c-toxcore/docs/av_api.md

# A/V API reference

## Take toxmsi/phone.c as a reference

### Initialization:

```
phone_t* initPhone(uint16_t _listen_port, uint16_t _send_port);
```

function initializes sample phone. `_listen_port` and `_send_port` are variables
only meant for local testing. You will not have to do anything regarding to that
since everything will be started within a messenger.

Phone requires one msi session and two rtp sessions (one for audio and one for
video).

```
msi_session_t* msi_init_session( void* _core_handler, const uint8_t* _user_agent );
```

initializes msi session. Params:

```
void* _core_handler - pointer to an object handling networking,
const uint8_t* _user_agent - string describing phone client version.
```

Return value: `msi_session_t*` - pointer to a newly created msi session handler.

### `msi_session_t` reference:

How to handle msi session: Controlling is done via callbacks and action
handlers. First register callbacks for every state/action received and make sure
NOT TO PLACE SOMETHING LIKE LOOPS THAT TAKES A LOT OF TIME TO EXECUTE; every
callback is being called directly from event loop. You can find examples in
phone.c.

Register callbacks:

```
void msi_register_callback_call_started ( MCALLBACK );
void msi_register_callback_call_canceled ( MCALLBACK );
void msi_register_callback_call_rejected ( MCALLBACK );
void msi_register_callback_call_ended ( MCALLBACK );

void msi_register_callback_recv_invite ( MCALLBACK );
void msi_register_callback_recv_ringing ( MCALLBACK );
void msi_register_callback_recv_starting ( MCALLBACK );
void msi_register_callback_recv_ending ( MCALLBACK );
void msi_register_callback_recv_error ( MCALLBACK );

void msi_register_callback_requ_timeout ( MCALLBACK );
```

MCALLBACK is defined as: `void (*callback) (void* _arg)` `msi_session_t*`
handler is being thrown as `_arg` so you can use that and `_agent_handler` to
get to your own phone handler directly from callback.

Actions:

```
int msi_invite ( msi_session_t* _session, call_type _call_type, uint32_t _timeoutms );
```

Sends call invite. Before calling/sending invite `msi_session_t::_friend_id` is
needed to be set or else it will not work. `_call_type` is type of the call (
Audio/Video ) and `_timeoutms` is how long will poll wait until request is
terminated.

```
int msi_hangup ( msi_session_t* _session );
```

Hangs up active call

```
int msi_answer ( msi_session_t* _session, call_type _call_type );
```

Answer incoming call. `_call_type` set's callee call type.

```
int msi_cancel ( msi_session_t* _session );
```

Cancel current request.

```
int msi_reject ( msi_session_t* _session );
```

Reject incoming call.

### Now for rtp:

You will need 2 sessions; one for audio one for video. You start them with:

```
rtp_session_t* rtp_init_session ( int _max_users, int _multi_session );
```

Params:

```
int _max_users - max users. -1 if undefined
int _multi_session - any positive number means uses multi session; -1 if not.
```

Return value:

```
rtp_session_t* - pointer to a newly created rtp session handler.
```

### How to handle rtp session:

Take a look at

```
void* phone_handle_media_transport_poll ( void* _hmtc_args_p ) in phone.c
```

on example. Basically what you do is just receive a message via:

```
struct rtp_msg_s* rtp_recv_msg ( rtp_session_t* _session );
```

and then you use payload within the `rtp_msg_s` struct. Don't forget to
deallocate it with:
`void rtp_free_msg ( rtp_session_t* _session, struct rtp_msg_s* _msg );`
Receiving should be thread safe so don't worry about that.

When you capture and encode a payload you want to send it ( obviously ).

first create a new message with:

```
struct rtp_msg_s* rtp_msg_new ( rtp_session_t* _session, const uint8_t* _data, uint32_t _length );
```

and then send it with:

```
int rtp_send_msg ( rtp_session_t* _session, struct rtp_msg_s* _msg, void* _core_handler );
```

`_core_handler` is the same network handler as in `msi_session_s` struct.

## A/V initialization:

```
int init_receive_audio(codec_state *cs);
int init_receive_video(codec_state *cs);
Initialises the A/V decoders. On failure it will print the reason and return 0. On success it will return 1.

int init_send_audio(codec_state *cs);
int init_send_video(codec_state *cs);
Initialises the A/V encoders. On failure it will print the reason and return 0. On success it will return 1.
init_send_audio will also let the user select an input device. init_send_video will determine the webcam's output codec and initialise the appropriate decoder.

int video_encoder_refresh(codec_state *cs, int bps);
Reinitialises the video encoder with a new bitrate. ffmpeg does not expose the needed VP8 feature to change the bitrate on the fly, so this serves as a workaround.
In the future, VP8 should be used directly and ffmpeg should be dropped from the dependencies.
The variable bps is the required bitrate in bits per second.
```

### A/V encoding/decoding:

```
void *encode_video_thread(void *arg);
```

Spawns the video encoding thread. The argument should hold a pointer to a
`codec_state`. This function should only be called if video encoding is
supported (when `init_send_video` returns 1). Each video frame gets encoded into
a packet, which is sent via RTP. Every 60 frames a new bidirectional interframe
is encoded.

```
void *encode_audio_thread(void *arg);
```

Spawns the audio encoding thread. The argument should hold a pointer to a
`codec_state`. This function should only be called if audio encoding is
supported (when `init_send_audio` returns 1). Audio frames are read from the
selected audio capture device during initialisation. This audio capturing can be
rerouted to a different device on the fly. Each audio frame is encoded into a
packet, and sent via RTP. All audio frames have the same amount of samples,
which is defined in `AV_codec.h`.

```
int video_decoder_refresh(codec_state *cs, int width, int height);
```

Sets the SDL window dimensions and creates a pixel buffer with the requested
size. It also creates a scaling context, which will be used to convert the input
image format to YUV420P.

```
void *decode_video_thread(void *arg);
```

Spawns a video decoding thread. The argument should hold a pointer to a
`codec_state`. The `codec_state` is assumed to contain a successfully
initialised video decoder. This function reads video packets and feeds them to
the video decoder. If the video frame's resolution has changed,
`video_decoder_refresh()` is called. Afterwards, the frame is displayed on the
SDL window.

```
void *decode_audio_thread(void *arg);
```

Spawns an audio decoding thread. The argument should hold a pointer to a
`codec_state`. The `codec_state` is assumed to contain a successfully
initialised audio decoder. All received audio packets are pushed into a jitter
buffer and are reordered. If there is a missing packet, or a packet has arrived
too late, it is treated as a lost packet and the audio decoder is informed of
the packet loss. The audio decoder will then try to reconstruct the lost packet,
based on information from previous packets. Audio is played on the default
OpenAL output device.

If you have any more qustions/bug reports/feature request contact the following
users on the irc channel #tox-dev on irc.freenode.net: For RTP and MSI: mannol
For audio and video: Martijnvdc