Socket event epoll blocking in linux
Table of Contents
Whether server or client-side, the same basic thing is going to happen: send and receive data to and from at least a socket. I won't go explain how as there are plenty of guides covering that stuff around, most notably Beej's Guide to Network Programming (pretty much the standard).
There is one issue that will inevitably come up when implementing a polling system: blocking.
1. Setting up epoll
Constantly checking a socket for data will incur wasted CPU cycles on the thread that runs it. It would be better to wait for a signal and then resume operations then. This is where polling comes in with its ability to block and only resume on specified events.
First, we got to make sure the file descriptor used is set to non-blocking so we can "poll" it. In the case of multiple clients server-side, each client file descriptors need to be set.
::fcntl( client_fd, F_SETFL, O_NONBLOCK );
Creating a new epoll returns its file descriptor (or -1 on error). Note that the queue length argument is ignored since Linux 2.6.8.
int epoll_fd = ::epoll_create( EPOLL_PENDING_QUEUE_LENGTH );
All file descriptors that we want to keep an eye on for any activity must be added to the epoll queue.
struct epoll_event event = {};
event.events = EPOLLIN;
event.data.fd = fd; //this way we can retrieve the file descriptor via the event struct
if( ::epoll_ctl( epoll_fd, EPOLL_CTL_ADD, client_fd, &event ) < 0 ) {
//error
}
The second argument is the operation to perform with the given file descriptor:
EPOLL_CTL_ADD- Adds the file descriptor to the given epoll's watch list and associates it with the
eventgiven. EPOLL_CTL_MOD- Changes the associated event of the file descriptor with the new
eventgiven. EPOLL_CTL_DEL- Removes the file descriptor from the given epoll's watch list
To wait on events it's a simple matter of using epoll_wait and iterating through whatever events may have been triggered.
struct epoll_event event_buff[EPOLL_ARRAY_SIZE];
int event_count = epoll_wait( _epoll_fd, event_buff, EPOLL_ARRAY_SIZE, -1 );
for( int i = 0; i < event_count; ++i ) {
int client_fd = event_buff[i].data.fd; //since we set the client fd there previously
//deal with event
}
The last argument of epoll_wait is the timeout before unblocking - even if there are no events. A "-1" value sets it to indefinite so, unless an event is signaled, epoll_wait will stay blocked.
That concludes the basics for getting "epoll" off the ground. See the linux manual page for "epoll" for more.
2. Solution to blocking issue
Now onto the meat of the matter, and you might already see where I'm going with this... If epoll_wait is blocked and you want to exit the application cleanly, how the hell can you work around that?
There are 3 possible approaches outside of force-killing the application.
- Using
epoll_pwaitwith an interrupt mask (good for forwarding interrupt signals likeSIGINT,SIGTERM, etc...), - Using the "self-pipe" trick, or
- The event file descriptor.
The "self-pipe" trick uses a pipe (unidirectional data channel that can be used for interprocess communication
) whose "read" file descriptor end is added to the epoll watch list. This facilitates unblocking epoll_wait by sending some data to the "write" end of the pipe.
A simpler and much cleaner way to have that behaviour is eventfd. This creates a special file descriptor that only holds a uint64_t. This has the advantage of requiring only 1 file descriptor to do both the writing and reading unlike a pipe which has 1 for each.
if( ( _unblock_event_fd = ::eventfd( 0, EFD_NONBLOCK ) ) == -1 ) {
//error
}
The idea is to add an eventfd to our epoll watch list and write to it from another thread like 'main' to unblock it so that the thread in which epoll_wait and the event processing can be exited gracefully.
struct epoll_event event_buff[EPOLL_ARRAY_SIZE];
int event_count = epoll_wait( _epoll_fd, event_buff, EPOLL_ARRAY_SIZE, -1 );
for( int i = 0; i < event_count; ++i ) {
if( event_buff[i].data.fd == _unblock_event_fd ) {
//clean up/gracefully exit
} else {
//deal with event
}
}
And voila!