Listens to the file system change notifications and raises events when a directory, or file in a directory, changes.
File system watcher listens to the file system change notifications and raises events when a directory, or file in a directory, changes. A very useful tool since it notifies exactly at the moment when a file or directory is created, changed or deleted, without having to do polling on that directory. It can watch sub-directories too.
If that wasn’t enough it also provides filters. For instance u just want notifications about text files then you can provide a filter “*.txt”, now event will be raised only for text files.
Default size of the buffer is 4 KB, so obviously the immediate solution would be to increase this size. But again there is a downside the buffer is located in a portion of memory which cannot be swapped, in other words the memory will be allocated by default whether it is use or not. So it not advisable to not allocate large memory sizes.
Now the buffer discussed above holds the information for each event until the event is handled. The idea is to handle the event as quickly as possible so that the buffer is cleared.
But that’s easier said than done. To do some complex processing on the file would take some time, some CPU cycles, and the buffer will have to hold the data till the processing is over.
So instead of we do not write the proceeding logic in the handler instead the handler simply queues the file in a work queue. This queue is processed by an entirely independent thread. So the actual work done by the handler is just queuing and the control moves out of the handler thus releasing the buffer.
File system watcher listens to the file system change notifications and raises events when a directory, or file in a directory, changes. A very useful tool since it notifies exactly at the moment when a file or directory is created, changed or deleted, without having to do polling on that directory. It can watch sub-directories too.
If that wasn’t enough it also provides filters. For instance u just want notifications about text files then you can provide a filter “*.txt”, now event will be raised only for text files.
The Problem
However there is a downside. The file system watcher is not entirely reliable for working with large volumes of files. The reason for this is that there is a fixed buffer allocated to each file system watcher which is used to store the details such as file location for each file that raises an event. However when a large number of files raise an event then this buffer gets full.Default size of the buffer is 4 KB, so obviously the immediate solution would be to increase this size. But again there is a downside the buffer is located in a portion of memory which cannot be swapped, in other words the memory will be allocated by default whether it is use or not. So it not advisable to not allocate large memory sizes.
The Solution
If you Google for the solution to this problem you will find a large number of articles. In this article however we take an entirely different approach.Now the buffer discussed above holds the information for each event until the event is handled. The idea is to handle the event as quickly as possible so that the buffer is cleared.
But that’s easier said than done. To do some complex processing on the file would take some time, some CPU cycles, and the buffer will have to hold the data till the processing is over.
So instead of we do not write the proceeding logic in the handler instead the handler simply queues the file in a work queue. This queue is processed by an entirely independent thread. So the actual work done by the handler is just queuing and the control moves out of the handler thus releasing the buffer.
No comments:
Post a Comment