GSoC’21 Phase-1 | Smartpens
This is an update blog for the project Smartpen @caMicroscope in GSoC’21.
About caMicroscope,
caMicroscope is a web-based biomedical image and data viewer, with a strong emphasis on cancer pathology WSI (Whole Slide Imaging).
About the project,
Smartpen, This project involves adapting pathology annotation tools to prefer following edges in the base image when close and also add other smart techniques that will make it look perfect. This tool helps the user to draw almost 100% accurate drawing/ annotation free-hand with the help of Computer Vision. The project also includes extending other annotation related facilities as well.
Work done,
Features implemented:
- Smartpen
- Moving Points
Basic design,
UI
UI is super simple, contains features for modes (after-draw, realtime), settings and undo. Its very user-friendly, contains drop-down buttons as well as blinkers.
Usage
Smartpen needs to be activated using the mode button ( leftmost ), to any desired mode out of after-draw and realtime modes. Both the modes are quite obvious as the name sounds,
- Realtime aligns in real-time as the user draws, (perfect drawings)
- After-draw aligns after drawing. (almost perfect)
Also, alignment can be undone if its not up to the mark. Right settings can also be selected for better working.
Details
Smartpen is extremely lightweight and blazingly fast.
It uses canny edge detection. I didn’t want to use Open-CV because its kind of heavy and non-customizable, so I wrote my own canny edge detection module (got multiple pieces of code snippets from multiple opensource repos). Apart from nearest edge selection, there are slight heuristics also which make the curves smooth and realistic. Making this feature real-time was another important task, this is also one of the reasons why computer vision was preferred. Earlier, it required too much time to process all the algorithms but after using collected real-time edge data it became amazingly fast.
Smartpen is available in the annotations and preset labels tab. But any utility using openseadragon-api can use it.
Further technical details about the feature can be found here.
Moving Points,
This is a feature which allows the user to drag annotations after drawing them to perfect the shape. This is similar to anchor points but in my implementation each drawn point is an anchor point.
To drag the figure, user has to simply click the annotation boundary and start moving it with the mouse.
Some limitations,
- Smartpen works perfectly when the user is close to the boundaries and the real-time mode is exactly perfect. But as the user moves away from the boundaries, accuracy starts to worsen.
- Currently, this feature is utilized by few tools.
Challenges faced,
- Thinking of various ways to optimize the smartpen and come up with novel techniques/features was a challenge.
- Implementation of edge detection (canny) was difficult and confusing. Anyway by googling and own implementation, I could finish it.
- Implementing moving points was relatively hard and required a bit of logic and math.
Acknowledgment,
I’m really thankful to my mentors: Insiyah Hajoori and Ryan Birmingham for their constant support and help. It was a really pleasant journey.