I would like to scan and analyze recieipts to sprung more data. First step is reliable image scanning. This has definitely been done before however not in JavaScript. Plan of attack: build a scanner in NodeJS, then port it into browser application.

I played around with OpenCV’s cross-compile into the browser the other day and had reasonable luck. Definitely some translation which needs to occur, in addtion to custom building the script.

OpenCV.js

Buidling the WebASM port would have taken hours or at least that is what I assumed. So I grabbed the build they have on the tutorial site. The live examples are executed via iframes. The entire bundle size is 3.3MB for 4.1.2 however this isn’t minified or compressed in any way. The file is actually an AMD. This successfully loads, although takes a brief period of time to load with NodeJS.

Turns out trying to use this with NodeJS results in a number of explosions. In an effort to avoid explosiosn I am going back to using the straight browser. Perhaps I’ll have more luck outside of my primary application environment!

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Recongizer</title>
    <script src="opencv.js"></script>
    <script>
function onReady(){ 
	if( !window.cv || !window.cv.Mat ){ // (2)
		console.log("Waiting for OpenCV to finish loading...");
		setTimeout(onReady, 250);
		return;
    }
	console.log("Ready");
	loadImage("test.jpg", (img) => { // (3)
		const material = cv.imread(img);
    });
}

function loadImage( url, next ) {
	const img = new Image();
	img.onload = (e) => {
		console.log("Loaded...",e);

		next(img);
	};
	img.src = url;
}

document.addEventListener("DOMContentLoaded",onReady); // (1)
    </script>
</head>
<body>

</body>
</html>

This skelaton will load an image into OpenCV. Some notes:

  1. We exectue this after the the document has finished loading. Note that this does not mean the we’re entirely initialized, just the initial DOM setup is complete. OpenCV is not yet done loading.
  2. OpenCV takes about a second to initialize. Probably not a problem on an interact site or a SPA however in this example the cv.Mat object does not exist for a short period of time. This will result in a complaint from cv.imread Uncaught TypeError: cv.Mat is not a constructor nondeterminstically.
  3. We should be ready right after our source image is loaded.

Following the existing example, we need to resize our image. The particiluar example image being worked has a lot of detail. A great torture test to see what can be extracted, so a larger 1024px size is desired. There is quiet a bit mroe which should be deleted, such as resizedImage, if this was production code.

    const originalImage = cv.imread(img);
    try {
        const ratioChange = 1024 / originalImage.rows ;
        const resizedImge = new cv.Mat();
        const dsize = new cv.Size(originalImage.cols * ratioChange, originalImage.rows * ratioChange);
        cv.resize(originalImage, resizedImge, dsize, 0, 0, cv.INTER_AREA);

        appendImage(resizedImge);
    }finally {
        originalImage.delete();
    }

Next up is downsampling to a grayscale image:

    const gray = new cv.Mat();
    cv.cvtColor(resizedImge, gray, cv.COLOR_RGBA2GRAY, 0);
    appendImage(gray);

Then extract only the edges of the object:

    //Edge detect
    const blur = new cv.Mat();
    let ksize = new cv.Size(3, 3);
    cv.GaussianBlur(gray, blur, ksize, 0, 0, cv.BORDER_CONSTANT);
    const edgeMap = new cv.Mat();
    cv.Canny(blur, edgeMap, 75, 196, 3, true);
    appendImage(edgeMap);

In the example text they use a python filter to sort an entire array. Really we’re only looking for the largest, so I just manually iterated. Ideally we could use a reduce here, but meh.

    //Find contours
    let contours = new cv.MatVector();
    let hierarchy = new cv.Mat();
    cv.findContours(edgeMap, contours, hierarchy, cv.RETR_LIST, cv.CHAIN_APPROX_SIMPLE);
    let largest, largestArea;
    for( let i = 0 ; i < contours.size(); i++ ){
        const contour = contours.get(i);
        const area = cv.arcLength(contour, false);
    
        function update() {
            console.log("Using ", i);
            largest = contour;
            largestArea = area;
        }
        if( !largest ){ update() }
        if( area > largestArea ){ update() }
    }

To verify the image, let’s draw a bounding rectangle appropriatly rotated to capture the document:

    let rotatedRect = cv.minAreaRect(largest);
    let vertices = cv.RotatedRect.points(rotatedRect);
    let contoursColor = new cv.Scalar(255, 255, 255);
    let rectangleColor = new cv.Scalar(255, 0, 0);
    // draw rotatedRect
    for (let i = 0; i < 4; i++) {
        cv.line(resizedImge, vertices[i], vertices[(i + 1) % 4], rectangleColor, 2, cv.LINE_AA, 0);
    }

Alright. Well, this got me a rectangle. The image I’m using is definitely parallax. I’ve gotten it closer with adding a threshold function, however I am not able to get the bent corner of my image to display.