Watch what happens when you do:

Case3={"id":3,
       "interp":"none",
       "extents":(1,200,1,100)}

the image is shrunk horizontally and centered, when it should remain fully covering the axes. I think the bug is most likely in _draw_unsampled_image not taking into account non-real extents, but I don't yet have a solution. @leejjoon, I believe, is the original author of that code -- maybe he has some idea.

That Case3 looks like your viewer is trying to keep the aspect ratio somehow. Indeed this here fixes the problem for my default SVG gnome viewer (so cairo in general?):

diff --git a/lib/matplotlib/backends/backend_svg.py b/lib/matplotlib/backends/backend_svg.py
index 0b9b7ab..a4b72c9 100644
--- a/lib/matplotlib/backends/backend_svg.py
+++ b/lib/matplotlib/backends/backend_svg.py
@@ -801,6 +801,7 @@ class RendererSVG(RendererBase):
                 y *= -1.0
             attrib[u'transform'] = generate_transform(
                 [(u'matrix', flipped)])
+            attrib['preserveAspectRatio'] = "none"
             self.writer.element(
                 u'image',
                 x=unicode(x), y=unicode(y),

The bug is still shows up in Firefox and Chrome. Also the original post states that this problem highly depends on the viewer :/

And evaluating (x, y, height, width) in python instead of having the SVG viewers interpret the transformation fixes it for all my applications...

diff --git a/lib/matplotlib/backends/backend_svg.py b/lib/matplotlib/backends/backend_svg.py
index 0b9b7ab..f300b4a 100644
--- a/lib/matplotlib/backends/backend_svg.py
+++ b/lib/matplotlib/backends/backend_svg.py
@@ -793,18 +793,14 @@ class RendererSVG(RendererBase):
                 width=unicode(w), height=unicode(h),
                 attrib=attrib)
         else:
-            flipped = self._make_flip_transform(transform)
-            flipped = np.array(flipped.to_values())
-            y = y+dy
-            if dy > 0.0:
-                flipped[3] *= -1.0
-                y *= -1.0
-            attrib[u'transform'] = generate_transform(
-                [(u'matrix', flipped)])
+            x, y = transform.transform_point((x,y))
+            scale_x, _, _, scale_y, _, _ = transform.to_values()
+            w, h = dx * scale_x, dy * scale_y
+            attrib['preserveAspectRatio'] = "none"
             self.writer.element(
                 u'image',
-                x=unicode(x), y=unicode(y),
-                width=unicode(dx), height=unicode(abs(dy)),
+                x=unicode(x), y=unicode(self.height-y-h),
+                width=unicode(w), height=unicode(h),
                 attrib=attrib)

         if url is not None:

I suspect the essence of this fix can and should be in image.py so that it doesn't have to be repeated in each backend.

With interpolation="none", we cannot avoid using the matrix in the backend, and I doubt if pwuertz's solution will work for rotated and/or skewed images.
On the other hand, I think what we should try is a similar approach so that the matrix only contains rotation and skewing (i.e., matplotlib pre-calculates translation and scaling), and I think we can do this in image.py as Eric suggested. I will try to give a more look later.

It wouldn't have worked with any transformation but scaling, just wanted to share the observation that numerically the transformations shouldn't be doing stuff that exceeds a few orders of magnitudes as this seems to freak out some viewers.

I think above PR seem to resolve the issue. Please test it and see if it works.

@technobauble : Does the mentioned PR solve the issue for you?

This appears to solve the problem for me in both the test cases that I posted and the actual application that I am developing.

Thank you everyone for your time and the promptness of the fix.