Discussion:
[Numpy-discussion] Help with bit arrays
Henrique Almeida
2016-04-27 14:35:15 UTC
Permalink
Hello, what's the current status on numpy for loading bit-arrays ?

I'm currently unable to correctly load black and white (1-bit) TIFF
images. Code example follows:

from PIL import Image
import numpy
from matplotlib import pyplot

img = Image.open('oi-00.tiff')
a = numpy.array(img)

^ does not work for 1-bit TIFF images

PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
change this to '|t1', but I get :

TypeError: data type "|t1" not understood

My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
Henrique Almeida
2016-04-29 15:27:38 UTC
Permalink
Any help with this problem ?
Post by Henrique Almeida
Hello, what's the current status on numpy for loading bit-arrays ?
I'm currently unable to correctly load black and white (1-bit) TIFF
from PIL import Image
import numpy
from matplotlib import pyplot
img = Image.open('oi-00.tiff')
a = numpy.array(img)
^ does not work for 1-bit TIFF images
PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
TypeError: data type "|t1" not understood
My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
Paul Hobson
2016-04-29 16:06:24 UTC
Permalink
Does using pyplot.imgread work?
Post by Henrique Almeida
Any help with this problem ?
Post by Henrique Almeida
Hello, what's the current status on numpy for loading bit-arrays ?
I'm currently unable to correctly load black and white (1-bit) TIFF
from PIL import Image
import numpy
from matplotlib import pyplot
img = Image.open('oi-00.tiff')
a = numpy.array(img)
^ does not work for 1-bit TIFF images
PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
TypeError: data type "|t1" not understood
My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Henrique Almeida
2016-04-29 16:31:26 UTC
Permalink
Paul, yes, imread() worked for reading the black and white TIFF. The
situation improved, but now, there seems to be some problem with the
color map. Example code:

#!/usr/bin/env python3
import numpy
from matplotlib import pyplot, cm

img = pyplot.imread('oi-00.tiff')
pyplot.imshow(img)
pyplot.colorbar()
pyplot.show()

The code can open both 1-bit and 8-bit images, but only with 8 bits
the image is shown with the colormap colors. The 1 bit image is shown
as black and white.

The questions:
1) Should Image.open() behave like pyplot.imread() ? Is this a bug in PIL ?
2) Why isn't the colormap working with black and white images ?
Post by Paul Hobson
Does using pyplot.imgread work?
Post by Henrique Almeida
Any help with this problem ?
Post by Henrique Almeida
Hello, what's the current status on numpy for loading bit-arrays ?
I'm currently unable to correctly load black and white (1-bit) TIFF
from PIL import Image
import numpy
from matplotlib import pyplot
img = Image.open('oi-00.tiff')
a = numpy.array(img)
^ does not work for 1-bit TIFF images
PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
TypeError: data type "|t1" not understood
My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Benjamin Root
2016-04-29 16:38:57 UTC
Permalink
What kind of array is "img"? What is its dtype and shape?

plt.imshow() will use the default colormap for matplotlib if the given
array is just 2D. But if it is 3D (a 2D array of RGB[A] channels), then it
will forego the colormap and utilize that for the colors. It knows nothing
of the colormap contained in the TIFF.

Ben Root
Post by Henrique Almeida
Paul, yes, imread() worked for reading the black and white TIFF. The
situation improved, but now, there seems to be some problem with the
#!/usr/bin/env python3
import numpy
from matplotlib import pyplot, cm
img = pyplot.imread('oi-00.tiff')
pyplot.imshow(img)
pyplot.colorbar()
pyplot.show()
The code can open both 1-bit and 8-bit images, but only with 8 bits
the image is shown with the colormap colors. The 1 bit image is shown
as black and white.
1) Should Image.open() behave like pyplot.imread() ? Is this a bug in PIL ?
2) Why isn't the colormap working with black and white images ?
Post by Paul Hobson
Does using pyplot.imgread work?
Post by Henrique Almeida
Any help with this problem ?
Post by Henrique Almeida
Hello, what's the current status on numpy for loading bit-arrays ?
I'm currently unable to correctly load black and white (1-bit) TIFF
from PIL import Image
import numpy
from matplotlib import pyplot
img = Image.open('oi-00.tiff')
a = numpy.array(img)
^ does not work for 1-bit TIFF images
PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
TypeError: data type "|t1" not understood
My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Henrique Almeida
2016-04-29 16:43:06 UTC
Permalink
For 1 bit images, the resulting array has shape (256, 256, 4). For
grayscale images, the shape is (256, 256). So the image seems to have
been loaded as a color image.
Post by Benjamin Root
What kind of array is "img"? What is its dtype and shape?
plt.imshow() will use the default colormap for matplotlib if the given array
is just 2D. But if it is 3D (a 2D array of RGB[A] channels), then it will
forego the colormap and utilize that for the colors. It knows nothing of the
colormap contained in the TIFF.
Ben Root
Post by Henrique Almeida
Paul, yes, imread() worked for reading the black and white TIFF. The
situation improved, but now, there seems to be some problem with the
#!/usr/bin/env python3
import numpy
from matplotlib import pyplot, cm
img = pyplot.imread('oi-00.tiff')
pyplot.imshow(img)
pyplot.colorbar()
pyplot.show()
The code can open both 1-bit and 8-bit images, but only with 8 bits
the image is shown with the colormap colors. The 1 bit image is shown
as black and white.
1) Should Image.open() behave like pyplot.imread() ? Is this a bug in PIL ?
2) Why isn't the colormap working with black and white images ?
Post by Paul Hobson
Does using pyplot.imgread work?
On Fri, Apr 29, 2016 at 8:27 AM, Henrique Almeida
Post by Henrique Almeida
Any help with this problem ?
Post by Henrique Almeida
Hello, what's the current status on numpy for loading bit-arrays ?
I'm currently unable to correctly load black and white (1-bit) TIFF
from PIL import Image
import numpy
from matplotlib import pyplot
img = Image.open('oi-00.tiff')
a = numpy.array(img)
^ does not work for 1-bit TIFF images
PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
TypeError: data type "|t1" not understood
My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Henrique Almeida
2016-04-29 16:47:07 UTC
Permalink
I think in any case, the result is unexpected, PIL is loading garbage
from memory when loading black and white images because it sends the
wrong buffer size, and matplotlib correctly loads the black and white
image, but stores it in a 3D array.
Post by Henrique Almeida
For 1 bit images, the resulting array has shape (256, 256, 4). For
grayscale images, the shape is (256, 256). So the image seems to have
been loaded as a color image.
Post by Benjamin Root
What kind of array is "img"? What is its dtype and shape?
plt.imshow() will use the default colormap for matplotlib if the given array
is just 2D. But if it is 3D (a 2D array of RGB[A] channels), then it will
forego the colormap and utilize that for the colors. It knows nothing of the
colormap contained in the TIFF.
Ben Root
Post by Henrique Almeida
Paul, yes, imread() worked for reading the black and white TIFF. The
situation improved, but now, there seems to be some problem with the
#!/usr/bin/env python3
import numpy
from matplotlib import pyplot, cm
img = pyplot.imread('oi-00.tiff')
pyplot.imshow(img)
pyplot.colorbar()
pyplot.show()
The code can open both 1-bit and 8-bit images, but only with 8 bits
the image is shown with the colormap colors. The 1 bit image is shown
as black and white.
1) Should Image.open() behave like pyplot.imread() ? Is this a bug in PIL ?
2) Why isn't the colormap working with black and white images ?
Post by Paul Hobson
Does using pyplot.imgread work?
On Fri, Apr 29, 2016 at 8:27 AM, Henrique Almeida
Post by Henrique Almeida
Any help with this problem ?
Post by Henrique Almeida
Hello, what's the current status on numpy for loading bit-arrays ?
I'm currently unable to correctly load black and white (1-bit) TIFF
from PIL import Image
import numpy
from matplotlib import pyplot
img = Image.open('oi-00.tiff')
a = numpy.array(img)
^ does not work for 1-bit TIFF images
PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
TypeError: data type "|t1" not understood
My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Benjamin Root
2016-04-29 17:22:21 UTC
Permalink
What behavior is unexpected? For the (256, 256) images, matplotlib applies
its default colormap to the grayscale (v1.5 and previous, that is jet,
+v2.0, that will be viridis). The numpy array as loaded from PIL will never
carry any additional information that came from the TIFF.

As for PIL, it will return an RGB[A] array if there is colormap data in the
TIFF. If there is no colormap specified in the TIFF, it'll give you a
simple 2D array. Now, maybe you'd like it to always return an RGB[A] array,
but without a colormap in the TIFF, it makes sense to return the data
as-is. This makes sense for people treating the TIFF as a data format
rather than a visualization data format.

Ben Root
Post by Henrique Almeida
I think in any case, the result is unexpected, PIL is loading garbage
from memory when loading black and white images because it sends the
wrong buffer size, and matplotlib correctly loads the black and white
image, but stores it in a 3D array.
Post by Henrique Almeida
For 1 bit images, the resulting array has shape (256, 256, 4). For
grayscale images, the shape is (256, 256). So the image seems to have
been loaded as a color image.
Post by Benjamin Root
What kind of array is "img"? What is its dtype and shape?
plt.imshow() will use the default colormap for matplotlib if the given
array
Post by Henrique Almeida
Post by Benjamin Root
is just 2D. But if it is 3D (a 2D array of RGB[A] channels), then it
will
Post by Henrique Almeida
Post by Benjamin Root
forego the colormap and utilize that for the colors. It knows nothing
of the
Post by Henrique Almeida
Post by Benjamin Root
colormap contained in the TIFF.
Ben Root
On Fri, Apr 29, 2016 at 12:31 PM, Henrique Almeida <
Post by Henrique Almeida
Paul, yes, imread() worked for reading the black and white TIFF. The
situation improved, but now, there seems to be some problem with the
#!/usr/bin/env python3
import numpy
from matplotlib import pyplot, cm
img = pyplot.imread('oi-00.tiff')
pyplot.imshow(img)
pyplot.colorbar()
pyplot.show()
The code can open both 1-bit and 8-bit images, but only with 8 bits
the image is shown with the colormap colors. The 1 bit image is shown
as black and white.
1) Should Image.open() behave like pyplot.imread() ? Is this a bug in
PIL
Post by Henrique Almeida
Post by Benjamin Root
Post by Henrique Almeida
?
2) Why isn't the colormap working with black and white images ?
Post by Paul Hobson
Does using pyplot.imgread work?
On Fri, Apr 29, 2016 at 8:27 AM, Henrique Almeida
Post by Henrique Almeida
Any help with this problem ?
Post by Henrique Almeida
Hello, what's the current status on numpy for loading bit-arrays
?
Post by Henrique Almeida
Post by Benjamin Root
Post by Henrique Almeida
Post by Paul Hobson
Post by Henrique Almeida
Post by Henrique Almeida
I'm currently unable to correctly load black and white (1-bit)
TIFF
Post by Henrique Almeida
Post by Benjamin Root
Post by Henrique Almeida
Post by Paul Hobson
Post by Henrique Almeida
Post by Henrique Almeida
from PIL import Image
import numpy
from matplotlib import pyplot
img = Image.open('oi-00.tiff')
a = numpy.array(img)
^ does not work for 1-bit TIFF images
PIL source shows that it incorrectly uses typestr == '|b1'. I
tried
Post by Henrique Almeida
Post by Benjamin Root
Post by Henrique Almeida
Post by Paul Hobson
Post by Henrique Almeida
Post by Henrique Almeida
to
TypeError: data type "|t1" not understood
My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Henrique Almeida
2016-04-29 17:42:48 UTC
Permalink
I agree with everything, but the unexpected are something else.

The TIFF images have no colormap. The colormap that I'm referring to
is the GUI colormap, used by matplotlib to draw the image (imshow
parameter cmap).

The problematic image format is the black and white 1-bit TIFF
format. It is a bit array format, all bits are packed in sequence.

PIL passes this kind of image to numpy through the array_interface
getter, which returns an array description of shape = (256, 256), type
string "|b1" and data is a 8192 byte array (256 * 256 * 1 bit). This
description is invalid and causes numpy to load 65536 bytes from
memory, causing a buffer overflow (even though it does not crash).
This is unexpected #1.

matplotlib.imread(), when loading a 8-bit grayscale image creates an
array of shape (256, 256). matplotlib.imread(), when loading a 1-bit
black and white image creates an array of shape (256, 256, 4) by first
converting the black and white image to RGBA. This difference between
grayscale and black and white is unexpected #2.
Post by Benjamin Root
What behavior is unexpected? For the (256, 256) images, matplotlib applies
its default colormap to the grayscale (v1.5 and previous, that is jet,
+v2.0, that will be viridis). The numpy array as loaded from PIL will never
carry any additional information that came from the TIFF.
As for PIL, it will return an RGB[A] array if there is colormap data in the
TIFF. If there is no colormap specified in the TIFF, it'll give you a simple
2D array. Now, maybe you'd like it to always return an RGB[A] array, but
without a colormap in the TIFF, it makes sense to return the data as-is.
This makes sense for people treating the TIFF as a data format rather than a
visualization data format.
Ben Root
Post by Henrique Almeida
I think in any case, the result is unexpected, PIL is loading garbage
from memory when loading black and white images because it sends the
wrong buffer size, and matplotlib correctly loads the black and white
image, but stores it in a 3D array.
Post by Henrique Almeida
For 1 bit images, the resulting array has shape (256, 256, 4). For
grayscale images, the shape is (256, 256). So the image seems to have
been loaded as a color image.
Post by Benjamin Root
What kind of array is "img"? What is its dtype and shape?
plt.imshow() will use the default colormap for matplotlib if the given array
is just 2D. But if it is 3D (a 2D array of RGB[A] channels), then it will
forego the colormap and utilize that for the colors. It knows nothing of the
colormap contained in the TIFF.
Ben Root
On Fri, Apr 29, 2016 at 12:31 PM, Henrique Almeida
Post by Henrique Almeida
Paul, yes, imread() worked for reading the black and white TIFF. The
situation improved, but now, there seems to be some problem with the
#!/usr/bin/env python3
import numpy
from matplotlib import pyplot, cm
img = pyplot.imread('oi-00.tiff')
pyplot.imshow(img)
pyplot.colorbar()
pyplot.show()
The code can open both 1-bit and 8-bit images, but only with 8 bits
the image is shown with the colormap colors. The 1 bit image is shown
as black and white.
1) Should Image.open() behave like pyplot.imread() ? Is this a bug in
PIL
?
2) Why isn't the colormap working with black and white images ?
Post by Paul Hobson
Does using pyplot.imgread work?
On Fri, Apr 29, 2016 at 8:27 AM, Henrique Almeida
Post by Henrique Almeida
Any help with this problem ?
2016-04-27 11:35 GMT-03:00 Henrique Almeida
Post by Henrique Almeida
Hello, what's the current status on numpy for loading bit-arrays ?
I'm currently unable to correctly load black and white (1-bit) TIFF
from PIL import Image
import numpy
from matplotlib import pyplot
img = Image.open('oi-00.tiff')
a = numpy.array(img)
^ does not work for 1-bit TIFF images
PIL source shows that it incorrectly uses typestr == '|b1'. I
tried
to
TypeError: data type "|t1" not understood
My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Loading...