the poc in the maptools report seems to be just a copy of the report.

I have requested a test case from the reporter in upstream bug.

This looks very similar to bug 983440.

BEFORE

4.0.9

$ valgrind -q bmp2tiff -c packbits CVE-2016-5319.bmp out.tiff
CVE-2016-5319.bmp: Premature end of file.
$

$ valgrind -q bmp2tiff -c packbits cve-2017-17942-poc out.tiff
==19076== Invalid read of size 1
==19076==    at 0x4E858D6: PackBitsEncode (tif_packbits.c:88)
==19076==    by 0x4E95CD4: TIFFWriteScanline (tif_write.c:173)
==19076==    by 0x10ACE8: main (bmp2tiff.c:783)
==19076==  Address 0x62446d0 is 0 bytes after a block of size 1,024 alloc'd
==19076==    at 0x4C29110: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==19076==    by 0x4E98B63: _TIFFmalloc (tif_unix.c:316)
==19076==    by 0x10A601: main (bmp2tiff.c:678)
==19076== 
==19076== Invalid read of size 1
==19076==    at 0x4E858A4: PackBitsEncode (tif_packbits.c:85)
==19076==    by 0x4E95CD4: TIFFWriteScanline (tif_write.c:173)
==19076==    by 0x10ACE8: main (bmp2tiff.c:783)
==19076==  Address 0x62446d0 is 0 bytes after a block of size 1,024 alloc'd
==19076==    at 0x4C29110: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==19076==    by 0x4E98B63: _TIFFmalloc (tif_unix.c:316)
==19076==    by 0x10A601: main (bmp2tiff.c:678)
==19076== 
$

3.8.2

$ valgrind -q bmp2tiff CVE-2016-5319.bmp out.tiff
==19123== Conditional jump or move depends on uninitialised value(s)
==19123==    at 0x4E624ED: PackBitsEncode (tif_packbits.c:90)
==19123==    by 0x4E704E8: TIFFWriteScanline (tif_write.c:167)
==19123==    by 0x402363: main (bmp2tiff.c:759)
==19123== 
==19123== Syscall param write(buf) points to uninitialised byte(s)
==19123==    at 0x57F0F30: write (in /lib64/libc-2.9.so)
==19123==    by 0x4E6F975: _tiffWriteProc (tif_unix.c:64)
==19123==    by 0x4E71ABD: TIFFAppendToStrip (tif_write.c:680)
==19123==    by 0x4E71C1D: TIFFFlushData1 (tif_write.c:703)
==19123==    by 0x4E4BD75: TIFFFlushData (tif_flush.c:65)
==19123==    by 0x4E7027F: TIFFWriteScanline (tif_write.c:105)
==19123==    by 0x402363: main (bmp2tiff.c:759)
==19123==  Address 0x2a0c7031 is 1 bytes inside a block of size 9,474,192 alloc'd
==19123==    at 0x4C256AE: malloc (in /usr/lib64/valgrind/amd64-linux/vgpreload_memcheck.so)
==19123==    by 0x4E6FBF9: _TIFFmalloc (tif_unix.c:244)
==19123==    by 0x4E7147D: TIFFWriteBufferSetup (tif_write.c:571)
==19123==    by 0x4E70105: TIFFWriteScanline (tif_write.c:63)
==19123==    by 0x402363: main (bmp2tiff.c:759)
==19123== 
==19123== More than 10000000 total errors detected.  I'm not reporting any more.
==19123== Final error counts will be inaccurate.  Go fix your program!
==19123== Rerun with --error-limit=no to disable this cutoff.  Note
==19123== that errors may occur in your program without prior warning from
==19123== Valgrind, because errors are no longer being displayed.
==19123== 
$

$ valgrind -q bmp2tiff -c packbits cve-2017-17942-poc out.tiff
$ file out.tiff
out.tiff: TIFF image data, little-endian
$

(the testcase can be found at:
 https://github.com/xiaosatianyu/for-cve/blob/master/collect_poc/libtiff4.0.8/bmp2tiff/cve-2017-17942/cve-2017-17942-poc)

I think the issue lies in [1]

        } else if ( info_hdr.iCompression == BMPC_RLE8
                    || info_hdr.iCompression == BMPC_RLE4 ) {
           [..]
        }

branch of bmp2tiff.c. There is uncomprbuf allocated to handle uncompressed data, but only just width * length of size.

               uncompr_size = width * length;
               uncomprbuf = (unsigned char *)_TIFFmalloc(uncompr_size);

At the end of the branch [1], the buffer is read via for loop:

                for (row = 0; row < length; row++) {
                        if (TIFFWriteScanline(out,
                                  uncomprbuf + (length - row - 1) * width,
                                  row, 0) < 0) {
                            TIFFError(infilename,
                                      "scanline %lu: Write error.\n",
                                      (unsigned long) row);
                        }
                }


That would be ok, when info_hdr.iBitCount read from BMP header would be less than 16. However, info_hdr.iBitCount == 24, which translates to nbands == 3 and therefore TIFFSetField(out, TIFFTAG_SAMPLESPERPIXEL, 3). Now TIFFWriteScanline() in the loop above wants to read three times more than width, not just width, hence the buffer over-read.

We see that even comment reveals that info_hdr.iBitCount equal or more than 16 is not taken in [1] branch into account at all:

     if (info_hdr.iBitCount == 8) {          /* RLE8 */
         [..]
     }
     else {                              /* RLE4 */
         [..]
     }

I dare therefore to catch this and error out from the tool in this case, as I do not know tiff format to the extent I could say how the tool should behave better.

The issue lies with high probability in bmp2tiff.c tool itself, not in the library.

Also, conclusions from comment 6 seem to align with http://bugzilla.maptools.org/show_bug.cgi?id=2562 findings.

PATCH

12/tiff: I will create patch to achieve the behavior described at the end of comment 6.
10sp3,11/tiff: I will update bmp2tiff.c to the version we have in 12/tiff.

AFTER

10sp3,11,12/tiff:

$ valgrind -q bmp2tiff -c packbits CVE-2016-5319.bmp out.tiff
CVE-2016-5319.bmp: Handle of this type image is not implemented.
$

$ valgrind -q bmp2tiff cve-2017-17942-poc out.tiff
cve-2017-17942-poc: Handle of this type image is not implemented.
$

Will submit for 12/tiff, 11/tiff and 10sp3/tiff.

SUSE-SU-2018:2676-1: An update that fixes four vulnerabilities is now available.

Category: security (moderate)
Bug References: 1074186,1092480,960589,983440
CVE References: CVE-2015-8668,CVE-2016-5319,CVE-2017-17942,CVE-2018-10779
Sources used:
SUSE Linux Enterprise Software Development Kit 11-SP4 (src):    tiff-3.8.2-141.169.16.1
SUSE Linux Enterprise Server 11-SP4 (src):    tiff-3.8.2-141.169.16.1
SUSE Linux Enterprise Debuginfo 11-SP4 (src):    tiff-3.8.2-141.169.16.1

SUSE-SU-2018:2836-1: An update that fixes three vulnerabilities is now available.

Category: security (moderate)
Bug References: 1074186,1092480,983440
CVE References: CVE-2016-5319,CVE-2017-17942,CVE-2018-10779
Sources used:
SUSE Linux Enterprise Software Development Kit 12-SP3 (src):    tiff-4.0.9-44.21.1
SUSE Linux Enterprise Server 12-SP3 (src):    tiff-4.0.9-44.21.1
SUSE Linux Enterprise Desktop 12-SP3 (src):    tiff-4.0.9-44.21.1

openSUSE-SU-2018:2880-1: An update that fixes three vulnerabilities is now available.

Category: security (moderate)
Bug References: 1074186,1092480,983440
CVE References: CVE-2016-5319,CVE-2017-17942,CVE-2018-10779
Sources used:
openSUSE Leap 42.3 (src):    tiff-4.0.9-34.1

SUSE-SU-2018:3879-1: An update that fixes 11 vulnerabilities is now available.

Category: security (moderate)
Bug References: 1010163,1014461,1040080,1040322,1074186,1099257,1113672,974446,974447,974448,983440
CVE References: CVE-2015-8870,CVE-2016-3619,CVE-2016-3620,CVE-2016-3621,CVE-2016-5319,CVE-2016-9273,CVE-2017-17942,CVE-2017-9117,CVE-2017-9147,CVE-2018-12900,CVE-2018-18661
Sources used:
SUSE Linux Enterprise Software Development Kit 11-SP4 (src):    tiff-3.8.2-141.169.22.1
SUSE Linux Enterprise Server 11-SP4 (src):    tiff-3.8.2-141.169.22.1
SUSE Linux Enterprise Debuginfo 11-SP4 (src):    tiff-3.8.2-141.169.22.1

released