Debian Bug report logs - #748474
mk-origtargz: Please allow extra actions

Reply or subscribe to this bug.

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: David Prévot <taffit@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: mk-origtargz: Please allow extra actions

Date: Sat, 17 May 2014 11:15:11 -0400

[Message part 1 (text/plain, inline)]

Package: devscripts
Version: 2.14.2
Severity: wishlist
User: devscripts@packages.debian.org
Usertags: mk-origtargz

Hi,

Thanks for aiming at providing a reliable way to clean up upstream
tarballs with regard to non-DFSG compliant material.

It would be nice if one could also alter specific files within the
upstream tarball, e.g., by removing nonfree ICC profiles from images as
recently discussed on debian-devel. [0]

To do so, it would be convenient to execute arbitrary code during the
repack, e.g. via a --command switch allowing to execute things like:

	exiftool -icc_profile"-<=" $listoffiles

where $listoffiles would refer to some files within the upstream
tarball.

The sound of “execute arbitrary code during the repack” makes me feel
uneasy, but I’d like to make it possible for such argument to the
--command switch to be defined into the debian/ directory (e.g., a
debian/upstream/repack.commands file), and possibly deactivate this
behavior by default if there are security concerns (one then would need
to call mk-origtargz and uscan with a --i-trust-the-debian-dir switch in
order to take this file into account).

	0: https://lists.debian.org/debian-devel/2014/05/thrd2.html#00312

Regards

David

-- Package-specific info:

--- /etc/devscripts.conf ---

--- ~/.devscripts ---
DEBSIGN_KEYID=0xB82A217AFDFE09F2

-- System Information:
Debian Release: jessie/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (500, 'oldstable'), (100, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-1-rt-amd64 (SMP w/1 CPU core; PREEMPT)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages devscripts depends on:
ii  dpkg-dev     1.17.9
ii  libc6        2.18-6
ii  perl         5.18.2-4
ii  python3      3.3.4-1
pn  python3:any  <none>

Versions of packages devscripts recommends:
ii  at                          3.1.14-1
ii  curl                        7.36.0-2
ii  dctrl-tools                 2.23
ii  debian-keyring              2014.04.25
ii  dput                        0.9.6.4
ii  equivs                      2.0.9
ii  fakeroot                    1.20-3
ii  file                        1:5.18-1
ii  gnupg                       1.4.16-1.1
ii  libdistro-info-perl         0.13
ii  libencode-locale-perl       1.03-1
ii  libjson-perl                2.61-1
ii  liblwp-protocol-https-perl  6.04-3
ii  libparse-debcontrol-perl    2.005-4
ii  libsoap-lite-perl           1.11-1
ii  liburi-perl                 1.60-1
ii  libwww-perl                 6.06-1
ii  lintian                     2.5.22.1
ii  man-db                      2.6.7.1-1
ii  patch                       2.7.1-5
ii  patchutils                  0.3.3-1
ii  python3-debian              0.1.21+nmu3
ii  python3-magic               1:5.18-1
ii  sensible-utils              0.0.9
ii  strace                      4.5.20-2.3
ii  unzip                       6.0-12
ii  wdiff                       1.2.1-3
ii  wget                        1.15-1
ii  xz-utils                    5.1.1alpha+20120614-2

Versions of packages devscripts suggests:
ii  bsd-mailx [mailx]            8.1.2-0.20131005cvs-1
ii  build-essential              11.6
ii  cvs-buildpackage             5.23
pn  devscripts-el                <none>
ii  gnuplot                      4.6.5-1
ii  gpgv                         1.4.16-1.1
ii  libauthen-sasl-perl          2.1600-1
ii  libfile-desktopentry-perl    0.07-1
ii  libnet-smtp-ssl-perl         1.01-3
ii  libterm-size-perl            0.207-1+b1
ii  libtimedate-perl             2.3000-2
ii  libyaml-syck-perl            1.27-2+b1
ii  mutt                         1.5.23-1
ii  openssh-client [ssh-client]  1:6.6p1-5
pn  svn-buildpackage             <none>
ii  w3m                          0.5.3-15

-- no debconf information

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 748474@bugs.debian.org (full text, mbox, reply):

From: Osamu Aoki <osamu@debian.org>

To: 748474@bugs.debian.org, David Prévot <taffit@debian.org>

Subject: RE: mk-origtargz: Please allow extra actions

Date: Sun, 27 Sep 2015 00:24:39 +0900

Hi David,

I have been working on uscan related topic recently :-).  Since it is a
bit rough, my latest work is living in a branch off alioth's main branch
(multitar).  So far, multiple upstream tarball, separate URL search for
SIG file, better uupdate integration have been done.

As I see https://bugs.debian.org/748474 , this seems to be something
easily handled by the released code as is. (Maybe there is some corner
case where we may need to address if you wish to use command such as
"gbp import-orig --uscan" which I have not looked into yet.

Just create a script debian/uupdate and create watch file like

version=3
http://www.example.org/downloadguide \
        foo/bar-([\d.]+).tar.xz debian debian/uupdate

The content of debian/uupdate can be:

-------------------
#!/bin/sh
# Assume recent uscan v=3 "--upstream-version <version> <path>"
while [ "$1" ]; do
    case $1 in
    --upstream-version|-v)
        shift; NEW_VERSION="$1"
        ;;
    --force-bad-version|-b|--find|-f)
        :
        ;;
    esca
    shift
done
# ignore <path> and rely on ../ being the path to the orig.tar.xz
# using source package name $SPKG from debian/changelog
# This makes it more robust for newer uupdate in future
SPKG=$(dpkg-parsechangelog -SSource)
cd ..
# Now ${SPKG}_${NEW_VERSION}.orig.tar.xz is upstream tarball
mv -f ${SPKG}_${NEW_VERSION}.orig.tar.xz ${SPKG}_${NEW_VERSION}.orig.tar.xz.orig
tar --xz -xf ${SPKG}_${NEW_VERSION}.orig.tar.xz
# make any modification to files in ${SPKG}-${NEW_VERSION}
# exiftool -icc_profile"-<=" $listoffiles
tar --xz -cf ${SPKG}_${NEW_VERSION}.orig.tar.xz ${SPKG}-${NEW_VERSION}
cd -
uupdate --no-symlink --upstream-version ${NEW_VERSION} ../${SPKG}_${NEW_VERSION}.orig.tar.xz
-------------------

As for "gbp import-orig --uscan", this needs to be updated to cope with
multiple upstream tarballs.  Then we just need to make it honor
debian/uupdate in the watch file at that time.

Regards,

Osamu

Message #15 received at 748474@bugs.debian.org (full text, mbox, reply):

From: Osamu Aoki <osamu@debian.org>

To: 748474@bugs.debian.org

Cc: David Prévot <taffit@debian.org>

Subject: mk-origtargz: Please allow extra actions

Date: Sun, 15 Nov 2015 10:04:19 +0900

Hi,

After thinking about xpi-repack, this ICC_PROFILE may be another one to
solve nicely for many people to clean upstream as easy as possible.
debian/uupdate approach is not easy to use.

Here is my memo:

As for the action to take, 
 https://lists.debian.org/debian-devel/2014/05/msg00812.html
 https://github.com/danwent/Perspectives/pull/120
These are good guide.

As I checked my packages
$ find . -regextype posix-extended -iregex '.*\.(jpg|png)' -exec sh -c 'identify -verbose "$0" | grep -i copyright && echo "$0"' {} \; 2>/dev/null
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./ddp/ddp/release-notes/png/caution.png
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./webwml/manuals/release-notes/png/caution.png
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./ibus/github/ibus/ibus-hangul/setup/ibus-setup-hangul.png
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./ibus-hangul/osamu/ibus-hangul-keep/setup/ibus-setup-hangul.png
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./ibus-hangul/ibus-hangul/setup/ibus-setup-hangul.png
$ 

So not all ICC profile data are bad :-)

I wonder if we can add features like Files-Excluded ...
Files-ICC-Removed:
Files-ICC-Removed-<component>:

But this may be over-complication and sticking with adding a feature to
run a packaged script as originally proposed.

At any rate, if --report is used, this part of script is not run.
So security concern can be addressed for cases where uscan are run by
the server environment.

As for the copyright checker as presented with "find ..." in the above,
adding this to the debmake package may be more helpful.

Osamu

Message #20 received at 748474@bugs.debian.org (full text, mbox, reply):

From: Osamu Aoki <osamu@debian.org>

To: Nicholas Bamber <nicholas@periapt.co.uk>

Cc: 458789@bugs.debian.org, 748474@bugs.debian.org, David Prévot <taffit@debian.org>

Subject: clean-orig-sorce target (custom download scripts)

Date: Mon, 7 Dec 2015 22:13:21 +0900

Please respond to 748474@bugs.debian.org
Please drop       458789@bugs.debian.org 

Hi,

On Sun, Dec 06, 2015 at 04:03:34PM +0000, Nicholas Bamber wrote:
> Osamu,
> 	I was just thinking about this because the internet is the modern day
> equiavlent of the Wild West.

i.e., never trust codes from the internet :-)
 
>  It seems the bug report is fairly old and quite a lot of what is
>  talked about has been implemented. However the essential issue is
>  that some packages just do not fit into the normal mode.  "ksh" is an
>  example. (The project is dead upstream so it deserves no
>  consideration for that as well as being odd.) Such projects usually
>  have a get-orig-source target in their rules file as specified in the
>  policy:
> http://www.debian.org/doc/debian-policy/ch-source.html.

Script for uscan is sensitive issue.  Let's see the original bug:

  https://bugs.debian.org/458789

This is about running script while searching the new version string.
That is handled with mangling rules under safe substitution method.

Adding script running capability beats the whole purpose of
  sub safe_replace ($$)
to ensure safety of this part of action.  (uscan --report ...)

Please read Jakub's message: https://bugs.debian.org/cgi-bin/458789#34

I agree with him and I marked this #458789 as wontfix.

>  So what we could do is build on that. The "custom" type watch file
>  would run the "get-orig-source" target and treat a specified
>  directory as the remote web page and parse that directory as if it
>  were a web page.

This is another issue.  repacking of tarball is done by mk-origtargz
called from uscan.  See the bug report:
  https://bugs.debian.org/cgi-bin/748474

This talks about running script during repacking of tarball
This part of code will not be run if --report is used.  All packaging
crawling usages of uscan under the server environment do not come here.

So I am not all that negative about adding some support for this
feature of running script when repackaging.  (David!)

I was initially thinking to add a watch line option pointing to a
script... but as I read your message, your idea is better direction.

As the policy indicates, the get-orig-source provide downloading feature
too.  So there is an feature overlap with uscan.  We do not need
"download" part.  If there is the "clean-orig-source" target exist,
that is what mk-origtargz should run, I think.

This "clean-orig-source" target maybe better interface than setting
script name in watch file.  We can test script independently without
reading manual :-) Just "./debian/rules clean-orig-source"

Let's see what other people say.

Osamu

Message #25 received at 748474@bugs.debian.org (full text, mbox, reply):

From: Nicholas Bamber <nicholas@periapt.co.uk>

To: 748474@bugs.debian.org

Cc: 458789@bugs.debian.org, David Prévot <taffit@debian.org>

Subject: Re: clean-orig-sorce target (custom download scripts)

Date: Mon, 07 Dec 2015 14:29:29 +0000

Osamu,
	with the get-orig-source I think I really messed up my explanation of 
how I see things.

	Once a maintainer has dtermined  that the upstream download method is 
too whacky to expect uscan to cope, and has to resort to a 
get-orig-source target in the debian/rules file, then yes uscan is out 
of the picture for downloading the tarball.

	However uscan is also used in say the DDPO reports to identify when a 
new version is available. That function of uscan cannot be handled by 
debian/rules but uscan could offer a wrapper around the get-orig-source 
target to make that infomration available via the debian/watch interface.

	Does that make my point clearer?

On 07/12/15 13:13, Osamu Aoki wrote:
> Please respond to 748474@bugs.debian.org
> Please drop       458789@bugs.debian.org
>
> Hi,
>
> On Sun, Dec 06, 2015 at 04:03:34PM +0000, Nicholas Bamber wrote:
>> Osamu,
>> 	I was just thinking about this because the internet is the modern day
>> equiavlent of the Wild West.
>
> i.e., never trust codes from the internet :-)
>
>>   It seems the bug report is fairly old and quite a lot of what is
>>   talked about has been implemented. However the essential issue is
>>   that some packages just do not fit into the normal mode.  "ksh" is an
>>   example. (The project is dead upstream so it deserves no
>>   consideration for that as well as being odd.) Such projects usually
>>   have a get-orig-source target in their rules file as specified in the
>>   policy:
>> http://www.debian.org/doc/debian-policy/ch-source.html.
>
> Script for uscan is sensitive issue.  Let's see the original bug:
>
>    https://bugs.debian.org/458789
>
> This is about running script while searching the new version string.
> That is handled with mangling rules under safe substitution method.
>
> Adding script running capability beats the whole purpose of
>    sub safe_replace ($$)
> to ensure safety of this part of action.  (uscan --report ...)
>
> Please read Jakub's message: https://bugs.debian.org/cgi-bin/458789#34
>
> I agree with him and I marked this #458789 as wontfix.
>
>>   So what we could do is build on that. The "custom" type watch file
>>   would run the "get-orig-source" target and treat a specified
>>   directory as the remote web page and parse that directory as if it
>>   were a web page.
>
> This is another issue.  repacking of tarball is done by mk-origtargz
> called from uscan.  See the bug report:
>    https://bugs.debian.org/cgi-bin/748474
>
> This talks about running script during repacking of tarball
> This part of code will not be run if --report is used.  All packaging
> crawling usages of uscan under the server environment do not come here.
>
> So I am not all that negative about adding some support for this
> feature of running script when repackaging.  (David!)
>
> I was initially thinking to add a watch line option pointing to a
> script... but as I read your message, your idea is better direction.
>
> As the policy indicates, the get-orig-source provide downloading feature
> too.  So there is an feature overlap with uscan.  We do not need
> "download" part.  If there is the "clean-orig-source" target exist,
> that is what mk-origtargz should run, I think.
>
> This "clean-orig-source" target maybe better interface than setting
> script name in watch file.  We can test script independently without
> reading manual :-) Just "./debian/rules clean-orig-source"
>
> Let's see what other people say.
>
> Osamu
>
>
>

Message #30 received at 748474@bugs.debian.org (full text, mbox, reply):

From: Osamu Aoki <osamu@debian.org>

To: Nicholas Bamber <nicholas@periapt.co.uk>, 748474@bugs.debian.org

Cc: 458789@bugs.debian.org

Subject: Re: Bug#748474: clean-orig-sorce target (custom download scripts)

Date: Wed, 9 Dec 2015 22:24:33 +0900

Hi,

On Mon, Dec 07, 2015 at 02:29:29PM +0000, Nicholas Bamber wrote:
> Osamu,
> 	with the get-orig-source I think I really messed up my explanation of how I
> see things.
> 
> 	Once a maintainer has dtermined  that the upstream download method is too
> whacky to expect uscan to cope, and has to resort to a get-orig-source
> target in the debian/rules file, then yes uscan is out of the picture for
> downloading the tarball.
> 
> 	However uscan is also used in say the DDPO reports to identify when a new
> version is available. That function of uscan cannot be handled by
> debian/rules but uscan could offer a wrapper around the get-orig-source
> target to make that infomration available via the debian/watch interface.
> 
> 	Does that make my point clearer?


Watch file to identify if there is new version or not
get-orig-source to download etc.

Hmmmm, that's easy addition.  Question is is it valuable. Probably.

Let me think a bit more.

(clean-orig-source idea may not be too far as this report....)

Some download script run after --report barrier ...  interesting
thought.

Osamu

Send a report that this bug log contains spam.