Re: Python 2 and test/vm/netbsd

[Thomas Huth]

On 18/10/2019 00.55, Eduardo Habkost wrote:
> On Thu, Oct 17, 2019 at 07:05:41PM -0300, Eduardo Habkost wrote:
>> On Wed, Oct 16, 2019 at 07:41:24PM -0300, Eduardo Habkost wrote:
>>> On Wed, Oct 16, 2019 at 08:11:57AM +0200, Thomas Huth wrote:
>>>> On 16/10/2019 05.00, Eduardo Habkost wrote:
>>>>> On Tue, Sep 17, 2019 at 08:31:40PM -0300, Eduardo Habkost wrote:
>>>>>> On Mon, Jul 01, 2019 at 07:25:27PM -0300, Eduardo Habkost wrote:
>>>>>>> On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:
>>>>> [...]
>>>>>>>> The configure check also spits out deprecation warnings for
>>>>>>>> the NetBSD/FreeBSD/OpenBSD tests/vm configurations. It would be nice
>>>>>>>> to get those updated.
>>>>>>>
>>>>>>> CCing the test/vm maintainers.
>>>>>>>
>>>>>>> Fam, Alex, are you able to fix this and create new BSD VM images
>>>>>>> with Python 3 available?  I thought the VM image configurations
>>>>>>> were stored in the source tree, but they are downloaded from
>>>>>>> download.patchew.org.
>>>>>>
>>>>>> Fam, Alex, can you help us on this?  Python 2 won't be supported
>>>>>> anymore, so we need the VM images to be updated.
>>>>>
>>>>> Anyone?
>>>>>
>>>>> I'm about to submit patches to remove Python 2 support, and this
>>>>> will break tests/vm/netbsd.
>>>>>
>>>>> I'm powerless to fix this issue, because the netbsd image is
>>>>> hosted at download.patchew.org.
>>>>
>>>> Gerd had a patch to convert the netbsd VM script to ad hoc image
>>>> creation, too:
>>>>
>>>> https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg04459.html
>>>>
>>>> But there was a regression with the serial port between QEMU v3.0 and
>>>> v4.x, so it was not included:
>>>>
>>>> https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg06784.html
>>>
>>> The URL above has this error:
>>>
>>>   con recv:  x: Exitqqqqqqqqqqqqqqqqqqqqqqqqqj
>>>   con recv: To be able to use the network, we need answers to the
>>>   following:Network media type
>>>   con send: <enter>
>>>   con recv: : qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk Perform autoconfiguration?
>>>      >a: Yes b: Noqqqqqqqqqqqqqqqqq
>>>   console: *** read timeout ***
>>>   console: waiting for: 'a: Yes'
>>>   console: line buffer:
>>>   
>>>   con recv: qqqqqqqqqqqqqqj
>>>
>>> I believe that problem was solved in v4, because v4 was reading
>>> the serial output 1 byte at a time.
>>>
>>> The issue that caused the netbsd patch to be dropped was:
>>> https://lore.kernel.org/qemu-devel/address@hidden/
>>>
>>> Possibly this is the same issue we saw at:
>>> https://lore.kernel.org/qemu-devel/address@hidden/
>>>
>>> The test script must either close the console socket, or keep
>>> reading from it.  Otherwise, the QEMU VCPU threads might get
>>> stuck waiting for the chardev to be writeable.
>>
>> It doesn't seem to be the same issue.  Even if the console socket is closed,
>> I'm seeing results similar to the ones reported by Peter (the "pkgin -y
>> install" step is unreasonably slow).
>>
>> Running with V=1, I see packages being downloaded at reasonable speeds, but
>> there's a huge interval (of various minutes) between each package download.
> 
> I've found the cause for the slowness I'm seeing: for each file
> being downloaded, the guest spents at least 75 seconds trying to
> connect to the IPv6 address of ftp.NetBSD.org, before trying
> IPv4.  I don't know if this is a NetBSD bug, or a slirp bug.

Does it work better if you turn IPv6 off? E.g.:

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -81,7 +81,7 @@ class BaseVM(object):
         self._args = [ \
             "-nodefaults", "-m", "4G",
             "-cpu", "max",
-            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22",
+            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22,ipv6=off",
             "-device", "virtio-net-pci,netdev=vnet",
             "-vnc", "127.0.0.1:0,to=20"]
         if vcpus and vcpus > 1:

 Thomas


> Output of `strace -e trace=network` below:
> 
> 1571352260.348566 recvfrom(30, 
> "~[\201\200\0\1\0\1\0\0\0\0\3ftp\6NetBSD\3org\0\0\1\0\1"..., 1500, 0, 
> {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.5.30.160")}, 
> [128->16]) = 48 <0.000016>
> 1571352260.349030 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 31 
> <0.000041>
> 1571352260.349142 setsockopt(31, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
> <0.000009>
> 1571352260.349179 setsockopt(31, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
> <0.000007>
> 1571352260.349207 setsockopt(31, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>
> 1571352260.349239 connect(31, {sa_family=AF_INET6, sin6_port=htons(80), 
> sin6_flowinfo=htonl(67108864), inet_pton(AF_INET6, "2001:470:a085:999::21", 
> &sin6_addr), sin6_scope_id=377348480}, 28) = -1 ENETUNREACH (Network is 
> unreachable) <0.000
> 021>
> 1571352266.350112 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 31 
> <0.000131>
> 1571352266.350603 setsockopt(31, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
> <0.000183>
> 1571352266.350946 setsockopt(31, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
> <0.000047>
> 1571352266.351109 setsockopt(31, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000043>
> 1571352266.351260 connect(31, {sa_family=AF_INET6, sin6_port=htons(80), 
> sin6_flowinfo=htonl(67108864), inet_pton(AF_INET6, "2001:470:a085:999::21", 
> &sin6_addr), sin6_scope_id=377348480}, 28) = -1 ENETUNREACH (Network is 
> unreachable) <0.000070>
> 1571352278.357962 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 29 
> <0.000166>
> 1571352278.358524 setsockopt(29, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
> <0.000046>
> 1571352278.358757 setsockopt(29, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
> <0.000046>
> 1571352278.358950 setsockopt(29, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000039>
> 1571352278.359103 connect(29, {sa_family=AF_INET6, sin6_port=htons(80), 
> sin6_flowinfo=htonl(67108864), inet_pton(AF_INET6, "2001:470:a085:999::21", 
> &sin6_addr), sin6_scope_id=377348480}, 28) = -1 ENETUNREACH (Network is 
> unreachable) <0.000065>
> 1571352302.373056 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 29 
> <0.000323>
> 1571352302.373909 setsockopt(29, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
> <0.000162>
> 1571352302.374331 setsockopt(29, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
> <0.000159>
> 1571352302.374626 setsockopt(29, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000050>
> 1571352302.374857 connect(29, {sa_family=AF_INET6, sin6_port=htons(80), 
> sin6_flowinfo=htonl(67108864), inet_pton(AF_INET6, "2001:470:a085:999::21", 
> &sin6_addr), sin6_scope_id=377348480}, 28) = -1 ENETUNREACH (Network is 
> unreachable) <0.000070>
> 1571352335.394568 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 29 
> <0.000091>
> 1571352335.394796 setsockopt(29, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
> <0.000011>
> 1571352335.394848 setsockopt(29, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
> <0.000009>
> 1571352335.394883 setsockopt(29, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>
> 1571352335.394913 connect(29, {sa_family=AF_INET, sin_port=htons(80), 
> sin_addr=inet_addr("199.233.217.201")}, 16) = -1 EINPROGRESS (Operation now 
> in progress) <0.000055>
> 1571352335.587395 sendto(29, "", 0, 0, NULL, 0) = 0 <0.000220>
> 1571352335.589650 sendto(29, "GET /pub/pkgsrc/packages/NetBSD/"..., 81, 0, 
> NULL, 0) = 81 <0.000088>
>