From 605978b9c3f7bd84b33dfbcecd7392280cb62469 Mon Sep 17 00:00:00 2001 From: Simon Menke Date: Thu, 28 Aug 2014 20:49:41 +0200 Subject: [PATCH 0001/3965] e3x: working on channel implementation --- LICENSE | 191 ++++++++++++++++++++++++++++++++ README.md | 3 + _examples/nat-tester.go | 67 +++++++++++ nat.go | 42 +++++++ natpmp.go | 178 ++++++++++++++++++++++++++++++ upnp.go | 239 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 720 insertions(+) create mode 100644 LICENSE create mode 100644 README.md create mode 100644 _examples/nat-tester.go create mode 100644 nat.go create mode 100644 natpmp.go create mode 100644 upnp.go diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000..37ec93a14f --- /dev/null +++ b/LICENSE @@ -0,0 +1,191 @@ +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and +distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright +owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities +that control, are controlled by, or are under common control with that entity. +For the purposes of this definition, "control" means (i) the power, direct or +indirect, to cause the direction or management of such entity, whether by +contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the +outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising +permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including +but not limited to software source code, documentation source, and configuration +files. + +"Object" form shall mean any form resulting from mechanical transformation or +translation of a Source form, including but not limited to compiled object code, +generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made +available under the License, as indicated by a copyright notice that is included +in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that +is based on (or derived from) the Work and for which the editorial revisions, +annotations, elaborations, or other modifications represent, as a whole, an +original work of authorship. For the purposes of this License, Derivative Works +shall not include works that remain separable from, or merely link (or bind by +name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version +of the Work and any modifications or additions to that Work or Derivative Works +thereof, that is intentionally submitted to Licensor for inclusion in the Work +by the copyright owner or by an individual or Legal Entity authorized to submit +on behalf of the copyright owner. For the purposes of this definition, +"submitted" means any form of electronic, verbal, or written communication sent +to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, and +issue tracking systems that are managed by, or on behalf of, the Licensor for +the purpose of discussing and improving the Work, but excluding communication +that is conspicuously marked or otherwise designated in writing by the copyright +owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf +of whom a Contribution has been received by Licensor and subsequently +incorporated within the Work. + +2. Grant of Copyright License. + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable copyright license to reproduce, prepare Derivative Works of, +publicly display, publicly perform, sublicense, and distribute the Work and such +Derivative Works in Source or Object form. + +3. Grant of Patent License. + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable (except as stated in this section) patent license to make, have +made, use, offer to sell, sell, import, and otherwise transfer the Work, where +such license applies only to those patent claims licensable by such Contributor +that are necessarily infringed by their Contribution(s) alone or by combination +of their Contribution(s) with the Work to which such Contribution(s) was +submitted. If You institute patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Work or a +Contribution incorporated within the Work constitutes direct or contributory +patent infringement, then any patent licenses granted to You under this License +for that Work shall terminate as of the date such litigation is filed. + +4. Redistribution. + +You may reproduce and distribute copies of the Work or Derivative Works thereof +in any medium, with or without modifications, and in Source or Object form, +provided that You meet the following conditions: + +You must give any other recipients of the Work or Derivative Works a copy of +this License; and +You must cause any modified files to carry prominent notices stating that You +changed the files; and +You must retain, in the Source form of any Derivative Works that You distribute, +all copyright, patent, trademark, and attribution notices from the Source form +of the Work, excluding those notices that do not pertain to any part of the +Derivative Works; and +If the Work includes a "NOTICE" text file as part of its distribution, then any +Derivative Works that You distribute must include a readable copy of the +attribution notices contained within such NOTICE file, excluding those notices +that do not pertain to any part of the Derivative Works, in at least one of the +following places: within a NOTICE text file distributed as part of the +Derivative Works; within the Source form or documentation, if provided along +with the Derivative Works; or, within a display generated by the Derivative +Works, if and wherever such third-party notices normally appear. The contents of +the NOTICE file are for informational purposes only and do not modify the +License. You may add Your own attribution notices within Derivative Works that +You distribute, alongside or as an addendum to the NOTICE text from the Work, +provided that such additional attribution notices cannot be construed as +modifying the License. +You may add Your own copyright statement to Your modifications and may provide +additional or different license terms and conditions for use, reproduction, or +distribution of Your modifications, or for any such Derivative Works as a whole, +provided Your use, reproduction, and distribution of the Work otherwise complies +with the conditions stated in this License. + +5. Submission of Contributions. + +Unless You explicitly state otherwise, any Contribution intentionally submitted +for inclusion in the Work by You to the Licensor shall be under the terms and +conditions of this License, without any additional terms or conditions. +Notwithstanding the above, nothing herein shall supersede or modify the terms of +any separate license agreement you may have executed with Licensor regarding +such Contributions. + +6. Trademarks. + +This License does not grant permission to use the trade names, trademarks, +service marks, or product names of the Licensor, except as required for +reasonable and customary use in describing the origin of the Work and +reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. + +Unless required by applicable law or agreed to in writing, Licensor provides the +Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, +including, without limitation, any warranties or conditions of TITLE, +NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are +solely responsible for determining the appropriateness of using or +redistributing the Work and assume any risks associated with Your exercise of +permissions under this License. + +8. Limitation of Liability. + +In no event and under no legal theory, whether in tort (including negligence), +contract, or otherwise, unless required by applicable law (such as deliberate +and grossly negligent acts) or agreed to in writing, shall any Contributor be +liable to You for damages, including any direct, indirect, special, incidental, +or consequential damages of any character arising as a result of this License or +out of the use or inability to use the Work (including but not limited to +damages for loss of goodwill, work stoppage, computer failure or malfunction, or +any and all other commercial damages or losses), even if such Contributor has +been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. + +While redistributing the Work or Derivative Works thereof, You may choose to +offer, and charge a fee for, acceptance of support, warranty, indemnity, or +other liability obligations and/or rights consistent with this License. However, +in accepting such obligations, You may act only on Your own behalf and on Your +sole responsibility, not on behalf of any other Contributor, and only if You +agree to indemnify, defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason of your +accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work + +To apply the Apache License to your work, attach the following boilerplate +notice, with the fields enclosed by brackets "[]" replaced with your own +identifying information. (Don't include the brackets!) The text should be +enclosed in the appropriate comment syntax for the file format. We also +recommend that a file or class name and description of purpose be included on +the same "printed page" as the copyright notice for easier identification within +third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000000..24ddea09e6 --- /dev/null +++ b/README.md @@ -0,0 +1,3 @@ +# go-nat + +[![GoDoc](https://godoc.org/github.com/fd/go-nat?status.svg)](https://godoc.org/github.com/fd/go-nat) [![status](https://sourcegraph.com/api/repos/github.com/fd/go-nat/.badges/status.png)](https://sourcegraph.com/github.com/fd/go-nat) diff --git a/_examples/nat-tester.go b/_examples/nat-tester.go new file mode 100644 index 0000000000..1033293888 --- /dev/null +++ b/_examples/nat-tester.go @@ -0,0 +1,67 @@ +package main + +import ( + "fmt" + "log" + "net/http" + "time" + + "bitbucket.org/simonmenke/go-telehash/util/nat" +) + +func main() { + nat, err := nat.Discover() + if err != nil { + log.Fatalf("error: %s", err) + } + log.Printf("nat type: %s", nat.Type()) + + daddr, err := nat.GetDeviceAddress() + if err != nil { + log.Fatalf("error: %s", err) + } + log.Printf("device address: %s", daddr) + + iaddr, err := nat.GetInternalAddress() + if err != nil { + log.Fatalf("error: %s", err) + } + log.Printf("internal address: %s", iaddr) + + eaddr, err := nat.GetExternalAddress() + if err != nil { + log.Fatalf("error: %s", err) + } + log.Printf("external address: %s", eaddr) + + eport, err := nat.AddPortMapping("tcp", 3080, "http", 60) + if err != nil { + log.Fatalf("error: %s", err) + } + + log.Printf("test-page: http://%s:%d/", eaddr, eport) + + go func() { + for { + time.Sleep(30 * time.Second) + + _, err = nat.AddPortMapping("tcp", 3080, "http", 60) + if err != nil { + log.Fatalf("error: %s", err) + } + } + }() + + defer nat.DeletePortMapping("txp", 3080) + + http.ListenAndServe(":3080", http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) { + rw.Header().Set("Content-Type", "text/plain") + rw.WriteHeader(200) + fmt.Fprintf(rw, "Hello there!\n") + fmt.Fprintf(rw, "nat type: %s\n", nat.Type()) + fmt.Fprintf(rw, "device address: %s\n", daddr) + fmt.Fprintf(rw, "internal address: %s\n", iaddr) + fmt.Fprintf(rw, "external address: %s\n", eaddr) + fmt.Fprintf(rw, "test-page: http://%s:%d/\n", eaddr, eport) + })) +} diff --git a/nat.go b/nat.go new file mode 100644 index 0000000000..59b970f4f4 --- /dev/null +++ b/nat.go @@ -0,0 +1,42 @@ +package nat + +import ( + "errors" + "math" + "math/rand" + "net" + "time" +) + +var ErrNoExternalAddress = errors.New("nat: no external address") +var ErrNoInternalAddress = errors.New("nat: no internal address") +var ErrNoNATFound = errors.New("nat: no NAT found") + +// protocol is either "udp" or "tcp" +type NAT interface { + Type() string + GetDeviceAddress() (addr net.IP, err error) + GetInternalAddress() (addr net.IP, err error) + GetExternalAddress() (addr net.IP, err error) + + AddPortMapping(protocol string, internalPort int, description string, timeout time.Duration) (mappedExternalPort int, err error) + DeletePortMapping(protocol string, internalPort int) (err error) +} + +func Discover() (NAT, error) { + select { + case nat := <-discoverUPNP_IG1(): + return nat, nil + case nat := <-discoverUPNP_IG2(): + return nat, nil + case nat := <-discoverNATPMP(): + return nat, nil + case <-time.After(10 * time.Second): + return nil, ErrNoNATFound + } +} + +func randomPort() int { + rand.Seed(time.Now().UnixNano()) + return rand.Intn(math.MaxUint16-10000) + 10000 +} diff --git a/natpmp.go b/natpmp.go new file mode 100644 index 0000000000..8bc6c07171 --- /dev/null +++ b/natpmp.go @@ -0,0 +1,178 @@ +package nat + +import ( + "net" + "time" + + "github.com/jackpal/go-nat-pmp" +) + +var ( + _ NAT = (*natpmp_NAT)(nil) +) + +func natpmp_PotentialGateways() ([]net.IP, error) { + _, ipNet_10, err := net.ParseCIDR("10.0.0.0/8") + if err != nil { + panic(err) + } + + _, ipNet_172_16, err := net.ParseCIDR("172.16.0.0/12") + if err != nil { + panic(err) + } + + _, ipNet_192_168, err := net.ParseCIDR("192.168.0.0/16") + if err != nil { + panic(err) + } + + ifaces, err := net.Interfaces() + if err != nil { + return nil, err + } + + var ips []net.IP + + for _, iface := range ifaces { + addrs, err := iface.Addrs() + if err != nil { + return nil, err + } + + for _, addr := range addrs { + switch x := addr.(type) { + case *net.IPNet: + var ipNet *net.IPNet + if ipNet_10.Contains(x.IP) { + ipNet = ipNet_10 + } else if ipNet_172_16.Contains(x.IP) { + ipNet = ipNet_172_16 + } else if ipNet_192_168.Contains(x.IP) { + ipNet = ipNet_192_168 + } + + if ipNet != nil { + ip := x.IP.Mask(x.Mask) + ip = ip.To4() + if ip != nil { + ip[3] = ip[3] | 0x01 + ips = append(ips, ip) + } + } + } + } + } + + if len(ips) == 0 { + return nil, ErrNoNATFound + } + + return ips, nil +} + +func discoverNATPMP() <-chan NAT { + ips, err := natpmp_PotentialGateways() + if err != nil { + return nil + } + + res := make(chan NAT, len(ips)) + + for _, ip := range ips { + go discoverNATPMPWithAddr(res, ip) + } + + return res +} + +func discoverNATPMPWithAddr(c chan NAT, ip net.IP) { + client := natpmp.NewClient(ip) + _, err := client.GetExternalAddress() + if err != nil { + return + } + + c <- &natpmp_NAT{client, ip, make(map[int]int)} +} + +type natpmp_NAT struct { + c *natpmp.Client + gateway net.IP + ports map[int]int +} + +func (n *natpmp_NAT) GetDeviceAddress() (addr net.IP, err error) { + return n.gateway, nil +} + +func (n *natpmp_NAT) GetInternalAddress() (addr net.IP, err error) { + ifaces, err := net.Interfaces() + if err != nil { + return nil, err + } + + for _, iface := range ifaces { + addrs, err := iface.Addrs() + if err != nil { + return nil, err + } + + for _, addr := range addrs { + switch x := addr.(type) { + case *net.IPNet: + if x.Contains(n.gateway) { + return x.IP, nil + } + } + } + } + + return nil, ErrNoInternalAddress +} + +func (n *natpmp_NAT) GetExternalAddress() (addr net.IP, err error) { + res, err := n.c.GetExternalAddress() + if err != nil { + return nil, err + } + + d := res.ExternalIPAddress + return net.IPv4(d[0], d[1], d[2], d[3]), nil +} + +func (n *natpmp_NAT) AddPortMapping(protocol string, internalPort int, description string, timeout time.Duration) (int, error) { + var ( + err error + ) + + timeoutInSeconds := int(timeout / time.Second) + + if externalPort := n.ports[internalPort]; externalPort > 0 { + _, err = n.c.AddPortMapping(protocol, internalPort, externalPort, timeoutInSeconds) + if err == nil { + n.ports[internalPort] = externalPort + return externalPort, nil + } + } + + for i := 0; i < 3; i++ { + externalPort := randomPort() + _, err = n.c.AddPortMapping(protocol, internalPort, externalPort, timeoutInSeconds) + if err == nil { + n.ports[internalPort] = externalPort + return externalPort, nil + } + } + + return 0, err +} + +func (n *natpmp_NAT) DeletePortMapping(protocol string, internalPort int) (err error) { + delete(n.ports, internalPort) + return nil +} + +func (n *natpmp_NAT) Type() string { + return "NAT-PMP" +} diff --git a/upnp.go b/upnp.go new file mode 100644 index 0000000000..b27ca1e9a9 --- /dev/null +++ b/upnp.go @@ -0,0 +1,239 @@ +package nat + +import ( + "net" + "time" + + "github.com/huin/goupnp" + "github.com/huin/goupnp/dcps/internetgateway1" + "github.com/huin/goupnp/dcps/internetgateway2" +) + +var ( + _ NAT = (*upnp_NAT)(nil) +) + +func discoverUPNP_IG1() <-chan NAT { + res := make(chan NAT, 1) + go func() { + + // find devices + devs, err := goupnp.DiscoverDevices(internetgateway1.URN_WANConnectionDevice_1) + if err != nil { + return + } + + for _, dev := range devs { + if dev.Root == nil { + continue + } + + dev.Root.Device.VisitServices(func(srv *goupnp.Service) { + switch srv.ServiceType { + case internetgateway1.URN_WANIPConnection_1: + client := &internetgateway1.WANIPConnection1{goupnp.ServiceClient{ + SOAPClient: srv.NewSOAPClient(), + RootDevice: dev.Root, + Service: srv, + }} + _, isNat, err := client.GetNATRSIPStatus() + if err == nil && isNat { + res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-IP1)", dev.Root} + return + } + + case internetgateway1.URN_WANPPPConnection_1: + client := &internetgateway1.WANPPPConnection1{goupnp.ServiceClient{ + SOAPClient: srv.NewSOAPClient(), + RootDevice: dev.Root, + Service: srv, + }} + _, isNat, err := client.GetNATRSIPStatus() + if err == nil && isNat { + res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-PPP1)", dev.Root} + return + } + + } + }) + } + + }() + return res +} + +func discoverUPNP_IG2() <-chan NAT { + res := make(chan NAT, 1) + go func() { + + // find devices + devs, err := goupnp.DiscoverDevices(internetgateway2.URN_WANConnectionDevice_2) + if err != nil { + return + } + + for _, dev := range devs { + if dev.Root == nil { + continue + } + + dev.Root.Device.VisitServices(func(srv *goupnp.Service) { + switch srv.ServiceType { + case internetgateway2.URN_WANIPConnection_1: + client := &internetgateway2.WANIPConnection1{goupnp.ServiceClient{ + SOAPClient: srv.NewSOAPClient(), + RootDevice: dev.Root, + Service: srv, + }} + _, isNat, err := client.GetNATRSIPStatus() + if err == nil && isNat { + res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-IP1)", dev.Root} + return + } + + case internetgateway2.URN_WANIPConnection_2: + client := &internetgateway2.WANIPConnection2{goupnp.ServiceClient{ + SOAPClient: srv.NewSOAPClient(), + RootDevice: dev.Root, + Service: srv, + }} + _, isNat, err := client.GetNATRSIPStatus() + if err == nil && isNat { + res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-IP2)", dev.Root} + return + } + + case internetgateway2.URN_WANPPPConnection_1: + client := &internetgateway2.WANPPPConnection1{goupnp.ServiceClient{ + SOAPClient: srv.NewSOAPClient(), + RootDevice: dev.Root, + Service: srv, + }} + _, isNat, err := client.GetNATRSIPStatus() + if err == nil && isNat { + res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-PPP1)", dev.Root} + return + } + + } + }) + } + + }() + return res +} + +type upnp_NAT_Client interface { + GetExternalIPAddress() (string, error) + AddPortMapping(string, uint16, string, uint16, string, bool, string, uint32) error + DeletePortMapping(string, uint16, string) error +} + +type upnp_NAT struct { + c upnp_NAT_Client + ports map[int]int + typ string + rootDevice *goupnp.RootDevice +} + +func (u *upnp_NAT) GetExternalAddress() (addr net.IP, err error) { + ipString, err := u.c.GetExternalIPAddress() + if err != nil { + return nil, err + } + + ip := net.ParseIP(ipString) + if ip == nil { + return nil, ErrNoExternalAddress + } + + return ip, nil +} + +func mapProtocol(s string) string { + switch s { + case "udp": + return "UDP" + case "tcp": + return "TCP" + default: + panic("invalid protocol: " + s) + } +} + +func (u *upnp_NAT) AddPortMapping(protocol string, internalPort int, description string, timeout time.Duration) (int, error) { + ip, err := u.GetInternalAddress() + if err != nil { + return 0, nil + } + + timeoutInSeconds := uint32(timeout / time.Second) + + if externalPort := u.ports[internalPort]; externalPort > 0 { + err = u.c.AddPortMapping("", uint16(externalPort), mapProtocol(protocol), uint16(internalPort), ip.String(), true, description, timeoutInSeconds) + if err == nil { + return externalPort, nil + } + } + + for i := 0; i < 3; i++ { + externalPort := randomPort() + err = u.c.AddPortMapping("", uint16(externalPort), mapProtocol(protocol), uint16(internalPort), ip.String(), true, description, timeoutInSeconds) + if err == nil { + u.ports[internalPort] = externalPort + return externalPort, nil + } + } + + return 0, err +} + +func (u *upnp_NAT) DeletePortMapping(protocol string, internalPort int) error { + if externalPort := u.ports[internalPort]; externalPort > 0 { + delete(u.ports, internalPort) + return u.c.DeletePortMapping("", uint16(externalPort), mapProtocol(protocol)) + } + + return nil +} + +func (u *upnp_NAT) GetDeviceAddress() (net.IP, error) { + addr, err := net.ResolveUDPAddr("udp4", u.rootDevice.URLBase.Host) + if err != nil { + return nil, err + } + + return addr.IP, nil +} + +func (u *upnp_NAT) GetInternalAddress() (net.IP, error) { + devAddr, err := u.GetDeviceAddress() + if err != nil { + return nil, err + } + + ifaces, err := net.Interfaces() + if err != nil { + return nil, err + } + + for _, iface := range ifaces { + addrs, err := iface.Addrs() + if err != nil { + return nil, err + } + + for _, addr := range addrs { + switch x := addr.(type) { + case *net.IPNet: + if x.Contains(devAddr) { + return x.IP, nil + } + } + } + } + + return nil, ErrNoInternalAddress +} + +func (n *upnp_NAT) Type() string { return n.typ } From d827c54a1e8044b91c5c20885ad53524432fa83e Mon Sep 17 00:00:00 2001 From: Simon Menke Date: Mon, 27 Oct 2014 08:07:06 +0100 Subject: [PATCH 0002/3965] Moving to github --- _examples/nat-tester.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_examples/nat-tester.go b/_examples/nat-tester.go index 1033293888..592125639f 100644 --- a/_examples/nat-tester.go +++ b/_examples/nat-tester.go @@ -6,7 +6,7 @@ import ( "net/http" "time" - "bitbucket.org/simonmenke/go-telehash/util/nat" + "github.com/telehash/gogotelehash/util/nat" ) func main() { From 83bf98db65f0e298f73df271e80b35d42eebe4a5 Mon Sep 17 00:00:00 2001 From: Simon Menke Date: Mon, 27 Oct 2014 17:39:55 +0100 Subject: [PATCH 0003/3965] Vendored dependencies --- natpmp.go | 2 +- upnp.go | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/natpmp.go b/natpmp.go index 8bc6c07171..33fb5028a7 100644 --- a/natpmp.go +++ b/natpmp.go @@ -4,7 +4,7 @@ import ( "net" "time" - "github.com/jackpal/go-nat-pmp" + "github.com/telehash/gogotelehash/Godeps/_workspace/src/github.com/jackpal/go-nat-pmp" ) var ( diff --git a/upnp.go b/upnp.go index b27ca1e9a9..4cabd661aa 100644 --- a/upnp.go +++ b/upnp.go @@ -4,9 +4,9 @@ import ( "net" "time" - "github.com/huin/goupnp" - "github.com/huin/goupnp/dcps/internetgateway1" - "github.com/huin/goupnp/dcps/internetgateway2" + "github.com/telehash/gogotelehash/Godeps/_workspace/src/github.com/huin/goupnp" + "github.com/telehash/gogotelehash/Godeps/_workspace/src/github.com/huin/goupnp/dcps/internetgateway1" + "github.com/telehash/gogotelehash/Godeps/_workspace/src/github.com/huin/goupnp/dcps/internetgateway2" ) var ( From 4dbe9af9ab7666bc2ee453e9d08cd59448ca4bbb Mon Sep 17 00:00:00 2001 From: Simon Menke Date: Thu, 30 Oct 2014 16:41:18 +0100 Subject: [PATCH 0004/3965] WIP go vet and golint --- upnp.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/upnp.go b/upnp.go index 4cabd661aa..b6c3bfa1ab 100644 --- a/upnp.go +++ b/upnp.go @@ -31,7 +31,7 @@ func discoverUPNP_IG1() <-chan NAT { dev.Root.Device.VisitServices(func(srv *goupnp.Service) { switch srv.ServiceType { case internetgateway1.URN_WANIPConnection_1: - client := &internetgateway1.WANIPConnection1{goupnp.ServiceClient{ + client := &internetgateway1.WANIPConnection1{ServiceClient: goupnp.ServiceClient{ SOAPClient: srv.NewSOAPClient(), RootDevice: dev.Root, Service: srv, @@ -43,7 +43,7 @@ func discoverUPNP_IG1() <-chan NAT { } case internetgateway1.URN_WANPPPConnection_1: - client := &internetgateway1.WANPPPConnection1{goupnp.ServiceClient{ + client := &internetgateway1.WANPPPConnection1{ServiceClient: goupnp.ServiceClient{ SOAPClient: srv.NewSOAPClient(), RootDevice: dev.Root, Service: srv, @@ -80,7 +80,7 @@ func discoverUPNP_IG2() <-chan NAT { dev.Root.Device.VisitServices(func(srv *goupnp.Service) { switch srv.ServiceType { case internetgateway2.URN_WANIPConnection_1: - client := &internetgateway2.WANIPConnection1{goupnp.ServiceClient{ + client := &internetgateway2.WANIPConnection1{ServiceClient: goupnp.ServiceClient{ SOAPClient: srv.NewSOAPClient(), RootDevice: dev.Root, Service: srv, @@ -92,7 +92,7 @@ func discoverUPNP_IG2() <-chan NAT { } case internetgateway2.URN_WANIPConnection_2: - client := &internetgateway2.WANIPConnection2{goupnp.ServiceClient{ + client := &internetgateway2.WANIPConnection2{ServiceClient: goupnp.ServiceClient{ SOAPClient: srv.NewSOAPClient(), RootDevice: dev.Root, Service: srv, @@ -104,7 +104,7 @@ func discoverUPNP_IG2() <-chan NAT { } case internetgateway2.URN_WANPPPConnection_1: - client := &internetgateway2.WANPPPConnection1{goupnp.ServiceClient{ + client := &internetgateway2.WANPPPConnection1{ServiceClient: goupnp.ServiceClient{ SOAPClient: srv.NewSOAPClient(), RootDevice: dev.Root, Service: srv, From c0902fb9c1dabc41dd65e0189061594b37726af0 Mon Sep 17 00:00:00 2001 From: Simon Menke Date: Wed, 5 Nov 2014 12:43:11 +0100 Subject: [PATCH 0005/3965] Extracted from gogotelehash --- _examples/nat-tester.go | 2 +- nat.go | 22 +++++++++++++++++----- natpmp.go | 2 +- upnp.go | 6 +++--- 4 files changed, 22 insertions(+), 10 deletions(-) diff --git a/_examples/nat-tester.go b/_examples/nat-tester.go index 592125639f..e534d3f11d 100644 --- a/_examples/nat-tester.go +++ b/_examples/nat-tester.go @@ -6,7 +6,7 @@ import ( "net/http" "time" - "github.com/telehash/gogotelehash/util/nat" + "github.com/fd/go-nat" ) func main() { diff --git a/nat.go b/nat.go index 59b970f4f4..3cf9648585 100644 --- a/nat.go +++ b/nat.go @@ -1,3 +1,4 @@ +// Package nat implements NAT handling facilities package nat import ( @@ -8,22 +9,33 @@ import ( "time" ) -var ErrNoExternalAddress = errors.New("nat: no external address") -var ErrNoInternalAddress = errors.New("nat: no internal address") -var ErrNoNATFound = errors.New("nat: no NAT found") +var ErrNoExternalAddress = errors.New("no external address") +var ErrNoInternalAddress = errors.New("no internal address") +var ErrNoNATFound = errors.New("no NAT found") // protocol is either "udp" or "tcp" type NAT interface { + // Type returns the kind of NAT port mapping service that is used Type() string + + // GetDeviceAddress returns the internal address of the gateway device. GetDeviceAddress() (addr net.IP, err error) - GetInternalAddress() (addr net.IP, err error) + + // GetExternalAddress returns the external address of the gateway device. GetExternalAddress() (addr net.IP, err error) + // GetInternalAddress returns the address of the local host. + GetInternalAddress() (addr net.IP, err error) + + // AddPortMapping maps a port on the local host to an external port. AddPortMapping(protocol string, internalPort int, description string, timeout time.Duration) (mappedExternalPort int, err error) + + // DeletePortMapping removes a port mapping. DeletePortMapping(protocol string, internalPort int) (err error) } -func Discover() (NAT, error) { +// DiscoverGateway attempts to find a gateway device. +func DiscoverGateway() (NAT, error) { select { case nat := <-discoverUPNP_IG1(): return nat, nil diff --git a/natpmp.go b/natpmp.go index 33fb5028a7..8bc6c07171 100644 --- a/natpmp.go +++ b/natpmp.go @@ -4,7 +4,7 @@ import ( "net" "time" - "github.com/telehash/gogotelehash/Godeps/_workspace/src/github.com/jackpal/go-nat-pmp" + "github.com/jackpal/go-nat-pmp" ) var ( diff --git a/upnp.go b/upnp.go index b6c3bfa1ab..86d6e9b3ba 100644 --- a/upnp.go +++ b/upnp.go @@ -4,9 +4,9 @@ import ( "net" "time" - "github.com/telehash/gogotelehash/Godeps/_workspace/src/github.com/huin/goupnp" - "github.com/telehash/gogotelehash/Godeps/_workspace/src/github.com/huin/goupnp/dcps/internetgateway1" - "github.com/telehash/gogotelehash/Godeps/_workspace/src/github.com/huin/goupnp/dcps/internetgateway2" + "github.com/huin/goupnp" + "github.com/huin/goupnp/dcps/internetgateway1" + "github.com/huin/goupnp/dcps/internetgateway2" ) var ( From 50e7633d5f27d81490026a13e5b92d2e42d8c6bb Mon Sep 17 00:00:00 2001 From: Simon Menke Date: Wed, 5 Nov 2014 13:04:22 +0100 Subject: [PATCH 0006/3965] Fixed the example --- _examples/nat-tester.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_examples/nat-tester.go b/_examples/nat-tester.go index e534d3f11d..f2f3bc9afc 100644 --- a/_examples/nat-tester.go +++ b/_examples/nat-tester.go @@ -10,7 +10,7 @@ import ( ) func main() { - nat, err := nat.Discover() + nat, err := nat.DiscoverGateway() if err != nil { log.Fatalf("error: %s", err) } From dcaf50131e4810440bed2cbb6f7f32c4f4cc95dd Mon Sep 17 00:00:00 2001 From: Simon Menke Date: Fri, 8 May 2015 11:37:58 +0200 Subject: [PATCH 0007/3965] Using github.com/jackpal/gateway to discover NAT-PMP/PCP gateways This change also removes the restriction on what IP ranges can be used with NAT-PMP/PCP. Fixes: #1, #2 --- natpmp.go | 89 ++++++++----------------------------------------------- 1 file changed, 13 insertions(+), 76 deletions(-) diff --git a/natpmp.go b/natpmp.go index 8bc6c07171..395a5dd578 100644 --- a/natpmp.go +++ b/natpmp.go @@ -4,82 +4,19 @@ import ( "net" "time" + "github.com/jackpal/gateway" "github.com/jackpal/go-nat-pmp" ) var ( - _ NAT = (*natpmp_NAT)(nil) + _ NAT = (*natpmpNAT)(nil) ) -func natpmp_PotentialGateways() ([]net.IP, error) { - _, ipNet_10, err := net.ParseCIDR("10.0.0.0/8") - if err != nil { - panic(err) - } - - _, ipNet_172_16, err := net.ParseCIDR("172.16.0.0/12") - if err != nil { - panic(err) - } - - _, ipNet_192_168, err := net.ParseCIDR("192.168.0.0/16") - if err != nil { - panic(err) - } - - ifaces, err := net.Interfaces() - if err != nil { - return nil, err - } - - var ips []net.IP - - for _, iface := range ifaces { - addrs, err := iface.Addrs() - if err != nil { - return nil, err - } - - for _, addr := range addrs { - switch x := addr.(type) { - case *net.IPNet: - var ipNet *net.IPNet - if ipNet_10.Contains(x.IP) { - ipNet = ipNet_10 - } else if ipNet_172_16.Contains(x.IP) { - ipNet = ipNet_172_16 - } else if ipNet_192_168.Contains(x.IP) { - ipNet = ipNet_192_168 - } - - if ipNet != nil { - ip := x.IP.Mask(x.Mask) - ip = ip.To4() - if ip != nil { - ip[3] = ip[3] | 0x01 - ips = append(ips, ip) - } - } - } - } - } - - if len(ips) == 0 { - return nil, ErrNoNATFound - } - - return ips, nil -} - func discoverNATPMP() <-chan NAT { - ips, err := natpmp_PotentialGateways() - if err != nil { - return nil - } - - res := make(chan NAT, len(ips)) + res := make(chan NAT, 1) - for _, ip := range ips { + ip, err := gateway.DiscoverGateway() + if err == nil { go discoverNATPMPWithAddr(res, ip) } @@ -93,20 +30,20 @@ func discoverNATPMPWithAddr(c chan NAT, ip net.IP) { return } - c <- &natpmp_NAT{client, ip, make(map[int]int)} + c <- &natpmpNAT{client, ip, make(map[int]int)} } -type natpmp_NAT struct { +type natpmpNAT struct { c *natpmp.Client gateway net.IP ports map[int]int } -func (n *natpmp_NAT) GetDeviceAddress() (addr net.IP, err error) { +func (n *natpmpNAT) GetDeviceAddress() (addr net.IP, err error) { return n.gateway, nil } -func (n *natpmp_NAT) GetInternalAddress() (addr net.IP, err error) { +func (n *natpmpNAT) GetInternalAddress() (addr net.IP, err error) { ifaces, err := net.Interfaces() if err != nil { return nil, err @@ -131,7 +68,7 @@ func (n *natpmp_NAT) GetInternalAddress() (addr net.IP, err error) { return nil, ErrNoInternalAddress } -func (n *natpmp_NAT) GetExternalAddress() (addr net.IP, err error) { +func (n *natpmpNAT) GetExternalAddress() (addr net.IP, err error) { res, err := n.c.GetExternalAddress() if err != nil { return nil, err @@ -141,7 +78,7 @@ func (n *natpmp_NAT) GetExternalAddress() (addr net.IP, err error) { return net.IPv4(d[0], d[1], d[2], d[3]), nil } -func (n *natpmp_NAT) AddPortMapping(protocol string, internalPort int, description string, timeout time.Duration) (int, error) { +func (n *natpmpNAT) AddPortMapping(protocol string, internalPort int, description string, timeout time.Duration) (int, error) { var ( err error ) @@ -168,11 +105,11 @@ func (n *natpmp_NAT) AddPortMapping(protocol string, internalPort int, descripti return 0, err } -func (n *natpmp_NAT) DeletePortMapping(protocol string, internalPort int) (err error) { +func (n *natpmpNAT) DeletePortMapping(protocol string, internalPort int) (err error) { delete(n.ports, internalPort) return nil } -func (n *natpmp_NAT) Type() string { +func (n *natpmpNAT) Type() string { return "NAT-PMP" } From 3023356beb10273993a8a6d2f9c069194f9b35ca Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 18:51:47 -0700 Subject: [PATCH 0008/3965] extracted from go-peerstream --- p2p/muxer/mplex/multiplex.go | 44 +++++++++++++++++++++++++++++++ p2p/muxer/mplex/multiplex_test.go | 11 ++++++++ 2 files changed, 55 insertions(+) create mode 100644 p2p/muxer/mplex/multiplex.go create mode 100644 p2p/muxer/mplex/multiplex_test.go diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go new file mode 100644 index 0000000000..62868a8410 --- /dev/null +++ b/p2p/muxer/mplex/multiplex.go @@ -0,0 +1,44 @@ +package peerstream_multiplex + +import ( + "net" + + smux "github.com/jbenet/go-stream-mux" + mp "github.com/jbenet/go-stream-mux/Godeps/_workspace/src/github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. +) + +type conn struct { + *mp.Multiplex +} + +func (c *conn) Close() error { + return c.Multiplex.Close() +} + +func (c *conn) IsClosed() bool { + return c.Multiplex.IsClosed() +} + +// OpenStream creates a new stream. +func (c *conn) OpenStream() (smux.Stream, error) { + return c.Multiplex.NewStream(), nil +} + +// Serve starts listening for incoming requests and handles them +// using given StreamHandler +func (c *conn) Serve(handler smux.StreamHandler) { + c.Multiplex.Serve(func(s *mp.Stream) { + handler(s) + }) +} + +// Transport is a go-peerstream transport that constructs +// multiplex-backed connections. +type Transport struct{} + +// DefaultTransport has default settings for multiplex +var DefaultTransport = &Transport{} + +func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { + return &conn{mp.NewMultiplex(nc, isServer)}, nil +} diff --git a/p2p/muxer/mplex/multiplex_test.go b/p2p/muxer/mplex/multiplex_test.go new file mode 100644 index 0000000000..1b8ffd76a3 --- /dev/null +++ b/p2p/muxer/mplex/multiplex_test.go @@ -0,0 +1,11 @@ +package peerstream_multiplex + +import ( + "testing" + + test "github.com/jbenet/go-stream-mux/test" +) + +func TestMultiplexTransport(t *testing.T) { + test.SubtestAll(t, DefaultTransport) +} From 62cf6119be59e169d72f022c5248203d7a79a2f6 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 18:51:47 -0700 Subject: [PATCH 0009/3965] extracted from go-peerstream --- p2p/muxer/yamux/yamux.go | 94 +++++++++++++++++++++++++++++++++++ p2p/muxer/yamux/yamux_test.go | 11 ++++ 2 files changed, 105 insertions(+) create mode 100644 p2p/muxer/yamux/yamux.go create mode 100644 p2p/muxer/yamux/yamux_test.go diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go new file mode 100644 index 0000000000..c5861d1a28 --- /dev/null +++ b/p2p/muxer/yamux/yamux.go @@ -0,0 +1,94 @@ +package sm_yamux + +import ( + "io/ioutil" + "net" + "time" + + smux "github.com/jbenet/go-stream-mux" + yamux "github.com/jbenet/go-stream-mux/Godeps/_workspace/src/github.com/hashicorp/yamux" +) + +// stream implements smux.Stream using a ss.Stream +type stream yamux.Stream + +func (s *stream) yamuxStream() *yamux.Stream { + return (*yamux.Stream)(s) +} + +func (s *stream) Read(buf []byte) (int, error) { + return s.yamuxStream().Read(buf) +} + +func (s *stream) Write(buf []byte) (int, error) { + return s.yamuxStream().Write(buf) +} + +func (s *stream) Close() error { + return s.yamuxStream().Close() +} + +// Conn is a connection to a remote peer. +type conn yamux.Session + +func (c *conn) yamuxSession() *yamux.Session { + return (*yamux.Session)(c) +} + +func (c *conn) Close() error { + return c.yamuxSession().Close() +} + +func (c *conn) IsClosed() bool { + return c.yamuxSession().IsClosed() +} + +// OpenStream creates a new stream. +func (c *conn) OpenStream() (smux.Stream, error) { + s, err := c.yamuxSession().OpenStream() + if err != nil { + return nil, err + } + + return (*stream)(s), nil +} + +// Serve starts listening for incoming requests and handles them +// using given StreamHandler +func (c *conn) Serve(handler smux.StreamHandler) { + for { // accept loop + s, err := c.yamuxSession().AcceptStream() + if err != nil { + return // err always means closed. + } + go handler((*stream)(s)) + } +} + +// Transport is a go-peerstream transport that constructs +// yamux-backed connections. +type Transport yamux.Config + +// DefaultTransport has default settings for yamux +var DefaultTransport = (*Transport)(&yamux.Config{ + AcceptBacklog: 256, // from yamux.DefaultConfig + EnableKeepAlive: true, // from yamux.DefaultConfig + KeepAliveInterval: 30 * time.Second, // from yamux.DefaultConfig + MaxStreamWindowSize: uint32(256 * 1024), // from yamux.DefaultConfig + LogOutput: ioutil.Discard, +}) + +func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { + var s *yamux.Session + var err error + if isServer { + s, err = yamux.Server(nc, t.Config()) + } else { + s, err = yamux.Client(nc, t.Config()) + } + return (*conn)(s), err +} + +func (t *Transport) Config() *yamux.Config { + return (*yamux.Config)(t) +} diff --git a/p2p/muxer/yamux/yamux_test.go b/p2p/muxer/yamux/yamux_test.go new file mode 100644 index 0000000000..2a319570c2 --- /dev/null +++ b/p2p/muxer/yamux/yamux_test.go @@ -0,0 +1,11 @@ +package sm_yamux + +import ( + "testing" + + test "github.com/jbenet/go-stream-mux/test" +) + +func TestYamuxTransport(t *testing.T) { + test.SubtestAll(t, DefaultTransport) +} From 811f0c738998da13d1406e8510b1b75e05325f2a Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 18:51:47 -0700 Subject: [PATCH 0010/3965] extracted from go-peerstream --- p2p/muxer/muxer-multistream/multistream.go | 60 +++++++++++++++++++ .../muxer-multistream/multistream_test.go | 11 ++++ 2 files changed, 71 insertions(+) create mode 100644 p2p/muxer/muxer-multistream/multistream.go create mode 100644 p2p/muxer/muxer-multistream/multistream_test.go diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go new file mode 100644 index 0000000000..1d8e8c2b7c --- /dev/null +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -0,0 +1,60 @@ +// package multistream implements a peerstream transport using +// go-multistream to select the underlying stream muxer +package multistream + +import ( + "net" + + mss "github.com/jbenet/go-stream-mux/Godeps/_workspace/src/github.com/whyrusleeping/go-multistream" + + smux "github.com/jbenet/go-stream-mux" + multiplex "github.com/jbenet/go-stream-mux/multiplex" + spdy "github.com/jbenet/go-stream-mux/spdystream" + yamux "github.com/jbenet/go-stream-mux/yamux" +) + +type transport struct { + mux *mss.MultistreamMuxer + + tpts map[string]smux.Transport +} + +func NewTransport() smux.Transport { + mux := mss.NewMultistreamMuxer() + mux.AddHandler("/multiplex", nil) + mux.AddHandler("/spdystream", nil) + mux.AddHandler("/yamux", nil) + + tpts := map[string]smux.Transport{ + "/multiplex": multiplex.DefaultTransport, + "/spdystream": spdy.Transport, + "/yamux": yamux.DefaultTransport, + } + + return &transport{ + mux: mux, + tpts: tpts, + } +} + +func (t *transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { + var proto string + if isServer { + selected, _, err := t.mux.Negotiate(nc) + if err != nil { + return nil, err + } + proto = selected + } else { + // prefer yamux + selected, err := mss.SelectOneOf([]string{"/yamux", "/spdystream", "/multiplex"}, nc) + if err != nil { + return nil, err + } + proto = selected + } + + tpt := t.tpts[proto] + + return tpt.NewConn(nc, isServer) +} diff --git a/p2p/muxer/muxer-multistream/multistream_test.go b/p2p/muxer/muxer-multistream/multistream_test.go new file mode 100644 index 0000000000..95a9264c69 --- /dev/null +++ b/p2p/muxer/muxer-multistream/multistream_test.go @@ -0,0 +1,11 @@ +package multistream + +import ( + "testing" + + test "github.com/jbenet/go-stream-mux/test" +) + +func TestMultiStreamTransport(t *testing.T) { + test.SubtestAll(t, NewTransport()) +} From 5b2626d3699a2977a3a9f4a327f0ca0941758044 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 19:17:39 -0700 Subject: [PATCH 0011/3965] fixed urls --- p2p/muxer/mplex/multiplex.go | 2 +- p2p/muxer/mplex/multiplex_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 62868a8410..6d2387ca5d 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -4,7 +4,7 @@ import ( "net" smux "github.com/jbenet/go-stream-mux" - mp "github.com/jbenet/go-stream-mux/Godeps/_workspace/src/github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. + mp "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. ) type conn struct { diff --git a/p2p/muxer/mplex/multiplex_test.go b/p2p/muxer/mplex/multiplex_test.go index 1b8ffd76a3..e35b3b038d 100644 --- a/p2p/muxer/mplex/multiplex_test.go +++ b/p2p/muxer/mplex/multiplex_test.go @@ -3,7 +3,7 @@ package peerstream_multiplex import ( "testing" - test "github.com/jbenet/go-stream-mux/test" + test "github.com/jbenet/go-stream-muxer/test" ) func TestMultiplexTransport(t *testing.T) { From de05b59cdf2cec0cc706775bd3fbf9558c6882b8 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 19:17:39 -0700 Subject: [PATCH 0012/3965] fixed urls --- p2p/muxer/yamux/yamux.go | 2 +- p2p/muxer/yamux/yamux_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index c5861d1a28..4e6794fc56 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -6,7 +6,7 @@ import ( "time" smux "github.com/jbenet/go-stream-mux" - yamux "github.com/jbenet/go-stream-mux/Godeps/_workspace/src/github.com/hashicorp/yamux" + yamux "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/hashicorp/yamux" ) // stream implements smux.Stream using a ss.Stream diff --git a/p2p/muxer/yamux/yamux_test.go b/p2p/muxer/yamux/yamux_test.go index 2a319570c2..f95409ec24 100644 --- a/p2p/muxer/yamux/yamux_test.go +++ b/p2p/muxer/yamux/yamux_test.go @@ -3,7 +3,7 @@ package sm_yamux import ( "testing" - test "github.com/jbenet/go-stream-mux/test" + test "github.com/jbenet/go-stream-muxer/test" ) func TestYamuxTransport(t *testing.T) { From 5f17d146eb78ae6810a2107a42396a238c04f1ec Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 19:17:39 -0700 Subject: [PATCH 0013/3965] fixed urls --- p2p/muxer/muxer-multistream/multistream.go | 8 ++++---- p2p/muxer/muxer-multistream/multistream_test.go | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 1d8e8c2b7c..267c6e460b 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -5,12 +5,12 @@ package multistream import ( "net" - mss "github.com/jbenet/go-stream-mux/Godeps/_workspace/src/github.com/whyrusleeping/go-multistream" + mss "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/whyrusleeping/go-multistream" smux "github.com/jbenet/go-stream-mux" - multiplex "github.com/jbenet/go-stream-mux/multiplex" - spdy "github.com/jbenet/go-stream-mux/spdystream" - yamux "github.com/jbenet/go-stream-mux/yamux" + multiplex "github.com/jbenet/go-stream-muxer/multiplex" + spdy "github.com/jbenet/go-stream-muxer/spdystream" + yamux "github.com/jbenet/go-stream-muxer/yamux" ) type transport struct { diff --git a/p2p/muxer/muxer-multistream/multistream_test.go b/p2p/muxer/muxer-multistream/multistream_test.go index 95a9264c69..de1a9e4cd8 100644 --- a/p2p/muxer/muxer-multistream/multistream_test.go +++ b/p2p/muxer/muxer-multistream/multistream_test.go @@ -3,7 +3,7 @@ package multistream import ( "testing" - test "github.com/jbenet/go-stream-mux/test" + test "github.com/jbenet/go-stream-muxer/test" ) func TestMultiStreamTransport(t *testing.T) { From 73c3d70a0850ca2a82e2606edab7877091954855 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 19:39:04 -0700 Subject: [PATCH 0014/3965] links fix --- p2p/muxer/mplex/multiplex.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 6d2387ca5d..16e10006ef 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -3,7 +3,7 @@ package peerstream_multiplex import ( "net" - smux "github.com/jbenet/go-stream-mux" + smux "github.com/jbenet/go-stream-muxer" mp "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. ) From 59832ab002103f730060d260d570fb9514e1eb4e Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 19:39:04 -0700 Subject: [PATCH 0015/3965] links fix --- p2p/muxer/yamux/yamux.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 4e6794fc56..0f3a76f0a8 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -5,7 +5,7 @@ import ( "net" "time" - smux "github.com/jbenet/go-stream-mux" + smux "github.com/jbenet/go-stream-muxer" yamux "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/hashicorp/yamux" ) From 49ce9ea2a51e1819a97e851ec807e97f8a228c70 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 19:39:04 -0700 Subject: [PATCH 0016/3965] links fix --- p2p/muxer/muxer-multistream/multistream.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 267c6e460b..d13605531b 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -7,7 +7,7 @@ import ( mss "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/whyrusleeping/go-multistream" - smux "github.com/jbenet/go-stream-mux" + smux "github.com/jbenet/go-stream-muxer" multiplex "github.com/jbenet/go-stream-muxer/multiplex" spdy "github.com/jbenet/go-stream-muxer/spdystream" yamux "github.com/jbenet/go-stream-muxer/yamux" From aab33b03745aab68b3fdf155e1a92400c1ea72f2 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 20:55:24 -0700 Subject: [PATCH 0017/3965] implement AcceptStream --- p2p/muxer/mplex/multiplex.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 16e10006ef..0c626480e9 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -1,12 +1,15 @@ package peerstream_multiplex import ( + "errors" "net" smux "github.com/jbenet/go-stream-muxer" mp "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. ) +var ErrUseServe = errors.New("not implemented, use Serve") + type conn struct { *mp.Multiplex } @@ -24,6 +27,11 @@ func (c *conn) OpenStream() (smux.Stream, error) { return c.Multiplex.NewStream(), nil } +// AcceptStream accepts a stream opened by the other side. +func (c *conn) AcceptStream() (smux.Stream, error) { + return nil, ErrUseServe +} + // Serve starts listening for incoming requests and handles them // using given StreamHandler func (c *conn) Serve(handler smux.StreamHandler) { From d42b66198c20332f6f4fc95fdd048f813c9e6962 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Sat, 11 Jul 2015 20:55:24 -0700 Subject: [PATCH 0018/3965] implement AcceptStream --- p2p/muxer/yamux/yamux.go | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 0f3a76f0a8..0e13843878 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -53,15 +53,21 @@ func (c *conn) OpenStream() (smux.Stream, error) { return (*stream)(s), nil } +// AcceptStream accepts a stream opened by the other side. +func (c *conn) AcceptStream() (smux.Stream, error) { + s, err := c.yamuxSession().AcceptStream() + return (*stream)(s), err +} + // Serve starts listening for incoming requests and handles them // using given StreamHandler func (c *conn) Serve(handler smux.StreamHandler) { for { // accept loop - s, err := c.yamuxSession().AcceptStream() + s, err := c.AcceptStream() if err != nil { return // err always means closed. } - go handler((*stream)(s)) + go handler(s) } } From 357317776cc1f37a7cd8893d14224b1e5cc82303 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 14 Jul 2015 09:18:42 -0700 Subject: [PATCH 0019/3965] update multistream and multiplex --- p2p/muxer/mplex/multiplex.go | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 0c626480e9..d07d53e057 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -29,15 +29,19 @@ func (c *conn) OpenStream() (smux.Stream, error) { // AcceptStream accepts a stream opened by the other side. func (c *conn) AcceptStream() (smux.Stream, error) { - return nil, ErrUseServe + return c.Multiplex.Accept() } // Serve starts listening for incoming requests and handles them // using given StreamHandler func (c *conn) Serve(handler smux.StreamHandler) { - c.Multiplex.Serve(func(s *mp.Stream) { - handler(s) - }) + for { + s, err := c.AcceptStream() + if err != nil { + return + } + go handler(s) + } } // Transport is a go-peerstream transport that constructs From 9bb09c13869c755e2367103f2d5062a110454394 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Wed, 30 Sep 2015 18:42:55 -0400 Subject: [PATCH 0020/3965] move to p2p dir --- p2p/host/peerstore/addr/addrsrcs.go | 70 ++++++++ p2p/host/peerstore/addr/addrsrcs_test.go | 78 ++++++++ p2p/host/peerstore/addr_manager.go | 188 ++++++++++++++++++++ p2p/host/peerstore/addr_manager_test.go | 180 +++++++++++++++++++ p2p/host/peerstore/metrics.go | 63 +++++++ p2p/host/peerstore/metrics_test.go | 40 +++++ p2p/host/peerstore/peer.go | 180 +++++++++++++++++++ p2p/host/peerstore/peer_test.go | 163 +++++++++++++++++ p2p/host/peerstore/peerstore.go | 216 +++++++++++++++++++++++ p2p/host/peerstore/queue/distance.go | 101 +++++++++++ p2p/host/peerstore/queue/interface.go | 18 ++ p2p/host/peerstore/queue/queue_test.go | 141 +++++++++++++++ p2p/host/peerstore/queue/sync.go | 84 +++++++++ 13 files changed, 1522 insertions(+) create mode 100644 p2p/host/peerstore/addr/addrsrcs.go create mode 100644 p2p/host/peerstore/addr/addrsrcs_test.go create mode 100644 p2p/host/peerstore/addr_manager.go create mode 100644 p2p/host/peerstore/addr_manager_test.go create mode 100644 p2p/host/peerstore/metrics.go create mode 100644 p2p/host/peerstore/metrics_test.go create mode 100644 p2p/host/peerstore/peer.go create mode 100644 p2p/host/peerstore/peer_test.go create mode 100644 p2p/host/peerstore/peerstore.go create mode 100644 p2p/host/peerstore/queue/distance.go create mode 100644 p2p/host/peerstore/queue/interface.go create mode 100644 p2p/host/peerstore/queue/queue_test.go create mode 100644 p2p/host/peerstore/queue/sync.go diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go new file mode 100644 index 0000000000..d8b942bef9 --- /dev/null +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -0,0 +1,70 @@ +// Package addr provides utility functions to handle peer addresses. +package addr + +import ( + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" +) + +// AddrSource is a source of addresses. It allows clients to retrieve +// a set of addresses at a last possible moment in time. It is used +// to query a set of addresses that may change over time, as a result +// of the network changing interfaces or mappings. +type Source interface { + Addrs() []ma.Multiaddr +} + +// CombineSources returns a new AddrSource which is the +// concatenation of all input AddrSources: +// +// combined := CombinedSources(a, b) +// combined.Addrs() // append(a.Addrs(), b.Addrs()...) +// +func CombineSources(srcs ...Source) Source { + return combinedAS(srcs) +} + +type combinedAS []Source + +func (cas combinedAS) Addrs() []ma.Multiaddr { + var addrs []ma.Multiaddr + for _, s := range cas { + addrs = append(addrs, s.Addrs()...) + } + return addrs +} + +// UniqueSource returns a new AddrSource which omits duplicate +// addresses from the inputs: +// +// unique := UniqueSource(a, b) +// unique.Addrs() // append(a.Addrs(), b.Addrs()...) +// // but only adds each addr once. +// +func UniqueSource(srcs ...Source) Source { + return uniqueAS(srcs) +} + +type uniqueAS []Source + +func (uas uniqueAS) Addrs() []ma.Multiaddr { + seen := make(map[string]struct{}) + var addrs []ma.Multiaddr + for _, s := range uas { + for _, a := range s.Addrs() { + s := a.String() + if _, found := seen[s]; !found { + addrs = append(addrs, a) + seen[s] = struct{}{} + } + } + } + return addrs +} + +// Slice is a simple slice of addresses that implements +// the AddrSource interface. +type Slice []ma.Multiaddr + +func (as Slice) Addrs() []ma.Multiaddr { + return as +} diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go new file mode 100644 index 0000000000..ea75a2bd6a --- /dev/null +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -0,0 +1,78 @@ +package addr + +import ( + "fmt" + "testing" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" +) + +func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { + a, err := ma.NewMultiaddr(s) + if err != nil { + t.Fatal("error parsing multiaddr", err) + } + return a +} + +func newAddrs(t *testing.T, n int) []ma.Multiaddr { + addrs := make([]ma.Multiaddr, n) + for i := 0; i < n; i++ { + s := fmt.Sprintf("/ip4/1.2.3.4/tcp/%d", i) + addrs[i] = newAddrOrFatal(t, s) + } + return addrs +} + +func addrSetsSame(a, b []ma.Multiaddr) bool { + if len(a) != len(b) { + return false + } + for i, aa := range a { + bb := b[i] + if !aa.Equal(bb) { + return false + } + } + return true +} + +func addrSourcesSame(a, b Source) bool { + return addrSetsSame(a.Addrs(), b.Addrs()) +} + +func TestAddrCombine(t *testing.T) { + addrs := newAddrs(t, 30) + a := Slice(addrs[0:10]) + b := Slice(addrs[10:20]) + c := Slice(addrs[20:30]) + d := CombineSources(a, b, c) + if !addrSetsSame(addrs, d.Addrs()) { + t.Error("addrs differ") + } + if !addrSourcesSame(Slice(addrs), d) { + t.Error("addrs differ") + } +} + +func TestAddrUnique(t *testing.T) { + + addrs := newAddrs(t, 40) + a := Slice(addrs[0:20]) + b := Slice(addrs[10:30]) + c := Slice(addrs[20:40]) + d := CombineSources(a, b, c) + e := UniqueSource(a, b, c) + if addrSetsSame(addrs, d.Addrs()) { + t.Error("addrs same") + } + if addrSourcesSame(Slice(addrs), d) { + t.Error("addrs same") + } + if !addrSetsSame(addrs, e.Addrs()) { + t.Error("addrs differ", addrs, "\n\n", e.Addrs(), "\n\n") + } + if !addrSourcesSame(Slice(addrs), e) { + t.Error("addrs differ", addrs, "\n\n", e.Addrs(), "\n\n") + } +} diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go new file mode 100644 index 0000000000..02b0fd1187 --- /dev/null +++ b/p2p/host/peerstore/addr_manager.go @@ -0,0 +1,188 @@ +package peer + +import ( + "sync" + "time" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" +) + +const ( + + // TempAddrTTL is the ttl used for a short lived address + TempAddrTTL = time.Second * 10 + + // ProviderAddrTTL is the TTL of an address we've received from a provider. + // This is also a temporary address, but lasts longer. After this expires, + // the records we return will require an extra lookup. + ProviderAddrTTL = time.Minute * 10 + + // RecentlyConnectedAddrTTL is used when we recently connected to a peer. + // It means that we are reasonably certain of the peer's address. + RecentlyConnectedAddrTTL = time.Minute * 10 + + // OwnObservedAddrTTL is used for our own external addresses observed by peers. + OwnObservedAddrTTL = time.Minute * 10 + + // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes) + // if we haven't shipped you an update to ipfs in 356 days + // we probably arent running the same bootstrap nodes... + PermanentAddrTTL = time.Hour * 24 * 356 + + // ConnectedAddrTTL is the ttl used for the addresses of a peer to whom + // we're connected directly. This is basically permanent, as we will + // clear them + re-add under a TempAddrTTL after disconnecting. + ConnectedAddrTTL = PermanentAddrTTL +) + +type expiringAddr struct { + Addr ma.Multiaddr + TTL time.Time +} + +func (e *expiringAddr) ExpiredBy(t time.Time) bool { + return t.After(e.TTL) +} + +type addrSet map[string]expiringAddr + +// AddrManager manages addresses. +// The zero-value is ready to be used. +type AddrManager struct { + addrmu sync.Mutex // guards addrs + addrs map[ID]addrSet +} + +// ensures the AddrManager is initialized. +// So we can use the zero value. +func (mgr *AddrManager) init() { + if mgr.addrs == nil { + mgr.addrs = make(map[ID]addrSet) + } +} + +func (mgr *AddrManager) Peers() []ID { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + if mgr.addrs == nil { + return nil + } + + pids := make([]ID, 0, len(mgr.addrs)) + for pid := range mgr.addrs { + pids = append(pids, pid) + } + return pids +} + +// AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) +func (mgr *AddrManager) AddAddr(p ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) +} + +// AddAddrs gives AddrManager addresses to use, with a given ttl +// (time-to-live), after which the address is no longer valid. +// If the manager has a longer TTL, the operation is a no-op for that address +func (mgr *AddrManager) AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + + // if ttl is zero, exit. nothing to do. + if ttl <= 0 { + return + } + + // so zero value can be used + mgr.init() + + amap, found := mgr.addrs[p] + if !found { + amap = make(addrSet) + mgr.addrs[p] = amap + } + + // only expand ttls + exp := time.Now().Add(ttl) + for _, addr := range addrs { + addrstr := addr.String() + a, found := amap[addrstr] + if !found || exp.After(a.TTL) { + amap[addrstr] = expiringAddr{Addr: addr, TTL: exp} + } + } +} + +// SetAddr calls mgr.SetAddrs(p, addr, ttl) +func (mgr *AddrManager) SetAddr(p ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) +} + +// SetAddrs sets the ttl on addresses. This clears any TTL there previously. +// This is used when we receive the best estimate of the validity of an address. +func (mgr *AddrManager) SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + + // so zero value can be used + mgr.init() + + amap, found := mgr.addrs[p] + if !found { + amap = make(addrSet) + mgr.addrs[p] = amap + } + + exp := time.Now().Add(ttl) + for _, addr := range addrs { + // re-set all of them for new ttl. + addrs := addr.String() + + if ttl > 0 { + amap[addrs] = expiringAddr{Addr: addr, TTL: exp} + } else { + delete(amap, addrs) + } + } +} + +// Addresses returns all known (and valid) addresses for a given +func (mgr *AddrManager) Addrs(p ID) []ma.Multiaddr { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + + // not initialized? nothing to give. + if mgr.addrs == nil { + return nil + } + + maddrs, found := mgr.addrs[p] + if !found { + return nil + } + + now := time.Now() + good := make([]ma.Multiaddr, 0, len(maddrs)) + var expired []string + for s, m := range maddrs { + if m.ExpiredBy(now) { + expired = append(expired, s) + } else { + good = append(good, m.Addr) + } + } + + // clean up the expired ones. + for _, s := range expired { + delete(maddrs, s) + } + return good +} + +// ClearAddresses removes all previously stored addresses +func (mgr *AddrManager) ClearAddrs(p ID) { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + mgr.init() + + mgr.addrs[p] = make(addrSet) // clear what was there before +} diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go new file mode 100644 index 0000000000..2bd480fdd5 --- /dev/null +++ b/p2p/host/peerstore/addr_manager_test.go @@ -0,0 +1,180 @@ +package peer + +import ( + "testing" + "time" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" +) + +func IDS(t *testing.T, ids string) ID { + id, err := IDB58Decode(ids) + if err != nil { + t.Fatal(err) + } + return id +} + +func MA(t *testing.T, m string) ma.Multiaddr { + maddr, err := ma.NewMultiaddr(m) + if err != nil { + t.Fatal(err) + } + return maddr +} + +func testHas(t *testing.T, exp, act []ma.Multiaddr) { + if len(exp) != len(act) { + t.Fatal("lengths not the same") + } + + for _, a := range exp { + found := false + + for _, b := range act { + if a.Equal(b) { + found = true + break + } + } + + if !found { + t.Fatal("expected address %s not found", a) + } + } +} + +func TestAddresses(t *testing.T) { + + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := IDS(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") + id3 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ6Kn") + id4 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Kn") + id5 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Km") + + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + ma21 := MA(t, "/ip4/2.2.3.2/tcp/1111") + ma22 := MA(t, "/ip4/2.2.3.2/tcp/2222") + ma31 := MA(t, "/ip4/3.2.3.3/tcp/1111") + ma32 := MA(t, "/ip4/3.2.3.3/tcp/2222") + ma33 := MA(t, "/ip4/3.2.3.3/tcp/3333") + ma41 := MA(t, "/ip4/4.2.3.3/tcp/1111") + ma42 := MA(t, "/ip4/4.2.3.3/tcp/2222") + ma43 := MA(t, "/ip4/4.2.3.3/tcp/3333") + ma44 := MA(t, "/ip4/4.2.3.3/tcp/4444") + ma51 := MA(t, "/ip4/5.2.3.3/tcp/1111") + ma52 := MA(t, "/ip4/5.2.3.3/tcp/2222") + ma53 := MA(t, "/ip4/5.2.3.3/tcp/3333") + ma54 := MA(t, "/ip4/5.2.3.3/tcp/4444") + ma55 := MA(t, "/ip4/5.2.3.3/tcp/5555") + + ttl := time.Hour + m := AddrManager{} + m.AddAddr(id1, ma11, ttl) + + m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) + m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) // idempotency + + m.AddAddr(id3, ma31, ttl) + m.AddAddr(id3, ma32, ttl) + m.AddAddr(id3, ma33, ttl) + m.AddAddr(id3, ma33, ttl) // idempotency + m.AddAddr(id3, ma33, ttl) + + m.AddAddrs(id4, []ma.Multiaddr{ma41, ma42, ma43, ma44}, ttl) // multiple + + m.AddAddrs(id5, []ma.Multiaddr{ma21, ma22}, ttl) // clearing + m.AddAddrs(id5, []ma.Multiaddr{ma41, ma42, ma43, ma44}, ttl) // clearing + m.ClearAddrs(id5) + m.AddAddrs(id5, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, ttl) // clearing + + // test the Addresses return value + testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + testHas(t, []ma.Multiaddr{ma31, ma32, ma33}, m.Addrs(id3)) + testHas(t, []ma.Multiaddr{ma41, ma42, ma43, ma44}, m.Addrs(id4)) + testHas(t, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, m.Addrs(id5)) + +} + +func TestAddressesExpire(t *testing.T) { + + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := MA(t, "/ip4/2.2.3.2/tcp/2222") + ma13 := MA(t, "/ip4/3.2.3.3/tcp/3333") + ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") + ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") + + m := AddrManager{} + m.AddAddr(id1, ma11, time.Hour) + m.AddAddr(id1, ma12, time.Hour) + m.AddAddr(id1, ma13, time.Hour) + m.AddAddr(id2, ma24, time.Hour) + m.AddAddr(id2, ma25, time.Hour) + + testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.SetAddr(id1, ma11, 2*time.Hour) + m.SetAddr(id1, ma12, 2*time.Hour) + m.SetAddr(id1, ma13, 2*time.Hour) + m.SetAddr(id2, ma24, 2*time.Hour) + m.SetAddr(id2, ma25, 2*time.Hour) + + testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.SetAddr(id1, ma11, time.Millisecond) + <-time.After(time.Millisecond) + testHas(t, []ma.Multiaddr{ma12, ma13}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.SetAddr(id1, ma13, time.Millisecond) + <-time.After(time.Millisecond) + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.SetAddr(id2, ma24, time.Millisecond) + <-time.After(time.Millisecond) + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma25}, m.Addrs(id2)) + + m.SetAddr(id2, ma25, time.Millisecond) + <-time.After(time.Millisecond) + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, nil, m.Addrs(id2)) + + m.SetAddr(id1, ma12, time.Millisecond) + <-time.After(time.Millisecond) + testHas(t, nil, m.Addrs(id1)) + testHas(t, nil, m.Addrs(id2)) +} + +func TestClearWorks(t *testing.T) { + + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := MA(t, "/ip4/2.2.3.2/tcp/2222") + ma13 := MA(t, "/ip4/3.2.3.3/tcp/3333") + ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") + ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") + + m := AddrManager{} + m.AddAddr(id1, ma11, time.Hour) + m.AddAddr(id1, ma12, time.Hour) + m.AddAddr(id1, ma13, time.Hour) + m.AddAddr(id2, ma24, time.Hour) + m.AddAddr(id2, ma25, time.Hour) + + testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.ClearAddrs(id1) + m.ClearAddrs(id2) + + testHas(t, nil, m.Addrs(id1)) + testHas(t, nil, m.Addrs(id2)) +} diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go new file mode 100644 index 0000000000..6cad11cdfd --- /dev/null +++ b/p2p/host/peerstore/metrics.go @@ -0,0 +1,63 @@ +package peer + +import ( + "sync" + "time" +) + +// LatencyEWMASmooting governs the decay of the EWMA (the speed +// at which it changes). This must be a normalized (0-1) value. +// 1 is 100% change, 0 is no change. +var LatencyEWMASmoothing = 0.1 + +// Metrics is just an object that tracks metrics +// across a set of peers. +type Metrics interface { + + // RecordLatency records a new latency measurement + RecordLatency(ID, time.Duration) + + // LatencyEWMA returns an exponentially-weighted moving avg. + // of all measurements of a peer's latency. + LatencyEWMA(ID) time.Duration +} + +type metrics struct { + latmap map[ID]time.Duration + latmu sync.RWMutex +} + +func NewMetrics() Metrics { + return &metrics{ + latmap: make(map[ID]time.Duration), + } +} + +// RecordLatency records a new latency measurement +func (m *metrics) RecordLatency(p ID, next time.Duration) { + nextf := float64(next) + s := LatencyEWMASmoothing + if s > 1 || s < 0 { + s = 0.1 // ignore the knob. it's broken. look, it jiggles. + } + + m.latmu.Lock() + ewma, found := m.latmap[p] + ewmaf := float64(ewma) + if !found { + m.latmap[p] = next // when no data, just take it as the mean. + } else { + nextf = ((1.0 - s) * ewmaf) + (s * nextf) + m.latmap[p] = time.Duration(nextf) + } + m.latmu.Unlock() +} + +// LatencyEWMA returns an exponentially-weighted moving avg. +// of all measurements of a peer's latency. +func (m *metrics) LatencyEWMA(p ID) time.Duration { + m.latmu.RLock() + lat := m.latmap[p] + m.latmu.RUnlock() + return time.Duration(lat) +} diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go new file mode 100644 index 0000000000..db5f3a94d8 --- /dev/null +++ b/p2p/host/peerstore/metrics_test.go @@ -0,0 +1,40 @@ +package peer_test + +import ( + "fmt" + "math/rand" + "testing" + "time" + + peer "github.com/ipfs/go-ipfs/p2p/peer" + testutil "github.com/ipfs/go-ipfs/util/testutil" +) + +func TestLatencyEWMAFun(t *testing.T) { + t.Skip("run it for fun") + + m := peer.NewMetrics() + id, err := testutil.RandPeerID() + if err != nil { + t.Fatal(err) + } + + mu := 100.0 + sig := 10.0 + next := func() time.Duration { + mu = (rand.NormFloat64() * sig) + mu + return time.Duration(mu) + } + + print := func() { + fmt.Printf("%3.f %3.f --> %d\n", sig, mu, m.LatencyEWMA(id)) + } + + for { + select { + case <-time.After(200 * time.Millisecond): + m.RecordLatency(id, next()) + print() + } + } +} diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go new file mode 100644 index 0000000000..4e1e3e3ab1 --- /dev/null +++ b/p2p/host/peerstore/peer.go @@ -0,0 +1,180 @@ +// package peer implements an object used to represent peers in the ipfs network. +package peer + +import ( + "encoding/hex" + "encoding/json" + "fmt" + "strings" + + b58 "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-base58" + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + mh "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multihash" + + ic "github.com/ipfs/go-ipfs/p2p/crypto" + u "github.com/ipfs/go-ipfs/util" + logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" +) + +var log = logging.Logger("peer") + +// ID represents the identity of a peer. +type ID string + +// Pretty returns a b58-encoded string of the ID +func (id ID) Pretty() string { + return IDB58Encode(id) +} + +func (id ID) Loggable() map[string]interface{} { + return map[string]interface{}{ + "peerID": id.Pretty(), + } +} + +// String prints out the peer. +// +// TODO(brian): ensure correctness at ID generation and +// enforce this by only exposing functions that generate +// IDs safely. Then any peer.ID type found in the +// codebase is known to be correct. +func (id ID) String() string { + pid := id.Pretty() + + //All sha256 nodes start with Qm + //We can skip the Qm to make the peer.ID more useful + if strings.HasPrefix(pid, "Qm") { + pid = pid[2:] + } + + maxRunes := 6 + if len(pid) < maxRunes { + maxRunes = len(pid) + } + return fmt.Sprintf("", pid[:maxRunes]) +} + +// MatchesPrivateKey tests whether this ID was derived from sk +func (id ID) MatchesPrivateKey(sk ic.PrivKey) bool { + return id.MatchesPublicKey(sk.GetPublic()) +} + +// MatchesPublicKey tests whether this ID was derived from pk +func (id ID) MatchesPublicKey(pk ic.PubKey) bool { + oid, err := IDFromPublicKey(pk) + if err != nil { + return false + } + return oid == id +} + +// IDFromString cast a string to ID type, and validate +// the id to make sure it is a multihash. +func IDFromString(s string) (ID, error) { + if _, err := mh.Cast([]byte(s)); err != nil { + return ID(""), err + } + return ID(s), nil +} + +// IDFromBytes cast a string to ID type, and validate +// the id to make sure it is a multihash. +func IDFromBytes(b []byte) (ID, error) { + if _, err := mh.Cast(b); err != nil { + return ID(""), err + } + return ID(b), nil +} + +// IDB58Decode returns a b58-decoded Peer +func IDB58Decode(s string) (ID, error) { + m, err := mh.FromB58String(s) + if err != nil { + return "", err + } + return ID(m), err +} + +// IDB58Encode returns b58-encoded string +func IDB58Encode(id ID) string { + return b58.Encode([]byte(id)) +} + +// IDHexDecode returns a b58-decoded Peer +func IDHexDecode(s string) (ID, error) { + m, err := mh.FromHexString(s) + if err != nil { + return "", err + } + return ID(m), err +} + +// IDHexEncode returns b58-encoded string +func IDHexEncode(id ID) string { + return hex.EncodeToString([]byte(id)) +} + +// IDFromPublicKey returns the Peer ID corresponding to pk +func IDFromPublicKey(pk ic.PubKey) (ID, error) { + b, err := pk.Bytes() + if err != nil { + return "", err + } + hash := u.Hash(b) + return ID(hash), nil +} + +// IDFromPrivateKey returns the Peer ID corresponding to sk +func IDFromPrivateKey(sk ic.PrivKey) (ID, error) { + return IDFromPublicKey(sk.GetPublic()) +} + +// Map maps a Peer ID to a struct. +type Set map[ID]struct{} + +// PeerInfo is a small struct used to pass around a peer with +// a set of addresses (and later, keys?). This is not meant to be +// a complete view of the system, but rather to model updates to +// the peerstore. It is used by things like the routing system. +type PeerInfo struct { + ID ID + Addrs []ma.Multiaddr +} + +func (pi *PeerInfo) MarshalJSON() ([]byte, error) { + out := make(map[string]interface{}) + out["ID"] = IDB58Encode(pi.ID) + var addrs []string + for _, a := range pi.Addrs { + addrs = append(addrs, a.String()) + } + out["Addrs"] = addrs + return json.Marshal(out) +} + +func (pi *PeerInfo) UnmarshalJSON(b []byte) error { + var data map[string]interface{} + err := json.Unmarshal(b, &data) + if err != nil { + return err + } + pid, err := IDB58Decode(data["ID"].(string)) + if err != nil { + return err + } + pi.ID = pid + addrs, ok := data["Addrs"].([]interface{}) + if ok { + for _, a := range addrs { + pi.Addrs = append(pi.Addrs, ma.StringCast(a.(string))) + } + } + return nil +} + +// IDSlice for sorting peers +type IDSlice []ID + +func (es IDSlice) Len() int { return len(es) } +func (es IDSlice) Swap(i, j int) { es[i], es[j] = es[j], es[i] } +func (es IDSlice) Less(i, j int) bool { return string(es[i]) < string(es[j]) } diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go new file mode 100644 index 0000000000..80d6d9f079 --- /dev/null +++ b/p2p/host/peerstore/peer_test.go @@ -0,0 +1,163 @@ +package peer_test + +import ( + "encoding/base64" + "fmt" + "strings" + "testing" + + ic "github.com/ipfs/go-ipfs/p2p/crypto" + . "github.com/ipfs/go-ipfs/p2p/peer" + u "github.com/ipfs/go-ipfs/util" + tu "github.com/ipfs/go-ipfs/util/testutil" + + b58 "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-base58" +) + +var gen1 keyset // generated +var gen2 keyset // generated +var man keyset // manual + +func init() { + if err := gen1.generate(); err != nil { + panic(err) + } + if err := gen2.generate(); err != nil { + panic(err) + } + + skManBytes = strings.Replace(skManBytes, "\n", "", -1) + if err := man.load(hpkpMan, skManBytes); err != nil { + panic(err) + } +} + +type keyset struct { + sk ic.PrivKey + pk ic.PubKey + hpk string + hpkp string +} + +func (ks *keyset) generate() error { + var err error + ks.sk, ks.pk, err = tu.RandTestKeyPair(512) + if err != nil { + return err + } + + bpk, err := ks.pk.Bytes() + if err != nil { + return err + } + + ks.hpk = string(u.Hash(bpk)) + ks.hpkp = b58.Encode([]byte(ks.hpk)) + return nil +} + +func (ks *keyset) load(hpkp, skBytesStr string) error { + skBytes, err := base64.StdEncoding.DecodeString(skBytesStr) + if err != nil { + return err + } + + ks.sk, err = ic.UnmarshalPrivateKey(skBytes) + if err != nil { + return err + } + + ks.pk = ks.sk.GetPublic() + bpk, err := ks.pk.Bytes() + if err != nil { + return err + } + + ks.hpk = string(u.Hash(bpk)) + ks.hpkp = b58.Encode([]byte(ks.hpk)) + if ks.hpkp != hpkp { + return fmt.Errorf("hpkp doesn't match key. %s", hpkp) + } + return nil +} + +func TestIDMatchesPublicKey(t *testing.T) { + + test := func(ks keyset) { + p1, err := IDB58Decode(ks.hpkp) + if err != nil { + t.Fatal(err) + } + + if ks.hpk != string(p1) { + t.Error("p1 and hpk differ") + } + + if !p1.MatchesPublicKey(ks.pk) { + t.Fatal("p1 does not match pk") + } + + p2, err := IDFromPublicKey(ks.pk) + if err != nil { + t.Fatal(err) + } + + if p1 != p2 { + t.Error("p1 and p2 differ", p1.Pretty(), p2.Pretty()) + } + + if p2.Pretty() != ks.hpkp { + t.Error("hpkp and p2.Pretty differ", ks.hpkp, p2.Pretty()) + } + } + + test(gen1) + test(gen2) + test(man) +} + +func TestIDMatchesPrivateKey(t *testing.T) { + + test := func(ks keyset) { + p1, err := IDB58Decode(ks.hpkp) + if err != nil { + t.Fatal(err) + } + + if ks.hpk != string(p1) { + t.Error("p1 and hpk differ") + } + + if !p1.MatchesPrivateKey(ks.sk) { + t.Fatal("p1 does not match sk") + } + + p2, err := IDFromPrivateKey(ks.sk) + if err != nil { + t.Fatal(err) + } + + if p1 != p2 { + t.Error("p1 and p2 differ", p1.Pretty(), p2.Pretty()) + } + } + + test(gen1) + test(gen2) + test(man) +} + +var hpkpMan = `QmRK3JgmVEGiewxWbhpXLJyjWuGuLeSTMTndA1coMHEy5o` +var skManBytes = ` +CAAS4AQwggJcAgEAAoGBAL7w+Wc4VhZhCdM/+Hccg5Nrf4q9NXWwJylbSrXz/unFS24wyk6pEk0zi3W +7li+vSNVO+NtJQw9qGNAMtQKjVTP+3Vt/jfQRnQM3s6awojtjueEWuLYVt62z7mofOhCtj+VwIdZNBo +/EkLZ0ETfcvN5LVtLYa8JkXybnOPsLvK+PAgMBAAECgYBdk09HDM7zzL657uHfzfOVrdslrTCj6p5mo +DzvCxLkkjIzYGnlPuqfNyGjozkpSWgSUc+X+EGLLl3WqEOVdWJtbM61fewEHlRTM5JzScvwrJ39t7o6 +CCAjKA0cBWBd6UWgbN/t53RoWvh9HrA2AW5YrT0ZiAgKe9y7EMUaENVJ8QJBAPhpdmb4ZL4Fkm4OKia +NEcjzn6mGTlZtef7K/0oRC9+2JkQnCuf6HBpaRhJoCJYg7DW8ZY+AV6xClKrgjBOfERMCQQDExhnzu2 +dsQ9k8QChBlpHO0TRbZBiQfC70oU31kM1AeLseZRmrxv9Yxzdl8D693NNWS2JbKOXl0kMHHcuGQLMVA +kBZ7WvkmPV3aPL6jnwp2pXepntdVnaTiSxJ1dkXShZ/VSSDNZMYKY306EtHrIu3NZHtXhdyHKcggDXr +qkBrdgErAkAlpGPojUwemOggr4FD8sLX1ot2hDJyyV7OK2FXfajWEYJyMRL1Gm9Uk1+Un53RAkJneqp +JGAzKpyttXBTIDO51AkEA98KTiROMnnU8Y6Mgcvr68/SMIsvCYMt9/mtwSBGgl80VaTQ5Hpaktl6Xbh +VUt5Wv0tRxlXZiViCGCD1EtrrwTw== +` diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go new file mode 100644 index 0000000000..30a12ebbc9 --- /dev/null +++ b/p2p/host/peerstore/peerstore.go @@ -0,0 +1,216 @@ +package peer + +import ( + "errors" + "sync" + "time" + + ic "github.com/ipfs/go-ipfs/p2p/crypto" + + ds "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore" + dssync "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore/sync" + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" +) + +const ( + // AddressTTL is the expiration time of addresses. + AddressTTL = time.Hour +) + +// Peerstore provides a threadsafe store of Peer related +// information. +type Peerstore interface { + AddrBook + KeyBook + Metrics + + // Peers returns a list of all peer.IDs in this Peerstore + Peers() []ID + + // PeerInfo returns a peer.PeerInfo struct for given peer.ID. + // This is a small slice of the information Peerstore has on + // that peer, useful to other services. + PeerInfo(ID) PeerInfo + + // Get/Put is a simple registry for other peer-related key/value pairs. + // if we find something we use often, it should become its own set of + // methods. this is a last resort. + Get(id ID, key string) (interface{}, error) + Put(id ID, key string, val interface{}) error +} + +// AddrBook is an interface that fits the new AddrManager. I'm patching +// it up in here to avoid changing a ton of the codebase. +type AddrBook interface { + + // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) + AddAddr(p ID, addr ma.Multiaddr, ttl time.Duration) + + // AddAddrs gives AddrManager addresses to use, with a given ttl + // (time-to-live), after which the address is no longer valid. + // If the manager has a longer TTL, the operation is a no-op for that address + AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) + + // SetAddr calls mgr.SetAddrs(p, addr, ttl) + SetAddr(p ID, addr ma.Multiaddr, ttl time.Duration) + + // SetAddrs sets the ttl on addresses. This clears any TTL there previously. + // This is used when we receive the best estimate of the validity of an address. + SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) + + // Addresses returns all known (and valid) addresses for a given + Addrs(p ID) []ma.Multiaddr + + // ClearAddresses removes all previously stored addresses + ClearAddrs(p ID) +} + +// KeyBook tracks the Public keys of Peers. +type KeyBook interface { + PubKey(ID) ic.PubKey + AddPubKey(ID, ic.PubKey) error + + PrivKey(ID) ic.PrivKey + AddPrivKey(ID, ic.PrivKey) error +} + +type keybook struct { + pks map[ID]ic.PubKey + sks map[ID]ic.PrivKey + + sync.RWMutex // same lock. wont happen a ton. +} + +func newKeybook() *keybook { + return &keybook{ + pks: map[ID]ic.PubKey{}, + sks: map[ID]ic.PrivKey{}, + } +} + +func (kb *keybook) Peers() []ID { + kb.RLock() + ps := make([]ID, 0, len(kb.pks)+len(kb.sks)) + for p := range kb.pks { + ps = append(ps, p) + } + for p := range kb.sks { + if _, found := kb.pks[p]; !found { + ps = append(ps, p) + } + } + kb.RUnlock() + return ps +} + +func (kb *keybook) PubKey(p ID) ic.PubKey { + kb.RLock() + pk := kb.pks[p] + kb.RUnlock() + return pk +} + +func (kb *keybook) AddPubKey(p ID, pk ic.PubKey) error { + + // check it's correct first + if !p.MatchesPublicKey(pk) { + return errors.New("ID does not match PublicKey") + } + + kb.Lock() + kb.pks[p] = pk + kb.Unlock() + return nil +} + +func (kb *keybook) PrivKey(p ID) ic.PrivKey { + kb.RLock() + sk := kb.sks[p] + kb.RUnlock() + return sk +} + +func (kb *keybook) AddPrivKey(p ID, sk ic.PrivKey) error { + + if sk == nil { + return errors.New("sk is nil (PrivKey)") + } + + // check it's correct first + if !p.MatchesPrivateKey(sk) { + return errors.New("ID does not match PrivateKey") + } + + kb.Lock() + kb.sks[p] = sk + kb.Unlock() + return nil +} + +type peerstore struct { + keybook + metrics + AddrManager + + // store other data, like versions + ds ds.ThreadSafeDatastore +} + +// NewPeerstore creates a threadsafe collection of peers. +func NewPeerstore() Peerstore { + return &peerstore{ + keybook: *newKeybook(), + metrics: *(NewMetrics()).(*metrics), + AddrManager: AddrManager{}, + ds: dssync.MutexWrap(ds.NewMapDatastore()), + } +} + +func (ps *peerstore) Put(p ID, key string, val interface{}) error { + dsk := ds.NewKey(string(p) + "/" + key) + return ps.ds.Put(dsk, val) +} + +func (ps *peerstore) Get(p ID, key string) (interface{}, error) { + dsk := ds.NewKey(string(p) + "/" + key) + return ps.ds.Get(dsk) +} + +func (ps *peerstore) Peers() []ID { + set := map[ID]struct{}{} + for _, p := range ps.keybook.Peers() { + set[p] = struct{}{} + } + for _, p := range ps.AddrManager.Peers() { + set[p] = struct{}{} + } + + pps := make([]ID, 0, len(set)) + for p := range set { + pps = append(pps, p) + } + return pps +} + +func (ps *peerstore) PeerInfo(p ID) PeerInfo { + return PeerInfo{ + ID: p, + Addrs: ps.AddrManager.Addrs(p), + } +} + +func PeerInfos(ps Peerstore, peers []ID) []PeerInfo { + pi := make([]PeerInfo, len(peers)) + for i, p := range peers { + pi[i] = ps.PeerInfo(p) + } + return pi +} + +func PeerInfoIDs(pis []PeerInfo) []ID { + ps := make([]ID, len(pis)) + for i, pi := range pis { + ps[i] = pi.ID + } + return ps +} diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go new file mode 100644 index 0000000000..2b3bce82d3 --- /dev/null +++ b/p2p/host/peerstore/queue/distance.go @@ -0,0 +1,101 @@ +package queue + +import ( + "container/heap" + "math/big" + "sync" + + key "github.com/ipfs/go-ipfs/blocks/key" + peer "github.com/ipfs/go-ipfs/p2p/peer" + ks "github.com/ipfs/go-ipfs/routing/keyspace" +) + +// peerMetric tracks a peer and its distance to something else. +type peerMetric struct { + // the peer + peer peer.ID + + // big.Int for XOR metric + metric *big.Int +} + +// peerMetricHeap implements a heap of peerDistances +type peerMetricHeap []*peerMetric + +func (ph peerMetricHeap) Len() int { + return len(ph) +} + +func (ph peerMetricHeap) Less(i, j int) bool { + return -1 == ph[i].metric.Cmp(ph[j].metric) +} + +func (ph peerMetricHeap) Swap(i, j int) { + ph[i], ph[j] = ph[j], ph[i] +} + +func (ph *peerMetricHeap) Push(x interface{}) { + item := x.(*peerMetric) + *ph = append(*ph, item) +} + +func (ph *peerMetricHeap) Pop() interface{} { + old := *ph + n := len(old) + item := old[n-1] + *ph = old[0 : n-1] + return item +} + +// distancePQ implements heap.Interface and PeerQueue +type distancePQ struct { + // from is the Key this PQ measures against + from ks.Key + + // heap is a heap of peerDistance items + heap peerMetricHeap + + sync.RWMutex +} + +func (pq *distancePQ) Len() int { + pq.Lock() + defer pq.Unlock() + return len(pq.heap) +} + +func (pq *distancePQ) Enqueue(p peer.ID) { + pq.Lock() + defer pq.Unlock() + + distance := ks.XORKeySpace.Key([]byte(p)).Distance(pq.from) + + heap.Push(&pq.heap, &peerMetric{ + peer: p, + metric: distance, + }) +} + +func (pq *distancePQ) Dequeue() peer.ID { + pq.Lock() + defer pq.Unlock() + + if len(pq.heap) < 1 { + panic("called Dequeue on an empty PeerQueue") + // will panic internally anyway, but we can help debug here + } + + o := heap.Pop(&pq.heap) + p := o.(*peerMetric) + return p.peer +} + +// NewXORDistancePQ returns a PeerQueue which maintains its peers sorted +// in terms of their distances to each other in an XORKeySpace (i.e. using +// XOR as a metric of distance). +func NewXORDistancePQ(fromKey key.Key) PeerQueue { + return &distancePQ{ + from: ks.XORKeySpace.Key([]byte(fromKey)), + heap: peerMetricHeap{}, + } +} diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go new file mode 100644 index 0000000000..6b0f8c5c9e --- /dev/null +++ b/p2p/host/peerstore/queue/interface.go @@ -0,0 +1,18 @@ +package queue + +import peer "github.com/ipfs/go-ipfs/p2p/peer" + +// PeerQueue maintains a set of peers ordered according to a metric. +// Implementations of PeerQueue could order peers based on distances along +// a KeySpace, latency measurements, trustworthiness, reputation, etc. +type PeerQueue interface { + + // Len returns the number of items in PeerQueue + Len() int + + // Enqueue adds this node to the queue. + Enqueue(peer.ID) + + // Dequeue retrieves the highest (smallest int) priority node + Dequeue() peer.ID +} diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go new file mode 100644 index 0000000000..c303550a92 --- /dev/null +++ b/p2p/host/peerstore/queue/queue_test.go @@ -0,0 +1,141 @@ +package queue + +import ( + "fmt" + "sync" + "testing" + "time" + + key "github.com/ipfs/go-ipfs/blocks/key" + peer "github.com/ipfs/go-ipfs/p2p/peer" + u "github.com/ipfs/go-ipfs/util" + + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +func TestQueue(t *testing.T) { + + p1 := peer.ID("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a31") // these aren't valid, because need to hex-decode. + p2 := peer.ID("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a32") // these aren't valid, because need to hex-decode. + p3 := peer.ID("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33") // these aren't valid, because need to hex-decode. + p4 := peer.ID("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a34") // these aren't valid, because need to hex-decode. + p5 := peer.ID("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a31") // these aren't valid, because need to hex-decode. + // but they work. + + // these are the peer.IDs' XORKeySpace Key values: + // [228 47 151 130 156 102 222 232 218 31 132 94 170 208 80 253 120 103 55 35 91 237 48 157 81 245 57 247 66 150 9 40] + // [26 249 85 75 54 49 25 30 21 86 117 62 85 145 48 175 155 194 210 216 58 14 241 143 28 209 129 144 122 28 163 6] + // [78 135 26 216 178 181 224 181 234 117 2 248 152 115 255 103 244 34 4 152 193 88 9 225 8 127 216 158 226 8 236 246] + // [125 135 124 6 226 160 101 94 192 57 39 12 18 79 121 140 190 154 147 55 44 83 101 151 63 255 94 179 51 203 241 51] + + pq := NewXORDistancePQ(key.Key("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a31")) + pq.Enqueue(p3) + pq.Enqueue(p1) + pq.Enqueue(p2) + pq.Enqueue(p4) + pq.Enqueue(p5) + pq.Enqueue(p1) + + // should come out as: p1, p4, p3, p2 + + if d := pq.Dequeue(); d != p1 && d != p5 { + t.Error("ordering failed") + } + + if d := pq.Dequeue(); d != p1 && d != p5 { + t.Error("ordering failed") + } + + if d := pq.Dequeue(); d != p1 && d != p5 { + t.Error("ordering failed") + } + + if pq.Dequeue() != p4 { + t.Error("ordering failed") + } + + if pq.Dequeue() != p3 { + t.Error("ordering failed") + } + + if pq.Dequeue() != p2 { + t.Error("ordering failed") + } + +} + +func newPeerTime(t time.Time) peer.ID { + s := fmt.Sprintf("hmmm time: %v", t) + h := u.Hash([]byte(s)) + return peer.ID(h) +} + +func TestSyncQueue(t *testing.T) { + tickT := time.Microsecond * 50 + max := 5000 + consumerN := 10 + countsIn := make([]int, consumerN*2) + countsOut := make([]int, consumerN) + + if testing.Short() { + max = 1000 + } + + ctx := context.Background() + pq := NewXORDistancePQ(key.Key("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a31")) + cq := NewChanQueue(ctx, pq) + wg := sync.WaitGroup{} + + produce := func(p int) { + defer wg.Done() + + tick := time.Tick(tickT) + for i := 0; i < max; i++ { + select { + case tim := <-tick: + countsIn[p]++ + cq.EnqChan <- newPeerTime(tim) + case <-ctx.Done(): + return + } + } + } + + consume := func(c int) { + defer wg.Done() + + for { + select { + case <-cq.DeqChan: + countsOut[c]++ + if countsOut[c] >= max*2 { + return + } + case <-ctx.Done(): + return + } + } + } + + // make n * 2 producers and n consumers + for i := 0; i < consumerN; i++ { + wg.Add(3) + go produce(i) + go produce(consumerN + i) + go consume(i) + } + + wg.Wait() + + sum := func(ns []int) int { + total := 0 + for _, n := range ns { + total += n + } + return total + } + + if sum(countsIn) != sum(countsOut) { + t.Errorf("didnt get all of them out: %d/%d", sum(countsOut), sum(countsIn)) + } +} diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go new file mode 100644 index 0000000000..04775a0f99 --- /dev/null +++ b/p2p/host/peerstore/queue/sync.go @@ -0,0 +1,84 @@ +package queue + +import ( + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + peer "github.com/ipfs/go-ipfs/p2p/peer" + logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" +) + +var log = logging.Logger("peerqueue") + +// ChanQueue makes any PeerQueue synchronizable through channels. +type ChanQueue struct { + Queue PeerQueue + EnqChan chan<- peer.ID + DeqChan <-chan peer.ID +} + +// NewChanQueue creates a ChanQueue by wrapping pq. +func NewChanQueue(ctx context.Context, pq PeerQueue) *ChanQueue { + cq := &ChanQueue{Queue: pq} + cq.process(ctx) + return cq +} + +func (cq *ChanQueue) process(ctx context.Context) { + // construct the channels here to be able to use them bidirectionally + enqChan := make(chan peer.ID) + deqChan := make(chan peer.ID) + + cq.EnqChan = enqChan + cq.DeqChan = deqChan + + go func() { + log.Debug("processing") + defer log.Debug("closed") + defer close(deqChan) + + var next peer.ID + var item peer.ID + var more bool + + for { + if cq.Queue.Len() == 0 { + // log.Debug("wait for enqueue") + select { + case next, more = <-enqChan: + if !more { + return + } + // log.Debug("got", next) + + case <-ctx.Done(): + return + } + + } else { + next = cq.Queue.Dequeue() + // log.Debug("peek", next) + } + + select { + case item, more = <-enqChan: + if !more { + if cq.Queue.Len() > 0 { + return // we're done done. + } + enqChan = nil // closed, so no use. + } + // log.Debug("got", item) + cq.Queue.Enqueue(item) + cq.Queue.Enqueue(next) // order may have changed. + next = "" + + case deqChan <- next: + // log.Debug("dequeued", next) + next = "" + + case <-ctx.Done(): + return + } + } + + }() +} From 4ab021d79b7e56886e8683dafbb4f140fe2cc172 Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Wed, 30 Sep 2015 18:42:55 -0400 Subject: [PATCH 0021/3965] move to p2p dir --- p2p/net/swarm/addr/addr.go | 275 +++++++++++++++++ p2p/net/swarm/addr/addr_test.go | 232 ++++++++++++++ p2p/net/swarm/dial_test.go | 440 ++++++++++++++++++++++++++ p2p/net/swarm/peers_test.go | 72 +++++ p2p/net/swarm/simul_test.go | 75 +++++ p2p/net/swarm/swarm.go | 321 +++++++++++++++++++ p2p/net/swarm/swarm_addr.go | 39 +++ p2p/net/swarm/swarm_addr_test.go | 123 ++++++++ p2p/net/swarm/swarm_conn.go | 141 +++++++++ p2p/net/swarm/swarm_dial.go | 493 ++++++++++++++++++++++++++++++ p2p/net/swarm/swarm_listen.go | 142 +++++++++ p2p/net/swarm/swarm_net.go | 172 +++++++++++ p2p/net/swarm/swarm_net_test.go | 77 +++++ p2p/net/swarm/swarm_notif_test.go | 210 +++++++++++++ p2p/net/swarm/swarm_stream.go | 54 ++++ p2p/net/swarm/swarm_test.go | 330 ++++++++++++++++++++ 16 files changed, 3196 insertions(+) create mode 100644 p2p/net/swarm/addr/addr.go create mode 100644 p2p/net/swarm/addr/addr_test.go create mode 100644 p2p/net/swarm/dial_test.go create mode 100644 p2p/net/swarm/peers_test.go create mode 100644 p2p/net/swarm/simul_test.go create mode 100644 p2p/net/swarm/swarm.go create mode 100644 p2p/net/swarm/swarm_addr.go create mode 100644 p2p/net/swarm/swarm_addr_test.go create mode 100644 p2p/net/swarm/swarm_conn.go create mode 100644 p2p/net/swarm/swarm_dial.go create mode 100644 p2p/net/swarm/swarm_listen.go create mode 100644 p2p/net/swarm/swarm_net.go create mode 100644 p2p/net/swarm/swarm_net_test.go create mode 100644 p2p/net/swarm/swarm_notif_test.go create mode 100644 p2p/net/swarm/swarm_stream.go create mode 100644 p2p/net/swarm/swarm_test.go diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go new file mode 100644 index 0000000000..649d54b3c0 --- /dev/null +++ b/p2p/net/swarm/addr/addr.go @@ -0,0 +1,275 @@ +package addrutil + +import ( + "fmt" + + logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +var log = logging.Logger("p2p/net/swarm/addr") + +// SupportedTransportStrings is the list of supported transports for the swarm. +// These are strings of encapsulated multiaddr protocols. E.g.: +// /ip4/tcp +var SupportedTransportStrings = []string{ + "/ip4/tcp", + "/ip6/tcp", + // "/ip4/udp/utp", disabled because the lib is broken + // "/ip6/udp/utp", disabled because the lib is broken + // "/ip4/udp/udt", disabled because the lib doesnt work on arm + // "/ip6/udp/udt", disabled because the lib doesnt work on arm +} + +// SupportedTransportProtocols is the list of supported transports for the swarm. +// These are []ma.Protocol lists. Populated at runtime from SupportedTransportStrings +var SupportedTransportProtocols = [][]ma.Protocol{} + +func init() { + // initialize SupportedTransportProtocols + transports := make([][]ma.Protocol, len(SupportedTransportStrings)) + for _, s := range SupportedTransportStrings { + t, err := ma.ProtocolsWithString(s) + if err != nil { + panic(err) // important to fix this in the codebase + } + transports = append(transports, t) + } + SupportedTransportProtocols = transports +} + +// FilterAddrs is a filter that removes certain addresses, according to filter. +// if filter returns true, the address is kept. +func FilterAddrs(a []ma.Multiaddr, filter func(ma.Multiaddr) bool) []ma.Multiaddr { + b := make([]ma.Multiaddr, 0, len(a)) + for _, addr := range a { + if filter(addr) { + b = append(b, addr) + } + } + return b +} + +// FilterUsableAddrs removes certain addresses +// from a list. the addresses removed are those known NOT +// to work with our network. Namely, addresses with UTP. +func FilterUsableAddrs(a []ma.Multiaddr) []ma.Multiaddr { + return FilterAddrs(a, func(m ma.Multiaddr) bool { + return AddrUsable(m, false) + }) +} + +// AddrOverNonLocalIP returns whether the addr uses a non-local ip link +func AddrOverNonLocalIP(a ma.Multiaddr) bool { + split := ma.Split(a) + if len(split) < 1 { + return false + } + if manet.IsIP6LinkLocal(split[0]) { + return false + } + return true +} + +// AddrUsable returns whether our network can use this addr. +// We only use the transports in SupportedTransportStrings, +// and we do not link local addresses. Loopback is ok +// as we need to be able to connect to multiple ipfs nodes +// in the same machine. +func AddrUsable(a ma.Multiaddr, partial bool) bool { + if a == nil { + return false + } + + if !AddrOverNonLocalIP(a) { + return false + } + + // test the address protocol list is in SupportedTransportProtocols + matches := func(supported, test []ma.Protocol) bool { + if len(test) > len(supported) { + return false + } + + // when partial, it's ok if test < supported. + if !partial && len(supported) != len(test) { + return false + } + + for i := range test { + if supported[i].Code != test[i].Code { + return false + } + } + return true + } + + transport := a.Protocols() + for _, supported := range SupportedTransportProtocols { + if matches(supported, transport) { + return true + } + } + + return false +} + +// ResolveUnspecifiedAddress expands an unspecified ip addresses (/ip4/0.0.0.0, /ip6/::) to +// use the known local interfaces. If ifaceAddr is nil, we request interface addresses +// from the network stack. (this is so you can provide a cached value if resolving many addrs) +func ResolveUnspecifiedAddress(resolve ma.Multiaddr, ifaceAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { + // split address into its components + split := ma.Split(resolve) + + // if first component (ip) is not unspecified, use it as is. + if !manet.IsIPUnspecified(split[0]) { + return []ma.Multiaddr{resolve}, nil + } + + out := make([]ma.Multiaddr, 0, len(ifaceAddrs)) + for _, ia := range ifaceAddrs { + // must match the first protocol to be resolve. + if ia.Protocols()[0].Code != resolve.Protocols()[0].Code { + continue + } + + split[0] = ia + joined := ma.Join(split...) + out = append(out, joined) + log.Debug("adding resolved addr:", resolve, joined, out) + } + if len(out) < 1 { + return nil, fmt.Errorf("failed to resolve: %s", resolve) + } + return out, nil +} + +// ResolveUnspecifiedAddresses expands unspecified ip addresses (/ip4/0.0.0.0, /ip6/::) to +// use the known local interfaces. +func ResolveUnspecifiedAddresses(unspecAddrs, ifaceAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { + + // todo optimize: only fetch these if we have a "any" addr. + if len(ifaceAddrs) < 1 { + var err error + ifaceAddrs, err = InterfaceAddresses() + if err != nil { + return nil, err + } + // log.Debug("InterfaceAddresses:", ifaceAddrs) + } + + var outputAddrs []ma.Multiaddr + for _, a := range unspecAddrs { + // unspecified? + resolved, err := ResolveUnspecifiedAddress(a, ifaceAddrs) + if err != nil { + continue // optimistic. if we cant resolve anything, we'll know at the bottom. + } + // log.Debug("resolved:", a, resolved) + outputAddrs = append(outputAddrs, resolved...) + } + + if len(outputAddrs) < 1 { + return nil, fmt.Errorf("failed to specify addrs: %s", unspecAddrs) + } + + log.Event(context.TODO(), "interfaceListenAddresses", func() logging.Loggable { + var addrs []string + for _, addr := range outputAddrs { + addrs = append(addrs, addr.String()) + } + return logging.Metadata{"addresses": addrs} + }()) + + log.Debug("ResolveUnspecifiedAddresses:", unspecAddrs, ifaceAddrs, outputAddrs) + return outputAddrs, nil +} + +// InterfaceAddresses returns a list of addresses associated with local machine +// Note: we do not return link local addresses. IP loopback is ok, because we +// may be connecting to other nodes in the same machine. +func InterfaceAddresses() ([]ma.Multiaddr, error) { + maddrs, err := manet.InterfaceMultiaddrs() + if err != nil { + return nil, err + } + log.Debug("InterfaceAddresses: from manet:", maddrs) + + var out []ma.Multiaddr + for _, a := range maddrs { + if !AddrUsable(a, true) { // partial + // log.Debug("InterfaceAddresses: skipping unusable:", a) + continue + } + + out = append(out, a) + } + + log.Debug("InterfaceAddresses: usable:", out) + return out, nil +} + +// AddrInList returns whether or not an address is part of a list. +// this is useful to check if NAT is happening (or other bugs?) +func AddrInList(addr ma.Multiaddr, list []ma.Multiaddr) bool { + for _, addr2 := range list { + if addr.Equal(addr2) { + return true + } + } + return false +} + +// AddrIsShareableOnWAN returns whether the given address should be shareable on the +// wide area network (wide internet). +func AddrIsShareableOnWAN(addr ma.Multiaddr) bool { + s := ma.Split(addr) + if len(s) < 1 { + return false + } + a := s[0] + if manet.IsIPLoopback(a) || manet.IsIP6LinkLocal(a) || manet.IsIPUnspecified(a) { + return false + } + return manet.IsThinWaist(a) +} + +// WANShareableAddrs filters addresses based on whether they're shareable on WAN +func WANShareableAddrs(inp []ma.Multiaddr) []ma.Multiaddr { + return FilterAddrs(inp, AddrIsShareableOnWAN) +} + +// Subtract filters out all addrs in b from a +func Subtract(a, b []ma.Multiaddr) []ma.Multiaddr { + return FilterAddrs(a, func(m ma.Multiaddr) bool { + for _, bb := range b { + if m.Equal(bb) { + return false + } + } + return true + }) +} + +// CheckNATWarning checks if our observed addresses differ. if so, +// informs the user that certain things might not work yet +func CheckNATWarning(observed, expected ma.Multiaddr, listen []ma.Multiaddr) { + if observed.Equal(expected) { + return + } + + if !AddrInList(observed, listen) { // probably a nat + log.Warningf(natWarning, observed, listen) + } +} + +const natWarning = `Remote peer observed our address to be: %s +The local addresses are: %s +Thus, connection is going through NAT, and other connections may fail. + +IPFS NAT traversal is still under development. Please bug us on github or irc to fix this. +Baby steps: http://jbenet.static.s3.amazonaws.com/271dfcf/baby-steps.gif +` diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go new file mode 100644 index 0000000000..eb843ffc09 --- /dev/null +++ b/p2p/net/swarm/addr/addr_test.go @@ -0,0 +1,232 @@ +package addrutil + +import ( + "testing" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" +) + +func newMultiaddr(t *testing.T, s string) ma.Multiaddr { + maddr, err := ma.NewMultiaddr(s) + if err != nil { + t.Fatal(err) + } + return maddr +} + +func TestFilterAddrs(t *testing.T) { + + bad := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/1.2.3.4/udp/1234"), // unreliable + newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/sctp/1234"), // not in manet + newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/utp"), // utp is broken + newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/udt"), // udt is broken on arm + newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), // link local + newMultiaddr(t, "/ip6/fe80::100/tcp/1234"), // link local + } + + good := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), + newMultiaddr(t, "/ip6/::1/tcp/1234"), + } + + goodAndBad := append(good, bad...) + + // test filters + + for _, a := range bad { + if AddrUsable(a, false) { + t.Errorf("addr %s should be unusable", a) + } + if AddrUsable(a, true) { + t.Errorf("addr %s should be unusable", a) + } + } + + for _, a := range good { + if !AddrUsable(a, false) { + t.Errorf("addr %s should be usable", a) + } + if !AddrUsable(a, true) { + t.Errorf("addr %s should be usable", a) + } + } + + subtestAddrsEqual(t, FilterUsableAddrs(bad), []ma.Multiaddr{}) + subtestAddrsEqual(t, FilterUsableAddrs(good), good) + subtestAddrsEqual(t, FilterUsableAddrs(goodAndBad), good) +} + +func subtestAddrsEqual(t *testing.T, a, b []ma.Multiaddr) { + if len(a) != len(b) { + t.Error(t) + } + + in := func(addr ma.Multiaddr, l []ma.Multiaddr) bool { + for _, addr2 := range l { + if addr.Equal(addr2) { + return true + } + } + return false + } + + for _, aa := range a { + if !in(aa, b) { + t.Errorf("%s not in %s", aa, b) + } + } +} + +func TestInterfaceAddrs(t *testing.T) { + addrs, err := InterfaceAddresses() + if err != nil { + t.Fatal(err) + } + + if len(addrs) < 1 { + t.Error("no addresses") + } + + for _, a := range addrs { + if manet.IsIP6LinkLocal(a) { + t.Error("should not return ip link local addresses", a) + } + } + + if len(addrs) < 1 { + t.Error("no good interface addrs") + } +} + +func TestResolvingAddrs(t *testing.T) { + + unspec := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/0.0.0.0/tcp/1234"), + newMultiaddr(t, "/ip4/1.2.3.4/tcp/1234"), + newMultiaddr(t, "/ip6/::/tcp/1234"), + newMultiaddr(t, "/ip6/::100/tcp/1234"), + } + + iface := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/127.0.0.1"), + newMultiaddr(t, "/ip4/10.20.30.40"), + newMultiaddr(t, "/ip6/::1"), + newMultiaddr(t, "/ip6/::f"), + } + + spec := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), + newMultiaddr(t, "/ip4/10.20.30.40/tcp/1234"), + newMultiaddr(t, "/ip4/1.2.3.4/tcp/1234"), + newMultiaddr(t, "/ip6/::1/tcp/1234"), + newMultiaddr(t, "/ip6/::f/tcp/1234"), + newMultiaddr(t, "/ip6/::100/tcp/1234"), + } + + actual, err := ResolveUnspecifiedAddresses(unspec, iface) + if err != nil { + t.Fatal(err) + } + + for i, a := range actual { + if !a.Equal(spec[i]) { + t.Error(a, " != ", spec[i]) + } + } + + ip4u := []ma.Multiaddr{newMultiaddr(t, "/ip4/0.0.0.0")} + ip4i := []ma.Multiaddr{newMultiaddr(t, "/ip4/1.2.3.4")} + + ip6u := []ma.Multiaddr{newMultiaddr(t, "/ip6/::")} + ip6i := []ma.Multiaddr{newMultiaddr(t, "/ip6/::1")} + + if _, err := ResolveUnspecifiedAddress(ip4u[0], ip6i); err == nil { + t.Fatal("should have failed") + } + if _, err := ResolveUnspecifiedAddress(ip6u[0], ip4i); err == nil { + t.Fatal("should have failed") + } + + if _, err := ResolveUnspecifiedAddresses(ip6u, ip4i); err == nil { + t.Fatal("should have failed") + } + if _, err := ResolveUnspecifiedAddresses(ip4u, ip6i); err == nil { + t.Fatal("should have failed") + } + +} + +func TestWANShareable(t *testing.T) { + + wanok := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/1.2.3.4/tcp/1234"), + newMultiaddr(t, "/ip6/abcd::1/tcp/1234"), + } + + wanbad := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), + newMultiaddr(t, "/ip4/0.0.0.0/tcp/1234"), + newMultiaddr(t, "/ip6/::1/tcp/1234"), + newMultiaddr(t, "/ip6/::/tcp/1234"), + newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), + newMultiaddr(t, "/ip6/fe80::/tcp/1234"), + } + + for _, a := range wanok { + if !AddrIsShareableOnWAN(a) { + t.Error("should be true", a) + } + } + + for _, a := range wanbad { + if AddrIsShareableOnWAN(a) { + t.Error("should be false", a) + } + } + + wanok2 := WANShareableAddrs(wanok) + if len(wanok) != len(wanok2) { + t.Error("should be the same") + } + + wanbad2 := WANShareableAddrs(wanbad) + if len(wanbad2) != 0 { + t.Error("should be zero") + } +} + +func TestSubtract(t *testing.T) { + + a := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), + newMultiaddr(t, "/ip4/0.0.0.0/tcp/1234"), + newMultiaddr(t, "/ip6/::1/tcp/1234"), + newMultiaddr(t, "/ip6/::/tcp/1234"), + newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), + newMultiaddr(t, "/ip6/fe80::/tcp/1234"), + } + + b := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), + newMultiaddr(t, "/ip6/::1/tcp/1234"), + newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), + } + + c1 := []ma.Multiaddr{ + newMultiaddr(t, "/ip4/0.0.0.0/tcp/1234"), + newMultiaddr(t, "/ip6/::/tcp/1234"), + newMultiaddr(t, "/ip6/fe80::/tcp/1234"), + } + + c2 := Subtract(a, b) + if len(c1) != len(c2) { + t.Error("should be the same") + } + for i, ca := range c1 { + if !c2[i].Equal(ca) { + t.Error("should be the same", ca, c2[i]) + } + } +} diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go new file mode 100644 index 0000000000..0ed10f4d30 --- /dev/null +++ b/p2p/net/swarm/dial_test.go @@ -0,0 +1,440 @@ +package swarm + +import ( + "net" + "sync" + "testing" + "time" + + addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" + peer "github.com/ipfs/go-ipfs/p2p/peer" + + testutil "github.com/ipfs/go-ipfs/util/testutil" + ci "github.com/ipfs/go-ipfs/util/testutil/ci" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +func acceptAndHang(l net.Listener) { + conns := make([]net.Conn, 0, 10) + for { + c, err := l.Accept() + if err != nil { + break + } + if c != nil { + conns = append(conns, c) + } + } + for _, c := range conns { + c.Close() + } +} + +func TestSimultDials(t *testing.T) { + // t.Skip("skipping for another test") + t.Parallel() + + ctx := context.Background() + swarms := makeSwarms(ctx, t, 2) + + // connect everyone + { + var wg sync.WaitGroup + connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { + // copy for other peer + log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.local, dst, addr) + s.peers.AddAddr(dst, addr, peer.TempAddrTTL) + if _, err := s.Dial(ctx, dst); err != nil { + t.Fatal("error swarm dialing to peer", err) + } + wg.Done() + } + + ifaceAddrs0, err := swarms[0].InterfaceListenAddresses() + if err != nil { + t.Fatal(err) + } + ifaceAddrs1, err := swarms[1].InterfaceListenAddresses() + if err != nil { + t.Fatal(err) + } + + log.Info("Connecting swarms simultaneously.") + for i := 0; i < 10; i++ { // connect 10x for each. + wg.Add(2) + go connect(swarms[0], swarms[1].local, ifaceAddrs1[0]) + go connect(swarms[1], swarms[0].local, ifaceAddrs0[0]) + } + wg.Wait() + } + + // should still just have 1, at most 2 connections :) + c01l := len(swarms[0].ConnectionsToPeer(swarms[1].local)) + if c01l > 2 { + t.Error("0->1 has", c01l) + } + c10l := len(swarms[1].ConnectionsToPeer(swarms[0].local)) + if c10l > 2 { + t.Error("1->0 has", c10l) + } + + for _, s := range swarms { + s.Close() + } +} + +func newSilentPeer(t *testing.T) (peer.ID, ma.Multiaddr, net.Listener) { + dst := testutil.RandPeerIDFatal(t) + lst, err := net.Listen("tcp", ":0") + if err != nil { + t.Fatal(err) + } + addr, err := manet.FromNetAddr(lst.Addr()) + if err != nil { + t.Fatal(err) + } + addrs := []ma.Multiaddr{addr} + addrs, err = addrutil.ResolveUnspecifiedAddresses(addrs, nil) + if err != nil { + t.Fatal(err) + } + t.Log("new silent peer:", dst, addrs[0]) + return dst, addrs[0], lst +} + +func TestDialWait(t *testing.T) { + // t.Skip("skipping for another test") + t.Parallel() + + ctx := context.Background() + swarms := makeSwarms(ctx, t, 1) + s1 := swarms[0] + defer s1.Close() + + s1.dialT = time.Millisecond * 300 // lower timeout for tests. + if ci.IsRunning() { + s1.dialT = time.Second + } + + // dial to a non-existent peer. + s2p, s2addr, s2l := newSilentPeer(t) + go acceptAndHang(s2l) + defer s2l.Close() + s1.peers.AddAddr(s2p, s2addr, peer.PermanentAddrTTL) + + before := time.Now() + if c, err := s1.Dial(ctx, s2p); err == nil { + defer c.Close() + t.Fatal("error swarm dialing to unknown peer worked...", err) + } else { + t.Log("correctly got error:", err) + } + duration := time.Now().Sub(before) + + dt := s1.dialT + if duration < dt*dialAttempts { + t.Error("< DialTimeout * dialAttempts not being respected", duration, dt*dialAttempts) + } + if duration > 2*dt*dialAttempts { + t.Error("> 2*DialTimeout * dialAttempts not being respected", duration, 2*dt*dialAttempts) + } + + if !s1.backf.Backoff(s2p) { + t.Error("s2 should now be on backoff") + } +} + +func TestDialBackoff(t *testing.T) { + // t.Skip("skipping for another test") + if ci.IsRunning() { + t.Skip("travis and jenkins will never have fun with this test") + } + + t.Parallel() + + ctx := context.Background() + swarms := makeSwarms(ctx, t, 2) + s1 := swarms[0] + s2 := swarms[1] + defer s1.Close() + defer s2.Close() + + s1.dialT = time.Second // lower timeout for tests. + s2.dialT = time.Second // lower timeout for tests. + + s2addrs, err := s2.InterfaceListenAddresses() + if err != nil { + t.Fatal(err) + } + s1.peers.AddAddrs(s2.local, s2addrs, peer.PermanentAddrTTL) + + // dial to a non-existent peer. + s3p, s3addr, s3l := newSilentPeer(t) + go acceptAndHang(s3l) + defer s3l.Close() + s1.peers.AddAddr(s3p, s3addr, peer.PermanentAddrTTL) + + // in this test we will: + // 1) dial 10x to each node. + // 2) all dials should hang + // 3) s1->s2 should succeed. + // 4) s1->s3 should not (and should place s3 on backoff) + // 5) disconnect entirely + // 6) dial 10x to each node again + // 7) s3 dials should all return immediately (except 1) + // 8) s2 dials should all hang, and succeed + // 9) last s3 dial ends, unsuccessful + + dialOnlineNode := func(dst peer.ID, times int) <-chan bool { + ch := make(chan bool) + for i := 0; i < times; i++ { + go func() { + if _, err := s1.Dial(ctx, dst); err != nil { + t.Error("error dialing", dst, err) + ch <- false + } else { + ch <- true + } + }() + } + return ch + } + + dialOfflineNode := func(dst peer.ID, times int) <-chan bool { + ch := make(chan bool) + for i := 0; i < times; i++ { + go func() { + if c, err := s1.Dial(ctx, dst); err != nil { + ch <- false + } else { + t.Error("succeeded in dialing", dst) + ch <- true + c.Close() + } + }() + } + return ch + } + + { + // 1) dial 10x to each node. + N := 10 + s2done := dialOnlineNode(s2.local, N) + s3done := dialOfflineNode(s3p, N) + + // when all dials should be done by: + dialTimeout1x := time.After(s1.dialT) + // dialTimeout1Ax := time.After(s1.dialT * 2) // dialAttempts) + dialTimeout10Ax := time.After(s1.dialT * 2 * 10) // dialAttempts * 10) + + // 2) all dials should hang + select { + case <-s2done: + t.Error("s2 should not happen immediately") + case <-s3done: + t.Error("s3 should not happen yet") + case <-time.After(time.Millisecond): + // s2 may finish very quickly, so let's get out. + } + + // 3) s1->s2 should succeed. + for i := 0; i < N; i++ { + select { + case r := <-s2done: + if !r { + t.Error("s2 should not fail") + } + case <-s3done: + t.Error("s3 should not happen yet") + case <-dialTimeout1x: + t.Error("s2 took too long") + } + } + + select { + case <-s2done: + t.Error("s2 should have no more") + case <-s3done: + t.Error("s3 should not happen yet") + case <-dialTimeout1x: // let it pass + } + + // 4) s1->s3 should not (and should place s3 on backoff) + // N-1 should finish before dialTimeout1x * 2 + for i := 0; i < N; i++ { + select { + case <-s2done: + t.Error("s2 should have no more") + case r := <-s3done: + if r { + t.Error("s3 should not succeed") + } + case <-(dialTimeout1x): + if i < (N - 1) { + t.Fatal("s3 took too long") + } + t.Log("dialTimeout1x * 1.3 hit for last peer") + case <-dialTimeout10Ax: + t.Fatal("s3 took too long") + } + } + + // check backoff state + if s1.backf.Backoff(s2.local) { + t.Error("s2 should not be on backoff") + } + if !s1.backf.Backoff(s3p) { + t.Error("s3 should be on backoff") + } + + // 5) disconnect entirely + + for _, c := range s1.Connections() { + c.Close() + } + for i := 0; i < 100 && len(s1.Connections()) > 0; i++ { + <-time.After(time.Millisecond) + } + if len(s1.Connections()) > 0 { + t.Fatal("s1 conns must exit") + } + } + + { + // 6) dial 10x to each node again + N := 10 + s2done := dialOnlineNode(s2.local, N) + s3done := dialOfflineNode(s3p, N) + + // when all dials should be done by: + dialTimeout1x := time.After(s1.dialT) + // dialTimeout1Ax := time.After(s1.dialT * 2) // dialAttempts) + dialTimeout10Ax := time.After(s1.dialT * 2 * 10) // dialAttempts * 10) + + // 7) s3 dials should all return immediately (except 1) + for i := 0; i < N-1; i++ { + select { + case <-s2done: + t.Error("s2 should not succeed yet") + case r := <-s3done: + if r { + t.Error("s3 should not succeed") + } + case <-dialTimeout1x: + t.Fatal("s3 took too long") + } + } + + // 8) s2 dials should all hang, and succeed + for i := 0; i < N; i++ { + select { + case r := <-s2done: + if !r { + t.Error("s2 should succeed") + } + // case <-s3done: + case <-(dialTimeout1x): + t.Fatal("s3 took too long") + } + } + + // 9) the last s3 should return, failed. + select { + case <-s2done: + t.Error("s2 should have no more") + case r := <-s3done: + if r { + t.Error("s3 should not succeed") + } + case <-dialTimeout10Ax: + t.Fatal("s3 took too long") + } + + // check backoff state (the same) + if s1.backf.Backoff(s2.local) { + t.Error("s2 should not be on backoff") + } + if !s1.backf.Backoff(s3p) { + t.Error("s3 should be on backoff") + } + + } +} + +func TestDialBackoffClears(t *testing.T) { + // t.Skip("skipping for another test") + t.Parallel() + + ctx := context.Background() + swarms := makeSwarms(ctx, t, 2) + s1 := swarms[0] + s2 := swarms[1] + defer s1.Close() + defer s2.Close() + s1.dialT = time.Millisecond * 300 // lower timeout for tests. + s2.dialT = time.Millisecond * 300 // lower timeout for tests. + if ci.IsRunning() { + s1.dialT = 2 * time.Second + s2.dialT = 2 * time.Second + } + + // use another address first, that accept and hang on conns + _, s2bad, s2l := newSilentPeer(t) + go acceptAndHang(s2l) + defer s2l.Close() + + // phase 1 -- dial to non-operational addresses + s1.peers.AddAddr(s2.local, s2bad, peer.PermanentAddrTTL) + + before := time.Now() + if c, err := s1.Dial(ctx, s2.local); err == nil { + t.Fatal("dialing to broken addr worked...", err) + defer c.Close() + } else { + t.Log("correctly got error:", err) + } + duration := time.Now().Sub(before) + + dt := s1.dialT + if duration < dt*dialAttempts { + t.Error("< DialTimeout * dialAttempts not being respected", duration, dt*dialAttempts) + } + if duration > 2*dt*dialAttempts { + t.Error("> 2*DialTimeout * dialAttempts not being respected", duration, 2*dt*dialAttempts) + } + + if !s1.backf.Backoff(s2.local) { + t.Error("s2 should now be on backoff") + } else { + t.Log("correctly added to backoff") + } + + // phase 2 -- add the working address. dial should succeed. + ifaceAddrs1, err := swarms[1].InterfaceListenAddresses() + if err != nil { + t.Fatal(err) + } + s1.peers.AddAddrs(s2.local, ifaceAddrs1, peer.PermanentAddrTTL) + + before = time.Now() + if c, err := s1.Dial(ctx, s2.local); err != nil { + t.Fatal(err) + } else { + c.Close() + t.Log("correctly connected") + } + duration = time.Now().Sub(before) + + if duration >= dt { + // t.Error("took too long", duration, dt) + } + + if s1.backf.Backoff(s2.local) { + t.Error("s2 should no longer be on backoff") + } else { + t.Log("correctly cleared backoff") + } +} diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go new file mode 100644 index 0000000000..2f8b07ef0b --- /dev/null +++ b/p2p/net/swarm/peers_test.go @@ -0,0 +1,72 @@ +package swarm + +import ( + "testing" + + peer "github.com/ipfs/go-ipfs/p2p/peer" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +func TestPeers(t *testing.T) { + // t.Skip("skipping for another test") + + ctx := context.Background() + swarms := makeSwarms(ctx, t, 2) + s1 := swarms[0] + s2 := swarms[1] + + connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { + // TODO: make a DialAddr func. + s.peers.AddAddr(dst, addr, peer.PermanentAddrTTL) + // t.Logf("connections from %s", s.LocalPeer()) + // for _, c := range s.ConnectionsToPeer(dst) { + // t.Logf("connection from %s to %s: %v", s.LocalPeer(), dst, c) + // } + // t.Logf("") + if _, err := s.Dial(ctx, dst); err != nil { + t.Fatal("error swarm dialing to peer", err) + } + // t.Log(s.swarm.Dump()) + } + + s1GotConn := make(chan struct{}, 0) + s2GotConn := make(chan struct{}, 0) + s1.SetConnHandler(func(c *Conn) { + s1GotConn <- struct{}{} + }) + s2.SetConnHandler(func(c *Conn) { + s2GotConn <- struct{}{} + }) + + connect(s1, s2.LocalPeer(), s2.ListenAddresses()[0]) + <-s2GotConn // have to wait here so the other side catches up. + connect(s2, s1.LocalPeer(), s1.ListenAddresses()[0]) + + for i := 0; i < 100; i++ { + connect(s1, s2.LocalPeer(), s2.ListenAddresses()[0]) + connect(s2, s1.LocalPeer(), s1.ListenAddresses()[0]) + } + + for _, s := range swarms { + log.Infof("%s swarm routing table: %s", s.local, s.Peers()) + } + + test := func(s *Swarm) { + expect := 1 + actual := len(s.Peers()) + if actual != expect { + t.Errorf("%s has %d peers, not %d: %v", s.LocalPeer(), actual, expect, s.Peers()) + t.Log(s.swarm.Dump()) + } + actual = len(s.Connections()) + if actual != expect { + t.Errorf("%s has %d conns, not %d: %v", s.LocalPeer(), actual, expect, s.Connections()) + t.Log(s.swarm.Dump()) + } + } + + test(s1) + test(s2) +} diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go new file mode 100644 index 0000000000..dfb775c971 --- /dev/null +++ b/p2p/net/swarm/simul_test.go @@ -0,0 +1,75 @@ +package swarm + +import ( + "sync" + "testing" + "time" + + peer "github.com/ipfs/go-ipfs/p2p/peer" + ci "github.com/ipfs/go-ipfs/util/testutil/ci" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +func TestSimultOpen(t *testing.T) { + // t.Skip("skipping for another test") + t.Parallel() + + ctx := context.Background() + swarms := makeSwarms(ctx, t, 2) + + // connect everyone + { + var wg sync.WaitGroup + connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { + // copy for other peer + log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.local, dst, addr) + s.peers.AddAddr(dst, addr, peer.PermanentAddrTTL) + if _, err := s.Dial(ctx, dst); err != nil { + t.Fatal("error swarm dialing to peer", err) + } + wg.Done() + } + + log.Info("Connecting swarms simultaneously.") + wg.Add(2) + go connect(swarms[0], swarms[1].local, swarms[1].ListenAddresses()[0]) + go connect(swarms[1], swarms[0].local, swarms[0].ListenAddresses()[0]) + wg.Wait() + } + + for _, s := range swarms { + s.Close() + } +} + +func TestSimultOpenMany(t *testing.T) { + // t.Skip("very very slow") + + addrs := 20 + rounds := 10 + if ci.IsRunning() { + addrs = 10 + rounds = 5 + } + SubtestSwarm(t, addrs, rounds) +} + +func TestSimultOpenFewStress(t *testing.T) { + if testing.Short() { + t.SkipNow() + } + // t.Skip("skipping for another test") + t.Parallel() + + msgs := 40 + swarms := 2 + rounds := 10 + // rounds := 100 + + for i := 0; i < rounds; i++ { + SubtestSwarm(t, swarms, msgs) + <-time.After(10 * time.Millisecond) + } +} diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go new file mode 100644 index 0000000000..512369013c --- /dev/null +++ b/p2p/net/swarm/swarm.go @@ -0,0 +1,321 @@ +// package swarm implements a connection muxer with a pair of channels +// to synchronize all network communication. +package swarm + +import ( + "fmt" + "sync" + "time" + + metrics "github.com/ipfs/go-ipfs/metrics" + inet "github.com/ipfs/go-ipfs/p2p/net" + filter "github.com/ipfs/go-ipfs/p2p/net/filter" + addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" + peer "github.com/ipfs/go-ipfs/p2p/peer" + logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" + pst "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-stream-muxer" + psy "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-stream-muxer/yamux" + "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" + goprocessctx "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context" + prom "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus" + mafilter "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/whyrusleeping/multiaddr-filter" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +var log = logging.Logger("swarm2") + +var PSTransport pst.Transport + +var peersTotal = prom.NewGaugeVec(prom.GaugeOpts{ + Namespace: "ipfs", + Subsystem: "p2p", + Name: "peers_total", + Help: "Number of connected peers", +}, []string{"peer_id"}) + +func init() { + tpt := *psy.DefaultTransport + tpt.MaxStreamWindowSize = 512 * 1024 + PSTransport = &tpt +} + +// Swarm is a connection muxer, allowing connections to other peers to +// be opened and closed, while still using the same Chan for all +// communication. The Chan sends/receives Messages, which note the +// destination or source Peer. +// +// Uses peerstream.Swarm +type Swarm struct { + swarm *ps.Swarm + local peer.ID + peers peer.Peerstore + connh ConnHandler + + dsync dialsync + backf dialbackoff + dialT time.Duration // mainly for tests + + notifmu sync.RWMutex + notifs map[inet.Notifiee]ps.Notifiee + + // filters for addresses that shouldnt be dialed + Filters *filter.Filters + + proc goprocess.Process + ctx context.Context + bwc metrics.Reporter +} + +// NewSwarm constructs a Swarm, with a Chan. +func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, + local peer.ID, peers peer.Peerstore, bwc metrics.Reporter) (*Swarm, error) { + + listenAddrs, err := filterAddrs(listenAddrs) + if err != nil { + return nil, err + } + + s := &Swarm{ + swarm: ps.NewSwarm(PSTransport), + local: local, + peers: peers, + ctx: ctx, + dialT: DialTimeout, + notifs: make(map[inet.Notifiee]ps.Notifiee), + bwc: bwc, + Filters: filter.NewFilters(), + } + + // configure Swarm + s.proc = goprocessctx.WithContextAndTeardown(ctx, s.teardown) + s.SetConnHandler(nil) // make sure to setup our own conn handler. + + // setup swarm metrics + prom.MustRegisterOrGet(peersTotal) + s.Notify((*metricsNotifiee)(s)) + + return s, s.listen(listenAddrs) +} + +func (s *Swarm) teardown() error { + return s.swarm.Close() +} + +func (s *Swarm) AddAddrFilter(f string) error { + m, err := mafilter.NewMask(f) + if err != nil { + return err + } + + s.Filters.AddDialFilter(m) + return nil +} +func filterAddrs(listenAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { + if len(listenAddrs) > 0 { + filtered := addrutil.FilterUsableAddrs(listenAddrs) + if len(filtered) < 1 { + return nil, fmt.Errorf("swarm cannot use any addr in: %s", listenAddrs) + } + listenAddrs = filtered + } + return listenAddrs, nil +} + +func (s *Swarm) Listen(addrs ...ma.Multiaddr) error { + addrs, err := filterAddrs(addrs) + if err != nil { + return err + } + + return s.listen(addrs) +} + +// Process returns the Process of the swarm +func (s *Swarm) Process() goprocess.Process { + return s.proc +} + +// Context returns the context of the swarm +func (s *Swarm) Context() context.Context { + return s.ctx +} + +// Close stops the Swarm. +func (s *Swarm) Close() error { + return s.proc.Close() +} + +// StreamSwarm returns the underlying peerstream.Swarm +func (s *Swarm) StreamSwarm() *ps.Swarm { + return s.swarm +} + +// SetConnHandler assigns the handler for new connections. +// See peerstream. You will rarely use this. See SetStreamHandler +func (s *Swarm) SetConnHandler(handler ConnHandler) { + + // handler is nil if user wants to clear the old handler. + if handler == nil { + s.swarm.SetConnHandler(func(psconn *ps.Conn) { + s.connHandler(psconn) + }) + return + } + + s.swarm.SetConnHandler(func(psconn *ps.Conn) { + // sc is nil if closed in our handler. + if sc := s.connHandler(psconn); sc != nil { + // call the user's handler. in a goroutine for sync safety. + go handler(sc) + } + }) +} + +// SetStreamHandler assigns the handler for new streams. +// See peerstream. +func (s *Swarm) SetStreamHandler(handler inet.StreamHandler) { + s.swarm.SetStreamHandler(func(s *ps.Stream) { + handler(wrapStream(s)) + }) +} + +// NewStreamWithPeer creates a new stream on any available connection to p +func (s *Swarm) NewStreamWithPeer(p peer.ID) (*Stream, error) { + // if we have no connections, try connecting. + if len(s.ConnectionsToPeer(p)) == 0 { + log.Debug("Swarm: NewStreamWithPeer no connections. Attempting to connect...") + if _, err := s.Dial(s.Context(), p); err != nil { + return nil, err + } + } + log.Debug("Swarm: NewStreamWithPeer...") + + st, err := s.swarm.NewStreamWithGroup(p) + return wrapStream(st), err +} + +// StreamsWithPeer returns all the live Streams to p +func (s *Swarm) StreamsWithPeer(p peer.ID) []*Stream { + return wrapStreams(ps.StreamsWithGroup(p, s.swarm.Streams())) +} + +// ConnectionsToPeer returns all the live connections to p +func (s *Swarm) ConnectionsToPeer(p peer.ID) []*Conn { + return wrapConns(ps.ConnsWithGroup(p, s.swarm.Conns())) +} + +// Connections returns a slice of all connections. +func (s *Swarm) Connections() []*Conn { + return wrapConns(s.swarm.Conns()) +} + +// CloseConnection removes a given peer from swarm + closes the connection +func (s *Swarm) CloseConnection(p peer.ID) error { + conns := s.swarm.ConnsWithGroup(p) // boom. + for _, c := range conns { + c.Close() + } + return nil +} + +// Peers returns a copy of the set of peers swarm is connected to. +func (s *Swarm) Peers() []peer.ID { + conns := s.Connections() + + seen := make(map[peer.ID]struct{}) + peers := make([]peer.ID, 0, len(conns)) + for _, c := range conns { + p := c.RemotePeer() + if _, found := seen[p]; found { + continue + } + + seen[p] = struct{}{} + peers = append(peers, p) + } + return peers +} + +// LocalPeer returns the local peer swarm is associated to. +func (s *Swarm) LocalPeer() peer.ID { + return s.local +} + +// notifyAll sends a signal to all Notifiees +func (s *Swarm) notifyAll(notify func(inet.Notifiee)) { + s.notifmu.RLock() + for f := range s.notifs { + go notify(f) + } + s.notifmu.RUnlock() +} + +// Notify signs up Notifiee to receive signals when events happen +func (s *Swarm) Notify(f inet.Notifiee) { + // wrap with our notifiee, to translate function calls + n := &ps2netNotifee{net: (*Network)(s), not: f} + + s.notifmu.Lock() + s.notifs[f] = n + s.notifmu.Unlock() + + // register for notifications in the peer swarm. + s.swarm.Notify(n) +} + +// StopNotify unregisters Notifiee fromr receiving signals +func (s *Swarm) StopNotify(f inet.Notifiee) { + s.notifmu.Lock() + n, found := s.notifs[f] + if found { + delete(s.notifs, f) + } + s.notifmu.Unlock() + + if found { + s.swarm.StopNotify(n) + } +} + +type ps2netNotifee struct { + net *Network + not inet.Notifiee +} + +func (n *ps2netNotifee) Connected(c *ps.Conn) { + n.not.Connected(n.net, inet.Conn((*Conn)(c))) +} + +func (n *ps2netNotifee) Disconnected(c *ps.Conn) { + n.not.Disconnected(n.net, inet.Conn((*Conn)(c))) +} + +func (n *ps2netNotifee) OpenedStream(s *ps.Stream) { + n.not.OpenedStream(n.net, inet.Stream((*Stream)(s))) +} + +func (n *ps2netNotifee) ClosedStream(s *ps.Stream) { + n.not.ClosedStream(n.net, inet.Stream((*Stream)(s))) +} + +type metricsNotifiee Swarm + +func (nn *metricsNotifiee) Connected(n inet.Network, v inet.Conn) { + peersTotalGauge(n.LocalPeer()).Set(float64(len(n.Conns()))) +} + +func (nn *metricsNotifiee) Disconnected(n inet.Network, v inet.Conn) { + peersTotalGauge(n.LocalPeer()).Set(float64(len(n.Conns()))) +} + +func (nn *metricsNotifiee) OpenedStream(n inet.Network, v inet.Stream) {} +func (nn *metricsNotifiee) ClosedStream(n inet.Network, v inet.Stream) {} +func (nn *metricsNotifiee) Listen(n inet.Network, a ma.Multiaddr) {} +func (nn *metricsNotifiee) ListenClose(n inet.Network, a ma.Multiaddr) {} + +func peersTotalGauge(id peer.ID) prom.Gauge { + return peersTotal.With(prom.Labels{"peer_id": id.Pretty()}) +} diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go new file mode 100644 index 0000000000..8f9360d9d0 --- /dev/null +++ b/p2p/net/swarm/swarm_addr.go @@ -0,0 +1,39 @@ +package swarm + +import ( + conn "github.com/ipfs/go-ipfs/p2p/net/conn" + addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" +) + +// ListenAddresses returns a list of addresses at which this swarm listens. +func (s *Swarm) ListenAddresses() []ma.Multiaddr { + listeners := s.swarm.Listeners() + addrs := make([]ma.Multiaddr, 0, len(listeners)) + for _, l := range listeners { + if l2, ok := l.NetListener().(conn.Listener); ok { + addrs = append(addrs, l2.Multiaddr()) + } + } + return addrs +} + +// InterfaceListenAddresses returns a list of addresses at which this swarm +// listens. It expands "any interface" addresses (/ip4/0.0.0.0, /ip6/::) to +// use the known local interfaces. +func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { + return addrutil.ResolveUnspecifiedAddresses(s.ListenAddresses(), nil) +} + +// checkNATWarning checks if our observed addresses differ. if so, +// informs the user that certain things might not work yet +func checkNATWarning(s *Swarm, observed ma.Multiaddr, expected ma.Multiaddr) { + listen, err := s.InterfaceListenAddresses() + if err != nil { + log.Debugf("Error retrieving swarm.InterfaceListenAddresses: %s", err) + return + } + + addrutil.CheckNATWarning(observed, expected, listen) +} diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go new file mode 100644 index 0000000000..b75b491c42 --- /dev/null +++ b/p2p/net/swarm/swarm_addr_test.go @@ -0,0 +1,123 @@ +package swarm + +import ( + "testing" + + metrics "github.com/ipfs/go-ipfs/metrics" + addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" + peer "github.com/ipfs/go-ipfs/p2p/peer" + testutil "github.com/ipfs/go-ipfs/util/testutil" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +func TestFilterAddrs(t *testing.T) { + + m := func(s string) ma.Multiaddr { + maddr, err := ma.NewMultiaddr(s) + if err != nil { + t.Fatal(err) + } + return maddr + } + + bad := []ma.Multiaddr{ + m("/ip4/1.2.3.4/udp/1234"), // unreliable + m("/ip4/1.2.3.4/udp/1234/sctp/1234"), // not in manet + m("/ip4/1.2.3.4/udp/1234/utp"), // utp is broken + m("/ip4/1.2.3.4/udp/1234/udt"), // udt is broken on arm + m("/ip6/fe80::1/tcp/0"), // link local + m("/ip6/fe80::100/tcp/1234"), // link local + } + + good := []ma.Multiaddr{ + m("/ip4/127.0.0.1/tcp/0"), + m("/ip6/::1/tcp/0"), + } + + goodAndBad := append(good, bad...) + + // test filters + + for _, a := range bad { + if addrutil.AddrUsable(a, true) { + t.Errorf("addr %s should be unusable", a) + } + } + + for _, a := range good { + if !addrutil.AddrUsable(a, true) { + t.Errorf("addr %s should be usable", a) + } + } + + subtestAddrsEqual(t, addrutil.FilterUsableAddrs(bad), []ma.Multiaddr{}) + subtestAddrsEqual(t, addrutil.FilterUsableAddrs(good), good) + subtestAddrsEqual(t, addrutil.FilterUsableAddrs(goodAndBad), good) + + // now test it with swarm + + id, err := testutil.RandPeerID() + if err != nil { + t.Fatal(err) + } + + ps := peer.NewPeerstore() + ctx := context.Background() + + if _, err := NewNetwork(ctx, bad, id, ps, metrics.NewBandwidthCounter()); err == nil { + t.Fatal("should have failed to create swarm") + } + + if _, err := NewNetwork(ctx, goodAndBad, id, ps, metrics.NewBandwidthCounter()); err != nil { + t.Fatal("should have succeeded in creating swarm", err) + } +} + +func subtestAddrsEqual(t *testing.T, a, b []ma.Multiaddr) { + if len(a) != len(b) { + t.Error(t) + } + + in := func(addr ma.Multiaddr, l []ma.Multiaddr) bool { + for _, addr2 := range l { + if addr.Equal(addr2) { + return true + } + } + return false + } + + for _, aa := range a { + if !in(aa, b) { + t.Errorf("%s not in %s", aa, b) + } + } +} + +func TestDialBadAddrs(t *testing.T) { + + m := func(s string) ma.Multiaddr { + maddr, err := ma.NewMultiaddr(s) + if err != nil { + t.Fatal(err) + } + return maddr + } + + ctx := context.Background() + s := makeSwarms(ctx, t, 1)[0] + + test := func(a ma.Multiaddr) { + p := testutil.RandPeerIDFatal(t) + s.peers.AddAddr(p, a, peer.PermanentAddrTTL) + if _, err := s.Dial(ctx, p); err == nil { + t.Error("swarm should not dial: %s", m) + } + } + + test(m("/ip6/fe80::1")) // link local + test(m("/ip6/fe80::100")) // link local + test(m("/ip4/127.0.0.1/udp/1234/utp")) // utp +} diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go new file mode 100644 index 0000000000..126ace627a --- /dev/null +++ b/p2p/net/swarm/swarm_conn.go @@ -0,0 +1,141 @@ +package swarm + +import ( + "fmt" + + ic "github.com/ipfs/go-ipfs/p2p/crypto" + inet "github.com/ipfs/go-ipfs/p2p/net" + conn "github.com/ipfs/go-ipfs/p2p/net/conn" + peer "github.com/ipfs/go-ipfs/p2p/peer" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +// a Conn is a simple wrapper around a ps.Conn that also exposes +// some of the methods from the underlying conn.Conn. +// There's **five** "layers" to each connection: +// * 0. the net.Conn - underlying net.Conn (TCP/UDP/UTP/etc) +// * 1. the manet.Conn - provides multiaddr friendly Conn +// * 2. the conn.Conn - provides Peer friendly Conn (inc Secure channel) +// * 3. the peerstream.Conn - provides peerstream / spdysptream happiness +// * 4. the Conn - abstracts everyting out, exposing only key parts of underlying layers +// (I know, this is kinda crazy. it's more historical than a good design. though the +// layers do build up pieces of functionality. and they're all just io.RW :) ) +type Conn ps.Conn + +// ConnHandler is called when new conns are opened from remote peers. +// See peerstream.ConnHandler +type ConnHandler func(*Conn) + +func (c *Conn) StreamConn() *ps.Conn { + return (*ps.Conn)(c) +} + +func (c *Conn) RawConn() conn.Conn { + // righly panic if these things aren't true. it is an expected + // invariant that these Conns are all of the typewe expect: + // ps.Conn wrapping a conn.Conn + // if we get something else it is programmer error. + return (*ps.Conn)(c).NetConn().(conn.Conn) +} + +func (c *Conn) String() string { + return fmt.Sprintf("", c.RawConn()) +} + +// LocalMultiaddr is the Multiaddr on this side +func (c *Conn) LocalMultiaddr() ma.Multiaddr { + return c.RawConn().LocalMultiaddr() +} + +// LocalPeer is the Peer on our side of the connection +func (c *Conn) LocalPeer() peer.ID { + return c.RawConn().LocalPeer() +} + +// RemoteMultiaddr is the Multiaddr on the remote side +func (c *Conn) RemoteMultiaddr() ma.Multiaddr { + return c.RawConn().RemoteMultiaddr() +} + +// RemotePeer is the Peer on the remote side +func (c *Conn) RemotePeer() peer.ID { + return c.RawConn().RemotePeer() +} + +// LocalPrivateKey is the public key of the peer on this side +func (c *Conn) LocalPrivateKey() ic.PrivKey { + return c.RawConn().LocalPrivateKey() +} + +// RemotePublicKey is the public key of the peer on the remote side +func (c *Conn) RemotePublicKey() ic.PubKey { + return c.RawConn().RemotePublicKey() +} + +// NewSwarmStream returns a new Stream from this connection +func (c *Conn) NewSwarmStream() (*Stream, error) { + s, err := c.StreamConn().NewStream() + return wrapStream(s), err +} + +// NewStream returns a new Stream from this connection +func (c *Conn) NewStream() (inet.Stream, error) { + s, err := c.NewSwarmStream() + return inet.Stream(s), err +} + +func (c *Conn) Close() error { + return c.StreamConn().Close() +} + +func wrapConn(psc *ps.Conn) (*Conn, error) { + // grab the underlying connection. + if _, ok := psc.NetConn().(conn.Conn); !ok { + // this should never happen. if we see it ocurring it means that we added + // a Listener to the ps.Swarm that is NOT one of our net/conn.Listener. + return nil, fmt.Errorf("swarm connHandler: invalid conn (not a conn.Conn): %s", psc) + } + return (*Conn)(psc), nil +} + +// wrapConns returns a *Conn for all these ps.Conns +func wrapConns(conns1 []*ps.Conn) []*Conn { + conns2 := make([]*Conn, len(conns1)) + for i, c1 := range conns1 { + if c2, err := wrapConn(c1); err == nil { + conns2[i] = c2 + } + } + return conns2 +} + +// newConnSetup does the swarm's "setup" for a connection. returns the underlying +// conn.Conn this method is used by both swarm.Dial and ps.Swarm connHandler +func (s *Swarm) newConnSetup(ctx context.Context, psConn *ps.Conn) (*Conn, error) { + + // wrap with a Conn + sc, err := wrapConn(psConn) + if err != nil { + return nil, err + } + + // if we have a public key, make sure we add it to our peerstore! + // This is an important detail. Otherwise we must fetch the public + // key from the DHT or some other system. + if pk := sc.RemotePublicKey(); pk != nil { + s.peers.AddPubKey(sc.RemotePeer(), pk) + } + + // ok great! we can use it. add it to our group. + + // set the RemotePeer as a group on the conn. this lets us group + // connections in the StreamSwarm by peer, and get a streams from + // any available connection in the group (better multiconn): + // swarm.StreamSwarm().NewStreamWithGroup(remotePeer) + psConn.AddGroup(sc.RemotePeer()) + + return sc, nil +} diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go new file mode 100644 index 0000000000..a68d288059 --- /dev/null +++ b/p2p/net/swarm/swarm_dial.go @@ -0,0 +1,493 @@ +package swarm + +import ( + "errors" + "fmt" + "math/rand" + "net" + "sync" + "time" + + mconn "github.com/ipfs/go-ipfs/metrics/conn" + conn "github.com/ipfs/go-ipfs/p2p/net/conn" + addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" + peer "github.com/ipfs/go-ipfs/p2p/peer" + lgbl "github.com/ipfs/go-ipfs/util/eventlog/loggables" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" + process "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" + processctx "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context" + ratelimit "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/ratelimit" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +// Diagram of dial sync: +// +// many callers of Dial() synched w. dials many addrs results to callers +// ----------------------\ dialsync use earliest /-------------- +// -----------------------\ |----------\ /---------------- +// ------------------------>------------<------- >---------<----------------- +// -----------------------| \----x \---------------- +// ----------------------| \-----x \--------------- +// any may fail if no addr at end +// retry dialAttempt x + +var ( + ErrDialBackoff = errors.New("dial backoff") + ErrDialFailed = errors.New("dial attempt failed") + ErrDialToSelf = errors.New("dial to self attempted") +) + +// dialAttempts governs how many times a goroutine will try to dial a given peer. +// Note: this is down to one, as we have _too many dials_ atm. To add back in, +// add loop back in Dial(.) +const dialAttempts = 1 + +// DialTimeout is the amount of time each dial attempt has. We can think about making +// this larger down the road, or putting more granular timeouts (i.e. within each +// subcomponent of Dial) +var DialTimeout time.Duration = time.Second * 10 + +// dialsync is a small object that helps manage ongoing dials. +// this way, if we receive many simultaneous dial requests, one +// can do its thing, while the rest wait. +// +// this interface is so would-be dialers can just: +// +// for { +// c := findConnectionToPeer(peer) +// if c != nil { +// return c +// } +// +// // ok, no connections. should we dial? +// if ok, wait := dialsync.Lock(peer); !ok { +// <-wait // can optionally wait +// continue +// } +// defer dialsync.Unlock(peer) +// +// c := actuallyDial(peer) +// return c +// } +// +type dialsync struct { + // ongoing is a map of tickets for the current peers being dialed. + // this way, we dont kick off N dials simultaneously. + ongoing map[peer.ID]chan struct{} + lock sync.Mutex +} + +// Lock governs the beginning of a dial attempt. +// If there are no ongoing dials, it returns true, and the client is now +// scheduled to dial. Every other goroutine that calls startDial -- with +//the same dst -- will block until client is done. The client MUST call +// ds.Unlock(p) when it is done, to unblock the other callers. +// The client is not reponsible for achieving a successful dial, only for +// reporting the end of the attempt (calling ds.Unlock(p)). +// +// see the example below `dialsync` +func (ds *dialsync) Lock(dst peer.ID) (bool, chan struct{}) { + ds.lock.Lock() + if ds.ongoing == nil { // init if not ready + ds.ongoing = make(map[peer.ID]chan struct{}) + } + wait, found := ds.ongoing[dst] + if !found { + ds.ongoing[dst] = make(chan struct{}) + } + ds.lock.Unlock() + + if found { + return false, wait + } + + // ok! you're signed up to dial! + return true, nil +} + +// Unlock releases waiters to a dial attempt. see Lock. +// if Unlock(p) is called without calling Lock(p) first, Unlock panics. +func (ds *dialsync) Unlock(dst peer.ID) { + ds.lock.Lock() + wait, found := ds.ongoing[dst] + if !found { + panic("called dialDone with no ongoing dials to peer: " + dst.Pretty()) + } + delete(ds.ongoing, dst) // remove ongoing dial + close(wait) // release everyone else + ds.lock.Unlock() +} + +// dialbackoff is a struct used to avoid over-dialing the same, dead peers. +// Whenever we totally time out on a peer (all three attempts), we add them +// to dialbackoff. Then, whenevers goroutines would _wait_ (dialsync), they +// check dialbackoff. If it's there, they don't wait and exit promptly with +// an error. (the single goroutine that is actually dialing continues to +// dial). If a dial is successful, the peer is removed from backoff. +// Example: +// +// for { +// if ok, wait := dialsync.Lock(p); !ok { +// if backoff.Backoff(p) { +// return errDialFailed +// } +// <-wait +// continue +// } +// defer dialsync.Unlock(p) +// c, err := actuallyDial(p) +// if err != nil { +// dialbackoff.AddBackoff(p) +// continue +// } +// dialbackoff.Clear(p) +// } +// +type dialbackoff struct { + entries map[peer.ID]struct{} + lock sync.RWMutex +} + +func (db *dialbackoff) init() { + if db.entries == nil { + db.entries = make(map[peer.ID]struct{}) + } +} + +// Backoff returns whether the client should backoff from dialing +// peeer p +func (db *dialbackoff) Backoff(p peer.ID) bool { + db.lock.Lock() + db.init() + _, found := db.entries[p] + db.lock.Unlock() + return found +} + +// AddBackoff lets other nodes know that we've entered backoff with +// peer p, so dialers should not wait unnecessarily. We still will +// attempt to dial with one goroutine, in case we get through. +func (db *dialbackoff) AddBackoff(p peer.ID) { + db.lock.Lock() + db.init() + db.entries[p] = struct{}{} + db.lock.Unlock() +} + +// Clear removes a backoff record. Clients should call this after a +// successful Dial. +func (db *dialbackoff) Clear(p peer.ID) { + db.lock.Lock() + db.init() + delete(db.entries, p) + db.lock.Unlock() +} + +// Dial connects to a peer. +// +// The idea is that the client of Swarm does not need to know what network +// the connection will happen over. Swarm can use whichever it choses. +// This allows us to use various transport protocols, do NAT traversal/relay, +// etc. to achive connection. +func (s *Swarm) Dial(ctx context.Context, p peer.ID) (*Conn, error) { + var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) + if p == s.local { + log.Event(ctx, "swarmDialSelf", logdial) + return nil, ErrDialToSelf + } + + return s.gatedDialAttempt(ctx, p) +} + +func (s *Swarm) bestConnectionToPeer(p peer.ID) *Conn { + cs := s.ConnectionsToPeer(p) + for _, conn := range cs { + if conn != nil { // dump out the first one we find. (TODO pick better) + return conn + } + } + return nil +} + +// gatedDialAttempt is an attempt to dial a node. It is gated by the swarm's +// dial synchronization systems: dialsync and dialbackoff. +func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) { + var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) + defer log.EventBegin(ctx, "swarmDialAttemptSync", logdial).Done() + + // check if we already have an open connection first + conn := s.bestConnectionToPeer(p) + if conn != nil { + return conn, nil + } + + // check if there's an ongoing dial to this peer + if ok, wait := s.dsync.Lock(p); ok { + // ok, we have been charged to dial! let's do it. + // if it succeeds, dial will add the conn to the swarm itself. + + defer log.EventBegin(ctx, "swarmDialAttemptStart", logdial).Done() + ctxT, cancel := context.WithTimeout(ctx, s.dialT) + conn, err := s.dial(ctxT, p) + cancel() + s.dsync.Unlock(p) + log.Debugf("dial end %s", conn) + if err != nil { + log.Event(ctx, "swarmDialBackoffAdd", logdial) + s.backf.AddBackoff(p) // let others know to backoff + + // ok, we failed. try again. (if loop is done, our error is output) + return nil, fmt.Errorf("dial attempt failed: %s", err) + } + log.Event(ctx, "swarmDialBackoffClear", logdial) + s.backf.Clear(p) // okay, no longer need to backoff + return conn, nil + + } else { + // we did not dial. we must wait for someone else to dial. + + // check whether we should backoff first... + if s.backf.Backoff(p) { + log.Event(ctx, "swarmDialBackoff", logdial) + return nil, ErrDialBackoff + } + + defer log.EventBegin(ctx, "swarmDialWait", logdial).Done() + select { + case <-wait: // wait for that other dial to finish. + + // see if it worked, OR we got an incoming dial in the meantime... + conn := s.bestConnectionToPeer(p) + if conn != nil { + return conn, nil + } + return nil, ErrDialFailed + case <-ctx.Done(): // or we may have to bail... + return nil, ctx.Err() + } + } +} + +// dial is the actual swarm's dial logic, gated by Dial. +func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { + var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) + if p == s.local { + log.Event(ctx, "swarmDialDoDialSelf", logdial) + return nil, ErrDialToSelf + } + defer log.EventBegin(ctx, "swarmDialDo", logdial).Done() + logdial["dial"] = "failure" // start off with failure. set to "success" at the end. + + sk := s.peers.PrivKey(s.local) + logdial["encrypted"] = (sk != nil) // log wether this will be an encrypted dial or not. + if sk == nil { + // fine for sk to be nil, just log. + log.Debug("Dial not given PrivateKey, so WILL NOT SECURE conn.") + } + + // get our own addrs. try dialing out from our listener addresses (reusing ports) + // Note that using our peerstore's addresses here is incorrect, as that would + // include observed addresses. TODO: make peerstore's address book smarter. + localAddrs := s.ListenAddresses() + if len(localAddrs) == 0 { + log.Debug("Dialing out with no local addresses.") + } + + // get remote peer addrs + remoteAddrs := s.peers.Addrs(p) + // make sure we can use the addresses. + remoteAddrs = addrutil.FilterUsableAddrs(remoteAddrs) + // drop out any addrs that would just dial ourselves. use ListenAddresses + // as that is a more authoritative view than localAddrs. + ila, _ := s.InterfaceListenAddresses() + remoteAddrs = addrutil.Subtract(remoteAddrs, ila) + remoteAddrs = addrutil.Subtract(remoteAddrs, s.peers.Addrs(s.local)) + + log.Debugf("%s swarm dialing %s -- local:%s remote:%s", s.local, p, s.ListenAddresses(), remoteAddrs) + if len(remoteAddrs) == 0 { + err := errors.New("peer has no addresses") + logdial["error"] = err + return nil, err + } + + remoteAddrs = s.filterAddrs(remoteAddrs) + if len(remoteAddrs) == 0 { + err := errors.New("all adresses for peer have been filtered out") + logdial["error"] = err + return nil, err + } + + // open connection to peer + d := &conn.Dialer{ + Dialer: manet.Dialer{ + Dialer: net.Dialer{ + Timeout: s.dialT, + }, + }, + LocalPeer: s.local, + LocalAddrs: localAddrs, + PrivateKey: sk, + Wrapper: func(c manet.Conn) manet.Conn { + return mconn.WrapConn(s.bwc, c) + }, + } + + // try to get a connection to any addr + connC, err := s.dialAddrs(ctx, d, p, remoteAddrs) + if err != nil { + logdial["error"] = err + return nil, err + } + logdial["netconn"] = lgbl.NetConn(connC) + + // ok try to setup the new connection. + defer log.EventBegin(ctx, "swarmDialDoSetup", logdial, lgbl.NetConn(connC)).Done() + swarmC, err := dialConnSetup(ctx, s, connC) + if err != nil { + logdial["error"] = err + connC.Close() // close the connection. didn't work out :( + return nil, err + } + + logdial["dial"] = "success" + return swarmC, nil +} + +func (s *Swarm) dialAddrs(ctx context.Context, d *conn.Dialer, p peer.ID, remoteAddrs []ma.Multiaddr) (conn.Conn, error) { + + // try to connect to one of the peer's known addresses. + // we dial concurrently to each of the addresses, which: + // * makes the process faster overall + // * attempts to get the fastest connection available. + // * mitigates the waste of trying bad addresses + log.Debugf("%s swarm dialing %s %s", s.local, p, remoteAddrs) + + ctx, cancel := context.WithCancel(ctx) + defer cancel() // cancel work when we exit func + + foundConn := make(chan struct{}) + conns := make(chan conn.Conn, len(remoteAddrs)) + errs := make(chan error, len(remoteAddrs)) + + // dialSingleAddr is used in the rate-limited async thing below. + dialSingleAddr := func(addr ma.Multiaddr) { + connC, err := s.dialAddr(ctx, d, p, addr) + + // check parent still wants our results + select { + case <-foundConn: + if connC != nil { + connC.Close() + } + return + default: + } + + if err != nil { + errs <- err + } else if connC == nil { + errs <- fmt.Errorf("failed to dial %s %s", p, addr) + } else { + conns <- connC + } + } + + // this whole thing is in a goroutine so we can use foundConn + // to end early. + go func() { + // rate limiting just in case. at most 10 addrs at once. + limiter := ratelimit.NewRateLimiter(process.Background(), 10) + limiter.Go(func(worker process.Process) { + // permute addrs so we try different sets first each time. + for _, i := range rand.Perm(len(remoteAddrs)) { + select { + case <-foundConn: // if one of them succeeded already + break + case <-worker.Closing(): // our context was cancelled + break + default: + } + + workerAddr := remoteAddrs[i] // shadow variable to avoid race + limiter.LimitedGo(func(worker process.Process) { + dialSingleAddr(workerAddr) + }) + } + }) + + processctx.CloseAfterContext(limiter, ctx) + }() + + // wair fot the results. + exitErr := fmt.Errorf("failed to dial %s", p) + for i := 0; i < len(remoteAddrs); i++ { + select { + case exitErr = <-errs: // + log.Debug("dial error: ", exitErr) + case connC := <-conns: + // take the first + return asap + close(foundConn) + return connC, nil + } + } + return nil, exitErr +} + +func (s *Swarm) dialAddr(ctx context.Context, d *conn.Dialer, p peer.ID, addr ma.Multiaddr) (conn.Conn, error) { + log.Debugf("%s swarm dialing %s %s", s.local, p, addr) + + connC, err := d.Dial(ctx, addr, p) + if err != nil { + return nil, fmt.Errorf("%s --> %s dial attempt failed: %s", s.local, p, err) + } + + // if the connection is not to whom we thought it would be... + remotep := connC.RemotePeer() + if remotep != p { + connC.Close() + return nil, fmt.Errorf("misdial to %s through %s (got %s)", p, addr, remotep) + } + + // if the connection is to ourselves... + // this can happen TONS when Loopback addrs are advertized. + // (this should be caught by two checks above, but let's just make sure.) + if remotep == s.local { + connC.Close() + return nil, fmt.Errorf("misdial to %s through %s (got self)", p, addr) + } + + // success! we got one! + return connC, nil +} + +func (s *Swarm) filterAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { + var out []ma.Multiaddr + for _, a := range addrs { + if !s.Filters.AddrBlocked(a) { + out = append(out, a) + } + } + return out +} + +// dialConnSetup is the setup logic for a connection from the dial side. it +// needs to add the Conn to the StreamSwarm, then run newConnSetup +func dialConnSetup(ctx context.Context, s *Swarm, connC conn.Conn) (*Conn, error) { + + psC, err := s.swarm.AddConn(connC) + if err != nil { + // connC is closed by caller if we fail. + return nil, fmt.Errorf("failed to add conn to ps.Swarm: %s", err) + } + + // ok try to setup the new connection. (newConnSetup will add to group) + swarmC, err := s.newConnSetup(ctx, psC) + if err != nil { + psC.Close() // we need to make sure psC is Closed. + return nil, err + } + + return swarmC, err +} diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go new file mode 100644 index 0000000000..d1bcb07521 --- /dev/null +++ b/p2p/net/swarm/swarm_listen.go @@ -0,0 +1,142 @@ +package swarm + +import ( + "fmt" + + mconn "github.com/ipfs/go-ipfs/metrics/conn" + inet "github.com/ipfs/go-ipfs/p2p/net" + conn "github.com/ipfs/go-ipfs/p2p/net/conn" + addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" + lgbl "github.com/ipfs/go-ipfs/util/eventlog/loggables" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" + ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + multierr "github.com/ipfs/go-ipfs/thirdparty/multierr" +) + +// Open listeners for each network the swarm should listen on +func (s *Swarm) listen(addrs []ma.Multiaddr) error { + + for _, addr := range addrs { + if !addrutil.AddrUsable(addr, true) { + return fmt.Errorf("cannot use addr: %s", addr) + } + } + + retErr := multierr.New() + + // listen on every address + for i, addr := range addrs { + err := s.setupListener(addr) + if err != nil { + if retErr.Errors == nil { + retErr.Errors = make([]error, len(addrs)) + } + retErr.Errors[i] = err + log.Debugf("Failed to listen on: %s - %s", addr, err) + } + } + + if retErr.Errors != nil { + return retErr + } + return nil +} + +// Listen for new connections on the given multiaddr +func (s *Swarm) setupListener(maddr ma.Multiaddr) error { + + // TODO rethink how this has to work. (jbenet) + // + // resolved, err := resolveUnspecifiedAddresses([]ma.Multiaddr{maddr}) + // if err != nil { + // return err + // } + // for _, a := range resolved { + // s.peers.AddAddr(s.local, a) + // } + + sk := s.peers.PrivKey(s.local) + if sk == nil { + // may be fine for sk to be nil, just log a warning. + log.Warning("Listener not given PrivateKey, so WILL NOT SECURE conns.") + } + log.Debugf("Swarm Listening at %s", maddr) + list, err := conn.Listen(s.Context(), maddr, s.local, sk) + if err != nil { + return err + } + + list.SetAddrFilters(s.Filters) + + if cw, ok := list.(conn.ListenerConnWrapper); ok { + cw.SetConnWrapper(func(c manet.Conn) manet.Conn { + return mconn.WrapConn(s.bwc, c) + }) + } + + // AddListener to the peerstream Listener. this will begin accepting connections + // and streams! + sl, err := s.swarm.AddListener(list) + if err != nil { + return err + } + log.Debugf("Swarm Listeners at %s", s.ListenAddresses()) + + // signal to our notifiees on successful conn. + s.notifyAll(func(n inet.Notifiee) { + n.Listen((*Network)(s), maddr) + }) + + // go consume peerstream's listen accept errors. note, these ARE errors. + // they may be killing the listener, and if we get _any_ we should be + // fixing this in our conn.Listener (to ignore them or handle them + // differently.) + go func(ctx context.Context, sl *ps.Listener) { + + // signal to our notifiees closing + defer s.notifyAll(func(n inet.Notifiee) { + n.ListenClose((*Network)(s), maddr) + }) + + for { + select { + case err, more := <-sl.AcceptErrors(): + if !more { + return + } + log.Warningf("swarm listener accept error: %s", err) + case <-ctx.Done(): + return + } + } + }(s.Context(), sl) + + return nil +} + +// connHandler is called by the StreamSwarm whenever a new connection is added +// here we configure it slightly. Note that this is sequential, so if anything +// will take a while do it in a goroutine. +// See https://godoc.org/github.com/jbenet/go-peerstream for more information +func (s *Swarm) connHandler(c *ps.Conn) *Conn { + ctx := context.Background() + // this context is for running the handshake, which -- when receiveing connections + // -- we have no bound on beyond what the transport protocol bounds it at. + // note that setup + the handshake are bounded by underlying io. + // (i.e. if TCP or UDP disconnects (or the swarm closes), we're done. + // Q: why not have a shorter handshake? think about an HTTP server on really slow conns. + // as long as the conn is live (TCP says its online), it tries its best. we follow suit.) + + sc, err := s.newConnSetup(ctx, c) + if err != nil { + log.Debug(err) + log.Event(ctx, "newConnHandlerDisconnect", lgbl.NetConn(c.NetConn()), lgbl.Error(err)) + c.Close() // boom. close it. + return nil + } + + return sc +} diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go new file mode 100644 index 0000000000..da9b52251e --- /dev/null +++ b/p2p/net/swarm/swarm_net.go @@ -0,0 +1,172 @@ +package swarm + +import ( + "fmt" + + peer "github.com/ipfs/go-ipfs/p2p/peer" + + metrics "github.com/ipfs/go-ipfs/metrics" + inet "github.com/ipfs/go-ipfs/p2p/net" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +// Network implements the inet.Network interface. +// It is simply a swarm, with a few different functions +// to implement inet.Network. +type Network Swarm + +// NewNetwork constructs a new network and starts listening on given addresses. +func NewNetwork(ctx context.Context, listen []ma.Multiaddr, local peer.ID, + peers peer.Peerstore, bwc metrics.Reporter) (*Network, error) { + + s, err := NewSwarm(ctx, listen, local, peers, bwc) + if err != nil { + return nil, err + } + + return (*Network)(s), nil +} + +// DialPeer attempts to establish a connection to a given peer. +// Respects the context. +func (n *Network) DialPeer(ctx context.Context, p peer.ID) (inet.Conn, error) { + log.Debugf("[%s] network dialing peer [%s]", n.local, p) + sc, err := n.Swarm().Dial(ctx, p) + if err != nil { + return nil, err + } + + log.Debugf("network for %s finished dialing %s", n.local, p) + return inet.Conn(sc), nil +} + +// Process returns the network's Process +func (n *Network) Process() goprocess.Process { + return n.proc +} + +// Swarm returns the network's peerstream.Swarm +func (n *Network) Swarm() *Swarm { + return (*Swarm)(n) +} + +// LocalPeer the network's LocalPeer +func (n *Network) LocalPeer() peer.ID { + return n.Swarm().LocalPeer() +} + +// Peers returns the known peer IDs from the Peerstore +func (n *Network) Peers() []peer.ID { + return n.Swarm().Peers() +} + +// Peers returns the Peerstore, which tracks known peers +func (n *Network) Peerstore() peer.Peerstore { + return n.Swarm().peers +} + +// Conns returns the connected peers +func (n *Network) Conns() []inet.Conn { + conns1 := n.Swarm().Connections() + out := make([]inet.Conn, len(conns1)) + for i, c := range conns1 { + out[i] = inet.Conn(c) + } + return out +} + +// ConnsToPeer returns the connections in this Netowrk for given peer. +func (n *Network) ConnsToPeer(p peer.ID) []inet.Conn { + conns1 := n.Swarm().ConnectionsToPeer(p) + out := make([]inet.Conn, len(conns1)) + for i, c := range conns1 { + out[i] = inet.Conn(c) + } + return out +} + +// ClosePeer connection to peer +func (n *Network) ClosePeer(p peer.ID) error { + return n.Swarm().CloseConnection(p) +} + +// close is the real teardown function +func (n *Network) close() error { + return n.Swarm().Close() +} + +// Close calls the ContextCloser func +func (n *Network) Close() error { + return n.Swarm().proc.Close() +} + +// Listen tells the network to start listening on given multiaddrs. +func (n *Network) Listen(addrs ...ma.Multiaddr) error { + return n.Swarm().Listen(addrs...) +} + +// ListenAddresses returns a list of addresses at which this network listens. +func (n *Network) ListenAddresses() []ma.Multiaddr { + return n.Swarm().ListenAddresses() +} + +// InterfaceListenAddresses returns a list of addresses at which this network +// listens. It expands "any interface" addresses (/ip4/0.0.0.0, /ip6/::) to +// use the known local interfaces. +func (n *Network) InterfaceListenAddresses() ([]ma.Multiaddr, error) { + return n.Swarm().InterfaceListenAddresses() +} + +// Connectedness returns a state signaling connection capabilities +// For now only returns Connected || NotConnected. Expand into more later. +func (n *Network) Connectedness(p peer.ID) inet.Connectedness { + c := n.Swarm().ConnectionsToPeer(p) + if c != nil && len(c) > 0 { + return inet.Connected + } + return inet.NotConnected +} + +// NewStream returns a new stream to given peer p. +// If there is no connection to p, attempts to create one. +func (n *Network) NewStream(p peer.ID) (inet.Stream, error) { + log.Debugf("[%s] network opening stream to peer [%s]", n.local, p) + s, err := n.Swarm().NewStreamWithPeer(p) + if err != nil { + return nil, err + } + + return inet.Stream(s), nil +} + +// SetHandler sets the protocol handler on the Network's Muxer. +// This operation is threadsafe. +func (n *Network) SetStreamHandler(h inet.StreamHandler) { + n.Swarm().SetStreamHandler(h) +} + +// SetConnHandler sets the conn handler on the Network. +// This operation is threadsafe. +func (n *Network) SetConnHandler(h inet.ConnHandler) { + n.Swarm().SetConnHandler(func(c *Conn) { + h(inet.Conn(c)) + }) +} + +// String returns a string representation of Network. +func (n *Network) String() string { + return fmt.Sprintf("", n.LocalPeer()) +} + +// Notify signs up Notifiee to receive signals when events happen +func (n *Network) Notify(f inet.Notifiee) { + n.Swarm().Notify(f) +} + +// StopNotify unregisters Notifiee fromr receiving signals +func (n *Network) StopNotify(f inet.Notifiee) { + n.Swarm().StopNotify(f) +} diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go new file mode 100644 index 0000000000..b5f5c56996 --- /dev/null +++ b/p2p/net/swarm/swarm_net_test.go @@ -0,0 +1,77 @@ +package swarm_test + +import ( + "fmt" + "testing" + "time" + + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + inet "github.com/ipfs/go-ipfs/p2p/net" + testutil "github.com/ipfs/go-ipfs/p2p/test/util" +) + +// TestConnectednessCorrect starts a few networks, connects a few +// and tests Connectedness value is correct. +func TestConnectednessCorrect(t *testing.T) { + + ctx := context.Background() + + nets := make([]inet.Network, 4) + for i := 0; i < 4; i++ { + nets[i] = testutil.GenSwarmNetwork(t, ctx) + } + + // connect 0-1, 0-2, 0-3, 1-2, 2-3 + + dial := func(a, b inet.Network) { + testutil.DivulgeAddresses(b, a) + if _, err := a.DialPeer(ctx, b.LocalPeer()); err != nil { + t.Fatalf("Failed to dial: %s", err) + } + } + + dial(nets[0], nets[1]) + dial(nets[0], nets[3]) + dial(nets[1], nets[2]) + dial(nets[3], nets[2]) + + // there's something wrong with dial, i think. it's not finishing + // completely. there must be some async stuff. + <-time.After(100 * time.Millisecond) + + // test those connected show up correctly + + // test connected + expectConnectedness(t, nets[0], nets[1], inet.Connected) + expectConnectedness(t, nets[0], nets[3], inet.Connected) + expectConnectedness(t, nets[1], nets[2], inet.Connected) + expectConnectedness(t, nets[3], nets[2], inet.Connected) + + // test not connected + expectConnectedness(t, nets[0], nets[2], inet.NotConnected) + expectConnectedness(t, nets[1], nets[3], inet.NotConnected) + + for _, n := range nets { + n.Close() + } +} + +func expectConnectedness(t *testing.T, a, b inet.Network, expected inet.Connectedness) { + es := "%s is connected to %s, but Connectedness incorrect. %s %s" + if a.Connectedness(b.LocalPeer()) != expected { + t.Errorf(es, a, b, printConns(a), printConns(b)) + } + + // test symmetric case + if b.Connectedness(a.LocalPeer()) != expected { + t.Errorf(es, b, a, printConns(b), printConns(a)) + } +} + +func printConns(n inet.Network) string { + s := fmt.Sprintf("Connections in %s:\n", n) + for _, c := range n.Conns() { + s = s + fmt.Sprintf("- %s\n", c) + } + return s +} diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go new file mode 100644 index 0000000000..9b0e3a9bab --- /dev/null +++ b/p2p/net/swarm/swarm_notif_test.go @@ -0,0 +1,210 @@ +package swarm + +import ( + "testing" + "time" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + + inet "github.com/ipfs/go-ipfs/p2p/net" + peer "github.com/ipfs/go-ipfs/p2p/peer" +) + +func TestNotifications(t *testing.T) { + ctx := context.Background() + swarms := makeSwarms(ctx, t, 5) + defer func() { + for _, s := range swarms { + s.Close() + } + }() + + timeout := 5 * time.Second + + // signup notifs + notifiees := make([]*netNotifiee, len(swarms)) + for i, swarm := range swarms { + n := newNetNotifiee() + swarm.Notify(n) + notifiees[i] = n + } + + connectSwarms(t, ctx, swarms) + + <-time.After(time.Millisecond) + // should've gotten 5 by now. + + // test everyone got the correct connection opened calls + for i, s := range swarms { + n := notifiees[i] + notifs := make(map[peer.ID][]inet.Conn) + for j, s2 := range swarms { + if i == j { + continue + } + + // this feels a little sketchy, but its probably okay + for len(s.ConnectionsToPeer(s2.LocalPeer())) != len(notifs[s2.LocalPeer()]) { + select { + case c := <-n.connected: + nfp := notifs[c.RemotePeer()] + notifs[c.RemotePeer()] = append(nfp, c) + case <-time.After(timeout): + t.Fatal("timeout") + } + } + } + + for p, cons := range notifs { + expect := s.ConnectionsToPeer(p) + if len(expect) != len(cons) { + t.Fatal("got different number of connections") + } + + for _, c := range cons { + var found bool + for _, c2 := range expect { + if c == c2 { + found = true + break + } + } + + if !found { + t.Fatal("connection not found!") + } + } + } + } + + complement := func(c inet.Conn) (*Swarm, *netNotifiee, *Conn) { + for i, s := range swarms { + for _, c2 := range s.Connections() { + if c.LocalMultiaddr().Equal(c2.RemoteMultiaddr()) && + c2.LocalMultiaddr().Equal(c.RemoteMultiaddr()) { + return s, notifiees[i], c2 + } + } + } + t.Fatal("complementary conn not found", c) + return nil, nil, nil + } + + testOCStream := func(n *netNotifiee, s inet.Stream) { + var s2 inet.Stream + select { + case s2 = <-n.openedStream: + t.Log("got notif for opened stream") + case <-time.After(timeout): + t.Fatal("timeout") + } + if s != s2 { + t.Fatal("got incorrect stream", s.Conn(), s2.Conn()) + } + + select { + case s2 = <-n.closedStream: + t.Log("got notif for closed stream") + case <-time.After(timeout): + t.Fatal("timeout") + } + if s != s2 { + t.Fatal("got incorrect stream", s.Conn(), s2.Conn()) + } + } + + streams := make(chan inet.Stream) + for _, s := range swarms { + s.SetStreamHandler(func(s inet.Stream) { + streams <- s + s.Close() + }) + } + + // open a streams in each conn + for i, s := range swarms { + for _, c := range s.Connections() { + _, n2, _ := complement(c) + + st1, err := c.NewStream() + if err != nil { + t.Error(err) + } else { + st1.Write([]byte("hello")) + st1.Close() + testOCStream(notifiees[i], st1) + st2 := <-streams + testOCStream(n2, st2) + } + } + } + + // close conns + for i, s := range swarms { + n := notifiees[i] + for _, c := range s.Connections() { + _, n2, c2 := complement(c) + c.Close() + c2.Close() + + var c3, c4 inet.Conn + select { + case c3 = <-n.disconnected: + case <-time.After(timeout): + t.Fatal("timeout") + } + if c != c3 { + t.Fatal("got incorrect conn", c, c3) + } + + select { + case c4 = <-n2.disconnected: + case <-time.After(timeout): + t.Fatal("timeout") + } + if c2 != c4 { + t.Fatal("got incorrect conn", c, c2) + } + } + } +} + +type netNotifiee struct { + listen chan ma.Multiaddr + listenClose chan ma.Multiaddr + connected chan inet.Conn + disconnected chan inet.Conn + openedStream chan inet.Stream + closedStream chan inet.Stream +} + +func newNetNotifiee() *netNotifiee { + return &netNotifiee{ + listen: make(chan ma.Multiaddr), + listenClose: make(chan ma.Multiaddr), + connected: make(chan inet.Conn), + disconnected: make(chan inet.Conn), + openedStream: make(chan inet.Stream), + closedStream: make(chan inet.Stream), + } +} + +func (nn *netNotifiee) Listen(n inet.Network, a ma.Multiaddr) { + nn.listen <- a +} +func (nn *netNotifiee) ListenClose(n inet.Network, a ma.Multiaddr) { + nn.listenClose <- a +} +func (nn *netNotifiee) Connected(n inet.Network, v inet.Conn) { + nn.connected <- v +} +func (nn *netNotifiee) Disconnected(n inet.Network, v inet.Conn) { + nn.disconnected <- v +} +func (nn *netNotifiee) OpenedStream(n inet.Network, v inet.Stream) { + nn.openedStream <- v +} +func (nn *netNotifiee) ClosedStream(n inet.Network, v inet.Stream) { + nn.closedStream <- v +} diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go new file mode 100644 index 0000000000..66dd6353b7 --- /dev/null +++ b/p2p/net/swarm/swarm_stream.go @@ -0,0 +1,54 @@ +package swarm + +import ( + inet "github.com/ipfs/go-ipfs/p2p/net" + + ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" +) + +// a Stream is a wrapper around a ps.Stream that exposes a way to get +// our Conn and Swarm (instead of just the ps.Conn and ps.Swarm) +type Stream ps.Stream + +// Stream returns the underlying peerstream.Stream +func (s *Stream) Stream() *ps.Stream { + return (*ps.Stream)(s) +} + +// Conn returns the Conn associated with this Stream, as an inet.Conn +func (s *Stream) Conn() inet.Conn { + return s.SwarmConn() +} + +// SwarmConn returns the Conn associated with this Stream, as a *Conn +func (s *Stream) SwarmConn() *Conn { + return (*Conn)(s.Stream().Conn()) +} + +// Read reads bytes from a stream. +func (s *Stream) Read(p []byte) (n int, err error) { + return s.Stream().Read(p) +} + +// Write writes bytes to a stream, flushing for each call. +func (s *Stream) Write(p []byte) (n int, err error) { + return s.Stream().Write(p) +} + +// Close closes the stream, indicating this side is finished +// with the stream. +func (s *Stream) Close() error { + return s.Stream().Close() +} + +func wrapStream(pss *ps.Stream) *Stream { + return (*Stream)(pss) +} + +func wrapStreams(st []*ps.Stream) []*Stream { + out := make([]*Stream, len(st)) + for i, s := range st { + out[i] = wrapStream(s) + } + return out +} diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go new file mode 100644 index 0000000000..9193db0109 --- /dev/null +++ b/p2p/net/swarm/swarm_test.go @@ -0,0 +1,330 @@ +package swarm + +import ( + "bytes" + "fmt" + "io" + "net" + "sync" + "testing" + "time" + + metrics "github.com/ipfs/go-ipfs/metrics" + inet "github.com/ipfs/go-ipfs/p2p/net" + peer "github.com/ipfs/go-ipfs/p2p/peer" + testutil "github.com/ipfs/go-ipfs/util/testutil" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" +) + +func EchoStreamHandler(stream inet.Stream) { + go func() { + defer stream.Close() + + // pull out the ipfs conn + c := stream.Conn() + log.Infof("%s ponging to %s", c.LocalPeer(), c.RemotePeer()) + + buf := make([]byte, 4) + + for { + if _, err := stream.Read(buf); err != nil { + if err != io.EOF { + log.Info("ping receive error:", err) + } + return + } + + if !bytes.Equal(buf, []byte("ping")) { + log.Infof("ping receive error: ping != %s %v", buf, buf) + return + } + + if _, err := stream.Write([]byte("pong")); err != nil { + log.Info("pond send error:", err) + return + } + } + }() +} + +func makeSwarms(ctx context.Context, t *testing.T, num int) []*Swarm { + swarms := make([]*Swarm, 0, num) + + for i := 0; i < num; i++ { + localnp := testutil.RandPeerNetParamsOrFatal(t) + + peerstore := peer.NewPeerstore() + peerstore.AddPubKey(localnp.ID, localnp.PubKey) + peerstore.AddPrivKey(localnp.ID, localnp.PrivKey) + + addrs := []ma.Multiaddr{localnp.Addr} + swarm, err := NewSwarm(ctx, addrs, localnp.ID, peerstore, metrics.NewBandwidthCounter()) + if err != nil { + t.Fatal(err) + } + + swarm.SetStreamHandler(EchoStreamHandler) + swarms = append(swarms, swarm) + } + + return swarms +} + +func connectSwarms(t *testing.T, ctx context.Context, swarms []*Swarm) { + + var wg sync.WaitGroup + connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { + // TODO: make a DialAddr func. + s.peers.AddAddr(dst, addr, peer.PermanentAddrTTL) + if _, err := s.Dial(ctx, dst); err != nil { + t.Fatal("error swarm dialing to peer", err) + } + wg.Done() + } + + log.Info("Connecting swarms simultaneously.") + for _, s1 := range swarms { + for _, s2 := range swarms { + if s2.local != s1.local { // don't connect to self. + wg.Add(1) + connect(s1, s2.LocalPeer(), s2.ListenAddresses()[0]) // try the first. + } + } + } + wg.Wait() + + for _, s := range swarms { + log.Infof("%s swarm routing table: %s", s.local, s.Peers()) + } +} + +func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { + // t.Skip("skipping for another test") + + ctx := context.Background() + swarms := makeSwarms(ctx, t, SwarmNum) + + // connect everyone + connectSwarms(t, ctx, swarms) + + // ping/pong + for _, s1 := range swarms { + log.Debugf("-------------------------------------------------------") + log.Debugf("%s ping pong round", s1.local) + log.Debugf("-------------------------------------------------------") + + _, cancel := context.WithCancel(ctx) + got := map[peer.ID]int{} + errChan := make(chan error, MsgNum*len(swarms)) + streamChan := make(chan *Stream, MsgNum) + + // send out "ping" x MsgNum to every peer + go func() { + defer close(streamChan) + + var wg sync.WaitGroup + send := func(p peer.ID) { + defer wg.Done() + + // first, one stream per peer (nice) + stream, err := s1.NewStreamWithPeer(p) + if err != nil { + errChan <- err + return + } + + // send out ping! + for k := 0; k < MsgNum; k++ { // with k messages + msg := "ping" + log.Debugf("%s %s %s (%d)", s1.local, msg, p, k) + if _, err := stream.Write([]byte(msg)); err != nil { + errChan <- err + continue + } + } + + // read it later + streamChan <- stream + } + + for _, s2 := range swarms { + if s2.local == s1.local { + continue // dont send to self... + } + + wg.Add(1) + go send(s2.local) + } + wg.Wait() + }() + + // receive "pong" x MsgNum from every peer + go func() { + defer close(errChan) + count := 0 + countShouldBe := MsgNum * (len(swarms) - 1) + for stream := range streamChan { // one per peer + defer stream.Close() + + // get peer on the other side + p := stream.Conn().RemotePeer() + + // receive pings + msgCount := 0 + msg := make([]byte, 4) + for k := 0; k < MsgNum; k++ { // with k messages + + // read from the stream + if _, err := stream.Read(msg); err != nil { + errChan <- err + continue + } + + if string(msg) != "pong" { + errChan <- fmt.Errorf("unexpected message: %s", msg) + continue + } + + log.Debugf("%s %s %s (%d)", s1.local, msg, p, k) + msgCount++ + } + + got[p] = msgCount + count += msgCount + } + + if count != countShouldBe { + errChan <- fmt.Errorf("count mismatch: %d != %d", count, countShouldBe) + } + }() + + // check any errors (blocks till consumer is done) + for err := range errChan { + if err != nil { + t.Error(err.Error()) + } + } + + log.Debugf("%s got pongs", s1.local) + if (len(swarms) - 1) != len(got) { + t.Errorf("got (%d) less messages than sent (%d).", len(got), len(swarms)) + } + + for p, n := range got { + if n != MsgNum { + t.Error("peer did not get all msgs", p, n, "/", MsgNum) + } + } + + cancel() + <-time.After(10 * time.Millisecond) + } + + for _, s := range swarms { + s.Close() + } +} + +func TestSwarm(t *testing.T) { + // t.Skip("skipping for another test") + t.Parallel() + + // msgs := 1000 + msgs := 100 + swarms := 5 + SubtestSwarm(t, swarms, msgs) +} + +func TestConnHandler(t *testing.T) { + // t.Skip("skipping for another test") + t.Parallel() + + ctx := context.Background() + swarms := makeSwarms(ctx, t, 5) + + gotconn := make(chan struct{}, 10) + swarms[0].SetConnHandler(func(conn *Conn) { + gotconn <- struct{}{} + }) + + connectSwarms(t, ctx, swarms) + + <-time.After(time.Millisecond) + // should've gotten 5 by now. + + swarms[0].SetConnHandler(nil) + + expect := 4 + for i := 0; i < expect; i++ { + select { + case <-time.After(time.Second): + t.Fatal("failed to get connections") + case <-gotconn: + } + } + + select { + case <-gotconn: + t.Fatalf("should have connected to %d swarms", expect) + default: + } +} + +func TestAddrBlocking(t *testing.T) { + ctx := context.Background() + swarms := makeSwarms(ctx, t, 2) + + swarms[0].SetConnHandler(func(conn *Conn) { + t.Fatal("no connections should happen!") + }) + + _, block, err := net.ParseCIDR("127.0.0.1/8") + if err != nil { + t.Fatal(err) + } + + swarms[1].Filters.AddDialFilter(block) + + swarms[1].peers.AddAddr(swarms[0].LocalPeer(), swarms[0].ListenAddresses()[0], peer.PermanentAddrTTL) + _, err = swarms[1].Dial(ctx, swarms[0].LocalPeer()) + if err == nil { + t.Fatal("dial should have failed") + } + + swarms[0].peers.AddAddr(swarms[1].LocalPeer(), swarms[1].ListenAddresses()[0], peer.PermanentAddrTTL) + _, err = swarms[0].Dial(ctx, swarms[1].LocalPeer()) + if err == nil { + t.Fatal("dial should have failed") + } +} + +func TestFilterBounds(t *testing.T) { + ctx := context.Background() + swarms := makeSwarms(ctx, t, 2) + + conns := make(chan struct{}, 8) + swarms[0].SetConnHandler(func(conn *Conn) { + conns <- struct{}{} + }) + + // Address that we wont be dialing from + _, block, err := net.ParseCIDR("192.0.0.1/8") + if err != nil { + t.Fatal(err) + } + + // set filter on both sides, shouldnt matter + swarms[1].Filters.AddDialFilter(block) + swarms[0].Filters.AddDialFilter(block) + + connectSwarms(t, ctx, swarms) + + select { + case <-time.After(time.Second): + t.Fatal("should have gotten connection") + case <-conns: + t.Log("got connect") + } +} From 843a30cb9afc681538be65ca2fb8cb276ad247cf Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Wed, 30 Sep 2015 18:42:55 -0400 Subject: [PATCH 0022/3965] move to p2p dir --- p2p/net/nat/nat.go | 439 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 439 insertions(+) create mode 100644 p2p/net/nat/nat.go diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go new file mode 100644 index 0000000000..a8bdb81be6 --- /dev/null +++ b/p2p/net/nat/nat.go @@ -0,0 +1,439 @@ +package nat + +import ( + "errors" + "fmt" + "strconv" + "strings" + "sync" + "time" + + ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" + + nat "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/fd/go-nat" + goprocess "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" + periodic "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/periodic" + notifier "github.com/ipfs/go-ipfs/thirdparty/notifier" + logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" +) + +var ( + // ErrNoMapping signals no mapping exists for an address + ErrNoMapping = errors.New("mapping not established") +) + +var log = logging.Logger("nat") + +// MappingDuration is a default port mapping duration. +// Port mappings are renewed every (MappingDuration / 3) +const MappingDuration = time.Second * 60 + +// CacheTime is the time a mapping will cache an external address for +const CacheTime = time.Second * 15 + +// DiscoverNAT looks for a NAT device in the network and +// returns an object that can manage port mappings. +func DiscoverNAT() *NAT { + nat, err := nat.DiscoverGateway() + if err != nil { + log.Debug("DiscoverGateway error:", err) + return nil + } + addr, err := nat.GetDeviceAddress() + if err != nil { + log.Debug("DiscoverGateway address error:", err) + } else { + log.Debug("DiscoverGateway address:", addr) + } + return newNAT(nat) +} + +// NAT is an object that manages address port mappings in +// NATs (Network Address Translators). It is a long-running +// service that will periodically renew port mappings, +// and keep an up-to-date list of all the external addresses. +type NAT struct { + nat nat.NAT + proc goprocess.Process // manages nat mappings lifecycle + + mappingmu sync.RWMutex // guards mappings + mappings map[*mapping]struct{} + + Notifier +} + +func newNAT(realNAT nat.NAT) *NAT { + return &NAT{ + nat: realNAT, + proc: goprocess.WithParent(goprocess.Background()), + mappings: make(map[*mapping]struct{}), + } +} + +// Close shuts down all port mappings. NAT can no longer be used. +func (nat *NAT) Close() error { + return nat.proc.Close() +} + +// Process returns the nat's life-cycle manager, for making it listen +// to close signals. +func (nat *NAT) Process() goprocess.Process { + return nat.proc +} + +// Notifier is an object that assists NAT in notifying listeners. +// It is implemented using github.com/ipfs/go-ipfs/thirdparty/notifier +type Notifier struct { + n notifier.Notifier +} + +func (n *Notifier) notifyAll(notify func(n Notifiee)) { + n.n.NotifyAll(func(n notifier.Notifiee) { + notify(n.(Notifiee)) + }) +} + +// Notify signs up notifiee to listen to NAT events. +func (n *Notifier) Notify(notifiee Notifiee) { + n.n.Notify(n) +} + +// StopNotify stops signaling events to notifiee. +func (n *Notifier) StopNotify(notifiee Notifiee) { + n.n.StopNotify(notifiee) +} + +// Notifiee is an interface objects must implement to listen to NAT events. +type Notifiee interface { + + // Called every time a successful mapping happens + // Warning: the port mapping may have changed. If that is the + // case, both MappingSuccess and MappingChanged are called. + MappingSuccess(nat *NAT, m Mapping) + + // Called when mapping a port succeeds, but the mapping is + // with a different port than an earlier success. + MappingChanged(nat *NAT, m Mapping, oldport int, newport int) + + // Called when a port mapping fails. NAT will continue attempting after + // the next period. To stop trying, use: mapping.Close(). After this failure, + // mapping.ExternalPort() will be zero, and nat.ExternalAddrs() will not + // return the address for this mapping. With luck, the next attempt will + // succeed, without the client needing to do anything. + MappingFailed(nat *NAT, m Mapping, oldport int, err error) +} + +// Mapping represents a port mapping in a NAT. +type Mapping interface { + // NAT returns the NAT object this Mapping belongs to. + NAT() *NAT + + // Protocol returns the protocol of this port mapping. This is either + // "tcp" or "udp" as no other protocols are likely to be NAT-supported. + Protocol() string + + // InternalPort returns the internal device port. Mapping will continue to + // try to map InternalPort() to an external facing port. + InternalPort() int + + // ExternalPort returns the external facing port. If the mapping is not + // established, port will be 0 + ExternalPort() int + + // InternalAddr returns the internal address. + InternalAddr() ma.Multiaddr + + // ExternalAddr returns the external facing address. If the mapping is not + // established, addr will be nil, and and ErrNoMapping will be returned. + ExternalAddr() (addr ma.Multiaddr, err error) + + // Close closes the port mapping + Close() error +} + +// keeps republishing +type mapping struct { + sync.Mutex // guards all fields + + nat *NAT + proto string + intport int + extport int + intaddr ma.Multiaddr + proc goprocess.Process + + cached ma.Multiaddr + cacheTime time.Time +} + +func (m *mapping) NAT() *NAT { + m.Lock() + defer m.Unlock() + return m.nat +} + +func (m *mapping) Protocol() string { + m.Lock() + defer m.Unlock() + return m.proto +} + +func (m *mapping) InternalPort() int { + m.Lock() + defer m.Unlock() + return m.intport +} + +func (m *mapping) ExternalPort() int { + m.Lock() + defer m.Unlock() + return m.extport +} + +func (m *mapping) setExternalPort(p int) { + m.Lock() + defer m.Unlock() + m.extport = p +} + +func (m *mapping) InternalAddr() ma.Multiaddr { + m.Lock() + defer m.Unlock() + return m.intaddr +} + +func (m *mapping) ExternalAddr() (ma.Multiaddr, error) { + if time.Now().Sub(m.cacheTime) < CacheTime { + return m.cached, nil + } + + if m.ExternalPort() == 0 { // dont even try right now. + return nil, ErrNoMapping + } + + ip, err := m.nat.nat.GetExternalAddress() + if err != nil { + return nil, err + } + + ipmaddr, err := manet.FromIP(ip) + if err != nil { + return nil, fmt.Errorf("error parsing ip") + } + + // call m.ExternalPort again, as mapping may have changed under our feet. (tocttou) + extport := m.ExternalPort() + if extport == 0 { + return nil, ErrNoMapping + } + + tcp, err := ma.NewMultiaddr(fmt.Sprintf("/%s/%d", m.Protocol(), extport)) + if err != nil { + return nil, err + } + + maddr2 := ipmaddr.Encapsulate(tcp) + + m.cached = maddr2 + m.cacheTime = time.Now() + return maddr2, nil +} + +func (m *mapping) Close() error { + return m.proc.Close() +} + +// Mappings returns a slice of all NAT mappings +func (nat *NAT) Mappings() []Mapping { + nat.mappingmu.Lock() + maps2 := make([]Mapping, 0, len(nat.mappings)) + for m := range nat.mappings { + maps2 = append(maps2, m) + } + nat.mappingmu.Unlock() + return maps2 +} + +func (nat *NAT) addMapping(m *mapping) { + // make mapping automatically close when nat is closed. + nat.proc.AddChild(m.proc) + + nat.mappingmu.Lock() + nat.mappings[m] = struct{}{} + nat.mappingmu.Unlock() +} + +func (nat *NAT) rmMapping(m *mapping) { + nat.mappingmu.Lock() + delete(nat.mappings, m) + nat.mappingmu.Unlock() +} + +// NewMapping attemps to construct a mapping on protocol and internal port +// It will also periodically renew the mapping until the returned Mapping +// -- or its parent NAT -- is Closed. +// +// May not succeed, and mappings may change over time; +// NAT devices may not respect our port requests, and even lie. +// Clients should not store the mapped results, but rather always +// poll our object for the latest mappings. +func (nat *NAT) NewMapping(maddr ma.Multiaddr) (Mapping, error) { + if nat == nil { + return nil, fmt.Errorf("no nat available") + } + + network, addr, err := manet.DialArgs(maddr) + if err != nil { + return nil, fmt.Errorf("DialArgs failed on addr:", maddr.String()) + } + + switch network { + case "tcp", "tcp4", "tcp6": + network = "tcp" + case "udp", "udp4", "udp6": + network = "udp" + default: + return nil, fmt.Errorf("transport not supported by NAT: %s", network) + } + + intports := strings.Split(addr, ":")[1] + intport, err := strconv.Atoi(intports) + if err != nil { + return nil, err + } + + m := &mapping{ + nat: nat, + proto: network, + intport: intport, + intaddr: maddr, + } + m.proc = goprocess.WithTeardown(func() error { + nat.rmMapping(m) + return nil + }) + nat.addMapping(m) + + m.proc.AddChild(periodic.Every(MappingDuration/3, func(worker goprocess.Process) { + nat.establishMapping(m) + })) + + // do it once synchronously, so first mapping is done right away, and before exiting, + // allowing users -- in the optimistic case -- to use results right after. + nat.establishMapping(m) + return m, nil +} + +func (nat *NAT) establishMapping(m *mapping) { + oldport := m.ExternalPort() + log.Debugf("Attempting port map: %s/%d", m.Protocol(), m.InternalPort()) + newport, err := nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), "http", MappingDuration) + + failure := func() { + m.setExternalPort(0) // clear mapping + // TODO: log.Event + log.Debugf("failed to establish port mapping: %s", err) + nat.Notifier.notifyAll(func(n Notifiee) { + n.MappingFailed(nat, m, oldport, err) + }) + + // we do not close if the mapping failed, + // because it may work again next time. + } + + if err != nil || newport == 0 { + failure() + return + } + + m.setExternalPort(newport) + ext, err := m.ExternalAddr() + if err != nil { + log.Debugf("NAT Mapping addr error: %s %s", m.InternalAddr(), err) + failure() + return + } + + log.Debugf("NAT Mapping: %s --> %s", m.InternalAddr(), ext) + if oldport != 0 && newport != oldport { + log.Debugf("failed to renew same port mapping: ch %d -> %d", oldport, newport) + nat.Notifier.notifyAll(func(n Notifiee) { + n.MappingChanged(nat, m, oldport, newport) + }) + } + + nat.Notifier.notifyAll(func(n Notifiee) { + n.MappingSuccess(nat, m) + }) +} + +// PortMapAddrs attempts to open (and continue to keep open) +// port mappings for given addrs. This function blocks until +// all addresses have been tried. This allows clients to +// retrieve results immediately after: +// +// nat.PortMapAddrs(addrs) +// mapped := nat.ExternalAddrs() +// +// Some may not succeed, and mappings may change over time; +// NAT devices may not respect our port requests, and even lie. +// Clients should not store the mapped results, but rather always +// poll our object for the latest mappings. +func (nat *NAT) PortMapAddrs(addrs []ma.Multiaddr) { + // spin off addr mappings independently. + var wg sync.WaitGroup + for _, addr := range addrs { + // do all of them concurrently + wg.Add(1) + go func() { + defer wg.Done() + nat.NewMapping(addr) + }() + } + wg.Wait() +} + +// MappedAddrs returns address mappings NAT believes have been +// successfully established. Unsuccessful mappings are nil. This is: +// +// map[internalAddr]externalAddr +// +// This set of mappings _may not_ be correct, as NAT devices are finicky. +// Consider this with _best effort_ semantics. +func (nat *NAT) MappedAddrs() map[ma.Multiaddr]ma.Multiaddr { + + mappings := nat.Mappings() + addrmap := make(map[ma.Multiaddr]ma.Multiaddr, len(mappings)) + + for _, m := range mappings { + i := m.InternalAddr() + e, err := m.ExternalAddr() + if err != nil { + addrmap[i] = nil + } else { + addrmap[i] = e + } + } + return addrmap +} + +// ExternalAddrs returns a list of addresses that NAT believes have +// been successfully established. Unsuccessful mappings are omitted, +// so nat.ExternalAddrs() may return less addresses than nat.InternalAddrs(). +// To see which addresses are mapped, use nat.MappedAddrs(). +// +// This set of mappings _may not_ be correct, as NAT devices are finicky. +// Consider this with _best effort_ semantics. +func (nat *NAT) ExternalAddrs() []ma.Multiaddr { + mappings := nat.Mappings() + addrs := make([]ma.Multiaddr, 0, len(mappings)) + for _, m := range mappings { + a, err := m.ExternalAddr() + if err != nil { + continue // this mapping not currently successful. + } + addrs = append(addrs, a) + } + return addrs +} From 68f8d27d0713acee39febef5eee6da21a00c65ee Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Wed, 30 Sep 2015 19:24:00 -0400 Subject: [PATCH 0023/3965] rewrote imports to p2p --- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peer.go | 2 +- p2p/host/peerstore/peer_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/queue/distance.go | 2 +- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 2 +- p2p/host/peerstore/queue/sync.go | 2 +- 8 files changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index db5f3a94d8..7975ac2ddf 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -6,8 +6,8 @@ import ( "testing" "time" - peer "github.com/ipfs/go-ipfs/p2p/peer" testutil "github.com/ipfs/go-ipfs/util/testutil" + peer "github.com/ipfs/go-libp2p/p2p/peer" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 4e1e3e3ab1..d076ac862e 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -11,9 +11,9 @@ import ( ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" mh "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multihash" - ic "github.com/ipfs/go-ipfs/p2p/crypto" u "github.com/ipfs/go-ipfs/util" logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + ic "github.com/ipfs/go-libp2p/p2p/crypto" ) var log = logging.Logger("peer") diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 80d6d9f079..9d420d36e1 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -6,10 +6,10 @@ import ( "strings" "testing" - ic "github.com/ipfs/go-ipfs/p2p/crypto" - . "github.com/ipfs/go-ipfs/p2p/peer" u "github.com/ipfs/go-ipfs/util" tu "github.com/ipfs/go-ipfs/util/testutil" + ic "github.com/ipfs/go-libp2p/p2p/crypto" + . "github.com/ipfs/go-libp2p/p2p/peer" b58 "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-base58" ) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 30a12ebbc9..c9bc643500 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -5,7 +5,7 @@ import ( "sync" "time" - ic "github.com/ipfs/go-ipfs/p2p/crypto" + ic "github.com/ipfs/go-libp2p/p2p/crypto" ds "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore" dssync "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore/sync" diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 2b3bce82d3..ada2aed002 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -6,8 +6,8 @@ import ( "sync" key "github.com/ipfs/go-ipfs/blocks/key" - peer "github.com/ipfs/go-ipfs/p2p/peer" ks "github.com/ipfs/go-ipfs/routing/keyspace" + peer "github.com/ipfs/go-libp2p/p2p/peer" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index 6b0f8c5c9e..3c55e7e603 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import peer "github.com/ipfs/go-ipfs/p2p/peer" +import peer "github.com/ipfs/go-libp2p/p2p/peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index c303550a92..7e31b5f305 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,8 +7,8 @@ import ( "time" key "github.com/ipfs/go-ipfs/blocks/key" - peer "github.com/ipfs/go-ipfs/p2p/peer" u "github.com/ipfs/go-ipfs/util" + peer "github.com/ipfs/go-libp2p/p2p/peer" context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" ) diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 04775a0f99..04eac7e43f 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -2,8 +2,8 @@ package queue import ( context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" - peer "github.com/ipfs/go-ipfs/p2p/peer" logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + peer "github.com/ipfs/go-libp2p/p2p/peer" ) var log = logging.Logger("peerqueue") From 9bf24a5968e97f268ccc3bbf69e6bfee3fcfcede Mon Sep 17 00:00:00 2001 From: Juan Batiz-Benet Date: Wed, 30 Sep 2015 19:24:00 -0400 Subject: [PATCH 0024/3965] rewrote imports to p2p --- p2p/net/swarm/dial_test.go | 4 ++-- p2p/net/swarm/peers_test.go | 2 +- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm.go | 10 +++++----- p2p/net/swarm/swarm_addr.go | 4 ++-- p2p/net/swarm/swarm_addr_test.go | 6 +++--- p2p/net/swarm/swarm_conn.go | 8 ++++---- p2p/net/swarm/swarm_dial.go | 10 +++++----- p2p/net/swarm/swarm_listen.go | 10 +++++----- p2p/net/swarm/swarm_net.go | 6 +++--- p2p/net/swarm/swarm_net_test.go | 4 ++-- p2p/net/swarm/swarm_notif_test.go | 4 ++-- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 6 +++--- 14 files changed, 39 insertions(+), 39 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 0ed10f4d30..4d4ea1515e 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -6,8 +6,8 @@ import ( "testing" "time" - addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" - peer "github.com/ipfs/go-ipfs/p2p/peer" + addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" + peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-ipfs/util/testutil" ci "github.com/ipfs/go-ipfs/util/testutil/ci" diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 2f8b07ef0b..65756a6c2b 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -3,7 +3,7 @@ package swarm import ( "testing" - peer "github.com/ipfs/go-ipfs/p2p/peer" + peer "github.com/ipfs/go-libp2p/p2p/peer" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index dfb775c971..09be20a60a 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -5,8 +5,8 @@ import ( "testing" "time" - peer "github.com/ipfs/go-ipfs/p2p/peer" ci "github.com/ipfs/go-ipfs/util/testutil/ci" + peer "github.com/ipfs/go-libp2p/p2p/peer" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 512369013c..b20b128930 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -7,12 +7,12 @@ import ( "sync" "time" - metrics "github.com/ipfs/go-ipfs/metrics" - inet "github.com/ipfs/go-ipfs/p2p/net" - filter "github.com/ipfs/go-ipfs/p2p/net/filter" - addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" - peer "github.com/ipfs/go-ipfs/p2p/peer" logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + inet "github.com/ipfs/go-libp2p/p2p/net" + filter "github.com/ipfs/go-libp2p/p2p/net/filter" + addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" + peer "github.com/ipfs/go-libp2p/p2p/peer" + metrics "github.com/ipfs/go-libp2p/util/metrics" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 8f9360d9d0..c3eaa3b001 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -1,8 +1,8 @@ package swarm import ( - conn "github.com/ipfs/go-ipfs/p2p/net/conn" - addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" + conn "github.com/ipfs/go-libp2p/p2p/net/conn" + addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index b75b491c42..fbfbcb8c3b 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -3,10 +3,10 @@ package swarm import ( "testing" - metrics "github.com/ipfs/go-ipfs/metrics" - addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" - peer "github.com/ipfs/go-ipfs/p2p/peer" testutil "github.com/ipfs/go-ipfs/util/testutil" + addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" + peer "github.com/ipfs/go-libp2p/p2p/peer" + metrics "github.com/ipfs/go-libp2p/util/metrics" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 126ace627a..dd9f308eef 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -3,10 +3,10 @@ package swarm import ( "fmt" - ic "github.com/ipfs/go-ipfs/p2p/crypto" - inet "github.com/ipfs/go-ipfs/p2p/net" - conn "github.com/ipfs/go-ipfs/p2p/net/conn" - peer "github.com/ipfs/go-ipfs/p2p/peer" + ic "github.com/ipfs/go-libp2p/p2p/crypto" + inet "github.com/ipfs/go-libp2p/p2p/net" + conn "github.com/ipfs/go-libp2p/p2p/net/conn" + peer "github.com/ipfs/go-libp2p/p2p/peer" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index a68d288059..d48d8a95f7 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -8,11 +8,11 @@ import ( "sync" "time" - mconn "github.com/ipfs/go-ipfs/metrics/conn" - conn "github.com/ipfs/go-ipfs/p2p/net/conn" - addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" - peer "github.com/ipfs/go-ipfs/p2p/peer" - lgbl "github.com/ipfs/go-ipfs/util/eventlog/loggables" + conn "github.com/ipfs/go-libp2p/p2p/net/conn" + addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" + peer "github.com/ipfs/go-libp2p/p2p/peer" + lgbl "github.com/ipfs/go-libp2p/util/eventlog/loggables" + mconn "github.com/ipfs/go-libp2p/util/metrics/conn" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index d1bcb07521..d41e173574 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -3,11 +3,11 @@ package swarm import ( "fmt" - mconn "github.com/ipfs/go-ipfs/metrics/conn" - inet "github.com/ipfs/go-ipfs/p2p/net" - conn "github.com/ipfs/go-ipfs/p2p/net/conn" - addrutil "github.com/ipfs/go-ipfs/p2p/net/swarm/addr" - lgbl "github.com/ipfs/go-ipfs/util/eventlog/loggables" + inet "github.com/ipfs/go-libp2p/p2p/net" + conn "github.com/ipfs/go-libp2p/p2p/net/conn" + addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" + lgbl "github.com/ipfs/go-libp2p/util/eventlog/loggables" + mconn "github.com/ipfs/go-libp2p/util/metrics/conn" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index da9b52251e..863ef5e64f 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -3,10 +3,10 @@ package swarm import ( "fmt" - peer "github.com/ipfs/go-ipfs/p2p/peer" + peer "github.com/ipfs/go-libp2p/p2p/peer" - metrics "github.com/ipfs/go-ipfs/metrics" - inet "github.com/ipfs/go-ipfs/p2p/net" + inet "github.com/ipfs/go-libp2p/p2p/net" + metrics "github.com/ipfs/go-libp2p/util/metrics" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index b5f5c56996..59c4892b87 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -6,8 +6,8 @@ import ( "time" context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" - inet "github.com/ipfs/go-ipfs/p2p/net" - testutil "github.com/ipfs/go-ipfs/p2p/test/util" + inet "github.com/ipfs/go-libp2p/p2p/net" + testutil "github.com/ipfs/go-libp2p/p2p/test/util" ) // TestConnectednessCorrect starts a few networks, connects a few diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 9b0e3a9bab..9bbf43e529 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -7,8 +7,8 @@ import ( ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" - inet "github.com/ipfs/go-ipfs/p2p/net" - peer "github.com/ipfs/go-ipfs/p2p/peer" + inet "github.com/ipfs/go-libp2p/p2p/net" + peer "github.com/ipfs/go-libp2p/p2p/peer" ) func TestNotifications(t *testing.T) { diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 66dd6353b7..bcb188bc49 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -1,7 +1,7 @@ package swarm import ( - inet "github.com/ipfs/go-ipfs/p2p/net" + inet "github.com/ipfs/go-libp2p/p2p/net" ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" ) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 9193db0109..1d0479f2cb 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -9,10 +9,10 @@ import ( "testing" "time" - metrics "github.com/ipfs/go-ipfs/metrics" - inet "github.com/ipfs/go-ipfs/p2p/net" - peer "github.com/ipfs/go-ipfs/p2p/peer" testutil "github.com/ipfs/go-ipfs/util/testutil" + inet "github.com/ipfs/go-libp2p/p2p/net" + peer "github.com/ipfs/go-libp2p/p2p/peer" + metrics "github.com/ipfs/go-libp2p/util/metrics" ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" From 9604eaef7db3d5b896d20cf8d4b5ba9721674364 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 11 Nov 2015 16:15:47 -0800 Subject: [PATCH 0025/3965] extract from 0.4.0 --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peer.go | 10 +++++----- p2p/host/peerstore/peer_test.go | 6 +++--- p2p/host/peerstore/peerstore.go | 6 +++--- p2p/host/peerstore/queue/distance.go | 4 ++-- p2p/host/peerstore/queue/queue_test.go | 6 +++--- p2p/host/peerstore/queue/sync.go | 4 ++-- 11 files changed, 23 insertions(+), 23 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index d8b942bef9..a172373d8d 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index ea75a2bd6a..46140641ca 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 02b0fd1187..87b8206eee 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -4,7 +4,7 @@ import ( "sync" "time" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 2bd480fdd5..3e19530539 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) func IDS(t *testing.T, ids string) ID { diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 7975ac2ddf..4343c0875e 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -6,8 +6,8 @@ import ( "testing" "time" - testutil "github.com/ipfs/go-ipfs/util/testutil" peer "github.com/ipfs/go-libp2p/p2p/peer" + testutil "util/testutil" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index d076ac862e..549b2093a6 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -7,13 +7,13 @@ import ( "fmt" "strings" - b58 "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-base58" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - mh "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multihash" + b58 "github.com/jbenet/go-base58" + ma "github.com/jbenet/go-multiaddr" + mh "github.com/jbenet/go-multihash" - u "github.com/ipfs/go-ipfs/util" - logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ic "github.com/ipfs/go-libp2p/p2p/crypto" + u "util" ) var log = logging.Logger("peer") diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 9d420d36e1..1d4ca2595b 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -6,12 +6,12 @@ import ( "strings" "testing" - u "github.com/ipfs/go-ipfs/util" - tu "github.com/ipfs/go-ipfs/util/testutil" ic "github.com/ipfs/go-libp2p/p2p/crypto" . "github.com/ipfs/go-libp2p/p2p/peer" + u "util" + tu "util/testutil" - b58 "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-base58" + b58 "github.com/jbenet/go-base58" ) var gen1 keyset // generated diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index c9bc643500..9f3a98d25d 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -7,9 +7,9 @@ import ( ic "github.com/ipfs/go-libp2p/p2p/crypto" - ds "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore" - dssync "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore/sync" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + ds "github.com/jbenet/go-datastore" + dssync "github.com/jbenet/go-datastore/sync" + ma "github.com/jbenet/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index ada2aed002..6b721abb76 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,9 +5,9 @@ import ( "math/big" "sync" - key "github.com/ipfs/go-ipfs/blocks/key" - ks "github.com/ipfs/go-ipfs/routing/keyspace" + ks "Qma4vHVBYKDiKS5VpvtLNJHHDbL7S6VRsvxxmBnBFfKP3k/go-keyspace" peer "github.com/ipfs/go-libp2p/p2p/peer" + key "github.com/whyrusleeping/go-key" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 7e31b5f305..d85f84b7c2 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -6,11 +6,11 @@ import ( "testing" "time" - key "github.com/ipfs/go-ipfs/blocks/key" - u "github.com/ipfs/go-ipfs/util" peer "github.com/ipfs/go-libp2p/p2p/peer" + key "github.com/whyrusleeping/go-key" + u "util" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + context "golang.org/x/net/context" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 04eac7e43f..2a88683e9c 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -1,9 +1,9 @@ package queue import ( - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" - logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + context "golang.org/x/net/context" peer "github.com/ipfs/go-libp2p/p2p/peer" + logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ) var log = logging.Logger("peerqueue") From 9b779001c2c9aa0a3c0760d962b184b484863b36 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 11 Nov 2015 16:15:47 -0800 Subject: [PATCH 0026/3965] extract from 0.4.0 --- p2p/net/swarm/addr/addr.go | 10 +- p2p/net/swarm/addr/addr_test.go | 4 +- p2p/net/swarm/dial_test.go | 58 +++++-- p2p/net/swarm/peers_test.go | 4 +- p2p/net/swarm/simul_test.go | 10 +- p2p/net/swarm/swarm.go | 71 +++++--- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 8 +- p2p/net/swarm/swarm_conn.go | 6 +- p2p/net/swarm/swarm_dial.go | 265 ++++++++++++++++++++---------- p2p/net/swarm/swarm_listen.go | 104 +++++++----- p2p/net/swarm/swarm_net.go | 8 +- p2p/net/swarm/swarm_net_test.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 4 +- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 17 +- 16 files changed, 379 insertions(+), 196 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 649d54b3c0..cec777a25e 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,14 +3,14 @@ package addrutil import ( "fmt" - logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + context "golang.org/x/net/context" ) -var log = logging.Logger("p2p/net/swarm/addr") +var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") // SupportedTransportStrings is the list of supported transports for the swarm. // These are strings of encapsulated multiaddr protocols. E.g.: diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index eb843ffc09..49eec13511 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,8 +3,8 @@ package addrutil import ( "testing" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" ) func newMultiaddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 4d4ea1515e..f121f97f05 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -2,6 +2,7 @@ package swarm import ( "net" + "sort" "sync" "testing" "time" @@ -9,12 +10,12 @@ import ( addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - testutil "github.com/ipfs/go-ipfs/util/testutil" - ci "github.com/ipfs/go-ipfs/util/testutil/ci" + testutil "util/testutil" + ci "util/testutil/ci" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + context "golang.org/x/net/context" ) func acceptAndHang(l net.Listener) { @@ -419,18 +420,18 @@ func TestDialBackoffClears(t *testing.T) { } s1.peers.AddAddrs(s2.local, ifaceAddrs1, peer.PermanentAddrTTL) - before = time.Now() + if _, err := s1.Dial(ctx, s2.local); err == nil { + t.Fatal("should have failed to dial backed off peer") + } + + time.Sleep(baseBackoffTime) + if c, err := s1.Dial(ctx, s2.local); err != nil { t.Fatal(err) } else { c.Close() t.Log("correctly connected") } - duration = time.Now().Sub(before) - - if duration >= dt { - // t.Error("took too long", duration, dt) - } if s1.backf.Backoff(s2.local) { t.Error("s2 should no longer be on backoff") @@ -438,3 +439,38 @@ func TestDialBackoffClears(t *testing.T) { t.Log("correctly cleared backoff") } } + +func mkAddr(t *testing.T, s string) ma.Multiaddr { + a, err := ma.NewMultiaddr(s) + if err != nil { + t.Fatal(err) + } + + return a +} + +func TestAddressSorting(t *testing.T) { + u1 := mkAddr(t, "/ip4/152.12.23.53/udp/1234/utp") + u2l := mkAddr(t, "/ip4/127.0.0.1/udp/1234/utp") + local := mkAddr(t, "/ip4/127.0.0.1/tcp/1234") + norm := mkAddr(t, "/ip4/6.5.4.3/tcp/1234") + + l := AddrList{local, u1, u2l, norm} + sort.Sort(l) + + if !l[0].Equal(u2l) { + t.Fatal("expected utp local addr to be sorted first: ", l[0]) + } + + if !l[1].Equal(u1) { + t.Fatal("expected utp addr to be sorted second") + } + + if !l[2].Equal(local) { + t.Fatal("expected tcp localhost addr thid") + } + + if !l[3].Equal(norm) { + t.Fatal("expected normal addr last") + } +} diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 65756a6c2b..b107fbfe83 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -5,8 +5,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 09be20a60a..63e07ff8ac 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -1,15 +1,16 @@ package swarm import ( + "runtime" "sync" "testing" "time" - ci "github.com/ipfs/go-ipfs/util/testutil/ci" peer "github.com/ipfs/go-libp2p/p2p/peer" + ci "util/testutil/ci" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestSimultOpen(t *testing.T) { @@ -49,7 +50,8 @@ func TestSimultOpenMany(t *testing.T) { addrs := 20 rounds := 10 - if ci.IsRunning() { + if ci.IsRunning() || runtime.GOOS == "darwin" { + // osx has a limit of 256 file descriptors addrs = 10 rounds = 5 } diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index b20b128930..245b7204f7 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -7,22 +7,26 @@ import ( "sync" "time" - logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + metrics "github.com/ipfs/go-libp2p/p2p/metrics" + mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" inet "github.com/ipfs/go-libp2p/p2p/net" + conn "github.com/ipfs/go-libp2p/p2p/net/conn" filter "github.com/ipfs/go-libp2p/p2p/net/filter" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" + transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" - metrics "github.com/ipfs/go-libp2p/util/metrics" - - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" - pst "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-stream-muxer" - psy "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-stream-muxer/yamux" - "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" - goprocessctx "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context" - prom "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus" - mafilter "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/whyrusleeping/multiaddr-filter" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + pst "github.com/jbenet/go-stream-muxer" + psmss "github.com/jbenet/go-stream-muxer/multistream" + "github.com/jbenet/goprocess" + goprocessctx "github.com/jbenet/goprocess/context" + prom "github.com/prometheus/client_golang/prometheus" + mafilter "github.com/whyrusleeping/multiaddr-filter" + context "golang.org/x/net/context" + + logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ) var log = logging.Logger("swarm2") @@ -37,9 +41,7 @@ var peersTotal = prom.NewGaugeVec(prom.GaugeOpts{ }, []string{"peer_id"}) func init() { - tpt := *psy.DefaultTransport - tpt.MaxStreamWindowSize = 512 * 1024 - PSTransport = &tpt + PSTransport = psmss.NewTransport() } // Swarm is a connection muxer, allowing connections to other peers to @@ -58,12 +60,19 @@ type Swarm struct { backf dialbackoff dialT time.Duration // mainly for tests + dialer *conn.Dialer + notifmu sync.RWMutex notifs map[inet.Notifiee]ps.Notifiee + transports []transport.Transport + // filters for addresses that shouldnt be dialed Filters *filter.Filters + // file descriptor rate limited + fdRateLimit chan struct{} + proc goprocess.Process ctx context.Context bwc metrics.Reporter @@ -78,15 +87,22 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, return nil, err } + wrap := func(c transport.Conn) transport.Conn { + return mconn.WrapConn(bwc, c) + } + s := &Swarm{ - swarm: ps.NewSwarm(PSTransport), - local: local, - peers: peers, - ctx: ctx, - dialT: DialTimeout, - notifs: make(map[inet.Notifiee]ps.Notifiee), - bwc: bwc, - Filters: filter.NewFilters(), + swarm: ps.NewSwarm(PSTransport), + local: local, + peers: peers, + ctx: ctx, + dialT: DialTimeout, + notifs: make(map[inet.Notifiee]ps.Notifiee), + transports: []transport.Transport{transport.NewTCPTransport()}, + bwc: bwc, + fdRateLimit: make(chan struct{}, concurrentFdDials), + Filters: filter.NewFilters(), + dialer: conn.NewDialer(local, peers.PrivKey(local), wrap), } // configure Swarm @@ -97,7 +113,12 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, prom.MustRegisterOrGet(peersTotal) s.Notify((*metricsNotifiee)(s)) - return s, s.listen(listenAddrs) + err = s.setupInterfaces(listenAddrs) + if err != nil { + return nil, err + } + + return s, nil } func (s *Swarm) teardown() error { @@ -130,7 +151,7 @@ func (s *Swarm) Listen(addrs ...ma.Multiaddr) error { return err } - return s.listen(addrs) + return s.setupInterfaces(addrs) } // Process returns the Process of the swarm diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index c3eaa3b001..039683bfb9 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -4,7 +4,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index fbfbcb8c3b..b1f167ae49 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -3,13 +3,13 @@ package swarm import ( "testing" - testutil "github.com/ipfs/go-ipfs/util/testutil" + metrics "github.com/ipfs/go-libp2p/p2p/metrics" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - metrics "github.com/ipfs/go-libp2p/util/metrics" + testutil "util/testutil" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index dd9f308eef..840a9b465c 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,9 +8,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + context "golang.org/x/net/context" ) // a Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index d48d8a95f7..4790eb0738 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -1,25 +1,21 @@ package swarm import ( + "bytes" "errors" "fmt" - "math/rand" - "net" + "sort" "sync" "time" + "github.com/jbenet/go-multiaddr-net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - lgbl "github.com/ipfs/go-libp2p/util/eventlog/loggables" - mconn "github.com/ipfs/go-libp2p/util/metrics/conn" - - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" - process "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" - processctx "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context" - ratelimit "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/ratelimit" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + lgbl "util/eventlog/loggables" + + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) // Diagram of dial sync: @@ -44,6 +40,9 @@ var ( // add loop back in Dial(.) const dialAttempts = 1 +// number of concurrent outbound dials over transports that consume file descriptors +const concurrentFdDials = 160 + // DialTimeout is the amount of time each dial attempt has. We can think about making // this larger down the road, or putting more granular timeouts (i.e. within each // subcomponent of Dial) @@ -115,6 +114,7 @@ func (ds *dialsync) Unlock(dst peer.ID) { if !found { panic("called dialDone with no ongoing dials to peer: " + dst.Pretty()) } + delete(ds.ongoing, dst) // remove ongoing dial close(wait) // release everyone else ds.lock.Unlock() @@ -145,44 +145,71 @@ func (ds *dialsync) Unlock(dst peer.ID) { // dialbackoff.Clear(p) // } // + type dialbackoff struct { - entries map[peer.ID]struct{} + entries map[peer.ID]*backoffPeer lock sync.RWMutex } +type backoffPeer struct { + tries int + until time.Time +} + func (db *dialbackoff) init() { if db.entries == nil { - db.entries = make(map[peer.ID]struct{}) + db.entries = make(map[peer.ID]*backoffPeer) } } // Backoff returns whether the client should backoff from dialing -// peeer p -func (db *dialbackoff) Backoff(p peer.ID) bool { +// peer p +func (db *dialbackoff) Backoff(p peer.ID) (backoff bool) { db.lock.Lock() + defer db.lock.Unlock() db.init() - _, found := db.entries[p] - db.lock.Unlock() - return found + bp, found := db.entries[p] + if found && time.Now().Before(bp.until) { + return true + } + + return false } +const baseBackoffTime = time.Second * 5 +const maxBackoffTime = time.Minute * 5 + // AddBackoff lets other nodes know that we've entered backoff with // peer p, so dialers should not wait unnecessarily. We still will // attempt to dial with one goroutine, in case we get through. func (db *dialbackoff) AddBackoff(p peer.ID) { db.lock.Lock() + defer db.lock.Unlock() db.init() - db.entries[p] = struct{}{} - db.lock.Unlock() + bp, ok := db.entries[p] + if !ok { + db.entries[p] = &backoffPeer{ + tries: 1, + until: time.Now().Add(baseBackoffTime), + } + return + } + + expTimeAdd := time.Second * time.Duration(bp.tries*bp.tries) + if expTimeAdd > maxBackoffTime { + expTimeAdd = maxBackoffTime + } + bp.until = time.Now().Add(baseBackoffTime + expTimeAdd) + bp.tries++ } // Clear removes a backoff record. Clients should call this after a // successful Dial. func (db *dialbackoff) Clear(p peer.ID) { db.lock.Lock() + defer db.lock.Unlock() db.init() delete(db.entries, p) - db.lock.Unlock() } // Dial connects to a peer. @@ -225,14 +252,20 @@ func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) // check if there's an ongoing dial to this peer if ok, wait := s.dsync.Lock(p); ok { + defer s.dsync.Unlock(p) + + // if this peer has been backed off, lets get out of here + if s.backf.Backoff(p) { + log.Event(ctx, "swarmDialBackoff", logdial) + return nil, ErrDialBackoff + } + // ok, we have been charged to dial! let's do it. // if it succeeds, dial will add the conn to the swarm itself. - defer log.EventBegin(ctx, "swarmDialAttemptStart", logdial).Done() ctxT, cancel := context.WithTimeout(ctx, s.dialT) conn, err := s.dial(ctxT, p) cancel() - s.dsync.Unlock(p) log.Debugf("dial end %s", conn) if err != nil { log.Event(ctx, "swarmDialBackoffAdd", logdial) @@ -287,14 +320,6 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { log.Debug("Dial not given PrivateKey, so WILL NOT SECURE conn.") } - // get our own addrs. try dialing out from our listener addresses (reusing ports) - // Note that using our peerstore's addresses here is incorrect, as that would - // include observed addresses. TODO: make peerstore's address book smarter. - localAddrs := s.ListenAddresses() - if len(localAddrs) == 0 { - log.Debug("Dialing out with no local addresses.") - } - // get remote peer addrs remoteAddrs := s.peers.Addrs(p) // make sure we can use the addresses. @@ -319,23 +344,8 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { return nil, err } - // open connection to peer - d := &conn.Dialer{ - Dialer: manet.Dialer{ - Dialer: net.Dialer{ - Timeout: s.dialT, - }, - }, - LocalPeer: s.local, - LocalAddrs: localAddrs, - PrivateKey: sk, - Wrapper: func(c manet.Conn) manet.Conn { - return mconn.WrapConn(s.bwc, c) - }, - } - // try to get a connection to any addr - connC, err := s.dialAddrs(ctx, d, p, remoteAddrs) + connC, err := s.dialAddrs(ctx, p, remoteAddrs) if err != nil { logdial["error"] = err return nil, err @@ -355,7 +365,10 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { return swarmC, nil } -func (s *Swarm) dialAddrs(ctx context.Context, d *conn.Dialer, p peer.ID, remoteAddrs []ma.Multiaddr) (conn.Conn, error) { +func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs []ma.Multiaddr) (conn.Conn, error) { + + // sort addresses so preferred addresses are dialed sooner + sort.Sort(AddrList(remoteAddrs)) // try to connect to one of the peer's known addresses. // we dial concurrently to each of the addresses, which: @@ -367,78 +380,89 @@ func (s *Swarm) dialAddrs(ctx context.Context, d *conn.Dialer, p peer.ID, remote ctx, cancel := context.WithCancel(ctx) defer cancel() // cancel work when we exit func - foundConn := make(chan struct{}) - conns := make(chan conn.Conn, len(remoteAddrs)) + conns := make(chan conn.Conn) errs := make(chan error, len(remoteAddrs)) // dialSingleAddr is used in the rate-limited async thing below. dialSingleAddr := func(addr ma.Multiaddr) { - connC, err := s.dialAddr(ctx, d, p, addr) + // rebind chans in scope so we can nil them out easily + connsout := conns + errsout := errs + + connC, err := s.dialAddr(ctx, p, addr) + if err != nil { + connsout = nil + } else if connC == nil { + // NOTE: this really should never happen + log.Errorf("failed to dial %s %s and got no error!", p, addr) + err = fmt.Errorf("failed to dial %s %s", p, addr) + connsout = nil + } else { + errsout = nil + } // check parent still wants our results select { - case <-foundConn: + case <-ctx.Done(): if connC != nil { connC.Close() } - return - default: - } - - if err != nil { - errs <- err - } else if connC == nil { - errs <- fmt.Errorf("failed to dial %s %s", p, addr) - } else { - conns <- connC + case errsout <- err: + case connsout <- connC: } } // this whole thing is in a goroutine so we can use foundConn // to end early. go func() { - // rate limiting just in case. at most 10 addrs at once. - limiter := ratelimit.NewRateLimiter(process.Background(), 10) - limiter.Go(func(worker process.Process) { - // permute addrs so we try different sets first each time. - for _, i := range rand.Perm(len(remoteAddrs)) { - select { - case <-foundConn: // if one of them succeeded already - break - case <-worker.Closing(): // our context was cancelled - break - default: - } - - workerAddr := remoteAddrs[i] // shadow variable to avoid race - limiter.LimitedGo(func(worker process.Process) { - dialSingleAddr(workerAddr) - }) + limiter := make(chan struct{}, 8) + for _, addr := range remoteAddrs { + // returns whatever ratelimiting is acceptable for workerAddr. + // may not rate limit at all. + rl := s.addrDialRateLimit(addr) + select { + case <-ctx.Done(): // our context was cancelled + return + case rl <- struct{}{}: + // take the token, move on + } + + select { + case <-ctx.Done(): // our context was cancelled + return + case limiter <- struct{}{}: + // take the token, move on } - }) - processctx.CloseAfterContext(limiter, ctx) + go func(rlc <-chan struct{}, a ma.Multiaddr) { + dialSingleAddr(a) + <-limiter + <-rlc + }(rl, addr) + } }() - // wair fot the results. + // wair for the results. exitErr := fmt.Errorf("failed to dial %s", p) - for i := 0; i < len(remoteAddrs); i++ { + for range remoteAddrs { select { case exitErr = <-errs: // log.Debug("dial error: ", exitErr) case connC := <-conns: // take the first + return asap - close(foundConn) return connC, nil + case <-ctx.Done(): + // break out and return error + break } } return nil, exitErr } -func (s *Swarm) dialAddr(ctx context.Context, d *conn.Dialer, p peer.ID, addr ma.Multiaddr) (conn.Conn, error) { +func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (conn.Conn, error) { log.Debugf("%s swarm dialing %s %s", s.local, p, addr) - connC, err := d.Dial(ctx, addr, p) + connC, err := s.dialer.Dial(ctx, addr, p) if err != nil { return nil, fmt.Errorf("%s --> %s dial attempt failed: %s", s.local, p, err) } @@ -491,3 +515,72 @@ func dialConnSetup(ctx context.Context, s *Swarm, connC conn.Conn) (*Conn, error return swarmC, err } + +// addrDialRateLimit returns a ratelimiting channel for dialing transport +// addrs like a. for example, tcp is fd-ratelimited. utp is not ratelimited. +func (s *Swarm) addrDialRateLimit(a ma.Multiaddr) chan struct{} { + if isFDCostlyTransport(a) { + return s.fdRateLimit + } + + // do not rate limit it at all + return make(chan struct{}, 1) +} + +func isFDCostlyTransport(a ma.Multiaddr) bool { + return isTCPMultiaddr(a) +} + +func isTCPMultiaddr(a ma.Multiaddr) bool { + p := a.Protocols() + return len(p) == 2 && (p[0].Name == "ip4" || p[0].Name == "ip6") && p[1].Name == "tcp" +} + +type AddrList []ma.Multiaddr + +func (al AddrList) Len() int { + return len(al) +} + +func (al AddrList) Swap(i, j int) { + al[i], al[j] = al[j], al[i] +} + +func (al AddrList) Less(i, j int) bool { + a := al[i] + b := al[j] + + // dial localhost addresses next, they should fail immediately + lba := manet.IsIPLoopback(a) + lbb := manet.IsIPLoopback(b) + if lba { + if !lbb { + return true + } + } + + // dial utp and similar 'non-fd-consuming' addresses first + fda := isFDCostlyTransport(a) + fdb := isFDCostlyTransport(b) + if !fda { + if fdb { + return true + } + + // if neither consume fd's, assume equal ordering + return false + } + + // if 'b' doesnt take a file descriptor + if !fdb { + return false + } + + // if 'b' is loopback and both take file descriptors + if lbb { + return false + } + + // for the rest, just sort by bytes + return bytes.Compare(a.Bytes(), b.Bytes()) > 0 +} diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index d41e173574..f3da2eeea7 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -3,68 +3,81 @@ package swarm import ( "fmt" + mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" - addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - lgbl "github.com/ipfs/go-libp2p/util/eventlog/loggables" - mconn "github.com/ipfs/go-libp2p/util/metrics/conn" - - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" - ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" - multierr "github.com/ipfs/go-ipfs/thirdparty/multierr" + transport "github.com/ipfs/go-libp2p/p2p/net/transport" + lgbl "util/eventlog/loggables" + + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + context "golang.org/x/net/context" ) -// Open listeners for each network the swarm should listen on -func (s *Swarm) listen(addrs []ma.Multiaddr) error { +// Open listeners and reuse-dialers for the given addresses +func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { + errs := make([]error, len(addrs)) + var succeeded int + for i, a := range addrs { + tpt := s.transportForAddr(a) + if tpt == nil { + errs[i] = fmt.Errorf("no transport for address: %s", a) + continue + } - for _, addr := range addrs { - if !addrutil.AddrUsable(addr, true) { - return fmt.Errorf("cannot use addr: %s", addr) + d, err := tpt.Dialer(a, transport.TimeoutOpt(DialTimeout), transport.ReusePorts) + if err != nil { + errs[i] = err + continue } - } - retErr := multierr.New() + s.dialer.AddDialer(d) - // listen on every address - for i, addr := range addrs { - err := s.setupListener(addr) + list, err := tpt.Listen(a) if err != nil { - if retErr.Errors == nil { - retErr.Errors = make([]error, len(addrs)) - } - retErr.Errors[i] = err - log.Debugf("Failed to listen on: %s - %s", addr, err) + errs[i] = err + continue } + + err = s.addListener(list) + if err != nil { + errs[i] = err + continue + } + succeeded++ } - if retErr.Errors != nil { - return retErr + for i, e := range errs { + if e != nil { + log.Warning("listen on %s failed: %s", addrs[i], errs[i]) + } } + if succeeded == 0 && len(addrs) > 0 { + return fmt.Errorf("failed to listen on any addresses: %s", errs) + } + return nil } -// Listen for new connections on the given multiaddr -func (s *Swarm) setupListener(maddr ma.Multiaddr) error { +func (s *Swarm) transportForAddr(a ma.Multiaddr) transport.Transport { + for _, t := range s.transports { + if t.Matches(a) { + return t + } + } + + return nil +} - // TODO rethink how this has to work. (jbenet) - // - // resolved, err := resolveUnspecifiedAddresses([]ma.Multiaddr{maddr}) - // if err != nil { - // return err - // } - // for _, a := range resolved { - // s.peers.AddAddr(s.local, a) - // } +func (s *Swarm) addListener(tptlist transport.Listener) error { sk := s.peers.PrivKey(s.local) if sk == nil { // may be fine for sk to be nil, just log a warning. log.Warning("Listener not given PrivateKey, so WILL NOT SECURE conns.") } - log.Debugf("Swarm Listening at %s", maddr) - list, err := conn.Listen(s.Context(), maddr, s.local, sk) + + list, err := conn.WrapTransportListener(s.Context(), tptlist, s.local, sk) if err != nil { return err } @@ -72,11 +85,15 @@ func (s *Swarm) setupListener(maddr ma.Multiaddr) error { list.SetAddrFilters(s.Filters) if cw, ok := list.(conn.ListenerConnWrapper); ok { - cw.SetConnWrapper(func(c manet.Conn) manet.Conn { + cw.SetConnWrapper(func(c transport.Conn) transport.Conn { return mconn.WrapConn(s.bwc, c) }) } + return s.addConnListener(list) +} + +func (s *Swarm) addConnListener(list conn.Listener) error { // AddListener to the peerstream Listener. this will begin accepting connections // and streams! sl, err := s.swarm.AddListener(list) @@ -85,6 +102,8 @@ func (s *Swarm) setupListener(maddr ma.Multiaddr) error { } log.Debugf("Swarm Listeners at %s", s.ListenAddresses()) + maddr := list.Multiaddr() + // signal to our notifiees on successful conn. s.notifyAll(func(n inet.Notifiee) { n.Listen((*Network)(s), maddr) @@ -107,7 +126,7 @@ func (s *Swarm) setupListener(maddr ma.Multiaddr) error { if !more { return } - log.Warningf("swarm listener accept error: %s", err) + log.Errorf("swarm listener accept error: %s", err) case <-ctx.Done(): return } @@ -138,5 +157,8 @@ func (s *Swarm) connHandler(c *ps.Conn) *Conn { return nil } + // if a peer dials us, remove from dial backoff. + s.backf.Clear(sc.RemotePeer()) + return sc } diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 863ef5e64f..91c4208370 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -5,12 +5,12 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" + metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - metrics "github.com/ipfs/go-libp2p/util/metrics" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + "github.com/jbenet/goprocess" + context "golang.org/x/net/context" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 59c4892b87..647c53c74e 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -5,7 +5,7 @@ import ( "testing" "time" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + context "golang.org/x/net/context" inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/p2p/test/util" ) diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 9bbf43e529..0d976e81d0 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" inet "github.com/ipfs/go-libp2p/p2p/net" peer "github.com/ipfs/go-libp2p/p2p/peer" diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index bcb188bc49..7965d27439 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,7 +3,7 @@ package swarm import ( inet "github.com/ipfs/go-libp2p/p2p/net" - ps "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-peerstream" + ps "github.com/jbenet/go-peerstream" ) // a Stream is a wrapper around a ps.Stream that exposes a way to get diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 1d0479f2cb..2c98e5e854 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -9,13 +9,13 @@ import ( "testing" "time" - testutil "github.com/ipfs/go-ipfs/util/testutil" + metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" peer "github.com/ipfs/go-libp2p/p2p/peer" - metrics "github.com/ipfs/go-libp2p/util/metrics" + testutil "util/testutil" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - context "github.com/ipfs/go-ipfs/Godeps/_workspace/src/golang.org/x/net/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func EchoStreamHandler(stream inet.Stream) { @@ -237,6 +237,15 @@ func TestSwarm(t *testing.T) { SubtestSwarm(t, swarms, msgs) } +func TestBasicSwarm(t *testing.T) { + // t.Skip("skipping for another test") + t.Parallel() + + msgs := 1 + swarms := 2 + SubtestSwarm(t, swarms, msgs) +} + func TestConnHandler(t *testing.T) { // t.Skip("skipping for another test") t.Parallel() From 0be91aa3ba54cbdf84040dca47358f20f60f05de Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 11 Nov 2015 16:15:47 -0800 Subject: [PATCH 0027/3965] extract from 0.4.0 --- p2p/net/nat/nat.go | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index a8bdb81be6..3c29d4ee08 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,14 +8,14 @@ import ( "sync" "time" - ma "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" - manet "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multiaddr-net" - - nat "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/fd/go-nat" - goprocess "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess" - periodic "github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/periodic" - notifier "github.com/ipfs/go-ipfs/thirdparty/notifier" - logging "github.com/ipfs/go-ipfs/vendor/go-log-v1.0.0" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + + nat "github.com/fd/go-nat" + goprocess "github.com/jbenet/goprocess" + periodic "github.com/jbenet/goprocess/periodic" + notifier "thirdparty/notifier" + logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ) var ( @@ -83,7 +83,7 @@ func (nat *NAT) Process() goprocess.Process { } // Notifier is an object that assists NAT in notifying listeners. -// It is implemented using github.com/ipfs/go-ipfs/thirdparty/notifier +// It is implemented using thirdparty/notifier type Notifier struct { n notifier.Notifier } From d78342b382d0edc7d47fd9a16af222257091d264 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 13 Nov 2015 16:18:52 -0800 Subject: [PATCH 0028/3965] remove dependency on key --- p2p/host/peerstore/queue/distance.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 6b721abb76..9600f87ce9 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -7,7 +7,6 @@ import ( ks "Qma4vHVBYKDiKS5VpvtLNJHHDbL7S6VRsvxxmBnBFfKP3k/go-keyspace" peer "github.com/ipfs/go-libp2p/p2p/peer" - key "github.com/whyrusleeping/go-key" ) // peerMetric tracks a peer and its distance to something else. @@ -93,9 +92,9 @@ func (pq *distancePQ) Dequeue() peer.ID { // NewXORDistancePQ returns a PeerQueue which maintains its peers sorted // in terms of their distances to each other in an XORKeySpace (i.e. using // XOR as a metric of distance). -func NewXORDistancePQ(fromKey key.Key) PeerQueue { +func NewXORDistancePQ(from string) PeerQueue { return &distancePQ{ - from: ks.XORKeySpace.Key([]byte(fromKey)), + from: ks.XORKeySpace.Key([]byte(from)), heap: peerMetricHeap{}, } } From 5070870e7bda75c083e1dd2ab83a6e969018858c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 11:56:01 -0800 Subject: [PATCH 0029/3965] more vendoring --- p2p/net/swarm/addr/addr.go | 4 ++-- p2p/net/swarm/addr/addr_test.go | 4 ++-- p2p/net/swarm/dial_test.go | 4 ++-- p2p/net/swarm/peers_test.go | 2 +- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm.go | 4 ++-- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_dial.go | 4 ++-- p2p/net/swarm/swarm_listen.go | 2 +- p2p/net/swarm/swarm_net.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 2 +- p2p/net/swarm/swarm_test.go | 2 +- 14 files changed, 19 insertions(+), 19 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index cec777a25e..cdb7058590 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -5,8 +5,8 @@ import ( logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + manet "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 49eec13511..2015b974cd 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,8 +3,8 @@ package addrutil import ( "testing" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + manet "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" ) func newMultiaddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index f121f97f05..77f827c333 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -13,8 +13,8 @@ import ( testutil "util/testutil" ci "util/testutil/ci" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + manet "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index b107fbfe83..cc8e14328f 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -5,7 +5,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 63e07ff8ac..c046ac2e45 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -9,7 +9,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" ci "util/testutil/ci" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 245b7204f7..d09b62ad50 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -16,14 +16,14 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "github.com/jbenet/go-multiaddr" + mafilter "QmVdADza4QFVAR9xqAxRQjt9vTZJ6UrVLgBstKua1Xg7he/multiaddr-filter" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ps "github.com/jbenet/go-peerstream" pst "github.com/jbenet/go-stream-muxer" psmss "github.com/jbenet/go-stream-muxer/multistream" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" prom "github.com/prometheus/client_golang/prometheus" - mafilter "github.com/whyrusleeping/multiaddr-filter" context "golang.org/x/net/context" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 039683bfb9..87ee37bf17 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -4,7 +4,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index b1f167ae49..78685a4d38 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -8,7 +8,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "util/testutil" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 840a9b465c..c97a47dfcb 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,7 +8,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ps "github.com/jbenet/go-peerstream" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 4790eb0738..47caa69d65 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -8,13 +8,13 @@ import ( "sync" "time" - "github.com/jbenet/go-multiaddr-net" + "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" lgbl "util/eventlog/loggables" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index f3da2eeea7..394d5c2179 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -9,7 +9,7 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" lgbl "util/eventlog/loggables" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ps "github.com/jbenet/go-peerstream" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 91c4208370..3c5e69225b 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -8,7 +8,7 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" "github.com/jbenet/goprocess" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 0d976e81d0..3d2ef86d7b 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" context "golang.org/x/net/context" inet "github.com/ipfs/go-libp2p/p2p/net" diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 2c98e5e854..c6e4a981d1 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -14,7 +14,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "util/testutil" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" context "golang.org/x/net/context" ) From dc2ec7c327224a453e451b2115d1bd83e64c86c1 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 11:56:01 -0800 Subject: [PATCH 0030/3965] more vendoring --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/peer.go | 6 +++--- p2p/host/peerstore/peer_test.go | 2 +- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 5 ++--- 8 files changed, 11 insertions(+), 12 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index a172373d8d..f6ea1c1835 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 46140641ca..cc7fdd54b3 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 87b8206eee..1775069f7e 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -4,7 +4,7 @@ import ( "sync" "time" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 3e19530539..d22198284d 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "github.com/jbenet/go-multiaddr" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ) func IDS(t *testing.T, ids string) ID { diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 549b2093a6..1b59a821aa 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -7,9 +7,9 @@ import ( "fmt" "strings" - b58 "github.com/jbenet/go-base58" - ma "github.com/jbenet/go-multiaddr" - mh "github.com/jbenet/go-multihash" + b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + mh "QmdeauTdyf38KDQB4Cc4CurPWRRb5pej27NCXPA7kbPTJy/go-multihash" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ic "github.com/ipfs/go-libp2p/p2p/crypto" diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 1d4ca2595b..502d8aba06 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -11,7 +11,7 @@ import ( u "util" tu "util/testutil" - b58 "github.com/jbenet/go-base58" + b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" ) var gen1 keyset // generated diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 9f3a98d25d..8c2ae05a89 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -7,9 +7,9 @@ import ( ic "github.com/ipfs/go-libp2p/p2p/crypto" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" ds "github.com/jbenet/go-datastore" dssync "github.com/jbenet/go-datastore/sync" - ma "github.com/jbenet/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index d85f84b7c2..9d8a42718a 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,7 +7,6 @@ import ( "time" peer "github.com/ipfs/go-libp2p/p2p/peer" - key "github.com/whyrusleeping/go-key" u "util" context "golang.org/x/net/context" @@ -28,7 +27,7 @@ func TestQueue(t *testing.T) { // [78 135 26 216 178 181 224 181 234 117 2 248 152 115 255 103 244 34 4 152 193 88 9 225 8 127 216 158 226 8 236 246] // [125 135 124 6 226 160 101 94 192 57 39 12 18 79 121 140 190 154 147 55 44 83 101 151 63 255 94 179 51 203 241 51] - pq := NewXORDistancePQ(key.Key("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a31")) + pq := NewXORDistancePQ("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a31") pq.Enqueue(p3) pq.Enqueue(p1) pq.Enqueue(p2) @@ -82,7 +81,7 @@ func TestSyncQueue(t *testing.T) { } ctx := context.Background() - pq := NewXORDistancePQ(key.Key("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a31")) + pq := NewXORDistancePQ("11140beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a31") cq := NewChanQueue(ctx, pq) wg := sync.WaitGroup{} From fa375b9634bbc88356e5b6931765160416e14bdb Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 11:56:01 -0800 Subject: [PATCH 0031/3965] more vendoring --- p2p/net/nat/nat.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 3c29d4ee08..fe3a52114e 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,14 +8,14 @@ import ( "sync" "time" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" + ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + manet "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" + logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" nat "github.com/fd/go-nat" goprocess "github.com/jbenet/goprocess" periodic "github.com/jbenet/goprocess/periodic" notifier "thirdparty/notifier" - logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ) var ( From 21e5c03e6894b48d1b42951f7735ae78250506b6 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 18:28:35 -0800 Subject: [PATCH 0032/3965] fixes for sha3 --- p2p/net/swarm/addr/addr.go | 4 ++-- p2p/net/swarm/addr/addr_test.go | 4 ++-- p2p/net/swarm/dial_test.go | 4 ++-- p2p/net/swarm/peers_test.go | 2 +- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm.go | 4 ++-- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_dial.go | 4 ++-- p2p/net/swarm/swarm_listen.go | 2 +- p2p/net/swarm/swarm_net.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 2 +- p2p/net/swarm/swarm_test.go | 2 +- 14 files changed, 19 insertions(+), 19 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index cdb7058590..c0c4851c26 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -5,8 +5,8 @@ import ( logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" - manet "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" + manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 2015b974cd..21d645724c 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,8 +3,8 @@ package addrutil import ( "testing" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" - manet "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" + manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) func newMultiaddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 77f827c333..632ceb7af8 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -13,8 +13,8 @@ import ( testutil "util/testutil" ci "util/testutil/ci" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" - manet "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" + manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index cc8e14328f..29099de46f 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -5,7 +5,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index c046ac2e45..7adc022f8f 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -9,7 +9,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" ci "util/testutil/ci" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index d09b62ad50..2fcd275675 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -16,8 +16,8 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" - mafilter "QmVdADza4QFVAR9xqAxRQjt9vTZJ6UrVLgBstKua1Xg7he/multiaddr-filter" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + mafilter "QmYWqTn1i8yv9QRDzGPJ2yRudKzYCaC5Aqasbm8vwaG92E/multiaddr-filter" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ps "github.com/jbenet/go-peerstream" pst "github.com/jbenet/go-stream-muxer" psmss "github.com/jbenet/go-stream-muxer/multistream" diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 87ee37bf17..6149eb66e3 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -4,7 +4,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 78685a4d38..0b0d6fffa5 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -8,7 +8,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "util/testutil" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index c97a47dfcb..be95a4282b 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,7 +8,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ps "github.com/jbenet/go-peerstream" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 47caa69d65..2bedebc2c8 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -8,13 +8,13 @@ import ( "sync" "time" - "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" + "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" lgbl "util/eventlog/loggables" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 394d5c2179..ee8ee91366 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -9,7 +9,7 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" lgbl "util/eventlog/loggables" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ps "github.com/jbenet/go-peerstream" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 3c5e69225b..972afc1d02 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -8,7 +8,7 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" "github.com/jbenet/goprocess" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 3d2ef86d7b..2ba852d188 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" context "golang.org/x/net/context" inet "github.com/ipfs/go-libp2p/p2p/net" diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index c6e4a981d1..f9ba35ed04 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -14,7 +14,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "util/testutil" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" context "golang.org/x/net/context" ) From 35caf5995d1eb493a6ec2f74667ae3b693a8ef47 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 18:28:35 -0800 Subject: [PATCH 0033/3965] fixes for sha3 --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/peer.go | 2 +- p2p/host/peerstore/peerstore.go | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index f6ea1c1835..babfea5598 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index cc7fdd54b3..81b3a0518e 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 1775069f7e..d3781ced81 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -4,7 +4,7 @@ import ( "sync" "time" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index d22198284d..30799798ec 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) func IDS(t *testing.T, ids string) ID { diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 1b59a821aa..8958e2b299 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -8,7 +8,7 @@ import ( "strings" b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" mh "QmdeauTdyf38KDQB4Cc4CurPWRRb5pej27NCXPA7kbPTJy/go-multihash" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 8c2ae05a89..dea66f3a5a 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -7,7 +7,7 @@ import ( ic "github.com/ipfs/go-libp2p/p2p/crypto" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ds "github.com/jbenet/go-datastore" dssync "github.com/jbenet/go-datastore/sync" ) From aa5c8a270dc7cb463cca5e035213568eec545b99 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 18:28:35 -0800 Subject: [PATCH 0034/3965] fixes for sha3 --- p2p/net/nat/nat.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index fe3a52114e..3258417220 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,8 +8,8 @@ import ( "sync" "time" - ma "QmaA6aDzeHjZiuqBtgYRz8ZXb1qMCoyMHgyDjBEYQniUKF/go-multiaddr" - manet "QmanZCL6SXRfafiUEMCBLq2QR171uQSdXQ8YAdHXLd8Cwr/go-multiaddr-net" + manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" nat "github.com/fd/go-nat" From 7d2dd014c8f44d012c78722b328e578d13c5f0ba Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 18:42:27 -0800 Subject: [PATCH 0035/3965] vendor in notifier --- p2p/net/swarm/addr/addr.go | 2 +- p2p/net/swarm/dial_test.go | 2 +- p2p/net/swarm/peers_test.go | 2 +- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm.go | 6 +++--- p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_dial.go | 2 +- p2p/net/swarm/swarm_listen.go | 2 +- p2p/net/swarm/swarm_net.go | 4 ++-- p2p/net/swarm/swarm_net_test.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 2 +- p2p/net/swarm/swarm_test.go | 2 +- 13 files changed, 16 insertions(+), 16 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index c0c4851c26..857b0f7289 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -6,8 +6,8 @@ import ( logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - context "golang.org/x/net/context" ) var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 632ceb7af8..027e7945cb 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -14,8 +14,8 @@ import ( ci "util/testutil/ci" manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - context "golang.org/x/net/context" ) func acceptAndHang(l net.Listener) { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 29099de46f..0068b37901 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -5,8 +5,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - context "golang.org/x/net/context" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 7adc022f8f..549dad4ff8 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -9,8 +9,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" ci "util/testutil/ci" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - context "golang.org/x/net/context" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 2fcd275675..e5424802f7 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -16,15 +16,15 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" + "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" + goprocessctx "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess/context" mafilter "QmYWqTn1i8yv9QRDzGPJ2yRudKzYCaC5Aqasbm8vwaG92E/multiaddr-filter" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ps "github.com/jbenet/go-peerstream" pst "github.com/jbenet/go-stream-muxer" psmss "github.com/jbenet/go-stream-muxer/multistream" - "github.com/jbenet/goprocess" - goprocessctx "github.com/jbenet/goprocess/context" prom "github.com/prometheus/client_golang/prometheus" - context "golang.org/x/net/context" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ) diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 0b0d6fffa5..ec5c6576fa 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -8,8 +8,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "util/testutil" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - context "golang.org/x/net/context" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index be95a4282b..7b9c920fae 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,9 +8,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ps "github.com/jbenet/go-peerstream" - context "golang.org/x/net/context" ) // a Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 2bedebc2c8..3c7aa01d31 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -14,8 +14,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" lgbl "util/eventlog/loggables" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - context "golang.org/x/net/context" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index ee8ee91366..8355815869 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -9,9 +9,9 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" lgbl "util/eventlog/loggables" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ps "github.com/jbenet/go-peerstream" - context "golang.org/x/net/context" ) // Open listeners and reuse-dialers for the given addresses diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 972afc1d02..36416fa394 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -8,9 +8,9 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" + "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - "github.com/jbenet/goprocess" - context "golang.org/x/net/context" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 647c53c74e..2db22fbb35 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -5,7 +5,7 @@ import ( "testing" "time" - context "golang.org/x/net/context" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/p2p/test/util" ) diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 2ba852d188..1180d641ff 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - context "golang.org/x/net/context" inet "github.com/ipfs/go-libp2p/p2p/net" peer "github.com/ipfs/go-libp2p/p2p/peer" diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index f9ba35ed04..27331ade57 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -14,8 +14,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "util/testutil" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - context "golang.org/x/net/context" ) func EchoStreamHandler(stream inet.Stream) { From 18755d988d50d4186acff21e3fbbff027fd1f029 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 18:42:27 -0800 Subject: [PATCH 0036/3965] vendor in notifier --- p2p/host/peerstore/queue/queue_test.go | 2 +- p2p/host/peerstore/queue/sync.go | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 9d8a42718a..a094dd243f 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -9,7 +9,7 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" u "util" - context "golang.org/x/net/context" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 2a88683e9c..6d834e8617 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -1,9 +1,9 @@ package queue import ( - context "golang.org/x/net/context" - peer "github.com/ipfs/go-libp2p/p2p/peer" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" + context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + peer "github.com/ipfs/go-libp2p/p2p/peer" ) var log = logging.Logger("peerqueue") From 2c7dfee2f9e7696b29e36da1f99fd0c4ee934678 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 18:42:27 -0800 Subject: [PATCH 0037/3965] vendor in notifier --- p2p/net/nat/nat.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 3258417220..8e7fb95ea7 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -11,11 +11,11 @@ import ( manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + goprocess "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" + periodic "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess/periodic" + notifier "QmUtEiB6DmXs7eLJiwS9YFyTAtptqzaWutxCsjHy7UKEgo/go-notifier" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" nat "github.com/fd/go-nat" - goprocess "github.com/jbenet/goprocess" - periodic "github.com/jbenet/goprocess/periodic" - notifier "thirdparty/notifier" ) var ( From a82cbe7b2ff22d2f93e4b616d1de197bb539c3f4 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 19:59:59 -0800 Subject: [PATCH 0038/3965] move testutil up --- p2p/net/swarm/dial_test.go | 4 ++-- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_test.go | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 027e7945cb..128ddb5827 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -10,8 +10,8 @@ import ( addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - testutil "util/testutil" - ci "util/testutil/ci" + testutil "github.com/ipfs/go-libp2p/testutil" + ci "github.com/ipfs/go-libp2p/testutil/ci" manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 549dad4ff8..ec988bcd6c 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -7,7 +7,7 @@ import ( "time" peer "github.com/ipfs/go-libp2p/p2p/peer" - ci "util/testutil/ci" + ci "github.com/ipfs/go-libp2p/testutil/ci" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index ec5c6576fa..b397a07f62 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -6,7 +6,7 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - testutil "util/testutil" + testutil "github.com/ipfs/go-libp2p/testutil" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 27331ade57..670ea26592 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -12,7 +12,7 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" peer "github.com/ipfs/go-libp2p/p2p/peer" - testutil "util/testutil" + testutil "github.com/ipfs/go-libp2p/testutil" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" From e27e65720b7eb0dd41aa75a0a5864f7533f77563 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 19:59:59 -0800 Subject: [PATCH 0039/3965] move testutil up --- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peer_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 4343c0875e..84ae8ed6d7 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,7 +7,7 @@ import ( "time" peer "github.com/ipfs/go-libp2p/p2p/peer" - testutil "util/testutil" + testutil "github.com/ipfs/go-libp2p/testutil" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 502d8aba06..915a0eecee 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -9,7 +9,7 @@ import ( ic "github.com/ipfs/go-libp2p/p2p/crypto" . "github.com/ipfs/go-libp2p/p2p/peer" u "util" - tu "util/testutil" + tu "github.com/ipfs/go-libp2p/testutil" b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" ) From 8cf47173c898399496f4af16b42708428d598ed0 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 21:04:44 -0800 Subject: [PATCH 0040/3965] remove multiple multihash deps --- p2p/net/swarm/addr/addr.go | 2 +- p2p/net/swarm/addr/addr_test.go | 2 +- p2p/net/swarm/dial_test.go | 2 +- p2p/net/swarm/swarm.go | 2 +- p2p/net/swarm/swarm_dial.go | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 857b0f7289..00b8ed1c29 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -5,7 +5,7 @@ import ( logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" - manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + manet "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 21d645724c..8a0bc9c35a 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,7 +3,7 @@ package addrutil import ( "testing" - manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + manet "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 128ddb5827..f915808b75 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -13,7 +13,7 @@ import ( testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" - manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + manet "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index e5424802f7..b2b6b83835 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -18,7 +18,7 @@ import ( "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" goprocessctx "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess/context" - mafilter "QmYWqTn1i8yv9QRDzGPJ2yRudKzYCaC5Aqasbm8vwaG92E/multiaddr-filter" + mafilter "QmXE4GFk66B5Ts362YWZosjiZLpP4QDmBTBadRZFffZ58U/multiaddr-filter" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ps "github.com/jbenet/go-peerstream" diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 3c7aa01d31..df3b89d821 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -8,7 +8,7 @@ import ( "sync" "time" - "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" From 42b661a7d22bf708d9dcb3c8ca57fb089e908838 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 21:04:44 -0800 Subject: [PATCH 0041/3965] remove multiple multihash deps --- p2p/host/peerstore/peer.go | 2 +- p2p/host/peerstore/peer_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 8958e2b299..3b602f8c22 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -9,7 +9,7 @@ import ( b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - mh "QmdeauTdyf38KDQB4Cc4CurPWRRb5pej27NCXPA7kbPTJy/go-multihash" + mh "QmdsKjp5fcCT8PZ8JBMcdFsCbbmKwSLCU5xXbsnwb5DMxy/go-multihash" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ic "github.com/ipfs/go-libp2p/p2p/crypto" diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 915a0eecee..4a92ecf602 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -8,8 +8,8 @@ import ( ic "github.com/ipfs/go-libp2p/p2p/crypto" . "github.com/ipfs/go-libp2p/p2p/peer" - u "util" tu "github.com/ipfs/go-libp2p/testutil" + u "util" b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" ) From ac1dafff64a8f2bd718be0b1fe9184fbda9df098 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 21:04:44 -0800 Subject: [PATCH 0042/3965] remove multiple multihash deps --- p2p/net/nat/nat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 8e7fb95ea7..3583a1d3f9 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,7 +8,7 @@ import ( "sync" "time" - manet "QmU5s159q8cZuM1f9Vqti4LHu6y8zyVc5dxv2py81sdp6Q/go-multiaddr-net" + manet "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" goprocess "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" From 46df570e0eb2151b54ff332a3fefa318fa9fc5da Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 21:15:23 -0800 Subject: [PATCH 0043/3965] vendor go-ipfs-util --- p2p/net/swarm/swarm_dial.go | 2 +- p2p/net/swarm/swarm_listen.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index df3b89d821..11b3f20463 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -12,7 +12,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - lgbl "util/eventlog/loggables" + lgbl "github.com/ipfs/go-libp2p/loggables" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 8355815869..4dd95a579d 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -7,7 +7,7 @@ import ( inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" transport "github.com/ipfs/go-libp2p/p2p/net/transport" - lgbl "util/eventlog/loggables" + lgbl "github.com/ipfs/go-libp2p/loggables" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" From 53ce4e3a9cef7c56c290f76f3c885ed505819aa6 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 Nov 2015 21:15:23 -0800 Subject: [PATCH 0044/3965] vendor go-ipfs-util --- p2p/host/peerstore/peer.go | 2 +- p2p/host/peerstore/peer_test.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 3b602f8c22..4b907d3108 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -12,8 +12,8 @@ import ( mh "QmdsKjp5fcCT8PZ8JBMcdFsCbbmKwSLCU5xXbsnwb5DMxy/go-multihash" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" + u "Qmah3kfjwhVxBM4qGnrqJTqGzrF8svwByyhExPipA2U6LE/go-ipfs-util" ic "github.com/ipfs/go-libp2p/p2p/crypto" - u "util" ) var log = logging.Logger("peer") diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 4a92ecf602..7472db169f 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -9,7 +9,7 @@ import ( ic "github.com/ipfs/go-libp2p/p2p/crypto" . "github.com/ipfs/go-libp2p/p2p/peer" tu "github.com/ipfs/go-libp2p/testutil" - u "util" + u "Qmah3kfjwhVxBM4qGnrqJTqGzrF8svwByyhExPipA2U6LE/go-ipfs-util" b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" ) diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index a094dd243f..d42ebca50e 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,7 +7,7 @@ import ( "time" peer "github.com/ipfs/go-libp2p/p2p/peer" - u "util" + u "Qmah3kfjwhVxBM4qGnrqJTqGzrF8svwByyhExPipA2U6LE/go-ipfs-util" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) From 66d6d96cebf0bcdbb27a8190fd6b59b23e56a63b Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 16 Nov 2015 16:41:23 -0800 Subject: [PATCH 0045/3965] update multiaddr-filter --- p2p/net/swarm/swarm.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index b2b6b83835..5fc02a1a16 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -18,7 +18,7 @@ import ( "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" goprocessctx "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess/context" - mafilter "QmXE4GFk66B5Ts362YWZosjiZLpP4QDmBTBadRZFffZ58U/multiaddr-filter" + mafilter "QmYhewVqJhkgEsm3AYVUbpT14q2P9V4Xb7np9JXKua6y7A/multiaddr-filter" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" ps "github.com/jbenet/go-peerstream" From f4661d9a5bb7a4d077f23c2c41c99d2a26421478 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 16 Nov 2015 16:47:49 -0800 Subject: [PATCH 0046/3965] Vendor in go-peerstream --- p2p/net/swarm/swarm.go | 6 +++--- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_listen.go | 4 ++-- p2p/net/swarm/swarm_stream.go | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 5fc02a1a16..a6dfa61639 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -16,14 +16,14 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" + pst "QmPxuHs2NQjz16gnvndgkzHkm5PjtqbB5rwoSpLusBkQ7Q/go-stream-muxer" + psmss "QmPxuHs2NQjz16gnvndgkzHkm5PjtqbB5rwoSpLusBkQ7Q/go-stream-muxer/multistream" "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" goprocessctx "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess/context" + ps "QmTgxFwS1nDK126fH5XPnLFcxcDFsxKbPPnCBwyRWNAjDX/go-peerstream" mafilter "QmYhewVqJhkgEsm3AYVUbpT14q2P9V4Xb7np9JXKua6y7A/multiaddr-filter" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - ps "github.com/jbenet/go-peerstream" - pst "github.com/jbenet/go-stream-muxer" - psmss "github.com/jbenet/go-stream-muxer/multistream" prom "github.com/prometheus/client_golang/prometheus" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 7b9c920fae..e07e82f41f 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,9 +8,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" + ps "QmTgxFwS1nDK126fH5XPnLFcxcDFsxKbPPnCBwyRWNAjDX/go-peerstream" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - ps "github.com/jbenet/go-peerstream" ) // a Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 4dd95a579d..b986876825 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -3,15 +3,15 @@ package swarm import ( "fmt" + lgbl "github.com/ipfs/go-libp2p/loggables" mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" transport "github.com/ipfs/go-libp2p/p2p/net/transport" - lgbl "github.com/ipfs/go-libp2p/loggables" + ps "QmTgxFwS1nDK126fH5XPnLFcxcDFsxKbPPnCBwyRWNAjDX/go-peerstream" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - ps "github.com/jbenet/go-peerstream" ) // Open listeners and reuse-dialers for the given addresses diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 7965d27439..d52962ae83 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,7 +3,7 @@ package swarm import ( inet "github.com/ipfs/go-libp2p/p2p/net" - ps "github.com/jbenet/go-peerstream" + ps "QmTgxFwS1nDK126fH5XPnLFcxcDFsxKbPPnCBwyRWNAjDX/go-peerstream" ) // a Stream is a wrapper around a ps.Stream that exposes a way to get From 9a7fdeb38a4a03c75202d6b044984cf28c5a75f2 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 16 Nov 2015 17:25:35 -0800 Subject: [PATCH 0047/3965] remove prometheus dep --- p2p/net/swarm/swarm.go | 31 ------------------------------- 1 file changed, 31 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index a6dfa61639..3a37d30b88 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -24,7 +24,6 @@ import ( mafilter "QmYhewVqJhkgEsm3AYVUbpT14q2P9V4Xb7np9JXKua6y7A/multiaddr-filter" context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - prom "github.com/prometheus/client_golang/prometheus" logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" ) @@ -33,13 +32,6 @@ var log = logging.Logger("swarm2") var PSTransport pst.Transport -var peersTotal = prom.NewGaugeVec(prom.GaugeOpts{ - Namespace: "ipfs", - Subsystem: "p2p", - Name: "peers_total", - Help: "Number of connected peers", -}, []string{"peer_id"}) - func init() { PSTransport = psmss.NewTransport() } @@ -109,10 +101,6 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, s.proc = goprocessctx.WithContextAndTeardown(ctx, s.teardown) s.SetConnHandler(nil) // make sure to setup our own conn handler. - // setup swarm metrics - prom.MustRegisterOrGet(peersTotal) - s.Notify((*metricsNotifiee)(s)) - err = s.setupInterfaces(listenAddrs) if err != nil { return nil, err @@ -321,22 +309,3 @@ func (n *ps2netNotifee) OpenedStream(s *ps.Stream) { func (n *ps2netNotifee) ClosedStream(s *ps.Stream) { n.not.ClosedStream(n.net, inet.Stream((*Stream)(s))) } - -type metricsNotifiee Swarm - -func (nn *metricsNotifiee) Connected(n inet.Network, v inet.Conn) { - peersTotalGauge(n.LocalPeer()).Set(float64(len(n.Conns()))) -} - -func (nn *metricsNotifiee) Disconnected(n inet.Network, v inet.Conn) { - peersTotalGauge(n.LocalPeer()).Set(float64(len(n.Conns()))) -} - -func (nn *metricsNotifiee) OpenedStream(n inet.Network, v inet.Stream) {} -func (nn *metricsNotifiee) ClosedStream(n inet.Network, v inet.Stream) {} -func (nn *metricsNotifiee) Listen(n inet.Network, a ma.Multiaddr) {} -func (nn *metricsNotifiee) ListenClose(n inet.Network, a ma.Multiaddr) {} - -func peersTotalGauge(id peer.ID) prom.Gauge { - return peersTotal.With(prom.Labels{"peer_id": id.Pretty()}) -} From a120be4ee50459697721c086d27edb478a248d4a Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 18 Nov 2015 11:47:51 -0800 Subject: [PATCH 0048/3965] migrate to gx namespace --- p2p/net/swarm/addr/addr.go | 8 ++++---- p2p/net/swarm/addr/addr_test.go | 4 ++-- p2p/net/swarm/dial_test.go | 6 +++--- p2p/net/swarm/peers_test.go | 4 ++-- p2p/net/swarm/simul_test.go | 4 ++-- p2p/net/swarm/swarm.go | 20 ++++++++++---------- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 4 ++-- p2p/net/swarm/swarm_conn.go | 6 +++--- p2p/net/swarm/swarm_dial.go | 8 ++++---- p2p/net/swarm/swarm_listen.go | 6 +++--- p2p/net/swarm/swarm_net.go | 6 +++--- p2p/net/swarm/swarm_net_test.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 4 ++-- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 4 ++-- 16 files changed, 45 insertions(+), 45 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 00b8ed1c29..c74147527b 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,11 +3,11 @@ package addrutil import ( "fmt" - logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" + logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" - manet "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + manet "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 8a0bc9c35a..120195a122 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,8 +3,8 @@ package addrutil import ( "testing" - manet "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + manet "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" ) func newMultiaddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index f915808b75..0df87fd293 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -13,9 +13,9 @@ import ( testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" - manet "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + manet "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) func acceptAndHang(l net.Listener) { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 0068b37901..ef9c22ff60 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -5,8 +5,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index ec988bcd6c..408b39b244 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -9,8 +9,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" ci "github.com/ipfs/go-libp2p/testutil/ci" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 3a37d30b88..a6ddae3ccd 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -16,16 +16,16 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" - pst "QmPxuHs2NQjz16gnvndgkzHkm5PjtqbB5rwoSpLusBkQ7Q/go-stream-muxer" - psmss "QmPxuHs2NQjz16gnvndgkzHkm5PjtqbB5rwoSpLusBkQ7Q/go-stream-muxer/multistream" - "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" - goprocessctx "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess/context" - ps "QmTgxFwS1nDK126fH5XPnLFcxcDFsxKbPPnCBwyRWNAjDX/go-peerstream" - mafilter "QmYhewVqJhkgEsm3AYVUbpT14q2P9V4Xb7np9JXKua6y7A/multiaddr-filter" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - - logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" + ps "gx/QmQMkNFMuh1UJbdaggASZpLtCroTnAYcWNenYL5hrzLJrT/go-peerstream" + pst "gx/QmRmT6MSnfhRDW1PTUGSd3z4fqXK48GUequQAZzeT4c5iC/go-stream-muxer" + psmss "gx/QmRmT6MSnfhRDW1PTUGSd3z4fqXK48GUequQAZzeT4c5iC/go-stream-muxer/multistream" + mafilter "gx/QmVCmuhgDFer5MW5737Z8GtBEGpUyEkFnLUv4ASDWnLZdC/multiaddr-filter" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess" + goprocessctx "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess/context" + + logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" ) var log = logging.Logger("swarm2") diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 6149eb66e3..81a77bb8e0 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -4,7 +4,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index b397a07f62..7383beeabe 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -8,8 +8,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index e07e82f41f..e7d1ec91dc 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,9 +8,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" - ps "QmTgxFwS1nDK126fH5XPnLFcxcDFsxKbPPnCBwyRWNAjDX/go-peerstream" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ps "gx/QmQMkNFMuh1UJbdaggASZpLtCroTnAYcWNenYL5hrzLJrT/go-peerstream" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) // a Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 11b3f20463..fa4fce1e47 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -8,14 +8,14 @@ import ( "sync" "time" - "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" + lgbl "github.com/ipfs/go-libp2p/loggables" conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - lgbl "github.com/ipfs/go-libp2p/loggables" + "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index b986876825..bd1810b462 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -9,9 +9,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" transport "github.com/ipfs/go-libp2p/p2p/net/transport" - ps "QmTgxFwS1nDK126fH5XPnLFcxcDFsxKbPPnCBwyRWNAjDX/go-peerstream" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ps "gx/QmQMkNFMuh1UJbdaggASZpLtCroTnAYcWNenYL5hrzLJrT/go-peerstream" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) // Open listeners and reuse-dialers for the given addresses diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 36416fa394..d8724a49f9 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -8,9 +8,9 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 2db22fbb35..c40b89cc63 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -5,7 +5,7 @@ import ( "testing" "time" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/p2p/test/util" ) diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 1180d641ff..9b59bb24d9 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" inet "github.com/ipfs/go-libp2p/p2p/net" peer "github.com/ipfs/go-libp2p/p2p/peer" diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index d52962ae83..f4ee314958 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,7 +3,7 @@ package swarm import ( inet "github.com/ipfs/go-libp2p/p2p/net" - ps "QmTgxFwS1nDK126fH5XPnLFcxcDFsxKbPPnCBwyRWNAjDX/go-peerstream" + ps "gx/QmQMkNFMuh1UJbdaggASZpLtCroTnAYcWNenYL5hrzLJrT/go-peerstream" ) // a Stream is a wrapper around a ps.Stream that exposes a way to get diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 670ea26592..c8a1ce548e 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -14,8 +14,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) func EchoStreamHandler(stream inet.Stream) { From 08dc34f725456a277cd5b0977eea5e1acc01b8ca Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 18 Nov 2015 11:47:51 -0800 Subject: [PATCH 0049/3965] migrate to gx namespace --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/addr_manager_test.go | 4 +-- p2p/host/peerstore/peer.go | 12 ++++----- p2p/host/peerstore/peer_test.go | 4 +-- p2p/host/peerstore/peerstore.go | 34 +++++++++++++++++------- p2p/host/peerstore/queue/distance.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 4 +-- p2p/host/peerstore/queue/sync.go | 4 +-- 10 files changed, 43 insertions(+), 27 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index babfea5598..ae9073ace4 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 81b3a0518e..173fbca0fd 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index d3781ced81..1129b0637c 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -4,7 +4,7 @@ import ( "sync" "time" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 30799798ec..59bcf19047 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,13 +4,13 @@ import ( "testing" "time" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" ) func IDS(t *testing.T, ids string) ID { id, err := IDB58Decode(ids) if err != nil { - t.Fatal(err) + t.Fatalf("id %q is bad: %s", ids, err) } return id } diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 4b907d3108..0e7c009eef 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -7,13 +7,13 @@ import ( "fmt" "strings" - b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - mh "QmdsKjp5fcCT8PZ8JBMcdFsCbbmKwSLCU5xXbsnwb5DMxy/go-multihash" + b58 "gx/QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + mh "gx/Qma7dqy7ZVH4tkNJdC9TRrA82Uz5fQfbbwuvmNVVc17r7a/go-multihash" - logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" - u "Qmah3kfjwhVxBM4qGnrqJTqGzrF8svwByyhExPipA2U6LE/go-ipfs-util" ic "github.com/ipfs/go-libp2p/p2p/crypto" + u "gx/QmQA79FfVsUnGkH3TgKDqcDkupfjqLSJ6EYwDuDDZK8nhD/go-ipfs-util" + logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" ) var log = logging.Logger("peer") @@ -43,7 +43,7 @@ func (id ID) String() string { //All sha256 nodes start with Qm //We can skip the Qm to make the peer.ID more useful - if strings.HasPrefix(pid, "Qm") { + if strings.HasPrefix(pid, "gx/Qm") { pid = pid[2:] } diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 7472db169f..3ad9c23a22 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -9,9 +9,9 @@ import ( ic "github.com/ipfs/go-libp2p/p2p/crypto" . "github.com/ipfs/go-libp2p/p2p/peer" tu "github.com/ipfs/go-libp2p/testutil" - u "Qmah3kfjwhVxBM4qGnrqJTqGzrF8svwByyhExPipA2U6LE/go-ipfs-util" + u "gx/QmQA79FfVsUnGkH3TgKDqcDkupfjqLSJ6EYwDuDDZK8nhD/go-ipfs-util" - b58 "QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" + b58 "gx/QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" ) var gen1 keyset // generated diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index dea66f3a5a..ef86182653 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -7,9 +7,9 @@ import ( ic "github.com/ipfs/go-libp2p/p2p/crypto" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - ds "github.com/jbenet/go-datastore" - dssync "github.com/jbenet/go-datastore/sync" + //ds "github.com/jbenet/go-datastore" + //dssync "github.com/jbenet/go-datastore/sync" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" ) const ( @@ -153,7 +153,10 @@ type peerstore struct { AddrManager // store other data, like versions - ds ds.ThreadSafeDatastore + //ds ds.ThreadSafeDatastore + // TODO: use a datastore for this + ds map[string]interface{} + dslock sync.Mutex } // NewPeerstore creates a threadsafe collection of peers. @@ -162,18 +165,31 @@ func NewPeerstore() Peerstore { keybook: *newKeybook(), metrics: *(NewMetrics()).(*metrics), AddrManager: AddrManager{}, - ds: dssync.MutexWrap(ds.NewMapDatastore()), + //ds: dssync.MutexWrap(ds.NewMapDatastore()), + ds: make(map[string]interface{}), } } func (ps *peerstore) Put(p ID, key string, val interface{}) error { - dsk := ds.NewKey(string(p) + "/" + key) - return ps.ds.Put(dsk, val) + //dsk := ds.NewKey(string(p) + "/" + key) + //return ps.ds.Put(dsk, val) + ps.dslock.Lock() + defer ps.dslock.Unlock() + ps.ds[string(p)+"/"+key] = val + return nil } func (ps *peerstore) Get(p ID, key string) (interface{}, error) { - dsk := ds.NewKey(string(p) + "/" + key) - return ps.ds.Get(dsk) + //dsk := ds.NewKey(string(p) + "/" + key) + //return ps.ds.Get(dsk) + + ps.dslock.Lock() + defer ps.dslock.Unlock() + i, ok := ps.ds[string(p)+"/"+key] + if !ok { + return nil, errors.New("item not found") + } + return i, nil } func (ps *peerstore) Peers() []ID { diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 9600f87ce9..beb7605b33 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,8 +5,8 @@ import ( "math/big" "sync" - ks "Qma4vHVBYKDiKS5VpvtLNJHHDbL7S6VRsvxxmBnBFfKP3k/go-keyspace" peer "github.com/ipfs/go-libp2p/p2p/peer" + ks "gx/QmThbzo9oBE7FAVmgjp3xYv5Yra81uB7KXbrTei47gWT6o/go-keyspace" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index d42ebca50e..ca449fe698 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,9 +7,9 @@ import ( "time" peer "github.com/ipfs/go-libp2p/p2p/peer" - u "Qmah3kfjwhVxBM4qGnrqJTqGzrF8svwByyhExPipA2U6LE/go-ipfs-util" + u "gx/QmQA79FfVsUnGkH3TgKDqcDkupfjqLSJ6EYwDuDDZK8nhD/go-ipfs-util" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 6d834e8617..ccece46af4 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -1,9 +1,9 @@ package queue import ( - logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" - context "QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" peer "github.com/ipfs/go-libp2p/p2p/peer" + context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" ) var log = logging.Logger("peerqueue") From f2b4ef28344579a2da7782d918e75723806b8c31 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 18 Nov 2015 11:47:51 -0800 Subject: [PATCH 0050/3965] migrate to gx namespace --- p2p/net/nat/nat.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 3583a1d3f9..c549527c57 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,14 +8,14 @@ import ( "sync" "time" - manet "QmRCPT5WRph8aWXmaT2Rfn6ac98YRUUJnNURpD3hNAWp4f/go-multiaddr-net" - ma "QmbWxL1aXQhBjc1XGjGF1f2KGBMCBYSuT2ThA8YXnXJK83/go-multiaddr" - - goprocess "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess" - periodic "QmSir6qPL1tjuxd8LkR8VZq6v625ExAUVs2eCLeqQuaPGU/goprocess/periodic" - notifier "QmUtEiB6DmXs7eLJiwS9YFyTAtptqzaWutxCsjHy7UKEgo/go-notifier" - logging "QmWRypnfEwrgH4k93KEHN5hng7VjKYkWmzDYRuTZeh2Mgh/go-log" - nat "github.com/fd/go-nat" + manet "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" + ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + + notifier "gx/QmRV5aE4LZ4Kw9g2VAMpbEcaT7uz3Z3WxgdW8mzRPLL7g8/go-notifier" + nat "gx/QmWjjKcvWKtkwzbqF6t3Kz3wXA3Uj8DRAbPaTtcgFnvTQt/go-nat" + goprocess "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess" + periodic "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess/periodic" + logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" ) var ( From a478405a422fcfc24d8288d2232604208454e98d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 19 Nov 2015 16:20:59 -0800 Subject: [PATCH 0051/3965] make new stream calls accept a context --- p2p/net/swarm/swarm.go | 5 +++-- p2p/net/swarm/swarm_net.go | 4 ++-- p2p/net/swarm/swarm_test.go | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index a6ddae3ccd..41c2ca95b0 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -192,16 +192,17 @@ func (s *Swarm) SetStreamHandler(handler inet.StreamHandler) { } // NewStreamWithPeer creates a new stream on any available connection to p -func (s *Swarm) NewStreamWithPeer(p peer.ID) (*Stream, error) { +func (s *Swarm) NewStreamWithPeer(ctx context.Context, p peer.ID) (*Stream, error) { // if we have no connections, try connecting. if len(s.ConnectionsToPeer(p)) == 0 { log.Debug("Swarm: NewStreamWithPeer no connections. Attempting to connect...") - if _, err := s.Dial(s.Context(), p); err != nil { + if _, err := s.Dial(ctx, p); err != nil { return nil, err } } log.Debug("Swarm: NewStreamWithPeer...") + // TODO: think about passing a context down to NewStreamWithGroup st, err := s.swarm.NewStreamWithGroup(p) return wrapStream(st), err } diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index d8724a49f9..e5a6e0216a 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -132,9 +132,9 @@ func (n *Network) Connectedness(p peer.ID) inet.Connectedness { // NewStream returns a new stream to given peer p. // If there is no connection to p, attempts to create one. -func (n *Network) NewStream(p peer.ID) (inet.Stream, error) { +func (n *Network) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { log.Debugf("[%s] network opening stream to peer [%s]", n.local, p) - s, err := n.Swarm().NewStreamWithPeer(p) + s, err := n.Swarm().NewStreamWithPeer(ctx, p) if err != nil { return nil, err } diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index c8a1ce548e..b6aa692cb0 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -129,7 +129,7 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { defer wg.Done() // first, one stream per peer (nice) - stream, err := s1.NewStreamWithPeer(p) + stream, err := s1.NewStreamWithPeer(ctx, p) if err != nil { errChan <- err return From 2d1b1db0e4f15d351dd3aa7dd64815c885d8588c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 6 Dec 2015 23:00:10 -0800 Subject: [PATCH 0052/3965] WIP --- p2p/net/swarm/addr/addr.go | 8 ++++---- p2p/net/swarm/addr/addr_test.go | 4 ++-- p2p/net/swarm/dial_test.go | 6 +++--- p2p/net/swarm/peers_test.go | 4 ++-- p2p/net/swarm/simul_test.go | 4 ++-- p2p/net/swarm/swarm.go | 20 ++++++++++---------- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 4 ++-- p2p/net/swarm/swarm_conn.go | 6 +++--- p2p/net/swarm/swarm_dial.go | 6 +++--- p2p/net/swarm/swarm_listen.go | 6 +++--- p2p/net/swarm/swarm_net.go | 6 +++--- p2p/net/swarm/swarm_net_test.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 4 ++-- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 4 ++-- 16 files changed, 44 insertions(+), 44 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index c74147527b..39f00dba1c 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,11 +3,11 @@ package addrutil import ( "fmt" - logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" + logging "github.com/ipfs/go-log" - manet "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + context "golang.org/x/net/context" ) var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 120195a122..49eec13511 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,8 +3,8 @@ package addrutil import ( "testing" - manet "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" ) func newMultiaddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 0df87fd293..57a4d7f161 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -13,9 +13,9 @@ import ( testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" - manet "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + context "golang.org/x/net/context" ) func acceptAndHang(l net.Listener) { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index ef9c22ff60..b107fbfe83 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -5,8 +5,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 408b39b244..bd753e7660 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -9,8 +9,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" ci "github.com/ipfs/go-libp2p/testutil/ci" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 41c2ca95b0..c71ef359de 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -16,16 +16,16 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" - ps "gx/QmQMkNFMuh1UJbdaggASZpLtCroTnAYcWNenYL5hrzLJrT/go-peerstream" - pst "gx/QmRmT6MSnfhRDW1PTUGSd3z4fqXK48GUequQAZzeT4c5iC/go-stream-muxer" - psmss "gx/QmRmT6MSnfhRDW1PTUGSd3z4fqXK48GUequQAZzeT4c5iC/go-stream-muxer/multistream" - mafilter "gx/QmVCmuhgDFer5MW5737Z8GtBEGpUyEkFnLUv4ASDWnLZdC/multiaddr-filter" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess" - goprocessctx "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess/context" - - logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + pst "github.com/jbenet/go-stream-muxer" + psmss "github.com/jbenet/go-stream-muxer/multistream" + "github.com/jbenet/goprocess" + goprocessctx "github.com/jbenet/goprocess/context" + mafilter "github.com/whyrusleeping/multiaddr-filter" + context "golang.org/x/net/context" + + logging "github.com/ipfs/go-log" ) var log = logging.Logger("swarm2") diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 81a77bb8e0..039683bfb9 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -4,7 +4,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 7383beeabe..7db46d3444 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -8,8 +8,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index e7d1ec91dc..840a9b465c 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,9 +8,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" - ps "gx/QmQMkNFMuh1UJbdaggASZpLtCroTnAYcWNenYL5hrzLJrT/go-peerstream" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + context "golang.org/x/net/context" ) // a Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index fa4fce1e47..0cb2858768 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -12,10 +12,10 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" + "github.com/jbenet/go-multiaddr-net" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index bd1810b462..f184640658 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -9,9 +9,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" transport "github.com/ipfs/go-libp2p/p2p/net/transport" - ps "gx/QmQMkNFMuh1UJbdaggASZpLtCroTnAYcWNenYL5hrzLJrT/go-peerstream" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + context "golang.org/x/net/context" ) // Open listeners and reuse-dialers for the given addresses diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index e5a6e0216a..a8361fc5bb 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -8,9 +8,9 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess" + ma "github.com/jbenet/go-multiaddr" + "github.com/jbenet/goprocess" + context "golang.org/x/net/context" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index c40b89cc63..14c590b9d1 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -5,9 +5,9 @@ import ( "testing" "time" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/p2p/test/util" + context "golang.org/x/net/context" ) // TestConnectednessCorrect starts a few networks, connects a few diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 9b59bb24d9..0d976e81d0 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" inet "github.com/ipfs/go-libp2p/p2p/net" peer "github.com/ipfs/go-libp2p/p2p/peer" diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index f4ee314958..7965d27439 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,7 +3,7 @@ package swarm import ( inet "github.com/ipfs/go-libp2p/p2p/net" - ps "gx/QmQMkNFMuh1UJbdaggASZpLtCroTnAYcWNenYL5hrzLJrT/go-peerstream" + ps "github.com/jbenet/go-peerstream" ) // a Stream is a wrapper around a ps.Stream that exposes a way to get diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index b6aa692cb0..a1c35696dc 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -14,8 +14,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func EchoStreamHandler(stream inet.Stream) { From faae73da45d08ece661e90069f0c278c711ded34 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 6 Dec 2015 23:00:10 -0800 Subject: [PATCH 0053/3965] WIP --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/peer.go | 10 +++++----- p2p/host/peerstore/peer_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/queue/distance.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 4 ++-- p2p/host/peerstore/queue/sync.go | 4 ++-- 10 files changed, 17 insertions(+), 17 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index ae9073ace4..a172373d8d 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 173fbca0fd..46140641ca 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 1129b0637c..87b8206eee 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -4,7 +4,7 @@ import ( "sync" "time" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 59bcf19047..0de0c6ab3e 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) func IDS(t *testing.T, ids string) ID { diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 0e7c009eef..2f2a48683b 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -7,13 +7,13 @@ import ( "fmt" "strings" - b58 "gx/QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - mh "gx/Qma7dqy7ZVH4tkNJdC9TRrA82Uz5fQfbbwuvmNVVc17r7a/go-multihash" + b58 "github.com/jbenet/go-base58" + ma "github.com/jbenet/go-multiaddr" + mh "github.com/jbenet/go-multihash" + u "github.com/ipfs/go-ipfs-util" ic "github.com/ipfs/go-libp2p/p2p/crypto" - u "gx/QmQA79FfVsUnGkH3TgKDqcDkupfjqLSJ6EYwDuDDZK8nhD/go-ipfs-util" - logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" + logging "github.com/ipfs/go-log" ) var log = logging.Logger("peer") diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 3ad9c23a22..93cbfc86c8 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -6,12 +6,12 @@ import ( "strings" "testing" + u "github.com/ipfs/go-ipfs-util" ic "github.com/ipfs/go-libp2p/p2p/crypto" . "github.com/ipfs/go-libp2p/p2p/peer" tu "github.com/ipfs/go-libp2p/testutil" - u "gx/QmQA79FfVsUnGkH3TgKDqcDkupfjqLSJ6EYwDuDDZK8nhD/go-ipfs-util" - b58 "gx/QmNsoHoCVhgXcv1Yg45jtkMgimxorTAN36fV9AQMFXHHAQ/go-base58" + b58 "github.com/jbenet/go-base58" ) var gen1 keyset // generated diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index ef86182653..f8e88754e2 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -9,7 +9,7 @@ import ( //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index beb7605b33..09da0cb88a 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -6,7 +6,7 @@ import ( "sync" peer "github.com/ipfs/go-libp2p/p2p/peer" - ks "gx/QmThbzo9oBE7FAVmgjp3xYv5Yra81uB7KXbrTei47gWT6o/go-keyspace" + ks "github.com/whyrusleeping/go-keyspace" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index ca449fe698..8d43bb690e 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -6,10 +6,10 @@ import ( "testing" "time" + u "github.com/ipfs/go-ipfs-util" peer "github.com/ipfs/go-libp2p/p2p/peer" - u "gx/QmQA79FfVsUnGkH3TgKDqcDkupfjqLSJ6EYwDuDDZK8nhD/go-ipfs-util" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" + context "golang.org/x/net/context" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index ccece46af4..fd49e00545 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -2,8 +2,8 @@ package queue import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - context "gx/QmacZi9WygGK7Me8mH53pypyscHzU386aUZXpr28GZgUct/context" - logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" + logging "github.com/ipfs/go-log" + context "golang.org/x/net/context" ) var log = logging.Logger("peerqueue") From 8863821e46a309d65f8b34b59f832a4d7d07dd2d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 6 Dec 2015 23:00:10 -0800 Subject: [PATCH 0054/3965] WIP --- p2p/net/nat/nat.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index c549527c57..6032ccd5a1 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,14 +8,14 @@ import ( "sync" "time" - manet "gx/QmNT7d1e4Xcp3KcsvxyzUHVtqrR43uypoxLLzdKj6YZga2/go-multiaddr-net" - ma "gx/QmVUi2ncqnU48zsPgR1rQosDGwY3SSZ1Ndp33j33YjXdsj/go-multiaddr" - - notifier "gx/QmRV5aE4LZ4Kw9g2VAMpbEcaT7uz3Z3WxgdW8mzRPLL7g8/go-notifier" - nat "gx/QmWjjKcvWKtkwzbqF6t3Kz3wXA3Uj8DRAbPaTtcgFnvTQt/go-nat" - goprocess "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess" - periodic "gx/QmfDXyLfKNfja2XebomRZjZ2UZCa4BDyFoCymKtzNRVQ5b/goprocess/periodic" - logging "gx/QmfZZB1aVXWA4kaR5R4e9NifERT366TTCSagkfhmAbYLsu/go-log" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + + nat "github.com/fd/go-nat" + logging "github.com/ipfs/go-log" + goprocess "github.com/jbenet/goprocess" + periodic "github.com/jbenet/goprocess/periodic" + notifier "github.com/whyrusleeping/go-notifier" ) var ( From 3cdb2f50f7ac455ed13e646789f6ddb8f8ebe01a Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 4 Jan 2016 05:45:58 -0800 Subject: [PATCH 0055/3965] path rewrites --- p2p/net/swarm/addr/addr.go | 8 ++++---- p2p/net/swarm/addr/addr_test.go | 4 ++-- p2p/net/swarm/dial_test.go | 6 +++--- p2p/net/swarm/peers_test.go | 4 ++-- p2p/net/swarm/simul_test.go | 4 ++-- p2p/net/swarm/swarm.go | 20 ++++++++++---------- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 4 ++-- p2p/net/swarm/swarm_conn.go | 6 +++--- p2p/net/swarm/swarm_dial.go | 6 +++--- p2p/net/swarm/swarm_listen.go | 6 +++--- p2p/net/swarm/swarm_net.go | 6 +++--- p2p/net/swarm/swarm_net_test.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 4 ++-- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 4 ++-- 16 files changed, 44 insertions(+), 44 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 39f00dba1c..8711e8daec 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,11 +3,11 @@ package addrutil import ( "fmt" - logging "github.com/ipfs/go-log" + logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" - context "golang.org/x/net/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + manet "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 49eec13511..aa82097c3d 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,8 +3,8 @@ package addrutil import ( "testing" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + manet "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" ) func newMultiaddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 57a4d7f161..72dd85860a 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -13,9 +13,9 @@ import ( testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" - context "golang.org/x/net/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + manet "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) func acceptAndHang(l net.Listener) { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index b107fbfe83..3451ec1f51 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -5,8 +5,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index bd753e7660..a838f3044d 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -9,8 +9,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" ci "github.com/ipfs/go-libp2p/testutil/ci" - ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index c71ef359de..c422645c5c 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -16,16 +16,16 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "github.com/jbenet/go-multiaddr" - ps "github.com/jbenet/go-peerstream" - pst "github.com/jbenet/go-stream-muxer" - psmss "github.com/jbenet/go-stream-muxer/multistream" - "github.com/jbenet/goprocess" - goprocessctx "github.com/jbenet/goprocess/context" - mafilter "github.com/whyrusleeping/multiaddr-filter" - context "golang.org/x/net/context" - - logging "github.com/ipfs/go-log" + ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" + "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" + goprocessctx "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + pst "gx/ipfs/QmTYr6RrJs8b63LTVwahmtytnuqzsLfNPBJp6EvmFWHbGh/go-stream-muxer" + psmss "gx/ipfs/QmTYr6RrJs8b63LTVwahmtytnuqzsLfNPBJp6EvmFWHbGh/go-stream-muxer/multistream" + mafilter "gx/ipfs/QmW3Q7RQa8D1qCEEeyKCBV1drgFxvHBqAZ3zgCujEwKpHD/multiaddr-filter" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + + logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" ) var log = logging.Logger("swarm2") diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 039683bfb9..5b7aae5fb2 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -4,7 +4,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - ma "github.com/jbenet/go-multiaddr" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 7db46d3444..f48291bcd1 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -8,8 +8,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" - ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 840a9b465c..b8c5633f1e 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,9 +8,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "github.com/jbenet/go-multiaddr" - ps "github.com/jbenet/go-peerstream" - context "golang.org/x/net/context" + ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) // a Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 0cb2858768..e05a2a2f37 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -12,10 +12,10 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - "github.com/jbenet/go-multiaddr-net" + "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" - ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index f184640658..4584250f7d 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -9,9 +9,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" transport "github.com/ipfs/go-libp2p/p2p/net/transport" - ma "github.com/jbenet/go-multiaddr" - ps "github.com/jbenet/go-peerstream" - context "golang.org/x/net/context" + ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) // Open listeners and reuse-dialers for the given addresses diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index a8361fc5bb..1d75814da1 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -8,9 +8,9 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - ma "github.com/jbenet/go-multiaddr" - "github.com/jbenet/goprocess" - context "golang.org/x/net/context" + "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 14c590b9d1..1ffb44812e 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -7,7 +7,7 @@ import ( inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/p2p/test/util" - context "golang.org/x/net/context" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) // TestConnectednessCorrect starts a few networks, connects a few diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 0d976e81d0..d6f06da7ce 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" - ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" inet "github.com/ipfs/go-libp2p/p2p/net" peer "github.com/ipfs/go-libp2p/p2p/peer" diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 7965d27439..3f9115fc6c 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,7 +3,7 @@ package swarm import ( inet "github.com/ipfs/go-libp2p/p2p/net" - ps "github.com/jbenet/go-peerstream" + ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" ) // a Stream is a wrapper around a ps.Stream that exposes a way to get diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index a1c35696dc..1166f52f1f 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -14,8 +14,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" - ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) func EchoStreamHandler(stream inet.Stream) { From cc235880fc92e3912191ede9b2cb6af5ea24f36e Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 4 Jan 2016 05:45:58 -0800 Subject: [PATCH 0056/3965] path rewrites --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/peer.go | 10 +++++----- p2p/host/peerstore/peer_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 4 ++-- p2p/host/peerstore/queue/sync.go | 4 ++-- 9 files changed, 16 insertions(+), 16 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index a172373d8d..2769e4ba84 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "github.com/jbenet/go-multiaddr" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 46140641ca..03e9366847 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "github.com/jbenet/go-multiaddr" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 87b8206eee..b2243e101f 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -4,7 +4,7 @@ import ( "sync" "time" - ma "github.com/jbenet/go-multiaddr" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 0de0c6ab3e..66dce5ab7f 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "github.com/jbenet/go-multiaddr" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" ) func IDS(t *testing.T, ids string) ID { diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 2f2a48683b..e3544a1265 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -7,13 +7,13 @@ import ( "fmt" "strings" - b58 "github.com/jbenet/go-base58" - ma "github.com/jbenet/go-multiaddr" - mh "github.com/jbenet/go-multihash" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + b58 "gx/ipfs/QmT8rehPR3F6bmwL6zjUN8XpiDBFFpMP2myPdC6ApsWfJf/go-base58" + mh "gx/ipfs/QmYf7ng2hG5XBtJA3tN34DQ2GUN5HNksEw1rLDkmr6vGku/go-multihash" - u "github.com/ipfs/go-ipfs-util" ic "github.com/ipfs/go-libp2p/p2p/crypto" - logging "github.com/ipfs/go-log" + u "gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util" + logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" ) var log = logging.Logger("peer") diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 93cbfc86c8..013bcf2e02 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -6,12 +6,12 @@ import ( "strings" "testing" - u "github.com/ipfs/go-ipfs-util" ic "github.com/ipfs/go-libp2p/p2p/crypto" . "github.com/ipfs/go-libp2p/p2p/peer" tu "github.com/ipfs/go-libp2p/testutil" + u "gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util" - b58 "github.com/jbenet/go-base58" + b58 "gx/ipfs/QmT8rehPR3F6bmwL6zjUN8XpiDBFFpMP2myPdC6ApsWfJf/go-base58" ) var gen1 keyset // generated diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index f8e88754e2..9eb49e0731 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -9,7 +9,7 @@ import ( //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" - ma "github.com/jbenet/go-multiaddr" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 8d43bb690e..34179a5989 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -6,10 +6,10 @@ import ( "testing" "time" - u "github.com/ipfs/go-ipfs-util" peer "github.com/ipfs/go-libp2p/p2p/peer" + u "gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util" - context "golang.org/x/net/context" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index fd49e00545..b679a4930d 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -2,8 +2,8 @@ package queue import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - logging "github.com/ipfs/go-log" - context "golang.org/x/net/context" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" ) var log = logging.Logger("peerqueue") From eff52d23d3cd2a68a7177a95d25569e9e27f8626 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 4 Jan 2016 05:45:58 -0800 Subject: [PATCH 0057/3965] path rewrites --- p2p/net/nat/nat.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 6032ccd5a1..c6f775989f 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,14 +8,14 @@ import ( "sync" "time" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" - - nat "github.com/fd/go-nat" - logging "github.com/ipfs/go-log" - goprocess "github.com/jbenet/goprocess" - periodic "github.com/jbenet/goprocess/periodic" - notifier "github.com/whyrusleeping/go-notifier" + ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + manet "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" + + nat "gx/ipfs/QmNLvkCDV6ZjUJsEwGNporYBuZdhWT6q7TBVYQwwRv12HT/go-nat" + goprocess "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" + periodic "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/periodic" + logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" + notifier "gx/ipfs/QmbcS9XrwZkF1rZj8bBwwzoYhVuA2PCnPhFUL1pyWGgt2A/go-notifier" ) var ( From b9674575f9b130d076e1feda0ec5bf4d226d2870 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 5 Jan 2016 14:24:02 -0800 Subject: [PATCH 0058/3965] add utp to have feature parity with go-ipfs --- p2p/net/swarm/swarm.go | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index c422645c5c..572933f9da 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -84,13 +84,16 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, } s := &Swarm{ - swarm: ps.NewSwarm(PSTransport), - local: local, - peers: peers, - ctx: ctx, - dialT: DialTimeout, - notifs: make(map[inet.Notifiee]ps.Notifiee), - transports: []transport.Transport{transport.NewTCPTransport()}, + swarm: ps.NewSwarm(PSTransport), + local: local, + peers: peers, + ctx: ctx, + dialT: DialTimeout, + notifs: make(map[inet.Notifiee]ps.Notifiee), + transports: []transport.Transport{ + transport.NewTCPTransport(), + transport.NewUtpTransport(), + }, bwc: bwc, fdRateLimit: make(chan struct{}, concurrentFdDials), Filters: filter.NewFilters(), From a8f9f6caca31f32d4d04b38be3e26cb3da465809 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sat, 30 Jan 2016 17:12:33 -0800 Subject: [PATCH 0059/3965] update to utp code from master of go-ipfs --- p2p/net/swarm/addr/addr.go | 4 ++-- p2p/net/swarm/addr/addr_test.go | 8 +------- p2p/net/swarm/dial_test.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 6 +++--- 4 files changed, 7 insertions(+), 13 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 8711e8daec..4fdda74d4a 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -18,8 +18,8 @@ var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") var SupportedTransportStrings = []string{ "/ip4/tcp", "/ip6/tcp", - // "/ip4/udp/utp", disabled because the lib is broken - // "/ip6/udp/utp", disabled because the lib is broken + "/ip4/udp/utp", + "/ip6/udp/utp", // "/ip4/udp/udt", disabled because the lib doesnt work on arm // "/ip6/udp/udt", disabled because the lib doesnt work on arm } diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index aa82097c3d..2b0532ceb3 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -20,7 +20,6 @@ func TestFilterAddrs(t *testing.T) { bad := []ma.Multiaddr{ newMultiaddr(t, "/ip4/1.2.3.4/udp/1234"), // unreliable newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/sctp/1234"), // not in manet - newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/utp"), // utp is broken newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/udt"), // udt is broken on arm newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), // link local newMultiaddr(t, "/ip6/fe80::100/tcp/1234"), // link local @@ -29,6 +28,7 @@ func TestFilterAddrs(t *testing.T) { good := []ma.Multiaddr{ newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), newMultiaddr(t, "/ip6/::1/tcp/1234"), + newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/utp"), } goodAndBad := append(good, bad...) @@ -39,18 +39,12 @@ func TestFilterAddrs(t *testing.T) { if AddrUsable(a, false) { t.Errorf("addr %s should be unusable", a) } - if AddrUsable(a, true) { - t.Errorf("addr %s should be unusable", a) - } } for _, a := range good { if !AddrUsable(a, false) { t.Errorf("addr %s should be usable", a) } - if !AddrUsable(a, true) { - t.Errorf("addr %s should be usable", a) - } } subtestAddrsEqual(t, FilterUsableAddrs(bad), []ma.Multiaddr{}) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 72dd85860a..127edffd52 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -151,7 +151,7 @@ func TestDialWait(t *testing.T) { func TestDialBackoff(t *testing.T) { // t.Skip("skipping for another test") if ci.IsRunning() { - t.Skip("travis and jenkins will never have fun with this test") + t.Skip("travis will never have fun with this test") } t.Parallel() diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index f48291bcd1..214916e47e 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -25,7 +25,6 @@ func TestFilterAddrs(t *testing.T) { bad := []ma.Multiaddr{ m("/ip4/1.2.3.4/udp/1234"), // unreliable m("/ip4/1.2.3.4/udp/1234/sctp/1234"), // not in manet - m("/ip4/1.2.3.4/udp/1234/utp"), // utp is broken m("/ip4/1.2.3.4/udp/1234/udt"), // udt is broken on arm m("/ip6/fe80::1/tcp/0"), // link local m("/ip6/fe80::100/tcp/1234"), // link local @@ -34,6 +33,7 @@ func TestFilterAddrs(t *testing.T) { good := []ma.Multiaddr{ m("/ip4/127.0.0.1/tcp/0"), m("/ip6/::1/tcp/0"), + m("/ip4/1.2.3.4/udp/1234/utp"), } goodAndBad := append(good, bad...) @@ -41,13 +41,13 @@ func TestFilterAddrs(t *testing.T) { // test filters for _, a := range bad { - if addrutil.AddrUsable(a, true) { + if addrutil.AddrUsable(a, false) { t.Errorf("addr %s should be unusable", a) } } for _, a := range good { - if !addrutil.AddrUsable(a, true) { + if !addrutil.AddrUsable(a, false) { t.Errorf("addr %s should be usable", a) } } From a84e25a7c9390bb9ba361e01bfe1faa01a3a200f Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 10 Feb 2016 09:32:01 -0800 Subject: [PATCH 0060/3965] make PeerInfo loggable --- p2p/host/peerstore/peer.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index e3544a1265..e4a53d9f2b 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -141,6 +141,13 @@ type PeerInfo struct { Addrs []ma.Multiaddr } +func (pi *PeerInfo) Loggable() map[string]interface{} { + return map[string]interface{}{ + "peerID": pi.ID.Pretty(), + "addrs": pi.Addrs, + } +} + func (pi *PeerInfo) MarshalJSON() ([]byte, error) { out := make(map[string]interface{}) out["ID"] = IDB58Encode(pi.ID) From a7a48ae51108234952e077857cfe64b126525f10 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sat, 13 Feb 2016 16:22:02 -0800 Subject: [PATCH 0061/3965] change listener errors to warnings --- p2p/net/swarm/swarm_listen.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 4584250f7d..045ac9ccf8 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -126,7 +126,7 @@ func (s *Swarm) addConnListener(list conn.Listener) error { if !more { return } - log.Errorf("swarm listener accept error: %s", err) + log.Warningf("swarm listener accept error: %s", err) case <-ctx.Done(): return } From 94f0726bd7acad4fd6602f49ff6ebe551e2c231d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 16 Feb 2016 17:02:00 -0800 Subject: [PATCH 0062/3965] finish deps rewrite --- p2p/host/peerstore/queue/distance.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 09da0cb88a..337a752978 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -6,7 +6,7 @@ import ( "sync" peer "github.com/ipfs/go-libp2p/p2p/peer" - ks "github.com/whyrusleeping/go-keyspace" + ks "gx/ipfs/QmUusaX99BZoELh7dmPgirqRQ1FAmMnmnBn3oiqDFGBUSc/go-keyspace" ) // peerMetric tracks a peer and its distance to something else. From 55941459a299811d732c2482e02bffd49699a515 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 3 Mar 2016 13:03:24 -0800 Subject: [PATCH 0063/3965] update version of go-multiaddr --- p2p/net/swarm/addr/addr.go | 7 +++---- p2p/net/swarm/addr/addr_test.go | 4 ++-- p2p/net/swarm/dial_test.go | 4 ++-- p2p/net/swarm/peers_test.go | 3 +-- p2p/net/swarm/simul_test.go | 4 ++-- p2p/net/swarm/swarm.go | 5 ++--- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_dial.go | 5 ++--- p2p/net/swarm/swarm_listen.go | 5 ++--- p2p/net/swarm/swarm_net.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 5 ++--- p2p/net/swarm/swarm_test.go | 2 +- 14 files changed, 23 insertions(+), 29 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 4fdda74d4a..3cf1b7d149 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,11 +3,10 @@ package addrutil import ( "fmt" - logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" - - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" - manet "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" + manet "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 2b0532ceb3..af291fc3f7 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,8 +3,8 @@ package addrutil import ( "testing" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" - manet "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" + manet "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func newMultiaddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 127edffd52..66bac550a8 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -13,9 +13,9 @@ import ( testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" - manet "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" + manet "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func acceptAndHang(l net.Listener) { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 3451ec1f51..8b71f77f34 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -5,12 +5,11 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func TestPeers(t *testing.T) { - // t.Skip("skipping for another test") ctx := context.Background() swarms := makeSwarms(ctx, t, 2) diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index a838f3044d..8c626b805d 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -9,12 +9,12 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" ci "github.com/ipfs/go-libp2p/testutil/ci" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func TestSimultOpen(t *testing.T) { - // t.Skip("skipping for another test") + t.Parallel() ctx := context.Background() diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 572933f9da..8e4f32f2e1 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -19,13 +19,12 @@ import ( ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" goprocessctx "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/context" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" pst "gx/ipfs/QmTYr6RrJs8b63LTVwahmtytnuqzsLfNPBJp6EvmFWHbGh/go-stream-muxer" psmss "gx/ipfs/QmTYr6RrJs8b63LTVwahmtytnuqzsLfNPBJp6EvmFWHbGh/go-stream-muxer/multistream" - mafilter "gx/ipfs/QmW3Q7RQa8D1qCEEeyKCBV1drgFxvHBqAZ3zgCujEwKpHD/multiaddr-filter" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" + mafilter "gx/ipfs/QmcR6dLYF8Eozaae3wGd5wjq76bofzmmbvQmtwobxvfhEt/multiaddr-filter" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) var log = logging.Logger("swarm2") diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 5b7aae5fb2..faab0a3fdb 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -4,7 +4,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 214916e47e..2e69e14e11 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -8,8 +8,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index b8c5633f1e..a07818d48a 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -9,8 +9,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) // a Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index e05a2a2f37..96cccd3c97 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -12,10 +12,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" - - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 045ac9ccf8..50550248f5 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -10,11 +10,10 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" -) + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" +) // Open listeners and reuse-dialers for the given addresses -// Open listeners and reuse-dialers for the given addresses func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { errs := make([]error, len(addrs)) var succeeded int diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 1d75814da1..863da8efec 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -9,8 +9,8 @@ import ( inet "github.com/ipfs/go-libp2p/p2p/net" "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index d6f06da7ce..9a8052e9ef 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,11 +4,10 @@ import ( "testing" "time" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - inet "github.com/ipfs/go-libp2p/p2p/net" peer "github.com/ipfs/go-libp2p/p2p/peer" + context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func TestNotifications(t *testing.T) { diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 1166f52f1f..121d1b0f35 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -14,8 +14,8 @@ import ( peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func EchoStreamHandler(stream inet.Stream) { From c5375abe77c84e0b933b4a4c416a967960901da7 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 3 Mar 2016 13:03:24 -0800 Subject: [PATCH 0064/3965] update version of go-multiaddr --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/peer.go | 8 +++----- p2p/host/peerstore/peerstore.go | 2 +- 6 files changed, 8 insertions(+), 10 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index 2769e4ba84..280a23e522 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 03e9366847..703aabdb90 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index b2243e101f..7b00dbaf5c 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -4,7 +4,7 @@ import ( "sync" "time" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 66dce5ab7f..64f69f055b 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) func IDS(t *testing.T, ids string) ID { diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index e4a53d9f2b..dc529cf388 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -7,18 +7,16 @@ import ( "fmt" "strings" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + ic "github.com/ipfs/go-libp2p/p2p/crypto" b58 "gx/ipfs/QmT8rehPR3F6bmwL6zjUN8XpiDBFFpMP2myPdC6ApsWfJf/go-base58" mh "gx/ipfs/QmYf7ng2hG5XBtJA3tN34DQ2GUN5HNksEw1rLDkmr6vGku/go-multihash" - - ic "github.com/ipfs/go-libp2p/p2p/crypto" u "gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util" - logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" + logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" // ID represents the identity of a peer. + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) var log = logging.Logger("peer") -// ID represents the identity of a peer. type ID string // Pretty returns a b58-encoded string of the ID diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 9eb49e0731..22fe410536 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -9,7 +9,7 @@ import ( //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) const ( From 122bc10af3b818a6bb6464494d72fe7517ec96d5 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 3 Mar 2016 13:03:24 -0800 Subject: [PATCH 0065/3965] update version of go-multiaddr --- p2p/net/nat/nat.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index c6f775989f..d840a669ad 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,14 +8,13 @@ import ( "sync" "time" - ma "gx/ipfs/QmR3JkmZBKYXgNMNsNZawm914455Qof3PEopwuVSeXG7aV/go-multiaddr" - manet "gx/ipfs/QmYtzQmUwPFGxjCXctJ8e6GXS8sYfoXy2pdeMbS5SFWqRi/go-multiaddr-net" - nat "gx/ipfs/QmNLvkCDV6ZjUJsEwGNporYBuZdhWT6q7TBVYQwwRv12HT/go-nat" + manet "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" goprocess "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" periodic "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/periodic" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" notifier "gx/ipfs/QmbcS9XrwZkF1rZj8bBwwzoYhVuA2PCnPhFUL1pyWGgt2A/go-notifier" + ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) var ( From 5f0753d75f2980c79b514505511295c91d72e658 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 4 Mar 2016 09:27:52 -0800 Subject: [PATCH 0066/3965] Add fallback dialer Previously we were unable to dial on a given transport if there were no listener addresses defined for that. This meant that most people couldnt use utp (as they had no utp listener transport to dial from). --- p2p/net/swarm/dial_test.go | 55 +++++++++++++++++++++++++++++++++++++ p2p/net/swarm/swarm_test.go | 25 ++++++++++++++--- 2 files changed, 76 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 66bac550a8..a6bc84e861 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -18,6 +18,61 @@ import ( ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) +func closeSwarms(swarms []*Swarm) { + for _, s := range swarms { + s.Close() + } +} + +func TestBasicDial(t *testing.T) { + t.Parallel() + ctx := context.Background() + + swarms := makeSwarms(ctx, t, 2) + defer closeSwarms(swarms) + s1 := swarms[0] + s2 := swarms[1] + + s1.peers.AddAddrs(s2.local, s2.ListenAddresses(), peer.PermanentAddrTTL) + + c, err := s1.Dial(ctx, s2.local) + if err != nil { + t.Fatal(err) + } + + s, err := c.NewStream() + if err != nil { + t.Fatal(err) + } + + s.Close() +} + +func TestDialWithNoListeners(t *testing.T) { + t.Parallel() + ctx := context.Background() + + s1 := makeDialOnlySwarm(ctx, t) + + swarms := makeSwarms(ctx, t, 1) + defer closeSwarms(swarms) + s2 := swarms[0] + + s1.peers.AddAddrs(s2.local, s2.ListenAddresses(), peer.PermanentAddrTTL) + + c, err := s1.Dial(ctx, s2.local) + if err != nil { + t.Fatal(err) + } + + s, err := c.NewStream() + if err != nil { + t.Fatal(err) + } + + s.Close() +} + func acceptAndHang(l net.Listener) { conns := make([]net.Conn, 0, 10) for { diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 121d1b0f35..fe2ef2a207 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -24,31 +24,48 @@ func EchoStreamHandler(stream inet.Stream) { // pull out the ipfs conn c := stream.Conn() - log.Infof("%s ponging to %s", c.LocalPeer(), c.RemotePeer()) + log.Errorf("%s ponging to %s", c.LocalPeer(), c.RemotePeer()) buf := make([]byte, 4) for { if _, err := stream.Read(buf); err != nil { if err != io.EOF { - log.Info("ping receive error:", err) + log.Error("ping receive error:", err) } return } if !bytes.Equal(buf, []byte("ping")) { - log.Infof("ping receive error: ping != %s %v", buf, buf) + log.Errorf("ping receive error: ping != %s %v", buf, buf) return } if _, err := stream.Write([]byte("pong")); err != nil { - log.Info("pond send error:", err) + log.Error("pond send error:", err) return } } }() } +func makeDialOnlySwarm(ctx context.Context, t *testing.T) *Swarm { + id := testutil.RandIdentityOrFatal(t) + + peerstore := peer.NewPeerstore() + peerstore.AddPubKey(id.ID(), id.PublicKey()) + peerstore.AddPrivKey(id.ID(), id.PrivateKey()) + + swarm, err := NewSwarm(ctx, nil, id.ID(), peerstore, metrics.NewBandwidthCounter()) + if err != nil { + t.Fatal(err) + } + + swarm.SetStreamHandler(EchoStreamHandler) + + return swarm +} + func makeSwarms(ctx context.Context, t *testing.T, num int) []*Swarm { swarms := make([]*Swarm, 0, num) From 47680235d8dbe8ee1f02b8e2a8105d57eee0fef0 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 7 Mar 2016 19:55:13 -0800 Subject: [PATCH 0067/3965] make multistream multiplexer customizable --- p2p/muxer/muxer-multistream/multistream.go | 30 +++++++++++++++++----- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index d13605531b..94372e0c75 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -13,13 +13,15 @@ import ( yamux "github.com/jbenet/go-stream-muxer/yamux" ) -type transport struct { +type Transport struct { mux *mss.MultistreamMuxer tpts map[string]smux.Transport + + OrderPreference []string } -func NewTransport() smux.Transport { +func NewTransport() *Transport { mux := mss.NewMultistreamMuxer() mux.AddHandler("/multiplex", nil) mux.AddHandler("/spdystream", nil) @@ -31,13 +33,27 @@ func NewTransport() smux.Transport { "/yamux": yamux.DefaultTransport, } - return &transport{ - mux: mux, - tpts: tpts, + return &Transport{ + mux: mux, + tpts: tpts, + OrderPreference: []string{"/yamux", "/spdystream", "/multiplex"}, + } +} + +func NewBlankTransport() *Transport { + return &Transport{ + mux: mss.NewMultistreamMuxer(), + tpts: make(map[string]smux.Transport), } } -func (t *transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { +func (t *Transport) AddTransport(path string, tpt smux.Transport) { + t.mux.AddHandler(path, nil) + t.tpts[path] = tpt + t.OrderPreference = append(t.OrderPreference, path) +} + +func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { var proto string if isServer { selected, _, err := t.mux.Negotiate(nc) @@ -47,7 +63,7 @@ func (t *transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { proto = selected } else { // prefer yamux - selected, err := mss.SelectOneOf([]string{"/yamux", "/spdystream", "/multiplex"}, nc) + selected, err := mss.SelectOneOf(t.OrderPreference, nc) if err != nil { return nil, err } From 01d6273db10b17399e772652b497b885be4d9788 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 7 Mar 2016 22:43:13 -0800 Subject: [PATCH 0068/3965] switch to new version of go-stream-muxer --- p2p/net/swarm/swarm.go | 25 +++++++++++++++++++++---- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_listen.go | 6 +++--- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 2 +- 5 files changed, 27 insertions(+), 10 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 8e4f32f2e1..0fd7a25e45 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -4,6 +4,7 @@ package swarm import ( "fmt" + "io/ioutil" "sync" "time" @@ -16,11 +17,13 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" - ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" goprocessctx "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/context" - pst "gx/ipfs/QmTYr6RrJs8b63LTVwahmtytnuqzsLfNPBJp6EvmFWHbGh/go-stream-muxer" - psmss "gx/ipfs/QmTYr6RrJs8b63LTVwahmtytnuqzsLfNPBJp6EvmFWHbGh/go-stream-muxer/multistream" + pst "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer" + psmss "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/multistream" + spdy "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/spdystream" + yamux "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/yamux" + ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" mafilter "gx/ipfs/QmcR6dLYF8Eozaae3wGd5wjq76bofzmmbvQmtwobxvfhEt/multiaddr-filter" @@ -32,7 +35,21 @@ var log = logging.Logger("swarm2") var PSTransport pst.Transport func init() { - PSTransport = psmss.NewTransport() + msstpt := psmss.NewBlankTransport() + + ymxtpt := &yamux.Transport{ + AcceptBacklog: 2048, + ConnectionWriteTimeout: time.Second * 10, + KeepAliveInterval: time.Second * 30, + EnableKeepAlive: true, + MaxStreamWindowSize: uint32(1024 * 256), + LogOutput: ioutil.Discard, + } + + msstpt.AddTransport("/yamux", ymxtpt) + msstpt.AddTransport("/spdystream", spdy.Transport) + + PSTransport = msstpt } // Swarm is a connection muxer, allowing connections to other peers to diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index a07818d48a..04e9a4034f 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -8,7 +8,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" peer "github.com/ipfs/go-libp2p/p2p/peer" - ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" + ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 50550248f5..a4d5e943d1 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -9,10 +9,10 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" transport "github.com/ipfs/go-libp2p/p2p/net/transport" - ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" + ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" -) // Open listeners and reuse-dialers for the given addresses +) // Open listeners and reuse-dialers for the given addresses func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { errs := make([]error, len(addrs)) @@ -152,7 +152,7 @@ func (s *Swarm) connHandler(c *ps.Conn) *Conn { if err != nil { log.Debug(err) log.Event(ctx, "newConnHandlerDisconnect", lgbl.NetConn(c.NetConn()), lgbl.Error(err)) - c.Close() // boom. close it. + c.Close() // boom. close it. return nil } diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 3f9115fc6c..3fbcc5d37c 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,7 +3,7 @@ package swarm import ( inet "github.com/ipfs/go-libp2p/p2p/net" - ps "gx/ipfs/QmQDPXRFzRcCGPbPViQCKjzbQBkZGpLV1f9KwXnksSNcTK/go-peerstream" + ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" ) // a Stream is a wrapper around a ps.Stream that exposes a way to get diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index fe2ef2a207..c69c7db38f 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -24,7 +24,7 @@ func EchoStreamHandler(stream inet.Stream) { // pull out the ipfs conn c := stream.Conn() - log.Errorf("%s ponging to %s", c.LocalPeer(), c.RemotePeer()) + log.Infof("%s ponging to %s", c.LocalPeer(), c.RemotePeer()) buf := make([]byte, 4) From 7110cd7c80bc6ae2710f77a404b851b8757bcfe6 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 8 Mar 2016 15:47:25 -0800 Subject: [PATCH 0069/3965] bump yamux backlog setting way up --- p2p/net/swarm/swarm.go | 2 +- p2p/net/swarm/swarm_listen.go | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 0fd7a25e45..fd66f73027 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -38,7 +38,7 @@ func init() { msstpt := psmss.NewBlankTransport() ymxtpt := &yamux.Transport{ - AcceptBacklog: 2048, + AcceptBacklog: 8192, ConnectionWriteTimeout: time.Second * 10, KeepAliveInterval: time.Second * 30, EnableKeepAlive: true, diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index a4d5e943d1..b83df4551c 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -12,8 +12,9 @@ import ( ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" -) // Open listeners and reuse-dialers for the given addresses +) +// Open listeners and reuse-dialers for the given addresses func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { errs := make([]error, len(addrs)) var succeeded int @@ -152,7 +153,7 @@ func (s *Swarm) connHandler(c *ps.Conn) *Conn { if err != nil { log.Debug(err) log.Event(ctx, "newConnHandlerDisconnect", lgbl.NetConn(c.NetConn()), lgbl.Error(err)) - c.Close() // boom. close it. + c.Close() // boom. close it. return nil } From a2c64420db0613c31377894ea8f0400b3400cc1c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 28 Mar 2016 20:16:52 -0700 Subject: [PATCH 0070/3965] fix accidental find and replace fail --- p2p/host/peerstore/peer.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index dc529cf388..41126dce38 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -41,7 +41,7 @@ func (id ID) String() string { //All sha256 nodes start with Qm //We can skip the Qm to make the peer.ID more useful - if strings.HasPrefix(pid, "gx/Qm") { + if strings.HasPrefix(pid, "Qm") { pid = pid[2:] } From b4e7e8a85143b872c5078ecacbbc39c4b91655dc Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 29 Mar 2016 18:38:24 -0700 Subject: [PATCH 0071/3965] update utp lib --- p2p/net/swarm/addr/addr.go | 2 +- p2p/net/swarm/addr/addr_test.go | 2 +- p2p/net/swarm/dial_test.go | 2 +- p2p/net/swarm/swarm.go | 2 +- p2p/net/swarm/swarm_dial.go | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 3cf1b7d149..14da595090 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,7 +3,7 @@ package addrutil import ( "fmt" - manet "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" + manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index af291fc3f7..16599b4089 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,7 +3,7 @@ package addrutil import ( "testing" - manet "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" + manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index a6bc84e861..6e987e24dd 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -13,7 +13,7 @@ import ( testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" - manet "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" + manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index fd66f73027..543c01c214 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -17,6 +17,7 @@ import ( transport "github.com/ipfs/go-libp2p/p2p/net/transport" peer "github.com/ipfs/go-libp2p/p2p/peer" + mafilter "gx/ipfs/QmPwfFAHUmvWDucLHRS9Xz2Kb1TNX2cY4LJ7pQjg9kVcae/multiaddr-filter" "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" goprocessctx "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/context" pst "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer" @@ -26,7 +27,6 @@ import ( ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" - mafilter "gx/ipfs/QmcR6dLYF8Eozaae3wGd5wjq76bofzmmbvQmtwobxvfhEt/multiaddr-filter" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 96cccd3c97..283bef1afb 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -12,7 +12,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" peer "github.com/ipfs/go-libp2p/p2p/peer" - "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" + "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) From a61a7ada39097a144fbf33feb2ac78cdfd963c23 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 29 Mar 2016 18:38:24 -0700 Subject: [PATCH 0072/3965] update utp lib --- p2p/net/nat/nat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index d840a669ad..0c7d86ca55 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -9,9 +9,9 @@ import ( "time" nat "gx/ipfs/QmNLvkCDV6ZjUJsEwGNporYBuZdhWT6q7TBVYQwwRv12HT/go-nat" - manet "gx/ipfs/QmQB7mNP3QE7b4zP2MQmsyJDqG5hzYE2CL8k1VyLWky2Ed/go-multiaddr-net" goprocess "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" periodic "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/periodic" + manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" notifier "gx/ipfs/QmbcS9XrwZkF1rZj8bBwwzoYhVuA2PCnPhFUL1pyWGgt2A/go-notifier" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" From a81c312df58891cfbcacf5a825a2ee8398055f59 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 6 Apr 2016 12:14:06 -0700 Subject: [PATCH 0073/3965] version multistream protocol tags --- p2p/net/swarm/swarm.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 543c01c214..b8e041e811 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -46,8 +46,8 @@ func init() { LogOutput: ioutil.Discard, } - msstpt.AddTransport("/yamux", ymxtpt) - msstpt.AddTransport("/spdystream", spdy.Transport) + msstpt.AddTransport("/yamux/1.0.0", ymxtpt) + msstpt.AddTransport("/spdy/3.1.0", spdy.Transport) PSTransport = msstpt } From d4973ee01b624d92a93b3355d3db041662035863 Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Sun, 10 Apr 2016 22:58:16 -0700 Subject: [PATCH 0074/3965] Fix address filtering for /ip6, add tests License: MIT Signed-off-by: Lars Gierth --- p2p/net/swarm/swarm_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index c69c7db38f..6c81a9c14f 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -303,7 +303,7 @@ func TestAddrBlocking(t *testing.T) { swarms := makeSwarms(ctx, t, 2) swarms[0].SetConnHandler(func(conn *Conn) { - t.Fatal("no connections should happen!") + t.Fatalf("no connections should happen! -- %s", conn) }) _, block, err := net.ParseCIDR("127.0.0.1/8") From c218be3e56b1764e3648a9efdc933ba26259a1b9 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 11 Apr 2016 23:24:32 -0700 Subject: [PATCH 0075/3965] move some deps out as gx packages --- p2p/net/swarm/dial_test.go | 3 +-- p2p/net/swarm/peers_test.go | 3 +-- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm.go | 5 ++--- p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_conn.go | 4 ++-- p2p/net/swarm/swarm_dial.go | 4 ++-- p2p/net/swarm/swarm_listen.go | 5 ++--- p2p/net/swarm/swarm_net.go | 3 +-- p2p/net/swarm/swarm_notif_test.go | 2 +- p2p/net/swarm/swarm_test.go | 2 +- 11 files changed, 15 insertions(+), 20 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 6e987e24dd..d21af292d8 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -8,10 +8,9 @@ import ( "time" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - peer "github.com/ipfs/go-libp2p/p2p/peer" - testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 8b71f77f34..9976d3bf4c 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -3,8 +3,7 @@ package swarm import ( "testing" - peer "github.com/ipfs/go-libp2p/p2p/peer" - + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 8c626b805d..b0256b8489 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -6,8 +6,8 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p/p2p/peer" ci "github.com/ipfs/go-libp2p/testutil/ci" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index b8e041e811..5326bf4a7f 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -14,16 +14,15 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" filter "github.com/ipfs/go-libp2p/p2p/net/filter" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - transport "github.com/ipfs/go-libp2p/p2p/net/transport" - peer "github.com/ipfs/go-libp2p/p2p/peer" - mafilter "gx/ipfs/QmPwfFAHUmvWDucLHRS9Xz2Kb1TNX2cY4LJ7pQjg9kVcae/multiaddr-filter" "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" goprocessctx "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/context" + transport "gx/ipfs/QmR4HhZm1fL8epXz8661Ru8ks5Y1Uk2u51quAWaCTD1zHg/go-libp2p-transport" pst "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer" psmss "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/multistream" spdy "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/spdystream" yamux "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/yamux" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 2e69e14e11..9cef367c78 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -5,8 +5,8 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 04e9a4034f..3ff2b91fcc 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -3,13 +3,13 @@ package swarm import ( "fmt" - ic "github.com/ipfs/go-libp2p/p2p/crypto" inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" - peer "github.com/ipfs/go-libp2p/p2p/peer" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + ic "gx/ipfs/QmaP38GJApheTr84f8R89vsT7oJLQw1AeCz4HqrQgv2njB/go-libp2p-crypto" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 283bef1afb..b569f92fa4 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -8,10 +8,10 @@ import ( "sync" "time" - lgbl "github.com/ipfs/go-libp2p/loggables" conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - peer "github.com/ipfs/go-libp2p/p2p/peer" + lgbl "gx/ipfs/QmSyBhZt2upyQ3NJmTpab1pX6hesA59vcYTGmgoDorZZbw/go-libp2p-loggables" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index b83df4551c..cf0c3a3326 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -3,12 +3,11 @@ package swarm import ( "fmt" - lgbl "github.com/ipfs/go-libp2p/loggables" mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" - transport "github.com/ipfs/go-libp2p/p2p/net/transport" - + transport "gx/ipfs/QmR4HhZm1fL8epXz8661Ru8ks5Y1Uk2u51quAWaCTD1zHg/go-libp2p-transport" + lgbl "gx/ipfs/QmSyBhZt2upyQ3NJmTpab1pX6hesA59vcYTGmgoDorZZbw/go-libp2p-loggables" ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 863da8efec..d99e04e135 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -3,10 +3,9 @@ package swarm import ( "fmt" - peer "github.com/ipfs/go-libp2p/p2p/peer" - metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 9a8052e9ef..29292eedbd 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -5,7 +5,7 @@ import ( "time" inet "github.com/ipfs/go-libp2p/p2p/net" - peer "github.com/ipfs/go-libp2p/p2p/peer" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index c69c7db38f..f637203a30 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -11,8 +11,8 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - peer "github.com/ipfs/go-libp2p/p2p/peer" testutil "github.com/ipfs/go-libp2p/testutil" + peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" From 6136edbae7337e7ab57fe915f35f83f9f8f4cd8d Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Fri, 15 Apr 2016 20:27:15 -0700 Subject: [PATCH 0076/3965] Undo import path rewrites --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/peer.go | 10 +++++----- p2p/host/peerstore/peer_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/queue/distance.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 4 ++-- p2p/host/peerstore/queue/sync.go | 4 ++-- 10 files changed, 17 insertions(+), 17 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index 280a23e522..a172373d8d 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 703aabdb90..46140641ca 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 7b00dbaf5c..87b8206eee 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -4,7 +4,7 @@ import ( "sync" "time" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 64f69f055b..0de0c6ab3e 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) func IDS(t *testing.T, ids string) ID { diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index 41126dce38..d928b70425 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -7,12 +7,12 @@ import ( "fmt" "strings" + u "github.com/ipfs/go-ipfs-util" ic "github.com/ipfs/go-libp2p/p2p/crypto" - b58 "gx/ipfs/QmT8rehPR3F6bmwL6zjUN8XpiDBFFpMP2myPdC6ApsWfJf/go-base58" - mh "gx/ipfs/QmYf7ng2hG5XBtJA3tN34DQ2GUN5HNksEw1rLDkmr6vGku/go-multihash" - u "gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util" - logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" // ID represents the identity of a peer. - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + logging "github.com/ipfs/go-log" // ID represents the identity of a peer. + b58 "github.com/jbenet/go-base58" + ma "github.com/jbenet/go-multiaddr" + mh "github.com/jbenet/go-multihash" ) var log = logging.Logger("peer") diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 013bcf2e02..93cbfc86c8 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -6,12 +6,12 @@ import ( "strings" "testing" + u "github.com/ipfs/go-ipfs-util" ic "github.com/ipfs/go-libp2p/p2p/crypto" . "github.com/ipfs/go-libp2p/p2p/peer" tu "github.com/ipfs/go-libp2p/testutil" - u "gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util" - b58 "gx/ipfs/QmT8rehPR3F6bmwL6zjUN8XpiDBFFpMP2myPdC6ApsWfJf/go-base58" + b58 "github.com/jbenet/go-base58" ) var gen1 keyset // generated diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 22fe410536..f8e88754e2 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -9,7 +9,7 @@ import ( //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 337a752978..09da0cb88a 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -6,7 +6,7 @@ import ( "sync" peer "github.com/ipfs/go-libp2p/p2p/peer" - ks "gx/ipfs/QmUusaX99BZoELh7dmPgirqRQ1FAmMnmnBn3oiqDFGBUSc/go-keyspace" + ks "github.com/whyrusleeping/go-keyspace" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 34179a5989..8d43bb690e 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -6,10 +6,10 @@ import ( "testing" "time" + u "github.com/ipfs/go-ipfs-util" peer "github.com/ipfs/go-libp2p/p2p/peer" - u "gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + context "golang.org/x/net/context" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index b679a4930d..fd49e00545 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -2,8 +2,8 @@ package queue import ( peer "github.com/ipfs/go-libp2p/p2p/peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" + logging "github.com/ipfs/go-log" + context "golang.org/x/net/context" ) var log = logging.Logger("peerqueue") From 80e965c50311e6ceec04daf93d29f3b649750b51 Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Fri, 15 Apr 2016 22:56:37 -0700 Subject: [PATCH 0077/3965] Use post-extraction import paths --- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peer.go | 2 +- p2p/host/peerstore/peer_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/queue/distance.go | 2 +- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 2 +- p2p/host/peerstore/queue/sync.go | 2 +- 8 files changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 84ae8ed6d7..40429f529c 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -6,7 +6,7 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p/p2p/peer" + peer "github.com/ipfs/go-libp2p-peer" testutil "github.com/ipfs/go-libp2p/testutil" ) diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go index d928b70425..f42f5b623f 100644 --- a/p2p/host/peerstore/peer.go +++ b/p2p/host/peerstore/peer.go @@ -8,7 +8,7 @@ import ( "strings" u "github.com/ipfs/go-ipfs-util" - ic "github.com/ipfs/go-libp2p/p2p/crypto" + ic "github.com/ipfs/go-libp2p-crypto" logging "github.com/ipfs/go-log" // ID represents the identity of a peer. b58 "github.com/jbenet/go-base58" ma "github.com/jbenet/go-multiaddr" diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 93cbfc86c8..0ad4838cdc 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -7,8 +7,8 @@ import ( "testing" u "github.com/ipfs/go-ipfs-util" - ic "github.com/ipfs/go-libp2p/p2p/crypto" - . "github.com/ipfs/go-libp2p/p2p/peer" + ic "github.com/ipfs/go-libp2p-crypto" + . "github.com/ipfs/go-libp2p-peer" tu "github.com/ipfs/go-libp2p/testutil" b58 "github.com/jbenet/go-base58" diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index f8e88754e2..f941eaf662 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -5,7 +5,7 @@ import ( "sync" "time" - ic "github.com/ipfs/go-libp2p/p2p/crypto" + ic "github.com/ipfs/go-libp2p-crypto" //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 09da0cb88a..7249c79c03 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,7 +5,7 @@ import ( "math/big" "sync" - peer "github.com/ipfs/go-libp2p/p2p/peer" + peer "github.com/ipfs/go-libp2p-peer" ks "github.com/whyrusleeping/go-keyspace" ) diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index 3c55e7e603..352d8077ca 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import peer "github.com/ipfs/go-libp2p/p2p/peer" +import peer "github.com/ipfs/go-libp2p-peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 8d43bb690e..2daa5bdaf8 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,7 +7,7 @@ import ( "time" u "github.com/ipfs/go-ipfs-util" - peer "github.com/ipfs/go-libp2p/p2p/peer" + peer "github.com/ipfs/go-libp2p-peer" context "golang.org/x/net/context" ) diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index fd49e00545..0ff94f7628 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -1,7 +1,7 @@ package queue import ( - peer "github.com/ipfs/go-libp2p/p2p/peer" + peer "github.com/ipfs/go-libp2p-peer" logging "github.com/ipfs/go-log" context "golang.org/x/net/context" ) From 84bb5e8ba76b238c62989c7bd0c276d274b2017d Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Fri, 15 Apr 2016 22:56:55 -0700 Subject: [PATCH 0078/3965] Resolve circular dependency by porting testutil functions over --- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peer_test.go | 2 +- p2p/host/peerstore/test/utils.go | 26 ++++++++++++++++++++++++++ 3 files changed, 28 insertions(+), 2 deletions(-) create mode 100644 p2p/host/peerstore/test/utils.go diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 40429f529c..4d9bf25d7a 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,7 +7,7 @@ import ( "time" peer "github.com/ipfs/go-libp2p-peer" - testutil "github.com/ipfs/go-libp2p/testutil" + testutil "github.com/ipfs/go-libp2p-peer/test" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go index 0ad4838cdc..ffef1eec56 100644 --- a/p2p/host/peerstore/peer_test.go +++ b/p2p/host/peerstore/peer_test.go @@ -9,7 +9,7 @@ import ( u "github.com/ipfs/go-ipfs-util" ic "github.com/ipfs/go-libp2p-crypto" . "github.com/ipfs/go-libp2p-peer" - tu "github.com/ipfs/go-libp2p/testutil" + tu "github.com/ipfs/go-libp2p-peer/test" b58 "github.com/jbenet/go-base58" ) diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go new file mode 100644 index 0000000000..c9ee61c41f --- /dev/null +++ b/p2p/host/peerstore/test/utils.go @@ -0,0 +1,26 @@ +package testutil + +import ( + "io" + + u "github.com/ipfs/go-ipfs-util" + ci "github.com/ipfs/go-libp2p-crypto" + peer "github.com/ipfs/go-libp2p-peer" +) + +func RandPeerID() (peer.ID, error) { + buf := make([]byte, 16) + if _, err := io.ReadFull(u.NewTimeSeededRand(), buf); err != nil { + return "", err + } + h := u.Hash(buf) + return peer.ID(h), nil +} + +func RandTestKeyPair(bits int) (ci.PrivKey, ci.PubKey, error) { + return ci.GenerateKeyPairWithReader(ci.RSA, bits, u.NewTimeSeededRand()) +} + +func SeededTestKeyPair(seed int64) (ci.PrivKey, ci.PubKey, error) { + return ci.GenerateKeyPairWithReader(ci.RSA, 512, u.NewSeededRand(seed)) +} From 91fc7552be37200af4149ad3849fb084335357a3 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 20 Apr 2016 10:20:46 -0700 Subject: [PATCH 0079/3965] improve code coverage a bit --- p2p/host/peerstore/addr_manager_test.go | 22 +++++++++++++++++++ p2p/host/peerstore/metrics_test.go | 29 +++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 0de0c6ab3e..52f621c606 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -88,6 +88,10 @@ func TestAddresses(t *testing.T) { m.ClearAddrs(id5) m.AddAddrs(id5, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, ttl) // clearing + if len(m.Peers()) != 5 { + t.Fatal("should have exactly two peers in the address book") + } + // test the Addresses return value testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) @@ -114,6 +118,10 @@ func TestAddressesExpire(t *testing.T) { m.AddAddr(id2, ma24, time.Hour) m.AddAddr(id2, ma25, time.Hour) + if len(m.Peers()) != 2 { + t.Fatal("should have exactly two peers in the address book") + } + testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) @@ -178,3 +186,17 @@ func TestClearWorks(t *testing.T) { testHas(t, nil, m.Addrs(id1)) testHas(t, nil, m.Addrs(id2)) } + +func TestSetNegativeTTLClears(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + + m := AddrManager{} + m.SetAddr(id1, ma11, time.Hour) + + testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) + + m.SetAddr(id1, ma11, -1) + + testHas(t, nil, m.Addrs(id1)) +} diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 4d9bf25d7a..bd10a1a533 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -2,6 +2,7 @@ package peer_test import ( "fmt" + "math" "math/rand" "testing" "time" @@ -38,3 +39,31 @@ func TestLatencyEWMAFun(t *testing.T) { } } } + +func TestLatencyEWMA(t *testing.T) { + m := peer.NewMetrics() + id, err := testutil.RandPeerID() + if err != nil { + t.Fatal(err) + } + + exp := 100.0 + mu := exp + sig := 10.0 + next := func() time.Duration { + mu = (rand.NormFloat64() * sig) + mu + return time.Duration(mu) + } + + for i := 0; i < 10; i++ { + select { + case <-time.After(200 * time.Millisecond): + m.RecordLatency(id, next()) + } + } + + lat := m.LatencyEWMA(id) + if math.Abs(exp-float64(lat)) > sig { + t.Fatal("latency outside of expected range: ", exp, lat, sig) + } +} From ccb540dce3882b16c344e980034edc13045112cd Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 20 Apr 2016 11:52:03 -0700 Subject: [PATCH 0080/3965] make metrics test a little less picky --- p2p/host/peerstore/metrics_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index bd10a1a533..08d53ac83f 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -51,7 +51,7 @@ func TestLatencyEWMA(t *testing.T) { mu := exp sig := 10.0 next := func() time.Duration { - mu = (rand.NormFloat64() * sig) + mu + mu := (rand.NormFloat64() * sig) + mu return time.Duration(mu) } From 995f55cbe056c1448adfd2117e4da4a439c76691 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 27 Apr 2016 10:12:51 -0700 Subject: [PATCH 0081/3965] recursive dependency update of utp lib --- p2p/net/swarm/addr/addr.go | 2 +- p2p/net/swarm/addr/addr_test.go | 2 +- p2p/net/swarm/dial_test.go | 4 ++-- p2p/net/swarm/peers_test.go | 2 +- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm.go | 6 +++--- p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_conn.go | 4 ++-- p2p/net/swarm/swarm_dial.go | 6 +++--- p2p/net/swarm/swarm_listen.go | 4 ++-- p2p/net/swarm/swarm_net.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 2 +- p2p/net/swarm/swarm_test.go | 2 +- 13 files changed, 20 insertions(+), 20 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 14da595090..7cda66439c 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,7 +3,7 @@ package addrutil import ( "fmt" - manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" + manet "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 16599b4089..8793bc0917 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,7 +3,7 @@ package addrutil import ( "testing" - manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" + manet "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index d21af292d8..31efd1b467 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -10,9 +10,9 @@ import ( addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" + manet "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 9976d3bf4c..6429a86278 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -3,7 +3,7 @@ package swarm import ( "testing" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index b0256b8489..b0c5497743 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -7,7 +7,7 @@ import ( "time" ci "github.com/ipfs/go-libp2p/testutil/ci" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 5326bf4a7f..5f75b5494a 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -14,19 +14,19 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" filter "github.com/ipfs/go-libp2p/p2p/net/filter" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - mafilter "gx/ipfs/QmPwfFAHUmvWDucLHRS9Xz2Kb1TNX2cY4LJ7pQjg9kVcae/multiaddr-filter" "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" goprocessctx "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/context" - transport "gx/ipfs/QmR4HhZm1fL8epXz8661Ru8ks5Y1Uk2u51quAWaCTD1zHg/go-libp2p-transport" + transport "gx/ipfs/QmRHqYZs3Diy8YC3bW16zvs8VDDwS2ARKBuKwALxEMqibc/go-libp2p-transport" pst "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer" psmss "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/multistream" spdy "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/spdystream" yamux "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/yamux" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + mafilter "gx/ipfs/Qme8dipKnZAChkp5Kfgj2MYYyBbzjqqPXmxQx3g9v3MoxP/multiaddr-filter" ) var log = logging.Logger("swarm2") diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 9cef367c78..9822ff7d44 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -6,7 +6,7 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" testutil "github.com/ipfs/go-libp2p/testutil" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 3ff2b91fcc..d815f7dc0e 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -6,10 +6,10 @@ import ( inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" + ic "gx/ipfs/QmUEUu1CM8bxBJxc3ZLojAi8evhTr4byQogWstABet79oY/go-libp2p-crypto" ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ic "gx/ipfs/QmaP38GJApheTr84f8R89vsT7oJLQw1AeCz4HqrQgv2njB/go-libp2p-crypto" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index b569f92fa4..c069543d51 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -10,9 +10,9 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - lgbl "gx/ipfs/QmSyBhZt2upyQ3NJmTpab1pX6hesA59vcYTGmgoDorZZbw/go-libp2p-loggables" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" - "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" + "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" + lgbl "gx/ipfs/QmYqiDbGUGqatbPFie11Py8cnyduoJYqcgKtjfhu1SQLh1/go-libp2p-loggables" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index cf0c3a3326..e17cda6e7f 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -6,8 +6,8 @@ import ( mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" - transport "gx/ipfs/QmR4HhZm1fL8epXz8661Ru8ks5Y1Uk2u51quAWaCTD1zHg/go-libp2p-transport" - lgbl "gx/ipfs/QmSyBhZt2upyQ3NJmTpab1pX6hesA59vcYTGmgoDorZZbw/go-libp2p-loggables" + transport "gx/ipfs/QmRHqYZs3Diy8YC3bW16zvs8VDDwS2ARKBuKwALxEMqibc/go-libp2p-transport" + lgbl "gx/ipfs/QmYqiDbGUGqatbPFie11Py8cnyduoJYqcgKtjfhu1SQLh1/go-libp2p-loggables" ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index d99e04e135..14a3a953d4 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -5,7 +5,7 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 29292eedbd..0230ac4ab7 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -5,7 +5,7 @@ import ( "time" inet "github.com/ipfs/go-libp2p/p2p/net" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 5d56d29fbd..c231e3b248 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -12,7 +12,7 @@ import ( metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/testutil" - peer "gx/ipfs/QmY1xNhBfF9xA1pmD8yejyQAyd77K68qNN6JPM1CN2eiRu/go-libp2p-peer" + peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" From b6ac01bf1fbe3f7bf402eb687d8d91235de0f7b1 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 27 Apr 2016 10:12:51 -0700 Subject: [PATCH 0082/3965] recursive dependency update of utp lib --- p2p/net/nat/nat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 0c7d86ca55..a3dd83c6d0 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -11,7 +11,7 @@ import ( nat "gx/ipfs/QmNLvkCDV6ZjUJsEwGNporYBuZdhWT6q7TBVYQwwRv12HT/go-nat" goprocess "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" periodic "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/periodic" - manet "gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net" + manet "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" notifier "gx/ipfs/QmbcS9XrwZkF1rZj8bBwwzoYhVuA2PCnPhFUL1pyWGgt2A/go-notifier" ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" From b58b106ecd09285efbb77c47722e1eae9537be53 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 27 Apr 2016 12:45:16 -0700 Subject: [PATCH 0083/3965] rewrite all package paths to dvcs --- p2p/net/swarm/addr/addr.go | 8 ++++---- p2p/net/swarm/addr/addr_test.go | 4 ++-- p2p/net/swarm/dial_test.go | 8 ++++---- p2p/net/swarm/peers_test.go | 6 +++--- p2p/net/swarm/simul_test.go | 6 +++--- p2p/net/swarm/swarm.go | 26 +++++++++++++------------- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 6 +++--- p2p/net/swarm/swarm_conn.go | 10 +++++----- p2p/net/swarm/swarm_dial.go | 10 +++++----- p2p/net/swarm/swarm_listen.go | 10 +++++----- p2p/net/swarm/swarm_net.go | 8 ++++---- p2p/net/swarm/swarm_net_test.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 6 +++--- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 6 +++--- 16 files changed, 60 insertions(+), 60 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 7cda66439c..1e22bae261 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,10 +3,10 @@ package addrutil import ( "fmt" - manet "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + logging "github.com/ipfs/go-log" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + context "golang.org/x/net/context" ) var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 8793bc0917..17039967e3 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -3,8 +3,8 @@ package addrutil import ( "testing" - manet "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" ) func newMultiaddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 31efd1b467..988f7f7f01 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -7,14 +7,14 @@ import ( "testing" "time" + peer "github.com/ipfs/go-libp2p-peer" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - manet "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + context "golang.org/x/net/context" ) func closeSwarms(swarms []*Swarm) { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 6429a86278..cacd88ece8 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -3,9 +3,9 @@ package swarm import ( "testing" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + peer "github.com/ipfs/go-libp2p-peer" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index b0c5497743..39ba6411b6 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -6,11 +6,11 @@ import ( "testing" "time" + peer "github.com/ipfs/go-libp2p-peer" ci "github.com/ipfs/go-libp2p/testutil/ci" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 5f75b5494a..c812f37565 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -8,25 +8,25 @@ import ( "sync" "time" + peer "github.com/ipfs/go-libp2p-peer" + transport "github.com/ipfs/go-libp2p-transport" metrics "github.com/ipfs/go-libp2p/p2p/metrics" mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" filter "github.com/ipfs/go-libp2p/p2p/net/filter" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" - goprocessctx "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/context" - transport "gx/ipfs/QmRHqYZs3Diy8YC3bW16zvs8VDDwS2ARKBuKwALxEMqibc/go-libp2p-transport" - pst "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer" - psmss "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/multistream" - spdy "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/spdystream" - yamux "gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/yamux" - ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" - mafilter "gx/ipfs/Qme8dipKnZAChkp5Kfgj2MYYyBbzjqqPXmxQx3g9v3MoxP/multiaddr-filter" + logging "github.com/ipfs/go-log" + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + pst "github.com/jbenet/go-stream-muxer" + psmss "github.com/jbenet/go-stream-muxer/multistream" + spdy "github.com/jbenet/go-stream-muxer/spdystream" + yamux "github.com/jbenet/go-stream-muxer/yamux" + "github.com/jbenet/goprocess" + goprocessctx "github.com/jbenet/goprocess/context" + mafilter "github.com/whyrusleeping/multiaddr-filter" + context "golang.org/x/net/context" ) var log = logging.Logger("swarm2") diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index faab0a3fdb..039683bfb9 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -4,7 +4,7 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 9822ff7d44..42ee299e87 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -3,13 +3,13 @@ package swarm import ( "testing" + peer "github.com/ipfs/go-libp2p-peer" metrics "github.com/ipfs/go-libp2p/p2p/metrics" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" testutil "github.com/ipfs/go-libp2p/testutil" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index d815f7dc0e..96700ba835 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -6,11 +6,11 @@ import ( inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" - ic "gx/ipfs/QmUEUu1CM8bxBJxc3ZLojAi8evhTr4byQogWstABet79oY/go-libp2p-crypto" - ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ic "github.com/ipfs/go-libp2p-crypto" + peer "github.com/ipfs/go-libp2p-peer" + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + context "golang.org/x/net/context" ) // a Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index c069543d51..aa4a3514b4 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -8,13 +8,13 @@ import ( "sync" "time" + lgbl "github.com/ipfs/go-libp2p-loggables" + peer "github.com/ipfs/go-libp2p-peer" conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" - lgbl "gx/ipfs/QmYqiDbGUGqatbPFie11Py8cnyduoJYqcgKtjfhu1SQLh1/go-libp2p-loggables" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + "github.com/jbenet/go-multiaddr-net" + context "golang.org/x/net/context" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index e17cda6e7f..38457f16fe 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -3,14 +3,14 @@ package swarm import ( "fmt" + lgbl "github.com/ipfs/go-libp2p-loggables" + transport "github.com/ipfs/go-libp2p-transport" mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" inet "github.com/ipfs/go-libp2p/p2p/net" conn "github.com/ipfs/go-libp2p/p2p/net/conn" - transport "gx/ipfs/QmRHqYZs3Diy8YC3bW16zvs8VDDwS2ARKBuKwALxEMqibc/go-libp2p-transport" - lgbl "gx/ipfs/QmYqiDbGUGqatbPFie11Py8cnyduoJYqcgKtjfhu1SQLh1/go-libp2p-loggables" - ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + ps "github.com/jbenet/go-peerstream" + context "golang.org/x/net/context" ) // Open listeners and reuse-dialers for the given addresses diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 14a3a953d4..84cc6326eb 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -3,13 +3,13 @@ package swarm import ( "fmt" + peer "github.com/ipfs/go-libp2p-peer" metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + "github.com/jbenet/goprocess" + context "golang.org/x/net/context" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 1ffb44812e..14c590b9d1 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -7,7 +7,7 @@ import ( inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/p2p/test/util" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" + context "golang.org/x/net/context" ) // TestConnectednessCorrect starts a few networks, connects a few diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 0230ac4ab7..aa4fca2448 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,10 +4,10 @@ import ( "testing" "time" + peer "github.com/ipfs/go-libp2p-peer" inet "github.com/ipfs/go-libp2p/p2p/net" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func TestNotifications(t *testing.T) { diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 3fbcc5d37c..7965d27439 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,7 +3,7 @@ package swarm import ( inet "github.com/ipfs/go-libp2p/p2p/net" - ps "gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream" + ps "github.com/jbenet/go-peerstream" ) // a Stream is a wrapper around a ps.Stream that exposes a way to get diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index c231e3b248..6e9f121fcd 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -9,13 +9,13 @@ import ( "testing" "time" + peer "github.com/ipfs/go-libp2p-peer" metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/testutil" - peer "gx/ipfs/QmZwZjMVGss5rqYsJVGy18gNbkTJffFyq2x1uJ4e4p3ZAt/go-libp2p-peer" - context "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" ) func EchoStreamHandler(stream inet.Stream) { From 7c74ee6d74c4c8bee43c2795d1cc67f0fc0c385e Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 27 Apr 2016 12:45:16 -0700 Subject: [PATCH 0084/3965] rewrite all package paths to dvcs --- p2p/net/nat/nat.go | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index a3dd83c6d0..237c1296ca 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -8,13 +8,13 @@ import ( "sync" "time" - nat "gx/ipfs/QmNLvkCDV6ZjUJsEwGNporYBuZdhWT6q7TBVYQwwRv12HT/go-nat" - goprocess "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" - periodic "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/periodic" - manet "gx/ipfs/QmTrxSBY8Wqd5aBB4MeizeSzS5xFbK8dQBrYaMsiGnCBhb/go-multiaddr-net" - logging "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" - notifier "gx/ipfs/QmbcS9XrwZkF1rZj8bBwwzoYhVuA2PCnPhFUL1pyWGgt2A/go-notifier" - ma "gx/ipfs/QmcobAGsCjYt5DXoq9et9L8yR8er7o7Cu3DTvpaq12jYSz/go-multiaddr" + nat "github.com/fd/go-nat" + logging "github.com/ipfs/go-log" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + goprocess "github.com/jbenet/goprocess" + periodic "github.com/jbenet/goprocess/periodic" + notifier "github.com/whyrusleeping/go-notifier" ) var ( From ea09f2754ca7b13af811e3a7f705b161fd93d07d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 2 May 2016 15:24:02 -0700 Subject: [PATCH 0085/3965] gxify and update deps --- p2p/muxer/muxer-multistream/multistream.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 94372e0c75..1c90485def 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -5,7 +5,7 @@ package multistream import ( "net" - mss "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/whyrusleeping/go-multistream" + mss "github.com/whyrusleeping/go-multistream" smux "github.com/jbenet/go-stream-muxer" multiplex "github.com/jbenet/go-stream-muxer/multiplex" From 4332c8da95e04a75e5b6269d0dee02884c90a301 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 2 May 2016 15:24:02 -0700 Subject: [PATCH 0086/3965] gxify and update deps --- p2p/muxer/yamux/yamux.go | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 0e13843878..a6abb6f043 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -5,8 +5,8 @@ import ( "net" "time" + yamux "github.com/hashicorp/yamux" smux "github.com/jbenet/go-stream-muxer" - yamux "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/hashicorp/yamux" ) // stream implements smux.Stream using a ss.Stream @@ -77,11 +77,12 @@ type Transport yamux.Config // DefaultTransport has default settings for yamux var DefaultTransport = (*Transport)(&yamux.Config{ - AcceptBacklog: 256, // from yamux.DefaultConfig - EnableKeepAlive: true, // from yamux.DefaultConfig - KeepAliveInterval: 30 * time.Second, // from yamux.DefaultConfig - MaxStreamWindowSize: uint32(256 * 1024), // from yamux.DefaultConfig - LogOutput: ioutil.Discard, + AcceptBacklog: 256, // from yamux.DefaultConfig + EnableKeepAlive: true, // from yamux.DefaultConfig + KeepAliveInterval: 30 * time.Second, // from yamux.DefaultConfig + ConnectionWriteTimeout: 10 * time.Second, // from yamux.DefaultConfig + MaxStreamWindowSize: uint32(256 * 1024), // from yamux.DefaultConfig + LogOutput: ioutil.Discard, }) func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { From 9b37fe1717575d35c95858c93b0827e93c5fda11 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 2 May 2016 15:24:02 -0700 Subject: [PATCH 0087/3965] gxify and update deps --- p2p/muxer/mplex/multiplex.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index d07d53e057..7d7f53aa3f 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -5,7 +5,7 @@ import ( "net" smux "github.com/jbenet/go-stream-muxer" - mp "github.com/jbenet/go-stream-muxer/Godeps/_workspace/src/github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. + mp "github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. ) var ErrUseServe = errors.New("not implemented, use Serve") From df16981dae12451cdae187be52074875b4f6bb89 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 4 May 2016 15:51:32 -0700 Subject: [PATCH 0088/3965] rewrite paths to gx versions --- p2p/muxer/muxer-multistream/multistream.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 1c90485def..57d30b0ef1 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -5,7 +5,7 @@ package multistream import ( "net" - mss "github.com/whyrusleeping/go-multistream" + mss "gx/ipfs/Qmf91yhgRLo2dhhbc5zZ7TxjMaR1oxaWaoc9zRZdi1kU4a/go-multistream" smux "github.com/jbenet/go-stream-muxer" multiplex "github.com/jbenet/go-stream-muxer/multiplex" From 3a964c3cbfc261ed2e9af715384f8efe7edb29f4 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 4 May 2016 15:51:32 -0700 Subject: [PATCH 0089/3965] rewrite paths to gx versions --- p2p/muxer/yamux/yamux.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index a6abb6f043..86d722be8e 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -5,8 +5,8 @@ import ( "net" "time" - yamux "github.com/hashicorp/yamux" smux "github.com/jbenet/go-stream-muxer" + yamux "gx/ipfs/QmT8nkh6VVJ2fWgAshstDkDWssAY1EXBhoraqGDfGeVx9Q/yamux" ) // stream implements smux.Stream using a ss.Stream From 9a45ffbf09421da6d704507a03a89eee11eb9e6b Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 4 May 2016 15:51:32 -0700 Subject: [PATCH 0090/3965] rewrite paths to gx versions --- p2p/muxer/mplex/multiplex.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 7d7f53aa3f..bb17f61470 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -5,7 +5,7 @@ import ( "net" smux "github.com/jbenet/go-stream-muxer" - mp "github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. + mp "gx/ipfs/QmSLbJgmwYvjQuMpfW6kUt5KsJeEkjLwXTkRM5DkqR14if/go-multiplex" // Conn is a connection to a remote peer. ) var ErrUseServe = errors.New("not implemented, use Serve") From 41807c04ac6264ee6d9bf8208618a32c78086959 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 5 May 2016 20:38:17 -0700 Subject: [PATCH 0091/3965] add env var for overriding muxer prefs --- p2p/net/swarm/swarm.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index c812f37565..d31c03a63f 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -5,6 +5,8 @@ package swarm import ( "fmt" "io/ioutil" + "os" + "strings" "sync" "time" @@ -48,6 +50,11 @@ func init() { msstpt.AddTransport("/yamux/1.0.0", ymxtpt) msstpt.AddTransport("/spdy/3.1.0", spdy.Transport) + // allow overriding of muxer preferences + if prefs := os.Getenv("LIBP2P_MUX_PREFS"); prefs != "" { + msstpt.OrderPreference = strings.Fields(prefs) + } + PSTransport = msstpt } From 416d561682bb6d428075a5d08a975407858e0761 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 12 May 2016 22:41:03 -0700 Subject: [PATCH 0092/3965] WIP --- p2p/muxer/muxer-multistream/multistream.go | 22 ------------------- .../muxer-multistream/multistream_test.go | 5 ++++- 2 files changed, 4 insertions(+), 23 deletions(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 57d30b0ef1..4e4682e0d4 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -8,9 +8,6 @@ import ( mss "gx/ipfs/Qmf91yhgRLo2dhhbc5zZ7TxjMaR1oxaWaoc9zRZdi1kU4a/go-multistream" smux "github.com/jbenet/go-stream-muxer" - multiplex "github.com/jbenet/go-stream-muxer/multiplex" - spdy "github.com/jbenet/go-stream-muxer/spdystream" - yamux "github.com/jbenet/go-stream-muxer/yamux" ) type Transport struct { @@ -21,25 +18,6 @@ type Transport struct { OrderPreference []string } -func NewTransport() *Transport { - mux := mss.NewMultistreamMuxer() - mux.AddHandler("/multiplex", nil) - mux.AddHandler("/spdystream", nil) - mux.AddHandler("/yamux", nil) - - tpts := map[string]smux.Transport{ - "/multiplex": multiplex.DefaultTransport, - "/spdystream": spdy.Transport, - "/yamux": yamux.DefaultTransport, - } - - return &Transport{ - mux: mux, - tpts: tpts, - OrderPreference: []string{"/yamux", "/spdystream", "/multiplex"}, - } -} - func NewBlankTransport() *Transport { return &Transport{ mux: mss.NewMultistreamMuxer(), diff --git a/p2p/muxer/muxer-multistream/multistream_test.go b/p2p/muxer/muxer-multistream/multistream_test.go index de1a9e4cd8..f13a909ccc 100644 --- a/p2p/muxer/muxer-multistream/multistream_test.go +++ b/p2p/muxer/muxer-multistream/multistream_test.go @@ -3,9 +3,12 @@ package multistream import ( "testing" + spdy "github.com/jbenet/go-stream-muxer/spdystream" test "github.com/jbenet/go-stream-muxer/test" ) func TestMultiStreamTransport(t *testing.T) { - test.SubtestAll(t, NewTransport()) + tpt := NewBlankTransport() + tpt.AddTransport("/spdy", spdy.Transport) + test.SubtestAll(t, tpt) } From 4d9e0bd068c8ea0b12f5f836f566149e3b8f7598 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 12 May 2016 23:09:34 -0700 Subject: [PATCH 0093/3965] extract from go-stream-muxer --- p2p/muxer/muxer-multistream/multistream.go | 1 - p2p/muxer/muxer-multistream/multistream_test.go | 14 -------------- 2 files changed, 15 deletions(-) delete mode 100644 p2p/muxer/muxer-multistream/multistream_test.go diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 4e4682e0d4..8ac74235c0 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -40,7 +40,6 @@ func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { } proto = selected } else { - // prefer yamux selected, err := mss.SelectOneOf(t.OrderPreference, nc) if err != nil { return nil, err diff --git a/p2p/muxer/muxer-multistream/multistream_test.go b/p2p/muxer/muxer-multistream/multistream_test.go deleted file mode 100644 index f13a909ccc..0000000000 --- a/p2p/muxer/muxer-multistream/multistream_test.go +++ /dev/null @@ -1,14 +0,0 @@ -package multistream - -import ( - "testing" - - spdy "github.com/jbenet/go-stream-muxer/spdystream" - test "github.com/jbenet/go-stream-muxer/test" -) - -func TestMultiStreamTransport(t *testing.T) { - tpt := NewBlankTransport() - tpt.AddTransport("/spdy", spdy.Transport) - test.SubtestAll(t, tpt) -} From c6d96a9aa0650b351f063836abb0b98b6d29e593 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 13 May 2016 09:15:25 -0700 Subject: [PATCH 0094/3965] gx publish version 1.0.0 --- p2p/muxer/yamux/yamux.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 86d722be8e..a6abb6f043 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -5,8 +5,8 @@ import ( "net" "time" + yamux "github.com/hashicorp/yamux" smux "github.com/jbenet/go-stream-muxer" - yamux "gx/ipfs/QmT8nkh6VVJ2fWgAshstDkDWssAY1EXBhoraqGDfGeVx9Q/yamux" ) // stream implements smux.Stream using a ss.Stream From b3902adadb872a12e3bbf47cbc5e364330882a93 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 13 May 2016 09:16:00 -0700 Subject: [PATCH 0095/3965] gx publish version 1.0.0 --- p2p/muxer/mplex/multiplex.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index bb17f61470..7d7f53aa3f 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -5,7 +5,7 @@ import ( "net" smux "github.com/jbenet/go-stream-muxer" - mp "gx/ipfs/QmSLbJgmwYvjQuMpfW6kUt5KsJeEkjLwXTkRM5DkqR14if/go-multiplex" // Conn is a connection to a remote peer. + mp "github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. ) var ErrUseServe = errors.New("not implemented, use Serve") From c06505549e461b62a4476db52a7a8ce77f5cac96 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 13 May 2016 09:18:43 -0700 Subject: [PATCH 0096/3965] gx publish version 1.0.0 --- p2p/muxer/muxer-multistream/multistream.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 8ac74235c0..94b2320054 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -5,7 +5,7 @@ package multistream import ( "net" - mss "gx/ipfs/Qmf91yhgRLo2dhhbc5zZ7TxjMaR1oxaWaoc9zRZdi1kU4a/go-multistream" + mss "github.com/whyrusleeping/go-multistream" smux "github.com/jbenet/go-stream-muxer" ) From 936b1571c3f622092870c0d12d4b560284694b16 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 13 May 2016 10:59:16 -0700 Subject: [PATCH 0097/3965] update muxers, includes yamux deadlock fix --- p2p/net/swarm/swarm.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index d31c03a63f..e8b53885cc 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -22,11 +22,11 @@ import ( ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" pst "github.com/jbenet/go-stream-muxer" - psmss "github.com/jbenet/go-stream-muxer/multistream" - spdy "github.com/jbenet/go-stream-muxer/spdystream" - yamux "github.com/jbenet/go-stream-muxer/yamux" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" + psmss "github.com/whyrusleeping/go-smux-multistream" + spdy "github.com/whyrusleeping/go-smux-spdystream" + yamux "github.com/whyrusleeping/go-smux-yamux" mafilter "github.com/whyrusleeping/multiaddr-filter" context "golang.org/x/net/context" ) @@ -39,11 +39,11 @@ func init() { msstpt := psmss.NewBlankTransport() ymxtpt := &yamux.Transport{ - AcceptBacklog: 8192, + AcceptBacklog: 1024, ConnectionWriteTimeout: time.Second * 10, KeepAliveInterval: time.Second * 30, EnableKeepAlive: true, - MaxStreamWindowSize: uint32(1024 * 256), + MaxStreamWindowSize: uint32(1024 * 512), LogOutput: ioutil.Discard, } From c6d47eea103ed4c0cb592f799f1ae52b88df5ef6 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 15 May 2016 18:57:29 -0700 Subject: [PATCH 0098/3965] yamux patches to help mitigate hanging issue --- p2p/net/swarm/swarm.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index e8b53885cc..74e43dde44 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -39,7 +39,7 @@ func init() { msstpt := psmss.NewBlankTransport() ymxtpt := &yamux.Transport{ - AcceptBacklog: 1024, + AcceptBacklog: 8192, ConnectionWriteTimeout: time.Second * 10, KeepAliveInterval: time.Second * 30, EnableKeepAlive: true, From f928ecce2fde6eae83d2e30f9e589e86c4cc8cab Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 16 May 2016 12:29:11 -0700 Subject: [PATCH 0099/3965] first working commit --- p2p/transport/websocket/websocket.go | 214 +++++++++++++++++++++++++++ p2p/transport/websocket/ws_test.go | 65 ++++++++ 2 files changed, 279 insertions(+) create mode 100644 p2p/transport/websocket/websocket.go create mode 100644 p2p/transport/websocket/ws_test.go diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go new file mode 100644 index 0000000000..ba59fc290d --- /dev/null +++ b/p2p/transport/websocket/websocket.go @@ -0,0 +1,214 @@ +package websocket + +import ( + "fmt" + "log" + "net" + "net/http" + "net/url" + + tpt "github.com/ipfs/go-libp2p-transport" + manet "github.com/jbenet/go-multiaddr-net" + mafmt "github.com/whyrusleeping/mafmt" + ws "golang.org/x/net/websocket" + + ma "github.com/jbenet/go-multiaddr" + "golang.org/x/net/context" +) + +var WsProtocol = ma.Protocol{ + Code: 477, + Name: "ws", + VCode: ma.CodeToVarint(477), +} + +var WsFmt = mafmt.And(mafmt.TCP, mafmt.Base(WsProtocol.Code)) + +var WsCodec = &manet.NetCodec{ + NetAddrNetworks: []string{"websocket"}, + ProtocolName: "ws", + ConvertMultiaddr: ConvertWebsocketMultiaddrToNetAddr, + ParseNetAddr: ParseWebsocketNetAddr, +} + +func init() { + err := ma.AddProtocol(WsProtocol) + if err != nil { + log.Fatalf("error registering websocket protocol: %s", err) + } + + manet.RegisterNetCodec(WsCodec) +} + +func ConvertWebsocketMultiaddrToNetAddr(maddr ma.Multiaddr) (net.Addr, error) { + panic("not yet") +} + +func ParseWebsocketNetAddr(a net.Addr) (ma.Multiaddr, error) { + wsa, ok := a.(*ws.Addr) + if !ok { + return nil, fmt.Errorf("not a websocket address") + } + + tcpaddr, err := net.ResolveTCPAddr("tcp", wsa.Host) + if err != nil { + return nil, err + } + + tcpma, err := manet.FromNetAddr(tcpaddr) + if err != nil { + return nil, err + } + + wsma, err := ma.NewMultiaddr("/ws") + if err != nil { + return nil, err + } + + OUT := tcpma.Encapsulate(wsma) + return OUT, nil +} + +type WebsocketTransport struct{} + +func (t *WebsocketTransport) Matches(a ma.Multiaddr) bool { + return WsFmt.Matches(a) +} + +func (t *WebsocketTransport) Dialer(_ ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { + return &dialer{}, nil +} + +type dialer struct{} + +func parseMultiaddr(a ma.Multiaddr) (string, error) { + _, host, err := manet.DialArgs(a) + if err != nil { + return "", err + } + + return "ws://" + host, nil +} + +func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { + wsurl, err := parseMultiaddr(raddr) + if err != nil { + log.Println("parse multiaddr failed: ", err) + return nil, err + } + + wscon, err := ws.Dial(wsurl, "", "http://127.0.0.1:0/") + if err != nil { + return nil, err + } + + mnc, err := manet.WrapNetConn(wscon) + if err != nil { + return nil, err + } + + return &wsConn{ + Conn: mnc, + }, nil +} + +func (d *dialer) Matches(a ma.Multiaddr) bool { + return WsFmt.Matches(a) +} + +type wsConn struct { + manet.Conn + t tpt.Transport +} + +func (c *wsConn) Transport() tpt.Transport { + return c.t +} + +type listener struct { + manet.Listener + + incoming chan *conn + + tpt tpt.Transport +} + +type conn struct { + *ws.Conn + + done func() +} + +func (c *conn) Close() error { + c.done() + return c.Conn.Close() +} + +func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { + list, err := manet.Listen(a) + if err != nil { + return nil, err + } + + tlist := t.wrapListener(list) + + u, err := url.Parse("ws://" + list.Addr().String()) + if err != nil { + return nil, err + } + + s := &ws.Server{ + Handler: tlist.handleWsConn, + Config: ws.Config{Origin: u}, + } + + go http.Serve(list.NetListener(), s) + + return tlist, nil +} + +func (t *WebsocketTransport) wrapListener(l manet.Listener) *listener { + return &listener{ + Listener: l, + incoming: make(chan *conn), + tpt: t, + } +} + +func (l *listener) handleWsConn(s *ws.Conn) { + ctx, cancel := context.WithCancel(context.Background()) + + l.incoming <- &conn{ + Conn: s, + done: cancel, + } + + // wait until conn gets closed, otherwise the handler closes it early + <-ctx.Done() +} + +func (l *listener) Accept() (tpt.Conn, error) { + c, ok := <-l.incoming + if !ok { + return nil, fmt.Errorf("listener is closed") + } + + mnc, err := manet.WrapNetConn(c) + if err != nil { + return nil, err + } + + return &wsConn{ + Conn: mnc, + t: l.tpt, + }, nil +} + +func (l *listener) Multiaddr() ma.Multiaddr { + wsma, err := ma.NewMultiaddr("/ws") + if err != nil { + panic(err) + } + + return l.Listener.Multiaddr().Encapsulate(wsma) +} diff --git a/p2p/transport/websocket/ws_test.go b/p2p/transport/websocket/ws_test.go new file mode 100644 index 0000000000..5bda992d17 --- /dev/null +++ b/p2p/transport/websocket/ws_test.go @@ -0,0 +1,65 @@ +package websocket + +import ( + "log" + "testing" + "time" + + ma "github.com/jbenet/go-multiaddr" +) + +var _ = log.Println + +func TestMultiaddrParsing(t *testing.T) { + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") + if err != nil { + t.Fatal(err) + } + + _, err = parseMultiaddr(addr) + if err != nil { + t.Fatal(err) + } +} + +func TestWebsocketListen(t *testing.T) { + zero, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0/ws") + if err != nil { + t.Fatal(err) + } + + tpt := &WebsocketTransport{} + l, err := tpt.Listen(zero) + if err != nil { + t.Fatal(err) + } + + go func() { + d, _ := tpt.Dialer(nil) + c, err := d.Dial(l.Multiaddr()) + if err != nil { + t.Error(err) + return + } + + c.Write([]byte("HELLO WORLD!")) + time.Sleep(time.Second) + c.Close() + }() + + c, err := l.Accept() + if err != nil { + t.Fatal(err) + } + + buf := make([]byte, 32) + n, err := c.Read(buf) + if err != nil { + t.Fatal(err) + } + + log.Printf("READ: %s", buf[:n]) + c.Close() + l.Close() + time.Sleep(time.Second) +} From 83015ba5dcda52b6e4b265ecd422b587ea97445f Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 16 May 2016 15:17:21 -0700 Subject: [PATCH 0100/3965] complete parsers --- p2p/transport/websocket/websocket.go | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index ba59fc290d..e37cb98a85 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -41,7 +41,17 @@ func init() { } func ConvertWebsocketMultiaddrToNetAddr(maddr ma.Multiaddr) (net.Addr, error) { - panic("not yet") + _, host, err := manet.DialArgs(maddr) + if err != nil { + return nil, err + } + + a := &ws.Addr{ + URL: &url.URL{ + Host: host, + }, + } + return a, nil } func ParseWebsocketNetAddr(a net.Addr) (ma.Multiaddr, error) { @@ -65,8 +75,7 @@ func ParseWebsocketNetAddr(a net.Addr) (ma.Multiaddr, error) { return nil, err } - OUT := tcpma.Encapsulate(wsma) - return OUT, nil + return tcpma.Encapsulate(wsma), nil } type WebsocketTransport struct{} From 2eaeaff3d6b768b41a94c753f96006268c3b46f6 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 16 May 2016 17:04:08 -0700 Subject: [PATCH 0101/3965] don't panic on being passed nil addresses --- p2p/host/peerstore/addr_manager.go | 9 +++++++++ p2p/host/peerstore/addr_manager_test.go | 7 +++++++ 2 files changed, 16 insertions(+) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 87b8206eee..964cc39bda 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -104,6 +104,11 @@ func (mgr *AddrManager) AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) // only expand ttls exp := time.Now().Add(ttl) for _, addr := range addrs { + if addr == nil { + log.Warningf("was passed nil multiaddr for %s", p) + continue + } + addrstr := addr.String() a, found := amap[addrstr] if !found || exp.After(a.TTL) { @@ -134,6 +139,10 @@ func (mgr *AddrManager) SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) exp := time.Now().Add(ttl) for _, addr := range addrs { + if addr == nil { + log.Warningf("was passed nil multiaddr for %s", p) + continue + } // re-set all of them for new ttl. addrs := addr.String() diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 52f621c606..1d6f6b69f8 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -200,3 +200,10 @@ func TestSetNegativeTTLClears(t *testing.T) { testHas(t, nil, m.Addrs(id1)) } + +func TestNilAddrsDontBreak(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + m := AddrManager{} + m.SetAddr(id1, nil, time.Hour) + m.AddAddr(id1, nil, time.Hour) +} From 5f44cd0a9c3a1b932a2e39d82c6f21818f0b777e Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 16 May 2016 21:23:25 -0700 Subject: [PATCH 0102/3965] don't use string method for key of maps --- p2p/host/peerstore/addr_manager.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 964cc39bda..c06771c0fb 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -109,7 +109,7 @@ func (mgr *AddrManager) AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) continue } - addrstr := addr.String() + addrstr := string(addr.Bytes()) a, found := amap[addrstr] if !found || exp.After(a.TTL) { amap[addrstr] = expiringAddr{Addr: addr, TTL: exp} @@ -144,7 +144,7 @@ func (mgr *AddrManager) SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) continue } // re-set all of them for new ttl. - addrs := addr.String() + addrs := string(addr.Bytes()) if ttl > 0 { amap[addrs] = expiringAddr{Addr: addr, TTL: exp} From eebf64993c15867806ea601d02b5bd45c4895332 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 14 Apr 2016 16:43:42 -0700 Subject: [PATCH 0103/3965] Refactor the swarm dialer --- p2p/net/swarm/addr/addr.go | 16 +- p2p/net/swarm/addr/filter.go | 27 ++++ p2p/net/swarm/dial_test.go | 36 ----- p2p/net/swarm/limiter.go | 136 +++++++++++++++++ p2p/net/swarm/limiter_test.go | 265 ++++++++++++++++++++++++++++++++++ p2p/net/swarm/swarm.go | 5 + p2p/net/swarm/swarm_dial.go | 253 +++++++++----------------------- p2p/net/swarm/swarm_test.go | 2 +- 8 files changed, 516 insertions(+), 224 deletions(-) create mode 100644 p2p/net/swarm/addr/filter.go create mode 100644 p2p/net/swarm/limiter.go create mode 100644 p2p/net/swarm/limiter_test.go diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 1e22bae261..8c0c94b2d6 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -42,10 +42,14 @@ func init() { // FilterAddrs is a filter that removes certain addresses, according to filter. // if filter returns true, the address is kept. -func FilterAddrs(a []ma.Multiaddr, filter func(ma.Multiaddr) bool) []ma.Multiaddr { +func FilterAddrs(a []ma.Multiaddr, filters ...func(ma.Multiaddr) bool) []ma.Multiaddr { b := make([]ma.Multiaddr, 0, len(a)) for _, addr := range a { - if filter(addr) { + good := true + for _, filter := range filters { + good = good && filter(addr) + } + if good { b = append(b, addr) } } @@ -56,9 +60,11 @@ func FilterAddrs(a []ma.Multiaddr, filter func(ma.Multiaddr) bool) []ma.Multiadd // from a list. the addresses removed are those known NOT // to work with our network. Namely, addresses with UTP. func FilterUsableAddrs(a []ma.Multiaddr) []ma.Multiaddr { - return FilterAddrs(a, func(m ma.Multiaddr) bool { - return AddrUsable(m, false) - }) + return FilterAddrs(a, AddrUsableFunc) +} + +func AddrUsableFunc(m ma.Multiaddr) bool { + return AddrUsable(m, false) } // AddrOverNonLocalIP returns whether the addr uses a non-local ip link diff --git a/p2p/net/swarm/addr/filter.go b/p2p/net/swarm/addr/filter.go new file mode 100644 index 0000000000..c67e1c624e --- /dev/null +++ b/p2p/net/swarm/addr/filter.go @@ -0,0 +1,27 @@ +package addrutil + +import ( + ma "github.com/jbenet/go-multiaddr" + mafmt "github.com/whyrusleeping/mafmt" +) + +func SubtractFilter(addrs ...ma.Multiaddr) func(ma.Multiaddr) bool { + addrmap := make(map[string]bool) + for _, a := range addrs { + addrmap[string(a.Bytes())] = true + } + + return func(a ma.Multiaddr) bool { + return !addrmap[string(a.Bytes())] + } +} + +func IsFDCostlyTransport(a ma.Multiaddr) bool { + return mafmt.TCP.Matches(a) +} + +func FilterNeg(f func(ma.Multiaddr) bool) func(ma.Multiaddr) bool { + return func(a ma.Multiaddr) bool { + return !f(a) + } +} diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 988f7f7f01..aea0f728cc 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -2,7 +2,6 @@ package swarm import ( "net" - "sort" "sync" "testing" "time" @@ -493,38 +492,3 @@ func TestDialBackoffClears(t *testing.T) { t.Log("correctly cleared backoff") } } - -func mkAddr(t *testing.T, s string) ma.Multiaddr { - a, err := ma.NewMultiaddr(s) - if err != nil { - t.Fatal(err) - } - - return a -} - -func TestAddressSorting(t *testing.T) { - u1 := mkAddr(t, "/ip4/152.12.23.53/udp/1234/utp") - u2l := mkAddr(t, "/ip4/127.0.0.1/udp/1234/utp") - local := mkAddr(t, "/ip4/127.0.0.1/tcp/1234") - norm := mkAddr(t, "/ip4/6.5.4.3/tcp/1234") - - l := AddrList{local, u1, u2l, norm} - sort.Sort(l) - - if !l[0].Equal(u2l) { - t.Fatal("expected utp local addr to be sorted first: ", l[0]) - } - - if !l[1].Equal(u1) { - t.Fatal("expected utp addr to be sorted second") - } - - if !l[2].Equal(local) { - t.Fatal("expected tcp localhost addr thid") - } - - if !l[3].Equal(norm) { - t.Fatal("expected normal addr last") - } -} diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go new file mode 100644 index 0000000000..fe8162d7f1 --- /dev/null +++ b/p2p/net/swarm/limiter.go @@ -0,0 +1,136 @@ +package swarm + +import ( + "sync" + + peer "github.com/ipfs/go-libp2p-peer" + ma "github.com/jbenet/go-multiaddr" + context "golang.org/x/net/context" + + conn "github.com/ipfs/go-libp2p/p2p/net/conn" + addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" +) + +type dialResult struct { + Conn conn.Conn + Err error +} + +type dialJob struct { + addr ma.Multiaddr + peer peer.ID + ctx context.Context + resp chan dialResult + success bool +} + +type dialLimiter struct { + rllock sync.Mutex + fdConsuming int + fdLimit int + waitingOnFd []*dialJob + + dialFunc func(context.Context, peer.ID, ma.Multiaddr) (conn.Conn, error) + + activePerPeer map[peer.ID]int + perPeerLimit int + waitingOnPeerLimit map[peer.ID][]*dialJob +} + +type dialfunc func(context.Context, peer.ID, ma.Multiaddr) (conn.Conn, error) + +func newDialLimiter(df dialfunc) *dialLimiter { + return newDialLimiterWithParams(df, concurrentFdDials, defaultPerPeerRateLimit) +} + +func newDialLimiterWithParams(df dialfunc, fdl, ppl int) *dialLimiter { + return &dialLimiter{ + fdLimit: fdl, + perPeerLimit: ppl, + waitingOnPeerLimit: make(map[peer.ID][]*dialJob), + activePerPeer: make(map[peer.ID]int), + dialFunc: df, + } +} + +func (dl *dialLimiter) finishedDial(dj *dialJob) { + dl.rllock.Lock() + defer dl.rllock.Unlock() + + // release tokens in reverse order than we take them + dl.activePerPeer[dj.peer]-- + if dl.activePerPeer[dj.peer] == 0 { + delete(dl.activePerPeer, dj.peer) + } + + waitlist := dl.waitingOnPeerLimit[dj.peer] + if !dj.success && len(waitlist) > 0 { + next := waitlist[0] + if len(waitlist) == 1 { + delete(dl.waitingOnPeerLimit, dj.peer) + } else { + dl.waitingOnPeerLimit[dj.peer] = waitlist[1:] + } + dl.activePerPeer[dj.peer]++ // just kidding, we still want this token + + // can kick this off right here, dials in this list already + // have the other tokens needed + go dl.executeDial(next) + } + + if addrutil.IsFDCostlyTransport(dj.addr) { + dl.fdConsuming-- + if len(dl.waitingOnFd) > 0 { + next := dl.waitingOnFd[0] + dl.waitingOnFd = dl.waitingOnFd[1:] + dl.fdConsuming++ + + // now, attempt to take the 'per peer limit' token + dl.schedulePerPeerDial(next) + } + } +} + +// AddDialJob tries to take the needed tokens for starting the given dial job. +// If it acquires all needed tokens, it immediately starts the dial, otherwise +// it will put it on the waitlist for the requested token. +func (dl *dialLimiter) AddDialJob(dj *dialJob) { + dl.rllock.Lock() + defer dl.rllock.Unlock() + + if addrutil.IsFDCostlyTransport(dj.addr) { + if dl.fdConsuming >= dl.fdLimit { + dl.waitingOnFd = append(dl.waitingOnFd, dj) + return + } + + // take token + dl.fdConsuming++ + } + + dl.schedulePerPeerDial(dj) +} + +// executeDial calls the dialFunc, and reports the result through the response +// channel when finished. Once the response is sent it also releases all tokens +// it held during the dial. +func (dl *dialLimiter) executeDial(j *dialJob) { + defer dl.finishedDial(j) + con, err := dl.dialFunc(j.ctx, j.peer, j.addr) + select { + case j.resp <- dialResult{Conn: con, Err: err}: + case <-j.ctx.Done(): + } +} + +func (dl *dialLimiter) schedulePerPeerDial(j *dialJob) { + if dl.activePerPeer[j.peer] >= dl.perPeerLimit { + wlist := dl.waitingOnPeerLimit[j.peer] + dl.waitingOnPeerLimit[j.peer] = append(wlist, j) + return + } + + // take second needed token and start dial! + dl.activePerPeer[j.peer]++ + go dl.executeDial(j) +} diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go new file mode 100644 index 0000000000..b5bbde847e --- /dev/null +++ b/p2p/net/swarm/limiter_test.go @@ -0,0 +1,265 @@ +package swarm + +import ( + "fmt" + "strconv" + "testing" + "time" + + peer "github.com/ipfs/go-libp2p-peer" + ma "github.com/jbenet/go-multiaddr" + mafmt "github.com/whyrusleeping/mafmt" + context "golang.org/x/net/context" + + conn "github.com/ipfs/go-libp2p/p2p/net/conn" +) + +func mustAddr(t *testing.T, s string) ma.Multiaddr { + a, err := ma.NewMultiaddr(s) + if err != nil { + t.Fatal(err) + } + return a +} + +func addrWithPort(t *testing.T, p int) ma.Multiaddr { + return mustAddr(t, fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", p)) +} + +// in these tests I use addresses with tcp ports over a certain number to +// signify 'good' addresses that will succeed, and addresses below that number +// will fail. This lets us more easily test these different scenarios. +func tcpPortOver(a ma.Multiaddr, n int) bool { + port, err := a.ValueForProtocol(ma.P_TCP) + if err != nil { + panic(err) + } + + pnum, err := strconv.Atoi(port) + if err != nil { + panic(err) + } + + return pnum > n +} + +func tryDialAddrs(ctx context.Context, l *dialLimiter, p peer.ID, addrs []ma.Multiaddr, res chan dialResult) { + for _, a := range addrs { + l.AddDialJob(&dialJob{ + ctx: ctx, + peer: p, + addr: a, + resp: res, + }) + } +} + +func hangDialFunc(hang chan struct{}) dialfunc { + return func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { + if mafmt.UTP.Matches(a) { + return conn.Conn(nil), nil + } + + if tcpPortOver(a, 10) { + return conn.Conn(nil), nil + } else { + <-hang + return nil, fmt.Errorf("test bad dial") + } + } +} + +func TestLimiterBasicDials(t *testing.T) { + hang := make(chan struct{}) + defer close(hang) + + l := newDialLimiterWithParams(hangDialFunc(hang), concurrentFdDials, 4) + + bads := []ma.Multiaddr{ + addrWithPort(t, 1), + addrWithPort(t, 2), + addrWithPort(t, 3), + addrWithPort(t, 4), + } + + good := addrWithPort(t, 20) + + resch := make(chan dialResult) + pid := peer.ID("testpeer") + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + tryDialAddrs(ctx, l, pid, bads, resch) + + l.AddDialJob(&dialJob{ + ctx: ctx, + peer: pid, + addr: good, + resp: resch, + }) + + select { + case <-resch: + t.Fatal("no dials should have completed!") + case <-time.After(time.Millisecond * 100): + } + + // complete a single hung dial + hang <- struct{}{} + + select { + case r := <-resch: + if r.Err == nil { + t.Fatal("should have gotten failed dial result") + } + case <-time.After(time.Second): + t.Fatal("timed out waiting for dial completion") + } + + select { + case r := <-resch: + if r.Err != nil { + t.Fatal("expected second result to be success!") + } + case <-time.After(time.Second): + } +} + +func TestFDLimiting(t *testing.T) { + hang := make(chan struct{}) + defer close(hang) + l := newDialLimiterWithParams(hangDialFunc(hang), 16, 5) + + bads := []ma.Multiaddr{addrWithPort(t, 1), addrWithPort(t, 2), addrWithPort(t, 3), addrWithPort(t, 4)} + pids := []peer.ID{"testpeer1", "testpeer2", "testpeer3", "testpeer4"} + good_tcp := addrWithPort(t, 20) + + ctx := context.Background() + resch := make(chan dialResult) + + // take all fd limit tokens with hang dials + for _, pid := range pids { + tryDialAddrs(ctx, l, pid, bads, resch) + } + + // these dials should work normally, but will hang because we have taken + // up all the fd limiting + for _, pid := range pids { + l.AddDialJob(&dialJob{ + ctx: ctx, + peer: pid, + addr: good_tcp, + resp: resch, + }) + } + + select { + case <-resch: + t.Fatal("no dials should have completed!") + case <-time.After(time.Millisecond * 100): + } + + pid5 := peer.ID("testpeer5") + utpaddr := mustAddr(t, "/ip4/127.0.0.1/udp/7777/utp") + + l.AddDialJob(&dialJob{ctx: ctx, peer: pid5, addr: utpaddr, resp: resch}) + + select { + case res := <-resch: + if res.Err != nil { + t.Fatal("should have gotten successful response") + } + case <-time.After(time.Second * 5): + t.Fatal("timeout waiting for utp addr success") + } +} + +func TestTokenRedistribution(t *testing.T) { + hangchs := make(map[peer.ID]chan struct{}) + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { + if tcpPortOver(a, 10) { + return (conn.Conn)(nil), nil + } else { + <-hangchs[p] + return nil, fmt.Errorf("test bad dial") + } + } + l := newDialLimiterWithParams(df, 8, 4) + + bads := []ma.Multiaddr{addrWithPort(t, 1), addrWithPort(t, 2), addrWithPort(t, 3), addrWithPort(t, 4)} + pids := []peer.ID{"testpeer1", "testpeer2"} + + ctx := context.Background() + resch := make(chan dialResult) + + // take all fd limit tokens with hang dials + for _, pid := range pids { + hangchs[pid] = make(chan struct{}) + tryDialAddrs(ctx, l, pid, bads, resch) + } + + good := mustAddr(t, "/ip4/127.0.0.1/tcp/1001") + + // add a good dial job for peer 1 + l.AddDialJob(&dialJob{ + ctx: ctx, + peer: pids[1], + addr: good, + resp: resch, + }) + + select { + case <-resch: + t.Fatal("no dials should have completed!") + case <-time.After(time.Millisecond * 100): + } + + // unblock one dial for peer 0 + hangchs[pids[0]] <- struct{}{} + + select { + case res := <-resch: + if res.Err == nil { + t.Fatal("should have only been a failure here") + } + case <-time.After(time.Millisecond * 100): + t.Fatal("expected a dial failure here") + } + + select { + case <-resch: + t.Fatal("no more dials should have completed!") + case <-time.After(time.Millisecond * 100): + } + + // add a bad dial job to peer 0 to fill their rate limiter + // and test that more dials for this peer won't interfere with peer 1's successful dial incoming + l.AddDialJob(&dialJob{ + ctx: ctx, + peer: pids[0], + addr: addrWithPort(t, 7), + resp: resch, + }) + + hangchs[pids[1]] <- struct{}{} + + // now one failed dial from peer 1 should get through and fail + // which will in turn unblock the successful dial on peer 1 + select { + case res := <-resch: + if res.Err == nil { + t.Fatal("should have only been a failure here") + } + case <-time.After(time.Millisecond * 100): + t.Fatal("expected a dial failure here") + } + + select { + case res := <-resch: + if res.Err != nil { + t.Fatal("should have succeeded!") + } + case <-time.After(time.Millisecond * 100): + t.Fatal("should have gotten successful dial") + } +} diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 74e43dde44..54dd903056 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -90,6 +90,8 @@ type Swarm struct { proc goprocess.Process ctx context.Context bwc metrics.Reporter + + limiter *dialLimiter } // NewSwarm constructs a Swarm, with a Chan. @@ -122,6 +124,8 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, dialer: conn.NewDialer(local, peers.PrivKey(local), wrap), } + s.limiter = newDialLimiter(s.dialAddr) + // configure Swarm s.proc = goprocessctx.WithContextAndTeardown(ctx, s.teardown) s.SetConnHandler(nil) // make sure to setup our own conn handler. @@ -155,6 +159,7 @@ func filterAddrs(listenAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { } listenAddrs = filtered } + return listenAddrs, nil } diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index aa4a3514b4..c1b3a73922 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -1,10 +1,8 @@ package swarm import ( - "bytes" "errors" "fmt" - "sort" "sync" "time" @@ -13,7 +11,6 @@ import ( conn "github.com/ipfs/go-libp2p/p2p/net/conn" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" ma "github.com/jbenet/go-multiaddr" - "github.com/jbenet/go-multiaddr-net" context "golang.org/x/net/context" ) @@ -42,6 +39,9 @@ const dialAttempts = 1 // number of concurrent outbound dials over transports that consume file descriptors const concurrentFdDials = 160 +// number of concurrent outbound dials to make per peer +const defaultPerPeerRateLimit = 8 + // DialTimeout is the amount of time each dial attempt has. We can think about making // this larger down the road, or putting more granular timeouts (i.e. within each // subcomponent of Dial) @@ -319,32 +319,34 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { log.Debug("Dial not given PrivateKey, so WILL NOT SECURE conn.") } - // get remote peer addrs - remoteAddrs := s.peers.Addrs(p) - // make sure we can use the addresses. - remoteAddrs = addrutil.FilterUsableAddrs(remoteAddrs) - // drop out any addrs that would just dial ourselves. use ListenAddresses - // as that is a more authoritative view than localAddrs. ila, _ := s.InterfaceListenAddresses() - remoteAddrs = addrutil.Subtract(remoteAddrs, ila) - remoteAddrs = addrutil.Subtract(remoteAddrs, s.peers.Addrs(s.local)) - - log.Debugf("%s swarm dialing %s -- local:%s remote:%s", s.local, p, s.ListenAddresses(), remoteAddrs) - if len(remoteAddrs) == 0 { - err := errors.New("peer has no addresses") - logdial["error"] = err - return nil, err - } - - remoteAddrs = s.filterAddrs(remoteAddrs) - if len(remoteAddrs) == 0 { - err := errors.New("all adresses for peer have been filtered out") - logdial["error"] = err - return nil, err + subtract_filter := addrutil.SubtractFilter(append(ila, s.peers.Addrs(s.local)...)...) + + // get live channel of addresses for peer, filtered by the given filters + /* + remoteAddrChan := s.peers.AddrsChan(ctx, p, + addrutil.AddrUsableFilter, + subtract_filter, + s.Filters.AddrBlocked) + */ + + ////// TEMP UNTIL PEERSTORE GETS UPGRADED + // Ref: https://github.com/ipfs/go-libp2p-peer/pull/1 + paddrs := s.peers.Addrs(p) + good_addrs := addrutil.FilterAddrs(paddrs, + addrutil.AddrUsableFunc, + subtract_filter, + addrutil.FilterNeg(s.Filters.AddrBlocked), + ) + remoteAddrChan := make(chan ma.Multiaddr, len(good_addrs)) + for _, a := range good_addrs { + remoteAddrChan <- a } + close(remoteAddrChan) + ///////// // try to get a connection to any addr - connC, err := s.dialAddrs(ctx, p, remoteAddrs) + connC, err := s.dialAddrs(ctx, p, remoteAddrChan) if err != nil { logdial["error"] = err return nil, err @@ -364,98 +366,64 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { return swarmC, nil } -func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs []ma.Multiaddr) (conn.Conn, error) { - - // sort addresses so preferred addresses are dialed sooner - sort.Sort(AddrList(remoteAddrs)) - - // try to connect to one of the peer's known addresses. - // we dial concurrently to each of the addresses, which: - // * makes the process faster overall - // * attempts to get the fastest connection available. - // * mitigates the waste of trying bad addresses +func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (conn.Conn, error) { log.Debugf("%s swarm dialing %s %s", s.local, p, remoteAddrs) ctx, cancel := context.WithCancel(ctx) defer cancel() // cancel work when we exit func - conns := make(chan conn.Conn) - errs := make(chan error, len(remoteAddrs)) + // use a single response type instead of errs and conns, reduces complexity *a ton* + respch := make(chan dialResult) - // dialSingleAddr is used in the rate-limited async thing below. - dialSingleAddr := func(addr ma.Multiaddr) { - // rebind chans in scope so we can nil them out easily - connsout := conns - errsout := errs + defaultDialFail := fmt.Errorf("failed to dial %s (default failure)", p) + exitErr := defaultDialFail - connC, err := s.dialAddr(ctx, p, addr) - if err != nil { - connsout = nil - } else if connC == nil { - // NOTE: this really should never happen - log.Errorf("failed to dial %s %s and got no error!", p, addr) - err = fmt.Errorf("failed to dial %s %s", p, addr) - connsout = nil - } else { - errsout = nil - } - - // check parent still wants our results + var active int + for { select { - case <-ctx.Done(): - if connC != nil { - connC.Close() + case addr, ok := <-remoteAddrs: + if !ok { + remoteAddrs = nil + if active == 0 { + return nil, exitErr + } + continue } - case errsout <- err: - case connsout <- connC: - } - } - // this whole thing is in a goroutine so we can use foundConn - // to end early. - go func() { - limiter := make(chan struct{}, 8) - for _, addr := range remoteAddrs { - // returns whatever ratelimiting is acceptable for workerAddr. - // may not rate limit at all. - rl := s.addrDialRateLimit(addr) - select { - case <-ctx.Done(): // our context was cancelled - return - case rl <- struct{}{}: - // take the token, move on + // limitedDial will start a dial to the given peer when + // it is able, respecting the various different types of rate + // limiting that occur without using extra goroutines per addr + s.limitedDial(ctx, p, addr, respch) + active++ + case <-ctx.Done(): + if exitErr == defaultDialFail { + exitErr = ctx.Err() } - - select { - case <-ctx.Done(): // our context was cancelled - return - case limiter <- struct{}{}: - // take the token, move on + return nil, exitErr + case resp := <-respch: + active-- + if resp.Err != nil { + log.Error("got error on dial: ", resp.Err) + // Errors are normal, lots of dials will fail + exitErr = resp.Err + + if remoteAddrs == nil && active == 0 { + return nil, exitErr + } + } else if resp.Conn != nil { + return resp.Conn, nil } - - go func(rlc <-chan struct{}, a ma.Multiaddr) { - dialSingleAddr(a) - <-limiter - <-rlc - }(rl, addr) - } - }() - - // wair for the results. - exitErr := fmt.Errorf("failed to dial %s", p) - for range remoteAddrs { - select { - case exitErr = <-errs: // - log.Debug("dial error: ", exitErr) - case connC := <-conns: - // take the first + return asap - return connC, nil - case <-ctx.Done(): - // break out and return error - break } } - return nil, exitErr +} + +func (s *Swarm) limitedDial(ctx context.Context, p peer.ID, a ma.Multiaddr, resp chan dialResult) { + s.limiter.AddDialJob(&dialJob{ + addr: a, + peer: p, + resp: resp, + ctx: ctx, + }) } func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (conn.Conn, error) { @@ -485,16 +453,6 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (con return connC, nil } -func (s *Swarm) filterAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { - var out []ma.Multiaddr - for _, a := range addrs { - if !s.Filters.AddrBlocked(a) { - out = append(out, a) - } - } - return out -} - // dialConnSetup is the setup logic for a connection from the dial side. it // needs to add the Conn to the StreamSwarm, then run newConnSetup func dialConnSetup(ctx context.Context, s *Swarm, connC conn.Conn) (*Conn, error) { @@ -514,72 +472,3 @@ func dialConnSetup(ctx context.Context, s *Swarm, connC conn.Conn) (*Conn, error return swarmC, err } - -// addrDialRateLimit returns a ratelimiting channel for dialing transport -// addrs like a. for example, tcp is fd-ratelimited. utp is not ratelimited. -func (s *Swarm) addrDialRateLimit(a ma.Multiaddr) chan struct{} { - if isFDCostlyTransport(a) { - return s.fdRateLimit - } - - // do not rate limit it at all - return make(chan struct{}, 1) -} - -func isFDCostlyTransport(a ma.Multiaddr) bool { - return isTCPMultiaddr(a) -} - -func isTCPMultiaddr(a ma.Multiaddr) bool { - p := a.Protocols() - return len(p) == 2 && (p[0].Name == "ip4" || p[0].Name == "ip6") && p[1].Name == "tcp" -} - -type AddrList []ma.Multiaddr - -func (al AddrList) Len() int { - return len(al) -} - -func (al AddrList) Swap(i, j int) { - al[i], al[j] = al[j], al[i] -} - -func (al AddrList) Less(i, j int) bool { - a := al[i] - b := al[j] - - // dial localhost addresses next, they should fail immediately - lba := manet.IsIPLoopback(a) - lbb := manet.IsIPLoopback(b) - if lba { - if !lbb { - return true - } - } - - // dial utp and similar 'non-fd-consuming' addresses first - fda := isFDCostlyTransport(a) - fdb := isFDCostlyTransport(b) - if !fda { - if fdb { - return true - } - - // if neither consume fd's, assume equal ordering - return false - } - - // if 'b' doesnt take a file descriptor - if !fdb { - return false - } - - // if 'b' is loopback and both take file descriptors - if lbb { - return false - } - - // for the rest, just sort by bytes - return bytes.Compare(a.Bytes(), b.Bytes()) > 0 -} diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 6e9f121fcd..f5e454956c 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -303,7 +303,7 @@ func TestAddrBlocking(t *testing.T) { swarms := makeSwarms(ctx, t, 2) swarms[0].SetConnHandler(func(conn *Conn) { - t.Fatalf("no connections should happen! -- %s", conn) + t.Errorf("no connections should happen! -- %s", conn) }) _, block, err := net.ParseCIDR("127.0.0.1/8") From 19e1b7f9623a08993d39ae71f778b9ed28b755f1 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 6 May 2016 12:39:08 -0700 Subject: [PATCH 0104/3965] don't execute cancelled jobs --- p2p/net/swarm/limiter.go | 13 +++++++ p2p/net/swarm/limiter_test.go | 64 +++++++++++++++++++++++++++++++---- 2 files changed, 70 insertions(+), 7 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index fe8162d7f1..7835fe5729 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -24,6 +24,15 @@ type dialJob struct { success bool } +func (dj *dialJob) cancelled() bool { + select { + case <-dj.ctx.Done(): + return true + default: + return false + } +} + type dialLimiter struct { rllock sync.Mutex fdConsuming int @@ -116,6 +125,10 @@ func (dl *dialLimiter) AddDialJob(dj *dialJob) { // it held during the dial. func (dl *dialLimiter) executeDial(j *dialJob) { defer dl.finishedDial(j) + if j.cancelled() { + return + } + con, err := dl.dialFunc(j.ctx, j.peer, j.addr) select { case j.resp <- dialResult{Conn: con, Err: err}: diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index b5bbde847e..28733c5ab5 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -2,6 +2,7 @@ package swarm import ( "fmt" + "math/rand" "strconv" "testing" "time" @@ -75,13 +76,7 @@ func TestLimiterBasicDials(t *testing.T) { l := newDialLimiterWithParams(hangDialFunc(hang), concurrentFdDials, 4) - bads := []ma.Multiaddr{ - addrWithPort(t, 1), - addrWithPort(t, 2), - addrWithPort(t, 3), - addrWithPort(t, 4), - } - + bads := []ma.Multiaddr{addrWithPort(t, 1), addrWithPort(t, 2), addrWithPort(t, 3), addrWithPort(t, 4)} good := addrWithPort(t, 20) resch := make(chan dialResult) @@ -162,6 +157,7 @@ func TestFDLimiting(t *testing.T) { pid5 := peer.ID("testpeer5") utpaddr := mustAddr(t, "/ip4/127.0.0.1/udp/7777/utp") + // This should complete immediately since utp addresses arent blocked by fd rate limiting l.AddDialJob(&dialJob{ctx: ctx, peer: pid5, addr: utpaddr, resp: resch}) select { @@ -263,3 +259,57 @@ func TestTokenRedistribution(t *testing.T) { t.Fatal("should have gotten successful dial") } } + +func TestStressLimiter(t *testing.T) { + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { + if tcpPortOver(a, 1000) { + return conn.Conn(nil), nil + } else { + time.Sleep(time.Millisecond * time.Duration(5+rand.Intn(100))) + return nil, fmt.Errorf("test bad dial") + } + } + + l := newDialLimiterWithParams(df, 20, 5) + + var bads []ma.Multiaddr + for i := 0; i < 100; i++ { + bads = append(bads, addrWithPort(t, i)) + } + + addresses := append(bads, addrWithPort(t, 2000)) + success := make(chan struct{}) + + for i := 0; i < 20; i++ { + go func(id peer.ID) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + resp := make(chan dialResult) + time.Sleep(time.Duration(rand.Intn(10)) * time.Millisecond) + for _, i := range rand.Perm(len(addresses)) { + l.AddDialJob(&dialJob{ + addr: addresses[i], + ctx: ctx, + peer: id, + resp: resp, + }) + } + + for res := range resp { + if res.Err == nil { + success <- struct{}{} + return + } + } + }(peer.ID(fmt.Sprintf("testpeer%d", i))) + } + + for i := 0; i < 20; i++ { + select { + case <-success: + case <-time.After(time.Second * 5): + t.Fatal("expected a success within five seconds") + } + } +} From 475ca0cb712609716400b44a88758a941d4d2deb Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 13 May 2016 09:28:52 -0700 Subject: [PATCH 0105/3965] refactor locking order structure --- p2p/net/swarm/limiter.go | 59 +++++++++++++++++++++-------------- p2p/net/swarm/limiter_test.go | 4 +++ 2 files changed, 39 insertions(+), 24 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 7835fe5729..94ce05bb86 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -66,6 +66,20 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { dl.rllock.Lock() defer dl.rllock.Unlock() + if addrutil.IsFDCostlyTransport(dj.addr) { + dl.fdConsuming-- + if len(dl.waitingOnFd) > 0 { + next := dl.waitingOnFd[0] + dl.waitingOnFd = dl.waitingOnFd[1:] + if len(dl.waitingOnFd) == 0 { + dl.waitingOnFd = nil // clear out memory + } + dl.fdConsuming++ + + go dl.executeDial(next) + } + } + // release tokens in reverse order than we take them dl.activePerPeer[dj.peer]-- if dl.activePerPeer[dj.peer] == 0 { @@ -87,17 +101,6 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { go dl.executeDial(next) } - if addrutil.IsFDCostlyTransport(dj.addr) { - dl.fdConsuming-- - if len(dl.waitingOnFd) > 0 { - next := dl.waitingOnFd[0] - dl.waitingOnFd = dl.waitingOnFd[1:] - dl.fdConsuming++ - - // now, attempt to take the 'per peer limit' token - dl.schedulePerPeerDial(next) - } - } } // AddDialJob tries to take the needed tokens for starting the given dial job. @@ -107,6 +110,13 @@ func (dl *dialLimiter) AddDialJob(dj *dialJob) { dl.rllock.Lock() defer dl.rllock.Unlock() + if dl.activePerPeer[dj.peer] >= dl.perPeerLimit { + wlist := dl.waitingOnPeerLimit[dj.peer] + dl.waitingOnPeerLimit[dj.peer] = append(wlist, dj) + return + } + dl.activePerPeer[dj.peer]++ + if addrutil.IsFDCostlyTransport(dj.addr) { if dl.fdConsuming >= dl.fdLimit { dl.waitingOnFd = append(dl.waitingOnFd, dj) @@ -117,7 +127,20 @@ func (dl *dialLimiter) AddDialJob(dj *dialJob) { dl.fdConsuming++ } - dl.schedulePerPeerDial(dj) + // take second needed token and start dial! + go dl.executeDial(dj) +} + +func (dl *dialLimiter) schedulePerPeerDial(j *dialJob) { + if dl.activePerPeer[j.peer] >= dl.perPeerLimit { + wlist := dl.waitingOnPeerLimit[j.peer] + dl.waitingOnPeerLimit[j.peer] = append(wlist, j) + return + } + + // take second needed token and start dial! + dl.activePerPeer[j.peer]++ + go dl.executeDial(j) } // executeDial calls the dialFunc, and reports the result through the response @@ -135,15 +158,3 @@ func (dl *dialLimiter) executeDial(j *dialJob) { case <-j.ctx.Done(): } } - -func (dl *dialLimiter) schedulePerPeerDial(j *dialJob) { - if dl.activePerPeer[j.peer] >= dl.perPeerLimit { - wlist := dl.waitingOnPeerLimit[j.peer] - dl.waitingOnPeerLimit[j.peer] = append(wlist, j) - return - } - - // take second needed token and start dial! - dl.activePerPeer[j.peer]++ - go dl.executeDial(j) -} diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 28733c5ab5..fb1be191ee 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -3,6 +3,7 @@ package swarm import ( "fmt" "math/rand" + "runtime" "strconv" "testing" "time" @@ -262,6 +263,7 @@ func TestTokenRedistribution(t *testing.T) { func TestStressLimiter(t *testing.T) { df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { + fmt.Println("dial for peer: ", string(p)) if tcpPortOver(a, 1000) { return conn.Conn(nil), nil } else { @@ -305,6 +307,8 @@ func TestStressLimiter(t *testing.T) { }(peer.ID(fmt.Sprintf("testpeer%d", i))) } + time.Sleep(time.Millisecond * 1000) + fmt.Println("NUM GOROS: ", runtime.NumGoroutine()) for i := 0; i < 20; i++ { select { case <-success: From 37883b42dfe638c029926b1a9c887f5948123878 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sat, 14 May 2016 19:32:28 -0700 Subject: [PATCH 0106/3965] address CR feedback --- p2p/net/swarm/addr/addr.go | 4 ++-- p2p/net/swarm/addr/filter.go | 4 ++++ p2p/net/swarm/swarm_dial.go | 8 ++++---- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 8c0c94b2d6..d9ba87216a 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -40,8 +40,8 @@ func init() { SupportedTransportProtocols = transports } -// FilterAddrs is a filter that removes certain addresses, according to filter. -// if filter returns true, the address is kept. +// FilterAddrs is a filter that removes certain addresses, according the given filters. +// if all filters return true, the address is kept. func FilterAddrs(a []ma.Multiaddr, filters ...func(ma.Multiaddr) bool) []ma.Multiaddr { b := make([]ma.Multiaddr, 0, len(a)) for _, addr := range a { diff --git a/p2p/net/swarm/addr/filter.go b/p2p/net/swarm/addr/filter.go index c67e1c624e..d87ba816a7 100644 --- a/p2p/net/swarm/addr/filter.go +++ b/p2p/net/swarm/addr/filter.go @@ -5,6 +5,7 @@ import ( mafmt "github.com/whyrusleeping/mafmt" ) +// SubtractFilter returns a filter func that filters all of the given addresses func SubtractFilter(addrs ...ma.Multiaddr) func(ma.Multiaddr) bool { addrmap := make(map[string]bool) for _, a := range addrs { @@ -16,10 +17,13 @@ func SubtractFilter(addrs ...ma.Multiaddr) func(ma.Multiaddr) bool { } } +// IsFDCostlyTransport returns true for transports that require a new file +// descriptor per connection created func IsFDCostlyTransport(a ma.Multiaddr) bool { return mafmt.TCP.Matches(a) } +// FilterNeg returns a negated version of the passed in filter func FilterNeg(f func(ma.Multiaddr) bool) func(ma.Multiaddr) bool { return func(a ma.Multiaddr) bool { return !f(a) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index c1b3a73922..33b087efc3 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -390,9 +390,6 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. continue } - // limitedDial will start a dial to the given peer when - // it is able, respecting the various different types of rate - // limiting that occur without using extra goroutines per addr s.limitedDial(ctx, p, addr, respch) active++ case <-ctx.Done(): @@ -403,7 +400,7 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. case resp := <-respch: active-- if resp.Err != nil { - log.Error("got error on dial: ", resp.Err) + log.Info("got error on dial: ", resp.Err) // Errors are normal, lots of dials will fail exitErr = resp.Err @@ -417,6 +414,9 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. } } +// limitedDial will start a dial to the given peer when +// it is able, respecting the various different types of rate +// limiting that occur without using extra goroutines per addr func (s *Swarm) limitedDial(ctx context.Context, p peer.ID, a ma.Multiaddr, resp chan dialResult) { s.limiter.AddDialJob(&dialJob{ addr: a, From 4aea49efe54458febbb3ae8350a94bfe34241b0d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sat, 14 May 2016 22:51:25 -0700 Subject: [PATCH 0107/3965] test cleanup --- p2p/net/swarm/limiter_test.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index fb1be191ee..28733c5ab5 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -3,7 +3,6 @@ package swarm import ( "fmt" "math/rand" - "runtime" "strconv" "testing" "time" @@ -263,7 +262,6 @@ func TestTokenRedistribution(t *testing.T) { func TestStressLimiter(t *testing.T) { df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { - fmt.Println("dial for peer: ", string(p)) if tcpPortOver(a, 1000) { return conn.Conn(nil), nil } else { @@ -307,8 +305,6 @@ func TestStressLimiter(t *testing.T) { }(peer.ID(fmt.Sprintf("testpeer%d", i))) } - time.Sleep(time.Millisecond * 1000) - fmt.Println("NUM GOROS: ", runtime.NumGoroutine()) for i := 0; i < 20; i++ { select { case <-success: From 27cc64d4712db6dce32d981952d59fa04121f6ed Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 15 Apr 2016 09:30:59 -0700 Subject: [PATCH 0108/3965] Add peerstore method to subscribe to new addresses --- p2p/host/peerstore/addr_manager.go | 113 +++++++++++++++++++++++++++ p2p/host/peerstore/peerstore.go | 6 ++ p2p/host/peerstore/peerstore_test.go | 84 ++++++++++++++++++++ 3 files changed, 203 insertions(+) create mode 100644 p2p/host/peerstore/peerstore_test.go diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index c06771c0fb..6775b1bc68 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -5,6 +5,7 @@ import ( "time" ma "github.com/jbenet/go-multiaddr" + "golang.org/x/net/context" ) const ( @@ -51,6 +52,8 @@ type addrSet map[string]expiringAddr type AddrManager struct { addrmu sync.Mutex // guards addrs addrs map[ID]addrSet + + addrSubs map[ID][]*addrSub } // ensures the AddrManager is initialized. @@ -59,6 +62,9 @@ func (mgr *AddrManager) init() { if mgr.addrs == nil { mgr.addrs = make(map[ID]addrSet) } + if mgr.addrSubs == nil { + mgr.addrSubs = make(map[ID][]*addrSub) + } } func (mgr *AddrManager) Peers() []ID { @@ -101,6 +107,8 @@ func (mgr *AddrManager) AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) mgr.addrs[p] = amap } + subs := mgr.addrSubs[p] + // only expand ttls exp := time.Now().Add(ttl) for _, addr := range addrs { @@ -113,6 +121,10 @@ func (mgr *AddrManager) AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) a, found := amap[addrstr] if !found || exp.After(a.TTL) { amap[addrstr] = expiringAddr{Addr: addr, TTL: exp} + + for _, sub := range subs { + sub.pubAddr(addr) + } } } } @@ -137,6 +149,8 @@ func (mgr *AddrManager) SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) mgr.addrs[p] = amap } + subs := mgr.addrSubs[p] + exp := time.Now().Add(ttl) for _, addr := range addrs { if addr == nil { @@ -148,6 +162,10 @@ func (mgr *AddrManager) SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) if ttl > 0 { amap[addrs] = expiringAddr{Addr: addr, TTL: exp} + + for _, sub := range subs { + sub.pubAddr(addr) + } } else { delete(amap, addrs) } @@ -195,3 +213,98 @@ func (mgr *AddrManager) ClearAddrs(p ID) { mgr.addrs[p] = make(addrSet) // clear what was there before } + +func (mgr *AddrManager) removeSub(p ID, s *addrSub) { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + subs := mgr.addrSubs[p] + var filtered []*addrSub + for _, v := range subs { + if v != s { + filtered = append(filtered, v) + } + } + mgr.addrSubs[p] = filtered +} + +type addrSub struct { + pubch chan ma.Multiaddr + lk sync.Mutex + buffer []ma.Multiaddr + ctx context.Context +} + +func (s *addrSub) pubAddr(a ma.Multiaddr) { + select { + case s.pubch <- a: + case <-s.ctx.Done(): + } +} + +func (mgr *AddrManager) AddrStream(ctx context.Context, p ID) <-chan ma.Multiaddr { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + mgr.init() + + sub := &addrSub{pubch: make(chan ma.Multiaddr), ctx: ctx} + + out := make(chan ma.Multiaddr) + + mgr.addrSubs[p] = append(mgr.addrSubs[p], sub) + + baseaddrset := mgr.addrs[p] + var initial []ma.Multiaddr + for _, a := range baseaddrset { + initial = append(initial, a.Addr) + } + + // TODO: sort these? + + go func(buffer []ma.Multiaddr) { + defer close(out) + + sent := make(map[string]bool) + outch := out + + for _, a := range buffer { + sent[a.String()] = true + } + + var next ma.Multiaddr + if len(buffer) > 0 { + next = buffer[0] + buffer = buffer[1:] + } + + for { + select { + case outch <- next: + if len(buffer) > 0 { + next = buffer[0] + buffer = buffer[1:] + } else { + outch = nil + next = nil + } + case naddr := <-sub.pubch: + if sent[naddr.String()] { + continue + } + + sent[naddr.String()] = true + if next == nil { + next = naddr + outch = out + } else { + buffer = append(buffer, naddr) + } + case <-ctx.Done(): + mgr.removeSub(p, sub) + return + } + } + + }(initial) + + return out +} diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index f941eaf662..1cc25c01ed 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -10,6 +10,7 @@ import ( //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" ma "github.com/jbenet/go-multiaddr" + "golang.org/x/net/context" ) const ( @@ -32,6 +33,11 @@ type Peerstore interface { // that peer, useful to other services. PeerInfo(ID) PeerInfo + // AddrStream returns a channel that gets all addresses for a given + // peer sent on it. If new addresses are added after the call is made + // they will be sent along through the channel as well. + AddrStream(context.Context, ID) <-chan ma.Multiaddr + // Get/Put is a simple registry for other peer-related key/value pairs. // if we find something we use often, it should become its own set of // methods. this is a last resort. diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go new file mode 100644 index 0000000000..0ef84a48fc --- /dev/null +++ b/p2p/host/peerstore/peerstore_test.go @@ -0,0 +1,84 @@ +package peer + +import ( + "fmt" + "testing" + "time" + + ma "github.com/jbenet/go-multiaddr" + "golang.org/x/net/context" +) + +func TestAddrStream(t *testing.T) { + var addrs []ma.Multiaddr + for i := 0; i < 100; i++ { + a, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", i)) + if err != nil { + t.Fatal(err) + } + + addrs = append(addrs, a) + } + + pid := ID("testpeer") + + ps := NewPeerstore() + + ps.AddAddrs(pid, addrs[:10], time.Hour) + + ctx, cancel := context.WithCancel(context.Background()) + + addrch := ps.AddrStream(ctx, pid) + + // while that subscription is active, publish ten more addrs + // this tests that it doesnt hang + for i := 10; i < 20; i++ { + ps.AddAddr(pid, addrs[i], time.Hour) + } + + // now receive them (without hanging) + timeout := time.After(time.Second * 10) + for i := 0; i < 20; i++ { + select { + case <-addrch: + case <-timeout: + t.Fatal("timed out") + } + } + + done := make(chan struct{}) + go func() { + defer close(done) + // now send the rest of the addresses + for _, a := range addrs[20:80] { + ps.AddAddr(pid, a, time.Hour) + } + }() + + // receive some concurrently with the goroutine + timeout = time.After(time.Second * 10) + for i := 0; i < 40; i++ { + select { + case <-addrch: + case <-timeout: + } + } + + <-done + + // receive some more after waiting for that goroutine to complete + timeout = time.After(time.Second * 10) + for i := 0; i < 20; i++ { + select { + case <-addrch: + case <-timeout: + } + } + + // now cancel it, and add a few more addresses it doesnt hang afterwards + cancel() + + for _, a := range addrs[80:] { + ps.AddAddr(pid, a, time.Hour) + } +} From f05be332f907e38bc142daf0e064d7c74153f153 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 19 Apr 2016 15:18:24 -0700 Subject: [PATCH 0109/3965] do sorting on addresses before sending them to stream --- p2p/host/peerstore/addr/sorting.go | 62 ++++++++++++++++++++++++++++++ p2p/host/peerstore/addr_manager.go | 4 +- 2 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 p2p/host/peerstore/addr/sorting.go diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go new file mode 100644 index 0000000000..084722ab57 --- /dev/null +++ b/p2p/host/peerstore/addr/sorting.go @@ -0,0 +1,62 @@ +package addr + +import ( + "bytes" + + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + mafmt "github.com/whyrusleeping/mafmt" +) + +func isFDCostlyTransport(a ma.Multiaddr) bool { + return mafmt.TCP.Matches(a) +} + +type AddrList []ma.Multiaddr + +func (al AddrList) Len() int { + return len(al) +} + +func (al AddrList) Swap(i, j int) { + al[i], al[j] = al[j], al[i] +} + +func (al AddrList) Less(i, j int) bool { + a := al[i] + b := al[j] + + // dial localhost addresses next, they should fail immediately + lba := manet.IsIPLoopback(a) + lbb := manet.IsIPLoopback(b) + if lba { + if !lbb { + return true + } + } + + // dial utp and similar 'non-fd-consuming' addresses first + fda := isFDCostlyTransport(a) + fdb := isFDCostlyTransport(b) + if !fda { + if fdb { + return true + } + + // if neither consume fd's, assume equal ordering + return false + } + + // if 'b' doesnt take a file descriptor + if !fdb { + return false + } + + // if 'b' is loopback and both take file descriptors + if lbb { + return false + } + + // for the rest, just sort by bytes + return bytes.Compare(a.Bytes(), b.Bytes()) > 0 +} diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 6775b1bc68..e6c1a35529 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -1,9 +1,11 @@ package peer import ( + "sort" "sync" "time" + addr "github.com/ipfs/go-libp2p-peer/addr" ma "github.com/jbenet/go-multiaddr" "golang.org/x/net/context" ) @@ -258,7 +260,7 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p ID) <-chan ma.Multiadd initial = append(initial, a.Addr) } - // TODO: sort these? + sort.Sort(addr.AddrList(initial)) go func(buffer []ma.Multiaddr) { defer close(out) From 9b4db81718d257c57101e14f7a2983f75a6f44b2 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 20 Apr 2016 10:11:01 -0700 Subject: [PATCH 0110/3965] move interface method to more appropriate location --- p2p/host/peerstore/peerstore.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 1cc25c01ed..9b367f81e5 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -33,11 +33,6 @@ type Peerstore interface { // that peer, useful to other services. PeerInfo(ID) PeerInfo - // AddrStream returns a channel that gets all addresses for a given - // peer sent on it. If new addresses are added after the call is made - // they will be sent along through the channel as well. - AddrStream(context.Context, ID) <-chan ma.Multiaddr - // Get/Put is a simple registry for other peer-related key/value pairs. // if we find something we use often, it should become its own set of // methods. this is a last resort. @@ -67,6 +62,11 @@ type AddrBook interface { // Addresses returns all known (and valid) addresses for a given Addrs(p ID) []ma.Multiaddr + // AddrStream returns a channel that gets all addresses for a given + // peer sent on it. If new addresses are added after the call is made + // they will be sent along through the channel as well. + AddrStream(context.Context, ID) <-chan ma.Multiaddr + // ClearAddresses removes all previously stored addresses ClearAddrs(p ID) } From b1921949d5e1e7f5cc256b8ab67559ae19656639 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 20 Apr 2016 11:25:03 -0700 Subject: [PATCH 0111/3965] bugfix on sub before peer entry --- p2p/host/peerstore/addr_manager.go | 3 +- p2p/host/peerstore/peerstore_test.go | 100 ++++++++++++++++++++++++++- 2 files changed, 100 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index e6c1a35529..fae08e57ca 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -266,7 +266,7 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p ID) <-chan ma.Multiadd defer close(out) sent := make(map[string]bool) - outch := out + var outch chan ma.Multiaddr for _, a := range buffer { sent[a.String()] = true @@ -276,6 +276,7 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p ID) <-chan ma.Multiadd if len(buffer) > 0 { next = buffer[0] buffer = buffer[1:] + outch = out } for { diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 0ef84a48fc..cacdcf7eca 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -2,6 +2,7 @@ package peer import ( "fmt" + "math/rand" "testing" "time" @@ -9,9 +10,9 @@ import ( "golang.org/x/net/context" ) -func TestAddrStream(t *testing.T) { +func getAddrs(t *testing.T, n int) []ma.Multiaddr { var addrs []ma.Multiaddr - for i := 0; i < 100; i++ { + for i := 0; i < n; i++ { a, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", i)) if err != nil { t.Fatal(err) @@ -19,6 +20,11 @@ func TestAddrStream(t *testing.T) { addrs = append(addrs, a) } + return addrs +} + +func TestAddrStream(t *testing.T) { + addrs := getAddrs(t, 100) pid := ID("testpeer") @@ -82,3 +88,93 @@ func TestAddrStream(t *testing.T) { ps.AddAddr(pid, a, time.Hour) } } + +func TestGetStreamBeforePeerAdded(t *testing.T) { + addrs := getAddrs(t, 10) + pid := ID("testpeer") + + ps := NewPeerstore() + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + ach := ps.AddrStream(ctx, pid) + + for i := 0; i < 10; i++ { + ps.AddAddr(pid, addrs[i], time.Hour) + } + + received := make(map[string]bool) + var count int + + for i := 0; i < 10; i++ { + a, ok := <-ach + if !ok { + t.Fatal("channel shouldnt be closed yet") + } + if a == nil { + t.Fatal("got a nil address, thats weird") + } + count++ + if received[a.String()] { + t.Fatal("received duplicate address") + } + received[a.String()] = true + } + + select { + case <-ach: + t.Fatal("shouldnt have received any more addresses") + default: + } + + if count != 10 { + t.Fatal("should have received exactly ten addresses, got ", count) + } + + for _, a := range addrs { + if !received[a.String()] { + t.Log(received) + t.Fatalf("expected to receive address %s but didnt", a) + } + } +} + +func TestAddrStreamDuplicates(t *testing.T) { + addrs := getAddrs(t, 10) + pid := ID("testpeer") + + ps := NewPeerstore() + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + ach := ps.AddrStream(ctx, pid) + + go func() { + for i := 0; i < 10; i++ { + ps.AddAddr(pid, addrs[i], time.Hour) + ps.AddAddr(pid, addrs[rand.Intn(10)], time.Hour) + } + + // make sure that all addresses get processed before context is cancelled + time.Sleep(time.Millisecond * 50) + cancel() + }() + + received := make(map[string]bool) + var count int + for a := range ach { + if a == nil { + t.Fatal("got a nil address, thats weird") + } + count++ + if received[a.String()] { + t.Fatal("received duplicate address") + } + received[a.String()] = true + } + + if count != 10 { + t.Fatal("should have received exactly ten addresses") + } +} From 53bbe685aa88b385ec2c01bbdf8f479af688dd3d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 20 Apr 2016 14:35:27 -0700 Subject: [PATCH 0112/3965] migrate sorting tests --- p2p/host/peerstore/addr/sorting_test.go | 32 +++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 p2p/host/peerstore/addr/sorting_test.go diff --git a/p2p/host/peerstore/addr/sorting_test.go b/p2p/host/peerstore/addr/sorting_test.go new file mode 100644 index 0000000000..7dfa013d63 --- /dev/null +++ b/p2p/host/peerstore/addr/sorting_test.go @@ -0,0 +1,32 @@ +package addr + +import ( + "sort" + "testing" +) + +func TestAddressSorting(t *testing.T) { + u1 := newAddrOrFatal(t, "/ip4/152.12.23.53/udp/1234/utp") + u2l := newAddrOrFatal(t, "/ip4/127.0.0.1/udp/1234/utp") + local := newAddrOrFatal(t, "/ip4/127.0.0.1/tcp/1234") + norm := newAddrOrFatal(t, "/ip4/6.5.4.3/tcp/1234") + + l := AddrList{local, u1, u2l, norm} + sort.Sort(l) + + if !l[0].Equal(u2l) { + t.Fatal("expected utp local addr to be sorted first: ", l[0]) + } + + if !l[1].Equal(u1) { + t.Fatal("expected utp addr to be sorted second") + } + + if !l[2].Equal(local) { + t.Fatal("expected tcp localhost addr thid") + } + + if !l[3].Equal(norm) { + t.Fatal("expected normal addr last") + } +} From d1d8f53c8b7b67ef637ea49286433a9ace524f7e Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 31 May 2016 11:31:50 -0700 Subject: [PATCH 0113/3965] extract peerstore from peer package --- p2p/host/peerstore/addr_manager.go | 33 +++-- p2p/host/peerstore/addr_manager_test.go | 7 +- p2p/host/peerstore/metrics.go | 16 +- p2p/host/peerstore/metrics_test.go | 7 +- p2p/host/peerstore/peer.go | 185 ------------------------ p2p/host/peerstore/peer_test.go | 163 --------------------- p2p/host/peerstore/peerinfo.go | 55 +++++++ p2p/host/peerstore/peerstore.go | 74 +++++----- p2p/host/peerstore/peerstore_test.go | 9 +- 9 files changed, 132 insertions(+), 417 deletions(-) delete mode 100644 p2p/host/peerstore/peer.go delete mode 100644 p2p/host/peerstore/peer_test.go create mode 100644 p2p/host/peerstore/peerinfo.go diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index fae08e57ca..877af2c4cf 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -1,11 +1,12 @@ -package peer +package peerstore import ( "sort" "sync" "time" - addr "github.com/ipfs/go-libp2p-peer/addr" + peer "github.com/ipfs/go-libp2p-peer" + addr "github.com/ipfs/go-libp2p-peerstore/addr" ma "github.com/jbenet/go-multiaddr" "golang.org/x/net/context" ) @@ -53,30 +54,30 @@ type addrSet map[string]expiringAddr // The zero-value is ready to be used. type AddrManager struct { addrmu sync.Mutex // guards addrs - addrs map[ID]addrSet + addrs map[peer.ID]addrSet - addrSubs map[ID][]*addrSub + addrSubs map[peer.ID][]*addrSub } // ensures the AddrManager is initialized. // So we can use the zero value. func (mgr *AddrManager) init() { if mgr.addrs == nil { - mgr.addrs = make(map[ID]addrSet) + mgr.addrs = make(map[peer.ID]addrSet) } if mgr.addrSubs == nil { - mgr.addrSubs = make(map[ID][]*addrSub) + mgr.addrSubs = make(map[peer.ID][]*addrSub) } } -func (mgr *AddrManager) Peers() []ID { +func (mgr *AddrManager) Peers() []peer.ID { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() if mgr.addrs == nil { return nil } - pids := make([]ID, 0, len(mgr.addrs)) + pids := make([]peer.ID, 0, len(mgr.addrs)) for pid := range mgr.addrs { pids = append(pids, pid) } @@ -84,14 +85,14 @@ func (mgr *AddrManager) Peers() []ID { } // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) -func (mgr *AddrManager) AddAddr(p ID, addr ma.Multiaddr, ttl time.Duration) { +func (mgr *AddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } // AddAddrs gives AddrManager addresses to use, with a given ttl // (time-to-live), after which the address is no longer valid. // If the manager has a longer TTL, the operation is a no-op for that address -func (mgr *AddrManager) AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) { +func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() @@ -132,13 +133,13 @@ func (mgr *AddrManager) AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) } // SetAddr calls mgr.SetAddrs(p, addr, ttl) -func (mgr *AddrManager) SetAddr(p ID, addr ma.Multiaddr, ttl time.Duration) { +func (mgr *AddrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } // SetAddrs sets the ttl on addresses. This clears any TTL there previously. // This is used when we receive the best estimate of the validity of an address. -func (mgr *AddrManager) SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) { +func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() @@ -175,7 +176,7 @@ func (mgr *AddrManager) SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) } // Addresses returns all known (and valid) addresses for a given -func (mgr *AddrManager) Addrs(p ID) []ma.Multiaddr { +func (mgr *AddrManager) Addrs(p peer.ID) []ma.Multiaddr { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() @@ -208,7 +209,7 @@ func (mgr *AddrManager) Addrs(p ID) []ma.Multiaddr { } // ClearAddresses removes all previously stored addresses -func (mgr *AddrManager) ClearAddrs(p ID) { +func (mgr *AddrManager) ClearAddrs(p peer.ID) { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() mgr.init() @@ -216,7 +217,7 @@ func (mgr *AddrManager) ClearAddrs(p ID) { mgr.addrs[p] = make(addrSet) // clear what was there before } -func (mgr *AddrManager) removeSub(p ID, s *addrSub) { +func (mgr *AddrManager) removeSub(p peer.ID, s *addrSub) { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() subs := mgr.addrSubs[p] @@ -243,7 +244,7 @@ func (s *addrSub) pubAddr(a ma.Multiaddr) { } } -func (mgr *AddrManager) AddrStream(ctx context.Context, p ID) <-chan ma.Multiaddr { +func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() mgr.init() diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 1d6f6b69f8..a480ee71a3 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -1,14 +1,15 @@ -package peer +package peerstore import ( "testing" "time" + "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" ) -func IDS(t *testing.T, ids string) ID { - id, err := IDB58Decode(ids) +func IDS(t *testing.T, ids string) peer.ID { + id, err := peer.IDB58Decode(ids) if err != nil { t.Fatalf("id %q is bad: %s", ids, err) } diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index 6cad11cdfd..2045ed9c93 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -1,8 +1,10 @@ -package peer +package peerstore import ( "sync" "time" + + "github.com/ipfs/go-libp2p-peer" ) // LatencyEWMASmooting governs the decay of the EWMA (the speed @@ -15,26 +17,26 @@ var LatencyEWMASmoothing = 0.1 type Metrics interface { // RecordLatency records a new latency measurement - RecordLatency(ID, time.Duration) + RecordLatency(peer.ID, time.Duration) // LatencyEWMA returns an exponentially-weighted moving avg. // of all measurements of a peer's latency. - LatencyEWMA(ID) time.Duration + LatencyEWMA(peer.ID) time.Duration } type metrics struct { - latmap map[ID]time.Duration + latmap map[peer.ID]time.Duration latmu sync.RWMutex } func NewMetrics() Metrics { return &metrics{ - latmap: make(map[ID]time.Duration), + latmap: make(map[peer.ID]time.Duration), } } // RecordLatency records a new latency measurement -func (m *metrics) RecordLatency(p ID, next time.Duration) { +func (m *metrics) RecordLatency(p peer.ID, next time.Duration) { nextf := float64(next) s := LatencyEWMASmoothing if s > 1 || s < 0 { @@ -55,7 +57,7 @@ func (m *metrics) RecordLatency(p ID, next time.Duration) { // LatencyEWMA returns an exponentially-weighted moving avg. // of all measurements of a peer's latency. -func (m *metrics) LatencyEWMA(p ID) time.Duration { +func (m *metrics) LatencyEWMA(p peer.ID) time.Duration { m.latmu.RLock() lat := m.latmap[p] m.latmu.RUnlock() diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 08d53ac83f..110b6cf103 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -1,4 +1,4 @@ -package peer_test +package peerstore import ( "fmt" @@ -7,14 +7,13 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" testutil "github.com/ipfs/go-libp2p-peer/test" ) func TestLatencyEWMAFun(t *testing.T) { t.Skip("run it for fun") - m := peer.NewMetrics() + m := NewMetrics() id, err := testutil.RandPeerID() if err != nil { t.Fatal(err) @@ -41,7 +40,7 @@ func TestLatencyEWMAFun(t *testing.T) { } func TestLatencyEWMA(t *testing.T) { - m := peer.NewMetrics() + m := NewMetrics() id, err := testutil.RandPeerID() if err != nil { t.Fatal(err) diff --git a/p2p/host/peerstore/peer.go b/p2p/host/peerstore/peer.go deleted file mode 100644 index f42f5b623f..0000000000 --- a/p2p/host/peerstore/peer.go +++ /dev/null @@ -1,185 +0,0 @@ -// package peer implements an object used to represent peers in the ipfs network. -package peer - -import ( - "encoding/hex" - "encoding/json" - "fmt" - "strings" - - u "github.com/ipfs/go-ipfs-util" - ic "github.com/ipfs/go-libp2p-crypto" - logging "github.com/ipfs/go-log" // ID represents the identity of a peer. - b58 "github.com/jbenet/go-base58" - ma "github.com/jbenet/go-multiaddr" - mh "github.com/jbenet/go-multihash" -) - -var log = logging.Logger("peer") - -type ID string - -// Pretty returns a b58-encoded string of the ID -func (id ID) Pretty() string { - return IDB58Encode(id) -} - -func (id ID) Loggable() map[string]interface{} { - return map[string]interface{}{ - "peerID": id.Pretty(), - } -} - -// String prints out the peer. -// -// TODO(brian): ensure correctness at ID generation and -// enforce this by only exposing functions that generate -// IDs safely. Then any peer.ID type found in the -// codebase is known to be correct. -func (id ID) String() string { - pid := id.Pretty() - - //All sha256 nodes start with Qm - //We can skip the Qm to make the peer.ID more useful - if strings.HasPrefix(pid, "Qm") { - pid = pid[2:] - } - - maxRunes := 6 - if len(pid) < maxRunes { - maxRunes = len(pid) - } - return fmt.Sprintf("", pid[:maxRunes]) -} - -// MatchesPrivateKey tests whether this ID was derived from sk -func (id ID) MatchesPrivateKey(sk ic.PrivKey) bool { - return id.MatchesPublicKey(sk.GetPublic()) -} - -// MatchesPublicKey tests whether this ID was derived from pk -func (id ID) MatchesPublicKey(pk ic.PubKey) bool { - oid, err := IDFromPublicKey(pk) - if err != nil { - return false - } - return oid == id -} - -// IDFromString cast a string to ID type, and validate -// the id to make sure it is a multihash. -func IDFromString(s string) (ID, error) { - if _, err := mh.Cast([]byte(s)); err != nil { - return ID(""), err - } - return ID(s), nil -} - -// IDFromBytes cast a string to ID type, and validate -// the id to make sure it is a multihash. -func IDFromBytes(b []byte) (ID, error) { - if _, err := mh.Cast(b); err != nil { - return ID(""), err - } - return ID(b), nil -} - -// IDB58Decode returns a b58-decoded Peer -func IDB58Decode(s string) (ID, error) { - m, err := mh.FromB58String(s) - if err != nil { - return "", err - } - return ID(m), err -} - -// IDB58Encode returns b58-encoded string -func IDB58Encode(id ID) string { - return b58.Encode([]byte(id)) -} - -// IDHexDecode returns a b58-decoded Peer -func IDHexDecode(s string) (ID, error) { - m, err := mh.FromHexString(s) - if err != nil { - return "", err - } - return ID(m), err -} - -// IDHexEncode returns b58-encoded string -func IDHexEncode(id ID) string { - return hex.EncodeToString([]byte(id)) -} - -// IDFromPublicKey returns the Peer ID corresponding to pk -func IDFromPublicKey(pk ic.PubKey) (ID, error) { - b, err := pk.Bytes() - if err != nil { - return "", err - } - hash := u.Hash(b) - return ID(hash), nil -} - -// IDFromPrivateKey returns the Peer ID corresponding to sk -func IDFromPrivateKey(sk ic.PrivKey) (ID, error) { - return IDFromPublicKey(sk.GetPublic()) -} - -// Map maps a Peer ID to a struct. -type Set map[ID]struct{} - -// PeerInfo is a small struct used to pass around a peer with -// a set of addresses (and later, keys?). This is not meant to be -// a complete view of the system, but rather to model updates to -// the peerstore. It is used by things like the routing system. -type PeerInfo struct { - ID ID - Addrs []ma.Multiaddr -} - -func (pi *PeerInfo) Loggable() map[string]interface{} { - return map[string]interface{}{ - "peerID": pi.ID.Pretty(), - "addrs": pi.Addrs, - } -} - -func (pi *PeerInfo) MarshalJSON() ([]byte, error) { - out := make(map[string]interface{}) - out["ID"] = IDB58Encode(pi.ID) - var addrs []string - for _, a := range pi.Addrs { - addrs = append(addrs, a.String()) - } - out["Addrs"] = addrs - return json.Marshal(out) -} - -func (pi *PeerInfo) UnmarshalJSON(b []byte) error { - var data map[string]interface{} - err := json.Unmarshal(b, &data) - if err != nil { - return err - } - pid, err := IDB58Decode(data["ID"].(string)) - if err != nil { - return err - } - pi.ID = pid - addrs, ok := data["Addrs"].([]interface{}) - if ok { - for _, a := range addrs { - pi.Addrs = append(pi.Addrs, ma.StringCast(a.(string))) - } - } - return nil -} - -// IDSlice for sorting peers -type IDSlice []ID - -func (es IDSlice) Len() int { return len(es) } -func (es IDSlice) Swap(i, j int) { es[i], es[j] = es[j], es[i] } -func (es IDSlice) Less(i, j int) bool { return string(es[i]) < string(es[j]) } diff --git a/p2p/host/peerstore/peer_test.go b/p2p/host/peerstore/peer_test.go deleted file mode 100644 index ffef1eec56..0000000000 --- a/p2p/host/peerstore/peer_test.go +++ /dev/null @@ -1,163 +0,0 @@ -package peer_test - -import ( - "encoding/base64" - "fmt" - "strings" - "testing" - - u "github.com/ipfs/go-ipfs-util" - ic "github.com/ipfs/go-libp2p-crypto" - . "github.com/ipfs/go-libp2p-peer" - tu "github.com/ipfs/go-libp2p-peer/test" - - b58 "github.com/jbenet/go-base58" -) - -var gen1 keyset // generated -var gen2 keyset // generated -var man keyset // manual - -func init() { - if err := gen1.generate(); err != nil { - panic(err) - } - if err := gen2.generate(); err != nil { - panic(err) - } - - skManBytes = strings.Replace(skManBytes, "\n", "", -1) - if err := man.load(hpkpMan, skManBytes); err != nil { - panic(err) - } -} - -type keyset struct { - sk ic.PrivKey - pk ic.PubKey - hpk string - hpkp string -} - -func (ks *keyset) generate() error { - var err error - ks.sk, ks.pk, err = tu.RandTestKeyPair(512) - if err != nil { - return err - } - - bpk, err := ks.pk.Bytes() - if err != nil { - return err - } - - ks.hpk = string(u.Hash(bpk)) - ks.hpkp = b58.Encode([]byte(ks.hpk)) - return nil -} - -func (ks *keyset) load(hpkp, skBytesStr string) error { - skBytes, err := base64.StdEncoding.DecodeString(skBytesStr) - if err != nil { - return err - } - - ks.sk, err = ic.UnmarshalPrivateKey(skBytes) - if err != nil { - return err - } - - ks.pk = ks.sk.GetPublic() - bpk, err := ks.pk.Bytes() - if err != nil { - return err - } - - ks.hpk = string(u.Hash(bpk)) - ks.hpkp = b58.Encode([]byte(ks.hpk)) - if ks.hpkp != hpkp { - return fmt.Errorf("hpkp doesn't match key. %s", hpkp) - } - return nil -} - -func TestIDMatchesPublicKey(t *testing.T) { - - test := func(ks keyset) { - p1, err := IDB58Decode(ks.hpkp) - if err != nil { - t.Fatal(err) - } - - if ks.hpk != string(p1) { - t.Error("p1 and hpk differ") - } - - if !p1.MatchesPublicKey(ks.pk) { - t.Fatal("p1 does not match pk") - } - - p2, err := IDFromPublicKey(ks.pk) - if err != nil { - t.Fatal(err) - } - - if p1 != p2 { - t.Error("p1 and p2 differ", p1.Pretty(), p2.Pretty()) - } - - if p2.Pretty() != ks.hpkp { - t.Error("hpkp and p2.Pretty differ", ks.hpkp, p2.Pretty()) - } - } - - test(gen1) - test(gen2) - test(man) -} - -func TestIDMatchesPrivateKey(t *testing.T) { - - test := func(ks keyset) { - p1, err := IDB58Decode(ks.hpkp) - if err != nil { - t.Fatal(err) - } - - if ks.hpk != string(p1) { - t.Error("p1 and hpk differ") - } - - if !p1.MatchesPrivateKey(ks.sk) { - t.Fatal("p1 does not match sk") - } - - p2, err := IDFromPrivateKey(ks.sk) - if err != nil { - t.Fatal(err) - } - - if p1 != p2 { - t.Error("p1 and p2 differ", p1.Pretty(), p2.Pretty()) - } - } - - test(gen1) - test(gen2) - test(man) -} - -var hpkpMan = `QmRK3JgmVEGiewxWbhpXLJyjWuGuLeSTMTndA1coMHEy5o` -var skManBytes = ` -CAAS4AQwggJcAgEAAoGBAL7w+Wc4VhZhCdM/+Hccg5Nrf4q9NXWwJylbSrXz/unFS24wyk6pEk0zi3W -7li+vSNVO+NtJQw9qGNAMtQKjVTP+3Vt/jfQRnQM3s6awojtjueEWuLYVt62z7mofOhCtj+VwIdZNBo -/EkLZ0ETfcvN5LVtLYa8JkXybnOPsLvK+PAgMBAAECgYBdk09HDM7zzL657uHfzfOVrdslrTCj6p5mo -DzvCxLkkjIzYGnlPuqfNyGjozkpSWgSUc+X+EGLLl3WqEOVdWJtbM61fewEHlRTM5JzScvwrJ39t7o6 -CCAjKA0cBWBd6UWgbN/t53RoWvh9HrA2AW5YrT0ZiAgKe9y7EMUaENVJ8QJBAPhpdmb4ZL4Fkm4OKia -NEcjzn6mGTlZtef7K/0oRC9+2JkQnCuf6HBpaRhJoCJYg7DW8ZY+AV6xClKrgjBOfERMCQQDExhnzu2 -dsQ9k8QChBlpHO0TRbZBiQfC70oU31kM1AeLseZRmrxv9Yxzdl8D693NNWS2JbKOXl0kMHHcuGQLMVA -kBZ7WvkmPV3aPL6jnwp2pXepntdVnaTiSxJ1dkXShZ/VSSDNZMYKY306EtHrIu3NZHtXhdyHKcggDXr -qkBrdgErAkAlpGPojUwemOggr4FD8sLX1ot2hDJyyV7OK2FXfajWEYJyMRL1Gm9Uk1+Un53RAkJneqp -JGAzKpyttXBTIDO51AkEA98KTiROMnnU8Y6Mgcvr68/SMIsvCYMt9/mtwSBGgl80VaTQ5Hpaktl6Xbh -VUt5Wv0tRxlXZiViCGCD1EtrrwTw== -` diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go new file mode 100644 index 0000000000..584d363c1a --- /dev/null +++ b/p2p/host/peerstore/peerinfo.go @@ -0,0 +1,55 @@ +package peerstore + +import ( + "encoding/json" + + "github.com/ipfs/go-libp2p-peer" + ma "github.com/jbenet/go-multiaddr" +) + +// PeerInfo is a small struct used to pass around a peer with +// a set of addresses (and later, keys?). This is not meant to be +// a complete view of the system, but rather to model updates to +// the peerstore. It is used by things like the routing system. +type PeerInfo struct { + ID peer.ID + Addrs []ma.Multiaddr +} + +func (pi *PeerInfo) Loggable() map[string]interface{} { + return map[string]interface{}{ + "peerID": pi.ID.Pretty(), + "addrs": pi.Addrs, + } +} + +func (pi *PeerInfo) MarshalJSON() ([]byte, error) { + out := make(map[string]interface{}) + out["ID"] = pi.ID.Pretty() + var addrs []string + for _, a := range pi.Addrs { + addrs = append(addrs, a.String()) + } + out["Addrs"] = addrs + return json.Marshal(out) +} + +func (pi *PeerInfo) UnmarshalJSON(b []byte) error { + var data map[string]interface{} + err := json.Unmarshal(b, &data) + if err != nil { + return err + } + pid, err := peer.IDB58Decode(data["ID"].(string)) + if err != nil { + return err + } + pi.ID = pid + addrs, ok := data["Addrs"].([]interface{}) + if ok { + for _, a := range addrs { + pi.Addrs = append(pi.Addrs, ma.StringCast(a.(string))) + } + } + return nil +} diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 9b367f81e5..60ebcef523 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -1,4 +1,4 @@ -package peer +package peerstore import ( "errors" @@ -9,10 +9,14 @@ import ( //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" + "github.com/ipfs/go-libp2p-peer" + logging "github.com/ipfs/go-log" ma "github.com/jbenet/go-multiaddr" "golang.org/x/net/context" ) +var log = logging.Logger("peerstore") + const ( // AddressTTL is the expiration time of addresses. AddressTTL = time.Hour @@ -26,18 +30,18 @@ type Peerstore interface { Metrics // Peers returns a list of all peer.IDs in this Peerstore - Peers() []ID + Peers() []peer.ID // PeerInfo returns a peer.PeerInfo struct for given peer.ID. // This is a small slice of the information Peerstore has on // that peer, useful to other services. - PeerInfo(ID) PeerInfo + PeerInfo(peer.ID) PeerInfo // Get/Put is a simple registry for other peer-related key/value pairs. // if we find something we use often, it should become its own set of // methods. this is a last resort. - Get(id ID, key string) (interface{}, error) - Put(id ID, key string, val interface{}) error + Get(id peer.ID, key string) (interface{}, error) + Put(id peer.ID, key string, val interface{}) error } // AddrBook is an interface that fits the new AddrManager. I'm patching @@ -45,58 +49,58 @@ type Peerstore interface { type AddrBook interface { // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) - AddAddr(p ID, addr ma.Multiaddr, ttl time.Duration) + AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) // AddAddrs gives AddrManager addresses to use, with a given ttl // (time-to-live), after which the address is no longer valid. // If the manager has a longer TTL, the operation is a no-op for that address - AddAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) + AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) // SetAddr calls mgr.SetAddrs(p, addr, ttl) - SetAddr(p ID, addr ma.Multiaddr, ttl time.Duration) + SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) // SetAddrs sets the ttl on addresses. This clears any TTL there previously. // This is used when we receive the best estimate of the validity of an address. - SetAddrs(p ID, addrs []ma.Multiaddr, ttl time.Duration) + SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) // Addresses returns all known (and valid) addresses for a given - Addrs(p ID) []ma.Multiaddr + Addrs(p peer.ID) []ma.Multiaddr // AddrStream returns a channel that gets all addresses for a given // peer sent on it. If new addresses are added after the call is made // they will be sent along through the channel as well. - AddrStream(context.Context, ID) <-chan ma.Multiaddr + AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr // ClearAddresses removes all previously stored addresses - ClearAddrs(p ID) + ClearAddrs(p peer.ID) } // KeyBook tracks the Public keys of Peers. type KeyBook interface { - PubKey(ID) ic.PubKey - AddPubKey(ID, ic.PubKey) error + PubKey(peer.ID) ic.PubKey + AddPubKey(peer.ID, ic.PubKey) error - PrivKey(ID) ic.PrivKey - AddPrivKey(ID, ic.PrivKey) error + PrivKey(peer.ID) ic.PrivKey + AddPrivKey(peer.ID, ic.PrivKey) error } type keybook struct { - pks map[ID]ic.PubKey - sks map[ID]ic.PrivKey + pks map[peer.ID]ic.PubKey + sks map[peer.ID]ic.PrivKey sync.RWMutex // same lock. wont happen a ton. } func newKeybook() *keybook { return &keybook{ - pks: map[ID]ic.PubKey{}, - sks: map[ID]ic.PrivKey{}, + pks: map[peer.ID]ic.PubKey{}, + sks: map[peer.ID]ic.PrivKey{}, } } -func (kb *keybook) Peers() []ID { +func (kb *keybook) Peers() []peer.ID { kb.RLock() - ps := make([]ID, 0, len(kb.pks)+len(kb.sks)) + ps := make([]peer.ID, 0, len(kb.pks)+len(kb.sks)) for p := range kb.pks { ps = append(ps, p) } @@ -109,14 +113,14 @@ func (kb *keybook) Peers() []ID { return ps } -func (kb *keybook) PubKey(p ID) ic.PubKey { +func (kb *keybook) PubKey(p peer.ID) ic.PubKey { kb.RLock() pk := kb.pks[p] kb.RUnlock() return pk } -func (kb *keybook) AddPubKey(p ID, pk ic.PubKey) error { +func (kb *keybook) AddPubKey(p peer.ID, pk ic.PubKey) error { // check it's correct first if !p.MatchesPublicKey(pk) { @@ -129,14 +133,14 @@ func (kb *keybook) AddPubKey(p ID, pk ic.PubKey) error { return nil } -func (kb *keybook) PrivKey(p ID) ic.PrivKey { +func (kb *keybook) PrivKey(p peer.ID) ic.PrivKey { kb.RLock() sk := kb.sks[p] kb.RUnlock() return sk } -func (kb *keybook) AddPrivKey(p ID, sk ic.PrivKey) error { +func (kb *keybook) AddPrivKey(p peer.ID, sk ic.PrivKey) error { if sk == nil { return errors.New("sk is nil (PrivKey)") @@ -176,7 +180,7 @@ func NewPeerstore() Peerstore { } } -func (ps *peerstore) Put(p ID, key string, val interface{}) error { +func (ps *peerstore) Put(p peer.ID, key string, val interface{}) error { //dsk := ds.NewKey(string(p) + "/" + key) //return ps.ds.Put(dsk, val) ps.dslock.Lock() @@ -185,7 +189,7 @@ func (ps *peerstore) Put(p ID, key string, val interface{}) error { return nil } -func (ps *peerstore) Get(p ID, key string) (interface{}, error) { +func (ps *peerstore) Get(p peer.ID, key string) (interface{}, error) { //dsk := ds.NewKey(string(p) + "/" + key) //return ps.ds.Get(dsk) @@ -198,8 +202,8 @@ func (ps *peerstore) Get(p ID, key string) (interface{}, error) { return i, nil } -func (ps *peerstore) Peers() []ID { - set := map[ID]struct{}{} +func (ps *peerstore) Peers() []peer.ID { + set := map[peer.ID]struct{}{} for _, p := range ps.keybook.Peers() { set[p] = struct{}{} } @@ -207,21 +211,21 @@ func (ps *peerstore) Peers() []ID { set[p] = struct{}{} } - pps := make([]ID, 0, len(set)) + pps := make([]peer.ID, 0, len(set)) for p := range set { pps = append(pps, p) } return pps } -func (ps *peerstore) PeerInfo(p ID) PeerInfo { +func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { return PeerInfo{ ID: p, Addrs: ps.AddrManager.Addrs(p), } } -func PeerInfos(ps Peerstore, peers []ID) []PeerInfo { +func PeerInfos(ps Peerstore, peers []peer.ID) []PeerInfo { pi := make([]PeerInfo, len(peers)) for i, p := range peers { pi[i] = ps.PeerInfo(p) @@ -229,8 +233,8 @@ func PeerInfos(ps Peerstore, peers []ID) []PeerInfo { return pi } -func PeerInfoIDs(pis []PeerInfo) []ID { - ps := make([]ID, len(pis)) +func PeerInfoIDs(pis []PeerInfo) []peer.ID { + ps := make([]peer.ID, len(pis)) for i, pi := range pis { ps[i] = pi.ID } diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index cacdcf7eca..365cc32826 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -1,4 +1,4 @@ -package peer +package peerstore import ( "fmt" @@ -6,6 +6,7 @@ import ( "testing" "time" + peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" "golang.org/x/net/context" ) @@ -26,7 +27,7 @@ func getAddrs(t *testing.T, n int) []ma.Multiaddr { func TestAddrStream(t *testing.T) { addrs := getAddrs(t, 100) - pid := ID("testpeer") + pid := peer.ID("testpeer") ps := NewPeerstore() @@ -91,7 +92,7 @@ func TestAddrStream(t *testing.T) { func TestGetStreamBeforePeerAdded(t *testing.T) { addrs := getAddrs(t, 10) - pid := ID("testpeer") + pid := peer.ID("testpeer") ps := NewPeerstore() @@ -142,7 +143,7 @@ func TestGetStreamBeforePeerAdded(t *testing.T) { func TestAddrStreamDuplicates(t *testing.T) { addrs := getAddrs(t, 10) - pid := ID("testpeer") + pid := peer.ID("testpeer") ps := NewPeerstore() From 4e786eb9472b6be10c866c85fc2516b44694a754 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 31 May 2016 16:24:11 -0700 Subject: [PATCH 0114/3965] split peerstore from peer package --- p2p/net/swarm/dial_test.go | 19 ++++++++++--------- p2p/net/swarm/peers_test.go | 3 ++- p2p/net/swarm/simul_test.go | 5 +++-- p2p/net/swarm/swarm.go | 5 +++-- p2p/net/swarm/swarm_addr_test.go | 6 +++--- p2p/net/swarm/swarm_net.go | 5 +++-- p2p/net/swarm/swarm_test.go | 13 +++++++------ 7 files changed, 31 insertions(+), 25 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 988f7f7f01..833d1087ce 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -7,11 +7,12 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" testutil "github.com/ipfs/go-libp2p/testutil" ci "github.com/ipfs/go-libp2p/testutil/ci" + peer "github.com/ipfs/go-libp2p-peer" + pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" manet "github.com/jbenet/go-multiaddr-net" context "golang.org/x/net/context" @@ -32,7 +33,7 @@ func TestBasicDial(t *testing.T) { s1 := swarms[0] s2 := swarms[1] - s1.peers.AddAddrs(s2.local, s2.ListenAddresses(), peer.PermanentAddrTTL) + s1.peers.AddAddrs(s2.local, s2.ListenAddresses(), pstore.PermanentAddrTTL) c, err := s1.Dial(ctx, s2.local) if err != nil { @@ -57,7 +58,7 @@ func TestDialWithNoListeners(t *testing.T) { defer closeSwarms(swarms) s2 := swarms[0] - s1.peers.AddAddrs(s2.local, s2.ListenAddresses(), peer.PermanentAddrTTL) + s1.peers.AddAddrs(s2.local, s2.ListenAddresses(), pstore.PermanentAddrTTL) c, err := s1.Dial(ctx, s2.local) if err != nil { @@ -101,7 +102,7 @@ func TestSimultDials(t *testing.T) { connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // copy for other peer log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.local, dst, addr) - s.peers.AddAddr(dst, addr, peer.TempAddrTTL) + s.peers.AddAddr(dst, addr, pstore.TempAddrTTL) if _, err := s.Dial(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } @@ -178,7 +179,7 @@ func TestDialWait(t *testing.T) { s2p, s2addr, s2l := newSilentPeer(t) go acceptAndHang(s2l) defer s2l.Close() - s1.peers.AddAddr(s2p, s2addr, peer.PermanentAddrTTL) + s1.peers.AddAddr(s2p, s2addr, pstore.PermanentAddrTTL) before := time.Now() if c, err := s1.Dial(ctx, s2p); err == nil { @@ -224,13 +225,13 @@ func TestDialBackoff(t *testing.T) { if err != nil { t.Fatal(err) } - s1.peers.AddAddrs(s2.local, s2addrs, peer.PermanentAddrTTL) + s1.peers.AddAddrs(s2.local, s2addrs, pstore.PermanentAddrTTL) // dial to a non-existent peer. s3p, s3addr, s3l := newSilentPeer(t) go acceptAndHang(s3l) defer s3l.Close() - s1.peers.AddAddr(s3p, s3addr, peer.PermanentAddrTTL) + s1.peers.AddAddr(s3p, s3addr, pstore.PermanentAddrTTL) // in this test we will: // 1) dial 10x to each node. @@ -442,7 +443,7 @@ func TestDialBackoffClears(t *testing.T) { defer s2l.Close() // phase 1 -- dial to non-operational addresses - s1.peers.AddAddr(s2.local, s2bad, peer.PermanentAddrTTL) + s1.peers.AddAddr(s2.local, s2bad, pstore.PermanentAddrTTL) before := time.Now() if c, err := s1.Dial(ctx, s2.local); err == nil { @@ -472,7 +473,7 @@ func TestDialBackoffClears(t *testing.T) { if err != nil { t.Fatal(err) } - s1.peers.AddAddrs(s2.local, ifaceAddrs1, peer.PermanentAddrTTL) + s1.peers.AddAddrs(s2.local, ifaceAddrs1, pstore.PermanentAddrTTL) if _, err := s1.Dial(ctx, s2.local); err == nil { t.Fatal("should have failed to dial backed off peer") diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index cacd88ece8..b63e655b11 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -4,6 +4,7 @@ import ( "testing" peer "github.com/ipfs/go-libp2p-peer" + pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" context "golang.org/x/net/context" ) @@ -17,7 +18,7 @@ func TestPeers(t *testing.T) { connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // TODO: make a DialAddr func. - s.peers.AddAddr(dst, addr, peer.PermanentAddrTTL) + s.peers.AddAddr(dst, addr, pstore.PermanentAddrTTL) // t.Logf("connections from %s", s.LocalPeer()) // for _, c := range s.ConnectionsToPeer(dst) { // t.Logf("connection from %s to %s: %v", s.LocalPeer(), dst, c) diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 39ba6411b6..e0165262e3 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -6,9 +6,10 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" ci "github.com/ipfs/go-libp2p/testutil/ci" + peer "github.com/ipfs/go-libp2p-peer" + pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" context "golang.org/x/net/context" ) @@ -26,7 +27,7 @@ func TestSimultOpen(t *testing.T) { connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // copy for other peer log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.local, dst, addr) - s.peers.AddAddr(dst, addr, peer.PermanentAddrTTL) + s.peers.AddAddr(dst, addr, pstore.PermanentAddrTTL) if _, err := s.Dial(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 74e43dde44..69591d0caa 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -11,6 +11,7 @@ import ( "time" peer "github.com/ipfs/go-libp2p-peer" + pstore "github.com/ipfs/go-libp2p-peerstore" transport "github.com/ipfs/go-libp2p-transport" metrics "github.com/ipfs/go-libp2p/p2p/metrics" mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" @@ -67,7 +68,7 @@ func init() { type Swarm struct { swarm *ps.Swarm local peer.ID - peers peer.Peerstore + peers pstore.Peerstore connh ConnHandler dsync dialsync @@ -94,7 +95,7 @@ type Swarm struct { // NewSwarm constructs a Swarm, with a Chan. func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, - local peer.ID, peers peer.Peerstore, bwc metrics.Reporter) (*Swarm, error) { + local peer.ID, peers pstore.Peerstore, bwc metrics.Reporter) (*Swarm, error) { listenAddrs, err := filterAddrs(listenAddrs) if err != nil { diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 42ee299e87..0f9419bf34 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -3,11 +3,11 @@ package swarm import ( "testing" - peer "github.com/ipfs/go-libp2p-peer" metrics "github.com/ipfs/go-libp2p/p2p/metrics" addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" testutil "github.com/ipfs/go-libp2p/testutil" + pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" context "golang.org/x/net/context" ) @@ -63,7 +63,7 @@ func TestFilterAddrs(t *testing.T) { t.Fatal(err) } - ps := peer.NewPeerstore() + ps := pstore.NewPeerstore() ctx := context.Background() if _, err := NewNetwork(ctx, bad, id, ps, metrics.NewBandwidthCounter()); err == nil { @@ -111,7 +111,7 @@ func TestDialBadAddrs(t *testing.T) { test := func(a ma.Multiaddr) { p := testutil.RandPeerIDFatal(t) - s.peers.AddAddr(p, a, peer.PermanentAddrTTL) + s.peers.AddAddr(p, a, pstore.PermanentAddrTTL) if _, err := s.Dial(ctx, p); err == nil { t.Error("swarm should not dial: %s", m) } diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 84cc6326eb..f13da04083 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -4,6 +4,7 @@ import ( "fmt" peer "github.com/ipfs/go-libp2p-peer" + pstore "github.com/ipfs/go-libp2p-peerstore" metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" @@ -19,7 +20,7 @@ type Network Swarm // NewNetwork constructs a new network and starts listening on given addresses. func NewNetwork(ctx context.Context, listen []ma.Multiaddr, local peer.ID, - peers peer.Peerstore, bwc metrics.Reporter) (*Network, error) { + peers pstore.Peerstore, bwc metrics.Reporter) (*Network, error) { s, err := NewSwarm(ctx, listen, local, peers, bwc) if err != nil { @@ -63,7 +64,7 @@ func (n *Network) Peers() []peer.ID { } // Peers returns the Peerstore, which tracks known peers -func (n *Network) Peerstore() peer.Peerstore { +func (n *Network) Peerstore() pstore.Peerstore { return n.Swarm().peers } diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 6e9f121fcd..9160c4c6ae 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -9,11 +9,12 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" metrics "github.com/ipfs/go-libp2p/p2p/metrics" inet "github.com/ipfs/go-libp2p/p2p/net" testutil "github.com/ipfs/go-libp2p/testutil" + peer "github.com/ipfs/go-libp2p-peer" + pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" context "golang.org/x/net/context" ) @@ -52,7 +53,7 @@ func EchoStreamHandler(stream inet.Stream) { func makeDialOnlySwarm(ctx context.Context, t *testing.T) *Swarm { id := testutil.RandIdentityOrFatal(t) - peerstore := peer.NewPeerstore() + peerstore := pstore.NewPeerstore() peerstore.AddPubKey(id.ID(), id.PublicKey()) peerstore.AddPrivKey(id.ID(), id.PrivateKey()) @@ -72,7 +73,7 @@ func makeSwarms(ctx context.Context, t *testing.T, num int) []*Swarm { for i := 0; i < num; i++ { localnp := testutil.RandPeerNetParamsOrFatal(t) - peerstore := peer.NewPeerstore() + peerstore := pstore.NewPeerstore() peerstore.AddPubKey(localnp.ID, localnp.PubKey) peerstore.AddPrivKey(localnp.ID, localnp.PrivKey) @@ -94,7 +95,7 @@ func connectSwarms(t *testing.T, ctx context.Context, swarms []*Swarm) { var wg sync.WaitGroup connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // TODO: make a DialAddr func. - s.peers.AddAddr(dst, addr, peer.PermanentAddrTTL) + s.peers.AddAddr(dst, addr, pstore.PermanentAddrTTL) if _, err := s.Dial(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } @@ -313,13 +314,13 @@ func TestAddrBlocking(t *testing.T) { swarms[1].Filters.AddDialFilter(block) - swarms[1].peers.AddAddr(swarms[0].LocalPeer(), swarms[0].ListenAddresses()[0], peer.PermanentAddrTTL) + swarms[1].peers.AddAddr(swarms[0].LocalPeer(), swarms[0].ListenAddresses()[0], pstore.PermanentAddrTTL) _, err = swarms[1].Dial(ctx, swarms[0].LocalPeer()) if err == nil { t.Fatal("dial should have failed") } - swarms[0].peers.AddAddr(swarms[1].LocalPeer(), swarms[1].ListenAddresses()[0], peer.PermanentAddrTTL) + swarms[0].peers.AddAddr(swarms[1].LocalPeer(), swarms[1].ListenAddresses()[0], pstore.PermanentAddrTTL) _, err = swarms[0].Dial(ctx, swarms[1].LocalPeer()) if err == nil { t.Fatal("dial should have failed") From f374c546885999dd01d8e34d0b43922f1071f5d6 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 1 Jun 2016 10:02:26 -0700 Subject: [PATCH 0115/3965] improve comment on temp code --- p2p/net/swarm/swarm_dial.go | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 33b087efc3..1100ca544d 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -330,8 +330,14 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { s.Filters.AddrBlocked) */ - ////// TEMP UNTIL PEERSTORE GETS UPGRADED - // Ref: https://github.com/ipfs/go-libp2p-peer/pull/1 + ////// + /* + This code is temporary, the peerstore can currently provide + a channel as an interface for receiving addresses, but more thought + needs to be put into the execution. For now, this allows us to use + the improved rate limiter, while maintaining the outward behaviour + that we previously had (halting a dial when we run out of addrs) + */ paddrs := s.peers.Addrs(p) good_addrs := addrutil.FilterAddrs(paddrs, addrutil.AddrUsableFunc, From 21c40f8c976a36b6480dde98c17e1737e0224482 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 10 Jun 2016 16:13:11 -0700 Subject: [PATCH 0116/3965] add methods for setting protocols of a peer --- p2p/host/peerstore/peerstore.go | 21 +++++++++++++++++++++ p2p/host/peerstore/peerstore_test.go | 27 +++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 60ebcef523..3a88d58087 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -42,6 +42,9 @@ type Peerstore interface { // methods. this is a last resort. Get(id peer.ID, key string) (interface{}, error) Put(id peer.ID, key string, val interface{}) error + + GetProtocols(peer.ID) ([]string, error) + SetProtocols(peer.ID, []string) error } // AddrBook is an interface that fits the new AddrManager. I'm patching @@ -225,6 +228,24 @@ func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { } } +func (ps *peerstore) SetProtocols(p peer.ID, protos []string) error { + return ps.Put(p, "protocols", protos) +} + +func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { + protos, err := ps.Get(p, "protocols") + if err != nil { + return nil, err + } + + out, ok := protos.([]string) + if !ok { + return nil, errors.New("stored protocols array was not array of strings") + } + + return out, nil +} + func PeerInfos(ps Peerstore, peers []peer.ID) []PeerInfo { pi := make([]PeerInfo, len(peers)) for i, p := range peers { diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 365cc32826..2a0c28d575 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -179,3 +179,30 @@ func TestAddrStreamDuplicates(t *testing.T) { t.Fatal("should have received exactly ten addresses") } } + +func TestPeerstoreProtoStore(t *testing.T) { + ps := NewPeerstore() + p1 := peer.ID("TESTPEER") + + protos := []string{"a", "b", "c", "d"} + + err := ps.SetProtocols(p1, protos) + if err != nil { + t.Fatal(err) + } + + out, err := ps.GetProtocols(p1) + if err != nil { + t.Fatal(err) + } + + if len(out) != len(protos) { + t.Fatal("got wrong number of protocols back") + } + + for i, p := range protos { + if out[i] != p { + t.Fatal("got wrong protocol") + } + } +} From 47808266ff204de00bf33fdb78490741818cc175 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 15 Jun 2016 11:07:30 -0700 Subject: [PATCH 0117/3965] fix minor race condition in nat detection code --- p2p/net/nat/nat.go | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 237c1296ca..4dd6f0439e 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -164,6 +164,7 @@ type mapping struct { cached ma.Multiaddr cacheTime time.Time + cacheLk sync.Mutex } func (m *mapping) NAT() *NAT { @@ -203,8 +204,12 @@ func (m *mapping) InternalAddr() ma.Multiaddr { } func (m *mapping) ExternalAddr() (ma.Multiaddr, error) { - if time.Now().Sub(m.cacheTime) < CacheTime { - return m.cached, nil + m.cacheLk.Lock() + ctime := m.cacheTime + cval := m.cached + m.cacheLk.Unlock() + if time.Now().Sub(ctime) < CacheTime { + return cval, nil } if m.ExternalPort() == 0 { // dont even try right now. @@ -234,8 +239,10 @@ func (m *mapping) ExternalAddr() (ma.Multiaddr, error) { maddr2 := ipmaddr.Encapsulate(tcp) + m.cacheLk.Lock() m.cached = maddr2 m.cacheTime = time.Now() + m.cacheLk.Unlock() return maddr2, nil } From 7b26e0b818ce3a2bb855f76309556afa9ce2b63c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 3 Jul 2016 13:28:06 -0700 Subject: [PATCH 0118/3965] expose backoff manager in swarm --- p2p/net/swarm/swarm.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 100a9826d8..15e1ec5e1b 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -152,6 +152,7 @@ func (s *Swarm) AddAddrFilter(f string) error { s.Filters.AddDialFilter(m) return nil } + func filterAddrs(listenAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { if len(listenAddrs) > 0 { filtered := addrutil.FilterUsableAddrs(listenAddrs) @@ -285,6 +286,10 @@ func (s *Swarm) LocalPeer() peer.ID { return s.local } +func (s *Swarm) Backoff() *dialbackoff { + return &s.backf +} + // notifyAll sends a signal to all Notifiees func (s *Swarm) notifyAll(notify func(inet.Notifiee)) { s.notifmu.RLock() From 133ace853665ec95b2f40a2522de7aad44e453f4 Mon Sep 17 00:00:00 2001 From: John Steidley Date: Sat, 23 Jul 2016 13:24:45 -0700 Subject: [PATCH 0119/3965] gosimple --- p2p/net/swarm/dial_test.go | 4 ++-- p2p/net/swarm/swarm_net.go | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 515deb100f..031f3b3a5e 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -187,7 +187,7 @@ func TestDialWait(t *testing.T) { } else { t.Log("correctly got error:", err) } - duration := time.Now().Sub(before) + duration := time.Since(before) dt := s1.dialT if duration < dt*dialAttempts { @@ -451,7 +451,7 @@ func TestDialBackoffClears(t *testing.T) { } else { t.Log("correctly got error:", err) } - duration := time.Now().Sub(before) + duration := time.Since(before) dt := s1.dialT if duration < dt*dialAttempts { diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index f13da04083..92d2a801eb 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -124,7 +124,7 @@ func (n *Network) InterfaceListenAddresses() ([]ma.Multiaddr, error) { // For now only returns Connected || NotConnected. Expand into more later. func (n *Network) Connectedness(p peer.ID) inet.Connectedness { c := n.Swarm().ConnectionsToPeer(p) - if c != nil && len(c) > 0 { + if len(c) > 0 { return inet.Connected } return inet.NotConnected From 977485df98ca806114b17312b2c3f6dd62979240 Mon Sep 17 00:00:00 2001 From: John Steidley Date: Sat, 23 Jul 2016 13:24:45 -0700 Subject: [PATCH 0120/3965] gosimple --- p2p/net/nat/nat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 4dd6f0439e..7dac06df1d 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -208,7 +208,7 @@ func (m *mapping) ExternalAddr() (ma.Multiaddr, error) { ctime := m.cacheTime cval := m.cached m.cacheLk.Unlock() - if time.Now().Sub(ctime) < CacheTime { + if time.Since(ctime) < CacheTime { return cval, nil } From 06b858b3c09a706cbf8711c41eab5b9de84873c3 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 10 Aug 2016 09:15:53 -0700 Subject: [PATCH 0121/3965] gx publish v1.1.0 --- p2p/muxer/muxer-multistream/multistream.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 94b2320054..fdb76f02f6 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -3,6 +3,7 @@ package multistream import ( + "fmt" "net" mss "github.com/whyrusleeping/go-multistream" @@ -47,7 +48,10 @@ func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { proto = selected } - tpt := t.tpts[proto] + tpt, ok := t.tpts[proto] + if !ok { + return nil, fmt.Errorf("selected protocol we don't have a transport for") + } return tpt.NewConn(nc, isServer) } From f223e85a366e42106cdf8123980453a30a4d3600 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 10 Aug 2016 09:48:34 -0700 Subject: [PATCH 0122/3965] update multistream deps and fix code to work with new changes --- p2p/net/swarm/swarm.go | 4 ++-- p2p/net/swarm/swarm_notif_test.go | 10 ++++++++-- p2p/net/swarm/swarm_stream.go | 27 ++++++++++++++++++++------- 3 files changed, 30 insertions(+), 11 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 15e1ec5e1b..1fcf0f1c14 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -340,9 +340,9 @@ func (n *ps2netNotifee) Disconnected(c *ps.Conn) { } func (n *ps2netNotifee) OpenedStream(s *ps.Stream) { - n.not.OpenedStream(n.net, inet.Stream((*Stream)(s))) + n.not.OpenedStream(n.net, &Stream{stream: s}) } func (n *ps2netNotifee) ClosedStream(s *ps.Stream) { - n.not.ClosedStream(n.net, inet.Stream((*Stream)(s))) + n.not.ClosedStream(n.net, &Stream{stream: s}) } diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index aa4fca2448..baced67e61 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -10,6 +10,12 @@ import ( context "golang.org/x/net/context" ) +func streamsSame(a, b inet.Stream) bool { + sa := a.(*Stream) + sb := b.(*Stream) + return sa.Stream() == sb.Stream() +} + func TestNotifications(t *testing.T) { ctx := context.Background() swarms := makeSwarms(ctx, t, 5) @@ -98,7 +104,7 @@ func TestNotifications(t *testing.T) { case <-time.After(timeout): t.Fatal("timeout") } - if s != s2 { + if !streamsSame(s, s2) { t.Fatal("got incorrect stream", s.Conn(), s2.Conn()) } @@ -108,7 +114,7 @@ func TestNotifications(t *testing.T) { case <-time.After(timeout): t.Fatal("timeout") } - if s != s2 { + if !streamsSame(s, s2) { t.Fatal("got incorrect stream", s.Conn(), s2.Conn()) } } diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 7965d27439..16aa6e07c9 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -8,11 +8,14 @@ import ( // a Stream is a wrapper around a ps.Stream that exposes a way to get // our Conn and Swarm (instead of just the ps.Conn and ps.Swarm) -type Stream ps.Stream +type Stream struct { + stream *ps.Stream + protocol string +} // Stream returns the underlying peerstream.Stream func (s *Stream) Stream() *ps.Stream { - return (*ps.Stream)(s) + return s.stream } // Conn returns the Conn associated with this Stream, as an inet.Conn @@ -22,27 +25,37 @@ func (s *Stream) Conn() inet.Conn { // SwarmConn returns the Conn associated with this Stream, as a *Conn func (s *Stream) SwarmConn() *Conn { - return (*Conn)(s.Stream().Conn()) + return (*Conn)(s.stream.Conn()) } // Read reads bytes from a stream. func (s *Stream) Read(p []byte) (n int, err error) { - return s.Stream().Read(p) + return s.stream.Read(p) } // Write writes bytes to a stream, flushing for each call. func (s *Stream) Write(p []byte) (n int, err error) { - return s.Stream().Write(p) + return s.stream.Write(p) } // Close closes the stream, indicating this side is finished // with the stream. func (s *Stream) Close() error { - return s.Stream().Close() + return s.stream.Close() +} + +func (s *Stream) Protocol() string { + return s.protocol +} + +func (s *Stream) SetProtocol(p string) { + s.protocol = p } func wrapStream(pss *ps.Stream) *Stream { - return (*Stream)(pss) + return &Stream{ + stream: pss, + } } func wrapStreams(st []*ps.Stream) []*Stream { From a2a399b0ee18614276e5b75fe564df3bbd600bbd Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 10 Aug 2016 09:59:14 -0700 Subject: [PATCH 0123/3965] update local import paths to reflect org change --- p2p/net/swarm/addr/addr.go | 2 +- p2p/net/swarm/dial_test.go | 6 +++--- p2p/net/swarm/limiter.go | 4 ++-- p2p/net/swarm/limiter_test.go | 2 +- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm.go | 12 ++++++------ p2p/net/swarm/swarm_addr.go | 4 ++-- p2p/net/swarm/swarm_addr_test.go | 6 +++--- p2p/net/swarm/swarm_conn.go | 4 ++-- p2p/net/swarm/swarm_dial.go | 4 ++-- p2p/net/swarm/swarm_listen.go | 6 +++--- p2p/net/swarm/swarm_net.go | 4 ++-- p2p/net/swarm/swarm_net_test.go | 4 ++-- p2p/net/swarm/swarm_notif_test.go | 2 +- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 6 +++--- 16 files changed, 35 insertions(+), 35 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index d9ba87216a..7759625f11 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -9,7 +9,7 @@ import ( context "golang.org/x/net/context" ) -var log = logging.Logger("github.com/ipfs/go-libp2p/p2p/net/swarm/addr") +var log = logging.Logger("github.com/libp2p/go-libp2p/p2p/net/swarm/addr") // SupportedTransportStrings is the list of supported transports for the swarm. // These are strings of encapsulated multiaddr protocols. E.g.: diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 031f3b3a5e..cfdb996c28 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -6,9 +6,9 @@ import ( "testing" "time" - addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - testutil "github.com/ipfs/go-libp2p/testutil" - ci "github.com/ipfs/go-libp2p/testutil/ci" + addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" + testutil "github.com/libp2p/go-libp2p/testutil" + ci "github.com/libp2p/go-libp2p/testutil/ci" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 94ce05bb86..19fc84c329 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -7,8 +7,8 @@ import ( ma "github.com/jbenet/go-multiaddr" context "golang.org/x/net/context" - conn "github.com/ipfs/go-libp2p/p2p/net/conn" - addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" + conn "github.com/libp2p/go-libp2p/p2p/net/conn" + addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" ) type dialResult struct { diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 28733c5ab5..6ee6349a7b 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -12,7 +12,7 @@ import ( mafmt "github.com/whyrusleeping/mafmt" context "golang.org/x/net/context" - conn "github.com/ipfs/go-libp2p/p2p/net/conn" + conn "github.com/libp2p/go-libp2p/p2p/net/conn" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index e0165262e3..c6ee6cf985 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -6,7 +6,7 @@ import ( "testing" "time" - ci "github.com/ipfs/go-libp2p/testutil/ci" + ci "github.com/libp2p/go-libp2p/testutil/ci" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 1fcf0f1c14..a86edb372f 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -13,18 +13,18 @@ import ( peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" transport "github.com/ipfs/go-libp2p-transport" - metrics "github.com/ipfs/go-libp2p/p2p/metrics" - mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" - inet "github.com/ipfs/go-libp2p/p2p/net" - conn "github.com/ipfs/go-libp2p/p2p/net/conn" - filter "github.com/ipfs/go-libp2p/p2p/net/filter" - addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" logging "github.com/ipfs/go-log" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" pst "github.com/jbenet/go-stream-muxer" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" + metrics "github.com/libp2p/go-libp2p/p2p/metrics" + mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" + inet "github.com/libp2p/go-libp2p/p2p/net" + conn "github.com/libp2p/go-libp2p/p2p/net/conn" + filter "github.com/libp2p/go-libp2p/p2p/net/filter" + addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" psmss "github.com/whyrusleeping/go-smux-multistream" spdy "github.com/whyrusleeping/go-smux-spdystream" yamux "github.com/whyrusleeping/go-smux-yamux" diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 039683bfb9..85d4aafe85 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -1,8 +1,8 @@ package swarm import ( - conn "github.com/ipfs/go-libp2p/p2p/net/conn" - addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" + conn "github.com/libp2p/go-libp2p/p2p/net/conn" + addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" ma "github.com/jbenet/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 0f9419bf34..37eacbfd71 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -3,9 +3,9 @@ package swarm import ( "testing" - metrics "github.com/ipfs/go-libp2p/p2p/metrics" - addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" - testutil "github.com/ipfs/go-libp2p/testutil" + metrics "github.com/libp2p/go-libp2p/p2p/metrics" + addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" + testutil "github.com/libp2p/go-libp2p/testutil" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 96700ba835..36932784c9 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -3,8 +3,8 @@ package swarm import ( "fmt" - inet "github.com/ipfs/go-libp2p/p2p/net" - conn "github.com/ipfs/go-libp2p/p2p/net/conn" + inet "github.com/libp2p/go-libp2p/p2p/net" + conn "github.com/libp2p/go-libp2p/p2p/net/conn" ic "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 1100ca544d..2f4d29fa51 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -8,9 +8,9 @@ import ( lgbl "github.com/ipfs/go-libp2p-loggables" peer "github.com/ipfs/go-libp2p-peer" - conn "github.com/ipfs/go-libp2p/p2p/net/conn" - addrutil "github.com/ipfs/go-libp2p/p2p/net/swarm/addr" ma "github.com/jbenet/go-multiaddr" + conn "github.com/libp2p/go-libp2p/p2p/net/conn" + addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 38457f16fe..64fdb83bd0 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -5,11 +5,11 @@ import ( lgbl "github.com/ipfs/go-libp2p-loggables" transport "github.com/ipfs/go-libp2p-transport" - mconn "github.com/ipfs/go-libp2p/p2p/metrics/conn" - inet "github.com/ipfs/go-libp2p/p2p/net" - conn "github.com/ipfs/go-libp2p/p2p/net/conn" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" + mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" + inet "github.com/libp2p/go-libp2p/p2p/net" + conn "github.com/libp2p/go-libp2p/p2p/net/conn" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 92d2a801eb..8624dedba1 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -5,8 +5,8 @@ import ( peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" - metrics "github.com/ipfs/go-libp2p/p2p/metrics" - inet "github.com/ipfs/go-libp2p/p2p/net" + metrics "github.com/libp2p/go-libp2p/p2p/metrics" + inet "github.com/libp2p/go-libp2p/p2p/net" ma "github.com/jbenet/go-multiaddr" "github.com/jbenet/goprocess" diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 14c590b9d1..fddc8523fa 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -5,8 +5,8 @@ import ( "testing" "time" - inet "github.com/ipfs/go-libp2p/p2p/net" - testutil "github.com/ipfs/go-libp2p/p2p/test/util" + inet "github.com/libp2p/go-libp2p/p2p/net" + testutil "github.com/libp2p/go-libp2p/p2p/test/util" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index baced67e61..06777469f5 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -5,8 +5,8 @@ import ( "time" peer "github.com/ipfs/go-libp2p-peer" - inet "github.com/ipfs/go-libp2p/p2p/net" ma "github.com/jbenet/go-multiaddr" + inet "github.com/libp2p/go-libp2p/p2p/net" context "golang.org/x/net/context" ) diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 16aa6e07c9..b305bf917e 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -1,7 +1,7 @@ package swarm import ( - inet "github.com/ipfs/go-libp2p/p2p/net" + inet "github.com/libp2p/go-libp2p/p2p/net" ps "github.com/jbenet/go-peerstream" ) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index a19688d90b..c11377ed5e 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -9,9 +9,9 @@ import ( "testing" "time" - metrics "github.com/ipfs/go-libp2p/p2p/metrics" - inet "github.com/ipfs/go-libp2p/p2p/net" - testutil "github.com/ipfs/go-libp2p/testutil" + metrics "github.com/libp2p/go-libp2p/p2p/metrics" + inet "github.com/libp2p/go-libp2p/p2p/net" + testutil "github.com/libp2p/go-libp2p/testutil" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" From 2ca1eb131e40fb67233609157f1c17cee730b4a5 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 4 Aug 2016 05:47:26 -0700 Subject: [PATCH 0124/3965] lint: fixed a bunch of issues reported by gometalinter --- p2p/net/swarm/limiter_test.go | 22 +++++++++++----------- p2p/net/swarm/swarm.go | 8 +++++++- p2p/net/swarm/swarm_addr.go | 12 ------------ p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_conn.go | 3 ++- p2p/net/swarm/swarm_dial.go | 24 +++++++++++++++--------- p2p/net/swarm/swarm_net.go | 4 ++-- p2p/net/swarm/swarm_stream.go | 2 +- 8 files changed, 39 insertions(+), 38 deletions(-) diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 6ee6349a7b..f5fc18746f 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -63,10 +63,10 @@ func hangDialFunc(hang chan struct{}) dialfunc { if tcpPortOver(a, 10) { return conn.Conn(nil), nil - } else { - <-hang - return nil, fmt.Errorf("test bad dial") } + + <-hang + return nil, fmt.Errorf("test bad dial") } } @@ -127,7 +127,7 @@ func TestFDLimiting(t *testing.T) { bads := []ma.Multiaddr{addrWithPort(t, 1), addrWithPort(t, 2), addrWithPort(t, 3), addrWithPort(t, 4)} pids := []peer.ID{"testpeer1", "testpeer2", "testpeer3", "testpeer4"} - good_tcp := addrWithPort(t, 20) + goodTCP := addrWithPort(t, 20) ctx := context.Background() resch := make(chan dialResult) @@ -143,7 +143,7 @@ func TestFDLimiting(t *testing.T) { l.AddDialJob(&dialJob{ ctx: ctx, peer: pid, - addr: good_tcp, + addr: goodTCP, resp: resch, }) } @@ -175,10 +175,10 @@ func TestTokenRedistribution(t *testing.T) { df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { if tcpPortOver(a, 10) { return (conn.Conn)(nil), nil - } else { - <-hangchs[p] - return nil, fmt.Errorf("test bad dial") } + + <-hangchs[p] + return nil, fmt.Errorf("test bad dial") } l := newDialLimiterWithParams(df, 8, 4) @@ -264,10 +264,10 @@ func TestStressLimiter(t *testing.T) { df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { if tcpPortOver(a, 1000) { return conn.Conn(nil), nil - } else { - time.Sleep(time.Millisecond * time.Duration(5+rand.Intn(100))) - return nil, fmt.Errorf("test bad dial") } + + time.Sleep(time.Millisecond * time.Duration(5+rand.Intn(100))) + return nil, fmt.Errorf("test bad dial") } l := newDialLimiterWithParams(df, 20, 5) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index a86edb372f..704beb761e 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -1,4 +1,4 @@ -// package swarm implements a connection muxer with a pair of channels +// Package swarm implements a connection muxer with a pair of channels // to synchronize all network communication. package swarm @@ -34,6 +34,8 @@ import ( var log = logging.Logger("swarm2") +// PSTransport is the default peerstream transport that will be used by +// any libp2p swarms. var PSTransport pst.Transport func init() { @@ -143,6 +145,8 @@ func (s *Swarm) teardown() error { return s.swarm.Close() } +// AddAddrFilter adds a multiaddr filter to the set of filters the swarm will +// use to determine which addresses not to dial to. func (s *Swarm) AddAddrFilter(f string) error { m, err := mafilter.NewMask(f) if err != nil { @@ -165,6 +169,7 @@ func filterAddrs(listenAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { return listenAddrs, nil } +// Listen sets up listeners for all of the given addresses func (s *Swarm) Listen(addrs ...ma.Multiaddr) error { addrs, err := filterAddrs(addrs) if err != nil { @@ -286,6 +291,7 @@ func (s *Swarm) LocalPeer() peer.ID { return s.local } +// Backoff returns the dialbackoff object for this swarm. func (s *Swarm) Backoff() *dialbackoff { return &s.backf } diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 85d4aafe85..39fbe02afc 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -25,15 +25,3 @@ func (s *Swarm) ListenAddresses() []ma.Multiaddr { func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { return addrutil.ResolveUnspecifiedAddresses(s.ListenAddresses(), nil) } - -// checkNATWarning checks if our observed addresses differ. if so, -// informs the user that certain things might not work yet -func checkNATWarning(s *Swarm, observed ma.Multiaddr, expected ma.Multiaddr) { - listen, err := s.InterfaceListenAddresses() - if err != nil { - log.Debugf("Error retrieving swarm.InterfaceListenAddresses: %s", err) - return - } - - addrutil.CheckNATWarning(observed, expected, listen) -} diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 37eacbfd71..772413b286 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -113,7 +113,7 @@ func TestDialBadAddrs(t *testing.T) { p := testutil.RandPeerIDFatal(t) s.peers.AddAddr(p, a, pstore.PermanentAddrTTL) if _, err := s.Dial(ctx, p); err == nil { - t.Error("swarm should not dial: %s", m) + t.Errorf("swarm should not dial: %s", p) } } diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 36932784c9..762ba6e1d3 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -13,7 +13,7 @@ import ( context "golang.org/x/net/context" ) -// a Conn is a simple wrapper around a ps.Conn that also exposes +// Conn is a simple wrapper around a ps.Conn that also exposes // some of the methods from the underlying conn.Conn. // There's **five** "layers" to each connection: // * 0. the net.Conn - underlying net.Conn (TCP/UDP/UTP/etc) @@ -87,6 +87,7 @@ func (c *Conn) NewStream() (inet.Stream, error) { return inet.Stream(s), err } +// Close closes the underlying stream connection func (c *Conn) Close() error { return c.StreamConn().Close() } diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 2f4d29fa51..6774721247 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -26,9 +26,15 @@ import ( // retry dialAttempt x var ( + // ErrDialBackoff is returned by the backoff code when a given peer has + // been dialed too frequently ErrDialBackoff = errors.New("dial backoff") - ErrDialFailed = errors.New("dial attempt failed") - ErrDialToSelf = errors.New("dial to self attempted") + + // ErrDialFailed is returned when connecting to a peer has ultimately failed + ErrDialFailed = errors.New("dial attempt failed") + + // ErrDialToSelf is returned if we attempt to dial our own peer + ErrDialToSelf = errors.New("dial to self attempted") ) // dialAttempts governs how many times a goroutine will try to dial a given peer. @@ -45,7 +51,7 @@ const defaultPerPeerRateLimit = 8 // DialTimeout is the amount of time each dial attempt has. We can think about making // this larger down the road, or putting more granular timeouts (i.e. within each // subcomponent of Dial) -var DialTimeout time.Duration = time.Second * 10 +var DialTimeout = time.Second * 10 // dialsync is a small object that helps manage ongoing dials. // this way, if we receive many simultaneous dial requests, one @@ -320,13 +326,13 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { } ila, _ := s.InterfaceListenAddresses() - subtract_filter := addrutil.SubtractFilter(append(ila, s.peers.Addrs(s.local)...)...) + subtractFilter := addrutil.SubtractFilter(append(ila, s.peers.Addrs(s.local)...)...) // get live channel of addresses for peer, filtered by the given filters /* remoteAddrChan := s.peers.AddrsChan(ctx, p, addrutil.AddrUsableFilter, - subtract_filter, + subtractFilter, s.Filters.AddrBlocked) */ @@ -339,13 +345,13 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { that we previously had (halting a dial when we run out of addrs) */ paddrs := s.peers.Addrs(p) - good_addrs := addrutil.FilterAddrs(paddrs, + goodAddrs := addrutil.FilterAddrs(paddrs, addrutil.AddrUsableFunc, - subtract_filter, + subtractFilter, addrutil.FilterNeg(s.Filters.AddrBlocked), ) - remoteAddrChan := make(chan ma.Multiaddr, len(good_addrs)) - for _, a := range good_addrs { + remoteAddrChan := make(chan ma.Multiaddr, len(goodAddrs)) + for _, a := range goodAddrs { remoteAddrChan <- a } close(remoteAddrChan) diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 8624dedba1..48defb11ad 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -63,7 +63,7 @@ func (n *Network) Peers() []peer.ID { return n.Swarm().Peers() } -// Peers returns the Peerstore, which tracks known peers +// Peerstore returns the Peerstore, which tracks known peers func (n *Network) Peerstore() pstore.Peerstore { return n.Swarm().peers } @@ -142,7 +142,7 @@ func (n *Network) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) return inet.Stream(s), nil } -// SetHandler sets the protocol handler on the Network's Muxer. +// SetStreamHandler sets the protocol handler on the Network's Muxer. // This operation is threadsafe. func (n *Network) SetStreamHandler(h inet.StreamHandler) { n.Swarm().SetStreamHandler(h) diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index b305bf917e..86719819a5 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -6,7 +6,7 @@ import ( ps "github.com/jbenet/go-peerstream" ) -// a Stream is a wrapper around a ps.Stream that exposes a way to get +// Stream is a wrapper around a ps.Stream that exposes a way to get // our Conn and Swarm (instead of just the ps.Conn and ps.Swarm) type Stream struct { stream *ps.Stream From 3471cda23c3fd24e9e9de2c1197b20cce11a2402 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 16 Aug 2016 15:04:24 -0700 Subject: [PATCH 0125/3965] p2p/net/swarm: improve code coverage --- p2p/net/swarm/limiter.go | 12 ---- p2p/net/swarm/swarm_addr.go | 12 ---- p2p/net/swarm/swarm_net_test.go | 104 +++++++++++++++++++++++++++++--- 3 files changed, 96 insertions(+), 32 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 19fc84c329..4bbfb11862 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -131,18 +131,6 @@ func (dl *dialLimiter) AddDialJob(dj *dialJob) { go dl.executeDial(dj) } -func (dl *dialLimiter) schedulePerPeerDial(j *dialJob) { - if dl.activePerPeer[j.peer] >= dl.perPeerLimit { - wlist := dl.waitingOnPeerLimit[j.peer] - dl.waitingOnPeerLimit[j.peer] = append(wlist, j) - return - } - - // take second needed token and start dial! - dl.activePerPeer[j.peer]++ - go dl.executeDial(j) -} - // executeDial calls the dialFunc, and reports the result through the response // channel when finished. Once the response is sent it also releases all tokens // it held during the dial. diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 85d4aafe85..39fbe02afc 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -25,15 +25,3 @@ func (s *Swarm) ListenAddresses() []ma.Multiaddr { func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { return addrutil.ResolveUnspecifiedAddresses(s.ListenAddresses(), nil) } - -// checkNATWarning checks if our observed addresses differ. if so, -// informs the user that certain things might not work yet -func checkNATWarning(s *Swarm, observed ma.Multiaddr, expected ma.Multiaddr) { - listen, err := s.InterfaceListenAddresses() - if err != nil { - log.Debugf("Error retrieving swarm.InterfaceListenAddresses: %s", err) - return - } - - addrutil.CheckNATWarning(observed, expected, listen) -} diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index fddc8523fa..1294e899ba 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -35,9 +35,10 @@ func TestConnectednessCorrect(t *testing.T) { dial(nets[1], nets[2]) dial(nets[3], nets[2]) - // there's something wrong with dial, i think. it's not finishing - // completely. there must be some async stuff. - <-time.After(100 * time.Millisecond) + // The notifications for new connections get sent out asynchronously. + // There is the potential for a race condition here, so we sleep to ensure + // that they have been received. + time.Sleep(time.Millisecond * 100) // test those connected show up correctly @@ -51,20 +52,44 @@ func TestConnectednessCorrect(t *testing.T) { expectConnectedness(t, nets[0], nets[2], inet.NotConnected) expectConnectedness(t, nets[1], nets[3], inet.NotConnected) + if len(nets[0].Peers()) != 2 { + t.Fatal("expected net 0 to have two peers") + } + + if len(nets[2].Conns()) != 2 { + t.Fatal("expected net 2 to have two conns") + } + + if len(nets[1].ConnsToPeer(nets[3].LocalPeer())) != 0 { + t.Fatal("net 1 should have no connections to net 3") + } + + if err := nets[2].ClosePeer(nets[1].LocalPeer()); err != nil { + t.Fatal(err) + } + + expectConnectedness(t, nets[2], nets[1], inet.NotConnected) + for _, n := range nets { n.Close() } + + for _, n := range nets { + <-n.Process().Closed() + } } func expectConnectedness(t *testing.T, a, b inet.Network, expected inet.Connectedness) { - es := "%s is connected to %s, but Connectedness incorrect. %s %s" - if a.Connectedness(b.LocalPeer()) != expected { - t.Errorf(es, a, b, printConns(a), printConns(b)) + es := "%s is connected to %s, but Connectedness incorrect. %s %s %s" + atob := a.Connectedness(b.LocalPeer()) + btoa := b.Connectedness(a.LocalPeer()) + if atob != expected { + t.Errorf(es, a, b, printConns(a), printConns(b), atob) } // test symmetric case - if b.Connectedness(a.LocalPeer()) != expected { - t.Errorf(es, b, a, printConns(b), printConns(a)) + if btoa != expected { + t.Errorf(es, b, a, printConns(b), printConns(a), btoa) } } @@ -75,3 +100,66 @@ func printConns(n inet.Network) string { } return s } + +func TestNetworkOpenStream(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + nets := make([]inet.Network, 4) + for i := 0; i < 4; i++ { + nets[i] = testutil.GenSwarmNetwork(t, ctx) + } + + dial := func(a, b inet.Network) { + testutil.DivulgeAddresses(b, a) + if _, err := a.DialPeer(ctx, b.LocalPeer()); err != nil { + t.Fatalf("Failed to dial: %s", err) + } + } + + dial(nets[0], nets[1]) + dial(nets[0], nets[3]) + dial(nets[1], nets[2]) + + done := make(chan bool) + nets[1].SetStreamHandler(func(s inet.Stream) { + defer close(done) + defer s.Close() + + buf := make([]byte, 10) + _, err := s.Read(buf) + if err != nil { + t.Error(err) + return + } + if string(buf) != "hello ipfs" { + t.Error("got wrong message") + } + }) + + s, err := nets[0].NewStream(ctx, nets[1].LocalPeer()) + if err != nil { + t.Fatal(err) + } + + _, err = s.Write([]byte("hello ipfs")) + if err != nil { + t.Fatal(err) + } + + err = s.Close() + if err != nil { + t.Fatal(err) + } + + select { + case <-done: + case <-time.After(time.Millisecond * 100): + t.Fatal("timed out waiting on stream") + } + + _, err = nets[1].NewStream(ctx, nets[3].LocalPeer()) + if err == nil { + t.Fatal("expected stream open 1->3 to fail") + } +} From 873e29641af5dcda8a75f80c7de48fc5d5be5351 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 17 Aug 2016 18:40:08 -0700 Subject: [PATCH 0126/3965] swarm: make stream.Protocol() return type protocol.ID --- p2p/net/swarm/swarm_stream.go | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 86719819a5..dc365e8c09 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -2,6 +2,7 @@ package swarm import ( inet "github.com/libp2p/go-libp2p/p2p/net" + protocol "github.com/libp2p/go-libp2p/p2p/protocol" ps "github.com/jbenet/go-peerstream" ) @@ -10,7 +11,7 @@ import ( // our Conn and Swarm (instead of just the ps.Conn and ps.Swarm) type Stream struct { stream *ps.Stream - protocol string + protocol protocol.ID } // Stream returns the underlying peerstream.Stream @@ -44,11 +45,11 @@ func (s *Stream) Close() error { return s.stream.Close() } -func (s *Stream) Protocol() string { +func (s *Stream) Protocol() protocol.ID { return s.protocol } -func (s *Stream) SetProtocol(p string) { +func (s *Stream) SetProtocol(p protocol.ID) { s.protocol = p } From 275e18319f81683d7bc207da0e145da5cae3bc36 Mon Sep 17 00:00:00 2001 From: Jeromy Johnson Date: Thu, 18 Aug 2016 11:19:20 -0700 Subject: [PATCH 0127/3965] sleep before final connectedness check --- p2p/net/swarm/swarm_net_test.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 1294e899ba..4d564e3901 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -68,6 +68,8 @@ func TestConnectednessCorrect(t *testing.T) { t.Fatal(err) } + time.Sleep(time.Millisecond * 50) + expectConnectedness(t, nets[2], nets[1], inet.NotConnected) for _, n := range nets { From b4b69e1c9398819b3d94344eab43065531a4c4f2 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 19 Aug 2016 14:16:56 -0700 Subject: [PATCH 0128/3965] rework protocol storage and handling to better support upper layers needs --- p2p/host/peerstore/peerstore.go | 62 ++++++++++++++++++++++++---- p2p/host/peerstore/peerstore_test.go | 17 +++++++- 2 files changed, 70 insertions(+), 9 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 3a88d58087..166be36424 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -2,6 +2,7 @@ package peerstore import ( "errors" + "fmt" "sync" "time" @@ -44,7 +45,8 @@ type Peerstore interface { Put(id peer.ID, key string, val interface{}) error GetProtocols(peer.ID) ([]string, error) - SetProtocols(peer.ID, []string) error + AddProtocols(peer.ID, ...string) error + SupportsProtocols(peer.ID, ...string) ([]string, error) } // AddrBook is an interface that fits the new AddrManager. I'm patching @@ -192,6 +194,8 @@ func (ps *peerstore) Put(p peer.ID, key string, val interface{}) error { return nil } +var ErrNotFound = errors.New("item not found") + func (ps *peerstore) Get(p peer.ID, key string) (interface{}, error) { //dsk := ds.NewKey(string(p) + "/" + key) //return ps.ds.Get(dsk) @@ -200,7 +204,7 @@ func (ps *peerstore) Get(p peer.ID, key string) (interface{}, error) { defer ps.dslock.Unlock() i, ok := ps.ds[string(p)+"/"+key] if !ok { - return nil, errors.New("item not found") + return nil, ErrNotFound } return i, nil } @@ -228,19 +232,61 @@ func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { } } -func (ps *peerstore) SetProtocols(p peer.ID, protos []string) error { - return ps.Put(p, "protocols", protos) +func (ps *peerstore) AddProtocols(p peer.ID, protos ...string) error { + protomap, err := ps.getProtocolMap(p) + if err != nil { + return err + } + + for _, proto := range protos { + protomap[proto] = struct{}{} + } + + return ps.Put(p, "protocols", protomap) +} + +func (ps *peerstore) getProtocolMap(p peer.ID) (map[string]struct{}, error) { + iprotomap, err := ps.Get(p, "protocols") + switch err { + default: + return nil, err + case ErrNotFound: + return make(map[string]struct{}), nil + case nil: + cast, ok := iprotomap.(map[string]struct{}) + if !ok { + return nil, fmt.Errorf("stored protocol set was not a map") + } + + return cast, nil + } } func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { - protos, err := ps.Get(p, "protocols") + pmap, err := ps.getProtocolMap(p) if err != nil { return nil, err } - out, ok := protos.([]string) - if !ok { - return nil, errors.New("stored protocols array was not array of strings") + var out []string + for k, _ := range pmap { + out = append(out, k) + } + + return out, nil +} + +func (ps *peerstore) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { + pmap, err := ps.getProtocolMap(p) + if err != nil { + return nil, err + } + + var out []string + for _, proto := range protos { + if _, ok := pmap[proto]; ok { + out = append(out, proto) + } } return out, nil diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 2a0c28d575..7aedb9021a 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -3,6 +3,7 @@ package peerstore import ( "fmt" "math/rand" + "sort" "testing" "time" @@ -186,7 +187,7 @@ func TestPeerstoreProtoStore(t *testing.T) { protos := []string{"a", "b", "c", "d"} - err := ps.SetProtocols(p1, protos) + err := ps.AddProtocols(p1, protos...) if err != nil { t.Fatal(err) } @@ -200,9 +201,23 @@ func TestPeerstoreProtoStore(t *testing.T) { t.Fatal("got wrong number of protocols back") } + sort.Strings(out) for i, p := range protos { if out[i] != p { t.Fatal("got wrong protocol") } } + + supported, err := ps.SupportsProtocols(p1, "q", "w", "a", "y", "b") + if err != nil { + t.Fatal(err) + } + + if len(supported) != 2 { + t.Fatal("only expected 2 supported") + } + + if supported[0] != "a" || supported[1] != "b" { + t.Fatal("got wrong supported array: ", supported) + } } From 5becb9137951f46d44e3a02b9e4f45af9e48fb2c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 19 Aug 2016 14:30:10 -0700 Subject: [PATCH 0129/3965] improve test coverage --- p2p/host/peerstore/peerinfo_test.go | 57 ++++++++++++++++++++++++++++ p2p/host/peerstore/peerstore_test.go | 22 +++++++++++ 2 files changed, 79 insertions(+) create mode 100644 p2p/host/peerstore/peerinfo_test.go diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go new file mode 100644 index 0000000000..b78a662999 --- /dev/null +++ b/p2p/host/peerstore/peerinfo_test.go @@ -0,0 +1,57 @@ +package peerstore + +import ( + peer "github.com/ipfs/go-libp2p-peer" + ma "github.com/jbenet/go-multiaddr" + "testing" +) + +func mustAddr(t *testing.T, s string) ma.Multiaddr { + addr, err := ma.NewMultiaddr(s) + if err != nil { + t.Fatal(err) + } + + return addr +} + +func TestPeerInfoMarshal(t *testing.T) { + a := mustAddr(t, "/ip4/1.2.3.4/tcp/4536") + b := mustAddr(t, "/ip4/1.2.3.8/udp/7777") + id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") + if err != nil { + t.Fatal(err) + } + + pi := &PeerInfo{ + ID: id, + Addrs: []ma.Multiaddr{a, b}, + } + + data, err := pi.MarshalJSON() + if err != nil { + t.Fatal(err) + } + + pi2 := new(PeerInfo) + if err := pi2.UnmarshalJSON(data); err != nil { + t.Fatal(err) + } + + if pi2.ID != pi.ID { + t.Fatal("ids didnt match after marshal") + } + + if !pi.Addrs[0].Equal(pi2.Addrs[0]) { + t.Fatal("wrong addrs") + } + + if !pi.Addrs[1].Equal(pi2.Addrs[1]) { + t.Fatal("wrong addrs") + } + + lgbl := pi2.Loggable() + if lgbl["peerID"] != id.Pretty() { + t.Fatal("loggables gave wrong peerID output") + } +} diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 7aedb9021a..44d7c2259d 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -221,3 +221,25 @@ func TestPeerstoreProtoStore(t *testing.T) { t.Fatal("got wrong supported array: ", supported) } } + +func TestBasicPeerstore(t *testing.T) { + ps := NewPeerstore() + + var pids []peer.ID + addrs := getAddrs(t, 10) + for i, a := range addrs { + p := peer.ID(fmt.Sprint(i)) + pids = append(pids, p) + ps.AddAddr(p, a, PermanentAddrTTL) + } + + peers := ps.Peers() + if len(peers) != 10 { + t.Fatal("expected ten peers") + } + + pinfo := ps.PeerInfo(pids[0]) + if !pinfo.Addrs[0].Equal(addrs[0]) { + t.Fatal("stored wrong address") + } +} From f5f832ccfb8a9bff7966fc2a4a5cb6ba91f84409 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 19 Aug 2016 14:51:59 -0700 Subject: [PATCH 0130/3965] make protocol methods threadsafe --- p2p/host/peerstore/peerstore.go | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 166be36424..7518b453da 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -68,7 +68,7 @@ type AddrBook interface { // This is used when we receive the best estimate of the validity of an address. SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) - // Addresses returns all known (and valid) addresses for a given + // Addresses returns all known (and valid) addresses for a given peer Addrs(p peer.ID) []ma.Multiaddr // AddrStream returns a channel that gets all addresses for a given @@ -172,6 +172,9 @@ type peerstore struct { // TODO: use a datastore for this ds map[string]interface{} dslock sync.Mutex + + // lock for protocol information, separate from datastore lock + protolock sync.Mutex } // NewPeerstore creates a threadsafe collection of peers. @@ -233,6 +236,8 @@ func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { } func (ps *peerstore) AddProtocols(p peer.ID, protos ...string) error { + ps.protolock.Lock() + defer ps.protolock.Unlock() protomap, err := ps.getProtocolMap(p) if err != nil { return err @@ -263,6 +268,8 @@ func (ps *peerstore) getProtocolMap(p peer.ID) (map[string]struct{}, error) { } func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { + ps.protolock.Lock() + defer ps.protolock.Unlock() pmap, err := ps.getProtocolMap(p) if err != nil { return nil, err @@ -277,6 +284,8 @@ func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { } func (ps *peerstore) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { + ps.protolock.Lock() + defer ps.protolock.Unlock() pmap, err := ps.getProtocolMap(p) if err != nil { return nil, err From 5d09bed16b773696d277ee3908587bbd5e93b127 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 23 Aug 2016 17:31:26 -0700 Subject: [PATCH 0131/3965] gx publish version 1.0.2 --- p2p/transport/websocket/websocket.go | 8 ++++---- p2p/transport/websocket/ws_test.go | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index e37cb98a85..fef3673ddd 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -7,13 +7,13 @@ import ( "net/http" "net/url" - tpt "github.com/ipfs/go-libp2p-transport" - manet "github.com/jbenet/go-multiaddr-net" - mafmt "github.com/whyrusleeping/mafmt" ws "golang.org/x/net/websocket" + manet "gx/ipfs/QmPpRcbNUXauP3zWZ1NJMLWpe4QnmEHrd2ba2D3yqWznw7/go-multiaddr-net" + tpt "gx/ipfs/QmbRuJ16EfPGWuEguoNxwPacpeHFUAZb2XQXVvUt5Vxg5q/go-libp2p-transport" + mafmt "gx/ipfs/QmeLQ13LftT9XhNn22piZc3GP56fGqhijuL5Y8KdUaRn1g/mafmt" - ma "github.com/jbenet/go-multiaddr" "golang.org/x/net/context" + ma "gx/ipfs/QmYzDkkgAEmrcNzFCiYo6L1dTX4EAG1gZkbtdbd9trL4vd/go-multiaddr" ) var WsProtocol = ma.Protocol{ diff --git a/p2p/transport/websocket/ws_test.go b/p2p/transport/websocket/ws_test.go index 5bda992d17..9a75c9437d 100644 --- a/p2p/transport/websocket/ws_test.go +++ b/p2p/transport/websocket/ws_test.go @@ -5,7 +5,7 @@ import ( "testing" "time" - ma "github.com/jbenet/go-multiaddr" + ma "gx/ipfs/QmYzDkkgAEmrcNzFCiYo6L1dTX4EAG1gZkbtdbd9trL4vd/go-multiaddr" ) var _ = log.Println From 6cb2231c24118a34aa8b5b6fbfff11ff2a33c9a9 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 23 Aug 2016 19:21:43 -0700 Subject: [PATCH 0132/3965] gx publish version 1.0.3 --- p2p/transport/websocket/websocket.go | 10 ++++++---- p2p/transport/websocket/ws_test.go | 2 +- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index fef3673ddd..bdae6a37c5 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -7,13 +7,13 @@ import ( "net/http" "net/url" + tpt "github.com/ipfs/go-libp2p-transport" + manet "github.com/jbenet/go-multiaddr-net" + mafmt "github.com/whyrusleeping/mafmt" ws "golang.org/x/net/websocket" - manet "gx/ipfs/QmPpRcbNUXauP3zWZ1NJMLWpe4QnmEHrd2ba2D3yqWznw7/go-multiaddr-net" - tpt "gx/ipfs/QmbRuJ16EfPGWuEguoNxwPacpeHFUAZb2XQXVvUt5Vxg5q/go-libp2p-transport" - mafmt "gx/ipfs/QmeLQ13LftT9XhNn22piZc3GP56fGqhijuL5Y8KdUaRn1g/mafmt" + ma "github.com/jbenet/go-multiaddr" "golang.org/x/net/context" - ma "gx/ipfs/QmYzDkkgAEmrcNzFCiYo6L1dTX4EAG1gZkbtdbd9trL4vd/go-multiaddr" ) var WsProtocol = ma.Protocol{ @@ -221,3 +221,5 @@ func (l *listener) Multiaddr() ma.Multiaddr { return l.Listener.Multiaddr().Encapsulate(wsma) } + +var _ tpt.Transport = (*WebsocketTransport)(nil) diff --git a/p2p/transport/websocket/ws_test.go b/p2p/transport/websocket/ws_test.go index 9a75c9437d..5bda992d17 100644 --- a/p2p/transport/websocket/ws_test.go +++ b/p2p/transport/websocket/ws_test.go @@ -5,7 +5,7 @@ import ( "testing" "time" - ma "gx/ipfs/QmYzDkkgAEmrcNzFCiYo6L1dTX4EAG1gZkbtdbd9trL4vd/go-multiaddr" + ma "github.com/jbenet/go-multiaddr" ) var _ = log.Println From 10b6a10746ffd7618b8159f7fd54fcf7eff1cea5 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 23 Aug 2016 22:04:22 -0700 Subject: [PATCH 0133/3965] some cleanup --- p2p/transport/websocket/websocket.go | 4 +--- p2p/transport/websocket/ws_test.go | 19 +++++++++---------- 2 files changed, 10 insertions(+), 13 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index bdae6a37c5..bd66b99d82 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -2,7 +2,6 @@ package websocket import ( "fmt" - "log" "net" "net/http" "net/url" @@ -34,7 +33,7 @@ var WsCodec = &manet.NetCodec{ func init() { err := ma.AddProtocol(WsProtocol) if err != nil { - log.Fatalf("error registering websocket protocol: %s", err) + panic(fmt.Errorf("error registering websocket protocol: %s", err)) } manet.RegisterNetCodec(WsCodec) @@ -102,7 +101,6 @@ func parseMultiaddr(a ma.Multiaddr) (string, error) { func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { wsurl, err := parseMultiaddr(raddr) if err != nil { - log.Println("parse multiaddr failed: ", err) return nil, err } diff --git a/p2p/transport/websocket/ws_test.go b/p2p/transport/websocket/ws_test.go index 5bda992d17..678a4cbeb9 100644 --- a/p2p/transport/websocket/ws_test.go +++ b/p2p/transport/websocket/ws_test.go @@ -1,15 +1,12 @@ package websocket import ( - "log" + "bytes" "testing" - "time" ma "github.com/jbenet/go-multiaddr" ) -var _ = log.Println - func TestMultiaddrParsing(t *testing.T) { addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") if err != nil { @@ -33,6 +30,9 @@ func TestWebsocketListen(t *testing.T) { if err != nil { t.Fatal(err) } + defer l.Close() + + msg := []byte("HELLO WORLD") go func() { d, _ := tpt.Dialer(nil) @@ -42,8 +42,7 @@ func TestWebsocketListen(t *testing.T) { return } - c.Write([]byte("HELLO WORLD!")) - time.Sleep(time.Second) + c.Write(msg) c.Close() }() @@ -51,6 +50,7 @@ func TestWebsocketListen(t *testing.T) { if err != nil { t.Fatal(err) } + defer c.Close() buf := make([]byte, 32) n, err := c.Read(buf) @@ -58,8 +58,7 @@ func TestWebsocketListen(t *testing.T) { t.Fatal(err) } - log.Printf("READ: %s", buf[:n]) - c.Close() - l.Close() - time.Sleep(time.Second) + if !bytes.Equal(buf[:n], msg) { + t.Fatal("got wrong message", buf[:n], msg) + } } From f87a8406e9f14d606e739362853f6b7b3126e7ec Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 24 Aug 2016 09:27:17 -0700 Subject: [PATCH 0134/3965] gx publish version 1.0.5, use builtin context --- p2p/transport/websocket/websocket.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index bd66b99d82..f048c006cb 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -1,6 +1,7 @@ package websocket import ( + "context" "fmt" "net" "net/http" @@ -12,7 +13,6 @@ import ( ws "golang.org/x/net/websocket" ma "github.com/jbenet/go-multiaddr" - "golang.org/x/net/context" ) var WsProtocol = ma.Protocol{ From b619447584cafbce0304db59e24136c376568fb8 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 16 May 2016 15:27:58 -0700 Subject: [PATCH 0135/3965] add in experimental websocket support --- p2p/net/swarm/addr/addr.go | 4 ++++ p2p/net/swarm/addr/addr_test.go | 1 + p2p/net/swarm/swarm.go | 2 ++ 3 files changed, 7 insertions(+) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 7759625f11..08ca90ac46 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -7,6 +7,8 @@ import ( ma "github.com/jbenet/go-multiaddr" manet "github.com/jbenet/go-multiaddr-net" context "golang.org/x/net/context" + + _ "github.com/whyrusleeping/ws-transport" ) var log = logging.Logger("github.com/libp2p/go-libp2p/p2p/net/swarm/addr") @@ -19,6 +21,8 @@ var SupportedTransportStrings = []string{ "/ip6/tcp", "/ip4/udp/utp", "/ip6/udp/utp", + "/ip4/tcp/ws", + "/ip6/tcp/ws", // "/ip4/udp/udt", disabled because the lib doesnt work on arm // "/ip6/udp/udt", disabled because the lib doesnt work on arm } diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go index 17039967e3..213b7b9f1a 100644 --- a/p2p/net/swarm/addr/addr_test.go +++ b/p2p/net/swarm/addr/addr_test.go @@ -29,6 +29,7 @@ func TestFilterAddrs(t *testing.T) { newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), newMultiaddr(t, "/ip6/::1/tcp/1234"), newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/utp"), + newMultiaddr(t, "/ip4/1.2.3.4/tcp/1234/ws"), } goodAndBad := append(good, bad...) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 704beb761e..c342f411c4 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -29,6 +29,7 @@ import ( spdy "github.com/whyrusleeping/go-smux-spdystream" yamux "github.com/whyrusleeping/go-smux-yamux" mafilter "github.com/whyrusleeping/multiaddr-filter" + ws "github.com/whyrusleeping/ws-transport" context "golang.org/x/net/context" ) @@ -120,6 +121,7 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, transports: []transport.Transport{ transport.NewTCPTransport(), transport.NewUtpTransport(), + new(ws.WebsocketTransport), }, bwc: bwc, fdRateLimit: make(chan struct{}, concurrentFdDials), From afe0aa7f2b4f40da7770ab9f219730450f634e03 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 24 Aug 2016 13:03:30 -0700 Subject: [PATCH 0136/3965] add note to test failure --- p2p/net/swarm/swarm_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index c11377ed5e..db8a464423 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -294,7 +294,7 @@ func TestConnHandler(t *testing.T) { select { case <-gotconn: - t.Fatalf("should have connected to %d swarms", expect) + t.Fatalf("should have connected to %d swarms, got an extra.", expect) default: } } From a9132bea7b15e7735915bf23c4c86d34d5a0501e Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sat, 3 Sep 2016 09:38:00 -0700 Subject: [PATCH 0137/3965] swarm: refactor to make modular construction easier --- p2p/net/swarm/swarm.go | 39 ++++++++++++++++++++---- p2p/net/swarm/swarm_listen.go | 56 +++++++++++++++++++---------------- 2 files changed, 64 insertions(+), 31 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index c342f411c4..05402f5052 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -10,6 +10,14 @@ import ( "sync" "time" + metrics "github.com/libp2p/go-libp2p/p2p/metrics" + mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" + inet "github.com/libp2p/go-libp2p/p2p/net" + conn "github.com/libp2p/go-libp2p/p2p/net/conn" + filter "github.com/libp2p/go-libp2p/p2p/net/filter" + addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" + + ci "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" transport "github.com/ipfs/go-libp2p-transport" @@ -19,12 +27,6 @@ import ( pst "github.com/jbenet/go-stream-muxer" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" - metrics "github.com/libp2p/go-libp2p/p2p/metrics" - mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" - inet "github.com/libp2p/go-libp2p/p2p/net" - conn "github.com/libp2p/go-libp2p/p2p/net/conn" - filter "github.com/libp2p/go-libp2p/p2p/net/filter" - addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" psmss "github.com/whyrusleeping/go-smux-multistream" spdy "github.com/whyrusleeping/go-smux-spdystream" yamux "github.com/whyrusleeping/go-smux-yamux" @@ -143,6 +145,31 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, return s, nil } +func NewBlankSwarm(ctx context.Context, id peer.ID, privkey ci.PrivKey) *Swarm { + s := &Swarm{ + swarm: ps.NewSwarm(PSTransport), + local: id, + peers: pstore.NewPeerstore(), + ctx: ctx, + dialT: DialTimeout, + notifs: make(map[inet.Notifiee]ps.Notifiee), + fdRateLimit: make(chan struct{}, concurrentFdDials), + Filters: filter.NewFilters(), + dialer: conn.NewDialer(id, privkey, nil), + } + + // configure Swarm + s.limiter = newDialLimiter(s.dialAddr) + s.proc = goprocessctx.WithContextAndTeardown(ctx, s.teardown) + s.SetConnHandler(nil) // make sure to setup our own conn handler. + + return s +} + +func (s *Swarm) AddTransport(t transport.Transport) { + s.transports = append(s.transports, t) +} + func (s *Swarm) teardown() error { return s.swarm.Close() } diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 64fdb83bd0..4eb61d5a41 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -13,37 +13,42 @@ import ( context "golang.org/x/net/context" ) +func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { + tpt := s.transportForAddr(a) + if tpt == nil { + return fmt.Errorf("no transport for address: %s", a) + } + + d, err := tpt.Dialer(a, transport.TimeoutOpt(DialTimeout), transport.ReusePorts) + if err != nil { + return err + } + + s.dialer.AddDialer(d) + + list, err := tpt.Listen(a) + if err != nil { + return err + } + + err = s.addListener(list) + if err != nil { + return err + } + + return nil +} + // Open listeners and reuse-dialers for the given addresses func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { errs := make([]error, len(addrs)) var succeeded int for i, a := range addrs { - tpt := s.transportForAddr(a) - if tpt == nil { - errs[i] = fmt.Errorf("no transport for address: %s", a) - continue - } - - d, err := tpt.Dialer(a, transport.TimeoutOpt(DialTimeout), transport.ReusePorts) - if err != nil { - errs[i] = err - continue - } - - s.dialer.AddDialer(d) - - list, err := tpt.Listen(a) - if err != nil { + if err := s.AddListenAddr(a); err != nil { errs[i] = err - continue + } else { + succeeded++ } - - err = s.addListener(list) - if err != nil { - errs[i] = err - continue - } - succeeded++ } for i, e := range errs { @@ -51,6 +56,7 @@ func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { log.Warning("listen on %s failed: %s", addrs[i], errs[i]) } } + if succeeded == 0 && len(addrs) > 0 { return fmt.Errorf("failed to listen on any addresses: %s", errs) } @@ -83,7 +89,7 @@ func (s *Swarm) addListener(tptlist transport.Listener) error { list.SetAddrFilters(s.Filters) - if cw, ok := list.(conn.ListenerConnWrapper); ok { + if cw, ok := list.(conn.ListenerConnWrapper); ok && s.bwc != nil { cw.SetConnWrapper(func(c transport.Conn) transport.Conn { return mconn.WrapConn(s.bwc, c) }) From 423a82fa7aa8568260401cdb3544db5ecad00501 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 4 Sep 2016 08:13:36 -0700 Subject: [PATCH 0138/3965] swarm: pass in stream muxer on construct --- p2p/net/swarm/swarm.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 05402f5052..a3787ced0e 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -145,9 +145,9 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, return s, nil } -func NewBlankSwarm(ctx context.Context, id peer.ID, privkey ci.PrivKey) *Swarm { +func NewBlankSwarm(ctx context.Context, id peer.ID, privkey ci.PrivKey, pstpt pst.Transport) *Swarm { s := &Swarm{ - swarm: ps.NewSwarm(PSTransport), + swarm: ps.NewSwarm(pstpt), local: id, peers: pstore.NewPeerstore(), ctx: ctx, From 6856aa870c8c16b2a83fc34aae063d1fea9d0087 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 4 Sep 2016 10:50:07 -0700 Subject: [PATCH 0139/3965] don't wrap conns with metrics if bwc is nil --- p2p/net/swarm/swarm.go | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index a3787ced0e..05eecec59a 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -109,8 +109,11 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, return nil, err } - wrap := func(c transport.Conn) transport.Conn { - return mconn.WrapConn(bwc, c) + var wrap func(c transport.Conn) transport.Conn + if bwc != nil { + wrap = func(c transport.Conn) transport.Conn { + return mconn.WrapConn(bwc, c) + } } s := &Swarm{ From a7b803d838df91a528fab55c8e698da467377995 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 5 Sep 2016 13:25:24 -0700 Subject: [PATCH 0140/3965] swarm: add optimized method for checking connectedness to peer --- p2p/net/swarm/swarm.go | 9 +++++++++ p2p/net/swarm/swarm_net.go | 3 +-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 05eecec59a..abab8e8a43 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -286,6 +286,15 @@ func (s *Swarm) ConnectionsToPeer(p peer.ID) []*Conn { return wrapConns(ps.ConnsWithGroup(p, s.swarm.Conns())) } +func (s *Swarm) HaveConnsToPeer(p peer.ID) bool { + for _, c := range s.swarm.Conns() { + if c.InGroup(p) { + return true + } + } + return false +} + // Connections returns a slice of all connections. func (s *Swarm) Connections() []*Conn { return wrapConns(s.swarm.Conns()) diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 48defb11ad..abf2f21ce0 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -123,8 +123,7 @@ func (n *Network) InterfaceListenAddresses() ([]ma.Multiaddr, error) { // Connectedness returns a state signaling connection capabilities // For now only returns Connected || NotConnected. Expand into more later. func (n *Network) Connectedness(p peer.ID) inet.Connectedness { - c := n.Swarm().ConnectionsToPeer(p) - if len(c) > 0 { + if n.Swarm().HaveConnsToPeer(p) { return inet.Connected } return inet.NotConnected From ef69b8b035a3193179ff2509e3fd70211c848b43 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 5 Sep 2016 14:48:28 -0700 Subject: [PATCH 0141/3965] swarm: externalize secio handshake errors --- p2p/net/swarm/swarm_dial.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 6774721247..6f82204347 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -450,7 +450,8 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (con remotep := connC.RemotePeer() if remotep != p { connC.Close() - return nil, fmt.Errorf("misdial to %s through %s (got %s)", p, addr, remotep) + _, err := connC.Read(nil) // should return any potential errors (ex: from secio) + return nil, fmt.Errorf("misdial to %s through %s (got %s): %s", p, addr, remotep, err) } // if the connection is to ourselves... From b97c307993d6389a8860825112dc027bcd1cfdbf Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 7 Sep 2016 18:48:22 -0700 Subject: [PATCH 0142/3965] swarm: perform backoff check before taking dialsync lock --- p2p/net/swarm/swarm_dial.go | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 6f82204347..eaf8d1f63a 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -255,16 +255,16 @@ func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) return conn, nil } + // if this peer has been backed off, lets get out of here + if s.backf.Backoff(p) { + log.Event(ctx, "swarmDialBackoff", logdial) + return nil, ErrDialBackoff + } + // check if there's an ongoing dial to this peer if ok, wait := s.dsync.Lock(p); ok { defer s.dsync.Unlock(p) - // if this peer has been backed off, lets get out of here - if s.backf.Backoff(p) { - log.Event(ctx, "swarmDialBackoff", logdial) - return nil, ErrDialBackoff - } - // ok, we have been charged to dial! let's do it. // if it succeeds, dial will add the conn to the swarm itself. defer log.EventBegin(ctx, "swarmDialAttemptStart", logdial).Done() @@ -285,13 +285,6 @@ func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) } else { // we did not dial. we must wait for someone else to dial. - - // check whether we should backoff first... - if s.backf.Backoff(p) { - log.Event(ctx, "swarmDialBackoff", logdial) - return nil, ErrDialBackoff - } - defer log.EventBegin(ctx, "swarmDialWait", logdial).Done() select { case <-wait: // wait for that other dial to finish. From da731da3f5f4fdbd4ca29a1aa4078665528e5b80 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 7 Sep 2016 18:48:55 -0700 Subject: [PATCH 0143/3965] swarm: implement new DialSync manager code --- p2p/net/swarm/dial_sync.go | 92 +++++++++++++++ p2p/net/swarm/dial_sync_test.go | 203 ++++++++++++++++++++++++++++++++ 2 files changed, 295 insertions(+) create mode 100644 p2p/net/swarm/dial_sync.go create mode 100644 p2p/net/swarm/dial_sync_test.go diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go new file mode 100644 index 0000000000..a63c047ac8 --- /dev/null +++ b/p2p/net/swarm/dial_sync.go @@ -0,0 +1,92 @@ +package swarm + +import ( + "context" + "sync" + + peer "github.com/ipfs/go-libp2p-peer" +) + +type DialFunc func(context.Context, peer.ID) (*Conn, error) + +func NewDialSync(dfn DialFunc) *DialSync { + return &DialSync{ + dials: make(map[peer.ID]*activeDial), + dialFunc: dfn, + } +} + +type DialSync struct { + dials map[peer.ID]*activeDial + dialsLk sync.Mutex + dialFunc DialFunc +} + +type activeDial struct { + id peer.ID + refCnt int + refCntLk sync.Mutex + cancel func() + + err error + conn *Conn + waitch chan struct{} + + ds *DialSync +} + +func (dr *activeDial) wait(ctx context.Context) (*Conn, error) { + defer dr.decref() + select { + case <-dr.waitch: + return dr.conn, dr.err + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +func (ad *activeDial) incref() { + ad.refCntLk.Lock() + defer ad.refCntLk.Unlock() + ad.refCnt++ +} + +func (ad *activeDial) decref() { + ad.refCntLk.Lock() + defer ad.refCntLk.Unlock() + ad.refCnt-- + if ad.refCnt <= 0 { + ad.cancel() + ad.ds.dialsLk.Lock() + delete(ad.ds.dials, ad.id) + ad.ds.dialsLk.Unlock() + } +} + +func (ds *DialSync) DialLock(ctx context.Context, p peer.ID) (*Conn, error) { + ds.dialsLk.Lock() + + actd, ok := ds.dials[p] + if !ok { + ctx, cancel := context.WithCancel(context.Background()) + actd = &activeDial{ + id: p, + cancel: cancel, + waitch: make(chan struct{}), + ds: ds, + } + ds.dials[p] = actd + + go func(ctx context.Context, p peer.ID, ad *activeDial) { + ad.conn, ad.err = ds.dialFunc(ctx, p) + close(ad.waitch) + ad.cancel() + ad.waitch = nil // to ensure nobody tries reusing this + }(ctx, p, actd) + } + + actd.incref() + ds.dialsLk.Unlock() + + return actd.wait(ctx) +} diff --git a/p2p/net/swarm/dial_sync_test.go b/p2p/net/swarm/dial_sync_test.go new file mode 100644 index 0000000000..61a69ce386 --- /dev/null +++ b/p2p/net/swarm/dial_sync_test.go @@ -0,0 +1,203 @@ +package swarm + +import ( + "context" + "fmt" + "sync" + "testing" + "time" + + peer "github.com/ipfs/go-libp2p-peer" +) + +func getMockDialFunc() (DialFunc, func(), context.Context, <-chan struct{}) { + dfcalls := make(chan struct{}, 512) // buffer it large enough that we won't care + dialctx, cancel := context.WithCancel(context.Background()) + ch := make(chan struct{}) + f := func(ctx context.Context, p peer.ID) (*Conn, error) { + dfcalls <- struct{}{} + defer cancel() + select { + case <-ch: + return new(Conn), nil + case <-ctx.Done(): + return nil, ctx.Err() + } + } + + o := new(sync.Once) + + return f, func() { o.Do(func() { close(ch) }) }, dialctx, dfcalls +} + +func TestBasicDialSync(t *testing.T) { + df, done, _, callsch := getMockDialFunc() + + dsync := NewDialSync(df) + + p := peer.ID("testpeer") + + ctx := context.Background() + + finished := make(chan struct{}) + go func() { + _, err := dsync.DialLock(ctx, p) + if err != nil { + t.Error(err) + } + finished <- struct{}{} + }() + + go func() { + _, err := dsync.DialLock(ctx, p) + if err != nil { + t.Error(err) + } + finished <- struct{}{} + }() + + // short sleep just to make sure we've moved around in the scheduler + time.Sleep(time.Millisecond * 20) + done() + + <-finished + <-finished + + if len(callsch) > 1 { + t.Fatal("should only have called dial func once!") + } +} + +func TestDialSyncCancel(t *testing.T) { + df, done, _, dcall := getMockDialFunc() + + dsync := NewDialSync(df) + + p := peer.ID("testpeer") + + ctx1, cancel1 := context.WithCancel(context.Background()) + + finished := make(chan struct{}) + go func() { + _, err := dsync.DialLock(ctx1, p) + if err != ctx1.Err() { + t.Error("should have gotten context error") + } + finished <- struct{}{} + }() + + // make sure the above makes it through the wait code first + select { + case <-dcall: + case <-time.After(time.Second): + t.Fatal("timed out waiting for dial to start") + } + + // Add a second dialwait in so two actors are waiting on the same dial + go func() { + _, err := dsync.DialLock(context.Background(), p) + if err != nil { + t.Error(err) + } + finished <- struct{}{} + }() + + time.Sleep(time.Millisecond * 20) + + // cancel the first dialwait, it should not affect the second at all + cancel1() + select { + case <-finished: + case <-time.After(time.Second): + t.Fatal("timed out waiting for wait to exit") + } + + // short sleep just to make sure we've moved around in the scheduler + time.Sleep(time.Millisecond * 20) + done() + + <-finished +} + +func TestDialSyncAllCancel(t *testing.T) { + df, done, dctx, _ := getMockDialFunc() + + dsync := NewDialSync(df) + + p := peer.ID("testpeer") + + ctx1, cancel1 := context.WithCancel(context.Background()) + + finished := make(chan struct{}) + go func() { + _, err := dsync.DialLock(ctx1, p) + if err != ctx1.Err() { + t.Error("should have gotten context error") + } + finished <- struct{}{} + }() + + // Add a second dialwait in so two actors are waiting on the same dial + go func() { + _, err := dsync.DialLock(ctx1, p) + if err != ctx1.Err() { + t.Error("should have gotten context error") + } + finished <- struct{}{} + }() + + cancel1() + for i := 0; i < 2; i++ { + select { + case <-finished: + case <-time.After(time.Second): + t.Fatal("timed out waiting for wait to exit") + } + } + + // the dial should have exited now + select { + case <-dctx.Done(): + case <-time.After(time.Second): + t.Fatal("timed out waiting for dial to return") + } + + // should be able to successfully dial that peer again + done() + _, err := dsync.DialLock(context.Background(), p) + if err != nil { + t.Fatal(err) + } +} + +func TestFailFirst(t *testing.T) { + var count int + f := func(ctx context.Context, p peer.ID) (*Conn, error) { + if count > 0 { + return new(Conn), nil + } + count++ + return nil, fmt.Errorf("gophers ate the modem") + } + + ds := NewDialSync(f) + + p := peer.ID("testing") + + ctx, cancel := context.WithTimeout(context.Background(), time.Second*5) + defer cancel() + + _, err := ds.DialLock(ctx, p) + if err == nil { + t.Fatal("expected gophers to have eaten the modem") + } + + c, err := ds.DialLock(ctx, p) + if err != nil { + t.Fatal(err) + } + + if c == nil { + t.Fatal("should have gotten a 'real' conn back") + } +} From ad0576ff0feae24bf0a3ca9ab18fcfd0d0d61800 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 8 Sep 2016 09:59:28 -0700 Subject: [PATCH 0144/3965] swarm: integrate new dialsync code into swarm --- p2p/net/swarm/dial_test.go | 1 - p2p/net/swarm/limiter.go | 2 +- p2p/net/swarm/limiter_test.go | 2 +- p2p/net/swarm/swarm.go | 5 +- p2p/net/swarm/swarm_dial.go | 135 ++++++---------------------------- 5 files changed, 29 insertions(+), 116 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index cfdb996c28..aeeb6ae42d 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -415,7 +415,6 @@ func TestDialBackoff(t *testing.T) { if !s1.backf.Backoff(s3p) { t.Error("s3 should be on backoff") } - } } diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 4bbfb11862..c34954a527 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -1,11 +1,11 @@ package swarm import ( + "context" "sync" peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" conn "github.com/libp2p/go-libp2p/p2p/net/conn" addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index f5fc18746f..93761a5048 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -1,6 +1,7 @@ package swarm import ( + "context" "fmt" "math/rand" "strconv" @@ -10,7 +11,6 @@ import ( peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" mafmt "github.com/whyrusleeping/mafmt" - context "golang.org/x/net/context" conn "github.com/libp2p/go-libp2p/p2p/net/conn" ) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index abab8e8a43..e36f763fb9 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -3,6 +3,7 @@ package swarm import ( + "context" "fmt" "io/ioutil" "os" @@ -32,7 +33,6 @@ import ( yamux "github.com/whyrusleeping/go-smux-yamux" mafilter "github.com/whyrusleeping/multiaddr-filter" ws "github.com/whyrusleeping/ws-transport" - context "golang.org/x/net/context" ) var log = logging.Logger("swarm2") @@ -76,7 +76,7 @@ type Swarm struct { peers pstore.Peerstore connh ConnHandler - dsync dialsync + dsync *DialSync backf dialbackoff dialT time.Duration // mainly for tests @@ -134,6 +134,7 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, dialer: conn.NewDialer(local, peers.PrivKey(local), wrap), } + s.dsync = NewDialSync(s.doDial) s.limiter = newDialLimiter(s.dialAddr) // configure Swarm diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index eaf8d1f63a..573651778b 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -1,6 +1,7 @@ package swarm import ( + "context" "errors" "fmt" "sync" @@ -11,7 +12,6 @@ import ( ma "github.com/jbenet/go-multiaddr" conn "github.com/libp2p/go-libp2p/p2p/net/conn" addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" - context "golang.org/x/net/context" ) // Diagram of dial sync: @@ -53,78 +53,6 @@ const defaultPerPeerRateLimit = 8 // subcomponent of Dial) var DialTimeout = time.Second * 10 -// dialsync is a small object that helps manage ongoing dials. -// this way, if we receive many simultaneous dial requests, one -// can do its thing, while the rest wait. -// -// this interface is so would-be dialers can just: -// -// for { -// c := findConnectionToPeer(peer) -// if c != nil { -// return c -// } -// -// // ok, no connections. should we dial? -// if ok, wait := dialsync.Lock(peer); !ok { -// <-wait // can optionally wait -// continue -// } -// defer dialsync.Unlock(peer) -// -// c := actuallyDial(peer) -// return c -// } -// -type dialsync struct { - // ongoing is a map of tickets for the current peers being dialed. - // this way, we dont kick off N dials simultaneously. - ongoing map[peer.ID]chan struct{} - lock sync.Mutex -} - -// Lock governs the beginning of a dial attempt. -// If there are no ongoing dials, it returns true, and the client is now -// scheduled to dial. Every other goroutine that calls startDial -- with -//the same dst -- will block until client is done. The client MUST call -// ds.Unlock(p) when it is done, to unblock the other callers. -// The client is not reponsible for achieving a successful dial, only for -// reporting the end of the attempt (calling ds.Unlock(p)). -// -// see the example below `dialsync` -func (ds *dialsync) Lock(dst peer.ID) (bool, chan struct{}) { - ds.lock.Lock() - if ds.ongoing == nil { // init if not ready - ds.ongoing = make(map[peer.ID]chan struct{}) - } - wait, found := ds.ongoing[dst] - if !found { - ds.ongoing[dst] = make(chan struct{}) - } - ds.lock.Unlock() - - if found { - return false, wait - } - - // ok! you're signed up to dial! - return true, nil -} - -// Unlock releases waiters to a dial attempt. see Lock. -// if Unlock(p) is called without calling Lock(p) first, Unlock panics. -func (ds *dialsync) Unlock(dst peer.ID) { - ds.lock.Lock() - wait, found := ds.ongoing[dst] - if !found { - panic("called dialDone with no ongoing dials to peer: " + dst.Pretty()) - } - - delete(ds.ongoing, dst) // remove ongoing dial - close(wait) // release everyone else - ds.lock.Unlock() -} - // dialbackoff is a struct used to avoid over-dialing the same, dead peers. // Whenever we totally time out on a peer (all three attempts), we add them // to dialbackoff. Then, whenevers goroutines would _wait_ (dialsync), they @@ -246,8 +174,7 @@ func (s *Swarm) bestConnectionToPeer(p peer.ID) *Conn { // gatedDialAttempt is an attempt to dial a node. It is gated by the swarm's // dial synchronization systems: dialsync and dialbackoff. func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) { - var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) - defer log.EventBegin(ctx, "swarmDialAttemptSync", logdial).Done() + defer log.EventBegin(ctx, "swarmDialAttemptSync", p).Done() // check if we already have an open connection first conn := s.bestConnectionToPeer(p) @@ -257,48 +184,34 @@ func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) // if this peer has been backed off, lets get out of here if s.backf.Backoff(p) { - log.Event(ctx, "swarmDialBackoff", logdial) + log.Event(ctx, "swarmDialBackoff", p) return nil, ErrDialBackoff } - // check if there's an ongoing dial to this peer - if ok, wait := s.dsync.Lock(p); ok { - defer s.dsync.Unlock(p) - - // ok, we have been charged to dial! let's do it. - // if it succeeds, dial will add the conn to the swarm itself. - defer log.EventBegin(ctx, "swarmDialAttemptStart", logdial).Done() - ctxT, cancel := context.WithTimeout(ctx, s.dialT) - conn, err := s.dial(ctxT, p) - cancel() - log.Debugf("dial end %s", conn) - if err != nil { - log.Event(ctx, "swarmDialBackoffAdd", logdial) - s.backf.AddBackoff(p) // let others know to backoff - - // ok, we failed. try again. (if loop is done, our error is output) - return nil, fmt.Errorf("dial attempt failed: %s", err) - } - log.Event(ctx, "swarmDialBackoffClear", logdial) - s.backf.Clear(p) // okay, no longer need to backoff - return conn, nil + return s.dsync.DialLock(ctx, p) +} - } else { - // we did not dial. we must wait for someone else to dial. - defer log.EventBegin(ctx, "swarmDialWait", logdial).Done() - select { - case <-wait: // wait for that other dial to finish. +// doDial is an ugly shim method to retain all the logging and backoff logic +// of the old dialsync code +func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { + var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) + // ok, we have been charged to dial! let's do it. + // if it succeeds, dial will add the conn to the swarm itself. + defer log.EventBegin(ctx, "swarmDialAttemptStart", logdial).Done() + ctxT, cancel := context.WithTimeout(ctx, s.dialT) + conn, err := s.dial(ctxT, p) + cancel() + log.Debugf("dial end %s", conn) + if err != nil { + log.Event(ctx, "swarmDialBackoffAdd", logdial) + s.backf.AddBackoff(p) // let others know to backoff - // see if it worked, OR we got an incoming dial in the meantime... - conn := s.bestConnectionToPeer(p) - if conn != nil { - return conn, nil - } - return nil, ErrDialFailed - case <-ctx.Done(): // or we may have to bail... - return nil, ctx.Err() - } + // ok, we failed. try again. (if loop is done, our error is output) + return nil, fmt.Errorf("dial attempt failed: %s", err) } + log.Event(ctx, "swarmDialBackoffClear", logdial) + s.backf.Clear(p) // okay, no longer need to backoff + return conn, nil } // dial is the actual swarm's dial logic, gated by Dial. From 3339abd03e403c076c1f1b201e4afc7f61393a46 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Fri, 26 Aug 2016 18:15:27 +0200 Subject: [PATCH 0145/3965] Retry NAT punching without duration on mapping failure Some hardware doesn't support UPnP with durations. --- p2p/net/nat/nat.go | 43 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 8 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 7dac06df1d..6324715bf5 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -155,12 +155,13 @@ type Mapping interface { type mapping struct { sync.Mutex // guards all fields - nat *NAT - proto string - intport int - extport int - intaddr ma.Multiaddr - proc goprocess.Process + nat *NAT + proto string + intport int + extport int + permanent bool + intaddr ma.Multiaddr + proc goprocess.Process cached ma.Multiaddr cacheTime time.Time @@ -197,6 +198,18 @@ func (m *mapping) setExternalPort(p int) { m.extport = p } +func (m *mapping) setPermanent(p bool) { + m.Lock() + defer m.Unlock() + m.permanent = p +} + +func (m *mapping) isPermanent() bool { + m.Lock() + defer m.Unlock() + return m.permanent +} + func (m *mapping) InternalAddr() ma.Multiaddr { m.Lock() defer m.Unlock() @@ -333,13 +346,26 @@ func (nat *NAT) NewMapping(maddr ma.Multiaddr) (Mapping, error) { func (nat *NAT) establishMapping(m *mapping) { oldport := m.ExternalPort() + if oldport != 0 && m.isPermanent() { + // mapping was already established and it is permanent + return + } + log.Debugf("Attempting port map: %s/%d", m.Protocol(), m.InternalPort()) - newport, err := nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), "http", MappingDuration) + permanent := false + + newport, err := nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), "libp2p", MappingDuration) + + if err != nil { + // Some hardware does not support mappings with timeout, so try that + newport, err = nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), "libp2p", 0) + permanent = (err == nil) + } failure := func() { m.setExternalPort(0) // clear mapping // TODO: log.Event - log.Debugf("failed to establish port mapping: %s", err) + log.Warningf("failed to establish port mapping: %s", err) nat.Notifier.notifyAll(func(n Notifiee) { n.MappingFailed(nat, m, oldport, err) }) @@ -353,6 +379,7 @@ func (nat *NAT) establishMapping(m *mapping) { return } + m.setPermanent(permanent) m.setExternalPort(newport) ext, err := m.ExternalAddr() if err != nil { From 59a1cee3e674f1a57aceedc3bf9f0dbb72c96069 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Fri, 26 Aug 2016 18:45:28 +0200 Subject: [PATCH 0146/3965] nat: do not shortcircuit permanent mappings If we use same NAT agent and call for the same permanent mapping again we get the same mapping, no harm done. If router dies, we will remap again. Just pros, no cons. --- p2p/net/nat/nat.go | 30 ++++++++---------------------- 1 file changed, 8 insertions(+), 22 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 6324715bf5..a7fc75b0d3 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -163,6 +163,8 @@ type mapping struct { intaddr ma.Multiaddr proc goprocess.Process + comment string + cached ma.Multiaddr cacheTime time.Time cacheLk sync.Mutex @@ -198,18 +200,6 @@ func (m *mapping) setExternalPort(p int) { m.extport = p } -func (m *mapping) setPermanent(p bool) { - m.Lock() - defer m.Unlock() - m.permanent = p -} - -func (m *mapping) isPermanent() bool { - m.Lock() - defer m.Unlock() - return m.permanent -} - func (m *mapping) InternalAddr() ma.Multiaddr { m.Lock() defer m.Unlock() @@ -346,20 +336,17 @@ func (nat *NAT) NewMapping(maddr ma.Multiaddr) (Mapping, error) { func (nat *NAT) establishMapping(m *mapping) { oldport := m.ExternalPort() - if oldport != 0 && m.isPermanent() { - // mapping was already established and it is permanent - return - } log.Debugf("Attempting port map: %s/%d", m.Protocol(), m.InternalPort()) - permanent := false - - newport, err := nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), "libp2p", MappingDuration) + comment := "libp2p" + if m.comment != "" { + comment = "libp2p-" + m.comment + } + newport, err := nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), comment, MappingDuration) if err != nil { // Some hardware does not support mappings with timeout, so try that - newport, err = nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), "libp2p", 0) - permanent = (err == nil) + newport, err = nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), comment, 0) } failure := func() { @@ -379,7 +366,6 @@ func (nat *NAT) establishMapping(m *mapping) { return } - m.setPermanent(permanent) m.setExternalPort(newport) ext, err := m.ExternalAddr() if err != nil { From 72d32fc62f3109195b858c5b99689843f3fa985f Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 12 Sep 2016 15:59:01 +0200 Subject: [PATCH 0147/3965] nat: split into files --- p2p/net/nat/mapping.go | 141 ++++++++++++++++++++++++++++++++ p2p/net/nat/nat.go | 173 ---------------------------------------- p2p/net/nat/notifier.go | 47 +++++++++++ 3 files changed, 188 insertions(+), 173 deletions(-) create mode 100644 p2p/net/nat/mapping.go create mode 100644 p2p/net/nat/notifier.go diff --git a/p2p/net/nat/mapping.go b/p2p/net/nat/mapping.go new file mode 100644 index 0000000000..2f45726308 --- /dev/null +++ b/p2p/net/nat/mapping.go @@ -0,0 +1,141 @@ +package nat + +import ( + "fmt" + "sync" + "time" + + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + "github.com/jbenet/goprocess" +) + +// Mapping represents a port mapping in a NAT. +type Mapping interface { + // NAT returns the NAT object this Mapping belongs to. + NAT() *NAT + + // Protocol returns the protocol of this port mapping. This is either + // "tcp" or "udp" as no other protocols are likely to be NAT-supported. + Protocol() string + + // InternalPort returns the internal device port. Mapping will continue to + // try to map InternalPort() to an external facing port. + InternalPort() int + + // ExternalPort returns the external facing port. If the mapping is not + // established, port will be 0 + ExternalPort() int + + // InternalAddr returns the internal address. + InternalAddr() ma.Multiaddr + + // ExternalAddr returns the external facing address. If the mapping is not + // established, addr will be nil, and and ErrNoMapping will be returned. + ExternalAddr() (addr ma.Multiaddr, err error) + + // Close closes the port mapping + Close() error +} + +// keeps republishing +type mapping struct { + sync.Mutex // guards all fields + + nat *NAT + proto string + intport int + extport int + permanent bool + intaddr ma.Multiaddr + proc goprocess.Process + + comment string + + cached ma.Multiaddr + cacheTime time.Time + cacheLk sync.Mutex +} + +func (m *mapping) NAT() *NAT { + m.Lock() + defer m.Unlock() + return m.nat +} + +func (m *mapping) Protocol() string { + m.Lock() + defer m.Unlock() + return m.proto +} + +func (m *mapping) InternalPort() int { + m.Lock() + defer m.Unlock() + return m.intport +} + +func (m *mapping) ExternalPort() int { + m.Lock() + defer m.Unlock() + return m.extport +} + +func (m *mapping) setExternalPort(p int) { + m.Lock() + defer m.Unlock() + m.extport = p +} + +func (m *mapping) InternalAddr() ma.Multiaddr { + m.Lock() + defer m.Unlock() + return m.intaddr +} + +func (m *mapping) ExternalAddr() (ma.Multiaddr, error) { + m.cacheLk.Lock() + ctime := m.cacheTime + cval := m.cached + m.cacheLk.Unlock() + if time.Since(ctime) < CacheTime { + return cval, nil + } + + if m.ExternalPort() == 0 { // dont even try right now. + return nil, ErrNoMapping + } + + ip, err := m.nat.nat.GetExternalAddress() + if err != nil { + return nil, err + } + + ipmaddr, err := manet.FromIP(ip) + if err != nil { + return nil, fmt.Errorf("error parsing ip") + } + + // call m.ExternalPort again, as mapping may have changed under our feet. (tocttou) + extport := m.ExternalPort() + if extport == 0 { + return nil, ErrNoMapping + } + + tcp, err := ma.NewMultiaddr(fmt.Sprintf("/%s/%d", m.Protocol(), extport)) + if err != nil { + return nil, err + } + + maddr2 := ipmaddr.Encapsulate(tcp) + + m.cacheLk.Lock() + m.cached = maddr2 + m.cacheTime = time.Now() + m.cacheLk.Unlock() + return maddr2, nil +} + +func (m *mapping) Close() error { + return m.proc.Close() +} diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index a7fc75b0d3..210512cd1a 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -14,7 +14,6 @@ import ( manet "github.com/jbenet/go-multiaddr-net" goprocess "github.com/jbenet/goprocess" periodic "github.com/jbenet/goprocess/periodic" - notifier "github.com/whyrusleeping/go-notifier" ) var ( @@ -81,178 +80,6 @@ func (nat *NAT) Process() goprocess.Process { return nat.proc } -// Notifier is an object that assists NAT in notifying listeners. -// It is implemented using thirdparty/notifier -type Notifier struct { - n notifier.Notifier -} - -func (n *Notifier) notifyAll(notify func(n Notifiee)) { - n.n.NotifyAll(func(n notifier.Notifiee) { - notify(n.(Notifiee)) - }) -} - -// Notify signs up notifiee to listen to NAT events. -func (n *Notifier) Notify(notifiee Notifiee) { - n.n.Notify(n) -} - -// StopNotify stops signaling events to notifiee. -func (n *Notifier) StopNotify(notifiee Notifiee) { - n.n.StopNotify(notifiee) -} - -// Notifiee is an interface objects must implement to listen to NAT events. -type Notifiee interface { - - // Called every time a successful mapping happens - // Warning: the port mapping may have changed. If that is the - // case, both MappingSuccess and MappingChanged are called. - MappingSuccess(nat *NAT, m Mapping) - - // Called when mapping a port succeeds, but the mapping is - // with a different port than an earlier success. - MappingChanged(nat *NAT, m Mapping, oldport int, newport int) - - // Called when a port mapping fails. NAT will continue attempting after - // the next period. To stop trying, use: mapping.Close(). After this failure, - // mapping.ExternalPort() will be zero, and nat.ExternalAddrs() will not - // return the address for this mapping. With luck, the next attempt will - // succeed, without the client needing to do anything. - MappingFailed(nat *NAT, m Mapping, oldport int, err error) -} - -// Mapping represents a port mapping in a NAT. -type Mapping interface { - // NAT returns the NAT object this Mapping belongs to. - NAT() *NAT - - // Protocol returns the protocol of this port mapping. This is either - // "tcp" or "udp" as no other protocols are likely to be NAT-supported. - Protocol() string - - // InternalPort returns the internal device port. Mapping will continue to - // try to map InternalPort() to an external facing port. - InternalPort() int - - // ExternalPort returns the external facing port. If the mapping is not - // established, port will be 0 - ExternalPort() int - - // InternalAddr returns the internal address. - InternalAddr() ma.Multiaddr - - // ExternalAddr returns the external facing address. If the mapping is not - // established, addr will be nil, and and ErrNoMapping will be returned. - ExternalAddr() (addr ma.Multiaddr, err error) - - // Close closes the port mapping - Close() error -} - -// keeps republishing -type mapping struct { - sync.Mutex // guards all fields - - nat *NAT - proto string - intport int - extport int - permanent bool - intaddr ma.Multiaddr - proc goprocess.Process - - comment string - - cached ma.Multiaddr - cacheTime time.Time - cacheLk sync.Mutex -} - -func (m *mapping) NAT() *NAT { - m.Lock() - defer m.Unlock() - return m.nat -} - -func (m *mapping) Protocol() string { - m.Lock() - defer m.Unlock() - return m.proto -} - -func (m *mapping) InternalPort() int { - m.Lock() - defer m.Unlock() - return m.intport -} - -func (m *mapping) ExternalPort() int { - m.Lock() - defer m.Unlock() - return m.extport -} - -func (m *mapping) setExternalPort(p int) { - m.Lock() - defer m.Unlock() - m.extport = p -} - -func (m *mapping) InternalAddr() ma.Multiaddr { - m.Lock() - defer m.Unlock() - return m.intaddr -} - -func (m *mapping) ExternalAddr() (ma.Multiaddr, error) { - m.cacheLk.Lock() - ctime := m.cacheTime - cval := m.cached - m.cacheLk.Unlock() - if time.Since(ctime) < CacheTime { - return cval, nil - } - - if m.ExternalPort() == 0 { // dont even try right now. - return nil, ErrNoMapping - } - - ip, err := m.nat.nat.GetExternalAddress() - if err != nil { - return nil, err - } - - ipmaddr, err := manet.FromIP(ip) - if err != nil { - return nil, fmt.Errorf("error parsing ip") - } - - // call m.ExternalPort again, as mapping may have changed under our feet. (tocttou) - extport := m.ExternalPort() - if extport == 0 { - return nil, ErrNoMapping - } - - tcp, err := ma.NewMultiaddr(fmt.Sprintf("/%s/%d", m.Protocol(), extport)) - if err != nil { - return nil, err - } - - maddr2 := ipmaddr.Encapsulate(tcp) - - m.cacheLk.Lock() - m.cached = maddr2 - m.cacheTime = time.Now() - m.cacheLk.Unlock() - return maddr2, nil -} - -func (m *mapping) Close() error { - return m.proc.Close() -} - // Mappings returns a slice of all NAT mappings func (nat *NAT) Mappings() []Mapping { nat.mappingmu.Lock() diff --git a/p2p/net/nat/notifier.go b/p2p/net/nat/notifier.go new file mode 100644 index 0000000000..462a78b953 --- /dev/null +++ b/p2p/net/nat/notifier.go @@ -0,0 +1,47 @@ +package nat + +import ( + notifier "github.com/whyrusleeping/go-notifier" +) + +// Notifier is an object that assists NAT in notifying listeners. +// It is implemented using thirdparty/notifier +type Notifier struct { + n notifier.Notifier +} + +func (n *Notifier) notifyAll(notify func(n Notifiee)) { + n.n.NotifyAll(func(n notifier.Notifiee) { + notify(n.(Notifiee)) + }) +} + +// Notify signs up notifiee to listen to NAT events. +func (n *Notifier) Notify(notifiee Notifiee) { + n.n.Notify(n) +} + +// StopNotify stops signaling events to notifiee. +func (n *Notifier) StopNotify(notifiee Notifiee) { + n.n.StopNotify(notifiee) +} + +// Notifiee is an interface objects must implement to listen to NAT events. +type Notifiee interface { + + // Called every time a successful mapping happens + // Warning: the port mapping may have changed. If that is the + // case, both MappingSuccess and MappingChanged are called. + MappingSuccess(nat *NAT, m Mapping) + + // Called when mapping a port succeeds, but the mapping is + // with a different port than an earlier success. + MappingChanged(nat *NAT, m Mapping, oldport int, newport int) + + // Called when a port mapping fails. NAT will continue attempting after + // the next period. To stop trying, use: mapping.Close(). After this failure, + // mapping.ExternalPort() will be zero, and nat.ExternalAddrs() will not + // return the address for this mapping. With luck, the next attempt will + // succeed, without the client needing to do anything. + MappingFailed(nat *NAT, m Mapping, oldport int, err error) +} From d44566d5a54e275bcee0d24c0dfce4f1776271b7 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 12 Sep 2016 16:01:05 +0200 Subject: [PATCH 0148/3965] nat: add locks around nat --- p2p/net/nat/mapping.go | 2 ++ p2p/net/nat/nat.go | 7 +++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/p2p/net/nat/mapping.go b/p2p/net/nat/mapping.go index 2f45726308..b2aeb9c4d7 100644 --- a/p2p/net/nat/mapping.go +++ b/p2p/net/nat/mapping.go @@ -106,7 +106,9 @@ func (m *mapping) ExternalAddr() (ma.Multiaddr, error) { return nil, ErrNoMapping } + m.nat.natmu.Lock() ip, err := m.nat.nat.GetExternalAddress() + m.nat.natmu.Unlock() if err != nil { return nil, err } diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 210512cd1a..879b69c06d 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -52,8 +52,9 @@ func DiscoverNAT() *NAT { // service that will periodically renew port mappings, // and keep an up-to-date list of all the external addresses. type NAT struct { - nat nat.NAT - proc goprocess.Process // manages nat mappings lifecycle + natmu sync.Mutex + nat nat.NAT + proc goprocess.Process // manages nat mappings lifecycle mappingmu sync.RWMutex // guards mappings mappings map[*mapping]struct{} @@ -170,11 +171,13 @@ func (nat *NAT) establishMapping(m *mapping) { comment = "libp2p-" + m.comment } + nat.natmu.Lock() newport, err := nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), comment, MappingDuration) if err != nil { // Some hardware does not support mappings with timeout, so try that newport, err = nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), comment, 0) } + nat.natmu.Lock() failure := func() { m.setExternalPort(0) // clear mapping From 598f265e6f53fe6336c59e75c3050ae5838f527a Mon Sep 17 00:00:00 2001 From: Friedel Ziegelmayer Date: Thu, 15 Sep 2016 10:06:05 +0200 Subject: [PATCH 0149/3965] fix: use binary frames instead of text frames This fixes websocket interop with js-ipfs --- p2p/transport/websocket/websocket.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index f048c006cb..fed6bc3655 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -184,6 +184,7 @@ func (t *WebsocketTransport) wrapListener(l manet.Listener) *listener { func (l *listener) handleWsConn(s *ws.Conn) { ctx, cancel := context.WithCancel(context.Background()) + s.PayloadType = ws.BinaryFrame l.incoming <- &conn{ Conn: s, From b80c12878c795bf51a9f2b57aa9fba02d198975b Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 15 Sep 2016 22:24:15 -0700 Subject: [PATCH 0150/3965] gx publish v1.2.0 --- p2p/transport/websocket/websocket.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index fed6bc3655..331f1d2130 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -99,6 +99,10 @@ func parseMultiaddr(a ma.Multiaddr) (string, error) { } func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { + return d.DialContext(context.Background(), raddr) +} + +func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { wsurl, err := parseMultiaddr(raddr) if err != nil { return nil, err From 7f414b2b1426f7ec48de59f3bc4020a6c67776b9 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 18 Aug 2016 15:45:18 -0700 Subject: [PATCH 0151/3965] swarm: add deadline for connection setup --- p2p/net/swarm/swarm_dial.go | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 573651778b..6d1abd3342 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -372,10 +372,21 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (con return connC, nil } +var ConnSetupTimeout = time.Minute * 5 + // dialConnSetup is the setup logic for a connection from the dial side. it // needs to add the Conn to the StreamSwarm, then run newConnSetup func dialConnSetup(ctx context.Context, s *Swarm, connC conn.Conn) (*Conn, error) { + deadline, ok := ctx.Deadline() + if !ok { + deadline = time.Now().Add(ConnSetupTimeout) + } + + if err := connC.SetDeadline(deadline); err != nil { + return nil, err + } + psC, err := s.swarm.AddConn(connC) if err != nil { // connC is closed by caller if we fail. @@ -389,5 +400,10 @@ func dialConnSetup(ctx context.Context, s *Swarm, connC conn.Conn) (*Conn, error return nil, err } + if err := connC.SetDeadline(time.Time{}); err != nil { + log.Error("failed to reset connection deadline after setup: ", err) + return nil, err + } + return swarmC, err } From 7c54aa9763b542886407ddc2c3913b8d36e45407 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 25 Sep 2016 01:08:17 -0700 Subject: [PATCH 0152/3965] extract tcp transport from go-libp2p-transport --- p2p/transport/tcp/reuseport.go | 65 +++++++ p2p/transport/tcp/reuseport_test.go | 48 ++++++ p2p/transport/tcp/tcp.go | 252 ++++++++++++++++++++++++++++ p2p/transport/tcp/tcp_test.go | 30 ++++ 4 files changed, 395 insertions(+) create mode 100644 p2p/transport/tcp/reuseport.go create mode 100644 p2p/transport/tcp/reuseport_test.go create mode 100644 p2p/transport/tcp/tcp.go create mode 100644 p2p/transport/tcp/tcp_test.go diff --git a/p2p/transport/tcp/reuseport.go b/p2p/transport/tcp/reuseport.go new file mode 100644 index 0000000000..6ead3a74a4 --- /dev/null +++ b/p2p/transport/tcp/reuseport.go @@ -0,0 +1,65 @@ +package tcp + +import ( + "net" + "os" + "strings" + "syscall" + + reuseport "github.com/jbenet/go-reuseport" +) + +// envReuseport is the env variable name used to turn off reuse port. +// It default to true. +const envReuseport = "IPFS_REUSEPORT" + +// envReuseportVal stores the value of envReuseport. defaults to true. +var envReuseportVal = true + +func init() { + v := strings.ToLower(os.Getenv(envReuseport)) + if v == "false" || v == "f" || v == "0" { + envReuseportVal = false + log.Infof("REUSEPORT disabled (IPFS_REUSEPORT=%s)", v) + } +} + +// reuseportIsAvailable returns whether reuseport is available to be used. This +// is here because we want to be able to turn reuseport on and off selectively. +// For now we use an ENV variable, as this handles our pressing need: +// +// IPFS_REUSEPORT=false ipfs daemon +// +// If this becomes a sought after feature, we could add this to the config. +// In the end, reuseport is a stop-gap. +func ReuseportIsAvailable() bool { + return envReuseportVal && reuseport.Available() +} + +// ReuseErrShouldRetry diagnoses whether to retry after a reuse error. +// if we failed to bind, we should retry. if bind worked and this is a +// real dial error (remote end didnt answer) then we should not retry. +func ReuseErrShouldRetry(err error) bool { + if err == nil { + return false // hey, it worked! no need to retry. + } + + // if it's a network timeout error, it's a legitimate failure. + if nerr, ok := err.(net.Error); ok && nerr.Timeout() { + return false + } + + errno, ok := err.(syscall.Errno) + if !ok { // not an errno? who knows what this is. retry. + return true + } + + switch errno { + case syscall.EADDRINUSE, syscall.EADDRNOTAVAIL: + return true // failure to bind. retry. + case syscall.ECONNREFUSED: + return false // real dial error + default: + return true // optimistically default to retry. + } +} diff --git a/p2p/transport/tcp/reuseport_test.go b/p2p/transport/tcp/reuseport_test.go new file mode 100644 index 0000000000..ea746cd5f8 --- /dev/null +++ b/p2p/transport/tcp/reuseport_test.go @@ -0,0 +1,48 @@ +package tcp + +import ( + "net" + "syscall" + "testing" +) + +type netTimeoutErr struct { + timeout bool +} + +func (e netTimeoutErr) Error() string { + return "" +} + +func (e netTimeoutErr) Timeout() bool { + return e.timeout +} + +func (e netTimeoutErr) Temporary() bool { + panic("not checked") +} + +func TestReuseError(t *testing.T) { + var nte1 net.Error = &netTimeoutErr{true} + var nte2 net.Error = &netTimeoutErr{false} + + cases := map[error]bool{ + nil: false, + syscall.EADDRINUSE: true, + syscall.EADDRNOTAVAIL: true, + syscall.ECONNREFUSED: false, + + nte1: false, + nte2: true, // this ones a little weird... we should check neterror.Temporary() too + + // test 'default' to true + syscall.EBUSY: true, + } + + for k, v := range cases { + if ReuseErrShouldRetry(k) != v { + t.Fatalf("expected %b for %#v", v, k) + } + } + +} diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go new file mode 100644 index 0000000000..2285d165dc --- /dev/null +++ b/p2p/transport/tcp/tcp.go @@ -0,0 +1,252 @@ +package tcp + +import ( + "context" + "fmt" + "net" + "sync" + "time" + + lgbl "github.com/ipfs/go-libp2p-loggables" + logging "github.com/ipfs/go-log" + ma "github.com/jbenet/go-multiaddr" + manet "github.com/jbenet/go-multiaddr-net" + reuseport "github.com/jbenet/go-reuseport" + tpt "github.com/libp2p/go-libp2p-transport" + mafmt "github.com/whyrusleeping/mafmt" +) + +var log = logging.Logger("tcp-tpt") + +type TcpTransport struct { + dlock sync.Mutex + dialers map[string]tpt.Dialer + + llock sync.Mutex + listeners map[string]tpt.Listener +} + +// NewTCPTransport creates a tcp transport object that tracks dialers and listeners +// created. It represents an entire tcp stack (though it might not necessarily be) +func NewTCPTransport() *TcpTransport { + return &TcpTransport{ + dialers: make(map[string]tpt.Dialer), + listeners: make(map[string]tpt.Listener), + } +} + +func (t *TcpTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { + t.dlock.Lock() + defer t.dlock.Unlock() + s := laddr.String() + d, found := t.dialers[s] + if found { + return d, nil + } + var base manet.Dialer + + var doReuse bool + for _, o := range opts { + switch o := o.(type) { + case tpt.TimeoutOpt: + base.Timeout = time.Duration(o) + case tpt.ReuseportOpt: + doReuse = bool(o) + default: + return nil, fmt.Errorf("unrecognized option: %#v", o) + } + } + + tcpd, err := t.newTcpDialer(base, laddr, doReuse) + if err != nil { + return nil, err + } + + t.dialers[s] = tcpd + return tcpd, nil +} + +func (t *TcpTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { + if !t.Matches(laddr) { + return nil, fmt.Errorf("tcp transport cannot listen on %q", laddr) + } + + t.llock.Lock() + defer t.llock.Unlock() + s := laddr.String() + l, found := t.listeners[s] + if found { + return l, nil + } + + list, err := manetListen(laddr) + if err != nil { + return nil, err + } + + tlist := &tcpListener{ + list: list, + transport: t, + } + + t.listeners[s] = tlist + return tlist, nil +} + +func manetListen(addr ma.Multiaddr) (manet.Listener, error) { + network, naddr, err := manet.DialArgs(addr) + if err != nil { + return nil, err + } + + if ReuseportIsAvailable() { + nl, err := reuseport.Listen(network, naddr) + if err == nil { + // hey, it worked! + return manet.WrapNetListener(nl) + } + // reuseport is available, but we failed to listen. log debug, and retry normally. + log.Debugf("reuseport available, but failed to listen: %s %s, %s", network, naddr, err) + } + + // either reuseport not available, or it failed. try normally. + return manet.Listen(addr) +} + +func (t *TcpTransport) Matches(a ma.Multiaddr) bool { + return mafmt.TCP.Matches(a) +} + +type tcpDialer struct { + laddr ma.Multiaddr + + doReuse bool + + rd reuseport.Dialer + madialer manet.Dialer + + transport tpt.Transport +} + +func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReuse bool) (*tcpDialer, error) { + // get the local net.Addr manually + la, err := manet.ToNetAddr(laddr) + if err != nil { + return nil, err // something wrong with laddr. + } + + if doReuse && ReuseportIsAvailable() { + rd := reuseport.Dialer{ + D: net.Dialer{ + LocalAddr: la, + Timeout: base.Timeout, + }, + } + + return &tcpDialer{ + doReuse: true, + laddr: laddr, + rd: rd, + madialer: base, + transport: t, + }, nil + } + + return &tcpDialer{ + doReuse: false, + laddr: laddr, + madialer: base, + transport: t, + }, nil +} + +func (d *tcpDialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { + return d.DialContext(context.Background(), raddr) +} + +func (d *tcpDialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { + var c manet.Conn + var err error + if d.doReuse { + c, err = d.reuseDial(ctx, raddr) + } else { + c, err = d.madialer.DialContext(ctx, raddr) + } + + if err != nil { + return nil, err + } + + return &tpt.ConnWrap{ + Conn: c, + Tpt: d.transport, + }, nil +} + +func (d *tcpDialer) reuseDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { + logdial := lgbl.Dial("conn", "", "", d.laddr, raddr) + rpev := log.EventBegin(ctx, "tptDialReusePort", logdial) + + network, netraddr, err := manet.DialArgs(raddr) + if err != nil { + return nil, err + } + + _ = ctx // TODO: implement DialContext in reuseport + con, err := d.rd.Dial(network, netraddr) + if err == nil { + logdial["reuseport"] = "success" + rpev.Done() + return manet.WrapNetConn(con) + } + + if !ReuseErrShouldRetry(err) { + logdial["reuseport"] = "failure" + logdial["error"] = err + rpev.Done() + return nil, err + } + + logdial["reuseport"] = "retry" + logdial["error"] = err + rpev.Done() + + return d.madialer.DialContext(ctx, raddr) +} + +func (d *tcpDialer) Matches(a ma.Multiaddr) bool { + return mafmt.TCP.Matches(a) +} + +type tcpListener struct { + list manet.Listener + transport tpt.Transport +} + +func (d *tcpListener) Accept() (tpt.Conn, error) { + c, err := d.list.Accept() + if err != nil { + return nil, err + } + + return &tpt.ConnWrap{ + Conn: c, + Tpt: d.transport, + }, nil +} + +func (d *tcpListener) Addr() net.Addr { + return d.list.Addr() +} + +func (t *tcpListener) Multiaddr() ma.Multiaddr { + return t.list.Multiaddr() +} + +func (t *tcpListener) NetListener() net.Listener { + return t.list.NetListener() +} + +func (d *tcpListener) Close() error { + return d.list.Close() +} diff --git a/p2p/transport/tcp/tcp_test.go b/p2p/transport/tcp/tcp_test.go new file mode 100644 index 0000000000..5e1e29087e --- /dev/null +++ b/p2p/transport/tcp/tcp_test.go @@ -0,0 +1,30 @@ +package tcp + +import ( + "testing" + + ma "github.com/jbenet/go-multiaddr" + + utils "github.com/libp2p/go-libp2p-transport/test" +) + +func TestTcpTransport(t *testing.T) { + ta := NewTCPTransport() + tb := NewTCPTransport() + + zero := "/ip4/127.0.0.1/tcp/0" + utils.SubtestTransport(t, ta, tb, zero) +} + +func TestTcpTransportCantListenUtp(t *testing.T) { + utpa, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/utp") + if err != nil { + t.Fatal(err) + } + + tpt := NewTCPTransport() + _, err = tpt.Listen(utpa) + if err == nil { + t.Fatal("shouldnt be able to listen on utp addr with tcp transport") + } +} From f5f0af8297d64a4daab4b21d74a0620eface355b Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 25 Sep 2016 01:29:29 -0700 Subject: [PATCH 0153/3965] update libp2p-transport --- p2p/transport/websocket/websocket.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 331f1d2130..ee8678a2cc 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -7,8 +7,8 @@ import ( "net/http" "net/url" - tpt "github.com/ipfs/go-libp2p-transport" manet "github.com/jbenet/go-multiaddr-net" + tpt "github.com/libp2p/go-libp2p-transport" mafmt "github.com/whyrusleeping/mafmt" ws "golang.org/x/net/websocket" From af5031ca5e01cd7b20c3d898a3576ea3dc62c8bc Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 25 Sep 2016 04:51:27 -0700 Subject: [PATCH 0154/3965] split tcp transports out of the base package --- p2p/net/swarm/swarm.go | 6 +++--- p2p/net/swarm/swarm_listen.go | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index e36f763fb9..4eebf626be 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -21,13 +21,14 @@ import ( ci "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" - transport "github.com/ipfs/go-libp2p-transport" logging "github.com/ipfs/go-log" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" pst "github.com/jbenet/go-stream-muxer" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" + transport "github.com/libp2p/go-libp2p-transport" + tcpt "github.com/libp2p/go-tcp-transport" psmss "github.com/whyrusleeping/go-smux-multistream" spdy "github.com/whyrusleeping/go-smux-spdystream" yamux "github.com/whyrusleeping/go-smux-yamux" @@ -124,8 +125,7 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, dialT: DialTimeout, notifs: make(map[inet.Notifiee]ps.Notifiee), transports: []transport.Transport{ - transport.NewTCPTransport(), - transport.NewUtpTransport(), + tcpt.NewTCPTransport(), new(ws.WebsocketTransport), }, bwc: bwc, diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 4eb61d5a41..92ccd833af 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -4,9 +4,9 @@ import ( "fmt" lgbl "github.com/ipfs/go-libp2p-loggables" - transport "github.com/ipfs/go-libp2p-transport" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" + transport "github.com/libp2p/go-libp2p-transport" mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" inet "github.com/libp2p/go-libp2p/p2p/net" conn "github.com/libp2p/go-libp2p/p2p/net/conn" From c803102bdf8849be70985abb5f6025befabb8852 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Thu, 29 Sep 2016 15:32:27 +0200 Subject: [PATCH 0155/3965] fix locking issue introduced in 90eeff42fcd --- p2p/net/nat/nat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 879b69c06d..3a4ae60b4d 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -177,7 +177,7 @@ func (nat *NAT) establishMapping(m *mapping) { // Some hardware does not support mappings with timeout, so try that newport, err = nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), comment, 0) } - nat.natmu.Lock() + nat.natmu.Unlock() failure := func() { m.setExternalPort(0) // clear mapping From 8df46c82c359c825c6daecbf4a3251580d656ee8 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 3 Oct 2016 19:47:09 +0200 Subject: [PATCH 0156/3965] fix: use DialContext of go-reuseport It got skipped when go-reuseport got migrated to context --- p2p/transport/tcp/tcp.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 2285d165dc..4def4dfb00 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -193,7 +193,7 @@ func (d *tcpDialer) reuseDial(ctx context.Context, raddr ma.Multiaddr) (manet.Co } _ = ctx // TODO: implement DialContext in reuseport - con, err := d.rd.Dial(network, netraddr) + con, err := d.rd.DialContext(ctx, network, netraddr) if err == nil { logdial["reuseport"] = "success" rpev.Done() From 17194ccda6021fa8cd0621335324f14a79d5385f Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 3 Oct 2016 11:00:00 -0700 Subject: [PATCH 0157/3965] switch to using stdlib context package --- p2p/net/swarm/addr/addr.go | 2 +- p2p/net/swarm/dial_test.go | 2 +- p2p/net/swarm/peers_test.go | 2 +- p2p/net/swarm/simul_test.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 2 +- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_listen.go | 2 +- p2p/net/swarm/swarm_net.go | 2 +- p2p/net/swarm/swarm_net_test.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 2 +- p2p/net/swarm/swarm_test.go | 2 +- 11 files changed, 11 insertions(+), 11 deletions(-) diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go index 08ca90ac46..4a14c43778 100644 --- a/p2p/net/swarm/addr/addr.go +++ b/p2p/net/swarm/addr/addr.go @@ -3,10 +3,10 @@ package addrutil import ( "fmt" + "context" logging "github.com/ipfs/go-log" ma "github.com/jbenet/go-multiaddr" manet "github.com/jbenet/go-multiaddr-net" - context "golang.org/x/net/context" _ "github.com/whyrusleeping/ws-transport" ) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index aeeb6ae42d..ab65b8c59f 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -10,11 +10,11 @@ import ( testutil "github.com/libp2p/go-libp2p/testutil" ci "github.com/libp2p/go-libp2p/testutil/ci" + "context" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" manet "github.com/jbenet/go-multiaddr-net" - context "golang.org/x/net/context" ) func closeSwarms(swarms []*Swarm) { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index b63e655b11..5eadf4e671 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -3,10 +3,10 @@ package swarm import ( "testing" + "context" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index c6ee6cf985..bc9cf3ae0f 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -8,10 +8,10 @@ import ( ci "github.com/libp2p/go-libp2p/testutil/ci" + "context" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 772413b286..e221f7bc87 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -7,9 +7,9 @@ import ( addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" testutil "github.com/libp2p/go-libp2p/testutil" + "context" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 762ba6e1d3..40f5bdf0ab 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -6,11 +6,11 @@ import ( inet "github.com/libp2p/go-libp2p/p2p/net" conn "github.com/libp2p/go-libp2p/p2p/net/conn" + "context" ic "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" - context "golang.org/x/net/context" ) // Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 92ccd833af..e1b9a876fb 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -3,6 +3,7 @@ package swarm import ( "fmt" + "context" lgbl "github.com/ipfs/go-libp2p-loggables" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" @@ -10,7 +11,6 @@ import ( mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" inet "github.com/libp2p/go-libp2p/p2p/net" conn "github.com/libp2p/go-libp2p/p2p/net/conn" - context "golang.org/x/net/context" ) func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index abf2f21ce0..d9e83d5fdc 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -8,9 +8,9 @@ import ( metrics "github.com/libp2p/go-libp2p/p2p/metrics" inet "github.com/libp2p/go-libp2p/p2p/net" + "context" ma "github.com/jbenet/go-multiaddr" "github.com/jbenet/goprocess" - context "golang.org/x/net/context" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 4d564e3901..205b841a19 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -7,7 +7,7 @@ import ( inet "github.com/libp2p/go-libp2p/p2p/net" testutil "github.com/libp2p/go-libp2p/p2p/test/util" - context "golang.org/x/net/context" + "context" ) // TestConnectednessCorrect starts a few networks, connects a few diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 06777469f5..2bc663e0a5 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -4,10 +4,10 @@ import ( "testing" "time" + "context" peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" inet "github.com/libp2p/go-libp2p/p2p/net" - context "golang.org/x/net/context" ) func streamsSame(a, b inet.Stream) bool { diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index db8a464423..b7520dc757 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -13,10 +13,10 @@ import ( inet "github.com/libp2p/go-libp2p/p2p/net" testutil "github.com/libp2p/go-libp2p/testutil" + "context" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" - context "golang.org/x/net/context" ) func EchoStreamHandler(stream inet.Stream) { From aa9e1d0d7ebe9d30b76edfc3377ea08e79da955c Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 3 Oct 2016 20:04:59 +0200 Subject: [PATCH 0158/3965] fix(govet): correct fmt for bool is %t --- p2p/transport/tcp/reuseport_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/tcp/reuseport_test.go b/p2p/transport/tcp/reuseport_test.go index ea746cd5f8..1b03ffd33b 100644 --- a/p2p/transport/tcp/reuseport_test.go +++ b/p2p/transport/tcp/reuseport_test.go @@ -41,7 +41,7 @@ func TestReuseError(t *testing.T) { for k, v := range cases { if ReuseErrShouldRetry(k) != v { - t.Fatalf("expected %b for %#v", v, k) + t.Fatalf("expected %t for %#v", v, k) } } From c03a78ff28e77862a44faa836b3c8db2644111b9 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 3 Oct 2016 21:35:40 -0700 Subject: [PATCH 0159/3965] extract protocol and maddr-filter --- p2p/net/swarm/swarm.go | 2 +- p2p/net/swarm/swarm_stream.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 4eebf626be..19ccf13eef 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -15,7 +15,6 @@ import ( mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" inet "github.com/libp2p/go-libp2p/p2p/net" conn "github.com/libp2p/go-libp2p/p2p/net/conn" - filter "github.com/libp2p/go-libp2p/p2p/net/filter" addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" ci "github.com/ipfs/go-libp2p-crypto" @@ -28,6 +27,7 @@ import ( "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" transport "github.com/libp2p/go-libp2p-transport" + filter "github.com/libp2p/go-maddr-filter" tcpt "github.com/libp2p/go-tcp-transport" psmss "github.com/whyrusleeping/go-smux-multistream" spdy "github.com/whyrusleeping/go-smux-spdystream" diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index dc365e8c09..51c846001a 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -1,8 +1,8 @@ package swarm import ( + protocol "github.com/libp2p/go-libp2p-protocol" inet "github.com/libp2p/go-libp2p/p2p/net" - protocol "github.com/libp2p/go-libp2p/p2p/protocol" ps "github.com/jbenet/go-peerstream" ) From 7f10dbda8be07f737fab768d4e627c42cfe069a6 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 12:33:25 -0700 Subject: [PATCH 0160/3965] extract conn, addr-util, and testutil --- p2p/net/swarm/addr/addr.go | 284 ------------------------------- p2p/net/swarm/addr/addr_test.go | 227 ------------------------ p2p/net/swarm/addr/filter.go | 31 ---- p2p/net/swarm/dial_test.go | 9 +- p2p/net/swarm/limiter.go | 5 +- p2p/net/swarm/limiter_test.go | 3 +- p2p/net/swarm/simul_test.go | 5 +- p2p/net/swarm/swarm.go | 4 +- p2p/net/swarm/swarm_addr.go | 5 +- p2p/net/swarm/swarm_addr_test.go | 6 +- p2p/net/swarm/swarm_conn.go | 4 +- p2p/net/swarm/swarm_dial.go | 4 +- p2p/net/swarm/swarm_listen.go | 7 +- p2p/net/swarm/swarm_test.go | 4 +- 14 files changed, 26 insertions(+), 572 deletions(-) delete mode 100644 p2p/net/swarm/addr/addr.go delete mode 100644 p2p/net/swarm/addr/addr_test.go delete mode 100644 p2p/net/swarm/addr/filter.go diff --git a/p2p/net/swarm/addr/addr.go b/p2p/net/swarm/addr/addr.go deleted file mode 100644 index 4a14c43778..0000000000 --- a/p2p/net/swarm/addr/addr.go +++ /dev/null @@ -1,284 +0,0 @@ -package addrutil - -import ( - "fmt" - - "context" - logging "github.com/ipfs/go-log" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" - - _ "github.com/whyrusleeping/ws-transport" -) - -var log = logging.Logger("github.com/libp2p/go-libp2p/p2p/net/swarm/addr") - -// SupportedTransportStrings is the list of supported transports for the swarm. -// These are strings of encapsulated multiaddr protocols. E.g.: -// /ip4/tcp -var SupportedTransportStrings = []string{ - "/ip4/tcp", - "/ip6/tcp", - "/ip4/udp/utp", - "/ip6/udp/utp", - "/ip4/tcp/ws", - "/ip6/tcp/ws", - // "/ip4/udp/udt", disabled because the lib doesnt work on arm - // "/ip6/udp/udt", disabled because the lib doesnt work on arm -} - -// SupportedTransportProtocols is the list of supported transports for the swarm. -// These are []ma.Protocol lists. Populated at runtime from SupportedTransportStrings -var SupportedTransportProtocols = [][]ma.Protocol{} - -func init() { - // initialize SupportedTransportProtocols - transports := make([][]ma.Protocol, len(SupportedTransportStrings)) - for _, s := range SupportedTransportStrings { - t, err := ma.ProtocolsWithString(s) - if err != nil { - panic(err) // important to fix this in the codebase - } - transports = append(transports, t) - } - SupportedTransportProtocols = transports -} - -// FilterAddrs is a filter that removes certain addresses, according the given filters. -// if all filters return true, the address is kept. -func FilterAddrs(a []ma.Multiaddr, filters ...func(ma.Multiaddr) bool) []ma.Multiaddr { - b := make([]ma.Multiaddr, 0, len(a)) - for _, addr := range a { - good := true - for _, filter := range filters { - good = good && filter(addr) - } - if good { - b = append(b, addr) - } - } - return b -} - -// FilterUsableAddrs removes certain addresses -// from a list. the addresses removed are those known NOT -// to work with our network. Namely, addresses with UTP. -func FilterUsableAddrs(a []ma.Multiaddr) []ma.Multiaddr { - return FilterAddrs(a, AddrUsableFunc) -} - -func AddrUsableFunc(m ma.Multiaddr) bool { - return AddrUsable(m, false) -} - -// AddrOverNonLocalIP returns whether the addr uses a non-local ip link -func AddrOverNonLocalIP(a ma.Multiaddr) bool { - split := ma.Split(a) - if len(split) < 1 { - return false - } - if manet.IsIP6LinkLocal(split[0]) { - return false - } - return true -} - -// AddrUsable returns whether our network can use this addr. -// We only use the transports in SupportedTransportStrings, -// and we do not link local addresses. Loopback is ok -// as we need to be able to connect to multiple ipfs nodes -// in the same machine. -func AddrUsable(a ma.Multiaddr, partial bool) bool { - if a == nil { - return false - } - - if !AddrOverNonLocalIP(a) { - return false - } - - // test the address protocol list is in SupportedTransportProtocols - matches := func(supported, test []ma.Protocol) bool { - if len(test) > len(supported) { - return false - } - - // when partial, it's ok if test < supported. - if !partial && len(supported) != len(test) { - return false - } - - for i := range test { - if supported[i].Code != test[i].Code { - return false - } - } - return true - } - - transport := a.Protocols() - for _, supported := range SupportedTransportProtocols { - if matches(supported, transport) { - return true - } - } - - return false -} - -// ResolveUnspecifiedAddress expands an unspecified ip addresses (/ip4/0.0.0.0, /ip6/::) to -// use the known local interfaces. If ifaceAddr is nil, we request interface addresses -// from the network stack. (this is so you can provide a cached value if resolving many addrs) -func ResolveUnspecifiedAddress(resolve ma.Multiaddr, ifaceAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { - // split address into its components - split := ma.Split(resolve) - - // if first component (ip) is not unspecified, use it as is. - if !manet.IsIPUnspecified(split[0]) { - return []ma.Multiaddr{resolve}, nil - } - - out := make([]ma.Multiaddr, 0, len(ifaceAddrs)) - for _, ia := range ifaceAddrs { - // must match the first protocol to be resolve. - if ia.Protocols()[0].Code != resolve.Protocols()[0].Code { - continue - } - - split[0] = ia - joined := ma.Join(split...) - out = append(out, joined) - log.Debug("adding resolved addr:", resolve, joined, out) - } - if len(out) < 1 { - return nil, fmt.Errorf("failed to resolve: %s", resolve) - } - return out, nil -} - -// ResolveUnspecifiedAddresses expands unspecified ip addresses (/ip4/0.0.0.0, /ip6/::) to -// use the known local interfaces. -func ResolveUnspecifiedAddresses(unspecAddrs, ifaceAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { - - // todo optimize: only fetch these if we have a "any" addr. - if len(ifaceAddrs) < 1 { - var err error - ifaceAddrs, err = InterfaceAddresses() - if err != nil { - return nil, err - } - // log.Debug("InterfaceAddresses:", ifaceAddrs) - } - - var outputAddrs []ma.Multiaddr - for _, a := range unspecAddrs { - // unspecified? - resolved, err := ResolveUnspecifiedAddress(a, ifaceAddrs) - if err != nil { - continue // optimistic. if we cant resolve anything, we'll know at the bottom. - } - // log.Debug("resolved:", a, resolved) - outputAddrs = append(outputAddrs, resolved...) - } - - if len(outputAddrs) < 1 { - return nil, fmt.Errorf("failed to specify addrs: %s", unspecAddrs) - } - - log.Event(context.TODO(), "interfaceListenAddresses", func() logging.Loggable { - var addrs []string - for _, addr := range outputAddrs { - addrs = append(addrs, addr.String()) - } - return logging.Metadata{"addresses": addrs} - }()) - - log.Debug("ResolveUnspecifiedAddresses:", unspecAddrs, ifaceAddrs, outputAddrs) - return outputAddrs, nil -} - -// InterfaceAddresses returns a list of addresses associated with local machine -// Note: we do not return link local addresses. IP loopback is ok, because we -// may be connecting to other nodes in the same machine. -func InterfaceAddresses() ([]ma.Multiaddr, error) { - maddrs, err := manet.InterfaceMultiaddrs() - if err != nil { - return nil, err - } - log.Debug("InterfaceAddresses: from manet:", maddrs) - - var out []ma.Multiaddr - for _, a := range maddrs { - if !AddrUsable(a, true) { // partial - // log.Debug("InterfaceAddresses: skipping unusable:", a) - continue - } - - out = append(out, a) - } - - log.Debug("InterfaceAddresses: usable:", out) - return out, nil -} - -// AddrInList returns whether or not an address is part of a list. -// this is useful to check if NAT is happening (or other bugs?) -func AddrInList(addr ma.Multiaddr, list []ma.Multiaddr) bool { - for _, addr2 := range list { - if addr.Equal(addr2) { - return true - } - } - return false -} - -// AddrIsShareableOnWAN returns whether the given address should be shareable on the -// wide area network (wide internet). -func AddrIsShareableOnWAN(addr ma.Multiaddr) bool { - s := ma.Split(addr) - if len(s) < 1 { - return false - } - a := s[0] - if manet.IsIPLoopback(a) || manet.IsIP6LinkLocal(a) || manet.IsIPUnspecified(a) { - return false - } - return manet.IsThinWaist(a) -} - -// WANShareableAddrs filters addresses based on whether they're shareable on WAN -func WANShareableAddrs(inp []ma.Multiaddr) []ma.Multiaddr { - return FilterAddrs(inp, AddrIsShareableOnWAN) -} - -// Subtract filters out all addrs in b from a -func Subtract(a, b []ma.Multiaddr) []ma.Multiaddr { - return FilterAddrs(a, func(m ma.Multiaddr) bool { - for _, bb := range b { - if m.Equal(bb) { - return false - } - } - return true - }) -} - -// CheckNATWarning checks if our observed addresses differ. if so, -// informs the user that certain things might not work yet -func CheckNATWarning(observed, expected ma.Multiaddr, listen []ma.Multiaddr) { - if observed.Equal(expected) { - return - } - - if !AddrInList(observed, listen) { // probably a nat - log.Warningf(natWarning, observed, listen) - } -} - -const natWarning = `Remote peer observed our address to be: %s -The local addresses are: %s -Thus, connection is going through NAT, and other connections may fail. - -IPFS NAT traversal is still under development. Please bug us on github or irc to fix this. -Baby steps: http://jbenet.static.s3.amazonaws.com/271dfcf/baby-steps.gif -` diff --git a/p2p/net/swarm/addr/addr_test.go b/p2p/net/swarm/addr/addr_test.go deleted file mode 100644 index 213b7b9f1a..0000000000 --- a/p2p/net/swarm/addr/addr_test.go +++ /dev/null @@ -1,227 +0,0 @@ -package addrutil - -import ( - "testing" - - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" -) - -func newMultiaddr(t *testing.T, s string) ma.Multiaddr { - maddr, err := ma.NewMultiaddr(s) - if err != nil { - t.Fatal(err) - } - return maddr -} - -func TestFilterAddrs(t *testing.T) { - - bad := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/1.2.3.4/udp/1234"), // unreliable - newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/sctp/1234"), // not in manet - newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/udt"), // udt is broken on arm - newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), // link local - newMultiaddr(t, "/ip6/fe80::100/tcp/1234"), // link local - } - - good := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), - newMultiaddr(t, "/ip6/::1/tcp/1234"), - newMultiaddr(t, "/ip4/1.2.3.4/udp/1234/utp"), - newMultiaddr(t, "/ip4/1.2.3.4/tcp/1234/ws"), - } - - goodAndBad := append(good, bad...) - - // test filters - - for _, a := range bad { - if AddrUsable(a, false) { - t.Errorf("addr %s should be unusable", a) - } - } - - for _, a := range good { - if !AddrUsable(a, false) { - t.Errorf("addr %s should be usable", a) - } - } - - subtestAddrsEqual(t, FilterUsableAddrs(bad), []ma.Multiaddr{}) - subtestAddrsEqual(t, FilterUsableAddrs(good), good) - subtestAddrsEqual(t, FilterUsableAddrs(goodAndBad), good) -} - -func subtestAddrsEqual(t *testing.T, a, b []ma.Multiaddr) { - if len(a) != len(b) { - t.Error(t) - } - - in := func(addr ma.Multiaddr, l []ma.Multiaddr) bool { - for _, addr2 := range l { - if addr.Equal(addr2) { - return true - } - } - return false - } - - for _, aa := range a { - if !in(aa, b) { - t.Errorf("%s not in %s", aa, b) - } - } -} - -func TestInterfaceAddrs(t *testing.T) { - addrs, err := InterfaceAddresses() - if err != nil { - t.Fatal(err) - } - - if len(addrs) < 1 { - t.Error("no addresses") - } - - for _, a := range addrs { - if manet.IsIP6LinkLocal(a) { - t.Error("should not return ip link local addresses", a) - } - } - - if len(addrs) < 1 { - t.Error("no good interface addrs") - } -} - -func TestResolvingAddrs(t *testing.T) { - - unspec := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/0.0.0.0/tcp/1234"), - newMultiaddr(t, "/ip4/1.2.3.4/tcp/1234"), - newMultiaddr(t, "/ip6/::/tcp/1234"), - newMultiaddr(t, "/ip6/::100/tcp/1234"), - } - - iface := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/127.0.0.1"), - newMultiaddr(t, "/ip4/10.20.30.40"), - newMultiaddr(t, "/ip6/::1"), - newMultiaddr(t, "/ip6/::f"), - } - - spec := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), - newMultiaddr(t, "/ip4/10.20.30.40/tcp/1234"), - newMultiaddr(t, "/ip4/1.2.3.4/tcp/1234"), - newMultiaddr(t, "/ip6/::1/tcp/1234"), - newMultiaddr(t, "/ip6/::f/tcp/1234"), - newMultiaddr(t, "/ip6/::100/tcp/1234"), - } - - actual, err := ResolveUnspecifiedAddresses(unspec, iface) - if err != nil { - t.Fatal(err) - } - - for i, a := range actual { - if !a.Equal(spec[i]) { - t.Error(a, " != ", spec[i]) - } - } - - ip4u := []ma.Multiaddr{newMultiaddr(t, "/ip4/0.0.0.0")} - ip4i := []ma.Multiaddr{newMultiaddr(t, "/ip4/1.2.3.4")} - - ip6u := []ma.Multiaddr{newMultiaddr(t, "/ip6/::")} - ip6i := []ma.Multiaddr{newMultiaddr(t, "/ip6/::1")} - - if _, err := ResolveUnspecifiedAddress(ip4u[0], ip6i); err == nil { - t.Fatal("should have failed") - } - if _, err := ResolveUnspecifiedAddress(ip6u[0], ip4i); err == nil { - t.Fatal("should have failed") - } - - if _, err := ResolveUnspecifiedAddresses(ip6u, ip4i); err == nil { - t.Fatal("should have failed") - } - if _, err := ResolveUnspecifiedAddresses(ip4u, ip6i); err == nil { - t.Fatal("should have failed") - } - -} - -func TestWANShareable(t *testing.T) { - - wanok := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/1.2.3.4/tcp/1234"), - newMultiaddr(t, "/ip6/abcd::1/tcp/1234"), - } - - wanbad := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), - newMultiaddr(t, "/ip4/0.0.0.0/tcp/1234"), - newMultiaddr(t, "/ip6/::1/tcp/1234"), - newMultiaddr(t, "/ip6/::/tcp/1234"), - newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), - newMultiaddr(t, "/ip6/fe80::/tcp/1234"), - } - - for _, a := range wanok { - if !AddrIsShareableOnWAN(a) { - t.Error("should be true", a) - } - } - - for _, a := range wanbad { - if AddrIsShareableOnWAN(a) { - t.Error("should be false", a) - } - } - - wanok2 := WANShareableAddrs(wanok) - if len(wanok) != len(wanok2) { - t.Error("should be the same") - } - - wanbad2 := WANShareableAddrs(wanbad) - if len(wanbad2) != 0 { - t.Error("should be zero") - } -} - -func TestSubtract(t *testing.T) { - - a := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), - newMultiaddr(t, "/ip4/0.0.0.0/tcp/1234"), - newMultiaddr(t, "/ip6/::1/tcp/1234"), - newMultiaddr(t, "/ip6/::/tcp/1234"), - newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), - newMultiaddr(t, "/ip6/fe80::/tcp/1234"), - } - - b := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/127.0.0.1/tcp/1234"), - newMultiaddr(t, "/ip6/::1/tcp/1234"), - newMultiaddr(t, "/ip6/fe80::1/tcp/1234"), - } - - c1 := []ma.Multiaddr{ - newMultiaddr(t, "/ip4/0.0.0.0/tcp/1234"), - newMultiaddr(t, "/ip6/::/tcp/1234"), - newMultiaddr(t, "/ip6/fe80::/tcp/1234"), - } - - c2 := Subtract(a, b) - if len(c1) != len(c2) { - t.Error("should be the same") - } - for i, ca := range c1 { - if !c2[i].Equal(ca) { - t.Error("should be the same", ca, c2[i]) - } - } -} diff --git a/p2p/net/swarm/addr/filter.go b/p2p/net/swarm/addr/filter.go deleted file mode 100644 index d87ba816a7..0000000000 --- a/p2p/net/swarm/addr/filter.go +++ /dev/null @@ -1,31 +0,0 @@ -package addrutil - -import ( - ma "github.com/jbenet/go-multiaddr" - mafmt "github.com/whyrusleeping/mafmt" -) - -// SubtractFilter returns a filter func that filters all of the given addresses -func SubtractFilter(addrs ...ma.Multiaddr) func(ma.Multiaddr) bool { - addrmap := make(map[string]bool) - for _, a := range addrs { - addrmap[string(a.Bytes())] = true - } - - return func(a ma.Multiaddr) bool { - return !addrmap[string(a.Bytes())] - } -} - -// IsFDCostlyTransport returns true for transports that require a new file -// descriptor per connection created -func IsFDCostlyTransport(a ma.Multiaddr) bool { - return mafmt.TCP.Matches(a) -} - -// FilterNeg returns a negated version of the passed in filter -func FilterNeg(f func(ma.Multiaddr) bool) func(ma.Multiaddr) bool { - return func(a ma.Multiaddr) bool { - return !f(a) - } -} diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index ab65b8c59f..8e7b48fc2b 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -1,20 +1,19 @@ package swarm import ( + "context" "net" "sync" "testing" "time" - addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" - testutil "github.com/libp2p/go-libp2p/testutil" - ci "github.com/libp2p/go-libp2p/testutil/ci" - - "context" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" manet "github.com/jbenet/go-multiaddr-net" + addrutil "github.com/libp2p/go-addr-util" + testutil "github.com/libp2p/go-testutil" + ci "github.com/libp2p/go-testutil/ci" ) func closeSwarms(swarms []*Swarm) { diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index c34954a527..5c8d235b8e 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -6,9 +6,8 @@ import ( peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" - - conn "github.com/libp2p/go-libp2p/p2p/net/conn" - addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" + addrutil "github.com/libp2p/go-addr-util" + conn "github.com/libp2p/go-libp2p-conn" ) type dialResult struct { diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 93761a5048..7332c0e06a 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -10,9 +10,8 @@ import ( peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" + conn "github.com/libp2p/go-libp2p-conn" mafmt "github.com/whyrusleeping/mafmt" - - conn "github.com/libp2p/go-libp2p/p2p/net/conn" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index bc9cf3ae0f..3518f139e6 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -1,17 +1,16 @@ package swarm import ( + "context" "runtime" "sync" "testing" "time" - ci "github.com/libp2p/go-libp2p/testutil/ci" - - "context" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" + ci "github.com/libp2p/go-testutil/ci" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 19ccf13eef..9107d7160a 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -14,8 +14,6 @@ import ( metrics "github.com/libp2p/go-libp2p/p2p/metrics" mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" inet "github.com/libp2p/go-libp2p/p2p/net" - conn "github.com/libp2p/go-libp2p/p2p/net/conn" - addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" ci "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" @@ -26,6 +24,8 @@ import ( pst "github.com/jbenet/go-stream-muxer" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" + addrutil "github.com/libp2p/go-addr-util" + conn "github.com/libp2p/go-libp2p-conn" transport "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" tcpt "github.com/libp2p/go-tcp-transport" diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 39fbe02afc..87b0829da1 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -1,10 +1,9 @@ package swarm import ( - conn "github.com/libp2p/go-libp2p/p2p/net/conn" - addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" - ma "github.com/jbenet/go-multiaddr" + addrutil "github.com/libp2p/go-addr-util" + conn "github.com/libp2p/go-libp2p-conn" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index e221f7bc87..bd48d4f601 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -1,15 +1,15 @@ package swarm import ( + "context" "testing" metrics "github.com/libp2p/go-libp2p/p2p/metrics" - addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" - testutil "github.com/libp2p/go-libp2p/testutil" - "context" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" + addrutil "github.com/libp2p/go-addr-util" + testutil "github.com/libp2p/go-testutil" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 40f5bdf0ab..30ef800edb 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -1,16 +1,16 @@ package swarm import ( + "context" "fmt" inet "github.com/libp2p/go-libp2p/p2p/net" - conn "github.com/libp2p/go-libp2p/p2p/net/conn" - "context" ic "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" + conn "github.com/libp2p/go-libp2p-conn" ) // Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 6d1abd3342..29694df73b 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -10,8 +10,8 @@ import ( lgbl "github.com/ipfs/go-libp2p-loggables" peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" - conn "github.com/libp2p/go-libp2p/p2p/net/conn" - addrutil "github.com/libp2p/go-libp2p/p2p/net/swarm/addr" + addrutil "github.com/libp2p/go-addr-util" + conn "github.com/libp2p/go-libp2p-conn" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index e1b9a876fb..702ef54b7e 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -1,16 +1,17 @@ package swarm import ( + "context" "fmt" - "context" + inet "github.com/libp2p/go-libp2p/p2p/net" + lgbl "github.com/ipfs/go-libp2p-loggables" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" + conn "github.com/libp2p/go-libp2p-conn" transport "github.com/libp2p/go-libp2p-transport" mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" - inet "github.com/libp2p/go-libp2p/p2p/net" - conn "github.com/libp2p/go-libp2p/p2p/net/conn" ) func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index b7520dc757..e7099812db 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -2,6 +2,7 @@ package swarm import ( "bytes" + "context" "fmt" "io" "net" @@ -11,12 +12,11 @@ import ( metrics "github.com/libp2p/go-libp2p/p2p/metrics" inet "github.com/libp2p/go-libp2p/p2p/net" - testutil "github.com/libp2p/go-libp2p/testutil" - "context" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" + testutil "github.com/libp2p/go-testutil" ) func EchoStreamHandler(stream inet.Stream) { From dfae5c7dd6e6c6aff67f1ffc3002236e6b48ed1f Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 13:50:34 -0700 Subject: [PATCH 0161/3965] extract net interface and metrics --- p2p/net/swarm/swarm.go | 7 +++---- p2p/net/swarm/swarm_addr_test.go | 3 +-- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_listen.go | 5 ++--- p2p/net/swarm/swarm_net.go | 7 +++---- p2p/net/swarm/swarm_net_test.go | 4 ++-- p2p/net/swarm/swarm_notif_test.go | 2 +- p2p/net/swarm/swarm_stream.go | 2 +- p2p/net/swarm/swarm_test.go | 5 ++--- 9 files changed, 16 insertions(+), 21 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 9107d7160a..77c7daff04 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -11,10 +11,6 @@ import ( "sync" "time" - metrics "github.com/libp2p/go-libp2p/p2p/metrics" - mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" - inet "github.com/libp2p/go-libp2p/p2p/net" - ci "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" @@ -26,6 +22,9 @@ import ( goprocessctx "github.com/jbenet/goprocess/context" addrutil "github.com/libp2p/go-addr-util" conn "github.com/libp2p/go-libp2p-conn" + metrics "github.com/libp2p/go-libp2p-metrics" + mconn "github.com/libp2p/go-libp2p-metrics/conn" + inet "github.com/libp2p/go-libp2p-net" transport "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" tcpt "github.com/libp2p/go-tcp-transport" diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index bd48d4f601..5d2225405a 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -4,11 +4,10 @@ import ( "context" "testing" - metrics "github.com/libp2p/go-libp2p/p2p/metrics" - pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" addrutil "github.com/libp2p/go-addr-util" + metrics "github.com/libp2p/go-libp2p-metrics" testutil "github.com/libp2p/go-testutil" ) diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 30ef800edb..42fc8905da 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -4,7 +4,7 @@ import ( "context" "fmt" - inet "github.com/libp2p/go-libp2p/p2p/net" + inet "github.com/libp2p/go-libp2p-net" ic "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 702ef54b7e..0c9d859735 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -4,14 +4,13 @@ import ( "context" "fmt" - inet "github.com/libp2p/go-libp2p/p2p/net" - lgbl "github.com/ipfs/go-libp2p-loggables" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" conn "github.com/libp2p/go-libp2p-conn" + mconn "github.com/libp2p/go-libp2p-metrics/conn" + inet "github.com/libp2p/go-libp2p-net" transport "github.com/libp2p/go-libp2p-transport" - mconn "github.com/libp2p/go-libp2p/p2p/metrics/conn" ) func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index d9e83d5fdc..86755cf8be 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -1,16 +1,15 @@ package swarm import ( + "context" "fmt" peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" - metrics "github.com/libp2p/go-libp2p/p2p/metrics" - inet "github.com/libp2p/go-libp2p/p2p/net" - - "context" ma "github.com/jbenet/go-multiaddr" "github.com/jbenet/goprocess" + metrics "github.com/libp2p/go-libp2p-metrics" + inet "github.com/libp2p/go-libp2p-net" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 205b841a19..ebf3bb82a2 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -5,9 +5,9 @@ import ( "testing" "time" - inet "github.com/libp2p/go-libp2p/p2p/net" - testutil "github.com/libp2p/go-libp2p/p2p/test/util" "context" + inet "github.com/libp2p/go-libp2p-net" + testutil "github.com/libp2p/go-libp2p/p2p/test/util" ) // TestConnectednessCorrect starts a few networks, connects a few diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 2bc663e0a5..9e914b8218 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -7,7 +7,7 @@ import ( "context" peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" - inet "github.com/libp2p/go-libp2p/p2p/net" + inet "github.com/libp2p/go-libp2p-net" ) func streamsSame(a, b inet.Stream) bool { diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 51c846001a..ef3ab3c8fd 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -1,8 +1,8 @@ package swarm import ( + inet "github.com/libp2p/go-libp2p-net" protocol "github.com/libp2p/go-libp2p-protocol" - inet "github.com/libp2p/go-libp2p/p2p/net" ps "github.com/jbenet/go-peerstream" ) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index e7099812db..f0ed560cf2 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -10,12 +10,11 @@ import ( "testing" "time" - metrics "github.com/libp2p/go-libp2p/p2p/metrics" - inet "github.com/libp2p/go-libp2p/p2p/net" - peer "github.com/ipfs/go-libp2p-peer" pstore "github.com/ipfs/go-libp2p-peerstore" ma "github.com/jbenet/go-multiaddr" + metrics "github.com/libp2p/go-libp2p-metrics" + inet "github.com/libp2p/go-libp2p-net" testutil "github.com/libp2p/go-testutil" ) From 26422f282b81406b2ddbf985bb17a1642cee3af0 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 15:39:24 -0700 Subject: [PATCH 0162/3965] switching to separated conn and interface-conn packages --- p2p/net/swarm/limiter.go | 8 ++++---- p2p/net/swarm/limiter_test.go | 16 ++++++++-------- p2p/net/swarm/swarm_addr.go | 4 ++-- p2p/net/swarm/swarm_conn.go | 11 +++++------ p2p/net/swarm/swarm_dial.go | 8 ++++---- p2p/net/swarm/swarm_listen.go | 3 ++- 6 files changed, 25 insertions(+), 25 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 5c8d235b8e..bf4bcf0f5c 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -7,11 +7,11 @@ import ( peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" addrutil "github.com/libp2p/go-addr-util" - conn "github.com/libp2p/go-libp2p-conn" + iconn "github.com/libp2p/go-libp2p-interface-conn" ) type dialResult struct { - Conn conn.Conn + Conn iconn.Conn Err error } @@ -38,14 +38,14 @@ type dialLimiter struct { fdLimit int waitingOnFd []*dialJob - dialFunc func(context.Context, peer.ID, ma.Multiaddr) (conn.Conn, error) + dialFunc func(context.Context, peer.ID, ma.Multiaddr) (iconn.Conn, error) activePerPeer map[peer.ID]int perPeerLimit int waitingOnPeerLimit map[peer.ID][]*dialJob } -type dialfunc func(context.Context, peer.ID, ma.Multiaddr) (conn.Conn, error) +type dialfunc func(context.Context, peer.ID, ma.Multiaddr) (iconn.Conn, error) func newDialLimiter(df dialfunc) *dialLimiter { return newDialLimiterWithParams(df, concurrentFdDials, defaultPerPeerRateLimit) diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 7332c0e06a..31ac8435df 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -10,7 +10,7 @@ import ( peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" - conn "github.com/libp2p/go-libp2p-conn" + iconn "github.com/libp2p/go-libp2p-interface-conn" mafmt "github.com/whyrusleeping/mafmt" ) @@ -55,13 +55,13 @@ func tryDialAddrs(ctx context.Context, l *dialLimiter, p peer.ID, addrs []ma.Mul } func hangDialFunc(hang chan struct{}) dialfunc { - return func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { + return func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { if mafmt.UTP.Matches(a) { - return conn.Conn(nil), nil + return iconn.Conn(nil), nil } if tcpPortOver(a, 10) { - return conn.Conn(nil), nil + return iconn.Conn(nil), nil } <-hang @@ -171,9 +171,9 @@ func TestFDLimiting(t *testing.T) { func TestTokenRedistribution(t *testing.T) { hangchs := make(map[peer.ID]chan struct{}) - df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { if tcpPortOver(a, 10) { - return (conn.Conn)(nil), nil + return (iconn.Conn)(nil), nil } <-hangchs[p] @@ -260,9 +260,9 @@ func TestTokenRedistribution(t *testing.T) { } func TestStressLimiter(t *testing.T) { - df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (conn.Conn, error) { + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { if tcpPortOver(a, 1000) { - return conn.Conn(nil), nil + return iconn.Conn(nil), nil } time.Sleep(time.Millisecond * time.Duration(5+rand.Intn(100))) diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 87b0829da1..0ab2c7ea1c 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -3,7 +3,7 @@ package swarm import ( ma "github.com/jbenet/go-multiaddr" addrutil "github.com/libp2p/go-addr-util" - conn "github.com/libp2p/go-libp2p-conn" + iconn "github.com/libp2p/go-libp2p-interface-conn" ) // ListenAddresses returns a list of addresses at which this swarm listens. @@ -11,7 +11,7 @@ func (s *Swarm) ListenAddresses() []ma.Multiaddr { listeners := s.swarm.Listeners() addrs := make([]ma.Multiaddr, 0, len(listeners)) for _, l := range listeners { - if l2, ok := l.NetListener().(conn.Listener); ok { + if l2, ok := l.NetListener().(iconn.Listener); ok { addrs = append(addrs, l2.Multiaddr()) } } diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 42fc8905da..8128a3e6ab 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -4,13 +4,12 @@ import ( "context" "fmt" - inet "github.com/libp2p/go-libp2p-net" - ic "github.com/ipfs/go-libp2p-crypto" peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" - conn "github.com/libp2p/go-libp2p-conn" + iconn "github.com/libp2p/go-libp2p-interface-conn" + inet "github.com/libp2p/go-libp2p-net" ) // Conn is a simple wrapper around a ps.Conn that also exposes @@ -33,12 +32,12 @@ func (c *Conn) StreamConn() *ps.Conn { return (*ps.Conn)(c) } -func (c *Conn) RawConn() conn.Conn { +func (c *Conn) RawConn() iconn.Conn { // righly panic if these things aren't true. it is an expected // invariant that these Conns are all of the typewe expect: // ps.Conn wrapping a conn.Conn // if we get something else it is programmer error. - return (*ps.Conn)(c).NetConn().(conn.Conn) + return (*ps.Conn)(c).NetConn().(iconn.Conn) } func (c *Conn) String() string { @@ -94,7 +93,7 @@ func (c *Conn) Close() error { func wrapConn(psc *ps.Conn) (*Conn, error) { // grab the underlying connection. - if _, ok := psc.NetConn().(conn.Conn); !ok { + if _, ok := psc.NetConn().(iconn.Conn); !ok { // this should never happen. if we see it ocurring it means that we added // a Listener to the ps.Swarm that is NOT one of our net/conn.Listener. return nil, fmt.Errorf("swarm connHandler: invalid conn (not a conn.Conn): %s", psc) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 29694df73b..bfe9f0382e 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -11,7 +11,7 @@ import ( peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" addrutil "github.com/libp2p/go-addr-util" - conn "github.com/libp2p/go-libp2p-conn" + iconn "github.com/libp2p/go-libp2p-interface-conn" ) // Diagram of dial sync: @@ -284,7 +284,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { return swarmC, nil } -func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (conn.Conn, error) { +func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (iconn.Conn, error) { log.Debugf("%s swarm dialing %s %s", s.local, p, remoteAddrs) ctx, cancel := context.WithCancel(ctx) @@ -344,7 +344,7 @@ func (s *Swarm) limitedDial(ctx context.Context, p peer.ID, a ma.Multiaddr, resp }) } -func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (conn.Conn, error) { +func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (iconn.Conn, error) { log.Debugf("%s swarm dialing %s %s", s.local, p, addr) connC, err := s.dialer.Dial(ctx, addr, p) @@ -376,7 +376,7 @@ var ConnSetupTimeout = time.Minute * 5 // dialConnSetup is the setup logic for a connection from the dial side. it // needs to add the Conn to the StreamSwarm, then run newConnSetup -func dialConnSetup(ctx context.Context, s *Swarm, connC conn.Conn) (*Conn, error) { +func dialConnSetup(ctx context.Context, s *Swarm, connC iconn.Conn) (*Conn, error) { deadline, ok := ctx.Deadline() if !ok { diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 0c9d859735..756084abc7 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -8,6 +8,7 @@ import ( ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" conn "github.com/libp2p/go-libp2p-conn" + iconn "github.com/libp2p/go-libp2p-interface-conn" mconn "github.com/libp2p/go-libp2p-metrics/conn" inet "github.com/libp2p/go-libp2p-net" transport "github.com/libp2p/go-libp2p-transport" @@ -98,7 +99,7 @@ func (s *Swarm) addListener(tptlist transport.Listener) error { return s.addConnListener(list) } -func (s *Swarm) addConnListener(list conn.Listener) error { +func (s *Swarm) addConnListener(list iconn.Listener) error { // AddListener to the peerstream Listener. this will begin accepting connections // and streams! sl, err := s.swarm.AddListener(list) From dbd5a4a1597d5a5431e69725818dd7e09d0235fd Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 16:19:16 -0700 Subject: [PATCH 0163/3965] use context from stdlib --- p2p/host/peerstore/addr_manager.go | 2 +- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/peerstore_test.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 3 +-- p2p/host/peerstore/queue/sync.go | 3 ++- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 877af2c4cf..eb7afd3813 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -1,6 +1,7 @@ package peerstore import ( + "context" "sort" "sync" "time" @@ -8,7 +9,6 @@ import ( peer "github.com/ipfs/go-libp2p-peer" addr "github.com/ipfs/go-libp2p-peerstore/addr" ma "github.com/jbenet/go-multiaddr" - "golang.org/x/net/context" ) const ( diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 7518b453da..c87c7c19d4 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -1,6 +1,7 @@ package peerstore import ( + "context" "errors" "fmt" "sync" @@ -13,7 +14,6 @@ import ( "github.com/ipfs/go-libp2p-peer" logging "github.com/ipfs/go-log" ma "github.com/jbenet/go-multiaddr" - "golang.org/x/net/context" ) var log = logging.Logger("peerstore") diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 44d7c2259d..a1a3096ea9 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -1,6 +1,7 @@ package peerstore import ( + "context" "fmt" "math/rand" "sort" @@ -9,7 +10,6 @@ import ( peer "github.com/ipfs/go-libp2p-peer" ma "github.com/jbenet/go-multiaddr" - "golang.org/x/net/context" ) func getAddrs(t *testing.T, n int) []ma.Multiaddr { diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 2daa5bdaf8..94c24e326b 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -1,6 +1,7 @@ package queue import ( + "context" "fmt" "sync" "testing" @@ -8,8 +9,6 @@ import ( u "github.com/ipfs/go-ipfs-util" peer "github.com/ipfs/go-libp2p-peer" - - context "golang.org/x/net/context" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 0ff94f7628..8ceedce19c 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -1,9 +1,10 @@ package queue import ( + "context" + peer "github.com/ipfs/go-libp2p-peer" logging "github.com/ipfs/go-log" - context "golang.org/x/net/context" ) var log = logging.Logger("peerqueue") From f782339c6abc9e9bdcf0c0239ad1ad15449ab4e3 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 18:01:51 -0700 Subject: [PATCH 0164/3965] fix CI, vet, and annoying types --- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/metrics.go | 2 +- p2p/host/peerstore/peerstore.go | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index a480ee71a3..04a2a972f8 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -40,7 +40,7 @@ func testHas(t *testing.T, exp, act []ma.Multiaddr) { } if !found { - t.Fatal("expected address %s not found", a) + t.Fatalf("expected address %s not found", a) } } } diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index 2045ed9c93..94b0acc2d7 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -29,7 +29,7 @@ type metrics struct { latmu sync.RWMutex } -func NewMetrics() Metrics { +func NewMetrics() *metrics { return &metrics{ latmap: make(map[peer.ID]time.Duration), } diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index c87c7c19d4..e26c0ad969 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -163,8 +163,8 @@ func (kb *keybook) AddPrivKey(p peer.ID, sk ic.PrivKey) error { } type peerstore struct { - keybook - metrics + *keybook + *metrics AddrManager // store other data, like versions @@ -180,8 +180,8 @@ type peerstore struct { // NewPeerstore creates a threadsafe collection of peers. func NewPeerstore() Peerstore { return &peerstore{ - keybook: *newKeybook(), - metrics: *(NewMetrics()).(*metrics), + keybook: newKeybook(), + metrics: NewMetrics(), AddrManager: AddrManager{}, //ds: dssync.MutexWrap(ds.NewMapDatastore()), ds: make(map[string]interface{}), From f193b1e8e2a11b16f608b8c46f70ec97eb6a4f32 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 18:05:58 -0700 Subject: [PATCH 0165/3965] update package location --- p2p/host/peerstore/addr_manager.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index eb7afd3813..303e8e0ef7 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -7,8 +7,8 @@ import ( "time" peer "github.com/ipfs/go-libp2p-peer" - addr "github.com/ipfs/go-libp2p-peerstore/addr" ma "github.com/jbenet/go-multiaddr" + addr "github.com/libp2p/go-libp2p-peerstore/addr" ) const ( From 7752178a41c65e859aabf214fca664f24dafe220 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 18:38:14 -0700 Subject: [PATCH 0166/3965] gx publish 1.3.0 --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr/sorting.go | 4 ++-- p2p/host/peerstore/addr_manager.go | 4 ++-- p2p/host/peerstore/addr_manager_test.go | 4 ++-- p2p/host/peerstore/metrics.go | 2 +- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peerinfo.go | 4 ++-- p2p/host/peerstore/peerinfo_test.go | 5 +++-- p2p/host/peerstore/peerstore.go | 6 +++--- p2p/host/peerstore/peerstore_test.go | 4 ++-- p2p/host/peerstore/queue/distance.go | 2 +- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 2 +- p2p/host/peerstore/queue/sync.go | 2 +- p2p/host/peerstore/test/utils.go | 4 ++-- 16 files changed, 26 insertions(+), 25 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index a172373d8d..9be78d5ccc 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "github.com/jbenet/go-multiaddr" + ma "github.com/multiformats/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 46140641ca..06bea07bce 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "github.com/jbenet/go-multiaddr" + ma "github.com/multiformats/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index 084722ab57..f8a89150a2 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -3,8 +3,8 @@ package addr import ( "bytes" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" ) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 303e8e0ef7..a676f44252 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -6,9 +6,9 @@ import ( "sync" "time" - peer "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" + peer "github.com/libp2p/go-libp2p-peer" addr "github.com/libp2p/go-libp2p-peerstore/addr" + ma "github.com/multiformats/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 04a2a972f8..b9cbfd91b1 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" - "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func IDS(t *testing.T, ids string) peer.ID { diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index 94b0acc2d7..05b867692b 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -4,7 +4,7 @@ import ( "sync" "time" - "github.com/ipfs/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" ) // LatencyEWMASmooting governs the decay of the EWMA (the speed diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 110b6cf103..94a741182d 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - testutil "github.com/ipfs/go-libp2p-peer/test" + testutil "github.com/libp2p/go-libp2p-peer/test" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index 584d363c1a..a42d5ef006 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -3,8 +3,8 @@ package peerstore import ( "encoding/json" - "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) // PeerInfo is a small struct used to pass around a peer with diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index b78a662999..ca73430a18 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -1,9 +1,10 @@ package peerstore import ( - peer "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" "testing" + + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index e26c0ad969..98cc4605bc 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -7,13 +7,13 @@ import ( "sync" "time" - ic "github.com/ipfs/go-libp2p-crypto" + ic "github.com/libp2p/go-libp2p-crypto" //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" - "github.com/ipfs/go-libp2p-peer" logging "github.com/ipfs/go-log" - ma "github.com/jbenet/go-multiaddr" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) var log = logging.Logger("peerstore") diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index a1a3096ea9..29cf6fa7c5 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -8,8 +8,8 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func getAddrs(t *testing.T, n int) []ma.Multiaddr { diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 7249c79c03..f89e8d4210 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,7 +5,7 @@ import ( "math/big" "sync" - peer "github.com/ipfs/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ks "github.com/whyrusleeping/go-keyspace" ) diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index 352d8077ca..eab523a125 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import peer "github.com/ipfs/go-libp2p-peer" +import peer "github.com/libp2p/go-libp2p-peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 94c24e326b..cd3c80de12 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -8,7 +8,7 @@ import ( "time" u "github.com/ipfs/go-ipfs-util" - peer "github.com/ipfs/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 8ceedce19c..a1edc2ccbc 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -3,8 +3,8 @@ package queue import ( "context" - peer "github.com/ipfs/go-libp2p-peer" logging "github.com/ipfs/go-log" + peer "github.com/libp2p/go-libp2p-peer" ) var log = logging.Logger("peerqueue") diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index c9ee61c41f..6c4f0be5fc 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -4,8 +4,8 @@ import ( "io" u "github.com/ipfs/go-ipfs-util" - ci "github.com/ipfs/go-libp2p-crypto" - peer "github.com/ipfs/go-libp2p-peer" + ci "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" ) func RandPeerID() (peer.ID, error) { From fa8d0d27d1832e727926dc0789ba4c484e060cd1 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 19:24:36 -0700 Subject: [PATCH 0167/3965] gx publish 1.4.0 --- p2p/transport/websocket/websocket.go | 5 ++--- p2p/transport/websocket/ws_test.go | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index ee8678a2cc..69fc11c408 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -7,12 +7,11 @@ import ( "net/http" "net/url" - manet "github.com/jbenet/go-multiaddr-net" tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" ws "golang.org/x/net/websocket" - - ma "github.com/jbenet/go-multiaddr" ) var WsProtocol = ma.Protocol{ diff --git a/p2p/transport/websocket/ws_test.go b/p2p/transport/websocket/ws_test.go index 678a4cbeb9..1667ac42b8 100644 --- a/p2p/transport/websocket/ws_test.go +++ b/p2p/transport/websocket/ws_test.go @@ -4,7 +4,7 @@ import ( "bytes" "testing" - ma "github.com/jbenet/go-multiaddr" + ma "github.com/multiformats/go-multiaddr" ) func TestMultiaddrParsing(t *testing.T) { From 9532d3f16a8a68ef7bb5c7f8edd083607df68e89 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 19:38:14 -0700 Subject: [PATCH 0168/3965] update deps --- p2p/transport/tcp/tcp.go | 6 +++--- p2p/transport/tcp/tcp_test.go | 3 +-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 4def4dfb00..ae069cce63 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -7,12 +7,12 @@ import ( "sync" "time" - lgbl "github.com/ipfs/go-libp2p-loggables" logging "github.com/ipfs/go-log" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" reuseport "github.com/jbenet/go-reuseport" + lgbl "github.com/libp2p/go-libp2p-loggables" tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" ) diff --git a/p2p/transport/tcp/tcp_test.go b/p2p/transport/tcp/tcp_test.go index 5e1e29087e..8eeb5139f3 100644 --- a/p2p/transport/tcp/tcp_test.go +++ b/p2p/transport/tcp/tcp_test.go @@ -3,9 +3,8 @@ package tcp import ( "testing" - ma "github.com/jbenet/go-multiaddr" - utils "github.com/libp2p/go-libp2p-transport/test" + ma "github.com/multiformats/go-multiaddr" ) func TestTcpTransport(t *testing.T) { From 05b9ca603912fee3020476d425142732d8b10d64 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 21:18:07 -0700 Subject: [PATCH 0169/3965] update deps --- p2p/net/swarm/dial_sync.go | 2 +- p2p/net/swarm/dial_sync_test.go | 2 +- p2p/net/swarm/dial_test.go | 8 ++++---- p2p/net/swarm/limiter.go | 4 ++-- p2p/net/swarm/limiter_test.go | 4 ++-- p2p/net/swarm/peers_test.go | 6 +++--- p2p/net/swarm/simul_test.go | 6 +++--- p2p/net/swarm/swarm.go | 8 ++++---- p2p/net/swarm/swarm_addr.go | 2 +- p2p/net/swarm/swarm_addr_test.go | 4 ++-- p2p/net/swarm/swarm_conn.go | 6 +++--- p2p/net/swarm/swarm_dial.go | 6 +++--- p2p/net/swarm/swarm_listen.go | 4 ++-- p2p/net/swarm/swarm_net.go | 6 +++--- p2p/net/swarm/swarm_notif_test.go | 4 ++-- p2p/net/swarm/swarm_test.go | 6 +++--- 16 files changed, 39 insertions(+), 39 deletions(-) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index a63c047ac8..75d3f0fbee 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -4,7 +4,7 @@ import ( "context" "sync" - peer "github.com/ipfs/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ) type DialFunc func(context.Context, peer.ID) (*Conn, error) diff --git a/p2p/net/swarm/dial_sync_test.go b/p2p/net/swarm/dial_sync_test.go index 61a69ce386..ca81a9c872 100644 --- a/p2p/net/swarm/dial_sync_test.go +++ b/p2p/net/swarm/dial_sync_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ) func getMockDialFunc() (DialFunc, func(), context.Context, <-chan struct{}) { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 8e7b48fc2b..dabb0bdb00 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -7,13 +7,13 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" - pstore "github.com/ipfs/go-libp2p-peerstore" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" addrutil "github.com/libp2p/go-addr-util" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" testutil "github.com/libp2p/go-testutil" ci "github.com/libp2p/go-testutil/ci" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) func closeSwarms(swarms []*Swarm) { diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index bf4bcf0f5c..5e96908e41 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -4,10 +4,10 @@ import ( "context" "sync" - peer "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" addrutil "github.com/libp2p/go-addr-util" iconn "github.com/libp2p/go-libp2p-interface-conn" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) type dialResult struct { diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 31ac8435df..646019d265 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -8,9 +8,9 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" iconn "github.com/libp2p/go-libp2p-interface-conn" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" mafmt "github.com/whyrusleeping/mafmt" ) diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 5eadf4e671..3259264ae8 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -4,9 +4,9 @@ import ( "testing" "context" - peer "github.com/ipfs/go-libp2p-peer" - pstore "github.com/ipfs/go-libp2p-peerstore" - ma "github.com/jbenet/go-multiaddr" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 3518f139e6..8935773296 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -7,10 +7,10 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" - pstore "github.com/ipfs/go-libp2p-peerstore" - ma "github.com/jbenet/go-multiaddr" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" ci "github.com/libp2p/go-testutil/ci" + ma "github.com/multiformats/go-multiaddr" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 77c7daff04..bb179bbc69 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -11,23 +11,23 @@ import ( "sync" "time" - ci "github.com/ipfs/go-libp2p-crypto" - peer "github.com/ipfs/go-libp2p-peer" - pstore "github.com/ipfs/go-libp2p-peerstore" logging "github.com/ipfs/go-log" - ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" pst "github.com/jbenet/go-stream-muxer" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" addrutil "github.com/libp2p/go-addr-util" conn "github.com/libp2p/go-libp2p-conn" + ci "github.com/libp2p/go-libp2p-crypto" metrics "github.com/libp2p/go-libp2p-metrics" mconn "github.com/libp2p/go-libp2p-metrics/conn" inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" transport "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" tcpt "github.com/libp2p/go-tcp-transport" + ma "github.com/multiformats/go-multiaddr" psmss "github.com/whyrusleeping/go-smux-multistream" spdy "github.com/whyrusleeping/go-smux-spdystream" yamux "github.com/whyrusleeping/go-smux-yamux" diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 0ab2c7ea1c..78a3f5351a 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -1,9 +1,9 @@ package swarm import ( - ma "github.com/jbenet/go-multiaddr" addrutil "github.com/libp2p/go-addr-util" iconn "github.com/libp2p/go-libp2p-interface-conn" + ma "github.com/multiformats/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 5d2225405a..011b706d81 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -4,11 +4,11 @@ import ( "context" "testing" - pstore "github.com/ipfs/go-libp2p-peerstore" - ma "github.com/jbenet/go-multiaddr" addrutil "github.com/libp2p/go-addr-util" metrics "github.com/libp2p/go-libp2p-metrics" + pstore "github.com/libp2p/go-libp2p-peerstore" testutil "github.com/libp2p/go-testutil" + ma "github.com/multiformats/go-multiaddr" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 8128a3e6ab..72609f0f32 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -4,12 +4,12 @@ import ( "context" "fmt" - ic "github.com/ipfs/go-libp2p-crypto" - peer "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" + ic "github.com/libp2p/go-libp2p-crypto" iconn "github.com/libp2p/go-libp2p-interface-conn" inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) // Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index bfe9f0382e..0f15301698 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -7,11 +7,11 @@ import ( "sync" "time" - lgbl "github.com/ipfs/go-libp2p-loggables" - peer "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" addrutil "github.com/libp2p/go-addr-util" iconn "github.com/libp2p/go-libp2p-interface-conn" + lgbl "github.com/libp2p/go-libp2p-loggables" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 756084abc7..d76d1e640e 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -4,14 +4,14 @@ import ( "context" "fmt" - lgbl "github.com/ipfs/go-libp2p-loggables" - ma "github.com/jbenet/go-multiaddr" ps "github.com/jbenet/go-peerstream" conn "github.com/libp2p/go-libp2p-conn" iconn "github.com/libp2p/go-libp2p-interface-conn" + lgbl "github.com/libp2p/go-libp2p-loggables" mconn "github.com/libp2p/go-libp2p-metrics/conn" inet "github.com/libp2p/go-libp2p-net" transport "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" ) func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 86755cf8be..90356a0fc7 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -4,12 +4,12 @@ import ( "context" "fmt" - peer "github.com/ipfs/go-libp2p-peer" - pstore "github.com/ipfs/go-libp2p-peerstore" - ma "github.com/jbenet/go-multiaddr" "github.com/jbenet/goprocess" metrics "github.com/libp2p/go-libp2p-metrics" inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 9e914b8218..d9e9df62a5 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -5,9 +5,9 @@ import ( "time" "context" - peer "github.com/ipfs/go-libp2p-peer" - ma "github.com/jbenet/go-multiaddr" inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func streamsSame(a, b inet.Stream) bool { diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index f0ed560cf2..9deb918f3c 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -10,12 +10,12 @@ import ( "testing" "time" - peer "github.com/ipfs/go-libp2p-peer" - pstore "github.com/ipfs/go-libp2p-peerstore" - ma "github.com/jbenet/go-multiaddr" metrics "github.com/libp2p/go-libp2p-metrics" inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" testutil "github.com/libp2p/go-testutil" + ma "github.com/multiformats/go-multiaddr" ) func EchoStreamHandler(stream inet.Stream) { From e7fa567007593345bb94c71c726e1381faed78ce Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 4 Oct 2016 21:18:07 -0700 Subject: [PATCH 0170/3965] update deps --- p2p/net/nat/mapping.go | 4 ++-- p2p/net/nat/nat.go | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/net/nat/mapping.go b/p2p/net/nat/mapping.go index b2aeb9c4d7..b03b002d00 100644 --- a/p2p/net/nat/mapping.go +++ b/p2p/net/nat/mapping.go @@ -5,9 +5,9 @@ import ( "sync" "time" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" "github.com/jbenet/goprocess" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) // Mapping represents a port mapping in a NAT. diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 3a4ae60b4d..602667a3fc 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -10,10 +10,10 @@ import ( nat "github.com/fd/go-nat" logging "github.com/ipfs/go-log" - ma "github.com/jbenet/go-multiaddr" - manet "github.com/jbenet/go-multiaddr-net" goprocess "github.com/jbenet/goprocess" periodic "github.com/jbenet/goprocess/periodic" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) var ( From ba533150b4fd7df41e7718e3f374af727623b924 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 5 Oct 2016 10:05:05 -0700 Subject: [PATCH 0171/3965] extract from go-libp2p --- p2p/net/swarm/swarm_net_test.go | 35 ++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index ebf3bb82a2..ff13934088 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -1,15 +1,36 @@ -package swarm_test +package swarm import ( + "context" "fmt" "testing" "time" - "context" inet "github.com/libp2p/go-libp2p-net" - testutil "github.com/libp2p/go-libp2p/p2p/test/util" + pstore "github.com/libp2p/go-libp2p-peerstore" + tu "github.com/libp2p/go-testutil" + ma "github.com/multiformats/go-multiaddr" ) +func GenSwarmNetwork(t *testing.T, ctx context.Context) *Network { + p := tu.RandPeerNetParamsOrFatal(t) + ps := pstore.NewPeerstore() + ps.AddPubKey(p.ID, p.PubKey) + ps.AddPrivKey(p.ID, p.PrivKey) + n, err := NewNetwork(ctx, []ma.Multiaddr{p.Addr}, p.ID, ps, nil) + if err != nil { + t.Fatal(err) + } + ps.AddAddrs(p.ID, n.ListenAddresses(), pstore.PermanentAddrTTL) + return n +} + +func DivulgeAddresses(a, b inet.Network) { + id := a.LocalPeer() + addrs := a.Peerstore().Addrs(id) + b.Peerstore().AddAddrs(id, addrs, pstore.PermanentAddrTTL) +} + // TestConnectednessCorrect starts a few networks, connects a few // and tests Connectedness value is correct. func TestConnectednessCorrect(t *testing.T) { @@ -18,13 +39,13 @@ func TestConnectednessCorrect(t *testing.T) { nets := make([]inet.Network, 4) for i := 0; i < 4; i++ { - nets[i] = testutil.GenSwarmNetwork(t, ctx) + nets[i] = GenSwarmNetwork(t, ctx) } // connect 0-1, 0-2, 0-3, 1-2, 2-3 dial := func(a, b inet.Network) { - testutil.DivulgeAddresses(b, a) + DivulgeAddresses(b, a) if _, err := a.DialPeer(ctx, b.LocalPeer()); err != nil { t.Fatalf("Failed to dial: %s", err) } @@ -109,11 +130,11 @@ func TestNetworkOpenStream(t *testing.T) { nets := make([]inet.Network, 4) for i := 0; i < 4; i++ { - nets[i] = testutil.GenSwarmNetwork(t, ctx) + nets[i] = GenSwarmNetwork(t, ctx) } dial := func(a, b inet.Network) { - testutil.DivulgeAddresses(b, a) + DivulgeAddresses(b, a) if _, err := a.DialPeer(ctx, b.LocalPeer()); err != nil { t.Fatalf("Failed to dial: %s", err) } From 1fc8868191454ee03a626380f7e2b1b2f4283eb6 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 18 Oct 2016 17:10:22 -0700 Subject: [PATCH 0172/3965] fix race conditions in dial_sync and limiter tests --- p2p/net/swarm/dial_sync.go | 1 - p2p/net/swarm/limiter_test.go | 10 +++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index 75d3f0fbee..48b77899f2 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -81,7 +81,6 @@ func (ds *DialSync) DialLock(ctx context.Context, p peer.ID) (*Conn, error) { ad.conn, ad.err = ds.dialFunc(ctx, p) close(ad.waitch) ad.cancel() - ad.waitch = nil // to ensure nobody tries reusing this }(ctx, p, actd) } diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 646019d265..cbb6afdbc6 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -5,6 +5,7 @@ import ( "fmt" "math/rand" "strconv" + "sync" "testing" "time" @@ -170,13 +171,17 @@ func TestFDLimiting(t *testing.T) { } func TestTokenRedistribution(t *testing.T) { + var lk sync.Mutex hangchs := make(map[peer.ID]chan struct{}) df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { if tcpPortOver(a, 10) { return (iconn.Conn)(nil), nil } - <-hangchs[p] + lk.Lock() + ch := hangchs[p] + lk.Unlock() + <-ch return nil, fmt.Errorf("test bad dial") } l := newDialLimiterWithParams(df, 8, 4) @@ -190,6 +195,9 @@ func TestTokenRedistribution(t *testing.T) { // take all fd limit tokens with hang dials for _, pid := range pids { hangchs[pid] = make(chan struct{}) + } + + for _, pid := range pids { tryDialAddrs(ctx, l, pid, bads, resch) } From c88fe55f11a4e3b6baa4b94927154a496f52d200 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 24 Oct 2016 17:42:27 -0700 Subject: [PATCH 0173/3965] expose errors in log messages --- p2p/net/swarm/swarm_dial.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 0f15301698..d32d156dff 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -266,7 +266,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { // try to get a connection to any addr connC, err := s.dialAddrs(ctx, p, remoteAddrChan) if err != nil { - logdial["error"] = err + logdial["error"] = err.Error() return nil, err } logdial["netconn"] = lgbl.NetConn(connC) @@ -275,7 +275,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { defer log.EventBegin(ctx, "swarmDialDoSetup", logdial, lgbl.NetConn(connC)).Done() swarmC, err := dialConnSetup(ctx, s, connC) if err != nil { - logdial["error"] = err + logdial["error"] = err.Error() connC.Close() // close the connection. didn't work out :( return nil, err } From b5de300ca4d4af199ea39522d5c2f50e89f630c8 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Wed, 26 Oct 2016 12:14:48 +0200 Subject: [PATCH 0174/3965] Add pskConn --- p2p/net/pnet/psk_conn.go | 91 +++++++++++++++++++++++++++++++++++ p2p/net/pnet/psk_conn_test.go | 79 ++++++++++++++++++++++++++++++ 2 files changed, 170 insertions(+) create mode 100644 p2p/net/pnet/psk_conn.go create mode 100644 p2p/net/pnet/psk_conn_test.go diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go new file mode 100644 index 0000000000..29f5200765 --- /dev/null +++ b/p2p/net/pnet/psk_conn.go @@ -0,0 +1,91 @@ +package conn + +import ( + "context" + "crypto/cipher" + "crypto/rand" + "errors" + + salsa20 "github.com/davidlazar/go-crypto/salsa20" + mpool "github.com/jbenet/go-msgio/mpool" + iconn "github.com/libp2p/go-libp2p-interface-conn" +) + +// we are using buffer pool as user needs their slice back +// so we can't do XOR cripter in place +var bufPool = mpool.ByteSlicePool + +type pskConn struct { + iconn.Conn + psk *[32]byte + + writeS20 cipher.Stream + readS20 cipher.Stream +} + +func (c *pskConn) Read(out []byte) (int, error) { + if c.readS20 == nil { + nonce := make([]byte, 24) + n, err := c.Conn.Read(nonce) + if err != nil { + return 0, err + } + if n != 24 { + return 0, errors.New("could not read full nonce") + } + c.readS20 = salsa20.New(c.psk, nonce) + } + + maxn := uint32(len(out)) + in := bufPool.Get(maxn).([]byte) // get buffer + defer bufPool.Put(maxn, in) // put the buffer back + + in = in[:maxn] + n, err := c.Conn.Read(in) + if err != nil { + return 0, err + } + + c.readS20.XORKeyStream(out[:n], in[:n]) + + return n, nil +} + +func (c *pskConn) Write(in []byte) (int, error) { + if c.writeS20 == nil { + nonce := make([]byte, 24) + _, err := rand.Read(nonce) + if err != nil { + return 0, err + } + _, err = c.Conn.Write(nonce) + if err != nil { + return 0, err + } + + c.writeS20 = salsa20.New(c.psk, nonce) + } + n := uint32(len(in)) + out := bufPool.Get(n).([]byte) // get buffer + defer bufPool.Put(n, out) // put the buffer back + + out = out[:n] + c.writeS20.XORKeyStream(out, in) + + return c.Conn.Write(out) +} + +var _ iconn.Conn = (*pskConn)(nil) + +func newPSKConn(ctx context.Context, psk *[32]byte, insecure iconn.Conn) (iconn.Conn, error) { + if insecure == nil { + return nil, errors.New("insecure is nil") + } + if psk == nil { + return nil, errors.New("pre-shread key is nil") + } + return &pskConn{ + Conn: insecure, + psk: psk, + }, nil +} diff --git a/p2p/net/pnet/psk_conn_test.go b/p2p/net/pnet/psk_conn_test.go new file mode 100644 index 0000000000..7c8877cc89 --- /dev/null +++ b/p2p/net/pnet/psk_conn_test.go @@ -0,0 +1,79 @@ +package conn + +import ( + "bytes" + "context" + "math/rand" + "testing" + + iconn "github.com/libp2p/go-libp2p-interface-conn" +) + +var testPSK = [32]byte{} // null bytes are as good test key as any other key + +func setupPSKConns(ctx context.Context, t *testing.T) (iconn.Conn, iconn.Conn) { + + conn1, conn2, _, _ := setupSingleConn(t, ctx) + psk1, err := newPSKConn(ctx, &testPSK, conn1) + if err != nil { + t.Fatal(err) + } + psk2, err := newPSKConn(ctx, &testPSK, conn2) + if err != nil { + t.Fatal(err) + } + return psk1, psk2 +} + +func TestPSKSimpelMessges(t *testing.T) { + ctx, cancel := context.WithCancel(context.TODO()) + defer cancel() + + psk1, psk2 := setupPSKConns(ctx, t) + msg1 := []byte("hello world") + out1 := make([]byte, len(msg1)) + + _, err := psk1.Write(msg1) + if err != nil { + t.Fatal(err) + } + n, err := psk2.Read(out1) + if err != nil { + t.Fatal(err) + } + if n != len(out1) { + t.Fatalf("expected to read %d bytes, read: %d", len(out1), n) + } + + if !bytes.Equal(msg1, out1) { + t.Fatalf("input and output are not the same") + } +} + +func TestPSKFragmentation(t *testing.T) { + ctx, cancel := context.WithCancel(context.TODO()) + defer cancel() + + psk1, psk2 := setupPSKConns(ctx, t) + + in := make([]byte, 1000) + _, err := rand.Read(in) + if err != nil { + t.Fatal(err) + } + + out := make([]byte, 100) + + _, err = psk1.Write(in) + if err != nil { + t.Fatal(err) + } + + for i := 0; i < 10; i++ { + _, err = psk2.Read(out) + if !bytes.Equal(in[:100], out) { + t.Fatalf("input and output are not the same") + } + in = in[100:] + } +} From 55d3a73febbbd43674d4ee68cfb708a8f5e1dc61 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Thu, 27 Oct 2016 18:08:36 +0200 Subject: [PATCH 0175/3965] Disable tests for now and start implementing protector --- p2p/net/pnet/entry.go | 11 +++++++++++ p2p/net/pnet/psk_conn.go | 2 +- p2p/net/pnet/psk_conn_test.go | 10 ++++++---- 3 files changed, 18 insertions(+), 5 deletions(-) create mode 100644 p2p/net/pnet/entry.go diff --git a/p2p/net/pnet/entry.go b/p2p/net/pnet/entry.go new file mode 100644 index 0000000000..e4785c8ad7 --- /dev/null +++ b/p2p/net/pnet/entry.go @@ -0,0 +1,11 @@ +package pnet + +import ( + "io" + + ipnet "github.com/libp2p/go-libp2p-interface-pnet" +) + +func NewProtector(input io.Reader) (ipnet.Protector, error) { + return nil, nil +} diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index 29f5200765..d1f3aa1d2f 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -1,4 +1,4 @@ -package conn +package pnet import ( "context" diff --git a/p2p/net/pnet/psk_conn_test.go b/p2p/net/pnet/psk_conn_test.go index 7c8877cc89..40cca34b43 100644 --- a/p2p/net/pnet/psk_conn_test.go +++ b/p2p/net/pnet/psk_conn_test.go @@ -1,4 +1,4 @@ -package conn +package pnet import ( "bytes" @@ -12,8 +12,10 @@ import ( var testPSK = [32]byte{} // null bytes are as good test key as any other key func setupPSKConns(ctx context.Context, t *testing.T) (iconn.Conn, iconn.Conn) { + return nil, nil - conn1, conn2, _, _ := setupSingleConn(t, ctx) + //conn1, conn2, _, _ := setupSingleConn(t, ctx) + var conn1, conn2 iconn.Conn psk1, err := newPSKConn(ctx, &testPSK, conn1) if err != nil { t.Fatal(err) @@ -25,7 +27,7 @@ func setupPSKConns(ctx context.Context, t *testing.T) (iconn.Conn, iconn.Conn) { return psk1, psk2 } -func TestPSKSimpelMessges(t *testing.T) { +func XTestPSKSimpelMessges(t *testing.T) { ctx, cancel := context.WithCancel(context.TODO()) defer cancel() @@ -50,7 +52,7 @@ func TestPSKSimpelMessges(t *testing.T) { } } -func TestPSKFragmentation(t *testing.T) { +func XTestPSKFragmentation(t *testing.T) { ctx, cancel := context.WithCancel(context.TODO()) defer cancel() From c6ffeff383b87192a3822a082f5654ab4b75fcfa Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Fri, 28 Oct 2016 01:04:16 +0200 Subject: [PATCH 0176/3965] Start implementing codecs --- p2p/net/pnet/codec.go | 5 +++++ p2p/net/pnet/entry.go | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 p2p/net/pnet/codec.go diff --git a/p2p/net/pnet/codec.go b/p2p/net/pnet/codec.go new file mode 100644 index 0000000000..96c34b6e7a --- /dev/null +++ b/p2p/net/pnet/codec.go @@ -0,0 +1,5 @@ +package pnet + +var ( + pathPSKv1 = "/key/swarm/psk/1.0.0/" +) diff --git a/p2p/net/pnet/entry.go b/p2p/net/pnet/entry.go index e4785c8ad7..504b1a0476 100644 --- a/p2p/net/pnet/entry.go +++ b/p2p/net/pnet/entry.go @@ -4,8 +4,13 @@ import ( "io" ipnet "github.com/libp2p/go-libp2p-interface-pnet" + mc "github.com/multiformats/go-multicodec" + bmux "github.com/multiformats/go-multicodec/base/mux" ) func NewProtector(input io.Reader) (ipnet.Protector, error) { + input = mc.WrapTransformPathToHeader(input) + _ = bmux.AllBasesMux() + return nil, nil } From 3d3f679ee95e47cf86a309b78350938f6a1ba149 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Fri, 28 Oct 2016 17:30:42 +0200 Subject: [PATCH 0177/3965] Implement protector --- p2p/net/pnet/codec.go | 32 +++++++++++++++++++++++++++++++- p2p/net/pnet/entry.go | 11 +++++------ p2p/net/pnet/protector.go | 16 ++++++++++++++++ p2p/net/pnet/psk_conn.go | 3 +-- 4 files changed, 53 insertions(+), 9 deletions(-) create mode 100644 p2p/net/pnet/protector.go diff --git a/p2p/net/pnet/codec.go b/p2p/net/pnet/codec.go index 96c34b6e7a..8bf985ce26 100644 --- a/p2p/net/pnet/codec.go +++ b/p2p/net/pnet/codec.go @@ -1,5 +1,35 @@ package pnet +import ( + "io" + + mc "github.com/multiformats/go-multicodec" + bmux "github.com/multiformats/go-multicodec/base/mux" +) + var ( - pathPSKv1 = "/key/swarm/psk/1.0.0/" + pathPSKv1 = []byte("/key/swarm/psk/1.0.0/") + headerPSKv1 = mc.Header(pathPSKv1) ) + +func decodeV1PSKKey(in io.Reader) (*[32]byte, error) { + var err error + in, err = mc.WrapTransformPathToHeader(in) + if err != nil { + return nil, err + } + err = mc.ConsumeHeader(in, headerPSKv1) + if err != nil { + return nil, err + } + + in, err = mc.WrapTransformPathToHeader(in) + if err != nil { + return nil, err + } + + out := [32]byte{} + + err = bmux.AllBasesMux().Decoder(in).Decode(out) + return &out, err +} diff --git a/p2p/net/pnet/entry.go b/p2p/net/pnet/entry.go index 504b1a0476..e0cde4d550 100644 --- a/p2p/net/pnet/entry.go +++ b/p2p/net/pnet/entry.go @@ -4,13 +4,12 @@ import ( "io" ipnet "github.com/libp2p/go-libp2p-interface-pnet" - mc "github.com/multiformats/go-multicodec" - bmux "github.com/multiformats/go-multicodec/base/mux" ) func NewProtector(input io.Reader) (ipnet.Protector, error) { - input = mc.WrapTransformPathToHeader(input) - _ = bmux.AllBasesMux() - - return nil, nil + psk, err := decodeV1PSKKey(input) + if err != nil { + return nil, err + } + return &protector{psk}, nil } diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go new file mode 100644 index 0000000000..2759eefc27 --- /dev/null +++ b/p2p/net/pnet/protector.go @@ -0,0 +1,16 @@ +package pnet + +import ( + iconn "github.com/libp2p/go-libp2p-interface-conn" + ipnet "github.com/libp2p/go-libp2p-interface-pnet" +) + +type protector struct { + psk *[32]byte +} + +var _ ipnet.Protector = (*protector)(nil) + +func (p protector) Protect(in iconn.Conn) (iconn.Conn, error) { + return newPSKConn(p.psk, in) +} diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index d1f3aa1d2f..a96e277c21 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -1,7 +1,6 @@ package pnet import ( - "context" "crypto/cipher" "crypto/rand" "errors" @@ -77,7 +76,7 @@ func (c *pskConn) Write(in []byte) (int, error) { var _ iconn.Conn = (*pskConn)(nil) -func newPSKConn(ctx context.Context, psk *[32]byte, insecure iconn.Conn) (iconn.Conn, error) { +func newPSKConn(psk *[32]byte, insecure iconn.Conn) (iconn.Conn, error) { if insecure == nil { return nil, errors.New("insecure is nil") } From 3b5d44bb21262b5dd85b717f03e1442b2e444df4 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 28 Oct 2016 14:06:31 -0700 Subject: [PATCH 0178/3965] make sure to always take locks in correct order --- p2p/net/swarm/dial_sync.go | 42 +++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index 48b77899f2..8ef5183755 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -36,6 +36,7 @@ type activeDial struct { } func (dr *activeDial) wait(ctx context.Context) (*Conn, error) { + dr.incref() defer dr.decref() select { case <-dr.waitch: @@ -53,22 +54,38 @@ func (ad *activeDial) incref() { func (ad *activeDial) decref() { ad.refCntLk.Lock() - defer ad.refCntLk.Unlock() ad.refCnt-- - if ad.refCnt <= 0 { - ad.cancel() + maybeZero := (ad.refCnt <= 0) + ad.refCntLk.Unlock() + + // make sure to always take locks in correct order. + if maybeZero { ad.ds.dialsLk.Lock() - delete(ad.ds.dials, ad.id) + ad.refCntLk.Lock() + // check again after lock swap drop to make sure nobody else called incref + // in between locks + if ad.refCnt <= 0 { + ad.cancel() + delete(ad.ds.dials, ad.id) + } ad.ds.dialsLk.Unlock() + ad.refCntLk.Unlock() } } -func (ds *DialSync) DialLock(ctx context.Context, p peer.ID) (*Conn, error) { +func (ad *activeDial) start(ctx context.Context) { + ad.conn, ad.err = ad.ds.dialFunc(ctx, ad.id) + close(ad.waitch) + ad.cancel() +} + +func (ds *DialSync) getActiveDial(p peer.ID) *activeDial { ds.dialsLk.Lock() + defer ds.dialsLk.Unlock() actd, ok := ds.dials[p] if !ok { - ctx, cancel := context.WithCancel(context.Background()) + adctx, cancel := context.WithCancel(context.Background()) actd = &activeDial{ id: p, cancel: cancel, @@ -77,15 +94,12 @@ func (ds *DialSync) DialLock(ctx context.Context, p peer.ID) (*Conn, error) { } ds.dials[p] = actd - go func(ctx context.Context, p peer.ID, ad *activeDial) { - ad.conn, ad.err = ds.dialFunc(ctx, p) - close(ad.waitch) - ad.cancel() - }(ctx, p, actd) + go actd.start(adctx) } - actd.incref() - ds.dialsLk.Unlock() + return actd +} - return actd.wait(ctx) +func (ds *DialSync) DialLock(ctx context.Context, p peer.ID) (*Conn, error) { + return ds.getActiveDial(p).wait(ctx) } From 9bd9701f20c20c96119e23de7267042e0f61f031 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 28 Oct 2016 14:14:44 -0700 Subject: [PATCH 0179/3965] add a test to stress the lock order race condition --- p2p/net/swarm/dial_sync_test.go | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/p2p/net/swarm/dial_sync_test.go b/p2p/net/swarm/dial_sync_test.go index ca81a9c872..0d70e226af 100644 --- a/p2p/net/swarm/dial_sync_test.go +++ b/p2p/net/swarm/dial_sync_test.go @@ -201,3 +201,27 @@ func TestFailFirst(t *testing.T) { t.Fatal("should have gotten a 'real' conn back") } } + +func TestStressActiveDial(t *testing.T) { + ds := NewDialSync(func(ctx context.Context, p peer.ID) (*Conn, error) { + return nil, nil + }) + + wg := sync.WaitGroup{} + + pid := peer.ID("foo") + + makeDials := func() { + for i := 0; i < 10000; i++ { + ds.DialLock(context.Background(), pid) + } + wg.Done() + } + + for i := 0; i < 100; i++ { + wg.Add(1) + go makeDials() + } + + wg.Wait() +} From 0cd1fc7907ed83cb22ee3afbf86e48e9af6c49f9 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 28 Oct 2016 16:43:20 -0700 Subject: [PATCH 0180/3965] nitpick: lock release order --- p2p/net/swarm/dial_sync.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index 8ef5183755..734901fc15 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -68,8 +68,8 @@ func (ad *activeDial) decref() { ad.cancel() delete(ad.ds.dials, ad.id) } - ad.ds.dialsLk.Unlock() ad.refCntLk.Unlock() + ad.ds.dialsLk.Unlock() } } From 034bcab5e6761da4a5204d433b92d9883de22a3b Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 28 Oct 2016 17:09:45 -0700 Subject: [PATCH 0181/3965] increase refcount before dropping dialsLk initially --- p2p/net/swarm/dial_sync.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index 734901fc15..69739e54ec 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -36,7 +36,6 @@ type activeDial struct { } func (dr *activeDial) wait(ctx context.Context) (*Conn, error) { - dr.incref() defer dr.decref() select { case <-dr.waitch: @@ -97,6 +96,9 @@ func (ds *DialSync) getActiveDial(p peer.ID) *activeDial { go actd.start(adctx) } + // increase ref count before dropping dialsLk + actd.incref() + return actd } From 95707866880290775cf91d59dbd7cba0baec2a63 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Wed, 2 Nov 2016 17:37:03 +0100 Subject: [PATCH 0182/3965] Add decoding tests, make conn tests compile --- p2p/net/pnet/codec.go | 8 ++--- p2p/net/pnet/codec_test.go | 61 +++++++++++++++++++++++++++++++++++ p2p/net/pnet/psk_conn_test.go | 4 +-- 3 files changed, 67 insertions(+), 6 deletions(-) create mode 100644 p2p/net/pnet/codec_test.go diff --git a/p2p/net/pnet/codec.go b/p2p/net/pnet/codec.go index 8bf985ce26..c013abbc60 100644 --- a/p2p/net/pnet/codec.go +++ b/p2p/net/pnet/codec.go @@ -1,6 +1,7 @@ package pnet import ( + "fmt" "io" mc "github.com/multiformats/go-multicodec" @@ -20,16 +21,15 @@ func decodeV1PSKKey(in io.Reader) (*[32]byte, error) { } err = mc.ConsumeHeader(in, headerPSKv1) if err != nil { - return nil, err + return nil, fmt.Errorf("psk header error: %s", err.Error()) } in, err = mc.WrapTransformPathToHeader(in) if err != nil { - return nil, err + return nil, fmt.Errorf("wrapping error: %s", err.Error()) } - out := [32]byte{} - err = bmux.AllBasesMux().Decoder(in).Decode(out) + err = bmux.AllBasesMux().Decoder(in).Decode(out[:]) return &out, err } diff --git a/p2p/net/pnet/codec_test.go b/p2p/net/pnet/codec_test.go new file mode 100644 index 0000000000..11d912b62c --- /dev/null +++ b/p2p/net/pnet/codec_test.go @@ -0,0 +1,61 @@ +package pnet + +import ( + "bytes" + "encoding/base64" + "testing" +) + +func bufWithBase(base string) *bytes.Buffer { + + b := &bytes.Buffer{} + b.Write(pathPSKv1) + b.WriteString("\n") + b.WriteString(base) + b.WriteString("\n") + return b +} + +func TestDecodeHex(t *testing.T) { + b := bufWithBase("/base16/") + for i := 0; i < 32; i++ { + b.WriteString("FF") + } + + psk, err := decodeV1PSKKey(b) + if err != nil { + t.Fatal(err) + } + + for _, b := range psk { + if b != 255 { + t.Fatal("byte was wrong") + } + } +} + +func TestDecodeB64(t *testing.T) { + b := bufWithBase("/base64/") + key := make([]byte, 32) + for i := 0; i < 32; i++ { + key[i] = byte(i) + } + + e := base64.NewEncoder(base64.StdEncoding, b) + _, err := e.Write(key) + if err != nil { + t.Fatal(err) + } + + psk, err := decodeV1PSKKey(b) + if err != nil { + t.Fatal(err) + } + + for i, b := range psk { + if b != psk[i] { + t.Fatal("byte was wrong") + } + } + +} diff --git a/p2p/net/pnet/psk_conn_test.go b/p2p/net/pnet/psk_conn_test.go index 40cca34b43..d142fa3835 100644 --- a/p2p/net/pnet/psk_conn_test.go +++ b/p2p/net/pnet/psk_conn_test.go @@ -16,11 +16,11 @@ func setupPSKConns(ctx context.Context, t *testing.T) (iconn.Conn, iconn.Conn) { //conn1, conn2, _, _ := setupSingleConn(t, ctx) var conn1, conn2 iconn.Conn - psk1, err := newPSKConn(ctx, &testPSK, conn1) + psk1, err := newPSKConn(&testPSK, conn1) if err != nil { t.Fatal(err) } - psk2, err := newPSKConn(ctx, &testPSK, conn2) + psk2, err := newPSKConn(&testPSK, conn2) if err != nil { t.Fatal(err) } From 4edf3cfa4c6dc9e4f3effb748d797471493ef132 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 2 Nov 2016 23:35:03 -0700 Subject: [PATCH 0183/3965] move protocol methods down into peerstream --- p2p/net/swarm/swarm.go | 13 ++++--------- p2p/net/swarm/swarm_conn.go | 6 +++++- p2p/net/swarm/swarm_stream.go | 33 ++++++++------------------------- 3 files changed, 17 insertions(+), 35 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index bb179bbc69..ec156d4e4b 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -256,7 +256,7 @@ func (s *Swarm) SetConnHandler(handler ConnHandler) { // See peerstream. func (s *Swarm) SetStreamHandler(handler inet.StreamHandler) { s.swarm.SetStreamHandler(func(s *ps.Stream) { - handler(wrapStream(s)) + handler((*Stream)(s)) }) } @@ -273,12 +273,7 @@ func (s *Swarm) NewStreamWithPeer(ctx context.Context, p peer.ID) (*Stream, erro // TODO: think about passing a context down to NewStreamWithGroup st, err := s.swarm.NewStreamWithGroup(p) - return wrapStream(st), err -} - -// StreamsWithPeer returns all the live Streams to p -func (s *Swarm) StreamsWithPeer(p peer.ID) []*Stream { - return wrapStreams(ps.StreamsWithGroup(p, s.swarm.Streams())) + return (*Stream)(st), err } // ConnectionsToPeer returns all the live connections to p @@ -387,9 +382,9 @@ func (n *ps2netNotifee) Disconnected(c *ps.Conn) { } func (n *ps2netNotifee) OpenedStream(s *ps.Stream) { - n.not.OpenedStream(n.net, &Stream{stream: s}) + n.not.OpenedStream(n.net, (*Stream)(s)) } func (n *ps2netNotifee) ClosedStream(s *ps.Stream) { - n.not.ClosedStream(n.net, &Stream{stream: s}) + n.not.ClosedStream(n.net, (*Stream)(s)) } diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 72609f0f32..bc5b69e4ad 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -77,7 +77,7 @@ func (c *Conn) RemotePublicKey() ic.PubKey { // NewSwarmStream returns a new Stream from this connection func (c *Conn) NewSwarmStream() (*Stream, error) { s, err := c.StreamConn().NewStream() - return wrapStream(s), err + return (*Stream)(s), err } // NewStream returns a new Stream from this connection @@ -91,6 +91,10 @@ func (c *Conn) Close() error { return c.StreamConn().Close() } +func (c *Conn) GetStreams() ([]inet.Stream, error) { + return nil, fmt.Errorf("GetStreams() not yet implemented") +} + func wrapConn(psc *ps.Conn) (*Conn, error) { // grab the underlying connection. if _, ok := psc.NetConn().(iconn.Conn); !ok { diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index ef3ab3c8fd..e3b12563bb 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -9,14 +9,11 @@ import ( // Stream is a wrapper around a ps.Stream that exposes a way to get // our Conn and Swarm (instead of just the ps.Conn and ps.Swarm) -type Stream struct { - stream *ps.Stream - protocol protocol.ID -} +type Stream ps.Stream // Stream returns the underlying peerstream.Stream func (s *Stream) Stream() *ps.Stream { - return s.stream + return (*ps.Stream)(s) } // Conn returns the Conn associated with this Stream, as an inet.Conn @@ -26,43 +23,29 @@ func (s *Stream) Conn() inet.Conn { // SwarmConn returns the Conn associated with this Stream, as a *Conn func (s *Stream) SwarmConn() *Conn { - return (*Conn)(s.stream.Conn()) + return (*Conn)(s.Stream().Conn()) } // Read reads bytes from a stream. func (s *Stream) Read(p []byte) (n int, err error) { - return s.stream.Read(p) + return s.Stream().Read(p) } // Write writes bytes to a stream, flushing for each call. func (s *Stream) Write(p []byte) (n int, err error) { - return s.stream.Write(p) + return s.Stream().Write(p) } // Close closes the stream, indicating this side is finished // with the stream. func (s *Stream) Close() error { - return s.stream.Close() + return s.Stream().Close() } func (s *Stream) Protocol() protocol.ID { - return s.protocol + return (*ps.Stream)(s).Protocol() } func (s *Stream) SetProtocol(p protocol.ID) { - s.protocol = p -} - -func wrapStream(pss *ps.Stream) *Stream { - return &Stream{ - stream: pss, - } -} - -func wrapStreams(st []*ps.Stream) []*Stream { - out := make([]*Stream, len(st)) - for i, s := range st { - out[i] = wrapStream(s) - } - return out + (*ps.Stream)(s).SetProtocol(p) } From 0ac5cdf74458835bbcc18d9bfb703666f912bd1c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 3 Nov 2016 13:15:48 -0700 Subject: [PATCH 0184/3965] bubble up peerstream modifications --- p2p/net/swarm/swarm.go | 2 +- p2p/net/swarm/swarm_conn.go | 2 +- p2p/net/swarm/swarm_listen.go | 2 +- p2p/net/swarm/swarm_stream.go | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index ec156d4e4b..3a59517c34 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -12,7 +12,6 @@ import ( "time" logging "github.com/ipfs/go-log" - ps "github.com/jbenet/go-peerstream" pst "github.com/jbenet/go-stream-muxer" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" @@ -26,6 +25,7 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" transport "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" + ps "github.com/libp2p/go-peerstream" tcpt "github.com/libp2p/go-tcp-transport" ma "github.com/multiformats/go-multiaddr" psmss "github.com/whyrusleeping/go-smux-multistream" diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index bc5b69e4ad..4b13be54f1 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -4,11 +4,11 @@ import ( "context" "fmt" - ps "github.com/jbenet/go-peerstream" ic "github.com/libp2p/go-libp2p-crypto" iconn "github.com/libp2p/go-libp2p-interface-conn" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" + ps "github.com/libp2p/go-peerstream" ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index d76d1e640e..571a305e3a 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -4,13 +4,13 @@ import ( "context" "fmt" - ps "github.com/jbenet/go-peerstream" conn "github.com/libp2p/go-libp2p-conn" iconn "github.com/libp2p/go-libp2p-interface-conn" lgbl "github.com/libp2p/go-libp2p-loggables" mconn "github.com/libp2p/go-libp2p-metrics/conn" inet "github.com/libp2p/go-libp2p-net" transport "github.com/libp2p/go-libp2p-transport" + ps "github.com/libp2p/go-peerstream" ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index e3b12563bb..771038621f 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -4,7 +4,7 @@ import ( inet "github.com/libp2p/go-libp2p-net" protocol "github.com/libp2p/go-libp2p-protocol" - ps "github.com/jbenet/go-peerstream" + ps "github.com/libp2p/go-peerstream" ) // Stream is a wrapper around a ps.Stream that exposes a way to get From 23bb096d3ebaf22a7490507d73b76824a0c570c8 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 3 Nov 2016 14:01:18 -0700 Subject: [PATCH 0185/3965] implement GetStreams --- p2p/net/swarm/swarm_conn.go | 8 +++++++- p2p/net/swarm/swarm_net_test.go | 9 +++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 4b13be54f1..0b0c626cea 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -92,7 +92,13 @@ func (c *Conn) Close() error { } func (c *Conn) GetStreams() ([]inet.Stream, error) { - return nil, fmt.Errorf("GetStreams() not yet implemented") + ss := c.StreamConn().Streams() + out := make([]inet.Stream, len(ss)) + + for i, s := range ss { + out[i] = (*Stream)(s) + } + return out, nil } func wrapConn(psc *ps.Conn) (*Conn, error) { diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index ff13934088..95bc268307 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -165,6 +165,15 @@ func TestNetworkOpenStream(t *testing.T) { t.Fatal(err) } + streams, err := nets[0].ConnsToPeer(nets[1].LocalPeer())[0].GetStreams() + if err != nil { + t.Fatal(err) + } + + if len(streams) != 1 { + t.Fatal("should only have one stream there") + } + _, err = s.Write([]byte("hello ipfs")) if err != nil { t.Fatal(err) From 7d51c0fbb42cfb8c700d1ba6e33d847e1652885d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sat, 5 Nov 2016 15:55:49 -0700 Subject: [PATCH 0186/3965] add timeout for negotiate --- p2p/muxer/muxer-multistream/multistream.go | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index fdb76f02f6..87ac7f7c6c 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -5,24 +5,30 @@ package multistream import ( "fmt" "net" + "time" mss "github.com/whyrusleeping/go-multistream" smux "github.com/jbenet/go-stream-muxer" ) +var DefaultNegotiateTimeout = time.Second * 60 + type Transport struct { mux *mss.MultistreamMuxer tpts map[string]smux.Transport + NegotiateTimeout time.Duration + OrderPreference []string } func NewBlankTransport() *Transport { return &Transport{ - mux: mss.NewMultistreamMuxer(), - tpts: make(map[string]smux.Transport), + mux: mss.NewMultistreamMuxer(), + tpts: make(map[string]smux.Transport), + NegotiateTimeout: DefaultNegotiateTimeout, } } @@ -33,6 +39,12 @@ func (t *Transport) AddTransport(path string, tpt smux.Transport) { } func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { + if t.NegotiateTimeout != 0 { + if err := nc.SetDeadline(time.Now().Add(t.NegotiateTimeout)); err != nil { + return nil, err + } + } + var proto string if isServer { selected, _, err := t.mux.Negotiate(nc) @@ -48,6 +60,12 @@ func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { proto = selected } + if t.NegotiateTimeout != 0 { + if err := nc.SetDeadline(time.Time{}); err != nil { + return nil, err + } + } + tpt, ok := t.tpts[proto] if !ok { return nil, fmt.Errorf("selected protocol we don't have a transport for") From 2fc370b3884ac0ac5a198b4c9e94966719eb0726 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 6 Nov 2016 21:39:54 -0800 Subject: [PATCH 0187/3965] add deadline methods to stream --- p2p/muxer/yamux/yamux.go | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index a6abb6f043..6f62aa8ac2 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -10,6 +10,7 @@ import ( ) // stream implements smux.Stream using a ss.Stream +// TODO: do we actually need a wrapper here? type stream yamux.Stream func (s *stream) yamuxStream() *yamux.Stream { @@ -28,6 +29,18 @@ func (s *stream) Close() error { return s.yamuxStream().Close() } +func (s *stream) SetDeadline(t time.Time) error { + return s.yamuxStream().SetDeadline(t) +} + +func (s *stream) SetReadDeadline(t time.Time) error { + return s.yamuxStream().SetReadDeadline(t) +} + +func (s *stream) SetWriteDeadline(t time.Time) error { + return s.yamuxStream().SetWriteDeadline(t) +} + // Conn is a connection to a remote peer. type conn yamux.Session From 23d124341e362782372be72b6c0f724f3544e6bc Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 7 Nov 2016 17:35:13 +0100 Subject: [PATCH 0188/3965] Enable tests back again using dummy-conn --- p2p/net/pnet/psk_conn_test.go | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/p2p/net/pnet/psk_conn_test.go b/p2p/net/pnet/psk_conn_test.go index d142fa3835..f81b55cf48 100644 --- a/p2p/net/pnet/psk_conn_test.go +++ b/p2p/net/pnet/psk_conn_test.go @@ -6,16 +6,18 @@ import ( "math/rand" "testing" + dconn "github.com/Kubuxu/go-libp2p-dummy-conn" iconn "github.com/libp2p/go-libp2p-interface-conn" ) var testPSK = [32]byte{} // null bytes are as good test key as any other key func setupPSKConns(ctx context.Context, t *testing.T) (iconn.Conn, iconn.Conn) { - return nil, nil + conn1, conn2, err := dconn.NewDummyConnPair() + if err != nil { + t.Fatal(err) + } - //conn1, conn2, _, _ := setupSingleConn(t, ctx) - var conn1, conn2 iconn.Conn psk1, err := newPSKConn(&testPSK, conn1) if err != nil { t.Fatal(err) @@ -27,7 +29,7 @@ func setupPSKConns(ctx context.Context, t *testing.T) (iconn.Conn, iconn.Conn) { return psk1, psk2 } -func XTestPSKSimpelMessges(t *testing.T) { +func TestPSKSimpelMessges(t *testing.T) { ctx, cancel := context.WithCancel(context.TODO()) defer cancel() @@ -52,7 +54,7 @@ func XTestPSKSimpelMessges(t *testing.T) { } } -func XTestPSKFragmentation(t *testing.T) { +func TestPSKFragmentation(t *testing.T) { ctx, cancel := context.WithCancel(context.TODO()) defer cancel() From 02a743fd6f2faa3852b9e7bcdf2d65698360721e Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 8 Nov 2016 10:37:06 -0800 Subject: [PATCH 0189/3965] add deadline methods to stream --- p2p/net/swarm/swarm_stream.go | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 771038621f..adadca78ec 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -1,9 +1,10 @@ package swarm import ( + "time" + inet "github.com/libp2p/go-libp2p-net" protocol "github.com/libp2p/go-libp2p-protocol" - ps "github.com/libp2p/go-peerstream" ) @@ -49,3 +50,15 @@ func (s *Stream) Protocol() protocol.ID { func (s *Stream) SetProtocol(p protocol.ID) { (*ps.Stream)(s).SetProtocol(p) } + +func (s *Stream) SetDeadline(t time.Time) error { + return s.Stream().SetDeadline(t) +} + +func (s *Stream) SetReadDeadline(t time.Time) error { + return s.Stream().SetReadDeadline(t) +} + +func (s *Stream) SetWriteDeadline(t time.Time) error { + return s.Stream().SetWriteDeadline(t) +} From 82fb7d1a18c7b55f955d33066ca2a759ea378005 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Wed, 9 Nov 2016 12:04:55 +0100 Subject: [PATCH 0190/3965] Add Private Network protector to swarm --- p2p/net/swarm/swarm.go | 12 ++++++++---- p2p/net/swarm/swarm_listen.go | 2 +- p2p/net/swarm/swarm_net.go | 5 +++-- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 3a59517c34..fe03a8e111 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -18,6 +18,7 @@ import ( addrutil "github.com/libp2p/go-addr-util" conn "github.com/libp2p/go-libp2p-conn" ci "github.com/libp2p/go-libp2p-crypto" + ipnet "github.com/libp2p/go-libp2p-interface-pnet" metrics "github.com/libp2p/go-libp2p-metrics" mconn "github.com/libp2p/go-libp2p-metrics/conn" inet "github.com/libp2p/go-libp2p-net" @@ -98,11 +99,13 @@ type Swarm struct { bwc metrics.Reporter limiter *dialLimiter + + protec ipnet.Protector } // NewSwarm constructs a Swarm, with a Chan. -func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, - local peer.ID, peers pstore.Peerstore, bwc metrics.Reporter) (*Swarm, error) { +func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, local peer.ID, + peers pstore.Peerstore, protec ipnet.Protector, bwc metrics.Reporter) (*Swarm, error) { listenAddrs, err := filterAddrs(listenAddrs) if err != nil { @@ -130,7 +133,8 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, bwc: bwc, fdRateLimit: make(chan struct{}, concurrentFdDials), Filters: filter.NewFilters(), - dialer: conn.NewDialer(local, peers.PrivKey(local), wrap), + dialer: conn.NewDialer(local, peers.PrivKey(local), protec, wrap), + protec: protec, } s.dsync = NewDialSync(s.doDial) @@ -158,7 +162,7 @@ func NewBlankSwarm(ctx context.Context, id peer.ID, privkey ci.PrivKey, pstpt ps notifs: make(map[inet.Notifiee]ps.Notifiee), fdRateLimit: make(chan struct{}, concurrentFdDials), Filters: filter.NewFilters(), - dialer: conn.NewDialer(id, privkey, nil), + dialer: conn.NewDialer(id, privkey, nil, nil), } // configure Swarm diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 571a305e3a..edb075c18e 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -83,7 +83,7 @@ func (s *Swarm) addListener(tptlist transport.Listener) error { log.Warning("Listener not given PrivateKey, so WILL NOT SECURE conns.") } - list, err := conn.WrapTransportListener(s.Context(), tptlist, s.local, sk) + list, err := conn.WrapTransportListener(s.Context(), tptlist, s.local, sk, s.protec) if err != nil { return err } diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 90356a0fc7..d522f87262 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -5,6 +5,7 @@ import ( "fmt" "github.com/jbenet/goprocess" + ipnet "github.com/libp2p/go-libp2p-interface-pnet" metrics "github.com/libp2p/go-libp2p-metrics" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" @@ -19,9 +20,9 @@ type Network Swarm // NewNetwork constructs a new network and starts listening on given addresses. func NewNetwork(ctx context.Context, listen []ma.Multiaddr, local peer.ID, - peers pstore.Peerstore, bwc metrics.Reporter) (*Network, error) { + peers pstore.Peerstore, protec ipnet.Protector, bwc metrics.Reporter) (*Network, error) { - s, err := NewSwarm(ctx, listen, local, peers, bwc) + s, err := NewSwarm(ctx, listen, local, peers, protec, bwc) if err != nil { return nil, err } From 1128a7c6249e54f53c7efb48e4334892d3155cf9 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Wed, 9 Nov 2016 20:55:29 +0100 Subject: [PATCH 0191/3965] Move from reader base ctor to bytestring based one --- p2p/net/pnet/entry.go | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/p2p/net/pnet/entry.go b/p2p/net/pnet/entry.go index e0cde4d550..9add1b0187 100644 --- a/p2p/net/pnet/entry.go +++ b/p2p/net/pnet/entry.go @@ -1,13 +1,15 @@ package pnet import ( - "io" + "bytes" ipnet "github.com/libp2p/go-libp2p-interface-pnet" ) -func NewProtector(input io.Reader) (ipnet.Protector, error) { - psk, err := decodeV1PSKKey(input) +func NewProtector(key []byte) (ipnet.Protector, error) { + reader := bytes.NewReader(key) + + psk, err := decodeV1PSKKey(reader) if err != nil { return nil, err } From b4dc58aa3be1c8fcad51cd53f2ebab67a13d8ef3 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 10 Nov 2016 16:00:57 -0800 Subject: [PATCH 0192/3965] log addr with failures --- p2p/net/swarm/limiter.go | 3 ++- p2p/net/swarm/swarm_dial.go | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 5e96908e41..67c6af4ddf 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -12,6 +12,7 @@ import ( type dialResult struct { Conn iconn.Conn + Addr ma.Multiaddr Err error } @@ -141,7 +142,7 @@ func (dl *dialLimiter) executeDial(j *dialJob) { con, err := dl.dialFunc(j.ctx, j.peer, j.addr) select { - case j.resp <- dialResult{Conn: con, Err: err}: + case j.resp <- dialResult{Conn: con, Addr: j.addr, Err: err}: case <-j.ctx.Done(): } } diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index d32d156dff..c7cfe91110 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -318,7 +318,7 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. case resp := <-respch: active-- if resp.Err != nil { - log.Info("got error on dial: ", resp.Err) + log.Info("got error on dial to %s: ", resp.Addr, resp.Err) // Errors are normal, lots of dials will fail exitErr = resp.Err From 8dcdc4f994f4cab63f91a3e4dfd07a32cab66798 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Fri, 11 Nov 2016 21:34:55 +0100 Subject: [PATCH 0193/3965] Update for changed libp2p-conn --- p2p/net/swarm/swarm.go | 12 +++++++++--- p2p/net/swarm/swarm_addr_test.go | 4 ++-- p2p/net/swarm/swarm_listen.go | 2 +- p2p/net/swarm/swarm_net.go | 2 +- p2p/net/swarm/swarm_net_test.go | 2 +- 5 files changed, 14 insertions(+), 8 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index fe03a8e111..b1878ff0e2 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -103,8 +103,13 @@ type Swarm struct { protec ipnet.Protector } -// NewSwarm constructs a Swarm, with a Chan. func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, local peer.ID, + peers pstore.Peerstore, bwc metrics.Reporter) (*Swarm, error) { + return NewSwarmWithProtector(ctx, listenAddrs, local, peers, nil, bwc) +} + +// NewSwarm constructs a Swarm, with a Chan. +func NewSwarmWithProtector(ctx context.Context, listenAddrs []ma.Multiaddr, local peer.ID, peers pstore.Peerstore, protec ipnet.Protector, bwc metrics.Reporter) (*Swarm, error) { listenAddrs, err := filterAddrs(listenAddrs) @@ -133,9 +138,10 @@ func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, local peer.ID, bwc: bwc, fdRateLimit: make(chan struct{}, concurrentFdDials), Filters: filter.NewFilters(), - dialer: conn.NewDialer(local, peers.PrivKey(local), protec, wrap), + dialer: conn.NewDialer(local, peers.PrivKey(local), wrap), protec: protec, } + s.dialer.Protector = protec s.dsync = NewDialSync(s.doDial) s.limiter = newDialLimiter(s.dialAddr) @@ -162,7 +168,7 @@ func NewBlankSwarm(ctx context.Context, id peer.ID, privkey ci.PrivKey, pstpt ps notifs: make(map[inet.Notifiee]ps.Notifiee), fdRateLimit: make(chan struct{}, concurrentFdDials), Filters: filter.NewFilters(), - dialer: conn.NewDialer(id, privkey, nil, nil), + dialer: conn.NewDialer(id, privkey, nil), } // configure Swarm diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 011b706d81..ab0ad029dd 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -65,11 +65,11 @@ func TestFilterAddrs(t *testing.T) { ps := pstore.NewPeerstore() ctx := context.Background() - if _, err := NewNetwork(ctx, bad, id, ps, metrics.NewBandwidthCounter()); err == nil { + if _, err := NewNetwork(ctx, bad, id, ps, nil, metrics.NewBandwidthCounter()); err == nil { t.Fatal("should have failed to create swarm") } - if _, err := NewNetwork(ctx, goodAndBad, id, ps, metrics.NewBandwidthCounter()); err != nil { + if _, err := NewNetwork(ctx, goodAndBad, id, ps, nil, metrics.NewBandwidthCounter()); err != nil { t.Fatal("should have succeeded in creating swarm", err) } } diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index edb075c18e..93099fe398 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -83,7 +83,7 @@ func (s *Swarm) addListener(tptlist transport.Listener) error { log.Warning("Listener not given PrivateKey, so WILL NOT SECURE conns.") } - list, err := conn.WrapTransportListener(s.Context(), tptlist, s.local, sk, s.protec) + list, err := conn.WrapTransportListenerWithProtector(s.Context(), tptlist, s.local, sk, s.protec) if err != nil { return err } diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index d522f87262..f6fa8c5796 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -22,7 +22,7 @@ type Network Swarm func NewNetwork(ctx context.Context, listen []ma.Multiaddr, local peer.ID, peers pstore.Peerstore, protec ipnet.Protector, bwc metrics.Reporter) (*Network, error) { - s, err := NewSwarm(ctx, listen, local, peers, protec, bwc) + s, err := NewSwarmWithProtector(ctx, listen, local, peers, protec, bwc) if err != nil { return nil, err } diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 95bc268307..7dcdbba99f 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -17,7 +17,7 @@ func GenSwarmNetwork(t *testing.T, ctx context.Context) *Network { ps := pstore.NewPeerstore() ps.AddPubKey(p.ID, p.PubKey) ps.AddPrivKey(p.ID, p.PrivKey) - n, err := NewNetwork(ctx, []ma.Multiaddr{p.Addr}, p.ID, ps, nil) + n, err := NewNetwork(ctx, []ma.Multiaddr{p.Addr}, p.ID, ps, nil, nil) if err != nil { t.Fatal(err) } From 94cbca5f521d33a6adc51f3805ff8060e9431d12 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 14 Nov 2016 19:39:18 +0100 Subject: [PATCH 0194/3965] Revert to normal NewNetwork create NewNetworkWithProtector --- p2p/net/swarm/swarm_addr_test.go | 4 ++-- p2p/net/swarm/swarm_net.go | 8 +++++++- p2p/net/swarm/swarm_net_test.go | 2 +- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index ab0ad029dd..011b706d81 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -65,11 +65,11 @@ func TestFilterAddrs(t *testing.T) { ps := pstore.NewPeerstore() ctx := context.Background() - if _, err := NewNetwork(ctx, bad, id, ps, nil, metrics.NewBandwidthCounter()); err == nil { + if _, err := NewNetwork(ctx, bad, id, ps, metrics.NewBandwidthCounter()); err == nil { t.Fatal("should have failed to create swarm") } - if _, err := NewNetwork(ctx, goodAndBad, id, ps, nil, metrics.NewBandwidthCounter()); err != nil { + if _, err := NewNetwork(ctx, goodAndBad, id, ps, metrics.NewBandwidthCounter()); err != nil { t.Fatal("should have succeeded in creating swarm", err) } } diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index f6fa8c5796..b8f66d958a 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -18,8 +18,14 @@ import ( // to implement inet.Network. type Network Swarm -// NewNetwork constructs a new network and starts listening on given addresses. func NewNetwork(ctx context.Context, listen []ma.Multiaddr, local peer.ID, + peers pstore.Peerstore, bwc metrics.Reporter) (*Network, error) { + + return NewNetworkWithProtector(ctx, listen, local, peers, nil, bwc) +} + +// NewNetwork constructs a new network and starts listening on given addresses. +func NewNetworkWithProtector(ctx context.Context, listen []ma.Multiaddr, local peer.ID, peers pstore.Peerstore, protec ipnet.Protector, bwc metrics.Reporter) (*Network, error) { s, err := NewSwarmWithProtector(ctx, listen, local, peers, protec, bwc) diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 7dcdbba99f..95bc268307 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -17,7 +17,7 @@ func GenSwarmNetwork(t *testing.T, ctx context.Context) *Network { ps := pstore.NewPeerstore() ps.AddPubKey(p.ID, p.PubKey) ps.AddPrivKey(p.ID, p.PrivKey) - n, err := NewNetwork(ctx, []ma.Multiaddr{p.Addr}, p.ID, ps, nil, nil) + n, err := NewNetwork(ctx, []ma.Multiaddr{p.Addr}, p.ID, ps, nil) if err != nil { t.Fatal(err) } From cd20113f5623fc9e48e524c0b783e53cfed6d808 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 16 Nov 2016 18:21:51 -0800 Subject: [PATCH 0195/3965] implement barebones 'blank' host --- p2p/host/blank/blank.go | 142 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 p2p/host/blank/blank.go diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go new file mode 100644 index 0000000000..7c6a32e35f --- /dev/null +++ b/p2p/host/blank/blank.go @@ -0,0 +1,142 @@ +package blankhost + +import ( + "context" + "io" + + logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" + inet "gx/ipfs/QmU3pGGVT1riXp5dBJbNrGpxssVScfvk9236drRHZZbKJ1/go-libp2p-net" + ma "gx/ipfs/QmUAQaWbKxGCUTuoQVvvicbQNZ9APF5pDGWyAZSe93AtKH/go-multiaddr" + pstore "gx/ipfs/QmXXCcQ7CLg5a81Ui9TTR35QcR4y7ZyihxwfjqaHfUVcVo/go-libp2p-peerstore" + protocol "gx/ipfs/QmZNkThpqfVXs9GNbexPrfBbXSLNYeKrE7jwFM2oqHbyqN/go-libp2p-protocol" + mstream "gx/ipfs/QmatJnBK2qyjcy1AYq4Gb5YH16YM7uibdteQ589r46YLvB/go-multistream" + host "gx/ipfs/Qmb6UFbVu1grhv5o5KnouvtZ6cqdrjXj6zLejAHWunxgCt/go-libp2p-host" + peer "gx/ipfs/QmfMmLGoKzCHDN7cGgk64PJr4iipzidDRME8HABSJqvmhC/go-libp2p-peer" +) + +var log = logging.Logger("blankhost") + +// BlankHost is the thinnest implementation of the host.Host interface +type BlankHost struct { + n inet.Network + mux *mstream.MultistreamMuxer +} + +func NewBlankHost(n inet.Network) *BlankHost { + bh := &BlankHost{ + n: n, + mux: mstream.NewMultistreamMuxer(), + } + + n.SetStreamHandler(bh.newStreamHandler) + return bh +} + +var _ host.Host = (*BlankHost)(nil) + +func (bh *BlankHost) Addrs() []ma.Multiaddr { + addrs, err := bh.n.InterfaceListenAddresses() + if err != nil { + log.Debug("error retrieving network interface addrs: ", err) + return nil + } + + return addrs +} + +func (bh *BlankHost) Close() error { + return bh.n.Close() +} + +func (bh *BlankHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { + // absorb addresses into peerstore + bh.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) + + cs := bh.n.ConnsToPeer(pi.ID) + if len(cs) > 0 { + return nil + } + + _, err := bh.Network().DialPeer(ctx, pi.ID) + return err +} + +func (bh *BlankHost) Peerstore() pstore.Peerstore { + return bh.n.Peerstore() +} + +func (bh *BlankHost) ID() peer.ID { + return bh.n.LocalPeer() +} + +func (bh *BlankHost) NewStream(ctx context.Context, p peer.ID, protos ...protocol.ID) (inet.Stream, error) { + s, err := bh.n.NewStream(ctx, p) + if err != nil { + return nil, err + } + + var protoStrs []string + for _, pid := range protos { + protoStrs = append(protoStrs, string(pid)) + } + + log.Error("select: ", protoStrs) + selected, err := mstream.SelectOneOf(protoStrs, s) + if err != nil { + s.Close() + return nil, err + } + + selpid := protocol.ID(selected) + s.SetProtocol(selpid) + bh.Peerstore().AddProtocols(p, selected) + + return s, nil +} + +func (bh *BlankHost) RemoveStreamHandler(p protocol.ID) { + bh.Mux().RemoveHandler(string(p)) +} + +func (bh *BlankHost) SetStreamHandler(pid protocol.ID, handler inet.StreamHandler) { + bh.Mux().AddHandler(string(pid), func(p string, rwc io.ReadWriteCloser) error { + is := rwc.(inet.Stream) + is.SetProtocol(protocol.ID(p)) + handler(is) + return nil + }) +} + +func (bh *BlankHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler inet.StreamHandler) { + bh.Mux().AddHandlerWithFunc(string(pid), m, func(p string, rwc io.ReadWriteCloser) error { + is := rwc.(inet.Stream) + is.SetProtocol(protocol.ID(p)) + handler(is) + return nil + }) +} + +// newStreamHandler is the remote-opened stream handler for inet.Network +func (h *BlankHost) newStreamHandler(s inet.Stream) { + + protoID, handle, err := h.Mux().Negotiate(s) + if err != nil { + log.Warning("protocol mux failed: %s", err) + s.Close() + return + } + + s.SetProtocol(protocol.ID(protoID)) + + go handle(protoID, s) +} + +// TODO: i'm not sure this really needs to be here +func (bh *BlankHost) Mux() *mstream.MultistreamMuxer { + return bh.mux +} + +// TODO: also not sure this fits... Might be better ways around this (leaky abstractions) +func (bh *BlankHost) Network() inet.Network { + return bh.n +} From e334efac8763343b5973138b0789f75a65baf537 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 17 Nov 2016 15:49:21 -0800 Subject: [PATCH 0196/3965] Add SetProtocols method to peerstore This allows us to overwrite known protocols and remove potentially unwanted ones during a full knowledge update. The current AddProtocol just appends to the existing set. --- p2p/host/peerstore/peerstore.go | 13 +++++++++++++ p2p/host/peerstore/peerstore_test.go | 14 ++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 98cc4605bc..9747984317 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -46,6 +46,7 @@ type Peerstore interface { GetProtocols(peer.ID) ([]string, error) AddProtocols(peer.ID, ...string) error + SetProtocols(peer.ID, ...string) error SupportsProtocols(peer.ID, ...string) ([]string, error) } @@ -235,6 +236,18 @@ func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { } } +func (ps *peerstore) SetProtocols(p peer.ID, protos ...string) error { + ps.protolock.Lock() + defer ps.protolock.Unlock() + + protomap := make(map[string]struct{}) + for _, proto := range protos { + protomap[proto] = struct{}{} + } + + return ps.Put(p, "protocols", protomap) +} + func (ps *peerstore) AddProtocols(p peer.ID, protos ...string) error { ps.protolock.Lock() defer ps.protolock.Unlock() diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 29cf6fa7c5..6dbd8ab96c 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -220,6 +220,20 @@ func TestPeerstoreProtoStore(t *testing.T) { if supported[0] != "a" || supported[1] != "b" { t.Fatal("got wrong supported array: ", supported) } + + err = ps.SetProtocols(p1, "other") + if err != nil { + t.Fatal(err) + } + + supported, err = ps.SupportsProtocols(p1, "q", "w", "a", "y", "b") + if err != nil { + t.Fatal(err) + } + + if len(supported) != 0 { + t.Fatal("none of those protocols should have been supported") + } } func TestBasicPeerstore(t *testing.T) { From abf3fbb6157c2e303064c8964ed81d9e546e51d8 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 17 Nov 2016 16:58:28 -0800 Subject: [PATCH 0197/3965] update go-libp2p-net and host (with peerstore updates) --- p2p/host/blank/blank.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go index 7c6a32e35f..6fc227adf3 100644 --- a/p2p/host/blank/blank.go +++ b/p2p/host/blank/blank.go @@ -4,14 +4,14 @@ import ( "context" "io" - logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" - inet "gx/ipfs/QmU3pGGVT1riXp5dBJbNrGpxssVScfvk9236drRHZZbKJ1/go-libp2p-net" - ma "gx/ipfs/QmUAQaWbKxGCUTuoQVvvicbQNZ9APF5pDGWyAZSe93AtKH/go-multiaddr" - pstore "gx/ipfs/QmXXCcQ7CLg5a81Ui9TTR35QcR4y7ZyihxwfjqaHfUVcVo/go-libp2p-peerstore" - protocol "gx/ipfs/QmZNkThpqfVXs9GNbexPrfBbXSLNYeKrE7jwFM2oqHbyqN/go-libp2p-protocol" - mstream "gx/ipfs/QmatJnBK2qyjcy1AYq4Gb5YH16YM7uibdteQ589r46YLvB/go-multistream" - host "gx/ipfs/Qmb6UFbVu1grhv5o5KnouvtZ6cqdrjXj6zLejAHWunxgCt/go-libp2p-host" - peer "gx/ipfs/QmfMmLGoKzCHDN7cGgk64PJr4iipzidDRME8HABSJqvmhC/go-libp2p-peer" + logging "github.com/ipfs/go-log" + host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + protocol "github.com/libp2p/go-libp2p-protocol" + ma "github.com/multiformats/go-multiaddr" + mstream "github.com/whyrusleeping/go-multistream" ) var log = logging.Logger("blankhost") From ac8a3088a4311fd7cd78f089e69bd062f6f9435a Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 22 Nov 2016 15:19:54 -0800 Subject: [PATCH 0198/3965] update multiplex to 0.1.0 --- p2p/muxer/mplex/multiplex.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 7d7f53aa3f..6db90ca62f 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -24,7 +24,7 @@ func (c *conn) IsClosed() bool { // OpenStream creates a new stream. func (c *conn) OpenStream() (smux.Stream, error) { - return c.Multiplex.NewStream(), nil + return c.Multiplex.NewStream() } // AcceptStream accepts a stream opened by the other side. From c1991809dc7ed76f1fb8cd35cd5d31bb2a7667a8 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 22 Nov 2016 19:20:21 -0800 Subject: [PATCH 0199/3965] expose transport parameter to swarm creation --- p2p/net/swarm/swarm.go | 6 +++--- p2p/net/swarm/swarm_net.go | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index b1878ff0e2..6eba8f819b 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -105,12 +105,12 @@ type Swarm struct { func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, local peer.ID, peers pstore.Peerstore, bwc metrics.Reporter) (*Swarm, error) { - return NewSwarmWithProtector(ctx, listenAddrs, local, peers, nil, bwc) + return NewSwarmWithProtector(ctx, listenAddrs, local, peers, nil, PSTransport, bwc) } // NewSwarm constructs a Swarm, with a Chan. func NewSwarmWithProtector(ctx context.Context, listenAddrs []ma.Multiaddr, local peer.ID, - peers pstore.Peerstore, protec ipnet.Protector, bwc metrics.Reporter) (*Swarm, error) { + peers pstore.Peerstore, protec ipnet.Protector, tpt pst.Transport, bwc metrics.Reporter) (*Swarm, error) { listenAddrs, err := filterAddrs(listenAddrs) if err != nil { @@ -125,7 +125,7 @@ func NewSwarmWithProtector(ctx context.Context, listenAddrs []ma.Multiaddr, loca } s := &Swarm{ - swarm: ps.NewSwarm(PSTransport), + swarm: ps.NewSwarm(tpt), local: local, peers: peers, ctx: ctx, diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index b8f66d958a..eb11ecb7da 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -28,7 +28,7 @@ func NewNetwork(ctx context.Context, listen []ma.Multiaddr, local peer.ID, func NewNetworkWithProtector(ctx context.Context, listen []ma.Multiaddr, local peer.ID, peers pstore.Peerstore, protec ipnet.Protector, bwc metrics.Reporter) (*Network, error) { - s, err := NewSwarmWithProtector(ctx, listen, local, peers, protec, bwc) + s, err := NewSwarmWithProtector(ctx, listen, local, peers, protec, PSTransport, bwc) if err != nil { return nil, err } From 92a6e398d64f8efb124551e26af89ad5a509c4b6 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Thu, 24 Nov 2016 16:17:19 +0100 Subject: [PATCH 0200/3965] Implement fingerprint --- p2p/net/pnet/entry.go | 17 ----------------- p2p/net/pnet/fingerprint.go | 24 ++++++++++++++++++++++++ p2p/net/pnet/fingerprint_test.go | 18 ++++++++++++++++++ p2p/net/pnet/protector.go | 21 ++++++++++++++++++--- 4 files changed, 60 insertions(+), 20 deletions(-) delete mode 100644 p2p/net/pnet/entry.go create mode 100644 p2p/net/pnet/fingerprint.go create mode 100644 p2p/net/pnet/fingerprint_test.go diff --git a/p2p/net/pnet/entry.go b/p2p/net/pnet/entry.go deleted file mode 100644 index 9add1b0187..0000000000 --- a/p2p/net/pnet/entry.go +++ /dev/null @@ -1,17 +0,0 @@ -package pnet - -import ( - "bytes" - - ipnet "github.com/libp2p/go-libp2p-interface-pnet" -) - -func NewProtector(key []byte) (ipnet.Protector, error) { - reader := bytes.NewReader(key) - - psk, err := decodeV1PSKKey(reader) - if err != nil { - return nil, err - } - return &protector{psk}, nil -} diff --git a/p2p/net/pnet/fingerprint.go b/p2p/net/pnet/fingerprint.go new file mode 100644 index 0000000000..978a46b625 --- /dev/null +++ b/p2p/net/pnet/fingerprint.go @@ -0,0 +1,24 @@ +package pnet + +import ( + "golang.org/x/crypto/salsa20" + "golang.org/x/crypto/sha3" +) + +var zero64 = make([]byte, 64) + +func fingerprint(psk *[32]byte) []byte { + enc := make([]byte, 64) + + // We encrypt data first so we don't feed PSK to hash function. + // Salsa20 function is not reversible thus increasing our security margin. + salsa20.XORKeyStream(enc, zero64, []byte("finprint"), psk) + + out := make([]byte, 16) + // Then do Shake-128 hash to reduce its length. + // This way if for some reason Shake is broken and Salsa20 preimage is possible, + // attacker has only half of the bytes necessary to recreate psk. + sha3.ShakeSum128(out, enc) + + return out +} diff --git a/p2p/net/pnet/fingerprint_test.go b/p2p/net/pnet/fingerprint_test.go new file mode 100644 index 0000000000..04b238e6b4 --- /dev/null +++ b/p2p/net/pnet/fingerprint_test.go @@ -0,0 +1,18 @@ +package pnet + +import ( + "bytes" + "testing" +) + +var tpsk *[32]byte = &[32]byte{} + +func TestFingerprintGen(t *testing.T) { + f := fingerprint(tpsk) + exp := []byte{0x70, 0x8a, 0x75, 0xaf, 0xd0, 0x5a, 0xff, 0xb0, 0x87, 0x36, 0xcb, 0xf1, 0x7c, 0x73, 0x77, 0x3e} + + if !bytes.Equal(f, exp) { + t.Fatal("fingerprint different than expected") + } + +} diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index 2759eefc27..3bd15ecc4f 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -1,16 +1,31 @@ package pnet import ( + "io" + iconn "github.com/libp2p/go-libp2p-interface-conn" ipnet "github.com/libp2p/go-libp2p-interface-pnet" ) -type protector struct { - psk *[32]byte +var _ ipnet.Protector = (*protector)(nil) + +func NewProtector(input io.Reader) (ipnet.Protector, error) { + psk, err := decodeV1PSKKey(input) + if err != nil { + return nil, err + } + f := fingerprint(psk) + return &protector{psk, f}, nil } -var _ ipnet.Protector = (*protector)(nil) +type protector struct { + psk *[32]byte + fingerprint []byte +} func (p protector) Protect(in iconn.Conn) (iconn.Conn, error) { return newPSKConn(p.psk, in) } +func (p protector) Fingerprint() []byte { + return p.fingerprint +} From 8b54290486c11be0fc641cb91de241f226ff1096 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Thu, 24 Nov 2016 16:23:27 +0100 Subject: [PATCH 0201/3965] Add more info to error --- p2p/net/pnet/psk_conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index a96e277c21..6afa580c47 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -30,7 +30,7 @@ func (c *pskConn) Read(out []byte) (int, error) { return 0, err } if n != 24 { - return 0, errors.New("could not read full nonce") + return 0, errors.New("privnet: could not read full nonce") } c.readS20 = salsa20.New(c.psk, nonce) } From 85eab918ab68e991cc73ec10d45092aaf4044f86 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Fri, 25 Nov 2016 13:24:22 +0100 Subject: [PATCH 0202/3965] Update interface-pnet to 1.0.1 Add more docs, switch errors to ipnet.NewError --- p2p/net/pnet/protector.go | 6 +++++- p2p/net/pnet/psk_conn.go | 28 +++++++++++++++++----------- 2 files changed, 22 insertions(+), 12 deletions(-) diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index 3bd15ecc4f..bfaf15696d 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -15,7 +15,11 @@ func NewProtector(input io.Reader) (ipnet.Protector, error) { return nil, err } f := fingerprint(psk) - return &protector{psk, f}, nil + + return &protector{ + psk: psk, + fingerprint: f, + }, nil } type protector struct { diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index 6afa580c47..a98a692664 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -3,16 +3,22 @@ package pnet import ( "crypto/cipher" "crypto/rand" - "errors" salsa20 "github.com/davidlazar/go-crypto/salsa20" mpool "github.com/jbenet/go-msgio/mpool" iconn "github.com/libp2p/go-libp2p-interface-conn" + ipnet "github.com/libp2p/go-libp2p-interface-pnet" ) // we are using buffer pool as user needs their slice back // so we can't do XOR cripter in place -var bufPool = mpool.ByteSlicePool +var ( + bufPool = mpool.ByteSlicePool + + errShortNonce = ipnet.NewError("could not read full nonce") + errInsecureNil = ipnet.NewError("insecure is nil") + errPSKNil = ipnet.NewError("pre-shread key is nil") +) type pskConn struct { iconn.Conn @@ -30,7 +36,7 @@ func (c *pskConn) Read(out []byte) (int, error) { return 0, err } if n != 24 { - return 0, errors.New("privnet: could not read full nonce") + return 0, errShortNonce } c.readS20 = salsa20.New(c.psk, nonce) } @@ -39,13 +45,13 @@ func (c *pskConn) Read(out []byte) (int, error) { in := bufPool.Get(maxn).([]byte) // get buffer defer bufPool.Put(maxn, in) // put the buffer back - in = in[:maxn] - n, err := c.Conn.Read(in) + in = in[:maxn] // truncate to required length + n, err := c.Conn.Read(in) // read to in if err != nil { return 0, err } - c.readS20.XORKeyStream(out[:n], in[:n]) + c.readS20.XORKeyStream(out[:n], in[:n]) // decrypt to out buffer return n, nil } @@ -68,20 +74,20 @@ func (c *pskConn) Write(in []byte) (int, error) { out := bufPool.Get(n).([]byte) // get buffer defer bufPool.Put(n, out) // put the buffer back - out = out[:n] - c.writeS20.XORKeyStream(out, in) + out = out[:n] // truncate to required length + c.writeS20.XORKeyStream(out, in) // encrypt - return c.Conn.Write(out) + return c.Conn.Write(out) // send } var _ iconn.Conn = (*pskConn)(nil) func newPSKConn(psk *[32]byte, insecure iconn.Conn) (iconn.Conn, error) { if insecure == nil { - return nil, errors.New("insecure is nil") + return nil, errInsecureNil } if psk == nil { - return nil, errors.New("pre-shread key is nil") + return nil, errPSKNil } return &pskConn{ Conn: insecure, From 07cdcbbff2d22135068c96a972e09a2ab3ecac9c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 10 Jan 2017 06:32:49 -0800 Subject: [PATCH 0203/3965] fix multistream import paths --- p2p/host/blank/blank.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go index 6fc227adf3..1b7dae845a 100644 --- a/p2p/host/blank/blank.go +++ b/p2p/host/blank/blank.go @@ -11,7 +11,7 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" protocol "github.com/libp2p/go-libp2p-protocol" ma "github.com/multiformats/go-multiaddr" - mstream "github.com/whyrusleeping/go-multistream" + mstream "github.com/multiformats/go-multistream" ) var log = logging.Logger("blankhost") From 3d5df6b14f8de393b424319ad745d4602eabb85a Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 10 Jan 2017 06:46:38 -0800 Subject: [PATCH 0204/3965] fix multistream import paths --- p2p/muxer/muxer-multistream/multistream.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 87ac7f7c6c..16b10164a2 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -7,9 +7,8 @@ import ( "net" "time" - mss "github.com/whyrusleeping/go-multistream" - smux "github.com/jbenet/go-stream-muxer" + mss "github.com/multiformats/go-multistream" ) var DefaultNegotiateTimeout = time.Second * 60 From a72748927bc5fb9a2ca97bf08c5c112d21545957 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 10 Jan 2017 10:11:07 -0800 Subject: [PATCH 0205/3965] gx publish 0.1.6 --- p2p/host/blank/blank.go | 1 - 1 file changed, 1 deletion(-) diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go index 1b7dae845a..3c75331c48 100644 --- a/p2p/host/blank/blank.go +++ b/p2p/host/blank/blank.go @@ -80,7 +80,6 @@ func (bh *BlankHost) NewStream(ctx context.Context, p peer.ID, protos ...protoco protoStrs = append(protoStrs, string(pid)) } - log.Error("select: ", protoStrs) selected, err := mstream.SelectOneOf(protoStrs, s) if err != nil { s.Close() From addabcef30381b96e4893838e599c0aa48bff551 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 24 Jan 2017 10:20:46 -0800 Subject: [PATCH 0206/3965] fix log message --- p2p/net/swarm/swarm_listen.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 93099fe398..b04c47f11a 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -54,7 +54,7 @@ func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { for i, e := range errs { if e != nil { - log.Warning("listen on %s failed: %s", addrs[i], errs[i]) + log.Warningf("listen on %s failed: %s", addrs[i], errs[i]) } } From ec64623922d764d6d6da85064b9edab68312d494 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 3 Feb 2017 10:32:21 -0800 Subject: [PATCH 0207/3965] gx publish 1.4.2 --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr/sorting.go | 6 +++--- p2p/host/peerstore/addr_manager.go | 4 ++-- p2p/host/peerstore/addr_manager_test.go | 4 ++-- p2p/host/peerstore/metrics.go | 2 +- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peerinfo.go | 4 ++-- p2p/host/peerstore/peerinfo_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 8 ++++---- p2p/host/peerstore/peerstore_test.go | 4 ++-- p2p/host/peerstore/queue/distance.go | 4 ++-- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 4 ++-- p2p/host/peerstore/queue/sync.go | 4 ++-- p2p/host/peerstore/test/utils.go | 6 +++--- 16 files changed, 31 insertions(+), 31 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index 9be78d5ccc..2cbde1aeed 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 06bea07bce..23e3a22d93 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index f8a89150a2..74c6d911e1 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -3,9 +3,9 @@ package addr import ( "bytes" - ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" - mafmt "github.com/whyrusleeping/mafmt" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + manet "gx/ipfs/QmVCNGTyD4EkvNYaAp253uMQ9Rjsjy2oGMvcdJJUoVRfja/go-multiaddr-net" + mafmt "gx/ipfs/QmYjJnSTfXWhYL2cV1xFphPqjqowJqH7ZKLA1As8QrPHbn/mafmt" ) func isFDCostlyTransport(a ma.Multiaddr) bool { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index a676f44252..1c9e3a09ae 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -6,9 +6,9 @@ import ( "sync" "time" - peer "github.com/libp2p/go-libp2p-peer" addr "github.com/libp2p/go-libp2p-peerstore/addr" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index b9cbfd91b1..018474bda2 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) func IDS(t *testing.T, ids string) peer.ID { diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index 05b867692b..145d918679 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -4,7 +4,7 @@ import ( "sync" "time" - "github.com/libp2p/go-libp2p-peer" + "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) // LatencyEWMASmooting governs the decay of the EWMA (the speed diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 94a741182d..990b48fb96 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - testutil "github.com/libp2p/go-libp2p-peer/test" + testutil "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer/test" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index a42d5ef006..da76a7b521 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -3,8 +3,8 @@ package peerstore import ( "encoding/json" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) // PeerInfo is a small struct used to pass around a peer with diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index ca73430a18..d813679067 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -3,8 +3,8 @@ package peerstore import ( "testing" - peer "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 9747984317..8406db451f 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -7,13 +7,13 @@ import ( "sync" "time" - ic "github.com/libp2p/go-libp2p-crypto" + ic "gx/ipfs/QmTuX6VtWTbWgPwd5PMXHyp411RKsW5nBqLKVVRfJMNneb/go-libp2p-crypto" //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" - logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" + "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) var log = logging.Logger("peerstore") diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 6dbd8ab96c..a24a523b13 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -8,8 +8,8 @@ import ( "testing" "time" - peer "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) func getAddrs(t *testing.T, n int) []ma.Multiaddr { diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index f89e8d4210..224831806e 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,8 +5,8 @@ import ( "math/big" "sync" - peer "github.com/libp2p/go-libp2p-peer" - ks "github.com/whyrusleeping/go-keyspace" + ks "gx/ipfs/QmUusaX99BZoELh7dmPgirqRQ1FAmMnmnBn3oiqDFGBUSc/go-keyspace" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index eab523a125..6eb69fb500 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import peer "github.com/libp2p/go-libp2p-peer" +import peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index cd3c80de12..a81bc23f3b 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,8 +7,8 @@ import ( "testing" "time" - u "github.com/ipfs/go-ipfs-util" - peer "github.com/libp2p/go-libp2p-peer" + u "gx/ipfs/QmZuY8aV7zbNXVy6DyN9SmnuH3o9nG852F4aTiSBpts8d1/go-ipfs-util" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index a1edc2ccbc..9c7c514dd6 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -3,8 +3,8 @@ package queue import ( "context" - logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-peer" + logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) var log = logging.Logger("peerqueue") diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index 6c4f0be5fc..81795b12d8 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -3,9 +3,9 @@ package testutil import ( "io" - u "github.com/ipfs/go-ipfs-util" - ci "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" + ci "gx/ipfs/QmTuX6VtWTbWgPwd5PMXHyp411RKsW5nBqLKVVRfJMNneb/go-libp2p-crypto" + u "gx/ipfs/QmZuY8aV7zbNXVy6DyN9SmnuH3o9nG852F4aTiSBpts8d1/go-ipfs-util" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) func RandPeerID() (peer.ID, error) { From 5c58b97791cff762d9d821c13ea2223b42984aee Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 3 Feb 2017 11:02:21 -0800 Subject: [PATCH 0208/3965] gx publish 1.6.8 --- p2p/net/swarm/dial_sync.go | 2 +- p2p/net/swarm/dial_sync_test.go | 2 +- p2p/net/swarm/dial_test.go | 14 +++++----- p2p/net/swarm/limiter.go | 8 +++--- p2p/net/swarm/limiter_test.go | 8 +++--- p2p/net/swarm/peers_test.go | 6 ++-- p2p/net/swarm/simul_test.go | 8 +++--- p2p/net/swarm/swarm.go | 46 +++++++++++++++---------------- p2p/net/swarm/swarm_addr.go | 6 ++-- p2p/net/swarm/swarm_addr_test.go | 10 +++---- p2p/net/swarm/swarm_conn.go | 12 ++++---- p2p/net/swarm/swarm_dial.go | 10 +++---- p2p/net/swarm/swarm_listen.go | 16 +++++------ p2p/net/swarm/swarm_net.go | 14 +++++----- p2p/net/swarm/swarm_net_test.go | 8 +++--- p2p/net/swarm/swarm_notif_test.go | 6 ++-- p2p/net/swarm/swarm_stream.go | 6 ++-- p2p/net/swarm/swarm_test.go | 12 ++++---- 18 files changed, 97 insertions(+), 97 deletions(-) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index 69739e54ec..d22acb69f8 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -4,7 +4,7 @@ import ( "context" "sync" - peer "github.com/libp2p/go-libp2p-peer" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) type DialFunc func(context.Context, peer.ID) (*Conn, error) diff --git a/p2p/net/swarm/dial_sync_test.go b/p2p/net/swarm/dial_sync_test.go index 0d70e226af..bb189d2f2f 100644 --- a/p2p/net/swarm/dial_sync_test.go +++ b/p2p/net/swarm/dial_sync_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - peer "github.com/libp2p/go-libp2p-peer" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) func getMockDialFunc() (DialFunc, func(), context.Context, <-chan struct{}) { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index dabb0bdb00..84df140263 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -7,13 +7,13 @@ import ( "testing" "time" - addrutil "github.com/libp2p/go-addr-util" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - testutil "github.com/libp2p/go-testutil" - ci "github.com/libp2p/go-testutil/ci" - ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + manet "gx/ipfs/QmVCNGTyD4EkvNYaAp253uMQ9Rjsjy2oGMvcdJJUoVRfja/go-multiaddr-net" + addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + testutil "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil" + ci "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil/ci" + pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" ) func closeSwarms(swarms []*Swarm) { diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 67c6af4ddf..db29b874d3 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -4,10 +4,10 @@ import ( "context" "sync" - addrutil "github.com/libp2p/go-addr-util" - iconn "github.com/libp2p/go-libp2p-interface-conn" - peer "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) type dialResult struct { diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index cbb6afdbc6..948e7f1513 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -9,10 +9,10 @@ import ( "testing" "time" - iconn "github.com/libp2p/go-libp2p-interface-conn" - peer "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" - mafmt "github.com/whyrusleeping/mafmt" + iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + mafmt "gx/ipfs/QmYjJnSTfXWhYL2cV1xFphPqjqowJqH7ZKLA1As8QrPHbn/mafmt" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 3259264ae8..28d6d14cdf 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -4,9 +4,9 @@ import ( "testing" "context" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 8935773296..c6d1195b5d 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -7,10 +7,10 @@ import ( "testing" "time" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - ci "github.com/libp2p/go-testutil/ci" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + ci "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil/ci" + pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 6eba8f819b..90ec067372 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -11,29 +11,29 @@ import ( "sync" "time" - logging "github.com/ipfs/go-log" - pst "github.com/jbenet/go-stream-muxer" - "github.com/jbenet/goprocess" - goprocessctx "github.com/jbenet/goprocess/context" - addrutil "github.com/libp2p/go-addr-util" - conn "github.com/libp2p/go-libp2p-conn" - ci "github.com/libp2p/go-libp2p-crypto" - ipnet "github.com/libp2p/go-libp2p-interface-pnet" - metrics "github.com/libp2p/go-libp2p-metrics" - mconn "github.com/libp2p/go-libp2p-metrics/conn" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - transport "github.com/libp2p/go-libp2p-transport" - filter "github.com/libp2p/go-maddr-filter" - ps "github.com/libp2p/go-peerstream" - tcpt "github.com/libp2p/go-tcp-transport" - ma "github.com/multiformats/go-multiaddr" - psmss "github.com/whyrusleeping/go-smux-multistream" - spdy "github.com/whyrusleeping/go-smux-spdystream" - yamux "github.com/whyrusleeping/go-smux-yamux" - mafilter "github.com/whyrusleeping/multiaddr-filter" - ws "github.com/whyrusleeping/ws-transport" + metrics "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics" + mconn "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics/conn" + transport "gx/ipfs/QmR4YrM1DdvYmox249o8aday8oiotkcsT1Qzv42ujPEQDb/go-libp2p-transport" + psmss "gx/ipfs/QmRVYfZ7tWNHPBzWiG6KWGzvT2hcGems8srihsQE29x1U5/go-smux-multistream" + "gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess" + goprocessctx "gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess/context" + mafilter "gx/ipfs/QmSMZwvs3n4GBikZ7hKzT17c3bk65FmyZo2JqtJ16swqCv/multiaddr-filter" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" + ps "gx/ipfs/QmSxPTJ2Jb3nekq3AGMgDyXZR3dhPrSPVSggT3FqJWw4Sa/go-peerstream" + inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" + ci "gx/ipfs/QmTuX6VtWTbWgPwd5PMXHyp411RKsW5nBqLKVVRfJMNneb/go-libp2p-crypto" + filter "gx/ipfs/QmUuCcuet4eneEzto6PxvoC1MrHcxszy7MALrWSKH9inLy/go-maddr-filter" + ipnet "gx/ipfs/QmWNwP8oYzPUK6LqYEH1hoTMuVmQUUbhKQgHeT1AENGkEm/go-libp2p-interface-pnet" + spdy "gx/ipfs/QmWUNsat6Jb19nC5CiJCDXepTkxjdxi3eZqeoB6mrmmaGu/go-smux-spdystream" + tcpt "gx/ipfs/QmXJfmGL3GycHcuqft7vvhf6Xhzad6nuTiEnoEWwAv3zcd/go-tcp-transport" + addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + yamux "gx/ipfs/Qmbn7RYyWzBVXiUp9jZ1dA4VADHy9DtS7iZLwfhEUQvm3U/go-smux-yamux" + ws "gx/ipfs/QmeHbitozRtLYuPrtpcAoJFJixZzTctPtkSsBQGSQuz8pG/ws-transport" + pst "gx/ipfs/QmeZBgYBHvxMukGK5ojg28BCNLB9SeXqT7XXg6o7r2GbJy/go-stream-muxer" + pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" + conn "gx/ipfs/Qmf8Br5gq5XgMrSA6Y6dzY3kaBWwJu6aqXJxxuxCCKsbyD/go-libp2p-conn" ) var log = logging.Logger("swarm2") diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 78a3f5351a..518138f874 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -1,9 +1,9 @@ package swarm import ( - addrutil "github.com/libp2p/go-addr-util" - iconn "github.com/libp2p/go-libp2p-interface-conn" - ma "github.com/multiformats/go-multiaddr" + iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 011b706d81..e9db3697df 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -4,11 +4,11 @@ import ( "context" "testing" - addrutil "github.com/libp2p/go-addr-util" - metrics "github.com/libp2p/go-libp2p-metrics" - pstore "github.com/libp2p/go-libp2p-peerstore" - testutil "github.com/libp2p/go-testutil" - ma "github.com/multiformats/go-multiaddr" + metrics "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" + testutil "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil" + pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 0b0c626cea..f40dbdb5be 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -4,12 +4,12 @@ import ( "context" "fmt" - ic "github.com/libp2p/go-libp2p-crypto" - iconn "github.com/libp2p/go-libp2p-interface-conn" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - ps "github.com/libp2p/go-peerstream" - ma "github.com/multiformats/go-multiaddr" + iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + ps "gx/ipfs/QmSxPTJ2Jb3nekq3AGMgDyXZR3dhPrSPVSggT3FqJWw4Sa/go-peerstream" + inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" + ic "gx/ipfs/QmTuX6VtWTbWgPwd5PMXHyp411RKsW5nBqLKVVRfJMNneb/go-libp2p-crypto" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) // Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index c7cfe91110..84da324a43 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -7,11 +7,11 @@ import ( "sync" "time" - addrutil "github.com/libp2p/go-addr-util" - iconn "github.com/libp2p/go-libp2p-interface-conn" - lgbl "github.com/libp2p/go-libp2p-loggables" - peer "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + lgbl "gx/ipfs/Qma1v2AjS7d2ceYgye4wUVBFFSdB6UPp7jgL8kNtxvFzWu/go-libp2p-loggables" + addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index b04c47f11a..128668bf54 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -4,14 +4,14 @@ import ( "context" "fmt" - conn "github.com/libp2p/go-libp2p-conn" - iconn "github.com/libp2p/go-libp2p-interface-conn" - lgbl "github.com/libp2p/go-libp2p-loggables" - mconn "github.com/libp2p/go-libp2p-metrics/conn" - inet "github.com/libp2p/go-libp2p-net" - transport "github.com/libp2p/go-libp2p-transport" - ps "github.com/libp2p/go-peerstream" - ma "github.com/multiformats/go-multiaddr" + iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" + mconn "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics/conn" + transport "gx/ipfs/QmR4YrM1DdvYmox249o8aday8oiotkcsT1Qzv42ujPEQDb/go-libp2p-transport" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + ps "gx/ipfs/QmSxPTJ2Jb3nekq3AGMgDyXZR3dhPrSPVSggT3FqJWw4Sa/go-peerstream" + inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" + lgbl "gx/ipfs/Qma1v2AjS7d2ceYgye4wUVBFFSdB6UPp7jgL8kNtxvFzWu/go-libp2p-loggables" + conn "gx/ipfs/Qmf8Br5gq5XgMrSA6Y6dzY3kaBWwJu6aqXJxxuxCCKsbyD/go-libp2p-conn" ) func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index eb11ecb7da..2b3ba020f6 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -4,13 +4,13 @@ import ( "context" "fmt" - "github.com/jbenet/goprocess" - ipnet "github.com/libp2p/go-libp2p-interface-pnet" - metrics "github.com/libp2p/go-libp2p-metrics" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" + metrics "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics" + "gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" + ipnet "gx/ipfs/QmWNwP8oYzPUK6LqYEH1hoTMuVmQUUbhKQgHeT1AENGkEm/go-libp2p-interface-pnet" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 95bc268307..9130c8a082 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -6,10 +6,10 @@ import ( "testing" "time" - inet "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" - tu "github.com/libp2p/go-testutil" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" + tu "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil" + pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" ) func GenSwarmNetwork(t *testing.T, ctx context.Context) *Network { diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index d9e9df62a5..18eef7cd61 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -5,9 +5,9 @@ import ( "time" "context" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" ) func streamsSame(a, b inet.Stream) bool { diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index adadca78ec..cf59b169cd 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,9 +3,9 @@ package swarm import ( "time" - inet "github.com/libp2p/go-libp2p-net" - protocol "github.com/libp2p/go-libp2p-protocol" - ps "github.com/libp2p/go-peerstream" + ps "gx/ipfs/QmSxPTJ2Jb3nekq3AGMgDyXZR3dhPrSPVSggT3FqJWw4Sa/go-peerstream" + inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" + protocol "gx/ipfs/QmZNkThpqfVXs9GNbexPrfBbXSLNYeKrE7jwFM2oqHbyqN/go-libp2p-protocol" ) // Stream is a wrapper around a ps.Stream that exposes a way to get diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 9deb918f3c..0b5ab4bdba 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -10,12 +10,12 @@ import ( "testing" "time" - metrics "github.com/libp2p/go-libp2p-metrics" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - testutil "github.com/libp2p/go-testutil" - ma "github.com/multiformats/go-multiaddr" + metrics "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics" + ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" + peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + testutil "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil" + pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" ) func EchoStreamHandler(stream inet.Stream) { From b731baad7bbfcfb8d85fd01fecb6807412232a0c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 3 Feb 2017 12:49:59 -0800 Subject: [PATCH 0209/3965] dont commit with deps rewritten --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr/sorting.go | 6 +++--- p2p/host/peerstore/addr_manager.go | 4 ++-- p2p/host/peerstore/addr_manager_test.go | 4 ++-- p2p/host/peerstore/metrics.go | 2 +- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peerinfo.go | 4 ++-- p2p/host/peerstore/peerinfo_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 8 ++++---- p2p/host/peerstore/peerstore_test.go | 4 ++-- p2p/host/peerstore/queue/distance.go | 4 ++-- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 4 ++-- p2p/host/peerstore/queue/sync.go | 4 ++-- p2p/host/peerstore/test/utils.go | 6 +++--- 16 files changed, 31 insertions(+), 31 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index 2cbde1aeed..9be78d5ccc 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + ma "github.com/multiformats/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 23e3a22d93..06bea07bce 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" + ma "github.com/multiformats/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index 74c6d911e1..f8a89150a2 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -3,9 +3,9 @@ package addr import ( "bytes" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - manet "gx/ipfs/QmVCNGTyD4EkvNYaAp253uMQ9Rjsjy2oGMvcdJJUoVRfja/go-multiaddr-net" - mafmt "gx/ipfs/QmYjJnSTfXWhYL2cV1xFphPqjqowJqH7ZKLA1As8QrPHbn/mafmt" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" + mafmt "github.com/whyrusleeping/mafmt" ) func isFDCostlyTransport(a ma.Multiaddr) bool { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 1c9e3a09ae..a676f44252 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -6,9 +6,9 @@ import ( "sync" "time" + peer "github.com/libp2p/go-libp2p-peer" addr "github.com/libp2p/go-libp2p-peerstore/addr" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) const ( diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 018474bda2..b9cbfd91b1 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func IDS(t *testing.T, ids string) peer.ID { diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index 145d918679..05b867692b 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -4,7 +4,7 @@ import ( "sync" "time" - "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" ) // LatencyEWMASmooting governs the decay of the EWMA (the speed diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 990b48fb96..94a741182d 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - testutil "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer/test" + testutil "github.com/libp2p/go-libp2p-peer/test" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index da76a7b521..a42d5ef006 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -3,8 +3,8 @@ package peerstore import ( "encoding/json" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) // PeerInfo is a small struct used to pass around a peer with diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index d813679067..ca73430a18 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -3,8 +3,8 @@ package peerstore import ( "testing" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 8406db451f..9747984317 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -7,13 +7,13 @@ import ( "sync" "time" - ic "gx/ipfs/QmTuX6VtWTbWgPwd5PMXHyp411RKsW5nBqLKVVRfJMNneb/go-libp2p-crypto" + ic "github.com/libp2p/go-libp2p-crypto" //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" - "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + logging "github.com/ipfs/go-log" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) var log = logging.Logger("peerstore") diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index a24a523b13..6dbd8ab96c 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -8,8 +8,8 @@ import ( "testing" "time" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func getAddrs(t *testing.T, n int) []ma.Multiaddr { diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 224831806e..f89e8d4210 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,8 +5,8 @@ import ( "math/big" "sync" - ks "gx/ipfs/QmUusaX99BZoELh7dmPgirqRQ1FAmMnmnBn3oiqDFGBUSc/go-keyspace" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" + ks "github.com/whyrusleeping/go-keyspace" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index 6eb69fb500..eab523a125 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" +import peer "github.com/libp2p/go-libp2p-peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index a81bc23f3b..cd3c80de12 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,8 +7,8 @@ import ( "testing" "time" - u "gx/ipfs/QmZuY8aV7zbNXVy6DyN9SmnuH3o9nG852F4aTiSBpts8d1/go-ipfs-util" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + u "github.com/ipfs/go-ipfs-util" + peer "github.com/libp2p/go-libp2p-peer" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 9c7c514dd6..a1edc2ccbc 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -3,8 +3,8 @@ package queue import ( "context" - logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + logging "github.com/ipfs/go-log" + peer "github.com/libp2p/go-libp2p-peer" ) var log = logging.Logger("peerqueue") diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index 81795b12d8..6c4f0be5fc 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -3,9 +3,9 @@ package testutil import ( "io" - ci "gx/ipfs/QmTuX6VtWTbWgPwd5PMXHyp411RKsW5nBqLKVVRfJMNneb/go-libp2p-crypto" - u "gx/ipfs/QmZuY8aV7zbNXVy6DyN9SmnuH3o9nG852F4aTiSBpts8d1/go-ipfs-util" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + u "github.com/ipfs/go-ipfs-util" + ci "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" ) func RandPeerID() (peer.ID, error) { From 9661f5e7ab49772f9229e607990f823bb917f330 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 3 Feb 2017 13:05:16 -0800 Subject: [PATCH 0210/3965] gx publish 1.6.9 --- p2p/net/swarm/dial_sync.go | 2 +- p2p/net/swarm/dial_sync_test.go | 2 +- p2p/net/swarm/dial_test.go | 14 +++++----- p2p/net/swarm/limiter.go | 8 +++--- p2p/net/swarm/limiter_test.go | 8 +++--- p2p/net/swarm/peers_test.go | 6 ++-- p2p/net/swarm/simul_test.go | 8 +++--- p2p/net/swarm/swarm.go | 46 +++++++++++++++---------------- p2p/net/swarm/swarm_addr.go | 6 ++-- p2p/net/swarm/swarm_addr_test.go | 10 +++---- p2p/net/swarm/swarm_conn.go | 12 ++++---- p2p/net/swarm/swarm_dial.go | 10 +++---- p2p/net/swarm/swarm_listen.go | 16 +++++------ p2p/net/swarm/swarm_net.go | 14 +++++----- p2p/net/swarm/swarm_net_test.go | 8 +++--- p2p/net/swarm/swarm_notif_test.go | 6 ++-- p2p/net/swarm/swarm_stream.go | 6 ++-- p2p/net/swarm/swarm_test.go | 12 ++++---- 18 files changed, 97 insertions(+), 97 deletions(-) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index d22acb69f8..69739e54ec 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -4,7 +4,7 @@ import ( "context" "sync" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ) type DialFunc func(context.Context, peer.ID) (*Conn, error) diff --git a/p2p/net/swarm/dial_sync_test.go b/p2p/net/swarm/dial_sync_test.go index bb189d2f2f..0d70e226af 100644 --- a/p2p/net/swarm/dial_sync_test.go +++ b/p2p/net/swarm/dial_sync_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ) func getMockDialFunc() (DialFunc, func(), context.Context, <-chan struct{}) { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 84df140263..dabb0bdb00 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -7,13 +7,13 @@ import ( "testing" "time" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - manet "gx/ipfs/QmVCNGTyD4EkvNYaAp253uMQ9Rjsjy2oGMvcdJJUoVRfja/go-multiaddr-net" - addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" - testutil "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil" - ci "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil/ci" - pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" + addrutil "github.com/libp2p/go-addr-util" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + testutil "github.com/libp2p/go-testutil" + ci "github.com/libp2p/go-testutil/ci" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) func closeSwarms(swarms []*Swarm) { diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index db29b874d3..67c6af4ddf 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -4,10 +4,10 @@ import ( "context" "sync" - iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + addrutil "github.com/libp2p/go-addr-util" + iconn "github.com/libp2p/go-libp2p-interface-conn" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) type dialResult struct { diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 948e7f1513..cbb6afdbc6 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -9,10 +9,10 @@ import ( "testing" "time" - iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - mafmt "gx/ipfs/QmYjJnSTfXWhYL2cV1xFphPqjqowJqH7ZKLA1As8QrPHbn/mafmt" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + iconn "github.com/libp2p/go-libp2p-interface-conn" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" + mafmt "github.com/whyrusleeping/mafmt" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 28d6d14cdf..3259264ae8 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -4,9 +4,9 @@ import ( "testing" "context" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" - pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" ) func TestPeers(t *testing.T) { diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index c6d1195b5d..8935773296 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -7,10 +7,10 @@ import ( "testing" "time" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" - ci "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil/ci" - pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ci "github.com/libp2p/go-testutil/ci" + ma "github.com/multiformats/go-multiaddr" ) func TestSimultOpen(t *testing.T) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 90ec067372..6eba8f819b 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -11,29 +11,29 @@ import ( "sync" "time" - metrics "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics" - mconn "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics/conn" - transport "gx/ipfs/QmR4YrM1DdvYmox249o8aday8oiotkcsT1Qzv42ujPEQDb/go-libp2p-transport" - psmss "gx/ipfs/QmRVYfZ7tWNHPBzWiG6KWGzvT2hcGems8srihsQE29x1U5/go-smux-multistream" - "gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess" - goprocessctx "gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess/context" - mafilter "gx/ipfs/QmSMZwvs3n4GBikZ7hKzT17c3bk65FmyZo2JqtJ16swqCv/multiaddr-filter" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" - ps "gx/ipfs/QmSxPTJ2Jb3nekq3AGMgDyXZR3dhPrSPVSggT3FqJWw4Sa/go-peerstream" - inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" - ci "gx/ipfs/QmTuX6VtWTbWgPwd5PMXHyp411RKsW5nBqLKVVRfJMNneb/go-libp2p-crypto" - filter "gx/ipfs/QmUuCcuet4eneEzto6PxvoC1MrHcxszy7MALrWSKH9inLy/go-maddr-filter" - ipnet "gx/ipfs/QmWNwP8oYzPUK6LqYEH1hoTMuVmQUUbhKQgHeT1AENGkEm/go-libp2p-interface-pnet" - spdy "gx/ipfs/QmWUNsat6Jb19nC5CiJCDXepTkxjdxi3eZqeoB6mrmmaGu/go-smux-spdystream" - tcpt "gx/ipfs/QmXJfmGL3GycHcuqft7vvhf6Xhzad6nuTiEnoEWwAv3zcd/go-tcp-transport" - addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" - yamux "gx/ipfs/Qmbn7RYyWzBVXiUp9jZ1dA4VADHy9DtS7iZLwfhEUQvm3U/go-smux-yamux" - ws "gx/ipfs/QmeHbitozRtLYuPrtpcAoJFJixZzTctPtkSsBQGSQuz8pG/ws-transport" - pst "gx/ipfs/QmeZBgYBHvxMukGK5ojg28BCNLB9SeXqT7XXg6o7r2GbJy/go-stream-muxer" - pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" - conn "gx/ipfs/Qmf8Br5gq5XgMrSA6Y6dzY3kaBWwJu6aqXJxxuxCCKsbyD/go-libp2p-conn" + logging "github.com/ipfs/go-log" + pst "github.com/jbenet/go-stream-muxer" + "github.com/jbenet/goprocess" + goprocessctx "github.com/jbenet/goprocess/context" + addrutil "github.com/libp2p/go-addr-util" + conn "github.com/libp2p/go-libp2p-conn" + ci "github.com/libp2p/go-libp2p-crypto" + ipnet "github.com/libp2p/go-libp2p-interface-pnet" + metrics "github.com/libp2p/go-libp2p-metrics" + mconn "github.com/libp2p/go-libp2p-metrics/conn" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + transport "github.com/libp2p/go-libp2p-transport" + filter "github.com/libp2p/go-maddr-filter" + ps "github.com/libp2p/go-peerstream" + tcpt "github.com/libp2p/go-tcp-transport" + ma "github.com/multiformats/go-multiaddr" + psmss "github.com/whyrusleeping/go-smux-multistream" + spdy "github.com/whyrusleeping/go-smux-spdystream" + yamux "github.com/whyrusleeping/go-smux-yamux" + mafilter "github.com/whyrusleeping/multiaddr-filter" + ws "github.com/whyrusleeping/ws-transport" ) var log = logging.Logger("swarm2") diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 518138f874..78a3f5351a 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -1,9 +1,9 @@ package swarm import ( - iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" + addrutil "github.com/libp2p/go-addr-util" + iconn "github.com/libp2p/go-libp2p-interface-conn" + ma "github.com/multiformats/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index e9db3697df..011b706d81 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -4,11 +4,11 @@ import ( "context" "testing" - metrics "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" - testutil "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil" - pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" + addrutil "github.com/libp2p/go-addr-util" + metrics "github.com/libp2p/go-libp2p-metrics" + pstore "github.com/libp2p/go-libp2p-peerstore" + testutil "github.com/libp2p/go-testutil" + ma "github.com/multiformats/go-multiaddr" ) func TestFilterAddrs(t *testing.T) { diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index f40dbdb5be..0b0c626cea 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -4,12 +4,12 @@ import ( "context" "fmt" - iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - ps "gx/ipfs/QmSxPTJ2Jb3nekq3AGMgDyXZR3dhPrSPVSggT3FqJWw4Sa/go-peerstream" - inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" - ic "gx/ipfs/QmTuX6VtWTbWgPwd5PMXHyp411RKsW5nBqLKVVRfJMNneb/go-libp2p-crypto" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + ic "github.com/libp2p/go-libp2p-crypto" + iconn "github.com/libp2p/go-libp2p-interface-conn" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + ps "github.com/libp2p/go-peerstream" + ma "github.com/multiformats/go-multiaddr" ) // Conn is a simple wrapper around a ps.Conn that also exposes diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 84da324a43..c7cfe91110 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -7,11 +7,11 @@ import ( "sync" "time" - iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - lgbl "gx/ipfs/Qma1v2AjS7d2ceYgye4wUVBFFSdB6UPp7jgL8kNtxvFzWu/go-libp2p-loggables" - addrutil "gx/ipfs/Qmb86KiitngVm5AGQ18viwCBwwcTvULAtz3X74YWKAevnc/go-addr-util" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + addrutil "github.com/libp2p/go-addr-util" + iconn "github.com/libp2p/go-libp2p-interface-conn" + lgbl "github.com/libp2p/go-libp2p-loggables" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) // Diagram of dial sync: diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 128668bf54..b04c47f11a 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -4,14 +4,14 @@ import ( "context" "fmt" - iconn "gx/ipfs/QmNmKcQiJB5cUTXXXS1o44s4PdeYevmH8aHhQJCGYjwBj3/go-libp2p-interface-conn" - mconn "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics/conn" - transport "gx/ipfs/QmR4YrM1DdvYmox249o8aday8oiotkcsT1Qzv42ujPEQDb/go-libp2p-transport" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - ps "gx/ipfs/QmSxPTJ2Jb3nekq3AGMgDyXZR3dhPrSPVSggT3FqJWw4Sa/go-peerstream" - inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" - lgbl "gx/ipfs/Qma1v2AjS7d2ceYgye4wUVBFFSdB6UPp7jgL8kNtxvFzWu/go-libp2p-loggables" - conn "gx/ipfs/Qmf8Br5gq5XgMrSA6Y6dzY3kaBWwJu6aqXJxxuxCCKsbyD/go-libp2p-conn" + conn "github.com/libp2p/go-libp2p-conn" + iconn "github.com/libp2p/go-libp2p-interface-conn" + lgbl "github.com/libp2p/go-libp2p-loggables" + mconn "github.com/libp2p/go-libp2p-metrics/conn" + inet "github.com/libp2p/go-libp2p-net" + transport "github.com/libp2p/go-libp2p-transport" + ps "github.com/libp2p/go-peerstream" + ma "github.com/multiformats/go-multiaddr" ) func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go index 2b3ba020f6..eb11ecb7da 100644 --- a/p2p/net/swarm/swarm_net.go +++ b/p2p/net/swarm/swarm_net.go @@ -4,13 +4,13 @@ import ( "context" "fmt" - metrics "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics" - "gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" - ipnet "gx/ipfs/QmWNwP8oYzPUK6LqYEH1hoTMuVmQUUbhKQgHeT1AENGkEm/go-libp2p-interface-pnet" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" - pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" + "github.com/jbenet/goprocess" + ipnet "github.com/libp2p/go-libp2p-interface-pnet" + metrics "github.com/libp2p/go-libp2p-metrics" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" ) // Network implements the inet.Network interface. diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 9130c8a082..95bc268307 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -6,10 +6,10 @@ import ( "testing" "time" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" - tu "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil" - pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" + inet "github.com/libp2p/go-libp2p-net" + pstore "github.com/libp2p/go-libp2p-peerstore" + tu "github.com/libp2p/go-testutil" + ma "github.com/multiformats/go-multiaddr" ) func GenSwarmNetwork(t *testing.T, ctx context.Context) *Network { diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 18eef7cd61..d9e9df62a5 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -5,9 +5,9 @@ import ( "time" "context" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func streamsSame(a, b inet.Stream) bool { diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index cf59b169cd..adadca78ec 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -3,9 +3,9 @@ package swarm import ( "time" - ps "gx/ipfs/QmSxPTJ2Jb3nekq3AGMgDyXZR3dhPrSPVSggT3FqJWw4Sa/go-peerstream" - inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" - protocol "gx/ipfs/QmZNkThpqfVXs9GNbexPrfBbXSLNYeKrE7jwFM2oqHbyqN/go-libp2p-protocol" + inet "github.com/libp2p/go-libp2p-net" + protocol "github.com/libp2p/go-libp2p-protocol" + ps "github.com/libp2p/go-peerstream" ) // Stream is a wrapper around a ps.Stream that exposes a way to get diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 0b5ab4bdba..9deb918f3c 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -10,12 +10,12 @@ import ( "testing" "time" - metrics "gx/ipfs/QmPfeoLzBPsiwbHLwev8ecgjd1RPCYfsAHhdFA8avEnXQm/go-libp2p-metrics" - ma "gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG/go-multiaddr" - inet "gx/ipfs/QmTWdBq4ZaLGbXcMiNasSR5ywomkgQiJDqQknEU16z3pZR/go-libp2p-net" - peer "gx/ipfs/QmbKtZxyDqUJp7Ad8tGr5nrLqoi9nfgqFxcNbmLJbfaHPe/go-libp2p-peer" - testutil "gx/ipfs/QmbN6CcaGuVSWwhYq5nDQs98hWHgu5PTBgVjnGV28YeGRk/go-testutil" - pstore "gx/ipfs/QmekwEJM81TqzKzupMvK68hVNfBNvBengWkpc8nAw2bjBf/go-libp2p-peerstore" + metrics "github.com/libp2p/go-libp2p-metrics" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + testutil "github.com/libp2p/go-testutil" + ma "github.com/multiformats/go-multiaddr" ) func EchoStreamHandler(stream inet.Stream) { From 5f77c1f1b9ad151c96e2cdc72215a4cde383374d Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Thu, 16 Feb 2017 13:20:44 +0100 Subject: [PATCH 0211/3965] Add function to generate PSK key Resolves https://github.com/libp2p/go-libp2p-pnet/issues/5 --- p2p/net/pnet/generate.go | 25 +++++++++++++++++++++++++ p2p/net/pnet/generate_test.go | 32 ++++++++++++++++++++++++++++++++ p2p/net/pnet/protector.go | 2 ++ 3 files changed, 59 insertions(+) create mode 100644 p2p/net/pnet/generate.go create mode 100644 p2p/net/pnet/generate_test.go diff --git a/p2p/net/pnet/generate.go b/p2p/net/pnet/generate.go new file mode 100644 index 0000000000..f01257f56b --- /dev/null +++ b/p2p/net/pnet/generate.go @@ -0,0 +1,25 @@ +package pnet + +import ( + "bytes" + "crypto/rand" + "encoding/hex" + "io" +) + +func newLine() io.Reader { + return bytes.NewReader([]byte("\n")) +} + +func GenerateV1PSK() io.Reader { + psk := make([]byte, 32) + rand.Read(psk) + hexPsk := make([]byte, len(psk)*2) + hex.Encode(hexPsk, psk) + + // just a shortcut to NewReader + nr := func(b []byte) io.Reader { + return bytes.NewReader(b) + } + return io.MultiReader(nr(pathPSKv1), newLine(), nr([]byte("/base16/")), newLine(), nr(hexPsk)) +} diff --git a/p2p/net/pnet/generate_test.go b/p2p/net/pnet/generate_test.go new file mode 100644 index 0000000000..a672163525 --- /dev/null +++ b/p2p/net/pnet/generate_test.go @@ -0,0 +1,32 @@ +package pnet + +import ( + "bytes" + "io/ioutil" + "testing" +) + +func TestGeneratedPSKCanBeUsed(t *testing.T) { + psk := GenerateV1PSK() + + _, err := NewProtector(psk) + if err != nil { + t.Fatal(err) + } +} + +func TestGeneratedKeysAreDifferent(t *testing.T) { + psk1 := GenerateV1PSK() + psk2 := GenerateV1PSK() + bpsk1, err := ioutil.ReadAll(psk1) + if err != nil { + t.Fatal(err) + } + bpsk2, err := ioutil.ReadAll(psk2) + if err != nil { + t.Fatal(err) + } + if bytes.Equal(bpsk1, bpsk2) { + t.Fatal("generated keys are the same") + } +} diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index bfaf15696d..77cd6b6c71 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -9,6 +9,8 @@ import ( var _ ipnet.Protector = (*protector)(nil) +// NewProtector creates ipnet.Protector instance from a io.Reader stream +// that should include Multicodec encoded V1 PSK. func NewProtector(input io.Reader) (ipnet.Protector, error) { psk, err := decodeV1PSKKey(input) if err != nil { From e48088b6662fa4b1a4b428a32bc8177de85f8bb4 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Thu, 16 Feb 2017 14:56:28 +0100 Subject: [PATCH 0212/3965] Move to go-libp2p-transport.Conn --- p2p/net/pnet/protector.go | 4 ++-- p2p/net/pnet/psk_conn.go | 8 ++++---- p2p/net/pnet/psk_conn_test.go | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index 77cd6b6c71..d613cac764 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -3,8 +3,8 @@ package pnet import ( "io" - iconn "github.com/libp2p/go-libp2p-interface-conn" ipnet "github.com/libp2p/go-libp2p-interface-pnet" + tconn "github.com/libp2p/go-libp2p-transport" ) var _ ipnet.Protector = (*protector)(nil) @@ -29,7 +29,7 @@ type protector struct { fingerprint []byte } -func (p protector) Protect(in iconn.Conn) (iconn.Conn, error) { +func (p protector) Protect(in tconn.Conn) (tconn.Conn, error) { return newPSKConn(p.psk, in) } func (p protector) Fingerprint() []byte { diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index a98a692664..d60b7ef717 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -6,8 +6,8 @@ import ( salsa20 "github.com/davidlazar/go-crypto/salsa20" mpool "github.com/jbenet/go-msgio/mpool" - iconn "github.com/libp2p/go-libp2p-interface-conn" ipnet "github.com/libp2p/go-libp2p-interface-pnet" + tconn "github.com/libp2p/go-libp2p-transport" ) // we are using buffer pool as user needs their slice back @@ -21,7 +21,7 @@ var ( ) type pskConn struct { - iconn.Conn + tconn.Conn psk *[32]byte writeS20 cipher.Stream @@ -80,9 +80,9 @@ func (c *pskConn) Write(in []byte) (int, error) { return c.Conn.Write(out) // send } -var _ iconn.Conn = (*pskConn)(nil) +var _ tconn.Conn = (*pskConn)(nil) -func newPSKConn(psk *[32]byte, insecure iconn.Conn) (iconn.Conn, error) { +func newPSKConn(psk *[32]byte, insecure tconn.Conn) (tconn.Conn, error) { if insecure == nil { return nil, errInsecureNil } diff --git a/p2p/net/pnet/psk_conn_test.go b/p2p/net/pnet/psk_conn_test.go index f81b55cf48..17a3963d92 100644 --- a/p2p/net/pnet/psk_conn_test.go +++ b/p2p/net/pnet/psk_conn_test.go @@ -7,12 +7,12 @@ import ( "testing" dconn "github.com/Kubuxu/go-libp2p-dummy-conn" - iconn "github.com/libp2p/go-libp2p-interface-conn" + tconn "github.com/libp2p/go-libp2p-transport" ) var testPSK = [32]byte{} // null bytes are as good test key as any other key -func setupPSKConns(ctx context.Context, t *testing.T) (iconn.Conn, iconn.Conn) { +func setupPSKConns(ctx context.Context, t *testing.T) (tconn.Conn, tconn.Conn) { conn1, conn2, err := dconn.NewDummyConnPair() if err != nil { t.Fatal(err) From 93471c19633a5ad7b240220404d1e801022c424c Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Fri, 17 Feb 2017 14:24:50 +0100 Subject: [PATCH 0213/3965] Add docs to GenerateV1PSK --- p2p/net/pnet/generate.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/net/pnet/generate.go b/p2p/net/pnet/generate.go index f01257f56b..f44dfe643a 100644 --- a/p2p/net/pnet/generate.go +++ b/p2p/net/pnet/generate.go @@ -11,6 +11,7 @@ func newLine() io.Reader { return bytes.NewReader([]byte("\n")) } +// GenerateV1PSK generates new PSK key that can be used with NewProtector func GenerateV1PSK() io.Reader { psk := make([]byte, 32) rand.Read(psk) From 36c695a63fe2e47612854172352a496c6de4cce8 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 7 Mar 2017 17:16:30 +0700 Subject: [PATCH 0214/3965] initial commmit --- p2p/transport/quic/dialer.go | 24 ++++++++++++++++++++++++ p2p/transport/quic/listener.go | 28 ++++++++++++++++++++++++++++ p2p/transport/quic/transport.go | 26 ++++++++++++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100644 p2p/transport/quic/dialer.go create mode 100644 p2p/transport/quic/listener.go create mode 100644 p2p/transport/quic/transport.go diff --git a/p2p/transport/quic/dialer.go b/p2p/transport/quic/dialer.go new file mode 100644 index 0000000000..a5536d4f49 --- /dev/null +++ b/p2p/transport/quic/dialer.go @@ -0,0 +1,24 @@ +package libp2pquic + +import ( + "context" + + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" +) + +type dialer struct{} + +func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { + panic("not implemented") +} + +func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { + panic("not implemented") +} + +func (d *dialer) Matches(ma.Multiaddr) bool { + panic("not implemented") +} + +var _ tpt.Dialer = &dialer{} diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go new file mode 100644 index 0000000000..bf867770e4 --- /dev/null +++ b/p2p/transport/quic/listener.go @@ -0,0 +1,28 @@ +package libp2pquic + +import ( + "net" + + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" +) + +type listener struct{} + +func (l *listener) Accept() (tpt.Conn, error) { + panic("not implemented") +} + +func (l *listener) Close() error { + panic("not implemented") +} + +func (l *listener) Addr() net.Addr { + panic("not implemented") +} + +func (l *listener) Multiaddr() ma.Multiaddr { + panic("not implemented") +} + +var _ tpt.Listener = &listener{} diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go new file mode 100644 index 0000000000..8a35b4077d --- /dev/null +++ b/p2p/transport/quic/transport.go @@ -0,0 +1,26 @@ +package libp2pquic + +import ( + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" +) + +type QuicTransport struct{} + +func NewQuicTransport() *QuicTransport { + return &QuicTransport{} +} + +func (t *QuicTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { + panic("not implemented") +} + +func (t *QuicTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { + panic("not implemented") +} + +func (t *QuicTransport) Matches(ma.Multiaddr) bool { + panic("not implemented") +} + +var _ tpt.Transport = &QuicTransport{} From 67700695b0c165c1ac342b4dabf2082f674c4685 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 15 Mar 2017 19:41:08 +0700 Subject: [PATCH 0215/3965] implement the listener --- p2p/transport/quic/conn.go | 60 +++++++++++++++++++++++++++++++++ p2p/transport/quic/listener.go | 43 ++++++++++++++++++++--- p2p/transport/quic/transport.go | 39 ++++++++++++++++++--- 3 files changed, 133 insertions(+), 9 deletions(-) create mode 100644 p2p/transport/quic/conn.go diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go new file mode 100644 index 0000000000..f9340ad2c9 --- /dev/null +++ b/p2p/transport/quic/conn.go @@ -0,0 +1,60 @@ +package libp2pquic + +import ( + "net" + "time" + + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" +) + +type conn struct { + quicConn net.Conn + transport tpt.Transport +} + +func (c *conn) Read(p []byte) (int, error) { + return c.quicConn.Read(p) +} + +func (c *conn) Write(p []byte) (int, error) { + return c.quicConn.Write(p) +} + +func (c *conn) Close() error { + return c.quicConn.Close() +} + +func (c *conn) LocalAddr() net.Addr { + return c.quicConn.LocalAddr() +} + +func (c *conn) RemoteAddr() net.Addr { + return c.quicConn.RemoteAddr() +} + +func (c *conn) LocalMultiaddr() ma.Multiaddr { + panic("not implemented") +} + +func (c *conn) RemoteMultiaddr() ma.Multiaddr { + panic("not implemented") +} + +func (c *conn) Transport() tpt.Transport { + return c.transport +} + +func (c *conn) SetDeadline(t time.Time) error { + return nil +} + +func (c *conn) SetReadDeadline(t time.Time) error { + return nil +} + +func (c *conn) SetWriteDeadline(t time.Time) error { + return nil +} + +var _ tpt.Conn = &conn{} diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index bf867770e4..a0aea58e02 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -1,28 +1,61 @@ package libp2pquic import ( + "crypto/tls" "net" + pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" + quicconn "github.com/marten-seemann/quic-conn" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) -type listener struct{} +type listener struct { + laddr ma.Multiaddr + quicListener net.Listener + + transport tpt.Transport +} + +func newListener(laddr ma.Multiaddr, peers pstore.Peerstore, transport tpt.Transport) (*listener, error) { + tlsConf := &tls.Config{} + network, host, err := manet.DialArgs(laddr) + if err != nil { + return nil, err + } + qln, err := quicconn.Listen(network, host, tlsConf) + if err != nil { + return nil, err + } + return &listener{ + laddr: laddr, + quicListener: qln, + transport: transport, + }, nil +} func (l *listener) Accept() (tpt.Conn, error) { - panic("not implemented") + c, err := l.quicListener.Accept() + if err != nil { + return nil, err + } + return &conn{ + quicConn: c, + transport: l.transport, + }, nil } func (l *listener) Close() error { - panic("not implemented") + return l.quicListener.Close() } func (l *listener) Addr() net.Addr { - panic("not implemented") + return l.quicListener.Addr() } func (l *listener) Multiaddr() ma.Multiaddr { - panic("not implemented") + return l.laddr } var _ tpt.Listener = &listener{} diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 8a35b4077d..1982e9c053 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -1,22 +1,53 @@ package libp2pquic import ( + "sync" + + pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) -type QuicTransport struct{} +// QuicTransport implements a QUIC Transport +type QuicTransport struct { + mutex sync.Mutex + + peers pstore.Peerstore + + listeners map[string]tpt.Listener +} -func NewQuicTransport() *QuicTransport { - return &QuicTransport{} +// NewQuicTransport creates a new QUIC Transport +// it tracks dialers and listeners created +func NewQuicTransport(peers pstore.Peerstore) *QuicTransport { + return &QuicTransport{ + peers: peers, + listeners: make(map[string]tpt.Listener), + } } func (t *QuicTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { panic("not implemented") } +// Listen starts listening on laddr func (t *QuicTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { - panic("not implemented") + // TODO: check if laddr is actually a QUIC address + t.mutex.Lock() + defer t.mutex.Unlock() + + l, ok := t.listeners[laddr.String()] + if ok { + return l, nil + } + + ln, err := newListener(laddr, t.peers, t) + if err != nil { + return nil, err + } + + t.listeners[laddr.String()] = ln + return ln, nil } func (t *QuicTransport) Matches(ma.Multiaddr) bool { From 3a37e5880db42748ca54e04ce9623592e20feac5 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 17 Mar 2017 12:50:12 +0700 Subject: [PATCH 0216/3965] fix logging of dial errors --- p2p/net/swarm/swarm_dial.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index c7cfe91110..627cce45ad 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -318,7 +318,7 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. case resp := <-respch: active-- if resp.Err != nil { - log.Info("got error on dial to %s: ", resp.Addr, resp.Err) + log.Infof("got error on dial to %s: %s", resp.Addr, resp.Err) // Errors are normal, lots of dials will fail exitErr = resp.Err From 3b09dd9ce2c061f72cc3e3dce176d040dbc47771 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 17 Mar 2017 16:57:06 +0700 Subject: [PATCH 0217/3965] use conn wrapper functions to construct a go-libp2p-transport.Conn --- p2p/transport/quic/conn.go | 60 ---------------------------------- p2p/transport/quic/listener.go | 12 +++++-- 2 files changed, 9 insertions(+), 63 deletions(-) delete mode 100644 p2p/transport/quic/conn.go diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go deleted file mode 100644 index f9340ad2c9..0000000000 --- a/p2p/transport/quic/conn.go +++ /dev/null @@ -1,60 +0,0 @@ -package libp2pquic - -import ( - "net" - "time" - - tpt "github.com/libp2p/go-libp2p-transport" - ma "github.com/multiformats/go-multiaddr" -) - -type conn struct { - quicConn net.Conn - transport tpt.Transport -} - -func (c *conn) Read(p []byte) (int, error) { - return c.quicConn.Read(p) -} - -func (c *conn) Write(p []byte) (int, error) { - return c.quicConn.Write(p) -} - -func (c *conn) Close() error { - return c.quicConn.Close() -} - -func (c *conn) LocalAddr() net.Addr { - return c.quicConn.LocalAddr() -} - -func (c *conn) RemoteAddr() net.Addr { - return c.quicConn.RemoteAddr() -} - -func (c *conn) LocalMultiaddr() ma.Multiaddr { - panic("not implemented") -} - -func (c *conn) RemoteMultiaddr() ma.Multiaddr { - panic("not implemented") -} - -func (c *conn) Transport() tpt.Transport { - return c.transport -} - -func (c *conn) SetDeadline(t time.Time) error { - return nil -} - -func (c *conn) SetReadDeadline(t time.Time) error { - return nil -} - -func (c *conn) SetWriteDeadline(t time.Time) error { - return nil -} - -var _ tpt.Conn = &conn{} diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index a0aea58e02..9ef7386bd0 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -40,9 +40,15 @@ func (l *listener) Accept() (tpt.Conn, error) { if err != nil { return nil, err } - return &conn{ - quicConn: c, - transport: l.transport, + + mnc, err := manet.WrapNetConn(c) + if err != nil { + return nil, err + } + + return &tpt.ConnWrap{ + Conn: mnc, + Tpt: l.transport, }, nil } From 260693d121cef99cf0dedb5179e9ed1c29fe9ae1 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 18 Mar 2017 13:43:54 +0700 Subject: [PATCH 0218/3965] add (empty) test suite and travis config --- p2p/transport/quic/libp2pquic_suite_test.go | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 p2p/transport/quic/libp2pquic_suite_test.go diff --git a/p2p/transport/quic/libp2pquic_suite_test.go b/p2p/transport/quic/libp2pquic_suite_test.go new file mode 100644 index 0000000000..d3766da3e0 --- /dev/null +++ b/p2p/transport/quic/libp2pquic_suite_test.go @@ -0,0 +1,13 @@ +package libp2pquic + +import ( + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" + + "testing" +) + +func TestQuicGo(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "libp2p QUIC Transport Suite") +} From a06bda1f4ee1fcca6d0372718440c298d6c2c0dd Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 18 Mar 2017 15:20:41 +0700 Subject: [PATCH 0219/3965] implement transport.Matches --- p2p/transport/quic/transport.go | 5 +++-- p2p/transport/quic/transport_test.go | 25 +++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 2 deletions(-) create mode 100644 p2p/transport/quic/transport_test.go diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 1982e9c053..cfd61f01b4 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -6,6 +6,7 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" + "github.com/whyrusleeping/mafmt" ) // QuicTransport implements a QUIC Transport @@ -50,8 +51,8 @@ func (t *QuicTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { return ln, nil } -func (t *QuicTransport) Matches(ma.Multiaddr) bool { - panic("not implemented") +func (t *QuicTransport) Matches(a ma.Multiaddr) bool { + return mafmt.QUIC.Matches(a) } var _ tpt.Transport = &QuicTransport{} diff --git a/p2p/transport/quic/transport_test.go b/p2p/transport/quic/transport_test.go new file mode 100644 index 0000000000..a6cd01255d --- /dev/null +++ b/p2p/transport/quic/transport_test.go @@ -0,0 +1,25 @@ +package libp2pquic + +import ( + ma "github.com/multiformats/go-multiaddr" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +var _ = Describe("Transport", func() { + var t *QuicTransport + + BeforeEach(func() { + t = NewQuicTransport(nil) + }) + + It("matches", func() { + invalidAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") + Expect(err).ToNot(HaveOccurred()) + validAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") + Expect(err).ToNot(HaveOccurred()) + Expect(t.Matches(invalidAddr)).To(BeFalse()) + Expect(t.Matches(validAddr)).To(BeTrue()) + }) +}) From 9fafe5c100ab0ebf5ca423a832ab19babef37d40 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 18 Mar 2017 22:55:51 +0700 Subject: [PATCH 0220/3965] add tests for the listener --- p2p/transport/quic/listener_test.go | 105 ++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 p2p/transport/quic/listener_test.go diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go new file mode 100644 index 0000000000..9b1851a5e0 --- /dev/null +++ b/p2p/transport/quic/listener_test.go @@ -0,0 +1,105 @@ +package libp2pquic + +import ( + "errors" + "net" + + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +type mockNetListener struct { + connToAccept net.Conn + acceptErr error + closed bool +} + +func (m *mockNetListener) Accept() (net.Conn, error) { + if m.acceptErr != nil { + return nil, m.acceptErr + } + return m.connToAccept, nil +} + +func (m *mockNetListener) Close() error { + m.closed = true + return nil +} + +func (m *mockNetListener) Addr() net.Addr { + panic("not implemented") +} + +var _ net.Listener = &mockNetListener{} + +var _ = Describe("Listener", func() { + var ( + l *listener + netListener *mockNetListener + transport tpt.Transport + ) + + BeforeEach(func() { + netListener = &mockNetListener{} + transport = &QuicTransport{} + l = &listener{ + quicListener: netListener, + transport: transport, + } + }) + + It("returns its addr", func() { + laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/12345/quic") + Expect(err).ToNot(HaveOccurred()) + l, err = newListener(laddr, nil, nil) + Expect(err).ToNot(HaveOccurred()) + Expect(l.Addr().String()).To(Equal("127.0.0.1:12345")) + }) + + It("returns its multiaddr", func() { + laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/12346/quic") + Expect(err).ToNot(HaveOccurred()) + l, err = newListener(laddr, nil, nil) + Expect(err).ToNot(HaveOccurred()) + Expect(l.Multiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/12346/quic")) + }) + + It("closes", func() { + err := l.Close() + Expect(err).ToNot(HaveOccurred()) + Expect(netListener.closed).To(BeTrue()) + }) + + Context("accepting", func() { + It("accepts a new conn", func() { + remoteAddr, err := net.ResolveUDPAddr("udp", "127.0.0.1:1234") + Expect(err).ToNot(HaveOccurred()) + localAddr, err := net.ResolveUDPAddr("udp", "127.0.0.1:4321") + Expect(err).ToNot(HaveOccurred()) + udpConn, err := net.DialUDP("udp", localAddr, remoteAddr) + netListener.connToAccept = udpConn + conn, err := l.Accept() + Expect(err).ToNot(HaveOccurred()) + Expect(conn.LocalMultiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/4321")) + Expect(conn.RemoteMultiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/1234")) + Expect(conn.Transport()).To(Equal(transport)) + }) + + It("errors if it can't read the muliaddresses of a conn", func() { + netListener.connToAccept = &net.UDPConn{} + _, err := l.Accept() + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("nil multiaddr")) + }) + + It("errors if no connection can be accepted", func() { + testErr := errors.New("test error") + netListener.acceptErr = testErr + _, err := l.Accept() + Expect(err).To(MatchError(testErr)) + }) + }) +}) From 8ad1607e8071d35fdd4272f3d20d88e09584b780 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 18 Mar 2017 23:06:50 +0700 Subject: [PATCH 0221/3965] add tests for transport.Listen --- p2p/transport/quic/transport_test.go | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/p2p/transport/quic/transport_test.go b/p2p/transport/quic/transport_test.go index a6cd01255d..278f77129f 100644 --- a/p2p/transport/quic/transport_test.go +++ b/p2p/transport/quic/transport_test.go @@ -14,6 +14,28 @@ var _ = Describe("Transport", func() { t = NewQuicTransport(nil) }) + Context("listening", func() { + It("creates a new listener", func() { + maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") + Expect(err).ToNot(HaveOccurred()) + ln, err := t.Listen(maddr) + Expect(err).ToNot(HaveOccurred()) + Expect(ln.Multiaddr()).To(Equal(maddr)) + }) + + It("returns an existing listener", func() { + maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1235/quic") + Expect(err).ToNot(HaveOccurred()) + ln, err := t.Listen(maddr) + Expect(err).ToNot(HaveOccurred()) + Expect(ln.Multiaddr()).To(Equal(maddr)) + ln2, err := t.Listen(maddr) + Expect(err).ToNot(HaveOccurred()) + Expect(ln2).To(Equal(ln)) + Expect(t.listeners).To(HaveLen(1)) + }) + }) + It("matches", func() { invalidAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") Expect(err).ToNot(HaveOccurred()) From 4ebce2621bcdbef1340361813711bcc508f53632 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 18 Mar 2017 23:23:13 +0700 Subject: [PATCH 0222/3965] reject non-QUIC listen addresses --- p2p/transport/quic/transport.go | 6 +++++- p2p/transport/quic/transport_test.go | 7 +++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index cfd61f01b4..96465e11cd 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -1,6 +1,7 @@ package libp2pquic import ( + "fmt" "sync" pstore "github.com/libp2p/go-libp2p-peerstore" @@ -33,7 +34,10 @@ func (t *QuicTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dia // Listen starts listening on laddr func (t *QuicTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { - // TODO: check if laddr is actually a QUIC address + if !t.Matches(laddr) { + return nil, fmt.Errorf("quic transport cannot listen on %q", laddr) + } + t.mutex.Lock() defer t.mutex.Unlock() diff --git a/p2p/transport/quic/transport_test.go b/p2p/transport/quic/transport_test.go index 278f77129f..efe1b169e0 100644 --- a/p2p/transport/quic/transport_test.go +++ b/p2p/transport/quic/transport_test.go @@ -34,6 +34,13 @@ var _ = Describe("Transport", func() { Expect(ln2).To(Equal(ln)) Expect(t.listeners).To(HaveLen(1)) }) + + It("errors if the address is not a QUIC address", func() { + maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1235/utp") + Expect(err).ToNot(HaveOccurred()) + _, err = t.Listen(maddr) + Expect(err).To(MatchError("quic transport cannot listen on \"/ip4/127.0.0.1/udp/1235/utp\"")) + }) }) It("matches", func() { From 147f9baf98428f8bb19f177e51051490ce68a1fc Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 20 Mar 2017 14:53:30 +0700 Subject: [PATCH 0223/3965] initialize the listener with a valid TLS config MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The certificate doesn’t matter, since it is not validated anywehere (yet), but we need to send something. --- p2p/transport/quic/listener.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 9ef7386bd0..99eb96a5e7 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -1,11 +1,11 @@ package libp2pquic import ( - "crypto/tls" "net" pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" + testdata "github.com/lucas-clemente/quic-go/testdata" quicconn "github.com/marten-seemann/quic-conn" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" @@ -19,7 +19,9 @@ type listener struct { } func newListener(laddr ma.Multiaddr, peers pstore.Peerstore, transport tpt.Transport) (*listener, error) { - tlsConf := &tls.Config{} + // we need to provide a certificate here + // use the demo certificate from quic-go + tlsConf := testdata.GetTLSConfig() network, host, err := manet.DialArgs(laddr) if err != nil { return nil, err From 960bacbe0b3c902d120db466761528efef612ae6 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 20 Mar 2017 14:53:55 +0700 Subject: [PATCH 0224/3965] implement a dialer --- p2p/transport/quic/dialer.go | 41 +++++++++++++++++++--- p2p/transport/quic/dialer_test.go | 52 ++++++++++++++++++++++++++++ p2p/transport/quic/transport.go | 33 +++++++++++++++--- p2p/transport/quic/transport_test.go | 28 +++++++++++++++ 4 files changed, 144 insertions(+), 10 deletions(-) create mode 100644 p2p/transport/quic/dialer_test.go diff --git a/p2p/transport/quic/dialer.go b/p2p/transport/quic/dialer.go index a5536d4f49..f8e90ed93b 100644 --- a/p2p/transport/quic/dialer.go +++ b/p2p/transport/quic/dialer.go @@ -2,23 +2,54 @@ package libp2pquic import ( "context" + "crypto/tls" tpt "github.com/libp2p/go-libp2p-transport" + quicconn "github.com/marten-seemann/quic-conn" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" + "github.com/whyrusleeping/mafmt" ) -type dialer struct{} +type dialer struct { + transport tpt.Transport +} + +func newDialer(transport tpt.Transport) (*dialer, error) { + return &dialer{ + transport: transport, + }, nil +} func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { - panic("not implemented") + // TODO: check that raddr is a QUIC address + tlsConf := &tls.Config{InsecureSkipVerify: true} + _, host, err := manet.DialArgs(raddr) + if err != nil { + return nil, err + } + c, err := quicconn.Dial(host, tlsConf) + if err != nil { + return nil, err + } + + mnc, err := manet.WrapNetConn(c) + if err != nil { + return nil, err + } + + return &tpt.ConnWrap{ + Conn: mnc, + Tpt: d.transport, + }, nil } func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { - panic("not implemented") + return d.Dial(raddr) } -func (d *dialer) Matches(ma.Multiaddr) bool { - panic("not implemented") +func (d *dialer) Matches(a ma.Multiaddr) bool { + return mafmt.QUIC.Matches(a) } var _ tpt.Dialer = &dialer{} diff --git a/p2p/transport/quic/dialer_test.go b/p2p/transport/quic/dialer_test.go new file mode 100644 index 0000000000..f587cb65be --- /dev/null +++ b/p2p/transport/quic/dialer_test.go @@ -0,0 +1,52 @@ +package libp2pquic + +import ( + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +var _ = Describe("Listener", func() { + var ( + d *dialer + transport tpt.Transport + ) + + BeforeEach(func() { + var err error + transport = &QuicTransport{} + d, err = newDialer(transport) + Expect(err).ToNot(HaveOccurred()) + }) + + It("dials", func() { + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/8888") + Expect(err).ToNot(HaveOccurred()) + + // start a listener to connect to + var ln *listener + go func() { + defer GinkgoRecover() + ln, err = newListener(addr, nil, transport) + Expect(err).ToNot(HaveOccurred()) + _, err = ln.Accept() + Expect(err).ToNot(HaveOccurred()) + }() + + Eventually(func() *listener { return ln }).ShouldNot(BeNil()) + conn, err := d.Dial(addr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.Transport()).To(Equal(d.transport)) + }) + + It("matches", func() { + invalidAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") + Expect(err).ToNot(HaveOccurred()) + validAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") + Expect(err).ToNot(HaveOccurred()) + Expect(d.Matches(invalidAddr)).To(BeFalse()) + Expect(d.Matches(validAddr)).To(BeTrue()) + }) +}) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 96465e11cd..68968695f8 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -12,24 +12,47 @@ import ( // QuicTransport implements a QUIC Transport type QuicTransport struct { - mutex sync.Mutex - peers pstore.Peerstore + lmutex sync.Mutex listeners map[string]tpt.Listener + + dmutex sync.Mutex + dialers map[string]tpt.Dialer } // NewQuicTransport creates a new QUIC Transport // it tracks dialers and listeners created func NewQuicTransport(peers pstore.Peerstore) *QuicTransport { + // utils.SetLogLevel(utils.LogLevelDebug) return &QuicTransport{ peers: peers, listeners: make(map[string]tpt.Listener), + dialers: make(map[string]tpt.Dialer), } } func (t *QuicTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { - panic("not implemented") + if !t.Matches(laddr) { + return nil, fmt.Errorf("quic transport cannot dial %q", laddr) + } + + t.dmutex.Lock() + defer t.dmutex.Unlock() + + s := laddr.String() + d, ok := t.dialers[s] + if ok { + return d, nil + } + + // TODO: read opts + quicd, err := newDialer(t) + if err != nil { + return nil, err + } + t.dialers[s] = quicd + return quicd, nil } // Listen starts listening on laddr @@ -38,8 +61,8 @@ func (t *QuicTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { return nil, fmt.Errorf("quic transport cannot listen on %q", laddr) } - t.mutex.Lock() - defer t.mutex.Unlock() + t.lmutex.Lock() + defer t.lmutex.Unlock() l, ok := t.listeners[laddr.String()] if ok { diff --git a/p2p/transport/quic/transport_test.go b/p2p/transport/quic/transport_test.go index efe1b169e0..d7f8358c84 100644 --- a/p2p/transport/quic/transport_test.go +++ b/p2p/transport/quic/transport_test.go @@ -43,6 +43,34 @@ var _ = Describe("Transport", func() { }) }) + Context("dialing", func() { + It("creates a new dialer", func() { + maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") + Expect(err).ToNot(HaveOccurred()) + d, err := t.Dialer(maddr) + Expect(err).ToNot(HaveOccurred()) + Expect(d).ToNot(BeNil()) + }) + + It("returns an existing dialer", func() { + maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1235/quic") + Expect(err).ToNot(HaveOccurred()) + d, err := t.Dialer(maddr) + Expect(err).ToNot(HaveOccurred()) + d2, err := t.Dialer(maddr) + Expect(err).ToNot(HaveOccurred()) + Expect(d2).To(Equal(d)) + Expect(t.dialers).To(HaveLen(1)) + }) + + It("errors if the address is not a QUIC address", func() { + maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1235/utp") + Expect(err).ToNot(HaveOccurred()) + _, err = t.Dialer(maddr) + Expect(err).To(MatchError("quic transport cannot dial \"/ip4/127.0.0.1/udp/1235/utp\"")) + }) + }) + It("matches", func() { invalidAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") Expect(err).ToNot(HaveOccurred()) From e37704103a964f7e68b8680b4a2b4b1c17e22bb9 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 21 Mar 2017 16:59:49 -0700 Subject: [PATCH 0225/3965] fix ip4 vs ip6 dial failures --- p2p/transport/tcp/tcp.go | 23 ++++++++++++++++++++++- p2p/transport/tcp/tcp_test.go | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 1 deletion(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index ae069cce63..e38c2d1f53 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -124,6 +124,7 @@ type tcpDialer struct { rd reuseport.Dialer madialer manet.Dialer + pattern mafmt.Pattern transport tpt.Transport } @@ -135,6 +136,15 @@ func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReu return nil, err // something wrong with laddr. } + var pattern mafmt.Pattern + if TCP4.Matches(laddr) { + pattern = TCP4 + } else if TCP6.Matches(laddr) { + pattern = TCP6 + } else { + return nil, fmt.Errorf("local addr did not match TCP4 or TCP6: %s", laddr) + } + if doReuse && ReuseportIsAvailable() { rd := reuseport.Dialer{ D: net.Dialer{ @@ -149,6 +159,7 @@ func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReu rd: rd, madialer: base, transport: t, + pattern: pattern, }, nil } @@ -165,6 +176,13 @@ func (d *tcpDialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { } func (d *tcpDialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { + if raddr == nil { + zaddr, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") + if err != nil { + return nil, err + } + raddr = zaddr + } var c manet.Conn var err error if d.doReuse { @@ -214,8 +232,11 @@ func (d *tcpDialer) reuseDial(ctx context.Context, raddr ma.Multiaddr) (manet.Co return d.madialer.DialContext(ctx, raddr) } +var TCP4 = mafmt.And(mafmt.Base(ma.P_IP4), mafmt.Base(ma.P_TCP)) +var TCP6 = mafmt.And(mafmt.Base(ma.P_IP6), mafmt.Base(ma.P_TCP)) + func (d *tcpDialer) Matches(a ma.Multiaddr) bool { - return mafmt.TCP.Matches(a) + return d.pattern.Matches(a) } type tcpListener struct { diff --git a/p2p/transport/tcp/tcp_test.go b/p2p/transport/tcp/tcp_test.go index 8eeb5139f3..2a9a21b49d 100644 --- a/p2p/transport/tcp/tcp_test.go +++ b/p2p/transport/tcp/tcp_test.go @@ -3,6 +3,7 @@ package tcp import ( "testing" + tpt "github.com/libp2p/go-libp2p-transport" utils "github.com/libp2p/go-libp2p-transport/test" ma "github.com/multiformats/go-multiaddr" ) @@ -27,3 +28,34 @@ func TestTcpTransportCantListenUtp(t *testing.T) { t.Fatal("shouldnt be able to listen on utp addr with tcp transport") } } + +func TestCorrectIPVersionMatching(t *testing.T) { + ta := NewTCPTransport() + + addr4, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") + if err != nil { + t.Fatal(err) + } + addr6, err := ma.NewMultiaddr("/ip6/::1/tcp/0") + if err != nil { + t.Fatal(err) + } + + d4, err := ta.Dialer(addr4, tpt.ReuseportOpt(true)) + if err != nil { + t.Fatal(err) + } + + d6, err := ta.Dialer(addr6, tpt.ReuseportOpt(true)) + if err != nil { + t.Fatal(err) + } + + if d4.Matches(addr6) { + t.Fatal("tcp4 dialer should not match ipv6 address") + } + + if d6.Matches(addr4) { + t.Fatal("tcp4 dialer should not match ipv6 address") + } +} From cfef65cee378689dc823ed537d14ca17459ed356 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 21 Mar 2017 17:01:26 -0700 Subject: [PATCH 0226/3965] allow nil for dialer laddr --- p2p/transport/tcp/tcp.go | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index e38c2d1f53..0efa3ffe4e 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -36,6 +36,13 @@ func NewTCPTransport() *TcpTransport { } func (t *TcpTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { + if laddr == nil { + zaddr, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") + if err != nil { + return nil, err + } + laddr = zaddr + } t.dlock.Lock() defer t.dlock.Unlock() s := laddr.String() @@ -176,13 +183,6 @@ func (d *tcpDialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { } func (d *tcpDialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { - if raddr == nil { - zaddr, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") - if err != nil { - return nil, err - } - raddr = zaddr - } var c manet.Conn var err error if d.doReuse { From 59530f21f48196535bdbd056a81f49f29466e44e Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 21 Mar 2017 17:08:53 -0700 Subject: [PATCH 0227/3965] clean up reusedial and remove go-libp2p-logging --- p2p/transport/tcp/tcp.go | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 0efa3ffe4e..908229d0bc 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -9,7 +9,6 @@ import ( logging "github.com/ipfs/go-log" reuseport "github.com/jbenet/go-reuseport" - lgbl "github.com/libp2p/go-libp2p-loggables" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" @@ -202,33 +201,27 @@ func (d *tcpDialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Co } func (d *tcpDialer) reuseDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { - logdial := lgbl.Dial("conn", "", "", d.laddr, raddr) - rpev := log.EventBegin(ctx, "tptDialReusePort", logdial) - network, netraddr, err := manet.DialArgs(raddr) if err != nil { return nil, err } - _ = ctx // TODO: implement DialContext in reuseport + rpev := log.EventBegin(ctx, "tptDialReusePort", logging.LoggableMap{ + "raddr": raddr, + }) + con, err := d.rd.DialContext(ctx, network, netraddr) if err == nil { - logdial["reuseport"] = "success" rpev.Done() return manet.WrapNetConn(con) } + rpev.SetError(err) + rpev.Done() if !ReuseErrShouldRetry(err) { - logdial["reuseport"] = "failure" - logdial["error"] = err - rpev.Done() return nil, err } - logdial["reuseport"] = "retry" - logdial["error"] = err - rpev.Done() - return d.madialer.DialContext(ctx, raddr) } From ad5c86672f61b0e84657897bcea876226166a80a Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 23 Mar 2017 18:57:39 -0700 Subject: [PATCH 0228/3965] remove go-ipfs-util dep --- p2p/host/peerstore/queue/queue_test.go | 4 ++-- p2p/host/peerstore/test/utils.go | 20 +++++++++++++++----- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index cd3c80de12..e30794b0cf 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,8 +7,8 @@ import ( "testing" "time" - u "github.com/ipfs/go-ipfs-util" peer "github.com/libp2p/go-libp2p-peer" + mh "github.com/multiformats/go-multihash" ) func TestQueue(t *testing.T) { @@ -64,7 +64,7 @@ func TestQueue(t *testing.T) { func newPeerTime(t time.Time) peer.ID { s := fmt.Sprintf("hmmm time: %v", t) - h := u.Hash([]byte(s)) + h, _ := mh.Sum([]byte(s), mh.SHA2_256, -1) return peer.ID(h) } diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index 6c4f0be5fc..fc141b0d6c 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -2,25 +2,35 @@ package testutil import ( "io" + "math/rand" + "time" - u "github.com/ipfs/go-ipfs-util" ci "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" + mh "github.com/multiformats/go-multihash" ) +func timeSeededRand() io.Reader { + return rand.New(rand.NewSource(time.Now().UnixNano())) +} + func RandPeerID() (peer.ID, error) { buf := make([]byte, 16) - if _, err := io.ReadFull(u.NewTimeSeededRand(), buf); err != nil { + if _, err := io.ReadFull(timeSeededRand(), buf); err != nil { return "", err } - h := u.Hash(buf) + h, err := mh.Sum(buf, mh.SHA2_256, -1) + if err != nil { + return "", err + } + return peer.ID(h), nil } func RandTestKeyPair(bits int) (ci.PrivKey, ci.PubKey, error) { - return ci.GenerateKeyPairWithReader(ci.RSA, bits, u.NewTimeSeededRand()) + return ci.GenerateKeyPairWithReader(ci.RSA, bits, timeSeededRand()) } func SeededTestKeyPair(seed int64) (ci.PrivKey, ci.PubKey, error) { - return ci.GenerateKeyPairWithReader(ci.RSA, 512, u.NewSeededRand(seed)) + return ci.GenerateKeyPairWithReader(ci.RSA, 512, rand.New(rand.NewSource(seed))) } From 53da0c05ceeb11e00c72639584447eb99b23c8e8 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Thu, 23 Mar 2017 22:35:08 -0700 Subject: [PATCH 0229/3965] fix panic when reuseport was no available --- p2p/transport/tcp/tcp.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 908229d0bc..a30dcbb7b0 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -172,6 +172,7 @@ func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReu return &tcpDialer{ doReuse: false, laddr: laddr, + pattern: pattern, madialer: base, transport: t, }, nil From dc8b46a6582c05e59affd343721591bae51a74f5 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 29 Mar 2017 15:05:08 -0700 Subject: [PATCH 0230/3965] first commit, basic test works. Now for some integration work --- .../internal/circuitv1-deprecated/conn.go | 87 +++++++ .../internal/circuitv1-deprecated/relay.go | 217 ++++++++++++++++++ .../circuitv1-deprecated/relay_test.go | 106 +++++++++ .../internal/circuitv1-deprecated/status.go | 62 +++++ .../circuitv1-deprecated/status_test.go | 32 +++ .../circuitv1-deprecated/transport.go | 75 ++++++ .../internal/circuitv1-deprecated/util.go | 63 +++++ .../circuitv1-deprecated/util_test.go | 44 ++++ 8 files changed, 686 insertions(+) create mode 100644 p2p/protocol/internal/circuitv1-deprecated/conn.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/relay.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/relay_test.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/status.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/status_test.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/transport.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/util.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/util_test.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go new file mode 100644 index 0000000000..73a66b78d0 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -0,0 +1,87 @@ +package relay + +import ( + "fmt" + "net" + + ic "github.com/libp2p/go-libp2p-crypto" + iconn "github.com/libp2p/go-libp2p-interface-conn" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +type Conn struct { + inet.Stream + remoteMaddr ma.Multiaddr + remotePeer peer.ID +} + +var _ iconn.Conn = (*Conn)(nil) + +type NetAddr struct { + Relay string + Remote string +} + +func (n *NetAddr) Network() string { + return "libp2p-circuit-relay" +} + +func (n *NetAddr) String() string { + return fmt.Sprintf("relay[%s-%s]", n.Remote, n.Relay) +} + +func (c *Conn) RemoteAddr() net.Addr { + return &NetAddr{ + Relay: c.Conn().RemotePeer().Pretty(), + Remote: c.remotePeer.String(), + } +} + +func (c *Conn) RemoteMultiaddr() ma.Multiaddr { + a, err := ma.NewMultiaddr(fmt.Sprintf("%s/ipfs/%s/p2p-circuit/%s", c.remoteMaddr, c.remotePeer.Pretty(), c.Conn().RemotePeer())) + if err != nil { + panic(err) + } + return a +} + +func (c *Conn) LocalMultiaddr() ma.Multiaddr { + return c.Conn().LocalMultiaddr() +} + +func (c *Conn) LocalAddr() net.Addr { + na, err := manet.ToNetAddr(c.Conn().LocalMultiaddr()) + if err != nil { + log.Error("failed to convert local multiaddr to net addr:", err) + return nil + } + return na +} + +func (c *Conn) Transport() tpt.Transport { + panic("does anyone really call this?") +} + +func (c *Conn) LocalPeer() peer.ID { + return c.Conn().LocalPeer() +} + +func (c *Conn) RemotePeer() peer.ID { + return c.remotePeer +} + +func (c *Conn) LocalPrivateKey() ic.PrivKey { + return nil +} + +func (c *Conn) RemotePublicKey() ic.PubKey { + return nil +} + +func (c *Conn) ID() string { + return "TODO: relay conn ID" +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go new file mode 100644 index 0000000000..d4229567ae --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -0,0 +1,217 @@ +package relay + +import ( + "context" + "fmt" + "io" + "sync" + "time" + + logging "github.com/ipfs/go-log" + host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" +) + +var log = logging.Logger("relay") + +const HopID = "/libp2p/relay/circuit/1.0.0/hop" +const StopID = "/libp2p/relay/circuit/1.0.0/stop" + +const maxAddrLen = 1024 + +var RelayAcceptTimeout = time.Minute + +type Relay struct { + host host.Host + ctx context.Context + self peer.ID + + active bool + + incoming chan *Conn + + arLk sync.Mutex + activeRelays []*Conn +} + +type RelayOpt int + +var ( + OptActive = RelayOpt(0) + OptHop = RelayOpt(1) +) + +func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error) { + r := &Relay{ + host: h, + ctx: ctx, + self: h.ID(), + incoming: make(chan *Conn), + } + + h.SetStreamHandler(StopID, r.HandleNewStopStream) + + for _, opt := range opts { + switch opt { + case OptActive: + r.active = true + case OptHop: + h.SetStreamHandler(HopID, r.HandleNewHopStream) + default: + return nil, fmt.Errorf("unrecognized option: %d", opt) + } + } + + return r, nil +} + +func (r *Relay) Dial(ctx context.Context, relay peer.ID, dest ma.Multiaddr) (*Conn, error) { + s, err := r.host.NewStream(ctx, relay, HopID) + if err != nil { + return nil, err + } + + if err := writeLpMultiaddr(s, dest); err != nil { + return nil, err + } + + var stat RelayStatus + if err := stat.ReadFrom(s); err != nil { + return nil, err + } + + if stat.Code != StatusOK { + return nil, &stat + } + + return &Conn{Stream: s}, nil +} + +func (r *Relay) HandleNewStopStream(s inet.Stream) { + log.Infof("new stop stream from: %s", s.Conn().RemotePeer()) + status := r.handleNewStopStream(s) + if status != nil { + if err := status.WriteTo(s); err != nil { + log.Info("problem writing error status:", err) + } + s.Close() + return + } +} + +func (r *Relay) handleNewStopStream(s inet.Stream) *RelayStatus { + info, err := r.readInfo(s) + if err != nil { + return &RelayStatus{ + Code: StatusDstAddrErr, + Message: err.Error(), + } + } + + log.Infof("relay connection from: %s", info.ID) + select { + case r.incoming <- &Conn{Stream: s, remoteMaddr: info.Addrs[0], remotePeer: info.ID}: + return nil + case <-time.After(RelayAcceptTimeout): + return &RelayStatus{ + Code: StatusDstRelayRefused, + Message: "timed out waiting for relay to be accepted", + } + } +} + +func (r *Relay) HandleNewHopStream(s inet.Stream) { + log.Infof("new hop stream from: %s", s.Conn().RemotePeer()) + status := r.handleNewHopStream(s) + if status != nil { + if err := status.WriteTo(s); err != nil { + log.Debugf("problem writing error status back: %s", err) + s.Close() + return + } + } +} + +func (r *Relay) handleNewHopStream(s inet.Stream) *RelayStatus { + info, err := r.readInfo(s) + if err != nil { + return &RelayStatus{ + Code: StatusRelayAddrErr, + Message: err.Error(), + } + } + + if info.ID == r.self { + return &RelayStatus{ + Code: StatusRelayHopToSelf, + Message: "relay hop attempted to self", + } + } + + ctp := r.host.Network().ConnsToPeer(info.ID) + if len(ctp) == 0 { + return &RelayStatus{ + Code: StatusRelayNotConnected, + Message: "refusing to make new connection for relay", + } + } + + bs, err := r.host.NewStream(r.ctx, info.ID, StopID) + if err != nil { + return &RelayStatus{ + Code: StatusRelayStreamFailed, + Message: err.Error(), + } + } + + // TODO: add helper method 'PeerID to multiaddr' + paddr, err := ma.NewMultiaddr("/ipfs/" + s.Conn().RemotePeer().Pretty()) + if err != nil { + return &RelayStatus{ + Code: StatusRelayAddrErr, + Message: err.Error(), + } + } + + p2pa := s.Conn().RemoteMultiaddr().Encapsulate(paddr) + if err := writeLpMultiaddr(bs, p2pa); err != nil { + return &RelayStatus{ + Code: StatusRelayStreamFailed, + Message: err.Error(), + } + } + + go func() { + _, err := io.Copy(s, bs) + if err != io.EOF && err != nil { + log.Debugf("relay copy error: %s", err) + } + s.Close() + }() + go func() { + _, err := io.Copy(bs, s) + if err != io.EOF && err != nil { + log.Debugf("relay copy error: %s", err) + } + bs.Close() + }() + + return nil +} + +func (r *Relay) readInfo(s inet.Stream) (*pstore.PeerInfo, error) { + addr, err := readLpMultiaddr(s) + if err != nil { + return nil, err + } + + info, err := pstore.InfoFromP2pAddr(addr) + if err != nil { + return nil, err + } + + return info, nil +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go new file mode 100644 index 0000000000..ba5e867ef3 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -0,0 +1,106 @@ +package relay + +import ( + "bytes" + "context" + "io/ioutil" + "testing" + + bhost "github.com/libp2p/go-libp2p-blankhost" + host "github.com/libp2p/go-libp2p-host" + netutil "github.com/libp2p/go-libp2p-netutil" + ma "github.com/multiformats/go-multiaddr" +) + +/* TODO: add tests +- simple A -[R]-> B +- A tries to relay through R, R doesnt support relay +- A tries to relay through R to B, B doesnt support relay +- A sends invalid multiaddr +- A sends too long multiaddr +- R drops stream mid-message +- A relays through R, R has no connection to B +*/ + +func getNetHosts(t *testing.T, ctx context.Context, n int) []host.Host { + var out []host.Host + + for i := 0; i < n; i++ { + netw := netutil.GenSwarmNetwork(t, ctx) + h := bhost.NewBlankHost(netw) + out = append(out, h) + } + + return out +} + +func connect(t *testing.T, a, b host.Host) { + pinfo := a.Peerstore().PeerInfo(a.ID()) + err := b.Connect(context.Background(), pinfo) + if err != nil { + t.Fatal(err) + } +} + +func TestBasicRelay(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 3) + + connect(t, hosts[0], hosts[1]) + connect(t, hosts[1], hosts[2]) + + r1, err := NewRelay(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[1], OptHop) + if err != nil { + t.Fatal(err) + } + + r3, err := NewRelay(ctx, hosts[2]) + if err != nil { + t.Fatal(err) + } + + msg := []byte("relay works!") + go func() { + list, err := r3.Listener() + if err != nil { + t.Error(err) + } + + con, err := list.Accept() + if err != nil { + t.Error(err) + } + + _, err = con.Write(msg) + if err != nil { + t.Error("failed to write", err) + } + con.Close() + }() + + destma, err := ma.NewMultiaddr("/ipfs/" + hosts[2].ID().Pretty()) + if err != nil { + t.Fatal(err) + } + + con, err := r1.Dial(ctx, hosts[1].ID(), destma) + if err != nil { + t.Fatal(err) + } + + data, err := ioutil.ReadAll(con) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(data, msg) { + t.Fatal("message was incorrect:", string(data)) + } +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/status.go b/p2p/protocol/internal/circuitv1-deprecated/status.go new file mode 100644 index 0000000000..8d9c2582ed --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/status.go @@ -0,0 +1,62 @@ +package relay + +import ( + "encoding/binary" + "fmt" + "io" +) + +const maxStatusMessageLength = 1024 + +const ( + StatusOK = 100 + + StatusRelayAddrErr = 250 + StatusRelayNotConnected = 260 + StatusRelayDialFailed = 261 + StatusRelayStreamFailed = 262 + StatusRelayHopToSelf = 270 + + StatusDstAddrErr = 350 + StatusDstRelayRefused = 380 + StatusDstWrongDst = 381 +) + +type RelayStatus struct { + Code uint64 + Message string +} + +func (s *RelayStatus) WriteTo(w io.Writer) error { + outbuf := make([]byte, 2*binary.MaxVarintLen64+len(s.Message)) + n := binary.PutUvarint(outbuf, s.Code) + n += binary.PutUvarint(outbuf[n:], uint64(len(s.Message))) + n += copy(outbuf[n:], s.Message) + _, err := w.Write(outbuf[:n]) + return err +} + +func (s *RelayStatus) ReadFrom(r io.Reader) error { + br := &singleByteReader{r} + code, err := binary.ReadUvarint(br) + if err != nil { + return err + } + l, err := binary.ReadUvarint(br) + if err != nil { + return err + } + buf := make([]byte, l) + _, err = io.ReadFull(r, buf) + if err != nil { + return err + } + + s.Code = code + s.Message = string(buf) + return nil +} + +func (s *RelayStatus) Error() string { + return fmt.Sprintf("%d: %s", s.Code, s.Message) +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/status_test.go b/p2p/protocol/internal/circuitv1-deprecated/status_test.go new file mode 100644 index 0000000000..25fe4e0f32 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/status_test.go @@ -0,0 +1,32 @@ +package relay + +import ( + "bytes" + "testing" +) + +func TestStatusParsing(t *testing.T) { + s := &RelayStatus{ + Code: StatusDstAddrErr, + Message: "foo bar", + } + + buf := new(bytes.Buffer) + + if err := s.WriteTo(buf); err != nil { + t.Fatal(err) + } + + var ns RelayStatus + if err := ns.ReadFrom(buf); err != nil { + t.Fatal(err) + } + + if ns.Code != s.Code { + t.Fatal("codes didnt match") + } + + if ns.Message != s.Message { + t.Fatal("messages didnt match") + } +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go new file mode 100644 index 0000000000..b1c1baee40 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -0,0 +1,75 @@ +package relay + +import ( + "net" + + peer "github.com/libp2p/go-libp2p-peer" + tpt "github.com/libp2p/go-libp2p-transport" + filter "github.com/libp2p/go-maddr-filter" + ma "github.com/multiformats/go-multiaddr" +) + +var RelayMaddrProtocol = ma.Protocol{ + Code: 290, + Name: "p2p-circuit", + Size: 0, +} + +func init() { + ma.AddProtocol(RelayMaddrProtocol) +} + +var _ tpt.Listener = (*RelayListener)(nil) + +type RelayListener Relay + +func (l *RelayListener) Relay() *Relay { + return (*Relay)(l) +} + +func (r *Relay) Listener() (tpt.Listener, error) { + return (*RelayListener)(r), nil +} + +func (r *Relay) Matches(a ma.Multiaddr) bool { + return false +} + +func (l *RelayListener) Accept() (tpt.Conn, error) { + ctx := l.Relay().ctx + select { + case c := <-l.incoming: + log.Infof("accepted relay connection: %s", c.ID()) + s := RelayStatus{ + Code: StatusOK, + Message: "OK", + } + if err := s.WriteTo(c); err != nil { + return nil, err + } + return c, nil + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +func (l *RelayListener) Addr() net.Addr { + panic("oh no") +} + +func (l *RelayListener) Multiaddr() ma.Multiaddr { + panic("oh no") +} + +func (l *RelayListener) LocalPeer() peer.ID { + return l.Relay().self +} + +func (l *RelayListener) SetAddrFilters(f *filter.Filters) { + // noop ? +} + +func (l *RelayListener) Close() error { + // TODO: noop? + return nil +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go new file mode 100644 index 0000000000..15726d3d3d --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -0,0 +1,63 @@ +package relay + +import ( + "encoding/binary" + "fmt" + "io" + + ma "github.com/multiformats/go-multiaddr" +) + +type singleByteReader struct { + r io.Reader +} + +func (s *singleByteReader) ReadByte() (byte, error) { + var b [1]byte + n, err := s.r.Read(b[:]) + if err != nil { + return 0, err + } + if n == 0 { + return 0, io.ErrNoProgress + } + + return b[0], nil +} + +func writeLpMultiaddr(w io.Writer, a ma.Multiaddr) error { + buf := make([]byte, binary.MaxVarintLen32+len(a.Bytes())) + n := binary.PutUvarint(buf, uint64(len(a.Bytes()))) + n += copy(buf[n:], a.Bytes()) + nw, err := w.Write(buf[:n]) + if err != nil { + return err + } + if n != nw { + return fmt.Errorf("failed to write all bytes to writer") + } + return nil +} + +func readLpMultiaddr(r io.Reader) (ma.Multiaddr, error) { + l, err := binary.ReadUvarint(&singleByteReader{r}) + if err != nil { + return nil, err + } + + if l > maxAddrLen { + return nil, fmt.Errorf("address length was too long: %d > %d", l, maxAddrLen) + } + + if l == 0 { + return nil, fmt.Errorf("zero length multiaddr is invalid") + } + + buf := make([]byte, l) + _, err = io.ReadFull(r, buf) + if err != nil { + return nil, err + } + + return ma.NewMultiaddrBytes(buf) +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/util_test.go b/p2p/protocol/internal/circuitv1-deprecated/util_test.go new file mode 100644 index 0000000000..b0f3f68cd6 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/util_test.go @@ -0,0 +1,44 @@ +package relay + +import ( + "bytes" + "testing" + + ma "github.com/multiformats/go-multiaddr" +) + +func TestMultiaddrSerialization(t *testing.T) { + buf := new(bytes.Buffer) + + addr, err := ma.NewMultiaddr("/ip4/1.2.3.4/tcp/1234") + if err != nil { + t.Fatal(err) + } + + if err := writeLpMultiaddr(buf, addr); err != nil { + t.Fatal(err) + } + + out, err := readLpMultiaddr(buf) + if err != nil { + t.Fatal(err) + } + + if !addr.Equal(out) { + t.Fatal("addresses didnt match") + } +} + +func TestDecodeInvalid(t *testing.T) { + buf := bytes.NewBuffer([]byte{72, 0, 0}) + _, err := readLpMultiaddr(buf) + if err == nil { + t.Fatal("shouldnt have parsed correctly") + } + + buf = bytes.NewBuffer([]byte{0}) + _, err = readLpMultiaddr(buf) + if err == nil { + t.Fatal("shouldnt have parsed correctly") + } +} From 511c02bf9c4848279337c3cac4d92026f47ec06c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 31 Mar 2017 16:24:53 -0700 Subject: [PATCH 0231/3965] more tests --- .../circuitv1-deprecated/relay_test.go | 103 +++++++++++++++++- 1 file changed, 102 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index ba5e867ef3..c44416542b 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -3,7 +3,9 @@ package relay import ( "bytes" "context" + "fmt" "io/ioutil" + "strings" "testing" bhost "github.com/libp2p/go-libp2p-blankhost" @@ -16,7 +18,6 @@ import ( - simple A -[R]-> B - A tries to relay through R, R doesnt support relay - A tries to relay through R to B, B doesnt support relay -- A sends invalid multiaddr - A sends too long multiaddr - R drops stream mid-message - A relays through R, R has no connection to B @@ -71,16 +72,19 @@ func TestBasicRelay(t *testing.T) { list, err := r3.Listener() if err != nil { t.Error(err) + return } con, err := list.Accept() if err != nil { t.Error(err) + return } _, err = con.Write(msg) if err != nil { t.Error("failed to write", err) + return } con.Close() }() @@ -104,3 +108,100 @@ func TestBasicRelay(t *testing.T) { t.Fatal("message was incorrect:", string(data)) } } + +func TestRelayThroughNonHop(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 3) + + connect(t, hosts[0], hosts[1]) + connect(t, hosts[1], hosts[2]) + + r1, err := NewRelay(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[1]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[2]) + if err != nil { + t.Fatal(err) + } + + destma, err := ma.NewMultiaddr("/ipfs/" + hosts[2].ID().Pretty()) + if err != nil { + t.Fatal(err) + } + + _, err = r1.Dial(ctx, hosts[1].ID(), destma) + if err.Error() != "protocol not supported" { + t.Fatal("expected 'protocol not supported' error") + } +} + +func TestDestNoRelay(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 3) + + connect(t, hosts[0], hosts[1]) + connect(t, hosts[1], hosts[2]) + + r1, err := NewRelay(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[1], OptHop) + if err != nil { + t.Fatal(err) + } + + destma, err := ma.NewMultiaddr("/ipfs/" + hosts[2].ID().Pretty()) + if err != nil { + t.Fatal(err) + } + for i := 0; i < 10; i++ { + destma = ma.Join(destma, destma) + } + + _, err = r1.Dial(ctx, hosts[1].ID(), destma) + if !strings.HasPrefix(err.Error(), fmt.Sprintf("%d: address length was too long", StatusRelayAddrErr)) { + t.Fatal(err) + } +} + +func TestRelayNoDestConnection(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 3) + + connect(t, hosts[0], hosts[1]) + + r1, err := NewRelay(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[1], OptHop) + if err != nil { + t.Fatal(err) + } + + destma, err := ma.NewMultiaddr("/ipfs/" + hosts[2].ID().Pretty()) + if err != nil { + t.Fatal(err) + } + + _, err = r1.Dial(ctx, hosts[1].ID(), destma) + if err.Error() != "260: refusing to make new connection for relay" { + t.Fatal("expected this not to work") + } +} From ec6366ce59a06cce8605b5f1ad05b4b426883b0c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 31 Mar 2017 22:06:44 -0700 Subject: [PATCH 0232/3965] add dialer interface --- .../internal/circuitv1-deprecated/dial.go | 60 ++++++++++++++++ .../internal/circuitv1-deprecated/listen.go | 65 +++++++++++++++++ .../circuitv1-deprecated/relay_test.go | 70 ++++++++++++++++++- .../circuitv1-deprecated/transport.go | 64 +---------------- 4 files changed, 197 insertions(+), 62 deletions(-) create mode 100644 p2p/protocol/internal/circuitv1-deprecated/dial.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/listen.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go new file mode 100644 index 0000000000..1d2f103575 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -0,0 +1,60 @@ +package relay + +import ( + "context" + "fmt" + + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" +) + +type Dialer Relay + +func (d *Dialer) Relay() *Relay { + return (*Relay)(d) +} + +func (r *Relay) Dialer() *Dialer { + return (*Dialer)(r) +} + +func (d *Dialer) DialPeer(ctx context.Context, p peer.ID, a ma.Multiaddr) (tpt.Conn, error) { + if !d.Matches(a) { + return nil, fmt.Errorf("%s is not a relay address", a) + } + parts := ma.Split(a) + + spl, _ := ma.NewMultiaddr("/p2p-circuit") + + var relayaddr, destaddr ma.Multiaddr + for i, p := range parts { + if p.Equal(spl) { + relayaddr = ma.Join(parts[:i]...) + destaddr = ma.Join(parts[i+1:]...) + break + } + } + + rinfo, err := pstore.InfoFromP2pAddr(relayaddr) + if err != nil { + return nil, err + } + + destp2p, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s", p.Pretty())) + if err != nil { + return nil, err + } + + destp2p = destaddr.Encapsulate(destp2p) + + d.Relay().host.Peerstore().AddAddrs(rinfo.ID, rinfo.Addrs, pstore.TempAddrTTL) + + return d.Relay().Dial(ctx, rinfo.ID, destp2p) +} + +func (d *Dialer) Matches(a ma.Multiaddr) bool { + _, err := a.ValueForProtocol(P_CIRCUIT) + return err == nil +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go new file mode 100644 index 0000000000..1c73a65417 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -0,0 +1,65 @@ +package relay + +import ( + "net" + + peer "github.com/libp2p/go-libp2p-peer" + tpt "github.com/libp2p/go-libp2p-transport" + filter "github.com/libp2p/go-maddr-filter" + ma "github.com/multiformats/go-multiaddr" +) + +var _ tpt.Listener = (*RelayListener)(nil) + +type RelayListener Relay + +func (l *RelayListener) Relay() *Relay { + return (*Relay)(l) +} + +func (r *Relay) Listener() (tpt.Listener, error) { + return (*RelayListener)(r), nil +} + +func (r *Relay) Matches(a ma.Multiaddr) bool { + return false +} + +func (l *RelayListener) Accept() (tpt.Conn, error) { + ctx := l.Relay().ctx + select { + case c := <-l.incoming: + log.Infof("accepted relay connection: %s", c.ID()) + s := RelayStatus{ + Code: StatusOK, + Message: "OK", + } + if err := s.WriteTo(c); err != nil { + return nil, err + } + return c, nil + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +func (l *RelayListener) Addr() net.Addr { + panic("oh no") +} + +func (l *RelayListener) Multiaddr() ma.Multiaddr { + panic("oh no") +} + +func (l *RelayListener) LocalPeer() peer.ID { + return l.Relay().self +} + +func (l *RelayListener) SetAddrFilters(f *filter.Filters) { + // noop ? +} + +func (l *RelayListener) Close() error { + // TODO: noop? + return nil +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index c44416542b..cb1a855021 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -1,4 +1,4 @@ -package relay +package relay_test import ( "bytes" @@ -9,6 +9,7 @@ import ( "testing" bhost "github.com/libp2p/go-libp2p-blankhost" + . "github.com/libp2p/go-libp2p-circuit" host "github.com/libp2p/go-libp2p-host" netutil "github.com/libp2p/go-libp2p-netutil" ma "github.com/multiformats/go-multiaddr" @@ -109,6 +110,73 @@ func TestBasicRelay(t *testing.T) { } } +func TestBasicRelayDial(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 3) + + connect(t, hosts[0], hosts[1]) + connect(t, hosts[1], hosts[2]) + + r1, err := NewRelay(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[1], OptHop) + if err != nil { + t.Fatal(err) + } + + r3, err := NewRelay(ctx, hosts[2]) + if err != nil { + t.Fatal(err) + } + + msg := []byte("relay works!") + go func() { + list, err := r3.Listener() + if err != nil { + t.Error(err) + return + } + + con, err := list.Accept() + if err != nil { + t.Error(err) + return + } + + _, err = con.Write(msg) + if err != nil { + t.Error("failed to write", err) + return + } + con.Close() + }() + + relayaddr, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit", hosts[1].ID().Pretty())) + if err != nil { + t.Fatal(err) + } + + d := r1.Dialer() + con, err := d.DialPeer(ctx, hosts[2].ID(), relayaddr) + if err != nil { + t.Fatal(err) + } + + data, err := ioutil.ReadAll(con) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(data, msg) { + t.Fatal("message was incorrect:", string(data)) + } +} + func TestRelayThroughNonHop(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index b1c1baee40..24c09fd874 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -1,16 +1,13 @@ package relay import ( - "net" - - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" - filter "github.com/libp2p/go-maddr-filter" ma "github.com/multiformats/go-multiaddr" ) +const P_CIRCUIT = 290 + var RelayMaddrProtocol = ma.Protocol{ - Code: 290, + Code: P_CIRCUIT, Name: "p2p-circuit", Size: 0, } @@ -18,58 +15,3 @@ var RelayMaddrProtocol = ma.Protocol{ func init() { ma.AddProtocol(RelayMaddrProtocol) } - -var _ tpt.Listener = (*RelayListener)(nil) - -type RelayListener Relay - -func (l *RelayListener) Relay() *Relay { - return (*Relay)(l) -} - -func (r *Relay) Listener() (tpt.Listener, error) { - return (*RelayListener)(r), nil -} - -func (r *Relay) Matches(a ma.Multiaddr) bool { - return false -} - -func (l *RelayListener) Accept() (tpt.Conn, error) { - ctx := l.Relay().ctx - select { - case c := <-l.incoming: - log.Infof("accepted relay connection: %s", c.ID()) - s := RelayStatus{ - Code: StatusOK, - Message: "OK", - } - if err := s.WriteTo(c); err != nil { - return nil, err - } - return c, nil - case <-ctx.Done(): - return nil, ctx.Err() - } -} - -func (l *RelayListener) Addr() net.Addr { - panic("oh no") -} - -func (l *RelayListener) Multiaddr() ma.Multiaddr { - panic("oh no") -} - -func (l *RelayListener) LocalPeer() peer.ID { - return l.Relay().self -} - -func (l *RelayListener) SetAddrFilters(f *filter.Filters) { - // noop ? -} - -func (l *RelayListener) Close() error { - // TODO: noop? - return nil -} From 7047cea01a994183b90f53599e58da988baea73d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 31 Mar 2017 23:08:23 -0700 Subject: [PATCH 0233/3965] add method to parse /ipfs/ addresses into a peerinfo --- p2p/host/peerstore/peerinfo.go | 34 +++++++++++++++++++++++++++++ p2p/host/peerstore/peerinfo_test.go | 27 +++++++++++++++++++++++ 2 files changed, 61 insertions(+) diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index a42d5ef006..3226a50972 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -2,6 +2,8 @@ package peerstore import ( "encoding/json" + "fmt" + "strings" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" @@ -16,6 +18,38 @@ type PeerInfo struct { Addrs []ma.Multiaddr } +var ErrInvalidAddr = fmt.Errorf("invalid p2p multiaddr") + +func InfoFromP2pAddr(m ma.Multiaddr) (*PeerInfo, error) { + if m == nil { + return nil, ErrInvalidAddr + } + + // make sure it's an IPFS addr + parts := ma.Split(m) + if len(parts) < 1 { + return nil, ErrInvalidAddr + } + + ipfspart := parts[len(parts)-1] // last part + if ipfspart.Protocols()[0].Code != ma.P_IPFS { + return nil, ErrInvalidAddr + } + + // make sure 'ipfs id' parses as a peer.ID + peerIdParts := strings.Split(ipfspart.String(), "/") + peerIdStr := peerIdParts[len(peerIdParts)-1] + id, err := peer.IDB58Decode(peerIdStr) + if err != nil { + return nil, err + } + + return &PeerInfo{ + ID: id, + Addrs: []ma.Multiaddr{ma.Join(parts[:len(parts)-1]...)}, + }, nil +} + func (pi *PeerInfo) Loggable() map[string]interface{} { return map[string]interface{}{ "peerID": pi.ID.Pretty(), diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index ca73430a18..edad247c38 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -56,3 +56,30 @@ func TestPeerInfoMarshal(t *testing.T) { t.Fatal("loggables gave wrong peerID output") } } + +func TestP2pAddrParsing(t *testing.T) { + a := mustAddr(t, "/ip4/1.2.3.4/tcp/4536") + id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") + if err != nil { + t.Fatal(err) + } + + p2pa := a.String() + "/ipfs/" + id.Pretty() + p2pma, err := ma.NewMultiaddr(p2pa) + if err != nil { + t.Fatal(err) + } + + pinfo, err := InfoFromP2pAddr(p2pma) + if err != nil { + t.Fatal(err) + } + + if pinfo.ID != id { + t.Fatal("didnt get expected peerID") + } + + if !a.Equal(pinfo.Addrs[0]) { + t.Fatal("didnt get expected address") + } +} From cb1f88a8c037e7812cf75c82d37c7135f8e0ae68 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 12 Apr 2017 08:53:56 +0700 Subject: [PATCH 0234/3965] remove unused peerstore from listener --- p2p/transport/quic/dialer_test.go | 2 +- p2p/transport/quic/listener.go | 3 +-- p2p/transport/quic/listener_test.go | 4 ++-- p2p/transport/quic/transport.go | 8 ++------ p2p/transport/quic/transport_test.go | 2 +- 5 files changed, 7 insertions(+), 12 deletions(-) diff --git a/p2p/transport/quic/dialer_test.go b/p2p/transport/quic/dialer_test.go index f587cb65be..229e19a606 100644 --- a/p2p/transport/quic/dialer_test.go +++ b/p2p/transport/quic/dialer_test.go @@ -29,7 +29,7 @@ var _ = Describe("Listener", func() { var ln *listener go func() { defer GinkgoRecover() - ln, err = newListener(addr, nil, transport) + ln, err = newListener(addr, transport) Expect(err).ToNot(HaveOccurred()) _, err = ln.Accept() Expect(err).ToNot(HaveOccurred()) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 99eb96a5e7..4ca8487782 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -3,7 +3,6 @@ package libp2pquic import ( "net" - pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" testdata "github.com/lucas-clemente/quic-go/testdata" quicconn "github.com/marten-seemann/quic-conn" @@ -18,7 +17,7 @@ type listener struct { transport tpt.Transport } -func newListener(laddr ma.Multiaddr, peers pstore.Peerstore, transport tpt.Transport) (*listener, error) { +func newListener(laddr ma.Multiaddr, transport tpt.Transport) (*listener, error) { // we need to provide a certificate here // use the demo certificate from quic-go tlsConf := testdata.GetTLSConfig() diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index 9b1851a5e0..6c99e35b9f 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -54,7 +54,7 @@ var _ = Describe("Listener", func() { It("returns its addr", func() { laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/12345/quic") Expect(err).ToNot(HaveOccurred()) - l, err = newListener(laddr, nil, nil) + l, err = newListener(laddr, nil) Expect(err).ToNot(HaveOccurred()) Expect(l.Addr().String()).To(Equal("127.0.0.1:12345")) }) @@ -62,7 +62,7 @@ var _ = Describe("Listener", func() { It("returns its multiaddr", func() { laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/12346/quic") Expect(err).ToNot(HaveOccurred()) - l, err = newListener(laddr, nil, nil) + l, err = newListener(laddr, nil) Expect(err).ToNot(HaveOccurred()) Expect(l.Multiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/12346/quic")) }) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 68968695f8..7505f94a6d 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -4,7 +4,6 @@ import ( "fmt" "sync" - pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" "github.com/whyrusleeping/mafmt" @@ -12,8 +11,6 @@ import ( // QuicTransport implements a QUIC Transport type QuicTransport struct { - peers pstore.Peerstore - lmutex sync.Mutex listeners map[string]tpt.Listener @@ -23,10 +20,9 @@ type QuicTransport struct { // NewQuicTransport creates a new QUIC Transport // it tracks dialers and listeners created -func NewQuicTransport(peers pstore.Peerstore) *QuicTransport { +func NewQuicTransport() *QuicTransport { // utils.SetLogLevel(utils.LogLevelDebug) return &QuicTransport{ - peers: peers, listeners: make(map[string]tpt.Listener), dialers: make(map[string]tpt.Dialer), } @@ -69,7 +65,7 @@ func (t *QuicTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { return l, nil } - ln, err := newListener(laddr, t.peers, t) + ln, err := newListener(laddr, t) if err != nil { return nil, err } diff --git a/p2p/transport/quic/transport_test.go b/p2p/transport/quic/transport_test.go index d7f8358c84..141e468426 100644 --- a/p2p/transport/quic/transport_test.go +++ b/p2p/transport/quic/transport_test.go @@ -11,7 +11,7 @@ var _ = Describe("Transport", func() { var t *QuicTransport BeforeEach(func() { - t = NewQuicTransport(nil) + t = NewQuicTransport() }) Context("listening", func() { From 8013e9ce541577e6d81d610ffebc11564c0bb08c Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Wed, 26 Apr 2017 01:44:38 +0200 Subject: [PATCH 0235/3965] Add func to construct /ipfs addrs from PeerInfo --- p2p/host/peerstore/peerinfo.go | 13 ++++++++++ p2p/host/peerstore/peerinfo_test.go | 39 +++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index 3226a50972..2b86f08546 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -50,6 +50,19 @@ func InfoFromP2pAddr(m ma.Multiaddr) (*PeerInfo, error) { }, nil } +func InfoToP2pAddrs(pi *PeerInfo) ([]ma.Multiaddr, error) { + addrs := []ma.Multiaddr{} + tpl := "/" + ma.ProtocolWithCode(ma.P_IPFS).Name + "/" + for _, addr := range pi.Addrs { + p2paddr, err := ma.NewMultiaddr(tpl + peer.IDB58Encode(pi.ID)) + if err != nil { + return nil, err + } + addrs = append(addrs, addr.Encapsulate(p2paddr)) + } + return addrs, nil +} + func (pi *PeerInfo) Loggable() map[string]interface{} { return map[string]interface{}{ "peerID": pi.ID.Pretty(), diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index edad247c38..11d0737aba 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -83,3 +83,42 @@ func TestP2pAddrParsing(t *testing.T) { t.Fatal("didnt get expected address") } } + +func TestP2pAddrConstruction(t *testing.T) { + id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") + if err != nil { + t.Error(err) + } + addr := ma.StringCast("/ip4/1.2.3.4/tcp/4536") + p2paddr := ma.Join(addr, ma.StringCast("/ipfs/"+peer.IDB58Encode(id))) + + pi := &PeerInfo{ID: id, Addrs: []ma.Multiaddr{addr}} + p2paddrs, err := InfoToP2pAddrs(pi) + if err != nil { + t.Error(err) + } + + if len(p2paddrs) != 1 { + t.Fatalf("expected 1 addr, got %d", len(p2paddrs)) + } + + if !p2paddr.Equal(p2paddrs[0]) { + t.Fatalf("expected [%s], got [%s]", p2paddr, p2paddrs[0]) + } + + pi = &PeerInfo{ID: id} + p2paddrs, err = InfoToP2pAddrs(pi) + if err != nil { + t.Error(err) + } + + if len(p2paddrs) > 0 { + t.Fatalf("expected 0 addrs, got %d", len(p2paddrs)) + } + + pi = &PeerInfo{Addrs: []ma.Multiaddr{ma.StringCast("/ip4/1.2.3.4/tcp/4536")}} + _, err = InfoToP2pAddrs(pi) + if err == nil { + t.Fatalf("expected error, got none") + } +} From 3e9dccb742890ee06f788c5bbc72124d3c4349e5 Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Wed, 26 Apr 2017 01:58:43 +0200 Subject: [PATCH 0236/3965] Allow for parsing otherwise empty /ipfs addrs, improve tests --- p2p/host/peerstore/peerinfo.go | 13 +++++++-- p2p/host/peerstore/peerinfo_test.go | 45 ++++++++++++++++++++++------- 2 files changed, 45 insertions(+), 13 deletions(-) diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index 2b86f08546..cb216264b5 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -31,12 +31,13 @@ func InfoFromP2pAddr(m ma.Multiaddr) (*PeerInfo, error) { return nil, ErrInvalidAddr } - ipfspart := parts[len(parts)-1] // last part + // TODO(lgierth): we shouldn't assume /ipfs is the last part + ipfspart := parts[len(parts)-1] if ipfspart.Protocols()[0].Code != ma.P_IPFS { return nil, ErrInvalidAddr } - // make sure 'ipfs id' parses as a peer.ID + // make sure the /ipfs value parses as a peer.ID peerIdParts := strings.Split(ipfspart.String(), "/") peerIdStr := peerIdParts[len(peerIdParts)-1] id, err := peer.IDB58Decode(peerIdStr) @@ -44,9 +45,15 @@ func InfoFromP2pAddr(m ma.Multiaddr) (*PeerInfo, error) { return nil, err } + // we might have received just an /ipfs part, which means there's no addr. + var addrs []ma.Multiaddr + if len(parts) > 1 { + addrs = append(addrs, ma.Join(parts[:len(parts)-1]...)) + } + return &PeerInfo{ ID: id, - Addrs: []ma.Multiaddr{ma.Join(parts[:len(parts)-1]...)}, + Addrs: addrs, }, nil } diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index 11d0737aba..57ca5cc87f 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -58,29 +58,54 @@ func TestPeerInfoMarshal(t *testing.T) { } func TestP2pAddrParsing(t *testing.T) { - a := mustAddr(t, "/ip4/1.2.3.4/tcp/4536") id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") if err != nil { - t.Fatal(err) + t.Error(err) } + addr := ma.StringCast("/ip4/1.2.3.4/tcp/4536") + p2paddr := ma.Join(addr, ma.StringCast("/ipfs/"+peer.IDB58Encode(id))) - p2pa := a.String() + "/ipfs/" + id.Pretty() - p2pma, err := ma.NewMultiaddr(p2pa) + pinfo, err := InfoFromP2pAddr(p2paddr) if err != nil { - t.Fatal(err) + t.Error(err) + } + + if pinfo.ID != id { + t.Fatalf("expected PeerID [%s], got [%s]", id, pinfo.ID) + } + + if len(pinfo.Addrs) != 1 { + t.Fatalf("expected 1 addr, got %d", len(pinfo.Addrs)) } - pinfo, err := InfoFromP2pAddr(p2pma) + if !addr.Equal(pinfo.Addrs[0]) { + t.Fatalf("expected addr [%s], got [%s]", addr, pinfo.Addrs[0]) + } + + addr = ma.StringCast("/ipfs/" + peer.IDB58Encode(id)) + pinfo, err = InfoFromP2pAddr(addr) if err != nil { - t.Fatal(err) + t.Error(err) } if pinfo.ID != id { - t.Fatal("didnt get expected peerID") + t.Fatalf("expected PeerID [%s], got [%s]", id, pinfo.ID) + } + + if len(pinfo.Addrs) > 0 { + t.Fatalf("expected 0 addrs, got %d", len(pinfo.Addrs)) } - if !a.Equal(pinfo.Addrs[0]) { - t.Fatal("didnt get expected address") + addr = ma.StringCast("/ip4/1.2.3.4/tcp/4536") + pinfo, err = InfoFromP2pAddr(addr) + if err == nil { + t.Fatalf("expected error, got none") + } + + addr = ma.StringCast("/ip4/1.2.3.4/tcp/4536/http") + pinfo, err = InfoFromP2pAddr(addr) + if err == nil { + t.Fatalf("expected error, got none") } } From 0a1f5c07307597523809285fc36b289d5a6b0621 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 17 May 2017 16:57:51 +0800 Subject: [PATCH 0237/3965] add support for multi-stream connections --- p2p/transport/quic/conn.go | 113 ++++++++++++++++++++ p2p/transport/quic/conn_test.go | 159 ++++++++++++++++++++++++++++ p2p/transport/quic/dialer.go | 18 ++-- p2p/transport/quic/listener.go | 39 +++---- p2p/transport/quic/listener_test.go | 61 +++-------- p2p/transport/quic/stream.go | 29 +++++ p2p/transport/quic/transport.go | 2 + 7 files changed, 343 insertions(+), 78 deletions(-) create mode 100644 p2p/transport/quic/conn.go create mode 100644 p2p/transport/quic/conn_test.go create mode 100644 p2p/transport/quic/stream.go diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go new file mode 100644 index 0000000000..1239e191e9 --- /dev/null +++ b/p2p/transport/quic/conn.go @@ -0,0 +1,113 @@ +package libp2pquic + +import ( + "fmt" + "net" + + smux "github.com/jbenet/go-stream-muxer" + tpt "github.com/libp2p/go-libp2p-transport" + quic "github.com/lucas-clemente/quic-go" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +type quicConn struct { + sess quic.Session + transport tpt.Transport + + laddr ma.Multiaddr + raddr ma.Multiaddr +} + +var _ tpt.Conn = &quicConn{} +var _ tpt.MultiStreamConn = &quicConn{} + +func newQuicConn(sess quic.Session, t tpt.Transport) (*quicConn, error) { + // analogues to manet.WrapNetConn + + laddr, err := quicMultiAddress(sess.LocalAddr()) + if err != nil { + return nil, fmt.Errorf("failed to convert nconn.LocalAddr: %s", err) + } + + // analogues to manet.WrapNetConn + raddr, err := quicMultiAddress(sess.RemoteAddr()) + if err != nil { + return nil, fmt.Errorf("failed to convert nconn.RemoteAddr: %s", err) + } + + return &quicConn{ + sess: sess, + laddr: laddr, + raddr: raddr, + transport: t, + }, nil +} + +func (c *quicConn) AcceptStream() (smux.Stream, error) { + str, err := c.sess.AcceptStream() + if err != nil { + return nil, err + } + return &quicStream{Stream: str}, nil +} + +func (c *quicConn) OpenStream() (smux.Stream, error) { + str, err := c.sess.OpenStream() + if err != nil { + return nil, err + } + return &quicStream{Stream: str}, nil +} + +func (c *quicConn) Serve(handler smux.StreamHandler) { + for { // accept loop + s, err := c.AcceptStream() + if err != nil { + return // err always means closed. + } + go handler(s) + } +} + +func (c *quicConn) Close() error { + return c.sess.Close(nil) +} + +// TODO: implement this +func (c *quicConn) IsClosed() bool { + return false +} + +func (c *quicConn) LocalAddr() net.Addr { + return c.sess.LocalAddr() +} + +func (c *quicConn) LocalMultiaddr() ma.Multiaddr { + return c.laddr +} + +func (c *quicConn) RemoteAddr() net.Addr { + return c.sess.RemoteAddr() +} + +func (c *quicConn) RemoteMultiaddr() ma.Multiaddr { + return c.raddr +} + +func (c *quicConn) Transport() tpt.Transport { + return c.transport +} + +// TODO: there must be a better way to do this +func quicMultiAddress(na net.Addr) (ma.Multiaddr, error) { + udpMA, err := manet.FromNetAddr(na) + if err != nil { + return nil, err + } + quicMA, err := ma.NewMultiaddr(udpMA.String() + "/quic") + if err != nil { + return nil, err + } + return quicMA, nil +} diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go new file mode 100644 index 0000000000..bfce79c965 --- /dev/null +++ b/p2p/transport/quic/conn_test.go @@ -0,0 +1,159 @@ +package libp2pquic + +import ( + "errors" + "net" + + smux "github.com/jbenet/go-stream-muxer" + quic "github.com/lucas-clemente/quic-go" + "github.com/lucas-clemente/quic-go/protocol" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +type mockStream struct { + id protocol.StreamID +} + +func (s *mockStream) Close() error { return nil } +func (s *mockStream) Reset(error) { return } +func (s *mockStream) Read([]byte) (int, error) { return 0, nil } +func (s *mockStream) Write([]byte) (int, error) { return 0, nil } +func (s *mockStream) StreamID() protocol.StreamID { return s.id } + +var _ quic.Stream = &mockStream{} + +type mockQuicSession struct { + closed bool + + localAddr net.Addr + remoteAddr net.Addr + + streamToAccept quic.Stream + streamAcceptErr error + + streamToOpen quic.Stream + streamOpenErr error +} + +var _ quic.Session = &mockQuicSession{} + +func (s *mockQuicSession) AcceptStream() (quic.Stream, error) { + return s.streamToAccept, s.streamAcceptErr +} +func (s *mockQuicSession) OpenStream() (quic.Stream, error) { return s.streamToOpen, s.streamOpenErr } +func (s *mockQuicSession) OpenStreamSync() (quic.Stream, error) { return s.streamToOpen, nil } +func (s *mockQuicSession) Close(error) error { s.closed = true; return nil } +func (s *mockQuicSession) LocalAddr() net.Addr { return s.localAddr } +func (s *mockQuicSession) RemoteAddr() net.Addr { return s.remoteAddr } + +var _ = Describe("Conn", func() { + var ( + conn *quicConn + sess *mockQuicSession + ) + + BeforeEach(func() { + sess = &mockQuicSession{ + localAddr: &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 1337}, + remoteAddr: &net.UDPAddr{IP: net.IPv4(192, 168, 13, 37), Port: 1234}, + } + var err error + conn, err = newQuicConn(sess, nil) + Expect(err).ToNot(HaveOccurred()) + }) + + It("has the correct local address", func() { + Expect(conn.LocalAddr()).To(Equal(sess.localAddr)) + Expect(conn.LocalMultiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/1337/quic")) + }) + + It("has the correct remote address", func() { + Expect(conn.RemoteAddr()).To(Equal(sess.remoteAddr)) + Expect(conn.RemoteMultiaddr().String()).To(Equal("/ip4/192.168.13.37/udp/1234/quic")) + }) + + It("closes", func() { + err := conn.Close() + Expect(err).ToNot(HaveOccurred()) + Expect(sess.closed).To(BeTrue()) + }) + + Context("opening streams", func() { + It("opens streams", func() { + s := &mockStream{id: 1337} + sess.streamToOpen = s + str, err := conn.OpenStream() + Expect(err).ToNot(HaveOccurred()) + Expect(str.(*quicStream).Stream).To(Equal(s)) + }) + + It("errors when it can't open a stream", func() { + testErr := errors.New("stream open err") + sess.streamOpenErr = testErr + _, err := conn.OpenStream() + Expect(err).To(MatchError(testErr)) + }) + }) + + Context("accepting streams", func() { + It("accepts streams", func() { + s := &mockStream{id: 1337} + sess.streamToAccept = s + str, err := conn.AcceptStream() + Expect(err).ToNot(HaveOccurred()) + Expect(str.(*quicStream).Stream).To(Equal(s)) + }) + + It("errors when it can't open a stream", func() { + testErr := errors.New("stream open err") + sess.streamAcceptErr = testErr + _, err := conn.AcceptStream() + Expect(err).To(MatchError(testErr)) + }) + }) + + Context("serving", func() { + var ( + handler func(smux.Stream) + handlerCalled bool + handlerCalledWith smux.Stream + ) + + BeforeEach(func() { + handlerCalled = false + handlerCalledWith = nil + handler = func(s smux.Stream) { + handlerCalledWith = s + handlerCalled = true + } + }) + + It("calls the handler", func() { + str := &mockStream{id: 5} + sess.streamToAccept = str + var returned bool + go func() { + conn.Serve(handler) + returned = true + }() + Eventually(func() bool { return handlerCalled }).Should(BeTrue()) + Expect(handlerCalledWith.(*quicStream).Stream).To(Equal(str)) + // make the go-routine return + sess.streamAcceptErr = errors.New("stop test") + }) + + It("returns when accepting a stream errors", func() { + sess.streamAcceptErr = errors.New("accept err") + var returned bool + go func() { + conn.Serve(handler) + returned = true + }() + Eventually(func() bool { return returned }).Should(BeTrue()) + Expect(handlerCalled).To(BeFalse()) + }) + }) + +}) diff --git a/p2p/transport/quic/dialer.go b/p2p/transport/quic/dialer.go index f8e90ed93b..8563674369 100644 --- a/p2p/transport/quic/dialer.go +++ b/p2p/transport/quic/dialer.go @@ -5,7 +5,7 @@ import ( "crypto/tls" tpt "github.com/libp2p/go-libp2p-transport" - quicconn "github.com/marten-seemann/quic-conn" + quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" "github.com/whyrusleeping/mafmt" @@ -15,6 +15,8 @@ type dialer struct { transport tpt.Transport } +var _ tpt.Dialer = &dialer{} + func newDialer(transport tpt.Transport) (*dialer, error) { return &dialer{ transport: transport, @@ -28,28 +30,20 @@ func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { if err != nil { return nil, err } - c, err := quicconn.Dial(host, tlsConf) - if err != nil { - return nil, err - } - mnc, err := manet.WrapNetConn(c) + qsess, err := quic.DialAddr(host, &quic.Config{TLSConfig: tlsConf}) if err != nil { return nil, err } - return &tpt.ConnWrap{ - Conn: mnc, - Tpt: d.transport, - }, nil + return newQuicConn(qsess, d.transport) } func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { + // TODO: implement the ctx return d.Dial(raddr) } func (d *dialer) Matches(a ma.Multiaddr) bool { return mafmt.QUIC.Matches(a) } - -var _ tpt.Dialer = &dialer{} diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 4ca8487782..18ab38201c 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -4,53 +4,50 @@ import ( "net" tpt "github.com/libp2p/go-libp2p-transport" + quic "github.com/lucas-clemente/quic-go" testdata "github.com/lucas-clemente/quic-go/testdata" - quicconn "github.com/marten-seemann/quic-conn" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) type listener struct { laddr ma.Multiaddr - quicListener net.Listener + quicListener quic.Listener transport tpt.Transport } -func newListener(laddr ma.Multiaddr, transport tpt.Transport) (*listener, error) { - // we need to provide a certificate here - // use the demo certificate from quic-go - tlsConf := testdata.GetTLSConfig() - network, host, err := manet.DialArgs(laddr) +var _ tpt.Listener = &listener{} + +func newListener(laddr ma.Multiaddr, t tpt.Transport) (*listener, error) { + qconf := &quic.Config{ + // we need to provide a certificate here + // use the demo certificate from quic-go + TLSConfig: testdata.GetTLSConfig(), + } + + _, host, err := manet.DialArgs(laddr) if err != nil { return nil, err } - qln, err := quicconn.Listen(network, host, tlsConf) + qln, err := quic.ListenAddr(host, qconf) if err != nil { return nil, err } + return &listener{ laddr: laddr, quicListener: qln, - transport: transport, + transport: t, }, nil } func (l *listener) Accept() (tpt.Conn, error) { - c, err := l.quicListener.Accept() + sess, err := l.quicListener.Accept() if err != nil { return nil, err } - - mnc, err := manet.WrapNetConn(c) - if err != nil { - return nil, err - } - - return &tpt.ConnWrap{ - Conn: mnc, - Tpt: l.transport, - }, nil + return newQuicConn(sess, l.transport) } func (l *listener) Close() error { @@ -64,5 +61,3 @@ func (l *listener) Addr() net.Addr { func (l *listener) Multiaddr() ma.Multiaddr { return l.laddr } - -var _ tpt.Listener = &listener{} diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index 6c99e35b9f..726c54f5d7 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -5,48 +5,42 @@ import ( "net" tpt "github.com/libp2p/go-libp2p-transport" + quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) -type mockNetListener struct { +type mockQuicListener struct { connToAccept net.Conn - acceptErr error + serveErr error closed bool -} -func (m *mockNetListener) Accept() (net.Conn, error) { - if m.acceptErr != nil { - return nil, m.acceptErr - } - return m.connToAccept, nil + sessionToAccept quic.Session + acceptErr error } -func (m *mockNetListener) Close() error { - m.closed = true - return nil -} +var _ quic.Listener = &mockQuicListener{} + +func (m *mockQuicListener) Close() error { m.closed = true; return nil } +func (m *mockQuicListener) Accept() (quic.Session, error) { return m.sessionToAccept, m.acceptErr } +func (m *mockQuicListener) Addr() net.Addr { panic("not implemented") } -func (m *mockNetListener) Addr() net.Addr { - panic("not implemented") -} -var _ net.Listener = &mockNetListener{} var _ = Describe("Listener", func() { var ( - l *listener - netListener *mockNetListener - transport tpt.Transport + l *listener + quicListener *mockQuicListener + transport tpt.Transport ) BeforeEach(func() { - netListener = &mockNetListener{} + quicListener = &mockQuicListener{} transport = &QuicTransport{} l = &listener{ - quicListener: netListener, + quicListener: quicListener, transport: transport, } }) @@ -70,34 +64,13 @@ var _ = Describe("Listener", func() { It("closes", func() { err := l.Close() Expect(err).ToNot(HaveOccurred()) - Expect(netListener.closed).To(BeTrue()) + Expect(quicListener.closed).To(BeTrue()) }) Context("accepting", func() { - It("accepts a new conn", func() { - remoteAddr, err := net.ResolveUDPAddr("udp", "127.0.0.1:1234") - Expect(err).ToNot(HaveOccurred()) - localAddr, err := net.ResolveUDPAddr("udp", "127.0.0.1:4321") - Expect(err).ToNot(HaveOccurred()) - udpConn, err := net.DialUDP("udp", localAddr, remoteAddr) - netListener.connToAccept = udpConn - conn, err := l.Accept() - Expect(err).ToNot(HaveOccurred()) - Expect(conn.LocalMultiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/4321")) - Expect(conn.RemoteMultiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/1234")) - Expect(conn.Transport()).To(Equal(transport)) - }) - - It("errors if it can't read the muliaddresses of a conn", func() { - netListener.connToAccept = &net.UDPConn{} - _, err := l.Accept() - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("nil multiaddr")) - }) - It("errors if no connection can be accepted", func() { testErr := errors.New("test error") - netListener.acceptErr = testErr + quicListener.acceptErr = testErr _, err := l.Accept() Expect(err).To(MatchError(testErr)) }) diff --git a/p2p/transport/quic/stream.go b/p2p/transport/quic/stream.go new file mode 100644 index 0000000000..c41b2b324b --- /dev/null +++ b/p2p/transport/quic/stream.go @@ -0,0 +1,29 @@ +package libp2pquic + +import ( + "time" + + smux "github.com/jbenet/go-stream-muxer" + quic "github.com/lucas-clemente/quic-go" +) + +// The quicStream is a very thin wrapper for a quic.Stream, adding some methods +// required to fulfill the smux.Stream interface +// TODO: this can be removed once the quic.Stream supports deadlines (quic-go#514) +type quicStream struct { + quic.Stream +} + +var _ smux.Stream = &quicStream{} + +func (s *quicStream) SetDeadline(time.Time) error { + return nil +} + +func (s *quicStream) SetReadDeadline(time.Time) error { + return nil +} + +func (s *quicStream) SetWriteDeadline(time.Time) error { + return nil +} diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 7505f94a6d..04f6a82751 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -18,6 +18,8 @@ type QuicTransport struct { dialers map[string]tpt.Dialer } +var _ tpt.Transport = &QuicTransport{} + // NewQuicTransport creates a new QUIC Transport // it tracks dialers and listeners created func NewQuicTransport() *QuicTransport { From 80bf4d2a41b6f7b6db28fd2f73ec6df8bbd3293a Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 27 May 2017 20:17:34 +0800 Subject: [PATCH 0238/3965] read nonces, even if they are fragmented io.ReadFull is garantueed to read the complete length of the buffer (or to return an error if that fails). This way, reading the nonce also succeeds when multiple Read calls are necessary. --- p2p/net/pnet/psk_conn.go | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index d60b7ef717..cce585216f 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -3,6 +3,7 @@ package pnet import ( "crypto/cipher" "crypto/rand" + "io" salsa20 "github.com/davidlazar/go-crypto/salsa20" mpool "github.com/jbenet/go-msgio/mpool" @@ -31,11 +32,8 @@ type pskConn struct { func (c *pskConn) Read(out []byte) (int, error) { if c.readS20 == nil { nonce := make([]byte, 24) - n, err := c.Conn.Read(nonce) + _, err := io.ReadFull(c.Conn, nonce) if err != nil { - return 0, err - } - if n != 24 { return 0, errShortNonce } c.readS20 = salsa20.New(c.psk, nonce) From 3e754c4b0abd6c6ca80095fae4be51b1b9e4dff1 Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Tue, 6 Jun 2017 02:42:05 +0200 Subject: [PATCH 0239/3965] Fix go-ws-transport package name --- p2p/net/swarm/swarm.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 6eba8f819b..a282f7429b 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -28,12 +28,12 @@ import ( filter "github.com/libp2p/go-maddr-filter" ps "github.com/libp2p/go-peerstream" tcpt "github.com/libp2p/go-tcp-transport" + ws "github.com/libp2p/go-ws-transport" ma "github.com/multiformats/go-multiaddr" psmss "github.com/whyrusleeping/go-smux-multistream" spdy "github.com/whyrusleeping/go-smux-spdystream" yamux "github.com/whyrusleeping/go-smux-yamux" mafilter "github.com/whyrusleeping/multiaddr-filter" - ws "github.com/whyrusleeping/ws-transport" ) var log = logging.Logger("swarm2") From 15cc2f7e686896c250e7fac24157fa90974dcb67 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 15 Jun 2017 16:33:02 +0200 Subject: [PATCH 0240/3965] open new streams synchronously --- p2p/transport/quic/conn.go | 4 +++- p2p/transport/quic/conn_test.go | 12 +++++++----- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 1239e191e9..8d396e6c8d 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -52,8 +52,10 @@ func (c *quicConn) AcceptStream() (smux.Stream, error) { return &quicStream{Stream: str}, nil } +// OpenStream opens a new stream +// It blocks until a new stream can be opened (when limited by the QUIC maximum stream limit) func (c *quicConn) OpenStream() (smux.Stream, error) { - str, err := c.sess.OpenStream() + str, err := c.sess.OpenStreamSync() if err != nil { return nil, err } diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index bfce79c965..562c442eab 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -42,11 +42,13 @@ var _ quic.Session = &mockQuicSession{} func (s *mockQuicSession) AcceptStream() (quic.Stream, error) { return s.streamToAccept, s.streamAcceptErr } -func (s *mockQuicSession) OpenStream() (quic.Stream, error) { return s.streamToOpen, s.streamOpenErr } -func (s *mockQuicSession) OpenStreamSync() (quic.Stream, error) { return s.streamToOpen, nil } -func (s *mockQuicSession) Close(error) error { s.closed = true; return nil } -func (s *mockQuicSession) LocalAddr() net.Addr { return s.localAddr } -func (s *mockQuicSession) RemoteAddr() net.Addr { return s.remoteAddr } +func (s *mockQuicSession) OpenStream() (quic.Stream, error) { return s.streamToOpen, s.streamOpenErr } +func (s *mockQuicSession) OpenStreamSync() (quic.Stream, error) { + return s.streamToOpen, s.streamOpenErr +} +func (s *mockQuicSession) Close(error) error { s.closed = true; return nil } +func (s *mockQuicSession) LocalAddr() net.Addr { return s.localAddr } +func (s *mockQuicSession) RemoteAddr() net.Addr { return s.remoteAddr } var _ = Describe("Conn", func() { var ( From 1a9d35883b971f091837f1d87ab71fa9432d7940 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 15 Jun 2017 17:02:10 +0200 Subject: [PATCH 0241/3965] return resolved address in the listener --- p2p/transport/quic/listener.go | 6 +++++- p2p/transport/quic/listener_test.go | 15 +++++++++------ 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 18ab38201c..dbd644a0ff 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -34,9 +34,13 @@ func newListener(laddr ma.Multiaddr, t tpt.Transport) (*listener, error) { if err != nil { return nil, err } + addr, err := quicMultiAddress(qln.Addr()) + if err != nil { + return nil, err + } return &listener{ - laddr: laddr, + laddr: addr, quicListener: qln, transport: t, }, nil diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index 726c54f5d7..f4e03d63c8 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -27,8 +27,6 @@ func (m *mockQuicListener) Close() error { m.closed = true; ret func (m *mockQuicListener) Accept() (quic.Session, error) { return m.sessionToAccept, m.acceptErr } func (m *mockQuicListener) Addr() net.Addr { panic("not implemented") } - - var _ = Describe("Listener", func() { var ( l *listener @@ -46,19 +44,24 @@ var _ = Describe("Listener", func() { }) It("returns its addr", func() { - laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/12345/quic") + laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") Expect(err).ToNot(HaveOccurred()) l, err = newListener(laddr, nil) Expect(err).ToNot(HaveOccurred()) - Expect(l.Addr().String()).To(Equal("127.0.0.1:12345")) + as := l.Addr().String() + Expect(as).ToNot(Equal("127.0.0.1:0)")) + Expect(as).To(ContainSubstring("127.0.0.1:")) }) It("returns its multiaddr", func() { - laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/12346/quic") + laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") Expect(err).ToNot(HaveOccurred()) l, err = newListener(laddr, nil) Expect(err).ToNot(HaveOccurred()) - Expect(l.Multiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/12346/quic")) + mas := l.Multiaddr().String() + Expect(mas).ToNot(Equal("/ip4/127.0.0.1/udp/0/quic")) + Expect(mas).To(ContainSubstring("/ip4/127.0.0.1/udp/")) + Expect(mas).To(ContainSubstring("/quic")) }) It("closes", func() { From c7bf01e46dd6c63b4171cf0b72a810167a27ef3a Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 16 Jun 2017 23:26:39 +0200 Subject: [PATCH 0242/3965] implement IsClosed() for the connection --- p2p/transport/quic/conn.go | 26 +++++++++++++++++++++----- p2p/transport/quic/conn_test.go | 15 ++++++++++++--- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 8d396e6c8d..cbc6709f40 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -3,6 +3,7 @@ package libp2pquic import ( "fmt" "net" + "sync" smux "github.com/jbenet/go-stream-muxer" tpt "github.com/libp2p/go-libp2p-transport" @@ -12,11 +13,15 @@ import ( ) type quicConn struct { + mutex sync.RWMutex + sess quic.Session transport tpt.Transport laddr ma.Multiaddr raddr ma.Multiaddr + + closed bool } var _ tpt.Conn = &quicConn{} @@ -24,7 +29,6 @@ var _ tpt.MultiStreamConn = &quicConn{} func newQuicConn(sess quic.Session, t tpt.Transport) (*quicConn, error) { // analogues to manet.WrapNetConn - laddr, err := quicMultiAddress(sess.LocalAddr()) if err != nil { return nil, fmt.Errorf("failed to convert nconn.LocalAddr: %s", err) @@ -36,12 +40,16 @@ func newQuicConn(sess quic.Session, t tpt.Transport) (*quicConn, error) { return nil, fmt.Errorf("failed to convert nconn.RemoteAddr: %s", err) } - return &quicConn{ + c := &quicConn{ sess: sess, laddr: laddr, raddr: raddr, transport: t, - }, nil + } + + go c.watchClosed() + + return c, nil } func (c *quicConn) AcceptStream() (smux.Stream, error) { @@ -76,9 +84,17 @@ func (c *quicConn) Close() error { return c.sess.Close(nil) } -// TODO: implement this +func (c *quicConn) watchClosed() { + c.sess.WaitUntilClosed() + c.mutex.Lock() + c.closed = true + c.mutex.Unlock() +} + func (c *quicConn) IsClosed() bool { - return false + c.mutex.Lock() + defer c.mutex.Unlock() + return c.closed } func (c *quicConn) LocalAddr() net.Addr { diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 562c442eab..03e519d706 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -25,7 +25,8 @@ func (s *mockStream) StreamID() protocol.StreamID { return s.id } var _ quic.Stream = &mockStream{} type mockQuicSession struct { - closed bool + closed bool + waitUntilClosedChan chan struct{} // close this chan to make WaitUntilClosed return localAddr net.Addr remoteAddr net.Addr @@ -49,6 +50,7 @@ func (s *mockQuicSession) OpenStreamSync() (quic.Stream, error) { func (s *mockQuicSession) Close(error) error { s.closed = true; return nil } func (s *mockQuicSession) LocalAddr() net.Addr { return s.localAddr } func (s *mockQuicSession) RemoteAddr() net.Addr { return s.remoteAddr } +func (s *mockQuicSession) WaitUntilClosed() { <-s.waitUntilClosedChan } var _ = Describe("Conn", func() { var ( @@ -58,8 +60,9 @@ var _ = Describe("Conn", func() { BeforeEach(func() { sess = &mockQuicSession{ - localAddr: &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 1337}, - remoteAddr: &net.UDPAddr{IP: net.IPv4(192, 168, 13, 37), Port: 1234}, + localAddr: &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 1337}, + remoteAddr: &net.UDPAddr{IP: net.IPv4(192, 168, 13, 37), Port: 1234}, + waitUntilClosedChan: make(chan struct{}), } var err error conn, err = newQuicConn(sess, nil) @@ -82,6 +85,12 @@ var _ = Describe("Conn", func() { Expect(sess.closed).To(BeTrue()) }) + It("says if it is closed", func() { + Consistently(func() bool { return conn.IsClosed() }).Should(BeFalse()) + close(sess.waitUntilClosedChan) + Eventually(func() bool { return conn.IsClosed() }).Should(BeTrue()) + }) + Context("opening streams", func() { It("opens streams", func() { s := &mockStream{id: 1337} From 398fcd59b4ca90d36a72b7e4714f16b795f07032 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Sat, 1 Jul 2017 23:11:20 +0200 Subject: [PATCH 0243/3965] feat: upgrade to use gorilla/websocket Closes #12 --- p2p/transport/websocket/websocket.go | 106 ++++++++++++++++++++++----- 1 file changed, 89 insertions(+), 17 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 69fc11c408..46130c150c 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -6,7 +6,9 @@ import ( "net" "net/http" "net/url" + "time" + wsGorilla "github.com/gorilla/websocket" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" @@ -29,6 +31,9 @@ var WsCodec = &manet.NetCodec{ ParseNetAddr: ParseWebsocketNetAddr, } +// Default gorilla upgrader +var upgrader = wsGorilla.Upgrader{} + func init() { err := ma.AddProtocol(WsProtocol) if err != nil { @@ -107,12 +112,15 @@ func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, return nil, err } - wscon, err := ws.Dial(wsurl, "", "http://127.0.0.1:0/") + // TODO: figure out origins, probably don't work for us + // header := http.Header{} + // header.Set("Origin", "http://127.0.0.1:0/") + wscon, _, err := wsGorilla.DefaultDialer.Dial(wsurl, nil) if err != nil { return nil, err } - mnc, err := manet.WrapNetConn(wscon) + mnc, err := manet.WrapNetConn(NewGorillaNetConn(wscon)) if err != nil { return nil, err } @@ -141,17 +149,19 @@ type listener struct { incoming chan *conn tpt tpt.Transport + + origin *url.URL } type conn struct { - *ws.Conn + *GorillaNetConn done func() } func (c *conn) Close() error { c.done() - return c.Conn.Close() + return c.GorillaNetConn.Close() } func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { @@ -160,38 +170,41 @@ func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { return nil, err } - tlist := t.wrapListener(list) - - u, err := url.Parse("ws://" + list.Addr().String()) + u, err := url.Parse("http://" + list.Addr().String()) if err != nil { return nil, err } - s := &ws.Server{ - Handler: tlist.handleWsConn, - Config: ws.Config{Origin: u}, - } + tlist := t.wrapListener(list, u) - go http.Serve(list.NetListener(), s) + http.HandleFunc("/", tlist.handleWsConn) + go http.Serve(list.NetListener(), nil) return tlist, nil } -func (t *WebsocketTransport) wrapListener(l manet.Listener) *listener { +func (t *WebsocketTransport) wrapListener(l manet.Listener, origin *url.URL) *listener { return &listener{ Listener: l, incoming: make(chan *conn), tpt: t, + origin: origin, } } -func (l *listener) handleWsConn(s *ws.Conn) { +func (l *listener) handleWsConn(w http.ResponseWriter, r *http.Request) { + c, err := upgrader.Upgrade(w, r, nil) + if err != nil { + http.Error(w, "Failed to upgrade websocket", 400) + return + } + ctx, cancel := context.WithCancel(context.Background()) - s.PayloadType = ws.BinaryFrame + wrapped := NewGorillaNetConn(c) l.incoming <- &conn{ - Conn: s, - done: cancel, + GorillaNetConn: &wrapped, + done: cancel, } // wait until conn gets closed, otherwise the handler closes it early @@ -225,3 +238,62 @@ func (l *listener) Multiaddr() ma.Multiaddr { } var _ tpt.Transport = (*WebsocketTransport)(nil) + +type GorillaNetConn struct { + Inner *wsGorilla.Conn + DefaultMessageType int +} + +func (c GorillaNetConn) Read(b []byte) (n int, err error) { + fmt.Println("reading") + _, r, err := c.Inner.NextReader() + if err != nil { + return 0, err + } + + return r.Read(b) +} + +func (c GorillaNetConn) Write(b []byte) (n int, err error) { + fmt.Printf("write %s\n", string(b)) + if err := c.Inner.WriteMessage(c.DefaultMessageType, b); err != nil { + return 0, err + } + + return len(b), nil +} + +func (c GorillaNetConn) Close() error { + return c.Inner.Close() +} + +func (c GorillaNetConn) LocalAddr() net.Addr { + return c.Inner.LocalAddr() +} + +func (c GorillaNetConn) RemoteAddr() net.Addr { + return c.Inner.RemoteAddr() +} + +func (c GorillaNetConn) SetDeadline(t time.Time) error { + if err := c.SetReadDeadline(t); err != nil { + return err + } + + return c.SetReadDeadline(t) +} + +func (c GorillaNetConn) SetReadDeadline(t time.Time) error { + return c.Inner.SetReadDeadline(t) +} + +func (c GorillaNetConn) SetWriteDeadline(t time.Time) error { + return c.Inner.SetWriteDeadline(t) +} + +func NewGorillaNetConn(raw *wsGorilla.Conn) GorillaNetConn { + return GorillaNetConn{ + Inner: raw, + DefaultMessageType: wsGorilla.BinaryMessage, + } +} From 0cb827d19b2bbed0ee6bac0b739f2a757ee3d72b Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Tue, 4 Jul 2017 22:39:10 +0200 Subject: [PATCH 0244/3965] gxified gorilla/websocket --- p2p/transport/websocket/websocket.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 46130c150c..20ba9cd479 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -8,12 +8,12 @@ import ( "net/url" "time" - wsGorilla "github.com/gorilla/websocket" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" ws "golang.org/x/net/websocket" + wsGorilla "gx/ipfs/QmdKzkTPWmQ4nc8McYVb9TJYzRGmux9sqySQBD4nhbjQpf/." ) var WsProtocol = ma.Protocol{ From 32c4c20ddd78b3fdf3b423872c68270aa9120a41 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Tue, 4 Jul 2017 22:47:16 +0200 Subject: [PATCH 0245/3965] first round of cr fixes --- p2p/transport/websocket/websocket.go | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 20ba9cd479..d0855be534 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -13,7 +13,7 @@ import ( manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" ws "golang.org/x/net/websocket" - wsGorilla "gx/ipfs/QmdKzkTPWmQ4nc8McYVb9TJYzRGmux9sqySQBD4nhbjQpf/." + wsGorilla "gx/ipfs/QmdKzkTPWmQ4nc8McYVb9TJYzRGmux9sqySQBD4nhbjQpf" ) var WsProtocol = ma.Protocol{ @@ -245,7 +245,6 @@ type GorillaNetConn struct { } func (c GorillaNetConn) Read(b []byte) (n int, err error) { - fmt.Println("reading") _, r, err := c.Inner.NextReader() if err != nil { return 0, err @@ -255,7 +254,6 @@ func (c GorillaNetConn) Read(b []byte) (n int, err error) { } func (c GorillaNetConn) Write(b []byte) (n int, err error) { - fmt.Printf("write %s\n", string(b)) if err := c.Inner.WriteMessage(c.DefaultMessageType, b); err != nil { return 0, err } @@ -280,7 +278,7 @@ func (c GorillaNetConn) SetDeadline(t time.Time) error { return err } - return c.SetReadDeadline(t) + return c.SetWriteDeadline(t) } func (c GorillaNetConn) SetReadDeadline(t time.Time) error { From f5d0996f7a73d22dda07128cf482231b7bfe222a Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Tue, 4 Jul 2017 22:50:03 +0200 Subject: [PATCH 0246/3965] more cr fixes --- p2p/transport/websocket/websocket.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index d0855be534..b0dac81341 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -177,8 +177,7 @@ func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { tlist := t.wrapListener(list, u) - http.HandleFunc("/", tlist.handleWsConn) - go http.Serve(list.NetListener(), nil) + go http.Serve(list.NetListener(), tlist) return tlist, nil } @@ -192,7 +191,7 @@ func (t *WebsocketTransport) wrapListener(l manet.Listener, origin *url.URL) *li } } -func (l *listener) handleWsConn(w http.ResponseWriter, r *http.Request) { +func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { c, err := upgrader.Upgrade(w, r, nil) if err != nil { http.Error(w, "Failed to upgrade websocket", 400) From 661ec220fd14f169bbcea42aad17779d1d441520 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Wed, 5 Jul 2017 17:25:15 +0200 Subject: [PATCH 0247/3965] use more pointers --- p2p/transport/websocket/websocket.go | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index b0dac81341..4caffb5936 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -202,7 +202,7 @@ func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { wrapped := NewGorillaNetConn(c) l.incoming <- &conn{ - GorillaNetConn: &wrapped, + GorillaNetConn: wrapped, done: cancel, } @@ -243,7 +243,7 @@ type GorillaNetConn struct { DefaultMessageType int } -func (c GorillaNetConn) Read(b []byte) (n int, err error) { +func (c *GorillaNetConn) Read(b []byte) (n int, err error) { _, r, err := c.Inner.NextReader() if err != nil { return 0, err @@ -252,7 +252,7 @@ func (c GorillaNetConn) Read(b []byte) (n int, err error) { return r.Read(b) } -func (c GorillaNetConn) Write(b []byte) (n int, err error) { +func (c *GorillaNetConn) Write(b []byte) (n int, err error) { if err := c.Inner.WriteMessage(c.DefaultMessageType, b); err != nil { return 0, err } @@ -260,19 +260,19 @@ func (c GorillaNetConn) Write(b []byte) (n int, err error) { return len(b), nil } -func (c GorillaNetConn) Close() error { +func (c *GorillaNetConn) Close() error { return c.Inner.Close() } -func (c GorillaNetConn) LocalAddr() net.Addr { +func (c *GorillaNetConn) LocalAddr() net.Addr { return c.Inner.LocalAddr() } -func (c GorillaNetConn) RemoteAddr() net.Addr { +func (c *GorillaNetConn) RemoteAddr() net.Addr { return c.Inner.RemoteAddr() } -func (c GorillaNetConn) SetDeadline(t time.Time) error { +func (c *GorillaNetConn) SetDeadline(t time.Time) error { if err := c.SetReadDeadline(t); err != nil { return err } @@ -280,16 +280,16 @@ func (c GorillaNetConn) SetDeadline(t time.Time) error { return c.SetWriteDeadline(t) } -func (c GorillaNetConn) SetReadDeadline(t time.Time) error { +func (c *GorillaNetConn) SetReadDeadline(t time.Time) error { return c.Inner.SetReadDeadline(t) } -func (c GorillaNetConn) SetWriteDeadline(t time.Time) error { +func (c *GorillaNetConn) SetWriteDeadline(t time.Time) error { return c.Inner.SetWriteDeadline(t) } -func NewGorillaNetConn(raw *wsGorilla.Conn) GorillaNetConn { - return GorillaNetConn{ +func NewGorillaNetConn(raw *wsGorilla.Conn) *GorillaNetConn { + return &GorillaNetConn{ Inner: raw, DefaultMessageType: wsGorilla.BinaryMessage, } From be25e2eccbd361946097f8a3c41f5a98282538f8 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Wed, 5 Jul 2017 18:10:17 +0200 Subject: [PATCH 0248/3965] another import fix --- p2p/transport/websocket/websocket.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 4caffb5936..0a94b30cd6 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -13,7 +13,7 @@ import ( manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" ws "golang.org/x/net/websocket" - wsGorilla "gx/ipfs/QmdKzkTPWmQ4nc8McYVb9TJYzRGmux9sqySQBD4nhbjQpf" + wsGorilla "gx/ipfs/QmZH5VXfAJouGMyCCHTRPGCT3e5MG9Lu78Ln3YAYW1XTts/websocket" ) var WsProtocol = ma.Protocol{ From fd70584011fa525e15db520dcd1385bc4a162c2e Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Wed, 5 Jul 2017 19:30:09 +0200 Subject: [PATCH 0249/3965] gx unwrite --- p2p/transport/websocket/websocket.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 0a94b30cd6..83a4e90aab 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -8,12 +8,12 @@ import ( "net/url" "time" + wsGorilla "github.com/gorilla/websocket" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" ws "golang.org/x/net/websocket" - wsGorilla "gx/ipfs/QmZH5VXfAJouGMyCCHTRPGCT3e5MG9Lu78Ln3YAYW1XTts/websocket" ) var WsProtocol = ma.Protocol{ From 973781e05d15b5bfde97e45e2062eb77af339cb4 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sat, 8 Jul 2017 09:44:57 -0700 Subject: [PATCH 0250/3965] first commit --- p2p/net/connmgr/connmgr.go | 170 +++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 p2p/net/connmgr/connmgr.go diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go new file mode 100644 index 0000000000..94ea6c4f44 --- /dev/null +++ b/p2p/net/connmgr/connmgr.go @@ -0,0 +1,170 @@ +package connmgr + +import ( + "context" + "sort" + "sync" + "time" + + inet "gx/ipfs/QmRscs8KxrSmSv4iuevHv8JfuUzHBMoqiaHzxfDRiksd6e/go-libp2p-net" + logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" + ma "gx/ipfs/QmcyqRMCAXVtYPS4DiBrA7sezL9rRGfW8Ctx7cywL4TXJj/go-multiaddr" +) + +var log = logging.Logger("connmgr") + +type ConnManager struct { + HighWater int + LowWater int + + GracePeriod time.Duration + + conns map[inet.Conn]connInfo + + lk sync.Mutex +} + +func NewConnManager(low, hi int) *ConnManager { + return &ConnManager{ + HighWater: hi, + LowWater: low, + GracePeriod: time.Second * 10, + conns: make(map[inet.Conn]connInfo), + } +} + +type connInfo struct { + tags map[string]int + value int + c inet.Conn + closed bool + + firstSeen time.Time +} + +func (cm *ConnManager) TrimOpenConns(ctx context.Context) { + defer log.EventBegin(ctx, "connCleanup").Done() + cm.lk.Lock() + defer cm.lk.Unlock() + + if len(cm.conns) < cm.LowWater { + log.Info("open connection count below limit") + return + } + + var infos []connInfo + + for _, inf := range cm.conns { + infos = append(infos, inf) + } + + sort.Slice(infos, func(i, j int) bool { + if infos[i].closed { + return true + } + if infos[j].closed { + return false + } + + return infos[i].value < infos[j].value + }) + + toclose := infos[:len(infos)-cm.LowWater] + + for _, inf := range toclose { + if time.Since(inf.firstSeen) < cm.GracePeriod { + continue + } + + log.Info("closing conn: ", inf.c.RemotePeer()) + log.Event(ctx, "closeConn", inf.c.RemotePeer()) + inf.c.Close() + inf.closed = true + } + + if len(cm.conns) > cm.HighWater { + log.Error("still over high water mark after trimming connections") + } +} + +func (cm *ConnManager) TagConn(c inet.Conn, tag string, val int) { + cm.lk.Lock() + defer cm.lk.Unlock() + + ci, ok := cm.conns[c] + if !ok { + log.Error("tried to tag untracked conn: ", c.RemotePeer()) + return + } + + ci.value += (val - ci.tags[tag]) + ci.tags[tag] = val +} + +func (cm *ConnManager) UntagConn(c inet.Conn, tag string) { + cm.lk.Lock() + defer cm.lk.Unlock() + + ci, ok := cm.conns[c] + if !ok { + log.Error("tried to remove tag on untracked conn: ", c.RemotePeer()) + return + } + + ci.value -= ci.tags[tag] + delete(ci.tags, tag) +} + +func (cm *ConnManager) Notifee() *cmNotifee { + return (*cmNotifee)(cm) +} + +type cmNotifee ConnManager + +func (nn *cmNotifee) cm() *ConnManager { + return (*ConnManager)(nn) +} + +func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { + cm := nn.cm() + + cm.lk.Lock() + defer cm.lk.Unlock() + + cinfo, ok := cm.conns[c] + if ok { + log.Error("received connected notification for conn we are already tracking: ", c.RemotePeer()) + return + } + + cinfo = connInfo{ + firstSeen: time.Now(), + tags: make(map[string]int), + c: c, + } + cm.conns[c] = cinfo + + if len(cm.conns) > nn.HighWater { + go cm.TrimOpenConns(context.Background()) + } +} + +func (nn *cmNotifee) Disconnected(n inet.Network, c inet.Conn) { + cm := nn.cm() + + cm.lk.Lock() + defer cm.lk.Unlock() + + _, ok := cm.conns[c] + if !ok { + log.Error("received disconnected notification for conn we are not tracking: ", c.RemotePeer()) + return + } + + delete(cm.conns, c) +} + +func (nn *cmNotifee) Listen(n inet.Network, addr ma.Multiaddr) {} +func (nn *cmNotifee) ListenClose(n inet.Network, addr ma.Multiaddr) {} +func (nn *cmNotifee) OpenedStream(inet.Network, inet.Stream) {} +func (nn *cmNotifee) ClosedStream(inet.Network, inet.Stream) {} From 84a5d64f5a133eb4bcbc17d67b21361cf2a18463 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Sun, 9 Jul 2017 21:04:38 +0200 Subject: [PATCH 0251/3965] refactor: simplify and split into multiple files --- p2p/transport/websocket/addrs.go | 58 +++++ p2p/transport/websocket/addrs_test.go | 23 ++ p2p/transport/websocket/conn.go | 75 ++++++ p2p/transport/websocket/dialer.go | 41 +++ p2p/transport/websocket/listener.go | 73 ++++++ p2p/transport/websocket/websocket.go | 233 +----------------- .../{ws_test.go => websocket_test.go} | 12 - 7 files changed, 281 insertions(+), 234 deletions(-) create mode 100644 p2p/transport/websocket/addrs.go create mode 100644 p2p/transport/websocket/addrs_test.go create mode 100644 p2p/transport/websocket/conn.go create mode 100644 p2p/transport/websocket/dialer.go create mode 100644 p2p/transport/websocket/listener.go rename p2p/transport/websocket/{ws_test.go => websocket_test.go} (78%) diff --git a/p2p/transport/websocket/addrs.go b/p2p/transport/websocket/addrs.go new file mode 100644 index 0000000000..5e7cb396dd --- /dev/null +++ b/p2p/transport/websocket/addrs.go @@ -0,0 +1,58 @@ +package websocket + +import ( + "fmt" + "net" + "net/url" + + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" + ws "golang.org/x/net/websocket" +) + +func ConvertWebsocketMultiaddrToNetAddr(maddr ma.Multiaddr) (net.Addr, error) { + _, host, err := manet.DialArgs(maddr) + if err != nil { + return nil, err + } + + a := &ws.Addr{ + URL: &url.URL{ + Host: host, + }, + } + return a, nil +} + +func ParseWebsocketNetAddr(a net.Addr) (ma.Multiaddr, error) { + wsa, ok := a.(*ws.Addr) + if !ok { + return nil, fmt.Errorf("not a websocket address") + } + + tcpaddr, err := net.ResolveTCPAddr("tcp", wsa.Host) + if err != nil { + return nil, err + } + + tcpma, err := manet.FromNetAddr(tcpaddr) + if err != nil { + return nil, err + } + + wsma, err := ma.NewMultiaddr("/ws") + if err != nil { + return nil, err + } + + return tcpma.Encapsulate(wsma), nil +} + +func parseMultiaddr(a ma.Multiaddr) (string, error) { + _, host, err := manet.DialArgs(a) + if err != nil { + return "", err + } + + return "ws://" + host, nil +} diff --git a/p2p/transport/websocket/addrs_test.go b/p2p/transport/websocket/addrs_test.go new file mode 100644 index 0000000000..b85bdfda89 --- /dev/null +++ b/p2p/transport/websocket/addrs_test.go @@ -0,0 +1,23 @@ +package websocket + +import ( + "fmt" + "testing" + + ma "github.com/multiformats/go-multiaddr" +) + +func TestMultiaddrParsing(t *testing.T) { + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") + if err != nil { + t.Fatal(err) + } + + res, err := parseMultiaddr(addr) + if err != nil { + t.Fatal(err) + } + if res != "ws://127.0.0.1:5555" { + t.Fatal(fmt.Errorf("%s != ws://127.0.0.1:5555", res)) + } +} diff --git a/p2p/transport/websocket/conn.go b/p2p/transport/websocket/conn.go new file mode 100644 index 0000000000..9a48839109 --- /dev/null +++ b/p2p/transport/websocket/conn.go @@ -0,0 +1,75 @@ +package websocket + +import ( + "net" + "time" + + ws "github.com/gorilla/websocket" +) + +var _ net.Conn = (*Conn)(nil) + +// Conn implements net.Conn interface for gorilla/websocket. +type Conn struct { + *ws.Conn + DefaultMessageType int + done func() +} + +func (c *Conn) Read(b []byte) (n int, err error) { + _, r, err := c.Conn.NextReader() + if err != nil { + return 0, err + } + + return r.Read(b) +} + +func (c *Conn) Write(b []byte) (n int, err error) { + if err := c.Conn.WriteMessage(c.DefaultMessageType, b); err != nil { + return 0, err + } + + return len(b), nil +} + +func (c *Conn) Close() error { + if c.done != nil { + c.done() + } + + return c.Conn.Close() +} + +func (c *Conn) LocalAddr() net.Addr { + return c.Conn.LocalAddr() +} + +func (c *Conn) RemoteAddr() net.Addr { + return c.Conn.RemoteAddr() +} + +func (c *Conn) SetDeadline(t time.Time) error { + if err := c.SetReadDeadline(t); err != nil { + return err + } + + return c.SetWriteDeadline(t) +} + +func (c *Conn) SetReadDeadline(t time.Time) error { + return c.Conn.SetReadDeadline(t) +} + +func (c *Conn) SetWriteDeadline(t time.Time) error { + return c.Conn.SetWriteDeadline(t) +} + +// NewConn creates a Conn given a regular gorilla/websocket Conn. +func NewConn(raw *ws.Conn, done func()) *Conn { + return &Conn{ + Conn: raw, + DefaultMessageType: ws.BinaryMessage, + done: done, + } +} diff --git a/p2p/transport/websocket/dialer.go b/p2p/transport/websocket/dialer.go new file mode 100644 index 0000000000..4c74df39d7 --- /dev/null +++ b/p2p/transport/websocket/dialer.go @@ -0,0 +1,41 @@ +package websocket + +import ( + "context" + + ws "github.com/gorilla/websocket" + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +type dialer struct{} + +func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { + return d.DialContext(context.Background(), raddr) +} + +func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { + wsurl, err := parseMultiaddr(raddr) + if err != nil { + return nil, err + } + + wscon, _, err := ws.DefaultDialer.Dial(wsurl, nil) + if err != nil { + return nil, err + } + + mnc, err := manet.WrapNetConn(NewConn(wscon, nil)) + if err != nil { + return nil, err + } + + return &wsConn{ + Conn: mnc, + }, nil +} + +func (d *dialer) Matches(a ma.Multiaddr) bool { + return WsFmt.Matches(a) +} diff --git a/p2p/transport/websocket/listener.go b/p2p/transport/websocket/listener.go new file mode 100644 index 0000000000..785173a0c7 --- /dev/null +++ b/p2p/transport/websocket/listener.go @@ -0,0 +1,73 @@ +package websocket + +import ( + "context" + "fmt" + "net/http" + "net/url" + + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +type wsConn struct { + manet.Conn + t tpt.Transport +} + +var _ tpt.Conn = (*wsConn)(nil) + +func (c *wsConn) Transport() tpt.Transport { + return c.t +} + +type listener struct { + manet.Listener + + incoming chan *Conn + + tpt tpt.Transport + + origin *url.URL +} + +func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { + c, err := upgrader.Upgrade(w, r, nil) + if err != nil { + http.Error(w, "Failed to upgrade websocket", 400) + return + } + + ctx, cancel := context.WithCancel(context.Background()) + l.incoming <- NewConn(c, cancel) + + // wait until conn gets closed, otherwise the handler closes it early + <-ctx.Done() +} + +func (l *listener) Accept() (tpt.Conn, error) { + c, ok := <-l.incoming + if !ok { + return nil, fmt.Errorf("listener is closed") + } + + mnc, err := manet.WrapNetConn(c) + if err != nil { + return nil, err + } + + return &wsConn{ + Conn: mnc, + t: l.tpt, + }, nil +} + +func (l *listener) Multiaddr() ma.Multiaddr { + wsma, err := ma.NewMultiaddr("/ws") + if err != nil { + panic(err) + } + + return l.Listener.Multiaddr().Encapsulate(wsma) +} diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 83a4e90aab..55ea54d994 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -1,29 +1,30 @@ +// Package websocket implements a websocket based transport for go-libp2p. package websocket import ( - "context" "fmt" - "net" "net/http" "net/url" - "time" - wsGorilla "github.com/gorilla/websocket" + ws "github.com/gorilla/websocket" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" + mafmt "github.com/whyrusleeping/mafmt" - ws "golang.org/x/net/websocket" ) +// WsProtocol is the multiaddr protocol definition for this transport. var WsProtocol = ma.Protocol{ Code: 477, Name: "ws", VCode: ma.CodeToVarint(477), } +// WsFmt is multiaddr formatter for WsProtocol var WsFmt = mafmt.And(mafmt.TCP, mafmt.Base(WsProtocol.Code)) +// WsCodec is the multiaddr-net codec definition for the websocket transport var WsCodec = &manet.NetCodec{ NetAddrNetworks: []string{"websocket"}, ProtocolName: "ws", @@ -32,7 +33,7 @@ var WsCodec = &manet.NetCodec{ } // Default gorilla upgrader -var upgrader = wsGorilla.Upgrader{} +var upgrader = ws.Upgrader{} func init() { err := ma.AddProtocol(WsProtocol) @@ -43,46 +44,11 @@ func init() { manet.RegisterNetCodec(WsCodec) } -func ConvertWebsocketMultiaddrToNetAddr(maddr ma.Multiaddr) (net.Addr, error) { - _, host, err := manet.DialArgs(maddr) - if err != nil { - return nil, err - } - - a := &ws.Addr{ - URL: &url.URL{ - Host: host, - }, - } - return a, nil -} - -func ParseWebsocketNetAddr(a net.Addr) (ma.Multiaddr, error) { - wsa, ok := a.(*ws.Addr) - if !ok { - return nil, fmt.Errorf("not a websocket address") - } - - tcpaddr, err := net.ResolveTCPAddr("tcp", wsa.Host) - if err != nil { - return nil, err - } - - tcpma, err := manet.FromNetAddr(tcpaddr) - if err != nil { - return nil, err - } - - wsma, err := ma.NewMultiaddr("/ws") - if err != nil { - return nil, err - } - - return tcpma.Encapsulate(wsma), nil -} - +// WebsocketTransport is the actual go-libp2p transport type WebsocketTransport struct{} +var _ tpt.Transport = (*WebsocketTransport)(nil) + func (t *WebsocketTransport) Matches(a ma.Multiaddr) bool { return WsFmt.Matches(a) } @@ -91,79 +57,6 @@ func (t *WebsocketTransport) Dialer(_ ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Di return &dialer{}, nil } -type dialer struct{} - -func parseMultiaddr(a ma.Multiaddr) (string, error) { - _, host, err := manet.DialArgs(a) - if err != nil { - return "", err - } - - return "ws://" + host, nil -} - -func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { - return d.DialContext(context.Background(), raddr) -} - -func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { - wsurl, err := parseMultiaddr(raddr) - if err != nil { - return nil, err - } - - // TODO: figure out origins, probably don't work for us - // header := http.Header{} - // header.Set("Origin", "http://127.0.0.1:0/") - wscon, _, err := wsGorilla.DefaultDialer.Dial(wsurl, nil) - if err != nil { - return nil, err - } - - mnc, err := manet.WrapNetConn(NewGorillaNetConn(wscon)) - if err != nil { - return nil, err - } - - return &wsConn{ - Conn: mnc, - }, nil -} - -func (d *dialer) Matches(a ma.Multiaddr) bool { - return WsFmt.Matches(a) -} - -type wsConn struct { - manet.Conn - t tpt.Transport -} - -func (c *wsConn) Transport() tpt.Transport { - return c.t -} - -type listener struct { - manet.Listener - - incoming chan *conn - - tpt tpt.Transport - - origin *url.URL -} - -type conn struct { - *GorillaNetConn - - done func() -} - -func (c *conn) Close() error { - c.done() - return c.GorillaNetConn.Close() -} - func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { list, err := manet.Listen(a) if err != nil { @@ -185,112 +78,8 @@ func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { func (t *WebsocketTransport) wrapListener(l manet.Listener, origin *url.URL) *listener { return &listener{ Listener: l, - incoming: make(chan *conn), + incoming: make(chan *Conn), tpt: t, origin: origin, } } - -func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { - c, err := upgrader.Upgrade(w, r, nil) - if err != nil { - http.Error(w, "Failed to upgrade websocket", 400) - return - } - - ctx, cancel := context.WithCancel(context.Background()) - - wrapped := NewGorillaNetConn(c) - l.incoming <- &conn{ - GorillaNetConn: wrapped, - done: cancel, - } - - // wait until conn gets closed, otherwise the handler closes it early - <-ctx.Done() -} - -func (l *listener) Accept() (tpt.Conn, error) { - c, ok := <-l.incoming - if !ok { - return nil, fmt.Errorf("listener is closed") - } - - mnc, err := manet.WrapNetConn(c) - if err != nil { - return nil, err - } - - return &wsConn{ - Conn: mnc, - t: l.tpt, - }, nil -} - -func (l *listener) Multiaddr() ma.Multiaddr { - wsma, err := ma.NewMultiaddr("/ws") - if err != nil { - panic(err) - } - - return l.Listener.Multiaddr().Encapsulate(wsma) -} - -var _ tpt.Transport = (*WebsocketTransport)(nil) - -type GorillaNetConn struct { - Inner *wsGorilla.Conn - DefaultMessageType int -} - -func (c *GorillaNetConn) Read(b []byte) (n int, err error) { - _, r, err := c.Inner.NextReader() - if err != nil { - return 0, err - } - - return r.Read(b) -} - -func (c *GorillaNetConn) Write(b []byte) (n int, err error) { - if err := c.Inner.WriteMessage(c.DefaultMessageType, b); err != nil { - return 0, err - } - - return len(b), nil -} - -func (c *GorillaNetConn) Close() error { - return c.Inner.Close() -} - -func (c *GorillaNetConn) LocalAddr() net.Addr { - return c.Inner.LocalAddr() -} - -func (c *GorillaNetConn) RemoteAddr() net.Addr { - return c.Inner.RemoteAddr() -} - -func (c *GorillaNetConn) SetDeadline(t time.Time) error { - if err := c.SetReadDeadline(t); err != nil { - return err - } - - return c.SetWriteDeadline(t) -} - -func (c *GorillaNetConn) SetReadDeadline(t time.Time) error { - return c.Inner.SetReadDeadline(t) -} - -func (c *GorillaNetConn) SetWriteDeadline(t time.Time) error { - return c.Inner.SetWriteDeadline(t) -} - -func NewGorillaNetConn(raw *wsGorilla.Conn) *GorillaNetConn { - return &GorillaNetConn{ - Inner: raw, - DefaultMessageType: wsGorilla.BinaryMessage, - } -} diff --git a/p2p/transport/websocket/ws_test.go b/p2p/transport/websocket/websocket_test.go similarity index 78% rename from p2p/transport/websocket/ws_test.go rename to p2p/transport/websocket/websocket_test.go index 1667ac42b8..f56159f200 100644 --- a/p2p/transport/websocket/ws_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -7,18 +7,6 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -func TestMultiaddrParsing(t *testing.T) { - addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") - if err != nil { - t.Fatal(err) - } - - _, err = parseMultiaddr(addr) - if err != nil { - t.Fatal(err) - } -} - func TestWebsocketListen(t *testing.T) { zero, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0/ws") if err != nil { From 328039237d7868ee510c67b9cd681a41e501cfb0 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Sun, 9 Jul 2017 21:47:26 +0200 Subject: [PATCH 0252/3965] more tests --- p2p/transport/websocket/addrs.go | 31 +++++++++++----- p2p/transport/websocket/addrs_test.go | 52 ++++++++++++++++++++++++--- 2 files changed, 71 insertions(+), 12 deletions(-) diff --git a/p2p/transport/websocket/addrs.go b/p2p/transport/websocket/addrs.go index 5e7cb396dd..e5dbc46e43 100644 --- a/p2p/transport/websocket/addrs.go +++ b/p2p/transport/websocket/addrs.go @@ -7,25 +7,40 @@ import ( ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" - ws "golang.org/x/net/websocket" ) +// Addr is an implementation of net.Addr for WebSocket. +type Addr struct { + *url.URL +} + +var _ net.Addr = (*Addr)(nil) + +// Network returns the network type for a WebSocket, "websocket". +func (addr *Addr) Network() string { + return "websocket" +} + +// NewAddr creates a new Addr using the given host string +func NewAddr(host string) *Addr { + return &Addr{ + URL: &url.URL{ + Host: host, + }, + } +} + func ConvertWebsocketMultiaddrToNetAddr(maddr ma.Multiaddr) (net.Addr, error) { _, host, err := manet.DialArgs(maddr) if err != nil { return nil, err } - a := &ws.Addr{ - URL: &url.URL{ - Host: host, - }, - } - return a, nil + return NewAddr(host), nil } func ParseWebsocketNetAddr(a net.Addr) (ma.Multiaddr, error) { - wsa, ok := a.(*ws.Addr) + wsa, ok := a.(*Addr) if !ok { return nil, fmt.Errorf("not a websocket address") } diff --git a/p2p/transport/websocket/addrs_test.go b/p2p/transport/websocket/addrs_test.go index b85bdfda89..d962760088 100644 --- a/p2p/transport/websocket/addrs_test.go +++ b/p2p/transport/websocket/addrs_test.go @@ -1,7 +1,7 @@ package websocket import ( - "fmt" + "net/url" "testing" ma "github.com/multiformats/go-multiaddr" @@ -13,11 +13,55 @@ func TestMultiaddrParsing(t *testing.T) { t.Fatal(err) } - res, err := parseMultiaddr(addr) + wsaddr, err := parseMultiaddr(addr) if err != nil { t.Fatal(err) } - if res != "ws://127.0.0.1:5555" { - t.Fatal(fmt.Errorf("%s != ws://127.0.0.1:5555", res)) + if wsaddr != "ws://127.0.0.1:5555" { + t.Fatalf("expected ws://127.0.0.1:5555, got %s", wsaddr) + } +} + +type httpAddr struct { + *url.URL +} + +func (addr *httpAddr) Network() string { + return "http" +} + +func TestParseWebsocketNetAddr(t *testing.T) { + notWs := &httpAddr{&url.URL{Host: "http://127.0.0.1:1234"}} + _, err := ParseWebsocketNetAddr(notWs) + if err.Error() != "not a websocket address" { + t.Fatalf("expect \"not a websocket address\", got \"%s\"", err) + } + + wsAddr := NewAddr("127.0.0.1:5555") + parsed, err := ParseWebsocketNetAddr(wsAddr) + if err != nil { + t.Fatal(err) + } + + if parsed.String() != "/ip4/127.0.0.1/tcp/5555/ws" { + t.Fatalf("expected \"/ip4/127.0.0.1/tcp/5555/ws\", got \"%s\"", parsed.String()) + } +} + +func TestConvertWebsocketMultiaddrToNetAddr(t *testing.T) { + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") + if err != nil { + t.Fatal(err) + } + + wsaddr, err := ConvertWebsocketMultiaddrToNetAddr(addr) + if err != nil { + t.Fatal(err) + } + if wsaddr.String() != "//127.0.0.1:5555" { + t.Fatalf("expected //127.0.0.1:5555, got %s", wsaddr) + } + if wsaddr.Network() != "websocket" { + t.Fatalf("expected network: \"websocket\", got \"%s\"", wsaddr.Network()) } } From 9b327a62a9cc9e479cfff110f5a30ca016c6f3c0 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Sun, 9 Jul 2017 21:56:19 +0200 Subject: [PATCH 0253/3965] add dialer.Matches test --- p2p/transport/websocket/dialer_test.go | 31 ++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 p2p/transport/websocket/dialer_test.go diff --git a/p2p/transport/websocket/dialer_test.go b/p2p/transport/websocket/dialer_test.go new file mode 100644 index 0000000000..2e8b5caf59 --- /dev/null +++ b/p2p/transport/websocket/dialer_test.go @@ -0,0 +1,31 @@ +package websocket + +import ( + "testing" + + ma "github.com/multiformats/go-multiaddr" +) + +func TestDialerMatches(t *testing.T) { + addrWs, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") + if err != nil { + t.Fatal(err) + } + + addrTcp, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555") + if err != nil { + t.Fatal(err) + } + + d := &dialer{} + matchTrue := d.Matches(addrWs) + matchFalse := d.Matches(addrTcp) + + if !matchTrue { + t.Fatal("expected to match websocket maddr, but did not") + } + + if matchFalse { + t.Fatal("expected to not match tcp maddr, but did") + } +} From 29a8c4e1bfdc378810c7713bf21b1efdd7736878 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 10 Jul 2017 08:04:31 -0700 Subject: [PATCH 0254/3965] add base repo files --- p2p/net/connmgr/connmgr.go | 112 +++++++++++++++++++++---------------- 1 file changed, 65 insertions(+), 47 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 94ea6c4f44..f9f0b34f09 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -6,9 +6,10 @@ import ( "sync" "time" - inet "gx/ipfs/QmRscs8KxrSmSv4iuevHv8JfuUzHBMoqiaHzxfDRiksd6e/go-libp2p-net" - logging "gx/ipfs/QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52/go-log" - ma "gx/ipfs/QmcyqRMCAXVtYPS4DiBrA7sezL9rRGfW8Ctx7cywL4TXJj/go-multiaddr" + logging "github.com/ipfs/go-log" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) var log = logging.Logger("connmgr") @@ -19,9 +20,12 @@ type ConnManager struct { GracePeriod time.Duration - conns map[inet.Conn]connInfo + peers map[peer.ID]peerInfo + connCount int lk sync.Mutex + + lastTrim time.Time } func NewConnManager(low, hi int) *ConnManager { @@ -29,43 +33,37 @@ func NewConnManager(low, hi int) *ConnManager { HighWater: hi, LowWater: low, GracePeriod: time.Second * 10, - conns: make(map[inet.Conn]connInfo), + peers: make(map[peer.ID]peerInfo), } } -type connInfo struct { - tags map[string]int - value int - c inet.Conn - closed bool +type peerInfo struct { + tags map[string]int + value int + + conns map[inet.Conn]time.Time firstSeen time.Time } func (cm *ConnManager) TrimOpenConns(ctx context.Context) { - defer log.EventBegin(ctx, "connCleanup").Done() cm.lk.Lock() defer cm.lk.Unlock() + defer log.EventBegin(ctx, "connCleanup").Done() + cm.lastTrim = time.Now() - if len(cm.conns) < cm.LowWater { + if len(cm.peers) < cm.LowWater { log.Info("open connection count below limit") return } - var infos []connInfo + var infos []peerInfo - for _, inf := range cm.conns { + for _, inf := range cm.peers { infos = append(infos, inf) } sort.Slice(infos, func(i, j int) bool { - if infos[i].closed { - return true - } - if infos[j].closed { - return false - } - return infos[i].value < infos[j].value }) @@ -76,43 +74,45 @@ func (cm *ConnManager) TrimOpenConns(ctx context.Context) { continue } - log.Info("closing conn: ", inf.c.RemotePeer()) - log.Event(ctx, "closeConn", inf.c.RemotePeer()) - inf.c.Close() - inf.closed = true + // TODO: if a peer has more than one connection, maybe only close one? + for c, _ := range inf.conns { + log.Info("closing conn: ", c.RemotePeer()) + log.Event(ctx, "closeConn", c.RemotePeer()) + c.Close() + } } - if len(cm.conns) > cm.HighWater { + if len(cm.peers) > cm.HighWater { log.Error("still over high water mark after trimming connections") } } -func (cm *ConnManager) TagConn(c inet.Conn, tag string, val int) { +func (cm *ConnManager) TagConn(p peer.ID, tag string, val int) { cm.lk.Lock() defer cm.lk.Unlock() - ci, ok := cm.conns[c] + pi, ok := cm.peers[p] if !ok { - log.Error("tried to tag untracked conn: ", c.RemotePeer()) + log.Error("tried to tag conn from untracked peer: ", p) return } - ci.value += (val - ci.tags[tag]) - ci.tags[tag] = val + pi.value += (val - pi.tags[tag]) + pi.tags[tag] = val } -func (cm *ConnManager) UntagConn(c inet.Conn, tag string) { +func (cm *ConnManager) UntagConn(p peer.ID, tag string) { cm.lk.Lock() defer cm.lk.Unlock() - ci, ok := cm.conns[c] + pi, ok := cm.peers[p] if !ok { - log.Error("tried to remove tag on untracked conn: ", c.RemotePeer()) + log.Error("tried to remove tag from untracked peer: ", p) return } - ci.value -= ci.tags[tag] - delete(ci.tags, tag) + pi.value -= pi.tags[tag] + delete(pi.tags, tag) } func (cm *ConnManager) Notifee() *cmNotifee { @@ -131,21 +131,29 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { cm.lk.Lock() defer cm.lk.Unlock() - cinfo, ok := cm.conns[c] + pinfo, ok := cm.peers[c.RemotePeer()] + if !ok { + pinfo = peerInfo{ + firstSeen: time.Now(), + tags: make(map[string]int), + conns: make(map[inet.Conn]time.Time), + } + cm.peers[c.RemotePeer()] = pinfo + } + + _, ok = pinfo.conns[c] if ok { log.Error("received connected notification for conn we are already tracking: ", c.RemotePeer()) return } - cinfo = connInfo{ - firstSeen: time.Now(), - tags: make(map[string]int), - c: c, - } - cm.conns[c] = cinfo + pinfo.conns[c] = time.Now() + cm.connCount++ - if len(cm.conns) > nn.HighWater { - go cm.TrimOpenConns(context.Background()) + if cm.connCount > nn.HighWater { + if cm.lastTrim.IsZero() || time.Since(cm.lastTrim) > time.Second*10 { + go cm.TrimOpenConns(context.Background()) + } } } @@ -155,13 +163,23 @@ func (nn *cmNotifee) Disconnected(n inet.Network, c inet.Conn) { cm.lk.Lock() defer cm.lk.Unlock() - _, ok := cm.conns[c] + cinf, ok := cm.peers[c.RemotePeer()] + if !ok { + log.Error("received disconnected notification for peer we are not tracking: ", c.RemotePeer()) + return + } + + _, ok = cinf.conns[c] if !ok { log.Error("received disconnected notification for conn we are not tracking: ", c.RemotePeer()) return } - delete(cm.conns, c) + delete(cinf.conns, c) + cm.connCount-- + if len(cinf.conns) == 0 { + delete(cm.peers, c.RemotePeer()) + } } func (nn *cmNotifee) Listen(n inet.Network, addr ma.Multiaddr) {} From 4b860973dcbb27993f338401d422a51867587baf Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 10 Jul 2017 10:45:11 -0700 Subject: [PATCH 0255/3965] Add a test --- p2p/net/connmgr/connmgr.go | 60 +++++++++++++++++------------ p2p/net/connmgr/connmgr_test.go | 67 +++++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+), 25 deletions(-) create mode 100644 p2p/net/connmgr/connmgr_test.go diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index f9f0b34f09..fb77484e01 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -14,13 +14,20 @@ import ( var log = logging.Logger("connmgr") -type ConnManager struct { - HighWater int - LowWater int +type ConnManager interface { + TagPeer(peer.ID, string, int) + UntagPeer(peer.ID, string) + TrimOpenConns(context.Context) + Notifee() inet.Notifiee +} + +type connManager struct { + highWater int + lowWater int - GracePeriod time.Duration + gracePeriod time.Duration - peers map[peer.ID]peerInfo + peers map[peer.ID]*peerInfo connCount int lk sync.Mutex @@ -28,12 +35,14 @@ type ConnManager struct { lastTrim time.Time } -func NewConnManager(low, hi int) *ConnManager { - return &ConnManager{ - HighWater: hi, - LowWater: low, - GracePeriod: time.Second * 10, - peers: make(map[peer.ID]peerInfo), +var DefaultGracePeriod = time.Second * 10 + +func NewConnManager(low, hi int, grace time.Duration) ConnManager { + return &connManager{ + highWater: hi, + lowWater: low, + gracePeriod: grace, + peers: make(map[peer.ID]*peerInfo), } } @@ -46,18 +55,18 @@ type peerInfo struct { firstSeen time.Time } -func (cm *ConnManager) TrimOpenConns(ctx context.Context) { +func (cm *connManager) TrimOpenConns(ctx context.Context) { cm.lk.Lock() defer cm.lk.Unlock() defer log.EventBegin(ctx, "connCleanup").Done() cm.lastTrim = time.Now() - if len(cm.peers) < cm.LowWater { + if len(cm.peers) < cm.lowWater { log.Info("open connection count below limit") return } - var infos []peerInfo + var infos []*peerInfo for _, inf := range cm.peers { infos = append(infos, inf) @@ -67,10 +76,11 @@ func (cm *ConnManager) TrimOpenConns(ctx context.Context) { return infos[i].value < infos[j].value }) - toclose := infos[:len(infos)-cm.LowWater] + close_count := len(infos) - cm.lowWater + toclose := infos[:close_count] for _, inf := range toclose { - if time.Since(inf.firstSeen) < cm.GracePeriod { + if time.Since(inf.firstSeen) < cm.gracePeriod { continue } @@ -82,12 +92,12 @@ func (cm *ConnManager) TrimOpenConns(ctx context.Context) { } } - if len(cm.peers) > cm.HighWater { + if len(cm.peers) > cm.highWater { log.Error("still over high water mark after trimming connections") } } -func (cm *ConnManager) TagConn(p peer.ID, tag string, val int) { +func (cm *connManager) TagPeer(p peer.ID, tag string, val int) { cm.lk.Lock() defer cm.lk.Unlock() @@ -101,7 +111,7 @@ func (cm *ConnManager) TagConn(p peer.ID, tag string, val int) { pi.tags[tag] = val } -func (cm *ConnManager) UntagConn(p peer.ID, tag string) { +func (cm *connManager) UntagPeer(p peer.ID, tag string) { cm.lk.Lock() defer cm.lk.Unlock() @@ -115,14 +125,14 @@ func (cm *ConnManager) UntagConn(p peer.ID, tag string) { delete(pi.tags, tag) } -func (cm *ConnManager) Notifee() *cmNotifee { +func (cm *connManager) Notifee() inet.Notifiee { return (*cmNotifee)(cm) } -type cmNotifee ConnManager +type cmNotifee connManager -func (nn *cmNotifee) cm() *ConnManager { - return (*ConnManager)(nn) +func (nn *cmNotifee) cm() *connManager { + return (*connManager)(nn) } func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { @@ -133,7 +143,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { pinfo, ok := cm.peers[c.RemotePeer()] if !ok { - pinfo = peerInfo{ + pinfo = &peerInfo{ firstSeen: time.Now(), tags: make(map[string]int), conns: make(map[inet.Conn]time.Time), @@ -150,7 +160,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { pinfo.conns[c] = time.Now() cm.connCount++ - if cm.connCount > nn.HighWater { + if cm.connCount > nn.highWater { if cm.lastTrim.IsZero() || time.Since(cm.lastTrim) > time.Second*10 { go cm.TrimOpenConns(context.Background()) } diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go new file mode 100644 index 0000000000..8c30e9e5d9 --- /dev/null +++ b/p2p/net/connmgr/connmgr_test.go @@ -0,0 +1,67 @@ +package connmgr + +import ( + "context" + "testing" + + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + tu "github.com/libp2p/go-testutil" +) + +type tconn struct { + inet.Conn + peer peer.ID + closed bool +} + +func (c *tconn) Close() error { + c.closed = true + return nil +} + +func (c *tconn) RemotePeer() peer.ID { + return c.peer +} + +func randConn(t *testing.T) inet.Conn { + pid := tu.RandPeerIDFatal(t) + return &tconn{peer: pid} +} + +func TestConnTrimming(t *testing.T) { + cm := NewConnManager(200, 300, 0) + not := cm.Notifee() + + var conns []inet.Conn + for i := 0; i < 300; i++ { + rc := randConn(t) + conns = append(conns, rc) + not.Connected(nil, rc) + } + + for _, c := range conns { + if c.(*tconn).closed { + t.Fatal("nothing should be closed yet") + } + } + + for i := 0; i < 100; i++ { + cm.TagPeer(conns[i].RemotePeer(), "foo", 10) + } + + cm.TagPeer(conns[299].RemotePeer(), "badfoo", -5) + + cm.TrimOpenConns(context.Background()) + + for i := 0; i < 100; i++ { + c := conns[i] + if c.(*tconn).closed { + t.Fatal("these shouldnt be closed") + } + } + + if !conns[299].(*tconn).closed { + t.Fatal("conn with bad tag should have gotten closed") + } +} From e4074aaa497c7698e293267fa280c95b9f049b8c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 10 Jul 2017 12:22:53 -0700 Subject: [PATCH 0256/3965] make it inspectable and disableable --- p2p/net/connmgr/connmgr.go | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index fb77484e01..cbad648130 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -17,6 +17,7 @@ var log = logging.Logger("connmgr") type ConnManager interface { TagPeer(peer.ID, string, int) UntagPeer(peer.ID, string) + GetTagInfo(peer.ID) *TagInfo TrimOpenConns(context.Context) Notifee() inet.Notifiee } @@ -55,10 +56,21 @@ type peerInfo struct { firstSeen time.Time } +type TagInfo struct { + FirstSeen time.Time + Value int + Tags map[string]int + Conns map[string]time.Time +} + func (cm *connManager) TrimOpenConns(ctx context.Context) { cm.lk.Lock() defer cm.lk.Unlock() defer log.EventBegin(ctx, "connCleanup").Done() + if cm.lowWater == 0 || cm.highWater == 0 { + // disabled + return + } cm.lastTrim = time.Now() if len(cm.peers) < cm.lowWater { @@ -97,6 +109,32 @@ func (cm *connManager) TrimOpenConns(ctx context.Context) { } } +func (cm *connManager) GetTagInfo(p peer.ID) *TagInfo { + cm.lk.Lock() + defer cm.lk.Unlock() + + pi, ok := cm.peers[p] + if !ok { + return nil + } + + out := &TagInfo{ + FirstSeen: pi.firstSeen, + Value: pi.value, + Tags: make(map[string]int), + Conns: make(map[string]time.Time), + } + + for t, v := range pi.tags { + out.Tags[t] = v + } + for c, t := range pi.conns { + out.Conns[c.RemoteMultiaddr().String()] = t + } + + return out +} + func (cm *connManager) TagPeer(p peer.ID, tag string, val int) { cm.lk.Lock() defer cm.lk.Unlock() From ce255882ef84dab4f05ff3aa9820cb26c16fc767 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 11 Jul 2017 23:02:35 -0700 Subject: [PATCH 0257/3965] add in connmgr interfaces --- p2p/host/blank/blank.go | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go index 3c75331c48..ec4fe010ae 100644 --- a/p2p/host/blank/blank.go +++ b/p2p/host/blank/blank.go @@ -5,6 +5,7 @@ import ( "io" logging "github.com/ipfs/go-log" + connmgr "github.com/libp2p/go-libp2p-connmgr" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" @@ -18,14 +19,16 @@ var log = logging.Logger("blankhost") // BlankHost is the thinnest implementation of the host.Host interface type BlankHost struct { - n inet.Network - mux *mstream.MultistreamMuxer + n inet.Network + mux *mstream.MultistreamMuxer + cmgr connmgr.ConnManager } func NewBlankHost(n inet.Network) *BlankHost { bh := &BlankHost{ - n: n, - mux: mstream.NewMultistreamMuxer(), + n: n, + cmgr: connmgr.NewConnManager(0, 0, 0), + mux: mstream.NewMultistreamMuxer(), } n.SetStreamHandler(bh.newStreamHandler) @@ -139,3 +142,7 @@ func (bh *BlankHost) Mux() *mstream.MultistreamMuxer { func (bh *BlankHost) Network() inet.Network { return bh.n } + +func (bh *BlankHost) ConnManager() connmgr.ConnManager { + return bh.cmgr +} From 554bd36fb626e89a3a515851634ff5c468be1b79 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 14 Jul 2017 16:08:35 +0700 Subject: [PATCH 0258/3965] use the new quic-go API The tls.Config was removed from the quic.Config in https://github.com/lucas-clemente/quic-go/pull/713. --- p2p/transport/quic/dialer.go | 3 +-- p2p/transport/quic/listener.go | 10 +++------- 2 files changed, 4 insertions(+), 9 deletions(-) diff --git a/p2p/transport/quic/dialer.go b/p2p/transport/quic/dialer.go index 8563674369..fd47186b2a 100644 --- a/p2p/transport/quic/dialer.go +++ b/p2p/transport/quic/dialer.go @@ -25,13 +25,12 @@ func newDialer(transport tpt.Transport) (*dialer, error) { func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { // TODO: check that raddr is a QUIC address - tlsConf := &tls.Config{InsecureSkipVerify: true} _, host, err := manet.DialArgs(raddr) if err != nil { return nil, err } - qsess, err := quic.DialAddr(host, &quic.Config{TLSConfig: tlsConf}) + qsess, err := quic.DialAddr(host, &tls.Config{InsecureSkipVerify: true}, nil) if err != nil { return nil, err } diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index dbd644a0ff..6ef1427378 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -20,17 +20,13 @@ type listener struct { var _ tpt.Listener = &listener{} func newListener(laddr ma.Multiaddr, t tpt.Transport) (*listener, error) { - qconf := &quic.Config{ - // we need to provide a certificate here - // use the demo certificate from quic-go - TLSConfig: testdata.GetTLSConfig(), - } - _, host, err := manet.DialArgs(laddr) if err != nil { return nil, err } - qln, err := quic.ListenAddr(host, qconf) + // we need to provide a certificate here + // use the demo certificate from quic-go + qln, err := quic.ListenAddr(host, testdata.GetTLSConfig(), nil) if err != nil { return nil, err } From 68db754e53bc1bac65f783489114f6a3b3c749e3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 14 Jul 2017 20:51:46 +0300 Subject: [PATCH 0259/3965] pb/relay: CiruitRelay protobuf --- .../internal/circuitv1-deprecated/pb/Makefile | 10 + .../circuitv1-deprecated/pb/relay.pb.go | 199 ++++++++++++++++++ .../circuitv1-deprecated/pb/relay.proto | 41 ++++ 3 files changed, 250 insertions(+) create mode 100644 p2p/protocol/internal/circuitv1-deprecated/pb/Makefile create mode 100644 p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go create mode 100644 p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/Makefile b/p2p/protocol/internal/circuitv1-deprecated/pb/Makefile new file mode 100644 index 0000000000..95e2e1c206 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/Makefile @@ -0,0 +1,10 @@ +PB = $(wildcard *.proto) +GO = $(PB:.proto=.pb.go) + +all: $(GO) + +%.pb.go: %.proto + protoc --gogo_out=. $< + +clean: + rm *.pb.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go new file mode 100644 index 0000000000..fee05f8ee5 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go @@ -0,0 +1,199 @@ +// Code generated by protoc-gen-gogo. +// source: relay.proto +// DO NOT EDIT! + +/* +Package relay_pb is a generated protocol buffer package. + +It is generated from these files: + relay.proto + +It has these top-level messages: + CircuitRelay +*/ +package relay_pb + +import proto "github.com/gogo/protobuf/proto" +import fmt "fmt" +import math "math" + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +type CircuitRelay_Status int32 + +const ( + CircuitRelay_SUCCESS CircuitRelay_Status = 100 + CircuitRelay_HOP_SRC_ADDR_TOO_LONG CircuitRelay_Status = 220 + CircuitRelay_HOP_DST_ADDR_TOO_LONG CircuitRelay_Status = 221 + CircuitRelay_HOP_SRC_MULTIADDR_INVALID CircuitRelay_Status = 250 + CircuitRelay_HOP_DST_MULTIADDR_INVALID CircuitRelay_Status = 251 + CircuitRelay_HOP_NO_CONN_TO_DST CircuitRelay_Status = 260 + CircuitRelay_HOP_CANT_DIAL_DST CircuitRelay_Status = 261 + CircuitRelay_HOP_CANT_OPEN_DST_STREAM CircuitRelay_Status = 262 + CircuitRelay_HOP_CANT_SPEAK_RELAY CircuitRelay_Status = 270 + CircuitRelay_HOP_CANT_REALAY_TO_SELF CircuitRelay_Status = 280 + CircuitRelay_STOP_SRC_ADDR_TOO_LONG CircuitRelay_Status = 320 + CircuitRelay_STOP_DST_ADDR_TOO_LONG CircuitRelay_Status = 321 + CircuitRelay_STOP_SRC_MULTIADDR_INVALID CircuitRelay_Status = 350 + CircuitRelay_STOP_DST_MULTIADDR_INVALID CircuitRelay_Status = 351 +) + +var CircuitRelay_Status_name = map[int32]string{ + 100: "SUCCESS", + 220: "HOP_SRC_ADDR_TOO_LONG", + 221: "HOP_DST_ADDR_TOO_LONG", + 250: "HOP_SRC_MULTIADDR_INVALID", + 251: "HOP_DST_MULTIADDR_INVALID", + 260: "HOP_NO_CONN_TO_DST", + 261: "HOP_CANT_DIAL_DST", + 262: "HOP_CANT_OPEN_DST_STREAM", + 270: "HOP_CANT_SPEAK_RELAY", + 280: "HOP_CANT_REALAY_TO_SELF", + 320: "STOP_SRC_ADDR_TOO_LONG", + 321: "STOP_DST_ADDR_TOO_LONG", + 350: "STOP_SRC_MULTIADDR_INVALID", + 351: "STOP_DST_MULTIADDR_INVALID", +} +var CircuitRelay_Status_value = map[string]int32{ + "SUCCESS": 100, + "HOP_SRC_ADDR_TOO_LONG": 220, + "HOP_DST_ADDR_TOO_LONG": 221, + "HOP_SRC_MULTIADDR_INVALID": 250, + "HOP_DST_MULTIADDR_INVALID": 251, + "HOP_NO_CONN_TO_DST": 260, + "HOP_CANT_DIAL_DST": 261, + "HOP_CANT_OPEN_DST_STREAM": 262, + "HOP_CANT_SPEAK_RELAY": 270, + "HOP_CANT_REALAY_TO_SELF": 280, + "STOP_SRC_ADDR_TOO_LONG": 320, + "STOP_DST_ADDR_TOO_LONG": 321, + "STOP_SRC_MULTIADDR_INVALID": 350, + "STOP_DST_MULTIADDR_INVALID": 351, +} + +func (x CircuitRelay_Status) Enum() *CircuitRelay_Status { + p := new(CircuitRelay_Status) + *p = x + return p +} +func (x CircuitRelay_Status) String() string { + return proto.EnumName(CircuitRelay_Status_name, int32(x)) +} +func (x *CircuitRelay_Status) UnmarshalJSON(data []byte) error { + value, err := proto.UnmarshalJSONEnum(CircuitRelay_Status_value, data, "CircuitRelay_Status") + if err != nil { + return err + } + *x = CircuitRelay_Status(value) + return nil +} + +type CircuitRelay_Type int32 + +const ( + CircuitRelay_HOP CircuitRelay_Type = 1 + CircuitRelay_STOP CircuitRelay_Type = 2 + CircuitRelay_STATUS CircuitRelay_Type = 3 +) + +var CircuitRelay_Type_name = map[int32]string{ + 1: "HOP", + 2: "STOP", + 3: "STATUS", +} +var CircuitRelay_Type_value = map[string]int32{ + "HOP": 1, + "STOP": 2, + "STATUS": 3, +} + +func (x CircuitRelay_Type) Enum() *CircuitRelay_Type { + p := new(CircuitRelay_Type) + *p = x + return p +} +func (x CircuitRelay_Type) String() string { + return proto.EnumName(CircuitRelay_Type_name, int32(x)) +} +func (x *CircuitRelay_Type) UnmarshalJSON(data []byte) error { + value, err := proto.UnmarshalJSONEnum(CircuitRelay_Type_value, data, "CircuitRelay_Type") + if err != nil { + return err + } + *x = CircuitRelay_Type(value) + return nil +} + +type CircuitRelay struct { + Type *CircuitRelay_Type `protobuf:"varint,1,opt,name=type,enum=relay.pb.CircuitRelay_Type" json:"type,omitempty"` + SrcPeer *CircuitRelay_Peer `protobuf:"bytes,2,opt,name=srcPeer" json:"srcPeer,omitempty"` + DstPeer *CircuitRelay_Peer `protobuf:"bytes,3,opt,name=dstPeer" json:"dstPeer,omitempty"` + Code *CircuitRelay_Status `protobuf:"varint,4,opt,name=code,enum=relay.pb.CircuitRelay_Status" json:"code,omitempty"` + XXX_unrecognized []byte `json:"-"` +} + +func (m *CircuitRelay) Reset() { *m = CircuitRelay{} } +func (m *CircuitRelay) String() string { return proto.CompactTextString(m) } +func (*CircuitRelay) ProtoMessage() {} + +func (m *CircuitRelay) GetType() CircuitRelay_Type { + if m != nil && m.Type != nil { + return *m.Type + } + return CircuitRelay_HOP +} + +func (m *CircuitRelay) GetSrcPeer() *CircuitRelay_Peer { + if m != nil { + return m.SrcPeer + } + return nil +} + +func (m *CircuitRelay) GetDstPeer() *CircuitRelay_Peer { + if m != nil { + return m.DstPeer + } + return nil +} + +func (m *CircuitRelay) GetCode() CircuitRelay_Status { + if m != nil && m.Code != nil { + return *m.Code + } + return CircuitRelay_SUCCESS +} + +type CircuitRelay_Peer struct { + Id []byte `protobuf:"bytes,1,req,name=id" json:"id,omitempty"` + Addrs [][]byte `protobuf:"bytes,2,rep,name=addrs" json:"addrs,omitempty"` + XXX_unrecognized []byte `json:"-"` +} + +func (m *CircuitRelay_Peer) Reset() { *m = CircuitRelay_Peer{} } +func (m *CircuitRelay_Peer) String() string { return proto.CompactTextString(m) } +func (*CircuitRelay_Peer) ProtoMessage() {} + +func (m *CircuitRelay_Peer) GetId() []byte { + if m != nil { + return m.Id + } + return nil +} + +func (m *CircuitRelay_Peer) GetAddrs() [][]byte { + if m != nil { + return m.Addrs + } + return nil +} + +func init() { + proto.RegisterType((*CircuitRelay)(nil), "relay.pb.CircuitRelay") + proto.RegisterType((*CircuitRelay_Peer)(nil), "relay.pb.CircuitRelay.Peer") + proto.RegisterEnum("relay.pb.CircuitRelay_Status", CircuitRelay_Status_name, CircuitRelay_Status_value) + proto.RegisterEnum("relay.pb.CircuitRelay_Type", CircuitRelay_Type_name, CircuitRelay_Type_value) +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto new file mode 100644 index 0000000000..46a7787e39 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto @@ -0,0 +1,41 @@ +syntax = "proto2"; + +package relay.pb; + +message CircuitRelay { + + enum Status { + SUCCESS = 100; + HOP_SRC_ADDR_TOO_LONG = 220; + HOP_DST_ADDR_TOO_LONG = 221; + HOP_SRC_MULTIADDR_INVALID = 250; + HOP_DST_MULTIADDR_INVALID = 251; + HOP_NO_CONN_TO_DST = 260; + HOP_CANT_DIAL_DST = 261; + HOP_CANT_OPEN_DST_STREAM = 262; + HOP_CANT_SPEAK_RELAY = 270; + HOP_CANT_REALAY_TO_SELF = 280; + STOP_SRC_ADDR_TOO_LONG = 320; + STOP_DST_ADDR_TOO_LONG = 321; + STOP_SRC_MULTIADDR_INVALID = 350; + STOP_DST_MULTIADDR_INVALID = 351; + } + + enum Type { // RPC identifier, either HOP, STOP or STATUS + HOP = 1; + STOP = 2; + STATUS = 3; + } + + message Peer { + required bytes id = 1; // peer id + repeated bytes addrs = 2; // peer's known addresses + } + + optional Type type = 1; // Type of the message + + optional Peer srcPeer = 2; // srcPeer and dstPeer are used when Type is HOP or STATUS + optional Peer dstPeer = 3; + + optional Status code = 4; // Status code, used when Type is STATUS +} From 59ce49e385b259e78bf3163da31087091c8a70c1 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 14 Jul 2017 20:52:02 +0300 Subject: [PATCH 0260/3965] delete unnecessary status/util --- .../internal/circuitv1-deprecated/status.go | 62 ------------------ .../circuitv1-deprecated/status_test.go | 32 ---------- .../internal/circuitv1-deprecated/util.go | 63 ------------------- .../circuitv1-deprecated/util_test.go | 44 ------------- 4 files changed, 201 deletions(-) delete mode 100644 p2p/protocol/internal/circuitv1-deprecated/status.go delete mode 100644 p2p/protocol/internal/circuitv1-deprecated/status_test.go delete mode 100644 p2p/protocol/internal/circuitv1-deprecated/util.go delete mode 100644 p2p/protocol/internal/circuitv1-deprecated/util_test.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/status.go b/p2p/protocol/internal/circuitv1-deprecated/status.go deleted file mode 100644 index 8d9c2582ed..0000000000 --- a/p2p/protocol/internal/circuitv1-deprecated/status.go +++ /dev/null @@ -1,62 +0,0 @@ -package relay - -import ( - "encoding/binary" - "fmt" - "io" -) - -const maxStatusMessageLength = 1024 - -const ( - StatusOK = 100 - - StatusRelayAddrErr = 250 - StatusRelayNotConnected = 260 - StatusRelayDialFailed = 261 - StatusRelayStreamFailed = 262 - StatusRelayHopToSelf = 270 - - StatusDstAddrErr = 350 - StatusDstRelayRefused = 380 - StatusDstWrongDst = 381 -) - -type RelayStatus struct { - Code uint64 - Message string -} - -func (s *RelayStatus) WriteTo(w io.Writer) error { - outbuf := make([]byte, 2*binary.MaxVarintLen64+len(s.Message)) - n := binary.PutUvarint(outbuf, s.Code) - n += binary.PutUvarint(outbuf[n:], uint64(len(s.Message))) - n += copy(outbuf[n:], s.Message) - _, err := w.Write(outbuf[:n]) - return err -} - -func (s *RelayStatus) ReadFrom(r io.Reader) error { - br := &singleByteReader{r} - code, err := binary.ReadUvarint(br) - if err != nil { - return err - } - l, err := binary.ReadUvarint(br) - if err != nil { - return err - } - buf := make([]byte, l) - _, err = io.ReadFull(r, buf) - if err != nil { - return err - } - - s.Code = code - s.Message = string(buf) - return nil -} - -func (s *RelayStatus) Error() string { - return fmt.Sprintf("%d: %s", s.Code, s.Message) -} diff --git a/p2p/protocol/internal/circuitv1-deprecated/status_test.go b/p2p/protocol/internal/circuitv1-deprecated/status_test.go deleted file mode 100644 index 25fe4e0f32..0000000000 --- a/p2p/protocol/internal/circuitv1-deprecated/status_test.go +++ /dev/null @@ -1,32 +0,0 @@ -package relay - -import ( - "bytes" - "testing" -) - -func TestStatusParsing(t *testing.T) { - s := &RelayStatus{ - Code: StatusDstAddrErr, - Message: "foo bar", - } - - buf := new(bytes.Buffer) - - if err := s.WriteTo(buf); err != nil { - t.Fatal(err) - } - - var ns RelayStatus - if err := ns.ReadFrom(buf); err != nil { - t.Fatal(err) - } - - if ns.Code != s.Code { - t.Fatal("codes didnt match") - } - - if ns.Message != s.Message { - t.Fatal("messages didnt match") - } -} diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go deleted file mode 100644 index 15726d3d3d..0000000000 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ /dev/null @@ -1,63 +0,0 @@ -package relay - -import ( - "encoding/binary" - "fmt" - "io" - - ma "github.com/multiformats/go-multiaddr" -) - -type singleByteReader struct { - r io.Reader -} - -func (s *singleByteReader) ReadByte() (byte, error) { - var b [1]byte - n, err := s.r.Read(b[:]) - if err != nil { - return 0, err - } - if n == 0 { - return 0, io.ErrNoProgress - } - - return b[0], nil -} - -func writeLpMultiaddr(w io.Writer, a ma.Multiaddr) error { - buf := make([]byte, binary.MaxVarintLen32+len(a.Bytes())) - n := binary.PutUvarint(buf, uint64(len(a.Bytes()))) - n += copy(buf[n:], a.Bytes()) - nw, err := w.Write(buf[:n]) - if err != nil { - return err - } - if n != nw { - return fmt.Errorf("failed to write all bytes to writer") - } - return nil -} - -func readLpMultiaddr(r io.Reader) (ma.Multiaddr, error) { - l, err := binary.ReadUvarint(&singleByteReader{r}) - if err != nil { - return nil, err - } - - if l > maxAddrLen { - return nil, fmt.Errorf("address length was too long: %d > %d", l, maxAddrLen) - } - - if l == 0 { - return nil, fmt.Errorf("zero length multiaddr is invalid") - } - - buf := make([]byte, l) - _, err = io.ReadFull(r, buf) - if err != nil { - return nil, err - } - - return ma.NewMultiaddrBytes(buf) -} diff --git a/p2p/protocol/internal/circuitv1-deprecated/util_test.go b/p2p/protocol/internal/circuitv1-deprecated/util_test.go deleted file mode 100644 index b0f3f68cd6..0000000000 --- a/p2p/protocol/internal/circuitv1-deprecated/util_test.go +++ /dev/null @@ -1,44 +0,0 @@ -package relay - -import ( - "bytes" - "testing" - - ma "github.com/multiformats/go-multiaddr" -) - -func TestMultiaddrSerialization(t *testing.T) { - buf := new(bytes.Buffer) - - addr, err := ma.NewMultiaddr("/ip4/1.2.3.4/tcp/1234") - if err != nil { - t.Fatal(err) - } - - if err := writeLpMultiaddr(buf, addr); err != nil { - t.Fatal(err) - } - - out, err := readLpMultiaddr(buf) - if err != nil { - t.Fatal(err) - } - - if !addr.Equal(out) { - t.Fatal("addresses didnt match") - } -} - -func TestDecodeInvalid(t *testing.T) { - buf := bytes.NewBuffer([]byte{72, 0, 0}) - _, err := readLpMultiaddr(buf) - if err == nil { - t.Fatal("shouldnt have parsed correctly") - } - - buf = bytes.NewBuffer([]byte{0}) - _, err = readLpMultiaddr(buf) - if err == nil { - t.Fatal("shouldnt have parsed correctly") - } -} From 55c4d357b769649ffba4f8ef96fc9b220d70c70e Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 14 Jul 2017 21:21:19 +0300 Subject: [PATCH 0261/3965] pb/relay: fix typos 1. it's not REALAY 2. srcaddr/dstaddr not meaningful for STATUS --- p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go | 6 +++--- p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go index fee05f8ee5..a7f7d9d8b9 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go @@ -34,7 +34,7 @@ const ( CircuitRelay_HOP_CANT_DIAL_DST CircuitRelay_Status = 261 CircuitRelay_HOP_CANT_OPEN_DST_STREAM CircuitRelay_Status = 262 CircuitRelay_HOP_CANT_SPEAK_RELAY CircuitRelay_Status = 270 - CircuitRelay_HOP_CANT_REALAY_TO_SELF CircuitRelay_Status = 280 + CircuitRelay_HOP_CANT_RELAY_TO_SELF CircuitRelay_Status = 280 CircuitRelay_STOP_SRC_ADDR_TOO_LONG CircuitRelay_Status = 320 CircuitRelay_STOP_DST_ADDR_TOO_LONG CircuitRelay_Status = 321 CircuitRelay_STOP_SRC_MULTIADDR_INVALID CircuitRelay_Status = 350 @@ -51,7 +51,7 @@ var CircuitRelay_Status_name = map[int32]string{ 261: "HOP_CANT_DIAL_DST", 262: "HOP_CANT_OPEN_DST_STREAM", 270: "HOP_CANT_SPEAK_RELAY", - 280: "HOP_CANT_REALAY_TO_SELF", + 280: "HOP_CANT_RELAY_TO_SELF", 320: "STOP_SRC_ADDR_TOO_LONG", 321: "STOP_DST_ADDR_TOO_LONG", 350: "STOP_SRC_MULTIADDR_INVALID", @@ -67,7 +67,7 @@ var CircuitRelay_Status_value = map[string]int32{ "HOP_CANT_DIAL_DST": 261, "HOP_CANT_OPEN_DST_STREAM": 262, "HOP_CANT_SPEAK_RELAY": 270, - "HOP_CANT_REALAY_TO_SELF": 280, + "HOP_CANT_RELAY_TO_SELF": 280, "STOP_SRC_ADDR_TOO_LONG": 320, "STOP_DST_ADDR_TOO_LONG": 321, "STOP_SRC_MULTIADDR_INVALID": 350, diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto index 46a7787e39..0b2290048a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto @@ -14,7 +14,7 @@ message CircuitRelay { HOP_CANT_DIAL_DST = 261; HOP_CANT_OPEN_DST_STREAM = 262; HOP_CANT_SPEAK_RELAY = 270; - HOP_CANT_REALAY_TO_SELF = 280; + HOP_CANT_RELAY_TO_SELF = 280; STOP_SRC_ADDR_TOO_LONG = 320; STOP_DST_ADDR_TOO_LONG = 321; STOP_SRC_MULTIADDR_INVALID = 350; @@ -34,7 +34,7 @@ message CircuitRelay { optional Type type = 1; // Type of the message - optional Peer srcPeer = 2; // srcPeer and dstPeer are used when Type is HOP or STATUS + optional Peer srcPeer = 2; // srcPeer and dstPeer are used when Type is HOP or STOP optional Peer dstPeer = 3; optional Status code = 4; // Status code, used when Type is STATUS From 7c6f9c07cfedec1056eeac677d1efeea72c4aacf Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 16 Jul 2017 12:24:43 +0700 Subject: [PATCH 0262/3965] use the new quic-go stream deadline Support for stream deadlines was added in https://github.com/lucas-clemente/quic-go/pull/579. --- p2p/transport/quic/conn.go | 13 ++----------- p2p/transport/quic/conn_test.go | 20 ++++++++++++-------- p2p/transport/quic/stream.go | 29 ----------------------------- 3 files changed, 14 insertions(+), 48 deletions(-) delete mode 100644 p2p/transport/quic/stream.go diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index cbc6709f40..511fa89a36 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -46,28 +46,19 @@ func newQuicConn(sess quic.Session, t tpt.Transport) (*quicConn, error) { raddr: raddr, transport: t, } - go c.watchClosed() return c, nil } func (c *quicConn) AcceptStream() (smux.Stream, error) { - str, err := c.sess.AcceptStream() - if err != nil { - return nil, err - } - return &quicStream{Stream: str}, nil + return c.sess.AcceptStream() } // OpenStream opens a new stream // It blocks until a new stream can be opened (when limited by the QUIC maximum stream limit) func (c *quicConn) OpenStream() (smux.Stream, error) { - str, err := c.sess.OpenStreamSync() - if err != nil { - return nil, err - } - return &quicStream{Stream: str}, nil + return c.sess.OpenStreamSync() } func (c *quicConn) Serve(handler smux.StreamHandler) { diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 03e519d706..7c80681736 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -3,6 +3,7 @@ package libp2pquic import ( "errors" "net" + "time" smux "github.com/jbenet/go-stream-muxer" quic "github.com/lucas-clemente/quic-go" @@ -16,11 +17,14 @@ type mockStream struct { id protocol.StreamID } -func (s *mockStream) Close() error { return nil } -func (s *mockStream) Reset(error) { return } -func (s *mockStream) Read([]byte) (int, error) { return 0, nil } -func (s *mockStream) Write([]byte) (int, error) { return 0, nil } -func (s *mockStream) StreamID() protocol.StreamID { return s.id } +func (s *mockStream) Close() error { return nil } +func (s *mockStream) Reset(error) { return } +func (s *mockStream) Read([]byte) (int, error) { return 0, nil } +func (s *mockStream) Write([]byte) (int, error) { return 0, nil } +func (s *mockStream) StreamID() protocol.StreamID { return s.id } +func (s *mockStream) SetReadDeadline(time.Time) error { panic("not implemented") } +func (s *mockStream) SetWriteDeadline(time.Time) error { panic("not implemented") } +func (s *mockStream) SetDeadline(time.Time) error { panic("not implemented") } var _ quic.Stream = &mockStream{} @@ -97,7 +101,7 @@ var _ = Describe("Conn", func() { sess.streamToOpen = s str, err := conn.OpenStream() Expect(err).ToNot(HaveOccurred()) - Expect(str.(*quicStream).Stream).To(Equal(s)) + Expect(str).To(Equal(s)) }) It("errors when it can't open a stream", func() { @@ -114,7 +118,7 @@ var _ = Describe("Conn", func() { sess.streamToAccept = s str, err := conn.AcceptStream() Expect(err).ToNot(HaveOccurred()) - Expect(str.(*quicStream).Stream).To(Equal(s)) + Expect(str).To(Equal(s)) }) It("errors when it can't open a stream", func() { @@ -150,7 +154,7 @@ var _ = Describe("Conn", func() { returned = true }() Eventually(func() bool { return handlerCalled }).Should(BeTrue()) - Expect(handlerCalledWith.(*quicStream).Stream).To(Equal(str)) + Expect(handlerCalledWith).To(Equal(str)) // make the go-routine return sess.streamAcceptErr = errors.New("stop test") }) diff --git a/p2p/transport/quic/stream.go b/p2p/transport/quic/stream.go deleted file mode 100644 index c41b2b324b..0000000000 --- a/p2p/transport/quic/stream.go +++ /dev/null @@ -1,29 +0,0 @@ -package libp2pquic - -import ( - "time" - - smux "github.com/jbenet/go-stream-muxer" - quic "github.com/lucas-clemente/quic-go" -) - -// The quicStream is a very thin wrapper for a quic.Stream, adding some methods -// required to fulfill the smux.Stream interface -// TODO: this can be removed once the quic.Stream supports deadlines (quic-go#514) -type quicStream struct { - quic.Stream -} - -var _ smux.Stream = &quicStream{} - -func (s *quicStream) SetDeadline(time.Time) error { - return nil -} - -func (s *quicStream) SetReadDeadline(time.Time) error { - return nil -} - -func (s *quicStream) SetWriteDeadline(time.Time) error { - return nil -} From bae0cbcb2f98c888beb549b9d6b3198782ef7aed Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 16 Jul 2017 13:42:31 -0700 Subject: [PATCH 0263/3965] cleanup and change import --- p2p/muxer/yamux/yamux.go | 38 +++----------------------------------- 1 file changed, 3 insertions(+), 35 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 6f62aa8ac2..a8877c213b 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -5,42 +5,10 @@ import ( "net" "time" - yamux "github.com/hashicorp/yamux" smux "github.com/jbenet/go-stream-muxer" + yamux "github.com/whyrusleeping/yamux" ) -// stream implements smux.Stream using a ss.Stream -// TODO: do we actually need a wrapper here? -type stream yamux.Stream - -func (s *stream) yamuxStream() *yamux.Stream { - return (*yamux.Stream)(s) -} - -func (s *stream) Read(buf []byte) (int, error) { - return s.yamuxStream().Read(buf) -} - -func (s *stream) Write(buf []byte) (int, error) { - return s.yamuxStream().Write(buf) -} - -func (s *stream) Close() error { - return s.yamuxStream().Close() -} - -func (s *stream) SetDeadline(t time.Time) error { - return s.yamuxStream().SetDeadline(t) -} - -func (s *stream) SetReadDeadline(t time.Time) error { - return s.yamuxStream().SetReadDeadline(t) -} - -func (s *stream) SetWriteDeadline(t time.Time) error { - return s.yamuxStream().SetWriteDeadline(t) -} - // Conn is a connection to a remote peer. type conn yamux.Session @@ -63,13 +31,13 @@ func (c *conn) OpenStream() (smux.Stream, error) { return nil, err } - return (*stream)(s), nil + return s, nil } // AcceptStream accepts a stream opened by the other side. func (c *conn) AcceptStream() (smux.Stream, error) { s, err := c.yamuxSession().AcceptStream() - return (*stream)(s), err + return s, err } // Serve starts listening for incoming requests and handles them From ce5e59807c2cfb07a16afb98e6033c37fa11620d Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 12:48:33 +0300 Subject: [PATCH 0264/3965] relay.proto: add more error codes --- .../circuitv1-deprecated/pb/relay.pb.go | 21 +++++++++++++------ .../circuitv1-deprecated/pb/relay.proto | 9 +++++--- 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go index a7f7d9d8b9..9c3fc2b466 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go @@ -39,6 +39,8 @@ const ( CircuitRelay_STOP_DST_ADDR_TOO_LONG CircuitRelay_Status = 321 CircuitRelay_STOP_SRC_MULTIADDR_INVALID CircuitRelay_Status = 350 CircuitRelay_STOP_DST_MULTIADDR_INVALID CircuitRelay_Status = 351 + CircuitRelay_STOP_RELAY_REFUSED CircuitRelay_Status = 390 + CircuitRelay_MALFORMED_MESSAGE CircuitRelay_Status = 400 ) var CircuitRelay_Status_name = map[int32]string{ @@ -56,6 +58,8 @@ var CircuitRelay_Status_name = map[int32]string{ 321: "STOP_DST_ADDR_TOO_LONG", 350: "STOP_SRC_MULTIADDR_INVALID", 351: "STOP_DST_MULTIADDR_INVALID", + 390: "STOP_RELAY_REFUSED", + 400: "MALFORMED_MESSAGE", } var CircuitRelay_Status_value = map[string]int32{ "SUCCESS": 100, @@ -72,6 +76,8 @@ var CircuitRelay_Status_value = map[string]int32{ "STOP_DST_ADDR_TOO_LONG": 321, "STOP_SRC_MULTIADDR_INVALID": 350, "STOP_DST_MULTIADDR_INVALID": 351, + "STOP_RELAY_REFUSED": 390, + "MALFORMED_MESSAGE": 400, } func (x CircuitRelay_Status) Enum() *CircuitRelay_Status { @@ -94,20 +100,23 @@ func (x *CircuitRelay_Status) UnmarshalJSON(data []byte) error { type CircuitRelay_Type int32 const ( - CircuitRelay_HOP CircuitRelay_Type = 1 - CircuitRelay_STOP CircuitRelay_Type = 2 - CircuitRelay_STATUS CircuitRelay_Type = 3 + CircuitRelay_HOP CircuitRelay_Type = 1 + CircuitRelay_STOP CircuitRelay_Type = 2 + CircuitRelay_STATUS CircuitRelay_Type = 3 + CircuitRelay_CAN_HOP CircuitRelay_Type = 4 ) var CircuitRelay_Type_name = map[int32]string{ 1: "HOP", 2: "STOP", 3: "STATUS", + 4: "CAN_HOP", } var CircuitRelay_Type_value = map[string]int32{ - "HOP": 1, - "STOP": 2, - "STATUS": 3, + "HOP": 1, + "STOP": 2, + "STATUS": 3, + "CAN_HOP": 4, } func (x CircuitRelay_Type) Enum() *CircuitRelay_Type { diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto index 0b2290048a..de3e637b12 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.proto @@ -19,12 +19,15 @@ message CircuitRelay { STOP_DST_ADDR_TOO_LONG = 321; STOP_SRC_MULTIADDR_INVALID = 350; STOP_DST_MULTIADDR_INVALID = 351; + STOP_RELAY_REFUSED = 390; + MALFORMED_MESSAGE = 400; } enum Type { // RPC identifier, either HOP, STOP or STATUS - HOP = 1; - STOP = 2; - STATUS = 3; + HOP = 1; + STOP = 2; + STATUS = 3; + CAN_HOP = 4; } message Peer { From f18fc04bd4a7a69ff9ba219bb904acc0cb1b7909 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 13:16:07 +0300 Subject: [PATCH 0265/3965] implement ciruit/relay/0.1.0 spec --- .../internal/circuitv1-deprecated/conn.go | 10 +- .../internal/circuitv1-deprecated/dial.go | 11 +- .../internal/circuitv1-deprecated/listen.go | 20 +- .../internal/circuitv1-deprecated/relay.go | 250 +++++++++++------- .../internal/circuitv1-deprecated/util.go | 47 ++++ 5 files changed, 226 insertions(+), 112 deletions(-) create mode 100644 p2p/protocol/internal/circuitv1-deprecated/util.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 73a66b78d0..f165647028 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -8,6 +8,7 @@ import ( iconn "github.com/libp2p/go-libp2p-interface-conn" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" @@ -15,8 +16,7 @@ import ( type Conn struct { inet.Stream - remoteMaddr ma.Multiaddr - remotePeer peer.ID + remote pstore.PeerInfo } var _ iconn.Conn = (*Conn)(nil) @@ -37,12 +37,12 @@ func (n *NetAddr) String() string { func (c *Conn) RemoteAddr() net.Addr { return &NetAddr{ Relay: c.Conn().RemotePeer().Pretty(), - Remote: c.remotePeer.String(), + Remote: c.remote.ID.Pretty(), } } func (c *Conn) RemoteMultiaddr() ma.Multiaddr { - a, err := ma.NewMultiaddr(fmt.Sprintf("%s/ipfs/%s/p2p-circuit/%s", c.remoteMaddr, c.remotePeer.Pretty(), c.Conn().RemotePeer())) + a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/%s", c.remote.ID.Pretty(), c.Conn().RemotePeer())) if err != nil { panic(err) } @@ -71,7 +71,7 @@ func (c *Conn) LocalPeer() peer.ID { } func (c *Conn) RemotePeer() peer.ID { - return c.remotePeer + return c.remote.ID } func (c *Conn) LocalPrivateKey() ic.PrivKey { diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 1d2f103575..936cc21d71 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -42,16 +42,9 @@ func (d *Dialer) DialPeer(ctx context.Context, p peer.ID, a ma.Multiaddr) (tpt.C return nil, err } - destp2p, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s", p.Pretty())) - if err != nil { - return nil, err - } - - destp2p = destaddr.Encapsulate(destp2p) - - d.Relay().host.Peerstore().AddAddrs(rinfo.ID, rinfo.Addrs, pstore.TempAddrTTL) + dinfo := pstore.PeerInfo{ID: p, Addrs: []ma.Multiaddr{destaddr}} - return d.Relay().Dial(ctx, rinfo.ID, destp2p) + return d.Relay().Dial(ctx, *rinfo, dinfo) } func (d *Dialer) Matches(a ma.Multiaddr) bool { diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index 1c73a65417..8cc096eb8c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -3,6 +3,8 @@ package relay import ( "net" + pb "github.com/libp2p/go-libp2p-circuit/pb" + peer "github.com/libp2p/go-libp2p-peer" tpt "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" @@ -26,20 +28,20 @@ func (r *Relay) Matches(a ma.Multiaddr) bool { } func (l *RelayListener) Accept() (tpt.Conn, error) { - ctx := l.Relay().ctx select { case c := <-l.incoming: - log.Infof("accepted relay connection: %s", c.ID()) - s := RelayStatus{ - Code: StatusOK, - Message: "OK", - } - if err := s.WriteTo(c); err != nil { + err := l.Relay().writeResponse(c.Stream, pb.CircuitRelay_SUCCESS) + if err != nil { + log.Debugf("error writing relay response: %s", err.Error()) + c.Stream.Close() return nil, err } + + log.Infof("accepted relay connection: %s", c.ID()) + return c, nil - case <-ctx.Done(): - return nil, ctx.Err() + case <-l.ctx.Done(): + return nil, l.ctx.Err() } } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index d4229567ae..65a04375a1 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -7,20 +7,21 @@ import ( "sync" "time" + pb "github.com/libp2p/go-libp2p-circuit/pb" + + ggio "github.com/gogo/protobuf/io" logging "github.com/ipfs/go-log" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" ) var log = logging.Logger("relay") -const HopID = "/libp2p/relay/circuit/1.0.0/hop" -const StopID = "/libp2p/relay/circuit/1.0.0/stop" +const ProtoID = "/libp2p/circuit/relay/0.1.0" -const maxAddrLen = 1024 +const maxMessageSize = 4096 var RelayAcceptTimeout = time.Minute @@ -30,6 +31,7 @@ type Relay struct { self peer.ID active bool + hop bool incoming chan *Conn @@ -44,6 +46,14 @@ var ( OptHop = RelayOpt(1) ) +type RelayError struct { + Code int32 +} + +func (e RelayError) Error() string { + return fmt.Sprintf("error opening relay circuit: %s (%d)", pb.CircuitRelay_Status_name[e.Code], e.Code) +} + func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error) { r := &Relay{ host: h, @@ -52,138 +62,156 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error incoming: make(chan *Conn), } - h.SetStreamHandler(StopID, r.HandleNewStopStream) - for _, opt := range opts { switch opt { case OptActive: r.active = true case OptHop: - h.SetStreamHandler(HopID, r.HandleNewHopStream) + r.hop = true default: return nil, fmt.Errorf("unrecognized option: %d", opt) } } + h.SetStreamHandler(ProtoID, r.handleNewStream) + return r, nil } -func (r *Relay) Dial(ctx context.Context, relay peer.ID, dest ma.Multiaddr) (*Conn, error) { - s, err := r.host.NewStream(ctx, relay, HopID) +func (r *Relay) Dial(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { + err := r.host.Connect(ctx, relay) if err != nil { return nil, err } - if err := writeLpMultiaddr(s, dest); err != nil { + s, err := r.host.NewStream(ctx, relay.ID, ProtoID) + if err != nil { return nil, err } - var stat RelayStatus - if err := stat.ReadFrom(s); err != nil { + rd := ggio.NewDelimitedReader(s, maxMessageSize) + wr := ggio.NewDelimitedWriter(s) + + var msg pb.CircuitRelay + + msg.Type = pb.CircuitRelay_HOP.Enum() + msg.SrcPeer = peerInfoToPeer(pstore.PeerInfo{r.self, r.host.Addrs()}) + msg.DstPeer = peerInfoToPeer(dest) + + err = wr.WriteMsg(&msg) + if err != nil { + s.Close() return nil, err } - if stat.Code != StatusOK { - return nil, &stat + msg.Reset() + + err = rd.ReadMsg(&msg) + if err != nil { + s.Close() + return nil, err } - return &Conn{Stream: s}, nil -} + if msg.GetType() != pb.CircuitRelay_STATUS { + s.Close() + return nil, fmt.Errorf("unexpected relay response; not a status message (%d)", msg.GetType()) + } -func (r *Relay) HandleNewStopStream(s inet.Stream) { - log.Infof("new stop stream from: %s", s.Conn().RemotePeer()) - status := r.handleNewStopStream(s) - if status != nil { - if err := status.WriteTo(s); err != nil { - log.Info("problem writing error status:", err) - } + if msg.GetCode() != pb.CircuitRelay_SUCCESS { s.Close() - return + return nil, RelayError{int32(msg.GetCode())} } + + return &Conn{Stream: s}, nil } -func (r *Relay) handleNewStopStream(s inet.Stream) *RelayStatus { - info, err := r.readInfo(s) +func (r *Relay) handleNewStream(s inet.Stream) { + log.Infof("new relay stream from: %s", s.Conn().RemotePeer()) + + rd := ggio.NewDelimitedReader(s, maxMessageSize) + + var msg pb.CircuitRelay + + err := rd.ReadMsg(&msg) if err != nil { - return &RelayStatus{ - Code: StatusDstAddrErr, - Message: err.Error(), - } + r.handleError(s, pb.CircuitRelay_MALFORMED_MESSAGE) + return } - log.Infof("relay connection from: %s", info.ID) - select { - case r.incoming <- &Conn{Stream: s, remoteMaddr: info.Addrs[0], remotePeer: info.ID}: - return nil - case <-time.After(RelayAcceptTimeout): - return &RelayStatus{ - Code: StatusDstRelayRefused, - Message: "timed out waiting for relay to be accepted", - } + switch msg.GetType() { + case pb.CircuitRelay_HOP: + r.handleHopStream(s, &msg) + case pb.CircuitRelay_STOP: + r.handleStopStream(s, &msg) + case pb.CircuitRelay_CAN_HOP: + r.handleCanHop(s, &msg) + default: + log.Warningf("unexpected relay handshake: %d", msg.GetType()) + r.handleError(s, pb.CircuitRelay_MALFORMED_MESSAGE) } } -func (r *Relay) HandleNewHopStream(s inet.Stream) { - log.Infof("new hop stream from: %s", s.Conn().RemotePeer()) - status := r.handleNewHopStream(s) - if status != nil { - if err := status.WriteTo(s); err != nil { - log.Debugf("problem writing error status back: %s", err) - s.Close() - return - } +func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { + if !r.hop { + r.handleError(s, pb.CircuitRelay_HOP_CANT_SPEAK_RELAY) + return } -} -func (r *Relay) handleNewHopStream(s inet.Stream) *RelayStatus { - info, err := r.readInfo(s) + src, err := peerToPeerInfo(msg.GetSrcPeer()) if err != nil { - return &RelayStatus{ - Code: StatusRelayAddrErr, - Message: err.Error(), - } + r.handleError(s, pb.CircuitRelay_HOP_SRC_MULTIADDR_INVALID) + return } - if info.ID == r.self { - return &RelayStatus{ - Code: StatusRelayHopToSelf, - Message: "relay hop attempted to self", - } + if src.ID != s.Conn().RemotePeer() { + r.handleError(s, pb.CircuitRelay_HOP_SRC_MULTIADDR_INVALID) + return } - ctp := r.host.Network().ConnsToPeer(info.ID) + dst, err := peerToPeerInfo(msg.GetDstPeer()) + if err != nil { + r.handleError(s, pb.CircuitRelay_HOP_DST_MULTIADDR_INVALID) + return + } + + if dst.ID == r.self { + r.handleError(s, pb.CircuitRelay_HOP_CANT_RELAY_TO_SELF) + return + } + + ctp := r.host.Network().ConnsToPeer(dst.ID) if len(ctp) == 0 { - return &RelayStatus{ - Code: StatusRelayNotConnected, - Message: "refusing to make new connection for relay", + if !r.active { + r.handleError(s, pb.CircuitRelay_HOP_NO_CONN_TO_DST) + return } - } - bs, err := r.host.NewStream(r.ctx, info.ID, StopID) - if err != nil { - return &RelayStatus{ - Code: StatusRelayStreamFailed, - Message: err.Error(), + ctx, cancel := context.WithTimeout(r.ctx, time.Second*10) + defer cancel() + err = r.host.Connect(ctx, dst) + if err != nil { + log.Debugf("error opening relay connection to %s: %s", dst.ID.Pretty(), err.Error()) + r.handleError(s, pb.CircuitRelay_HOP_CANT_DIAL_DST) + return } } - // TODO: add helper method 'PeerID to multiaddr' - paddr, err := ma.NewMultiaddr("/ipfs/" + s.Conn().RemotePeer().Pretty()) + bs, err := r.host.NewStream(r.ctx, dst.ID, ProtoID) if err != nil { - return &RelayStatus{ - Code: StatusRelayAddrErr, - Message: err.Error(), - } + log.Debugf("error opening relay stream to %s: %s", dst.ID.Pretty(), err.Error()) + r.handleError(s, pb.CircuitRelay_HOP_CANT_OPEN_DST_STREAM) + return } - p2pa := s.Conn().RemoteMultiaddr().Encapsulate(paddr) - if err := writeLpMultiaddr(bs, p2pa); err != nil { - return &RelayStatus{ - Code: StatusRelayStreamFailed, - Message: err.Error(), - } + err = r.writeResponse(s, pb.CircuitRelay_SUCCESS) + if err != nil { + log.Debugf("error writing relay response: %s", err.Error()) + s.Close() + return } + log.Infof("relaying connection between %s and %s", src.ID.Pretty(), dst.ID.Pretty()) + go func() { _, err := io.Copy(s, bs) if err != io.EOF && err != nil { @@ -191,6 +219,7 @@ func (r *Relay) handleNewHopStream(s inet.Stream) *RelayStatus { } s.Close() }() + go func() { _, err := io.Copy(bs, s) if err != io.EOF && err != nil { @@ -198,20 +227,63 @@ func (r *Relay) handleNewHopStream(s inet.Stream) *RelayStatus { } bs.Close() }() +} - return nil +func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { + src, err := peerToPeerInfo(msg.GetSrcPeer()) + if err != nil || len(src.Addrs) == 0 { + r.handleError(s, pb.CircuitRelay_STOP_SRC_MULTIADDR_INVALID) + return + } + + dst, err := peerToPeerInfo(msg.GetDstPeer()) + if err != nil || dst.ID != r.self { + r.handleError(s, pb.CircuitRelay_STOP_DST_MULTIADDR_INVALID) + return + } + + log.Infof("relay connection from: %s", src.ID) + + r.host.Peerstore().AddAddrs(src.ID, src.Addrs, pstore.TempAddrTTL) + + select { + case r.incoming <- &Conn{Stream: s, remote: src}: + case <-time.After(RelayAcceptTimeout): + r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) + } } -func (r *Relay) readInfo(s inet.Stream) (*pstore.PeerInfo, error) { - addr, err := readLpMultiaddr(s) +func (r *Relay) handleCanHop(s inet.Stream, msg *pb.CircuitRelay) { + var err error + + if r.hop { + err = r.writeResponse(s, pb.CircuitRelay_SUCCESS) + } else { + err = r.writeResponse(s, pb.CircuitRelay_HOP_CANT_SPEAK_RELAY) + } + if err != nil { - return nil, err + log.Debugf("error writing relay response: %s", err.Error()) } - info, err := pstore.InfoFromP2pAddr(addr) + s.Close() +} + +func (r *Relay) handleError(s inet.Stream, code pb.CircuitRelay_Status) { + log.Warningf("relay error: %s (%d)", pb.CircuitRelay_Status_name[int32(code)], code) + err := r.writeResponse(s, code) if err != nil { - return nil, err + log.Debugf("error writing relay response: %s", err.Error()) } + s.Close() +} + +func (r *Relay) writeResponse(s inet.Stream, code pb.CircuitRelay_Status) error { + wr := ggio.NewDelimitedWriter(s) + + var msg pb.CircuitRelay + msg.Type = pb.CircuitRelay_STATUS.Enum() + msg.Code = code.Enum() - return info, nil + return wr.WriteMsg(&msg) } diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go new file mode 100644 index 0000000000..d18970414d --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -0,0 +1,47 @@ +package relay + +import ( + "errors" + + pb "github.com/libp2p/go-libp2p-circuit/pb" + + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" + mh "github.com/multiformats/go-multihash" +) + +func peerToPeerInfo(p *pb.CircuitRelay_Peer) (empty pstore.PeerInfo, err error) { + if p == nil { + return empty, errors.New("nil peer") + } + + h, err := mh.Cast(p.Id) + if err != nil { + return empty, err + } + + addrs := make([]ma.Multiaddr, len(p.Addrs)) + for i := 0; i < len(addrs); i++ { + a, err := ma.NewMultiaddrBytes(p.Addrs[i]) + if err != nil { + return empty, err + } + addrs[i] = a + } + + return pstore.PeerInfo{ID: peer.ID(h), Addrs: addrs}, nil +} + +func peerInfoToPeer(pi pstore.PeerInfo) *pb.CircuitRelay_Peer { + addrs := make([][]byte, len(pi.Addrs)) + for i := 0; i < len(addrs); i++ { + addrs[i] = pi.Addrs[i].Bytes() + } + + p := new(pb.CircuitRelay_Peer) + p.Id = []byte(pi.ID) + p.Addrs = addrs + + return p +} From ebdec667d319e8bd4cddfb2c1db31d8f4bfb0943 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 13:57:31 +0300 Subject: [PATCH 0266/3965] preserve pb status code type in RelayError --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 65a04375a1..1176841a85 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -47,11 +47,11 @@ var ( ) type RelayError struct { - Code int32 + Code pb.CircuitRelay_Status } func (e RelayError) Error() string { - return fmt.Sprintf("error opening relay circuit: %s (%d)", pb.CircuitRelay_Status_name[e.Code], e.Code) + return fmt.Sprintf("error opening relay circuit: %s (%d)", pb.CircuitRelay_Status_name[int32(e.Code)], e.Code) } func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error) { @@ -119,7 +119,7 @@ func (r *Relay) Dial(ctx context.Context, relay pstore.PeerInfo, dest pstore.Pee if msg.GetCode() != pb.CircuitRelay_SUCCESS { s.Close() - return nil, RelayError{int32(msg.GetCode())} + return nil, RelayError{msg.GetCode()} } return &Conn{Stream: s}, nil From 9e3ef8701d2b52d61afe71fca33d1602f9e7b206 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 14:18:44 +0300 Subject: [PATCH 0267/3965] hop relay must perform stop handshake; duh. --- .../internal/circuitv1-deprecated/relay.go | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 1176841a85..27cb51f43c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -179,6 +179,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { return } + // open stream ctp := r.host.Network().ConnsToPeer(dst.ID) if len(ctp) == 0 { if !r.active { @@ -203,6 +204,44 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { return } + // stop handshake + rd := ggio.NewDelimitedReader(bs, maxMessageSize) + wr := ggio.NewDelimitedWriter(bs) + + msg.Type = pb.CircuitRelay_STOP.Enum() + + err = wr.WriteMsg(msg) + if err != nil { + log.Debugf("error writing stop handshake: %s", err.Error()) + bs.Close() + r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) + return + } + + msg.Reset() + + err = rd.ReadMsg(msg) + if err != nil { + log.Debugf("error reading stop response: %s", err.Error()) + bs.Close() + r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) + return + } + + if msg.GetType() != pb.CircuitRelay_STATUS { + log.Debugf("unexpected relay stop response: not a status message (%d)", msg.GetType()) + bs.Close() + r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) + return + } + + if msg.GetCode() != pb.CircuitRelay_SUCCESS { + log.Debugf("relay stop failure: %d", msg.GetCode()) + bs.Close() + r.handleError(s, msg.GetCode()) + return + } + err = r.writeResponse(s, pb.CircuitRelay_SUCCESS) if err != nil { log.Debugf("error writing relay response: %s", err.Error()) @@ -210,6 +249,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { return } + // relay connection log.Infof("relaying connection between %s and %s", src.ID.Pretty(), dst.ID.Pretty()) go func() { From 8941943d8dcd005419dadcf6c2d1b1c524da9b31 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 14:25:00 +0300 Subject: [PATCH 0268/3965] update test --- .../circuitv1-deprecated/relay_test.go | 119 +++++++++++++----- 1 file changed, 89 insertions(+), 30 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index cb1a855021..c6f4516548 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -5,11 +5,13 @@ import ( "context" "fmt" "io/ioutil" - "strings" "testing" + "time" - bhost "github.com/libp2p/go-libp2p-blankhost" . "github.com/libp2p/go-libp2p-circuit" + pb "github.com/libp2p/go-libp2p-circuit/pb" + + bhost "github.com/libp2p/go-libp2p-blankhost" host "github.com/libp2p/go-libp2p-host" netutil "github.com/libp2p/go-libp2p-netutil" ma "github.com/multiformats/go-multiaddr" @@ -90,12 +92,13 @@ func TestBasicRelay(t *testing.T) { con.Close() }() - destma, err := ma.NewMultiaddr("/ipfs/" + hosts[2].ID().Pretty()) - if err != nil { - t.Fatal(err) - } + rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) + dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) - con, err := r1.Dial(ctx, hosts[1].ID(), destma) + rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + defer rcancel() + + con, err := r1.Dial(rctx, rinfo, dinfo) if err != nil { t.Fatal(err) } @@ -161,8 +164,11 @@ func TestBasicRelayDial(t *testing.T) { t.Fatal(err) } + rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + defer rcancel() + d := r1.Dialer() - con, err := d.DialPeer(ctx, hosts[2].ID(), relayaddr) + con, err := d.DialPeer(rctx, hosts[2].ID(), relayaddr) if err != nil { t.Fatal(err) } @@ -201,25 +207,34 @@ func TestRelayThroughNonHop(t *testing.T) { t.Fatal(err) } - destma, err := ma.NewMultiaddr("/ipfs/" + hosts[2].ID().Pretty()) - if err != nil { - t.Fatal(err) + rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) + dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) + + rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + defer rcancel() + + _, err = r1.Dial(rctx, rinfo, dinfo) + if err == nil { + t.Fatal("expected error") } - _, err = r1.Dial(ctx, hosts[1].ID(), destma) - if err.Error() != "protocol not supported" { - t.Fatal("expected 'protocol not supported' error") + rerr, ok := err.(RelayError) + if !ok { + t.Fatalf("expected RelayError: %#v", err) + } + + if rerr.Code != pb.CircuitRelay_HOP_CANT_SPEAK_RELAY { + t.Fatal("expected 'HOP_CANT_SPEAK_RELAY' error") } } -func TestDestNoRelay(t *testing.T) { +func TestRelayNoDestConnection(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() hosts := getNetHosts(t, ctx, 3) connect(t, hosts[0], hosts[1]) - connect(t, hosts[1], hosts[2]) r1, err := NewRelay(ctx, hosts[0]) if err != nil { @@ -231,21 +246,28 @@ func TestDestNoRelay(t *testing.T) { t.Fatal(err) } - destma, err := ma.NewMultiaddr("/ipfs/" + hosts[2].ID().Pretty()) - if err != nil { - t.Fatal(err) + rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) + dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) + + rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + defer rcancel() + + _, err = r1.Dial(rctx, rinfo, dinfo) + if err == nil { + t.Fatal("expected error") } - for i := 0; i < 10; i++ { - destma = ma.Join(destma, destma) + + rerr, ok := err.(RelayError) + if !ok { + t.Fatalf("expected RelayError: %#v", err) } - _, err = r1.Dial(ctx, hosts[1].ID(), destma) - if !strings.HasPrefix(err.Error(), fmt.Sprintf("%d: address length was too long", StatusRelayAddrErr)) { - t.Fatal(err) + if rerr.Code != pb.CircuitRelay_HOP_NO_CONN_TO_DST { + t.Fatal("expected 'HOP_NO_CONN_TO_DST' error") } } -func TestRelayNoDestConnection(t *testing.T) { +func TestActiveRelay(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -258,18 +280,55 @@ func TestRelayNoDestConnection(t *testing.T) { t.Fatal(err) } - _, err = NewRelay(ctx, hosts[1], OptHop) + _, err = NewRelay(ctx, hosts[1], OptHop, OptActive) + if err != nil { + t.Fatal(err) + } + + r3, err := NewRelay(ctx, hosts[2]) + if err != nil { + t.Fatal(err) + } + + msg := []byte("relay works!") + go func() { + list, err := r3.Listener() + if err != nil { + t.Error(err) + return + } + + con, err := list.Accept() + if err != nil { + t.Error(err) + return + } + + _, err = con.Write(msg) + if err != nil { + t.Error("failed to write", err) + return + } + con.Close() + }() + + rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) + dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) + + rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + defer rcancel() + + con, err := r1.Dial(rctx, rinfo, dinfo) if err != nil { t.Fatal(err) } - destma, err := ma.NewMultiaddr("/ipfs/" + hosts[2].ID().Pretty()) + data, err := ioutil.ReadAll(con) if err != nil { t.Fatal(err) } - _, err = r1.Dial(ctx, hosts[1].ID(), destma) - if err.Error() != "260: refusing to make new connection for relay" { - t.Fatal("expected this not to work") + if !bytes.Equal(data, msg) { + t.Fatal("message was incorrect:", string(data)) } } From 3943597bcec0f6159e772b94bbc4957bf51ee3c2 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 15:26:54 +0300 Subject: [PATCH 0269/3965] relay: return HOP_CANT_OPEN_DST_STREAM on stop stream errors --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 27cb51f43c..03cb38a6b4 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -214,7 +214,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { if err != nil { log.Debugf("error writing stop handshake: %s", err.Error()) bs.Close() - r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) + r.handleError(s, pb.CircuitRelay_HOP_CANT_OPEN_DST_STREAM) return } @@ -224,14 +224,14 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { if err != nil { log.Debugf("error reading stop response: %s", err.Error()) bs.Close() - r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) + r.handleError(s, pb.CircuitRelay_HOP_CANT_OPEN_DST_STREAM) return } if msg.GetType() != pb.CircuitRelay_STATUS { log.Debugf("unexpected relay stop response: not a status message (%d)", msg.GetType()) bs.Close() - r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) + r.handleError(s, pb.CircuitRelay_HOP_CANT_OPEN_DST_STREAM) return } From 1cee41522ddca3fc0e392d401e511a69a9804c21 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 15:37:24 +0300 Subject: [PATCH 0270/3965] hop relay: close bs on status response error --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 03cb38a6b4..d37cab213a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -245,6 +245,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { err = r.writeResponse(s, pb.CircuitRelay_SUCCESS) if err != nil { log.Debugf("error writing relay response: %s", err.Error()) + bs.Close() s.Close() return } From 1ad69518d15546f34ac4ef9e52bd72d88c4959c1 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 16:28:30 +0300 Subject: [PATCH 0271/3965] relay: use Peerstore to get self peer info, log bytes relayed --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index d37cab213a..7b855edca8 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -95,7 +95,7 @@ func (r *Relay) Dial(ctx context.Context, relay pstore.PeerInfo, dest pstore.Pee var msg pb.CircuitRelay msg.Type = pb.CircuitRelay_HOP.Enum() - msg.SrcPeer = peerInfoToPeer(pstore.PeerInfo{r.self, r.host.Addrs()}) + msg.SrcPeer = peerInfoToPeer(r.host.Peerstore().PeerInfo(r.self)) msg.DstPeer = peerInfoToPeer(dest) err = wr.WriteMsg(&msg) @@ -254,19 +254,21 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { log.Infof("relaying connection between %s and %s", src.ID.Pretty(), dst.ID.Pretty()) go func() { - _, err := io.Copy(s, bs) + count, err := io.Copy(s, bs) if err != io.EOF && err != nil { log.Debugf("relay copy error: %s", err) } s.Close() + log.Debugf("relayed %d bytes from %s to %s", count, dst.ID.Pretty(), src.ID.Pretty()) }() go func() { - _, err := io.Copy(bs, s) + count, err := io.Copy(bs, s) if err != io.EOF && err != nil { log.Debugf("relay copy error: %s", err) } bs.Close() + log.Debugf("relayed %d bytes from %s to %s", count, src.ID.Pretty(), dst.ID.Pretty()) }() } From 9883786568b913954e76c36b0ba1d8bd1171dd0b Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 18:32:07 +0300 Subject: [PATCH 0272/3965] relay: poluate remote field in dialed Conn --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 7b855edca8..25977e6522 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -122,7 +122,7 @@ func (r *Relay) Dial(ctx context.Context, relay pstore.PeerInfo, dest pstore.Pee return nil, RelayError{msg.GetCode()} } - return &Conn{Stream: s}, nil + return &Conn{Stream: s, remote: dest}, nil } func (r *Relay) handleNewStream(s inet.Stream) { From 37e58194250f31d57158779cf2504c84624e2a97 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 21:39:34 +0300 Subject: [PATCH 0273/3965] conn: connection ID, fix RemoteMultiaddr panic --- p2p/protocol/internal/circuitv1-deprecated/conn.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index f165647028..16c08459aa 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -42,7 +42,7 @@ func (c *Conn) RemoteAddr() net.Addr { } func (c *Conn) RemoteMultiaddr() ma.Multiaddr { - a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/%s", c.remote.ID.Pretty(), c.Conn().RemotePeer())) + a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", c.remote.ID.Pretty(), c.Conn().RemotePeer().Pretty())) if err != nil { panic(err) } @@ -83,5 +83,5 @@ func (c *Conn) RemotePublicKey() ic.PubKey { } func (c *Conn) ID() string { - return "TODO: relay conn ID" + return iconn.ID(c) } From 30adc968b1777e22d400842911a971d8a1a6e29f Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 17 Jul 2017 22:20:28 +0300 Subject: [PATCH 0274/3965] fix relay message input: ggio delimited reader buffers so we hand-craft a compatible delimited reader which doesn't. --- .../internal/circuitv1-deprecated/relay.go | 13 +++--- .../internal/circuitv1-deprecated/util.go | 43 +++++++++++++++++++ 2 files changed, 49 insertions(+), 7 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 25977e6522..7f926fd1c9 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -9,7 +9,6 @@ import ( pb "github.com/libp2p/go-libp2p-circuit/pb" - ggio "github.com/gogo/protobuf/io" logging "github.com/ipfs/go-log" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" @@ -89,8 +88,8 @@ func (r *Relay) Dial(ctx context.Context, relay pstore.PeerInfo, dest pstore.Pee return nil, err } - rd := ggio.NewDelimitedReader(s, maxMessageSize) - wr := ggio.NewDelimitedWriter(s) + rd := newDelimitedReader(s, maxMessageSize) + wr := newDelimitedWriter(s) var msg pb.CircuitRelay @@ -128,7 +127,7 @@ func (r *Relay) Dial(ctx context.Context, relay pstore.PeerInfo, dest pstore.Pee func (r *Relay) handleNewStream(s inet.Stream) { log.Infof("new relay stream from: %s", s.Conn().RemotePeer()) - rd := ggio.NewDelimitedReader(s, maxMessageSize) + rd := newDelimitedReader(s, maxMessageSize) var msg pb.CircuitRelay @@ -205,8 +204,8 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { } // stop handshake - rd := ggio.NewDelimitedReader(bs, maxMessageSize) - wr := ggio.NewDelimitedWriter(bs) + rd := newDelimitedReader(bs, maxMessageSize) + wr := newDelimitedWriter(bs) msg.Type = pb.CircuitRelay_STOP.Enum() @@ -322,7 +321,7 @@ func (r *Relay) handleError(s inet.Stream, code pb.CircuitRelay_Status) { } func (r *Relay) writeResponse(s inet.Stream, code pb.CircuitRelay_Status) error { - wr := ggio.NewDelimitedWriter(s) + wr := newDelimitedWriter(s) var msg pb.CircuitRelay msg.Type = pb.CircuitRelay_STATUS.Enum() diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index d18970414d..58e64657c2 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -1,10 +1,14 @@ package relay import ( + "encoding/binary" "errors" + "io" pb "github.com/libp2p/go-libp2p-circuit/pb" + ggio "github.com/gogo/protobuf/io" + proto "github.com/gogo/protobuf/proto" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" @@ -45,3 +49,42 @@ func peerInfoToPeer(pi pstore.PeerInfo) *pb.CircuitRelay_Peer { return p } + +type delimitedReader struct { + r io.Reader + buf []byte +} + +// the gogo protobuf NewDelimitedReader is buffered, which may eat up stream data +func newDelimitedReader(r io.Reader, maxSize int) *delimitedReader { + return &delimitedReader{r: r, buf: make([]byte, maxSize)} +} + +func (d *delimitedReader) ReadByte() (byte, error) { + buf := d.buf[:1] + _, err := d.r.Read(buf) + return buf[0], err +} + +func (d *delimitedReader) ReadMsg(msg proto.Message) error { + mlen, err := binary.ReadUvarint(d) + if err != nil { + return err + } + + if uint64(len(d.buf)) < mlen { + return errors.New("Message too large") + } + + buf := d.buf[:mlen] + _, err = io.ReadFull(d.r, buf) + if err != nil { + return err + } + + return proto.Unmarshal(buf, msg) +} + +func newDelimitedWriter(w io.Writer) ggio.WriteCloser { + return ggio.NewDelimitedWriter(w) +} From aadf0c53716bf7c49c79b2af48eddb0581eb4e71 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 10:59:03 +0300 Subject: [PATCH 0275/3965] conn: fix RemoteMultiaddr consistency so that it works for both active and passive connections. --- p2p/protocol/internal/circuitv1-deprecated/conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 16c08459aa..86cdd14718 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -42,7 +42,7 @@ func (c *Conn) RemoteAddr() net.Addr { } func (c *Conn) RemoteMultiaddr() ma.Multiaddr { - a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", c.remote.ID.Pretty(), c.Conn().RemotePeer().Pretty())) + a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", c.Conn().RemotePeer().Pretty(), c.remote.ID.Pretty())) if err != nil { panic(err) } From 93ae30dd90903b1f6e013a100840e568dbf5c93b Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 13:41:04 +0300 Subject: [PATCH 0276/3965] don't connect explicitly (unnecessary), add comments for non obvious things per @Stebalien's review comments. --- .../internal/circuitv1-deprecated/listen.go | 2 ++ .../internal/circuitv1-deprecated/relay.go | 32 ++++++++----------- .../internal/circuitv1-deprecated/util.go | 19 ++++++++--- 3 files changed, 29 insertions(+), 24 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index 8cc096eb8c..db6c7e4e2f 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -33,6 +33,8 @@ func (l *RelayListener) Accept() (tpt.Conn, error) { err := l.Relay().writeResponse(c.Stream, pb.CircuitRelay_SUCCESS) if err != nil { log.Debugf("error writing relay response: %s", err.Error()) + // this won't prevent the other side from continuing to write + // TODO fully close the stream when Reset is implemented c.Stream.Close() return nil, err } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 7f926fd1c9..7c36c7cbd8 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -23,6 +23,7 @@ const ProtoID = "/libp2p/circuit/relay/0.1.0" const maxMessageSize = 4096 var RelayAcceptTimeout = time.Minute +var HopConnectTimeout = 2 * time.Second type Relay struct { host host.Host @@ -78,10 +79,8 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error } func (r *Relay) Dial(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { - err := r.host.Connect(ctx, relay) - if err != nil { - return nil, err - } + + r.host.Peerstore().AddAddrs(relay.ID, relay.Addrs, pstore.TempAddrTTL) s, err := r.host.NewStream(ctx, relay.ID, ProtoID) if err != nil { @@ -180,26 +179,21 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { // open stream ctp := r.host.Network().ConnsToPeer(dst.ID) - if len(ctp) == 0 { - if !r.active { - r.handleError(s, pb.CircuitRelay_HOP_NO_CONN_TO_DST) - return - } - ctx, cancel := context.WithTimeout(r.ctx, time.Second*10) - defer cancel() - err = r.host.Connect(ctx, dst) - if err != nil { - log.Debugf("error opening relay connection to %s: %s", dst.ID.Pretty(), err.Error()) - r.handleError(s, pb.CircuitRelay_HOP_CANT_DIAL_DST) - return - } + if len(ctp) == 0 && !r.active { + r.handleError(s, pb.CircuitRelay_HOP_NO_CONN_TO_DST) + return } - bs, err := r.host.NewStream(r.ctx, dst.ID, ProtoID) + r.host.Peerstore().AddAddrs(dst.ID, dst.Addrs, pstore.TempAddrTTL) + + ctx, cancel := context.WithTimeout(r.ctx, HopConnectTimeout) + defer cancel() + + bs, err := r.host.NewStream(ctx, dst.ID, ProtoID) if err != nil { log.Debugf("error opening relay stream to %s: %s", dst.ID.Pretty(), err.Error()) - r.handleError(s, pb.CircuitRelay_HOP_CANT_OPEN_DST_STREAM) + r.handleError(s, pb.CircuitRelay_HOP_CANT_DIAL_DST) return } diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index 58e64657c2..b06c1aafe1 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -15,21 +15,21 @@ import ( mh "github.com/multiformats/go-multihash" ) -func peerToPeerInfo(p *pb.CircuitRelay_Peer) (empty pstore.PeerInfo, err error) { +func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { if p == nil { - return empty, errors.New("nil peer") + return pstore.PeerInfo{}, errors.New("nil peer") } h, err := mh.Cast(p.Id) if err != nil { - return empty, err + return pstore.PeerInfo{}, err } addrs := make([]ma.Multiaddr, len(p.Addrs)) for i := 0; i < len(addrs); i++ { a, err := ma.NewMultiaddrBytes(p.Addrs[i]) if err != nil { - return empty, err + return pstore.PeerInfo{}, err } addrs[i] = a } @@ -55,7 +55,16 @@ type delimitedReader struct { buf []byte } -// the gogo protobuf NewDelimitedReader is buffered, which may eat up stream data +// The gogo protobuf NewDelimitedReader is buffered, which may eat up stream data. +// So we need to implement a compatible delimited reader that reads unbuffered. +// There is a slowdown from unbuffered reading: when reading the message +// it can take multiple single byte Reads to read the length and another Read +// to read the message payload. +// However, this is not critical performance degradation as +// - the reader is utilized to read one (dialer, stop) or two messages (hop) during +// the handshake, so it's a drop in the water for the connection lifetime. +// - messages are small (max 4k) and the length fits in a couple of bytes, +// so overall we have at most three reads per message. func newDelimitedReader(r io.Reader, maxSize int) *delimitedReader { return &delimitedReader{r: r, buf: make([]byte, maxSize)} } From 2b8bc8164622a776de6a98021d96fd4013be2671 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 13:49:51 +0300 Subject: [PATCH 0277/3965] set HopConnectTimeout to 10s connection might have to resolve through the DHT, so it can take a while. --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 7c36c7cbd8..3b57452c57 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -23,7 +23,7 @@ const ProtoID = "/libp2p/circuit/relay/0.1.0" const maxMessageSize = 4096 var RelayAcceptTimeout = time.Minute -var HopConnectTimeout = 2 * time.Second +var HopConnectTimeout = 10 * time.Second type Relay struct { host host.Host From a556e2d9ebe35686043e7909d639c089d1e1944d Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 14:26:32 +0300 Subject: [PATCH 0278/3965] relay_test: more robust test --- .../circuitv1-deprecated/relay_test.go | 26 +++++++++++++------ 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index c6f4516548..6fe1255f23 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -55,6 +55,8 @@ func TestBasicRelay(t *testing.T) { connect(t, hosts[0], hosts[1]) connect(t, hosts[1], hosts[2]) + time.Sleep(10 * time.Millisecond) + r1, err := NewRelay(ctx, hosts[0]) if err != nil { t.Fatal(err) @@ -86,7 +88,7 @@ func TestBasicRelay(t *testing.T) { _, err = con.Write(msg) if err != nil { - t.Error("failed to write", err) + t.Error(err) return } con.Close() @@ -95,7 +97,7 @@ func TestBasicRelay(t *testing.T) { rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) - rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() con, err := r1.Dial(rctx, rinfo, dinfo) @@ -122,6 +124,8 @@ func TestBasicRelayDial(t *testing.T) { connect(t, hosts[0], hosts[1]) connect(t, hosts[1], hosts[2]) + time.Sleep(10 * time.Millisecond) + r1, err := NewRelay(ctx, hosts[0]) if err != nil { t.Fatal(err) @@ -153,7 +157,7 @@ func TestBasicRelayDial(t *testing.T) { _, err = con.Write(msg) if err != nil { - t.Error("failed to write", err) + t.Error(err) return } con.Close() @@ -164,7 +168,7 @@ func TestBasicRelayDial(t *testing.T) { t.Fatal(err) } - rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() d := r1.Dialer() @@ -192,6 +196,8 @@ func TestRelayThroughNonHop(t *testing.T) { connect(t, hosts[0], hosts[1]) connect(t, hosts[1], hosts[2]) + time.Sleep(10 * time.Millisecond) + r1, err := NewRelay(ctx, hosts[0]) if err != nil { t.Fatal(err) @@ -210,7 +216,7 @@ func TestRelayThroughNonHop(t *testing.T) { rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) - rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() _, err = r1.Dial(rctx, rinfo, dinfo) @@ -236,6 +242,8 @@ func TestRelayNoDestConnection(t *testing.T) { connect(t, hosts[0], hosts[1]) + time.Sleep(10 * time.Millisecond) + r1, err := NewRelay(ctx, hosts[0]) if err != nil { t.Fatal(err) @@ -249,7 +257,7 @@ func TestRelayNoDestConnection(t *testing.T) { rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) - rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() _, err = r1.Dial(rctx, rinfo, dinfo) @@ -275,6 +283,8 @@ func TestActiveRelay(t *testing.T) { connect(t, hosts[0], hosts[1]) + time.Sleep(10 * time.Millisecond) + r1, err := NewRelay(ctx, hosts[0]) if err != nil { t.Fatal(err) @@ -306,7 +316,7 @@ func TestActiveRelay(t *testing.T) { _, err = con.Write(msg) if err != nil { - t.Error("failed to write", err) + t.Error(err) return } con.Close() @@ -315,7 +325,7 @@ func TestActiveRelay(t *testing.T) { rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) - rctx, rcancel := context.WithTimeout(ctx, 10*time.Second) + rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() con, err := r1.Dial(rctx, rinfo, dinfo) From 80dbe02b30a1c75b62f87094102d4b6e196641e8 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 15:15:40 +0300 Subject: [PATCH 0279/3965] dialer: implement transport.Dialer interface --- .../internal/circuitv1-deprecated/dial.go | 16 ++++++++++++---- .../internal/circuitv1-deprecated/relay.go | 2 +- .../internal/circuitv1-deprecated/relay_test.go | 12 ++++++------ 3 files changed, 19 insertions(+), 11 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 936cc21d71..f48e4138de 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -4,12 +4,13 @@ import ( "context" "fmt" - peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) +var _ tpt.Dialer = (*Dialer)(nil) + type Dialer Relay func (d *Dialer) Relay() *Relay { @@ -20,7 +21,11 @@ func (r *Relay) Dialer() *Dialer { return (*Dialer)(r) } -func (d *Dialer) DialPeer(ctx context.Context, p peer.ID, a ma.Multiaddr) (tpt.Conn, error) { +func (d *Dialer) Dial(a ma.Multiaddr) (tpt.Conn, error) { + return d.DialContext(d.ctx, a) +} + +func (d *Dialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn, error) { if !d.Matches(a) { return nil, fmt.Errorf("%s is not a relay address", a) } @@ -42,9 +47,12 @@ func (d *Dialer) DialPeer(ctx context.Context, p peer.ID, a ma.Multiaddr) (tpt.C return nil, err } - dinfo := pstore.PeerInfo{ID: p, Addrs: []ma.Multiaddr{destaddr}} + dinfo, err := pstore.InfoFromP2pAddr(destaddr) + if err != nil { + return nil, err + } - return d.Relay().Dial(ctx, *rinfo, dinfo) + return d.Relay().DialPeer(ctx, *rinfo, *dinfo) } func (d *Dialer) Matches(a ma.Multiaddr) bool { diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 3b57452c57..09ec1eb73c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -78,7 +78,7 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error return r, nil } -func (r *Relay) Dial(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { +func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { r.host.Peerstore().AddAddrs(relay.ID, relay.Addrs, pstore.TempAddrTTL) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 6fe1255f23..bf5f322d91 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -100,7 +100,7 @@ func TestBasicRelay(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - con, err := r1.Dial(rctx, rinfo, dinfo) + con, err := r1.DialPeer(rctx, rinfo, dinfo) if err != nil { t.Fatal(err) } @@ -163,7 +163,7 @@ func TestBasicRelayDial(t *testing.T) { con.Close() }() - relayaddr, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit", hosts[1].ID().Pretty())) + addr, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", hosts[1].ID().Pretty(), hosts[2].ID().Pretty())) if err != nil { t.Fatal(err) } @@ -172,7 +172,7 @@ func TestBasicRelayDial(t *testing.T) { defer rcancel() d := r1.Dialer() - con, err := d.DialPeer(rctx, hosts[2].ID(), relayaddr) + con, err := d.DialContext(rctx, addr) if err != nil { t.Fatal(err) } @@ -219,7 +219,7 @@ func TestRelayThroughNonHop(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - _, err = r1.Dial(rctx, rinfo, dinfo) + _, err = r1.DialPeer(rctx, rinfo, dinfo) if err == nil { t.Fatal("expected error") } @@ -260,7 +260,7 @@ func TestRelayNoDestConnection(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - _, err = r1.Dial(rctx, rinfo, dinfo) + _, err = r1.DialPeer(rctx, rinfo, dinfo) if err == nil { t.Fatal("expected error") } @@ -328,7 +328,7 @@ func TestActiveRelay(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - con, err := r1.Dial(rctx, rinfo, dinfo) + con, err := r1.DialPeer(rctx, rinfo, dinfo) if err != nil { t.Fatal(err) } From d442f301dee94a63137b7777faa8d514eb350873 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 15:28:54 +0300 Subject: [PATCH 0280/3965] listen: implement transport.Listener panic stubs --- .../internal/circuitv1-deprecated/listen.go | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index db6c7e4e2f..b271c0f465 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -1,6 +1,7 @@ package relay import ( + "fmt" "net" pb "github.com/libp2p/go-libp2p-circuit/pb" @@ -23,10 +24,6 @@ func (r *Relay) Listener() (tpt.Listener, error) { return (*RelayListener)(r), nil } -func (r *Relay) Matches(a ma.Multiaddr) bool { - return false -} - func (l *RelayListener) Accept() (tpt.Conn, error) { select { case c := <-l.incoming: @@ -48,15 +45,22 @@ func (l *RelayListener) Accept() (tpt.Conn, error) { } func (l *RelayListener) Addr() net.Addr { - panic("oh no") + return &NetAddr{ + Relay: "any", + Remote: "any", + } } func (l *RelayListener) Multiaddr() ma.Multiaddr { - panic("oh no") + a, err := ma.NewMultiaddr(fmt.Sprintf("/p2p-circuit/ipfs/%s", l.self.Pretty())) + if err != nil { + panic(err) + } + return a } func (l *RelayListener) LocalPeer() peer.ID { - return l.Relay().self + return l.self } func (l *RelayListener) SetAddrFilters(f *filter.Filters) { From f26747379662f443dc0855be7aa9a96b79e56e90 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 18:23:30 +0300 Subject: [PATCH 0281/3965] transport: implement transport.Transport interface --- .../internal/circuitv1-deprecated/conn.go | 5 +-- .../internal/circuitv1-deprecated/relay.go | 4 +-- .../circuitv1-deprecated/transport.go | 33 +++++++++++++++++++ 3 files changed, 38 insertions(+), 4 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 86cdd14718..35043e5e39 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -16,7 +16,8 @@ import ( type Conn struct { inet.Stream - remote pstore.PeerInfo + remote pstore.PeerInfo + transport tpt.Transport } var _ iconn.Conn = (*Conn)(nil) @@ -63,7 +64,7 @@ func (c *Conn) LocalAddr() net.Addr { } func (c *Conn) Transport() tpt.Transport { - panic("does anyone really call this?") + return c.transport } func (c *Conn) LocalPeer() peer.ID { diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 09ec1eb73c..ef8bb7e4fc 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -120,7 +120,7 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore return nil, RelayError{msg.GetCode()} } - return &Conn{Stream: s, remote: dest}, nil + return &Conn{Stream: s, remote: dest, transport: r.Transport()}, nil } func (r *Relay) handleNewStream(s inet.Stream) { @@ -283,7 +283,7 @@ func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { r.host.Peerstore().AddAddrs(src.ID, src.Addrs, pstore.TempAddrTTL) select { - case r.incoming <- &Conn{Stream: s, remote: src}: + case r.incoming <- &Conn{Stream: s, remote: src, transport: r.Transport()}: case <-time.After(RelayAcceptTimeout): r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) } diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 24c09fd874..2a3d71ea49 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -1,6 +1,9 @@ package relay import ( + "fmt" + + tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) @@ -15,3 +18,33 @@ var RelayMaddrProtocol = ma.Protocol{ func init() { ma.AddProtocol(RelayMaddrProtocol) } + +var _ tpt.Transport = (*Transport)(nil) + +type Transport Relay + +func (t *Transport) Relay() *Relay { + return (*Relay)(t) +} + +func (r *Relay) Transport() *Transport { + return (*Transport)(r) +} + +func (t *Transport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { + if !t.Matches(laddr) { + return nil, fmt.Errorf("%s is not a relay address", laddr) + } + return t.Relay().Dialer(), nil +} + +func (t *Transport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { + if !t.Matches(laddr) { + return nil, fmt.Errorf("%s is not a relay address", laddr) + } + return t.Relay().Listener() +} + +func (t *Transport) Matches(a ma.Multiaddr) bool { + return t.Relay().Dialer().Matches(a) +} From d0ae18b80943eb082daef6ae7593c83378dabedc Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 19:02:13 +0300 Subject: [PATCH 0282/3965] relay: only AddAddrs if there are some --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index ef8bb7e4fc..b0ba572f99 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -80,7 +80,9 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { - r.host.Peerstore().AddAddrs(relay.ID, relay.Addrs, pstore.TempAddrTTL) + if len(relay.Addrs) > 0 { + r.host.Peerstore().AddAddrs(relay.ID, relay.Addrs, pstore.TempAddrTTL) + } s, err := r.host.NewStream(ctx, relay.ID, ProtoID) if err != nil { @@ -185,7 +187,9 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { return } - r.host.Peerstore().AddAddrs(dst.ID, dst.Addrs, pstore.TempAddrTTL) + if len(dst.Addrs) > 0 { + r.host.Peerstore().AddAddrs(dst.ID, dst.Addrs, pstore.TempAddrTTL) + } ctx, cancel := context.WithTimeout(r.ctx, HopConnectTimeout) defer cancel() From d6401c780381e4a5e97cb5099243da218ddbc31e Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 20:16:26 +0300 Subject: [PATCH 0283/3965] relay: remove unused fields from Relay --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index b0ba572f99..7e7a0ad9f1 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -4,7 +4,6 @@ import ( "context" "fmt" "io" - "sync" "time" pb "github.com/libp2p/go-libp2p-circuit/pb" @@ -34,9 +33,6 @@ type Relay struct { hop bool incoming chan *Conn - - arLk sync.Mutex - activeRelays []*Conn } type RelayOpt int From 7428b11e553b147816435c9d7835894d9787b621 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 18 Jul 2017 20:28:32 +0300 Subject: [PATCH 0284/3965] interface - type naming consistency: RelayDialer and RelayTransport --- .../internal/circuitv1-deprecated/dial.go | 16 ++++++++-------- .../internal/circuitv1-deprecated/listen.go | 4 ++-- .../circuitv1-deprecated/relay_test.go | 18 +++--------------- .../internal/circuitv1-deprecated/transport.go | 18 +++++++++--------- 4 files changed, 22 insertions(+), 34 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index f48e4138de..8e543e7b86 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -9,23 +9,23 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -var _ tpt.Dialer = (*Dialer)(nil) +var _ tpt.Dialer = (*RelayDialer)(nil) -type Dialer Relay +type RelayDialer Relay -func (d *Dialer) Relay() *Relay { +func (d *RelayDialer) Relay() *Relay { return (*Relay)(d) } -func (r *Relay) Dialer() *Dialer { - return (*Dialer)(r) +func (r *Relay) Dialer() *RelayDialer { + return (*RelayDialer)(r) } -func (d *Dialer) Dial(a ma.Multiaddr) (tpt.Conn, error) { +func (d *RelayDialer) Dial(a ma.Multiaddr) (tpt.Conn, error) { return d.DialContext(d.ctx, a) } -func (d *Dialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn, error) { +func (d *RelayDialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn, error) { if !d.Matches(a) { return nil, fmt.Errorf("%s is not a relay address", a) } @@ -55,7 +55,7 @@ func (d *Dialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn, err return d.Relay().DialPeer(ctx, *rinfo, *dinfo) } -func (d *Dialer) Matches(a ma.Multiaddr) bool { +func (d *RelayDialer) Matches(a ma.Multiaddr) bool { _, err := a.ValueForProtocol(P_CIRCUIT) return err == nil } diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index b271c0f465..dfec479ab3 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -20,8 +20,8 @@ func (l *RelayListener) Relay() *Relay { return (*Relay)(l) } -func (r *Relay) Listener() (tpt.Listener, error) { - return (*RelayListener)(r), nil +func (r *Relay) Listener() *RelayListener { + return (*RelayListener)(r) } func (l *RelayListener) Accept() (tpt.Conn, error) { diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index bf5f322d91..5ad1c7b2fb 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -74,11 +74,7 @@ func TestBasicRelay(t *testing.T) { msg := []byte("relay works!") go func() { - list, err := r3.Listener() - if err != nil { - t.Error(err) - return - } + list := r3.Listener() con, err := list.Accept() if err != nil { @@ -143,11 +139,7 @@ func TestBasicRelayDial(t *testing.T) { msg := []byte("relay works!") go func() { - list, err := r3.Listener() - if err != nil { - t.Error(err) - return - } + list := r3.Listener() con, err := list.Accept() if err != nil { @@ -302,11 +294,7 @@ func TestActiveRelay(t *testing.T) { msg := []byte("relay works!") go func() { - list, err := r3.Listener() - if err != nil { - t.Error(err) - return - } + list := r3.Listener() con, err := list.Accept() if err != nil { diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 2a3d71ea49..52dc86bd5f 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -19,32 +19,32 @@ func init() { ma.AddProtocol(RelayMaddrProtocol) } -var _ tpt.Transport = (*Transport)(nil) +var _ tpt.Transport = (*RelayTransport)(nil) -type Transport Relay +type RelayTransport Relay -func (t *Transport) Relay() *Relay { +func (t *RelayTransport) Relay() *Relay { return (*Relay)(t) } -func (r *Relay) Transport() *Transport { - return (*Transport)(r) +func (r *Relay) Transport() *RelayTransport { + return (*RelayTransport)(r) } -func (t *Transport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { +func (t *RelayTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { if !t.Matches(laddr) { return nil, fmt.Errorf("%s is not a relay address", laddr) } return t.Relay().Dialer(), nil } -func (t *Transport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { +func (t *RelayTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { if !t.Matches(laddr) { return nil, fmt.Errorf("%s is not a relay address", laddr) } - return t.Relay().Listener() + return t.Relay().Listener(), nil } -func (t *Transport) Matches(a ma.Multiaddr) bool { +func (t *RelayTransport) Matches(a ma.Multiaddr) bool { return t.Relay().Dialer().Matches(a) } From fd916f70a701add4b3a39c0e3ea1fc95e604c85d Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 19 Jul 2017 10:03:47 +0700 Subject: [PATCH 0285/3965] import the correct go-stream-muxer repo --- p2p/transport/quic/conn.go | 2 +- p2p/transport/quic/conn_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 511fa89a36..3b0310fca3 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -5,8 +5,8 @@ import ( "net" "sync" - smux "github.com/jbenet/go-stream-muxer" tpt "github.com/libp2p/go-libp2p-transport" + smux "github.com/libp2p/go-stream-muxer" quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 7c80681736..62288cbc86 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -5,7 +5,7 @@ import ( "net" "time" - smux "github.com/jbenet/go-stream-muxer" + smux "github.com/libp2p/go-stream-muxer" quic "github.com/lucas-clemente/quic-go" "github.com/lucas-clemente/quic-go/protocol" From 6d20f17c385925c6a180ba56a86f184556c0b02f Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 19 Jul 2017 11:37:01 +0700 Subject: [PATCH 0286/3965] import the correct go-stream-muxer repo --- p2p/muxer/muxer-multistream/multistream.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index 16b10164a2..ad453a9958 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -7,7 +7,7 @@ import ( "net" "time" - smux "github.com/jbenet/go-stream-muxer" + smux "github.com/libp2p/go-stream-muxer" mss "github.com/multiformats/go-multistream" ) From b2e8111dbcd149aac80d5c61ce2cdd86db184519 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Wed, 19 Jul 2017 09:58:25 +0200 Subject: [PATCH 0287/3965] add NewV1ProtectorFromBytes and tests for it --- p2p/net/pnet/codec.go | 2 +- p2p/net/pnet/codec_test.go | 4 ++-- p2p/net/pnet/generate.go | 22 +++++++++++++++++----- p2p/net/pnet/generate_test.go | 31 +++++++++++++++++++++++++++---- p2p/net/pnet/protector.go | 9 ++++++--- 5 files changed, 53 insertions(+), 15 deletions(-) diff --git a/p2p/net/pnet/codec.go b/p2p/net/pnet/codec.go index c013abbc60..ac6982834f 100644 --- a/p2p/net/pnet/codec.go +++ b/p2p/net/pnet/codec.go @@ -13,7 +13,7 @@ var ( headerPSKv1 = mc.Header(pathPSKv1) ) -func decodeV1PSKKey(in io.Reader) (*[32]byte, error) { +func decodeV1PSK(in io.Reader) (*[32]byte, error) { var err error in, err = mc.WrapTransformPathToHeader(in) if err != nil { diff --git a/p2p/net/pnet/codec_test.go b/p2p/net/pnet/codec_test.go index 11d912b62c..04afe65f08 100644 --- a/p2p/net/pnet/codec_test.go +++ b/p2p/net/pnet/codec_test.go @@ -22,7 +22,7 @@ func TestDecodeHex(t *testing.T) { b.WriteString("FF") } - psk, err := decodeV1PSKKey(b) + psk, err := decodeV1PSK(b) if err != nil { t.Fatal(err) } @@ -47,7 +47,7 @@ func TestDecodeB64(t *testing.T) { t.Fatal(err) } - psk, err := decodeV1PSKKey(b) + psk, err := decodeV1PSK(b) if err != nil { t.Fatal(err) } diff --git a/p2p/net/pnet/generate.go b/p2p/net/pnet/generate.go index f44dfe643a..cf220ced22 100644 --- a/p2p/net/pnet/generate.go +++ b/p2p/net/pnet/generate.go @@ -12,15 +12,27 @@ func newLine() io.Reader { } // GenerateV1PSK generates new PSK key that can be used with NewProtector -func GenerateV1PSK() io.Reader { - psk := make([]byte, 32) - rand.Read(psk) +func GenerateV1PSK() (io.Reader, error) { + psk, err := GenerateV1Bytes() + if err != nil { + return nil, err + } + hexPsk := make([]byte, len(psk)*2) - hex.Encode(hexPsk, psk) + hex.Encode(hexPsk, psk[:]) // just a shortcut to NewReader nr := func(b []byte) io.Reader { return bytes.NewReader(b) } - return io.MultiReader(nr(pathPSKv1), newLine(), nr([]byte("/base16/")), newLine(), nr(hexPsk)) + return io.MultiReader(nr(pathPSKv1), newLine(), nr([]byte("/base16/")), newLine(), nr(hexPsk)), nil +} + +func GenerateV1Bytes() (*[32]byte, error) { + psk := [32]byte{} + _, err := rand.Read(psk[:]) + if err != nil { + return nil, err + } + return &psk, nil } diff --git a/p2p/net/pnet/generate_test.go b/p2p/net/pnet/generate_test.go index a672163525..e06e4c1e7b 100644 --- a/p2p/net/pnet/generate_test.go +++ b/p2p/net/pnet/generate_test.go @@ -7,17 +7,26 @@ import ( ) func TestGeneratedPSKCanBeUsed(t *testing.T) { - psk := GenerateV1PSK() + psk, err := GenerateV1PSK() + if err != nil { + t.Fatal(err) + } - _, err := NewProtector(psk) + _, err = NewProtector(psk) if err != nil { t.Fatal(err) } } func TestGeneratedKeysAreDifferent(t *testing.T) { - psk1 := GenerateV1PSK() - psk2 := GenerateV1PSK() + psk1, err := GenerateV1PSK() + if err != nil { + t.Fatal(err) + } + psk2, err := GenerateV1PSK() + if err != nil { + t.Fatal(err) + } bpsk1, err := ioutil.ReadAll(psk1) if err != nil { t.Fatal(err) @@ -30,3 +39,17 @@ func TestGeneratedKeysAreDifferent(t *testing.T) { t.Fatal("generated keys are the same") } } + +func TestGeneratedV1BytesAreDifferent(t *testing.T) { + b1, err := GenerateV1Bytes() + if err != nil { + t.Fatal(err) + } + b2, err := GenerateV1Bytes() + if err != nil { + t.Fatal(err) + } + if bytes.Equal(b1[:], b2[:]) { + t.Fatal("generated keys are the same") + } +} diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index d613cac764..3442150a93 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -12,15 +12,18 @@ var _ ipnet.Protector = (*protector)(nil) // NewProtector creates ipnet.Protector instance from a io.Reader stream // that should include Multicodec encoded V1 PSK. func NewProtector(input io.Reader) (ipnet.Protector, error) { - psk, err := decodeV1PSKKey(input) + psk, err := decodeV1PSK(input) if err != nil { return nil, err } - f := fingerprint(psk) + return NewV1ProtectorFromBytes(psk) +} +// NewV1ProtectorFromBytes creates ipnet.Protector of the V1 version. +func NewV1ProtectorFromBytes(psk *[32]byte) (ipnet.Protector, error) { return &protector{ psk: psk, - fingerprint: f, + fingerprint: fingerprint(psk), }, nil } From ac48c753abf4324e45702caa5de6df40bfd8e05e Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 19 Jul 2017 20:49:01 +0300 Subject: [PATCH 0288/3965] AddRelayTransport: use relays for host transport --- .../internal/circuitv1-deprecated/dial.go | 2 +- .../circuitv1-deprecated/transport.go | 35 ++++++--- .../circuitv1-deprecated/transport_test.go | 77 +++++++++++++++++++ 3 files changed, 101 insertions(+), 13 deletions(-) create mode 100644 p2p/protocol/internal/circuitv1-deprecated/transport_test.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 8e543e7b86..53252644ba 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -56,6 +56,6 @@ func (d *RelayDialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn } func (d *RelayDialer) Matches(a ma.Multiaddr) bool { - _, err := a.ValueForProtocol(P_CIRCUIT) + _, err := a.ValueForProtocol(ma.P_CIRCUIT) return err == nil } diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 52dc86bd5f..6fb79304cf 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -1,24 +1,15 @@ package relay import ( + "context" "fmt" + host "github.com/libp2p/go-libp2p-host" + swarm "github.com/libp2p/go-libp2p-swarm" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) -const P_CIRCUIT = 290 - -var RelayMaddrProtocol = ma.Protocol{ - Code: P_CIRCUIT, - Name: "p2p-circuit", - Size: 0, -} - -func init() { - ma.AddProtocol(RelayMaddrProtocol) -} - var _ tpt.Transport = (*RelayTransport)(nil) type RelayTransport Relay @@ -48,3 +39,23 @@ func (t *RelayTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { func (t *RelayTransport) Matches(a ma.Multiaddr) bool { return t.Relay().Dialer().Matches(a) } + +// AddRelayTransport constructs a relay and adds it as a transport to the host network. +func AddRelayTransport(ctx context.Context, h host.Host, opts ...RelayOpt) error { + // the necessary methods are not part of the Network interface, only exported by Swarm + // TODO: generalize the network interface for adding tranports + n, ok := h.Network().(*swarm.Network) + if !ok { + return fmt.Errorf("%s is not a swarm network", h.Network()) + } + + s := n.Swarm() + + r, err := NewRelay(ctx, h, opts...) + if err != nil { + return err + } + + s.AddTransport(r.Transport()) + return s.AddListenAddr(r.Listener().Multiaddr()) +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go new file mode 100644 index 0000000000..6eb12efbe0 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go @@ -0,0 +1,77 @@ +package relay_test + +import ( + "bytes" + "context" + "fmt" + "io/ioutil" + "testing" + "time" + + . "github.com/libp2p/go-libp2p-circuit" + + inet "github.com/libp2p/go-libp2p-net" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" +) + +func TestRelayTransport(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 3) + + connect(t, hosts[0], hosts[1]) + connect(t, hosts[1], hosts[2]) + + time.Sleep(10 * time.Millisecond) + + err := AddRelayTransport(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + err = AddRelayTransport(ctx, hosts[1], OptHop) + if err != nil { + t.Fatal(err) + } + + err = AddRelayTransport(ctx, hosts[2]) + if err != nil { + t.Fatal(err) + } + + const proto = "test/relay-transport" + + msg := []byte("relay works!") + handler := func(s inet.Stream) { + s.Write(msg) + s.Close() + } + + hosts[2].SetStreamHandler(proto, handler) + + addr, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", hosts[1].ID().Pretty(), hosts[2].ID().Pretty())) + if err != nil { + t.Fatal(err) + } + + rctx, rcancel := context.WithTimeout(ctx, time.Second) + defer rcancel() + + hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) + + s, err := hosts[0].NewStream(rctx, hosts[2].ID(), proto) + if err != nil { + t.Fatal(err) + } + + data, err := ioutil.ReadAll(s) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(data, msg) { + t.Fatal("message was incorrect:", string(data)) + } +} From 264939d2e027dfe8f873f093863b1c0cb97bc6d5 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 19 Jul 2017 21:28:59 +0300 Subject: [PATCH 0289/3965] transport_test: check error in stream handler --- p2p/protocol/internal/circuitv1-deprecated/transport_test.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go index 6eb12efbe0..101622fb5d 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go @@ -45,7 +45,10 @@ func TestRelayTransport(t *testing.T) { msg := []byte("relay works!") handler := func(s inet.Stream) { - s.Write(msg) + _, err := s.Write(msg) + if err != nil { + t.Error(err) + } s.Close() } From 3cb456daa060d7cd2ab955391193712f857c7651 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 19 Jul 2017 22:59:44 +0300 Subject: [PATCH 0290/3965] dynamic protocol registration --- .../internal/circuitv1-deprecated/dial.go | 2 +- .../circuitv1-deprecated/transport.go | 30 +++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 53252644ba..8e543e7b86 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -56,6 +56,6 @@ func (d *RelayDialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn } func (d *RelayDialer) Matches(a ma.Multiaddr) bool { - _, err := a.ValueForProtocol(ma.P_CIRCUIT) + _, err := a.ValueForProtocol(P_CIRCUIT) return err == nil } diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 6fb79304cf..480ebdd3ae 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -4,12 +4,42 @@ import ( "context" "fmt" + addrutil "github.com/libp2p/go-addr-util" host "github.com/libp2p/go-libp2p-host" swarm "github.com/libp2p/go-libp2p-swarm" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) +const P_CIRCUIT = 290 + +var Protocol = ma.Protocol{ + Code: P_CIRCUIT, + Size: 0, + Name: "p2p-circuit", + VCode: ma.CodeToVarint(P_CIRCUIT), +} + +func init() { + ma.AddProtocol(Protocol) + + // Add dialer transport + const proto = "/ipfs/p2p-circuit/ipfs" + tps := addrutil.SupportedTransportStrings + + err := addrutil.AddTransport(proto) + if err != nil { + panic(err) + } + + for _, tp := range tps { + err = addrutil.AddTransport(tp + proto) + if err != nil { + panic(err) + } + } +} + var _ tpt.Transport = (*RelayTransport)(nil) type RelayTransport Relay From b671627aafc890992946cfbb2878990495619d33 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 20 Jul 2017 10:55:46 +0300 Subject: [PATCH 0291/3965] transport: use correct format specifier for error --- p2p/protocol/internal/circuitv1-deprecated/transport.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 480ebdd3ae..614b252e4c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -76,7 +76,7 @@ func AddRelayTransport(ctx context.Context, h host.Host, opts ...RelayOpt) error // TODO: generalize the network interface for adding tranports n, ok := h.Network().(*swarm.Network) if !ok { - return fmt.Errorf("%s is not a swarm network", h.Network()) + return fmt.Errorf("%v is not a swarm network", h.Network()) } s := n.Swarm() From 1536c8a0637b63ecec654edc689659d8a173b723 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 20 Jul 2017 19:46:19 +0300 Subject: [PATCH 0292/3965] relay: no need to error on empty src multiaddrs --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 7e7a0ad9f1..5e2aaec60e 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -267,7 +267,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { src, err := peerToPeerInfo(msg.GetSrcPeer()) - if err != nil || len(src.Addrs) == 0 { + if err != nil { r.handleError(s, pb.CircuitRelay_STOP_SRC_MULTIADDR_INVALID) return } @@ -280,7 +280,9 @@ func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { log.Infof("relay connection from: %s", src.ID) - r.host.Peerstore().AddAddrs(src.ID, src.Addrs, pstore.TempAddrTTL) + if len(src.Addrs) > 0 { + r.host.Peerstore().AddAddrs(src.ID, src.Addrs, pstore.TempAddrTTL) + } select { case r.incoming <- &Conn{Stream: s, remote: src, transport: r.Transport()}: From 4333b4cf3029e2931cfa3944ee4443ef32fd9148 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 20 Jul 2017 01:24:31 -0700 Subject: [PATCH 0293/3965] update deps * Also, make notifications async to prevent them from deadlocking. --- p2p/net/swarm/swarm_notif_test.go | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index d9e9df62a5..8ebcccb973 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -17,8 +17,10 @@ func streamsSame(a, b inet.Stream) bool { } func TestNotifications(t *testing.T) { + const swarmSize = 5 + ctx := context.Background() - swarms := makeSwarms(ctx, t, 5) + swarms := makeSwarms(ctx, t, swarmSize) defer func() { for _, s := range swarms { s.Close() @@ -30,7 +32,7 @@ func TestNotifications(t *testing.T) { // signup notifs notifiees := make([]*netNotifiee, len(swarms)) for i, swarm := range swarms { - n := newNetNotifiee() + n := newNetNotifiee(swarmSize) swarm.Notify(n) notifiees[i] = n } @@ -184,14 +186,14 @@ type netNotifiee struct { closedStream chan inet.Stream } -func newNetNotifiee() *netNotifiee { +func newNetNotifiee(buffer int) *netNotifiee { return &netNotifiee{ - listen: make(chan ma.Multiaddr), - listenClose: make(chan ma.Multiaddr), - connected: make(chan inet.Conn), - disconnected: make(chan inet.Conn), - openedStream: make(chan inet.Stream), - closedStream: make(chan inet.Stream), + listen: make(chan ma.Multiaddr, buffer), + listenClose: make(chan ma.Multiaddr, buffer), + connected: make(chan inet.Conn, buffer), + disconnected: make(chan inet.Conn, buffer), + openedStream: make(chan inet.Stream, buffer), + closedStream: make(chan inet.Stream, buffer), } } From 201ce9e9cd67670d0570fe66b6d01dfcb281aeb7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 24 Jul 2017 12:30:40 +0300 Subject: [PATCH 0294/3965] relay: CanHop for testing hop relays --- .../internal/circuitv1-deprecated/relay.go | 40 +++++++++++++++++++ .../circuitv1-deprecated/relay_test.go | 30 ++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 5e2aaec60e..56e45bee40 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "io" + // "sync" "time" pb "github.com/libp2p/go-libp2p-circuit/pb" @@ -33,6 +34,9 @@ type Relay struct { hop bool incoming chan *Conn + + // relays map[peer.ID]struct{} + // mx sync.Mutex } type RelayOpt int @@ -56,6 +60,7 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error ctx: ctx, self: h.ID(), incoming: make(chan *Conn), + // relays: make(map[peer.ID]struct{}), } for _, opt := range opts { @@ -70,6 +75,7 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error } h.SetStreamHandler(ProtoID, r.handleNewStream) + // h.Network().Notify(r.Notifiee()) return r, nil } @@ -121,6 +127,40 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore return &Conn{Stream: s, remote: dest, transport: r.Transport()}, nil } +func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { + s, err := r.host.NewStream(ctx, id, ProtoID) + if err != nil { + return false, err + } + + defer s.Close() + + rd := newDelimitedReader(s, maxMessageSize) + wr := newDelimitedWriter(s) + + var msg pb.CircuitRelay + + msg.Type = pb.CircuitRelay_CAN_HOP.Enum() + + err = wr.WriteMsg(&msg) + if err != nil { + return false, err + } + + msg.Reset() + + err = rd.ReadMsg(&msg) + if err != nil { + return false, err + } + + if msg.GetType() != pb.CircuitRelay_STATUS { + return false, fmt.Errorf("unexpected relay response; not a status message (%d)", msg.GetType()) + } + + return msg.GetCode() == pb.CircuitRelay_SUCCESS, nil +} + func (r *Relay) handleNewStream(s inet.Stream) { log.Infof("new relay stream from: %s", s.Conn().RemotePeer()) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 5ad1c7b2fb..79017cdd8a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -330,3 +330,33 @@ func TestActiveRelay(t *testing.T) { t.Fatal("message was incorrect:", string(data)) } } + +func TestRelayCanHop(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 2) + + connect(t, hosts[0], hosts[1]) + + time.Sleep(10 * time.Millisecond) + + r1, err := NewRelay(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[1], OptHop) + if err != nil { + t.Fatal(err) + } + + canhop, err := r1.CanHop(ctx, hosts[1].ID()) + if err != nil { + t.Fatal(err) + } + + if !canhop { + t.Fatal("Relay can't hop") + } +} From 60fe9d2fd040ec5de8ce8b919bd9b32a405f37b7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 24 Jul 2017 12:34:14 +0300 Subject: [PATCH 0295/3965] notify: passive hop relay discovery --- .../internal/circuitv1-deprecated/notify.go | 58 +++++++++++++++++++ .../internal/circuitv1-deprecated/relay.go | 10 ++-- 2 files changed, 63 insertions(+), 5 deletions(-) create mode 100644 p2p/protocol/internal/circuitv1-deprecated/notify.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/notify.go b/p2p/protocol/internal/circuitv1-deprecated/notify.go new file mode 100644 index 0000000000..4608434478 --- /dev/null +++ b/p2p/protocol/internal/circuitv1-deprecated/notify.go @@ -0,0 +1,58 @@ +package relay + +import ( + "context" + "time" + + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" +) + +var _ inet.Notifiee = (*RelayNotifiee)(nil) + +type RelayNotifiee Relay + +func (r *Relay) Notifiee() inet.Notifiee { + return (*RelayNotifiee)(r) +} + +func (n *RelayNotifiee) Relay() *Relay { + return (*Relay)(n) +} + +func (n *RelayNotifiee) Listen(net inet.Network, a ma.Multiaddr) {} +func (n *RelayNotifiee) ListenClose(net inet.Network, a ma.Multiaddr) {} +func (n *RelayNotifiee) OpenedStream(net inet.Network, s inet.Stream) {} +func (n *RelayNotifiee) ClosedStream(net inet.Network, s inet.Stream) {} + +func (n *RelayNotifiee) Connected(s inet.Network, c inet.Conn) { + if n.Relay().Transport().Matches(c.RemoteMultiaddr()) { + return + } + + go func(id peer.ID) { + ctx, cancel := context.WithTimeout(n.ctx, time.Second) + defer cancel() + + canhop, err := n.Relay().CanHop(ctx, id) + + if err != nil { + log.Debugf("Error testing relay hop: %s", err.Error()) + return + } + + if canhop { + log.Debugf("Discovered hop relay %s", id.Pretty()) + n.mx.Lock() + n.relays[id] = struct{}{} + n.mx.Unlock() + } + }(c.RemotePeer()) +} + +func (n *RelayNotifiee) Disconnected(s inet.Network, c inet.Conn) { + n.mx.Lock() + delete(n.relays, c.RemotePeer()) + n.mx.Unlock() +} diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 56e45bee40..1f3f756060 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -4,7 +4,7 @@ import ( "context" "fmt" "io" - // "sync" + "sync" "time" pb "github.com/libp2p/go-libp2p-circuit/pb" @@ -35,8 +35,8 @@ type Relay struct { incoming chan *Conn - // relays map[peer.ID]struct{} - // mx sync.Mutex + relays map[peer.ID]struct{} + mx sync.Mutex } type RelayOpt int @@ -60,7 +60,7 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error ctx: ctx, self: h.ID(), incoming: make(chan *Conn), - // relays: make(map[peer.ID]struct{}), + relays: make(map[peer.ID]struct{}), } for _, opt := range opts { @@ -75,7 +75,7 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error } h.SetStreamHandler(ProtoID, r.handleNewStream) - // h.Network().Notify(r.Notifiee()) + h.Network().Notify(r.Notifiee()) return r, nil } From 12e647fd851e2b85007ebc5e8aa0cf28664b433d Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 24 Jul 2017 13:34:29 +0300 Subject: [PATCH 0296/3965] dial: support unspecific relay address dialing --- .../internal/circuitv1-deprecated/dial.go | 35 +++++++++++++++++-- .../circuitv1-deprecated/transport.go | 9 ++++- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 8e543e7b86..b86dc81295 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -4,6 +4,9 @@ import ( "context" "fmt" + pb "github.com/libp2p/go-libp2p-circuit/pb" + + peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" @@ -42,12 +45,17 @@ func (d *RelayDialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn } } - rinfo, err := pstore.InfoFromP2pAddr(relayaddr) + dinfo, err := pstore.InfoFromP2pAddr(destaddr) if err != nil { return nil, err } - dinfo, err := pstore.InfoFromP2pAddr(destaddr) + if len(relayaddr.Bytes()) == 0 { + // unspecific relay address, try dialing using known hop relays + return d.tryDialRelays(ctx, *dinfo) + } + + rinfo, err := pstore.InfoFromP2pAddr(relayaddr) if err != nil { return nil, err } @@ -55,6 +63,29 @@ func (d *RelayDialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn return d.Relay().DialPeer(ctx, *rinfo, *dinfo) } +func (d *RelayDialer) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) (tpt.Conn, error) { + var relays []peer.ID + d.mx.Lock() + for p := range d.relays { + relays = append(relays, p) + } + d.mx.Unlock() + + for _, relay := range relays { + rctx, cancel := context.WithTimeout(ctx, HopConnectTimeout) + c, err := d.Relay().DialPeer(rctx, pstore.PeerInfo{ID: relay}, dinfo) + cancel() + + if err == nil { + return c, nil + } + + log.Debugf("Error opening relay connection to %s: %s", dinfo.ID, err.Error()) + } + + return nil, RelayError{pb.CircuitRelay_HOP_NO_CONN_TO_DST} +} + func (d *RelayDialer) Matches(a ma.Multiaddr) bool { _, err := a.ValueForProtocol(P_CIRCUIT) return err == nil diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 614b252e4c..1396828747 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -24,10 +24,17 @@ func init() { ma.AddProtocol(Protocol) // Add dialer transport + const unspecific = "/p2p-circuit/ipfs" const proto = "/ipfs/p2p-circuit/ipfs" + tps := addrutil.SupportedTransportStrings - err := addrutil.AddTransport(proto) + err := addrutil.AddTransport(unspecific) + if err != nil { + panic(err) + } + + err = addrutil.AddTransport(proto) if err != nil { panic(err) } From 0672a1a6ce564f1dc2c4bd373acf6573b0596cec Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 24 Jul 2017 13:35:30 +0300 Subject: [PATCH 0297/3965] tests for unspecific relay address dialing also test full address dialing in transport. --- .../circuitv1-deprecated/relay_test.go | 68 ++++++++++++++++ .../circuitv1-deprecated/transport_test.go | 77 ++++++++++++++++--- 2 files changed, 136 insertions(+), 9 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 79017cdd8a..05c1154015 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -179,6 +179,74 @@ func TestBasicRelayDial(t *testing.T) { } } +func TestUnspecificRelayDial(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 3) + + r1, err := NewRelay(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[1], OptHop) + if err != nil { + t.Fatal(err) + } + + r3, err := NewRelay(ctx, hosts[2]) + if err != nil { + t.Fatal(err) + } + + connect(t, hosts[0], hosts[1]) + connect(t, hosts[1], hosts[2]) + + time.Sleep(100 * time.Millisecond) + + msg := []byte("relay works!") + go func() { + list := r3.Listener() + + con, err := list.Accept() + if err != nil { + t.Error(err) + return + } + + _, err = con.Write(msg) + if err != nil { + t.Error(err) + return + } + con.Close() + }() + + addr, err := ma.NewMultiaddr(fmt.Sprintf("/p2p-circuit/ipfs/%s", hosts[2].ID().Pretty())) + if err != nil { + t.Fatal(err) + } + + rctx, rcancel := context.WithTimeout(ctx, time.Second) + defer rcancel() + + d := r1.Dialer() + con, err := d.DialContext(rctx, addr) + if err != nil { + t.Fatal(err) + } + + data, err := ioutil.ReadAll(con) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(data, msg) { + t.Fatal("message was incorrect:", string(data)) + } +} + func TestRelayThroughNonHop(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go index 101622fb5d..734a9dc104 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go @@ -10,22 +10,20 @@ import ( . "github.com/libp2p/go-libp2p-circuit" + host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) +const TestProto = "test/relay-transport" + func TestRelayTransport(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() hosts := getNetHosts(t, ctx, 3) - connect(t, hosts[0], hosts[1]) - connect(t, hosts[1], hosts[2]) - - time.Sleep(10 * time.Millisecond) - err := AddRelayTransport(ctx, hosts[0]) if err != nil { t.Fatal(err) @@ -41,7 +39,10 @@ func TestRelayTransport(t *testing.T) { t.Fatal(err) } - const proto = "test/relay-transport" + connect(t, hosts[0], hosts[1]) + connect(t, hosts[1], hosts[2]) + + time.Sleep(100 * time.Millisecond) msg := []byte("relay works!") handler := func(s inet.Stream) { @@ -52,19 +53,77 @@ func TestRelayTransport(t *testing.T) { s.Close() } - hosts[2].SetStreamHandler(proto, handler) + hosts[2].SetStreamHandler(TestProto, handler) + + testFullAddressDial(t, hosts, msg) + testSpecificRelayDial(t, hosts, msg) + testUnspecificRelayDial(t, hosts, msg) +} + +func testFullAddressDial(t *testing.T, hosts []host.Host, msg []byte) { + addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/ipfs/%s/p2p-circuit/ipfs/%s", hosts[1].Addrs()[0].String(), hosts[1].ID().Pretty(), hosts[2].ID().Pretty())) + if err != nil { + t.Fatal(err) + } + + rctx, rcancel := context.WithTimeout(context.Background(), time.Second) + defer rcancel() + + hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) + + s, err := hosts[0].NewStream(rctx, hosts[2].ID(), TestProto) + if err != nil { + t.Fatal(err) + } + + data, err := ioutil.ReadAll(s) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(data, msg) { + t.Fatal("message was incorrect:", string(data)) + } +} +func testSpecificRelayDial(t *testing.T, hosts []host.Host, msg []byte) { addr, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", hosts[1].ID().Pretty(), hosts[2].ID().Pretty())) if err != nil { t.Fatal(err) } - rctx, rcancel := context.WithTimeout(ctx, time.Second) + rctx, rcancel := context.WithTimeout(context.Background(), time.Second) + defer rcancel() + + hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) + + s, err := hosts[0].NewStream(rctx, hosts[2].ID(), TestProto) + if err != nil { + t.Fatal(err) + } + + data, err := ioutil.ReadAll(s) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(data, msg) { + t.Fatal("message was incorrect:", string(data)) + } +} + +func testUnspecificRelayDial(t *testing.T, hosts []host.Host, msg []byte) { + addr, err := ma.NewMultiaddr(fmt.Sprintf("/p2p-circuit/ipfs/%s", hosts[2].ID().Pretty())) + if err != nil { + t.Fatal(err) + } + + rctx, rcancel := context.WithTimeout(context.Background(), time.Second) defer rcancel() hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) - s, err := hosts[0].NewStream(rctx, hosts[2].ID(), proto) + s, err := hosts[0].NewStream(rctx, hosts[2].ID(), TestProto) if err != nil { t.Fatal(err) } From 2bb102a04afcf4a8e2996c407f6a913b2a68cf66 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 24 Jul 2017 13:51:43 +0300 Subject: [PATCH 0298/3965] notify: avoid discarding relays on disconnects it's racey, plus relay is a sticky property of nodes; test if we have active connections instead at the time of dial. --- p2p/protocol/internal/circuitv1-deprecated/dial.go | 4 ++++ p2p/protocol/internal/circuitv1-deprecated/notify.go | 6 +----- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index b86dc81295..015fff5c4c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -72,6 +72,10 @@ func (d *RelayDialer) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) d.mx.Unlock() for _, relay := range relays { + if len(d.host.Network().ConnsToPeer(relay)) == 0 { + continue + } + rctx, cancel := context.WithTimeout(ctx, HopConnectTimeout) c, err := d.Relay().DialPeer(rctx, pstore.PeerInfo{ID: relay}, dinfo) cancel() diff --git a/p2p/protocol/internal/circuitv1-deprecated/notify.go b/p2p/protocol/internal/circuitv1-deprecated/notify.go index 4608434478..f3ea08cec3 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/notify.go +++ b/p2p/protocol/internal/circuitv1-deprecated/notify.go @@ -51,8 +51,4 @@ func (n *RelayNotifiee) Connected(s inet.Network, c inet.Conn) { }(c.RemotePeer()) } -func (n *RelayNotifiee) Disconnected(s inet.Network, c inet.Conn) { - n.mx.Lock() - delete(n.relays, c.RemotePeer()) - n.mx.Unlock() -} +func (n *RelayNotifiee) Disconnected(s inet.Network, c inet.Conn) {} From 208ee02c7e5563e139f0c0e2995e23e58d7ad6b7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 24 Jul 2017 14:02:48 +0300 Subject: [PATCH 0299/3965] dial: shuffle relays --- p2p/protocol/internal/circuitv1-deprecated/dial.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 015fff5c4c..5eeece3c0e 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -3,6 +3,7 @@ package relay import ( "context" "fmt" + "math/rand" pb "github.com/libp2p/go-libp2p-circuit/pb" @@ -71,6 +72,12 @@ func (d *RelayDialer) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) } d.mx.Unlock() + // shuffle list of relays, avoid overloading a specific relay + for i := range relays { + j := rand.Intn(i + 1) + relays[i], relays[j] = relays[j], relays[i] + } + for _, relay := range relays { if len(d.host.Network().ConnsToPeer(relay)) == 0 { continue From 3e901489de58035a4b0b101faf1b4255fe62cad0 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 24 Jul 2017 14:15:55 +0300 Subject: [PATCH 0300/3965] refactor transport test --- .../circuitv1-deprecated/transport_test.go | 37 ++++++++++++------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go index 734a9dc104..512ba8644b 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go @@ -18,10 +18,9 @@ import ( const TestProto = "test/relay-transport" -func TestRelayTransport(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() +var msg = []byte("relay works!") +func testSetupRelay(t *testing.T, ctx context.Context) []host.Host { hosts := getNetHosts(t, ctx, 3) err := AddRelayTransport(ctx, hosts[0]) @@ -44,7 +43,6 @@ func TestRelayTransport(t *testing.T) { time.Sleep(100 * time.Millisecond) - msg := []byte("relay works!") handler := func(s inet.Stream) { _, err := s.Write(msg) if err != nil { @@ -55,18 +53,21 @@ func TestRelayTransport(t *testing.T) { hosts[2].SetStreamHandler(TestProto, handler) - testFullAddressDial(t, hosts, msg) - testSpecificRelayDial(t, hosts, msg) - testUnspecificRelayDial(t, hosts, msg) + return hosts } -func testFullAddressDial(t *testing.T, hosts []host.Host, msg []byte) { +func TestFullAddressTransportDial(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := testSetupRelay(t, ctx) + addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/ipfs/%s/p2p-circuit/ipfs/%s", hosts[1].Addrs()[0].String(), hosts[1].ID().Pretty(), hosts[2].ID().Pretty())) if err != nil { t.Fatal(err) } - rctx, rcancel := context.WithTimeout(context.Background(), time.Second) + rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) @@ -86,13 +87,18 @@ func testFullAddressDial(t *testing.T, hosts []host.Host, msg []byte) { } } -func testSpecificRelayDial(t *testing.T, hosts []host.Host, msg []byte) { +func TestSpecificRelayTransportDial(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := testSetupRelay(t, ctx) + addr, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", hosts[1].ID().Pretty(), hosts[2].ID().Pretty())) if err != nil { t.Fatal(err) } - rctx, rcancel := context.WithTimeout(context.Background(), time.Second) + rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) @@ -112,13 +118,18 @@ func testSpecificRelayDial(t *testing.T, hosts []host.Host, msg []byte) { } } -func testUnspecificRelayDial(t *testing.T, hosts []host.Host, msg []byte) { +func TestUnspecificRelayTransportDial(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := testSetupRelay(t, ctx) + addr, err := ma.NewMultiaddr(fmt.Sprintf("/p2p-circuit/ipfs/%s", hosts[2].ID().Pretty())) if err != nil { t.Fatal(err) } - rctx, rcancel := context.WithTimeout(context.Background(), time.Second) + rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) From 332879a640a3c203af8e4f6494db00763dc0b4c6 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 25 Jul 2017 11:05:21 +0300 Subject: [PATCH 0301/3965] somewhat better logging around dialing --- p2p/protocol/internal/circuitv1-deprecated/dial.go | 2 +- p2p/protocol/internal/circuitv1-deprecated/relay.go | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 5eeece3c0e..a13fc4e650 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -91,7 +91,7 @@ func (d *RelayDialer) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) return c, nil } - log.Debugf("Error opening relay connection to %s: %s", dinfo.ID, err.Error()) + log.Debugf("error opening relay connection through %s: %s", dinfo.ID, err.Error()) } return nil, RelayError{pb.CircuitRelay_HOP_NO_CONN_TO_DST} diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 1f3f756060..4eb3c16a83 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -82,6 +82,8 @@ func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { + log.Debugf("dialing peer %s through relay %s", dest.ID, relay.ID) + if len(relay.Addrs) > 0 { r.host.Peerstore().AddAddrs(relay.ID, relay.Addrs, pstore.TempAddrTTL) } From 88cc56330bf9cc03b7aa0d5d9c41466a4f01c53c Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 25 Jul 2017 11:25:36 +0300 Subject: [PATCH 0302/3965] dial: better error for uspecific relay dial failure it's not a HOP_NO_CONN_TO_DST; the intermediate errors are in the logs. --- p2p/protocol/internal/circuitv1-deprecated/dial.go | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index a13fc4e650..1f01be6aa6 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -5,8 +5,6 @@ import ( "fmt" "math/rand" - pb "github.com/libp2p/go-libp2p-circuit/pb" - peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" tpt "github.com/libp2p/go-libp2p-transport" @@ -94,7 +92,7 @@ func (d *RelayDialer) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) log.Debugf("error opening relay connection through %s: %s", dinfo.ID, err.Error()) } - return nil, RelayError{pb.CircuitRelay_HOP_NO_CONN_TO_DST} + return nil, fmt.Errorf("Failed to dial through %d known relay hosts", len(relays)) } func (d *RelayDialer) Matches(a ma.Multiaddr) bool { From af1302af3ea4c560159cf4f4ff3379ea6595d7f4 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Wed, 26 Jul 2017 16:40:00 +0200 Subject: [PATCH 0303/3965] fix source address not being set in non REUSEPORT dialer --- p2p/transport/tcp/tcp.go | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index a30dcbb7b0..e851a2084d 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -51,6 +51,12 @@ func (t *TcpTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dial } var base manet.Dialer + la, err := manet.ToNetAddr(laddr) + if err != nil { + return nil, err // something wrong with laddr. + } + base.Dialer.LocalAddr = la + var doReuse bool for _, o := range opts { switch o := o.(type) { @@ -136,12 +142,6 @@ type tcpDialer struct { } func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReuse bool) (*tcpDialer, error) { - // get the local net.Addr manually - la, err := manet.ToNetAddr(laddr) - if err != nil { - return nil, err // something wrong with laddr. - } - var pattern mafmt.Pattern if TCP4.Matches(laddr) { pattern = TCP4 @@ -153,10 +153,7 @@ func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReu if doReuse && ReuseportIsAvailable() { rd := reuseport.Dialer{ - D: net.Dialer{ - LocalAddr: la, - Timeout: base.Timeout, - }, + D: base.Dialer, } return &tcpDialer{ From 0bd829259869523c43afd2b7258ee9dd32758dce Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 29 Jul 2017 10:42:25 +0300 Subject: [PATCH 0304/3965] connection RemoteMultiaddr returns partial relay address for consistency --- p2p/protocol/internal/circuitv1-deprecated/conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 35043e5e39..51693df59b 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -43,7 +43,7 @@ func (c *Conn) RemoteAddr() net.Addr { } func (c *Conn) RemoteMultiaddr() ma.Multiaddr { - a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", c.Conn().RemotePeer().Pretty(), c.remote.ID.Pretty())) + a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit", c.Conn().RemotePeer().Pretty())) if err != nil { panic(err) } From 38fbd1cd2be7f0d82e6e68fe0d3d3063646e50cf Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 29 Jul 2017 11:50:33 +0300 Subject: [PATCH 0305/3965] Revert "Report partial circuit address in Conn.RemoteMultiaddr" --- p2p/protocol/internal/circuitv1-deprecated/conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 51693df59b..35043e5e39 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -43,7 +43,7 @@ func (c *Conn) RemoteAddr() net.Addr { } func (c *Conn) RemoteMultiaddr() ma.Multiaddr { - a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit", c.Conn().RemotePeer().Pretty())) + a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", c.Conn().RemotePeer().Pretty(), c.remote.ID.Pretty())) if err != nil { panic(err) } From 07e3a6db8d09e654c1f32d489e4ff665b87b3cf2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Sun, 30 Jul 2017 23:25:59 +0200 Subject: [PATCH 0306/3965] Fix dialLimiter.fdConsuming counting --- p2p/net/swarm/limiter.go | 19 +++++++--- p2p/net/swarm/limiter_test.go | 66 +++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 67c6af4ddf..f2513d8e10 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -52,10 +52,10 @@ func newDialLimiter(df dialfunc) *dialLimiter { return newDialLimiterWithParams(df, concurrentFdDials, defaultPerPeerRateLimit) } -func newDialLimiterWithParams(df dialfunc, fdl, ppl int) *dialLimiter { +func newDialLimiterWithParams(df dialfunc, fdLimit, perPeerLimit int) *dialLimiter { return &dialLimiter{ - fdLimit: fdl, - perPeerLimit: ppl, + fdLimit: fdLimit, + perPeerLimit: perPeerLimit, waitingOnPeerLimit: make(map[peer.ID][]*dialJob), activePerPeer: make(map[peer.ID]int), dialFunc: df, @@ -68,6 +68,7 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { if addrutil.IsFDCostlyTransport(dj.addr) { dl.fdConsuming-- + if len(dl.waitingOnFd) > 0 { next := dl.waitingOnFd[0] dl.waitingOnFd = dl.waitingOnFd[1:] @@ -89,6 +90,7 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { waitlist := dl.waitingOnPeerLimit[dj.peer] if !dj.success && len(waitlist) > 0 { next := waitlist[0] + if len(waitlist) == 1 { delete(dl.waitingOnPeerLimit, dj.peer) } else { @@ -96,11 +98,20 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { } dl.activePerPeer[dj.peer]++ // just kidding, we still want this token + if addrutil.IsFDCostlyTransport(next.addr) { + if dl.fdConsuming >= dl.fdLimit { + dl.waitingOnFd = append(dl.waitingOnFd, next) + return + } + + // take token + dl.fdConsuming++ + } + // can kick this off right here, dials in this list already // have the other tokens needed go dl.executeDial(next) } - } // AddDialJob tries to take the needed tokens for starting the given dial job. diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index cbb6afdbc6..3111a5d107 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -320,3 +320,69 @@ func TestStressLimiter(t *testing.T) { } } } + +func TestFDLimitUnderflow(t *testing.T) { + dials := 0 + + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { + dials++ + + timeout := make(chan bool, 1) + go func() { + time.Sleep(time.Second * 5) + timeout <- true + }() + + select { + case <-ctx.Done(): + case <-timeout: + } + + return nil, fmt.Errorf("df timed out") + } + + l := newDialLimiterWithParams(df, 20, 3) + + var addrs []ma.Multiaddr + for i := 0; i <= 1000; i++ { + addrs = append(addrs, addrWithPort(t, i)) + } + + for i := 0; i < 1000; i++ { + go func(id peer.ID, i int) { + ctx, cancel := context.WithCancel(context.Background()) + + resp := make(chan dialResult) + l.AddDialJob(&dialJob{ + addr: addrs[i], + ctx: ctx, + peer: id, + resp: resp, + }) + + //cancel first 60 after 1s, next 60 after 2s + if i > 60 { + time.Sleep(time.Second * 1) + } + if i < 120 { + time.Sleep(time.Second * 1) + cancel() + return + } + defer cancel() + + for res := range resp { + if res.Err != nil { + return + } + t.Fatal("got dial res, shouldn't") + } + }(peer.ID(fmt.Sprintf("testpeer%d", i % 20)), i) + } + + time.Sleep(time.Second * 3) + + if l.fdConsuming < 0 { + t.Fatalf("l.fdConsuming < 0") + } +} From bb8f8d0696b43d3dbb02ec80b93b50a27bdd08d8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 31 Jul 2017 13:40:11 -0700 Subject: [PATCH 0307/3965] go fmt --- p2p/net/swarm/limiter_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 3111a5d107..6c6eeb68fa 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -377,7 +377,7 @@ func TestFDLimitUnderflow(t *testing.T) { } t.Fatal("got dial res, shouldn't") } - }(peer.ID(fmt.Sprintf("testpeer%d", i % 20)), i) + }(peer.ID(fmt.Sprintf("testpeer%d", i%20)), i) } time.Sleep(time.Second * 3) From 214392a00d418eb402d2ec411d77fb5cea007167 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 8 Aug 2017 11:00:42 +0700 Subject: [PATCH 0308/3965] implement the new stream reset recently added to smux.Stream It was added in https://github.com/libp2p/go-stream-muxer/commit/be2992bb9 --- p2p/transport/quic/conn.go | 6 ++++-- p2p/transport/quic/conn_test.go | 6 +++--- p2p/transport/quic/stream.go | 17 +++++++++++++++++ 3 files changed, 24 insertions(+), 5 deletions(-) create mode 100644 p2p/transport/quic/stream.go diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 3b0310fca3..1805334870 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -52,13 +52,15 @@ func newQuicConn(sess quic.Session, t tpt.Transport) (*quicConn, error) { } func (c *quicConn) AcceptStream() (smux.Stream, error) { - return c.sess.AcceptStream() + str, err := c.sess.AcceptStream() + return &stream{str}, err } // OpenStream opens a new stream // It blocks until a new stream can be opened (when limited by the QUIC maximum stream limit) func (c *quicConn) OpenStream() (smux.Stream, error) { - return c.sess.OpenStreamSync() + str, err := c.sess.OpenStreamSync() + return &stream{str}, err } func (c *quicConn) Serve(handler smux.StreamHandler) { diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 62288cbc86..8ef5755ae6 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -101,7 +101,7 @@ var _ = Describe("Conn", func() { sess.streamToOpen = s str, err := conn.OpenStream() Expect(err).ToNot(HaveOccurred()) - Expect(str).To(Equal(s)) + Expect(str.(*stream).Stream).To(Equal(s)) }) It("errors when it can't open a stream", func() { @@ -118,7 +118,7 @@ var _ = Describe("Conn", func() { sess.streamToAccept = s str, err := conn.AcceptStream() Expect(err).ToNot(HaveOccurred()) - Expect(str).To(Equal(s)) + Expect(str.(*stream).Stream).To(Equal(s)) }) It("errors when it can't open a stream", func() { @@ -154,7 +154,7 @@ var _ = Describe("Conn", func() { returned = true }() Eventually(func() bool { return handlerCalled }).Should(BeTrue()) - Expect(handlerCalledWith).To(Equal(str)) + Expect(handlerCalledWith.(*stream).Stream).To(Equal(str)) // make the go-routine return sess.streamAcceptErr = errors.New("stop test") }) diff --git a/p2p/transport/quic/stream.go b/p2p/transport/quic/stream.go new file mode 100644 index 0000000000..5d5ad242cf --- /dev/null +++ b/p2p/transport/quic/stream.go @@ -0,0 +1,17 @@ +package libp2pquic + +import ( + smux "github.com/libp2p/go-stream-muxer" + "github.com/lucas-clemente/quic-go" +) + +type stream struct { + quic.Stream +} + +var _ smux.Stream = &stream{} + +func (s *stream) Reset() error { + s.Stream.Reset(nil) + return nil +} From 2f4cb702e37d1b11ed1117dd780cb24d2019e754 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 8 Aug 2017 11:07:24 +0700 Subject: [PATCH 0309/3965] use the new quic-go session context The session context was introduced in https://github.com/lucas-clemente/quic-go/pull/774. --- p2p/transport/quic/conn.go | 2 +- p2p/transport/quic/conn_test.go | 29 +++++++++++++++++------------ 2 files changed, 18 insertions(+), 13 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 1805334870..3f8ebe91a7 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -78,7 +78,7 @@ func (c *quicConn) Close() error { } func (c *quicConn) watchClosed() { - c.sess.WaitUntilClosed() + <-c.sess.Context().Done() c.mutex.Lock() c.closed = true c.mutex.Unlock() diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 8ef5755ae6..4a38d222a8 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -1,6 +1,7 @@ package libp2pquic import ( + "context" "errors" "net" "time" @@ -25,12 +26,13 @@ func (s *mockStream) StreamID() protocol.StreamID { return s.id } func (s *mockStream) SetReadDeadline(time.Time) error { panic("not implemented") } func (s *mockStream) SetWriteDeadline(time.Time) error { panic("not implemented") } func (s *mockStream) SetDeadline(time.Time) error { panic("not implemented") } +func (s *mockStream) Context() context.Context { panic("not implemented") } var _ quic.Stream = &mockStream{} type mockQuicSession struct { - closed bool - waitUntilClosedChan chan struct{} // close this chan to make WaitUntilClosed return + closed bool + context context.Context localAddr net.Addr remoteAddr net.Addr @@ -51,22 +53,25 @@ func (s *mockQuicSession) OpenStream() (quic.Stream, error) { return s.streamToO func (s *mockQuicSession) OpenStreamSync() (quic.Stream, error) { return s.streamToOpen, s.streamOpenErr } -func (s *mockQuicSession) Close(error) error { s.closed = true; return nil } -func (s *mockQuicSession) LocalAddr() net.Addr { return s.localAddr } -func (s *mockQuicSession) RemoteAddr() net.Addr { return s.remoteAddr } -func (s *mockQuicSession) WaitUntilClosed() { <-s.waitUntilClosedChan } +func (s *mockQuicSession) Close(error) error { s.closed = true; return nil } +func (s *mockQuicSession) LocalAddr() net.Addr { return s.localAddr } +func (s *mockQuicSession) RemoteAddr() net.Addr { return s.remoteAddr } +func (s *mockQuicSession) Context() context.Context { return s.context } var _ = Describe("Conn", func() { var ( - conn *quicConn - sess *mockQuicSession + conn *quicConn + sess *mockQuicSession + ctxCancel context.CancelFunc ) BeforeEach(func() { + var ctx context.Context + ctx, ctxCancel = context.WithCancel(context.Background()) sess = &mockQuicSession{ - localAddr: &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 1337}, - remoteAddr: &net.UDPAddr{IP: net.IPv4(192, 168, 13, 37), Port: 1234}, - waitUntilClosedChan: make(chan struct{}), + localAddr: &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 1337}, + remoteAddr: &net.UDPAddr{IP: net.IPv4(192, 168, 13, 37), Port: 1234}, + context: ctx, } var err error conn, err = newQuicConn(sess, nil) @@ -91,7 +96,7 @@ var _ = Describe("Conn", func() { It("says if it is closed", func() { Consistently(func() bool { return conn.IsClosed() }).Should(BeFalse()) - close(sess.waitUntilClosedChan) + ctxCancel() Eventually(func() bool { return conn.IsClosed() }).Should(BeTrue()) }) From e61b4ddf16110864728d2421b64bb8a5134c59a4 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 8 Aug 2017 11:19:37 +0700 Subject: [PATCH 0310/3965] remove Serve from the quicConn --- p2p/transport/quic/conn.go | 10 -------- p2p/transport/quic/conn_test.go | 44 --------------------------------- 2 files changed, 54 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 3f8ebe91a7..4881f99315 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -63,16 +63,6 @@ func (c *quicConn) OpenStream() (smux.Stream, error) { return &stream{str}, err } -func (c *quicConn) Serve(handler smux.StreamHandler) { - for { // accept loop - s, err := c.AcceptStream() - if err != nil { - return // err always means closed. - } - go handler(s) - } -} - func (c *quicConn) Close() error { return c.sess.Close(nil) } diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 4a38d222a8..774df1d2ce 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -6,7 +6,6 @@ import ( "net" "time" - smux "github.com/libp2p/go-stream-muxer" quic "github.com/lucas-clemente/quic-go" "github.com/lucas-clemente/quic-go/protocol" @@ -133,47 +132,4 @@ var _ = Describe("Conn", func() { Expect(err).To(MatchError(testErr)) }) }) - - Context("serving", func() { - var ( - handler func(smux.Stream) - handlerCalled bool - handlerCalledWith smux.Stream - ) - - BeforeEach(func() { - handlerCalled = false - handlerCalledWith = nil - handler = func(s smux.Stream) { - handlerCalledWith = s - handlerCalled = true - } - }) - - It("calls the handler", func() { - str := &mockStream{id: 5} - sess.streamToAccept = str - var returned bool - go func() { - conn.Serve(handler) - returned = true - }() - Eventually(func() bool { return handlerCalled }).Should(BeTrue()) - Expect(handlerCalledWith.(*stream).Stream).To(Equal(str)) - // make the go-routine return - sess.streamAcceptErr = errors.New("stop test") - }) - - It("returns when accepting a stream errors", func() { - sess.streamAcceptErr = errors.New("accept err") - var returned bool - go func() { - conn.Serve(handler) - returned = true - }() - Eventually(func() bool { return returned }).Should(BeTrue()) - Expect(handlerCalled).To(BeFalse()) - }) - }) - }) From 43db8f270ac548bdd0b39b3739c0fa0fb6e082bf Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 18 Aug 2017 00:54:48 -0700 Subject: [PATCH 0311/3965] add support for the new Reset method. --- p2p/muxer/mplex/multiplex.go | 2 +- p2p/muxer/mplex/multiplex_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 6db90ca62f..77227ccf25 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -4,7 +4,7 @@ import ( "errors" "net" - smux "github.com/jbenet/go-stream-muxer" + smux "github.com/libp2p/go-stream-muxer" mp "github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. ) diff --git a/p2p/muxer/mplex/multiplex_test.go b/p2p/muxer/mplex/multiplex_test.go index e35b3b038d..2b392e231d 100644 --- a/p2p/muxer/mplex/multiplex_test.go +++ b/p2p/muxer/mplex/multiplex_test.go @@ -3,7 +3,7 @@ package peerstream_multiplex import ( "testing" - test "github.com/jbenet/go-stream-muxer/test" + test "github.com/libp2p/go-stream-muxer/test" ) func TestMultiplexTransport(t *testing.T) { From 3576d5ec8ef85d0e04ea78078ac411b7a4013727 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 18 Aug 2017 14:04:32 -0700 Subject: [PATCH 0312/3965] go-stream-muxer has moved to libp2p --- p2p/muxer/yamux/yamux.go | 2 +- p2p/muxer/yamux/yamux_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index a8877c213b..6fbef11bbd 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -5,7 +5,7 @@ import ( "net" "time" - smux "github.com/jbenet/go-stream-muxer" + smux "github.com/libp2p/go-stream-muxer" yamux "github.com/whyrusleeping/yamux" ) diff --git a/p2p/muxer/yamux/yamux_test.go b/p2p/muxer/yamux/yamux_test.go index f95409ec24..fbd0188b8d 100644 --- a/p2p/muxer/yamux/yamux_test.go +++ b/p2p/muxer/yamux/yamux_test.go @@ -3,7 +3,7 @@ package sm_yamux import ( "testing" - test "github.com/jbenet/go-stream-muxer/test" + test "github.com/libp2p/go-stream-muxer/test" ) func TestYamuxTransport(t *testing.T) { From a258c5039fee4159afe9b657decde3bd49af822c Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 19 Jul 2017 11:53:57 +0700 Subject: [PATCH 0313/3965] remove conn.Serve This method was recently removed from the interface. --- p2p/muxer/yamux/yamux.go | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 6fbef11bbd..7a602d0ba6 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -40,18 +40,6 @@ func (c *conn) AcceptStream() (smux.Stream, error) { return s, err } -// Serve starts listening for incoming requests and handles them -// using given StreamHandler -func (c *conn) Serve(handler smux.StreamHandler) { - for { // accept loop - s, err := c.AcceptStream() - if err != nil { - return // err always means closed. - } - go handler(s) - } -} - // Transport is a go-peerstream transport that constructs // yamux-backed connections. type Transport yamux.Config From a920757f7e8d15d4bb999eb70506e49ba9c84ecb Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 19 Aug 2017 14:25:42 +0700 Subject: [PATCH 0314/3965] remove conn.Serve --- p2p/muxer/mplex/multiplex.go | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 77227ccf25..0eebb7262f 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -1,15 +1,12 @@ package peerstream_multiplex import ( - "errors" "net" smux "github.com/libp2p/go-stream-muxer" mp "github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. ) -var ErrUseServe = errors.New("not implemented, use Serve") - type conn struct { *mp.Multiplex } @@ -32,18 +29,6 @@ func (c *conn) AcceptStream() (smux.Stream, error) { return c.Multiplex.Accept() } -// Serve starts listening for incoming requests and handles them -// using given StreamHandler -func (c *conn) Serve(handler smux.StreamHandler) { - for { - s, err := c.AcceptStream() - if err != nil { - return - } - go handler(s) - } -} - // Transport is a go-peerstream transport that constructs // multiplex-backed connections. type Transport struct{} From 969c12de2213743056a975f156ceb0669b045626 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 28 Aug 2017 18:06:33 +0700 Subject: [PATCH 0315/3965] generate a private key and a self-signed certificate for the listener --- p2p/transport/quic/listener.go | 36 ++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 6ef1427378..b0d61c6966 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -1,11 +1,16 @@ package libp2pquic import ( + "crypto/rand" + "crypto/rsa" + "crypto/tls" + "crypto/x509" + "encoding/pem" + "math/big" "net" tpt "github.com/libp2p/go-libp2p-transport" quic "github.com/lucas-clemente/quic-go" - testdata "github.com/lucas-clemente/quic-go/testdata" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) @@ -24,9 +29,11 @@ func newListener(laddr ma.Multiaddr, t tpt.Transport) (*listener, error) { if err != nil { return nil, err } - // we need to provide a certificate here - // use the demo certificate from quic-go - qln, err := quic.ListenAddr(host, testdata.GetTLSConfig(), nil) + tlsConf, err := generateTLSConfig() + if err != nil { + return nil, err + } + qln, err := quic.ListenAddr(host, tlsConf, nil) if err != nil { return nil, err } @@ -61,3 +68,24 @@ func (l *listener) Addr() net.Addr { func (l *listener) Multiaddr() ma.Multiaddr { return l.laddr } + +// Generate a bare-bones TLS config for the server. +// The client doesn't verify the certificate yet. +func generateTLSConfig() (*tls.Config, error) { + key, err := rsa.GenerateKey(rand.Reader, 2048) + if err != nil { + return nil, err + } + template := x509.Certificate{SerialNumber: big.NewInt(1)} + certDER, err := x509.CreateCertificate(rand.Reader, &template, &template, &key.PublicKey, key) + if err != nil { + return nil, err + } + keyPEM := pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(key)}) + certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: certDER}) + tlsCert, err := tls.X509KeyPair(certPEM, keyPEM) + if err != nil { + return nil, err + } + return &tls.Config{Certificates: []tls.Certificate{tlsCert}}, nil +} From 755830ea064e0af3b89deb7caaf0eae182dd23b1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 30 Aug 2017 18:17:44 -0700 Subject: [PATCH 0316/3965] test: connect each peer precisely once. Otherwise, some tests will get confused by duplicate (temporary, we close duplicates quickly) connections. fixes libp2p/go-libp2p#100 --- p2p/net/swarm/swarm_test.go | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 9deb918f3c..e14de3faad 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -102,12 +102,10 @@ func connectSwarms(t *testing.T, ctx context.Context, swarms []*Swarm) { } log.Info("Connecting swarms simultaneously.") - for _, s1 := range swarms { - for _, s2 := range swarms { - if s2.local != s1.local { // don't connect to self. - wg.Add(1) - connect(s1, s2.LocalPeer(), s2.ListenAddresses()[0]) // try the first. - } + for i, s1 := range swarms { + for _, s2 := range swarms[i+1:] { + wg.Add(1) + connect(s1, s2.LocalPeer(), s2.ListenAddresses()[0]) // try the first. } } wg.Wait() From 36115aef16dfef70c6a960fc1e09a11042f23e93 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 1 Sep 2017 12:45:06 +0700 Subject: [PATCH 0317/3965] use the new exported types in the quic-go API This also requires dropping support for Go 1.8. --- p2p/transport/quic/conn_test.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 774df1d2ce..815e269a0e 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -7,21 +7,20 @@ import ( "time" quic "github.com/lucas-clemente/quic-go" - "github.com/lucas-clemente/quic-go/protocol" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) type mockStream struct { - id protocol.StreamID + id quic.StreamID } func (s *mockStream) Close() error { return nil } func (s *mockStream) Reset(error) { return } func (s *mockStream) Read([]byte) (int, error) { return 0, nil } func (s *mockStream) Write([]byte) (int, error) { return 0, nil } -func (s *mockStream) StreamID() protocol.StreamID { return s.id } +func (s *mockStream) StreamID() quic.StreamID { return s.id } func (s *mockStream) SetReadDeadline(time.Time) error { panic("not implemented") } func (s *mockStream) SetWriteDeadline(time.Time) error { panic("not implemented") } func (s *mockStream) SetDeadline(time.Time) error { panic("not implemented") } From efba55492e4ca25ce3ff1597909dde851ae5749d Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 4 Sep 2017 21:54:39 -0700 Subject: [PATCH 0318/3965] fix reading from conn and addresses --- p2p/transport/websocket/conn.go | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/p2p/transport/websocket/conn.go b/p2p/transport/websocket/conn.go index 9a48839109..9f1973474c 100644 --- a/p2p/transport/websocket/conn.go +++ b/p2p/transport/websocket/conn.go @@ -1,6 +1,7 @@ package websocket import ( + "io" "net" "time" @@ -14,15 +15,37 @@ type Conn struct { *ws.Conn DefaultMessageType int done func() + reader io.Reader } -func (c *Conn) Read(b []byte) (n int, err error) { +func (c *Conn) Read(b []byte) (int, error) { + if c.reader == nil { + if err := c.prepNextReader(); err != nil { + return 0, err + } + } + + n, err := c.reader.Read(b) + if err == io.EOF { + if err := c.prepNextReader(); err != nil { + return 0, err + } + if n == 0 { + n, err = c.reader.Read(b) + } + } + + return n, err +} + +func (c *Conn) prepNextReader() error { _, r, err := c.Conn.NextReader() if err != nil { - return 0, err + return err } - return r.Read(b) + c.reader = r + return nil } func (c *Conn) Write(b []byte) (n int, err error) { @@ -42,11 +65,11 @@ func (c *Conn) Close() error { } func (c *Conn) LocalAddr() net.Addr { - return c.Conn.LocalAddr() + return NewAddr(c.Conn.LocalAddr().String()) } func (c *Conn) RemoteAddr() net.Addr { - return c.Conn.RemoteAddr() + return NewAddr(c.Conn.RemoteAddr().String()) } func (c *Conn) SetDeadline(t time.Time) error { From cb1cddaeb6421ded8886e1278d055c21a9ff4e3e Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 4 Sep 2017 22:27:51 -0700 Subject: [PATCH 0319/3965] fix review and tests --- p2p/transport/websocket/conn.go | 38 +++++++++++++++++------ p2p/transport/websocket/websocket_test.go | 11 ++++--- 2 files changed, 35 insertions(+), 14 deletions(-) diff --git a/p2p/transport/websocket/conn.go b/p2p/transport/websocket/conn.go index 9f1973474c..1d8de2a01f 100644 --- a/p2p/transport/websocket/conn.go +++ b/p2p/transport/websocket/conn.go @@ -25,25 +25,42 @@ func (c *Conn) Read(b []byte) (int, error) { } } - n, err := c.reader.Read(b) - if err == io.EOF { - if err := c.prepNextReader(); err != nil { - return 0, err - } - if n == 0 { - n, err = c.reader.Read(b) + for { + n, err := c.reader.Read(b) + switch err { + case io.EOF: + c.reader = nil + + if n > 0 { + return n, nil + } + + if err := c.prepNextReader(); err != nil { + return 0, err + } + + // explicitly looping + default: + return n, err } } - - return n, err } func (c *Conn) prepNextReader() error { - _, r, err := c.Conn.NextReader() + t, r, err := c.Conn.NextReader() if err != nil { + if wserr, ok := err.(*ws.CloseError); ok { + if wserr.Code == 1000 || wserr.Code == 1005 { + return io.EOF + } + } return err } + if t == ws.CloseMessage { + return io.EOF + } + c.reader = r return nil } @@ -61,6 +78,7 @@ func (c *Conn) Close() error { c.done() } + c.Conn.WriteMessage(ws.CloseMessage, nil) return c.Conn.Close() } diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index f56159f200..7aab918576 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -2,7 +2,9 @@ package websocket import ( "bytes" + "io/ioutil" "testing" + "testing/iotest" ma "github.com/multiformats/go-multiaddr" ) @@ -40,13 +42,14 @@ func TestWebsocketListen(t *testing.T) { } defer c.Close() - buf := make([]byte, 32) - n, err := c.Read(buf) + obr := iotest.OneByteReader(c) + + out, err := ioutil.ReadAll(obr) if err != nil { t.Fatal(err) } - if !bytes.Equal(buf[:n], msg) { - t.Fatal("got wrong message", buf[:n], msg) + if !bytes.Equal(out, msg) { + t.Fatal("got wrong message", out, msg) } } From 707639d5429588add14d555c28f2083cc6beaab4 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Mon, 4 Sep 2017 22:53:46 -0700 Subject: [PATCH 0320/3965] gx publish 1.7.6 --- p2p/net/swarm/swarm_dial.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 627cce45ad..d866c57b99 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -285,7 +285,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { } func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (iconn.Conn, error) { - log.Debugf("%s swarm dialing %s %s", s.local, p, remoteAddrs) + log.Debugf("%s swarm dialing %s", s.local, p) ctx, cancel := context.WithCancel(ctx) defer cancel() // cancel work when we exit func From a293989e7c4c82d2228bff8c8470a71f6f8d8df8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Sep 2017 11:48:46 -0700 Subject: [PATCH 0321/3965] Make close thread safe 1. Ensure we only close the connection once. Especially, don't call the done function multiple times and/or concurrently. 2. Call WriteControl instead of WriteMessage in Close. WriteControl is thread-safe, WriteMessage isn't. Also, this sets a 100ms deadline on gracefully closing connections. --- p2p/transport/websocket/conn.go | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/p2p/transport/websocket/conn.go b/p2p/transport/websocket/conn.go index 1d8de2a01f..0ab71ce220 100644 --- a/p2p/transport/websocket/conn.go +++ b/p2p/transport/websocket/conn.go @@ -3,11 +3,16 @@ package websocket import ( "io" "net" + "sync" "time" ws "github.com/gorilla/websocket" ) +// GracefulCloseTimeout is the time to wait trying to gracefully close a +// connection before simply cutting it. +var GracefulCloseTimeout = 100 * time.Millisecond + var _ net.Conn = (*Conn)(nil) // Conn implements net.Conn interface for gorilla/websocket. @@ -16,6 +21,7 @@ type Conn struct { DefaultMessageType int done func() reader io.Reader + closeOnce sync.Once } func (c *Conn) Read(b []byte) (int, error) { @@ -73,13 +79,22 @@ func (c *Conn) Write(b []byte) (n int, err error) { return len(b), nil } +// Close closes the connection. Only the first call to Close will receive the +// close error, subsequent and concurrent calls will return nil. +// This method is thread-safe. func (c *Conn) Close() error { - if c.done != nil { - c.done() - } + var err error = nil + c.closeOnce.Do(func() { + if c.done != nil { + c.done() + // Be nice to GC + c.done = nil + } - c.Conn.WriteMessage(ws.CloseMessage, nil) - return c.Conn.Close() + c.Conn.WriteControl(ws.CloseMessage, nil, time.Now().Add(GracefulCloseTimeout)) + err = c.Conn.Close() + }) + return err } func (c *Conn) LocalAddr() net.Addr { From b863f44dabd3c8a230a6c4f0b57cfeee4576abc8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Sep 2017 12:12:21 -0700 Subject: [PATCH 0322/3965] test concurrent connection closing --- p2p/transport/websocket/websocket_test.go | 38 +++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index 7aab918576..d2e166a685 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -53,3 +53,41 @@ func TestWebsocketListen(t *testing.T) { t.Fatal("got wrong message", out, msg) } } + +func TestConcurrentClose(t *testing.T) { + zero, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0/ws") + if err != nil { + t.Fatal(err) + } + + tpt := &WebsocketTransport{} + l, err := tpt.Listen(zero) + if err != nil { + t.Fatal(err) + } + defer l.Close() + + msg := []byte("HELLO WORLD") + + go func() { + d, _ := tpt.Dialer(nil) + for i := 0; i < 100; i++ { + c, err := d.Dial(l.Multiaddr()) + if err != nil { + t.Error(err) + return + } + + go c.Write(msg) + go c.Close() + } + }() + + for i := 0; i < 100; i++ { + c, err := l.Accept() + if err != nil { + t.Fatal(err) + } + c.Close() + } +} From a88b185db2f234ec5743b9ec9810437fa856e669 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Sep 2017 12:25:29 -0700 Subject: [PATCH 0323/3965] Add write-zero test. Just to be thorough. --- p2p/transport/websocket/websocket_test.go | 52 +++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index d2e166a685..61bfa14230 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -2,6 +2,7 @@ package websocket import ( "bytes" + "io" "io/ioutil" "testing" "testing/iotest" @@ -91,3 +92,54 @@ func TestConcurrentClose(t *testing.T) { c.Close() } } + +func TestWriteZero(t *testing.T) { + zero, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0/ws") + if err != nil { + t.Fatal(err) + } + + tpt := &WebsocketTransport{} + l, err := tpt.Listen(zero) + if err != nil { + t.Fatal(err) + } + defer l.Close() + + msg := []byte(nil) + + go func() { + d, _ := tpt.Dialer(nil) + c, err := d.Dial(l.Multiaddr()) + defer c.Close() + if err != nil { + t.Error(err) + return + } + + for i := 0; i < 100; i++ { + n, err := c.Write(msg) + if n != 0 { + t.Errorf("expected to write 0 bytes, wrote %d", n) + } + if err != nil { + t.Error(err) + return + } + } + }() + + c, err := l.Accept() + defer c.Close() + if err != nil { + t.Fatal(err) + } + buf := make([]byte, 100) + n, err := c.Read(buf) + if n != 0 { + t.Errorf("read %d bytes, expected 0", n) + } + if err != io.EOF { + t.Errorf("expected EOF, got err: %s", err) + } +} From da59505d57802250e79733282e7a296c8f81db13 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Sep 2017 12:27:45 -0700 Subject: [PATCH 0324/3965] don't explicitly nil err (make @whyrusleeping happy) --- p2p/transport/websocket/conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/conn.go b/p2p/transport/websocket/conn.go index 0ab71ce220..168497bf39 100644 --- a/p2p/transport/websocket/conn.go +++ b/p2p/transport/websocket/conn.go @@ -83,7 +83,7 @@ func (c *Conn) Write(b []byte) (n int, err error) { // close error, subsequent and concurrent calls will return nil. // This method is thread-safe. func (c *Conn) Close() error { - var err error = nil + var err error c.closeOnce.Do(func() { if c.done != nil { c.done() From f4559306534f037b0fa71200b678e13b7cebdf37 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Sep 2017 14:56:26 -0700 Subject: [PATCH 0325/3965] fix requests from other origins We don't care about origins and *want* any code running on any origin to be able to connect to any peer. fixes ipfs/go-ipfs#4202 --- p2p/transport/websocket/websocket.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 55ea54d994..6a2c6722e5 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -33,7 +33,12 @@ var WsCodec = &manet.NetCodec{ } // Default gorilla upgrader -var upgrader = ws.Upgrader{} +var upgrader = ws.Upgrader{ + // Allow requests from *all* origins. + CheckOrigin: func(r *http.Request) bool { + return true + }, +} func init() { err := ma.AddProtocol(WsProtocol) From aaf9fcb7135baa542baf6218858ba06d399fc335 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 9 Apr 2017 02:46:53 +0700 Subject: [PATCH 0326/3965] be explicit about the interfaces implemented --- p2p/transport/tcp/tcp.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index a30dcbb7b0..e4534b7544 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -25,6 +25,8 @@ type TcpTransport struct { listeners map[string]tpt.Listener } +var _ tpt.Transport = &TcpTransport{} + // NewTCPTransport creates a tcp transport object that tracks dialers and listeners // created. It represents an entire tcp stack (though it might not necessarily be) func NewTCPTransport() *TcpTransport { @@ -135,6 +137,8 @@ type tcpDialer struct { transport tpt.Transport } +var _ tpt.Dialer = &tcpDialer{} + func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReuse bool) (*tcpDialer, error) { // get the local net.Addr manually la, err := manet.ToNetAddr(laddr) @@ -238,6 +242,8 @@ type tcpListener struct { transport tpt.Transport } +var _ tpt.Listener = &tcpListener{} + func (d *tcpListener) Accept() (tpt.Conn, error) { c, err := d.list.Accept() if err != nil { From d29927f2f7700141df77a8ebd0be02ca507fb51d Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 9 Apr 2017 02:49:05 +0700 Subject: [PATCH 0327/3965] replace tpt.ConnWrap by a tcpConn The tpt.ConnWrap is being removed from go-libp2p-transport --- p2p/transport/tcp/tcp.go | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index e4534b7544..066fffe3df 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -199,9 +199,9 @@ func (d *tcpDialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Co return nil, err } - return &tpt.ConnWrap{ + return &tcpConn{ Conn: c, - Tpt: d.transport, + t: d.transport, }, nil } @@ -250,9 +250,9 @@ func (d *tcpListener) Accept() (tpt.Conn, error) { return nil, err } - return &tpt.ConnWrap{ + return &tcpConn{ Conn: c, - Tpt: d.transport, + t: d.transport, }, nil } @@ -271,3 +271,15 @@ func (t *tcpListener) NetListener() net.Listener { func (d *tcpListener) Close() error { return d.list.Close() } + +type tcpConn struct { + manet.Conn + t tpt.Transport +} + +var _ tpt.Conn = &tcpConn{} +var _ tpt.SingleStreamConn = &tcpConn{} + +func (c *tcpConn) Transport() tpt.Transport { + return c.t +} From d7cda66e0f5e84ef598e2283331c2991f35f6c19 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 6 Sep 2017 12:05:11 +0200 Subject: [PATCH 0328/3965] remove the timeoutOpt Timeout should only be set by using contexts. --- p2p/transport/tcp/tcp.go | 3 --- 1 file changed, 3 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 066fffe3df..991981ca9c 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -5,7 +5,6 @@ import ( "fmt" "net" "sync" - "time" logging "github.com/ipfs/go-log" reuseport "github.com/jbenet/go-reuseport" @@ -56,8 +55,6 @@ func (t *TcpTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dial var doReuse bool for _, o := range opts { switch o := o.(type) { - case tpt.TimeoutOpt: - base.Timeout = time.Duration(o) case tpt.ReuseportOpt: doReuse = bool(o) default: From 21fd167df35acbda11a41e4458c3e6a4b558729c Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 6 Sep 2017 12:05:25 +0200 Subject: [PATCH 0329/3965] use the renamed transport interfaces --- p2p/transport/tcp/tcp.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 991981ca9c..87aef6e92c 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -275,7 +275,7 @@ type tcpConn struct { } var _ tpt.Conn = &tcpConn{} -var _ tpt.SingleStreamConn = &tcpConn{} +var _ tpt.DuplexConn = &tcpConn{} func (c *tcpConn) Transport() tpt.Transport { return c.t From 443d2a836a3965b6607810fdcc1fd546658cd9c9 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 6 Sep 2017 12:56:04 +0200 Subject: [PATCH 0330/3965] use the renamed transport interfaces --- p2p/transport/quic/conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 4881f99315..1cc72686ed 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -25,7 +25,7 @@ type quicConn struct { } var _ tpt.Conn = &quicConn{} -var _ tpt.MultiStreamConn = &quicConn{} +var _ tpt.MultiplexConn = &quicConn{} func newQuicConn(sess quic.Session, t tpt.Transport) (*quicConn, error) { // analogues to manet.WrapNetConn From 10de0c316cbefb08d1f80ff8ee0b9dac214eeead Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 12 Sep 2017 17:38:02 -0700 Subject: [PATCH 0331/3965] Update stream muxer. * Add in the reset method and use it instead of Close. Close now only closes one side of the connection. --- p2p/net/swarm/swarm.go | 2 +- p2p/net/swarm/swarm_listen.go | 2 +- p2p/net/swarm/swarm_notif_test.go | 4 ++-- p2p/net/swarm/swarm_stream.go | 5 +++++ 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index a282f7429b..8794048b5c 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -12,7 +12,6 @@ import ( "time" logging "github.com/ipfs/go-log" - pst "github.com/jbenet/go-stream-muxer" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" addrutil "github.com/libp2p/go-addr-util" @@ -27,6 +26,7 @@ import ( transport "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" ps "github.com/libp2p/go-peerstream" + pst "github.com/libp2p/go-stream-muxer" tcpt "github.com/libp2p/go-tcp-transport" ws "github.com/libp2p/go-ws-transport" ma "github.com/multiformats/go-multiaddr" diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index b04c47f11a..98d3549a4d 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -145,7 +145,7 @@ func (s *Swarm) addConnListener(list iconn.Listener) error { // connHandler is called by the StreamSwarm whenever a new connection is added // here we configure it slightly. Note that this is sequential, so if anything // will take a while do it in a goroutine. -// See https://godoc.org/github.com/jbenet/go-peerstream for more information +// See https://godoc.org/github.com/libp2p/go-peerstream for more information func (s *Swarm) connHandler(c *ps.Conn) *Conn { ctx := context.Background() // this context is for running the handshake, which -- when receiveing connections diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 8ebcccb973..8a574aa070 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -125,7 +125,7 @@ func TestNotifications(t *testing.T) { for _, s := range swarms { s.SetStreamHandler(func(s inet.Stream) { streams <- s - s.Close() + s.Reset() }) } @@ -139,7 +139,7 @@ func TestNotifications(t *testing.T) { t.Error(err) } else { st1.Write([]byte("hello")) - st1.Close() + st1.Reset() testOCStream(notifiees[i], st1) st2 := <-streams testOCStream(n2, st2) diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index adadca78ec..7e83dde1d8 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -43,6 +43,11 @@ func (s *Stream) Close() error { return s.Stream().Close() } +// Reset resets the stream, closing both ends. +func (s *Stream) Reset() error { + return s.Stream().Reset() +} + func (s *Stream) Protocol() protocol.ID { return (*ps.Stream)(s).Protocol() } From b36e3738981fc4a7b928ed40506401a31e3a04f0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 12 Sep 2017 23:03:25 -0700 Subject: [PATCH 0332/3965] update go-stream-muxer Use the new Reset method. --- .../internal/circuitv1-deprecated/listen.go | 4 +-- .../internal/circuitv1-deprecated/relay.go | 31 +++++++++++-------- 2 files changed, 19 insertions(+), 16 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index dfec479ab3..9eac888913 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -30,9 +30,7 @@ func (l *RelayListener) Accept() (tpt.Conn, error) { err := l.Relay().writeResponse(c.Stream, pb.CircuitRelay_SUCCESS) if err != nil { log.Debugf("error writing relay response: %s", err.Error()) - // this won't prevent the other side from continuing to write - // TODO fully close the stream when Reset is implemented - c.Stream.Close() + c.Stream.Reset() return nil, err } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 4eb3c16a83..5e33f70d26 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -104,7 +104,7 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore err = wr.WriteMsg(&msg) if err != nil { - s.Close() + s.Reset() return nil, err } @@ -112,17 +112,17 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore err = rd.ReadMsg(&msg) if err != nil { - s.Close() + s.Reset() return nil, err } if msg.GetType() != pb.CircuitRelay_STATUS { - s.Close() + s.Reset() return nil, fmt.Errorf("unexpected relay response; not a status message (%d)", msg.GetType()) } if msg.GetCode() != pb.CircuitRelay_SUCCESS { - s.Close() + s.Reset() return nil, RelayError{msg.GetCode()} } @@ -135,8 +135,6 @@ func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { return false, err } - defer s.Close() - rd := newDelimitedReader(s, maxMessageSize) wr := newDelimitedWriter(s) @@ -146,12 +144,15 @@ func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { err = wr.WriteMsg(&msg) if err != nil { + s.Reset() return false, err } msg.Reset() err = rd.ReadMsg(&msg) + s.Close() + if err != nil { return false, err } @@ -248,7 +249,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { err = wr.WriteMsg(msg) if err != nil { log.Debugf("error writing stop handshake: %s", err.Error()) - bs.Close() + bs.Reset() r.handleError(s, pb.CircuitRelay_HOP_CANT_OPEN_DST_STREAM) return } @@ -258,21 +259,21 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { err = rd.ReadMsg(msg) if err != nil { log.Debugf("error reading stop response: %s", err.Error()) - bs.Close() + bs.Reset() r.handleError(s, pb.CircuitRelay_HOP_CANT_OPEN_DST_STREAM) return } if msg.GetType() != pb.CircuitRelay_STATUS { log.Debugf("unexpected relay stop response: not a status message (%d)", msg.GetType()) - bs.Close() + bs.Reset() r.handleError(s, pb.CircuitRelay_HOP_CANT_OPEN_DST_STREAM) return } if msg.GetCode() != pb.CircuitRelay_SUCCESS { log.Debugf("relay stop failure: %d", msg.GetCode()) - bs.Close() + bs.Reset() r.handleError(s, msg.GetCode()) return } @@ -280,14 +281,16 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { err = r.writeResponse(s, pb.CircuitRelay_SUCCESS) if err != nil { log.Debugf("error writing relay response: %s", err.Error()) - bs.Close() - s.Close() + bs.Reset() + s.Reset() return } // relay connection log.Infof("relaying connection between %s and %s", src.ID.Pretty(), dst.ID.Pretty()) + // Don't reset streams after finishing or the other side will get an + // error, not an EOF. go func() { count, err := io.Copy(s, bs) if err != io.EOF && err != nil { @@ -353,9 +356,11 @@ func (r *Relay) handleError(s inet.Stream, code pb.CircuitRelay_Status) { log.Warningf("relay error: %s (%d)", pb.CircuitRelay_Status_name[int32(code)], code) err := r.writeResponse(s, code) if err != nil { + s.Reset() log.Debugf("error writing relay response: %s", err.Error()) + } else { + s.Close() } - s.Close() } func (r *Relay) writeResponse(s inet.Stream, code pb.CircuitRelay_Status) error { From 5253314fb208aff9788d617337bc3f3803831fe1 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 13 Sep 2017 11:40:11 +0300 Subject: [PATCH 0333/3965] handleCanHop: reset stream on errors --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 5e33f70d26..edeafc9a84 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -346,10 +346,11 @@ func (r *Relay) handleCanHop(s inet.Stream, msg *pb.CircuitRelay) { } if err != nil { + s.Reset() log.Debugf("error writing relay response: %s", err.Error()) + } else { + s.Close() } - - s.Close() } func (r *Relay) handleError(s inet.Stream, code pb.CircuitRelay_Status) { From 93b0a9abf653fcbc0e36439416232653ff366090 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 13 Sep 2017 17:34:13 -0700 Subject: [PATCH 0334/3965] don't try to close a connection while holding a lock that blocks the notifiees --- p2p/net/connmgr/connmgr.go | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index cbad648130..d06440d014 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -2,6 +2,7 @@ package connmgr import ( "context" + "io" "sort" "sync" "time" @@ -64,18 +65,24 @@ type TagInfo struct { } func (cm *connManager) TrimOpenConns(ctx context.Context) { + for _, c := range cm.getConnsToClose(ctx) { + c.Close() + } +} + +func (cm *connManager) getConnsToClose(ctx context.Context) []io.Closer { cm.lk.Lock() defer cm.lk.Unlock() defer log.EventBegin(ctx, "connCleanup").Done() if cm.lowWater == 0 || cm.highWater == 0 { // disabled - return + return nil } cm.lastTrim = time.Now() if len(cm.peers) < cm.lowWater { log.Info("open connection count below limit") - return + return nil } var infos []*peerInfo @@ -91,6 +98,8 @@ func (cm *connManager) TrimOpenConns(ctx context.Context) { close_count := len(infos) - cm.lowWater toclose := infos[:close_count] + var closed []io.Closer + for _, inf := range toclose { if time.Since(inf.firstSeen) < cm.gracePeriod { continue @@ -100,13 +109,12 @@ func (cm *connManager) TrimOpenConns(ctx context.Context) { for c, _ := range inf.conns { log.Info("closing conn: ", c.RemotePeer()) log.Event(ctx, "closeConn", c.RemotePeer()) - c.Close() + // TODO: probably don't want to always do this in a goroutine + closed = append(closed, c) } } - if len(cm.peers) > cm.highWater { - log.Error("still over high water mark after trimming connections") - } + return closed } func (cm *connManager) GetTagInfo(p peer.ID) *TagInfo { From 5be8da502992ad261bfe975fa001105314c0da7c Mon Sep 17 00:00:00 2001 From: Jeromy Date: Wed, 20 Sep 2017 15:04:07 -0700 Subject: [PATCH 0335/3965] use optimized 'HaveConnToPeer' checks --- p2p/net/swarm/swarm.go | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 8794048b5c..dac2624d96 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -273,7 +273,7 @@ func (s *Swarm) SetStreamHandler(handler inet.StreamHandler) { // NewStreamWithPeer creates a new stream on any available connection to p func (s *Swarm) NewStreamWithPeer(ctx context.Context, p peer.ID) (*Stream, error) { // if we have no connections, try connecting. - if len(s.ConnectionsToPeer(p)) == 0 { + if !s.HaveConnsToPeer(p) { log.Debug("Swarm: NewStreamWithPeer no connections. Attempting to connect...") if _, err := s.Dial(ctx, p); err != nil { return nil, err @@ -288,16 +288,11 @@ func (s *Swarm) NewStreamWithPeer(ctx context.Context, p peer.ID) (*Stream, erro // ConnectionsToPeer returns all the live connections to p func (s *Swarm) ConnectionsToPeer(p peer.ID) []*Conn { - return wrapConns(ps.ConnsWithGroup(p, s.swarm.Conns())) + return wrapConns(s.swarm.ConnsWithGroup(p)) } func (s *Swarm) HaveConnsToPeer(p peer.ID) bool { - for _, c := range s.swarm.Conns() { - if c.InGroup(p) { - return true - } - } - return false + return len(s.swarm.ConnsWithGroup(p)) > 0 } // Connections returns a slice of all connections. From 8a2f0cbb0f00ced21af084d8f6257fc3e81c73da Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 6 Oct 2017 03:51:51 -0700 Subject: [PATCH 0336/3965] gx publish 0.2.1 --- p2p/host/blank/blank.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go index ec4fe010ae..1c896b54b1 100644 --- a/p2p/host/blank/blank.go +++ b/p2p/host/blank/blank.go @@ -5,8 +5,8 @@ import ( "io" logging "github.com/ipfs/go-log" - connmgr "github.com/libp2p/go-libp2p-connmgr" host "github.com/libp2p/go-libp2p-host" + ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" @@ -21,13 +21,13 @@ var log = logging.Logger("blankhost") type BlankHost struct { n inet.Network mux *mstream.MultistreamMuxer - cmgr connmgr.ConnManager + cmgr ifconnmgr.ConnManager } func NewBlankHost(n inet.Network) *BlankHost { bh := &BlankHost{ n: n, - cmgr: connmgr.NewConnManager(0, 0, 0), + cmgr: &ifconnmgr.NullConnMgr{}, mux: mstream.NewMultistreamMuxer(), } @@ -143,6 +143,6 @@ func (bh *BlankHost) Network() inet.Network { return bh.n } -func (bh *BlankHost) ConnManager() connmgr.ConnManager { +func (bh *BlankHost) ConnManager() ifconnmgr.ConnManager { return bh.cmgr } From 1fd3dd4bbfa883c03640b085251b27f9a48856dd Mon Sep 17 00:00:00 2001 From: Jeromy Date: Fri, 6 Oct 2017 09:01:08 -0700 Subject: [PATCH 0337/3965] extract interface --- p2p/net/connmgr/connmgr.go | 24 ++++++------------------ 1 file changed, 6 insertions(+), 18 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index d06440d014..610e3bec1d 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -8,6 +8,7 @@ import ( "time" logging "github.com/ipfs/go-log" + ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" @@ -15,14 +16,6 @@ import ( var log = logging.Logger("connmgr") -type ConnManager interface { - TagPeer(peer.ID, string, int) - UntagPeer(peer.ID, string) - GetTagInfo(peer.ID) *TagInfo - TrimOpenConns(context.Context) - Notifee() inet.Notifiee -} - type connManager struct { highWater int lowWater int @@ -37,9 +30,11 @@ type connManager struct { lastTrim time.Time } +var _ ifconnmgr.ConnManager = (*connManager)(nil) + var DefaultGracePeriod = time.Second * 10 -func NewConnManager(low, hi int, grace time.Duration) ConnManager { +func NewConnManager(low, hi int, grace time.Duration) ifconnmgr.ConnManager { return &connManager{ highWater: hi, lowWater: low, @@ -57,13 +52,6 @@ type peerInfo struct { firstSeen time.Time } -type TagInfo struct { - FirstSeen time.Time - Value int - Tags map[string]int - Conns map[string]time.Time -} - func (cm *connManager) TrimOpenConns(ctx context.Context) { for _, c := range cm.getConnsToClose(ctx) { c.Close() @@ -117,7 +105,7 @@ func (cm *connManager) getConnsToClose(ctx context.Context) []io.Closer { return closed } -func (cm *connManager) GetTagInfo(p peer.ID) *TagInfo { +func (cm *connManager) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { cm.lk.Lock() defer cm.lk.Unlock() @@ -126,7 +114,7 @@ func (cm *connManager) GetTagInfo(p peer.ID) *TagInfo { return nil } - out := &TagInfo{ + out := &ifconnmgr.TagInfo{ FirstSeen: pi.firstSeen, Value: pi.value, Tags: make(map[string]int), From b0909a3412cc01d77d7e24b58d160ce7148c3716 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sun, 8 Oct 2017 10:15:03 -0700 Subject: [PATCH 0338/3965] tag 'can-hop' peers to aid in connection closing logic --- p2p/protocol/internal/circuitv1-deprecated/notify.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/notify.go b/p2p/protocol/internal/circuitv1-deprecated/notify.go index f3ea08cec3..dec311671a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/notify.go +++ b/p2p/protocol/internal/circuitv1-deprecated/notify.go @@ -47,6 +47,7 @@ func (n *RelayNotifiee) Connected(s inet.Network, c inet.Conn) { n.mx.Lock() n.relays[id] = struct{}{} n.mx.Unlock() + n.host.ConnManager().TagPeer(id, "relay-hop", 2) } }(c.RemotePeer()) } From ef6047635c8d3af7b1e9d60b124d7880a34cacc4 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 16 Oct 2017 10:30:52 -0700 Subject: [PATCH 0339/3965] faster, cleaner connection closing 1. Don't ask the system for the time when checking if we should close each connection (potentially thousands of systemcalls!). 2. Log from outside the lock. 3. Log the event around the entire connection closing operation. 4. Preallocate the slice holding the connections to be closed. --- p2p/net/connmgr/connmgr.go | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 610e3bec1d..4305424400 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -2,7 +2,6 @@ package connmgr import ( "context" - "io" "sort" "sync" "time" @@ -53,20 +52,23 @@ type peerInfo struct { } func (cm *connManager) TrimOpenConns(ctx context.Context) { + defer log.EventBegin(ctx, "connCleanup").Done() for _, c := range cm.getConnsToClose(ctx) { + log.Info("closing conn: ", c.RemotePeer()) + log.Event(ctx, "closeConn", c.RemotePeer()) c.Close() } } -func (cm *connManager) getConnsToClose(ctx context.Context) []io.Closer { +func (cm *connManager) getConnsToClose(ctx context.Context) []inet.Conn { cm.lk.Lock() defer cm.lk.Unlock() - defer log.EventBegin(ctx, "connCleanup").Done() if cm.lowWater == 0 || cm.highWater == 0 { // disabled return nil } - cm.lastTrim = time.Now() + now := time.Now() + cm.lastTrim = now if len(cm.peers) < cm.lowWater { log.Info("open connection count below limit") @@ -83,20 +85,22 @@ func (cm *connManager) getConnsToClose(ctx context.Context) []io.Closer { return infos[i].value < infos[j].value }) - close_count := len(infos) - cm.lowWater - toclose := infos[:close_count] + closeCount := len(infos) - cm.lowWater + toclose := infos[:closeCount] - var closed []io.Closer + // 2x number of peers we're disconnecting from because we may have more + // than one connection per peer. Slightly over allocating isn't an issue + // as this is a very short-lived array. + closed := make([]inet.Conn, 0, len(toclose)*2) for _, inf := range toclose { - if time.Since(inf.firstSeen) < cm.gracePeriod { + // TODO: should we be using firstSeen or the time associated with the connection itself? + if inf.firstSeen.Add(cm.gracePeriod).After(now) { continue } // TODO: if a peer has more than one connection, maybe only close one? - for c, _ := range inf.conns { - log.Info("closing conn: ", c.RemotePeer()) - log.Event(ctx, "closeConn", c.RemotePeer()) + for c := range inf.conns { // TODO: probably don't want to always do this in a goroutine closed = append(closed, c) } From 25616aa1782c1147da53cada8d31ae9dbffd032f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 16 Oct 2017 11:41:15 -0700 Subject: [PATCH 0340/3965] remove the DefaultGracePeriod It's unused. --- p2p/net/connmgr/connmgr.go | 2 -- 1 file changed, 2 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 4305424400..cd712b1e3d 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -31,8 +31,6 @@ type connManager struct { var _ ifconnmgr.ConnManager = (*connManager)(nil) -var DefaultGracePeriod = time.Second * 10 - func NewConnManager(low, hi int, grace time.Duration) ifconnmgr.ConnManager { return &connManager{ highWater: hi, From bedd5ad2bbcbe39f5c2f7b5b616e01e0c1650db6 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Wed, 18 Oct 2017 19:21:36 +0200 Subject: [PATCH 0341/3965] Import and adapt code from go-libp2p-http --- p2p/net/gostream/addr.go | 14 ++++ p2p/net/gostream/conn.go | 85 +++++++++++++++++++++ p2p/net/gostream/coverage.out | 37 ++++++++++ p2p/net/gostream/gostream.go | 14 ++++ p2p/net/gostream/gostream_test.go | 118 ++++++++++++++++++++++++++++++ p2p/net/gostream/listener.go | 68 +++++++++++++++++ 6 files changed, 336 insertions(+) create mode 100644 p2p/net/gostream/addr.go create mode 100644 p2p/net/gostream/conn.go create mode 100644 p2p/net/gostream/coverage.out create mode 100644 p2p/net/gostream/gostream.go create mode 100644 p2p/net/gostream/gostream_test.go create mode 100644 p2p/net/gostream/listener.go diff --git a/p2p/net/gostream/addr.go b/p2p/net/gostream/addr.go new file mode 100644 index 0000000000..3e7f699387 --- /dev/null +++ b/p2p/net/gostream/addr.go @@ -0,0 +1,14 @@ +package gostream + +import peer "github.com/libp2p/go-libp2p-peer" + +// Addr implements net.Addr and holds a libp2p peer ID. +type Addr struct{ id peer.ID } + +// Network returns the name of the network that this address belongs to +// (libp2p). +func (a *Addr) Network() string { return "libp2p" } + +// String returns the peer ID of this address in string form +// (B58-encoded). +func (a *Addr) String() string { return a.id.Pretty() } diff --git a/p2p/net/gostream/conn.go b/p2p/net/gostream/conn.go new file mode 100644 index 0000000000..ce5d7feee7 --- /dev/null +++ b/p2p/net/gostream/conn.go @@ -0,0 +1,85 @@ +package gostream + +import ( + "context" + "net" + "time" + + host "github.com/libp2p/go-libp2p-host" + pnet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + protocol "github.com/libp2p/go-libp2p-protocol" +) + +// Conn is an implementation of net.Conn which wraps +// libp2p streams. +type Conn struct { + s pnet.Stream +} + +// NewConn creates a Conn given a libp2p stream +func NewConn(s pnet.Stream) net.Conn { + return &Conn{s} +} + +// Read reads data from the connection. +func (c *Conn) Read(b []byte) (n int, err error) { + return c.s.Read(b) +} + +// Write writes data to the connection. +func (c *Conn) Write(b []byte) (n int, err error) { + return c.s.Write(b) +} + +// Close closes the connection. +// Any blocked Read or Write operations will be unblocked and return errors. +func (c *Conn) Close() error { + return c.s.Close() +} + +// LocalAddr returns the local network address. +func (c *Conn) LocalAddr() net.Addr { + return &Addr{c.s.Conn().LocalPeer()} +} + +// RemoteAddr returns the remote network address. +func (c *Conn) RemoteAddr() net.Addr { + return &Addr{c.s.Conn().RemotePeer()} +} + +// SetDeadline sets the read and write deadlines associated +// with the connection. It is equivalent to calling both +// SetReadDeadline and SetWriteDeadline. +// See https://golang.org/pkg/net/#Conn for more details. +func (c *Conn) SetDeadline(t time.Time) error { + return c.s.SetDeadline(t) +} + +// SetReadDeadline sets the deadline for future Read calls. +// A zero value for t means Read will not time out. +func (c *Conn) SetReadDeadline(t time.Time) error { + return c.s.SetReadDeadline(t) +} + +// SetWriteDeadline sets the deadline for future Write calls. +// Even if write times out, it may return n > 0, indicating that +// some of the data was successfully written. +// A zero value for t means Write will not time out. +func (c *Conn) SetWriteDeadline(t time.Time) error { + return c.s.SetWriteDeadline(t) +} + +// Dial opens a stream to the destination address +// (which should parseable to a peer ID) using the given +// host and returns it. +func Dial(h host.Host, pid peer.ID, tag protocol.ID) (net.Conn, error) { + ctx, cancel := context.WithTimeout(context.Background(), time.Second*30) + defer cancel() + + s, err := h.NewStream(ctx, pid, tag) + if err != nil { + return nil, err + } + return NewConn(s), nil +} diff --git a/p2p/net/gostream/coverage.out b/p2p/net/gostream/coverage.out new file mode 100644 index 0000000000..77b128ca5a --- /dev/null +++ b/p2p/net/gostream/coverage.out @@ -0,0 +1,37 @@ +mode: count +github.com/hsanjuan/go-libp2p-http/addr.go:8.33,8.52 1 0 +github.com/hsanjuan/go-libp2p-http/addr.go:9.33,9.57 1 1 +github.com/hsanjuan/go-libp2p-http/conn.go:17.38,19.2 1 2 +github.com/hsanjuan/go-libp2p-http/conn.go:22.50,24.2 1 3 +github.com/hsanjuan/go-libp2p-http/conn.go:27.51,29.2 1 2 +github.com/hsanjuan/go-libp2p-http/conn.go:33.30,35.2 1 2 +github.com/hsanjuan/go-libp2p-http/conn.go:38.37,40.2 1 0 +github.com/hsanjuan/go-libp2p-http/conn.go:43.38,45.2 1 1 +github.com/hsanjuan/go-libp2p-http/conn.go:51.47,53.2 1 0 +github.com/hsanjuan/go-libp2p-http/conn.go:57.51,59.2 1 0 +github.com/hsanjuan/go-libp2p-http/conn.go:65.52,67.2 1 0 +github.com/hsanjuan/go-libp2p-http/listener.go:25.47,26.9 1 2 +github.com/hsanjuan/go-libp2p-http/listener.go:27.2,28.25 1 1 +github.com/hsanjuan/go-libp2p-http/listener.go:29.2,30.26 1 1 +github.com/hsanjuan/go-libp2p-http/listener.go:36.34,40.2 3 2 +github.com/hsanjuan/go-libp2p-http/listener.go:43.36,45.2 1 1 +github.com/hsanjuan/go-libp2p-http/listener.go:49.48,59.54 3 1 +github.com/hsanjuan/go-libp2p-http/listener.go:67.2,67.15 1 1 +github.com/hsanjuan/go-libp2p-http/listener.go:59.54,60.10 1 1 +github.com/hsanjuan/go-libp2p-http/listener.go:61.3,61.24 0 1 +github.com/hsanjuan/go-libp2p-http/listener.go:62.3,63.13 1 0 +github.com/hsanjuan/go-libp2p-http/listener.go:73.58,78.16 4 1 +github.com/hsanjuan/go-libp2p-http/listener.go:82.2,83.16 2 1 +github.com/hsanjuan/go-libp2p-http/listener.go:86.2,86.24 1 1 +github.com/hsanjuan/go-libp2p-http/listener.go:78.16,80.3 1 0 +github.com/hsanjuan/go-libp2p-http/listener.go:83.16,85.3 1 0 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:16.76,17.19 1 1 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:21.2,22.16 2 1 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:26.2,27.16 2 1 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:30.2,33.16 3 1 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:37.2,38.37 2 1 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:17.19,19.3 1 0 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:22.16,24.3 1 0 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:27.16,29.3 1 0 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:33.16,35.3 1 0 +github.com/hsanjuan/go-libp2p-http/roundtripper.go:51.46,53.2 1 1 diff --git a/p2p/net/gostream/gostream.go b/p2p/net/gostream/gostream.go new file mode 100644 index 0000000000..ae292c59ec --- /dev/null +++ b/p2p/net/gostream/gostream.go @@ -0,0 +1,14 @@ +// Package `gostream` allows to replace the standard net stack in Go +// with [LibP2P](https://github.com/libp2p/libp2p) streams. +// +// Given a libp2p.Host, `gostream` provides Dial() and Listen() methods which +// return implementations of net.Conn and net.Listener. +// +// Instead of the regular "host:port" addressing, `gostream` uses a Peer ID, +// and rather than a raw TCP connection, gostream will use libp2p's net.Stream. +// This means your connections will take advantage of LibP2P's multi-routes, +// NAT transversal and stream multiplexing. +// +// Note that LibP2P hosts cannot dial to themselves, so there is no possibility +// of using the same Host as server and as client. +package gostream diff --git a/p2p/net/gostream/gostream_test.go b/p2p/net/gostream/gostream_test.go new file mode 100644 index 0000000000..5f7634d95b --- /dev/null +++ b/p2p/net/gostream/gostream_test.go @@ -0,0 +1,118 @@ +package gostream + +import ( + "bufio" + "context" + "io/ioutil" + "testing" + + crypto "github.com/libp2p/go-libp2p-crypto" + host "github.com/libp2p/go-libp2p-host" + peer "github.com/libp2p/go-libp2p-peer" + peerstore "github.com/libp2p/go-libp2p-peerstore" + protocol "github.com/libp2p/go-libp2p-protocol" + swarm "github.com/libp2p/go-libp2p-swarm" + bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + multiaddr "github.com/multiformats/go-multiaddr" +) + +// newHost illustrates how to build a libp2p host with secio using +// a randomly generated key-pair +func newHost(t *testing.T, listen multiaddr.Multiaddr) host.Host { + priv, pub, err := crypto.GenerateKeyPair(crypto.RSA, 2048) + if err != nil { + t.Fatal(err) + } + pid, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Fatal(err) + } + ps := peerstore.NewPeerstore() + err = ps.AddPubKey(pid, pub) + if err != nil { + t.Fatal(err) + } + err = ps.AddPrivKey(pid, priv) + if err != nil { + t.Fatal(err) + } + + network, err := swarm.NewNetwork( + context.Background(), + []multiaddr.Multiaddr{listen}, + pid, + ps, + nil) + + if err != nil { + t.Fatal(err) + } + + host := bhost.New(network) + return host +} + +func TestServerClient(t *testing.T) { + m1, _ := multiaddr.NewMultiaddr("/ip4/127.0.0.1/tcp/10000") + m2, _ := multiaddr.NewMultiaddr("/ip4/127.0.0.1/tcp/10001") + srvHost := newHost(t, m1) + clientHost := newHost(t, m2) + defer srvHost.Close() + defer clientHost.Close() + + srvHost.Peerstore().AddAddrs(clientHost.ID(), clientHost.Addrs(), peerstore.PermanentAddrTTL) + clientHost.Peerstore().AddAddrs(srvHost.ID(), srvHost.Addrs(), peerstore.PermanentAddrTTL) + + var tag protocol.ID = "/testitytest" + + go func() { + listener, err := Listen(srvHost, tag) + if err != nil { + t.Fatal(err) + } + defer listener.Close() + servConn, err := listener.Accept() + if err != nil { + t.Fatal(err) + } + defer servConn.Close() + + reader := bufio.NewReader(servConn) + msg, err := reader.ReadString('\n') + if err != nil { + t.Fatal(err) + } + if string(msg) != "is libp2p awesome?\n" { + t.Fatalf("Bad incoming message: %s", msg) + } + + _, err = servConn.Write([]byte("yes it is")) + if err != nil { + t.Fatal(err) + } + }() + + clientConn, err := Dial(clientHost, srvHost.ID(), tag) + if err != nil { + t.Fatal(err) + } + + _, err = clientConn.Write([]byte("is libp2p awesome?\n")) + if err != nil { + t.Fatal(err) + } + + resp, err := ioutil.ReadAll(clientConn) + if err != nil { + t.Fatal(err) + } + + if string(resp) != "yes it is" { + t.Errorf("Bad response: %s", resp) + } + + err = clientConn.Close() + if err != nil { + t.Fatal(err) + } +} diff --git a/p2p/net/gostream/listener.go b/p2p/net/gostream/listener.go new file mode 100644 index 0000000000..c64aa3373b --- /dev/null +++ b/p2p/net/gostream/listener.go @@ -0,0 +1,68 @@ +package gostream + +import ( + "context" + "net" + + host "github.com/libp2p/go-libp2p-host" + pnet "github.com/libp2p/go-libp2p-net" + protocol "github.com/libp2p/go-libp2p-protocol" +) + +// Listener is an implementation of net.Listener which handles +// http-tagged streams from a libp2p connection. +// A listener can be built with Listen() +type Listener struct { + host host.Host + ctx context.Context + tag protocol.ID + cancel func() + streamCh chan pnet.Stream +} + +// Accept returns a connection from this listener. It blocks if there +// are no connections. +func (l *Listener) Accept() (net.Conn, error) { + select { + case s := <-l.streamCh: + return NewConn(s), nil + case <-l.ctx.Done(): + return nil, l.ctx.Err() + } +} + +// Close terminates this listener. It will no longer handle any +// incoming streams +func (l *Listener) Close() error { + l.cancel() + l.host.RemoveStreamHandler(l.tag) + return nil +} + +// Addr returns the address for this listener, which is its libp2p Peer ID. +func (l *Listener) Addr() net.Addr { + return &Addr{l.host.ID()} +} + +// Listen creates a new listener ready to accept streams received by a host. +func Listen(h host.Host, tag protocol.ID) (net.Listener, error) { + ctx, cancel := context.WithCancel(context.Background()) + + l := &Listener{ + host: h, + ctx: ctx, + cancel: cancel, + tag: tag, + streamCh: make(chan pnet.Stream), + } + + h.SetStreamHandler(tag, func(s pnet.Stream) { + select { + case l.streamCh <- s: + case <-ctx.Done(): + s.Close() + } + }) + + return l, nil +} From 8b11532c4c1b8318668d576b7ed0dd76691f0af3 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Wed, 18 Oct 2017 20:02:41 +0200 Subject: [PATCH 0342/3965] Increase coverage --- p2p/net/gostream/gostream_test.go | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/p2p/net/gostream/gostream_test.go b/p2p/net/gostream/gostream_test.go index 5f7634d95b..19ea63b536 100644 --- a/p2p/net/gostream/gostream_test.go +++ b/p2p/net/gostream/gostream_test.go @@ -5,6 +5,7 @@ import ( "context" "io/ioutil" "testing" + "time" crypto "github.com/libp2p/go-libp2p-crypto" host "github.com/libp2p/go-libp2p-host" @@ -71,6 +72,11 @@ func TestServerClient(t *testing.T) { t.Fatal(err) } defer listener.Close() + + if listener.Addr().String() != srvHost.ID().Pretty() { + t.Fatal("bad listener address") + } + servConn, err := listener.Accept() if err != nil { t.Fatal(err) @@ -97,6 +103,29 @@ func TestServerClient(t *testing.T) { t.Fatal(err) } + if clientConn.LocalAddr().String() != clientHost.ID().Pretty() { + t.Fatal("Bad LocalAddr") + } + + if clientConn.RemoteAddr().String() != srvHost.ID().Pretty() { + t.Fatal("Bad RemoteAddr") + } + + err = clientConn.SetDeadline(time.Now().Add(time.Second)) + if err != nil { + t.Fatal(err) + } + + err = clientConn.SetReadDeadline(time.Now().Add(time.Second)) + if err != nil { + t.Fatal(err) + } + + err = clientConn.SetWriteDeadline(time.Now().Add(time.Second)) + if err != nil { + t.Fatal(err) + } + _, err = clientConn.Write([]byte("is libp2p awesome?\n")) if err != nil { t.Fatal(err) From 43f337097797bdde2963f9af84cc2952f2ef12d7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Oct 2017 13:42:31 -0700 Subject: [PATCH 0343/3965] don't backoff dialing when the context is canceled fixes libp2p/go-libp2p-kad-dht#96 --- p2p/net/swarm/swarm_dial.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index d866c57b99..3b0b676cd5 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -203,8 +203,10 @@ func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { cancel() log.Debugf("dial end %s", conn) if err != nil { - log.Event(ctx, "swarmDialBackoffAdd", logdial) - s.backf.AddBackoff(p) // let others know to backoff + if err != context.Canceled { + log.Event(ctx, "swarmDialBackoffAdd", logdial) + s.backf.AddBackoff(p) // let others know to backoff + } // ok, we failed. try again. (if loop is done, our error is output) return nil, fmt.Errorf("dial attempt failed: %s", err) From 5fc7f01b2be5ad5e0598d38d06ec3104942b5e47 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Thu, 19 Oct 2017 13:09:49 +0200 Subject: [PATCH 0344/3965] Make Listener, Conn and Addr private. They aren't meant to be used directly, only as implementations of net.Listener, net.Conn and net.Addr. Accessible via Dial() and Listen(). --- p2p/net/gostream/addr.go | 8 ++++---- p2p/net/gostream/conn.go | 34 +++++++++++++++---------------- p2p/net/gostream/gostream.go | 9 ++++++-- p2p/net/gostream/gostream_test.go | 4 ++++ p2p/net/gostream/listener.go | 25 +++++++++++++---------- 5 files changed, 46 insertions(+), 34 deletions(-) diff --git a/p2p/net/gostream/addr.go b/p2p/net/gostream/addr.go index 3e7f699387..db3b25c50f 100644 --- a/p2p/net/gostream/addr.go +++ b/p2p/net/gostream/addr.go @@ -2,13 +2,13 @@ package gostream import peer "github.com/libp2p/go-libp2p-peer" -// Addr implements net.Addr and holds a libp2p peer ID. -type Addr struct{ id peer.ID } +// addr implements net.Addr and holds a libp2p peer ID. +type addr struct{ id peer.ID } // Network returns the name of the network that this address belongs to // (libp2p). -func (a *Addr) Network() string { return "libp2p" } +func (a *addr) Network() string { return Network } // String returns the peer ID of this address in string form // (B58-encoded). -func (a *Addr) String() string { return a.id.Pretty() } +func (a *addr) String() string { return a.id.Pretty() } diff --git a/p2p/net/gostream/conn.go b/p2p/net/gostream/conn.go index ce5d7feee7..1351beec60 100644 --- a/p2p/net/gostream/conn.go +++ b/p2p/net/gostream/conn.go @@ -11,54 +11,54 @@ import ( protocol "github.com/libp2p/go-libp2p-protocol" ) -// Conn is an implementation of net.Conn which wraps +// conn is an implementation of net.Conn which wraps // libp2p streams. -type Conn struct { +type conn struct { s pnet.Stream } -// NewConn creates a Conn given a libp2p stream -func NewConn(s pnet.Stream) net.Conn { - return &Conn{s} +// newConn creates a conn given a libp2p stream +func newConn(s pnet.Stream) net.Conn { + return &conn{s} } // Read reads data from the connection. -func (c *Conn) Read(b []byte) (n int, err error) { +func (c *conn) Read(b []byte) (n int, err error) { return c.s.Read(b) } // Write writes data to the connection. -func (c *Conn) Write(b []byte) (n int, err error) { +func (c *conn) Write(b []byte) (n int, err error) { return c.s.Write(b) } // Close closes the connection. // Any blocked Read or Write operations will be unblocked and return errors. -func (c *Conn) Close() error { +func (c *conn) Close() error { return c.s.Close() } // LocalAddr returns the local network address. -func (c *Conn) LocalAddr() net.Addr { - return &Addr{c.s.Conn().LocalPeer()} +func (c *conn) LocalAddr() net.Addr { + return &addr{c.s.Conn().LocalPeer()} } // RemoteAddr returns the remote network address. -func (c *Conn) RemoteAddr() net.Addr { - return &Addr{c.s.Conn().RemotePeer()} +func (c *conn) RemoteAddr() net.Addr { + return &addr{c.s.Conn().RemotePeer()} } // SetDeadline sets the read and write deadlines associated // with the connection. It is equivalent to calling both // SetReadDeadline and SetWriteDeadline. // See https://golang.org/pkg/net/#Conn for more details. -func (c *Conn) SetDeadline(t time.Time) error { +func (c *conn) SetDeadline(t time.Time) error { return c.s.SetDeadline(t) } // SetReadDeadline sets the deadline for future Read calls. // A zero value for t means Read will not time out. -func (c *Conn) SetReadDeadline(t time.Time) error { +func (c *conn) SetReadDeadline(t time.Time) error { return c.s.SetReadDeadline(t) } @@ -66,13 +66,13 @@ func (c *Conn) SetReadDeadline(t time.Time) error { // Even if write times out, it may return n > 0, indicating that // some of the data was successfully written. // A zero value for t means Write will not time out. -func (c *Conn) SetWriteDeadline(t time.Time) error { +func (c *conn) SetWriteDeadline(t time.Time) error { return c.s.SetWriteDeadline(t) } // Dial opens a stream to the destination address // (which should parseable to a peer ID) using the given -// host and returns it. +// host and returns it as a standard net.Conn. func Dial(h host.Host, pid peer.ID, tag protocol.ID) (net.Conn, error) { ctx, cancel := context.WithTimeout(context.Background(), time.Second*30) defer cancel() @@ -81,5 +81,5 @@ func Dial(h host.Host, pid peer.ID, tag protocol.ID) (net.Conn, error) { if err != nil { return nil, err } - return NewConn(s), nil + return newConn(s), nil } diff --git a/p2p/net/gostream/gostream.go b/p2p/net/gostream/gostream.go index ae292c59ec..a15125be32 100644 --- a/p2p/net/gostream/gostream.go +++ b/p2p/net/gostream/gostream.go @@ -1,7 +1,7 @@ -// Package `gostream` allows to replace the standard net stack in Go +// Package gostream allows to replace the standard net stack in Go // with [LibP2P](https://github.com/libp2p/libp2p) streams. // -// Given a libp2p.Host, `gostream` provides Dial() and Listen() methods which +// Given a libp2p.Host, gostream provides Dial() and Listen() methods which // return implementations of net.Conn and net.Listener. // // Instead of the regular "host:port" addressing, `gostream` uses a Peer ID, @@ -12,3 +12,8 @@ // Note that LibP2P hosts cannot dial to themselves, so there is no possibility // of using the same Host as server and as client. package gostream + +// Network is the "net.Addr.Network()" name returned by +// addresses used by gostream connections. In turn, the "net.Addr.String()" will +// be a peer ID. +var Network = "libp2p" diff --git a/p2p/net/gostream/gostream_test.go b/p2p/net/gostream/gostream_test.go index 19ea63b536..60d20f57b4 100644 --- a/p2p/net/gostream/gostream_test.go +++ b/p2p/net/gostream/gostream_test.go @@ -111,6 +111,10 @@ func TestServerClient(t *testing.T) { t.Fatal("Bad RemoteAddr") } + if clientConn.LocalAddr().Network() != Network { + t.Fatal("Bad Network()") + } + err = clientConn.SetDeadline(time.Now().Add(time.Second)) if err != nil { t.Fatal(err) diff --git a/p2p/net/gostream/listener.go b/p2p/net/gostream/listener.go index c64aa3373b..9f37a956ca 100644 --- a/p2p/net/gostream/listener.go +++ b/p2p/net/gostream/listener.go @@ -9,10 +9,10 @@ import ( protocol "github.com/libp2p/go-libp2p-protocol" ) -// Listener is an implementation of net.Listener which handles +// listener is an implementation of net.Listener which handles // http-tagged streams from a libp2p connection. // A listener can be built with Listen() -type Listener struct { +type listener struct { host host.Host ctx context.Context tag protocol.ID @@ -20,12 +20,13 @@ type Listener struct { streamCh chan pnet.Stream } -// Accept returns a connection from this listener. It blocks if there -// are no connections. -func (l *Listener) Accept() (net.Conn, error) { +// Accept returns the next a connection to this listener. +// It blocks if there are no connections. Under the hood, +// connections are libp2p streams. +func (l *listener) Accept() (net.Conn, error) { select { case s := <-l.streamCh: - return NewConn(s), nil + return newConn(s), nil case <-l.ctx.Done(): return nil, l.ctx.Err() } @@ -33,22 +34,24 @@ func (l *Listener) Accept() (net.Conn, error) { // Close terminates this listener. It will no longer handle any // incoming streams -func (l *Listener) Close() error { +func (l *listener) Close() error { l.cancel() l.host.RemoveStreamHandler(l.tag) return nil } // Addr returns the address for this listener, which is its libp2p Peer ID. -func (l *Listener) Addr() net.Addr { - return &Addr{l.host.ID()} +func (l *listener) Addr() net.Addr { + return &addr{l.host.ID()} } -// Listen creates a new listener ready to accept streams received by a host. +// Listen provides a standard net.Listener ready to accept "connections". +// Under the hood, these connections are libp2p streams tagged with the +// given protocol.ID. func Listen(h host.Host, tag protocol.ID) (net.Listener, error) { ctx, cancel := context.WithCancel(context.Background()) - l := &Listener{ + l := &listener{ host: h, ctx: ctx, cancel: cancel, From a7ae57bbfc7a9094f218a9219d4408fd782cd6f0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 19 Oct 2017 21:40:57 -0700 Subject: [PATCH 0345/3965] Revert "use the renamed transport interfaces" This reverts commit 21fd167df35acbda11a41e4458c3e6a4b558729c. --- p2p/transport/tcp/tcp.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 87aef6e92c..991981ca9c 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -275,7 +275,7 @@ type tcpConn struct { } var _ tpt.Conn = &tcpConn{} -var _ tpt.DuplexConn = &tcpConn{} +var _ tpt.SingleStreamConn = &tcpConn{} func (c *tcpConn) Transport() tpt.Transport { return c.t From f4deba088c4ceccff9d1761795128ff11f72f3b8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 19 Oct 2017 21:43:38 -0700 Subject: [PATCH 0346/3965] remove reference to SingleStreamConn --- p2p/transport/tcp/tcp.go | 1 - 1 file changed, 1 deletion(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 991981ca9c..5501ae8a56 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -275,7 +275,6 @@ type tcpConn struct { } var _ tpt.Conn = &tcpConn{} -var _ tpt.SingleStreamConn = &tcpConn{} func (c *tcpConn) Transport() tpt.Transport { return c.t From d662cced87b74dc27044edad1d5aca3b2352fb49 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 24 Oct 2017 11:18:46 -0700 Subject: [PATCH 0347/3965] add a way to expose info from the conn manager --- p2p/net/connmgr/connmgr.go | 47 +++++++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index cd712b1e3d..3b4aa4a8fb 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -15,7 +15,7 @@ import ( var log = logging.Logger("connmgr") -type connManager struct { +type BasicConnMgr struct { highWater int lowWater int @@ -29,10 +29,10 @@ type connManager struct { lastTrim time.Time } -var _ ifconnmgr.ConnManager = (*connManager)(nil) +var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) -func NewConnManager(low, hi int, grace time.Duration) ifconnmgr.ConnManager { - return &connManager{ +func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { + return &BasicConnMgr{ highWater: hi, lowWater: low, gracePeriod: grace, @@ -49,7 +49,7 @@ type peerInfo struct { firstSeen time.Time } -func (cm *connManager) TrimOpenConns(ctx context.Context) { +func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { defer log.EventBegin(ctx, "connCleanup").Done() for _, c := range cm.getConnsToClose(ctx) { log.Info("closing conn: ", c.RemotePeer()) @@ -58,7 +58,7 @@ func (cm *connManager) TrimOpenConns(ctx context.Context) { } } -func (cm *connManager) getConnsToClose(ctx context.Context) []inet.Conn { +func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { cm.lk.Lock() defer cm.lk.Unlock() if cm.lowWater == 0 || cm.highWater == 0 { @@ -107,7 +107,7 @@ func (cm *connManager) getConnsToClose(ctx context.Context) []inet.Conn { return closed } -func (cm *connManager) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { +func (cm *BasicConnMgr) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { cm.lk.Lock() defer cm.lk.Unlock() @@ -133,7 +133,7 @@ func (cm *connManager) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { return out } -func (cm *connManager) TagPeer(p peer.ID, tag string, val int) { +func (cm *BasicConnMgr) TagPeer(p peer.ID, tag string, val int) { cm.lk.Lock() defer cm.lk.Unlock() @@ -147,7 +147,7 @@ func (cm *connManager) TagPeer(p peer.ID, tag string, val int) { pi.tags[tag] = val } -func (cm *connManager) UntagPeer(p peer.ID, tag string) { +func (cm *BasicConnMgr) UntagPeer(p peer.ID, tag string) { cm.lk.Lock() defer cm.lk.Unlock() @@ -161,14 +161,35 @@ func (cm *connManager) UntagPeer(p peer.ID, tag string) { delete(pi.tags, tag) } -func (cm *connManager) Notifee() inet.Notifiee { +type CMInfo struct { + LowWater int + HighWater int + LastTrim time.Time + GracePeriod time.Duration + ConnCount int +} + +func (cm *BasicConnMgr) GetInfo() CMInfo { + cm.lk.Lock() + defer cm.lk.Unlock() + + return CMInfo{ + HighWater: cm.highWater, + LowWater: cm.lowWater, + LastTrim: cm.lastTrim, + GracePeriod: cm.gracePeriod, + ConnCount: cm.connCount, + } +} + +func (cm *BasicConnMgr) Notifee() inet.Notifiee { return (*cmNotifee)(cm) } -type cmNotifee connManager +type cmNotifee BasicConnMgr -func (nn *cmNotifee) cm() *connManager { - return (*connManager)(nn) +func (nn *cmNotifee) cm() *BasicConnMgr { + return (*BasicConnMgr)(nn) } func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { From 88f2b707a88112dd622408325e08fac6abae2b3f Mon Sep 17 00:00:00 2001 From: Jeromy Date: Sat, 28 Oct 2017 06:08:21 -0700 Subject: [PATCH 0348/3965] demote error message to info --- p2p/net/connmgr/connmgr.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 3b4aa4a8fb..fc406bf8b8 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -139,7 +139,7 @@ func (cm *BasicConnMgr) TagPeer(p peer.ID, tag string, val int) { pi, ok := cm.peers[p] if !ok { - log.Error("tried to tag conn from untracked peer: ", p) + log.Info("tried to tag conn from untracked peer: ", p) return } @@ -153,7 +153,7 @@ func (cm *BasicConnMgr) UntagPeer(p peer.ID, tag string) { pi, ok := cm.peers[p] if !ok { - log.Error("tried to remove tag from untracked peer: ", p) + log.Info("tried to remove tag from untracked peer: ", p) return } From 055327437924d45ee50403c34528f2f34e006657 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 9 Nov 2017 11:58:38 -0800 Subject: [PATCH 0349/3965] update go-reuseport 1. Migrate to libp2p repo. 2. Pull in fixes for edge platforms. Related: use `x/sys` instead of `syscall` when possible. 3. Pull in less complicated and slightly faster dial logic. --- p2p/transport/tcp/reuseport.go | 2 +- p2p/transport/tcp/tcp.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/transport/tcp/reuseport.go b/p2p/transport/tcp/reuseport.go index 6ead3a74a4..db2b3da698 100644 --- a/p2p/transport/tcp/reuseport.go +++ b/p2p/transport/tcp/reuseport.go @@ -6,7 +6,7 @@ import ( "strings" "syscall" - reuseport "github.com/jbenet/go-reuseport" + reuseport "github.com/libp2p/go-reuseport" ) // envReuseport is the env variable name used to turn off reuse port. diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 5501ae8a56..442b3d9fc3 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -7,7 +7,7 @@ import ( "sync" logging "github.com/ipfs/go-log" - reuseport "github.com/jbenet/go-reuseport" + reuseport "github.com/libp2p/go-reuseport" tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" From 4381ea23a48c256a77e5d0080a1a4cb92c75103f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 9 Nov 2017 16:02:02 -0800 Subject: [PATCH 0350/3965] don't pass the dial timeout option It doesn't exist anymore. This is now handled entirely though contexts. --- p2p/net/swarm/swarm.go | 4 ++-- p2p/net/swarm/swarm_dial.go | 5 ----- p2p/net/swarm/swarm_listen.go | 2 +- 3 files changed, 3 insertions(+), 8 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index dac2624d96..eabd6d67f6 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -129,7 +129,7 @@ func NewSwarmWithProtector(ctx context.Context, listenAddrs []ma.Multiaddr, loca local: local, peers: peers, ctx: ctx, - dialT: DialTimeout, + dialT: conn.DialTimeout, notifs: make(map[inet.Notifiee]ps.Notifiee), transports: []transport.Transport{ tcpt.NewTCPTransport(), @@ -164,7 +164,7 @@ func NewBlankSwarm(ctx context.Context, id peer.ID, privkey ci.PrivKey, pstpt ps local: id, peers: pstore.NewPeerstore(), ctx: ctx, - dialT: DialTimeout, + dialT: conn.DialTimeout, notifs: make(map[inet.Notifiee]ps.Notifiee), fdRateLimit: make(chan struct{}, concurrentFdDials), Filters: filter.NewFilters(), diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 3b0b676cd5..1cc03580fd 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -48,11 +48,6 @@ const concurrentFdDials = 160 // number of concurrent outbound dials to make per peer const defaultPerPeerRateLimit = 8 -// DialTimeout is the amount of time each dial attempt has. We can think about making -// this larger down the road, or putting more granular timeouts (i.e. within each -// subcomponent of Dial) -var DialTimeout = time.Second * 10 - // dialbackoff is a struct used to avoid over-dialing the same, dead peers. // Whenever we totally time out on a peer (all three attempts), we add them // to dialbackoff. Then, whenevers goroutines would _wait_ (dialsync), they diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 98d3549a4d..1e4a92efc9 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -20,7 +20,7 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { return fmt.Errorf("no transport for address: %s", a) } - d, err := tpt.Dialer(a, transport.TimeoutOpt(DialTimeout), transport.ReusePorts) + d, err := tpt.Dialer(a, transport.ReusePorts) if err != nil { return err } From 0ee4048d653a8ac65d6db2be406340e80e78f1f8 Mon Sep 17 00:00:00 2001 From: Kevin Atkinson Date: Fri, 17 Nov 2017 03:57:35 -0500 Subject: [PATCH 0351/3965] Fix "go vet" errors. --- p2p/net/nat/nat.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 602667a3fc..d68aca0752 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -122,7 +122,7 @@ func (nat *NAT) NewMapping(maddr ma.Multiaddr) (Mapping, error) { network, addr, err := manet.DialArgs(maddr) if err != nil { - return nil, fmt.Errorf("DialArgs failed on addr:", maddr.String()) + return nil, fmt.Errorf("DialArgs failed on addr: %s", maddr.String()) } switch network { @@ -235,10 +235,10 @@ func (nat *NAT) PortMapAddrs(addrs []ma.Multiaddr) { for _, addr := range addrs { // do all of them concurrently wg.Add(1) - go func() { + go func(addr ma.Multiaddr) { defer wg.Done() nat.NewMapping(addr) - }() + }(addr) } wg.Wait() } From 191577982ff857e14c46907e0b94c8ff62c30714 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 20 Nov 2017 13:16:19 -0800 Subject: [PATCH 0352/3965] update msgio --- p2p/net/pnet/psk_conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index cce585216f..a24968a1c1 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -6,9 +6,9 @@ import ( "io" salsa20 "github.com/davidlazar/go-crypto/salsa20" - mpool "github.com/jbenet/go-msgio/mpool" ipnet "github.com/libp2p/go-libp2p-interface-pnet" tconn "github.com/libp2p/go-libp2p-transport" + mpool "github.com/libp2p/go-msgio/mpool" ) // we are using buffer pool as user needs their slice back From 9e983dad027b62c91427102002d3b08d6a8b1057 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 20 Nov 2017 13:17:46 -0800 Subject: [PATCH 0353/3965] don't copy the buffer pool use the global one --- p2p/net/pnet/psk_conn.go | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index a24968a1c1..a6e4247185 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -14,8 +14,6 @@ import ( // we are using buffer pool as user needs their slice back // so we can't do XOR cripter in place var ( - bufPool = mpool.ByteSlicePool - errShortNonce = ipnet.NewError("could not read full nonce") errInsecureNil = ipnet.NewError("insecure is nil") errPSKNil = ipnet.NewError("pre-shread key is nil") @@ -40,8 +38,8 @@ func (c *pskConn) Read(out []byte) (int, error) { } maxn := uint32(len(out)) - in := bufPool.Get(maxn).([]byte) // get buffer - defer bufPool.Put(maxn, in) // put the buffer back + in := mpool.ByteSlicePool.Get(maxn).([]byte) // get buffer + defer mpool.ByteSlicePool.Put(maxn, in) // put the buffer back in = in[:maxn] // truncate to required length n, err := c.Conn.Read(in) // read to in @@ -69,8 +67,8 @@ func (c *pskConn) Write(in []byte) (int, error) { c.writeS20 = salsa20.New(c.psk, nonce) } n := uint32(len(in)) - out := bufPool.Get(n).([]byte) // get buffer - defer bufPool.Put(n, out) // put the buffer back + out := mpool.ByteSlicePool.Get(n).([]byte) // get buffer + defer mpool.ByteSlicePool.Put(n, out) // put the buffer back out = out[:n] // truncate to required length c.writeS20.XORKeyStream(out, in) // encrypt From a1ed19f911945c3c28da82635d5cac02811851ea Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 20 Nov 2017 16:02:11 -0800 Subject: [PATCH 0354/3965] gx: update deps * pull through libp2p-transport update * update moved go-libp2p-dummy-conn --- p2p/net/pnet/psk_conn_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/pnet/psk_conn_test.go b/p2p/net/pnet/psk_conn_test.go index 17a3963d92..c76f474af7 100644 --- a/p2p/net/pnet/psk_conn_test.go +++ b/p2p/net/pnet/psk_conn_test.go @@ -6,7 +6,7 @@ import ( "math/rand" "testing" - dconn "github.com/Kubuxu/go-libp2p-dummy-conn" + dconn "github.com/libp2p/go-libp2p-dummy-conn" tconn "github.com/libp2p/go-libp2p-transport" ) From 140bad3585f20b59b19413d8777b149fc963a532 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 20 Nov 2017 17:52:09 -0800 Subject: [PATCH 0355/3965] nil out references to finished dials We need a better solution (this is a memory hotspot when mass dialing) but that will take some thought and effort. This is a simple fix that should reduce memory usage a bit. --- p2p/net/swarm/limiter.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index f2513d8e10..f6cf6dda26 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -71,6 +71,7 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { if len(dl.waitingOnFd) > 0 { next := dl.waitingOnFd[0] + dl.waitingOnFd[0] = nil // clear out memory dl.waitingOnFd = dl.waitingOnFd[1:] if len(dl.waitingOnFd) == 0 { dl.waitingOnFd = nil // clear out memory @@ -94,6 +95,7 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { if len(waitlist) == 1 { delete(dl.waitingOnPeerLimit, dj.peer) } else { + waitlist[0] = nil // clear out memory dl.waitingOnPeerLimit[dj.peer] = waitlist[1:] } dl.activePerPeer[dj.peer]++ // just kidding, we still want this token From 2d2035f7575dc3b339d5f81529741eb71bee7045 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 29 Nov 2017 10:31:36 -0800 Subject: [PATCH 0356/3965] close connections we fail to wrap Maybe related to #21. Probably not. --- p2p/transport/websocket/listener.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/transport/websocket/listener.go b/p2p/transport/websocket/listener.go index 785173a0c7..146b371a9f 100644 --- a/p2p/transport/websocket/listener.go +++ b/p2p/transport/websocket/listener.go @@ -54,6 +54,7 @@ func (l *listener) Accept() (tpt.Conn, error) { mnc, err := manet.WrapNetConn(c) if err != nil { + c.Close() return nil, err } From 250a23c6dbf6a658d094e297ce4ac3fa254879f2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 3 Dec 2017 14:24:53 -0800 Subject: [PATCH 0357/3965] close connections we fail to dial The previous commit did this for connections we accept but I forgot to do this for dialed connections. --- p2p/transport/websocket/dialer.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/transport/websocket/dialer.go b/p2p/transport/websocket/dialer.go index 4c74df39d7..7a5a91579a 100644 --- a/p2p/transport/websocket/dialer.go +++ b/p2p/transport/websocket/dialer.go @@ -28,6 +28,7 @@ func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, mnc, err := manet.WrapNetConn(NewConn(wscon, nil)) if err != nil { + wscon.Close() return nil, err } From 5ec6564e7d7201d9c76841e181913791cb5716f0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 3 Dec 2017 21:11:42 -0800 Subject: [PATCH 0358/3965] gx publish 1.2.6 --- p2p/transport/tcp/tcp.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index edc3b2fc10..bd9b99431e 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -7,8 +7,8 @@ import ( "sync" logging "github.com/ipfs/go-log" - reuseport "github.com/libp2p/go-reuseport" tpt "github.com/libp2p/go-libp2p-transport" + reuseport "github.com/libp2p/go-reuseport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" From 00ded6820ef6b410a88266227b56af69fcde9494 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Dec 2017 16:20:51 -0800 Subject: [PATCH 0359/3965] preallocate duplicate checking map in Peers We call this method in `FindProvidersAsync`. Really, we shouldn't even be doing that (but this is "good enough" for now). --- p2p/net/swarm/swarm.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index eabd6d67f6..e7d6b9a7f4 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -313,7 +313,7 @@ func (s *Swarm) CloseConnection(p peer.ID) error { func (s *Swarm) Peers() []peer.ID { conns := s.Connections() - seen := make(map[peer.ID]struct{}) + seen := make(map[peer.ID]struct{}, len(conns)) peers := make([]peer.ID, 0, len(conns)) for _, c := range conns { p := c.RemotePeer() From 3a9c576182f49f4bc0a58aecc7325eb3056ff84c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 Nov 2017 18:08:58 -0800 Subject: [PATCH 0360/3965] more effectively free memory * Completely free memory related to subscriptions when all subscribers go away. * Completely free peer address records when they all expire. * Don't copy the *entire* subscription array when unsubscribing from a peer address stream. --- p2p/host/peerstore/addr_manager.go | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index a676f44252..4dba9b0032 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -205,6 +205,9 @@ func (mgr *AddrManager) Addrs(p peer.ID) []ma.Multiaddr { for _, s := range expired { delete(maddrs, s) } + if len(maddrs) == 0 { + delete(mgr.addrs, p) + } return good } @@ -214,20 +217,28 @@ func (mgr *AddrManager) ClearAddrs(p peer.ID) { defer mgr.addrmu.Unlock() mgr.init() - mgr.addrs[p] = make(addrSet) // clear what was there before + delete(mgr.addrs, p) } func (mgr *AddrManager) removeSub(p peer.ID, s *addrSub) { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() subs := mgr.addrSubs[p] - var filtered []*addrSub - for _, v := range subs { - if v != s { - filtered = append(filtered, v) + if len(subs) == 1 { + if subs[0] != s { + return + } + delete(mgr.addrSubs, p) + return + } + for i, v := range subs { + if v == s { + subs[i] = subs[len(subs)-1] + subs[len(subs)-1] = nil + mgr.addrSubs[p] = subs[:len(subs)-1] + return } } - mgr.addrSubs[p] = filtered } type addrSub struct { From f18e1dc754cc2dd30b5daeac3620fcc592bd4bfa Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Dec 2017 17:37:20 -0800 Subject: [PATCH 0361/3965] preallocate AddrStream buffer --- p2p/host/peerstore/addr_manager.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 4dba9b0032..262f6cf95a 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -267,7 +267,7 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mul mgr.addrSubs[p] = append(mgr.addrSubs[p], sub) baseaddrset := mgr.addrs[p] - var initial []ma.Multiaddr + initial := make([]ma.Multiaddr, 0, len(baseaddrset)) for _, a := range baseaddrset { initial = append(initial, a.Addr) } @@ -277,7 +277,7 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mul go func(buffer []ma.Multiaddr) { defer close(out) - sent := make(map[string]bool) + sent := make(map[string]bool, len(buffer)) var outch chan ma.Multiaddr for _, a := range buffer { From 9cb48cfed861fa5b8ea10028022d200b3caaac37 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Dec 2017 17:38:43 -0800 Subject: [PATCH 0362/3965] faster duplicate check in AddrStream --- p2p/host/peerstore/addr_manager.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 262f6cf95a..b94b37b3a8 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -281,7 +281,7 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mul var outch chan ma.Multiaddr for _, a := range buffer { - sent[a.String()] = true + sent[string(a.Bytes())] = true } var next ma.Multiaddr @@ -302,11 +302,11 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mul next = nil } case naddr := <-sub.pubch: - if sent[naddr.String()] { + if sent[string(naddr.Bytes())] { continue } - sent[naddr.String()] = true + sent[string(naddr.Bytes())] = true if next == nil { next = naddr outch = out From 47495fcb6f33e9f6cd43d8d3ba5021d5a0c4a0ef Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Dec 2017 18:03:03 -0800 Subject: [PATCH 0363/3965] add test for multiple subscriptions --- p2p/host/peerstore/peerstore_test.go | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 6dbd8ab96c..6c4d27cbaf 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -54,6 +54,10 @@ func TestAddrStream(t *testing.T) { } } + // start a second stream + ctx2, cancel2 := context.WithCancel(context.Background()) + addrch2 := ps.AddrStream(ctx2, pid) + done := make(chan struct{}) go func() { defer close(done) @@ -83,9 +87,17 @@ func TestAddrStream(t *testing.T) { } } - // now cancel it, and add a few more addresses it doesnt hang afterwards + // now cancel it cancel() + // now check the *second* subscription. We should see 80 addresses. + for i := 0; i < 80; i++ { + <-addrch2 + } + + cancel2() + + // and add a few more addresses it doesnt hang afterwards for _, a := range addrs[80:] { ps.AddAddr(pid, a, time.Hour) } From ec33afbcaae0faabc76544a55fe9578149f59230 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 27 Dec 2017 16:26:37 -0800 Subject: [PATCH 0364/3965] Revert "fix source address not being set in non REUSEPORT dialer" This reverts commit af1302af3ea4c560159cf4f4ff3379ea6595d7f4. If we do this, we end up using the source *port* for dialing. That's not what we want when not using reuseport. fixes #19 --- p2p/transport/tcp/tcp.go | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index bd9b99431e..cd8466aae7 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -52,12 +52,6 @@ func (t *TcpTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dial } var base manet.Dialer - la, err := manet.ToNetAddr(laddr) - if err != nil { - return nil, err // something wrong with laddr. - } - base.Dialer.LocalAddr = la - var doReuse bool for _, o := range opts { switch o := o.(type) { @@ -143,6 +137,12 @@ type tcpDialer struct { var _ tpt.Dialer = &tcpDialer{} func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReuse bool) (*tcpDialer, error) { + // get the local net.Addr manually + la, err := manet.ToNetAddr(laddr) + if err != nil { + return nil, err // something wrong with laddr. + } + var pattern mafmt.Pattern if TCP4.Matches(laddr) { pattern = TCP4 @@ -154,7 +154,10 @@ func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReu if doReuse && ReuseportIsAvailable() { rd := reuseport.Dialer{ - D: base.Dialer, + D: net.Dialer{ + LocalAddr: la, + Timeout: base.Timeout, + }, } return &tcpDialer{ From 69b7572bb71f922970f198b53cb52d6cfe075ca8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 27 Dec 2017 16:52:04 -0800 Subject: [PATCH 0365/3965] set the source IP, but not port, when not using the reuseport dialer --- p2p/transport/tcp/tcp.go | 59 +++++++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 25 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index cd8466aae7..0711edebd6 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -50,8 +50,6 @@ func (t *TcpTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dial if found { return d, nil } - var base manet.Dialer - var doReuse bool for _, o := range opts { switch o := o.(type) { @@ -62,7 +60,7 @@ func (t *TcpTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dial } } - tcpd, err := t.newTcpDialer(base, laddr, doReuse) + tcpd, err := t.newTcpDialer(laddr, doReuse) if err != nil { return nil, err } @@ -136,11 +134,23 @@ type tcpDialer struct { var _ tpt.Dialer = &tcpDialer{} -func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReuse bool) (*tcpDialer, error) { +func maddrToTcp(addr ma.Multiaddr) (*net.TCPAddr, error) { + la, err := manet.ToNetAddr(addr) + if err != nil { + return nil, err // something wrong with addr. + } + latcp, ok := la.(*net.TCPAddr) + if !ok { + return nil, fmt.Errorf("not a tcp multiaddr: %s", addr) + } + return latcp, nil +} + +func (t *TcpTransport) newTcpDialer(laddr ma.Multiaddr, doReuse bool) (*tcpDialer, error) { // get the local net.Addr manually - la, err := manet.ToNetAddr(laddr) + la, err := maddrToTcp(laddr) if err != nil { - return nil, err // something wrong with laddr. + return nil, err } var pattern mafmt.Pattern @@ -152,31 +162,30 @@ func (t *TcpTransport) newTcpDialer(base manet.Dialer, laddr ma.Multiaddr, doReu return nil, fmt.Errorf("local addr did not match TCP4 or TCP6: %s", laddr) } + // Ignore the port when constructing the default (non-reuseport) dialer. + labase := *la + labase.Port = 0 + + dialer := &tcpDialer{ + laddr: laddr, + pattern: pattern, + madialer: manet.Dialer{ + Dialer: net.Dialer{ + LocalAddr: &labase, + }, + }, + transport: t, + } + if doReuse && ReuseportIsAvailable() { - rd := reuseport.Dialer{ + dialer.doReuse = true + dialer.rd = reuseport.Dialer{ D: net.Dialer{ LocalAddr: la, - Timeout: base.Timeout, }, } - - return &tcpDialer{ - doReuse: true, - laddr: laddr, - rd: rd, - madialer: base, - transport: t, - pattern: pattern, - }, nil } - - return &tcpDialer{ - doReuse: false, - laddr: laddr, - pattern: pattern, - madialer: base, - transport: t, - }, nil + return dialer, nil } func (d *tcpDialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { From 6138f692cd631105856b962b56db525e09763723 Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 2 Jan 2018 21:09:00 -0800 Subject: [PATCH 0366/3965] clear out extra dial jobs after dial finishes --- p2p/net/swarm/limiter.go | 9 +++++++++ p2p/net/swarm/swarm_dial.go | 2 ++ 2 files changed, 11 insertions(+) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index f6cf6dda26..c3d8951894 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -144,6 +144,15 @@ func (dl *dialLimiter) AddDialJob(dj *dialJob) { go dl.executeDial(dj) } +func (dl *dialLimiter) clearAllPeerDials(p peer.ID) { + dl.rllock.Lock() + defer dl.rllock.Unlock() + delete(dl.waitingOnPeerLimit, p) + // NB: the waitingOnFd list doesnt need to be cleaned out here, we will + // remove them as we encounter them because they are 'cancelled' at this + // point +} + // executeDial calls the dialFunc, and reports the result through the response // channel when finished. Once the response is sent it also releases all tokens // it held during the dial. diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 1cc03580fd..1531bb2ed9 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -293,6 +293,8 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. defaultDialFail := fmt.Errorf("failed to dial %s (default failure)", p) exitErr := defaultDialFail + defer s.limiter.clearAllPeerDials(p) + var active int for { select { From 3ea2727f00f928419b3592407750bd1c48fd59f9 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jan 2018 12:59:43 -0800 Subject: [PATCH 0367/3965] mark test helpers --- p2p/host/peerstore/addr_manager_test.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index b9cbfd91b1..e1ae0fda23 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -9,6 +9,7 @@ import ( ) func IDS(t *testing.T, ids string) peer.ID { + t.Helper() id, err := peer.IDB58Decode(ids) if err != nil { t.Fatalf("id %q is bad: %s", ids, err) @@ -17,6 +18,7 @@ func IDS(t *testing.T, ids string) peer.ID { } func MA(t *testing.T, m string) ma.Multiaddr { + t.Helper() maddr, err := ma.NewMultiaddr(m) if err != nil { t.Fatal(err) @@ -25,6 +27,7 @@ func MA(t *testing.T, m string) ma.Multiaddr { } func testHas(t *testing.T, exp, act []ma.Multiaddr) { + t.Helper() if len(exp) != len(act) { t.Fatal("lengths not the same") } From ecdac8658bb0e4ce12533ff5603407d21bc11f6c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jan 2018 13:00:37 -0800 Subject: [PATCH 0368/3965] add method to atomically update TTLs This allows one to find all addresses with a given TTL and update them to have a new TTL. --- p2p/host/peerstore/addr_manager.go | 38 +++++++++++++++++++++++++----- p2p/host/peerstore/peerstore.go | 4 ++++ 2 files changed, 36 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index b94b37b3a8..c7b5b35c39 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -40,12 +40,13 @@ const ( ) type expiringAddr struct { - Addr ma.Multiaddr - TTL time.Time + Addr ma.Multiaddr + TTL time.Duration + Expires time.Time } func (e *expiringAddr) ExpiredBy(t time.Time) bool { - return t.After(e.TTL) + return t.After(e.Expires) } type addrSet map[string]expiringAddr @@ -122,8 +123,8 @@ func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat addrstr := string(addr.Bytes()) a, found := amap[addrstr] - if !found || exp.After(a.TTL) { - amap[addrstr] = expiringAddr{Addr: addr, TTL: exp} + if !found || exp.After(a.Expires) { + amap[addrstr] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} for _, sub := range subs { sub.pubAddr(addr) @@ -164,7 +165,7 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat addrs := string(addr.Bytes()) if ttl > 0 { - amap[addrs] = expiringAddr{Addr: addr, TTL: exp} + amap[addrs] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} for _, sub := range subs { sub.pubAddr(addr) @@ -175,6 +176,31 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat } } +// UpdateAddrs updates the addresses associated with the given peer that have +// the given oldTTL to have the given newTTL. +func (mgr *AddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + + if mgr.addrs == nil { + return + } + + amap, found := mgr.addrs[p] + if !found { + return + } + + exp := time.Now().Add(newTTL) + for addrstr, aexp := range amap { + if oldTTL == aexp.TTL { + aexp.TTL = newTTL + aexp.Expires = exp + amap[addrstr] = aexp + } + } +} + // Addresses returns all known (and valid) addresses for a given func (mgr *AddrManager) Addrs(p peer.ID) []ma.Multiaddr { mgr.addrmu.Lock() diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 9747984317..d0fe38c646 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -69,6 +69,10 @@ type AddrBook interface { // This is used when we receive the best estimate of the validity of an address. SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) + // UpdateAddrs updates the addresses associated with the given peer that have + // the given oldTTL to have the given newTTL. + UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) + // Addresses returns all known (and valid) addresses for a given peer Addrs(p peer.ID) []ma.Multiaddr From acef05b7c675f1bc35187e6f5dee6f0034674b38 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jan 2018 13:01:24 -0800 Subject: [PATCH 0369/3965] add a test for UpdateAddrs --- p2p/host/peerstore/addr_manager_test.go | 38 +++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index e1ae0fda23..06ff66c983 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -205,6 +205,44 @@ func TestSetNegativeTTLClears(t *testing.T) { testHas(t, nil, m.Addrs(id1)) } +func TestUpdateTTLs(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := MA(t, "/ip4/1.2.3.1/tcp/1112") + ma21 := MA(t, "/ip4/1.2.3.1/tcp/1121") + ma22 := MA(t, "/ip4/1.2.3.1/tcp/1122") + + m := AddrManager{} + m.SetAddr(id1, ma11, time.Hour) + m.SetAddr(id1, ma12, time.Minute) + m.SetAddr(id2, ma21, time.Hour) + m.SetAddr(id2, ma22, time.Minute) + + testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + + m.UpdateAddrs(id1, time.Hour, time.Millisecond) + + testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + + time.Sleep(time.Millisecond) + + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + + m.UpdateAddrs(id2, time.Hour, time.Millisecond) + + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + + time.Sleep(time.Millisecond) + + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma22}, m.Addrs(id2)) +} + func TestNilAddrsDontBreak(t *testing.T) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") m := AddrManager{} From e64bf7cd5e83afcf1d94cfe30db8d9248a79ff04 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jan 2018 14:51:36 -0800 Subject: [PATCH 0370/3965] make all TTLs distinct (also, make permanent TTLs actually permanent) --- p2p/host/peerstore/addr_manager.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index c7b5b35c39..0ffa3cfc3c 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -2,6 +2,7 @@ package peerstore import ( "context" + "math" "sort" "sync" "time" @@ -27,16 +28,19 @@ const ( // OwnObservedAddrTTL is used for our own external addresses observed by peers. OwnObservedAddrTTL = time.Minute * 10 +) +// Perminent TTLs (distinct so we can distinguish between them) +const ( // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes) // if we haven't shipped you an update to ipfs in 356 days // we probably arent running the same bootstrap nodes... - PermanentAddrTTL = time.Hour * 24 * 356 + PermanentAddrTTL = math.MaxInt64 - iota // ConnectedAddrTTL is the ttl used for the addresses of a peer to whom // we're connected directly. This is basically permanent, as we will // clear them + re-add under a TempAddrTTL after disconnecting. - ConnectedAddrTTL = PermanentAddrTTL + ConnectedAddrTTL ) type expiringAddr struct { From 1762e47d7829693ab698b6c2c2b723f014c45c44 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jan 2018 16:32:58 -0800 Subject: [PATCH 0371/3965] fix PerminentAddrTTL comment --- p2p/host/peerstore/addr_manager.go | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 0ffa3cfc3c..a12af8d38b 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -32,9 +32,7 @@ const ( // Perminent TTLs (distinct so we can distinguish between them) const ( - // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes) - // if we haven't shipped you an update to ipfs in 356 days - // we probably arent running the same bootstrap nodes... + // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes). PermanentAddrTTL = math.MaxInt64 - iota // ConnectedAddrTTL is the ttl used for the addresses of a peer to whom From ce1f3b3ef92a1695db64a75ff2997d348aef8d76 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jan 2018 18:15:01 -0800 Subject: [PATCH 0372/3965] spelling --- p2p/host/peerstore/addr_manager.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index a12af8d38b..1dfe9dc72b 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -30,7 +30,7 @@ const ( OwnObservedAddrTTL = time.Minute * 10 ) -// Perminent TTLs (distinct so we can distinguish between them) +// Permanent TTLs (distinct so we can distinguish between them) const ( // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes). PermanentAddrTTL = math.MaxInt64 - iota From 2f06feab490fc0f958541b620028259ebbef9290 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jan 2018 18:23:02 -0800 Subject: [PATCH 0373/3965] improve UpdateTTLs test coverage --- p2p/host/peerstore/addr_manager_test.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 06ff66c983..9ade2c5950 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -214,8 +214,16 @@ func TestUpdateTTLs(t *testing.T) { ma22 := MA(t, "/ip4/1.2.3.1/tcp/1122") m := AddrManager{} + + // Shouldn't panic. + m.UpdateAddrs(id1, time.Hour, time.Minute) + m.SetAddr(id1, ma11, time.Hour) m.SetAddr(id1, ma12, time.Minute) + + // Shouldn't panic. + m.UpdateAddrs(id2, time.Hour, time.Minute) + m.SetAddr(id2, ma21, time.Hour) m.SetAddr(id2, ma22, time.Minute) From b62f422394bae86b14092b0a404b2630a6c6df83 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jan 2018 19:30:22 -0800 Subject: [PATCH 0374/3965] make it possible to modify non-permanent TTLs useful for testing, at the very least. --- p2p/host/peerstore/addr_manager.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 1dfe9dc72b..aa913b87a4 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -12,7 +12,7 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -const ( +var ( // TempAddrTTL is the ttl used for a short lived address TempAddrTTL = time.Second * 10 @@ -30,7 +30,8 @@ const ( OwnObservedAddrTTL = time.Minute * 10 ) -// Permanent TTLs (distinct so we can distinguish between them) +// Permanent TTLs (distinct so we can distinguish between them, constant as they +// are, in fact, permanent) const ( // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes). PermanentAddrTTL = math.MaxInt64 - iota From 22b78018de8df8e4d84b0530d3f72b41fca44af2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 9 Jan 2018 11:14:54 -0800 Subject: [PATCH 0375/3965] cleanup connection setup We already setup the connection from within the new connection handler. No need to do it *again* on dial. --- p2p/net/swarm/swarm_conn.go | 29 ----------------------------- p2p/net/swarm/swarm_dial.go | 22 ++++++++++------------ p2p/net/swarm/swarm_listen.go | 25 +++++++++++++------------ 3 files changed, 23 insertions(+), 53 deletions(-) diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 0b0c626cea..c143062cee 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -1,7 +1,6 @@ package swarm import ( - "context" "fmt" ic "github.com/libp2p/go-libp2p-crypto" @@ -121,31 +120,3 @@ func wrapConns(conns1 []*ps.Conn) []*Conn { } return conns2 } - -// newConnSetup does the swarm's "setup" for a connection. returns the underlying -// conn.Conn this method is used by both swarm.Dial and ps.Swarm connHandler -func (s *Swarm) newConnSetup(ctx context.Context, psConn *ps.Conn) (*Conn, error) { - - // wrap with a Conn - sc, err := wrapConn(psConn) - if err != nil { - return nil, err - } - - // if we have a public key, make sure we add it to our peerstore! - // This is an important detail. Otherwise we must fetch the public - // key from the DHT or some other system. - if pk := sc.RemotePublicKey(); pk != nil { - s.peers.AddPubKey(sc.RemotePeer(), pk) - } - - // ok great! we can use it. add it to our group. - - // set the RemotePeer as a group on the conn. this lets us group - // connections in the StreamSwarm by peer, and get a streams from - // any available connection in the group (better multiconn): - // swarm.StreamSwarm().NewStreamWithGroup(remotePeer) - psConn.AddGroup(sc.RemotePeer()) - - return sc, nil -} diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 1531bb2ed9..d977d3752b 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -189,14 +189,16 @@ func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) // doDial is an ugly shim method to retain all the logging and backoff logic // of the old dialsync code func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { - var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) + ctx, cancel := context.WithTimeout(ctx, s.dialT) + defer cancel() + + logdial := lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) + // ok, we have been charged to dial! let's do it. // if it succeeds, dial will add the conn to the swarm itself. defer log.EventBegin(ctx, "swarmDialAttemptStart", logdial).Done() - ctxT, cancel := context.WithTimeout(ctx, s.dialT) - conn, err := s.dial(ctxT, p) - cancel() - log.Debugf("dial end %s", conn) + + conn, err := s.dial(ctx, p) if err != nil { if err != context.Canceled { log.Event(ctx, "swarmDialBackoffAdd", logdial) @@ -206,8 +208,6 @@ func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { // ok, we failed. try again. (if loop is done, our error is output) return nil, fmt.Errorf("dial attempt failed: %s", err) } - log.Event(ctx, "swarmDialBackoffClear", logdial) - s.backf.Clear(p) // okay, no longer need to backoff return conn, nil } @@ -373,10 +373,8 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (ico var ConnSetupTimeout = time.Minute * 5 -// dialConnSetup is the setup logic for a connection from the dial side. it -// needs to add the Conn to the StreamSwarm, then run newConnSetup +// dialConnSetup is the setup logic for a connection from the dial side. func dialConnSetup(ctx context.Context, s *Swarm, connC iconn.Conn) (*Conn, error) { - deadline, ok := ctx.Deadline() if !ok { deadline = time.Now().Add(ConnSetupTimeout) @@ -386,14 +384,14 @@ func dialConnSetup(ctx context.Context, s *Swarm, connC iconn.Conn) (*Conn, erro return nil, err } + // Add conn to ps swarm. Setup will be done by the connection handler. psC, err := s.swarm.AddConn(connC) if err != nil { // connC is closed by caller if we fail. return nil, fmt.Errorf("failed to add conn to ps.Swarm: %s", err) } - // ok try to setup the new connection. (newConnSetup will add to group) - swarmC, err := s.newConnSetup(ctx, psC) + swarmC, err := wrapConn(psC) if err != nil { psC.Close() // we need to make sure psC is Closed. return nil, err diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 1e4a92efc9..148d9a8881 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -147,23 +147,24 @@ func (s *Swarm) addConnListener(list iconn.Listener) error { // will take a while do it in a goroutine. // See https://godoc.org/github.com/libp2p/go-peerstream for more information func (s *Swarm) connHandler(c *ps.Conn) *Conn { - ctx := context.Background() - // this context is for running the handshake, which -- when receiveing connections - // -- we have no bound on beyond what the transport protocol bounds it at. - // note that setup + the handshake are bounded by underlying io. - // (i.e. if TCP or UDP disconnects (or the swarm closes), we're done. - // Q: why not have a shorter handshake? think about an HTTP server on really slow conns. - // as long as the conn is live (TCP says its online), it tries its best. we follow suit.) - - sc, err := s.newConnSetup(ctx, c) + sc, err := wrapConn(c) if err != nil { - log.Debug(err) - log.Event(ctx, "newConnHandlerDisconnect", lgbl.NetConn(c.NetConn()), lgbl.Error(err)) + log.Event(s.Context(), "newConnHandlerDisconnect", lgbl.NetConn(c.NetConn()), lgbl.Error(err)) c.Close() // boom. close it. return nil } - // if a peer dials us, remove from dial backoff. + // Add the public key. + if pk := sc.RemotePublicKey(); pk != nil { + s.peers.AddPubKey(sc.RemotePeer(), pk) + } + + // Add the group + c.AddGroup(sc.RemotePeer()) + + // clear backoff on successful connection. + logdial := lgbl.Dial("swarm", sc.LocalPeer(), sc.RemotePeer(), nil, nil) + log.Event(s.Context(), "swarmDialBackoffClear", logdial) s.backf.Clear(sc.RemotePeer()) return sc From 7f335f0accce3397e188b6256fe33c814740d168 Mon Sep 17 00:00:00 2001 From: Arceliar Date: Tue, 16 Jan 2018 17:36:29 -0600 Subject: [PATCH 0376/3965] use []expiringAddr instead map[string]expiringAddr in addr_manager.go --- p2p/host/peerstore/addr_manager.go | 53 +++++++++++++++++------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index aa913b87a4..96e11e28d0 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -52,13 +52,13 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { return t.After(e.Expires) } -type addrSet map[string]expiringAddr +type addrSlice []expiringAddr // AddrManager manages addresses. // The zero-value is ready to be used. type AddrManager struct { addrmu sync.Mutex // guards addrs - addrs map[peer.ID]addrSet + addrs map[peer.ID]addrSlice addrSubs map[peer.ID][]*addrSub } @@ -67,7 +67,7 @@ type AddrManager struct { // So we can use the zero value. func (mgr *AddrManager) init() { if mgr.addrs == nil { - mgr.addrs = make(map[peer.ID]addrSet) + mgr.addrs = make(map[peer.ID]addrSlice) } if mgr.addrSubs == nil { mgr.addrSubs = make(map[peer.ID][]*addrSub) @@ -108,10 +108,9 @@ func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat // so zero value can be used mgr.init() - amap, found := mgr.addrs[p] - if !found { - amap = make(addrSet) - mgr.addrs[p] = amap + amap := make(map[string]expiringAddr) + for _, ea := range mgr.addrs[p] { + amap[string(ea.Addr.Bytes())] = ea } subs := mgr.addrSubs[p] @@ -134,6 +133,11 @@ func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat } } } + newAddrs := make([]expiringAddr, 0, len(amap)) + for _, ea := range amap { + newAddrs = append(newAddrs, ea) + } + mgr.addrs[p] = newAddrs } // SetAddr calls mgr.SetAddrs(p, addr, ttl) @@ -150,10 +154,9 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat // so zero value can be used mgr.init() - amap, found := mgr.addrs[p] - if !found { - amap = make(addrSet) - mgr.addrs[p] = amap + amap := make(map[string]expiringAddr) + for _, ea := range mgr.addrs[p] { + amap[string(ea.Addr.Bytes())] = ea } subs := mgr.addrSubs[p] @@ -177,6 +180,11 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat delete(amap, addrs) } } + newAddrs := make([]expiringAddr, 0, len(amap)) + for _, ea := range amap { + newAddrs = append(newAddrs, ea) + } + mgr.addrs[p] = newAddrs } // UpdateAddrs updates the addresses associated with the given peer that have @@ -221,21 +229,20 @@ func (mgr *AddrManager) Addrs(p peer.ID) []ma.Multiaddr { now := time.Now() good := make([]ma.Multiaddr, 0, len(maddrs)) - var expired []string - for s, m := range maddrs { - if m.ExpiredBy(now) { - expired = append(expired, s) - } else { + cleaned := make([]expiringAddr, 0, len(maddrs)) + for _, m := range maddrs { + if !m.ExpiredBy(now) { + cleaned = append(cleaned, m) good = append(good, m.Addr) } } // clean up the expired ones. - for _, s := range expired { - delete(maddrs, s) - } - if len(maddrs) == 0 { + mgr.addrs[p] = cleaned + if len(cleaned) == 0 { delete(mgr.addrs, p) + } else { + mgr.addrs[p] = cleaned } return good } @@ -295,9 +302,9 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mul mgr.addrSubs[p] = append(mgr.addrSubs[p], sub) - baseaddrset := mgr.addrs[p] - initial := make([]ma.Multiaddr, 0, len(baseaddrset)) - for _, a := range baseaddrset { + baseaddrslice := mgr.addrs[p] + initial := make([]ma.Multiaddr, 0, len(baseaddrslice)) + for _, a := range baseaddrslice { initial = append(initial, a.Addr) } From 45bb9ebc2f652dc9fc79a735956f210207a94575 Mon Sep 17 00:00:00 2001 From: Arceliar Date: Wed, 17 Jan 2018 08:06:35 -0600 Subject: [PATCH 0377/3965] Preallocate maps and remove extra cleanup line. --- p2p/host/peerstore/addr_manager.go | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 96e11e28d0..6f348f5952 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -108,8 +108,9 @@ func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat // so zero value can be used mgr.init() - amap := make(map[string]expiringAddr) - for _, ea := range mgr.addrs[p] { + oldAddrs := mgr.addrs[p] + amap := make(map[string]expiringAddr, len(oldAddrs)) + for _, ea := range oldAddrs { amap[string(ea.Addr.Bytes())] = ea } @@ -154,8 +155,9 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat // so zero value can be used mgr.init() - amap := make(map[string]expiringAddr) - for _, ea := range mgr.addrs[p] { + oldAddrs := mgr.addrs[p] + amap := make(map[string]expiringAddr, len(oldAddrs)) + for _, ea := range oldAddrs { amap[string(ea.Addr.Bytes())] = ea } @@ -238,7 +240,6 @@ func (mgr *AddrManager) Addrs(p peer.ID) []ma.Multiaddr { } // clean up the expired ones. - mgr.addrs[p] = cleaned if len(cleaned) == 0 { delete(mgr.addrs, p) } else { From 6df75729b55d690a6d5acc54087d60fde49c53e6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 17 Jan 2018 22:07:04 -0800 Subject: [PATCH 0378/3965] initial commit --- p2p/net/conn-security-multistream/ssms.go | 85 +++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 p2p/net/conn-security-multistream/ssms.go diff --git a/p2p/net/conn-security-multistream/ssms.go b/p2p/net/conn-security-multistream/ssms.go new file mode 100644 index 0000000000..2b6b4666b6 --- /dev/null +++ b/p2p/net/conn-security-multistream/ssms.go @@ -0,0 +1,85 @@ +package ssms + +import ( + "context" + "fmt" + "net" + + peer "github.com/libp2p/go-libp2p-peer" + ss "github.com/libp2p/go-stream-security" + mss "github.com/multiformats/go-multistream" +) + +// SSMuxer is a multistream stream security transport multiplexer. +// +// SSMuxer is safe to use without initialization. However, it's not safe to move +// after use. +type SSMuxer struct { + mux mss.MultistreamMuxer + tpts map[string]ss.Transport + OrderPreference []string +} + +// AddTransport adds a stream security transport to this multistream muxer. +// +// This method is *not* thread-safe. It should be called only when initializing +// the SSMuxer. +func (sm *SSMuxer) AddTransport(path string, transport ss.Transport) { + if sm.tpts == nil { + sm.tpts = make(map[string]ss.Transport, 1) + } + + sm.mux.AddHandler(path, nil) + sm.tpts[path] = transport + sm.OrderPreference = append(sm.OrderPreference, path) +} + +// SecureInbound secures an inbound connection using this multistream +// multiplexed stream security transport. +func (sm *SSMuxer) SecureInbound(ctx context.Context, insecure net.Conn) (ss.Conn, error) { + tpt, err := sm.selectProto(ctx, insecure, true) + if err != nil { + return nil, err + } + return tpt.SecureInbound(ctx, insecure) +} + +// SecureOutbound secures an outbound connection using this multistream +// multiplexed stream security transport. +func (sm *SSMuxer) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (ss.Conn, error) { + tpt, err := sm.selectProto(ctx, insecure, false) + if err != nil { + return nil, err + } + return tpt.SecureOutbound(ctx, insecure, p) +} + +func (sm *SSMuxer) selectProto(ctx context.Context, insecure net.Conn, server bool) (ss.Transport, error) { + var proto string + var err error + done := make(chan struct{}) + go func() { + defer close(done) + if server { + proto, _, err = sm.mux.Negotiate(insecure) + } else { + proto, err = mss.SelectOneOf(sm.OrderPreference, insecure) + } + }() + + select { + case <-done: + if err != nil { + return nil, err + } + if tpt, ok := sm.tpts[proto]; ok { + return tpt, nil + } + return nil, fmt.Errorf("selected unknown security transport") + case <-ctx.Done(): + // We *must* do this. We have outstanding work on the connection + // and it's no longer safe to use. + insecure.Close() + return nil, ctx.Err() + } +} From 58590694774217460d1e6e863f8e1f0803b6642b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 18 Jan 2018 11:24:21 -0800 Subject: [PATCH 0379/3965] type assert transport --- p2p/net/conn-security-multistream/ssms.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/net/conn-security-multistream/ssms.go b/p2p/net/conn-security-multistream/ssms.go index 2b6b4666b6..0adbe0f7a8 100644 --- a/p2p/net/conn-security-multistream/ssms.go +++ b/p2p/net/conn-security-multistream/ssms.go @@ -20,6 +20,8 @@ type SSMuxer struct { OrderPreference []string } +var _ ss.Transport = (*SSMuxer)(nil) + // AddTransport adds a stream security transport to this multistream muxer. // // This method is *not* thread-safe. It should be called only when initializing From cae12315981846556c42fa3b64d7ceaa2db0f6e1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 18 Jan 2018 11:45:08 -0800 Subject: [PATCH 0380/3965] add tests --- .../conn-security-multistream/ssms_test.go | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 p2p/net/conn-security-multistream/ssms_test.go diff --git a/p2p/net/conn-security-multistream/ssms_test.go b/p2p/net/conn-security-multistream/ssms_test.go new file mode 100644 index 0000000000..faa1c87dfd --- /dev/null +++ b/p2p/net/conn-security-multistream/ssms_test.go @@ -0,0 +1,55 @@ +package ssms + +import ( + "context" + "net" + "sync" + "testing" + + ss "github.com/libp2p/go-stream-security" + sst "github.com/libp2p/go-stream-security/test" +) + +func TestCommonProto(t *testing.T) { + var at, bt SSMuxer + atInsecure := ss.InsecureTransport("peerA") + btInsecure := ss.InsecureTransport("peerB") + at.AddTransport("/plaintext/1.0.0", &atInsecure) + bt.AddTransport("/plaintext/1.1.0", &btInsecure) + bt.AddTransport("/plaintext/1.0.0", &btInsecure) + sst.SubtestRW(t, &at, &bt, "peerA", "peerB") +} + +func TestNoCommonProto(t *testing.T) { + var at, bt SSMuxer + atInsecure := ss.InsecureTransport("peerA") + btInsecure := ss.InsecureTransport("peerB") + + at.AddTransport("/plaintext/1.0.0", &atInsecure) + bt.AddTransport("/plaintext/1.1.0", &btInsecure) + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + a, b := net.Pipe() + + var wg sync.WaitGroup + wg.Add(2) + go func() { + defer wg.Done() + defer a.Close() + _, err := at.SecureInbound(ctx, a) + if err == nil { + t.Fatal("conection should have failed") + } + }() + + go func() { + defer wg.Done() + defer b.Close() + _, err := bt.SecureOutbound(ctx, b, "peerA") + if err == nil { + t.Fatal("connection should have failed") + } + }() + wg.Wait() +} From 4a9cd1b556f089fec212e6915efa5d67a09f2845 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 18 Jan 2018 22:05:52 -0800 Subject: [PATCH 0381/3965] initial commit --- p2p/net/reuseport/dial.go | 102 ++++++++++++++++++++++++++++ p2p/net/reuseport/listen.go | 72 ++++++++++++++++++++ p2p/net/reuseport/multidialer.go | 88 ++++++++++++++++++++++++ p2p/net/reuseport/reuseport.go | 63 +++++++++++++++++ p2p/net/reuseport/reuseport_test.go | 48 +++++++++++++ p2p/net/reuseport/singledialer.go | 16 +++++ p2p/net/reuseport/transport.go | 25 +++++++ 7 files changed, 414 insertions(+) create mode 100644 p2p/net/reuseport/dial.go create mode 100644 p2p/net/reuseport/listen.go create mode 100644 p2p/net/reuseport/multidialer.go create mode 100644 p2p/net/reuseport/reuseport.go create mode 100644 p2p/net/reuseport/reuseport_test.go create mode 100644 p2p/net/reuseport/singledialer.go create mode 100644 p2p/net/reuseport/transport.go diff --git a/p2p/net/reuseport/dial.go b/p2p/net/reuseport/dial.go new file mode 100644 index 0000000000..c599667502 --- /dev/null +++ b/p2p/net/reuseport/dial.go @@ -0,0 +1,102 @@ +package tcpreuse + +import ( + "context" + "net" + + reuseport "github.com/libp2p/go-reuseport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +type dialer interface { + Dial(network, addr string) (net.Conn, error) + DialContext(ctx context.Context, network, addr string) (net.Conn, error) +} + +func (t *Transport) Dial(raddr ma.Multiaddr) (manet.Conn, error) { + return t.DialContext(context.Background(), raddr) +} + +func (t *Transport) DialContext(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { + network, addr, err := manet.DialArgs(raddr) + if err != nil { + return nil, err + } + var d dialer + switch network { + case "tcp4": + d = t.v4.getDialer(network) + case "tcp6": + d = t.v6.getDialer(network) + default: + return nil, ErrWrongProto + } + conn, err := d.DialContext(ctx, network, addr) + if err != nil { + return nil, err + } + maconn, err := manet.WrapNetConn(conn) + if err != nil { + conn.Close() + return nil, err + } + return maconn, nil +} + +func (n *network) getDialer(network string) dialer { + n.mu.RLock() + d := n.dialer + n.mu.RUnlock() + if d == nil { + n.mu.Lock() + defer n.mu.Unlock() + + if n.dialer == nil { + n.dialer = n.makeDialer(network) + } + d = n.dialer + } + return d +} + +func (n *network) makeDialer(network string) dialer { + if !reuseport.Available() { + log.Debug("reuseport not available") + return &net.Dialer{} + } + + var unspec net.IP + switch network { + case "tcp4": + unspec = net.IPv4zero + case "tcp6": + unspec = net.IPv6unspecified + default: + panic("invalid network: must be either tcp4 or tcp6") + } + + // How many ports are we listening on. + var port = 0 + for l := range n.listeners { + if port == 0 { + port = l.Addr().(*net.TCPAddr).Port + } else { + // > 1 + return newMultiDialer(unspec, n.listeners) + } + } + + // None. + if port == 0 { + return &net.Dialer{} + } + + // One. Always dial from the single port we're listening on. + laddr := &net.TCPAddr{ + IP: unspec, + Port: port, + } + + return (*singleDialer)(laddr) +} diff --git a/p2p/net/reuseport/listen.go b/p2p/net/reuseport/listen.go new file mode 100644 index 0000000000..483b33eec6 --- /dev/null +++ b/p2p/net/reuseport/listen.go @@ -0,0 +1,72 @@ +package tcpreuse + +import ( + "net" + + reuseport "github.com/libp2p/go-reuseport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +type listener struct { + manet.Listener + network *network +} + +func (l *listener) Close() error { + l.network.mu.Lock() + delete(l.network.listeners, l) + l.network.mu.Unlock() + return l.Listener.Close() +} + +func (t *Transport) Listen(laddr ma.Multiaddr) (manet.Listener, error) { + nw, naddr, err := manet.DialArgs(laddr) + if err != nil { + return nil, err + } + var n *network + switch nw { + case "tcp4": + n = &t.v4 + case "tcp6": + n = &t.v6 + default: + return nil, ErrWrongProto + } + + if !reuseport.Available() { + return manet.Listen(laddr) + } + nl, err := reuseport.Listen(nw, naddr) + if err != nil { + return manet.Listen(laddr) + } + + if _, ok := nl.Addr().(*net.TCPAddr); !ok { + nl.Close() + return nil, ErrWrongProto + } + + malist, err := manet.WrapNetListener(nl) + if err != nil { + nl.Close() + return nil, err + } + + list := &listener{ + Listener: malist, + network: n, + } + + n.mu.Lock() + defer n.mu.Unlock() + + if n.listeners == nil { + n.listeners = make(map[*listener]struct{}) + } + n.listeners[list] = struct{}{} + n.dialer = nil + + return list, nil +} diff --git a/p2p/net/reuseport/multidialer.go b/p2p/net/reuseport/multidialer.go new file mode 100644 index 0000000000..b8f4129f14 --- /dev/null +++ b/p2p/net/reuseport/multidialer.go @@ -0,0 +1,88 @@ +package tcpreuse + +import ( + "context" + "fmt" + "math/rand" + "net" +) + +type multiDialer struct { + loopback []*net.TCPAddr + unspecified []*net.TCPAddr + global *net.TCPAddr +} + +func (d *multiDialer) Dial(network, addr string) (net.Conn, error) { + return d.DialContext(context.Background(), network, addr) +} + +func randAddr(addrs []*net.TCPAddr) *net.TCPAddr { + if len(addrs) > 0 { + return addrs[rand.Intn(len(addrs))] + } + return nil +} + +func (d *multiDialer) DialContext(ctx context.Context, network, addr string) (net.Conn, error) { + tcpAddr, err := net.ResolveTCPAddr(network, addr) + if err != nil { + return nil, err + } + + ip := tcpAddr.IP + source := d.global + switch { + case ip.IsLoopback(): + switch { + case len(d.loopback) > 0: + source = randAddr(d.loopback) + case len(d.unspecified) > 0: + source = randAddr(d.unspecified) + } + case ip.IsGlobalUnicast(): + switch { + case len(d.unspecified) > 0: + source = randAddr(d.unspecified) + } + default: + return nil, fmt.Errorf("undialable IP: %s", tcpAddr.IP) + } + return reuseDial(ctx, source, network, addr) +} + +func newMultiDialer(unspec net.IP, listeners map[*listener]struct{}) dialer { + m := new(multiDialer) + for l := range listeners { + laddr := l.Addr().(*net.TCPAddr) + switch { + case laddr.IP.IsLoopback(): + m.loopback = append(m.loopback, laddr) + case laddr.IP.IsGlobalUnicast(): + // Different global ports? Crap. + // + // The *proper* way to deal with this is to, e.g., use + // netlink to figure out which source address we would + // normally use to dial a destination address and then + // pick one of the ports we're listening on on that + // source address. However, this is a pain in the ass. + // + // Instead, we're just going to always dial from the + // unspecified address with the first global port we + // find. + // + // TODO: Port priority? Addr priority? + if m.global != nil { + m.global = &net.TCPAddr{ + IP: unspec, + Port: laddr.Port, + } + } else { + log.Warning("listening on external interfaces on multiple ports, will dial from %d, not %s", m.global, laddr) + } + case laddr.IP.IsUnspecified(): + m.unspecified = append(m.unspecified, laddr) + } + } + return m +} diff --git a/p2p/net/reuseport/reuseport.go b/p2p/net/reuseport/reuseport.go new file mode 100644 index 0000000000..c85f1180b9 --- /dev/null +++ b/p2p/net/reuseport/reuseport.go @@ -0,0 +1,63 @@ +package tcpreuse + +import ( + "context" + "net" + "syscall" + + reuseport "github.com/libp2p/go-reuseport" +) + +// ReuseErrShouldRetry diagnoses whether to retry after a reuse error. +// if we failed to bind, we should retry. if bind worked and this is a +// real dial error (remote end didnt answer) then we should not retry. +func ReuseErrShouldRetry(err error) bool { + if err == nil { + return false // hey, it worked! no need to retry. + } + + // if it's a network timeout error, it's a legitimate failure. + if nerr, ok := err.(net.Error); ok && nerr.Timeout() { + return false + } + + errno, ok := err.(syscall.Errno) + if !ok { // not an errno? who knows what this is. retry. + return true + } + + switch errno { + case syscall.EADDRINUSE, syscall.EADDRNOTAVAIL: + return true // failure to bind. retry. + case syscall.ECONNREFUSED: + return false // real dial error + default: + return true // optimistically default to retry. + } +} + +// Dials using reusport and then redials normally if that fails. +func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) (net.Conn, error) { + if laddr == nil { + var d net.Dialer + return d.DialContext(ctx, network, raddr) + } + + d := reuseport.Dialer{ + D: net.Dialer{ + LocalAddr: laddr, + }, + } + + con, err := d.DialContext(ctx, network, raddr) + if err != nil { + return con, err + } + + if ReuseErrShouldRetry(err) && ctx.Err() == nil { + log.Debug("failed to reuse port, dialing with a random port") + var d net.Dialer + con, err = d.DialContext(ctx, network, raddr) + } + return con, err +} diff --git a/p2p/net/reuseport/reuseport_test.go b/p2p/net/reuseport/reuseport_test.go new file mode 100644 index 0000000000..a4d8b1e2b6 --- /dev/null +++ b/p2p/net/reuseport/reuseport_test.go @@ -0,0 +1,48 @@ +package tcpreuse + +import ( + "net" + "syscall" + "testing" +) + +type netTimeoutErr struct { + timeout bool +} + +func (e netTimeoutErr) Error() string { + return "" +} + +func (e netTimeoutErr) Timeout() bool { + return e.timeout +} + +func (e netTimeoutErr) Temporary() bool { + panic("not checked") +} + +func TestReuseError(t *testing.T) { + var nte1 net.Error = &netTimeoutErr{true} + var nte2 net.Error = &netTimeoutErr{false} + + cases := map[error]bool{ + nil: false, + syscall.EADDRINUSE: true, + syscall.EADDRNOTAVAIL: true, + syscall.ECONNREFUSED: false, + + nte1: false, + nte2: true, // this ones a little weird... we should check neterror.Temporary() too + + // test 'default' to true + syscall.EBUSY: true, + } + + for k, v := range cases { + if ReuseErrShouldRetry(k) != v { + t.Fatalf("expected %t for %#v", v, k) + } + } + +} diff --git a/p2p/net/reuseport/singledialer.go b/p2p/net/reuseport/singledialer.go new file mode 100644 index 0000000000..efb96eb1ae --- /dev/null +++ b/p2p/net/reuseport/singledialer.go @@ -0,0 +1,16 @@ +package tcpreuse + +import ( + "context" + "net" +) + +type singleDialer net.TCPAddr + +func (d *singleDialer) Dial(network, address string) (net.Conn, error) { + return d.DialContext(context.Background(), network, address) +} + +func (d *singleDialer) DialContext(ctx context.Context, network, address string) (net.Conn, error) { + return reuseDial(ctx, (*net.TCPAddr)(d), network, address) +} diff --git a/p2p/net/reuseport/transport.go b/p2p/net/reuseport/transport.go new file mode 100644 index 0000000000..5f094d1a12 --- /dev/null +++ b/p2p/net/reuseport/transport.go @@ -0,0 +1,25 @@ +package tcpreuse + +import ( + "errors" + "sync" + + logging "github.com/ipfs/go-log" +) + +var log = logging.Logger("reuseport-transport") + +// ErrWrongProto is returned when dialing a protocol other than tcp. +var ErrWrongProto = errors.New("can only dial TCP over IPv4 or IPv6") + +// Transport is a TCP reuse transport that reuses listener ports. +type Transport struct { + v4 network + v6 network +} + +type network struct { + mu sync.RWMutex + listeners map[*listener]struct{} + dialer dialer +} From ad8c7eec539ccf0776e5d573dcd267936ee9e876 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 19 Jan 2018 14:45:26 -0800 Subject: [PATCH 0382/3965] add some documentation --- p2p/net/reuseport/dial.go | 8 ++++++++ p2p/net/reuseport/listen.go | 7 +++++++ 2 files changed, 15 insertions(+) diff --git a/p2p/net/reuseport/dial.go b/p2p/net/reuseport/dial.go index c599667502..0fa6e99b84 100644 --- a/p2p/net/reuseport/dial.go +++ b/p2p/net/reuseport/dial.go @@ -14,10 +14,18 @@ type dialer interface { DialContext(ctx context.Context, network, addr string) (net.Conn, error) } +// Dial dials the given multiaddr, reusing ports we're currently listening on if +// possible. +// +// Dial attempts to be smart about choosing the source port. For example, If +// we're dialing a loopback address and we're listening on one or more loopback +// ports, Dial will randomly choose one of the loopback ports and addresses and +// reuse it. func (t *Transport) Dial(raddr ma.Multiaddr) (manet.Conn, error) { return t.DialContext(context.Background(), raddr) } +// DialContext is like Dial but takes a context. func (t *Transport) DialContext(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { network, addr, err := manet.DialArgs(raddr) if err != nil { diff --git a/p2p/net/reuseport/listen.go b/p2p/net/reuseport/listen.go index 483b33eec6..04bd01b569 100644 --- a/p2p/net/reuseport/listen.go +++ b/p2p/net/reuseport/listen.go @@ -20,6 +20,13 @@ func (l *listener) Close() error { return l.Listener.Close() } +// Listen listens on the given multiaddr. +// +// If reuseport is supported, it will be enabled for this listener and future +// dials from this transport may reuse the port. +// +// Note: You can listen on the same multiaddr as many times as you want +// (although only *one* listener will end up handling the inbound connection). func (t *Transport) Listen(laddr ma.Multiaddr) (manet.Listener, error) { nw, naddr, err := manet.DialArgs(laddr) if err != nil { From 154ae5c47c25c1e2ebea8ba4aa2e3d10bee044f6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 19 Jan 2018 14:45:33 -0800 Subject: [PATCH 0383/3965] add minimal test --- p2p/net/reuseport/transport_test.go | 40 +++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 p2p/net/reuseport/transport_test.go diff --git a/p2p/net/reuseport/transport_test.go b/p2p/net/reuseport/transport_test.go new file mode 100644 index 0000000000..1b3941f448 --- /dev/null +++ b/p2p/net/reuseport/transport_test.go @@ -0,0 +1,40 @@ +package tcpreuse + +import ( + "testing" + + ma "github.com/multiformats/go-multiaddr" +) + +func TestAll(t *testing.T) { + var trA Transport + var trB Transport + laddr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") + listenerA, err := trA.Listen(laddr) + if err != nil { + t.Fatal(err) + } + defer listenerA.Close() + listenerB, err := trB.Listen(laddr) + if err != nil { + t.Fatal(err) + } + defer listenerB.Close() + + done := make(chan struct{}) + go func() { + defer close(done) + c, err := listenerA.Accept() + if err != nil { + t.Fatal(err) + } + c.Close() + }() + + c, err := trB.Dial(listenerA.Multiaddr()) + if err != nil { + t.Fatal(err) + } + <-done + c.Close() +} From ede46e0eeafad55421952ef4c594c28df36105c9 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 19 Jan 2018 14:56:27 -0800 Subject: [PATCH 0384/3965] initial commit --- p2p/net/upgrader/conn.go | 17 +++++ p2p/net/upgrader/listener.go | 142 +++++++++++++++++++++++++++++++++++ p2p/net/upgrader/upgrader.go | 120 +++++++++++++++++++++++++++++ 3 files changed, 279 insertions(+) create mode 100644 p2p/net/upgrader/conn.go create mode 100644 p2p/net/upgrader/listener.go create mode 100644 p2p/net/upgrader/upgrader.go diff --git a/p2p/net/upgrader/conn.go b/p2p/net/upgrader/conn.go new file mode 100644 index 0000000000..a9f5f36bd7 --- /dev/null +++ b/p2p/net/upgrader/conn.go @@ -0,0 +1,17 @@ +package stream + +import ( + transport "github.com/libp2p/go-libp2p-transport" + smux "github.com/libp2p/go-stream-muxer" +) + +type transportConn struct { + smux.Conn + transport.ConnMultiaddrs + transport.ConnSecurity + transport transport.Transport +} + +func (t *transportConn) Transport() transport.Transport { + return t.transport +} diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go new file mode 100644 index 0000000000..cc8e16743c --- /dev/null +++ b/p2p/net/upgrader/listener.go @@ -0,0 +1,142 @@ +package stream + +import ( + "context" + "fmt" + "sync" + + logging "github.com/ipfs/go-log" + tec "github.com/jbenet/go-temp-err-catcher" + transport "github.com/libp2p/go-libp2p-transport" + manet "github.com/multiformats/go-multiaddr-net" +) + +var log = logging.Logger("stream-upgrader") + +type connErr struct { + conn transport.Conn + err error +} + +type listener struct { + manet.Listener + + transport transport.Transport + upgrader *Upgrader + + incoming chan transport.Conn + err error + + ticket chan struct{} + + ctx context.Context + cancel func() +} + +// Close closes the listener. +func (l *listener) Close() error { + // Do this first to try to get any relevent errors. + err := l.Listener.Close() + + l.cancel() + // Drain and wait. + for c := range l.incoming { + c.Close() + } + return err +} + +func (l *listener) handleIncoming() { + var wg sync.WaitGroup + defer func() { + // make sure we're closed + l.Listener.Close() + if l.err == nil { + l.err = fmt.Errorf("listener closed") + } + + wg.Wait() + close(l.incoming) + }() + + var catcher tec.TempErrCatcher + for { + + select { + case <-l.ticket: + case <-l.ctx.Done(): + return + } + + maconn, err := l.Listener.Accept() + if err != nil { + if catcher.IsTemporary(err) { + log.Infof("temporary accept error: %s", err) + continue + } + l.err = err + return + } + + log.Debugf("listener %s got connection: %s <---> %s", + l, + maconn.LocalMultiaddr(), + maconn.RemoteMultiaddr()) + + wg.Add(1) + go func() { + defer wg.Done() + + ctx, cancel := context.WithTimeout(l.ctx, transport.AcceptTimeout) + defer cancel() + + conn, err := l.upgrader.UpgradeInbound(ctx, l.transport, maconn) + if err != nil { + // Don't bother bubbling this up. We just failed + // to completely negotiate the connection. + log.Debugf("accept upgrade error: %s (%s <--> %s)", + err, + conn.LocalMultiaddr(), + conn.RemoteMultiaddr()) + return + } + + log.Debugf("listener %s accepted connection: %s (%s) <---> %s (%s)", + l, + conn.LocalMultiaddr(), + conn.LocalPeer(), + conn.RemoteMultiaddr(), + conn.RemotePeer()) + + select { + case l.incoming <- conn: + case <-ctx.Done(): + if l.ctx.Err() == nil { + // Listener *not* closed but the accept timeout expired. + log.Warningf("listener dropped connection due to slow accept") + } + // Wait on the context with a timeout. This way, + // if we stop accepting connections for some reason, + // we'll eventually close all the open ones + // instead of hanging onto them. + conn.Close() + } + }() + } +} + +// Accept accepts a connection. +func (l *listener) Accept() (transport.Conn, error) { + for { + select { + case l.ticket <- struct{}{}: + case c, ok := <-l.incoming: + if !ok { + return nil, l.err + } + return c, nil + } + } +} + +var _ transport.Listener = (*listener)(nil) diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go new file mode 100644 index 0000000000..3da10bf798 --- /dev/null +++ b/p2p/net/upgrader/upgrader.go @@ -0,0 +1,120 @@ +package stream + +import ( + "context" + "errors" + "fmt" + "net" + + pnet "github.com/libp2p/go-libp2p-interface-pnet" + peer "github.com/libp2p/go-libp2p-peer" + transport "github.com/libp2p/go-libp2p-transport" + filter "github.com/libp2p/go-maddr-filter" + smux "github.com/libp2p/go-stream-muxer" + ss "github.com/libp2p/go-stream-security" + manet "github.com/multiformats/go-multiaddr-net" +) + +// ErrNilPeer is returned when attempting to upgrade an outbound connection +// without specifying a peer ID. +var ErrNilPeer = errors.New("nil peer") + +// Upgrader is a multistream upgrader that can upgrade an underlying connection +// to a full transport connection (secure and multiplexed). +type Upgrader struct { + Protector pnet.Protector + Secure ss.Transport + Muxer smux.Transport + Filters *filter.Filters +} + +// UpgradeListener upgrades the passed multiaddr-net listener into a full libp2p-transport listener. +func (u *Upgrader) UpgradeListener(t transport.Transport, list manet.Listener) transport.Listener { + ctx, cancel := context.WithCancel(context.Background()) + l := &listener{ + Listener: list, + upgrader: u, + transport: t, + ticket: make(chan struct{}), + incoming: make(chan transport.Conn), + cancel: cancel, + ctx: ctx, + } + go l.handleIncoming() + return l +} + +// UpgradeOutbound upgrades the given outbound multiaddr-net connection into a +// full libp2p-transport connection. +func (u *Upgrader) UpgradeOutbound(ctx context.Context, t transport.Transport, maconn manet.Conn, p peer.ID) (transport.Conn, error) { + if p == "" { + return nil, ErrNilPeer + } + return u.upgrade(ctx, t, maconn, p) +} + +// UpgradeInbound upgrades the given inbound multiaddr-net connection into a +// full libp2p-transport connection. +func (u *Upgrader) UpgradeInbound(ctx context.Context, t transport.Transport, maconn manet.Conn) (transport.Conn, error) { + return u.upgrade(ctx, t, maconn, "") +} + +func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn manet.Conn, p peer.ID) (transport.Conn, error) { + if u.Filters != nil && u.Filters.AddrBlocked(maconn.RemoteMultiaddr()) { + log.Debugf("blocked connection from %s", maconn.RemoteMultiaddr()) + maconn.Close() + return nil, fmt.Errorf("blocked connection from %s", maconn.RemoteMultiaddr()) + } + + var conn net.Conn = maconn + if u.Protector != nil { + pconn, err := u.Protector.Protect(conn) + if err != nil { + conn.Close() + return nil, err + } + conn = pconn + } + sconn, err := u.setupSecurity(ctx, conn, p) + if err != nil { + conn.Close() + return nil, err + } + smconn, err := u.setupMuxer(ctx, sconn, p) + if err != nil { + conn.Close() + return nil, err + } + return &transportConn{ + Conn: smconn, + ConnMultiaddrs: maconn, + ConnSecurity: sconn, + transport: t, + }, nil +} + +func (u *Upgrader) setupSecurity(ctx context.Context, conn net.Conn, p peer.ID) (ss.Conn, error) { + if p == "" { + return u.Secure.SecureInbound(ctx, conn) + } + return u.Secure.SecureOutbound(ctx, conn, p) +} + +func (u *Upgrader) setupMuxer(ctx context.Context, conn net.Conn, p peer.ID) (smux.Conn, error) { + // TODO: The muxer should take a context. + done := make(chan struct{}) + + var smconn smux.Conn + var err error + go func() { + defer close(done) + smconn, err = u.Muxer.NewConn(conn, p == "") + }() + + select { + case <-done: + return smconn, err + case <-ctx.Done(): + return nil, ctx.Err() + } +} From 48221760c6128bcb1e86de789fb6b0ccba823f8e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 19 Jan 2018 16:27:04 -0800 Subject: [PATCH 0385/3965] fix nil dereference --- p2p/net/upgrader/listener.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index cc8e16743c..3c7408955e 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -96,8 +96,8 @@ func (l *listener) handleIncoming() { // to completely negotiate the connection. log.Debugf("accept upgrade error: %s (%s <--> %s)", err, - conn.LocalMultiaddr(), - conn.RemoteMultiaddr()) + maconn.LocalMultiaddr(), + maconn.RemoteMultiaddr()) return } From cf38072bb6220b471503a4fd627568007e7bc21f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 19 Jan 2018 16:48:12 -0800 Subject: [PATCH 0386/3965] nicer stringers --- p2p/net/upgrader/conn.go | 17 +++++++++++++++++ p2p/net/upgrader/listener.go | 14 ++++++++------ 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/p2p/net/upgrader/conn.go b/p2p/net/upgrader/conn.go index a9f5f36bd7..50d5a5560b 100644 --- a/p2p/net/upgrader/conn.go +++ b/p2p/net/upgrader/conn.go @@ -1,6 +1,8 @@ package stream import ( + "fmt" + transport "github.com/libp2p/go-libp2p-transport" smux "github.com/libp2p/go-stream-muxer" ) @@ -15,3 +17,18 @@ type transportConn struct { func (t *transportConn) Transport() transport.Transport { return t.transport } + +func (t *transportConn) String() string { + ts := "" + if s, ok := t.transport.(fmt.Stringer); ok { + ts = "[" + s.String() + "]" + } + return fmt.Sprintf( + " %s (%s)>", + ts, + t.LocalMultiaddr(), + t.LocalPeer(), + t.RemoteMultiaddr(), + t.RemotePeer(), + ) +} diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index 3c7408955e..c1d73ae570 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -101,12 +101,7 @@ func (l *listener) handleIncoming() { return } - log.Debugf("listener %s accepted connection: %s (%s) <---> %s (%s)", - l, - conn.LocalMultiaddr(), - conn.LocalPeer(), - conn.RemoteMultiaddr(), - conn.RemotePeer()) + log.Debugf("listener %s accepted connection: %s", l, conn) select { case l.incoming <- conn: @@ -139,4 +134,11 @@ func (l *listener) Accept() (transport.Conn, error) { } } +func (l *listener) String() string { + if s, ok := l.transport.(fmt.Stringer); ok { + return fmt.Sprintf("", s, l.Multiaddr()) + } + return fmt.Sprintf("", l.Multiaddr()) +} + var _ transport.Listener = (*listener)(nil) From 61cd627405e18bcd591d7861be051683265b612e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 22 Jan 2018 00:05:13 -0800 Subject: [PATCH 0387/3965] keep accepting and negotiating connections until we have 16 ready and queued up Otherwise, we'll have an annoying saw-tooth pattern where we'll accept a bunch of connections, set them up in parallel, and then wait until we pass them all back up to the swarm before accepting any more. This commit allows us to queue up 16 ready connections before ceasing to accept any new connections. We'll still probably have a saw-tooth pattern under heavy load but it should be less pronounced (and we can improve the situation by upping the queue size. --- p2p/net/upgrader/listener.go | 30 +++++++++---------- p2p/net/upgrader/threshold.go | 56 +++++++++++++++++++++++++++++++++++ p2p/net/upgrader/upgrader.go | 5 +++- 3 files changed, 74 insertions(+), 17 deletions(-) create mode 100644 p2p/net/upgrader/threshold.go diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index c1d73ae570..72d129ced1 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -27,8 +27,11 @@ type listener struct { incoming chan transport.Conn err error - ticket chan struct{} + // Used for backpressure + threshold *threshold + // Canceling this context isn't sufficient to tear down the listener. + // Call close. ctx context.Context cancel func() } @@ -61,12 +64,10 @@ func (l *listener) handleIncoming() { var catcher tec.TempErrCatcher for { - - select { - case <-l.ticket: - case <-l.ctx.Done(): - return - } + // no need to wait on the context. + // Close will drain the incoming channel which will cause the + // threshold to drop. + l.threshold.Wait() maconn, err := l.Listener.Accept() if err != nil { @@ -103,6 +104,9 @@ func (l *listener) handleIncoming() { log.Debugf("listener %s accepted connection: %s", l, conn) + l.threshold.Acquire() + defer l.threshold.Release() + select { case l.incoming <- conn: case <-ctx.Done(): @@ -122,16 +126,10 @@ func (l *listener) handleIncoming() { // Accept accepts a connection. func (l *listener) Accept() (transport.Conn, error) { - for { - select { - case l.ticket <- struct{}{}: - case c, ok := <-l.incoming: - if !ok { - return nil, l.err - } - return c, nil - } + if c, ok := <-l.incoming; ok { + return c, nil } + return nil, l.err } func (l *listener) String() string { diff --git a/p2p/net/upgrader/threshold.go b/p2p/net/upgrader/threshold.go new file mode 100644 index 0000000000..091e35c075 --- /dev/null +++ b/p2p/net/upgrader/threshold.go @@ -0,0 +1,56 @@ +package stream + +import ( + "sync" +) + +func newThreshold(cutoff int) *threshold { + t := &threshold{ + threshold: cutoff, + } + t.cond.L = &t.mu + return t +} + +type threshold struct { + mu sync.Mutex + cond sync.Cond + + count int + threshold int +} + +func (t *threshold) Count() int { + t.mu.Lock() + defer t.mu.Unlock() + return t.count +} + +// Acquire increments the counter. It will not block. +func (t *threshold) Acquire() { + t.mu.Lock() + t.count++ + t.mu.Unlock() +} + +// Release decrements the counter. +func (t *threshold) Release() { + t.mu.Lock() + if t.count == 0 { + panic("negative count") + } + if t.threshold == t.count { + t.cond.Broadcast() + } + t.count-- + t.mu.Unlock() +} + +// Wait waits for the counter to drop below the threshold +func (t *threshold) Wait() { + t.mu.Lock() + for t.count > t.threshold { + t.cond.Wait() + } + t.mu.Unlock() +} diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index 3da10bf798..c494ea2374 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -19,6 +19,9 @@ import ( // without specifying a peer ID. var ErrNilPeer = errors.New("nil peer") +// AcceptQueueLength is the number of connections to fully setup before not accepting any new connections +var AcceptQueueLength = 16 + // Upgrader is a multistream upgrader that can upgrade an underlying connection // to a full transport connection (secure and multiplexed). type Upgrader struct { @@ -35,7 +38,7 @@ func (u *Upgrader) UpgradeListener(t transport.Transport, list manet.Listener) t Listener: list, upgrader: u, transport: t, - ticket: make(chan struct{}), + threshold: newThreshold(AcceptQueueLength), incoming: make(chan transport.Conn), cancel: cancel, ctx: ctx, From a950fa7dafb8b761489e32a2ae99f657815b38bf Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 22 Jan 2018 09:16:28 -0800 Subject: [PATCH 0388/3965] fix comment on why we don't need to wait on the context Also, add a context canceled check to the loop just in case (accept could, if buggy, return temporary errors after it's closed). --- p2p/net/upgrader/listener.go | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index 72d129ced1..c74772f90b 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -63,12 +63,7 @@ func (l *listener) handleIncoming() { }() var catcher tec.TempErrCatcher - for { - // no need to wait on the context. - // Close will drain the incoming channel which will cause the - // threshold to drop. - l.threshold.Wait() - + for l.ctx.Err() == nil { maconn, err := l.Listener.Accept() if err != nil { if catcher.IsTemporary(err) { @@ -121,6 +116,10 @@ func (l *listener) handleIncoming() { conn.Close() } }() + + // The go routine above calls Release when the context is + // canceled so there's no need to wait on it here. + l.threshold.Wait() } } From cec0d9ff0cae84a411593f68f6f74dbb39e6b269 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 22 Jan 2018 09:21:46 -0800 Subject: [PATCH 0389/3965] remove unused count function from threshold --- p2p/net/upgrader/threshold.go | 6 ------ 1 file changed, 6 deletions(-) diff --git a/p2p/net/upgrader/threshold.go b/p2p/net/upgrader/threshold.go index 091e35c075..8185e3a53e 100644 --- a/p2p/net/upgrader/threshold.go +++ b/p2p/net/upgrader/threshold.go @@ -20,12 +20,6 @@ type threshold struct { threshold int } -func (t *threshold) Count() int { - t.mu.Lock() - defer t.mu.Unlock() - return t.count -} - // Acquire increments the counter. It will not block. func (t *threshold) Acquire() { t.mu.Lock() From fc057b07c10eabe7425c2293c23602d33ec6d735 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 22 Jan 2018 10:54:31 -0800 Subject: [PATCH 0390/3965] handle all possible port configurations when creating dialers --- p2p/net/reuseport/dial.go | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/p2p/net/reuseport/dial.go b/p2p/net/reuseport/dial.go index 0fa6e99b84..a16b5b2b75 100644 --- a/p2p/net/reuseport/dial.go +++ b/p2p/net/reuseport/dial.go @@ -87,10 +87,13 @@ func (n *network) makeDialer(network string) dialer { // How many ports are we listening on. var port = 0 for l := range n.listeners { - if port == 0 { - port = l.Addr().(*net.TCPAddr).Port - } else { - // > 1 + newPort := l.Addr().(*net.TCPAddr).Port + switch { + case newPort == 0: // Any port, ignore (really, we shouldn't get this case...). + case port == 0: // Haven't selected a port yet, choose this one. + port = newPort + case newPort == port: // Same as the selected port, continue... + default: // Multiple ports, use the multi dialer return newMultiDialer(unspec, n.listeners) } } From 1ad436a818c4046024325f9e7ea5ac3f0e0a437c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 22 Jan 2018 10:55:09 -0800 Subject: [PATCH 0391/3965] fix incorrect condition in reuseDial --- p2p/net/reuseport/reuseport.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/reuseport/reuseport.go b/p2p/net/reuseport/reuseport.go index c85f1180b9..307f4ae4d1 100644 --- a/p2p/net/reuseport/reuseport.go +++ b/p2p/net/reuseport/reuseport.go @@ -50,12 +50,12 @@ func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) ( } con, err := d.DialContext(ctx, network, raddr) - if err != nil { - return con, err + if err == nil { + return con, nil } if ReuseErrShouldRetry(err) && ctx.Err() == nil { - log.Debug("failed to reuse port, dialing with a random port") + log.Debugf("failed to reuse port, dialing with a random port: %s", err) var d net.Dialer con, err = d.DialContext(ctx, network, raddr) } From be4a5f410f36edf99d92151afa47a8e48f954c1d Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 23 Jan 2018 23:31:15 +1100 Subject: [PATCH 0392/3965] fix off-by-one error in the threshold --- p2p/net/upgrader/threshold.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/upgrader/threshold.go b/p2p/net/upgrader/threshold.go index 8185e3a53e..2c278bd398 100644 --- a/p2p/net/upgrader/threshold.go +++ b/p2p/net/upgrader/threshold.go @@ -43,7 +43,7 @@ func (t *threshold) Release() { // Wait waits for the counter to drop below the threshold func (t *threshold) Wait() { t.mu.Lock() - for t.count > t.threshold { + for t.count >= t.threshold { t.cond.Wait() } t.mu.Unlock() From 90789da3cb30fce78ebd3f692b5cc4defd7ece6e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 23 Jan 2018 23:41:58 +1100 Subject: [PATCH 0393/3965] add tests for the listener --- .../go_libp2p_stream_transport_suite_test.go | 13 + p2p/net/upgrader/listener_test.go | 254 ++++++++++++++++++ 2 files changed, 267 insertions(+) create mode 100644 p2p/net/upgrader/go_libp2p_stream_transport_suite_test.go create mode 100644 p2p/net/upgrader/listener_test.go diff --git a/p2p/net/upgrader/go_libp2p_stream_transport_suite_test.go b/p2p/net/upgrader/go_libp2p_stream_transport_suite_test.go new file mode 100644 index 0000000000..1a40574419 --- /dev/null +++ b/p2p/net/upgrader/go_libp2p_stream_transport_suite_test.go @@ -0,0 +1,13 @@ +package stream_test + +import ( + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" + + "testing" +) + +func TestGoLibp2pStreamTransport(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "GoLibp2pStreamTransport Suite") +} diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go new file mode 100644 index 0000000000..c2eaf6c72b --- /dev/null +++ b/p2p/net/upgrader/listener_test.go @@ -0,0 +1,254 @@ +package stream_test + +import ( + "context" + "errors" + "net" + "sync" + "time" + + peer "github.com/libp2p/go-libp2p-peer" + st "github.com/libp2p/go-libp2p-stream-transport" + tpt "github.com/libp2p/go-libp2p-transport" + smux "github.com/libp2p/go-stream-muxer" + ss "github.com/libp2p/go-stream-security" + tcp "github.com/libp2p/go-tcp-transport" + ma "github.com/multiformats/go-multiaddr" + yamux "github.com/whyrusleeping/go-smux-yamux" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +// negotiatingMuxer sets up a new yamux connection +// It makes sure that this happens at the same time for client and server. +type negotiatingMuxer struct{} + +func (m *negotiatingMuxer) NewConn(c net.Conn, isServer bool) (smux.Conn, error) { + var err error + // run a fake muxer negotiation + if isServer { + _, err = c.Write([]byte("setup")) + } else { + _, err = c.Read(make([]byte, 5)) + } + if err != nil { + return nil, err + } + return yamux.DefaultTransport.NewConn(c, isServer) +} + +// blockingMuxer blocks the muxer negotiation until the contain chan is closed +type blockingMuxer struct { + unblock chan struct{} +} + +var _ smux.Transport = &blockingMuxer{} + +func newBlockingMuxer() *blockingMuxer { return &blockingMuxer{unblock: make(chan struct{})} } +func (m *blockingMuxer) NewConn(c net.Conn, isServer bool) (smux.Conn, error) { + <-m.unblock + return (&negotiatingMuxer{}).NewConn(c, isServer) +} +func (m *blockingMuxer) Unblock() { close(m.unblock) } + +// errorMuxer is a muxer that errors while setting up +type errorMuxer struct{} + +var _ smux.Transport = &errorMuxer{} + +func (m *errorMuxer) NewConn(c net.Conn, isServer bool) (smux.Conn, error) { + return nil, errors.New("mux error") +} + +var _ = Describe("Listener", func() { + var ( + defaultUpgrader = &st.Upgrader{ + Secure: ss.NewInsecureTransport(peer.ID(1)), + Muxer: &negotiatingMuxer{}, + } + ) + + testConn := func(clientConn, serverConn tpt.Conn) { + cstr, err := clientConn.OpenStream() + ExpectWithOffset(0, err).ToNot(HaveOccurred()) + _, err = cstr.Write([]byte("foobar")) + ExpectWithOffset(0, err).ToNot(HaveOccurred()) + sstr, err := serverConn.AcceptStream() + ExpectWithOffset(0, err).ToNot(HaveOccurred()) + b := make([]byte, 6) + _, err = sstr.Read(b) + ExpectWithOffset(0, err).ToNot(HaveOccurred()) + ExpectWithOffset(0, b).To(Equal([]byte("foobar"))) + } + + createListener := func(upgrader *st.Upgrader) tpt.Listener { + addr, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") + ExpectWithOffset(0, err).ToNot(HaveOccurred()) + ln, err := tcp.NewTCPTransport(upgrader).Listen(addr) + ExpectWithOffset(0, err).ToNot(HaveOccurred()) + return ln + } + + BeforeEach(func() { + tpt.AcceptTimeout = time.Hour + }) + + It("accepts a single connection", func() { + ln := createListener(defaultUpgrader) + cconn, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(1)) + Expect(err).ToNot(HaveOccurred()) + sconn, err := ln.Accept() + Expect(err).ToNot(HaveOccurred()) + testConn(cconn, sconn) + }) + + It("accepts multiple connections", func() { + ln := createListener(defaultUpgrader) + const num = 10 + for i := 0; i < 10; i++ { + cconn, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(1)) + Expect(err).ToNot(HaveOccurred()) + sconn, err := ln.Accept() + Expect(err).ToNot(HaveOccurred()) + testConn(cconn, sconn) + } + }) + + It("closes connections if they are not accepted", func() { + const timeout = 200 * time.Millisecond + tpt.AcceptTimeout = timeout + ln := createListener(defaultUpgrader) + conn, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + Expect(err).ToNot(HaveOccurred()) + done := make(chan struct{}) + go func() { + defer GinkgoRecover() + str, err := conn.OpenStream() + Expect(err).ToNot(HaveOccurred()) + // start a Read. It will block until the connection is closed + str.Read([]byte{0}) + close(done) + }() + Consistently(done, timeout/2).ShouldNot(BeClosed()) + Eventually(done, timeout).Should(BeClosed()) + }) + + It("doesn't accept connections that fail to setup", func() { + upgrader := &st.Upgrader{ + Secure: ss.NewInsecureTransport(peer.ID(1)), + Muxer: &errorMuxer{}, + } + ln := createListener(upgrader) + done := make(chan struct{}) + go func() { + defer GinkgoRecover() + _, _ = ln.Accept() + close(done) + }() + _, _ = tcp.NewTCPTransport(upgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + Consistently(done).ShouldNot(BeClosed()) + // make the goroutine return + ln.Close() + Eventually(done).Should(BeClosed()) + }) + + Context("concurrency", func() { + It("sets up connections concurrently", func() { + num := 3 * st.AcceptQueueLength + bm := newBlockingMuxer() + upgrader := &st.Upgrader{ + Secure: ss.NewInsecureTransport(peer.ID(1)), + Muxer: bm, + } + ln := createListener(upgrader) + accepted := make(chan tpt.Conn, num) + go func() { + defer GinkgoRecover() + for { + conn, err := ln.Accept() + if err != nil { + return + } + accepted <- conn + } + }() + var wg sync.WaitGroup + // start num dials, which all block while setting up the muxer + for i := 0; i < num; i++ { + wg.Add(1) + go func() { + defer GinkgoRecover() + _, err := tcp.NewTCPTransport(upgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + Expect(err).ToNot(HaveOccurred()) + wg.Done() + }() + } + // the dials are still blocked, so we shouldn't have any connection available yet + Consistently(accepted).Should(BeEmpty()) + bm.Unblock() // make all dials succeed + Eventually(accepted).Should(HaveLen(num)) + wg.Wait() + }) + + It("stops setting up when the more than AcceptQueueLength connections are waiting to get accepted", func() { + ln := createListener(defaultUpgrader) + // setup AcceptQueueLength connections, but don't accept any of them + dialed := make(chan struct{}, 10*st.AcceptQueueLength) // used as a thread-safe counter + for i := 0; i < st.AcceptQueueLength; i++ { + go func() { + defer GinkgoRecover() + _, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + Expect(err).ToNot(HaveOccurred()) + dialed <- struct{}{} + }() + } + Eventually(dialed).Should(HaveLen(st.AcceptQueueLength)) + // dial a new connection. This connection should not complete setup, since the queue is full + go func() { + defer GinkgoRecover() + _, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + Expect(err).ToNot(HaveOccurred()) + dialed <- struct{}{} + }() + Consistently(dialed).Should(HaveLen(st.AcceptQueueLength)) + // accept a single connection. Now the new connection should be set up, and fill the queue again + _, err := ln.Accept() + Expect(err).ToNot(HaveOccurred()) + Eventually(dialed).Should(HaveLen(st.AcceptQueueLength + 1)) + }) + }) + + Context("closing", func() { + It("unblocks Accept when it is closed", func() { + ln := createListener(defaultUpgrader) + done := make(chan struct{}) + go func() { + defer GinkgoRecover() + _, err := ln.Accept() + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("use of closed network connection")) + close(done) + }() + Consistently(done).ShouldNot(BeClosed()) + Expect(ln.Close()).To(Succeed()) + Eventually(done).Should(BeClosed()) + }) + + It("doesn't accept new connections when it is closed", func() { + ln := createListener(defaultUpgrader) + Expect(ln.Close()).To(Succeed()) + _, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(1)) + Expect(err).To(HaveOccurred()) + }) + + It("closes incoming connections that have not yet been accepted", func() { + ln := createListener(defaultUpgrader) + conn, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + Expect(conn.IsClosed()).To(BeFalse()) + Expect(err).ToNot(HaveOccurred()) + Expect(ln.Close()).To(Succeed()) + Eventually(conn.IsClosed).Should(BeTrue()) + }) + }) +}) From 516978c8494484ccf8588a5b9cc7da668f1462ea Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 23 Jan 2018 23:34:12 +1100 Subject: [PATCH 0394/3965] wait after accepting a new connection if the queue is full --- p2p/net/upgrader/listener.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index c74772f90b..18005593aa 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -74,6 +74,10 @@ func (l *listener) handleIncoming() { return } + // The go routine above calls Release when the context is + // canceled so there's no need to wait on it here. + l.threshold.Wait() + log.Debugf("listener %s got connection: %s <---> %s", l, maconn.LocalMultiaddr(), @@ -116,10 +120,6 @@ func (l *listener) handleIncoming() { conn.Close() } }() - - // The go routine above calls Release when the context is - // canceled so there's no need to wait on it here. - l.threshold.Wait() } } From 72062ab86038dd8e8e5c0dc6eb5b536f3632fa0f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 26 Jan 2018 20:31:11 -0800 Subject: [PATCH 0395/3965] we no longer use maps This technically worked, but was extremely misleading. --- p2p/host/peerstore/addr_manager.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 6f348f5952..8b06c484da 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -199,17 +199,17 @@ func (mgr *AddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time return } - amap, found := mgr.addrs[p] + addrs, found := mgr.addrs[p] if !found { return } exp := time.Now().Add(newTTL) - for addrstr, aexp := range amap { + for i := range addrs { + aexp := &addrs[i] if oldTTL == aexp.TTL { aexp.TTL = newTTL aexp.Expires = exp - amap[addrstr] = aexp } } } From 5eaaf4c892a85fbfba11bf4d72a52b8a322bfdb8 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 28 Jan 2018 14:34:18 +0700 Subject: [PATCH 0396/3965] refactor for transport interface changes --- p2p/transport/quic/conn.go | 104 +++++++--------- p2p/transport/quic/conn_test.go | 179 +++++++++++---------------- p2p/transport/quic/crypto.go | 126 +++++++++++++++++++ p2p/transport/quic/dialer.go | 48 ------- p2p/transport/quic/dialer_test.go | 52 -------- p2p/transport/quic/listener.go | 89 +++++++------ p2p/transport/quic/listener_test.go | 92 ++++++-------- p2p/transport/quic/stream.go | 8 +- p2p/transport/quic/transport.go | 120 ++++++++++-------- p2p/transport/quic/transport_test.go | 74 ++--------- 10 files changed, 410 insertions(+), 482 deletions(-) create mode 100644 p2p/transport/quic/crypto.go delete mode 100644 p2p/transport/quic/dialer.go delete mode 100644 p2p/transport/quic/dialer_test.go diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 1cc72686ed..e5ccdc53c8 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -1,10 +1,10 @@ package libp2pquic import ( - "fmt" "net" - "sync" + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" tpt "github.com/libp2p/go-libp2p-transport" smux "github.com/libp2p/go-stream-muxer" quic "github.com/lucas-clemente/quic-go" @@ -12,96 +12,78 @@ import ( manet "github.com/multiformats/go-multiaddr-net" ) -type quicConn struct { - mutex sync.RWMutex - +type conn struct { sess quic.Session transport tpt.Transport - laddr ma.Multiaddr - raddr ma.Multiaddr + localPeer peer.ID + privKey ic.PrivKey + localMultiaddr ma.Multiaddr - closed bool + remotePeerID peer.ID + remotePubKey ic.PubKey + remoteMultiaddr ma.Multiaddr } -var _ tpt.Conn = &quicConn{} -var _ tpt.MultiplexConn = &quicConn{} - -func newQuicConn(sess quic.Session, t tpt.Transport) (*quicConn, error) { - // analogues to manet.WrapNetConn - laddr, err := quicMultiAddress(sess.LocalAddr()) - if err != nil { - return nil, fmt.Errorf("failed to convert nconn.LocalAddr: %s", err) - } +var _ tpt.Conn = &conn{} - // analogues to manet.WrapNetConn - raddr, err := quicMultiAddress(sess.RemoteAddr()) - if err != nil { - return nil, fmt.Errorf("failed to convert nconn.RemoteAddr: %s", err) - } - - c := &quicConn{ - sess: sess, - laddr: laddr, - raddr: raddr, - transport: t, - } - go c.watchClosed() - - return c, nil +func (c *conn) Close() error { + return c.sess.Close(nil) } -func (c *quicConn) AcceptStream() (smux.Stream, error) { - str, err := c.sess.AcceptStream() - return &stream{str}, err +// IsClosed returns whether a connection is fully closed. +func (c *conn) IsClosed() bool { + return c.sess.Context().Err() != nil } -// OpenStream opens a new stream -// It blocks until a new stream can be opened (when limited by the QUIC maximum stream limit) -func (c *quicConn) OpenStream() (smux.Stream, error) { - str, err := c.sess.OpenStreamSync() - return &stream{str}, err +// OpenStream creates a new stream. +func (c *conn) OpenStream() (smux.Stream, error) { + qstr, err := c.sess.OpenStreamSync() + return &stream{Stream: qstr}, err } -func (c *quicConn) Close() error { - return c.sess.Close(nil) +// AcceptStream accepts a stream opened by the other side. +func (c *conn) AcceptStream() (smux.Stream, error) { + qstr, err := c.sess.AcceptStream() + return &stream{Stream: qstr}, err } -func (c *quicConn) watchClosed() { - <-c.sess.Context().Done() - c.mutex.Lock() - c.closed = true - c.mutex.Unlock() +// LocalPeer returns our peer ID +func (c *conn) LocalPeer() peer.ID { + return c.localPeer } -func (c *quicConn) IsClosed() bool { - c.mutex.Lock() - defer c.mutex.Unlock() - return c.closed +// LocalPrivateKey returns our private key +func (c *conn) LocalPrivateKey() ic.PrivKey { + return c.privKey } -func (c *quicConn) LocalAddr() net.Addr { - return c.sess.LocalAddr() +// RemotePeer returns the peer ID of the remote peer. +func (c *conn) RemotePeer() peer.ID { + return c.remotePeerID } -func (c *quicConn) LocalMultiaddr() ma.Multiaddr { - return c.laddr +// RemotePublicKey returns the public key of the remote peer. +func (c *conn) RemotePublicKey() ic.PubKey { + return c.remotePubKey } -func (c *quicConn) RemoteAddr() net.Addr { - return c.sess.RemoteAddr() +// LocalMultiaddr returns the local Multiaddr associated +func (c *conn) LocalMultiaddr() ma.Multiaddr { + return c.localMultiaddr } -func (c *quicConn) RemoteMultiaddr() ma.Multiaddr { - return c.raddr +// RemoteMultiaddr returns the remote Multiaddr associated +func (c *conn) RemoteMultiaddr() ma.Multiaddr { + return c.remoteMultiaddr } -func (c *quicConn) Transport() tpt.Transport { +func (c *conn) Transport() tpt.Transport { return c.transport } // TODO: there must be a better way to do this -func quicMultiAddress(na net.Addr) (ma.Multiaddr, error) { +func quicMultiaddr(na net.Addr) (ma.Multiaddr, error) { udpMA, err := manet.FromNetAddr(na) if err != nil { return nil, err diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 815e269a0e..473a50c6e9 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -2,133 +2,92 @@ package libp2pquic import ( "context" - "errors" - "net" - "time" + "crypto/rand" + "crypto/rsa" + "crypto/x509" - quic "github.com/lucas-clemente/quic-go" + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" + tpt "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) -type mockStream struct { - id quic.StreamID -} - -func (s *mockStream) Close() error { return nil } -func (s *mockStream) Reset(error) { return } -func (s *mockStream) Read([]byte) (int, error) { return 0, nil } -func (s *mockStream) Write([]byte) (int, error) { return 0, nil } -func (s *mockStream) StreamID() quic.StreamID { return s.id } -func (s *mockStream) SetReadDeadline(time.Time) error { panic("not implemented") } -func (s *mockStream) SetWriteDeadline(time.Time) error { panic("not implemented") } -func (s *mockStream) SetDeadline(time.Time) error { panic("not implemented") } -func (s *mockStream) Context() context.Context { panic("not implemented") } - -var _ quic.Stream = &mockStream{} - -type mockQuicSession struct { - closed bool - context context.Context - - localAddr net.Addr - remoteAddr net.Addr - - streamToAccept quic.Stream - streamAcceptErr error - - streamToOpen quic.Stream - streamOpenErr error -} - -var _ quic.Session = &mockQuicSession{} - -func (s *mockQuicSession) AcceptStream() (quic.Stream, error) { - return s.streamToAccept, s.streamAcceptErr -} -func (s *mockQuicSession) OpenStream() (quic.Stream, error) { return s.streamToOpen, s.streamOpenErr } -func (s *mockQuicSession) OpenStreamSync() (quic.Stream, error) { - return s.streamToOpen, s.streamOpenErr -} -func (s *mockQuicSession) Close(error) error { s.closed = true; return nil } -func (s *mockQuicSession) LocalAddr() net.Addr { return s.localAddr } -func (s *mockQuicSession) RemoteAddr() net.Addr { return s.remoteAddr } -func (s *mockQuicSession) Context() context.Context { return s.context } - -var _ = Describe("Conn", func() { +var _ = Describe("Connection", func() { var ( - conn *quicConn - sess *mockQuicSession - ctxCancel context.CancelFunc + serverKey, clientKey ic.PrivKey + serverID, clientID peer.ID ) - BeforeEach(func() { - var ctx context.Context - ctx, ctxCancel = context.WithCancel(context.Background()) - sess = &mockQuicSession{ - localAddr: &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 1337}, - remoteAddr: &net.UDPAddr{IP: net.IPv4(192, 168, 13, 37), Port: 1234}, - context: ctx, - } - var err error - conn, err = newQuicConn(sess, nil) + createPeer := func() ic.PrivKey { + key, err := rsa.GenerateKey(rand.Reader, 1024) Expect(err).ToNot(HaveOccurred()) - }) - - It("has the correct local address", func() { - Expect(conn.LocalAddr()).To(Equal(sess.localAddr)) - Expect(conn.LocalMultiaddr().String()).To(Equal("/ip4/127.0.0.1/udp/1337/quic")) - }) - - It("has the correct remote address", func() { - Expect(conn.RemoteAddr()).To(Equal(sess.remoteAddr)) - Expect(conn.RemoteMultiaddr().String()).To(Equal("/ip4/192.168.13.37/udp/1234/quic")) - }) + priv, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(key)) + Expect(err).ToNot(HaveOccurred()) + return priv + } - It("closes", func() { - err := conn.Close() + runServer := func() (<-chan ma.Multiaddr, <-chan tpt.Conn) { + serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - Expect(sess.closed).To(BeTrue()) - }) + addrChan := make(chan ma.Multiaddr) + connChan := make(chan tpt.Conn) + go func() { + defer GinkgoRecover() + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + Expect(err).ToNot(HaveOccurred()) + ln, err := serverTransport.Listen(addr) + Expect(err).ToNot(HaveOccurred()) + addrChan <- ln.Multiaddr() + conn, err := ln.Accept() + Expect(err).ToNot(HaveOccurred()) + connChan <- conn + }() + return addrChan, connChan + } - It("says if it is closed", func() { - Consistently(func() bool { return conn.IsClosed() }).Should(BeFalse()) - ctxCancel() - Eventually(func() bool { return conn.IsClosed() }).Should(BeTrue()) + BeforeEach(func() { + var err error + serverKey = createPeer() + serverID, err = peer.IDFromPrivateKey(serverKey) + Expect(err).ToNot(HaveOccurred()) + clientKey = createPeer() + clientID, err = peer.IDFromPrivateKey(clientKey) + Expect(err).ToNot(HaveOccurred()) }) - Context("opening streams", func() { - It("opens streams", func() { - s := &mockStream{id: 1337} - sess.streamToOpen = s - str, err := conn.OpenStream() - Expect(err).ToNot(HaveOccurred()) - Expect(str.(*stream).Stream).To(Equal(s)) - }) - - It("errors when it can't open a stream", func() { - testErr := errors.New("stream open err") - sess.streamOpenErr = testErr - _, err := conn.OpenStream() - Expect(err).To(MatchError(testErr)) - }) + It("handshakes", func() { + serverAddrChan, serverConnChan := runServer() + clientTransport, err := NewTransport(clientKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr := <-serverAddrChan + conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) + Expect(err).ToNot(HaveOccurred()) + serverConn := <-serverConnChan + Expect(conn.LocalPeer()).To(Equal(clientID)) + Expect(conn.LocalPrivateKey()).To(Equal(clientKey)) + Expect(conn.RemotePeer()).To(Equal(serverID)) + Expect(conn.RemotePublicKey()).To(Equal(serverKey.GetPublic())) + Expect(serverConn.LocalPeer()).To(Equal(serverID)) + Expect(serverConn.LocalPrivateKey()).To(Equal(serverKey)) + Expect(serverConn.RemotePeer()).To(Equal(clientID)) + Expect(serverConn.RemotePublicKey()).To(Equal(clientKey.GetPublic())) }) - Context("accepting streams", func() { - It("accepts streams", func() { - s := &mockStream{id: 1337} - sess.streamToAccept = s - str, err := conn.AcceptStream() - Expect(err).ToNot(HaveOccurred()) - Expect(str.(*stream).Stream).To(Equal(s)) - }) + It("fails if the peer ID doesn't match", func() { + thirdPartyID, err := peer.IDFromPrivateKey(createPeer()) + Expect(err).ToNot(HaveOccurred()) - It("errors when it can't open a stream", func() { - testErr := errors.New("stream open err") - sess.streamAcceptErr = testErr - _, err := conn.AcceptStream() - Expect(err).To(MatchError(testErr)) - }) + serverAddrChan, _ := runServer() + clientTransport, err := NewTransport(clientKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr := <-serverAddrChan + // dial, but expect the wrong peer ID + _, err = clientTransport.Dial(context.Background(), serverAddr, thirdPartyID) + Expect(err).To(MatchError("peer IDs don't match")) + // TODO(#2): don't accept a connection if the client's peer verification fails + // Consistently(serverConnChan).ShouldNot(Receive()) }) }) diff --git a/p2p/transport/quic/crypto.go b/p2p/transport/quic/crypto.go new file mode 100644 index 0000000000..a6dee953d7 --- /dev/null +++ b/p2p/transport/quic/crypto.go @@ -0,0 +1,126 @@ +package libp2pquic + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "errors" + "math/big" + "time" + + "github.com/lucas-clemente/quic-go" + + "github.com/gogo/protobuf/proto" + ic "github.com/libp2p/go-libp2p-crypto" + pb "github.com/libp2p/go-libp2p-crypto/pb" +) + +// mint certificate selection is broken. +const hostname = "quic.ipfs" + +type connectionStater interface { + ConnectionState() quic.ConnectionState +} + +// TODO: make this private +func GenerateConfig(privKey ic.PrivKey) (*tls.Config, error) { + key, hostCert, err := keyToCertificate(privKey) + if err != nil { + return nil, err + } + // The ephemeral key used just for a couple of connections (or a limited time). + ephemeralKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return nil, err + } + // Sign the ephemeral key using the host key. + // This is the only time that the host's private key of the peer is needed. + // Note that this step could be done asynchronously, such that a running node doesn't need access its private key at all. + certTemplate := &x509.Certificate{ + DNSNames: []string{hostname}, + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-24 * time.Hour), + NotAfter: time.Now().Add(30 * 24 * time.Hour), + } + certDER, err := x509.CreateCertificate(rand.Reader, certTemplate, hostCert, ephemeralKey.Public(), key) + if err != nil { + return nil, err + } + cert, err := x509.ParseCertificate(certDER) + if err != nil { + return nil, err + } + return &tls.Config{ + ServerName: hostname, + InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. + ClientAuth: tls.RequireAnyClientCert, + Certificates: []tls.Certificate{{ + Certificate: [][]byte{cert.Raw, hostCert.Raw}, + PrivateKey: ephemeralKey, + }}, + }, nil +} + +func getRemotePubKey(conn connectionStater) (ic.PubKey, error) { + certChain := conn.ConnectionState().PeerCertificates + if len(certChain) != 2 { + return nil, errors.New("expected 2 certificates in the chain") + } + pool := x509.NewCertPool() + pool.AddCert(certChain[1]) + if _, err := certChain[0].Verify(x509.VerifyOptions{Roots: pool}); err != nil { + return nil, err + } + remotePubKey, err := x509.MarshalPKIXPublicKey(certChain[1].PublicKey) + if err != nil { + return nil, err + } + return ic.UnmarshalRsaPublicKey(remotePubKey) +} + +func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { + sn, err := rand.Int(rand.Reader, big.NewInt(1<<62)) + if err != nil { + return nil, nil, err + } + tmpl := &x509.Certificate{ + SerialNumber: sn, + NotBefore: time.Now().Add(-24 * time.Hour), + NotAfter: time.Now().Add(30 * 24 * time.Hour), + IsCA: true, + BasicConstraintsValid: true, + } + + var publicKey, privateKey interface{} + keyBytes, err := sk.Bytes() + if err != nil { + return nil, nil, err + } + pbmes := new(pb.PrivateKey) + if err := proto.Unmarshal(keyBytes, pbmes); err != nil { + return nil, nil, err + } + switch pbmes.GetType() { + case pb.KeyType_RSA: + k, err := x509.ParsePKCS1PrivateKey(pbmes.GetData()) + if err != nil { + return nil, nil, err + } + publicKey = &k.PublicKey + privateKey = k + // TODO: add support for ECDSA + default: + return nil, nil, errors.New("unsupported key type for TLS") + } + certDER, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, publicKey, privateKey) + if err != nil { + return nil, nil, err + } + cert, err := x509.ParseCertificate(certDER) + if err != nil { + return nil, nil, err + } + return privateKey, cert, nil +} diff --git a/p2p/transport/quic/dialer.go b/p2p/transport/quic/dialer.go deleted file mode 100644 index fd47186b2a..0000000000 --- a/p2p/transport/quic/dialer.go +++ /dev/null @@ -1,48 +0,0 @@ -package libp2pquic - -import ( - "context" - "crypto/tls" - - tpt "github.com/libp2p/go-libp2p-transport" - quic "github.com/lucas-clemente/quic-go" - ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" - "github.com/whyrusleeping/mafmt" -) - -type dialer struct { - transport tpt.Transport -} - -var _ tpt.Dialer = &dialer{} - -func newDialer(transport tpt.Transport) (*dialer, error) { - return &dialer{ - transport: transport, - }, nil -} - -func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { - // TODO: check that raddr is a QUIC address - _, host, err := manet.DialArgs(raddr) - if err != nil { - return nil, err - } - - qsess, err := quic.DialAddr(host, &tls.Config{InsecureSkipVerify: true}, nil) - if err != nil { - return nil, err - } - - return newQuicConn(qsess, d.transport) -} - -func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { - // TODO: implement the ctx - return d.Dial(raddr) -} - -func (d *dialer) Matches(a ma.Multiaddr) bool { - return mafmt.QUIC.Matches(a) -} diff --git a/p2p/transport/quic/dialer_test.go b/p2p/transport/quic/dialer_test.go deleted file mode 100644 index 229e19a606..0000000000 --- a/p2p/transport/quic/dialer_test.go +++ /dev/null @@ -1,52 +0,0 @@ -package libp2pquic - -import ( - tpt "github.com/libp2p/go-libp2p-transport" - ma "github.com/multiformats/go-multiaddr" - - . "github.com/onsi/ginkgo" - . "github.com/onsi/gomega" -) - -var _ = Describe("Listener", func() { - var ( - d *dialer - transport tpt.Transport - ) - - BeforeEach(func() { - var err error - transport = &QuicTransport{} - d, err = newDialer(transport) - Expect(err).ToNot(HaveOccurred()) - }) - - It("dials", func() { - addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/8888") - Expect(err).ToNot(HaveOccurred()) - - // start a listener to connect to - var ln *listener - go func() { - defer GinkgoRecover() - ln, err = newListener(addr, transport) - Expect(err).ToNot(HaveOccurred()) - _, err = ln.Accept() - Expect(err).ToNot(HaveOccurred()) - }() - - Eventually(func() *listener { return ln }).ShouldNot(BeNil()) - conn, err := d.Dial(addr) - Expect(err).ToNot(HaveOccurred()) - Expect(conn.Transport()).To(Equal(d.transport)) - }) - - It("matches", func() { - invalidAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") - Expect(err).ToNot(HaveOccurred()) - validAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") - Expect(err).ToNot(HaveOccurred()) - Expect(d.Matches(invalidAddr)).To(BeFalse()) - Expect(d.Matches(validAddr)).To(BeTrue()) - }) -}) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index b0d61c6966..ab044add60 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -1,91 +1,100 @@ package libp2pquic import ( - "crypto/rand" - "crypto/rsa" - "crypto/tls" - "crypto/x509" - "encoding/pem" - "math/big" "net" + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" tpt "github.com/libp2p/go-libp2p-transport" quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) +var quicListenAddr = quic.ListenAddr + +// A listener listens for QUIC connections. type listener struct { - laddr ma.Multiaddr quicListener quic.Listener + transport tpt.Transport + + acceptQueue chan tpt.Conn - transport tpt.Transport + privKey ic.PrivKey + localPeer peer.ID + localMultiaddr ma.Multiaddr } var _ tpt.Listener = &listener{} -func newListener(laddr ma.Multiaddr, t tpt.Transport) (*listener, error) { - _, host, err := manet.DialArgs(laddr) +func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, key ic.PrivKey) (tpt.Listener, error) { + _, host, err := manet.DialArgs(addr) if err != nil { return nil, err } - tlsConf, err := generateTLSConfig() + tlsConf, err := GenerateConfig(key) if err != nil { return nil, err } - qln, err := quic.ListenAddr(host, tlsConf, nil) + ln, err := quicListenAddr(host, tlsConf, &quic.Config{Versions: []quic.VersionNumber{101}}) if err != nil { return nil, err } - addr, err := quicMultiAddress(qln.Addr()) + localMultiaddr, err := quicMultiaddr(ln.Addr()) if err != nil { return nil, err } - return &listener{ - laddr: addr, - quicListener: qln, - transport: t, + quicListener: ln, + transport: transport, + privKey: key, + localPeer: localPeer, + localMultiaddr: localMultiaddr, }, nil } +// Accept accepts new connections. +// TODO(#2): don't accept a connection if the client's peer verification fails func (l *listener) Accept() (tpt.Conn, error) { sess, err := l.quicListener.Accept() if err != nil { return nil, err } - return newQuicConn(sess, l.transport) + remotePubKey, err := getRemotePubKey(sess) + if err != nil { + return nil, err + } + remotePeerID, err := peer.IDFromPublicKey(remotePubKey) + if err != nil { + return nil, err + } + remoteMultiaddr, err := quicMultiaddr(sess.RemoteAddr()) + if err != nil { + return nil, err + } + return &conn{ + sess: sess, + transport: l.transport, + localPeer: l.localPeer, + localMultiaddr: l.localMultiaddr, + privKey: l.privKey, + remoteMultiaddr: remoteMultiaddr, + remotePeerID: remotePeerID, + remotePubKey: remotePubKey, + }, nil } +// Close closes the listener. func (l *listener) Close() error { return l.quicListener.Close() } +// Addr returns the address of this listener. func (l *listener) Addr() net.Addr { return l.quicListener.Addr() } +// Multiaddr returns the multiaddress of this listener. func (l *listener) Multiaddr() ma.Multiaddr { - return l.laddr -} - -// Generate a bare-bones TLS config for the server. -// The client doesn't verify the certificate yet. -func generateTLSConfig() (*tls.Config, error) { - key, err := rsa.GenerateKey(rand.Reader, 2048) - if err != nil { - return nil, err - } - template := x509.Certificate{SerialNumber: big.NewInt(1)} - certDER, err := x509.CreateCertificate(rand.Reader, &template, &template, &key.PublicKey, key) - if err != nil { - return nil, err - } - keyPEM := pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(key)}) - certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: certDER}) - tlsCert, err := tls.X509KeyPair(certPEM, keyPEM) - if err != nil { - return nil, err - } - return &tls.Config{Certificates: []tls.Certificate{tlsCert}}, nil + return l.localMultiaddr } diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index f4e03d63c8..1069d57e99 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -1,81 +1,67 @@ package libp2pquic import ( - "errors" + "crypto/rand" + "crypto/rsa" + "crypto/x509" + "fmt" "net" + ic "github.com/libp2p/go-libp2p-crypto" tpt "github.com/libp2p/go-libp2p-transport" - quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" - . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) -type mockQuicListener struct { - connToAccept net.Conn - serveErr error - closed bool - - sessionToAccept quic.Session - acceptErr error -} - -var _ quic.Listener = &mockQuicListener{} - -func (m *mockQuicListener) Close() error { m.closed = true; return nil } -func (m *mockQuicListener) Accept() (quic.Session, error) { return m.sessionToAccept, m.acceptErr } -func (m *mockQuicListener) Addr() net.Addr { panic("not implemented") } - var _ = Describe("Listener", func() { var ( - l *listener - quicListener *mockQuicListener - transport tpt.Transport + t tpt.Transport + localAddr ma.Multiaddr ) BeforeEach(func() { - quicListener = &mockQuicListener{} - transport = &QuicTransport{} - l = &listener{ - quicListener: quicListener, - transport: transport, - } - }) - - It("returns its addr", func() { - laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + rsaKey, err := rsa.GenerateKey(rand.Reader, 1024) + Expect(err).ToNot(HaveOccurred()) + key, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(rsaKey)) + Expect(err).ToNot(HaveOccurred()) + t, err = NewTransport(key) Expect(err).ToNot(HaveOccurred()) - l, err = newListener(laddr, nil) + localAddr, err = ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") Expect(err).ToNot(HaveOccurred()) - as := l.Addr().String() - Expect(as).ToNot(Equal("127.0.0.1:0)")) - Expect(as).To(ContainSubstring("127.0.0.1:")) }) - It("returns its multiaddr", func() { - laddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + It("returns the address it is listening on", func() { + ln, err := t.Listen(localAddr) Expect(err).ToNot(HaveOccurred()) - l, err = newListener(laddr, nil) - Expect(err).ToNot(HaveOccurred()) - mas := l.Multiaddr().String() - Expect(mas).ToNot(Equal("/ip4/127.0.0.1/udp/0/quic")) - Expect(mas).To(ContainSubstring("/ip4/127.0.0.1/udp/")) - Expect(mas).To(ContainSubstring("/quic")) + netAddr := ln.Addr() + Expect(netAddr).To(BeAssignableToTypeOf(&net.UDPAddr{})) + port := netAddr.(*net.UDPAddr).Port + Expect(port).ToNot(BeZero()) + Expect(ln.Multiaddr().String()).To(Equal(fmt.Sprintf("/ip4/127.0.0.1/udp/%d/quic", port))) }) - It("closes", func() { - err := l.Close() + It("returns Accept when it is closed", func() { + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + Expect(err).ToNot(HaveOccurred()) + ln, err := t.Listen(addr) Expect(err).ToNot(HaveOccurred()) - Expect(quicListener.closed).To(BeTrue()) + done := make(chan struct{}) + go func() { + defer GinkgoRecover() + ln.Accept() + close(done) + }() + Consistently(done).ShouldNot(BeClosed()) + Expect(ln.Close()).To(Succeed()) + Eventually(done).Should(BeClosed()) }) - Context("accepting", func() { - It("errors if no connection can be accepted", func() { - testErr := errors.New("test error") - quicListener.acceptErr = testErr - _, err := l.Accept() - Expect(err).To(MatchError(testErr)) - }) + It("doesn't accept Accept calls after it is closed", func() { + ln, err := t.Listen(localAddr) + Expect(err).ToNot(HaveOccurred()) + Expect(ln.Close()).To(Succeed()) + _, err = ln.Accept() + Expect(err).To(HaveOccurred()) }) }) diff --git a/p2p/transport/quic/stream.go b/p2p/transport/quic/stream.go index 5d5ad242cf..3fcf0d7d79 100644 --- a/p2p/transport/quic/stream.go +++ b/p2p/transport/quic/stream.go @@ -2,7 +2,7 @@ package libp2pquic import ( smux "github.com/libp2p/go-stream-muxer" - "github.com/lucas-clemente/quic-go" + quic "github.com/lucas-clemente/quic-go" ) type stream struct { @@ -12,6 +12,8 @@ type stream struct { var _ smux.Stream = &stream{} func (s *stream) Reset() error { - s.Stream.Reset(nil) - return nil + if err := s.Stream.CancelRead(0); err != nil { + return err + } + return s.Stream.CancelWrite(0) } diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 04f6a82751..6e05f63d64 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -1,83 +1,97 @@ package libp2pquic import ( - "fmt" - "sync" + "context" + "errors" + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" tpt "github.com/libp2p/go-libp2p-transport" + quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" "github.com/whyrusleeping/mafmt" ) -// QuicTransport implements a QUIC Transport -type QuicTransport struct { - lmutex sync.Mutex - listeners map[string]tpt.Listener +var quicDialAddr = quic.DialAddr - dmutex sync.Mutex - dialers map[string]tpt.Dialer +// The Transport implements the tpt.Transport interface for QUIC connections. +type transport struct { + privKey ic.PrivKey + localPeer peer.ID } -var _ tpt.Transport = &QuicTransport{} +var _ tpt.Transport = &transport{} -// NewQuicTransport creates a new QUIC Transport -// it tracks dialers and listeners created -func NewQuicTransport() *QuicTransport { - // utils.SetLogLevel(utils.LogLevelDebug) - return &QuicTransport{ - listeners: make(map[string]tpt.Listener), - dialers: make(map[string]tpt.Dialer), +// NewTransport creates a new QUIC transport +func NewTransport(key ic.PrivKey) (tpt.Transport, error) { + localPeer, err := peer.IDFromPrivateKey(key) + if err != nil { + return nil, err } + return &transport{ + privKey: key, + localPeer: localPeer, + }, nil } -func (t *QuicTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { - if !t.Matches(laddr) { - return nil, fmt.Errorf("quic transport cannot dial %q", laddr) - } - - t.dmutex.Lock() - defer t.dmutex.Unlock() - - s := laddr.String() - d, ok := t.dialers[s] - if ok { - return d, nil +// Dial dials a new QUIC connection +func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { + _, host, err := manet.DialArgs(raddr) + if err != nil { + return nil, err } - - // TODO: read opts - quicd, err := newDialer(t) + tlsConf, err := GenerateConfig(t.privKey) if err != nil { return nil, err } - t.dialers[s] = quicd - return quicd, nil -} - -// Listen starts listening on laddr -func (t *QuicTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { - if !t.Matches(laddr) { - return nil, fmt.Errorf("quic transport cannot listen on %q", laddr) + sess, err := quicDialAddr(host, tlsConf, &quic.Config{Versions: []quic.VersionNumber{101}}) + if err != nil { + return nil, err } - - t.lmutex.Lock() - defer t.lmutex.Unlock() - - l, ok := t.listeners[laddr.String()] - if ok { - return l, nil + remotePubKey, err := getRemotePubKey(sess) + if err != nil { + return nil, err } - - ln, err := newListener(laddr, t) + localMultiaddr, err := quicMultiaddr(sess.LocalAddr()) if err != nil { return nil, err } + if !p.MatchesPublicKey(remotePubKey) { + err := errors.New("peer IDs don't match") + sess.Close(err) + return nil, err + } + return &conn{ + privKey: t.privKey, + localPeer: t.localPeer, + localMultiaddr: localMultiaddr, + remotePubKey: remotePubKey, + remotePeerID: p, + remoteMultiaddr: raddr, + }, nil +} + +// CanDial determines if we can dial to an address +func (t *transport) CanDial(addr ma.Multiaddr) bool { + return mafmt.QUIC.Matches(addr) +} - t.listeners[laddr.String()] = ln - return ln, nil +// Listen listens for new QUIC connections on the passed multiaddr. +func (t *transport) Listen(addr ma.Multiaddr) (tpt.Listener, error) { + return newListener(addr, t, t.localPeer, t.privKey) } -func (t *QuicTransport) Matches(a ma.Multiaddr) bool { - return mafmt.QUIC.Matches(a) +// Proxy returns true if this transport proxies. +func (t *transport) Proxy() bool { + return false } -var _ tpt.Transport = &QuicTransport{} +// Protocols returns the set of protocols handled by this transport. +func (t *transport) Protocols() []int { + return []int{ma.P_QUIC} +} + +func (t *transport) String() string { + return "QUIC" +} diff --git a/p2p/transport/quic/transport_test.go b/p2p/transport/quic/transport_test.go index 141e468426..0decd920a7 100644 --- a/p2p/transport/quic/transport_test.go +++ b/p2p/transport/quic/transport_test.go @@ -1,6 +1,7 @@ package libp2pquic import ( + tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" . "github.com/onsi/ginkgo" @@ -8,75 +9,24 @@ import ( ) var _ = Describe("Transport", func() { - var t *QuicTransport + var t tpt.Transport BeforeEach(func() { - t = NewQuicTransport() + t = &transport{} }) - Context("listening", func() { - It("creates a new listener", func() { - maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") - Expect(err).ToNot(HaveOccurred()) - ln, err := t.Listen(maddr) - Expect(err).ToNot(HaveOccurred()) - Expect(ln.Multiaddr()).To(Equal(maddr)) - }) - - It("returns an existing listener", func() { - maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1235/quic") - Expect(err).ToNot(HaveOccurred()) - ln, err := t.Listen(maddr) - Expect(err).ToNot(HaveOccurred()) - Expect(ln.Multiaddr()).To(Equal(maddr)) - ln2, err := t.Listen(maddr) - Expect(err).ToNot(HaveOccurred()) - Expect(ln2).To(Equal(ln)) - Expect(t.listeners).To(HaveLen(1)) - }) - - It("errors if the address is not a QUIC address", func() { - maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1235/utp") - Expect(err).ToNot(HaveOccurred()) - _, err = t.Listen(maddr) - Expect(err).To(MatchError("quic transport cannot listen on \"/ip4/127.0.0.1/udp/1235/utp\"")) - }) - }) - - Context("dialing", func() { - It("creates a new dialer", func() { - maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") - Expect(err).ToNot(HaveOccurred()) - d, err := t.Dialer(maddr) - Expect(err).ToNot(HaveOccurred()) - Expect(d).ToNot(BeNil()) - }) - - It("returns an existing dialer", func() { - maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1235/quic") - Expect(err).ToNot(HaveOccurred()) - d, err := t.Dialer(maddr) - Expect(err).ToNot(HaveOccurred()) - d2, err := t.Dialer(maddr) - Expect(err).ToNot(HaveOccurred()) - Expect(d2).To(Equal(d)) - Expect(t.dialers).To(HaveLen(1)) - }) - - It("errors if the address is not a QUIC address", func() { - maddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1235/utp") - Expect(err).ToNot(HaveOccurred()) - _, err = t.Dialer(maddr) - Expect(err).To(MatchError("quic transport cannot dial \"/ip4/127.0.0.1/udp/1235/utp\"")) - }) - }) - - It("matches", func() { + It("says if it can dial an address", func() { invalidAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") Expect(err).ToNot(HaveOccurred()) validAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") Expect(err).ToNot(HaveOccurred()) - Expect(t.Matches(invalidAddr)).To(BeFalse()) - Expect(t.Matches(validAddr)).To(BeTrue()) + Expect(t.CanDial(invalidAddr)).To(BeFalse()) + Expect(t.CanDial(validAddr)).To(BeTrue()) + }) + + It("supports the QUIC protocol", func() { + protocols := t.Protocols() + Expect(protocols).To(HaveLen(1)) + Expect(protocols[0]).To(Equal(ma.P_QUIC)) }) }) From 1317fb9f54a89e1dfadf813f8b2478672c24fc93 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 28 Jan 2018 17:56:38 -0800 Subject: [PATCH 0397/3965] complete rename --- p2p/net/upgrader/upgrader.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index c494ea2374..392d65da03 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -11,7 +11,7 @@ import ( transport "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" smux "github.com/libp2p/go-stream-muxer" - ss "github.com/libp2p/go-stream-security" + ss "github.com/libp2p/go-conn-security" manet "github.com/multiformats/go-multiaddr-net" ) From fc8c4a8fbf05e9787057c82c5b0e5da26a3b8b33 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 28 Jan 2018 18:04:12 -0800 Subject: [PATCH 0398/3965] complete rename --- p2p/net/conn-security-multistream/ssms.go | 18 +++++++++--------- p2p/net/conn-security-multistream/ssms_test.go | 14 +++++++------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/p2p/net/conn-security-multistream/ssms.go b/p2p/net/conn-security-multistream/ssms.go index 0adbe0f7a8..75bbb10f44 100644 --- a/p2p/net/conn-security-multistream/ssms.go +++ b/p2p/net/conn-security-multistream/ssms.go @@ -1,4 +1,4 @@ -package ssms +package csms import ( "context" @@ -6,7 +6,7 @@ import ( "net" peer "github.com/libp2p/go-libp2p-peer" - ss "github.com/libp2p/go-stream-security" + connsec "github.com/libp2p/go-conn-security" mss "github.com/multiformats/go-multistream" ) @@ -16,19 +16,19 @@ import ( // after use. type SSMuxer struct { mux mss.MultistreamMuxer - tpts map[string]ss.Transport + tpts map[string]connsec.Transport OrderPreference []string } -var _ ss.Transport = (*SSMuxer)(nil) +var _ connsec.Transport = (*SSMuxer)(nil) // AddTransport adds a stream security transport to this multistream muxer. // // This method is *not* thread-safe. It should be called only when initializing // the SSMuxer. -func (sm *SSMuxer) AddTransport(path string, transport ss.Transport) { +func (sm *SSMuxer) AddTransport(path string, transport connsec.Transport) { if sm.tpts == nil { - sm.tpts = make(map[string]ss.Transport, 1) + sm.tpts = make(map[string]connsec.Transport, 1) } sm.mux.AddHandler(path, nil) @@ -38,7 +38,7 @@ func (sm *SSMuxer) AddTransport(path string, transport ss.Transport) { // SecureInbound secures an inbound connection using this multistream // multiplexed stream security transport. -func (sm *SSMuxer) SecureInbound(ctx context.Context, insecure net.Conn) (ss.Conn, error) { +func (sm *SSMuxer) SecureInbound(ctx context.Context, insecure net.Conn) (connsec.Conn, error) { tpt, err := sm.selectProto(ctx, insecure, true) if err != nil { return nil, err @@ -48,7 +48,7 @@ func (sm *SSMuxer) SecureInbound(ctx context.Context, insecure net.Conn) (ss.Con // SecureOutbound secures an outbound connection using this multistream // multiplexed stream security transport. -func (sm *SSMuxer) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (ss.Conn, error) { +func (sm *SSMuxer) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (connsec.Conn, error) { tpt, err := sm.selectProto(ctx, insecure, false) if err != nil { return nil, err @@ -56,7 +56,7 @@ func (sm *SSMuxer) SecureOutbound(ctx context.Context, insecure net.Conn, p peer return tpt.SecureOutbound(ctx, insecure, p) } -func (sm *SSMuxer) selectProto(ctx context.Context, insecure net.Conn, server bool) (ss.Transport, error) { +func (sm *SSMuxer) selectProto(ctx context.Context, insecure net.Conn, server bool) (connsec.Transport, error) { var proto string var err error done := make(chan struct{}) diff --git a/p2p/net/conn-security-multistream/ssms_test.go b/p2p/net/conn-security-multistream/ssms_test.go index faa1c87dfd..5d72458562 100644 --- a/p2p/net/conn-security-multistream/ssms_test.go +++ b/p2p/net/conn-security-multistream/ssms_test.go @@ -1,4 +1,4 @@ -package ssms +package csms import ( "context" @@ -6,14 +6,14 @@ import ( "sync" "testing" - ss "github.com/libp2p/go-stream-security" - sst "github.com/libp2p/go-stream-security/test" + connsec "github.com/libp2p/go-conn-security" + sst "github.com/libp2p/go-conn-security/test" ) func TestCommonProto(t *testing.T) { var at, bt SSMuxer - atInsecure := ss.InsecureTransport("peerA") - btInsecure := ss.InsecureTransport("peerB") + atInsecure := connsec.InsecureTransport("peerA") + btInsecure := connsec.InsecureTransport("peerB") at.AddTransport("/plaintext/1.0.0", &atInsecure) bt.AddTransport("/plaintext/1.1.0", &btInsecure) bt.AddTransport("/plaintext/1.0.0", &btInsecure) @@ -22,8 +22,8 @@ func TestCommonProto(t *testing.T) { func TestNoCommonProto(t *testing.T) { var at, bt SSMuxer - atInsecure := ss.InsecureTransport("peerA") - btInsecure := ss.InsecureTransport("peerB") + atInsecure := connsec.InsecureTransport("peerA") + btInsecure := connsec.InsecureTransport("peerB") at.AddTransport("/plaintext/1.0.0", &atInsecure) bt.AddTransport("/plaintext/1.1.0", &btInsecure) From dc4b4aef0cc6a31d7bc60dee82128bbb19ad86f1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 28 Jan 2018 18:57:37 -0800 Subject: [PATCH 0399/3965] fix tests for rename --- p2p/net/upgrader/listener_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go index c2eaf6c72b..7be94667a0 100644 --- a/p2p/net/upgrader/listener_test.go +++ b/p2p/net/upgrader/listener_test.go @@ -8,10 +8,10 @@ import ( "time" peer "github.com/libp2p/go-libp2p-peer" - st "github.com/libp2p/go-libp2p-stream-transport" + st "github.com/libp2p/go-libp2p-transport-upgrader" tpt "github.com/libp2p/go-libp2p-transport" smux "github.com/libp2p/go-stream-muxer" - ss "github.com/libp2p/go-stream-security" + ss "github.com/libp2p/go-conn-security" tcp "github.com/libp2p/go-tcp-transport" ma "github.com/multiformats/go-multiaddr" yamux "github.com/whyrusleeping/go-smux-yamux" From 0db3c0aa8861d80e8e91d21401be2400e33b0e0d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 15 Feb 2018 12:15:41 -0800 Subject: [PATCH 0400/3965] fix for interface move --- p2p/net/upgrader/conn.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/net/upgrader/conn.go b/p2p/net/upgrader/conn.go index 50d5a5560b..48e4814166 100644 --- a/p2p/net/upgrader/conn.go +++ b/p2p/net/upgrader/conn.go @@ -3,14 +3,15 @@ package stream import ( "fmt" + inet "github.com/libp2p/go-libp2p-net" transport "github.com/libp2p/go-libp2p-transport" smux "github.com/libp2p/go-stream-muxer" ) type transportConn struct { smux.Conn - transport.ConnMultiaddrs - transport.ConnSecurity + inet.ConnMultiaddrs + inet.ConnSecurity transport transport.Transport } From 605544869cb72af1d514e701e85e0030f3cbf6c4 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 15 Feb 2018 12:16:22 -0800 Subject: [PATCH 0401/3965] fix for moved insecure security transport --- p2p/net/upgrader/listener_test.go | 10 +++++----- p2p/net/upgrader/upgrader.go | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go index 7be94667a0..45418053d7 100644 --- a/p2p/net/upgrader/listener_test.go +++ b/p2p/net/upgrader/listener_test.go @@ -7,11 +7,11 @@ import ( "sync" "time" + insecure "github.com/libp2p/go-conn-security/insecure" peer "github.com/libp2p/go-libp2p-peer" - st "github.com/libp2p/go-libp2p-transport-upgrader" tpt "github.com/libp2p/go-libp2p-transport" + st "github.com/libp2p/go-libp2p-transport-upgrader" smux "github.com/libp2p/go-stream-muxer" - ss "github.com/libp2p/go-conn-security" tcp "github.com/libp2p/go-tcp-transport" ma "github.com/multiformats/go-multiaddr" yamux "github.com/whyrusleeping/go-smux-yamux" @@ -64,7 +64,7 @@ func (m *errorMuxer) NewConn(c net.Conn, isServer bool) (smux.Conn, error) { var _ = Describe("Listener", func() { var ( defaultUpgrader = &st.Upgrader{ - Secure: ss.NewInsecureTransport(peer.ID(1)), + Secure: insecure.New(peer.ID(1)), Muxer: &negotiatingMuxer{}, } ) @@ -136,7 +136,7 @@ var _ = Describe("Listener", func() { It("doesn't accept connections that fail to setup", func() { upgrader := &st.Upgrader{ - Secure: ss.NewInsecureTransport(peer.ID(1)), + Secure: insecure.New(peer.ID(1)), Muxer: &errorMuxer{}, } ln := createListener(upgrader) @@ -158,7 +158,7 @@ var _ = Describe("Listener", func() { num := 3 * st.AcceptQueueLength bm := newBlockingMuxer() upgrader := &st.Upgrader{ - Secure: ss.NewInsecureTransport(peer.ID(1)), + Secure: insecure.New(peer.ID(1)), Muxer: bm, } ln := createListener(upgrader) diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index 392d65da03..7bc8fd7675 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -6,12 +6,12 @@ import ( "fmt" "net" + ss "github.com/libp2p/go-conn-security" pnet "github.com/libp2p/go-libp2p-interface-pnet" peer "github.com/libp2p/go-libp2p-peer" transport "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" smux "github.com/libp2p/go-stream-muxer" - ss "github.com/libp2p/go-conn-security" manet "github.com/multiformats/go-multiaddr-net" ) From 6fbbb671e114db456181288f89bac9fc37d96138 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 15 Feb 2018 19:57:41 -0800 Subject: [PATCH 0402/3965] check if connection is closed before returning it from Accept It could have been sitting around for a while. --- p2p/net/upgrader/listener.go | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index 18005593aa..1a4ddd57b8 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -125,8 +125,11 @@ func (l *listener) handleIncoming() { // Accept accepts a connection. func (l *listener) Accept() (transport.Conn, error) { - if c, ok := <-l.incoming; ok { - return c, nil + for c := range l.incoming { + // Could have been sitting there for a while. + if !c.IsClosed() { + return c, nil + } } return nil, l.err } From 1b0d97811f07a32b01c3edf6d85af81a8cc4ae66 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 2 Feb 2018 11:14:00 +0800 Subject: [PATCH 0403/3965] verify the server's certificate using tls.Config.VerifyPeerCertificate Fixes #2. --- p2p/transport/quic/conn_test.go | 8 ++++---- p2p/transport/quic/crypto.go | 17 +++++------------ p2p/transport/quic/listener.go | 2 +- p2p/transport/quic/transport.go | 30 +++++++++++++++++++++--------- 4 files changed, 31 insertions(+), 26 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 473a50c6e9..66d9d2f7cb 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -80,14 +80,14 @@ var _ = Describe("Connection", func() { thirdPartyID, err := peer.IDFromPrivateKey(createPeer()) Expect(err).ToNot(HaveOccurred()) - serverAddrChan, _ := runServer() + serverAddrChan, serverConnChan := runServer() clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) serverAddr := <-serverAddrChan // dial, but expect the wrong peer ID _, err = clientTransport.Dial(context.Background(), serverAddr, thirdPartyID) - Expect(err).To(MatchError("peer IDs don't match")) - // TODO(#2): don't accept a connection if the client's peer verification fails - // Consistently(serverConnChan).ShouldNot(Receive()) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("TLS handshake error: bad certificate")) + Consistently(serverConnChan).ShouldNot(Receive()) }) }) diff --git a/p2p/transport/quic/crypto.go b/p2p/transport/quic/crypto.go index a6dee953d7..1c6a236000 100644 --- a/p2p/transport/quic/crypto.go +++ b/p2p/transport/quic/crypto.go @@ -10,8 +10,6 @@ import ( "math/big" "time" - "github.com/lucas-clemente/quic-go" - "github.com/gogo/protobuf/proto" ic "github.com/libp2p/go-libp2p-crypto" pb "github.com/libp2p/go-libp2p-crypto/pb" @@ -20,10 +18,6 @@ import ( // mint certificate selection is broken. const hostname = "quic.ipfs" -type connectionStater interface { - ConnectionState() quic.ConnectionState -} - // TODO: make this private func GenerateConfig(privKey ic.PrivKey) (*tls.Config, error) { key, hostCert, err := keyToCertificate(privKey) @@ -63,17 +57,16 @@ func GenerateConfig(privKey ic.PrivKey) (*tls.Config, error) { }, nil } -func getRemotePubKey(conn connectionStater) (ic.PubKey, error) { - certChain := conn.ConnectionState().PeerCertificates - if len(certChain) != 2 { +func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { + if len(chain) != 2 { return nil, errors.New("expected 2 certificates in the chain") } pool := x509.NewCertPool() - pool.AddCert(certChain[1]) - if _, err := certChain[0].Verify(x509.VerifyOptions{Roots: pool}); err != nil { + pool.AddCert(chain[1]) + if _, err := chain[0].Verify(x509.VerifyOptions{Roots: pool}); err != nil { return nil, err } - remotePubKey, err := x509.MarshalPKIXPublicKey(certChain[1].PublicKey) + remotePubKey, err := x509.MarshalPKIXPublicKey(chain[1].PublicKey) if err != nil { return nil, err } diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index ab044add60..3ff6b8945a 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -60,7 +60,7 @@ func (l *listener) Accept() (tpt.Conn, error) { if err != nil { return nil, err } - remotePubKey, err := getRemotePubKey(sess) + remotePubKey, err := getRemotePubKey(sess.ConnectionState().PeerCertificates) if err != nil { return nil, err } diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 6e05f63d64..c655569d30 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -2,6 +2,7 @@ package libp2pquic import ( "context" + "crypto/x509" "errors" ic "github.com/libp2p/go-libp2p-crypto" @@ -45,11 +46,27 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - sess, err := quicDialAddr(host, tlsConf, &quic.Config{Versions: []quic.VersionNumber{101}}) - if err != nil { - return nil, err + var remotePubKey ic.PubKey + tlsConf.VerifyPeerCertificate = func(rawCerts [][]byte, _ [][]*x509.Certificate) error { + chain := make([]*x509.Certificate, len(rawCerts)) + for i := 0; i < len(rawCerts); i++ { + cert, err := x509.ParseCertificate(rawCerts[i]) + if err != nil { + return err + } + chain[i] = cert + } + var err error + remotePubKey, err = getRemotePubKey(chain) + if err != nil { + return err + } + if !p.MatchesPublicKey(remotePubKey) { + return errors.New("peer IDs don't match") + } + return nil } - remotePubKey, err := getRemotePubKey(sess) + sess, err := quicDialAddr(host, tlsConf, &quic.Config{Versions: []quic.VersionNumber{101}}) if err != nil { return nil, err } @@ -57,11 +74,6 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - if !p.MatchesPublicKey(remotePubKey) { - err := errors.New("peer IDs don't match") - sess.Close(err) - return nil, err - } return &conn{ privKey: t.privKey, localPeer: t.localPeer, From 55a8a75431a6f10ff318bbc6f229f21de6f19872 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 22 Feb 2018 12:32:38 +0800 Subject: [PATCH 0404/3965] fix opening and accepting of streams --- p2p/transport/quic/conn_test.go | 21 +++++++++++++++++++++ p2p/transport/quic/transport.go | 1 + 2 files changed, 22 insertions(+) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 66d9d2f7cb..765e711fc8 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -5,6 +5,7 @@ import ( "crypto/rand" "crypto/rsa" "crypto/x509" + "io/ioutil" ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" @@ -76,6 +77,26 @@ var _ = Describe("Connection", func() { Expect(serverConn.RemotePublicKey()).To(Equal(clientKey.GetPublic())) }) + It("opens and accepts streams", func() { + serverAddrChan, serverConnChan := runServer() + clientTransport, err := NewTransport(clientKey) + Expect(err).ToNot(HaveOccurred()) + conn, err := clientTransport.Dial(context.Background(), <-serverAddrChan, serverID) + Expect(err).ToNot(HaveOccurred()) + serverConn := <-serverConnChan + + str, err := conn.OpenStream() + Expect(err).ToNot(HaveOccurred()) + _, err = str.Write([]byte("foobar")) + Expect(err).ToNot(HaveOccurred()) + str.Close() + sstr, err := serverConn.AcceptStream() + Expect(err).ToNot(HaveOccurred()) + data, err := ioutil.ReadAll(sstr) + Expect(err).ToNot(HaveOccurred()) + Expect(data).To(Equal([]byte("foobar"))) + }) + It("fails if the peer ID doesn't match", func() { thirdPartyID, err := peer.IDFromPrivateKey(createPeer()) Expect(err).ToNot(HaveOccurred()) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index c655569d30..5339a0ab85 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -75,6 +75,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp return nil, err } return &conn{ + sess: sess, privKey: t.privKey, localPeer: t.localPeer, localMultiaddr: localMultiaddr, From 93810dcfc9bbb7f6dbaf13fbd18bdff509425c5e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 22 Feb 2018 12:38:57 +0800 Subject: [PATCH 0405/3965] privatize generating the tls.Config --- p2p/transport/quic/crypto.go | 3 +-- p2p/transport/quic/listener.go | 2 +- p2p/transport/quic/transport.go | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/p2p/transport/quic/crypto.go b/p2p/transport/quic/crypto.go index 1c6a236000..5834a83fcc 100644 --- a/p2p/transport/quic/crypto.go +++ b/p2p/transport/quic/crypto.go @@ -18,8 +18,7 @@ import ( // mint certificate selection is broken. const hostname = "quic.ipfs" -// TODO: make this private -func GenerateConfig(privKey ic.PrivKey) (*tls.Config, error) { +func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { key, hostCert, err := keyToCertificate(privKey) if err != nil { return nil, err diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 3ff6b8945a..b85800c57e 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -32,7 +32,7 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, if err != nil { return nil, err } - tlsConf, err := GenerateConfig(key) + tlsConf, err := generateConfig(key) if err != nil { return nil, err } diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 5339a0ab85..eae98de6eb 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -42,7 +42,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - tlsConf, err := GenerateConfig(t.privKey) + tlsConf, err := generateConfig(t.privKey) if err != nil { return nil, err } From 02aa6480abcfdaf8887dcfb186be29a4044098c6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 7 Mar 2018 20:00:03 -0800 Subject: [PATCH 0406/3965] correctly handle Read errors/EOF Read can read data *and* return an error. --- p2p/net/pnet/psk_conn.go | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index a6e4247185..b9fc750b3c 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -43,13 +43,10 @@ func (c *pskConn) Read(out []byte) (int, error) { in = in[:maxn] // truncate to required length n, err := c.Conn.Read(in) // read to in - if err != nil { - return 0, err + if n > 0 { + c.readS20.XORKeyStream(out[:n], in[:n]) // decrypt to out buffer } - - c.readS20.XORKeyStream(out[:n], in[:n]) // decrypt to out buffer - - return n, nil + return n, err } func (c *pskConn) Write(in []byte) (int, error) { From 9bcb1d53a6b0b474c1aab9373d323d210b60e6b0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 8 Mar 2018 10:46:19 -0800 Subject: [PATCH 0407/3965] more correctly handle close and errors 1. Only propagate close on successful EOF. 2. Reset *both* streams on error. --- .../internal/circuitv1-deprecated/relay.go | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index edeafc9a84..91ba7f23c6 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -293,19 +293,29 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { // error, not an EOF. go func() { count, err := io.Copy(s, bs) - if err != io.EOF && err != nil { + if err != nil { log.Debugf("relay copy error: %s", err) + // Reset both. + s.Reset() + bs.Reset() + } else { + // propagate the close + s.Close() } - s.Close() log.Debugf("relayed %d bytes from %s to %s", count, dst.ID.Pretty(), src.ID.Pretty()) }() go func() { count, err := io.Copy(bs, s) - if err != io.EOF && err != nil { + if err != nil { log.Debugf("relay copy error: %s", err) + // Reset both. + bs.Reset() + s.Reset() + } else { + // propagate the close + bs.Close() } - bs.Close() log.Debugf("relayed %d bytes from %s to %s", count, src.ID.Pretty(), dst.ID.Pretty()) }() } From b64da1fd89c2a263ec22c1613d9a4c195cf41e32 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 8 Mar 2018 10:51:35 -0800 Subject: [PATCH 0408/3965] test resetting relay connections --- .../circuitv1-deprecated/relay_test.go | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 05c1154015..0b655e2ad3 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -111,6 +111,67 @@ func TestBasicRelay(t *testing.T) { } } +func TestRelayReset(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 3) + + connect(t, hosts[0], hosts[1]) + connect(t, hosts[1], hosts[2]) + + time.Sleep(10 * time.Millisecond) + + r1, err := NewRelay(ctx, hosts[0]) + if err != nil { + t.Fatal(err) + } + + _, err = NewRelay(ctx, hosts[1], OptHop) + if err != nil { + t.Fatal(err) + } + + r3, err := NewRelay(ctx, hosts[2]) + if err != nil { + t.Fatal(err) + } + + msg := []byte("relay works!") + go func() { + list := r3.Listener() + + con, err := list.Accept() + if err != nil { + t.Error(err) + return + } + + _, err = con.Write(msg) + if err != nil { + t.Error(err) + return + } + hosts[2].Close() + }() + + rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) + dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) + + rctx, rcancel := context.WithTimeout(ctx, time.Second) + defer rcancel() + + con, err := r1.DialPeer(rctx, rinfo, dinfo) + if err != nil { + t.Fatal(err) + } + + _, err = ioutil.ReadAll(con) + if err == nil { + t.Fatal("expected error for reset relayed connection") + } +} + func TestBasicRelayDial(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() From 7d45aa84ea7b79aadb66e1c457e806bcedd610a1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 8 Mar 2018 20:51:56 -0800 Subject: [PATCH 0409/3965] don't leak connections when canceling dials --- p2p/net/swarm/limiter.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index c3d8951894..836edbc9f5 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -166,5 +166,8 @@ func (dl *dialLimiter) executeDial(j *dialJob) { select { case j.resp <- dialResult{Conn: con, Addr: j.addr, Err: err}: case <-j.ctx.Done(): + if err == nil { + con.Close() + } } } From 07e87ba7ae4cc75f7497ce6e68a3955068721602 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 8 Mar 2018 21:29:16 -0800 Subject: [PATCH 0410/3965] ensure we force use of the libp2p protector if ForcePrivateNetwork is set --- p2p/net/upgrader/upgrader.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index 7bc8fd7675..bfbfa7d017 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -77,6 +77,10 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma return nil, err } conn = pconn + } else if pnet.ForcePrivateNetwork { + log.Error("tried to dial with no Private Network Protector but usage" + + " of Private Networks is forced by the enviroment") + return nil, pnet.ErrNotInPrivateNetwork } sconn, err := u.setupSecurity(ctx, conn, p) if err != nil { From 32b0643327821ba8b63adf9766808a8fb5865d5d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 9 Mar 2018 10:44:29 -0800 Subject: [PATCH 0411/3965] better document handleIncoming --- p2p/net/upgrader/listener.go | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index 1a4ddd57b8..fa26825fd5 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -49,6 +49,15 @@ func (l *listener) Close() error { return err } +// handles inbound connections. +// +// This function does a few interesting things that should be noted: +// +// 1. It logs and discards temporary/transient errors (errors with a Temporary() +// function that returns true). +// 2. It stops accepting new connections once AcceptQueueLength connections have +// been fully negotiated but not accepted. This gives us a basic backpressure +// mechanism while still allowing us to negotiate connections in parallel. func (l *listener) handleIncoming() { var wg sync.WaitGroup defer func() { @@ -66,6 +75,7 @@ func (l *listener) handleIncoming() { for l.ctx.Err() == nil { maconn, err := l.Listener.Accept() if err != nil { + // Note: function may pause the accept loop. if catcher.IsTemporary(err) { log.Infof("temporary accept error: %s", err) continue @@ -74,7 +84,7 @@ func (l *listener) handleIncoming() { return } - // The go routine above calls Release when the context is + // The go routine below calls Release when the context is // canceled so there's no need to wait on it here. l.threshold.Wait() @@ -103,6 +113,11 @@ func (l *listener) handleIncoming() { log.Debugf("listener %s accepted connection: %s", l, conn) + // This records the fact that the connection has been + // setup and is waiting to be accepted. This call + // *never* blocks, even if we go over the threshold. It + // simply ensures that calls to Wait block while we're + // over the threshold. l.threshold.Acquire() defer l.threshold.Release() From dbf3f835e261a2ad06100d0ca45bbdb78b809fa9 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 9 Mar 2018 16:46:51 -0800 Subject: [PATCH 0412/3965] remove dependency on the tcp transport fixes #7 --- p2p/net/upgrader/listener_test.go | 32 +++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go index 45418053d7..e15ca0db6d 100644 --- a/p2p/net/upgrader/listener_test.go +++ b/p2p/net/upgrader/listener_test.go @@ -12,8 +12,8 @@ import ( tpt "github.com/libp2p/go-libp2p-transport" st "github.com/libp2p/go-libp2p-transport-upgrader" smux "github.com/libp2p/go-stream-muxer" - tcp "github.com/libp2p/go-tcp-transport" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" yamux "github.com/whyrusleeping/go-smux-yamux" . "github.com/onsi/ginkgo" @@ -85,9 +85,17 @@ var _ = Describe("Listener", func() { createListener := func(upgrader *st.Upgrader) tpt.Listener { addr, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") ExpectWithOffset(0, err).ToNot(HaveOccurred()) - ln, err := tcp.NewTCPTransport(upgrader).Listen(addr) + ln, err := manet.Listen(addr) ExpectWithOffset(0, err).ToNot(HaveOccurred()) - return ln + return upgrader.UpgradeListener(nil, ln) + } + + dial := func(upgrader *st.Upgrader, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { + macon, err := manet.Dial(raddr) + if err != nil { + return nil, err + } + return upgrader.UpgradeOutbound(context.Background(), nil, macon, p) } BeforeEach(func() { @@ -96,7 +104,7 @@ var _ = Describe("Listener", func() { It("accepts a single connection", func() { ln := createListener(defaultUpgrader) - cconn, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(1)) + cconn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) Expect(err).ToNot(HaveOccurred()) sconn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) @@ -107,7 +115,7 @@ var _ = Describe("Listener", func() { ln := createListener(defaultUpgrader) const num = 10 for i := 0; i < 10; i++ { - cconn, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(1)) + cconn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) Expect(err).ToNot(HaveOccurred()) sconn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) @@ -119,7 +127,7 @@ var _ = Describe("Listener", func() { const timeout = 200 * time.Millisecond tpt.AcceptTimeout = timeout ln := createListener(defaultUpgrader) - conn, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(err).ToNot(HaveOccurred()) done := make(chan struct{}) go func() { @@ -146,7 +154,7 @@ var _ = Describe("Listener", func() { _, _ = ln.Accept() close(done) }() - _, _ = tcp.NewTCPTransport(upgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + _, _ = dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Consistently(done).ShouldNot(BeClosed()) // make the goroutine return ln.Close() @@ -179,7 +187,7 @@ var _ = Describe("Listener", func() { wg.Add(1) go func() { defer GinkgoRecover() - _, err := tcp.NewTCPTransport(upgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + _, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(err).ToNot(HaveOccurred()) wg.Done() }() @@ -198,7 +206,7 @@ var _ = Describe("Listener", func() { for i := 0; i < st.AcceptQueueLength; i++ { go func() { defer GinkgoRecover() - _, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + _, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(err).ToNot(HaveOccurred()) dialed <- struct{}{} }() @@ -207,7 +215,7 @@ var _ = Describe("Listener", func() { // dial a new connection. This connection should not complete setup, since the queue is full go func() { defer GinkgoRecover() - _, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + _, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(err).ToNot(HaveOccurred()) dialed <- struct{}{} }() @@ -238,13 +246,13 @@ var _ = Describe("Listener", func() { It("doesn't accept new connections when it is closed", func() { ln := createListener(defaultUpgrader) Expect(ln.Close()).To(Succeed()) - _, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(1)) + _, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) Expect(err).To(HaveOccurred()) }) It("closes incoming connections that have not yet been accepted", func() { ln := createListener(defaultUpgrader) - conn, err := tcp.NewTCPTransport(defaultUpgrader).Dial(context.Background(), ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(conn.IsClosed()).To(BeFalse()) Expect(err).ToNot(HaveOccurred()) Expect(ln.Close()).To(Succeed()) From 66b1a470e5b36bc01ebf2ef496109df735e875c6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 9 Mar 2018 16:48:23 -0800 Subject: [PATCH 0413/3965] use mplex for tests instead of yamux --- p2p/net/upgrader/listener_test.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go index e15ca0db6d..b216c462e3 100644 --- a/p2p/net/upgrader/listener_test.go +++ b/p2p/net/upgrader/listener_test.go @@ -14,13 +14,13 @@ import ( smux "github.com/libp2p/go-stream-muxer" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" - yamux "github.com/whyrusleeping/go-smux-yamux" + mplex "github.com/whyrusleeping/go-smux-multiplex" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) -// negotiatingMuxer sets up a new yamux connection +// negotiatingMuxer sets up a new mplex connection // It makes sure that this happens at the same time for client and server. type negotiatingMuxer struct{} @@ -35,7 +35,7 @@ func (m *negotiatingMuxer) NewConn(c net.Conn, isServer bool) (smux.Conn, error) if err != nil { return nil, err } - return yamux.DefaultTransport.NewConn(c, isServer) + return mplex.DefaultTransport.NewConn(c, isServer) } // blockingMuxer blocks the muxer negotiation until the contain chan is closed From a88cd13c3f7334722acadefdd53904c4156d85fc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 9 Mar 2018 22:11:41 -0800 Subject: [PATCH 0414/3965] unexport ReuseErrShouldRetry There's no reason a consumer of this package would use this error as *we* retry internally. Exporting it will just confuse users. --- p2p/net/reuseport/reuseport.go | 6 +++--- p2p/net/reuseport/reuseport_test.go | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/net/reuseport/reuseport.go b/p2p/net/reuseport/reuseport.go index 307f4ae4d1..1d9602e456 100644 --- a/p2p/net/reuseport/reuseport.go +++ b/p2p/net/reuseport/reuseport.go @@ -8,10 +8,10 @@ import ( reuseport "github.com/libp2p/go-reuseport" ) -// ReuseErrShouldRetry diagnoses whether to retry after a reuse error. +// reuseErrShouldRetry diagnoses whether to retry after a reuse error. // if we failed to bind, we should retry. if bind worked and this is a // real dial error (remote end didnt answer) then we should not retry. -func ReuseErrShouldRetry(err error) bool { +func reuseErrShouldRetry(err error) bool { if err == nil { return false // hey, it worked! no need to retry. } @@ -54,7 +54,7 @@ func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) ( return con, nil } - if ReuseErrShouldRetry(err) && ctx.Err() == nil { + if reuseErrShouldRetry(err) && ctx.Err() == nil { log.Debugf("failed to reuse port, dialing with a random port: %s", err) var d net.Dialer con, err = d.DialContext(ctx, network, raddr) diff --git a/p2p/net/reuseport/reuseport_test.go b/p2p/net/reuseport/reuseport_test.go index a4d8b1e2b6..b0ee4de6a1 100644 --- a/p2p/net/reuseport/reuseport_test.go +++ b/p2p/net/reuseport/reuseport_test.go @@ -40,7 +40,7 @@ func TestReuseError(t *testing.T) { } for k, v := range cases { - if ReuseErrShouldRetry(k) != v { + if reuseErrShouldRetry(k) != v { t.Fatalf("expected %t for %#v", v, k) } } From 8eef351ca804a99701ae461ecf27d14e7e2d48c3 Mon Sep 17 00:00:00 2001 From: Dmitriy Ryajov Date: Sun, 11 Mar 2018 12:26:42 -0600 Subject: [PATCH 0415/3965] fix: dont error out on unknown addrs in peer info --- p2p/protocol/internal/circuitv1-deprecated/util.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index b06c1aafe1..18aec9ebcc 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -29,9 +29,8 @@ func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { for i := 0; i < len(addrs); i++ { a, err := ma.NewMultiaddrBytes(p.Addrs[i]) if err != nil { - return pstore.PeerInfo{}, err + addrs[i] = a } - addrs[i] = a } return pstore.PeerInfo{ID: peer.ID(h), Addrs: addrs}, nil From 3e810f1c9570c3c2db288966cd37386f36f49aa9 Mon Sep 17 00:00:00 2001 From: Dmitriy Ryajov Date: Sun, 11 Mar 2018 13:25:15 -0600 Subject: [PATCH 0416/3965] fix: != to == --- p2p/protocol/internal/circuitv1-deprecated/util.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index 18aec9ebcc..b35c3cd504 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -28,7 +28,7 @@ func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { addrs := make([]ma.Multiaddr, len(p.Addrs)) for i := 0; i < len(addrs); i++ { a, err := ma.NewMultiaddrBytes(p.Addrs[i]) - if err != nil { + if err == nil { addrs[i] = a } } From b78f1451884d34cc708eeee3f8511c650867286d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 11 Mar 2018 12:59:37 -0700 Subject: [PATCH 0417/3965] fix race in reset test --- p2p/protocol/internal/circuitv1-deprecated/relay_test.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 0b655e2ad3..30181f0604 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -137,6 +137,8 @@ func TestRelayReset(t *testing.T) { t.Fatal(err) } + ready := make(chan struct{}) + msg := []byte("relay works!") go func() { list := r3.Listener() @@ -152,6 +154,8 @@ func TestRelayReset(t *testing.T) { t.Error(err) return } + <-ready + hosts[2].Close() }() @@ -166,6 +170,8 @@ func TestRelayReset(t *testing.T) { t.Fatal(err) } + close(ready) + _, err = ioutil.ReadAll(con) if err == nil { t.Fatal("expected error for reset relayed connection") From 5e68d0d1baedf24d66a05372b1d6a0027132ba7f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 12 Mar 2018 09:52:20 -0700 Subject: [PATCH 0418/3965] avoid putting nil multadders in the PeerInfo code nit: use the range operator when possible. --- p2p/protocol/internal/circuitv1-deprecated/util.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index b35c3cd504..71f4684ea8 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -25,11 +25,11 @@ func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { return pstore.PeerInfo{}, err } - addrs := make([]ma.Multiaddr, len(p.Addrs)) - for i := 0; i < len(addrs); i++ { - a, err := ma.NewMultiaddrBytes(p.Addrs[i]) + addrs := make([]ma.Multiaddr, 0, len(p.Addrs)) + for _, addrBytes := range p.Addrs { + a, err := ma.NewMultiaddrBytes(addrBytes) if err == nil { - addrs[i] = a + addrs = append(addrs, a) } } @@ -38,8 +38,8 @@ func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { func peerInfoToPeer(pi pstore.PeerInfo) *pb.CircuitRelay_Peer { addrs := make([][]byte, len(pi.Addrs)) - for i := 0; i < len(addrs); i++ { - addrs[i] = pi.Addrs[i].Bytes() + for i, addr := range pi.Addrs { + addrs[i] = addr.Bytes() } p := new(pb.CircuitRelay_Peer) From e8586130fba9ca0c6a326cc74f5eb6d5ca2ea096 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 14 Mar 2018 15:54:59 -0700 Subject: [PATCH 0419/3965] go fmt --- p2p/net/conn-security-multistream/ssms.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/conn-security-multistream/ssms.go b/p2p/net/conn-security-multistream/ssms.go index 75bbb10f44..312b775cd3 100644 --- a/p2p/net/conn-security-multistream/ssms.go +++ b/p2p/net/conn-security-multistream/ssms.go @@ -5,8 +5,8 @@ import ( "fmt" "net" - peer "github.com/libp2p/go-libp2p-peer" connsec "github.com/libp2p/go-conn-security" + peer "github.com/libp2p/go-libp2p-peer" mss "github.com/multiformats/go-multistream" ) From 5a60c83e50de398cab169951149e07fec3791c52 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 14 Mar 2018 15:55:05 -0700 Subject: [PATCH 0420/3965] fix tests for go-conn-security refactor --- .../conn-security-multistream/ssms_test.go | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/p2p/net/conn-security-multistream/ssms_test.go b/p2p/net/conn-security-multistream/ssms_test.go index 5d72458562..457c1ce644 100644 --- a/p2p/net/conn-security-multistream/ssms_test.go +++ b/p2p/net/conn-security-multistream/ssms_test.go @@ -6,27 +6,27 @@ import ( "sync" "testing" - connsec "github.com/libp2p/go-conn-security" + insecure "github.com/libp2p/go-conn-security/insecure" sst "github.com/libp2p/go-conn-security/test" ) func TestCommonProto(t *testing.T) { var at, bt SSMuxer - atInsecure := connsec.InsecureTransport("peerA") - btInsecure := connsec.InsecureTransport("peerB") - at.AddTransport("/plaintext/1.0.0", &atInsecure) - bt.AddTransport("/plaintext/1.1.0", &btInsecure) - bt.AddTransport("/plaintext/1.0.0", &btInsecure) + atInsecure := insecure.New("peerA") + btInsecure := insecure.New("peerB") + at.AddTransport("/plaintext/1.0.0", atInsecure) + bt.AddTransport("/plaintext/1.1.0", btInsecure) + bt.AddTransport("/plaintext/1.0.0", btInsecure) sst.SubtestRW(t, &at, &bt, "peerA", "peerB") } func TestNoCommonProto(t *testing.T) { var at, bt SSMuxer - atInsecure := connsec.InsecureTransport("peerA") - btInsecure := connsec.InsecureTransport("peerB") + atInsecure := insecure.New("peerA") + btInsecure := insecure.New("peerB") - at.AddTransport("/plaintext/1.0.0", &atInsecure) - bt.AddTransport("/plaintext/1.1.0", &btInsecure) + at.AddTransport("/plaintext/1.0.0", atInsecure) + bt.AddTransport("/plaintext/1.1.0", btInsecure) ctx, cancel := context.WithCancel(context.Background()) defer cancel() From 12f1c233332596bd20a04c77bf7748ff4296b39e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 15 Mar 2018 12:20:13 -0700 Subject: [PATCH 0421/3965] fix the relay reset test again This should be the final time I need to do this. We need to close the connection to the relay *before* closing the connection to other node. --- p2p/protocol/internal/circuitv1-deprecated/relay_test.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 30181f0604..5213e122bd 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -149,14 +149,15 @@ func TestRelayReset(t *testing.T) { return } + <-ready + _, err = con.Write(msg) if err != nil { t.Error(err) return } - <-ready - hosts[2].Close() + hosts[2].Network().ClosePeer(hosts[1].ID()) }() rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) From 1af0f4224ccb650592e96fe2f5ee188f0ab0c30d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 27 Mar 2018 13:07:48 -0700 Subject: [PATCH 0422/3965] document source port choosing algorithm --- p2p/net/reuseport/multidialer.go | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/p2p/net/reuseport/multidialer.go b/p2p/net/reuseport/multidialer.go index b8f4129f14..7098637fc4 100644 --- a/p2p/net/reuseport/multidialer.go +++ b/p2p/net/reuseport/multidialer.go @@ -30,6 +30,28 @@ func (d *multiDialer) DialContext(ctx context.Context, network, addr string) (ne return nil, err } + // We pick the source *port* based on the following algorithm. + // + // 1. If we're dialing loopback, choose a source-port in order of + // preference: + // 1. A port in-use by an explicit loopback listener. + // 2. A port in-use by a listener on an unspecified address (must + // also be listening on localhost). + // 3. A port in-use by a listener on a global address. We don't have + // any other better options (other than picking a random port). + // 2. If we're dialing a global address, choose a source-port in order + // of preference: + // 1. A port in-use by a listener on an unspecified address (the most + // general case). + // 2. A port in-use by a listener on the global address. + // 3. Fail on link-local dials (go-ipfs currently forbids this and I + // figured we could try lifting this restriction later). + // + // + // Note: We *always* dial from the unspecified address (regardless of + // the port we pick). In the future, we could use netlink (on Linux) to + // figure out the right source address but we're going to punt on that. + ip := tcpAddr.IP source := d.global switch { From 0b6d56b5e46bb74c8a8c63e9f36f295c3474ddcf Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 27 Mar 2018 13:07:56 -0700 Subject: [PATCH 0423/3965] define a global fallbackDialer indirectly addresses: https://github.com/libp2p/go-reuseport-transport/issues/1#issuecomment-376625179 --- p2p/net/reuseport/reuseport.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/net/reuseport/reuseport.go b/p2p/net/reuseport/reuseport.go index 1d9602e456..47ceac2278 100644 --- a/p2p/net/reuseport/reuseport.go +++ b/p2p/net/reuseport/reuseport.go @@ -8,6 +8,8 @@ import ( reuseport "github.com/libp2p/go-reuseport" ) +var fallbackDialer net.Dialer + // reuseErrShouldRetry diagnoses whether to retry after a reuse error. // if we failed to bind, we should retry. if bind worked and this is a // real dial error (remote end didnt answer) then we should not retry. @@ -39,8 +41,7 @@ func reuseErrShouldRetry(err error) bool { // Dials using reusport and then redials normally if that fails. func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) (net.Conn, error) { if laddr == nil { - var d net.Dialer - return d.DialContext(ctx, network, raddr) + return fallbackDialer.DialContext(ctx, network, raddr) } d := reuseport.Dialer{ @@ -56,8 +57,7 @@ func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) ( if reuseErrShouldRetry(err) && ctx.Err() == nil { log.Debugf("failed to reuse port, dialing with a random port: %s", err) - var d net.Dialer - con, err = d.DialContext(ctx, network, raddr) + con, err = fallbackDialer.DialContext(ctx, network, raddr) } return con, err } From 2f5a5fe89afb20d6334d948d61802dabab8f559e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 29 Mar 2018 00:08:11 -0700 Subject: [PATCH 0424/3965] add another test case --- p2p/net/reuseport/transport_test.go | 48 ++++++++++++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/p2p/net/reuseport/transport_test.go b/p2p/net/reuseport/transport_test.go index 1b3941f448..2564595f0c 100644 --- a/p2p/net/reuseport/transport_test.go +++ b/p2p/net/reuseport/transport_test.go @@ -1,12 +1,13 @@ package tcpreuse import ( + "net" "testing" ma "github.com/multiformats/go-multiaddr" ) -func TestAll(t *testing.T) { +func TestSingle(t *testing.T) { var trA Transport var trB Transport laddr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") @@ -38,3 +39,48 @@ func TestAll(t *testing.T) { <-done c.Close() } + +func TestTwoLocal(t *testing.T) { + var trA Transport + var trB Transport + laddr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") + listenerA, err := trA.Listen(laddr) + if err != nil { + t.Fatal(err) + } + defer listenerA.Close() + + listenerB1, err := trB.Listen(laddr) + if err != nil { + t.Fatal(err) + } + defer listenerB1.Close() + + listenerB2, err := trB.Listen(laddr) + if err != nil { + t.Fatal(err) + } + defer listenerB2.Close() + + done := make(chan struct{}) + go func() { + defer close(done) + c, err := listenerA.Accept() + if err != nil { + t.Fatal(err) + } + c.Close() + }() + + c, err := trB.Dial(listenerA.Multiaddr()) + if err != nil { + t.Fatal(err) + } + localPort := c.LocalAddr().(*net.TCPAddr).Port + if localPort != listenerB1.Addr().(*net.TCPAddr).Port && + localPort != listenerB2.Addr().(*net.TCPAddr).Port { + t.Fatal("didn't dial from one of our listener ports") + } + <-done + c.Close() +} From 95f137a647c33a34a97d836982193175ebd48716 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 29 Mar 2018 00:14:19 -0700 Subject: [PATCH 0425/3965] reset the dialer (which ports to dial from) when closing a listener --- p2p/net/reuseport/listen.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/net/reuseport/listen.go b/p2p/net/reuseport/listen.go index 04bd01b569..7b2a4c39d2 100644 --- a/p2p/net/reuseport/listen.go +++ b/p2p/net/reuseport/listen.go @@ -16,6 +16,7 @@ type listener struct { func (l *listener) Close() error { l.network.mu.Lock() delete(l.network.listeners, l) + l.network.dialer = nil l.network.mu.Unlock() return l.Listener.Close() } From bf8af04add48645254b4a224106f8cda54a5d96a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 29 Mar 2018 00:14:24 -0700 Subject: [PATCH 0426/3965] add a test for dialing with a local and an unspecified listener --- p2p/net/reuseport/transport_test.go | 71 +++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/p2p/net/reuseport/transport_test.go b/p2p/net/reuseport/transport_test.go index 2564595f0c..44d90b3956 100644 --- a/p2p/net/reuseport/transport_test.go +++ b/p2p/net/reuseport/transport_test.go @@ -84,3 +84,74 @@ func TestTwoLocal(t *testing.T) { <-done c.Close() } + +func TestLocalAndUnspecified(t *testing.T) { + var trA Transport + var trB Transport + laddr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") + unspec, _ := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") + listenerA, err := trA.Listen(laddr) + if err != nil { + t.Fatal(err) + } + defer listenerA.Close() + + listenerB1, err := trB.Listen(laddr) + if err != nil { + t.Fatal(err) + } + defer listenerB1.Close() + + listenerB2, err := trB.Listen(unspec) + if err != nil { + t.Fatal(err) + } + defer listenerB2.Close() + + done := make(chan struct{}) + go func() { + defer close(done) + c, err := listenerA.Accept() + if err != nil { + t.Fatal(err) + } + c.Close() + }() + + c, err := trB.Dial(listenerA.Multiaddr()) + if err != nil { + t.Fatal(err) + } + actual := c.LocalAddr().(*net.TCPAddr).Port + expected := listenerB1.Addr().(*net.TCPAddr).Port + if actual != expected { + t.Errorf("expected to use port %d, used port %d", expected, actual) + } + <-done + c.Close() + + // Closing the listener should reset the dialer. + listenerB1.Close() + + done = make(chan struct{}) + go func() { + defer close(done) + c, err := listenerA.Accept() + if err != nil { + t.Fatal(err) + } + c.Close() + }() + + c, err = trB.Dial(listenerA.Multiaddr()) + if err != nil { + t.Fatal(err) + } + actual = c.LocalAddr().(*net.TCPAddr).Port + expected = listenerB2.Addr().(*net.TCPAddr).Port + if actual != expected { + t.Errorf("expected to use port %d, used port %d", expected, actual) + } + <-done + c.Close() +} From 9eafb986cf7df5c75e13d6d869a7a1a3b829c3b1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 29 Mar 2018 00:18:10 -0700 Subject: [PATCH 0427/3965] test dialing with no listeners --- p2p/net/reuseport/transport_test.go | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/p2p/net/reuseport/transport_test.go b/p2p/net/reuseport/transport_test.go index 44d90b3956..4b15503530 100644 --- a/p2p/net/reuseport/transport_test.go +++ b/p2p/net/reuseport/transport_test.go @@ -7,7 +7,7 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -func TestSingle(t *testing.T) { +func TestNoneAndSingle(t *testing.T) { var trA Transport var trB Transport laddr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") @@ -16,13 +16,31 @@ func TestSingle(t *testing.T) { t.Fatal(err) } defer listenerA.Close() + + done := make(chan struct{}) + go func() { + defer close(done) + c, err := listenerA.Accept() + if err != nil { + t.Fatal(err) + } + c.Close() + }() + + c, err := trB.Dial(listenerA.Multiaddr()) + if err != nil { + t.Fatal(err) + } + <-done + c.Close() + listenerB, err := trB.Listen(laddr) if err != nil { t.Fatal(err) } defer listenerB.Close() - done := make(chan struct{}) + done = make(chan struct{}) go func() { defer close(done) c, err := listenerA.Accept() @@ -32,10 +50,15 @@ func TestSingle(t *testing.T) { c.Close() }() - c, err := trB.Dial(listenerA.Multiaddr()) + c, err = trB.Dial(listenerA.Multiaddr()) if err != nil { t.Fatal(err) } + actual := c.LocalAddr().(*net.TCPAddr).Port + expected := listenerB.Addr().(*net.TCPAddr).Port + if actual != expected { + t.Errorf("expected to use port %d, used port %d", expected, actual) + } <-done c.Close() } From bd345d7a98c4dbb19822d93d24d19d16911bd53d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 29 Mar 2018 00:55:57 -0700 Subject: [PATCH 0428/3965] fix use of global address condition --- p2p/net/reuseport/multidialer.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/reuseport/multidialer.go b/p2p/net/reuseport/multidialer.go index 7098637fc4..c7d388c144 100644 --- a/p2p/net/reuseport/multidialer.go +++ b/p2p/net/reuseport/multidialer.go @@ -94,7 +94,7 @@ func newMultiDialer(unspec net.IP, listeners map[*listener]struct{}) dialer { // find. // // TODO: Port priority? Addr priority? - if m.global != nil { + if m.global == nil { m.global = &net.TCPAddr{ IP: unspec, Port: laddr.Port, From 910deaeb2f8f5af0f25281faa9a34ba58b5ba82f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 29 Mar 2018 01:02:43 -0700 Subject: [PATCH 0429/3965] more tests --- p2p/net/reuseport/transport_test.go | 235 +++++++++++++++++----------- 1 file changed, 144 insertions(+), 91 deletions(-) diff --git a/p2p/net/reuseport/transport_test.go b/p2p/net/reuseport/transport_test.go index 4b15503530..e78d8c8221 100644 --- a/p2p/net/reuseport/transport_test.go +++ b/p2p/net/reuseport/transport_test.go @@ -5,176 +5,229 @@ import ( "testing" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) -func TestNoneAndSingle(t *testing.T) { - var trA Transport - var trB Transport - laddr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") - listenerA, err := trA.Listen(laddr) +var loopback, _ = ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") +var unspec, _ = ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") + +var global ma.Multiaddr + +func init() { + addrs, err := manet.InterfaceMultiaddrs() if err != nil { - t.Fatal(err) + return } - defer listenerA.Close() + for _, addr := range addrs { + if !manet.IsIP6LinkLocal(addr) && !manet.IsIPLoopback(addr) { + tcp, _ := ma.NewMultiaddr("/tcp/0") + global = addr.Encapsulate(tcp) + return + } + } +} +func acceptOne(t *testing.T, listener manet.Listener) <-chan struct{} { + t.Helper() done := make(chan struct{}) go func() { defer close(done) - c, err := listenerA.Accept() + c, err := listener.Accept() if err != nil { - t.Fatal(err) + t.Error(err) + return } c.Close() }() + return done +} + +func dialOne(t *testing.T, tr *Transport, listener manet.Listener, expected ...int) int { + t.Helper() - c, err := trB.Dial(listenerA.Multiaddr()) + done := acceptOne(t, listener) + c, err := tr.Dial(listener.Multiaddr()) if err != nil { t.Fatal(err) } + port := c.LocalAddr().(*net.TCPAddr).Port <-done c.Close() + if len(expected) == 0 { + return port + } + for _, p := range expected { + if p == port { + return port + } + } + t.Errorf("dialed from %d, expected to dial from one of %v", port, expected) + return 0 +} - listenerB, err := trB.Listen(laddr) +func TestNoneAndSingle(t *testing.T) { + var trA Transport + var trB Transport + listenerA, err := trA.Listen(loopback) if err != nil { t.Fatal(err) } - defer listenerB.Close() + defer listenerA.Close() - done = make(chan struct{}) - go func() { - defer close(done) - c, err := listenerA.Accept() - if err != nil { - t.Fatal(err) - } - c.Close() - }() + dialOne(t, &trB, listenerA) - c, err = trB.Dial(listenerA.Multiaddr()) + listenerB, err := trB.Listen(loopback) if err != nil { t.Fatal(err) } - actual := c.LocalAddr().(*net.TCPAddr).Port - expected := listenerB.Addr().(*net.TCPAddr).Port - if actual != expected { - t.Errorf("expected to use port %d, used port %d", expected, actual) - } - <-done - c.Close() + defer listenerB.Close() + + dialOne(t, &trB, listenerA, listenerB.Addr().(*net.TCPAddr).Port) } func TestTwoLocal(t *testing.T) { var trA Transport var trB Transport - laddr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") - listenerA, err := trA.Listen(laddr) + listenerA, err := trA.Listen(loopback) if err != nil { t.Fatal(err) } defer listenerA.Close() - listenerB1, err := trB.Listen(laddr) + listenerB1, err := trB.Listen(loopback) if err != nil { t.Fatal(err) } defer listenerB1.Close() - listenerB2, err := trB.Listen(laddr) + listenerB2, err := trB.Listen(loopback) if err != nil { t.Fatal(err) } defer listenerB2.Close() - done := make(chan struct{}) - go func() { - defer close(done) - c, err := listenerA.Accept() - if err != nil { - t.Fatal(err) - } - c.Close() - }() + dialOne(t, &trB, listenerA, + listenerB1.Addr().(*net.TCPAddr).Port, + listenerB2.Addr().(*net.TCPAddr).Port) +} - c, err := trB.Dial(listenerA.Multiaddr()) - if err != nil { - t.Fatal(err) - } - localPort := c.LocalAddr().(*net.TCPAddr).Port - if localPort != listenerB1.Addr().(*net.TCPAddr).Port && - localPort != listenerB2.Addr().(*net.TCPAddr).Port { - t.Fatal("didn't dial from one of our listener ports") +func TestGlobalPreference(t *testing.T) { + if global == nil { + t.Skip("no global addresses configured") + return } - <-done - c.Close() + testPrefer(t, loopback, loopback, global) + testPrefer(t, loopback, unspec, global) + + testPrefer(t, global, unspec, global) + testPrefer(t, global, unspec, loopback) } -func TestLocalAndUnspecified(t *testing.T) { +func TestLoopbackPreference(t *testing.T) { + testPrefer(t, loopback, loopback, unspec) +} + +func testPrefer(t *testing.T, listen, prefer, avoid ma.Multiaddr) { var trA Transport var trB Transport - laddr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") - unspec, _ := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") - listenerA, err := trA.Listen(laddr) + listenerA, err := trA.Listen(listen) if err != nil { t.Fatal(err) } defer listenerA.Close() - listenerB1, err := trB.Listen(laddr) + listenerB1, err := trB.Listen(avoid) if err != nil { t.Fatal(err) } defer listenerB1.Close() - listenerB2, err := trB.Listen(unspec) + dialOne(t, &trB, listenerA, listenerB1.Addr().(*net.TCPAddr).Port) + + listenerB2, err := trB.Listen(prefer) if err != nil { t.Fatal(err) } defer listenerB2.Close() - done := make(chan struct{}) - go func() { - defer close(done) - c, err := listenerA.Accept() - if err != nil { - t.Fatal(err) - } - c.Close() - }() + dialOne(t, &trB, listenerA, listenerB2.Addr().(*net.TCPAddr).Port) + + // Closing the listener should reset the dialer. + listenerB2.Close() + + dialOne(t, &trB, listenerA, listenerB1.Addr().(*net.TCPAddr).Port) +} + +func TestGlobalToGlobal(t *testing.T) { + if global == nil { + t.Skip("no global addresses configured") + return + } + + var trA Transport + var trB Transport + listenerA, err := trA.Listen(global) + if err != nil { + t.Fatal(err) + } + defer listenerA.Close() - c, err := trB.Dial(listenerA.Multiaddr()) + listenerB1, err := trB.Listen(loopback) if err != nil { t.Fatal(err) } - actual := c.LocalAddr().(*net.TCPAddr).Port - expected := listenerB1.Addr().(*net.TCPAddr).Port - if actual != expected { - t.Errorf("expected to use port %d, used port %d", expected, actual) + defer listenerB1.Close() + + // It works (random port) + dialOne(t, &trB, listenerA) + + listenerB2, err := trB.Listen(global) + if err != nil { + t.Fatal(err) } - <-done - c.Close() + defer listenerB2.Close() + + // Uses global port. + dialOne(t, &trB, listenerA, listenerB2.Addr().(*net.TCPAddr).Port) // Closing the listener should reset the dialer. - listenerB1.Close() + listenerB2.Close() - done = make(chan struct{}) - go func() { - defer close(done) - c, err := listenerA.Accept() - if err != nil { - t.Fatal(err) - } - c.Close() - }() + // It still works. + dialOne(t, &trB, listenerA) +} + +func TestDuplicateGlobal(t *testing.T) { + if global == nil { + t.Skip("no global addresses configured") + return + } + + var trA Transport + var trB Transport + listenerA, err := trA.Listen(global) + if err != nil { + t.Fatal(err) + } + defer listenerA.Close() - c, err = trB.Dial(listenerA.Multiaddr()) + listenerB1, err := trB.Listen(global) if err != nil { t.Fatal(err) } - actual = c.LocalAddr().(*net.TCPAddr).Port - expected = listenerB2.Addr().(*net.TCPAddr).Port - if actual != expected { - t.Errorf("expected to use port %d, used port %d", expected, actual) + defer listenerB1.Close() + + listenerB2, err := trB.Listen(global) + if err != nil { + t.Fatal(err) + } + defer listenerB2.Close() + + // Check which port we're using + port := dialOne(t, &trB, listenerA) + + // Check consistency + for i := 0; i < 10; i++ { + dialOne(t, &trB, listenerA, port) } - <-done - c.Close() } From 33bd60aa147f336e07db02e711e5b65f44608694 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 29 Mar 2018 01:14:22 -0700 Subject: [PATCH 0430/3965] test ipv6 --- p2p/net/reuseport/transport_test.go | 91 ++++++++++++++++++++--------- 1 file changed, 62 insertions(+), 29 deletions(-) diff --git a/p2p/net/reuseport/transport_test.go b/p2p/net/reuseport/transport_test.go index e78d8c8221..942019b0b4 100644 --- a/p2p/net/reuseport/transport_test.go +++ b/p2p/net/reuseport/transport_test.go @@ -8,10 +8,13 @@ import ( manet "github.com/multiformats/go-multiaddr-net" ) -var loopback, _ = ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") -var unspec, _ = ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") +var loopbackV4, _ = ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") +var loopbackV6, _ = ma.NewMultiaddr("/ip6/::1/tcp/0") +var unspecV6, _ = ma.NewMultiaddr("/ip6/::/tcp/0") +var unspecV4, _ = ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") -var global ma.Multiaddr +var globalV4 ma.Multiaddr +var globalV6 ma.Multiaddr func init() { addrs, err := manet.InterfaceMultiaddrs() @@ -21,8 +24,16 @@ func init() { for _, addr := range addrs { if !manet.IsIP6LinkLocal(addr) && !manet.IsIPLoopback(addr) { tcp, _ := ma.NewMultiaddr("/tcp/0") - global = addr.Encapsulate(tcp) - return + switch addr.Protocols()[0].Code { + case ma.P_IP4: + if globalV4 == nil { + globalV4 = addr.Encapsulate(tcp) + } + case ma.P_IP6: + if globalV6 == nil { + globalV6 = addr.Encapsulate(tcp) + } + } } } } @@ -68,7 +79,7 @@ func dialOne(t *testing.T, tr *Transport, listener manet.Listener, expected ...i func TestNoneAndSingle(t *testing.T) { var trA Transport var trB Transport - listenerA, err := trA.Listen(loopback) + listenerA, err := trA.Listen(loopbackV4) if err != nil { t.Fatal(err) } @@ -76,7 +87,7 @@ func TestNoneAndSingle(t *testing.T) { dialOne(t, &trB, listenerA) - listenerB, err := trB.Listen(loopback) + listenerB, err := trB.Listen(loopbackV4) if err != nil { t.Fatal(err) } @@ -88,19 +99,19 @@ func TestNoneAndSingle(t *testing.T) { func TestTwoLocal(t *testing.T) { var trA Transport var trB Transport - listenerA, err := trA.Listen(loopback) + listenerA, err := trA.Listen(loopbackV4) if err != nil { t.Fatal(err) } defer listenerA.Close() - listenerB1, err := trB.Listen(loopback) + listenerB1, err := trB.Listen(loopbackV4) if err != nil { t.Fatal(err) } defer listenerB1.Close() - listenerB2, err := trB.Listen(loopback) + listenerB2, err := trB.Listen(loopbackV4) if err != nil { t.Fatal(err) } @@ -111,20 +122,33 @@ func TestTwoLocal(t *testing.T) { listenerB2.Addr().(*net.TCPAddr).Port) } -func TestGlobalPreference(t *testing.T) { - if global == nil { - t.Skip("no global addresses configured") +func TestGlobalPreferenceV4(t *testing.T) { + if globalV4 == nil { + t.Skip("no global IPv4 addresses configured") return } - testPrefer(t, loopback, loopback, global) - testPrefer(t, loopback, unspec, global) + testPrefer(t, loopbackV4, loopbackV4, globalV4) + testPrefer(t, loopbackV4, unspecV4, globalV4) - testPrefer(t, global, unspec, global) - testPrefer(t, global, unspec, loopback) + testPrefer(t, globalV4, unspecV4, globalV4) + testPrefer(t, globalV4, unspecV4, loopbackV4) +} + +func TestGlobalPreferenceV6(t *testing.T) { + if globalV6 == nil { + t.Skip("no global IPv6 addresses configured") + return + } + testPrefer(t, loopbackV6, loopbackV6, globalV6) + testPrefer(t, loopbackV6, unspecV6, globalV6) + + testPrefer(t, globalV6, unspecV6, globalV6) + testPrefer(t, globalV6, unspecV6, loopbackV6) } func TestLoopbackPreference(t *testing.T) { - testPrefer(t, loopback, loopback, unspec) + testPrefer(t, loopbackV4, loopbackV4, unspecV4) + testPrefer(t, loopbackV6, loopbackV6, unspecV6) } func testPrefer(t *testing.T, listen, prefer, avoid ma.Multiaddr) { @@ -158,21 +182,30 @@ func testPrefer(t *testing.T, listen, prefer, avoid ma.Multiaddr) { dialOne(t, &trB, listenerA, listenerB1.Addr().(*net.TCPAddr).Port) } +func TestV6V4(t *testing.T) { + testUseFirst(t, loopbackV4, loopbackV4, loopbackV6) + testUseFirst(t, loopbackV6, loopbackV6, loopbackV4) +} + func TestGlobalToGlobal(t *testing.T) { - if global == nil { - t.Skip("no global addresses configured") + if globalV4 == nil { + t.Skip("no globalV4 addresses configured") return } + testUseFirst(t, globalV4, globalV4, loopbackV4) + testUseFirst(t, globalV6, globalV6, loopbackV6) +} +func testUseFirst(t *testing.T, listen, use, never ma.Multiaddr) { var trA Transport var trB Transport - listenerA, err := trA.Listen(global) + listenerA, err := trA.Listen(globalV4) if err != nil { t.Fatal(err) } defer listenerA.Close() - listenerB1, err := trB.Listen(loopback) + listenerB1, err := trB.Listen(loopbackV4) if err != nil { t.Fatal(err) } @@ -181,13 +214,13 @@ func TestGlobalToGlobal(t *testing.T) { // It works (random port) dialOne(t, &trB, listenerA) - listenerB2, err := trB.Listen(global) + listenerB2, err := trB.Listen(globalV4) if err != nil { t.Fatal(err) } defer listenerB2.Close() - // Uses global port. + // Uses globalV4 port. dialOne(t, &trB, listenerA, listenerB2.Addr().(*net.TCPAddr).Port) // Closing the listener should reset the dialer. @@ -198,26 +231,26 @@ func TestGlobalToGlobal(t *testing.T) { } func TestDuplicateGlobal(t *testing.T) { - if global == nil { - t.Skip("no global addresses configured") + if globalV4 == nil { + t.Skip("no globalV4 addresses configured") return } var trA Transport var trB Transport - listenerA, err := trA.Listen(global) + listenerA, err := trA.Listen(globalV4) if err != nil { t.Fatal(err) } defer listenerA.Close() - listenerB1, err := trB.Listen(global) + listenerB1, err := trB.Listen(globalV4) if err != nil { t.Fatal(err) } defer listenerB1.Close() - listenerB2, err := trB.Listen(global) + listenerB2, err := trB.Listen(globalV4) if err != nil { t.Fatal(err) } From 6ef12b596a0a6395d510cd2904d6658ae20e7990 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 16 Apr 2018 13:17:06 +0900 Subject: [PATCH 0431/3965] support extracting inlined public keys Technically, the caller can do this but this is more convenient. --- p2p/host/peerstore/peerstore.go | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index d0fe38c646..abfdeae6d3 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -127,6 +127,15 @@ func (kb *keybook) PubKey(p peer.ID) ic.PubKey { kb.RLock() pk := kb.pks[p] kb.RUnlock() + if pk != nil { + return pk + } + pk, err := p.ExtractPublicKey() + if err == nil && pk != nil { + kb.Lock() + kb.pks[p] = pk + kb.Unlock() + } return pk } From cec0aa6b9c219dfe31edccde608e3a6e5bafd2b1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 20 Apr 2018 09:28:12 +0900 Subject: [PATCH 0432/3965] annotate the "malformed key" error from pnet protector --- p2p/net/pnet/protector.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index 3442150a93..72446c2c6e 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -1,6 +1,7 @@ package pnet import ( + "fmt" "io" ipnet "github.com/libp2p/go-libp2p-interface-pnet" @@ -14,7 +15,7 @@ var _ ipnet.Protector = (*protector)(nil) func NewProtector(input io.Reader) (ipnet.Protector, error) { psk, err := decodeV1PSK(input) if err != nil { - return nil, err + return nil, fmt.Errorf("malformed private network key: %s", err) } return NewV1ProtectorFromBytes(psk) } From f0b7fc9cf3b1195bed4762b630917a96ace6dc4c Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 5 May 2018 16:12:39 +0300 Subject: [PATCH 0433/3965] protobuf --- p2p/host/autonat/pb/autonat.pb.go | 208 ++++++++++++++++++++++++++++++ p2p/host/autonat/pb/autonat.proto | 34 +++++ 2 files changed, 242 insertions(+) create mode 100644 p2p/host/autonat/pb/autonat.pb.go create mode 100644 p2p/host/autonat/pb/autonat.proto diff --git a/p2p/host/autonat/pb/autonat.pb.go b/p2p/host/autonat/pb/autonat.pb.go new file mode 100644 index 0000000000..79a09913b8 --- /dev/null +++ b/p2p/host/autonat/pb/autonat.pb.go @@ -0,0 +1,208 @@ +// Code generated by protoc-gen-gogo. +// source: autonat.proto +// DO NOT EDIT! + +/* +Package autonat_pb is a generated protocol buffer package. + +It is generated from these files: + autonat.proto + +It has these top-level messages: + Message +*/ +package autonat_pb + +import proto "github.com/gogo/protobuf/proto" +import fmt "fmt" +import math "math" + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +type Message_MessageType int32 + +const ( + Message_DIAL Message_MessageType = 0 + Message_DIAL_RESPONSE Message_MessageType = 1 +) + +var Message_MessageType_name = map[int32]string{ + 0: "DIAL", + 1: "DIAL_RESPONSE", +} +var Message_MessageType_value = map[string]int32{ + "DIAL": 0, + "DIAL_RESPONSE": 1, +} + +func (x Message_MessageType) Enum() *Message_MessageType { + p := new(Message_MessageType) + *p = x + return p +} +func (x Message_MessageType) String() string { + return proto.EnumName(Message_MessageType_name, int32(x)) +} +func (x *Message_MessageType) UnmarshalJSON(data []byte) error { + value, err := proto.UnmarshalJSONEnum(Message_MessageType_value, data, "Message_MessageType") + if err != nil { + return err + } + *x = Message_MessageType(value) + return nil +} + +type Message_ResponseStatus int32 + +const ( + Message_OK Message_ResponseStatus = 0 + Message_E_DIAL_ERROR Message_ResponseStatus = 100 + Message_E_DIAL_REFUSED Message_ResponseStatus = 101 + Message_E_INTERNAL_ERROR Message_ResponseStatus = 200 +) + +var Message_ResponseStatus_name = map[int32]string{ + 0: "OK", + 100: "E_DIAL_ERROR", + 101: "E_DIAL_REFUSED", + 200: "E_INTERNAL_ERROR", +} +var Message_ResponseStatus_value = map[string]int32{ + "OK": 0, + "E_DIAL_ERROR": 100, + "E_DIAL_REFUSED": 101, + "E_INTERNAL_ERROR": 200, +} + +func (x Message_ResponseStatus) Enum() *Message_ResponseStatus { + p := new(Message_ResponseStatus) + *p = x + return p +} +func (x Message_ResponseStatus) String() string { + return proto.EnumName(Message_ResponseStatus_name, int32(x)) +} +func (x *Message_ResponseStatus) UnmarshalJSON(data []byte) error { + value, err := proto.UnmarshalJSONEnum(Message_ResponseStatus_value, data, "Message_ResponseStatus") + if err != nil { + return err + } + *x = Message_ResponseStatus(value) + return nil +} + +type Message struct { + Type *Message_MessageType `protobuf:"varint,1,opt,name=type,enum=autonat.pb.Message_MessageType" json:"type,omitempty"` + Dial *Message_Dial `protobuf:"bytes,2,opt,name=dial" json:"dial,omitempty"` + DialResponse *Message_DialResponse `protobuf:"bytes,3,opt,name=dialResponse" json:"dialResponse,omitempty"` + XXX_unrecognized []byte `json:"-"` +} + +func (m *Message) Reset() { *m = Message{} } +func (m *Message) String() string { return proto.CompactTextString(m) } +func (*Message) ProtoMessage() {} + +func (m *Message) GetType() Message_MessageType { + if m != nil && m.Type != nil { + return *m.Type + } + return Message_DIAL +} + +func (m *Message) GetDial() *Message_Dial { + if m != nil { + return m.Dial + } + return nil +} + +func (m *Message) GetDialResponse() *Message_DialResponse { + if m != nil { + return m.DialResponse + } + return nil +} + +type Message_PeerInfo struct { + Id []byte `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"` + Addrs [][]byte `protobuf:"bytes,2,rep,name=addrs" json:"addrs,omitempty"` + XXX_unrecognized []byte `json:"-"` +} + +func (m *Message_PeerInfo) Reset() { *m = Message_PeerInfo{} } +func (m *Message_PeerInfo) String() string { return proto.CompactTextString(m) } +func (*Message_PeerInfo) ProtoMessage() {} + +func (m *Message_PeerInfo) GetId() []byte { + if m != nil { + return m.Id + } + return nil +} + +func (m *Message_PeerInfo) GetAddrs() [][]byte { + if m != nil { + return m.Addrs + } + return nil +} + +type Message_Dial struct { + Peer *Message_PeerInfo `protobuf:"bytes,1,opt,name=peer" json:"peer,omitempty"` + XXX_unrecognized []byte `json:"-"` +} + +func (m *Message_Dial) Reset() { *m = Message_Dial{} } +func (m *Message_Dial) String() string { return proto.CompactTextString(m) } +func (*Message_Dial) ProtoMessage() {} + +func (m *Message_Dial) GetPeer() *Message_PeerInfo { + if m != nil { + return m.Peer + } + return nil +} + +type Message_DialResponse struct { + Status *Message_ResponseStatus `protobuf:"varint,1,opt,name=status,enum=autonat.pb.Message_ResponseStatus" json:"status,omitempty"` + StatusText *string `protobuf:"bytes,2,opt,name=statusText" json:"statusText,omitempty"` + Addr []byte `protobuf:"bytes,3,opt,name=addr" json:"addr,omitempty"` + XXX_unrecognized []byte `json:"-"` +} + +func (m *Message_DialResponse) Reset() { *m = Message_DialResponse{} } +func (m *Message_DialResponse) String() string { return proto.CompactTextString(m) } +func (*Message_DialResponse) ProtoMessage() {} + +func (m *Message_DialResponse) GetStatus() Message_ResponseStatus { + if m != nil && m.Status != nil { + return *m.Status + } + return Message_OK +} + +func (m *Message_DialResponse) GetStatusText() string { + if m != nil && m.StatusText != nil { + return *m.StatusText + } + return "" +} + +func (m *Message_DialResponse) GetAddr() []byte { + if m != nil { + return m.Addr + } + return nil +} + +func init() { + proto.RegisterType((*Message)(nil), "autonat.pb.Message") + proto.RegisterType((*Message_PeerInfo)(nil), "autonat.pb.Message.PeerInfo") + proto.RegisterType((*Message_Dial)(nil), "autonat.pb.Message.Dial") + proto.RegisterType((*Message_DialResponse)(nil), "autonat.pb.Message.DialResponse") + proto.RegisterEnum("autonat.pb.Message_MessageType", Message_MessageType_name, Message_MessageType_value) + proto.RegisterEnum("autonat.pb.Message_ResponseStatus", Message_ResponseStatus_name, Message_ResponseStatus_value) +} diff --git a/p2p/host/autonat/pb/autonat.proto b/p2p/host/autonat/pb/autonat.proto new file mode 100644 index 0000000000..53999193c5 --- /dev/null +++ b/p2p/host/autonat/pb/autonat.proto @@ -0,0 +1,34 @@ +package autonat.pb; + +message Message { + enum MessageType { + DIAL = 0; + DIAL_RESPONSE = 1; + } + + enum ResponseStatus { + OK = 0; + E_DIAL_ERROR = 100; + E_DIAL_REFUSED = 101; + E_INTERNAL_ERROR = 200; + } + + message PeerInfo { + optional bytes id = 1; + repeated bytes addrs = 2; + } + + message Dial { + optional PeerInfo peer = 1; + } + + message DialResponse { + optional ResponseStatus status = 1; + optional string statusText = 2; + optional bytes addr = 3; + } + + optional MessageType type = 1; + optional Dial dial = 2; + optional DialResponse dialResponse = 3; +} From 1ad785cc023fc33f8a4fd0fdc3f0177992afde84 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 5 May 2018 16:45:58 +0300 Subject: [PATCH 0434/3965] basic client --- p2p/host/autonat/client.go | 83 ++++++++++++++++++++++++++++++++++++++ p2p/host/autonat/proto.go | 23 +++++++++++ 2 files changed, 106 insertions(+) create mode 100644 p2p/host/autonat/client.go create mode 100644 p2p/host/autonat/proto.go diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go new file mode 100644 index 0000000000..6709abcd1a --- /dev/null +++ b/p2p/host/autonat/client.go @@ -0,0 +1,83 @@ +package autonat + +import ( + "context" + "fmt" + + pb "github.com/libp2p/go-libp2p-autonat/pb" + + ggio "github.com/gogo/protobuf/io" + host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" +) + +type AutoNATClient interface { + Dial(ctx context.Context) (ma.Multiaddr, error) +} + +type AutoNATError struct { + Status pb.Message_ResponseStatus + Text string +} + +func NewAutoNATClient(h host.Host, p peer.ID) AutoNATClient { + return &client{h: h, p: p} +} + +type client struct { + h host.Host + p peer.ID +} + +func (c *client) Dial(ctx context.Context) (ma.Multiaddr, error) { + s, err := c.h.NewStream(ctx, c.p, AutoNATProto) + if err != nil { + return nil, err + } + defer s.Close() + + r := ggio.NewDelimitedReader(s, inet.MessageSizeMax) + w := ggio.NewDelimitedWriter(s) + + req := newDialMessage(pstore.PeerInfo{ID: c.h.ID(), Addrs: c.h.Addrs()}) + err = w.WriteMsg(req) + if err != nil { + return nil, err + } + + var res pb.Message + err = r.ReadMsg(&res) + if err != nil { + return nil, err + } + + if res.GetType() != pb.Message_DIAL_RESPONSE { + return nil, fmt.Errorf("Unexpected response: %s", res.GetType().String()) + } + + status := res.GetDialResponse().GetStatus() + switch status { + case pb.Message_OK: + addr := res.GetDialResponse().GetAddr() + return ma.NewMultiaddrBytes(addr) + + default: + return nil, AutoNATError{Status: status, Text: res.GetDialResponse().GetStatusText()} + } +} + +func (e AutoNATError) Error() string { + return fmt.Sprintf("AutoNAT error: %s (%s)", e.Text, e.Status.String()) +} + +func (e AutoNATError) IsDialError() bool { + return e.Status == pb.Message_E_DIAL_ERROR +} + +func IsDialError(e error) bool { + ae, ok := e.(AutoNATError) + return ok && ae.IsDialError() +} diff --git a/p2p/host/autonat/proto.go b/p2p/host/autonat/proto.go new file mode 100644 index 0000000000..142469ce65 --- /dev/null +++ b/p2p/host/autonat/proto.go @@ -0,0 +1,23 @@ +package autonat + +import ( + pb "github.com/libp2p/go-libp2p-autonat/pb" + + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +const AutoNATProto = "/autonat/1.0.0" + +func newDialMessage(pi pstore.PeerInfo) *pb.Message { + msg := new(pb.Message) + msg.Type = pb.Message_DIAL.Enum() + msg.Dial = new(pb.Message_Dial) + msg.Dial.Peer = new(pb.Message_PeerInfo) + msg.Dial.Peer.Id = []byte(pi.ID) + msg.Dial.Peer.Addrs = make([][]byte, len(pi.Addrs)) + for i, addr := range pi.Addrs { + msg.Dial.Peer.Addrs[i] = addr.Bytes() + } + + return msg +} From 76d49e17f1aed6298609f71bb0af2d8e5f02c71b Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 09:45:30 +0300 Subject: [PATCH 0435/3965] add E_BAD_REQUEST to protobuf --- p2p/host/autonat/pb/autonat.pb.go | 9 ++++++--- p2p/host/autonat/pb/autonat.proto | 3 ++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/p2p/host/autonat/pb/autonat.pb.go b/p2p/host/autonat/pb/autonat.pb.go index 79a09913b8..d1c981c01c 100644 --- a/p2p/host/autonat/pb/autonat.pb.go +++ b/p2p/host/autonat/pb/autonat.pb.go @@ -61,20 +61,23 @@ const ( Message_OK Message_ResponseStatus = 0 Message_E_DIAL_ERROR Message_ResponseStatus = 100 Message_E_DIAL_REFUSED Message_ResponseStatus = 101 - Message_E_INTERNAL_ERROR Message_ResponseStatus = 200 + Message_E_BAD_REQUEST Message_ResponseStatus = 200 + Message_E_INTERNAL_ERROR Message_ResponseStatus = 300 ) var Message_ResponseStatus_name = map[int32]string{ 0: "OK", 100: "E_DIAL_ERROR", 101: "E_DIAL_REFUSED", - 200: "E_INTERNAL_ERROR", + 200: "E_BAD_REQUEST", + 300: "E_INTERNAL_ERROR", } var Message_ResponseStatus_value = map[string]int32{ "OK": 0, "E_DIAL_ERROR": 100, "E_DIAL_REFUSED": 101, - "E_INTERNAL_ERROR": 200, + "E_BAD_REQUEST": 200, + "E_INTERNAL_ERROR": 300, } func (x Message_ResponseStatus) Enum() *Message_ResponseStatus { diff --git a/p2p/host/autonat/pb/autonat.proto b/p2p/host/autonat/pb/autonat.proto index 53999193c5..7107e1c07f 100644 --- a/p2p/host/autonat/pb/autonat.proto +++ b/p2p/host/autonat/pb/autonat.proto @@ -10,7 +10,8 @@ message Message { OK = 0; E_DIAL_ERROR = 100; E_DIAL_REFUSED = 101; - E_INTERNAL_ERROR = 200; + E_BAD_REQUEST = 200; + E_INTERNAL_ERROR = 300; } message PeerInfo { From 6d61b523ff8152e64b950f77c541e6da6bc5d332 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 09:49:02 +0300 Subject: [PATCH 0436/3965] service implementation --- p2p/host/autonat/proto.go | 18 ++++ p2p/host/autonat/svc.go | 175 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 193 insertions(+) create mode 100644 p2p/host/autonat/svc.go diff --git a/p2p/host/autonat/proto.go b/p2p/host/autonat/proto.go index 142469ce65..7d28d4bdb6 100644 --- a/p2p/host/autonat/proto.go +++ b/p2p/host/autonat/proto.go @@ -3,11 +3,15 @@ package autonat import ( pb "github.com/libp2p/go-libp2p-autonat/pb" + logging "github.com/ipfs/go-log" pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" ) const AutoNATProto = "/autonat/1.0.0" +var log = logging.Logger("autonat") + func newDialMessage(pi pstore.PeerInfo) *pb.Message { msg := new(pb.Message) msg.Type = pb.Message_DIAL.Enum() @@ -21,3 +25,17 @@ func newDialMessage(pi pstore.PeerInfo) *pb.Message { return msg } + +func newDialResponseOK(addr ma.Multiaddr) *pb.Message_DialResponse { + dr := new(pb.Message_DialResponse) + dr.Status = pb.Message_OK.Enum() + dr.Addr = addr.Bytes() + return dr +} + +func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Message_DialResponse { + dr := new(pb.Message_DialResponse) + dr.Status = status.Enum() + dr.StatusText = &text + return dr +} diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go new file mode 100644 index 0000000000..863f97cd90 --- /dev/null +++ b/p2p/host/autonat/svc.go @@ -0,0 +1,175 @@ +package autonat + +import ( + "context" + "sync" + "time" + + pb "github.com/libp2p/go-libp2p-autonat/pb" + + ggio "github.com/gogo/protobuf/io" + libp2p "github.com/libp2p/go-libp2p" + host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" +) + +const P_CIRCUIT = 290 + +type AutoNATService struct { + ctx context.Context + dialer host.Host + peers map[peer.ID]struct{} + mx sync.Mutex +} + +func NewAutoNATService(ctx context.Context, h host.Host) (*AutoNATService, error) { + dialer, err := libp2p.New(ctx) + if err != nil { + return nil, err + } + + as := &AutoNATService{ + ctx: ctx, + dialer: dialer, + peers: make(map[peer.ID]struct{}), + } + h.SetStreamHandler(AutoNATProto, as.handleStream) + + go as.resetPeers() + + return as, nil +} + +func (as *AutoNATService) handleStream(s inet.Stream) { + defer s.Close() + + pid := s.Conn().RemotePeer() + log.Debugf("New stream from %s", pid.Pretty()) + + r := ggio.NewDelimitedReader(s, inet.MessageSizeMax) + w := ggio.NewDelimitedWriter(s) + + var req pb.Message + var res pb.Message + + err := r.ReadMsg(&req) + if err != nil { + s.Reset() + return + } + + t := req.GetType() + if t != pb.Message_DIAL { + log.Debugf("Unexpected message from: %s", t.String()) + s.Reset() + return + } + + dr := as.handleDial(pid, req.GetDial().GetPeer()) + res.Type = pb.Message_DIAL_RESPONSE.Enum() + res.DialResponse = dr + + err = w.WriteMsg(&res) + if err != nil { + log.Debugf("Error writing response: %s", err.Error()) + s.Reset() + return + } +} + +func (as *AutoNATService) handleDial(p peer.ID, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { + if mpi == nil { + return newDialResponseError(pb.Message_E_BAD_REQUEST, "missing peer info") + } + + mpid := mpi.GetId() + if mpid != nil { + mp, err := peer.IDFromBytes(mpid) + if err != nil { + return newDialResponseError(pb.Message_E_BAD_REQUEST, "bad peer id") + } + + if mp != p { + return newDialResponseError(pb.Message_E_BAD_REQUEST, "peer id mismatch") + } + } + + addrs := make([]ma.Multiaddr, 0) + for _, maddr := range mpi.GetAddrs() { + addr, err := ma.NewMultiaddrBytes(maddr) + if err != nil { + log.Debugf("Error parsing multiaddr: %s", err.Error()) + continue + } + + // skip relay addresses + _, err = addr.ValueForProtocol(P_CIRCUIT) + if err == nil { + continue + } + + addrs = append(addrs, addr) + } + + if len(addrs) == 0 { + return newDialResponseError(pb.Message_E_DIAL_ERROR, "no dialable addresses") + } + + return as.doDial(pstore.PeerInfo{ID: p, Addrs: addrs}) +} + +func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { + // rate limit check + as.mx.Lock() + _, ok := as.peers[pi.ID] + if ok { + as.mx.Unlock() + return newDialResponseError(pb.Message_E_DIAL_REFUSED, "too many dials") + } + as.peers[pi.ID] = struct{}{} + as.mx.Unlock() + + ctx, cancel := context.WithTimeout(as.ctx, 42*time.Second) + defer cancel() + + err := as.dialer.Connect(ctx, pi) + if err != nil { + log.Debugf("error dialing %s: %s", pi.ID.Pretty(), err.Error()) + // wait for the context to timeout to avoid leaking timing information + // this renders the service ineffective as a port scanner + select { + case <-ctx.Done(): + return newDialResponseError(pb.Message_E_DIAL_ERROR, "dial failed") + } + } + + conns := as.dialer.Network().ConnsToPeer(pi.ID) + if len(conns) == 0 { + log.Errorf("supposedly connected to %s, but no connection to peer", pi.ID.Pretty()) + return newDialResponseError(pb.Message_E_INTERNAL_ERROR, "internal service error") + } + + ra := conns[0].RemoteMultiaddr() + conns[0].Close() + return newDialResponseOK(ra) +} + +func (as *AutoNATService) resetPeers() { + ticker := time.NewTicker(1 * time.Minute) + defer ticker.Stop() + + for { + select { + case <-ticker.C: + as.mx.Lock() + as.peers = make(map[peer.ID]struct{}) + as.mx.Unlock() + + case <-as.ctx.Done(): + return + } + } +} From 645e280349943d4c77e937bc6b1045aea75cde5e Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 11:22:32 +0300 Subject: [PATCH 0437/3965] NAT autodetection --- p2p/host/autonat/autonat.go | 143 ++++++++++++++++++++++++++++++++++++ p2p/host/autonat/client.go | 9 +++ p2p/host/autonat/notify.go | 35 +++++++++ 3 files changed, 187 insertions(+) create mode 100644 p2p/host/autonat/autonat.go create mode 100644 p2p/host/autonat/notify.go diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go new file mode 100644 index 0000000000..77b9b35070 --- /dev/null +++ b/p2p/host/autonat/autonat.go @@ -0,0 +1,143 @@ +package autonat + +import ( + "context" + "errors" + "math/rand" + "sync" + "time" + + host "github.com/libp2p/go-libp2p-host" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" +) + +// attach to any host and auto-discovery autonat servers +// periodically query them to deal with changing NAT situation + +type NATStatus int + +const ( + NATStatusUnknown = iota + NATStatusPublic + NATStatusPrivate +) + +type AutoNAT interface { + Status() NATStatus + PublicAddr() (ma.Multiaddr, error) +} + +type AutoNATState struct { + ctx context.Context + host host.Host + peers map[peer.ID]bool + status NATStatus + addr ma.Multiaddr + mx sync.Mutex +} + +func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { + as := &AutoNATState{ + ctx: ctx, + host: h, + peers: make(map[peer.ID]bool), + status: NATStatusUnknown, + } + + go as.background() + return as +} + +func (as *AutoNATState) Status() NATStatus { + return as.status +} + +func (as *AutoNATState) PublicAddr() (ma.Multiaddr, error) { + as.mx.Lock() + defer as.mx.Unlock() + + if as.status != NATStatusPublic { + return nil, errors.New("NAT Status is not public") + } + + return as.addr, nil +} + +func (as *AutoNATState) background() { + // wait a bit for the node to come online and establish some connections + // before starting autodetection + time.Sleep(10 * time.Second) + for { + as.autodetect() + select { + case <-time.After(15 * time.Minute): + case <-as.ctx.Done(): + return + } + } +} + +func (as *AutoNATState) autodetect() { + if len(as.peers) == 0 { + log.Debugf("skipping NAT auto detection; no autonat peers") + return + } + + as.mx.Lock() + peers := make([]peer.ID, 0, len(as.peers)) + for p, c := range as.peers { + if c { + peers = append(peers, p) + } + } + + if len(peers) == 0 { + // we don't have any open connections, try any autonat peer that we know about + for p := range as.peers { + peers = append(peers, p) + } + } + + as.mx.Unlock() + + shufflePeers(peers) + + for _, p := range peers { + cli := NewAutoNATClient(as.host, p) + ctx, cancel := context.WithTimeout(as.ctx, 60*time.Second) + a, err := cli.Dial(ctx) + cancel() + + switch { + case err == nil: + log.Debugf("NAT status is public; address through %s: %s", p.Pretty(), a.String()) + as.mx.Lock() + as.addr = a + as.status = NATStatusPublic + as.mx.Unlock() + return + + case IsDialError(err): + log.Debugf("NAT status is private; dial error through %s: %s", p.Pretty(), err.Error()) + as.mx.Lock() + as.status = NATStatusPrivate + as.mx.Unlock() + return + + default: + log.Debugf("Error dialing through %s: %s", p.Pretty(), err.Error()) + } + } + + as.mx.Lock() + as.status = NATStatusUnknown + as.mx.Unlock() +} + +func shufflePeers(peers []peer.ID) { + for i := range peers { + j := rand.Intn(i + 1) + peers[i], peers[j] = peers[j], peers[i] + } +} diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index 6709abcd1a..50c5426674 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -77,7 +77,16 @@ func (e AutoNATError) IsDialError() bool { return e.Status == pb.Message_E_DIAL_ERROR } +func (e AutoNATError) IsDialRefused() bool { + return e.Status == pb.Message_E_DIAL_REFUSED +} + func IsDialError(e error) bool { ae, ok := e.(AutoNATError) return ok && ae.IsDialError() } + +func IsDialRefused(e error) bool { + ae, ok := e.(AutoNATError) + return ok && ae.IsDialRefused() +} diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go new file mode 100644 index 0000000000..958f2e0a08 --- /dev/null +++ b/p2p/host/autonat/notify.go @@ -0,0 +1,35 @@ +package autonat + +import ( + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" +) + +var _ inet.Notifiee = (*AutoNATState)(nil) + +func (as *AutoNATState) Listen(net inet.Network, a ma.Multiaddr) {} +func (as *AutoNATState) ListenClose(net inet.Network, a ma.Multiaddr) {} +func (as *AutoNATState) OpenedStream(net inet.Network, s inet.Stream) {} +func (as *AutoNATState) ClosedStream(net inet.Network, s inet.Stream) {} + +func (as *AutoNATState) Connected(net inet.Network, c inet.Conn) { + go func(p peer.ID) { + s, err := as.host.NewStream(as.ctx, p, AutoNATProto) + if err != nil { + return + } + s.Close() + + log.Infof("Discovered AutoNAT peer %s", p.Pretty()) + as.mx.Lock() + as.peers[p] = true + as.mx.Unlock() + }(c.RemotePeer()) +} + +func (as *AutoNATState) Disconnected(net inet.Network, c inet.Conn) { + as.mx.Lock() + delete(as.peers, c.RemotePeer()) + as.mx.Unlock() +} From a713affd39d98cb8b502fa61b8ac0ba23a4500f9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 11:28:24 +0300 Subject: [PATCH 0438/3965] remove left-over design notes --- p2p/host/autonat/autonat.go | 3 --- 1 file changed, 3 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 77b9b35070..d809d0d2a3 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -12,9 +12,6 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -// attach to any host and auto-discovery autonat servers -// periodically query them to deal with changing NAT situation - type NATStatus int const ( From 7fc002407f43534a7a619b6a05cf438f4906cd21 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 12:00:17 +0300 Subject: [PATCH 0439/3965] don't delete autonat peers on disconnect, just mark them as disconnected --- p2p/host/autonat/notify.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 958f2e0a08..2c5f6c78bd 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -30,6 +30,6 @@ func (as *AutoNATState) Connected(net inet.Network, c inet.Conn) { func (as *AutoNATState) Disconnected(net inet.Network, c inet.Conn) { as.mx.Lock() - delete(as.peers, c.RemotePeer()) + as.peers[c.RemotePeer()] = false as.mx.Unlock() } From ae9117749d8d9b58ac750661a7344e87c95d376b Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 12:02:54 +0300 Subject: [PATCH 0440/3965] we only track autonat peers fix bug introduced in last commit --- p2p/host/autonat/notify.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 2c5f6c78bd..6a1a138370 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -30,6 +30,10 @@ func (as *AutoNATState) Connected(net inet.Network, c inet.Conn) { func (as *AutoNATState) Disconnected(net inet.Network, c inet.Conn) { as.mx.Lock() - as.peers[c.RemotePeer()] = false + p := c.RemotePeer() + _, ok := as.peers[p] + if ok { + as.peers[p] = false + } as.mx.Unlock() } From edf41b43f16a83024ca7daf36b630811d35dbe5e Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 13:13:19 +0300 Subject: [PATCH 0441/3965] fix autonat peer tracking connection state can be determined through the network interface on demand --- p2p/host/autonat/autonat.go | 8 ++++---- p2p/host/autonat/notify.go | 12 ++---------- 2 files changed, 6 insertions(+), 14 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index d809d0d2a3..5dabd31b23 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -28,7 +28,7 @@ type AutoNAT interface { type AutoNATState struct { ctx context.Context host host.Host - peers map[peer.ID]bool + peers map[peer.ID]struct{} status NATStatus addr ma.Multiaddr mx sync.Mutex @@ -38,7 +38,7 @@ func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { as := &AutoNATState{ ctx: ctx, host: h, - peers: make(map[peer.ID]bool), + peers: make(map[peer.ID]struct{}), status: NATStatusUnknown, } @@ -83,8 +83,8 @@ func (as *AutoNATState) autodetect() { as.mx.Lock() peers := make([]peer.ID, 0, len(as.peers)) - for p, c := range as.peers { - if c { + for p := range as.peers { + if len(as.host.Network().ConnsToPeer(p)) > 0 { peers = append(peers, p) } } diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 6a1a138370..2c68e8ea6e 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -23,17 +23,9 @@ func (as *AutoNATState) Connected(net inet.Network, c inet.Conn) { log.Infof("Discovered AutoNAT peer %s", p.Pretty()) as.mx.Lock() - as.peers[p] = true + as.peers[p] = struct{}{} as.mx.Unlock() }(c.RemotePeer()) } -func (as *AutoNATState) Disconnected(net inet.Network, c inet.Conn) { - as.mx.Lock() - p := c.RemotePeer() - _, ok := as.peers[p] - if ok { - as.peers[p] = false - } - as.mx.Unlock() -} +func (as *AutoNATState) Disconnected(net inet.Network, c inet.Conn) {} From 82a7aa9c0fc30e3461ef531ab302e995eaaff9ec Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 13:24:16 +0300 Subject: [PATCH 0442/3965] add TODO in service about skipping private network addresses --- p2p/host/autonat/svc.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 863f97cd90..e2aa252a88 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -111,6 +111,8 @@ func (as *AutoNATService) handleDial(p peer.ID, mpi *pb.Message_PeerInfo) *pb.Me continue } + // TODO we probably also want to skip all addresses that are not IANA public routable + addrs = append(addrs, addr) } From bd18d4edd9cf3390e4c21155d22dd6668fc87eb4 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 15:39:24 +0300 Subject: [PATCH 0443/3965] add network notifee --- p2p/host/autonat/autonat.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 5dabd31b23..6f3ecd38f1 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -42,7 +42,9 @@ func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { status: NATStatusUnknown, } + h.Network().Notify(as) go as.background() + return as } From e9f8bf996a91407f2db2167ab08607be0ed02b74 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 17:47:53 +0300 Subject: [PATCH 0444/3965] don't try to dial private network addresses addresses service TODO --- p2p/host/autonat/addr.go | 62 ++++++++++++++++++++++++++++++++++++++++ p2p/host/autonat/svc.go | 5 +++- 2 files changed, 66 insertions(+), 1 deletion(-) create mode 100644 p2p/host/autonat/addr.go diff --git a/p2p/host/autonat/addr.go b/p2p/host/autonat/addr.go new file mode 100644 index 0000000000..0eecb4f084 --- /dev/null +++ b/p2p/host/autonat/addr.go @@ -0,0 +1,62 @@ +package autonat + +import ( + "net" + + ma "github.com/multiformats/go-multiaddr" +) + +var private4, private6 []*net.IPNet +var privateCIDR4 = []string{ + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16", + "100.64.0.0/10", + "169.254.0.0/16", +} +var privateCIDR6 = []string{ + "fc00::/7", + "fe80::/10", +} + +func init() { + private4 = parsePrivateCIDR(privateCIDR4) + private6 = parsePrivateCIDR(privateCIDR6) +} + +func parsePrivateCIDR(cidrs []string) []*net.IPNet { + ipnets := make([]*net.IPNet, len(cidrs)) + for i, cidr := range cidrs { + _, ipnet, err := net.ParseCIDR(cidr) + if err != nil { + panic(err) + } + ipnets[i] = ipnet + } + return ipnets +} + +func isPublicAddr(a ma.Multiaddr) bool { + ip, err := a.ValueForProtocol(ma.P_IP4) + if err == nil { + return !inAddrRange(ip, private4) + } + + ip, err = a.ValueForProtocol(ma.P_IP6) + if err == nil { + return !inAddrRange(ip, private6) + } + + return false +} + +func inAddrRange(s string, ipnets []*net.IPNet) bool { + ip := net.ParseIP(s) + for _, ipnet := range ipnets { + if ipnet.Contains(ip) { + return true + } + } + + return false +} diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index e2aa252a88..73cff7699a 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -111,7 +111,10 @@ func (as *AutoNATService) handleDial(p peer.ID, mpi *pb.Message_PeerInfo) *pb.Me continue } - // TODO we probably also want to skip all addresses that are not IANA public routable + // skip private network (unroutable) addresses + if !isPublicAddr(addr) { + continue + } addrs = append(addrs, addr) } From 8af7aacd7a75c34e88b9a66176bc69982ee85b72 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 6 May 2018 18:33:21 +0300 Subject: [PATCH 0445/3965] add localhost to private addr ranges --- p2p/host/autonat/addr.go | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/p2p/host/autonat/addr.go b/p2p/host/autonat/addr.go index 0eecb4f084..4e078e3765 100644 --- a/p2p/host/autonat/addr.go +++ b/p2p/host/autonat/addr.go @@ -8,14 +8,22 @@ import ( var private4, private6 []*net.IPNet var privateCIDR4 = []string{ + // localhost + "127.0.0.0/8", + // private networks "10.0.0.0/8", + "100.64.0.0/10", "172.16.0.0/12", "192.168.0.0/16", - "100.64.0.0/10", + // link local "169.254.0.0/16", } var privateCIDR6 = []string{ + // localhost + "::1/128", + // ULA reserved "fc00::/7", + // link local "fe80::/10", } From ea2e2875722727b2f13a53a71369c0d2b1c98354 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 7 May 2018 12:03:21 +0300 Subject: [PATCH 0446/3965] no need to select; it's a one shot sync --- p2p/host/autonat/svc.go | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 73cff7699a..31520983ae 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -145,10 +145,8 @@ func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { log.Debugf("error dialing %s: %s", pi.ID.Pretty(), err.Error()) // wait for the context to timeout to avoid leaking timing information // this renders the service ineffective as a port scanner - select { - case <-ctx.Done(): - return newDialResponseError(pb.Message_E_DIAL_ERROR, "dial failed") - } + <-ctx.Done() + return newDialResponseError(pb.Message_E_DIAL_ERROR, "dial failed") } conns := as.dialer.Network().ConnsToPeer(pi.ID) From 1c0355d3e87993c8a3c9773833d70748d2c0f177 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 7 May 2018 12:08:13 +0300 Subject: [PATCH 0447/3965] typed NATStatus constants --- p2p/host/autonat/autonat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 6f3ecd38f1..4c0212cb24 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -15,7 +15,7 @@ import ( type NATStatus int const ( - NATStatusUnknown = iota + NATStatusUnknown NATStatus = iota NATStatusPublic NATStatusPrivate ) From da3f7b55271a15d3725a926532fe6d2e6ea2e2fc Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 7 May 2018 12:25:35 +0300 Subject: [PATCH 0448/3965] bump initial autodiscovery delay to 15s --- p2p/host/autonat/autonat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 4c0212cb24..79b0c89ab8 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -66,7 +66,7 @@ func (as *AutoNATState) PublicAddr() (ma.Multiaddr, error) { func (as *AutoNATState) background() { // wait a bit for the node to come online and establish some connections // before starting autodetection - time.Sleep(10 * time.Second) + time.Sleep(15 * time.Second) for { as.autodetect() select { From 2584bc51ffaec27e77a6f00971030dc7800a7660 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 8 May 2018 16:11:33 +0300 Subject: [PATCH 0449/3965] AutoNATState is AmbientAutoNAT --- p2p/host/autonat/autonat.go | 12 ++++++------ p2p/host/autonat/notify.go | 14 +++++++------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 79b0c89ab8..cad8a8d3f9 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -25,7 +25,7 @@ type AutoNAT interface { PublicAddr() (ma.Multiaddr, error) } -type AutoNATState struct { +type AmbientAutoNAT struct { ctx context.Context host host.Host peers map[peer.ID]struct{} @@ -35,7 +35,7 @@ type AutoNATState struct { } func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { - as := &AutoNATState{ + as := &AmbientAutoNAT{ ctx: ctx, host: h, peers: make(map[peer.ID]struct{}), @@ -48,11 +48,11 @@ func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { return as } -func (as *AutoNATState) Status() NATStatus { +func (as *AmbientAutoNAT) Status() NATStatus { return as.status } -func (as *AutoNATState) PublicAddr() (ma.Multiaddr, error) { +func (as *AmbientAutoNAT) PublicAddr() (ma.Multiaddr, error) { as.mx.Lock() defer as.mx.Unlock() @@ -63,7 +63,7 @@ func (as *AutoNATState) PublicAddr() (ma.Multiaddr, error) { return as.addr, nil } -func (as *AutoNATState) background() { +func (as *AmbientAutoNAT) background() { // wait a bit for the node to come online and establish some connections // before starting autodetection time.Sleep(15 * time.Second) @@ -77,7 +77,7 @@ func (as *AutoNATState) background() { } } -func (as *AutoNATState) autodetect() { +func (as *AmbientAutoNAT) autodetect() { if len(as.peers) == 0 { log.Debugf("skipping NAT auto detection; no autonat peers") return diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 2c68e8ea6e..4e2bc9d984 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -6,14 +6,14 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -var _ inet.Notifiee = (*AutoNATState)(nil) +var _ inet.Notifiee = (*AmbientAutoNAT)(nil) -func (as *AutoNATState) Listen(net inet.Network, a ma.Multiaddr) {} -func (as *AutoNATState) ListenClose(net inet.Network, a ma.Multiaddr) {} -func (as *AutoNATState) OpenedStream(net inet.Network, s inet.Stream) {} -func (as *AutoNATState) ClosedStream(net inet.Network, s inet.Stream) {} +func (as *AmbientAutoNAT) Listen(net inet.Network, a ma.Multiaddr) {} +func (as *AmbientAutoNAT) ListenClose(net inet.Network, a ma.Multiaddr) {} +func (as *AmbientAutoNAT) OpenedStream(net inet.Network, s inet.Stream) {} +func (as *AmbientAutoNAT) ClosedStream(net inet.Network, s inet.Stream) {} -func (as *AutoNATState) Connected(net inet.Network, c inet.Conn) { +func (as *AmbientAutoNAT) Connected(net inet.Network, c inet.Conn) { go func(p peer.ID) { s, err := as.host.NewStream(as.ctx, p, AutoNATProto) if err != nil { @@ -28,4 +28,4 @@ func (as *AutoNATState) Connected(net inet.Network, c inet.Conn) { }(c.RemotePeer()) } -func (as *AutoNATState) Disconnected(net inet.Network, c inet.Conn) {} +func (as *AmbientAutoNAT) Disconnected(net inet.Network, c inet.Conn) {} From 003e697e6ab2088171537dc045656e652a57ca6f Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 8 May 2018 16:17:05 +0300 Subject: [PATCH 0450/3965] variables for background delays --- p2p/host/autonat/autonat.go | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index cad8a8d3f9..54b2b56527 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -20,6 +20,11 @@ const ( NATStatusPrivate ) +var ( + AutoNATBootDelay = 15 * time.Second + AutoNATRefreshInterval = 15 * time.Minute +) + type AutoNAT interface { Status() NATStatus PublicAddr() (ma.Multiaddr, error) @@ -66,11 +71,11 @@ func (as *AmbientAutoNAT) PublicAddr() (ma.Multiaddr, error) { func (as *AmbientAutoNAT) background() { // wait a bit for the node to come online and establish some connections // before starting autodetection - time.Sleep(15 * time.Second) + time.Sleep(AutoNATBootDelay) for { as.autodetect() select { - case <-time.After(15 * time.Minute): + case <-time.After(AutoNATRefreshInterval): case <-as.ctx.Done(): return } From f36100a1f9e6cada2da04cc0dd455a1b404b8530 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 9 May 2018 11:37:49 +0300 Subject: [PATCH 0451/3965] named magic number incantations --- p2p/host/autonat/autonat.go | 4 +++- p2p/host/autonat/svc.go | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 54b2b56527..30255328a7 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -23,6 +23,8 @@ const ( var ( AutoNATBootDelay = 15 * time.Second AutoNATRefreshInterval = 15 * time.Minute + + AutoNATRequestTimeout = 60 * time.Second ) type AutoNAT interface { @@ -109,7 +111,7 @@ func (as *AmbientAutoNAT) autodetect() { for _, p := range peers { cli := NewAutoNATClient(as.host, p) - ctx, cancel := context.WithTimeout(as.ctx, 60*time.Second) + ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) a, err := cli.Dial(ctx) cancel() diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 31520983ae..26d08f1ff9 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -18,6 +18,8 @@ import ( const P_CIRCUIT = 290 +var AutoNATServiceResetInterval = 1 * time.Minute + type AutoNATService struct { ctx context.Context dialer host.Host @@ -161,7 +163,7 @@ func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { } func (as *AutoNATService) resetPeers() { - ticker := time.NewTicker(1 * time.Minute) + ticker := time.NewTicker(AutoNATServiceResetInterval) defer ticker.Stop() for { From 7f690dcc6888b504b249d2b75b3273046b26aeda Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 9 May 2018 11:47:16 +0300 Subject: [PATCH 0452/3965] refactor getPeers for locked scope --- p2p/host/autonat/autonat.go | 50 ++++++++++++++++++++++--------------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 30255328a7..765996b1d5 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -85,30 +85,13 @@ func (as *AmbientAutoNAT) background() { } func (as *AmbientAutoNAT) autodetect() { - if len(as.peers) == 0 { - log.Debugf("skipping NAT auto detection; no autonat peers") - return - } - - as.mx.Lock() - peers := make([]peer.ID, 0, len(as.peers)) - for p := range as.peers { - if len(as.host.Network().ConnsToPeer(p)) > 0 { - peers = append(peers, p) - } - } + peers := as.getPeers() if len(peers) == 0 { - // we don't have any open connections, try any autonat peer that we know about - for p := range as.peers { - peers = append(peers, p) - } + log.Debugf("skipping NAT auto detection; no autonat peers") + return } - as.mx.Unlock() - - shufflePeers(peers) - for _, p := range peers { cli := NewAutoNATClient(as.host, p) ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) @@ -141,6 +124,33 @@ func (as *AmbientAutoNAT) autodetect() { as.mx.Unlock() } +func (as *AmbientAutoNAT) getPeers() []peer.ID { + as.mx.Lock() + defer as.mx.Unlock() + + if len(as.peers) == 0 { + return nil + } + + peers := make([]peer.ID, 0, len(as.peers)) + for p := range as.peers { + if len(as.host.Network().ConnsToPeer(p)) > 0 { + peers = append(peers, p) + } + } + + if len(peers) == 0 { + // we don't have any open connections, try any autonat peer that we know about + for p := range as.peers { + peers = append(peers, p) + } + } + + shufflePeers(peers) + + return peers +} + func shufflePeers(peers []peer.ID) { for i := range peers { j := rand.Intn(i + 1) From bdb57814505f657780d82ffd5d6db3ffab617145 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 9 May 2018 11:53:15 +0300 Subject: [PATCH 0453/3965] don't throw away read errors; log them. --- p2p/host/autonat/svc.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 26d08f1ff9..068b2723b5 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -59,13 +59,14 @@ func (as *AutoNATService) handleStream(s inet.Stream) { err := r.ReadMsg(&req) if err != nil { + log.Debugf("Error reading message from %s: %s", pid.Pretty(), err.Error()) s.Reset() return } t := req.GetType() if t != pb.Message_DIAL { - log.Debugf("Unexpected message from: %s", t.String()) + log.Debugf("Unexpected message from %s: %s (%d)", pid.Pretty(), t.String(), t) s.Reset() return } @@ -76,7 +77,7 @@ func (as *AutoNATService) handleStream(s inet.Stream) { err = w.WriteMsg(&res) if err != nil { - log.Debugf("Error writing response: %s", err.Error()) + log.Debugf("Error writing response to %s: %s", pid.Pretty(), err.Error()) s.Reset() return } From 9c5851a80616b2c46ffcb887caa30a713f8e867b Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 9 May 2018 12:16:16 +0300 Subject: [PATCH 0454/3965] simplify autonat client itnerface --- p2p/host/autonat/autonat.go | 5 +++-- p2p/host/autonat/client.go | 11 +++++------ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 765996b1d5..3a58e2b9fd 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -92,10 +92,11 @@ func (as *AmbientAutoNAT) autodetect() { return } + cli := NewAutoNATClient(as.host) + for _, p := range peers { - cli := NewAutoNATClient(as.host, p) ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) - a, err := cli.Dial(ctx) + a, err := cli.Dial(ctx, p) cancel() switch { diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index 50c5426674..f996ff3781 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -15,7 +15,7 @@ import ( ) type AutoNATClient interface { - Dial(ctx context.Context) (ma.Multiaddr, error) + Dial(ctx context.Context, p peer.ID) (ma.Multiaddr, error) } type AutoNATError struct { @@ -23,17 +23,16 @@ type AutoNATError struct { Text string } -func NewAutoNATClient(h host.Host, p peer.ID) AutoNATClient { - return &client{h: h, p: p} +func NewAutoNATClient(h host.Host) AutoNATClient { + return &client{h: h} } type client struct { h host.Host - p peer.ID } -func (c *client) Dial(ctx context.Context) (ma.Multiaddr, error) { - s, err := c.h.NewStream(ctx, c.p, AutoNATProto) +func (c *client) Dial(ctx context.Context, p peer.ID) (ma.Multiaddr, error) { + s, err := c.h.NewStream(ctx, p, AutoNATProto) if err != nil { return nil, err } From 55cd73a0d861cade43ec239e3732b2c2334d1502 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 11 May 2018 08:13:53 +0300 Subject: [PATCH 0455/3965] mutex hat --- p2p/host/autonat/autonat.go | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 3a58e2b9fd..8d96fd1ee9 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -33,12 +33,13 @@ type AutoNAT interface { } type AmbientAutoNAT struct { - ctx context.Context - host host.Host + ctx context.Context + host host.Host + + mx sync.Mutex peers map[peer.ID]struct{} status NATStatus addr ma.Multiaddr - mx sync.Mutex } func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { From 889081eeba86962f7b1dbcb968d582690c8c525b Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 11 May 2018 09:40:36 +0300 Subject: [PATCH 0456/3965] docstrings and another mutex hat. --- p2p/host/autonat/autonat.go | 9 +++++++++ p2p/host/autonat/client.go | 6 ++++++ p2p/host/autonat/svc.go | 7 +++++-- 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 8d96fd1ee9..972cae2126 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -15,8 +15,11 @@ import ( type NATStatus int const ( + // NAT status is unknown NATStatusUnknown NATStatus = iota + // NAT status is publicly dialable NATStatusPublic + // NAT status is private network NATStatusPrivate ) @@ -27,11 +30,16 @@ var ( AutoNATRequestTimeout = 60 * time.Second ) +// AutoNAT is the interface for ambient NAT autodiscovery type AutoNAT interface { + // Status returns the current NAT status Status() NATStatus + // PublicAddr returns the public dial address when NAT status is public and an + // error otherwise PublicAddr() (ma.Multiaddr, error) } +// AmbientAutoNAT is the implementation of ambient NAT autodiscovery type AmbientAutoNAT struct { ctx context.Context host host.Host @@ -42,6 +50,7 @@ type AmbientAutoNAT struct { addr ma.Multiaddr } +// NewAutoNAT creates a new ambient NAT auto-discovery instance func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { as := &AmbientAutoNAT{ ctx: ctx, diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index f996ff3781..812ce47628 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -14,15 +14,19 @@ import ( ma "github.com/multiformats/go-multiaddr" ) +// AutoNATClient is a stateless client interface to AutoNAT peers type AutoNATClient interface { + // Dial requests from a peer providing AutoNAT services to test dial back Dial(ctx context.Context, p peer.ID) (ma.Multiaddr, error) } +// AutoNATError is the class of errors signalled by AutoNAT services type AutoNATError struct { Status pb.Message_ResponseStatus Text string } +// NewAutoNATClient creates a fresh instance of an AutoNATClient func NewAutoNATClient(h host.Host) AutoNATClient { return &client{h: h} } @@ -80,11 +84,13 @@ func (e AutoNATError) IsDialRefused() bool { return e.Status == pb.Message_E_DIAL_REFUSED } +// IsDialError returns true if the AutoNAT peer signalled an error dialing back func IsDialError(e error) bool { ae, ok := e.(AutoNATError) return ok && ae.IsDialError() } +// IsDialRefused returns true if the AutoNAT peer signalled refusal to dial back func IsDialRefused(e error) bool { ae, ok := e.(AutoNATError) return ok && ae.IsDialRefused() diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 068b2723b5..236f162e23 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -20,13 +20,16 @@ const P_CIRCUIT = 290 var AutoNATServiceResetInterval = 1 * time.Minute +// AutoNATService provides NAT autodetection services to other peers type AutoNATService struct { ctx context.Context dialer host.Host - peers map[peer.ID]struct{} - mx sync.Mutex + + mx sync.Mutex + peers map[peer.ID]struct{} } +// NewAutoNATService creates a new AutoNATService instance attached to a host func NewAutoNATService(ctx context.Context, h host.Host) (*AutoNATService, error) { dialer, err := libp2p.New(ctx) if err != nil { From 7ea67c6344a4a25ca4dc6a3aeed8b510ef1c3866 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 11 May 2018 09:46:16 +0300 Subject: [PATCH 0457/3965] improve docstring for NewAutoNAT --- p2p/host/autonat/autonat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 972cae2126..c059659b11 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -50,7 +50,7 @@ type AmbientAutoNAT struct { addr ma.Multiaddr } -// NewAutoNAT creates a new ambient NAT auto-discovery instance +// NewAutoNAT creates a new ambient NAT autodiscovery instance attached to a host func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { as := &AmbientAutoNAT{ ctx: ctx, From 81477e5d5316c7f9bab8b106d141013af886b0cf Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 11 May 2018 15:50:50 +0300 Subject: [PATCH 0458/3965] improve NATStatusUknown docstring --- p2p/host/autonat/autonat.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index c059659b11..bb6561a71d 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -12,10 +12,13 @@ import ( ma "github.com/multiformats/go-multiaddr" ) +// NATStatus is the state of NAT as detected by the ambient service. type NATStatus int const ( - // NAT status is unknown + // NAT status is unknown; this means that the ambient serice has not been + // able to decide the presence of NAT in the most recent attempt to test + // dial through known autonat peers. initial state. NATStatusUnknown NATStatus = iota // NAT status is publicly dialable NATStatusPublic From ce8aa438321571468c296511fbf06b12eede09ae Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 12 May 2018 08:20:04 +0300 Subject: [PATCH 0459/3965] fix typo --- p2p/host/autonat/autonat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index bb6561a71d..bdf1e8d8ba 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -16,7 +16,7 @@ import ( type NATStatus int const ( - // NAT status is unknown; this means that the ambient serice has not been + // NAT status is unknown; this means that the ambient service has not been // able to decide the presence of NAT in the most recent attempt to test // dial through known autonat peers. initial state. NATStatusUnknown NATStatus = iota From bad65a492f32121a87197f4a085905c35e2a367e Mon Sep 17 00:00:00 2001 From: Simon Menke Date: Fri, 25 May 2018 15:02:20 +0200 Subject: [PATCH 0460/3965] Added go.mod file --- go.mod | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 go.mod diff --git a/go.mod b/go.mod new file mode 100644 index 0000000000..d7f4b28787 --- /dev/null +++ b/go.mod @@ -0,0 +1,9 @@ +module github.com/fd/go-nat + +require ( + github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324 + github.com/jackpal/gateway v1.0.4 + github.com/jackpal/go-nat-pmp v1.0.1 + golang.org/x/net v0.0.0-20180524181706-dfa909b99c79 + golang.org/x/text v0.3.0 +) From a6371e76ba8bb846eb14c97ef40be7143a77a436 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Sun, 27 May 2018 16:58:17 +0200 Subject: [PATCH 0461/3965] Remove go-metrics from package.json: it does not seem used here or anywhere. License: MIT Signed-off-by: Hector Sanjuan --- package.json | 5 ----- 1 file changed, 5 deletions(-) diff --git a/package.json b/package.json index f1b93a680f..4df6f10232 100644 --- a/package.json +++ b/package.json @@ -83,11 +83,6 @@ "name": "go-multiaddr", "version": "1.2.6" }, - { - "hash": "QmeYJHEk8UjVVZ4XCRTZe6dFQrb8pGWD81LYCgeLp8CvMB", - "name": "go-metrics", - "version": "0.0.0" - }, { "hash": "QmYvsG72GsfLgUeSojXArjnU6L4Wmwk7wuAxtNLuyXcc1T", "name": "randbo", From 588159b82b06e80fadb0992794373194e47f1f97 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Tue, 29 May 2018 11:58:34 +0200 Subject: [PATCH 0462/3965] gx: update go-multistream License: MIT Signed-off-by: Hector Sanjuan --- package.json | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/package.json b/package.json index 4df6f10232..3b0a135790 100644 --- a/package.json +++ b/package.json @@ -34,9 +34,9 @@ "version": "1.0.0" }, { - "hash": "QmTnsezaB1wWNRHeHnYrm8K4d5i9wtyj3GsqjC3Rt5b5v5", + "hash": "QmbXRda5H2K3MSQyWWxTMtd8DWuguEBUCe6hpxfXVpFUGj", "name": "go-multistream", - "version": "0.3.6" + "version": "0.3.7" }, { "hash": "Qmf7HqcW7LtCi1W8y2bdx2eJpze74jkbKqpByxgXikdbLF", @@ -171,9 +171,9 @@ }, { "author": "whyrusleeping", - "hash": "QmaHsbK8b39AzQWEwDsysCutdJXyfa3k9oFh1cr6dfMhHT", + "hash": "QmYSQpi68jBLVUx62u543RVvnjjaQQDDxyopWWG31kiUkG", "name": "go-libp2p-conn", - "version": "1.7.6" + "version": "1.7.7" }, { "author": "whyrusleeping", @@ -195,15 +195,15 @@ }, { "author": "whyrusleeping", - "hash": "QmfZTdmunzKzAGJrSvXXQbQ5kLLUiEMX5vdwux7iXkdk7D", + "hash": "QmaSfSMvc1VPZ8JbMponFs4WHvF9FgEruF56opm5E1RgQA", "name": "go-libp2p-host", - "version": "2.1.7" + "version": "2.1.8" }, { "author": "whyrusleeping", - "hash": "QmRqfgh56f8CrqpwH7D2s6t8zQRsvPoftT3sp5Y6SUhNA3", + "hash": "QmWhhwxrH6msgQ3UFiDs2a5DQKwQQHroNJVkPkK8e8AuEX", "name": "go-libp2p-swarm", - "version": "2.1.7" + "version": "2.1.8" }, { "author": "whyrusleeping", @@ -213,15 +213,15 @@ }, { "author": "whyrusleeping", - "hash": "Qmb6BsZf6Y3kxffXMNTubGPF1w1bkHtpvhfYbmnwP3NQyw", + "hash": "QmYELA7ZYpWS1MYbpxEq1mMJsoBC33EvKfdWwfUkjc9wHP", "name": "go-libp2p-netutil", - "version": "0.3.11" + "version": "0.3.12" }, { "author": "whyrusleeping", - "hash": "Qmc64U41EEB4nPG7wxjEqFwKJajS2f8kk5q2TvUrQf78Xu", + "hash": "QmYEmPwCBe7ZUFfuymozopHTuF3JXejvJPDAjwtyQCrsDi", "name": "go-libp2p-blankhost", - "version": "0.2.7" + "version": "0.2.8" }, { "author": "whyrusleeping", @@ -243,9 +243,9 @@ }, { "author": "whyrusleeping", - "hash": "QmVniQJkdzLZaZwzwMdd3dJTvWiJ1DQEkreVy6hs6h7Vk5", + "hash": "QmWzjXAyBTygw6CeCTUnhJzhFucfxY5FJivSoiGuiSbPjS", "name": "go-smux-multistream", - "version": "2.0.0" + "version": "2.0.1" }, { "author": "whyrusleeping", @@ -255,9 +255,9 @@ }, { "author": "vyzo", - "hash": "QmZRbCo2gw7ghw5m7L77a8FvvQTVr62J4hmy8ozpdq7dHF", + "hash": "QmcE2pXXX3aLmNFoh75F3qZ2k5KaicZjvbuGPtBZ225K12", "name": "go-libp2p-circuit", - "version": "2.0.12" + "version": "2.0.13" }, { "author": "lgierth", From a018a64001bc93520a6a349971c548fa9d5dd9a1 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Fri, 1 Jun 2018 17:35:35 +0200 Subject: [PATCH 0463/3965] gx publish 5.0.18 License: MIT Signed-off-by: Hector Sanjuan --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 23914f7206..dceb6e2290 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -5.0.17: QmWsV6kzPaYGBDVyuUfWBvyQygEc9Qrv9vzo8vZ7X4mdLN +5.0.18: QmdKZLFmdWyCyF1FzFoKHsLg4iHr44bX6Q7WqqV8oicmmE diff --git a/package.json b/package.json index 3b0a135790..fbf49997b5 100644 --- a/package.json +++ b/package.json @@ -295,6 +295,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "5.0.17" + "version": "5.0.18" } From d9dda7e931af0840d85997d3e556d1a8d5004054 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Mon, 4 Jun 2018 12:49:21 +0200 Subject: [PATCH 0464/3965] gx: update yamux --- package.json | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/package.json b/package.json index fbf49997b5..ec82c8a1ed 100644 --- a/package.json +++ b/package.json @@ -201,9 +201,9 @@ }, { "author": "whyrusleeping", - "hash": "QmWhhwxrH6msgQ3UFiDs2a5DQKwQQHroNJVkPkK8e8AuEX", + "hash": "QmRpKdg1xs4Yyrn9yrVYRBp7AQqyRxMLpD6Jgp1eZAGqEr", "name": "go-libp2p-swarm", - "version": "2.1.8" + "version": "2.1.9" }, { "author": "whyrusleeping", @@ -213,9 +213,9 @@ }, { "author": "whyrusleeping", - "hash": "QmYELA7ZYpWS1MYbpxEq1mMJsoBC33EvKfdWwfUkjc9wHP", + "hash": "Qma2UuHusnaFV24DgeZ5hyrM9uc4UdyVaZbtn2FQsPRhES", "name": "go-libp2p-netutil", - "version": "0.3.12" + "version": "0.3.13" }, { "author": "whyrusleeping", @@ -231,9 +231,9 @@ }, { "author": "whyrusleeping", - "hash": "QmNWCEvi7bPRcvqAV8AKLGVNoQdArWi7NJayka2SM4XtRe", + "hash": "QmcsgrV3nCAKjiHKZhKVXWc4oY3WBECJCqahXEMpHeMrev", "name": "go-smux-yamux", - "version": "2.0.1" + "version": "2.0.3" }, { "author": "whyrusleeping", @@ -255,9 +255,9 @@ }, { "author": "vyzo", - "hash": "QmcE2pXXX3aLmNFoh75F3qZ2k5KaicZjvbuGPtBZ225K12", + "hash": "QmR5sXZi68rm9m2E3KiXj6hE5m3GeLaDjbLPUeV6W3MLR8", "name": "go-libp2p-circuit", - "version": "2.0.13" + "version": "2.0.14" }, { "author": "lgierth", From 83e823ac893ab0bb38f5f7363fb34490dee1a0a1 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Mon, 4 Jun 2018 12:49:49 +0200 Subject: [PATCH 0465/3965] gx publish 5.0.19 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index dceb6e2290..bb10b19600 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -5.0.18: QmdKZLFmdWyCyF1FzFoKHsLg4iHr44bX6Q7WqqV8oicmmE +5.0.19: QmbMvtGDKVBEunMNUeCYJALe1LSzFAWRcK5adqySpKdXpf diff --git a/package.json b/package.json index ec82c8a1ed..68801c7f73 100644 --- a/package.json +++ b/package.json @@ -295,6 +295,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "5.0.18" + "version": "5.0.19" } From c8617e01825824cf25e8e90456a8f24d7dadab96 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 4 Jun 2018 09:25:58 -0700 Subject: [PATCH 0466/3965] gx publish 5.0.20 --- .gx/lastpubver | 2 +- package.json | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index bb10b19600..19e85ccc36 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -5.0.19: QmbMvtGDKVBEunMNUeCYJALe1LSzFAWRcK5adqySpKdXpf +5.0.20: Qmbok4DkFi1nFBgM6ZPUM7BMGsUt6iYEdSdStCksQNLEeM diff --git a/package.json b/package.json index 68801c7f73..0602e3e3c2 100644 --- a/package.json +++ b/package.json @@ -273,9 +273,9 @@ }, { "author": "whyrusleeping", - "hash": "QmenmFuirGzv8S1R3DyvbZ6tFmQapkGeDCebgYzni1Ntn3", + "hash": "QmeDXFtW7tXBzgu2bKFkHkcabw1QxiusmPbYU4XXiUuXDJ", "name": "go-smux-multiplex", - "version": "3.0.6" + "version": "3.0.7" }, { "author": "multiformats", @@ -295,6 +295,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "5.0.19" + "version": "5.0.20" } From 9c69293c7be6a087c811f7657b1705e2ced30a31 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 4 Jun 2018 09:58:28 -0700 Subject: [PATCH 0467/3965] gx publish 5.0.21 --- .gx/lastpubver | 2 +- package.json | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 19e85ccc36..ce1a045622 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -5.0.20: Qmbok4DkFi1nFBgM6ZPUM7BMGsUt6iYEdSdStCksQNLEeM +5.0.21: QmY6iAoG9DVgZwh5ZRcQEpa2uErAe1Hbei8qXPCjpDS9Ge diff --git a/package.json b/package.json index 0602e3e3c2..f70b3baafb 100644 --- a/package.json +++ b/package.json @@ -273,9 +273,9 @@ }, { "author": "whyrusleeping", - "hash": "QmeDXFtW7tXBzgu2bKFkHkcabw1QxiusmPbYU4XXiUuXDJ", + "hash": "QmZeGmoJ3bEwEe6Huz6GKcHENWZCx7DReuAS5va4zP24PB", "name": "go-smux-multiplex", - "version": "3.0.7" + "version": "3.0.8" }, { "author": "multiformats", @@ -295,6 +295,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "5.0.20" + "version": "5.0.21" } From 4784a35f5244d8d365682c465e773cb93e8b6a33 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 18 Jan 2018 22:04:35 -0800 Subject: [PATCH 0468/3965] refactor for transport interface changes Also, refactor out reuseport logic into a separate package. --- p2p/transport/tcp/reuseport.go | 30 --- p2p/transport/tcp/reuseport_test.go | 48 ----- p2p/transport/tcp/tcp.go | 286 +++++----------------------- p2p/transport/tcp/tcp_test.go | 77 ++++---- 4 files changed, 79 insertions(+), 362 deletions(-) delete mode 100644 p2p/transport/tcp/reuseport_test.go diff --git a/p2p/transport/tcp/reuseport.go b/p2p/transport/tcp/reuseport.go index db2b3da698..58105ce537 100644 --- a/p2p/transport/tcp/reuseport.go +++ b/p2p/transport/tcp/reuseport.go @@ -1,10 +1,8 @@ package tcp import ( - "net" "os" "strings" - "syscall" reuseport "github.com/libp2p/go-reuseport" ) @@ -35,31 +33,3 @@ func init() { func ReuseportIsAvailable() bool { return envReuseportVal && reuseport.Available() } - -// ReuseErrShouldRetry diagnoses whether to retry after a reuse error. -// if we failed to bind, we should retry. if bind worked and this is a -// real dial error (remote end didnt answer) then we should not retry. -func ReuseErrShouldRetry(err error) bool { - if err == nil { - return false // hey, it worked! no need to retry. - } - - // if it's a network timeout error, it's a legitimate failure. - if nerr, ok := err.(net.Error); ok && nerr.Timeout() { - return false - } - - errno, ok := err.(syscall.Errno) - if !ok { // not an errno? who knows what this is. retry. - return true - } - - switch errno { - case syscall.EADDRINUSE, syscall.EADDRNOTAVAIL: - return true // failure to bind. retry. - case syscall.ECONNREFUSED: - return false // real dial error - default: - return true // optimistically default to retry. - } -} diff --git a/p2p/transport/tcp/reuseport_test.go b/p2p/transport/tcp/reuseport_test.go deleted file mode 100644 index 1b03ffd33b..0000000000 --- a/p2p/transport/tcp/reuseport_test.go +++ /dev/null @@ -1,48 +0,0 @@ -package tcp - -import ( - "net" - "syscall" - "testing" -) - -type netTimeoutErr struct { - timeout bool -} - -func (e netTimeoutErr) Error() string { - return "" -} - -func (e netTimeoutErr) Timeout() bool { - return e.timeout -} - -func (e netTimeoutErr) Temporary() bool { - panic("not checked") -} - -func TestReuseError(t *testing.T) { - var nte1 net.Error = &netTimeoutErr{true} - var nte2 net.Error = &netTimeoutErr{false} - - cases := map[error]bool{ - nil: false, - syscall.EADDRINUSE: true, - syscall.EADDRNOTAVAIL: true, - syscall.ECONNREFUSED: false, - - nte1: false, - nte2: true, // this ones a little weird... we should check neterror.Temporary() too - - // test 'default' to true - syscall.EBUSY: true, - } - - for k, v := range cases { - if ReuseErrShouldRetry(k) != v { - t.Fatalf("expected %t for %#v", v, k) - } - } - -} diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 0711edebd6..3ed2ff3f2f 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -2,13 +2,12 @@ package tcp import ( "context" - "fmt" - "net" - "sync" logging "github.com/ipfs/go-log" + peer "github.com/libp2p/go-libp2p-peer" tpt "github.com/libp2p/go-libp2p-transport" - reuseport "github.com/libp2p/go-reuseport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + rtpt "github.com/libp2p/go-reuseport-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" @@ -16,275 +15,80 @@ import ( var log = logging.Logger("tcp-tpt") +// TcpTransport is the TCP transport. type TcpTransport struct { - dlock sync.Mutex - dialers map[string]tpt.Dialer + // Connection upgrader for upgrading insecure stream connections to + // secure multiplex connections. + Upgrader *tptu.Upgrader - llock sync.Mutex - listeners map[string]tpt.Listener + // Explicitly disable reuseport. + DisableReuseport bool + + reuse rtpt.Transport } var _ tpt.Transport = &TcpTransport{} // NewTCPTransport creates a tcp transport object that tracks dialers and listeners // created. It represents an entire tcp stack (though it might not necessarily be) -func NewTCPTransport() *TcpTransport { - return &TcpTransport{ - dialers: make(map[string]tpt.Dialer), - listeners: make(map[string]tpt.Listener), - } -} - -func (t *TcpTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { - if laddr == nil { - zaddr, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") - if err != nil { - return nil, err - } - laddr = zaddr - } - t.dlock.Lock() - defer t.dlock.Unlock() - s := laddr.String() - d, found := t.dialers[s] - if found { - return d, nil - } - var doReuse bool - for _, o := range opts { - switch o := o.(type) { - case tpt.ReuseportOpt: - doReuse = bool(o) - default: - return nil, fmt.Errorf("unrecognized option: %#v", o) - } - } - - tcpd, err := t.newTcpDialer(laddr, doReuse) - if err != nil { - return nil, err - } - - t.dialers[s] = tcpd - return tcpd, nil -} - -func (t *TcpTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { - if !t.Matches(laddr) { - return nil, fmt.Errorf("tcp transport cannot listen on %q", laddr) - } - - t.llock.Lock() - defer t.llock.Unlock() - s := laddr.String() - l, found := t.listeners[s] - if found { - return l, nil - } - - list, err := manetListen(laddr) - if err != nil { - return nil, err - } - - tlist := &tcpListener{ - list: list, - transport: t, - } - - t.listeners[s] = tlist - return tlist, nil -} - -func manetListen(addr ma.Multiaddr) (manet.Listener, error) { - network, naddr, err := manet.DialArgs(addr) - if err != nil { - return nil, err - } - - if ReuseportIsAvailable() { - nl, err := reuseport.Listen(network, naddr) - if err == nil { - // hey, it worked! - return manet.WrapNetListener(nl) - } - // reuseport is available, but we failed to listen. log debug, and retry normally. - log.Debugf("reuseport available, but failed to listen: %s %s, %s", network, naddr, err) - } - - // either reuseport not available, or it failed. try normally. - return manet.Listen(addr) -} - -func (t *TcpTransport) Matches(a ma.Multiaddr) bool { - return mafmt.TCP.Matches(a) +func NewTCPTransport(upgrader *tptu.Upgrader) *TcpTransport { + return &TcpTransport{Upgrader: upgrader} } -type tcpDialer struct { - laddr ma.Multiaddr - - doReuse bool - - rd reuseport.Dialer - madialer manet.Dialer - pattern mafmt.Pattern - - transport tpt.Transport +// CanDial returns true if this transport believes it can dial the given +// multiaddr. +func (t *TcpTransport) CanDial(addr ma.Multiaddr) bool { + return mafmt.TCP.Matches(addr) } -var _ tpt.Dialer = &tcpDialer{} - -func maddrToTcp(addr ma.Multiaddr) (*net.TCPAddr, error) { - la, err := manet.ToNetAddr(addr) - if err != nil { - return nil, err // something wrong with addr. +func (t *TcpTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { + if t.UseReuseport() { + return t.reuse.DialContext(ctx, raddr) } - latcp, ok := la.(*net.TCPAddr) - if !ok { - return nil, fmt.Errorf("not a tcp multiaddr: %s", addr) - } - return latcp, nil + var d manet.Dialer + return d.DialContext(ctx, raddr) } -func (t *TcpTransport) newTcpDialer(laddr ma.Multiaddr, doReuse bool) (*tcpDialer, error) { - // get the local net.Addr manually - la, err := maddrToTcp(laddr) +// Dial dials the peer at the remote address. +func (t *TcpTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { + conn, err := t.maDial(ctx, raddr) if err != nil { return nil, err } - - var pattern mafmt.Pattern - if TCP4.Matches(laddr) { - pattern = TCP4 - } else if TCP6.Matches(laddr) { - pattern = TCP6 - } else { - return nil, fmt.Errorf("local addr did not match TCP4 or TCP6: %s", laddr) - } - - // Ignore the port when constructing the default (non-reuseport) dialer. - labase := *la - labase.Port = 0 - - dialer := &tcpDialer{ - laddr: laddr, - pattern: pattern, - madialer: manet.Dialer{ - Dialer: net.Dialer{ - LocalAddr: &labase, - }, - }, - transport: t, - } - - if doReuse && ReuseportIsAvailable() { - dialer.doReuse = true - dialer.rd = reuseport.Dialer{ - D: net.Dialer{ - LocalAddr: la, - }, - } - } - return dialer, nil -} - -func (d *tcpDialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { - return d.DialContext(context.Background(), raddr) + return t.Upgrader.UpgradeOutbound(ctx, t, conn, p) } -func (d *tcpDialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { - var c manet.Conn - var err error - if d.doReuse { - c, err = d.reuseDial(ctx, raddr) - } else { - c, err = d.madialer.DialContext(ctx, raddr) - } - - if err != nil { - return nil, err - } - - return &tcpConn{ - Conn: c, - t: d.transport, - }, nil +// UseReuseport returns true if reuseport is enabled and available. +func (t *TcpTransport) UseReuseport() bool { + return !t.DisableReuseport && ReuseportIsAvailable() } -func (d *tcpDialer) reuseDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { - network, netraddr, err := manet.DialArgs(raddr) - if err != nil { - return nil, err - } - - rpev := log.EventBegin(ctx, "tptDialReusePort", logging.LoggableMap{ - "raddr": raddr, - }) - - con, err := d.rd.DialContext(ctx, network, netraddr) - if err == nil { - rpev.Done() - return manet.WrapNetConn(con) +func (t *TcpTransport) maListen(laddr ma.Multiaddr) (manet.Listener, error) { + if t.UseReuseport() { + return t.reuse.Listen(laddr) } - rpev.SetError(err) - rpev.Done() - - if !ReuseErrShouldRetry(err) { - return nil, err - } - - return d.madialer.DialContext(ctx, raddr) -} - -var TCP4 = mafmt.And(mafmt.Base(ma.P_IP4), mafmt.Base(ma.P_TCP)) -var TCP6 = mafmt.And(mafmt.Base(ma.P_IP6), mafmt.Base(ma.P_TCP)) - -func (d *tcpDialer) Matches(a ma.Multiaddr) bool { - return d.pattern.Matches(a) -} - -type tcpListener struct { - list manet.Listener - transport tpt.Transport + return manet.Listen(laddr) } -var _ tpt.Listener = &tcpListener{} - -func (d *tcpListener) Accept() (tpt.Conn, error) { - c, err := d.list.Accept() +// Listen listens on the given multiaddr. +func (t *TcpTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { + list, err := t.maListen(laddr) if err != nil { return nil, err } - - return &tcpConn{ - Conn: c, - t: d.transport, - }, nil -} - -func (d *tcpListener) Addr() net.Addr { - return d.list.Addr() + return t.Upgrader.UpgradeListener(t, list), nil } -func (t *tcpListener) Multiaddr() ma.Multiaddr { - return t.list.Multiaddr() +// Protocols returns the list of terminal protocols this transport can dial. +func (t *TcpTransport) Protocols() []int { + return []int{ma.P_TCP} } -func (t *tcpListener) NetListener() net.Listener { - return t.list.NetListener() +// Proxy always returns false for the TCP transport. +func (t *TcpTransport) Proxy() bool { + return false } -func (d *tcpListener) Close() error { - return d.list.Close() -} - -type tcpConn struct { - manet.Conn - t tpt.Transport -} - -var _ tpt.Conn = &tcpConn{} - -func (c *tcpConn) Transport() tpt.Transport { - return c.t +func (t *TcpTransport) String() string { + return "TCP" } diff --git a/p2p/transport/tcp/tcp_test.go b/p2p/transport/tcp/tcp_test.go index 2a9a21b49d..4467e8a2d7 100644 --- a/p2p/transport/tcp/tcp_test.go +++ b/p2p/transport/tcp/tcp_test.go @@ -3,59 +3,50 @@ package tcp import ( "testing" - tpt "github.com/libp2p/go-libp2p-transport" + insecure "github.com/libp2p/go-conn-security/insecure" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" utils "github.com/libp2p/go-libp2p-transport/test" ma "github.com/multiformats/go-multiaddr" + mplex "github.com/whyrusleeping/go-smux-multiplex" ) func TestTcpTransport(t *testing.T) { - ta := NewTCPTransport() - tb := NewTCPTransport() - - zero := "/ip4/127.0.0.1/tcp/0" - utils.SubtestTransport(t, ta, tb, zero) + for i := 0; i < 2; i++ { + ta := NewTCPTransport(&tptu.Upgrader{ + Secure: insecure.New("peerA"), + Muxer: new(mplex.Transport), + }) + tb := NewTCPTransport(&tptu.Upgrader{ + Secure: insecure.New("peerB"), + Muxer: new(mplex.Transport), + }) + + zero := "/ip4/127.0.0.1/tcp/0" + utils.SubtestTransport(t, ta, tb, zero, "peerA") + + envReuseportVal = false + } + envReuseportVal = true } func TestTcpTransportCantListenUtp(t *testing.T) { - utpa, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/utp") - if err != nil { - t.Fatal(err) - } + for i := 0; i < 2; i++ { + utpa, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/utp") + if err != nil { + t.Fatal(err) + } - tpt := NewTCPTransport() - _, err = tpt.Listen(utpa) - if err == nil { - t.Fatal("shouldnt be able to listen on utp addr with tcp transport") - } -} + tpt := NewTCPTransport(&tptu.Upgrader{ + Secure: insecure.New("peerB"), + Muxer: new(mplex.Transport), + }) -func TestCorrectIPVersionMatching(t *testing.T) { - ta := NewTCPTransport() - - addr4, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") - if err != nil { - t.Fatal(err) - } - addr6, err := ma.NewMultiaddr("/ip6/::1/tcp/0") - if err != nil { - t.Fatal(err) - } - - d4, err := ta.Dialer(addr4, tpt.ReuseportOpt(true)) - if err != nil { - t.Fatal(err) - } - - d6, err := ta.Dialer(addr6, tpt.ReuseportOpt(true)) - if err != nil { - t.Fatal(err) - } - - if d4.Matches(addr6) { - t.Fatal("tcp4 dialer should not match ipv6 address") - } + _, err = tpt.Listen(utpa) + if err == nil { + t.Fatal("shouldnt be able to listen on utp addr with tcp transport") + } - if d6.Matches(addr4) { - t.Fatal("tcp4 dialer should not match ipv6 address") + envReuseportVal = false } + envReuseportVal = true } From 42780280b424e6b3aca737481fdc6be983049340 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 15 Feb 2018 20:01:38 -0800 Subject: [PATCH 0469/3965] fixup for transport refactor Also, use the now standardized transport tests (and fix bugs around closing the listener). --- p2p/transport/websocket/dialer.go | 42 ---------- p2p/transport/websocket/dialer_test.go | 31 ------- p2p/transport/websocket/listener.go | 88 +++++++++++--------- p2p/transport/websocket/websocket.go | 98 +++++++++++++++++++---- p2p/transport/websocket/websocket_test.go | 58 +++++++++++--- 5 files changed, 182 insertions(+), 135 deletions(-) delete mode 100644 p2p/transport/websocket/dialer.go delete mode 100644 p2p/transport/websocket/dialer_test.go diff --git a/p2p/transport/websocket/dialer.go b/p2p/transport/websocket/dialer.go deleted file mode 100644 index 7a5a91579a..0000000000 --- a/p2p/transport/websocket/dialer.go +++ /dev/null @@ -1,42 +0,0 @@ -package websocket - -import ( - "context" - - ws "github.com/gorilla/websocket" - tpt "github.com/libp2p/go-libp2p-transport" - ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" -) - -type dialer struct{} - -func (d *dialer) Dial(raddr ma.Multiaddr) (tpt.Conn, error) { - return d.DialContext(context.Background(), raddr) -} - -func (d *dialer) DialContext(ctx context.Context, raddr ma.Multiaddr) (tpt.Conn, error) { - wsurl, err := parseMultiaddr(raddr) - if err != nil { - return nil, err - } - - wscon, _, err := ws.DefaultDialer.Dial(wsurl, nil) - if err != nil { - return nil, err - } - - mnc, err := manet.WrapNetConn(NewConn(wscon, nil)) - if err != nil { - wscon.Close() - return nil, err - } - - return &wsConn{ - Conn: mnc, - }, nil -} - -func (d *dialer) Matches(a ma.Multiaddr) bool { - return WsFmt.Matches(a) -} diff --git a/p2p/transport/websocket/dialer_test.go b/p2p/transport/websocket/dialer_test.go deleted file mode 100644 index 2e8b5caf59..0000000000 --- a/p2p/transport/websocket/dialer_test.go +++ /dev/null @@ -1,31 +0,0 @@ -package websocket - -import ( - "testing" - - ma "github.com/multiformats/go-multiaddr" -) - -func TestDialerMatches(t *testing.T) { - addrWs, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") - if err != nil { - t.Fatal(err) - } - - addrTcp, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555") - if err != nil { - t.Fatal(err) - } - - d := &dialer{} - matchTrue := d.Matches(addrWs) - matchFalse := d.Matches(addrTcp) - - if !matchTrue { - t.Fatal("expected to match websocket maddr, but did not") - } - - if matchFalse { - t.Fatal("expected to not match tcp maddr, but did") - } -} diff --git a/p2p/transport/websocket/listener.go b/p2p/transport/websocket/listener.go index 146b371a9f..3d774f09dc 100644 --- a/p2p/transport/websocket/listener.go +++ b/p2p/transport/websocket/listener.go @@ -3,33 +3,25 @@ package websocket import ( "context" "fmt" + "net" "net/http" - "net/url" - tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) -type wsConn struct { - manet.Conn - t tpt.Transport -} - -var _ tpt.Conn = (*wsConn)(nil) - -func (c *wsConn) Transport() tpt.Transport { - return c.t -} - type listener struct { - manet.Listener + net.Listener - incoming chan *Conn + laddr ma.Multiaddr - tpt tpt.Transport + closed chan struct{} + incoming chan *Conn +} - origin *url.URL +func (l *listener) serve() { + defer close(l.closed) + http.Serve(l.Listener, l) } func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { @@ -40,35 +32,55 @@ func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { } ctx, cancel := context.WithCancel(context.Background()) - l.incoming <- NewConn(c, cancel) - - // wait until conn gets closed, otherwise the handler closes it early - <-ctx.Done() -} -func (l *listener) Accept() (tpt.Conn, error) { - c, ok := <-l.incoming - if !ok { - return nil, fmt.Errorf("listener is closed") + var cnCh <-chan bool + if cn, ok := w.(http.CloseNotifier); ok { + cnCh = cn.CloseNotify() } - mnc, err := manet.WrapNetConn(c) - if err != nil { + wscon := NewConn(c, cancel) + // Just to make sure. + defer wscon.Close() + + select { + case l.incoming <- wscon: + case <-l.closed: c.Close() - return nil, err + return + case <-cnCh: + return } - return &wsConn{ - Conn: mnc, - t: l.tpt, - }, nil + // wait until conn gets closed, otherwise the handler closes it early + select { + case <-ctx.Done(): + case <-l.closed: + c.Close() + return + case <-cnCh: + return + } } -func (l *listener) Multiaddr() ma.Multiaddr { - wsma, err := ma.NewMultiaddr("/ws") - if err != nil { - panic(err) +func (l *listener) Accept() (manet.Conn, error) { + select { + case c, ok := <-l.incoming: + if !ok { + return nil, fmt.Errorf("listener is closed") + } + + mnc, err := manet.WrapNetConn(c) + if err != nil { + c.Close() + return nil, err + } + + return mnc, nil + case <-l.closed: + return nil, fmt.Errorf("listener is closed") } +} - return l.Listener.Multiaddr().Encapsulate(wsma) +func (l *listener) Multiaddr() ma.Multiaddr { + return l.laddr } diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 6a2c6722e5..b9ec1d6a9a 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -2,15 +2,18 @@ package websocket import ( + "context" "fmt" + "net" "net/http" "net/url" ws "github.com/gorilla/websocket" + peer "github.com/libp2p/go-libp2p-peer" tpt "github.com/libp2p/go-libp2p-transport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" - mafmt "github.com/whyrusleeping/mafmt" ) @@ -50,41 +53,106 @@ func init() { } // WebsocketTransport is the actual go-libp2p transport -type WebsocketTransport struct{} +type WebsocketTransport struct { + Upgrader *tptu.Upgrader +} + +func New(u *tptu.Upgrader) *WebsocketTransport { + return &WebsocketTransport{u} +} var _ tpt.Transport = (*WebsocketTransport)(nil) -func (t *WebsocketTransport) Matches(a ma.Multiaddr) bool { +func (t *WebsocketTransport) CanDial(a ma.Multiaddr) bool { return WsFmt.Matches(a) } -func (t *WebsocketTransport) Dialer(_ ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { - return &dialer{}, nil +func (t *WebsocketTransport) Protocols() []int { + return []int{WsProtocol.Code} } -func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { - list, err := manet.Listen(a) +func (t *WebsocketTransport) Proxy() bool { + return false +} + +func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { + wsurl, err := parseMultiaddr(raddr) + if err != nil { + return nil, err + } + + wscon, _, err := ws.DefaultDialer.Dial(wsurl, nil) + if err != nil { + return nil, err + } + + mnc, err := manet.WrapNetConn(NewConn(wscon, nil)) + if err != nil { + wscon.Close() + return nil, err + } + return mnc, nil +} + +func (t *WebsocketTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { + macon, err := t.maDial(ctx, raddr) + if err != nil { + return nil, err + } + return t.Upgrader.UpgradeOutbound(ctx, t, macon, p) +} + +func (t *WebsocketTransport) maListen(a ma.Multiaddr) (manet.Listener, error) { + lnet, lnaddr, err := manet.DialArgs(a) + if err != nil { + return nil, err + } + + nl, err := net.Listen(lnet, lnaddr) if err != nil { return nil, err } - u, err := url.Parse("http://" + list.Addr().String()) + u, err := url.Parse("http://" + nl.Addr().String()) if err != nil { + nl.Close() return nil, err } - tlist := t.wrapListener(list, u) + malist, err := t.wrapListener(nl, u) + if err != nil { + nl.Close() + return nil, err + } - go http.Serve(list.NetListener(), tlist) + go malist.serve() - return tlist, nil + return malist, nil } -func (t *WebsocketTransport) wrapListener(l manet.Listener, origin *url.URL) *listener { +func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { + malist, err := t.maListen(a) + if err != nil { + return nil, err + } + return t.Upgrader.UpgradeListener(t, malist), nil +} + +func (t *WebsocketTransport) wrapListener(l net.Listener, origin *url.URL) (*listener, error) { + laddr, err := manet.FromNetAddr(l.Addr()) + if err != nil { + return nil, err + } + wsma, err := ma.NewMultiaddr("/ws") + if err != nil { + return nil, err + } + laddr = laddr.Encapsulate(wsma) + return &listener{ + laddr: laddr, Listener: l, incoming: make(chan *Conn), - tpt: t, - origin: origin, - } + closed: make(chan struct{}), + }, nil } diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index 61bfa14230..2c9ad32441 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -2,14 +2,57 @@ package websocket import ( "bytes" + "context" "io" "io/ioutil" "testing" "testing/iotest" + insecure "github.com/libp2p/go-conn-security/insecure" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + utils "github.com/libp2p/go-libp2p-transport/test" ma "github.com/multiformats/go-multiaddr" + mplex "github.com/whyrusleeping/go-smux-multiplex" ) +func TestCanDial(t *testing.T) { + addrWs, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") + if err != nil { + t.Fatal(err) + } + + addrTCP, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555") + if err != nil { + t.Fatal(err) + } + + d := &WebsocketTransport{} + matchTrue := d.CanDial(addrWs) + matchFalse := d.CanDial(addrTCP) + + if !matchTrue { + t.Fatal("expected to match websocket maddr, but did not") + } + + if matchFalse { + t.Fatal("expected to not match tcp maddr, but did") + } +} + +func TestWebsocketTransport(t *testing.T) { + ta := New(&tptu.Upgrader{ + Secure: insecure.New("peerA"), + Muxer: new(mplex.Transport), + }) + tb := New(&tptu.Upgrader{ + Secure: insecure.New("peerB"), + Muxer: new(mplex.Transport), + }) + + zero := "/ip4/127.0.0.1/tcp/0/ws" + utils.SubtestTransport(t, ta, tb, zero, "peerA") +} + func TestWebsocketListen(t *testing.T) { zero, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0/ws") if err != nil { @@ -17,7 +60,7 @@ func TestWebsocketListen(t *testing.T) { } tpt := &WebsocketTransport{} - l, err := tpt.Listen(zero) + l, err := tpt.maListen(zero) if err != nil { t.Fatal(err) } @@ -26,8 +69,7 @@ func TestWebsocketListen(t *testing.T) { msg := []byte("HELLO WORLD") go func() { - d, _ := tpt.Dialer(nil) - c, err := d.Dial(l.Multiaddr()) + c, err := tpt.maDial(context.Background(), l.Multiaddr()) if err != nil { t.Error(err) return @@ -62,7 +104,7 @@ func TestConcurrentClose(t *testing.T) { } tpt := &WebsocketTransport{} - l, err := tpt.Listen(zero) + l, err := tpt.maListen(zero) if err != nil { t.Fatal(err) } @@ -71,9 +113,8 @@ func TestConcurrentClose(t *testing.T) { msg := []byte("HELLO WORLD") go func() { - d, _ := tpt.Dialer(nil) for i := 0; i < 100; i++ { - c, err := d.Dial(l.Multiaddr()) + c, err := tpt.maDial(context.Background(), l.Multiaddr()) if err != nil { t.Error(err) return @@ -100,7 +141,7 @@ func TestWriteZero(t *testing.T) { } tpt := &WebsocketTransport{} - l, err := tpt.Listen(zero) + l, err := tpt.maListen(zero) if err != nil { t.Fatal(err) } @@ -109,8 +150,7 @@ func TestWriteZero(t *testing.T) { msg := []byte(nil) go func() { - d, _ := tpt.Dialer(nil) - c, err := d.Dial(l.Multiaddr()) + c, err := tpt.maDial(context.Background(), l.Multiaddr()) defer c.Close() if err != nil { t.Error(err) From 4e39954ac98603db3510b257e5d2d4287ea0b5a3 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 19 Jan 2018 16:28:45 -0800 Subject: [PATCH 0470/3965] refactor for new transports Removes: * go-libp2p-conn logic (moved to transports) * go-libp2p-peerstream (moved here) Changes: * New transport interface. * Explicit over implicit (the implicit automagic now all lives in go-libp2p): * No more default transports, muxers, etc. * No more fallback dialer. Transports are now *required*. * Stream opening: * Connection picking logic (instead of just picking the first). * Tries harder to open a stream of some connections happen to be closed. * Stream closing: * No longer treats half-closed streams as fully closed. Users *must* read the an EOF or reset the stream for it to be garbage collected. * No more polling for dead connections. * Multiplexers are now *non-optional*. Really, they haven't been optional for a while but we still pretended that they were. * No more Network type alias. It added a bunch of code and didn't really provide anything but an alternative set of methods that do the same thing. * Notifications: * New guarantee: connection open notifications will complete before connection close notifications begin. * Given that, notifications are now delivered in parallel. No more notification backlogs blocking connection closing/opening. --- p2p/net/swarm/dial_sync.go | 23 +- p2p/net/swarm/dial_sync_test.go | 4 +- p2p/net/swarm/dial_test.go | 125 +++--- p2p/net/swarm/limiter.go | 10 +- p2p/net/swarm/limiter_test.go | 30 +- p2p/net/swarm/peers_test.go | 23 +- p2p/net/swarm/simul_test.go | 17 +- p2p/net/swarm/swarm.go | 610 +++++++++++++++++------------- p2p/net/swarm/swarm_addr.go | 12 +- p2p/net/swarm/swarm_addr_test.go | 92 +---- p2p/net/swarm/swarm_conn.go | 229 +++++++---- p2p/net/swarm/swarm_dial.go | 211 ++++++----- p2p/net/swarm/swarm_listen.go | 159 ++------ p2p/net/swarm/swarm_net.go | 177 --------- p2p/net/swarm/swarm_net_test.go | 32 +- p2p/net/swarm/swarm_notif_test.go | 26 +- p2p/net/swarm/swarm_stream.go | 146 +++++-- p2p/net/swarm/swarm_test.go | 75 ++-- p2p/net/swarm/swarm_transport.go | 87 +++++ p2p/net/swarm/testing/testing.go | 97 +++++ 20 files changed, 1151 insertions(+), 1034 deletions(-) delete mode 100644 p2p/net/swarm/swarm_net.go create mode 100644 p2p/net/swarm/swarm_transport.go create mode 100644 p2p/net/swarm/testing/testing.go diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index 69739e54ec..82256eca48 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -7,8 +7,10 @@ import ( peer "github.com/libp2p/go-libp2p-peer" ) +// DialFunc is the type of function expected by DialSync. type DialFunc func(context.Context, peer.ID) (*Conn, error) +// NewDialSync constructs a new DialSync func NewDialSync(dfn DialFunc) *DialSync { return &DialSync{ dials: make(map[peer.ID]*activeDial), @@ -16,6 +18,8 @@ func NewDialSync(dfn DialFunc) *DialSync { } } +// DialSync is a dial synchronization helper that ensures that at most one dial +// to any given peer is active at any given time. type DialSync struct { dials map[peer.ID]*activeDial dialsLk sync.Mutex @@ -35,11 +39,11 @@ type activeDial struct { ds *DialSync } -func (dr *activeDial) wait(ctx context.Context) (*Conn, error) { - defer dr.decref() +func (ad *activeDial) wait(ctx context.Context) (*Conn, error) { + defer ad.decref() select { - case <-dr.waitch: - return dr.conn, dr.err + case <-ad.waitch: + return ad.conn, ad.err case <-ctx.Done(): return nil, ctx.Err() } @@ -102,6 +106,17 @@ func (ds *DialSync) getActiveDial(p peer.ID) *activeDial { return actd } +// DialLock initiates a dial to the given peer if there are none in progress +// then waits for the dial to that peer to complete. func (ds *DialSync) DialLock(ctx context.Context, p peer.ID) (*Conn, error) { return ds.getActiveDial(p).wait(ctx) } + +// CancelDial cancels all in-progress dials to the given peer. +func (ds *DialSync) CancelDial(p peer.ID) { + ds.dialsLk.Lock() + defer ds.dialsLk.Unlock() + if ad, ok := ds.dials[p]; ok { + ad.cancel() + } +} diff --git a/p2p/net/swarm/dial_sync_test.go b/p2p/net/swarm/dial_sync_test.go index 0d70e226af..53c3fc25b4 100644 --- a/p2p/net/swarm/dial_sync_test.go +++ b/p2p/net/swarm/dial_sync_test.go @@ -1,4 +1,4 @@ -package swarm +package swarm_test import ( "context" @@ -7,6 +7,8 @@ import ( "testing" "time" + . "github.com/libp2p/go-libp2p-swarm" + peer "github.com/libp2p/go-libp2p-peer" ) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index dabb0bdb00..a7eebcea1f 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -1,4 +1,4 @@ -package swarm +package swarm_test import ( "context" @@ -10,19 +10,26 @@ import ( addrutil "github.com/libp2p/go-addr-util" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" testutil "github.com/libp2p/go-testutil" ci "github.com/libp2p/go-testutil/ci" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" + + . "github.com/libp2p/go-libp2p-swarm" ) +func init() { + DialTimeout = time.Second +} + func closeSwarms(swarms []*Swarm) { for _, s := range swarms { s.Close() } } -func TestBasicDial(t *testing.T) { +func TestBasicDialPeer(t *testing.T) { t.Parallel() ctx := context.Background() @@ -31,9 +38,9 @@ func TestBasicDial(t *testing.T) { s1 := swarms[0] s2 := swarms[1] - s1.peers.AddAddrs(s2.local, s2.ListenAddresses(), pstore.PermanentAddrTTL) + s1.Peerstore().AddAddrs(s2.LocalPeer(), s2.ListenAddresses(), pstore.PermanentAddrTTL) - c, err := s1.Dial(ctx, s2.local) + c, err := s1.DialPeer(ctx, s2.LocalPeer()) if err != nil { t.Fatal(err) } @@ -56,9 +63,9 @@ func TestDialWithNoListeners(t *testing.T) { defer closeSwarms(swarms) s2 := swarms[0] - s1.peers.AddAddrs(s2.local, s2.ListenAddresses(), pstore.PermanentAddrTTL) + s1.Peerstore().AddAddrs(s2.LocalPeer(), s2.ListenAddresses(), pstore.PermanentAddrTTL) - c, err := s1.Dial(ctx, s2.local) + c, err := s1.DialPeer(ctx, s2.LocalPeer()) if err != nil { t.Fatal(err) } @@ -92,16 +99,16 @@ func TestSimultDials(t *testing.T) { t.Parallel() ctx := context.Background() - swarms := makeSwarms(ctx, t, 2) + swarms := makeSwarms(ctx, t, 2, swarmt.OptDisableReuseport) // connect everyone { var wg sync.WaitGroup connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // copy for other peer - log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.local, dst, addr) - s.peers.AddAddr(dst, addr, pstore.TempAddrTTL) - if _, err := s.Dial(ctx, dst); err != nil { + log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.LocalPeer(), dst, addr) + s.Peerstore().AddAddr(dst, addr, pstore.TempAddrTTL) + if _, err := s.DialPeer(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } wg.Done() @@ -119,18 +126,18 @@ func TestSimultDials(t *testing.T) { log.Info("Connecting swarms simultaneously.") for i := 0; i < 10; i++ { // connect 10x for each. wg.Add(2) - go connect(swarms[0], swarms[1].local, ifaceAddrs1[0]) - go connect(swarms[1], swarms[0].local, ifaceAddrs0[0]) + go connect(swarms[0], swarms[1].LocalPeer(), ifaceAddrs1[0]) + go connect(swarms[1], swarms[0].LocalPeer(), ifaceAddrs0[0]) } wg.Wait() } // should still just have 1, at most 2 connections :) - c01l := len(swarms[0].ConnectionsToPeer(swarms[1].local)) + c01l := len(swarms[0].ConnsToPeer(swarms[1].LocalPeer())) if c01l > 2 { t.Error("0->1 has", c01l) } - c10l := len(swarms[1].ConnectionsToPeer(swarms[0].local)) + c10l := len(swarms[1].ConnsToPeer(swarms[0].LocalPeer())) if c10l > 2 { t.Error("1->0 has", c10l) } @@ -168,19 +175,14 @@ func TestDialWait(t *testing.T) { s1 := swarms[0] defer s1.Close() - s1.dialT = time.Millisecond * 300 // lower timeout for tests. - if ci.IsRunning() { - s1.dialT = time.Second - } - // dial to a non-existent peer. s2p, s2addr, s2l := newSilentPeer(t) go acceptAndHang(s2l) defer s2l.Close() - s1.peers.AddAddr(s2p, s2addr, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddr(s2p, s2addr, pstore.PermanentAddrTTL) before := time.Now() - if c, err := s1.Dial(ctx, s2p); err == nil { + if c, err := s1.DialPeer(ctx, s2p); err == nil { defer c.Close() t.Fatal("error swarm dialing to unknown peer worked...", err) } else { @@ -188,15 +190,14 @@ func TestDialWait(t *testing.T) { } duration := time.Since(before) - dt := s1.dialT - if duration < dt*dialAttempts { - t.Error("< DialTimeout * dialAttempts not being respected", duration, dt*dialAttempts) + if duration < DialTimeout*DialAttempts { + t.Error("< DialTimeout * DialAttempts not being respected", duration, DialTimeout*DialAttempts) } - if duration > 2*dt*dialAttempts { - t.Error("> 2*DialTimeout * dialAttempts not being respected", duration, 2*dt*dialAttempts) + if duration > 2*DialTimeout*DialAttempts { + t.Error("> 2*DialTimeout * DialAttempts not being respected", duration, 2*DialTimeout*DialAttempts) } - if !s1.backf.Backoff(s2p) { + if !s1.Backoff().Backoff(s2p) { t.Error("s2 should now be on backoff") } } @@ -216,20 +217,17 @@ func TestDialBackoff(t *testing.T) { defer s1.Close() defer s2.Close() - s1.dialT = time.Second // lower timeout for tests. - s2.dialT = time.Second // lower timeout for tests. - s2addrs, err := s2.InterfaceListenAddresses() if err != nil { t.Fatal(err) } - s1.peers.AddAddrs(s2.local, s2addrs, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddrs(s2.LocalPeer(), s2addrs, pstore.PermanentAddrTTL) // dial to a non-existent peer. s3p, s3addr, s3l := newSilentPeer(t) go acceptAndHang(s3l) defer s3l.Close() - s1.peers.AddAddr(s3p, s3addr, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddr(s3p, s3addr, pstore.PermanentAddrTTL) // in this test we will: // 1) dial 10x to each node. @@ -246,7 +244,7 @@ func TestDialBackoff(t *testing.T) { ch := make(chan bool) for i := 0; i < times; i++ { go func() { - if _, err := s1.Dial(ctx, dst); err != nil { + if _, err := s1.DialPeer(ctx, dst); err != nil { t.Error("error dialing", dst, err) ch <- false } else { @@ -261,7 +259,7 @@ func TestDialBackoff(t *testing.T) { ch := make(chan bool) for i := 0; i < times; i++ { go func() { - if c, err := s1.Dial(ctx, dst); err != nil { + if c, err := s1.DialPeer(ctx, dst); err != nil { ch <- false } else { t.Error("succeeded in dialing", dst) @@ -276,13 +274,12 @@ func TestDialBackoff(t *testing.T) { { // 1) dial 10x to each node. N := 10 - s2done := dialOnlineNode(s2.local, N) + s2done := dialOnlineNode(s2.LocalPeer(), N) s3done := dialOfflineNode(s3p, N) // when all dials should be done by: - dialTimeout1x := time.After(s1.dialT) - // dialTimeout1Ax := time.After(s1.dialT * 2) // dialAttempts) - dialTimeout10Ax := time.After(s1.dialT * 2 * 10) // dialAttempts * 10) + dialTimeout1x := time.After(DialTimeout) + dialTimeout10Ax := time.After(DialTimeout * 2 * 10) // DialAttempts * 10) // 2) all dials should hang select { @@ -337,22 +334,22 @@ func TestDialBackoff(t *testing.T) { } // check backoff state - if s1.backf.Backoff(s2.local) { + if s1.Backoff().Backoff(s2.LocalPeer()) { t.Error("s2 should not be on backoff") } - if !s1.backf.Backoff(s3p) { + if !s1.Backoff().Backoff(s3p) { t.Error("s3 should be on backoff") } // 5) disconnect entirely - for _, c := range s1.Connections() { + for _, c := range s1.Conns() { c.Close() } - for i := 0; i < 100 && len(s1.Connections()) > 0; i++ { + for i := 0; i < 100 && len(s1.Conns()) > 0; i++ { <-time.After(time.Millisecond) } - if len(s1.Connections()) > 0 { + if len(s1.Conns()) > 0 { t.Fatal("s1 conns must exit") } } @@ -360,13 +357,12 @@ func TestDialBackoff(t *testing.T) { { // 6) dial 10x to each node again N := 10 - s2done := dialOnlineNode(s2.local, N) + s2done := dialOnlineNode(s2.LocalPeer(), N) s3done := dialOfflineNode(s3p, N) // when all dials should be done by: - dialTimeout1x := time.After(s1.dialT) - // dialTimeout1Ax := time.After(s1.dialT * 2) // dialAttempts) - dialTimeout10Ax := time.After(s1.dialT * 2 * 10) // dialAttempts * 10) + dialTimeout1x := time.After(DialTimeout) + dialTimeout10Ax := time.After(DialTimeout * 2 * 10) // DialAttempts * 10) // 7) s3 dials should all return immediately (except 1) for i := 0; i < N-1; i++ { @@ -408,10 +404,10 @@ func TestDialBackoff(t *testing.T) { } // check backoff state (the same) - if s1.backf.Backoff(s2.local) { + if s1.Backoff().Backoff(s2.LocalPeer()) { t.Error("s2 should not be on backoff") } - if !s1.backf.Backoff(s3p) { + if !s1.Backoff().Backoff(s3p) { t.Error("s3 should be on backoff") } } @@ -427,12 +423,6 @@ func TestDialBackoffClears(t *testing.T) { s2 := swarms[1] defer s1.Close() defer s2.Close() - s1.dialT = time.Millisecond * 300 // lower timeout for tests. - s2.dialT = time.Millisecond * 300 // lower timeout for tests. - if ci.IsRunning() { - s1.dialT = 2 * time.Second - s2.dialT = 2 * time.Second - } // use another address first, that accept and hang on conns _, s2bad, s2l := newSilentPeer(t) @@ -440,10 +430,10 @@ func TestDialBackoffClears(t *testing.T) { defer s2l.Close() // phase 1 -- dial to non-operational addresses - s1.peers.AddAddr(s2.local, s2bad, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddr(s2.LocalPeer(), s2bad, pstore.PermanentAddrTTL) before := time.Now() - if c, err := s1.Dial(ctx, s2.local); err == nil { + if c, err := s1.DialPeer(ctx, s2.LocalPeer()); err == nil { t.Fatal("dialing to broken addr worked...", err) defer c.Close() } else { @@ -451,15 +441,14 @@ func TestDialBackoffClears(t *testing.T) { } duration := time.Since(before) - dt := s1.dialT - if duration < dt*dialAttempts { - t.Error("< DialTimeout * dialAttempts not being respected", duration, dt*dialAttempts) + if duration < DialTimeout*DialAttempts { + t.Error("< DialTimeout * DialAttempts not being respected", duration, DialTimeout*DialAttempts) } - if duration > 2*dt*dialAttempts { - t.Error("> 2*DialTimeout * dialAttempts not being respected", duration, 2*dt*dialAttempts) + if duration > 2*DialTimeout*DialAttempts { + t.Error("> 2*DialTimeout * DialAttempts not being respected", duration, 2*DialTimeout*DialAttempts) } - if !s1.backf.Backoff(s2.local) { + if !s1.Backoff().Backoff(s2.LocalPeer()) { t.Error("s2 should now be on backoff") } else { t.Log("correctly added to backoff") @@ -470,22 +459,22 @@ func TestDialBackoffClears(t *testing.T) { if err != nil { t.Fatal(err) } - s1.peers.AddAddrs(s2.local, ifaceAddrs1, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddrs(s2.LocalPeer(), ifaceAddrs1, pstore.PermanentAddrTTL) - if _, err := s1.Dial(ctx, s2.local); err == nil { + if _, err := s1.DialPeer(ctx, s2.LocalPeer()); err == nil { t.Fatal("should have failed to dial backed off peer") } - time.Sleep(baseBackoffTime) + time.Sleep(BackoffBase) - if c, err := s1.Dial(ctx, s2.local); err != nil { + if c, err := s1.DialPeer(ctx, s2.LocalPeer()); err != nil { t.Fatal(err) } else { c.Close() t.Log("correctly connected") } - if s1.backf.Backoff(s2.local) { + if s1.Backoff().Backoff(s2.LocalPeer()) { t.Error("s2 should no longer be on backoff") } else { t.Log("correctly cleared backoff") diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 836edbc9f5..3d3437b95f 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -5,13 +5,13 @@ import ( "sync" addrutil "github.com/libp2p/go-addr-util" - iconn "github.com/libp2p/go-libp2p-interface-conn" peer "github.com/libp2p/go-libp2p-peer" + transport "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) type dialResult struct { - Conn iconn.Conn + Conn transport.Conn Addr ma.Multiaddr Err error } @@ -39,17 +39,17 @@ type dialLimiter struct { fdLimit int waitingOnFd []*dialJob - dialFunc func(context.Context, peer.ID, ma.Multiaddr) (iconn.Conn, error) + dialFunc func(context.Context, peer.ID, ma.Multiaddr) (transport.Conn, error) activePerPeer map[peer.ID]int perPeerLimit int waitingOnPeerLimit map[peer.ID][]*dialJob } -type dialfunc func(context.Context, peer.ID, ma.Multiaddr) (iconn.Conn, error) +type dialfunc func(context.Context, peer.ID, ma.Multiaddr) (transport.Conn, error) func newDialLimiter(df dialfunc) *dialLimiter { - return newDialLimiterWithParams(df, concurrentFdDials, defaultPerPeerRateLimit) + return newDialLimiterWithParams(df, ConcurrentFdDials, DefaultPerPeerRateLimit) } func newDialLimiterWithParams(df dialfunc, fdLimit, perPeerLimit int) *dialLimiter { diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 6c6eeb68fa..6ae3935caf 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -9,8 +9,8 @@ import ( "testing" "time" - iconn "github.com/libp2p/go-libp2p-interface-conn" peer "github.com/libp2p/go-libp2p-peer" + transport "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" mafmt "github.com/whyrusleeping/mafmt" ) @@ -56,13 +56,13 @@ func tryDialAddrs(ctx context.Context, l *dialLimiter, p peer.ID, addrs []ma.Mul } func hangDialFunc(hang chan struct{}) dialfunc { - return func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { + return func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.Conn, error) { if mafmt.UTP.Matches(a) { - return iconn.Conn(nil), nil + return transport.Conn(nil), nil } if tcpPortOver(a, 10) { - return iconn.Conn(nil), nil + return transport.Conn(nil), nil } <-hang @@ -74,7 +74,7 @@ func TestLimiterBasicDials(t *testing.T) { hang := make(chan struct{}) defer close(hang) - l := newDialLimiterWithParams(hangDialFunc(hang), concurrentFdDials, 4) + l := newDialLimiterWithParams(hangDialFunc(hang), ConcurrentFdDials, 4) bads := []ma.Multiaddr{addrWithPort(t, 1), addrWithPort(t, 2), addrWithPort(t, 3), addrWithPort(t, 4)} good := addrWithPort(t, 20) @@ -173,9 +173,9 @@ func TestFDLimiting(t *testing.T) { func TestTokenRedistribution(t *testing.T) { var lk sync.Mutex hangchs := make(map[peer.ID]chan struct{}) - df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.Conn, error) { if tcpPortOver(a, 10) { - return (iconn.Conn)(nil), nil + return (transport.Conn)(nil), nil } lk.Lock() @@ -268,9 +268,9 @@ func TestTokenRedistribution(t *testing.T) { } func TestStressLimiter(t *testing.T) { - df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.Conn, error) { if tcpPortOver(a, 1000) { - return iconn.Conn(nil), nil + return transport.Conn(nil), nil } time.Sleep(time.Millisecond * time.Duration(5+rand.Intn(100))) @@ -322,11 +322,7 @@ func TestStressLimiter(t *testing.T) { } func TestFDLimitUnderflow(t *testing.T) { - dials := 0 - - df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (iconn.Conn, error) { - dials++ - + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.Conn, error) { timeout := make(chan bool, 1) go func() { time.Sleep(time.Second * 5) @@ -382,7 +378,11 @@ func TestFDLimitUnderflow(t *testing.T) { time.Sleep(time.Second * 3) - if l.fdConsuming < 0 { + l.rllock.Lock() + fdConsuming := l.fdConsuming + l.rllock.Unlock() + + if fdConsuming < 0 { t.Fatalf("l.fdConsuming < 0") } } diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 3259264ae8..76cc00b022 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -1,12 +1,15 @@ -package swarm +package swarm_test import ( "testing" "context" + inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" + + . "github.com/libp2p/go-libp2p-swarm" ) func TestPeers(t *testing.T) { @@ -18,13 +21,13 @@ func TestPeers(t *testing.T) { connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // TODO: make a DialAddr func. - s.peers.AddAddr(dst, addr, pstore.PermanentAddrTTL) + s.Peerstore().AddAddr(dst, addr, pstore.PermanentAddrTTL) // t.Logf("connections from %s", s.LocalPeer()) - // for _, c := range s.ConnectionsToPeer(dst) { + // for _, c := range s.ConnsToPeer(dst) { // t.Logf("connection from %s to %s: %v", s.LocalPeer(), dst, c) // } // t.Logf("") - if _, err := s.Dial(ctx, dst); err != nil { + if _, err := s.DialPeer(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } // t.Log(s.swarm.Dump()) @@ -32,10 +35,10 @@ func TestPeers(t *testing.T) { s1GotConn := make(chan struct{}, 0) s2GotConn := make(chan struct{}, 0) - s1.SetConnHandler(func(c *Conn) { + s1.SetConnHandler(func(c inet.Conn) { s1GotConn <- struct{}{} }) - s2.SetConnHandler(func(c *Conn) { + s2.SetConnHandler(func(c inet.Conn) { s2GotConn <- struct{}{} }) @@ -49,7 +52,7 @@ func TestPeers(t *testing.T) { } for _, s := range swarms { - log.Infof("%s swarm routing table: %s", s.local, s.Peers()) + log.Infof("%s swarm routing table: %s", s.LocalPeer(), s.Peers()) } test := func(s *Swarm) { @@ -57,12 +60,10 @@ func TestPeers(t *testing.T) { actual := len(s.Peers()) if actual != expect { t.Errorf("%s has %d peers, not %d: %v", s.LocalPeer(), actual, expect, s.Peers()) - t.Log(s.swarm.Dump()) } - actual = len(s.Connections()) + actual = len(s.Conns()) if actual != expect { - t.Errorf("%s has %d conns, not %d: %v", s.LocalPeer(), actual, expect, s.Connections()) - t.Log(s.swarm.Dump()) + t.Errorf("%s has %d conns, not %d: %v", s.LocalPeer(), actual, expect, s.Conns()) } } diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 8935773296..22b0dbf330 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -1,4 +1,4 @@ -package swarm +package swarm_test import ( "context" @@ -11,6 +11,9 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" ci "github.com/libp2p/go-testutil/ci" ma "github.com/multiformats/go-multiaddr" + + . "github.com/libp2p/go-libp2p-swarm" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" ) func TestSimultOpen(t *testing.T) { @@ -18,16 +21,16 @@ func TestSimultOpen(t *testing.T) { t.Parallel() ctx := context.Background() - swarms := makeSwarms(ctx, t, 2) + swarms := makeSwarms(ctx, t, 2, swarmt.OptDisableReuseport) // connect everyone { var wg sync.WaitGroup connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // copy for other peer - log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.local, dst, addr) - s.peers.AddAddr(dst, addr, pstore.PermanentAddrTTL) - if _, err := s.Dial(ctx, dst); err != nil { + log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.LocalPeer(), dst, addr) + s.Peerstore().AddAddr(dst, addr, pstore.PermanentAddrTTL) + if _, err := s.DialPeer(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } wg.Done() @@ -35,8 +38,8 @@ func TestSimultOpen(t *testing.T) { log.Info("Connecting swarms simultaneously.") wg.Add(2) - go connect(swarms[0], swarms[1].local, swarms[1].ListenAddresses()[0]) - go connect(swarms[1], swarms[0].local, swarms[0].ListenAddresses()[0]) + go connect(swarms[0], swarms[1].LocalPeer(), swarms[1].ListenAddresses()[0]) + go connect(swarms[1], swarms[0].LocalPeer(), swarms[0].ListenAddresses()[0]) wg.Wait() } diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index e7d6b9a7f4..dbf81613f9 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -1,190 +1,150 @@ -// Package swarm implements a connection muxer with a pair of channels -// to synchronize all network communication. package swarm import ( "context" + "errors" "fmt" - "io/ioutil" - "os" "strings" "sync" + "sync/atomic" "time" logging "github.com/ipfs/go-log" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" - addrutil "github.com/libp2p/go-addr-util" - conn "github.com/libp2p/go-libp2p-conn" - ci "github.com/libp2p/go-libp2p-crypto" - ipnet "github.com/libp2p/go-libp2p-interface-pnet" metrics "github.com/libp2p/go-libp2p-metrics" - mconn "github.com/libp2p/go-libp2p-metrics/conn" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" transport "github.com/libp2p/go-libp2p-transport" filter "github.com/libp2p/go-maddr-filter" - ps "github.com/libp2p/go-peerstream" - pst "github.com/libp2p/go-stream-muxer" - tcpt "github.com/libp2p/go-tcp-transport" - ws "github.com/libp2p/go-ws-transport" - ma "github.com/multiformats/go-multiaddr" - psmss "github.com/whyrusleeping/go-smux-multistream" - spdy "github.com/whyrusleeping/go-smux-spdystream" - yamux "github.com/whyrusleeping/go-smux-yamux" mafilter "github.com/whyrusleeping/multiaddr-filter" ) -var log = logging.Logger("swarm2") - -// PSTransport is the default peerstream transport that will be used by -// any libp2p swarms. -var PSTransport pst.Transport - -func init() { - msstpt := psmss.NewBlankTransport() - - ymxtpt := &yamux.Transport{ - AcceptBacklog: 8192, - ConnectionWriteTimeout: time.Second * 10, - KeepAliveInterval: time.Second * 30, - EnableKeepAlive: true, - MaxStreamWindowSize: uint32(1024 * 512), - LogOutput: ioutil.Discard, - } +// DialTimeout is the maximum duration a Dial is allowed to take. +// This includes the time between dialing the raw network connection, +// protocol selection as well the handshake, if applicable. +var DialTimeout = 60 * time.Second - msstpt.AddTransport("/yamux/1.0.0", ymxtpt) - msstpt.AddTransport("/spdy/3.1.0", spdy.Transport) +var log = logging.Logger("swarm2") - // allow overriding of muxer preferences - if prefs := os.Getenv("LIBP2P_MUX_PREFS"); prefs != "" { - msstpt.OrderPreference = strings.Fields(prefs) - } +// ErrSwarmClosed is returned when one attempts to operate on a closed swarm. +var ErrSwarmClosed = errors.New("swarm closed") - PSTransport = msstpt -} +// ErrAddrFiltered is returned when trying to register a connection to a +// filtered address. You shouldn't see this error unless some underlying +// transport is misbehaving. +var ErrAddrFiltered = errors.New("address filtered") // Swarm is a connection muxer, allowing connections to other peers to // be opened and closed, while still using the same Chan for all // communication. The Chan sends/receives Messages, which note the // destination or source Peer. -// -// Uses peerstream.Swarm type Swarm struct { - swarm *ps.Swarm + // Close refcount. This allows us to fully wait for the swarm to be torn + // down before continuing. + refs sync.WaitGroup + local peer.ID peers pstore.Peerstore - connh ConnHandler - dsync *DialSync - backf dialbackoff - dialT time.Duration // mainly for tests + conns struct { + sync.RWMutex + m map[peer.ID][]*Conn + } - dialer *conn.Dialer + listeners struct { + sync.RWMutex + m map[transport.Listener]struct{} + } - notifmu sync.RWMutex - notifs map[inet.Notifiee]ps.Notifiee + notifs struct { + sync.RWMutex + m map[inet.Notifiee]struct{} + } - transports []transport.Transport + transports struct { + sync.RWMutex + m map[int]transport.Transport + } - // filters for addresses that shouldnt be dialed - Filters *filter.Filters + // new connection and stream handlers + connh atomic.Value + streamh atomic.Value - // file descriptor rate limited - fdRateLimit chan struct{} + // dialing helpers + dsync *DialSync + backf DialBackoff + limiter *dialLimiter + + // filters for addresses that shouldnt be dialed (or accepted) + Filters *filter.Filters proc goprocess.Process ctx context.Context bwc metrics.Reporter - - limiter *dialLimiter - - protec ipnet.Protector -} - -func NewSwarm(ctx context.Context, listenAddrs []ma.Multiaddr, local peer.ID, - peers pstore.Peerstore, bwc metrics.Reporter) (*Swarm, error) { - return NewSwarmWithProtector(ctx, listenAddrs, local, peers, nil, PSTransport, bwc) } -// NewSwarm constructs a Swarm, with a Chan. -func NewSwarmWithProtector(ctx context.Context, listenAddrs []ma.Multiaddr, local peer.ID, - peers pstore.Peerstore, protec ipnet.Protector, tpt pst.Transport, bwc metrics.Reporter) (*Swarm, error) { - - listenAddrs, err := filterAddrs(listenAddrs) - if err != nil { - return nil, err - } - - var wrap func(c transport.Conn) transport.Conn - if bwc != nil { - wrap = func(c transport.Conn) transport.Conn { - return mconn.WrapConn(bwc, c) - } - } - +// NewSwarm constructs a Swarm +func NewSwarm(ctx context.Context, local peer.ID, peers pstore.Peerstore, bwc metrics.Reporter) *Swarm { s := &Swarm{ - swarm: ps.NewSwarm(tpt), - local: local, - peers: peers, - ctx: ctx, - dialT: conn.DialTimeout, - notifs: make(map[inet.Notifiee]ps.Notifiee), - transports: []transport.Transport{ - tcpt.NewTCPTransport(), - new(ws.WebsocketTransport), - }, - bwc: bwc, - fdRateLimit: make(chan struct{}, concurrentFdDials), - Filters: filter.NewFilters(), - dialer: conn.NewDialer(local, peers.PrivKey(local), wrap), - protec: protec, + local: local, + peers: peers, + bwc: bwc, + Filters: filter.NewFilters(), } - s.dialer.Protector = protec + + s.conns.m = make(map[peer.ID][]*Conn) + s.listeners.m = make(map[transport.Listener]struct{}) + s.transports.m = make(map[int]transport.Transport) + s.notifs.m = make(map[inet.Notifiee]struct{}) s.dsync = NewDialSync(s.doDial) s.limiter = newDialLimiter(s.dialAddr) - - // configure Swarm s.proc = goprocessctx.WithContextAndTeardown(ctx, s.teardown) - s.SetConnHandler(nil) // make sure to setup our own conn handler. - - err = s.setupInterfaces(listenAddrs) - if err != nil { - return nil, err - } + s.ctx = goprocessctx.OnClosingContext(s.proc) - return s, nil + return s } -func NewBlankSwarm(ctx context.Context, id peer.ID, privkey ci.PrivKey, pstpt pst.Transport) *Swarm { - s := &Swarm{ - swarm: ps.NewSwarm(pstpt), - local: id, - peers: pstore.NewPeerstore(), - ctx: ctx, - dialT: conn.DialTimeout, - notifs: make(map[inet.Notifiee]ps.Notifiee), - fdRateLimit: make(chan struct{}, concurrentFdDials), - Filters: filter.NewFilters(), - dialer: conn.NewDialer(id, privkey, nil), +func (s *Swarm) teardown() error { + // Prevents new connections and/or listeners from being added to the swarm. + + s.listeners.Lock() + listeners := s.listeners.m + s.listeners.m = nil + s.listeners.Unlock() + + s.conns.Lock() + conns := s.conns.m + s.conns.m = nil + s.conns.Unlock() + + // Lots of go routines but we might as well do this in parallel. We want + // to shut down as fast as possible. + + for l := range listeners { + go func(l transport.Listener) { + if err := l.Close(); err != nil { + log.Errorf("error when shutting down listener: %s", err) + } + }(l) } - // configure Swarm - s.limiter = newDialLimiter(s.dialAddr) - s.proc = goprocessctx.WithContextAndTeardown(ctx, s.teardown) - s.SetConnHandler(nil) // make sure to setup our own conn handler. + for _, cs := range conns { + for _, c := range cs { + go func(c *Conn) { + if err := c.Close(); err != nil { + log.Errorf("error when shutting down connection: %s", err) + } + }(c) + } + } - return s -} + // Wait for everything to finish. + s.refs.Wait() -func (s *Swarm) AddTransport(t transport.Transport) { - s.transports = append(s.transports, t) -} - -func (s *Swarm) teardown() error { - return s.swarm.Close() + return nil } // AddAddrFilter adds a multiaddr filter to the set of filters the swarm will @@ -199,31 +159,81 @@ func (s *Swarm) AddAddrFilter(f string) error { return nil } -func filterAddrs(listenAddrs []ma.Multiaddr) ([]ma.Multiaddr, error) { - if len(listenAddrs) > 0 { - filtered := addrutil.FilterUsableAddrs(listenAddrs) - if len(filtered) < 1 { - return nil, fmt.Errorf("swarm cannot use any addr in: %s", listenAddrs) - } - listenAddrs = filtered +// Process returns the Process of the swarm +func (s *Swarm) Process() goprocess.Process { + return s.proc +} + +func (s *Swarm) addConn(tc transport.Conn) (*Conn, error) { + // The underlying transport (or the dialer) *should* filter it's own + // connections but we should double check anyways. + raddr := tc.RemoteMultiaddr() + if s.Filters.AddrBlocked(raddr) { + tc.Close() + return nil, ErrAddrFiltered } - return listenAddrs, nil -} + p := tc.RemotePeer() -// Listen sets up listeners for all of the given addresses -func (s *Swarm) Listen(addrs ...ma.Multiaddr) error { - addrs, err := filterAddrs(addrs) - if err != nil { - return err + // Add the public key. + if pk := tc.RemotePublicKey(); pk != nil { + s.peers.AddPubKey(p, pk) + } + + // Clear any backoffs + s.backf.Clear(p) + + // Finally, add the peer. + s.conns.Lock() + // Check if we're still online + if s.conns.m == nil { + s.conns.Unlock() + tc.Close() + return nil, ErrSwarmClosed + } + + // Wrap and register the connection. + c := &Conn{ + conn: tc, + swarm: s, } + c.streams.m = make(map[*Stream]struct{}) + s.conns.m[p] = append(s.conns.m[p], c) + + // Add two swarm refs: + // * One will be decremented after the close notifications fire in Conn.doClose + // * The other will be decremented when Conn.start exits. + s.refs.Add(2) + + // Take the notification lock before releasing the conns lock to block + // Disconnect notifications until after the Connect notifications done. + c.notifyLk.Lock() + s.conns.Unlock() + + // We have a connection now. Cancel all other in-progress dials. + // This should be fast, no reason to wait till later. + s.dsync.CancelDial(p) + + s.notifyAll(func(f inet.Notifiee) { + f.Connected(s, c) + }) + c.notifyLk.Unlock() - return s.setupInterfaces(addrs) + c.start() + + // TODO: Get rid of this. We use it for identify but that happen much + // earlier (really, inside the transport and, if not then, during the + // notifications). + if h := s.ConnHandler(); h != nil { + go h(c) + } + + return c, nil } -// Process returns the Process of the swarm -func (s *Swarm) Process() goprocess.Process { - return s.proc +// Peerstore returns this swarms internal Peerstore. +func (s *Swarm) Peerstore() pstore.Peerstore { + return s.peers } // Context returns the context of the swarm @@ -236,94 +246,180 @@ func (s *Swarm) Close() error { return s.proc.Close() } -// StreamSwarm returns the underlying peerstream.Swarm -func (s *Swarm) StreamSwarm() *ps.Swarm { - return s.swarm -} +// TODO: We probably don't need the conn handlers. // SetConnHandler assigns the handler for new connections. -// See peerstream. You will rarely use this. See SetStreamHandler -func (s *Swarm) SetConnHandler(handler ConnHandler) { - - // handler is nil if user wants to clear the old handler. - if handler == nil { - s.swarm.SetConnHandler(func(psconn *ps.Conn) { - s.connHandler(psconn) - }) - return - } +// You will rarely use this. See SetStreamHandler +func (s *Swarm) SetConnHandler(handler inet.ConnHandler) { + s.connh.Store(handler) +} - s.swarm.SetConnHandler(func(psconn *ps.Conn) { - // sc is nil if closed in our handler. - if sc := s.connHandler(psconn); sc != nil { - // call the user's handler. in a goroutine for sync safety. - go handler(sc) - } - }) +// ConnHandler gets the handler for new connections. +func (s *Swarm) ConnHandler() inet.ConnHandler { + handler, _ := s.connh.Load().(inet.ConnHandler) + return handler } // SetStreamHandler assigns the handler for new streams. -// See peerstream. func (s *Swarm) SetStreamHandler(handler inet.StreamHandler) { - s.swarm.SetStreamHandler(func(s *ps.Stream) { - handler((*Stream)(s)) - }) -} - -// NewStreamWithPeer creates a new stream on any available connection to p -func (s *Swarm) NewStreamWithPeer(ctx context.Context, p peer.ID) (*Stream, error) { - // if we have no connections, try connecting. - if !s.HaveConnsToPeer(p) { - log.Debug("Swarm: NewStreamWithPeer no connections. Attempting to connect...") - if _, err := s.Dial(ctx, p); err != nil { + s.streamh.Store(handler) +} + +// StreamHandler gets the handler for new streams. +func (s *Swarm) StreamHandler() inet.StreamHandler { + handler, _ := s.streamh.Load().(inet.StreamHandler) + return handler +} + +// NewStream creates a new stream on any available connection to peer, dialing +// if necessary. +func (s *Swarm) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { + log.Debugf("[%s] opening stream to peer [%s]", s.local, p) + + // Algorithm: + // 1. Find the best connection, otherwise, dial. + // 2. Try opening a stream. + // 3. If the underlying connection is, in fact, closed, close the outer + // connection and try again. We do this in case we have a closed + // connection but don't notice it until we actually try to open a + // stream. + // + // Note: We only dial once. + // + // TODO: Try all connections even if we get an error opening a stream on + // a non-closed connection. + dials := 0 + for { + c := s.bestConnToPeer(p) + if c == nil { + if dials >= DialAttempts { + return nil, errors.New("max dial attempts exceeded") + } + dials++ + + var err error + c, err = s.dialPeer(ctx, p) + if err != nil { + return nil, err + } + } + s, err := c.NewStream() + if err != nil { + if c.conn.IsClosed() { + continue + } return nil, err } + return s, nil } - log.Debug("Swarm: NewStreamWithPeer...") - - // TODO: think about passing a context down to NewStreamWithGroup - st, err := s.swarm.NewStreamWithGroup(p) - return (*Stream)(st), err } -// ConnectionsToPeer returns all the live connections to p -func (s *Swarm) ConnectionsToPeer(p peer.ID) []*Conn { - return wrapConns(s.swarm.ConnsWithGroup(p)) -} +// ConnsToPeer returns all the live connections to peer. +func (s *Swarm) ConnsToPeer(p peer.ID) []inet.Conn { + // TODO: Consider sorting the connection list best to worst. Currently, + // it's sorted oldest to newest. + s.conns.RLock() + defer s.conns.RUnlock() + conns := s.conns.m[p] + output := make([]inet.Conn, len(conns)) + for i, c := range conns { + output[i] = c + } + return output +} + +// bestConnToPeer returns the best connection to peer. +func (s *Swarm) bestConnToPeer(p peer.ID) *Conn { + // Selects the best connection we have to the peer. + // TODO: Prefer some transports over others. Currently, we just select + // the newest non-closed connection with the most streams. + s.conns.RLock() + defer s.conns.RUnlock() + + var best *Conn + bestLen := 0 + for _, c := range s.conns.m[p] { + if c.conn.IsClosed() { + // We *will* garbage collect this soon anyways. + continue + } + c.streams.Lock() + cLen := len(c.streams.m) + c.streams.Unlock() -func (s *Swarm) HaveConnsToPeer(p peer.ID) bool { - return len(s.swarm.ConnsWithGroup(p)) > 0 -} + if cLen >= bestLen { + best = c + bestLen = cLen + } -// Connections returns a slice of all connections. -func (s *Swarm) Connections() []*Conn { - return wrapConns(s.swarm.Conns()) + } + return best } -// CloseConnection removes a given peer from swarm + closes the connection -func (s *Swarm) CloseConnection(p peer.ID) error { - conns := s.swarm.ConnsWithGroup(p) // boom. - for _, c := range conns { - c.Close() +// Connectedness returns our "connectedness" state with the given peer. +// +// To check if we have an open connection, use `s.Connectedness(p) == +// inet.Connected`. +func (s *Swarm) Connectedness(p peer.ID) inet.Connectedness { + if s.bestConnToPeer(p) != nil { + return inet.Connected } - return nil + return inet.NotConnected } -// Peers returns a copy of the set of peers swarm is connected to. -func (s *Swarm) Peers() []peer.ID { - conns := s.Connections() +// Conns returns a slice of all connections. +func (s *Swarm) Conns() []inet.Conn { + s.conns.RLock() + defer s.conns.RUnlock() - seen := make(map[peer.ID]struct{}, len(conns)) - peers := make([]peer.ID, 0, len(conns)) - for _, c := range conns { - p := c.RemotePeer() - if _, found := seen[p]; found { - continue + conns := make([]inet.Conn, 0, len(s.conns.m)) + for _, cs := range s.conns.m { + for _, c := range cs { + conns = append(conns, c) + } + } + return conns +} + +// ClosePeer closes all connections to the given peer. +func (s *Swarm) ClosePeer(p peer.ID) error { + conns := s.ConnsToPeer(p) + switch len(conns) { + case 0: + return nil + case 1: + return conns[0].Close() + default: + errCh := make(chan error) + for _, c := range conns { + go func(c inet.Conn) { + errCh <- c.Close() + }(c) } - seen[p] = struct{}{} + var errs []string + for _ = range conns { + err := <-errCh + if err != nil { + errs = append(errs, err.Error()) + } + } + if len(errs) > 0 { + return fmt.Errorf("when disconnecting from peer %s: %s", p, strings.Join(errs, ", ")) + } + return nil + } +} + +// Peers returns a copy of the set of peers swarm is connected to. +func (s *Swarm) Peers() []peer.ID { + s.conns.RLock() + defer s.conns.RUnlock() + peers := make([]peer.ID, 0, len(s.conns.m)) + for p := range s.conns.m { peers = append(peers, p) } + return peers } @@ -332,64 +428,70 @@ func (s *Swarm) LocalPeer() peer.ID { return s.local } -// Backoff returns the dialbackoff object for this swarm. -func (s *Swarm) Backoff() *dialbackoff { +// Backoff returns the DialBackoff object for this swarm. +func (s *Swarm) Backoff() *DialBackoff { return &s.backf } // notifyAll sends a signal to all Notifiees func (s *Swarm) notifyAll(notify func(inet.Notifiee)) { - s.notifmu.RLock() - for f := range s.notifs { - go notify(f) + var wg sync.WaitGroup + + s.notifs.RLock() + wg.Add(len(s.notifs.m)) + for f := range s.notifs.m { + go func(f inet.Notifiee) { + defer wg.Done() + notify(f) + }(f) } - s.notifmu.RUnlock() + + wg.Wait() + s.notifs.RUnlock() } // Notify signs up Notifiee to receive signals when events happen func (s *Swarm) Notify(f inet.Notifiee) { - // wrap with our notifiee, to translate function calls - n := &ps2netNotifee{net: (*Network)(s), not: f} - - s.notifmu.Lock() - s.notifs[f] = n - s.notifmu.Unlock() - - // register for notifications in the peer swarm. - s.swarm.Notify(n) + s.notifs.Lock() + s.notifs.m[f] = struct{}{} + s.notifs.Unlock() } // StopNotify unregisters Notifiee fromr receiving signals func (s *Swarm) StopNotify(f inet.Notifiee) { - s.notifmu.Lock() - n, found := s.notifs[f] - if found { - delete(s.notifs, f) - } - s.notifmu.Unlock() - - if found { - s.swarm.StopNotify(n) + s.notifs.Lock() + delete(s.notifs.m, f) + s.notifs.Unlock() +} + +func (s *Swarm) removeConn(c *Conn) { + p := c.RemotePeer() + + s.conns.Lock() + defer s.conns.Unlock() + cs := s.conns.m[p] + for i, ci := range cs { + if ci == c { + if len(cs) == 1 { + delete(s.conns.m, p) + } else { + // NOTE: We're intentionally preserving order. + // This way, connections to a peer are always + // sorted oldest to newest. + copy(cs[i:], cs[i+1:]) + cs[len(cs)-1] = nil + s.conns.m[p] = cs[:len(cs)-1] + } + return + } } } -type ps2netNotifee struct { - net *Network - not inet.Notifiee +// String returns a string representation of Network. +func (s *Swarm) String() string { + return fmt.Sprintf("", s.LocalPeer()) } -func (n *ps2netNotifee) Connected(c *ps.Conn) { - n.not.Connected(n.net, inet.Conn((*Conn)(c))) -} - -func (n *ps2netNotifee) Disconnected(c *ps.Conn) { - n.not.Disconnected(n.net, inet.Conn((*Conn)(c))) -} - -func (n *ps2netNotifee) OpenedStream(s *ps.Stream) { - n.not.OpenedStream(n.net, (*Stream)(s)) -} - -func (n *ps2netNotifee) ClosedStream(s *ps.Stream) { - n.not.ClosedStream(n.net, (*Stream)(s)) -} +// Swarm is a Network. +var _ inet.Network = (*Swarm)(nil) +var _ transport.Network = (*Swarm)(nil) diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 78a3f5351a..c86d58bb4d 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -2,18 +2,16 @@ package swarm import ( addrutil "github.com/libp2p/go-addr-util" - iconn "github.com/libp2p/go-libp2p-interface-conn" ma "github.com/multiformats/go-multiaddr" ) // ListenAddresses returns a list of addresses at which this swarm listens. func (s *Swarm) ListenAddresses() []ma.Multiaddr { - listeners := s.swarm.Listeners() - addrs := make([]ma.Multiaddr, 0, len(listeners)) - for _, l := range listeners { - if l2, ok := l.NetListener().(iconn.Listener); ok { - addrs = append(addrs, l2.Multiaddr()) - } + s.listeners.RLock() + defer s.listeners.RUnlock() + addrs := make([]ma.Multiaddr, 0, len(s.listeners.m)) + for l := range s.listeners.m { + addrs = append(addrs, l.Multiaddr()) } return addrs } diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 011b706d81..bafcfb6697 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -1,100 +1,14 @@ -package swarm +package swarm_test import ( "context" "testing" - addrutil "github.com/libp2p/go-addr-util" - metrics "github.com/libp2p/go-libp2p-metrics" pstore "github.com/libp2p/go-libp2p-peerstore" testutil "github.com/libp2p/go-testutil" ma "github.com/multiformats/go-multiaddr" ) -func TestFilterAddrs(t *testing.T) { - - m := func(s string) ma.Multiaddr { - maddr, err := ma.NewMultiaddr(s) - if err != nil { - t.Fatal(err) - } - return maddr - } - - bad := []ma.Multiaddr{ - m("/ip4/1.2.3.4/udp/1234"), // unreliable - m("/ip4/1.2.3.4/udp/1234/sctp/1234"), // not in manet - m("/ip4/1.2.3.4/udp/1234/udt"), // udt is broken on arm - m("/ip6/fe80::1/tcp/0"), // link local - m("/ip6/fe80::100/tcp/1234"), // link local - } - - good := []ma.Multiaddr{ - m("/ip4/127.0.0.1/tcp/0"), - m("/ip6/::1/tcp/0"), - m("/ip4/1.2.3.4/udp/1234/utp"), - } - - goodAndBad := append(good, bad...) - - // test filters - - for _, a := range bad { - if addrutil.AddrUsable(a, false) { - t.Errorf("addr %s should be unusable", a) - } - } - - for _, a := range good { - if !addrutil.AddrUsable(a, false) { - t.Errorf("addr %s should be usable", a) - } - } - - subtestAddrsEqual(t, addrutil.FilterUsableAddrs(bad), []ma.Multiaddr{}) - subtestAddrsEqual(t, addrutil.FilterUsableAddrs(good), good) - subtestAddrsEqual(t, addrutil.FilterUsableAddrs(goodAndBad), good) - - // now test it with swarm - - id, err := testutil.RandPeerID() - if err != nil { - t.Fatal(err) - } - - ps := pstore.NewPeerstore() - ctx := context.Background() - - if _, err := NewNetwork(ctx, bad, id, ps, metrics.NewBandwidthCounter()); err == nil { - t.Fatal("should have failed to create swarm") - } - - if _, err := NewNetwork(ctx, goodAndBad, id, ps, metrics.NewBandwidthCounter()); err != nil { - t.Fatal("should have succeeded in creating swarm", err) - } -} - -func subtestAddrsEqual(t *testing.T, a, b []ma.Multiaddr) { - if len(a) != len(b) { - t.Error(t) - } - - in := func(addr ma.Multiaddr, l []ma.Multiaddr) bool { - for _, addr2 := range l { - if addr.Equal(addr2) { - return true - } - } - return false - } - - for _, aa := range a { - if !in(aa, b) { - t.Errorf("%s not in %s", aa, b) - } - } -} - func TestDialBadAddrs(t *testing.T) { m := func(s string) ma.Multiaddr { @@ -110,8 +24,8 @@ func TestDialBadAddrs(t *testing.T) { test := func(a ma.Multiaddr) { p := testutil.RandPeerIDFatal(t) - s.peers.AddAddr(p, a, pstore.PermanentAddrTTL) - if _, err := s.Dial(ctx, p); err == nil { + s.Peerstore().AddAddr(p, a, pstore.PermanentAddrTTL) + if _, err := s.DialPeer(ctx, p); err == nil { t.Errorf("swarm should not dial: %s", p) } } diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index c143062cee..5b2420cd5f 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -1,122 +1,213 @@ package swarm import ( + "errors" "fmt" + "sync" ic "github.com/libp2p/go-libp2p-crypto" - iconn "github.com/libp2p/go-libp2p-interface-conn" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" - ps "github.com/libp2p/go-peerstream" + transport "github.com/libp2p/go-libp2p-transport" + smux "github.com/libp2p/go-stream-muxer" ma "github.com/multiformats/go-multiaddr" ) -// Conn is a simple wrapper around a ps.Conn that also exposes -// some of the methods from the underlying conn.Conn. -// There's **five** "layers" to each connection: -// * 0. the net.Conn - underlying net.Conn (TCP/UDP/UTP/etc) -// * 1. the manet.Conn - provides multiaddr friendly Conn -// * 2. the conn.Conn - provides Peer friendly Conn (inc Secure channel) -// * 3. the peerstream.Conn - provides peerstream / spdysptream happiness -// * 4. the Conn - abstracts everyting out, exposing only key parts of underlying layers -// (I know, this is kinda crazy. it's more historical than a good design. though the -// layers do build up pieces of functionality. and they're all just io.RW :) ) -type Conn ps.Conn - -// ConnHandler is called when new conns are opened from remote peers. -// See peerstream.ConnHandler -type ConnHandler func(*Conn) - -func (c *Conn) StreamConn() *ps.Conn { - return (*ps.Conn)(c) +// TODO: Put this elsewhere. + +// ErrConnClosed is returned when operating on a closed connection. +var ErrConnClosed = errors.New("connection closed") + +// Conn is the connection type used by swarm. In general, you won't use this +// type directly. +type Conn struct { + conn transport.Conn + swarm *Swarm + + closeOnce sync.Once + err error + + notifyLk sync.Mutex + + streams struct { + sync.Mutex + m map[*Stream]struct{} + } +} + +// Close closes this connection. +// +// Note: This method won't wait for the close notifications to finish as that +// would create a deadlock when called from an open notification (because all +// open notifications must finish before we can fire off the close +// notifications). +func (c *Conn) Close() error { + c.closeOnce.Do(c.doClose) + return c.err +} + +func (c *Conn) doClose() { + c.swarm.removeConn(c) + + // Prevent new streams from opening. + c.streams.Lock() + streams := c.streams.m + c.streams.m = nil + c.streams.Unlock() + + c.err = c.conn.Close() + + // This is just for cleaning up state. The connection has already been closed. + // We *could* optimize this but it really isn't worth it. + for s := range streams { + s.Reset() + } + + // do this in a goroutine to avoid deadlocking if we call close in an open notification. + go func() { + // prevents us from issuing close notifications before finishing the open notifications + c.notifyLk.Lock() + defer c.notifyLk.Unlock() + + c.swarm.notifyAll(func(f inet.Notifiee) { + f.Disconnected(c.swarm, c) + }) + c.swarm.refs.Done() // taken in Swarm.addConn + }() } -func (c *Conn) RawConn() iconn.Conn { - // righly panic if these things aren't true. it is an expected - // invariant that these Conns are all of the typewe expect: - // ps.Conn wrapping a conn.Conn - // if we get something else it is programmer error. - return (*ps.Conn)(c).NetConn().(iconn.Conn) +func (c *Conn) removeStream(s *Stream) { + c.streams.Lock() + delete(c.streams.m, s) + c.streams.Unlock() +} + +// listens for new streams. +// +// The caller must take a swarm ref before calling. This function decrements the +// swarm ref count. +func (c *Conn) start() { + go func() { + defer c.swarm.refs.Done() + defer c.Close() + + for { + ts, err := c.conn.AcceptStream() + if err != nil { + return + } + c.swarm.refs.Add(1) + go func() { + s, err := c.addStream(ts) + + // Don't defer this. We don't want to block + // swarm shutdown on the connection handler. + c.swarm.refs.Done() + + // We only get an error here when the swarm is closed or closing. + if err != nil { + return + } + + if h := c.swarm.StreamHandler(); h != nil { + h(s) + } + }() + } + }() } func (c *Conn) String() string { - return fmt.Sprintf("", c.RawConn()) + return fmt.Sprintf( + " %s (%s)>", + c.conn.Transport(), + c.conn.LocalMultiaddr(), + c.conn.LocalPeer().Pretty(), + c.conn.RemoteMultiaddr(), + c.conn.RemotePeer().Pretty(), + ) } // LocalMultiaddr is the Multiaddr on this side func (c *Conn) LocalMultiaddr() ma.Multiaddr { - return c.RawConn().LocalMultiaddr() + return c.conn.LocalMultiaddr() } // LocalPeer is the Peer on our side of the connection func (c *Conn) LocalPeer() peer.ID { - return c.RawConn().LocalPeer() + return c.conn.LocalPeer() } // RemoteMultiaddr is the Multiaddr on the remote side func (c *Conn) RemoteMultiaddr() ma.Multiaddr { - return c.RawConn().RemoteMultiaddr() + return c.conn.RemoteMultiaddr() } // RemotePeer is the Peer on the remote side func (c *Conn) RemotePeer() peer.ID { - return c.RawConn().RemotePeer() + return c.conn.RemotePeer() } // LocalPrivateKey is the public key of the peer on this side func (c *Conn) LocalPrivateKey() ic.PrivKey { - return c.RawConn().LocalPrivateKey() + return c.conn.LocalPrivateKey() } // RemotePublicKey is the public key of the peer on the remote side func (c *Conn) RemotePublicKey() ic.PubKey { - return c.RawConn().RemotePublicKey() -} - -// NewSwarmStream returns a new Stream from this connection -func (c *Conn) NewSwarmStream() (*Stream, error) { - s, err := c.StreamConn().NewStream() - return (*Stream)(s), err + return c.conn.RemotePublicKey() } // NewStream returns a new Stream from this connection func (c *Conn) NewStream() (inet.Stream, error) { - s, err := c.NewSwarmStream() - return inet.Stream(s), err -} - -// Close closes the underlying stream connection -func (c *Conn) Close() error { - return c.StreamConn().Close() + ts, err := c.conn.OpenStream() + if err != nil { + return nil, err + } + return c.addStream(ts) } -func (c *Conn) GetStreams() ([]inet.Stream, error) { - ss := c.StreamConn().Streams() - out := make([]inet.Stream, len(ss)) - - for i, s := range ss { - out[i] = (*Stream)(s) +func (c *Conn) addStream(ts smux.Stream) (*Stream, error) { + c.streams.Lock() + // Are we still online? + if c.streams.m == nil { + c.streams.Unlock() + ts.Reset() + return nil, ErrConnClosed } - return out, nil -} -func wrapConn(psc *ps.Conn) (*Conn, error) { - // grab the underlying connection. - if _, ok := psc.NetConn().(iconn.Conn); !ok { - // this should never happen. if we see it ocurring it means that we added - // a Listener to the ps.Swarm that is NOT one of our net/conn.Listener. - return nil, fmt.Errorf("swarm connHandler: invalid conn (not a conn.Conn): %s", psc) + // Wrap and register the stream. + s := &Stream{ + stream: ts, + conn: c, } - return (*Conn)(psc), nil + c.streams.m[s] = struct{}{} + + // Released once the stream disconnect notifications have finished + // firing (in Swarm.remove). + c.swarm.refs.Add(1) + + // Take the notification lock before releasing the streams lock to block + // StreamClose notifications until after the StreamOpen notifications + // done. + s.notifyLk.Lock() + c.streams.Unlock() + + c.swarm.notifyAll(func(f inet.Notifiee) { + f.OpenedStream(c.swarm, s) + }) + s.notifyLk.Unlock() + + return s, nil } -// wrapConns returns a *Conn for all these ps.Conns -func wrapConns(conns1 []*ps.Conn) []*Conn { - conns2 := make([]*Conn, len(conns1)) - for i, c1 := range conns1 { - if c2, err := wrapConn(c1); err == nil { - conns2[i] = c2 - } +// GetStreams returns the streams associated with this connection. +func (c *Conn) GetStreams() []inet.Stream { + c.streams.Lock() + defer c.streams.Unlock() + streams := make([]inet.Stream, 0, len(c.streams.m)) + for s := range c.streams.m { + streams = append(streams, s) } - return conns2 + return streams } diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index d977d3752b..7125bc3cb2 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -7,10 +7,12 @@ import ( "sync" "time" + logging "github.com/ipfs/go-log" addrutil "github.com/libp2p/go-addr-util" - iconn "github.com/libp2p/go-libp2p-interface-conn" lgbl "github.com/libp2p/go-libp2p-loggables" + inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" + transport "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) @@ -35,18 +37,24 @@ var ( // ErrDialToSelf is returned if we attempt to dial our own peer ErrDialToSelf = errors.New("dial to self attempted") + + // ErrNoTransport is returned when we don't know a transport for the + // given multiaddr. + ErrNoTransport = errors.New("no transport for protocol") ) -// dialAttempts governs how many times a goroutine will try to dial a given peer. +// DialAttempts governs how many times a goroutine will try to dial a given peer. // Note: this is down to one, as we have _too many dials_ atm. To add back in, // add loop back in Dial(.) -const dialAttempts = 1 +const DialAttempts = 1 -// number of concurrent outbound dials over transports that consume file descriptors -const concurrentFdDials = 160 +// ConcurrentFdDials is the number of concurrent outbound dials over transports +// that consume file descriptors +const ConcurrentFdDials = 160 -// number of concurrent outbound dials to make per peer -const defaultPerPeerRateLimit = 8 +// DefaultPerPeerRateLimit is the number of concurrent outbound dials to make +// per peer +const DefaultPerPeerRateLimit = 8 // dialbackoff is a struct used to avoid over-dialing the same, dead peers. // Whenever we totally time out on a peer (all three attempts), we add them @@ -74,7 +82,12 @@ const defaultPerPeerRateLimit = 8 // } // -type dialbackoff struct { +// DialBackoff is a type for tracking peer dial backoffs. +// +// * It's safe to use it's zero value. +// * It's thread-safe. +// * It's *not* safe to move this type after using. +type DialBackoff struct { entries map[peer.ID]*backoffPeer lock sync.RWMutex } @@ -84,7 +97,7 @@ type backoffPeer struct { until time.Time } -func (db *dialbackoff) init() { +func (db *DialBackoff) init() { if db.entries == nil { db.entries = make(map[peer.ID]*backoffPeer) } @@ -92,7 +105,7 @@ func (db *dialbackoff) init() { // Backoff returns whether the client should backoff from dialing // peer p -func (db *dialbackoff) Backoff(p peer.ID) (backoff bool) { +func (db *DialBackoff) Backoff(p peer.ID) (backoff bool) { db.lock.Lock() defer db.lock.Unlock() db.init() @@ -104,13 +117,26 @@ func (db *dialbackoff) Backoff(p peer.ID) (backoff bool) { return false } -const baseBackoffTime = time.Second * 5 -const maxBackoffTime = time.Minute * 5 +// BackoffBase is the base amount of time to backoff (default: 5s). +var BackoffBase = time.Second * 5 + +// BackoffCoef is the backoff coefficient (default: 1s). +var BackoffCoef = time.Second + +// BackoffMax is the maximum backoff time (default: 5m). +var BackoffMax = time.Minute * 5 // AddBackoff lets other nodes know that we've entered backoff with // peer p, so dialers should not wait unnecessarily. We still will // attempt to dial with one goroutine, in case we get through. -func (db *dialbackoff) AddBackoff(p peer.ID) { +// +// Backoff is not exponential, it's quadratic and computed according to the +// following formula: +// +// BackoffBase + BakoffCoef * PriorBackoffs^2 +// +// Where PriorBackoffs is the number of previous backoffs. +func (db *DialBackoff) AddBackoff(p peer.ID) { db.lock.Lock() defer db.lock.Unlock() db.init() @@ -118,61 +144,54 @@ func (db *dialbackoff) AddBackoff(p peer.ID) { if !ok { db.entries[p] = &backoffPeer{ tries: 1, - until: time.Now().Add(baseBackoffTime), + until: time.Now().Add(BackoffBase), } return } - expTimeAdd := time.Second * time.Duration(bp.tries*bp.tries) - if expTimeAdd > maxBackoffTime { - expTimeAdd = maxBackoffTime + backoffTime := BackoffBase + BackoffCoef*time.Duration(bp.tries*bp.tries) + if backoffTime > BackoffMax { + backoffTime = BackoffMax } - bp.until = time.Now().Add(baseBackoffTime + expTimeAdd) + bp.until = time.Now().Add(backoffTime) bp.tries++ } // Clear removes a backoff record. Clients should call this after a // successful Dial. -func (db *dialbackoff) Clear(p peer.ID) { +func (db *DialBackoff) Clear(p peer.ID) { db.lock.Lock() defer db.lock.Unlock() db.init() delete(db.entries, p) } -// Dial connects to a peer. +// DialPeer connects to a peer. // // The idea is that the client of Swarm does not need to know what network // the connection will happen over. Swarm can use whichever it choses. // This allows us to use various transport protocols, do NAT traversal/relay, -// etc. to achive connection. -func (s *Swarm) Dial(ctx context.Context, p peer.ID) (*Conn, error) { +// etc. to achieve connection. +func (s *Swarm) DialPeer(ctx context.Context, p peer.ID) (inet.Conn, error) { + return s.dialPeer(ctx, p) +} + +// internal dial method that returns an unwrapped conn +// +// It is gated by the swarm's dial synchronization systems: dialsync and +// dialbackoff. +func (s *Swarm) dialPeer(ctx context.Context, p peer.ID) (*Conn, error) { + log.Debugf("[%s] swarm dialing peer [%s]", s.local, p) var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) if p == s.local { log.Event(ctx, "swarmDialSelf", logdial) return nil, ErrDialToSelf } - return s.gatedDialAttempt(ctx, p) -} - -func (s *Swarm) bestConnectionToPeer(p peer.ID) *Conn { - cs := s.ConnectionsToPeer(p) - for _, conn := range cs { - if conn != nil { // dump out the first one we find. (TODO pick better) - return conn - } - } - return nil -} - -// gatedDialAttempt is an attempt to dial a node. It is gated by the swarm's -// dial synchronization systems: dialsync and dialbackoff. -func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) { defer log.EventBegin(ctx, "swarmDialAttemptSync", p).Done() // check if we already have an open connection first - conn := s.bestConnectionToPeer(p) + conn := s.bestConnToPeer(p) if conn != nil { return conn, nil } @@ -183,13 +202,26 @@ func (s *Swarm) gatedDialAttempt(ctx context.Context, p peer.ID) (*Conn, error) return nil, ErrDialBackoff } - return s.dsync.DialLock(ctx, p) + conn, err := s.dsync.DialLock(ctx, p) + if err != nil { + return nil, err + } + log.Debugf("network for %s finished dialing %s", s.local, p) + return conn, err } // doDial is an ugly shim method to retain all the logging and backoff logic // of the old dialsync code func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { - ctx, cancel := context.WithTimeout(ctx, s.dialT) + // Short circuit. + // By the time we take the dial lock, we may already *have* a connection + // to the peer. + c := s.bestConnToPeer(p) + if c != nil { + return c, nil + } + + ctx, cancel := context.WithTimeout(ctx, DialTimeout) defer cancel() logdial := lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) @@ -200,17 +232,31 @@ func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { conn, err := s.dial(ctx, p) if err != nil { + conn = s.bestConnToPeer(p) + if conn != nil { + // Hm? What error? + // Could have canceled the dial because we received a + // connection or some other random reason. + // Just ignore the error and return the connection. + log.Debugf("ignoring dial error because we have a connection: %s", err) + return conn, nil + } if err != context.Canceled { log.Event(ctx, "swarmDialBackoffAdd", logdial) s.backf.AddBackoff(p) // let others know to backoff } - // ok, we failed. try again. (if loop is done, our error is output) + // ok, we failed. return nil, fmt.Errorf("dial attempt failed: %s", err) } return conn, nil } +func (s *Swarm) canDial(addr ma.Multiaddr) bool { + t := s.TransportForDialing(addr) + return t != nil && t.CanDial(addr) +} + // dial is the actual swarm's dial logic, gated by Dial. func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) @@ -249,8 +295,10 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { */ paddrs := s.peers.Addrs(p) goodAddrs := addrutil.FilterAddrs(paddrs, - addrutil.AddrUsableFunc, subtractFilter, + s.canDial, + // TODO: Consider allowing this? + addrutil.AddrOverNonLocalIP, addrutil.FilterNeg(s.Filters.AddrBlocked), ) remoteAddrChan := make(chan ma.Multiaddr, len(goodAddrs)) @@ -266,11 +314,11 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { logdial["error"] = err.Error() return nil, err } - logdial["netconn"] = lgbl.NetConn(connC) - - // ok try to setup the new connection. - defer log.EventBegin(ctx, "swarmDialDoSetup", logdial, lgbl.NetConn(connC)).Done() - swarmC, err := dialConnSetup(ctx, s, connC) + logdial["conn"] = logging.Metadata{ + "localAddr": connC.LocalMultiaddr(), + "remoteAddr": connC.RemoteMultiaddr(), + } + swarmC, err := s.addConn(connC) if err != nil { logdial["error"] = err.Error() connC.Close() // close the connection. didn't work out :( @@ -281,7 +329,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { return swarmC, nil } -func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (iconn.Conn, error) { +func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (transport.Conn, error) { log.Debugf("%s swarm dialing %s", s.local, p) ctx, cancel := context.WithCancel(ctx) @@ -343,64 +391,31 @@ func (s *Swarm) limitedDial(ctx context.Context, p peer.ID, a ma.Multiaddr, resp }) } -func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (iconn.Conn, error) { +func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (transport.Conn, error) { + // Just to double check. Costs nothing. + if s.local == p { + return nil, ErrDialToSelf + } log.Debugf("%s swarm dialing %s %s", s.local, p, addr) - connC, err := s.dialer.Dial(ctx, addr, p) - if err != nil { - return nil, fmt.Errorf("%s --> %s dial attempt failed: %s", s.local, p, err) + transport := s.TransportForDialing(addr) + if transport == nil { + return nil, ErrNoTransport } - // if the connection is not to whom we thought it would be... - remotep := connC.RemotePeer() - if remotep != p { - connC.Close() - _, err := connC.Read(nil) // should return any potential errors (ex: from secio) - return nil, fmt.Errorf("misdial to %s through %s (got %s): %s", p, addr, remotep, err) + connC, err := transport.Dial(ctx, addr, p) + if err != nil { + return nil, fmt.Errorf("%s --> %s dial attempt failed: %s", s.local, p, err) } - // if the connection is to ourselves... - // this can happen TONS when Loopback addrs are advertized. - // (this should be caught by two checks above, but let's just make sure.) - if remotep == s.local { + // Trust the transport? Yeah... right. + if connC.RemotePeer() != p { connC.Close() - return nil, fmt.Errorf("misdial to %s through %s (got self)", p, addr) + err = fmt.Errorf("BUG in transport %T: tried to dial %s, dialed %s", p, connC.RemotePeer(), transport) + log.Error(err) + return nil, err } // success! we got one! return connC, nil } - -var ConnSetupTimeout = time.Minute * 5 - -// dialConnSetup is the setup logic for a connection from the dial side. -func dialConnSetup(ctx context.Context, s *Swarm, connC iconn.Conn) (*Conn, error) { - deadline, ok := ctx.Deadline() - if !ok { - deadline = time.Now().Add(ConnSetupTimeout) - } - - if err := connC.SetDeadline(deadline); err != nil { - return nil, err - } - - // Add conn to ps swarm. Setup will be done by the connection handler. - psC, err := s.swarm.AddConn(connC) - if err != nil { - // connC is closed by caller if we fail. - return nil, fmt.Errorf("failed to add conn to ps.Swarm: %s", err) - } - - swarmC, err := wrapConn(psC) - if err != nil { - psC.Close() // we need to make sure psC is Closed. - return nil, err - } - - if err := connC.SetDeadline(time.Time{}); err != nil { - log.Error("failed to reset connection deadline after setup: ", err) - return nil, err - } - - return swarmC, err -} diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 148d9a8881..df0a97f248 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -1,47 +1,15 @@ package swarm import ( - "context" "fmt" - conn "github.com/libp2p/go-libp2p-conn" - iconn "github.com/libp2p/go-libp2p-interface-conn" - lgbl "github.com/libp2p/go-libp2p-loggables" - mconn "github.com/libp2p/go-libp2p-metrics/conn" inet "github.com/libp2p/go-libp2p-net" - transport "github.com/libp2p/go-libp2p-transport" - ps "github.com/libp2p/go-peerstream" ma "github.com/multiformats/go-multiaddr" ) -func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { - tpt := s.transportForAddr(a) - if tpt == nil { - return fmt.Errorf("no transport for address: %s", a) - } - - d, err := tpt.Dialer(a, transport.ReusePorts) - if err != nil { - return err - } - - s.dialer.AddDialer(d) - - list, err := tpt.Listen(a) - if err != nil { - return err - } - - err = s.addListener(list) - if err != nil { - return err - } - - return nil -} - -// Open listeners and reuse-dialers for the given addresses -func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { +// Listen sets up listeners for all of the given addresses. +// It returns as long as we successfully listen on at least *one* address. +func (s *Swarm) Listen(addrs ...ma.Multiaddr) error { errs := make([]error, len(addrs)) var succeeded int for i, a := range addrs { @@ -65,107 +33,62 @@ func (s *Swarm) setupInterfaces(addrs []ma.Multiaddr) error { return nil } -func (s *Swarm) transportForAddr(a ma.Multiaddr) transport.Transport { - for _, t := range s.transports { - if t.Matches(a) { - return t - } - } - - return nil -} - -func (s *Swarm) addListener(tptlist transport.Listener) error { - - sk := s.peers.PrivKey(s.local) - if sk == nil { - // may be fine for sk to be nil, just log a warning. - log.Warning("Listener not given PrivateKey, so WILL NOT SECURE conns.") +// AddListenAddr tells the swarm to listen on a single address. Unlike Listen, +// this method does not attempt to filter out bad addresses. +func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { + tpt := s.TransportForListening(a) + if tpt == nil { + return ErrNoTransport } - list, err := conn.WrapTransportListenerWithProtector(s.Context(), tptlist, s.local, sk, s.protec) + list, err := tpt.Listen(a) if err != nil { return err } - list.SetAddrFilters(s.Filters) - - if cw, ok := list.(conn.ListenerConnWrapper); ok && s.bwc != nil { - cw.SetConnWrapper(func(c transport.Conn) transport.Conn { - return mconn.WrapConn(s.bwc, c) - }) - } - - return s.addConnListener(list) -} - -func (s *Swarm) addConnListener(list iconn.Listener) error { - // AddListener to the peerstream Listener. this will begin accepting connections - // and streams! - sl, err := s.swarm.AddListener(list) - if err != nil { - return err + s.listeners.Lock() + if s.listeners.m == nil { + s.listeners.Unlock() + list.Close() + return ErrSwarmClosed } - log.Debugf("Swarm Listeners at %s", s.ListenAddresses()) + s.refs.Add(1) + s.listeners.m[list] = struct{}{} + s.listeners.Unlock() maddr := list.Multiaddr() // signal to our notifiees on successful conn. s.notifyAll(func(n inet.Notifiee) { - n.Listen((*Network)(s), maddr) + n.Listen(s, maddr) }) - // go consume peerstream's listen accept errors. note, these ARE errors. - // they may be killing the listener, and if we get _any_ we should be - // fixing this in our conn.Listener (to ignore them or handle them - // differently.) - go func(ctx context.Context, sl *ps.Listener) { - - // signal to our notifiees closing - defer s.notifyAll(func(n inet.Notifiee) { - n.ListenClose((*Network)(s), maddr) - }) - + go func() { + defer func() { + list.Close() + s.listeners.Lock() + delete(s.listeners.m, list) + s.listeners.Unlock() + s.refs.Done() + }() for { - select { - case err, more := <-sl.AcceptErrors(): - if !more { - return - } + c, err := list.Accept() + if err != nil { log.Warningf("swarm listener accept error: %s", err) - case <-ctx.Done(): return } + log.Debugf("swarm listener accepted connection: %s", c) + s.refs.Add(1) + go func() { + defer s.refs.Done() + _, err := s.addConn(c) + if err != nil { + // Probably just means that the swarm has been closed. + log.Warningf("add conn failed: ", err) + return + } + }() } - }(s.Context(), sl) - + }() return nil } - -// connHandler is called by the StreamSwarm whenever a new connection is added -// here we configure it slightly. Note that this is sequential, so if anything -// will take a while do it in a goroutine. -// See https://godoc.org/github.com/libp2p/go-peerstream for more information -func (s *Swarm) connHandler(c *ps.Conn) *Conn { - sc, err := wrapConn(c) - if err != nil { - log.Event(s.Context(), "newConnHandlerDisconnect", lgbl.NetConn(c.NetConn()), lgbl.Error(err)) - c.Close() // boom. close it. - return nil - } - - // Add the public key. - if pk := sc.RemotePublicKey(); pk != nil { - s.peers.AddPubKey(sc.RemotePeer(), pk) - } - - // Add the group - c.AddGroup(sc.RemotePeer()) - - // clear backoff on successful connection. - logdial := lgbl.Dial("swarm", sc.LocalPeer(), sc.RemotePeer(), nil, nil) - log.Event(s.Context(), "swarmDialBackoffClear", logdial) - s.backf.Clear(sc.RemotePeer()) - - return sc -} diff --git a/p2p/net/swarm/swarm_net.go b/p2p/net/swarm/swarm_net.go deleted file mode 100644 index eb11ecb7da..0000000000 --- a/p2p/net/swarm/swarm_net.go +++ /dev/null @@ -1,177 +0,0 @@ -package swarm - -import ( - "context" - "fmt" - - "github.com/jbenet/goprocess" - ipnet "github.com/libp2p/go-libp2p-interface-pnet" - metrics "github.com/libp2p/go-libp2p-metrics" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" -) - -// Network implements the inet.Network interface. -// It is simply a swarm, with a few different functions -// to implement inet.Network. -type Network Swarm - -func NewNetwork(ctx context.Context, listen []ma.Multiaddr, local peer.ID, - peers pstore.Peerstore, bwc metrics.Reporter) (*Network, error) { - - return NewNetworkWithProtector(ctx, listen, local, peers, nil, bwc) -} - -// NewNetwork constructs a new network and starts listening on given addresses. -func NewNetworkWithProtector(ctx context.Context, listen []ma.Multiaddr, local peer.ID, - peers pstore.Peerstore, protec ipnet.Protector, bwc metrics.Reporter) (*Network, error) { - - s, err := NewSwarmWithProtector(ctx, listen, local, peers, protec, PSTransport, bwc) - if err != nil { - return nil, err - } - - return (*Network)(s), nil -} - -// DialPeer attempts to establish a connection to a given peer. -// Respects the context. -func (n *Network) DialPeer(ctx context.Context, p peer.ID) (inet.Conn, error) { - log.Debugf("[%s] network dialing peer [%s]", n.local, p) - sc, err := n.Swarm().Dial(ctx, p) - if err != nil { - return nil, err - } - - log.Debugf("network for %s finished dialing %s", n.local, p) - return inet.Conn(sc), nil -} - -// Process returns the network's Process -func (n *Network) Process() goprocess.Process { - return n.proc -} - -// Swarm returns the network's peerstream.Swarm -func (n *Network) Swarm() *Swarm { - return (*Swarm)(n) -} - -// LocalPeer the network's LocalPeer -func (n *Network) LocalPeer() peer.ID { - return n.Swarm().LocalPeer() -} - -// Peers returns the known peer IDs from the Peerstore -func (n *Network) Peers() []peer.ID { - return n.Swarm().Peers() -} - -// Peerstore returns the Peerstore, which tracks known peers -func (n *Network) Peerstore() pstore.Peerstore { - return n.Swarm().peers -} - -// Conns returns the connected peers -func (n *Network) Conns() []inet.Conn { - conns1 := n.Swarm().Connections() - out := make([]inet.Conn, len(conns1)) - for i, c := range conns1 { - out[i] = inet.Conn(c) - } - return out -} - -// ConnsToPeer returns the connections in this Netowrk for given peer. -func (n *Network) ConnsToPeer(p peer.ID) []inet.Conn { - conns1 := n.Swarm().ConnectionsToPeer(p) - out := make([]inet.Conn, len(conns1)) - for i, c := range conns1 { - out[i] = inet.Conn(c) - } - return out -} - -// ClosePeer connection to peer -func (n *Network) ClosePeer(p peer.ID) error { - return n.Swarm().CloseConnection(p) -} - -// close is the real teardown function -func (n *Network) close() error { - return n.Swarm().Close() -} - -// Close calls the ContextCloser func -func (n *Network) Close() error { - return n.Swarm().proc.Close() -} - -// Listen tells the network to start listening on given multiaddrs. -func (n *Network) Listen(addrs ...ma.Multiaddr) error { - return n.Swarm().Listen(addrs...) -} - -// ListenAddresses returns a list of addresses at which this network listens. -func (n *Network) ListenAddresses() []ma.Multiaddr { - return n.Swarm().ListenAddresses() -} - -// InterfaceListenAddresses returns a list of addresses at which this network -// listens. It expands "any interface" addresses (/ip4/0.0.0.0, /ip6/::) to -// use the known local interfaces. -func (n *Network) InterfaceListenAddresses() ([]ma.Multiaddr, error) { - return n.Swarm().InterfaceListenAddresses() -} - -// Connectedness returns a state signaling connection capabilities -// For now only returns Connected || NotConnected. Expand into more later. -func (n *Network) Connectedness(p peer.ID) inet.Connectedness { - if n.Swarm().HaveConnsToPeer(p) { - return inet.Connected - } - return inet.NotConnected -} - -// NewStream returns a new stream to given peer p. -// If there is no connection to p, attempts to create one. -func (n *Network) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { - log.Debugf("[%s] network opening stream to peer [%s]", n.local, p) - s, err := n.Swarm().NewStreamWithPeer(ctx, p) - if err != nil { - return nil, err - } - - return inet.Stream(s), nil -} - -// SetStreamHandler sets the protocol handler on the Network's Muxer. -// This operation is threadsafe. -func (n *Network) SetStreamHandler(h inet.StreamHandler) { - n.Swarm().SetStreamHandler(h) -} - -// SetConnHandler sets the conn handler on the Network. -// This operation is threadsafe. -func (n *Network) SetConnHandler(h inet.ConnHandler) { - n.Swarm().SetConnHandler(func(c *Conn) { - h(inet.Conn(c)) - }) -} - -// String returns a string representation of Network. -func (n *Network) String() string { - return fmt.Sprintf("", n.LocalPeer()) -} - -// Notify signs up Notifiee to receive signals when events happen -func (n *Network) Notify(f inet.Notifiee) { - n.Swarm().Notify(f) -} - -// StopNotify unregisters Notifiee fromr receiving signals -func (n *Network) StopNotify(f inet.Notifiee) { - n.Swarm().StopNotify(f) -} diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 95bc268307..425a1707d1 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -1,4 +1,4 @@ -package swarm +package swarm_test import ( "context" @@ -7,29 +7,9 @@ import ( "time" inet "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" - tu "github.com/libp2p/go-testutil" - ma "github.com/multiformats/go-multiaddr" -) - -func GenSwarmNetwork(t *testing.T, ctx context.Context) *Network { - p := tu.RandPeerNetParamsOrFatal(t) - ps := pstore.NewPeerstore() - ps.AddPubKey(p.ID, p.PubKey) - ps.AddPrivKey(p.ID, p.PrivKey) - n, err := NewNetwork(ctx, []ma.Multiaddr{p.Addr}, p.ID, ps, nil) - if err != nil { - t.Fatal(err) - } - ps.AddAddrs(p.ID, n.ListenAddresses(), pstore.PermanentAddrTTL) - return n -} -func DivulgeAddresses(a, b inet.Network) { - id := a.LocalPeer() - addrs := a.Peerstore().Addrs(id) - b.Peerstore().AddAddrs(id, addrs, pstore.PermanentAddrTTL) -} + . "github.com/libp2p/go-libp2p-swarm/testing" +) // TestConnectednessCorrect starts a few networks, connects a few // and tests Connectedness value is correct. @@ -39,7 +19,7 @@ func TestConnectednessCorrect(t *testing.T) { nets := make([]inet.Network, 4) for i := 0; i < 4; i++ { - nets[i] = GenSwarmNetwork(t, ctx) + nets[i] = GenSwarm(t, ctx) } // connect 0-1, 0-2, 0-3, 1-2, 2-3 @@ -130,7 +110,7 @@ func TestNetworkOpenStream(t *testing.T) { nets := make([]inet.Network, 4) for i := 0; i < 4; i++ { - nets[i] = GenSwarmNetwork(t, ctx) + nets[i] = GenSwarm(t, ctx) } dial := func(a, b inet.Network) { @@ -165,7 +145,7 @@ func TestNetworkOpenStream(t *testing.T) { t.Fatal(err) } - streams, err := nets[0].ConnsToPeer(nets[1].LocalPeer())[0].GetStreams() + streams := nets[0].ConnsToPeer(nets[1].LocalPeer())[0].GetStreams() if err != nil { t.Fatal(err) } diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index 8a574aa070..d05766d6c7 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -1,4 +1,4 @@ -package swarm +package swarm_test import ( "testing" @@ -8,13 +8,9 @@ import ( inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" -) -func streamsSame(a, b inet.Stream) bool { - sa := a.(*Stream) - sb := b.(*Stream) - return sa.Stream() == sb.Stream() -} + . "github.com/libp2p/go-libp2p-swarm" +) func TestNotifications(t *testing.T) { const swarmSize = 5 @@ -52,7 +48,7 @@ func TestNotifications(t *testing.T) { } // this feels a little sketchy, but its probably okay - for len(s.ConnectionsToPeer(s2.LocalPeer())) != len(notifs[s2.LocalPeer()]) { + for len(s.ConnsToPeer(s2.LocalPeer())) != len(notifs[s2.LocalPeer()]) { select { case c := <-n.connected: nfp := notifs[c.RemotePeer()] @@ -64,7 +60,7 @@ func TestNotifications(t *testing.T) { } for p, cons := range notifs { - expect := s.ConnectionsToPeer(p) + expect := s.ConnsToPeer(p) if len(expect) != len(cons) { t.Fatal("got different number of connections") } @@ -87,10 +83,10 @@ func TestNotifications(t *testing.T) { complement := func(c inet.Conn) (*Swarm, *netNotifiee, *Conn) { for i, s := range swarms { - for _, c2 := range s.Connections() { + for _, c2 := range s.Conns() { if c.LocalMultiaddr().Equal(c2.RemoteMultiaddr()) && c2.LocalMultiaddr().Equal(c.RemoteMultiaddr()) { - return s, notifiees[i], c2 + return s, notifiees[i], c2.(*Conn) } } } @@ -106,7 +102,7 @@ func TestNotifications(t *testing.T) { case <-time.After(timeout): t.Fatal("timeout") } - if !streamsSame(s, s2) { + if s != s2 { t.Fatal("got incorrect stream", s.Conn(), s2.Conn()) } @@ -116,7 +112,7 @@ func TestNotifications(t *testing.T) { case <-time.After(timeout): t.Fatal("timeout") } - if !streamsSame(s, s2) { + if s != s2 { t.Fatal("got incorrect stream", s.Conn(), s2.Conn()) } } @@ -131,7 +127,7 @@ func TestNotifications(t *testing.T) { // open a streams in each conn for i, s := range swarms { - for _, c := range s.Connections() { + for _, c := range s.Conns() { _, n2, _ := complement(c) st1, err := c.NewStream() @@ -150,7 +146,7 @@ func TestNotifications(t *testing.T) { // close conns for i, s := range swarms { n := notifiees[i] - for _, c := range s.Connections() { + for _, c := range s.Conns() { _, n2, c2 := complement(c) c.Close() c2.Close() diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 7e83dde1d8..30b44bd28e 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -1,69 +1,167 @@ package swarm import ( + "fmt" + "io" + "sync" + "sync/atomic" "time" inet "github.com/libp2p/go-libp2p-net" protocol "github.com/libp2p/go-libp2p-protocol" - ps "github.com/libp2p/go-peerstream" + smux "github.com/libp2p/go-stream-muxer" ) -// Stream is a wrapper around a ps.Stream that exposes a way to get -// our Conn and Swarm (instead of just the ps.Conn and ps.Swarm) -type Stream ps.Stream +type streamState int -// Stream returns the underlying peerstream.Stream -func (s *Stream) Stream() *ps.Stream { - return (*ps.Stream)(s) +const ( + streamOpen streamState = iota + streamCloseRead + streamCloseWrite + streamCloseBoth + streamReset +) + +// Stream is the stream type used by swarm. In general, you won't use this type +// directly. +type Stream struct { + stream smux.Stream + conn *Conn + + state struct { + sync.Mutex + v streamState + } + + notifyLk sync.Mutex + + protocol atomic.Value } -// Conn returns the Conn associated with this Stream, as an inet.Conn -func (s *Stream) Conn() inet.Conn { - return s.SwarmConn() +func (s *Stream) String() string { + return fmt.Sprintf( + " %s (%s)>", + s.conn.conn.Transport(), + s.conn.LocalMultiaddr(), + s.conn.LocalPeer(), + s.conn.RemoteMultiaddr(), + s.conn.RemotePeer(), + ) } -// SwarmConn returns the Conn associated with this Stream, as a *Conn -func (s *Stream) SwarmConn() *Conn { - return (*Conn)(s.Stream().Conn()) +// Conn returns the Conn associated with this stream, as an inet.Conn +func (s *Stream) Conn() inet.Conn { + return s.conn } // Read reads bytes from a stream. -func (s *Stream) Read(p []byte) (n int, err error) { - return s.Stream().Read(p) +func (s *Stream) Read(p []byte) (int, error) { + n, err := s.stream.Read(p) + // TODO: push this down to a lower level for better accuracy. + if s.conn.swarm.bwc != nil { + s.conn.swarm.bwc.LogRecvMessage(int64(n)) + s.conn.swarm.bwc.LogRecvMessageStream(int64(n), s.Protocol(), s.Conn().RemotePeer()) + } + // If we observe an EOF, this stream is now closed for reading. + // If we're already closed for writing, this stream is now fully closed. + if err == io.EOF { + s.state.Lock() + switch s.state.v { + case streamCloseWrite: + s.state.v = streamCloseBoth + s.remove() + case streamOpen: + s.state.v = streamCloseRead + } + s.state.Unlock() + } + return n, err } // Write writes bytes to a stream, flushing for each call. -func (s *Stream) Write(p []byte) (n int, err error) { - return s.Stream().Write(p) +func (s *Stream) Write(p []byte) (int, error) { + n, err := s.stream.Write(p) + // TODO: push this down to a lower level for better accuracy. + if s.conn.swarm.bwc != nil { + s.conn.swarm.bwc.LogSentMessage(int64(n)) + s.conn.swarm.bwc.LogSentMessageStream(int64(n), s.Protocol(), s.Conn().RemotePeer()) + } + return n, err } // Close closes the stream, indicating this side is finished // with the stream. func (s *Stream) Close() error { - return s.Stream().Close() + err := s.stream.Close() + + s.state.Lock() + switch s.state.v { + case streamCloseRead: + s.state.v = streamCloseBoth + s.remove() + case streamOpen: + s.state.v = streamCloseWrite + } + s.state.Unlock() + return err } // Reset resets the stream, closing both ends. func (s *Stream) Reset() error { - return s.Stream().Reset() + err := s.stream.Reset() + s.state.Lock() + switch s.state.v { + case streamOpen, streamCloseRead, streamCloseWrite: + s.state.v = streamReset + s.remove() + } + s.state.Unlock() + return err +} + +func (s *Stream) remove() { + s.conn.removeStream(s) + + // We *must* do this in a goroutine. This can be called during a + // an open notification and will block until that notification is done. + go func() { + s.notifyLk.Lock() + defer s.notifyLk.Unlock() + + s.conn.swarm.notifyAll(func(f inet.Notifiee) { + f.ClosedStream(s.conn.swarm, s) + }) + s.conn.swarm.refs.Done() + }() } +// Protocol returns the protocol negotiated on this stream (if set). func (s *Stream) Protocol() protocol.ID { - return (*ps.Stream)(s).Protocol() + // Ignore type error. It means that the protocol is unset. + p, _ := s.protocol.Load().(protocol.ID) + return p } +// SetProtocol sets the protocol for this stream. +// +// This doesn't actually *do* anything other than record the fact that we're +// speaking the given protocol over this stream. It's still up to the user to +// negotiate the protocol. This is usually done by the Host. func (s *Stream) SetProtocol(p protocol.ID) { - (*ps.Stream)(s).SetProtocol(p) + s.protocol.Store(p) } +// SetDeadline sets the read and write deadlines for this stream. func (s *Stream) SetDeadline(t time.Time) error { - return s.Stream().SetDeadline(t) + return s.stream.SetDeadline(t) } +// SetReadDeadline sets the read deadline for this stream. func (s *Stream) SetReadDeadline(t time.Time) error { - return s.Stream().SetReadDeadline(t) + return s.stream.SetReadDeadline(t) } +// SetWriteDeadline sets the write deadline for this stream. func (s *Stream) SetWriteDeadline(t time.Time) error { - return s.Stream().SetWriteDeadline(t) + return s.stream.SetWriteDeadline(t) } diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index e14de3faad..4911fc96a1 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -1,4 +1,4 @@ -package swarm +package swarm_test import ( "bytes" @@ -10,14 +10,18 @@ import ( "testing" "time" - metrics "github.com/libp2p/go-libp2p-metrics" + logging "github.com/ipfs/go-log" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" - testutil "github.com/libp2p/go-testutil" ma "github.com/multiformats/go-multiaddr" + + . "github.com/libp2p/go-libp2p-swarm" + . "github.com/libp2p/go-libp2p-swarm/testing" ) +var log = logging.Logger("swarm_test") + func EchoStreamHandler(stream inet.Stream) { go func() { defer stream.Close() @@ -50,38 +54,17 @@ func EchoStreamHandler(stream inet.Stream) { } func makeDialOnlySwarm(ctx context.Context, t *testing.T) *Swarm { - id := testutil.RandIdentityOrFatal(t) - - peerstore := pstore.NewPeerstore() - peerstore.AddPubKey(id.ID(), id.PublicKey()) - peerstore.AddPrivKey(id.ID(), id.PrivateKey()) - - swarm, err := NewSwarm(ctx, nil, id.ID(), peerstore, metrics.NewBandwidthCounter()) - if err != nil { - t.Fatal(err) - } - + swarm := GenSwarm(t, ctx, OptDialOnly) swarm.SetStreamHandler(EchoStreamHandler) return swarm } -func makeSwarms(ctx context.Context, t *testing.T, num int) []*Swarm { +func makeSwarms(ctx context.Context, t *testing.T, num int, opts ...Option) []*Swarm { swarms := make([]*Swarm, 0, num) for i := 0; i < num; i++ { - localnp := testutil.RandPeerNetParamsOrFatal(t) - - peerstore := pstore.NewPeerstore() - peerstore.AddPubKey(localnp.ID, localnp.PubKey) - peerstore.AddPrivKey(localnp.ID, localnp.PrivKey) - - addrs := []ma.Multiaddr{localnp.Addr} - swarm, err := NewSwarm(ctx, addrs, localnp.ID, peerstore, metrics.NewBandwidthCounter()) - if err != nil { - t.Fatal(err) - } - + swarm := GenSwarm(t, ctx, opts...) swarm.SetStreamHandler(EchoStreamHandler) swarms = append(swarms, swarm) } @@ -94,8 +77,8 @@ func connectSwarms(t *testing.T, ctx context.Context, swarms []*Swarm) { var wg sync.WaitGroup connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // TODO: make a DialAddr func. - s.peers.AddAddr(dst, addr, pstore.PermanentAddrTTL) - if _, err := s.Dial(ctx, dst); err != nil { + s.Peerstore().AddAddr(dst, addr, pstore.PermanentAddrTTL) + if _, err := s.DialPeer(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } wg.Done() @@ -111,7 +94,7 @@ func connectSwarms(t *testing.T, ctx context.Context, swarms []*Swarm) { wg.Wait() for _, s := range swarms { - log.Infof("%s swarm routing table: %s", s.local, s.Peers()) + log.Infof("%s swarm routing table: %s", s.LocalPeer(), s.Peers()) } } @@ -119,7 +102,7 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { // t.Skip("skipping for another test") ctx := context.Background() - swarms := makeSwarms(ctx, t, SwarmNum) + swarms := makeSwarms(ctx, t, SwarmNum, OptDisableReuseport) // connect everyone connectSwarms(t, ctx, swarms) @@ -127,13 +110,13 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { // ping/pong for _, s1 := range swarms { log.Debugf("-------------------------------------------------------") - log.Debugf("%s ping pong round", s1.local) + log.Debugf("%s ping pong round", s1.LocalPeer()) log.Debugf("-------------------------------------------------------") _, cancel := context.WithCancel(ctx) got := map[peer.ID]int{} errChan := make(chan error, MsgNum*len(swarms)) - streamChan := make(chan *Stream, MsgNum) + streamChan := make(chan inet.Stream, MsgNum) // send out "ping" x MsgNum to every peer go func() { @@ -144,7 +127,7 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { defer wg.Done() // first, one stream per peer (nice) - stream, err := s1.NewStreamWithPeer(ctx, p) + stream, err := s1.NewStream(ctx, p) if err != nil { errChan <- err return @@ -153,7 +136,7 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { // send out ping! for k := 0; k < MsgNum; k++ { // with k messages msg := "ping" - log.Debugf("%s %s %s (%d)", s1.local, msg, p, k) + log.Debugf("%s %s %s (%d)", s1.LocalPeer(), msg, p, k) if _, err := stream.Write([]byte(msg)); err != nil { errChan <- err continue @@ -165,12 +148,12 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { } for _, s2 := range swarms { - if s2.local == s1.local { + if s2.LocalPeer() == s1.LocalPeer() { continue // dont send to self... } wg.Add(1) - go send(s2.local) + go send(s2.LocalPeer()) } wg.Wait() }() @@ -202,7 +185,7 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { continue } - log.Debugf("%s %s %s (%d)", s1.local, msg, p, k) + log.Debugf("%s %s %s (%d)", s1.LocalPeer(), msg, p, k) msgCount++ } @@ -222,7 +205,7 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { } } - log.Debugf("%s got pongs", s1.local) + log.Debugf("%s got pongs", s1.LocalPeer()) if (len(swarms) - 1) != len(got) { t.Errorf("got (%d) less messages than sent (%d).", len(got), len(swarms)) } @@ -269,7 +252,7 @@ func TestConnHandler(t *testing.T) { swarms := makeSwarms(ctx, t, 5) gotconn := make(chan struct{}, 10) - swarms[0].SetConnHandler(func(conn *Conn) { + swarms[0].SetConnHandler(func(conn inet.Conn) { gotconn <- struct{}{} }) @@ -300,7 +283,7 @@ func TestAddrBlocking(t *testing.T) { ctx := context.Background() swarms := makeSwarms(ctx, t, 2) - swarms[0].SetConnHandler(func(conn *Conn) { + swarms[0].SetConnHandler(func(conn inet.Conn) { t.Errorf("no connections should happen! -- %s", conn) }) @@ -311,14 +294,14 @@ func TestAddrBlocking(t *testing.T) { swarms[1].Filters.AddDialFilter(block) - swarms[1].peers.AddAddr(swarms[0].LocalPeer(), swarms[0].ListenAddresses()[0], pstore.PermanentAddrTTL) - _, err = swarms[1].Dial(ctx, swarms[0].LocalPeer()) + swarms[1].Peerstore().AddAddr(swarms[0].LocalPeer(), swarms[0].ListenAddresses()[0], pstore.PermanentAddrTTL) + _, err = swarms[1].DialPeer(ctx, swarms[0].LocalPeer()) if err == nil { t.Fatal("dial should have failed") } - swarms[0].peers.AddAddr(swarms[1].LocalPeer(), swarms[1].ListenAddresses()[0], pstore.PermanentAddrTTL) - _, err = swarms[0].Dial(ctx, swarms[1].LocalPeer()) + swarms[0].Peerstore().AddAddr(swarms[1].LocalPeer(), swarms[1].ListenAddresses()[0], pstore.PermanentAddrTTL) + _, err = swarms[0].DialPeer(ctx, swarms[1].LocalPeer()) if err == nil { t.Fatal("dial should have failed") } @@ -329,7 +312,7 @@ func TestFilterBounds(t *testing.T) { swarms := makeSwarms(ctx, t, 2) conns := make(chan struct{}, 8) - swarms[0].SetConnHandler(func(conn *Conn) { + swarms[0].SetConnHandler(func(conn inet.Conn) { conns <- struct{}{} }) diff --git a/p2p/net/swarm/swarm_transport.go b/p2p/net/swarm/swarm_transport.go new file mode 100644 index 0000000000..bc60dcf993 --- /dev/null +++ b/p2p/net/swarm/swarm_transport.go @@ -0,0 +1,87 @@ +package swarm + +import ( + "fmt" + "strings" + + transport "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" +) + +// TransportForDialing retrieves the appropriate transport for dialing the given +// multiaddr. +func (s *Swarm) TransportForDialing(a ma.Multiaddr) transport.Transport { + protocols := a.Protocols() + if len(protocols) == 0 { + return nil + } + + s.transports.RLock() + defer s.transports.RUnlock() + for _, p := range protocols { + transport, ok := s.transports.m[p.Code] + if !ok { + continue + } + if transport.Proxy() { + return transport + } + } + + return s.transports.m[protocols[len(protocols)-1].Code] +} + +// TransportForListening retrieves the appropriate transport for listening on +// the given multiaddr. +func (s *Swarm) TransportForListening(a ma.Multiaddr) transport.Transport { + protocols := a.Protocols() + if len(protocols) == 0 { + return nil + } + + s.transports.RLock() + defer s.transports.RUnlock() + selected := s.transports.m[protocols[len(protocols)-1].Code] + for _, p := range protocols { + transport, ok := s.transports.m[p.Code] + if !ok { + continue + } + if transport.Proxy() { + selected = transport + } + } + return selected +} + +// AddTransport adds a transport to this swarm. +// +// Satisfies the Network interface from go-libp2p-transport. +func (s *Swarm) AddTransport(t transport.Transport) error { + protocols := t.Protocols() + + s.transports.Lock() + defer s.transports.Unlock() + var registered []string + for _, p := range protocols { + if _, ok := s.transports.m[p]; ok { + proto := ma.ProtocolWithCode(p) + name := proto.Name + if name == "" { + name = fmt.Sprintf("unknown (%d)", p) + } + registered = append(registered, name) + } + } + if len(registered) > 0 { + return fmt.Errorf( + "transports already registered for protocol(s): %s", + strings.Join(registered, ", "), + ) + } + + for _, p := range protocols { + s.transports.m[p] = t + } + return nil +} diff --git a/p2p/net/swarm/testing/testing.go b/p2p/net/swarm/testing/testing.go new file mode 100644 index 0000000000..29017df808 --- /dev/null +++ b/p2p/net/swarm/testing/testing.go @@ -0,0 +1,97 @@ +package testing + +import ( + "context" + "testing" + + csms "github.com/libp2p/go-conn-security-multistream" + metrics "github.com/libp2p/go-libp2p-metrics" + inet "github.com/libp2p/go-libp2p-net" + pstore "github.com/libp2p/go-libp2p-peerstore" + secio "github.com/libp2p/go-libp2p-secio" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + tcp "github.com/libp2p/go-tcp-transport" + tu "github.com/libp2p/go-testutil" + msmux "github.com/whyrusleeping/go-smux-multistream" + yamux "github.com/whyrusleeping/go-smux-yamux" + + swarm "github.com/libp2p/go-libp2p-swarm" +) + +type config struct { + disableReuseport bool + dialOnly bool +} + +// Option is an option that can be passed when constructing a test swarm. +type Option func(*testing.T, *config) + +// OptDisableReuseport disables reuseport in this test swarm. +var OptDisableReuseport Option = func(_ *testing.T, c *config) { + c.disableReuseport = true +} + +// OptDialOnly prevents the test swarm from listening. +var OptDialOnly Option = func(_ *testing.T, c *config) { + c.dialOnly = true +} + +// GenUpgrader creates a new connection upgrader for use with this swarm. +func GenUpgrader(n *swarm.Swarm) *tptu.Upgrader { + id := n.LocalPeer() + pk := n.Peerstore().PrivKey(id) + secMuxer := new(csms.SSMuxer) + secMuxer.AddTransport(secio.ID, &secio.Transport{ + LocalID: id, + PrivateKey: pk, + }) + + stMuxer := msmux.NewBlankTransport() + stMuxer.AddTransport("/yamux/1.0.0", yamux.DefaultTransport) + + return &tptu.Upgrader{ + Secure: secMuxer, + Muxer: stMuxer, + Filters: n.Filters, + } + +} + +// GenSwarm generates a new test swarm. +func GenSwarm(t *testing.T, ctx context.Context, opts ...Option) *swarm.Swarm { + var cfg config + for _, o := range opts { + o(t, &cfg) + } + + p := tu.RandPeerNetParamsOrFatal(t) + + ps := pstore.NewPeerstore() + ps.AddPubKey(p.ID, p.PubKey) + ps.AddPrivKey(p.ID, p.PrivKey) + s := swarm.NewSwarm(ctx, p.ID, ps, metrics.NewBandwidthCounter()) + + tcpTransport := tcp.NewTCPTransport(GenUpgrader(s)) + tcpTransport.DisableReuseport = cfg.disableReuseport + + if err := s.AddTransport(tcpTransport); err != nil { + t.Fatal(err) + } + + if !cfg.dialOnly { + if err := s.Listen(p.Addr); err != nil { + t.Fatal(err) + } + + s.Peerstore().AddAddrs(p.ID, s.ListenAddresses(), pstore.PermanentAddrTTL) + } + + return s +} + +// DivulgeAddresses adds swarm a's addresses to swarm b's peerstore. +func DivulgeAddresses(a, b inet.Network) { + id := a.LocalPeer() + addrs := a.Peerstore().Addrs(id) + b.Peerstore().AddAddrs(id, addrs, pstore.PermanentAddrTTL) +} From 6c449aa083f4a47e402504d932788add4dda1d9d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 15 Feb 2018 12:09:54 -0800 Subject: [PATCH 0471/3965] fixup for transport refactor. --- .../internal/circuitv1-deprecated/conn.go | 33 +---- .../internal/circuitv1-deprecated/dial.go | 48 +++---- .../internal/circuitv1-deprecated/listen.go | 20 +-- .../internal/circuitv1-deprecated/notify.go | 2 +- .../internal/circuitv1-deprecated/relay.go | 22 ++- .../circuitv1-deprecated/relay_test.go | 134 +++++------------- .../circuitv1-deprecated/transport.go | 72 ++++------ .../circuitv1-deprecated/transport_test.go | 8 +- 8 files changed, 111 insertions(+), 228 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 35043e5e39..b0afbd063f 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -4,24 +4,17 @@ import ( "fmt" "net" - ic "github.com/libp2p/go-libp2p-crypto" - iconn "github.com/libp2p/go-libp2p-interface-conn" inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" - tpt "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) type Conn struct { inet.Stream - remote pstore.PeerInfo - transport tpt.Transport + remote pstore.PeerInfo } -var _ iconn.Conn = (*Conn)(nil) - type NetAddr struct { Relay string Remote string @@ -62,27 +55,3 @@ func (c *Conn) LocalAddr() net.Addr { } return na } - -func (c *Conn) Transport() tpt.Transport { - return c.transport -} - -func (c *Conn) LocalPeer() peer.ID { - return c.Conn().LocalPeer() -} - -func (c *Conn) RemotePeer() peer.ID { - return c.remote.ID -} - -func (c *Conn) LocalPrivateKey() ic.PrivKey { - return nil -} - -func (c *Conn) RemotePublicKey() ic.PubKey { - return nil -} - -func (c *Conn) ID() string { - return iconn.ID(c) -} diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 1f01be6aa6..3f021fa434 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -11,24 +11,16 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -var _ tpt.Dialer = (*RelayDialer)(nil) - -type RelayDialer Relay - -func (d *RelayDialer) Relay() *Relay { - return (*Relay)(d) -} - -func (r *Relay) Dialer() *RelayDialer { - return (*RelayDialer)(r) -} - -func (d *RelayDialer) Dial(a ma.Multiaddr) (tpt.Conn, error) { - return d.DialContext(d.ctx, a) +func (d *RelayTransport) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (tpt.Conn, error) { + c, err := d.Relay().Dial(ctx, a) + if err != nil { + return nil, err + } + return d.upgrader.UpgradeOutbound(ctx, d, c, p) } -func (d *RelayDialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn, error) { - if !d.Matches(a) { +func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr) (*Conn, error) { + if !r.Matches(a) { return nil, fmt.Errorf("%s is not a relay address", a) } parts := ma.Split(a) @@ -51,24 +43,25 @@ func (d *RelayDialer) DialContext(ctx context.Context, a ma.Multiaddr) (tpt.Conn if len(relayaddr.Bytes()) == 0 { // unspecific relay address, try dialing using known hop relays - return d.tryDialRelays(ctx, *dinfo) + return r.tryDialRelays(ctx, *dinfo) } - rinfo, err := pstore.InfoFromP2pAddr(relayaddr) + var rinfo *pstore.PeerInfo + rinfo, err = pstore.InfoFromP2pAddr(relayaddr) if err != nil { return nil, err } - return d.Relay().DialPeer(ctx, *rinfo, *dinfo) + return r.DialPeer(ctx, *rinfo, *dinfo) } -func (d *RelayDialer) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) (tpt.Conn, error) { +func (r *Relay) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) (*Conn, error) { var relays []peer.ID - d.mx.Lock() - for p := range d.relays { + r.mx.Lock() + for p := range r.relays { relays = append(relays, p) } - d.mx.Unlock() + r.mx.Unlock() // shuffle list of relays, avoid overloading a specific relay for i := range relays { @@ -77,12 +70,12 @@ func (d *RelayDialer) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) } for _, relay := range relays { - if len(d.host.Network().ConnsToPeer(relay)) == 0 { + if len(r.host.Network().ConnsToPeer(relay)) == 0 { continue } rctx, cancel := context.WithTimeout(ctx, HopConnectTimeout) - c, err := d.Relay().DialPeer(rctx, pstore.PeerInfo{ID: relay}, dinfo) + c, err := r.DialPeer(rctx, pstore.PeerInfo{ID: relay}, dinfo) cancel() if err == nil { @@ -94,8 +87,3 @@ func (d *RelayDialer) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) return nil, fmt.Errorf("Failed to dial through %d known relay hosts", len(relays)) } - -func (d *RelayDialer) Matches(a ma.Multiaddr) bool { - _, err := a.ValueForProtocol(P_CIRCUIT) - return err == nil -} diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index 9eac888913..ee1f92e15d 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -6,13 +6,11 @@ import ( pb "github.com/libp2p/go-libp2p-circuit/pb" - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" - filter "github.com/libp2p/go-maddr-filter" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) -var _ tpt.Listener = (*RelayListener)(nil) +var _ manet.Listener = (*RelayListener)(nil) type RelayListener Relay @@ -21,10 +19,11 @@ func (l *RelayListener) Relay() *Relay { } func (r *Relay) Listener() *RelayListener { + // TODO: Only allow one! return (*RelayListener)(r) } -func (l *RelayListener) Accept() (tpt.Conn, error) { +func (l *RelayListener) Accept() (manet.Conn, error) { select { case c := <-l.incoming: err := l.Relay().writeResponse(c.Stream, pb.CircuitRelay_SUCCESS) @@ -34,7 +33,8 @@ func (l *RelayListener) Accept() (tpt.Conn, error) { return nil, err } - log.Infof("accepted relay connection: %s", c.ID()) + // TODO: Pretty print. + log.Infof("accepted relay connection: %s", c) return c, nil case <-l.ctx.Done(): @@ -57,14 +57,6 @@ func (l *RelayListener) Multiaddr() ma.Multiaddr { return a } -func (l *RelayListener) LocalPeer() peer.ID { - return l.self -} - -func (l *RelayListener) SetAddrFilters(f *filter.Filters) { - // noop ? -} - func (l *RelayListener) Close() error { // TODO: noop? return nil diff --git a/p2p/protocol/internal/circuitv1-deprecated/notify.go b/p2p/protocol/internal/circuitv1-deprecated/notify.go index dec311671a..7ded2b623b 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/notify.go +++ b/p2p/protocol/internal/circuitv1-deprecated/notify.go @@ -27,7 +27,7 @@ func (n *RelayNotifiee) OpenedStream(net inet.Network, s inet.Stream) {} func (n *RelayNotifiee) ClosedStream(net inet.Network, s inet.Stream) {} func (n *RelayNotifiee) Connected(s inet.Network, c inet.Conn) { - if n.Relay().Transport().Matches(c.RemoteMultiaddr()) { + if n.Relay().Matches(c.RemoteMultiaddr()) { return } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 91ba7f23c6..4a8a9e2851 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -14,6 +14,8 @@ import ( inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + ma "github.com/multiformats/go-multiaddr" ) var log = logging.Logger("relay") @@ -26,9 +28,10 @@ var RelayAcceptTimeout = time.Minute var HopConnectTimeout = 10 * time.Second type Relay struct { - host host.Host - ctx context.Context - self peer.ID + host host.Host + upgrader *tptu.Upgrader + ctx context.Context + self peer.ID active bool hop bool @@ -54,8 +57,9 @@ func (e RelayError) Error() string { return fmt.Sprintf("error opening relay circuit: %s (%d)", pb.CircuitRelay_Status_name[int32(e.Code)], e.Code) } -func NewRelay(ctx context.Context, h host.Host, opts ...RelayOpt) (*Relay, error) { +func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts ...RelayOpt) (*Relay, error) { r := &Relay{ + upgrader: upgrader, host: h, ctx: ctx, self: h.ID(), @@ -126,7 +130,13 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore return nil, RelayError{msg.GetCode()} } - return &Conn{Stream: s, remote: dest, transport: r.Transport()}, nil + return &Conn{Stream: s, remote: dest}, nil +} + +func (r *Relay) Matches(addr ma.Multiaddr) bool { + // TODO: Look at the prefix transport as well. + _, err := addr.ValueForProtocol(P_CIRCUIT) + return err == nil } func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { @@ -340,7 +350,7 @@ func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { } select { - case r.incoming <- &Conn{Stream: s, remote: src, transport: r.Transport()}: + case r.incoming <- &Conn{Stream: s, remote: src}: case <-time.After(RelayAcceptTimeout): r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 5213e122bd..04feb6ca80 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -13,7 +13,8 @@ import ( bhost "github.com/libp2p/go-libp2p-blankhost" host "github.com/libp2p/go-libp2p-host" - netutil "github.com/libp2p/go-libp2p-netutil" + swarm "github.com/libp2p/go-libp2p-swarm" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" ) @@ -30,7 +31,7 @@ func getNetHosts(t *testing.T, ctx context.Context, n int) []host.Host { var out []host.Host for i := 0; i < n; i++ { - netw := netutil.GenSwarmNetwork(t, ctx) + netw := swarmt.GenSwarm(t, ctx) h := bhost.NewBlankHost(netw) out = append(out, h) } @@ -38,6 +39,14 @@ func getNetHosts(t *testing.T, ctx context.Context, n int) []host.Host { return out } +func newTestRelay(t *testing.T, ctx context.Context, host host.Host, opts ...RelayOpt) *Relay { + r, err := NewRelay(ctx, host, swarmt.GenUpgrader(host.Network().(*swarm.Swarm)), opts...) + if err != nil { + t.Fatal(err) + } + return r +} + func connect(t *testing.T, a, b host.Host) { pinfo := a.Peerstore().PeerInfo(a.ID()) err := b.Connect(context.Background(), pinfo) @@ -57,20 +66,11 @@ func TestBasicRelay(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1, err := NewRelay(ctx, hosts[0]) - if err != nil { - t.Fatal(err) - } + r1 := newTestRelay(t, ctx, hosts[0]) - _, err = NewRelay(ctx, hosts[1], OptHop) - if err != nil { - t.Fatal(err) - } + newTestRelay(t, ctx, hosts[1], OptHop) - r3, err := NewRelay(ctx, hosts[2]) - if err != nil { - t.Fatal(err) - } + r3 := newTestRelay(t, ctx, hosts[2]) msg := []byte("relay works!") go func() { @@ -122,20 +122,11 @@ func TestRelayReset(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1, err := NewRelay(ctx, hosts[0]) - if err != nil { - t.Fatal(err) - } + r1 := newTestRelay(t, ctx, hosts[0]) - _, err = NewRelay(ctx, hosts[1], OptHop) - if err != nil { - t.Fatal(err) - } + newTestRelay(t, ctx, hosts[1], OptHop) - r3, err := NewRelay(ctx, hosts[2]) - if err != nil { - t.Fatal(err) - } + r3 := newTestRelay(t, ctx, hosts[2]) ready := make(chan struct{}) @@ -190,20 +181,10 @@ func TestBasicRelayDial(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1, err := NewRelay(ctx, hosts[0]) - if err != nil { - t.Fatal(err) - } - - _, err = NewRelay(ctx, hosts[1], OptHop) - if err != nil { - t.Fatal(err) - } + r1 := newTestRelay(t, ctx, hosts[0]) - r3, err := NewRelay(ctx, hosts[2]) - if err != nil { - t.Fatal(err) - } + _ = newTestRelay(t, ctx, hosts[1], OptHop) + r3 := newTestRelay(t, ctx, hosts[2]) msg := []byte("relay works!") go func() { @@ -231,8 +212,7 @@ func TestBasicRelayDial(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - d := r1.Dialer() - con, err := d.DialContext(rctx, addr) + con, err := r1.Dial(rctx, addr) if err != nil { t.Fatal(err) } @@ -253,20 +233,11 @@ func TestUnspecificRelayDial(t *testing.T) { hosts := getNetHosts(t, ctx, 3) - r1, err := NewRelay(ctx, hosts[0]) - if err != nil { - t.Fatal(err) - } + r1 := newTestRelay(t, ctx, hosts[0]) - _, err = NewRelay(ctx, hosts[1], OptHop) - if err != nil { - t.Fatal(err) - } + newTestRelay(t, ctx, hosts[1], OptHop) - r3, err := NewRelay(ctx, hosts[2]) - if err != nil { - t.Fatal(err) - } + r3 := newTestRelay(t, ctx, hosts[2]) connect(t, hosts[0], hosts[1]) connect(t, hosts[1], hosts[2]) @@ -299,8 +270,7 @@ func TestUnspecificRelayDial(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - d := r1.Dialer() - con, err := d.DialContext(rctx, addr) + con, err := r1.Dial(rctx, addr) if err != nil { t.Fatal(err) } @@ -326,20 +296,11 @@ func TestRelayThroughNonHop(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1, err := NewRelay(ctx, hosts[0]) - if err != nil { - t.Fatal(err) - } + r1 := newTestRelay(t, ctx, hosts[0]) - _, err = NewRelay(ctx, hosts[1]) - if err != nil { - t.Fatal(err) - } + newTestRelay(t, ctx, hosts[1]) - _, err = NewRelay(ctx, hosts[2]) - if err != nil { - t.Fatal(err) - } + newTestRelay(t, ctx, hosts[2]) rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) @@ -347,7 +308,7 @@ func TestRelayThroughNonHop(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - _, err = r1.DialPeer(rctx, rinfo, dinfo) + _, err := r1.DialPeer(rctx, rinfo, dinfo) if err == nil { t.Fatal("expected error") } @@ -372,15 +333,9 @@ func TestRelayNoDestConnection(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1, err := NewRelay(ctx, hosts[0]) - if err != nil { - t.Fatal(err) - } + r1 := newTestRelay(t, ctx, hosts[0]) - _, err = NewRelay(ctx, hosts[1], OptHop) - if err != nil { - t.Fatal(err) - } + newTestRelay(t, ctx, hosts[1], OptHop) rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) dinfo := hosts[2].Peerstore().PeerInfo(hosts[2].ID()) @@ -388,7 +343,7 @@ func TestRelayNoDestConnection(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - _, err = r1.DialPeer(rctx, rinfo, dinfo) + _, err := r1.DialPeer(rctx, rinfo, dinfo) if err == nil { t.Fatal("expected error") } @@ -413,20 +368,11 @@ func TestActiveRelay(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1, err := NewRelay(ctx, hosts[0]) - if err != nil { - t.Fatal(err) - } + r1 := newTestRelay(t, ctx, hosts[0]) - _, err = NewRelay(ctx, hosts[1], OptHop, OptActive) - if err != nil { - t.Fatal(err) - } + newTestRelay(t, ctx, hosts[1], OptHop, OptActive) - r3, err := NewRelay(ctx, hosts[2]) - if err != nil { - t.Fatal(err) - } + r3 := newTestRelay(t, ctx, hosts[2]) msg := []byte("relay works!") go func() { @@ -477,15 +423,9 @@ func TestRelayCanHop(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1, err := NewRelay(ctx, hosts[0]) - if err != nil { - t.Fatal(err) - } + r1 := newTestRelay(t, ctx, hosts[0]) - _, err = NewRelay(ctx, hosts[1], OptHop) - if err != nil { - t.Fatal(err) - } + newTestRelay(t, ctx, hosts[1], OptHop) canhop, err := r1.CanHop(ctx, hosts[1].ID()) if err != nil { diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 1396828747..86821d327c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -4,10 +4,9 @@ import ( "context" "fmt" - addrutil "github.com/libp2p/go-addr-util" host "github.com/libp2p/go-libp2p-host" - swarm "github.com/libp2p/go-libp2p-swarm" tpt "github.com/libp2p/go-libp2p-transport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" ma "github.com/multiformats/go-multiaddr" ) @@ -22,29 +21,6 @@ var Protocol = ma.Protocol{ func init() { ma.AddProtocol(Protocol) - - // Add dialer transport - const unspecific = "/p2p-circuit/ipfs" - const proto = "/ipfs/p2p-circuit/ipfs" - - tps := addrutil.SupportedTransportStrings - - err := addrutil.AddTransport(unspecific) - if err != nil { - panic(err) - } - - err = addrutil.AddTransport(proto) - if err != nil { - panic(err) - } - - for _, tp := range tps { - err = addrutil.AddTransport(tp + proto) - if err != nil { - panic(err) - } - } } var _ tpt.Transport = (*RelayTransport)(nil) @@ -59,40 +35,46 @@ func (r *Relay) Transport() *RelayTransport { return (*RelayTransport)(r) } -func (t *RelayTransport) Dialer(laddr ma.Multiaddr, opts ...tpt.DialOpt) (tpt.Dialer, error) { - if !t.Matches(laddr) { +func (t *RelayTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { + // TODO: Ensure we have a connection to the relay, if specified. Also, + // make sure the multiaddr makes sense. + if !t.Relay().Matches(laddr) { return nil, fmt.Errorf("%s is not a relay address", laddr) } - return t.Relay().Dialer(), nil + return t.upgrader.UpgradeListener(t, t.Relay().Listener()), nil } -func (t *RelayTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { - if !t.Matches(laddr) { - return nil, fmt.Errorf("%s is not a relay address", laddr) - } - return t.Relay().Listener(), nil +func (t *RelayTransport) CanDial(raddr ma.Multiaddr) bool { + return t.Relay().Matches(raddr) } -func (t *RelayTransport) Matches(a ma.Multiaddr) bool { - return t.Relay().Dialer().Matches(a) +func (t *RelayTransport) Proxy() bool { + return true +} + +func (t *RelayTransport) Protocols() []int { + return []int{P_CIRCUIT} } // AddRelayTransport constructs a relay and adds it as a transport to the host network. -func AddRelayTransport(ctx context.Context, h host.Host, opts ...RelayOpt) error { - // the necessary methods are not part of the Network interface, only exported by Swarm - // TODO: generalize the network interface for adding tranports - n, ok := h.Network().(*swarm.Network) +func AddRelayTransport(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts ...RelayOpt) error { + n, ok := h.Network().(tpt.Network) if !ok { - return fmt.Errorf("%v is not a swarm network", h.Network()) + return fmt.Errorf("%v is not a transport network", h.Network()) } - s := n.Swarm() - - r, err := NewRelay(ctx, h, opts...) + r, err := NewRelay(ctx, h, upgrader, opts...) if err != nil { return err } - s.AddTransport(r.Transport()) - return s.AddListenAddr(r.Listener().Multiaddr()) + // There's no nice way to handle these errors as we have no way to tear + // down the relay. + // TODO + if err := n.AddTransport(r.Transport()); err != nil { + log.Error("failed to add relay transport:", err) + } else if err := n.Listen(r.Listener().Multiaddr()); err != nil { + log.Error("failed to listen on relay transport:", err) + } + return nil } diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go index 512ba8644b..b718a90fe7 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go @@ -13,6 +13,8 @@ import ( host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" pstore "github.com/libp2p/go-libp2p-peerstore" + swarm "github.com/libp2p/go-libp2p-swarm" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" ) @@ -23,17 +25,17 @@ var msg = []byte("relay works!") func testSetupRelay(t *testing.T, ctx context.Context) []host.Host { hosts := getNetHosts(t, ctx, 3) - err := AddRelayTransport(ctx, hosts[0]) + err := AddRelayTransport(ctx, hosts[0], swarmt.GenUpgrader(hosts[0].Network().(*swarm.Swarm))) if err != nil { t.Fatal(err) } - err = AddRelayTransport(ctx, hosts[1], OptHop) + err = AddRelayTransport(ctx, hosts[1], swarmt.GenUpgrader(hosts[1].Network().(*swarm.Swarm)), OptHop) if err != nil { t.Fatal(err) } - err = AddRelayTransport(ctx, hosts[2]) + err = AddRelayTransport(ctx, hosts[2], swarmt.GenUpgrader(hosts[2].Network().(*swarm.Swarm))) if err != nil { t.Fatal(err) } From 98d61475ebaf893acbbc52632b8f762a32977db7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 8 Mar 2018 12:15:51 -0800 Subject: [PATCH 0472/3965] fully close streams when testing CanHop --- .../internal/circuitv1-deprecated/relay.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 4a8a9e2851..99e89d45ff 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -152,18 +152,18 @@ func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { msg.Type = pb.CircuitRelay_CAN_HOP.Enum() - err = wr.WriteMsg(&msg) - if err != nil { + if err := wr.WriteMsg(&msg); err != nil { s.Reset() return false, err } msg.Reset() - err = rd.ReadMsg(&msg) - s.Close() - - if err != nil { + if err := rd.ReadMsg(&msg); err != nil { + s.Reset() + return false, err + } + if err := inet.FullClose(s); err != nil { return false, err } @@ -369,7 +369,7 @@ func (r *Relay) handleCanHop(s inet.Stream, msg *pb.CircuitRelay) { s.Reset() log.Debugf("error writing relay response: %s", err.Error()) } else { - s.Close() + inet.FullClose(s) } } @@ -380,7 +380,7 @@ func (r *Relay) handleError(s inet.Stream, code pb.CircuitRelay_Status) { s.Reset() log.Debugf("error writing relay response: %s", err.Error()) } else { - s.Close() + inet.FullClose(s) } } From 41c683485015bca19626138f18fa0027b5a8243d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 9 Mar 2018 17:56:02 -0800 Subject: [PATCH 0473/3965] refactor for transport changes Also, make the libp2p constructor fully useful. There should now be no need to manually construct a swarm/host. --- config/config.go | 176 +++++++++++++ config/constructor_types.go | 62 +++++ config/muxer.go | 62 +++++ config/muxer_test.go | 54 ++++ config/reflection_magic.go | 126 +++++++++ config/reflection_magic_test.go | 142 ++++++++++ config/security.go | 76 ++++++ config/transport.go | 66 +++++ defaults.go | 109 ++++++++ error_util.go | 17 ++ examples/chat/chat.go | 30 +-- examples/echo/main.go | 2 +- examples/http-proxy/proxy.go | 24 +- examples/multipro/main.go | 17 +- .../main.go | 22 +- libp2p.go | 241 ++--------------- libp2p_test.go | 29 ++- options.go | 243 ++++++++++++++++++ p2p/discovery/mdns_test.go | 6 +- p2p/host/basic/basic_host.go | 66 +---- p2p/host/basic/basic_host_test.go | 20 +- p2p/net/mock/mock_conn.go | 4 +- p2p/protocol/identify/id.go | 24 +- p2p/protocol/identify/id_test.go | 8 +- p2p/protocol/ping/ping_test.go | 6 +- p2p/test/backpressure/backpressure_test.go | 10 +- p2p/test/reconnects/reconnect_test.go | 33 +-- package.json | 171 +++++------- 28 files changed, 1335 insertions(+), 511 deletions(-) create mode 100644 config/config.go create mode 100644 config/constructor_types.go create mode 100644 config/muxer.go create mode 100644 config/muxer_test.go create mode 100644 config/reflection_magic.go create mode 100644 config/reflection_magic_test.go create mode 100644 config/security.go create mode 100644 config/transport.go create mode 100644 defaults.go create mode 100644 error_util.go create mode 100644 options.go diff --git a/config/config.go b/config/config.go new file mode 100644 index 0000000000..b017fea60e --- /dev/null +++ b/config/config.go @@ -0,0 +1,176 @@ +package config + +import ( + "context" + "fmt" + + bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + + logging "github.com/ipfs/go-log" + circuit "github.com/libp2p/go-libp2p-circuit" + crypto "github.com/libp2p/go-libp2p-crypto" + host "github.com/libp2p/go-libp2p-host" + ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" + pnet "github.com/libp2p/go-libp2p-interface-pnet" + metrics "github.com/libp2p/go-libp2p-metrics" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + swarm "github.com/libp2p/go-libp2p-swarm" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + filter "github.com/libp2p/go-maddr-filter" + ma "github.com/multiformats/go-multiaddr" +) + +var log = logging.Logger("p2p-config") + +// AddrsFactory is a function that takes a set of multiaddrs we're listening on and +// returns the set of multiaddrs we should advertise to the network. +type AddrsFactory = bhost.AddrsFactory + +// NATManagerC is a NATManager constructor. +type NATManagerC func(inet.Network) bhost.NATManager + +// Config describes a set of settings for a libp2p node +// +// This is *not* a stable interface. Use the options defined in the root +// package. +type Config struct { + PeerKey crypto.PrivKey + + Transports []TptC + Muxers []MsMuxC + SecurityTransports []MsSecC + Insecure bool + Protector pnet.Protector + + Relay bool + RelayOpts []circuit.RelayOpt + + ListenAddrs []ma.Multiaddr + AddrsFactory bhost.AddrsFactory + Filters *filter.Filters + + ConnManager ifconnmgr.ConnManager + NATManager NATManagerC + Peerstore pstore.Peerstore + Reporter metrics.Reporter +} + +// NewNode constructs a new libp2p Host from the Config. +// +// This function consumes the config. Do not reuse it (really!). +func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { + // Check this early. Prevents us from even *starting* without verifying this. + if pnet.ForcePrivateNetwork && cfg.Protector == nil { + log.Error("tried to create a libp2p node with no Private" + + " Network Protector but usage of Private Networks" + + " is forced by the enviroment") + // Note: This is *also* checked the upgrader itself so it'll be + // enforced even *if* you don't use the libp2p constructor. + return nil, pnet.ErrNotInPrivateNetwork + } + + if cfg.PeerKey == nil { + return nil, fmt.Errorf("no peer key specified") + } + + // Obtain Peer ID from public key + pid, err := peer.IDFromPublicKey(cfg.PeerKey.GetPublic()) + if err != nil { + return nil, err + } + + if cfg.Peerstore == nil { + return nil, fmt.Errorf("no peerstore specified") + } + + if !cfg.Insecure { + cfg.Peerstore.AddPrivKey(pid, cfg.PeerKey) + cfg.Peerstore.AddPubKey(pid, cfg.PeerKey.GetPublic()) + } + + // TODO: Make the swarm implementation configurable. + swrm := swarm.NewSwarm(ctx, pid, cfg.Peerstore, cfg.Reporter) + if cfg.Filters != nil { + swrm.Filters = cfg.Filters + } + + // TODO: make host implementation configurable. + h, err := bhost.NewHost(ctx, swrm, &bhost.HostOpts{ + ConnManager: cfg.ConnManager, + AddrsFactory: cfg.AddrsFactory, + NATManager: cfg.NATManager, + }) + if err != nil { + swrm.Close() + return nil, err + } + + upgrader := new(tptu.Upgrader) + upgrader.Protector = cfg.Protector + upgrader.Filters = swrm.Filters + if cfg.Insecure { + upgrader.Secure = makeInsecureTransport(pid) + } else { + upgrader.Secure, err = makeSecurityTransport(h, cfg.SecurityTransports) + if err != nil { + h.Close() + return nil, err + } + } + + upgrader.Muxer, err = makeMuxer(h, cfg.Muxers) + if err != nil { + h.Close() + return nil, err + } + + tpts, err := makeTransports(h, upgrader, cfg.Transports) + if err != nil { + h.Close() + return nil, err + } + for _, t := range tpts { + err = swrm.AddTransport(t) + if err != nil { + h.Close() + return nil, err + } + } + + if cfg.Relay { + err := circuit.AddRelayTransport(swrm.Context(), h, upgrader, cfg.RelayOpts...) + if err != nil { + h.Close() + return nil, err + } + } + + // TODO: This method succeeds if listening on one address succeeds. We + // should probably fail if listening on *any* addr fails. + if err := h.Network().Listen(cfg.ListenAddrs...); err != nil { + h.Close() + return nil, err + } + + // TODO: Configure routing (it's a pain to setup). + // TODO: Bootstrapping. + + return h, nil +} + +// Option is a libp2p config option that can be given to the libp2p constructor +// (`libp2p.New`). +type Option func(cfg *Config) error + +// Apply applies the given options to the config, returning the first error +// encountered (if any). +func (cfg *Config) Apply(opts ...Option) error { + for _, opt := range opts { + if err := opt(cfg); err != nil { + return err + } + } + return nil +} diff --git a/config/constructor_types.go b/config/constructor_types.go new file mode 100644 index 0000000000..0cf4163fc1 --- /dev/null +++ b/config/constructor_types.go @@ -0,0 +1,62 @@ +package config + +import ( + "fmt" + "reflect" + + security "github.com/libp2p/go-conn-security" + crypto "github.com/libp2p/go-libp2p-crypto" + host "github.com/libp2p/go-libp2p-host" + pnet "github.com/libp2p/go-libp2p-interface-pnet" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + transport "github.com/libp2p/go-libp2p-transport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + filter "github.com/libp2p/go-maddr-filter" + mux "github.com/libp2p/go-stream-muxer" +) + +var ( + // interfaces + hostType = reflect.TypeOf((*host.Host)(nil)).Elem() + networkType = reflect.TypeOf((*inet.Network)(nil)).Elem() + transportType = reflect.TypeOf((*transport.Transport)(nil)).Elem() + muxType = reflect.TypeOf((*mux.Transport)(nil)).Elem() + securityType = reflect.TypeOf((*security.Transport)(nil)).Elem() + protectorType = reflect.TypeOf((*pnet.Protector)(nil)).Elem() + privKeyType = reflect.TypeOf((*crypto.PrivKey)(nil)).Elem() + pubKeyType = reflect.TypeOf((*crypto.PubKey)(nil)).Elem() + pstoreType = reflect.TypeOf((*pstore.Peerstore)(nil)).Elem() + + // concrete types + peerIDType = reflect.TypeOf((peer.ID)("")) + filtersType = reflect.TypeOf((*filter.Filters)(nil)) + upgraderType = reflect.TypeOf((*tptu.Upgrader)(nil)) +) + +var argTypes = map[reflect.Type]constructor{ + upgraderType: func(h host.Host, u *tptu.Upgrader) interface{} { return u }, + hostType: func(h host.Host, u *tptu.Upgrader) interface{} { return h }, + networkType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Network() }, + muxType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Muxer }, + securityType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Secure }, + protectorType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Protector }, + filtersType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Filters }, + peerIDType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.ID() }, + privKeyType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore().PrivKey(h.ID()) }, + pubKeyType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore().PubKey(h.ID()) }, + pstoreType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore() }, +} + +func newArgTypeSet(types ...reflect.Type) map[reflect.Type]constructor { + result := make(map[reflect.Type]constructor, len(types)) + for _, ty := range types { + c, ok := argTypes[ty] + if !ok { + panic(fmt.Sprintf("missing constructor for type %s", ty)) + } + result[ty] = c + } + return result +} diff --git a/config/muxer.go b/config/muxer.go new file mode 100644 index 0000000000..6a48cb3bd4 --- /dev/null +++ b/config/muxer.go @@ -0,0 +1,62 @@ +package config + +import ( + "fmt" + + host "github.com/libp2p/go-libp2p-host" + mux "github.com/libp2p/go-stream-muxer" + msmux "github.com/whyrusleeping/go-smux-multistream" +) + +// MuxC is a stream multiplex transport constructor +type MuxC func(h host.Host) (mux.Transport, error) + +// MsMuxC is a tuple containing a multiplex transport constructor and a protocol +// ID. +type MsMuxC struct { + MuxC + ID string +} + +var muxArgTypes = newArgTypeSet(hostType, networkType, peerIDType, pstoreType) + +// MuxerConstructor creates a multiplex constructor from the passed parameter +// using reflection. +func MuxerConstructor(m interface{}) (MuxC, error) { + // Already constructed? + if t, ok := m.(mux.Transport); ok { + return func(_ host.Host) (mux.Transport, error) { + return t, nil + }, nil + } + + ctor, err := makeConstructor(m, muxType, muxArgTypes) + if err != nil { + return nil, err + } + return func(h host.Host) (mux.Transport, error) { + t, err := ctor(h, nil) + if err != nil { + return nil, err + } + return t.(mux.Transport), nil + }, nil +} + +func makeMuxer(h host.Host, tpts []MsMuxC) (mux.Transport, error) { + muxMuxer := msmux.NewBlankTransport() + transportSet := make(map[string]struct{}, len(tpts)) + for _, tptC := range tpts { + if _, ok := transportSet[tptC.ID]; ok { + return nil, fmt.Errorf("duplicate muxer transport: %s", tptC.ID) + } + } + for _, tptC := range tpts { + tpt, err := tptC.MuxC(h) + if err != nil { + return nil, err + } + muxMuxer.AddTransport(tptC.ID, tpt) + } + return muxMuxer, nil +} diff --git a/config/muxer_test.go b/config/muxer_test.go new file mode 100644 index 0000000000..5e7bab41ff --- /dev/null +++ b/config/muxer_test.go @@ -0,0 +1,54 @@ +package config + +import ( + "testing" + + peer "github.com/libp2p/go-libp2p-peer" + mux "github.com/libp2p/go-stream-muxer" + yamux "github.com/whyrusleeping/go-smux-yamux" +) + +func TestMuxerSimple(t *testing.T) { + // single + _, err := MuxerConstructor(func(_ peer.ID) mux.Transport { return nil }) + if err != nil { + t.Fatal(err) + } +} + +func TestMuxerByValue(t *testing.T) { + _, err := MuxerConstructor(yamux.DefaultTransport) + if err != nil { + t.Fatal(err) + } +} +func TestMuxerDuplicate(t *testing.T) { + _, err := MuxerConstructor(func(_ peer.ID, _ peer.ID) mux.Transport { return nil }) + if err != nil { + t.Fatal(err) + } +} + +func TestMuxerError(t *testing.T) { + _, err := MuxerConstructor(func() (mux.Transport, error) { return nil, nil }) + if err != nil { + t.Fatal(err) + } +} + +func TestMuxerBadTypes(t *testing.T) { + for i, f := range []interface{}{ + func() error { return nil }, + func() string { return "" }, + func() {}, + func(string) mux.Transport { return nil }, + func(string) (mux.Transport, error) { return nil, nil }, + nil, + "testing", + } { + + if _, err := MuxerConstructor(f); err == nil { + t.Fatalf("constructor %d with type %T should have failed", i, f) + } + } +} diff --git a/config/reflection_magic.go b/config/reflection_magic.go new file mode 100644 index 0000000000..c23658e9cc --- /dev/null +++ b/config/reflection_magic.go @@ -0,0 +1,126 @@ +package config + +import ( + "fmt" + "reflect" + "runtime" + + host "github.com/libp2p/go-libp2p-host" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" +) + +var errorType = reflect.TypeOf((*error)(nil)).Elem() + +// checks if a function returns either the specified type or the specified type +// and an error. +func checkReturnType(fnType, tptType reflect.Type) error { + switch fnType.NumOut() { + case 2: + if fnType.Out(1) != errorType { + return fmt.Errorf("expected (optional) second return value from transport constructor to be an error") + } + + fallthrough + case 1: + if !fnType.Out(0).Implements(tptType) { + return fmt.Errorf("expected first return value from transport constructor to be a transport") + } + default: + return fmt.Errorf("expected transport constructor to return a transport and, optionally, an error") + } + return nil +} + +// Handles return values with optional errors. That is, return values of the +// form `(something, error)` or just `something`. +// +// Panics if the return value isn't of the correct form. +func handleReturnValue(out []reflect.Value) (interface{}, error) { + switch len(out) { + case 2: + err := out[1] + if err != (reflect.Value{}) && !err.IsNil() { + return nil, err.Interface().(error) + } + fallthrough + case 1: + tpt := out[0] + + // Check for nil value and nil error. + if tpt == (reflect.Value{}) { + return nil, fmt.Errorf("unspecified error") + } + switch tpt.Kind() { + case reflect.Ptr, reflect.Interface, reflect.Func: + if tpt.IsNil() { + return nil, fmt.Errorf("unspecified error") + } + } + + return tpt.Interface(), nil + default: + panic("expected 1 or 2 return values from transport constructor") + } +} + +// calls the transport constructor and annotates the error with the name of the constructor. +func callConstructor(c reflect.Value, args []reflect.Value) (interface{}, error) { + val, err := handleReturnValue(c.Call(args)) + if err != nil { + name := runtime.FuncForPC(c.Pointer()).Name() + if name != "" { + // makes debugging easier + return nil, fmt.Errorf("transport constructor %s failed: %s", name, err) + } + } + return val, err +} + +type constructor func(h host.Host, u *tptu.Upgrader) interface{} + +func makeArgumentConstructors(fnType reflect.Type, argTypes map[reflect.Type]constructor) ([]constructor, error) { + out := make([]constructor, fnType.NumIn()) + for i := range out { + argType := fnType.In(i) + c, ok := argTypes[argType] + if !ok { + return nil, fmt.Errorf("argument %d has an unexpected type %s", i, argType.Name()) + } + out[i] = c + } + return out, nil +} + +// makes a transport constructor. +func makeConstructor( + tpt interface{}, + tptType reflect.Type, + argTypes map[reflect.Type]constructor, +) (func(host.Host, *tptu.Upgrader) (interface{}, error), error) { + v := reflect.ValueOf(tpt) + // avoid panicing on nil/zero value. + if v == (reflect.Value{}) { + return nil, fmt.Errorf("expected a transport or transport constructor, got a %T", tpt) + } + t := v.Type() + if t.Kind() != reflect.Func { + return nil, fmt.Errorf("expected a transport or transport constructor, got a %T", tpt) + } + + if err := checkReturnType(t, tptType); err != nil { + return nil, err + } + + argConstructors, err := makeArgumentConstructors(t, argTypes) + if err != nil { + return nil, err + } + + return func(h host.Host, u *tptu.Upgrader) (interface{}, error) { + arguments := make([]reflect.Value, len(argConstructors)) + for i, makeArg := range argConstructors { + arguments[i] = reflect.ValueOf(makeArg(h, u)) + } + return callConstructor(v, arguments) + }, nil +} diff --git a/config/reflection_magic_test.go b/config/reflection_magic_test.go new file mode 100644 index 0000000000..ce9f3986f3 --- /dev/null +++ b/config/reflection_magic_test.go @@ -0,0 +1,142 @@ +package config + +import ( + "errors" + "reflect" + "strings" + "testing" +) + +func TestHandleReturnValue(t *testing.T) { + // one value + v, err := handleReturnValue([]reflect.Value{reflect.ValueOf(1)}) + if v.(int) != 1 { + t.Fatal("expected value") + } + if err != nil { + t.Fatal(err) + } + + // Nil value + v, err = handleReturnValue([]reflect.Value{reflect.ValueOf(nil)}) + if v != nil { + t.Fatal("expected no value") + } + if err == nil { + t.Fatal("expected an error") + } + + // Nil value, nil err + v, err = handleReturnValue([]reflect.Value{reflect.ValueOf(nil), reflect.ValueOf(nil)}) + if v != nil { + t.Fatal("expected no value") + } + if err == nil { + t.Fatal("expected an error") + } + + // two values + v, err = handleReturnValue([]reflect.Value{reflect.ValueOf(1), reflect.ValueOf(nil)}) + if v, ok := v.(int); !ok || v != 1 { + t.Fatalf("expected value of 1, got %v", v) + } + if err != nil { + t.Fatal("expected no error") + } + + // an error + myError := errors.New("my error") + _, err = handleReturnValue([]reflect.Value{reflect.ValueOf(1), reflect.ValueOf(myError)}) + if err != myError { + t.Fatal(err) + } + + for _, vals := range [][]reflect.Value{ + {reflect.ValueOf(1), reflect.ValueOf("not an error")}, + {}, + {reflect.ValueOf(1), reflect.ValueOf(myError), reflect.ValueOf(myError)}, + } { + func() { + defer func() { recover() }() + handleReturnValue(vals) + t.Fatal("expected a panic") + }() + } +} + +type foo interface { + foo() foo +} + +var fooType = reflect.TypeOf((*foo)(nil)).Elem() + +func TestCheckReturnType(t *testing.T) { + for i, fn := range []interface{}{ + func() { panic("") }, + func() error { panic("") }, + func() (error, error) { panic("") }, + func() (foo, error, error) { panic("") }, + func() (foo, foo) { panic("") }, + } { + if checkReturnType(reflect.TypeOf(fn), fooType) == nil { + t.Errorf("expected falure for case %d (type %T)", i, fn) + } + } + + for i, fn := range []interface{}{ + func() foo { panic("") }, + func() (foo, error) { panic("") }, + } { + if err := checkReturnType(reflect.TypeOf(fn), fooType); err != nil { + t.Errorf("expected success for case %d (type %T), got: %s", i, fn, err) + } + } +} + +func constructFoo() foo { + return nil +} + +type fooImpl struct{} + +func (f *fooImpl) foo() foo { return nil } + +func TestCallConstructor(t *testing.T) { + _, err := callConstructor(reflect.ValueOf(constructFoo), nil) + if err == nil { + t.Fatal("expected constructor to fail") + } + + if !strings.Contains(err.Error(), "constructFoo") { + t.Errorf("expected error to contain the constructor name: %s", err) + } + + v, err := callConstructor(reflect.ValueOf(func() foo { return &fooImpl{} }), nil) + if err != nil { + t.Fatal(err) + } + if _, ok := v.(*fooImpl); !ok { + t.Fatal("expected a fooImpl") + } + + v, err = callConstructor(reflect.ValueOf(func() *fooImpl { return new(fooImpl) }), nil) + if err != nil { + t.Fatal(err) + } + if _, ok := v.(*fooImpl); !ok { + t.Fatal("expected a fooImpl") + } + + _, err = callConstructor(reflect.ValueOf(func() (*fooImpl, error) { return nil, nil }), nil) + if err == nil { + t.Fatal("expected error") + } + + v, err = callConstructor(reflect.ValueOf(func() (*fooImpl, error) { return new(fooImpl), nil }), nil) + if err != nil { + t.Fatal(err) + } + if _, ok := v.(*fooImpl); !ok { + t.Fatal("expected a fooImpl") + } +} diff --git a/config/security.go b/config/security.go new file mode 100644 index 0000000000..c3639728ff --- /dev/null +++ b/config/security.go @@ -0,0 +1,76 @@ +package config + +import ( + "fmt" + + security "github.com/libp2p/go-conn-security" + csms "github.com/libp2p/go-conn-security-multistream" + insecure "github.com/libp2p/go-conn-security/insecure" + host "github.com/libp2p/go-libp2p-host" + peer "github.com/libp2p/go-libp2p-peer" +) + +// SecC is a security transport constructor +type SecC func(h host.Host) (security.Transport, error) + +// MsSecC is a tuple containing a security transport constructor and a protocol +// ID. +type MsSecC struct { + SecC + ID string +} + +var securityArgTypes = newArgTypeSet( + hostType, networkType, peerIDType, + privKeyType, pubKeyType, pstoreType, +) + +// SecurityConstructor creates a security constructor from the passed parameter +// using reflection. +func SecurityConstructor(sec interface{}) (SecC, error) { + // Already constructed? + if t, ok := sec.(security.Transport); ok { + return func(_ host.Host) (security.Transport, error) { + return t, nil + }, nil + } + + ctor, err := makeConstructor(sec, securityType, securityArgTypes) + if err != nil { + return nil, err + } + return func(h host.Host) (security.Transport, error) { + t, err := ctor(h, nil) + if err != nil { + return nil, err + } + return t.(security.Transport), nil + }, nil +} + +func makeInsecureTransport(id peer.ID) security.Transport { + secMuxer := new(csms.SSMuxer) + secMuxer.AddTransport(insecure.ID, insecure.New(id)) + return secMuxer +} + +func makeSecurityTransport(h host.Host, tpts []MsSecC) (security.Transport, error) { + secMuxer := new(csms.SSMuxer) + transportSet := make(map[string]struct{}, len(tpts)) + for _, tptC := range tpts { + if _, ok := transportSet[tptC.ID]; ok { + return nil, fmt.Errorf("duplicate security transport: %s", tptC.ID) + } + } + for _, tptC := range tpts { + tpt, err := tptC.SecC(h) + if err != nil { + return nil, err + } + if _, ok := tpt.(*insecure.Transport); ok { + return nil, fmt.Errorf("cannot construct libp2p with an insecure transport, set the Insecure config option instead") + } + secMuxer.AddTransport(tptC.ID, tpt) + } + return secMuxer, nil +} diff --git a/config/transport.go b/config/transport.go new file mode 100644 index 0000000000..d9b4414d38 --- /dev/null +++ b/config/transport.go @@ -0,0 +1,66 @@ +package config + +import ( + host "github.com/libp2p/go-libp2p-host" + transport "github.com/libp2p/go-libp2p-transport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" +) + +// TptC is the type for libp2p transport constructors. You probably won't ever +// implement this function interface directly. Instead, pass your transport +// constructor to TransportConstructor. +type TptC func(h host.Host, u *tptu.Upgrader) (transport.Transport, error) + +var transportArgTypes = argTypes + +// TransportConstructor uses reflection to turn a function that constructs a +// transport into a TptC. +// +// You can pass either a constructed transport (something that implements +// `transport.Transport`) or a function that takes any of: +// +// * The local peer ID. +// * A transport connection upgrader. +// * A private key. +// * A public key. +// * A Host. +// * A Network. +// * A Peerstore. +// * An address filter. +// * A security transport. +// * A stream multiplexer transport. +// * A private network protector. +// +// And returns a type implementing transport.Transport and, optionally, an error +// (as the second argument). +func TransportConstructor(tpt interface{}) (TptC, error) { + // Already constructed? + if t, ok := tpt.(transport.Transport); ok { + return func(_ host.Host, _ *tptu.Upgrader) (transport.Transport, error) { + return t, nil + }, nil + } + ctor, err := makeConstructor(tpt, transportType, transportArgTypes) + if err != nil { + return nil, err + } + return func(h host.Host, u *tptu.Upgrader) (transport.Transport, error) { + t, err := ctor(h, u) + if err != nil { + return nil, err + } + return t.(transport.Transport), nil + }, nil +} + +func makeTransports(h host.Host, u *tptu.Upgrader, tpts []TptC) ([]transport.Transport, error) { + transports := make([]transport.Transport, len(tpts)) + for i, tC := range tpts { + t, err := tC(h, u) + if err != nil { + return nil, err + } + transports[i] = t + } + return transports, nil +} diff --git a/defaults.go b/defaults.go new file mode 100644 index 0000000000..d3fa618d6f --- /dev/null +++ b/defaults.go @@ -0,0 +1,109 @@ +package libp2p + +// This file contains all the default configuration options. + +import ( + "crypto/rand" + + crypto "github.com/libp2p/go-libp2p-crypto" + pstore "github.com/libp2p/go-libp2p-peerstore" + secio "github.com/libp2p/go-libp2p-secio" + tcp "github.com/libp2p/go-tcp-transport" + ws "github.com/libp2p/go-ws-transport" + mplex "github.com/whyrusleeping/go-smux-multiplex" + yamux "github.com/whyrusleeping/go-smux-yamux" +) + +// DefaultSecurity is the default security option. +// +// Useful when you want to extend, but not replace, the supported transport +// security protocols. +var DefaultSecurity = Security(secio.ID, secio.New) + +// DefaultMuxer configures libp2p to use the stream connection multiplexers. +// +// Use this option when you want to *extend* the set of multiplexers used by +// libp2p instead of replacing them. +var DefaultMuxers = ChainOptions( + Muxer("/yamux/1.0.0", yamux.DefaultTransport), + Muxer("/mplex/6.3.0", mplex.DefaultTransport), +) + +// DefaultTransports are the default libp2p transports. +// +// Use this option when you want to *extend* the set of multiplexers used by +// libp2p instead of replacing them. +var DefaultTransports = ChainOptions( + Transport(tcp.NewTCPTransport), + Transport(ws.New), +) + +// DefaultPeerstore configures libp2p to use the default peerstore. +var DefaultPeerstore Option = func(cfg *Config) error { + return cfg.Apply(Peerstore(pstore.NewPeerstore())) +} + +// RandomIdentity generates a random identity (default behaviour) +var RandomIdentity = func(cfg *Config) error { + priv, _, err := crypto.GenerateKeyPairWithReader(crypto.RSA, 2048, rand.Reader) + if err != nil { + return err + } + return cfg.Apply(Identity(priv)) +} + +// Complete list of default options and when to fallback on them. +// +// Please *DON'T* specify default options any other way. Putting this all here +// makes tracking defaults *much* easier. +var defaults = []struct { + fallback func(cfg *Config) bool + opt Option +}{ + { + fallback: func(cfg *Config) bool { return cfg.Transports == nil }, + opt: DefaultTransports, + }, + { + fallback: func(cfg *Config) bool { return cfg.Muxers == nil }, + opt: DefaultMuxers, + }, + { + fallback: func(cfg *Config) bool { return !cfg.Insecure && cfg.SecurityTransports == nil }, + opt: DefaultSecurity, + }, + { + fallback: func(cfg *Config) bool { return cfg.PeerKey == nil }, + opt: RandomIdentity, + }, + { + fallback: func(cfg *Config) bool { return cfg.Peerstore == nil }, + opt: DefaultPeerstore, + }, +} + +// Defaults configures libp2p to use the default options. Can be combined with +// other options to *extend* the default options. +var Defaults Option = func(cfg *Config) error { + for _, def := range defaults { + if err := cfg.Apply(def.opt); err != nil { + return err + } + } + return nil +} + +// FallbackDefaults applies default options to the libp2p node if and only if no +// other relevent options have been applied. will be appended to the options +// passed into New. +var FallbackDefaults Option = func(cfg *Config) error { + for _, def := range defaults { + if !def.fallback(cfg) { + continue + } + if err := cfg.Apply(def.opt); err != nil { + return err + } + } + return nil +} diff --git a/error_util.go b/error_util.go new file mode 100644 index 0000000000..86827f4eac --- /dev/null +++ b/error_util.go @@ -0,0 +1,17 @@ +package libp2p + +import ( + "fmt" + "runtime" +) + +func traceError(err error, skip int) error { + if err == nil { + return nil + } + _, file, line, ok := runtime.Caller(skip + 1) + if !ok { + return err + } + return fmt.Errorf("%s:%d: %s", file, line, err) +} diff --git a/examples/chat/chat.go b/examples/chat/chat.go index 8f211ba949..942cb1e32e 100644 --- a/examples/chat/chat.go +++ b/examples/chat/chat.go @@ -39,13 +39,13 @@ import ( mrand "math/rand" "os" + "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p-crypto" "github.com/libp2p/go-libp2p-host" "github.com/libp2p/go-libp2p-net" "github.com/libp2p/go-libp2p-peer" "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-swarm" - "github.com/libp2p/go-libp2p/p2p/host/basic" "github.com/multiformats/go-multiaddr" ) @@ -155,37 +155,27 @@ func main() { } // Creates a new RSA key pair for this host - prvKey, pubKey, err := crypto.GenerateKeyPairWithReader(crypto.RSA, 2048, r) + prvKey, _, err := crypto.GenerateKeyPairWithReader(crypto.RSA, 2048, r) if err != nil { panic(err) } - // Getting host ID from public key. - // host ID is the hash of public key - nodeID, _ := peer.IDFromPublicKey(pubKey) - // 0.0.0.0 will listen on any interface device sourceMultiAddr, _ := multiaddr.NewMultiaddr(fmt.Sprintf("/ip4/0.0.0.0/tcp/%d", *sourcePort)) - // Adding self to the peerstore. - ps := peerstore.NewPeerstore() - ps.AddPrivKey(nodeID, prvKey) - ps.AddPubKey(nodeID, pubKey) - - // Creating a new Swarm network. - network, err := swarm.NewNetwork(context.Background(), []multiaddr.Multiaddr{sourceMultiAddr}, nodeID, ps, nil) + // libp2p.New constructs a new libp2p Host. + // Other options can be added here. + host, err := libp2p.New( + context.Background(), + libp2p.ListenAddrs(sourceMultiAddr), + libp2p.Identity(prvKey), + ) if err != nil { panic(err) } - // NewHost constructs a new *BasicHost and activates it by attaching its - // stream and connection handlers to the given inet.Network (network). - // Other options like NATManager can also be added here. - // See docs: https://godoc.org/github.com/libp2p/go-libp2p/p2p/host/basic#HostOpts - host := basichost.New(network) - if *dest == "" { // Set a function as stream handler. // This function is called when a peer initiate a connection and starts a stream with this peer. diff --git a/examples/echo/main.go b/examples/echo/main.go index 412df18fc1..32d156b73e 100644 --- a/examples/echo/main.go +++ b/examples/echo/main.go @@ -49,7 +49,7 @@ func makeBasicHost(listenPort int, secio bool, randseed int64) (host.Host, error } if !secio { - opts = append(opts, libp2p.NoEncryption()) + opts = append(opts, libp2p.NoSecurity) } basicHost, err := libp2p.New(context.Background(), opts...) diff --git a/examples/http-proxy/proxy.go b/examples/http-proxy/proxy.go index f8206a8ca1..07e36b0dbe 100644 --- a/examples/http-proxy/proxy.go +++ b/examples/http-proxy/proxy.go @@ -12,16 +12,14 @@ import ( // We need to import libp2p's libraries that we use in this project. // In order to work, these libraries need to be rewritten by gx-go. - crypto "github.com/libp2p/go-libp2p-crypto" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" ps "github.com/libp2p/go-libp2p-peerstore" - swarm "github.com/libp2p/go-libp2p-swarm" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" - bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + libp2p "github.com/libp2p/go-libp2p" ) // Protocol defines the libp2p protocol that we will use for the libp2p proxy @@ -33,27 +31,11 @@ const Protocol = "/proxy-example/0.0.1" // makeRandomHost creates a libp2p host with a randomly generated identity. // This step is described in depth in other tutorials. func makeRandomHost(port int) host.Host { - priv, pub, err := crypto.GenerateKeyPair(crypto.RSA, 2048) + host, err := libp2p.New(context.Background(), libp2p.ListenAddrStrings(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port))) if err != nil { log.Fatalln(err) } - pid, err := peer.IDFromPublicKey(pub) - if err != nil { - log.Fatalln(err) - } - listen, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port)) - if err != nil { - log.Fatalln(err) - } - ps := ps.NewPeerstore() - ps.AddPrivKey(pid, priv) - ps.AddPubKey(pid, pub) - n, err := swarm.NewNetwork(context.Background(), - []ma.Multiaddr{listen}, pid, ps, nil) - if err != nil { - log.Fatalln(err) - } - return bhost.New(n) + return host } // ProxyService provides HTTP proxying on top of libp2p by launching an diff --git a/examples/multipro/main.go b/examples/multipro/main.go index 4f9e144c2a..3350ea3834 100644 --- a/examples/multipro/main.go +++ b/examples/multipro/main.go @@ -6,11 +6,9 @@ import ( "log" "math/rand" + libp2p "github.com/libp2p/go-libp2p" crypto "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" ps "github.com/libp2p/go-libp2p-peerstore" - swarm "github.com/libp2p/go-libp2p-swarm" - bhost "github.com/libp2p/go-libp2p/p2p/host/basic" ma "github.com/multiformats/go-multiaddr" ) @@ -18,14 +16,13 @@ import ( func makeRandomNode(port int, done chan bool) *Node { // Ignoring most errors for brevity // See echo example for more details and better implementation - priv, pub, _ := crypto.GenerateKeyPair(crypto.Secp256k1, 256) - pid, _ := peer.IDFromPublicKey(pub) + priv, _, _ := crypto.GenerateKeyPair(crypto.Secp256k1, 256) listen, _ := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port)) - peerStore := ps.NewPeerstore() - peerStore.AddPrivKey(pid, priv) - peerStore.AddPubKey(pid, pub) - n, _ := swarm.NewNetwork(context.Background(), []ma.Multiaddr{listen}, pid, peerStore, nil) - host := bhost.New(n) + host, _ := libp2p.New( + context.Background(), + libp2p.ListenAddrs(listen), + libp2p.Identity(priv), + ) return NewNode(host, done) } diff --git a/examples/protocol-multiplexing-with-multicodecs/main.go b/examples/protocol-multiplexing-with-multicodecs/main.go index 86c069b29c..5aa4f6fd6b 100644 --- a/examples/protocol-multiplexing-with-multicodecs/main.go +++ b/examples/protocol-multiplexing-with-multicodecs/main.go @@ -8,15 +8,11 @@ import ( "math/rand" "time" - crypto "github.com/libp2p/go-libp2p-crypto" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" ps "github.com/libp2p/go-libp2p-peerstore" - swarm "github.com/libp2p/go-libp2p-swarm" - ma "github.com/multiformats/go-multiaddr" - bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + libp2p "github.com/libp2p/go-libp2p" multicodec "github.com/multiformats/go-multicodec" json "github.com/multiformats/go-multicodec/json" ) @@ -80,17 +76,11 @@ var conversationMsgs = []string{ } func makeRandomHost(port int) host.Host { - // Ignoring most errors for brevity - // See echo example for more details and better implementation - priv, pub, _ := crypto.GenerateKeyPair(crypto.RSA, 2048) - pid, _ := peer.IDFromPublicKey(pub) - listen, _ := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port)) - ps := ps.NewPeerstore() - ps.AddPrivKey(pid, priv) - ps.AddPubKey(pid, pub) - n, _ := swarm.NewNetwork(context.Background(), - []ma.Multiaddr{listen}, pid, ps, nil) - return bhost.New(n) + h, err := libp2p.New(context.Background(), libp2p.ListenAddrStrings(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port))) + if err != nil { + panic(err) + } + return h } func main() { diff --git a/libp2p.go b/libp2p.go index bfb0c18193..936b2220d0 100644 --- a/libp2p.go +++ b/libp2p.go @@ -2,240 +2,49 @@ package libp2p import ( "context" - "crypto/rand" - "fmt" - crypto "github.com/libp2p/go-libp2p-crypto" + config "github.com/libp2p/go-libp2p/config" + host "github.com/libp2p/go-libp2p-host" - pnet "github.com/libp2p/go-libp2p-interface-pnet" - metrics "github.com/libp2p/go-libp2p-metrics" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - swarm "github.com/libp2p/go-libp2p-swarm" - transport "github.com/libp2p/go-libp2p-transport" - bhost "github.com/libp2p/go-libp2p/p2p/host/basic" - mux "github.com/libp2p/go-stream-muxer" - ma "github.com/multiformats/go-multiaddr" - mplex "github.com/whyrusleeping/go-smux-multiplex" - msmux "github.com/whyrusleeping/go-smux-multistream" - yamux "github.com/whyrusleeping/go-smux-yamux" ) // Config describes a set of settings for a libp2p node -type Config struct { - Transports []transport.Transport - Muxer mux.Transport - ListenAddrs []ma.Multiaddr - PeerKey crypto.PrivKey - Peerstore pstore.Peerstore - Protector pnet.Protector - Reporter metrics.Reporter - DisableSecio bool - EnableNAT bool -} - -type Option func(cfg *Config) error +type Config = config.Config -func Transports(tpts ...transport.Transport) Option { - return func(cfg *Config) error { - cfg.Transports = append(cfg.Transports, tpts...) - return nil - } -} +// Option is a libp2p config option that can be given to the libp2p constructor +// (`libp2p.New`). +type Option = config.Option -func ListenAddrStrings(s ...string) Option { +// ChainOptions chains multiple options into a single option. +func ChainOptions(opts ...Option) Option { return func(cfg *Config) error { - for _, addrstr := range s { - a, err := ma.NewMultiaddr(addrstr) - if err != nil { + for _, opt := range opts { + if err := opt(cfg); err != nil { return err } - cfg.ListenAddrs = append(cfg.ListenAddrs, a) - } - return nil - } -} - -func ListenAddrs(addrs ...ma.Multiaddr) Option { - return func(cfg *Config) error { - cfg.ListenAddrs = append(cfg.ListenAddrs, addrs...) - return nil - } -} - -type transportEncOpt int - -const ( - EncPlaintext = transportEncOpt(0) - EncSecio = transportEncOpt(1) -) - -func TransportEncryption(tenc ...transportEncOpt) Option { - return func(cfg *Config) error { - if len(tenc) != 1 { - return fmt.Errorf("can only specify a single transport encryption option right now") - } - - // TODO: actually make this pluggable, otherwise tls will get tricky - switch tenc[0] { - case EncPlaintext: - cfg.DisableSecio = true - case EncSecio: - // noop - default: - return fmt.Errorf("unrecognized transport encryption option: %d", tenc[0]) - } - return nil - } -} - -func NoEncryption() Option { - return TransportEncryption(EncPlaintext) -} - -func NATPortMap() Option { - return func(cfg *Config) error { - cfg.EnableNAT = true - return nil - } -} - -func Muxer(m mux.Transport) Option { - return func(cfg *Config) error { - if cfg.Muxer != nil { - return fmt.Errorf("cannot specify multiple muxer options") } - - cfg.Muxer = m - return nil - } -} - -func Peerstore(ps pstore.Peerstore) Option { - return func(cfg *Config) error { - if cfg.Peerstore != nil { - return fmt.Errorf("cannot specify multiple peerstore options") - } - - cfg.Peerstore = ps - return nil - } -} - -func PrivateNetwork(prot pnet.Protector) Option { - return func(cfg *Config) error { - if cfg.Protector != nil { - return fmt.Errorf("cannot specify multiple private network options") - } - - cfg.Protector = prot - return nil - } -} - -func BandwidthReporter(rep metrics.Reporter) Option { - return func(cfg *Config) error { - if cfg.Reporter != nil { - return fmt.Errorf("cannot specify multiple bandwidth reporter options") - } - - cfg.Reporter = rep - return nil - } -} - -func Identity(sk crypto.PrivKey) Option { - return func(cfg *Config) error { - if cfg.PeerKey != nil { - return fmt.Errorf("cannot specify multiple identities") - } - - cfg.PeerKey = sk return nil } } +// New constructs a new libp2p node with the given options (falling back on +// reasonable defaults). +// +// Canceling the passed context will stop the returned libp2p node. func New(ctx context.Context, opts ...Option) (host.Host, error) { - var cfg Config - for _, opt := range opts { - if err := opt(&cfg); err != nil { - return nil, err - } - } - - return newWithCfg(ctx, &cfg) + return NewWithoutDefaults(ctx, append(opts, FallbackDefaults)...) } -func newWithCfg(ctx context.Context, cfg *Config) (host.Host, error) { - // If no key was given, generate a random 2048 bit RSA key - if cfg.PeerKey == nil { - priv, _, err := crypto.GenerateKeyPairWithReader(crypto.RSA, 2048, rand.Reader) - if err != nil { - return nil, err - } - cfg.PeerKey = priv - } - - // Obtain Peer ID from public key - pid, err := peer.IDFromPublicKey(cfg.PeerKey.GetPublic()) - if err != nil { - return nil, err - } - - // Create a new blank peerstore if none was passed in - ps := cfg.Peerstore - if ps == nil { - ps = pstore.NewPeerstore() - } - - // Set default muxer if none was passed in - muxer := cfg.Muxer - if muxer == nil { - muxer = DefaultMuxer() - } - - // If secio is disabled, don't add our private key to the peerstore - if !cfg.DisableSecio { - ps.AddPrivKey(pid, cfg.PeerKey) - ps.AddPubKey(pid, cfg.PeerKey.GetPublic()) - } - - swrm, err := swarm.NewSwarmWithProtector(ctx, cfg.ListenAddrs, pid, ps, cfg.Protector, muxer, cfg.Reporter) - if err != nil { +// NewWithoutDefaults constructs a new libp2p node with the given options but +// *without* falling back on reasonable defaults. +// +// Warning: This function should not be considered a stable interface. We may +// choose to add required services at any time and, by using this function, you +// opt-out of any defaults we may provide. +func NewWithoutDefaults(ctx context.Context, opts ...Option) (host.Host, error) { + var cfg Config + if err := cfg.Apply(opts...); err != nil { return nil, err } - - netw := (*swarm.Network)(swrm) - - hostOpts := &bhost.HostOpts{} - - if cfg.EnableNAT { - hostOpts.NATManager = bhost.NewNATManager(netw) - } - - return bhost.NewHost(ctx, netw, hostOpts) -} - -func DefaultMuxer() mux.Transport { - // Set up stream multiplexer - tpt := msmux.NewBlankTransport() - - // By default, support yamux and multiplex - tpt.AddTransport("/yamux/1.0.0", yamux.DefaultTransport) - tpt.AddTransport("/mplex/6.3.0", mplex.DefaultTransport) - - return tpt -} - -func Defaults(cfg *Config) error { - // Create a multiaddress that listens on a random port on all interfaces - addr, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/0") - if err != nil { - return err - } - - cfg.ListenAddrs = []ma.Multiaddr{addr} - cfg.Peerstore = pstore.NewPeerstore() - cfg.Muxer = DefaultMuxer() - return nil + return cfg.NewNode(ctx) } diff --git a/libp2p_test.go b/libp2p_test.go index d2ab62e4d1..1b38bbc369 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -3,6 +3,7 @@ package libp2p import ( "context" "fmt" + "strings" "testing" crypto "github.com/libp2p/go-libp2p-crypto" @@ -10,10 +11,32 @@ import ( ) func TestNewHost(t *testing.T) { - _, err := makeRandomHost(t, 9000) + h, err := makeRandomHost(t, 9000) if err != nil { t.Fatal(err) } + h.Close() +} + +func TestBadTransportConstructor(t *testing.T) { + ctx := context.Background() + h, err := New(ctx, Transport(func() {})) + if err == nil { + h.Close() + t.Fatal("expected an error") + } + if !strings.Contains(err.Error(), "libp2p_test.go") { + t.Error("expected error to contain debugging info") + } +} + +func TestInsecure(t *testing.T) { + ctx := context.Background() + h, err := New(ctx, NoSecurity) + if err != nil { + t.Fatal(err) + } + h.Close() } func makeRandomHost(t *testing.T, port int) (host.Host, error) { @@ -26,7 +49,9 @@ func makeRandomHost(t *testing.T, port int) (host.Host, error) { opts := []Option{ ListenAddrStrings(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port)), Identity(priv), - Muxer(DefaultMuxer()), + DefaultTransports, + DefaultMuxers, + DefaultSecurity, NATPortMap(), } diff --git a/options.go b/options.go new file mode 100644 index 0000000000..a17fc02e89 --- /dev/null +++ b/options.go @@ -0,0 +1,243 @@ +package libp2p + +// This file contains all libp2p configuration options (except the defaults, +// those are in defaults.go) + +import ( + "fmt" + "net" + + config "github.com/libp2p/go-libp2p/config" + bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + + circuit "github.com/libp2p/go-libp2p-circuit" + crypto "github.com/libp2p/go-libp2p-crypto" + ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" + pnet "github.com/libp2p/go-libp2p-interface-pnet" + metrics "github.com/libp2p/go-libp2p-metrics" + pstore "github.com/libp2p/go-libp2p-peerstore" + filter "github.com/libp2p/go-maddr-filter" + ma "github.com/multiformats/go-multiaddr" +) + +// ListenAddrStrings configures libp2p to listen on the given (unparsed) +// addresses. +func ListenAddrStrings(s ...string) Option { + return func(cfg *Config) error { + for _, addrstr := range s { + a, err := ma.NewMultiaddr(addrstr) + if err != nil { + return err + } + cfg.ListenAddrs = append(cfg.ListenAddrs, a) + } + return nil + } +} + +// ListenAddrs configures libp2p to listen on the given addresses. +func ListenAddrs(addrs ...ma.Multiaddr) Option { + return func(cfg *Config) error { + cfg.ListenAddrs = append(cfg.ListenAddrs, addrs...) + return nil + } +} + +// Security configures libp2p to use the given security transport (or transport +// constructor). +// +// Name is the protocol name. +// +// The transport can be a constructed security.Transport or a function taking +// any subset of this libp2p node's: +// * Public key +// * Private key +// * Peer ID +// * Host +// * Network +// * Peerstore +func Security(name string, tpt interface{}) Option { + stpt, err := config.SecurityConstructor(tpt) + err = traceError(err, 1) + return func(cfg *Config) error { + if err != nil { + return err + } + if cfg.Insecure { + return fmt.Errorf("cannot use security transports with an insecure libp2p configuration") + } + cfg.SecurityTransports = append(cfg.SecurityTransports, config.MsSecC{SecC: stpt, ID: name}) + return nil + } +} + +// NoSecurity is an option that completely disables all transport security. +// It's incompatible with all other transport security protocols. +var NoSecurity Option = func(cfg *Config) error { + if len(cfg.SecurityTransports) > 0 { + return fmt.Errorf("cannot use security transports with an insecure libp2p configuration") + } + cfg.Insecure = true + return nil +} + +// Muxer configures libp2p to use the given stream multiplexer (or stream +// multiplexer constructor). +// +// Name is the protocol name. +// +// The transport can be a constructed mux.Transport or a function taking any +// subset of this libp2p node's: +// * Peer ID +// * Host +// * Network +// * Peerstore +func Muxer(name string, tpt interface{}) Option { + mtpt, err := config.MuxerConstructor(tpt) + err = traceError(err, 1) + return func(cfg *Config) error { + if err != nil { + return err + } + cfg.Muxers = append(cfg.Muxers, config.MsMuxC{MuxC: mtpt, ID: name}) + return nil + } +} + +// Transport configures libp2p to use the given transport (or transport +// constructor). +// +// The transport can be a constructed transport.Transport or a function taking +// any subset of this libp2p node's: +// * Transport Upgrader (*tptu.Upgrader) +// * Host +// * Stream muxer (muxer.Transport) +// * Security transport (security.Transport) +// * Private network protector (pnet.Protector) +// * Peer ID +// * Private Key +// * Public Key +// * Address filter (filter.Filter) +// * Peerstore +func Transport(tpt interface{}) Option { + tptc, err := config.TransportConstructor(tpt) + err = traceError(err, 1) + return func(cfg *Config) error { + if err != nil { + return err + } + cfg.Transports = append(cfg.Transports, tptc) + return nil + } +} + +// Peerstore configures libp2p to use the given peerstore. +func Peerstore(ps pstore.Peerstore) Option { + return func(cfg *Config) error { + if cfg.Peerstore != nil { + return fmt.Errorf("cannot specify multiple peerstore options") + } + + cfg.Peerstore = ps + return nil + } +} + +// PrivateNetwork configures libp2p to use the given private network protector. +func PrivateNetwork(prot pnet.Protector) Option { + return func(cfg *Config) error { + if cfg.Protector != nil { + return fmt.Errorf("cannot specify multiple private network options") + } + + cfg.Protector = prot + return nil + } +} + +// BandwidthReporter configures libp2p to use the given bandwidth reporter. +func BandwidthReporter(rep metrics.Reporter) Option { + return func(cfg *Config) error { + if cfg.Reporter != nil { + return fmt.Errorf("cannot specify multiple bandwidth reporter options") + } + + cfg.Reporter = rep + return nil + } +} + +// Identity configures libp2p to use the given private key to identify itself. +func Identity(sk crypto.PrivKey) Option { + return func(cfg *Config) error { + if cfg.PeerKey != nil { + return fmt.Errorf("cannot specify multiple identities") + } + + cfg.PeerKey = sk + return nil + } +} + +// ConnectionManager configures libp2p to use the given connection manager. +func ConnectionManager(connman ifconnmgr.ConnManager) Option { + return func(cfg *Config) error { + if cfg.ConnManager != nil { + return fmt.Errorf("cannot specify multiple connection managers") + } + cfg.ConnManager = connman + return nil + } +} + +// AddrsFactory configures libp2p to use the given address factory. +func AddrsFactory(factory config.AddrsFactory) Option { + return func(cfg *Config) error { + if cfg.AddrsFactory != nil { + return fmt.Errorf("cannot specify multiple address factories") + } + cfg.AddrsFactory = factory + return nil + } +} + +// EnableRelay configures libp2p to enable the relay transport. +func EnableRelay(options ...circuit.RelayOpt) Option { + return func(cfg *Config) error { + cfg.Relay = true + cfg.RelayOpts = options + return nil + } +} + +// FilterAddresses configures libp2p to never dial nor accept connections from +// the given addresses. +func FilterAddresses(addrs ...*net.IPNet) Option { + return func(cfg *Config) error { + if cfg.Filters == nil { + cfg.Filters = filter.NewFilters() + } + for _, addr := range addrs { + cfg.Filters.AddDialFilter(addr) + } + return nil + } +} + +// NATPortMap configures libp2p to use the default NATManager. The default +// NATManager will attempt to open a port in your network's firewall using UPnP. +func NATPortMap() Option { + return NATManager(bhost.NewNATManager) +} + +// NATManager will configure libp2p to use the requested NATManager. This +// function should be passed a NATManager *constructor* that takes a libp2p Network. +func NATManager(nm config.NATManagerC) Option { + return func(cfg *Config) error { + if cfg.NATManager != nil { + return fmt.Errorf("cannot specify multiple NATManagers") + } + cfg.NATManager = nm + return nil + } +} diff --git a/p2p/discovery/mdns_test.go b/p2p/discovery/mdns_test.go index 579fc5bf69..bb0e84d6f1 100644 --- a/p2p/discovery/mdns_test.go +++ b/p2p/discovery/mdns_test.go @@ -8,7 +8,7 @@ import ( bhost "github.com/libp2p/go-libp2p/p2p/host/basic" host "github.com/libp2p/go-libp2p-host" - netutil "github.com/libp2p/go-libp2p-netutil" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" pstore "github.com/libp2p/go-libp2p-peerstore" ) @@ -28,8 +28,8 @@ func TestMdnsDiscovery(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - a := bhost.New(netutil.GenSwarmNetwork(t, ctx)) - b := bhost.New(netutil.GenSwarmNetwork(t, ctx)) + a := bhost.New(swarmt.GenSwarm(t, ctx)) + b := bhost.New(swarmt.GenSwarm(t, ctx)) sa, err := NewMdnsService(ctx, a, time.Second, "someTag") if err != nil { diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 0d566f433a..018f2691d8 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -9,10 +9,8 @@ import ( logging "github.com/ipfs/go-log" goprocess "github.com/jbenet/goprocess" - circuit "github.com/libp2p/go-libp2p-circuit" + goprocessctx "github.com/jbenet/goprocess/context" ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" - metrics "github.com/libp2p/go-libp2p-metrics" - mstream "github.com/libp2p/go-libp2p-metrics/stream" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" @@ -67,8 +65,6 @@ type BasicHost struct { negtimeout time.Duration proc goprocess.Process - - bwc metrics.Reporter } // HostOpts holds options that can be passed to NewHost in order to @@ -97,25 +93,14 @@ type HostOpts struct { // NATManager takes care of setting NAT port mappings, and discovering external addresses. // If omitted, this will simply be disabled. - NATManager NATManager - - // BandwidthReporter is used for collecting aggregate metrics of the - // bandwidth used by various protocols. - BandwidthReporter metrics.Reporter + NATManager func(inet.Network) NATManager // ConnManager is a libp2p connection manager ConnManager ifconnmgr.ConnManager - - // Relay indicates whether the host should use circuit relay transport - EnableRelay bool - - // RelayOpts are options for the relay transport; only meaningful when Relay=true - RelayOpts []circuit.RelayOpt } // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, error) { - ctx, cancel := context.WithCancel(ctx) h := &BasicHost{ network: net, mux: msmux.NewMultistreamMuxer(), @@ -124,11 +109,10 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, maResolver: madns.DefaultResolver, } - h.proc = goprocess.WithTeardown(func() error { + h.proc = goprocessctx.WithContextAndTeardown(ctx, func() error { if h.natmgr != nil { h.natmgr.Close() } - cancel() return h.Network().Close() }) @@ -152,18 +136,13 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, } if opts.NATManager != nil { - h.natmgr = opts.NATManager + h.natmgr = opts.NATManager(net) } if opts.MultiaddrResolver != nil { h.maResolver = opts.MultiaddrResolver } - if opts.BandwidthReporter != nil { - h.bwc = opts.BandwidthReporter - h.ids.Reporter = opts.BandwidthReporter - } - if opts.ConnManager == nil { h.cmgr = &ifconnmgr.NullConnMgr{} } else { @@ -173,20 +152,16 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, net.SetConnHandler(h.newConnHandler) net.SetStreamHandler(h.newStreamHandler) - - if opts.EnableRelay { - err := circuit.AddRelayTransport(ctx, h, opts.RelayOpts...) - if err != nil { - h.Close() - return nil, err - } - } - return h, nil } // New constructs and sets up a new *BasicHost with given Network and options. -// Three options can be passed: NATPortMap, AddrsFactory, and metrics.Reporter. +// The following options can be passed: +// * NATPortMap +// * AddrsFactory +// * ifconnmgr.ConnManager +// * madns.Resolver +// // This function is deprecated in favor of NewHost and HostOpts. func New(net inet.Network, opts ...interface{}) *BasicHost { hostopts := &HostOpts{} @@ -196,10 +171,8 @@ func New(net inet.Network, opts ...interface{}) *BasicHost { case Option: switch o { case NATPortMap: - hostopts.NATManager = newNatManager(net) + hostopts.NATManager = NewNATManager } - case metrics.Reporter: - hostopts.BandwidthReporter = o case AddrsFactory: hostopts.AddrsFactory = AddrsFactory(o) case ifconnmgr.ConnManager: @@ -270,10 +243,6 @@ func (h *BasicHost) newStreamHandler(s inet.Stream) { } s.SetProtocol(protocol.ID(protoID)) - - if h.bwc != nil { - s = mstream.WrapStream(s, h.bwc) - } log.Debugf("protocol negotiation took %s", took) go handle(protoID, s) @@ -366,10 +335,6 @@ func (h *BasicHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol.I s.SetProtocol(selpid) h.Peerstore().AddProtocols(p, selected) - if h.bwc != nil { - s = mstream.WrapStream(s, h.bwc) - } - return s, nil } @@ -403,10 +368,6 @@ func (h *BasicHost) newStream(ctx context.Context, p peer.ID, pid protocol.ID) ( s.SetProtocol(pid) - if h.bwc != nil { - s = mstream.WrapStream(s, h.bwc) - } - lzcon := msmux.NewMSSelect(s, string(pid)) return &streamWrapper{ Stream: s, @@ -536,11 +497,6 @@ func (h *BasicHost) Close() error { return h.proc.Close() } -// GetBandwidthReporter exposes the Host's bandiwth metrics reporter -func (h *BasicHost) GetBandwidthReporter() metrics.Reporter { - return h.bwc -} - type streamWrapper struct { inet.Stream rw io.ReadWriter diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index 7cf8136e62..27c81d552d 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -8,11 +8,13 @@ import ( "testing" "time" + testutil "github.com/libp2p/go-testutil" + host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" - testutil "github.com/libp2p/go-libp2p-netutil" pstore "github.com/libp2p/go-libp2p-peerstore" protocol "github.com/libp2p/go-libp2p-protocol" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" madns "github.com/multiformats/go-multiaddr-dns" ) @@ -20,8 +22,8 @@ import ( func TestHostSimple(t *testing.T) { ctx := context.Background() - h1 := New(testutil.GenSwarmNetwork(t, ctx)) - h2 := New(testutil.GenSwarmNetwork(t, ctx)) + h1 := New(swarmt.GenSwarm(t, ctx)) + h2 := New(swarmt.GenSwarm(t, ctx)) defer h1.Close() defer h2.Close() @@ -74,7 +76,7 @@ func TestHostAddrsFactory(t *testing.T) { } ctx := context.Background() - h := New(testutil.GenSwarmNetwork(t, ctx), AddrsFactory(addrsFactory)) + h := New(swarmt.GenSwarm(t, ctx), AddrsFactory(addrsFactory)) defer h.Close() addrs := h.Addrs() @@ -87,8 +89,8 @@ func TestHostAddrsFactory(t *testing.T) { } func getHostPair(ctx context.Context, t *testing.T) (host.Host, host.Host) { - h1 := New(testutil.GenSwarmNetwork(t, ctx)) - h2 := New(testutil.GenSwarmNetwork(t, ctx)) + h1 := New(swarmt.GenSwarm(t, ctx)) + h2 := New(swarmt.GenSwarm(t, ctx)) h2pi := h2.Peerstore().PeerInfo(h2.ID()) if err := h1.Connect(ctx, h2pi); err != nil { @@ -193,8 +195,8 @@ func TestHostProtoPreknowledge(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - h1 := New(testutil.GenSwarmNetwork(t, ctx)) - h2 := New(testutil.GenSwarmNetwork(t, ctx)) + h1 := New(swarmt.GenSwarm(t, ctx)) + h2 := New(swarmt.GenSwarm(t, ctx)) conn := make(chan protocol.ID) handler := func(s inet.Stream) { @@ -358,7 +360,7 @@ func TestAddrResolution(t *testing.T) { } resolver := &madns.Resolver{Backend: backend} - h := New(testutil.GenSwarmNetwork(t, ctx), resolver) + h := New(swarmt.GenSwarm(t, ctx), resolver) defer h.Close() pi, err := pstore.InfoFromP2pAddr(p2paddr1) diff --git a/p2p/net/mock/mock_conn.go b/p2p/net/mock/mock_conn.go index fcb634624c..68e9c840cc 100644 --- a/p2p/net/mock/mock_conn.go +++ b/p2p/net/mock/mock_conn.go @@ -118,12 +118,12 @@ func (c *conn) NewStream() (inet.Stream, error) { return s, nil } -func (c *conn) GetStreams() ([]inet.Stream, error) { +func (c *conn) GetStreams() []inet.Stream { var out []inet.Stream for e := c.streams.Front(); e != nil; e = e.Next() { out = append(out, e.Value.(*stream)) } - return out, nil + return out } // LocalMultiaddr is the Multiaddr on this side diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 66930a3b46..8f8394fbd1 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -13,8 +13,6 @@ import ( ic "github.com/libp2p/go-libp2p-crypto" host "github.com/libp2p/go-libp2p-host" lgbl "github.com/libp2p/go-libp2p-loggables" - metrics "github.com/libp2p/go-libp2p-metrics" - mstream "github.com/libp2p/go-libp2p-metrics/stream" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" @@ -44,7 +42,6 @@ var ClientVersion = "go-libp2p/3.3.4" type IDService struct { Host host.Host - Reporter metrics.Reporter // connections undergoing identification // for wait purposes currid map[inet.Conn]chan struct{} @@ -64,7 +61,7 @@ func NewIDService(h host.Host) *IDService { Host: h, currid: make(map[inet.Conn]chan struct{}), } - h.SetStreamHandler(ID, s.RequestHandler) + h.SetStreamHandler(ID, s.requestHandler) h.Network().Notify((*netNotifiee)(s)) return s } @@ -95,21 +92,17 @@ func (ids *IDService) IdentifyConn(c inet.Conn) { c.Close() return } - defer s.Close() + defer inet.FullClose(s) s.SetProtocol(ID) - if ids.Reporter != nil { - s = mstream.WrapStream(s, ids.Reporter) - } - // ok give the response to our handler. if err := msmux.SelectProtoOrFail(ID, s); err != nil { log.Event(context.TODO(), "IdentifyOpenFailed", c.RemotePeer(), logging.Metadata{"error": err}) return } - ids.ResponseHandler(s) + ids.responseHandler(s) ids.currmu.Lock() _, found := ids.currid[c] @@ -122,14 +115,10 @@ func (ids *IDService) IdentifyConn(c inet.Conn) { } } -func (ids *IDService) RequestHandler(s inet.Stream) { - defer s.Close() +func (ids *IDService) requestHandler(s inet.Stream) { + defer inet.FullClose(s) c := s.Conn() - if ids.Reporter != nil { - s = mstream.WrapStream(s, ids.Reporter) - } - w := ggio.NewDelimitedWriter(s) mes := pb.Identify{} ids.populateMessage(&mes, s.Conn()) @@ -139,8 +128,7 @@ func (ids *IDService) RequestHandler(s inet.Stream) { c.RemotePeer(), c.RemoteMultiaddr()) } -func (ids *IDService) ResponseHandler(s inet.Stream) { - defer s.Close() +func (ids *IDService) responseHandler(s inet.Stream) { c := s.Conn() r := ggio.NewDelimitedReader(s, 2048) diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index 7e82f9528c..79a09d2f9b 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -6,9 +6,9 @@ import ( "time" ic "github.com/libp2p/go-libp2p-crypto" - testutil "github.com/libp2p/go-libp2p-netutil" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" identify "github.com/libp2p/go-libp2p/p2p/protocol/identify" blhost "github.com/libp2p/go-libp2p-blankhost" @@ -20,8 +20,8 @@ func subtestIDService(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - h1 := blhost.NewBlankHost(testutil.GenSwarmNetwork(t, ctx)) - h2 := blhost.NewBlankHost(testutil.GenSwarmNetwork(t, ctx)) + h1 := blhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h2 := blhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) h1p := h1.ID() h2p := h2.ID() @@ -65,7 +65,7 @@ func subtestIDService(t *testing.T) { ids2.IdentifyConn(c[0]) addrs := h1.Peerstore().Addrs(h1p) - addrs = append(addrs, c[0].RemoteMultiaddr(), forgetMe) + addrs = append(addrs, forgetMe) // and the protocol versions. t.Log("test peer2 has peer1 addrs correctly") diff --git a/p2p/protocol/ping/ping_test.go b/p2p/protocol/ping/ping_test.go index a260cb81d4..adcb3ca696 100644 --- a/p2p/protocol/ping/ping_test.go +++ b/p2p/protocol/ping/ping_test.go @@ -5,17 +5,17 @@ import ( "testing" "time" - netutil "github.com/libp2p/go-libp2p-netutil" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" ) func TestPing(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - h1 := bhost.New(netutil.GenSwarmNetwork(t, ctx)) - h2 := bhost.New(netutil.GenSwarmNetwork(t, ctx)) + h1 := bhost.New(swarmt.GenSwarm(t, ctx)) + h2 := bhost.New(swarmt.GenSwarm(t, ctx)) err := h1.Connect(ctx, pstore.PeerInfo{ ID: h2.ID(), diff --git a/p2p/test/backpressure/backpressure_test.go b/p2p/test/backpressure/backpressure_test.go index 8e783118db..ab84ab2ab5 100644 --- a/p2p/test/backpressure/backpressure_test.go +++ b/p2p/test/backpressure/backpressure_test.go @@ -13,9 +13,9 @@ import ( logging "github.com/ipfs/go-log" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" - testutil "github.com/libp2p/go-libp2p-netutil" peer "github.com/libp2p/go-libp2p-peer" protocol "github.com/libp2p/go-libp2p-protocol" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" ) var log = logging.Logger("backpressure") @@ -137,8 +137,8 @@ a problem. // ok that's enough setup. let's do it! ctx := context.Background() - h1 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) - h2 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) + h1 := bhost.New(swarmt.GenSwarm(t, ctx)) + h2 := bhost.New(swarmt.GenSwarm(t, ctx)) // setup receiver handler h1.SetStreamHandler(protocol.TestingID, receiver) @@ -274,8 +274,8 @@ func TestStBackpressureStreamWrite(t *testing.T) { // setup the networks ctx := context.Background() - h1 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) - h2 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) + h1 := bhost.New(swarmt.GenSwarm(t, ctx)) + h2 := bhost.New(swarmt.GenSwarm(t, ctx)) // setup sender handler on 1 h1.SetStreamHandler(protocol.TestingID, sender) diff --git a/p2p/test/reconnects/reconnect_test.go b/p2p/test/reconnects/reconnect_test.go index e8851efe8c..2f721c3609 100644 --- a/p2p/test/reconnects/reconnect_test.go +++ b/p2p/test/reconnects/reconnect_test.go @@ -14,17 +14,10 @@ import ( logging "github.com/ipfs/go-log" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" - testutil "github.com/libp2p/go-libp2p-netutil" protocol "github.com/libp2p/go-libp2p-protocol" - swarm "github.com/libp2p/go-libp2p-swarm" - ps "github.com/libp2p/go-peerstream" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" ) -func init() { - // change the garbage collect timeout for testing. - ps.GarbageCollectTimeout = 10 * time.Millisecond -} - var log = logging.Logger("reconnect") func EchoStreamHandler(stream inet.Stream) { @@ -109,8 +102,8 @@ func newSender() (chan sendChans, func(s inet.Stream)) { // TestReconnect tests whether hosts are able to disconnect and reconnect. func TestReconnect2(t *testing.T) { ctx := context.Background() - h1 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) - h2 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) + h1 := bhost.New(swarmt.GenSwarm(t, ctx)) + h2 := bhost.New(swarmt.GenSwarm(t, ctx)) hosts := []host.Host{h1, h2} h1.SetStreamHandler(protocol.TestingID, EchoStreamHandler) @@ -129,11 +122,11 @@ func TestReconnect2(t *testing.T) { // TestReconnect tests whether hosts are able to disconnect and reconnect. func TestReconnect5(t *testing.T) { ctx := context.Background() - h1 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) - h2 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) - h3 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) - h4 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) - h5 := bhost.New(testutil.GenSwarmNetwork(t, ctx)) + h1 := bhost.New(swarmt.GenSwarm(t, ctx)) + h2 := bhost.New(swarmt.GenSwarm(t, ctx)) + h3 := bhost.New(swarmt.GenSwarm(t, ctx)) + h4 := bhost.New(swarmt.GenSwarm(t, ctx)) + h5 := bhost.New(swarmt.GenSwarm(t, ctx)) hosts := []host.Host{h1, h2, h3, h4, h5} h1.SetStreamHandler(protocol.TestingID, EchoStreamHandler) @@ -219,13 +212,11 @@ func SubtestConnSendDisc(t *testing.T, hosts []host.Host) { // close connection cs := h1.Network().Conns() for _, c := range cs { - sc := c.(*swarm.Conn) - if sc.LocalPeer() > sc.RemotePeer() { - continue // only close it on one side. + if c.LocalPeer() > c.RemotePeer() { + continue } - - log.Debugf("closing: %s", sc.RawConn()) - sc.Close() + log.Debugf("closing: %s", c) + c.Close() } } diff --git a/package.json b/package.json index f70b3baafb..98fd293af0 100644 --- a/package.json +++ b/package.json @@ -18,21 +18,11 @@ "name": "mdns", "version": "0.1.1" }, - { - "hash": "QmWBug6eBS7AxRdCDVuSY5CnSit7cS2XnPFYJWqWDumhCG", - "name": "go-msgio", - "version": "0.0.3" - }, { "hash": "QmNiJuT8Ja3hMVpBHXv3Q6dwmperaQ6JjLtpMQgMCD7xvx", "name": "go-ipfs-util", "version": "1.2.7" }, - { - "hash": "QmUusaX99BZoELh7dmPgirqRQ1FAmMnmnBn3oiqDFGBUSc", - "name": "go-keyspace", - "version": "1.0.0" - }, { "hash": "QmbXRda5H2K3MSQyWWxTMtd8DWuguEBUCe6hpxfXVpFUGj", "name": "go-multistream", @@ -54,24 +44,9 @@ "version": "1.4.1" }, { - "hash": "QmRK2LxanhK2gZq6k6R7vk5ZoYZk8ULSSTB7FzDsMUX6CB", + "hash": "QmcGXGdw9BWDysPJQHxJinjGHha3eEg4vzFETre4woNwcX", "name": "go-multiaddr-net", - "version": "1.5.7" - }, - { - "hash": "QmZyZDi491cCNTLfAhwcaDii2Kg4pwKRkhqQzURGDvY6ua", - "name": "go-multihash", - "version": "1.0.7" - }, - { - "hash": "QmSMZwvs3n4GBikZ7hKzT17c3bk65FmyZo2JqtJ16swqCv", - "name": "multiaddr-filter", - "version": "1.0.2" - }, - { - "hash": "QmaPHkZLbQQbvcyavn8q1GFHg6o6yeceyHFSJ3Pjf3p3TQ", - "name": "go-crypto", - "version": "0.0.0" + "version": "1.6.0" }, { "hash": "QmZ4Qi3GaRbjcx28Sme5eMH7RQjGkt8wHxt2a65oLaeFEV", @@ -83,38 +58,6 @@ "name": "go-multiaddr", "version": "1.2.6" }, - { - "hash": "QmYvsG72GsfLgUeSojXArjnU6L4Wmwk7wuAxtNLuyXcc1T", - "name": "randbo", - "version": "0.0.0" - }, - { - "hash": "QmeQW4ayVqi7Jjay1SrP2wYydsH9KwSrzQBnqyC25gPFnG", - "name": "go-notifier", - "version": "1.0.0" - }, - { - "hash": "QmWHgLqrghM9zw77nF6gdvT9ExQ2RB9pLxkd8sDHZf1rWb", - "name": "go-temp-err-catcher", - "version": "0.0.0" - }, - { - "hash": "QmVXXxPsnDY16szK4gPy1oz4qKd8HHshemX1miZR2frtJo", - "name": "go-peerstream", - "version": "2.1.5" - }, - { - "author": "whyrusleeping", - "hash": "QmTy17Jm1foTnvUS9JXRhLbRQ3XuC64jPTjUfpB4mHz2QM", - "name": "mafmt", - "version": "1.2.5" - }, - { - "author": "whyrusleeping", - "hash": "QmTd4Jgb4nbJq5uR55KJgGLyHWmM3dovS21D1HcwRneSLu", - "name": "gorocheck", - "version": "0.0.0" - }, { "author": "whyrusleeping", "hash": "QmPDZJxtWGfcwLPazJxD4h3v3aDs43V7UNAVs3Jz1Wo7o4", @@ -123,33 +66,33 @@ }, { "author": "whyrusleeping", - "hash": "QmP47neqyP4NR9CKbjVogZ8U9Gybxfcfsa8HtPSPSxwiA8", + "hash": "QmQS7P1VV4JuqbXEEaPQZ5ERHDQuGp8qk26Gfdg9ZtP1Eb", "name": "go-libp2p-secio", - "version": "1.2.7" + "version": "2.0.0" }, { "author": "whyrusleeping", - "hash": "QmdeiKhUy1TVGBaKxt7y1QmBDLBdisSrLJ1x58Eoj4PXUh", + "hash": "QmZb7hAgQEhW9dBbzBudU39gCeD4zbe6xafD52LUuF4cUN", "name": "go-libp2p-peerstore", - "version": "1.4.17" + "version": "1.4.18" }, { "author": "whyrusleeping", - "hash": "QmPUHzTLPZFYqv8WqcBTuMFYTgeom4uHHEaxzk7bd5GYZB", + "hash": "QmYnjSGtvn7LhrxCvwrU9uDWxKyg28uBYeXvgzTDDDzVy4", "name": "go-libp2p-transport", - "version": "2.2.14" + "version": "3.0.0" }, { "author": "whyrusleeping", - "hash": "QmdxKHpkZCTV3C7xdE1iJdPfFm5LVvMPvirdFmKu1TimzY", + "hash": "QmU7NiV3ZRJoNk8XZEKazs4AW7XQT1rQCrhh8cmhSjZgrC", "name": "go-tcp-transport", - "version": "1.2.9" + "version": "2.0.0" }, { "author": "whyrusleeping", - "hash": "Qmf2UAmRwDG4TvnkQpHZWPAzw7rpCYVhxmRXmYxXr5LD1g", + "hash": "QmYF6XMumtqvXtZfQKrCokXEdFvfy2i7tbpvSRzRrHyY8c", "name": "go-maddr-filter", - "version": "1.1.6" + "version": "1.1.7" }, { "author": "whyrusleeping", @@ -157,12 +100,6 @@ "name": "go-libp2p-protocol", "version": "1.0.0" }, - { - "author": "whyrusleeping", - "hash": "QmTGSre9j1otFgsr1opCUQDXTPSM6BTZnMWwPeA5nYJM7w", - "name": "go-addr-util", - "version": "1.2.7" - }, { "author": "whyrusleeping", "hash": "QmUJzxQQ2kzwQubsMqBTr1NGDpLfh7pGA2E1oaJULcKDPq", @@ -171,57 +108,45 @@ }, { "author": "whyrusleeping", - "hash": "QmYSQpi68jBLVUx62u543RVvnjjaQQDDxyopWWG31kiUkG", - "name": "go-libp2p-conn", - "version": "1.7.7" - }, - { - "author": "whyrusleeping", - "hash": "QmXoz9o2PT3tEzf7hicegwex5UgVP54n3k82K7jrWFyN86", + "hash": "QmYj8wdn5sZEHX2XMDWGBvcXJNdzVbaVpHmXvhHBVZepen", "name": "go-libp2p-net", - "version": "2.0.7" + "version": "3.0.0" }, { "author": "whyrusleeping", - "hash": "QmVvu4bS5QLfS19ePkp5Wgzn2ZUma5oXTT9BgDFyQLxUZF", + "hash": "QmNnaGds3p4hfTqSH4KURKh8pBRcisAWYbNDEGeMZ7c3Hv", "name": "go-libp2p-metrics", - "version": "2.0.6" + "version": "2.1.0" }, { "author": "whyrusleeping", - "hash": "QmYDNqBAMWVMHKndYR35Sd8PfEVWBiDmpHYkuRJTunJDeJ", - "name": "go-libp2p-interface-conn", - "version": "0.4.13" - }, - { - "author": "whyrusleeping", - "hash": "QmaSfSMvc1VPZ8JbMponFs4WHvF9FgEruF56opm5E1RgQA", + "hash": "QmdHyfNVTZ5VtUx4Xz23z8wtnioSrFQ28XSfpVkdhQBkGA", "name": "go-libp2p-host", - "version": "2.1.8" + "version": "3.0.0" }, { "author": "whyrusleeping", - "hash": "QmRpKdg1xs4Yyrn9yrVYRBp7AQqyRxMLpD6Jgp1eZAGqEr", + "hash": "QmPzT3rJnSP8VFP1kw7Ly7HP8AprKNZtwLHXHnxfVSbWT3", "name": "go-libp2p-swarm", - "version": "2.1.9" + "version": "3.0.0" }, { "author": "whyrusleeping", - "hash": "QmXtFH52dAPCq5i4iYjr1g8xVFVJD3fwKWWyNHjVB4sHRp", + "hash": "QmQF7htcTXeVqdTg4fKPGU59PeeTpsFgn9UquiixwbTPG1", "name": "go-libp2p-nat", - "version": "0.0.8" + "version": "0.8.0" }, { "author": "whyrusleeping", - "hash": "Qma2UuHusnaFV24DgeZ5hyrM9uc4UdyVaZbtn2FQsPRhES", + "hash": "Qmb3r9qUR7PnkyUKztmXp8sQhzXZHGmRg7fR5zsB1ebWMj", "name": "go-libp2p-netutil", - "version": "0.3.13" + "version": "0.4.0" }, { "author": "whyrusleeping", - "hash": "QmYEmPwCBe7ZUFfuymozopHTuF3JXejvJPDAjwtyQCrsDi", + "hash": "QmQ6ASb73YCy77TLfxzKnzQFUyFKMQzDhmjwjaQp6rxK34", "name": "go-libp2p-blankhost", - "version": "0.2.8" + "version": "0.3.0" }, { "author": "whyrusleeping", @@ -255,9 +180,9 @@ }, { "author": "vyzo", - "hash": "QmR5sXZi68rm9m2E3KiXj6hE5m3GeLaDjbLPUeV6W3MLR8", + "hash": "QmNXLcLAcfo8yp59FxFQJNa7pDbUUw97QN9GwefWWFK4hk", "name": "go-libp2p-circuit", - "version": "2.0.14" + "version": "2.1.0" }, { "author": "lgierth", @@ -267,9 +192,9 @@ }, { "author": "why", - "hash": "QmfQNieWBPwmnUjXWPZbjJPzhNwFFabTb5RQ79dyVWGujQ", + "hash": "QmWCWsDQnnQ9Mo9V3GK8TSR91662FdFxjjqPX8YbHC8Ltz", "name": "go-libp2p-interface-connmgr", - "version": "0.0.8" + "version": "0.0.7" }, { "author": "whyrusleeping", @@ -288,6 +213,42 @@ "hash": "QmcBWojPoNh4qm7zvv4qiepvCnnc7ALS9qcp7TNwwxT1gT", "name": "go.uuid", "version": "1.1.0" + }, + { + "author": "whyrusleeping", + "hash": "QmTeRSFgnXRCh13sxsZkLTVCc1diUbZiT5mkGUgkR1J1on", + "name": "go-ws-transport", + "version": "2.0.0" + }, + { + "author": "stebalien", + "hash": "QmRYk8zWrXSkXsE16vM8yxByqM6eVvnXzDXKGvHFJJubVc", + "name": "go-conn-security-multistream", + "version": "0.1.0" + }, + { + "author": "Stebalien", + "hash": "QmSieFUauuYnroStqmRAEgu9BMXDNY5LbtNgzXcFitBKXQ", + "name": "go-conn-security", + "version": "0.1.1" + }, + { + "author": "libp2p", + "hash": "QmW7Ump7YyBMr712Ta3iEVh3ZYcfVvJaPryfbCnyE826b4", + "name": "go-libp2p-interface-pnet", + "version": "3.0.0" + }, + { + "author": "whyrusleeping", + "hash": "QmY9JXR3FupnYAYJWK9aMr9bCpqWKcToQ1tz8DVGTrHpHw", + "name": "go-stream-muxer", + "version": "3.0.0" + }, + { + "author": "steb", + "hash": "Qmf3ejfGWR8Bd3wKFBvwYGFMJ9TeKJwYJUc2WchXjMxzg7", + "name": "go-libp2p-transport-upgrader", + "version": "0.1.0" } ], "gxVersion": "0.4.0", From a612c7ddee0c24de1f47bfca4236ac186865f955 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 4 Jun 2018 21:23:01 -0700 Subject: [PATCH 0474/3965] gx publish 6.0.0 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index ce1a045622..70da42f40f 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -5.0.21: QmY6iAoG9DVgZwh5ZRcQEpa2uErAe1Hbei8qXPCjpDS9Ge +6.0.0: QmRvoAami8AAf5Yy6jcPq5KqQT1ZCaoi9dF1vdKAghmq9X diff --git a/package.json b/package.json index 98fd293af0..60e39dbd9f 100644 --- a/package.json +++ b/package.json @@ -256,6 +256,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "5.0.21" + "version": "6.0.0" } From 34c0789b9de1128edfbd225c5c5c1864dd7e4bd6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 18 Jan 2018 13:31:54 -0800 Subject: [PATCH 0475/3965] wrap net conns, not transport conns We now "protect" below the transport layer. Also, make the tests work without the dummy conn package (one fewer deps). --- p2p/net/pnet/protector.go | 4 ++-- p2p/net/pnet/psk_conn.go | 8 ++++---- p2p/net/pnet/psk_conn_test.go | 36 +++++++++++++++++++++-------------- 3 files changed, 28 insertions(+), 20 deletions(-) diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index 72446c2c6e..38706817ab 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -3,9 +3,9 @@ package pnet import ( "fmt" "io" + "net" ipnet "github.com/libp2p/go-libp2p-interface-pnet" - tconn "github.com/libp2p/go-libp2p-transport" ) var _ ipnet.Protector = (*protector)(nil) @@ -33,7 +33,7 @@ type protector struct { fingerprint []byte } -func (p protector) Protect(in tconn.Conn) (tconn.Conn, error) { +func (p protector) Protect(in net.Conn) (net.Conn, error) { return newPSKConn(p.psk, in) } func (p protector) Fingerprint() []byte { diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index b9fc750b3c..18dbada1fe 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -4,10 +4,10 @@ import ( "crypto/cipher" "crypto/rand" "io" + "net" salsa20 "github.com/davidlazar/go-crypto/salsa20" ipnet "github.com/libp2p/go-libp2p-interface-pnet" - tconn "github.com/libp2p/go-libp2p-transport" mpool "github.com/libp2p/go-msgio/mpool" ) @@ -20,7 +20,7 @@ var ( ) type pskConn struct { - tconn.Conn + net.Conn psk *[32]byte writeS20 cipher.Stream @@ -73,9 +73,9 @@ func (c *pskConn) Write(in []byte) (int, error) { return c.Conn.Write(out) // send } -var _ tconn.Conn = (*pskConn)(nil) +var _ net.Conn = (*pskConn)(nil) -func newPSKConn(psk *[32]byte, insecure tconn.Conn) (tconn.Conn, error) { +func newPSKConn(psk *[32]byte, insecure net.Conn) (net.Conn, error) { if insecure == nil { return nil, errInsecureNil } diff --git a/p2p/net/pnet/psk_conn_test.go b/p2p/net/pnet/psk_conn_test.go index c76f474af7..23108f4072 100644 --- a/p2p/net/pnet/psk_conn_test.go +++ b/p2p/net/pnet/psk_conn_test.go @@ -4,19 +4,14 @@ import ( "bytes" "context" "math/rand" + "net" "testing" - - dconn "github.com/libp2p/go-libp2p-dummy-conn" - tconn "github.com/libp2p/go-libp2p-transport" ) var testPSK = [32]byte{} // null bytes are as good test key as any other key -func setupPSKConns(ctx context.Context, t *testing.T) (tconn.Conn, tconn.Conn) { - conn1, conn2, err := dconn.NewDummyConnPair() - if err != nil { - t.Fatal(err) - } +func setupPSKConns(ctx context.Context, t *testing.T) (net.Conn, net.Conn) { + conn1, conn2 := net.Pipe() psk1, err := newPSKConn(&testPSK, conn1) if err != nil { @@ -37,14 +32,21 @@ func TestPSKSimpelMessges(t *testing.T) { msg1 := []byte("hello world") out1 := make([]byte, len(msg1)) - _, err := psk1.Write(msg1) + wch := make(chan error) + go func() { + _, err := psk1.Write(msg1) + wch <- err + }() + n, err := psk2.Read(out1) if err != nil { t.Fatal(err) } - n, err := psk2.Read(out1) + + err = <-wch if err != nil { t.Fatal(err) } + if n != len(out1) { t.Fatalf("expected to read %d bytes, read: %d", len(out1), n) } @@ -68,10 +70,11 @@ func TestPSKFragmentation(t *testing.T) { out := make([]byte, 100) - _, err = psk1.Write(in) - if err != nil { - t.Fatal(err) - } + wch := make(chan error) + go func() { + _, err := psk1.Write(in) + wch <- err + }() for i := 0; i < 10; i++ { _, err = psk2.Read(out) @@ -80,4 +83,9 @@ func TestPSKFragmentation(t *testing.T) { } in = in[100:] } + + err = <-wch + if err != nil { + t.Fatal(err) + } } From bb9d55ac083a1edf89b386caa6ad3e784e8c63e3 Mon Sep 17 00:00:00 2001 From: Erin Swenson-Healey Date: Tue, 5 Jun 2018 17:35:18 -0700 Subject: [PATCH 0476/3965] add test which demonstrates how Mocknet latency works --- p2p/net/mock/mock_test.go | 53 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index 09f480c8ee..f2f6795fee 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -582,3 +582,56 @@ func TestLimitedStreams(t *testing.T) { t.Fatal("Expected 2ish seconds but got ", time.Since(before)) } } + +func TestStreamsWithLatency(t *testing.T) { + latency := time.Millisecond * 500 + + mn, err := WithNPeers(context.Background(), 2) + if err != nil { + t.Fatal(err) + } + + // configure the Mocknet with some latency and link/connect its peers + mn.SetLinkDefaults(LinkOptions{Latency: latency}) + mn.LinkAll() + mn.ConnectAllButSelf() + + msg := []byte("ping") + mln := len(msg) + + var wg sync.WaitGroup + + // we'll write once to a single stream + wg.Add(1) + + handler := func(s inet.Stream) { + b := make([]byte, mln) + + if _, err := io.ReadFull(s, b); err != nil { + t.Fatal(err) + } + + wg.Done() + s.Close() + } + + mn.Hosts()[0].SetStreamHandler(protocol.TestingID, handler) + mn.Hosts()[1].SetStreamHandler(protocol.TestingID, handler) + + s, err := mn.Hosts()[0].NewStream(context.Background(), mn.Hosts()[1].ID(), protocol.TestingID) + if err != nil { + t.Fatal(err) + } + + // writing to the stream will be subject to our configured latency + checkpoint := time.Now() + if _, err := s.Write(msg); err != nil { + t.Fatal(err) + } + wg.Wait() + + delta := time.Since(checkpoint) + if !within(delta, latency, time.Millisecond*50) { + t.Fatalf("Expected write to take ~%s, but took %s", latency.String(), delta.String()) + } +} From 11564d0a35a921015355becedd9b5a235bcba915 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 6 Jun 2018 07:25:43 +0200 Subject: [PATCH 0477/3965] generate the certificate chain on initialisation --- p2p/transport/quic/listener.go | 7 ++----- p2p/transport/quic/transport.go | 14 +++++++++----- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index b85800c57e..5e2c7cc0cc 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -1,6 +1,7 @@ package libp2pquic import ( + "crypto/tls" "net" ic "github.com/libp2p/go-libp2p-crypto" @@ -27,15 +28,11 @@ type listener struct { var _ tpt.Listener = &listener{} -func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, key ic.PrivKey) (tpt.Listener, error) { +func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, key ic.PrivKey, tlsConf *tls.Config) (tpt.Listener, error) { _, host, err := manet.DialArgs(addr) if err != nil { return nil, err } - tlsConf, err := generateConfig(key) - if err != nil { - return nil, err - } ln, err := quicListenAddr(host, tlsConf, &quic.Config{Versions: []quic.VersionNumber{101}}) if err != nil { return nil, err diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index eae98de6eb..5458fdb9ba 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -2,6 +2,7 @@ package libp2pquic import ( "context" + "crypto/tls" "crypto/x509" "errors" @@ -20,6 +21,7 @@ var quicDialAddr = quic.DialAddr type transport struct { privKey ic.PrivKey localPeer peer.ID + tlsConf *tls.Config } var _ tpt.Transport = &transport{} @@ -30,9 +32,14 @@ func NewTransport(key ic.PrivKey) (tpt.Transport, error) { if err != nil { return nil, err } + tlsConf, err := generateConfig(key) + if err != nil { + return nil, err + } return &transport{ privKey: key, localPeer: localPeer, + tlsConf: tlsConf, }, nil } @@ -42,11 +49,8 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - tlsConf, err := generateConfig(t.privKey) - if err != nil { - return nil, err - } var remotePubKey ic.PubKey + tlsConf := t.tlsConf.Clone() tlsConf.VerifyPeerCertificate = func(rawCerts [][]byte, _ [][]*x509.Certificate) error { chain := make([]*x509.Certificate, len(rawCerts)) for i := 0; i < len(rawCerts); i++ { @@ -92,7 +96,7 @@ func (t *transport) CanDial(addr ma.Multiaddr) bool { // Listen listens for new QUIC connections on the passed multiaddr. func (t *transport) Listen(addr ma.Multiaddr) (tpt.Listener, error) { - return newListener(addr, t, t.localPeer, t.privKey) + return newListener(addr, t, t.localPeer, t.privKey, t.tlsConf) } // Proxy returns true if this transport proxies. From 3cb5f3d15137f43df8b01c64eb021d17195a4a23 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 6 Jun 2018 07:27:43 +0200 Subject: [PATCH 0478/3965] use the same quic.Config values for dialing and listening --- p2p/transport/quic/listener.go | 2 +- p2p/transport/quic/transport.go | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 5e2c7cc0cc..aedd4ece92 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -33,7 +33,7 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, if err != nil { return nil, err } - ln, err := quicListenAddr(host, tlsConf, &quic.Config{Versions: []quic.VersionNumber{101}}) + ln, err := quicListenAddr(host, tlsConf, quicConfig) if err != nil { return nil, err } diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 5458fdb9ba..cebdc9caff 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -15,6 +15,12 @@ import ( "github.com/whyrusleeping/mafmt" ) +var quicConfig = &quic.Config{ + MaxReceiveStreamFlowControlWindow: 3 * (1 << 20), // 3 MB + MaxReceiveConnectionFlowControlWindow: 4.5 * (1 << 20), // 4.5 MB + Versions: []quic.VersionNumber{101}, +} + var quicDialAddr = quic.DialAddr // The Transport implements the tpt.Transport interface for QUIC connections. @@ -70,7 +76,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp } return nil } - sess, err := quicDialAddr(host, tlsConf, &quic.Config{Versions: []quic.VersionNumber{101}}) + sess, err := quicDialAddr(host, tlsConf, quicConfig) if err != nil { return nil, err } From f842b5569da44ec30c780cc79f4dfea7b473dd4f Mon Sep 17 00:00:00 2001 From: Christophe de Carvalho Pereira Martins Date: Wed, 6 Jun 2018 14:37:30 +0200 Subject: [PATCH 0479/3965] update README of the host example fixes #333 --- examples/libp2p-host/README.md | 61 ++++++++++++---------------------- 1 file changed, 21 insertions(+), 40 deletions(-) diff --git a/examples/libp2p-host/README.md b/examples/libp2p-host/README.md index f0cac08374..4714e8fe87 100644 --- a/examples/libp2p-host/README.md +++ b/examples/libp2p-host/README.md @@ -4,73 +4,54 @@ For most applications, the host is the basic building block you'll need to get s The host is an abstraction that manages services on top of a swarm. It provides a clean interface to connect to a service on a given remote peer. -First, you'll need an ID, and a place to store that ID. To generate an ID, you can do the following: +If you want to create a host with a default configuration, you can do the following: ```go import ( + "context" "crypto/rand" + "fmt" + libp2p "github.com/libp2p/go-libp2p" crypto "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" ) -// Generate an identity keypair using go's cryptographic randomness source -priv, pub, err := crypto.GenerateEd25519Key(rand.Reader) -if err != nil { - panic(err) -} -// A peers ID is the hash of its public key -pid, err := peer.IDFromPublicKey(pub) +// The context governs the lifetime of the libp2p node +ctx, cancel := context.WithCancel(context.Background()) +defer cancel() + +// To construct a simple host with all the default settings, just use `New` +h, err := libp2p.New(ctx) if err != nil { panic(err) } -// We've created the identity, now we need to store it. -// A peerstore holds information about peers, including your own -ps := pstore.NewPeerstore() -ps.AddPrivKey(pid, priv) -ps.AddPubKey(pid, pub) +fmt.Printf("Hello World, my hosts ID is %s\n", h.ID()) ``` -Next, you'll need at least one address that you want to listen on. You can go from a string to a multiaddr like this: +If you want more control over the configuration, you can specify some options to the constructor. In this snippet we generate our own ID and specified on which address we want to listen: ```go -import ma "github.com/multiformats/go-multiaddr" - -... - -maddr, err := ma.NewMultiaddr("/ip4/0.0.0.0/tcp/9000") +// Set your own keypair +priv, _, err := crypto.GenerateEd25519Key(rand.Reader) if err != nil { panic(err) } -``` -Now you know who you are, and where you live (in a manner of speaking). The next step is setting up a 'swarm network' to handle all the peers you will connect to. The swarm handles incoming connections from other peers, and handles the negotiation of new outbound connections. +h2, err := libp2p.New(ctx, + // Use your own created keypair + libp2p.Identity(priv), -```go -import ( - "context" - swarm "github.com/libp2p/go-libp2p-swarm" + // Set your own listen address + // The config takes an array of addresses, specify as many as you want. + libp2p.ListenAddrStrings("/ip4/0.0.0.0/tcp/9000"), ) - -// Make a context to govern the lifespan of the swarm -ctx := context.Background() - -// Put all this together -netw, err := swarm.NewNetwork(ctx, []ma.Multiaddr{maddr}, pid, ps, nil) if err != nil { panic(err) } -``` - -At this point, we have everything needed to finally construct a host. That call is the simplest one so far: - -```go -import bhost "github.com/libp2p/go-libp2p/p2p/host/basic" -myhost := bhost.New(netw) +fmt.Printf("Hello World, my second hosts ID is %s\n", h2.ID()) ``` And thats it, you have a libp2p host and you're ready to start doing some awesome p2p networking! From e6f29fdadcc8df11ce18223c2d0c7d0ee7c0e783 Mon Sep 17 00:00:00 2001 From: Christophe de Carvalho Pereira Martins Date: Wed, 6 Jun 2018 14:54:51 +0200 Subject: [PATCH 0480/3965] add a link to options.go in the host example so reader can see all the supported configuration for the host constructor --- examples/libp2p-host/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/examples/libp2p-host/README.md b/examples/libp2p-host/README.md index 4714e8fe87..88532053bb 100644 --- a/examples/libp2p-host/README.md +++ b/examples/libp2p-host/README.md @@ -30,7 +30,9 @@ if err != nil { fmt.Printf("Hello World, my hosts ID is %s\n", h.ID()) ``` -If you want more control over the configuration, you can specify some options to the constructor. In this snippet we generate our own ID and specified on which address we want to listen: +If you want more control over the configuration, you can specify some options to the constructor. For a full list of all the configuration supported by the constructor see: [options.go](https://github.com/libp2p/go-libp2p/blob/master/options.go) + +In this snippet we generate our own ID and specified on which address we want to listen: ```go // Set your own keypair From 2f001009f28244f650b775d0fd24875fd2643fbc Mon Sep 17 00:00:00 2001 From: Erin Swenson-Healey Date: Wed, 6 Jun 2018 07:49:29 -0700 Subject: [PATCH 0481/3965] loosen tolerance in latency test --- p2p/net/mock/mock_test.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index f2f6795fee..d22294c8aa 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -631,7 +631,8 @@ func TestStreamsWithLatency(t *testing.T) { wg.Wait() delta := time.Since(checkpoint) - if !within(delta, latency, time.Millisecond*50) { - t.Fatalf("Expected write to take ~%s, but took %s", latency.String(), delta.String()) + tolerance := time.Millisecond * 100 + if !within(delta, latency, tolerance) { + t.Fatalf("Expected write to take ~%s (+/- %s), but took %s", latency.String(), tolerance.String(), delta.String()) } } From f70bce27236ec9fa16bc66ac5be91962c485e5be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Thu, 7 Jun 2018 18:47:58 +0200 Subject: [PATCH 0482/3965] feat: lower timeout for local address dials --- p2p/net/swarm/addrs.go | 35 +++++++++++++++++++++++++++++++++++ p2p/net/swarm/limiter.go | 15 ++++++++++++++- p2p/net/swarm/swarm.go | 8 +++++++- 3 files changed, 56 insertions(+), 2 deletions(-) create mode 100644 p2p/net/swarm/addrs.go diff --git a/p2p/net/swarm/addrs.go b/p2p/net/swarm/addrs.go new file mode 100644 index 0000000000..ed510f2626 --- /dev/null +++ b/p2p/net/swarm/addrs.go @@ -0,0 +1,35 @@ +package swarm + +import ( + mafilter "github.com/libp2p/go-maddr-filter" + mamask "github.com/whyrusleeping/multiaddr-filter" +) + +// http://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml +var lowTimeoutFilters = mafilter.NewFilters() + +func init() { + for _, p := range []string{ + "/ip4/10.0.0.0/ipcidr/8", + "/ip4/100.64.0.0/ipcidr/10", + "/ip4/169.254.0.0/ipcidr/16", + "/ip4/172.16.0.0/ipcidr/12", + "/ip4/192.0.0.0/ipcidr/24", + "/ip4/192.0.0.0/ipcidr/29", + "/ip4/192.0.0.8/ipcidr/32", + "/ip4/192.0.0.170/ipcidr/32", + "/ip4/192.0.0.171/ipcidr/32", + "/ip4/192.0.2.0/ipcidr/24", + "/ip4/192.168.0.0/ipcidr/16", + "/ip4/198.18.0.0/ipcidr/15", + "/ip4/198.51.100.0/ipcidr/24", + "/ip4/203.0.113.0/ipcidr/24", + "/ip4/240.0.0.0/ipcidr/4", + } { + f, err := mamask.NewMask(p) + if err != nil { + panic("error in lowTimeoutFilters init: " + err.Error()) + } + lowTimeoutFilters.AddDialFilter(f) + } +} diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 3d3437b95f..a740e27efd 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -3,6 +3,7 @@ package swarm import ( "context" "sync" + "time" addrutil "github.com/libp2p/go-addr-util" peer "github.com/libp2p/go-libp2p-peer" @@ -33,6 +34,15 @@ func (dj *dialJob) cancelled() bool { } } +func (dj *dialJob) dialTimeout() time.Duration { + timeout := DialTimeout + if lowTimeoutFilters.AddrBlocked(dj.addr) { + timeout = DialTimeoutLocal + } + + return timeout +} + type dialLimiter struct { rllock sync.Mutex fdConsuming int @@ -162,7 +172,10 @@ func (dl *dialLimiter) executeDial(j *dialJob) { return } - con, err := dl.dialFunc(j.ctx, j.peer, j.addr) + dctx, cancel := context.WithTimeout(j.ctx, j.dialTimeout()) + defer cancel() + + con, err := dl.dialFunc(dctx, j.peer, j.addr) select { case j.resp <- dialResult{Conn: con, Addr: j.addr, Err: err}: case <-j.ctx.Done(): diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index dbf81613f9..002abd2d9f 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -22,9 +22,15 @@ import ( ) // DialTimeout is the maximum duration a Dial is allowed to take. +// This includes the time spent waiting in dial limiter, between dialing the raw +// network connection, protocol selection as well the handshake, if applicable. +var DialTimeout = 60 * time.Second + +// DialTimeoutLocal is the maximum duration a Dial to local network address +// is allowed to take. // This includes the time between dialing the raw network connection, // protocol selection as well the handshake, if applicable. -var DialTimeout = 60 * time.Second +var DialTimeoutLocal = 5 * time.Second var log = logging.Logger("swarm2") From bd1a6b0bdd25eef53c01870a480e3fa1dcced734 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 8 Jun 2018 10:11:41 -0700 Subject: [PATCH 0483/3965] add a TCP connect timeout --- p2p/transport/tcp/tcp.go | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 3ed2ff3f2f..18e119ec71 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -2,6 +2,7 @@ package tcp import ( "context" + "time" logging "github.com/ipfs/go-log" peer "github.com/libp2p/go-libp2p-peer" @@ -13,6 +14,10 @@ import ( mafmt "github.com/whyrusleeping/mafmt" ) +// DefaultConnectTimeout is the (default) maximum amount of time the TCP +// transport will spend on the initial TCP connect before giving up. +var DefaultConnectTimeout = 5 * time.Second + var log = logging.Logger("tcp-tpt") // TcpTransport is the TCP transport. @@ -24,6 +29,9 @@ type TcpTransport struct { // Explicitly disable reuseport. DisableReuseport bool + // TCP connect timeout + ConnectTimeout time.Duration + reuse rtpt.Transport } @@ -32,7 +40,7 @@ var _ tpt.Transport = &TcpTransport{} // NewTCPTransport creates a tcp transport object that tracks dialers and listeners // created. It represents an entire tcp stack (though it might not necessarily be) func NewTCPTransport(upgrader *tptu.Upgrader) *TcpTransport { - return &TcpTransport{Upgrader: upgrader} + return &TcpTransport{Upgrader: upgrader, ConnectTimeout: DefaultConnectTimeout} } // CanDial returns true if this transport believes it can dial the given @@ -42,6 +50,16 @@ func (t *TcpTransport) CanDial(addr ma.Multiaddr) bool { } func (t *TcpTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { + // Apply the deadline iff applicable + if t.ConnectTimeout > 0 { + deadline := time.Now().Add(t.ConnectTimeout) + if d, ok := ctx.Deadline(); !ok || deadline.Before(d) { + var cancel func() + ctx, cancel = context.WithDeadline(ctx, deadline) + defer cancel() + } + } + if t.UseReuseport() { return t.reuse.DialContext(ctx, raddr) } From 0c681d879019b6f602ae0a11973ea3a9cd68e45a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 8 Jun 2018 20:25:16 -0700 Subject: [PATCH 0484/3965] gx publish 6.0.1 --- .gx/lastpubver | 2 +- package.json | 110 ++++++++++++++++++++++++------------------------- 2 files changed, 56 insertions(+), 56 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 70da42f40f..747c00bb37 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.0: QmRvoAami8AAf5Yy6jcPq5KqQT1ZCaoi9dF1vdKAghmq9X +6.0.1: QmNVS2uXnjYf6TRgBHDEffvASQT2oFtnZkQK3Ds6eWEwdP diff --git a/package.json b/package.json index 60e39dbd9f..43e8733c74 100644 --- a/package.json +++ b/package.json @@ -19,9 +19,9 @@ "version": "0.1.1" }, { - "hash": "QmNiJuT8Ja3hMVpBHXv3Q6dwmperaQ6JjLtpMQgMCD7xvx", + "hash": "QmPdKqUcHGFdeSpvjVoaTRPPstGif9GBZb5Q56RVw9o69A", "name": "go-ipfs-util", - "version": "1.2.7" + "version": "1.2.8" }, { "hash": "QmbXRda5H2K3MSQyWWxTMtd8DWuguEBUCe6hpxfXVpFUGj", @@ -39,14 +39,14 @@ "version": "1.0.0" }, { - "hash": "QmTG23dvpBCBjqQwyDxV8CQT6jmS4PSftNr1VqHhE3MLy7", + "hash": "QmTsjVoib7EJhT7tmcqKPhEaATcxd6UVZu2qxdthk85vvv", "name": "go-log", - "version": "1.4.1" + "version": "1.5.1" }, { - "hash": "QmcGXGdw9BWDysPJQHxJinjGHha3eEg4vzFETre4woNwcX", + "hash": "QmNqRnejxJxjRroz7buhrjfU8i3yNBLa81hFtmf2pXEffN", "name": "go-multiaddr-net", - "version": "1.6.0" + "version": "1.6.1" }, { "hash": "QmZ4Qi3GaRbjcx28Sme5eMH7RQjGkt8wHxt2a65oLaeFEV", @@ -54,45 +54,45 @@ "version": "0.0.0" }, { - "hash": "QmWWQ2Txc2c6tqjsBpzg5Ar652cHPGNsQQp2SejkNmkUMb", + "hash": "QmUxSEGbv2nmYNnfXi7839wwQqTN3kwQeUxe8dTjZWZs7J", "name": "go-multiaddr", - "version": "1.2.6" + "version": "1.2.7" }, { "author": "whyrusleeping", - "hash": "QmPDZJxtWGfcwLPazJxD4h3v3aDs43V7UNAVs3Jz1Wo7o4", + "hash": "QmXKoqMcJaQM6agXCaPgDVUx11UxX8wBwKCkTsnDBCJZn6", "name": "go-libp2p-loggables", - "version": "1.1.14" + "version": "1.1.15" }, { "author": "whyrusleeping", - "hash": "QmQS7P1VV4JuqbXEEaPQZ5ERHDQuGp8qk26Gfdg9ZtP1Eb", + "hash": "QmfFWxQ5f2r664LwD6dASEfgYjojn4QkdCnuycu27oEeDo", "name": "go-libp2p-secio", - "version": "2.0.0" + "version": "2.0.1" }, { "author": "whyrusleeping", - "hash": "QmZb7hAgQEhW9dBbzBudU39gCeD4zbe6xafD52LUuF4cUN", + "hash": "QmPsgPxijUWE8fGcPmVDLJERZxmssBUHbsLNtJkeuWxUmm", "name": "go-libp2p-peerstore", - "version": "1.4.18" + "version": "1.4.19" }, { "author": "whyrusleeping", - "hash": "QmYnjSGtvn7LhrxCvwrU9uDWxKyg28uBYeXvgzTDDDzVy4", + "hash": "QmdMqcLHF29Wc2ty1Mm85XrYfPpRxKp3YXkfpeRZhUFZwX", "name": "go-libp2p-transport", - "version": "3.0.0" + "version": "3.0.1" }, { "author": "whyrusleeping", - "hash": "QmU7NiV3ZRJoNk8XZEKazs4AW7XQT1rQCrhh8cmhSjZgrC", + "hash": "QmcxaqKffd6114W97GuZtuDmNyftnagbT4o4B4jdyvt8oK", "name": "go-tcp-transport", - "version": "2.0.0" + "version": "2.0.1" }, { "author": "whyrusleeping", - "hash": "QmYF6XMumtqvXtZfQKrCokXEdFvfy2i7tbpvSRzRrHyY8c", + "hash": "QmNey9DW3QjsNh7tLfroFhk3994k99PC5Ta6aqCNA6hwYZ", "name": "go-maddr-filter", - "version": "1.1.7" + "version": "1.1.8" }, { "author": "whyrusleeping", @@ -102,51 +102,51 @@ }, { "author": "whyrusleeping", - "hash": "QmUJzxQQ2kzwQubsMqBTr1NGDpLfh7pGA2E1oaJULcKDPq", + "hash": "QmZaxPD9vvCgGq8sUBEZvjjyMrtL3kNto3DHiJ3qFSHKYQ", "name": "go-testutil", - "version": "1.2.1" + "version": "1.2.2" }, { "author": "whyrusleeping", - "hash": "QmYj8wdn5sZEHX2XMDWGBvcXJNdzVbaVpHmXvhHBVZepen", + "hash": "QmQGF5k4VyYwKXiKL1VxALixk9kHe7pSrNpXyJ8UmfLVBL", "name": "go-libp2p-net", - "version": "3.0.0" + "version": "3.0.1" }, { "author": "whyrusleeping", - "hash": "QmNnaGds3p4hfTqSH4KURKh8pBRcisAWYbNDEGeMZ7c3Hv", + "hash": "QmdZJXN4HVM9LDsn31zm6WByU1MtvJJCZQ62Qtf6ygQRSy", "name": "go-libp2p-metrics", - "version": "2.1.0" + "version": "2.1.1" }, { "author": "whyrusleeping", - "hash": "QmdHyfNVTZ5VtUx4Xz23z8wtnioSrFQ28XSfpVkdhQBkGA", + "hash": "QmSpDkHcp2mvSZ3uE9n3hMZs3tsau6iwbQzJXdbQ2BrmsF", "name": "go-libp2p-host", - "version": "3.0.0" + "version": "3.0.1" }, { "author": "whyrusleeping", - "hash": "QmPzT3rJnSP8VFP1kw7Ly7HP8AprKNZtwLHXHnxfVSbWT3", + "hash": "QmbFX2NDH7dfQFfiL8sEgwNko7Hsee3j9gfdqz8iB55Jpb", "name": "go-libp2p-swarm", - "version": "3.0.0" + "version": "3.0.1" }, { "author": "whyrusleeping", - "hash": "QmQF7htcTXeVqdTg4fKPGU59PeeTpsFgn9UquiixwbTPG1", + "hash": "QmSejn1nrguWhYCgf2NELNXUcA7TZ6nFmpMdB9rkiCYaMn", "name": "go-libp2p-nat", - "version": "0.8.0" + "version": "0.8.1" }, { "author": "whyrusleeping", - "hash": "Qmb3r9qUR7PnkyUKztmXp8sQhzXZHGmRg7fR5zsB1ebWMj", + "hash": "Qma7SgR78iRe6uY8gJ5LmgoeMse7i1bvggWi6sZPWePQfg", "name": "go-libp2p-netutil", - "version": "0.4.0" + "version": "0.4.1" }, { "author": "whyrusleeping", - "hash": "QmQ6ASb73YCy77TLfxzKnzQFUyFKMQzDhmjwjaQp6rxK34", + "hash": "QmQnF8bWyrdEjELunrfvMc71dAZSyWeb2MGTx18ZnR3E67", "name": "go-libp2p-blankhost", - "version": "0.3.0" + "version": "0.3.1" }, { "author": "whyrusleeping", @@ -174,33 +174,33 @@ }, { "author": "whyrusleeping", - "hash": "QmcJukH2sAFjY3HdBKq35WDzWoL3UUu2gt9wdfqZTUyM74", + "hash": "QmVh36bmhX2ATxs67tzRBs3SBJeEjiNFJHdmCHYotDZBUh", "name": "go-libp2p-peer", - "version": "2.3.2" + "version": "2.3.3" }, { "author": "vyzo", - "hash": "QmNXLcLAcfo8yp59FxFQJNa7pDbUUw97QN9GwefWWFK4hk", + "hash": "QmXez527Hb7hmFDfkihzPHFgSN17rNfE7kCH1E43P48AJG", "name": "go-libp2p-circuit", - "version": "2.1.0" + "version": "2.1.1" }, { "author": "lgierth", - "hash": "QmQMRYmPn77CKRFf4YFjX3M5e6uw6DFAgsQffCX6mwZ4mA", + "hash": "QmT8461vVVyBPyHJHQ6mvm8UdQ8UZNA5n6Z7kBk7GRf1xu", "name": "go-multiaddr-dns", - "version": "0.2.2" + "version": "0.2.3" }, { "author": "why", - "hash": "QmWCWsDQnnQ9Mo9V3GK8TSR91662FdFxjjqPX8YbHC8Ltz", + "hash": "QmfQqkLtaHhFEzcGVAJd9CFK3EMD7P9yByiR3uodMa6SAM", "name": "go-libp2p-interface-connmgr", - "version": "0.0.7" + "version": "0.0.8" }, { "author": "whyrusleeping", - "hash": "QmZeGmoJ3bEwEe6Huz6GKcHENWZCx7DReuAS5va4zP24PB", + "hash": "QmdXYoU9pWwnio622Pvh8ukkhqR4r5nrCQ4EyTN7QfRNU4", "name": "go-smux-multiplex", - "version": "3.0.8" + "version": "3.0.9" }, { "author": "multiformats", @@ -216,21 +216,21 @@ }, { "author": "whyrusleeping", - "hash": "QmTeRSFgnXRCh13sxsZkLTVCc1diUbZiT5mkGUgkR1J1on", + "hash": "QmUSNCZY313sLrA2fGWzsxt76bLixgry23WE9hjfEEPcoD", "name": "go-ws-transport", - "version": "2.0.0" + "version": "2.0.1" }, { "author": "stebalien", - "hash": "QmRYk8zWrXSkXsE16vM8yxByqM6eVvnXzDXKGvHFJJubVc", + "hash": "QmS8YJogJ7BZwqNU2KShJhJZ1MSph5XTsRGjZgrR48kjye", "name": "go-conn-security-multistream", - "version": "0.1.0" + "version": "0.1.1" }, { "author": "Stebalien", - "hash": "QmSieFUauuYnroStqmRAEgu9BMXDNY5LbtNgzXcFitBKXQ", + "hash": "QmTTeLLszzMh7ccM9zcHpJoucVbULf3VLgF4r1oBQSdfqB", "name": "go-conn-security", - "version": "0.1.1" + "version": "0.1.2" }, { "author": "libp2p", @@ -246,9 +246,9 @@ }, { "author": "steb", - "hash": "Qmf3ejfGWR8Bd3wKFBvwYGFMJ9TeKJwYJUc2WchXjMxzg7", + "hash": "QmVy8qVK2CVuQyXMe94SCs85pvBjEWfFHnircioWku71DX", "name": "go-libp2p-transport-upgrader", - "version": "0.1.0" + "version": "0.1.1" } ], "gxVersion": "0.4.0", @@ -256,6 +256,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.0" + "version": "6.0.1" } From a08d9e63dbf006c905a532d5f645a2bf6fae3bd1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 8 Jun 2018 22:30:45 -0700 Subject: [PATCH 0485/3965] gx publish 6.0.2 --- .gx/lastpubver | 2 +- package.json | 90 +++++++++++++++++++++++++------------------------- 2 files changed, 46 insertions(+), 46 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 747c00bb37..9bcc921fea 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.1: QmNVS2uXnjYf6TRgBHDEffvASQT2oFtnZkQK3Ds6eWEwdP +6.0.2: QmUEAR2pS7fP1GPseS3i8MWFyENs7oDp4CZrgn8FCjbsBu diff --git a/package.json b/package.json index 43e8733c74..d68dd06e9d 100644 --- a/package.json +++ b/package.json @@ -39,9 +39,9 @@ "version": "1.0.0" }, { - "hash": "QmTsjVoib7EJhT7tmcqKPhEaATcxd6UVZu2qxdthk85vvv", + "hash": "Qmbi1CTJsbnBZjCEgc2otwu8cUFPsGpzWXG7edVCLZ7Gvk", "name": "go-log", - "version": "1.5.1" + "version": "1.5.2" }, { "hash": "QmNqRnejxJxjRroz7buhrjfU8i3yNBLa81hFtmf2pXEffN", @@ -60,33 +60,33 @@ }, { "author": "whyrusleeping", - "hash": "QmXKoqMcJaQM6agXCaPgDVUx11UxX8wBwKCkTsnDBCJZn6", + "hash": "QmcBbMF4UyZFRTvH9S2h3rbSRBvvEGLqgt4sdvVugG8rX1", "name": "go-libp2p-loggables", - "version": "1.1.15" + "version": "1.1.16" }, { "author": "whyrusleeping", - "hash": "QmfFWxQ5f2r664LwD6dASEfgYjojn4QkdCnuycu27oEeDo", + "hash": "QmYLNezw2bwwSDoA82Teh9VSUvPGFmYdAmitxqVb19Y4xR", "name": "go-libp2p-secio", - "version": "2.0.1" + "version": "2.0.2" }, { "author": "whyrusleeping", - "hash": "QmPsgPxijUWE8fGcPmVDLJERZxmssBUHbsLNtJkeuWxUmm", + "hash": "QmZhsmorLpD9kmQ4ynbAu4vbKv2goMUnXazwGA4gnWHDjB", "name": "go-libp2p-peerstore", - "version": "1.4.19" + "version": "1.4.20" }, { "author": "whyrusleeping", - "hash": "QmdMqcLHF29Wc2ty1Mm85XrYfPpRxKp3YXkfpeRZhUFZwX", + "hash": "QmVMBFZqRZDA6TrQkVGGJEDSp5jC3UUMUjLcvaZ3fLCqh4", "name": "go-libp2p-transport", - "version": "3.0.1" + "version": "3.0.2" }, { "author": "whyrusleeping", - "hash": "QmcxaqKffd6114W97GuZtuDmNyftnagbT4o4B4jdyvt8oK", + "hash": "QmQkxTEhkY3ZsZJJnh5XqJH9tXrfHucpS8MgM6Jtyt6T5W", "name": "go-tcp-transport", - "version": "2.0.1" + "version": "2.0.2" }, { "author": "whyrusleeping", @@ -102,51 +102,51 @@ }, { "author": "whyrusleeping", - "hash": "QmZaxPD9vvCgGq8sUBEZvjjyMrtL3kNto3DHiJ3qFSHKYQ", + "hash": "QmPdxCaVp4jZ9RbxqZADvKH6kiCR5jHvdR5f2ycjAY6T2a", "name": "go-testutil", - "version": "1.2.2" + "version": "1.2.3" }, { "author": "whyrusleeping", - "hash": "QmQGF5k4VyYwKXiKL1VxALixk9kHe7pSrNpXyJ8UmfLVBL", + "hash": "QmXdgNhVEgjLxjUoMs5ViQL7pboAt3Y7V7eGHRiE4qrmTE", "name": "go-libp2p-net", - "version": "3.0.1" + "version": "3.0.2" }, { "author": "whyrusleeping", - "hash": "QmdZJXN4HVM9LDsn31zm6WByU1MtvJJCZQ62Qtf6ygQRSy", + "hash": "Qme71fYNbz1wLFqoYjLveUHX1CpPNfGmhBx8tvNKFjqUaC", "name": "go-libp2p-metrics", - "version": "2.1.1" + "version": "2.1.2" }, { "author": "whyrusleeping", - "hash": "QmSpDkHcp2mvSZ3uE9n3hMZs3tsau6iwbQzJXdbQ2BrmsF", + "hash": "QmQQGtcp6nVUrQjNsnU53YWV1q8fK1Kd9S7FEkYbRZzxry", "name": "go-libp2p-host", - "version": "3.0.1" + "version": "3.0.2" }, { "author": "whyrusleeping", - "hash": "QmbFX2NDH7dfQFfiL8sEgwNko7Hsee3j9gfdqz8iB55Jpb", + "hash": "QmSvhbgtjQJKdT5avEeb7cvjYs7YrhebJyM1K6GAnkKgfd", "name": "go-libp2p-swarm", - "version": "3.0.1" + "version": "3.0.2" }, { "author": "whyrusleeping", - "hash": "QmSejn1nrguWhYCgf2NELNXUcA7TZ6nFmpMdB9rkiCYaMn", + "hash": "QmVxBEipMPHG5SfX3LYJ7SCzSWySYwoabb6qeWuxfZS5QQ", "name": "go-libp2p-nat", - "version": "0.8.1" + "version": "0.8.2" }, { "author": "whyrusleeping", - "hash": "Qma7SgR78iRe6uY8gJ5LmgoeMse7i1bvggWi6sZPWePQfg", + "hash": "QmeBUY1BsMjkacVAJ2u76XBGNiRCHq6dkqT2VWG59N3d7b", "name": "go-libp2p-netutil", - "version": "0.4.1" + "version": "0.4.2" }, { "author": "whyrusleeping", - "hash": "QmQnF8bWyrdEjELunrfvMc71dAZSyWeb2MGTx18ZnR3E67", + "hash": "QmXkocWqNmLkoH6wbYi3dLeZJmTX3gZtVfzWTTyTsuMxje", "name": "go-libp2p-blankhost", - "version": "0.3.1" + "version": "0.3.2" }, { "author": "whyrusleeping", @@ -174,15 +174,15 @@ }, { "author": "whyrusleeping", - "hash": "QmVh36bmhX2ATxs67tzRBs3SBJeEjiNFJHdmCHYotDZBUh", + "hash": "QmVf8hTAsLLFtn4WPCRNdnaF2Eag2qTBS6uR8AiHPZARXy", "name": "go-libp2p-peer", - "version": "2.3.3" + "version": "2.3.4" }, { "author": "vyzo", - "hash": "QmXez527Hb7hmFDfkihzPHFgSN17rNfE7kCH1E43P48AJG", + "hash": "QmPavh4h3Edx5cv8GJQPC35AQAws7faXmzEZyru7k9b9Mn", "name": "go-libp2p-circuit", - "version": "2.1.1" + "version": "2.1.2" }, { "author": "lgierth", @@ -192,15 +192,15 @@ }, { "author": "why", - "hash": "QmfQqkLtaHhFEzcGVAJd9CFK3EMD7P9yByiR3uodMa6SAM", + "hash": "Qmav3fJzdn43FDvHyGkPdbQ5JVqqiDPmNdnuGa3vatpmwj", "name": "go-libp2p-interface-connmgr", - "version": "0.0.8" + "version": "0.0.9" }, { "author": "whyrusleeping", - "hash": "QmdXYoU9pWwnio622Pvh8ukkhqR4r5nrCQ4EyTN7QfRNU4", + "hash": "QmXKndvAxPwUuHmZDP7qizgNWovCyDkh4y43DiYwLtyQ4d", "name": "go-smux-multiplex", - "version": "3.0.9" + "version": "3.0.10" }, { "author": "multiformats", @@ -216,21 +216,21 @@ }, { "author": "whyrusleeping", - "hash": "QmUSNCZY313sLrA2fGWzsxt76bLixgry23WE9hjfEEPcoD", + "hash": "QmQSSwdqHWSVQyGZAaajnMCMpSJDqTG4zMCbqxh6bWTTuf", "name": "go-ws-transport", - "version": "2.0.1" + "version": "2.0.2" }, { "author": "stebalien", - "hash": "QmS8YJogJ7BZwqNU2KShJhJZ1MSph5XTsRGjZgrR48kjye", + "hash": "QmeRP5m2PBsVaragzoDqdDSS54nV2T7w8yfiiNZb7jfCMK", "name": "go-conn-security-multistream", - "version": "0.1.1" + "version": "0.1.2" }, { "author": "Stebalien", - "hash": "QmTTeLLszzMh7ccM9zcHpJoucVbULf3VLgF4r1oBQSdfqB", + "hash": "QmfCQHZGXiEqRgWBmJmWBD8p1rP3Z2T5Y5pvidfGTjsEPs", "name": "go-conn-security", - "version": "0.1.2" + "version": "0.1.3" }, { "author": "libp2p", @@ -246,9 +246,9 @@ }, { "author": "steb", - "hash": "QmVy8qVK2CVuQyXMe94SCs85pvBjEWfFHnircioWku71DX", + "hash": "QmdbjG1eui2spsiFLmBjmET6N9c4wfpDzAoKFyN72935Ec", "name": "go-libp2p-transport-upgrader", - "version": "0.1.1" + "version": "0.1.2" } ], "gxVersion": "0.4.0", @@ -256,6 +256,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.1" + "version": "6.0.2" } From a361bf03942b4b5e4b42b2e494ff3d45e4646579 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 9 Jun 2018 10:23:09 -0700 Subject: [PATCH 0486/3965] check for connectedness using Connectedness (counting open connections requires copying) --- p2p/host/basic/basic_host.go | 3 +-- p2p/host/routed/routed.go | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 018f2691d8..8437e086b9 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -384,8 +384,7 @@ func (h *BasicHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { // absorb addresses into peerstore h.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) - cs := h.Network().ConnsToPeer(pi.ID) - if len(cs) > 0 { + if h.Network().Connectedness(pi.ID) == inet.Connected { return nil } diff --git a/p2p/host/routed/routed.go b/p2p/host/routed/routed.go index df837fc7a9..99926f2b24 100644 --- a/p2p/host/routed/routed.go +++ b/p2p/host/routed/routed.go @@ -47,7 +47,7 @@ func Wrap(h host.Host, r Routing) *RoutedHost { // given peer, it will use its routing system to try to find some. func (rh *RoutedHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { // first, check if we're already connected. - if len(rh.Network().ConnsToPeer(pi.ID)) > 0 { + if rh.Network().Connectedness(pi.ID) == inet.Connected { return nil } From aeda168d9e692ef2b6c7c7339f684f74e8de18f1 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 11 Jun 2018 07:44:27 +0200 Subject: [PATCH 0487/3965] remove unused acceptQueue from the listener --- p2p/transport/quic/listener.go | 2 -- 1 file changed, 2 deletions(-) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index aedd4ece92..12b7f53a9e 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -19,8 +19,6 @@ type listener struct { quicListener quic.Listener transport tpt.Transport - acceptQueue chan tpt.Conn - privKey ic.PrivKey localPeer peer.ID localMultiaddr ma.Multiaddr From e08e8cc0d068283a938fa812d849d094da5523c4 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 11 Jun 2018 14:21:29 +0200 Subject: [PATCH 0488/3965] close and discard a connection if the client's cert chain is invalid --- p2p/transport/quic/conn_test.go | 82 ++++++++++++++++++++++++++++----- p2p/transport/quic/listener.go | 18 ++++++-- p2p/transport/quic/transport.go | 3 ++ 3 files changed, 88 insertions(+), 15 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 765e711fc8..55e81c37c9 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -4,6 +4,7 @@ import ( "context" "crypto/rand" "crypto/rsa" + "crypto/tls" "crypto/x509" "io/ioutil" @@ -30,23 +31,26 @@ var _ = Describe("Connection", func() { return priv } - runServer := func() (<-chan ma.Multiaddr, <-chan tpt.Conn) { - serverTransport, err := NewTransport(serverKey) - Expect(err).ToNot(HaveOccurred()) + runServer := func(tr tpt.Transport) (ma.Multiaddr, <-chan tpt.Conn) { addrChan := make(chan ma.Multiaddr) connChan := make(chan tpt.Conn) go func() { defer GinkgoRecover() addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") Expect(err).ToNot(HaveOccurred()) - ln, err := serverTransport.Listen(addr) + ln, err := tr.Listen(addr) Expect(err).ToNot(HaveOccurred()) addrChan <- ln.Multiaddr() conn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) connChan <- conn }() - return addrChan, connChan + return <-addrChan, connChan + } + + // modify the cert chain such that verificiation will fail + invalidateCertChain := func(tlsConf *tls.Config) { + tlsConf.Certificates[0].Certificate = [][]byte{tlsConf.Certificates[0].Certificate[0]} } BeforeEach(func() { @@ -60,10 +64,12 @@ var _ = Describe("Connection", func() { }) It("handshakes", func() { - serverAddrChan, serverConnChan := runServer() + serverTransport, err := NewTransport(serverKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr, serverConnChan := runServer(serverTransport) + clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) - serverAddr := <-serverAddrChan conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) Expect(err).ToNot(HaveOccurred()) serverConn := <-serverConnChan @@ -78,10 +84,13 @@ var _ = Describe("Connection", func() { }) It("opens and accepts streams", func() { - serverAddrChan, serverConnChan := runServer() + serverTransport, err := NewTransport(serverKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr, serverConnChan := runServer(serverTransport) + clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) - conn, err := clientTransport.Dial(context.Background(), <-serverAddrChan, serverID) + conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) Expect(err).ToNot(HaveOccurred()) serverConn := <-serverConnChan @@ -101,14 +110,65 @@ var _ = Describe("Connection", func() { thirdPartyID, err := peer.IDFromPrivateKey(createPeer()) Expect(err).ToNot(HaveOccurred()) - serverAddrChan, serverConnChan := runServer() + serverTransport, err := NewTransport(serverKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr, serverConnChan := runServer(serverTransport) + clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) - serverAddr := <-serverAddrChan // dial, but expect the wrong peer ID _, err = clientTransport.Dial(context.Background(), serverAddr, thirdPartyID) Expect(err).To(HaveOccurred()) Expect(err.Error()).To(ContainSubstring("TLS handshake error: bad certificate")) Consistently(serverConnChan).ShouldNot(Receive()) }) + + It("fails if the client presents an invalid cert chain", func() { + serverTransport, err := NewTransport(serverKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr, serverConnChan := runServer(serverTransport) + + clientTransport, err := NewTransport(clientKey) + invalidateCertChain(clientTransport.(*transport).tlsConf) + Expect(err).ToNot(HaveOccurred()) + conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) + Expect(err).ToNot(HaveOccurred()) + Eventually(func() bool { return conn.IsClosed() }).Should(BeTrue()) + Consistently(serverConnChan).ShouldNot(Receive()) + }) + + It("fails if the server presents an invalid cert chain", func() { + serverTransport, err := NewTransport(serverKey) + invalidateCertChain(serverTransport.(*transport).tlsConf) + Expect(err).ToNot(HaveOccurred()) + serverAddr, serverConnChan := runServer(serverTransport) + + clientTransport, err := NewTransport(clientKey) + Expect(err).ToNot(HaveOccurred()) + _, err = clientTransport.Dial(context.Background(), serverAddr, serverID) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("TLS handshake error: bad certificate")) + Consistently(serverConnChan).ShouldNot(Receive()) + }) + + It("keeps accepting connections after a failed connection attempt", func() { + serverTransport, err := NewTransport(serverKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr, serverConnChan := runServer(serverTransport) + + // first dial with an invalid cert chain + clientTransport1, err := NewTransport(clientKey) + invalidateCertChain(clientTransport1.(*transport).tlsConf) + Expect(err).ToNot(HaveOccurred()) + _, err = clientTransport1.Dial(context.Background(), serverAddr, serverID) + Expect(err).ToNot(HaveOccurred()) + Consistently(serverConnChan).ShouldNot(Receive()) + + // then dial with a valid client + clientTransport2, err := NewTransport(clientKey) + Expect(err).ToNot(HaveOccurred()) + _, err = clientTransport2.Dial(context.Background(), serverAddr, serverID) + Expect(err).ToNot(HaveOccurred()) + Eventually(serverConnChan).Should(Receive()) + }) }) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 12b7f53a9e..a3638dbe98 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -49,12 +49,22 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, } // Accept accepts new connections. -// TODO(#2): don't accept a connection if the client's peer verification fails func (l *listener) Accept() (tpt.Conn, error) { - sess, err := l.quicListener.Accept() - if err != nil { - return nil, err + for { + sess, err := l.quicListener.Accept() + if err != nil { + return nil, err + } + conn, err := l.setupConn(sess) + if err != nil { + sess.Close(err) + continue + } + return conn, nil } +} + +func (l *listener) setupConn(sess quic.Session) (tpt.Conn, error) { remotePubKey, err := getRemotePubKey(sess.ConnectionState().PeerCertificates) if err != nil { return nil, err diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index cebdc9caff..98b83eccf1 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -57,6 +57,9 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp } var remotePubKey ic.PubKey tlsConf := t.tlsConf.Clone() + // We need to check the peer ID in the VerifyPeerCertificate callback. + // The tls.Config it is also used for listening, and we might also have concurrent dials. + // Clone it so we can check for the specific peer ID we're dialing here. tlsConf.VerifyPeerCertificate = func(rawCerts [][]byte, _ [][]*x509.Certificate) error { chain := make([]*x509.Certificate, len(rawCerts)) for i := 0; i < len(rawCerts); i++ { From af53412341665ec5c178caca54336e8dad2c5288 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Tue, 12 Jun 2018 00:30:55 +0200 Subject: [PATCH 0489/3965] limiter: cleanup the code --- p2p/net/swarm/limiter.go | 107 ++++++++++++++++++---------------- p2p/net/swarm/limiter_test.go | 4 +- 2 files changed, 59 insertions(+), 52 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 3d3437b95f..1d9ff1a2ab 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -34,12 +34,13 @@ func (dj *dialJob) cancelled() bool { } type dialLimiter struct { - rllock sync.Mutex + lk sync.Mutex + fdConsuming int fdLimit int waitingOnFd []*dialJob - dialFunc func(context.Context, peer.ID, ma.Multiaddr) (transport.Conn, error) + dialFunc dialfunc activePerPeer map[peer.ID]int perPeerLimit int @@ -62,26 +63,26 @@ func newDialLimiterWithParams(df dialfunc, fdLimit, perPeerLimit int) *dialLimit } } -func (dl *dialLimiter) finishedDial(dj *dialJob) { - dl.rllock.Lock() - defer dl.rllock.Unlock() - - if addrutil.IsFDCostlyTransport(dj.addr) { - dl.fdConsuming-- - - if len(dl.waitingOnFd) > 0 { - next := dl.waitingOnFd[0] - dl.waitingOnFd[0] = nil // clear out memory - dl.waitingOnFd = dl.waitingOnFd[1:] - if len(dl.waitingOnFd) == 0 { - dl.waitingOnFd = nil // clear out memory - } - dl.fdConsuming++ - - go dl.executeDial(next) +// freeFDToken frees FD token and if there are any schedules another waiting dialJob +// in it's place +func (dl *dialLimiter) freeFDToken() { + dl.fdConsuming-- + + if len(dl.waitingOnFd) > 0 { + next := dl.waitingOnFd[0] + dl.waitingOnFd[0] = nil // clear out memory + dl.waitingOnFd = dl.waitingOnFd[1:] + if len(dl.waitingOnFd) == 0 { + dl.waitingOnFd = nil // clear out memory } + dl.fdConsuming++ + + // we already have activePerPeer token at this point so we can just dial + go dl.executeDial(next) } +} +func (dl *dialLimiter) freePeerToken(dj *dialJob) { // release tokens in reverse order than we take them dl.activePerPeer[dj.peer]-- if dl.activePerPeer[dj.peer] == 0 { @@ -91,45 +92,31 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { waitlist := dl.waitingOnPeerLimit[dj.peer] if !dj.success && len(waitlist) > 0 { next := waitlist[0] - if len(waitlist) == 1 { - delete(dl.waitingOnPeerLimit, dj.peer) + delete(dl.waitingOnPeerLimit, next.peer) } else { waitlist[0] = nil // clear out memory - dl.waitingOnPeerLimit[dj.peer] = waitlist[1:] + dl.waitingOnPeerLimit[next.peer] = waitlist[1:] } - dl.activePerPeer[dj.peer]++ // just kidding, we still want this token - if addrutil.IsFDCostlyTransport(next.addr) { - if dl.fdConsuming >= dl.fdLimit { - dl.waitingOnFd = append(dl.waitingOnFd, next) - return - } - - // take token - dl.fdConsuming++ - } + dl.activePerPeer[next.peer]++ // just kidding, we still want this token - // can kick this off right here, dials in this list already - // have the other tokens needed - go dl.executeDial(next) + dl.addCheckFdLimit(next) } } -// AddDialJob tries to take the needed tokens for starting the given dial job. -// If it acquires all needed tokens, it immediately starts the dial, otherwise -// it will put it on the waitlist for the requested token. -func (dl *dialLimiter) AddDialJob(dj *dialJob) { - dl.rllock.Lock() - defer dl.rllock.Unlock() +func (dl *dialLimiter) finishedDial(dj *dialJob) { + dl.lk.Lock() + defer dl.lk.Unlock() - if dl.activePerPeer[dj.peer] >= dl.perPeerLimit { - wlist := dl.waitingOnPeerLimit[dj.peer] - dl.waitingOnPeerLimit[dj.peer] = append(wlist, dj) - return + if addrutil.IsFDCostlyTransport(dj.addr) { + dl.freeFDToken() } - dl.activePerPeer[dj.peer]++ + dl.freePeerToken(dj) +} + +func (dl *dialLimiter) addCheckFdLimit(dj *dialJob) { if addrutil.IsFDCostlyTransport(dj.addr) { if dl.fdConsuming >= dl.fdLimit { dl.waitingOnFd = append(dl.waitingOnFd, dj) @@ -140,15 +127,35 @@ func (dl *dialLimiter) AddDialJob(dj *dialJob) { dl.fdConsuming++ } - // take second needed token and start dial! go dl.executeDial(dj) } +func (dl *dialLimiter) addCheckPeerLimit(dj *dialJob) { + if dl.activePerPeer[dj.peer] >= dl.perPeerLimit { + wlist := dl.waitingOnPeerLimit[dj.peer] + dl.waitingOnPeerLimit[dj.peer] = append(wlist, dj) + return + } + dl.activePerPeer[dj.peer]++ + + dl.addCheckFdLimit(dj) +} + +// AddDialJob tries to take the needed tokens for starting the given dial job. +// If it acquires all needed tokens, it immediately starts the dial, otherwise +// it will put it on the waitlist for the requested token. +func (dl *dialLimiter) AddDialJob(dj *dialJob) { + dl.lk.Lock() + defer dl.lk.Unlock() + + dl.addCheckPeerLimit(dj) +} + func (dl *dialLimiter) clearAllPeerDials(p peer.ID) { - dl.rllock.Lock() - defer dl.rllock.Unlock() + dl.lk.Lock() + defer dl.lk.Unlock() delete(dl.waitingOnPeerLimit, p) - // NB: the waitingOnFd list doesnt need to be cleaned out here, we will + // NB: the waitingOnFd list doesn't need to be cleaned out here, we will // remove them as we encounter them because they are 'cancelled' at this // point } diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 6ae3935caf..4338cad50a 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -378,9 +378,9 @@ func TestFDLimitUnderflow(t *testing.T) { time.Sleep(time.Second * 3) - l.rllock.Lock() + l.lk.Lock() fdConsuming := l.fdConsuming - l.rllock.Unlock() + l.lk.Unlock() if fdConsuming < 0 { t.Fatalf("l.fdConsuming < 0") From 964a7f10ea4c60a0783bc831ad049fec0c021e92 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 12 Jun 2018 13:46:34 +0200 Subject: [PATCH 0490/3965] disable source address validation The certificate chain used for the handshake is relatively small (it fits in 2 packets). This reduces the risk that an attacker will use libp2p nodes for reflection attacks. --- p2p/transport/quic/transport.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 98b83eccf1..c782d2a955 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -5,6 +5,7 @@ import ( "crypto/tls" "crypto/x509" "errors" + "net" ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" @@ -19,6 +20,10 @@ var quicConfig = &quic.Config{ MaxReceiveStreamFlowControlWindow: 3 * (1 << 20), // 3 MB MaxReceiveConnectionFlowControlWindow: 4.5 * (1 << 20), // 4.5 MB Versions: []quic.VersionNumber{101}, + AcceptCookie: func(clientAddr net.Addr, cookie *quic.Cookie) bool { + // TODO(#6): require source address validation when under load + return true + }, } var quicDialAddr = quic.DialAddr From 39791b176ccf77036524eafb456f51fa94227111 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Kurowski?= Date: Tue, 12 Jun 2018 21:39:29 +0200 Subject: [PATCH 0491/3965] function cannot be formatted as string --- p2p/host/basic/basic_host.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 018f2691d8..8eda0e46b2 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -432,7 +432,7 @@ func (h *BasicHost) resolveAddrs(ctx context.Context, pi pstore.PeerInfo) ([]ma. // dialPeer opens a connection to peer, and makes sure to identify // the connection once it has been opened. func (h *BasicHost) dialPeer(ctx context.Context, p peer.ID) error { - log.Debugf("host %s dialing %s", h.ID, p) + log.Debugf("host %s dialing %s", h.ID(), p) c, err := h.Network().DialPeer(ctx, p) if err != nil { return err From 555afdb8a5e577fbcb6f2baf6697eeaebf2f6fc4 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 12 Jun 2018 13:53:39 -0700 Subject: [PATCH 0492/3965] improve error message in libp2p reflection-based transport option --- config/reflection_magic.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/reflection_magic.go b/config/reflection_magic.go index c23658e9cc..d66a26e27c 100644 --- a/config/reflection_magic.go +++ b/config/reflection_magic.go @@ -23,7 +23,7 @@ func checkReturnType(fnType, tptType reflect.Type) error { fallthrough case 1: if !fnType.Out(0).Implements(tptType) { - return fmt.Errorf("expected first return value from transport constructor to be a transport") + return fmt.Errorf("transport constructor returns %s which doesn't implement %s", fnType.Out(0), tptType) } default: return fmt.Errorf("expected transport constructor to return a transport and, optionally, an error") From ec6358669c122b83aa7aa6d448ffd7f4d26e0dcc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Jun 2018 02:10:18 -0700 Subject: [PATCH 0493/3965] add news --- NEWS.md | 304 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 304 insertions(+) create mode 100644 NEWS.md diff --git a/NEWS.md b/NEWS.md new file mode 100644 index 0000000000..7b7e27af2d --- /dev/null +++ b/NEWS.md @@ -0,0 +1,304 @@ +# go-libp2p release notes + +## 6.0.0 + +We're pleased to announce go-libp2p 6.0.0. This release includes a massive +refactor of go-libp2p that paves the way for new transports such as QUIC. +Unfortunately, as it is broad sweeping, there are some breaking changes, +*especially* for maintainers of custom transports. + +Below, we cover the changes you'll likely care about. For convenience, we've +broken this into a section for users and transport authors/maintainers. However, +transport maintainers should really read both sections. + +### For Users + +Libp2p users should be aware of a few major changes. + +* Guarantees and performance concerning connect/disconnect notification + processing have improved. +* Handling of half-closed streams has changed (READ THIS SECTION). +* Some constructors and method signatures have changed slightly. + +#### Dialing And Source Addresses + +We've improved the logic that selects the source address when dialing. In the +past, you may have run into an issue where you couldn't dial non-local nodes +when listening on 127.0.0.1 as opposed to 0.0.0.0. This happened because +go-libp2p would randomly pick the source address from the set of addresses on +which the node was listening. We did this to ensure that the other end could +dial us back at the same address. Unfortunately, one can't use 127.0.0.1 as a +source address when dialing any non-local address so outbound dials failed. + +go-libp2p now tries to be smarter about this and avoids picking source addresses +that have no route to the destination address. + +#### Bandwidth Metrics + +To start out on an unhappy note, bandwidth metrics are now less accurate. In the +past, transports returned "minimal" connections (e.g., a TCP connection) so we +could wrap these transport connections in "metrics" connections that counted +every byte sent and received. + +Unfortunately, now that we've moved encryption and multiplexing down into the +transport layer, the connection we're wrapping has significantly more +under-the-covers overhead. + +However, we do hope to improve this and get even *better* bandwidth metrics than +we did before. See [libp2p/go-libp2p-transport#31][] for details. + +[libp2p/go-libp2p-transport#31]: https://github.com/libp2p/go-libp2p-transport/issues/31 + +#### Notifications + +This release brings performance improvements and easy to reason about ordering +guarantees libp2p connect/disconnect events: + +1. For any given connection/stream, libp2p will wait for all connect/open event + handlers to finish exit before triggering a disconnect/close event for the + connection/stream. +2. When a user calls the Close (or `Reset`) method on a connection or stream, + go-libp2p will process the close event asynchronously (i.e., not block the + call to `Close`). Otherwise, a call to `Close` from within a connect event + handler would deadlock. +3. Unless otherwise noted, events will be handled in parallel. + +What does this mean for end users? Well: + +1. Reference counting connections to a peer using connect/disconnect events + should "just work" and should never go negative. +2. Under heavy connect/disconnect loads, connecting to new peers should be + faster (usually). + +For those interested in the history of this issue, ... + +In the past, (dis)connect and stream open/close notifications have been a bit of +a pain point. For a long time, they were fired off in parallel and one could, for +example, process a disconnect notification before a connect notification (we had +to support *negative* ref-counts in several places to account for this). + +After no end of trouble, we finally "fixed" this by synchronizing notification +delivery. We still delivered notifications to all notifiees in parallel, we just +processed the events in series. + +Unfortunately, under heavy connect/disconnect load, new connections could easily +get stuck on open behind a queue of connect events all being handled in series. +In theory, these events should have been handled quickly but in practice, it's +very hard to avoid locks *entirely* (bitswap's event handlers were especially +problematic). + +Worse, this serial delivery guarantee didn't actually provide us with an +*in-order* delivery guarantee as it was still possible for a disconnect to +happen before we even *started* to fire the connect event. The situation was +slightly better than before because the events couldn't overlap but still far +from optimal. + +However, this has all been resolved now. From now on, you'll never receive a +disconnect event before a connect event. + +#### Conn.GetStreams Signature Change + +The signature of the `GetStreams` method on `go-libp2p-net.Conn` has changed from: + +```go +GetStreams() ([]Stream, error) +``` + +To: + +```go +GetStreams() []Stream +``` + +Listing the streams on an open connection should never involve IO or do anything +that can fail so we removed this error to improve usability. + +#### Libp2p Constructor + +If you're not already doing so, you should be using the `libp2p.New` constructor +to make your libp2p nodes. This release brings quite a few new options to the +libp2p constructor so if it hasn't been flexible enough for you in the past, I +recommend that you try again. A simple example can be found in the +[echo][example:echo] example. + +Given this work and in an attempt to consolidate all of our configuration logic +in one place, we've removed all default transports from go-libp2p-swarm. + +TL;DR: Please use the libp2p constructor. + +#### Zombie Streams + +From this release on, when you're done with a stream, you must either call +`Reset` on it (in case of an error) or close it and read an EOF (or some other +error). Otherwise, libp2p can't determine if the stream is *actually* closed and +will hang onto it indefinitely. + +To make properly closing streams a bit easier, we've added two methods to +[go-libp2p-net][]: `AwaitEOF` and `FullClose`. + +* `AwaitEOF(stream)` tries to read a single byte from the stream. If `Read` +returns an EOF, `AwaitEOF` returns success. Otherwise, if `Read` either reads +some data or returns some other error, `AwaitEOF` resets the stream and returns +an error. To avoid waiting indefinitely, `AwaitEOF` resets the stream +unconditionally after 1 minute. +* `FullClose(stream)` is a convenience function that closes the stream and then +calls `AwaitEOF` on it. + +Like with libp2p notifications, this issue has a bit of history... + +In the beginning, libp2p assumed that calling `Close` on a stream would close +the stream for both reading and writing. Unfortunately, *none* of our stream +multiplexers actually behaved this way. In practice, `Close` always closed the +stream for writing. + +After realizing this, we made two changes: + +1. We accepted the fact that `Close` only closed the stream for writing. +2. We added a `Reset` method for killing the stream (closing it in both + directions, throwing away any buffered data). + +However, we ran into a bit of a snag because we try to track open streams and +need some way to tell when a stream has been closed. In the past this was easy: +when the user calls `Close` on the stream, stop tracking it. However, now that +`Close` only closes the stream for writing, we still *technically* needed to +track it until the *other* end closed the stream as well. Unfortunately, without +actually reading from the stream, we have no way of knowing about this. +Therefore, if the user calls `Close` on a stream and then walks away, we'd have +to hang onto the stream indefinitely. + +Our solution was to simply stop tracking streams once they were closed for +writing. This wasn't the *correct* behavior but it avoided leaking memory in the +common case: + +1. The user calls `Close` and drops all references to the stream. +2. The other end calls `Close` without writing any additional data. +3. The stream multiplexer observes both closes and drops *its* reference to the stream. +4. The garbage collector garbage collects the stream. + +However, this meant that: + +1. The list of "open" streams was technically incomplete. +2. If the other side either failed to call `Close` or tried to send data before + closing, the stream would remain "open" (until the connection was closed). + +In this release, we've changed this behavior. Now, when you `Close` a stream for +writing, libp2p *continues* to track it. We only stop tracking it when either: + +1. You call `Reset` (throwing away the stream). +2. You finish reading any data off of it and observe either an EOF or an error. + +This way, we never "forget" about open streams or leave them in a half-forgotten +state. + +In the future, I'd like to add a `CloseAndForget` method to streams that: + +1. Closes the stream (sends an EOF). +2. Tells the swarm to stop tracking the stream. +3. Tells the stream muxer to stop tracking the stream and throw away any data + the other side may send (possibly resetting the stream on unexpected data). + +However: + +1. This would likely require modifying our stream muxers which may not be + feasible. +2. Explicitly waiting for an EOF is still the correct thing to do unless you + really don't care if the operation succeeded. + +### For Transport Maintainers + +For transport maintainers, quite a bit has changed. Before this change, +transports created simple, unencrypted, stream connections and it was the job of +the libp2p Network (go-libp2p-swarm) to negotiate security, multiplexing, etc. + +However, when attempting to add support for the QUIC protocol, we realized that +this was going to be a problem: QUIC already handles authentication and +encryption (using TLS1.3) and multiplexing. After much debate, we inverted our +current architecture and made transports responsible for encrypting/multiplexing +their connections (before returning them). + +To make this palatable, we've also introduced a new ["upgrader" +library][go-libp2p-transport-upgrader] for upgrading go-multiaddr-net +connections/listeners to full libp2p transport connections/listeners. Transports +that don't support encryption/multiplexing out of the box can expect to have an +upgrader passed into the constructor. + +To get a feel for how this new transport system works, take a look at the TCP +and WebSocket transports and the transport interface documentation: + +* [TCP Transport][go-tcp-transport] +* [WebSocket Transport][go-ws-transport] +* [Transport Interface][doc:go-libp2p-transport] + +#### Deprecated Packages + +This release sees the deprecation of a few packages: + +* [go-peerstream][] has been deprecated and all functionality has been merged + into [go-libp2p-swarm][]. [go-peerstream][] was written as a general-purpose + (not libp2p specific) listener, connection, and stream manager. However, this + package caused more problems than it solved and was incompatible with the new + transport interface. +* [go-libp2p-interface-conn][] has been deprecated. These interfaces to bridge + the gap between transport-level connections and [go-libp2p-net][] connections + however, now that transport connections are fully multiplexed/encrypted, this + is no longer needed. +* [go-libp2p-conn][] has also been deprecated and most of the functionality has + been moved to [go-libp2p-transport-upgrader][]. This package used to provide + connection "upgrade" logic for upgrading transport-level connections to + [go-libp2p-interface-conn][] connections however, transport-level connections + now provide the required functionality out of the box. + +#### Testing + +We've moved `GenSwarmNetwork` in [go-libp2p-netutil][] to `GenSwarm` in +[go-libp2p-swarm/testing][] because: + +1. The swarm duplicated this exact function for its own tests. +2. The swarm couldn't depend on [go-libp2p-netutil][] because + [go-libp2p-netutil][] depends on [go-libp2p-swarm][]. + +We've also added a new transport test suit +[go-libp2p-transport/test][]. If you implement a new transport, please consider +testing against these suite. If you find a bug in an existing transport, please +consider adding a test to this suite. + +#### go-addr-util + +In go-addr-util, we've removed the `SupportedTransportStrings` and +`SupportedTransportProtocols` transport registries and the associated +`AddTransport` function. These registries were updated by `init` functions in +packages providing transports and were used to keep track of known transports. + +However, *importing* a transport doesn't mean any libp2p nodes have been +configured to actually *use* that transport. Therefore, in the new go-libp2p, +it's go-libp2p-swarm's job to keep track of which transports are supported +(i.e., which transports have been registered with the swarm). + +We've also removed the associated `AddrUsable`, `FilterUsableAddrs`, and +`AddrUsableFunc` functions. + +#### Pluggable Security Transports + +This release brings a new pluggable security transport framework. Implementing a +new security framework is now as simple as: + +1. Implement the interfaces defined in [go-conn-security][]. +2. Pass it into the libp2p constructor using the `Security` option. + +[go-conn-security]: https://github.com/libp2p/go-conn-security +[go-libp2p-conn]: https://github.com/libp2p/go-libp2p-conn +[go-libp2p-interface-conn]: https://github.com/libp2p/go-libp2p-interface-conn +[go-libp2p-net]: https://github.com/libp2p/go-libp2p-net +[go-libp2p-netutil]: https://github.com/libp2p/go-libp2p/go-libp2p-netutil +[go-libp2p-swarm]: https://github.com/libp2p/go-libp2p/go-libp2p-swarm +[go-libp2p-swarm/testing]: https://github.com/libp2p/go-libp2p/go-libp2p-swarm/testing +[go-libp2p-transport]: https://github.com/libp2p/go-libp2p-transport +[go-libp2p-transport/test]: https://github.com/libp2p/go-libp2p-transport/test +[go-libp2p-transport-upgrader]: https://github.com/libp2p/go-libp2p-transport-upgrader +[go-peerstream]: https://github.com/libp2p/go-peerstream +[go-tcp-transport]: https://github.com/libp2p/go-tcp-transport +[go-ws-transport]: https://github.com/libp2p/go-ws-transport + +[example:echo]: https://github.com/libp2p/go-libp2p/tree/master/examples/echo + +[doc:go-libp2p-transport]: https://godoc.org/github.com/libp2p/go-libp2p-transport From 55f0adc1a33f63d1c68e82ad4c9d05695e750a92 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 15 Jun 2018 06:42:44 +0700 Subject: [PATCH 0494/3965] use the context for dialing --- p2p/transport/quic/transport.go | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index c782d2a955..a763c73451 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -26,8 +26,6 @@ var quicConfig = &quic.Config{ }, } -var quicDialAddr = quic.DialAddr - // The Transport implements the tpt.Transport interface for QUIC connections. type transport struct { privKey ic.PrivKey @@ -84,7 +82,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp } return nil } - sess, err := quicDialAddr(host, tlsConf, quicConfig) + sess, err := quic.DialAddrContext(ctx, host, tlsConf, quicConfig) if err != nil { return nil, err } From 3439696ebe1636b00e6eb3a3d9dc014f9160753e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 15 Jun 2018 10:42:19 +0700 Subject: [PATCH 0495/3965] fix the name of test ginkgo case --- p2p/transport/quic/libp2pquic_suite_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/quic/libp2pquic_suite_test.go b/p2p/transport/quic/libp2pquic_suite_test.go index d3766da3e0..9675f74995 100644 --- a/p2p/transport/quic/libp2pquic_suite_test.go +++ b/p2p/transport/quic/libp2pquic_suite_test.go @@ -7,7 +7,7 @@ import ( "testing" ) -func TestQuicGo(t *testing.T) { +func TestLibp2pQuicTransport(t *testing.T) { RegisterFailHandler(Fail) RunSpecs(t, "libp2p QUIC Transport Suite") } From b4428fda587f5faaa4529f336fee057a166db268 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 15 Jun 2018 14:27:22 -0700 Subject: [PATCH 0496/3965] reduce error to warning and log the error This happens all the time in tests where we intentionally use fake keys for performance. Anyways, users probably don't want their logs spammed with errors they can't do anything about. --- p2p/protocol/identify/id.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 8f8394fbd1..c548fd3272 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -255,7 +255,7 @@ func (ids *IDService) consumeReceivedPubKey(c inet.Conn, kb []byte) { newKey, err := ic.UnmarshalPublicKey(kb) if err != nil { - log.Errorf("%s cannot unmarshal key from remote peer: %s", lp, rp) + log.Warningf("%s cannot unmarshal key from remote peer: %s, %s", lp, rp, err) return } From 3524ae26b7476f073b12b36f3bb32542076fe16f Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 16 Jun 2018 22:23:12 +0700 Subject: [PATCH 0497/3965] use multiaddr.Encapsulate to create QUIC multiaddrs --- p2p/transport/quic/conn.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index e5ccdc53c8..312da6d1de 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -82,15 +82,14 @@ func (c *conn) Transport() tpt.Transport { return c.transport } -// TODO: there must be a better way to do this func quicMultiaddr(na net.Addr) (ma.Multiaddr, error) { udpMA, err := manet.FromNetAddr(na) if err != nil { return nil, err } - quicMA, err := ma.NewMultiaddr(udpMA.String() + "/quic") + quicMA, err := ma.NewMultiaddr("/quic") if err != nil { return nil, err } - return quicMA, nil + return udpMA.Encapsulate(quicMA), nil } From c0e1cce41218bc160c2bb6b8910c3ba96d112b7d Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 17 Jun 2018 11:56:33 +0700 Subject: [PATCH 0498/3965] add a function to decapsulate a quic multiaddr --- p2p/transport/quic/conn.go | 15 ------------ p2p/transport/quic/listener.go | 4 +-- p2p/transport/quic/quic_multiaddr.go | 30 +++++++++++++++++++++++ p2p/transport/quic/quic_multiaddr_test.go | 30 +++++++++++++++++++++++ p2p/transport/quic/transport.go | 2 +- 5 files changed, 63 insertions(+), 18 deletions(-) create mode 100644 p2p/transport/quic/quic_multiaddr.go create mode 100644 p2p/transport/quic/quic_multiaddr_test.go diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 312da6d1de..fef587395f 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -1,15 +1,12 @@ package libp2pquic import ( - "net" - ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" tpt "github.com/libp2p/go-libp2p-transport" smux "github.com/libp2p/go-stream-muxer" quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" ) type conn struct { @@ -81,15 +78,3 @@ func (c *conn) RemoteMultiaddr() ma.Multiaddr { func (c *conn) Transport() tpt.Transport { return c.transport } - -func quicMultiaddr(na net.Addr) (ma.Multiaddr, error) { - udpMA, err := manet.FromNetAddr(na) - if err != nil { - return nil, err - } - quicMA, err := ma.NewMultiaddr("/quic") - if err != nil { - return nil, err - } - return udpMA.Encapsulate(quicMA), nil -} diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index a3638dbe98..52558872d6 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -35,7 +35,7 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, if err != nil { return nil, err } - localMultiaddr, err := quicMultiaddr(ln.Addr()) + localMultiaddr, err := toQuicMultiaddr(ln.Addr()) if err != nil { return nil, err } @@ -73,7 +73,7 @@ func (l *listener) setupConn(sess quic.Session) (tpt.Conn, error) { if err != nil { return nil, err } - remoteMultiaddr, err := quicMultiaddr(sess.RemoteAddr()) + remoteMultiaddr, err := toQuicMultiaddr(sess.RemoteAddr()) if err != nil { return nil, err } diff --git a/p2p/transport/quic/quic_multiaddr.go b/p2p/transport/quic/quic_multiaddr.go new file mode 100644 index 0000000000..8b182b76ba --- /dev/null +++ b/p2p/transport/quic/quic_multiaddr.go @@ -0,0 +1,30 @@ +package libp2pquic + +import ( + "net" + + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +var quicMA ma.Multiaddr + +func init() { + var err error + quicMA, err = ma.NewMultiaddr("/quic") + if err != nil { + panic(err) + } +} + +func toQuicMultiaddr(na net.Addr) (ma.Multiaddr, error) { + udpMA, err := manet.FromNetAddr(na) + if err != nil { + return nil, err + } + return udpMA.Encapsulate(quicMA), nil +} + +func fromQuicMultiaddr(addr ma.Multiaddr) (net.Addr, error) { + return manet.ToNetAddr(addr.Decapsulate(quicMA)) +} diff --git a/p2p/transport/quic/quic_multiaddr_test.go b/p2p/transport/quic/quic_multiaddr_test.go new file mode 100644 index 0000000000..48949d3aa1 --- /dev/null +++ b/p2p/transport/quic/quic_multiaddr_test.go @@ -0,0 +1,30 @@ +package libp2pquic + +import ( + "net" + + ma "github.com/multiformats/go-multiaddr" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +var _ = Describe("QUIC Multiaddr", func() { + It("converts a net.Addr to a QUIC Multiaddr", func() { + addr := &net.UDPAddr{IP: net.IPv4(192, 168, 0, 42), Port: 1337} + maddr, err := toQuicMultiaddr(addr) + Expect(err).ToNot(HaveOccurred()) + Expect(maddr.String()).To(Equal("/ip4/192.168.0.42/udp/1337/quic")) + }) + + It("converts a QUIC Multiaddr to a net.Addr", func() { + maddr, err := ma.NewMultiaddr("/ip4/192.168.0.42/udp/1337/quic") + Expect(err).ToNot(HaveOccurred()) + addr, err := fromQuicMultiaddr(maddr) + Expect(err).ToNot(HaveOccurred()) + Expect(addr).To(BeAssignableToTypeOf(&net.UDPAddr{})) + udpAddr := addr.(*net.UDPAddr) + Expect(udpAddr.IP).To(Equal(net.IPv4(192, 168, 0, 42))) + Expect(udpAddr.Port).To(Equal(1337)) + }) +}) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index a763c73451..909f850837 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -86,7 +86,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - localMultiaddr, err := quicMultiaddr(sess.LocalAddr()) + localMultiaddr, err := toQuicMultiaddr(sess.LocalAddr()) if err != nil { return nil, err } From 308b97b2ac88dea92e7058ee774501337b5dea8e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 17 Jun 2018 12:55:35 +0700 Subject: [PATCH 0499/3965] use a single packet conn for all outgoing connections --- p2p/transport/quic/conn_test.go | 73 ++++++++++++++++++++++++++++----- p2p/transport/quic/transport.go | 19 ++++++++- 2 files changed, 80 insertions(+), 12 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 55e81c37c9..1a1e3f4a67 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -1,12 +1,14 @@ package libp2pquic import ( + "bytes" "context" "crypto/rand" "crypto/rsa" "crypto/tls" "crypto/x509" "io/ioutil" + "time" ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" @@ -23,12 +25,14 @@ var _ = Describe("Connection", func() { serverID, clientID peer.ID ) - createPeer := func() ic.PrivKey { + createPeer := func() (peer.ID, ic.PrivKey) { key, err := rsa.GenerateKey(rand.Reader, 1024) Expect(err).ToNot(HaveOccurred()) priv, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(key)) Expect(err).ToNot(HaveOccurred()) - return priv + id, err := peer.IDFromPrivateKey(priv) + Expect(err).ToNot(HaveOccurred()) + return id, priv } runServer := func(tr tpt.Transport) (ma.Multiaddr, <-chan tpt.Conn) { @@ -54,13 +58,8 @@ var _ = Describe("Connection", func() { } BeforeEach(func() { - var err error - serverKey = createPeer() - serverID, err = peer.IDFromPrivateKey(serverKey) - Expect(err).ToNot(HaveOccurred()) - clientKey = createPeer() - clientID, err = peer.IDFromPrivateKey(clientKey) - Expect(err).ToNot(HaveOccurred()) + serverID, serverKey = createPeer() + clientID, clientKey = createPeer() }) It("handshakes", func() { @@ -107,8 +106,7 @@ var _ = Describe("Connection", func() { }) It("fails if the peer ID doesn't match", func() { - thirdPartyID, err := peer.IDFromPrivateKey(createPeer()) - Expect(err).ToNot(HaveOccurred()) + thirdPartyID, _ := createPeer() serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) @@ -171,4 +169,57 @@ var _ = Describe("Connection", func() { Expect(err).ToNot(HaveOccurred()) Eventually(serverConnChan).Should(Receive()) }) + + It("dials to two servers at the same time", func() { + serverID2, serverKey2 := createPeer() + + serverTransport, err := NewTransport(serverKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr, serverConnChan := runServer(serverTransport) + serverTransport2, err := NewTransport(serverKey2) + Expect(err).ToNot(HaveOccurred()) + serverAddr2, serverConnChan2 := runServer(serverTransport2) + + data := bytes.Repeat([]byte{'a'}, 5*1<<20) // 5 MB + // wait for both servers to accept a connection + // then send some data + go func() { + for _, c := range []tpt.Conn{<-serverConnChan, <-serverConnChan2} { + go func(conn tpt.Conn) { + defer GinkgoRecover() + str, err := conn.OpenStream() + Expect(err).ToNot(HaveOccurred()) + defer str.Close() + _, err = str.Write(data) + Expect(err).ToNot(HaveOccurred()) + }(c) + } + }() + + clientTransport, err := NewTransport(clientKey) + Expect(err).ToNot(HaveOccurred()) + c1, err := clientTransport.Dial(context.Background(), serverAddr, serverID) + Expect(err).ToNot(HaveOccurred()) + c2, err := clientTransport.Dial(context.Background(), serverAddr2, serverID2) + Expect(err).ToNot(HaveOccurred()) + + done := make(chan struct{}, 2) + // receive the data on both connections at the same time + for _, c := range []tpt.Conn{c1, c2} { + go func(conn tpt.Conn) { + defer GinkgoRecover() + str, err := conn.AcceptStream() + Expect(err).ToNot(HaveOccurred()) + str.Close() + d, err := ioutil.ReadAll(str) + Expect(err).ToNot(HaveOccurred()) + Expect(d).To(Equal(data)) + conn.Close() + done <- struct{}{} + }(c) + } + + Eventually(done, 5*time.Second).Should(Receive()) + Eventually(done, 5*time.Second).Should(Receive()) + }) }) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 909f850837..814906b4d7 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -31,6 +31,7 @@ type transport struct { privKey ic.PrivKey localPeer peer.ID tlsConf *tls.Config + pconn net.PacketConn } var _ tpt.Transport = &transport{} @@ -45,10 +46,22 @@ func NewTransport(key ic.PrivKey) (tpt.Transport, error) { if err != nil { return nil, err } + + // create a packet conn for outgoing connections + addr, err := net.ResolveUDPAddr("udp", "localhost:0") + if err != nil { + return nil, err + } + conn, err := net.ListenUDP("udp", addr) + if err != nil { + return nil, err + } + return &transport{ privKey: key, localPeer: localPeer, tlsConf: tlsConf, + pconn: conn, }, nil } @@ -58,6 +71,10 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } + addr, err := fromQuicMultiaddr(raddr) + if err != nil { + return nil, err + } var remotePubKey ic.PubKey tlsConf := t.tlsConf.Clone() // We need to check the peer ID in the VerifyPeerCertificate callback. @@ -82,7 +99,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp } return nil } - sess, err := quic.DialAddrContext(ctx, host, tlsConf, quicConfig) + sess, err := quic.DialContext(ctx, t.pconn, addr, host, tlsConf, quicConfig) if err != nil { return nil, err } From a58964041052f25879a241b117058170a4b062d9 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 25 Jun 2018 17:21:02 -0700 Subject: [PATCH 0500/3965] gx publish 3.0.11 --- p2p/muxer/mplex/multiplex.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 0eebb7262f..4a217c3048 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -3,7 +3,7 @@ package peerstream_multiplex import ( "net" - smux "github.com/libp2p/go-stream-muxer" + smux "github.com/libp2p/go-stream-muxer" // Conn is a connection to a remote peer. mp "github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. ) From ecdd30ece946465822cf2f865270ba50b6d5f6c8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 25 Jun 2018 17:22:33 -0700 Subject: [PATCH 0501/3965] gx publish 6.0.3 --- .gx/lastpubver | 2 +- package.json | 90 +++++++++++++++++++++++++------------------------- 2 files changed, 46 insertions(+), 46 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 9bcc921fea..15288eb266 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.2: QmUEAR2pS7fP1GPseS3i8MWFyENs7oDp4CZrgn8FCjbsBu +6.0.3: QmddH8SGt2vYLw7hsv7tBfY6XV2rCagtSdZECUnpVftWLr diff --git a/package.json b/package.json index d68dd06e9d..3bfe64343a 100644 --- a/package.json +++ b/package.json @@ -39,9 +39,9 @@ "version": "1.0.0" }, { - "hash": "Qmbi1CTJsbnBZjCEgc2otwu8cUFPsGpzWXG7edVCLZ7Gvk", + "hash": "QmcVVHfdyv15GVPk7NrxdWjh2hLVccXnoD8j2tyQShiXJb", "name": "go-log", - "version": "1.5.2" + "version": "1.5.3" }, { "hash": "QmNqRnejxJxjRroz7buhrjfU8i3yNBLa81hFtmf2pXEffN", @@ -60,33 +60,33 @@ }, { "author": "whyrusleeping", - "hash": "QmcBbMF4UyZFRTvH9S2h3rbSRBvvEGLqgt4sdvVugG8rX1", + "hash": "Qmb3WWGVSrvhkDmuajyQWfvNdURCa1q9TVuj6giAGuq3Bk", "name": "go-libp2p-loggables", - "version": "1.1.16" + "version": "1.1.17" }, { "author": "whyrusleeping", - "hash": "QmYLNezw2bwwSDoA82Teh9VSUvPGFmYdAmitxqVb19Y4xR", + "hash": "QmeP5zmVqYZGQJfwzLt3Vw4vxiqn4wTwcdgSqD59rndFrF", "name": "go-libp2p-secio", - "version": "2.0.2" + "version": "2.0.3" }, { "author": "whyrusleeping", - "hash": "QmZhsmorLpD9kmQ4ynbAu4vbKv2goMUnXazwGA4gnWHDjB", + "hash": "QmeXmxAE5URbpqgAMAwQiR3zEbd2wvSPUXbdytvFdGNELF", "name": "go-libp2p-peerstore", - "version": "1.4.20" + "version": "1.4.21" }, { "author": "whyrusleeping", - "hash": "QmVMBFZqRZDA6TrQkVGGJEDSp5jC3UUMUjLcvaZ3fLCqh4", + "hash": "QmXLroMsvaP8q2W5BPmrLH4fPbXYoS6j23NeybxSNSt94h", "name": "go-libp2p-transport", - "version": "3.0.2" + "version": "3.0.4" }, { "author": "whyrusleeping", - "hash": "QmQkxTEhkY3ZsZJJnh5XqJH9tXrfHucpS8MgM6Jtyt6T5W", + "hash": "QmWAubjUxRp6B4DzfnW1fuJxoFyrEPULyie3StyFJ8kArT", "name": "go-tcp-transport", - "version": "2.0.2" + "version": "2.0.3" }, { "author": "whyrusleeping", @@ -102,51 +102,51 @@ }, { "author": "whyrusleeping", - "hash": "QmPdxCaVp4jZ9RbxqZADvKH6kiCR5jHvdR5f2ycjAY6T2a", + "hash": "QmRerBQ7QDTPjhnnzn4JVTWNpPenJaauT9YiWRbR4MipEP", "name": "go-testutil", - "version": "1.2.3" + "version": "1.2.4" }, { "author": "whyrusleeping", - "hash": "QmXdgNhVEgjLxjUoMs5ViQL7pboAt3Y7V7eGHRiE4qrmTE", + "hash": "Qme1bbEkDJmtCQ2XCN1TNFSGTmcE5rSmBYwJ2VdsMVoziM", "name": "go-libp2p-net", - "version": "3.0.2" + "version": "3.0.4" }, { "author": "whyrusleeping", - "hash": "Qme71fYNbz1wLFqoYjLveUHX1CpPNfGmhBx8tvNKFjqUaC", + "hash": "QmcoBbyTiL9PFjo1GFixJwqQ8mZLJ36CribuqyKmS1okPu", "name": "go-libp2p-metrics", - "version": "2.1.2" + "version": "2.1.3" }, { "author": "whyrusleeping", - "hash": "QmQQGtcp6nVUrQjNsnU53YWV1q8fK1Kd9S7FEkYbRZzxry", + "hash": "QmRBdiF83ZFxEpgN6UtkuGu7phpvDLrXDQqrM8ei7W4owt", "name": "go-libp2p-host", - "version": "3.0.2" + "version": "3.0.3" }, { "author": "whyrusleeping", - "hash": "QmSvhbgtjQJKdT5avEeb7cvjYs7YrhebJyM1K6GAnkKgfd", + "hash": "Qmf1Gzu2dPFCP2svEcRMfXtNupxhd37Zpye2XcY4uhJbpG", "name": "go-libp2p-swarm", - "version": "3.0.2" + "version": "3.0.3" }, { "author": "whyrusleeping", - "hash": "QmVxBEipMPHG5SfX3LYJ7SCzSWySYwoabb6qeWuxfZS5QQ", + "hash": "QmPdbV7V5Nh2kC9J6EGGjHuiGNSLB9TKbfkYUZBVgu4sm1", "name": "go-libp2p-nat", - "version": "0.8.2" + "version": "0.8.3" }, { "author": "whyrusleeping", - "hash": "QmeBUY1BsMjkacVAJ2u76XBGNiRCHq6dkqT2VWG59N3d7b", + "hash": "QmeEZbDUCSMXY3dJ2knx3H8G7EGYJEAFYCeif89BioE5eS", "name": "go-libp2p-netutil", - "version": "0.4.2" + "version": "0.4.3" }, { "author": "whyrusleeping", - "hash": "QmXkocWqNmLkoH6wbYi3dLeZJmTX3gZtVfzWTTyTsuMxje", + "hash": "QmfKx9StZWMskz8Sd4R4ux1DRY2wCCQrM38gr9jmmqrCh7", "name": "go-libp2p-blankhost", - "version": "0.3.2" + "version": "0.3.3" }, { "author": "whyrusleeping", @@ -174,15 +174,15 @@ }, { "author": "whyrusleeping", - "hash": "QmVf8hTAsLLFtn4WPCRNdnaF2Eag2qTBS6uR8AiHPZARXy", + "hash": "QmdVrMn1LhB4ybb8hMVaMLXnA8XRSewMnK6YqXKXoTcRvN", "name": "go-libp2p-peer", - "version": "2.3.4" + "version": "2.3.5" }, { "author": "vyzo", - "hash": "QmPavh4h3Edx5cv8GJQPC35AQAws7faXmzEZyru7k9b9Mn", + "hash": "QmXDRufVdhgmF9CW6Hn6ru7iDBhiebaLNNBMqQoX8oYQs6", "name": "go-libp2p-circuit", - "version": "2.1.2" + "version": "2.1.3" }, { "author": "lgierth", @@ -192,15 +192,15 @@ }, { "author": "why", - "hash": "Qmav3fJzdn43FDvHyGkPdbQ5JVqqiDPmNdnuGa3vatpmwj", + "hash": "QmZhvZjDir4uGxbfEZz8iBVsPgW9VPNNY7RsmvEAH9dTNS", "name": "go-libp2p-interface-connmgr", - "version": "0.0.9" + "version": "0.0.10" }, { "author": "whyrusleeping", - "hash": "QmXKndvAxPwUuHmZDP7qizgNWovCyDkh4y43DiYwLtyQ4d", + "hash": "QmZHiqdRuNXujvSPNu1ZWxxzV6a2WhoZpfYkesdgyaKF9f", "name": "go-smux-multiplex", - "version": "3.0.10" + "version": "3.0.11" }, { "author": "multiformats", @@ -216,21 +216,21 @@ }, { "author": "whyrusleeping", - "hash": "QmQSSwdqHWSVQyGZAaajnMCMpSJDqTG4zMCbqxh6bWTTuf", + "hash": "QmToH3sx9dM26AVNi2AftfcA6ui1XRXcUe65nRrLKL2dgt", "name": "go-ws-transport", - "version": "2.0.2" + "version": "2.0.3" }, { "author": "stebalien", - "hash": "QmeRP5m2PBsVaragzoDqdDSS54nV2T7w8yfiiNZb7jfCMK", + "hash": "QmZor4eKP3zxqFhbjsQjPS4XbZS476cMi8QwPRkaGz9Rh4", "name": "go-conn-security-multistream", - "version": "0.1.2" + "version": "0.1.3" }, { "author": "Stebalien", - "hash": "QmfCQHZGXiEqRgWBmJmWBD8p1rP3Z2T5Y5pvidfGTjsEPs", + "hash": "Qmbdu8oFHUejreAs7ZcrDG77ZyxgcGjk3ZbJAjpMrsBrrL", "name": "go-conn-security", - "version": "0.1.3" + "version": "0.1.4" }, { "author": "libp2p", @@ -246,9 +246,9 @@ }, { "author": "steb", - "hash": "QmdbjG1eui2spsiFLmBjmET6N9c4wfpDzAoKFyN72935Ec", + "hash": "QmbfWNtinqMcNRwyG1z2BvcHHWGPoCuQdZgfkLmZtEwn19", "name": "go-libp2p-transport-upgrader", - "version": "0.1.2" + "version": "0.1.3" } ], "gxVersion": "0.4.0", @@ -256,6 +256,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.2" + "version": "6.0.3" } From ebc7440631bfeccc4cd4da7b92d1f597a7827691 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 8 Jun 2018 10:23:07 -0700 Subject: [PATCH 0502/3965] add a per-dial transport-level dial timeout --- p2p/net/swarm/dial_test.go | 27 ++++++++++++++------------- p2p/net/swarm/swarm.go | 10 +++++----- p2p/net/swarm/swarm_dial.go | 17 ++++++++++------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index a7eebcea1f..dd93bd7cec 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -11,6 +11,7 @@ import ( peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" swarmt "github.com/libp2p/go-libp2p-swarm/testing" + transport "github.com/libp2p/go-libp2p-transport" testutil "github.com/libp2p/go-testutil" ci "github.com/libp2p/go-testutil/ci" ma "github.com/multiformats/go-multiaddr" @@ -20,7 +21,7 @@ import ( ) func init() { - DialTimeout = time.Second + transport.DialTimeout = time.Second } func closeSwarms(swarms []*Swarm) { @@ -190,11 +191,11 @@ func TestDialWait(t *testing.T) { } duration := time.Since(before) - if duration < DialTimeout*DialAttempts { - t.Error("< DialTimeout * DialAttempts not being respected", duration, DialTimeout*DialAttempts) + if duration < transport.DialTimeout*DialAttempts { + t.Error("< transport.DialTimeout * DialAttempts not being respected", duration, transport.DialTimeout*DialAttempts) } - if duration > 2*DialTimeout*DialAttempts { - t.Error("> 2*DialTimeout * DialAttempts not being respected", duration, 2*DialTimeout*DialAttempts) + if duration > 2*transport.DialTimeout*DialAttempts { + t.Error("> 2*transport.DialTimeout * DialAttempts not being respected", duration, 2*transport.DialTimeout*DialAttempts) } if !s1.Backoff().Backoff(s2p) { @@ -278,8 +279,8 @@ func TestDialBackoff(t *testing.T) { s3done := dialOfflineNode(s3p, N) // when all dials should be done by: - dialTimeout1x := time.After(DialTimeout) - dialTimeout10Ax := time.After(DialTimeout * 2 * 10) // DialAttempts * 10) + dialTimeout1x := time.After(transport.DialTimeout) + dialTimeout10Ax := time.After(transport.DialTimeout * 2 * 10) // DialAttempts * 10) // 2) all dials should hang select { @@ -361,8 +362,8 @@ func TestDialBackoff(t *testing.T) { s3done := dialOfflineNode(s3p, N) // when all dials should be done by: - dialTimeout1x := time.After(DialTimeout) - dialTimeout10Ax := time.After(DialTimeout * 2 * 10) // DialAttempts * 10) + dialTimeout1x := time.After(transport.DialTimeout) + dialTimeout10Ax := time.After(transport.DialTimeout * 2 * 10) // DialAttempts * 10) // 7) s3 dials should all return immediately (except 1) for i := 0; i < N-1; i++ { @@ -441,11 +442,11 @@ func TestDialBackoffClears(t *testing.T) { } duration := time.Since(before) - if duration < DialTimeout*DialAttempts { - t.Error("< DialTimeout * DialAttempts not being respected", duration, DialTimeout*DialAttempts) + if duration < transport.DialTimeout*DialAttempts { + t.Error("< transport.DialTimeout * DialAttempts not being respected", duration, transport.DialTimeout*DialAttempts) } - if duration > 2*DialTimeout*DialAttempts { - t.Error("> 2*DialTimeout * DialAttempts not being respected", duration, 2*DialTimeout*DialAttempts) + if duration > 2*transport.DialTimeout*DialAttempts { + t.Error("> 2*transport.DialTimeout * DialAttempts not being respected", duration, 2*transport.DialTimeout*DialAttempts) } if !s1.Backoff().Backoff(s2.LocalPeer()) { diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 002abd2d9f..0a50bed9e6 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -21,11 +21,6 @@ import ( mafilter "github.com/whyrusleeping/multiaddr-filter" ) -// DialTimeout is the maximum duration a Dial is allowed to take. -// This includes the time spent waiting in dial limiter, between dialing the raw -// network connection, protocol selection as well the handshake, if applicable. -var DialTimeout = 60 * time.Second - // DialTimeoutLocal is the maximum duration a Dial to local network address // is allowed to take. // This includes the time between dialing the raw network connection, @@ -42,6 +37,11 @@ var ErrSwarmClosed = errors.New("swarm closed") // transport is misbehaving. var ErrAddrFiltered = errors.New("address filtered") +// DialTimeout is the maximum duration a Dial is allowed to take. +// This includes the time between dialing the raw network connection, +// protocol selection as well the handshake, if applicable. +var DialTimeout = 60 * time.Second + // Swarm is a connection muxer, allowing connections to other peers to // be opened and closed, while still using the same Chan for all // communication. The Chan sends/receives Messages, which note the diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 7125bc3cb2..fe7a4cbfd5 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -221,9 +221,6 @@ func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { return c, nil } - ctx, cancel := context.WithTimeout(ctx, DialTimeout) - defer cancel() - logdial := lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) // ok, we have been charged to dial! let's do it. @@ -259,6 +256,9 @@ func (s *Swarm) canDial(addr ma.Multiaddr) bool { // dial is the actual swarm's dial logic, gated by Dial. func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { + ctx, cancel := context.WithTimeout(ctx, DialTimeout) + defer cancel() + var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) if p == s.local { log.Event(ctx, "swarmDialDoDialSelf", logdial) @@ -398,12 +398,15 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (tra } log.Debugf("%s swarm dialing %s %s", s.local, p, addr) - transport := s.TransportForDialing(addr) - if transport == nil { + tpt := s.TransportForDialing(addr) + if tpt == nil { return nil, ErrNoTransport } - connC, err := transport.Dial(ctx, addr, p) + ctx, cancel := context.WithTimeout(ctx, transport.DialTimeout) + defer cancel() + + connC, err := tpt.Dial(ctx, addr, p) if err != nil { return nil, fmt.Errorf("%s --> %s dial attempt failed: %s", s.local, p, err) } @@ -411,7 +414,7 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (tra // Trust the transport? Yeah... right. if connC.RemotePeer() != p { connC.Close() - err = fmt.Errorf("BUG in transport %T: tried to dial %s, dialed %s", p, connC.RemotePeer(), transport) + err = fmt.Errorf("BUG in transport %T: tried to dial %s, dialed %s", p, connC.RemotePeer(), tpt) log.Error(err) return nil, err } From d2b407f84ddaf3a2572b3242b147b1e36ecf9a94 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 11 Jun 2018 09:14:37 -0700 Subject: [PATCH 0503/3965] don't hang tests on dial failure --- p2p/net/swarm/simul_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index 22b0dbf330..b162ab380f 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -27,13 +27,13 @@ func TestSimultOpen(t *testing.T) { { var wg sync.WaitGroup connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { + defer wg.Done() // copy for other peer log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.LocalPeer(), dst, addr) s.Peerstore().AddAddr(dst, addr, pstore.PermanentAddrTTL) if _, err := s.DialPeer(ctx, dst); err != nil { - t.Fatal("error swarm dialing to peer", err) + t.Error("error swarm dialing to peer", err) } - wg.Done() } log.Info("Connecting swarms simultaneously.") From cbd2028e00fb9926ee9b0f1b49b560b5a44e4868 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 14 Jun 2018 13:26:42 -0700 Subject: [PATCH 0504/3965] remove unused success boolean --- p2p/net/swarm/limiter.go | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 32b75cf6b6..391b828f01 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -18,11 +18,10 @@ type dialResult struct { } type dialJob struct { - addr ma.Multiaddr - peer peer.ID - ctx context.Context - resp chan dialResult - success bool + addr ma.Multiaddr + peer peer.ID + ctx context.Context + resp chan dialResult } func (dj *dialJob) cancelled() bool { @@ -100,7 +99,7 @@ func (dl *dialLimiter) freePeerToken(dj *dialJob) { } waitlist := dl.waitingOnPeerLimit[dj.peer] - if !dj.success && len(waitlist) > 0 { + if len(waitlist) > 0 { next := waitlist[0] if len(waitlist) == 1 { delete(dl.waitingOnPeerLimit, next.peer) From c30ac0e7437909a2c0db367a37b05f59d76c7013 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 14 Jun 2018 15:48:25 -0700 Subject: [PATCH 0505/3965] switch to a per DialPeer/NewStream timeout defined in go-libp2p-net The global per-peer dial timeout had a significant drawback: When dialing many peers, this timeout could cause libp2p to cancel dials while they were still stuck in the limiter. A better but more complicated approach is a time budget system but we can implement that later. This change simply applies the limit to each `DialPeer`/`NewStream` call independently and makes it easy to override. While old timeout tried to account for how much we're willing to spend dialing a single peer, this new timeout tries to account for the amount of time a single "client" is willing to wait for a dial to complete before they no longer care. --- p2p/net/swarm/limiter.go | 2 +- p2p/net/swarm/swarm.go | 5 ----- p2p/net/swarm/swarm_dial.go | 8 +++++--- 3 files changed, 6 insertions(+), 9 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 391b828f01..02aed50a2e 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -34,7 +34,7 @@ func (dj *dialJob) cancelled() bool { } func (dj *dialJob) dialTimeout() time.Duration { - timeout := DialTimeout + timeout := transport.DialTimeout if lowTimeoutFilters.AddrBlocked(dj.addr) { timeout = DialTimeoutLocal } diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 0a50bed9e6..7d795abd86 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -37,11 +37,6 @@ var ErrSwarmClosed = errors.New("swarm closed") // transport is misbehaving. var ErrAddrFiltered = errors.New("address filtered") -// DialTimeout is the maximum duration a Dial is allowed to take. -// This includes the time between dialing the raw network connection, -// protocol selection as well the handshake, if applicable. -var DialTimeout = 60 * time.Second - // Swarm is a connection muxer, allowing connections to other peers to // be opened and closed, while still using the same Chan for all // communication. The Chan sends/receives Messages, which note the diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index fe7a4cbfd5..3e0cfafac2 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -202,10 +202,15 @@ func (s *Swarm) dialPeer(ctx context.Context, p peer.ID) (*Conn, error) { return nil, ErrDialBackoff } + // apply the DialPeer timeout + ctx, cancel := context.WithTimeout(ctx, inet.GetDialPeerTimeout(ctx)) + defer cancel() + conn, err := s.dsync.DialLock(ctx, p) if err != nil { return nil, err } + log.Debugf("network for %s finished dialing %s", s.local, p) return conn, err } @@ -256,9 +261,6 @@ func (s *Swarm) canDial(addr ma.Multiaddr) bool { // dial is the actual swarm's dial logic, gated by Dial. func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { - ctx, cancel := context.WithTimeout(ctx, DialTimeout) - defer cancel() - var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) if p == s.local { log.Event(ctx, "swarmDialDoDialSelf", logdial) From eb986b0c00f21555e6916eb437ed123dcd5b09dc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 25 Jun 2018 20:47:44 -0700 Subject: [PATCH 0506/3965] remove redundant dial timeout This timeout is already set by the dial limiter. --- p2p/net/swarm/swarm_dial.go | 3 --- 1 file changed, 3 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 3e0cfafac2..59adb037cb 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -405,9 +405,6 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (tra return nil, ErrNoTransport } - ctx, cancel := context.WithTimeout(ctx, transport.DialTimeout) - defer cancel() - connC, err := tpt.Dial(ctx, addr, p) if err != nil { return nil, fmt.Errorf("%s --> %s dial attempt failed: %s", s.local, p, err) From f24a6d86d87905f56c94de49cdafbfe1ebb83c7e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 26 Jun 2018 16:05:00 -0700 Subject: [PATCH 0507/3965] gx publish 6.0.4 --- .gx/lastpubver | 2 +- package.json | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 15288eb266..c4c6c7fc5b 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.3: QmddH8SGt2vYLw7hsv7tBfY6XV2rCagtSdZECUnpVftWLr +6.0.4: QmTrr3R3EKbapH1zAGzfaZambsjfskGC9tWqqLWP3yZcwq diff --git a/package.json b/package.json index 3bfe64343a..54c4b83697 100644 --- a/package.json +++ b/package.json @@ -126,9 +126,9 @@ }, { "author": "whyrusleeping", - "hash": "Qmf1Gzu2dPFCP2svEcRMfXtNupxhd37Zpye2XcY4uhJbpG", + "hash": "QmdmbsggnhPra4uue4tQsZKfSYn2i7vcMhraXJJYcrzcMu", "name": "go-libp2p-swarm", - "version": "3.0.3" + "version": "3.0.4" }, { "author": "whyrusleeping", @@ -180,9 +180,9 @@ }, { "author": "vyzo", - "hash": "QmXDRufVdhgmF9CW6Hn6ru7iDBhiebaLNNBMqQoX8oYQs6", + "hash": "QmWJ1iDKinRZ3jMdp4KM5TNoour7SmdGMdiu4r8NSqVJtY", "name": "go-libp2p-circuit", - "version": "2.1.3" + "version": "2.1.4" }, { "author": "lgierth", @@ -256,6 +256,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.3" + "version": "6.0.4" } From cdc6d1940763222a9f338a14903bb383384dc46c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 26 Jun 2018 16:30:07 -0700 Subject: [PATCH 0508/3965] gx publish 6.0.5 --- .gx/lastpubver | 2 +- package.json | 90 +++++++++++++++++++++++++------------------------- 2 files changed, 46 insertions(+), 46 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index c4c6c7fc5b..aa8d7846dc 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.4: QmTrr3R3EKbapH1zAGzfaZambsjfskGC9tWqqLWP3yZcwq +6.0.5: QmZ86eLPtXkQ1Dfa992Q8NpXArUoWWh3y728JDcWvzRrvC diff --git a/package.json b/package.json index 54c4b83697..439a7e24e6 100644 --- a/package.json +++ b/package.json @@ -44,9 +44,9 @@ "version": "1.5.3" }, { - "hash": "QmNqRnejxJxjRroz7buhrjfU8i3yNBLa81hFtmf2pXEffN", + "hash": "QmV6FjemM1K8oXjrvuq3wuVWWoU2TLDPmNnKrxHzY3v6Ai", "name": "go-multiaddr-net", - "version": "1.6.1" + "version": "1.6.3" }, { "hash": "QmZ4Qi3GaRbjcx28Sme5eMH7RQjGkt8wHxt2a65oLaeFEV", @@ -54,45 +54,45 @@ "version": "0.0.0" }, { - "hash": "QmUxSEGbv2nmYNnfXi7839wwQqTN3kwQeUxe8dTjZWZs7J", + "hash": "QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7", "name": "go-multiaddr", - "version": "1.2.7" + "version": "1.3.0" }, { "author": "whyrusleeping", - "hash": "Qmb3WWGVSrvhkDmuajyQWfvNdURCa1q9TVuj6giAGuq3Bk", + "hash": "QmRPkGkHLB72caXgdDYnoaWigXNWx95BcYDKV1n3KTEpaG", "name": "go-libp2p-loggables", - "version": "1.1.17" + "version": "1.1.18" }, { "author": "whyrusleeping", - "hash": "QmeP5zmVqYZGQJfwzLt3Vw4vxiqn4wTwcdgSqD59rndFrF", + "hash": "QmXzYnwYnJiDDV9y4QmbmtSeVyJ1R8HAmGnrnY2D6G3YcV", "name": "go-libp2p-secio", - "version": "2.0.3" + "version": "2.0.4" }, { "author": "whyrusleeping", - "hash": "QmeXmxAE5URbpqgAMAwQiR3zEbd2wvSPUXbdytvFdGNELF", + "hash": "QmZR2XWVVBCtbgBWnQhWk2xcQfaR3W8faQPriAiaaj7rsr", "name": "go-libp2p-peerstore", - "version": "1.4.21" + "version": "1.4.22" }, { "author": "whyrusleeping", - "hash": "QmXLroMsvaP8q2W5BPmrLH4fPbXYoS6j23NeybxSNSt94h", + "hash": "QmW5LxJm2Yo3S7uVsfLM7NsJn2QnKbvZD7uYsZVYR7YViE", "name": "go-libp2p-transport", - "version": "3.0.4" + "version": "3.0.5" }, { "author": "whyrusleeping", - "hash": "QmWAubjUxRp6B4DzfnW1fuJxoFyrEPULyie3StyFJ8kArT", + "hash": "Qmbu73y1Vomwtksx9evxQdgXAjr2pjEsATnGrS7NCk6KLU", "name": "go-tcp-transport", - "version": "2.0.3" + "version": "2.0.4" }, { "author": "whyrusleeping", - "hash": "QmNey9DW3QjsNh7tLfroFhk3994k99PC5Ta6aqCNA6hwYZ", + "hash": "QmSW4uNHbvQia8iZDXzbwjiyHQtnyo9aFqfQAMasj3TJ6Y", "name": "go-maddr-filter", - "version": "1.1.8" + "version": "1.1.9" }, { "author": "whyrusleeping", @@ -102,15 +102,15 @@ }, { "author": "whyrusleeping", - "hash": "QmRerBQ7QDTPjhnnzn4JVTWNpPenJaauT9YiWRbR4MipEP", + "hash": "QmcW4FGAt24fdK1jBgWQn3yP4R9ZLyWQqjozv9QK7epRhL", "name": "go-testutil", - "version": "1.2.4" + "version": "1.2.5" }, { "author": "whyrusleeping", - "hash": "Qme1bbEkDJmtCQ2XCN1TNFSGTmcE5rSmBYwJ2VdsMVoziM", + "hash": "QmPjvxTpVH8qJyQDnxnsxF9kv9jezKD1kozz1hs3fCGsNh", "name": "go-libp2p-net", - "version": "3.0.4" + "version": "3.0.5" }, { "author": "whyrusleeping", @@ -120,33 +120,33 @@ }, { "author": "whyrusleeping", - "hash": "QmRBdiF83ZFxEpgN6UtkuGu7phpvDLrXDQqrM8ei7W4owt", + "hash": "Qmb8T6YBBsjYsVGfrihQLfCJveczZnneSBqBKkYEBWDjge", "name": "go-libp2p-host", - "version": "3.0.3" + "version": "3.0.4" }, { "author": "whyrusleeping", - "hash": "QmdmbsggnhPra4uue4tQsZKfSYn2i7vcMhraXJJYcrzcMu", + "hash": "QmVqCSwuzgDfhLMTmFfUePTGX78PBjzuHcbSWWNPrnrmKy", "name": "go-libp2p-swarm", - "version": "3.0.4" + "version": "3.0.5" }, { "author": "whyrusleeping", - "hash": "QmPdbV7V5Nh2kC9J6EGGjHuiGNSLB9TKbfkYUZBVgu4sm1", + "hash": "QmaAXUjzKjKGAymaTJVZ6dkW8zqfLtwxFjCLvteh4fM4aC", "name": "go-libp2p-nat", - "version": "0.8.3" + "version": "0.8.4" }, { "author": "whyrusleeping", - "hash": "QmeEZbDUCSMXY3dJ2knx3H8G7EGYJEAFYCeif89BioE5eS", + "hash": "QmcxUtMB5sJrXR3znSvkrDd2ghvwGM8rLRqwJiPUdgQwat", "name": "go-libp2p-netutil", - "version": "0.4.3" + "version": "0.4.4" }, { "author": "whyrusleeping", - "hash": "QmfKx9StZWMskz8Sd4R4ux1DRY2wCCQrM38gr9jmmqrCh7", + "hash": "QmT1Q44YeQXSiK3LBXbXBRdxJAFZZSTarseqt7DVe9ASNA", "name": "go-libp2p-blankhost", - "version": "0.3.3" + "version": "0.3.4" }, { "author": "whyrusleeping", @@ -180,21 +180,21 @@ }, { "author": "vyzo", - "hash": "QmWJ1iDKinRZ3jMdp4KM5TNoour7SmdGMdiu4r8NSqVJtY", + "hash": "QmUTvbhkVteiGNkYJ5CN953NtGTPbgctcLk2WU1JVDV6TR", "name": "go-libp2p-circuit", - "version": "2.1.4" + "version": "2.1.5" }, { "author": "lgierth", - "hash": "QmT8461vVVyBPyHJHQ6mvm8UdQ8UZNA5n6Z7kBk7GRf1xu", + "hash": "QmfXU2MhWoegxHoeMd3A2ytL2P6CY4FfqGWc23LTNWBwZt", "name": "go-multiaddr-dns", - "version": "0.2.3" + "version": "0.2.4" }, { "author": "why", - "hash": "QmZhvZjDir4uGxbfEZz8iBVsPgW9VPNNY7RsmvEAH9dTNS", + "hash": "QmXuucFcuvAWYAJfhHV2h4BYreHEAsLSsiquosiXeuduTN", "name": "go-libp2p-interface-connmgr", - "version": "0.0.10" + "version": "0.0.11" }, { "author": "whyrusleeping", @@ -216,21 +216,21 @@ }, { "author": "whyrusleeping", - "hash": "QmToH3sx9dM26AVNi2AftfcA6ui1XRXcUe65nRrLKL2dgt", + "hash": "QmUTQ9rzQf52c4MU4ESFbTF8RaUKtQztcAp6ww8wF3ahxC", "name": "go-ws-transport", - "version": "2.0.3" + "version": "2.0.4" }, { "author": "stebalien", - "hash": "QmZor4eKP3zxqFhbjsQjPS4XbZS476cMi8QwPRkaGz9Rh4", + "hash": "QmZaZ4nkadScQtG7KaGUWyh24jz5tD8d6AyetJmGnMo49t", "name": "go-conn-security-multistream", - "version": "0.1.3" + "version": "0.1.4" }, { "author": "Stebalien", - "hash": "Qmbdu8oFHUejreAs7ZcrDG77ZyxgcGjk3ZbJAjpMrsBrrL", + "hash": "QmYMGCwtKbLXfh2MUTbLoP66N9Y24DSg8PFJRG21eE2qwv", "name": "go-conn-security", - "version": "0.1.4" + "version": "0.1.5" }, { "author": "libp2p", @@ -246,9 +246,9 @@ }, { "author": "steb", - "hash": "QmbfWNtinqMcNRwyG1z2BvcHHWGPoCuQdZgfkLmZtEwn19", + "hash": "QmbzWHWFpwCm7GDgrGyKnqUzh48PnRp86VD2Bb8wq6ZfDP", "name": "go-libp2p-transport-upgrader", - "version": "0.1.3" + "version": "0.1.4" } ], "gxVersion": "0.4.0", @@ -256,6 +256,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.4" + "version": "6.0.5" } From 9b46b6e50d5b9258911ba995d3dba5e79bf54da4 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Wed, 27 Jun 2018 23:04:53 +0200 Subject: [PATCH 0509/3965] gx publish 0.0.10 --- p2p/net/gostream/gostream_test.go | 36 +++++-------------------------- 1 file changed, 5 insertions(+), 31 deletions(-) diff --git a/p2p/net/gostream/gostream_test.go b/p2p/net/gostream/gostream_test.go index 60d20f57b4..d2e6937c9e 100644 --- a/p2p/net/gostream/gostream_test.go +++ b/p2p/net/gostream/gostream_test.go @@ -7,50 +7,24 @@ import ( "testing" "time" - crypto "github.com/libp2p/go-libp2p-crypto" + libp2p "github.com/libp2p/go-libp2p" host "github.com/libp2p/go-libp2p-host" - peer "github.com/libp2p/go-libp2p-peer" peerstore "github.com/libp2p/go-libp2p-peerstore" protocol "github.com/libp2p/go-libp2p-protocol" - swarm "github.com/libp2p/go-libp2p-swarm" - bhost "github.com/libp2p/go-libp2p/p2p/host/basic" multiaddr "github.com/multiformats/go-multiaddr" ) // newHost illustrates how to build a libp2p host with secio using // a randomly generated key-pair func newHost(t *testing.T, listen multiaddr.Multiaddr) host.Host { - priv, pub, err := crypto.GenerateKeyPair(crypto.RSA, 2048) - if err != nil { - t.Fatal(err) - } - pid, err := peer.IDFromPublicKey(pub) - if err != nil { - t.Fatal(err) - } - ps := peerstore.NewPeerstore() - err = ps.AddPubKey(pid, pub) - if err != nil { - t.Fatal(err) - } - err = ps.AddPrivKey(pid, priv) - if err != nil { - t.Fatal(err) - } - - network, err := swarm.NewNetwork( + h, err := libp2p.New( context.Background(), - []multiaddr.Multiaddr{listen}, - pid, - ps, - nil) - + libp2p.ListenAddrs(listen), + ) if err != nil { t.Fatal(err) } - - host := bhost.New(network) - return host + return h } func TestServerClient(t *testing.T) { From 61782c904c2d4b13ce42d526cc322ce564af7c37 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Thu, 28 Jun 2018 01:02:23 +0200 Subject: [PATCH 0510/3965] Reset streams when no reading is going to be done anymore --- p2p/net/gostream/conn.go | 2 +- p2p/net/gostream/listener.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/gostream/conn.go b/p2p/net/gostream/conn.go index 1351beec60..5d28cd3047 100644 --- a/p2p/net/gostream/conn.go +++ b/p2p/net/gostream/conn.go @@ -35,7 +35,7 @@ func (c *conn) Write(b []byte) (n int, err error) { // Close closes the connection. // Any blocked Read or Write operations will be unblocked and return errors. func (c *conn) Close() error { - return c.s.Close() + return c.s.Reset() } // LocalAddr returns the local network address. diff --git a/p2p/net/gostream/listener.go b/p2p/net/gostream/listener.go index 9f37a956ca..e66c262a70 100644 --- a/p2p/net/gostream/listener.go +++ b/p2p/net/gostream/listener.go @@ -63,7 +63,7 @@ func Listen(h host.Host, tag protocol.ID) (net.Listener, error) { select { case l.streamCh <- s: case <-ctx.Done(): - s.Close() + s.Reset() } }) From d33d7e8c28500b79d3bac836d1329ff62593dca9 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Thu, 28 Jun 2018 01:17:33 +0200 Subject: [PATCH 0511/3965] Reset streams: do not close server while reading --- p2p/net/gostream/gostream_test.go | 40 ++++++++++++++++++------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/p2p/net/gostream/gostream_test.go b/p2p/net/gostream/gostream_test.go index d2e6937c9e..cdbd5be124 100644 --- a/p2p/net/gostream/gostream_test.go +++ b/p2p/net/gostream/gostream_test.go @@ -3,7 +3,6 @@ package gostream import ( "bufio" "context" - "io/ioutil" "testing" "time" @@ -39,8 +38,10 @@ func TestServerClient(t *testing.T) { clientHost.Peerstore().AddAddrs(srvHost.ID(), srvHost.Addrs(), peerstore.PermanentAddrTTL) var tag protocol.ID = "/testitytest" + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() - go func() { + go func(ctx context.Context) { listener, err := Listen(srvHost, tag) if err != nil { t.Fatal(err) @@ -58,19 +59,25 @@ func TestServerClient(t *testing.T) { defer servConn.Close() reader := bufio.NewReader(servConn) - msg, err := reader.ReadString('\n') - if err != nil { - t.Fatal(err) - } - if string(msg) != "is libp2p awesome?\n" { - t.Fatalf("Bad incoming message: %s", msg) - } - - _, err = servConn.Write([]byte("yes it is")) - if err != nil { - t.Fatal(err) + for { + msg, err := reader.ReadString('\n') + if err != nil { + t.Fatal(err) + } + if string(msg) != "is libp2p awesome?\n" { + t.Fatalf("Bad incoming message: %s", msg) + } + + _, err = servConn.Write([]byte("yes it is\n")) + if err != nil { + t.Fatal(err) + } + select { + case <-ctx.Done(): + return + } } - }() + }(ctx) clientConn, err := Dial(clientHost, srvHost.ID(), tag) if err != nil { @@ -109,12 +116,13 @@ func TestServerClient(t *testing.T) { t.Fatal(err) } - resp, err := ioutil.ReadAll(clientConn) + reader := bufio.NewReader(clientConn) + resp, err := reader.ReadString('\n') if err != nil { t.Fatal(err) } - if string(resp) != "yes it is" { + if string(resp) != "yes it is\n" { t.Errorf("Bad response: %s", resp) } From d0d288b9d2fc599f5fb50dd019dc3fd63e135b8a Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Thu, 28 Jun 2018 02:13:42 +0200 Subject: [PATCH 0512/3965] Do not reset streams on close immediately. Close first. The reset after 1 min --- p2p/net/gostream/conn.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/p2p/net/gostream/conn.go b/p2p/net/gostream/conn.go index 5d28cd3047..dc5050ffaa 100644 --- a/p2p/net/gostream/conn.go +++ b/p2p/net/gostream/conn.go @@ -35,7 +35,12 @@ func (c *conn) Write(b []byte) (n int, err error) { // Close closes the connection. // Any blocked Read or Write operations will be unblocked and return errors. func (c *conn) Close() error { - return c.s.Reset() + if err := c.s.Close(); err != nil { + c.s.Reset() + return err + } + go pnet.AwaitEOF(c.s) + return nil } // LocalAddr returns the local network address. From 934606d0f292f265eab890101233d18ffeabfda0 Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 6 Jul 2018 19:16:45 +0200 Subject: [PATCH 0513/3965] docs: update logo url --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b44b25dec3..7aa6f366bd 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@

- libp2p hex logo + libp2p hex logo

The Go implementation of the libp2p Networking Stack.

From c75c51aecbfbc1a491a5e39c0f49eac40f566d0f Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 10 Jul 2018 12:25:11 +0200 Subject: [PATCH 0514/3965] use QUIC Keep Alives --- p2p/transport/quic/transport.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 814906b4d7..70f2d6d357 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -24,6 +24,7 @@ var quicConfig = &quic.Config{ // TODO(#6): require source address validation when under load return true }, + KeepAlive: true, } // The Transport implements the tpt.Transport interface for QUIC connections. From 67fb60d95b49f83cdd02fe7bec01ca17431cc09c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 12 Jul 2018 13:25:02 +0200 Subject: [PATCH 0515/3965] switch to google's uuid library --- .gx/post-install | 0 examples/multipro/echo.go | 4 ++-- examples/multipro/ping.go | 4 ++-- package.json | 12 ++++++------ 4 files changed, 10 insertions(+), 10 deletions(-) create mode 100644 .gx/post-install diff --git a/.gx/post-install b/.gx/post-install new file mode 100644 index 0000000000..e69de29bb2 diff --git a/examples/multipro/echo.go b/examples/multipro/echo.go index 48403d7f72..a3d39f8b89 100644 --- a/examples/multipro/echo.go +++ b/examples/multipro/echo.go @@ -8,10 +8,10 @@ import ( inet "github.com/libp2p/go-libp2p-net" + uuid "github.com/google/uuid" "github.com/libp2p/go-libp2p-host" pb "github.com/libp2p/go-libp2p/examples/multipro/pb" protobufCodec "github.com/multiformats/go-multicodec/protobuf" - uuid "github.com/satori/go.uuid" ) // pattern: /protocol-name/request-or-response-message/version @@ -126,7 +126,7 @@ func (e *EchoProtocol) Echo(host host.Host) bool { // create message data req := &pb.EchoRequest{ - MessageData: e.node.NewMessageData(uuid.Must(uuid.NewV4()).String(), false), + MessageData: e.node.NewMessageData(uuid.New().String(), false), Message: fmt.Sprintf("Echo from %s", e.node.ID())} signature, err := e.node.signProtoMessage(req) diff --git a/examples/multipro/ping.go b/examples/multipro/ping.go index bcc63dc1cf..f342aa69b3 100644 --- a/examples/multipro/ping.go +++ b/examples/multipro/ping.go @@ -6,11 +6,11 @@ import ( "fmt" "log" + uuid "github.com/google/uuid" "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" p2p "github.com/libp2p/go-libp2p/examples/multipro/pb" protobufCodec "github.com/multiformats/go-multicodec/protobuf" - uuid "github.com/satori/go.uuid" ) // pattern: /protocol-name/request-or-response-message/version @@ -116,7 +116,7 @@ func (p *PingProtocol) Ping(host host.Host) bool { log.Printf("%s: Sending ping to: %s....", p.node.ID(), host.ID()) // create message data - req := &p2p.PingRequest{MessageData: p.node.NewMessageData(uuid.Must(uuid.NewV4()).String(), false), + req := &p2p.PingRequest{MessageData: p.node.NewMessageData(uuid.New().String(), false), Message: fmt.Sprintf("Ping from %s", p.node.ID())} // sign the data diff --git a/package.json b/package.json index 439a7e24e6..74760290b2 100644 --- a/package.json +++ b/package.json @@ -208,12 +208,6 @@ "name": "go-multicodec", "version": "0.1.5" }, - { - "author": "satori", - "hash": "QmcBWojPoNh4qm7zvv4qiepvCnnc7ALS9qcp7TNwwxT1gT", - "name": "go.uuid", - "version": "1.1.0" - }, { "author": "whyrusleeping", "hash": "QmUTQ9rzQf52c4MU4ESFbTF8RaUKtQztcAp6ww8wF3ahxC", @@ -249,6 +243,12 @@ "hash": "QmbzWHWFpwCm7GDgrGyKnqUzh48PnRp86VD2Bb8wq6ZfDP", "name": "go-libp2p-transport-upgrader", "version": "0.1.4" + }, + { + "author": "google", + "hash": "QmSSeQqc5QeuefkaM6JFV5tSF9knLUkXKVhW1eYRiqe72W", + "name": "uuid", + "version": "0.1.0" } ], "gxVersion": "0.4.0", From 835abcbfe298fccad4270b33aca6213dd350d37b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 20 Jul 2018 19:41:04 -0700 Subject: [PATCH 0516/3965] gx: update go-libp2p-transport-upgrader, go-libp2p-nat, mdns --- .gx/post-install | 0 package.json | 28 ++++++++++++++-------------- 2 files changed, 14 insertions(+), 14 deletions(-) delete mode 100644 .gx/post-install diff --git a/.gx/post-install b/.gx/post-install deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/package.json b/package.json index 74760290b2..79249bb440 100644 --- a/package.json +++ b/package.json @@ -14,9 +14,9 @@ "version": "0.0.0" }, { - "hash": "QmWSvDKkcno2UyDg13rUBwWfhRsdj7uR3daAq57VoG5QeN", + "hash": "QmNeRzgrSpwbMU7VKLRyfvbqf1nRrAiQ7fEXaBxWGT5Ygr", "name": "mdns", - "version": "0.1.1" + "version": "0.1.3" }, { "hash": "QmPdKqUcHGFdeSpvjVoaTRPPstGif9GBZb5Q56RVw9o69A", @@ -84,9 +84,9 @@ }, { "author": "whyrusleeping", - "hash": "Qmbu73y1Vomwtksx9evxQdgXAjr2pjEsATnGrS7NCk6KLU", + "hash": "QmSggepaKhRfgEDiz2B6eGaCTK3TqDHXzK6F6HfsU7KPXt", "name": "go-tcp-transport", - "version": "2.0.4" + "version": "2.0.5" }, { "author": "whyrusleeping", @@ -126,15 +126,15 @@ }, { "author": "whyrusleeping", - "hash": "QmVqCSwuzgDfhLMTmFfUePTGX78PBjzuHcbSWWNPrnrmKy", + "hash": "QmemVjhp1UuWPQqrWSvPcaqH3QJRMjMqNm4T2RULMkDDQe", "name": "go-libp2p-swarm", - "version": "3.0.5" + "version": "3.0.6" }, { "author": "whyrusleeping", - "hash": "QmaAXUjzKjKGAymaTJVZ6dkW8zqfLtwxFjCLvteh4fM4aC", + "hash": "QmW25V77H8nb1ucUe8BDUbR3QXLNzTearBLYMW5DqDVWd5", "name": "go-libp2p-nat", - "version": "0.8.4" + "version": "0.8.5" }, { "author": "whyrusleeping", @@ -180,9 +180,9 @@ }, { "author": "vyzo", - "hash": "QmUTvbhkVteiGNkYJ5CN953NtGTPbgctcLk2WU1JVDV6TR", + "hash": "QmcQ56iqKP8ZRhRGLe5EReJVvrJZDaGzkuatrPv4Z1B6cG", "name": "go-libp2p-circuit", - "version": "2.1.5" + "version": "2.1.6" }, { "author": "lgierth", @@ -210,9 +210,9 @@ }, { "author": "whyrusleeping", - "hash": "QmUTQ9rzQf52c4MU4ESFbTF8RaUKtQztcAp6ww8wF3ahxC", + "hash": "QmX4UrmHGPnFxwfsunZjPbykzyv8Frg9AVmNariXqrLsMs", "name": "go-ws-transport", - "version": "2.0.4" + "version": "2.0.5" }, { "author": "stebalien", @@ -240,9 +240,9 @@ }, { "author": "steb", - "hash": "QmbzWHWFpwCm7GDgrGyKnqUzh48PnRp86VD2Bb8wq6ZfDP", + "hash": "Qma7AuxEA7dd1wAy95hTxXgxy4q7mU4Pyd1x4PRAzGP1fs", "name": "go-libp2p-transport-upgrader", - "version": "0.1.4" + "version": "0.1.5" }, { "author": "google", From 9e5ef7a4e6ced5eb5eb9982b2a0541b331ba5387 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 20 Jul 2018 19:45:25 -0700 Subject: [PATCH 0517/3965] gx publish 6.0.6 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index aa8d7846dc..e964909df0 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.5: QmZ86eLPtXkQ1Dfa992Q8NpXArUoWWh3y728JDcWvzRrvC +6.0.6: QmY51bqSM5XgxQZqsBrQcRkKTnCb8EKpJpR9K6Qax7Njco diff --git a/package.json b/package.json index 79249bb440..1df54e44e0 100644 --- a/package.json +++ b/package.json @@ -256,6 +256,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.5" + "version": "6.0.6" } From f82d4ec6aaaa38116f059b85d75fce329d117e94 Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Wed, 25 Jul 2018 21:50:54 +0530 Subject: [PATCH 0518/3965] default listener --- defaults.go | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/defaults.go b/defaults.go index d3fa618d6f..7d68eae7ae 100644 --- a/defaults.go +++ b/defaults.go @@ -10,6 +10,7 @@ import ( secio "github.com/libp2p/go-libp2p-secio" tcp "github.com/libp2p/go-tcp-transport" ws "github.com/libp2p/go-ws-transport" + multiaddr "github.com/multiformats/go-multiaddr" mplex "github.com/whyrusleeping/go-smux-multiplex" yamux "github.com/whyrusleeping/go-smux-yamux" ) @@ -52,6 +53,14 @@ var RandomIdentity = func(cfg *Config) error { return cfg.Apply(Identity(priv)) } +var DefaultListenAddrs = func(cfg *Config) error { + defaultListenAddr, err := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/0") + if err != nil { + return err + } + return cfg.Apply(ListenAddrs(defaultListenAddr)) +} + // Complete list of default options and when to fallback on them. // // Please *DON'T* specify default options any other way. Putting this all here @@ -80,6 +89,10 @@ var defaults = []struct { fallback: func(cfg *Config) bool { return cfg.Peerstore == nil }, opt: DefaultPeerstore, }, + { + fallback: func(cfg *Config) bool { return cfg.ListenAddrs == nil }, + opt: DefaultListenAddrs, + }, } // Defaults configures libp2p to use the default options. Can be combined with From d104ff8dea24608646e83c39e95e0acea494f02d Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Wed, 25 Jul 2018 22:02:37 +0530 Subject: [PATCH 0519/3965] making go fmt happy --- defaults.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/defaults.go b/defaults.go index 7d68eae7ae..cca571edef 100644 --- a/defaults.go +++ b/defaults.go @@ -21,7 +21,7 @@ import ( // security protocols. var DefaultSecurity = Security(secio.ID, secio.New) -// DefaultMuxer configures libp2p to use the stream connection multiplexers. +// DefaultMuxers configures libp2p to use the stream connection multiplexers. // // Use this option when you want to *extend* the set of multiplexers used by // libp2p instead of replacing them. @@ -53,6 +53,7 @@ var RandomIdentity = func(cfg *Config) error { return cfg.Apply(Identity(priv)) } +// DefaultListenAddrs configures libp2p to use default listen address var DefaultListenAddrs = func(cfg *Config) error { defaultListenAddr, err := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/0") if err != nil { @@ -91,7 +92,7 @@ var defaults = []struct { }, { fallback: func(cfg *Config) bool { return cfg.ListenAddrs == nil }, - opt: DefaultListenAddrs, + opt: DefaultListenAddrs, }, } From 836d042f547e1d48211205b979320c544c04e7d3 Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Wed, 25 Jul 2018 23:42:03 +0530 Subject: [PATCH 0520/3965] added ipv6 + checking for default transport --- defaults.go | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/defaults.go b/defaults.go index cca571edef..973cf5c168 100644 --- a/defaults.go +++ b/defaults.go @@ -55,11 +55,19 @@ var RandomIdentity = func(cfg *Config) error { // DefaultListenAddrs configures libp2p to use default listen address var DefaultListenAddrs = func(cfg *Config) error { - defaultListenAddr, err := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/0") + defaultIP4ListenAddr, err := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/0") if err != nil { return err } - return cfg.Apply(ListenAddrs(defaultListenAddr)) + + defaultIP6ListenAddr, err := multiaddr.NewMultiaddr("/ip6/0.0.0.0/tcp/0") + if err != nil { + return err + } + return cfg.Apply(ListenAddrs( + defaultIP4ListenAddr, + defaultIP6ListenAddr, + )) } // Complete list of default options and when to fallback on them. @@ -70,6 +78,10 @@ var defaults = []struct { fallback func(cfg *Config) bool opt Option }{ + { + fallback: func(cfg *Config) bool { return cfg.Transports == nil && cfg.ListenAddrs == nil }, + opt: DefaultListenAddrs, + }, { fallback: func(cfg *Config) bool { return cfg.Transports == nil }, opt: DefaultTransports, @@ -90,10 +102,6 @@ var defaults = []struct { fallback: func(cfg *Config) bool { return cfg.Peerstore == nil }, opt: DefaultPeerstore, }, - { - fallback: func(cfg *Config) bool { return cfg.ListenAddrs == nil }, - opt: DefaultListenAddrs, - }, } // Defaults configures libp2p to use the default options. Can be combined with From fb4019575dd1cfbfe22b24efeef6bf82faa936e8 Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Thu, 26 Jul 2018 01:08:20 +0530 Subject: [PATCH 0521/3965] Added test for DefaultListenAddrs --- libp2p_test.go | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/libp2p_test.go b/libp2p_test.go index 1b38bbc369..f4f4427643 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -8,6 +8,7 @@ import ( crypto "github.com/libp2p/go-libp2p-crypto" host "github.com/libp2p/go-libp2p-host" + "github.com/libp2p/go-tcp-transport" ) func TestNewHost(t *testing.T) { @@ -39,6 +40,39 @@ func TestInsecure(t *testing.T) { h.Close() } +func TestDefaultListenAddrs(t *testing.T) { + ctx := context.Background() + + // Test 1: Listen addr should not set if user defined transport is passed. + h, err := New( + ctx, + Transport(tcp.TcpTransport{}), + ) + if err != nil { + t.Fatal(err) + } + + if len(h.Addrs()) != 0 { + t.Error("expected zero listen addrs as none is set with user defined transport") + } + h.Close() + + // Test 2: User defined listener addrs should overwrite the default options. + h, err = New( + ctx, + Transport(tcp.TcpTransport{}), + ListenAddrStrings("/ip4/127.0.0.1/tcp/0"), + ) + if err != nil { + t.Fatal(err) + } + + if len(h.Addrs()) != 1 { + t.Error("expected one listen addr") + } + h.Close() +} + func makeRandomHost(t *testing.T, port int) (host.Host, error) { ctx := context.Background() priv, _, err := crypto.GenerateKeyPair(crypto.RSA, 2048) From 29af134a7103e8ae1c8660a14ddc533c0115ae40 Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Thu, 26 Jul 2018 01:14:09 +0530 Subject: [PATCH 0522/3965] corrected transport init --- libp2p_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libp2p_test.go b/libp2p_test.go index f4f4427643..7662fcf67b 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -46,7 +46,7 @@ func TestDefaultListenAddrs(t *testing.T) { // Test 1: Listen addr should not set if user defined transport is passed. h, err := New( ctx, - Transport(tcp.TcpTransport{}), + Transport(tcp.NewTCPTransport), ) if err != nil { t.Fatal(err) @@ -60,7 +60,7 @@ func TestDefaultListenAddrs(t *testing.T) { // Test 2: User defined listener addrs should overwrite the default options. h, err = New( ctx, - Transport(tcp.TcpTransport{}), + Transport(tcp.NewTCPTransport), ListenAddrStrings("/ip4/127.0.0.1/tcp/0"), ) if err != nil { From 6372f928d4f214d5f32bb73c860f00db2909a53f Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Fri, 27 Jul 2018 02:26:29 +0530 Subject: [PATCH 0523/3965] Correct behaviour test for DefaultListenAddrs --- libp2p_test.go | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/libp2p_test.go b/libp2p_test.go index 7662fcf67b..20370d540c 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -43,8 +43,18 @@ func TestInsecure(t *testing.T) { func TestDefaultListenAddrs(t *testing.T) { ctx := context.Background() - // Test 1: Listen addr should not set if user defined transport is passed. - h, err := New( + // Test 1: Setting the correct listen addresses if userDefined.Transport == nil && userDefined.ListenAddrs == nil + h, err := New(ctx) + if err != nil { + t.Fatal(err) + } + if len(h.Addrs()) != 2 { + t.Error("expected 2 default listen addrs") + } + h.Close() + + // Test 2: Listen addr should not set if user defined transport is passed. + h, err = New( ctx, Transport(tcp.NewTCPTransport), ) @@ -57,7 +67,7 @@ func TestDefaultListenAddrs(t *testing.T) { } h.Close() - // Test 2: User defined listener addrs should overwrite the default options. + // Test 3: User defined listener addrs should overwrite the default options. h, err = New( ctx, Transport(tcp.NewTCPTransport), From 72df25af76e4c5019c80893d3b5664e15ce53d96 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 27 Jul 2018 10:54:34 -0700 Subject: [PATCH 0524/3965] always reset ping streams when done Currently, we leak a stream. We could use FullClose but, IMO, it's not worth it. We might as well just reset and walk away. --- p2p/protocol/ping/ping.go | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/p2p/protocol/ping/ping.go b/p2p/protocol/ping/ping.go index 54464f3080..ae0c47f75d 100644 --- a/p2p/protocol/ping/ping.go +++ b/p2p/protocol/ping/ping.go @@ -44,19 +44,14 @@ func (p *PingService) PingHandler(s inet.Stream) { select { case <-timer.C: log.Debug("ping timeout") - s.Reset() case err, ok := <-errCh: if ok { log.Debug(err) - if err == io.EOF { - s.Close() - } else { - s.Reset() - } } else { log.Error("ping loop failed without error") } } + s.Reset() }() for { @@ -85,7 +80,7 @@ func (ps *PingService) Ping(ctx context.Context, p peer.ID) (<-chan time.Duratio out := make(chan time.Duration) go func() { defer close(out) - defer s.Close() + defer s.Reset() for { select { case <-ctx.Done(): @@ -93,7 +88,6 @@ func (ps *PingService) Ping(ctx context.Context, p peer.ID) (<-chan time.Duratio default: t, err := ping(s) if err != nil { - s.Reset() log.Debugf("ping error: %s", err) return } From 0ee6c5a2180d9f2b2895a49ede5bb1ce4ec5f498 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 27 Jul 2018 11:41:28 -0700 Subject: [PATCH 0525/3965] extract libp2p examples to go-libp2p-examples Repo: https://github.com/libp2p/go-lib2p-examples --- .travis.yml | 2 +- Makefile | 4 - README.md | 2 +- examples/README.md | 16 +- examples/chat/README.md | 49 ---- examples/chat/chat.go | 221 -------------- examples/echo/README.md | 52 ---- examples/echo/main.go | 178 ----------- examples/http-proxy/README.md | 58 ---- examples/http-proxy/proxy.go | 276 ------------------ examples/libp2p-host/README.md | 63 ---- examples/libp2p-host/host.go | 47 --- examples/multipro/AUTHORS | 3 - examples/multipro/LICENSE | 21 -- examples/multipro/README.md | 62 ---- examples/multipro/echo.go | 157 ---------- examples/multipro/main.go | 56 ---- examples/multipro/node.go | 150 ---------- examples/multipro/pb/p2p.pb.go | 121 -------- examples/multipro/pb/p2p.proto | 56 ---- examples/multipro/pb/readme.md | 4 - examples/multipro/ping.go | 148 ---------- .../README.md | 41 --- .../main.go | 171 ----------- package.json | 18 -- 25 files changed, 4 insertions(+), 1972 deletions(-) delete mode 100644 examples/chat/README.md delete mode 100644 examples/chat/chat.go delete mode 100644 examples/echo/README.md delete mode 100644 examples/echo/main.go delete mode 100644 examples/http-proxy/README.md delete mode 100644 examples/http-proxy/proxy.go delete mode 100644 examples/libp2p-host/README.md delete mode 100644 examples/libp2p-host/host.go delete mode 100644 examples/multipro/AUTHORS delete mode 100644 examples/multipro/LICENSE delete mode 100644 examples/multipro/README.md delete mode 100644 examples/multipro/echo.go delete mode 100644 examples/multipro/main.go delete mode 100644 examples/multipro/node.go delete mode 100644 examples/multipro/pb/p2p.pb.go delete mode 100644 examples/multipro/pb/p2p.proto delete mode 100644 examples/multipro/pb/readme.md delete mode 100644 examples/multipro/ping.go delete mode 100644 examples/protocol-multiplexing-with-multicodecs/README.md delete mode 100644 examples/protocol-multiplexing-with-multicodecs/main.go diff --git a/.travis.yml b/.travis.yml index 7be88a9369..e1be0bfa30 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,7 +10,7 @@ go: - 1.9.x install: - - make deps-protocol-muxing + - make deps script: - bash <(curl -s https://raw.githubusercontent.com/ipfs/ci-helpers/master/travis-ci/run-standard-tests.sh) diff --git a/Makefile b/Makefile index 075e51f7f5..f9dabde9b2 100644 --- a/Makefile +++ b/Makefile @@ -2,10 +2,6 @@ gx: go get github.com/whyrusleeping/gx go get github.com/whyrusleeping/gx-go -deps-protocol-muxing: deps - go get -u github.com/multiformats/go-multicodec - go get -u github.com/libp2p/go-msgio - deps: gx gx --verbose install --global gx-go rewrite diff --git a/README.md b/README.md index 7aa6f366bd..044e9290ed 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,7 @@ There is currently only one bundle of `go-libp2p`, this package. This bundle is ### Examples -Examples can be found on the [examples folder](examples). +Examples can be found in the [examples repo](https://github.com/libp2p/go-libp2p-examples). ## Development diff --git a/examples/README.md b/examples/README.md index a47661a635..c5e2e94318 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,15 +1,3 @@ +# go-libp2p Examples -# `go-libp2p` examples and tutorials - -In this folder, you can find a variety of examples to help you get started in using go-libp2p. Every example as a specific purpose and some of each incorporate a full tutorial that you can follow through, helping you expand your knowledge about libp2p and p2p networks in general. - -Let us know if you find any issue or if you want to contribute and add a new tutorial, feel welcome to submit a pr, thank you! - -## Examples and Tutorials - -- [The libp2p 'host'](./libp2p-host) -- [Building an http proxy with libp2p](./http-proxy) -- [Protocol Multiplexing with multicodecs](./protocol-multiplexing-with-multicodecs) -- [An echo host](./echo) -- [Multicodecs with protobufs](./multipro) -- [P2P chat application](./chat) \ No newline at end of file +The go-libp2p examples have moved to [go-libp2p-examples](https://github.com/libp2p/go-libp2p-examples). diff --git a/examples/chat/README.md b/examples/chat/README.md deleted file mode 100644 index da01591db2..0000000000 --- a/examples/chat/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# p2p chat app with libp2p - -This program demonstrates a simple p2p chat application. It can work between two peers if -1. Both have private IP address (same network). -2. At least one of them has a public IP address. - -Assume if 'A' and 'B' are on different networks host 'A' may or may not have a public IP address but host 'B' has one. - -Usage: Run `./chat -sp ` on host 'B' where can be any port number. Now run `./chat -d ` on node 'A' [`` is multiaddress of host 'B' which can be obtained from host 'B' console]. - -## Build - -To build the example, first run `make deps` in the root directory. - -``` -> make deps -> go build ./examples/chat -``` - -## Usage - -On node 'B' - -``` -> ./chat -sp 3001 -Run ./chat -d /ip4/127.0.0.1/tcp/3001/ipfs/QmdXGaeGiVA745XorV1jr11RHxB9z4fqykm6xCUPX1aTJo - -2018/02/27 01:21:32 Got a new stream! -> hi (received messages in green colour) -> hello (sent messages in white colour) -> no -``` - -On node 'A'. Replace 127.0.0.1 with if node 'B' has one. - -``` -> ./chat -d /ip4/127.0.0.1/tcp/3001/ipfs/QmdXGaeGiVA745XorV1jr11RHxB9z4fqykm6xCUPX1aTJo -Run ./chat -d /ip4/127.0.0.1/tcp/3001/ipfs/QmdXGaeGiVA745XorV1jr11RHxB9z4fqykm6xCUPX1aTJo - -This node's multiaddress: -/ip4/0.0.0.0/tcp/0/ipfs/QmWVx9NwsgaVWMRHNCpesq1WQAw2T3JurjGDNeVNWifPS7 -> hi -> hello -``` - -**NOTE: debug mode is enabled by default, debug mode will always generate same node id (on each node) on every execution. Disable debug using `--debug false` flag while running your executable.** - -## Authors -1. Abhishek Upperwal \ No newline at end of file diff --git a/examples/chat/chat.go b/examples/chat/chat.go deleted file mode 100644 index 942cb1e32e..0000000000 --- a/examples/chat/chat.go +++ /dev/null @@ -1,221 +0,0 @@ -/* -* -* The MIT License (MIT) -* -* Copyright (c) 2014 Juan Batiz-Benet -* -* Permission is hereby granted, free of charge, to any person obtaining a copy -* of this software and associated documentation files (the "Software"), to deal -* in the Software without restriction, including without limitation the rights -* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -* copies of the Software, and to permit persons to whom the Software is -* furnished to do so, subject to the following conditions: -* -* The above copyright notice and this permission notice shall be included in -* all copies or substantial portions of the Software. -* -* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -* THE SOFTWARE. -* -* This program demonstrate a simple chat application using p2p communication. -* - */ - -package main - -import ( - "bufio" - "context" - "crypto/rand" - "flag" - "fmt" - "io" - "log" - mrand "math/rand" - "os" - - "github.com/libp2p/go-libp2p" - - "github.com/libp2p/go-libp2p-crypto" - "github.com/libp2p/go-libp2p-host" - "github.com/libp2p/go-libp2p-net" - "github.com/libp2p/go-libp2p-peer" - "github.com/libp2p/go-libp2p-peerstore" - "github.com/multiformats/go-multiaddr" -) - -/* -* addAddrToPeerstore parses a peer multiaddress and adds -* it to the given host's peerstore, so it knows how to -* contact it. It returns the peer ID of the remote peer. -* @credit examples/http-proxy/proxy.go - */ -func addAddrToPeerstore(h host.Host, addr string) peer.ID { - // The following code extracts target's the peer ID from the - // given multiaddress - ipfsaddr, err := multiaddr.NewMultiaddr(addr) - if err != nil { - log.Fatalln(err) - } - pid, err := ipfsaddr.ValueForProtocol(multiaddr.P_IPFS) - if err != nil { - log.Fatalln(err) - } - - peerid, err := peer.IDB58Decode(pid) - if err != nil { - log.Fatalln(err) - } - - // Decapsulate the /ipfs/ part from the target - // /ip4//ipfs/ becomes /ip4/ - targetPeerAddr, _ := multiaddr.NewMultiaddr( - fmt.Sprintf("/ipfs/%s", peer.IDB58Encode(peerid))) - targetAddr := ipfsaddr.Decapsulate(targetPeerAddr) - - // We have a peer ID and a targetAddr so we add - // it to the peerstore so LibP2P knows how to contact it - h.Peerstore().AddAddr(peerid, targetAddr, peerstore.PermanentAddrTTL) - return peerid -} - -func handleStream(s net.Stream) { - log.Println("Got a new stream!") - - // Create a buffer stream for non blocking read and write. - rw := bufio.NewReadWriter(bufio.NewReader(s), bufio.NewWriter(s)) - - go readData(rw) - go writeData(rw) - - // stream 's' will stay open until you close it (or the other side closes it). -} -func readData(rw *bufio.ReadWriter) { - for { - str, _ := rw.ReadString('\n') - - if str == "" { - return - } - if str != "\n" { - // Green console colour: \x1b[32m - // Reset console colour: \x1b[0m - fmt.Printf("\x1b[32m%s\x1b[0m> ", str) - } - - } -} - -func writeData(rw *bufio.ReadWriter) { - stdReader := bufio.NewReader(os.Stdin) - - for { - fmt.Print("> ") - sendData, err := stdReader.ReadString('\n') - - if err != nil { - panic(err) - } - - rw.WriteString(fmt.Sprintf("%s\n", sendData)) - rw.Flush() - } - -} - -func main() { - - sourcePort := flag.Int("sp", 0, "Source port number") - dest := flag.String("d", "", "Dest MultiAddr String") - help := flag.Bool("help", false, "Display Help") - debug := flag.Bool("debug", true, "Debug generated same node id on every execution.") - - flag.Parse() - - if *help { - fmt.Printf("This program demonstrates a simple p2p chat application using libp2p\n\n") - fmt.Printf("Usage: Run './chat -sp ' where can be any port number. Now run './chat -d ' where is multiaddress of previous listener host.\n") - - os.Exit(0) - } - - // If debug is enabled used constant random source else cryptographic randomness. - var r io.Reader - if *debug { - // Constant random source. This will always generate the same host ID on multiple execution. - // Don't do this in production code. - r = mrand.New(mrand.NewSource(int64(*sourcePort))) - } else { - r = rand.Reader - } - - // Creates a new RSA key pair for this host - prvKey, _, err := crypto.GenerateKeyPairWithReader(crypto.RSA, 2048, r) - - if err != nil { - panic(err) - } - - // 0.0.0.0 will listen on any interface device - sourceMultiAddr, _ := multiaddr.NewMultiaddr(fmt.Sprintf("/ip4/0.0.0.0/tcp/%d", *sourcePort)) - - // libp2p.New constructs a new libp2p Host. - // Other options can be added here. - host, err := libp2p.New( - context.Background(), - libp2p.ListenAddrs(sourceMultiAddr), - libp2p.Identity(prvKey), - ) - - if err != nil { - panic(err) - } - - if *dest == "" { - // Set a function as stream handler. - // This function is called when a peer initiate a connection and starts a stream with this peer. - // Only applicable on the receiving side. - host.SetStreamHandler("/chat/1.0.0", handleStream) - - fmt.Printf("Run './chat -d /ip4/127.0.0.1/tcp/%d/ipfs/%s' on another console.\n You can replace 127.0.0.1 with public IP as well.\n", *sourcePort, host.ID().Pretty()) - fmt.Printf("\nWaiting for incoming connection\n\n") - // Hang forever - <-make(chan struct{}) - - } else { - - // Add destination peer multiaddress in the peerstore. - // This will be used during connection and stream creation by libp2p. - peerID := addAddrToPeerstore(host, *dest) - - fmt.Println("This node's multiaddress: ") - // IP will be 0.0.0.0 (listen on any interface) and port will be 0 (choose one for me). - // Although this node will not listen for any connection. It will just initiate a connect with - // one of its peer and use that stream to communicate. - fmt.Printf("%s/ipfs/%s\n", sourceMultiAddr, host.ID().Pretty()) - - // Start a stream with peer with peer Id: 'peerId'. - // Multiaddress of the destination peer is fetched from the peerstore using 'peerId'. - s, err := host.NewStream(context.Background(), peerID, "/chat/1.0.0") - - if err != nil { - panic(err) - } - - // Create a buffered stream so that read and writes are non blocking. - rw := bufio.NewReadWriter(bufio.NewReader(s), bufio.NewWriter(s)) - - // Create a thread to read and write data. - go writeData(rw) - go readData(rw) - - // Hang forever. - select {} - - } -} diff --git a/examples/echo/README.md b/examples/echo/README.md deleted file mode 100644 index 2b0dc24efa..0000000000 --- a/examples/echo/README.md +++ /dev/null @@ -1,52 +0,0 @@ -# Echo client/server with libp2p - -This is an example that quickly shows how to use the `go-libp2p` stack, including Host/Basichost, Network/Swarm, Streams, Peerstores and Multiaddresses. - -This example can be started in either listen mode, or dial mode. - -In listen mode, it will sit and wait for incoming connections on the `/echo/1.0.0` protocol. Whenever it receives a stream, it will write the message `"Hello, world!"` over the stream and close it. - -In dial mode, the node will start up, connect to the given address, open a stream to the target peer, and read a message on the protocol `/echo/1.0.0`. - -## Build - -From `go-libp2p` base folder: - -``` -> make deps -> go build ./examples/echo -``` - -## Usage - -``` -> ./echo -secio -l 10000 -2017/03/15 14:11:32 I am /ip4/127.0.0.1/tcp/10000/ipfs/QmYo41GybvrXk8y8Xnm1P7pfA4YEXCpfnLyzgRPnNbG35e -2017/03/15 14:11:32 Now run "./echo -l 10001 -d /ip4/127.0.0.1/tcp/10000/ipfs/QmYo41GybvrXk8y8Xnm1P7pfA4YEXCpfnLyzgRPnNbG35e -secio" on a different terminal -2017/03/15 14:11:32 listening for connections -``` - -The listener libp2p host will print its `Multiaddress`, which indicates how it can be reached (ip4+tcp) and its randomly generated ID (`QmYo41Gyb...`) - -Now, launch another node that talks to the listener: - -``` -> ./echo -secio -l 10001 -d /ip4/127.0.0.1/tcp/10000/ipfs/QmYo41GybvrXk8y8Xnm1P7pfA4YEXCpfnLyzgRPnNbG35e -``` - -The new node with send the message `"Hello, world!"` to the listener, which will in turn echo it over the stream and close it. The listener logs the message, and the sender logs the response. - -## Details - -The `makeBasicHost()` function creates a [go-libp2p-basichost](https://godoc.org/github.com/libp2p/go-libp2p/p2p/host/basic) object. `basichost` objects wrap [go-libp2 swarms](https://godoc.org/github.com/libp2p/go-libp2p-swarm#Swarm) and should be used preferentially. A [go-libp2p-swarm Network](https://godoc.org/github.com/libp2p/go-libp2p-swarm#Network) is a `swarm` which complies to the [go-libp2p-net Network interface](https://godoc.org/github.com/libp2p/go-libp2p-net#Network) and takes care of maintaining streams, connections, multiplexing different protocols on them, handling incoming connections etc. - -In order to create the swarm (and a `basichost`), the example needs: - -- An [ipfs-procotol ID](https://godoc.org/github.com/libp2p/go-libp2p-peer#ID) like `QmNtX1cvrm2K6mQmMEaMxAuB4rTexhd87vpYVot4sEZzxc`. The example autogenerates a key pair on every run and uses an ID extracted from the public key (the hash of the public key). When using `-secio`, it uses the key pair to encrypt communications (otherwise, it leaves the connections unencrypted). -- A [Multiaddress](https://godoc.org/github.com/multiformats/go-multiaddr), which indicates how to reach this peer. There can be several of them (using different protocols or locations for example). Example: `/ip4/127.0.0.1/tcp/1234`. -- A [go-libp2p-peerstore](https://godoc.org/github.com/libp2p/go-libp2p-peerstore), which is used as a address book which matches node IDs to the multiaddresses through which they can be contacted. This peerstore gets autopopulated when manually opening a connection (with [`Connect()`](https://godoc.org/github.com/libp2p/go-libp2p/p2p/host/basic#BasicHost.Connect). Alternatively, we can manually [`AddAddr()`](https://godoc.org/github.com/libp2p/go-libp2p-peerstore#AddrManager.AddAddr) as in the example. - -A `basichost` can now open streams (bi-directional channel between to peers) using [NewStream](https://godoc.org/github.com/libp2p/go-libp2p/p2p/host/basic#BasicHost.NewStream) and use them to send and receive data tagged with a `Protocol.ID` (a string). The host can also listen for incoming connections for a given -`Protocol` with [`SetStreamHandle()`](https://godoc.org/github.com/libp2p/go-libp2p/p2p/host/basic#BasicHost.SetStreamHandler). - -The example makes use of all of this to enable communication between a listener and a sender using protocol `/echo/1.0.0` (which could be any other thing). diff --git a/examples/echo/main.go b/examples/echo/main.go deleted file mode 100644 index 32d156b73e..0000000000 --- a/examples/echo/main.go +++ /dev/null @@ -1,178 +0,0 @@ -package main - -import ( - "bufio" - "context" - "crypto/rand" - "flag" - "fmt" - "io" - "io/ioutil" - "log" - mrand "math/rand" - - golog "github.com/ipfs/go-log" - libp2p "github.com/libp2p/go-libp2p" - crypto "github.com/libp2p/go-libp2p-crypto" - host "github.com/libp2p/go-libp2p-host" - net "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" - gologging "github.com/whyrusleeping/go-logging" -) - -// makeBasicHost creates a LibP2P host with a random peer ID listening on the -// given multiaddress. It will use secio if secio is true. -func makeBasicHost(listenPort int, secio bool, randseed int64) (host.Host, error) { - - // If the seed is zero, use real cryptographic randomness. Otherwise, use a - // deterministic randomness source to make generated keys stay the same - // across multiple runs - var r io.Reader - if randseed == 0 { - r = rand.Reader - } else { - r = mrand.New(mrand.NewSource(randseed)) - } - - // Generate a key pair for this host. We will use it at least - // to obtain a valid host ID. - priv, _, err := crypto.GenerateKeyPairWithReader(crypto.RSA, 2048, r) - if err != nil { - return nil, err - } - - opts := []libp2p.Option{ - libp2p.ListenAddrStrings(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", listenPort)), - libp2p.Identity(priv), - } - - if !secio { - opts = append(opts, libp2p.NoSecurity) - } - - basicHost, err := libp2p.New(context.Background(), opts...) - if err != nil { - return nil, err - } - - // Build host multiaddress - hostAddr, _ := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s", basicHost.ID().Pretty())) - - // Now we can build a full multiaddress to reach this host - // by encapsulating both addresses: - addr := basicHost.Addrs()[0] - fullAddr := addr.Encapsulate(hostAddr) - log.Printf("I am %s\n", fullAddr) - if secio { - log.Printf("Now run \"./echo -l %d -d %s -secio\" on a different terminal\n", listenPort+1, fullAddr) - } else { - log.Printf("Now run \"./echo -l %d -d %s\" on a different terminal\n", listenPort+1, fullAddr) - } - - return basicHost, nil -} - -func main() { - // LibP2P code uses golog to log messages. They log with different - // string IDs (i.e. "swarm"). We can control the verbosity level for - // all loggers with: - golog.SetAllLoggers(gologging.INFO) // Change to DEBUG for extra info - - // Parse options from the command line - listenF := flag.Int("l", 0, "wait for incoming connections") - target := flag.String("d", "", "target peer to dial") - secio := flag.Bool("secio", false, "enable secio") - seed := flag.Int64("seed", 0, "set random seed for id generation") - flag.Parse() - - if *listenF == 0 { - log.Fatal("Please provide a port to bind on with -l") - } - - // Make a host that listens on the given multiaddress - ha, err := makeBasicHost(*listenF, *secio, *seed) - if err != nil { - log.Fatal(err) - } - - // Set a stream handler on host A. /echo/1.0.0 is - // a user-defined protocol name. - ha.SetStreamHandler("/echo/1.0.0", func(s net.Stream) { - log.Println("Got a new stream!") - if err := doEcho(s); err != nil { - log.Println(err) - s.Reset() - } else { - s.Close() - } - }) - - if *target == "" { - log.Println("listening for connections") - select {} // hang forever - } - /**** This is where the listener code ends ****/ - - // The following code extracts target's the peer ID from the - // given multiaddress - ipfsaddr, err := ma.NewMultiaddr(*target) - if err != nil { - log.Fatalln(err) - } - - pid, err := ipfsaddr.ValueForProtocol(ma.P_IPFS) - if err != nil { - log.Fatalln(err) - } - - peerid, err := peer.IDB58Decode(pid) - if err != nil { - log.Fatalln(err) - } - - // Decapsulate the /ipfs/ part from the target - // /ip4//ipfs/ becomes /ip4/ - targetPeerAddr, _ := ma.NewMultiaddr( - fmt.Sprintf("/ipfs/%s", peer.IDB58Encode(peerid))) - targetAddr := ipfsaddr.Decapsulate(targetPeerAddr) - - // We have a peer ID and a targetAddr so we add it to the peerstore - // so LibP2P knows how to contact it - ha.Peerstore().AddAddr(peerid, targetAddr, pstore.PermanentAddrTTL) - - log.Println("opening stream") - // make a new stream from host B to host A - // it should be handled on host A by the handler we set above because - // we use the same /echo/1.0.0 protocol - s, err := ha.NewStream(context.Background(), peerid, "/echo/1.0.0") - if err != nil { - log.Fatalln(err) - } - - _, err = s.Write([]byte("Hello, world!\n")) - if err != nil { - log.Fatalln(err) - } - - out, err := ioutil.ReadAll(s) - if err != nil { - log.Fatalln(err) - } - - log.Printf("read reply: %q\n", out) -} - -// doEcho reads a line of data a stream and writes it back -func doEcho(s net.Stream) error { - buf := bufio.NewReader(s) - str, err := buf.ReadString('\n') - if err != nil { - return err - } - - log.Printf("read: %s\n", str) - _, err = s.Write([]byte(str)) - return err -} diff --git a/examples/http-proxy/README.md b/examples/http-proxy/README.md deleted file mode 100644 index 063cefc925..0000000000 --- a/examples/http-proxy/README.md +++ /dev/null @@ -1,58 +0,0 @@ -# HTTP proxy service with libp2p - -This examples shows how to create a simple HTTP proxy service with libp2p: - -``` - XXX - XX XXXXXX - X XX - XXXXXXX XX XX XXXXXXXXXX - +----------------+ +-----------------+ XXX XXX XXX XXX - HTTP Request | | | | XX XX -+-----------------> | libp2p stream | | HTTP X X - | Local peer <----------------> Remote peer <-------------> HTTP SERVER - THE INTERNET XX -<-----------------+ | | | Req & Resp XX X - HTTP Response | libp2p host | | libp2p host | XXXX XXXX XXXXXXXXXXXXXXXXXXXX XXXX - +----------------+ +-----------------+ XXXXX -``` - -In order to proxy an HTTP request, we create a local peer which listens on `localhost:9900`. HTTP requests performed to that address are tunneled via a libp2p stream to a remote peer, which then performs the HTTP requests and sends the response back to the local peer, which relays it to the user. - -Note that this is a very simple approach to a proxy, and does not perform any header management, nor supports HTTPS. The `proxy.go` code is thoroughly commeted, detailing what is happening in every step. - -## Build - -From `go-libp2p` base folder: - -``` -> make deps -> go build ./examples/http-proxy -``` - -## Usage - -First run the "remote" peer as follows. It will print a local peer address. If you would like to run this on a separate machine, please replace the IP accordingly: - -```sh -> ./http-proxy -Proxy server is ready -libp2p-peer addresses: -/ip4/127.0.0.1/tcp/12000/ipfs/QmddTrQXhA9AkCpXPTkcY7e22NK73TwkUms3a44DhTKJTD -``` - -The run the local peer, indicating that it will need to forward http requests to the remote peer as follows: - -``` -> ./http-proxy -d /ip4/127.0.0.1/tcp/12000/ipfs/QmddTrQXhA9AkCpXPTkcY7e22NK73TwkUms3a44DhTKJTD -Proxy server is ready -libp2p-peer addresses: -/ip4/127.0.0.1/tcp/12001/ipfs/Qmaa2AYTha1UqcFVX97p9R1UP7vbzDLY7bqWsZw1135QvN -proxy listening on 127.0.0.1:9900 -``` - -As you can see, the proxy prints the listening address `127.0.0.1:9900`. You can now use this address as proxy, for example with `curl`: - -``` -> curl -x "127.0.0.1:9900" "http://ipfs.io/ipfs/QmfUX75pGRBRDnjeoMkQzuQczuCup2aYbeLxz5NzeSu9G6" -it works! -``` diff --git a/examples/http-proxy/proxy.go b/examples/http-proxy/proxy.go deleted file mode 100644 index 07e36b0dbe..0000000000 --- a/examples/http-proxy/proxy.go +++ /dev/null @@ -1,276 +0,0 @@ -package main - -import ( - "bufio" - "context" - "flag" - "fmt" - "io" - "log" - "net/http" - "strings" - - // We need to import libp2p's libraries that we use in this project. - // In order to work, these libraries need to be rewritten by gx-go. - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - ps "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" - - libp2p "github.com/libp2p/go-libp2p" -) - -// Protocol defines the libp2p protocol that we will use for the libp2p proxy -// service that we are going to provide. This will tag the streams used for -// this service. Streams are multiplexed and their protocol tag helps -// libp2p handle them to the right handler functions. -const Protocol = "/proxy-example/0.0.1" - -// makeRandomHost creates a libp2p host with a randomly generated identity. -// This step is described in depth in other tutorials. -func makeRandomHost(port int) host.Host { - host, err := libp2p.New(context.Background(), libp2p.ListenAddrStrings(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port))) - if err != nil { - log.Fatalln(err) - } - return host -} - -// ProxyService provides HTTP proxying on top of libp2p by launching an -// HTTP server which tunnels the requests to a destination peer running -// ProxyService too. -type ProxyService struct { - host host.Host - dest peer.ID - proxyAddr ma.Multiaddr -} - -// NewProxyService attaches a proxy service to the given libp2p Host. -// The proxyAddr parameter specifies the address on which the -// HTTP proxy server listens. The dest parameter specifies the peer -// ID of the remote peer in charge of performing the HTTP requests. -// -// ProxyAddr/dest may be nil/"" it is not necessary that this host -// provides a listening HTTP server (and instead its only function is to -// perform the proxied http requests it receives from a different peer. -// -// The addresses for the dest peer should be part of the host's peerstore. -func NewProxyService(h host.Host, proxyAddr ma.Multiaddr, dest peer.ID) *ProxyService { - // We let our host know that it needs to handle streams tagged with the - // protocol id that we have defined, and then handle them to - // our own streamHandling function. - h.SetStreamHandler(Protocol, streamHandler) - - fmt.Println("Proxy server is ready") - fmt.Println("libp2p-peer addresses:") - for _, a := range h.Addrs() { - fmt.Printf("%s/ipfs/%s\n", a, peer.IDB58Encode(h.ID())) - } - - return &ProxyService{ - host: h, - dest: dest, - proxyAddr: proxyAddr, - } -} - -// streamHandler is our function to handle any libp2p-net streams that belong -// to our protocol. The streams should contain an HTTP request which we need -// to parse, make on behalf of the original node, and then write the response -// on the stream, before closing it. -func streamHandler(stream inet.Stream) { - // Remember to close the stream when we are done. - defer stream.Close() - - // Create a new buffered reader, as ReadRequest needs one. - // The buffered reader reads from our stream, on which we - // have sent the HTTP request (see ServeHTTP()) - buf := bufio.NewReader(stream) - // Read the HTTP request from the buffer - req, err := http.ReadRequest(buf) - if err != nil { - stream.Reset() - log.Println(err) - return - } - defer req.Body.Close() - - // We need to reset these fields in the request - // URL as they are not maintained. - req.URL.Scheme = "http" - hp := strings.Split(req.Host, ":") - if len(hp) > 1 && hp[1] == "443" { - req.URL.Scheme = "https" - } else { - req.URL.Scheme = "http" - } - req.URL.Host = req.Host - - outreq := new(http.Request) - *outreq = *req - - // We now make the request - fmt.Printf("Making request to %s\n", req.URL) - resp, err := http.DefaultTransport.RoundTrip(outreq) - if err != nil { - stream.Reset() - log.Println(err) - return - } - - // resp.Write writes whatever response we obtained for our - // request back to the stream. - resp.Write(stream) -} - -// Serve listens on the ProxyService's proxy address. This effectively -// allows to set the listening address as http proxy. -func (p *ProxyService) Serve() { - _, serveArgs, _ := manet.DialArgs(p.proxyAddr) - fmt.Println("proxy listening on ", serveArgs) - if p.dest != "" { - http.ListenAndServe(serveArgs, p) - } -} - -// ServeHTTP implements the http.Handler interface. WARNING: This is the -// simplest approach to a proxy. Therefore we do not do any of the things -// that should be done when implementing a reverse proxy (like handling -// headers correctly). For how to do it properly, see: -// https://golang.org/src/net/http/httputil/reverseproxy.go?s=3845:3920#L121 -// -// ServeHTTP opens a stream to the dest peer for every HTTP request. -// Streams are multiplexed over single connections so, unlike connections -// themselves, they are cheap to create and dispose of. -func (p *ProxyService) ServeHTTP(w http.ResponseWriter, r *http.Request) { - fmt.Printf("proxying request for %s to peer %s\n", r.URL, p.dest.Pretty()) - // We need to send the request to the remote libp2p peer, so - // we open a stream to it - stream, err := p.host.NewStream(context.Background(), p.dest, Protocol) - // If an error happens, we write an error for response. - if err != nil { - log.Println(err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - defer stream.Close() - - // r.Write() writes the HTTP request to the stream. - err = r.Write(stream) - if err != nil { - stream.Reset() - log.Println(err) - http.Error(w, err.Error(), http.StatusServiceUnavailable) - return - } - - // Now we read the response that was sent from the dest - // peer - buf := bufio.NewReader(stream) - resp, err := http.ReadResponse(buf, r) - if err != nil { - stream.Reset() - log.Println(err) - http.Error(w, err.Error(), http.StatusServiceUnavailable) - return - } - - // Copy any headers - for k, v := range resp.Header { - for _, s := range v { - w.Header().Add(k, s) - } - } - - // Write response status and headers - w.WriteHeader(resp.StatusCode) - - // Finally copy the body - io.Copy(w, resp.Body) - resp.Body.Close() -} - -// addAddrToPeerstore parses a peer multiaddress and adds -// it to the given host's peerstore, so it knows how to -// contact it. It returns the peer ID of the remote peer. -func addAddrToPeerstore(h host.Host, addr string) peer.ID { - // The following code extracts target's the peer ID from the - // given multiaddress - ipfsaddr, err := ma.NewMultiaddr(addr) - if err != nil { - log.Fatalln(err) - } - pid, err := ipfsaddr.ValueForProtocol(ma.P_IPFS) - if err != nil { - log.Fatalln(err) - } - - peerid, err := peer.IDB58Decode(pid) - if err != nil { - log.Fatalln(err) - } - - // Decapsulate the /ipfs/ part from the target - // /ip4//ipfs/ becomes /ip4/ - targetPeerAddr, _ := ma.NewMultiaddr( - fmt.Sprintf("/ipfs/%s", peer.IDB58Encode(peerid))) - targetAddr := ipfsaddr.Decapsulate(targetPeerAddr) - - // We have a peer ID and a targetAddr so we add - // it to the peerstore so LibP2P knows how to contact it - h.Peerstore().AddAddr(peerid, targetAddr, ps.PermanentAddrTTL) - return peerid -} - -const help = ` -This example creates a simple HTTP Proxy using two libp2p peers. The first peer -provides an HTTP server locally which tunnels the HTTP requests with libp2p -to a remote peer. The remote peer performs the requests and -send the sends the response back. - -Usage: Start remote peer first with: ./proxy - Then start the local peer with: ./proxy -d - -Then you can do something like: curl -x "localhost:9900" "http://ipfs.io". -This proxies sends the request through the local peer, which proxies it to -the remote peer, which makes it and sends the response back. -` - -func main() { - flag.Usage = func() { - fmt.Println(help) - flag.PrintDefaults() - } - - // Parse some flags - destPeer := flag.String("d", "", "destination peer address") - port := flag.Int("p", 9900, "proxy port") - p2pport := flag.Int("l", 12000, "libp2p listen port") - flag.Parse() - - // If we have a destination peer we will start a local server - if *destPeer != "" { - // We use p2pport+1 in order to not collide if the user - // is running the remote peer locally on that port - host := makeRandomHost(*p2pport + 1) - // Make sure our host knows how to reach destPeer - destPeerID := addAddrToPeerstore(host, *destPeer) - proxyAddr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", *port)) - if err != nil { - log.Fatalln(err) - } - // Create the proxy service and start the http server - proxy := NewProxyService(host, proxyAddr, destPeerID) - proxy.Serve() // serve hangs forever - } else { - host := makeRandomHost(*p2pport) - // In this case we only need to make sure our host - // knows how to handle incoming proxied requests from - // another peer. - _ = NewProxyService(host, nil, "") - <-make(chan struct{}) // hang forever - } - -} diff --git a/examples/libp2p-host/README.md b/examples/libp2p-host/README.md deleted file mode 100644 index 88532053bb..0000000000 --- a/examples/libp2p-host/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# The libp2p 'host' - -For most applications, the host is the basic building block you'll need to get started. This guide will show how to construct and use a simple host. - -The host is an abstraction that manages services on top of a swarm. It provides a clean interface to connect to a service on a given remote peer. - -If you want to create a host with a default configuration, you can do the following: - -```go -import ( - "context" - "crypto/rand" - "fmt" - - libp2p "github.com/libp2p/go-libp2p" - crypto "github.com/libp2p/go-libp2p-crypto" -) - - -// The context governs the lifetime of the libp2p node -ctx, cancel := context.WithCancel(context.Background()) -defer cancel() - -// To construct a simple host with all the default settings, just use `New` -h, err := libp2p.New(ctx) -if err != nil { - panic(err) -} - -fmt.Printf("Hello World, my hosts ID is %s\n", h.ID()) -``` - -If you want more control over the configuration, you can specify some options to the constructor. For a full list of all the configuration supported by the constructor see: [options.go](https://github.com/libp2p/go-libp2p/blob/master/options.go) - -In this snippet we generate our own ID and specified on which address we want to listen: - -```go -// Set your own keypair -priv, _, err := crypto.GenerateEd25519Key(rand.Reader) -if err != nil { - panic(err) -} - -h2, err := libp2p.New(ctx, - // Use your own created keypair - libp2p.Identity(priv), - - // Set your own listen address - // The config takes an array of addresses, specify as many as you want. - libp2p.ListenAddrStrings("/ip4/0.0.0.0/tcp/9000"), -) -if err != nil { - panic(err) -} - -fmt.Printf("Hello World, my second hosts ID is %s\n", h2.ID()) -``` - -And thats it, you have a libp2p host and you're ready to start doing some awesome p2p networking! - -In future guides we will go over ways to use hosts, configure them differently (hint: there are a huge number of ways to set these up), and interesting ways to apply this technology to various applications you might want to build. - -To see this code all put together, take a look at [host.go](host.go). diff --git a/examples/libp2p-host/host.go b/examples/libp2p-host/host.go deleted file mode 100644 index 793cdf4fd8..0000000000 --- a/examples/libp2p-host/host.go +++ /dev/null @@ -1,47 +0,0 @@ -package main - -import ( - "context" - "crypto/rand" - "fmt" - - libp2p "github.com/libp2p/go-libp2p" - crypto "github.com/libp2p/go-libp2p-crypto" -) - -func main() { - // The context governs the lifetime of the libp2p node - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - // To construct a simple host with all the default settings, just use `New` - h, err := libp2p.New(ctx) - if err != nil { - panic(err) - } - - fmt.Printf("Hello World, my hosts ID is %s\n", h.ID()) - - // If you want more control over the configuration, you can specify some - // options to the constructor - - // Set your own keypair - priv, _, err := crypto.GenerateEd25519Key(rand.Reader) - if err != nil { - panic(err) - } - - h2, err := libp2p.New(ctx, - // Use your own created keypair - libp2p.Identity(priv), - - // Set your own listen address - // The config takes an array of addresses, specify as many as you want. - libp2p.ListenAddrStrings("/ip4/0.0.0.0/tcp/9000"), - ) - if err != nil { - panic(err) - } - - fmt.Printf("Hello World, my second hosts ID is %s\n", h2.ID()) -} diff --git a/examples/multipro/AUTHORS b/examples/multipro/AUTHORS deleted file mode 100644 index 70a81f0a5d..0000000000 --- a/examples/multipro/AUTHORS +++ /dev/null @@ -1,3 +0,0 @@ -# This is the official list of authors for copyright purposes. - -Aviv Eyal diff --git a/examples/multipro/LICENSE b/examples/multipro/LICENSE deleted file mode 100644 index df71bca6ad..0000000000 --- a/examples/multipro/LICENSE +++ /dev/null @@ -1,21 +0,0 @@ -The MIT License (MIT) - -Copyright (c) 2017 Aviv Eyal - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. diff --git a/examples/multipro/README.md b/examples/multipro/README.md deleted file mode 100644 index c60da8b173..0000000000 --- a/examples/multipro/README.md +++ /dev/null @@ -1,62 +0,0 @@ -# Protocol Multiplexing using rpc-style multicodecs, protobufs with libp2p - -This examples shows how to use multicodecs (i.e. protobufs) to encode and transmit information between LibP2P hosts using LibP2P Streams. -Multicodecs present a common interface, making it very easy to swap the codec implementation if needed. -This example expects that you area already familiar with the [echo example](https://github.com/libp2p/go-libp2p/tree/master/examples/echo). - -## Build - -Install gx: -```sh -> go get -u github.com/whyrusleeping/gx - -``` - -Run GX from the root libp2p source dir: -```sh ->gx install -``` - -Build libp2p: -```sh -> make deps -> make -``` - -Run from `multipro` directory - -```sh -> go build -``` - - -## Usage - -```sh -> ./multipro - -``` - -## Details - -The example creates two LibP2P Hosts supporting 2 protocols: ping and echo. - -Each protocol consists RPC-style requests and responses and each request and response is a typed protobufs message (and a go data object). - -This is a different pattern then defining a whole p2p protocol as one protobuf message with lots of optional fields (as can be observed in various p2p-lib protocols using protobufs such as dht). - -The example shows how to match async received responses with their requests. This is useful when processing a response requires access to the request data. - -The idea is to use lib-p2p protocol multiplexing on a per-message basis. - -### Features -1. 2 fully implemented protocols using an RPC-like request-response pattern - Ping and Echo -2. Scaffolding for quickly implementing new app-level versioned RPC-like protocols -3. Full authentication of incoming message data by author (who might not be the message's sender peer) -4. Base p2p format in protobufs with fields shared by all protocol messages -5. Full access to request data when processing a response. - -## Author -@avive - - diff --git a/examples/multipro/echo.go b/examples/multipro/echo.go deleted file mode 100644 index a3d39f8b89..0000000000 --- a/examples/multipro/echo.go +++ /dev/null @@ -1,157 +0,0 @@ -package main - -import ( - "bufio" - "context" - "fmt" - "log" - - inet "github.com/libp2p/go-libp2p-net" - - uuid "github.com/google/uuid" - "github.com/libp2p/go-libp2p-host" - pb "github.com/libp2p/go-libp2p/examples/multipro/pb" - protobufCodec "github.com/multiformats/go-multicodec/protobuf" -) - -// pattern: /protocol-name/request-or-response-message/version -const echoRequest = "/echo/echoreq/0.0.1" -const echoResponse = "/echo/echoresp/0.0.1" - -type EchoProtocol struct { - node *Node // local host - requests map[string]*pb.EchoRequest // used to access request data from response handlers - done chan bool // only for demo purposes to hold main from terminating -} - -func NewEchoProtocol(node *Node, done chan bool) *EchoProtocol { - e := EchoProtocol{node: node, requests: make(map[string]*pb.EchoRequest), done: done} - node.SetStreamHandler(echoRequest, e.onEchoRequest) - node.SetStreamHandler(echoResponse, e.onEchoResponse) - - // design note: to implement fire-and-forget style messages you may just skip specifying a response callback. - // a fire-and-forget message will just include a request and not specify a response object - - return &e -} - -// remote peer requests handler -func (e *EchoProtocol) onEchoRequest(s inet.Stream) { - // get request data - data := &pb.EchoRequest{} - decoder := protobufCodec.Multicodec(nil).Decoder(bufio.NewReader(s)) - err := decoder.Decode(data) - if err != nil { - log.Println(err) - return - } - - log.Printf("%s: Received echo request from %s. Message: %s", s.Conn().LocalPeer(), s.Conn().RemotePeer(), data.Message) - - valid := e.node.authenticateMessage(data, data.MessageData) - - if !valid { - log.Println("Failed to authenticate message") - return - } - - log.Printf("%s: Sending echo response to %s. Message id: %s...", s.Conn().LocalPeer(), s.Conn().RemotePeer(), data.MessageData.Id) - - // send response to the request using the message string he provided - - resp := &pb.EchoResponse{ - MessageData: e.node.NewMessageData(data.MessageData.Id, false), - Message: data.Message} - - // sign the data - signature, err := e.node.signProtoMessage(resp) - if err != nil { - log.Println("failed to sign response") - return - } - - // add the signature to the message - resp.MessageData.Sign = string(signature) - - s, respErr := e.node.NewStream(context.Background(), s.Conn().RemotePeer(), echoResponse) - if respErr != nil { - log.Println(respErr) - return - } - - ok := e.node.sendProtoMessage(resp, s) - - if ok { - log.Printf("%s: Echo response to %s sent.", s.Conn().LocalPeer().String(), s.Conn().RemotePeer().String()) - } -} - -// remote echo response handler -func (e *EchoProtocol) onEchoResponse(s inet.Stream) { - data := &pb.EchoResponse{} - decoder := protobufCodec.Multicodec(nil).Decoder(bufio.NewReader(s)) - err := decoder.Decode(data) - if err != nil { - return - } - - // authenticate message content - valid := e.node.authenticateMessage(data, data.MessageData) - - if !valid { - log.Println("Failed to authenticate message") - return - } - - // locate request data and remove it if found - req, ok := e.requests[data.MessageData.Id] - if ok { - // remove request from map as we have processed it here - delete(e.requests, data.MessageData.Id) - } else { - log.Println("Failed to locate request data boject for response") - return - } - - if req.Message != data.Message { - log.Fatalln("Expected echo to respond with request message") - } - - log.Printf("%s: Received echo response from %s. Message id:%s. Message: %s.", s.Conn().LocalPeer(), s.Conn().RemotePeer(), data.MessageData.Id, data.Message) - e.done <- true -} - -func (e *EchoProtocol) Echo(host host.Host) bool { - log.Printf("%s: Sending echo to: %s....", e.node.ID(), host.ID()) - - // create message data - req := &pb.EchoRequest{ - MessageData: e.node.NewMessageData(uuid.New().String(), false), - Message: fmt.Sprintf("Echo from %s", e.node.ID())} - - signature, err := e.node.signProtoMessage(req) - if err != nil { - log.Println("failed to sign message") - return false - } - - // add the signature to the message - req.MessageData.Sign = string(signature) - - s, err := e.node.NewStream(context.Background(), host.ID(), echoRequest) - if err != nil { - log.Println(err) - return false - } - - ok := e.node.sendProtoMessage(req, s) - - if !ok { - return false - } - - // store request so response handler has access to it - e.requests[req.MessageData.Id] = req - log.Printf("%s: Echo to: %s was sent. Message Id: %s, Message: %s", e.node.ID(), host.ID(), req.MessageData.Id, req.Message) - return true -} diff --git a/examples/multipro/main.go b/examples/multipro/main.go deleted file mode 100644 index 3350ea3834..0000000000 --- a/examples/multipro/main.go +++ /dev/null @@ -1,56 +0,0 @@ -package main - -import ( - "context" - "fmt" - "log" - "math/rand" - - libp2p "github.com/libp2p/go-libp2p" - crypto "github.com/libp2p/go-libp2p-crypto" - ps "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" -) - -// helper method - create a lib-p2p host to listen on a port -func makeRandomNode(port int, done chan bool) *Node { - // Ignoring most errors for brevity - // See echo example for more details and better implementation - priv, _, _ := crypto.GenerateKeyPair(crypto.Secp256k1, 256) - listen, _ := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port)) - host, _ := libp2p.New( - context.Background(), - libp2p.ListenAddrs(listen), - libp2p.Identity(priv), - ) - - return NewNode(host, done) -} - -func main() { - // Choose random ports between 10000-10100 - rand.Seed(666) - port1 := rand.Intn(100) + 10000 - port2 := port1 + 1 - - done := make(chan bool, 1) - - // Make 2 hosts - h1 := makeRandomNode(port1, done) - h2 := makeRandomNode(port2, done) - h1.Peerstore().AddAddrs(h2.ID(), h2.Addrs(), ps.PermanentAddrTTL) - h2.Peerstore().AddAddrs(h1.ID(), h1.Addrs(), ps.PermanentAddrTTL) - - log.Printf("This is a conversation between %s and %s\n", h1.ID(), h2.ID()) - - // send messages using the protocols - h1.Ping(h2.Host) - h2.Ping(h1.Host) - h1.Echo(h2.Host) - h2.Echo(h1.Host) - - // block until all responses have been processed - for i := 0; i < 4; i++ { - <-done - } -} diff --git a/examples/multipro/node.go b/examples/multipro/node.go deleted file mode 100644 index 58b42330e1..0000000000 --- a/examples/multipro/node.go +++ /dev/null @@ -1,150 +0,0 @@ -package main - -import ( - "bufio" - "log" - "time" - - "github.com/gogo/protobuf/proto" - crypto "github.com/libp2p/go-libp2p-crypto" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - p2p "github.com/libp2p/go-libp2p/examples/multipro/pb" - protobufCodec "github.com/multiformats/go-multicodec/protobuf" -) - -// node client version -const clientVersion = "go-p2p-node/0.0.1" - -// Node type - a p2p host implementing one or more p2p protocols -type Node struct { - host.Host // lib-p2p host - *PingProtocol // ping protocol impl - *EchoProtocol // echo protocol impl - // add other protocols here... -} - -// Create a new node with its implemented protocols -func NewNode(host host.Host, done chan bool) *Node { - node := &Node{Host: host} - node.PingProtocol = NewPingProtocol(node, done) - node.EchoProtocol = NewEchoProtocol(node, done) - return node -} - -// Authenticate incoming p2p message -// message: a protobufs go data object -// data: common p2p message data -func (n *Node) authenticateMessage(message proto.Message, data *p2p.MessageData) bool { - // store a temp ref to signature and remove it from message data - // sign is a string to allow easy reset to zero-value (empty string) - sign := data.Sign - data.Sign = "" - - // marshall data without the signature to protobufs3 binary format - bin, err := proto.Marshal(message) - if err != nil { - log.Println(err, "failed to marshal pb message") - return false - } - - // restore sig in message data (for possible future use) - data.Sign = sign - - // restore peer id binary format from base58 encoded node id data - peerId, err := peer.IDB58Decode(data.NodeId) - if err != nil { - log.Println(err, "Failed to decode node id from base58") - return false - } - - // verify the data was authored by the signing peer identified by the public key - // and signature included in the message - return n.verifyData(bin, []byte(sign), peerId, data.NodePubKey) -} - -// sign an outgoing p2p message payload -func (n *Node) signProtoMessage(message proto.Message) ([]byte, error) { - data, err := proto.Marshal(message) - if err != nil { - return nil, err - } - return n.signData(data) -} - -// sign binary data using the local node's private key -func (n *Node) signData(data []byte) ([]byte, error) { - key := n.Peerstore().PrivKey(n.ID()) - res, err := key.Sign(data) - return res, err -} - -// Verify incoming p2p message data integrity -// data: data to verify -// signature: author signature provided in the message payload -// peerId: author peer id from the message payload -// pubKeyData: author public key from the message payload -func (n *Node) verifyData(data []byte, signature []byte, peerId peer.ID, pubKeyData []byte) bool { - key, err := crypto.UnmarshalPublicKey(pubKeyData) - if err != nil { - log.Println(err, "Failed to extract key from message key data") - return false - } - - // extract node id from the provided public key - idFromKey, err := peer.IDFromPublicKey(key) - - if err != nil { - log.Println(err, "Failed to extract peer id from public key") - return false - } - - // verify that message author node id matches the provided node public key - if idFromKey != peerId { - log.Println(err, "Node id and provided public key mismatch") - return false - } - - res, err := key.Verify(data, signature) - if err != nil { - log.Println(err, "Error authenticating data") - return false - } - - return res -} - -// helper method - generate message data shared between all node's p2p protocols -// messageId: unique for requests, copied from request for responses -func (n *Node) NewMessageData(messageId string, gossip bool) *p2p.MessageData { - // Add protobufs bin data for message author public key - // this is useful for authenticating messages forwarded by a node authored by another node - nodePubKey, err := n.Peerstore().PubKey(n.ID()).Bytes() - - if err != nil { - panic("Failed to get public key for sender from local peer store.") - } - - return &p2p.MessageData{ClientVersion: clientVersion, - NodeId: peer.IDB58Encode(n.ID()), - NodePubKey: nodePubKey, - Timestamp: time.Now().Unix(), - Id: messageId, - Gossip: gossip} -} - -// helper method - writes a protobuf go data object to a network stream -// data: reference of protobuf go data object to send (not the object itself) -// s: network stream to write the data to -func (n *Node) sendProtoMessage(data proto.Message, s inet.Stream) bool { - writer := bufio.NewWriter(s) - enc := protobufCodec.Multicodec(nil).Encoder(writer) - err := enc.Encode(data) - if err != nil { - log.Println(err) - return false - } - writer.Flush() - return true -} diff --git a/examples/multipro/pb/p2p.pb.go b/examples/multipro/pb/p2p.pb.go deleted file mode 100644 index 091ef4afec..0000000000 --- a/examples/multipro/pb/p2p.pb.go +++ /dev/null @@ -1,121 +0,0 @@ -// Code generated by protoc-gen-gogo. -// source: p2p.proto -// DO NOT EDIT! - -/* -Package protocols_p2p is a generated protocol buffer package. - -It is generated from these files: - p2p.proto - -It has these top-level messages: - MessageData - PingRequest - PingResponse - EchoRequest - EchoResponse -*/ -package protocols_p2p - -import proto "github.com/gogo/protobuf/proto" -import fmt "fmt" -import math "math" - -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf - -// designed to be shared between all app protocols -type MessageData struct { - // shared between all requests - ClientVersion string `protobuf:"bytes,1,opt,name=clientVersion,proto3" json:"clientVersion,omitempty"` - Timestamp int64 `protobuf:"varint,2,opt,name=timestamp,proto3" json:"timestamp,omitempty"` - Id string `protobuf:"bytes,3,opt,name=id,proto3" json:"id,omitempty"` - Gossip bool `protobuf:"varint,4,opt,name=gossip,proto3" json:"gossip,omitempty"` - NodeId string `protobuf:"bytes,5,opt,name=nodeId,proto3" json:"nodeId,omitempty"` - NodePubKey []byte `protobuf:"bytes,6,opt,name=nodePubKey,proto3" json:"nodePubKey,omitempty"` - Sign string `protobuf:"bytes,7,opt,name=sign,proto3" json:"sign,omitempty"` -} - -func (m *MessageData) Reset() { *m = MessageData{} } -func (m *MessageData) String() string { return proto.CompactTextString(m) } -func (*MessageData) ProtoMessage() {} - -// a protocol define a set of reuqest and responses -type PingRequest struct { - MessageData *MessageData `protobuf:"bytes,1,opt,name=messageData" json:"messageData,omitempty"` - // method specific data - Message string `protobuf:"bytes,2,opt,name=message,proto3" json:"message,omitempty"` -} - -func (m *PingRequest) Reset() { *m = PingRequest{} } -func (m *PingRequest) String() string { return proto.CompactTextString(m) } -func (*PingRequest) ProtoMessage() {} - -func (m *PingRequest) GetMessageData() *MessageData { - if m != nil { - return m.MessageData - } - return nil -} - -type PingResponse struct { - MessageData *MessageData `protobuf:"bytes,1,opt,name=messageData" json:"messageData,omitempty"` - // response specific data - Message string `protobuf:"bytes,2,opt,name=message,proto3" json:"message,omitempty"` -} - -func (m *PingResponse) Reset() { *m = PingResponse{} } -func (m *PingResponse) String() string { return proto.CompactTextString(m) } -func (*PingResponse) ProtoMessage() {} - -func (m *PingResponse) GetMessageData() *MessageData { - if m != nil { - return m.MessageData - } - return nil -} - -// a protocol define a set of reuqest and responses -type EchoRequest struct { - MessageData *MessageData `protobuf:"bytes,1,opt,name=messageData" json:"messageData,omitempty"` - // method specific data - Message string `protobuf:"bytes,2,opt,name=message,proto3" json:"message,omitempty"` -} - -func (m *EchoRequest) Reset() { *m = EchoRequest{} } -func (m *EchoRequest) String() string { return proto.CompactTextString(m) } -func (*EchoRequest) ProtoMessage() {} - -func (m *EchoRequest) GetMessageData() *MessageData { - if m != nil { - return m.MessageData - } - return nil -} - -type EchoResponse struct { - MessageData *MessageData `protobuf:"bytes,1,opt,name=messageData" json:"messageData,omitempty"` - // response specific data - Message string `protobuf:"bytes,2,opt,name=message,proto3" json:"message,omitempty"` -} - -func (m *EchoResponse) Reset() { *m = EchoResponse{} } -func (m *EchoResponse) String() string { return proto.CompactTextString(m) } -func (*EchoResponse) ProtoMessage() {} - -func (m *EchoResponse) GetMessageData() *MessageData { - if m != nil { - return m.MessageData - } - return nil -} - -func init() { - proto.RegisterType((*MessageData)(nil), "protocols.p2p.MessageData") - proto.RegisterType((*PingRequest)(nil), "protocols.p2p.PingRequest") - proto.RegisterType((*PingResponse)(nil), "protocols.p2p.PingResponse") - proto.RegisterType((*EchoRequest)(nil), "protocols.p2p.EchoRequest") - proto.RegisterType((*EchoResponse)(nil), "protocols.p2p.EchoResponse") -} diff --git a/examples/multipro/pb/p2p.proto b/examples/multipro/pb/p2p.proto deleted file mode 100644 index 53e652154b..0000000000 --- a/examples/multipro/pb/p2p.proto +++ /dev/null @@ -1,56 +0,0 @@ -syntax = "proto3"; - -package protocols.p2p; - -// designed to be shared between all app protocols -message MessageData { - // shared between all requests - string clientVersion = 1; // client version - int64 timestamp = 2; // unix time - string id = 3; // allows requesters to use request data when processing a response - bool gossip = 4; // true to have receiver peer gossip the message to neighbors - string nodeId = 5; // id of node that created the message (not the peer that may have sent it). =base58(mh(sha256(nodePubKey))) - bytes nodePubKey = 6; // Authoring node Secp256k1 public key (32bytes) - protobufs serielized - string sign = 7; // signature of message data + method specific data by message authoring node. format: string([]bytes) -} - -//// ping protocol - -// a protocol define a set of reuqest and responses -message PingRequest { - MessageData messageData = 1; - - // method specific data - string message = 2; - // add any data here.... -} - -message PingResponse { - MessageData messageData = 1; - - // response specific data - string message = 2; - - // ... add any additional message data here -} - -//// echo protocol - -// a protocol define a set of reuqest and responses -message EchoRequest { - MessageData messageData = 1; - - // method specific data - string message = 2; - - // add any additional message data here.... -} - -message EchoResponse { - MessageData messageData = 1; - - // response specific data - string message = 2; - - // ... add any additional message data here.... -} diff --git a/examples/multipro/pb/readme.md b/examples/multipro/pb/readme.md deleted file mode 100644 index ec7114f6ad..0000000000 --- a/examples/multipro/pb/readme.md +++ /dev/null @@ -1,4 +0,0 @@ -# building p2p.pb.go: -protoc --gogo_out=. --proto_path=../../../../../../:/usr/local/opt/protobuf/include:. *.proto - - diff --git a/examples/multipro/ping.go b/examples/multipro/ping.go deleted file mode 100644 index f342aa69b3..0000000000 --- a/examples/multipro/ping.go +++ /dev/null @@ -1,148 +0,0 @@ -package main - -import ( - "bufio" - "context" - "fmt" - "log" - - uuid "github.com/google/uuid" - "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - p2p "github.com/libp2p/go-libp2p/examples/multipro/pb" - protobufCodec "github.com/multiformats/go-multicodec/protobuf" -) - -// pattern: /protocol-name/request-or-response-message/version -const pingRequest = "/ping/pingreq/0.0.1" -const pingResponse = "/ping/pingresp/0.0.1" - -// PingProtocol type -type PingProtocol struct { - node *Node // local host - requests map[string]*p2p.PingRequest // used to access request data from response handlers - done chan bool // only for demo purposes to stop main from terminating -} - -func NewPingProtocol(node *Node, done chan bool) *PingProtocol { - p := &PingProtocol{node: node, requests: make(map[string]*p2p.PingRequest), done: done} - node.SetStreamHandler(pingRequest, p.onPingRequest) - node.SetStreamHandler(pingResponse, p.onPingResponse) - return p -} - -// remote peer requests handler -func (p *PingProtocol) onPingRequest(s inet.Stream) { - - // get request data - data := &p2p.PingRequest{} - decoder := protobufCodec.Multicodec(nil).Decoder(bufio.NewReader(s)) - err := decoder.Decode(data) - if err != nil { - log.Println(err) - return - } - - log.Printf("%s: Received ping request from %s. Message: %s", s.Conn().LocalPeer(), s.Conn().RemotePeer(), data.Message) - - valid := p.node.authenticateMessage(data, data.MessageData) - - if !valid { - log.Println("Failed to authenticate message") - return - } - - // generate response message - log.Printf("%s: Sending ping response to %s. Message id: %s...", s.Conn().LocalPeer(), s.Conn().RemotePeer(), data.MessageData.Id) - - resp := &p2p.PingResponse{MessageData: p.node.NewMessageData(data.MessageData.Id, false), - Message: fmt.Sprintf("Ping response from %s", p.node.ID())} - - // sign the data - signature, err := p.node.signProtoMessage(resp) - if err != nil { - log.Println("failed to sign response") - return - } - - // add the signature to the message - resp.MessageData.Sign = string(signature) - - // send the response - s, respErr := p.node.NewStream(context.Background(), s.Conn().RemotePeer(), pingResponse) - if respErr != nil { - log.Println(respErr) - return - } - - ok := p.node.sendProtoMessage(resp, s) - - if ok { - log.Printf("%s: Ping response to %s sent.", s.Conn().LocalPeer().String(), s.Conn().RemotePeer().String()) - } -} - -// remote ping response handler -func (p *PingProtocol) onPingResponse(s inet.Stream) { - data := &p2p.PingResponse{} - decoder := protobufCodec.Multicodec(nil).Decoder(bufio.NewReader(s)) - err := decoder.Decode(data) - if err != nil { - return - } - - valid := p.node.authenticateMessage(data, data.MessageData) - - if !valid { - log.Println("Failed to authenticate message") - return - } - - // locate request data and remove it if found - _, ok := p.requests[data.MessageData.Id] - if ok { - // remove request from map as we have processed it here - delete(p.requests, data.MessageData.Id) - } else { - log.Println("Failed to locate request data boject for response") - return - } - - log.Printf("%s: Received ping response from %s. Message id:%s. Message: %s.", s.Conn().LocalPeer(), s.Conn().RemotePeer(), data.MessageData.Id, data.Message) - p.done <- true -} - -func (p *PingProtocol) Ping(host host.Host) bool { - log.Printf("%s: Sending ping to: %s....", p.node.ID(), host.ID()) - - // create message data - req := &p2p.PingRequest{MessageData: p.node.NewMessageData(uuid.New().String(), false), - Message: fmt.Sprintf("Ping from %s", p.node.ID())} - - // sign the data - signature, err := p.node.signProtoMessage(req) - if err != nil { - log.Println("failed to sign pb data") - return false - } - - // add the signature to the message - req.MessageData.Sign = string(signature) - - s, err := p.node.NewStream(context.Background(), host.ID(), pingRequest) - if err != nil { - log.Println(err) - return false - } - - ok := p.node.sendProtoMessage(req, s) - - if !ok { - return false - } - - // store ref request so response handler has access to it - p.requests[req.MessageData.Id] = req - log.Printf("%s: Ping to: %s was sent. Message Id: %s, Message: %s", p.node.ID(), host.ID(), req.MessageData.Id, req.Message) - return true -} diff --git a/examples/protocol-multiplexing-with-multicodecs/README.md b/examples/protocol-multiplexing-with-multicodecs/README.md deleted file mode 100644 index 0b873f51a6..0000000000 --- a/examples/protocol-multiplexing-with-multicodecs/README.md +++ /dev/null @@ -1,41 +0,0 @@ - - -# Protocol Multiplexing using multicodecs with libp2p - -This examples shows how to use multicodecs (i.e. json) to encode and transmit information between LibP2P hosts using LibP2P Streams. - -Multicodecs present a common interface, making it very easy to swap the codec implementation if needed. - -This example expects that you area already familiar with the [echo example](https://github.com/libp2p/go-libp2p/tree/master/examples/echo). - -## Build - -From `go-libp2p` base folder: - -``` -> make deps-protocol-muxing -> go build -o multicodecs ./examples/protocol-multiplexing-with-multicodecs -``` - -## Usage - -``` -> ./multicodecs - -``` - -## Details - -The example creates two LibP2P Hosts. Host1 opens a stream to Host2. Host2 has an `StreamHandler` to deal with the incoming stream. This is covered in the `echo` example. - -Both hosts simulate a conversation. But rather than sending raw messages on the stream, each message in the conversation is encoded under a `json` object (using the `json` multicodec). For example: - -``` -{ - "Msg": "This is the message", - "Index": 3, - "HangUp": false -} -``` - -The stream lasts until one of the sides closes it when the HangUp field is `true`. diff --git a/examples/protocol-multiplexing-with-multicodecs/main.go b/examples/protocol-multiplexing-with-multicodecs/main.go deleted file mode 100644 index 5aa4f6fd6b..0000000000 --- a/examples/protocol-multiplexing-with-multicodecs/main.go +++ /dev/null @@ -1,171 +0,0 @@ -package main - -import ( - "bufio" - "context" - "fmt" - "log" - "math/rand" - "time" - - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - ps "github.com/libp2p/go-libp2p-peerstore" - - libp2p "github.com/libp2p/go-libp2p" - multicodec "github.com/multiformats/go-multicodec" - json "github.com/multiformats/go-multicodec/json" -) - -const proto = "/example/1.0.0" - -// Message is a serializable/encodable object that we will send -// on a Stream. -type Message struct { - Msg string - Index int - HangUp bool -} - -// streamWrap wraps a libp2p stream. We encode/decode whenever we -// write/read from a stream, so we can just carry the encoders -// and bufios with us -type WrappedStream struct { - stream inet.Stream - enc multicodec.Encoder - dec multicodec.Decoder - w *bufio.Writer - r *bufio.Reader -} - -// wrapStream takes a stream and complements it with r/w bufios and -// decoder/encoder. In order to write raw data to the stream we can use -// wrap.w.Write(). To encode something into it we can wrap.enc.Encode(). -// Finally, we should wrap.w.Flush() to actually send the data. Handling -// incoming data works similarly with wrap.r.Read() for raw-reading and -// wrap.dec.Decode() to decode. -func WrapStream(s inet.Stream) *WrappedStream { - reader := bufio.NewReader(s) - writer := bufio.NewWriter(s) - // This is where we pick our specific multicodec. In order to change the - // codec, we only need to change this place. - // See https://godoc.org/github.com/multiformats/go-multicodec/json - dec := json.Multicodec(false).Decoder(reader) - enc := json.Multicodec(false).Encoder(writer) - return &WrappedStream{ - stream: s, - r: reader, - w: writer, - enc: enc, - dec: dec, - } -} - -// messages that will be sent between the hosts. -var conversationMsgs = []string{ - "Hello!", - "Hey!", - "How are you doing?", - "Very good! It is great that you can send data on a stream to me!", - "Not only that, the data is encoded in a JSON object.", - "Yeah, and we are using the multicodecs interface to encode and decode.", - "This way we could swap it easily for, say, cbor, or msgpack!", - "Let's leave that as an excercise for the reader...", - "Agreed, our last message should activate the HangUp flag", - "Yes, and the example code will close streams. So sad :(. Bye!", -} - -func makeRandomHost(port int) host.Host { - h, err := libp2p.New(context.Background(), libp2p.ListenAddrStrings(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", port))) - if err != nil { - panic(err) - } - return h -} - -func main() { - // Choose random ports between 10000-10100 - rand.Seed(666) - port1 := rand.Intn(100) + 10000 - port2 := port1 + 1 - - // Make 2 hosts - h1 := makeRandomHost(port1) - h2 := makeRandomHost(port2) - h1.Peerstore().AddAddrs(h2.ID(), h2.Addrs(), ps.PermanentAddrTTL) - h2.Peerstore().AddAddrs(h1.ID(), h1.Addrs(), ps.PermanentAddrTTL) - - log.Printf("This is a conversation between %s and %s\n", h1.ID(), h2.ID()) - - // Define a stream handler for host number 2 - h2.SetStreamHandler(proto, func(stream inet.Stream) { - log.Printf("%s: Received a stream", h2.ID()) - wrappedStream := WrapStream(stream) - defer stream.Close() - handleStream(wrappedStream) - }) - - // Create new stream from h1 to h2 and start the conversation - stream, err := h1.NewStream(context.Background(), h2.ID(), proto) - if err != nil { - log.Fatal(err) - } - wrappedStream := WrapStream(stream) - // This sends the first message - sendMessage(0, wrappedStream) - // We keep the conversation on the created stream so we launch - // this to handle any responses - handleStream(wrappedStream) - // When we are done, close the stream on our side and exit. - stream.Close() -} - -// receiveMessage reads and decodes a message from the stream -func receiveMessage(ws *WrappedStream) (*Message, error) { - var msg Message - err := ws.dec.Decode(&msg) - if err != nil { - return nil, err - } - return &msg, nil -} - -// sendMessage encodes and writes a message to the stream -func sendMessage(index int, ws *WrappedStream) error { - msg := &Message{ - Msg: conversationMsgs[index], - Index: index, - HangUp: index >= len(conversationMsgs)-1, - } - - err := ws.enc.Encode(msg) - // Because output is buffered with bufio, we need to flush! - ws.w.Flush() - return err -} - -// handleStream is a for loop which receives and then sends a message -// an artificial delay of 500ms happens in-between. -// When Message.HangUp is true, it exists. This will close the stream -// on one of the sides. The other side's receiveMessage() will error -// with EOF, thus also breaking out from the loop. -func handleStream(ws *WrappedStream) { - for { - // Read - msg, err := receiveMessage(ws) - if err != nil { - break - } - pid := ws.stream.Conn().LocalPeer() - log.Printf("%s says: %s\n", pid, msg.Msg) - time.Sleep(500 * time.Millisecond) - if msg.HangUp { - break - } - // Send response - err = sendMessage(msg.Index+1, ws) - if err != nil { - break - } - } -} diff --git a/package.json b/package.json index 1df54e44e0..367fa5491e 100644 --- a/package.json +++ b/package.json @@ -160,12 +160,6 @@ "name": "go-smux-yamux", "version": "2.0.3" }, - { - "author": "whyrusleeping", - "hash": "QmQvJiADDe7JR4m968MwXobTCCzUqQkP87aRHe29MEBGHV", - "name": "go-logging", - "version": "0.0.0" - }, { "author": "whyrusleeping", "hash": "QmWzjXAyBTygw6CeCTUnhJzhFucfxY5FJivSoiGuiSbPjS", @@ -202,12 +196,6 @@ "name": "go-smux-multiplex", "version": "3.0.11" }, - { - "author": "multiformats", - "hash": "QmRDePEiL4Yupq5EkcK3L3ko3iMgYaqUdLu7xc1kqs7dnV", - "name": "go-multicodec", - "version": "0.1.5" - }, { "author": "whyrusleeping", "hash": "QmX4UrmHGPnFxwfsunZjPbykzyv8Frg9AVmNariXqrLsMs", @@ -243,12 +231,6 @@ "hash": "Qma7AuxEA7dd1wAy95hTxXgxy4q7mU4Pyd1x4PRAzGP1fs", "name": "go-libp2p-transport-upgrader", "version": "0.1.5" - }, - { - "author": "google", - "hash": "QmSSeQqc5QeuefkaM6JFV5tSF9knLUkXKVhW1eYRiqe72W", - "name": "uuid", - "version": "0.1.0" } ], "gxVersion": "0.4.0", From 075d336c2fb00c9da474934fe332a3fabe79a391 Mon Sep 17 00:00:00 2001 From: qidu Date: Mon, 12 Mar 2018 18:12:27 +0800 Subject: [PATCH 0526/3965] Update README.md to get the source right --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b44b25dec3..778f6eb865 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ There is currently only one bundle of `go-libp2p`, this package. This bundle is ### Install ```bash -> go get -d github.com/libp2p/go-libp2p/... +> go get -u -d github.com/libp2p/go-libp2p/... > cd $GOPATH/src/github.com/libp2p/go-libp2p > make > make deps From ecf27b599f6636e9cefa1b1153679afe2bdbb794 Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Sat, 28 Jul 2018 02:08:45 +0530 Subject: [PATCH 0527/3965] strict DefaultListenAddrs test w/ regex --- libp2p_test.go | 27 +++++++++------------------ 1 file changed, 9 insertions(+), 18 deletions(-) diff --git a/libp2p_test.go b/libp2p_test.go index 20370d540c..a93082440f 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -3,6 +3,7 @@ package libp2p import ( "context" "fmt" + "regexp" "strings" "testing" @@ -43,14 +44,19 @@ func TestInsecure(t *testing.T) { func TestDefaultListenAddrs(t *testing.T) { ctx := context.Background() + re := regexp.MustCompile("/ip[4|6]/0.0.0.0/tcp/") + // Test 1: Setting the correct listen addresses if userDefined.Transport == nil && userDefined.ListenAddrs == nil h, err := New(ctx) if err != nil { t.Fatal(err) } - if len(h.Addrs()) != 2 { - t.Error("expected 2 default listen addrs") + for _, addr := range h.Network().ListenAddresses() { + if re.FindStringSubmatchIndex(addr.String()) == nil { + t.Error("expected ip4 or ip6 interface") + } } + h.Close() // Test 2: Listen addr should not set if user defined transport is passed. @@ -62,25 +68,10 @@ func TestDefaultListenAddrs(t *testing.T) { t.Fatal(err) } - if len(h.Addrs()) != 0 { + if len(h.Network().ListenAddresses()) != 0 { t.Error("expected zero listen addrs as none is set with user defined transport") } h.Close() - - // Test 3: User defined listener addrs should overwrite the default options. - h, err = New( - ctx, - Transport(tcp.NewTCPTransport), - ListenAddrStrings("/ip4/127.0.0.1/tcp/0"), - ) - if err != nil { - t.Fatal(err) - } - - if len(h.Addrs()) != 1 { - t.Error("expected one listen addr") - } - h.Close() } func makeRandomHost(t *testing.T, port int) (host.Host, error) { From 617d7a5797e550e4655ba8a8d240fb6d4e11766a Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Sat, 28 Jul 2018 02:47:27 +0530 Subject: [PATCH 0528/3965] Corrected ip6 string --- defaults.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/defaults.go b/defaults.go index 973cf5c168..1bfb77f789 100644 --- a/defaults.go +++ b/defaults.go @@ -60,7 +60,7 @@ var DefaultListenAddrs = func(cfg *Config) error { return err } - defaultIP6ListenAddr, err := multiaddr.NewMultiaddr("/ip6/0.0.0.0/tcp/0") + defaultIP6ListenAddr, err := multiaddr.NewMultiaddr("/ip6/::/tcp/0") if err != nil { return err } From 9e952be41289e2cc3d4890c6c186a58883be46ca Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Sat, 28 Jul 2018 02:48:09 +0530 Subject: [PATCH 0529/3965] modified regex for ipv6 --- libp2p_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libp2p_test.go b/libp2p_test.go index a93082440f..293e0fef7c 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -44,7 +44,7 @@ func TestInsecure(t *testing.T) { func TestDefaultListenAddrs(t *testing.T) { ctx := context.Background() - re := regexp.MustCompile("/ip[4|6]/0.0.0.0/tcp/") + re := regexp.MustCompile("/(ip)[4|6]/((0.0.0.0)|(::))/tcp/") // Test 1: Setting the correct listen addresses if userDefined.Transport == nil && userDefined.ListenAddrs == nil h, err := New(ctx) From fcaa18df9a78301b78e8a61bc3f1e320fd244e39 Mon Sep 17 00:00:00 2001 From: Florian Lenz Date: Mon, 30 Jul 2018 22:29:16 +0300 Subject: [PATCH 0530/3965] [muxer] set mplex version to 6.7.0 --- defaults.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/defaults.go b/defaults.go index 1bfb77f789..90ef45bb8a 100644 --- a/defaults.go +++ b/defaults.go @@ -27,7 +27,7 @@ var DefaultSecurity = Security(secio.ID, secio.New) // libp2p instead of replacing them. var DefaultMuxers = ChainOptions( Muxer("/yamux/1.0.0", yamux.DefaultTransport), - Muxer("/mplex/6.3.0", mplex.DefaultTransport), + Muxer("/mplex/6.7.0", mplex.DefaultTransport), ) // DefaultTransports are the default libp2p transports. From 9a55577ec532567cf931c191c4d68ed2a9fc1c37 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 14 Jul 2018 11:52:35 +0200 Subject: [PATCH 0531/3965] obey the multiaddr's IP version when listening --- p2p/transport/quic/listener.go | 12 +++- p2p/transport/quic/listener_test.go | 102 ++++++++++++++++++---------- 2 files changed, 77 insertions(+), 37 deletions(-) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 52558872d6..1cdc79e208 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -27,11 +27,19 @@ type listener struct { var _ tpt.Listener = &listener{} func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, key ic.PrivKey, tlsConf *tls.Config) (tpt.Listener, error) { - _, host, err := manet.DialArgs(addr) + lnet, host, err := manet.DialArgs(addr) if err != nil { return nil, err } - ln, err := quicListenAddr(host, tlsConf, quicConfig) + laddr, err := net.ResolveUDPAddr(lnet, host) + if err != nil { + return nil, err + } + conn, err := net.ListenUDP(lnet, laddr) + if err != nil { + return nil, err + } + ln, err := quic.Listen(conn, tlsConf, quicConfig) if err != nil { return nil, err } diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index 1069d57e99..dd7974cf45 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -15,10 +15,7 @@ import ( ) var _ = Describe("Listener", func() { - var ( - t tpt.Transport - localAddr ma.Multiaddr - ) + var t tpt.Transport BeforeEach(func() { rsaKey, err := rsa.GenerateKey(rand.Reader, 1024) @@ -27,41 +24,76 @@ var _ = Describe("Listener", func() { Expect(err).ToNot(HaveOccurred()) t, err = NewTransport(key) Expect(err).ToNot(HaveOccurred()) - localAddr, err = ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") - Expect(err).ToNot(HaveOccurred()) }) - It("returns the address it is listening on", func() { - ln, err := t.Listen(localAddr) - Expect(err).ToNot(HaveOccurred()) - netAddr := ln.Addr() - Expect(netAddr).To(BeAssignableToTypeOf(&net.UDPAddr{})) - port := netAddr.(*net.UDPAddr).Port - Expect(port).ToNot(BeZero()) - Expect(ln.Multiaddr().String()).To(Equal(fmt.Sprintf("/ip4/127.0.0.1/udp/%d/quic", port))) - }) + Context("listening on the right address", func() { + It("returns the address it is listening on", func() { + localAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + ln, err := t.Listen(localAddr) + Expect(err).ToNot(HaveOccurred()) + netAddr := ln.Addr() + Expect(netAddr).To(BeAssignableToTypeOf(&net.UDPAddr{})) + port := netAddr.(*net.UDPAddr).Port + Expect(port).ToNot(BeZero()) + Expect(ln.Multiaddr().String()).To(Equal(fmt.Sprintf("/ip4/127.0.0.1/udp/%d/quic", port))) + }) - It("returns Accept when it is closed", func() { - addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") - Expect(err).ToNot(HaveOccurred()) - ln, err := t.Listen(addr) - Expect(err).ToNot(HaveOccurred()) - done := make(chan struct{}) - go func() { - defer GinkgoRecover() - ln.Accept() - close(done) - }() - Consistently(done).ShouldNot(BeClosed()) - Expect(ln.Close()).To(Succeed()) - Eventually(done).Should(BeClosed()) + It("returns the address it is listening on, for listening on IPv4", func() { + localAddr, err := ma.NewMultiaddr("/ip4/0.0.0.0/udp/0/quic") + Expect(err).ToNot(HaveOccurred()) + ln, err := t.Listen(localAddr) + Expect(err).ToNot(HaveOccurred()) + netAddr := ln.Addr() + Expect(netAddr).To(BeAssignableToTypeOf(&net.UDPAddr{})) + port := netAddr.(*net.UDPAddr).Port + Expect(port).ToNot(BeZero()) + Expect(ln.Multiaddr().String()).To(Equal(fmt.Sprintf("/ip4/0.0.0.0/udp/%d/quic", port))) + }) + + It("returns the address it is listening on, for listening on IPv6", func() { + localAddr, err := ma.NewMultiaddr("/ip6/::/udp/0/quic") + Expect(err).ToNot(HaveOccurred()) + ln, err := t.Listen(localAddr) + Expect(err).ToNot(HaveOccurred()) + netAddr := ln.Addr() + Expect(netAddr).To(BeAssignableToTypeOf(&net.UDPAddr{})) + port := netAddr.(*net.UDPAddr).Port + Expect(port).ToNot(BeZero()) + Expect(ln.Multiaddr().String()).To(Equal(fmt.Sprintf("/ip6/::/udp/%d/quic", port))) + }) }) - It("doesn't accept Accept calls after it is closed", func() { - ln, err := t.Listen(localAddr) - Expect(err).ToNot(HaveOccurred()) - Expect(ln.Close()).To(Succeed()) - _, err = ln.Accept() - Expect(err).To(HaveOccurred()) + Context("accepting connections", func() { + var localAddr ma.Multiaddr + + BeforeEach(func() { + var err error + localAddr, err = ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + Expect(err).ToNot(HaveOccurred()) + }) + + It("returns Accept when it is closed", func() { + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + Expect(err).ToNot(HaveOccurred()) + ln, err := t.Listen(addr) + Expect(err).ToNot(HaveOccurred()) + done := make(chan struct{}) + go func() { + defer GinkgoRecover() + ln.Accept() + close(done) + }() + Consistently(done).ShouldNot(BeClosed()) + Expect(ln.Close()).To(Succeed()) + Eventually(done).Should(BeClosed()) + }) + + It("doesn't accept Accept calls after it is closed", func() { + ln, err := t.Listen(localAddr) + Expect(err).ToNot(HaveOccurred()) + Expect(ln.Close()).To(Succeed()) + _, err = ln.Accept() + Expect(err).To(HaveOccurred()) + }) }) }) From ffe5aa8693a07c80f23d624d857243f7f9a20f1a Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 14 Jul 2018 13:54:36 +0200 Subject: [PATCH 0532/3965] obey the multiaddr's IP version when dialing --- p2p/transport/quic/conn_test.go | 42 ++++++++++++++----- p2p/transport/quic/transport.go | 71 +++++++++++++++++++++++---------- 2 files changed, 82 insertions(+), 31 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 1a1e3f4a67..d4744882a0 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -35,12 +35,12 @@ var _ = Describe("Connection", func() { return id, priv } - runServer := func(tr tpt.Transport) (ma.Multiaddr, <-chan tpt.Conn) { + runServer := func(tr tpt.Transport, multiaddr string) (ma.Multiaddr, <-chan tpt.Conn) { addrChan := make(chan ma.Multiaddr) connChan := make(chan tpt.Conn) go func() { defer GinkgoRecover() - addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + addr, err := ma.NewMultiaddr(multiaddr) Expect(err).ToNot(HaveOccurred()) ln, err := tr.Listen(addr) Expect(err).ToNot(HaveOccurred()) @@ -62,10 +62,30 @@ var _ = Describe("Connection", func() { clientID, clientKey = createPeer() }) - It("handshakes", func() { + It("handshakes on IPv4", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport) + serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + + clientTransport, err := NewTransport(clientKey) + Expect(err).ToNot(HaveOccurred()) + conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) + Expect(err).ToNot(HaveOccurred()) + serverConn := <-serverConnChan + Expect(conn.LocalPeer()).To(Equal(clientID)) + Expect(conn.LocalPrivateKey()).To(Equal(clientKey)) + Expect(conn.RemotePeer()).To(Equal(serverID)) + Expect(conn.RemotePublicKey()).To(Equal(serverKey.GetPublic())) + Expect(serverConn.LocalPeer()).To(Equal(serverID)) + Expect(serverConn.LocalPrivateKey()).To(Equal(serverKey)) + Expect(serverConn.RemotePeer()).To(Equal(clientID)) + Expect(serverConn.RemotePublicKey()).To(Equal(clientKey.GetPublic())) + }) + + It("handshakes on IPv6", func() { + serverTransport, err := NewTransport(serverKey) + Expect(err).ToNot(HaveOccurred()) + serverAddr, serverConnChan := runServer(serverTransport, "/ip6/::1/udp/0/quic") clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) @@ -85,7 +105,7 @@ var _ = Describe("Connection", func() { It("opens and accepts streams", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport) + serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) @@ -110,7 +130,7 @@ var _ = Describe("Connection", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport) + serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) @@ -124,7 +144,7 @@ var _ = Describe("Connection", func() { It("fails if the client presents an invalid cert chain", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport) + serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") clientTransport, err := NewTransport(clientKey) invalidateCertChain(clientTransport.(*transport).tlsConf) @@ -139,7 +159,7 @@ var _ = Describe("Connection", func() { serverTransport, err := NewTransport(serverKey) invalidateCertChain(serverTransport.(*transport).tlsConf) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport) + serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) @@ -152,7 +172,7 @@ var _ = Describe("Connection", func() { It("keeps accepting connections after a failed connection attempt", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport) + serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") // first dial with an invalid cert chain clientTransport1, err := NewTransport(clientKey) @@ -175,10 +195,10 @@ var _ = Describe("Connection", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport) + serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") serverTransport2, err := NewTransport(serverKey2) Expect(err).ToNot(HaveOccurred()) - serverAddr2, serverConnChan2 := runServer(serverTransport2) + serverAddr2, serverConnChan2 := runServer(serverTransport2, "/ip4/127.0.0.1/udp/0/quic") data := bytes.Repeat([]byte{'a'}, 5*1<<20) // 5 MB // wait for both servers to accept a connection diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 70f2d6d357..2fb16bec11 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -5,7 +5,9 @@ import ( "crypto/tls" "crypto/x509" "errors" + "fmt" "net" + "sync" ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" @@ -27,12 +29,47 @@ var quicConfig = &quic.Config{ KeepAlive: true, } +type connManager struct { + connIPv4Once sync.Once + connIPv4 net.PacketConn + + connIPv6Once sync.Once + connIPv6 net.PacketConn +} + +func (c *connManager) GetConnForAddr(network string) (net.PacketConn, error) { + switch network { + case "udp4": + var err error + c.connIPv4Once.Do(func() { + c.connIPv4, err = c.createConn(network, "0.0.0.0:0") + }) + return c.connIPv4, err + case "udp6": + var err error + c.connIPv6Once.Do(func() { + c.connIPv6, err = c.createConn(network, ":0") + }) + return c.connIPv6, err + default: + return nil, fmt.Errorf("unsupported network: %s", network) + } +} + +func (c *connManager) createConn(network, host string) (net.PacketConn, error) { + addr, err := net.ResolveUDPAddr(network, host) + if err != nil { + return nil, err + } + return net.ListenUDP(network, addr) +} + // The Transport implements the tpt.Transport interface for QUIC connections. type transport struct { - privKey ic.PrivKey - localPeer peer.ID - tlsConf *tls.Config - pconn net.PacketConn + privKey ic.PrivKey + localPeer peer.ID + tlsConf *tls.Config + connManager *connManager } var _ tpt.Transport = &transport{} @@ -48,27 +85,21 @@ func NewTransport(key ic.PrivKey) (tpt.Transport, error) { return nil, err } - // create a packet conn for outgoing connections - addr, err := net.ResolveUDPAddr("udp", "localhost:0") - if err != nil { - return nil, err - } - conn, err := net.ListenUDP("udp", addr) - if err != nil { - return nil, err - } - return &transport{ - privKey: key, - localPeer: localPeer, - tlsConf: tlsConf, - pconn: conn, + privKey: key, + localPeer: localPeer, + tlsConf: tlsConf, + connManager: &connManager{}, }, nil } // Dial dials a new QUIC connection func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { - _, host, err := manet.DialArgs(raddr) + network, host, err := manet.DialArgs(raddr) + if err != nil { + return nil, err + } + pconn, err := t.connManager.GetConnForAddr(network) if err != nil { return nil, err } @@ -100,7 +131,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp } return nil } - sess, err := quic.DialContext(ctx, t.pconn, addr, host, tlsConf, quicConfig) + sess, err := quic.DialContext(ctx, pconn, addr, host, tlsConf, quicConfig) if err != nil { return nil, err } From 7133f4124f7cac148da779c750aaf02506336f6c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Aug 2018 14:18:47 -0700 Subject: [PATCH 0533/3965] when we do a nat port mapping, explicitly remember the mapped address related to #383 I won't call this *fixed* yet but it should help. --- p2p/host/basic/natmgr.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/p2p/host/basic/natmgr.go b/p2p/host/basic/natmgr.go index 349f79e789..be4837503e 100644 --- a/p2p/host/basic/natmgr.go +++ b/p2p/host/basic/natmgr.go @@ -8,6 +8,7 @@ import ( lgbl "github.com/libp2p/go-libp2p-loggables" inat "github.com/libp2p/go-libp2p-nat" inet "github.com/libp2p/go-libp2p-net" + pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) @@ -187,6 +188,9 @@ func addPortMapping(nmgr *natManager, intaddr ma.Multiaddr) { return } + // TODO: make these temporary and rediscover them. + nmgr.net.Peerstore().AddAddr(nmgr.net.LocalPeer(), extaddr, pstore.PermanentAddrTTL) + lm["outcome"] = "success" lm["externalAddr"] = func() interface{} { return extaddr.String() } log.Infof("established nat port mapping: %s <--> %s", intaddr, extaddr) From 8dcf45bc813c0441f74fbddef838020e47d63638 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 5 Aug 2018 07:40:08 +0700 Subject: [PATCH 0534/3965] generate certificates that are valid for 180 days --- p2p/transport/quic/crypto.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/transport/quic/crypto.go b/p2p/transport/quic/crypto.go index 5834a83fcc..a9aabeefd2 100644 --- a/p2p/transport/quic/crypto.go +++ b/p2p/transport/quic/crypto.go @@ -18,6 +18,8 @@ import ( // mint certificate selection is broken. const hostname = "quic.ipfs" +const certValidityPeriod = 180 * 24 * time.Hour + func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { key, hostCert, err := keyToCertificate(privKey) if err != nil { @@ -35,7 +37,7 @@ func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { DNSNames: []string{hostname}, SerialNumber: big.NewInt(1), NotBefore: time.Now().Add(-24 * time.Hour), - NotAfter: time.Now().Add(30 * 24 * time.Hour), + NotAfter: time.Now().Add(certValidityPeriod), } certDER, err := x509.CreateCertificate(rand.Reader, certTemplate, hostCert, ephemeralKey.Public(), key) if err != nil { @@ -80,7 +82,7 @@ func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { tmpl := &x509.Certificate{ SerialNumber: sn, NotBefore: time.Now().Add(-24 * time.Hour), - NotAfter: time.Now().Add(30 * 24 * time.Hour), + NotAfter: time.Now().Add(certValidityPeriod), IsCA: true, BasicConstraintsValid: true, } From 197b42caeafd32c916fc14e844f9374c2e220eeb Mon Sep 17 00:00:00 2001 From: gpestana Date: Sun, 5 Aug 2018 20:28:00 +0300 Subject: [PATCH 0535/3965] docs: Document defaults #382 --- libp2p.go | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/libp2p.go b/libp2p.go index 936b2220d0..7d0621c94c 100644 --- a/libp2p.go +++ b/libp2p.go @@ -27,8 +27,27 @@ func ChainOptions(opts ...Option) Option { } } -// New constructs a new libp2p node with the given options (falling back on -// reasonable defaults). +// New constructs a new libp2p node with the given options, falling back on +// reasonable defaults. The defaults are: +// +// - If no transport and listen addresses are provided, the node listens to +// the multiaddresses "/ip4/0.0.0.0/tcp/0" and "/ip6/::/tcp/0"; +// +// - If no transport options are provided, the node uses TPC and websocket +// transport protocols; +// +// - If no multiplexer configuration is provided, the node is configured by +// default to use the "yamux/1.0.0" and "mplux/6.7.0" stream connection +// multiplexers; +// +// - If no security transport is provided, the host uses the go-libp2p's secio +// encrypted transport to encrypt all traffic; +// +// - If no peer identity is provided, it generates a random RSA 2048 key-par +// and derives a new identity from it; +// +// - If no peerstore is provided, the host is initialized with an empty +// peerstore. // // Canceling the passed context will stop the returned libp2p node. func New(ctx context.Context, opts ...Option) (host.Host, error) { From f80c91f49fa0b5dfc89a50798aa517183e4a8ca4 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 Aug 2018 10:33:44 -0700 Subject: [PATCH 0536/3965] update gogo protobuf and regenerate protobuf file --- .../internal/circuitv1-deprecated/pb/Makefile | 5 +- .../circuitv1-deprecated/pb/relay.pb.go | 671 +++++++++++++++++- 2 files changed, 655 insertions(+), 21 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/Makefile b/p2p/protocol/internal/circuitv1-deprecated/pb/Makefile index 95e2e1c206..eb14b5768a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/pb/Makefile +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/Makefile @@ -4,7 +4,8 @@ GO = $(PB:.proto=.pb.go) all: $(GO) %.pb.go: %.proto - protoc --gogo_out=. $< + protoc --proto_path=$(GOPATH)/src:. --gogofast_out=. $< clean: - rm *.pb.go + rm -f *.pb.go + rm -f *.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go index 9c3fc2b466..5a98753bab 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go @@ -1,27 +1,27 @@ -// Code generated by protoc-gen-gogo. +// Code generated by protoc-gen-gogo. DO NOT EDIT. // source: relay.proto -// DO NOT EDIT! -/* -Package relay_pb is a generated protocol buffer package. - -It is generated from these files: - relay.proto - -It has these top-level messages: - CircuitRelay -*/ package relay_pb import proto "github.com/gogo/protobuf/proto" import fmt "fmt" import math "math" +import github_com_gogo_protobuf_proto "github.com/gogo/protobuf/proto" + +import io "io" + // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal var _ = fmt.Errorf var _ = math.Inf +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + type CircuitRelay_Status int32 const ( @@ -96,6 +96,9 @@ func (x *CircuitRelay_Status) UnmarshalJSON(data []byte) error { *x = CircuitRelay_Status(value) return nil } +func (CircuitRelay_Status) EnumDescriptor() ([]byte, []int) { + return fileDescriptor_relay_cd6009b961688bf5, []int{0, 0} +} type CircuitRelay_Type int32 @@ -135,18 +138,52 @@ func (x *CircuitRelay_Type) UnmarshalJSON(data []byte) error { *x = CircuitRelay_Type(value) return nil } +func (CircuitRelay_Type) EnumDescriptor() ([]byte, []int) { + return fileDescriptor_relay_cd6009b961688bf5, []int{0, 1} +} type CircuitRelay struct { - Type *CircuitRelay_Type `protobuf:"varint,1,opt,name=type,enum=relay.pb.CircuitRelay_Type" json:"type,omitempty"` - SrcPeer *CircuitRelay_Peer `protobuf:"bytes,2,opt,name=srcPeer" json:"srcPeer,omitempty"` - DstPeer *CircuitRelay_Peer `protobuf:"bytes,3,opt,name=dstPeer" json:"dstPeer,omitempty"` - Code *CircuitRelay_Status `protobuf:"varint,4,opt,name=code,enum=relay.pb.CircuitRelay_Status" json:"code,omitempty"` - XXX_unrecognized []byte `json:"-"` + Type *CircuitRelay_Type `protobuf:"varint,1,opt,name=type,enum=relay.pb.CircuitRelay_Type" json:"type,omitempty"` + SrcPeer *CircuitRelay_Peer `protobuf:"bytes,2,opt,name=srcPeer" json:"srcPeer,omitempty"` + DstPeer *CircuitRelay_Peer `protobuf:"bytes,3,opt,name=dstPeer" json:"dstPeer,omitempty"` + Code *CircuitRelay_Status `protobuf:"varint,4,opt,name=code,enum=relay.pb.CircuitRelay_Status" json:"code,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *CircuitRelay) Reset() { *m = CircuitRelay{} } func (m *CircuitRelay) String() string { return proto.CompactTextString(m) } func (*CircuitRelay) ProtoMessage() {} +func (*CircuitRelay) Descriptor() ([]byte, []int) { + return fileDescriptor_relay_cd6009b961688bf5, []int{0} +} +func (m *CircuitRelay) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *CircuitRelay) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_CircuitRelay.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *CircuitRelay) XXX_Merge(src proto.Message) { + xxx_messageInfo_CircuitRelay.Merge(dst, src) +} +func (m *CircuitRelay) XXX_Size() int { + return m.Size() +} +func (m *CircuitRelay) XXX_DiscardUnknown() { + xxx_messageInfo_CircuitRelay.DiscardUnknown(m) +} + +var xxx_messageInfo_CircuitRelay proto.InternalMessageInfo func (m *CircuitRelay) GetType() CircuitRelay_Type { if m != nil && m.Type != nil { @@ -177,14 +214,45 @@ func (m *CircuitRelay) GetCode() CircuitRelay_Status { } type CircuitRelay_Peer struct { - Id []byte `protobuf:"bytes,1,req,name=id" json:"id,omitempty"` - Addrs [][]byte `protobuf:"bytes,2,rep,name=addrs" json:"addrs,omitempty"` - XXX_unrecognized []byte `json:"-"` + Id []byte `protobuf:"bytes,1,req,name=id" json:"id,omitempty"` + Addrs [][]byte `protobuf:"bytes,2,rep,name=addrs" json:"addrs,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *CircuitRelay_Peer) Reset() { *m = CircuitRelay_Peer{} } func (m *CircuitRelay_Peer) String() string { return proto.CompactTextString(m) } func (*CircuitRelay_Peer) ProtoMessage() {} +func (*CircuitRelay_Peer) Descriptor() ([]byte, []int) { + return fileDescriptor_relay_cd6009b961688bf5, []int{0, 0} +} +func (m *CircuitRelay_Peer) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *CircuitRelay_Peer) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_CircuitRelay_Peer.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *CircuitRelay_Peer) XXX_Merge(src proto.Message) { + xxx_messageInfo_CircuitRelay_Peer.Merge(dst, src) +} +func (m *CircuitRelay_Peer) XXX_Size() int { + return m.Size() +} +func (m *CircuitRelay_Peer) XXX_DiscardUnknown() { + xxx_messageInfo_CircuitRelay_Peer.DiscardUnknown(m) +} + +var xxx_messageInfo_CircuitRelay_Peer proto.InternalMessageInfo func (m *CircuitRelay_Peer) GetId() []byte { if m != nil { @@ -206,3 +274,568 @@ func init() { proto.RegisterEnum("relay.pb.CircuitRelay_Status", CircuitRelay_Status_name, CircuitRelay_Status_value) proto.RegisterEnum("relay.pb.CircuitRelay_Type", CircuitRelay_Type_name, CircuitRelay_Type_value) } +func (m *CircuitRelay) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *CircuitRelay) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Type != nil { + dAtA[i] = 0x8 + i++ + i = encodeVarintRelay(dAtA, i, uint64(*m.Type)) + } + if m.SrcPeer != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintRelay(dAtA, i, uint64(m.SrcPeer.Size())) + n1, err := m.SrcPeer.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n1 + } + if m.DstPeer != nil { + dAtA[i] = 0x1a + i++ + i = encodeVarintRelay(dAtA, i, uint64(m.DstPeer.Size())) + n2, err := m.DstPeer.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n2 + } + if m.Code != nil { + dAtA[i] = 0x20 + i++ + i = encodeVarintRelay(dAtA, i, uint64(*m.Code)) + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func (m *CircuitRelay_Peer) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *CircuitRelay_Peer) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Id == nil { + return 0, github_com_gogo_protobuf_proto.NewRequiredNotSetError("id") + } else { + dAtA[i] = 0xa + i++ + i = encodeVarintRelay(dAtA, i, uint64(len(m.Id))) + i += copy(dAtA[i:], m.Id) + } + if len(m.Addrs) > 0 { + for _, b := range m.Addrs { + dAtA[i] = 0x12 + i++ + i = encodeVarintRelay(dAtA, i, uint64(len(b))) + i += copy(dAtA[i:], b) + } + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func encodeVarintRelay(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func (m *CircuitRelay) Size() (n int) { + var l int + _ = l + if m.Type != nil { + n += 1 + sovRelay(uint64(*m.Type)) + } + if m.SrcPeer != nil { + l = m.SrcPeer.Size() + n += 1 + l + sovRelay(uint64(l)) + } + if m.DstPeer != nil { + l = m.DstPeer.Size() + n += 1 + l + sovRelay(uint64(l)) + } + if m.Code != nil { + n += 1 + sovRelay(uint64(*m.Code)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *CircuitRelay_Peer) Size() (n int) { + var l int + _ = l + if m.Id != nil { + l = len(m.Id) + n += 1 + l + sovRelay(uint64(l)) + } + if len(m.Addrs) > 0 { + for _, b := range m.Addrs { + l = len(b) + n += 1 + l + sovRelay(uint64(l)) + } + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func sovRelay(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozRelay(x uint64) (n int) { + return sovRelay(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *CircuitRelay) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowRelay + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: CircuitRelay: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: CircuitRelay: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Type", wireType) + } + var v CircuitRelay_Type + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowRelay + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= (CircuitRelay_Type(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + m.Type = &v + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field SrcPeer", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowRelay + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthRelay + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.SrcPeer == nil { + m.SrcPeer = &CircuitRelay_Peer{} + } + if err := m.SrcPeer.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field DstPeer", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowRelay + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthRelay + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.DstPeer == nil { + m.DstPeer = &CircuitRelay_Peer{} + } + if err := m.DstPeer.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + case 4: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Code", wireType) + } + var v CircuitRelay_Status + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowRelay + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= (CircuitRelay_Status(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + m.Code = &v + default: + iNdEx = preIndex + skippy, err := skipRelay(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthRelay + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *CircuitRelay_Peer) Unmarshal(dAtA []byte) error { + var hasFields [1]uint64 + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowRelay + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Peer: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Peer: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Id", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowRelay + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthRelay + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Id = append(m.Id[:0], dAtA[iNdEx:postIndex]...) + if m.Id == nil { + m.Id = []byte{} + } + iNdEx = postIndex + hasFields[0] |= uint64(0x00000001) + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Addrs", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowRelay + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthRelay + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Addrs = append(m.Addrs, make([]byte, postIndex-iNdEx)) + copy(m.Addrs[len(m.Addrs)-1], dAtA[iNdEx:postIndex]) + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipRelay(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthRelay + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + if hasFields[0]&uint64(0x00000001) == 0 { + return github_com_gogo_protobuf_proto.NewRequiredNotSetError("id") + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipRelay(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowRelay + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowRelay + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowRelay + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + iNdEx += length + if length < 0 { + return 0, ErrInvalidLengthRelay + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowRelay + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipRelay(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthRelay = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowRelay = fmt.Errorf("proto: integer overflow") +) + +func init() { proto.RegisterFile("relay.proto", fileDescriptor_relay_cd6009b961688bf5) } + +var fileDescriptor_relay_cd6009b961688bf5 = []byte{ + // 473 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x92, 0x4f, 0x6f, 0xd3, 0x3e, + 0x18, 0xc7, 0x65, 0x27, 0xbf, 0x76, 0x7a, 0x5a, 0x4d, 0xfe, 0x59, 0x63, 0x64, 0x9d, 0x56, 0xaa, + 0x9e, 0x7a, 0x40, 0x45, 0x4c, 0xe2, 0x05, 0x98, 0xc4, 0xdd, 0x2a, 0xd2, 0x38, 0xb2, 0x5d, 0x24, + 0x4e, 0x56, 0x69, 0x72, 0xa8, 0x84, 0xd4, 0x2a, 0xcd, 0x0e, 0xbd, 0xc3, 0xb8, 0x21, 0x8e, 0xbc, + 0x1c, 0xe0, 0xc4, 0x91, 0x17, 0xc0, 0x3f, 0xf5, 0x65, 0xc0, 0x05, 0xd9, 0x5d, 0x33, 0x44, 0x37, + 0x89, 0xa3, 0x9f, 0xef, 0xe7, 0xe3, 0x3c, 0xf9, 0x26, 0xd0, 0x28, 0xf2, 0x17, 0x93, 0x55, 0x7f, + 0x51, 0xcc, 0xcb, 0x39, 0xdd, 0xbb, 0x3a, 0x3c, 0xef, 0xbe, 0xae, 0x41, 0x33, 0x9c, 0x15, 0xd3, + 0x8b, 0x59, 0x29, 0xed, 0x8c, 0x3e, 0x00, 0xbf, 0x5c, 0x2d, 0xf2, 0x00, 0x75, 0x50, 0x6f, 0xff, + 0xf4, 0xb8, 0xbf, 0x25, 0xfb, 0x7f, 0x52, 0x7d, 0xbd, 0x5a, 0xe4, 0xd2, 0x81, 0xf4, 0x11, 0xd4, + 0x97, 0xc5, 0x34, 0xcd, 0xf3, 0x22, 0xc0, 0x1d, 0xd4, 0x6b, 0xdc, 0xea, 0x58, 0x44, 0x6e, 0x59, + 0xab, 0x65, 0xcb, 0xd2, 0x69, 0xde, 0x3f, 0x68, 0x57, 0x2c, 0x7d, 0x08, 0xfe, 0x74, 0x9e, 0xe5, + 0x81, 0xef, 0xd6, 0x3b, 0xb9, 0xc5, 0x51, 0xe5, 0xa4, 0xbc, 0x58, 0x4a, 0x87, 0xb6, 0xee, 0x83, + 0xef, 0xd4, 0x7d, 0xc0, 0xb3, 0x2c, 0x40, 0x1d, 0xdc, 0x6b, 0x4a, 0x3c, 0xcb, 0xe8, 0x01, 0xfc, + 0x37, 0xc9, 0xb2, 0x62, 0x19, 0xe0, 0x8e, 0xd7, 0x6b, 0xca, 0xcd, 0xa1, 0xfb, 0xd1, 0x83, 0xda, + 0x46, 0xa7, 0x0d, 0xa8, 0xab, 0x71, 0x18, 0x72, 0xa5, 0x48, 0x46, 0x5b, 0x70, 0xe7, 0x5c, 0xa4, + 0x46, 0xc9, 0xd0, 0xb0, 0x28, 0x92, 0x46, 0x0b, 0x61, 0x62, 0x91, 0x9c, 0x91, 0x2f, 0x68, 0x9b, + 0x45, 0x4a, 0xff, 0x95, 0x7d, 0x45, 0xb4, 0x0d, 0x47, 0x5b, 0x6f, 0x34, 0x8e, 0xf5, 0xd0, 0x01, + 0xc3, 0xe4, 0x29, 0x8b, 0x87, 0x11, 0xf9, 0x59, 0xe5, 0xd6, 0xdd, 0xcd, 0x7f, 0x21, 0x7a, 0x17, + 0xa8, 0xcd, 0x13, 0x61, 0x42, 0x91, 0x24, 0x46, 0x0b, 0x8b, 0x92, 0x97, 0x98, 0x1e, 0xc2, 0xff, + 0x36, 0x08, 0x59, 0xa2, 0x4d, 0x34, 0x64, 0xb1, 0x9b, 0xbf, 0xc2, 0xf4, 0x04, 0x82, 0x6a, 0x2e, + 0x52, 0x9e, 0xb8, 0xab, 0x95, 0x96, 0x9c, 0x8d, 0xc8, 0x25, 0xa6, 0x47, 0x70, 0x50, 0xc5, 0x2a, + 0xe5, 0xec, 0x89, 0x91, 0x3c, 0x66, 0xcf, 0xc8, 0x1b, 0x4c, 0x8f, 0xe1, 0xb0, 0x8a, 0xdc, 0xd0, + 0x3e, 0x4d, 0xf1, 0x78, 0x40, 0xde, 0xb9, 0x50, 0xe9, 0x1b, 0x0b, 0x78, 0x7f, 0x1d, 0xee, 0x36, + 0xf0, 0x01, 0xd3, 0x7b, 0xd0, 0xaa, 0xcc, 0xdd, 0x57, 0xfc, 0x76, 0x0d, 0xdc, 0xdc, 0xc1, 0x77, + 0x6c, 0x3b, 0x70, 0xc0, 0x66, 0x29, 0xc9, 0x07, 0x63, 0xc5, 0x23, 0x72, 0xe9, 0xd9, 0x0e, 0x46, + 0x2c, 0x1e, 0x08, 0x39, 0xe2, 0x91, 0x19, 0x71, 0xa5, 0xd8, 0x19, 0x27, 0x6f, 0xbd, 0xee, 0x29, + 0xf8, 0xf6, 0x0f, 0xa5, 0x75, 0xf0, 0xce, 0x45, 0x4a, 0x10, 0xdd, 0x03, 0xdf, 0xde, 0x40, 0x30, + 0x05, 0xa8, 0x29, 0xcd, 0xf4, 0x58, 0x11, 0xcf, 0x7e, 0xe0, 0x90, 0x25, 0xc6, 0x22, 0xfe, 0xe3, + 0xe6, 0xa7, 0x75, 0x1b, 0x7d, 0x5e, 0xb7, 0xd1, 0x8f, 0x75, 0x1b, 0xfd, 0x0e, 0x00, 0x00, 0xff, + 0xff, 0x6b, 0x22, 0x33, 0xbb, 0x2f, 0x03, 0x00, 0x00, +} From 2e13245085bac046518b2f742765f02f2ac117f0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 Aug 2018 11:48:45 -0700 Subject: [PATCH 0537/3965] reset when we fail in identify --- p2p/protocol/identify/id.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index c548fd3272..fb949851e0 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -92,13 +92,13 @@ func (ids *IDService) IdentifyConn(c inet.Conn) { c.Close() return } - defer inet.FullClose(s) s.SetProtocol(ID) // ok give the response to our handler. if err := msmux.SelectProtoOrFail(ID, s); err != nil { log.Event(context.TODO(), "IdentifyOpenFailed", c.RemotePeer(), logging.Metadata{"error": err}) + s.Reset() return } @@ -111,8 +111,8 @@ func (ids *IDService) IdentifyConn(c inet.Conn) { if !found { log.Errorf("IdentifyConn failed to find channel (programmer error) for %s", c) - return } + inet.FullClose(s) } func (ids *IDService) requestHandler(s inet.Stream) { From f56473fb013ededdc63021ca80467be784e45358 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 Aug 2018 13:36:01 -0700 Subject: [PATCH 0538/3965] make sure reset works on half-closed streams --- p2p/net/mock/mock_stream.go | 73 ++++++++++++++++++++++++------------- 1 file changed, 47 insertions(+), 26 deletions(-) diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index 79a3834c0a..6d056c8ea0 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -22,7 +22,7 @@ type stream struct { close chan struct{} closed chan struct{} - state error + writeErr error protocol protocol.ID } @@ -56,7 +56,7 @@ func (s *stream) Write(p []byte) (n int, err error) { t := time.Now().Add(delay) select { case <-s.closed: // bail out if we're closing. - return 0, s.state + return 0, s.writeErr case s.toDeliver <- &transportObject{msg: p, arrivalTime: t}: } return len(p), nil @@ -76,30 +76,28 @@ func (s *stream) Close() error { default: } <-s.closed - if s.state != ErrClosed { - return s.state + if s.writeErr != ErrClosed { + return s.writeErr } return nil } func (s *stream) Reset() error { - // Cancel any pending writes. - s.write.Close() + // Cancel any pending reads/writes with an error. + s.write.CloseWithError(ErrReset) + s.read.CloseWithError(ErrReset) select { case s.reset <- struct{}{}: default: } <-s.closed - if s.state != ErrReset { - return s.state - } + + // No meaningful error case here. return nil } func (s *stream) teardown() { - s.write.Close() - // at this point, no streams are writing. s.conn.removeStream(s) @@ -151,20 +149,21 @@ func (s *stream) transport() { // writeBuf writes the contents of buf through to the s.Writer. // done only when arrival time makes sense. - drainBuf := func() { + drainBuf := func() error { if buf.Len() > 0 { _, err := s.write.Write(buf.Bytes()) if err != nil { - return + return err } buf.Reset() } + return nil } // deliverOrWait is a helper func that processes // an incoming packet. it waits until the arrival time, // and then writes things out. - deliverOrWait := func(o *transportObject) { + deliverOrWait := func(o *transportObject) error { buffered := len(o.msg) + buf.Len() // Yes, we can end up extending a timer multiple times if we @@ -189,43 +188,65 @@ func (s *stream) transport() { select { case <-timer.C: case <-s.reset: - s.reset <- struct{}{} - return + select { + case s.reset <- struct{}{}: + default: + } + return ErrReset + } + if err := drainBuf(); err != nil { + return err } - drainBuf() // write this message. _, err := s.write.Write(o.msg) if err != nil { - log.Error("mock_stream", err) + return err } } else { buf.Write(o.msg) } + return nil } for { // Reset takes precedent. select { case <-s.reset: - s.state = ErrReset - s.read.CloseWithError(ErrReset) + s.writeErr = ErrReset return default: } select { case <-s.reset: - s.state = ErrReset - s.read.CloseWithError(ErrReset) + s.writeErr = ErrReset return case <-s.close: - s.state = ErrClosed - drainBuf() + if err := drainBuf(); err != nil { + s.resetWith(err) + return + } + s.writeErr = s.write.Close() + if s.writeErr == nil { + s.writeErr = ErrClosed + } return case o := <-s.toDeliver: - deliverOrWait(o) + if err := deliverOrWait(o); err != nil { + s.resetWith(err) + return + } case <-timer.C: // ok, due to write it out. - drainBuf() + if err := drainBuf(); err != nil { + s.resetWith(err) + return + } } } } + +func (s *stream) resetWith(err error) { + s.write.CloseWithError(err) + s.read.CloseWithError(err) + s.writeErr = err +} From 784ff9c5a540c42de696c9cc0472b71662cea2f2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 Aug 2018 13:37:29 -0700 Subject: [PATCH 0539/3965] fix closing of mock streams in test --- p2p/net/mock/mock_notif_test.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/mock/mock_notif_test.go b/p2p/net/mock/mock_notif_test.go index 3521d6f869..13758ff756 100644 --- a/p2p/net/mock/mock_notif_test.go +++ b/p2p/net/mock/mock_notif_test.go @@ -114,7 +114,7 @@ func TestNotifications(t *testing.T) { for _, s := range nets { s.SetStreamHandler(func(s inet.Stream) { - s.Close() + inet.FullClose(s) }) } @@ -131,7 +131,7 @@ func TestNotifications(t *testing.T) { for _, s := range nets { s.SetStreamHandler(func(s inet.Stream) { streams <- s - s.Close() + inet.FullClose(s) }) } @@ -146,7 +146,7 @@ func TestNotifications(t *testing.T) { } else { t.Logf("%s %s <--%p--> %s %s", c.LocalPeer(), c.LocalMultiaddr(), st1, c.RemotePeer(), c.RemoteMultiaddr()) // st1.Write([]byte("hello")) - st1.Close() + go inet.FullClose(st1) st2 := <-streams t.Logf("%s %s <--%p--> %s %s", c2.LocalPeer(), c2.LocalMultiaddr(), st2, c2.RemotePeer(), c2.RemoteMultiaddr()) testOCStream(notifiees[i], st1) From f6ad991bb5a8c2e9b9115b3701005bc696edcb61 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 Aug 2018 11:04:14 -0700 Subject: [PATCH 0540/3965] update gogo protobuf and regenerate protobuf files --- p2p/protocol/identify/pb/Makefile | 6 +- p2p/protocol/identify/pb/identify.pb.go | 538 +++++++++++++++++++++++- p2p/protocol/identify/pb/identify.proto | 2 + package.json | 10 +- 4 files changed, 535 insertions(+), 21 deletions(-) diff --git a/p2p/protocol/identify/pb/Makefile b/p2p/protocol/identify/pb/Makefile index d08f1c3ebb..eb14b5768a 100644 --- a/p2p/protocol/identify/pb/Makefile +++ b/p2p/protocol/identify/pb/Makefile @@ -1,11 +1,11 @@ - PB = $(wildcard *.proto) GO = $(PB:.proto=.pb.go) all: $(GO) %.pb.go: %.proto - protoc --gogo_out=. --proto_path=../../../../../../:/usr/local/opt/protobuf/include:. $< + protoc --proto_path=$(GOPATH)/src:. --gogofast_out=. $< clean: - rm *.pb.go + rm -f *.pb.go + rm -f *.go diff --git a/p2p/protocol/identify/pb/identify.pb.go b/p2p/protocol/identify/pb/identify.pb.go index cc4cba4975..774ab2b89c 100644 --- a/p2p/protocol/identify/pb/identify.pb.go +++ b/p2p/protocol/identify/pb/identify.pb.go @@ -1,25 +1,25 @@ -// Code generated by protoc-gen-gogo. +// Code generated by protoc-gen-gogo. DO NOT EDIT. // source: identify.proto -// DO NOT EDIT! -/* -Package identify_pb is a generated protocol buffer package. - -It is generated from these files: - identify.proto - -It has these top-level messages: - Identify -*/ package identify_pb import proto "github.com/gogo/protobuf/proto" +import fmt "fmt" import math "math" +import io "io" + // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal +var _ = fmt.Errorf var _ = math.Inf +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + type Identify struct { // protocolVersion determines compatibility between peers ProtocolVersion *string `protobuf:"bytes,5,opt,name=protocolVersion" json:"protocolVersion,omitempty"` @@ -37,13 +37,44 @@ type Identify struct { // determine whether its connection to the local peer goes through NAT. ObservedAddr []byte `protobuf:"bytes,4,opt,name=observedAddr" json:"observedAddr,omitempty"` // protocols are the services this node is running - Protocols []string `protobuf:"bytes,3,rep,name=protocols" json:"protocols,omitempty"` - XXX_unrecognized []byte `json:"-"` + Protocols []string `protobuf:"bytes,3,rep,name=protocols" json:"protocols,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *Identify) Reset() { *m = Identify{} } func (m *Identify) String() string { return proto.CompactTextString(m) } func (*Identify) ProtoMessage() {} +func (*Identify) Descriptor() ([]byte, []int) { + return fileDescriptor_identify_daaec8baf46eae80, []int{0} +} +func (m *Identify) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Identify) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Identify.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *Identify) XXX_Merge(src proto.Message) { + xxx_messageInfo_Identify.Merge(dst, src) +} +func (m *Identify) XXX_Size() int { + return m.Size() +} +func (m *Identify) XXX_DiscardUnknown() { + xxx_messageInfo_Identify.DiscardUnknown(m) +} + +var xxx_messageInfo_Identify proto.InternalMessageInfo func (m *Identify) GetProtocolVersion() string { if m != nil && m.ProtocolVersion != nil { @@ -88,4 +119,485 @@ func (m *Identify) GetProtocols() []string { } func init() { + proto.RegisterType((*Identify)(nil), "identify.pb.Identify") +} +func (m *Identify) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Identify) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.PublicKey != nil { + dAtA[i] = 0xa + i++ + i = encodeVarintIdentify(dAtA, i, uint64(len(m.PublicKey))) + i += copy(dAtA[i:], m.PublicKey) + } + if len(m.ListenAddrs) > 0 { + for _, b := range m.ListenAddrs { + dAtA[i] = 0x12 + i++ + i = encodeVarintIdentify(dAtA, i, uint64(len(b))) + i += copy(dAtA[i:], b) + } + } + if len(m.Protocols) > 0 { + for _, s := range m.Protocols { + dAtA[i] = 0x1a + i++ + l = len(s) + for l >= 1<<7 { + dAtA[i] = uint8(uint64(l)&0x7f | 0x80) + l >>= 7 + i++ + } + dAtA[i] = uint8(l) + i++ + i += copy(dAtA[i:], s) + } + } + if m.ObservedAddr != nil { + dAtA[i] = 0x22 + i++ + i = encodeVarintIdentify(dAtA, i, uint64(len(m.ObservedAddr))) + i += copy(dAtA[i:], m.ObservedAddr) + } + if m.ProtocolVersion != nil { + dAtA[i] = 0x2a + i++ + i = encodeVarintIdentify(dAtA, i, uint64(len(*m.ProtocolVersion))) + i += copy(dAtA[i:], *m.ProtocolVersion) + } + if m.AgentVersion != nil { + dAtA[i] = 0x32 + i++ + i = encodeVarintIdentify(dAtA, i, uint64(len(*m.AgentVersion))) + i += copy(dAtA[i:], *m.AgentVersion) + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func encodeVarintIdentify(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func (m *Identify) Size() (n int) { + var l int + _ = l + if m.PublicKey != nil { + l = len(m.PublicKey) + n += 1 + l + sovIdentify(uint64(l)) + } + if len(m.ListenAddrs) > 0 { + for _, b := range m.ListenAddrs { + l = len(b) + n += 1 + l + sovIdentify(uint64(l)) + } + } + if len(m.Protocols) > 0 { + for _, s := range m.Protocols { + l = len(s) + n += 1 + l + sovIdentify(uint64(l)) + } + } + if m.ObservedAddr != nil { + l = len(m.ObservedAddr) + n += 1 + l + sovIdentify(uint64(l)) + } + if m.ProtocolVersion != nil { + l = len(*m.ProtocolVersion) + n += 1 + l + sovIdentify(uint64(l)) + } + if m.AgentVersion != nil { + l = len(*m.AgentVersion) + n += 1 + l + sovIdentify(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func sovIdentify(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozIdentify(x uint64) (n int) { + return sovIdentify(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *Identify) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Identify: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Identify: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field PublicKey", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.PublicKey = append(m.PublicKey[:0], dAtA[iNdEx:postIndex]...) + if m.PublicKey == nil { + m.PublicKey = []byte{} + } + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field ListenAddrs", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.ListenAddrs = append(m.ListenAddrs, make([]byte, postIndex-iNdEx)) + copy(m.ListenAddrs[len(m.ListenAddrs)-1], dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Protocols", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Protocols = append(m.Protocols, string(dAtA[iNdEx:postIndex])) + iNdEx = postIndex + case 4: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field ObservedAddr", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.ObservedAddr = append(m.ObservedAddr[:0], dAtA[iNdEx:postIndex]...) + if m.ObservedAddr == nil { + m.ObservedAddr = []byte{} + } + iNdEx = postIndex + case 5: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field ProtocolVersion", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + s := string(dAtA[iNdEx:postIndex]) + m.ProtocolVersion = &s + iNdEx = postIndex + case 6: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field AgentVersion", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + s := string(dAtA[iNdEx:postIndex]) + m.AgentVersion = &s + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipIdentify(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthIdentify + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipIdentify(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowIdentify + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowIdentify + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowIdentify + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + iNdEx += length + if length < 0 { + return 0, ErrInvalidLengthIdentify + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowIdentify + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipIdentify(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthIdentify = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowIdentify = fmt.Errorf("proto: integer overflow") +) + +func init() { proto.RegisterFile("identify.proto", fileDescriptor_identify_daaec8baf46eae80) } + +var fileDescriptor_identify_daaec8baf46eae80 = []byte{ + // 187 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0xcb, 0x4c, 0x49, 0xcd, + 0x2b, 0xc9, 0x4c, 0xab, 0xd4, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x46, 0xf0, 0x93, 0x94, + 0x6e, 0x31, 0x72, 0x71, 0x78, 0x42, 0xf9, 0x42, 0x32, 0x5c, 0x9c, 0x05, 0xa5, 0x49, 0x39, 0x99, + 0xc9, 0xde, 0xa9, 0x95, 0x12, 0x8c, 0x0a, 0x8c, 0x1a, 0x3c, 0x41, 0x08, 0x01, 0x21, 0x05, 0x2e, + 0xee, 0x9c, 0xcc, 0xe2, 0x92, 0xd4, 0x3c, 0xc7, 0x94, 0x94, 0xa2, 0x62, 0x09, 0x26, 0x05, 0x66, + 0x0d, 0x9e, 0x20, 0x64, 0x21, 0xb0, 0x7e, 0x90, 0x15, 0xc9, 0xf9, 0x39, 0xc5, 0x12, 0xcc, 0x0a, + 0xcc, 0x1a, 0x9c, 0x41, 0x08, 0x01, 0x21, 0x25, 0x2e, 0x9e, 0xfc, 0xa4, 0xe2, 0xd4, 0xa2, 0xb2, + 0xd4, 0x14, 0x90, 0x72, 0x09, 0x16, 0xb0, 0x05, 0x28, 0x62, 0x42, 0x1a, 0x5c, 0xfc, 0x30, 0x0d, + 0x61, 0xa9, 0x45, 0xc5, 0x99, 0xf9, 0x79, 0x12, 0xac, 0x0a, 0x8c, 0x1a, 0x9c, 0x41, 0xe8, 0xc2, + 0x20, 0xd3, 0x12, 0xd3, 0x53, 0xf3, 0x4a, 0x60, 0xca, 0xd8, 0xc0, 0xca, 0x50, 0xc4, 0x9c, 0x78, + 0x4e, 0x3c, 0x92, 0x63, 0xbc, 0xf0, 0x48, 0x8e, 0xf1, 0xc1, 0x23, 0x39, 0x46, 0x40, 0x00, 0x00, + 0x00, 0xff, 0xff, 0x4b, 0x9c, 0x90, 0x7a, 0x08, 0x01, 0x00, 0x00, } diff --git a/p2p/protocol/identify/pb/identify.proto b/p2p/protocol/identify/pb/identify.proto index 7d31e0474a..5270c4cf58 100644 --- a/p2p/protocol/identify/pb/identify.proto +++ b/p2p/protocol/identify/pb/identify.proto @@ -1,3 +1,5 @@ +syntax = "proto2"; + package identify.pb; message Identify { diff --git a/package.json b/package.json index 367fa5491e..e553adf472 100644 --- a/package.json +++ b/package.json @@ -48,11 +48,6 @@ "name": "go-multiaddr-net", "version": "1.6.3" }, - { - "hash": "QmZ4Qi3GaRbjcx28Sme5eMH7RQjGkt8wHxt2a65oLaeFEV", - "name": "gogo-protobuf", - "version": "0.0.0" - }, { "hash": "QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7", "name": "go-multiaddr", @@ -231,6 +226,11 @@ "hash": "Qma7AuxEA7dd1wAy95hTxXgxy4q7mU4Pyd1x4PRAzGP1fs", "name": "go-libp2p-transport-upgrader", "version": "0.1.5" + }, + { + "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", + "name": "gogo-protobuf", + "version": "0.0.0" } ], "gxVersion": "0.4.0", From 8cdaeaa1620f6f5b58ef0aedc50f2cf9792fa829 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 Aug 2018 18:09:22 -0700 Subject: [PATCH 0541/3965] gx publish 6.0.7 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index e964909df0..61e7743ad3 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.6: QmY51bqSM5XgxQZqsBrQcRkKTnCb8EKpJpR9K6Qax7Njco +6.0.7: QmeLdnnwEHw4foJGfPij1sHYHmJRgRd9TPVPZsvLGXhbzw diff --git a/package.json b/package.json index e553adf472..f5d8591e71 100644 --- a/package.json +++ b/package.json @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.6" + "version": "6.0.7" } From cc70c4be36f765ba9fb92d6e0528722773af607d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 Aug 2018 18:50:40 -0700 Subject: [PATCH 0542/3965] gx publish 6.0.8 --- .gx/lastpubver | 2 +- package.json | 94 +++++++++++++++++++++++++------------------------- 2 files changed, 48 insertions(+), 48 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 61e7743ad3..9c3db95ba2 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.7: QmeLdnnwEHw4foJGfPij1sHYHmJRgRd9TPVPZsvLGXhbzw +6.0.8: QmUDzeFgYrRmHL2hUB6NZmqcBVQtUzETwmFRUc9onfSSHr diff --git a/package.json b/package.json index f5d8591e71..326b87d1ef 100644 --- a/package.json +++ b/package.json @@ -39,9 +39,9 @@ "version": "1.0.0" }, { - "hash": "QmcVVHfdyv15GVPk7NrxdWjh2hLVccXnoD8j2tyQShiXJb", + "hash": "QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr", "name": "go-log", - "version": "1.5.3" + "version": "1.5.5" }, { "hash": "QmV6FjemM1K8oXjrvuq3wuVWWoU2TLDPmNnKrxHzY3v6Ai", @@ -55,33 +55,33 @@ }, { "author": "whyrusleeping", - "hash": "QmRPkGkHLB72caXgdDYnoaWigXNWx95BcYDKV1n3KTEpaG", + "hash": "QmcEC2rbyMxUMgpLwt16wquaZdG1aPXcpbKYf4Fedt7hkD", "name": "go-libp2p-loggables", - "version": "1.1.18" + "version": "1.1.20" }, { "author": "whyrusleeping", - "hash": "QmXzYnwYnJiDDV9y4QmbmtSeVyJ1R8HAmGnrnY2D6G3YcV", + "hash": "QmZ69yKFZ7V5hLCYFWzJF1Nz4dH1Es4LaE8jGXAFucLe3K", "name": "go-libp2p-secio", - "version": "2.0.4" + "version": "2.0.6" }, { "author": "whyrusleeping", - "hash": "QmZR2XWVVBCtbgBWnQhWk2xcQfaR3W8faQPriAiaaj7rsr", + "hash": "QmYLXCWN2myozZpx8Wx4UjrRuQuhY3YtWoMi6SHaXii6aM", "name": "go-libp2p-peerstore", - "version": "1.4.22" + "version": "1.4.23" }, { "author": "whyrusleeping", - "hash": "QmW5LxJm2Yo3S7uVsfLM7NsJn2QnKbvZD7uYsZVYR7YViE", + "hash": "QmUMTtHxeyVJPrpcpvEQppH3uTf3g1NnkRC6C36LpXy2no", "name": "go-libp2p-transport", - "version": "3.0.5" + "version": "3.0.6" }, { "author": "whyrusleeping", - "hash": "QmSggepaKhRfgEDiz2B6eGaCTK3TqDHXzK6F6HfsU7KPXt", + "hash": "QmSYxVNTGJ4MEy7UH26qgLbidfSgdYiEkV6affSitghk4S", "name": "go-tcp-transport", - "version": "2.0.5" + "version": "2.0.6" }, { "author": "whyrusleeping", @@ -97,57 +97,57 @@ }, { "author": "whyrusleeping", - "hash": "QmcW4FGAt24fdK1jBgWQn3yP4R9ZLyWQqjozv9QK7epRhL", + "hash": "QmXG74iiKQnDstVQq9fPFQEB6JTNSWBbAWE1qsq6L4E5sR", "name": "go-testutil", - "version": "1.2.5" + "version": "1.2.6" }, { "author": "whyrusleeping", - "hash": "QmPjvxTpVH8qJyQDnxnsxF9kv9jezKD1kozz1hs3fCGsNh", + "hash": "QmVwU7Mgwg6qaPn9XXz93ANfq1PTxcduGRzfe41Sygg4mR", "name": "go-libp2p-net", - "version": "3.0.5" + "version": "3.0.6" }, { "author": "whyrusleeping", - "hash": "QmcoBbyTiL9PFjo1GFixJwqQ8mZLJ36CribuqyKmS1okPu", + "hash": "QmWne2EKHBvVpSTYuWuWch3D9KqAx78Te83UXWFKQDcksJ", "name": "go-libp2p-metrics", - "version": "2.1.3" + "version": "2.1.4" }, { "author": "whyrusleeping", - "hash": "Qmb8T6YBBsjYsVGfrihQLfCJveczZnneSBqBKkYEBWDjge", + "hash": "QmQ1hwb95uSSZR8jSPJysnfHxBDQAykSXsmz5TwTzxjq2Z", "name": "go-libp2p-host", - "version": "3.0.4" + "version": "3.0.5" }, { "author": "whyrusleeping", - "hash": "QmemVjhp1UuWPQqrWSvPcaqH3QJRMjMqNm4T2RULMkDDQe", + "hash": "QmdjC8HtKZpEufBL1u7WxvQn78Lqq2Wk31NJS8WvFX3crB", "name": "go-libp2p-swarm", - "version": "3.0.6" + "version": "3.0.7" }, { "author": "whyrusleeping", - "hash": "QmW25V77H8nb1ucUe8BDUbR3QXLNzTearBLYMW5DqDVWd5", + "hash": "QmScWrjfrMXw8AuBjtdDgdt97SuRhvf4KfR4y8djfNXCKT", "name": "go-libp2p-nat", - "version": "0.8.5" + "version": "0.8.6" }, { "author": "whyrusleeping", - "hash": "QmcxUtMB5sJrXR3znSvkrDd2ghvwGM8rLRqwJiPUdgQwat", + "hash": "QmVkii6Urke4akPoToq96XsFB8AuY81MzHKYE8DfJvDJq9", "name": "go-libp2p-netutil", - "version": "0.4.4" + "version": "0.4.5" }, { "author": "whyrusleeping", - "hash": "QmT1Q44YeQXSiK3LBXbXBRdxJAFZZSTarseqt7DVe9ASNA", + "hash": "QmVdU678uqPAfGgrrrn7zPz2yttmZTYbowF56TeNFecWzW", "name": "go-libp2p-blankhost", - "version": "0.3.4" + "version": "0.3.5" }, { "author": "whyrusleeping", - "hash": "Qme1knMqwt1hKZbc1BmQFmnm9f36nyQGwXxPGVpVJ9rMK5", + "hash": "QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n", "name": "go-libp2p-crypto", - "version": "1.6.2" + "version": "2.0.1" }, { "author": "whyrusleeping", @@ -163,15 +163,15 @@ }, { "author": "whyrusleeping", - "hash": "QmdVrMn1LhB4ybb8hMVaMLXnA8XRSewMnK6YqXKXoTcRvN", + "hash": "QmcZSzKEM5yDfpZbeEEZaVmaZ1zXm6JWTbrQZSB8hCVPzk", "name": "go-libp2p-peer", - "version": "2.3.5" + "version": "2.3.6" }, { "author": "vyzo", - "hash": "QmcQ56iqKP8ZRhRGLe5EReJVvrJZDaGzkuatrPv4Z1B6cG", + "hash": "QmPMRK5yTc2KhnaxQN4R7vRqEfZo5hW1aF5x6W97RKnXZq", "name": "go-libp2p-circuit", - "version": "2.1.6" + "version": "2.1.8" }, { "author": "lgierth", @@ -181,33 +181,33 @@ }, { "author": "why", - "hash": "QmXuucFcuvAWYAJfhHV2h4BYreHEAsLSsiquosiXeuduTN", + "hash": "QmeJbAMK4cZc1RMChb68h9t2jqvK8miqE8oQiwGAf4EdQq", "name": "go-libp2p-interface-connmgr", - "version": "0.0.11" + "version": "0.0.12" }, { "author": "whyrusleeping", - "hash": "QmZHiqdRuNXujvSPNu1ZWxxzV6a2WhoZpfYkesdgyaKF9f", + "hash": "QmdiBZzwGtN2yHJrWD9ojQ7ASS48nv7BcojWLkYd1ZtrV2", "name": "go-smux-multiplex", - "version": "3.0.11" + "version": "3.0.12" }, { "author": "whyrusleeping", - "hash": "QmX4UrmHGPnFxwfsunZjPbykzyv8Frg9AVmNariXqrLsMs", + "hash": "QmPUS1GXLAqz3aaeiaBuqusLWexaKVfJ9DW5ArEDF4VEUM", "name": "go-ws-transport", - "version": "2.0.5" + "version": "2.0.6" }, { "author": "stebalien", - "hash": "QmZaZ4nkadScQtG7KaGUWyh24jz5tD8d6AyetJmGnMo49t", + "hash": "Qmahr4UDXGj8zXFhXzQZpmoJgeVi6XX8pEnuVi2bFkENqj", "name": "go-conn-security-multistream", - "version": "0.1.4" + "version": "0.1.5" }, { "author": "Stebalien", - "hash": "QmYMGCwtKbLXfh2MUTbLoP66N9Y24DSg8PFJRG21eE2qwv", + "hash": "QmaKSpLZuCobQa8tfcKkZYdabfTPuihz113WM7RT9moeVS", "name": "go-conn-security", - "version": "0.1.5" + "version": "0.1.6" }, { "author": "libp2p", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "Qma7AuxEA7dd1wAy95hTxXgxy4q7mU4Pyd1x4PRAzGP1fs", + "hash": "QmP7znopdZogwxPJyRKEZSNnP7HfnUCaQjaMNDmPw8VE2Y", "name": "go-libp2p-transport-upgrader", - "version": "0.1.5" + "version": "0.1.6" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.7" + "version": "6.0.8" } From c3a5e869081a1852593e7745f82595f4c2b1e21a Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 11 Aug 2018 11:31:00 +0700 Subject: [PATCH 0543/3965] allow 1000 bidirectional streams, disable unidirectional streams --- p2p/transport/quic/transport.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 2fb16bec11..d13300fa71 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -19,6 +19,8 @@ import ( ) var quicConfig = &quic.Config{ + MaxIncomingStreams: 1000, + MaxIncomingUniStreams: -1, // disable unidirectional streams MaxReceiveStreamFlowControlWindow: 3 * (1 << 20), // 3 MB MaxReceiveConnectionFlowControlWindow: 4.5 * (1 << 20), // 4.5 MB Versions: []quic.VersionNumber{101}, From 163d7d40dd00bf4c275a164d60fc55e3a34a6073 Mon Sep 17 00:00:00 2001 From: David Dias Date: Sat, 11 Aug 2018 13:26:00 +0200 Subject: [PATCH 0544/3965] docs: add url to weekly dev syncs --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index a3864fb8ee..0fa5b21858 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,8 @@ [![Throughput Graph](https://graphs.waffle.io/libp2p/go-libp2p/throughput.svg)](https://waffle.io/libp2p/go-libp2p/metrics/throughput) +[**`Weekly Core Dev Calls`**](https://github.com/ipfs/pm/issues/674) + # Table of Contents - [Background](#background) From c83318bf08cb8651ebc2ac36483325ccda14b84c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 13 Aug 2018 16:52:49 -0700 Subject: [PATCH 0545/3965] Revert "Merge pull request #386 from libp2p/fix/correct-external-addr" This reverts commit aca83b9b6302000167c95dfb96eb4812cf1c0e93, reversing changes made to 86b8929d4ddf3d38d4ee4cca6651abb9d05f16da. This was is not the correct fix. We already expose these addresses via the host's AllAddrs method. The real problem is probably that we just don't ever tell anyone about them (unless we disconnect and reconnect to our nearby DHT nodes). We need an address gossip protocol. --- p2p/host/basic/natmgr.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/p2p/host/basic/natmgr.go b/p2p/host/basic/natmgr.go index be4837503e..349f79e789 100644 --- a/p2p/host/basic/natmgr.go +++ b/p2p/host/basic/natmgr.go @@ -8,7 +8,6 @@ import ( lgbl "github.com/libp2p/go-libp2p-loggables" inat "github.com/libp2p/go-libp2p-nat" inet "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) @@ -188,9 +187,6 @@ func addPortMapping(nmgr *natManager, intaddr ma.Multiaddr) { return } - // TODO: make these temporary and rediscover them. - nmgr.net.Peerstore().AddAddr(nmgr.net.LocalPeer(), extaddr, pstore.PermanentAddrTTL) - lm["outcome"] = "success" lm["externalAddr"] = func() interface{} { return extaddr.String() } log.Infof("established nat port mapping: %s <--> %s", intaddr, extaddr) From ed85445dc3ea856f2fc15f53363f1e6aa8542b68 Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Wed, 15 Aug 2018 18:36:57 +0800 Subject: [PATCH 0546/3965] Fix a typo in doc --- libp2p.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libp2p.go b/libp2p.go index 7d0621c94c..ad026ad4bd 100644 --- a/libp2p.go +++ b/libp2p.go @@ -33,7 +33,7 @@ func ChainOptions(opts ...Option) Option { // - If no transport and listen addresses are provided, the node listens to // the multiaddresses "/ip4/0.0.0.0/tcp/0" and "/ip6/::/tcp/0"; // -// - If no transport options are provided, the node uses TPC and websocket +// - If no transport options are provided, the node uses TCP and websocket // transport protocols; // // - If no multiplexer configuration is provided, the node is configured by From 005db422e6ef5b57ed60b0ca884c0e82fe106453 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 16 Aug 2018 11:29:43 +0700 Subject: [PATCH 0547/3965] update quic-go to v0.9.0, use the quic-go milestone version --- p2p/transport/quic/conn.go | 2 +- p2p/transport/quic/listener.go | 2 +- p2p/transport/quic/transport.go | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index fef587395f..ed837835cb 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -25,7 +25,7 @@ type conn struct { var _ tpt.Conn = &conn{} func (c *conn) Close() error { - return c.sess.Close(nil) + return c.sess.Close() } // IsClosed returns whether a connection is fully closed. diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 1cdc79e208..ef8b019048 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -65,7 +65,7 @@ func (l *listener) Accept() (tpt.Conn, error) { } conn, err := l.setupConn(sess) if err != nil { - sess.Close(err) + sess.CloseWithError(0, err) continue } return conn, nil diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index d13300fa71..d8c984af27 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -19,11 +19,11 @@ import ( ) var quicConfig = &quic.Config{ + Versions: []quic.VersionNumber{quic.VersionMilestone0_9_0}, MaxIncomingStreams: 1000, MaxIncomingUniStreams: -1, // disable unidirectional streams MaxReceiveStreamFlowControlWindow: 3 * (1 << 20), // 3 MB MaxReceiveConnectionFlowControlWindow: 4.5 * (1 << 20), // 4.5 MB - Versions: []quic.VersionNumber{101}, AcceptCookie: func(clientAddr net.Addr, cookie *quic.Cookie) bool { // TODO(#6): require source address validation when under load return true From 5a9655ca4de3d57f8cad3a7da35dee6413bb6b18 Mon Sep 17 00:00:00 2001 From: Kishan Sagathiya Date: Thu, 16 Aug 2018 18:12:11 +0530 Subject: [PATCH 0548/3965] Issue #370 Check if empty peer ID On trying to open a new stream against an empty peer ID, we get an error saying that failed to dial a peer. This commit changes that to a more informative error message: `empty peer ID` --- p2p/net/swarm/swarm_dial.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 59adb037cb..2af26ed1a2 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -183,6 +183,11 @@ func (s *Swarm) DialPeer(ctx context.Context, p peer.ID) (inet.Conn, error) { func (s *Swarm) dialPeer(ctx context.Context, p peer.ID) (*Conn, error) { log.Debugf("[%s] swarm dialing peer [%s]", s.local, p) var logdial = lgbl.Dial("swarm", s.LocalPeer(), p, nil, nil) + err := p.Validate() + if err != nil { + return nil, err + } + if p == s.local { log.Event(ctx, "swarmDialSelf", logdial) return nil, ErrDialToSelf @@ -206,7 +211,7 @@ func (s *Swarm) dialPeer(ctx context.Context, p peer.ID) (*Conn, error) { ctx, cancel := context.WithTimeout(ctx, inet.GetDialPeerTimeout(ctx)) defer cancel() - conn, err := s.dsync.DialLock(ctx, p) + conn, err = s.dsync.DialLock(ctx, p) if err != nil { return nil, err } From 0ec0019d06cd4df0744d50770be21a4cdcb0de47 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 16 Aug 2018 15:56:03 -0700 Subject: [PATCH 0549/3965] only map *usable* addresses Otherwise, we'll try to port-map localhost. --- p2p/net/nat/nat.go | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index d68aca0752..0c47a4c2c2 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -3,6 +3,7 @@ package nat import ( "errors" "fmt" + "net" "strconv" "strings" "sync" @@ -125,15 +126,40 @@ func (nat *NAT) NewMapping(maddr ma.Multiaddr) (Mapping, error) { return nil, fmt.Errorf("DialArgs failed on addr: %s", maddr.String()) } + var ip net.IP switch network { case "tcp", "tcp4", "tcp6": + addr, err := net.ResolveTCPAddr(network, addr) + if err != nil { + return nil, err + } + ip = addr.IP network = "tcp" case "udp", "udp4", "udp6": + addr, err := net.ResolveUDPAddr(network, addr) + if err != nil { + return nil, err + } + ip = addr.IP network = "udp" default: return nil, fmt.Errorf("transport not supported by NAT: %s", network) } + // XXX: Known limitation: doesn't handle multiple internal addresses. + // If this applies to you, you can figure it out yourself. Ideally, the + // NAT library would allow us to handle this case but the "go way" + // appears to be to just "shrug" at edge-cases. + if !ip.IsUnspecified() { + internalAddr, err := nat.nat.GetInternalAddress() + if err != nil { + return nil, fmt.Errorf("failed to discover address on nat: %s", err) + } + if !ip.Equal(internalAddr) { + return nil, fmt.Errorf("nat address is %s, refusing to map %s", internalAddr, ip) + } + } + intports := strings.Split(addr, ":")[1] intport, err := strconv.Atoi(intports) if err != nil { From 87255e4ff0ce9b0def941200329b650d172738b3 Mon Sep 17 00:00:00 2001 From: Raghav Gulati Date: Mon, 20 Aug 2018 17:09:53 -0700 Subject: [PATCH 0550/3965] Ensure duplicate transports are filtered Without adding the transport to the set, we'll never identify if there are duplicates in the slice of secure transports provided by the config. --- config/security.go | 1 + 1 file changed, 1 insertion(+) diff --git a/config/security.go b/config/security.go index c3639728ff..0aa6989248 100644 --- a/config/security.go +++ b/config/security.go @@ -61,6 +61,7 @@ func makeSecurityTransport(h host.Host, tpts []MsSecC) (security.Transport, erro if _, ok := transportSet[tptC.ID]; ok { return nil, fmt.Errorf("duplicate security transport: %s", tptC.ID) } + transportSet[tptC.ID] = tptC } for _, tptC := range tpts { tpt, err := tptC.SecC(h) From 8695530d729285f9024d514f8a5bd2c992358864 Mon Sep 17 00:00:00 2001 From: Raghav Gulati Date: Mon, 20 Aug 2018 18:17:30 -0700 Subject: [PATCH 0551/3965] Change value to struct --- config/security.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/security.go b/config/security.go index 0aa6989248..2798fda85f 100644 --- a/config/security.go +++ b/config/security.go @@ -61,7 +61,7 @@ func makeSecurityTransport(h host.Host, tpts []MsSecC) (security.Transport, erro if _, ok := transportSet[tptC.ID]; ok { return nil, fmt.Errorf("duplicate security transport: %s", tptC.ID) } - transportSet[tptC.ID] = tptC + transportSet[tptC.ID] = struct{}{} } for _, tptC := range tpts { tpt, err := tptC.SecC(h) From 3c19cb12b975c08c664941a580466ef95f168243 Mon Sep 17 00:00:00 2001 From: Raghav Gulati Date: Mon, 20 Aug 2018 18:18:37 -0700 Subject: [PATCH 0552/3965] Ensure we filter duplicate transports in muxer Similar to secure transports in config, if we don't add the transport to the set, we'll never identify duplicates. --- config/muxer.go | 1 + 1 file changed, 1 insertion(+) diff --git a/config/muxer.go b/config/muxer.go index 6a48cb3bd4..ee2adeb08a 100644 --- a/config/muxer.go +++ b/config/muxer.go @@ -50,6 +50,7 @@ func makeMuxer(h host.Host, tpts []MsMuxC) (mux.Transport, error) { if _, ok := transportSet[tptC.ID]; ok { return nil, fmt.Errorf("duplicate muxer transport: %s", tptC.ID) } + transportSet[tptC.ID] = struct{}{} } for _, tptC := range tpts { tpt, err := tptC.MuxC(h) From 6e72e88461db971db11ca389c72a51e6f3684b3d Mon Sep 17 00:00:00 2001 From: Raghav Gulati Date: Tue, 21 Aug 2018 09:25:52 -0700 Subject: [PATCH 0553/3965] Add test for duplicate tpts in muxer --- config/muxer_test.go | 47 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/config/muxer_test.go b/config/muxer_test.go index 5e7bab41ff..d7f7baeb5d 100644 --- a/config/muxer_test.go +++ b/config/muxer_test.go @@ -1,9 +1,13 @@ package config import ( + "context" "testing" + host "github.com/libp2p/go-libp2p-host" peer "github.com/libp2p/go-libp2p-peer" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" + bhost "github.com/libp2p/go-libp2p/p2p/host/basic" mux "github.com/libp2p/go-stream-muxer" yamux "github.com/whyrusleeping/go-smux-yamux" ) @@ -52,3 +56,46 @@ func TestMuxerBadTypes(t *testing.T) { } } } + +func TestCatchDuplicateTransportsMuxer(t *testing.T) { + ctx := context.Background() + h := bhost.New(swarmt.GenSwarm(t, ctx)) + yamuxMuxer, err := MuxerConstructor(yamux.DefaultTransport) + if err != nil { + t.Fatal(err) + } + + var tests = map[string]struct { + h host.Host + transports []MsMuxC + expectedError string + }{ + "no duplicate transports": { + h: h, + transports: []MsMuxC{MsMuxC{yamuxMuxer, "yamux"}}, + expectedError: "", + }, + "duplicate transports": { + h: h, + transports: []MsMuxC{ + MsMuxC{yamuxMuxer, "yamux"}, + MsMuxC{yamuxMuxer, "yamux"}, + }, + expectedError: "duplicate muxer transport: yamux", + }, + } + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + _, err = makeMuxer(test.h, test.transports) + if err != nil { + if err.Error() != test.expectedError { + t.Errorf( + "\nexpected: [%v]\nactual: [%v]\n", + test.expectedError, + err, + ) + } + } + }) + } +} From ad5b7b9ba65d7e9509b3f52f2697bdf03b9fcb3e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 16 Aug 2018 11:16:02 -0700 Subject: [PATCH 0554/3965] add options for disabling the default listen address/transports So, in go-ipfs at least, we *don't* pass the listen addresses into the libp2p constructor. Instead, we specify them later. This option allows us to tell libp2p to not listen by default. --- options.go | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/options.go b/options.go index a17fc02e89..ee147fa972 100644 --- a/options.go +++ b/options.go @@ -241,3 +241,21 @@ func NATManager(nm config.NATManagerC) Option { return nil } } + +// NoListen will configure libp2p to not listen by default. +// +// This will both clear any configured listen addrs and prevent libp2p from +// applying the default listen address option. +var NoListen = func(cfg *Config) error { + cfg.ListenAddrs = []ma.Multiaddr{} + return nil +} + +// NoTransports will configure libp2p to not enable any transports. +// +// This will both clear any configured transports (specified in prior libp2p +// options) and prevent libp2p from applying the default transports. +var NoTransports = func(cfg *Config) error { + cfg.Transports = []config.TptC{} + return nil +} From a36d3834139ee632285ff745321010a97441be37 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 16 Aug 2018 13:20:17 -0700 Subject: [PATCH 0555/3965] add tests for NoTransports and NoListen --- libp2p_test.go | 36 ++++++++++++++++++++++++++++++++++++ options.go | 4 ++-- 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/libp2p_test.go b/libp2p_test.go index 293e0fef7c..0685f4b6ef 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -9,6 +9,7 @@ import ( crypto "github.com/libp2p/go-libp2p-crypto" host "github.com/libp2p/go-libp2p-host" + pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-tcp-transport" ) @@ -32,6 +33,41 @@ func TestBadTransportConstructor(t *testing.T) { } } +func TestNoListenAddrs(t *testing.T) { + ctx := context.Background() + h, err := New(ctx, NoListenAddrs) + if err != nil { + t.Fatal(err) + } + defer h.Close() + if len(h.Addrs()) != 0 { + t.Fatal("expected no addresses") + } +} + +func TestNoTransports(t *testing.T) { + ctx := context.Background() + a, err := New(ctx, NoTransports) + if err != nil { + t.Fatal(err) + } + defer a.Close() + + b, err := New(ctx, ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) + if err != nil { + t.Fatal(err) + } + defer b.Close() + + err = a.Connect(ctx, pstore.PeerInfo{ + ID: b.ID(), + Addrs: b.Addrs(), + }) + if err == nil { + t.Error("dial should have failed as no transports have been configured") + } +} + func TestInsecure(t *testing.T) { ctx := context.Background() h, err := New(ctx, NoSecurity) diff --git a/options.go b/options.go index ee147fa972..ed8e89cfd5 100644 --- a/options.go +++ b/options.go @@ -242,11 +242,11 @@ func NATManager(nm config.NATManagerC) Option { } } -// NoListen will configure libp2p to not listen by default. +// NoListenAddrs will configure libp2p to not listen by default. // // This will both clear any configured listen addrs and prevent libp2p from // applying the default listen address option. -var NoListen = func(cfg *Config) error { +var NoListenAddrs = func(cfg *Config) error { cfg.ListenAddrs = []ma.Multiaddr{} return nil } From ab978fcf9dcf614300807c9f9003ce2fa7d504a3 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 Aug 2018 17:09:12 -0700 Subject: [PATCH 0556/3965] gx publish 6.0.9 --- .gx/lastpubver | 2 +- package.json | 78 +++++++++++++++++++++++++------------------------- 2 files changed, 40 insertions(+), 40 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 9c3db95ba2..16d7d87965 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.8: QmUDzeFgYrRmHL2hUB6NZmqcBVQtUzETwmFRUc9onfSSHr +6.0.9: QmQiaskfWpdRJ4x2spEQjPFTUkEB87KDYu91qnNYBqvvcX diff --git a/package.json b/package.json index 326b87d1ef..3665899df4 100644 --- a/package.json +++ b/package.json @@ -55,33 +55,33 @@ }, { "author": "whyrusleeping", - "hash": "QmcEC2rbyMxUMgpLwt16wquaZdG1aPXcpbKYf4Fedt7hkD", + "hash": "QmZ4zF1mBrt8C2mSCM4ZYE4aAnv78f7GvrzufJC4G5tecK", "name": "go-libp2p-loggables", - "version": "1.1.20" + "version": "1.1.21" }, { "author": "whyrusleeping", - "hash": "QmZ69yKFZ7V5hLCYFWzJF1Nz4dH1Es4LaE8jGXAFucLe3K", + "hash": "QmWKKkNLFRcznF5vDqt2eeRsnQqQhwbjVf8zJ9KC2RXrzN", "name": "go-libp2p-secio", - "version": "2.0.6" + "version": "2.0.7" }, { "author": "whyrusleeping", - "hash": "QmYLXCWN2myozZpx8Wx4UjrRuQuhY3YtWoMi6SHaXii6aM", + "hash": "QmeKD8YT7887Xu6Z86iZmpYNxrLogJexqxEugSmaf14k64", "name": "go-libp2p-peerstore", - "version": "1.4.23" + "version": "1.4.24" }, { "author": "whyrusleeping", - "hash": "QmUMTtHxeyVJPrpcpvEQppH3uTf3g1NnkRC6C36LpXy2no", + "hash": "QmYr9RHifaqHTFZdAsUPLmiMAi2oNeEqA48AFKxXJAsLpJ", "name": "go-libp2p-transport", - "version": "3.0.6" + "version": "3.0.7" }, { "author": "whyrusleeping", - "hash": "QmSYxVNTGJ4MEy7UH26qgLbidfSgdYiEkV6affSitghk4S", + "hash": "QmcK89iqkFV8TqpRUgx1481YZbhjPFnBjqkpBQJfJqmSfm", "name": "go-tcp-transport", - "version": "2.0.6" + "version": "2.0.7" }, { "author": "whyrusleeping", @@ -97,33 +97,33 @@ }, { "author": "whyrusleeping", - "hash": "QmXG74iiKQnDstVQq9fPFQEB6JTNSWBbAWE1qsq6L4E5sR", + "hash": "QmRNhSdqzMcuRxX9A1egBeQ3BhDTguDV5HPwi8wRykkPU8", "name": "go-testutil", - "version": "1.2.6" + "version": "1.2.7" }, { "author": "whyrusleeping", - "hash": "QmVwU7Mgwg6qaPn9XXz93ANfq1PTxcduGRzfe41Sygg4mR", + "hash": "QmX5J1q63BrrDTbpcHifrFbxH3cMZsvaNajy6u3zCpzBXs", "name": "go-libp2p-net", - "version": "3.0.6" + "version": "3.0.7" }, { "author": "whyrusleeping", - "hash": "QmWne2EKHBvVpSTYuWuWch3D9KqAx78Te83UXWFKQDcksJ", + "hash": "QmdhwKw53CTV8EJSAsR1bpmMT5kXiWBgeAyv1EXeeDiXqR", "name": "go-libp2p-metrics", - "version": "2.1.4" + "version": "2.1.5" }, { "author": "whyrusleeping", - "hash": "QmQ1hwb95uSSZR8jSPJysnfHxBDQAykSXsmz5TwTzxjq2Z", + "hash": "QmRRCrNRs4qxotXx7WJT6SpCvSNEhXvyBcVjXY2K71pcjE", "name": "go-libp2p-host", - "version": "3.0.5" + "version": "3.0.6" }, { "author": "whyrusleeping", - "hash": "QmdjC8HtKZpEufBL1u7WxvQn78Lqq2Wk31NJS8WvFX3crB", + "hash": "QmPWNZRUybw3nwJH3mpkrwB97YEQmXRkzvyh34rpJiih6Q", "name": "go-libp2p-swarm", - "version": "3.0.7" + "version": "3.0.8" }, { "author": "whyrusleeping", @@ -133,15 +133,15 @@ }, { "author": "whyrusleeping", - "hash": "QmVkii6Urke4akPoToq96XsFB8AuY81MzHKYE8DfJvDJq9", + "hash": "QmXeofKUbQf6diwT12ZmpmAjW9c47aXT43t7qG1bGTRQTj", "name": "go-libp2p-netutil", - "version": "0.4.5" + "version": "0.4.6" }, { "author": "whyrusleeping", - "hash": "QmVdU678uqPAfGgrrrn7zPz2yttmZTYbowF56TeNFecWzW", + "hash": "QmRAsmNHjzKVuscipvMVjB3NTyLW1HTBiSP8LaKD1iUJmH", "name": "go-libp2p-blankhost", - "version": "0.3.5" + "version": "0.3.6" }, { "author": "whyrusleeping", @@ -163,15 +163,15 @@ }, { "author": "whyrusleeping", - "hash": "QmcZSzKEM5yDfpZbeEEZaVmaZ1zXm6JWTbrQZSB8hCVPzk", + "hash": "QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W", "name": "go-libp2p-peer", - "version": "2.3.6" + "version": "2.3.7" }, { "author": "vyzo", - "hash": "QmPMRK5yTc2KhnaxQN4R7vRqEfZo5hW1aF5x6W97RKnXZq", + "hash": "Qmbc6WjgbkaYhPw5dd6X2RLGiJz854dPPKkyZf52vE2PTS", "name": "go-libp2p-circuit", - "version": "2.1.8" + "version": "2.1.9" }, { "author": "lgierth", @@ -181,9 +181,9 @@ }, { "author": "why", - "hash": "QmeJbAMK4cZc1RMChb68h9t2jqvK8miqE8oQiwGAf4EdQq", + "hash": "QmUPz6FCzCCU7sTY9Sore5NGSUA8YSF2yMkLPjDFq7wGqD", "name": "go-libp2p-interface-connmgr", - "version": "0.0.12" + "version": "0.0.13" }, { "author": "whyrusleeping", @@ -193,21 +193,21 @@ }, { "author": "whyrusleeping", - "hash": "QmPUS1GXLAqz3aaeiaBuqusLWexaKVfJ9DW5ArEDF4VEUM", + "hash": "QmTDv8XeECZaYBZXo5SAEGueMiPTszAkGZQ5YiM5KCAJai", "name": "go-ws-transport", - "version": "2.0.6" + "version": "2.0.7" }, { "author": "stebalien", - "hash": "Qmahr4UDXGj8zXFhXzQZpmoJgeVi6XX8pEnuVi2bFkENqj", + "hash": "Qma6UXLMHjdVFExQZLYqdb5KAesbnoXuthQzovrwRZ64fG", "name": "go-conn-security-multistream", - "version": "0.1.5" + "version": "0.1.6" }, { "author": "Stebalien", - "hash": "QmaKSpLZuCobQa8tfcKkZYdabfTPuihz113WM7RT9moeVS", + "hash": "QmcGgFLHMFLcNMMvxsBC5LeLqubLR5djxjShVU3koVMtVq", "name": "go-conn-security", - "version": "0.1.6" + "version": "0.1.7" }, { "author": "libp2p", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "QmP7znopdZogwxPJyRKEZSNnP7HfnUCaQjaMNDmPw8VE2Y", + "hash": "QmfNvpHX396fhMeauERV6eFnSJg78rUjhjpFf1JvbjxaYM", "name": "go-libp2p-transport-upgrader", - "version": "0.1.6" + "version": "0.1.7" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.8" + "version": "6.0.9" } From 8dc8dd09db1c05ed964e4f2e8596442859347ec5 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 22 Aug 2018 12:17:25 +0100 Subject: [PATCH 0557/3965] feat: use package-table --- README.md | 63 +++++++++++++++++++++++++++++++++++--------- package-list.json | 67 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+), 13 deletions(-) create mode 100644 package-list.json diff --git a/README.md b/README.md index 0fa5b21858..1893a3645c 100644 --- a/README.md +++ b/README.md @@ -109,21 +109,58 @@ $ gx test ./p2p/ ### Packages -> **WIP** - List of packages currently in existence for libp2p: -| Package | Version | CI | -|--------------------|---------|---------------------| -| **Transports** | -| **Connection Upgrades** | -| **Stream Muxers** | -| **Discovery** | -| **Crypto Channels** | -| **Peer Routing** | -| **Content Routing** | -| **Miscellaneous** | -| **Data Types** | +| Package | CI | Coverage | +| ---------|---------|--------- | +| **Libp2p** | +| [`interface-libp2p`](//github.com/libp2p/interface-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/interface-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-libp2p) | +| [`libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | +| **Connection** | +| [`interface-connection`](//github.com/libp2p/interface-connection) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-connection/master)](https://ci.ipfs.team/job/libp2p/job/interface-connection/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-connection/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-connection) | +| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | +| **Transport** | +| [`interface-transport`](//github.com/libp2p/interface-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-transport/master)](https://ci.ipfs.team/job/libp2p/job/interface-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-transport) | +| [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | +| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | +| [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | +| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | +| **Crypto Channels** | +| [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-secio) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | +| **Stream Muxers** | +| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.org/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.org/libp2p/go-stream-muxer) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | +| **Discovery** | +| [`interface-peer-discovery`](//github.com/libp2p/interface-peer-discovery) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-peer-discovery/master)](https://ci.ipfs.team/job/libp2p/job/interface-peer-discovery/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-peer-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-peer-discovery) | +| **NAT Traversal** | +| [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | +| [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | +| **Data Types** | +| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | +| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | +| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | +| **Content Routing** | +| [`interface-content-routing`](//github.com/libp2p/interface-content-routing) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-content-routing/master)](https://ci.ipfs.team/job/libp2p/job/interface-content-routing/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-content-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-content-routing) | +| **Peer Routing** | +| [`interface-peer-routing`](//github.com/libp2p/interface-peer-routing) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-peer-routing/master)](https://ci.ipfs.team/job/libp2p/job/interface-peer-routing/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-peer-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-peer-routing) | +| **Record Store** | +| [`interface-record-store`](//github.com/libp2p/interface-record-store) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-record-store/master)](https://ci.ipfs.team/job/libp2p/job/interface-record-store/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-record-store/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-record-store) | +| **Miscellaneous** | +| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | +| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | +| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-swarm) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | +| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-host) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | +| [`go-libp2p-blankhost`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-blankhost) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | +| [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security-multistream) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | +| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | +| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | +| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | +| **Utilities** | +| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | +| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | +| [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-netutil) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | +| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | + +> This table is generated using the module `package-table` with `package-table --data=package-list.json`. # Contribute diff --git a/package-list.json b/package-list.json new file mode 100644 index 0000000000..f8546b801d --- /dev/null +++ b/package-list.json @@ -0,0 +1,67 @@ +{ + "columns": [ + "Package", + "CI", + "Coverage" + ], + "rows": [ + "Libp2p", + ["libp2p/interface-libp2p", "interface-libp2p"], + ["libp2p/go-libp2p", "libp2p"], + + "Connection", + ["libp2p/interface-connection", "interface-connection"], + ["libp2p/go-libp2p-net", "go-libp2p-net"], + + "Transport", + ["libp2p/interface-transport", "interface-transport"], + ["libp2p/go-ws-transport", "go-ws-transport"], + ["libp2p/go-libp2p-transport", "go-libp2p-transport"], + ["libp2p/go-tcp-transport", "go-tcp-transport"], + ["libp2p/go-libp2p-transport-upgrader", "go-libp2p-transport-upgrader"], + + "Crypto Channels", + ["libp2p/go-libp2p-secio", "go-libp2p-secio"], + + "Stream Muxers", + ["libp2p/go-stream-muxer", "go-stream-muxer"], + + "Discovery", + ["libp2p/interface-peer-discovery", "interface-peer-discovery"], + + "NAT Traversal", + ["libp2p/go-libp2p-circuit", "go-libp2p-circuit"], + ["libp2p/go-libp2p-nat", "go-libp2p-nat"], + + "Data Types", + ["libp2p/go-libp2p-peer", "go-libp2p-peer"], + ["libp2p/go-libp2p-peerstore", "go-libp2p-peerstore"], + ["libp2p/go-libp2p-protocol", "go-libp2p-protocol"], + + "Content Routing", + ["libp2p/interface-content-routing", "interface-content-routing"], + + "Peer Routing", + ["libp2p/interface-peer-routing", "interface-peer-routing"], + + "Record Store", + ["libp2p/interface-record-store", "interface-record-store"], + + "Miscellaneous", + ["libp2p/go-libp2p-crypto", "go-libp2p-crypto"], + ["libp2p/go-libp2p-interface-connmgr", "go-libp2p-interface-connmgr"], + ["libp2p/go-libp2p-swarm", "go-libp2p-swarm"], + ["libp2p/go-libp2p-host", "go-libp2p-host"], + ["libp2p/go-libp2p-blankhost", "go-libp2p-blankhost"], + ["libp2p/go-conn-security-multistream", "go-conn-security-multistream"], + ["libp2p/go-conn-security", "go-conn-security"], + ["libp2p/go-libp2p-interface-pnet", "go-libp2p-interface-pnet"], + ["libp2p/go-libp2p-metrics", "go-libp2p-metrics"], + + "Utilities", + ["libp2p/go-libp2p-loggables", "go-libp2p-loggables"], + ["libp2p/go-maddr-filter", "go-maddr-filter"], + ["libp2p/go-libp2p-netutil", "go-libp2p-netutil"], + ["libp2p/go-testutil", "go-testutil"] + ] +} From 6c9315e809c162522deef5964da9bc71ee03cfd5 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 22 Aug 2018 12:47:57 +0100 Subject: [PATCH 0558/3965] fix: use name instead of pakage License: MIT Signed-off-by: Henrique Dias --- README.md | 2 +- package-list.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1893a3645c..e2fdc779ef 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,7 @@ $ gx test ./p2p/ List of packages currently in existence for libp2p: -| Package | CI | Coverage | +| Name | CI | Coverage | | ---------|---------|--------- | | **Libp2p** | | [`interface-libp2p`](//github.com/libp2p/interface-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/interface-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-libp2p) | diff --git a/package-list.json b/package-list.json index f8546b801d..04ca7efe09 100644 --- a/package-list.json +++ b/package-list.json @@ -1,6 +1,6 @@ { "columns": [ - "Package", + "Name", "CI", "Coverage" ], From 1de376200e61520d92643901d506ec3bf0e2138c Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 22 Aug 2018 13:26:20 +0100 Subject: [PATCH 0559/3965] add link to package-table License: MIT Signed-off-by: Henrique Dias --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e2fdc779ef..c726c19b1a 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,8 @@ $ gx test ./p2p/ ### Packages +> This table is generated using the module [`package-table`](https://github.com/ipfs-shipyard/package-table) with `package-table --data=package-list.json`. + List of packages currently in existence for libp2p: | Name | CI | Coverage | @@ -160,8 +162,6 @@ List of packages currently in existence for libp2p: | [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-netutil) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | | [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | -> This table is generated using the module `package-table` with `package-table --data=package-list.json`. - # Contribute go-libp2p is part of [The IPFS Project](https://github.com/ipfs/ipfs), and is MIT licensed open source software. We welcome contributions big and small! Take a look at the [community contributing notes](https://github.com/ipfs/community/blob/master/contributing.md). Please make sure to check the [issues](https://github.com/ipfs/go-libp2p/issues). Search the closed ones before reporting things, and help us with the open ones. From a7b1927769c2e896965053ab32681696e850d4a3 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 22 Aug 2018 14:09:37 +0100 Subject: [PATCH 0560/3965] travis and jenkins License: MIT Signed-off-by: Henrique Dias --- README.md | 70 +++++++++++++++++++++++------------------------ package-list.json | 3 +- 2 files changed, 37 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index c726c19b1a..f18642a560 100644 --- a/README.md +++ b/README.md @@ -113,54 +113,54 @@ $ gx test ./p2p/ List of packages currently in existence for libp2p: -| Name | CI | Coverage | -| ---------|---------|--------- | +| Name | CI/Travis | CI/Jenkins | Coverage | +| ---------|---------|---------|--------- | | **Libp2p** | -| [`interface-libp2p`](//github.com/libp2p/interface-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/interface-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-libp2p) | -| [`libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | +| [`interface-libp2p`](//github.com/libp2p/interface-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/interface-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/interface-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/interface-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-libp2p) | +| [`libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | | **Connection** | -| [`interface-connection`](//github.com/libp2p/interface-connection) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-connection/master)](https://ci.ipfs.team/job/libp2p/job/interface-connection/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-connection/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-connection) | -| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | +| [`interface-connection`](//github.com/libp2p/interface-connection) | [![Travis CI](https://travis-ci.org/libp2p/interface-connection.svg?branch=master)](https://travis-ci.org/libp2p/interface-connection) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-connection/master)](https://ci.ipfs.team/job/libp2p/job/interface-connection/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-connection/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-connection) | +| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-net/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-net/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | | **Transport** | -| [`interface-transport`](//github.com/libp2p/interface-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-transport/master)](https://ci.ipfs.team/job/libp2p/job/interface-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-transport) | -| [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | -| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | -| [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | -| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | +| [`interface-transport`](//github.com/libp2p/interface-transport) | [![Travis CI](https://travis-ci.org/libp2p/interface-transport.svg?branch=master)](https://travis-ci.org/libp2p/interface-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-transport/master)](https://ci.ipfs.team/job/libp2p/job/interface-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-transport) | +| [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-ws-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-ws-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | +| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | +| [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-tcp-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-tcp-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | +| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-transport-upgrader/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-transport-upgrader/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | | **Crypto Channels** | -| [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-secio) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | +| [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-secio) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-secio/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-secio/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | | **Stream Muxers** | -| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.org/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.org/libp2p/go-stream-muxer) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | +| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.org/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.org/libp2p/go-stream-muxer) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-stream-muxer/master)](https://ci.ipfs.team/job/libp2p/job/go-stream-muxer/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | | **Discovery** | -| [`interface-peer-discovery`](//github.com/libp2p/interface-peer-discovery) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-peer-discovery/master)](https://ci.ipfs.team/job/libp2p/job/interface-peer-discovery/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-peer-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-peer-discovery) | +| [`interface-peer-discovery`](//github.com/libp2p/interface-peer-discovery) | [![Travis CI](https://travis-ci.org/libp2p/interface-peer-discovery.svg?branch=master)](https://travis-ci.org/libp2p/interface-peer-discovery) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-peer-discovery/master)](https://ci.ipfs.team/job/libp2p/job/interface-peer-discovery/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-peer-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-peer-discovery) | | **NAT Traversal** | -| [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | -| [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | +| [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-circuit/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-circuit/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | +| [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-nat/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-nat/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | | **Data Types** | -| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | -| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | -| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | +| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-peer/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-peer/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | +| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-peerstore/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-peerstore/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | +| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-protocol/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-protocol/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | | **Content Routing** | -| [`interface-content-routing`](//github.com/libp2p/interface-content-routing) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-content-routing/master)](https://ci.ipfs.team/job/libp2p/job/interface-content-routing/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-content-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-content-routing) | +| [`interface-content-routing`](//github.com/libp2p/interface-content-routing) | [![Travis CI](https://travis-ci.org/libp2p/interface-content-routing.svg?branch=master)](https://travis-ci.org/libp2p/interface-content-routing) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-content-routing/master)](https://ci.ipfs.team/job/libp2p/job/interface-content-routing/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-content-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-content-routing) | | **Peer Routing** | -| [`interface-peer-routing`](//github.com/libp2p/interface-peer-routing) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-peer-routing/master)](https://ci.ipfs.team/job/libp2p/job/interface-peer-routing/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-peer-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-peer-routing) | +| [`interface-peer-routing`](//github.com/libp2p/interface-peer-routing) | [![Travis CI](https://travis-ci.org/libp2p/interface-peer-routing.svg?branch=master)](https://travis-ci.org/libp2p/interface-peer-routing) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-peer-routing/master)](https://ci.ipfs.team/job/libp2p/job/interface-peer-routing/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-peer-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-peer-routing) | | **Record Store** | -| [`interface-record-store`](//github.com/libp2p/interface-record-store) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-record-store/master)](https://ci.ipfs.team/job/libp2p/job/interface-record-store/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-record-store/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-record-store) | +| [`interface-record-store`](//github.com/libp2p/interface-record-store) | [![Travis CI](https://travis-ci.org/libp2p/interface-record-store.svg?branch=master)](https://travis-ci.org/libp2p/interface-record-store) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-record-store/master)](https://ci.ipfs.team/job/libp2p/job/interface-record-store/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-record-store/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-record-store) | | **Miscellaneous** | -| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | -| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | -| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-swarm) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | -| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-host) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | -| [`go-libp2p-blankhost`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-blankhost) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | -| [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security-multistream) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | -| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | -| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | -| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | +| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-crypto/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-crypto/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | +| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-interface-connmgr/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-interface-connmgr/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | +| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-swarm) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-swarm/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-swarm/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | +| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-host) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-host/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-host/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | +| [`go-libp2p-blankhost`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-blankhost) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-blankhost/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-blankhost/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | +| [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security-multistream) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-conn-security-multistream/master)](https://ci.ipfs.team/job/libp2p/job/go-conn-security-multistream/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | +| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-conn-security/master)](https://ci.ipfs.team/job/libp2p/job/go-conn-security/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | +| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-pnet) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-interface-pnet/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-interface-pnet/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | +| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-metrics/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-metrics/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | | **Utilities** | -| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | -| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | -| [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-netutil) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | -| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | +| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-loggables/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-loggables/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | +| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-maddr-filter/master)](https://ci.ipfs.team/job/libp2p/job/go-maddr-filter/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | +| [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-netutil) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-netutil/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-netutil/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | +| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-testutil/master)](https://ci.ipfs.team/job/libp2p/job/go-testutil/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | # Contribute diff --git a/package-list.json b/package-list.json index 04ca7efe09..9667207b32 100644 --- a/package-list.json +++ b/package-list.json @@ -1,7 +1,8 @@ { "columns": [ "Name", - "CI", + "CI/Travis", + "CI/Jenkins", "Coverage" ], "rows": [ From 84c701dbfe82a83a53c6aea6d357066ee1767af2 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 22 Aug 2018 14:13:05 +0100 Subject: [PATCH 0561/3965] remove badges for interface- License: MIT Signed-off-by: Henrique Dias --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index f18642a560..23ddf3b0ec 100644 --- a/README.md +++ b/README.md @@ -116,13 +116,13 @@ List of packages currently in existence for libp2p: | Name | CI/Travis | CI/Jenkins | Coverage | | ---------|---------|---------|--------- | | **Libp2p** | -| [`interface-libp2p`](//github.com/libp2p/interface-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/interface-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/interface-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/interface-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-libp2p) | +| [`interface-libp2p`](//github.com/libp2p/interface-libp2p) | | | | | [`libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | | **Connection** | -| [`interface-connection`](//github.com/libp2p/interface-connection) | [![Travis CI](https://travis-ci.org/libp2p/interface-connection.svg?branch=master)](https://travis-ci.org/libp2p/interface-connection) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-connection/master)](https://ci.ipfs.team/job/libp2p/job/interface-connection/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-connection/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-connection) | +| [`interface-connection`](//github.com/libp2p/interface-connection) | | | | | [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-net/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-net/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | | **Transport** | -| [`interface-transport`](//github.com/libp2p/interface-transport) | [![Travis CI](https://travis-ci.org/libp2p/interface-transport.svg?branch=master)](https://travis-ci.org/libp2p/interface-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-transport/master)](https://ci.ipfs.team/job/libp2p/job/interface-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-transport) | +| [`interface-transport`](//github.com/libp2p/interface-transport) | | | | | [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-ws-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-ws-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | | [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | | [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-tcp-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-tcp-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | @@ -132,7 +132,7 @@ List of packages currently in existence for libp2p: | **Stream Muxers** | | [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.org/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.org/libp2p/go-stream-muxer) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-stream-muxer/master)](https://ci.ipfs.team/job/libp2p/job/go-stream-muxer/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | | **Discovery** | -| [`interface-peer-discovery`](//github.com/libp2p/interface-peer-discovery) | [![Travis CI](https://travis-ci.org/libp2p/interface-peer-discovery.svg?branch=master)](https://travis-ci.org/libp2p/interface-peer-discovery) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-peer-discovery/master)](https://ci.ipfs.team/job/libp2p/job/interface-peer-discovery/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-peer-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-peer-discovery) | +| [`interface-peer-discovery`](//github.com/libp2p/interface-peer-discovery) | | | | | **NAT Traversal** | | [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-circuit/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-circuit/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | | [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-nat/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-nat/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | @@ -141,11 +141,11 @@ List of packages currently in existence for libp2p: | [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-peerstore/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-peerstore/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | | [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-protocol/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-protocol/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | | **Content Routing** | -| [`interface-content-routing`](//github.com/libp2p/interface-content-routing) | [![Travis CI](https://travis-ci.org/libp2p/interface-content-routing.svg?branch=master)](https://travis-ci.org/libp2p/interface-content-routing) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-content-routing/master)](https://ci.ipfs.team/job/libp2p/job/interface-content-routing/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-content-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-content-routing) | +| [`interface-content-routing`](//github.com/libp2p/interface-content-routing) | | | | | **Peer Routing** | -| [`interface-peer-routing`](//github.com/libp2p/interface-peer-routing) | [![Travis CI](https://travis-ci.org/libp2p/interface-peer-routing.svg?branch=master)](https://travis-ci.org/libp2p/interface-peer-routing) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-peer-routing/master)](https://ci.ipfs.team/job/libp2p/job/interface-peer-routing/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-peer-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-peer-routing) | +| [`interface-peer-routing`](//github.com/libp2p/interface-peer-routing) | | | | | **Record Store** | -| [`interface-record-store`](//github.com/libp2p/interface-record-store) | [![Travis CI](https://travis-ci.org/libp2p/interface-record-store.svg?branch=master)](https://travis-ci.org/libp2p/interface-record-store) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/interface-record-store/master)](https://ci.ipfs.team/job/libp2p/job/interface-record-store/job/master/) | [![codecov](https://codecov.io/gh/libp2p/interface-record-store/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/interface-record-store) | +| [`interface-record-store`](//github.com/libp2p/interface-record-store) | | | | | **Miscellaneous** | | [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-crypto/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-crypto/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | | [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-interface-connmgr/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-interface-connmgr/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | From c13bbde3cf162bc927647b6847708f6b0e4c0e12 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 22 Aug 2018 18:35:05 +0200 Subject: [PATCH 0562/3965] fix: update package-list --- package-list.json | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/package-list.json b/package-list.json index 9667207b32..51735ba3ad 100644 --- a/package-list.json +++ b/package-list.json @@ -7,15 +7,12 @@ ], "rows": [ "Libp2p", - ["libp2p/interface-libp2p", "interface-libp2p"], ["libp2p/go-libp2p", "libp2p"], "Connection", - ["libp2p/interface-connection", "interface-connection"], ["libp2p/go-libp2p-net", "go-libp2p-net"], "Transport", - ["libp2p/interface-transport", "interface-transport"], ["libp2p/go-ws-transport", "go-ws-transport"], ["libp2p/go-libp2p-transport", "go-libp2p-transport"], ["libp2p/go-tcp-transport", "go-tcp-transport"], @@ -25,6 +22,7 @@ ["libp2p/go-libp2p-secio", "go-libp2p-secio"], "Stream Muxers", + ["libp2p/interface-stream-muxer", "interface-stream-muxer"], ["libp2p/go-stream-muxer", "go-stream-muxer"], "Discovery", From e7739aadfce3108907edc13ffc94c3a5d7a72873 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 22 Aug 2018 18:36:46 +0200 Subject: [PATCH 0563/3965] docs: update packages table --- README.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/README.md b/README.md index 23ddf3b0ec..4777744307 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,6 @@

- -
@@ -116,13 +114,10 @@ List of packages currently in existence for libp2p: | Name | CI/Travis | CI/Jenkins | Coverage | | ---------|---------|---------|--------- | | **Libp2p** | -| [`interface-libp2p`](//github.com/libp2p/interface-libp2p) | | | | | [`libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | | **Connection** | -| [`interface-connection`](//github.com/libp2p/interface-connection) | | | | | [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-net/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-net/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | | **Transport** | -| [`interface-transport`](//github.com/libp2p/interface-transport) | | | | | [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-ws-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-ws-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | | [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | | [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-tcp-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-tcp-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | @@ -130,6 +125,7 @@ List of packages currently in existence for libp2p: | **Crypto Channels** | | [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-secio) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-secio/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-secio/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | | **Stream Muxers** | +| [`interface-stream-muxer`](//github.com/libp2p/interface-stream-muxer) | | | | | [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.org/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.org/libp2p/go-stream-muxer) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-stream-muxer/master)](https://ci.ipfs.team/job/libp2p/job/go-stream-muxer/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | | **Discovery** | | [`interface-peer-discovery`](//github.com/libp2p/interface-peer-discovery) | | | | @@ -141,11 +137,7 @@ List of packages currently in existence for libp2p: | [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-peerstore/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-peerstore/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | | [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-protocol/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-protocol/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | | **Content Routing** | -| [`interface-content-routing`](//github.com/libp2p/interface-content-routing) | | | | | **Peer Routing** | -| [`interface-peer-routing`](//github.com/libp2p/interface-peer-routing) | | | | -| **Record Store** | -| [`interface-record-store`](//github.com/libp2p/interface-record-store) | | | | | **Miscellaneous** | | [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-crypto/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-crypto/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | | [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-interface-connmgr/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-interface-connmgr/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | From e1ef4d206d6c995e135afa3f1edca3008bb69815 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 22 Aug 2018 18:37:13 +0200 Subject: [PATCH 0564/3965] Update package-list.json --- package-list.json | 4 ---- 1 file changed, 4 deletions(-) diff --git a/package-list.json b/package-list.json index 51735ba3ad..ae59093146 100644 --- a/package-list.json +++ b/package-list.json @@ -38,13 +38,9 @@ ["libp2p/go-libp2p-protocol", "go-libp2p-protocol"], "Content Routing", - ["libp2p/interface-content-routing", "interface-content-routing"], "Peer Routing", - ["libp2p/interface-peer-routing", "interface-peer-routing"], - "Record Store", - ["libp2p/interface-record-store", "interface-record-store"], "Miscellaneous", ["libp2p/go-libp2p-crypto", "go-libp2p-crypto"], From cc7c13b9b5d2995d2e2f72163be80bc275759ff2 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 22 Aug 2018 19:10:32 +0100 Subject: [PATCH 0565/3965] update table License: MIT Signed-off-by: Henrique Dias --- README.md | 80 ++++++++++++++++++++++++++------------------- package-list.json | 83 +++++++++++++++++++++++++++-------------------- 2 files changed, 94 insertions(+), 69 deletions(-) diff --git a/README.md b/README.md index 4777744307..258f320527 100644 --- a/README.md +++ b/README.md @@ -111,48 +111,60 @@ $ gx test ./p2p/ List of packages currently in existence for libp2p: -| Name | CI/Travis | CI/Jenkins | Coverage | -| ---------|---------|---------|--------- | +| Name | CI/Travis | CI/Jenkins | Coverage | Description | +| ---------|---------|---------|---------|--------- | | **Libp2p** | -| [`libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | -| **Connection** | -| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-net/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-net/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | +| [`go-libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | go-libp2p entry point | +| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-host) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | libp2p "host" interface | +| [`minimal implementation of the "host" interface`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-blankhost) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | | +| **Network** | +| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | libp2p connection and "network" interfaces | +| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-swarm) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-swarm/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-swarm/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | reference implementation | | **Transport** | -| [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-ws-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-ws-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | -| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | -| [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-tcp-transport/master)](https://ci.ipfs.team/job/libp2p/job/go-tcp-transport/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | -| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-transport-upgrader/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-transport-upgrader/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | +| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | interfaces | +| [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | WebSocket transport | +| [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | TCP transport | +| [`go-libp2p-quic-transport`](//github.com/libp2p/go-libp2p-quic-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-quic-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-quic-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-quic-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-quic-transport) | QUIC transport | +| [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-circuit/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-circuit/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | relay transport | +| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | helpers for implementing new transports | +| [`go-libp2p-reuseport-transport`](//github.com/libp2p/go-libp2p-reuseport-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-reuseport-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-reuseport-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport) | partial transport for building transports that reuse ports | | **Crypto Channels** | -| [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-secio) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-secio/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-secio/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | +| [`go-conn-security`](//github.com/libp2p/go-libp2p-security) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-security.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-security) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-security) | interfaces | +| [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-secio) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | SecIO crypto channel | +| [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security-multistream) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | multistream multiplexed meta crypto channel | +| **Private Network** | +| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-pnet) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | interfaces | +| [`go-libp2p-pnet`](//github.com/libp2p/go-libp2p-pnet) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pnet.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pnet) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pnet) | reference implementation | | **Stream Muxers** | -| [`interface-stream-muxer`](//github.com/libp2p/interface-stream-muxer) | | | | -| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.org/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.org/libp2p/go-stream-muxer) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-stream-muxer/master)](https://ci.ipfs.team/job/libp2p/job/go-stream-muxer/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | -| **Discovery** | -| [`interface-peer-discovery`](//github.com/libp2p/interface-peer-discovery) | | | | +| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.org/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.org/libp2p/go-stream-muxer) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | interfaces | +| [`go-smux-yamux`](//github.com/whyrusleeping/go-smux-yamux) | [![Travis CI](https://travis-ci.org/whyrusleeping/go-smux-yamux.svg?branch=master)](https://travis-ci.org/whyrusleeping/go-smux-yamux) | N/A | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-yamux/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-yamux) | YAMUX stream multiplexer | +| [`go-smux-mplex`](//github.com/whyrusleeping/go-smux-mplex) | [![Travis CI](https://travis-ci.org/whyrusleeping/go-smux-mplex.svg?branch=master)](https://travis-ci.org/whyrusleeping/go-smux-mplex) | N/A | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-mplex/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-mplex) | MPLEX stream multiplexer | | **NAT Traversal** | -| [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-circuit/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-circuit/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | -| [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-nat/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-nat/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | +| [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | | +| **Peerstore** | +| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | interfaces and reference implementation | +| **Connection Manager** | +| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | interface | +| [`go-libp2p-connmgr`](//github.com/libp2p/go-libp2p-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-connmgr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-connmgr) | reference implementation | | **Data Types** | -| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-peer/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-peer/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | -| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-peerstore/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-peerstore/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | -| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-protocol/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-protocol/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | -| **Content Routing** | -| **Peer Routing** | +| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | libp2p peer-ID datatype | +| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | libp2p key types | +| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | libp2p protocol datatype | +| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kbucket) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademila routing table helper types | +| **Routing** | +| [`go-libp2p-routing`](//github.com/libp2p/go-libp2p-routing) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing) | routing interfaces | +| [`go-libp2p-record`](//github.com/libp2p/go-libp2p-record) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-record.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-record) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-record/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-record) | record type and validator logic | +| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | +| [`go-libp2p-kad-dht`](//github.com/libp2p/go-libp2p-kad-dht) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kad-dht) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-kad-dht/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-kad-dht/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kad-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kad-dht) | Kademlia-like router | +| [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub-router) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | | **Miscellaneous** | -| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-crypto/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-crypto/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | -| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-interface-connmgr/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-interface-connmgr/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | -| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-swarm) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-swarm/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-swarm/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | -| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-host) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-host/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-host/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | -| [`go-libp2p-blankhost`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-blankhost) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-blankhost/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-blankhost/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | -| [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security-multistream) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-conn-security-multistream/master)](https://ci.ipfs.team/job/libp2p/job/go-conn-security-multistream/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | -| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-conn-security/master)](https://ci.ipfs.team/job/libp2p/job/go-conn-security/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | -| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-pnet) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-interface-pnet/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-interface-pnet/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | -| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-metrics/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-metrics/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | +| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | libp2p metrics interfaces/collectors | +| [`go-msgio`](//github.com/libp2p/go-msgio) | [![Travis CI](https://travis-ci.org/libp2p/go-msgio.svg?branch=master)](https://travis-ci.org/libp2p/go-msgio) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-msgio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-msgio) | length prefixed data channel | | **Utilities** | -| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-loggables/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-loggables/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | -| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-maddr-filter/master)](https://ci.ipfs.team/job/libp2p/job/go-maddr-filter/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | -| [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-netutil) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-netutil/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-netutil/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | -| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-testutil/master)](https://ci.ipfs.team/job/libp2p/job/go-testutil/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | +| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | +| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | multiaddr filtering helpers | +| [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-netutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | misc utilities | +| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | misc utilities | # Contribute diff --git a/package-list.json b/package-list.json index ae59093146..9bc7b70781 100644 --- a/package-list.json +++ b/package-list.json @@ -3,60 +3,73 @@ "Name", "CI/Travis", "CI/Jenkins", - "Coverage" + "Coverage", + "Description" ], "rows": [ "Libp2p", - ["libp2p/go-libp2p", "libp2p"], + ["libp2p/go-libp2p", "go-libp2p", "go-libp2p entry point"], + ["libp2p/go-libp2p-host", "go-libp2p-host", "libp2p \"host\" interface"], + ["libp2p/go-libp2p-blankhost", "minimal implementation of the \"host\" interface"], - "Connection", - ["libp2p/go-libp2p-net", "go-libp2p-net"], + "Network", + ["libp2p/go-libp2p-net", "go-libp2p-net", "libp2p connection and \"network\" interfaces"], + ["libp2p/go-libp2p-swarm", "go-libp2p-swarm", "reference implementation"], "Transport", - ["libp2p/go-ws-transport", "go-ws-transport"], - ["libp2p/go-libp2p-transport", "go-libp2p-transport"], - ["libp2p/go-tcp-transport", "go-tcp-transport"], - ["libp2p/go-libp2p-transport-upgrader", "go-libp2p-transport-upgrader"], + ["libp2p/go-libp2p-transport", "go-libp2p-transport", "interfaces"], + ["libp2p/go-ws-transport", "go-ws-transport", "WebSocket transport"], + ["libp2p/go-tcp-transport", "go-tcp-transport", "TCP transport"], + ["libp2p/go-libp2p-quic-transport", "go-libp2p-quic-transport", "QUIC transport"], + ["libp2p/go-libp2p-circuit", "go-libp2p-circuit", "relay transport"], + ["libp2p/go-libp2p-transport-upgrader", "go-libp2p-transport-upgrader", "helpers for implementing new transports"], + ["libp2p/go-libp2p-reuseport-transport", "go-libp2p-reuseport-transport", "partial transport for building transports that reuse ports"], "Crypto Channels", - ["libp2p/go-libp2p-secio", "go-libp2p-secio"], + ["libp2p/go-libp2p-security", "go-conn-security", "interfaces"], + ["libp2p/go-libp2p-secio", "go-libp2p-secio", "SecIO crypto channel"], + ["libp2p/go-conn-security-multistream", "go-conn-security-multistream", "multistream multiplexed meta crypto channel"], - "Stream Muxers", - ["libp2p/interface-stream-muxer", "interface-stream-muxer"], - ["libp2p/go-stream-muxer", "go-stream-muxer"], + "Private Network", + ["libp2p/go-libp2p-interface-pnet", "go-libp2p-interface-pnet", "interfaces"], + ["libp2p/go-libp2p-pnet", "go-libp2p-pnet", "reference implementation"], - "Discovery", - ["libp2p/interface-peer-discovery", "interface-peer-discovery"], + "Stream Muxers", + ["libp2p/go-stream-muxer", "go-stream-muxer", "interfaces"], + ["whyrusleeping/go-smux-yamux", "go-smux-yamux", "YAMUX stream multiplexer"], + ["whyrusleeping/go-smux-mplex", "go-smux-mplex", "MPLEX stream multiplexer"], "NAT Traversal", - ["libp2p/go-libp2p-circuit", "go-libp2p-circuit"], ["libp2p/go-libp2p-nat", "go-libp2p-nat"], - - "Data Types", - ["libp2p/go-libp2p-peer", "go-libp2p-peer"], - ["libp2p/go-libp2p-peerstore", "go-libp2p-peerstore"], - ["libp2p/go-libp2p-protocol", "go-libp2p-protocol"], - "Content Routing", + "Peerstore", + ["libp2p/go-libp2p-peerstore", "go-libp2p-peerstore", "interfaces and reference implementation"], - "Peer Routing", + "Connection Manager", + ["libp2p/go-libp2p-interface-connmgr", "go-libp2p-interface-connmgr", "interface"], + ["libp2p/go-libp2p-connmgr", "go-libp2p-connmgr", "reference implementation"], + + "Data Types", + ["libp2p/go-libp2p-peer", "go-libp2p-peer", "libp2p peer-ID datatype"], + ["libp2p/go-libp2p-crypto", "go-libp2p-crypto", "libp2p key types"], + ["libp2p/go-libp2p-protocol", "go-libp2p-protocol", "libp2p protocol datatype"], + ["libp2p/go-libp2p-kbucket", "go-libp2p-kbucket", "Kademila routing table helper types"], + "Routing", + ["libp2p/go-libp2p-routing", "go-libp2p-routing", "routing interfaces"], + ["libp2p/go-libp2p-record", "go-libp2p-record", "record type and validator logic"], + ["libp2p/go-libp2p-routing-helpers", "go-libp2p-routing-helpers", "helpers for composing routers"], + ["libp2p/go-libp2p-kad-dht", "go-libp2p-kad-dht", "Kademlia-like router"], + ["libp2p/go-libp2p-pubsub-router", "go-libp2p-pubsub-router", "record-store over pubsub adapter"], "Miscellaneous", - ["libp2p/go-libp2p-crypto", "go-libp2p-crypto"], - ["libp2p/go-libp2p-interface-connmgr", "go-libp2p-interface-connmgr"], - ["libp2p/go-libp2p-swarm", "go-libp2p-swarm"], - ["libp2p/go-libp2p-host", "go-libp2p-host"], - ["libp2p/go-libp2p-blankhost", "go-libp2p-blankhost"], - ["libp2p/go-conn-security-multistream", "go-conn-security-multistream"], - ["libp2p/go-conn-security", "go-conn-security"], - ["libp2p/go-libp2p-interface-pnet", "go-libp2p-interface-pnet"], - ["libp2p/go-libp2p-metrics", "go-libp2p-metrics"], + ["libp2p/go-libp2p-metrics", "go-libp2p-metrics", "libp2p metrics interfaces/collectors"], + ["libp2p/go-msgio", "go-msgio", "length prefixed data channel"], "Utilities", - ["libp2p/go-libp2p-loggables", "go-libp2p-loggables"], - ["libp2p/go-maddr-filter", "go-maddr-filter"], - ["libp2p/go-libp2p-netutil", "go-libp2p-netutil"], - ["libp2p/go-testutil", "go-testutil"] + ["libp2p/go-libp2p-loggables", "go-libp2p-loggables", "logging helpers"], + ["libp2p/go-maddr-filter", "go-maddr-filter", "multiaddr filtering helpers"], + ["libp2p/go-libp2p-netutil", "go-libp2p-netutil", "misc utilities"], + ["libp2p/go-testutil", "go-testutil", "misc utilities"] ] } From 30901d57f94d22943ce8244002acdb5b150638e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rau=CC=81l=20Kripalani?= Date: Thu, 23 Aug 2018 13:42:19 +0100 Subject: [PATCH 0566/3965] enhance package list with new items. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit License: MIT Signed-off-by: RauÌl Kripalani --- README.md | 47 +++++++++++++++++++++++++++++---------- package-list.json | 56 +++++++++++++++++++++++++++++++++++------------ 2 files changed, 77 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index 258f320527..8d9cf4f898 100644 --- a/README.md +++ b/README.md @@ -125,11 +125,13 @@ List of packages currently in existence for libp2p: | [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | WebSocket transport | | [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | TCP transport | | [`go-libp2p-quic-transport`](//github.com/libp2p/go-libp2p-quic-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-quic-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-quic-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-quic-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-quic-transport) | QUIC transport | +| [`go-udp-transport`](//github.com/libp2p/go-udp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-udp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-udp-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-udp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-udp-transport) | UDP transport | +| [`go-utp-transport`](//github.com/libp2p/go-utp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-utp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-utp-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-utp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-utp-transport) | uTorrent transport (UTP) | | [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-circuit/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-circuit/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | relay transport | -| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | helpers for implementing new transports | +| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | upgrades multiaddr-net connections into full libp2p transports | | [`go-libp2p-reuseport-transport`](//github.com/libp2p/go-libp2p-reuseport-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-reuseport-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-reuseport-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport) | partial transport for building transports that reuse ports | -| **Crypto Channels** | -| [`go-conn-security`](//github.com/libp2p/go-libp2p-security) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-security.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-security) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-security) | interfaces | +| **Encrypted Channels** | +| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | interfaces | | [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-secio) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | SecIO crypto channel | | [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security-multistream) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | multistream multiplexed meta crypto channel | | **Private Network** | @@ -146,25 +148,46 @@ List of packages currently in existence for libp2p: | **Connection Manager** | | [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | interface | | [`go-libp2p-connmgr`](//github.com/libp2p/go-libp2p-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-connmgr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-connmgr) | reference implementation | -| **Data Types** | -| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | libp2p peer-ID datatype | -| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | libp2p key types | -| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | libp2p protocol datatype | -| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kbucket) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademila routing table helper types | | **Routing** | | [`go-libp2p-routing`](//github.com/libp2p/go-libp2p-routing) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing) | routing interfaces | | [`go-libp2p-record`](//github.com/libp2p/go-libp2p-record) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-record.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-record) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-record/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-record) | record type and validator logic | | [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | | [`go-libp2p-kad-dht`](//github.com/libp2p/go-libp2p-kad-dht) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kad-dht) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-kad-dht/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-kad-dht/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kad-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kad-dht) | Kademlia-like router | | [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub-router) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | -| **Miscellaneous** | +| [`go-libp2p-coral-dht`](//github.com/libp2p/go-libp2p-coral-dht) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-coral-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-coral-dht) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-coral-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-coral-dht) | CoralDHT implementation of the routing interfaces | +| **Consensus** | +| [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-consensus) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | +| [`go-libp2p-raft`](//github.com/libp2p/go-libp2p-raft) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-raft.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-raft) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-raft/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-raft/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-raft/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-raft) | consensus implementation over raft | +| **Pubsub** | +| [`go-libp2p-pubsub`](//github.com/libp2p/go-libp2p-pubsub) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub) | pubsub interfaces | +| [`go-floodsub`](//github.com/libp2p/go-floodsub) | [![Travis CI](https://travis-ci.org/libp2p/go-floodsub.svg?branch=master)](https://travis-ci.org/libp2p/go-floodsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-floodsub/master)](https://ci.ipfs.team/job/libp2p/job/go-floodsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-floodsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-floodsub) | basic implementation of pubsub driven by flooding | +| **Protocols** | +| [`go-libp2p-ping`](//github.com/libp2p/go-libp2p-ping) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-ping.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-ping) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-ping/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-ping) | a ping-pong protocol implementation | +| **Metrics** | | [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | libp2p metrics interfaces/collectors | -| [`go-msgio`](//github.com/libp2p/go-msgio) | [![Travis CI](https://travis-ci.org/libp2p/go-msgio.svg?branch=master)](https://travis-ci.org/libp2p/go-msgio) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-msgio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-msgio) | length prefixed data channel | -| **Utilities** | +| **Data Types** | +| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | libp2p peer-ID datatype | +| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | libp2p key types | +| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | libp2p protocol datatype | +| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kbucket) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademila routing table helper types | +| **Utilities/miscellaneous** | | [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | | [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | multiaddr filtering helpers | | [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-netutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | misc utilities | -| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | misc utilities | +| [`go-msgio`](//github.com/libp2p/go-msgio) | [![Travis CI](https://travis-ci.org/libp2p/go-msgio.svg?branch=master)](https://travis-ci.org/libp2p/go-msgio) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-msgio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-msgio) | length prefixed data channel | +| [`go-addr-util`](//github.com/libp2p/go-addr-util) | [![Travis CI](https://travis-ci.org/libp2p/go-addr-util.svg?branch=master)](https://travis-ci.org/libp2p/go-addr-util) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-addr-util/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-addr-util) | address utilities for libp2p swarm | +| [`go-buffer-pool`](//github.com/libp2p/go-buffer-pool) | [![Travis CI](https://travis-ci.org/libp2p/go-buffer-pool.svg?branch=master)](https://travis-ci.org/libp2p/go-buffer-pool) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-buffer-pool/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-buffer-pool) | a variable size buffer pool for go | +| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | +| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | routing helpers | +| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | a library to perform filtering of multiaddrs. | +| [`go-reuseport`](//github.com/libp2p/go-reuseport) | [![Travis CI](https://travis-ci.org/libp2p/go-reuseport.svg?branch=master)](https://travis-ci.org/libp2p/go-reuseport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-reuseport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-reuseport) | enables reuse of addresses | +| [`go-sockaddr`](//github.com/libp2p/go-sockaddr) | [![Travis CI](https://travis-ci.org/libp2p/go-sockaddr.svg?branch=master)](https://travis-ci.org/libp2p/go-sockaddr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-sockaddr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-sockaddr) | utils for sockaddr conversions | +| [`go-flow-metrics`](//github.com/libp2p/go-flow-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-flow-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-flow-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-flow-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-flow-metrics) | metrics library | +| **Testing and examples** | +| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | a collection of testing utilities for ipfs and libp2p | +| [`go-libp2p-dummy-conn`](//github.com/libp2p/go-libp2p-dummy-conn) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-dummy-conn.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-dummy-conn) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-dummy-conn/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-dummy-conn) | a dummy libp2p connection for testing | +| [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-examples) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-examples/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-examples/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | +| [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit-progs) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | # Contribute diff --git a/package-list.json b/package-list.json index 9bc7b70781..51b8eaef3c 100644 --- a/package-list.json +++ b/package-list.json @@ -21,12 +21,14 @@ ["libp2p/go-ws-transport", "go-ws-transport", "WebSocket transport"], ["libp2p/go-tcp-transport", "go-tcp-transport", "TCP transport"], ["libp2p/go-libp2p-quic-transport", "go-libp2p-quic-transport", "QUIC transport"], + ["libp2p/go-udp-transport", "go-udp-transport", "UDP transport"], + ["libp2p/go-utp-transport", "go-utp-transport", "uTorrent transport (UTP)"], ["libp2p/go-libp2p-circuit", "go-libp2p-circuit", "relay transport"], - ["libp2p/go-libp2p-transport-upgrader", "go-libp2p-transport-upgrader", "helpers for implementing new transports"], + ["libp2p/go-libp2p-transport-upgrader", "go-libp2p-transport-upgrader", "upgrades multiaddr-net connections into full libp2p transports"], ["libp2p/go-libp2p-reuseport-transport", "go-libp2p-reuseport-transport", "partial transport for building transports that reuse ports"], - "Crypto Channels", - ["libp2p/go-libp2p-security", "go-conn-security", "interfaces"], + "Encrypted Channels", + ["libp2p/go-conn-security", "go-conn-security", "interfaces"], ["libp2p/go-libp2p-secio", "go-libp2p-secio", "SecIO crypto channel"], ["libp2p/go-conn-security-multistream", "go-conn-security-multistream", "multistream multiplexed meta crypto channel"], @@ -48,12 +50,6 @@ "Connection Manager", ["libp2p/go-libp2p-interface-connmgr", "go-libp2p-interface-connmgr", "interface"], ["libp2p/go-libp2p-connmgr", "go-libp2p-connmgr", "reference implementation"], - - "Data Types", - ["libp2p/go-libp2p-peer", "go-libp2p-peer", "libp2p peer-ID datatype"], - ["libp2p/go-libp2p-crypto", "go-libp2p-crypto", "libp2p key types"], - ["libp2p/go-libp2p-protocol", "go-libp2p-protocol", "libp2p protocol datatype"], - ["libp2p/go-libp2p-kbucket", "go-libp2p-kbucket", "Kademila routing table helper types"], "Routing", ["libp2p/go-libp2p-routing", "go-libp2p-routing", "routing interfaces"], @@ -61,15 +57,47 @@ ["libp2p/go-libp2p-routing-helpers", "go-libp2p-routing-helpers", "helpers for composing routers"], ["libp2p/go-libp2p-kad-dht", "go-libp2p-kad-dht", "Kademlia-like router"], ["libp2p/go-libp2p-pubsub-router", "go-libp2p-pubsub-router", "record-store over pubsub adapter"], + ["libp2p/go-libp2p-coral-dht", "go-libp2p-coral-dht", "CoralDHT implementation of the routing interfaces"], + + "Consensus", + ["libp2p/go-libp2p-consensus", "go-libp2p-consensus", "consensus protocols interfaces"], + ["libp2p/go-libp2p-raft", "go-libp2p-raft", "consensus implementation over raft"], + + "Pubsub", + ["libp2p/go-libp2p-pubsub", "go-libp2p-pubsub", "pubsub interfaces"], + ["libp2p/go-floodsub", "go-floodsub", "basic implementation of pubsub driven by flooding"], - "Miscellaneous", + "Protocols", + ["libp2p/go-libp2p-ping", "go-libp2p-ping", "a ping-pong protocol implementation"], + + "Metrics", ["libp2p/go-libp2p-metrics", "go-libp2p-metrics", "libp2p metrics interfaces/collectors"], - ["libp2p/go-msgio", "go-msgio", "length prefixed data channel"], + + "Data Types", + ["libp2p/go-libp2p-peer", "go-libp2p-peer", "libp2p peer-ID datatype"], + ["libp2p/go-libp2p-crypto", "go-libp2p-crypto", "libp2p key types"], + ["libp2p/go-libp2p-protocol", "go-libp2p-protocol", "libp2p protocol datatype"], + ["libp2p/go-libp2p-kbucket", "go-libp2p-kbucket", "Kademila routing table helper types"], - "Utilities", + "Utilities/miscellaneous", ["libp2p/go-libp2p-loggables", "go-libp2p-loggables", "logging helpers"], ["libp2p/go-maddr-filter", "go-maddr-filter", "multiaddr filtering helpers"], ["libp2p/go-libp2p-netutil", "go-libp2p-netutil", "misc utilities"], - ["libp2p/go-testutil", "go-testutil", "misc utilities"] + ["libp2p/go-msgio", "go-msgio", "length prefixed data channel"], + ["libp2p/go-addr-util", "go-addr-util", "address utilities for libp2p swarm"], + ["libp2p/go-buffer-pool", "go-buffer-pool", "a variable size buffer pool for go"], + ["libp2p/go-libp2p-loggables", "go-libp2p-loggables", "logging helpers"], + ["libp2p/go-libp2p-routing-helpers", "go-libp2p-routing-helpers", "routing helpers"], + ["libp2p/go-maddr-filter", "go-maddr-filter", "a library to perform filtering of multiaddrs."], + ["libp2p/go-reuseport", "go-reuseport", "enables reuse of addresses"], + ["libp2p/go-sockaddr", "go-sockaddr", "utils for sockaddr conversions"], + ["libp2p/go-flow-metrics", "go-flow-metrics", "metrics library"], + + "Testing and examples", + ["libp2p/go-testutil", "go-testutil", "a collection of testing utilities for ipfs and libp2p"], + ["libp2p/go-libp2p-dummy-conn", "go-libp2p-dummy-conn", "a dummy libp2p connection for testing"], + ["libp2p/go-libp2p-examples", "go-libp2p-examples", "go-libp2p examples and tutorials"], + ["libp2p/go-libp2p-circuit-progs", "go-libp2p-circuit-progs", "testing programs for go-libp2p-circuit"] + ] -} +} \ No newline at end of file From 2d5c64cdd374c62b5668d76804824a5eee3f6bbc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 23 Aug 2018 18:37:54 +0100 Subject: [PATCH 0567/3965] Fix blankhost entry. --- package-list.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/package-list.json b/package-list.json index 51b8eaef3c..7188742af4 100644 --- a/package-list.json +++ b/package-list.json @@ -10,7 +10,7 @@ "Libp2p", ["libp2p/go-libp2p", "go-libp2p", "go-libp2p entry point"], ["libp2p/go-libp2p-host", "go-libp2p-host", "libp2p \"host\" interface"], - ["libp2p/go-libp2p-blankhost", "minimal implementation of the \"host\" interface"], + ["libp2p/go-libp2p-blankhost", "go-libp2p-blankhost", "minimal implementation of the \"host\" interface"], "Network", ["libp2p/go-libp2p-net", "go-libp2p-net", "libp2p connection and \"network\" interfaces"], @@ -100,4 +100,4 @@ ["libp2p/go-libp2p-circuit-progs", "go-libp2p-circuit-progs", "testing programs for go-libp2p-circuit"] ] -} \ No newline at end of file +} From a6d2132260c8460dc3fa21390359dbd97e432c32 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rau=CC=81l=20Kripalani?= Date: Fri, 24 Aug 2018 19:49:07 +0100 Subject: [PATCH 0568/3965] README: fix go-libp2p-blankhost row. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit License: MIT Signed-off-by: RauÌl Kripalani --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8d9cf4f898..1192ca862c 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ List of packages currently in existence for libp2p: | **Libp2p** | | [`go-libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | go-libp2p entry point | | [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-host) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | libp2p "host" interface | -| [`minimal implementation of the "host" interface`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-blankhost) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | | +| [`go-libp2p-blankhost`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-blankhost) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | minimal implementation of the "host" interface | | **Network** | | [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | libp2p connection and "network" interfaces | | [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-swarm) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-swarm/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-swarm/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | reference implementation | From 788a09fbba5f4283fec56942ecf3fe9dbe82fc32 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 24 Aug 2018 12:52:47 -0700 Subject: [PATCH 0569/3965] remove some packages from the package table * go-libp2p-coral-dht is currently being written and shouldn't go in this list until it's reasonable usable (unless I'm missing the point of this list). * go-libp2p-ping is an unmaintained extraction of go-libp2p/p2p/ping * go-libp2p-pubsub is slated to be removed. * go-libpp2-dummy-conn was implemented to test go-libp2p-pnet. (1) go-libp2p-pnet can now be tested with normal `io.ReadWriteCloser`s so it doesn't need this package and (2) I didn't bother updating this package when I did the libp2p transport refactor. --- README.md | 5 ----- package-list.json | 6 ------ 2 files changed, 11 deletions(-) diff --git a/README.md b/README.md index 1192ca862c..b62cdb6ce7 100644 --- a/README.md +++ b/README.md @@ -154,15 +154,11 @@ List of packages currently in existence for libp2p: | [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | | [`go-libp2p-kad-dht`](//github.com/libp2p/go-libp2p-kad-dht) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kad-dht) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-kad-dht/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-kad-dht/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kad-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kad-dht) | Kademlia-like router | | [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub-router) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | -| [`go-libp2p-coral-dht`](//github.com/libp2p/go-libp2p-coral-dht) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-coral-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-coral-dht) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-coral-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-coral-dht) | CoralDHT implementation of the routing interfaces | | **Consensus** | | [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-consensus) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | | [`go-libp2p-raft`](//github.com/libp2p/go-libp2p-raft) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-raft.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-raft) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-raft/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-raft/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-raft/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-raft) | consensus implementation over raft | | **Pubsub** | -| [`go-libp2p-pubsub`](//github.com/libp2p/go-libp2p-pubsub) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub) | pubsub interfaces | | [`go-floodsub`](//github.com/libp2p/go-floodsub) | [![Travis CI](https://travis-ci.org/libp2p/go-floodsub.svg?branch=master)](https://travis-ci.org/libp2p/go-floodsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-floodsub/master)](https://ci.ipfs.team/job/libp2p/job/go-floodsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-floodsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-floodsub) | basic implementation of pubsub driven by flooding | -| **Protocols** | -| [`go-libp2p-ping`](//github.com/libp2p/go-libp2p-ping) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-ping.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-ping) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-ping/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-ping) | a ping-pong protocol implementation | | **Metrics** | | [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | libp2p metrics interfaces/collectors | | **Data Types** | @@ -185,7 +181,6 @@ List of packages currently in existence for libp2p: | [`go-flow-metrics`](//github.com/libp2p/go-flow-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-flow-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-flow-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-flow-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-flow-metrics) | metrics library | | **Testing and examples** | | [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | a collection of testing utilities for ipfs and libp2p | -| [`go-libp2p-dummy-conn`](//github.com/libp2p/go-libp2p-dummy-conn) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-dummy-conn.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-dummy-conn) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-dummy-conn/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-dummy-conn) | a dummy libp2p connection for testing | | [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-examples) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-examples/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-examples/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | | [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit-progs) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | diff --git a/package-list.json b/package-list.json index 7188742af4..bfcb5057e7 100644 --- a/package-list.json +++ b/package-list.json @@ -57,19 +57,14 @@ ["libp2p/go-libp2p-routing-helpers", "go-libp2p-routing-helpers", "helpers for composing routers"], ["libp2p/go-libp2p-kad-dht", "go-libp2p-kad-dht", "Kademlia-like router"], ["libp2p/go-libp2p-pubsub-router", "go-libp2p-pubsub-router", "record-store over pubsub adapter"], - ["libp2p/go-libp2p-coral-dht", "go-libp2p-coral-dht", "CoralDHT implementation of the routing interfaces"], "Consensus", ["libp2p/go-libp2p-consensus", "go-libp2p-consensus", "consensus protocols interfaces"], ["libp2p/go-libp2p-raft", "go-libp2p-raft", "consensus implementation over raft"], "Pubsub", - ["libp2p/go-libp2p-pubsub", "go-libp2p-pubsub", "pubsub interfaces"], ["libp2p/go-floodsub", "go-floodsub", "basic implementation of pubsub driven by flooding"], - "Protocols", - ["libp2p/go-libp2p-ping", "go-libp2p-ping", "a ping-pong protocol implementation"], - "Metrics", ["libp2p/go-libp2p-metrics", "go-libp2p-metrics", "libp2p metrics interfaces/collectors"], @@ -95,7 +90,6 @@ "Testing and examples", ["libp2p/go-testutil", "go-testutil", "a collection of testing utilities for ipfs and libp2p"], - ["libp2p/go-libp2p-dummy-conn", "go-libp2p-dummy-conn", "a dummy libp2p connection for testing"], ["libp2p/go-libp2p-examples", "go-libp2p-examples", "go-libp2p examples and tutorials"], ["libp2p/go-libp2p-circuit-progs", "go-libp2p-circuit-progs", "testing programs for go-libp2p-circuit"] From cf61930207bba73913211654c6ee14cad28ff1f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rau=CC=81l=20Kripalani?= Date: Mon, 27 Aug 2018 11:08:40 +0100 Subject: [PATCH 0570/3965] package-table: correct small typo. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit License: MIT Signed-off-by: RauÌl Kripalani --- README.md | 2 +- package-list.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b62cdb6ce7..af67156e19 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,7 @@ List of packages currently in existence for libp2p: | [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | libp2p peer-ID datatype | | [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | libp2p key types | | [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | libp2p protocol datatype | -| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kbucket) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademila routing table helper types | +| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kbucket) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademlia routing table helper types | | **Utilities/miscellaneous** | | [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | | [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | multiaddr filtering helpers | diff --git a/package-list.json b/package-list.json index bfcb5057e7..0d741ae13c 100644 --- a/package-list.json +++ b/package-list.json @@ -72,7 +72,7 @@ ["libp2p/go-libp2p-peer", "go-libp2p-peer", "libp2p peer-ID datatype"], ["libp2p/go-libp2p-crypto", "go-libp2p-crypto", "libp2p key types"], ["libp2p/go-libp2p-protocol", "go-libp2p-protocol", "libp2p protocol datatype"], - ["libp2p/go-libp2p-kbucket", "go-libp2p-kbucket", "Kademila routing table helper types"], + ["libp2p/go-libp2p-kbucket", "go-libp2p-kbucket", "Kademlia routing table helper types"], "Utilities/miscellaneous", ["libp2p/go-libp2p-loggables", "go-libp2p-loggables", "logging helpers"], From f724bff2f9e66aaf44e9a832c5aea0a6fc72508c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rau=CC=81l=20Kripalani?= Date: Mon, 27 Aug 2018 14:42:43 +0100 Subject: [PATCH 0571/3965] add docs to BasicConnMgr --- p2p/net/connmgr/connmgr.go | 70 ++++++++++++++++++++++++++++++++------ 1 file changed, 60 insertions(+), 10 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index fc406bf8b8..94eca7d445 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -7,14 +7,21 @@ import ( "time" logging "github.com/ipfs/go-log" - ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" + "github.com/libp2p/go-libp2p-interface-connmgr" inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) var log = logging.Logger("connmgr") +// BasicConnMgr is a ConnManager that trims connections whenever the count exceeds the +// high watermark. New connections are given a grace period before they're subject +// to trimming. Trims are automatically run on demand, only if the time from the +// previous trim is higher than 10 seconds. Furthermore, trims can be explicitly +// requested through the public interface of this struct (see TrimOpenConns). + +// See configuration parameters in NewConnManager. type BasicConnMgr struct { highWater int lowWater int @@ -31,6 +38,12 @@ type BasicConnMgr struct { var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) +// NewConnManager creates a new BasicConnMgr with the provided params: +// * lo and hi are watermarks governing the number of connections that'll be maintained. +// When the peer count exceeds the 'high watermark', as many peers will be pruned (and +// their connections terminated) until 'low watermark' peers remain. +// * grace is the amount of time a newly opened connection is given before it becomes +// subject to pruning. func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { return &BasicConnMgr{ highWater: hi, @@ -40,15 +53,20 @@ func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { } } +// peerInfo stores metadata for a given peer. type peerInfo struct { - tags map[string]int - value int + tags map[string]int // value for each tag + value int // cached sum of all tag values - conns map[inet.Conn]time.Time + conns map[inet.Conn]time.Time // start time of each connection - firstSeen time.Time + firstSeen time.Time // timestamp when we began tracking this peer. } +// TrimOpenConns closes the connections of as many peers as needed to make the peer count +// equal the low watermark. Peers are sorted in ascending order based on their total value, +// pruning those peers with the lowest scores first, as long as they are not within their +// grace period. func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { defer log.EventBegin(ctx, "connCleanup").Done() for _, c := range cm.getConnsToClose(ctx) { @@ -58,9 +76,12 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { } } +// getConnsToClose runs the heuristics described in TrimOpenConns and returns the +// connections to close. func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { cm.lk.Lock() defer cm.lk.Unlock() + if cm.lowWater == 0 || cm.highWater == 0 { // disabled return nil @@ -79,6 +100,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { infos = append(infos, inf) } + // Sort peers according to their value. sort.Slice(infos, func(i, j int) bool { return infos[i].value < infos[j].value }) @@ -143,6 +165,7 @@ func (cm *BasicConnMgr) TagPeer(p peer.ID, tag string, val int) { return } + // Update the total value of the peer. pi.value += (val - pi.tags[tag]) pi.tags[tag] = val } @@ -157,18 +180,30 @@ func (cm *BasicConnMgr) UntagPeer(p peer.ID, tag string) { return } + // Update the total value of the peer. pi.value -= pi.tags[tag] delete(pi.tags, tag) } +// CMInfo holds the configuration for BasicConnMgr, as well as status data. type CMInfo struct { - LowWater int - HighWater int - LastTrim time.Time + // The low watermark, as described in NewConnManager. + LowWater int + + // The high watermark, as described in NewConnManager. + HighWater int + + // The timestamp when the last trim was triggered. + LastTrim time.Time + + // The configured grace period, as described in NewConnManager. GracePeriod time.Duration - ConnCount int + + // The current connection count. + ConnCount int } +// GetInfo returns the configuration and status data for this connection manager. func (cm *BasicConnMgr) GetInfo() CMInfo { cm.lk.Lock() defer cm.lk.Unlock() @@ -182,6 +217,9 @@ func (cm *BasicConnMgr) GetInfo() CMInfo { } } +// Notifee returns a sink through which Notifiers can inform the BasicConnMgr when +// events occur. Currently, the notifee only reacts upon connection events +// {Connected, Disconnected}. func (cm *BasicConnMgr) Notifee() inet.Notifiee { return (*cmNotifee)(cm) } @@ -192,6 +230,9 @@ func (nn *cmNotifee) cm() *BasicConnMgr { return (*BasicConnMgr)(nn) } +// Connected is called by notifiers to inform that a new connection has been established. +// The notifee updates the BasicConnMgr to start tracking the connection. If the new connection +// count exceeds the high watermark, a trim may be triggered. func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { cm := nn.cm() @@ -224,6 +265,8 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { } } +// Disconnected is called by notifiers to inform that an existing connection has been closed or terminated. +// The notifee updates the BasicConnMgr accordingly to stop tracking the connection, and performs housekeeping. func (nn *cmNotifee) Disconnected(n inet.Network, c inet.Conn) { cm := nn.cm() @@ -249,7 +292,14 @@ func (nn *cmNotifee) Disconnected(n inet.Network, c inet.Conn) { } } +// Listen is no-op in this implementation. func (nn *cmNotifee) Listen(n inet.Network, addr ma.Multiaddr) {} + +// ListenClose is no-op in this implementation. func (nn *cmNotifee) ListenClose(n inet.Network, addr ma.Multiaddr) {} + +// OpenedStream is no-op in this implementation. func (nn *cmNotifee) OpenedStream(inet.Network, inet.Stream) {} + +// ClosedStream is no-op in this implementation. func (nn *cmNotifee) ClosedStream(inet.Network, inet.Stream) {} From 02481e7afdaa14e659266b9791aaea0b7fd02523 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rau=CC=81l=20Kripalani?= Date: Mon, 27 Aug 2018 15:38:09 +0100 Subject: [PATCH 0572/3965] gofmt. --- p2p/net/connmgr/connmgr.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 94eca7d445..56ca27ec59 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -293,13 +293,13 @@ func (nn *cmNotifee) Disconnected(n inet.Network, c inet.Conn) { } // Listen is no-op in this implementation. -func (nn *cmNotifee) Listen(n inet.Network, addr ma.Multiaddr) {} +func (nn *cmNotifee) Listen(n inet.Network, addr ma.Multiaddr) {} // ListenClose is no-op in this implementation. func (nn *cmNotifee) ListenClose(n inet.Network, addr ma.Multiaddr) {} // OpenedStream is no-op in this implementation. -func (nn *cmNotifee) OpenedStream(inet.Network, inet.Stream) {} +func (nn *cmNotifee) OpenedStream(inet.Network, inet.Stream) {} // ClosedStream is no-op in this implementation. -func (nn *cmNotifee) ClosedStream(inet.Network, inet.Stream) {} +func (nn *cmNotifee) ClosedStream(inet.Network, inet.Stream) {} From f6a0d2222b53bc1d53994aab71a60ffa708829b2 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 5 Jun 2018 02:18:06 -0400 Subject: [PATCH 0573/3965] Add simple benchmarking for peerstore --- p2p/host/peerstore/benchmark_utils.go | 74 +++++++++++++++++++++++++++ p2p/host/peerstore/peerstore_test.go | 37 ++++++++++++++ 2 files changed, 111 insertions(+) create mode 100644 p2p/host/peerstore/benchmark_utils.go diff --git a/p2p/host/peerstore/benchmark_utils.go b/p2p/host/peerstore/benchmark_utils.go new file mode 100644 index 0000000000..9e812f9c8d --- /dev/null +++ b/p2p/host/peerstore/benchmark_utils.go @@ -0,0 +1,74 @@ +package peerstore + +import ( + cr "crypto/rand" + "fmt" + "testing" + "time" + + "github.com/mr-tron/base58/base58" + ma "github.com/multiformats/go-multiaddr" + mh "github.com/multiformats/go-multihash" + "math/rand" +) + +type peerpair struct { + ID string + Addr ma.Multiaddr +} + +func randomPeer(b *testing.B) *peerpair { + buf := make([]byte, 50) + + for { + n, err := cr.Read(buf) + if err != nil { + b.Fatal(err) + } + if n > 0 { + break + } + } + + id, err := mh.Encode(buf, mh.SHA2_256) + if err != nil { + b.Fatal(err) + } + b58ID := base58.Encode(id) + + addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/6666/ipfs/%s", b58ID)) + if err != nil { + b.Fatal(err) + } + + return &peerpair{b58ID, addr} +} + +func addressProducer(b *testing.B, addrs chan *peerpair, n int) { + defer close(addrs) + for i := 0; i < n; i++ { + p := randomPeer(b) + addrs <- p + } +} + +func rateLimitedAddressProducer(b *testing.B, addrs chan *peerpair, producer chan *peerpair, n int, avgTime time.Duration, errBound time.Duration) { + defer close(addrs) + + go addressProducer(b, producer, n) + + eb := int64(errBound) + for { + addr, ok := <-producer + if !ok { + break + } + addrs <- addr + wiggle := time.Duration(0) + if eb > 0 { + wiggle = time.Duration(rand.Int63n(eb*2) - eb) + } + sleepDur := avgTime + wiggle + time.Sleep(sleepDur) + } +} diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 6c4d27cbaf..a1e650ee73 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -269,3 +269,40 @@ func TestBasicPeerstore(t *testing.T) { t.Fatal("stored wrong address") } } + +func BenchmarkBasicPeerstore(b *testing.B) { + ps := NewPeerstore() + + addrs := make(chan *peerpair, 100) + + go addressProducer(b, addrs, 50000) + + b.ResetTimer() + for { + pp, ok := <-addrs + if !ok { + break + } + pid := peer.ID(pp.ID) + ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) + } +} + +func BenchmarkBasicPeerstoreRateLimited(b *testing.B) { + ps := NewPeerstore() + + producer := make(chan *peerpair, 100) + addrs := make(chan *peerpair, 100) + + go rateLimitedAddressProducer(b, addrs, producer, 5000, time.Millisecond, 500 * time.Microsecond) + + b.ResetTimer() + for { + pp, ok := <-addrs + if !ok { + break + } + pid := peer.ID(pp.ID) + ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) + } +} \ No newline at end of file From 38775891bd70da1c492dae377530a9c4360cede9 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 5 Jun 2018 02:18:23 -0400 Subject: [PATCH 0574/3965] WIP: Begin work on a native badger peerstore --- p2p/host/peerstore/addrmanager_badger.go | 130 +++++++++++++++++++++++ p2p/host/peerstore/peerstore_badger.go | 91 ++++++++++++++++ 2 files changed, 221 insertions(+) create mode 100644 p2p/host/peerstore/addrmanager_badger.go create mode 100644 p2p/host/peerstore/peerstore_badger.go diff --git a/p2p/host/peerstore/addrmanager_badger.go b/p2p/host/peerstore/addrmanager_badger.go new file mode 100644 index 0000000000..55ebbecc55 --- /dev/null +++ b/p2p/host/peerstore/addrmanager_badger.go @@ -0,0 +1,130 @@ +package peerstore + +import ( + "context" + "time" + + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" + "github.com/hashicorp/golang-lru" + "github.com/dgraph-io/badger" + "github.com/multiformats/go-multihash" + "encoding/gob" + "bytes" + "encoding/binary" +) + +type addrmanager_badger struct { + cache *lru.Cache + db *badger.DB +} + +type addrentry struct { + Addr []byte + TTL time.Duration +} + +func (mgr *addrmanager_badger) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) +} + +// use murmur3 because it's the most compact +func hashMultiaddr(addr *ma.Multiaddr) ([]byte, error) { + return multihash.Encode((*addr).Bytes(), multihash.MURMUR3) +} + +// not relevant w/ key prefixing +func createAddrEntry(addr ma.Multiaddr, ttl time.Duration) ([]byte, error) { + entry := addrentry{Addr: addr.Bytes(), TTL: ttl} + buf := bytes.Buffer{} + enc := gob.NewEncoder(&buf) + if err := enc.Encode(entry); err != nil { + return nil, err + } + + return buf.Bytes(), nil +} + +func createKeyPrefix(p *peer.ID, ttl time.Duration) ([]byte, error) { + buf := bytes.NewBufferString(string(*p)) + if err := binary.Write(buf, binary.LittleEndian, ttl); err != nil { + return nil, err + } + return buf.Bytes(), nil +} + +func createUniqueKey(p *peer.ID, addr *ma.Multiaddr, ttl time.Duration) ([]byte, error) { + prefix, err := createKeyPrefix(p, ttl) + if err != nil { + return nil, err + } + addrHash, err := hashMultiaddr(addr) + if err != nil { + return nil, err + } + + return append(prefix, addrHash...), nil +} + +func (mgr *addrmanager_badger) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + // if ttl is zero, exit. nothing to do. + if ttl <= 0 { + log.Debugf("short circuiting AddAddrs with ttl %d", ttl) + return + } + + txn := mgr.db.NewTransaction(true) + defer txn.Discard() + + for _, addr := range addrs { + key, err := createUniqueKey(&p, &addr, ttl) + if err != nil { + log.Error(err) + return + } + txn.SetWithTTL(key, addr.Bytes(), ttl) + } + + txn.Commit(func (err error) { + log.Error(err) + }) +} + +func (mgr *addrmanager_badger) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.AddAddr(p, addr, ttl) +} + +func (mgr *addrmanager_badger) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + mgr.AddAddrs(p, addrs, ttl) +} + +func (mgr *addrmanager_badger) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { + prefix, err := createKeyPrefix(&p, oldTTL) + if err != nil { + log.Error(err) + } + txn := mgr.db.NewTransaction(true) + opts := badger.DefaultIteratorOptions + iter := txn.NewIterator(opts) + + iter.Seek(prefix) + for iter.Valid() && iter.ValidForPrefix(prefix) { + item := iter.Item() + // TODO: + + iter.Next() + } + +} + +func (mgr *addrmanager_badger) Addrs(p peer.ID) []ma.Multiaddr { + panic("implement me") +} + +func (mgr *addrmanager_badger) AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr { + panic("implement me") +} + +func (mgr *addrmanager_badger) ClearAddrs(p peer.ID) { + panic("implement me") +} diff --git a/p2p/host/peerstore/peerstore_badger.go b/p2p/host/peerstore/peerstore_badger.go new file mode 100644 index 0000000000..49dad24c38 --- /dev/null +++ b/p2p/host/peerstore/peerstore_badger.go @@ -0,0 +1,91 @@ +package peerstore + +import ( + "github.com/dgraph-io/badger" + "github.com/libp2p/go-libp2p-crypto" + "time" + "github.com/libp2p/go-libp2p-peer" +) + +type keybook_badger struct { + +} + +func (kb *keybook_badger) PubKey(peer.ID) crypto.PubKey { + panic("implement me") +} + +func (kb *keybook_badger) AddPubKey(peer.ID, crypto.PubKey) error { + panic("implement me") +} + +func (kb *keybook_badger) PrivKey(peer.ID) crypto.PrivKey { + panic("implement me") +} + +func (kb *keybook_badger) AddPrivKey(peer.ID, crypto.PrivKey) error { + panic("implement me") +} + +type peerstore_badger struct { + *addrmanager_badger + *keybook_badger + *badger.DB +} + +func (store *peerstore_badger) PubKey(peer.ID) crypto.PubKey { + panic("implement me") +} + +func (store *peerstore_badger) AddPubKey(peer.ID, crypto.PubKey) error { + panic("implement me") +} + +func (store *peerstore_badger) PrivKey(peer.ID) crypto.PrivKey { + panic("implement me") +} + +func (store *peerstore_badger) AddPrivKey(peer.ID, crypto.PrivKey) error { + panic("implement me") +} + +func (store *peerstore_badger) RecordLatency(peer.ID, time.Duration) { + panic("implement me") +} + +func (store *peerstore_badger) LatencyEWMA(peer.ID) time.Duration { + panic("implement me") +} + +func (store *peerstore_badger) Peers() []peer.ID { + panic("implement me") +} + +func (store *peerstore_badger) PeerInfo(peer.ID) PeerInfo { + panic("implement me") +} + +func (store *peerstore_badger) Get(id peer.ID, key string) (interface{}, error) { + panic("implement me") +} + +func (store *peerstore_badger) Put(id peer.ID, key string, val interface{}) error { + panic("implement me") +} + +func (store *peerstore_badger) GetProtocols(peer.ID) ([]string, error) { + panic("implement me") +} + +func (store *peerstore_badger) AddProtocols(peer.ID, ...string) error { + panic("implement me") +} + +func (store *peerstore_badger) SetProtocols(peer.ID, ...string) error { + panic("implement me") +} + +func (store *peerstore_badger) SupportsProtocols(peer.ID, ...string) ([]string, error) { + panic("implement me") +} + From ae8fbb92deac180064bd6fc4a7ad9c60215983e6 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 5 Jun 2018 16:54:20 -0400 Subject: [PATCH 0575/3965] Finish initial pass (sans stream) of addr manager --- ...nager_badger.go => addr_manager_badger.go} | 111 +++++++++++++++--- 1 file changed, 95 insertions(+), 16 deletions(-) rename p2p/host/peerstore/{addrmanager_badger.go => addr_manager_badger.go} (61%) diff --git a/p2p/host/peerstore/addrmanager_badger.go b/p2p/host/peerstore/addr_manager_badger.go similarity index 61% rename from p2p/host/peerstore/addrmanager_badger.go rename to p2p/host/peerstore/addr_manager_badger.go index 55ebbecc55..03efa909eb 100644 --- a/p2p/host/peerstore/addrmanager_badger.go +++ b/p2p/host/peerstore/addr_manager_badger.go @@ -6,7 +6,7 @@ import ( "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" - "github.com/hashicorp/golang-lru" + //"github.com/hashicorp/golang-lru" "github.com/dgraph-io/badger" "github.com/multiformats/go-multihash" "encoding/gob" @@ -15,7 +15,7 @@ import ( ) type addrmanager_badger struct { - cache *lru.Cache + //cache *lru.Cache db *badger.DB } @@ -24,6 +24,14 @@ type addrentry struct { TTL time.Duration } +func NewBadgerAddrManager() *addrmanager_badger { + db, err := badger.Open(badger.DefaultOptions) + if err != nil { + panic(err) + } + return &addrmanager_badger{db: db} +} + func (mgr *addrmanager_badger) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } @@ -66,6 +74,18 @@ func createUniqueKey(p *peer.ID, addr *ma.Multiaddr, ttl time.Duration) ([]byte, return append(prefix, addrHash...), nil } +func addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, txn *badger.Txn) { + for _, addr := range addrs { + key, err := createUniqueKey(&p, &addr, ttl) + if err != nil { + log.Error(err) + txn.Discard() + return + } + txn.SetWithTTL(key, addr.Bytes(), ttl) + } +} + func (mgr *addrmanager_badger) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { // if ttl is zero, exit. nothing to do. if ttl <= 0 { @@ -76,17 +96,12 @@ func (mgr *addrmanager_badger) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl tim txn := mgr.db.NewTransaction(true) defer txn.Discard() - for _, addr := range addrs { - key, err := createUniqueKey(&p, &addr, ttl) + addAddrs(p, addrs, ttl, txn) + + txn.Commit(func (err error) { if err != nil { log.Error(err) - return } - txn.SetWithTTL(key, addr.Bytes(), ttl) - } - - txn.Commit(func (err error) { - log.Error(err) }) } @@ -102,29 +117,93 @@ func (mgr *addrmanager_badger) UpdateAddrs(p peer.ID, oldTTL time.Duration, newT prefix, err := createKeyPrefix(&p, oldTTL) if err != nil { log.Error(err) + return } txn := mgr.db.NewTransaction(true) + defer txn.Discard() opts := badger.DefaultIteratorOptions iter := txn.NewIterator(opts) + var addrs []ma.Multiaddr iter.Seek(prefix) - for iter.Valid() && iter.ValidForPrefix(prefix) { + for iter.ValidForPrefix(prefix) { item := iter.Item() - // TODO: + addrbytes, err := item.Value() + if err != nil { + log.Error(err) + return + } + addrs = append(addrs, ma.Cast(addrbytes)) iter.Next() } + addAddrs(p, addrs, newTTL, txn) + + txn.Commit(func (err error) { + if err != nil { + log.Error(err) + } + }) } func (mgr *addrmanager_badger) Addrs(p peer.ID) []ma.Multiaddr { - panic("implement me") + txn := mgr.db.NewTransaction(false) + defer txn.Discard() + + prefix := []byte(p) + opts := badger.DefaultIteratorOptions + iter := txn.NewIterator(opts) + iter.Seek(prefix) + + var addrs []ma.Multiaddr + + for iter.ValidForPrefix(prefix) { + item := iter.Item() + + if !item.IsDeletedOrExpired() { + value, err := item.Value() + if err != nil { + log.Error(err) + } else { + addrs = append(addrs, ma.Cast(value)) + } + } + + iter.Next() + } + + txn.Commit(nil) + + return addrs } -func (mgr *addrmanager_badger) AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr { - panic("implement me") +func (mgr *addrmanager_badger) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { + addrs := make(chan ma.Multiaddr) + + //mgr.db.View(func (txn *badger.Txn) error { + // defer close(addrs) + // + // return nil + //}) + + return addrs } func (mgr *addrmanager_badger) ClearAddrs(p peer.ID) { - panic("implement me") + err := mgr.db.Update(func (txn *badger.Txn) error { + iter := txn.NewIterator(badger.DefaultIteratorOptions) + prefix := []byte(p) + iter.Seek(prefix) + + for iter.ValidForPrefix(prefix) { + txn.Delete(iter.Item().Key()) + iter.Next() + } + + return nil + }) + if err != nil { + log.Error(err) + } } From 89713eb5a8fcf02d14d90c9f97fe692aca6e878a Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 8 Jun 2018 13:26:53 -0400 Subject: [PATCH 0576/3965] Implement badger backed addr manager, add to tests --- p2p/host/peerstore/addr_manager_badger.go | 212 +++++++++++++--------- p2p/host/peerstore/addr_manager_test.go | 116 +++++++++--- 2 files changed, 214 insertions(+), 114 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_badger.go b/p2p/host/peerstore/addr_manager_badger.go index 03efa909eb..e118066eb7 100644 --- a/p2p/host/peerstore/addr_manager_badger.go +++ b/p2p/host/peerstore/addr_manager_badger.go @@ -6,17 +6,15 @@ import ( "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" - //"github.com/hashicorp/golang-lru" "github.com/dgraph-io/badger" "github.com/multiformats/go-multihash" "encoding/gob" "bytes" - "encoding/binary" ) -type addrmanager_badger struct { - //cache *lru.Cache - db *badger.DB +type AddrManagerBadger struct { + DB *badger.DB + addrSubs map[peer.ID][]*addrSub } type addrentry struct { @@ -24,48 +22,43 @@ type addrentry struct { TTL time.Duration } -func NewBadgerAddrManager() *addrmanager_badger { - db, err := badger.Open(badger.DefaultOptions) - if err != nil { - panic(err) +func (mgr *AddrManagerBadger) sendSubscriptionUpdates(p *peer.ID, addrs []ma.Multiaddr) { + subs := mgr.addrSubs[*p] + for _, sub := range subs { + for _, addr := range addrs { + sub.pubAddr(addr) + } } - return &addrmanager_badger{db: db} -} - -func (mgr *addrmanager_badger) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } -// use murmur3 because it's the most compact -func hashMultiaddr(addr *ma.Multiaddr) ([]byte, error) { - return multihash.Encode((*addr).Bytes(), multihash.MURMUR3) +func (mgr *AddrManagerBadger) Close() { + if err := mgr.DB.Close(); err != nil { + log.Error(err) + } } -// not relevant w/ key prefixing -func createAddrEntry(addr ma.Multiaddr, ttl time.Duration) ([]byte, error) { - entry := addrentry{Addr: addr.Bytes(), TTL: ttl} - buf := bytes.Buffer{} - enc := gob.NewEncoder(&buf) - if err := enc.Encode(entry); err != nil { +func NewBadgerAddrManager(dataPath string) (*AddrManagerBadger, error) { + opts := badger.DefaultOptions + opts.Dir = dataPath + opts.ValueDir = dataPath + db, err := badger.Open(opts) + if err != nil { return nil, err } + return &AddrManagerBadger{DB: db}, nil +} - return buf.Bytes(), nil +func (mgr *AddrManagerBadger) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } -func createKeyPrefix(p *peer.ID, ttl time.Duration) ([]byte, error) { - buf := bytes.NewBufferString(string(*p)) - if err := binary.Write(buf, binary.LittleEndian, ttl); err != nil { - return nil, err - } - return buf.Bytes(), nil +// use murmur3 because it's the most compact +func hashMultiaddr(addr *ma.Multiaddr) ([]byte, error) { + return multihash.Encode((*addr).Bytes(), multihash.MURMUR3) } -func createUniqueKey(p *peer.ID, addr *ma.Multiaddr, ttl time.Duration) ([]byte, error) { - prefix, err := createKeyPrefix(p, ttl) - if err != nil { - return nil, err - } +func createUniqueKey(p *peer.ID, addr *ma.Multiaddr) ([]byte, error) { + prefix := []byte(*p) addrHash, err := hashMultiaddr(addr) if err != nil { return nil, err @@ -76,55 +69,86 @@ func createUniqueKey(p *peer.ID, addr *ma.Multiaddr, ttl time.Duration) ([]byte, func addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, txn *badger.Txn) { for _, addr := range addrs { - key, err := createUniqueKey(&p, &addr, ttl) + if addr == nil { + continue + } + entry := &addrentry{Addr: addr.Bytes(), TTL: ttl} + key, err := createUniqueKey(&p, &addr) if err != nil { log.Error(err) txn.Discard() return } - txn.SetWithTTL(key, addr.Bytes(), ttl) + buf := &bytes.Buffer{} + enc := gob.NewEncoder(buf) + if err := enc.Encode(entry); err != nil { + log.Error(err) + txn.Discard() + return + } + txn.SetWithTTL(key, buf.Bytes(), ttl) } } -func (mgr *addrmanager_badger) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - // if ttl is zero, exit. nothing to do. +func (mgr *AddrManagerBadger) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { if ttl <= 0 { log.Debugf("short circuiting AddAddrs with ttl %d", ttl) return } - txn := mgr.db.NewTransaction(true) + txn := mgr.DB.NewTransaction(true) defer txn.Discard() + + go mgr.sendSubscriptionUpdates(&p, addrs) addAddrs(p, addrs, ttl, txn) - txn.Commit(func (err error) { - if err != nil { - log.Error(err) - } - }) + txn.Commit(nil) } -func (mgr *addrmanager_badger) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.AddAddr(p, addr, ttl) +func (mgr *AddrManagerBadger) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } -func (mgr *addrmanager_badger) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - mgr.AddAddrs(p, addrs, ttl) -} +func (mgr *AddrManagerBadger) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + txn := mgr.DB.NewTransaction(true) + defer txn.Discard() -func (mgr *addrmanager_badger) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { - prefix, err := createKeyPrefix(&p, oldTTL) - if err != nil { - log.Error(err) - return + for _, addr := range addrs { + if addr == nil { + continue + } + key, err := createUniqueKey(&p, &addr) + if err != nil { + log.Error(err) + continue + } + if ttl <= 0 { + if err := txn.Delete(key); err != nil { + log.Error(err) + } + } else { + entry := &addrentry{Addr: addr.Bytes(), TTL: ttl} + buf := &bytes.Buffer{} + enc := gob.NewEncoder(buf) + if err := enc.Encode(entry); err != nil { + log.Error(err) + continue + } + txn.SetWithTTL(key, buf.Bytes(), ttl) + } } - txn := mgr.db.NewTransaction(true) + + txn.Commit(nil) +} + +func (mgr *AddrManagerBadger) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { + prefix := []byte(p) + txn := mgr.DB.NewTransaction(true) defer txn.Discard() opts := badger.DefaultIteratorOptions iter := txn.NewIterator(opts) - var addrs []ma.Multiaddr iter.Seek(prefix) for iter.ValidForPrefix(prefix) { item := iter.Item() @@ -133,22 +157,32 @@ func (mgr *addrmanager_badger) UpdateAddrs(p peer.ID, oldTTL time.Duration, newT log.Error(err) return } - addrs = append(addrs, ma.Cast(addrbytes)) + entry := &addrentry{} + buf := bytes.NewBuffer(addrbytes) + dec := gob.NewDecoder(buf) + if err := dec.Decode(&entry); err != nil { + log.Error(err) + return + } + if entry.TTL == oldTTL { + entry.TTL = newTTL + buf := &bytes.Buffer{} + enc := gob.NewEncoder(buf) + if err := enc.Encode(&entry); err != nil { + log.Error(err) + return + } + txn.SetWithTTL(item.Key(), buf.Bytes(), newTTL) + } iter.Next() } - addAddrs(p, addrs, newTTL, txn) - - txn.Commit(func (err error) { - if err != nil { - log.Error(err) - } - }) + txn.Commit(nil) } -func (mgr *addrmanager_badger) Addrs(p peer.ID) []ma.Multiaddr { - txn := mgr.db.NewTransaction(false) +func (mgr *AddrManagerBadger) Addrs(p peer.ID) []ma.Multiaddr { + txn := mgr.DB.NewTransaction(false) defer txn.Discard() prefix := []byte(p) @@ -166,7 +200,15 @@ func (mgr *addrmanager_badger) Addrs(p peer.ID) []ma.Multiaddr { if err != nil { log.Error(err) } else { - addrs = append(addrs, ma.Cast(value)) + entry := &addrentry{} + buf := bytes.NewBuffer(value) + dec := gob.NewDecoder(buf) + if err := dec.Decode(&entry); err != nil { + log.Error("deleting bad entry in peerstore for peer", p.String()) + txn.Delete(item.Key()) + } else { + addrs = append(addrs, ma.Cast(entry.Addr)) + } } } @@ -178,32 +220,28 @@ func (mgr *addrmanager_badger) Addrs(p peer.ID) []ma.Multiaddr { return addrs } -func (mgr *addrmanager_badger) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { +func (mgr *AddrManagerBadger) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { addrs := make(chan ma.Multiaddr) - //mgr.db.View(func (txn *badger.Txn) error { - // defer close(addrs) - // - // return nil - //}) + // TODO: impl return addrs } -func (mgr *addrmanager_badger) ClearAddrs(p peer.ID) { - err := mgr.db.Update(func (txn *badger.Txn) error { - iter := txn.NewIterator(badger.DefaultIteratorOptions) - prefix := []byte(p) - iter.Seek(prefix) +func (mgr *AddrManagerBadger) ClearAddrs(p peer.ID) { + txn := mgr.DB.NewTransaction(true) + defer txn.Discard() + it := txn.NewIterator(badger.DefaultIteratorOptions) + prefix := []byte(p) + it.Seek(prefix) - for iter.ValidForPrefix(prefix) { - txn.Delete(iter.Item().Key()) - iter.Next() + count := 0 + for it.ValidForPrefix(prefix) { + count++ + if err := txn.Delete(it.Item().Key()); err != nil { + log.Error(err) } - - return nil - }) - if err != nil { - log.Error(err) + it.Next() } + txn.Commit(nil) } diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 9ade2c5950..e41e9655a8 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -6,6 +6,7 @@ import ( "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" + "os" ) func IDS(t *testing.T, ids string) peer.ID { @@ -29,7 +30,7 @@ func MA(t *testing.T, m string) ma.Multiaddr { func testHas(t *testing.T, exp, act []ma.Multiaddr) { t.Helper() if len(exp) != len(act) { - t.Fatal("lengths not the same") + t.Fatalf("lengths not the same. expected %d, got %d\n", len(exp), len(act)) } for _, a := range exp { @@ -48,8 +49,7 @@ func testHas(t *testing.T, exp, act []ma.Multiaddr) { } } -func TestAddresses(t *testing.T) { - +func testAddresses(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") id2 := IDS(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") id3 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ6Kn") @@ -73,7 +73,6 @@ func TestAddresses(t *testing.T) { ma55 := MA(t, "/ip4/5.2.3.3/tcp/5555") ttl := time.Hour - m := AddrManager{} m.AddAddr(id1, ma11, ttl) m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) @@ -92,21 +91,40 @@ func TestAddresses(t *testing.T) { m.ClearAddrs(id5) m.AddAddrs(id5, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, ttl) // clearing - if len(m.Peers()) != 5 { - t.Fatal("should have exactly two peers in the address book") - } - // test the Addresses return value testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) testHas(t, []ma.Multiaddr{ma31, ma32, ma33}, m.Addrs(id3)) testHas(t, []ma.Multiaddr{ma41, ma42, ma43, ma44}, m.Addrs(id4)) testHas(t, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, m.Addrs(id5)) +} +func setupBadgerAddrManager(t *testing.T) (*AddrManagerBadger, func ()) { + dataPath := os.TempDir() + mgr, err := NewBadgerAddrManager(dataPath) + if err != nil { + t.Fatal(err) + } + closer := func () { + mgr.Close() + os.RemoveAll(dataPath) + } + return mgr, closer } -func TestAddressesExpire(t *testing.T) { +func TestAddresses(t *testing.T) { + t.Log("AddrManager") + mgr1 := AddrManager{} + testAddresses(t, &mgr1) + + t.Log("AddrManager") + + mgr2, closer := setupBadgerAddrManager(t) + defer closer() + testAddresses(t, mgr2) +} +func testAddressesExpire(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") @@ -115,17 +133,12 @@ func TestAddressesExpire(t *testing.T) { ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") - m := AddrManager{} m.AddAddr(id1, ma11, time.Hour) m.AddAddr(id1, ma12, time.Hour) m.AddAddr(id1, ma13, time.Hour) m.AddAddr(id2, ma24, time.Hour) m.AddAddr(id2, ma25, time.Hour) - if len(m.Peers()) != 2 { - t.Fatal("should have exactly two peers in the address book") - } - testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) @@ -164,8 +177,18 @@ func TestAddressesExpire(t *testing.T) { testHas(t, nil, m.Addrs(id2)) } -func TestClearWorks(t *testing.T) { +func TestAddressesExpire(t *testing.T) { + t.Log("AddrManager") + m1 := &AddrManager{} + testAddressesExpire(t, m1) + + t.Log("AddrManagerBadger") + m2, closer := setupBadgerAddrManager(t) + defer closer() + testAddressesExpire(t, m2) +} +func testClearWorks(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") @@ -174,7 +197,6 @@ func TestClearWorks(t *testing.T) { ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") - m := AddrManager{} m.AddAddr(id1, ma11, time.Hour) m.AddAddr(id1, ma12, time.Hour) m.AddAddr(id1, ma13, time.Hour) @@ -191,11 +213,21 @@ func TestClearWorks(t *testing.T) { testHas(t, nil, m.Addrs(id2)) } -func TestSetNegativeTTLClears(t *testing.T) { +func TestClearWorks(t *testing.T) { + t.Log("AddrManager") + m1 := &AddrManager{} + testClearWorks(t, m1) + + t.Log("AddrManagerBadger") + m2, closer := setupBadgerAddrManager(t) + defer closer() + testClearWorks(t, m2) +} + +func testSetNegativeTTLClears(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - m := AddrManager{} m.SetAddr(id1, ma11, time.Hour) testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) @@ -204,8 +236,18 @@ func TestSetNegativeTTLClears(t *testing.T) { testHas(t, nil, m.Addrs(id1)) } +func TestSetNegativeTTLClears(t *testing.T) { + t.Log("AddrManager") + m1 := &AddrManager{} + testSetNegativeTTLClears(t, m1) + + t.Log("AddrManagerBadger") + m2, closer := setupBadgerAddrManager(t) + defer closer() + testSetNegativeTTLClears(t, m2) +} -func TestUpdateTTLs(t *testing.T) { +func testUpdateTTLs(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") @@ -213,8 +255,6 @@ func TestUpdateTTLs(t *testing.T) { ma21 := MA(t, "/ip4/1.2.3.1/tcp/1121") ma22 := MA(t, "/ip4/1.2.3.1/tcp/1122") - m := AddrManager{} - // Shouldn't panic. m.UpdateAddrs(id1, time.Hour, time.Minute) @@ -230,30 +270,52 @@ func TestUpdateTTLs(t *testing.T) { testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - m.UpdateAddrs(id1, time.Hour, time.Millisecond) + m.UpdateAddrs(id1, time.Hour, time.Second) testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - time.Sleep(time.Millisecond) + time.Sleep(time.Second) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - m.UpdateAddrs(id2, time.Hour, time.Millisecond) + m.UpdateAddrs(id2, time.Hour, time.Second) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - time.Sleep(time.Millisecond) + time.Sleep(time.Second) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma22}, m.Addrs(id2)) } -func TestNilAddrsDontBreak(t *testing.T) { +func TestUpdateTTLs(t *testing.T) { + t.Log("AddrManager") + m1 := &AddrManager{} + testUpdateTTLs(t, m1) + + t.Log("AddrManagerBadger") + m2, closer := setupBadgerAddrManager(t) + defer closer() + testUpdateTTLs(t, m2) +} + +func testNilAddrsDontBreak(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - m := AddrManager{} m.SetAddr(id1, nil, time.Hour) m.AddAddr(id1, nil, time.Hour) } + +func TestNilAddrsDontBreak(t *testing.T) { + t.Log("AddrManager") + m1 := &AddrManager{} + testNilAddrsDontBreak(t, m1) + t.Log("OK") + t.Log("AddrManagerBadger") + m2, closer := setupBadgerAddrManager(t) + defer closer() + testNilAddrsDontBreak(t, m2) + t.Log("OK") +} From 61a8c212e76170643903b165932d8312d0ce606c Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 8 Jun 2018 13:56:43 -0400 Subject: [PATCH 0577/3965] Implement naive benchmark --- p2p/host/peerstore/peerstore_badger.go | 159 ++++++++++++++++--------- p2p/host/peerstore/peerstore_test.go | 51 +++++++- 2 files changed, 153 insertions(+), 57 deletions(-) diff --git a/p2p/host/peerstore/peerstore_badger.go b/p2p/host/peerstore/peerstore_badger.go index 49dad24c38..55b585e415 100644 --- a/p2p/host/peerstore/peerstore_badger.go +++ b/p2p/host/peerstore/peerstore_badger.go @@ -1,91 +1,140 @@ package peerstore import ( - "github.com/dgraph-io/badger" - "github.com/libp2p/go-libp2p-crypto" - "time" + "fmt" + "sync" + "github.com/libp2p/go-libp2p-peer" ) -type keybook_badger struct { +type PeerstoreBadger struct { + *keybook + *metrics + *AddrManagerBadger -} + ds map[string]interface{} + dslock sync.Mutex -func (kb *keybook_badger) PubKey(peer.ID) crypto.PubKey { - panic("implement me") + // lock for protocol information, separate from datastore lock + protolock sync.Mutex } -func (kb *keybook_badger) AddPubKey(peer.ID, crypto.PubKey) error { - panic("implement me") +func NewPeerstoreBadger(dataPath string) *PeerstoreBadger { + mgr, err := NewBadgerAddrManager(dataPath) + if err != nil { + panic(err) + } + return &PeerstoreBadger{ + keybook: newKeybook(), + metrics: NewMetrics(), + AddrManagerBadger: mgr, + ds: make(map[string]interface{}), + } } -func (kb *keybook_badger) PrivKey(peer.ID) crypto.PrivKey { - panic("implement me") +func (ps *PeerstoreBadger) Close() error { + ps.AddrManagerBadger.Close() + return nil } -func (kb *keybook_badger) AddPrivKey(peer.ID, crypto.PrivKey) error { - panic("implement me") +func (ps *PeerstoreBadger) PeerInfo(p peer.ID) PeerInfo { + return PeerInfo{ + ID: p, + Addrs: ps.AddrManagerBadger.Addrs(p), + } } -type peerstore_badger struct { - *addrmanager_badger - *keybook_badger - *badger.DB +func (ps *PeerstoreBadger) Get(p peer.ID, key string) (interface{}, error) { + ps.dslock.Lock() + defer ps.dslock.Unlock() + i, ok := ps.ds[string(p)+"/"+key] + if !ok { + return nil, ErrNotFound + } + return i, nil } -func (store *peerstore_badger) PubKey(peer.ID) crypto.PubKey { - panic("implement me") +func (ps *PeerstoreBadger) Put(p peer.ID, key string, val interface{}) error { + ps.dslock.Lock() + defer ps.dslock.Unlock() + ps.ds[string(p)+"/"+key] = val + return nil } -func (store *peerstore_badger) AddPubKey(peer.ID, crypto.PubKey) error { - panic("implement me") -} +func (ps *PeerstoreBadger) getProtocolMap(p peer.ID) (map[string]struct{}, error) { + iprotomap, err := ps.Get(p, "protocols") + switch err { + default: + return nil, err + case ErrNotFound: + return make(map[string]struct{}), nil + case nil: + cast, ok := iprotomap.(map[string]struct{}) + if !ok { + return nil, fmt.Errorf("stored protocol set was not a map") + } -func (store *peerstore_badger) PrivKey(peer.ID) crypto.PrivKey { - panic("implement me") + return cast, nil + } } -func (store *peerstore_badger) AddPrivKey(peer.ID, crypto.PrivKey) error { - panic("implement me") -} +func (ps *PeerstoreBadger) GetProtocols(p peer.ID) ([]string, error) { + ps.protolock.Lock() + defer ps.protolock.Unlock() + pmap, err := ps.getProtocolMap(p) + if err != nil { + return nil, err + } -func (store *peerstore_badger) RecordLatency(peer.ID, time.Duration) { - panic("implement me") -} + var out []string + for k, _ := range pmap { + out = append(out, k) + } -func (store *peerstore_badger) LatencyEWMA(peer.ID) time.Duration { - panic("implement me") + return out, nil } -func (store *peerstore_badger) Peers() []peer.ID { - panic("implement me") -} +func (ps *PeerstoreBadger) AddProtocols(p peer.ID, protos ...string) error { + ps.protolock.Lock() + defer ps.protolock.Unlock() + protomap, err := ps.getProtocolMap(p) + if err != nil { + return err + } -func (store *peerstore_badger) PeerInfo(peer.ID) PeerInfo { - panic("implement me") -} + for _, proto := range protos { + protomap[proto] = struct{}{} + } -func (store *peerstore_badger) Get(id peer.ID, key string) (interface{}, error) { - panic("implement me") + return ps.Put(p, "protocols", protomap) } -func (store *peerstore_badger) Put(id peer.ID, key string, val interface{}) error { - panic("implement me") -} +func (ps *PeerstoreBadger) SetProtocols(p peer.ID, protos ...string) error { + ps.protolock.Lock() + defer ps.protolock.Unlock() -func (store *peerstore_badger) GetProtocols(peer.ID) ([]string, error) { - panic("implement me") -} + protomap := make(map[string]struct{}) + for _, proto := range protos { + protomap[proto] = struct{}{} + } -func (store *peerstore_badger) AddProtocols(peer.ID, ...string) error { - panic("implement me") -} + return ps.Put(p, "protocols", protomap) +} + +func (ps *PeerstoreBadger) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { + ps.protolock.Lock() + defer ps.protolock.Unlock() + pmap, err := ps.getProtocolMap(p) + if err != nil { + return nil, err + } -func (store *peerstore_badger) SetProtocols(peer.ID, ...string) error { - panic("implement me") -} + var out []string + for _, proto := range protos { + if _, ok := pmap[proto]; ok { + out = append(out, proto) + } + } -func (store *peerstore_badger) SupportsProtocols(peer.ID, ...string) ([]string, error) { - panic("implement me") + return out, nil } - diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index a1e650ee73..55a8e901a1 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -8,6 +8,8 @@ import ( "testing" "time" + "os" + peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) @@ -277,6 +279,27 @@ func BenchmarkBasicPeerstore(b *testing.B) { go addressProducer(b, addrs, 50000) + for { + pp, ok := <-addrs + if !ok { + break + } + pid := peer.ID(pp.ID) + ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) + } +} + +func BenchmarkBadgerPeerstore(b *testing.B) { + dataPath := os.TempDir() + ps := NewPeerstoreBadger(dataPath) + defer func() { + ps.Close() + os.RemoveAll(dataPath) + }() + + addrs := make(chan *peerpair, 100) + go addressProducer(b, addrs, 50000) + b.ResetTimer() for { pp, ok := <-addrs @@ -294,7 +317,31 @@ func BenchmarkBasicPeerstoreRateLimited(b *testing.B) { producer := make(chan *peerpair, 100) addrs := make(chan *peerpair, 100) - go rateLimitedAddressProducer(b, addrs, producer, 5000, time.Millisecond, 500 * time.Microsecond) + go rateLimitedAddressProducer(b, addrs, producer, 5000, time.Millisecond, 500*time.Microsecond) + + b.ResetTimer() + for { + pp, ok := <-addrs + if !ok { + break + } + pid := peer.ID(pp.ID) + ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) + } +} + +func BenchmarkBadgerPeerstoreRateLimited(b *testing.B) { + dataPath := os.TempDir() + ps := NewPeerstoreBadger(dataPath) + defer func() { + ps.Close() + os.RemoveAll(dataPath) + }() + + producer := make(chan *peerpair, 100) + addrs := make(chan *peerpair, 100) + + go rateLimitedAddressProducer(b, addrs, producer, 5000, time.Millisecond, 500*time.Microsecond) b.ResetTimer() for { @@ -305,4 +352,4 @@ func BenchmarkBasicPeerstoreRateLimited(b *testing.B) { pid := peer.ID(pp.ID) ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) } -} \ No newline at end of file +} From babbd1d476d72196dca53f260f39ea2e8c63c92d Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Mon, 11 Jun 2018 17:56:57 -0400 Subject: [PATCH 0578/3965] Rename AddrManagerBadger -> BadgerAddrManager --- p2p/host/peerstore/addr_manager_badger.go | 26 +++---- p2p/host/peerstore/addr_manager_test.go | 82 ++++++++++++----------- p2p/host/peerstore/peerstore_badger.go | 8 +-- 3 files changed, 60 insertions(+), 56 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_badger.go b/p2p/host/peerstore/addr_manager_badger.go index e118066eb7..0a612d6804 100644 --- a/p2p/host/peerstore/addr_manager_badger.go +++ b/p2p/host/peerstore/addr_manager_badger.go @@ -12,7 +12,7 @@ import ( "bytes" ) -type AddrManagerBadger struct { +type BadgerAddrManager struct { DB *badger.DB addrSubs map[peer.ID][]*addrSub } @@ -22,7 +22,7 @@ type addrentry struct { TTL time.Duration } -func (mgr *AddrManagerBadger) sendSubscriptionUpdates(p *peer.ID, addrs []ma.Multiaddr) { +func (mgr *BadgerAddrManager) sendSubscriptionUpdates(p *peer.ID, addrs []ma.Multiaddr) { subs := mgr.addrSubs[*p] for _, sub := range subs { for _, addr := range addrs { @@ -31,13 +31,13 @@ func (mgr *AddrManagerBadger) sendSubscriptionUpdates(p *peer.ID, addrs []ma.Mul } } -func (mgr *AddrManagerBadger) Close() { +func (mgr *BadgerAddrManager) Close() { if err := mgr.DB.Close(); err != nil { log.Error(err) } } -func NewBadgerAddrManager(dataPath string) (*AddrManagerBadger, error) { +func NewBadgerAddrManager(dataPath string) (*BadgerAddrManager, error) { opts := badger.DefaultOptions opts.Dir = dataPath opts.ValueDir = dataPath @@ -45,10 +45,10 @@ func NewBadgerAddrManager(dataPath string) (*AddrManagerBadger, error) { if err != nil { return nil, err } - return &AddrManagerBadger{DB: db}, nil + return &BadgerAddrManager{DB: db}, nil } -func (mgr *AddrManagerBadger) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { +func (mgr *BadgerAddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } @@ -90,7 +90,7 @@ func addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, txn *badger.Tx } } -func (mgr *AddrManagerBadger) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { +func (mgr *BadgerAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { if ttl <= 0 { log.Debugf("short circuiting AddAddrs with ttl %d", ttl) return @@ -106,11 +106,11 @@ func (mgr *AddrManagerBadger) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time txn.Commit(nil) } -func (mgr *AddrManagerBadger) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { +func (mgr *BadgerAddrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } -func (mgr *AddrManagerBadger) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { +func (mgr *BadgerAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { txn := mgr.DB.NewTransaction(true) defer txn.Discard() @@ -142,7 +142,7 @@ func (mgr *AddrManagerBadger) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time txn.Commit(nil) } -func (mgr *AddrManagerBadger) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { +func (mgr *BadgerAddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { prefix := []byte(p) txn := mgr.DB.NewTransaction(true) defer txn.Discard() @@ -181,7 +181,7 @@ func (mgr *AddrManagerBadger) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTT txn.Commit(nil) } -func (mgr *AddrManagerBadger) Addrs(p peer.ID) []ma.Multiaddr { +func (mgr *BadgerAddrManager) Addrs(p peer.ID) []ma.Multiaddr { txn := mgr.DB.NewTransaction(false) defer txn.Discard() @@ -220,7 +220,7 @@ func (mgr *AddrManagerBadger) Addrs(p peer.ID) []ma.Multiaddr { return addrs } -func (mgr *AddrManagerBadger) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { +func (mgr *BadgerAddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { addrs := make(chan ma.Multiaddr) // TODO: impl @@ -228,7 +228,7 @@ func (mgr *AddrManagerBadger) AddrStream(ctx context.Context, p peer.ID) <-chan return addrs } -func (mgr *AddrManagerBadger) ClearAddrs(p peer.ID) { +func (mgr *BadgerAddrManager) ClearAddrs(p peer.ID) { txn := mgr.DB.NewTransaction(true) defer txn.Discard() it := txn.NewIterator(badger.DefaultIteratorOptions) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index e41e9655a8..ae9c724d54 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -7,6 +7,7 @@ import ( "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" "os" + "io/ioutil" ) func IDS(t *testing.T, ids string) peer.ID { @@ -49,6 +50,22 @@ func testHas(t *testing.T, exp, act []ma.Multiaddr) { } } +func setupBadgerAddrManager(t *testing.T) (*BadgerAddrManager, func ()) { + dataPath, err := ioutil.TempDir(os.TempDir(), "badger") + if err != nil { + t.Fatal(err) + } + mgr, err := NewBadgerAddrManager(dataPath) + if err != nil { + t.Fatal(err) + } + closer := func () { + mgr.Close() + os.RemoveAll(dataPath) + } + return mgr, closer +} + func testAddresses(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") id2 := IDS(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") @@ -99,28 +116,14 @@ func testAddresses(t *testing.T, m AddrBook) { testHas(t, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, m.Addrs(id5)) } -func setupBadgerAddrManager(t *testing.T) (*AddrManagerBadger, func ()) { - dataPath := os.TempDir() - mgr, err := NewBadgerAddrManager(dataPath) - if err != nil { - t.Fatal(err) - } - closer := func () { - mgr.Close() - os.RemoveAll(dataPath) - } - return mgr, closer -} - func TestAddresses(t *testing.T) { t.Log("AddrManager") mgr1 := AddrManager{} testAddresses(t, &mgr1) - t.Log("AddrManager") - - mgr2, closer := setupBadgerAddrManager(t) - defer closer() + t.Log("BadgerAddrManager") + mgr2, closer2 := setupBadgerAddrManager(t) + defer closer2() testAddresses(t, mgr2) } @@ -152,27 +155,27 @@ func testAddressesExpire(t *testing.T, m AddrBook) { testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) m.SetAddr(id1, ma11, time.Millisecond) - <-time.After(time.Millisecond) + <-time.After(time.Millisecond * 2) testHas(t, []ma.Multiaddr{ma12, ma13}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) m.SetAddr(id1, ma13, time.Millisecond) - <-time.After(time.Millisecond) + <-time.After(time.Millisecond * 2) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) m.SetAddr(id2, ma24, time.Millisecond) - <-time.After(time.Millisecond) + <-time.After(time.Millisecond * 2) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma25}, m.Addrs(id2)) m.SetAddr(id2, ma25, time.Millisecond) - <-time.After(time.Millisecond) + <-time.After(time.Millisecond * 2) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, nil, m.Addrs(id2)) m.SetAddr(id1, ma12, time.Millisecond) - <-time.After(time.Millisecond) + <-time.After(time.Millisecond * 2) testHas(t, nil, m.Addrs(id1)) testHas(t, nil, m.Addrs(id2)) } @@ -182,9 +185,9 @@ func TestAddressesExpire(t *testing.T) { m1 := &AddrManager{} testAddressesExpire(t, m1) - t.Log("AddrManagerBadger") - m2, closer := setupBadgerAddrManager(t) - defer closer() + t.Log("BadgerAddrManager") + m2, closer2 := setupBadgerAddrManager(t) + defer closer2() testAddressesExpire(t, m2) } @@ -218,9 +221,9 @@ func TestClearWorks(t *testing.T) { m1 := &AddrManager{} testClearWorks(t, m1) - t.Log("AddrManagerBadger") - m2, closer := setupBadgerAddrManager(t) - defer closer() + t.Log("BadgerAddrManager") + m2, closer2 := setupBadgerAddrManager(t) + defer closer2() testClearWorks(t, m2) } @@ -241,9 +244,9 @@ func TestSetNegativeTTLClears(t *testing.T) { m1 := &AddrManager{} testSetNegativeTTLClears(t, m1) - t.Log("AddrManagerBadger") - m2, closer := setupBadgerAddrManager(t) - defer closer() + t.Log("BadgerAddrManager") + m2, closer2 := setupBadgerAddrManager(t) + defer closer2() testSetNegativeTTLClears(t, m2) } @@ -275,7 +278,7 @@ func testUpdateTTLs(t *testing.T, m AddrBook) { testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - time.Sleep(time.Second) + time.Sleep(1200 * time.Millisecond) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) @@ -285,7 +288,7 @@ func testUpdateTTLs(t *testing.T, m AddrBook) { testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - time.Sleep(time.Second) + time.Sleep(1200 * time.Millisecond) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma22}, m.Addrs(id2)) @@ -296,9 +299,9 @@ func TestUpdateTTLs(t *testing.T) { m1 := &AddrManager{} testUpdateTTLs(t, m1) - t.Log("AddrManagerBadger") - m2, closer := setupBadgerAddrManager(t) - defer closer() + t.Log("BadgerAddrManager") + m2, closer2 := setupBadgerAddrManager(t) + defer closer2() testUpdateTTLs(t, m2) } @@ -313,9 +316,10 @@ func TestNilAddrsDontBreak(t *testing.T) { m1 := &AddrManager{} testNilAddrsDontBreak(t, m1) t.Log("OK") - t.Log("AddrManagerBadger") - m2, closer := setupBadgerAddrManager(t) - defer closer() + + t.Log("BadgerAddrManager") + m2, closer2 := setupBadgerAddrManager(t) + defer closer2() testNilAddrsDontBreak(t, m2) t.Log("OK") } diff --git a/p2p/host/peerstore/peerstore_badger.go b/p2p/host/peerstore/peerstore_badger.go index 55b585e415..859edccb79 100644 --- a/p2p/host/peerstore/peerstore_badger.go +++ b/p2p/host/peerstore/peerstore_badger.go @@ -10,7 +10,7 @@ import ( type PeerstoreBadger struct { *keybook *metrics - *AddrManagerBadger + *BadgerAddrManager ds map[string]interface{} dslock sync.Mutex @@ -27,20 +27,20 @@ func NewPeerstoreBadger(dataPath string) *PeerstoreBadger { return &PeerstoreBadger{ keybook: newKeybook(), metrics: NewMetrics(), - AddrManagerBadger: mgr, + BadgerAddrManager: mgr, ds: make(map[string]interface{}), } } func (ps *PeerstoreBadger) Close() error { - ps.AddrManagerBadger.Close() + ps.BadgerAddrManager.Close() return nil } func (ps *PeerstoreBadger) PeerInfo(p peer.ID) PeerInfo { return PeerInfo{ ID: p, - Addrs: ps.AddrManagerBadger.Addrs(p), + Addrs: ps.BadgerAddrManager.Addrs(p), } } From 5feafbe1d06cf9c652c9992f74def17cec740c40 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Mon, 11 Jun 2018 17:58:10 -0400 Subject: [PATCH 0579/3965] Implement and test Datastore-backed AddrManager --- p2p/host/peerstore/addr_manager_ds.go | 239 ++++++++++++++++++++++++ p2p/host/peerstore/addr_manager_test.go | 50 +++++ 2 files changed, 289 insertions(+) create mode 100644 p2p/host/peerstore/addr_manager_ds.go diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go new file mode 100644 index 0000000000..b3b3a1974c --- /dev/null +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -0,0 +1,239 @@ +package peerstore + +import ( + "context" + "sync" + "time" + + ds "github.com/ipfs/go-datastore" + "github.com/ipfs/go-datastore/query" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" + mh "github.com/multiformats/go-multihash" +) + +type DatastoreAddrManager struct { + ds ds.Datastore + ttlManager *ttlmanager + addrSubs map[peer.ID][]*addrSub +} + +func NewDatastoreAddrManager(ds ds.Datastore, ttlInterval time.Duration) *DatastoreAddrManager { + mgr := &DatastoreAddrManager{ + ds: ds, + ttlManager: newTTLManager(context.Background(), ds, ttlInterval), + addrSubs: make(map[peer.ID][]*addrSub), + } + return mgr +} + +func (mgr *DatastoreAddrManager) Stop() { + mgr.ttlManager.Stop() +} + +func peerAddressKey(p *peer.ID, addr *ma.Multiaddr) (ds.Key, error) { + hash, err := mh.Sum((*addr).Bytes(), mh.MURMUR3, -1) + if err != nil { + return ds.Key{}, nil + } + return ds.NewKey(p.Pretty()).ChildString(hash.B58String()), nil +} + +func (mgr *DatastoreAddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) +} + +func (mgr *DatastoreAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + if ttl <= 0 { + return + } + + mgr.SetAddrs(p, addrs, ttl) +} + +func (mgr *DatastoreAddrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) +} + +func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + var keys []ds.Key + for _, addr := range addrs { + if addr == nil { + continue + } + + key, err := peerAddressKey(&p, &addr) + if err != nil { + log.Error(err) + continue + } + keys = append(keys, key) + + if ttl <= 0 { + mgr.ds.Delete(key) + } else { + if err := mgr.ds.Put(key, addr.Bytes()); err != nil { + log.Error(err) + } + } + } + mgr.ttlManager.SetTTLs(keys, ttl) +} + +func (mgr *DatastoreAddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { + prefix := ds.NewKey(p.Pretty()) + mgr.ttlManager.UpdateTTLs(prefix, oldTTL, newTTL) +} + +func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { + prefix := ds.NewKey(p.Pretty()) + q := query.Query{Prefix: prefix.String()} + results, err := mgr.ds.Query(q) + if err != nil { + log.Error(err) + return []ma.Multiaddr{} + } + + var addrs []ma.Multiaddr + for result := range results.Next() { + addrbytes := result.Value.([]byte) + addr, err := ma.NewMultiaddrBytes(addrbytes) + if err != nil { + continue + } + addrs = append(addrs, addr) + } + + return addrs +} + +func (mgr *DatastoreAddrManager) AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr { + panic("implement me") + stream := make(chan ma.Multiaddr) + return stream +} + +func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { + prefix := ds.NewKey(p.Pretty()) + q := query.Query{Prefix: prefix.String()} + results, err := mgr.ds.Query(q) + if err != nil { + log.Error(err) + return + } + + for result := range results.Next() { + mgr.ds.Delete(ds.NewKey(result.Key)) + } + mgr.ttlManager.Clear(ds.NewKey(p.Pretty())) +} + +// ttlmanager + +type ttlentry struct { + TTL time.Duration + ExpiresAt time.Time +} + +type ttlmanager struct { + sync.Mutex + entries map[ds.Key]*ttlentry + ctx context.Context + cancel context.CancelFunc + ticker *time.Ticker + done chan struct{} + ds ds.Datastore +} + +func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) *ttlmanager { + ctx, cancel := context.WithCancel(parent) + mgr := &ttlmanager{ + Mutex: sync.Mutex{}, + entries: make(map[ds.Key]*ttlentry), + ctx: ctx, + cancel: cancel, + ticker: time.NewTicker(tick), + ds: d, + done: make(chan struct{}), + } + + go func() { + stop := false + for { + select { + case <-mgr.ctx.Done(): + stop = true + case <-mgr.ticker.C: + mgr.tick() + } + + if stop { + break + } + } + mgr.done <- struct{}{} + }() + + return mgr +} + +func (mgr *ttlmanager) Stop() { + mgr.cancel() + <-mgr.done +} + +// For internal use only +func (mgr *ttlmanager) tick() { + mgr.Lock() + defer mgr.Unlock() + + now := time.Now() + for key, entry := range mgr.entries { + if entry.ExpiresAt.Before(now) { + if err := mgr.ds.Delete(key); err != nil { + log.Error(err) + } + delete(mgr.entries, key) + } + } +} + +func (mgr *ttlmanager) SetTTLs(keys []ds.Key, ttl time.Duration) { + mgr.Lock() + defer mgr.Unlock() + + expiration := time.Now().Add(ttl) + for _, key := range keys { + if ttl <= 0 { + delete(mgr.entries, key) + } else { + mgr.entries[key] = &ttlentry{TTL: ttl, ExpiresAt: expiration} + } + } +} + +func (mgr *ttlmanager) UpdateTTLs(prefix ds.Key, oldTTL, newTTL time.Duration) { + mgr.Lock() + defer mgr.Unlock() + + now := time.Now() + var keys []ds.Key + for key, entry := range mgr.entries { + if key.IsDescendantOf(prefix) && entry.TTL == oldTTL { + keys = append(keys, key) + entry.TTL = newTTL + entry.ExpiresAt = now.Add(newTTL) + } + } +} + +func (mgr *ttlmanager) Clear(prefix ds.Key) { + mgr.Lock() + defer mgr.Unlock() + + for key := range mgr.entries { + if key.IsDescendantOf(prefix) { + delete(mgr.entries, key) + } + } +} diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index ae9c724d54..c9805c3bf7 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -7,6 +7,7 @@ import ( "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" "os" + "github.com/ipfs/go-ds-badger" "io/ioutil" ) @@ -66,6 +67,24 @@ func setupBadgerAddrManager(t *testing.T) (*BadgerAddrManager, func ()) { return mgr, closer } +func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func ()) { + dataPath, err := ioutil.TempDir(os.TempDir(), "badger") + if err != nil { + t.Fatal(err) + } + ds, err := badger.NewDatastore(dataPath, nil) + if err != nil { + t.Fatal(err) + } + mgr := NewDatastoreAddrManager(ds, 100 * time.Microsecond) + closer := func () { + mgr.Stop() + ds.Close() + os.RemoveAll(dataPath) + } + return mgr, closer +} + func testAddresses(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") id2 := IDS(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") @@ -125,6 +144,11 @@ func TestAddresses(t *testing.T) { mgr2, closer2 := setupBadgerAddrManager(t) defer closer2() testAddresses(t, mgr2) + + t.Log("DatastoreAddrManager") + mgr3, closer3 := setupDatastoreAddrManager(t) + defer closer3() + testAddresses(t, mgr3) } func testAddressesExpire(t *testing.T, m AddrBook) { @@ -189,6 +213,11 @@ func TestAddressesExpire(t *testing.T) { m2, closer2 := setupBadgerAddrManager(t) defer closer2() testAddressesExpire(t, m2) + + t.Log("DatastoreAddrManager") + m3, closer3 := setupDatastoreAddrManager(t) + defer closer3() + testAddressesExpire(t, m3) } func testClearWorks(t *testing.T, m AddrBook) { @@ -225,6 +254,11 @@ func TestClearWorks(t *testing.T) { m2, closer2 := setupBadgerAddrManager(t) defer closer2() testClearWorks(t, m2) + + t.Log("DatastoreAddrManager") + m3, closer3 := setupDatastoreAddrManager(t) + defer closer3() + testClearWorks(t, m3) } func testSetNegativeTTLClears(t *testing.T, m AddrBook) { @@ -248,6 +282,11 @@ func TestSetNegativeTTLClears(t *testing.T) { m2, closer2 := setupBadgerAddrManager(t) defer closer2() testSetNegativeTTLClears(t, m2) + + t.Log("DatastoreAddrManager") + m3, closer3 := setupDatastoreAddrManager(t) + defer closer3() + testSetNegativeTTLClears(t, m3) } func testUpdateTTLs(t *testing.T, m AddrBook) { @@ -303,6 +342,11 @@ func TestUpdateTTLs(t *testing.T) { m2, closer2 := setupBadgerAddrManager(t) defer closer2() testUpdateTTLs(t, m2) + + t.Log("DatastoreAddrManager") + m3, closer3 := setupDatastoreAddrManager(t) + defer closer3() + testUpdateTTLs(t, m3) } func testNilAddrsDontBreak(t *testing.T, m AddrBook) { @@ -322,4 +366,10 @@ func TestNilAddrsDontBreak(t *testing.T) { defer closer2() testNilAddrsDontBreak(t, m2) t.Log("OK") + + t.Log("DatastoreAddrManager") + m3, closer3 := setupDatastoreAddrManager(t) + defer closer3() + testNilAddrsDontBreak(t, m3) + t.Log("OK") } From be2b4ba3e29475322ba0482f2d41f33e0c014cff Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Mon, 11 Jun 2018 18:15:09 -0400 Subject: [PATCH 0580/3965] goimports file --- p2p/host/peerstore/addr_manager_test.go | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index c9805c3bf7..a9519f53e3 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -4,11 +4,12 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + "io/ioutil" "os" + "github.com/ipfs/go-ds-badger" - "io/ioutil" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func IDS(t *testing.T, ids string) peer.ID { @@ -51,7 +52,7 @@ func testHas(t *testing.T, exp, act []ma.Multiaddr) { } } -func setupBadgerAddrManager(t *testing.T) (*BadgerAddrManager, func ()) { +func setupBadgerAddrManager(t *testing.T) (*BadgerAddrManager, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) @@ -60,14 +61,14 @@ func setupBadgerAddrManager(t *testing.T) (*BadgerAddrManager, func ()) { if err != nil { t.Fatal(err) } - closer := func () { + closer := func() { mgr.Close() os.RemoveAll(dataPath) } return mgr, closer } -func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func ()) { +func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) @@ -76,8 +77,8 @@ func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func ()) { if err != nil { t.Fatal(err) } - mgr := NewDatastoreAddrManager(ds, 100 * time.Microsecond) - closer := func () { + mgr := NewDatastoreAddrManager(ds, 100*time.Microsecond) + closer := func() { mgr.Stop() ds.Close() os.RemoveAll(dataPath) From 1f253695d41ef7a3731a51657f3d2a1c033c8569 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Mon, 11 Jun 2018 19:54:17 -0400 Subject: [PATCH 0581/3965] Accept parent ctx in constructor, simplify TTL loop --- p2p/host/peerstore/addr_manager_ds.go | 14 +++++--------- p2p/host/peerstore/addr_manager_test.go | 3 ++- 2 files changed, 7 insertions(+), 10 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index b3b3a1974c..ba5165273a 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -18,10 +18,10 @@ type DatastoreAddrManager struct { addrSubs map[peer.ID][]*addrSub } -func NewDatastoreAddrManager(ds ds.Datastore, ttlInterval time.Duration) *DatastoreAddrManager { +func NewDatastoreAddrManager(ctx context.Context, ds ds.Datastore, ttlInterval time.Duration) *DatastoreAddrManager { mgr := &DatastoreAddrManager{ ds: ds, - ttlManager: newTTLManager(context.Background(), ds, ttlInterval), + ttlManager: newTTLManager(ctx, ds, ttlInterval), addrSubs: make(map[peer.ID][]*addrSub), } return mgr @@ -158,20 +158,16 @@ func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) * } go func() { - stop := false for { select { case <-mgr.ctx.Done(): - stop = true + mgr.ticker.Stop() + mgr.done <- struct{}{} + return case <-mgr.ticker.C: mgr.tick() } - - if stop { - break - } } - mgr.done <- struct{}{} }() return mgr diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index a9519f53e3..d3eb0c0a62 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -10,6 +10,7 @@ import ( "github.com/ipfs/go-ds-badger" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" + "context" ) func IDS(t *testing.T, ids string) peer.ID { @@ -77,7 +78,7 @@ func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func()) { if err != nil { t.Fatal(err) } - mgr := NewDatastoreAddrManager(ds, 100*time.Microsecond) + mgr := NewDatastoreAddrManager(context.Background(), ds, 100*time.Microsecond) closer := func() { mgr.Stop() ds.Close() From ba6527d831aa0938e1411f2e650fb1217c2be810 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Mon, 11 Jun 2018 19:59:50 -0400 Subject: [PATCH 0582/3965] Don't export ttl manager functions --- p2p/host/peerstore/addr_manager_ds.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index ba5165273a..a96dcdf28e 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -28,7 +28,7 @@ func NewDatastoreAddrManager(ctx context.Context, ds ds.Datastore, ttlInterval t } func (mgr *DatastoreAddrManager) Stop() { - mgr.ttlManager.Stop() + mgr.ttlManager.stop() } func peerAddressKey(p *peer.ID, addr *ma.Multiaddr) (ds.Key, error) { @@ -77,12 +77,12 @@ func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t } } } - mgr.ttlManager.SetTTLs(keys, ttl) + mgr.ttlManager.setTTLs(keys, ttl) } func (mgr *DatastoreAddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { prefix := ds.NewKey(p.Pretty()) - mgr.ttlManager.UpdateTTLs(prefix, oldTTL, newTTL) + mgr.ttlManager.updateTTLs(prefix, oldTTL, newTTL) } func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { @@ -125,7 +125,7 @@ func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { for result := range results.Next() { mgr.ds.Delete(ds.NewKey(result.Key)) } - mgr.ttlManager.Clear(ds.NewKey(p.Pretty())) + mgr.ttlManager.clear(ds.NewKey(p.Pretty())) } // ttlmanager @@ -173,7 +173,7 @@ func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) * return mgr } -func (mgr *ttlmanager) Stop() { +func (mgr *ttlmanager) stop() { mgr.cancel() <-mgr.done } @@ -194,7 +194,7 @@ func (mgr *ttlmanager) tick() { } } -func (mgr *ttlmanager) SetTTLs(keys []ds.Key, ttl time.Duration) { +func (mgr *ttlmanager) setTTLs(keys []ds.Key, ttl time.Duration) { mgr.Lock() defer mgr.Unlock() @@ -208,7 +208,7 @@ func (mgr *ttlmanager) SetTTLs(keys []ds.Key, ttl time.Duration) { } } -func (mgr *ttlmanager) UpdateTTLs(prefix ds.Key, oldTTL, newTTL time.Duration) { +func (mgr *ttlmanager) updateTTLs(prefix ds.Key, oldTTL, newTTL time.Duration) { mgr.Lock() defer mgr.Unlock() @@ -223,7 +223,7 @@ func (mgr *ttlmanager) UpdateTTLs(prefix ds.Key, oldTTL, newTTL time.Duration) { } } -func (mgr *ttlmanager) Clear(prefix ds.Key) { +func (mgr *ttlmanager) clear(prefix ds.Key) { mgr.Lock() defer mgr.Unlock() From 054d92828479e7ca26dd1f6d38ff2a68e1b1ddcf Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 12 Jun 2018 17:37:26 -0400 Subject: [PATCH 0583/3965] Extract subscription manager from AddrManager --- p2p/host/peerstore/addr_manager.go | 106 +++++++++++++++++------------ 1 file changed, 64 insertions(+), 42 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 8b06c484da..f5f4a3801f 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -7,8 +7,8 @@ import ( "sync" "time" - peer "github.com/libp2p/go-libp2p-peer" - addr "github.com/libp2p/go-libp2p-peerstore/addr" + "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peerstore/addr" ma "github.com/multiformats/go-multiaddr" ) @@ -60,7 +60,7 @@ type AddrManager struct { addrmu sync.Mutex // guards addrs addrs map[peer.ID]addrSlice - addrSubs map[peer.ID][]*addrSub + subManager *AddrSubManager } // ensures the AddrManager is initialized. @@ -69,8 +69,10 @@ func (mgr *AddrManager) init() { if mgr.addrs == nil { mgr.addrs = make(map[peer.ID]addrSlice) } - if mgr.addrSubs == nil { - mgr.addrSubs = make(map[peer.ID][]*addrSub) + if mgr.subManager == nil { + mgr.subManager = &AddrSubManager{ + subs: make(map[peer.ID][]*addrSub), + } } } @@ -114,8 +116,6 @@ func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat amap[string(ea.Addr.Bytes())] = ea } - subs := mgr.addrSubs[p] - // only expand ttls exp := time.Now().Add(ttl) for _, addr := range addrs { @@ -129,9 +129,7 @@ func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat if !found || exp.After(a.Expires) { amap[addrstr] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} - for _, sub := range subs { - sub.pubAddr(addr) - } + mgr.subManager.BroadcastAddr(p, addr) } } newAddrs := make([]expiringAddr, 0, len(amap)) @@ -161,8 +159,6 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat amap[string(ea.Addr.Bytes())] = ea } - subs := mgr.addrSubs[p] - exp := time.Now().Add(ttl) for _, addr := range addrs { if addr == nil { @@ -175,9 +171,7 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat if ttl > 0 { amap[addrs] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} - for _, sub := range subs { - sub.pubAddr(addr) - } + mgr.subManager.BroadcastAddr(p, addr) } else { delete(amap, addrs) } @@ -257,57 +251,71 @@ func (mgr *AddrManager) ClearAddrs(p peer.ID) { delete(mgr.addrs, p) } -func (mgr *AddrManager) removeSub(p peer.ID, s *addrSub) { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() - subs := mgr.addrSubs[p] +func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { + baseaddrslice := mgr.addrs[p] + initial := make([]ma.Multiaddr, 0, len(baseaddrslice)) + for _, a := range baseaddrslice { + initial = append(initial, a.Addr) + } + + return mgr.subManager.AddrStream(ctx, p, initial) +} + +type AddrSubManager struct { + mu sync.RWMutex + subs map[peer.ID][]*addrSub +} + +func NewAddrSubManager() *AddrSubManager { + return &AddrSubManager{ + subs: make(map[peer.ID][]*addrSub), + } +} + +func (mgr *AddrSubManager) removeSub(p peer.ID, s *addrSub) { + mgr.mu.Lock() + defer mgr.mu.Unlock() + subs := mgr.subs[p] if len(subs) == 1 { if subs[0] != s { return } - delete(mgr.addrSubs, p) + delete(mgr.subs, p) return } for i, v := range subs { if v == s { subs[i] = subs[len(subs)-1] subs[len(subs)-1] = nil - mgr.addrSubs[p] = subs[:len(subs)-1] + mgr.subs[p] = subs[:len(subs)-1] return } } } -type addrSub struct { - pubch chan ma.Multiaddr - lk sync.Mutex - buffer []ma.Multiaddr - ctx context.Context -} +func (mgr *AddrSubManager) BroadcastAddr(p peer.ID, addr ma.Multiaddr) { + mgr.mu.RLock() + defer mgr.mu.RUnlock() -func (s *addrSub) pubAddr(a ma.Multiaddr) { - select { - case s.pubch <- a: - case <-s.ctx.Done(): + subs, ok := mgr.subs[p] + if !ok { + return + } + + for _, sub := range subs { + sub.pubAddr(addr) } } -func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() - mgr.init() +func (mgr *AddrSubManager) AddrStream(ctx context.Context, p peer.ID, initial []ma.Multiaddr) <-chan ma.Multiaddr { + mgr.mu.Lock() + defer mgr.mu.Unlock() sub := &addrSub{pubch: make(chan ma.Multiaddr), ctx: ctx} out := make(chan ma.Multiaddr) - mgr.addrSubs[p] = append(mgr.addrSubs[p], sub) - - baseaddrslice := mgr.addrs[p] - initial := make([]ma.Multiaddr, 0, len(baseaddrslice)) - for _, a := range baseaddrslice { - initial = append(initial, a.Addr) - } + mgr.subs[p] = append(mgr.subs[p], sub) sort.Sort(addr.AddrList(initial)) @@ -360,3 +368,17 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mul return out } + +type addrSub struct { + pubch chan ma.Multiaddr + lk sync.Mutex + buffer []ma.Multiaddr + ctx context.Context +} + +func (s *addrSub) pubAddr(a ma.Multiaddr) { + select { + case s.pubch <- a: + case <-s.ctx.Done(): + } +} From f5aa83e9586deaa51158cfa5d23bad95ece546ce Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 12 Jun 2018 17:41:05 -0400 Subject: [PATCH 0584/3965] Use NewAddrSubManager --- p2p/host/peerstore/addr_manager.go | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index f5f4a3801f..c916197487 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -70,9 +70,7 @@ func (mgr *AddrManager) init() { mgr.addrs = make(map[peer.ID]addrSlice) } if mgr.subManager == nil { - mgr.subManager = &AddrSubManager{ - subs: make(map[peer.ID][]*addrSub), - } + mgr.subManager = NewAddrSubManager() } } From 7d9c93dea6a306c7fa9d6e9c057744c8ef60c73d Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 12 Jun 2018 17:46:12 -0400 Subject: [PATCH 0585/3965] Use mutex hat pattern --- p2p/host/peerstore/peerstore_badger.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/peerstore_badger.go b/p2p/host/peerstore/peerstore_badger.go index 859edccb79..ac5adb423e 100644 --- a/p2p/host/peerstore/peerstore_badger.go +++ b/p2p/host/peerstore/peerstore_badger.go @@ -12,8 +12,8 @@ type PeerstoreBadger struct { *metrics *BadgerAddrManager - ds map[string]interface{} dslock sync.Mutex + ds map[string]interface{} // lock for protocol information, separate from datastore lock protolock sync.Mutex From 84866e56bf7db105f0a5ea52021fc964724ebc84 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 12 Jun 2018 17:54:11 -0400 Subject: [PATCH 0586/3965] Break nesting via continues --- p2p/host/peerstore/addr_manager_badger.go | 62 +++++++++++------------ 1 file changed, 30 insertions(+), 32 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_badger.go b/p2p/host/peerstore/addr_manager_badger.go index 0a612d6804..39c04dcb42 100644 --- a/p2p/host/peerstore/addr_manager_badger.go +++ b/p2p/host/peerstore/addr_manager_badger.go @@ -4,22 +4,22 @@ import ( "context" "time" + "bytes" + "encoding/gob" + "github.com/dgraph-io/badger" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" - "github.com/dgraph-io/badger" "github.com/multiformats/go-multihash" - "encoding/gob" - "bytes" ) type BadgerAddrManager struct { - DB *badger.DB + DB *badger.DB addrSubs map[peer.ID][]*addrSub } type addrentry struct { Addr []byte - TTL time.Duration + TTL time.Duration } func (mgr *BadgerAddrManager) sendSubscriptionUpdates(p *peer.ID, addrs []ma.Multiaddr) { @@ -99,7 +99,6 @@ func (mgr *BadgerAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time txn := mgr.DB.NewTransaction(true) defer txn.Discard() - go mgr.sendSubscriptionUpdates(&p, addrs) addAddrs(p, addrs, ttl, txn) @@ -126,17 +125,17 @@ func (mgr *BadgerAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time if ttl <= 0 { if err := txn.Delete(key); err != nil { log.Error(err) - } - } else { - entry := &addrentry{Addr: addr.Bytes(), TTL: ttl} - buf := &bytes.Buffer{} - enc := gob.NewEncoder(buf) - if err := enc.Encode(entry); err != nil { - log.Error(err) continue } - txn.SetWithTTL(key, buf.Bytes(), ttl) } + entry := &addrentry{Addr: addr.Bytes(), TTL: ttl} + buf := &bytes.Buffer{} + enc := gob.NewEncoder(buf) + if err := enc.Encode(entry); err != nil { + log.Error(err) + continue + } + txn.SetWithTTL(key, buf.Bytes(), ttl) } txn.Commit(nil) @@ -192,27 +191,26 @@ func (mgr *BadgerAddrManager) Addrs(p peer.ID) []ma.Multiaddr { var addrs []ma.Multiaddr - for iter.ValidForPrefix(prefix) { + for ; iter.ValidForPrefix(prefix); iter.Next() { item := iter.Item() - - if !item.IsDeletedOrExpired() { - value, err := item.Value() - if err != nil { - log.Error(err) - } else { - entry := &addrentry{} - buf := bytes.NewBuffer(value) - dec := gob.NewDecoder(buf) - if err := dec.Decode(&entry); err != nil { - log.Error("deleting bad entry in peerstore for peer", p.String()) - txn.Delete(item.Key()) - } else { - addrs = append(addrs, ma.Cast(entry.Addr)) - } - } + if item.IsDeletedOrExpired() { + continue } - iter.Next() + value, err := item.Value() + if err != nil { + log.Error(err) + continue + } + entry := &addrentry{} + buf := bytes.NewBuffer(value) + dec := gob.NewDecoder(buf) + if err := dec.Decode(&entry); err != nil { + log.Error("deleting bad entry in peerstore for peer", p.String()) + txn.Delete(item.Key()) + } else { + addrs = append(addrs, ma.Cast(entry.Addr)) + } } txn.Commit(nil) From a0d7719a1bec20120ef953d3b866d0ff052a2018 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 12 Jun 2018 17:58:57 -0400 Subject: [PATCH 0587/3965] Flatten loops --- p2p/host/peerstore/addr_manager_ds.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index a96dcdf28e..2d41378bab 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -71,10 +71,10 @@ func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t if ttl <= 0 { mgr.ds.Delete(key) - } else { - if err := mgr.ds.Put(key, addr.Bytes()); err != nil { - log.Error(err) - } + continue + } + if err := mgr.ds.Put(key, addr.Bytes()); err != nil { + log.Error(err) } } mgr.ttlManager.setTTLs(keys, ttl) From dd10058a1b863d04e9cb667ca3c076924c18822a Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 12 Jun 2018 18:02:42 -0400 Subject: [PATCH 0588/3965] Instrument DatastoreAddrManager with AddrSubManager --- p2p/host/peerstore/addr_manager_ds.go | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 2d41378bab..1c94fe0ca6 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -13,16 +13,16 @@ import ( ) type DatastoreAddrManager struct { - ds ds.Datastore - ttlManager *ttlmanager - addrSubs map[peer.ID][]*addrSub + ds ds.Datastore + ttlManager *ttlmanager + subsManager *AddrSubManager } func NewDatastoreAddrManager(ctx context.Context, ds ds.Datastore, ttlInterval time.Duration) *DatastoreAddrManager { mgr := &DatastoreAddrManager{ - ds: ds, - ttlManager: newTTLManager(ctx, ds, ttlInterval), - addrSubs: make(map[peer.ID][]*addrSub), + ds: ds, + ttlManager: newTTLManager(ctx, ds, ttlInterval), + subsManager: NewAddrSubManager(), } return mgr } @@ -73,6 +73,9 @@ func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t mgr.ds.Delete(key) continue } + if has, err := mgr.ds.Has(key); err != nil && has == false { + mgr.subsManager.BroadcastAddr(p, addr) + } if err := mgr.ds.Put(key, addr.Bytes()); err != nil { log.Error(err) } @@ -107,10 +110,9 @@ func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { return addrs } -func (mgr *DatastoreAddrManager) AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr { - panic("implement me") - stream := make(chan ma.Multiaddr) - return stream +func (mgr *DatastoreAddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { + initial := mgr.Addrs(p) + return mgr.subsManager.AddrStream(ctx, p, initial) } func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { From 8e9b6dc1ebbb622fc7a02281f5c9abd1df1e9f46 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 13 Jun 2018 19:26:06 -0400 Subject: [PATCH 0589/3965] Update native badger impl to support Peers --- p2p/host/peerstore/addr_manager_badger.go | 143 ++++++++++++++-------- 1 file changed, 90 insertions(+), 53 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_badger.go b/p2p/host/peerstore/addr_manager_badger.go index 39c04dcb42..16b696692b 100644 --- a/p2p/host/peerstore/addr_manager_badger.go +++ b/p2p/host/peerstore/addr_manager_badger.go @@ -5,7 +5,10 @@ import ( "time" "bytes" + "encoding/binary" "encoding/gob" + "errors" + "github.com/dgraph-io/badger" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" @@ -13,8 +16,8 @@ import ( ) type BadgerAddrManager struct { - DB *badger.DB - addrSubs map[peer.ID][]*addrSub + DB *badger.DB + subsManager *AddrSubManager } type addrentry struct { @@ -22,15 +25,6 @@ type addrentry struct { TTL time.Duration } -func (mgr *BadgerAddrManager) sendSubscriptionUpdates(p *peer.ID, addrs []ma.Multiaddr) { - subs := mgr.addrSubs[*p] - for _, sub := range subs { - for _, addr := range addrs { - sub.pubAddr(addr) - } - } -} - func (mgr *BadgerAddrManager) Close() { if err := mgr.DB.Close(); err != nil { log.Error(err) @@ -45,11 +39,7 @@ func NewBadgerAddrManager(dataPath string) (*BadgerAddrManager, error) { if err != nil { return nil, err } - return &BadgerAddrManager{DB: db}, nil -} - -func (mgr *BadgerAddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) + return &BadgerAddrManager{DB: db, subsManager: NewAddrSubManager()}, nil } // use murmur3 because it's the most compact @@ -57,37 +47,58 @@ func hashMultiaddr(addr *ma.Multiaddr) ([]byte, error) { return multihash.Encode((*addr).Bytes(), multihash.MURMUR3) } +func createKeyPrefix(p *peer.ID) ([]byte, error) { + buf := &bytes.Buffer{} + prefix := []byte(*p) + prefixlen := uint64(len(prefix)) + if err := binary.Write(buf, binary.LittleEndian, prefixlen); err != nil { + return nil, err + } + buf.Write(prefix) + return buf.Bytes(), nil +} + func createUniqueKey(p *peer.ID, addr *ma.Multiaddr) ([]byte, error) { + buf := &bytes.Buffer{} prefix := []byte(*p) - addrHash, err := hashMultiaddr(addr) + prefixlen := uint64(len(prefix)) + if err := binary.Write(buf, binary.LittleEndian, prefixlen); err != nil { + return nil, err + } + buf.Write(prefix) + addrhash, err := hashMultiaddr(addr) if err != nil { return nil, err } + addrhashlen := uint64(len(addrhash)) + + if err := binary.Write(buf, binary.LittleEndian, addrhashlen); err != nil { + return nil, err + } - return append(prefix, addrHash...), nil + buf.Write(addrhash) + return buf.Bytes(), nil } -func addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, txn *badger.Txn) { - for _, addr := range addrs { - if addr == nil { - continue - } - entry := &addrentry{Addr: addr.Bytes(), TTL: ttl} - key, err := createUniqueKey(&p, &addr) - if err != nil { - log.Error(err) - txn.Discard() - return - } - buf := &bytes.Buffer{} - enc := gob.NewEncoder(buf) - if err := enc.Encode(entry); err != nil { - log.Error(err) - txn.Discard() - return - } - txn.SetWithTTL(key, buf.Bytes(), ttl) +func peerIDFromBinaryKey(key []byte) (peer.ID, error) { + buf := bytes.NewBuffer(key) + idlen, err := binary.ReadUvarint(buf) + if err != nil { + return peer.ID(""), err } + idbytes := make([]byte, idlen) + read, err := buf.Read(idbytes) + if err != nil { + return peer.ID(""), err + } + if uint64(read) != idlen { + return peer.ID(""), errors.New("invalid key length") + } + return peer.IDFromBytes(idbytes) +} + +func (mgr *BadgerAddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } func (mgr *BadgerAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { @@ -96,13 +107,7 @@ func (mgr *BadgerAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time return } - txn := mgr.DB.NewTransaction(true) - defer txn.Discard() - - go mgr.sendSubscriptionUpdates(&p, addrs) - addAddrs(p, addrs, ttl, txn) - - txn.Commit(nil) + mgr.SetAddrs(p, addrs, ttl) } func (mgr *BadgerAddrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { @@ -128,6 +133,9 @@ func (mgr *BadgerAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time continue } } + if item, err := txn.Get(key); err != nil || item.IsDeletedOrExpired() { + mgr.subsManager.BroadcastAddr(p, addr) + } entry := &addrentry{Addr: addr.Bytes(), TTL: ttl} buf := &bytes.Buffer{} enc := gob.NewEncoder(buf) @@ -142,7 +150,11 @@ func (mgr *BadgerAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time } func (mgr *BadgerAddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { - prefix := []byte(p) + prefix, err := createKeyPrefix(&p) + if err != nil { + log.Error(err) + return + } txn := mgr.DB.NewTransaction(true) defer txn.Discard() opts := badger.DefaultIteratorOptions @@ -184,7 +196,11 @@ func (mgr *BadgerAddrManager) Addrs(p peer.ID) []ma.Multiaddr { txn := mgr.DB.NewTransaction(false) defer txn.Discard() - prefix := []byte(p) + prefix, err := createKeyPrefix(&p) + if err != nil { + log.Error(err) + return []ma.Multiaddr{} + } opts := badger.DefaultIteratorOptions iter := txn.NewIterator(opts) iter.Seek(prefix) @@ -218,19 +234,40 @@ func (mgr *BadgerAddrManager) Addrs(p peer.ID) []ma.Multiaddr { return addrs } -func (mgr *BadgerAddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { - addrs := make(chan ma.Multiaddr) - - // TODO: impl +func (mgr *BadgerAddrManager) Peers() []peer.ID { + txn := mgr.DB.NewTransaction(false) + defer txn.Commit(nil) + it := txn.NewIterator(badger.DefaultIteratorOptions) + idset := make(map[peer.ID]struct{}) + for ; it.Valid(); it.Next() { + item := it.Item() + id, err := peerIDFromBinaryKey(item.Key()) + if err != nil { + continue + } + idset[id] = struct{}{} + } + ids := make([]peer.ID, 0, len(idset)) + for id := range idset { + ids = append(ids, id) + } + return ids +} - return addrs +func (mgr *BadgerAddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { + initial := mgr.Addrs(p) + return mgr.subsManager.AddrStream(ctx, p, initial) } func (mgr *BadgerAddrManager) ClearAddrs(p peer.ID) { txn := mgr.DB.NewTransaction(true) defer txn.Discard() it := txn.NewIterator(badger.DefaultIteratorOptions) - prefix := []byte(p) + prefix, err := createKeyPrefix(&p) + if err != nil { + log.Error(err) + return + } it.Seek(prefix) count := 0 From 53f0d200752e3573564413d202b6c277ad5aadb2 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 13 Jun 2018 19:27:14 -0400 Subject: [PATCH 0590/3965] Move Peers from Peerstore to AddrBook --- p2p/host/peerstore/addr_manager_ds.go | 30 +++++++++++++++++++++++++++ p2p/host/peerstore/peerstore.go | 12 +++++------ 2 files changed, 36 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 1c94fe0ca6..ed397099ce 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -39,6 +39,11 @@ func peerAddressKey(p *peer.ID, addr *ma.Multiaddr) (ds.Key, error) { return ds.NewKey(p.Pretty()).ChildString(hash.B58String()), nil } +func peerIDFromKey(key ds.Key) (peer.ID, error) { + idstring := key.Parent().Type() + return peer.IDB58Decode(idstring) +} + func (mgr *DatastoreAddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } @@ -110,6 +115,31 @@ func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { return addrs } +func (mgr *DatastoreAddrManager) Peers() []peer.ID { + q := query.Query{} + results, err := mgr.ds.Query(q) + if err != nil { + log.Error(err) + return []peer.ID{} + } + + idset := make(map[peer.ID]struct{}) + for result := range results.Next() { + key := ds.RawKey(result.Key) + id, err := peerIDFromKey(key) + if err != nil { + continue + } + idset[id] = struct{}{} + } + + ids := make([]peer.ID, 0, len(idset)) + for id := range idset { + ids = append(ids, id) + } + return ids +} + func (mgr *DatastoreAddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { initial := mgr.Addrs(p) return mgr.subsManager.AddrStream(ctx, p, initial) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index abfdeae6d3..c47d8aae1c 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -30,9 +30,6 @@ type Peerstore interface { KeyBook Metrics - // Peers returns a list of all peer.IDs in this Peerstore - Peers() []peer.ID - // PeerInfo returns a peer.PeerInfo struct for given peer.ID. // This is a small slice of the information Peerstore has on // that peer, useful to other services. @@ -83,6 +80,9 @@ type AddrBook interface { // ClearAddresses removes all previously stored addresses ClearAddrs(p peer.ID) + + // Peers returns all of the peer IDs stored in the AddrBook + Peers() []peer.ID } // KeyBook tracks the Public keys of Peers. @@ -179,7 +179,7 @@ func (kb *keybook) AddPrivKey(p peer.ID, sk ic.PrivKey) error { type peerstore struct { *keybook *metrics - AddrManager + AddrBook // store other data, like versions //ds ds.ThreadSafeDatastore @@ -231,7 +231,7 @@ func (ps *peerstore) Peers() []peer.ID { for _, p := range ps.keybook.Peers() { set[p] = struct{}{} } - for _, p := range ps.AddrManager.Peers() { + for _, p := range ps.AddrBook.Peers() { set[p] = struct{}{} } @@ -245,7 +245,7 @@ func (ps *peerstore) Peers() []peer.ID { func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { return PeerInfo{ ID: p, - Addrs: ps.AddrManager.Addrs(p), + Addrs: ps.AddrBook.Addrs(p), } } From b9ca3813dfa5bbfec2e99bb0de09beec63a4b0a5 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 13 Jun 2018 19:27:43 -0400 Subject: [PATCH 0591/3965] Clean tests --- p2p/host/peerstore/addr_manager_test.go | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index d3eb0c0a62..c7b9b5f799 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -7,10 +7,11 @@ import ( "io/ioutil" "os" + "context" + "github.com/ipfs/go-ds-badger" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" - "context" ) func IDS(t *testing.T, ids string) peer.ID { @@ -361,17 +362,14 @@ func TestNilAddrsDontBreak(t *testing.T) { t.Log("AddrManager") m1 := &AddrManager{} testNilAddrsDontBreak(t, m1) - t.Log("OK") t.Log("BadgerAddrManager") m2, closer2 := setupBadgerAddrManager(t) defer closer2() testNilAddrsDontBreak(t, m2) - t.Log("OK") t.Log("DatastoreAddrManager") m3, closer3 := setupDatastoreAddrManager(t) defer closer3() testNilAddrsDontBreak(t, m3) - t.Log("OK") } From 181e26e24e6012a88408e5165150e113be220269 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 13 Jun 2018 19:27:57 -0400 Subject: [PATCH 0592/3965] Add NewPeerstoreDatastore --- p2p/host/peerstore/peerstore.go | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index c47d8aae1c..f964322bb1 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -11,6 +11,7 @@ import ( //ds "github.com/jbenet/go-datastore" //dssync "github.com/jbenet/go-datastore/sync" + "github.com/ipfs/go-datastore" logging "github.com/ipfs/go-log" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" @@ -194,11 +195,21 @@ type peerstore struct { // NewPeerstore creates a threadsafe collection of peers. func NewPeerstore() Peerstore { return &peerstore{ - keybook: newKeybook(), - metrics: NewMetrics(), - AddrManager: AddrManager{}, - //ds: dssync.MutexWrap(ds.NewMapDatastore()), - ds: make(map[string]interface{}), + keybook: newKeybook(), + metrics: NewMetrics(), + AddrBook: &AddrManager{}, + ds: make(map[string]interface{}), + } +} + +// NewPeerstoreDatastore creates a threadsafe collection of peers backed by a +// Datastore to prevent excess memory pressure. +func NewPeerstoreDatastore(ctx context.Context, ds datastore.Datastore) Peerstore { + return &peerstore{ + keybook: newKeybook(), + metrics: NewMetrics(), + AddrBook: NewDatastoreAddrManager(ctx, ds, time.Second), + ds: make(map[string]interface{}), } } From 8d9891b5db7fbdaf0be0ed95919b73ef3e0bd2a5 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Thu, 14 Jun 2018 16:17:50 -0400 Subject: [PATCH 0593/3965] Abstract Peerstore tests, fix bugs --- p2p/host/peerstore/addr_manager.go | 15 ++++-- p2p/host/peerstore/addr_manager_ds.go | 6 +-- p2p/host/peerstore/addr_manager_test.go | 25 ++++++--- p2p/host/peerstore/peerstore_test.go | 70 +++++++++++++++++++------ 4 files changed, 85 insertions(+), 31 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index c916197487..09319af74f 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -250,6 +250,10 @@ func (mgr *AddrManager) ClearAddrs(p peer.ID) { } func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { + mgr.addrmu.Lock() + defer mgr.addrmu.Unlock() + mgr.init() + baseaddrslice := mgr.addrs[p] initial := make([]ma.Multiaddr, 0, len(baseaddrslice)) for _, a := range baseaddrslice { @@ -266,6 +270,7 @@ type AddrSubManager struct { func NewAddrSubManager() *AddrSubManager { return &AddrSubManager{ + mu: sync.RWMutex{}, subs: make(map[peer.ID][]*addrSub), } } @@ -292,8 +297,8 @@ func (mgr *AddrSubManager) removeSub(p peer.ID, s *addrSub) { } func (mgr *AddrSubManager) BroadcastAddr(p peer.ID, addr ma.Multiaddr) { - mgr.mu.RLock() - defer mgr.mu.RUnlock() + mgr.mu.Lock() + defer mgr.mu.Unlock() subs, ok := mgr.subs[p] if !ok { @@ -313,7 +318,11 @@ func (mgr *AddrSubManager) AddrStream(ctx context.Context, p peer.ID, initial [] out := make(chan ma.Multiaddr) - mgr.subs[p] = append(mgr.subs[p], sub) + if _, ok := mgr.subs[p]; ok { + mgr.subs[p] = append(mgr.subs[p], sub) + } else { + mgr.subs[p] = []*addrSub{sub} + } sort.Sort(addr.AddrList(initial)) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index ed397099ce..125d6f0f85 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -36,11 +36,11 @@ func peerAddressKey(p *peer.ID, addr *ma.Multiaddr) (ds.Key, error) { if err != nil { return ds.Key{}, nil } - return ds.NewKey(p.Pretty()).ChildString(hash.B58String()), nil + return ds.NewKey(peer.IDB58Encode(*p)).ChildString(hash.B58String()), nil } func peerIDFromKey(key ds.Key) (peer.ID, error) { - idstring := key.Parent().Type() + idstring := key.Parent().Name() return peer.IDB58Decode(idstring) } @@ -78,7 +78,7 @@ func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t mgr.ds.Delete(key) continue } - if has, err := mgr.ds.Has(key); err != nil && has == false { + if has, err := mgr.ds.Has(key); err != nil || !has { mgr.subsManager.BroadcastAddr(p, addr) } if err := mgr.ds.Put(key, addr.Bytes()); err != nil { diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index c7b9b5f799..1f491887a3 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -9,6 +9,7 @@ import ( "context" + "github.com/ipfs/go-datastore" "github.com/ipfs/go-ds-badger" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" @@ -54,36 +55,44 @@ func testHas(t *testing.T, exp, act []ma.Multiaddr) { } } -func setupBadgerAddrManager(t *testing.T) (*BadgerAddrManager, func()) { +func setupBadgerDatastore(t *testing.T) (datastore.Datastore, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) } - mgr, err := NewBadgerAddrManager(dataPath) + ds, err := badger.NewDatastore(dataPath, nil) if err != nil { t.Fatal(err) } closer := func() { - mgr.Close() + ds.Close() os.RemoveAll(dataPath) } - return mgr, closer + return ds, closer } -func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func()) { +func setupBadgerAddrManager(t *testing.T) (*BadgerAddrManager, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) } - ds, err := badger.NewDatastore(dataPath, nil) + mgr, err := NewBadgerAddrManager(dataPath) if err != nil { t.Fatal(err) } + closer := func() { + mgr.Close() + os.RemoveAll(dataPath) + } + return mgr, closer +} + +func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func()) { + ds, closeDB := setupBadgerDatastore(t) mgr := NewDatastoreAddrManager(context.Background(), ds, 100*time.Microsecond) closer := func() { mgr.Stop() - ds.Close() - os.RemoveAll(dataPath) + closeDB() } return mgr, closer } diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 55a8e901a1..16261324f0 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -10,7 +10,8 @@ import ( "os" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-crypto" + "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) @@ -27,13 +28,34 @@ func getAddrs(t *testing.T, n int) []ma.Multiaddr { return addrs } -func TestAddrStream(t *testing.T) { +func runTestWithPeerstores(t *testing.T, testFunc func(*testing.T, Peerstore)) { + t.Helper() + t.Log("NewPeerstore") + ps1 := NewPeerstore() + testFunc(t, ps1) + + t.Log("NewPeerstoreDatastore") + ps2, closer2 := setupDatastorePeerstore(t) + defer closer2() + testFunc(t, ps2) +} + +func setupDatastorePeerstore(t *testing.T) (Peerstore, func()) { + ds, closeDB := setupBadgerDatastore(t) + ctx, cancel := context.WithCancel(context.Background()) + ps := NewPeerstoreDatastore(ctx, ds) + closer := func() { + cancel() + closeDB() + } + return ps, closer +} + +func testAddrStream(t *testing.T, ps Peerstore) { addrs := getAddrs(t, 100) pid := peer.ID("testpeer") - ps := NewPeerstore() - ps.AddAddrs(pid, addrs[:10], time.Hour) ctx, cancel := context.WithCancel(context.Background()) @@ -105,12 +127,14 @@ func TestAddrStream(t *testing.T) { } } -func TestGetStreamBeforePeerAdded(t *testing.T) { +func TestAddrStream(t *testing.T) { + runTestWithPeerstores(t, testAddrStream) +} + +func testGetStreamBeforePeerAdded(t *testing.T, ps Peerstore) { addrs := getAddrs(t, 10) pid := peer.ID("testpeer") - ps := NewPeerstore() - ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -156,12 +180,14 @@ func TestGetStreamBeforePeerAdded(t *testing.T) { } } -func TestAddrStreamDuplicates(t *testing.T) { +func TestGetStreamBeforePeerAdded(t *testing.T) { + runTestWithPeerstores(t, testGetStreamBeforePeerAdded) +} + +func testAddrStreamDuplicates(t *testing.T, ps Peerstore) { addrs := getAddrs(t, 10) pid := peer.ID("testpeer") - ps := NewPeerstore() - ctx, cancel := context.WithCancel(context.Background()) defer cancel() ach := ps.AddrStream(ctx, pid) @@ -195,8 +221,11 @@ func TestAddrStreamDuplicates(t *testing.T) { } } -func TestPeerstoreProtoStore(t *testing.T) { - ps := NewPeerstore() +func TestAddrStreamDuplicates(t *testing.T) { + runTestWithPeerstores(t, testAddrStreamDuplicates) +} + +func testPeerstoreProtoStore(t *testing.T, ps Peerstore) { p1 := peer.ID("TESTPEER") protos := []string{"a", "b", "c", "d"} @@ -250,20 +279,23 @@ func TestPeerstoreProtoStore(t *testing.T) { } } -func TestBasicPeerstore(t *testing.T) { - ps := NewPeerstore() +func TestPeerstoreProtoStore(t *testing.T) { + runTestWithPeerstores(t, testAddrStreamDuplicates) +} +func testBasicPeerstore(t *testing.T, ps Peerstore) { var pids []peer.ID addrs := getAddrs(t, 10) - for i, a := range addrs { - p := peer.ID(fmt.Sprint(i)) + for _, a := range addrs { + priv, _, _ := crypto.GenerateKeyPair(crypto.RSA, 512) + p, _ := peer.IDFromPrivateKey(priv) pids = append(pids, p) ps.AddAddr(p, a, PermanentAddrTTL) } peers := ps.Peers() if len(peers) != 10 { - t.Fatal("expected ten peers") + t.Fatal("expected ten peers, got", len(peers)) } pinfo := ps.PeerInfo(pids[0]) @@ -272,6 +304,10 @@ func TestBasicPeerstore(t *testing.T) { } } +func TestBasicPeerstore(t *testing.T) { + runTestWithPeerstores(t, testBasicPeerstore) +} + func BenchmarkBasicPeerstore(b *testing.B) { ps := NewPeerstore() From be4bdb1af92a026bc72f0cec0f8d3073628db81c Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 13:46:35 -0400 Subject: [PATCH 0594/3965] Update AddrManagerDatastore to require a Batching Datastore --- p2p/host/peerstore/addr_manager_ds.go | 60 ++++++++++++++++++++----- p2p/host/peerstore/addr_manager_test.go | 10 ++--- p2p/host/peerstore/peerstore.go | 2 +- 3 files changed, 53 insertions(+), 19 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 125d6f0f85..22ac30da7d 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -12,13 +12,18 @@ import ( mh "github.com/multiformats/go-multihash" ) +// DatastoreAddrManager is an address manager backed by a Datastore with both an +// in-memory TTL manager and an in-memory address stream manager. type DatastoreAddrManager struct { - ds ds.Datastore + ds ds.Batching ttlManager *ttlmanager subsManager *AddrSubManager } -func NewDatastoreAddrManager(ctx context.Context, ds ds.Datastore, ttlInterval time.Duration) *DatastoreAddrManager { +// NewDatastoreAddrManager initializes a new DatastoreAddrManager given a +// Datastore instance, a context for managing the TTL manager, and the interval +// at which the TTL manager should sweep the Datastore. +func NewDatastoreAddrManager(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) *DatastoreAddrManager { mgr := &DatastoreAddrManager{ ds: ds, ttlManager: newTTLManager(ctx, ds, ttlInterval), @@ -27,6 +32,7 @@ func NewDatastoreAddrManager(ctx context.Context, ds ds.Datastore, ttlInterval t return mgr } +// Stop will signal the TTL manager to stop and block until it returns. func (mgr *DatastoreAddrManager) Stop() { mgr.ttlManager.stop() } @@ -44,24 +50,38 @@ func peerIDFromKey(key ds.Key) (peer.ID, error) { return peer.IDB58Decode(idstring) } +// AddAddr will add a new address if it's not already in the AddrBook. func (mgr *DatastoreAddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } +// AddAddrs will add many new addresses if they're not already in the AddrBook. func (mgr *DatastoreAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { if ttl <= 0 { return } - mgr.SetAddrs(p, addrs, ttl) + mgr.setAddrs(p, addrs, ttl, true) } +// SetAddr will add or update the TTL of an address in the AddrBook. func (mgr *DatastoreAddrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } +// SetAddrs will add or update the TTLs of addresses in the AddrBook. func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + mgr.setAddrs(p, addrs, ttl, false) +} + +func (mgr *DatastoreAddrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, add bool) { var keys []ds.Key + batch, err := mgr.ds.Batch() + if err != nil { + log.Error(err) + return + } + defer batch.Commit() for _, addr := range addrs { if addr == nil { continue @@ -75,24 +95,31 @@ func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t keys = append(keys, key) if ttl <= 0 { - mgr.ds.Delete(key) + batch.Delete(key) continue } - if has, err := mgr.ds.Has(key); err != nil || !has { + has, err := mgr.ds.Has(key) + if err != nil || !has { mgr.subsManager.BroadcastAddr(p, addr) } - if err := mgr.ds.Put(key, addr.Bytes()); err != nil { - log.Error(err) + + if !has || !add { + if err := batch.Put(key, addr.Bytes()); err != nil { + log.Error(err) + } } } mgr.ttlManager.setTTLs(keys, ttl) } +// UpdateAddrs will update any addresses for a given peer and TTL combination to +// have a new TTL. func (mgr *DatastoreAddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { prefix := ds.NewKey(p.Pretty()) mgr.ttlManager.updateTTLs(prefix, oldTTL, newTTL) } +// Addrs Returns all of the non-expired addresses for a given peer. func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { prefix := ds.NewKey(p.Pretty()) q := query.Query{Prefix: prefix.String()} @@ -115,6 +142,7 @@ func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { return addrs } +// Peers returns all of the peer IDs for which the AddrBook has addresses. func (mgr *DatastoreAddrManager) Peers() []peer.ID { q := query.Query{} results, err := mgr.ds.Query(q) @@ -140,11 +168,14 @@ func (mgr *DatastoreAddrManager) Peers() []peer.ID { return ids } +// AddrStream returns a channel on which all new addresses discovered for a +// given peer ID will be published. func (mgr *DatastoreAddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { initial := mgr.Addrs(p) return mgr.subsManager.AddrStream(ctx, p, initial) } +// ClearAddrs will delete all known addresses for a peer ID. func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { prefix := ds.NewKey(p.Pretty()) q := query.Query{Prefix: prefix.String()} @@ -153,9 +184,15 @@ func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { log.Error(err) return } + batch, err := mgr.ds.Batch() + if err != nil { + log.Error(err) + return + } + defer batch.Commit() for result := range results.Next() { - mgr.ds.Delete(ds.NewKey(result.Key)) + batch.Delete(ds.NewKey(result.Key)) } mgr.ttlManager.clear(ds.NewKey(p.Pretty())) } @@ -168,7 +205,7 @@ type ttlentry struct { } type ttlmanager struct { - sync.Mutex + sync.RWMutex entries map[ds.Key]*ttlentry ctx context.Context cancel context.CancelFunc @@ -180,7 +217,6 @@ type ttlmanager struct { func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) *ttlmanager { ctx, cancel := context.WithCancel(parent) mgr := &ttlmanager{ - Mutex: sync.Mutex{}, entries: make(map[ds.Key]*ttlentry), ctx: ctx, cancel: cancel, @@ -212,8 +248,8 @@ func (mgr *ttlmanager) stop() { // For internal use only func (mgr *ttlmanager) tick() { - mgr.Lock() - defer mgr.Unlock() + mgr.RLock() + defer mgr.RUnlock() now := time.Now() for key, entry := range mgr.entries { diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 1f491887a3..f3c2254c0e 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -1,13 +1,11 @@ package peerstore import ( - "testing" - "time" - + "context" "io/ioutil" "os" - - "context" + "testing" + "time" "github.com/ipfs/go-datastore" "github.com/ipfs/go-ds-badger" @@ -55,7 +53,7 @@ func testHas(t *testing.T, exp, act []ma.Multiaddr) { } } -func setupBadgerDatastore(t *testing.T) (datastore.Datastore, func()) { +func setupBadgerDatastore(t *testing.T) (datastore.Batching, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index f964322bb1..38296a3d8d 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -204,7 +204,7 @@ func NewPeerstore() Peerstore { // NewPeerstoreDatastore creates a threadsafe collection of peers backed by a // Datastore to prevent excess memory pressure. -func NewPeerstoreDatastore(ctx context.Context, ds datastore.Datastore) Peerstore { +func NewPeerstoreDatastore(ctx context.Context, ds datastore.Batching) Peerstore { return &peerstore{ keybook: newKeybook(), metrics: NewMetrics(), From d93a04f936bee38eb4cb2e14ee239f42cad185f4 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 13:54:04 -0400 Subject: [PATCH 0595/3965] Document and clean AddrSubManager --- p2p/host/peerstore/addr_manager.go | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index 09319af74f..d03ad77090 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -240,7 +240,7 @@ func (mgr *AddrManager) Addrs(p peer.ID) []ma.Multiaddr { return good } -// ClearAddresses removes all previously stored addresses +// ClearAddrs removes all previously stored addresses func (mgr *AddrManager) ClearAddrs(p peer.ID) { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() @@ -249,6 +249,8 @@ func (mgr *AddrManager) ClearAddrs(p peer.ID) { delete(mgr.addrs, p) } +// AddrStream returns a channel on which all new addresses discovered for a +// given peer ID will be published. func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() @@ -263,18 +265,22 @@ func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mul return mgr.subManager.AddrStream(ctx, p, initial) } +// An abstracted, pub-sub manager for address streams. Extracted from +// AddrManager in order to support additional implementations. type AddrSubManager struct { mu sync.RWMutex subs map[peer.ID][]*addrSub } +// NewAddrSubManager initializes an AddrSubManager. func NewAddrSubManager() *AddrSubManager { return &AddrSubManager{ - mu: sync.RWMutex{}, subs: make(map[peer.ID][]*addrSub), } } +// Used internally by the address stream coroutine to remove a subscription +// from the manager. func (mgr *AddrSubManager) removeSub(p peer.ID, s *addrSub) { mgr.mu.Lock() defer mgr.mu.Unlock() @@ -296,20 +302,20 @@ func (mgr *AddrSubManager) removeSub(p peer.ID, s *addrSub) { } } +// BroadcastAddr broadcasts a new address to all subscribed streams. func (mgr *AddrSubManager) BroadcastAddr(p peer.ID, addr ma.Multiaddr) { - mgr.mu.Lock() - defer mgr.mu.Unlock() - - subs, ok := mgr.subs[p] - if !ok { - return - } + mgr.mu.RLock() + defer mgr.mu.RUnlock() - for _, sub := range subs { - sub.pubAddr(addr) + if subs, ok := mgr.subs[p]; ok { + for _, sub := range subs { + sub.pubAddr(addr) + } } } +// AddrStream creates a new subscription for a given peer ID, pre-populating the +// channel with any addresses we might already have on file. func (mgr *AddrSubManager) AddrStream(ctx context.Context, p peer.ID, initial []ma.Multiaddr) <-chan ma.Multiaddr { mgr.mu.Lock() defer mgr.mu.Unlock() From ebc90a153e6222aa80c8032fcc2b742b4b4af5f1 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 13:56:35 -0400 Subject: [PATCH 0596/3965] Clarify comment for tick function --- p2p/host/peerstore/addr_manager_ds.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 22ac30da7d..dd987eae57 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -246,7 +246,7 @@ func (mgr *ttlmanager) stop() { <-mgr.done } -// For internal use only +// To be called by TTL manager's coroutine only. func (mgr *ttlmanager) tick() { mgr.RLock() defer mgr.RUnlock() From 0a596fdf4f83437ba5de0e23c5c849e91855182b Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 17:30:47 -0400 Subject: [PATCH 0597/3965] Remove pure badger impl of AddrBook --- p2p/host/peerstore/addr_manager_badger.go | 282 ---------------------- p2p/host/peerstore/addr_manager_test.go | 111 ++------- p2p/host/peerstore/peerstore_badger.go | 140 ----------- p2p/host/peerstore/peerstore_test.go | 48 ---- 4 files changed, 17 insertions(+), 564 deletions(-) delete mode 100644 p2p/host/peerstore/addr_manager_badger.go delete mode 100644 p2p/host/peerstore/peerstore_badger.go diff --git a/p2p/host/peerstore/addr_manager_badger.go b/p2p/host/peerstore/addr_manager_badger.go deleted file mode 100644 index 16b696692b..0000000000 --- a/p2p/host/peerstore/addr_manager_badger.go +++ /dev/null @@ -1,282 +0,0 @@ -package peerstore - -import ( - "context" - "time" - - "bytes" - "encoding/binary" - "encoding/gob" - "errors" - - "github.com/dgraph-io/badger" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" - "github.com/multiformats/go-multihash" -) - -type BadgerAddrManager struct { - DB *badger.DB - subsManager *AddrSubManager -} - -type addrentry struct { - Addr []byte - TTL time.Duration -} - -func (mgr *BadgerAddrManager) Close() { - if err := mgr.DB.Close(); err != nil { - log.Error(err) - } -} - -func NewBadgerAddrManager(dataPath string) (*BadgerAddrManager, error) { - opts := badger.DefaultOptions - opts.Dir = dataPath - opts.ValueDir = dataPath - db, err := badger.Open(opts) - if err != nil { - return nil, err - } - return &BadgerAddrManager{DB: db, subsManager: NewAddrSubManager()}, nil -} - -// use murmur3 because it's the most compact -func hashMultiaddr(addr *ma.Multiaddr) ([]byte, error) { - return multihash.Encode((*addr).Bytes(), multihash.MURMUR3) -} - -func createKeyPrefix(p *peer.ID) ([]byte, error) { - buf := &bytes.Buffer{} - prefix := []byte(*p) - prefixlen := uint64(len(prefix)) - if err := binary.Write(buf, binary.LittleEndian, prefixlen); err != nil { - return nil, err - } - buf.Write(prefix) - return buf.Bytes(), nil -} - -func createUniqueKey(p *peer.ID, addr *ma.Multiaddr) ([]byte, error) { - buf := &bytes.Buffer{} - prefix := []byte(*p) - prefixlen := uint64(len(prefix)) - if err := binary.Write(buf, binary.LittleEndian, prefixlen); err != nil { - return nil, err - } - buf.Write(prefix) - addrhash, err := hashMultiaddr(addr) - if err != nil { - return nil, err - } - addrhashlen := uint64(len(addrhash)) - - if err := binary.Write(buf, binary.LittleEndian, addrhashlen); err != nil { - return nil, err - } - - buf.Write(addrhash) - return buf.Bytes(), nil -} - -func peerIDFromBinaryKey(key []byte) (peer.ID, error) { - buf := bytes.NewBuffer(key) - idlen, err := binary.ReadUvarint(buf) - if err != nil { - return peer.ID(""), err - } - idbytes := make([]byte, idlen) - read, err := buf.Read(idbytes) - if err != nil { - return peer.ID(""), err - } - if uint64(read) != idlen { - return peer.ID(""), errors.New("invalid key length") - } - return peer.IDFromBytes(idbytes) -} - -func (mgr *BadgerAddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) -} - -func (mgr *BadgerAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - if ttl <= 0 { - log.Debugf("short circuiting AddAddrs with ttl %d", ttl) - return - } - - mgr.SetAddrs(p, addrs, ttl) -} - -func (mgr *BadgerAddrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) -} - -func (mgr *BadgerAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - txn := mgr.DB.NewTransaction(true) - defer txn.Discard() - - for _, addr := range addrs { - if addr == nil { - continue - } - key, err := createUniqueKey(&p, &addr) - if err != nil { - log.Error(err) - continue - } - if ttl <= 0 { - if err := txn.Delete(key); err != nil { - log.Error(err) - continue - } - } - if item, err := txn.Get(key); err != nil || item.IsDeletedOrExpired() { - mgr.subsManager.BroadcastAddr(p, addr) - } - entry := &addrentry{Addr: addr.Bytes(), TTL: ttl} - buf := &bytes.Buffer{} - enc := gob.NewEncoder(buf) - if err := enc.Encode(entry); err != nil { - log.Error(err) - continue - } - txn.SetWithTTL(key, buf.Bytes(), ttl) - } - - txn.Commit(nil) -} - -func (mgr *BadgerAddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { - prefix, err := createKeyPrefix(&p) - if err != nil { - log.Error(err) - return - } - txn := mgr.DB.NewTransaction(true) - defer txn.Discard() - opts := badger.DefaultIteratorOptions - iter := txn.NewIterator(opts) - - iter.Seek(prefix) - for iter.ValidForPrefix(prefix) { - item := iter.Item() - addrbytes, err := item.Value() - if err != nil { - log.Error(err) - return - } - entry := &addrentry{} - buf := bytes.NewBuffer(addrbytes) - dec := gob.NewDecoder(buf) - if err := dec.Decode(&entry); err != nil { - log.Error(err) - return - } - if entry.TTL == oldTTL { - entry.TTL = newTTL - buf := &bytes.Buffer{} - enc := gob.NewEncoder(buf) - if err := enc.Encode(&entry); err != nil { - log.Error(err) - return - } - txn.SetWithTTL(item.Key(), buf.Bytes(), newTTL) - } - - iter.Next() - } - - txn.Commit(nil) -} - -func (mgr *BadgerAddrManager) Addrs(p peer.ID) []ma.Multiaddr { - txn := mgr.DB.NewTransaction(false) - defer txn.Discard() - - prefix, err := createKeyPrefix(&p) - if err != nil { - log.Error(err) - return []ma.Multiaddr{} - } - opts := badger.DefaultIteratorOptions - iter := txn.NewIterator(opts) - iter.Seek(prefix) - - var addrs []ma.Multiaddr - - for ; iter.ValidForPrefix(prefix); iter.Next() { - item := iter.Item() - if item.IsDeletedOrExpired() { - continue - } - - value, err := item.Value() - if err != nil { - log.Error(err) - continue - } - entry := &addrentry{} - buf := bytes.NewBuffer(value) - dec := gob.NewDecoder(buf) - if err := dec.Decode(&entry); err != nil { - log.Error("deleting bad entry in peerstore for peer", p.String()) - txn.Delete(item.Key()) - } else { - addrs = append(addrs, ma.Cast(entry.Addr)) - } - } - - txn.Commit(nil) - - return addrs -} - -func (mgr *BadgerAddrManager) Peers() []peer.ID { - txn := mgr.DB.NewTransaction(false) - defer txn.Commit(nil) - it := txn.NewIterator(badger.DefaultIteratorOptions) - idset := make(map[peer.ID]struct{}) - for ; it.Valid(); it.Next() { - item := it.Item() - id, err := peerIDFromBinaryKey(item.Key()) - if err != nil { - continue - } - idset[id] = struct{}{} - } - ids := make([]peer.ID, 0, len(idset)) - for id := range idset { - ids = append(ids, id) - } - return ids -} - -func (mgr *BadgerAddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { - initial := mgr.Addrs(p) - return mgr.subsManager.AddrStream(ctx, p, initial) -} - -func (mgr *BadgerAddrManager) ClearAddrs(p peer.ID) { - txn := mgr.DB.NewTransaction(true) - defer txn.Discard() - it := txn.NewIterator(badger.DefaultIteratorOptions) - prefix, err := createKeyPrefix(&p) - if err != nil { - log.Error(err) - return - } - it.Seek(prefix) - - count := 0 - for it.ValidForPrefix(prefix) { - count++ - if err := txn.Delete(it.Item().Key()); err != nil { - log.Error(err) - } - it.Next() - } - txn.Commit(nil) -} diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index f3c2254c0e..6c26ed4a95 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -69,22 +69,6 @@ func setupBadgerDatastore(t *testing.T) (datastore.Batching, func()) { return ds, closer } -func setupBadgerAddrManager(t *testing.T) (*BadgerAddrManager, func()) { - dataPath, err := ioutil.TempDir(os.TempDir(), "badger") - if err != nil { - t.Fatal(err) - } - mgr, err := NewBadgerAddrManager(dataPath) - if err != nil { - t.Fatal(err) - } - closer := func() { - mgr.Close() - os.RemoveAll(dataPath) - } - return mgr, closer -} - func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func()) { ds, closeDB := setupBadgerDatastore(t) mgr := NewDatastoreAddrManager(context.Background(), ds, 100*time.Microsecond) @@ -95,6 +79,17 @@ func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func()) { return mgr, closer } +func runTestWithAddrManagers(t *testing.T, test func(*testing.T, AddrBook)) { + t.Log("AddrManager") + mgr1 := &AddrManager{} + test(t, mgr1) + + t.Log("DatastoreAddrManager") + mgr2, closer2 := setupDatastoreAddrManager(t) + defer closer2() + test(t, mgr2) +} + func testAddresses(t *testing.T, m AddrBook) { id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") id2 := IDS(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") @@ -146,19 +141,7 @@ func testAddresses(t *testing.T, m AddrBook) { } func TestAddresses(t *testing.T) { - t.Log("AddrManager") - mgr1 := AddrManager{} - testAddresses(t, &mgr1) - - t.Log("BadgerAddrManager") - mgr2, closer2 := setupBadgerAddrManager(t) - defer closer2() - testAddresses(t, mgr2) - - t.Log("DatastoreAddrManager") - mgr3, closer3 := setupDatastoreAddrManager(t) - defer closer3() - testAddresses(t, mgr3) + runTestWithAddrManagers(t, testAddresses) } func testAddressesExpire(t *testing.T, m AddrBook) { @@ -215,19 +198,7 @@ func testAddressesExpire(t *testing.T, m AddrBook) { } func TestAddressesExpire(t *testing.T) { - t.Log("AddrManager") - m1 := &AddrManager{} - testAddressesExpire(t, m1) - - t.Log("BadgerAddrManager") - m2, closer2 := setupBadgerAddrManager(t) - defer closer2() - testAddressesExpire(t, m2) - - t.Log("DatastoreAddrManager") - m3, closer3 := setupDatastoreAddrManager(t) - defer closer3() - testAddressesExpire(t, m3) + runTestWithAddrManagers(t, testAddressesExpire) } func testClearWorks(t *testing.T, m AddrBook) { @@ -256,19 +227,7 @@ func testClearWorks(t *testing.T, m AddrBook) { } func TestClearWorks(t *testing.T) { - t.Log("AddrManager") - m1 := &AddrManager{} - testClearWorks(t, m1) - - t.Log("BadgerAddrManager") - m2, closer2 := setupBadgerAddrManager(t) - defer closer2() - testClearWorks(t, m2) - - t.Log("DatastoreAddrManager") - m3, closer3 := setupDatastoreAddrManager(t) - defer closer3() - testClearWorks(t, m3) + runTestWithAddrManagers(t, testClearWorks) } func testSetNegativeTTLClears(t *testing.T, m AddrBook) { @@ -284,19 +243,7 @@ func testSetNegativeTTLClears(t *testing.T, m AddrBook) { testHas(t, nil, m.Addrs(id1)) } func TestSetNegativeTTLClears(t *testing.T) { - t.Log("AddrManager") - m1 := &AddrManager{} - testSetNegativeTTLClears(t, m1) - - t.Log("BadgerAddrManager") - m2, closer2 := setupBadgerAddrManager(t) - defer closer2() - testSetNegativeTTLClears(t, m2) - - t.Log("DatastoreAddrManager") - m3, closer3 := setupDatastoreAddrManager(t) - defer closer3() - testSetNegativeTTLClears(t, m3) + runTestWithAddrManagers(t, testSetNegativeTTLClears) } func testUpdateTTLs(t *testing.T, m AddrBook) { @@ -344,19 +291,7 @@ func testUpdateTTLs(t *testing.T, m AddrBook) { } func TestUpdateTTLs(t *testing.T) { - t.Log("AddrManager") - m1 := &AddrManager{} - testUpdateTTLs(t, m1) - - t.Log("BadgerAddrManager") - m2, closer2 := setupBadgerAddrManager(t) - defer closer2() - testUpdateTTLs(t, m2) - - t.Log("DatastoreAddrManager") - m3, closer3 := setupDatastoreAddrManager(t) - defer closer3() - testUpdateTTLs(t, m3) + runTestWithAddrManagers(t, testUpdateTTLs) } func testNilAddrsDontBreak(t *testing.T, m AddrBook) { @@ -366,17 +301,5 @@ func testNilAddrsDontBreak(t *testing.T, m AddrBook) { } func TestNilAddrsDontBreak(t *testing.T) { - t.Log("AddrManager") - m1 := &AddrManager{} - testNilAddrsDontBreak(t, m1) - - t.Log("BadgerAddrManager") - m2, closer2 := setupBadgerAddrManager(t) - defer closer2() - testNilAddrsDontBreak(t, m2) - - t.Log("DatastoreAddrManager") - m3, closer3 := setupDatastoreAddrManager(t) - defer closer3() - testNilAddrsDontBreak(t, m3) + runTestWithAddrManagers(t, testNilAddrsDontBreak) } diff --git a/p2p/host/peerstore/peerstore_badger.go b/p2p/host/peerstore/peerstore_badger.go deleted file mode 100644 index ac5adb423e..0000000000 --- a/p2p/host/peerstore/peerstore_badger.go +++ /dev/null @@ -1,140 +0,0 @@ -package peerstore - -import ( - "fmt" - "sync" - - "github.com/libp2p/go-libp2p-peer" -) - -type PeerstoreBadger struct { - *keybook - *metrics - *BadgerAddrManager - - dslock sync.Mutex - ds map[string]interface{} - - // lock for protocol information, separate from datastore lock - protolock sync.Mutex -} - -func NewPeerstoreBadger(dataPath string) *PeerstoreBadger { - mgr, err := NewBadgerAddrManager(dataPath) - if err != nil { - panic(err) - } - return &PeerstoreBadger{ - keybook: newKeybook(), - metrics: NewMetrics(), - BadgerAddrManager: mgr, - ds: make(map[string]interface{}), - } -} - -func (ps *PeerstoreBadger) Close() error { - ps.BadgerAddrManager.Close() - return nil -} - -func (ps *PeerstoreBadger) PeerInfo(p peer.ID) PeerInfo { - return PeerInfo{ - ID: p, - Addrs: ps.BadgerAddrManager.Addrs(p), - } -} - -func (ps *PeerstoreBadger) Get(p peer.ID, key string) (interface{}, error) { - ps.dslock.Lock() - defer ps.dslock.Unlock() - i, ok := ps.ds[string(p)+"/"+key] - if !ok { - return nil, ErrNotFound - } - return i, nil -} - -func (ps *PeerstoreBadger) Put(p peer.ID, key string, val interface{}) error { - ps.dslock.Lock() - defer ps.dslock.Unlock() - ps.ds[string(p)+"/"+key] = val - return nil -} - -func (ps *PeerstoreBadger) getProtocolMap(p peer.ID) (map[string]struct{}, error) { - iprotomap, err := ps.Get(p, "protocols") - switch err { - default: - return nil, err - case ErrNotFound: - return make(map[string]struct{}), nil - case nil: - cast, ok := iprotomap.(map[string]struct{}) - if !ok { - return nil, fmt.Errorf("stored protocol set was not a map") - } - - return cast, nil - } -} - -func (ps *PeerstoreBadger) GetProtocols(p peer.ID) ([]string, error) { - ps.protolock.Lock() - defer ps.protolock.Unlock() - pmap, err := ps.getProtocolMap(p) - if err != nil { - return nil, err - } - - var out []string - for k, _ := range pmap { - out = append(out, k) - } - - return out, nil -} - -func (ps *PeerstoreBadger) AddProtocols(p peer.ID, protos ...string) error { - ps.protolock.Lock() - defer ps.protolock.Unlock() - protomap, err := ps.getProtocolMap(p) - if err != nil { - return err - } - - for _, proto := range protos { - protomap[proto] = struct{}{} - } - - return ps.Put(p, "protocols", protomap) -} - -func (ps *PeerstoreBadger) SetProtocols(p peer.ID, protos ...string) error { - ps.protolock.Lock() - defer ps.protolock.Unlock() - - protomap := make(map[string]struct{}) - for _, proto := range protos { - protomap[proto] = struct{}{} - } - - return ps.Put(p, "protocols", protomap) -} - -func (ps *PeerstoreBadger) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { - ps.protolock.Lock() - defer ps.protolock.Unlock() - pmap, err := ps.getProtocolMap(p) - if err != nil { - return nil, err - } - - var out []string - for _, proto := range protos { - if _, ok := pmap[proto]; ok { - out = append(out, proto) - } - } - - return out, nil -} diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 16261324f0..647cfcf64a 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -8,8 +8,6 @@ import ( "testing" "time" - "os" - "github.com/libp2p/go-libp2p-crypto" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" @@ -325,28 +323,6 @@ func BenchmarkBasicPeerstore(b *testing.B) { } } -func BenchmarkBadgerPeerstore(b *testing.B) { - dataPath := os.TempDir() - ps := NewPeerstoreBadger(dataPath) - defer func() { - ps.Close() - os.RemoveAll(dataPath) - }() - - addrs := make(chan *peerpair, 100) - go addressProducer(b, addrs, 50000) - - b.ResetTimer() - for { - pp, ok := <-addrs - if !ok { - break - } - pid := peer.ID(pp.ID) - ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) - } -} - func BenchmarkBasicPeerstoreRateLimited(b *testing.B) { ps := NewPeerstore() @@ -365,27 +341,3 @@ func BenchmarkBasicPeerstoreRateLimited(b *testing.B) { ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) } } - -func BenchmarkBadgerPeerstoreRateLimited(b *testing.B) { - dataPath := os.TempDir() - ps := NewPeerstoreBadger(dataPath) - defer func() { - ps.Close() - os.RemoveAll(dataPath) - }() - - producer := make(chan *peerpair, 100) - addrs := make(chan *peerpair, 100) - - go rateLimitedAddressProducer(b, addrs, producer, 5000, time.Millisecond, 500*time.Microsecond) - - b.ResetTimer() - for { - pp, ok := <-addrs - if !ok { - break - } - pid := peer.ID(pp.ID) - ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) - } -} From 4dfef4c9de67ede7331e6b61c65e531bb779c20d Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 17:33:18 -0400 Subject: [PATCH 0598/3965] Only lock critical section --- p2p/host/peerstore/addr_manager.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index d03ad77090..c20e54cbb3 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -317,18 +317,17 @@ func (mgr *AddrSubManager) BroadcastAddr(p peer.ID, addr ma.Multiaddr) { // AddrStream creates a new subscription for a given peer ID, pre-populating the // channel with any addresses we might already have on file. func (mgr *AddrSubManager) AddrStream(ctx context.Context, p peer.ID, initial []ma.Multiaddr) <-chan ma.Multiaddr { - mgr.mu.Lock() - defer mgr.mu.Unlock() - sub := &addrSub{pubch: make(chan ma.Multiaddr), ctx: ctx} out := make(chan ma.Multiaddr) + mgr.mu.Lock() if _, ok := mgr.subs[p]; ok { mgr.subs[p] = append(mgr.subs[p], sub) } else { mgr.subs[p] = []*addrSub{sub} } + mgr.mu.Unlock() sort.Sort(addr.AddrList(initial)) From 3eb102a69214124ecb787571465ea37a3cc876d4 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 17:35:18 -0400 Subject: [PATCH 0599/3965] Remove redundant channel from ttlmanager --- p2p/host/peerstore/addr_manager_ds.go | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index dd987eae57..cfe979f30c 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -34,7 +34,7 @@ func NewDatastoreAddrManager(ctx context.Context, ds ds.Batching, ttlInterval ti // Stop will signal the TTL manager to stop and block until it returns. func (mgr *DatastoreAddrManager) Stop() { - mgr.ttlManager.stop() + mgr.ttlManager.cancel() } func peerAddressKey(p *peer.ID, addr *ma.Multiaddr) (ds.Key, error) { @@ -210,7 +210,6 @@ type ttlmanager struct { ctx context.Context cancel context.CancelFunc ticker *time.Ticker - done chan struct{} ds ds.Datastore } @@ -222,7 +221,6 @@ func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) * cancel: cancel, ticker: time.NewTicker(tick), ds: d, - done: make(chan struct{}), } go func() { @@ -230,7 +228,6 @@ func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) * select { case <-mgr.ctx.Done(): mgr.ticker.Stop() - mgr.done <- struct{}{} return case <-mgr.ticker.C: mgr.tick() @@ -241,11 +238,6 @@ func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) * return mgr } -func (mgr *ttlmanager) stop() { - mgr.cancel() - <-mgr.done -} - // To be called by TTL manager's coroutine only. func (mgr *ttlmanager) tick() { mgr.RLock() From 884733f84fc97df63680b71391c68a3dcb47d00b Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 17:35:47 -0400 Subject: [PATCH 0600/3965] Mutex hat pattern --- p2p/host/peerstore/addr_manager_ds.go | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index cfe979f30c..7cb0b60f8d 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -207,10 +207,11 @@ type ttlentry struct { type ttlmanager struct { sync.RWMutex entries map[ds.Key]*ttlentry - ctx context.Context - cancel context.CancelFunc - ticker *time.Ticker - ds ds.Datastore + + ctx context.Context + cancel context.CancelFunc + ticker *time.Ticker + ds ds.Datastore } func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) *ttlmanager { From 353a43da1e85aca4bfed7a746d1e8084590e3f54 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 17:48:39 -0400 Subject: [PATCH 0601/3965] Implement error retry logic for transactions --- p2p/host/peerstore/addr_manager_ds.go | 102 +++++++++++++++----------- 1 file changed, 61 insertions(+), 41 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 7cb0b60f8d..71ed061764 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -12,6 +12,9 @@ import ( mh "github.com/multiformats/go-multihash" ) +// Number of times to retry transactional writes +var dsWriteRetries = 5 + // DatastoreAddrManager is an address manager backed by a Datastore with both an // in-memory TTL manager and an in-memory address stream manager. type DatastoreAddrManager struct { @@ -75,41 +78,51 @@ func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t } func (mgr *DatastoreAddrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, add bool) { - var keys []ds.Key - batch, err := mgr.ds.Batch() - if err != nil { - log.Error(err) - return - } - defer batch.Commit() - for _, addr := range addrs { - if addr == nil { - continue - } - - key, err := peerAddressKey(&p, &addr) + for i := 0; i < dsWriteRetries; i++ { + // keys to add to the TTL manager + var keys []ds.Key + batch, err := mgr.ds.Batch() if err != nil { log.Error(err) - continue + return } - keys = append(keys, key) - if ttl <= 0 { - batch.Delete(key) - continue - } - has, err := mgr.ds.Has(key) - if err != nil || !has { - mgr.subsManager.BroadcastAddr(p, addr) - } + for _, addr := range addrs { + if addr == nil { + continue + } - if !has || !add { - if err := batch.Put(key, addr.Bytes()); err != nil { + key, err := peerAddressKey(&p, &addr) + if err != nil { log.Error(err) + continue + } + keys = append(keys, key) + + if ttl <= 0 { + batch.Delete(key) + continue } + has, err := mgr.ds.Has(key) + if err != nil || !has { + mgr.subsManager.BroadcastAddr(p, addr) + } + + // Allows us to support AddAddr and SetAddr in one function + if !has || !add { + if err := batch.Put(key, addr.Bytes()); err != nil { + log.Error(err) + } + } + } + if err := batch.Commit(); err != nil { + log.Errorf("failed to write addresses for peer %s: %s\n", p.Pretty(), err) + continue } + mgr.ttlManager.setTTLs(keys, ttl) + return } - mgr.ttlManager.setTTLs(keys, ttl) + log.Errorf("failed to avoid write conflict for peer %s after %d retries\n", p.Pretty(), dsWriteRetries) } // UpdateAddrs will update any addresses for a given peer and TTL combination to @@ -178,23 +191,30 @@ func (mgr *DatastoreAddrManager) AddrStream(ctx context.Context, p peer.ID) <-ch // ClearAddrs will delete all known addresses for a peer ID. func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { prefix := ds.NewKey(p.Pretty()) - q := query.Query{Prefix: prefix.String()} - results, err := mgr.ds.Query(q) - if err != nil { - log.Error(err) - return - } - batch, err := mgr.ds.Batch() - if err != nil { - log.Error(err) - return - } - defer batch.Commit() + for i := 0; i < dsWriteRetries; i++ { + q := query.Query{Prefix: prefix.String()} + results, err := mgr.ds.Query(q) + if err != nil { + log.Error(err) + return + } + batch, err := mgr.ds.Batch() + if err != nil { + log.Error(err) + return + } - for result := range results.Next() { - batch.Delete(ds.NewKey(result.Key)) + for result := range results.Next() { + batch.Delete(ds.NewKey(result.Key)) + } + if err = batch.Commit(); err != nil { + log.Errorf("failed to clear addresses for peer %s: %s\n", p.Pretty(), err) + continue + } + mgr.ttlManager.clear(ds.NewKey(p.Pretty())) + return } - mgr.ttlManager.clear(ds.NewKey(p.Pretty())) + log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), dsWriteRetries) } // ttlmanager From e7ce3569b670f29aec80851e3eec225bf86ad82e Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 26 Jun 2018 17:22:04 -0400 Subject: [PATCH 0602/3965] Update ttlmanager to use ds.Batching --- p2p/host/peerstore/addr_manager_ds.go | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 71ed061764..75b16c80a6 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -231,7 +231,7 @@ type ttlmanager struct { ctx context.Context cancel context.CancelFunc ticker *time.Ticker - ds ds.Datastore + ds ds.Batching } func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) *ttlmanager { @@ -265,14 +265,23 @@ func (mgr *ttlmanager) tick() { defer mgr.RUnlock() now := time.Now() + batch, err := mgr.ds.Batch() + if err != nil { + log.Error(err) + return + } for key, entry := range mgr.entries { if entry.ExpiresAt.Before(now) { - if err := mgr.ds.Delete(key); err != nil { + if err := batch.Delete(key); err != nil { log.Error(err) } delete(mgr.entries, key) } } + err := batch.Commit() + if err != nil { + log.Error(err) + } } func (mgr *ttlmanager) setTTLs(keys []ds.Key, ttl time.Duration) { From 02a67d2b713ece6f00c6fa3741c2c8383088d51a Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 26 Jun 2018 17:58:38 -0400 Subject: [PATCH 0603/3965] Fix logic for AddAddrs --- p2p/host/peerstore/addr_manager_ds.go | 32 +++++++++++++++++++-------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 75b16c80a6..aa7b8a0cc7 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -109,7 +109,7 @@ func (mgr *DatastoreAddrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t } // Allows us to support AddAddr and SetAddr in one function - if !has || !add { + if !has { if err := batch.Put(key, addr.Bytes()); err != nil { log.Error(err) } @@ -119,7 +119,7 @@ func (mgr *DatastoreAddrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t log.Errorf("failed to write addresses for peer %s: %s\n", p.Pretty(), err) continue } - mgr.ttlManager.setTTLs(keys, ttl) + mgr.ttlManager.setTTLs(keys, ttl, add) return } log.Errorf("failed to avoid write conflict for peer %s after %d retries\n", p.Pretty(), dsWriteRetries) @@ -236,12 +236,16 @@ type ttlmanager struct { func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) *ttlmanager { ctx, cancel := context.WithCancel(parent) + batching, ok := d.(ds.Batching) + if !ok { + panic("must construct ttlmanager with batching datastore") + } mgr := &ttlmanager{ entries: make(map[ds.Key]*ttlentry), ctx: ctx, cancel: cancel, ticker: time.NewTicker(tick), - ds: d, + ds: batching, } go func() { @@ -278,22 +282,32 @@ func (mgr *ttlmanager) tick() { delete(mgr.entries, key) } } - err := batch.Commit() + err = batch.Commit() if err != nil { log.Error(err) } } -func (mgr *ttlmanager) setTTLs(keys []ds.Key, ttl time.Duration) { +func (mgr *ttlmanager) setTTLs(keys []ds.Key, ttl time.Duration, add bool) { mgr.Lock() defer mgr.Unlock() expiration := time.Now().Add(ttl) for _, key := range keys { - if ttl <= 0 { - delete(mgr.entries, key) - } else { - mgr.entries[key] = &ttlentry{TTL: ttl, ExpiresAt: expiration} + update := true + if add { + if entry, ok := mgr.entries[key]; ok { + if entry.ExpiresAt.After(expiration) { + update = false + } + } + } + if update { + if ttl <= 0 { + delete(mgr.entries, key) + } else { + mgr.entries[key] = &ttlentry{TTL: ttl, ExpiresAt: expiration} + } } } } From 1707dfc12afce7579a5b698620a1fbbee7a2003e Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 26 Jun 2018 18:00:36 -0400 Subject: [PATCH 0604/3965] Use nil instead of empty Multiaddr --- p2p/host/peerstore/addr_manager_ds.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index aa7b8a0cc7..c2c75f39cd 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -139,7 +139,7 @@ func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { results, err := mgr.ds.Query(q) if err != nil { log.Error(err) - return []ma.Multiaddr{} + return nil } var addrs []ma.Multiaddr From 251df966152cb41dc6b881c7f6209c8ccdfa7afd Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 26 Jun 2018 18:06:34 -0400 Subject: [PATCH 0605/3965] Handle batch Delete error --- p2p/host/peerstore/addr_manager_ds.go | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index c2c75f39cd..795239d13a 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -147,6 +147,7 @@ func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { addrbytes := result.Value.([]byte) addr, err := ma.NewMultiaddrBytes(addrbytes) if err != nil { + log.Error(err) continue } addrs = append(addrs, addr) @@ -205,7 +206,13 @@ func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { } for result := range results.Next() { - batch.Delete(ds.NewKey(result.Key)) + err := batch.Delete(ds.NewKey(result.Key)) + if err != nil { + // From inspectin badger, errors here signify a problem with + // the transaction as a whole, so we can log and abort. + log.Error(err) + return + } } if err = batch.Commit(); err != nil { log.Errorf("failed to clear addresses for peer %s: %s\n", p.Pretty(), err) From e46ef64067204f0e5209886fcffae2bb2353b9df Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 26 Jun 2018 18:10:10 -0400 Subject: [PATCH 0606/3965] Acquire full lock when ticking ttlmanager --- p2p/host/peerstore/addr_manager_ds.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 795239d13a..82c8637110 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -272,8 +272,8 @@ func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) * // To be called by TTL manager's coroutine only. func (mgr *ttlmanager) tick() { - mgr.RLock() - defer mgr.RUnlock() + mgr.Lock() + defer mgr.Unlock() now := time.Now() batch, err := mgr.ds.Batch() From 139fe246390ea3a8d6dfd39a024a90f4d91c580f Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 26 Jun 2018 19:06:05 -0400 Subject: [PATCH 0607/3965] Add ARCCache to DatastoreAddrManager --- p2p/host/peerstore/addr_manager_ds.go | 56 ++++++++++++++++++++----- p2p/host/peerstore/addr_manager_test.go | 5 ++- p2p/host/peerstore/peerstore.go | 12 ++++-- p2p/host/peerstore/peerstore_test.go | 5 ++- 4 files changed, 62 insertions(+), 16 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/addr_manager_ds.go index 82c8637110..a5b71abcab 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/addr_manager_ds.go @@ -5,6 +5,7 @@ import ( "sync" "time" + "github.com/hashicorp/golang-lru" ds "github.com/ipfs/go-datastore" "github.com/ipfs/go-datastore/query" "github.com/libp2p/go-libp2p-peer" @@ -18,6 +19,7 @@ var dsWriteRetries = 5 // DatastoreAddrManager is an address manager backed by a Datastore with both an // in-memory TTL manager and an in-memory address stream manager. type DatastoreAddrManager struct { + cache *lru.ARCCache ds ds.Batching ttlManager *ttlmanager subsManager *AddrSubManager @@ -26,13 +28,19 @@ type DatastoreAddrManager struct { // NewDatastoreAddrManager initializes a new DatastoreAddrManager given a // Datastore instance, a context for managing the TTL manager, and the interval // at which the TTL manager should sweep the Datastore. -func NewDatastoreAddrManager(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) *DatastoreAddrManager { +func NewDatastoreAddrManager(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) (*DatastoreAddrManager, error) { + cache, err := lru.NewARC(1024) + if err != nil { + return nil, err + } + mgr := &DatastoreAddrManager{ + cache: cache, ds: ds, - ttlManager: newTTLManager(ctx, ds, ttlInterval), + ttlManager: newTTLManager(ctx, ds, cache, ttlInterval), subsManager: NewAddrSubManager(), } - return mgr + return mgr, nil } // Stop will signal the TTL manager to stop and block until it returns. @@ -100,10 +108,18 @@ func (mgr *DatastoreAddrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t keys = append(keys, key) if ttl <= 0 { - batch.Delete(key) + if err := batch.Delete(key); err != nil { + log.Error(err) + } else { + mgr.cache.Remove(key) + } continue } - has, err := mgr.ds.Has(key) + + has := mgr.cache.Contains(key) + if !has { + has, err = mgr.ds.Has(key) + } if err != nil || !has { mgr.subsManager.BroadcastAddr(p, addr) } @@ -112,6 +128,8 @@ func (mgr *DatastoreAddrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t if !has { if err := batch.Put(key, addr.Bytes()); err != nil { log.Error(err) + } else { + mgr.cache.Add(key, addr.Bytes()) } } } @@ -135,7 +153,7 @@ func (mgr *DatastoreAddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, ne // Addrs Returns all of the non-expired addresses for a given peer. func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { prefix := ds.NewKey(p.Pretty()) - q := query.Query{Prefix: prefix.String()} + q := query.Query{Prefix: prefix.String(), KeysOnly: true} results, err := mgr.ds.Query(q) if err != nil { log.Error(err) @@ -144,7 +162,17 @@ func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { var addrs []ma.Multiaddr for result := range results.Next() { - addrbytes := result.Value.([]byte) + key := ds.RawKey(result.Key) + var addri interface{} + addri, ok := mgr.cache.Get(key) + if !ok { + addri, err = mgr.ds.Get(key) + if err != nil { + log.Error(err) + continue + } + } + addrbytes := addri.([]byte) addr, err := ma.NewMultiaddrBytes(addrbytes) if err != nil { log.Error(err) @@ -158,7 +186,7 @@ func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { // Peers returns all of the peer IDs for which the AddrBook has addresses. func (mgr *DatastoreAddrManager) Peers() []peer.ID { - q := query.Query{} + q := query.Query{KeysOnly: true} results, err := mgr.ds.Query(q) if err != nil { log.Error(err) @@ -193,7 +221,7 @@ func (mgr *DatastoreAddrManager) AddrStream(ctx context.Context, p peer.ID) <-ch func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { prefix := ds.NewKey(p.Pretty()) for i := 0; i < dsWriteRetries; i++ { - q := query.Query{Prefix: prefix.String()} + q := query.Query{Prefix: prefix.String(), KeysOnly: true} results, err := mgr.ds.Query(q) if err != nil { log.Error(err) @@ -206,13 +234,15 @@ func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { } for result := range results.Next() { - err := batch.Delete(ds.NewKey(result.Key)) + key := ds.NewKey(result.Key) + err := batch.Delete(key) if err != nil { // From inspectin badger, errors here signify a problem with // the transaction as a whole, so we can log and abort. log.Error(err) return } + mgr.cache.Remove(key) } if err = batch.Commit(); err != nil { log.Errorf("failed to clear addresses for peer %s: %s\n", p.Pretty(), err) @@ -239,9 +269,10 @@ type ttlmanager struct { cancel context.CancelFunc ticker *time.Ticker ds ds.Batching + cache *lru.ARCCache } -func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) *ttlmanager { +func newTTLManager(parent context.Context, d ds.Datastore, c *lru.ARCCache, tick time.Duration) *ttlmanager { ctx, cancel := context.WithCancel(parent) batching, ok := d.(ds.Batching) if !ok { @@ -253,6 +284,7 @@ func newTTLManager(parent context.Context, d ds.Datastore, tick time.Duration) * cancel: cancel, ticker: time.NewTicker(tick), ds: batching, + cache: c, } go func() { @@ -285,6 +317,8 @@ func (mgr *ttlmanager) tick() { if entry.ExpiresAt.Before(now) { if err := batch.Delete(key); err != nil { log.Error(err) + } else { + mgr.cache.Remove(key) } delete(mgr.entries, key) } diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 6c26ed4a95..4adf84df84 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -71,7 +71,10 @@ func setupBadgerDatastore(t *testing.T) (datastore.Batching, func()) { func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func()) { ds, closeDB := setupBadgerDatastore(t) - mgr := NewDatastoreAddrManager(context.Background(), ds, 100*time.Microsecond) + mgr, err := NewDatastoreAddrManager(context.Background(), ds, 100*time.Microsecond) + if err != nil { + t.Fatal(err) + } closer := func() { mgr.Stop() closeDB() diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 38296a3d8d..84f1a68211 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -204,13 +204,19 @@ func NewPeerstore() Peerstore { // NewPeerstoreDatastore creates a threadsafe collection of peers backed by a // Datastore to prevent excess memory pressure. -func NewPeerstoreDatastore(ctx context.Context, ds datastore.Batching) Peerstore { - return &peerstore{ +func NewPeerstoreDatastore(ctx context.Context, ds datastore.Batching) (Peerstore, error) { + addrBook, err := NewDatastoreAddrManager(ctx, ds, time.Second) + if err != nil { + return nil, err + } + + ps := &peerstore{ keybook: newKeybook(), metrics: NewMetrics(), - AddrBook: NewDatastoreAddrManager(ctx, ds, time.Second), + AddrBook: addrBook, ds: make(map[string]interface{}), } + return ps, nil } func (ps *peerstore) Put(p peer.ID, key string, val interface{}) error { diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 647cfcf64a..1799ce89b5 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -41,7 +41,10 @@ func runTestWithPeerstores(t *testing.T, testFunc func(*testing.T, Peerstore)) { func setupDatastorePeerstore(t *testing.T) (Peerstore, func()) { ds, closeDB := setupBadgerDatastore(t) ctx, cancel := context.WithCancel(context.Background()) - ps := NewPeerstoreDatastore(ctx, ds) + ps, err := NewPeerstoreDatastore(ctx, ds) + if err != nil { + t.Fatal(err) + } closer := func() { cancel() closeDB() From a19c6fefc014edc6d59a726c6c2729c1aca4f3f5 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 26 Jun 2018 20:10:17 -0400 Subject: [PATCH 0608/3965] Update benchmarks to run against both peerstores --- p2p/host/peerstore/addr_manager_test.go | 2 +- p2p/host/peerstore/peerstore_test.go | 39 +++++++++++++++---------- 2 files changed, 25 insertions(+), 16 deletions(-) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 4adf84df84..8498bc9368 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -53,7 +53,7 @@ func testHas(t *testing.T, exp, act []ma.Multiaddr) { } } -func setupBadgerDatastore(t *testing.T) (datastore.Batching, func()) { +func setupBadgerDatastore(t testing.TB) (datastore.Batching, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 1799ce89b5..58b0619e2f 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -38,7 +38,7 @@ func runTestWithPeerstores(t *testing.T, testFunc func(*testing.T, Peerstore)) { testFunc(t, ps2) } -func setupDatastorePeerstore(t *testing.T) (Peerstore, func()) { +func setupDatastorePeerstore(t testing.TB) (Peerstore, func()) { ds, closeDB := setupBadgerDatastore(t) ctx, cancel := context.WithCancel(context.Background()) ps, err := NewPeerstoreDatastore(ctx, ds) @@ -326,21 +326,30 @@ func BenchmarkBasicPeerstore(b *testing.B) { } } +func benchmarkPeerstoreRateLimited(ps Peerstore) func(*testing.B) { + return func(b *testing.B) { + producer := make(chan *peerpair, 100) + addrs := make(chan *peerpair, 100) + + go rateLimitedAddressProducer(b, addrs, producer, 10000, time.Millisecond, 500*time.Microsecond) + + b.ResetTimer() + for { + pp, ok := <-addrs + if !ok { + break + } + pid := peer.ID(pp.ID) + ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) + } + } +} + func BenchmarkBasicPeerstoreRateLimited(b *testing.B) { ps := NewPeerstore() + b.Run("PeerstoreBasic", benchmarkPeerstoreRateLimited(ps)) - producer := make(chan *peerpair, 100) - addrs := make(chan *peerpair, 100) - - go rateLimitedAddressProducer(b, addrs, producer, 5000, time.Millisecond, 500*time.Microsecond) - - b.ResetTimer() - for { - pp, ok := <-addrs - if !ok { - break - } - pid := peer.ID(pp.ID) - ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) - } + dsps, closer := setupDatastorePeerstore(b) + defer closer() + b.Run("PeerstoreDatastore", benchmarkPeerstoreRateLimited(dsps)) } From df805d22f0d867af0e2323d6ecb674d45ab4228d Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 26 Jun 2018 20:18:25 -0400 Subject: [PATCH 0609/3965] Update benchmark timings to stress peerstores more --- p2p/host/peerstore/benchmark_utils.go | 3 ++- p2p/host/peerstore/peerstore_test.go | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/benchmark_utils.go b/p2p/host/peerstore/benchmark_utils.go index 9e812f9c8d..c66fd07a7c 100644 --- a/p2p/host/peerstore/benchmark_utils.go +++ b/p2p/host/peerstore/benchmark_utils.go @@ -6,10 +6,11 @@ import ( "testing" "time" + "math/rand" + "github.com/mr-tron/base58/base58" ma "github.com/multiformats/go-multiaddr" mh "github.com/multiformats/go-multihash" - "math/rand" ) type peerpair struct { diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 58b0619e2f..e6d9407704 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -331,7 +331,7 @@ func benchmarkPeerstoreRateLimited(ps Peerstore) func(*testing.B) { producer := make(chan *peerpair, 100) addrs := make(chan *peerpair, 100) - go rateLimitedAddressProducer(b, addrs, producer, 10000, time.Millisecond, 500*time.Microsecond) + go rateLimitedAddressProducer(b, addrs, producer, 60000, 100*time.Microsecond, 100*time.Microsecond) b.ResetTimer() for { From 60f3f27d2d1ee0b52f2eb0d692906c768f1583de Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 27 Jun 2018 13:25:16 -0400 Subject: [PATCH 0610/3965] Run benchmarks 10s, report rates --- p2p/host/peerstore/benchmark_utils.go | 17 ++++++---- p2p/host/peerstore/peerstore_test.go | 46 ++++++++++++++++++++------- 2 files changed, 45 insertions(+), 18 deletions(-) diff --git a/p2p/host/peerstore/benchmark_utils.go b/p2p/host/peerstore/benchmark_utils.go index c66fd07a7c..7921090820 100644 --- a/p2p/host/peerstore/benchmark_utils.go +++ b/p2p/host/peerstore/benchmark_utils.go @@ -45,18 +45,23 @@ func randomPeer(b *testing.B) *peerpair { return &peerpair{b58ID, addr} } -func addressProducer(b *testing.B, addrs chan *peerpair, n int) { +func addressProducer(b *testing.B, addrs chan *peerpair, done <-chan time.Time) { defer close(addrs) - for i := 0; i < n; i++ { - p := randomPeer(b) - addrs <- p + for { + select { + case <-done: + return + default: + p := randomPeer(b) + addrs <- p + } } } -func rateLimitedAddressProducer(b *testing.B, addrs chan *peerpair, producer chan *peerpair, n int, avgTime time.Duration, errBound time.Duration) { +func rateLimitedAddressProducer(b *testing.B, addrs chan *peerpair, producer chan *peerpair, done <-chan time.Time, avgTime time.Duration, errBound time.Duration) { defer close(addrs) - go addressProducer(b, producer, n) + go addressProducer(b, producer, done) eb := int64(errBound) for { diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index e6d9407704..0efbf4cbe6 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -309,39 +309,61 @@ func TestBasicPeerstore(t *testing.T) { runTestWithPeerstores(t, testBasicPeerstore) } -func BenchmarkBasicPeerstore(b *testing.B) { - ps := NewPeerstore() - - addrs := make(chan *peerpair, 100) +func benchmarkPeerstore(ps Peerstore) func(*testing.B) { + return func(b *testing.B) { + addrs := make(chan *peerpair, 100) - go addressProducer(b, addrs, 50000) + go addressProducer(b, addrs, time.After(10*time.Second)) - for { - pp, ok := <-addrs - if !ok { - break + start := time.Now() + count := 0 + b.ResetTimer() + for { + pp, ok := <-addrs + if !ok { + break + } + count++ + pid := peer.ID(pp.ID) + ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) } - pid := peer.ID(pp.ID) - ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) + elapsed := time.Since(start) + rate := float64(count) / elapsed.Seconds() + b.Logf("Added %d addresses in %s, rate: %f addrs/s\n", count, elapsed, rate) } } +func BenchmarkPeerstore(b *testing.B) { + ps := NewPeerstore() + b.Run("PeerstoreBasic", benchmarkPeerstore(ps)) + + dsps, closer := setupDatastorePeerstore(b) + defer closer() + b.Run("PeerstoreDatastore", benchmarkPeerstore(dsps)) +} + func benchmarkPeerstoreRateLimited(ps Peerstore) func(*testing.B) { return func(b *testing.B) { producer := make(chan *peerpair, 100) addrs := make(chan *peerpair, 100) - go rateLimitedAddressProducer(b, addrs, producer, 60000, 100*time.Microsecond, 100*time.Microsecond) + go rateLimitedAddressProducer(b, addrs, producer, time.After(10*time.Second), 100*time.Microsecond, 100*time.Microsecond) + start := time.Now() + count := 0 b.ResetTimer() for { pp, ok := <-addrs if !ok { break } + count++ pid := peer.ID(pp.ID) ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) } + elapsed := time.Since(start) + rate := float64(count) / elapsed.Seconds() + b.Logf("Added %d addresses in %s, rate: %f addrs/s\n", count, elapsed, rate) } } From 5b1ae1595f4baea40c1f8331df0bc8cbd48aaca3 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Thu, 28 Jun 2018 09:31:36 -0400 Subject: [PATCH 0611/3965] Use b.N in benchmarks --- p2p/host/peerstore/benchmark_utils.go | 17 ++++++----------- p2p/host/peerstore/peerstore_test.go | 18 +++--------------- 2 files changed, 9 insertions(+), 26 deletions(-) diff --git a/p2p/host/peerstore/benchmark_utils.go b/p2p/host/peerstore/benchmark_utils.go index 7921090820..b85a7633a2 100644 --- a/p2p/host/peerstore/benchmark_utils.go +++ b/p2p/host/peerstore/benchmark_utils.go @@ -45,23 +45,18 @@ func randomPeer(b *testing.B) *peerpair { return &peerpair{b58ID, addr} } -func addressProducer(b *testing.B, addrs chan *peerpair, done <-chan time.Time) { +func addressProducer(b *testing.B, addrs chan *peerpair) { defer close(addrs) - for { - select { - case <-done: - return - default: - p := randomPeer(b) - addrs <- p - } + for i := 0; i < b.N; i++ { + p := randomPeer(b) + addrs <- p } } -func rateLimitedAddressProducer(b *testing.B, addrs chan *peerpair, producer chan *peerpair, done <-chan time.Time, avgTime time.Duration, errBound time.Duration) { +func rateLimitedAddressProducer(b *testing.B, addrs chan *peerpair, producer chan *peerpair, avgTime time.Duration, errBound time.Duration) { defer close(addrs) - go addressProducer(b, producer, done) + go addressProducer(b, producer) eb := int64(errBound) for { diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 0efbf4cbe6..5ad4d481d8 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -313,23 +313,17 @@ func benchmarkPeerstore(ps Peerstore) func(*testing.B) { return func(b *testing.B) { addrs := make(chan *peerpair, 100) - go addressProducer(b, addrs, time.After(10*time.Second)) + go addressProducer(b, addrs) - start := time.Now() - count := 0 b.ResetTimer() for { pp, ok := <-addrs if !ok { break } - count++ pid := peer.ID(pp.ID) ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) } - elapsed := time.Since(start) - rate := float64(count) / elapsed.Seconds() - b.Logf("Added %d addresses in %s, rate: %f addrs/s\n", count, elapsed, rate) } } @@ -347,27 +341,21 @@ func benchmarkPeerstoreRateLimited(ps Peerstore) func(*testing.B) { producer := make(chan *peerpair, 100) addrs := make(chan *peerpair, 100) - go rateLimitedAddressProducer(b, addrs, producer, time.After(10*time.Second), 100*time.Microsecond, 100*time.Microsecond) + go rateLimitedAddressProducer(b, addrs, producer, 100*time.Microsecond, 100*time.Microsecond) - start := time.Now() - count := 0 b.ResetTimer() for { pp, ok := <-addrs if !ok { break } - count++ pid := peer.ID(pp.ID) ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) } - elapsed := time.Since(start) - rate := float64(count) / elapsed.Seconds() - b.Logf("Added %d addresses in %s, rate: %f addrs/s\n", count, elapsed, rate) } } -func BenchmarkBasicPeerstoreRateLimited(b *testing.B) { +func BenchmarkPeerstoreRateLimited(b *testing.B) { ps := NewPeerstore() b.Run("PeerstoreBasic", benchmarkPeerstoreRateLimited(ps)) From 7660dfaa6a239273bf4e115f9c37678f80ad376d Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 24 Jul 2018 19:48:16 -0400 Subject: [PATCH 0612/3965] Implement proper benchmark with b.N --- p2p/host/peerstore/benchmark_utils.go | 33 +++++------------------ p2p/host/peerstore/peerstore_test.go | 39 ++++----------------------- 2 files changed, 12 insertions(+), 60 deletions(-) diff --git a/p2p/host/peerstore/benchmark_utils.go b/p2p/host/peerstore/benchmark_utils.go index b85a7633a2..e838894366 100644 --- a/p2p/host/peerstore/benchmark_utils.go +++ b/p2p/host/peerstore/benchmark_utils.go @@ -1,12 +1,10 @@ package peerstore import ( + "context" cr "crypto/rand" "fmt" "testing" - "time" - - "math/rand" "github.com/mr-tron/base58/base58" ma "github.com/multiformats/go-multiaddr" @@ -45,31 +43,14 @@ func randomPeer(b *testing.B) *peerpair { return &peerpair{b58ID, addr} } -func addressProducer(b *testing.B, addrs chan *peerpair) { - defer close(addrs) - for i := 0; i < b.N; i++ { - p := randomPeer(b) - addrs <- p - } -} - -func rateLimitedAddressProducer(b *testing.B, addrs chan *peerpair, producer chan *peerpair, avgTime time.Duration, errBound time.Duration) { +func addressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair) { defer close(addrs) - - go addressProducer(b, producer) - - eb := int64(errBound) for { - addr, ok := <-producer - if !ok { - break - } - addrs <- addr - wiggle := time.Duration(0) - if eb > 0 { - wiggle = time.Duration(rand.Int63n(eb*2) - eb) + p := randomPeer(b) + select { + case addrs <- p: + case <-ctx.Done(): + return } - sleepDur := avgTime + wiggle - time.Sleep(sleepDur) } } diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index 5ad4d481d8..d402be07d8 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -313,17 +313,16 @@ func benchmarkPeerstore(ps Peerstore) func(*testing.B) { return func(b *testing.B) { addrs := make(chan *peerpair, 100) - go addressProducer(b, addrs) + ctx, cancel := context.WithCancel(context.Background()) + go addressProducer(ctx, b, addrs) b.ResetTimer() - for { - pp, ok := <-addrs - if !ok { - break - } + for i := 0; i < b.N; i++ { + pp := <-addrs pid := peer.ID(pp.ID) ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) } + cancel() } } @@ -335,31 +334,3 @@ func BenchmarkPeerstore(b *testing.B) { defer closer() b.Run("PeerstoreDatastore", benchmarkPeerstore(dsps)) } - -func benchmarkPeerstoreRateLimited(ps Peerstore) func(*testing.B) { - return func(b *testing.B) { - producer := make(chan *peerpair, 100) - addrs := make(chan *peerpair, 100) - - go rateLimitedAddressProducer(b, addrs, producer, 100*time.Microsecond, 100*time.Microsecond) - - b.ResetTimer() - for { - pp, ok := <-addrs - if !ok { - break - } - pid := peer.ID(pp.ID) - ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) - } - } -} - -func BenchmarkPeerstoreRateLimited(b *testing.B) { - ps := NewPeerstore() - b.Run("PeerstoreBasic", benchmarkPeerstoreRateLimited(ps)) - - dsps, closer := setupDatastorePeerstore(b) - defer closer() - b.Run("PeerstoreDatastore", benchmarkPeerstoreRateLimited(dsps)) -} From 6d836d558002287beaea3202f2a4366399ffb5a1 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 15 Jun 2018 19:07:37 -0400 Subject: [PATCH 0613/3965] Add support for Stat() to Swarm Stream and Conn --- p2p/net/swarm/swarm.go | 3 ++- p2p/net/swarm/swarm_conn.go | 16 +++++++++++++--- p2p/net/swarm/swarm_dial.go | 3 ++- p2p/net/swarm/swarm_listen.go | 3 ++- p2p/net/swarm/swarm_stream.go | 10 ++++++++++ 5 files changed, 29 insertions(+), 6 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 7d795abd86..78d8e127dc 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -165,7 +165,7 @@ func (s *Swarm) Process() goprocess.Process { return s.proc } -func (s *Swarm) addConn(tc transport.Conn) (*Conn, error) { +func (s *Swarm) addConn(tc transport.Conn, stat inet.Stat) (*Conn, error) { // The underlying transport (or the dialer) *should* filter it's own // connections but we should double check anyways. raddr := tc.RemoteMultiaddr() @@ -197,6 +197,7 @@ func (s *Swarm) addConn(tc transport.Conn) (*Conn, error) { c := &Conn{ conn: tc, swarm: s, + stat: stat, } c.streams.m = make(map[*Stream]struct{}) s.conns.m[p] = append(s.conns.m[p], c) diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 5b2420cd5f..05e6bfca1a 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -33,6 +33,8 @@ type Conn struct { sync.Mutex m map[*Stream]struct{} } + + stat inet.Stat } // Close closes this connection. @@ -98,7 +100,8 @@ func (c *Conn) start() { } c.swarm.refs.Add(1) go func() { - s, err := c.addStream(ts) + stat := inet.Stat{Direction: inet.DirInbound} + s, err := c.addStream(ts, stat) // Don't defer this. We don't want to block // swarm shutdown on the connection handler. @@ -158,16 +161,22 @@ func (c *Conn) RemotePublicKey() ic.PubKey { return c.conn.RemotePublicKey() } +// Stat returns metadata pertaining to this connection +func (c *Conn) Stat() inet.Stat { + return c.stat +} + // NewStream returns a new Stream from this connection func (c *Conn) NewStream() (inet.Stream, error) { ts, err := c.conn.OpenStream() if err != nil { return nil, err } - return c.addStream(ts) + stat := inet.Stat{Direction: inet.DirOutbound} + return c.addStream(ts, stat) } -func (c *Conn) addStream(ts smux.Stream) (*Stream, error) { +func (c *Conn) addStream(ts smux.Stream, stat inet.Stat) (*Stream, error) { c.streams.Lock() // Are we still online? if c.streams.m == nil { @@ -180,6 +189,7 @@ func (c *Conn) addStream(ts smux.Stream) (*Stream, error) { s := &Stream{ stream: ts, conn: c, + stat: stat, } c.streams.m[s] = struct{}{} diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 2af26ed1a2..246935ab58 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -325,7 +325,8 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { "localAddr": connC.LocalMultiaddr(), "remoteAddr": connC.RemoteMultiaddr(), } - swarmC, err := s.addConn(connC) + stat := inet.Stat{Direction: inet.DirOutbound} + swarmC, err := s.addConn(connC, stat) if err != nil { logdial["error"] = err.Error() connC.Close() // close the connection. didn't work out :( diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index df0a97f248..cb56b42f06 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -81,7 +81,8 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { s.refs.Add(1) go func() { defer s.refs.Done() - _, err := s.addConn(c) + stat := inet.Stat{Direction: inet.DirInbound} + _, err := s.addConn(c, stat) if err != nil { // Probably just means that the swarm has been closed. log.Warningf("add conn failed: ", err) diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 30b44bd28e..754dbd50e7 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -22,6 +22,9 @@ const ( streamReset ) +// Validate Stream conforms to the go-libp2p-net Stream interface +var _ inet.Stream = &Stream{} + // Stream is the stream type used by swarm. In general, you won't use this type // directly. type Stream struct { @@ -36,6 +39,8 @@ type Stream struct { notifyLk sync.Mutex protocol atomic.Value + + stat inet.Stat } func (s *Stream) String() string { @@ -165,3 +170,8 @@ func (s *Stream) SetReadDeadline(t time.Time) error { func (s *Stream) SetWriteDeadline(t time.Time) error { return s.stream.SetWriteDeadline(t) } + +// Stat returns metadata information for this stream. +func (s *Stream) Stat() inet.Stat { + return s.stat +} From 3f9733aeb6bd764420c91d5fc1c31b6e871acaef Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Sun, 24 Jun 2018 20:54:56 -0400 Subject: [PATCH 0614/3965] Update addStream/Conn to accept a Direction --- p2p/net/swarm/swarm.go | 3 ++- p2p/net/swarm/swarm_conn.go | 9 ++++----- p2p/net/swarm/swarm_dial.go | 3 +-- p2p/net/swarm/swarm_listen.go | 3 +-- 4 files changed, 8 insertions(+), 10 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 78d8e127dc..c8d6fa02d1 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -165,7 +165,7 @@ func (s *Swarm) Process() goprocess.Process { return s.proc } -func (s *Swarm) addConn(tc transport.Conn, stat inet.Stat) (*Conn, error) { +func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { // The underlying transport (or the dialer) *should* filter it's own // connections but we should double check anyways. raddr := tc.RemoteMultiaddr() @@ -194,6 +194,7 @@ func (s *Swarm) addConn(tc transport.Conn, stat inet.Stat) (*Conn, error) { } // Wrap and register the connection. + stat := inet.Stat{Direction: dir} c := &Conn{ conn: tc, swarm: s, diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 05e6bfca1a..26a7794d15 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -100,8 +100,7 @@ func (c *Conn) start() { } c.swarm.refs.Add(1) go func() { - stat := inet.Stat{Direction: inet.DirInbound} - s, err := c.addStream(ts, stat) + s, err := c.addStream(ts, inet.DirInbound) // Don't defer this. We don't want to block // swarm shutdown on the connection handler. @@ -172,11 +171,10 @@ func (c *Conn) NewStream() (inet.Stream, error) { if err != nil { return nil, err } - stat := inet.Stat{Direction: inet.DirOutbound} - return c.addStream(ts, stat) + return c.addStream(ts, inet.DirOutbound) } -func (c *Conn) addStream(ts smux.Stream, stat inet.Stat) (*Stream, error) { +func (c *Conn) addStream(ts smux.Stream, dir inet.Direction) (*Stream, error) { c.streams.Lock() // Are we still online? if c.streams.m == nil { @@ -186,6 +184,7 @@ func (c *Conn) addStream(ts smux.Stream, stat inet.Stat) (*Stream, error) { } // Wrap and register the stream. + stat := inet.Stat{Direction: dir} s := &Stream{ stream: ts, conn: c, diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 246935ab58..64971b07e9 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -325,8 +325,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { "localAddr": connC.LocalMultiaddr(), "remoteAddr": connC.RemoteMultiaddr(), } - stat := inet.Stat{Direction: inet.DirOutbound} - swarmC, err := s.addConn(connC, stat) + swarmC, err := s.addConn(connC, inet.DirOutbound) if err != nil { logdial["error"] = err.Error() connC.Close() // close the connection. didn't work out :( diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index cb56b42f06..31b4a04208 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -81,8 +81,7 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { s.refs.Add(1) go func() { defer s.refs.Done() - stat := inet.Stat{Direction: inet.DirInbound} - _, err := s.addConn(c, stat) + _, err := s.addConn(c, inet.DirInbound) if err != nil { // Probably just means that the swarm has been closed. log.Warningf("add conn failed: ", err) From 10a44085f511bf75724cccd6f874edb44f9eb4e4 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 27 Aug 2018 17:28:07 -0700 Subject: [PATCH 0615/3965] gx: update go-libp2p-net --- package.json | 52 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/package.json b/package.json index 3665899df4..6d42c01c10 100644 --- a/package.json +++ b/package.json @@ -61,9 +61,9 @@ }, { "author": "whyrusleeping", - "hash": "QmWKKkNLFRcznF5vDqt2eeRsnQqQhwbjVf8zJ9KC2RXrzN", + "hash": "QmWri2HWdxHjWBUermhWy7QWJqN1cV8Gd1QbDiB5m86f1H", "name": "go-libp2p-secio", - "version": "2.0.7" + "version": "2.0.8" }, { "author": "whyrusleeping", @@ -73,15 +73,15 @@ }, { "author": "whyrusleeping", - "hash": "QmYr9RHifaqHTFZdAsUPLmiMAi2oNeEqA48AFKxXJAsLpJ", + "hash": "QmcDUyb52N62J8ZamGgUWUyWc1MtuCBce7WFA4D9xA6cwF", "name": "go-libp2p-transport", - "version": "3.0.7" + "version": "3.0.8" }, { "author": "whyrusleeping", - "hash": "QmcK89iqkFV8TqpRUgx1481YZbhjPFnBjqkpBQJfJqmSfm", + "hash": "Qmcw9fndogcYwyGs4a5TPDbnZPBLxvtrBZzpvyyVDzxDWT", "name": "go-tcp-transport", - "version": "2.0.7" + "version": "2.0.8" }, { "author": "whyrusleeping", @@ -103,9 +103,9 @@ }, { "author": "whyrusleeping", - "hash": "QmX5J1q63BrrDTbpcHifrFbxH3cMZsvaNajy6u3zCpzBXs", + "hash": "QmQSbtGXCyNrj34LWL8EgXyNNYDZ8r3SwQcpW5pPxVhLnM", "name": "go-libp2p-net", - "version": "3.0.7" + "version": "3.0.8" }, { "author": "whyrusleeping", @@ -115,15 +115,15 @@ }, { "author": "whyrusleeping", - "hash": "QmRRCrNRs4qxotXx7WJT6SpCvSNEhXvyBcVjXY2K71pcjE", + "hash": "QmPMtD39NN63AEUNghk1LFQcTLcCmYL8MtRzdv8BRUsC4Z", "name": "go-libp2p-host", - "version": "3.0.6" + "version": "3.0.7" }, { "author": "whyrusleeping", - "hash": "QmPWNZRUybw3nwJH3mpkrwB97YEQmXRkzvyh34rpJiih6Q", + "hash": "QmYSM6PKnCe9YVPNMisfpoBmczzHkA7h5Wrnc36DtdJhGo", "name": "go-libp2p-swarm", - "version": "3.0.8" + "version": "3.0.9" }, { "author": "whyrusleeping", @@ -139,9 +139,9 @@ }, { "author": "whyrusleeping", - "hash": "QmRAsmNHjzKVuscipvMVjB3NTyLW1HTBiSP8LaKD1iUJmH", + "hash": "QmbKt1yZEnCWUmQhZR9BMiXK4e4FoaqnieAARz1s88MSzf", "name": "go-libp2p-blankhost", - "version": "0.3.6" + "version": "0.3.7" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "Qmbc6WjgbkaYhPw5dd6X2RLGiJz854dPPKkyZf52vE2PTS", + "hash": "QmVYqPFBGi5wiuxpKxf4mUePEDdfXh8H7HKtq4mH8SeqML", "name": "go-libp2p-circuit", - "version": "2.1.9" + "version": "2.1.10" }, { "author": "lgierth", @@ -181,9 +181,9 @@ }, { "author": "why", - "hash": "QmUPz6FCzCCU7sTY9Sore5NGSUA8YSF2yMkLPjDFq7wGqD", + "hash": "QmVz2p8ZVZ5GcWPNWGs2HZHiZyHumZcJpQdMRpxkMDhc2C", "name": "go-libp2p-interface-connmgr", - "version": "0.0.13" + "version": "0.0.14" }, { "author": "whyrusleeping", @@ -193,21 +193,21 @@ }, { "author": "whyrusleeping", - "hash": "QmTDv8XeECZaYBZXo5SAEGueMiPTszAkGZQ5YiM5KCAJai", + "hash": "QmSQhZyqoEepHPHr6kCpAoYVHzvVkTHAAXnNqFEFbVaaQr", "name": "go-ws-transport", - "version": "2.0.7" + "version": "2.0.8" }, { "author": "stebalien", - "hash": "Qma6UXLMHjdVFExQZLYqdb5KAesbnoXuthQzovrwRZ64fG", + "hash": "QmSbkqfiFmJCdczVQ7mkFZf5FUUNpuP5Ne2LxY2htXGtrZ", "name": "go-conn-security-multistream", - "version": "0.1.6" + "version": "0.1.7" }, { "author": "Stebalien", - "hash": "QmcGgFLHMFLcNMMvxsBC5LeLqubLR5djxjShVU3koVMtVq", + "hash": "QmNPQvc3jUA7L6oHB8en9PittPGazFMdoveN2oycHL6yWF", "name": "go-conn-security", - "version": "0.1.7" + "version": "0.1.8" }, { "author": "libp2p", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "QmfNvpHX396fhMeauERV6eFnSJg78rUjhjpFf1JvbjxaYM", + "hash": "QmefQrpDSYX6jQRtUyhcASFVBDkoAsDTPXemyxGMzA3phK", "name": "go-libp2p-transport-upgrader", - "version": "0.1.7" + "version": "0.1.8" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", From 5a704d2daa5aa4a3afd70ea61f08d5c4482e3fe3 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Thu, 28 Jun 2018 17:57:19 -0400 Subject: [PATCH 0616/3965] Add Stat to mock net conns and streams --- p2p/net/mock/mock_conn.go | 9 ++++++++- p2p/net/mock/mock_link.go | 8 ++++---- p2p/net/mock/mock_stream.go | 8 +++++++- 3 files changed, 19 insertions(+), 6 deletions(-) diff --git a/p2p/net/mock/mock_conn.go b/p2p/net/mock/mock_conn.go index 68e9c840cc..f4cf10358b 100644 --- a/p2p/net/mock/mock_conn.go +++ b/p2p/net/mock/mock_conn.go @@ -29,14 +29,16 @@ type conn struct { rconn *conn // counterpart streams list.List proc process.Process + stat inet.Stat sync.RWMutex } -func newConn(ln, rn *peernet, l *link) *conn { +func newConn(ln, rn *peernet, l *link, dir inet.Direction) *conn { c := &conn{net: ln, link: l} c.local = ln.peer c.remote = rn.peer + c.stat = inet.Stat{Direction: dir} c.localAddr = ln.ps.Addrs(ln.peer)[0] c.remoteAddr = rn.ps.Addrs(rn.peer)[0] @@ -155,3 +157,8 @@ func (c *conn) RemotePeer() peer.ID { func (c *conn) RemotePublicKey() ic.PubKey { return c.remotePubKey } + +// Stat returns metadata about the connection +func (c *conn) Stat() inet.Stat { + return c.stat +} diff --git a/p2p/net/mock/mock_link.go b/p2p/net/mock/mock_link.go index d4d605310d..9c6049ac88 100644 --- a/p2p/net/mock/mock_link.go +++ b/p2p/net/mock/mock_link.go @@ -33,8 +33,8 @@ func (l *link) newConnPair(dialer *peernet) (*conn, *conn) { l.RLock() defer l.RUnlock() - c1 := newConn(l.nets[0], l.nets[1], l) - c2 := newConn(l.nets[1], l.nets[0], l) + c1 := newConn(l.nets[0], l.nets[1], l, inet.DirOutbound) + c2 := newConn(l.nets[1], l.nets[0], l, inet.DirInbound) c1.rconn = c2 c2.rconn = c1 @@ -48,8 +48,8 @@ func (l *link) newStreamPair() (*stream, *stream) { ra, wb := io.Pipe() rb, wa := io.Pipe() - sa := NewStream(wa, ra) - sb := NewStream(wb, rb) + sa := NewStream(wa, ra, inet.DirOutbound) + sb := NewStream(wb, rb, inet.DirInbound) return sa, sb } diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index 6d056c8ea0..4f0c395fec 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -25,6 +25,7 @@ type stream struct { writeErr error protocol protocol.ID + stat inet.Stat } var ErrReset error = errors.New("stream reset") @@ -35,7 +36,7 @@ type transportObject struct { arrivalTime time.Time } -func NewStream(w *io.PipeWriter, r *io.PipeReader) *stream { +func NewStream(w *io.PipeWriter, r *io.PipeReader, dir inet.Direction) *stream { s := &stream{ read: r, write: w, @@ -43,6 +44,7 @@ func NewStream(w *io.PipeWriter, r *io.PipeReader) *stream { close: make(chan struct{}, 1), closed: make(chan struct{}), toDeliver: make(chan *transportObject), + stat: inet.Stat{Direction: dir}, } go s.transport() @@ -66,6 +68,10 @@ func (s *stream) Protocol() protocol.ID { return s.protocol } +func (s *stream) Stat() inet.Stat { + return s.stat +} + func (s *stream) SetProtocol(proto protocol.ID) { s.protocol = proto } From 1f1aca1c28616769519c4f16d4d7362a109ecffe Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 27 Aug 2018 17:42:02 -0700 Subject: [PATCH 0617/3965] gx publish 6.0.10 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 16d7d87965..cbf4243541 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.9: QmQiaskfWpdRJ4x2spEQjPFTUkEB87KDYu91qnNYBqvvcX +6.0.10: QmVM6VuGaWcAaYjxG2om6XxMmpP3Rt9rw4nbMXVNYAPLhS diff --git a/package.json b/package.json index 6d42c01c10..9ef415a006 100644 --- a/package.json +++ b/package.json @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.9" + "version": "6.0.10" } From 88f8b4c38af708e5096c238f0370cfc53cd5aea9 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 28 Aug 2018 12:17:51 +0700 Subject: [PATCH 0618/3965] update quic-go to 0.10.0 --- p2p/transport/quic/transport.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index d8c984af27..c58b7b0a4e 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -19,7 +19,7 @@ import ( ) var quicConfig = &quic.Config{ - Versions: []quic.VersionNumber{quic.VersionMilestone0_9_0}, + Versions: []quic.VersionNumber{quic.VersionMilestone0_10_0}, MaxIncomingStreams: 1000, MaxIncomingUniStreams: -1, // disable unidirectional streams MaxReceiveStreamFlowControlWindow: 3 * (1 << 20), // 3 MB From cd2735b8b517ed99993b122597866e46a681a64d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rau=CC=81l=20Kripalani?= Date: Wed, 29 Aug 2018 15:12:41 +0100 Subject: [PATCH 0619/3965] refactor peerstore (+) 1. Create interface.go with abstract interfaces. 2. Extract memory-based keybook into file. 3. Spin off PeerMetadata interface. --- p2p/host/peerstore/addr_manager.go | 8 +- p2p/host/peerstore/addr_manager_test.go | 1 - .../addr_manager.go} | 50 ++-- p2p/host/peerstore/ds/peerstore.go | 20 ++ p2p/host/peerstore/interface.go | 93 +++++++ p2p/host/peerstore/keybook.go | 91 +++++++ p2p/host/peerstore/metadata.go | 43 ++++ p2p/host/peerstore/peerinfo.go | 2 +- p2p/host/peerstore/peerstore.go | 242 ++---------------- p2p/host/peerstore/peerstore_test.go | 3 +- 10 files changed, 310 insertions(+), 243 deletions(-) rename p2p/host/peerstore/{addr_manager_ds.go => ds/addr_manager.go} (83%) create mode 100644 p2p/host/peerstore/ds/peerstore.go create mode 100644 p2p/host/peerstore/interface.go create mode 100644 p2p/host/peerstore/keybook.go create mode 100644 p2p/host/peerstore/metadata.go diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/addr_manager.go index c20e54cbb3..b0db2b60e9 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/addr_manager.go @@ -7,11 +7,14 @@ import ( "sync" "time" + logging "github.com/ipfs/go-log" "github.com/libp2p/go-libp2p-peer" "github.com/libp2p/go-libp2p-peerstore/addr" ma "github.com/multiformats/go-multiaddr" ) +var log = logging.Logger("peerstore") + var ( // TempAddrTTL is the ttl used for a short lived address @@ -54,6 +57,8 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { type addrSlice []expiringAddr +var _ AddrBook = (*AddrManager)(nil) + // AddrManager manages addresses. // The zero-value is ready to be used. type AddrManager struct { @@ -74,7 +79,7 @@ func (mgr *AddrManager) init() { } } -func (mgr *AddrManager) Peers() []peer.ID { +func (mgr *AddrManager) AddrsPeers() []peer.ID { mgr.addrmu.Lock() defer mgr.addrmu.Unlock() if mgr.addrs == nil { @@ -197,6 +202,7 @@ func (mgr *AddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time } exp := time.Now().Add(newTTL) + // TODO: RK - Shorthand. for i := range addrs { aexp := &addrs[i] if oldTTL == aexp.TTL { diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go index 8498bc9368..50eda9bc08 100644 --- a/p2p/host/peerstore/addr_manager_test.go +++ b/p2p/host/peerstore/addr_manager_test.go @@ -8,7 +8,6 @@ import ( "time" "github.com/ipfs/go-datastore" - "github.com/ipfs/go-ds-badger" "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/host/peerstore/addr_manager_ds.go b/p2p/host/peerstore/ds/addr_manager.go similarity index 83% rename from p2p/host/peerstore/addr_manager_ds.go rename to p2p/host/peerstore/ds/addr_manager.go index a5b71abcab..e00023de50 100644 --- a/p2p/host/peerstore/addr_manager_ds.go +++ b/p2p/host/peerstore/ds/addr_manager.go @@ -1,4 +1,4 @@ -package peerstore +package ds import ( "context" @@ -8,43 +8,49 @@ import ( "github.com/hashicorp/golang-lru" ds "github.com/ipfs/go-datastore" "github.com/ipfs/go-datastore/query" + logging "github.com/ipfs/go-log" "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" mh "github.com/multiformats/go-multihash" ) +var log = logging.Logger("peerstore/ds") + // Number of times to retry transactional writes var dsWriteRetries = 5 -// DatastoreAddrManager is an address manager backed by a Datastore with both an +var _ pstore.AddrBook = (*addrManager)(nil) + +// addrManager is an address manager backed by a Datastore with both an // in-memory TTL manager and an in-memory address stream manager. -type DatastoreAddrManager struct { +type addrManager struct { cache *lru.ARCCache ds ds.Batching ttlManager *ttlmanager - subsManager *AddrSubManager + subsManager *pstore.AddrSubManager } -// NewDatastoreAddrManager initializes a new DatastoreAddrManager given a -// Datastore instance, a context for managing the TTL manager, and the interval -// at which the TTL manager should sweep the Datastore. -func NewDatastoreAddrManager(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) (*DatastoreAddrManager, error) { +// NewAddrManager initializes a new address manager given a +// Datastore instance, a context for managing the TTL manager, +// and the interval at which the TTL manager should sweep the Datastore. +func NewAddrManager(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) (*addrManager, error) { cache, err := lru.NewARC(1024) if err != nil { return nil, err } - mgr := &DatastoreAddrManager{ + mgr := &addrManager{ cache: cache, ds: ds, ttlManager: newTTLManager(ctx, ds, cache, ttlInterval), - subsManager: NewAddrSubManager(), + subsManager: pstore.NewAddrSubManager(), } return mgr, nil } // Stop will signal the TTL manager to stop and block until it returns. -func (mgr *DatastoreAddrManager) Stop() { +func (mgr *addrManager) Stop() { mgr.ttlManager.cancel() } @@ -62,12 +68,12 @@ func peerIDFromKey(key ds.Key) (peer.ID, error) { } // AddAddr will add a new address if it's not already in the AddrBook. -func (mgr *DatastoreAddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { +func (mgr *addrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } // AddAddrs will add many new addresses if they're not already in the AddrBook. -func (mgr *DatastoreAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { +func (mgr *addrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { if ttl <= 0 { return } @@ -76,16 +82,16 @@ func (mgr *DatastoreAddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t } // SetAddr will add or update the TTL of an address in the AddrBook. -func (mgr *DatastoreAddrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { +func (mgr *addrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } // SetAddrs will add or update the TTLs of addresses in the AddrBook. -func (mgr *DatastoreAddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { +func (mgr *addrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { mgr.setAddrs(p, addrs, ttl, false) } -func (mgr *DatastoreAddrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, add bool) { +func (mgr *addrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, add bool) { for i := 0; i < dsWriteRetries; i++ { // keys to add to the TTL manager var keys []ds.Key @@ -145,13 +151,13 @@ func (mgr *DatastoreAddrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl t // UpdateAddrs will update any addresses for a given peer and TTL combination to // have a new TTL. -func (mgr *DatastoreAddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { +func (mgr *addrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { prefix := ds.NewKey(p.Pretty()) mgr.ttlManager.updateTTLs(prefix, oldTTL, newTTL) } // Addrs Returns all of the non-expired addresses for a given peer. -func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { +func (mgr *addrManager) Addrs(p peer.ID) []ma.Multiaddr { prefix := ds.NewKey(p.Pretty()) q := query.Query{Prefix: prefix.String(), KeysOnly: true} results, err := mgr.ds.Query(q) @@ -185,7 +191,7 @@ func (mgr *DatastoreAddrManager) Addrs(p peer.ID) []ma.Multiaddr { } // Peers returns all of the peer IDs for which the AddrBook has addresses. -func (mgr *DatastoreAddrManager) Peers() []peer.ID { +func (mgr *addrManager) AddrsPeers() []peer.ID { q := query.Query{KeysOnly: true} results, err := mgr.ds.Query(q) if err != nil { @@ -212,13 +218,13 @@ func (mgr *DatastoreAddrManager) Peers() []peer.ID { // AddrStream returns a channel on which all new addresses discovered for a // given peer ID will be published. -func (mgr *DatastoreAddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { +func (mgr *addrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { initial := mgr.Addrs(p) return mgr.subsManager.AddrStream(ctx, p, initial) } // ClearAddrs will delete all known addresses for a peer ID. -func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { +func (mgr *addrManager) ClearAddrs(p peer.ID) { prefix := ds.NewKey(p.Pretty()) for i := 0; i < dsWriteRetries; i++ { q := query.Query{Prefix: prefix.String(), KeysOnly: true} @@ -254,8 +260,6 @@ func (mgr *DatastoreAddrManager) ClearAddrs(p peer.ID) { log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), dsWriteRetries) } -// ttlmanager - type ttlentry struct { TTL time.Duration ExpiresAt time.Time diff --git a/p2p/host/peerstore/ds/peerstore.go b/p2p/host/peerstore/ds/peerstore.go new file mode 100644 index 0000000000..f48263379b --- /dev/null +++ b/p2p/host/peerstore/ds/peerstore.go @@ -0,0 +1,20 @@ +package ds + +import ( + "context" + "time" + + "github.com/ipfs/go-datastore" + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +// NewPeerstore creates a peerstore backed by the provided persistent datastore. +func NewPeerstore(ctx context.Context, ds datastore.Batching) (pstore.Peerstore, error) { + addrBook, err := NewAddrManager(ctx, ds, time.Second) + if err != nil { + return nil, err + } + + ps := pstore.NewPeerstoreWith(pstore.NewKeybook(), addrBook, pstore.NewPeerMetadata()) + return ps, nil +} diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go new file mode 100644 index 0000000000..e0c0a6233c --- /dev/null +++ b/p2p/host/peerstore/interface.go @@ -0,0 +1,93 @@ +package peerstore + +import ( + "context" + "time" + + "gx/ipfs/QmVmDhyTTUcQXFD1rRQ64fGLMSAoaQvNH3hwuaCFAPq2hy/errors" + + ic "github.com/libp2p/go-libp2p-crypto" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" +) + +var ErrNotFound = errors.New("item not found") + +// Peerstore provides a threadsafe store of Peer related +// information. +type Peerstore interface { + AddrBook + KeyBook + PeerMetadata + Metrics + + // PeerInfo returns a peer.PeerInfo struct for given peer.ID. + // This is a small slice of the information Peerstore has on + // that peer, useful to other services. + PeerInfo(peer.ID) PeerInfo + + GetProtocols(peer.ID) ([]string, error) + AddProtocols(peer.ID, ...string) error + SetProtocols(peer.ID, ...string) error + SupportsProtocols(peer.ID, ...string) ([]string, error) + + // Peers returns all of the peer IDs stored across all inner stores. + Peers() []peer.ID +} + +type PeerMetadata interface { + // Get/Put is a simple registry for other peer-related key/value pairs. + // if we find something we use often, it should become its own set of + // methods. this is a last resort. + Get(p peer.ID, key string) (interface{}, error) + Put(p peer.ID, key string, val interface{}) error +} + +// AddrBook is an interface that fits the new AddrManager. I'm patching +// it up in here to avoid changing a ton of the codebase. +type AddrBook interface { + + // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) + AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) + + // AddAddrs gives AddrManager addresses to use, with a given ttl + // (time-to-live), after which the address is no longer valid. + // If the manager has a longer TTL, the operation is a no-op for that address + AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) + + // SetAddr calls mgr.SetAddrs(p, addr, ttl) + SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) + + // SetAddrs sets the ttl on addresses. This clears any TTL there previously. + // This is used when we receive the best estimate of the validity of an address. + SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) + + // UpdateAddrs updates the addresses associated with the given peer that have + // the given oldTTL to have the given newTTL. + UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) + + // Addresses returns all known (and valid) addresses for a given peer + Addrs(p peer.ID) []ma.Multiaddr + + // AddrStream returns a channel that gets all addresses for a given + // peer sent on it. If new addresses are added after the call is made + // they will be sent along through the channel as well. + AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr + + // ClearAddresses removes all previously stored addresses + ClearAddrs(p peer.ID) + + // Peers returns all of the peer IDs stored in the AddrBook + AddrsPeers() []peer.ID +} + +// KeyBook tracks the Public keys of Peers. +type KeyBook interface { + KeyBookPeers() []peer.ID + + PubKey(peer.ID) ic.PubKey + AddPubKey(peer.ID, ic.PubKey) error + + PrivKey(peer.ID) ic.PrivKey + AddPrivKey(peer.ID, ic.PrivKey) error +} diff --git a/p2p/host/peerstore/keybook.go b/p2p/host/peerstore/keybook.go new file mode 100644 index 0000000000..1a9e6b95b8 --- /dev/null +++ b/p2p/host/peerstore/keybook.go @@ -0,0 +1,91 @@ +package peerstore + +import ( + "errors" + "sync" + + ic "github.com/libp2p/go-libp2p-crypto" + "github.com/libp2p/go-libp2p-peer" +) + +type memoryKeyBook struct { + sync.RWMutex // same lock. wont happen a ton. + pks map[peer.ID]ic.PubKey + sks map[peer.ID]ic.PrivKey +} + +var _ KeyBook = (*memoryKeyBook)(nil) + +// noop new, but in the future we may want to do some init work. +func NewKeybook() KeyBook { + return &memoryKeyBook{ + pks: map[peer.ID]ic.PubKey{}, + sks: map[peer.ID]ic.PrivKey{}, + } +} + +func (mkb *memoryKeyBook) KeyBookPeers() []peer.ID { + mkb.RLock() + ps := make([]peer.ID, 0, len(mkb.pks)+len(mkb.sks)) + for p := range mkb.pks { + ps = append(ps, p) + } + for p := range mkb.sks { + if _, found := mkb.pks[p]; !found { + ps = append(ps, p) + } + } + mkb.RUnlock() + return ps +} + +func (mkb *memoryKeyBook) PubKey(p peer.ID) ic.PubKey { + mkb.RLock() + pk := mkb.pks[p] + mkb.RUnlock() + if pk != nil { + return pk + } + pk, err := p.ExtractPublicKey() + if err == nil && pk != nil { + mkb.Lock() + mkb.pks[p] = pk + mkb.Unlock() + } + return pk +} + +func (mkb *memoryKeyBook) AddPubKey(p peer.ID, pk ic.PubKey) error { + // check it's correct first + if !p.MatchesPublicKey(pk) { + return errors.New("ID does not match PublicKey") + } + + mkb.Lock() + mkb.pks[p] = pk + mkb.Unlock() + return nil +} + +func (mkb *memoryKeyBook) PrivKey(p peer.ID) ic.PrivKey { + mkb.RLock() + sk := mkb.sks[p] + mkb.RUnlock() + return sk +} + +func (mkb *memoryKeyBook) AddPrivKey(p peer.ID, sk ic.PrivKey) error { + if sk == nil { + return errors.New("sk is nil (PrivKey)") + } + + // check it's correct first + if !p.MatchesPrivateKey(sk) { + return errors.New("ID does not match PrivateKey") + } + + mkb.Lock() + mkb.sks[p] = sk + mkb.Unlock() + return nil +} diff --git a/p2p/host/peerstore/metadata.go b/p2p/host/peerstore/metadata.go new file mode 100644 index 0000000000..1f6f5ce3de --- /dev/null +++ b/p2p/host/peerstore/metadata.go @@ -0,0 +1,43 @@ +package peerstore + +import ( + "sync" + + "github.com/libp2p/go-libp2p-peer" +) + +type memoryPeerMetadata struct { + // store other data, like versions + //ds ds.ThreadSafeDatastore + // TODO: use a datastore for this + ds map[string]interface{} + dslock sync.Mutex +} + +func NewPeerMetadata() PeerMetadata { + return &memoryPeerMetadata{ + ds: make(map[string]interface{}), + } +} + +func (ps *memoryPeerMetadata) Put(p peer.ID, key string, val interface{}) error { + //dsk := ds.NewKey(string(p) + "/" + key) + //return ps.ds.Put(dsk, val) + ps.dslock.Lock() + defer ps.dslock.Unlock() + ps.ds[string(p)+"/"+key] = val + return nil +} + +func (ps *memoryPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { + //dsk := ds.NewKey(string(p) + "/" + key) + //return ps.ds.Get(dsk) + + ps.dslock.Lock() + defer ps.dslock.Unlock() + i, ok := ps.ds[string(p)+"/"+key] + if !ok { + return nil, ErrNotFound + } + return i, nil +} diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index cb216264b5..f41b9f85ad 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -58,7 +58,7 @@ func InfoFromP2pAddr(m ma.Multiaddr) (*PeerInfo, error) { } func InfoToP2pAddrs(pi *PeerInfo) ([]ma.Multiaddr, error) { - addrs := []ma.Multiaddr{} + var addrs []ma.Multiaddr tpl := "/" + ma.ProtocolWithCode(ma.P_IPFS).Name + "/" for _, addr := range pi.Addrs { p2paddr, err := ma.NewMultiaddr(tpl + peer.IDB58Encode(pi.ID)) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 84f1a68211..b127c4e30e 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -1,254 +1,64 @@ package peerstore import ( - "context" - "errors" "fmt" "sync" "time" - ic "github.com/libp2p/go-libp2p-crypto" - - //ds "github.com/jbenet/go-datastore" - //dssync "github.com/jbenet/go-datastore/sync" - "github.com/ipfs/go-datastore" - logging "github.com/ipfs/go-log" "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" ) -var log = logging.Logger("peerstore") - const ( // AddressTTL is the expiration time of addresses. AddressTTL = time.Hour ) -// Peerstore provides a threadsafe store of Peer related -// information. -type Peerstore interface { - AddrBook - KeyBook - Metrics - - // PeerInfo returns a peer.PeerInfo struct for given peer.ID. - // This is a small slice of the information Peerstore has on - // that peer, useful to other services. - PeerInfo(peer.ID) PeerInfo - - // Get/Put is a simple registry for other peer-related key/value pairs. - // if we find something we use often, it should become its own set of - // methods. this is a last resort. - Get(id peer.ID, key string) (interface{}, error) - Put(id peer.ID, key string, val interface{}) error - - GetProtocols(peer.ID) ([]string, error) - AddProtocols(peer.ID, ...string) error - SetProtocols(peer.ID, ...string) error - SupportsProtocols(peer.ID, ...string) ([]string, error) -} - -// AddrBook is an interface that fits the new AddrManager. I'm patching -// it up in here to avoid changing a ton of the codebase. -type AddrBook interface { - - // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) - AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) - - // AddAddrs gives AddrManager addresses to use, with a given ttl - // (time-to-live), after which the address is no longer valid. - // If the manager has a longer TTL, the operation is a no-op for that address - AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) - - // SetAddr calls mgr.SetAddrs(p, addr, ttl) - SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) - - // SetAddrs sets the ttl on addresses. This clears any TTL there previously. - // This is used when we receive the best estimate of the validity of an address. - SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) - - // UpdateAddrs updates the addresses associated with the given peer that have - // the given oldTTL to have the given newTTL. - UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) - - // Addresses returns all known (and valid) addresses for a given peer - Addrs(p peer.ID) []ma.Multiaddr - - // AddrStream returns a channel that gets all addresses for a given - // peer sent on it. If new addresses are added after the call is made - // they will be sent along through the channel as well. - AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr - - // ClearAddresses removes all previously stored addresses - ClearAddrs(p peer.ID) - - // Peers returns all of the peer IDs stored in the AddrBook - Peers() []peer.ID -} - -// KeyBook tracks the Public keys of Peers. -type KeyBook interface { - PubKey(peer.ID) ic.PubKey - AddPubKey(peer.ID, ic.PubKey) error - - PrivKey(peer.ID) ic.PrivKey - AddPrivKey(peer.ID, ic.PrivKey) error -} - -type keybook struct { - pks map[peer.ID]ic.PubKey - sks map[peer.ID]ic.PrivKey - - sync.RWMutex // same lock. wont happen a ton. -} - -func newKeybook() *keybook { - return &keybook{ - pks: map[peer.ID]ic.PubKey{}, - sks: map[peer.ID]ic.PrivKey{}, - } -} - -func (kb *keybook) Peers() []peer.ID { - kb.RLock() - ps := make([]peer.ID, 0, len(kb.pks)+len(kb.sks)) - for p := range kb.pks { - ps = append(ps, p) - } - for p := range kb.sks { - if _, found := kb.pks[p]; !found { - ps = append(ps, p) - } - } - kb.RUnlock() - return ps -} - -func (kb *keybook) PubKey(p peer.ID) ic.PubKey { - kb.RLock() - pk := kb.pks[p] - kb.RUnlock() - if pk != nil { - return pk - } - pk, err := p.ExtractPublicKey() - if err == nil && pk != nil { - kb.Lock() - kb.pks[p] = pk - kb.Unlock() - } - return pk -} - -func (kb *keybook) AddPubKey(p peer.ID, pk ic.PubKey) error { - - // check it's correct first - if !p.MatchesPublicKey(pk) { - return errors.New("ID does not match PublicKey") - } - - kb.Lock() - kb.pks[p] = pk - kb.Unlock() - return nil -} - -func (kb *keybook) PrivKey(p peer.ID) ic.PrivKey { - kb.RLock() - sk := kb.sks[p] - kb.RUnlock() - return sk -} - -func (kb *keybook) AddPrivKey(p peer.ID, sk ic.PrivKey) error { - - if sk == nil { - return errors.New("sk is nil (PrivKey)") - } - - // check it's correct first - if !p.MatchesPrivateKey(sk) { - return errors.New("ID does not match PrivateKey") - } - - kb.Lock() - kb.sks[p] = sk - kb.Unlock() - return nil -} +var _ Peerstore = (*peerstore)(nil) type peerstore struct { - *keybook - *metrics - AddrBook + Metrics - // store other data, like versions - //ds ds.ThreadSafeDatastore - // TODO: use a datastore for this - ds map[string]interface{} - dslock sync.Mutex + KeyBook + AddrBook + PeerMetadata // lock for protocol information, separate from datastore lock protolock sync.Mutex } // NewPeerstore creates a threadsafe collection of peers. +// +// TODO: (RK) move in-memory AddrBook, KeyBook and PeerMetadata to mem +// package. Adapt consumers (IPFS) to instantiate mem.NewPeerstore(). +// +// Preserving this function for backwards compatibility is not possible, as +// we would have to import the mem package, which in turn imports this one, thus +// creating a cycle. +// +// DEPRECATED: use an implementation-specific peerstore. func NewPeerstore() Peerstore { - return &peerstore{ - keybook: newKeybook(), - metrics: NewMetrics(), - AddrBook: &AddrManager{}, - ds: make(map[string]interface{}), - } + return NewPeerstoreWith(NewKeybook(), &AddrManager{}, NewPeerMetadata()) } -// NewPeerstoreDatastore creates a threadsafe collection of peers backed by a -// Datastore to prevent excess memory pressure. -func NewPeerstoreDatastore(ctx context.Context, ds datastore.Batching) (Peerstore, error) { - addrBook, err := NewDatastoreAddrManager(ctx, ds, time.Second) - if err != nil { - return nil, err - } - - ps := &peerstore{ - keybook: newKeybook(), - metrics: NewMetrics(), - AddrBook: addrBook, - ds: make(map[string]interface{}), - } - return ps, nil -} - -func (ps *peerstore) Put(p peer.ID, key string, val interface{}) error { - //dsk := ds.NewKey(string(p) + "/" + key) - //return ps.ds.Put(dsk, val) - ps.dslock.Lock() - defer ps.dslock.Unlock() - ps.ds[string(p)+"/"+key] = val - return nil -} - -var ErrNotFound = errors.New("item not found") - -func (ps *peerstore) Get(p peer.ID, key string) (interface{}, error) { - //dsk := ds.NewKey(string(p) + "/" + key) - //return ps.ds.Get(dsk) +// NewPeerstore creates a threadsafe collection of peers. +func NewPeerstoreWith(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { + return &peerstore{ + KeyBook: kb, + PeerMetadata: md, - ps.dslock.Lock() - defer ps.dslock.Unlock() - i, ok := ps.ds[string(p)+"/"+key] - if !ok { - return nil, ErrNotFound + // TODO: normalise the naming here AddrBook vs AddrManager. + // TODO: add a constructor NewAddrBook and make the struct private. + AddrBook: ab, + Metrics: NewMetrics(), } - return i, nil } func (ps *peerstore) Peers() []peer.ID { set := map[peer.ID]struct{}{} - for _, p := range ps.keybook.Peers() { + for _, p := range ps.KeyBookPeers() { set[p] = struct{}{} } - for _, p := range ps.AddrBook.Peers() { + for _, p := range ps.AddrsPeers() { set[p] = struct{}{} } diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go index d402be07d8..ff3bf3b93d 100644 --- a/p2p/host/peerstore/peerstore_test.go +++ b/p2p/host/peerstore/peerstore_test.go @@ -10,6 +10,7 @@ import ( "github.com/libp2p/go-libp2p-crypto" "github.com/libp2p/go-libp2p-peer" + dspstore "github.com/libp2p/go-libp2p-peerstore/ds" ma "github.com/multiformats/go-multiaddr" ) @@ -41,7 +42,7 @@ func runTestWithPeerstores(t *testing.T, testFunc func(*testing.T, Peerstore)) { func setupDatastorePeerstore(t testing.TB) (Peerstore, func()) { ds, closeDB := setupBadgerDatastore(t) ctx, cancel := context.WithCancel(context.Background()) - ps, err := NewPeerstoreDatastore(ctx, ds) + ps, err := dspstore.NewPeerstore(ctx, ds) if err != nil { t.Fatal(err) } From b8d574c8c3ecf742168080de3ee597a7de1f5b69 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 29 Aug 2018 15:54:02 -0700 Subject: [PATCH 0620/3965] use peer.IDFromBytes instead of parsing multihash directly This is technically the "more correct" option and will make it easier to change both peer IDs and multihash. --- p2p/protocol/internal/circuitv1-deprecated/util.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index 71f4684ea8..5812dbad72 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -12,7 +12,6 @@ import ( peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" - mh "github.com/multiformats/go-multihash" ) func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { @@ -20,7 +19,7 @@ func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { return pstore.PeerInfo{}, errors.New("nil peer") } - h, err := mh.Cast(p.Id) + id, err := peer.IDFromBytes(p.Id) if err != nil { return pstore.PeerInfo{}, err } @@ -33,7 +32,7 @@ func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { } } - return pstore.PeerInfo{ID: peer.ID(h), Addrs: addrs}, nil + return pstore.PeerInfo{ID: id, Addrs: addrs}, nil } func peerInfoToPeer(pi pstore.PeerInfo) *pb.CircuitRelay_Peer { From d1f0a910813905718c6798a83500f6f24387099f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rau=CC=81l=20Kripalani?= Date: Thu, 30 Aug 2018 13:44:50 +0100 Subject: [PATCH 0621/3965] refactor tests into suites. This enables each implementation to subject itself to a common test suite. --- p2p/host/peerstore/addr_manager_test.go | 307 ---------------- p2p/host/peerstore/ds/ds_test.go | 68 ++++ p2p/host/peerstore/inmem_test.go | 25 ++ p2p/host/peerstore/peerstore_test.go | 337 ------------------ p2p/host/peerstore/test/addr_manager_suite.go | 279 +++++++++++++++ .../peerstore/{ => test}/benchmark_utils.go | 2 +- p2p/host/peerstore/test/peerstore_suite.go | 316 ++++++++++++++++ .../test/{utils.go => test_utils.go} | 2 +- 8 files changed, 690 insertions(+), 646 deletions(-) delete mode 100644 p2p/host/peerstore/addr_manager_test.go create mode 100644 p2p/host/peerstore/ds/ds_test.go create mode 100644 p2p/host/peerstore/inmem_test.go delete mode 100644 p2p/host/peerstore/peerstore_test.go create mode 100644 p2p/host/peerstore/test/addr_manager_suite.go rename p2p/host/peerstore/{ => test}/benchmark_utils.go (97%) create mode 100644 p2p/host/peerstore/test/peerstore_suite.go rename p2p/host/peerstore/test/{utils.go => test_utils.go} (97%) diff --git a/p2p/host/peerstore/addr_manager_test.go b/p2p/host/peerstore/addr_manager_test.go deleted file mode 100644 index 50eda9bc08..0000000000 --- a/p2p/host/peerstore/addr_manager_test.go +++ /dev/null @@ -1,307 +0,0 @@ -package peerstore - -import ( - "context" - "io/ioutil" - "os" - "testing" - "time" - - "github.com/ipfs/go-datastore" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" -) - -func IDS(t *testing.T, ids string) peer.ID { - t.Helper() - id, err := peer.IDB58Decode(ids) - if err != nil { - t.Fatalf("id %q is bad: %s", ids, err) - } - return id -} - -func MA(t *testing.T, m string) ma.Multiaddr { - t.Helper() - maddr, err := ma.NewMultiaddr(m) - if err != nil { - t.Fatal(err) - } - return maddr -} - -func testHas(t *testing.T, exp, act []ma.Multiaddr) { - t.Helper() - if len(exp) != len(act) { - t.Fatalf("lengths not the same. expected %d, got %d\n", len(exp), len(act)) - } - - for _, a := range exp { - found := false - - for _, b := range act { - if a.Equal(b) { - found = true - break - } - } - - if !found { - t.Fatalf("expected address %s not found", a) - } - } -} - -func setupBadgerDatastore(t testing.TB) (datastore.Batching, func()) { - dataPath, err := ioutil.TempDir(os.TempDir(), "badger") - if err != nil { - t.Fatal(err) - } - ds, err := badger.NewDatastore(dataPath, nil) - if err != nil { - t.Fatal(err) - } - closer := func() { - ds.Close() - os.RemoveAll(dataPath) - } - return ds, closer -} - -func setupDatastoreAddrManager(t *testing.T) (*DatastoreAddrManager, func()) { - ds, closeDB := setupBadgerDatastore(t) - mgr, err := NewDatastoreAddrManager(context.Background(), ds, 100*time.Microsecond) - if err != nil { - t.Fatal(err) - } - closer := func() { - mgr.Stop() - closeDB() - } - return mgr, closer -} - -func runTestWithAddrManagers(t *testing.T, test func(*testing.T, AddrBook)) { - t.Log("AddrManager") - mgr1 := &AddrManager{} - test(t, mgr1) - - t.Log("DatastoreAddrManager") - mgr2, closer2 := setupDatastoreAddrManager(t) - defer closer2() - test(t, mgr2) -} - -func testAddresses(t *testing.T, m AddrBook) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := IDS(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") - id3 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ6Kn") - id4 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Kn") - id5 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Km") - - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - ma21 := MA(t, "/ip4/2.2.3.2/tcp/1111") - ma22 := MA(t, "/ip4/2.2.3.2/tcp/2222") - ma31 := MA(t, "/ip4/3.2.3.3/tcp/1111") - ma32 := MA(t, "/ip4/3.2.3.3/tcp/2222") - ma33 := MA(t, "/ip4/3.2.3.3/tcp/3333") - ma41 := MA(t, "/ip4/4.2.3.3/tcp/1111") - ma42 := MA(t, "/ip4/4.2.3.3/tcp/2222") - ma43 := MA(t, "/ip4/4.2.3.3/tcp/3333") - ma44 := MA(t, "/ip4/4.2.3.3/tcp/4444") - ma51 := MA(t, "/ip4/5.2.3.3/tcp/1111") - ma52 := MA(t, "/ip4/5.2.3.3/tcp/2222") - ma53 := MA(t, "/ip4/5.2.3.3/tcp/3333") - ma54 := MA(t, "/ip4/5.2.3.3/tcp/4444") - ma55 := MA(t, "/ip4/5.2.3.3/tcp/5555") - - ttl := time.Hour - m.AddAddr(id1, ma11, ttl) - - m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) - m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) // idempotency - - m.AddAddr(id3, ma31, ttl) - m.AddAddr(id3, ma32, ttl) - m.AddAddr(id3, ma33, ttl) - m.AddAddr(id3, ma33, ttl) // idempotency - m.AddAddr(id3, ma33, ttl) - - m.AddAddrs(id4, []ma.Multiaddr{ma41, ma42, ma43, ma44}, ttl) // multiple - - m.AddAddrs(id5, []ma.Multiaddr{ma21, ma22}, ttl) // clearing - m.AddAddrs(id5, []ma.Multiaddr{ma41, ma42, ma43, ma44}, ttl) // clearing - m.ClearAddrs(id5) - m.AddAddrs(id5, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, ttl) // clearing - - // test the Addresses return value - testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - testHas(t, []ma.Multiaddr{ma31, ma32, ma33}, m.Addrs(id3)) - testHas(t, []ma.Multiaddr{ma41, ma42, ma43, ma44}, m.Addrs(id4)) - testHas(t, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, m.Addrs(id5)) -} - -func TestAddresses(t *testing.T) { - runTestWithAddrManagers(t, testAddresses) -} - -func testAddressesExpire(t *testing.T, m AddrBook) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := MA(t, "/ip4/2.2.3.2/tcp/2222") - ma13 := MA(t, "/ip4/3.2.3.3/tcp/3333") - ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") - ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") - - m.AddAddr(id1, ma11, time.Hour) - m.AddAddr(id1, ma12, time.Hour) - m.AddAddr(id1, ma13, time.Hour) - m.AddAddr(id2, ma24, time.Hour) - m.AddAddr(id2, ma25, time.Hour) - - testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) - - m.SetAddr(id1, ma11, 2*time.Hour) - m.SetAddr(id1, ma12, 2*time.Hour) - m.SetAddr(id1, ma13, 2*time.Hour) - m.SetAddr(id2, ma24, 2*time.Hour) - m.SetAddr(id2, ma25, 2*time.Hour) - - testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) - - m.SetAddr(id1, ma11, time.Millisecond) - <-time.After(time.Millisecond * 2) - testHas(t, []ma.Multiaddr{ma12, ma13}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) - - m.SetAddr(id1, ma13, time.Millisecond) - <-time.After(time.Millisecond * 2) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) - - m.SetAddr(id2, ma24, time.Millisecond) - <-time.After(time.Millisecond * 2) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma25}, m.Addrs(id2)) - - m.SetAddr(id2, ma25, time.Millisecond) - <-time.After(time.Millisecond * 2) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, nil, m.Addrs(id2)) - - m.SetAddr(id1, ma12, time.Millisecond) - <-time.After(time.Millisecond * 2) - testHas(t, nil, m.Addrs(id1)) - testHas(t, nil, m.Addrs(id2)) -} - -func TestAddressesExpire(t *testing.T) { - runTestWithAddrManagers(t, testAddressesExpire) -} - -func testClearWorks(t *testing.T, m AddrBook) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := MA(t, "/ip4/2.2.3.2/tcp/2222") - ma13 := MA(t, "/ip4/3.2.3.3/tcp/3333") - ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") - ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") - - m.AddAddr(id1, ma11, time.Hour) - m.AddAddr(id1, ma12, time.Hour) - m.AddAddr(id1, ma13, time.Hour) - m.AddAddr(id2, ma24, time.Hour) - m.AddAddr(id2, ma25, time.Hour) - - testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) - - m.ClearAddrs(id1) - m.ClearAddrs(id2) - - testHas(t, nil, m.Addrs(id1)) - testHas(t, nil, m.Addrs(id2)) -} - -func TestClearWorks(t *testing.T) { - runTestWithAddrManagers(t, testClearWorks) -} - -func testSetNegativeTTLClears(t *testing.T, m AddrBook) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - - m.SetAddr(id1, ma11, time.Hour) - - testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) - - m.SetAddr(id1, ma11, -1) - - testHas(t, nil, m.Addrs(id1)) -} -func TestSetNegativeTTLClears(t *testing.T) { - runTestWithAddrManagers(t, testSetNegativeTTLClears) -} - -func testUpdateTTLs(t *testing.T, m AddrBook) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := MA(t, "/ip4/1.2.3.1/tcp/1112") - ma21 := MA(t, "/ip4/1.2.3.1/tcp/1121") - ma22 := MA(t, "/ip4/1.2.3.1/tcp/1122") - - // Shouldn't panic. - m.UpdateAddrs(id1, time.Hour, time.Minute) - - m.SetAddr(id1, ma11, time.Hour) - m.SetAddr(id1, ma12, time.Minute) - - // Shouldn't panic. - m.UpdateAddrs(id2, time.Hour, time.Minute) - - m.SetAddr(id2, ma21, time.Hour) - m.SetAddr(id2, ma22, time.Minute) - - testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - - m.UpdateAddrs(id1, time.Hour, time.Second) - - testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - - time.Sleep(1200 * time.Millisecond) - - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - - m.UpdateAddrs(id2, time.Hour, time.Second) - - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - - time.Sleep(1200 * time.Millisecond) - - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma22}, m.Addrs(id2)) -} - -func TestUpdateTTLs(t *testing.T) { - runTestWithAddrManagers(t, testUpdateTTLs) -} - -func testNilAddrsDontBreak(t *testing.T, m AddrBook) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - m.SetAddr(id1, nil, time.Hour) - m.AddAddr(id1, nil, time.Hour) -} - -func TestNilAddrsDontBreak(t *testing.T) { - runTestWithAddrManagers(t, testNilAddrsDontBreak) -} diff --git a/p2p/host/peerstore/ds/ds_test.go b/p2p/host/peerstore/ds/ds_test.go new file mode 100644 index 0000000000..6f7adcd78e --- /dev/null +++ b/p2p/host/peerstore/ds/ds_test.go @@ -0,0 +1,68 @@ +package ds + +import ( + "context" + "io/ioutil" + "os" + "testing" + "time" + + "github.com/ipfs/go-datastore" + "github.com/ipfs/go-ds-badger" + "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-peerstore/test" +) + +func setupBadgerDatastore(t testing.TB) (datastore.Batching, func()) { + dataPath, err := ioutil.TempDir(os.TempDir(), "badger") + if err != nil { + t.Fatal(err) + } + ds, err := badger.NewDatastore(dataPath, nil) + if err != nil { + t.Fatal(err) + } + closer := func() { + ds.Close() + os.RemoveAll(dataPath) + } + return ds, closer +} + +func newPeerstoreFactory(tb testing.TB) test.PeerstoreFactory { + return func() (peerstore.Peerstore, func()) { + ds, closeFunc := setupBadgerDatastore(tb) + + ps, err := NewPeerstore(context.Background(), ds) + if err != nil { + tb.Fatal(err) + } + + return ps, closeFunc + } +} + +func TestBadgerDsPeerstore(t *testing.T) { + test.TestPeerstore(t, newPeerstoreFactory(t)) +} + +func TestBadgerDsAddrBook(t *testing.T) { + test.TestAddrMgr(t, func() (peerstore.AddrBook, func()) { + ds, closeDB := setupBadgerDatastore(t) + + mgr, err := NewAddrManager(context.Background(), ds, 100*time.Microsecond) + if err != nil { + t.Fatal(err) + } + + closeFunc := func() { + mgr.Stop() + closeDB() + } + return mgr, closeFunc + }) +} + +func BenchmarkBadgerDsPeerstore(b *testing.B) { + test.BenchmarkPeerstore(b, newPeerstoreFactory(b)) +} \ No newline at end of file diff --git a/p2p/host/peerstore/inmem_test.go b/p2p/host/peerstore/inmem_test.go new file mode 100644 index 0000000000..6b1a012c0f --- /dev/null +++ b/p2p/host/peerstore/inmem_test.go @@ -0,0 +1,25 @@ +package peerstore + +import ( + "testing" + + "github.com/libp2p/go-libp2p-peerstore/test" +) + +func TestInMemoryPeerstore(t *testing.T) { + test.TestPeerstore(t, func() (Peerstore, func()) { + return NewPeerstore(), nil + }) +} + +func TestInMemoryAddrMgr(t *testing.T) { + test.TestAddrMgr(t, func() (AddrBook, func()) { + return &AddrManager{}, nil + }) +} + +func BenchmarkInMemoryPeerstore(b *testing.B) { + test.BenchmarkPeerstore(b, func() (Peerstore, func()) { + return NewPeerstore(), nil + }) +} \ No newline at end of file diff --git a/p2p/host/peerstore/peerstore_test.go b/p2p/host/peerstore/peerstore_test.go deleted file mode 100644 index ff3bf3b93d..0000000000 --- a/p2p/host/peerstore/peerstore_test.go +++ /dev/null @@ -1,337 +0,0 @@ -package peerstore - -import ( - "context" - "fmt" - "math/rand" - "sort" - "testing" - "time" - - "github.com/libp2p/go-libp2p-crypto" - "github.com/libp2p/go-libp2p-peer" - dspstore "github.com/libp2p/go-libp2p-peerstore/ds" - ma "github.com/multiformats/go-multiaddr" -) - -func getAddrs(t *testing.T, n int) []ma.Multiaddr { - var addrs []ma.Multiaddr - for i := 0; i < n; i++ { - a, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", i)) - if err != nil { - t.Fatal(err) - } - - addrs = append(addrs, a) - } - return addrs -} - -func runTestWithPeerstores(t *testing.T, testFunc func(*testing.T, Peerstore)) { - t.Helper() - t.Log("NewPeerstore") - ps1 := NewPeerstore() - testFunc(t, ps1) - - t.Log("NewPeerstoreDatastore") - ps2, closer2 := setupDatastorePeerstore(t) - defer closer2() - testFunc(t, ps2) -} - -func setupDatastorePeerstore(t testing.TB) (Peerstore, func()) { - ds, closeDB := setupBadgerDatastore(t) - ctx, cancel := context.WithCancel(context.Background()) - ps, err := dspstore.NewPeerstore(ctx, ds) - if err != nil { - t.Fatal(err) - } - closer := func() { - cancel() - closeDB() - } - return ps, closer -} - -func testAddrStream(t *testing.T, ps Peerstore) { - addrs := getAddrs(t, 100) - - pid := peer.ID("testpeer") - - ps.AddAddrs(pid, addrs[:10], time.Hour) - - ctx, cancel := context.WithCancel(context.Background()) - - addrch := ps.AddrStream(ctx, pid) - - // while that subscription is active, publish ten more addrs - // this tests that it doesnt hang - for i := 10; i < 20; i++ { - ps.AddAddr(pid, addrs[i], time.Hour) - } - - // now receive them (without hanging) - timeout := time.After(time.Second * 10) - for i := 0; i < 20; i++ { - select { - case <-addrch: - case <-timeout: - t.Fatal("timed out") - } - } - - // start a second stream - ctx2, cancel2 := context.WithCancel(context.Background()) - addrch2 := ps.AddrStream(ctx2, pid) - - done := make(chan struct{}) - go func() { - defer close(done) - // now send the rest of the addresses - for _, a := range addrs[20:80] { - ps.AddAddr(pid, a, time.Hour) - } - }() - - // receive some concurrently with the goroutine - timeout = time.After(time.Second * 10) - for i := 0; i < 40; i++ { - select { - case <-addrch: - case <-timeout: - } - } - - <-done - - // receive some more after waiting for that goroutine to complete - timeout = time.After(time.Second * 10) - for i := 0; i < 20; i++ { - select { - case <-addrch: - case <-timeout: - } - } - - // now cancel it - cancel() - - // now check the *second* subscription. We should see 80 addresses. - for i := 0; i < 80; i++ { - <-addrch2 - } - - cancel2() - - // and add a few more addresses it doesnt hang afterwards - for _, a := range addrs[80:] { - ps.AddAddr(pid, a, time.Hour) - } -} - -func TestAddrStream(t *testing.T) { - runTestWithPeerstores(t, testAddrStream) -} - -func testGetStreamBeforePeerAdded(t *testing.T, ps Peerstore) { - addrs := getAddrs(t, 10) - pid := peer.ID("testpeer") - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - ach := ps.AddrStream(ctx, pid) - - for i := 0; i < 10; i++ { - ps.AddAddr(pid, addrs[i], time.Hour) - } - - received := make(map[string]bool) - var count int - - for i := 0; i < 10; i++ { - a, ok := <-ach - if !ok { - t.Fatal("channel shouldnt be closed yet") - } - if a == nil { - t.Fatal("got a nil address, thats weird") - } - count++ - if received[a.String()] { - t.Fatal("received duplicate address") - } - received[a.String()] = true - } - - select { - case <-ach: - t.Fatal("shouldnt have received any more addresses") - default: - } - - if count != 10 { - t.Fatal("should have received exactly ten addresses, got ", count) - } - - for _, a := range addrs { - if !received[a.String()] { - t.Log(received) - t.Fatalf("expected to receive address %s but didnt", a) - } - } -} - -func TestGetStreamBeforePeerAdded(t *testing.T) { - runTestWithPeerstores(t, testGetStreamBeforePeerAdded) -} - -func testAddrStreamDuplicates(t *testing.T, ps Peerstore) { - addrs := getAddrs(t, 10) - pid := peer.ID("testpeer") - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - ach := ps.AddrStream(ctx, pid) - - go func() { - for i := 0; i < 10; i++ { - ps.AddAddr(pid, addrs[i], time.Hour) - ps.AddAddr(pid, addrs[rand.Intn(10)], time.Hour) - } - - // make sure that all addresses get processed before context is cancelled - time.Sleep(time.Millisecond * 50) - cancel() - }() - - received := make(map[string]bool) - var count int - for a := range ach { - if a == nil { - t.Fatal("got a nil address, thats weird") - } - count++ - if received[a.String()] { - t.Fatal("received duplicate address") - } - received[a.String()] = true - } - - if count != 10 { - t.Fatal("should have received exactly ten addresses") - } -} - -func TestAddrStreamDuplicates(t *testing.T) { - runTestWithPeerstores(t, testAddrStreamDuplicates) -} - -func testPeerstoreProtoStore(t *testing.T, ps Peerstore) { - p1 := peer.ID("TESTPEER") - - protos := []string{"a", "b", "c", "d"} - - err := ps.AddProtocols(p1, protos...) - if err != nil { - t.Fatal(err) - } - - out, err := ps.GetProtocols(p1) - if err != nil { - t.Fatal(err) - } - - if len(out) != len(protos) { - t.Fatal("got wrong number of protocols back") - } - - sort.Strings(out) - for i, p := range protos { - if out[i] != p { - t.Fatal("got wrong protocol") - } - } - - supported, err := ps.SupportsProtocols(p1, "q", "w", "a", "y", "b") - if err != nil { - t.Fatal(err) - } - - if len(supported) != 2 { - t.Fatal("only expected 2 supported") - } - - if supported[0] != "a" || supported[1] != "b" { - t.Fatal("got wrong supported array: ", supported) - } - - err = ps.SetProtocols(p1, "other") - if err != nil { - t.Fatal(err) - } - - supported, err = ps.SupportsProtocols(p1, "q", "w", "a", "y", "b") - if err != nil { - t.Fatal(err) - } - - if len(supported) != 0 { - t.Fatal("none of those protocols should have been supported") - } -} - -func TestPeerstoreProtoStore(t *testing.T) { - runTestWithPeerstores(t, testAddrStreamDuplicates) -} - -func testBasicPeerstore(t *testing.T, ps Peerstore) { - var pids []peer.ID - addrs := getAddrs(t, 10) - for _, a := range addrs { - priv, _, _ := crypto.GenerateKeyPair(crypto.RSA, 512) - p, _ := peer.IDFromPrivateKey(priv) - pids = append(pids, p) - ps.AddAddr(p, a, PermanentAddrTTL) - } - - peers := ps.Peers() - if len(peers) != 10 { - t.Fatal("expected ten peers, got", len(peers)) - } - - pinfo := ps.PeerInfo(pids[0]) - if !pinfo.Addrs[0].Equal(addrs[0]) { - t.Fatal("stored wrong address") - } -} - -func TestBasicPeerstore(t *testing.T) { - runTestWithPeerstores(t, testBasicPeerstore) -} - -func benchmarkPeerstore(ps Peerstore) func(*testing.B) { - return func(b *testing.B) { - addrs := make(chan *peerpair, 100) - - ctx, cancel := context.WithCancel(context.Background()) - go addressProducer(ctx, b, addrs) - - b.ResetTimer() - for i := 0; i < b.N; i++ { - pp := <-addrs - pid := peer.ID(pp.ID) - ps.AddAddr(pid, pp.Addr, PermanentAddrTTL) - } - cancel() - } -} - -func BenchmarkPeerstore(b *testing.B) { - ps := NewPeerstore() - b.Run("PeerstoreBasic", benchmarkPeerstore(ps)) - - dsps, closer := setupDatastorePeerstore(b) - defer closer() - b.Run("PeerstoreDatastore", benchmarkPeerstore(dsps)) -} diff --git a/p2p/host/peerstore/test/addr_manager_suite.go b/p2p/host/peerstore/test/addr_manager_suite.go new file mode 100644 index 0000000000..96408d787b --- /dev/null +++ b/p2p/host/peerstore/test/addr_manager_suite.go @@ -0,0 +1,279 @@ +package test + +import ( + "testing" + "time" + + "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" +) + +var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ + "Addresses": testAddresses, + "Clear": testClearWorks, + "SetNegativeTTLClears": testSetNegativeTTLClears, + "UpdateTTLs": testUpdateTTLs, + "NilAddrsDontBreak": testNilAddrsDontBreak, + "AddressesExpire": testAddressesExpire, +} + +type AddrMgrFactory func() (pstore.AddrBook, func()) + +func TestAddrMgr(t *testing.T, factory AddrMgrFactory) { + for name, test := range addressBookSuite { + // Create a new peerstore. + ab, closeFunc := factory() + + // Run the test. + t.Run(name, test(ab)) + + // Cleanup. + if closeFunc != nil { + closeFunc() + } + } +} + +func testAddresses(m pstore.AddrBook) func(*testing.T) { + return func(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := IDS(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") + id3 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ6Kn") + id4 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Kn") + id5 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Km") + + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + ma21 := MA(t, "/ip4/2.2.3.2/tcp/1111") + ma22 := MA(t, "/ip4/2.2.3.2/tcp/2222") + ma31 := MA(t, "/ip4/3.2.3.3/tcp/1111") + ma32 := MA(t, "/ip4/3.2.3.3/tcp/2222") + ma33 := MA(t, "/ip4/3.2.3.3/tcp/3333") + ma41 := MA(t, "/ip4/4.2.3.3/tcp/1111") + ma42 := MA(t, "/ip4/4.2.3.3/tcp/2222") + ma43 := MA(t, "/ip4/4.2.3.3/tcp/3333") + ma44 := MA(t, "/ip4/4.2.3.3/tcp/4444") + ma51 := MA(t, "/ip4/5.2.3.3/tcp/1111") + ma52 := MA(t, "/ip4/5.2.3.3/tcp/2222") + ma53 := MA(t, "/ip4/5.2.3.3/tcp/3333") + ma54 := MA(t, "/ip4/5.2.3.3/tcp/4444") + ma55 := MA(t, "/ip4/5.2.3.3/tcp/5555") + + ttl := time.Hour + m.AddAddr(id1, ma11, ttl) + + m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) + m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) // idempotency + + m.AddAddr(id3, ma31, ttl) + m.AddAddr(id3, ma32, ttl) + m.AddAddr(id3, ma33, ttl) + m.AddAddr(id3, ma33, ttl) // idempotency + m.AddAddr(id3, ma33, ttl) + + m.AddAddrs(id4, []ma.Multiaddr{ma41, ma42, ma43, ma44}, ttl) // multiple + + m.AddAddrs(id5, []ma.Multiaddr{ma21, ma22}, ttl) // clearing + m.AddAddrs(id5, []ma.Multiaddr{ma41, ma42, ma43, ma44}, ttl) // clearing + m.ClearAddrs(id5) + m.AddAddrs(id5, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, ttl) // clearing + + // test the Addresses return value + testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + testHas(t, []ma.Multiaddr{ma31, ma32, ma33}, m.Addrs(id3)) + testHas(t, []ma.Multiaddr{ma41, ma42, ma43, ma44}, m.Addrs(id4)) + testHas(t, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, m.Addrs(id5)) + } +} + +func testClearWorks(m pstore.AddrBook) func(t *testing.T) { + return func(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := MA(t, "/ip4/2.2.3.2/tcp/2222") + ma13 := MA(t, "/ip4/3.2.3.3/tcp/3333") + ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") + ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") + + m.AddAddr(id1, ma11, time.Hour) + m.AddAddr(id1, ma12, time.Hour) + m.AddAddr(id1, ma13, time.Hour) + m.AddAddr(id2, ma24, time.Hour) + m.AddAddr(id2, ma25, time.Hour) + + testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.ClearAddrs(id1) + m.ClearAddrs(id2) + + testHas(t, nil, m.Addrs(id1)) + testHas(t, nil, m.Addrs(id2)) + } +} + +func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { + return func(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + + m.SetAddr(id1, ma11, time.Hour) + + testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) + + m.SetAddr(id1, ma11, -1) + + testHas(t, nil, m.Addrs(id1)) + } +} + +func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { + return func(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := MA(t, "/ip4/1.2.3.1/tcp/1112") + ma21 := MA(t, "/ip4/1.2.3.1/tcp/1121") + ma22 := MA(t, "/ip4/1.2.3.1/tcp/1122") + + // Shouldn't panic. + m.UpdateAddrs(id1, time.Hour, time.Minute) + + m.SetAddr(id1, ma11, time.Hour) + m.SetAddr(id1, ma12, time.Minute) + + // Shouldn't panic. + m.UpdateAddrs(id2, time.Hour, time.Minute) + + m.SetAddr(id2, ma21, time.Hour) + m.SetAddr(id2, ma22, time.Minute) + + testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + + m.UpdateAddrs(id1, time.Hour, time.Second) + + testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + + time.Sleep(1200 * time.Millisecond) + + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + + m.UpdateAddrs(id2, time.Hour, time.Second) + + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + + time.Sleep(1200 * time.Millisecond) + + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma22}, m.Addrs(id2)) + } +} + +func testNilAddrsDontBreak(m pstore.AddrBook) func(t *testing.T) { + return func(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + m.SetAddr(id1, nil, time.Hour) + m.AddAddr(id1, nil, time.Hour) + } +} + +func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { + return func(t *testing.T) { + id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := MA(t, "/ip4/2.2.3.2/tcp/2222") + ma13 := MA(t, "/ip4/3.2.3.3/tcp/3333") + ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") + ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") + + m.AddAddr(id1, ma11, time.Hour) + m.AddAddr(id1, ma12, time.Hour) + m.AddAddr(id1, ma13, time.Hour) + m.AddAddr(id2, ma24, time.Hour) + m.AddAddr(id2, ma25, time.Hour) + + testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.SetAddr(id1, ma11, 2*time.Hour) + m.SetAddr(id1, ma12, 2*time.Hour) + m.SetAddr(id1, ma13, 2*time.Hour) + m.SetAddr(id2, ma24, 2*time.Hour) + m.SetAddr(id2, ma25, 2*time.Hour) + + testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.SetAddr(id1, ma11, time.Millisecond) + <-time.After(time.Millisecond * 2) + testHas(t, []ma.Multiaddr{ma12, ma13}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.SetAddr(id1, ma13, time.Millisecond) + <-time.After(time.Millisecond * 2) + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + + m.SetAddr(id2, ma24, time.Millisecond) + <-time.After(time.Millisecond * 2) + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, []ma.Multiaddr{ma25}, m.Addrs(id2)) + + m.SetAddr(id2, ma25, time.Millisecond) + <-time.After(time.Millisecond * 2) + testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) + testHas(t, nil, m.Addrs(id2)) + + m.SetAddr(id1, ma12, time.Millisecond) + <-time.After(time.Millisecond * 2) + testHas(t, nil, m.Addrs(id1)) + testHas(t, nil, m.Addrs(id2)) + } +} + +func IDS(t *testing.T, ids string) peer.ID { + t.Helper() + id, err := peer.IDB58Decode(ids) + if err != nil { + t.Fatalf("id %q is bad: %s", ids, err) + } + return id +} + +func MA(t *testing.T, m string) ma.Multiaddr { + t.Helper() + maddr, err := ma.NewMultiaddr(m) + if err != nil { + t.Fatal(err) + } + return maddr +} + +func testHas(t *testing.T, exp, act []ma.Multiaddr) { + t.Helper() + if len(exp) != len(act) { + t.Fatalf("lengths not the same. expected %d, got %d\n", len(exp), len(act)) + } + + for _, a := range exp { + found := false + + for _, b := range act { + if a.Equal(b) { + found = true + break + } + } + + if !found { + t.Fatalf("expected address %s not found", a) + } + } +} diff --git a/p2p/host/peerstore/benchmark_utils.go b/p2p/host/peerstore/test/benchmark_utils.go similarity index 97% rename from p2p/host/peerstore/benchmark_utils.go rename to p2p/host/peerstore/test/benchmark_utils.go index e838894366..b6d096cd5c 100644 --- a/p2p/host/peerstore/benchmark_utils.go +++ b/p2p/host/peerstore/test/benchmark_utils.go @@ -1,4 +1,4 @@ -package peerstore +package test import ( "context" diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go new file mode 100644 index 0000000000..fed435ae1a --- /dev/null +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -0,0 +1,316 @@ +package test + +import ( + "context" + "fmt" + "math/rand" + "sort" + "testing" + "time" + + "github.com/libp2p/go-libp2p-crypto" + "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" +) + +var peerstoreSuite = map[string]func(book pstore.Peerstore) func(*testing.T){ + "AddrStream": testAddrStream, + "GetStreamBeforePeerAdded": testGetStreamBeforePeerAdded, + "AddStreamDuplicates": testAddrStreamDuplicates, + "PeerstoreProtoStore": testPeerstoreProtoStore, + "BasicPeerstore": testBasicPeerstore, +} + +type PeerstoreFactory func() (pstore.Peerstore, func()) + +func TestPeerstore(t *testing.T, factory PeerstoreFactory) { + for name, test := range peerstoreSuite { + // Create a new peerstore. + ps, closeFunc := factory() + + // Run the test. + t.Run(name, test(ps)) + + // Cleanup. + if closeFunc != nil { + closeFunc() + } + } +} + +func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory) { + ps, closeFunc := factory() + defer closeFunc() + + b.Run("Peerstore", benchmarkPeerstore(ps)) +} + +func testAddrStream(ps pstore.Peerstore) func(t *testing.T) { + return func(t *testing.T) { + addrs, pid := getAddrs(t, 100), peer.ID("testpeer") + ps.AddAddrs(pid, addrs[:10], time.Hour) + + ctx, cancel := context.WithCancel(context.Background()) + addrch := ps.AddrStream(ctx, pid) + + // while that subscription is active, publish ten more addrs + // this tests that it doesnt hang + for i := 10; i < 20; i++ { + ps.AddAddr(pid, addrs[i], time.Hour) + } + + // now receive them (without hanging) + timeout := time.After(time.Second * 10) + for i := 0; i < 20; i++ { + select { + case <-addrch: + case <-timeout: + t.Fatal("timed out") + } + } + + // start a second stream + ctx2, cancel2 := context.WithCancel(context.Background()) + addrch2 := ps.AddrStream(ctx2, pid) + + done := make(chan struct{}) + go func() { + defer close(done) + // now send the rest of the addresses + for _, a := range addrs[20:80] { + ps.AddAddr(pid, a, time.Hour) + } + }() + + // receive some concurrently with the goroutine + timeout = time.After(time.Second * 10) + for i := 0; i < 40; i++ { + select { + case <-addrch: + case <-timeout: + } + } + + <-done + + // receive some more after waiting for that goroutine to complete + timeout = time.After(time.Second * 10) + for i := 0; i < 20; i++ { + select { + case <-addrch: + case <-timeout: + } + } + + // now cancel it + cancel() + + // now check the *second* subscription. We should see 80 addresses. + for i := 0; i < 80; i++ { + <-addrch2 + } + + cancel2() + + // and add a few more addresses it doesnt hang afterwards + for _, a := range addrs[80:] { + ps.AddAddr(pid, a, time.Hour) + } + } +} + +func testGetStreamBeforePeerAdded(ps pstore.Peerstore) func(t *testing.T) { + return func(t *testing.T) { + addrs, pid := getAddrs(t, 10), peer.ID("testpeer") + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + ach := ps.AddrStream(ctx, pid) + for i := 0; i < 10; i++ { + ps.AddAddr(pid, addrs[i], time.Hour) + } + + received := make(map[string]bool) + var count int + + for i := 0; i < 10; i++ { + a, ok := <-ach + if !ok { + t.Fatal("channel shouldnt be closed yet") + } + if a == nil { + t.Fatal("got a nil address, thats weird") + } + count++ + if received[a.String()] { + t.Fatal("received duplicate address") + } + received[a.String()] = true + } + + select { + case <-ach: + t.Fatal("shouldnt have received any more addresses") + default: + } + + if count != 10 { + t.Fatal("should have received exactly ten addresses, got ", count) + } + + for _, a := range addrs { + if !received[a.String()] { + t.Log(received) + t.Fatalf("expected to receive address %s but didnt", a) + } + } + } +} + +func testAddrStreamDuplicates(ps pstore.Peerstore) func(t *testing.T) { + return func(t *testing.T) { + addrs, pid := getAddrs(t, 10), peer.ID("testpeer") + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + ach := ps.AddrStream(ctx, pid) + go func() { + for i := 0; i < 10; i++ { + ps.AddAddr(pid, addrs[i], time.Hour) + ps.AddAddr(pid, addrs[rand.Intn(10)], time.Hour) + } + + // make sure that all addresses get processed before context is cancelled + time.Sleep(time.Millisecond * 50) + cancel() + }() + + received := make(map[string]bool) + var count int + for a := range ach { + if a == nil { + t.Fatal("got a nil address, thats weird") + } + count++ + if received[a.String()] { + t.Fatal("received duplicate address") + } + received[a.String()] = true + } + + if count != 10 { + t.Fatal("should have received exactly ten addresses") + } + } +} + +func testPeerstoreProtoStore(ps pstore.Peerstore) func(t *testing.T) { + return func(t *testing.T) { + p1, protos := peer.ID("TESTPEER"), []string{"a", "b", "c", "d"} + + err := ps.AddProtocols(p1, protos...) + if err != nil { + t.Fatal(err) + } + + out, err := ps.GetProtocols(p1) + if err != nil { + t.Fatal(err) + } + + if len(out) != len(protos) { + t.Fatal("got wrong number of protocols back") + } + + sort.Strings(out) + for i, p := range protos { + if out[i] != p { + t.Fatal("got wrong protocol") + } + } + + supported, err := ps.SupportsProtocols(p1, "q", "w", "a", "y", "b") + if err != nil { + t.Fatal(err) + } + + if len(supported) != 2 { + t.Fatal("only expected 2 supported") + } + + if supported[0] != "a" || supported[1] != "b" { + t.Fatal("got wrong supported array: ", supported) + } + + err = ps.SetProtocols(p1, "other") + if err != nil { + t.Fatal(err) + } + + supported, err = ps.SupportsProtocols(p1, "q", "w", "a", "y", "b") + if err != nil { + t.Fatal(err) + } + + if len(supported) != 0 { + t.Fatal("none of those protocols should have been supported") + } + } +} + +func testBasicPeerstore(ps pstore.Peerstore) func(t *testing.T) { + return func(t *testing.T) { + var pids []peer.ID + addrs := getAddrs(t, 10) + + for _, a := range addrs { + priv, _, _ := crypto.GenerateKeyPair(crypto.RSA, 512) + p, _ := peer.IDFromPrivateKey(priv) + pids = append(pids, p) + ps.AddAddr(p, a, pstore.PermanentAddrTTL) + } + + peers := ps.Peers() + if len(peers) != 10 { + t.Fatal("expected ten peers, got", len(peers)) + } + + pinfo := ps.PeerInfo(pids[0]) + if !pinfo.Addrs[0].Equal(addrs[0]) { + t.Fatal("stored wrong address") + } + } +} + +func benchmarkPeerstore(ps pstore.Peerstore) func(*testing.B) { + return func(b *testing.B) { + addrs := make(chan *peerpair, 100) + + ctx, cancel := context.WithCancel(context.Background()) + go addressProducer(ctx, b, addrs) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + pp := <-addrs + pid := peer.ID(pp.ID) + ps.AddAddr(pid, pp.Addr, pstore.PermanentAddrTTL) + } + cancel() + } +} + +func getAddrs(t *testing.T, n int) []ma.Multiaddr { + var addrs []ma.Multiaddr + for i := 0; i < n; i++ { + a, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/%d", i)) + if err != nil { + t.Fatal(err) + } + + addrs = append(addrs, a) + } + return addrs +} diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/test_utils.go similarity index 97% rename from p2p/host/peerstore/test/utils.go rename to p2p/host/peerstore/test/test_utils.go index fc141b0d6c..3bca8512ea 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/test_utils.go @@ -1,4 +1,4 @@ -package testutil +package test import ( "io" From d66b5d5e29ccdaded4b0fef56a637fdd4dd46bde Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 30 Aug 2018 14:13:27 +0100 Subject: [PATCH 0622/3965] move in-memory peerstore into mem package. --- p2p/host/peerstore/ds/addr_manager.go | 5 +-- p2p/host/peerstore/ds/peerstore.go | 3 +- p2p/host/peerstore/interface.go | 34 ++++++++++++++++++ p2p/host/peerstore/{ => mem}/addr_manager.go | 36 ++------------------ p2p/host/peerstore/{ => mem}/inmem_test.go | 11 +++--- p2p/host/peerstore/{ => mem}/keybook.go | 7 ++-- p2p/host/peerstore/{ => mem}/metadata.go | 7 ++-- p2p/host/peerstore/mem/peerstore.go | 11 ++++++ p2p/host/peerstore/peerstore.go | 20 ----------- 9 files changed, 67 insertions(+), 67 deletions(-) rename p2p/host/peerstore/{ => mem}/addr_manager.go (87%) rename p2p/host/peerstore/{ => mem}/inmem_test.go (56%) rename p2p/host/peerstore/{ => mem}/keybook.go (92%) rename p2p/host/peerstore/{ => mem}/metadata.go (85%) create mode 100644 p2p/host/peerstore/mem/peerstore.go diff --git a/p2p/host/peerstore/ds/addr_manager.go b/p2p/host/peerstore/ds/addr_manager.go index e00023de50..0c76747461 100644 --- a/p2p/host/peerstore/ds/addr_manager.go +++ b/p2p/host/peerstore/ds/addr_manager.go @@ -11,6 +11,7 @@ import ( logging "github.com/ipfs/go-log" "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-peerstore/mem" ma "github.com/multiformats/go-multiaddr" mh "github.com/multiformats/go-multihash" ) @@ -28,7 +29,7 @@ type addrManager struct { cache *lru.ARCCache ds ds.Batching ttlManager *ttlmanager - subsManager *pstore.AddrSubManager + subsManager *mem.AddrSubManager } // NewAddrManager initializes a new address manager given a @@ -44,7 +45,7 @@ func NewAddrManager(ctx context.Context, ds ds.Batching, ttlInterval time.Durati cache: cache, ds: ds, ttlManager: newTTLManager(ctx, ds, cache, ttlInterval), - subsManager: pstore.NewAddrSubManager(), + subsManager: mem.NewAddrSubManager(), } return mgr, nil } diff --git a/p2p/host/peerstore/ds/peerstore.go b/p2p/host/peerstore/ds/peerstore.go index f48263379b..06b4fd351d 100644 --- a/p2p/host/peerstore/ds/peerstore.go +++ b/p2p/host/peerstore/ds/peerstore.go @@ -6,6 +6,7 @@ import ( "github.com/ipfs/go-datastore" pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-peerstore/mem" ) // NewPeerstore creates a peerstore backed by the provided persistent datastore. @@ -15,6 +16,6 @@ func NewPeerstore(ctx context.Context, ds datastore.Batching) (pstore.Peerstore, return nil, err } - ps := pstore.NewPeerstoreWith(pstore.NewKeybook(), addrBook, pstore.NewPeerMetadata()) + ps := pstore.NewPeerstoreWith(mem.NewKeybook(), addrBook, mem.NewPeerMetadata()) return ps, nil } diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index e0c0a6233c..4206ca41b1 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -2,6 +2,7 @@ package peerstore import ( "context" + "math" "time" "gx/ipfs/QmVmDhyTTUcQXFD1rRQ64fGLMSAoaQvNH3hwuaCFAPq2hy/errors" @@ -13,6 +14,39 @@ import ( var ErrNotFound = errors.New("item not found") +var ( + // AddressTTL is the expiration time of addresses. + AddressTTL = time.Hour + + // TempAddrTTL is the ttl used for a short lived address + TempAddrTTL = time.Second * 10 + + // ProviderAddrTTL is the TTL of an address we've received from a provider. + // This is also a temporary address, but lasts longer. After this expires, + // the records we return will require an extra lookup. + ProviderAddrTTL = time.Minute * 10 + + // RecentlyConnectedAddrTTL is used when we recently connected to a peer. + // It means that we are reasonably certain of the peer's address. + RecentlyConnectedAddrTTL = time.Minute * 10 + + // OwnObservedAddrTTL is used for our own external addresses observed by peers. + OwnObservedAddrTTL = time.Minute * 10 +) + +// Permanent TTLs (distinct so we can distinguish between them, constant as they +// are, in fact, permanent) +const ( + + // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes). + PermanentAddrTTL = math.MaxInt64 - iota + + // ConnectedAddrTTL is the ttl used for the addresses of a peer to whom + // we're connected directly. This is basically permanent, as we will + // clear them + re-add under a TempAddrTTL after disconnecting. + ConnectedAddrTTL +) + // Peerstore provides a threadsafe store of Peer related // information. type Peerstore interface { diff --git a/p2p/host/peerstore/addr_manager.go b/p2p/host/peerstore/mem/addr_manager.go similarity index 87% rename from p2p/host/peerstore/addr_manager.go rename to p2p/host/peerstore/mem/addr_manager.go index b0db2b60e9..e6a9d8c280 100644 --- a/p2p/host/peerstore/addr_manager.go +++ b/p2p/host/peerstore/mem/addr_manager.go @@ -1,50 +1,20 @@ -package peerstore +package mem import ( "context" - "math" "sort" "sync" "time" logging "github.com/ipfs/go-log" "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/addr" ma "github.com/multiformats/go-multiaddr" ) var log = logging.Logger("peerstore") -var ( - - // TempAddrTTL is the ttl used for a short lived address - TempAddrTTL = time.Second * 10 - - // ProviderAddrTTL is the TTL of an address we've received from a provider. - // This is also a temporary address, but lasts longer. After this expires, - // the records we return will require an extra lookup. - ProviderAddrTTL = time.Minute * 10 - - // RecentlyConnectedAddrTTL is used when we recently connected to a peer. - // It means that we are reasonably certain of the peer's address. - RecentlyConnectedAddrTTL = time.Minute * 10 - - // OwnObservedAddrTTL is used for our own external addresses observed by peers. - OwnObservedAddrTTL = time.Minute * 10 -) - -// Permanent TTLs (distinct so we can distinguish between them, constant as they -// are, in fact, permanent) -const ( - // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes). - PermanentAddrTTL = math.MaxInt64 - iota - - // ConnectedAddrTTL is the ttl used for the addresses of a peer to whom - // we're connected directly. This is basically permanent, as we will - // clear them + re-add under a TempAddrTTL after disconnecting. - ConnectedAddrTTL -) - type expiringAddr struct { Addr ma.Multiaddr TTL time.Duration @@ -57,7 +27,7 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { type addrSlice []expiringAddr -var _ AddrBook = (*AddrManager)(nil) +var _ pstore.AddrBook = (*AddrManager)(nil) // AddrManager manages addresses. // The zero-value is ready to be used. diff --git a/p2p/host/peerstore/inmem_test.go b/p2p/host/peerstore/mem/inmem_test.go similarity index 56% rename from p2p/host/peerstore/inmem_test.go rename to p2p/host/peerstore/mem/inmem_test.go index 6b1a012c0f..cedc65bd27 100644 --- a/p2p/host/peerstore/inmem_test.go +++ b/p2p/host/peerstore/mem/inmem_test.go @@ -1,25 +1,26 @@ -package peerstore +package mem import ( "testing" + pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/test" ) func TestInMemoryPeerstore(t *testing.T) { - test.TestPeerstore(t, func() (Peerstore, func()) { + test.TestPeerstore(t, func() (pstore.Peerstore, func()) { return NewPeerstore(), nil }) } func TestInMemoryAddrMgr(t *testing.T) { - test.TestAddrMgr(t, func() (AddrBook, func()) { + test.TestAddrMgr(t, func() (pstore.AddrBook, func()) { return &AddrManager{}, nil }) } func BenchmarkInMemoryPeerstore(b *testing.B) { - test.BenchmarkPeerstore(b, func() (Peerstore, func()) { + test.BenchmarkPeerstore(b, func() (pstore.Peerstore, func()) { return NewPeerstore(), nil }) -} \ No newline at end of file +} diff --git a/p2p/host/peerstore/keybook.go b/p2p/host/peerstore/mem/keybook.go similarity index 92% rename from p2p/host/peerstore/keybook.go rename to p2p/host/peerstore/mem/keybook.go index 1a9e6b95b8..e67e8f2af1 100644 --- a/p2p/host/peerstore/keybook.go +++ b/p2p/host/peerstore/mem/keybook.go @@ -1,4 +1,4 @@ -package peerstore +package mem import ( "errors" @@ -6,6 +6,7 @@ import ( ic "github.com/libp2p/go-libp2p-crypto" "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" ) type memoryKeyBook struct { @@ -14,10 +15,10 @@ type memoryKeyBook struct { sks map[peer.ID]ic.PrivKey } -var _ KeyBook = (*memoryKeyBook)(nil) +var _ pstore.KeyBook = (*memoryKeyBook)(nil) // noop new, but in the future we may want to do some init work. -func NewKeybook() KeyBook { +func NewKeybook() pstore.KeyBook { return &memoryKeyBook{ pks: map[peer.ID]ic.PubKey{}, sks: map[peer.ID]ic.PrivKey{}, diff --git a/p2p/host/peerstore/metadata.go b/p2p/host/peerstore/mem/metadata.go similarity index 85% rename from p2p/host/peerstore/metadata.go rename to p2p/host/peerstore/mem/metadata.go index 1f6f5ce3de..26324b82fa 100644 --- a/p2p/host/peerstore/metadata.go +++ b/p2p/host/peerstore/mem/metadata.go @@ -1,9 +1,10 @@ -package peerstore +package mem import ( "sync" "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" ) type memoryPeerMetadata struct { @@ -14,7 +15,7 @@ type memoryPeerMetadata struct { dslock sync.Mutex } -func NewPeerMetadata() PeerMetadata { +func NewPeerMetadata() pstore.PeerMetadata { return &memoryPeerMetadata{ ds: make(map[string]interface{}), } @@ -37,7 +38,7 @@ func (ps *memoryPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { defer ps.dslock.Unlock() i, ok := ps.ds[string(p)+"/"+key] if !ok { - return nil, ErrNotFound + return nil, pstore.ErrNotFound } return i, nil } diff --git a/p2p/host/peerstore/mem/peerstore.go b/p2p/host/peerstore/mem/peerstore.go new file mode 100644 index 0000000000..a706e0785e --- /dev/null +++ b/p2p/host/peerstore/mem/peerstore.go @@ -0,0 +1,11 @@ +package mem + +import pstore "github.com/libp2p/go-libp2p-peerstore" + +// NewPeerstore creates an in-memory threadsafe collection of peers. +func NewPeerstore() pstore.Peerstore { + return pstore.NewPeerstoreWith( + NewKeybook(), + &AddrManager{}, + NewPeerMetadata()) +} diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index b127c4e30e..b4d35a2cc1 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -3,16 +3,10 @@ package peerstore import ( "fmt" "sync" - "time" "github.com/libp2p/go-libp2p-peer" ) -const ( - // AddressTTL is the expiration time of addresses. - AddressTTL = time.Hour -) - var _ Peerstore = (*peerstore)(nil) type peerstore struct { @@ -26,20 +20,6 @@ type peerstore struct { protolock sync.Mutex } -// NewPeerstore creates a threadsafe collection of peers. -// -// TODO: (RK) move in-memory AddrBook, KeyBook and PeerMetadata to mem -// package. Adapt consumers (IPFS) to instantiate mem.NewPeerstore(). -// -// Preserving this function for backwards compatibility is not possible, as -// we would have to import the mem package, which in turn imports this one, thus -// creating a cycle. -// -// DEPRECATED: use an implementation-specific peerstore. -func NewPeerstore() Peerstore { - return NewPeerstoreWith(NewKeybook(), &AddrManager{}, NewPeerMetadata()) -} - // NewPeerstore creates a threadsafe collection of peers. func NewPeerstoreWith(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { return &peerstore{ From e40add1e333d9c3e3a5ba690c6c9e96992d1d6e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 30 Aug 2018 15:24:06 +0100 Subject: [PATCH 0623/3965] gx-go rw. --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr/sorting.go | 6 +++--- p2p/host/peerstore/ds/addr_manager.go | 14 +++++++------- p2p/host/peerstore/ds/ds_test.go | 6 +++--- p2p/host/peerstore/ds/peerstore.go | 2 +- p2p/host/peerstore/interface.go | 6 +++--- p2p/host/peerstore/mem/addr_manager.go | 6 +++--- p2p/host/peerstore/mem/keybook.go | 4 ++-- p2p/host/peerstore/mem/metadata.go | 2 +- p2p/host/peerstore/metrics.go | 2 +- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peerinfo.go | 4 ++-- p2p/host/peerstore/peerinfo_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/queue/distance.go | 4 ++-- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 4 ++-- p2p/host/peerstore/queue/sync.go | 4 ++-- p2p/host/peerstore/test/addr_manager_suite.go | 4 ++-- p2p/host/peerstore/test/benchmark_utils.go | 6 +++--- p2p/host/peerstore/test/peerstore_suite.go | 6 +++--- p2p/host/peerstore/test/test_utils.go | 6 +++--- 23 files changed, 50 insertions(+), 50 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index 9be78d5ccc..fd959b1fae 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index 06bea07bce..a186012f06 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "github.com/multiformats/go-multiaddr" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index f8a89150a2..afeab089b6 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -3,9 +3,9 @@ package addr import ( "bytes" - ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" - mafmt "github.com/whyrusleeping/mafmt" + mafmt "gx/ipfs/QmQscWDtDBDsWAM58aY6gU2KtxyFFmvvZgdfJExYPLgtXA/mafmt" + manet "gx/ipfs/QmV6FjemM1K8oXjrvuq3wuVWWoU2TLDPmNnKrxHzY3v6Ai/go-multiaddr-net" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) func isFDCostlyTransport(a ma.Multiaddr) bool { diff --git a/p2p/host/peerstore/ds/addr_manager.go b/p2p/host/peerstore/ds/addr_manager.go index 0c76747461..2bf8c28c87 100644 --- a/p2p/host/peerstore/ds/addr_manager.go +++ b/p2p/host/peerstore/ds/addr_manager.go @@ -5,15 +5,15 @@ import ( "sync" "time" - "github.com/hashicorp/golang-lru" - ds "github.com/ipfs/go-datastore" - "github.com/ipfs/go-datastore/query" - logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/mem" - ma "github.com/multiformats/go-multiaddr" - mh "github.com/multiformats/go-multihash" + mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" + "gx/ipfs/QmQjMHF8ptRgx4E57UFMiT4YM6kqaJeYxZ1MCDX23aw4rK/golang-lru" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log" + ds "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" + "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore/query" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) var log = logging.Logger("peerstore/ds") diff --git a/p2p/host/peerstore/ds/ds_test.go b/p2p/host/peerstore/ds/ds_test.go index 6f7adcd78e..b2f70c0439 100644 --- a/p2p/host/peerstore/ds/ds_test.go +++ b/p2p/host/peerstore/ds/ds_test.go @@ -7,10 +7,10 @@ import ( "testing" "time" - "github.com/ipfs/go-datastore" - "github.com/ipfs/go-ds-badger" "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/test" + "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" + "gx/ipfs/QmUCfrikzKVGAfpE31RPwPd32fu1DYxSG7HTGCadba5Wza/go-ds-badger" ) func setupBadgerDatastore(t testing.TB) (datastore.Batching, func()) { @@ -65,4 +65,4 @@ func TestBadgerDsAddrBook(t *testing.T) { func BenchmarkBadgerDsPeerstore(b *testing.B) { test.BenchmarkPeerstore(b, newPeerstoreFactory(b)) -} \ No newline at end of file +} diff --git a/p2p/host/peerstore/ds/peerstore.go b/p2p/host/peerstore/ds/peerstore.go index 06b4fd351d..8e2070a163 100644 --- a/p2p/host/peerstore/ds/peerstore.go +++ b/p2p/host/peerstore/ds/peerstore.go @@ -4,9 +4,9 @@ import ( "context" "time" - "github.com/ipfs/go-datastore" pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/mem" + "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" ) // NewPeerstore creates a peerstore backed by the provided persistent datastore. diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 4206ca41b1..b1e3f3f3ba 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -7,9 +7,9 @@ import ( "gx/ipfs/QmVmDhyTTUcQXFD1rRQ64fGLMSAoaQvNH3hwuaCFAPq2hy/errors" - ic "github.com/libp2p/go-libp2p-crypto" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + ic "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) var ErrNotFound = errors.New("item not found") diff --git a/p2p/host/peerstore/mem/addr_manager.go b/p2p/host/peerstore/mem/addr_manager.go index e6a9d8c280..210b0e25df 100644 --- a/p2p/host/peerstore/mem/addr_manager.go +++ b/p2p/host/peerstore/mem/addr_manager.go @@ -6,11 +6,11 @@ import ( "sync" "time" - logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/addr" - ma "github.com/multiformats/go-multiaddr" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) var log = logging.Logger("peerstore") diff --git a/p2p/host/peerstore/mem/keybook.go b/p2p/host/peerstore/mem/keybook.go index e67e8f2af1..6d9b0124fc 100644 --- a/p2p/host/peerstore/mem/keybook.go +++ b/p2p/host/peerstore/mem/keybook.go @@ -4,9 +4,9 @@ import ( "errors" "sync" - ic "github.com/libp2p/go-libp2p-crypto" - "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + ic "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ) type memoryKeyBook struct { diff --git a/p2p/host/peerstore/mem/metadata.go b/p2p/host/peerstore/mem/metadata.go index 26324b82fa..0908fb5b68 100644 --- a/p2p/host/peerstore/mem/metadata.go +++ b/p2p/host/peerstore/mem/metadata.go @@ -3,8 +3,8 @@ package mem import ( "sync" - "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ) type memoryPeerMetadata struct { diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index 05b867692b..e269612e9a 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -4,7 +4,7 @@ import ( "sync" "time" - "github.com/libp2p/go-libp2p-peer" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ) // LatencyEWMASmooting governs the decay of the EWMA (the speed diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 94a741182d..63aef40c3d 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - testutil "github.com/libp2p/go-libp2p-peer/test" + testutil "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer/test" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index f41b9f85ad..f7110440e6 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -5,8 +5,8 @@ import ( "fmt" "strings" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) // PeerInfo is a small struct used to pass around a peer with diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index 57ca5cc87f..ae80f0cc5b 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -3,8 +3,8 @@ package peerstore import ( "testing" - peer "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" + peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index b4d35a2cc1..66979b164c 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -4,7 +4,7 @@ import ( "fmt" "sync" - "github.com/libp2p/go-libp2p-peer" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ) var _ Peerstore = (*peerstore)(nil) diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index f89e8d4210..ca67ea8d8c 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,8 +5,8 @@ import ( "math/big" "sync" - peer "github.com/libp2p/go-libp2p-peer" - ks "github.com/whyrusleeping/go-keyspace" + peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + ks "gx/ipfs/QmUusaX99BZoELh7dmPgirqRQ1FAmMnmnBn3oiqDFGBUSc/go-keyspace" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index eab523a125..44ba1ca0c3 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import peer "github.com/libp2p/go-libp2p-peer" +import peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index e30794b0cf..cc91118be2 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,8 +7,8 @@ import ( "testing" "time" - peer "github.com/libp2p/go-libp2p-peer" - mh "github.com/multiformats/go-multihash" + mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" + peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index a1edc2ccbc..12bc24afe5 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -3,8 +3,8 @@ package queue import ( "context" - logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-peer" + peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log" ) var log = logging.Logger("peerqueue") diff --git a/p2p/host/peerstore/test/addr_manager_suite.go b/p2p/host/peerstore/test/addr_manager_suite.go index 96408d787b..4b29efabae 100644 --- a/p2p/host/peerstore/test/addr_manager_suite.go +++ b/p2p/host/peerstore/test/addr_manager_suite.go @@ -4,9 +4,9 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ diff --git a/p2p/host/peerstore/test/benchmark_utils.go b/p2p/host/peerstore/test/benchmark_utils.go index b6d096cd5c..7008d5081d 100644 --- a/p2p/host/peerstore/test/benchmark_utils.go +++ b/p2p/host/peerstore/test/benchmark_utils.go @@ -6,9 +6,9 @@ import ( "fmt" "testing" - "github.com/mr-tron/base58/base58" - ma "github.com/multiformats/go-multiaddr" - mh "github.com/multiformats/go-multihash" + mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" + "gx/ipfs/QmWFAMPqsEyUX7gDUsRVmMWz59FxSpJ1b2v6bJ1yYzo7jY/go-base58-fast/base58" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) type peerpair struct { diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index fed435ae1a..5d1a81bf0d 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -8,10 +8,10 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-crypto" - "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" + "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) var peerstoreSuite = map[string]func(book pstore.Peerstore) func(*testing.T){ diff --git a/p2p/host/peerstore/test/test_utils.go b/p2p/host/peerstore/test/test_utils.go index 3bca8512ea..26ab537496 100644 --- a/p2p/host/peerstore/test/test_utils.go +++ b/p2p/host/peerstore/test/test_utils.go @@ -5,9 +5,9 @@ import ( "math/rand" "time" - ci "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - mh "github.com/multiformats/go-multihash" + mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" + ci "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" + peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ) func timeSeededRand() io.Reader { From 4fa0970e3127b33615eccfa2b1be5b82f4e93b8b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 30 Aug 2018 16:24:09 +0100 Subject: [PATCH 0624/3965] organise imports. --- p2p/host/peerstore/addr/sorting.go | 4 ++-- p2p/host/peerstore/ds/addr_manager.go | 5 +++-- p2p/host/peerstore/ds/ds_test.go | 5 +++-- p2p/host/peerstore/ds/peerstore.go | 3 ++- p2p/host/peerstore/mem/addr_manager.go | 5 +++-- p2p/host/peerstore/mem/keybook.go | 3 ++- p2p/host/peerstore/mem/metadata.go | 3 ++- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peerinfo_test.go | 2 +- p2p/host/peerstore/queue/distance.go | 2 +- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 2 +- p2p/host/peerstore/queue/sync.go | 2 +- p2p/host/peerstore/test/addr_manager_suite.go | 3 ++- p2p/host/peerstore/test/peerstore_suite.go | 3 ++- p2p/host/peerstore/test/test_utils.go | 2 +- 16 files changed, 28 insertions(+), 20 deletions(-) diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index afeab089b6..f66dc6cb18 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -3,8 +3,8 @@ package addr import ( "bytes" - mafmt "gx/ipfs/QmQscWDtDBDsWAM58aY6gU2KtxyFFmvvZgdfJExYPLgtXA/mafmt" - manet "gx/ipfs/QmV6FjemM1K8oXjrvuq3wuVWWoU2TLDPmNnKrxHzY3v6Ai/go-multiaddr-net" + "gx/ipfs/QmQscWDtDBDsWAM58aY6gU2KtxyFFmvvZgdfJExYPLgtXA/mafmt" + "gx/ipfs/QmV6FjemM1K8oXjrvuq3wuVWWoU2TLDPmNnKrxHzY3v6Ai/go-multiaddr-net" ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) diff --git a/p2p/host/peerstore/ds/addr_manager.go b/p2p/host/peerstore/ds/addr_manager.go index 2bf8c28c87..2b0fb55e36 100644 --- a/p2p/host/peerstore/ds/addr_manager.go +++ b/p2p/host/peerstore/ds/addr_manager.go @@ -5,8 +5,6 @@ import ( "sync" "time" - pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/mem" mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" "gx/ipfs/QmQjMHF8ptRgx4E57UFMiT4YM6kqaJeYxZ1MCDX23aw4rK/golang-lru" "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" @@ -14,6 +12,9 @@ import ( ds "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore/query" ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + + pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-peerstore/mem" ) var log = logging.Logger("peerstore/ds") diff --git a/p2p/host/peerstore/ds/ds_test.go b/p2p/host/peerstore/ds/ds_test.go index b2f70c0439..7e1d60821a 100644 --- a/p2p/host/peerstore/ds/ds_test.go +++ b/p2p/host/peerstore/ds/ds_test.go @@ -7,10 +7,11 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/test" "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" "gx/ipfs/QmUCfrikzKVGAfpE31RPwPd32fu1DYxSG7HTGCadba5Wza/go-ds-badger" + + "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-peerstore/test" ) func setupBadgerDatastore(t testing.TB) (datastore.Batching, func()) { diff --git a/p2p/host/peerstore/ds/peerstore.go b/p2p/host/peerstore/ds/peerstore.go index 8e2070a163..cb96fec9f4 100644 --- a/p2p/host/peerstore/ds/peerstore.go +++ b/p2p/host/peerstore/ds/peerstore.go @@ -4,9 +4,10 @@ import ( "context" "time" + "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" + pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/mem" - "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" ) // NewPeerstore creates a peerstore backed by the provided persistent datastore. diff --git a/p2p/host/peerstore/mem/addr_manager.go b/p2p/host/peerstore/mem/addr_manager.go index 210b0e25df..8a7c3c6914 100644 --- a/p2p/host/peerstore/mem/addr_manager.go +++ b/p2p/host/peerstore/mem/addr_manager.go @@ -6,11 +6,12 @@ import ( "sync" "time" - pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/addr" "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log" ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + + pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-peerstore/addr" ) var log = logging.Logger("peerstore") diff --git a/p2p/host/peerstore/mem/keybook.go b/p2p/host/peerstore/mem/keybook.go index 6d9b0124fc..ddfe1e0c0e 100644 --- a/p2p/host/peerstore/mem/keybook.go +++ b/p2p/host/peerstore/mem/keybook.go @@ -4,9 +4,10 @@ import ( "errors" "sync" - pstore "github.com/libp2p/go-libp2p-peerstore" ic "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + + pstore "github.com/libp2p/go-libp2p-peerstore" ) type memoryKeyBook struct { diff --git a/p2p/host/peerstore/mem/metadata.go b/p2p/host/peerstore/mem/metadata.go index 0908fb5b68..4a56035408 100644 --- a/p2p/host/peerstore/mem/metadata.go +++ b/p2p/host/peerstore/mem/metadata.go @@ -3,8 +3,9 @@ package mem import ( "sync" - pstore "github.com/libp2p/go-libp2p-peerstore" "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + + pstore "github.com/libp2p/go-libp2p-peerstore" ) type memoryPeerMetadata struct { diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 63aef40c3d..a69e2fb969 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - testutil "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer/test" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer/test" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index ae80f0cc5b..eaff1978fc 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -3,7 +3,7 @@ package peerstore import ( "testing" - peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" ) diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index ca67ea8d8c..b33363dc97 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,7 +5,7 @@ import ( "math/big" "sync" - peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ks "gx/ipfs/QmUusaX99BZoELh7dmPgirqRQ1FAmMnmnBn3oiqDFGBUSc/go-keyspace" ) diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index 44ba1ca0c3..321f5f30e1 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" +import "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index cc91118be2..15a6e00b9c 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -8,7 +8,7 @@ import ( "time" mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" - peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 12bc24afe5..830fdebadf 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -3,7 +3,7 @@ package queue import ( "context" - peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log" ) diff --git a/p2p/host/peerstore/test/addr_manager_suite.go b/p2p/host/peerstore/test/addr_manager_suite.go index 4b29efabae..a66918c224 100644 --- a/p2p/host/peerstore/test/addr_manager_suite.go +++ b/p2p/host/peerstore/test/addr_manager_suite.go @@ -4,9 +4,10 @@ import ( "testing" "time" - pstore "github.com/libp2p/go-libp2p-peerstore" "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + + pstore "github.com/libp2p/go-libp2p-peerstore" ) var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index 5d1a81bf0d..fdecd20ef0 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -8,10 +8,11 @@ import ( "testing" "time" - pstore "github.com/libp2p/go-libp2p-peerstore" "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + + pstore "github.com/libp2p/go-libp2p-peerstore" ) var peerstoreSuite = map[string]func(book pstore.Peerstore) func(*testing.T){ diff --git a/p2p/host/peerstore/test/test_utils.go b/p2p/host/peerstore/test/test_utils.go index 26ab537496..4d0b90a697 100644 --- a/p2p/host/peerstore/test/test_utils.go +++ b/p2p/host/peerstore/test/test_utils.go @@ -7,7 +7,7 @@ import ( mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" ci "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" - peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" ) func timeSeededRand() io.Reader { From ffdc59c20587506435c1a00808730cbed2039c90 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 30 Aug 2018 16:43:40 +0100 Subject: [PATCH 0625/3965] normalize naming: AddrManager => AddrBook; ctor for memory-backed AddrBook. --- .../ds/{addr_manager.go => addr_book.go} | 34 ++-- p2p/host/peerstore/ds/ds_test.go | 2 +- p2p/host/peerstore/ds/peerstore.go | 4 +- p2p/host/peerstore/interface.go | 19 ++- .../mem/{addr_manager.go => addr_book.go} | 150 ++++++++---------- p2p/host/peerstore/mem/inmem_test.go | 2 +- p2p/host/peerstore/mem/keybook.go | 2 +- p2p/host/peerstore/mem/peerstore.go | 4 +- p2p/host/peerstore/peerstore.go | 3 - 9 files changed, 106 insertions(+), 114 deletions(-) rename p2p/host/peerstore/ds/{addr_manager.go => addr_book.go} (87%) rename p2p/host/peerstore/mem/{addr_manager.go => addr_book.go} (72%) diff --git a/p2p/host/peerstore/ds/addr_manager.go b/p2p/host/peerstore/ds/addr_book.go similarity index 87% rename from p2p/host/peerstore/ds/addr_manager.go rename to p2p/host/peerstore/ds/addr_book.go index 2b0fb55e36..2b57c0781f 100644 --- a/p2p/host/peerstore/ds/addr_manager.go +++ b/p2p/host/peerstore/ds/addr_book.go @@ -22,27 +22,27 @@ var log = logging.Logger("peerstore/ds") // Number of times to retry transactional writes var dsWriteRetries = 5 -var _ pstore.AddrBook = (*addrManager)(nil) +var _ pstore.AddrBook = (*dsAddrBook)(nil) -// addrManager is an address manager backed by a Datastore with both an +// dsAddrBook is an address manager backed by a Datastore with both an // in-memory TTL manager and an in-memory address stream manager. -type addrManager struct { +type dsAddrBook struct { cache *lru.ARCCache ds ds.Batching ttlManager *ttlmanager subsManager *mem.AddrSubManager } -// NewAddrManager initializes a new address manager given a +// NewAddrBook initializes a new address manager given a // Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. -func NewAddrManager(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) (*addrManager, error) { +func NewAddrBook(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) (*dsAddrBook, error) { cache, err := lru.NewARC(1024) if err != nil { return nil, err } - mgr := &addrManager{ + mgr := &dsAddrBook{ cache: cache, ds: ds, ttlManager: newTTLManager(ctx, ds, cache, ttlInterval), @@ -52,7 +52,7 @@ func NewAddrManager(ctx context.Context, ds ds.Batching, ttlInterval time.Durati } // Stop will signal the TTL manager to stop and block until it returns. -func (mgr *addrManager) Stop() { +func (mgr *dsAddrBook) Stop() { mgr.ttlManager.cancel() } @@ -70,12 +70,12 @@ func peerIDFromKey(key ds.Key) (peer.ID, error) { } // AddAddr will add a new address if it's not already in the AddrBook. -func (mgr *addrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { +func (mgr *dsAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } // AddAddrs will add many new addresses if they're not already in the AddrBook. -func (mgr *addrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { +func (mgr *dsAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { if ttl <= 0 { return } @@ -84,16 +84,16 @@ func (mgr *addrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat } // SetAddr will add or update the TTL of an address in the AddrBook. -func (mgr *addrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { +func (mgr *dsAddrBook) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } // SetAddrs will add or update the TTLs of addresses in the AddrBook. -func (mgr *addrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { +func (mgr *dsAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { mgr.setAddrs(p, addrs, ttl, false) } -func (mgr *addrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, add bool) { +func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, add bool) { for i := 0; i < dsWriteRetries; i++ { // keys to add to the TTL manager var keys []ds.Key @@ -153,13 +153,13 @@ func (mgr *addrManager) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat // UpdateAddrs will update any addresses for a given peer and TTL combination to // have a new TTL. -func (mgr *addrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { +func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { prefix := ds.NewKey(p.Pretty()) mgr.ttlManager.updateTTLs(prefix, oldTTL, newTTL) } // Addrs Returns all of the non-expired addresses for a given peer. -func (mgr *addrManager) Addrs(p peer.ID) []ma.Multiaddr { +func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { prefix := ds.NewKey(p.Pretty()) q := query.Query{Prefix: prefix.String(), KeysOnly: true} results, err := mgr.ds.Query(q) @@ -193,7 +193,7 @@ func (mgr *addrManager) Addrs(p peer.ID) []ma.Multiaddr { } // Peers returns all of the peer IDs for which the AddrBook has addresses. -func (mgr *addrManager) AddrsPeers() []peer.ID { +func (mgr *dsAddrBook) AddrsPeers() []peer.ID { q := query.Query{KeysOnly: true} results, err := mgr.ds.Query(q) if err != nil { @@ -220,13 +220,13 @@ func (mgr *addrManager) AddrsPeers() []peer.ID { // AddrStream returns a channel on which all new addresses discovered for a // given peer ID will be published. -func (mgr *addrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { +func (mgr *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { initial := mgr.Addrs(p) return mgr.subsManager.AddrStream(ctx, p, initial) } // ClearAddrs will delete all known addresses for a peer ID. -func (mgr *addrManager) ClearAddrs(p peer.ID) { +func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { prefix := ds.NewKey(p.Pretty()) for i := 0; i < dsWriteRetries; i++ { q := query.Query{Prefix: prefix.String(), KeysOnly: true} diff --git a/p2p/host/peerstore/ds/ds_test.go b/p2p/host/peerstore/ds/ds_test.go index 7e1d60821a..c3a19d50a1 100644 --- a/p2p/host/peerstore/ds/ds_test.go +++ b/p2p/host/peerstore/ds/ds_test.go @@ -51,7 +51,7 @@ func TestBadgerDsAddrBook(t *testing.T) { test.TestAddrMgr(t, func() (peerstore.AddrBook, func()) { ds, closeDB := setupBadgerDatastore(t) - mgr, err := NewAddrManager(context.Background(), ds, 100*time.Microsecond) + mgr, err := NewAddrBook(context.Background(), ds, 100*time.Microsecond) if err != nil { t.Fatal(err) } diff --git a/p2p/host/peerstore/ds/peerstore.go b/p2p/host/peerstore/ds/peerstore.go index cb96fec9f4..2c63f1bcd6 100644 --- a/p2p/host/peerstore/ds/peerstore.go +++ b/p2p/host/peerstore/ds/peerstore.go @@ -12,11 +12,11 @@ import ( // NewPeerstore creates a peerstore backed by the provided persistent datastore. func NewPeerstore(ctx context.Context, ds datastore.Batching) (pstore.Peerstore, error) { - addrBook, err := NewAddrManager(ctx, ds, time.Second) + addrBook, err := NewAddrBook(ctx, ds, time.Second) if err != nil { return nil, err } - ps := pstore.NewPeerstoreWith(mem.NewKeybook(), addrBook, mem.NewPeerMetadata()) + ps := pstore.NewPeerstoreWith(mem.NewKeyBook(), addrBook, mem.NewPeerMetadata()) return ps, nil } diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index b1e3f3f3ba..9483b5132a 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -77,14 +77,13 @@ type PeerMetadata interface { Put(p peer.ID, key string, val interface{}) error } -// AddrBook is an interface that fits the new AddrManager. I'm patching -// it up in here to avoid changing a ton of the codebase. +// AddrBook holds the multiaddrs of peers. type AddrBook interface { // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) - // AddAddrs gives AddrManager addresses to use, with a given ttl + // AddAddrs gives this AddrBook addresses to use, with a given ttl // (time-to-live), after which the address is no longer valid. // If the manager has a longer TTL, the operation is a no-op for that address AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) @@ -111,17 +110,25 @@ type AddrBook interface { // ClearAddresses removes all previously stored addresses ClearAddrs(p peer.ID) - // Peers returns all of the peer IDs stored in the AddrBook + // AddrsPeers returns all of the peer IDs stored in the AddrBook AddrsPeers() []peer.ID } -// KeyBook tracks the Public keys of Peers. +// KeyBook tracks the keys of Peers. type KeyBook interface { - KeyBookPeers() []peer.ID + // PubKey stores the public key of a peer. PubKey(peer.ID) ic.PubKey + + // AddPubKey stores the public key of a peer. AddPubKey(peer.ID, ic.PubKey) error + // PrivKey returns the private key of a peer. PrivKey(peer.ID) ic.PrivKey + + // AddPrivKey stores the private key of a peer. AddPrivKey(peer.ID, ic.PrivKey) error + + // KeyBookPeers returns all the peer IDs stored in the KeyBook + KeyBookPeers() []peer.ID } diff --git a/p2p/host/peerstore/mem/addr_manager.go b/p2p/host/peerstore/mem/addr_book.go similarity index 72% rename from p2p/host/peerstore/mem/addr_manager.go rename to p2p/host/peerstore/mem/addr_book.go index 8a7c3c6914..ba343468c7 100644 --- a/p2p/host/peerstore/mem/addr_manager.go +++ b/p2p/host/peerstore/mem/addr_book.go @@ -28,63 +28,55 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { type addrSlice []expiringAddr -var _ pstore.AddrBook = (*AddrManager)(nil) +var _ pstore.AddrBook = (*memoryAddrBook)(nil) -// AddrManager manages addresses. -// The zero-value is ready to be used. -type AddrManager struct { - addrmu sync.Mutex // guards addrs +// memoryAddrBook manages addresses. +type memoryAddrBook struct { + addrmu sync.Mutex addrs map[peer.ID]addrSlice subManager *AddrSubManager } -// ensures the AddrManager is initialized. -// So we can use the zero value. -func (mgr *AddrManager) init() { - if mgr.addrs == nil { - mgr.addrs = make(map[peer.ID]addrSlice) - } - if mgr.subManager == nil { - mgr.subManager = NewAddrSubManager() +func NewAddrBook() pstore.AddrBook { + return &memoryAddrBook{ + addrs: make(map[peer.ID]addrSlice), + subManager: NewAddrSubManager(), } } -func (mgr *AddrManager) AddrsPeers() []peer.ID { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() - if mgr.addrs == nil { +func (mab *memoryAddrBook) AddrsPeers() []peer.ID { + mab.addrmu.Lock() + defer mab.addrmu.Unlock() + if mab.addrs == nil { return nil } - pids := make([]peer.ID, 0, len(mgr.addrs)) - for pid := range mgr.addrs { + pids := make([]peer.ID, 0, len(mab.addrs)) + for pid := range mab.addrs { pids = append(pids, pid) } return pids } // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) -func (mgr *AddrManager) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) +func (mab *memoryAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mab.AddAddrs(p, []ma.Multiaddr{addr}, ttl) } -// AddAddrs gives AddrManager addresses to use, with a given ttl +// AddAddrs gives memoryAddrBook addresses to use, with a given ttl // (time-to-live), after which the address is no longer valid. // If the manager has a longer TTL, the operation is a no-op for that address -func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() +func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + mab.addrmu.Lock() + defer mab.addrmu.Unlock() // if ttl is zero, exit. nothing to do. if ttl <= 0 { return } - // so zero value can be used - mgr.init() - - oldAddrs := mgr.addrs[p] + oldAddrs := mab.addrs[p] amap := make(map[string]expiringAddr, len(oldAddrs)) for _, ea := range oldAddrs { amap[string(ea.Addr.Bytes())] = ea @@ -103,31 +95,28 @@ func (mgr *AddrManager) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat if !found || exp.After(a.Expires) { amap[addrstr] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} - mgr.subManager.BroadcastAddr(p, addr) + mab.subManager.BroadcastAddr(p, addr) } } newAddrs := make([]expiringAddr, 0, len(amap)) for _, ea := range amap { newAddrs = append(newAddrs, ea) } - mgr.addrs[p] = newAddrs + mab.addrs[p] = newAddrs } // SetAddr calls mgr.SetAddrs(p, addr, ttl) -func (mgr *AddrManager) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) +func (mab *memoryAddrBook) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + mab.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } // SetAddrs sets the ttl on addresses. This clears any TTL there previously. // This is used when we receive the best estimate of the validity of an address. -func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() - - // so zero value can be used - mgr.init() +func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + mab.addrmu.Lock() + defer mab.addrmu.Unlock() - oldAddrs := mgr.addrs[p] + oldAddrs := mab.addrs[p] amap := make(map[string]expiringAddr, len(oldAddrs)) for _, ea := range oldAddrs { amap[string(ea.Addr.Bytes())] = ea @@ -145,7 +134,7 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat if ttl > 0 { amap[addrs] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} - mgr.subManager.BroadcastAddr(p, addr) + mab.subManager.BroadcastAddr(p, addr) } else { delete(amap, addrs) } @@ -154,20 +143,20 @@ func (mgr *AddrManager) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durat for _, ea := range amap { newAddrs = append(newAddrs, ea) } - mgr.addrs[p] = newAddrs + mab.addrs[p] = newAddrs } // UpdateAddrs updates the addresses associated with the given peer that have // the given oldTTL to have the given newTTL. -func (mgr *AddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() +func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { + mab.addrmu.Lock() + defer mab.addrmu.Unlock() - if mgr.addrs == nil { + if mab.addrs == nil { return } - addrs, found := mgr.addrs[p] + addrs, found := mab.addrs[p] if !found { return } @@ -184,16 +173,16 @@ func (mgr *AddrManager) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time } // Addresses returns all known (and valid) addresses for a given -func (mgr *AddrManager) Addrs(p peer.ID) []ma.Multiaddr { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() +func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { + mab.addrmu.Lock() + defer mab.addrmu.Unlock() // not initialized? nothing to give. - if mgr.addrs == nil { + if mab.addrs == nil { return nil } - maddrs, found := mgr.addrs[p] + maddrs, found := mab.addrs[p] if !found { return nil } @@ -210,40 +199,52 @@ func (mgr *AddrManager) Addrs(p peer.ID) []ma.Multiaddr { // clean up the expired ones. if len(cleaned) == 0 { - delete(mgr.addrs, p) + delete(mab.addrs, p) } else { - mgr.addrs[p] = cleaned + mab.addrs[p] = cleaned } return good } // ClearAddrs removes all previously stored addresses -func (mgr *AddrManager) ClearAddrs(p peer.ID) { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() - mgr.init() +func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { + mab.addrmu.Lock() + defer mab.addrmu.Unlock() - delete(mgr.addrs, p) + delete(mab.addrs, p) } // AddrStream returns a channel on which all new addresses discovered for a // given peer ID will be published. -func (mgr *AddrManager) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { - mgr.addrmu.Lock() - defer mgr.addrmu.Unlock() - mgr.init() +func (mab *memoryAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { + mab.addrmu.Lock() + defer mab.addrmu.Unlock() - baseaddrslice := mgr.addrs[p] + baseaddrslice := mab.addrs[p] initial := make([]ma.Multiaddr, 0, len(baseaddrslice)) for _, a := range baseaddrslice { initial = append(initial, a.Addr) } - return mgr.subManager.AddrStream(ctx, p, initial) + return mab.subManager.AddrStream(ctx, p, initial) +} + +type addrSub struct { + pubch chan ma.Multiaddr + lk sync.Mutex + buffer []ma.Multiaddr + ctx context.Context +} + +func (s *addrSub) pubAddr(a ma.Multiaddr) { + select { + case s.pubch <- a: + case <-s.ctx.Done(): + } } // An abstracted, pub-sub manager for address streams. Extracted from -// AddrManager in order to support additional implementations. +// memoryAddrBook in order to support additional implementations. type AddrSubManager struct { mu sync.RWMutex subs map[peer.ID][]*addrSub @@ -261,6 +262,7 @@ func NewAddrSubManager() *AddrSubManager { func (mgr *AddrSubManager) removeSub(p peer.ID, s *addrSub) { mgr.mu.Lock() defer mgr.mu.Unlock() + subs := mgr.subs[p] if len(subs) == 1 { if subs[0] != s { @@ -269,6 +271,7 @@ func (mgr *AddrSubManager) removeSub(p peer.ID, s *addrSub) { delete(mgr.subs, p) return } + for i, v := range subs { if v == s { subs[i] = subs[len(subs)-1] @@ -295,7 +298,6 @@ func (mgr *AddrSubManager) BroadcastAddr(p peer.ID, addr ma.Multiaddr) { // channel with any addresses we might already have on file. func (mgr *AddrSubManager) AddrStream(ctx context.Context, p peer.ID, initial []ma.Multiaddr) <-chan ma.Multiaddr { sub := &addrSub{pubch: make(chan ma.Multiaddr), ctx: ctx} - out := make(chan ma.Multiaddr) mgr.mu.Lock() @@ -357,17 +359,3 @@ func (mgr *AddrSubManager) AddrStream(ctx context.Context, p peer.ID, initial [] return out } - -type addrSub struct { - pubch chan ma.Multiaddr - lk sync.Mutex - buffer []ma.Multiaddr - ctx context.Context -} - -func (s *addrSub) pubAddr(a ma.Multiaddr) { - select { - case s.pubch <- a: - case <-s.ctx.Done(): - } -} diff --git a/p2p/host/peerstore/mem/inmem_test.go b/p2p/host/peerstore/mem/inmem_test.go index cedc65bd27..6f9b4a8d58 100644 --- a/p2p/host/peerstore/mem/inmem_test.go +++ b/p2p/host/peerstore/mem/inmem_test.go @@ -15,7 +15,7 @@ func TestInMemoryPeerstore(t *testing.T) { func TestInMemoryAddrMgr(t *testing.T) { test.TestAddrMgr(t, func() (pstore.AddrBook, func()) { - return &AddrManager{}, nil + return NewAddrBook(), nil }) } diff --git a/p2p/host/peerstore/mem/keybook.go b/p2p/host/peerstore/mem/keybook.go index ddfe1e0c0e..37e184e2af 100644 --- a/p2p/host/peerstore/mem/keybook.go +++ b/p2p/host/peerstore/mem/keybook.go @@ -19,7 +19,7 @@ type memoryKeyBook struct { var _ pstore.KeyBook = (*memoryKeyBook)(nil) // noop new, but in the future we may want to do some init work. -func NewKeybook() pstore.KeyBook { +func NewKeyBook() pstore.KeyBook { return &memoryKeyBook{ pks: map[peer.ID]ic.PubKey{}, sks: map[peer.ID]ic.PrivKey{}, diff --git a/p2p/host/peerstore/mem/peerstore.go b/p2p/host/peerstore/mem/peerstore.go index a706e0785e..ba4c5a3b79 100644 --- a/p2p/host/peerstore/mem/peerstore.go +++ b/p2p/host/peerstore/mem/peerstore.go @@ -5,7 +5,7 @@ import pstore "github.com/libp2p/go-libp2p-peerstore" // NewPeerstore creates an in-memory threadsafe collection of peers. func NewPeerstore() pstore.Peerstore { return pstore.NewPeerstoreWith( - NewKeybook(), - &AddrManager{}, + NewKeyBook(), + NewAddrBook(), NewPeerMetadata()) } diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 66979b164c..fc9bf2ef34 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -25,9 +25,6 @@ func NewPeerstoreWith(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { return &peerstore{ KeyBook: kb, PeerMetadata: md, - - // TODO: normalise the naming here AddrBook vs AddrManager. - // TODO: add a constructor NewAddrBook and make the struct private. AddrBook: ab, Metrics: NewMetrics(), } From 0d07edd8c784bb3ffe52e8d9d7fd41b5bef7a1a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 30 Aug 2018 18:05:22 +0100 Subject: [PATCH 0626/3965] go fmt. --- p2p/host/peerstore/peerstore.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index fc9bf2ef34..d15d129edb 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -25,8 +25,8 @@ func NewPeerstoreWith(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { return &peerstore{ KeyBook: kb, PeerMetadata: md, - AddrBook: ab, - Metrics: NewMetrics(), + AddrBook: ab, + Metrics: NewMetrics(), } } From 0d38750db8cd53ab468e0c758c18e621c1c1dd3f Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 30 Aug 2018 19:37:21 +0700 Subject: [PATCH 0627/3965] use the new DisableLogging mdns option, don't turn off logging globally --- p2p/discovery/mdns.go | 6 ++---- package.json | 4 ++-- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/p2p/discovery/mdns.go b/p2p/discovery/mdns.go index a7ec72ab12..691b51a29b 100644 --- a/p2p/discovery/mdns.go +++ b/p2p/discovery/mdns.go @@ -4,8 +4,6 @@ import ( "context" "errors" "io" - "io/ioutil" - golog "log" "net" "sync" "time" @@ -64,8 +62,8 @@ func getDialableListenAddrs(ph host.Host) ([]*net.TCPAddr, error) { func NewMdnsService(ctx context.Context, peerhost host.Host, interval time.Duration, serviceTag string) (Service, error) { - // TODO: dont let mdns use logging... - golog.SetOutput(ioutil.Discard) + // don't let mdns use logging... + mdns.DisableLogging = true var ipaddrs []net.IP port := 4001 diff --git a/package.json b/package.json index 9ef415a006..f048ac7d2d 100644 --- a/package.json +++ b/package.json @@ -14,9 +14,9 @@ "version": "0.0.0" }, { - "hash": "QmNeRzgrSpwbMU7VKLRyfvbqf1nRrAiQ7fEXaBxWGT5Ygr", + "hash": "QmekaTKpWkYGcn4ZEC5PwJDRCQHapwugmmG86g2Xpz5GBH", "name": "mdns", - "version": "0.1.3" + "version": "0.2.0" }, { "hash": "QmPdKqUcHGFdeSpvjVoaTRPPstGif9GBZb5Q56RVw9o69A", From 9692260f180abb455362f6a2c7102aad507ba7fe Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 31 Aug 2018 13:35:18 +0700 Subject: [PATCH 0628/3965] run Go 1.11 gofmt, use Go 1.10.x and 1.11.x on Travis --- p2p/transport/quic/crypto.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/transport/quic/crypto.go b/p2p/transport/quic/crypto.go index a9aabeefd2..1ce18bcb9d 100644 --- a/p2p/transport/quic/crypto.go +++ b/p2p/transport/quic/crypto.go @@ -80,10 +80,10 @@ func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { return nil, nil, err } tmpl := &x509.Certificate{ - SerialNumber: sn, - NotBefore: time.Now().Add(-24 * time.Hour), - NotAfter: time.Now().Add(certValidityPeriod), - IsCA: true, + SerialNumber: sn, + NotBefore: time.Now().Add(-24 * time.Hour), + NotAfter: time.Now().Add(certValidityPeriod), + IsCA: true, BasicConstraintsValid: true, } From 15272ee4215ce1cdbf20202b6fbd3bc7b6d6c857 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 31 Aug 2018 13:23:02 +0100 Subject: [PATCH 0629/3965] go fmt. --- p2p/host/peerstore/mem/addr_book.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/mem/addr_book.go b/p2p/host/peerstore/mem/addr_book.go index ba343468c7..e0c3f83109 100644 --- a/p2p/host/peerstore/mem/addr_book.go +++ b/p2p/host/peerstore/mem/addr_book.go @@ -40,7 +40,7 @@ type memoryAddrBook struct { func NewAddrBook() pstore.AddrBook { return &memoryAddrBook{ - addrs: make(map[peer.ID]addrSlice), + addrs: make(map[peer.ID]addrSlice), subManager: NewAddrSubManager(), } } @@ -262,7 +262,7 @@ func NewAddrSubManager() *AddrSubManager { func (mgr *AddrSubManager) removeSub(p peer.ID, s *addrSub) { mgr.mu.Lock() defer mgr.mu.Unlock() - + subs := mgr.subs[p] if len(subs) == 1 { if subs[0] != s { From f2db33fee7efc9ee4f970d8b802258ae0167dd1d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 31 Aug 2018 13:30:13 +0100 Subject: [PATCH 0630/3965] fix nil dereference in benchmarks. --- p2p/host/peerstore/test/peerstore_suite.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index fdecd20ef0..045d32d0bc 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -42,9 +42,12 @@ func TestPeerstore(t *testing.T, factory PeerstoreFactory) { func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory) { ps, closeFunc := factory() - defer closeFunc() b.Run("Peerstore", benchmarkPeerstore(ps)) + + if closeFunc != nil { + closeFunc() + } } func testAddrStream(ps pstore.Peerstore) func(t *testing.T) { From 7dffae8a14e99b5042872ba790fd89c57593ef50 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 31 Aug 2018 09:54:24 -0700 Subject: [PATCH 0631/3965] gx publish 6.0.11 --- .gx/lastpubver | 2 +- package.json | 30 +++++++++++++++--------------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index cbf4243541..070e64e803 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.10: QmVM6VuGaWcAaYjxG2om6XxMmpP3Rt9rw4nbMXVNYAPLhS +6.0.11: Qmf1u2efhjXYtuyP8SMHYtw4dCkbghnniex2PSp7baA7FP diff --git a/package.json b/package.json index 9ef415a006..80b2356596 100644 --- a/package.json +++ b/package.json @@ -24,9 +24,9 @@ "version": "1.2.8" }, { - "hash": "QmbXRda5H2K3MSQyWWxTMtd8DWuguEBUCe6hpxfXVpFUGj", + "hash": "QmabLh8TrJ3emfAoQk5AbqbLTbMyj7XqumMFmAFxa9epo8", "name": "go-multistream", - "version": "0.3.7" + "version": "0.3.9" }, { "hash": "Qmf7HqcW7LtCi1W8y2bdx2eJpze74jkbKqpByxgXikdbLF", @@ -115,15 +115,15 @@ }, { "author": "whyrusleeping", - "hash": "QmPMtD39NN63AEUNghk1LFQcTLcCmYL8MtRzdv8BRUsC4Z", + "hash": "QmfH9FKYv3Jp1xiyL8sPchGBUBg6JA6XviwajAo3qgnT3B", "name": "go-libp2p-host", - "version": "3.0.7" + "version": "3.0.8" }, { "author": "whyrusleeping", - "hash": "QmYSM6PKnCe9YVPNMisfpoBmczzHkA7h5Wrnc36DtdJhGo", + "hash": "Qmcc5CPuKyfDZNmqXNkk6j23CyZqZGypUv952NLHYGbeni", "name": "go-libp2p-swarm", - "version": "3.0.9" + "version": "3.0.10" }, { "author": "whyrusleeping", @@ -139,9 +139,9 @@ }, { "author": "whyrusleeping", - "hash": "QmbKt1yZEnCWUmQhZR9BMiXK4e4FoaqnieAARz1s88MSzf", + "hash": "QmeKQeTT46oH3ezWeNLLaa8RtFbvijmxtyRiRuQD5wASDk", "name": "go-libp2p-blankhost", - "version": "0.3.7" + "version": "0.3.8" }, { "author": "whyrusleeping", @@ -157,9 +157,9 @@ }, { "author": "whyrusleeping", - "hash": "QmWzjXAyBTygw6CeCTUnhJzhFucfxY5FJivSoiGuiSbPjS", + "hash": "QmRYdszNNq7ykPqavVNKMVyyjX59AcTisHqzussDfhwHkK", "name": "go-smux-multistream", - "version": "2.0.1" + "version": "2.0.2" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "QmVYqPFBGi5wiuxpKxf4mUePEDdfXh8H7HKtq4mH8SeqML", + "hash": "QmUveY3vGb6uQuzccihinJshn3KUFjvb5PaBZUVhqGx8TJ", "name": "go-libp2p-circuit", - "version": "2.1.10" + "version": "2.1.11" }, { "author": "lgierth", @@ -199,9 +199,9 @@ }, { "author": "stebalien", - "hash": "QmSbkqfiFmJCdczVQ7mkFZf5FUUNpuP5Ne2LxY2htXGtrZ", + "hash": "Qmd4VzS3T7ckhHcLCpWwgVMYe6RcKhYjNKfy1g7LJ5myjv", "name": "go-conn-security-multistream", - "version": "0.1.7" + "version": "0.1.8" }, { "author": "Stebalien", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.10" + "version": "6.0.11" } From 124155a079ec92fcd0d081cecc462785572237f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 2 Sep 2018 12:03:02 +0100 Subject: [PATCH 0632/3965] rename ds package to pstoreds. --- p2p/host/peerstore/{ds => pstoreds}/addr_book.go | 2 +- p2p/host/peerstore/{ds => pstoreds}/ds_test.go | 2 +- p2p/host/peerstore/{ds => pstoreds}/peerstore.go | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) rename p2p/host/peerstore/{ds => pstoreds}/addr_book.go (99%) rename p2p/host/peerstore/{ds => pstoreds}/ds_test.go (98%) rename p2p/host/peerstore/{ds => pstoreds}/peerstore.go (97%) diff --git a/p2p/host/peerstore/ds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go similarity index 99% rename from p2p/host/peerstore/ds/addr_book.go rename to p2p/host/peerstore/pstoreds/addr_book.go index 2b57c0781f..8531a70ad9 100644 --- a/p2p/host/peerstore/ds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -1,4 +1,4 @@ -package ds +package pstoreds import ( "context" diff --git a/p2p/host/peerstore/ds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go similarity index 98% rename from p2p/host/peerstore/ds/ds_test.go rename to p2p/host/peerstore/pstoreds/ds_test.go index c3a19d50a1..f637cf1132 100644 --- a/p2p/host/peerstore/ds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -1,4 +1,4 @@ -package ds +package pstoreds import ( "context" diff --git a/p2p/host/peerstore/ds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go similarity index 97% rename from p2p/host/peerstore/ds/peerstore.go rename to p2p/host/peerstore/pstoreds/peerstore.go index 2c63f1bcd6..8551993e84 100644 --- a/p2p/host/peerstore/ds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -1,4 +1,4 @@ -package ds +package pstoreds import ( "context" From e65fb60b9767e9b2244cab24f1845b9f5c45114e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 2 Sep 2018 12:10:55 +0100 Subject: [PATCH 0633/3965] gx-go unrewrite. --- p2p/host/peerstore/addr/addrsrcs.go | 2 +- p2p/host/peerstore/addr/addrsrcs_test.go | 2 +- p2p/host/peerstore/addr/sorting.go | 6 +++--- p2p/host/peerstore/interface.go | 8 ++++---- p2p/host/peerstore/mem/addr_book.go | 6 +++--- p2p/host/peerstore/mem/keybook.go | 4 ++-- p2p/host/peerstore/mem/metadata.go | 2 +- p2p/host/peerstore/metrics.go | 2 +- p2p/host/peerstore/metrics_test.go | 2 +- p2p/host/peerstore/peerinfo.go | 4 ++-- p2p/host/peerstore/peerinfo_test.go | 4 ++-- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/pstoreds/addr_book.go | 14 +++++++------- p2p/host/peerstore/pstoreds/ds_test.go | 4 ++-- p2p/host/peerstore/pstoreds/peerstore.go | 2 +- p2p/host/peerstore/queue/distance.go | 4 ++-- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 4 ++-- p2p/host/peerstore/queue/sync.go | 4 ++-- p2p/host/peerstore/test/addr_manager_suite.go | 4 ++-- p2p/host/peerstore/test/benchmark_utils.go | 6 +++--- p2p/host/peerstore/test/peerstore_suite.go | 6 +++--- p2p/host/peerstore/test/test_utils.go | 6 +++--- 23 files changed, 50 insertions(+), 50 deletions(-) diff --git a/p2p/host/peerstore/addr/addrsrcs.go b/p2p/host/peerstore/addr/addrsrcs.go index fd959b1fae..9be78d5ccc 100644 --- a/p2p/host/peerstore/addr/addrsrcs.go +++ b/p2p/host/peerstore/addr/addrsrcs.go @@ -2,7 +2,7 @@ package addr import ( - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + ma "github.com/multiformats/go-multiaddr" ) // AddrSource is a source of addresses. It allows clients to retrieve diff --git a/p2p/host/peerstore/addr/addrsrcs_test.go b/p2p/host/peerstore/addr/addrsrcs_test.go index a186012f06..06bea07bce 100644 --- a/p2p/host/peerstore/addr/addrsrcs_test.go +++ b/p2p/host/peerstore/addr/addrsrcs_test.go @@ -4,7 +4,7 @@ import ( "fmt" "testing" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + ma "github.com/multiformats/go-multiaddr" ) func newAddrOrFatal(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index f66dc6cb18..5ca91d787f 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -3,9 +3,9 @@ package addr import ( "bytes" - "gx/ipfs/QmQscWDtDBDsWAM58aY6gU2KtxyFFmvvZgdfJExYPLgtXA/mafmt" - "gx/ipfs/QmV6FjemM1K8oXjrvuq3wuVWWoU2TLDPmNnKrxHzY3v6Ai/go-multiaddr-net" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + ma "github.com/multiformats/go-multiaddr" + "github.com/multiformats/go-multiaddr-net" + "github.com/whyrusleeping/mafmt" ) func isFDCostlyTransport(a ma.Multiaddr) bool { diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 9483b5132a..0f4d5ba8c7 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -5,11 +5,11 @@ import ( "math" "time" - "gx/ipfs/QmVmDhyTTUcQXFD1rRQ64fGLMSAoaQvNH3hwuaCFAPq2hy/errors" + "github.com/pkg/errors" - ic "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + ic "github.com/libp2p/go-libp2p-crypto" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) var ErrNotFound = errors.New("item not found") diff --git a/p2p/host/peerstore/mem/addr_book.go b/p2p/host/peerstore/mem/addr_book.go index e0c3f83109..72b7a6f5b9 100644 --- a/p2p/host/peerstore/mem/addr_book.go +++ b/p2p/host/peerstore/mem/addr_book.go @@ -6,9 +6,9 @@ import ( "sync" "time" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + logging "github.com/ipfs/go-log" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/addr" diff --git a/p2p/host/peerstore/mem/keybook.go b/p2p/host/peerstore/mem/keybook.go index 37e184e2af..290e6bd67d 100644 --- a/p2p/host/peerstore/mem/keybook.go +++ b/p2p/host/peerstore/mem/keybook.go @@ -4,8 +4,8 @@ import ( "errors" "sync" - ic "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + ic "github.com/libp2p/go-libp2p-crypto" + "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ) diff --git a/p2p/host/peerstore/mem/metadata.go b/p2p/host/peerstore/mem/metadata.go index 4a56035408..f7c0da7179 100644 --- a/p2p/host/peerstore/mem/metadata.go +++ b/p2p/host/peerstore/mem/metadata.go @@ -3,7 +3,7 @@ package mem import ( "sync" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ) diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index e269612e9a..05b867692b 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -4,7 +4,7 @@ import ( "sync" "time" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" ) // LatencyEWMASmooting governs the decay of the EWMA (the speed diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index a69e2fb969..2010db3089 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer/test" + "github.com/libp2p/go-libp2p-peer/test" ) func TestLatencyEWMAFun(t *testing.T) { diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index f7110440e6..f41b9f85ad 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -5,8 +5,8 @@ import ( "fmt" "strings" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) // PeerInfo is a small struct used to pass around a peer with diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index eaff1978fc..84a44b1797 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -3,8 +3,8 @@ package peerstore import ( "testing" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index d15d129edb..c29efdb15e 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -4,7 +4,7 @@ import ( "fmt" "sync" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" ) var _ Peerstore = (*peerstore)(nil) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 8531a70ad9..37425904ac 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -5,13 +5,13 @@ import ( "sync" "time" - mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" - "gx/ipfs/QmQjMHF8ptRgx4E57UFMiT4YM6kqaJeYxZ1MCDX23aw4rK/golang-lru" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log" - ds "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" - "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore/query" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + "github.com/hashicorp/golang-lru" + ds "github.com/ipfs/go-datastore" + "github.com/ipfs/go-datastore/query" + logging "github.com/ipfs/go-log" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" + mh "github.com/multiformats/go-multihash" pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/mem" diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index f637cf1132..29d6513cee 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -7,8 +7,8 @@ import ( "testing" "time" - "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" - "gx/ipfs/QmUCfrikzKVGAfpE31RPwPd32fu1DYxSG7HTGCadba5Wza/go-ds-badger" + "github.com/ipfs/go-datastore" + "github.com/ipfs/go-ds-badger" "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/test" diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 8551993e84..0ce3926a15 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -4,7 +4,7 @@ import ( "context" "time" - "gx/ipfs/QmSpg1CvpXQQow5ernt1gNBXaXV6yxyNqi7XoeerWfzB5w/go-datastore" + "github.com/ipfs/go-datastore" pstore "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/mem" diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index b33363dc97..9c3e026e55 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,8 +5,8 @@ import ( "math/big" "sync" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - ks "gx/ipfs/QmUusaX99BZoELh7dmPgirqRQ1FAmMnmnBn3oiqDFGBUSc/go-keyspace" + "github.com/libp2p/go-libp2p-peer" + ks "github.com/whyrusleeping/go-keyspace" ) // peerMetric tracks a peer and its distance to something else. diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index 321f5f30e1..26e1d48b7d 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" +import "github.com/libp2p/go-libp2p-peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 15a6e00b9c..93b0e9f570 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,8 +7,8 @@ import ( "testing" "time" - mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" + mh "github.com/multiformats/go-multihash" ) func TestQueue(t *testing.T) { diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 830fdebadf..3815ec782c 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -3,8 +3,8 @@ package queue import ( "context" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log" + logging "github.com/ipfs/go-log" + "github.com/libp2p/go-libp2p-peer" ) var log = logging.Logger("peerqueue") diff --git a/p2p/host/peerstore/test/addr_manager_suite.go b/p2p/host/peerstore/test/addr_manager_suite.go index a66918c224..acc0165744 100644 --- a/p2p/host/peerstore/test/addr_manager_suite.go +++ b/p2p/host/peerstore/test/addr_manager_suite.go @@ -4,8 +4,8 @@ import ( "testing" "time" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" pstore "github.com/libp2p/go-libp2p-peerstore" ) diff --git a/p2p/host/peerstore/test/benchmark_utils.go b/p2p/host/peerstore/test/benchmark_utils.go index 7008d5081d..b6d096cd5c 100644 --- a/p2p/host/peerstore/test/benchmark_utils.go +++ b/p2p/host/peerstore/test/benchmark_utils.go @@ -6,9 +6,9 @@ import ( "fmt" "testing" - mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" - "gx/ipfs/QmWFAMPqsEyUX7gDUsRVmMWz59FxSpJ1b2v6bJ1yYzo7jY/go-base58-fast/base58" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + "github.com/mr-tron/base58/base58" + ma "github.com/multiformats/go-multiaddr" + mh "github.com/multiformats/go-multihash" ) type peerpair struct { diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index 045d32d0bc..5317a4964f 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -8,9 +8,9 @@ import ( "testing" "time" - "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" - ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr" + "github.com/libp2p/go-libp2p-crypto" + "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" pstore "github.com/libp2p/go-libp2p-peerstore" ) diff --git a/p2p/host/peerstore/test/test_utils.go b/p2p/host/peerstore/test/test_utils.go index 4d0b90a697..fd66d73da9 100644 --- a/p2p/host/peerstore/test/test_utils.go +++ b/p2p/host/peerstore/test/test_utils.go @@ -5,9 +5,9 @@ import ( "math/rand" "time" - mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash" - ci "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto" - "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer" + ci "github.com/libp2p/go-libp2p-crypto" + "github.com/libp2p/go-libp2p-peer" + mh "github.com/multiformats/go-multihash" ) func timeSeededRand() io.Reader { From 8bdb5ba6941d62a5751c0de072ebfd1ec196ca5f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 2 Sep 2018 12:18:00 +0100 Subject: [PATCH 0634/3965] rename *Peers() to PeersWith*(). --- p2p/host/peerstore/interface.go | 8 ++++---- p2p/host/peerstore/mem/addr_book.go | 2 +- p2p/host/peerstore/mem/keybook.go | 2 +- p2p/host/peerstore/peerstore.go | 4 ++-- p2p/host/peerstore/pstoreds/addr_book.go | 2 +- 5 files changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 0f4d5ba8c7..468a8a12a9 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -110,8 +110,8 @@ type AddrBook interface { // ClearAddresses removes all previously stored addresses ClearAddrs(p peer.ID) - // AddrsPeers returns all of the peer IDs stored in the AddrBook - AddrsPeers() []peer.ID + // PeersWithAddrs returns all of the peer IDs stored in the AddrBook + PeersWithAddrs() []peer.ID } // KeyBook tracks the keys of Peers. @@ -129,6 +129,6 @@ type KeyBook interface { // AddPrivKey stores the private key of a peer. AddPrivKey(peer.ID, ic.PrivKey) error - // KeyBookPeers returns all the peer IDs stored in the KeyBook - KeyBookPeers() []peer.ID + // PeersWithKeys returns all the peer IDs stored in the KeyBook + PeersWithKeys() []peer.ID } diff --git a/p2p/host/peerstore/mem/addr_book.go b/p2p/host/peerstore/mem/addr_book.go index 72b7a6f5b9..d0cf980195 100644 --- a/p2p/host/peerstore/mem/addr_book.go +++ b/p2p/host/peerstore/mem/addr_book.go @@ -45,7 +45,7 @@ func NewAddrBook() pstore.AddrBook { } } -func (mab *memoryAddrBook) AddrsPeers() []peer.ID { +func (mab *memoryAddrBook) PeersWithAddrs() []peer.ID { mab.addrmu.Lock() defer mab.addrmu.Unlock() if mab.addrs == nil { diff --git a/p2p/host/peerstore/mem/keybook.go b/p2p/host/peerstore/mem/keybook.go index 290e6bd67d..053ffe401a 100644 --- a/p2p/host/peerstore/mem/keybook.go +++ b/p2p/host/peerstore/mem/keybook.go @@ -26,7 +26,7 @@ func NewKeyBook() pstore.KeyBook { } } -func (mkb *memoryKeyBook) KeyBookPeers() []peer.ID { +func (mkb *memoryKeyBook) PeersWithKeys() []peer.ID { mkb.RLock() ps := make([]peer.ID, 0, len(mkb.pks)+len(mkb.sks)) for p := range mkb.pks { diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index c29efdb15e..047926d2c6 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -32,10 +32,10 @@ func NewPeerstoreWith(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { func (ps *peerstore) Peers() []peer.ID { set := map[peer.ID]struct{}{} - for _, p := range ps.KeyBookPeers() { + for _, p := range ps.PeersWithKeys() { set[p] = struct{}{} } - for _, p := range ps.AddrsPeers() { + for _, p := range ps.PeersWithAddrs() { set[p] = struct{}{} } diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 37425904ac..8aebdd888d 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -193,7 +193,7 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { } // Peers returns all of the peer IDs for which the AddrBook has addresses. -func (mgr *dsAddrBook) AddrsPeers() []peer.ID { +func (mgr *dsAddrBook) PeersWithAddrs() []peer.ID { q := query.Query{KeysOnly: true} results, err := mgr.ds.Query(q) if err != nil { From bb971794b7850bce824a2d7472864a5b20775d48 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 2 Sep 2018 14:31:46 -0700 Subject: [PATCH 0635/3965] disable mdns logging from init (avoids the race condition) --- p2p/discovery/mdns.go | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/p2p/discovery/mdns.go b/p2p/discovery/mdns.go index 691b51a29b..e41c175caa 100644 --- a/p2p/discovery/mdns.go +++ b/p2p/discovery/mdns.go @@ -17,6 +17,11 @@ import ( "github.com/whyrusleeping/mdns" ) +func init() { + // don't let mdns use logging... + mdns.DisableLogging = true +} + var log = logging.Logger("mdns") const ServiceTag = "_ipfs-discovery._udp" @@ -62,9 +67,6 @@ func getDialableListenAddrs(ph host.Host) ([]*net.TCPAddr, error) { func NewMdnsService(ctx context.Context, peerhost host.Host, interval time.Duration, serviceTag string) (Service, error) { - // don't let mdns use logging... - mdns.DisableLogging = true - var ipaddrs []net.IP port := 4001 From b292631515ea85bc02ddb2e1849662f75c63c1e1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Sep 2018 11:42:42 +0100 Subject: [PATCH 0636/3965] readd import alias for manet. --- p2p/host/peerstore/addr/sorting.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index 5ca91d787f..b42b0410e3 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -4,7 +4,7 @@ import ( "bytes" ma "github.com/multiformats/go-multiaddr" - "github.com/multiformats/go-multiaddr-net" + manet "github.com/multiformats/go-multiaddr-net" "github.com/whyrusleeping/mafmt" ) From 1df741becc51c78450eb8dddec6d37f61282e14e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 31 Aug 2018 18:35:23 +0100 Subject: [PATCH 0637/3965] another perf improvement iteration. --- p2p/host/peerstore/pstoreds/addr_book.go | 374 ++++++++++++++++------- p2p/host/peerstore/pstoreds/ds_test.go | 2 +- p2p/host/peerstore/pstoreds/peerstore.go | 2 +- 3 files changed, 265 insertions(+), 113 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 5bb8b5e479..26ee580eb4 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -17,10 +17,12 @@ import ( "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) -var log = logging.Logger("peerstore/ds") +var ( + log = logging.Logger("peerstore/ds") -// Number of times to retry transactional writes -var dsWriteRetries = 5 + // Number of times to retry transactional writes + dsWriteRetries = 5 +) var _ pstore.AddrBook = (*dsAddrBook)(nil) @@ -28,15 +30,15 @@ var _ pstore.AddrBook = (*dsAddrBook)(nil) // in-memory TTL manager and an in-memory address stream manager. type dsAddrBook struct { cache *lru.ARCCache - ds ds.Batching - ttlManager *ttlmanager + ds ds.TxnDatastore + ttlManager *ttlManager subsManager *pstoremem.AddrSubManager } // NewAddrBook initializes a new address book given a // Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. -func NewAddrBook(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) (*dsAddrBook, error) { +func NewAddrBook(ctx context.Context, ds ds.TxnDatastore, ttlInterval time.Duration) (*dsAddrBook, error) { cache, err := lru.NewARC(1024) if err != nil { return nil, err @@ -56,12 +58,30 @@ func (mgr *dsAddrBook) Stop() { mgr.ttlManager.cancel() } -func peerAddressKey(p *peer.ID, addr *ma.Multiaddr) (ds.Key, error) { - hash, err := mh.Sum((*addr).Bytes(), mh.MURMUR3, -1) - if err != nil { - return ds.Key{}, nil +func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, error) { + var ( + keys = make([]ds.Key, len(addrs)) + clean = make([]ma.Multiaddr, len(addrs)) + parentKey = ds.NewKey(peer.IDB58Encode(p)) + i = 0 + ) + + for _, addr := range addrs { + if addr == nil { + continue + } + + hash, err := mh.Sum((addr).Bytes(), mh.MURMUR3, -1) + if err != nil { + return nil, nil, err + } + + keys[i] = parentKey.ChildString(hash.B58String()) + clean[i] = addr + i++ } - return ds.NewKey(peer.IDB58Encode(*p)).ChildString(hash.B58String()), nil + + return keys[:i], clean[:i], nil } func peerIDFromKey(key ds.Key) (peer.ID, error) { @@ -79,83 +99,164 @@ func (mgr *dsAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati if ttl <= 0 { return } - - mgr.setAddrs(p, addrs, ttl, true) + mgr.setAddrs(p, addrs, ttl, false) } // SetAddr will add or update the TTL of an address in the AddrBook. func (mgr *dsAddrBook) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.SetAddrs(p, []ma.Multiaddr{addr}, ttl) + addrs := []ma.Multiaddr{addr} + mgr.SetAddrs(p, addrs, ttl) } // SetAddrs will add or update the TTLs of addresses in the AddrBook. func (mgr *dsAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - mgr.setAddrs(p, addrs, ttl, false) + if ttl <= 0 { + mgr.deleteAddrs(p, addrs) + return + } + mgr.setAddrs(p, addrs, ttl, true) } -func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, add bool) { +func (mgr *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) error { + // Keys and cleaned up addresses. + keys, addrs, err := keysAndAddrs(p, addrs) + if err != nil { + return err + } + + // Evict all keys from cache. + for _, key := range keys { + mgr.cache.Remove(key) + } + + // Attempt transactional KV deletion. for i := 0; i < dsWriteRetries; i++ { - // keys to add to the TTL manager - var keys []ds.Key - batch, err := mgr.ds.Batch() - if err != nil { - log.Error(err) - return + if err = mgr.dbDelete(keys); err == nil { + break } + log.Errorf("failed to delete addresses for peer %s: %s\n", p.Pretty(), err) + } - for _, addr := range addrs { - if addr == nil { - continue - } + if err != nil { + log.Errorf("failed to avoid write conflict for peer %s after %d retries: %v\n", p.Pretty(), dsWriteRetries, err) + return err + } - key, err := peerAddressKey(&p, &addr) - if err != nil { - log.Error(err) - continue - } - keys = append(keys, key) + mgr.ttlManager.deleteTTLs(keys) + return nil +} - if ttl <= 0 { - if err := batch.Delete(key); err != nil { - log.Error(err) - } else { - mgr.cache.Remove(key) - } - continue - } +func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, ttlReset bool) error { + // Keys and cleaned up addresses. + keys, addrs, err := keysAndAddrs(p, addrs) + if err != nil { + return err + } - has := mgr.cache.Contains(key) - if !has { - has, err = mgr.ds.Has(key) - } - if err != nil || !has { - mgr.subsManager.BroadcastAddr(p, addr) - } + // Evict all keys from cache before the update. + for _, key := range keys { + mgr.cache.Remove(key) + } - // Allows us to support AddAddr and SetAddr in one function - if !has { - if err := batch.Put(key, addr.Bytes()); err != nil { - log.Error(err) - } else { - mgr.cache.Add(key, addr.Bytes()) - } - } + // Attempt transactional KV insertion. + var existed []bool + for i := 0; i < dsWriteRetries; i++ { + if existed, err = mgr.dbInsert(keys, addrs); err == nil { + break + } + log.Errorf("failed to write addresses for peer %s: %s\n", p.Pretty(), err) + } + + if err != nil { + log.Errorf("failed to avoid write conflict for peer %s after %d retries: %v\n", p.Pretty(), dsWriteRetries, err) + return err + } + + // Successful. Update cache and broadcast event. + for i, key := range keys { + addr := addrs[i] + mgr.cache.Add(key, addr.Bytes()) + + if !existed[i] { + mgr.subsManager.BroadcastAddr(p, addr) + } + } + + // Force update TTLs only if TTL reset was requested; otherwise + // insert the appropriate TTL entries if they don't already exist. + if ttlReset { + mgr.ttlManager.setTTLs(keys, ttl) + } else { + mgr.ttlManager.insertTTLs(keys, ttl) + } + + return nil +} + +// dbDelete performs a transactional delete of the provided keys. +func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { + var err error + + txn := mgr.ds.NewTransaction(false) + defer txn.Discard() + + // Attempt to delete all keys. + for _, key := range keys { + if err = txn.Delete(key); err != nil { + log.Errorf("transaction failed and aborted while deleting key: %s, cause: %v", key.String(), err) + return err } - if err := batch.Commit(); err != nil { - log.Errorf("failed to write addresses for peer %s: %s\n", p.Pretty(), err) + } + + if err = txn.Commit(); err != nil { + log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) + return err + } + return nil +} + +// dbInsert performs a transactional insert of the provided keys and values. +func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr) ([]bool, error) { + var ( + err error + existed = make([]bool, len(keys)) + ) + + txn := mgr.ds.NewTransaction(false) + defer txn.Discard() + + for i, key := range keys { + // Check if the key existed previously. + if existed[i], err = txn.Has(key); err != nil { + log.Errorf("transaction failed and aborted while checking key existence: %s, cause: %v", key.String(), err) + return nil, err + } + + // The key embeds a hash of the value, so if it existed, we can safely skip the insert. + if existed[i] { continue } - mgr.ttlManager.setTTLs(keys, ttl, add) - return + + // Attempt to add the key. + if err = txn.Put(key, addrs[i].Bytes()); err != nil { + log.Errorf("transaction failed and aborted while setting key: %s, cause: %v", key.String(), err) + return nil, err + } } - log.Errorf("failed to avoid write conflict for peer %s after %d retries\n", p.Pretty(), dsWriteRetries) + + if err = txn.Commit(); err != nil { + log.Errorf("failed to commit transaction when setting keys, cause: %v", err) + return nil, err + } + + return existed, nil } // UpdateAddrs will update any addresses for a given peer and TTL combination to // have a new TTL. func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { prefix := ds.NewKey(p.Pretty()) - mgr.ttlManager.updateTTLs(prefix, oldTTL, newTTL) + mgr.ttlManager.adjustTTLs(prefix, oldTTL, newTTL) } // Addrs Returns all of the non-expired addresses for a given peer. @@ -163,6 +264,7 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { prefix := ds.NewKey(p.Pretty()) q := query.Query{Prefix: prefix.String(), KeysOnly: true} results, err := mgr.ds.Query(q) + if err != nil { log.Error(err) return nil @@ -196,6 +298,7 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { q := query.Query{KeysOnly: true} results, err := mgr.ds.Query(q) + if err != nil { log.Error(err) return peer.IDSlice{} @@ -227,39 +330,81 @@ func (mgr *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mult // ClearAddrs will delete all known addresses for a peer ID. func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { - prefix := ds.NewKey(p.Pretty()) + var ( + err error + keys []ds.Key + prefix = ds.NewKey(p.Pretty()) + ) + + // Attempt transactional KV deletion. for i := 0; i < dsWriteRetries; i++ { - q := query.Query{Prefix: prefix.String(), KeysOnly: true} - results, err := mgr.ds.Query(q) - if err != nil { - log.Error(err) - return - } - batch, err := mgr.ds.Batch() - if err != nil { - log.Error(err) - return + if keys, err = mgr.dbClear(prefix); err == nil { + break } + log.Errorf("failed to clear addresses for peer %s: %s\n", p.Pretty(), err) + } - for result := range results.Next() { - key := ds.NewKey(result.Key) - err := batch.Delete(key) - if err != nil { - // From inspectin badger, errors here signify a problem with - // the transaction as a whole, so we can log and abort. - log.Error(err) - return - } - mgr.cache.Remove(key) - } - if err = batch.Commit(); err != nil { - log.Errorf("failed to clear addresses for peer %s: %s\n", p.Pretty(), err) - continue + if err != nil { + log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), dsWriteRetries) + // TODO: return error + } + + // Perform housekeeping. + mgr.ttlManager.clear(prefix) + for _, key := range keys { + mgr.cache.Remove(key) + } +} + +func (mgr *dsAddrBook) dbClear(prefix ds.Key) ([]ds.Key, error) { + q := query.Query{Prefix: prefix.String(), KeysOnly: true} + + txn := mgr.ds.NewTransaction(false) + defer txn.Discard() + + results, err := txn.Query(q) + if err != nil { + log.Errorf("failed to fetch all keys prefixed with: %s, cause: %v", prefix.String(), err) + return nil, err + } + + var keys []ds.Key + for result := range results.Next() { + key := ds.RawKey(result.Key) + keys = append(keys, key) + + if err = txn.Delete(key); err != nil { + log.Errorf("failed to delete key: %s, cause: %v", key.String(), err) + return nil, err } - mgr.ttlManager.clear(ds.NewKey(p.Pretty())) - return } - log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), dsWriteRetries) + + if err := results.Close(); err != nil { + log.Errorf("failed to close cursor: %s, cause: %v", err) + return nil, err + } + + if err = txn.Commit(); err != nil { + log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) + return nil, err + } + + return keys, nil +} + +func (mgr *dsAddrBook) doInTransaction(readOnly bool, op func(txn ds.Txn) error) error { + txn := mgr.ds.NewTransaction(false) + defer txn.Discard() + + if err := op(txn); err != nil { + return err + } + + if err := txn.Commit(); err != nil { + return err + } + + return nil } type ttlentry struct { @@ -267,7 +412,7 @@ type ttlentry struct { ExpiresAt time.Time } -type ttlmanager struct { +type ttlManager struct { sync.RWMutex entries map[ds.Key]*ttlentry @@ -278,13 +423,13 @@ type ttlmanager struct { cache *lru.ARCCache } -func newTTLManager(parent context.Context, d ds.Datastore, c *lru.ARCCache, tick time.Duration) *ttlmanager { +func newTTLManager(parent context.Context, d ds.Datastore, c *lru.ARCCache, tick time.Duration) *ttlManager { ctx, cancel := context.WithCancel(parent) batching, ok := d.(ds.Batching) if !ok { - panic("must construct ttlmanager with batching datastore") + panic("must construct ttlManager with batching datastore") } - mgr := &ttlmanager{ + mgr := &ttlManager{ entries: make(map[ds.Key]*ttlentry), ctx: ctx, cancel: cancel, @@ -309,7 +454,7 @@ func newTTLManager(parent context.Context, d ds.Datastore, c *lru.ARCCache, tick } // To be called by TTL manager's coroutine only. -func (mgr *ttlmanager) tick() { +func (mgr *ttlManager) tick() { mgr.Lock() defer mgr.Unlock() @@ -335,31 +480,38 @@ func (mgr *ttlmanager) tick() { } } -func (mgr *ttlmanager) setTTLs(keys []ds.Key, ttl time.Duration, add bool) { +func (mgr *ttlManager) deleteTTLs(keys []ds.Key) { + mgr.Lock() + defer mgr.Unlock() + + for _, key := range keys { + delete(mgr.entries, key) + } +} + +func (mgr *ttlManager) insertTTLs(keys []ds.Key, ttl time.Duration) { mgr.Lock() defer mgr.Unlock() expiration := time.Now().Add(ttl) for _, key := range keys { - update := true - if add { - if entry, ok := mgr.entries[key]; ok { - if entry.ExpiresAt.After(expiration) { - update = false - } - } - } - if update { - if ttl <= 0 { - delete(mgr.entries, key) - } else { - mgr.entries[key] = &ttlentry{TTL: ttl, ExpiresAt: expiration} - } + if entry, ok := mgr.entries[key]; !ok || (ok && entry.ExpiresAt.Before(expiration)) { + mgr.entries[key] = &ttlentry{TTL: ttl, ExpiresAt: expiration} } } } -func (mgr *ttlmanager) updateTTLs(prefix ds.Key, oldTTL, newTTL time.Duration) { +func (mgr *ttlManager) setTTLs(keys []ds.Key, ttl time.Duration) { + mgr.Lock() + defer mgr.Unlock() + + expiration := time.Now().Add(ttl) + for _, key := range keys { + mgr.entries[key] = &ttlentry{TTL: ttl, ExpiresAt: expiration} + } +} + +func (mgr *ttlManager) adjustTTLs(prefix ds.Key, oldTTL, newTTL time.Duration) { mgr.Lock() defer mgr.Unlock() @@ -374,7 +526,7 @@ func (mgr *ttlmanager) updateTTLs(prefix ds.Key, oldTTL, newTTL time.Duration) { } } -func (mgr *ttlmanager) clear(prefix ds.Key) { +func (mgr *ttlManager) clear(prefix ds.Key) { mgr.Lock() defer mgr.Unlock() diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 4ed24d0e9d..58a0fd29f0 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -14,7 +14,7 @@ import ( "github.com/libp2p/go-libp2p-peerstore/test" ) -func setupBadgerDatastore(t testing.TB) (datastore.Batching, func()) { +func setupBadgerDatastore(t testing.TB) (datastore.TxnDatastore, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index e973988bd5..3a219b21ba 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -11,7 +11,7 @@ import ( ) // NewPeerstore creates a peerstore backed by the provided persistent datastore. -func NewPeerstore(ctx context.Context, ds datastore.Batching) (pstore.Peerstore, error) { +func NewPeerstore(ctx context.Context, ds datastore.TxnDatastore) (pstore.Peerstore, error) { addrBook, err := NewAddrBook(ctx, ds, time.Second) if err != nil { return nil, err From 5e3c5f3ed3193c07ec47819355304407c9996aa7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Sep 2018 11:34:49 +0100 Subject: [PATCH 0638/3965] regroup test and benchmark utils. --- p2p/host/peerstore/test/test_utils.go | 22 ------------------- .../test/{benchmark_utils.go => utils.go} | 18 ++++++++++++++- 2 files changed, 17 insertions(+), 23 deletions(-) delete mode 100644 p2p/host/peerstore/test/test_utils.go rename p2p/host/peerstore/test/{benchmark_utils.go => utils.go} (76%) diff --git a/p2p/host/peerstore/test/test_utils.go b/p2p/host/peerstore/test/test_utils.go deleted file mode 100644 index 4b282c446e..0000000000 --- a/p2p/host/peerstore/test/test_utils.go +++ /dev/null @@ -1,22 +0,0 @@ -package test - -import ( - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" -) - -func peerId(ids string) peer.ID { - id, err := peer.IDB58Decode(ids) - if err != nil { - panic(err) - } - return id -} - -func multiaddr(m string) ma.Multiaddr { - maddr, err := ma.NewMultiaddr(m) - if err != nil { - panic(err) - } - return maddr -} diff --git a/p2p/host/peerstore/test/benchmark_utils.go b/p2p/host/peerstore/test/utils.go similarity index 76% rename from p2p/host/peerstore/test/benchmark_utils.go rename to p2p/host/peerstore/test/utils.go index 714bf0d3e7..ebb049b583 100644 --- a/p2p/host/peerstore/test/benchmark_utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -10,6 +10,22 @@ import ( ma "github.com/multiformats/go-multiaddr" ) +func peerId(ids string) peer.ID { + id, err := peer.IDB58Decode(ids) + if err != nil { + panic(err) + } + return id +} + +func multiaddr(m string) ma.Multiaddr { + maddr, err := ma.NewMultiaddr(m) + if err != nil { + panic(err) + } + return maddr +} + type peerpair struct { ID peer.ID Addr ma.Multiaddr @@ -41,4 +57,4 @@ func addressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair) { return } } -} +} \ No newline at end of file From 5f855964cf4d573196f8de60cc8448e2db2b1acf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Sep 2018 11:34:55 +0100 Subject: [PATCH 0639/3965] increase benchmark coverage; introduce config struct. * benchmark caching and cacheless variants of DS addrbook. * benchmark peers with 1, 10, 100 addrs. --- p2p/host/peerstore/pstoreds/addr_book.go | 63 +++++++++++---------- p2p/host/peerstore/pstoreds/cache.go | 33 +++++++++++ p2p/host/peerstore/pstoreds/ds_test.go | 36 ++++++++++-- p2p/host/peerstore/pstoreds/peerstore.go | 28 ++++++++- p2p/host/peerstore/pstoremem/inmem_test.go | 2 +- p2p/host/peerstore/test/benchmarks_suite.go | 52 +++++++++++++++++ p2p/host/peerstore/test/peerstore_suite.go | 50 ++++++++++------ p2p/host/peerstore/test/utils.go | 30 ++++++---- 8 files changed, 228 insertions(+), 66 deletions(-) create mode 100644 p2p/host/peerstore/pstoreds/cache.go create mode 100644 p2p/host/peerstore/test/benchmarks_suite.go diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 26ee580eb4..a2d5b1b3ca 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -19,9 +19,6 @@ import ( var ( log = logging.Logger("peerstore/ds") - - // Number of times to retry transactional writes - dsWriteRetries = 5 ) var _ pstore.AddrBook = (*dsAddrBook)(nil) @@ -29,26 +26,34 @@ var _ pstore.AddrBook = (*dsAddrBook)(nil) // dsAddrBook is an address book backed by a Datastore with both an // in-memory TTL manager and an in-memory address stream manager. type dsAddrBook struct { - cache *lru.ARCCache - ds ds.TxnDatastore - ttlManager *ttlManager - subsManager *pstoremem.AddrSubManager + cache cache + ds ds.TxnDatastore + ttlManager *ttlManager + subsManager *pstoremem.AddrSubManager + writeRetries int } // NewAddrBook initializes a new address book given a // Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. -func NewAddrBook(ctx context.Context, ds ds.TxnDatastore, ttlInterval time.Duration) (*dsAddrBook, error) { - cache, err := lru.NewARC(1024) - if err != nil { - return nil, err +func NewAddrBook(ctx context.Context, ds ds.TxnDatastore, opts PeerstoreOpts) (*dsAddrBook, error) { + var ( + cache cache = &noopCache{} + err error + ) + + if opts.CacheSize > 0 { + if cache, err = lru.NewARC(int(opts.CacheSize)); err != nil { + return nil, err + } } mgr := &dsAddrBook{ - cache: cache, - ds: ds, - ttlManager: newTTLManager(ctx, ds, cache, ttlInterval), - subsManager: pstoremem.NewAddrSubManager(), + cache: cache, + ds: ds, + ttlManager: newTTLManager(ctx, ds, &cache, opts.TTLInterval), + subsManager: pstoremem.NewAddrSubManager(), + writeRetries: int(opts.WriteRetries), } return mgr, nil } @@ -130,7 +135,7 @@ func (mgr *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) error { } // Attempt transactional KV deletion. - for i := 0; i < dsWriteRetries; i++ { + for i := 0; i < mgr.writeRetries; i++ { if err = mgr.dbDelete(keys); err == nil { break } @@ -138,7 +143,7 @@ func (mgr *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) error { } if err != nil { - log.Errorf("failed to avoid write conflict for peer %s after %d retries: %v\n", p.Pretty(), dsWriteRetries, err) + log.Errorf("failed to avoid write conflict for peer %s after %d retries: %v\n", p.Pretty(), mgr.writeRetries, err) return err } @@ -160,7 +165,7 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati // Attempt transactional KV insertion. var existed []bool - for i := 0; i < dsWriteRetries; i++ { + for i := 0; i < mgr.writeRetries; i++ { if existed, err = mgr.dbInsert(keys, addrs); err == nil { break } @@ -168,7 +173,7 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati } if err != nil { - log.Errorf("failed to avoid write conflict for peer %s after %d retries: %v\n", p.Pretty(), dsWriteRetries, err) + log.Errorf("failed to avoid write conflict for peer %s after %d retries: %v\n", p.Pretty(), mgr.writeRetries, err) return err } @@ -337,7 +342,7 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { ) // Attempt transactional KV deletion. - for i := 0; i < dsWriteRetries; i++ { + for i := 0; i < mgr.writeRetries; i++ { if keys, err = mgr.dbClear(prefix); err == nil { break } @@ -345,7 +350,7 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { } if err != nil { - log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), dsWriteRetries) + log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), mgr.writeRetries) // TODO: return error } @@ -407,35 +412,35 @@ func (mgr *dsAddrBook) doInTransaction(readOnly bool, op func(txn ds.Txn) error) return nil } -type ttlentry struct { +type ttlEntry struct { TTL time.Duration ExpiresAt time.Time } type ttlManager struct { sync.RWMutex - entries map[ds.Key]*ttlentry + entries map[ds.Key]*ttlEntry ctx context.Context cancel context.CancelFunc ticker *time.Ticker ds ds.Batching - cache *lru.ARCCache + cache cache } -func newTTLManager(parent context.Context, d ds.Datastore, c *lru.ARCCache, tick time.Duration) *ttlManager { +func newTTLManager(parent context.Context, d ds.Datastore, c *cache, tick time.Duration) *ttlManager { ctx, cancel := context.WithCancel(parent) batching, ok := d.(ds.Batching) if !ok { panic("must construct ttlManager with batching datastore") } mgr := &ttlManager{ - entries: make(map[ds.Key]*ttlentry), + entries: make(map[ds.Key]*ttlEntry), ctx: ctx, cancel: cancel, ticker: time.NewTicker(tick), ds: batching, - cache: c, + cache: *c, } go func() { @@ -496,7 +501,7 @@ func (mgr *ttlManager) insertTTLs(keys []ds.Key, ttl time.Duration) { expiration := time.Now().Add(ttl) for _, key := range keys { if entry, ok := mgr.entries[key]; !ok || (ok && entry.ExpiresAt.Before(expiration)) { - mgr.entries[key] = &ttlentry{TTL: ttl, ExpiresAt: expiration} + mgr.entries[key] = &ttlEntry{TTL: ttl, ExpiresAt: expiration} } } } @@ -507,7 +512,7 @@ func (mgr *ttlManager) setTTLs(keys []ds.Key, ttl time.Duration) { expiration := time.Now().Add(ttl) for _, key := range keys { - mgr.entries[key] = &ttlentry{TTL: ttl, ExpiresAt: expiration} + mgr.entries[key] = &ttlEntry{TTL: ttl, ExpiresAt: expiration} } } diff --git a/p2p/host/peerstore/pstoreds/cache.go b/p2p/host/peerstore/pstoreds/cache.go new file mode 100644 index 0000000000..2e20ae6294 --- /dev/null +++ b/p2p/host/peerstore/pstoreds/cache.go @@ -0,0 +1,33 @@ +package pstoreds + +// cache abstracts all methods we access from ARCCache, to enable alternate +// implementations such as a no-op one. +type cache interface { + Get(key interface{}) (value interface{}, ok bool) + Add(key, value interface{}) + Remove(key interface{}) + Contains(key interface{}) bool + Peek(key interface{}) (value interface{}, ok bool) +} + +// noopCache is a dummy implementation that's used when the cache is disabled. +type noopCache struct { +} + +func (*noopCache) Get(key interface{}) (value interface{}, ok bool) { + return nil, false +} + +func (*noopCache) Add(key, value interface{}) { +} + +func (*noopCache) Remove(key interface{}) { +} + +func (*noopCache) Contains(key interface{}) bool { + return false +} + +func (*noopCache) Peek(key interface{}) (value interface{}, ok bool) { + return nil, false +} diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 58a0fd29f0..e14e6e1c7e 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -30,11 +30,32 @@ func setupBadgerDatastore(t testing.TB) (datastore.TxnDatastore, func()) { return ds, closer } -func newPeerstoreFactory(tb testing.TB) test.PeerstoreFactory { +func cachingPeerstore(tb testing.TB) test.PeerstoreFactory { + opts := DefaultOpts() + opts.CacheSize = 1024 + opts.TTLInterval = 100 * time.Microsecond + + return func() (peerstore.Peerstore, func()) { + ds, closeFunc := setupBadgerDatastore(tb) + + ps, err := NewPeerstore(context.Background(), ds, opts) + if err != nil { + tb.Fatal(err) + } + + return ps, closeFunc + } +} + +func cachelessPeerstore(tb testing.TB) test.PeerstoreFactory { + opts := DefaultOpts() + opts.CacheSize = 0 + opts.TTLInterval = 100 * time.Microsecond + return func() (peerstore.Peerstore, func()) { ds, closeFunc := setupBadgerDatastore(tb) - ps, err := NewPeerstore(context.Background(), ds) + ps, err := NewPeerstore(context.Background(), ds, opts) if err != nil { tb.Fatal(err) } @@ -44,14 +65,18 @@ func newPeerstoreFactory(tb testing.TB) test.PeerstoreFactory { } func TestBadgerDsPeerstore(t *testing.T) { - test.TestPeerstore(t, newPeerstoreFactory(t)) + test.TestPeerstore(t, cachingPeerstore(t)) } func TestBadgerDsAddrBook(t *testing.T) { + opts := DefaultOpts() + opts.CacheSize = 0 + opts.TTLInterval = 100 * time.Microsecond + test.TestAddrBook(t, func() (peerstore.AddrBook, func()) { ds, closeDB := setupBadgerDatastore(t) - mgr, err := NewAddrBook(context.Background(), ds, 100*time.Microsecond) + mgr, err := NewAddrBook(context.Background(), ds, opts) if err != nil { t.Fatal(err) } @@ -65,5 +90,6 @@ func TestBadgerDsAddrBook(t *testing.T) { } func BenchmarkBadgerDsPeerstore(b *testing.B) { - test.BenchmarkPeerstore(b, newPeerstoreFactory(b)) + test.BenchmarkPeerstore(b, cachingPeerstore(b), "Caching") + test.BenchmarkPeerstore(b, cachelessPeerstore(b), "Cacheless") } diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 3a219b21ba..480edcd9e6 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -10,9 +10,33 @@ import ( "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) +// Configuration object for the peerstore. +type PeerstoreOpts struct { + // The size of the in-memory cache. A value of 0 or lower disables the cache. + CacheSize uint + + // Sweep interval to expire entries when TTL is not managed by underlying datastore. + TTLInterval time.Duration + + // Number of times to retry transactional writes. + WriteRetries uint +} + +// DefaultOpts returns the default options for a persistent peerstore: +// * Cache size: 1024 +// * TTL sweep interval: 1 second +// * WriteRetries: 5 +func DefaultOpts() PeerstoreOpts { + return PeerstoreOpts{ + CacheSize: 1024, + TTLInterval: time.Second, + WriteRetries: 5, + } +} + // NewPeerstore creates a peerstore backed by the provided persistent datastore. -func NewPeerstore(ctx context.Context, ds datastore.TxnDatastore) (pstore.Peerstore, error) { - addrBook, err := NewAddrBook(ctx, ds, time.Second) +func NewPeerstore(ctx context.Context, ds datastore.TxnDatastore, opts PeerstoreOpts) (pstore.Peerstore, error) { + addrBook, err := NewAddrBook(ctx, ds, opts) if err != nil { return nil, err } diff --git a/p2p/host/peerstore/pstoremem/inmem_test.go b/p2p/host/peerstore/pstoremem/inmem_test.go index e13135ea34..a03b4dd5a8 100644 --- a/p2p/host/peerstore/pstoremem/inmem_test.go +++ b/p2p/host/peerstore/pstoremem/inmem_test.go @@ -28,5 +28,5 @@ func TestInMemoryKeyBook(t *testing.T) { func BenchmarkInMemoryPeerstore(b *testing.B) { test.BenchmarkPeerstore(b, func() (pstore.Peerstore, func()) { return NewPeerstore(), nil - }) + }, "InMem") } diff --git a/p2p/host/peerstore/test/benchmarks_suite.go b/p2p/host/peerstore/test/benchmarks_suite.go new file mode 100644 index 0000000000..cd5f2df733 --- /dev/null +++ b/p2p/host/peerstore/test/benchmarks_suite.go @@ -0,0 +1,52 @@ +package test + +import ( + "context" + "fmt" + "testing" + + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +var peerstoreBenchmarks = map[string]func(pstore.Peerstore, chan *peerpair) func(*testing.B){ + "AddAddr": benchmarkAddAddr, + "AddAddrs": benchmarkAddAddrs, + "SetAddrs": benchmarkSetAddrs, + "GetAddrs": benchmarkGetAddrs, +} + +func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory, variant string) { + // Parameterises benchmarks to tackle peers with 1, 10, 100 multiaddrs. + params := []struct { + n int + ch chan *peerpair + }{ + {1, make(chan *peerpair, 100)}, + {10, make(chan *peerpair, 100)}, + {100, make(chan *peerpair, 100)}, + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // Start all test peer producing goroutines, where each produces peers with as many + // multiaddrs as the n field in the param struct. + for _, p := range params { + go addressProducer(ctx, b, p.ch, p.n) + } + + for name, bench := range peerstoreBenchmarks { + for _, p := range params { + // Create a new peerstore. + ps, closeFunc := factory() + + // Run the test. + b.Run(fmt.Sprintf("%s-%dAddrs-%s", name, p.n, variant), bench(ps, p.ch)) + + // Cleanup. + if closeFunc != nil { + closeFunc() + } + } + } +} diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index eac6aad119..0a057f0aad 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -15,7 +15,7 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" ) -var peerstoreSuite = map[string]func(book pstore.Peerstore) func(*testing.T){ +var peerstoreSuite = map[string]func(pstore.Peerstore) func(*testing.T){ "AddrStream": testAddrStream, "GetStreamBeforePeerAdded": testGetStreamBeforePeerAdded, "AddStreamDuplicates": testAddrStreamDuplicates, @@ -40,16 +40,6 @@ func TestPeerstore(t *testing.T, factory PeerstoreFactory) { } } -func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory) { - ps, closeFunc := factory() - - b.Run("Peerstore", benchmarkPeerstore(ps)) - - if closeFunc != nil { - closeFunc() - } -} - func testAddrStream(ps pstore.Peerstore) func(t *testing.T) { return func(t *testing.T) { addrs, pid := getAddrs(t, 100), peer.ID("testpeer") @@ -289,19 +279,45 @@ func testBasicPeerstore(ps pstore.Peerstore) func(t *testing.T) { } } -func benchmarkPeerstore(ps pstore.Peerstore) func(*testing.B) { +func benchmarkAddAddr(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { return func(b *testing.B) { - addrs := make(chan *peerpair, 100) + b.ResetTimer() + for i := 0; i < b.N; i++ { + pp := <-addrs + ps.AddAddr(pp.ID, pp.Addr[0], pstore.PermanentAddrTTL) + } + } +} - ctx, cancel := context.WithCancel(context.Background()) - go addressProducer(ctx, b, addrs) +func benchmarkAddAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { + return func(b *testing.B) { + b.ResetTimer() + for i := 0; i < b.N; i++ { + pp := <-addrs + ps.AddAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + } + } +} +func benchmarkSetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { + return func(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { pp := <-addrs - ps.AddAddr(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + ps.SetAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + } + } +} + +func benchmarkGetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { + return func(b *testing.B) { + pp := <-addrs + ps.SetAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + _ = ps.Addrs(pp.ID) } - cancel() } } diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index ebb049b583..8b7a86a9f0 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -28,33 +28,39 @@ func multiaddr(m string) ma.Multiaddr { type peerpair struct { ID peer.ID - Addr ma.Multiaddr + Addr []ma.Multiaddr } -func randomPeer(b *testing.B) *peerpair { - var pid peer.ID - var err error - var addr ma.Multiaddr +func randomPeer(b *testing.B, addrCount int) *peerpair { + var ( + pid peer.ID + err error + addrs = make([]ma.Multiaddr, addrCount) + aFmt = "/ip4/127.0.0.1/tcp/%d/ipfs/%s" + ) + b.Helper() if pid, err = pt.RandPeerID(); err != nil { b.Fatal(err) } - if addr, err = ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/6666/ipfs/%s", pid.Pretty())); err != nil { - b.Fatal(err) + for i := 0; i < addrCount; i++ { + if addrs[i], err = ma.NewMultiaddr(fmt.Sprintf(aFmt, i, pid.Pretty())); err != nil { + b.Fatal(err) + } } - - return &peerpair{pid, addr} + return &peerpair{pid, addrs} } -func addressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair) { +func addressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair, addrsPerPeer int) { + b.Helper() defer close(addrs) for { - p := randomPeer(b) + p := randomPeer(b, addrsPerPeer) select { case addrs <- p: case <-ctx.Done(): return } } -} \ No newline at end of file +} From 6e88c538ee0342d0391de5a724eb28fdbe616034 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Sep 2018 14:30:52 +0100 Subject: [PATCH 0640/3965] migrate TTL manager to ds.TxnDatastore. --- p2p/host/peerstore/pstoreds/addr_book.go | 37 ++++++++++++------------ 1 file changed, 18 insertions(+), 19 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index a2d5b1b3ca..2ffcf128c3 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -424,22 +424,22 @@ type ttlManager struct { ctx context.Context cancel context.CancelFunc ticker *time.Ticker - ds ds.Batching + ds ds.TxnDatastore cache cache } func newTTLManager(parent context.Context, d ds.Datastore, c *cache, tick time.Duration) *ttlManager { ctx, cancel := context.WithCancel(parent) - batching, ok := d.(ds.Batching) + txnDs, ok := d.(ds.TxnDatastore) if !ok { - panic("must construct ttlManager with batching datastore") + panic("must construct ttlManager with transactional datastore") } mgr := &ttlManager{ entries: make(map[ds.Key]*ttlEntry), ctx: ctx, cancel: cancel, ticker: time.NewTicker(tick), - ds: batching, + ds: txnDs, cache: *c, } @@ -463,25 +463,24 @@ func (mgr *ttlManager) tick() { mgr.Lock() defer mgr.Unlock() + txn := mgr.ds.NewTransaction(false) + defer txn.Discard() + now := time.Now() - batch, err := mgr.ds.Batch() - if err != nil { - log.Error(err) - return - } for key, entry := range mgr.entries { - if entry.ExpiresAt.Before(now) { - if err := batch.Delete(key); err != nil { - log.Error(err) - } else { - mgr.cache.Remove(key) - } - delete(mgr.entries, key) + if entry.ExpiresAt.After(now) { + continue + } + if err := txn.Delete(key); err != nil { + log.Error("failed to delete TTL key: %v, cause: %v", key.String(), err) + break } + mgr.cache.Remove(key) + delete(mgr.entries, key) } - err = batch.Commit() - if err != nil { - log.Error(err) + + if err := txn.Commit(); err != nil { + log.Error("failed to commit TTL deletion, cause: %v", err) } } From 9f64a46d219df89f338d61c7fa2cb3e1974cc437 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Sep 2018 14:52:26 +0100 Subject: [PATCH 0641/3965] ttl manager: create transaction only when needed. Also adjust test logic. Last changes were ticking the ttl manager too often unnecessarily. This was skewing benchmarks. --- p2p/host/peerstore/pstoreds/addr_book.go | 15 ++++- p2p/host/peerstore/pstoreds/ds_test.go | 73 ++++++++++-------------- 2 files changed, 42 insertions(+), 46 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 2ffcf128c3..ba4ee275bc 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -463,14 +463,23 @@ func (mgr *ttlManager) tick() { mgr.Lock() defer mgr.Unlock() - txn := mgr.ds.NewTransaction(false) - defer txn.Discard() - now := time.Now() + var toDel []ds.Key for key, entry := range mgr.entries { if entry.ExpiresAt.After(now) { continue } + toDel = append(toDel, key) + } + + if len(toDel) == 0 { + return + } + + txn := mgr.ds.NewTransaction(false) + defer txn.Discard() + + for _, key := range toDel { if err := txn.Delete(key); err != nil { log.Error("failed to delete TTL key: %v, cause: %v", key.String(), err) break diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index e14e6e1c7e..5ed07dd9e5 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -9,12 +9,33 @@ import ( "github.com/ipfs/go-datastore" "github.com/ipfs/go-ds-badger" - "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/test" ) -func setupBadgerDatastore(t testing.TB) (datastore.TxnDatastore, func()) { +func TestBadgerDsPeerstore(t *testing.T) { + test.TestPeerstore(t, peerstoreFactory(t, DefaultOpts())) +} + +func TestBadgerDsAddrBook(t *testing.T) { + opts := DefaultOpts() + opts.TTLInterval = 100 * time.Microsecond + + test.TestAddrBook(t, addressBookFactory(t, opts)) +} + +func BenchmarkBadgerDsPeerstore(b *testing.B) { + caching := DefaultOpts() + caching.CacheSize = 1024 + + cacheless := DefaultOpts() + cacheless.CacheSize = 0 + + test.BenchmarkPeerstore(b, peerstoreFactory(b, caching), "Caching") + test.BenchmarkPeerstore(b, peerstoreFactory(b, cacheless), "Cacheless") +} + +func badgerStore(t testing.TB) (datastore.TxnDatastore, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) @@ -30,30 +51,9 @@ func setupBadgerDatastore(t testing.TB) (datastore.TxnDatastore, func()) { return ds, closer } -func cachingPeerstore(tb testing.TB) test.PeerstoreFactory { - opts := DefaultOpts() - opts.CacheSize = 1024 - opts.TTLInterval = 100 * time.Microsecond - - return func() (peerstore.Peerstore, func()) { - ds, closeFunc := setupBadgerDatastore(tb) - - ps, err := NewPeerstore(context.Background(), ds, opts) - if err != nil { - tb.Fatal(err) - } - - return ps, closeFunc - } -} - -func cachelessPeerstore(tb testing.TB) test.PeerstoreFactory { - opts := DefaultOpts() - opts.CacheSize = 0 - opts.TTLInterval = 100 * time.Microsecond - +func peerstoreFactory(tb testing.TB, opts PeerstoreOpts) test.PeerstoreFactory { return func() (peerstore.Peerstore, func()) { - ds, closeFunc := setupBadgerDatastore(tb) + ds, closeFunc := badgerStore(tb) ps, err := NewPeerstore(context.Background(), ds, opts) if err != nil { @@ -64,21 +64,13 @@ func cachelessPeerstore(tb testing.TB) test.PeerstoreFactory { } } -func TestBadgerDsPeerstore(t *testing.T) { - test.TestPeerstore(t, cachingPeerstore(t)) -} - -func TestBadgerDsAddrBook(t *testing.T) { - opts := DefaultOpts() - opts.CacheSize = 0 - opts.TTLInterval = 100 * time.Microsecond - - test.TestAddrBook(t, func() (peerstore.AddrBook, func()) { - ds, closeDB := setupBadgerDatastore(t) +func addressBookFactory(tb testing.TB, opts PeerstoreOpts) test.AddrBookFactory { + return func() (peerstore.AddrBook, func()) { + ds, closeDB := badgerStore(tb) mgr, err := NewAddrBook(context.Background(), ds, opts) if err != nil { - t.Fatal(err) + tb.Fatal(err) } closeFunc := func() { @@ -86,10 +78,5 @@ func TestBadgerDsAddrBook(t *testing.T) { closeDB() } return mgr, closeFunc - }) -} - -func BenchmarkBadgerDsPeerstore(b *testing.B) { - test.BenchmarkPeerstore(b, cachingPeerstore(b), "Caching") - test.BenchmarkPeerstore(b, cachelessPeerstore(b), "Cacheless") + } } From 748d06e8e0bec735122b82378f4f7e5aa4ecb49b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Sep 2018 16:21:02 +0100 Subject: [PATCH 0642/3965] move benchmarks test cases to benchmarks_suite.go. --- p2p/host/peerstore/test/benchmarks_suite.go | 42 +++++++++++++++++++++ p2p/host/peerstore/test/peerstore_suite.go | 42 --------------------- 2 files changed, 42 insertions(+), 42 deletions(-) diff --git a/p2p/host/peerstore/test/benchmarks_suite.go b/p2p/host/peerstore/test/benchmarks_suite.go index cd5f2df733..e112befdf9 100644 --- a/p2p/host/peerstore/test/benchmarks_suite.go +++ b/p2p/host/peerstore/test/benchmarks_suite.go @@ -50,3 +50,45 @@ func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory, variant string) } } } + +func benchmarkAddAddr(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { + return func(b *testing.B) { + b.ResetTimer() + for i := 0; i < b.N; i++ { + pp := <-addrs + ps.AddAddr(pp.ID, pp.Addr[0], pstore.PermanentAddrTTL) + } + } +} + +func benchmarkAddAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { + return func(b *testing.B) { + b.ResetTimer() + for i := 0; i < b.N; i++ { + pp := <-addrs + ps.AddAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + } + } +} + +func benchmarkSetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { + return func(b *testing.B) { + b.ResetTimer() + for i := 0; i < b.N; i++ { + pp := <-addrs + ps.SetAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + } + } +} + +func benchmarkGetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { + return func(b *testing.B) { + pp := <-addrs + ps.SetAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + _ = ps.Addrs(pp.ID) + } + } +} diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index 0a057f0aad..7f3ad58819 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -279,48 +279,6 @@ func testBasicPeerstore(ps pstore.Peerstore) func(t *testing.T) { } } -func benchmarkAddAddr(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { - return func(b *testing.B) { - b.ResetTimer() - for i := 0; i < b.N; i++ { - pp := <-addrs - ps.AddAddr(pp.ID, pp.Addr[0], pstore.PermanentAddrTTL) - } - } -} - -func benchmarkAddAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { - return func(b *testing.B) { - b.ResetTimer() - for i := 0; i < b.N; i++ { - pp := <-addrs - ps.AddAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) - } - } -} - -func benchmarkSetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { - return func(b *testing.B) { - b.ResetTimer() - for i := 0; i < b.N; i++ { - pp := <-addrs - ps.SetAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) - } - } -} - -func benchmarkGetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { - return func(b *testing.B) { - pp := <-addrs - ps.SetAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) - - b.ResetTimer() - for i := 0; i < b.N; i++ { - _ = ps.Addrs(pp.ID) - } - } -} - func getAddrs(t *testing.T, n int) []ma.Multiaddr { var addrs []ma.Multiaddr for i := 0; i < n; i++ { From b60a3ceab32a509cd8c3495e174594d2b88c644b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Sep 2018 18:02:00 +0100 Subject: [PATCH 0643/3965] add benchmark for ClearAddrs. --- p2p/host/peerstore/test/benchmarks_suite.go | 35 ++++++++++++--------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/p2p/host/peerstore/test/benchmarks_suite.go b/p2p/host/peerstore/test/benchmarks_suite.go index e112befdf9..6e1bcf9d75 100644 --- a/p2p/host/peerstore/test/benchmarks_suite.go +++ b/p2p/host/peerstore/test/benchmarks_suite.go @@ -9,10 +9,10 @@ import ( ) var peerstoreBenchmarks = map[string]func(pstore.Peerstore, chan *peerpair) func(*testing.B){ - "AddAddr": benchmarkAddAddr, - "AddAddrs": benchmarkAddAddrs, - "SetAddrs": benchmarkSetAddrs, - "GetAddrs": benchmarkGetAddrs, + "AddAddrs": benchmarkAddAddrs, + "SetAddrs": benchmarkSetAddrs, + "GetAddrs": benchmarkGetAddrs, + "AddAndClearAddrs": benchmarkAddAndClearAddrs, } func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory, variant string) { @@ -51,16 +51,6 @@ func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory, variant string) } } -func benchmarkAddAddr(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { - return func(b *testing.B) { - b.ResetTimer() - for i := 0; i < b.N; i++ { - pp := <-addrs - ps.AddAddr(pp.ID, pp.Addr[0], pstore.PermanentAddrTTL) - } - } -} - func benchmarkAddAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { return func(b *testing.B) { b.ResetTimer() @@ -92,3 +82,20 @@ func benchmarkGetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing. } } } + +func benchmarkAddAndClearAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { + // 1000 peers. + return func(b *testing.B) { + var peers = make([]*peerpair, 1000) + for i := 0; i < len(peers); peers[i], i = <-addrs, i+1 { + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + for _, pp := range peers { + ps.AddAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + ps.ClearAddrs(pp.ID) + } + } + } +} From d20a98cac8b9fd295f68a12496824c8535cbd203 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Sep 2018 18:07:44 +0100 Subject: [PATCH 0644/3965] rename package mem => pstoremem. --- p2p/host/peerstore/pstoreds/addr_book.go | 6 +++--- p2p/host/peerstore/pstoreds/peerstore.go | 4 ++-- p2p/host/peerstore/{mem => pstoremem}/addr_book.go | 2 +- p2p/host/peerstore/{mem => pstoremem}/inmem_test.go | 2 +- p2p/host/peerstore/{mem => pstoremem}/keybook.go | 2 +- p2p/host/peerstore/{mem => pstoremem}/metadata.go | 2 +- p2p/host/peerstore/{mem => pstoremem}/peerstore.go | 2 +- 7 files changed, 10 insertions(+), 10 deletions(-) rename p2p/host/peerstore/{mem => pstoremem}/addr_book.go (99%) rename p2p/host/peerstore/{mem => pstoremem}/inmem_test.go (96%) rename p2p/host/peerstore/{mem => pstoremem}/keybook.go (98%) rename p2p/host/peerstore/{mem => pstoremem}/metadata.go (98%) rename p2p/host/peerstore/{mem => pstoremem}/peerstore.go (93%) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 8aebdd888d..2db922a2ea 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -14,7 +14,7 @@ import ( mh "github.com/multiformats/go-multihash" pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/mem" + "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) var log = logging.Logger("peerstore/ds") @@ -30,7 +30,7 @@ type dsAddrBook struct { cache *lru.ARCCache ds ds.Batching ttlManager *ttlmanager - subsManager *mem.AddrSubManager + subsManager *pstoremem.AddrSubManager } // NewAddrBook initializes a new address manager given a @@ -46,7 +46,7 @@ func NewAddrBook(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) cache: cache, ds: ds, ttlManager: newTTLManager(ctx, ds, cache, ttlInterval), - subsManager: mem.NewAddrSubManager(), + subsManager: pstoremem.NewAddrSubManager(), } return mgr, nil } diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 0ce3926a15..9423df6377 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -7,7 +7,7 @@ import ( "github.com/ipfs/go-datastore" pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/mem" + "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) // NewPeerstore creates a peerstore backed by the provided persistent datastore. @@ -17,6 +17,6 @@ func NewPeerstore(ctx context.Context, ds datastore.Batching) (pstore.Peerstore, return nil, err } - ps := pstore.NewPeerstoreWith(mem.NewKeyBook(), addrBook, mem.NewPeerMetadata()) + ps := pstore.NewPeerstoreWith(pstoremem.NewKeyBook(), addrBook, pstoremem.NewPeerMetadata()) return ps, nil } diff --git a/p2p/host/peerstore/mem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go similarity index 99% rename from p2p/host/peerstore/mem/addr_book.go rename to p2p/host/peerstore/pstoremem/addr_book.go index d0cf980195..3bd1c6bf7b 100644 --- a/p2p/host/peerstore/mem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -1,4 +1,4 @@ -package mem +package pstoremem import ( "context" diff --git a/p2p/host/peerstore/mem/inmem_test.go b/p2p/host/peerstore/pstoremem/inmem_test.go similarity index 96% rename from p2p/host/peerstore/mem/inmem_test.go rename to p2p/host/peerstore/pstoremem/inmem_test.go index 6f9b4a8d58..5e56462e11 100644 --- a/p2p/host/peerstore/mem/inmem_test.go +++ b/p2p/host/peerstore/pstoremem/inmem_test.go @@ -1,4 +1,4 @@ -package mem +package pstoremem import ( "testing" diff --git a/p2p/host/peerstore/mem/keybook.go b/p2p/host/peerstore/pstoremem/keybook.go similarity index 98% rename from p2p/host/peerstore/mem/keybook.go rename to p2p/host/peerstore/pstoremem/keybook.go index 053ffe401a..13d5009e54 100644 --- a/p2p/host/peerstore/mem/keybook.go +++ b/p2p/host/peerstore/pstoremem/keybook.go @@ -1,4 +1,4 @@ -package mem +package pstoremem import ( "errors" diff --git a/p2p/host/peerstore/mem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go similarity index 98% rename from p2p/host/peerstore/mem/metadata.go rename to p2p/host/peerstore/pstoremem/metadata.go index f7c0da7179..b4ba6056ed 100644 --- a/p2p/host/peerstore/mem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -1,4 +1,4 @@ -package mem +package pstoremem import ( "sync" diff --git a/p2p/host/peerstore/mem/peerstore.go b/p2p/host/peerstore/pstoremem/peerstore.go similarity index 93% rename from p2p/host/peerstore/mem/peerstore.go rename to p2p/host/peerstore/pstoremem/peerstore.go index ba4c5a3b79..f8fde4ff00 100644 --- a/p2p/host/peerstore/mem/peerstore.go +++ b/p2p/host/peerstore/pstoremem/peerstore.go @@ -1,4 +1,4 @@ -package mem +package pstoremem import pstore "github.com/libp2p/go-libp2p-peerstore" From 0bc531b8860fdd84518b540c227de96dc77852a2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 13 Jun 2018 11:20:29 -0700 Subject: [PATCH 0645/3965] log an error when we have no transports configured Modify our two transport lookup functions to log loudly when there are no transports registered. --- p2p/net/swarm/swarm_transport.go | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/p2p/net/swarm/swarm_transport.go b/p2p/net/swarm/swarm_transport.go index bc60dcf993..c96a079d9e 100644 --- a/p2p/net/swarm/swarm_transport.go +++ b/p2p/net/swarm/swarm_transport.go @@ -18,6 +18,11 @@ func (s *Swarm) TransportForDialing(a ma.Multiaddr) transport.Transport { s.transports.RLock() defer s.transports.RUnlock() + if len(s.transports.m) == 0 { + log.Error("you have no transports configured") + return nil + } + for _, p := range protocols { transport, ok := s.transports.m[p.Code] if !ok { @@ -41,6 +46,11 @@ func (s *Swarm) TransportForListening(a ma.Multiaddr) transport.Transport { s.transports.RLock() defer s.transports.RUnlock() + if len(s.transports.m) == 0 { + log.Error("you have no transports configured") + return nil + } + selected := s.transports.m[protocols[len(protocols)-1].Code] for _, p := range protocols { transport, ok := s.transports.m[p.Code] From 3138e45b752b428131567c2b2c408a3f26ac525b Mon Sep 17 00:00:00 2001 From: Jeromy Date: Tue, 5 Jun 2018 05:42:50 -0700 Subject: [PATCH 0646/3965] add in hop tracking --- .../internal/circuitv1-deprecated/notify.go | 1 - .../internal/circuitv1-deprecated/relay.go | 46 +++++++++++++++++++ 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/notify.go b/p2p/protocol/internal/circuitv1-deprecated/notify.go index 7ded2b623b..a82ef552b4 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/notify.go +++ b/p2p/protocol/internal/circuitv1-deprecated/notify.go @@ -36,7 +36,6 @@ func (n *RelayNotifiee) Connected(s inet.Network, c inet.Conn) { defer cancel() canhop, err := n.Relay().CanHop(ctx, id) - if err != nil { log.Debugf("Error testing relay hop: %s", err.Error()) return diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 99e89d45ff..6f28f00681 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -40,6 +40,10 @@ type Relay struct { relays map[peer.ID]struct{} mx sync.Mutex + + liveHops map[peer.ID]map[peer.ID]struct{} + lhCount uint64 + lhLk sync.Mutex } type RelayOpt int @@ -65,6 +69,7 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. self: h.ID(), incoming: make(chan *Conn), relays: make(map[peer.ID]struct{}), + liveHops: make(map[peer.ID]map[peer.ID]struct{}), } for _, opt := range opts { @@ -84,6 +89,43 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. return r, nil } +func (r *Relay) addLiveHop(from, to peer.ID) { + r.lhLk.Lock() + defer r.lhLk.Unlock() + + r.lhCount++ + trg, ok := r.liveHops[from] + if !ok { + trg = make(map[peer.ID]struct{}) + r.liveHops[from] = trg + } + + trg[to] = struct{}{} +} + +func (r *Relay) rmLiveHop(from, to peer.ID) { + r.lhLk.Lock() + defer r.lhLk.Unlock() + + r.lhCount-- + trg, ok := r.liveHops[from] + if !ok { + return + } + + delete(trg, to) + if len(trg) == 0 { + delete(r.liveHops, from) + } +} + +func (r *Relay) GetActiveHops() uint64 { + r.lhLk.Lock() + defer r.lhLk.Unlock() + + return r.lhCount +} + func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { log.Debugf("dialing peer %s through relay %s", dest.ID, relay.ID) @@ -299,9 +341,13 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { // relay connection log.Infof("relaying connection between %s and %s", src.ID.Pretty(), dst.ID.Pretty()) + r.addLiveHop(src.ID, dst.ID) + // Don't reset streams after finishing or the other side will get an // error, not an EOF. go func() { + defer r.rmLiveHop(src.ID, dst.ID) + count, err := io.Copy(s, bs) if err != nil { log.Debugf("relay copy error: %s", err) From d43feab8f4b453c5a486fac2c2b3b5a7c80f2ff0 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 4 Sep 2018 18:45:36 -0400 Subject: [PATCH 0647/3965] Only decrement hop counter when actually removing live hop --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 6f28f00681..436061cc17 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -107,12 +107,12 @@ func (r *Relay) rmLiveHop(from, to peer.ID) { r.lhLk.Lock() defer r.lhLk.Unlock() - r.lhCount-- trg, ok := r.liveHops[from] if !ok { return } + r.lhCount-- delete(trg, to) if len(trg) == 0 { delete(r.liveHops, from) From 7f38aaf25424d87e8d8c6fd081b6c3a513465b29 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 5 Sep 2018 15:08:33 +0100 Subject: [PATCH 0648/3965] rename AddrMgr => AddrBook elsewhere. --- p2p/host/peerstore/pstoreds/ds_test.go | 2 +- p2p/host/peerstore/pstoremem/inmem_test.go | 4 ++-- .../test/{addr_manager_suite.go => addr_book_suite.go} | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) rename p2p/host/peerstore/test/{addr_manager_suite.go => addr_book_suite.go} (98%) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 29d6513cee..4ed24d0e9d 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -48,7 +48,7 @@ func TestBadgerDsPeerstore(t *testing.T) { } func TestBadgerDsAddrBook(t *testing.T) { - test.TestAddrMgr(t, func() (peerstore.AddrBook, func()) { + test.TestAddrBook(t, func() (peerstore.AddrBook, func()) { ds, closeDB := setupBadgerDatastore(t) mgr, err := NewAddrBook(context.Background(), ds, 100*time.Microsecond) diff --git a/p2p/host/peerstore/pstoremem/inmem_test.go b/p2p/host/peerstore/pstoremem/inmem_test.go index 5e56462e11..8f5782d41e 100644 --- a/p2p/host/peerstore/pstoremem/inmem_test.go +++ b/p2p/host/peerstore/pstoremem/inmem_test.go @@ -13,8 +13,8 @@ func TestInMemoryPeerstore(t *testing.T) { }) } -func TestInMemoryAddrMgr(t *testing.T) { - test.TestAddrMgr(t, func() (pstore.AddrBook, func()) { +func TestInMemoryAddrBook(t *testing.T) { + test.TestAddrBook(t, func() (pstore.AddrBook, func()) { return NewAddrBook(), nil }) } diff --git a/p2p/host/peerstore/test/addr_manager_suite.go b/p2p/host/peerstore/test/addr_book_suite.go similarity index 98% rename from p2p/host/peerstore/test/addr_manager_suite.go rename to p2p/host/peerstore/test/addr_book_suite.go index acc0165744..f9e52aa7cb 100644 --- a/p2p/host/peerstore/test/addr_manager_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -19,9 +19,9 @@ var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ "AddressesExpire": testAddressesExpire, } -type AddrMgrFactory func() (pstore.AddrBook, func()) +type AddrBookFactory func() (pstore.AddrBook, func()) -func TestAddrMgr(t *testing.T, factory AddrMgrFactory) { +func TestAddrBook(t *testing.T, factory AddrBookFactory) { for name, test := range addressBookSuite { // Create a new peerstore. ab, closeFunc := factory() From 6959b235bf581fdd5e607e7c6b4af08c314c66bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 5 Sep 2018 15:14:40 +0100 Subject: [PATCH 0649/3965] rename NewPeerstoreWith(...) => NewPeerstore(...); adjust godoc. --- p2p/host/peerstore/peerstore.go | 7 ++++--- p2p/host/peerstore/pstoreds/addr_book.go | 4 ++-- p2p/host/peerstore/pstoreds/peerstore.go | 2 +- p2p/host/peerstore/pstoremem/peerstore.go | 2 +- 4 files changed, 8 insertions(+), 7 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 047926d2c6..c09743bfb9 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -20,12 +20,13 @@ type peerstore struct { protolock sync.Mutex } -// NewPeerstore creates a threadsafe collection of peers. -func NewPeerstoreWith(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { +// NewPeerstore creates a data structure that stores peer data, backed by the +// supplied implementations of KeyBook, AddrBook and PeerMetadata. +func NewPeerstore(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { return &peerstore{ KeyBook: kb, - PeerMetadata: md, AddrBook: ab, + PeerMetadata: md, Metrics: NewMetrics(), } } diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 2db922a2ea..90f147f5bd 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -24,7 +24,7 @@ var dsWriteRetries = 5 var _ pstore.AddrBook = (*dsAddrBook)(nil) -// dsAddrBook is an address manager backed by a Datastore with both an +// dsAddrBook is an address book backed by a Datastore with both an // in-memory TTL manager and an in-memory address stream manager. type dsAddrBook struct { cache *lru.ARCCache @@ -33,7 +33,7 @@ type dsAddrBook struct { subsManager *pstoremem.AddrSubManager } -// NewAddrBook initializes a new address manager given a +// NewAddrBook initializes a new address book given a // Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. func NewAddrBook(ctx context.Context, ds ds.Batching, ttlInterval time.Duration) (*dsAddrBook, error) { diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 9423df6377..e973988bd5 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -17,6 +17,6 @@ func NewPeerstore(ctx context.Context, ds datastore.Batching) (pstore.Peerstore, return nil, err } - ps := pstore.NewPeerstoreWith(pstoremem.NewKeyBook(), addrBook, pstoremem.NewPeerMetadata()) + ps := pstore.NewPeerstore(pstoremem.NewKeyBook(), addrBook, pstoremem.NewPeerMetadata()) return ps, nil } diff --git a/p2p/host/peerstore/pstoremem/peerstore.go b/p2p/host/peerstore/pstoremem/peerstore.go index f8fde4ff00..7d87313de1 100644 --- a/p2p/host/peerstore/pstoremem/peerstore.go +++ b/p2p/host/peerstore/pstoremem/peerstore.go @@ -4,7 +4,7 @@ import pstore "github.com/libp2p/go-libp2p-peerstore" // NewPeerstore creates an in-memory threadsafe collection of peers. func NewPeerstore() pstore.Peerstore { - return pstore.NewPeerstoreWith( + return pstore.NewPeerstore( NewKeyBook(), NewAddrBook(), NewPeerMetadata()) From d410f091b465b6df171bcc174d405a13d1bfa8f9 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 5 Sep 2018 15:22:52 -0400 Subject: [PATCH 0650/3965] Fix counting logic to avoid double counting --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 436061cc17..51dc1c10a5 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -93,13 +93,16 @@ func (r *Relay) addLiveHop(from, to peer.ID) { r.lhLk.Lock() defer r.lhLk.Unlock() - r.lhCount++ trg, ok := r.liveHops[from] if !ok { trg = make(map[peer.ID]struct{}) r.liveHops[from] = trg } + if _, ok := trg[to]; ok { + return + } + r.lhCount++ trg[to] = struct{}{} } @@ -111,6 +114,9 @@ func (r *Relay) rmLiveHop(from, to peer.ID) { if !ok { return } + if _, ok := trg[to]; !ok { + return + } r.lhCount-- delete(trg, to) From 2818d6589b785695576936003afefdcbc91c6fef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 6 Sep 2018 20:18:25 +0100 Subject: [PATCH 0651/3965] increase wait times in tests. --- p2p/host/peerstore/test/addr_book_suite.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index f9e52aa7cb..91326d4a59 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -213,27 +213,27 @@ func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) m.SetAddr(id1, ma11, time.Millisecond) - <-time.After(time.Millisecond * 2) + <-time.After(time.Millisecond * 5) testHas(t, []ma.Multiaddr{ma12, ma13}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) m.SetAddr(id1, ma13, time.Millisecond) - <-time.After(time.Millisecond * 2) + <-time.After(time.Millisecond * 5) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) m.SetAddr(id2, ma24, time.Millisecond) - <-time.After(time.Millisecond * 2) + <-time.After(time.Millisecond * 5) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, []ma.Multiaddr{ma25}, m.Addrs(id2)) m.SetAddr(id2, ma25, time.Millisecond) - <-time.After(time.Millisecond * 2) + <-time.After(time.Millisecond * 5) testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) testHas(t, nil, m.Addrs(id2)) m.SetAddr(id1, ma12, time.Millisecond) - <-time.After(time.Millisecond * 2) + <-time.After(time.Millisecond * 5) testHas(t, nil, m.Addrs(id1)) testHas(t, nil, m.Addrs(id2)) } From ef51fdbe09ccc47e7b64ef29515b2c1c43a31fdb Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Fri, 7 Sep 2018 16:06:16 +0800 Subject: [PATCH 0652/3965] Deduplicate addresses of host output --- p2p/host/basic/basic_host.go | 37 +++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 2deb016f1f..0265819e48 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -469,26 +469,41 @@ func (h *BasicHost) Addrs() []ma.Multiaddr { return h.addrs(h.AllAddrs()) } +// mergeAddrs merges input address lists, leave only unique addresses +func mergeAddrs(addrLists ...[]ma.Multiaddr) (uniqueAddrs []ma.Multiaddr) { + exists := make(map[string]bool) + for _, addrList := range addrLists { + for _, addr := range addrList { + if exists[addr.String()] { + continue + } + exists[addr.String()] = true + uniqueAddrs = append(uniqueAddrs, addr) + } + } + return uniqueAddrs +} + // AllAddrs returns all the addresses of BasicHost at this moment in time. // It's ok to not include addresses if they're not available to be used now. func (h *BasicHost) AllAddrs() []ma.Multiaddr { - addrs, err := h.Network().InterfaceListenAddresses() + listenAddrs, err := h.Network().InterfaceListenAddresses() if err != nil { log.Debug("error retrieving network interface addrs") } - - if h.ids != nil { // add external observed addresses - addrs = append(addrs, h.ids.OwnObservedAddrs()...) + var observedAddrs []ma.Multiaddr + if h.ids != nil { + // peer observed addresses + observedAddrs = h.ids.OwnObservedAddrs() } - - if h.natmgr != nil { // natmgr is nil if we do not use nat option. - nat := h.natmgr.NAT() - if nat != nil { // nat is nil if not ready, or no nat is available. - addrs = append(addrs, nat.ExternalAddrs()...) - } + var natAddrs []ma.Multiaddr + // natmgr is nil if we do not use nat option; + // h.natmgr.NAT() is nil if not ready, or no nat is available. + if h.natmgr != nil && h.natmgr.NAT() != nil { + natAddrs = h.natmgr.NAT().ExternalAddrs() } - return addrs + return mergeAddrs(listenAddrs, observedAddrs, natAddrs) } // Close shuts down the Host's services (network, etc). From 65adc76e657859f2b19c53a86bb9f68b4efdbdcd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 7 Sep 2018 12:43:27 +0100 Subject: [PATCH 0653/3965] add mafmt import alias. --- p2p/host/peerstore/addr/sorting.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index b42b0410e3..f8a89150a2 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -5,7 +5,7 @@ import ( ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" - "github.com/whyrusleeping/mafmt" + mafmt "github.com/whyrusleeping/mafmt" ) func isFDCostlyTransport(a ma.Multiaddr) bool { From 888f6317eb109d59a94561f4523d0a74be5f58b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 31 Aug 2018 12:59:46 +0100 Subject: [PATCH 0654/3965] add tests for keybook; []peer.ID => peer.IDSlice. --- p2p/host/peerstore/interface.go | 6 +- p2p/host/peerstore/peerstore.go | 10 +- p2p/host/peerstore/pstoreds/addr_book.go | 6 +- p2p/host/peerstore/pstoremem/addr_book.go | 4 +- p2p/host/peerstore/pstoremem/inmem_test.go | 6 + p2p/host/peerstore/pstoremem/keybook.go | 4 +- p2p/host/peerstore/test/addr_book_suite.go | 110 ++++++--------- p2p/host/peerstore/test/keybook_suite.go | 154 +++++++++++++++++++++ p2p/host/peerstore/test/test_utils.go | 40 ++---- 9 files changed, 235 insertions(+), 105 deletions(-) create mode 100644 p2p/host/peerstore/test/keybook_suite.go diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 468a8a12a9..94b18aa632 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -66,7 +66,7 @@ type Peerstore interface { SupportsProtocols(peer.ID, ...string) ([]string, error) // Peers returns all of the peer IDs stored across all inner stores. - Peers() []peer.ID + Peers() peer.IDSlice } type PeerMetadata interface { @@ -111,7 +111,7 @@ type AddrBook interface { ClearAddrs(p peer.ID) // PeersWithAddrs returns all of the peer IDs stored in the AddrBook - PeersWithAddrs() []peer.ID + PeersWithAddrs() peer.IDSlice } // KeyBook tracks the keys of Peers. @@ -130,5 +130,5 @@ type KeyBook interface { AddPrivKey(peer.ID, ic.PrivKey) error // PeersWithKeys returns all the peer IDs stored in the KeyBook - PeersWithKeys() []peer.ID + PeersWithKeys() peer.IDSlice } diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index c09743bfb9..d25eb4344e 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -31,7 +31,7 @@ func NewPeerstore(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { } } -func (ps *peerstore) Peers() []peer.ID { +func (ps *peerstore) Peers() peer.IDSlice { set := map[peer.ID]struct{}{} for _, p := range ps.PeersWithKeys() { set[p] = struct{}{} @@ -40,7 +40,7 @@ func (ps *peerstore) Peers() []peer.ID { set[p] = struct{}{} } - pps := make([]peer.ID, 0, len(set)) + pps := make(peer.IDSlice, 0, len(set)) for p := range set { pps = append(pps, p) } @@ -132,7 +132,7 @@ func (ps *peerstore) SupportsProtocols(p peer.ID, protos ...string) ([]string, e return out, nil } -func PeerInfos(ps Peerstore, peers []peer.ID) []PeerInfo { +func PeerInfos(ps Peerstore, peers peer.IDSlice) []PeerInfo { pi := make([]PeerInfo, len(peers)) for i, p := range peers { pi[i] = ps.PeerInfo(p) @@ -140,8 +140,8 @@ func PeerInfos(ps Peerstore, peers []peer.ID) []PeerInfo { return pi } -func PeerInfoIDs(pis []PeerInfo) []peer.ID { - ps := make([]peer.ID, len(pis)) +func PeerInfoIDs(pis []PeerInfo) peer.IDSlice { + ps := make(peer.IDSlice, len(pis)) for i, pi := range pis { ps[i] = pi.ID } diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 90f147f5bd..5bb8b5e479 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -193,12 +193,12 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { } // Peers returns all of the peer IDs for which the AddrBook has addresses. -func (mgr *dsAddrBook) PeersWithAddrs() []peer.ID { +func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { q := query.Query{KeysOnly: true} results, err := mgr.ds.Query(q) if err != nil { log.Error(err) - return []peer.ID{} + return peer.IDSlice{} } idset := make(map[peer.ID]struct{}) @@ -211,7 +211,7 @@ func (mgr *dsAddrBook) PeersWithAddrs() []peer.ID { idset[id] = struct{}{} } - ids := make([]peer.ID, 0, len(idset)) + ids := make(peer.IDSlice, 0, len(idset)) for id := range idset { ids = append(ids, id) } diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 3bd1c6bf7b..ebd2de1a46 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -45,14 +45,14 @@ func NewAddrBook() pstore.AddrBook { } } -func (mab *memoryAddrBook) PeersWithAddrs() []peer.ID { +func (mab *memoryAddrBook) PeersWithAddrs() peer.IDSlice { mab.addrmu.Lock() defer mab.addrmu.Unlock() if mab.addrs == nil { return nil } - pids := make([]peer.ID, 0, len(mab.addrs)) + pids := make(peer.IDSlice, 0, len(mab.addrs)) for pid := range mab.addrs { pids = append(pids, pid) } diff --git a/p2p/host/peerstore/pstoremem/inmem_test.go b/p2p/host/peerstore/pstoremem/inmem_test.go index 8f5782d41e..e13135ea34 100644 --- a/p2p/host/peerstore/pstoremem/inmem_test.go +++ b/p2p/host/peerstore/pstoremem/inmem_test.go @@ -19,6 +19,12 @@ func TestInMemoryAddrBook(t *testing.T) { }) } +func TestInMemoryKeyBook(t *testing.T) { + test.TestKeyBook(t, func() (pstore.KeyBook, func()) { + return NewKeyBook(), nil + }) +} + func BenchmarkInMemoryPeerstore(b *testing.B) { test.BenchmarkPeerstore(b, func() (pstore.Peerstore, func()) { return NewPeerstore(), nil diff --git a/p2p/host/peerstore/pstoremem/keybook.go b/p2p/host/peerstore/pstoremem/keybook.go index 13d5009e54..878cde9915 100644 --- a/p2p/host/peerstore/pstoremem/keybook.go +++ b/p2p/host/peerstore/pstoremem/keybook.go @@ -26,9 +26,9 @@ func NewKeyBook() pstore.KeyBook { } } -func (mkb *memoryKeyBook) PeersWithKeys() []peer.ID { +func (mkb *memoryKeyBook) PeersWithKeys() peer.IDSlice { mkb.RLock() - ps := make([]peer.ID, 0, len(mkb.pks)+len(mkb.sks)) + ps := make(peer.IDSlice, 0, len(mkb.pks)+len(mkb.sks)) for p := range mkb.pks { ps = append(ps, p) } diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index 91326d4a59..329a1804c7 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -4,10 +4,8 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" - pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" ) var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ @@ -38,27 +36,27 @@ func TestAddrBook(t *testing.T, factory AddrBookFactory) { func testAddresses(m pstore.AddrBook) func(*testing.T) { return func(t *testing.T) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := IDS(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") - id3 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ6Kn") - id4 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Kn") - id5 := IDS(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Km") - - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - ma21 := MA(t, "/ip4/2.2.3.2/tcp/1111") - ma22 := MA(t, "/ip4/2.2.3.2/tcp/2222") - ma31 := MA(t, "/ip4/3.2.3.3/tcp/1111") - ma32 := MA(t, "/ip4/3.2.3.3/tcp/2222") - ma33 := MA(t, "/ip4/3.2.3.3/tcp/3333") - ma41 := MA(t, "/ip4/4.2.3.3/tcp/1111") - ma42 := MA(t, "/ip4/4.2.3.3/tcp/2222") - ma43 := MA(t, "/ip4/4.2.3.3/tcp/3333") - ma44 := MA(t, "/ip4/4.2.3.3/tcp/4444") - ma51 := MA(t, "/ip4/5.2.3.3/tcp/1111") - ma52 := MA(t, "/ip4/5.2.3.3/tcp/2222") - ma53 := MA(t, "/ip4/5.2.3.3/tcp/3333") - ma54 := MA(t, "/ip4/5.2.3.3/tcp/4444") - ma55 := MA(t, "/ip4/5.2.3.3/tcp/5555") + id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := peerId(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") + id3 := peerId(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ6Kn") + id4 := peerId(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Kn") + id5 := peerId(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Km") + + ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") + ma21 := multiaddr(t, "/ip4/2.2.3.2/tcp/1111") + ma22 := multiaddr(t, "/ip4/2.2.3.2/tcp/2222") + ma31 := multiaddr(t, "/ip4/3.2.3.3/tcp/1111") + ma32 := multiaddr(t, "/ip4/3.2.3.3/tcp/2222") + ma33 := multiaddr(t, "/ip4/3.2.3.3/tcp/3333") + ma41 := multiaddr(t, "/ip4/4.2.3.3/tcp/1111") + ma42 := multiaddr(t, "/ip4/4.2.3.3/tcp/2222") + ma43 := multiaddr(t, "/ip4/4.2.3.3/tcp/3333") + ma44 := multiaddr(t, "/ip4/4.2.3.3/tcp/4444") + ma51 := multiaddr(t, "/ip4/5.2.3.3/tcp/1111") + ma52 := multiaddr(t, "/ip4/5.2.3.3/tcp/2222") + ma53 := multiaddr(t, "/ip4/5.2.3.3/tcp/3333") + ma54 := multiaddr(t, "/ip4/5.2.3.3/tcp/4444") + ma55 := multiaddr(t, "/ip4/5.2.3.3/tcp/5555") ttl := time.Hour m.AddAddr(id1, ma11, ttl) @@ -90,13 +88,13 @@ func testAddresses(m pstore.AddrBook) func(*testing.T) { func testClearWorks(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := MA(t, "/ip4/2.2.3.2/tcp/2222") - ma13 := MA(t, "/ip4/3.2.3.3/tcp/3333") - ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") - ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") + id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := multiaddr(t, "/ip4/2.2.3.2/tcp/2222") + ma13 := multiaddr(t, "/ip4/3.2.3.3/tcp/3333") + ma24 := multiaddr(t, "/ip4/4.2.3.3/tcp/4444") + ma25 := multiaddr(t, "/ip4/5.2.3.3/tcp/5555") m.AddAddr(id1, ma11, time.Hour) m.AddAddr(id1, ma12, time.Hour) @@ -117,8 +115,8 @@ func testClearWorks(m pstore.AddrBook) func(t *testing.T) { func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") + id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") m.SetAddr(id1, ma11, time.Hour) @@ -132,12 +130,12 @@ func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := MA(t, "/ip4/1.2.3.1/tcp/1112") - ma21 := MA(t, "/ip4/1.2.3.1/tcp/1121") - ma22 := MA(t, "/ip4/1.2.3.1/tcp/1122") + id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := multiaddr(t, "/ip4/1.2.3.1/tcp/1112") + ma21 := multiaddr(t, "/ip4/1.2.3.1/tcp/1121") + ma22 := multiaddr(t, "/ip4/1.2.3.1/tcp/1122") // Shouldn't panic. m.UpdateAddrs(id1, time.Hour, time.Minute) @@ -178,7 +176,7 @@ func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { func testNilAddrsDontBreak(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") m.SetAddr(id1, nil, time.Hour) m.AddAddr(id1, nil, time.Hour) } @@ -186,13 +184,13 @@ func testNilAddrsDontBreak(m pstore.AddrBook) func(t *testing.T) { func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := IDS(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := MA(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := MA(t, "/ip4/2.2.3.2/tcp/2222") - ma13 := MA(t, "/ip4/3.2.3.3/tcp/3333") - ma24 := MA(t, "/ip4/4.2.3.3/tcp/4444") - ma25 := MA(t, "/ip4/5.2.3.3/tcp/5555") + id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") + id2 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") + ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") + ma12 := multiaddr(t, "/ip4/2.2.3.2/tcp/2222") + ma13 := multiaddr(t, "/ip4/3.2.3.3/tcp/3333") + ma24 := multiaddr(t, "/ip4/4.2.3.3/tcp/4444") + ma25 := multiaddr(t, "/ip4/5.2.3.3/tcp/5555") m.AddAddr(id1, ma11, time.Hour) m.AddAddr(id1, ma12, time.Hour) @@ -239,24 +237,6 @@ func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { } } -func IDS(t *testing.T, ids string) peer.ID { - t.Helper() - id, err := peer.IDB58Decode(ids) - if err != nil { - t.Fatalf("id %q is bad: %s", ids, err) - } - return id -} - -func MA(t *testing.T, m string) ma.Multiaddr { - t.Helper() - maddr, err := ma.NewMultiaddr(m) - if err != nil { - t.Fatal(err) - } - return maddr -} - func testHas(t *testing.T, exp, act []ma.Multiaddr) { t.Helper() if len(exp) != len(act) { diff --git a/p2p/host/peerstore/test/keybook_suite.go b/p2p/host/peerstore/test/keybook_suite.go new file mode 100644 index 0000000000..c48a6eedfd --- /dev/null +++ b/p2p/host/peerstore/test/keybook_suite.go @@ -0,0 +1,154 @@ +package test + +import ( + "sort" + "testing" + + "github.com/libp2p/go-libp2p-peer" + pt "github.com/libp2p/go-libp2p-peer/test" + + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +var keyBookSuite = map[string]func(kb pstore.KeyBook) func(*testing.T){ + "AddGetPrivKey": testKeybookPrivKey, + "AddGetPubKey": testKeyBookPubKey, + "PeersWithKeys": testKeyBookPeers, + "PubKeyAddedOnRetrieve": testInlinedPubKeyAddedOnRetrieve, +} + +type KeyBookFactory func() (pstore.KeyBook, func()) + +func TestKeyBook(t *testing.T, factory KeyBookFactory) { + for name, test := range keyBookSuite { + // Create a new peerstore. + kb, closeFunc := factory() + + // Run the test. + t.Run(name, test(kb)) + + // Cleanup. + if closeFunc != nil { + closeFunc() + } + } +} + +func testKeybookPrivKey(kb pstore.KeyBook) func(t *testing.T) { + return func(t *testing.T) { + if peers := kb.PeersWithKeys(); len(peers) > 0 { + t.Error("expected peers to be empty on init") + } + + priv, _, err := pt.RandTestKeyPair(512) + if err != nil { + t.Error(err) + } + + id, err := peer.IDFromPrivateKey(priv) + if err != nil { + t.Error(err) + } + + err = kb.AddPrivKey(id, priv) + if err != nil { + t.Error(err) + } + + if res := kb.PrivKey(id); !priv.Equals(res) { + t.Error("retrieved private key did not match stored private key") + } + + if peers := kb.PeersWithKeys(); len(peers) != 1 || peers[0] != id { + t.Error("list of peers did not include test peer") + } + } +} + +func testKeyBookPubKey(kb pstore.KeyBook) func(t *testing.T) { + return func(t *testing.T) { + if peers := kb.PeersWithKeys(); len(peers) > 0 { + t.Error("expected peers to be empty on init") + } + + _, pub, err := pt.RandTestKeyPair(512) + if err != nil { + t.Error(err) + } + + id, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Error(err) + } + + err = kb.AddPubKey(id, pub) + if err != nil { + t.Error(err) + } + + if res := kb.PubKey(id); !pub.Equals(res) { + t.Error("retrieved public key did not match stored public key") + } + + if peers := kb.PeersWithKeys(); len(peers) != 1 || peers[0] != id { + t.Error("list of peers did not include test peer") + } + } +} + +func testKeyBookPeers(kb pstore.KeyBook) func(t *testing.T) { + return func(t *testing.T) { + if peers := kb.PeersWithKeys(); len(peers) > 0 { + t.Error("expected peers to be empty on init") + } + + var peers peer.IDSlice + for i := 0; i < 10; i++ { + // Add a public key. + _, pub, _ := pt.RandTestKeyPair(512) + p1, _ := peer.IDFromPublicKey(pub) + kb.AddPubKey(p1, pub) + + // Add a private key. + priv, _, _ := pt.RandTestKeyPair(512) + p2, _ := peer.IDFromPrivateKey(priv) + kb.AddPrivKey(p2, priv) + + peers = append(peers, []peer.ID{p1, p2}...) + } + + kbPeers := kb.PeersWithKeys() + sort.Sort(kbPeers) + sort.Sort(peers) + + for i, p := range kbPeers { + if p != peers[i] { + t.Errorf("mismatch of peer at index %d", i) + } + } + } +} + +func testInlinedPubKeyAddedOnRetrieve(kb pstore.KeyBook) func(t *testing.T) { + return func(t *testing.T) { + if peers := kb.PeersWithKeys(); len(peers) > 0 { + t.Error("expected peers to be empty on init") + } + + // Key small enough for inlining. + _, pub, err := pt.RandTestKeyPair(32) + if err != nil { + t.Error(err) + } + + id, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Error(err) + } + + pubKey := kb.PubKey(id) + if !pubKey.Equals(pub) { + t.Error("mismatch between original public key and keybook-calculated one") + } + } +} diff --git a/p2p/host/peerstore/test/test_utils.go b/p2p/host/peerstore/test/test_utils.go index fd66d73da9..0b308a78e0 100644 --- a/p2p/host/peerstore/test/test_utils.go +++ b/p2p/host/peerstore/test/test_utils.go @@ -1,36 +1,26 @@ package test import ( - "io" - "math/rand" - "time" + "testing" - ci "github.com/libp2p/go-libp2p-crypto" "github.com/libp2p/go-libp2p-peer" - mh "github.com/multiformats/go-multihash" + ma "github.com/multiformats/go-multiaddr" ) -func timeSeededRand() io.Reader { - return rand.New(rand.NewSource(time.Now().UnixNano())) -} - -func RandPeerID() (peer.ID, error) { - buf := make([]byte, 16) - if _, err := io.ReadFull(timeSeededRand(), buf); err != nil { - return "", err - } - h, err := mh.Sum(buf, mh.SHA2_256, -1) +func peerId(t *testing.T, ids string) peer.ID { + t.Helper() + id, err := peer.IDB58Decode(ids) if err != nil { - return "", err + t.Fatalf("id %q is bad: %s", ids, err) } - - return peer.ID(h), nil + return id } -func RandTestKeyPair(bits int) (ci.PrivKey, ci.PubKey, error) { - return ci.GenerateKeyPairWithReader(ci.RSA, bits, timeSeededRand()) -} - -func SeededTestKeyPair(seed int64) (ci.PrivKey, ci.PubKey, error) { - return ci.GenerateKeyPairWithReader(ci.RSA, 512, rand.New(rand.NewSource(seed))) -} +func multiaddr(t *testing.T, m string) ma.Multiaddr { + t.Helper() + maddr, err := ma.NewMultiaddr(m) + if err != nil { + t.Fatal(err) + } + return maddr +} \ No newline at end of file From 92298708b8925bbe0ec902f3a301091070526161 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 31 Aug 2018 18:49:07 +0100 Subject: [PATCH 0655/3965] optimise benchmark code. --- p2p/host/peerstore/test/benchmark_utils.go | 30 +++++++--------------- p2p/host/peerstore/test/peerstore_suite.go | 3 +-- p2p/host/peerstore/test/test_utils.go | 2 +- 3 files changed, 11 insertions(+), 24 deletions(-) diff --git a/p2p/host/peerstore/test/benchmark_utils.go b/p2p/host/peerstore/test/benchmark_utils.go index b6d096cd5c..714bf0d3e7 100644 --- a/p2p/host/peerstore/test/benchmark_utils.go +++ b/p2p/host/peerstore/test/benchmark_utils.go @@ -2,45 +2,33 @@ package test import ( "context" - cr "crypto/rand" "fmt" "testing" - "github.com/mr-tron/base58/base58" + "github.com/libp2p/go-libp2p-peer" + pt "github.com/libp2p/go-libp2p-peer/test" ma "github.com/multiformats/go-multiaddr" - mh "github.com/multiformats/go-multihash" ) type peerpair struct { - ID string + ID peer.ID Addr ma.Multiaddr } func randomPeer(b *testing.B) *peerpair { - buf := make([]byte, 50) + var pid peer.ID + var err error + var addr ma.Multiaddr - for { - n, err := cr.Read(buf) - if err != nil { - b.Fatal(err) - } - if n > 0 { - break - } - } - - id, err := mh.Encode(buf, mh.SHA2_256) - if err != nil { + if pid, err = pt.RandPeerID(); err != nil { b.Fatal(err) } - b58ID := base58.Encode(id) - addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/6666/ipfs/%s", b58ID)) - if err != nil { + if addr, err = ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/6666/ipfs/%s", pid.Pretty())); err != nil { b.Fatal(err) } - return &peerpair{b58ID, addr} + return &peerpair{pid, addr} } func addressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair) { diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index 5317a4964f..eac6aad119 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -299,8 +299,7 @@ func benchmarkPeerstore(ps pstore.Peerstore) func(*testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { pp := <-addrs - pid := peer.ID(pp.ID) - ps.AddAddr(pid, pp.Addr, pstore.PermanentAddrTTL) + ps.AddAddr(pp.ID, pp.Addr, pstore.PermanentAddrTTL) } cancel() } diff --git a/p2p/host/peerstore/test/test_utils.go b/p2p/host/peerstore/test/test_utils.go index 0b308a78e0..d94f640dca 100644 --- a/p2p/host/peerstore/test/test_utils.go +++ b/p2p/host/peerstore/test/test_utils.go @@ -23,4 +23,4 @@ func multiaddr(t *testing.T, m string) ma.Multiaddr { t.Fatal(err) } return maddr -} \ No newline at end of file +} From cd0cdec8b990245b0922b6870c6e554a13d19308 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 7 Sep 2018 16:49:19 +0100 Subject: [PATCH 0656/3965] nested test cases in addr_book suite. --- p2p/host/peerstore/test/addr_book_suite.go | 303 ++++++++++----------- p2p/host/peerstore/test/test_utils.go | 12 +- 2 files changed, 154 insertions(+), 161 deletions(-) diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index 329a1804c7..441e2fa77f 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -1,15 +1,18 @@ package test import ( + "fmt" "testing" "time" + peer "github.com/libp2p/go-libp2p-peer" + pt "github.com/libp2p/go-libp2p-peer/test" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ - "Addresses": testAddresses, + "AddAddress": testAddAddress, "Clear": testClearWorks, "SetNegativeTTLClears": testSetNegativeTTLClears, "UpdateTTLs": testUpdateTTLs, @@ -34,206 +37,200 @@ func TestAddrBook(t *testing.T, factory AddrBookFactory) { } } -func testAddresses(m pstore.AddrBook) func(*testing.T) { - return func(t *testing.T) { - id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := peerId(t, "QmRmPL3FDZKE3Qiwv1RosLdwdvbvg17b2hB39QPScgWKKZ") - id3 := peerId(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ6Kn") - id4 := peerId(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Kn") - id5 := peerId(t, "QmPhi7vBsChP7sjRoZGgg7bcKqF6MmCcQwvRbDte8aJ5Km") - - ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") - ma21 := multiaddr(t, "/ip4/2.2.3.2/tcp/1111") - ma22 := multiaddr(t, "/ip4/2.2.3.2/tcp/2222") - ma31 := multiaddr(t, "/ip4/3.2.3.3/tcp/1111") - ma32 := multiaddr(t, "/ip4/3.2.3.3/tcp/2222") - ma33 := multiaddr(t, "/ip4/3.2.3.3/tcp/3333") - ma41 := multiaddr(t, "/ip4/4.2.3.3/tcp/1111") - ma42 := multiaddr(t, "/ip4/4.2.3.3/tcp/2222") - ma43 := multiaddr(t, "/ip4/4.2.3.3/tcp/3333") - ma44 := multiaddr(t, "/ip4/4.2.3.3/tcp/4444") - ma51 := multiaddr(t, "/ip4/5.2.3.3/tcp/1111") - ma52 := multiaddr(t, "/ip4/5.2.3.3/tcp/2222") - ma53 := multiaddr(t, "/ip4/5.2.3.3/tcp/3333") - ma54 := multiaddr(t, "/ip4/5.2.3.3/tcp/4444") - ma55 := multiaddr(t, "/ip4/5.2.3.3/tcp/5555") - - ttl := time.Hour - m.AddAddr(id1, ma11, ttl) - - m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) - m.AddAddrs(id2, []ma.Multiaddr{ma21, ma22}, ttl) // idempotency - - m.AddAddr(id3, ma31, ttl) - m.AddAddr(id3, ma32, ttl) - m.AddAddr(id3, ma33, ttl) - m.AddAddr(id3, ma33, ttl) // idempotency - m.AddAddr(id3, ma33, ttl) - - m.AddAddrs(id4, []ma.Multiaddr{ma41, ma42, ma43, ma44}, ttl) // multiple - - m.AddAddrs(id5, []ma.Multiaddr{ma21, ma22}, ttl) // clearing - m.AddAddrs(id5, []ma.Multiaddr{ma41, ma42, ma43, ma44}, ttl) // clearing - m.ClearAddrs(id5) - m.AddAddrs(id5, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, ttl) // clearing - - // test the Addresses return value - testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) - testHas(t, []ma.Multiaddr{ma31, ma32, ma33}, m.Addrs(id3)) - testHas(t, []ma.Multiaddr{ma41, ma42, ma43, ma44}, m.Addrs(id4)) - testHas(t, []ma.Multiaddr{ma51, ma52, ma53, ma54, ma55}, m.Addrs(id5)) +func generateAddrs(count int) []ma.Multiaddr { + var addrs = make([]ma.Multiaddr, count) + for i := 0; i < count; i++ { + addrs[i] = multiaddr(fmt.Sprintf("/ip4/1.1.1.%d/tcp/1111", i)) + } + return addrs +} + +func generatePeerIds(count int) []peer.ID { + var ids = make([]peer.ID, count) + for i := 0; i < count; i++ { + ids[i], _ = pt.RandPeerID() } + return ids } -func testClearWorks(m pstore.AddrBook) func(t *testing.T) { +func testAddAddress(ab pstore.AddrBook) func(*testing.T) { return func(t *testing.T) { - id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := multiaddr(t, "/ip4/2.2.3.2/tcp/2222") - ma13 := multiaddr(t, "/ip4/3.2.3.3/tcp/3333") - ma24 := multiaddr(t, "/ip4/4.2.3.3/tcp/4444") - ma25 := multiaddr(t, "/ip4/5.2.3.3/tcp/5555") - - m.AddAddr(id1, ma11, time.Hour) - m.AddAddr(id1, ma12, time.Hour) - m.AddAddr(id1, ma13, time.Hour) - m.AddAddr(id2, ma24, time.Hour) - m.AddAddr(id2, ma25, time.Hour) - - testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) - - m.ClearAddrs(id1) - m.ClearAddrs(id2) - - testHas(t, nil, m.Addrs(id1)) - testHas(t, nil, m.Addrs(id2)) + t.Run("add a single address", func(t *testing.T) { + id := generatePeerIds(1)[0] + addrs := generateAddrs(1) + + ab.AddAddr(id, addrs[0], time.Hour) + + testHas(t, addrs, ab.Addrs(id)) + }) + + t.Run("idempotent add single address", func(t *testing.T) { + id := generatePeerIds(1)[0] + addrs := generateAddrs(1) + + ab.AddAddr(id, addrs[0], time.Hour) + ab.AddAddr(id, addrs[0], time.Hour) + + testHas(t, addrs, ab.Addrs(id)) + }) + + t.Run("add multiple addresses", func(t *testing.T) { + id := generatePeerIds(1)[0] + addrs := generateAddrs(3) + + ab.AddAddrs(id, addrs, time.Hour) + testHas(t, addrs, ab.Addrs(id)) + }) + + t.Run("idempotent add multiple addresses", func(t *testing.T) { + id := generatePeerIds(1)[0] + addrs := generateAddrs(3) + + ab.AddAddrs(id, addrs, time.Hour) + ab.AddAddrs(id, addrs, time.Hour) + + testHas(t, addrs, ab.Addrs(id)) + }) } } -func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { +func testClearWorks(ab pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") + ids := generatePeerIds(2) + addrs := generateAddrs(5) - m.SetAddr(id1, ma11, time.Hour) + ab.AddAddrs(ids[0], addrs[0:3], time.Hour) + ab.AddAddrs(ids[1], addrs[3:], time.Hour) - testHas(t, []ma.Multiaddr{ma11}, m.Addrs(id1)) + testHas(t, addrs[0:3], ab.Addrs(ids[0])) + testHas(t, addrs[3:], ab.Addrs(ids[1])) - m.SetAddr(id1, ma11, -1) + ab.ClearAddrs(ids[0]) + testHas(t, nil, ab.Addrs(ids[0])) + testHas(t, addrs[3:], ab.Addrs(ids[1])) - testHas(t, nil, m.Addrs(id1)) + ab.ClearAddrs(ids[1]) + testHas(t, nil, ab.Addrs(ids[0])) + testHas(t, nil, ab.Addrs(ids[1])) } } -func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { +func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := multiaddr(t, "/ip4/1.2.3.1/tcp/1112") - ma21 := multiaddr(t, "/ip4/1.2.3.1/tcp/1121") - ma22 := multiaddr(t, "/ip4/1.2.3.1/tcp/1122") + id := generatePeerIds(1)[0] + addr := generateAddrs(1)[0] - // Shouldn't panic. - m.UpdateAddrs(id1, time.Hour, time.Minute) + m.SetAddr(id, addr, time.Hour) + testHas(t, []ma.Multiaddr{addr}, m.Addrs(id)) + + m.SetAddr(id, addr, -1) + testHas(t, nil, m.Addrs(id)) + } +} + +func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { + return func(t *testing.T) { + t.Run("update ttl of peer with no addrs", func(t *testing.T) { + id := generatePeerIds(1)[0] - m.SetAddr(id1, ma11, time.Hour) - m.SetAddr(id1, ma12, time.Minute) + // Shouldn't panic. + m.UpdateAddrs(id, time.Hour, time.Minute) + }) - // Shouldn't panic. - m.UpdateAddrs(id2, time.Hour, time.Minute) + t.Run("update ttls successfully", func(t *testing.T) { + ids := generatePeerIds(2) + addrs1, addrs2 := generateAddrs(2), generateAddrs(2) - m.SetAddr(id2, ma21, time.Hour) - m.SetAddr(id2, ma22, time.Minute) + // set two keys with different ttls for each peer. + m.SetAddr(ids[0], addrs1[0], time.Hour) + m.SetAddr(ids[0], addrs1[1], time.Minute) + m.SetAddr(ids[1], addrs2[0], time.Hour) + m.SetAddr(ids[1], addrs2[1], time.Minute) - testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + // Sanity check. + testHas(t, addrs1, m.Addrs(ids[0])) + testHas(t, addrs2, m.Addrs(ids[1])) - m.UpdateAddrs(id1, time.Hour, time.Second) + // Will only affect addrs1[0]. + m.UpdateAddrs(ids[0], time.Hour, time.Second) - testHas(t, []ma.Multiaddr{ma11, ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + // No immediate effect. + testHas(t, addrs1, m.Addrs(ids[0])) + testHas(t, addrs2, m.Addrs(ids[1])) - time.Sleep(1200 * time.Millisecond) + // After a wait, addrs[0] is gone. + time.Sleep(1200 * time.Millisecond) + testHas(t, addrs1[1:2], m.Addrs(ids[0])) + testHas(t, addrs2, m.Addrs(ids[1])) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + // Will only affect addrs2[0]. + m.UpdateAddrs(ids[1], time.Hour, time.Second) - m.UpdateAddrs(id2, time.Hour, time.Second) + // No immediate effect. + testHas(t, addrs1[1:2], m.Addrs(ids[0])) + testHas(t, addrs2, m.Addrs(ids[1])) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma21, ma22}, m.Addrs(id2)) + time.Sleep(1200 * time.Millisecond) - time.Sleep(1200 * time.Millisecond) + // First addrs is gone in both. + testHas(t, addrs1[1:], m.Addrs(ids[0])) + testHas(t, addrs2[1:], m.Addrs(ids[1])) + }) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma22}, m.Addrs(id2)) } } func testNilAddrsDontBreak(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - m.SetAddr(id1, nil, time.Hour) - m.AddAddr(id1, nil, time.Hour) + id := generatePeerIds(1)[0] + + m.SetAddr(id, nil, time.Hour) + m.AddAddr(id, nil, time.Hour) } } func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id1 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQN") - id2 := peerId(t, "QmcNstKuwBBoVTpSCSDrwzjgrRcaYXK833Psuz2EMHwyQM") - ma11 := multiaddr(t, "/ip4/1.2.3.1/tcp/1111") - ma12 := multiaddr(t, "/ip4/2.2.3.2/tcp/2222") - ma13 := multiaddr(t, "/ip4/3.2.3.3/tcp/3333") - ma24 := multiaddr(t, "/ip4/4.2.3.3/tcp/4444") - ma25 := multiaddr(t, "/ip4/5.2.3.3/tcp/5555") - - m.AddAddr(id1, ma11, time.Hour) - m.AddAddr(id1, ma12, time.Hour) - m.AddAddr(id1, ma13, time.Hour) - m.AddAddr(id2, ma24, time.Hour) - m.AddAddr(id2, ma25, time.Hour) - - testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) - - m.SetAddr(id1, ma11, 2*time.Hour) - m.SetAddr(id1, ma12, 2*time.Hour) - m.SetAddr(id1, ma13, 2*time.Hour) - m.SetAddr(id2, ma24, 2*time.Hour) - m.SetAddr(id2, ma25, 2*time.Hour) - - testHas(t, []ma.Multiaddr{ma11, ma12, ma13}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) - - m.SetAddr(id1, ma11, time.Millisecond) + ids := generatePeerIds(2) + addrs1 := generateAddrs(3) + addrs2 := generateAddrs(2) + + m.AddAddrs(ids[0], addrs1, time.Hour) + m.AddAddrs(ids[1], addrs2, time.Hour) + + testHas(t, addrs1, m.Addrs(ids[0])) + testHas(t, addrs2, m.Addrs(ids[1])) + + m.AddAddrs(ids[0], addrs1, 2*time.Hour) + m.AddAddrs(ids[1], addrs2, 2*time.Hour) + + testHas(t, addrs1, m.Addrs(ids[0])) + testHas(t, addrs2, m.Addrs(ids[1])) + + m.SetAddr(ids[0], addrs1[0], time.Millisecond) <-time.After(time.Millisecond * 5) - testHas(t, []ma.Multiaddr{ma12, ma13}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + testHas(t, addrs1[1:3], m.Addrs(ids[0])) + testHas(t, addrs2, m.Addrs(ids[1])) - m.SetAddr(id1, ma13, time.Millisecond) + m.SetAddr(ids[0], addrs1[2], time.Millisecond) <-time.After(time.Millisecond * 5) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma24, ma25}, m.Addrs(id2)) + testHas(t, addrs1[1:2], m.Addrs(ids[0])) + testHas(t, addrs2, m.Addrs(ids[1])) - m.SetAddr(id2, ma24, time.Millisecond) + m.SetAddr(ids[1], addrs2[0], time.Millisecond) <-time.After(time.Millisecond * 5) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, []ma.Multiaddr{ma25}, m.Addrs(id2)) + testHas(t, addrs1[1:2], m.Addrs(ids[0])) + testHas(t, addrs2[1:], m.Addrs(ids[1])) - m.SetAddr(id2, ma25, time.Millisecond) + m.SetAddr(ids[1], addrs2[1], time.Millisecond) <-time.After(time.Millisecond * 5) - testHas(t, []ma.Multiaddr{ma12}, m.Addrs(id1)) - testHas(t, nil, m.Addrs(id2)) + testHas(t, addrs1[1:2], m.Addrs(ids[0])) + testHas(t, nil, m.Addrs(ids[1])) - m.SetAddr(id1, ma12, time.Millisecond) + m.SetAddr(ids[0], addrs1[1], time.Millisecond) <-time.After(time.Millisecond * 5) - testHas(t, nil, m.Addrs(id1)) - testHas(t, nil, m.Addrs(id2)) + testHas(t, nil, m.Addrs(ids[0])) + testHas(t, nil, m.Addrs(ids[1])) } } diff --git a/p2p/host/peerstore/test/test_utils.go b/p2p/host/peerstore/test/test_utils.go index d94f640dca..4b282c446e 100644 --- a/p2p/host/peerstore/test/test_utils.go +++ b/p2p/host/peerstore/test/test_utils.go @@ -1,26 +1,22 @@ package test import ( - "testing" - "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) -func peerId(t *testing.T, ids string) peer.ID { - t.Helper() +func peerId(ids string) peer.ID { id, err := peer.IDB58Decode(ids) if err != nil { - t.Fatalf("id %q is bad: %s", ids, err) + panic(err) } return id } -func multiaddr(t *testing.T, m string) ma.Multiaddr { - t.Helper() +func multiaddr(m string) ma.Multiaddr { maddr, err := ma.NewMultiaddr(m) if err != nil { - t.Fatal(err) + panic(err) } return maddr } From 8abb8a37574a60818d55c65a88e1c971288ce80b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 7 Sep 2018 18:37:01 +0100 Subject: [PATCH 0657/3965] migrate Addrs() and PeersWithAddrs() to use txns, improve perf, benchmarks. --- p2p/host/peerstore/pstoreds/addr_book.go | 82 ++++++++++----------- p2p/host/peerstore/test/benchmarks_suite.go | 31 +++++--- 2 files changed, 59 insertions(+), 54 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index ba4ee275bc..e600b49b22 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -89,11 +89,6 @@ func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, er return keys[:i], clean[:i], nil } -func peerIDFromKey(key ds.Key) (peer.ID, error) { - idstring := key.Parent().Name() - return peer.IDB58Decode(idstring) -} - // AddAddr will add a new address if it's not already in the AddrBook. func (mgr *dsAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) @@ -264,36 +259,41 @@ func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time. mgr.ttlManager.adjustTTLs(prefix, oldTTL, newTTL) } -// Addrs Returns all of the non-expired addresses for a given peer. +// Addrs returns all of the non-expired addresses for a given peer. func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { - prefix := ds.NewKey(p.Pretty()) - q := query.Query{Prefix: prefix.String(), KeysOnly: true} - results, err := mgr.ds.Query(q) + var ( + prefix = ds.NewKey(p.Pretty()) + q = query.Query{Prefix: prefix.String(), KeysOnly: true} + results query.Results + err error + ) - if err != nil { + txn := mgr.ds.NewTransaction(true) + defer txn.Discard() + + if results, err = txn.Query(q); err != nil { log.Error(err) return nil } + defer results.Close() + var addrs []ma.Multiaddr for result := range results.Next() { key := ds.RawKey(result.Key) var addri interface{} addri, ok := mgr.cache.Get(key) + if !ok { - addri, err = mgr.ds.Get(key) - if err != nil { + if addri, err = txn.Get(key); err != nil { log.Error(err) continue } } - addrbytes := addri.([]byte) - addr, err := ma.NewMultiaddrBytes(addrbytes) - if err != nil { - log.Error(err) - continue + + if addr, err := ma.NewMultiaddrBytes(addri.([]byte)); err != nil { + addrs = append(addrs, addr) } - addrs = append(addrs, addr) } return addrs @@ -301,27 +301,36 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { // Peers returns all of the peer IDs for which the AddrBook has addresses. func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { - q := query.Query{KeysOnly: true} - results, err := mgr.ds.Query(q) + var ( + q = query.Query{KeysOnly: true} + results query.Results + err error + ) - if err != nil { + txn := mgr.ds.NewTransaction(true) + defer txn.Discard() + + if results, err = txn.Query(q); err != nil { log.Error(err) return peer.IDSlice{} } - idset := make(map[peer.ID]struct{}) + defer results.Close() + + idset := make(map[string]struct{}) for result := range results.Next() { key := ds.RawKey(result.Key) - id, err := peerIDFromKey(key) - if err != nil { - continue - } - idset[id] = struct{}{} + idset[key.Parent().Name()] = struct{}{} + } + + if len(idset) == 0 { + return peer.IDSlice{} } - ids := make(peer.IDSlice, 0, len(idset)) + ids := make(peer.IDSlice, 1, len(idset)) for id := range idset { - ids = append(ids, id) + i, _ := peer.IDB58Decode(id) + ids = append(ids, i) } return ids } @@ -397,21 +406,6 @@ func (mgr *dsAddrBook) dbClear(prefix ds.Key) ([]ds.Key, error) { return keys, nil } -func (mgr *dsAddrBook) doInTransaction(readOnly bool, op func(txn ds.Txn) error) error { - txn := mgr.ds.NewTransaction(false) - defer txn.Discard() - - if err := op(txn); err != nil { - return err - } - - if err := txn.Commit(); err != nil { - return err - } - - return nil -} - type ttlEntry struct { TTL time.Duration ExpiresAt time.Time diff --git a/p2p/host/peerstore/test/benchmarks_suite.go b/p2p/host/peerstore/test/benchmarks_suite.go index 6e1bcf9d75..61c611eb82 100644 --- a/p2p/host/peerstore/test/benchmarks_suite.go +++ b/p2p/host/peerstore/test/benchmarks_suite.go @@ -9,10 +9,11 @@ import ( ) var peerstoreBenchmarks = map[string]func(pstore.Peerstore, chan *peerpair) func(*testing.B){ - "AddAddrs": benchmarkAddAddrs, - "SetAddrs": benchmarkSetAddrs, - "GetAddrs": benchmarkGetAddrs, - "AddAndClearAddrs": benchmarkAddAndClearAddrs, + "AddAddrs": benchmarkAddAddrs, + "SetAddrs": benchmarkSetAddrs, + "GetAddrs": benchmarkGetAddrs, + "AddAndClearAddrs": benchmarkAddAndClearAddrs, + "Get1000PeersWithAddrs": benchmarkGet1000PeersWithAddrs, } func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory, variant string) { @@ -84,18 +85,28 @@ func benchmarkGetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing. } func benchmarkAddAndClearAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { - // 1000 peers. + return func(b *testing.B) { + b.ResetTimer() + for i := 0; i < b.N; i++ { + pp := <-addrs + ps.AddAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + ps.ClearAddrs(pp.ID) + } + } +} + +func benchmarkGet1000PeersWithAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { return func(b *testing.B) { var peers = make([]*peerpair, 1000) - for i := 0; i < len(peers); peers[i], i = <-addrs, i+1 { + for i, _ := range peers { + pp := <-addrs + ps.AddAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + peers[i] = pp } b.ResetTimer() for i := 0; i < b.N; i++ { - for _, pp := range peers { - ps.AddAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) - ps.ClearAddrs(pp.ID) - } + _ = ps.PeersWithAddrs() } } } From 9a5262383c526b9f6a944ee51a9de7034be763ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 7 Sep 2018 18:45:58 +0100 Subject: [PATCH 0658/3965] regroup test utils; remove unused function. --- p2p/host/peerstore/test/test_utils.go | 22 ------------------- .../test/{benchmark_utils.go => utils.go} | 0 2 files changed, 22 deletions(-) delete mode 100644 p2p/host/peerstore/test/test_utils.go rename p2p/host/peerstore/test/{benchmark_utils.go => utils.go} (100%) diff --git a/p2p/host/peerstore/test/test_utils.go b/p2p/host/peerstore/test/test_utils.go deleted file mode 100644 index 4b282c446e..0000000000 --- a/p2p/host/peerstore/test/test_utils.go +++ /dev/null @@ -1,22 +0,0 @@ -package test - -import ( - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" -) - -func peerId(ids string) peer.ID { - id, err := peer.IDB58Decode(ids) - if err != nil { - panic(err) - } - return id -} - -func multiaddr(m string) ma.Multiaddr { - maddr, err := ma.NewMultiaddr(m) - if err != nil { - panic(err) - } - return maddr -} diff --git a/p2p/host/peerstore/test/benchmark_utils.go b/p2p/host/peerstore/test/utils.go similarity index 100% rename from p2p/host/peerstore/test/benchmark_utils.go rename to p2p/host/peerstore/test/utils.go From 55afe7b05803c7d4cd497d75a7fe7a54aa7d69ef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 7 Sep 2018 18:46:23 +0100 Subject: [PATCH 0659/3965] normalise import aliases. --- p2p/host/peerstore/interface.go | 5 ++--- p2p/host/peerstore/peerstore.go | 2 +- p2p/host/peerstore/pstoreds/addr_book.go | 8 ++++---- p2p/host/peerstore/pstoreds/ds_test.go | 18 +++++++++--------- p2p/host/peerstore/pstoreds/peerstore.go | 6 +++--- p2p/host/peerstore/pstoremem/addr_book.go | 4 ++-- p2p/host/peerstore/pstoremem/inmem_test.go | 10 +++++----- p2p/host/peerstore/pstoremem/keybook.go | 2 +- p2p/host/peerstore/pstoremem/metadata.go | 3 +-- p2p/host/peerstore/test/keybook_suite.go | 2 +- p2p/host/peerstore/test/peerstore_suite.go | 4 ++-- p2p/host/peerstore/test/utils.go | 10 +++++++++- 12 files changed, 40 insertions(+), 34 deletions(-) diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 94b18aa632..068211c846 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -2,13 +2,12 @@ package peerstore import ( "context" + "errors" "math" "time" - "github.com/pkg/errors" - ic "github.com/libp2p/go-libp2p-crypto" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index d25eb4344e..e460b1c612 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -4,7 +4,7 @@ import ( "fmt" "sync" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ) var _ Peerstore = (*peerstore)(nil) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 5bb8b5e479..14c006a048 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -5,16 +5,16 @@ import ( "sync" "time" - "github.com/hashicorp/golang-lru" + lru "github.com/hashicorp/golang-lru" ds "github.com/ipfs/go-datastore" - "github.com/ipfs/go-datastore/query" + query "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" mh "github.com/multiformats/go-multihash" pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/pstoremem" + pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) var log = logging.Logger("peerstore/ds") diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 4ed24d0e9d..34b5d33b83 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -7,14 +7,14 @@ import ( "testing" "time" - "github.com/ipfs/go-datastore" + ds "github.com/ipfs/go-datastore" "github.com/ipfs/go-ds-badger" - "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/test" + pstore "github.com/libp2p/go-libp2p-peerstore" + pt "github.com/libp2p/go-libp2p-peerstore/test" ) -func setupBadgerDatastore(t testing.TB) (datastore.Batching, func()) { +func setupBadgerDatastore(t testing.TB) (ds.Batching, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { t.Fatal(err) @@ -30,8 +30,8 @@ func setupBadgerDatastore(t testing.TB) (datastore.Batching, func()) { return ds, closer } -func newPeerstoreFactory(tb testing.TB) test.PeerstoreFactory { - return func() (peerstore.Peerstore, func()) { +func newPeerstoreFactory(tb testing.TB) pt.PeerstoreFactory { + return func() (pstore.Peerstore, func()) { ds, closeFunc := setupBadgerDatastore(tb) ps, err := NewPeerstore(context.Background(), ds) @@ -44,11 +44,11 @@ func newPeerstoreFactory(tb testing.TB) test.PeerstoreFactory { } func TestBadgerDsPeerstore(t *testing.T) { - test.TestPeerstore(t, newPeerstoreFactory(t)) + pt.TestPeerstore(t, newPeerstoreFactory(t)) } func TestBadgerDsAddrBook(t *testing.T) { - test.TestAddrBook(t, func() (peerstore.AddrBook, func()) { + pt.TestAddrBook(t, func() (pstore.AddrBook, func()) { ds, closeDB := setupBadgerDatastore(t) mgr, err := NewAddrBook(context.Background(), ds, 100*time.Microsecond) @@ -65,5 +65,5 @@ func TestBadgerDsAddrBook(t *testing.T) { } func BenchmarkBadgerDsPeerstore(b *testing.B) { - test.BenchmarkPeerstore(b, newPeerstoreFactory(b)) + pt.BenchmarkPeerstore(b, newPeerstoreFactory(b)) } diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index e973988bd5..5125c38649 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -4,14 +4,14 @@ import ( "context" "time" - "github.com/ipfs/go-datastore" + ds "github.com/ipfs/go-datastore" pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/pstoremem" + pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) // NewPeerstore creates a peerstore backed by the provided persistent datastore. -func NewPeerstore(ctx context.Context, ds datastore.Batching) (pstore.Peerstore, error) { +func NewPeerstore(ctx context.Context, ds ds.Batching) (pstore.Peerstore, error) { addrBook, err := NewAddrBook(ctx, ds, time.Second) if err != nil { return nil, err diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index ebd2de1a46..6db498ad2e 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -7,11 +7,11 @@ import ( "time" logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/addr" + addr "github.com/libp2p/go-libp2p-peerstore/addr" ) var log = logging.Logger("peerstore") diff --git a/p2p/host/peerstore/pstoremem/inmem_test.go b/p2p/host/peerstore/pstoremem/inmem_test.go index e13135ea34..5d0d4f3283 100644 --- a/p2p/host/peerstore/pstoremem/inmem_test.go +++ b/p2p/host/peerstore/pstoremem/inmem_test.go @@ -4,29 +4,29 @@ import ( "testing" pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/test" + pt "github.com/libp2p/go-libp2p-peerstore/test" ) func TestInMemoryPeerstore(t *testing.T) { - test.TestPeerstore(t, func() (pstore.Peerstore, func()) { + pt.TestPeerstore(t, func() (pstore.Peerstore, func()) { return NewPeerstore(), nil }) } func TestInMemoryAddrBook(t *testing.T) { - test.TestAddrBook(t, func() (pstore.AddrBook, func()) { + pt.TestAddrBook(t, func() (pstore.AddrBook, func()) { return NewAddrBook(), nil }) } func TestInMemoryKeyBook(t *testing.T) { - test.TestKeyBook(t, func() (pstore.KeyBook, func()) { + pt.TestKeyBook(t, func() (pstore.KeyBook, func()) { return NewKeyBook(), nil }) } func BenchmarkInMemoryPeerstore(b *testing.B) { - test.BenchmarkPeerstore(b, func() (pstore.Peerstore, func()) { + pt.BenchmarkPeerstore(b, func() (pstore.Peerstore, func()) { return NewPeerstore(), nil }) } diff --git a/p2p/host/peerstore/pstoremem/keybook.go b/p2p/host/peerstore/pstoremem/keybook.go index 878cde9915..37437ebdbb 100644 --- a/p2p/host/peerstore/pstoremem/keybook.go +++ b/p2p/host/peerstore/pstoremem/keybook.go @@ -5,7 +5,7 @@ import ( "sync" ic "github.com/libp2p/go-libp2p-crypto" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ) diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index b4ba6056ed..8b58256c16 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -3,8 +3,7 @@ package pstoremem import ( "sync" - "github.com/libp2p/go-libp2p-peer" - + peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ) diff --git a/p2p/host/peerstore/test/keybook_suite.go b/p2p/host/peerstore/test/keybook_suite.go index c48a6eedfd..cb305e1e2c 100644 --- a/p2p/host/peerstore/test/keybook_suite.go +++ b/p2p/host/peerstore/test/keybook_suite.go @@ -4,7 +4,7 @@ import ( "sort" "testing" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" pt "github.com/libp2p/go-libp2p-peer/test" pstore "github.com/libp2p/go-libp2p-peerstore" diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index eac6aad119..6b08cab1da 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -8,8 +8,8 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-crypto" - "github.com/libp2p/go-libp2p-peer" + crypto "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" pstore "github.com/libp2p/go-libp2p-peerstore" diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index 714bf0d3e7..8630e4e914 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -5,7 +5,7 @@ import ( "fmt" "testing" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" pt "github.com/libp2p/go-libp2p-peer/test" ma "github.com/multiformats/go-multiaddr" ) @@ -42,3 +42,11 @@ func addressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair) { } } } + +func multiaddr(m string) ma.Multiaddr { + maddr, err := ma.NewMultiaddr(m) + if err != nil { + panic(err) + } + return maddr +} From 602f9d4cef57928d948e79adb693f5f54c3ea2c6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 7 Sep 2018 22:05:41 -0700 Subject: [PATCH 0660/3965] gx: update go-libp2p-peerstore --- p2p/net/swarm/testing/testing.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/testing/testing.go b/p2p/net/swarm/testing/testing.go index 29017df808..e3a62c5283 100644 --- a/p2p/net/swarm/testing/testing.go +++ b/p2p/net/swarm/testing/testing.go @@ -8,6 +8,7 @@ import ( metrics "github.com/libp2p/go-libp2p-metrics" inet "github.com/libp2p/go-libp2p-net" pstore "github.com/libp2p/go-libp2p-peerstore" + pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" secio "github.com/libp2p/go-libp2p-secio" tptu "github.com/libp2p/go-libp2p-transport-upgrader" tcp "github.com/libp2p/go-tcp-transport" @@ -66,7 +67,7 @@ func GenSwarm(t *testing.T, ctx context.Context, opts ...Option) *swarm.Swarm { p := tu.RandPeerNetParamsOrFatal(t) - ps := pstore.NewPeerstore() + ps := pstoremem.NewPeerstore() ps.AddPubKey(p.ID, p.PubKey) ps.AddPrivKey(p.ID, p.PrivKey) s := swarm.NewSwarm(ctx, p.ID, ps, metrics.NewBandwidthCounter()) From c46e8d6e363917452047839f995a20079fb3fc61 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 7 Sep 2018 22:11:23 -0700 Subject: [PATCH 0661/3965] gx: update go-libp2p-peerstore --- defaults.go | 4 +-- p2p/net/mock/mock_net.go | 3 +- package.json | 60 ++++++++++++++++++++-------------------- 3 files changed, 34 insertions(+), 33 deletions(-) diff --git a/defaults.go b/defaults.go index 90ef45bb8a..ba9bdc4063 100644 --- a/defaults.go +++ b/defaults.go @@ -6,7 +6,7 @@ import ( "crypto/rand" crypto "github.com/libp2p/go-libp2p-crypto" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" secio "github.com/libp2p/go-libp2p-secio" tcp "github.com/libp2p/go-tcp-transport" ws "github.com/libp2p/go-ws-transport" @@ -41,7 +41,7 @@ var DefaultTransports = ChainOptions( // DefaultPeerstore configures libp2p to use the default peerstore. var DefaultPeerstore Option = func(cfg *Config) error { - return cfg.Apply(Peerstore(pstore.NewPeerstore())) + return cfg.Apply(Peerstore(pstoremem.NewPeerstore())) } // RandomIdentity generates a random identity (default behaviour) diff --git a/p2p/net/mock/mock_net.go b/p2p/net/mock/mock_net.go index 898292af8e..9242cea549 100644 --- a/p2p/net/mock/mock_net.go +++ b/p2p/net/mock/mock_net.go @@ -16,6 +16,7 @@ import ( p2putil "github.com/libp2p/go-libp2p-netutil" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" testutil "github.com/libp2p/go-testutil" ma "github.com/multiformats/go-multiaddr" ) @@ -70,7 +71,7 @@ func (mn *mocknet) AddPeer(k ic.PrivKey, a ma.Multiaddr) (host.Host, error) { return nil, err } - ps := pstore.NewPeerstore() + ps := pstoremem.NewPeerstore() ps.AddAddr(p, a, pstore.PermanentAddrTTL) ps.AddPrivKey(p, k) ps.AddPubKey(p, k.GetPublic()) diff --git a/package.json b/package.json index ffed4490c6..ec20288212 100644 --- a/package.json +++ b/package.json @@ -61,27 +61,27 @@ }, { "author": "whyrusleeping", - "hash": "QmWri2HWdxHjWBUermhWy7QWJqN1cV8Gd1QbDiB5m86f1H", + "hash": "QmReYSQGHjf28pKf93FwyD72mLXoZo94MB2Cq6VBSUHvFB", "name": "go-libp2p-secio", - "version": "2.0.8" + "version": "2.0.9" }, { "author": "whyrusleeping", - "hash": "QmeKD8YT7887Xu6Z86iZmpYNxrLogJexqxEugSmaf14k64", + "hash": "Qmda4cPRvSRyox3SqgJN6DfSZGU5TtHufPTp9uXjFj71X6", "name": "go-libp2p-peerstore", - "version": "1.4.24" + "version": "2.0.0" }, { "author": "whyrusleeping", - "hash": "QmcDUyb52N62J8ZamGgUWUyWc1MtuCBce7WFA4D9xA6cwF", + "hash": "QmU129xU8dM79BgR97hu4fsiUDkTQrNHbzkiYfyrkNci8o", "name": "go-libp2p-transport", - "version": "3.0.8" + "version": "3.0.9" }, { "author": "whyrusleeping", - "hash": "Qmcw9fndogcYwyGs4a5TPDbnZPBLxvtrBZzpvyyVDzxDWT", + "hash": "QmV8KW6eBanaxCxGNrXx8Q3fZUqvumCz2Hwd2FGpb3vzYC", "name": "go-tcp-transport", - "version": "2.0.8" + "version": "2.0.9" }, { "author": "whyrusleeping", @@ -103,9 +103,9 @@ }, { "author": "whyrusleeping", - "hash": "QmQSbtGXCyNrj34LWL8EgXyNNYDZ8r3SwQcpW5pPxVhLnM", + "hash": "QmZNJyx9GGCX4GeuHnLB8fxaxMLs4MjTjHokxfQcCd6Nve", "name": "go-libp2p-net", - "version": "3.0.8" + "version": "3.0.9" }, { "author": "whyrusleeping", @@ -115,15 +115,15 @@ }, { "author": "whyrusleeping", - "hash": "QmfH9FKYv3Jp1xiyL8sPchGBUBg6JA6XviwajAo3qgnT3B", + "hash": "QmeMYW7Nj8jnnEfs9qhm7SxKkoDPUWXu3MsxX6BFwz34tf", "name": "go-libp2p-host", - "version": "3.0.8" + "version": "3.0.9" }, { "author": "whyrusleeping", - "hash": "Qmcc5CPuKyfDZNmqXNkk6j23CyZqZGypUv952NLHYGbeni", + "hash": "QmeDpqUwwdye8ABKVMPXKuWwPVURFdqTqssbTUB39E2Nwd", "name": "go-libp2p-swarm", - "version": "3.0.10" + "version": "3.0.11" }, { "author": "whyrusleeping", @@ -133,15 +133,15 @@ }, { "author": "whyrusleeping", - "hash": "QmXeofKUbQf6diwT12ZmpmAjW9c47aXT43t7qG1bGTRQTj", + "hash": "QmfDapjsRAfzVpjeEm2tSmX19QpCrkLDXRCDDWJcbbUsFn", "name": "go-libp2p-netutil", - "version": "0.4.6" + "version": "0.4.7" }, { "author": "whyrusleeping", - "hash": "QmeKQeTT46oH3ezWeNLLaa8RtFbvijmxtyRiRuQD5wASDk", + "hash": "QmQ4bjZSEC5drCRqssuXRymCswHPmW3Z46ibgBtg9XGd34", "name": "go-libp2p-blankhost", - "version": "0.3.8" + "version": "0.3.9" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "QmUveY3vGb6uQuzccihinJshn3KUFjvb5PaBZUVhqGx8TJ", + "hash": "QmWX6RySJ3yAYmfjLSw1LtRZnDh5oVeA9kM3scNQJkysqa", "name": "go-libp2p-circuit", - "version": "2.1.11" + "version": "2.2.1" }, { "author": "lgierth", @@ -181,9 +181,9 @@ }, { "author": "why", - "hash": "QmVz2p8ZVZ5GcWPNWGs2HZHiZyHumZcJpQdMRpxkMDhc2C", + "hash": "QmWGGN1nysi1qgqto31bENwESkmZBY4YGK4sZC3qhnqhSv", "name": "go-libp2p-interface-connmgr", - "version": "0.0.14" + "version": "0.0.15" }, { "author": "whyrusleeping", @@ -193,21 +193,21 @@ }, { "author": "whyrusleeping", - "hash": "QmSQhZyqoEepHPHr6kCpAoYVHzvVkTHAAXnNqFEFbVaaQr", + "hash": "Qmewujzx2quBisChBARfNDsMjTgUgs8qMkvCercSPQPLur", "name": "go-ws-transport", - "version": "2.0.8" + "version": "2.0.9" }, { "author": "stebalien", - "hash": "Qmd4VzS3T7ckhHcLCpWwgVMYe6RcKhYjNKfy1g7LJ5myjv", + "hash": "QmPGWMxTWBJwfH1dJo4J7eV4Fgpzb9gukPRkeEN1ycAkNu", "name": "go-conn-security-multistream", - "version": "0.1.8" + "version": "0.1.9" }, { "author": "Stebalien", - "hash": "QmNPQvc3jUA7L6oHB8en9PittPGazFMdoveN2oycHL6yWF", + "hash": "QmXaH5S5aHmbYyKSTMQDhWX8LzXfX91VGQFxaMdCuTUgBv", "name": "go-conn-security", - "version": "0.1.8" + "version": "0.1.9" }, { "author": "libp2p", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "QmefQrpDSYX6jQRtUyhcASFVBDkoAsDTPXemyxGMzA3phK", + "hash": "QmWRcKvbFVND1vSTJZv5imdBxmkj9FFJ5Jku1qWxasAAMo", "name": "go-libp2p-transport-upgrader", - "version": "0.1.8" + "version": "0.1.9" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", From fe20b1dd30913c09fbb11119c1506ecc5f61496b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 7 Sep 2018 22:18:29 -0700 Subject: [PATCH 0662/3965] gx publish 6.0.12 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 070e64e803..b29ec9e334 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.11: Qmf1u2efhjXYtuyP8SMHYtw4dCkbghnniex2PSp7baA7FP +6.0.12: QmUEqyXr97aUbNmQADHYNknjwjjdVpJXEt1UZXmSG81EV4 diff --git a/package.json b/package.json index ec20288212..add9ca786d 100644 --- a/package.json +++ b/package.json @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.11" + "version": "6.0.12" } From 30134cf5103e36112b02dcfe4382fd899f07ab91 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 7 Sep 2018 13:37:10 +0300 Subject: [PATCH 0663/3965] regenerate protobuf --- p2p/host/autonat/pb/autonat.pb.go | 1018 ++++++++++++++++++++++++++++- 1 file changed, 994 insertions(+), 24 deletions(-) diff --git a/p2p/host/autonat/pb/autonat.pb.go b/p2p/host/autonat/pb/autonat.pb.go index d1c981c01c..3f2db2e1a1 100644 --- a/p2p/host/autonat/pb/autonat.pb.go +++ b/p2p/host/autonat/pb/autonat.pb.go @@ -1,27 +1,25 @@ -// Code generated by protoc-gen-gogo. +// Code generated by protoc-gen-gogo. DO NOT EDIT. // source: autonat.proto -// DO NOT EDIT! -/* -Package autonat_pb is a generated protocol buffer package. - -It is generated from these files: - autonat.proto - -It has these top-level messages: - Message -*/ package autonat_pb import proto "github.com/gogo/protobuf/proto" import fmt "fmt" import math "math" +import io "io" + // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal var _ = fmt.Errorf var _ = math.Inf +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + type Message_MessageType int32 const ( @@ -54,6 +52,9 @@ func (x *Message_MessageType) UnmarshalJSON(data []byte) error { *x = Message_MessageType(value) return nil } +func (Message_MessageType) EnumDescriptor() ([]byte, []int) { + return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 0} +} type Message_ResponseStatus int32 @@ -96,17 +97,51 @@ func (x *Message_ResponseStatus) UnmarshalJSON(data []byte) error { *x = Message_ResponseStatus(value) return nil } +func (Message_ResponseStatus) EnumDescriptor() ([]byte, []int) { + return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 1} +} type Message struct { - Type *Message_MessageType `protobuf:"varint,1,opt,name=type,enum=autonat.pb.Message_MessageType" json:"type,omitempty"` - Dial *Message_Dial `protobuf:"bytes,2,opt,name=dial" json:"dial,omitempty"` - DialResponse *Message_DialResponse `protobuf:"bytes,3,opt,name=dialResponse" json:"dialResponse,omitempty"` - XXX_unrecognized []byte `json:"-"` + Type *Message_MessageType `protobuf:"varint,1,opt,name=type,enum=autonat.pb.Message_MessageType" json:"type,omitempty"` + Dial *Message_Dial `protobuf:"bytes,2,opt,name=dial" json:"dial,omitempty"` + DialResponse *Message_DialResponse `protobuf:"bytes,3,opt,name=dialResponse" json:"dialResponse,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *Message) Reset() { *m = Message{} } func (m *Message) String() string { return proto.CompactTextString(m) } func (*Message) ProtoMessage() {} +func (*Message) Descriptor() ([]byte, []int) { + return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0} +} +func (m *Message) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Message) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Message.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *Message) XXX_Merge(src proto.Message) { + xxx_messageInfo_Message.Merge(dst, src) +} +func (m *Message) XXX_Size() int { + return m.Size() +} +func (m *Message) XXX_DiscardUnknown() { + xxx_messageInfo_Message.DiscardUnknown(m) +} + +var xxx_messageInfo_Message proto.InternalMessageInfo func (m *Message) GetType() Message_MessageType { if m != nil && m.Type != nil { @@ -130,14 +165,45 @@ func (m *Message) GetDialResponse() *Message_DialResponse { } type Message_PeerInfo struct { - Id []byte `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"` - Addrs [][]byte `protobuf:"bytes,2,rep,name=addrs" json:"addrs,omitempty"` - XXX_unrecognized []byte `json:"-"` + Id []byte `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"` + Addrs [][]byte `protobuf:"bytes,2,rep,name=addrs" json:"addrs,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *Message_PeerInfo) Reset() { *m = Message_PeerInfo{} } func (m *Message_PeerInfo) String() string { return proto.CompactTextString(m) } func (*Message_PeerInfo) ProtoMessage() {} +func (*Message_PeerInfo) Descriptor() ([]byte, []int) { + return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 0} +} +func (m *Message_PeerInfo) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Message_PeerInfo) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Message_PeerInfo.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *Message_PeerInfo) XXX_Merge(src proto.Message) { + xxx_messageInfo_Message_PeerInfo.Merge(dst, src) +} +func (m *Message_PeerInfo) XXX_Size() int { + return m.Size() +} +func (m *Message_PeerInfo) XXX_DiscardUnknown() { + xxx_messageInfo_Message_PeerInfo.DiscardUnknown(m) +} + +var xxx_messageInfo_Message_PeerInfo proto.InternalMessageInfo func (m *Message_PeerInfo) GetId() []byte { if m != nil { @@ -154,13 +220,44 @@ func (m *Message_PeerInfo) GetAddrs() [][]byte { } type Message_Dial struct { - Peer *Message_PeerInfo `protobuf:"bytes,1,opt,name=peer" json:"peer,omitempty"` - XXX_unrecognized []byte `json:"-"` + Peer *Message_PeerInfo `protobuf:"bytes,1,opt,name=peer" json:"peer,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *Message_Dial) Reset() { *m = Message_Dial{} } func (m *Message_Dial) String() string { return proto.CompactTextString(m) } func (*Message_Dial) ProtoMessage() {} +func (*Message_Dial) Descriptor() ([]byte, []int) { + return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 1} +} +func (m *Message_Dial) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Message_Dial) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Message_Dial.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *Message_Dial) XXX_Merge(src proto.Message) { + xxx_messageInfo_Message_Dial.Merge(dst, src) +} +func (m *Message_Dial) XXX_Size() int { + return m.Size() +} +func (m *Message_Dial) XXX_DiscardUnknown() { + xxx_messageInfo_Message_Dial.DiscardUnknown(m) +} + +var xxx_messageInfo_Message_Dial proto.InternalMessageInfo func (m *Message_Dial) GetPeer() *Message_PeerInfo { if m != nil { @@ -170,15 +267,46 @@ func (m *Message_Dial) GetPeer() *Message_PeerInfo { } type Message_DialResponse struct { - Status *Message_ResponseStatus `protobuf:"varint,1,opt,name=status,enum=autonat.pb.Message_ResponseStatus" json:"status,omitempty"` - StatusText *string `protobuf:"bytes,2,opt,name=statusText" json:"statusText,omitempty"` - Addr []byte `protobuf:"bytes,3,opt,name=addr" json:"addr,omitempty"` - XXX_unrecognized []byte `json:"-"` + Status *Message_ResponseStatus `protobuf:"varint,1,opt,name=status,enum=autonat.pb.Message_ResponseStatus" json:"status,omitempty"` + StatusText *string `protobuf:"bytes,2,opt,name=statusText" json:"statusText,omitempty"` + Addr []byte `protobuf:"bytes,3,opt,name=addr" json:"addr,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *Message_DialResponse) Reset() { *m = Message_DialResponse{} } func (m *Message_DialResponse) String() string { return proto.CompactTextString(m) } func (*Message_DialResponse) ProtoMessage() {} +func (*Message_DialResponse) Descriptor() ([]byte, []int) { + return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 2} +} +func (m *Message_DialResponse) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Message_DialResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Message_DialResponse.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *Message_DialResponse) XXX_Merge(src proto.Message) { + xxx_messageInfo_Message_DialResponse.Merge(dst, src) +} +func (m *Message_DialResponse) XXX_Size() int { + return m.Size() +} +func (m *Message_DialResponse) XXX_DiscardUnknown() { + xxx_messageInfo_Message_DialResponse.DiscardUnknown(m) +} + +var xxx_messageInfo_Message_DialResponse proto.InternalMessageInfo func (m *Message_DialResponse) GetStatus() Message_ResponseStatus { if m != nil && m.Status != nil { @@ -209,3 +337,845 @@ func init() { proto.RegisterEnum("autonat.pb.Message_MessageType", Message_MessageType_name, Message_MessageType_value) proto.RegisterEnum("autonat.pb.Message_ResponseStatus", Message_ResponseStatus_name, Message_ResponseStatus_value) } +func (m *Message) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Message) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Type != nil { + dAtA[i] = 0x8 + i++ + i = encodeVarintAutonat(dAtA, i, uint64(*m.Type)) + } + if m.Dial != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintAutonat(dAtA, i, uint64(m.Dial.Size())) + n1, err := m.Dial.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n1 + } + if m.DialResponse != nil { + dAtA[i] = 0x1a + i++ + i = encodeVarintAutonat(dAtA, i, uint64(m.DialResponse.Size())) + n2, err := m.DialResponse.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n2 + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func (m *Message_PeerInfo) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Message_PeerInfo) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Id != nil { + dAtA[i] = 0xa + i++ + i = encodeVarintAutonat(dAtA, i, uint64(len(m.Id))) + i += copy(dAtA[i:], m.Id) + } + if len(m.Addrs) > 0 { + for _, b := range m.Addrs { + dAtA[i] = 0x12 + i++ + i = encodeVarintAutonat(dAtA, i, uint64(len(b))) + i += copy(dAtA[i:], b) + } + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func (m *Message_Dial) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Message_Dial) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Peer != nil { + dAtA[i] = 0xa + i++ + i = encodeVarintAutonat(dAtA, i, uint64(m.Peer.Size())) + n3, err := m.Peer.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n3 + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func (m *Message_DialResponse) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Message_DialResponse) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Status != nil { + dAtA[i] = 0x8 + i++ + i = encodeVarintAutonat(dAtA, i, uint64(*m.Status)) + } + if m.StatusText != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintAutonat(dAtA, i, uint64(len(*m.StatusText))) + i += copy(dAtA[i:], *m.StatusText) + } + if m.Addr != nil { + dAtA[i] = 0x1a + i++ + i = encodeVarintAutonat(dAtA, i, uint64(len(m.Addr))) + i += copy(dAtA[i:], m.Addr) + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func encodeVarintAutonat(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func (m *Message) Size() (n int) { + var l int + _ = l + if m.Type != nil { + n += 1 + sovAutonat(uint64(*m.Type)) + } + if m.Dial != nil { + l = m.Dial.Size() + n += 1 + l + sovAutonat(uint64(l)) + } + if m.DialResponse != nil { + l = m.DialResponse.Size() + n += 1 + l + sovAutonat(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *Message_PeerInfo) Size() (n int) { + var l int + _ = l + if m.Id != nil { + l = len(m.Id) + n += 1 + l + sovAutonat(uint64(l)) + } + if len(m.Addrs) > 0 { + for _, b := range m.Addrs { + l = len(b) + n += 1 + l + sovAutonat(uint64(l)) + } + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *Message_Dial) Size() (n int) { + var l int + _ = l + if m.Peer != nil { + l = m.Peer.Size() + n += 1 + l + sovAutonat(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *Message_DialResponse) Size() (n int) { + var l int + _ = l + if m.Status != nil { + n += 1 + sovAutonat(uint64(*m.Status)) + } + if m.StatusText != nil { + l = len(*m.StatusText) + n += 1 + l + sovAutonat(uint64(l)) + } + if m.Addr != nil { + l = len(m.Addr) + n += 1 + l + sovAutonat(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func sovAutonat(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozAutonat(x uint64) (n int) { + return sovAutonat(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *Message) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Message: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Message: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Type", wireType) + } + var v Message_MessageType + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= (Message_MessageType(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + m.Type = &v + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Dial", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthAutonat + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.Dial == nil { + m.Dial = &Message_Dial{} + } + if err := m.Dial.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field DialResponse", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthAutonat + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.DialResponse == nil { + m.DialResponse = &Message_DialResponse{} + } + if err := m.DialResponse.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAutonat(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthAutonat + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *Message_PeerInfo) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: PeerInfo: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: PeerInfo: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Id", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAutonat + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Id = append(m.Id[:0], dAtA[iNdEx:postIndex]...) + if m.Id == nil { + m.Id = []byte{} + } + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Addrs", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAutonat + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Addrs = append(m.Addrs, make([]byte, postIndex-iNdEx)) + copy(m.Addrs[len(m.Addrs)-1], dAtA[iNdEx:postIndex]) + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAutonat(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthAutonat + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *Message_Dial) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Dial: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Dial: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Peer", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthAutonat + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.Peer == nil { + m.Peer = &Message_PeerInfo{} + } + if err := m.Peer.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAutonat(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthAutonat + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: DialResponse: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: DialResponse: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Status", wireType) + } + var v Message_ResponseStatus + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= (Message_ResponseStatus(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + m.Status = &v + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field StatusText", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthAutonat + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + s := string(dAtA[iNdEx:postIndex]) + m.StatusText = &s + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Addr", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAutonat + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAutonat + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Addr = append(m.Addr[:0], dAtA[iNdEx:postIndex]...) + if m.Addr == nil { + m.Addr = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAutonat(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthAutonat + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipAutonat(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowAutonat + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowAutonat + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowAutonat + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + iNdEx += length + if length < 0 { + return 0, ErrInvalidLengthAutonat + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowAutonat + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipAutonat(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthAutonat = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowAutonat = fmt.Errorf("proto: integer overflow") +) + +func init() { proto.RegisterFile("autonat.proto", fileDescriptor_autonat_bd0ec7a019b57e9d) } + +var fileDescriptor_autonat_bd0ec7a019b57e9d = []byte{ + // 372 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x74, 0x90, 0xcf, 0x8a, 0xda, 0x50, + 0x14, 0xc6, 0xbd, 0x31, 0xb5, 0xf6, 0x18, 0xc3, 0xed, 0xa1, 0x85, 0x20, 0x25, 0x0d, 0x59, 0x49, + 0x29, 0x22, 0x76, 0x53, 0xba, 0x53, 0x72, 0x0b, 0xd2, 0x56, 0xed, 0x49, 0x5c, 0x87, 0x94, 0xdc, + 0x0e, 0x01, 0x31, 0x21, 0x89, 0x30, 0x6e, 0xe6, 0x89, 0x66, 0x3b, 0xef, 0xe0, 0x72, 0x1e, 0x61, + 0xf0, 0x49, 0x86, 0x5c, 0xa3, 0xa3, 0xe0, 0xac, 0xce, 0x1f, 0x7e, 0xdf, 0x39, 0x1f, 0x1f, 0x74, + 0xa3, 0x4d, 0x99, 0xae, 0xa3, 0x72, 0x90, 0xe5, 0x69, 0x99, 0x22, 0x9c, 0xc6, 0x7f, 0xee, 0x83, + 0x0e, 0x6f, 0xff, 0xc8, 0xa2, 0x88, 0x6e, 0x24, 0x7e, 0x03, 0xbd, 0xdc, 0x66, 0xd2, 0x62, 0x0e, + 0xeb, 0x9b, 0xa3, 0xcf, 0x83, 0x17, 0x6c, 0x50, 0x23, 0xc7, 0x1a, 0x6c, 0x33, 0x49, 0x0a, 0xc6, + 0xaf, 0xa0, 0xc7, 0x49, 0xb4, 0xb2, 0x34, 0x87, 0xf5, 0x3b, 0x23, 0xeb, 0x9a, 0xc8, 0x4b, 0xa2, + 0x15, 0x29, 0x0a, 0x3d, 0x30, 0xaa, 0x4a, 0xb2, 0xc8, 0xd2, 0x75, 0x21, 0xad, 0xa6, 0x52, 0x39, + 0xaf, 0xaa, 0x6a, 0x8e, 0x2e, 0x54, 0xbd, 0x21, 0xb4, 0x17, 0x52, 0xe6, 0xd3, 0xf5, 0xff, 0x14, + 0x4d, 0xd0, 0x92, 0x58, 0x59, 0x36, 0x48, 0x4b, 0x62, 0xfc, 0x00, 0x6f, 0xa2, 0x38, 0xce, 0x0b, + 0x4b, 0x73, 0x9a, 0x7d, 0x83, 0x0e, 0x43, 0xef, 0x3b, 0xe8, 0xd5, 0x3d, 0x1c, 0x82, 0x9e, 0x49, + 0x99, 0x2b, 0xbe, 0x33, 0xfa, 0x74, 0xed, 0xef, 0xf1, 0x32, 0x29, 0xb2, 0x77, 0x07, 0xc6, 0xb9, + 0x13, 0xfc, 0x01, 0xad, 0xa2, 0x8c, 0xca, 0x4d, 0x51, 0xc7, 0xe4, 0x5e, 0xbb, 0x71, 0xa4, 0x7d, + 0x45, 0x52, 0xad, 0x40, 0x1b, 0xe0, 0xd0, 0x05, 0xf2, 0xb6, 0x54, 0x89, 0xbd, 0xa3, 0xb3, 0x0d, + 0x22, 0xe8, 0x95, 0x5d, 0x95, 0x8a, 0x41, 0xaa, 0x77, 0xbf, 0x40, 0xe7, 0x2c, 0x74, 0x6c, 0x83, + 0xee, 0x4d, 0xc7, 0xbf, 0x79, 0x03, 0xdf, 0x43, 0xb7, 0xea, 0x42, 0x12, 0xfe, 0x62, 0x3e, 0xf3, + 0x05, 0x67, 0x6e, 0x02, 0xe6, 0xe5, 0x67, 0x6c, 0x81, 0x36, 0xff, 0xc5, 0x1b, 0xc8, 0xc1, 0x10, + 0xa1, 0xc2, 0x05, 0xd1, 0x9c, 0x78, 0x8c, 0x08, 0x66, 0xbd, 0x21, 0xf1, 0x73, 0xe9, 0x0b, 0x8f, + 0x4b, 0x44, 0xe8, 0x8a, 0x70, 0x32, 0xf6, 0x42, 0x12, 0x7f, 0x97, 0xc2, 0x0f, 0xf8, 0x8e, 0xe1, + 0x47, 0xe0, 0x22, 0x9c, 0xce, 0x02, 0x41, 0xb3, 0x93, 0xfa, 0x5e, 0x9b, 0x18, 0xbb, 0xbd, 0xcd, + 0x1e, 0xf7, 0x36, 0x7b, 0xda, 0xdb, 0xec, 0x39, 0x00, 0x00, 0xff, 0xff, 0x8e, 0xe2, 0x93, 0x4e, + 0x61, 0x02, 0x00, 0x00, +} From fc32ec33156c9e659ebc55fdf4811196e7653a09 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 8 Sep 2018 11:30:56 +0300 Subject: [PATCH 0664/3965] svc: construct dialer host without listen addrs --- p2p/host/autonat/svc.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 236f162e23..32d089e079 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -31,7 +31,7 @@ type AutoNATService struct { // NewAutoNATService creates a new AutoNATService instance attached to a host func NewAutoNATService(ctx context.Context, h host.Host) (*AutoNATService, error) { - dialer, err := libp2p.New(ctx) + dialer, err := libp2p.New(ctx, libp2p.NoListenAddrs) if err != nil { return nil, err } From 9d20474431fc5d736c5a3a7e76a382cad9fda73a Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 8 Sep 2018 14:21:04 +0300 Subject: [PATCH 0665/3965] accept libp2p options for the dialer constructor in NewAutoNATService --- p2p/host/autonat/svc.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 32d089e079..fd0d08f146 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -30,8 +30,9 @@ type AutoNATService struct { } // NewAutoNATService creates a new AutoNATService instance attached to a host -func NewAutoNATService(ctx context.Context, h host.Host) (*AutoNATService, error) { - dialer, err := libp2p.New(ctx, libp2p.NoListenAddrs) +func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) (*AutoNATService, error) { + opts = append(opts, libp2p.NoListenAddrs) + dialer, err := libp2p.New(ctx, opts...) if err != nil { return nil, err } From a0234678eb4656cd5fe77a6225dc73931b429e92 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 8 Sep 2018 15:57:48 +0300 Subject: [PATCH 0666/3965] make service dialback timeout configurable; useful for tests --- p2p/host/autonat/svc.go | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index fd0d08f146..6ec7e7c9af 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -18,7 +18,10 @@ import ( const P_CIRCUIT = 290 -var AutoNATServiceResetInterval = 1 * time.Minute +var ( + AutoNATServiceDialTimeout = 42 * time.Second + AutoNATServiceResetInterval = 1 * time.Minute +) // AutoNATService provides NAT autodetection services to other peers type AutoNATService struct { @@ -144,7 +147,7 @@ func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { as.peers[pi.ID] = struct{}{} as.mx.Unlock() - ctx, cancel := context.WithTimeout(as.ctx, 42*time.Second) + ctx, cancel := context.WithTimeout(as.ctx, AutoNATServiceDialTimeout) defer cancel() err := as.dialer.Connect(ctx, pi) From bc61d559c5bf598cc26f4a763b11e91acf9f827e Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 8 Sep 2018 16:10:11 +0300 Subject: [PATCH 0667/3965] basic service tests --- p2p/host/autonat/svc_test.go | 128 +++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 p2p/host/autonat/svc_test.go diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go new file mode 100644 index 0000000000..8cc7f667a2 --- /dev/null +++ b/p2p/host/autonat/svc_test.go @@ -0,0 +1,128 @@ +package autonat + +import ( + "context" + "net" + "testing" + "time" + + libp2p "github.com/libp2p/go-libp2p" + host "github.com/libp2p/go-libp2p-host" + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATService) { + h, err := libp2p.New(ctx) + if err != nil { + t.Fatal(err) + } + + as, err := NewAutoNATService(ctx, h) + if err != nil { + t.Fatal(err) + } + + return h, as +} + +func makeAutoNATClient(ctx context.Context, t *testing.T) (host.Host, AutoNATClient) { + h, err := libp2p.New(ctx) + if err != nil { + t.Fatal(err) + } + + cli := NewAutoNATClient(h) + return h, cli +} + +func connect(t *testing.T, a, b host.Host) { + pinfo := pstore.PeerInfo{ID: a.ID(), Addrs: a.Addrs()} + err := b.Connect(context.Background(), pinfo) + if err != nil { + t.Fatal(err) + } +} + +// Note: these tests assume that the host has only private inet addresses! +func TestAutoNATServiceDialError(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save := AutoNATServiceDialTimeout + AutoNATServiceDialTimeout = 1 * time.Second + + hs, _ := makeAutoNATService(ctx, t) + hc, ac := makeAutoNATClient(ctx, t) + connect(t, hs, hc) + + _, err := ac.Dial(ctx, hs.ID()) + if err == nil { + t.Fatal("Dial back succeeded unexpectedly!") + } + + if !IsDialError(err) { + t.Fatal(err) + } + + AutoNATServiceDialTimeout = save +} + +func TestAutoNATServiceDialSuccess(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save := private4 + private4 = []*net.IPNet{} + + hs, _ := makeAutoNATService(ctx, t) + hc, ac := makeAutoNATClient(ctx, t) + connect(t, hs, hc) + + _, err := ac.Dial(ctx, hs.ID()) + if err != nil { + t.Fatalf("Dial back failed: %s", err.Error()) + } + + private4 = save +} + +func TestAutoNATServiceDialRateLimiter(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save1 := AutoNATServiceDialTimeout + AutoNATServiceDialTimeout = 1 * time.Second + save2 := AutoNATServiceResetInterval + AutoNATServiceResetInterval = 1 * time.Second + save3 := private4 + private4 = []*net.IPNet{} + + hs, _ := makeAutoNATService(ctx, t) + hc, ac := makeAutoNATClient(ctx, t) + connect(t, hs, hc) + + _, err := ac.Dial(ctx, hs.ID()) + if err != nil { + t.Fatal(err) + } + + _, err = ac.Dial(ctx, hs.ID()) + if err == nil { + t.Fatal("Dial back succeeded unexpectedly!") + } + + if !IsDialRefused(err) { + t.Fatal(err) + } + + time.Sleep(2 * time.Second) + + _, err = ac.Dial(ctx, hs.ID()) + if err != nil { + t.Fatal(err) + } + + AutoNATServiceDialTimeout = save1 + AutoNATServiceResetInterval = save2 + private4 = save3 +} From b88d3b600facb55df0fdfedf3b909fb812d96fb8 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 8 Sep 2018 17:01:21 +0300 Subject: [PATCH 0668/3965] test for ambient autonat functionality --- p2p/host/autonat/autonat_test.go | 84 ++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 p2p/host/autonat/autonat_test.go diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go new file mode 100644 index 0000000000..019a5906b5 --- /dev/null +++ b/p2p/host/autonat/autonat_test.go @@ -0,0 +1,84 @@ +package autonat + +import ( + "context" + "net" + "testing" + "time" + + libp2p "github.com/libp2p/go-libp2p" + host "github.com/libp2p/go-libp2p-host" +) + +func makeAutoNAT(ctx context.Context, t *testing.T) (host.Host, AutoNAT) { + h, err := libp2p.New(ctx) + if err != nil { + t.Fatal(err) + } + + a := NewAutoNAT(ctx, h) + + return h, a +} + +// Note: these tests assume the host has only private inet addresses! +func TestAutoNATPrivate(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save1 := AutoNATBootDelay + AutoNATBootDelay = 1 * time.Second + save2 := AutoNATRefreshInterval + AutoNATRefreshInterval = 1 * time.Second + + hs, _ := makeAutoNATService(ctx, t) + hc, an := makeAutoNAT(ctx, t) + + status := an.Status() + if status != NATStatusUnknown { + t.Fatalf("unexpected NAT status: %d", status) + } + + connect(t, hs, hc) + time.Sleep(2 * time.Second) + + status = an.Status() + if status != NATStatusPrivate { + t.Fatalf("unexpected NAT status: %d", status) + } + + AutoNATBootDelay = save1 + AutoNATRefreshInterval = save2 +} + +func TestAutoNATPublic(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save1 := AutoNATBootDelay + AutoNATBootDelay = 1 * time.Second + save2 := AutoNATRefreshInterval + AutoNATRefreshInterval = 1 * time.Second + save3 := private4 + private4 = []*net.IPNet{} + + hs, _ := makeAutoNATService(ctx, t) + hc, an := makeAutoNAT(ctx, t) + + status := an.Status() + if status != NATStatusUnknown { + t.Fatalf("unexpected NAT status: %d", status) + } + + connect(t, hs, hc) + time.Sleep(2 * time.Second) + + status = an.Status() + if status != NATStatusPublic { + t.Fatalf("unexpected NAT status: %d", status) + } + + AutoNATBootDelay = save1 + AutoNATRefreshInterval = save2 + private4 = save3 +} From ea75b1845bedffcae2f7cc9ea718bd0f96023286 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 10 Sep 2018 19:59:27 +0100 Subject: [PATCH 0669/3965] add baseline benchmarks. --- p2p/host/peerstore/pstoreds/ds_test.go | 104 +++++++++++++++++++++++++ 1 file changed, 104 insertions(+) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 5ed07dd9e5..5fc9d74fe2 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -2,17 +2,121 @@ package pstoreds import ( "context" + "fmt" "io/ioutil" "os" "testing" "time" + bd "github.com/dgraph-io/badger" + "github.com/ipfs/go-datastore" "github.com/ipfs/go-ds-badger" + "github.com/libp2p/go-libp2p-peerstore" "github.com/libp2p/go-libp2p-peerstore/test" ) +func BenchmarkBaselineBadgerDatastorePutEntry(b *testing.B) { + bds, closer := badgerStore(b) + defer closer() + + b.ResetTimer() + for i := 0; i < b.N; i++ { + txn := bds.NewTransaction(false) + + key := datastore.RawKey(fmt.Sprintf("/key/%d", i)) + txn.Put(key, []byte(fmt.Sprintf("/value/%d", i))) + + txn.Commit() + txn.Discard() + } +} + +func BenchmarkBaselineBadgerDatastoreGetEntry(b *testing.B) { + bds, closer := badgerStore(b) + defer closer() + + txn := bds.NewTransaction(false) + keys := make([]datastore.Key, 1000) + for i := 0; i < 1000; i++ { + key := datastore.RawKey(fmt.Sprintf("/key/%d", i)) + txn.Put(key, []byte(fmt.Sprintf("/value/%d", i))) + keys[i] = key + } + if err := txn.Commit(); err != nil { + b.Fatal(err) + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + txn := bds.NewTransaction(true) + if _, err := txn.Get(keys[i%1000]); err != nil { + b.Fatal(err) + } + txn.Discard() + } +} + +func BenchmarkBaselineBadgerDirectPutEntry(b *testing.B) { + opts := bd.DefaultOptions + + dataPath, err := ioutil.TempDir(os.TempDir(), "badger") + if err != nil { + b.Fatal(err) + } + + opts.Dir = dataPath + opts.ValueDir = dataPath + opts.SyncWrites = false + + db, err := bd.Open(opts) + if err != nil { + b.Fatal(err) + } + + defer db.Close() + + b.ResetTimer() + for i := 0; i < b.N; i++ { + txn := db.NewTransaction(true) + txn.Set([]byte(fmt.Sprintf("/key/%d", i)), []byte(fmt.Sprintf("/value/%d", i))) + txn.Commit(nil) + } +} + +func BenchmarkBaselineBadgerDirectGetEntry(b *testing.B) { + opts := bd.DefaultOptions + + dataPath, err := ioutil.TempDir(os.TempDir(), "badger") + if err != nil { + b.Fatal(err) + } + + opts.Dir = dataPath + opts.ValueDir = dataPath + + db, err := bd.Open(opts) + if err != nil { + b.Fatal(err) + } + + defer db.Close() + + txn := db.NewTransaction(true) + for i := 0; i < 1000; i++ { + txn.Set([]byte(fmt.Sprintf("/key/%d", i)), []byte(fmt.Sprintf("/value/%d", i))) + } + txn.Commit(nil) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + txn := db.NewTransaction(false) + txn.Get([]byte(fmt.Sprintf("/key/%d", i%1000))) + txn.Discard() + } +} + func TestBadgerDsPeerstore(t *testing.T) { test.TestPeerstore(t, peerstoreFactory(t, DefaultOpts())) } From e436bcc3917fca066bc328221e679a388fdb9738 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 10 Sep 2018 15:51:09 -0700 Subject: [PATCH 0670/3965] always remove connection from identify service map fixes #419 Also call FullClose in a goroutine; no need to block this. (not happy with that but I'm starting to think we need to rethink stream closing, again...) --- p2p/protocol/identify/id.go | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index fb949851e0..1f8dc01203 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -83,7 +83,12 @@ func (ids *IDService) IdentifyConn(c inet.Conn) { ids.currid[c] = ch ids.currmu.Unlock() - defer close(ch) + defer func() { + close(ch) + ids.currmu.Lock() + delete(ids.currid, c) + ids.currmu.Unlock() + }() s, err := c.NewStream() if err != nil { @@ -103,16 +108,6 @@ func (ids *IDService) IdentifyConn(c inet.Conn) { } ids.responseHandler(s) - - ids.currmu.Lock() - _, found := ids.currid[c] - delete(ids.currid, c) - ids.currmu.Unlock() - - if !found { - log.Errorf("IdentifyConn failed to find channel (programmer error) for %s", c) - } - inet.FullClose(s) } func (ids *IDService) requestHandler(s inet.Stream) { @@ -135,12 +130,14 @@ func (ids *IDService) responseHandler(s inet.Stream) { mes := pb.Identify{} if err := r.ReadMsg(&mes); err != nil { log.Warning("error reading identify message: ", err) + s.Reset() return } ids.consumeMessage(&mes, c) - log.Debugf("%s received message from %s %s", ID, c.RemotePeer(), c.RemoteMultiaddr()) + + go inet.FullClose(s) } func (ids *IDService) populateMessage(mes *pb.Identify, c inet.Conn) { From 5c6b53e1f6269f57964eb607c98b9435ce7a03d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 11 Sep 2018 11:48:23 +0100 Subject: [PATCH 0671/3965] change cache layout, now peer.ID => []ma.Multiaddr. --- p2p/host/peerstore/pstoreds/addr_book.go | 59 ++++++++++-------------- 1 file changed, 24 insertions(+), 35 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index e600b49b22..147e953961 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -124,11 +124,7 @@ func (mgr *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) error { return err } - // Evict all keys from cache. - for _, key := range keys { - mgr.cache.Remove(key) - } - + mgr.cache.Remove(p.Pretty()) // Attempt transactional KV deletion. for i := 0; i < mgr.writeRetries; i++ { if err = mgr.dbDelete(keys); err == nil { @@ -153,11 +149,7 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati return err } - // Evict all keys from cache before the update. - for _, key := range keys { - mgr.cache.Remove(key) - } - + mgr.cache.Remove(p.Pretty()) // Attempt transactional KV insertion. var existed []bool for i := 0; i < mgr.writeRetries; i++ { @@ -173,12 +165,9 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati } // Successful. Update cache and broadcast event. - for i, key := range keys { - addr := addrs[i] - mgr.cache.Add(key, addr.Bytes()) - + for i, _ := range keys { if !existed[i] { - mgr.subsManager.BroadcastAddr(p, addr) + mgr.subsManager.BroadcastAddr(p, addrs[i]) } } @@ -263,11 +252,19 @@ func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time. func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { var ( prefix = ds.NewKey(p.Pretty()) - q = query.Query{Prefix: prefix.String(), KeysOnly: true} + q = query.Query{Prefix: prefix.String(), KeysOnly: false} results query.Results err error ) + // Check the cache. + if entry, ok := mgr.cache.Get(p.Pretty()); ok { + e := entry.([]ma.Multiaddr) + addrs := make([]ma.Multiaddr, len(e)) + copy(addrs, e) + return addrs + } + txn := mgr.ds.NewTransaction(true) defer txn.Discard() @@ -275,27 +272,20 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { log.Error(err) return nil } - defer results.Close() var addrs []ma.Multiaddr for result := range results.Next() { - key := ds.RawKey(result.Key) - var addri interface{} - addri, ok := mgr.cache.Get(key) - - if !ok { - if addri, err = txn.Get(key); err != nil { - log.Error(err) - continue - } - } - - if addr, err := ma.NewMultiaddrBytes(addri.([]byte)); err != nil { + if addr, err := ma.NewMultiaddrBytes(result.Value); err == nil { addrs = append(addrs, addr) } } + // Store a copy in the cache. + addrsCpy := make([]ma.Multiaddr, len(addrs)) + copy(addrsCpy, addrs) + mgr.cache.Add(p.Pretty(), addrsCpy) + return addrs } @@ -346,13 +336,13 @@ func (mgr *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mult func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { var ( err error - keys []ds.Key prefix = ds.NewKey(p.Pretty()) ) + mgr.cache.Remove(p.Pretty()) // Attempt transactional KV deletion. for i := 0; i < mgr.writeRetries; i++ { - if keys, err = mgr.dbClear(prefix); err == nil { + if _, err = mgr.dbClear(prefix); err == nil { break } log.Errorf("failed to clear addresses for peer %s: %s\n", p.Pretty(), err) @@ -365,11 +355,10 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { // Perform housekeeping. mgr.ttlManager.clear(prefix) - for _, key := range keys { - mgr.cache.Remove(key) - } } +// dbClear removes all entries whose keys are prefixed with the argument. +// it returns a slice of the removed keys in case it's needed func (mgr *dsAddrBook) dbClear(prefix ds.Key) ([]ds.Key, error) { q := query.Query{Prefix: prefix.String(), KeysOnly: true} @@ -478,7 +467,7 @@ func (mgr *ttlManager) tick() { log.Error("failed to delete TTL key: %v, cause: %v", key.String(), err) break } - mgr.cache.Remove(key) + mgr.cache.Remove(key.Parent().Name()) delete(mgr.entries, key) } From de70a7544fe7df8a72798caa9611d2ef19c0a1af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 11 Sep 2018 12:43:36 +0100 Subject: [PATCH 0672/3965] check cache in ClearAddrs and perform targeted delete. --- p2p/host/peerstore/pstoreds/addr_book.go | 70 ++++++++++++--------- p2p/host/peerstore/test/benchmarks_suite.go | 13 ++-- 2 files changed, 48 insertions(+), 35 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 147e953961..5d87873a27 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -80,7 +80,6 @@ func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, er if err != nil { return nil, nil, err } - keys[i] = parentKey.ChildString(hash.B58String()) clean[i] = addr i++ @@ -182,28 +181,6 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati return nil } -// dbDelete performs a transactional delete of the provided keys. -func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { - var err error - - txn := mgr.ds.NewTransaction(false) - defer txn.Discard() - - // Attempt to delete all keys. - for _, key := range keys { - if err = txn.Delete(key); err != nil { - log.Errorf("transaction failed and aborted while deleting key: %s, cause: %v", key.String(), err) - return err - } - } - - if err = txn.Commit(); err != nil { - log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) - return err - } - return nil -} - // dbInsert performs a transactional insert of the provided keys and values. func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr) ([]bool, error) { var ( @@ -335,14 +312,28 @@ func (mgr *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mult // ClearAddrs will delete all known addresses for a peer ID. func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { var ( - err error - prefix = ds.NewKey(p.Pretty()) + err error + prefix = ds.NewKey(p.Pretty()) + deleteFn func() error ) - mgr.cache.Remove(p.Pretty()) + if e, ok := mgr.cache.Peek(p.Pretty()); ok { + mgr.cache.Remove(p.Pretty()) + keys, _, _ := keysAndAddrs(p, e.([]ma.Multiaddr)) + + deleteFn = func() error { + return mgr.dbDelete(keys) + } + } else { + deleteFn = func() error { + _, err := mgr.dbClearIterator(prefix) + return err + } + } + // Attempt transactional KV deletion. for i := 0; i < mgr.writeRetries; i++ { - if _, err = mgr.dbClear(prefix); err == nil { + if err = deleteFn(); err == nil { break } log.Errorf("failed to clear addresses for peer %s: %s\n", p.Pretty(), err) @@ -350,16 +341,35 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { if err != nil { log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), mgr.writeRetries) - // TODO: return error } // Perform housekeeping. mgr.ttlManager.clear(prefix) } -// dbClear removes all entries whose keys are prefixed with the argument. +// dbDelete transactionally deletes the provided keys. +func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { + txn := mgr.ds.NewTransaction(false) + defer txn.Discard() + + for _, key := range keys { + if err := txn.Delete(key); err != nil { + log.Errorf("failed to delete key: %s, cause: %v", key.String(), err) + return err + } + } + + if err := txn.Commit(); err != nil { + log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) + return err + } + + return nil +} + +// dbClearIterator removes all entries whose keys are prefixed with the argument. // it returns a slice of the removed keys in case it's needed -func (mgr *dsAddrBook) dbClear(prefix ds.Key) ([]ds.Key, error) { +func (mgr *dsAddrBook) dbClearIterator(prefix ds.Key) ([]ds.Key, error) { q := query.Query{Prefix: prefix.String(), KeysOnly: true} txn := mgr.ds.NewTransaction(false) diff --git a/p2p/host/peerstore/test/benchmarks_suite.go b/p2p/host/peerstore/test/benchmarks_suite.go index 61c611eb82..69fde2d2a4 100644 --- a/p2p/host/peerstore/test/benchmarks_suite.go +++ b/p2p/host/peerstore/test/benchmarks_suite.go @@ -9,10 +9,12 @@ import ( ) var peerstoreBenchmarks = map[string]func(pstore.Peerstore, chan *peerpair) func(*testing.B){ - "AddAddrs": benchmarkAddAddrs, - "SetAddrs": benchmarkSetAddrs, - "GetAddrs": benchmarkGetAddrs, - "AddAndClearAddrs": benchmarkAddAndClearAddrs, + "AddAddrs": benchmarkAddAddrs, + "SetAddrs": benchmarkSetAddrs, + "GetAddrs": benchmarkGetAddrs, + // The in-between get allows us to benchmark the read-through cache. + "AddGetAndClearAddrs": benchmarkAddGetAndClearAddrs, + // Calls PeersWithAddr on a peerstore with 1000 peers. "Get1000PeersWithAddrs": benchmarkGet1000PeersWithAddrs, } @@ -84,12 +86,13 @@ func benchmarkGetAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing. } } -func benchmarkAddAndClearAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { +func benchmarkAddGetAndClearAddrs(ps pstore.Peerstore, addrs chan *peerpair) func(*testing.B) { return func(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { pp := <-addrs ps.AddAddrs(pp.ID, pp.Addr, pstore.PermanentAddrTTL) + ps.Addrs(pp.ID) ps.ClearAddrs(pp.ID) } } From 6b61869f76bcc54ddf680688ac99f8348ccb2c1f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 11 Sep 2018 14:14:00 +0100 Subject: [PATCH 0673/3965] fix bug in PeersWithAddrs(). --- p2p/host/peerstore/pstoreds/addr_book.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 6dddcf7f38..8f2da2b757 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -294,7 +294,7 @@ func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { return peer.IDSlice{} } - ids := make(peer.IDSlice, 1, len(idset)) + ids := make(peer.IDSlice, 0, len(idset)) for id := range idset { i, _ := peer.IDB58Decode(id) ids = append(ids, i) From 22da14178a234242b39c135e384ea91a16984911 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 11 Sep 2018 14:34:26 +0100 Subject: [PATCH 0674/3965] small fixes. --- p2p/host/peerstore/pstoreds/addr_book.go | 15 ++++++++------- p2p/host/peerstore/test/utils.go | 10 +--------- 2 files changed, 9 insertions(+), 16 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 8f2da2b757..53c3006b03 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -294,10 +294,11 @@ func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { return peer.IDSlice{} } - ids := make(peer.IDSlice, 0, len(idset)) + ids, i := make(peer.IDSlice, len(idset)), 0 for id := range idset { - i, _ := peer.IDB58Decode(id) - ids = append(ids, i) + pid, _ := peer.IDB58Decode(id) + ids[i] = pid + i++ } return ids } @@ -326,7 +327,7 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { } } else { deleteFn = func() error { - _, err := mgr.dbClearIterator(prefix) + _, err := mgr.dbDeleteIter(prefix) return err } } @@ -367,9 +368,9 @@ func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { return nil } -// dbClearIterator removes all entries whose keys are prefixed with the argument. +// dbDeleteIter removes all entries whose keys are prefixed with the argument. // it returns a slice of the removed keys in case it's needed -func (mgr *dsAddrBook) dbClearIterator(prefix ds.Key) ([]ds.Key, error) { +func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) ([]ds.Key, error) { q := query.Query{Prefix: prefix.String(), KeysOnly: true} txn := mgr.ds.NewTransaction(false) @@ -393,7 +394,7 @@ func (mgr *dsAddrBook) dbClearIterator(prefix ds.Key) ([]ds.Key, error) { } if err := results.Close(); err != nil { - log.Errorf("failed to close cursor: %s, cause: %v", err) + log.Errorf("failed to close cursor, cause: %v", err) return nil, err } diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index 8b7a86a9f0..6649820785 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -5,19 +5,11 @@ import ( "fmt" "testing" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" pt "github.com/libp2p/go-libp2p-peer/test" ma "github.com/multiformats/go-multiaddr" ) -func peerId(ids string) peer.ID { - id, err := peer.IDB58Decode(ids) - if err != nil { - panic(err) - } - return id -} - func multiaddr(m string) ma.Multiaddr { maddr, err := ma.NewMultiaddr(m) if err != nil { From 78f457d97ddfeaedcebec0c7875f77fbacb02293 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 11 Sep 2018 18:10:27 +0100 Subject: [PATCH 0675/3965] increase test coverage. --- p2p/host/peerstore/pstoreds/addr_book.go | 1 - p2p/host/peerstore/pstoreds/ds_test.go | 21 +++++++-- p2p/host/peerstore/test/addr_book_suite.go | 55 ++++++++++++++++++++++ 3 files changed, 73 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 53c3006b03..9bb5a4572a 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -321,7 +321,6 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { if e, ok := mgr.cache.Peek(p.Pretty()); ok { mgr.cache.Remove(p.Pretty()) keys, _, _ := keysAndAddrs(p, e.([]ma.Multiaddr)) - deleteFn = func() error { return mgr.dbDelete(keys) } diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index f1972dc7de..4a8d04c8d9 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -122,10 +122,25 @@ func TestBadgerDsPeerstore(t *testing.T) { } func TestBadgerDsAddrBook(t *testing.T) { - opts := DefaultOpts() - opts.TTLInterval = 100 * time.Microsecond + t.Run("Cacheful", func(t *testing.T) { + t.Parallel() - pt.TestAddrBook(t, addressBookFactory(t, opts)) + opts := DefaultOpts() + opts.TTLInterval = 100 * time.Microsecond + opts.CacheSize = 1024 + + pt.TestAddrBook(t, addressBookFactory(t, opts)) + }) + + t.Run("Cacheless", func(t *testing.T) { + t.Parallel() + + opts := DefaultOpts() + opts.TTLInterval = 100 * time.Microsecond + opts.CacheSize = 0 + + pt.TestAddrBook(t, addressBookFactory(t, opts)) + }) } func BenchmarkBadgerDsPeerstore(b *testing.B) { diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index 441e2fa77f..dbff04a27f 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -18,6 +18,8 @@ var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ "UpdateTTLs": testUpdateTTLs, "NilAddrsDontBreak": testNilAddrsDontBreak, "AddressesExpire": testAddressesExpire, + "ClearWithIter": testClearWithIterator, + "PeersWithAddresses": testPeersWithAddrs, } type AddrBookFactory func() (pstore.AddrBook, func()) @@ -234,6 +236,59 @@ func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { } } +func testClearWithIterator(m pstore.AddrBook) func(t *testing.T) { + return func(t *testing.T) { + ids := generatePeerIds(2) + addrs := generateAddrs(100) + + // Add the peers with 50 addresses each. + m.AddAddrs(ids[0], addrs[:50], pstore.PermanentAddrTTL) + m.AddAddrs(ids[1], addrs[50:], pstore.PermanentAddrTTL) + + if all := append(m.Addrs(ids[0]), m.Addrs(ids[1])...); len(all) != 100 { + t.Fatal("expected pstore to contain both peers with all their maddrs") + } + + // Since we don't fetch these peers, they won't be present in cache. + + m.ClearAddrs(ids[0]) + if all := append(m.Addrs(ids[0]), m.Addrs(ids[1])...); len(all) != 50 { + t.Fatal("expected pstore to contain only addrs of peer 2") + } + + m.ClearAddrs(ids[1]) + if all := append(m.Addrs(ids[0]), m.Addrs(ids[1])...); len(all) != 0 { + t.Fatal("expected pstore to contain no addresses") + } + } +} + +func testPeersWithAddrs(m pstore.AddrBook) func(t *testing.T) { + return func(t *testing.T) { + // cannot run in parallel as the store is modified. + // go runs sequentially in the specified order + // see https://blog.golang.org/subtests + + t.Run("empty addrbook", func(t *testing.T) { + if peers := m.PeersWithAddrs(); len(peers) != 0 { + t.Fatal("expected to find no peers") + } + }) + + t.Run("non-empty addrbook", func(t *testing.T) { + ids := generatePeerIds(2) + addrs := generateAddrs(10) + + m.AddAddrs(ids[0], addrs[:5], pstore.PermanentAddrTTL) + m.AddAddrs(ids[1], addrs[5:], pstore.PermanentAddrTTL) + + if peers := m.PeersWithAddrs(); len(peers) != 2 { + t.Fatal("expected to find 2 peers") + } + }) + } +} + func testHas(t *testing.T, exp, act []ma.Multiaddr) { t.Helper() if len(exp) != len(act) { From 71e08d9f5d545bf4d289658d7c25ed85ec860408 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 12 Sep 2018 13:42:26 +0100 Subject: [PATCH 0676/3965] Resolve TODO. --- p2p/host/peerstore/pstoremem/addr_book.go | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 6db498ad2e..311c436ccb 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -162,10 +162,8 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t } exp := time.Now().Add(newTTL) - // TODO: RK - Shorthand. for i := range addrs { - aexp := &addrs[i] - if oldTTL == aexp.TTL { + if aexp := &addrs[i]; oldTTL == aexp.TTL { aexp.TTL = newTTL aexp.Expires = exp } From 202fdc25da95be21dd838ebd927cf43b3b6c41d7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 12 Sep 2018 13:44:14 +0100 Subject: [PATCH 0677/3965] address review comments. --- p2p/host/peerstore/pstoreds/addr_book.go | 11 ++++++----- p2p/host/peerstore/pstoreds/ds_test.go | 4 ++-- p2p/host/peerstore/pstoreds/peerstore.go | 11 ++++++----- 3 files changed, 14 insertions(+), 12 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 9bb5a4572a..f338cb449c 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -36,7 +36,7 @@ type dsAddrBook struct { // NewAddrBook initializes a new address book given a // Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. -func NewAddrBook(ctx context.Context, ds ds.TxnDatastore, opts PeerstoreOpts) (*dsAddrBook, error) { +func NewAddrBook(ctx context.Context, ds ds.TxnDatastore, opts Options) (*dsAddrBook, error) { var ( cache cache = &noopCache{} err error @@ -163,7 +163,7 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati return err } - // Successful. Update cache and broadcast event. + // Update was successful, so broadcast event only for new addresses. for i, _ := range keys { if !existed[i] { mgr.subsManager.BroadcastAddr(p, addrs[i]) @@ -175,7 +175,7 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati if ttlReset { mgr.ttlManager.setTTLs(keys, ttl) } else { - mgr.ttlManager.insertTTLs(keys, ttl) + mgr.ttlManager.insertOrExtendTTLs(keys, ttl) } return nil @@ -294,7 +294,8 @@ func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { return peer.IDSlice{} } - ids, i := make(peer.IDSlice, len(idset)), 0 + ids := make(peer.IDSlice, len(idset)) + i := 0 for id := range idset { pid, _ := peer.IDB58Decode(id) ids[i] = pid @@ -495,7 +496,7 @@ func (mgr *ttlManager) deleteTTLs(keys []ds.Key) { } } -func (mgr *ttlManager) insertTTLs(keys []ds.Key, ttl time.Duration) { +func (mgr *ttlManager) insertOrExtendTTLs(keys []ds.Key, ttl time.Duration) { mgr.Lock() defer mgr.Unlock() diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 4a8d04c8d9..20bef6a9c3 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -170,7 +170,7 @@ func badgerStore(t testing.TB) (ds.TxnDatastore, func()) { return ds, closer } -func peerstoreFactory(tb testing.TB, opts PeerstoreOpts) pt.PeerstoreFactory { +func peerstoreFactory(tb testing.TB, opts Options) pt.PeerstoreFactory { return func() (pstore.Peerstore, func()) { ds, closeFunc := badgerStore(tb) @@ -183,7 +183,7 @@ func peerstoreFactory(tb testing.TB, opts PeerstoreOpts) pt.PeerstoreFactory { } } -func addressBookFactory(tb testing.TB, opts PeerstoreOpts) pt.AddrBookFactory { +func addressBookFactory(tb testing.TB, opts Options) pt.AddrBookFactory { return func() (pstore.AddrBook, func()) { ds, closeDB := badgerStore(tb) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 74576d0532..a3b4d3f174 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -11,11 +11,12 @@ import ( ) // Configuration object for the peerstore. -type PeerstoreOpts struct { +type Options struct { // The size of the in-memory cache. A value of 0 or lower disables the cache. CacheSize uint - // Sweep interval to expire entries when TTL is not managed by underlying datastore. + // Sweep interval to expire entries, only used when TTL is *not* natively managed + // by the underlying datastore. TTLInterval time.Duration // Number of times to retry transactional writes. @@ -26,8 +27,8 @@ type PeerstoreOpts struct { // * Cache size: 1024 // * TTL sweep interval: 1 second // * WriteRetries: 5 -func DefaultOpts() PeerstoreOpts { - return PeerstoreOpts{ +func DefaultOpts() Options { + return Options{ CacheSize: 1024, TTLInterval: time.Second, WriteRetries: 5, @@ -35,7 +36,7 @@ func DefaultOpts() PeerstoreOpts { } // NewPeerstore creates a peerstore backed by the provided persistent datastore. -func NewPeerstore(ctx context.Context, store ds.TxnDatastore, opts PeerstoreOpts) (pstore.Peerstore, error) { +func NewPeerstore(ctx context.Context, store ds.TxnDatastore, opts Options) (pstore.Peerstore, error) { addrBook, err := NewAddrBook(ctx, store, opts) if err != nil { return nil, err From 4433afc8d419bd95ab9233ff88a59aadde145b17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 12 Sep 2018 13:43:08 +0100 Subject: [PATCH 0678/3965] hopefully fix intermittent time-dependent test failure. --- p2p/host/peerstore/test/addr_book_suite.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index dbff04a27f..b103aecc7c 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -154,25 +154,25 @@ func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { testHas(t, addrs2, m.Addrs(ids[1])) // Will only affect addrs1[0]. - m.UpdateAddrs(ids[0], time.Hour, time.Second) + m.UpdateAddrs(ids[0], time.Hour, 100*time.Microsecond) // No immediate effect. testHas(t, addrs1, m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) // After a wait, addrs[0] is gone. - time.Sleep(1200 * time.Millisecond) + time.Sleep(100 * time.Millisecond) testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) // Will only affect addrs2[0]. - m.UpdateAddrs(ids[1], time.Hour, time.Second) + m.UpdateAddrs(ids[1], time.Hour, 100*time.Microsecond) // No immediate effect. testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) - time.Sleep(1200 * time.Millisecond) + time.Sleep(100 * time.Millisecond) // First addrs is gone in both. testHas(t, addrs1[1:], m.Addrs(ids[0])) From b84518b6c9e247f5d1724582d9244f917bbfbde1 Mon Sep 17 00:00:00 2001 From: gukq Date: Thu, 13 Sep 2018 01:02:07 +0800 Subject: [PATCH 0679/3965] Just repair spelling mistake Signed-off-by: gukq --- defaults.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/defaults.go b/defaults.go index ba9bdc4063..f19e5ab08c 100644 --- a/defaults.go +++ b/defaults.go @@ -32,7 +32,7 @@ var DefaultMuxers = ChainOptions( // DefaultTransports are the default libp2p transports. // -// Use this option when you want to *extend* the set of multiplexers used by +// Use this option when you want to *extend* the set of transports used by // libp2p instead of replacing them. var DefaultTransports = ChainOptions( Transport(tcp.NewTCPTransport), From e6b1e07f11df5f373b2508743213c13cbf317003 Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Thu, 13 Sep 2018 11:38:16 +0800 Subject: [PATCH 0680/3965] Updates due to code review --- p2p/host/basic/basic_host.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 0265819e48..5379bed4b2 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -474,10 +474,11 @@ func mergeAddrs(addrLists ...[]ma.Multiaddr) (uniqueAddrs []ma.Multiaddr) { exists := make(map[string]bool) for _, addrList := range addrLists { for _, addr := range addrList { - if exists[addr.String()] { + k := string(addr.Bytes()) + if exists[k] { continue } - exists[addr.String()] = true + exists[k] = true uniqueAddrs = append(uniqueAddrs, addr) } } From 66a3ad4c648a674f402de64e127780bff064126d Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Thu, 13 Sep 2018 15:04:26 +0800 Subject: [PATCH 0681/3965] Fix test --- _examples/nat-tester.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_examples/nat-tester.go b/_examples/nat-tester.go index f2f3bc9afc..e3b616b2a5 100644 --- a/_examples/nat-tester.go +++ b/_examples/nat-tester.go @@ -6,7 +6,7 @@ import ( "net/http" "time" - "github.com/fd/go-nat" + "github.com/libp2p/go-nat" ) func main() { @@ -52,7 +52,7 @@ func main() { } }() - defer nat.DeletePortMapping("txp", 3080) + defer nat.DeletePortMapping("tcp", 3080) http.ListenAndServe(":3080", http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) { rw.Header().Set("Content-Type", "text/plain") From 3fa58ced20b2ff6110ae7eca6c22e18c06c8da6f Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Thu, 13 Sep 2018 15:11:48 +0800 Subject: [PATCH 0682/3965] Try to map external port the same as internal port --- natpmp.go | 7 +++++++ upnp.go | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/natpmp.go b/natpmp.go index 395a5dd578..9a2c27d9e7 100644 --- a/natpmp.go +++ b/natpmp.go @@ -93,6 +93,13 @@ func (n *natpmpNAT) AddPortMapping(protocol string, internalPort int, descriptio } } + // try to map external port the same as internal port + _, err = n.c.AddPortMapping(protocol, internalPort, internalPort, timeoutInSeconds) + if err == nil { + n.ports[internalPort] = internalPort + return internalPort, nil + } + for i := 0; i < 3; i++ { externalPort := randomPort() _, err = n.c.AddPortMapping(protocol, internalPort, externalPort, timeoutInSeconds) diff --git a/upnp.go b/upnp.go index 86d6e9b3ba..8b0975783b 100644 --- a/upnp.go +++ b/upnp.go @@ -176,6 +176,12 @@ func (u *upnp_NAT) AddPortMapping(protocol string, internalPort int, description } } + // try to map external port the same as internal port + err = u.c.AddPortMapping("", uint16(internalPort), mapProtocol(protocol), uint16(internalPort), ip.String(), true, description, timeoutInSeconds) + if err == nil { + return internalPort, nil + } + for i := 0; i < 3; i++ { externalPort := randomPort() err = u.c.AddPortMapping("", uint16(externalPort), mapProtocol(protocol), uint16(internalPort), ip.String(), true, description, timeoutInSeconds) From b9b39391e592df70eafaa287dafbb952760ddf65 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 13 Sep 2018 10:35:03 +0100 Subject: [PATCH 0683/3965] attempt to fix time-dependent tests in CI. --- p2p/host/peerstore/test/addr_book_suite.go | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index b103aecc7c..d78e7ccd34 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -209,28 +209,28 @@ func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { testHas(t, addrs1, m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) - m.SetAddr(ids[0], addrs1[0], time.Millisecond) - <-time.After(time.Millisecond * 5) + m.SetAddr(ids[0], addrs1[0], 100*time.Microsecond) + <-time.After(100 * time.Millisecond) testHas(t, addrs1[1:3], m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) - m.SetAddr(ids[0], addrs1[2], time.Millisecond) - <-time.After(time.Millisecond * 5) + m.SetAddr(ids[0], addrs1[2], 100*time.Microsecond) + <-time.After(100 * time.Millisecond) testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) - m.SetAddr(ids[1], addrs2[0], time.Millisecond) - <-time.After(time.Millisecond * 5) + m.SetAddr(ids[1], addrs2[0], 100*time.Microsecond) + <-time.After(100 * time.Millisecond) testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, addrs2[1:], m.Addrs(ids[1])) - m.SetAddr(ids[1], addrs2[1], time.Millisecond) - <-time.After(time.Millisecond * 5) + m.SetAddr(ids[1], addrs2[1], 100*time.Microsecond) + <-time.After(100 * time.Millisecond) testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, nil, m.Addrs(ids[1])) - m.SetAddr(ids[0], addrs1[1], time.Millisecond) - <-time.After(time.Millisecond * 5) + m.SetAddr(ids[0], addrs1[1], 100*time.Microsecond) + <-time.After(100 * time.Millisecond) testHas(t, nil, m.Addrs(ids[0])) testHas(t, nil, m.Addrs(ids[1])) } From 80894bed8d5a2404828a3e059445f00f540c6176 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 12 Sep 2018 20:40:51 +0100 Subject: [PATCH 0684/3965] migrate from bespoke TTL manager to db-managed TTLs. --- p2p/host/peerstore/pstoreds/addr_book.go | 183 ++++++++++++++++----- p2p/host/peerstore/test/addr_book_suite.go | 4 +- 2 files changed, 142 insertions(+), 45 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 9bb5a4572a..7ee3de91b5 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -1,24 +1,33 @@ package pstoreds import ( + "bytes" "context" + "encoding/binary" + "errors" "sync" "time" - lru "github.com/hashicorp/golang-lru" + "github.com/hashicorp/golang-lru" ds "github.com/ipfs/go-datastore" - query "github.com/ipfs/go-datastore/query" + "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" mh "github.com/multiformats/go-multihash" pstore "github.com/libp2p/go-libp2p-peerstore" - pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" + "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) var ( log = logging.Logger("peerstore/ds") + // The maximum representable value in time.Time is time.Unix(1<<63-62135596801, 999999999). + // But it's too brittle and implementation-dependent, so we prefer to use 1<<62, which is in the + // year 146138514283. We're safe. + maxTime = time.Unix(1<<62, 0) + + ErrTTLDatastore = errors.New("datastore must provide TTL support") ) var _ pstore.AddrBook = (*dsAddrBook)(nil) @@ -28,15 +37,30 @@ var _ pstore.AddrBook = (*dsAddrBook)(nil) type dsAddrBook struct { cache cache ds ds.TxnDatastore - ttlManager *ttlManager subsManager *pstoremem.AddrSubManager writeRetries int } +type ttlWriteMode int + +const ( + ttlOverride ttlWriteMode = iota + ttlExtend +) + +type cacheEntry struct { + expiration time.Time + addrs []ma.Multiaddr +} + // NewAddrBook initializes a new address book given a // Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. -func NewAddrBook(ctx context.Context, ds ds.TxnDatastore, opts PeerstoreOpts) (*dsAddrBook, error) { +func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts PeerstoreOpts) (*dsAddrBook, error) { + if _, ok := store.(ds.TTLDatastore); !ok { + return nil, ErrTTLDatastore + } + var ( cache cache = &noopCache{} err error @@ -50,8 +74,7 @@ func NewAddrBook(ctx context.Context, ds ds.TxnDatastore, opts PeerstoreOpts) (* mgr := &dsAddrBook{ cache: cache, - ds: ds, - ttlManager: newTTLManager(ctx, ds, &cache, opts.TTLInterval), + ds: store, subsManager: pstoremem.NewAddrSubManager(), writeRetries: int(opts.WriteRetries), } @@ -60,7 +83,7 @@ func NewAddrBook(ctx context.Context, ds ds.TxnDatastore, opts PeerstoreOpts) (* // Stop will signal the TTL manager to stop and block until it returns. func (mgr *dsAddrBook) Stop() { - mgr.ttlManager.cancel() + // noop } func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, error) { @@ -98,7 +121,7 @@ func (mgr *dsAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati if ttl <= 0 { return } - mgr.setAddrs(p, addrs, ttl, false) + mgr.setAddrs(p, addrs, ttl, ttlExtend) } // SetAddr will add or update the TTL of an address in the AddrBook. @@ -113,7 +136,7 @@ func (mgr *dsAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati mgr.deleteAddrs(p, addrs) return } - mgr.setAddrs(p, addrs, ttl, true) + mgr.setAddrs(p, addrs, ttl, ttlOverride) } func (mgr *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) error { @@ -137,11 +160,10 @@ func (mgr *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) error { return err } - mgr.ttlManager.deleteTTLs(keys) return nil } -func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, ttlReset bool) error { +func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode) error { // Keys and cleaned up addresses. keys, addrs, err := keysAndAddrs(p, addrs) if err != nil { @@ -149,10 +171,11 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati } mgr.cache.Remove(p.Pretty()) + // Attempt transactional KV insertion. var existed []bool for i := 0; i < mgr.writeRetries; i++ { - if existed, err = mgr.dbInsert(keys, addrs); err == nil { + if existed, err = mgr.dbInsert(keys, addrs, ttl, mode); err == nil { break } log.Errorf("failed to write addresses for peer %s: %s\n", p.Pretty(), err) @@ -170,41 +193,54 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati } } - // Force update TTLs only if TTL reset was requested; otherwise - // insert the appropriate TTL entries if they don't already exist. - if ttlReset { - mgr.ttlManager.setTTLs(keys, ttl) - } else { - mgr.ttlManager.insertTTLs(keys, ttl) - } - return nil } // dbInsert performs a transactional insert of the provided keys and values. -func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr) ([]bool, error) { +func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode) ([]bool, error) { var ( err error existed = make([]bool, len(keys)) + exp = time.Now().Add(ttl) + ttlB = make([]byte, 8) ) + binary.LittleEndian.PutUint64(ttlB, uint64(ttl)) + txn := mgr.ds.NewTransaction(false) defer txn.Discard() + ttltxn := txn.(ds.TTLDatastore) for i, key := range keys { // Check if the key existed previously. - if existed[i], err = txn.Has(key); err != nil { + if existed[i], err = ttltxn.Has(key); err != nil { log.Errorf("transaction failed and aborted while checking key existence: %s, cause: %v", key.String(), err) return nil, err } - // The key embeds a hash of the value, so if it existed, we can safely skip the insert. + // The key embeds a hash of the value, so if it existed, we can safely skip the insert and + // just update the TTL. if existed[i] { + switch mode { + case ttlOverride: + err = ttltxn.SetTTL(key, ttl) + case ttlExtend: + var curr time.Time + if curr, err = ttltxn.GetExpiration(key); err != nil && exp.After(curr) { + err = ttltxn.SetTTL(key, ttl) + } + } + if err != nil { + // mode will be printed as an int + log.Errorf("failed while updating the ttl for key: %s, mode: %v, cause: %v", key.String(), mode, err) + return nil, err + } continue } - // Attempt to add the key. - if err = txn.Put(key, addrs[i].Bytes()); err != nil { + // format: bytes(ttl) || bytes(multiaddr) + value := append(ttlB, addrs[i].Bytes()...) + if err = ttltxn.PutWithTTL(key, value, ttl); err != nil { log.Errorf("transaction failed and aborted while setting key: %s, cause: %v", key.String(), err) return nil, err } @@ -221,25 +257,81 @@ func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr) ([]bool, er // UpdateAddrs will update any addresses for a given peer and TTL combination to // have a new TTL. func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { - prefix := ds.NewKey(p.Pretty()) - mgr.ttlManager.adjustTTLs(prefix, oldTTL, newTTL) + mgr.cache.Remove(p.Pretty()) + + var err error + for i := 0; i < mgr.writeRetries; i++ { + if err = mgr.dbUpdateTTL(p, oldTTL, newTTL); err == nil { + break + } + log.Errorf("failed to update ttlsfor peer %s: %s\n", p.Pretty(), err) + } + + if err != nil { + log.Errorf("failed to avoid write conflict when updating ttls for peer %s after %d retries: %v\n", + p.Pretty(), mgr.writeRetries, err) + } +} + +func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time.Duration) error { + var ( + prefix = ds.NewKey(p.Pretty()) + q = query.Query{Prefix: prefix.String(), KeysOnly: false} + oldb, newb = make([]byte, 8), make([]byte, 8) + results query.Results + err error + ) + + binary.LittleEndian.PutUint64(oldb, uint64(oldTTL)) + binary.LittleEndian.PutUint64(newb, uint64(newTTL)) + + txn := mgr.ds.NewTransaction(false) + defer txn.Discard() + + if results, err = txn.Query(q); err != nil { + return err + } + defer results.Close() + + ttltxn := txn.(ds.TTLDatastore) + for result := range results.Next() { + // format: bytes(ttl) || bytes(multiaddr) + if curr := result.Value[:8]; !bytes.Equal(curr, oldb) { + continue + } + newVal := append(newb, result.Value[8:]...) + if err = ttltxn.PutWithTTL(ds.RawKey(result.Key), newVal, newTTL); err != nil { + return err + } + } + + if err := txn.Commit(); err != nil { + log.Errorf("failed to commit transaction when updating ttls, cause: %v", err) + return err + } + + return nil } // Addrs returns all of the non-expired addresses for a given peer. func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { var ( prefix = ds.NewKey(p.Pretty()) - q = query.Query{Prefix: prefix.String(), KeysOnly: false} + q = query.Query{Prefix: prefix.String(), KeysOnly: false, ReturnExpirations: true} results query.Results err error ) - // Check the cache. - if entry, ok := mgr.cache.Get(p.Pretty()); ok { - e := entry.([]ma.Multiaddr) - addrs := make([]ma.Multiaddr, len(e)) - copy(addrs, e) - return addrs + // Check the cache and return the entry only if it hasn't expired; if expired, remove. + if e, ok := mgr.cache.Get(p.Pretty()); ok { + entry := e.(cacheEntry) + if entry.expiration.After(time.Now()) { + addrs := make([]ma.Multiaddr, len(entry.addrs)) + copy(addrs, entry.addrs) + return addrs + } else { + mgr.cache.Remove(p.Pretty()) + } } txn := mgr.ds.NewTransaction(true) @@ -252,16 +344,24 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { defer results.Close() var addrs []ma.Multiaddr + // used to set the expiration for the entire cache entry + earliestExp := maxTime for result := range results.Next() { - if addr, err := ma.NewMultiaddrBytes(result.Value); err == nil { + // extract multiaddr from value: bytes(ttl) || bytes(multiaddr) + if addr, err := ma.NewMultiaddrBytes(result.Value[8:]); err == nil { addrs = append(addrs, addr) } + + if exp := result.Expiration; !exp.IsZero() && exp.Before(earliestExp) { + earliestExp = exp + } } // Store a copy in the cache. addrsCpy := make([]ma.Multiaddr, len(addrs)) copy(addrsCpy, addrs) - mgr.cache.Add(p.Pretty(), addrsCpy) + entry := cacheEntry{addrs: addrsCpy, expiration: earliestExp} + mgr.cache.Add(p.Pretty(), entry) return addrs } @@ -320,7 +420,7 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { if e, ok := mgr.cache.Peek(p.Pretty()); ok { mgr.cache.Remove(p.Pretty()) - keys, _, _ := keysAndAddrs(p, e.([]ma.Multiaddr)) + keys, _, _ := keysAndAddrs(p, e.(cacheEntry).addrs) deleteFn = func() error { return mgr.dbDelete(keys) } @@ -342,9 +442,6 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { if err != nil { log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), mgr.writeRetries) } - - // Perform housekeeping. - mgr.ttlManager.clear(prefix) } // dbDelete transactionally deletes the provided keys. @@ -495,7 +592,7 @@ func (mgr *ttlManager) deleteTTLs(keys []ds.Key) { } } -func (mgr *ttlManager) insertTTLs(keys []ds.Key, ttl time.Duration) { +func (mgr *ttlManager) bumpTTLs(keys []ds.Key, ttl time.Duration) { mgr.Lock() defer mgr.Unlock() diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index dbff04a27f..c80f3f0bff 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -161,7 +161,7 @@ func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { testHas(t, addrs2, m.Addrs(ids[1])) // After a wait, addrs[0] is gone. - time.Sleep(1200 * time.Millisecond) + time.Sleep(3000 * time.Millisecond) testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) @@ -172,7 +172,7 @@ func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) - time.Sleep(1200 * time.Millisecond) + time.Sleep(3000 * time.Millisecond) // First addrs is gone in both. testHas(t, addrs1[1:], m.Addrs(ids[0])) From db643f3805af8762e37f1026581e2e7b14712cc0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 13 Sep 2018 12:48:34 +0100 Subject: [PATCH 0685/3965] remove deprecated ttlManager. --- p2p/host/peerstore/pstoreds/addr_book.go | 139 ----------------------- 1 file changed, 139 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 7ee3de91b5..ef3309d983 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -5,7 +5,6 @@ import ( "context" "encoding/binary" "errors" - "sync" "time" "github.com/hashicorp/golang-lru" @@ -501,141 +500,3 @@ func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) ([]ds.Key, error) { return keys, nil } - -type ttlEntry struct { - TTL time.Duration - ExpiresAt time.Time -} - -type ttlManager struct { - sync.RWMutex - entries map[ds.Key]*ttlEntry - - ctx context.Context - cancel context.CancelFunc - ticker *time.Ticker - ds ds.TxnDatastore - cache cache -} - -func newTTLManager(parent context.Context, d ds.Datastore, c *cache, tick time.Duration) *ttlManager { - ctx, cancel := context.WithCancel(parent) - txnDs, ok := d.(ds.TxnDatastore) - if !ok { - panic("must construct ttlManager with transactional datastore") - } - mgr := &ttlManager{ - entries: make(map[ds.Key]*ttlEntry), - ctx: ctx, - cancel: cancel, - ticker: time.NewTicker(tick), - ds: txnDs, - cache: *c, - } - - go func() { - for { - select { - case <-mgr.ctx.Done(): - mgr.ticker.Stop() - return - case <-mgr.ticker.C: - mgr.tick() - } - } - }() - - return mgr -} - -// To be called by TTL manager's coroutine only. -func (mgr *ttlManager) tick() { - mgr.Lock() - defer mgr.Unlock() - - now := time.Now() - var toDel []ds.Key - for key, entry := range mgr.entries { - if entry.ExpiresAt.After(now) { - continue - } - toDel = append(toDel, key) - } - - if len(toDel) == 0 { - return - } - - txn := mgr.ds.NewTransaction(false) - defer txn.Discard() - - for _, key := range toDel { - if err := txn.Delete(key); err != nil { - log.Error("failed to delete TTL key: %v, cause: %v", key.String(), err) - break - } - mgr.cache.Remove(key.Parent().Name()) - delete(mgr.entries, key) - } - - if err := txn.Commit(); err != nil { - log.Error("failed to commit TTL deletion, cause: %v", err) - } -} - -func (mgr *ttlManager) deleteTTLs(keys []ds.Key) { - mgr.Lock() - defer mgr.Unlock() - - for _, key := range keys { - delete(mgr.entries, key) - } -} - -func (mgr *ttlManager) bumpTTLs(keys []ds.Key, ttl time.Duration) { - mgr.Lock() - defer mgr.Unlock() - - expiration := time.Now().Add(ttl) - for _, key := range keys { - if entry, ok := mgr.entries[key]; !ok || (ok && entry.ExpiresAt.Before(expiration)) { - mgr.entries[key] = &ttlEntry{TTL: ttl, ExpiresAt: expiration} - } - } -} - -func (mgr *ttlManager) setTTLs(keys []ds.Key, ttl time.Duration) { - mgr.Lock() - defer mgr.Unlock() - - expiration := time.Now().Add(ttl) - for _, key := range keys { - mgr.entries[key] = &ttlEntry{TTL: ttl, ExpiresAt: expiration} - } -} - -func (mgr *ttlManager) adjustTTLs(prefix ds.Key, oldTTL, newTTL time.Duration) { - mgr.Lock() - defer mgr.Unlock() - - now := time.Now() - var keys []ds.Key - for key, entry := range mgr.entries { - if key.IsDescendantOf(prefix) && entry.TTL == oldTTL { - keys = append(keys, key) - entry.TTL = newTTL - entry.ExpiresAt = now.Add(newTTL) - } - } -} - -func (mgr *ttlManager) clear(prefix ds.Key) { - mgr.Lock() - defer mgr.Unlock() - - for key := range mgr.entries { - if key.IsDescendantOf(prefix) { - delete(mgr.entries, key) - } - } -} From e38ee5c2a742d78d61a83de2b949d36a9d90caf9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 13 Sep 2018 12:49:29 +0100 Subject: [PATCH 0686/3965] performance optimisation round. --- p2p/host/peerstore/pstoreds/addr_book.go | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index ef3309d983..5da4d4d4f0 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -445,17 +445,19 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { // dbDelete transactionally deletes the provided keys. func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { + var err error + txn := mgr.ds.NewTransaction(false) defer txn.Discard() for _, key := range keys { - if err := txn.Delete(key); err != nil { + if err = txn.Delete(key); err != nil { log.Errorf("failed to delete key: %s, cause: %v", key.String(), err) return err } } - if err := txn.Commit(); err != nil { + if err = txn.Commit(); err != nil { log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) return err } @@ -477,9 +479,10 @@ func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) ([]ds.Key, error) { return nil, err } - var keys []ds.Key + var keys = make([]ds.Key, 0, 4) // cap: 4 to reduce allocs + var key ds.Key for result := range results.Next() { - key := ds.RawKey(result.Key) + key = ds.RawKey(result.Key) keys = append(keys, key) if err = txn.Delete(key); err != nil { @@ -488,7 +491,7 @@ func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) ([]ds.Key, error) { } } - if err := results.Close(); err != nil { + if err = results.Close(); err != nil { log.Errorf("failed to close cursor, cause: %v", err) return nil, err } From 18e0eebfcec8b6402dd144b96055963d7858ae36 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 13 Sep 2018 13:28:27 +0100 Subject: [PATCH 0687/3965] do not prettify cache keys (peer ids). --- p2p/host/peerstore/pstoreds/addr_book.go | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 5da4d4d4f0..c9a0cd798d 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -145,7 +145,7 @@ func (mgr *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) error { return err } - mgr.cache.Remove(p.Pretty()) + mgr.cache.Remove(p) // Attempt transactional KV deletion. for i := 0; i < mgr.writeRetries; i++ { if err = mgr.dbDelete(keys); err == nil { @@ -169,8 +169,7 @@ func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Durati return err } - mgr.cache.Remove(p.Pretty()) - + mgr.cache.Remove(p) // Attempt transactional KV insertion. var existed []bool for i := 0; i < mgr.writeRetries; i++ { @@ -256,7 +255,7 @@ func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr, ttl time.Du // UpdateAddrs will update any addresses for a given peer and TTL combination to // have a new TTL. func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { - mgr.cache.Remove(p.Pretty()) + mgr.cache.Remove(p) var err error for i := 0; i < mgr.writeRetries; i++ { @@ -322,14 +321,14 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { ) // Check the cache and return the entry only if it hasn't expired; if expired, remove. - if e, ok := mgr.cache.Get(p.Pretty()); ok { + if e, ok := mgr.cache.Get(p); ok { entry := e.(cacheEntry) if entry.expiration.After(time.Now()) { addrs := make([]ma.Multiaddr, len(entry.addrs)) copy(addrs, entry.addrs) return addrs } else { - mgr.cache.Remove(p.Pretty()) + mgr.cache.Remove(p) } } @@ -360,7 +359,7 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { addrsCpy := make([]ma.Multiaddr, len(addrs)) copy(addrsCpy, addrs) entry := cacheEntry{addrs: addrsCpy, expiration: earliestExp} - mgr.cache.Add(p.Pretty(), entry) + mgr.cache.Add(p, entry) return addrs } @@ -417,8 +416,8 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { deleteFn func() error ) - if e, ok := mgr.cache.Peek(p.Pretty()); ok { - mgr.cache.Remove(p.Pretty()) + if e, ok := mgr.cache.Peek(p); ok { + mgr.cache.Remove(p) keys, _, _ := keysAndAddrs(p, e.(cacheEntry).addrs) deleteFn = func() error { return mgr.dbDelete(keys) From 57ae658096cdb4cfe48007f875df62ce07fa7ac5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 13 Sep 2018 16:09:32 +0100 Subject: [PATCH 0688/3965] adjust ttl tests as badger doesn't support sub-second ttl. --- p2p/host/peerstore/test/addr_book_suite.go | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index d78e7ccd34..92bc1fa98c 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -154,25 +154,27 @@ func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { testHas(t, addrs2, m.Addrs(ids[1])) // Will only affect addrs1[0]. - m.UpdateAddrs(ids[0], time.Hour, 100*time.Microsecond) + // Badger does not support subsecond TTLs. + // https://github.com/dgraph-io/badger/issues/339 + m.UpdateAddrs(ids[0], time.Hour, 1*time.Second) // No immediate effect. testHas(t, addrs1, m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) // After a wait, addrs[0] is gone. - time.Sleep(100 * time.Millisecond) + time.Sleep(1500 * time.Millisecond) testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) // Will only affect addrs2[0]. - m.UpdateAddrs(ids[1], time.Hour, 100*time.Microsecond) + m.UpdateAddrs(ids[1], time.Hour, 1*time.Second) // No immediate effect. testHas(t, addrs1[1:2], m.Addrs(ids[0])) testHas(t, addrs2, m.Addrs(ids[1])) - time.Sleep(100 * time.Millisecond) + time.Sleep(1500 * time.Millisecond) // First addrs is gone in both. testHas(t, addrs1[1:], m.Addrs(ids[0])) From ad173aea32709fda183f8333eadcd2844cdf732f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 13 Sep 2018 16:09:47 +0100 Subject: [PATCH 0689/3965] reintroduce import prefixes. --- p2p/host/peerstore/pstoreds/addr_book.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index f8fafa8a4a..9946243a1c 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -7,16 +7,16 @@ import ( "errors" "time" - "github.com/hashicorp/golang-lru" + lru "github.com/hashicorp/golang-lru" ds "github.com/ipfs/go-datastore" - "github.com/ipfs/go-datastore/query" + query "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" mh "github.com/multiformats/go-multihash" pstore "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/pstoremem" + pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) var ( From 2ea753ece024d4f4d1dda059388966b545705e56 Mon Sep 17 00:00:00 2001 From: gukq Date: Fri, 14 Sep 2018 17:03:39 +0800 Subject: [PATCH 0690/3965] Just repair spelling mistake Signed-off-by: gukq --- p2p/net/mock/mock_peernet.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_peernet.go b/p2p/net/mock/mock_peernet.go index 42361ac34a..3c5710bf91 100644 --- a/p2p/net/mock/mock_peernet.go +++ b/p2p/net/mock/mock_peernet.go @@ -372,7 +372,7 @@ func (pn *peernet) Notify(f inet.Notifiee) { pn.notifmu.Unlock() } -// StopNotify unregisters Notifiee fromr receiving signals +// StopNotify unregisters Notifiee from receiving signals func (pn *peernet) StopNotify(f inet.Notifiee) { pn.notifmu.Lock() delete(pn.notifs, f) From e77452b5c53b196ad57a0158210e80a097169d4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 14 Sep 2018 15:21:53 +0100 Subject: [PATCH 0691/3965] remove unnecessary return values. --- p2p/host/peerstore/pstoreds/addr_book.go | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 9946243a1c..684007dac6 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -424,8 +424,7 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { } } else { deleteFn = func() error { - _, err := mgr.dbDeleteIter(prefix) - return err + return mgr.dbDeleteIter(prefix) } } @@ -466,7 +465,7 @@ func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { // dbDeleteIter removes all entries whose keys are prefixed with the argument. // it returns a slice of the removed keys in case it's needed -func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) ([]ds.Key, error) { +func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) error { q := query.Query{Prefix: prefix.String(), KeysOnly: true} txn := mgr.ds.NewTransaction(false) @@ -475,7 +474,7 @@ func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) ([]ds.Key, error) { results, err := txn.Query(q) if err != nil { log.Errorf("failed to fetch all keys prefixed with: %s, cause: %v", prefix.String(), err) - return nil, err + return err } var keys = make([]ds.Key, 0, 4) // cap: 4 to reduce allocs @@ -486,19 +485,19 @@ func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) ([]ds.Key, error) { if err = txn.Delete(key); err != nil { log.Errorf("failed to delete key: %s, cause: %v", key.String(), err) - return nil, err + return err } } if err = results.Close(); err != nil { log.Errorf("failed to close cursor, cause: %v", err) - return nil, err + return err } if err = txn.Commit(); err != nil { log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) - return nil, err + return err } - return keys, nil + return nil } From a6fcd7f9b86d4b7ca8b117badbe855908585a6fb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 14 Sep 2018 18:27:23 +0100 Subject: [PATCH 0692/3965] add a test and fix broken logic. --- p2p/host/peerstore/pstoreds/addr_book.go | 2 +- p2p/host/peerstore/test/addr_book_suite.go | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 684007dac6..d9d762c7db 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -223,7 +223,7 @@ func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr, ttl time.Du err = ttltxn.SetTTL(key, ttl) case ttlExtend: var curr time.Time - if curr, err = ttltxn.GetExpiration(key); err != nil && exp.After(curr) { + if curr, err = ttltxn.GetExpiration(key); err == nil && exp.After(curr) { err = ttltxn.SetTTL(key, ttl) } } diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index 92bc1fa98c..7cb4cf8deb 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -93,6 +93,20 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { testHas(t, addrs, ab.Addrs(id)) }) + + t.Run("adding an existing address with a later expiration extends its ttl", func(t *testing.T) { + id := generatePeerIds(1)[0] + addrs := generateAddrs(3) + + ab.AddAddrs(id, addrs, time.Second) + + // same address as before but with a higher TTL + ab.AddAddrs(id, addrs[2:], time.Hour) + + // after the initial TTL has expired, check that only the third address is present. + time.Sleep(1200 * time.Millisecond) + testHas(t, addrs[2:], ab.Addrs(id)) + }) } } From 2ae3cfd7ef9e91999bf9db309da58af7614ebb8b Mon Sep 17 00:00:00 2001 From: James Wetter Date: Mon, 17 Sep 2018 01:50:30 +0900 Subject: [PATCH 0693/3965] lift test coverage --- p2p/net/connmgr/connmgr_test.go | 229 ++++++++++++++++++++++++++++++++ 1 file changed, 229 insertions(+) diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index 8c30e9e5d9..cf11f208db 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -3,10 +3,12 @@ package connmgr import ( "context" "testing" + "time" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" tu "github.com/libp2p/go-testutil" + ma "github.com/multiformats/go-multiaddr" ) type tconn struct { @@ -24,6 +26,14 @@ func (c *tconn) RemotePeer() peer.ID { return c.peer } +func (c *tconn) RemoteMultiaddr() ma.Multiaddr { + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") + if err != nil { + panic("cannot create multiaddr") + } + return addr +} + func randConn(t *testing.T) inet.Conn { pid := tu.RandPeerIDFatal(t) return &tconn{peer: pid} @@ -65,3 +75,222 @@ func TestConnTrimming(t *testing.T) { t.Fatal("conn with bad tag should have gotten closed") } } + +func TestConnsToClose(t *testing.T) { + cm := NewConnManager(0, 10, 0) + conns := cm.getConnsToClose(context.Background()) + if conns != nil { + t.Fatal("expected no connections") + } + + cm = NewConnManager(10, 0, 0) + conns = cm.getConnsToClose(context.Background()) + if conns != nil { + t.Fatal("expected no connections") + } + + cm = NewConnManager(1, 1, 0) + conns = cm.getConnsToClose(context.Background()) + if conns != nil { + t.Fatal("expected no connections") + } + + cm = NewConnManager(1, 1, time.Duration(10*time.Minute)) + not := cm.Notifee() + for i := 0; i < 5; i++ { + conn := randConn(t) + not.Connected(nil, conn) + } + conns = cm.getConnsToClose(context.Background()) + if len(conns) != 0 { + t.Fatal("expected no connections") + } +} + +func TestGetTagInfo(t *testing.T) { + start := time.Now() + cm := NewConnManager(1, 1, time.Duration(10*time.Minute)) + not := cm.Notifee() + conn := randConn(t) + not.Connected(nil, conn) + end := time.Now() + + other := tu.RandPeerIDFatal(t) + tag := cm.GetTagInfo(other) + if tag != nil { + t.Fatal("expected no tag") + } + + tag = cm.GetTagInfo(conn.RemotePeer()) + if tag == nil { + t.Fatal("expected tag") + } + if tag.FirstSeen.Before(start) || tag.FirstSeen.After(end) { + t.Fatal("expected first seen time") + } + if tag.Value != 0 { + t.Fatal("expected zero value") + } + if len(tag.Tags) != 0 { + t.Fatal("expected no tags") + } + if len(tag.Conns) != 1 { + t.Fatal("expected one connection") + } + for s, tm := range tag.Conns { + if s != conn.RemoteMultiaddr().String() { + t.Fatal("unexpected multiaddr") + } + if tm.Before(start) || tm.After(end) { + t.Fatal("unexpected connection time") + } + } + + cm.TagPeer(conn.RemotePeer(), "tag", 5) + tag = cm.GetTagInfo(conn.RemotePeer()) + if tag == nil { + t.Fatal("expected tag") + } + if tag.FirstSeen.Before(start) || tag.FirstSeen.After(end) { + t.Fatal("expected first seen time") + } + if tag.Value != 5 { + t.Fatal("expected five value") + } + if len(tag.Tags) != 1 { + t.Fatal("expected no tags") + } + for tString, v := range tag.Tags { + if tString != "tag" || v != 5 { + t.Fatal("expected tag value") + } + } + if len(tag.Conns) != 1 { + t.Fatal("expected one connection") + } + for s, tm := range tag.Conns { + if s != conn.RemoteMultiaddr().String() { + t.Fatal("unexpected multiaddr") + } + if tm.Before(start) || tm.After(end) { + t.Fatal("unexpected connection time") + } + } +} + +func TestTagPeerNonExistant(t *testing.T) { + cm := NewConnManager(1, 1, time.Duration(10*time.Minute)) + + id := tu.RandPeerIDFatal(t) + cm.TagPeer(id, "test", 1) + + if len(cm.peers) != 0 { + t.Fatal("expected zero peers") + } +} + +func TestUntagPeer(t *testing.T) { + cm := NewConnManager(1, 1, time.Duration(10*time.Minute)) + not := cm.Notifee() + conn := randConn(t) + not.Connected(nil, conn) + rp := conn.RemotePeer() + cm.TagPeer(rp, "tag", 5) + cm.TagPeer(rp, "tag two", 5) + + id := tu.RandPeerIDFatal(t) + cm.UntagPeer(id, "test") + if len(cm.peers[rp].tags) != 2 { + t.Fatal("expected tags to be uneffected") + } + + cm.UntagPeer(conn.RemotePeer(), "test") + if len(cm.peers[rp].tags) != 2 { + t.Fatal("expected tags to be uneffected") + } + + cm.UntagPeer(conn.RemotePeer(), "tag") + if len(cm.peers[rp].tags) != 1 { + t.Fatal("expected tag to be removed") + } + if cm.peers[rp].value != 5 { + t.Fatal("expected aggreagte tag value to be 5") + } +} + +func TestGetInfo(t *testing.T) { + start := time.Now() + gp := time.Duration(10 * time.Minute) + cm := NewConnManager(1, 5, gp) + not := cm.Notifee() + conn := randConn(t) + not.Connected(nil, conn) + cm.TrimOpenConns(context.Background()) + end := time.Now() + + info := cm.GetInfo() + if info.HighWater != 5 { + t.Fatal("expected highwater to be 5") + } + if info.LowWater != 1 { + t.Fatal("expected highwater to be 1") + } + if info.LastTrim.Before(start) || info.LastTrim.After(end) { + t.Fatal("unexpected last trim time") + } + if info.GracePeriod != gp { + t.Fatal("unexpected grace period") + } + if info.ConnCount != 1 { + t.Fatal("unexpected number of connections") + } +} + +func TestDoubleConnection(t *testing.T) { + gp := time.Duration(10 * time.Minute) + cm := NewConnManager(1, 5, gp) + not := cm.Notifee() + conn := randConn(t) + not.Connected(nil, conn) + cm.TagPeer(conn.RemotePeer(), "foo", 10) + not.Connected(nil, conn) + if cm.connCount != 1 { + t.Fatal("unexpected number of connections") + } + if cm.peers[conn.RemotePeer()].value != 10 { + t.Fatal("unexpected peer value") + } +} + +func TestDisconnected(t *testing.T) { + gp := time.Duration(10 * time.Minute) + cm := NewConnManager(1, 5, gp) + not := cm.Notifee() + conn := randConn(t) + not.Connected(nil, conn) + cm.TagPeer(conn.RemotePeer(), "foo", 10) + + not.Disconnected(nil, randConn(t)) + if cm.connCount != 1 { + t.Fatal("unexpected number of connections") + } + if cm.peers[conn.RemotePeer()].value != 10 { + t.Fatal("unexpected peer value") + } + + not.Disconnected(nil, &tconn{peer: conn.RemotePeer()}) + if cm.connCount != 1 { + t.Fatal("unexpected number of connections") + } + if cm.peers[conn.RemotePeer()].value != 10 { + t.Fatal("unexpected peer value") + } + + not.Disconnected(nil, conn) + if cm.connCount != 0 { + t.Fatal("unexpected number of connections") + } + if len(cm.peers) != 0 { + t.Fatal("unexpected number of peers") + } +} From 2e2a793e30d87e75cb76f09c51fe14f1e8452caa Mon Sep 17 00:00:00 2001 From: James Wetter Date: Mon, 17 Sep 2018 01:56:46 +0900 Subject: [PATCH 0694/3965] add comments for tag functions to silence golint --- p2p/net/connmgr/connmgr.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 56ca27ec59..8e7d5aaafb 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -20,7 +20,7 @@ var log = logging.Logger("connmgr") // to trimming. Trims are automatically run on demand, only if the time from the // previous trim is higher than 10 seconds. Furthermore, trims can be explicitly // requested through the public interface of this struct (see TrimOpenConns). - +// // See configuration parameters in NewConnManager. type BasicConnMgr struct { highWater int @@ -129,6 +129,8 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { return closed } +// GetTagInfo is called to fetch the tag information associated with a given +// peer, nil is returned if p refers to an unknown peer. func (cm *BasicConnMgr) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { cm.lk.Lock() defer cm.lk.Unlock() @@ -155,6 +157,7 @@ func (cm *BasicConnMgr) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { return out } +// TagPeer is called to associate a string and integer with a given peer. func (cm *BasicConnMgr) TagPeer(p peer.ID, tag string, val int) { cm.lk.Lock() defer cm.lk.Unlock() @@ -170,6 +173,7 @@ func (cm *BasicConnMgr) TagPeer(p peer.ID, tag string, val int) { pi.tags[tag] = val } +// UntagPeer is called to disassociate a string and integer from a given peer. func (cm *BasicConnMgr) UntagPeer(p peer.ID, tag string) { cm.lk.Lock() defer cm.lk.Unlock() From 329756a8946a72449a3d3277865331ff52c0fd7b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 17 Sep 2018 13:51:20 -0700 Subject: [PATCH 0695/3965] gx import --- .gx/lastpubver | 1 + package.json | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) create mode 100644 .gx/lastpubver create mode 100644 package.json diff --git a/.gx/lastpubver b/.gx/lastpubver new file mode 100644 index 0000000000..79ffaca958 --- /dev/null +++ b/.gx/lastpubver @@ -0,0 +1 @@ +1.0.1: QmZ4g5TSk6EKQHrR61FgnE1ytJfirDHKvquL93a3Na1wJT diff --git a/package.json b/package.json new file mode 100644 index 0000000000..1155730974 --- /dev/null +++ b/package.json @@ -0,0 +1,35 @@ +{ + "author": "fd", + "bugs": { + "url": "https://github.com/issues/go-nat/issues" + }, + "gx": { + "dvcsimport": "github.com/libp2p/go-nat" + }, + "gxDependencies": [ + { + "author": "huin", + "hash": "QmVwfv63beSAAirq3tiLY6fkNqvjThLnCofL7363YkaScy", + "name": "goupnp", + "version": "0.1.0" + }, + { + "author": "jackpal", + "hash": "Qmf2fBLzCvFxs3vvZaoQyKSTv2rjApek4F1kzRxAfK6P4P", + "name": "gateway", + "version": "1.0.4" + }, + { + "author": "jackpal", + "hash": "QmYsYNh6saxUYHajdj49uiRzdxQgiFTtymrjf3d1f2Cer4", + "name": "go-nat-pmp", + "version": "1.0.1" + } + ], + "gxVersion": "0.9.0", + "language": "go", + "license": "Apache-2.0", + "name": "go-nat", + "version": "1.0.1" +} + From 78a6e612d20233659382cd253145d38d6d24cdd7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 17 Sep 2018 13:56:12 -0700 Subject: [PATCH 0696/3965] finish fd -> libp2p migration --- README.md | 4 +++- go.mod | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 24ddea09e6..eec79193e6 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ # go-nat -[![GoDoc](https://godoc.org/github.com/fd/go-nat?status.svg)](https://godoc.org/github.com/fd/go-nat) [![status](https://sourcegraph.com/api/repos/github.com/fd/go-nat/.badges/status.png)](https://sourcegraph.com/github.com/fd/go-nat) +[![GoDoc](https://godoc.org/github.com/libp2p/go-nat?status.svg)](https://godoc.org/github.com/libp2p/go-nat) [![status](https://sourcegraph.com/api/repos/github.com/libp2p/go-nat/.badges/status.png)](https://sourcegraph.com/github.com/libp2p/go-nat) + +Forked from: [fd/go-nat](https://github.com/fd/go-nat). diff --git a/go.mod b/go.mod index d7f4b28787..4ac4a3dec7 100644 --- a/go.mod +++ b/go.mod @@ -1,4 +1,4 @@ -module github.com/fd/go-nat +module github.com/libp2p/go-nat require ( github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324 From ab584c7bce7cd20eb179845a3c93059661fedc0c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 17 Sep 2018 13:56:40 -0700 Subject: [PATCH 0697/3965] gx release 1.0.2 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 79ffaca958..d816451dee 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -1.0.1: QmZ4g5TSk6EKQHrR61FgnE1ytJfirDHKvquL93a3Na1wJT +1.0.2: QmUTkdfn3e6EDb15C4XMHfooD5AoQxt5joYMeaqU56LU8h diff --git a/package.json b/package.json index 1155730974..61cdf82719 100644 --- a/package.json +++ b/package.json @@ -30,6 +30,6 @@ "language": "go", "license": "Apache-2.0", "name": "go-nat", - "version": "1.0.1" + "version": "1.0.2" } From ad42428e6bb50a1a3a4388984496d6be11492739 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 17 Sep 2018 14:01:07 -0700 Subject: [PATCH 0698/3965] gx: update go-libp2p-nat Pulls in update to fix #415. Thanks @cannium! --- package.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/package.json b/package.json index add9ca786d..1d328344f3 100644 --- a/package.json +++ b/package.json @@ -127,9 +127,9 @@ }, { "author": "whyrusleeping", - "hash": "QmScWrjfrMXw8AuBjtdDgdt97SuRhvf4KfR4y8djfNXCKT", + "hash": "QmcoQ3D7ypAFWSWexqhC9uSCeNrNhoYQGLJNBzUkDEf1CB", "name": "go-libp2p-nat", - "version": "0.8.6" + "version": "0.8.7" }, { "author": "whyrusleeping", From b6e2680be2cd8b6c5a48c66371d7c7019aa6cc86 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 18 Sep 2018 18:01:10 +0100 Subject: [PATCH 0699/3965] remove noop Stop() in address book. --- p2p/host/peerstore/pstoreds/addr_book.go | 5 ----- p2p/host/peerstore/pstoreds/ds_test.go | 14 +++++--------- 2 files changed, 5 insertions(+), 14 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index d9d762c7db..493ba8a80b 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -80,11 +80,6 @@ func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (*dsA return mgr, nil } -// Stop will signal the TTL manager to stop and block until it returns. -func (mgr *dsAddrBook) Stop() { - // noop -} - func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, error) { var ( keys = make([]ds.Key, len(addrs)) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 20bef6a9c3..fbfdf8a23c 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -172,9 +172,9 @@ func badgerStore(t testing.TB) (ds.TxnDatastore, func()) { func peerstoreFactory(tb testing.TB, opts Options) pt.PeerstoreFactory { return func() (pstore.Peerstore, func()) { - ds, closeFunc := badgerStore(tb) + store, closeFunc := badgerStore(tb) - ps, err := NewPeerstore(context.Background(), ds, opts) + ps, err := NewPeerstore(context.Background(), store, opts) if err != nil { tb.Fatal(err) } @@ -185,17 +185,13 @@ func peerstoreFactory(tb testing.TB, opts Options) pt.PeerstoreFactory { func addressBookFactory(tb testing.TB, opts Options) pt.AddrBookFactory { return func() (pstore.AddrBook, func()) { - ds, closeDB := badgerStore(tb) + store, closeFunc := badgerStore(tb) - mgr, err := NewAddrBook(context.Background(), ds, opts) + ab, err := NewAddrBook(context.Background(), store, opts) if err != nil { tb.Fatal(err) } - closeFunc := func() { - mgr.Stop() - closeDB() - } - return mgr, closeFunc + return ab, closeFunc } } From 35a1fb3b8ad12c2b42e96918a6255fb470db7b00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 18 Sep 2018 19:01:24 +0100 Subject: [PATCH 0700/3965] introduce struct for persisted value. --- p2p/host/peerstore/pstoreds/addr_book.go | 58 +++++++++++++++--------- 1 file changed, 37 insertions(+), 21 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 493ba8a80b..35fb3140ca 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -1,7 +1,6 @@ package pstoreds import ( - "bytes" "context" "encoding/binary" "errors" @@ -52,6 +51,24 @@ type cacheEntry struct { addrs []ma.Multiaddr } +type addrRecord struct { + ttl time.Duration + addr ma.Multiaddr +} + +func (ar *addrRecord) MarshalBinary() ([]byte, error) { + ttlB := make([]byte, 8) + binary.LittleEndian.PutUint64(ttlB, uint64(ar.ttl)) + return append(ttlB, ar.addr.Bytes()...), nil +} + +func (ar *addrRecord) UnmarshalBinary(b []byte) error { + ar.ttl = time.Duration(binary.LittleEndian.Uint64(b)) + // this had been serialized by us, no need to check for errors + ar.addr, _ = ma.NewMultiaddrBytes(b[8:]) + return nil +} + // NewAddrBook initializes a new address book given a // Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. @@ -194,11 +211,8 @@ func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr, ttl time.Du err error existed = make([]bool, len(keys)) exp = time.Now().Add(ttl) - ttlB = make([]byte, 8) ) - binary.LittleEndian.PutUint64(ttlB, uint64(ttl)) - txn := mgr.ds.NewTransaction(false) defer txn.Discard() @@ -230,8 +244,11 @@ func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr, ttl time.Du continue } - // format: bytes(ttl) || bytes(multiaddr) - value := append(ttlB, addrs[i].Bytes()...) + r := &addrRecord{ + ttl: ttl, + addr: addrs[i], + } + value, _ := r.MarshalBinary() if err = ttltxn.PutWithTTL(key, value, ttl); err != nil { log.Errorf("transaction failed and aborted while setting key: %s, cause: %v", key.String(), err) return nil, err @@ -267,16 +284,12 @@ func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time. func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time.Duration) error { var ( - prefix = ds.NewKey(p.Pretty()) - q = query.Query{Prefix: prefix.String(), KeysOnly: false} - oldb, newb = make([]byte, 8), make([]byte, 8) - results query.Results - err error + prefix = ds.NewKey(p.Pretty()) + q = query.Query{Prefix: prefix.String(), KeysOnly: false} + results query.Results + err error ) - binary.LittleEndian.PutUint64(oldb, uint64(oldTTL)) - binary.LittleEndian.PutUint64(newb, uint64(newTTL)) - txn := mgr.ds.NewTransaction(false) defer txn.Discard() @@ -286,13 +299,16 @@ func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time. defer results.Close() ttltxn := txn.(ds.TTLDatastore) + r := &addrRecord{} for result := range results.Next() { - // format: bytes(ttl) || bytes(multiaddr) - if curr := result.Value[:8]; !bytes.Equal(curr, oldb) { + r.UnmarshalBinary(result.Value) + if r.ttl != oldTTL { continue } - newVal := append(newb, result.Value[8:]...) - if err = ttltxn.PutWithTTL(ds.RawKey(result.Key), newVal, newTTL); err != nil { + + r.ttl = newTTL + value, _ := r.MarshalBinary() + if err = ttltxn.PutWithTTL(ds.RawKey(result.Key), value, newTTL); err != nil { return err } } @@ -336,12 +352,12 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { defer results.Close() var addrs []ma.Multiaddr + var r addrRecord // used to set the expiration for the entire cache entry earliestExp := maxTime for result := range results.Next() { - // extract multiaddr from value: bytes(ttl) || bytes(multiaddr) - if addr, err := ma.NewMultiaddrBytes(result.Value[8:]); err == nil { - addrs = append(addrs, addr) + if err = r.UnmarshalBinary(result.Value); err == nil { + addrs = append(addrs, r.addr) } if exp := result.Expiration; !exp.IsZero() && exp.Before(earliestExp) { From 38abbc9128256e247de8b2eaa41660abf37f9523 Mon Sep 17 00:00:00 2001 From: QYuan Date: Fri, 21 Sep 2018 10:33:15 +0800 Subject: [PATCH 0701/3965] Delete extra comments --- p2p/protocol/identify/id_test.go | 1 - 1 file changed, 1 deletion(-) diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index 79a09d2f9b..2c9b279f56 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -113,7 +113,6 @@ func testKnowsAddrs(t *testing.T, h host.Host, p peer.ID, expected []ma.Multiadd for _, addr := range expected { if _, found := have[addr.String()]; !found { t.Errorf("%s did not have addr for %s: %s", h.ID(), p, addr) - // panic("ahhhhhhh") } } } From 2787133b0469926b58dd819707cc4297a3ca7306 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 24 Sep 2018 05:11:17 -0700 Subject: [PATCH 0702/3965] gx publish 6.0.13 --- .gx/lastpubver | 2 +- package.json | 88 +++++++++++++++++++++++++------------------------- 2 files changed, 45 insertions(+), 45 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index b29ec9e334..5aaaf8d507 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.12: QmUEqyXr97aUbNmQADHYNknjwjjdVpJXEt1UZXmSG81EV4 +6.0.13: QmVsVARb86uSe1qYouewFMNd2p2sp2NGWm1JGPReVDWchW diff --git a/package.json b/package.json index 1d328344f3..7c55a90498 100644 --- a/package.json +++ b/package.json @@ -39,9 +39,9 @@ "version": "1.0.0" }, { - "hash": "QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr", + "hash": "QmZChCsSt8DctjceaL56Eibc29CVQq4dGKRXC5JRZ6Ppae", "name": "go-log", - "version": "1.5.5" + "version": "1.5.7" }, { "hash": "QmV6FjemM1K8oXjrvuq3wuVWWoU2TLDPmNnKrxHzY3v6Ai", @@ -55,33 +55,33 @@ }, { "author": "whyrusleeping", - "hash": "QmZ4zF1mBrt8C2mSCM4ZYE4aAnv78f7GvrzufJC4G5tecK", + "hash": "QmNLzS18jsmwTxXewTm3YnZVLftWCeegNZEBFjMrnvnBrH", "name": "go-libp2p-loggables", - "version": "1.1.21" + "version": "1.1.22" }, { "author": "whyrusleeping", - "hash": "QmReYSQGHjf28pKf93FwyD72mLXoZo94MB2Cq6VBSUHvFB", + "hash": "QmPJ6U14u5D2abGVkF3dopCeM1JqigVTUJ7jbAHjTnGZrt", "name": "go-libp2p-secio", - "version": "2.0.9" + "version": "2.0.10" }, { "author": "whyrusleeping", - "hash": "Qmda4cPRvSRyox3SqgJN6DfSZGU5TtHufPTp9uXjFj71X6", + "hash": "QmfAQMFpgDU2U4BXG64qVr8HSiictfWvkSBz7Y2oDj65st", "name": "go-libp2p-peerstore", - "version": "2.0.0" + "version": "2.0.1" }, { "author": "whyrusleeping", - "hash": "QmU129xU8dM79BgR97hu4fsiUDkTQrNHbzkiYfyrkNci8o", + "hash": "QmUFARWmz7LKUCzoWn1M3Z4Q2F8NK319452CBVFu4AgwzL", "name": "go-libp2p-transport", - "version": "3.0.9" + "version": "3.0.10" }, { "author": "whyrusleeping", - "hash": "QmV8KW6eBanaxCxGNrXx8Q3fZUqvumCz2Hwd2FGpb3vzYC", + "hash": "QmY3ymwbfNPb9KirX2DpgjpB4FaSGQ65edP9kSKyCHUtKQ", "name": "go-tcp-transport", - "version": "2.0.9" + "version": "2.0.10" }, { "author": "whyrusleeping", @@ -97,51 +97,51 @@ }, { "author": "whyrusleeping", - "hash": "QmRNhSdqzMcuRxX9A1egBeQ3BhDTguDV5HPwi8wRykkPU8", + "hash": "QmNfQbgBfARAtrYsBguChX6VJ5nbjeoYy1KdC36aaYWqG8", "name": "go-testutil", - "version": "1.2.7" + "version": "1.2.8" }, { "author": "whyrusleeping", - "hash": "QmZNJyx9GGCX4GeuHnLB8fxaxMLs4MjTjHokxfQcCd6Nve", + "hash": "QmfDPh144WGBqRxZb1TGDHerbMnZATrHZggAPw7putNnBq", "name": "go-libp2p-net", - "version": "3.0.9" + "version": "3.0.10" }, { "author": "whyrusleeping", - "hash": "QmdhwKw53CTV8EJSAsR1bpmMT5kXiWBgeAyv1EXeeDiXqR", + "hash": "QmNn6gcjBXpg8kccr9zEV7UVBpqAw8FZEiQ6DksvzyTQ5K", "name": "go-libp2p-metrics", - "version": "2.1.5" + "version": "2.1.6" }, { "author": "whyrusleeping", - "hash": "QmeMYW7Nj8jnnEfs9qhm7SxKkoDPUWXu3MsxX6BFwz34tf", + "hash": "QmeA5hsqgLryvkeyqeQdvGDqurLkYi3XEPLZP3pzuBJXh2", "name": "go-libp2p-host", - "version": "3.0.9" + "version": "3.0.10" }, { "author": "whyrusleeping", - "hash": "QmeDpqUwwdye8ABKVMPXKuWwPVURFdqTqssbTUB39E2Nwd", + "hash": "QmPQoCVRHaGD25VffyB7DFV5qP65hFSQJdSDy75P1vYBKe", "name": "go-libp2p-swarm", - "version": "3.0.11" + "version": "3.0.12" }, { "author": "whyrusleeping", - "hash": "QmcoQ3D7ypAFWSWexqhC9uSCeNrNhoYQGLJNBzUkDEf1CB", + "hash": "QmVicCVLxf62MkDBRXC27UTeqEvhNpBUouXNCqnLf4emJp", "name": "go-libp2p-nat", "version": "0.8.7" }, { "author": "whyrusleeping", - "hash": "QmfDapjsRAfzVpjeEm2tSmX19QpCrkLDXRCDDWJcbbUsFn", + "hash": "QmQgzDvAmKDGGthTLcqzF7VrxtzFbmH7946TwjkX9ruB2Q", "name": "go-libp2p-netutil", - "version": "0.4.7" + "version": "0.4.8" }, { "author": "whyrusleeping", - "hash": "QmQ4bjZSEC5drCRqssuXRymCswHPmW3Z46ibgBtg9XGd34", + "hash": "QmPBiDDeTyHpKypigUYcGivr6Lvo9GjoA7HeAGjoMyKHUK", "name": "go-libp2p-blankhost", - "version": "0.3.9" + "version": "0.3.10" }, { "author": "whyrusleeping", @@ -163,15 +163,15 @@ }, { "author": "whyrusleeping", - "hash": "QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W", + "hash": "QmbNepETomvmXfz1X5pHNFD2QuPqnqi47dTd94QJWSorQ3", "name": "go-libp2p-peer", - "version": "2.3.7" + "version": "2.3.8" }, { "author": "vyzo", - "hash": "QmWX6RySJ3yAYmfjLSw1LtRZnDh5oVeA9kM3scNQJkysqa", + "hash": "QmS5X31wA86RSjQqCUHG2CLPoFNnBZ3UMy1ikfBHQ3G6LG", "name": "go-libp2p-circuit", - "version": "2.2.1" + "version": "2.2.2" }, { "author": "lgierth", @@ -181,33 +181,33 @@ }, { "author": "why", - "hash": "QmWGGN1nysi1qgqto31bENwESkmZBY4YGK4sZC3qhnqhSv", + "hash": "QmV9T3rTezwJhMJQBzVrGj2tncJwv23dTKdxzXrUxAvWFi", "name": "go-libp2p-interface-connmgr", - "version": "0.0.15" + "version": "0.0.16" }, { "author": "whyrusleeping", - "hash": "QmdiBZzwGtN2yHJrWD9ojQ7ASS48nv7BcojWLkYd1ZtrV2", + "hash": "QmYsbnP57nNUKWijs8o2q3ZFrSUcAGTdKoQPuSJC3Uxt1Y", "name": "go-smux-multiplex", - "version": "3.0.12" + "version": "3.0.13" }, { "author": "whyrusleeping", - "hash": "Qmewujzx2quBisChBARfNDsMjTgUgs8qMkvCercSPQPLur", + "hash": "QmXawDNdHwcY8yf1AmuRCYrFgJAy4NiXY3RMZJ45XcNqRv", "name": "go-ws-transport", - "version": "2.0.9" + "version": "2.0.10" }, { "author": "stebalien", - "hash": "QmPGWMxTWBJwfH1dJo4J7eV4Fgpzb9gukPRkeEN1ycAkNu", + "hash": "QmUuM6nebEptQ53bjpAtfmhtbvbAy6WwhebUoEEg17fTxE", "name": "go-conn-security-multistream", - "version": "0.1.9" + "version": "0.1.10" }, { "author": "Stebalien", - "hash": "QmXaH5S5aHmbYyKSTMQDhWX8LzXfX91VGQFxaMdCuTUgBv", + "hash": "QmcnVbkHxvzHKYvdbDxiUVGbn5yQjkSbb9qqkGHcgwvZff", "name": "go-conn-security", - "version": "0.1.9" + "version": "0.1.10" }, { "author": "libp2p", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "QmWRcKvbFVND1vSTJZv5imdBxmkj9FFJ5Jku1qWxasAAMo", + "hash": "QmNRRKZjRHJTuEvbAkkmJLpBQX9FZ7rAXdD8zuknZv6mmF", "name": "go-libp2p-transport-upgrader", - "version": "0.1.9" + "version": "0.1.10" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.12" + "version": "6.0.13" } From c31fdfddc79a5e7b79c55235f9470a442987941d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 25 Sep 2018 15:18:08 -0700 Subject: [PATCH 0703/3965] bump the stream window size to 1MiB work towards https://github.com/libp2p/go-libp2p/issues/435 --- p2p/muxer/yamux/yamux.go | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 7a602d0ba6..12212eeb8f 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -46,12 +46,17 @@ type Transport yamux.Config // DefaultTransport has default settings for yamux var DefaultTransport = (*Transport)(&yamux.Config{ - AcceptBacklog: 256, // from yamux.DefaultConfig - EnableKeepAlive: true, // from yamux.DefaultConfig - KeepAliveInterval: 30 * time.Second, // from yamux.DefaultConfig - ConnectionWriteTimeout: 10 * time.Second, // from yamux.DefaultConfig - MaxStreamWindowSize: uint32(256 * 1024), // from yamux.DefaultConfig - LogOutput: ioutil.Discard, + AcceptBacklog: 256, // from yamux.DefaultConfig + EnableKeepAlive: true, // from yamux.DefaultConfig + KeepAliveInterval: 30 * time.Second, // from yamux.DefaultConfig + ConnectionWriteTimeout: 10 * time.Second, // from yamux.DefaultConfig + // We've bumped this to 1MiB as this critically limits throughput. + // + // 1MiB means a best case of 10MiB/s (83.89Mbps) on a connection with + // 100ms latency. The default gave us 2.4MiB *best case* which was + // totally unacceptable. + MaxStreamWindowSize: uint32(1024 * 1024), + LogOutput: ioutil.Discard, }) func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { From fbcd00db5fb4b81f0459ca07024cd463d7418b0e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 25 Sep 2018 15:52:41 -0700 Subject: [PATCH 0704/3965] avoid the buffer-pool on read We can xor in-place (this is what secio does). --- p2p/net/pnet/psk_conn.go | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index 18dbada1fe..a2e9e1e188 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -37,14 +37,9 @@ func (c *pskConn) Read(out []byte) (int, error) { c.readS20 = salsa20.New(c.psk, nonce) } - maxn := uint32(len(out)) - in := mpool.ByteSlicePool.Get(maxn).([]byte) // get buffer - defer mpool.ByteSlicePool.Put(maxn, in) // put the buffer back - - in = in[:maxn] // truncate to required length - n, err := c.Conn.Read(in) // read to in + n, err := c.Conn.Read(out) // read to in if n > 0 { - c.readS20.XORKeyStream(out[:n], in[:n]) // decrypt to out buffer + c.readS20.XORKeyStream(out[:n], out[:n]) // decrypt to out buffer } return n, err } From f9f82398e1a763e2666cf0238e00dd64f7252478 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 25 Sep 2018 15:53:24 -0700 Subject: [PATCH 0705/3965] switch to go-buffer-pool It has a nicer interface and we don't even need the rest of the msgio stuff. --- p2p/net/pnet/psk_conn.go | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index a2e9e1e188..5b16ef7a6c 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -7,8 +7,8 @@ import ( "net" salsa20 "github.com/davidlazar/go-crypto/salsa20" + pool "github.com/libp2p/go-buffer-pool" ipnet "github.com/libp2p/go-libp2p-interface-pnet" - mpool "github.com/libp2p/go-msgio/mpool" ) // we are using buffer pool as user needs their slice back @@ -58,11 +58,9 @@ func (c *pskConn) Write(in []byte) (int, error) { c.writeS20 = salsa20.New(c.psk, nonce) } - n := uint32(len(in)) - out := mpool.ByteSlicePool.Get(n).([]byte) // get buffer - defer mpool.ByteSlicePool.Put(n, out) // put the buffer back + out := pool.Get(len(in)) + defer pool.Put(out) - out = out[:n] // truncate to required length c.writeS20.XORKeyStream(out, in) // encrypt return c.Conn.Write(out) // send From fad7d9b963f4ee2f34956ccb96877cea23fe8c72 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 25 Sep 2018 16:11:49 -0700 Subject: [PATCH 0706/3965] switch to the correct mplex package --- p2p/muxer/mplex/multiplex.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 4a217c3048..1cef4ef3ff 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -3,8 +3,8 @@ package peerstream_multiplex import ( "net" - smux "github.com/libp2p/go-stream-muxer" // Conn is a connection to a remote peer. - mp "github.com/whyrusleeping/go-multiplex" // Conn is a connection to a remote peer. + mp "github.com/libp2p/go-mplex" // Conn is a connection to a remote peer. + smux "github.com/libp2p/go-stream-muxer" // Conn is a connection to a remote peer. ) type conn struct { From d859fb0dd7c2e5db2d5cf7f1aeb232557b6cca5d Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Tue, 18 Sep 2018 11:33:52 +0800 Subject: [PATCH 0707/3965] Track more info for observed addresses --- p2p/protocol/identify/id.go | 3 ++- p2p/protocol/identify/obsaddr.go | 23 +++++++++++++++-------- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 1f8dc01203..4fc88dd450 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -393,7 +393,8 @@ func (ids *IDService) consumeObservedAddress(observed []byte, c inet.Conn) { // ok! we have the observed version of one of our ListenAddresses! log.Debugf("added own observed listen addr: %s --> %s", c.LocalMultiaddr(), maddr) - ids.observedAddrs.Add(maddr, c.RemoteMultiaddr()) + ids.observedAddrs.Add(maddr, c.LocalMultiaddr(), c.RemoteMultiaddr(), + c.Stat().Direction) } func addrInAddrs(a ma.Multiaddr, as []ma.Multiaddr) bool { diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 0e72ab3433..b8aa84cf5d 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -4,6 +4,7 @@ import ( "sync" "time" + net "github.com/libp2p/go-libp2p-net" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) @@ -16,10 +17,12 @@ const ActivationThresh = 4 // - have been observed at least once recently (1h), because our position in the // network, or network port mapppings, may have changed. type ObservedAddr struct { - Addr ma.Multiaddr - SeenBy map[string]time.Time - LastSeen time.Time - Activated bool + Addr ma.Multiaddr // observed address by peers + InternalAddr ma.Multiaddr // corresponding internal address + SeenBy map[string]time.Time + LastSeen time.Time + ConnDirection net.Direction + Activated bool } func (oa *ObservedAddr) TryActivate(ttl time.Duration) bool { @@ -70,7 +73,9 @@ func (oas *ObservedAddrSet) Addrs() []ma.Multiaddr { return addrs } -func (oas *ObservedAddrSet) Add(addr ma.Multiaddr, observer ma.Multiaddr) { +func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, + direction net.Direction) { + oas.Lock() defer oas.Unlock() @@ -80,14 +85,16 @@ func (oas *ObservedAddrSet) Add(addr ma.Multiaddr, observer ma.Multiaddr) { oas.ttl = pstore.OwnObservedAddrTTL } - s := addr.String() + s := observed.String() oa, found := oas.addrs[s] // first time seeing address. if !found { oa = &ObservedAddr{ - Addr: addr, - SeenBy: make(map[string]time.Time), + Addr: observed, + InternalAddr: local, + SeenBy: make(map[string]time.Time), + ConnDirection: direction, } oas.addrs[s] = oa } From b4e4d496eae823931f454a7b2ba9ebd2df1cc693 Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Tue, 18 Sep 2018 11:49:22 +0800 Subject: [PATCH 0708/3965] Fix test --- p2p/protocol/identify/obsaddr_test.go | 56 +++++++++++++++------------ 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/p2p/protocol/identify/obsaddr_test.go b/p2p/protocol/identify/obsaddr_test.go index acf3d30d00..7291326218 100644 --- a/p2p/protocol/identify/obsaddr_test.go +++ b/p2p/protocol/identify/obsaddr_test.go @@ -4,6 +4,7 @@ import ( "testing" "time" + net "github.com/libp2p/go-libp2p-net" ma "github.com/multiformats/go-multiaddr" ) @@ -49,15 +50,22 @@ func TestObsAddrSet(t *testing.T) { b4 := m("/ip4/1.2.3.9/tcp/1237") b5 := m("/ip4/1.2.3.10/tcp/1237") - oas := ObservedAddrSet{} + oas := &ObservedAddrSet{} if !addrsMarch(oas.Addrs(), nil) { t.Error("addrs should be empty") } - oas.Add(a1, a4) - oas.Add(a2, a4) - oas.Add(a3, a4) + add := func(oas *ObservedAddrSet, observed, observer ma.Multiaddr) { + dummyLocal := m("/ip4/127.0.0.1/tcp/10086") + dummyDirection := net.DirOutbound + + oas.Add(observed, dummyLocal, observer, dummyDirection) + } + + add(oas, a1, a4) + add(oas, a2, a4) + add(oas, a3, a4) // these are all different so we should not yet get them. if !addrsMarch(oas.Addrs(), nil) { @@ -65,39 +73,39 @@ func TestObsAddrSet(t *testing.T) { } // same observer, so should not yet get them. - oas.Add(a1, a4) - oas.Add(a2, a4) - oas.Add(a3, a4) + add(oas, a1, a4) + add(oas, a2, a4) + add(oas, a3, a4) if !addrsMarch(oas.Addrs(), nil) { t.Error("addrs should _still_ be empty (same obs)") } // different observer, but same observer group. - oas.Add(a1, a5) - oas.Add(a2, a5) - oas.Add(a3, a5) + add(oas, a1, a5) + add(oas, a2, a5) + add(oas, a3, a5) if !addrsMarch(oas.Addrs(), nil) { t.Error("addrs should _still_ be empty (same obs group)") } - oas.Add(a1, b1) - oas.Add(a1, b2) - oas.Add(a1, b3) + add(oas, a1, b1) + add(oas, a1, b2) + add(oas, a1, b3) if !addrsMarch(oas.Addrs(), []ma.Multiaddr{a1}) { t.Error("addrs should only have a1") } - oas.Add(a2, a5) - oas.Add(a1, a5) - oas.Add(a1, a5) - oas.Add(a2, b1) - oas.Add(a1, b1) - oas.Add(a1, b1) - oas.Add(a2, b2) - oas.Add(a1, b2) - oas.Add(a1, b2) - oas.Add(a2, b4) - oas.Add(a2, b5) + add(oas, a2, a5) + add(oas, a1, a5) + add(oas, a1, a5) + add(oas, a2, b1) + add(oas, a1, b1) + add(oas, a1, b1) + add(oas, a2, b2) + add(oas, a1, b2) + add(oas, a1, b2) + add(oas, a2, b4) + add(oas, a2, b5) if !addrsMarch(oas.Addrs(), []ma.Multiaddr{a1, a2}) { t.Error("addrs should only have a1, a2") } From 96df62bd5bae73d0136c975bbe6230cdeb9f1dbc Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Wed, 26 Sep 2018 16:52:35 +0800 Subject: [PATCH 0709/3965] Changes on discussion - map internal -> []{external -> { observer -> [time, direction] } } - some cleaning --- p2p/protocol/identify/obsaddr.go | 95 ++++++++++++++++++++------------ 1 file changed, 59 insertions(+), 36 deletions(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index b8aa84cf5d..61ccc0bede 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -11,25 +11,27 @@ import ( const ActivationThresh = 4 +type observation struct { + seenTime time.Time + connDirection net.Direction +} + // ObservedAddr is an entry for an address reported by our peers. // We only use addresses that: // - have been observed at least 4 times in last 1h. (counter symmetric nats) // - have been observed at least once recently (1h), because our position in the // network, or network port mapppings, may have changed. type ObservedAddr struct { - Addr ma.Multiaddr // observed address by peers - InternalAddr ma.Multiaddr // corresponding internal address - SeenBy map[string]time.Time - LastSeen time.Time - ConnDirection net.Direction - Activated bool + Addr ma.Multiaddr + SeenBy map[string]observation // peer(observer) address -> observation info + LastSeen time.Time } -func (oa *ObservedAddr) TryActivate(ttl time.Duration) bool { +func (oa *ObservedAddr) activated(ttl time.Duration) bool { // cleanup SeenBy set now := time.Now() - for k, t := range oa.SeenBy { - if now.Sub(t) > ttl*ActivationThresh { + for k, ob := range oa.SeenBy { + if now.Sub(ob.seenTime) > ttl*ActivationThresh { delete(oa.SeenBy, k) } } @@ -44,32 +46,37 @@ func (oa *ObservedAddr) TryActivate(ttl time.Duration) bool { type ObservedAddrSet struct { sync.Mutex // guards whole datastruct. - addrs map[string]*ObservedAddr + // local(internal) address -> list of observed(external) addresses + addrs map[string][]*ObservedAddr ttl time.Duration } -func (oas *ObservedAddrSet) Addrs() []ma.Multiaddr { +// Addrs return all activated observed addresses +func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { oas.Lock() defer oas.Unlock() // for zero-value. - if oas.addrs == nil { + if len(oas.addrs) == 0 { return nil } now := time.Now() - addrs := make([]ma.Multiaddr, 0, len(oas.addrs)) - for s, a := range oas.addrs { - // remove timed out addresses. - if now.Sub(a.LastSeen) > oas.ttl { - delete(oas.addrs, s) - continue - } - - if a.Activated || a.TryActivate(oas.ttl) { - addrs = append(addrs, a.Addr) + filteredAddrMap := make(map[string][]*ObservedAddr) + for local, observedAddrs := range oas.addrs { + filteredAddrs := make([]*ObservedAddr, 0, len(observedAddrs)) + for _, a := range observedAddrs { + // leave only alive observed addresses + if now.Sub(a.LastSeen) <= oas.ttl { + filteredAddrs = append(filteredAddrs, a) + if a.activated(oas.ttl) { + addrs = append(addrs, a.Addr) + } + } } + filteredAddrMap[local] = filteredAddrs } + oas.addrs = filteredAddrMap return addrs } @@ -81,27 +88,43 @@ func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, // for zero-value. if oas.addrs == nil { - oas.addrs = make(map[string]*ObservedAddr) + oas.addrs = make(map[string][]*ObservedAddr) oas.ttl = pstore.OwnObservedAddrTTL } - s := observed.String() - oa, found := oas.addrs[s] + now := time.Now() + observerString := observerGroup(observer) + localString := local.String() + observedAddr := &ObservedAddr{ + Addr: observed, + SeenBy: map[string]observation{ + observerString: { + seenTime: now, + connDirection: direction, + }, + }, + LastSeen: now, + } - // first time seeing address. + observedAddrs, found := oas.addrs[localString] + // map key not exist yet, init with new values if !found { - oa = &ObservedAddr{ - Addr: observed, - InternalAddr: local, - SeenBy: make(map[string]time.Time), - ConnDirection: direction, + oas.addrs[localString] = []*ObservedAddr{observedAddr} + return + } + // check if observed address seen yet, if so, update it + for i, previousObserved := range observedAddrs { + if previousObserved.Addr.Equal(observed) { + observedAddrs[i].SeenBy[observerString] = observation{ + seenTime: now, + connDirection: direction, + } + observedAddrs[i].LastSeen = now + return } - oas.addrs[s] = oa } - - // mark the observer - oa.SeenBy[observerGroup(observer)] = time.Now() - oa.LastSeen = time.Now() + // observed address not seen yet, append it + oas.addrs[localString] = append(oas.addrs[localString], observedAddr) } // observerGroup is a function that determines what part of From 22c953c74c72d8aff3ca38049cf6858a3ad201b6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 27 Sep 2018 20:52:25 +0100 Subject: [PATCH 0710/3965] ds.NewTransaction() can now return an error. --- p2p/host/peerstore/pstoreds/addr_book.go | 31 +++++++++++++++++++----- p2p/host/peerstore/pstoreds/ds_test.go | 6 ++--- 2 files changed, 28 insertions(+), 9 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 35fb3140ca..df486217cc 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -213,7 +213,10 @@ func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr, ttl time.Du exp = time.Now().Add(ttl) ) - txn := mgr.ds.NewTransaction(false) + txn, err := mgr.ds.NewTransaction(false) + if err != nil { + return nil, err + } defer txn.Discard() ttltxn := txn.(ds.TTLDatastore) @@ -290,7 +293,10 @@ func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time. err error ) - txn := mgr.ds.NewTransaction(false) + txn, err := mgr.ds.NewTransaction(false) + if err != nil { + return err + } defer txn.Discard() if results, err = txn.Query(q); err != nil { @@ -342,7 +348,10 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { } } - txn := mgr.ds.NewTransaction(true) + txn, err := mgr.ds.NewTransaction(true) + if err != nil { + return nil + } defer txn.Discard() if results, err = txn.Query(q); err != nil { @@ -382,7 +391,11 @@ func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { err error ) - txn := mgr.ds.NewTransaction(true) + txn, err := mgr.ds.NewTransaction(true) + if err != nil { + log.Error(err) + return peer.IDSlice{} + } defer txn.Discard() if results, err = txn.Query(q); err != nil { @@ -456,7 +469,10 @@ func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { var err error - txn := mgr.ds.NewTransaction(false) + txn, err := mgr.ds.NewTransaction(false) + if err != nil { + return err + } defer txn.Discard() for _, key := range keys { @@ -479,7 +495,10 @@ func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) error { q := query.Query{Prefix: prefix.String(), KeysOnly: true} - txn := mgr.ds.NewTransaction(false) + txn, err := mgr.ds.NewTransaction(false) + if err != nil { + return err + } defer txn.Discard() results, err := txn.Query(q) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index fbfdf8a23c..253fd8b5d3 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -23,7 +23,7 @@ func BenchmarkBaselineBadgerDatastorePutEntry(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { - txn := bds.NewTransaction(false) + txn, _ := bds.NewTransaction(false) key := ds.RawKey(fmt.Sprintf("/key/%d", i)) txn.Put(key, []byte(fmt.Sprintf("/value/%d", i))) @@ -37,7 +37,7 @@ func BenchmarkBaselineBadgerDatastoreGetEntry(b *testing.B) { bds, closer := badgerStore(b) defer closer() - txn := bds.NewTransaction(false) + txn, _ := bds.NewTransaction(false) keys := make([]ds.Key, 1000) for i := 0; i < 1000; i++ { key := ds.RawKey(fmt.Sprintf("/key/%d", i)) @@ -50,7 +50,7 @@ func BenchmarkBaselineBadgerDatastoreGetEntry(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { - txn := bds.NewTransaction(true) + txn, _ := bds.NewTransaction(true) if _, err := txn.Get(keys[i%1000]); err != nil { b.Fatal(err) } From 0600392e23054dbe31ede5e77fc4781da5f6815c Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 28 Sep 2018 11:30:45 +0300 Subject: [PATCH 0711/3965] enable relay by default in New --- config/config.go | 5 +++-- defaults.go | 9 +++++++++ libp2p_test.go | 15 ++++++++++----- options.go | 20 ++++++++++++++++++-- 4 files changed, 40 insertions(+), 9 deletions(-) diff --git a/config/config.go b/config/config.go index b017fea60e..0348f19140 100644 --- a/config/config.go +++ b/config/config.go @@ -44,8 +44,9 @@ type Config struct { Insecure bool Protector pnet.Protector - Relay bool - RelayOpts []circuit.RelayOpt + RelayCustom bool + Relay bool + RelayOpts []circuit.RelayOpt ListenAddrs []ma.Multiaddr AddrsFactory bhost.AddrsFactory diff --git a/defaults.go b/defaults.go index f19e5ab08c..679fd7af5a 100644 --- a/defaults.go +++ b/defaults.go @@ -70,6 +70,11 @@ var DefaultListenAddrs = func(cfg *Config) error { )) } +// DefaultEnableRelay enables relay dialing and listening by default +var DefaultEnableRelay = func(cfg *Config) error { + return cfg.Apply(EnableRelay()) +} + // Complete list of default options and when to fallback on them. // // Please *DON'T* specify default options any other way. Putting this all here @@ -102,6 +107,10 @@ var defaults = []struct { fallback: func(cfg *Config) bool { return cfg.Peerstore == nil }, opt: DefaultPeerstore, }, + { + fallback: func(cfg *Config) bool { return !cfg.RelayCustom }, + opt: DefaultEnableRelay, + }, } // Defaults configures libp2p to use the default options. Can be combined with diff --git a/libp2p_test.go b/libp2p_test.go index 0685f4b6ef..3ecedb3b83 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -81,6 +81,7 @@ func TestDefaultListenAddrs(t *testing.T) { ctx := context.Background() re := regexp.MustCompile("/(ip)[4|6]/((0.0.0.0)|(::))/tcp/") + re2 := regexp.MustCompile("/p2p-circuit") // Test 1: Setting the correct listen addresses if userDefined.Transport == nil && userDefined.ListenAddrs == nil h, err := New(ctx) @@ -88,14 +89,15 @@ func TestDefaultListenAddrs(t *testing.T) { t.Fatal(err) } for _, addr := range h.Network().ListenAddresses() { - if re.FindStringSubmatchIndex(addr.String()) == nil { - t.Error("expected ip4 or ip6 interface") + if re.FindStringSubmatchIndex(addr.String()) == nil && + re2.FindStringSubmatchIndex(addr.String()) == nil { + t.Error("expected ip4 or ip6 or relay interface") } } h.Close() - // Test 2: Listen addr should not set if user defined transport is passed. + // Test 2: Listen addr only include relay if user defined transport is passed. h, err = New( ctx, Transport(tcp.NewTCPTransport), @@ -104,8 +106,11 @@ func TestDefaultListenAddrs(t *testing.T) { t.Fatal(err) } - if len(h.Network().ListenAddresses()) != 0 { - t.Error("expected zero listen addrs as none is set with user defined transport") + if len(h.Network().ListenAddresses()) != 1 { + t.Error("expected one listen addr with user defined transport") + } + if re2.FindStringSubmatchIndex(h.Network().ListenAddresses()[0].String()) == nil { + t.Error("expected relay address") } h.Close() } diff --git a/options.go b/options.go index ed8e89cfd5..ca8b505646 100644 --- a/options.go +++ b/options.go @@ -201,15 +201,25 @@ func AddrsFactory(factory config.AddrsFactory) Option { } } -// EnableRelay configures libp2p to enable the relay transport. +// EnableRelay configures libp2p to enable the relay transport with configuration options. func EnableRelay(options ...circuit.RelayOpt) Option { return func(cfg *Config) error { + cfg.RelayCustom = true cfg.Relay = true cfg.RelayOpts = options return nil } } +// DisableRelay configures libp2p to disable the relay transport +func DisableRelay() Option { + return func(cfg *Config) error { + cfg.RelayCustom = true + cfg.Relay = false + return nil + } +} + // FilterAddresses configures libp2p to never dial nor accept connections from // the given addresses. func FilterAddresses(addrs ...*net.IPNet) Option { @@ -245,9 +255,15 @@ func NATManager(nm config.NATManagerC) Option { // NoListenAddrs will configure libp2p to not listen by default. // // This will both clear any configured listen addrs and prevent libp2p from -// applying the default listen address option. +// applying the default listen address option. It also disables relay, unless the +// user explicitly specifies with an option, as the transport creates an implicit +// listen address that would make the node diable through any relay it was connected to. var NoListenAddrs = func(cfg *Config) error { cfg.ListenAddrs = []ma.Multiaddr{} + if !cfg.RelayCustom { + cfg.RelayCustom = true + cfg.Relay = false + } return nil } From 5834054ad8d6be538128b8ddf3450bee15fffc30 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 28 Sep 2018 12:50:53 +0300 Subject: [PATCH 0712/3965] fix typo --- options.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/options.go b/options.go index ca8b505646..224336ead6 100644 --- a/options.go +++ b/options.go @@ -257,7 +257,7 @@ func NATManager(nm config.NATManagerC) Option { // This will both clear any configured listen addrs and prevent libp2p from // applying the default listen address option. It also disables relay, unless the // user explicitly specifies with an option, as the transport creates an implicit -// listen address that would make the node diable through any relay it was connected to. +// listen address that would make the node dialable through any relay it was connected to. var NoListenAddrs = func(cfg *Config) error { cfg.ListenAddrs = []ma.Multiaddr{} if !cfg.RelayCustom { From 9961608ef0369efff20a82a98232f162b20f6f3e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 28 Sep 2018 14:04:52 +0100 Subject: [PATCH 0713/3965] datastore-backed impls of KeyBook and PeerMetadata. --- p2p/host/peerstore/interface.go | 7 ++ p2p/host/peerstore/pstoreds/addr_book.go | 55 +++------- p2p/host/peerstore/pstoreds/ds_test.go | 18 +++- p2p/host/peerstore/pstoreds/keybook.go | 128 +++++++++++++++++++++++ p2p/host/peerstore/pstoreds/metadata.go | 65 ++++++++++++ p2p/host/peerstore/pstoreds/peerstore.go | 58 +++++++++- p2p/host/peerstore/pstoremem/metadata.go | 3 +- 7 files changed, 288 insertions(+), 46 deletions(-) create mode 100644 p2p/host/peerstore/pstoreds/keybook.go create mode 100644 p2p/host/peerstore/pstoreds/metadata.go diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 068211c846..25d2c08578 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -68,6 +68,13 @@ type Peerstore interface { Peers() peer.IDSlice } +// PeerMetadata can handle values of any type. Serializing values is +// up to the implementation. Dynamic type introspection may not be +// supported, in which case explicitly enlisting types in the +// serializer may be required. +// +// Refer to the docs of the underlying implementation for more +// information. type PeerMetadata interface { // Get/Put is a simple registry for other peer-related key/value pairs. // if we find something we use often, it should become its own set of diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index df486217cc..ad37529a5c 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -7,13 +7,15 @@ import ( "time" lru "github.com/hashicorp/golang-lru" + ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" mh "github.com/multiformats/go-multihash" + peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) @@ -28,6 +30,10 @@ var ( ErrTTLDatastore = errors.New("datastore must provide TTL support") ) +// Peer addresses are stored under the following db key pattern: +// /peers/addr// +var abBase = ds.NewKey("/peers/addrs") + var _ pstore.AddrBook = (*dsAddrBook)(nil) // dsAddrBook is an address book backed by a Datastore with both an @@ -101,7 +107,7 @@ func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, er var ( keys = make([]ds.Key, len(addrs)) clean = make([]ma.Multiaddr, len(addrs)) - parentKey = ds.NewKey(peer.IDB58Encode(p)) + parentKey = abBase.ChildString(peer.IDB58Encode(p)) i = 0 ) @@ -287,7 +293,7 @@ func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time. func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time.Duration) error { var ( - prefix = ds.NewKey(p.Pretty()) + prefix = abBase.ChildString(peer.IDB58Encode(p)) q = query.Query{Prefix: prefix.String(), KeysOnly: false} results query.Results err error @@ -330,7 +336,7 @@ func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time. // Addrs returns all of the non-expired addresses for a given peer. func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { var ( - prefix = ds.NewKey(p.Pretty()) + prefix = abBase.ChildString(peer.IDB58Encode(p)) q = query.Query{Prefix: prefix.String(), KeysOnly: false, ReturnExpirations: true} results query.Results err error @@ -385,42 +391,11 @@ func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { // Peers returns all of the peer IDs for which the AddrBook has addresses. func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { - var ( - q = query.Query{KeysOnly: true} - results query.Results - err error - ) - - txn, err := mgr.ds.NewTransaction(true) + ids, err := uniquePeerIds(mgr.ds, abBase, func(result query.Result) string { + return ds.RawKey(result.Key).Parent().Name() + }) if err != nil { - log.Error(err) - return peer.IDSlice{} - } - defer txn.Discard() - - if results, err = txn.Query(q); err != nil { - log.Error(err) - return peer.IDSlice{} - } - - defer results.Close() - - idset := make(map[string]struct{}) - for result := range results.Next() { - key := ds.RawKey(result.Key) - idset[key.Parent().Name()] = struct{}{} - } - - if len(idset) == 0 { - return peer.IDSlice{} - } - - ids := make(peer.IDSlice, len(idset)) - i := 0 - for id := range idset { - pid, _ := peer.IDB58Decode(id) - ids[i] = pid - i++ + log.Errorf("error while retrieving peers with addresses: %v", err) } return ids } @@ -436,7 +411,7 @@ func (mgr *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mult func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { var ( err error - prefix = ds.NewKey(p.Pretty()) + prefix = abBase.ChildString(peer.IDB58Encode(p)) deleteFn func() error ) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 253fd8b5d3..2eb3f995d4 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -143,6 +143,10 @@ func TestBadgerDsAddrBook(t *testing.T) { }) } +func TestBadgerDsKeyBook(t *testing.T) { + pt.TestKeyBook(t, keyBookFactory(t, DefaultOpts())) +} + func BenchmarkBadgerDsPeerstore(b *testing.B) { caching := DefaultOpts() caching.CacheSize = 1024 @@ -159,15 +163,15 @@ func badgerStore(t testing.TB) (ds.TxnDatastore, func()) { if err != nil { t.Fatal(err) } - ds, err := badger.NewDatastore(dataPath, nil) + store, err := badger.NewDatastore(dataPath, nil) if err != nil { t.Fatal(err) } closer := func() { - ds.Close() + store.Close() os.RemoveAll(dataPath) } - return ds, closer + return store, closer } func peerstoreFactory(tb testing.TB, opts Options) pt.PeerstoreFactory { @@ -195,3 +199,11 @@ func addressBookFactory(tb testing.TB, opts Options) pt.AddrBookFactory { return ab, closeFunc } } + +func keyBookFactory(tb testing.TB, opts Options) pt.KeyBookFactory { + return func() (pstore.KeyBook, func()) { + store, closeFunc := badgerStore(tb) + kb, _ := NewKeyBook(context.Background(), store, opts) + return kb, closeFunc + } +} diff --git a/p2p/host/peerstore/pstoreds/keybook.go b/p2p/host/peerstore/pstoreds/keybook.go new file mode 100644 index 0000000000..4801e6b338 --- /dev/null +++ b/p2p/host/peerstore/pstoreds/keybook.go @@ -0,0 +1,128 @@ +package pstoreds + +import ( + "context" + "errors" + + ds "github.com/ipfs/go-datastore" + query "github.com/ipfs/go-datastore/query" + + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +// Public and private keys are stored under the following db key pattern: +// /peers/keys//{pub, priv} +var ( + kbBase = ds.NewKey("/peers/keys") + pubSuffix = ds.NewKey("/pub") + privSuffix = ds.NewKey("/priv") +) + +type dsKeyBook struct { + ds ds.TxnDatastore +} + +var _ pstore.KeyBook = (*dsKeyBook)(nil) + +func NewKeyBook(_ context.Context, store ds.TxnDatastore, _ Options) (pstore.KeyBook, error) { + return &dsKeyBook{store}, nil +} + +func (kb *dsKeyBook) PubKey(p peer.ID) ic.PubKey { + key := kbBase.ChildString(peer.IDB58Encode(p)).Child(pubSuffix) + + var pk ic.PubKey + if value, err := kb.ds.Get(key); err == nil { + pk, err = ic.UnmarshalPublicKey(value) + if err != nil { + log.Errorf("error when unmarshalling pubkey from datastore for peer %s: %s\n", p.Pretty(), err) + } + } else if err == ds.ErrNotFound { + pk, err = p.ExtractPublicKey() + if err != nil { + log.Errorf("error when extracting pubkey from peer ID for peer %s: %s\n", p.Pretty(), err) + return nil + } + pkb, err := pk.Bytes() + if err != nil { + log.Errorf("error when turning extracted pubkey into bytes for peer %s: %s\n", p.Pretty(), err) + return nil + } + err = kb.ds.Put(key, pkb) + if err != nil { + log.Errorf("error when adding extracted pubkey to peerstore for peer %s: %s\n", p.Pretty(), err) + return nil + } + } else { + log.Errorf("error when fetching pubkey from datastore for peer %s: %s\n", p.Pretty(), err) + } + + return pk +} + +func (kb *dsKeyBook) AddPubKey(p peer.ID, pk ic.PubKey) error { + // check it's correct. + if !p.MatchesPublicKey(pk) { + return errors.New("peer ID does not match public key") + } + + key := kbBase.ChildString(peer.IDB58Encode(p)).Child(pubSuffix) + val, err := pk.Bytes() + if err != nil { + log.Errorf("error while converting pubkey byte string for peer %s: %s\n", p.Pretty(), err) + return err + } + err = kb.ds.Put(key, val) + if err != nil { + log.Errorf("error while updating pubkey in datastore for peer %s: %s\n", p.Pretty(), err) + } + return err +} + +func (kb *dsKeyBook) PrivKey(p peer.ID) ic.PrivKey { + key := kbBase.ChildString(peer.IDB58Encode(p)).Child(privSuffix) + value, err := kb.ds.Get(key) + if err != nil { + log.Errorf("error while fetching privkey from datastore for peer %s: %s\n", p.Pretty(), err) + return nil + } + sk, err := ic.UnmarshalPrivateKey(value) + if err != nil { + return nil + } + return sk +} + +func (kb *dsKeyBook) AddPrivKey(p peer.ID, sk ic.PrivKey) error { + if sk == nil { + return errors.New("private key is nil") + } + // check it's correct. + if !p.MatchesPrivateKey(sk) { + return errors.New("peer ID does not match private key") + } + + key := kbBase.ChildString(peer.IDB58Encode(p)).Child(privSuffix) + val, err := sk.Bytes() + if err != nil { + log.Errorf("error while converting privkey byte string for peer %s: %s\n", p.Pretty(), err) + return err + } + err = kb.ds.Put(key, val) + if err != nil { + log.Errorf("error while updating privkey in datastore for peer %s: %s\n", p.Pretty(), err) + } + return err +} + +func (kb *dsKeyBook) PeersWithKeys() peer.IDSlice { + ids, err := uniquePeerIds(kb.ds, kbBase, func(result query.Result) string { + return ds.RawKey(result.Key).Parent().Name() + }) + if err != nil { + log.Errorf("error while retrieving peers with keys: %v", err) + } + return ids +} diff --git a/p2p/host/peerstore/pstoreds/metadata.go b/p2p/host/peerstore/pstoreds/metadata.go new file mode 100644 index 0000000000..de21434d6e --- /dev/null +++ b/p2p/host/peerstore/pstoreds/metadata.go @@ -0,0 +1,65 @@ +package pstoreds + +import ( + "bytes" + "context" + "encoding/gob" + + ds "github.com/ipfs/go-datastore" + + pool "github.com/libp2p/go-buffer-pool" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +// Metadata is stored under the following db key pattern: +// /peers/metadata// +var pmBase = ds.NewKey("/peers/metadata") + +type dsPeerMetadata struct { + ds ds.Datastore +} + +var _ pstore.PeerMetadata = (*dsPeerMetadata)(nil) + +func init() { + // Gob registers basic types by default. + // + // Register complex types used by the peerstore itself. + gob.Register(make(map[string]struct{})) +} + +// NewPeerMetadata creates a metadata store backed by a persistent db. It uses gob for serialisation. +// +// See `init()` to learn which types are registered by default. Modules wishing to store +// values of other types will need to `gob.Register()` them explicitly, or else callers +// will receive runtime errors. +func NewPeerMetadata(_ context.Context, store ds.Datastore, _ Options) (pstore.PeerMetadata, error) { + return &dsPeerMetadata{store}, nil +} + +func (pm *dsPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { + k := pmBase.ChildString(peer.IDB58Encode(p)).ChildString(key) + value, err := pm.ds.Get(k) + if err != nil { + if err == ds.ErrNotFound { + err = pstore.ErrNotFound + } + return nil, err + } + + var res interface{} + if err := gob.NewDecoder(bytes.NewReader(value)).Decode(&res); err != nil { + return nil, err + } + return res, nil +} + +func (pm *dsPeerMetadata) Put(p peer.ID, key string, val interface{}) error { + k := pmBase.ChildString(peer.IDB58Encode(p)).ChildString(key) + var buf pool.Buffer + if err := gob.NewEncoder(&buf).Encode(&val); err != nil { + return err + } + return pm.ds.Put(k, buf.Bytes()) +} diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index a3b4d3f174..fc3b6d70e1 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -5,9 +5,10 @@ import ( "time" ds "github.com/ipfs/go-datastore" + "github.com/ipfs/go-datastore/query" + "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" - pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" ) // Configuration object for the peerstore. @@ -42,6 +43,59 @@ func NewPeerstore(ctx context.Context, store ds.TxnDatastore, opts Options) (pst return nil, err } - ps := pstore.NewPeerstore(pstoremem.NewKeyBook(), addrBook, pstoremem.NewPeerMetadata()) + keyBook, err := NewKeyBook(ctx, store, opts) + if err != nil { + return nil, err + } + + peerMetadata, err := NewPeerMetadata(ctx, store, opts) + if err != nil { + return nil, err + } + + ps := pstore.NewPeerstore(keyBook, addrBook, peerMetadata) return ps, nil } + +// uniquePeerIds extracts and returns unique peer IDs from database keys. +func uniquePeerIds(ds ds.TxnDatastore, prefix ds.Key, extractor func(result query.Result) string) (peer.IDSlice, error) { + var ( + q = query.Query{Prefix: prefix.String(), KeysOnly: true} + results query.Results + err error + ) + + txn, err := ds.NewTransaction(true) + if err != nil { + return peer.IDSlice{}, err + } + defer txn.Discard() + + if results, err = txn.Query(q); err != nil { + log.Error(err) + return peer.IDSlice{}, err + } + + defer results.Close() + + idset := make(map[string]struct{}) + for result := range results.Next() { + k := extractor(result) + idset[k] = struct{}{} + //key := ds.RawKey(result.Key) + //idset[key.Parent().Name()] = struct{}{} + } + + if len(idset) == 0 { + return peer.IDSlice{}, nil + } + + ids := make(peer.IDSlice, len(idset)) + i := 0 + for id := range idset { + pid, _ := peer.IDB58Decode(id) + ids[i] = pid + i++ + } + return ids, nil +} diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index 8b58256c16..ba57dd7fbf 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -10,11 +10,12 @@ import ( type memoryPeerMetadata struct { // store other data, like versions //ds ds.ThreadSafeDatastore - // TODO: use a datastore for this ds map[string]interface{} dslock sync.Mutex } +var _ pstore.PeerMetadata = (*memoryPeerMetadata)(nil) + func NewPeerMetadata() pstore.PeerMetadata { return &memoryPeerMetadata{ ds: make(map[string]interface{}), From 0531abfbccae5d7685afbbeaa28ff6f7f2566cfb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 28 Sep 2018 14:18:05 +0100 Subject: [PATCH 0714/3965] add explicit import aliases. --- p2p/host/peerstore/pstoreds/peerstore.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index fc3b6d70e1..34b150649e 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -5,9 +5,9 @@ import ( "time" ds "github.com/ipfs/go-datastore" - "github.com/ipfs/go-datastore/query" - "github.com/libp2p/go-libp2p-peer" + query "github.com/ipfs/go-datastore/query" + peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ) From 78cfb8ba63731b0b218ed53a32a567b0be5af6a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 28 Sep 2018 18:47:41 +0100 Subject: [PATCH 0715/3965] review comments. --- p2p/host/peerstore/pstoreds/peerstore.go | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 34b150649e..cc3c6f88ab 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -67,13 +67,13 @@ func uniquePeerIds(ds ds.TxnDatastore, prefix ds.Key, extractor func(result quer txn, err := ds.NewTransaction(true) if err != nil { - return peer.IDSlice{}, err + return nil, err } defer txn.Discard() if results, err = txn.Query(q); err != nil { log.Error(err) - return peer.IDSlice{}, err + return nil, err } defer results.Close() @@ -82,8 +82,6 @@ func uniquePeerIds(ds ds.TxnDatastore, prefix ds.Key, extractor func(result quer for result := range results.Next() { k := extractor(result) idset[k] = struct{}{} - //key := ds.RawKey(result.Key) - //idset[key.Parent().Name()] = struct{}{} } if len(idset) == 0 { From 9d3db84029c63c1625911edc37a5a2de8ee415d3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 29 Sep 2018 11:09:48 +0300 Subject: [PATCH 0716/3965] address review comments --- p2p/host/autonat/autonat.go | 15 ++++++++++----- p2p/host/autonat/client.go | 7 ++++--- p2p/host/autonat/svc.go | 2 +- p2p/host/autonat/svc_test.go | 10 +++++----- 4 files changed, 20 insertions(+), 14 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index bdf1e8d8ba..037a094f56 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -8,6 +8,7 @@ import ( "time" host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) @@ -29,8 +30,7 @@ const ( var ( AutoNATBootDelay = 15 * time.Second AutoNATRefreshInterval = 15 * time.Minute - - AutoNATRequestTimeout = 60 * time.Second + AutoNATRequestTimeout = 60 * time.Second ) // AutoNAT is the interface for ambient NAT autodiscovery @@ -86,7 +86,12 @@ func (as *AmbientAutoNAT) PublicAddr() (ma.Multiaddr, error) { func (as *AmbientAutoNAT) background() { // wait a bit for the node to come online and establish some connections // before starting autodetection - time.Sleep(AutoNATBootDelay) + select { + case <-time.After(AutoNATBootDelay): + case <-as.ctx.Done(): + return + } + for { as.autodetect() select { @@ -109,7 +114,7 @@ func (as *AmbientAutoNAT) autodetect() { for _, p := range peers { ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) - a, err := cli.Dial(ctx, p) + a, err := cli.DialBack(ctx, p) cancel() switch { @@ -148,7 +153,7 @@ func (as *AmbientAutoNAT) getPeers() []peer.ID { peers := make([]peer.ID, 0, len(as.peers)) for p := range as.peers { - if len(as.host.Network().ConnsToPeer(p)) > 0 { + if as.host.Network().Connectedness(p) == inet.Connected { peers = append(peers, p) } } diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index 812ce47628..468bc68106 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -16,8 +16,9 @@ import ( // AutoNATClient is a stateless client interface to AutoNAT peers type AutoNATClient interface { - // Dial requests from a peer providing AutoNAT services to test dial back - Dial(ctx context.Context, p peer.ID) (ma.Multiaddr, error) + // DialBack requests from a peer providing AutoNAT services to test dial back + // and report the address on a successful connection. + DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) } // AutoNATError is the class of errors signalled by AutoNAT services @@ -35,7 +36,7 @@ type client struct { h host.Host } -func (c *client) Dial(ctx context.Context, p peer.ID) (ma.Multiaddr, error) { +func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) { s, err := c.h.NewStream(ctx, p, AutoNATProto) if err != nil { return nil, err diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 6ec7e7c9af..b25ead6eb1 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -166,7 +166,7 @@ func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { } ra := conns[0].RemoteMultiaddr() - conns[0].Close() + as.dialer.Network().ClosePeer(pi.ID) return newDialResponseOK(ra) } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 8cc7f667a2..77d482be83 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -55,7 +55,7 @@ func TestAutoNATServiceDialError(t *testing.T) { hc, ac := makeAutoNATClient(ctx, t) connect(t, hs, hc) - _, err := ac.Dial(ctx, hs.ID()) + _, err := ac.DialBack(ctx, hs.ID()) if err == nil { t.Fatal("Dial back succeeded unexpectedly!") } @@ -78,7 +78,7 @@ func TestAutoNATServiceDialSuccess(t *testing.T) { hc, ac := makeAutoNATClient(ctx, t) connect(t, hs, hc) - _, err := ac.Dial(ctx, hs.ID()) + _, err := ac.DialBack(ctx, hs.ID()) if err != nil { t.Fatalf("Dial back failed: %s", err.Error()) } @@ -101,12 +101,12 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { hc, ac := makeAutoNATClient(ctx, t) connect(t, hs, hc) - _, err := ac.Dial(ctx, hs.ID()) + _, err := ac.DialBack(ctx, hs.ID()) if err != nil { t.Fatal(err) } - _, err = ac.Dial(ctx, hs.ID()) + _, err = ac.DialBack(ctx, hs.ID()) if err == nil { t.Fatal("Dial back succeeded unexpectedly!") } @@ -117,7 +117,7 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { time.Sleep(2 * time.Second) - _, err = ac.Dial(ctx, hs.ID()) + _, err = ac.DialBack(ctx, hs.ID()) if err != nil { t.Fatal(err) } From 744bf71d9158cbce7ae26822748975090ffde062 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 29 Sep 2018 11:38:53 +0300 Subject: [PATCH 0717/3965] use the protocol list by identify, don't emit chatter on every connection --- p2p/host/autonat/notify.go | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 4e2bc9d984..30ad9cd0c1 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -1,8 +1,9 @@ package autonat import ( + "time" + inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) @@ -14,18 +15,25 @@ func (as *AmbientAutoNAT) OpenedStream(net inet.Network, s inet.Stream) {} func (as *AmbientAutoNAT) ClosedStream(net inet.Network, s inet.Stream) {} func (as *AmbientAutoNAT) Connected(net inet.Network, c inet.Conn) { - go func(p peer.ID) { - s, err := as.host.NewStream(as.ctx, p, AutoNATProto) + p := c.RemotePeer() + + go func() { + // add some delay for identify + time.Sleep(250 * time.Millisecond) + + protos, err := as.host.Peerstore().SupportsProtocols(p, AutoNATProto) if err != nil { + log.Debugf("error retrieving supported protocols for peer %s: %s", p, err) return } - s.Close() - log.Infof("Discovered AutoNAT peer %s", p.Pretty()) - as.mx.Lock() - as.peers[p] = struct{}{} - as.mx.Unlock() - }(c.RemotePeer()) + if len(protos) > 0 { + log.Infof("Discovered AutoNAT peer %s", p.Pretty()) + as.mx.Lock() + as.peers[p] = struct{}{} + as.mx.Unlock() + } + }() } func (as *AmbientAutoNAT) Disconnected(net inet.Network, c inet.Conn) {} From 92ec4b2d186c5e7db0be9e47753956a52ae9d05c Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Sat, 29 Sep 2018 11:00:23 +0800 Subject: [PATCH 0718/3965] Updates due to code review --- p2p/host/basic/basic_host.go | 3 +-- p2p/protocol/identify/obsaddr.go | 36 ++++++++++++-------------------- 2 files changed, 14 insertions(+), 25 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 5379bed4b2..1838cbc2fb 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -5,8 +5,6 @@ import ( "io" "time" - identify "github.com/libp2p/go-libp2p/p2p/protocol/identify" - logging "github.com/ipfs/go-log" goprocess "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" @@ -15,6 +13,7 @@ import ( peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" protocol "github.com/libp2p/go-libp2p-protocol" + identify "github.com/libp2p/go-libp2p/p2p/protocol/identify" ma "github.com/multiformats/go-multiaddr" madns "github.com/multiformats/go-multiaddr-dns" msmux "github.com/multiformats/go-multistream" diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 61ccc0bede..40c6f78df2 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -62,7 +62,6 @@ func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { } now := time.Now() - filteredAddrMap := make(map[string][]*ObservedAddr) for local, observedAddrs := range oas.addrs { filteredAddrs := make([]*ObservedAddr, 0, len(observedAddrs)) for _, a := range observedAddrs { @@ -74,9 +73,8 @@ func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { } } } - filteredAddrMap[local] = filteredAddrs + oas.addrs[local] = filteredAddrs } - oas.addrs = filteredAddrMap return addrs } @@ -95,36 +93,28 @@ func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, now := time.Now() observerString := observerGroup(observer) localString := local.String() - observedAddr := &ObservedAddr{ - Addr: observed, - SeenBy: map[string]observation{ - observerString: { - seenTime: now, - connDirection: direction, - }, - }, - LastSeen: now, + ob := observation{ + seenTime: now, + connDirection: direction, } - observedAddrs, found := oas.addrs[localString] - // map key not exist yet, init with new values - if !found { - oas.addrs[localString] = []*ObservedAddr{observedAddr} - return - } + observedAddrs := oas.addrs[localString] // check if observed address seen yet, if so, update it for i, previousObserved := range observedAddrs { if previousObserved.Addr.Equal(observed) { - observedAddrs[i].SeenBy[observerString] = observation{ - seenTime: now, - connDirection: direction, - } + observedAddrs[i].SeenBy[observerString] = ob observedAddrs[i].LastSeen = now return } } // observed address not seen yet, append it - oas.addrs[localString] = append(oas.addrs[localString], observedAddr) + oas.addrs[localString] = append(oas.addrs[localString], &ObservedAddr{ + Addr: observed, + SeenBy: map[string]observation{ + observerString: ob, + }, + LastSeen: now, + }) } // observerGroup is a function that determines what part of From e7fc8861f9c899107242aa4fb46dd0396039f8f1 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 1 Oct 2018 10:34:39 +0300 Subject: [PATCH 0719/3965] downgrade protocol mux failure log to debug --- p2p/host/basic/basic_host.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 5379bed4b2..a8f1ebb91b 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -223,7 +223,7 @@ func (h *BasicHost) newStreamHandler(s inet.Stream) { } logf("protocol EOF: %s (took %s)", s.Conn().RemotePeer(), took) } else { - log.Infof("protocol mux failed: %s (took %s)", err, took) + log.Debugf("protocol mux failed: %s (took %s)", err, took) } s.Reset() return From c8096184f855f18a5df4565275f841c1f74cd731 Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Tue, 2 Oct 2018 04:59:24 +0200 Subject: [PATCH 0720/3965] Refactor and relax filtering of known undialable addresses This commit moves filtering of dial candidates into its own little function. Things that are being filtered: addresses configured to be blocked, IPv6 link-local addresses, addresses without a dial-capable transport, and addresses that we know to be our own. It's is an optimization to avoid wasting time on dials that we know are going to fail. This also relaxes the filtering of addresses that we consider our own. Previously, any address would get filtered that's registered in peerstore for or own PeerID. For e.g. /ip4/1.2.3.4/tcp/4001 that's fine, but for ephemeral ports it can already cause problems. In addition, now that go-libp2p-circuit is being fixed to handle its multiaddrs slightly differently, /p2p-circuit addresses won't contain the PeerID anymore. That means they stand for themselves, and would get filtered too. (/p2p-circuit is the address we want to dial, but it's also on of "our own addresses"). In the future we'd want to use the mafmt package here, and also consider /quic, /ws, etc. addresses as our own. --- p2p/net/swarm/swarm_dial.go | 55 +++++++++++++++++++++---------------- 1 file changed, 31 insertions(+), 24 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 64971b07e9..6030ca79d7 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -281,42 +281,24 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { log.Debug("Dial not given PrivateKey, so WILL NOT SECURE conn.") } - ila, _ := s.InterfaceListenAddresses() - subtractFilter := addrutil.SubtractFilter(append(ila, s.peers.Addrs(s.local)...)...) - - // get live channel of addresses for peer, filtered by the given filters - /* - remoteAddrChan := s.peers.AddrsChan(ctx, p, - addrutil.AddrUsableFilter, - subtractFilter, - s.Filters.AddrBlocked) - */ - ////// /* - This code is temporary, the peerstore can currently provide + This slice-to-chan code is temporary, the peerstore can currently provide a channel as an interface for receiving addresses, but more thought needs to be put into the execution. For now, this allows us to use the improved rate limiter, while maintaining the outward behaviour that we previously had (halting a dial when we run out of addrs) */ - paddrs := s.peers.Addrs(p) - goodAddrs := addrutil.FilterAddrs(paddrs, - subtractFilter, - s.canDial, - // TODO: Consider allowing this? - addrutil.AddrOverNonLocalIP, - addrutil.FilterNeg(s.Filters.AddrBlocked), - ) - remoteAddrChan := make(chan ma.Multiaddr, len(goodAddrs)) + goodAddrs := s.filterKnownUndialables(s.peers.Addrs(p)) + goodAddrsChan := make(chan ma.Multiaddr, len(goodAddrs)) for _, a := range goodAddrs { - remoteAddrChan <- a + goodAddrsChan <- a } - close(remoteAddrChan) + close(goodAddrsChan) ///////// // try to get a connection to any addr - connC, err := s.dialAddrs(ctx, p, remoteAddrChan) + connC, err := s.dialAddrs(ctx, p, goodAddrsChan) if err != nil { logdial["error"] = err.Error() return nil, err @@ -336,6 +318,31 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { return swarmC, nil } +// filterKnownUndialables takes a list of multiaddrs, and removes those +// that we definitely don't want to dial: addresses configured to be blocked, +// IPv6 link-local addresses, addresses without a dial-capable transport, +// and addresses that we know to be our own. +// This is an optimization to avoid wasting time on dials that we know are going to fail. +func (s *Swarm) filterKnownUndialables(addrs []ma.Multiaddr) []ma.Multiaddr { + lisAddrs, _ := s.InterfaceListenAddresses() + var ourAddrs []ma.Multiaddr + for _, addr := range lisAddrs { + protos := addr.Protocols() + // we're only sure about filtering out /ip4 and /ip6 addresses, so far + if len(protos) == 2 && (protos[0].Code == ma.P_IP4 || protos[0].Code == ma.P_IP6) { + ourAddrs = append(ourAddrs, addr) + } + } + + return addrutil.FilterAddrs(addrs, + addrutil.SubtractFilter(ourAddrs...), + s.canDial, + // TODO: Consider allowing link-local addresses + addrutil.AddrOverNonLocalIP, + addrutil.FilterNeg(s.Filters.AddrBlocked), + ) +} + func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (transport.Conn, error) { log.Debugf("%s swarm dialing %s", s.local, p) From 0e80d957363e85fd21d416afd3c3134346824dcb Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Wed, 26 Sep 2018 18:04:41 +0200 Subject: [PATCH 0721/3965] Change transport address to not include /p2p part --- .../internal/circuitv1-deprecated/conn.go | 12 +++++++----- .../internal/circuitv1-deprecated/dial.go | 16 ++++++++-------- .../internal/circuitv1-deprecated/listen.go | 7 +------ .../internal/circuitv1-deprecated/relay_test.go | 14 ++++---------- 4 files changed, 20 insertions(+), 29 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index b0afbd063f..df39f119f5 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -35,12 +35,14 @@ func (c *Conn) RemoteAddr() net.Addr { } } +// TODO: is it okay to cast c.Conn().RemotePeer() into a multiaddr? might be "user input" func (c *Conn) RemoteMultiaddr() ma.Multiaddr { - a, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", c.Conn().RemotePeer().Pretty(), c.remote.ID.Pretty())) - if err != nil { - panic(err) - } - return a + proto := ma.ProtocolWithCode(ma.P_P2P).Name + peerid := c.Conn().RemotePeer().Pretty() + p2paddr := ma.StringCast(fmt.Sprintf("/%s/%s", proto, peerid)) + + circaddr := ma.Cast(ma.CodeToVarint(P_CIRCUIT)) + return p2paddr.Encapsulate(circaddr) } func (c *Conn) LocalMultiaddr() ma.Multiaddr { diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 3f021fa434..ecfd23fe6b 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -12,20 +12,20 @@ import ( ) func (d *RelayTransport) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (tpt.Conn, error) { - c, err := d.Relay().Dial(ctx, a) + c, err := d.Relay().Dial(ctx, a, p) if err != nil { return nil, err } return d.upgrader.UpgradeOutbound(ctx, d, c, p) } -func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr) (*Conn, error) { +func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (*Conn, error) { if !r.Matches(a) { return nil, fmt.Errorf("%s is not a relay address", a) } parts := ma.Split(a) - spl, _ := ma.NewMultiaddr("/p2p-circuit") + spl := ma.Cast(ma.CodeToVarint(P_CIRCUIT)) var relayaddr, destaddr ma.Multiaddr for i, p := range parts { @@ -36,9 +36,9 @@ func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr) (*Conn, error) { } } - dinfo, err := pstore.InfoFromP2pAddr(destaddr) - if err != nil { - return nil, err + dinfo := &pstore.PeerInfo{ID: p, Addrs: []ma.Multiaddr{}} + if len(destaddr.Bytes()) > 0 { + dinfo.Addrs = append(dinfo.Addrs, destaddr) } if len(relayaddr.Bytes()) == 0 { @@ -47,9 +47,9 @@ func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr) (*Conn, error) { } var rinfo *pstore.PeerInfo - rinfo, err = pstore.InfoFromP2pAddr(relayaddr) + rinfo, err := pstore.InfoFromP2pAddr(relayaddr) if err != nil { - return nil, err + return nil, fmt.Errorf("error parsing multiaddr '%s': %s", relayaddr.String(), err) } return r.DialPeer(ctx, *rinfo, *dinfo) diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index ee1f92e15d..0b1885be03 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -1,7 +1,6 @@ package relay import ( - "fmt" "net" pb "github.com/libp2p/go-libp2p-circuit/pb" @@ -50,11 +49,7 @@ func (l *RelayListener) Addr() net.Addr { } func (l *RelayListener) Multiaddr() ma.Multiaddr { - a, err := ma.NewMultiaddr(fmt.Sprintf("/p2p-circuit/ipfs/%s", l.self.Pretty())) - if err != nil { - panic(err) - } - return a + return ma.Cast(ma.CodeToVarint(P_CIRCUIT)) } func (l *RelayListener) Close() error { diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 04feb6ca80..6fdc1d62e8 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -204,15 +204,12 @@ func TestBasicRelayDial(t *testing.T) { con.Close() }() - addr, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit/ipfs/%s", hosts[1].ID().Pretty(), hosts[2].ID().Pretty())) - if err != nil { - t.Fatal(err) - } + addr := ma.StringCast(fmt.Sprintf("/ipfs/%s/p2p-circuit", hosts[1].ID().Pretty())) rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - con, err := r1.Dial(rctx, addr) + con, err := r1.Dial(rctx, addr, hosts[2].ID()) if err != nil { t.Fatal(err) } @@ -262,15 +259,12 @@ func TestUnspecificRelayDial(t *testing.T) { con.Close() }() - addr, err := ma.NewMultiaddr(fmt.Sprintf("/p2p-circuit/ipfs/%s", hosts[2].ID().Pretty())) - if err != nil { - t.Fatal(err) - } + addr := ma.StringCast(fmt.Sprintf("/p2p-circuit")) rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - con, err := r1.Dial(rctx, addr) + con, err := r1.Dial(rctx, addr, hosts[2].ID()) if err != nil { t.Fatal(err) } From 382eb8356615bae9687fa186c1c4927afe1f8d6c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 1 Oct 2018 22:16:08 -0700 Subject: [PATCH 0722/3965] revert to storing peer addresses in a map We switched to a slice to reduce the amount of memory the peerstore ended up taking up unfortunately, this really killed us in allocations. Looking at go-ipfs profiles, I'm worried that memory fragmentation is killing us so I'd like to revert to the old behavior. Note: The real solution here is dealing with "address abusers". --- p2p/host/peerstore/pstoremem/addr_book.go | 68 ++++++++--------------- 1 file changed, 24 insertions(+), 44 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 311c436ccb..1cd1a2a6c8 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -26,21 +26,19 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { return t.After(e.Expires) } -type addrSlice []expiringAddr - var _ pstore.AddrBook = (*memoryAddrBook)(nil) // memoryAddrBook manages addresses. type memoryAddrBook struct { addrmu sync.Mutex - addrs map[peer.ID]addrSlice + addrs map[peer.ID]map[string]expiringAddr subManager *AddrSubManager } func NewAddrBook() pstore.AddrBook { return &memoryAddrBook{ - addrs: make(map[peer.ID]addrSlice), + addrs: make(map[peer.ID]map[string]expiringAddr), subManager: NewAddrSubManager(), } } @@ -76,20 +74,17 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du return } - oldAddrs := mab.addrs[p] - amap := make(map[string]expiringAddr, len(oldAddrs)) - for _, ea := range oldAddrs { - amap[string(ea.Addr.Bytes())] = ea + amap := mab.addrs[p] + if amap == nil { + amap = make(map[string]expiringAddr, len(addrs)) + mab.addrs[p] = amap } - - // only expand ttls exp := time.Now().Add(ttl) for _, addr := range addrs { if addr == nil { log.Warningf("was passed nil multiaddr for %s", p) continue } - addrstr := string(addr.Bytes()) a, found := amap[addrstr] if !found || exp.After(a.Expires) { @@ -98,11 +93,6 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du mab.subManager.BroadcastAddr(p, addr) } } - newAddrs := make([]expiringAddr, 0, len(amap)) - for _, ea := range amap { - newAddrs = append(newAddrs, ea) - } - mab.addrs[p] = newAddrs } // SetAddr calls mgr.SetAddrs(p, addr, ttl) @@ -116,10 +106,10 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du mab.addrmu.Lock() defer mab.addrmu.Unlock() - oldAddrs := mab.addrs[p] - amap := make(map[string]expiringAddr, len(oldAddrs)) - for _, ea := range oldAddrs { - amap[string(ea.Addr.Bytes())] = ea + amap := mab.addrs[p] + if amap == nil { + amap = make(map[string]expiringAddr, len(addrs)) + mab.addrs[p] = amap } exp := time.Now().Add(ttl) @@ -129,21 +119,16 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du continue } // re-set all of them for new ttl. - addrs := string(addr.Bytes()) + addrstr := string(addr.Bytes()) if ttl > 0 { - amap[addrs] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} + amap[addrstr] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} mab.subManager.BroadcastAddr(p, addr) } else { - delete(amap, addrs) + delete(amap, addrstr) } } - newAddrs := make([]expiringAddr, 0, len(amap)) - for _, ea := range amap { - newAddrs = append(newAddrs, ea) - } - mab.addrs[p] = newAddrs } // UpdateAddrs updates the addresses associated with the given peer that have @@ -156,16 +141,17 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t return } - addrs, found := mab.addrs[p] + amap, found := mab.addrs[p] if !found { return } exp := time.Now().Add(newTTL) - for i := range addrs { - if aexp := &addrs[i]; oldTTL == aexp.TTL { - aexp.TTL = newTTL - aexp.Expires = exp + for k, addr := range amap { + if oldTTL == addr.TTL { + addr.TTL = newTTL + addr.Expires = exp + amap[k] = addr } } } @@ -180,27 +166,21 @@ func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { return nil } - maddrs, found := mab.addrs[p] + amap, found := mab.addrs[p] if !found { return nil } now := time.Now() - good := make([]ma.Multiaddr, 0, len(maddrs)) - cleaned := make([]expiringAddr, 0, len(maddrs)) - for _, m := range maddrs { + good := make([]ma.Multiaddr, 0, len(amap)) + for k, m := range amap { if !m.ExpiredBy(now) { - cleaned = append(cleaned, m) good = append(good, m.Addr) + } else { + delete(amap, k) } } - // clean up the expired ones. - if len(cleaned) == 0 { - delete(mab.addrs, p) - } else { - mab.addrs[p] = cleaned - } return good } From a0a98a9d92895eeb88576f2f5a0901687014a47d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 2 Oct 2018 10:49:55 -0700 Subject: [PATCH 0723/3965] add garbage collection for in-memory peerstore There are better ways to do this but pausing dialing once an hour likely isn't going to break anything and is the simplest approach. --- p2p/host/peerstore/pstoremem/addr_book.go | 35 +++++++++++++++-------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 1cd1a2a6c8..e72edd6d22 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -33,6 +33,8 @@ type memoryAddrBook struct { addrmu sync.Mutex addrs map[peer.ID]map[string]expiringAddr + nextGC time.Time + subManager *AddrSubManager } @@ -43,12 +45,27 @@ func NewAddrBook() pstore.AddrBook { } } +func (mab *memoryAddrBook) gc() { + now := time.Now() + if !now.After(mab.nextGC) { + return + } + for p, amap := range mab.addrs { + for k, addr := range amap { + if addr.ExpiredBy(now) { + delete(amap, k) + } + } + if len(amap) == 0 { + delete(mab.addrs, p) + } + } + mab.nextGC = time.Now().Add(pstore.AddressTTL) +} + func (mab *memoryAddrBook) PeersWithAddrs() peer.IDSlice { mab.addrmu.Lock() defer mab.addrmu.Unlock() - if mab.addrs == nil { - return nil - } pids := make(peer.IDSlice, 0, len(mab.addrs)) for pid := range mab.addrs { @@ -93,6 +110,7 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du mab.subManager.BroadcastAddr(p, addr) } } + mab.gc() } // SetAddr calls mgr.SetAddrs(p, addr, ttl) @@ -129,6 +147,7 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du delete(amap, addrstr) } } + mab.gc() } // UpdateAddrs updates the addresses associated with the given peer that have @@ -137,10 +156,6 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t mab.addrmu.Lock() defer mab.addrmu.Unlock() - if mab.addrs == nil { - return - } - amap, found := mab.addrs[p] if !found { return @@ -154,6 +169,7 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t amap[k] = addr } } + mab.gc() } // Addresses returns all known (and valid) addresses for a given @@ -161,11 +177,6 @@ func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { mab.addrmu.Lock() defer mab.addrmu.Unlock() - // not initialized? nothing to give. - if mab.addrs == nil { - return nil - } - amap, found := mab.addrs[p] if !found { return nil From c640cd812b60b1cf14f841c908785f9374610748 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Oct 2018 16:15:58 +0300 Subject: [PATCH 0724/3965] add observed address to the dialback set --- p2p/host/autonat/svc.go | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index b25ead6eb1..479629f0f3 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -78,7 +78,7 @@ func (as *AutoNATService) handleStream(s inet.Stream) { return } - dr := as.handleDial(pid, req.GetDial().GetPeer()) + dr := as.handleDial(pid, s.Conn().RemoteMultiaddr(), req.GetDial().GetPeer()) res.Type = pb.Message_DIAL_RESPONSE.Enum() res.DialResponse = dr @@ -90,7 +90,7 @@ func (as *AutoNATService) handleStream(s inet.Stream) { } } -func (as *AutoNATService) handleDial(p peer.ID, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { +func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { if mpi == nil { return newDialResponseError(pb.Message_E_BAD_REQUEST, "missing peer info") } @@ -108,6 +108,14 @@ func (as *AutoNATService) handleDial(p peer.ID, mpi *pb.Message_PeerInfo) *pb.Me } addrs := make([]ma.Multiaddr, 0) + seen := make(map[string]struct{}) + + // add observed addr to the list of addresses to dial + if !as.skipDial(obsaddr) { + addrs = append(addrs, obsaddr) + seen[obsaddr.String()] = struct{}{} + } + for _, maddr := range mpi.GetAddrs() { addr, err := ma.NewMultiaddrBytes(maddr) if err != nil { @@ -115,18 +123,18 @@ func (as *AutoNATService) handleDial(p peer.ID, mpi *pb.Message_PeerInfo) *pb.Me continue } - // skip relay addresses - _, err = addr.ValueForProtocol(P_CIRCUIT) - if err == nil { + if as.skipDial(addr) { continue } - // skip private network (unroutable) addresses - if !isPublicAddr(addr) { + str := addr.String() + _, ok := seen[str] + if ok { continue } addrs = append(addrs, addr) + seen[str] = struct{}{} } if len(addrs) == 0 { @@ -136,6 +144,21 @@ func (as *AutoNATService) handleDial(p peer.ID, mpi *pb.Message_PeerInfo) *pb.Me return as.doDial(pstore.PeerInfo{ID: p, Addrs: addrs}) } +func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { + // skip relay addresses + _, err := addr.ValueForProtocol(P_CIRCUIT) + if err == nil { + return true + } + + // skip private network (unroutable) addresses + if !isPublicAddr(addr) { + return true + } + + return false +} + func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { // rate limit check as.mx.Lock() From 8c66f57ec6347466d63926c153cfdab2c5dab4ed Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Oct 2018 16:22:58 +0300 Subject: [PATCH 0725/3965] ensure test hosts are only on loopback --- p2p/host/autonat/autonat_test.go | 2 +- p2p/host/autonat/svc_test.go | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 019a5906b5..570c4a269c 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -11,7 +11,7 @@ import ( ) func makeAutoNAT(ctx context.Context, t *testing.T) (host.Host, AutoNAT) { - h, err := libp2p.New(ctx) + h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) if err != nil { t.Fatal(err) } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 77d482be83..ba2395b2f5 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -12,7 +12,7 @@ import ( ) func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATService) { - h, err := libp2p.New(ctx) + h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) if err != nil { t.Fatal(err) } @@ -26,7 +26,7 @@ func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATS } func makeAutoNATClient(ctx context.Context, t *testing.T) (host.Host, AutoNATClient) { - h, err := libp2p.New(ctx) + h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) if err != nil { t.Fatal(err) } From cb0e1930e3bd1501a3825b84dd764abafdfcddfa Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Oct 2018 16:28:21 +0300 Subject: [PATCH 0726/3965] add /libp2p prefix in protocol string --- p2p/host/autonat/proto.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/proto.go b/p2p/host/autonat/proto.go index 7d28d4bdb6..c9768cbdbd 100644 --- a/p2p/host/autonat/proto.go +++ b/p2p/host/autonat/proto.go @@ -8,7 +8,7 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -const AutoNATProto = "/autonat/1.0.0" +const AutoNATProto = "/libp2p/autonat/1.0.0" var log = logging.Logger("autonat") From 88ad7b74ec183c46427660eb4e2d5338876e06ab Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Oct 2018 19:41:08 +0300 Subject: [PATCH 0727/3965] configurable throttle for service rate limiter --- p2p/host/autonat/svc.go | 15 +++++++++------ p2p/host/autonat/svc_test.go | 7 +++++-- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 479629f0f3..95c66bdd82 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -21,6 +21,8 @@ const P_CIRCUIT = 290 var ( AutoNATServiceDialTimeout = 42 * time.Second AutoNATServiceResetInterval = 1 * time.Minute + + AutoNATServiceThrottle = 3 ) // AutoNATService provides NAT autodetection services to other peers @@ -28,8 +30,9 @@ type AutoNATService struct { ctx context.Context dialer host.Host + // rate limiter mx sync.Mutex - peers map[peer.ID]struct{} + peers map[peer.ID]int } // NewAutoNATService creates a new AutoNATService instance attached to a host @@ -43,7 +46,7 @@ func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) as := &AutoNATService{ ctx: ctx, dialer: dialer, - peers: make(map[peer.ID]struct{}), + peers: make(map[peer.ID]int), } h.SetStreamHandler(AutoNATProto, as.handleStream) @@ -162,12 +165,12 @@ func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { // rate limit check as.mx.Lock() - _, ok := as.peers[pi.ID] - if ok { + count := as.peers[pi.ID] + if count >= AutoNATServiceThrottle { as.mx.Unlock() return newDialResponseError(pb.Message_E_DIAL_REFUSED, "too many dials") } - as.peers[pi.ID] = struct{}{} + as.peers[pi.ID] = count + 1 as.mx.Unlock() ctx, cancel := context.WithTimeout(as.ctx, AutoNATServiceDialTimeout) @@ -201,7 +204,7 @@ func (as *AutoNATService) resetPeers() { select { case <-ticker.C: as.mx.Lock() - as.peers = make(map[peer.ID]struct{}) + as.peers = make(map[peer.ID]int) as.mx.Unlock() case <-as.ctx.Done(): diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index ba2395b2f5..cfa4068bfa 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -94,7 +94,9 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { AutoNATServiceDialTimeout = 1 * time.Second save2 := AutoNATServiceResetInterval AutoNATServiceResetInterval = 1 * time.Second - save3 := private4 + save3 := AutoNATServiceThrottle + AutoNATServiceThrottle = 1 + save4 := private4 private4 = []*net.IPNet{} hs, _ := makeAutoNATService(ctx, t) @@ -124,5 +126,6 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { AutoNATServiceDialTimeout = save1 AutoNATServiceResetInterval = save2 - private4 = save3 + AutoNATServiceThrottle = save3 + private4 = save4 } From 2d5d58ce51b5e5cde93243b6e2d6ae8bbb17d308 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Oct 2018 19:52:42 +0300 Subject: [PATCH 0728/3965] call AutoNATService.peers something else (reqs) --- p2p/host/autonat/svc.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 95c66bdd82..4d395d22ec 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -31,8 +31,8 @@ type AutoNATService struct { dialer host.Host // rate limiter - mx sync.Mutex - peers map[peer.ID]int + mx sync.Mutex + reqs map[peer.ID]int } // NewAutoNATService creates a new AutoNATService instance attached to a host @@ -46,11 +46,11 @@ func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) as := &AutoNATService{ ctx: ctx, dialer: dialer, - peers: make(map[peer.ID]int), + reqs: make(map[peer.ID]int), } h.SetStreamHandler(AutoNATProto, as.handleStream) - go as.resetPeers() + go as.resetRateLimiter() return as, nil } @@ -165,12 +165,12 @@ func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { // rate limit check as.mx.Lock() - count := as.peers[pi.ID] + count := as.reqs[pi.ID] if count >= AutoNATServiceThrottle { as.mx.Unlock() return newDialResponseError(pb.Message_E_DIAL_REFUSED, "too many dials") } - as.peers[pi.ID] = count + 1 + as.reqs[pi.ID] = count + 1 as.mx.Unlock() ctx, cancel := context.WithTimeout(as.ctx, AutoNATServiceDialTimeout) @@ -196,7 +196,7 @@ func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { return newDialResponseOK(ra) } -func (as *AutoNATService) resetPeers() { +func (as *AutoNATService) resetRateLimiter() { ticker := time.NewTicker(AutoNATServiceResetInterval) defer ticker.Stop() @@ -204,7 +204,7 @@ func (as *AutoNATService) resetPeers() { select { case <-ticker.C: as.mx.Lock() - as.peers = make(map[peer.ID]int) + as.reqs = make(map[peer.ID]int) as.mx.Unlock() case <-as.ctx.Done(): From 3390cc400110350930b429c5459ad16dd08149af Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 3 Oct 2018 19:03:13 -0400 Subject: [PATCH 0729/3965] Count hops --- .../internal/circuitv1-deprecated/relay.go | 28 ++++++++++++------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 51dc1c10a5..ec88bd82d3 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -41,7 +41,7 @@ type Relay struct { relays map[peer.ID]struct{} mx sync.Mutex - liveHops map[peer.ID]map[peer.ID]struct{} + liveHops map[peer.ID]map[peer.ID]int lhCount uint64 lhLk sync.Mutex } @@ -69,7 +69,7 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. self: h.ID(), incoming: make(chan *Conn), relays: make(map[peer.ID]struct{}), - liveHops: make(map[peer.ID]map[peer.ID]struct{}), + liveHops: make(map[peer.ID]map[peer.ID]int), } for _, opt := range opts { @@ -95,15 +95,17 @@ func (r *Relay) addLiveHop(from, to peer.ID) { trg, ok := r.liveHops[from] if !ok { - trg = make(map[peer.ID]struct{}) + trg = make(map[peer.ID]int) r.liveHops[from] = trg } - if _, ok := trg[to]; ok { - return + var count int + if count, ok = trg[to]; !ok { + count = 0 } + count++ r.lhCount++ - trg[to] = struct{}{} + trg[to] = count } func (r *Relay) rmLiveHop(from, to peer.ID) { @@ -114,14 +116,20 @@ func (r *Relay) rmLiveHop(from, to peer.ID) { if !ok { return } - if _, ok := trg[to]; !ok { + var count int + if count, ok = trg[to]; !ok { return } + count-- r.lhCount-- - delete(trg, to) - if len(trg) == 0 { - delete(r.liveHops, from) + if count <= 0 { + delete(trg, to) + if len(trg) == 0 { + delete(r.liveHops, from) + } + } else { + trg[to] = count } } From 708455127bbd4ef3b6c63ee544850a9a20981947 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 3 Oct 2018 17:42:54 -0700 Subject: [PATCH 0730/3965] memoryAddrBook: document gc precondition --- p2p/host/peerstore/pstoremem/addr_book.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index e72edd6d22..f0f8a2c87c 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -45,6 +45,7 @@ func NewAddrBook() pstore.AddrBook { } } +// gc garbage collects the in-memory address book. The caller *must* hold the addrmu lock. func (mab *memoryAddrBook) gc() { now := time.Now() if !now.After(mab.nextGC) { From 0cc0faf7eb62fbf7b909d55538256fbc7176c7ea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 4 Oct 2018 10:50:45 +0100 Subject: [PATCH 0731/3965] run tests against multiple datastores. leveldb disabled for now, until TTL shim is implemented. --- p2p/host/peerstore/pstoreds/ds_test.go | 201 +++++++++---------------- 1 file changed, 72 insertions(+), 129 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 2eb3f995d4..29ac9eb04d 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -2,170 +2,92 @@ package pstoreds import ( "context" - "fmt" "io/ioutil" "os" "testing" "time" - bd "github.com/dgraph-io/badger" - ds "github.com/ipfs/go-datastore" badger "github.com/ipfs/go-ds-badger" + leveldb "github.com/ipfs/go-ds-leveldb" pstore "github.com/libp2p/go-libp2p-peerstore" pt "github.com/libp2p/go-libp2p-peerstore/test" ) -func BenchmarkBaselineBadgerDatastorePutEntry(b *testing.B) { - bds, closer := badgerStore(b) - defer closer() - - b.ResetTimer() - for i := 0; i < b.N; i++ { - txn, _ := bds.NewTransaction(false) +type datastoreFactory func(tb testing.TB) (ds.TxnDatastore, func()) - key := ds.RawKey(fmt.Sprintf("/key/%d", i)) - txn.Put(key, []byte(fmt.Sprintf("/value/%d", i))) - - txn.Commit() - txn.Discard() - } +var dstores = map[string]datastoreFactory{ + "Badger": badgerStore, + // TODO: Enable once go-ds-leveldb supports TTL via a shim. + // "Leveldb": leveldbStore, } -func BenchmarkBaselineBadgerDatastoreGetEntry(b *testing.B) { - bds, closer := badgerStore(b) - defer closer() - - txn, _ := bds.NewTransaction(false) - keys := make([]ds.Key, 1000) - for i := 0; i < 1000; i++ { - key := ds.RawKey(fmt.Sprintf("/key/%d", i)) - txn.Put(key, []byte(fmt.Sprintf("/value/%d", i))) - keys[i] = key - } - if err := txn.Commit(); err != nil { - b.Fatal(err) - } - - b.ResetTimer() - for i := 0; i < b.N; i++ { - txn, _ := bds.NewTransaction(true) - if _, err := txn.Get(keys[i%1000]); err != nil { - b.Fatal(err) - } - txn.Discard() +func TestDsPeerstore(t *testing.T) { + for name, dsFactory := range dstores { + t.Run(name, func(t *testing.T) { + pt.TestPeerstore(t, peerstoreFactory(t, dsFactory, DefaultOpts())) + }) } } -func BenchmarkBaselineBadgerDirectPutEntry(b *testing.B) { - opts := bd.DefaultOptions +func TestDsAddrBook(t *testing.T) { + for name, dsFactory := range dstores { + t.Run(name, func(t *testing.T) { + t.Run("Cacheful", func(t *testing.T) { + t.Parallel() - dataPath, err := ioutil.TempDir(os.TempDir(), "badger") - if err != nil { - b.Fatal(err) - } + opts := DefaultOpts() + opts.TTLInterval = 100 * time.Microsecond + opts.CacheSize = 1024 - opts.Dir = dataPath - opts.ValueDir = dataPath - opts.SyncWrites = false + pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) + }) - db, err := bd.Open(opts) - if err != nil { - b.Fatal(err) - } + t.Run("Cacheless", func(t *testing.T) { + t.Parallel() - defer db.Close() + opts := DefaultOpts() + opts.TTLInterval = 100 * time.Microsecond + opts.CacheSize = 0 - b.ResetTimer() - for i := 0; i < b.N; i++ { - txn := db.NewTransaction(true) - txn.Set([]byte(fmt.Sprintf("/key/%d", i)), []byte(fmt.Sprintf("/value/%d", i))) - txn.Commit(nil) + pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) + }) + }) } } -func BenchmarkBaselineBadgerDirectGetEntry(b *testing.B) { - opts := bd.DefaultOptions - - dataPath, err := ioutil.TempDir(os.TempDir(), "badger") - if err != nil { - b.Fatal(err) - } - - opts.Dir = dataPath - opts.ValueDir = dataPath - - db, err := bd.Open(opts) - if err != nil { - b.Fatal(err) +func TestDsKeyBook(t *testing.T) { + for name, dsFactory := range dstores { + t.Run(name, func(t *testing.T) { + pt.TestKeyBook(t, keyBookFactory(t, dsFactory, DefaultOpts())) + }) } - - defer db.Close() - - txn := db.NewTransaction(true) - for i := 0; i < 1000; i++ { - txn.Set([]byte(fmt.Sprintf("/key/%d", i)), []byte(fmt.Sprintf("/value/%d", i))) - } - txn.Commit(nil) - - b.ResetTimer() - for i := 0; i < b.N; i++ { - txn := db.NewTransaction(false) - txn.Get([]byte(fmt.Sprintf("/key/%d", i%1000))) - txn.Discard() - } -} - -func TestBadgerDsPeerstore(t *testing.T) { - pt.TestPeerstore(t, peerstoreFactory(t, DefaultOpts())) -} - -func TestBadgerDsAddrBook(t *testing.T) { - t.Run("Cacheful", func(t *testing.T) { - t.Parallel() - - opts := DefaultOpts() - opts.TTLInterval = 100 * time.Microsecond - opts.CacheSize = 1024 - - pt.TestAddrBook(t, addressBookFactory(t, opts)) - }) - - t.Run("Cacheless", func(t *testing.T) { - t.Parallel() - - opts := DefaultOpts() - opts.TTLInterval = 100 * time.Microsecond - opts.CacheSize = 0 - - pt.TestAddrBook(t, addressBookFactory(t, opts)) - }) -} - -func TestBadgerDsKeyBook(t *testing.T) { - pt.TestKeyBook(t, keyBookFactory(t, DefaultOpts())) } -func BenchmarkBadgerDsPeerstore(b *testing.B) { +func BenchmarkDsPeerstore(b *testing.B) { caching := DefaultOpts() caching.CacheSize = 1024 cacheless := DefaultOpts() cacheless.CacheSize = 0 - pt.BenchmarkPeerstore(b, peerstoreFactory(b, caching), "Caching") - pt.BenchmarkPeerstore(b, peerstoreFactory(b, cacheless), "Cacheless") + for name, dsFactory := range dstores { + b.Run(name, func(b *testing.B) { + pt.BenchmarkPeerstore(b, peerstoreFactory(b, dsFactory, caching), "Caching") + pt.BenchmarkPeerstore(b, peerstoreFactory(b, dsFactory, cacheless), "Cacheless") + }) + } } -func badgerStore(t testing.TB) (ds.TxnDatastore, func()) { +func badgerStore(tb testing.TB) (ds.TxnDatastore, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { - t.Fatal(err) + tb.Fatal(err) } store, err := badger.NewDatastore(dataPath, nil) if err != nil { - t.Fatal(err) + tb.Fatal(err) } closer := func() { store.Close() @@ -174,9 +96,25 @@ func badgerStore(t testing.TB) (ds.TxnDatastore, func()) { return store, closer } -func peerstoreFactory(tb testing.TB, opts Options) pt.PeerstoreFactory { +func leveldbStore(tb testing.TB) (ds.TxnDatastore, func()) { + dataPath, err := ioutil.TempDir(os.TempDir(), "leveldb") + if err != nil { + tb.Fatal(err) + } + store, err := leveldb.NewDatastore(dataPath, nil) + if err != nil { + tb.Fatal(err) + } + closer := func() { + store.Close() + os.RemoveAll(dataPath) + } + return store, closer +} + +func peerstoreFactory(tb testing.TB, storeFactory datastoreFactory, opts Options) pt.PeerstoreFactory { return func() (pstore.Peerstore, func()) { - store, closeFunc := badgerStore(tb) + store, closeFunc := storeFactory(tb) ps, err := NewPeerstore(context.Background(), store, opts) if err != nil { @@ -187,9 +125,9 @@ func peerstoreFactory(tb testing.TB, opts Options) pt.PeerstoreFactory { } } -func addressBookFactory(tb testing.TB, opts Options) pt.AddrBookFactory { +func addressBookFactory(tb testing.TB, storeFactory datastoreFactory, opts Options) pt.AddrBookFactory { return func() (pstore.AddrBook, func()) { - store, closeFunc := badgerStore(tb) + store, closeFunc := storeFactory(tb) ab, err := NewAddrBook(context.Background(), store, opts) if err != nil { @@ -200,10 +138,15 @@ func addressBookFactory(tb testing.TB, opts Options) pt.AddrBookFactory { } } -func keyBookFactory(tb testing.TB, opts Options) pt.KeyBookFactory { +func keyBookFactory(tb testing.TB, storeFactory datastoreFactory, opts Options) pt.KeyBookFactory { return func() (pstore.KeyBook, func()) { - store, closeFunc := badgerStore(tb) - kb, _ := NewKeyBook(context.Background(), store, opts) + store, closeFunc := storeFactory(tb) + + kb, err := NewKeyBook(context.Background(), store, opts) + if err != nil { + tb.Fatal(err) + } + return kb, closeFunc } } From c95b4a88d2d5aec89171cb4406e958b5cf656a29 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 4 Oct 2018 10:51:57 +0100 Subject: [PATCH 0732/3965] ds key encoding: b58 => b32 (no padding). --- p2p/host/peerstore/pstoreds/addr_book.go | 13 +++++++------ p2p/host/peerstore/pstoreds/keybook.go | 12 +++++++----- p2p/host/peerstore/pstoreds/metadata.go | 8 +++++--- p2p/host/peerstore/pstoreds/peerstore.go | 6 ++++-- 4 files changed, 23 insertions(+), 16 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index ad37529a5c..9e92336c45 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -7,6 +7,7 @@ import ( "time" lru "github.com/hashicorp/golang-lru" + base32 "github.com/whyrusleeping/base32" ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" @@ -31,7 +32,7 @@ var ( ) // Peer addresses are stored under the following db key pattern: -// /peers/addr// +// /peers/addr// var abBase = ds.NewKey("/peers/addrs") var _ pstore.AddrBook = (*dsAddrBook)(nil) @@ -107,7 +108,7 @@ func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, er var ( keys = make([]ds.Key, len(addrs)) clean = make([]ma.Multiaddr, len(addrs)) - parentKey = abBase.ChildString(peer.IDB58Encode(p)) + parentKey = abBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))) i = 0 ) @@ -120,7 +121,7 @@ func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, er if err != nil { return nil, nil, err } - keys[i] = parentKey.ChildString(hash.B58String()) + keys[i] = parentKey.ChildString(base32.RawStdEncoding.EncodeToString(hash)) clean[i] = addr i++ } @@ -293,7 +294,7 @@ func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time. func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time.Duration) error { var ( - prefix = abBase.ChildString(peer.IDB58Encode(p)) + prefix = abBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))) q = query.Query{Prefix: prefix.String(), KeysOnly: false} results query.Results err error @@ -336,7 +337,7 @@ func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time. // Addrs returns all of the non-expired addresses for a given peer. func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { var ( - prefix = abBase.ChildString(peer.IDB58Encode(p)) + prefix = abBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))) q = query.Query{Prefix: prefix.String(), KeysOnly: false, ReturnExpirations: true} results query.Results err error @@ -411,7 +412,7 @@ func (mgr *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Mult func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { var ( err error - prefix = abBase.ChildString(peer.IDB58Encode(p)) + prefix = abBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))) deleteFn func() error ) diff --git a/p2p/host/peerstore/pstoreds/keybook.go b/p2p/host/peerstore/pstoreds/keybook.go index 4801e6b338..307f7dfd4a 100644 --- a/p2p/host/peerstore/pstoreds/keybook.go +++ b/p2p/host/peerstore/pstoreds/keybook.go @@ -4,6 +4,8 @@ import ( "context" "errors" + base32 "github.com/whyrusleeping/base32" + ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" @@ -13,7 +15,7 @@ import ( ) // Public and private keys are stored under the following db key pattern: -// /peers/keys//{pub, priv} +// /peers/keys//{pub, priv} var ( kbBase = ds.NewKey("/peers/keys") pubSuffix = ds.NewKey("/pub") @@ -31,7 +33,7 @@ func NewKeyBook(_ context.Context, store ds.TxnDatastore, _ Options) (pstore.Key } func (kb *dsKeyBook) PubKey(p peer.ID) ic.PubKey { - key := kbBase.ChildString(peer.IDB58Encode(p)).Child(pubSuffix) + key := kbBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))).Child(pubSuffix) var pk ic.PubKey if value, err := kb.ds.Get(key); err == nil { @@ -68,7 +70,7 @@ func (kb *dsKeyBook) AddPubKey(p peer.ID, pk ic.PubKey) error { return errors.New("peer ID does not match public key") } - key := kbBase.ChildString(peer.IDB58Encode(p)).Child(pubSuffix) + key := kbBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))).Child(pubSuffix) val, err := pk.Bytes() if err != nil { log.Errorf("error while converting pubkey byte string for peer %s: %s\n", p.Pretty(), err) @@ -82,7 +84,7 @@ func (kb *dsKeyBook) AddPubKey(p peer.ID, pk ic.PubKey) error { } func (kb *dsKeyBook) PrivKey(p peer.ID) ic.PrivKey { - key := kbBase.ChildString(peer.IDB58Encode(p)).Child(privSuffix) + key := kbBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))).Child(privSuffix) value, err := kb.ds.Get(key) if err != nil { log.Errorf("error while fetching privkey from datastore for peer %s: %s\n", p.Pretty(), err) @@ -104,7 +106,7 @@ func (kb *dsKeyBook) AddPrivKey(p peer.ID, sk ic.PrivKey) error { return errors.New("peer ID does not match private key") } - key := kbBase.ChildString(peer.IDB58Encode(p)).Child(privSuffix) + key := kbBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))).Child(privSuffix) val, err := sk.Bytes() if err != nil { log.Errorf("error while converting privkey byte string for peer %s: %s\n", p.Pretty(), err) diff --git a/p2p/host/peerstore/pstoreds/metadata.go b/p2p/host/peerstore/pstoreds/metadata.go index de21434d6e..7c1d124f73 100644 --- a/p2p/host/peerstore/pstoreds/metadata.go +++ b/p2p/host/peerstore/pstoreds/metadata.go @@ -5,6 +5,8 @@ import ( "context" "encoding/gob" + base32 "github.com/whyrusleeping/base32" + ds "github.com/ipfs/go-datastore" pool "github.com/libp2p/go-buffer-pool" @@ -13,7 +15,7 @@ import ( ) // Metadata is stored under the following db key pattern: -// /peers/metadata// +// /peers/metadata// var pmBase = ds.NewKey("/peers/metadata") type dsPeerMetadata struct { @@ -39,7 +41,7 @@ func NewPeerMetadata(_ context.Context, store ds.Datastore, _ Options) (pstore.P } func (pm *dsPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { - k := pmBase.ChildString(peer.IDB58Encode(p)).ChildString(key) + k := pmBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))).ChildString(key) value, err := pm.ds.Get(k) if err != nil { if err == ds.ErrNotFound { @@ -56,7 +58,7 @@ func (pm *dsPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { } func (pm *dsPeerMetadata) Put(p peer.ID, key string, val interface{}) error { - k := pmBase.ChildString(peer.IDB58Encode(p)).ChildString(key) + k := pmBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))).ChildString(key) var buf pool.Buffer if err := gob.NewEncoder(&buf).Encode(&val); err != nil { return err diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index cc3c6f88ab..b1a64f076e 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -4,6 +4,8 @@ import ( "context" "time" + base32 "github.com/whyrusleeping/base32" + ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" @@ -91,8 +93,8 @@ func uniquePeerIds(ds ds.TxnDatastore, prefix ds.Key, extractor func(result quer ids := make(peer.IDSlice, len(idset)) i := 0 for id := range idset { - pid, _ := peer.IDB58Decode(id) - ids[i] = pid + pid, _ := base32.RawStdEncoding.DecodeString(id) + ids[i], _ = peer.IDFromBytes(pid) i++ } return ids, nil From 5ada32f26c9f796b674b0759469284a827403b77 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 4 Oct 2018 13:37:51 +0300 Subject: [PATCH 0733/3965] use more peer dial errors for increased confidence in NATPrivate state 3 times is enemy action; this is more resilient to transient dial errors. --- p2p/host/autonat/autonat.go | 45 ++++++++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 037a094f56..e006900fc3 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -29,6 +29,7 @@ const ( var ( AutoNATBootDelay = 15 * time.Second + AutoNATRetryInterval = 60 * time.Second AutoNATRefreshInterval = 15 * time.Minute AutoNATRequestTimeout = 60 * time.Second ) @@ -47,10 +48,11 @@ type AmbientAutoNAT struct { ctx context.Context host host.Host - mx sync.Mutex - peers map[peer.ID]struct{} - status NATStatus - addr ma.Multiaddr + mx sync.Mutex + peers map[peer.ID]struct{} + status NATStatus + addr ma.Multiaddr + confidence int } // NewAutoNAT creates a new ambient NAT autodiscovery instance attached to a host @@ -94,8 +96,14 @@ func (as *AmbientAutoNAT) background() { for { as.autodetect() + + delay := AutoNATRefreshInterval + if as.status == NATStatusUnknown { + delay = AutoNATRetryInterval + } + select { - case <-time.After(AutoNATRefreshInterval): + case <-time.After(delay): case <-as.ctx.Done(): return } @@ -111,6 +119,7 @@ func (as *AmbientAutoNAT) autodetect() { } cli := NewAutoNATClient(as.host) + failures := 0 for _, p := range peers { ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) @@ -123,15 +132,21 @@ func (as *AmbientAutoNAT) autodetect() { as.mx.Lock() as.addr = a as.status = NATStatusPublic + as.confidence = 0 as.mx.Unlock() return case IsDialError(err): - log.Debugf("NAT status is private; dial error through %s: %s", p.Pretty(), err.Error()) - as.mx.Lock() - as.status = NATStatusPrivate - as.mx.Unlock() - return + log.Debugf("dial error through %s: %s", p.Pretty(), err.Error()) + failures++ + if failures >= 3 || as.confidence >= 3 { // 3 times is enemy action + log.Debugf("NAT status is private") + as.mx.Lock() + as.status = NATStatusPrivate + as.confidence = 3 + as.mx.Unlock() + return + } default: log.Debugf("Error dialing through %s: %s", p.Pretty(), err.Error()) @@ -139,7 +154,15 @@ func (as *AmbientAutoNAT) autodetect() { } as.mx.Lock() - as.status = NATStatusUnknown + if failures > 0 { + as.status = NATStatusPrivate + as.confidence++ + log.Debugf("NAT status is private") + } else { + as.status = NATStatusUnknown + as.confidence = 0 + log.Debugf("NAT status is unknown") + } as.mx.Unlock() } From ab670969010a8b2fd69f7f55bf54e3ebb4f450ed Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 4 Oct 2018 13:46:39 +0300 Subject: [PATCH 0734/3965] use more peers if less than 3 are connected --- p2p/host/autonat/autonat.go | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index e006900fc3..4f766ce7c9 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -29,7 +29,7 @@ const ( var ( AutoNATBootDelay = 15 * time.Second - AutoNATRetryInterval = 60 * time.Second + AutoNATRetryInterval = 90 * time.Second AutoNATRefreshInterval = 15 * time.Minute AutoNATRequestTimeout = 60 * time.Second ) @@ -174,23 +174,24 @@ func (as *AmbientAutoNAT) getPeers() []peer.ID { return nil } - peers := make([]peer.ID, 0, len(as.peers)) + var connected, others []peer.ID + for p := range as.peers { if as.host.Network().Connectedness(p) == inet.Connected { - peers = append(peers, p) + connected = append(connected, p) + } else { + others = append(others, p) } } - if len(peers) == 0 { - // we don't have any open connections, try any autonat peer that we know about - for p := range as.peers { - peers = append(peers, p) - } - } + shufflePeers(connected) - shufflePeers(peers) - - return peers + if len(connected) < 3 { + shufflePeers(others) + return append(connected, others...) + } else { + return connected + } } func shufflePeers(peers []peer.ID) { From bff416bf7e21792001adea6a946c7719dd30ca4a Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 4 Oct 2018 16:57:02 +0300 Subject: [PATCH 0735/3965] adjust AutoNATRetryInterval in autonat tests --- p2p/host/autonat/autonat_test.go | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 570c4a269c..ac438f8482 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -30,6 +30,8 @@ func TestAutoNATPrivate(t *testing.T) { AutoNATBootDelay = 1 * time.Second save2 := AutoNATRefreshInterval AutoNATRefreshInterval = 1 * time.Second + save3 := AutoNATRetryInterval + AutoNATRetryInterval = 1 * time.Second hs, _ := makeAutoNATService(ctx, t) hc, an := makeAutoNAT(ctx, t) @@ -49,6 +51,7 @@ func TestAutoNATPrivate(t *testing.T) { AutoNATBootDelay = save1 AutoNATRefreshInterval = save2 + AutoNATRetryInterval = save3 } func TestAutoNATPublic(t *testing.T) { @@ -59,7 +62,9 @@ func TestAutoNATPublic(t *testing.T) { AutoNATBootDelay = 1 * time.Second save2 := AutoNATRefreshInterval AutoNATRefreshInterval = 1 * time.Second - save3 := private4 + save3 := AutoNATRetryInterval + AutoNATRetryInterval = 1 * time.Second + save4 := private4 private4 = []*net.IPNet{} hs, _ := makeAutoNATService(ctx, t) @@ -80,5 +85,6 @@ func TestAutoNATPublic(t *testing.T) { AutoNATBootDelay = save1 AutoNATRefreshInterval = save2 - private4 = save3 + AutoNATRetryInterval = save3 + private4 = save4 } From 9c7d8298e3c21a7be4049bba6538aa953748bf42 Mon Sep 17 00:00:00 2001 From: Kevin Atkinson Date: Thu, 4 Oct 2018 18:42:18 -0400 Subject: [PATCH 0736/3965] gx publish 6.0.14 --- .gx/lastpubver | 2 +- package.json | 62 +++++++++++++++++++++++++------------------------- 2 files changed, 32 insertions(+), 32 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 5aaaf8d507..8645b19e54 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.13: QmVsVARb86uSe1qYouewFMNd2p2sp2NGWm1JGPReVDWchW +6.0.14: Qmd9zWxAeeDJoLdxqvaDXAGtoafX5cc9Tp25DNm9W7fVnB diff --git a/package.json b/package.json index 7c55a90498..21853804ca 100644 --- a/package.json +++ b/package.json @@ -61,27 +61,27 @@ }, { "author": "whyrusleeping", - "hash": "QmPJ6U14u5D2abGVkF3dopCeM1JqigVTUJ7jbAHjTnGZrt", + "hash": "QmUdVuLDQCQD4WPBpgDsbBAaqui3GW4d7exBHK2PUKRQpv", "name": "go-libp2p-secio", - "version": "2.0.10" + "version": "2.0.11" }, { "author": "whyrusleeping", - "hash": "QmfAQMFpgDU2U4BXG64qVr8HSiictfWvkSBz7Y2oDj65st", + "hash": "QmSJ36wcYQyEViJUWUEhJU81tw1KdakTKqLLHbvYbA9zDv", "name": "go-libp2p-peerstore", - "version": "2.0.1" + "version": "2.0.2" }, { "author": "whyrusleeping", - "hash": "QmUFARWmz7LKUCzoWn1M3Z4Q2F8NK319452CBVFu4AgwzL", + "hash": "QmRJFmMd5ed1fy5pGLfmWPGGSb1FFm5tJm62euuFPBQXjk", "name": "go-libp2p-transport", - "version": "3.0.10" + "version": "3.0.11" }, { "author": "whyrusleeping", - "hash": "QmY3ymwbfNPb9KirX2DpgjpB4FaSGQ65edP9kSKyCHUtKQ", + "hash": "Qmf61d1En1KVfwc6A3fn4AYvSCKx6LiMivG5Ljyok1CXTW", "name": "go-tcp-transport", - "version": "2.0.10" + "version": "2.0.11" }, { "author": "whyrusleeping", @@ -103,9 +103,9 @@ }, { "author": "whyrusleeping", - "hash": "QmfDPh144WGBqRxZb1TGDHerbMnZATrHZggAPw7putNnBq", + "hash": "QmQdLcvoy3JuSqhV6iwQ9T6Cv7hWLAdzob4jUZRPqFL67Z", "name": "go-libp2p-net", - "version": "3.0.10" + "version": "3.0.11" }, { "author": "whyrusleeping", @@ -115,15 +115,15 @@ }, { "author": "whyrusleeping", - "hash": "QmeA5hsqgLryvkeyqeQdvGDqurLkYi3XEPLZP3pzuBJXh2", + "hash": "QmX9pw5dSUZ2FozbppcSDJiS7eEh1RFwJNwrbmyLoUMS9x", "name": "go-libp2p-host", - "version": "3.0.10" + "version": "3.0.11" }, { "author": "whyrusleeping", - "hash": "QmPQoCVRHaGD25VffyB7DFV5qP65hFSQJdSDy75P1vYBKe", + "hash": "QmcyQj1V6Ht8uSrRqj865UXGUo5Sc8GxNA4U8bLQxCSmfX", "name": "go-libp2p-swarm", - "version": "3.0.12" + "version": "3.0.14" }, { "author": "whyrusleeping", @@ -133,15 +133,15 @@ }, { "author": "whyrusleeping", - "hash": "QmQgzDvAmKDGGthTLcqzF7VrxtzFbmH7946TwjkX9ruB2Q", + "hash": "Qmek9Aad1NQUz3Fb1gXn6F1EuGrV4XoDQ33jEU3Wvt3XPv", "name": "go-libp2p-netutil", - "version": "0.4.8" + "version": "0.4.9" }, { "author": "whyrusleeping", - "hash": "QmPBiDDeTyHpKypigUYcGivr6Lvo9GjoA7HeAGjoMyKHUK", + "hash": "QmSG4P432Eomm1s8rkgUuBk9Nzp8YjY35RUcJTGT2Huwki", "name": "go-libp2p-blankhost", - "version": "0.3.10" + "version": "0.3.11" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "QmS5X31wA86RSjQqCUHG2CLPoFNnBZ3UMy1ikfBHQ3G6LG", + "hash": "QmeBpq5pAf2yaVdbpaKVcijEXu1QVjnDqPqYq4MACaEHLo", "name": "go-libp2p-circuit", - "version": "2.2.2" + "version": "2.2.3" }, { "author": "lgierth", @@ -181,9 +181,9 @@ }, { "author": "why", - "hash": "QmV9T3rTezwJhMJQBzVrGj2tncJwv23dTKdxzXrUxAvWFi", + "hash": "QmYbLGD41kmPZfgGs5B8DMMwxtY61Wz4KPeGFZ2qoetsPY", "name": "go-libp2p-interface-connmgr", - "version": "0.0.16" + "version": "0.0.17" }, { "author": "whyrusleeping", @@ -193,21 +193,21 @@ }, { "author": "whyrusleeping", - "hash": "QmXawDNdHwcY8yf1AmuRCYrFgJAy4NiXY3RMZJ45XcNqRv", + "hash": "QmVKVakzxomMUPZMivoWTuWzRUv5VngfaWNmmqrAgaSc8S", "name": "go-ws-transport", - "version": "2.0.10" + "version": "2.0.11" }, { "author": "stebalien", - "hash": "QmUuM6nebEptQ53bjpAtfmhtbvbAy6WwhebUoEEg17fTxE", + "hash": "QmVivEuGVUTj48rXwhQVxrMGfhzEEanPuXidSmP9ASYoY9", "name": "go-conn-security-multistream", - "version": "0.1.10" + "version": "0.1.11" }, { "author": "Stebalien", - "hash": "QmcnVbkHxvzHKYvdbDxiUVGbn5yQjkSbb9qqkGHcgwvZff", + "hash": "QmTSeQWcL14juHXsqDfNjCADy89XS7sQpfJ3KFiiajXzJn", "name": "go-conn-security", - "version": "0.1.10" + "version": "0.1.11" }, { "author": "libp2p", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "QmNRRKZjRHJTuEvbAkkmJLpBQX9FZ7rAXdD8zuknZv6mmF", + "hash": "QmUhdmaWBQaxSBX8PXQme85Wgj1FFj29DePskiv44MDKPi", "name": "go-libp2p-transport-upgrader", - "version": "0.1.10" + "version": "0.1.11" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.13" + "version": "6.0.14" } From 394c637102ad82ecb08b1bf0331ac3a0813a7c38 Mon Sep 17 00:00:00 2001 From: Kevin Atkinson Date: Thu, 4 Oct 2018 19:26:11 -0400 Subject: [PATCH 0737/3965] gx publish 6.0.15 --- .gx/lastpubver | 2 +- package.json | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 8645b19e54..466c63e2fd 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.14: Qmd9zWxAeeDJoLdxqvaDXAGtoafX5cc9Tp25DNm9W7fVnB +6.0.15: QmNmj2AeM46ZQqHARnWidb5qqHoZJFeYWzmG65jviJDRQY diff --git a/package.json b/package.json index 21853804ca..e53e365f34 100644 --- a/package.json +++ b/package.json @@ -61,9 +61,9 @@ }, { "author": "whyrusleeping", - "hash": "QmUdVuLDQCQD4WPBpgDsbBAaqui3GW4d7exBHK2PUKRQpv", + "hash": "QmNkxudvyUjdKw5ComDSCbJTHCjrjNGYjEAUdhoEUUjFsX", "name": "go-libp2p-secio", - "version": "2.0.11" + "version": "2.0.12" }, { "author": "whyrusleeping", @@ -121,9 +121,9 @@ }, { "author": "whyrusleeping", - "hash": "QmcyQj1V6Ht8uSrRqj865UXGUo5Sc8GxNA4U8bLQxCSmfX", + "hash": "QmcALnLPFXgLLaZkXzGv99yt8xPfiWqFTumiXpWBJbU9cY", "name": "go-libp2p-swarm", - "version": "3.0.14" + "version": "3.0.15" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "QmeBpq5pAf2yaVdbpaKVcijEXu1QVjnDqPqYq4MACaEHLo", + "hash": "QmPq6vgdtDmuui5FU9qskJE7ijpmJKZ9rfmtdMW2jPvaos", "name": "go-libp2p-circuit", - "version": "2.2.3" + "version": "2.2.4" }, { "author": "lgierth", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.14" + "version": "6.0.15" } From 70586dda6665f94279ff52f21eda4e1f8fdc31d6 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 5 Oct 2018 10:05:47 -0400 Subject: [PATCH 0738/3965] Simplify hop increase logic --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index ec88bd82d3..96c30c9b2b 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -98,14 +98,8 @@ func (r *Relay) addLiveHop(from, to peer.ID) { trg = make(map[peer.ID]int) r.liveHops[from] = trg } - var count int - if count, ok = trg[to]; !ok { - count = 0 - } - count++ - + trg[to]++ r.lhCount++ - trg[to] = count } func (r *Relay) rmLiveHop(from, to peer.ID) { From f3c4a0c48b1e46210f1d101c4869cc6d63e7123c Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 5 Oct 2018 23:01:22 +0300 Subject: [PATCH 0739/3965] fix relay-specific address dialing in routed host --- p2p/host/routed/routed.go | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/p2p/host/routed/routed.go b/p2p/host/routed/routed.go index 99926f2b24..6522c3e88c 100644 --- a/p2p/host/routed/routed.go +++ b/p2p/host/routed/routed.go @@ -8,6 +8,7 @@ import ( host "github.com/libp2p/go-libp2p-host" logging "github.com/ipfs/go-log" + circuit "github.com/libp2p/go-libp2p-circuit" ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" lgbl "github.com/libp2p/go-libp2p-loggables" inet "github.com/libp2p/go-libp2p-net" @@ -67,6 +68,42 @@ func (rh *RoutedHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { } } + // Issue 448: if our address set includes specific relay addrs, we need + // to make sure the relay's addr itself is in the peerstore or else + // we wont be able to dial it. + for _, addr := range addrs { + _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) + if err != nil { + // not a relay address + continue + } + + relay, err := addr.ValueForProtocol(ma.P_P2P) + if err != nil { + // not a specific relay address + continue + } + + relayID, err := peer.IDFromString(relay) + if err != nil { + log.Debugf("failed to parse relay ID in address %s: %s", relay, err) + continue + } + + if len(rh.Peerstore().Addrs(relayID)) > 0 { + // we already have addrs for this relay + continue + } + + relayAddrs, err := rh.findPeerAddrs(ctx, relayID) + if err != nil { + log.Debugf("failed to find relay %s: %s", relay, err) + continue + } + + rh.Peerstore().AddAddrs(relayID, relayAddrs, pstore.AddressTTL) + } + // if we're here, we got some addrs. let's use our wrapped host to connect. pi.Addrs = addrs return rh.host.Connect(ctx, pi) From 42346ec3a162fe6c4a734e793e35902b6e171e4e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 5 Oct 2018 13:40:12 -0700 Subject: [PATCH 0740/3965] gx publish 6.0.16 --- .gx/lastpubver | 2 +- package.json | 62 +++++++++++++++++++++++++------------------------- 2 files changed, 32 insertions(+), 32 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 466c63e2fd..31bc9ce627 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.15: QmNmj2AeM46ZQqHARnWidb5qqHoZJFeYWzmG65jviJDRQY +6.0.16: QmWeqvGxDHjtjvRaHNLj1bDkeToAWJsq9HVCvqGEDZjiaG diff --git a/package.json b/package.json index e53e365f34..172938f67e 100644 --- a/package.json +++ b/package.json @@ -61,27 +61,27 @@ }, { "author": "whyrusleeping", - "hash": "QmNkxudvyUjdKw5ComDSCbJTHCjrjNGYjEAUdhoEUUjFsX", + "hash": "QmeUYMcQMwV9GvNhuh35G52EXCV77qPZUFkQc2TCPt27Fy", "name": "go-libp2p-secio", - "version": "2.0.12" + "version": "2.0.13" }, { "author": "whyrusleeping", - "hash": "QmSJ36wcYQyEViJUWUEhJU81tw1KdakTKqLLHbvYbA9zDv", + "hash": "QmXEyLwySuDMXejWBu8XwdkX2WuGKk8x9jFwz8js7j72UX", "name": "go-libp2p-peerstore", - "version": "2.0.2" + "version": "2.0.3" }, { "author": "whyrusleeping", - "hash": "QmRJFmMd5ed1fy5pGLfmWPGGSb1FFm5tJm62euuFPBQXjk", + "hash": "QmajSo5YEqW898AuVcY77oa5Jffx4N7wvQBGxyLnaD5zxo", "name": "go-libp2p-transport", - "version": "3.0.11" + "version": "3.0.12" }, { "author": "whyrusleeping", - "hash": "Qmf61d1En1KVfwc6A3fn4AYvSCKx6LiMivG5Ljyok1CXTW", + "hash": "QmdbYtEnF2imic8M5UmZtxgsvxPo6CdZ7tJNF1iyY7E59g", "name": "go-tcp-transport", - "version": "2.0.11" + "version": "2.0.12" }, { "author": "whyrusleeping", @@ -103,9 +103,9 @@ }, { "author": "whyrusleeping", - "hash": "QmQdLcvoy3JuSqhV6iwQ9T6Cv7hWLAdzob4jUZRPqFL67Z", + "hash": "QmWUPYHpNv4YahaBYXovuEJttgfqcNcN9Gg4arhQYcRoqa", "name": "go-libp2p-net", - "version": "3.0.11" + "version": "3.0.12" }, { "author": "whyrusleeping", @@ -115,15 +115,15 @@ }, { "author": "whyrusleeping", - "hash": "QmX9pw5dSUZ2FozbppcSDJiS7eEh1RFwJNwrbmyLoUMS9x", + "hash": "QmQKod7iLxQK6X3aFYvhDDnFdXo3QjxKeL2F7UrPtdKQR2", "name": "go-libp2p-host", - "version": "3.0.11" + "version": "3.0.12" }, { "author": "whyrusleeping", - "hash": "QmcALnLPFXgLLaZkXzGv99yt8xPfiWqFTumiXpWBJbU9cY", + "hash": "QmYssXVP4anzJGaq7VD1SJSxsr6AjbYD6gGjWoWYL5H3PR", "name": "go-libp2p-swarm", - "version": "3.0.15" + "version": "3.0.16" }, { "author": "whyrusleeping", @@ -133,15 +133,15 @@ }, { "author": "whyrusleeping", - "hash": "Qmek9Aad1NQUz3Fb1gXn6F1EuGrV4XoDQ33jEU3Wvt3XPv", + "hash": "QmcLwSgC3kFVe1Es6CSocPjF2At7WT1RRUfxLBxEWiQ8tD", "name": "go-libp2p-netutil", - "version": "0.4.9" + "version": "0.4.10" }, { "author": "whyrusleeping", - "hash": "QmSG4P432Eomm1s8rkgUuBk9Nzp8YjY35RUcJTGT2Huwki", + "hash": "QmfKYJYEAKy2eMTTyE8gHBNEVxfzPKnBH429EkdCP1csrP", "name": "go-libp2p-blankhost", - "version": "0.3.11" + "version": "0.3.12" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "QmPq6vgdtDmuui5FU9qskJE7ijpmJKZ9rfmtdMW2jPvaos", + "hash": "QmdTNkY7zXsoYcKUUbnsDnHVVGELKxvmkA6caUzxY8sF1j", "name": "go-libp2p-circuit", - "version": "2.2.4" + "version": "2.2.5" }, { "author": "lgierth", @@ -181,9 +181,9 @@ }, { "author": "why", - "hash": "QmYbLGD41kmPZfgGs5B8DMMwxtY61Wz4KPeGFZ2qoetsPY", + "hash": "QmXPqoktJEKkWPT6rSdhCM2G3E1yR7uUBdjnJrhJYeBgHU", "name": "go-libp2p-interface-connmgr", - "version": "0.0.17" + "version": "0.0.18" }, { "author": "whyrusleeping", @@ -193,21 +193,21 @@ }, { "author": "whyrusleeping", - "hash": "QmVKVakzxomMUPZMivoWTuWzRUv5VngfaWNmmqrAgaSc8S", + "hash": "Qmak2GyYnxBvEqhK8jX7sULZTpQC1U1wDUoFNb2SEUJCXF", "name": "go-ws-transport", - "version": "2.0.11" + "version": "2.0.12" }, { "author": "stebalien", - "hash": "QmVivEuGVUTj48rXwhQVxrMGfhzEEanPuXidSmP9ASYoY9", + "hash": "QmVTBq7VEo9M6cwsFWndC9q6RHRmL6BzMQsNL5WdZTMQEd", "name": "go-conn-security-multistream", - "version": "0.1.11" + "version": "0.1.12" }, { "author": "Stebalien", - "hash": "QmTSeQWcL14juHXsqDfNjCADy89XS7sQpfJ3KFiiajXzJn", + "hash": "QmPsFSk7bKhBeRkELpLagpWLJywG3bxgiiMcXXyBnZKiPK", "name": "go-conn-security", - "version": "0.1.11" + "version": "0.1.12" }, { "author": "libp2p", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "QmUhdmaWBQaxSBX8PXQme85Wgj1FFj29DePskiv44MDKPi", + "hash": "QmQkCDQnrQgS2StMmLhro8Bd9LxRQXrysq27cj7hPxEvsb", "name": "go-libp2p-transport-upgrader", - "version": "0.1.11" + "version": "0.1.12" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.15" + "version": "6.0.16" } From 2fac973565025e3cedccc1762f8f00cb45862dd0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 5 Oct 2018 14:40:46 -0700 Subject: [PATCH 0741/3965] gx publish 6.0.17 --- .gx/lastpubver | 2 +- package.json | 34 +++++++++++++++++----------------- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 31bc9ce627..a99285b412 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.16: QmWeqvGxDHjtjvRaHNLj1bDkeToAWJsq9HVCvqGEDZjiaG +6.0.17: QmU9Cf9q5TBCAC3kg74Fqr6K7DQTwa41C44YypYqB2GfR8 diff --git a/package.json b/package.json index 172938f67e..adde00a689 100644 --- a/package.json +++ b/package.json @@ -61,9 +61,9 @@ }, { "author": "whyrusleeping", - "hash": "QmeUYMcQMwV9GvNhuh35G52EXCV77qPZUFkQc2TCPt27Fy", + "hash": "QmZSHe9kqEcqUG6DSBrKYULk8EeZLDd1skpHNoSaK6rW4M", "name": "go-libp2p-secio", - "version": "2.0.13" + "version": "2.0.14" }, { "author": "whyrusleeping", @@ -79,9 +79,9 @@ }, { "author": "whyrusleeping", - "hash": "QmdbYtEnF2imic8M5UmZtxgsvxPo6CdZ7tJNF1iyY7E59g", + "hash": "Qmd995jEtfoM7dipVGFSvpGdXJpwUxF1Lqof296kgexh7q", "name": "go-tcp-transport", - "version": "2.0.12" + "version": "2.0.13" }, { "author": "whyrusleeping", @@ -121,9 +121,9 @@ }, { "author": "whyrusleeping", - "hash": "QmYssXVP4anzJGaq7VD1SJSxsr6AjbYD6gGjWoWYL5H3PR", + "hash": "Qmb55o5PuhvqwjdVLvc4VV3ouLziYc6TfwM9LC6GwBQokn", "name": "go-libp2p-swarm", - "version": "3.0.16" + "version": "3.0.17" }, { "author": "whyrusleeping", @@ -151,9 +151,9 @@ }, { "author": "whyrusleeping", - "hash": "QmcsgrV3nCAKjiHKZhKVXWc4oY3WBECJCqahXEMpHeMrev", + "hash": "QmejxafYByb7s4rEMoKjmThZm6HfQ1seASNejVNKde6LBZ", "name": "go-smux-yamux", - "version": "2.0.3" + "version": "2.0.4" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "QmdTNkY7zXsoYcKUUbnsDnHVVGELKxvmkA6caUzxY8sF1j", + "hash": "Qmf5M8FECT1H1NGZ9kM6Ri4G4LfQGJcyMpfMPEA1herPuH", "name": "go-libp2p-circuit", - "version": "2.2.5" + "version": "2.2.6" }, { "author": "lgierth", @@ -187,15 +187,15 @@ }, { "author": "whyrusleeping", - "hash": "QmYsbnP57nNUKWijs8o2q3ZFrSUcAGTdKoQPuSJC3Uxt1Y", + "hash": "QmQvMxiavoMJzuLPuqrzVX4nXZpMks29hqpv13d81aJNEF", "name": "go-smux-multiplex", - "version": "3.0.13" + "version": "3.0.15" }, { "author": "whyrusleeping", - "hash": "Qmak2GyYnxBvEqhK8jX7sULZTpQC1U1wDUoFNb2SEUJCXF", + "hash": "QmUXXKJ71CHKSnJYzgtLuAkebk2Hj43Pk7aW42dZ8vT1Qm", "name": "go-ws-transport", - "version": "2.0.12" + "version": "2.0.13" }, { "author": "stebalien", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "QmQkCDQnrQgS2StMmLhro8Bd9LxRQXrysq27cj7hPxEvsb", + "hash": "QmXffKE4ikCV3medQiCzbzuoE5QUuLd2rGr4EyiC6b9Mye", "name": "go-libp2p-transport-upgrader", - "version": "0.1.12" + "version": "0.1.13" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.16" + "version": "6.0.17" } From dd9b96f70272a52b9c7fdf19bb4b3b23efe19ffe Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 6 Oct 2018 09:40:24 +0300 Subject: [PATCH 0742/3965] check for old-style relay addrs --- p2p/host/routed/routed.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/p2p/host/routed/routed.go b/p2p/host/routed/routed.go index 6522c3e88c..5435bcd31c 100644 --- a/p2p/host/routed/routed.go +++ b/p2p/host/routed/routed.go @@ -90,6 +90,11 @@ func (rh *RoutedHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { continue } + if relayID == pi.ID { + // it's an old style p2p-circuit address that includes the peer + continue + } + if len(rh.Peerstore().Addrs(relayID)) > 0 { // we already have addrs for this relay continue From c52f45235d1b2a27d220c0e7944ea2b0c2489037 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 8 Oct 2018 15:04:43 +0300 Subject: [PATCH 0743/3965] simplify routed relay addr check the first component must be P_P2P, or else we have a specific address to dial --- p2p/host/routed/routed.go | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/p2p/host/routed/routed.go b/p2p/host/routed/routed.go index 5435bcd31c..97b58a9f63 100644 --- a/p2p/host/routed/routed.go +++ b/p2p/host/routed/routed.go @@ -68,8 +68,8 @@ func (rh *RoutedHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { } } - // Issue 448: if our address set includes specific relay addrs, we need - // to make sure the relay's addr itself is in the peerstore or else + // Issue 448: if our address set includes routed specific relay addrs, + // we need to make sure the relay's addr itself is in the peerstore or else // we wont be able to dial it. for _, addr := range addrs { _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) @@ -78,23 +78,19 @@ func (rh *RoutedHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { continue } - relay, err := addr.ValueForProtocol(ma.P_P2P) - if err != nil { - // not a specific relay address + if addr.Protocols()[0].Code != ma.P_P2P { + // not a routed relay specific address continue } + relay, _ := addr.ValueForProtocol(ma.P_P2P) + relayID, err := peer.IDFromString(relay) if err != nil { log.Debugf("failed to parse relay ID in address %s: %s", relay, err) continue } - if relayID == pi.ID { - // it's an old style p2p-circuit address that includes the peer - continue - } - if len(rh.Peerstore().Addrs(relayID)) > 0 { // we already have addrs for this relay continue From 6d641cac9fc91e7f9b8dd4adf064a63971ecb6e7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Oct 2018 12:04:02 +0300 Subject: [PATCH 0744/3965] fix README info for pubsub --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index af67156e19..f3075e948a 100644 --- a/README.md +++ b/README.md @@ -158,7 +158,7 @@ List of packages currently in existence for libp2p: | [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-consensus) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | | [`go-libp2p-raft`](//github.com/libp2p/go-libp2p-raft) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-raft.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-raft) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-raft/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-raft/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-raft/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-raft) | consensus implementation over raft | | **Pubsub** | -| [`go-floodsub`](//github.com/libp2p/go-floodsub) | [![Travis CI](https://travis-ci.org/libp2p/go-floodsub.svg?branch=master)](https://travis-ci.org/libp2p/go-floodsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-floodsub/master)](https://ci.ipfs.team/job/libp2p/job/go-floodsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-floodsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-floodsub) | basic implementation of pubsub driven by flooding | +| [`go-floodsub`](//github.com/libp2p/go-floodsub) | [![Travis CI](https://travis-ci.org/libp2p/go-floodsub.svg?branch=master)](https://travis-ci.org/libp2p/go-floodsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-floodsub/master)](https://ci.ipfs.team/job/libp2p/job/go-floodsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-floodsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-floodsub) | pubsub implementation | | **Metrics** | | [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | libp2p metrics interfaces/collectors | | **Data Types** | From 9750574e2eb77892de61e6cf494a35ebabf30060 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 10 Oct 2018 11:44:03 +0100 Subject: [PATCH 0745/3965] gx publish 6.0.18 --- .gx/lastpubver | 2 +- package.json | 70 +++++++++++++++++++++++++------------------------- 2 files changed, 36 insertions(+), 36 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index a99285b412..9882bbabba 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.17: QmU9Cf9q5TBCAC3kg74Fqr6K7DQTwa41C44YypYqB2GfR8 +6.0.18: QmcmNfbQznhk66ipFiaRHmZU8DVpvDKFfHrRo9q5wsHzZP diff --git a/package.json b/package.json index adde00a689..3725e588b6 100644 --- a/package.json +++ b/package.json @@ -61,27 +61,27 @@ }, { "author": "whyrusleeping", - "hash": "QmZSHe9kqEcqUG6DSBrKYULk8EeZLDd1skpHNoSaK6rW4M", + "hash": "QmZUBPE2W68FpZgUSmhVyjVD55P5Bm51wjnU8hTH14jHpk", "name": "go-libp2p-secio", - "version": "2.0.14" + "version": "2.0.15" }, { "author": "whyrusleeping", - "hash": "QmXEyLwySuDMXejWBu8XwdkX2WuGKk8x9jFwz8js7j72UX", + "hash": "QmWtCpWB39Rzc2xTB75MKorsxNpo3TyecTEN24CJ3KVohE", "name": "go-libp2p-peerstore", - "version": "2.0.3" + "version": "2.0.4" }, { "author": "whyrusleeping", - "hash": "QmajSo5YEqW898AuVcY77oa5Jffx4N7wvQBGxyLnaD5zxo", + "hash": "QmQx3dbLZqokcGsMbaj2KYoQQ6PBZZGvqiTn8rRNyk7H51", "name": "go-libp2p-transport", - "version": "3.0.12" + "version": "3.0.13" }, { "author": "whyrusleeping", - "hash": "Qmd995jEtfoM7dipVGFSvpGdXJpwUxF1Lqof296kgexh7q", + "hash": "QmRJ7i85QeZoR1LJbbAjmTu2jDnrcNH5NqsXirbv6c9SX7", "name": "go-tcp-transport", - "version": "2.0.13" + "version": "2.0.14" }, { "author": "whyrusleeping", @@ -103,9 +103,9 @@ }, { "author": "whyrusleeping", - "hash": "QmWUPYHpNv4YahaBYXovuEJttgfqcNcN9Gg4arhQYcRoqa", + "hash": "QmSTaEYUgDe1r581hxyd2u9582Hgp3KX4wGwYbRqz2u9Qh", "name": "go-libp2p-net", - "version": "3.0.12" + "version": "3.0.13" }, { "author": "whyrusleeping", @@ -115,15 +115,15 @@ }, { "author": "whyrusleeping", - "hash": "QmQKod7iLxQK6X3aFYvhDDnFdXo3QjxKeL2F7UrPtdKQR2", + "hash": "Qmf5yHzmWAyHSJRPAmZzfk3Yd7icydBLi7eec5741aov7v", "name": "go-libp2p-host", - "version": "3.0.12" + "version": "3.0.13" }, { "author": "whyrusleeping", - "hash": "Qmb55o5PuhvqwjdVLvc4VV3ouLziYc6TfwM9LC6GwBQokn", + "hash": "QmR2Yj7Eod2U77DqUHRwNSbV2mtqeRHmVdCfsLkXZwJVLy", "name": "go-libp2p-swarm", - "version": "3.0.17" + "version": "3.0.18" }, { "author": "whyrusleeping", @@ -133,15 +133,15 @@ }, { "author": "whyrusleeping", - "hash": "QmcLwSgC3kFVe1Es6CSocPjF2At7WT1RRUfxLBxEWiQ8tD", + "hash": "QmTqBqbGpNZnuPwW5AxUrooejubQuWnPpaatPozykZpXfi", "name": "go-libp2p-netutil", - "version": "0.4.10" + "version": "0.4.11" }, { "author": "whyrusleeping", - "hash": "QmfKYJYEAKy2eMTTyE8gHBNEVxfzPKnBH429EkdCP1csrP", + "hash": "QmcZbC2qrMF6DAsHKzk7xgWBDiJHHc15jJ6aDNrv84DG6S", "name": "go-libp2p-blankhost", - "version": "0.3.12" + "version": "0.3.13" }, { "author": "whyrusleeping", @@ -151,9 +151,9 @@ }, { "author": "whyrusleeping", - "hash": "QmejxafYByb7s4rEMoKjmThZm6HfQ1seASNejVNKde6LBZ", + "hash": "QmSNsTp6CjGoPj5yqjTMruAEoKcqicrzjsXfsaBLtKToVd", "name": "go-smux-yamux", - "version": "2.0.4" + "version": "2.0.5" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "Qmf5M8FECT1H1NGZ9kM6Ri4G4LfQGJcyMpfMPEA1herPuH", + "hash": "QmXpERwA2SDeruV41ohW4v9i2CSqFcHLEJsBwY2YxqsGLM", "name": "go-libp2p-circuit", - "version": "2.2.6" + "version": "2.2.7" }, { "author": "lgierth", @@ -181,33 +181,33 @@ }, { "author": "why", - "hash": "QmXPqoktJEKkWPT6rSdhCM2G3E1yR7uUBdjnJrhJYeBgHU", + "hash": "QmYfDsxmFGuMSdvFFgw5Ss9pD4YvTFWSFFXYT7LjtwKnY7", "name": "go-libp2p-interface-connmgr", - "version": "0.0.18" + "version": "0.0.19" }, { "author": "whyrusleeping", - "hash": "QmQvMxiavoMJzuLPuqrzVX4nXZpMks29hqpv13d81aJNEF", + "hash": "QmaveCPGVaKJU57tBErGCDjzLaqEMZkFygoiv4BhYwWUGc", "name": "go-smux-multiplex", - "version": "3.0.15" + "version": "3.0.16" }, { "author": "whyrusleeping", - "hash": "QmUXXKJ71CHKSnJYzgtLuAkebk2Hj43Pk7aW42dZ8vT1Qm", + "hash": "QmefRFPqDrW6xzNqfct7gWXSDQQV1PQEK2nhTxhN9YATJF", "name": "go-ws-transport", - "version": "2.0.13" + "version": "2.0.14" }, { "author": "stebalien", - "hash": "QmVTBq7VEo9M6cwsFWndC9q6RHRmL6BzMQsNL5WdZTMQEd", + "hash": "QmQYTdbqdgLDikde7dHKh4svNG925EDYA9ZKZHwqidX277", "name": "go-conn-security-multistream", - "version": "0.1.12" + "version": "0.1.13" }, { "author": "Stebalien", - "hash": "QmPsFSk7bKhBeRkELpLagpWLJywG3bxgiiMcXXyBnZKiPK", + "hash": "QmSjwssnWNJvzwD77yWruH2ebqwMNR6PCn49hbDURYH9d8", "name": "go-conn-security", - "version": "0.1.12" + "version": "0.1.13" }, { "author": "libp2p", @@ -223,9 +223,9 @@ }, { "author": "steb", - "hash": "QmXffKE4ikCV3medQiCzbzuoE5QUuLd2rGr4EyiC6b9Mye", + "hash": "QmPM1NCCnMMY3GaPVkG2sTVh8Xys1gCPsk1X8Z1HuUJePs", "name": "go-libp2p-transport-upgrader", - "version": "0.1.13" + "version": "0.1.14" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.17" + "version": "6.0.18" } From 257cea01fb7fa5a225c07939e5340388b55d0c4d Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 12 Oct 2018 14:32:46 +0300 Subject: [PATCH 0746/3965] interface and utility functions --- p2p/discovery/util/util.go | 56 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 p2p/discovery/util/util.go diff --git a/p2p/discovery/util/util.go b/p2p/discovery/util/util.go new file mode 100644 index 0000000000..be43a4b15c --- /dev/null +++ b/p2p/discovery/util/util.go @@ -0,0 +1,56 @@ +package discovery + +import ( + "context" + "time" + + logging "github.com/ipfs/go-log" + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +var log = logging.Logger("discovery") + +// FindPeers is a utility function that synchonously collects peers from a Discoverer +func FindPeers(ctx context.Context, d Discoverer, ns string, limit int) ([]pstore.PeerInfo, error) { + res := make([]pstore.PeerInfo, 0, limit) + + ch, err := d.FindPeers(ctx, ns, Limit(limit)) + if err != nil { + return nil, err + } + + for pi := range ch { + res = append(res, pi) + } + + return res, nil +} + +// Advertise is a utility function that persistently advertises a service through an Advertiser +func Advertise(ctx context.Context, a Advertiser, ns string) { + go func() { + for { + ttl, err := a.Advertise(ctx, ns) + if err != nil { + log.Debugf("Error advertising %s: %s", ns, err.Error()) + if ctx.Err() != nil { + return + } + + select { + case <-time.After(2 * time.Minute): + continue + case <-ctx.Done(): + return + } + } + + wait := 7 * ttl / 8 + select { + case <-time.After(wait): + case <-ctx.Done(): + return + } + } + }() +} From d0766bb0acc78eff0090f6c5d76879e83b1617ca Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 12 Oct 2018 14:56:09 +0300 Subject: [PATCH 0747/3965] discovery implementation using ContentRouting --- p2p/discovery/routing/routing.go | 64 ++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 p2p/discovery/routing/routing.go diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go new file mode 100644 index 0000000000..f1cec03a0f --- /dev/null +++ b/p2p/discovery/routing/routing.go @@ -0,0 +1,64 @@ +package discovery + +import ( + "context" + "time" + + cid "github.com/ipfs/go-cid" + pstore "github.com/libp2p/go-libp2p-peerstore" + routing "github.com/libp2p/go-libp2p-routing" + mh "github.com/multiformats/go-multihash" +) + +// RoutingDiscovery is an implementation of discovery using ContentRouting +type RoutingDiscovery struct { + router routing.ContentRouting +} + +func NewRoutingDiscovery(router routing.ContentRouting) Discovery { + return &RoutingDiscovery{router} +} + +func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Option) (time.Duration, error) { + cid, err := nsToCid(ns) + if err != nil { + return 0, err + } + + err = d.router.Provide(ctx, cid, true) + if err != nil { + return 0, err + } + + // this is the dht provide validity + return 24 * time.Hour, nil +} + +func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Option) (<-chan pstore.PeerInfo, error) { + options := &Options{} + err := options.Apply(opts...) + if err != nil { + return nil, err + } + + limit := options.Limit + if limit == 0 { + limit = 100 // that's just arbitrary, but FindProvidersAsync needs a count + } + + cid, err := nsToCid(ns) + if err != nil { + return nil, err + } + + return d.router.FindProvidersAsync(ctx, cid, limit), nil +} + +func nsToCid(ns string) (cid.Cid, error) { + h, err := mh.Encode([]byte(ns), mh.SHA2_256) + if err != nil { + return cid.Undef, err + } + + return cid.NewCidV1(cid.Raw, h), nil +} From 061038402a29fa17158afd7d9b3355f0be897166 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 12 Oct 2018 15:59:24 +0100 Subject: [PATCH 0748/3965] gx publish 6.0.19 --- .gx/lastpubver | 2 +- package.json | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 9882bbabba..b9e31f7c95 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.18: QmcmNfbQznhk66ipFiaRHmZU8DVpvDKFfHrRo9q5wsHzZP +6.0.19: QmPL3AKtiaQyYpchZceXBZhZ3MSnoGqJvLZrc7fzDTTQdJ diff --git a/package.json b/package.json index 3725e588b6..48628ba0b6 100644 --- a/package.json +++ b/package.json @@ -121,9 +121,9 @@ }, { "author": "whyrusleeping", - "hash": "QmR2Yj7Eod2U77DqUHRwNSbV2mtqeRHmVdCfsLkXZwJVLy", + "hash": "QmYyFNDLTSy6rW99K8vHPvFPLa5bB4zoy8gGZENoiaBN6R", "name": "go-libp2p-swarm", - "version": "3.0.18" + "version": "3.0.19" }, { "author": "whyrusleeping", @@ -151,9 +151,9 @@ }, { "author": "whyrusleeping", - "hash": "QmSNsTp6CjGoPj5yqjTMruAEoKcqicrzjsXfsaBLtKToVd", + "hash": "QmVwYCtShoL74Xi8TgEg9jXvHVVtWwZ3Hg1QZqQvQA9xji", "name": "go-smux-yamux", - "version": "2.0.5" + "version": "2.0.6" }, { "author": "whyrusleeping", @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "QmXpERwA2SDeruV41ohW4v9i2CSqFcHLEJsBwY2YxqsGLM", + "hash": "Qma2mxvX5zvPEmjXC4CktjExnRjVp4DFcog4GNubpqSXsD", "name": "go-libp2p-circuit", - "version": "2.2.7" + "version": "2.2.8" }, { "author": "lgierth", @@ -238,6 +238,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.18" + "version": "6.0.19" } From d15e65b95bdf4066aecd3bc40059ace606ac2251 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 12 Oct 2018 18:06:04 +0300 Subject: [PATCH 0749/3965] increase identify delay to 500ms --- p2p/host/autonat/notify.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 30ad9cd0c1..63ed44390d 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -19,7 +19,7 @@ func (as *AmbientAutoNAT) Connected(net inet.Network, c inet.Conn) { go func() { // add some delay for identify - time.Sleep(250 * time.Millisecond) + time.Sleep(500 * time.Millisecond) protos, err := as.host.Peerstore().SupportsProtocols(p, AutoNATProto) if err != nil { From 810238c3e7784309433359501ba2c5d07c76df5d Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 12 Oct 2018 18:07:41 +0300 Subject: [PATCH 0750/3965] jenkins file --- p2p/host/autonat/ci/Jenkinsfile | 1 + 1 file changed, 1 insertion(+) create mode 100644 p2p/host/autonat/ci/Jenkinsfile diff --git a/p2p/host/autonat/ci/Jenkinsfile b/p2p/host/autonat/ci/Jenkinsfile new file mode 100644 index 0000000000..b2067e6232 --- /dev/null +++ b/p2p/host/autonat/ci/Jenkinsfile @@ -0,0 +1 @@ +golang() From c2190b48fef845ae587ba92690b765e02ac526a1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 14 Oct 2018 15:11:35 +0100 Subject: [PATCH 0751/3965] store expiring addrs by pointer Hopefully, this reduces the space wasted by using maps (see the comment in the committed code). --- p2p/host/peerstore/pstoremem/addr_book.go | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index f0f8a2c87c..723b4337cb 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -31,7 +31,10 @@ var _ pstore.AddrBook = (*memoryAddrBook)(nil) // memoryAddrBook manages addresses. type memoryAddrBook struct { addrmu sync.Mutex - addrs map[peer.ID]map[string]expiringAddr + // Use pointers to save memory. Maps always leave some fraction of their + // space unused. storing the *values* directly in the map will + // drastically increase the space waste. In our case, by 6x. + addrs map[peer.ID]map[string]*expiringAddr nextGC time.Time @@ -40,7 +43,7 @@ type memoryAddrBook struct { func NewAddrBook() pstore.AddrBook { return &memoryAddrBook{ - addrs: make(map[peer.ID]map[string]expiringAddr), + addrs: make(map[peer.ID]map[string]*expiringAddr), subManager: NewAddrSubManager(), } } @@ -94,7 +97,7 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du amap := mab.addrs[p] if amap == nil { - amap = make(map[string]expiringAddr, len(addrs)) + amap = make(map[string]*expiringAddr, len(addrs)) mab.addrs[p] = amap } exp := time.Now().Add(ttl) @@ -106,7 +109,7 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du addrstr := string(addr.Bytes()) a, found := amap[addrstr] if !found || exp.After(a.Expires) { - amap[addrstr] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} + amap[addrstr] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} mab.subManager.BroadcastAddr(p, addr) } @@ -127,7 +130,7 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du amap := mab.addrs[p] if amap == nil { - amap = make(map[string]expiringAddr, len(addrs)) + amap = make(map[string]*expiringAddr, len(addrs)) mab.addrs[p] = amap } @@ -141,7 +144,7 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du addrstr := string(addr.Bytes()) if ttl > 0 { - amap[addrstr] = expiringAddr{Addr: addr, Expires: exp, TTL: ttl} + amap[addrstr] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} mab.subManager.BroadcastAddr(p, addr) } else { From 590aed0cef2ea52544fc3ba6be13ec098677d42b Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Oct 2018 12:12:10 +0300 Subject: [PATCH 0752/3965] Add configurable Identify delay, with default value of 5 secs --- p2p/host/autonat/autonat_test.go | 31 +++++++++---------------------- p2p/host/autonat/notify.go | 4 +++- 2 files changed, 12 insertions(+), 23 deletions(-) diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index ac438f8482..6d31d46a87 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -10,6 +10,13 @@ import ( host "github.com/libp2p/go-libp2p-host" ) +func init() { + AutoNATBootDelay = 1 * time.Second + AutoNATRefreshInterval = 1 * time.Second + AutoNATRetryInterval = 1 * time.Second + AutoNATIdentifyDelay = 100 * time.Millisecond +} + func makeAutoNAT(ctx context.Context, t *testing.T) (host.Host, AutoNAT) { h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) if err != nil { @@ -26,13 +33,6 @@ func TestAutoNATPrivate(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save1 := AutoNATBootDelay - AutoNATBootDelay = 1 * time.Second - save2 := AutoNATRefreshInterval - AutoNATRefreshInterval = 1 * time.Second - save3 := AutoNATRetryInterval - AutoNATRetryInterval = 1 * time.Second - hs, _ := makeAutoNATService(ctx, t) hc, an := makeAutoNAT(ctx, t) @@ -48,23 +48,13 @@ func TestAutoNATPrivate(t *testing.T) { if status != NATStatusPrivate { t.Fatalf("unexpected NAT status: %d", status) } - - AutoNATBootDelay = save1 - AutoNATRefreshInterval = save2 - AutoNATRetryInterval = save3 } func TestAutoNATPublic(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save1 := AutoNATBootDelay - AutoNATBootDelay = 1 * time.Second - save2 := AutoNATRefreshInterval - AutoNATRefreshInterval = 1 * time.Second - save3 := AutoNATRetryInterval - AutoNATRetryInterval = 1 * time.Second - save4 := private4 + save := private4 private4 = []*net.IPNet{} hs, _ := makeAutoNATService(ctx, t) @@ -83,8 +73,5 @@ func TestAutoNATPublic(t *testing.T) { t.Fatalf("unexpected NAT status: %d", status) } - AutoNATBootDelay = save1 - AutoNATRefreshInterval = save2 - AutoNATRetryInterval = save3 - private4 = save4 + private4 = save } diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 63ed44390d..ebe709baf6 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -9,6 +9,8 @@ import ( var _ inet.Notifiee = (*AmbientAutoNAT)(nil) +var AutoNATIdentifyDelay = 5 * time.Second + func (as *AmbientAutoNAT) Listen(net inet.Network, a ma.Multiaddr) {} func (as *AmbientAutoNAT) ListenClose(net inet.Network, a ma.Multiaddr) {} func (as *AmbientAutoNAT) OpenedStream(net inet.Network, s inet.Stream) {} @@ -19,7 +21,7 @@ func (as *AmbientAutoNAT) Connected(net inet.Network, c inet.Conn) { go func() { // add some delay for identify - time.Sleep(500 * time.Millisecond) + time.Sleep(AutoNATIdentifyDelay) protos, err := as.host.Peerstore().SupportsProtocols(p, AutoNATProto) if err != nil { From 0710dd37a5a00842ed728af56b1517fe93942515 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Oct 2018 12:48:28 +0300 Subject: [PATCH 0753/3965] add docstring for confidence --- p2p/host/autonat/autonat.go | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 4f766ce7c9..1eace01770 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -48,10 +48,15 @@ type AmbientAutoNAT struct { ctx context.Context host host.Host - mx sync.Mutex - peers map[peer.ID]struct{} - status NATStatus - addr ma.Multiaddr + mx sync.Mutex + peers map[peer.ID]struct{} + status NATStatus + addr ma.Multiaddr + // Reflects the confidence on of the NATStatus being private, as a single + // dialback may fail for reasons unrelated to NAT. + // If it is <3, then multiple autoNAT peers may be contacted for dialback + // If only a single autoNAT peer is known, then the confidence increases + // for each failure until it reaches 3. confidence int } From 8ddcc58f27c9037b7395a46b1a409961bacd183f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 16 Oct 2018 11:25:41 +0100 Subject: [PATCH 0754/3965] update to match new ExtractPublicKey API This is a "more correct" API and we might as well fix this before it becomes used all over the place. The new API returns ErrNoPublicKey when there is no inlined key. This *also* fixes a bug where the datastore-backed keystore would panic if `ExtractPublicKey` returned `nil, nil`. --- p2p/host/peerstore/pstoreds/keybook.go | 6 +++++- p2p/host/peerstore/pstoremem/keybook.go | 2 +- p2p/host/peerstore/test/keybook_suite.go | 8 ++++++++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/keybook.go b/p2p/host/peerstore/pstoreds/keybook.go index 307f7dfd4a..dc54b19239 100644 --- a/p2p/host/peerstore/pstoreds/keybook.go +++ b/p2p/host/peerstore/pstoreds/keybook.go @@ -43,7 +43,11 @@ func (kb *dsKeyBook) PubKey(p peer.ID) ic.PubKey { } } else if err == ds.ErrNotFound { pk, err = p.ExtractPublicKey() - if err != nil { + switch err { + case nil: + case peer.ErrNoPublicKey: + return nil + default: log.Errorf("error when extracting pubkey from peer ID for peer %s: %s\n", p.Pretty(), err) return nil } diff --git a/p2p/host/peerstore/pstoremem/keybook.go b/p2p/host/peerstore/pstoremem/keybook.go index 37437ebdbb..87baea24f7 100644 --- a/p2p/host/peerstore/pstoremem/keybook.go +++ b/p2p/host/peerstore/pstoremem/keybook.go @@ -49,7 +49,7 @@ func (mkb *memoryKeyBook) PubKey(p peer.ID) ic.PubKey { return pk } pk, err := p.ExtractPublicKey() - if err == nil && pk != nil { + if err == nil { mkb.Lock() mkb.pks[p] = pk mkb.Unlock() diff --git a/p2p/host/peerstore/test/keybook_suite.go b/p2p/host/peerstore/test/keybook_suite.go index cb305e1e2c..9379d9cd99 100644 --- a/p2p/host/peerstore/test/keybook_suite.go +++ b/p2p/host/peerstore/test/keybook_suite.go @@ -50,6 +50,10 @@ func testKeybookPrivKey(kb pstore.KeyBook) func(t *testing.T) { t.Error(err) } + if res := kb.PrivKey(id); res != nil { + t.Error("retrieving private key should have failed") + } + err = kb.AddPrivKey(id, priv) if err != nil { t.Error(err) @@ -81,6 +85,10 @@ func testKeyBookPubKey(kb pstore.KeyBook) func(t *testing.T) { t.Error(err) } + if res := kb.PubKey(id); res != nil { + t.Error("retrieving public key should have failed") + } + err = kb.AddPubKey(id, pub) if err != nil { t.Error(err) From 2c67a96430bb0e459014ec79765d709367fa9643 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Oct 2018 14:39:19 +0300 Subject: [PATCH 0755/3965] extract service from package --- p2p/host/autonat/addr.go | 12 +- p2p/host/autonat/autonat_test.go | 85 +++++++++--- p2p/host/autonat/proto.go | 15 --- p2p/host/autonat/svc.go | 214 ------------------------------- p2p/host/autonat/svc_test.go | 131 ------------------- 5 files changed, 75 insertions(+), 382 deletions(-) delete mode 100644 p2p/host/autonat/svc.go delete mode 100644 p2p/host/autonat/svc_test.go diff --git a/p2p/host/autonat/addr.go b/p2p/host/autonat/addr.go index 4e078e3765..eee254f8a0 100644 --- a/p2p/host/autonat/addr.go +++ b/p2p/host/autonat/addr.go @@ -6,7 +6,7 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -var private4, private6 []*net.IPNet +var Private4, Private6 []*net.IPNet var privateCIDR4 = []string{ // localhost "127.0.0.0/8", @@ -28,8 +28,8 @@ var privateCIDR6 = []string{ } func init() { - private4 = parsePrivateCIDR(privateCIDR4) - private6 = parsePrivateCIDR(privateCIDR6) + Private4 = parsePrivateCIDR(privateCIDR4) + Private6 = parsePrivateCIDR(privateCIDR6) } func parsePrivateCIDR(cidrs []string) []*net.IPNet { @@ -44,15 +44,15 @@ func parsePrivateCIDR(cidrs []string) []*net.IPNet { return ipnets } -func isPublicAddr(a ma.Multiaddr) bool { +func IsPublicAddr(a ma.Multiaddr) bool { ip, err := a.ValueForProtocol(ma.P_IP4) if err == nil { - return !inAddrRange(ip, private4) + return !inAddrRange(ip, Private4) } ip, err = a.ValueForProtocol(ma.P_IP6) if err == nil { - return !inAddrRange(ip, private6) + return !inAddrRange(ip, Private6) } return false diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 6d31d46a87..6170d01d19 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -2,12 +2,18 @@ package autonat import ( "context" - "net" "testing" "time" - libp2p "github.com/libp2p/go-libp2p" + pb "github.com/libp2p/go-libp2p-autonat/pb" + + ggio "github.com/gogo/protobuf/io" + bhost "github.com/libp2p/go-libp2p-blankhost" host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" + pstore "github.com/libp2p/go-libp2p-peerstore" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" + ma "github.com/multiformats/go-multiaddr" ) func init() { @@ -17,24 +23,76 @@ func init() { AutoNATIdentifyDelay = 100 * time.Millisecond } -func makeAutoNAT(ctx context.Context, t *testing.T) (host.Host, AutoNAT) { - h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) - if err != nil { - t.Fatal(err) +// these are mock service implementations for testing +func makeAutoNATServicePrivate(ctx context.Context, t *testing.T) host.Host { + h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h.SetStreamHandler(AutoNATProto, sayAutoNATPrivate) + return h +} + +func makeAutoNATServicePublic(ctx context.Context, t *testing.T) host.Host { + h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h.SetStreamHandler(AutoNATProto, sayAutoNATPublic) + return h +} + +func sayAutoNATPrivate(s inet.Stream) { + defer s.Close() + w := ggio.NewDelimitedWriter(s) + res := pb.Message{ + Type: pb.Message_DIAL_RESPONSE.Enum(), + DialResponse: newDialResponseError(pb.Message_E_DIAL_ERROR, "no dialable addresses"), } + w.WriteMsg(&res) +} + +func sayAutoNATPublic(s inet.Stream) { + defer s.Close() + w := ggio.NewDelimitedWriter(s) + res := pb.Message{ + Type: pb.Message_DIAL_RESPONSE.Enum(), + DialResponse: newDialResponseOK(s.Conn().RemoteMultiaddr()), + } + w.WriteMsg(&res) +} +func newDialResponseOK(addr ma.Multiaddr) *pb.Message_DialResponse { + dr := new(pb.Message_DialResponse) + dr.Status = pb.Message_OK.Enum() + dr.Addr = addr.Bytes() + return dr +} + +func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Message_DialResponse { + dr := new(pb.Message_DialResponse) + dr.Status = status.Enum() + dr.StatusText = &text + return dr +} + +func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, AutoNAT) { + h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) a := NewAutoNAT(ctx, h) + a.(*AmbientAutoNAT).peers[ash.ID()] = struct{}{} return h, a } -// Note: these tests assume the host has only private inet addresses! +func connect(t *testing.T, a, b host.Host) { + pinfo := pstore.PeerInfo{ID: a.ID(), Addrs: a.Addrs()} + err := b.Connect(context.Background(), pinfo) + if err != nil { + t.Fatal(err) + } +} + +// tests func TestAutoNATPrivate(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - hs, _ := makeAutoNATService(ctx, t) - hc, an := makeAutoNAT(ctx, t) + hs := makeAutoNATServicePrivate(ctx, t) + hc, an := makeAutoNAT(ctx, t, hs) status := an.Status() if status != NATStatusUnknown { @@ -54,11 +112,8 @@ func TestAutoNATPublic(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save := private4 - private4 = []*net.IPNet{} - - hs, _ := makeAutoNATService(ctx, t) - hc, an := makeAutoNAT(ctx, t) + hs := makeAutoNATServicePublic(ctx, t) + hc, an := makeAutoNAT(ctx, t, hs) status := an.Status() if status != NATStatusUnknown { @@ -72,6 +127,4 @@ func TestAutoNATPublic(t *testing.T) { if status != NATStatusPublic { t.Fatalf("unexpected NAT status: %d", status) } - - private4 = save } diff --git a/p2p/host/autonat/proto.go b/p2p/host/autonat/proto.go index c9768cbdbd..2dbb8ebaa7 100644 --- a/p2p/host/autonat/proto.go +++ b/p2p/host/autonat/proto.go @@ -5,7 +5,6 @@ import ( logging "github.com/ipfs/go-log" pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" ) const AutoNATProto = "/libp2p/autonat/1.0.0" @@ -25,17 +24,3 @@ func newDialMessage(pi pstore.PeerInfo) *pb.Message { return msg } - -func newDialResponseOK(addr ma.Multiaddr) *pb.Message_DialResponse { - dr := new(pb.Message_DialResponse) - dr.Status = pb.Message_OK.Enum() - dr.Addr = addr.Bytes() - return dr -} - -func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Message_DialResponse { - dr := new(pb.Message_DialResponse) - dr.Status = status.Enum() - dr.StatusText = &text - return dr -} diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go deleted file mode 100644 index 4d395d22ec..0000000000 --- a/p2p/host/autonat/svc.go +++ /dev/null @@ -1,214 +0,0 @@ -package autonat - -import ( - "context" - "sync" - "time" - - pb "github.com/libp2p/go-libp2p-autonat/pb" - - ggio "github.com/gogo/protobuf/io" - libp2p "github.com/libp2p/go-libp2p" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" -) - -const P_CIRCUIT = 290 - -var ( - AutoNATServiceDialTimeout = 42 * time.Second - AutoNATServiceResetInterval = 1 * time.Minute - - AutoNATServiceThrottle = 3 -) - -// AutoNATService provides NAT autodetection services to other peers -type AutoNATService struct { - ctx context.Context - dialer host.Host - - // rate limiter - mx sync.Mutex - reqs map[peer.ID]int -} - -// NewAutoNATService creates a new AutoNATService instance attached to a host -func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) (*AutoNATService, error) { - opts = append(opts, libp2p.NoListenAddrs) - dialer, err := libp2p.New(ctx, opts...) - if err != nil { - return nil, err - } - - as := &AutoNATService{ - ctx: ctx, - dialer: dialer, - reqs: make(map[peer.ID]int), - } - h.SetStreamHandler(AutoNATProto, as.handleStream) - - go as.resetRateLimiter() - - return as, nil -} - -func (as *AutoNATService) handleStream(s inet.Stream) { - defer s.Close() - - pid := s.Conn().RemotePeer() - log.Debugf("New stream from %s", pid.Pretty()) - - r := ggio.NewDelimitedReader(s, inet.MessageSizeMax) - w := ggio.NewDelimitedWriter(s) - - var req pb.Message - var res pb.Message - - err := r.ReadMsg(&req) - if err != nil { - log.Debugf("Error reading message from %s: %s", pid.Pretty(), err.Error()) - s.Reset() - return - } - - t := req.GetType() - if t != pb.Message_DIAL { - log.Debugf("Unexpected message from %s: %s (%d)", pid.Pretty(), t.String(), t) - s.Reset() - return - } - - dr := as.handleDial(pid, s.Conn().RemoteMultiaddr(), req.GetDial().GetPeer()) - res.Type = pb.Message_DIAL_RESPONSE.Enum() - res.DialResponse = dr - - err = w.WriteMsg(&res) - if err != nil { - log.Debugf("Error writing response to %s: %s", pid.Pretty(), err.Error()) - s.Reset() - return - } -} - -func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { - if mpi == nil { - return newDialResponseError(pb.Message_E_BAD_REQUEST, "missing peer info") - } - - mpid := mpi.GetId() - if mpid != nil { - mp, err := peer.IDFromBytes(mpid) - if err != nil { - return newDialResponseError(pb.Message_E_BAD_REQUEST, "bad peer id") - } - - if mp != p { - return newDialResponseError(pb.Message_E_BAD_REQUEST, "peer id mismatch") - } - } - - addrs := make([]ma.Multiaddr, 0) - seen := make(map[string]struct{}) - - // add observed addr to the list of addresses to dial - if !as.skipDial(obsaddr) { - addrs = append(addrs, obsaddr) - seen[obsaddr.String()] = struct{}{} - } - - for _, maddr := range mpi.GetAddrs() { - addr, err := ma.NewMultiaddrBytes(maddr) - if err != nil { - log.Debugf("Error parsing multiaddr: %s", err.Error()) - continue - } - - if as.skipDial(addr) { - continue - } - - str := addr.String() - _, ok := seen[str] - if ok { - continue - } - - addrs = append(addrs, addr) - seen[str] = struct{}{} - } - - if len(addrs) == 0 { - return newDialResponseError(pb.Message_E_DIAL_ERROR, "no dialable addresses") - } - - return as.doDial(pstore.PeerInfo{ID: p, Addrs: addrs}) -} - -func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { - // skip relay addresses - _, err := addr.ValueForProtocol(P_CIRCUIT) - if err == nil { - return true - } - - // skip private network (unroutable) addresses - if !isPublicAddr(addr) { - return true - } - - return false -} - -func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { - // rate limit check - as.mx.Lock() - count := as.reqs[pi.ID] - if count >= AutoNATServiceThrottle { - as.mx.Unlock() - return newDialResponseError(pb.Message_E_DIAL_REFUSED, "too many dials") - } - as.reqs[pi.ID] = count + 1 - as.mx.Unlock() - - ctx, cancel := context.WithTimeout(as.ctx, AutoNATServiceDialTimeout) - defer cancel() - - err := as.dialer.Connect(ctx, pi) - if err != nil { - log.Debugf("error dialing %s: %s", pi.ID.Pretty(), err.Error()) - // wait for the context to timeout to avoid leaking timing information - // this renders the service ineffective as a port scanner - <-ctx.Done() - return newDialResponseError(pb.Message_E_DIAL_ERROR, "dial failed") - } - - conns := as.dialer.Network().ConnsToPeer(pi.ID) - if len(conns) == 0 { - log.Errorf("supposedly connected to %s, but no connection to peer", pi.ID.Pretty()) - return newDialResponseError(pb.Message_E_INTERNAL_ERROR, "internal service error") - } - - ra := conns[0].RemoteMultiaddr() - as.dialer.Network().ClosePeer(pi.ID) - return newDialResponseOK(ra) -} - -func (as *AutoNATService) resetRateLimiter() { - ticker := time.NewTicker(AutoNATServiceResetInterval) - defer ticker.Stop() - - for { - select { - case <-ticker.C: - as.mx.Lock() - as.reqs = make(map[peer.ID]int) - as.mx.Unlock() - - case <-as.ctx.Done(): - return - } - } -} diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go deleted file mode 100644 index cfa4068bfa..0000000000 --- a/p2p/host/autonat/svc_test.go +++ /dev/null @@ -1,131 +0,0 @@ -package autonat - -import ( - "context" - "net" - "testing" - "time" - - libp2p "github.com/libp2p/go-libp2p" - host "github.com/libp2p/go-libp2p-host" - pstore "github.com/libp2p/go-libp2p-peerstore" -) - -func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATService) { - h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) - if err != nil { - t.Fatal(err) - } - - as, err := NewAutoNATService(ctx, h) - if err != nil { - t.Fatal(err) - } - - return h, as -} - -func makeAutoNATClient(ctx context.Context, t *testing.T) (host.Host, AutoNATClient) { - h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) - if err != nil { - t.Fatal(err) - } - - cli := NewAutoNATClient(h) - return h, cli -} - -func connect(t *testing.T, a, b host.Host) { - pinfo := pstore.PeerInfo{ID: a.ID(), Addrs: a.Addrs()} - err := b.Connect(context.Background(), pinfo) - if err != nil { - t.Fatal(err) - } -} - -// Note: these tests assume that the host has only private inet addresses! -func TestAutoNATServiceDialError(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - save := AutoNATServiceDialTimeout - AutoNATServiceDialTimeout = 1 * time.Second - - hs, _ := makeAutoNATService(ctx, t) - hc, ac := makeAutoNATClient(ctx, t) - connect(t, hs, hc) - - _, err := ac.DialBack(ctx, hs.ID()) - if err == nil { - t.Fatal("Dial back succeeded unexpectedly!") - } - - if !IsDialError(err) { - t.Fatal(err) - } - - AutoNATServiceDialTimeout = save -} - -func TestAutoNATServiceDialSuccess(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - save := private4 - private4 = []*net.IPNet{} - - hs, _ := makeAutoNATService(ctx, t) - hc, ac := makeAutoNATClient(ctx, t) - connect(t, hs, hc) - - _, err := ac.DialBack(ctx, hs.ID()) - if err != nil { - t.Fatalf("Dial back failed: %s", err.Error()) - } - - private4 = save -} - -func TestAutoNATServiceDialRateLimiter(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - save1 := AutoNATServiceDialTimeout - AutoNATServiceDialTimeout = 1 * time.Second - save2 := AutoNATServiceResetInterval - AutoNATServiceResetInterval = 1 * time.Second - save3 := AutoNATServiceThrottle - AutoNATServiceThrottle = 1 - save4 := private4 - private4 = []*net.IPNet{} - - hs, _ := makeAutoNATService(ctx, t) - hc, ac := makeAutoNATClient(ctx, t) - connect(t, hs, hc) - - _, err := ac.DialBack(ctx, hs.ID()) - if err != nil { - t.Fatal(err) - } - - _, err = ac.DialBack(ctx, hs.ID()) - if err == nil { - t.Fatal("Dial back succeeded unexpectedly!") - } - - if !IsDialRefused(err) { - t.Fatal(err) - } - - time.Sleep(2 * time.Second) - - _, err = ac.DialBack(ctx, hs.ID()) - if err != nil { - t.Fatal(err) - } - - AutoNATServiceDialTimeout = save1 - AutoNATServiceResetInterval = save2 - AutoNATServiceThrottle = save3 - private4 = save4 -} From 0bc8c8010da218451a14e6ce72f4a572a5b1806b Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Oct 2018 14:53:35 +0300 Subject: [PATCH 0756/3965] add docstring for IsPublicAddr --- p2p/host/autonat/addr.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/host/autonat/addr.go b/p2p/host/autonat/addr.go index eee254f8a0..d653914150 100644 --- a/p2p/host/autonat/addr.go +++ b/p2p/host/autonat/addr.go @@ -44,6 +44,7 @@ func parsePrivateCIDR(cidrs []string) []*net.IPNet { return ipnets } +// IsPublicAddr retruns true if the IP part of the multiaddr is not in a private network func IsPublicAddr(a ma.Multiaddr) bool { ip, err := a.ValueForProtocol(ma.P_IP4) if err == nil { From 33423558c527d3e03a71e9937067cfb9c3b5677d Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Oct 2018 15:14:34 +0300 Subject: [PATCH 0757/3965] service implementation --- p2p/host/autonat/ci/Jenkinsfile | 1 + p2p/host/autonat/proto.go | 24 ++++ p2p/host/autonat/svc.go | 215 ++++++++++++++++++++++++++++++++ p2p/host/autonat/svc_test.go | 132 ++++++++++++++++++++ 4 files changed, 372 insertions(+) create mode 100644 p2p/host/autonat/ci/Jenkinsfile create mode 100644 p2p/host/autonat/proto.go create mode 100644 p2p/host/autonat/svc.go create mode 100644 p2p/host/autonat/svc_test.go diff --git a/p2p/host/autonat/ci/Jenkinsfile b/p2p/host/autonat/ci/Jenkinsfile new file mode 100644 index 0000000000..b2067e6232 --- /dev/null +++ b/p2p/host/autonat/ci/Jenkinsfile @@ -0,0 +1 @@ +golang() diff --git a/p2p/host/autonat/proto.go b/p2p/host/autonat/proto.go new file mode 100644 index 0000000000..c0bd44a869 --- /dev/null +++ b/p2p/host/autonat/proto.go @@ -0,0 +1,24 @@ +package autonat + +import ( + pb "github.com/libp2p/go-libp2p-autonat/pb" + + logging "github.com/ipfs/go-log" + ma "github.com/multiformats/go-multiaddr" +) + +var log = logging.Logger("autonat-svc") + +func newDialResponseOK(addr ma.Multiaddr) *pb.Message_DialResponse { + dr := new(pb.Message_DialResponse) + dr.Status = pb.Message_OK.Enum() + dr.Addr = addr.Bytes() + return dr +} + +func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Message_DialResponse { + dr := new(pb.Message_DialResponse) + dr.Status = status.Enum() + dr.StatusText = &text + return dr +} diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go new file mode 100644 index 0000000000..e523af69f6 --- /dev/null +++ b/p2p/host/autonat/svc.go @@ -0,0 +1,215 @@ +package autonat + +import ( + "context" + "sync" + "time" + + pb "github.com/libp2p/go-libp2p-autonat/pb" + + ggio "github.com/gogo/protobuf/io" + libp2p "github.com/libp2p/go-libp2p" + autonat "github.com/libp2p/go-libp2p-autonat" + host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" +) + +const P_CIRCUIT = 290 + +var ( + AutoNATServiceDialTimeout = 42 * time.Second + AutoNATServiceResetInterval = 1 * time.Minute + + AutoNATServiceThrottle = 3 +) + +// AutoNATService provides NAT autodetection services to other peers +type AutoNATService struct { + ctx context.Context + dialer host.Host + + // rate limiter + mx sync.Mutex + reqs map[peer.ID]int +} + +// NewAutoNATService creates a new AutoNATService instance attached to a host +func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) (*AutoNATService, error) { + opts = append(opts, libp2p.NoListenAddrs) + dialer, err := libp2p.New(ctx, opts...) + if err != nil { + return nil, err + } + + as := &AutoNATService{ + ctx: ctx, + dialer: dialer, + reqs: make(map[peer.ID]int), + } + h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) + + go as.resetRateLimiter() + + return as, nil +} + +func (as *AutoNATService) handleStream(s inet.Stream) { + defer s.Close() + + pid := s.Conn().RemotePeer() + log.Debugf("New stream from %s", pid.Pretty()) + + r := ggio.NewDelimitedReader(s, inet.MessageSizeMax) + w := ggio.NewDelimitedWriter(s) + + var req pb.Message + var res pb.Message + + err := r.ReadMsg(&req) + if err != nil { + log.Debugf("Error reading message from %s: %s", pid.Pretty(), err.Error()) + s.Reset() + return + } + + t := req.GetType() + if t != pb.Message_DIAL { + log.Debugf("Unexpected message from %s: %s (%d)", pid.Pretty(), t.String(), t) + s.Reset() + return + } + + dr := as.handleDial(pid, s.Conn().RemoteMultiaddr(), req.GetDial().GetPeer()) + res.Type = pb.Message_DIAL_RESPONSE.Enum() + res.DialResponse = dr + + err = w.WriteMsg(&res) + if err != nil { + log.Debugf("Error writing response to %s: %s", pid.Pretty(), err.Error()) + s.Reset() + return + } +} + +func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { + if mpi == nil { + return newDialResponseError(pb.Message_E_BAD_REQUEST, "missing peer info") + } + + mpid := mpi.GetId() + if mpid != nil { + mp, err := peer.IDFromBytes(mpid) + if err != nil { + return newDialResponseError(pb.Message_E_BAD_REQUEST, "bad peer id") + } + + if mp != p { + return newDialResponseError(pb.Message_E_BAD_REQUEST, "peer id mismatch") + } + } + + addrs := make([]ma.Multiaddr, 0) + seen := make(map[string]struct{}) + + // add observed addr to the list of addresses to dial + if !as.skipDial(obsaddr) { + addrs = append(addrs, obsaddr) + seen[obsaddr.String()] = struct{}{} + } + + for _, maddr := range mpi.GetAddrs() { + addr, err := ma.NewMultiaddrBytes(maddr) + if err != nil { + log.Debugf("Error parsing multiaddr: %s", err.Error()) + continue + } + + if as.skipDial(addr) { + continue + } + + str := addr.String() + _, ok := seen[str] + if ok { + continue + } + + addrs = append(addrs, addr) + seen[str] = struct{}{} + } + + if len(addrs) == 0 { + return newDialResponseError(pb.Message_E_DIAL_ERROR, "no dialable addresses") + } + + return as.doDial(pstore.PeerInfo{ID: p, Addrs: addrs}) +} + +func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { + // skip relay addresses + _, err := addr.ValueForProtocol(P_CIRCUIT) + if err == nil { + return true + } + + // skip private network (unroutable) addresses + if !autonat.IsPublicAddr(addr) { + return true + } + + return false +} + +func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { + // rate limit check + as.mx.Lock() + count := as.reqs[pi.ID] + if count >= AutoNATServiceThrottle { + as.mx.Unlock() + return newDialResponseError(pb.Message_E_DIAL_REFUSED, "too many dials") + } + as.reqs[pi.ID] = count + 1 + as.mx.Unlock() + + ctx, cancel := context.WithTimeout(as.ctx, AutoNATServiceDialTimeout) + defer cancel() + + err := as.dialer.Connect(ctx, pi) + if err != nil { + log.Debugf("error dialing %s: %s", pi.ID.Pretty(), err.Error()) + // wait for the context to timeout to avoid leaking timing information + // this renders the service ineffective as a port scanner + <-ctx.Done() + return newDialResponseError(pb.Message_E_DIAL_ERROR, "dial failed") + } + + conns := as.dialer.Network().ConnsToPeer(pi.ID) + if len(conns) == 0 { + log.Errorf("supposedly connected to %s, but no connection to peer", pi.ID.Pretty()) + return newDialResponseError(pb.Message_E_INTERNAL_ERROR, "internal service error") + } + + ra := conns[0].RemoteMultiaddr() + as.dialer.Network().ClosePeer(pi.ID) + return newDialResponseOK(ra) +} + +func (as *AutoNATService) resetRateLimiter() { + ticker := time.NewTicker(AutoNATServiceResetInterval) + defer ticker.Stop() + + for { + select { + case <-ticker.C: + as.mx.Lock() + as.reqs = make(map[peer.ID]int) + as.mx.Unlock() + + case <-as.ctx.Done(): + return + } + } +} diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go new file mode 100644 index 0000000000..65b2a9dd74 --- /dev/null +++ b/p2p/host/autonat/svc_test.go @@ -0,0 +1,132 @@ +package autonat + +import ( + "context" + "net" + "testing" + "time" + + libp2p "github.com/libp2p/go-libp2p" + autonat "github.com/libp2p/go-libp2p-autonat" + host "github.com/libp2p/go-libp2p-host" + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATService) { + h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) + if err != nil { + t.Fatal(err) + } + + as, err := NewAutoNATService(ctx, h) + if err != nil { + t.Fatal(err) + } + + return h, as +} + +func makeAutoNATClient(ctx context.Context, t *testing.T) (host.Host, autonat.AutoNATClient) { + h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) + if err != nil { + t.Fatal(err) + } + + cli := autonat.NewAutoNATClient(h) + return h, cli +} + +func connect(t *testing.T, a, b host.Host) { + pinfo := pstore.PeerInfo{ID: a.ID(), Addrs: a.Addrs()} + err := b.Connect(context.Background(), pinfo) + if err != nil { + t.Fatal(err) + } +} + +// Note: these tests assume that the host has only private inet addresses! +func TestAutoNATServiceDialError(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save := AutoNATServiceDialTimeout + AutoNATServiceDialTimeout = 1 * time.Second + + hs, _ := makeAutoNATService(ctx, t) + hc, ac := makeAutoNATClient(ctx, t) + connect(t, hs, hc) + + _, err := ac.DialBack(ctx, hs.ID()) + if err == nil { + t.Fatal("Dial back succeeded unexpectedly!") + } + + if !autonat.IsDialError(err) { + t.Fatal(err) + } + + AutoNATServiceDialTimeout = save +} + +func TestAutoNATServiceDialSuccess(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save := autonat.Private4 + autonat.Private4 = []*net.IPNet{} + + hs, _ := makeAutoNATService(ctx, t) + hc, ac := makeAutoNATClient(ctx, t) + connect(t, hs, hc) + + _, err := ac.DialBack(ctx, hs.ID()) + if err != nil { + t.Fatalf("Dial back failed: %s", err.Error()) + } + + autonat.Private4 = save +} + +func TestAutoNATServiceDialRateLimiter(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save1 := AutoNATServiceDialTimeout + AutoNATServiceDialTimeout = 1 * time.Second + save2 := AutoNATServiceResetInterval + AutoNATServiceResetInterval = 1 * time.Second + save3 := AutoNATServiceThrottle + AutoNATServiceThrottle = 1 + save4 := autonat.Private4 + autonat.Private4 = []*net.IPNet{} + + hs, _ := makeAutoNATService(ctx, t) + hc, ac := makeAutoNATClient(ctx, t) + connect(t, hs, hc) + + _, err := ac.DialBack(ctx, hs.ID()) + if err != nil { + t.Fatal(err) + } + + _, err = ac.DialBack(ctx, hs.ID()) + if err == nil { + t.Fatal("Dial back succeeded unexpectedly!") + } + + if !autonat.IsDialRefused(err) { + t.Fatal(err) + } + + time.Sleep(2 * time.Second) + + _, err = ac.DialBack(ctx, hs.ID()) + if err != nil { + t.Fatal(err) + } + + AutoNATServiceDialTimeout = save1 + AutoNATServiceResetInterval = save2 + AutoNATServiceThrottle = save3 + autonat.Private4 = save4 +} From b11fa1c0604446c90f32cdf5429b095d776b9f50 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Oct 2018 16:33:22 +0300 Subject: [PATCH 0758/3965] move addr.go to go-multiaddr-net --- p2p/host/autonat/addr.go | 71 ---------------------------------------- 1 file changed, 71 deletions(-) delete mode 100644 p2p/host/autonat/addr.go diff --git a/p2p/host/autonat/addr.go b/p2p/host/autonat/addr.go deleted file mode 100644 index d653914150..0000000000 --- a/p2p/host/autonat/addr.go +++ /dev/null @@ -1,71 +0,0 @@ -package autonat - -import ( - "net" - - ma "github.com/multiformats/go-multiaddr" -) - -var Private4, Private6 []*net.IPNet -var privateCIDR4 = []string{ - // localhost - "127.0.0.0/8", - // private networks - "10.0.0.0/8", - "100.64.0.0/10", - "172.16.0.0/12", - "192.168.0.0/16", - // link local - "169.254.0.0/16", -} -var privateCIDR6 = []string{ - // localhost - "::1/128", - // ULA reserved - "fc00::/7", - // link local - "fe80::/10", -} - -func init() { - Private4 = parsePrivateCIDR(privateCIDR4) - Private6 = parsePrivateCIDR(privateCIDR6) -} - -func parsePrivateCIDR(cidrs []string) []*net.IPNet { - ipnets := make([]*net.IPNet, len(cidrs)) - for i, cidr := range cidrs { - _, ipnet, err := net.ParseCIDR(cidr) - if err != nil { - panic(err) - } - ipnets[i] = ipnet - } - return ipnets -} - -// IsPublicAddr retruns true if the IP part of the multiaddr is not in a private network -func IsPublicAddr(a ma.Multiaddr) bool { - ip, err := a.ValueForProtocol(ma.P_IP4) - if err == nil { - return !inAddrRange(ip, Private4) - } - - ip, err = a.ValueForProtocol(ma.P_IP6) - if err == nil { - return !inAddrRange(ip, Private6) - } - - return false -} - -func inAddrRange(s string, ipnets []*net.IPNet) bool { - ip := net.ParseIP(s) - for _, ipnet := range ipnets { - if ipnet.Contains(ip) { - return true - } - } - - return false -} From 605e4e6ebe6b726a0f96255a9f57e52fd6e7de36 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Oct 2018 16:48:46 +0300 Subject: [PATCH 0759/3965] IsPublicAddr lives in multiaddr-net --- p2p/host/autonat/svc.go | 3 ++- p2p/host/autonat/svc_test.go | 13 +++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index e523af69f6..b874b7d1a6 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -15,6 +15,7 @@ import ( peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) const P_CIRCUIT = 290 @@ -156,7 +157,7 @@ func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { } // skip private network (unroutable) addresses - if !autonat.IsPublicAddr(addr) { + if !manet.IsPublicAddr(addr) { return true } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 65b2a9dd74..9c8d969188 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -10,6 +10,7 @@ import ( autonat "github.com/libp2p/go-libp2p-autonat" host "github.com/libp2p/go-libp2p-host" pstore "github.com/libp2p/go-libp2p-peerstore" + manet "github.com/multiformats/go-multiaddr-net" ) func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATService) { @@ -72,8 +73,8 @@ func TestAutoNATServiceDialSuccess(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save := autonat.Private4 - autonat.Private4 = []*net.IPNet{} + save := manet.Private4 + manet.Private4 = []*net.IPNet{} hs, _ := makeAutoNATService(ctx, t) hc, ac := makeAutoNATClient(ctx, t) @@ -84,7 +85,7 @@ func TestAutoNATServiceDialSuccess(t *testing.T) { t.Fatalf("Dial back failed: %s", err.Error()) } - autonat.Private4 = save + manet.Private4 = save } func TestAutoNATServiceDialRateLimiter(t *testing.T) { @@ -97,8 +98,8 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { AutoNATServiceResetInterval = 1 * time.Second save3 := AutoNATServiceThrottle AutoNATServiceThrottle = 1 - save4 := autonat.Private4 - autonat.Private4 = []*net.IPNet{} + save4 := manet.Private4 + manet.Private4 = []*net.IPNet{} hs, _ := makeAutoNATService(ctx, t) hc, ac := makeAutoNATClient(ctx, t) @@ -128,5 +129,5 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { AutoNATServiceDialTimeout = save1 AutoNATServiceResetInterval = save2 AutoNATServiceThrottle = save3 - autonat.Private4 = save4 + manet.Private4 = save4 } From 1d7a9075e98d09b08230609892e64ab39f8ab272 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 12:20:02 +0300 Subject: [PATCH 0760/3965] make relay discovery optional (disabled by default) --- .../internal/circuitv1-deprecated/relay.go | 17 ++++++++++++----- .../internal/circuitv1-deprecated/relay_test.go | 14 +++++++------- .../circuitv1-deprecated/transport_test.go | 2 +- 3 files changed, 20 insertions(+), 13 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 96c30c9b2b..a5323e7bbf 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -33,8 +33,9 @@ type Relay struct { ctx context.Context self peer.ID - active bool - hop bool + active bool + hop bool + discovery bool incoming chan *Conn @@ -49,8 +50,9 @@ type Relay struct { type RelayOpt int var ( - OptActive = RelayOpt(0) - OptHop = RelayOpt(1) + OptActive = RelayOpt(0) + OptHop = RelayOpt(1) + OptDiscovery = RelayOpt(2) ) type RelayError struct { @@ -78,13 +80,18 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. r.active = true case OptHop: r.hop = true + case OptDiscovery: + r.discovery = true default: return nil, fmt.Errorf("unrecognized option: %d", opt) } } h.SetStreamHandler(ProtoID, r.handleNewStream) - h.Network().Notify(r.Notifiee()) + + if r.discovery { + h.Network().Notify(r.Notifiee()) + } return r, nil } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 6fdc1d62e8..58f641f473 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -66,7 +66,7 @@ func TestBasicRelay(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0]) + r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) newTestRelay(t, ctx, hosts[1], OptHop) @@ -122,7 +122,7 @@ func TestRelayReset(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0]) + r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) newTestRelay(t, ctx, hosts[1], OptHop) @@ -181,7 +181,7 @@ func TestBasicRelayDial(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0]) + r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) _ = newTestRelay(t, ctx, hosts[1], OptHop) r3 := newTestRelay(t, ctx, hosts[2]) @@ -230,7 +230,7 @@ func TestUnspecificRelayDial(t *testing.T) { hosts := getNetHosts(t, ctx, 3) - r1 := newTestRelay(t, ctx, hosts[0]) + r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) newTestRelay(t, ctx, hosts[1], OptHop) @@ -290,7 +290,7 @@ func TestRelayThroughNonHop(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0]) + r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) newTestRelay(t, ctx, hosts[1]) @@ -327,7 +327,7 @@ func TestRelayNoDestConnection(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0]) + r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) newTestRelay(t, ctx, hosts[1], OptHop) @@ -362,7 +362,7 @@ func TestActiveRelay(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0]) + r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) newTestRelay(t, ctx, hosts[1], OptHop, OptActive) diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go index b718a90fe7..23e428783f 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go @@ -25,7 +25,7 @@ var msg = []byte("relay works!") func testSetupRelay(t *testing.T, ctx context.Context) []host.Host { hosts := getNetHosts(t, ctx, 3) - err := AddRelayTransport(ctx, hosts[0], swarmt.GenUpgrader(hosts[0].Network().(*swarm.Swarm))) + err := AddRelayTransport(ctx, hosts[0], swarmt.GenUpgrader(hosts[0].Network().(*swarm.Swarm)), OptDiscovery) if err != nil { t.Fatal(err) } From 80a994aeb0773a6ba6de34d94d60be00b82a4d88 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Wed, 17 Oct 2018 15:56:04 +0200 Subject: [PATCH 0761/3965] Add go-libp2p-gorpc to package list. Apparently this also includes some new Jenkins badges. License: MIT Signed-off-by: Hector Sanjuan --- README.md | 11 +++++++---- package-list.json | 5 ++++- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index f3075e948a..bb774c34a6 100644 --- a/README.md +++ b/README.md @@ -151,14 +151,16 @@ List of packages currently in existence for libp2p: | **Routing** | | [`go-libp2p-routing`](//github.com/libp2p/go-libp2p-routing) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing) | routing interfaces | | [`go-libp2p-record`](//github.com/libp2p/go-libp2p-record) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-record.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-record) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-record/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-record) | record type and validator logic | -| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | +| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-routing-helpers/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-routing-helpers/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | | [`go-libp2p-kad-dht`](//github.com/libp2p/go-libp2p-kad-dht) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kad-dht) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-kad-dht/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-kad-dht/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kad-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kad-dht) | Kademlia-like router | -| [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub-router) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | +| [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub-router) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-pubsub-router/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-pubsub-router/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | | **Consensus** | | [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-consensus) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | | [`go-libp2p-raft`](//github.com/libp2p/go-libp2p-raft) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-raft.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-raft) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-raft/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-raft/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-raft/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-raft) | consensus implementation over raft | | **Pubsub** | -| [`go-floodsub`](//github.com/libp2p/go-floodsub) | [![Travis CI](https://travis-ci.org/libp2p/go-floodsub.svg?branch=master)](https://travis-ci.org/libp2p/go-floodsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-floodsub/master)](https://ci.ipfs.team/job/libp2p/job/go-floodsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-floodsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-floodsub) | pubsub implementation | +| [`go-floodsub`](//github.com/libp2p/go-floodsub) | [![Travis CI](https://travis-ci.org/libp2p/go-floodsub.svg?branch=master)](https://travis-ci.org/libp2p/go-floodsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-floodsub/master)](https://ci.ipfs.team/job/libp2p/job/go-floodsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-floodsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-floodsub) | basic implementation of pubsub driven by flooding | +| **RPC** | +| [`go-libp2p-gorpc`](//github.com/libp2p/go-libp2p-gorpc) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-gorpc.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-gorpc) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-gorpc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-gorpc) | a simple RPC library for libp2p | | **Metrics** | | [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | libp2p metrics interfaces/collectors | | **Data Types** | @@ -174,7 +176,7 @@ List of packages currently in existence for libp2p: | [`go-addr-util`](//github.com/libp2p/go-addr-util) | [![Travis CI](https://travis-ci.org/libp2p/go-addr-util.svg?branch=master)](https://travis-ci.org/libp2p/go-addr-util) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-addr-util/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-addr-util) | address utilities for libp2p swarm | | [`go-buffer-pool`](//github.com/libp2p/go-buffer-pool) | [![Travis CI](https://travis-ci.org/libp2p/go-buffer-pool.svg?branch=master)](https://travis-ci.org/libp2p/go-buffer-pool) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-buffer-pool/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-buffer-pool) | a variable size buffer pool for go | | [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | -| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | routing helpers | +| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-routing-helpers/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-routing-helpers/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | routing helpers | | [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | a library to perform filtering of multiaddrs. | | [`go-reuseport`](//github.com/libp2p/go-reuseport) | [![Travis CI](https://travis-ci.org/libp2p/go-reuseport.svg?branch=master)](https://travis-ci.org/libp2p/go-reuseport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-reuseport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-reuseport) | enables reuse of addresses | | [`go-sockaddr`](//github.com/libp2p/go-sockaddr) | [![Travis CI](https://travis-ci.org/libp2p/go-sockaddr.svg?branch=master)](https://travis-ci.org/libp2p/go-sockaddr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-sockaddr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-sockaddr) | utils for sockaddr conversions | @@ -184,6 +186,7 @@ List of packages currently in existence for libp2p: | [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-examples) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-examples/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-examples/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | | [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit-progs) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | + # Contribute go-libp2p is part of [The IPFS Project](https://github.com/ipfs/ipfs), and is MIT licensed open source software. We welcome contributions big and small! Take a look at the [community contributing notes](https://github.com/ipfs/community/blob/master/contributing.md). Please make sure to check the [issues](https://github.com/ipfs/go-libp2p/issues). Search the closed ones before reporting things, and help us with the open ones. diff --git a/package-list.json b/package-list.json index 0d741ae13c..9f3af9c198 100644 --- a/package-list.json +++ b/package-list.json @@ -65,6 +65,9 @@ "Pubsub", ["libp2p/go-floodsub", "go-floodsub", "basic implementation of pubsub driven by flooding"], + "RPC", + ["libp2p/go-libp2p-gorpc", "go-libp2p-gorpc", "a simple RPC library for libp2p"], + "Metrics", ["libp2p/go-libp2p-metrics", "go-libp2p-metrics", "libp2p metrics interfaces/collectors"], @@ -73,7 +76,7 @@ ["libp2p/go-libp2p-crypto", "go-libp2p-crypto", "libp2p key types"], ["libp2p/go-libp2p-protocol", "go-libp2p-protocol", "libp2p protocol datatype"], ["libp2p/go-libp2p-kbucket", "go-libp2p-kbucket", "Kademlia routing table helper types"], - + "Utilities/miscellaneous", ["libp2p/go-libp2p-loggables", "go-libp2p-loggables", "logging helpers"], ["libp2p/go-maddr-filter", "go-maddr-filter", "multiaddr filtering helpers"], From 9a4502abb523346d1c005142ecb654def0f66024 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 23:51:32 +0300 Subject: [PATCH 0762/3965] add address factory --- p2p/host/autonat/autonat.go | 20 ++++++++++++++------ p2p/host/autonat/client.go | 16 ++++++++++++---- 2 files changed, 26 insertions(+), 10 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 1eace01770..176c77a2eb 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -48,6 +48,8 @@ type AmbientAutoNAT struct { ctx context.Context host host.Host + getAddrs GetAddrs + mx sync.Mutex peers map[peer.ID]struct{} status NATStatus @@ -61,12 +63,18 @@ type AmbientAutoNAT struct { } // NewAutoNAT creates a new ambient NAT autodiscovery instance attached to a host -func NewAutoNAT(ctx context.Context, h host.Host) AutoNAT { +func NewAutoNAT(ctx context.Context, h host.Host, ga ...GetAddrs) AutoNAT { + getAddrs := h.Addrs + if len(ga) > 0 { + getAddrs = ga[0] + } + as := &AmbientAutoNAT{ - ctx: ctx, - host: h, - peers: make(map[peer.ID]struct{}), - status: NATStatusUnknown, + ctx: ctx, + host: h, + getAddrs: getAddrs, + peers: make(map[peer.ID]struct{}), + status: NATStatusUnknown, } h.Network().Notify(as) @@ -123,7 +131,7 @@ func (as *AmbientAutoNAT) autodetect() { return } - cli := NewAutoNATClient(as.host) + cli := NewAutoNATClient(as.host, as.getAddrs) failures := 0 for _, p := range peers { diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index 468bc68106..96c5255faa 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -27,13 +27,21 @@ type AutoNATError struct { Text string } +// GetAddrs is a function that returns the addresses to dial back +type GetAddrs func() []ma.Multiaddr + // NewAutoNATClient creates a fresh instance of an AutoNATClient -func NewAutoNATClient(h host.Host) AutoNATClient { - return &client{h: h} +func NewAutoNATClient(h host.Host, ga ...GetAddrs) AutoNATClient { + getAddrs := h.Addrs + if len(ga) > 0 { + getAddrs = ga[0] + } + return &client{h: h, getAddrs: getAddrs} } type client struct { - h host.Host + h host.Host + getAddrs GetAddrs } func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) { @@ -46,7 +54,7 @@ func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) r := ggio.NewDelimitedReader(s, inet.MessageSizeMax) w := ggio.NewDelimitedWriter(s) - req := newDialMessage(pstore.PeerInfo{ID: c.h.ID(), Addrs: c.h.Addrs()}) + req := newDialMessage(pstore.PeerInfo{ID: c.h.ID(), Addrs: c.getAddrs()}) err = w.WriteMsg(req) if err != nil { return nil, err From 6e5e92862db11e30fcbb0e7770f88e3e17d23f2e Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 12:59:01 +0300 Subject: [PATCH 0763/3965] niceties and best practices --- p2p/discovery/routing/routing.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index f1cec03a0f..6606267d75 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -12,10 +12,10 @@ import ( // RoutingDiscovery is an implementation of discovery using ContentRouting type RoutingDiscovery struct { - router routing.ContentRouting + routing.ContentRouting } -func NewRoutingDiscovery(router routing.ContentRouting) Discovery { +func NewRoutingDiscovery(router routing.ContentRouting) *RoutingDiscovery { return &RoutingDiscovery{router} } @@ -25,7 +25,7 @@ func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Opt return 0, err } - err = d.router.Provide(ctx, cid, true) + err = d.Provide(ctx, cid, true) if err != nil { return 0, err } @@ -35,7 +35,7 @@ func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Opt } func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Option) (<-chan pstore.PeerInfo, error) { - options := &Options{} + var options Options err := options.Apply(opts...) if err != nil { return nil, err @@ -51,7 +51,7 @@ func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Opt return nil, err } - return d.router.FindProvidersAsync(ctx, cid, limit), nil + return d.FindProvidersAsync(ctx, cid, limit), nil } func nsToCid(ns string) (cid.Cid, error) { From 0cd9c1bf2d671a5abb1e8c166105111653d93ce3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 14:12:18 +0300 Subject: [PATCH 0764/3965] test and gx package update --- p2p/discovery/routing/routing_test.go | 105 ++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 p2p/discovery/routing/routing_test.go diff --git a/p2p/discovery/routing/routing_test.go b/p2p/discovery/routing/routing_test.go new file mode 100644 index 0000000000..82cebf31e1 --- /dev/null +++ b/p2p/discovery/routing/routing_test.go @@ -0,0 +1,105 @@ +package discovery + +import ( + "context" + "sync" + "testing" + + cid "github.com/ipfs/go-cid" + bhost "github.com/libp2p/go-libp2p-blankhost" + host "github.com/libp2p/go-libp2p-host" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" +) + +type mockRoutingTable struct { + mx sync.Mutex + providers map[string]map[peer.ID]pstore.PeerInfo +} + +type mockRouting struct { + h host.Host + tab *mockRoutingTable +} + +func NewMockRoutingTable() *mockRoutingTable { + return &mockRoutingTable{providers: make(map[string]map[peer.ID]pstore.PeerInfo)} +} + +func NewMockRouting(h host.Host, tab *mockRoutingTable) *mockRouting { + return &mockRouting{h: h, tab: tab} +} + +func (m *mockRouting) Provide(ctx context.Context, cid cid.Cid, bcast bool) error { + m.tab.mx.Lock() + defer m.tab.mx.Unlock() + + pmap, ok := m.tab.providers[cid.String()] + if !ok { + pmap = make(map[peer.ID]pstore.PeerInfo) + m.tab.providers[cid.String()] = pmap + } + + pmap[m.h.ID()] = pstore.PeerInfo{ID: m.h.ID(), Addrs: m.h.Addrs()} + + return nil +} + +func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit int) <-chan pstore.PeerInfo { + ch := make(chan pstore.PeerInfo) + go func() { + defer close(ch) + m.tab.mx.Lock() + defer m.tab.mx.Unlock() + + pmap, ok := m.tab.providers[cid.String()] + if !ok { + return + } + + for _, pi := range pmap { + select { + case ch <- pi: + case <-ctx.Done(): + return + } + } + }() + + return ch +} + +func TestRoutingDiscovery(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + h1 := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h2 := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + + mtab := NewMockRoutingTable() + mr1 := NewMockRouting(h1, mtab) + mr2 := NewMockRouting(h2, mtab) + + d1 := NewRoutingDiscovery(mr1) + d2 := NewRoutingDiscovery(mr2) + + _, err := d1.Advertise(ctx, "/test") + if err != nil { + t.Fatal(err) + } + + pis, err := FindPeers(ctx, d2, "/test", 20) + if err != nil { + t.Fatal(err) + } + + if len(pis) != 1 { + t.Fatalf("Expected 1 peer, got %d", len(pis)) + } + + pi := pis[0] + if pi.ID != h1.ID() { + t.Fatalf("Unexpected peer: %s", pi.ID) + } +} From 25577e94fddabbac17a7ec3f5ef2ff1a3738939c Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 14:15:52 +0300 Subject: [PATCH 0765/3965] document the use of SHA256 as the cid hash function --- p2p/discovery/routing/routing.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index 6606267d75..f79726c930 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -11,6 +11,7 @@ import ( ) // RoutingDiscovery is an implementation of discovery using ContentRouting +// Namespaces are translated to Cids using the SHA256 hash. type RoutingDiscovery struct { routing.ContentRouting } From c3acc1c21b4bdec0b88cc23f4a84ea1c0a362d1b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 18 Oct 2018 15:23:29 +0100 Subject: [PATCH 0766/3965] gx: update go-libp2p-circuit (disables discovery by default) --- package.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/package.json b/package.json index 48628ba0b6..e7e7d50177 100644 --- a/package.json +++ b/package.json @@ -169,9 +169,9 @@ }, { "author": "vyzo", - "hash": "Qma2mxvX5zvPEmjXC4CktjExnRjVp4DFcog4GNubpqSXsD", + "hash": "Qmbqn6qCocM5Dk9idyUw6wU1vsZCd1aZ3yJpwPA27YpE6J", "name": "go-libp2p-circuit", - "version": "2.2.8" + "version": "2.3.0" }, { "author": "lgierth", From 9237347ad27b6affc620db21b5a202f2005f238b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 18 Oct 2018 15:29:21 +0100 Subject: [PATCH 0767/3965] ci: update go --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index e1be0bfa30..6f8b950982 100644 --- a/.travis.yml +++ b/.travis.yml @@ -7,7 +7,7 @@ sudo: false language: go go: - - 1.9.x + - 1.11.1 install: - make deps From 00f3153d29959965f0564042ebdbf08cb5a5c3e3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 18:05:26 +0300 Subject: [PATCH 0768/3965] make getAddrs a static argument (not variadic) --- p2p/host/autonat/autonat.go | 8 ++++---- p2p/host/autonat/autonat_test.go | 2 +- p2p/host/autonat/client.go | 8 ++++---- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 176c77a2eb..0aa987ca63 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -63,10 +63,10 @@ type AmbientAutoNAT struct { } // NewAutoNAT creates a new ambient NAT autodiscovery instance attached to a host -func NewAutoNAT(ctx context.Context, h host.Host, ga ...GetAddrs) AutoNAT { - getAddrs := h.Addrs - if len(ga) > 0 { - getAddrs = ga[0] +// If getAddrs is nil, h.Addrs will be used +func NewAutoNAT(ctx context.Context, h host.Host, getAddrs GetAddrs) AutoNAT { + if getAddrs == nil { + getAddrs = h.Addrs } as := &AmbientAutoNAT{ diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 6170d01d19..aa8d879135 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -72,7 +72,7 @@ func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Mes func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, AutoNAT) { h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) - a := NewAutoNAT(ctx, h) + a := NewAutoNAT(ctx, h, nil) a.(*AmbientAutoNAT).peers[ash.ID()] = struct{}{} return h, a diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index 96c5255faa..0fd665d3df 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -31,10 +31,10 @@ type AutoNATError struct { type GetAddrs func() []ma.Multiaddr // NewAutoNATClient creates a fresh instance of an AutoNATClient -func NewAutoNATClient(h host.Host, ga ...GetAddrs) AutoNATClient { - getAddrs := h.Addrs - if len(ga) > 0 { - getAddrs = ga[0] +// If getAddrs is nil, h.Addrs will be used +func NewAutoNATClient(h host.Host, getAddrs GetAddrs) AutoNATClient { + if getAddrs == nil { + getAddrs = h.Addrs } return &client{h: h, getAddrs: getAddrs} } From 1536433f6b64d0171cecffda745a9a0ede14dc59 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 18:37:35 +0300 Subject: [PATCH 0769/3965] use TempAddrTTL for relay addrs --- p2p/host/routed/routed.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/routed/routed.go b/p2p/host/routed/routed.go index 97b58a9f63..d921536895 100644 --- a/p2p/host/routed/routed.go +++ b/p2p/host/routed/routed.go @@ -102,7 +102,7 @@ func (rh *RoutedHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { continue } - rh.Peerstore().AddAddrs(relayID, relayAddrs, pstore.AddressTTL) + rh.Peerstore().AddAddrs(relayID, relayAddrs, pstore.TempAddrTTL) } // if we're here, we got some addrs. let's use our wrapped host to connect. From 9a8a3d7437b5ce6dc68cdef46aafb2c04af57377 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 18:55:10 +0300 Subject: [PATCH 0770/3965] update package list for discovery and autonat --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index bb774c34a6..d5a5f1e9d3 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,9 @@ List of packages currently in existence for libp2p: | [`go-reuseport`](//github.com/libp2p/go-reuseport) | [![Travis CI](https://travis-ci.org/libp2p/go-reuseport.svg?branch=master)](https://travis-ci.org/libp2p/go-reuseport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-reuseport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-reuseport) | enables reuse of addresses | | [`go-sockaddr`](//github.com/libp2p/go-sockaddr) | [![Travis CI](https://travis-ci.org/libp2p/go-sockaddr.svg?branch=master)](https://travis-ci.org/libp2p/go-sockaddr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-sockaddr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-sockaddr) | utils for sockaddr conversions | | [`go-flow-metrics`](//github.com/libp2p/go-flow-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-flow-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-flow-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-flow-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-flow-metrics) | metrics library | +| [`go-libp2p-discovery`](//github.com/libp2p/go-libp2p-discovery) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-discovery.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-discovery) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-discovery) | active peer disovery interface | +| [`go-libp2p-autonat`](//github.com/libp2p/go-libp2p-autonat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat) | NAT autodetection (client) | +| [`go-libp2p-autonat-svc`](//github.com/libp2p/go-libp2p-autonat-svc) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat-svc.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat-svc) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc) | NAT autodetection (service) | | **Testing and examples** | | [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | a collection of testing utilities for ipfs and libp2p | | [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-examples) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-examples/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-examples/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | From 55f24aa3fca8766e89f1e52ed448c80e0bb39bc0 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 21:28:39 +0300 Subject: [PATCH 0771/3965] shuffle packages to better spots --- README.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d5a5f1e9d3..a876f815d2 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,9 @@ List of packages currently in existence for libp2p: | [`go-smux-mplex`](//github.com/whyrusleeping/go-smux-mplex) | [![Travis CI](https://travis-ci.org/whyrusleeping/go-smux-mplex.svg?branch=master)](https://travis-ci.org/whyrusleeping/go-smux-mplex) | N/A | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-mplex/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-mplex) | MPLEX stream multiplexer | | **NAT Traversal** | | [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | | +| [`go-libp2p-autonat`](//github.com/libp2p/go-libp2p-autonat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat) | NAT autodetection (client) | +| [`go-libp2p-autonat-svc`](//github.com/libp2p/go-libp2p-autonat-svc) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat-svc.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat-svc) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc) | NAT autodetection (service) | + | **Peerstore** | | [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | interfaces and reference implementation | | **Connection Manager** | @@ -154,6 +157,9 @@ List of packages currently in existence for libp2p: | [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-routing-helpers/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-routing-helpers/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | | [`go-libp2p-kad-dht`](//github.com/libp2p/go-libp2p-kad-dht) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kad-dht) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-kad-dht/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-kad-dht/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kad-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kad-dht) | Kademlia-like router | | [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub-router) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-pubsub-router/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-pubsub-router/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | +| **Discovery** | +| [`go-libp2p-discovery`](//github.com/libp2p/go-libp2p-discovery) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-discovery.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-discovery) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-discovery) | active peer disovery interface | + | **Consensus** | | [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-consensus) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | | [`go-libp2p-raft`](//github.com/libp2p/go-libp2p-raft) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-raft.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-raft) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-raft/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-raft/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-raft/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-raft) | consensus implementation over raft | @@ -181,9 +187,6 @@ List of packages currently in existence for libp2p: | [`go-reuseport`](//github.com/libp2p/go-reuseport) | [![Travis CI](https://travis-ci.org/libp2p/go-reuseport.svg?branch=master)](https://travis-ci.org/libp2p/go-reuseport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-reuseport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-reuseport) | enables reuse of addresses | | [`go-sockaddr`](//github.com/libp2p/go-sockaddr) | [![Travis CI](https://travis-ci.org/libp2p/go-sockaddr.svg?branch=master)](https://travis-ci.org/libp2p/go-sockaddr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-sockaddr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-sockaddr) | utils for sockaddr conversions | | [`go-flow-metrics`](//github.com/libp2p/go-flow-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-flow-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-flow-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-flow-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-flow-metrics) | metrics library | -| [`go-libp2p-discovery`](//github.com/libp2p/go-libp2p-discovery) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-discovery.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-discovery) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-discovery) | active peer disovery interface | -| [`go-libp2p-autonat`](//github.com/libp2p/go-libp2p-autonat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat) | NAT autodetection (client) | -| [`go-libp2p-autonat-svc`](//github.com/libp2p/go-libp2p-autonat-svc) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat-svc.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat-svc) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc) | NAT autodetection (service) | | **Testing and examples** | | [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | a collection of testing utilities for ipfs and libp2p | | [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-examples) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-examples/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-examples/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | From 9592a3eab676268a42e2a7e5f7c3e2f61460ade0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 18 Oct 2018 21:20:34 +0100 Subject: [PATCH 0772/3965] package-list: fix pubsub link (for rename) --- README.md | 4 +--- package-list.json | 2 +- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index a876f815d2..2b76e680d9 100644 --- a/README.md +++ b/README.md @@ -145,7 +145,6 @@ List of packages currently in existence for libp2p: | [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | | | [`go-libp2p-autonat`](//github.com/libp2p/go-libp2p-autonat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat) | NAT autodetection (client) | | [`go-libp2p-autonat-svc`](//github.com/libp2p/go-libp2p-autonat-svc) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat-svc.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat-svc) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc) | NAT autodetection (service) | - | **Peerstore** | | [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | interfaces and reference implementation | | **Connection Manager** | @@ -159,12 +158,11 @@ List of packages currently in existence for libp2p: | [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub-router) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-pubsub-router/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-pubsub-router/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | | **Discovery** | | [`go-libp2p-discovery`](//github.com/libp2p/go-libp2p-discovery) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-discovery.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-discovery) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-discovery) | active peer disovery interface | - | **Consensus** | | [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-consensus) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | | [`go-libp2p-raft`](//github.com/libp2p/go-libp2p-raft) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-raft.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-raft) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-raft/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-raft/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-raft/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-raft) | consensus implementation over raft | | **Pubsub** | -| [`go-floodsub`](//github.com/libp2p/go-floodsub) | [![Travis CI](https://travis-ci.org/libp2p/go-floodsub.svg?branch=master)](https://travis-ci.org/libp2p/go-floodsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-floodsub/master)](https://ci.ipfs.team/job/libp2p/job/go-floodsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-floodsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-floodsub) | basic implementation of pubsub driven by flooding | +| [`go-libp2p-pubsub`](//github.com/libp2p/go-libp2p-pubsub) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-pubsub/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-pubsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub) | multiple pubsub over libp2p implementations | | **RPC** | | [`go-libp2p-gorpc`](//github.com/libp2p/go-libp2p-gorpc) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-gorpc.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-gorpc) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-gorpc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-gorpc) | a simple RPC library for libp2p | | **Metrics** | diff --git a/package-list.json b/package-list.json index 9f3af9c198..7e1f62c3c7 100644 --- a/package-list.json +++ b/package-list.json @@ -63,7 +63,7 @@ ["libp2p/go-libp2p-raft", "go-libp2p-raft", "consensus implementation over raft"], "Pubsub", - ["libp2p/go-floodsub", "go-floodsub", "basic implementation of pubsub driven by flooding"], + ["libp2p/go-libp2p-pubsub", "go-libp2p-pubsub", "multiple pubsub over libp2p implementations"], "RPC", ["libp2p/go-libp2p-gorpc", "go-libp2p-gorpc", "a simple RPC library for libp2p"], From a768598c501753847270655a5df5af9c4db451bd Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Oct 2018 12:22:13 +0300 Subject: [PATCH 0773/3965] free standing Ping function --- p2p/protocol/ping/ping.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/ping/ping.go b/p2p/protocol/ping/ping.go index ae0c47f75d..b76133cc16 100644 --- a/p2p/protocol/ping/ping.go +++ b/p2p/protocol/ping/ping.go @@ -72,7 +72,11 @@ func (p *PingService) PingHandler(s inet.Stream) { } func (ps *PingService) Ping(ctx context.Context, p peer.ID) (<-chan time.Duration, error) { - s, err := ps.Host.NewStream(ctx, p, ID) + return Ping(ctx, ps.Host, p) +} + +func Ping(ctx context.Context, h host.Host, p peer.ID) (<-chan time.Duration, error) { + s, err := h.NewStream(ctx, p, ID) if err != nil { return nil, err } @@ -92,7 +96,7 @@ func (ps *PingService) Ping(ctx context.Context, p peer.ID) (<-chan time.Duratio return } - ps.Host.Peerstore().RecordLatency(p, t) + h.Peerstore().RecordLatency(p, t) select { case out <- t: case <-ctx.Done(): From 0fa1c254793da6ce8eab0989e6ff4fd9ccaa2505 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Oct 2018 12:28:58 +0300 Subject: [PATCH 0774/3965] ping service in basic host --- p2p/host/basic/basic_host.go | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 094fcd36b0..d1bd0d653a 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -14,6 +14,7 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" protocol "github.com/libp2p/go-libp2p-protocol" identify "github.com/libp2p/go-libp2p/p2p/protocol/identify" + ping "github.com/libp2p/go-libp2p/p2p/protocol/ping" ma "github.com/multiformats/go-multiaddr" madns "github.com/multiformats/go-multiaddr-dns" msmux "github.com/multiformats/go-multistream" @@ -56,6 +57,7 @@ type BasicHost struct { network inet.Network mux *msmux.MultistreamMuxer ids *identify.IDService + pings *ping.PingService natmgr NATManager addrs AddrsFactory maResolver *madns.Resolver @@ -96,6 +98,9 @@ type HostOpts struct { // ConnManager is a libp2p connection manager ConnManager ifconnmgr.ConnManager + + // EnablePing indicates whether to instantiate the ping service + EnablePing bool } // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. @@ -149,6 +154,10 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, net.Notify(h.cmgr.Notifee()) } + if opts.EnablePing { + h.pings = ping.NewPingService(h) + } + net.SetConnHandler(h.newConnHandler) net.SetStreamHandler(h.newStreamHandler) return h, nil From 1056fa811928a85e07fd1e95a5998db555dc5e93 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Oct 2018 12:37:20 +0300 Subject: [PATCH 0775/3965] Ping constructor option, enabled by default --- config/config.go | 4 ++++ defaults.go | 9 +++++++++ options.go | 9 +++++++++ 3 files changed, 22 insertions(+) diff --git a/config/config.go b/config/config.go index 0348f19140..520cfb9e81 100644 --- a/config/config.go +++ b/config/config.go @@ -56,6 +56,9 @@ type Config struct { NATManager NATManagerC Peerstore pstore.Peerstore Reporter metrics.Reporter + + PingCustom bool + Ping bool } // NewNode constructs a new libp2p Host from the Config. @@ -102,6 +105,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { ConnManager: cfg.ConnManager, AddrsFactory: cfg.AddrsFactory, NATManager: cfg.NATManager, + EnablePing: cfg.Ping, }) if err != nil { swrm.Close() diff --git a/defaults.go b/defaults.go index 679fd7af5a..f44f84d4a5 100644 --- a/defaults.go +++ b/defaults.go @@ -75,6 +75,11 @@ var DefaultEnableRelay = func(cfg *Config) error { return cfg.Apply(EnableRelay()) } +// DefaultEnablePing enables the ping service by default +var DefaultEnablePing = func(cfg *Config) error { + return cfg.Apply(Ping(true)) +} + // Complete list of default options and when to fallback on them. // // Please *DON'T* specify default options any other way. Putting this all here @@ -111,6 +116,10 @@ var defaults = []struct { fallback: func(cfg *Config) bool { return !cfg.RelayCustom }, opt: DefaultEnableRelay, }, + { + fallback: func(cfg *Config) bool { return !cfg.PingCustom }, + opt: DefaultEnablePing, + }, } // Defaults configures libp2p to use the default options. Can be combined with diff --git a/options.go b/options.go index 224336ead6..9c69744cf9 100644 --- a/options.go +++ b/options.go @@ -252,6 +252,15 @@ func NATManager(nm config.NATManagerC) Option { } } +// Ping will configure libp2p to support the ping service; enable by default. +func Ping(enable bool) Option { + return func(cfg *Config) error { + cfg.PingCustom = true + cfg.Ping = enable + return nil + } +} + // NoListenAddrs will configure libp2p to not listen by default. // // This will both clear any configured listen addrs and prevent libp2p from From 48e0effd6afcadb1c93419257d98e0420ddea826 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Oct 2018 12:45:44 +0300 Subject: [PATCH 0776/3965] resolve import cycle in ping test --- p2p/protocol/ping/ping_test.go | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/p2p/protocol/ping/ping_test.go b/p2p/protocol/ping/ping_test.go index adcb3ca696..f8ea0cc79e 100644 --- a/p2p/protocol/ping/ping_test.go +++ b/p2p/protocol/ping/ping_test.go @@ -1,4 +1,4 @@ -package ping +package ping_test import ( "context" @@ -9,6 +9,7 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" swarmt "github.com/libp2p/go-libp2p-swarm/testing" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + ping "github.com/libp2p/go-libp2p/p2p/protocol/ping" ) func TestPing(t *testing.T) { @@ -26,14 +27,14 @@ func TestPing(t *testing.T) { t.Fatal(err) } - ps1 := NewPingService(h1) - ps2 := NewPingService(h2) + ps1 := ping.NewPingService(h1) + ps2 := ping.NewPingService(h2) testPing(t, ps1, h2.ID()) testPing(t, ps2, h1.ID()) } -func testPing(t *testing.T, ps *PingService, p peer.ID) { +func testPing(t *testing.T, ps *ping.PingService, p peer.ID) { pctx, cancel := context.WithCancel(context.Background()) defer cancel() ts, err := ps.Ping(pctx, p) From dd7868a1a7a26e2c2bc2c4a5771371a3d68db862 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 07:00:42 -0700 Subject: [PATCH 0777/3965] don't disconnect on protocol version mismatch Sending a protocol version is nice. However, this "disconnect if our versions are different" logic makes the version entirely useless (because we can't change it). Really, each indevidual protocol is versioned so let's just leave it at that. If we make a breaking change that requires a protocol bump, we can do that and then switch on the other side's version. However, we'll have to wait for the entire network to upgrade for that to work. --- p2p/protocol/identify/id.go | 36 ------------------------------------ package.json | 5 ----- 2 files changed, 41 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 4fc88dd450..2d32d1f418 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -2,12 +2,10 @@ package identify import ( "context" - "strings" "sync" pb "github.com/libp2p/go-libp2p/p2p/protocol/identify/pb" - semver "github.com/coreos/go-semver/semver" ggio "github.com/gogo/protobuf/io" logging "github.com/ipfs/go-log" ic "github.com/libp2p/go-libp2p-crypto" @@ -225,15 +223,6 @@ func (ids *IDService) consumeMessage(mes *pb.Identify, c inet.Conn) { pv := mes.GetProtocolVersion() av := mes.GetAgentVersion() - // version check. if we shouldn't talk, bail. - // TODO: at this point, we've already exchanged information. - // move this into a first handshake before the connection can open streams. - if !protocolVersionsAreCompatible(pv, LibP2PVersion) { - logProtocolMismatchDisconnect(c, pv, av) - c.Close() - return - } - ids.Host.Peerstore().Put(p, "ProtocolVersion", pv) ids.Host.Peerstore().Put(p, "AgentVersion", av) @@ -406,31 +395,6 @@ func addrInAddrs(a ma.Multiaddr, as []ma.Multiaddr) bool { return false } -// protocolVersionsAreCompatible checks that the two implementations -// can talk to each other. It will use semver, but for now while -// we're in tight development, we will return false for minor version -// changes too. -func protocolVersionsAreCompatible(v1, v2 string) bool { - if strings.HasPrefix(v1, "ipfs/") { - v1 = v1[5:] - } - if strings.HasPrefix(v2, "ipfs/") { - v2 = v2[5:] - } - - v1s, err := semver.NewVersion(v1) - if err != nil { - return false - } - - v2s, err := semver.NewVersion(v2) - if err != nil { - return false - } - - return v1s.Major == v2s.Major && v1s.Minor == v2s.Minor -} - // netNotifiee defines methods to be used with the IpfsDHT type netNotifiee IDService diff --git a/package.json b/package.json index e7e7d50177..18298b06d6 100644 --- a/package.json +++ b/package.json @@ -8,11 +8,6 @@ "goversion": "1.5.2" }, "gxDependencies": [ - { - "hash": "QmcrrEpx3VMUbrbgVroH3YiYyUS5c4YAykzyPJWKspUYLa", - "name": "go-semver", - "version": "0.0.0" - }, { "hash": "QmekaTKpWkYGcn4ZEC5PwJDRCQHapwugmmG86g2Xpz5GBH", "name": "mdns", From d59ca83d48b65c977e19fb16b23c27801dcf7d36 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 07:44:25 -0700 Subject: [PATCH 0778/3965] switch to DisablePing instead of PingCustom --- config/config.go | 5 ++--- defaults.go | 9 --------- options.go | 3 +-- 3 files changed, 3 insertions(+), 14 deletions(-) diff --git a/config/config.go b/config/config.go index 520cfb9e81..dd3d3996b7 100644 --- a/config/config.go +++ b/config/config.go @@ -57,8 +57,7 @@ type Config struct { Peerstore pstore.Peerstore Reporter metrics.Reporter - PingCustom bool - Ping bool + DisablePing bool } // NewNode constructs a new libp2p Host from the Config. @@ -105,7 +104,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { ConnManager: cfg.ConnManager, AddrsFactory: cfg.AddrsFactory, NATManager: cfg.NATManager, - EnablePing: cfg.Ping, + EnablePing: !cfg.DisablePing, }) if err != nil { swrm.Close() diff --git a/defaults.go b/defaults.go index f44f84d4a5..679fd7af5a 100644 --- a/defaults.go +++ b/defaults.go @@ -75,11 +75,6 @@ var DefaultEnableRelay = func(cfg *Config) error { return cfg.Apply(EnableRelay()) } -// DefaultEnablePing enables the ping service by default -var DefaultEnablePing = func(cfg *Config) error { - return cfg.Apply(Ping(true)) -} - // Complete list of default options and when to fallback on them. // // Please *DON'T* specify default options any other way. Putting this all here @@ -116,10 +111,6 @@ var defaults = []struct { fallback: func(cfg *Config) bool { return !cfg.RelayCustom }, opt: DefaultEnableRelay, }, - { - fallback: func(cfg *Config) bool { return !cfg.PingCustom }, - opt: DefaultEnablePing, - }, } // Defaults configures libp2p to use the default options. Can be combined with diff --git a/options.go b/options.go index 9c69744cf9..4ab4f39c73 100644 --- a/options.go +++ b/options.go @@ -255,8 +255,7 @@ func NATManager(nm config.NATManagerC) Option { // Ping will configure libp2p to support the ping service; enable by default. func Ping(enable bool) Option { return func(cfg *Config) error { - cfg.PingCustom = true - cfg.Ping = enable + cfg.DisablePing = !enable return nil } } From 63256192d7c5a1bbe00807afbed93d9c14ddcd0a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 07:45:45 -0700 Subject: [PATCH 0779/3965] fix buggy addr equal in test --- p2p/host/basic/basic_host_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index 27c81d552d..affdec7b3a 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -83,7 +83,7 @@ func TestHostAddrsFactory(t *testing.T) { if len(addrs) != 1 { t.Fatalf("expected 1 addr, got %d", len(addrs)) } - if addrs[0] != maddr { + if !addrs[0].Equal(maddr) { t.Fatalf("expected %s, got %s", maddr.String(), addrs[0].String()) } } From f046774e0abbbe85a58f8c43de4c869eac2c80c9 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 09:47:07 -0700 Subject: [PATCH 0780/3965] gx publish 6.0.20 --- .gx/lastpubver | 2 +- package.json | 102 ++++++++++++++++++++++++------------------------- 2 files changed, 52 insertions(+), 52 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index b9e31f7c95..b71eabdda7 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.19: QmPL3AKtiaQyYpchZceXBZhZ3MSnoGqJvLZrc7fzDTTQdJ +6.0.20: QmabWrc5aEQ36iWgJZonKgHpttvyDhHoWBoCtesuyMn9XF diff --git a/package.json b/package.json index 18298b06d6..19043d429d 100644 --- a/package.json +++ b/package.json @@ -39,50 +39,50 @@ "version": "1.5.7" }, { - "hash": "QmV6FjemM1K8oXjrvuq3wuVWWoU2TLDPmNnKrxHzY3v6Ai", + "hash": "Qmaabb1tJZ2CX5cp6MuuiGgns71NYoxdgQP6Xdid1dVceC", "name": "go-multiaddr-net", - "version": "1.6.3" + "version": "1.6.5" }, { - "hash": "QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7", + "hash": "QmT4U94DnD8FRfqr21obWY32HLM5VExccPKMjQHofeYqr9", "name": "go-multiaddr", - "version": "1.3.0" + "version": "1.3.5" }, { "author": "whyrusleeping", - "hash": "QmNLzS18jsmwTxXewTm3YnZVLftWCeegNZEBFjMrnvnBrH", + "hash": "QmVrDtvvQCUeMZaY9UFkae6c85kdQ1GvVEhPrjPTdjxRLv", "name": "go-libp2p-loggables", - "version": "1.1.22" + "version": "1.1.24" }, { "author": "whyrusleeping", - "hash": "QmZUBPE2W68FpZgUSmhVyjVD55P5Bm51wjnU8hTH14jHpk", + "hash": "QmXVoVfwEXZdG9zNNHE9in55SepeSJM5GSETEyWNSNMygc", "name": "go-libp2p-secio", - "version": "2.0.15" + "version": "2.0.17" }, { "author": "whyrusleeping", - "hash": "QmWtCpWB39Rzc2xTB75MKorsxNpo3TyecTEN24CJ3KVohE", + "hash": "QmTTJcDL3gsnGDALjh2fDGg1onGRUdVgNL2hU2WEZcVrMX", "name": "go-libp2p-peerstore", - "version": "2.0.4" + "version": "2.0.6" }, { "author": "whyrusleeping", - "hash": "QmQx3dbLZqokcGsMbaj2KYoQQ6PBZZGvqiTn8rRNyk7H51", + "hash": "QmbCkisBsdejwSzusQcdbYjpSX3yvUw1ek2YSsJ89QbZYX", "name": "go-libp2p-transport", - "version": "3.0.13" + "version": "3.0.15" }, { "author": "whyrusleeping", - "hash": "QmRJ7i85QeZoR1LJbbAjmTu2jDnrcNH5NqsXirbv6c9SX7", + "hash": "QmTkKN1x5Jvhc5Np55gJzD3PQ6GL74aKm9145t9WbvJyrB", "name": "go-tcp-transport", - "version": "2.0.14" + "version": "2.0.16" }, { "author": "whyrusleeping", - "hash": "QmSW4uNHbvQia8iZDXzbwjiyHQtnyo9aFqfQAMasj3TJ6Y", + "hash": "QmbuCmYjYK5GQo4zKrK2h3NVsyBYf81ZQXgiE69CLLGHgB", "name": "go-maddr-filter", - "version": "1.1.9" + "version": "1.1.10" }, { "author": "whyrusleeping", @@ -92,51 +92,51 @@ }, { "author": "whyrusleeping", - "hash": "QmNfQbgBfARAtrYsBguChX6VJ5nbjeoYy1KdC36aaYWqG8", + "hash": "Qma6ESRQTf1ZLPgzpCwDTqQJefPnU6uLvMjP18vK8EWp8L", "name": "go-testutil", - "version": "1.2.8" + "version": "1.2.10" }, { "author": "whyrusleeping", - "hash": "QmSTaEYUgDe1r581hxyd2u9582Hgp3KX4wGwYbRqz2u9Qh", + "hash": "QmXuRkCR7BNQa9uqfpTiFWsTQLzmTWYg91Ja1w95gnqb6u", "name": "go-libp2p-net", - "version": "3.0.13" + "version": "3.0.15" }, { "author": "whyrusleeping", - "hash": "QmNn6gcjBXpg8kccr9zEV7UVBpqAw8FZEiQ6DksvzyTQ5K", + "hash": "QmeaTjsfPf6vQ3WU2BUdjakgvKUHpuv3Fjxvb75N5iksMx", "name": "go-libp2p-metrics", - "version": "2.1.6" + "version": "2.1.7" }, { "author": "whyrusleeping", - "hash": "Qmf5yHzmWAyHSJRPAmZzfk3Yd7icydBLi7eec5741aov7v", + "hash": "QmdJfsSbKSZnMkfZ1kpopiyB9i3Hd6cp8VKWZmtWPa7Moc", "name": "go-libp2p-host", - "version": "3.0.13" + "version": "3.0.15" }, { "author": "whyrusleeping", - "hash": "QmYyFNDLTSy6rW99K8vHPvFPLa5bB4zoy8gGZENoiaBN6R", + "hash": "QmbJrDrS4zNqFEXAZuzJ1Jq6YZ1FS2PdVY1zmDFm9M8Lda", "name": "go-libp2p-swarm", - "version": "3.0.19" + "version": "3.0.21" }, { "author": "whyrusleeping", - "hash": "QmVicCVLxf62MkDBRXC27UTeqEvhNpBUouXNCqnLf4emJp", + "hash": "QmbLmhmtzD1rsrWC5Vjq6YHBmUzqh7LCR4m8W1e91BHQyg", "name": "go-libp2p-nat", - "version": "0.8.7" + "version": "0.8.8" }, { "author": "whyrusleeping", - "hash": "QmTqBqbGpNZnuPwW5AxUrooejubQuWnPpaatPozykZpXfi", + "hash": "QmdjF7Ubi5KNwUYMtKWyUsB6UY3B1wB5CDus9L3JKPWM2j", "name": "go-libp2p-netutil", - "version": "0.4.11" + "version": "0.4.12" }, { "author": "whyrusleeping", - "hash": "QmcZbC2qrMF6DAsHKzk7xgWBDiJHHc15jJ6aDNrv84DG6S", + "hash": "Qmcwep23T9AwwbuQRRhVyjk9PBRbMFCyGKCKGZfTPteiFU", "name": "go-libp2p-blankhost", - "version": "0.3.13" + "version": "0.3.15" }, { "author": "whyrusleeping", @@ -146,9 +146,9 @@ }, { "author": "whyrusleeping", - "hash": "QmVwYCtShoL74Xi8TgEg9jXvHVVtWwZ3Hg1QZqQvQA9xji", + "hash": "QmegfBRUMFMaHgwSzxTUDdgmtMLt8YAJBgWub8VvAFaX8V", "name": "go-smux-yamux", - "version": "2.0.6" + "version": "2.0.7" }, { "author": "whyrusleeping", @@ -158,27 +158,27 @@ }, { "author": "whyrusleeping", - "hash": "QmbNepETomvmXfz1X5pHNFD2QuPqnqi47dTd94QJWSorQ3", + "hash": "QmTRhk7cgjUf2gfQ3p2M9KPECNZEW9XUrmHcFCgog4cPgB", "name": "go-libp2p-peer", - "version": "2.3.8" + "version": "2.4.0" }, { "author": "vyzo", - "hash": "Qmbqn6qCocM5Dk9idyUw6wU1vsZCd1aZ3yJpwPA27YpE6J", + "hash": "QmWiBdRSS7f3aYVWePSrQsAk5nqBVswgV3eKvJMEH6NhHc", "name": "go-libp2p-circuit", - "version": "2.3.0" + "version": "2.3.1" }, { "author": "lgierth", - "hash": "QmfXU2MhWoegxHoeMd3A2ytL2P6CY4FfqGWc23LTNWBwZt", + "hash": "QmeHJXPqCNzSFbVkYM1uQLuM2L5FyJB9zukQ7EeqRP8ZC9", "name": "go-multiaddr-dns", - "version": "0.2.4" + "version": "0.2.5" }, { "author": "why", - "hash": "QmYfDsxmFGuMSdvFFgw5Ss9pD4YvTFWSFFXYT7LjtwKnY7", + "hash": "QmWRvjn5BHMLCGkf48Hk1LDc4W72RPA9H59AAVCXmn9esJ", "name": "go-libp2p-interface-connmgr", - "version": "0.0.19" + "version": "0.0.21" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmefRFPqDrW6xzNqfct7gWXSDQQV1PQEK2nhTxhN9YATJF", + "hash": "QmY957dCFYVPKpj21xRs6KA3XAGA9tBt73UE5kfUGdNgD9", "name": "go-ws-transport", - "version": "2.0.14" + "version": "2.0.15" }, { "author": "stebalien", - "hash": "QmQYTdbqdgLDikde7dHKh4svNG925EDYA9ZKZHwqidX277", + "hash": "QmVCtDMyervzvQTi3yZrfTLXtXSuUYQaowCG9hJdHu6gAM", "name": "go-conn-security-multistream", - "version": "0.1.13" + "version": "0.1.15" }, { "author": "Stebalien", - "hash": "QmSjwssnWNJvzwD77yWruH2ebqwMNR6PCn49hbDURYH9d8", + "hash": "QmZ3XKH272gU9px86XqWYeZHU65ayHxWs6Wbswvdj2VqVK", "name": "go-conn-security", - "version": "0.1.13" + "version": "0.1.15" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "QmPM1NCCnMMY3GaPVkG2sTVh8Xys1gCPsk1X8Z1HuUJePs", + "hash": "QmeUjhpfGkrMtE6s7JjU2xLhfzprSDh16QUxGVr3wTrKSx", "name": "go-libp2p-transport-upgrader", - "version": "0.1.14" + "version": "0.1.16" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -233,6 +233,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.19" + "version": "6.0.20" } From 841a6272c661fae506cd656ea884655f3c313743 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 12:41:12 -0700 Subject: [PATCH 0781/3965] fix a data-race in the mock net This was causing https://github.com/ipfs/go-ipfs/pull/5637#issuecomment-432792969 --- p2p/net/mock/mock_stream.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index 4f0c395fec..c680ead26d 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -56,6 +56,11 @@ func (s *stream) Write(p []byte) (n int, err error) { l := s.conn.link delay := l.GetLatency() + l.RateLimit(len(p)) t := time.Now().Add(delay) + + // Copy it. + cpy := make([]byte, len(p)) + copy(cpy, p) + select { case <-s.closed: // bail out if we're closing. return 0, s.writeErr From 8af9bc6228094f9725939a54f57fc6bb2224324d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 12:45:56 -0700 Subject: [PATCH 0782/3965] gx publish 6.0.21 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index b71eabdda7..72b9227fdb 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.20: QmabWrc5aEQ36iWgJZonKgHpttvyDhHoWBoCtesuyMn9XF +6.0.21: QmQLCaTaF2gJvpFwLYJUYVzQJxJvCGfjutBBDb65dssFVu diff --git a/package.json b/package.json index 19043d429d..c504d14ca2 100644 --- a/package.json +++ b/package.json @@ -233,6 +233,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.20" + "version": "6.0.21" } From 80c5998236da06d55c072957d9de5f2d37d35c7c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 12:58:51 -0700 Subject: [PATCH 0783/3965] *really* fix the race in mock --- p2p/net/mock/mock_stream.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index c680ead26d..8d01505049 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -64,7 +64,7 @@ func (s *stream) Write(p []byte) (n int, err error) { select { case <-s.closed: // bail out if we're closing. return 0, s.writeErr - case s.toDeliver <- &transportObject{msg: p, arrivalTime: t}: + case s.toDeliver <- &transportObject{msg: cpy, arrivalTime: t}: } return len(p), nil } From f6dce9ab90ce66f3505915eeaa5ceaeb80c8a833 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 13:00:46 -0700 Subject: [PATCH 0784/3965] gx publish 6.0.22 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 72b9227fdb..cd9084c2a7 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.21: QmQLCaTaF2gJvpFwLYJUYVzQJxJvCGfjutBBDb65dssFVu +6.0.22: QmRqtXHu5gsCLpf2s1R2jQuKJBowYKkg6FGQiGCbzttSd1 diff --git a/package.json b/package.json index c504d14ca2..3f9f7b0b19 100644 --- a/package.json +++ b/package.json @@ -233,6 +233,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.21" + "version": "6.0.22" } From 9356373d00ab1aef3e20c8202b682f93799acf78 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Oct 2018 14:56:21 -0700 Subject: [PATCH 0785/3965] gx publish 6.0.23 --- .gx/lastpubver | 2 +- package.json | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index cd9084c2a7..2ccffef679 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.22: QmRqtXHu5gsCLpf2s1R2jQuKJBowYKkg6FGQiGCbzttSd1 +6.0.23: QmUDTcnDp2WssbmiDLC6aYurUeyt7QeRakHUQMxA2mZ5iB diff --git a/package.json b/package.json index 3f9f7b0b19..e27a8272ad 100644 --- a/package.json +++ b/package.json @@ -116,9 +116,9 @@ }, { "author": "whyrusleeping", - "hash": "QmbJrDrS4zNqFEXAZuzJ1Jq6YZ1FS2PdVY1zmDFm9M8Lda", + "hash": "QmVHhT8NxtApPTndiZPe4JNGNUxGWtJe3ebyxtRz4HnbEp", "name": "go-libp2p-swarm", - "version": "3.0.21" + "version": "3.0.22" }, { "author": "whyrusleeping", @@ -146,9 +146,9 @@ }, { "author": "whyrusleeping", - "hash": "QmegfBRUMFMaHgwSzxTUDdgmtMLt8YAJBgWub8VvAFaX8V", + "hash": "Qmdps3CYh5htGQSrPvzg5PHouVexLmtpbuLCqc4vuej8PC", "name": "go-smux-yamux", - "version": "2.0.7" + "version": "2.0.8" }, { "author": "whyrusleeping", @@ -164,9 +164,9 @@ }, { "author": "vyzo", - "hash": "QmWiBdRSS7f3aYVWePSrQsAk5nqBVswgV3eKvJMEH6NhHc", + "hash": "QmVYDvJjiKb9iFEyHxx4i1TJSRBLkQhGb5Fc8XpmDuNCEA", "name": "go-libp2p-circuit", - "version": "2.3.1" + "version": "2.3.2" }, { "author": "lgierth", @@ -233,6 +233,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.22" + "version": "6.0.23" } From c98eb14a9004ed80fc1b86f10230a99a6dc7c037 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 25 Oct 2018 11:56:27 +0300 Subject: [PATCH 0786/3965] update test for AutoNATClient constructor change --- p2p/host/autonat/svc_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 9c8d969188..c0a676dece 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -33,7 +33,7 @@ func makeAutoNATClient(ctx context.Context, t *testing.T) (host.Host, autonat.Au t.Fatal(err) } - cli := autonat.NewAutoNATClient(h) + cli := autonat.NewAutoNATClient(h, nil) return h, cli } From e8b9e0d5e344d85ae438103834826b95fb6f6532 Mon Sep 17 00:00:00 2001 From: Can ZHANG Date: Tue, 30 Oct 2018 11:24:46 +0800 Subject: [PATCH 0787/3965] Deprecate IPFS_REUSEPORT, use LIBP2P_TCP_REUSEPORT --- p2p/transport/tcp/reuseport.go | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/p2p/transport/tcp/reuseport.go b/p2p/transport/tcp/reuseport.go index 58105ce537..04a5da3ed1 100644 --- a/p2p/transport/tcp/reuseport.go +++ b/p2p/transport/tcp/reuseport.go @@ -9,7 +9,8 @@ import ( // envReuseport is the env variable name used to turn off reuse port. // It default to true. -const envReuseport = "IPFS_REUSEPORT" +const envReuseport = "LIBP2P_TCP_REUSEPORT" +const deprecatedEnvReuseport = "IPFS_REUSEPORT" // envReuseportVal stores the value of envReuseport. defaults to true. var envReuseportVal = true @@ -18,7 +19,15 @@ func init() { v := strings.ToLower(os.Getenv(envReuseport)) if v == "false" || v == "f" || v == "0" { envReuseportVal = false - log.Infof("REUSEPORT disabled (IPFS_REUSEPORT=%s)", v) + log.Infof("REUSEPORT disabled (LIBP2P_TCP_REUSEPORT=%s)", v) + } + v, exist := os.LookupEnv(deprecatedEnvReuseport) + if exist { + log.Warning("IPFS_REUSEPORT is deprecated, use LIBP2P_TCP_REUSEPORT instead") + if v == "false" || v == "f" || v == "0" { + envReuseportVal = false + log.Infof("REUSEPORT disabled (IPFS_REUSEPORT=%s)", v) + } } } @@ -26,7 +35,7 @@ func init() { // is here because we want to be able to turn reuseport on and off selectively. // For now we use an ENV variable, as this handles our pressing need: // -// IPFS_REUSEPORT=false ipfs daemon +// LIBP2P_TCP_REUSEPORT=false ipfs daemon // // If this becomes a sought after feature, we could add this to the config. // In the end, reuseport is a stop-gap. From 5f4a118ea3ef90b99418176d2ab8b493f0402c6d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 2 Nov 2018 15:24:44 -0700 Subject: [PATCH 0788/3965] fix the inline key test RandTestKeyPair uses RSA keys but we now forbid any RSA key under 512 bits. --- p2p/host/peerstore/test/keybook_suite.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/test/keybook_suite.go b/p2p/host/peerstore/test/keybook_suite.go index 9379d9cd99..e713f0eb10 100644 --- a/p2p/host/peerstore/test/keybook_suite.go +++ b/p2p/host/peerstore/test/keybook_suite.go @@ -4,6 +4,7 @@ import ( "sort" "testing" + ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" pt "github.com/libp2p/go-libp2p-peer/test" @@ -144,7 +145,7 @@ func testInlinedPubKeyAddedOnRetrieve(kb pstore.KeyBook) func(t *testing.T) { } // Key small enough for inlining. - _, pub, err := pt.RandTestKeyPair(32) + _, pub, err := ic.GenerateKeyPair(ic.Ed25519, 256) if err != nil { t.Error(err) } From 5904fabec1e566ba7f8f859de1678b855a98389e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 2 Nov 2018 16:51:20 -0700 Subject: [PATCH 0789/3965] gx publish 6.0.24 --- .gx/lastpubver | 2 +- package.json | 114 ++++++++++++++++++++++++------------------------- 2 files changed, 58 insertions(+), 58 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 2ccffef679..147cdb8032 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.23: QmUDTcnDp2WssbmiDLC6aYurUeyt7QeRakHUQMxA2mZ5iB +6.0.24: QmW1qc6tENFpuvv3VfjSHRz1GiVYGiQa13nnEMJBaJDBqW diff --git a/package.json b/package.json index e27a8272ad..92f978c11c 100644 --- a/package.json +++ b/package.json @@ -14,9 +14,9 @@ "version": "0.2.0" }, { - "hash": "QmPdKqUcHGFdeSpvjVoaTRPPstGif9GBZb5Q56RVw9o69A", + "hash": "QmNohiVssaPw3KVLZik59DBVGTSm2dGvYT9eoXt5DQ36Yz", "name": "go-ipfs-util", - "version": "1.2.8" + "version": "1.2.9" }, { "hash": "QmabLh8TrJ3emfAoQk5AbqbLTbMyj7XqumMFmAFxa9epo8", @@ -34,55 +34,55 @@ "version": "1.0.0" }, { - "hash": "QmZChCsSt8DctjceaL56Eibc29CVQq4dGKRXC5JRZ6Ppae", + "hash": "QmcuXC5cxs79ro2cUuHs4HQ2bkDLJUYokwL8aivcX6HW3C", "name": "go-log", - "version": "1.5.7" + "version": "1.5.8" }, { - "hash": "Qmaabb1tJZ2CX5cp6MuuiGgns71NYoxdgQP6Xdid1dVceC", + "hash": "QmQVUtnrNGtCRkCMpXgpApfzQjc8FDaDVxHqWH8cnZQeh5", "name": "go-multiaddr-net", - "version": "1.6.5" + "version": "1.6.6" }, { - "hash": "QmT4U94DnD8FRfqr21obWY32HLM5VExccPKMjQHofeYqr9", + "hash": "QmRKLtwMw131aK7ugC3G7ybpumMz78YrJe5dzneyindvG1", "name": "go-multiaddr", - "version": "1.3.5" + "version": "1.3.6" }, { "author": "whyrusleeping", - "hash": "QmVrDtvvQCUeMZaY9UFkae6c85kdQ1GvVEhPrjPTdjxRLv", + "hash": "QmZdC12tc1dboSPXr6B5R7gQzuZtUg1Poa5Ly7GG5kwxdb", "name": "go-libp2p-loggables", - "version": "1.1.24" + "version": "1.1.25" }, { "author": "whyrusleeping", - "hash": "QmXVoVfwEXZdG9zNNHE9in55SepeSJM5GSETEyWNSNMygc", + "hash": "QmUGYhX1updeqaJJ2jDtUSch7N9x7xrVtBEh8RaaUEGB1a", "name": "go-libp2p-secio", - "version": "2.0.17" + "version": "2.0.19" }, { "author": "whyrusleeping", - "hash": "QmTTJcDL3gsnGDALjh2fDGg1onGRUdVgNL2hU2WEZcVrMX", + "hash": "QmNVzTd29YipdnuXB67b2VnFSerudTVS3ZrxyBhNxcdvAg", "name": "go-libp2p-peerstore", - "version": "2.0.6" + "version": "2.0.8" }, { "author": "whyrusleeping", - "hash": "QmbCkisBsdejwSzusQcdbYjpSX3yvUw1ek2YSsJ89QbZYX", + "hash": "QmTehjKGgeUrmvGVHxwoQYMWdLJJBpDUskbLz6o89guFN5", "name": "go-libp2p-transport", - "version": "3.0.15" + "version": "3.0.16" }, { "author": "whyrusleeping", - "hash": "QmTkKN1x5Jvhc5Np55gJzD3PQ6GL74aKm9145t9WbvJyrB", + "hash": "QmdWRMjmRNba29As5nPN6GHJYdBLUX52vi5sX6NW3K8Wb4", "name": "go-tcp-transport", - "version": "2.0.16" + "version": "2.0.17" }, { "author": "whyrusleeping", - "hash": "QmbuCmYjYK5GQo4zKrK2h3NVsyBYf81ZQXgiE69CLLGHgB", + "hash": "QmQJRvWaYAvU3Mdtk33ADXr9JAZwKMBYBGPkRQBDvyj2nn", "name": "go-maddr-filter", - "version": "1.1.10" + "version": "1.1.11" }, { "author": "whyrusleeping", @@ -92,57 +92,57 @@ }, { "author": "whyrusleeping", - "hash": "Qma6ESRQTf1ZLPgzpCwDTqQJefPnU6uLvMjP18vK8EWp8L", + "hash": "QmbtxgRVMQS2fZpuwowZRQUg3QNri9yuHSFGfXUrqoSABU", "name": "go-testutil", - "version": "1.2.10" + "version": "1.2.11" }, { "author": "whyrusleeping", - "hash": "QmXuRkCR7BNQa9uqfpTiFWsTQLzmTWYg91Ja1w95gnqb6u", + "hash": "QmX1VAqodkn8yFJe7gYFs8PvrSCVsCWcTUBgxswoeC3J4K", "name": "go-libp2p-net", - "version": "3.0.15" + "version": "3.0.17" }, { "author": "whyrusleeping", - "hash": "QmeaTjsfPf6vQ3WU2BUdjakgvKUHpuv3Fjxvb75N5iksMx", + "hash": "QmVoacHjdm3VRFp7W9gC6qftKgVFeBXarcJT518kKCWNqf", "name": "go-libp2p-metrics", - "version": "2.1.7" + "version": "2.1.8" }, { "author": "whyrusleeping", - "hash": "QmdJfsSbKSZnMkfZ1kpopiyB9i3Hd6cp8VKWZmtWPa7Moc", + "hash": "QmUR3nQ8nSP7vUnzTvz2kLxq4ncud6eC2s4BAEC8uiAsde", "name": "go-libp2p-host", - "version": "3.0.15" + "version": "3.0.16" }, { "author": "whyrusleeping", - "hash": "QmVHhT8NxtApPTndiZPe4JNGNUxGWtJe3ebyxtRz4HnbEp", + "hash": "QmdEdguaWGhjhQXUBvSLjSAT3idU1ECwASUo95ZW1obEEp", "name": "go-libp2p-swarm", - "version": "3.0.22" + "version": "3.0.23" }, { "author": "whyrusleeping", - "hash": "QmbLmhmtzD1rsrWC5Vjq6YHBmUzqh7LCR4m8W1e91BHQyg", + "hash": "QmYVUZr4KxAD7ginKydBtJpVzy2nEe7vXyKvJYjneAX273", "name": "go-libp2p-nat", - "version": "0.8.8" + "version": "0.8.9" }, { "author": "whyrusleeping", - "hash": "QmdjF7Ubi5KNwUYMtKWyUsB6UY3B1wB5CDus9L3JKPWM2j", + "hash": "QmSpjWUgMC2zWuuJvoBHfwhofmBLuKNNbimJbakboBUkHP", "name": "go-libp2p-netutil", - "version": "0.4.12" + "version": "0.4.13" }, { "author": "whyrusleeping", - "hash": "Qmcwep23T9AwwbuQRRhVyjk9PBRbMFCyGKCKGZfTPteiFU", + "hash": "QmNipGra5iGyJYw85wYwLf2RpvsUu1q93XESxCPWYpMmrF", "name": "go-libp2p-blankhost", - "version": "0.3.15" + "version": "0.3.16" }, { "author": "whyrusleeping", - "hash": "QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n", + "hash": "QmaZ1F8vQdJaQqA8hjAzH8a7Qnqqb2n2ZYeUbKMjMcxAZe", "name": "go-libp2p-crypto", - "version": "2.0.1" + "version": "2.0.2" }, { "author": "whyrusleeping", @@ -158,51 +158,51 @@ }, { "author": "whyrusleeping", - "hash": "QmTRhk7cgjUf2gfQ3p2M9KPECNZEW9XUrmHcFCgog4cPgB", + "hash": "QmUz7HmU4ws577TMEVNoTjNghVvq2jXnxnVhbkNXKPKpgK", "name": "go-libp2p-peer", - "version": "2.4.0" + "version": "2.4.1" }, { "author": "vyzo", - "hash": "QmVYDvJjiKb9iFEyHxx4i1TJSRBLkQhGb5Fc8XpmDuNCEA", + "hash": "QmTQttQVZNPrremo44RwPyCBQ5e2KKZPNgBQ5CuNxLZT8C", "name": "go-libp2p-circuit", - "version": "2.3.2" + "version": "2.3.3" }, { "author": "lgierth", - "hash": "QmeHJXPqCNzSFbVkYM1uQLuM2L5FyJB9zukQ7EeqRP8ZC9", + "hash": "QmT4zgnKCyZBpRyxzsvZqUjzUkMWLJ2pZCw7uk6M6Kto5m", "name": "go-multiaddr-dns", - "version": "0.2.5" + "version": "0.2.6" }, { "author": "why", - "hash": "QmWRvjn5BHMLCGkf48Hk1LDc4W72RPA9H59AAVCXmn9esJ", + "hash": "QmeiGg65GR8ifGTEiM5qoxu4PqiuwUQPkWjXEkuCqMJBKS", "name": "go-libp2p-interface-connmgr", - "version": "0.0.21" + "version": "0.0.22" }, { "author": "whyrusleeping", - "hash": "QmaveCPGVaKJU57tBErGCDjzLaqEMZkFygoiv4BhYwWUGc", + "hash": "QmZsejKNkeFSQe5TcmYXJ8iq6qPL1FpsP4eAA8j7RfE7xg", "name": "go-smux-multiplex", - "version": "3.0.16" + "version": "3.0.17" }, { "author": "whyrusleeping", - "hash": "QmY957dCFYVPKpj21xRs6KA3XAGA9tBt73UE5kfUGdNgD9", + "hash": "QmbwKd99wgauoCPN7zJLompWFkBZkqBruv8CwBkZmgtZQa", "name": "go-ws-transport", - "version": "2.0.15" + "version": "2.0.16" }, { "author": "stebalien", - "hash": "QmVCtDMyervzvQTi3yZrfTLXtXSuUYQaowCG9hJdHu6gAM", + "hash": "QmeL7iuPLMb6tcdgs2mKH8kkJAQUf5U91WTQgbw5TZDt2Y", "name": "go-conn-security-multistream", - "version": "0.1.15" + "version": "0.1.16" }, { "author": "Stebalien", - "hash": "QmZ3XKH272gU9px86XqWYeZHU65ayHxWs6Wbswvdj2VqVK", + "hash": "QmV5458E9oWTDkfw3SAGQY3GWsg9zmMvykFHkcdc6pYS7r", "name": "go-conn-security", - "version": "0.1.15" + "version": "0.1.17" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "QmeUjhpfGkrMtE6s7JjU2xLhfzprSDh16QUxGVr3wTrKSx", + "hash": "Qmbp39ii5NyHzkeXGZj1THb5MNKFrFTigJ5wHHmd3ukCfe", "name": "go-libp2p-transport-upgrader", - "version": "0.1.16" + "version": "0.1.17" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -233,6 +233,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.23" + "version": "6.0.24" } From 43636e4e28fed8fe4d5f0ea13e0c691b53e9fe95 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 2 Nov 2018 18:06:44 -0700 Subject: [PATCH 0790/3965] gx publish 6.0.25 --- .gx/lastpubver | 2 +- package.json | 82 +++++++++++++++++++++++++------------------------- 2 files changed, 42 insertions(+), 42 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 147cdb8032..d111583579 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.24: QmW1qc6tENFpuvv3VfjSHRz1GiVYGiQa13nnEMJBaJDBqW +6.0.25: QmRKGtF5iRsA1kNgagiKZGo2VeJxizF2tXPD3sysGX2vcc diff --git a/package.json b/package.json index 92f978c11c..68eb38c7bc 100644 --- a/package.json +++ b/package.json @@ -50,33 +50,33 @@ }, { "author": "whyrusleeping", - "hash": "QmZdC12tc1dboSPXr6B5R7gQzuZtUg1Poa5Ly7GG5kwxdb", + "hash": "QmaLkq87EbugRDQBRMJhuXv29LSCRE4D3AC9t1UAV3kyj1", "name": "go-libp2p-loggables", - "version": "1.1.25" + "version": "1.1.26" }, { "author": "whyrusleeping", - "hash": "QmUGYhX1updeqaJJ2jDtUSch7N9x7xrVtBEh8RaaUEGB1a", + "hash": "Qmb4TZphDDGhE5kegzhjA3C7iTURzKGVGQmaMi6dXCtG4a", "name": "go-libp2p-secio", - "version": "2.0.19" + "version": "2.0.20" }, { "author": "whyrusleeping", - "hash": "QmNVzTd29YipdnuXB67b2VnFSerudTVS3ZrxyBhNxcdvAg", + "hash": "QmYrNDh2z1sFB7gxYU6kX2f9N8LcWJkAzAKwZqUzvf6d1P", "name": "go-libp2p-peerstore", - "version": "2.0.8" + "version": "2.0.9" }, { "author": "whyrusleeping", - "hash": "QmTehjKGgeUrmvGVHxwoQYMWdLJJBpDUskbLz6o89guFN5", + "hash": "QmdwyRENdzpNQQhDi8YqEovKipGb4DtNcWPXcj1RoeSzbC", "name": "go-libp2p-transport", - "version": "3.0.16" + "version": "3.0.17" }, { "author": "whyrusleeping", - "hash": "QmdWRMjmRNba29As5nPN6GHJYdBLUX52vi5sX6NW3K8Wb4", + "hash": "QmXB5U1hRyF2eeT9Wqj7KMSgheidpTRkJ9MdEBJQiPwBb4", "name": "go-tcp-transport", - "version": "2.0.17" + "version": "2.0.18" }, { "author": "whyrusleeping", @@ -92,33 +92,33 @@ }, { "author": "whyrusleeping", - "hash": "QmbtxgRVMQS2fZpuwowZRQUg3QNri9yuHSFGfXUrqoSABU", + "hash": "Qmcxbk5F1rJ9KeJaKRoZNG7qrexftEM7Rz9y15hi5FHAuk", "name": "go-testutil", - "version": "1.2.11" + "version": "1.2.12" }, { "author": "whyrusleeping", - "hash": "QmX1VAqodkn8yFJe7gYFs8PvrSCVsCWcTUBgxswoeC3J4K", + "hash": "QmXmkWCiVumBA1MMfPHHoUaCoVo7H5uceEXjzYCVkwVxN9", "name": "go-libp2p-net", - "version": "3.0.17" + "version": "3.0.18" }, { "author": "whyrusleeping", - "hash": "QmVoacHjdm3VRFp7W9gC6qftKgVFeBXarcJT518kKCWNqf", + "hash": "QmVnxHtMVtYC3rP6vpZnWAVYxMsYVW5enEFbp8jD13vu45", "name": "go-libp2p-metrics", - "version": "2.1.8" + "version": "2.1.9" }, { "author": "whyrusleeping", - "hash": "QmUR3nQ8nSP7vUnzTvz2kLxq4ncud6eC2s4BAEC8uiAsde", + "hash": "QmYWxr8cuXMeisJ69SK5HCg3S7FVuARCi8JZWxDyb9Aatw", "name": "go-libp2p-host", - "version": "3.0.16" + "version": "3.0.17" }, { "author": "whyrusleeping", - "hash": "QmdEdguaWGhjhQXUBvSLjSAT3idU1ECwASUo95ZW1obEEp", + "hash": "QmXncr9XPENrEdKSVtSht8RafevVfFwvPCPRVoYhrZmMi8", "name": "go-libp2p-swarm", - "version": "3.0.23" + "version": "3.0.24" }, { "author": "whyrusleeping", @@ -128,21 +128,21 @@ }, { "author": "whyrusleeping", - "hash": "QmSpjWUgMC2zWuuJvoBHfwhofmBLuKNNbimJbakboBUkHP", + "hash": "QmUmuPKRdXA5PJVFUBZPMj1U87t4zmZGganw46xyfXm3gQ", "name": "go-libp2p-netutil", - "version": "0.4.13" + "version": "0.4.14" }, { "author": "whyrusleeping", - "hash": "QmNipGra5iGyJYw85wYwLf2RpvsUu1q93XESxCPWYpMmrF", + "hash": "QmVXYnW9fUXpJwpUyyN1uTSoAWUUHKreKmBaieqk6Q3Jjs", "name": "go-libp2p-blankhost", - "version": "0.3.16" + "version": "0.3.17" }, { "author": "whyrusleeping", - "hash": "QmaZ1F8vQdJaQqA8hjAzH8a7Qnqqb2n2ZYeUbKMjMcxAZe", + "hash": "QmcgyFuP2SVMhgkTjFaXwnyAi9Fheg6A7SWgpYJKuajPnK", "name": "go-libp2p-crypto", - "version": "2.0.2" + "version": "2.0.3" }, { "author": "whyrusleeping", @@ -158,15 +158,15 @@ }, { "author": "whyrusleeping", - "hash": "QmUz7HmU4ws577TMEVNoTjNghVvq2jXnxnVhbkNXKPKpgK", + "hash": "Qme2jJHLnAfULnkrSxmqAn7pGTp3HWVmFYLnuLtxFvH9nA", "name": "go-libp2p-peer", - "version": "2.4.1" + "version": "2.4.2" }, { "author": "vyzo", - "hash": "QmTQttQVZNPrremo44RwPyCBQ5e2KKZPNgBQ5CuNxLZT8C", + "hash": "QmW1EkbsbRj5ZmeuvdMJqoLmmobUmLQjG4e8KgrdvABwB6", "name": "go-libp2p-circuit", - "version": "2.3.3" + "version": "2.3.4" }, { "author": "lgierth", @@ -176,9 +176,9 @@ }, { "author": "why", - "hash": "QmeiGg65GR8ifGTEiM5qoxu4PqiuwUQPkWjXEkuCqMJBKS", + "hash": "Qmdv8PZ3Hdk3VxAZmTeuz8rmNGcjdGEqDRt7jJvz3hyTUH", "name": "go-libp2p-interface-connmgr", - "version": "0.0.22" + "version": "0.0.23" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmbwKd99wgauoCPN7zJLompWFkBZkqBruv8CwBkZmgtZQa", + "hash": "QmSU7jRUFH1hHA3Xqy7cypFZLEKzTTnE5sX7xsF4FYGZvD", "name": "go-ws-transport", - "version": "2.0.16" + "version": "2.0.17" }, { "author": "stebalien", - "hash": "QmeL7iuPLMb6tcdgs2mKH8kkJAQUf5U91WTQgbw5TZDt2Y", + "hash": "QmQDzMXb6BFiheZj7c6Dysa43DZKBJrdPPqJ7TFrzrZYAf", "name": "go-conn-security-multistream", - "version": "0.1.16" + "version": "0.1.17" }, { "author": "Stebalien", - "hash": "QmV5458E9oWTDkfw3SAGQY3GWsg9zmMvykFHkcdc6pYS7r", + "hash": "QmYreGEecvzC8WzrrQjAWVzj234Qi9XDC8Wj6dRdfpg5zT", "name": "go-conn-security", - "version": "0.1.17" + "version": "0.1.18" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "Qmbp39ii5NyHzkeXGZj1THb5MNKFrFTigJ5wHHmd3ukCfe", + "hash": "Qmcb6HnPAaqHznw45vW11G1vHCkbjZFHLKiifnrZafZW3C", "name": "go-libp2p-transport-upgrader", - "version": "0.1.17" + "version": "0.1.18" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -233,6 +233,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.24" + "version": "6.0.25" } From 2b6c3a228e6779af6161e78f57ac24e5992f717c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 2 Nov 2018 21:11:37 -0700 Subject: [PATCH 0791/3965] gx publish 6.0.26 --- .gx/lastpubver | 2 +- package.json | 82 +++++++++++++++++++++++++------------------------- 2 files changed, 42 insertions(+), 42 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index d111583579..30f352f3c9 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.25: QmRKGtF5iRsA1kNgagiKZGo2VeJxizF2tXPD3sysGX2vcc +6.0.26: QmXnpYYg2onGLXVxM4Q5PEFcx29k8zeJQkPeLAk9h9naxg diff --git a/package.json b/package.json index 68eb38c7bc..c83cc672a5 100644 --- a/package.json +++ b/package.json @@ -50,33 +50,33 @@ }, { "author": "whyrusleeping", - "hash": "QmaLkq87EbugRDQBRMJhuXv29LSCRE4D3AC9t1UAV3kyj1", + "hash": "QmbP5E6C5Ks2b43vTZoyjSUHx2PPgoe78PZpJSJqCJ7LYx", "name": "go-libp2p-loggables", - "version": "1.1.26" + "version": "1.1.27" }, { "author": "whyrusleeping", - "hash": "Qmb4TZphDDGhE5kegzhjA3C7iTURzKGVGQmaMi6dXCtG4a", + "hash": "QmcDSLssuPYMwC71biMiCvWMGg6rBzk8TKf4UJN64M68YK", "name": "go-libp2p-secio", - "version": "2.0.20" + "version": "2.0.21" }, { "author": "whyrusleeping", - "hash": "QmYrNDh2z1sFB7gxYU6kX2f9N8LcWJkAzAKwZqUzvf6d1P", + "hash": "QmUymf8fJtideyv3z727BcZUifGBjMZMpCJqu3Gxk5aRUk", "name": "go-libp2p-peerstore", - "version": "2.0.9" + "version": "2.0.10" }, { "author": "whyrusleeping", - "hash": "QmdwyRENdzpNQQhDi8YqEovKipGb4DtNcWPXcj1RoeSzbC", + "hash": "QmZJ5hXLAz8vrZ4cw4EFk355pqMuxWTZQ5Hs2xhYGjdvGr", "name": "go-libp2p-transport", - "version": "3.0.17" + "version": "3.0.18" }, { "author": "whyrusleeping", - "hash": "QmXB5U1hRyF2eeT9Wqj7KMSgheidpTRkJ9MdEBJQiPwBb4", + "hash": "QmetLkVrNTi6QGfv577kpKPVBXJFshKZrP87C7xHJp7SCX", "name": "go-tcp-transport", - "version": "2.0.18" + "version": "2.0.19" }, { "author": "whyrusleeping", @@ -92,33 +92,33 @@ }, { "author": "whyrusleeping", - "hash": "Qmcxbk5F1rJ9KeJaKRoZNG7qrexftEM7Rz9y15hi5FHAuk", + "hash": "QmZXjR5X1p4KrQ967cTsy4MymMzUM8mZECF3PV8UcN4o3g", "name": "go-testutil", - "version": "1.2.12" + "version": "1.2.13" }, { "author": "whyrusleeping", - "hash": "QmXmkWCiVumBA1MMfPHHoUaCoVo7H5uceEXjzYCVkwVxN9", + "hash": "QmRKbEchaYADxSCyyjhDh4cTrUby8ftXUb8MRLBTHQYupw", "name": "go-libp2p-net", - "version": "3.0.18" + "version": "3.0.19" }, { "author": "whyrusleeping", - "hash": "QmVnxHtMVtYC3rP6vpZnWAVYxMsYVW5enEFbp8jD13vu45", + "hash": "QmfBAmuDFoPTMC232UQenPDYAzHQ48crKaXG9AfQqFuRpN", "name": "go-libp2p-metrics", - "version": "2.1.9" + "version": "2.1.10" }, { "author": "whyrusleeping", - "hash": "QmYWxr8cuXMeisJ69SK5HCg3S7FVuARCi8JZWxDyb9Aatw", + "hash": "QmVrjR2KMe57y4YyfHdYa3yKD278gN8W7CTiqSuYmxjA7F", "name": "go-libp2p-host", - "version": "3.0.17" + "version": "3.0.18" }, { "author": "whyrusleeping", - "hash": "QmXncr9XPENrEdKSVtSht8RafevVfFwvPCPRVoYhrZmMi8", + "hash": "QmcYC4ayKi7bq8xecEZxHVEuTL6HREZWTTErrSRd1S3Spz", "name": "go-libp2p-swarm", - "version": "3.0.24" + "version": "3.0.25" }, { "author": "whyrusleeping", @@ -128,21 +128,21 @@ }, { "author": "whyrusleeping", - "hash": "QmUmuPKRdXA5PJVFUBZPMj1U87t4zmZGganw46xyfXm3gQ", + "hash": "QmVPQuFZTw2yVm5Uu1MGofaNBnqaySQddLHKCFjWyoKGhv", "name": "go-libp2p-netutil", - "version": "0.4.14" + "version": "0.4.15" }, { "author": "whyrusleeping", - "hash": "QmVXYnW9fUXpJwpUyyN1uTSoAWUUHKreKmBaieqk6Q3Jjs", + "hash": "QmVzYgadebqVb6iCVvjGfgZ7HB9zoCnDgBjMqFLD5aMrFX", "name": "go-libp2p-blankhost", - "version": "0.3.17" + "version": "0.3.18" }, { "author": "whyrusleeping", - "hash": "QmcgyFuP2SVMhgkTjFaXwnyAi9Fheg6A7SWgpYJKuajPnK", + "hash": "QmNiJiXwWE3kRhZrC5ej3kSjWHm337pYfhjLGSCDNKJP2s", "name": "go-libp2p-crypto", - "version": "2.0.3" + "version": "2.0.4" }, { "author": "whyrusleeping", @@ -158,15 +158,15 @@ }, { "author": "whyrusleeping", - "hash": "Qme2jJHLnAfULnkrSxmqAn7pGTp3HWVmFYLnuLtxFvH9nA", + "hash": "QmcqU6QUDSXprb1518vYDGczrTJTyGwLG9eUa5iNX4xUtS", "name": "go-libp2p-peer", - "version": "2.4.2" + "version": "2.4.3" }, { "author": "vyzo", - "hash": "QmW1EkbsbRj5ZmeuvdMJqoLmmobUmLQjG4e8KgrdvABwB6", + "hash": "QmddZ5gv3Gkicoqh5NDfHGjpij6zw92pQjvpa181yfnXm2", "name": "go-libp2p-circuit", - "version": "2.3.4" + "version": "2.3.5" }, { "author": "lgierth", @@ -176,9 +176,9 @@ }, { "author": "why", - "hash": "Qmdv8PZ3Hdk3VxAZmTeuz8rmNGcjdGEqDRt7jJvz3hyTUH", + "hash": "QmR8DgkC3Xnc1TnfH1DvZtLRzPKJBrWfeDKseeXnUY6CN5", "name": "go-libp2p-interface-connmgr", - "version": "0.0.23" + "version": "0.0.24" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmSU7jRUFH1hHA3Xqy7cypFZLEKzTTnE5sX7xsF4FYGZvD", + "hash": "QmTt8BCNdw8uowByuFbUpphdrkTKVLJEcMNgewoYzb4K1c", "name": "go-ws-transport", - "version": "2.0.17" + "version": "2.0.18" }, { "author": "stebalien", - "hash": "QmQDzMXb6BFiheZj7c6Dysa43DZKBJrdPPqJ7TFrzrZYAf", + "hash": "QmZdxWGQU7PdcqcRExtzXZvcbxqM9CxrFW8HuS33XHm9yn", "name": "go-conn-security-multistream", - "version": "0.1.17" + "version": "0.1.18" }, { "author": "Stebalien", - "hash": "QmYreGEecvzC8WzrrQjAWVzj234Qi9XDC8Wj6dRdfpg5zT", + "hash": "QmbNjbKRJKbek3jPV6rpbGEtrZd84cDxjpBtsiGYa9Z5Do", "name": "go-conn-security", - "version": "0.1.18" + "version": "0.1.19" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "Qmcb6HnPAaqHznw45vW11G1vHCkbjZFHLKiifnrZafZW3C", + "hash": "QmbrgvQMRBhWJtG9pjerPb3V9xb3JzCDR6m1tp6J44iynL", "name": "go-libp2p-transport-upgrader", - "version": "0.1.18" + "version": "0.1.19" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -233,6 +233,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.25" + "version": "6.0.26" } From 4ea04c23b3f9ef455d7e67b3fd7831f81566beaa Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 17:03:59 +0300 Subject: [PATCH 0792/3965] identify: implement identify push protocol --- p2p/protocol/identify/id.go | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 2d32d1f418..d49bdf28a3 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -3,6 +3,7 @@ package identify import ( "context" "sync" + "time" pb "github.com/libp2p/go-libp2p/p2p/protocol/identify/pb" @@ -23,6 +24,9 @@ var log = logging.Logger("net/identify") // ID is the protocol.ID of the Identify Service. const ID = "/ipfs/id/1.0.0" +// IDPush is the protocol.ID of the Identify push protocol +const IDPush = "/ipfs/id/push/1.0.0" + // LibP2PVersion holds the current protocol version for a client running this code // TODO(jbenet): fix the versioning mess. const LibP2PVersion = "ipfs/0.1.0" @@ -60,6 +64,7 @@ func NewIDService(h host.Host) *IDService { currid: make(map[inet.Conn]chan struct{}), } h.SetStreamHandler(ID, s.requestHandler) + h.SetStreamHandler(IDPush, s.pushHandler) h.Network().Notify((*netNotifiee)(s)) return s } @@ -138,6 +143,24 @@ func (ids *IDService) responseHandler(s inet.Stream) { go inet.FullClose(s) } +func (ids *IDService) pushHandler(s inet.Stream) { + ids.responseHandler(s) +} + +func (ids *IDService) Push() { + for _, p := range ids.Host.Network().Peers() { + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + s, err := ids.Host.NewStream(ctx, p, IDPush) + cancel() + if err != nil { + log.Debugf("error opening push stream: %s", err.Error()) + continue + } + + ids.requestHandler(s) + } +} + func (ids *IDService) populateMessage(mes *pb.Identify, c inet.Conn) { // set protocols this node is currently handling From a89e74a1553db4b110cd2bf55759f2529752be59 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 17:04:46 +0300 Subject: [PATCH 0793/3965] basic host: export AddrsFactory, provide method to push identify --- p2p/host/basic/basic_host.go | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index d1bd0d653a..b5d2108b95 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -59,10 +59,11 @@ type BasicHost struct { ids *identify.IDService pings *ping.PingService natmgr NATManager - addrs AddrsFactory maResolver *madns.Resolver cmgr ifconnmgr.ConnManager + AddrsFactory AddrsFactory + negtimeout time.Duration proc goprocess.Process @@ -106,11 +107,11 @@ type HostOpts struct { // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, error) { h := &BasicHost{ - network: net, - mux: msmux.NewMultistreamMuxer(), - negtimeout: DefaultNegotiationTimeout, - addrs: DefaultAddrsFactory, - maResolver: madns.DefaultResolver, + network: net, + mux: msmux.NewMultistreamMuxer(), + negtimeout: DefaultNegotiationTimeout, + AddrsFactory: DefaultAddrsFactory, + maResolver: madns.DefaultResolver, } h.proc = goprocessctx.WithContextAndTeardown(ctx, func() error { @@ -136,7 +137,7 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, } if opts.AddrsFactory != nil { - h.addrs = opts.AddrsFactory + h.AddrsFactory = opts.AddrsFactory } if opts.NATManager != nil { @@ -256,6 +257,11 @@ func (h *BasicHost) newStreamHandler(s inet.Stream) { go handle(protoID, s) } +// PushIdentify pushes an identify update through the identify push protocol +func (h *BasicHost) PushIdentify() { + h.ids.Push() +} + // ID returns the (local) peer.ID associated with this Host func (h *BasicHost) ID() peer.ID { return h.Network().LocalPeer() @@ -474,7 +480,7 @@ func (h *BasicHost) ConnManager() ifconnmgr.ConnManager { // Addrs returns listening addresses that are safe to announce to the network. // The output is the same as AllAddrs, but processed by AddrsFactory. func (h *BasicHost) Addrs() []ma.Multiaddr { - return h.addrs(h.AllAddrs()) + return h.AddrsFactory(h.AllAddrs()) } // mergeAddrs merges input address lists, leave only unique addresses From a5858912d6da151ea5c95ba62742bbdbb9ad3ac6 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 17:05:15 +0300 Subject: [PATCH 0794/3965] autorelay hosts --- p2p/host/relay/autorelay.go | 252 ++++++++++++++++++++++++++++++++++++ p2p/host/relay/log.go | 7 + p2p/host/relay/relay.go | 23 ++++ 3 files changed, 282 insertions(+) create mode 100644 p2p/host/relay/autorelay.go create mode 100644 p2p/host/relay/log.go create mode 100644 p2p/host/relay/relay.go diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go new file mode 100644 index 0000000000..09a9e9884f --- /dev/null +++ b/p2p/host/relay/autorelay.go @@ -0,0 +1,252 @@ +package relay + +import ( + "context" + "fmt" + "math/rand" + "sync" + "time" + + basic "github.com/libp2p/go-libp2p/p2p/host/basic" + + autonat "github.com/libp2p/go-libp2p-autonat" + discovery "github.com/libp2p/go-libp2p-discovery" + host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +var DesiredRelays = 3 + +// AutoRelayHost is a Host that uses relays for connectivity when a NAT is detected. +type AutoRelayHost struct { + *basic.BasicHost + discover discovery.Discoverer + autonat autonat.AutoNAT + addrsF basic.AddrsFactory + + disconnect chan struct{} + + mx sync.Mutex + relays map[peer.ID]pstore.PeerInfo + addrs []ma.Multiaddr +} + +func NewAutoRelayHost(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer) *AutoRelayHost { + autonat := autonat.NewAutoNAT(ctx, bhost) + h := &AutoRelayHost{ + BasicHost: bhost, + discover: discover, + autonat: autonat, + addrsF: bhost.AddrsFactory, + relays: make(map[peer.ID]pstore.PeerInfo), + disconnect: make(chan struct{}, 1), + } + bhost.AddrsFactory = h.hostAddrs + bhost.Network().Notify(h) + go h.background(ctx) + return h +} + +func (h *AutoRelayHost) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { + h.mx.Lock() + defer h.mx.Unlock() + if h.addrs != nil && h.autonat.Status() == autonat.NATStatusPrivate { + return h.addrs + } else { + return h.addrsF(addrs) + } +} + +func (h *AutoRelayHost) background(ctx context.Context) { + select { + case <-time.After(autonat.AutoNATBootDelay + 30*time.Second): + case <-ctx.Done(): + return + } + + for { + wait := autonat.AutoNATRefreshInterval + switch h.autonat.Status() { + case autonat.NATStatusUnknown: + wait = autonat.AutoNATRetryInterval + case autonat.NATStatusPublic: + case autonat.NATStatusPrivate: + h.findRelays(ctx) + } + + select { + case <-h.disconnect: + // invalidate addrs + h.mx.Lock() + h.addrs = nil + h.mx.Unlock() + case <-time.After(wait): + case <-ctx.Done(): + return + } + } +} + +func (h *AutoRelayHost) findRelays(ctx context.Context) { + h.mx.Lock() + if len(h.relays) >= DesiredRelays { + h.mx.Unlock() + return + } + need := DesiredRelays - len(h.relays) + h.mx.Unlock() + + limit := 20 + for ; need > limit; limit *= 2 { + } + + dctx, cancel := context.WithTimeout(ctx, 60*time.Second) + pis, err := discovery.FindPeers(dctx, h.discover, "/libp2p/relay", limit) + cancel() + if err != nil { + log.Debugf("error discovering relays: %s", err.Error()) + return + } + + // TODO better relay selection strategy; this just selects random relays + // but we should probably use ping latency as the selection metric + shuffleRelays(pis) + + update := 0 + + for _, pi := range pis { + h.mx.Lock() + if _, ok := h.relays[pi.ID]; ok { + h.mx.Unlock() + continue + } + h.mx.Unlock() + + cctx, cancel := context.WithTimeout(ctx, 60*time.Second) + err = h.Connect(cctx, pi) + cancel() + if err != nil { + log.Debugf("error connecting to relay %s: %s", pi.ID, err.Error()) + continue + } + + log.Debugf("connected to relay %s", pi.ID) + h.mx.Lock() + h.relays[pi.ID] = pi + h.mx.Unlock() + + update++ + need-- + if need == 0 { + break + } + } + + if update > 0 || h.addrs == nil { + h.updateAddrs() + } +} + +func (h *AutoRelayHost) updateAddrs() { + h.doUpdateAddrs() + h.PushIdentify() +} + +func (h *AutoRelayHost) doUpdateAddrs() { + h.mx.Lock() + defer h.mx.Unlock() + + addrs := h.addrsF(h.AllAddrs()) + raddrs := make([]ma.Multiaddr, 0, len(addrs)+len(h.relays)) + + // remove our public addresses from the list and replace them by just the public IP + for _, addr := range addrs { + if manet.IsPublicAddr(addr) { + ip, err := addr.ValueForProtocol(ma.P_IP4) + if err == nil { + pub, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s", ip)) + if err != nil { + panic(err) + } + + if !containsAddr(raddrs, pub) { + raddrs = append(raddrs, pub) + } + continue + } + + ip, err = addr.ValueForProtocol(ma.P_IP6) + if err == nil { + pub, err := ma.NewMultiaddr(fmt.Sprintf("/ip6/%s", ip)) + if err != nil { + panic(err) + } + if !containsAddr(raddrs, pub) { + raddrs = append(raddrs, pub) + } + continue + } + } else { + raddrs = append(raddrs, addr) + } + } + + circuit, err := ma.NewMultiaddr("/p2p-circuit") + if err != nil { + panic(err) + } + + for _, pi := range h.relays { + for _, addr := range pi.Addrs { + if !manet.IsPrivateAddr(addr) { + pub := addr.Encapsulate(circuit) + raddrs = append(raddrs, pub) + } + } + } + + h.addrs = raddrs +} + +func shuffleRelays(pis []pstore.PeerInfo) { + for i := range pis { + j := rand.Intn(i + 1) + pis[i], pis[j] = pis[j], pis[i] + } +} + +func containsAddr(lst []ma.Multiaddr, addr ma.Multiaddr) bool { + for _, xaddr := range lst { + if xaddr.Equal(addr) { + return true + } + } + return false +} + +// notify +func (h *AutoRelayHost) Listen(inet.Network, ma.Multiaddr) {} +func (h *AutoRelayHost) ListenClose(inet.Network, ma.Multiaddr) {} +func (h *AutoRelayHost) Connected(inet.Network, inet.Conn) {} + +func (h *AutoRelayHost) Disconnected(_ inet.Network, c inet.Conn) { + p := c.RemotePeer() + h.mx.Lock() + defer h.mx.Unlock() + if _, ok := h.relays[p]; ok { + delete(h.relays, p) + select { + case h.disconnect <- struct{}{}: + default: + } + } +} + +func (h *AutoRelayHost) OpenedStream(inet.Network, inet.Stream) {} +func (h *AutoRelayHost) ClosedStream(inet.Network, inet.Stream) {} + +var _ host.Host = (*AutoRelayHost)(nil) diff --git a/p2p/host/relay/log.go b/p2p/host/relay/log.go new file mode 100644 index 0000000000..9671dc760b --- /dev/null +++ b/p2p/host/relay/log.go @@ -0,0 +1,7 @@ +package relay + +import ( + logging "github.com/ipfs/go-log" +) + +var log = logging.Logger("relay") diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go new file mode 100644 index 0000000000..95b5079c4d --- /dev/null +++ b/p2p/host/relay/relay.go @@ -0,0 +1,23 @@ +package relay + +import ( + "context" + + discovery "github.com/libp2p/go-libp2p-discovery" + host "github.com/libp2p/go-libp2p-host" +) + +// RelayHost is a Host that provides Relay services. +type RelayHost struct { + host.Host + advertise discovery.Advertiser +} + +// New constructs a new RelayHost +func NewRelayHost(ctx context.Context, host host.Host, advertise discovery.Advertiser) *RelayHost { + h := &RelayHost{Host: host, advertise: advertise} + discovery.Advertise(ctx, advertise, "/libp2p/relay") + return h +} + +var _ host.Host = (*RelayHost)(nil) From 9795a01ba31444b1a3807d5b10f14e2cf229c505 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 17:13:55 +0300 Subject: [PATCH 0795/3965] configurable boot delay for autorelay --- p2p/host/relay/autorelay.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 09a9e9884f..164b72be4f 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -19,7 +19,11 @@ import ( manet "github.com/multiformats/go-multiaddr-net" ) -var DesiredRelays = 3 +var ( + DesiredRelays = 3 + + BootDelay = 90 * time.Second +) // AutoRelayHost is a Host that uses relays for connectivity when a NAT is detected. type AutoRelayHost struct { @@ -63,7 +67,7 @@ func (h *AutoRelayHost) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { func (h *AutoRelayHost) background(ctx context.Context) { select { - case <-time.After(autonat.AutoNATBootDelay + 30*time.Second): + case <-time.After(autonat.AutoNATBootDelay + BootDelay): case <-ctx.Done(): return } From 5c623f269fbf0391dd684f3593f9c4a95ee5936f Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 17:19:13 +0300 Subject: [PATCH 0796/3965] name the autorelay logger as such --- p2p/host/relay/log.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/log.go b/p2p/host/relay/log.go index 9671dc760b..eca0fa45a8 100644 --- a/p2p/host/relay/log.go +++ b/p2p/host/relay/log.go @@ -4,4 +4,4 @@ import ( logging "github.com/ipfs/go-log" ) -var log = logging.Logger("relay") +var log = logging.Logger("autorelay") From d24fe6ae9161e8e6ce475fe8e7554d2e2c9fdbcb Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 18:24:54 +0300 Subject: [PATCH 0797/3965] update libp2p.New constructor to construct relayed/routed hosts --- config/config.go | 46 +++++++++++++++++++++++++++++++++++++++++++--- options.go | 11 +++++++++++ 2 files changed, 54 insertions(+), 3 deletions(-) diff --git a/config/config.go b/config/config.go index dd3d3996b7..aa06705c4f 100644 --- a/config/config.go +++ b/config/config.go @@ -5,10 +5,13 @@ import ( "fmt" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + relay "github.com/libp2p/go-libp2p/p2p/host/relay" + routed "github.com/libp2p/go-libp2p/p2p/host/routed" logging "github.com/ipfs/go-log" circuit "github.com/libp2p/go-libp2p-circuit" crypto "github.com/libp2p/go-libp2p-crypto" + discovery "github.com/libp2p/go-libp2p-discovery" host "github.com/libp2p/go-libp2p-host" ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" pnet "github.com/libp2p/go-libp2p-interface-pnet" @@ -16,6 +19,7 @@ import ( inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + routing "github.com/libp2p/go-libp2p-routing" swarm "github.com/libp2p/go-libp2p-swarm" tptu "github.com/libp2p/go-libp2p-transport-upgrader" filter "github.com/libp2p/go-maddr-filter" @@ -31,6 +35,13 @@ type AddrsFactory = bhost.AddrsFactory // NATManagerC is a NATManager constructor. type NATManagerC func(inet.Network) bhost.NATManager +type Routing interface { + routing.ContentRouting + routing.PeerRouting +} + +type RoutingC func(host.Host) (Routing, error) + // Config describes a set of settings for a libp2p node // // This is *not* a stable interface. Use the options defined in the root @@ -58,6 +69,8 @@ type Config struct { Reporter metrics.Reporter DisablePing bool + + Routing RoutingC } // NewNode constructs a new libp2p Host from the Config. @@ -99,8 +112,8 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { swrm.Filters = cfg.Filters } - // TODO: make host implementation configurable. - h, err := bhost.NewHost(ctx, swrm, &bhost.HostOpts{ + var h host.Host + h, err = bhost.NewHost(ctx, swrm, &bhost.HostOpts{ ConnManager: cfg.ConnManager, AddrsFactory: cfg.AddrsFactory, NATManager: cfg.NATManager, @@ -158,7 +171,34 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return nil, err } - // TODO: Configure routing (it's a pain to setup). + if cfg.Routing != nil { + router, err := cfg.Routing(h) + if err != nil { + h.Close() + return nil, err + } + + if cfg.Relay { + discovery := discovery.NewRoutingDiscovery(router) + + hop := false + for _, opt := range cfg.RelayOpts { + if opt == circuit.OptHop { + hop = true + break + } + } + + if hop { + h = relay.NewRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + } else { + h = relay.NewAutoRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + } + } + + h = routed.Wrap(h, router) + } + // TODO: Bootstrapping. return h, nil diff --git a/options.go b/options.go index 4ab4f39c73..23c9f7d22b 100644 --- a/options.go +++ b/options.go @@ -260,6 +260,17 @@ func Ping(enable bool) Option { } } +// Routing will configure libp2p to use routing. +func Routing(rt config.RoutingC) Option { + return func(cfg *Config) error { + if cfg.Routing != nil { + return fmt.Errorf("cannot specified multiple routing options") + } + cfg.Routing = rt + return nil + } +} + // NoListenAddrs will configure libp2p to not listen by default. // // This will both clear any configured listen addrs and prevent libp2p from From 6bdfcac0b4984c773cb36761df4f13dc4e3a5de6 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 17 Oct 2018 23:53:28 +0300 Subject: [PATCH 0798/3965] use AllAddrs as the address factory in autonat --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 164b72be4f..409a5e25ff 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -40,7 +40,7 @@ type AutoRelayHost struct { } func NewAutoRelayHost(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer) *AutoRelayHost { - autonat := autonat.NewAutoNAT(ctx, bhost) + autonat := autonat.NewAutoNAT(ctx, bhost, bhost.AllAddrs) h := &AutoRelayHost{ BasicHost: bhost, discover: discover, From 0ff739334949f534b4b57307a8bb3beb98c3ad6d Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 11:32:35 +0300 Subject: [PATCH 0799/3965] fix announced relay address --- p2p/host/relay/autorelay.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 409a5e25ff..18858795b6 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -199,12 +199,12 @@ func (h *AutoRelayHost) doUpdateAddrs() { } } - circuit, err := ma.NewMultiaddr("/p2p-circuit") - if err != nil { - panic(err) - } - for _, pi := range h.relays { + circuit, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit", pi.ID.Pretty())) + if err != nil { + panic(err) + } + for _, addr := range pi.Addrs { if !manet.IsPrivateAddr(addr) { pub := addr.Encapsulate(circuit) From 477c19aac12bb1217e3a2c0e855ca373e60f4e1c Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 11:48:13 +0300 Subject: [PATCH 0800/3965] reduce boot delay to 60s --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 18858795b6..aaae496acb 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -22,7 +22,7 @@ import ( var ( DesiredRelays = 3 - BootDelay = 90 * time.Second + BootDelay = 60 * time.Second ) // AutoRelayHost is a Host that uses relays for connectivity when a NAT is detected. From 7fabe39290271ea6b8a2b52db1140e20d6563a14 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 12:00:22 +0300 Subject: [PATCH 0801/3965] parallel identify push --- p2p/protocol/identify/id.go | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index d49bdf28a3..5991fe6950 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -149,15 +149,17 @@ func (ids *IDService) pushHandler(s inet.Stream) { func (ids *IDService) Push() { for _, p := range ids.Host.Network().Peers() { - ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) - s, err := ids.Host.NewStream(ctx, p, IDPush) - cancel() - if err != nil { - log.Debugf("error opening push stream: %s", err.Error()) - continue - } + go func(p peer.ID) { + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + s, err := ids.Host.NewStream(ctx, p, IDPush) + if err != nil { + log.Debugf("error opening push stream: %s", err.Error()) + return + } - ids.requestHandler(s) + ids.requestHandler(s) + }(p) } } From c6d1eebf15dae80ea5684566f2fa8bcbd5647c0e Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 16:47:40 +0300 Subject: [PATCH 0802/3965] autorelay test --- p2p/host/relay/autorelay_test.go | 169 +++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 p2p/host/relay/autorelay_test.go diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go new file mode 100644 index 0000000000..f4e2d3a4e2 --- /dev/null +++ b/p2p/host/relay/autorelay_test.go @@ -0,0 +1,169 @@ +package relay_test + +import ( + "context" + "net" + "sync" + "testing" + "time" + + libp2p "github.com/libp2p/go-libp2p" + config "github.com/libp2p/go-libp2p/config" + relay "github.com/libp2p/go-libp2p/p2p/host/relay" + + ggio "github.com/gogo/protobuf/io" + cid "github.com/ipfs/go-cid" + autonat "github.com/libp2p/go-libp2p-autonat" + autonatpb "github.com/libp2p/go-libp2p-autonat/pb" + circuit "github.com/libp2p/go-libp2p-circuit" + host "github.com/libp2p/go-libp2p-host" + inet "github.com/libp2p/go-libp2p-net" + peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" + routing "github.com/libp2p/go-libp2p-routing" + manet "github.com/multiformats/go-multiaddr-net" +) + +// test specific parameters +func init() { + autonat.AutoNATIdentifyDelay = 10 * time.Millisecond + autonat.AutoNATBootDelay = 1 * time.Second + relay.BootDelay = 1 * time.Second + manet.Private4 = []*net.IPNet{} +} + +// mock routing +type mockRoutingTable struct { + mx sync.Mutex + providers map[string]map[peer.ID]pstore.PeerInfo +} + +type mockRouting struct { + h host.Host + tab *mockRoutingTable +} + +func newMockRoutingTable() *mockRoutingTable { + return &mockRoutingTable{providers: make(map[string]map[peer.ID]pstore.PeerInfo)} +} + +func newMockRouting(h host.Host, tab *mockRoutingTable) *mockRouting { + return &mockRouting{h: h, tab: tab} +} + +func (m *mockRouting) FindPeer(ctx context.Context, p peer.ID) (pstore.PeerInfo, error) { + return pstore.PeerInfo{}, routing.ErrNotFound +} + +func (m *mockRouting) Provide(ctx context.Context, cid cid.Cid, bcast bool) error { + m.tab.mx.Lock() + defer m.tab.mx.Unlock() + + pmap, ok := m.tab.providers[cid.String()] + if !ok { + pmap = make(map[peer.ID]pstore.PeerInfo) + m.tab.providers[cid.String()] = pmap + } + + pmap[m.h.ID()] = pstore.PeerInfo{ID: m.h.ID(), Addrs: m.h.Addrs()} + + return nil +} + +func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit int) <-chan pstore.PeerInfo { + ch := make(chan pstore.PeerInfo) + go func() { + defer close(ch) + m.tab.mx.Lock() + defer m.tab.mx.Unlock() + + pmap, ok := m.tab.providers[cid.String()] + if !ok { + return + } + + for _, pi := range pmap { + select { + case ch <- pi: + case <-ctx.Done(): + return + } + } + }() + + return ch +} + +// mock autonat +func makeAutoNATServicePrivate(ctx context.Context, t *testing.T) host.Host { + h, err := libp2p.New(ctx) + if err != nil { + t.Fatal(err) + } + h.SetStreamHandler(autonat.AutoNATProto, sayAutoNATPrivate) + return h +} + +func sayAutoNATPrivate(s inet.Stream) { + defer s.Close() + w := ggio.NewDelimitedWriter(s) + res := autonatpb.Message{ + Type: autonatpb.Message_DIAL_RESPONSE.Enum(), + DialResponse: newDialResponseError(autonatpb.Message_E_DIAL_ERROR, "no dialable addresses"), + } + w.WriteMsg(&res) +} + +func newDialResponseError(status autonatpb.Message_ResponseStatus, text string) *autonatpb.Message_DialResponse { + dr := new(autonatpb.Message_DialResponse) + dr.Status = status.Enum() + dr.StatusText = &text + return dr +} + +// connector +func connect(t *testing.T, a, b host.Host) { + pinfo := pstore.PeerInfo{ID: a.ID(), Addrs: a.Addrs()} + err := b.Connect(context.Background(), pinfo) + if err != nil { + t.Fatal(err) + } +} + +// and the actual test! +func TestAutoRelay(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + mtab := newMockRoutingTable() + makeRouting := func(h host.Host) (config.Routing, error) { + mr := newMockRouting(h, mtab) + return mr, nil + } + + h1 := makeAutoNATServicePrivate(ctx, t) + _, err := libp2p.New(ctx, libp2p.EnableRelay(circuit.OptHop), libp2p.Routing(makeRouting)) + if err != nil { + t.Fatal(err) + } + h3, err := libp2p.New(ctx, libp2p.EnableRelay(), libp2p.Routing(makeRouting)) + if err != nil { + t.Fatal(err) + } + + connect(t, h1, h3) + time.Sleep(3 * time.Second) + + haveRelay := false + for _, addr := range h3.Addrs() { + _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) + if err != nil { + haveRelay = true + break + } + } + + if !haveRelay { + t.Fatal("No relay addrs advertised") + } +} From f6c38c9d9191eb5a431136cc473e995b478783e9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 17:17:49 +0300 Subject: [PATCH 0803/3965] filter unspecific relay address --- p2p/host/relay/autorelay.go | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index aaae496acb..13020793c1 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -10,6 +10,7 @@ import ( basic "github.com/libp2p/go-libp2p/p2p/host/basic" autonat "github.com/libp2p/go-libp2p-autonat" + _ "github.com/libp2p/go-libp2p-circuit" discovery "github.com/libp2p/go-libp2p-discovery" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" @@ -23,8 +24,18 @@ var ( DesiredRelays = 3 BootDelay = 60 * time.Second + + unspecificRelay ma.Multiaddr ) +func init() { + var err error + unspecificRelay, err = ma.NewMultiaddr("/p2p-circuit") + if err != nil { + panic(err) + } +} + // AutoRelayHost is a Host that uses relays for connectivity when a NAT is detected. type AutoRelayHost struct { *basic.BasicHost @@ -61,7 +72,7 @@ func (h *AutoRelayHost) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { if h.addrs != nil && h.autonat.Status() == autonat.NATStatusPrivate { return h.addrs } else { - return h.addrsF(addrs) + return filterUnspecificRelay(h.addrsF(addrs)) } } @@ -164,7 +175,7 @@ func (h *AutoRelayHost) doUpdateAddrs() { h.mx.Lock() defer h.mx.Unlock() - addrs := h.addrsF(h.AllAddrs()) + addrs := filterUnspecificRelay(h.addrsF(h.AllAddrs())) raddrs := make([]ma.Multiaddr, 0, len(addrs)+len(h.relays)) // remove our public addresses from the list and replace them by just the public IP @@ -216,6 +227,17 @@ func (h *AutoRelayHost) doUpdateAddrs() { h.addrs = raddrs } +func filterUnspecificRelay(addrs []ma.Multiaddr) []ma.Multiaddr { + res := make([]ma.Multiaddr, 0, len(addrs)) + for _, addr := range addrs { + if addr.Equal(unspecificRelay) { + continue + } + res = append(res, addr) + } + return res +} + func shuffleRelays(pis []pstore.PeerInfo) { for i := range pis { j := rand.Intn(i + 1) From 8d108dddcb24d9efce386c63df444b7a04160335 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Oct 2018 18:22:49 +0300 Subject: [PATCH 0804/3965] import go-libp2p-discovery and go-libp2p-autonat --- package.json | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/package.json b/package.json index c83cc672a5..52c3f84b48 100644 --- a/package.json +++ b/package.json @@ -226,6 +226,18 @@ "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", "name": "gogo-protobuf", "version": "0.0.0" + }, + { + "author": "vyzo", + "hash": "QmYwn471zWPm7dLaEtNWMUj9ytMDFwnpFy1uMQDK3BZ4xj", + "name": "go-libp2p-discovery", + "version": "1.0.0" + }, + { + "author": "vyzo", + "hash": "QmNtEW59XTrreF9U4NAjYUmixEJ56HnfiJCj1p97bFnah2", + "name": "go-libp2p-autonat", + "version": "1.0.0" } ], "gxVersion": "0.4.0", From 8faf5403c544a7064d1e78605e9d4dcc9d7acb48 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Oct 2018 15:54:17 +0300 Subject: [PATCH 0805/3965] fix typo --- options.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/options.go b/options.go index 23c9f7d22b..0015c80818 100644 --- a/options.go +++ b/options.go @@ -264,7 +264,7 @@ func Ping(enable bool) Option { func Routing(rt config.RoutingC) Option { return func(cfg *Config) error { if cfg.Routing != nil { - return fmt.Errorf("cannot specified multiple routing options") + return fmt.Errorf("cannot specify multiple routing options") } cfg.Routing = rt return nil From b62c83bd5cf1fdcc9507a87ca3a1f6f4b4b70f34 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Oct 2018 15:56:45 +0300 Subject: [PATCH 0806/3965] better limit adjustment for relay discovery --- p2p/host/relay/autorelay.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 13020793c1..8d8302da3f 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -116,7 +116,8 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { h.mx.Unlock() limit := 20 - for ; need > limit; limit *= 2 { + if need > limit/2 { + limit = 2 * need } dctx, cancel := context.WithTimeout(ctx, 60*time.Second) From ee69383c2c49aabb23e790f61e679c91f869f29f Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Oct 2018 08:29:20 +0300 Subject: [PATCH 0807/3965] extend autorelay test to verify connectivity --- p2p/host/relay/autorelay_test.go | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index f4e2d3a4e2..5f34880d01 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -21,6 +21,7 @@ import ( peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" routing "github.com/libp2p/go-libp2p-routing" + ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) @@ -150,10 +151,12 @@ func TestAutoRelay(t *testing.T) { if err != nil { t.Fatal(err) } + h4, err := libp2p.New(ctx, libp2p.EnableRelay()) connect(t, h1, h3) time.Sleep(3 * time.Second) + // verify that we advertise relay addrs haveRelay := false for _, addr := range h3.Addrs() { _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) @@ -166,4 +169,18 @@ func TestAutoRelay(t *testing.T) { if !haveRelay { t.Fatal("No relay addrs advertised") } + + // check that we can connect through the relay + var raddrs []ma.Multiaddr + for _, addr := range h3.Addrs() { + _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) + if err != nil { + raddrs = append(raddrs, addr) + } + } + + err = h4.Connect(ctx, pstore.PeerInfo{h3.ID(), raddrs}) + if err != nil { + t.Fatal(err) + } } From dcb8fd28134ac92b18a3cc205a1251efa0e0d272 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Oct 2018 20:53:56 +0300 Subject: [PATCH 0808/3965] fix inverted address selection logic in test --- p2p/host/relay/autorelay_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index 5f34880d01..38e7f518c4 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -160,7 +160,7 @@ func TestAutoRelay(t *testing.T) { haveRelay := false for _, addr := range h3.Addrs() { _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) - if err != nil { + if err == nil { haveRelay = true break } @@ -174,7 +174,7 @@ func TestAutoRelay(t *testing.T) { var raddrs []ma.Multiaddr for _, addr := range h3.Addrs() { _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) - if err != nil { + if err == nil { raddrs = append(raddrs, addr) } } From 89aef893e6f3bc1c7abe7be89dc165b14338a7e2 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Oct 2018 21:01:57 +0300 Subject: [PATCH 0809/3965] don't adveretise unspecific relay addrs in RelayHost --- p2p/host/relay/relay.go | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index 95b5079c4d..17aa09cd49 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -3,21 +3,34 @@ package relay import ( "context" + basic "github.com/libp2p/go-libp2p/p2p/host/basic" + discovery "github.com/libp2p/go-libp2p-discovery" host "github.com/libp2p/go-libp2p-host" + ma "github.com/multiformats/go-multiaddr" ) // RelayHost is a Host that provides Relay services. type RelayHost struct { - host.Host + *basic.BasicHost advertise discovery.Advertiser + addrsF basic.AddrsFactory } // New constructs a new RelayHost -func NewRelayHost(ctx context.Context, host host.Host, advertise discovery.Advertiser) *RelayHost { - h := &RelayHost{Host: host, advertise: advertise} +func NewRelayHost(ctx context.Context, bhost *basic.BasicHost, advertise discovery.Advertiser) *RelayHost { + h := &RelayHost{ + BasicHost: bhost, + addrsF: bhost.AddrsFactory, + advertise: advertise, + } + bhost.AddrsFactory = h.hostAddrs discovery.Advertise(ctx, advertise, "/libp2p/relay") return h } +func (h *RelayHost) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { + return filterUnspecificRelay(h.addrsF(addrs)) +} + var _ host.Host = (*RelayHost)(nil) From a7e1bf0f15ffad372b07313769af6a542fd2f086 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 22 Oct 2018 21:20:31 +0300 Subject: [PATCH 0810/3965] call the routing interface BasicRouting, alias to top level type --- config/config.go | 4 ++-- libp2p.go | 3 +++ p2p/host/relay/autorelay_test.go | 3 +-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/config/config.go b/config/config.go index aa06705c4f..3d06fdca0a 100644 --- a/config/config.go +++ b/config/config.go @@ -35,12 +35,12 @@ type AddrsFactory = bhost.AddrsFactory // NATManagerC is a NATManager constructor. type NATManagerC func(inet.Network) bhost.NATManager -type Routing interface { +type BasicRouting interface { routing.ContentRouting routing.PeerRouting } -type RoutingC func(host.Host) (Routing, error) +type RoutingC func(host.Host) (BasicRouting, error) // Config describes a set of settings for a libp2p node // diff --git a/libp2p.go b/libp2p.go index ad026ad4bd..6c928c3159 100644 --- a/libp2p.go +++ b/libp2p.go @@ -15,6 +15,9 @@ type Config = config.Config // (`libp2p.New`). type Option = config.Option +// BasicRouting is the combination of PeerRouting and ContentRouting +type BasicRouting = config.BasicRouting + // ChainOptions chains multiple options into a single option. func ChainOptions(opts ...Option) Option { return func(cfg *Config) error { diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index 38e7f518c4..ff4d850715 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -8,7 +8,6 @@ import ( "time" libp2p "github.com/libp2p/go-libp2p" - config "github.com/libp2p/go-libp2p/config" relay "github.com/libp2p/go-libp2p/p2p/host/relay" ggio "github.com/gogo/protobuf/io" @@ -137,7 +136,7 @@ func TestAutoRelay(t *testing.T) { defer cancel() mtab := newMockRoutingTable() - makeRouting := func(h host.Host) (config.Routing, error) { + makeRouting := func(h host.Host) (libp2p.BasicRouting, error) { mr := newMockRouting(h, mtab) return mr, nil } From 2993fd9a688251c95240572c1c477fb5b5077900 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 23 Oct 2018 12:57:09 +0300 Subject: [PATCH 0811/3965] add autorelay documentation --- p2p/host/relay/doc.go | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 p2p/host/relay/doc.go diff --git a/p2p/host/relay/doc.go b/p2p/host/relay/doc.go new file mode 100644 index 0000000000..8e642b2ad6 --- /dev/null +++ b/p2p/host/relay/doc.go @@ -0,0 +1,28 @@ +/* +The relay package contains host implementations that automatically +advertise relay addresses when the presence of NAT is detected. This +feature is dubbed `autorelay`. + +System Components: +- AutoNATService instances -- see https://github.com/libp2p/go-libp2p-autonat-svc +- One or more relays, instances of `RelayHost` +- The autorelayed hosts, instances of `AutoRelayHost`. + +How it works: +- `AutoNATService` instances are instantiated in the + bootstrappers (or other well known publicly reachable hosts) + +- `RelayHost`s are constructed with + `libp2p.New(libp2p.EnableRelay(circuit.OptHop), libp2p.Routing(makeDHT))`. + They provide Relay Hop services, and advertise through the DHT + in the `/libp2p/relay` namespace + +- `AutoRelayHost`s are constructed with `libp2p.New(libp2p.Routing(makeDHT))` + They passively discover autonat service instances and test dialability of + their listen address set through them. When the presence of NAT is detected, + they discover relays through the DHT, connect to some of them and begin + advertising relay addresses. The new set of addresses is propagated to + connected peers through the `identify/push` protocol. + +*/ +package relay From cc2dd220289c8eb7ba5146ffc6c370b331b8cd40 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 24 Oct 2018 10:27:42 +0300 Subject: [PATCH 0812/3965] make randezvous key a constant --- p2p/host/relay/autorelay.go | 6 +++++- p2p/host/relay/relay.go | 2 +- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 8d8302da3f..f506a0baaa 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -20,6 +20,10 @@ import ( manet "github.com/multiformats/go-multiaddr-net" ) +const ( + RelayRenezvous = "/libp2p/relay" +) + var ( DesiredRelays = 3 @@ -121,7 +125,7 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { } dctx, cancel := context.WithTimeout(ctx, 60*time.Second) - pis, err := discovery.FindPeers(dctx, h.discover, "/libp2p/relay", limit) + pis, err := discovery.FindPeers(dctx, h.discover, RelayRenezvous, limit) cancel() if err != nil { log.Debugf("error discovering relays: %s", err.Error()) diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index 17aa09cd49..8ba99d5ed4 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -25,7 +25,7 @@ func NewRelayHost(ctx context.Context, bhost *basic.BasicHost, advertise discove advertise: advertise, } bhost.AddrsFactory = h.hostAddrs - discovery.Advertise(ctx, advertise, "/libp2p/relay") + discovery.Advertise(ctx, advertise, RelayRenezvous) return h } From 67aba4d50149406de36e5cc79b381b16ea7debfe Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 24 Oct 2018 10:38:15 +0300 Subject: [PATCH 0813/3965] move relay selection strategy out of line --- p2p/host/relay/autorelay.go | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index f506a0baaa..9e2352710d 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -132,9 +132,7 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { return } - // TODO better relay selection strategy; this just selects random relays - // but we should probably use ping latency as the selection metric - shuffleRelays(pis) + pis = h.selectRelays(pis) update := 0 @@ -171,6 +169,13 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { } } +func (h *AutoRelayHost) selectRelays(pis []pstore.PeerInfo) []pstore.PeerInfo { + // TODO better relay selection strategy; this just selects random relays + // but we should probably use ping latency as the selection metric + shuffleRelays(pis) + return pis +} + func (h *AutoRelayHost) updateAddrs() { h.doUpdateAddrs() h.PushIdentify() From 4f90393629b0cb984e9ae84820174e4918efcb5f Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 24 Oct 2018 10:41:43 +0300 Subject: [PATCH 0814/3965] add a comment --- p2p/host/relay/autorelay.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 9e2352710d..26140b022f 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -220,6 +220,7 @@ func (h *AutoRelayHost) doUpdateAddrs() { } } + // add relay specific addrs to the list for _, pi := range h.relays { circuit, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit", pi.ID.Pretty())) if err != nil { From 84400e22f2b6fc9b8346d578badfef1956c38f0c Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 24 Oct 2018 16:17:49 +0300 Subject: [PATCH 0815/3965] fix typo --- p2p/host/relay/autorelay.go | 4 ++-- p2p/host/relay/relay.go | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 26140b022f..664f82d682 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -21,7 +21,7 @@ import ( ) const ( - RelayRenezvous = "/libp2p/relay" + RelayRendezvous = "/libp2p/relay" ) var ( @@ -125,7 +125,7 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { } dctx, cancel := context.WithTimeout(ctx, 60*time.Second) - pis, err := discovery.FindPeers(dctx, h.discover, RelayRenezvous, limit) + pis, err := discovery.FindPeers(dctx, h.discover, RelayRendezvous, limit) cancel() if err != nil { log.Debugf("error discovering relays: %s", err.Error()) diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index 8ba99d5ed4..69b6737078 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -25,7 +25,7 @@ func NewRelayHost(ctx context.Context, bhost *basic.BasicHost, advertise discove advertise: advertise, } bhost.AddrsFactory = h.hostAddrs - discovery.Advertise(ctx, advertise, RelayRenezvous) + discovery.Advertise(ctx, advertise, RelayRendezvous) return h } From e96605d1a34a56a63ef5dc94a0d5b9dd86ed6d60 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 24 Oct 2018 16:44:02 +0300 Subject: [PATCH 0816/3965] use /p2p multiaddr --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 664f82d682..088a477f3e 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -222,7 +222,7 @@ func (h *AutoRelayHost) doUpdateAddrs() { // add relay specific addrs to the list for _, pi := range h.relays { - circuit, err := ma.NewMultiaddr(fmt.Sprintf("/ipfs/%s/p2p-circuit", pi.ID.Pretty())) + circuit, err := ma.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", pi.ID.Pretty())) if err != nil { panic(err) } From 4be7ada37bcdeadd2cab8a3010fe4f1939d05aa5 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 24 Oct 2018 16:58:06 +0300 Subject: [PATCH 0817/3965] tag relay connections --- p2p/host/relay/autorelay.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 088a477f3e..01451ceb24 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -157,6 +157,9 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { h.relays[pi.ID] = pi h.mx.Unlock() + // tag the connection as very important + h.ConnManager().TagPeer(pi.ID, "relay", 42) + update++ need-- if need == 0 { From fdfa2246991938f733ccef87e52b59a6d8bfb797 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 24 Oct 2018 17:09:09 +0300 Subject: [PATCH 0818/3965] document doUpdateAddrs --- p2p/host/relay/autorelay.go | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 01451ceb24..6ea85bed47 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -184,6 +184,15 @@ func (h *AutoRelayHost) updateAddrs() { h.PushIdentify() } +// This function updates our NATed advertised addrs (h.addrs) +// The public addrs are rewritten so that they only retain the public IP part; they +// become undialable but are useful as a hint to the dialer to determine whether or not +// to dial private addrs. +// The non-public addrs are included verbatim so that peers behind the same NAT/firewall +// can still dial us directly. +// On top of those, we add the relay-specific addrs for the relays to which we are +// connected. For each non-private relay addr, we encapsulate the p2p-circuit addr +// through which we can be dialed. func (h *AutoRelayHost) doUpdateAddrs() { h.mx.Lock() defer h.mx.Unlock() From 292b8a12801d51d6be50e2f7aff81405d5759526 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 25 Oct 2018 11:49:17 +0300 Subject: [PATCH 0819/3965] update gx deps --- package.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/package.json b/package.json index 52c3f84b48..0bfb05796f 100644 --- a/package.json +++ b/package.json @@ -229,15 +229,15 @@ }, { "author": "vyzo", - "hash": "QmYwn471zWPm7dLaEtNWMUj9ytMDFwnpFy1uMQDK3BZ4xj", + "hash": "QmfWLPvbWEHjMbEsLwCzg3igdeijJHDWhLth3k9E3imdEL", "name": "go-libp2p-discovery", - "version": "1.0.0" + "version": "1.0.1" }, { "author": "vyzo", - "hash": "QmNtEW59XTrreF9U4NAjYUmixEJ56HnfiJCj1p97bFnah2", + "hash": "QmUn8mtaf4tTFwKnFRzkNYYLc8XEo3yz6qBfp5ShVB1HYZ", "name": "go-libp2p-autonat", - "version": "1.0.0" + "version": "1.0.1" } ], "gxVersion": "0.4.0", From c2d846a7c38e1685babd19a77d7cc75c9ffdca98 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 25 Oct 2018 12:00:00 +0300 Subject: [PATCH 0820/3965] fix go vet issue --- p2p/host/relay/autorelay_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index ff4d850715..241e33f65e 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -178,7 +178,7 @@ func TestAutoRelay(t *testing.T) { } } - err = h4.Connect(ctx, pstore.PeerInfo{h3.ID(), raddrs}) + err = h4.Connect(ctx, pstore.PeerInfo{ID: h3.ID(), Addrs: raddrs}) if err != nil { t.Fatal(err) } From 7b324b14b5702044c118d07c3a41af1621b89652 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 26 Oct 2018 13:58:35 +0300 Subject: [PATCH 0821/3965] remove BasicRouting interface; use PeerRouting and upcast for discovery --- config/config.go | 36 +++++++++++++++----------------- libp2p.go | 3 --- p2p/host/relay/autorelay_test.go | 2 +- 3 files changed, 18 insertions(+), 23 deletions(-) diff --git a/config/config.go b/config/config.go index 3d06fdca0a..c503509465 100644 --- a/config/config.go +++ b/config/config.go @@ -35,12 +35,7 @@ type AddrsFactory = bhost.AddrsFactory // NATManagerC is a NATManager constructor. type NATManagerC func(inet.Network) bhost.NATManager -type BasicRouting interface { - routing.ContentRouting - routing.PeerRouting -} - -type RoutingC func(host.Host) (BasicRouting, error) +type RoutingC func(host.Host) (routing.PeerRouting, error) // Config describes a set of settings for a libp2p node // @@ -178,21 +173,24 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return nil, err } - if cfg.Relay { - discovery := discovery.NewRoutingDiscovery(router) - - hop := false - for _, opt := range cfg.RelayOpts { - if opt == circuit.OptHop { - hop = true - break + crouter, ok := router.(routing.ContentRouting) + if ok { + if cfg.Relay { + discovery := discovery.NewRoutingDiscovery(crouter) + + hop := false + for _, opt := range cfg.RelayOpts { + if opt == circuit.OptHop { + hop = true + break + } } - } - if hop { - h = relay.NewRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) - } else { - h = relay.NewAutoRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + if hop { + h = relay.NewRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + } else { + h = relay.NewAutoRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + } } } diff --git a/libp2p.go b/libp2p.go index 6c928c3159..ad026ad4bd 100644 --- a/libp2p.go +++ b/libp2p.go @@ -15,9 +15,6 @@ type Config = config.Config // (`libp2p.New`). type Option = config.Option -// BasicRouting is the combination of PeerRouting and ContentRouting -type BasicRouting = config.BasicRouting - // ChainOptions chains multiple options into a single option. func ChainOptions(opts ...Option) Option { return func(cfg *Config) error { diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index 241e33f65e..a3ac7dd295 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -136,7 +136,7 @@ func TestAutoRelay(t *testing.T) { defer cancel() mtab := newMockRoutingTable() - makeRouting := func(h host.Host) (libp2p.BasicRouting, error) { + makeRouting := func(h host.Host) (routing.PeerRouting, error) { mr := newMockRouting(h, mtab) return mr, nil } From 0dfca3ba5d068b178a3431caaa1e26b52b7be0a8 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 26 Oct 2018 14:06:19 +0300 Subject: [PATCH 0822/3965] increase test AutoNATIdentifyDelay to 100ms macosx seems to fail intermittently, and that race is the likely culprit. --- p2p/host/relay/autorelay_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index a3ac7dd295..bd738f5ce8 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -26,7 +26,7 @@ import ( // test specific parameters func init() { - autonat.AutoNATIdentifyDelay = 10 * time.Millisecond + autonat.AutoNATIdentifyDelay = 100 * time.Millisecond autonat.AutoNATBootDelay = 1 * time.Second relay.BootDelay = 1 * time.Second manet.Private4 = []*net.IPNet{} From 69144bd78bcc0868b063bc61462ae6a4eb9a0f1c Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 26 Oct 2018 15:05:20 +0300 Subject: [PATCH 0823/3965] extend autorelay test to verify pushing of relay addrs also check that we don't initially advertise any. --- p2p/host/relay/autorelay_test.go | 40 +++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index bd738f5ce8..ce50ea8c09 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -152,16 +152,33 @@ func TestAutoRelay(t *testing.T) { } h4, err := libp2p.New(ctx, libp2p.EnableRelay()) + // verify that we don't advertise relay addrs initially + for _, addr := range h3.Addrs() { + _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) + if err == nil { + t.Fatal("relay addr advertised before auto detection") + } + } + + // connect to AutoNAT and let detection/discovery work its magic connect(t, h1, h3) time.Sleep(3 * time.Second) - // verify that we advertise relay addrs + // verify that we now advertise relay addrs (but not unspecific relay addrs) + unspecificRelay, err := ma.NewMultiaddr("/p2p-circuit") + if err != nil { + t.Fatal(err) + } + haveRelay := false for _, addr := range h3.Addrs() { + if addr.Equal(unspecificRelay) { + t.Fatal("unspecific relay addr advertised") + } + _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) if err == nil { haveRelay = true - break } } @@ -169,7 +186,7 @@ func TestAutoRelay(t *testing.T) { t.Fatal("No relay addrs advertised") } - // check that we can connect through the relay + // verify that we can connect through the relay var raddrs []ma.Multiaddr for _, addr := range h3.Addrs() { _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) @@ -182,4 +199,21 @@ func TestAutoRelay(t *testing.T) { if err != nil { t.Fatal(err) } + + // verify that we have pushed relay addrs to connected peers + haveRelay = false + for _, addr := range h1.Peerstore().Addrs(h3.ID()) { + if addr.Equal(unspecificRelay) { + t.Fatal("unspecific relay addr advertised") + } + + _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) + if err == nil { + haveRelay = true + } + } + + if !haveRelay { + t.Fatal("No relay addrs pushed") + } } From 5d8988f3ee0d2ce8621972edbfc6ad090782fc6d Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 30 Oct 2018 11:08:02 +0200 Subject: [PATCH 0824/3965] add comment about the unstable nature of BasicHost.PushIdentify --- p2p/host/basic/basic_host.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index b5d2108b95..13452e2f9c 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -258,6 +258,7 @@ func (h *BasicHost) newStreamHandler(s inet.Stream) { } // PushIdentify pushes an identify update through the identify push protocol +// Warning: this interface is unstable and may disappear in the future. func (h *BasicHost) PushIdentify() { h.ids.Push() } From 5a1c09c4afe061e4a5913b66985315e043b8c665 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 2 Nov 2018 11:39:51 +0200 Subject: [PATCH 0825/3965] use advertised addrs for autonat dial back, not all addrs --- p2p/host/relay/autorelay.go | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 6ea85bed47..e97ce88321 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -55,15 +55,14 @@ type AutoRelayHost struct { } func NewAutoRelayHost(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer) *AutoRelayHost { - autonat := autonat.NewAutoNAT(ctx, bhost, bhost.AllAddrs) h := &AutoRelayHost{ BasicHost: bhost, discover: discover, - autonat: autonat, addrsF: bhost.AddrsFactory, relays: make(map[peer.ID]pstore.PeerInfo), disconnect: make(chan struct{}, 1), } + h.autonat = autonat.NewAutoNAT(ctx, bhost, h.baseAddrs) bhost.AddrsFactory = h.hostAddrs bhost.Network().Notify(h) go h.background(ctx) @@ -80,6 +79,10 @@ func (h *AutoRelayHost) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { } } +func (h *AutoRelayHost) baseAddrs() []ma.Multiaddr { + return filterUnspecificRelay(h.addrsF(h.AllAddrs())) +} + func (h *AutoRelayHost) background(ctx context.Context) { select { case <-time.After(autonat.AutoNATBootDelay + BootDelay): @@ -197,7 +200,7 @@ func (h *AutoRelayHost) doUpdateAddrs() { h.mx.Lock() defer h.mx.Unlock() - addrs := filterUnspecificRelay(h.addrsF(h.AllAddrs())) + addrs := h.baseAddrs() raddrs := make([]ma.Multiaddr, 0, len(addrs)+len(h.relays)) // remove our public addresses from the list and replace them by just the public IP From 4cb5d00e2cf34d9194c1414c2fb4f735b9cfbebc Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 4 Nov 2018 10:55:17 +0200 Subject: [PATCH 0826/3965] gx update --- package.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/package.json b/package.json index 0bfb05796f..65d8edec28 100644 --- a/package.json +++ b/package.json @@ -229,15 +229,15 @@ }, { "author": "vyzo", - "hash": "QmfWLPvbWEHjMbEsLwCzg3igdeijJHDWhLth3k9E3imdEL", + "hash": "QmT8eNT96scr6wcrARzyxrCcsBAPKn9GkEx2TeXZniHiMP", "name": "go-libp2p-discovery", - "version": "1.0.1" + "version": "1.0.2" }, { "author": "vyzo", - "hash": "QmUn8mtaf4tTFwKnFRzkNYYLc8XEo3yz6qBfp5ShVB1HYZ", + "hash": "QmU2XQcwPxikg1jZqDBMFgLQVan7zjT2HtQWrWui3vbUVS", "name": "go-libp2p-autonat", - "version": "1.0.1" + "version": "1.0.2" } ], "gxVersion": "0.4.0", From a309f096067c578402a4fdaec133160a9abb1351 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 6 Nov 2018 09:50:25 +0200 Subject: [PATCH 0827/3965] Add note to relay docs about internal interface instability --- p2p/host/relay/doc.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/relay/doc.go b/p2p/host/relay/doc.go index 8e642b2ad6..c0a5878113 100644 --- a/p2p/host/relay/doc.go +++ b/p2p/host/relay/doc.go @@ -3,6 +3,8 @@ The relay package contains host implementations that automatically advertise relay addresses when the presence of NAT is detected. This feature is dubbed `autorelay`. +Warning: the internal interfaces are unstable. + System Components: - AutoNATService instances -- see https://github.com/libp2p/go-libp2p-autonat-svc - One or more relays, instances of `RelayHost` From 8fb174c11f6efcf51779daa7c327256dcd29ce84 Mon Sep 17 00:00:00 2001 From: Adrian Lanzafame Date: Mon, 29 Oct 2018 20:14:51 +1000 Subject: [PATCH 0828/3965] prevent timeout by un-nesting tb.Runs License: MIT Signed-off-by: Adrian Lanzafame --- p2p/host/peerstore/pstoreds/ds_test.go | 30 +++++++++++++------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 29ac9eb04d..1557c65e1a 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -33,26 +33,24 @@ func TestDsPeerstore(t *testing.T) { func TestDsAddrBook(t *testing.T) { for name, dsFactory := range dstores { - t.Run(name, func(t *testing.T) { - t.Run("Cacheful", func(t *testing.T) { - t.Parallel() + t.Run(name+" Cacheful", func(t *testing.T) { + t.Parallel() - opts := DefaultOpts() - opts.TTLInterval = 100 * time.Microsecond - opts.CacheSize = 1024 + opts := DefaultOpts() + opts.TTLInterval = 100 * time.Microsecond + opts.CacheSize = 1024 - pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) - }) + pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) + }) - t.Run("Cacheless", func(t *testing.T) { - t.Parallel() + t.Run(name+" Cacheless", func(t *testing.T) { + t.Parallel() - opts := DefaultOpts() - opts.TTLInterval = 100 * time.Microsecond - opts.CacheSize = 0 + opts := DefaultOpts() + opts.TTLInterval = 100 * time.Microsecond + opts.CacheSize = 0 - pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) - }) + pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) }) } } @@ -75,6 +73,8 @@ func BenchmarkDsPeerstore(b *testing.B) { for name, dsFactory := range dstores { b.Run(name, func(b *testing.B) { pt.BenchmarkPeerstore(b, peerstoreFactory(b, dsFactory, caching), "Caching") + }) + b.Run(name, func(b *testing.B) { pt.BenchmarkPeerstore(b, peerstoreFactory(b, dsFactory, cacheless), "Cacheless") }) } From 9c6157d05479440c0858e9078a220528e572f7e0 Mon Sep 17 00:00:00 2001 From: Adrian Lanzafame Date: Mon, 29 Oct 2018 20:16:51 +1000 Subject: [PATCH 0829/3965] sort suite so benchmarks always run in the same order License: MIT Signed-off-by: Adrian Lanzafame --- p2p/host/peerstore/test/benchmarks_suite.go | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/test/benchmarks_suite.go b/p2p/host/peerstore/test/benchmarks_suite.go index 69fde2d2a4..f283b9b0a9 100644 --- a/p2p/host/peerstore/test/benchmarks_suite.go +++ b/p2p/host/peerstore/test/benchmarks_suite.go @@ -3,6 +3,7 @@ package test import ( "context" "fmt" + "sort" "testing" pstore "github.com/libp2p/go-libp2p-peerstore" @@ -38,7 +39,15 @@ func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory, variant string) go addressProducer(ctx, b, p.ch, p.n) } - for name, bench := range peerstoreBenchmarks { + // So tests are always run in the same order. + ordernames := make([]string, 0, len(peerstoreBenchmarks)) + for name := range peerstoreBenchmarks { + ordernames = append(ordernames, name) + } + sort.Strings(ordernames) + + for _, name := range ordernames { + bench := peerstoreBenchmarks[name] for _, p := range params { // Create a new peerstore. ps, closeFunc := factory() From 56ce954c2515dd718fd952124eb85e935dbbc3d4 Mon Sep 17 00:00:00 2001 From: Adrian Lanzafame Date: Mon, 29 Oct 2018 20:17:10 +1000 Subject: [PATCH 0830/3965] add keybook benchmarks License: MIT Signed-off-by: Adrian Lanzafame --- p2p/host/peerstore/pstoreds/ds_test.go | 8 ++ p2p/host/peerstore/pstoremem/inmem_test.go | 6 + p2p/host/peerstore/test/keybook_suite.go | 141 +++++++++++++++++++++ 3 files changed, 155 insertions(+) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 1557c65e1a..c98d4a6831 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -63,6 +63,14 @@ func TestDsKeyBook(t *testing.T) { } } +func BenchmarkDsKeyBook(b *testing.B) { + for name, dsFactory := range dstores { + b.Run(name, func(b *testing.B) { + pt.BenchmarkKeyBook(b, keyBookFactory(b, dsFactory, DefaultOpts())) + }) + } +} + func BenchmarkDsPeerstore(b *testing.B) { caching := DefaultOpts() caching.CacheSize = 1024 diff --git a/p2p/host/peerstore/pstoremem/inmem_test.go b/p2p/host/peerstore/pstoremem/inmem_test.go index 510399dabd..97e24e4288 100644 --- a/p2p/host/peerstore/pstoremem/inmem_test.go +++ b/p2p/host/peerstore/pstoremem/inmem_test.go @@ -30,3 +30,9 @@ func BenchmarkInMemoryPeerstore(b *testing.B) { return NewPeerstore(), nil }, "InMem") } + +func BenchmarkInMemoryKeyBook(b *testing.B) { + pt.BenchmarkKeyBook(b, func() (pstore.KeyBook, func()) { + return NewKeyBook(), nil + }) +} diff --git a/p2p/host/peerstore/test/keybook_suite.go b/p2p/host/peerstore/test/keybook_suite.go index e713f0eb10..f4f84967e2 100644 --- a/p2p/host/peerstore/test/keybook_suite.go +++ b/p2p/host/peerstore/test/keybook_suite.go @@ -161,3 +161,144 @@ func testInlinedPubKeyAddedOnRetrieve(kb pstore.KeyBook) func(t *testing.T) { } } } + +var keybookBenchmarkSuite = map[string]func(kb pstore.KeyBook) func(*testing.B){ + "PubKey": benchmarkPubKey, + "AddPubKey": benchmarkAddPubKey, + "PrivKey": benchmarkPrivKey, + "AddPrivKey": benchmarkAddPrivKey, + "PeersWithKeys": benchmarkPeersWithKeys, +} + +func BenchmarkKeyBook(b *testing.B, factory KeyBookFactory) { + ordernames := make([]string, 0, len(keybookBenchmarkSuite)) + for name := range keybookBenchmarkSuite { + ordernames = append(ordernames, name) + } + sort.Strings(ordernames) + for _, name := range ordernames { + bench := keybookBenchmarkSuite[name] + kb, closeFunc := factory() + + b.Run(name, bench(kb)) + + if closeFunc != nil { + closeFunc() + } + } +} + +func benchmarkPubKey(kb pstore.KeyBook) func(*testing.B) { + return func(b *testing.B) { + _, pub, err := pt.RandTestKeyPair(512) + if err != nil { + b.Error(err) + } + + id, err := peer.IDFromPublicKey(pub) + if err != nil { + b.Error(err) + } + + err = kb.AddPubKey(id, pub) + if err != nil { + b.Fatal(err) + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + kb.PubKey(id) + } + } +} + +func benchmarkAddPubKey(kb pstore.KeyBook) func(*testing.B) { + return func(b *testing.B) { + _, pub, err := pt.RandTestKeyPair(512) + if err != nil { + b.Error(err) + } + + id, err := peer.IDFromPublicKey(pub) + if err != nil { + b.Error(err) + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + kb.AddPubKey(id, pub) + } + } +} + +func benchmarkPrivKey(kb pstore.KeyBook) func(*testing.B) { + return func(b *testing.B) { + priv, _, err := pt.RandTestKeyPair(512) + if err != nil { + b.Error(err) + } + + id, err := peer.IDFromPrivateKey(priv) + if err != nil { + b.Error(err) + } + + err = kb.AddPrivKey(id, priv) + if err != nil { + b.Fatal(err) + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + kb.PrivKey(id) + } + } +} + +func benchmarkAddPrivKey(kb pstore.KeyBook) func(*testing.B) { + return func(b *testing.B) { + priv, _, err := pt.RandTestKeyPair(512) + if err != nil { + b.Error(err) + } + + id, err := peer.IDFromPrivateKey(priv) + if err != nil { + b.Error(err) + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + kb.AddPrivKey(id, priv) + } + } +} + +func benchmarkPeersWithKeys(kb pstore.KeyBook) func(*testing.B) { + return func(b *testing.B) { + for i := 0; i < 10; i++ { + priv, pub, err := pt.RandTestKeyPair(512) + if err != nil { + b.Error(err) + } + + id, err := peer.IDFromPublicKey(pub) + if err != nil { + b.Error(err) + } + + err = kb.AddPubKey(id, pub) + if err != nil { + b.Fatal(err) + } + err = kb.AddPrivKey(id, priv) + if err != nil { + b.Fatal(err) + } + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + kb.PeersWithKeys() + } + } +} From f4229470cbd178d05e5c5f352f43df66cd0fa52f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 7 Nov 2018 15:40:37 -0800 Subject: [PATCH 0831/3965] mocknet: use peer ID in peer address RandLocalTCPAddress is mostly useful when we *actually* want to listen on a real address. Unfortunately, when running a bunch of tests, we can actually run out. With this change, a collision means we have a duplicate peer ID so yeah... fixes #473 --- p2p/net/mock/mock_net.go | 22 +++++++++++++++++++--- p2p/net/mock/mock_test.go | 8 ++++++++ 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/p2p/net/mock/mock_net.go b/p2p/net/mock/mock_net.go index 9242cea549..2e34d3f74b 100644 --- a/p2p/net/mock/mock_net.go +++ b/p2p/net/mock/mock_net.go @@ -3,6 +3,7 @@ package mocknet import ( "context" "fmt" + "net" "sort" "sync" @@ -17,10 +18,13 @@ import ( peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" - testutil "github.com/libp2p/go-testutil" ma "github.com/multiformats/go-multiaddr" ) +// IP6 range that gets blackholed (in case our traffic ever makes it out onto +// the internet). +var blackholeIP6 = net.ParseIP("100::") + // mocknet implements mocknet.Mocknet type mocknet struct { nets map[peer.ID]*peernet @@ -54,8 +58,20 @@ func (mn *mocknet) GenPeer() (host.Host, error) { if err != nil { return nil, err } - - a := testutil.RandLocalTCPAddress() + id, err := peer.IDFromPrivateKey(sk) + if err != nil { + return nil, err + } + suffix := id + if len(id) > 8 { + suffix = id[len(id)-8:] + } + ip := append(net.IP{}, blackholeIP6...) + copy(ip[net.IPv6len-len(suffix):], suffix) + a, err := ma.NewMultiaddr(fmt.Sprintf("/ip6/%s/tcp/4242", ip)) + if err != nil { + return nil, fmt.Errorf("failed to create test multiaddr: %s", err) + } h, err := mn.AddPeer(sk, a) if err != nil { diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index d22294c8aa..0f888a7098 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -582,6 +582,14 @@ func TestLimitedStreams(t *testing.T) { t.Fatal("Expected 2ish seconds but got ", time.Since(before)) } } +func TestFuzzManyPeers(t *testing.T) { + for i := 0; i < 50000; i++ { + _, err := FullMeshConnected(context.Background(), 2) + if err != nil { + t.Fatal(err) + } + } +} func TestStreamsWithLatency(t *testing.T) { latency := time.Millisecond * 500 From 5d053a26b58491942d398d59ae78a7f013ccbee3 Mon Sep 17 00:00:00 2001 From: wzp Date: Thu, 8 Nov 2018 10:12:34 +0800 Subject: [PATCH 0832/3965] PeerInfo UnMarshal Error #393 --- p2p/host/peerstore/peerinfo.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index f41b9f85ad..5995b3ef99 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -70,14 +70,14 @@ func InfoToP2pAddrs(pi *PeerInfo) ([]ma.Multiaddr, error) { return addrs, nil } -func (pi *PeerInfo) Loggable() map[string]interface{} { +func (pi PeerInfo) Loggable() map[string]interface{} { return map[string]interface{}{ "peerID": pi.ID.Pretty(), "addrs": pi.Addrs, } } -func (pi *PeerInfo) MarshalJSON() ([]byte, error) { +func (pi PeerInfo) MarshalJSON() ([]byte, error) { out := make(map[string]interface{}) out["ID"] = pi.ID.Pretty() var addrs []string @@ -88,7 +88,7 @@ func (pi *PeerInfo) MarshalJSON() ([]byte, error) { return json.Marshal(out) } -func (pi *PeerInfo) UnmarshalJSON(b []byte) error { +func (pi PeerInfo) UnmarshalJSON(b []byte) error { var data map[string]interface{} err := json.Unmarshal(b, &data) if err != nil { From fc976fdfa96c55fbe5d8ebdc7c019229918997e5 Mon Sep 17 00:00:00 2001 From: Eli Wang Date: Thu, 8 Nov 2018 10:40:39 +0800 Subject: [PATCH 0833/3965] bug fix: UnmarshalJSON should take a pointer --- p2p/host/peerstore/peerinfo.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index 5995b3ef99..f0ff3bad0e 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -70,7 +70,7 @@ func InfoToP2pAddrs(pi *PeerInfo) ([]ma.Multiaddr, error) { return addrs, nil } -func (pi PeerInfo) Loggable() map[string]interface{} { +func (pi *PeerInfo) Loggable() map[string]interface{} { return map[string]interface{}{ "peerID": pi.ID.Pretty(), "addrs": pi.Addrs, @@ -88,7 +88,7 @@ func (pi PeerInfo) MarshalJSON() ([]byte, error) { return json.Marshal(out) } -func (pi PeerInfo) UnmarshalJSON(b []byte) error { +func (pi *PeerInfo) UnmarshalJSON(b []byte) error { var data map[string]interface{} err := json.Unmarshal(b, &data) if err != nil { From 00ba6a165ae63cf6710dade02a805398f435289b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 8 Nov 2018 10:49:57 -0800 Subject: [PATCH 0834/3965] deflake identify test This was probably failing (rarely) due to the fact that we're shrinking the timeout asynchronously (I think?). --- p2p/protocol/identify/id_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index 2c9b279f56..621ce1aa0c 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -88,7 +88,7 @@ func subtestIDService(t *testing.T) { // Forget the first one. testKnowsAddrs(t, h2, h1p, addrs[:len(addrs)-1]) - time.Sleep(500 * time.Millisecond) + time.Sleep(1 * time.Second) // Forget the rest. testKnowsAddrs(t, h1, h2p, []ma.Multiaddr{}) From a8e25bf270c59fba86319c7b73ec11efed07c617 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 8 Nov 2018 11:08:14 -0800 Subject: [PATCH 0835/3965] mocknet: refuse to connect to self The swarm does this as well and most of our services will fail if we don't have this. --- p2p/net/mock/mock.go | 11 +++-------- p2p/net/mock/mock_peernet.go | 4 ++++ p2p/net/mock/mock_test.go | 17 ++++++++++------- 3 files changed, 17 insertions(+), 15 deletions(-) diff --git a/p2p/net/mock/mock.go b/p2p/net/mock/mock.go index 8760b4d489..55256f867f 100644 --- a/p2p/net/mock/mock.go +++ b/p2p/net/mock/mock.go @@ -44,14 +44,9 @@ func FullMeshConnected(ctx context.Context, n int) (Mocknet, error) { return nil, err } - nets := m.Nets() - for _, n1 := range nets { - for _, n2 := range nets { - if _, err := m.ConnectNets(n1, n2); err != nil { - return nil, err - } - } + err = m.ConnectAllButSelf() + if err != nil { + return nil, err } - return m, nil } diff --git a/p2p/net/mock/mock_peernet.go b/p2p/net/mock/mock_peernet.go index 3c5710bf91..52331016ca 100644 --- a/p2p/net/mock/mock_peernet.go +++ b/p2p/net/mock/mock_peernet.go @@ -118,6 +118,10 @@ func (pn *peernet) DialPeer(ctx context.Context, p peer.ID) (inet.Conn, error) { } func (pn *peernet) connect(p peer.ID) (*conn, error) { + if p == pn.peer { + return nil, fmt.Errorf("attempted to dial self %s", p) + } + // first, check if we already have live connections pn.RLock() cs, found := pn.connsByPeer[p] diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index 0f888a7098..d6d3c72680 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -225,14 +225,14 @@ func TestNetworkSetup(t *testing.T) { t.Error("should not be able to connect") } - // connect p1->p1 (should work) - if _, err := n1.DialPeer(ctx, p1); err != nil { - t.Error("p1 should be able to dial self.", err) + // connect p1->p1 (should fail) + if _, err := n1.DialPeer(ctx, p1); err == nil { + t.Error("p1 shouldn't be able to dial self") } // and a stream too - if _, err := n1.NewStream(ctx, p1); err != nil { - t.Error(err) + if _, err := n1.NewStream(ctx, p1); err == nil { + t.Error("p1 shouldn't be able to dial self") } // connect p1->p2 @@ -383,8 +383,11 @@ func TestStreamsStress(t *testing.T) { wg.Add(1) go func(i int) { defer wg.Done() - from := rand.Intn(len(hosts)) - to := rand.Intn(len(hosts)) + var from, to int + for from == to { + from = rand.Intn(len(hosts)) + to = rand.Intn(len(hosts)) + } s, err := hosts[from].NewStream(ctx, hosts[to].ID(), protocol.TestingID) if err != nil { log.Debugf("%d (%s) %d (%s)", from, hosts[from], to, hosts[to]) From 79ba610d21eb5213906e959dc63d4bdf0b771ded Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 8 Nov 2018 11:08:38 -0800 Subject: [PATCH 0836/3965] mocknet: create a connection on NewStream if we need one That's what the Swarm does and that's what the function's documentation says it does. --- p2p/net/mock/mock_peernet.go | 22 +++------------------- 1 file changed, 3 insertions(+), 19 deletions(-) diff --git a/p2p/net/mock/mock_peernet.go b/p2p/net/mock/mock_peernet.go index 52331016ca..35c9bbe647 100644 --- a/p2p/net/mock/mock_peernet.go +++ b/p2p/net/mock/mock_peernet.go @@ -330,26 +330,10 @@ func (pn *peernet) Connectedness(p peer.ID) inet.Connectedness { // NewStream returns a new stream to given peer p. // If there is no connection to p, attempts to create one. func (pn *peernet) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { - pn.Lock() - cs, found := pn.connsByPeer[p] - if !found || len(cs) < 1 { - pn.Unlock() - return nil, fmt.Errorf("no connection to peer") + c, err := pn.DialPeer(ctx, p) + if err != nil { + return nil, err } - - // if many conns are found, how do we select? for now, randomly... - // this would be an interesting place to test logic that can measure - // links (network interfaces) and select properly - n := rand.Intn(len(cs)) - var c *conn - for c = range cs { - if n == 0 { - break - } - n-- - } - pn.Unlock() - return c.NewStream() } From e4c99a186655448f39ba8cc511c845e27d679c17 Mon Sep 17 00:00:00 2001 From: wzp Date: Fri, 9 Nov 2018 11:13:06 +0800 Subject: [PATCH 0837/3965] add marshal unittest #393 --- p2p/host/peerstore/peerinfo_test.go | 83 +++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go index 84a44b1797..336c719b03 100644 --- a/p2p/host/peerstore/peerinfo_test.go +++ b/p2p/host/peerstore/peerinfo_test.go @@ -1,6 +1,7 @@ package peerstore import ( + "encoding/json" "testing" "github.com/libp2p/go-libp2p-peer" @@ -57,6 +58,88 @@ func TestPeerInfoMarshal(t *testing.T) { } } +func TestPeerInfoMarshalWithPointer(t *testing.T) { + a := mustAddr(t, "/ip4/1.2.3.4/tcp/4536") + b := mustAddr(t, "/ip4/1.2.3.8/udp/7777") + id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") + if err != nil { + t.Fatal(err) + } + + pi := PeerInfo{ + ID: id, + Addrs: []ma.Multiaddr{a, b}, + } + + data, err := json.Marshal(&pi) + if err != nil { + t.Fatal(err) + } + + pi2 := new(PeerInfo) + if err := json.Unmarshal(data, pi2); err != nil { + t.Fatal(err) + } + + if pi2.ID != pi.ID { + t.Fatal("ids didnt match after marshal") + } + + if !pi.Addrs[0].Equal(pi2.Addrs[0]) { + t.Fatal("wrong addrs") + } + + if !pi.Addrs[1].Equal(pi2.Addrs[1]) { + t.Fatal("wrong addrs") + } + + lgbl := pi2.Loggable() + if lgbl["peerID"] != id.Pretty() { + t.Fatal("loggables gave wrong peerID output") + } +} + +func TestPeerInfoMarshalWithValue(t *testing.T) { + a := mustAddr(t, "/ip4/1.2.3.4/tcp/4536") + b := mustAddr(t, "/ip4/1.2.3.8/udp/7777") + id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") + if err != nil { + t.Fatal(err) + } + + pi := PeerInfo{ + ID: id, + Addrs: []ma.Multiaddr{a, b}, + } + + data, err := json.Marshal(pi) + if err != nil { + t.Fatal(err) + } + + pi2 := new(PeerInfo) + if err := json.Unmarshal(data, pi2); err != nil { + t.Fatal(err) + } + + if pi2.ID != pi.ID { + t.Fatal("ids didnt match after marshal") + } + + if !pi.Addrs[0].Equal(pi2.Addrs[0]) { + t.Fatal("wrong addrs") + } + + if !pi.Addrs[1].Equal(pi2.Addrs[1]) { + t.Fatal("wrong addrs") + } + + lgbl := pi2.Loggable() + if lgbl["peerID"] != id.Pretty() { + t.Fatal("loggables gave wrong peerID output") + } +} + func TestP2pAddrParsing(t *testing.T) { id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") if err != nil { From f7df58532f79ce3f8d9586fc287b73b855385eaa Mon Sep 17 00:00:00 2001 From: hannahhoward Date: Sat, 10 Nov 2018 11:05:01 -0800 Subject: [PATCH 0838/3965] gx publish 6.0.27 --- .gx/lastpubver | 2 +- package.json | 75 +++++++++++++++++++++++++++----------------------- 2 files changed, 41 insertions(+), 36 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 30f352f3c9..25c4296c72 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.26: QmXnpYYg2onGLXVxM4Q5PEFcx29k8zeJQkPeLAk9h9naxg +6.0.27: QmVvV8JQmmqPCwXAaesWJPheUiEFQJ9HWRhWhuFuxVQxpR diff --git a/package.json b/package.json index 65d8edec28..5afdc6cac0 100644 --- a/package.json +++ b/package.json @@ -56,27 +56,27 @@ }, { "author": "whyrusleeping", - "hash": "QmcDSLssuPYMwC71biMiCvWMGg6rBzk8TKf4UJN64M68YK", + "hash": "QmcE2awGbLPqCDH5PbqBTyqeoYpBAFNSnYK2Kw8azQkqXH", "name": "go-libp2p-secio", - "version": "2.0.21" + "version": "2.0.22" }, { "author": "whyrusleeping", - "hash": "QmUymf8fJtideyv3z727BcZUifGBjMZMpCJqu3Gxk5aRUk", + "hash": "QmQAGG1zxfePqj2t7bLxyN8AFccZ889DDR9Gn8kVLDrGZo", "name": "go-libp2p-peerstore", - "version": "2.0.10" + "version": "2.0.11" }, { "author": "whyrusleeping", - "hash": "QmZJ5hXLAz8vrZ4cw4EFk355pqMuxWTZQ5Hs2xhYGjdvGr", + "hash": "QmdQx4ZhKGdv9TvpCFpMxFzjTQFHRmFqjBxkRVwzT1JNes", "name": "go-libp2p-transport", - "version": "3.0.18" + "version": "3.0.19" }, { "author": "whyrusleeping", - "hash": "QmetLkVrNTi6QGfv577kpKPVBXJFshKZrP87C7xHJp7SCX", + "hash": "QmUmXYaTmtvefgKoVA7hY3uUN8D1CiqxufigSyWuWwQjXy", "name": "go-tcp-transport", - "version": "2.0.19" + "version": "2.0.20" }, { "author": "whyrusleeping", @@ -98,9 +98,9 @@ }, { "author": "whyrusleeping", - "hash": "QmRKbEchaYADxSCyyjhDh4cTrUby8ftXUb8MRLBTHQYupw", + "hash": "QmenvQQy4bFGSiHJUGupVmCRHfetg5rH3vTp9Z2f6v2KXR", "name": "go-libp2p-net", - "version": "3.0.19" + "version": "3.0.20" }, { "author": "whyrusleeping", @@ -110,15 +110,15 @@ }, { "author": "whyrusleeping", - "hash": "QmVrjR2KMe57y4YyfHdYa3yKD278gN8W7CTiqSuYmxjA7F", + "hash": "QmahxMNoNuSsgQefo9rkpcfRFmQrMN6Q99aztKXf63K7YJ", "name": "go-libp2p-host", - "version": "3.0.18" + "version": "3.0.19" }, { "author": "whyrusleeping", - "hash": "QmcYC4ayKi7bq8xecEZxHVEuTL6HREZWTTErrSRd1S3Spz", + "hash": "QmQrYHkcGprZBUFnRigeiZFkaFDBHtmRhDdPpSiiUTRNwv", "name": "go-libp2p-swarm", - "version": "3.0.25" + "version": "3.0.26" }, { "author": "whyrusleeping", @@ -128,15 +128,15 @@ }, { "author": "whyrusleeping", - "hash": "QmVPQuFZTw2yVm5Uu1MGofaNBnqaySQddLHKCFjWyoKGhv", + "hash": "QmPphQ7HMN49UGAwX3VhxibKkNP3fjLDdkcM5SjhksZSNq", "name": "go-libp2p-netutil", - "version": "0.4.15" + "version": "0.4.16" }, { "author": "whyrusleeping", - "hash": "QmVzYgadebqVb6iCVvjGfgZ7HB9zoCnDgBjMqFLD5aMrFX", + "hash": "QmWnd4JYBouYJMuW9Gt5a5hFpW2whwgAaxyxBWKXZXzVgy", "name": "go-libp2p-blankhost", - "version": "0.3.18" + "version": "0.3.19" }, { "author": "whyrusleeping", @@ -164,9 +164,9 @@ }, { "author": "vyzo", - "hash": "QmddZ5gv3Gkicoqh5NDfHGjpij6zw92pQjvpa181yfnXm2", + "hash": "QmWxCQ66YzAb24Py5D3qeDJcVTF4aW9AeAmppQ7FhQZxgZ", "name": "go-libp2p-circuit", - "version": "2.3.5" + "version": "2.3.6" }, { "author": "lgierth", @@ -176,9 +176,9 @@ }, { "author": "why", - "hash": "QmR8DgkC3Xnc1TnfH1DvZtLRzPKJBrWfeDKseeXnUY6CN5", + "hash": "QmQSucBpqUVQ5Q1stDmm2Bon4Tq4KNhNXuVmLMraARoUoh", "name": "go-libp2p-interface-connmgr", - "version": "0.0.24" + "version": "0.0.25" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmTt8BCNdw8uowByuFbUpphdrkTKVLJEcMNgewoYzb4K1c", + "hash": "QmP2qH43GBjhUo1XPrn5k6y4fvahSrLwNPf4ddjitxo2Kz", "name": "go-ws-transport", - "version": "2.0.18" + "version": "2.0.19" }, { "author": "stebalien", - "hash": "QmZdxWGQU7PdcqcRExtzXZvcbxqM9CxrFW8HuS33XHm9yn", + "hash": "QmPcoYfH3kjZmasoVuVWrjPvdX5zwG4U8iXgEgZJD14dBf", "name": "go-conn-security-multistream", - "version": "0.1.18" + "version": "0.1.19" }, { "author": "Stebalien", - "hash": "QmbNjbKRJKbek3jPV6rpbGEtrZd84cDxjpBtsiGYa9Z5Do", + "hash": "QmbyjEKtxXmZdiByBiNhfayzEuEPPBvuD2dLpHky8cHUvy", "name": "go-conn-security", - "version": "0.1.19" + "version": "0.1.20" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "QmbrgvQMRBhWJtG9pjerPb3V9xb3JzCDR6m1tp6J44iynL", + "hash": "QmPbKqnriyf7c2Kr5NHR2tw52SkWqdb1uHxLWT3h3qBmeS", "name": "go-libp2p-transport-upgrader", - "version": "0.1.19" + "version": "0.1.20" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -229,15 +229,20 @@ }, { "author": "vyzo", - "hash": "QmT8eNT96scr6wcrARzyxrCcsBAPKn9GkEx2TeXZniHiMP", + "hash": "QmejQHZsodEXxdeFQazcWsi4Dkmp4mX7QEZrWXHoVR5EtK", "name": "go-libp2p-discovery", - "version": "1.0.2" + "version": "1.0.3" }, { "author": "vyzo", - "hash": "QmU2XQcwPxikg1jZqDBMFgLQVan7zjT2HtQWrWui3vbUVS", + "hash": "QmYdJ7pcu5fbwBeiue9TPaNie3QwPaGsh6sPzZiz9gMgLQ", "name": "go-libp2p-autonat", - "version": "1.0.2" + "version": "1.0.3" + }, + { + "hash": "QmZBH87CAPFHcc7cYmBqeSQ98zQ3SX9KUxiYgzPmLWNVKz", + "name": "go-libp2p-routing", + "version": "2.7.5" } ], "gxVersion": "0.4.0", @@ -245,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.26" + "version": "6.0.27" } From e587133f9a6e1ec4dc5216605edefa5ea1bed4cb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 12 Nov 2018 22:28:12 +0000 Subject: [PATCH 0839/3965] pstoreds: migrate from strict TTL to GC-based expiry. Each entry in the address book KV store now represents a full peer with all its addresses. Expiries and TTLs are kept inline. Records are serialised with Protobuf. Housekeeping is performed on retrieve, and via a periodic GC routine. The architecture mimics a write-through cache, although not strictly. --- p2p/host/peerstore/pb/Makefile | 11 + p2p/host/peerstore/pb/pstore.pb.go | 936 +++++++++++++++++++++++ p2p/host/peerstore/pb/pstore.proto | 32 + p2p/host/peerstore/pb/pstorepb_test.go | 129 ++++ p2p/host/peerstore/pstoreds/addr_book.go | 693 ++++++++--------- p2p/host/peerstore/pstoreds/ds_test.go | 9 +- p2p/host/peerstore/pstoreds/peerstore.go | 7 +- 7 files changed, 1457 insertions(+), 360 deletions(-) create mode 100644 p2p/host/peerstore/pb/Makefile create mode 100644 p2p/host/peerstore/pb/pstore.pb.go create mode 100644 p2p/host/peerstore/pb/pstore.proto create mode 100644 p2p/host/peerstore/pb/pstorepb_test.go diff --git a/p2p/host/peerstore/pb/Makefile b/p2p/host/peerstore/pb/Makefile new file mode 100644 index 0000000000..eb14b5768a --- /dev/null +++ b/p2p/host/peerstore/pb/Makefile @@ -0,0 +1,11 @@ +PB = $(wildcard *.proto) +GO = $(PB:.proto=.pb.go) + +all: $(GO) + +%.pb.go: %.proto + protoc --proto_path=$(GOPATH)/src:. --gogofast_out=. $< + +clean: + rm -f *.pb.go + rm -f *.go diff --git a/p2p/host/peerstore/pb/pstore.pb.go b/p2p/host/peerstore/pb/pstore.pb.go new file mode 100644 index 0000000000..cf4b04b1d9 --- /dev/null +++ b/p2p/host/peerstore/pb/pstore.pb.go @@ -0,0 +1,936 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: pstore.proto + +package pstore_pb + +import proto "github.com/gogo/protobuf/proto" +import fmt "fmt" +import math "math" +import _ "github.com/gogo/protobuf/gogoproto" +import _ "github.com/golang/protobuf/ptypes/duration" +import _ "github.com/golang/protobuf/ptypes/timestamp" + +import time "time" + +import github_com_gogo_protobuf_types "github.com/gogo/protobuf/types" + +import io "io" + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf +var _ = time.Kitchen + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + +// AddrBookRecord represents a record for a peer in the address book. +type AddrBookRecord struct { + // The peer ID. + Id []byte `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` + // The timestamp where purging needs to happen. + NextVisit *time.Time `protobuf:"bytes,2,opt,name=nextVisit,stdtime" json:"nextVisit,omitempty"` + // The multiaddresses. + Addrs map[string]*AddrBookRecord_AddrEntry `protobuf:"bytes,3,rep,name=addrs" json:"addrs,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *AddrBookRecord) Reset() { *m = AddrBookRecord{} } +func (m *AddrBookRecord) String() string { return proto.CompactTextString(m) } +func (*AddrBookRecord) ProtoMessage() {} +func (*AddrBookRecord) Descriptor() ([]byte, []int) { + return fileDescriptor_pstore_842456c80d89ffef, []int{0} +} +func (m *AddrBookRecord) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *AddrBookRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_AddrBookRecord.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *AddrBookRecord) XXX_Merge(src proto.Message) { + xxx_messageInfo_AddrBookRecord.Merge(dst, src) +} +func (m *AddrBookRecord) XXX_Size() int { + return m.Size() +} +func (m *AddrBookRecord) XXX_DiscardUnknown() { + xxx_messageInfo_AddrBookRecord.DiscardUnknown(m) +} + +var xxx_messageInfo_AddrBookRecord proto.InternalMessageInfo + +func (m *AddrBookRecord) GetId() []byte { + if m != nil { + return m.Id + } + return nil +} + +func (m *AddrBookRecord) GetNextVisit() *time.Time { + if m != nil { + return m.NextVisit + } + return nil +} + +func (m *AddrBookRecord) GetAddrs() map[string]*AddrBookRecord_AddrEntry { + if m != nil { + return m.Addrs + } + return nil +} + +// AddrEntry represents a single multiaddress. +type AddrBookRecord_AddrEntry struct { + // The point in time when this address expires. + Expiry *time.Time `protobuf:"bytes,2,opt,name=expiry,stdtime" json:"expiry,omitempty"` + // The original TTL of this address. + Ttl *time.Duration `protobuf:"bytes,3,opt,name=ttl,stdduration" json:"ttl,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *AddrBookRecord_AddrEntry) Reset() { *m = AddrBookRecord_AddrEntry{} } +func (m *AddrBookRecord_AddrEntry) String() string { return proto.CompactTextString(m) } +func (*AddrBookRecord_AddrEntry) ProtoMessage() {} +func (*AddrBookRecord_AddrEntry) Descriptor() ([]byte, []int) { + return fileDescriptor_pstore_842456c80d89ffef, []int{0, 1} +} +func (m *AddrBookRecord_AddrEntry) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *AddrBookRecord_AddrEntry) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_AddrBookRecord_AddrEntry.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (dst *AddrBookRecord_AddrEntry) XXX_Merge(src proto.Message) { + xxx_messageInfo_AddrBookRecord_AddrEntry.Merge(dst, src) +} +func (m *AddrBookRecord_AddrEntry) XXX_Size() int { + return m.Size() +} +func (m *AddrBookRecord_AddrEntry) XXX_DiscardUnknown() { + xxx_messageInfo_AddrBookRecord_AddrEntry.DiscardUnknown(m) +} + +var xxx_messageInfo_AddrBookRecord_AddrEntry proto.InternalMessageInfo + +func (m *AddrBookRecord_AddrEntry) GetExpiry() *time.Time { + if m != nil { + return m.Expiry + } + return nil +} + +func (m *AddrBookRecord_AddrEntry) GetTtl() *time.Duration { + if m != nil { + return m.Ttl + } + return nil +} + +func init() { + proto.RegisterType((*AddrBookRecord)(nil), "pstore.pb.AddrBookRecord") + proto.RegisterMapType((map[string]*AddrBookRecord_AddrEntry)(nil), "pstore.pb.AddrBookRecord.AddrsEntry") + proto.RegisterType((*AddrBookRecord_AddrEntry)(nil), "pstore.pb.AddrBookRecord.AddrEntry") +} +func (m *AddrBookRecord) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *AddrBookRecord) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Id) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintPstore(dAtA, i, uint64(len(m.Id))) + i += copy(dAtA[i:], m.Id) + } + if m.NextVisit != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintPstore(dAtA, i, uint64(github_com_gogo_protobuf_types.SizeOfStdTime(*m.NextVisit))) + n1, err := github_com_gogo_protobuf_types.StdTimeMarshalTo(*m.NextVisit, dAtA[i:]) + if err != nil { + return 0, err + } + i += n1 + } + if len(m.Addrs) > 0 { + for k, _ := range m.Addrs { + dAtA[i] = 0x1a + i++ + v := m.Addrs[k] + msgSize := 0 + if v != nil { + msgSize = v.Size() + msgSize += 1 + sovPstore(uint64(msgSize)) + } + mapSize := 1 + len(k) + sovPstore(uint64(len(k))) + msgSize + i = encodeVarintPstore(dAtA, i, uint64(mapSize)) + dAtA[i] = 0xa + i++ + i = encodeVarintPstore(dAtA, i, uint64(len(k))) + i += copy(dAtA[i:], k) + if v != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintPstore(dAtA, i, uint64(v.Size())) + n2, err := v.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n2 + } + } + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func (m *AddrBookRecord_AddrEntry) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *AddrBookRecord_AddrEntry) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Expiry != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintPstore(dAtA, i, uint64(github_com_gogo_protobuf_types.SizeOfStdTime(*m.Expiry))) + n3, err := github_com_gogo_protobuf_types.StdTimeMarshalTo(*m.Expiry, dAtA[i:]) + if err != nil { + return 0, err + } + i += n3 + } + if m.Ttl != nil { + dAtA[i] = 0x1a + i++ + i = encodeVarintPstore(dAtA, i, uint64(github_com_gogo_protobuf_types.SizeOfStdDuration(*m.Ttl))) + n4, err := github_com_gogo_protobuf_types.StdDurationMarshalTo(*m.Ttl, dAtA[i:]) + if err != nil { + return 0, err + } + i += n4 + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil +} + +func encodeVarintPstore(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func NewPopulatedAddrBookRecord(r randyPstore, easy bool) *AddrBookRecord { + this := &AddrBookRecord{} + v1 := r.Intn(100) + this.Id = make([]byte, v1) + for i := 0; i < v1; i++ { + this.Id[i] = byte(r.Intn(256)) + } + if r.Intn(10) != 0 { + this.NextVisit = github_com_gogo_protobuf_types.NewPopulatedStdTime(r, easy) + } + if r.Intn(10) != 0 { + v2 := r.Intn(10) + this.Addrs = make(map[string]*AddrBookRecord_AddrEntry) + for i := 0; i < v2; i++ { + this.Addrs[randStringPstore(r)] = NewPopulatedAddrBookRecord_AddrEntry(r, easy) + } + } + if !easy && r.Intn(10) != 0 { + this.XXX_unrecognized = randUnrecognizedPstore(r, 4) + } + return this +} + +func NewPopulatedAddrBookRecord_AddrEntry(r randyPstore, easy bool) *AddrBookRecord_AddrEntry { + this := &AddrBookRecord_AddrEntry{} + if r.Intn(10) != 0 { + this.Expiry = github_com_gogo_protobuf_types.NewPopulatedStdTime(r, easy) + } + if r.Intn(10) != 0 { + this.Ttl = github_com_gogo_protobuf_types.NewPopulatedStdDuration(r, easy) + } + if !easy && r.Intn(10) != 0 { + this.XXX_unrecognized = randUnrecognizedPstore(r, 4) + } + return this +} + +type randyPstore interface { + Float32() float32 + Float64() float64 + Int63() int64 + Int31() int32 + Uint32() uint32 + Intn(n int) int +} + +func randUTF8RunePstore(r randyPstore) rune { + ru := r.Intn(62) + if ru < 10 { + return rune(ru + 48) + } else if ru < 36 { + return rune(ru + 55) + } + return rune(ru + 61) +} +func randStringPstore(r randyPstore) string { + v3 := r.Intn(100) + tmps := make([]rune, v3) + for i := 0; i < v3; i++ { + tmps[i] = randUTF8RunePstore(r) + } + return string(tmps) +} +func randUnrecognizedPstore(r randyPstore, maxFieldNumber int) (dAtA []byte) { + l := r.Intn(5) + for i := 0; i < l; i++ { + wire := r.Intn(4) + if wire == 3 { + wire = 5 + } + fieldNumber := maxFieldNumber + r.Intn(100) + dAtA = randFieldPstore(dAtA, r, fieldNumber, wire) + } + return dAtA +} +func randFieldPstore(dAtA []byte, r randyPstore, fieldNumber int, wire int) []byte { + key := uint32(fieldNumber)<<3 | uint32(wire) + switch wire { + case 0: + dAtA = encodeVarintPopulatePstore(dAtA, uint64(key)) + v4 := r.Int63() + if r.Intn(2) == 0 { + v4 *= -1 + } + dAtA = encodeVarintPopulatePstore(dAtA, uint64(v4)) + case 1: + dAtA = encodeVarintPopulatePstore(dAtA, uint64(key)) + dAtA = append(dAtA, byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256))) + case 2: + dAtA = encodeVarintPopulatePstore(dAtA, uint64(key)) + ll := r.Intn(100) + dAtA = encodeVarintPopulatePstore(dAtA, uint64(ll)) + for j := 0; j < ll; j++ { + dAtA = append(dAtA, byte(r.Intn(256))) + } + default: + dAtA = encodeVarintPopulatePstore(dAtA, uint64(key)) + dAtA = append(dAtA, byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256))) + } + return dAtA +} +func encodeVarintPopulatePstore(dAtA []byte, v uint64) []byte { + for v >= 1<<7 { + dAtA = append(dAtA, uint8(uint64(v)&0x7f|0x80)) + v >>= 7 + } + dAtA = append(dAtA, uint8(v)) + return dAtA +} +func (m *AddrBookRecord) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + l = len(m.Id) + if l > 0 { + n += 1 + l + sovPstore(uint64(l)) + } + if m.NextVisit != nil { + l = github_com_gogo_protobuf_types.SizeOfStdTime(*m.NextVisit) + n += 1 + l + sovPstore(uint64(l)) + } + if len(m.Addrs) > 0 { + for k, v := range m.Addrs { + _ = k + _ = v + l = 0 + if v != nil { + l = v.Size() + l += 1 + sovPstore(uint64(l)) + } + mapEntrySize := 1 + len(k) + sovPstore(uint64(len(k))) + l + n += mapEntrySize + 1 + sovPstore(uint64(mapEntrySize)) + } + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *AddrBookRecord_AddrEntry) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.Expiry != nil { + l = github_com_gogo_protobuf_types.SizeOfStdTime(*m.Expiry) + n += 1 + l + sovPstore(uint64(l)) + } + if m.Ttl != nil { + l = github_com_gogo_protobuf_types.SizeOfStdDuration(*m.Ttl) + n += 1 + l + sovPstore(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func sovPstore(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozPstore(x uint64) (n int) { + return sovPstore(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: AddrBookRecord: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: AddrBookRecord: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Id", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPstore + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Id = append(m.Id[:0], dAtA[iNdEx:postIndex]...) + if m.Id == nil { + m.Id = []byte{} + } + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field NextVisit", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthPstore + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.NextVisit == nil { + m.NextVisit = new(time.Time) + } + if err := github_com_gogo_protobuf_types.StdTimeUnmarshal(m.NextVisit, dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Addrs", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthPstore + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.Addrs == nil { + m.Addrs = make(map[string]*AddrBookRecord_AddrEntry) + } + var mapkey string + var mapvalue *AddrBookRecord_AddrEntry + for iNdEx < postIndex { + entryPreIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + if fieldNum == 1 { + var stringLenmapkey uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLenmapkey |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLenmapkey := int(stringLenmapkey) + if intStringLenmapkey < 0 { + return ErrInvalidLengthPstore + } + postStringIndexmapkey := iNdEx + intStringLenmapkey + if postStringIndexmapkey > l { + return io.ErrUnexpectedEOF + } + mapkey = string(dAtA[iNdEx:postStringIndexmapkey]) + iNdEx = postStringIndexmapkey + } else if fieldNum == 2 { + var mapmsglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + mapmsglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if mapmsglen < 0 { + return ErrInvalidLengthPstore + } + postmsgIndex := iNdEx + mapmsglen + if mapmsglen < 0 { + return ErrInvalidLengthPstore + } + if postmsgIndex > l { + return io.ErrUnexpectedEOF + } + mapvalue = &AddrBookRecord_AddrEntry{} + if err := mapvalue.Unmarshal(dAtA[iNdEx:postmsgIndex]); err != nil { + return err + } + iNdEx = postmsgIndex + } else { + iNdEx = entryPreIndex + skippy, err := skipPstore(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPstore + } + if (iNdEx + skippy) > postIndex { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + m.Addrs[mapkey] = mapvalue + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipPstore(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPstore + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: AddrEntry: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: AddrEntry: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Expiry", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthPstore + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.Expiry == nil { + m.Expiry = new(time.Time) + } + if err := github_com_gogo_protobuf_types.StdTimeUnmarshal(m.Expiry, dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Ttl", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthPstore + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.Ttl == nil { + m.Ttl = new(time.Duration) + } + if err := github_com_gogo_protobuf_types.StdDurationUnmarshal(m.Ttl, dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipPstore(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPstore + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipPstore(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPstore + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPstore + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPstore + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + iNdEx += length + if length < 0 { + return 0, ErrInvalidLengthPstore + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPstore + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipPstore(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthPstore = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowPstore = fmt.Errorf("proto: integer overflow") +) + +func init() { proto.RegisterFile("pstore.proto", fileDescriptor_pstore_842456c80d89ffef) } + +var fileDescriptor_pstore_842456c80d89ffef = []byte{ + // 330 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x91, 0xbf, 0x4e, 0xc3, 0x30, + 0x10, 0xc6, 0xe5, 0xa4, 0xad, 0x14, 0xb7, 0xaa, 0x90, 0xc5, 0x10, 0x32, 0xb8, 0x15, 0x30, 0x74, + 0xc1, 0x15, 0x65, 0x29, 0x1d, 0x90, 0x88, 0xe0, 0x05, 0x22, 0xc4, 0xc6, 0x90, 0xd4, 0x26, 0x58, + 0xfd, 0x73, 0x91, 0xe3, 0xa0, 0xf6, 0x2d, 0x18, 0x79, 0x1c, 0x46, 0x46, 0xde, 0x00, 0xc8, 0x3b, + 0x20, 0x31, 0xa2, 0x38, 0x69, 0x23, 0x40, 0x42, 0x6c, 0xf7, 0xdd, 0x7d, 0xbf, 0xf3, 0x77, 0x32, + 0xee, 0x24, 0xa9, 0x06, 0x25, 0x58, 0xa2, 0x40, 0x03, 0x71, 0x36, 0x2a, 0xf2, 0x7a, 0x31, 0x40, + 0x3c, 0x17, 0x43, 0x33, 0x88, 0xb2, 0xdb, 0xa1, 0x96, 0x0b, 0x91, 0xea, 0x70, 0x91, 0x94, 0x5e, + 0x8f, 0xfe, 0x34, 0xf0, 0x4c, 0x85, 0x5a, 0xc2, 0xb2, 0x9a, 0x1f, 0xc5, 0x52, 0xdf, 0x65, 0x11, + 0x9b, 0xc2, 0x62, 0x18, 0x43, 0x0c, 0xb5, 0xb1, 0x50, 0x46, 0x98, 0xaa, 0xb4, 0xef, 0x7f, 0x58, + 0xb8, 0x7b, 0xce, 0xb9, 0xf2, 0x01, 0x66, 0x81, 0x98, 0x82, 0xe2, 0xa4, 0x8b, 0x2d, 0xc9, 0x5d, + 0xd4, 0x47, 0x83, 0x4e, 0x60, 0x49, 0x4e, 0xce, 0xb0, 0xb3, 0x14, 0x2b, 0x7d, 0x2d, 0x53, 0xa9, + 0x5d, 0xab, 0x8f, 0x06, 0xed, 0x91, 0xc7, 0xca, 0x14, 0x6c, 0xb3, 0x9c, 0x5d, 0x6d, 0x62, 0xfa, + 0x8d, 0x87, 0xd7, 0x1e, 0x0a, 0x6a, 0x84, 0x4c, 0x70, 0x33, 0xe4, 0x5c, 0xa5, 0xae, 0xdd, 0xb7, + 0x07, 0xed, 0xd1, 0x21, 0xdb, 0x5e, 0xcb, 0xbe, 0xbf, 0x6c, 0x64, 0x7a, 0xb9, 0xd4, 0x6a, 0x1d, + 0x94, 0x88, 0x77, 0x83, 0x71, 0xdd, 0x24, 0x3b, 0xd8, 0x9e, 0x89, 0xb5, 0x89, 0xe6, 0x04, 0x45, + 0x49, 0x4e, 0x71, 0xf3, 0x3e, 0x9c, 0x67, 0xa2, 0xca, 0x75, 0xf0, 0xf7, 0xee, 0x6a, 0xb5, 0x21, + 0x26, 0xd6, 0x18, 0x79, 0x2b, 0xec, 0x6c, 0xfb, 0x64, 0x8c, 0x5b, 0x62, 0x95, 0x48, 0xb5, 0xfe, + 0xf7, 0x91, 0x95, 0x9f, 0x1c, 0x63, 0x5b, 0xeb, 0xb9, 0x6b, 0x1b, 0x6c, 0xef, 0x17, 0x76, 0x51, + 0xfd, 0x90, 0xdf, 0x78, 0x2c, 0xa8, 0xc2, 0xeb, 0xef, 0x7e, 0xbe, 0x53, 0xf4, 0x94, 0x53, 0xf4, + 0x9c, 0x53, 0xf4, 0x92, 0x53, 0xf4, 0x96, 0x53, 0x14, 0xb5, 0x0c, 0x73, 0xf2, 0x15, 0x00, 0x00, + 0xff, 0xff, 0x0c, 0x8d, 0xe8, 0x16, 0x1f, 0x02, 0x00, 0x00, +} diff --git a/p2p/host/peerstore/pb/pstore.proto b/p2p/host/peerstore/pb/pstore.proto new file mode 100644 index 0000000000..2a768462cd --- /dev/null +++ b/p2p/host/peerstore/pb/pstore.proto @@ -0,0 +1,32 @@ +syntax = "proto3"; +package pstore.pb; + +import "google/protobuf/timestamp.proto"; +import "google/protobuf/duration.proto"; +import "github.com/gogo/protobuf/gogoproto/gogo.proto"; + +option (gogoproto.marshaler_all) = true; +option (gogoproto.unmarshaler_all) = true; +option (gogoproto.benchgen_all) = true; +option (gogoproto.populate_all) = true; + +// AddrBookRecord represents a record for a peer in the address book. +message AddrBookRecord { + // The peer ID. + bytes id = 1; + + // The timestamp where purging needs to happen. + google.protobuf.Timestamp nextVisit = 2 [(gogoproto.stdtime) = true]; + + // The multiaddresses. + map addrs = 3; + + // AddrEntry represents a single multiaddress. + message AddrEntry { + // The point in time when this address expires. + google.protobuf.Timestamp expiry = 2 [(gogoproto.stdtime) = true]; + + // The original TTL of this address. + google.protobuf.Duration ttl = 3 [(gogoproto.stdduration) = true]; + } +} diff --git a/p2p/host/peerstore/pb/pstorepb_test.go b/p2p/host/peerstore/pb/pstorepb_test.go new file mode 100644 index 0000000000..6d8cbf3270 --- /dev/null +++ b/p2p/host/peerstore/pb/pstorepb_test.go @@ -0,0 +1,129 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: pstore.proto + +package pstore_pb + +import testing "testing" +import math_rand "math/rand" +import github_com_gogo_protobuf_proto "github.com/gogo/protobuf/proto" +import proto "github.com/gogo/protobuf/proto" +import fmt "fmt" +import math "math" +import _ "github.com/gogo/protobuf/gogoproto" +import _ "github.com/golang/protobuf/ptypes/duration" +import _ "github.com/golang/protobuf/ptypes/timestamp" + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +func BenchmarkAddrBookRecordProtoMarshal(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + pops := make([]*AddrBookRecord, 10000) + for i := 0; i < 10000; i++ { + pops[i] = NewPopulatedAddrBookRecord(popr, false) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + dAtA, err := github_com_gogo_protobuf_proto.Marshal(pops[i%10000]) + if err != nil { + panic(err) + } + total += len(dAtA) + } + b.SetBytes(int64(total / b.N)) +} + +func BenchmarkAddrBookRecordProtoUnmarshal(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + datas := make([][]byte, 10000) + for i := 0; i < 10000; i++ { + dAtA, err := github_com_gogo_protobuf_proto.Marshal(NewPopulatedAddrBookRecord(popr, false)) + if err != nil { + panic(err) + } + datas[i] = dAtA + } + msg := &AddrBookRecord{} + b.ResetTimer() + for i := 0; i < b.N; i++ { + total += len(datas[i%10000]) + if err := github_com_gogo_protobuf_proto.Unmarshal(datas[i%10000], msg); err != nil { + panic(err) + } + } + b.SetBytes(int64(total / b.N)) +} + +func BenchmarkAddrBookRecord_AddrEntryProtoMarshal(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + pops := make([]*AddrBookRecord_AddrEntry, 10000) + for i := 0; i < 10000; i++ { + pops[i] = NewPopulatedAddrBookRecord_AddrEntry(popr, false) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + dAtA, err := github_com_gogo_protobuf_proto.Marshal(pops[i%10000]) + if err != nil { + panic(err) + } + total += len(dAtA) + } + b.SetBytes(int64(total / b.N)) +} + +func BenchmarkAddrBookRecord_AddrEntryProtoUnmarshal(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + datas := make([][]byte, 10000) + for i := 0; i < 10000; i++ { + dAtA, err := github_com_gogo_protobuf_proto.Marshal(NewPopulatedAddrBookRecord_AddrEntry(popr, false)) + if err != nil { + panic(err) + } + datas[i] = dAtA + } + msg := &AddrBookRecord_AddrEntry{} + b.ResetTimer() + for i := 0; i < b.N; i++ { + total += len(datas[i%10000]) + if err := github_com_gogo_protobuf_proto.Unmarshal(datas[i%10000], msg); err != nil { + panic(err) + } + } + b.SetBytes(int64(total / b.N)) +} + +func BenchmarkAddrBookRecordSize(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + pops := make([]*AddrBookRecord, 1000) + for i := 0; i < 1000; i++ { + pops[i] = NewPopulatedAddrBookRecord(popr, false) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + total += pops[i%1000].Size() + } + b.SetBytes(int64(total / b.N)) +} + +func BenchmarkAddrBookRecord_AddrEntrySize(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + pops := make([]*AddrBookRecord_AddrEntry, 1000) + for i := 0; i < 1000; i++ { + pops[i] = NewPopulatedAddrBookRecord_AddrEntry(popr, false) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + total += pops[i%1000].Size() + } + b.SetBytes(int64(total / b.N)) +} + +//These tests are generated by github.com/gogo/protobuf/plugin/testgen diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 9e92336c45..7eb08b7831 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -2,50 +2,25 @@ package pstoreds import ( "context" - "encoding/binary" - "errors" + "fmt" + "sync" "time" - lru "github.com/hashicorp/golang-lru" - base32 "github.com/whyrusleeping/base32" - ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - ma "github.com/multiformats/go-multiaddr" - mh "github.com/multiformats/go-multihash" - + pool "github.com/libp2p/go-buffer-pool" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + pb "github.com/libp2p/go-libp2p-peerstore/pb" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" -) -var ( - log = logging.Logger("peerstore/ds") - // The maximum representable value in time.Time is time.Unix(1<<63-62135596801, 999999999). - // But it's too brittle and implementation-dependent, so we prefer to use 1<<62, which is in the - // year 146138514283. We're safe. - maxTime = time.Unix(1<<62, 0) - - ErrTTLDatastore = errors.New("datastore must provide TTL support") + lru "github.com/hashicorp/golang-lru" + ma "github.com/multiformats/go-multiaddr" + b32 "github.com/whyrusleeping/base32" ) -// Peer addresses are stored under the following db key pattern: -// /peers/addr// -var abBase = ds.NewKey("/peers/addrs") - -var _ pstore.AddrBook = (*dsAddrBook)(nil) - -// dsAddrBook is an address book backed by a Datastore with both an -// in-memory TTL manager and an in-memory address stream manager. -type dsAddrBook struct { - cache cache - ds ds.TxnDatastore - subsManager *pstoremem.AddrSubManager - writeRetries int -} - type ttlWriteMode int const ( @@ -53,347 +28,375 @@ const ( ttlExtend ) -type cacheEntry struct { - expiration time.Time - addrs []ma.Multiaddr -} +var ( + log = logging.Logger("peerstore/ds") + // Peer addresses are stored under the following db key pattern: + // /peers/addr// + addrBookBase = ds.NewKey("/peers/addrs") +) -type addrRecord struct { - ttl time.Duration - addr ma.Multiaddr -} +// addrsRecord decorates the AddrBookRecord with locks and metadata. +type addrsRecord struct { + sync.RWMutex + *pb.AddrBookRecord -func (ar *addrRecord) MarshalBinary() ([]byte, error) { - ttlB := make([]byte, 8) - binary.LittleEndian.PutUint64(ttlB, uint64(ar.ttl)) - return append(ttlB, ar.addr.Bytes()...), nil + delete bool } -func (ar *addrRecord) UnmarshalBinary(b []byte) error { - ar.ttl = time.Duration(binary.LittleEndian.Uint64(b)) - // this had been serialized by us, no need to check for errors - ar.addr, _ = ma.NewMultiaddrBytes(b[8:]) - return nil +// FlushInTxn flushes the record to the datastore by calling ds.Put, unless the record is +// marked for deletion, in which case the deletion is executed. +func (r *addrsRecord) FlushInTxn(txn ds.Txn) (err error) { + key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id))) + if r.delete { + return txn.Delete(key) + } + data := pool.Get(r.Size()) + defer pool.Put(data) + i, err := r.MarshalTo(data) + if err != nil { + return err + } + return txn.Put(key, data[:i]) } -// NewAddrBook initializes a new address book given a -// Datastore instance, a context for managing the TTL manager, -// and the interval at which the TTL manager should sweep the Datastore. -func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (*dsAddrBook, error) { - if _, ok := store.(ds.TTLDatastore); !ok { - return nil, ErrTTLDatastore +// Flush creates a ds.Txn, and calls FlushInTxn with it. +func (r *addrsRecord) Flush(ds ds.TxnDatastore) (err error) { + txn, err := ds.NewTransaction(false) + if err != nil { + return err } + defer txn.Discard() - var ( - cache cache = &noopCache{} - err error - ) + if err = r.FlushInTxn(txn); err != nil { + return err + } + return txn.Commit() +} - if opts.CacheSize > 0 { - if cache, err = lru.NewARC(int(opts.CacheSize)); err != nil { - return nil, err +// Refresh is called on in-memory entries to perform housekeeping. Refresh does the following: +// * removes all expired addresses. +// * recalculates the date in which the record needs to be revisited (earliest expiration of survivors). +// * marks the record for deletion if no addresses are left. +// +// A `true` value of `force` tells us to proceed with housekeeping even if the `NextVisit` date has not arrived. +// +// Refresh is called in several occasions: +// * with force=false, when accessing and loading an entry, or when performing GC. +// * with force=true, after an entry has been modified (e.g. addresses have been added or removed, +// TTLs updated, etc.) +func (r *addrsRecord) Refresh(force bool) (chgd bool) { + if len(r.Addrs) == 0 { + r.delete = true + r.NextVisit = nil + return true + } + + now := time.Now() + if !force && r.NextVisit != nil && !r.NextVisit.IsZero() && now.Before(*r.NextVisit) { + // no expired entries to purge, and no forced housekeeping. + return false + } + + // nv stores the next visit for surviving addresses following the purge + var nv time.Time + for addr, entry := range r.Addrs { + if entry.Expiry == nil { + continue + } + if nv.IsZero() { + nv = *entry.Expiry + } + if now.After(*entry.Expiry) { + // this entry is expired; remove it. + delete(r.Addrs, addr) + chgd = true + } else if nv.After(*entry.Expiry) { + // keep track of the earliest expiry across survivors. + nv = *entry.Expiry } } - mgr := &dsAddrBook{ - cache: cache, - ds: store, - subsManager: pstoremem.NewAddrSubManager(), - writeRetries: int(opts.WriteRetries), + if len(r.Addrs) == 0 { + r.delete = true + r.NextVisit = nil + return true } - return mgr, nil + + chgd = chgd || r.NextVisit == nil || nv != *r.NextVisit + r.NextVisit = &nv + return chgd } -func keysAndAddrs(p peer.ID, addrs []ma.Multiaddr) ([]ds.Key, []ma.Multiaddr, error) { - var ( - keys = make([]ds.Key, len(addrs)) - clean = make([]ma.Multiaddr, len(addrs)) - parentKey = abBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))) - i = 0 - ) +// dsAddrBook is an address book backed by a Datastore with a GC-like procedure +// to purge expired entries. It uses an in-memory address stream manager. +type dsAddrBook struct { + ctx context.Context + gcInterval time.Duration + gcMaxPurgePerCycle int - for _, addr := range addrs { - if addr == nil { - continue - } + cache cache + ds ds.TxnDatastore + subsManager *pstoremem.AddrSubManager + + flushJobCh chan *addrsRecord + cancelFn func() + closedCh chan struct{} +} + +var _ pstore.AddrBook = (*dsAddrBook)(nil) - hash, err := mh.Sum((addr).Bytes(), mh.MURMUR3, -1) - if err != nil { - return nil, nil, err +// NewAddrBook initializes a new address book given a +// Datastore instance, a context for managing the TTL manager, +// and the interval at which the TTL manager should sweep the Datastore. +func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab *dsAddrBook, err error) { + var cache cache = new(noopCache) + if opts.CacheSize > 0 { + if cache, err = lru.NewARC(int(opts.CacheSize)); err != nil { + return nil, err } - keys[i] = parentKey.ChildString(base32.RawStdEncoding.EncodeToString(hash)) - clean[i] = addr - i++ } - return keys[:i], clean[:i], nil -} + ctx, cancelFn := context.WithCancel(ctx) + mgr := &dsAddrBook{ + ctx: ctx, + cancelFn: cancelFn, + gcInterval: opts.GCInterval, + cache: cache, + ds: store, + subsManager: pstoremem.NewAddrSubManager(), + flushJobCh: make(chan *addrsRecord, 32), + closedCh: make(chan struct{}), + } -// AddAddr will add a new address if it's not already in the AddrBook. -func (mgr *dsAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - mgr.AddAddrs(p, []ma.Multiaddr{addr}, ttl) -} + go mgr.background() -// AddAddrs will add many new addresses if they're not already in the AddrBook. -func (mgr *dsAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - if ttl <= 0 { - return - } - mgr.setAddrs(p, addrs, ttl, ttlExtend) + return mgr, nil } -// SetAddr will add or update the TTL of an address in the AddrBook. -func (mgr *dsAddrBook) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { - addrs := []ma.Multiaddr{addr} - mgr.SetAddrs(p, addrs, ttl) +func (ab *dsAddrBook) Close() { + ab.cancelFn() + <-ab.closedCh } -// SetAddrs will add or update the TTLs of addresses in the AddrBook. -func (mgr *dsAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - if ttl <= 0 { - mgr.deleteAddrs(p, addrs) - return +func (ab *dsAddrBook) tryFlush(pr *addrsRecord) { + select { + case ab.flushJobCh <- pr: + default: + id, _ := peer.IDFromBytes(pr.Id) + log.Warningf("flush queue is full; could not flush peer %v", id) } - mgr.setAddrs(p, addrs, ttl, ttlOverride) } -func (mgr *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) error { - // Keys and cleaned up addresses. - keys, addrs, err := keysAndAddrs(p, addrs) - if err != nil { - return err - } - - mgr.cache.Remove(p) - // Attempt transactional KV deletion. - for i := 0; i < mgr.writeRetries; i++ { - if err = mgr.dbDelete(keys); err == nil { - break +func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrsRecord, err error) { + if e, ok := ab.cache.Get(id); ok { + pr = e.(*addrsRecord) + if pr.Refresh(false) && update { + ab.tryFlush(pr) } - log.Errorf("failed to delete addresses for peer %s: %s\n", p.Pretty(), err) + return pr, nil } + txn, err := ab.ds.NewTransaction(true) if err != nil { - log.Errorf("failed to avoid write conflict for peer %s after %d retries: %v\n", p.Pretty(), mgr.writeRetries, err) - return err + return nil, err } + defer txn.Discard() - return nil -} + key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(id))) + data, err := txn.Get(key) -func (mgr *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode) error { - // Keys and cleaned up addresses. - keys, addrs, err := keysAndAddrs(p, addrs) - if err != nil { - return err + if err == nil { + pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} + if err = pr.Unmarshal(data); err != nil { + return nil, err + } + if pr.Refresh(false) { + ab.tryFlush(pr) + } + if cache { + ab.cache.Add(id, pr) + } + return pr, nil } - mgr.cache.Remove(p) - // Attempt transactional KV insertion. - var existed []bool - for i := 0; i < mgr.writeRetries; i++ { - if existed, err = mgr.dbInsert(keys, addrs, ttl, mode); err == nil { - break + if err == ds.ErrNotFound { + pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{ + Id: []byte(id), + Addrs: make(map[string]*pb.AddrBookRecord_AddrEntry), + }} + if cache { + ab.cache.Add(id, pr) } - log.Errorf("failed to write addresses for peer %s: %s\n", p.Pretty(), err) + return pr, nil } - if err != nil { - log.Errorf("failed to avoid write conflict for peer %s after %d retries: %v\n", p.Pretty(), mgr.writeRetries, err) - return err - } + log.Error(err) + return nil, err +} - // Update was successful, so broadcast event only for new addresses. - for i, _ := range keys { - if !existed[i] { - mgr.subsManager.BroadcastAddr(p, addrs[i]) +// background is the peerstore process that takes care of: +// * purging expired addresses and peers with no addresses from the datastore at regular intervals. +// * asynchronously flushing cached entries with expired addresses to the datastore. +func (ab *dsAddrBook) background() { + timer := time.NewTicker(ab.gcInterval) + + for { + select { + case fj := <-ab.flushJobCh: + id, _ := peer.IDFromBytes(fj.Id) + if cached, ok := ab.cache.Peek(id); ok { + // Only continue flushing if the record we have in memory is the same as for which the flush + // job was requested. If it's not in memory, it has been evicted and we don't know if we hold + // the latest state or not. Similarly, if it's cached but the pointer is different, it means + // it was evicted and has been reloaded, so we're also uncertain if we hold the latest state. + if pr := cached.(*addrsRecord); pr == fj { + pr.RLock() + pr.Flush(ab.ds) + pr.RUnlock() + } + } + + case <-timer.C: + ab.purgeCycle() + + case <-ab.ctx.Done(): + timer.Stop() + close(ab.closedCh) + return } } - return nil } -// dbInsert performs a transactional insert of the provided keys and values. -func (mgr *dsAddrBook) dbInsert(keys []ds.Key, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode) ([]bool, error) { - var ( - err error - existed = make([]bool, len(keys)) - exp = time.Now().Add(ttl) - ) +func (ab *dsAddrBook) purgeCycle() { + var id peer.ID + record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} + q := query.Query{Prefix: addrBookBase.String()} - txn, err := mgr.ds.NewTransaction(false) + txn, err := ab.ds.NewTransaction(false) if err != nil { - return nil, err + log.Warningf("failed while purging entries: %v\n", err) + return } + defer txn.Discard() - ttltxn := txn.(ds.TTLDatastore) - for i, key := range keys { - // Check if the key existed previously. - if existed[i], err = ttltxn.Has(key); err != nil { - log.Errorf("transaction failed and aborted while checking key existence: %s, cause: %v", key.String(), err) - return nil, err - } + results, err := txn.Query(q) + if err != nil { + log.Warningf("failed while purging entries: %v\n", err) + return + } - // The key embeds a hash of the value, so if it existed, we can safely skip the insert and - // just update the TTL. - if existed[i] { - switch mode { - case ttlOverride: - err = ttltxn.SetTTL(key, ttl) - case ttlExtend: - var curr time.Time - if curr, err = ttltxn.GetExpiration(key); err == nil && exp.After(curr) { - err = ttltxn.SetTTL(key, ttl) - } - } - if err != nil { - // mode will be printed as an int - log.Errorf("failed while updating the ttl for key: %s, mode: %v, cause: %v", key.String(), mode, err) - return nil, err + defer results.Close() + + for result := range results.Next() { + id, _ = peer.IDFromBytes(record.Id) + + // if the record is in cache, let's refresh that one and flush it if necessary. + if e, ok := ab.cache.Peek(id); ok { + cached := e.(*addrsRecord) + cached.Lock() + if cached.Refresh(false) { + cached.FlushInTxn(txn) } + cached.Unlock() continue } - r := &addrRecord{ - ttl: ttl, - addr: addrs[i], + if err := record.Unmarshal(result.Value); err != nil { + log.Warningf("failed while purging entries: %v\n", err) + continue } - value, _ := r.MarshalBinary() - if err = ttltxn.PutWithTTL(key, value, ttl); err != nil { - log.Errorf("transaction failed and aborted while setting key: %s, cause: %v", key.String(), err) - return nil, err + + if record.Refresh(false) { + record.FlushInTxn(txn) } + record.Reset() } if err = txn.Commit(); err != nil { - log.Errorf("failed to commit transaction when setting keys, cause: %v", err) - return nil, err + log.Warningf("failed to commit GC transaction: %v\n", err) } - - return existed, nil } -// UpdateAddrs will update any addresses for a given peer and TTL combination to -// have a new TTL. -func (mgr *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { - mgr.cache.Remove(p) +// AddAddr will add a new address if it's not already in the AddrBook. +func (ab *dsAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + ab.AddAddrs(p, []ma.Multiaddr{addr}, ttl) +} - var err error - for i := 0; i < mgr.writeRetries; i++ { - if err = mgr.dbUpdateTTL(p, oldTTL, newTTL); err == nil { - break - } - log.Errorf("failed to update ttlsfor peer %s: %s\n", p.Pretty(), err) +// AddAddrs will add many new addresses if they're not already in the AddrBook. +func (ab *dsAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + if ttl <= 0 { + return } + addrs = cleanAddrs(addrs) + ab.setAddrs(p, addrs, ttl, ttlExtend) +} - if err != nil { - log.Errorf("failed to avoid write conflict when updating ttls for peer %s after %d retries: %v\n", - p.Pretty(), mgr.writeRetries, err) - } +// SetAddr will add or update the TTL of an address in the AddrBook. +func (ab *dsAddrBook) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + ab.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } -func (mgr *dsAddrBook) dbUpdateTTL(p peer.ID, oldTTL time.Duration, newTTL time.Duration) error { - var ( - prefix = abBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))) - q = query.Query{Prefix: prefix.String(), KeysOnly: false} - results query.Results - err error - ) +// SetAddrs will add or update the TTLs of addresses in the AddrBook. +func (ab *dsAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + addrs = cleanAddrs(addrs) + if ttl <= 0 { + ab.deleteAddrs(p, addrs) + return + } + ab.setAddrs(p, addrs, ttl, ttlOverride) +} - txn, err := mgr.ds.NewTransaction(false) +// UpdateAddrs will update any addresses for a given peer and TTL combination to +// have a new TTL. +func (ab *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { + pr, err := ab.loadRecord(p, true, false) if err != nil { - return err + log.Errorf("failed to update ttls for peer %s: %s\n", p.Pretty(), err) } - defer txn.Discard() - if results, err = txn.Query(q); err != nil { - return err - } - defer results.Close() + pr.Lock() + defer pr.Unlock() - ttltxn := txn.(ds.TTLDatastore) - r := &addrRecord{} - for result := range results.Next() { - r.UnmarshalBinary(result.Value) - if r.ttl != oldTTL { + chgd, newExp := false, time.Now().Add(newTTL) + for _, entry := range pr.Addrs { + if entry.Ttl == nil || *entry.Ttl != oldTTL { continue } - - r.ttl = newTTL - value, _ := r.MarshalBinary() - if err = ttltxn.PutWithTTL(ds.RawKey(result.Key), value, newTTL); err != nil { - return err - } + entry.Ttl, entry.Expiry = &newTTL, &newExp + chgd = true } - if err := txn.Commit(); err != nil { - log.Errorf("failed to commit transaction when updating ttls, cause: %v", err) - return err + if chgd { + pr.Refresh(true) + pr.Flush(ab.ds) } - - return nil } // Addrs returns all of the non-expired addresses for a given peer. -func (mgr *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { - var ( - prefix = abBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))) - q = query.Query{Prefix: prefix.String(), KeysOnly: false, ReturnExpirations: true} - results query.Results - err error - ) - - // Check the cache and return the entry only if it hasn't expired; if expired, remove. - if e, ok := mgr.cache.Get(p); ok { - entry := e.(cacheEntry) - if entry.expiration.After(time.Now()) { - addrs := make([]ma.Multiaddr, len(entry.addrs)) - copy(addrs, entry.addrs) - return addrs - } else { - mgr.cache.Remove(p) - } - } - - txn, err := mgr.ds.NewTransaction(true) +func (ab *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { + pr, err := ab.loadRecord(p, true, true) if err != nil { + log.Warning("failed to load peerstore entry for peer %v while querying addrs, err: %v", p, err) return nil } - defer txn.Discard() - if results, err = txn.Query(q); err != nil { - log.Error(err) - return nil - } - defer results.Close() - - var addrs []ma.Multiaddr - var r addrRecord - // used to set the expiration for the entire cache entry - earliestExp := maxTime - for result := range results.Next() { - if err = r.UnmarshalBinary(result.Value); err == nil { - addrs = append(addrs, r.addr) - } + pr.RLock() + defer pr.RUnlock() - if exp := result.Expiration; !exp.IsZero() && exp.Before(earliestExp) { - earliestExp = exp + addrs := make([]ma.Multiaddr, 0, len(pr.Addrs)) + for k, _ := range pr.Addrs { + if a, err := ma.NewMultiaddr(k); err == nil { + addrs = append(addrs, a) } } - - // Store a copy in the cache. - addrsCpy := make([]ma.Multiaddr, len(addrs)) - copy(addrsCpy, addrs) - entry := cacheEntry{addrs: addrsCpy, expiration: earliestExp} - mgr.cache.Add(p, entry) - return addrs } // Peers returns all of the peer IDs for which the AddrBook has addresses. -func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { - ids, err := uniquePeerIds(mgr.ds, abBase, func(result query.Result) string { - return ds.RawKey(result.Key).Parent().Name() +func (ab *dsAddrBook) PeersWithAddrs() peer.IDSlice { + ids, err := uniquePeerIds(ab.ds, addrBookBase, func(result query.Result) string { + return ds.RawKey(result.Key).Name() }) if err != nil { log.Errorf("error while retrieving peers with addresses: %v", err) @@ -403,107 +406,91 @@ func (mgr *dsAddrBook) PeersWithAddrs() peer.IDSlice { // AddrStream returns a channel on which all new addresses discovered for a // given peer ID will be published. -func (mgr *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { - initial := mgr.Addrs(p) - return mgr.subsManager.AddrStream(ctx, p, initial) +func (ab *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { + initial := ab.Addrs(p) + return ab.subsManager.AddrStream(ctx, p, initial) } // ClearAddrs will delete all known addresses for a peer ID. -func (mgr *dsAddrBook) ClearAddrs(p peer.ID) { - var ( - err error - prefix = abBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))) - deleteFn func() error - ) - - if e, ok := mgr.cache.Peek(p); ok { - mgr.cache.Remove(p) - keys, _, _ := keysAndAddrs(p, e.(cacheEntry).addrs) - deleteFn = func() error { - return mgr.dbDelete(keys) - } - } else { - deleteFn = func() error { - return mgr.dbDeleteIter(prefix) - } +func (ab *dsAddrBook) ClearAddrs(p peer.ID) { + ab.cache.Remove(p) + + key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(p))) + txn, err := ab.ds.NewTransaction(false) + if err != nil { + log.Errorf("failed to clear addresses for peer %s: %v\n", p.Pretty(), err) } + defer txn.Discard() - // Attempt transactional KV deletion. - for i := 0; i < mgr.writeRetries; i++ { - if err = deleteFn(); err == nil { - break - } - log.Errorf("failed to clear addresses for peer %s: %s\n", p.Pretty(), err) + if err := txn.Delete(key); err != nil { + log.Errorf("failed to clear addresses for peer %s: %v\n", p.Pretty(), err) } - if err != nil { - log.Errorf("failed to clear addresses for peer %s after %d attempts\n", p.Pretty(), mgr.writeRetries) + if err = txn.Commit(); err != nil { + log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) } } -// dbDelete transactionally deletes the provided keys. -func (mgr *dsAddrBook) dbDelete(keys []ds.Key) error { - var err error - - txn, err := mgr.ds.NewTransaction(false) +func (ab *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode) (err error) { + pr, err := ab.loadRecord(p, true, false) if err != nil { - return err + return fmt.Errorf("failed to load peerstore entry for peer %v while deleting addrs, err: %v", p, err) } - defer txn.Discard() - for _, key := range keys { - if err = txn.Delete(key); err != nil { - log.Errorf("failed to delete key: %s, cause: %v", key.String(), err) - return err + pr.Lock() + defer pr.Unlock() + + now := time.Now() + broadcast := make([]bool, len(addrs)) + newExp := now.Add(ttl) + for i, addr := range addrs { + e, ok := pr.Addrs[addr.String()] + if ok && mode == ttlExtend && e.Expiry.After(newExp) { + continue } + pr.Addrs[addr.String()] = &pb.AddrBookRecord_AddrEntry{Expiry: &newExp, Ttl: &ttl} + broadcast[i] = !ok } - if err = txn.Commit(); err != nil { - log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) - return err + // Update was successful, so broadcast event only for new addresses. + for i, v := range broadcast { + if v { + ab.subsManager.BroadcastAddr(p, addrs[i]) + } } - return nil + pr.Refresh(true) + return pr.Flush(ab.ds) } -// dbDeleteIter removes all entries whose keys are prefixed with the argument. -// it returns a slice of the removed keys in case it's needed -func (mgr *dsAddrBook) dbDeleteIter(prefix ds.Key) error { - q := query.Query{Prefix: prefix.String(), KeysOnly: true} - - txn, err := mgr.ds.NewTransaction(false) +func (ab *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) (err error) { + pr, err := ab.loadRecord(p, false, false) if err != nil { - return err + return fmt.Errorf("failed to load peerstore entry for peer %v while deleting addrs, err: %v", p, err) } - defer txn.Discard() - results, err := txn.Query(q) - if err != nil { - log.Errorf("failed to fetch all keys prefixed with: %s, cause: %v", prefix.String(), err) - return err + if pr.Addrs == nil { + return nil } - var keys = make([]ds.Key, 0, 4) // cap: 4 to reduce allocs - var key ds.Key - for result := range results.Next() { - key = ds.RawKey(result.Key) - keys = append(keys, key) + pr.Lock() + defer pr.Unlock() - if err = txn.Delete(key); err != nil { - log.Errorf("failed to delete key: %s, cause: %v", key.String(), err) - return err - } + for _, addr := range addrs { + delete(pr.Addrs, addr.String()) } - if err = results.Close(); err != nil { - log.Errorf("failed to close cursor, cause: %v", err) - return err - } + pr.Refresh(true) + return pr.Flush(ab.ds) +} - if err = txn.Commit(); err != nil { - log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) - return err +func cleanAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { + clean := make([]ma.Multiaddr, 0, len(addrs)) + for _, addr := range addrs { + if addr == nil { + continue + } + clean = append(clean, addr) } - - return nil + return clean } diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 29ac9eb04d..21a10db03b 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -38,7 +38,7 @@ func TestDsAddrBook(t *testing.T) { t.Parallel() opts := DefaultOpts() - opts.TTLInterval = 100 * time.Microsecond + opts.GCInterval = 100 * time.Microsecond opts.CacheSize = 1024 pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) @@ -48,7 +48,7 @@ func TestDsAddrBook(t *testing.T) { t.Parallel() opts := DefaultOpts() - opts.TTLInterval = 100 * time.Microsecond + opts.GCInterval = 100 * time.Microsecond opts.CacheSize = 0 pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) @@ -134,7 +134,10 @@ func addressBookFactory(tb testing.TB, storeFactory datastoreFactory, opts Optio tb.Fatal(err) } - return ab, closeFunc + return ab, func() { + ab.Close() + closeFunc() + } } } diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index b1a64f076e..65ac325a4b 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -18,9 +18,8 @@ type Options struct { // The size of the in-memory cache. A value of 0 or lower disables the cache. CacheSize uint - // Sweep interval to expire entries, only used when TTL is *not* natively managed - // by the underlying datastore. - TTLInterval time.Duration + // Sweep interval to purge expired addresses from the datastore. + GCInterval time.Duration // Number of times to retry transactional writes. WriteRetries uint @@ -33,7 +32,7 @@ type Options struct { func DefaultOpts() Options { return Options{ CacheSize: 1024, - TTLInterval: time.Second, + GCInterval: 5 * time.Minute, WriteRetries: 5, } } From 01bb5211566cae69655c903961e67e66a60542be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 15 Nov 2018 00:27:28 +0000 Subject: [PATCH 0840/3965] pstoreds: tighten up gc-based peerstore. Introduces custom types in protobuf to serde directly into multiaddrs and peer IDs. Simplify purge by ordering addrs by expiry. In general, getting this readier for merge. --- p2p/host/peerstore/pb/custom.go | 89 +++++ p2p/host/peerstore/pb/pstore.pb.go | 445 +++++++---------------- p2p/host/peerstore/pb/pstore.proto | 19 +- p2p/host/peerstore/pb/pstorepb_test.go | 2 - p2p/host/peerstore/pstoreds/addr_book.go | 279 ++++++++------ 5 files changed, 388 insertions(+), 446 deletions(-) create mode 100644 p2p/host/peerstore/pb/custom.go diff --git a/p2p/host/peerstore/pb/custom.go b/p2p/host/peerstore/pb/custom.go new file mode 100644 index 0000000000..e51b2ca7d3 --- /dev/null +++ b/p2p/host/peerstore/pb/custom.go @@ -0,0 +1,89 @@ +package pstore_pb + +import ( + "encoding/json" + + peer "github.com/libp2p/go-libp2p-peer" + pt "github.com/libp2p/go-libp2p-peer/test" + ma "github.com/multiformats/go-multiaddr" +) + +type ProtoPeerID struct { + peer.ID +} + +func NewPopulatedProtoPeerID(r randyPstore) *ProtoPeerID { + id, _ := pt.RandPeerID() + return &ProtoPeerID{ID: id} +} + +func (id ProtoPeerID) Marshal() ([]byte, error) { + return []byte(id.ID), nil +} + +func (id ProtoPeerID) MarshalTo(data []byte) (n int, err error) { + return copy(data, []byte(id.ID)), nil +} + +func (id ProtoPeerID) MarshalJSON() ([]byte, error) { + m, _ := id.Marshal() + return json.Marshal(m) +} + +func (id *ProtoPeerID) Unmarshal(data []byte) (err error) { + id.ID = peer.ID(string(data)) + return nil +} + +func (id *ProtoPeerID) UnmarshalJSON(data []byte) error { + v := new([]byte) + err := json.Unmarshal(data, v) + if err != nil { + return err + } + return id.Unmarshal(*v) +} + +func (id ProtoPeerID) Size() int { + return len([]byte(id.ID)) +} + +type ProtoAddr struct { + ma.Multiaddr +} + +func NewPopulatedProtoAddr(r randyPstore) *ProtoAddr { + a, _ := ma.NewMultiaddr("/ip4/123.123.123.123/tcp/7001") + return &ProtoAddr{Multiaddr: a} +} + +func (a ProtoAddr) Marshal() ([]byte, error) { + return a.Bytes(), nil +} + +func (a ProtoAddr) MarshalTo(data []byte) (n int, err error) { + return copy(data, a.Bytes()), nil +} + +func (a ProtoAddr) MarshalJSON() ([]byte, error) { + m, _ := a.Marshal() + return json.Marshal(m) +} + +func (a *ProtoAddr) Unmarshal(data []byte) (err error) { + a.Multiaddr, err = ma.NewMultiaddrBytes(data) + return err +} + +func (a *ProtoAddr) UnmarshalJSON(data []byte) error { + v := new([]byte) + err := json.Unmarshal(data, v) + if err != nil { + return err + } + return a.Unmarshal(*v) +} + +func (a ProtoAddr) Size() int { + return len(a.Bytes()) +} diff --git a/p2p/host/peerstore/pb/pstore.pb.go b/p2p/host/peerstore/pb/pstore.pb.go index cf4b04b1d9..ba5a51b63d 100644 --- a/p2p/host/peerstore/pb/pstore.pb.go +++ b/p2p/host/peerstore/pb/pstore.pb.go @@ -7,12 +7,6 @@ import proto "github.com/gogo/protobuf/proto" import fmt "fmt" import math "math" import _ "github.com/gogo/protobuf/gogoproto" -import _ "github.com/golang/protobuf/ptypes/duration" -import _ "github.com/golang/protobuf/ptypes/timestamp" - -import time "time" - -import github_com_gogo_protobuf_types "github.com/gogo/protobuf/types" import io "io" @@ -20,7 +14,6 @@ import io "io" var _ = proto.Marshal var _ = fmt.Errorf var _ = math.Inf -var _ = time.Kitchen // This is a compile-time assertion to ensure that this generated file // is compatible with the proto package it is being compiled against. @@ -31,21 +24,19 @@ const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package // AddrBookRecord represents a record for a peer in the address book. type AddrBookRecord struct { // The peer ID. - Id []byte `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` - // The timestamp where purging needs to happen. - NextVisit *time.Time `protobuf:"bytes,2,opt,name=nextVisit,stdtime" json:"nextVisit,omitempty"` - // The multiaddresses. - Addrs map[string]*AddrBookRecord_AddrEntry `protobuf:"bytes,3,rep,name=addrs" json:"addrs,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + Id *ProtoPeerID `protobuf:"bytes,1,opt,name=id,proto3,customtype=ProtoPeerID" json:"id,omitempty"` + // The multiaddresses. This is a sorted list where element 0 expires the soonest. + Addrs []*AddrBookRecord_AddrEntry `protobuf:"bytes,2,rep,name=addrs" json:"addrs,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *AddrBookRecord) Reset() { *m = AddrBookRecord{} } func (m *AddrBookRecord) String() string { return proto.CompactTextString(m) } func (*AddrBookRecord) ProtoMessage() {} func (*AddrBookRecord) Descriptor() ([]byte, []int) { - return fileDescriptor_pstore_842456c80d89ffef, []int{0} + return fileDescriptor_pstore_fcc3073dbc5464a9, []int{0} } func (m *AddrBookRecord) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -74,21 +65,7 @@ func (m *AddrBookRecord) XXX_DiscardUnknown() { var xxx_messageInfo_AddrBookRecord proto.InternalMessageInfo -func (m *AddrBookRecord) GetId() []byte { - if m != nil { - return m.Id - } - return nil -} - -func (m *AddrBookRecord) GetNextVisit() *time.Time { - if m != nil { - return m.NextVisit - } - return nil -} - -func (m *AddrBookRecord) GetAddrs() map[string]*AddrBookRecord_AddrEntry { +func (m *AddrBookRecord) GetAddrs() []*AddrBookRecord_AddrEntry { if m != nil { return m.Addrs } @@ -97,20 +74,21 @@ func (m *AddrBookRecord) GetAddrs() map[string]*AddrBookRecord_AddrEntry { // AddrEntry represents a single multiaddress. type AddrBookRecord_AddrEntry struct { + Addr *ProtoAddr `protobuf:"bytes,1,opt,name=addr,proto3,customtype=ProtoAddr" json:"addr,omitempty"` // The point in time when this address expires. - Expiry *time.Time `protobuf:"bytes,2,opt,name=expiry,stdtime" json:"expiry,omitempty"` + Expiry int64 `protobuf:"varint,2,opt,name=expiry,proto3" json:"expiry,omitempty"` // The original TTL of this address. - Ttl *time.Duration `protobuf:"bytes,3,opt,name=ttl,stdduration" json:"ttl,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + Ttl int64 `protobuf:"varint,3,opt,name=ttl,proto3" json:"ttl,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *AddrBookRecord_AddrEntry) Reset() { *m = AddrBookRecord_AddrEntry{} } func (m *AddrBookRecord_AddrEntry) String() string { return proto.CompactTextString(m) } func (*AddrBookRecord_AddrEntry) ProtoMessage() {} func (*AddrBookRecord_AddrEntry) Descriptor() ([]byte, []int) { - return fileDescriptor_pstore_842456c80d89ffef, []int{0, 1} + return fileDescriptor_pstore_fcc3073dbc5464a9, []int{0, 0} } func (m *AddrBookRecord_AddrEntry) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -139,23 +117,22 @@ func (m *AddrBookRecord_AddrEntry) XXX_DiscardUnknown() { var xxx_messageInfo_AddrBookRecord_AddrEntry proto.InternalMessageInfo -func (m *AddrBookRecord_AddrEntry) GetExpiry() *time.Time { +func (m *AddrBookRecord_AddrEntry) GetExpiry() int64 { if m != nil { return m.Expiry } - return nil + return 0 } -func (m *AddrBookRecord_AddrEntry) GetTtl() *time.Duration { +func (m *AddrBookRecord_AddrEntry) GetTtl() int64 { if m != nil { return m.Ttl } - return nil + return 0 } func init() { proto.RegisterType((*AddrBookRecord)(nil), "pstore.pb.AddrBookRecord") - proto.RegisterMapType((map[string]*AddrBookRecord_AddrEntry)(nil), "pstore.pb.AddrBookRecord.AddrsEntry") proto.RegisterType((*AddrBookRecord_AddrEntry)(nil), "pstore.pb.AddrBookRecord.AddrEntry") } func (m *AddrBookRecord) Marshal() (dAtA []byte, err error) { @@ -173,48 +150,26 @@ func (m *AddrBookRecord) MarshalTo(dAtA []byte) (int, error) { _ = i var l int _ = l - if len(m.Id) > 0 { + if m.Id != nil { dAtA[i] = 0xa i++ - i = encodeVarintPstore(dAtA, i, uint64(len(m.Id))) - i += copy(dAtA[i:], m.Id) - } - if m.NextVisit != nil { - dAtA[i] = 0x12 - i++ - i = encodeVarintPstore(dAtA, i, uint64(github_com_gogo_protobuf_types.SizeOfStdTime(*m.NextVisit))) - n1, err := github_com_gogo_protobuf_types.StdTimeMarshalTo(*m.NextVisit, dAtA[i:]) + i = encodeVarintPstore(dAtA, i, uint64(m.Id.Size())) + n1, err := m.Id.MarshalTo(dAtA[i:]) if err != nil { return 0, err } i += n1 } if len(m.Addrs) > 0 { - for k, _ := range m.Addrs { - dAtA[i] = 0x1a - i++ - v := m.Addrs[k] - msgSize := 0 - if v != nil { - msgSize = v.Size() - msgSize += 1 + sovPstore(uint64(msgSize)) - } - mapSize := 1 + len(k) + sovPstore(uint64(len(k))) + msgSize - i = encodeVarintPstore(dAtA, i, uint64(mapSize)) - dAtA[i] = 0xa + for _, msg := range m.Addrs { + dAtA[i] = 0x12 i++ - i = encodeVarintPstore(dAtA, i, uint64(len(k))) - i += copy(dAtA[i:], k) - if v != nil { - dAtA[i] = 0x12 - i++ - i = encodeVarintPstore(dAtA, i, uint64(v.Size())) - n2, err := v.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err - } - i += n2 + i = encodeVarintPstore(dAtA, i, uint64(msg.Size())) + n, err := msg.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err } + i += n } } if m.XXX_unrecognized != nil { @@ -238,25 +193,25 @@ func (m *AddrBookRecord_AddrEntry) MarshalTo(dAtA []byte) (int, error) { _ = i var l int _ = l - if m.Expiry != nil { - dAtA[i] = 0x12 + if m.Addr != nil { + dAtA[i] = 0xa i++ - i = encodeVarintPstore(dAtA, i, uint64(github_com_gogo_protobuf_types.SizeOfStdTime(*m.Expiry))) - n3, err := github_com_gogo_protobuf_types.StdTimeMarshalTo(*m.Expiry, dAtA[i:]) + i = encodeVarintPstore(dAtA, i, uint64(m.Addr.Size())) + n2, err := m.Addr.MarshalTo(dAtA[i:]) if err != nil { return 0, err } - i += n3 + i += n2 } - if m.Ttl != nil { - dAtA[i] = 0x1a + if m.Expiry != 0 { + dAtA[i] = 0x10 i++ - i = encodeVarintPstore(dAtA, i, uint64(github_com_gogo_protobuf_types.SizeOfStdDuration(*m.Ttl))) - n4, err := github_com_gogo_protobuf_types.StdDurationMarshalTo(*m.Ttl, dAtA[i:]) - if err != nil { - return 0, err - } - i += n4 + i = encodeVarintPstore(dAtA, i, uint64(m.Expiry)) + } + if m.Ttl != 0 { + dAtA[i] = 0x18 + i++ + i = encodeVarintPstore(dAtA, i, uint64(m.Ttl)) } if m.XXX_unrecognized != nil { i += copy(dAtA[i:], m.XXX_unrecognized) @@ -275,34 +230,30 @@ func encodeVarintPstore(dAtA []byte, offset int, v uint64) int { } func NewPopulatedAddrBookRecord(r randyPstore, easy bool) *AddrBookRecord { this := &AddrBookRecord{} - v1 := r.Intn(100) - this.Id = make([]byte, v1) - for i := 0; i < v1; i++ { - this.Id[i] = byte(r.Intn(256)) - } - if r.Intn(10) != 0 { - this.NextVisit = github_com_gogo_protobuf_types.NewPopulatedStdTime(r, easy) - } + this.Id = NewPopulatedProtoPeerID(r) if r.Intn(10) != 0 { - v2 := r.Intn(10) - this.Addrs = make(map[string]*AddrBookRecord_AddrEntry) - for i := 0; i < v2; i++ { - this.Addrs[randStringPstore(r)] = NewPopulatedAddrBookRecord_AddrEntry(r, easy) + v1 := r.Intn(5) + this.Addrs = make([]*AddrBookRecord_AddrEntry, v1) + for i := 0; i < v1; i++ { + this.Addrs[i] = NewPopulatedAddrBookRecord_AddrEntry(r, easy) } } if !easy && r.Intn(10) != 0 { - this.XXX_unrecognized = randUnrecognizedPstore(r, 4) + this.XXX_unrecognized = randUnrecognizedPstore(r, 3) } return this } func NewPopulatedAddrBookRecord_AddrEntry(r randyPstore, easy bool) *AddrBookRecord_AddrEntry { this := &AddrBookRecord_AddrEntry{} - if r.Intn(10) != 0 { - this.Expiry = github_com_gogo_protobuf_types.NewPopulatedStdTime(r, easy) + this.Addr = NewPopulatedProtoAddr(r) + this.Expiry = int64(r.Int63()) + if r.Intn(2) == 0 { + this.Expiry *= -1 } - if r.Intn(10) != 0 { - this.Ttl = github_com_gogo_protobuf_types.NewPopulatedStdDuration(r, easy) + this.Ttl = int64(r.Int63()) + if r.Intn(2) == 0 { + this.Ttl *= -1 } if !easy && r.Intn(10) != 0 { this.XXX_unrecognized = randUnrecognizedPstore(r, 4) @@ -329,9 +280,9 @@ func randUTF8RunePstore(r randyPstore) rune { return rune(ru + 61) } func randStringPstore(r randyPstore) string { - v3 := r.Intn(100) - tmps := make([]rune, v3) - for i := 0; i < v3; i++ { + v2 := r.Intn(100) + tmps := make([]rune, v2) + for i := 0; i < v2; i++ { tmps[i] = randUTF8RunePstore(r) } return string(tmps) @@ -353,11 +304,11 @@ func randFieldPstore(dAtA []byte, r randyPstore, fieldNumber int, wire int) []by switch wire { case 0: dAtA = encodeVarintPopulatePstore(dAtA, uint64(key)) - v4 := r.Int63() + v3 := r.Int63() if r.Intn(2) == 0 { - v4 *= -1 + v3 *= -1 } - dAtA = encodeVarintPopulatePstore(dAtA, uint64(v4)) + dAtA = encodeVarintPopulatePstore(dAtA, uint64(v3)) case 1: dAtA = encodeVarintPopulatePstore(dAtA, uint64(key)) dAtA = append(dAtA, byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256))) @@ -388,25 +339,14 @@ func (m *AddrBookRecord) Size() (n int) { } var l int _ = l - l = len(m.Id) - if l > 0 { - n += 1 + l + sovPstore(uint64(l)) - } - if m.NextVisit != nil { - l = github_com_gogo_protobuf_types.SizeOfStdTime(*m.NextVisit) + if m.Id != nil { + l = m.Id.Size() n += 1 + l + sovPstore(uint64(l)) } if len(m.Addrs) > 0 { - for k, v := range m.Addrs { - _ = k - _ = v - l = 0 - if v != nil { - l = v.Size() - l += 1 + sovPstore(uint64(l)) - } - mapEntrySize := 1 + len(k) + sovPstore(uint64(len(k))) + l - n += mapEntrySize + 1 + sovPstore(uint64(mapEntrySize)) + for _, e := range m.Addrs { + l = e.Size() + n += 1 + l + sovPstore(uint64(l)) } } if m.XXX_unrecognized != nil { @@ -421,13 +361,15 @@ func (m *AddrBookRecord_AddrEntry) Size() (n int) { } var l int _ = l - if m.Expiry != nil { - l = github_com_gogo_protobuf_types.SizeOfStdTime(*m.Expiry) + if m.Addr != nil { + l = m.Addr.Size() n += 1 + l + sovPstore(uint64(l)) } - if m.Ttl != nil { - l = github_com_gogo_protobuf_types.SizeOfStdDuration(*m.Ttl) - n += 1 + l + sovPstore(uint64(l)) + if m.Expiry != 0 { + n += 1 + sovPstore(uint64(m.Expiry)) + } + if m.Ttl != 0 { + n += 1 + sovPstore(uint64(m.Ttl)) } if m.XXX_unrecognized != nil { n += len(m.XXX_unrecognized) @@ -503,45 +445,13 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { if postIndex > l { return io.ErrUnexpectedEOF } - m.Id = append(m.Id[:0], dAtA[iNdEx:postIndex]...) - if m.Id == nil { - m.Id = []byte{} - } - iNdEx = postIndex - case 2: - if wireType != 2 { - return fmt.Errorf("proto: wrong wireType = %d for field NextVisit", wireType) - } - var msglen int - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return ErrIntOverflowPstore - } - if iNdEx >= l { - return io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - msglen |= (int(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - if msglen < 0 { - return ErrInvalidLengthPstore - } - postIndex := iNdEx + msglen - if postIndex > l { - return io.ErrUnexpectedEOF - } - if m.NextVisit == nil { - m.NextVisit = new(time.Time) - } - if err := github_com_gogo_protobuf_types.StdTimeUnmarshal(m.NextVisit, dAtA[iNdEx:postIndex]); err != nil { + var v ProtoPeerID + m.Id = &v + if err := m.Id.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { return err } iNdEx = postIndex - case 3: + case 2: if wireType != 2 { return fmt.Errorf("proto: wrong wireType = %d for field Addrs", wireType) } @@ -567,102 +477,10 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { if postIndex > l { return io.ErrUnexpectedEOF } - if m.Addrs == nil { - m.Addrs = make(map[string]*AddrBookRecord_AddrEntry) - } - var mapkey string - var mapvalue *AddrBookRecord_AddrEntry - for iNdEx < postIndex { - entryPreIndex := iNdEx - var wire uint64 - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return ErrIntOverflowPstore - } - if iNdEx >= l { - return io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - wire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - fieldNum := int32(wire >> 3) - if fieldNum == 1 { - var stringLenmapkey uint64 - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return ErrIntOverflowPstore - } - if iNdEx >= l { - return io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - stringLenmapkey |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - intStringLenmapkey := int(stringLenmapkey) - if intStringLenmapkey < 0 { - return ErrInvalidLengthPstore - } - postStringIndexmapkey := iNdEx + intStringLenmapkey - if postStringIndexmapkey > l { - return io.ErrUnexpectedEOF - } - mapkey = string(dAtA[iNdEx:postStringIndexmapkey]) - iNdEx = postStringIndexmapkey - } else if fieldNum == 2 { - var mapmsglen int - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return ErrIntOverflowPstore - } - if iNdEx >= l { - return io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - mapmsglen |= (int(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - if mapmsglen < 0 { - return ErrInvalidLengthPstore - } - postmsgIndex := iNdEx + mapmsglen - if mapmsglen < 0 { - return ErrInvalidLengthPstore - } - if postmsgIndex > l { - return io.ErrUnexpectedEOF - } - mapvalue = &AddrBookRecord_AddrEntry{} - if err := mapvalue.Unmarshal(dAtA[iNdEx:postmsgIndex]); err != nil { - return err - } - iNdEx = postmsgIndex - } else { - iNdEx = entryPreIndex - skippy, err := skipPstore(dAtA[iNdEx:]) - if err != nil { - return err - } - if skippy < 0 { - return ErrInvalidLengthPstore - } - if (iNdEx + skippy) > postIndex { - return io.ErrUnexpectedEOF - } - iNdEx += skippy - } + m.Addrs = append(m.Addrs, &AddrBookRecord_AddrEntry{}) + if err := m.Addrs[len(m.Addrs)-1].Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err } - m.Addrs[mapkey] = mapvalue iNdEx = postIndex default: iNdEx = preIndex @@ -715,11 +533,11 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { return fmt.Errorf("proto: AddrEntry: illegal tag %d (wire type %d)", fieldNum, wire) } switch fieldNum { - case 2: + case 1: if wireType != 2 { - return fmt.Errorf("proto: wrong wireType = %d for field Expiry", wireType) + return fmt.Errorf("proto: wrong wireType = %d for field Addr", wireType) } - var msglen int + var byteLen int for shift := uint(0); ; shift += 7 { if shift >= 64 { return ErrIntOverflowPstore @@ -729,30 +547,29 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - msglen |= (int(b) & 0x7F) << shift + byteLen |= (int(b) & 0x7F) << shift if b < 0x80 { break } } - if msglen < 0 { + if byteLen < 0 { return ErrInvalidLengthPstore } - postIndex := iNdEx + msglen + postIndex := iNdEx + byteLen if postIndex > l { return io.ErrUnexpectedEOF } - if m.Expiry == nil { - m.Expiry = new(time.Time) - } - if err := github_com_gogo_protobuf_types.StdTimeUnmarshal(m.Expiry, dAtA[iNdEx:postIndex]); err != nil { + var v ProtoAddr + m.Addr = &v + if err := m.Addr.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { return err } iNdEx = postIndex - case 3: - if wireType != 2 { - return fmt.Errorf("proto: wrong wireType = %d for field Ttl", wireType) + case 2: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Expiry", wireType) } - var msglen int + m.Expiry = 0 for shift := uint(0); ; shift += 7 { if shift >= 64 { return ErrIntOverflowPstore @@ -762,25 +579,30 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - msglen |= (int(b) & 0x7F) << shift + m.Expiry |= (int64(b) & 0x7F) << shift if b < 0x80 { break } } - if msglen < 0 { - return ErrInvalidLengthPstore - } - postIndex := iNdEx + msglen - if postIndex > l { - return io.ErrUnexpectedEOF - } - if m.Ttl == nil { - m.Ttl = new(time.Duration) + case 3: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Ttl", wireType) } - if err := github_com_gogo_protobuf_types.StdDurationUnmarshal(m.Ttl, dAtA[iNdEx:postIndex]); err != nil { - return err + m.Ttl = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.Ttl |= (int64(b) & 0x7F) << shift + if b < 0x80 { + break + } } - iNdEx = postIndex default: iNdEx = preIndex skippy, err := skipPstore(dAtA[iNdEx:]) @@ -908,29 +730,24 @@ var ( ErrIntOverflowPstore = fmt.Errorf("proto: integer overflow") ) -func init() { proto.RegisterFile("pstore.proto", fileDescriptor_pstore_842456c80d89ffef) } - -var fileDescriptor_pstore_842456c80d89ffef = []byte{ - // 330 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x91, 0xbf, 0x4e, 0xc3, 0x30, - 0x10, 0xc6, 0xe5, 0xa4, 0xad, 0x14, 0xb7, 0xaa, 0x90, 0xc5, 0x10, 0x32, 0xb8, 0x15, 0x30, 0x74, - 0xc1, 0x15, 0x65, 0x29, 0x1d, 0x90, 0x88, 0xe0, 0x05, 0x22, 0xc4, 0xc6, 0x90, 0xd4, 0x26, 0x58, - 0xfd, 0x73, 0x91, 0xe3, 0xa0, 0xf6, 0x2d, 0x18, 0x79, 0x1c, 0x46, 0x46, 0xde, 0x00, 0xc8, 0x3b, - 0x20, 0x31, 0xa2, 0x38, 0x69, 0x23, 0x40, 0x42, 0x6c, 0xf7, 0xdd, 0x7d, 0xbf, 0xf3, 0x77, 0x32, - 0xee, 0x24, 0xa9, 0x06, 0x25, 0x58, 0xa2, 0x40, 0x03, 0x71, 0x36, 0x2a, 0xf2, 0x7a, 0x31, 0x40, - 0x3c, 0x17, 0x43, 0x33, 0x88, 0xb2, 0xdb, 0xa1, 0x96, 0x0b, 0x91, 0xea, 0x70, 0x91, 0x94, 0x5e, - 0x8f, 0xfe, 0x34, 0xf0, 0x4c, 0x85, 0x5a, 0xc2, 0xb2, 0x9a, 0x1f, 0xc5, 0x52, 0xdf, 0x65, 0x11, - 0x9b, 0xc2, 0x62, 0x18, 0x43, 0x0c, 0xb5, 0xb1, 0x50, 0x46, 0x98, 0xaa, 0xb4, 0xef, 0x7f, 0x58, - 0xb8, 0x7b, 0xce, 0xb9, 0xf2, 0x01, 0x66, 0x81, 0x98, 0x82, 0xe2, 0xa4, 0x8b, 0x2d, 0xc9, 0x5d, - 0xd4, 0x47, 0x83, 0x4e, 0x60, 0x49, 0x4e, 0xce, 0xb0, 0xb3, 0x14, 0x2b, 0x7d, 0x2d, 0x53, 0xa9, - 0x5d, 0xab, 0x8f, 0x06, 0xed, 0x91, 0xc7, 0xca, 0x14, 0x6c, 0xb3, 0x9c, 0x5d, 0x6d, 0x62, 0xfa, - 0x8d, 0x87, 0xd7, 0x1e, 0x0a, 0x6a, 0x84, 0x4c, 0x70, 0x33, 0xe4, 0x5c, 0xa5, 0xae, 0xdd, 0xb7, - 0x07, 0xed, 0xd1, 0x21, 0xdb, 0x5e, 0xcb, 0xbe, 0xbf, 0x6c, 0x64, 0x7a, 0xb9, 0xd4, 0x6a, 0x1d, - 0x94, 0x88, 0x77, 0x83, 0x71, 0xdd, 0x24, 0x3b, 0xd8, 0x9e, 0x89, 0xb5, 0x89, 0xe6, 0x04, 0x45, - 0x49, 0x4e, 0x71, 0xf3, 0x3e, 0x9c, 0x67, 0xa2, 0xca, 0x75, 0xf0, 0xf7, 0xee, 0x6a, 0xb5, 0x21, - 0x26, 0xd6, 0x18, 0x79, 0x2b, 0xec, 0x6c, 0xfb, 0x64, 0x8c, 0x5b, 0x62, 0x95, 0x48, 0xb5, 0xfe, - 0xf7, 0x91, 0x95, 0x9f, 0x1c, 0x63, 0x5b, 0xeb, 0xb9, 0x6b, 0x1b, 0x6c, 0xef, 0x17, 0x76, 0x51, - 0xfd, 0x90, 0xdf, 0x78, 0x2c, 0xa8, 0xc2, 0xeb, 0xef, 0x7e, 0xbe, 0x53, 0xf4, 0x94, 0x53, 0xf4, - 0x9c, 0x53, 0xf4, 0x92, 0x53, 0xf4, 0x96, 0x53, 0x14, 0xb5, 0x0c, 0x73, 0xf2, 0x15, 0x00, 0x00, - 0xff, 0xff, 0x0c, 0x8d, 0xe8, 0x16, 0x1f, 0x02, 0x00, 0x00, +func init() { proto.RegisterFile("pstore.proto", fileDescriptor_pstore_fcc3073dbc5464a9) } + +var fileDescriptor_pstore_fcc3073dbc5464a9 = []byte{ + // 243 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x29, 0x28, 0x2e, 0xc9, + 0x2f, 0x4a, 0xd5, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x84, 0xf1, 0x92, 0xa4, 0x74, 0xd3, + 0x33, 0x4b, 0x32, 0x4a, 0x93, 0xf4, 0x92, 0xf3, 0x73, 0xf5, 0xd3, 0xf3, 0xd3, 0xf3, 0xf5, 0xc1, + 0x2a, 0x92, 0x4a, 0xd3, 0xc0, 0x3c, 0x30, 0x07, 0xcc, 0x82, 0xe8, 0x54, 0x3a, 0xc6, 0xc8, 0xc5, + 0xe7, 0x98, 0x92, 0x52, 0xe4, 0x94, 0x9f, 0x9f, 0x1d, 0x94, 0x9a, 0x9c, 0x5f, 0x94, 0x22, 0x24, + 0xcf, 0xc5, 0x94, 0x99, 0x22, 0xc1, 0xa8, 0xc0, 0xa8, 0xc1, 0xe3, 0xc4, 0x7f, 0xeb, 0x9e, 0x3c, + 0x77, 0x00, 0x48, 0x65, 0x40, 0x6a, 0x6a, 0x91, 0xa7, 0x4b, 0x10, 0x53, 0x66, 0x8a, 0x90, 0x25, + 0x17, 0x6b, 0x62, 0x4a, 0x4a, 0x51, 0xb1, 0x04, 0x93, 0x02, 0xb3, 0x06, 0xb7, 0x91, 0xb2, 0x1e, + 0xdc, 0x76, 0x3d, 0x54, 0xa3, 0xc0, 0x5c, 0xd7, 0xbc, 0x92, 0xa2, 0xca, 0x20, 0x88, 0x0e, 0xa9, + 0x08, 0x2e, 0x4e, 0xb8, 0x98, 0x90, 0x22, 0x17, 0x0b, 0x48, 0x14, 0x6a, 0x15, 0xef, 0xad, 0x7b, + 0xf2, 0x9c, 0x60, 0xab, 0x40, 0x2a, 0x82, 0xc0, 0x52, 0x42, 0x62, 0x5c, 0x6c, 0xa9, 0x15, 0x05, + 0x99, 0x45, 0x95, 0x12, 0x4c, 0x0a, 0x8c, 0x1a, 0xcc, 0x41, 0x50, 0x9e, 0x90, 0x00, 0x17, 0x73, + 0x49, 0x49, 0x8e, 0x04, 0x33, 0x58, 0x10, 0xc4, 0x74, 0x12, 0xf9, 0xf1, 0x50, 0x8e, 0xf1, 0xc0, + 0x23, 0x39, 0xc6, 0x13, 0x8f, 0xe4, 0x18, 0x2f, 0x3c, 0x92, 0x63, 0x7c, 0xf0, 0x48, 0x8e, 0x31, + 0x89, 0x0d, 0xec, 0x4b, 0x63, 0x40, 0x00, 0x00, 0x00, 0xff, 0xff, 0x2b, 0x91, 0xbf, 0xc2, 0x2f, + 0x01, 0x00, 0x00, } diff --git a/p2p/host/peerstore/pb/pstore.proto b/p2p/host/peerstore/pb/pstore.proto index 2a768462cd..3894534745 100644 --- a/p2p/host/peerstore/pb/pstore.proto +++ b/p2p/host/peerstore/pb/pstore.proto @@ -1,32 +1,27 @@ syntax = "proto3"; package pstore.pb; -import "google/protobuf/timestamp.proto"; -import "google/protobuf/duration.proto"; import "github.com/gogo/protobuf/gogoproto/gogo.proto"; -option (gogoproto.marshaler_all) = true; -option (gogoproto.unmarshaler_all) = true; option (gogoproto.benchgen_all) = true; option (gogoproto.populate_all) = true; // AddrBookRecord represents a record for a peer in the address book. message AddrBookRecord { // The peer ID. - bytes id = 1; + bytes id = 1 [(gogoproto.customtype) = "ProtoPeerID"]; - // The timestamp where purging needs to happen. - google.protobuf.Timestamp nextVisit = 2 [(gogoproto.stdtime) = true]; - - // The multiaddresses. - map addrs = 3; + // The multiaddresses. This is a sorted list where element 0 expires the soonest. + repeated AddrEntry addrs = 2; // AddrEntry represents a single multiaddress. message AddrEntry { + bytes addr = 1 [(gogoproto.customtype) = "ProtoAddr"]; + // The point in time when this address expires. - google.protobuf.Timestamp expiry = 2 [(gogoproto.stdtime) = true]; + int64 expiry = 2; // The original TTL of this address. - google.protobuf.Duration ttl = 3 [(gogoproto.stdduration) = true]; + int64 ttl = 3; } } diff --git a/p2p/host/peerstore/pb/pstorepb_test.go b/p2p/host/peerstore/pb/pstorepb_test.go index 6d8cbf3270..9b7ae039c5 100644 --- a/p2p/host/peerstore/pb/pstorepb_test.go +++ b/p2p/host/peerstore/pb/pstorepb_test.go @@ -10,8 +10,6 @@ import proto "github.com/gogo/protobuf/proto" import fmt "fmt" import math "math" import _ "github.com/gogo/protobuf/gogoproto" -import _ "github.com/golang/protobuf/ptypes/duration" -import _ "github.com/golang/protobuf/ptypes/timestamp" // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 7eb08b7831..0dfff50924 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -3,6 +3,7 @@ package pstoreds import ( "context" "fmt" + "sort" "sync" "time" @@ -39,24 +40,30 @@ var ( type addrsRecord struct { sync.RWMutex *pb.AddrBookRecord - - delete bool + dirty bool } -// FlushInTxn flushes the record to the datastore by calling ds.Put, unless the record is -// marked for deletion, in which case the deletion is executed. +// FlushInTxn writes the record to the datastore by calling ds.Put, unless the record is +// marked for deletion, in which case the deletion is executed via ds.Delete. func (r *addrsRecord) FlushInTxn(txn ds.Txn) (err error) { - key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id))) - if r.delete { + key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id.ID))) + if len(r.Addrs) == 0 { return txn.Delete(key) } data := pool.Get(r.Size()) defer pool.Put(data) + + // i is the number of bytes that were effectively written. i, err := r.MarshalTo(data) if err != nil { return err } - return txn.Put(key, data[:i]) + if err := txn.Put(key, data[:i]); err != nil { + return err + } + // write succeeded; record is no longer dirty. + r.dirty = false + return nil } // Flush creates a ds.Txn, and calls FlushInTxn with it. @@ -73,58 +80,56 @@ func (r *addrsRecord) Flush(ds ds.TxnDatastore) (err error) { return txn.Commit() } -// Refresh is called on in-memory entries to perform housekeeping. Refresh does the following: -// * removes all expired addresses. -// * recalculates the date in which the record needs to be revisited (earliest expiration of survivors). -// * marks the record for deletion if no addresses are left. +// Refresh is called on records to perform housekeeping. The return value signals if the record was changed +// as a result of the refresh. +// +// Refresh does the following: +// * sorts the addresses by expiration (soonest expiring first). +// * removes the addresses that have expired. // -// A `true` value of `force` tells us to proceed with housekeeping even if the `NextVisit` date has not arrived. +// It short-circuits optimistically when we know there's nothing to do. // -// Refresh is called in several occasions: -// * with force=false, when accessing and loading an entry, or when performing GC. -// * with force=true, after an entry has been modified (e.g. addresses have been added or removed, +// Refresh is called from several points: +// * when accessing and loading an entry. +// * when performing periodic GC. +// * after an entry has been modified (e.g. addresses have been added or removed, // TTLs updated, etc.) -func (r *addrsRecord) Refresh(force bool) (chgd bool) { +// +// If the return value is true, the caller can perform a flush immediately, or can schedule an async +// flush, depending on the context. +func (r *addrsRecord) Refresh() (chgd bool) { + now := time.Now().Unix() + if !r.dirty && len(r.Addrs) > 0 && r.Addrs[0].Expiry > now { + // record is not dirty, and we have no expired entries to purge. + return false + } + if len(r.Addrs) == 0 { - r.delete = true - r.NextVisit = nil + // this is a ghost record; let's signal it has to be written. + // Flush() will take care of doing the deletion. return true } - now := time.Now() - if !force && r.NextVisit != nil && !r.NextVisit.IsZero() && now.Before(*r.NextVisit) { - // no expired entries to purge, and no forced housekeeping. - return false + if r.dirty && len(r.Addrs) > 1 { + // the record has been modified, so it may need resorting. + // we keep addresses sorted by expiration, where 0 is the soonest expiring. + sort.Slice(r.Addrs, func(i, j int) bool { + return r.Addrs[i].Expiry < r.Addrs[j].Expiry + }) } - // nv stores the next visit for surviving addresses following the purge - var nv time.Time - for addr, entry := range r.Addrs { - if entry.Expiry == nil { - continue - } - if nv.IsZero() { - nv = *entry.Expiry - } - if now.After(*entry.Expiry) { - // this entry is expired; remove it. - delete(r.Addrs, addr) - chgd = true - } else if nv.After(*entry.Expiry) { - // keep track of the earliest expiry across survivors. - nv = *entry.Expiry + // since addresses are sorted by expiration, we find the first survivor and split the + // slice on its index. + pivot := -1 + for i, addr := range r.Addrs { + if addr.Expiry > now { + break } + pivot = i } - if len(r.Addrs) == 0 { - r.delete = true - r.NextVisit = nil - return true - } - - chgd = chgd || r.NextVisit == nil || nv != *r.NextVisit - r.NextVisit = &nv - return chgd + r.Addrs = r.Addrs[pivot+1:] + return r.dirty || pivot >= 0 } // dsAddrBook is an address book backed by a Datastore with a GC-like procedure @@ -145,8 +150,7 @@ type dsAddrBook struct { var _ pstore.AddrBook = (*dsAddrBook)(nil) -// NewAddrBook initializes a new address book given a -// Datastore instance, a context for managing the TTL manager, +// NewAddrBook initializes a new address book given a Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab *dsAddrBook, err error) { var cache cache = new(noopCache) @@ -168,6 +172,7 @@ func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab * closedCh: make(chan struct{}), } + // kick off periodic GC. go mgr.background() return mgr, nil @@ -178,20 +183,26 @@ func (ab *dsAddrBook) Close() { <-ab.closedCh } -func (ab *dsAddrBook) tryFlush(pr *addrsRecord) { +func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { select { case ab.flushJobCh <- pr: default: - id, _ := peer.IDFromBytes(pr.Id) - log.Warningf("flush queue is full; could not flush peer %v", id) + log.Warningf("flush queue is full; could not flush peer %v", pr.Id.ID.Pretty()) } } +// loadRecord is a read-through fetch. It fetches a record from cache, falling back to the +// datastore upon a miss, and returning an newly initialized record if the peer doesn't exist. +// +// loadRecord calls Refresh() on the record before returning it. If the record changes +// as a result and `update=true`, an async flush is scheduled. +// +// If `cache=true`, the record is inserted in the cache when loaded from the datastore. func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrsRecord, err error) { if e, ok := ab.cache.Get(id); ok { pr = e.(*addrsRecord) - if pr.Refresh(false) && update { - ab.tryFlush(pr) + if pr.Refresh() && update { + ab.asyncFlush(pr) } return pr, nil } @@ -205,46 +216,38 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(id))) data, err := txn.Get(key) + if err != nil && err != ds.ErrNotFound { + return nil, err + } + if err == nil { pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} if err = pr.Unmarshal(data); err != nil { return nil, err } - if pr.Refresh(false) { - ab.tryFlush(pr) + if pr.Refresh() && update { + ab.asyncFlush(pr) } - if cache { - ab.cache.Add(id, pr) - } - return pr, nil + } else { + pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{Id: &pb.ProtoPeerID{ID: id}}} } - if err == ds.ErrNotFound { - pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{ - Id: []byte(id), - Addrs: make(map[string]*pb.AddrBookRecord_AddrEntry), - }} - if cache { - ab.cache.Add(id, pr) - } - return pr, nil + if cache { + ab.cache.Add(id, pr) } - - log.Error(err) - return nil, err + return pr, nil } -// background is the peerstore process that takes care of: -// * purging expired addresses and peers with no addresses from the datastore at regular intervals. -// * asynchronously flushing cached entries with expired addresses to the datastore. +// background runs the housekeeping process that takes care of: +// +// * purging expired addresses from the datastore at regular intervals. +// * persisting asynchronous flushes to the datastore. func (ab *dsAddrBook) background() { timer := time.NewTicker(ab.gcInterval) - for { select { case fj := <-ab.flushJobCh: - id, _ := peer.IDFromBytes(fj.Id) - if cached, ok := ab.cache.Peek(id); ok { + if cached, ok := ab.cache.Peek(fj.Id.ID); ok { // Only continue flushing if the record we have in memory is the same as for which the flush // job was requested. If it's not in memory, it has been evicted and we don't know if we hold // the latest state or not. Similarly, if it's cached but the pointer is different, it means @@ -267,35 +270,44 @@ func (ab *dsAddrBook) background() { } } +var purgeQuery = query.Query{Prefix: addrBookBase.String()} + +// purgeCycle runs a GC cycle func (ab *dsAddrBook) purgeCycle() { var id peer.ID record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} - q := query.Query{Prefix: addrBookBase.String()} - txn, err := ab.ds.NewTransaction(false) if err != nil { log.Warningf("failed while purging entries: %v\n", err) return } - defer txn.Discard() - results, err := txn.Query(q) + results, err := txn.Query(purgeQuery) if err != nil { log.Warningf("failed while purging entries: %v\n", err) return } - defer results.Close() for result := range results.Next() { - id, _ = peer.IDFromBytes(record.Id) - - // if the record is in cache, let's refresh that one and flush it if necessary. + k, err := b32.RawStdEncoding.DecodeString(ds.RawKey(result.Key).Name()) + if err != nil { + // TODO: drop the record? this will keep failing forever. + log.Warningf("failed while purging record: %v, err: %v\n", result.Key, err) + continue + } + id, err = peer.IDFromBytes(k) + if err != nil { + // TODO: drop the record? this will keep failing forever. + log.Warningf("failed to get extract peer ID from bytes (hex): %x, err: %v\n", k, err) + continue + } + // if the record is in cache, we refresh it and flush it if necessary. if e, ok := ab.cache.Peek(id); ok { cached := e.(*addrsRecord) cached.Lock() - if cached.Refresh(false) { + if cached.Refresh() { cached.FlushInTxn(txn) } cached.Unlock() @@ -303,11 +315,11 @@ func (ab *dsAddrBook) purgeCycle() { } if err := record.Unmarshal(result.Value); err != nil { - log.Warningf("failed while purging entries: %v\n", err) + // TODO: drop the record? this will keep failing forever. + log.Warningf("failed while deserializing entry with key: %v, err: %v\n", result.Key, err) continue } - - if record.Refresh(false) { + if record.Refresh() { record.FlushInTxn(txn) } record.Reset() @@ -358,17 +370,16 @@ func (ab *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.D pr.Lock() defer pr.Unlock() - chgd, newExp := false, time.Now().Add(newTTL) + newExp := time.Now().Add(newTTL).Unix() for _, entry := range pr.Addrs { - if entry.Ttl == nil || *entry.Ttl != oldTTL { + if entry.Ttl != int64(oldTTL) { continue } - entry.Ttl, entry.Expiry = &newTTL, &newExp - chgd = true + entry.Ttl, entry.Expiry = int64(newTTL), newExp + pr.dirty = true } - if chgd { - pr.Refresh(true) + if pr.Refresh() { pr.Flush(ab.ds) } } @@ -385,10 +396,8 @@ func (ab *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { defer pr.RUnlock() addrs := make([]ma.Multiaddr, 0, len(pr.Addrs)) - for k, _ := range pr.Addrs { - if a, err := ma.NewMultiaddr(k); err == nil { - addrs = append(addrs, a) - } + for _, a := range pr.Addrs { + addrs = append(addrs, a.Addr) } return addrs } @@ -434,32 +443,54 @@ func (ab *dsAddrBook) ClearAddrs(p peer.ID) { func (ab *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode) (err error) { pr, err := ab.loadRecord(p, true, false) if err != nil { - return fmt.Errorf("failed to load peerstore entry for peer %v while deleting addrs, err: %v", p, err) + return fmt.Errorf("failed to load peerstore entry for peer %v while setting addrs, err: %v", p, err) } pr.Lock() defer pr.Unlock() - now := time.Now() - broadcast := make([]bool, len(addrs)) - newExp := now.Add(ttl) - for i, addr := range addrs { - e, ok := pr.Addrs[addr.String()] - if ok && mode == ttlExtend && e.Expiry.After(newExp) { - continue + newExp := time.Now().Add(ttl).Unix() + existed := make([]bool, len(addrs)) // keeps track of which addrs we found + +Outer: + for i, incoming := range addrs { + for _, have := range pr.Addrs { + if incoming.Equal(have.Addr) { + existed[i] = true + if mode == ttlExtend && have.Expiry > newExp { + // if we're only extending TTLs but the addr already has a longer one, we skip it. + continue Outer + } + have.Expiry = newExp + // we found the address, and addresses cannot be duplicate, + // so let's move on to the next. + continue Outer + } } - pr.Addrs[addr.String()] = &pb.AddrBookRecord_AddrEntry{Expiry: &newExp, Ttl: &ttl} - broadcast[i] = !ok } - // Update was successful, so broadcast event only for new addresses. - for i, v := range broadcast { - if v { - ab.subsManager.BroadcastAddr(p, addrs[i]) + // add addresses we didn't hold. + var added []*pb.AddrBookRecord_AddrEntry + for i, e := range existed { + if e { + continue + } + addr := addrs[i] + entry := &pb.AddrBookRecord_AddrEntry{ + Addr: &pb.ProtoAddr{Multiaddr: addr}, + Ttl: int64(ttl), + Expiry: newExp, } + added = append(added, entry) + // TODO: should we only broadcast if we updated the store successfully? + // we have no way of rolling back the state of the in-memory record, although we + // could at the expense of allocs. But is it worthwhile? + ab.subsManager.BroadcastAddr(p, addr) } - pr.Refresh(true) + pr.Addrs = append(pr.Addrs, added...) + pr.dirty = true + pr.Refresh() return pr.Flush(ab.ds) } @@ -476,11 +507,23 @@ func (ab *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) (err error) { pr.Lock() defer pr.Unlock() - for _, addr := range addrs { - delete(pr.Addrs, addr.String()) + // deletes addresses in place, and avoiding copies until we encounter the first deletion. + survived := 0 + for i, addr := range pr.Addrs { + for _, del := range addrs { + if addr.Addr.Equal(del) { + continue + } + if i != survived { + pr.Addrs[survived] = pr.Addrs[i] + } + survived++ + } } + pr.Addrs = pr.Addrs[:survived] - pr.Refresh(true) + pr.dirty = true + pr.Refresh() return pr.Flush(ab.ds) } From 909782c975cfafaea4551470aee64b8ecd15d983 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 15 Nov 2018 13:40:42 +0000 Subject: [PATCH 0841/3965] fix makefile for protobuf. --- p2p/host/peerstore/pb/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pb/Makefile b/p2p/host/peerstore/pb/Makefile index eb14b5768a..cc99b57246 100644 --- a/p2p/host/peerstore/pb/Makefile +++ b/p2p/host/peerstore/pb/Makefile @@ -8,4 +8,5 @@ all: $(GO) clean: rm -f *.pb.go - rm -f *.go + +.PHONY: clean \ No newline at end of file From 664ef3235f3beacd2a15a532d137c52ab745938f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 15 Nov 2018 13:55:53 +0000 Subject: [PATCH 0842/3965] adjust GC interval in tests. --- p2p/host/peerstore/pstoreds/ds_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index c7d673ab3b..a3cc24f6cf 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -37,7 +37,7 @@ func TestDsAddrBook(t *testing.T) { t.Parallel() opts := DefaultOpts() - opts.GCInterval = 100 * time.Microsecond + opts.GCInterval = 1 * time.Second opts.CacheSize = 1024 pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) @@ -47,7 +47,7 @@ func TestDsAddrBook(t *testing.T) { t.Parallel() opts := DefaultOpts() - opts.GCInterval = 100 * time.Microsecond + opts.GCInterval = 1 * time.Second opts.CacheSize = 0 pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) From a6534f5f441aa6a5daf750e3d7639d6612fd6978 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 15 Nov 2018 18:39:38 +0000 Subject: [PATCH 0843/3965] add Keys() function to private cache interface for completeness. --- p2p/host/peerstore/pstoreds/cache.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/host/peerstore/pstoreds/cache.go b/p2p/host/peerstore/pstoreds/cache.go index 2e20ae6294..d3d43aae1f 100644 --- a/p2p/host/peerstore/pstoreds/cache.go +++ b/p2p/host/peerstore/pstoreds/cache.go @@ -8,12 +8,15 @@ type cache interface { Remove(key interface{}) Contains(key interface{}) bool Peek(key interface{}) (value interface{}, ok bool) + Keys() []interface{} } // noopCache is a dummy implementation that's used when the cache is disabled. type noopCache struct { } +var _ cache = (*noopCache)(nil) + func (*noopCache) Get(key interface{}) (value interface{}, ok bool) { return nil, false } @@ -31,3 +34,7 @@ func (*noopCache) Contains(key interface{}) bool { func (*noopCache) Peek(key interface{}) (value interface{}, ok bool) { return nil, false } + +func (*noopCache) Keys() (keys []interface{}) { + return keys +} From 10c51a8e887d84a7fbfdb591304274c2effeab0d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 15 Nov 2018 18:41:25 +0000 Subject: [PATCH 0844/3965] use a Pool for address record objects. --- p2p/host/peerstore/pstoreds/addr_book.go | 30 ++++++++++++++++++++---- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 0dfff50924..3eae391d10 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -17,8 +17,8 @@ import ( pb "github.com/libp2p/go-libp2p-peerstore/pb" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" - lru "github.com/hashicorp/golang-lru" ma "github.com/multiformats/go-multiaddr" + lru "github.com/raulk/golang-lru" b32 "github.com/whyrusleeping/base32" ) @@ -34,6 +34,14 @@ var ( // Peer addresses are stored under the following db key pattern: // /peers/addr// addrBookBase = ds.NewKey("/peers/addrs") + arPool = &sync.Pool{ + New: func() interface{} { + return &addrsRecord{ + AddrBookRecord: &pb.AddrBookRecord{}, + dirty: false, + } + }, + } ) // addrsRecord decorates the AddrBookRecord with locks and metadata. @@ -43,6 +51,11 @@ type addrsRecord struct { dirty bool } +func (r *addrsRecord) Reset() { + r.AddrBookRecord.Reset() + r.dirty = false +} + // FlushInTxn writes the record to the datastore by calling ds.Put, unless the record is // marked for deletion, in which case the deletion is executed via ds.Delete. func (r *addrsRecord) FlushInTxn(txn ds.Txn) (err error) { @@ -155,7 +168,11 @@ var _ pstore.AddrBook = (*dsAddrBook)(nil) func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab *dsAddrBook, err error) { var cache cache = new(noopCache) if opts.CacheSize > 0 { - if cache, err = lru.NewARC(int(opts.CacheSize)); err != nil { + evictCallback := func(key interface{}, value interface{}) { + value.(*addrsRecord).Reset() + arPool.Put(value) + } + if cache, err = lru.NewARCWithEvict(int(opts.CacheSize), evictCallback); err != nil { return nil, err } } @@ -221,7 +238,7 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs } if err == nil { - pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} + pr = arPool.Get().(*addrsRecord) if err = pr.Unmarshal(data); err != nil { return nil, err } @@ -229,7 +246,8 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs ab.asyncFlush(pr) } } else { - pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{Id: &pb.ProtoPeerID{ID: id}}} + pr = arPool.Get().(*addrsRecord) + pr.Id = &pb.ProtoPeerID{ID: id} } if cache { @@ -275,7 +293,9 @@ var purgeQuery = query.Query{Prefix: addrBookBase.String()} // purgeCycle runs a GC cycle func (ab *dsAddrBook) purgeCycle() { var id peer.ID - record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} + record := arPool.Get().(*addrsRecord) + defer arPool.Put(record) + txn, err := ab.ds.NewTransaction(false) if err != nil { log.Warningf("failed while purging entries: %v\n", err) From 374d8d9fa4bf12961fc804f5c0479225a5405d03 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 15 Nov 2018 21:11:10 -0800 Subject: [PATCH 0845/3965] warn when we encounter a useless transport --- p2p/net/swarm/swarm_transport.go | 4 +++ p2p/net/swarm/transport_test.go | 47 ++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100644 p2p/net/swarm/transport_test.go diff --git a/p2p/net/swarm/swarm_transport.go b/p2p/net/swarm/swarm_transport.go index c96a079d9e..0431488777 100644 --- a/p2p/net/swarm/swarm_transport.go +++ b/p2p/net/swarm/swarm_transport.go @@ -70,6 +70,10 @@ func (s *Swarm) TransportForListening(a ma.Multiaddr) transport.Transport { func (s *Swarm) AddTransport(t transport.Transport) error { protocols := t.Protocols() + if len(protocols) == 0 { + return fmt.Errorf("useless transport handles no protocols: %T", t) + } + s.transports.Lock() defer s.transports.Unlock() var registered []string diff --git a/p2p/net/swarm/transport_test.go b/p2p/net/swarm/transport_test.go new file mode 100644 index 0000000000..018775b261 --- /dev/null +++ b/p2p/net/swarm/transport_test.go @@ -0,0 +1,47 @@ +package swarm_test + +import ( + "context" + "testing" + + swarmt "github.com/libp2p/go-libp2p-swarm/testing" + + peer "github.com/libp2p/go-libp2p-peer" + transport "github.com/libp2p/go-libp2p-transport" + ma "github.com/multiformats/go-multiaddr" +) + +type dummyTransport struct { + protocols []int + proxy bool +} + +func (dt *dummyTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (transport.Conn, error) { + panic("unimplemented") +} + +func (dt *dummyTransport) CanDial(addr ma.Multiaddr) bool { + panic("unimplemented") +} + +func (dt *dummyTransport) Listen(laddr ma.Multiaddr) (transport.Listener, error) { + panic("unimplemented") +} + +func (dt *dummyTransport) Proxy() bool { + return dt.proxy +} + +func (dt *dummyTransport) Protocols() []int { + return dt.protocols +} + +func TestUselessTransport(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + swarm := swarmt.GenSwarm(t, ctx) + err := swarm.AddTransport(new(dummyTransport)) + if err == nil { + t.Fatal("adding a transport that supports no protocols should have failed") + } +} From 388a2945786bdff68b0815a6be4e7e0c3fa11bed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 16 Nov 2018 12:16:03 +0000 Subject: [PATCH 0846/3965] Fix badges and links on README.md --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 2b76e680d9..6d02c0764a 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@

The Go implementation of the libp2p Networking Stack.

- - - + + +

@@ -16,7 +16,7 @@
- +

@@ -73,7 +73,7 @@ There is currently only one bundle of `go-libp2p`, this package. This bundle is ### API -[![GoDoc](https://godoc.org/github.com/ipfs/go-libp2p?status.svg)](https://godoc.org/github.com/libp2p/go-libp2p) +[![GoDoc](https://godoc.org/github.com/libp2p/go-libp2p?status.svg)](https://godoc.org/github.com/libp2p/go-libp2p) ### Examples From b3274ae95015be38b771a7e7f3d2cb098ea7cc9d Mon Sep 17 00:00:00 2001 From: can Date: Tue, 20 Nov 2018 18:00:12 +0800 Subject: [PATCH 0847/3965] Fix missing transport in dialed connection --- p2p/transport/quic/transport.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index c58b7b0a4e..212ddf4e46 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -143,6 +143,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp } return &conn{ sess: sess, + transport: t, privKey: t.privKey, localPeer: t.localPeer, localMultiaddr: localMultiaddr, From e15828972d59c96952829fe19e0784ad71884721 Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 23 Nov 2018 09:41:49 +0000 Subject: [PATCH 0848/3965] docs: libp2p yellow --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 6d02c0764a..bfced51bde 100644 --- a/README.md +++ b/README.md @@ -6,10 +6,10 @@

The Go implementation of the libp2p Networking Stack.

- - - - + + + +

From 82f9c87252708a75ebf6d0f204ed438f636681c0 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 24 Nov 2018 15:33:50 +0700 Subject: [PATCH 0849/3965] initial commit --- p2p/security/tls/conn.go | 37 +++++ p2p/security/tls/crypto.go | 140 +++++++++++++++++++ p2p/security/tls/libp2p_tls_suite_test.go | 13 ++ p2p/security/tls/transport.go | 81 +++++++++++ p2p/security/tls/transport_test.go | 161 ++++++++++++++++++++++ 5 files changed, 432 insertions(+) create mode 100644 p2p/security/tls/conn.go create mode 100644 p2p/security/tls/crypto.go create mode 100644 p2p/security/tls/libp2p_tls_suite_test.go create mode 100644 p2p/security/tls/transport.go create mode 100644 p2p/security/tls/transport_test.go diff --git a/p2p/security/tls/conn.go b/p2p/security/tls/conn.go new file mode 100644 index 0000000000..f4eb600089 --- /dev/null +++ b/p2p/security/tls/conn.go @@ -0,0 +1,37 @@ +package libp2ptls + +import ( + "crypto/tls" + + cs "github.com/libp2p/go-conn-security" + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" +) + +type conn struct { + *tls.Conn + + localPeer peer.ID + privKey ic.PrivKey + + remotePeer peer.ID + remotePubKey ic.PubKey +} + +var _ cs.Conn = &conn{} + +func (c *conn) LocalPeer() peer.ID { + return c.localPeer +} + +func (c *conn) LocalPrivateKey() ic.PrivKey { + return c.privKey +} + +func (c *conn) RemotePeer() peer.ID { + return c.remotePeer +} + +func (c *conn) RemotePublicKey() ic.PubKey { + return c.remotePubKey +} diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go new file mode 100644 index 0000000000..1a6812a18e --- /dev/null +++ b/p2p/security/tls/crypto.go @@ -0,0 +1,140 @@ +package libp2ptls + +import ( + "crypto/rand" + "crypto/tls" + "crypto/x509" + "errors" + "math/big" + "time" + + "github.com/gogo/protobuf/proto" + ic "github.com/libp2p/go-libp2p-crypto" + pb "github.com/libp2p/go-libp2p-crypto/pb" + peer "github.com/libp2p/go-libp2p-peer" +) + +// Identity is used to secure connections +type Identity struct { + *tls.Config +} + +// NewIdentity creates a new identity +func NewIdentity(privKey ic.PrivKey) (*Identity, error) { + conf, err := generateConfig(privKey) + if err != nil { + return nil, err + } + return &Identity{conf}, nil +} + +// ConfigForPeer creates a new tls.Config that verifies the peers certificate chain. +// It should be used to create a new tls.Config before dialing. +func (i *Identity) ConfigForPeer(remote peer.ID) *tls.Config { + // We need to check the peer ID in the VerifyPeerCertificate callback. + // The tls.Config it is also used for listening, and we might also have concurrent dials. + // Clone it so we can check for the specific peer ID we're dialing here. + conf := i.Config.Clone() + // We're using InsecureSkipVerify, so the verifiedChains parameter will always be empty. + // We need to parse the certificates ourselves from the raw certs. + conf.VerifyPeerCertificate = func(rawCerts [][]byte, _ [][]*x509.Certificate) error { + chain := make([]*x509.Certificate, len(rawCerts)) + for i := 0; i < len(rawCerts); i++ { + cert, err := x509.ParseCertificate(rawCerts[i]) + if err != nil { + return err + } + chain[i] = cert + } + pubKey, err := getRemotePubKey(chain) + if err != nil { + return err + } + if !remote.MatchesPublicKey(pubKey) { + return errors.New("peer IDs don't match") + } + return nil + } + return conf +} + +// KeyFromChain takes a chain of x509.Certificates and returns the peer's public key. +func KeyFromChain(chain []*x509.Certificate) (ic.PubKey, error) { + return getRemotePubKey(chain) +} + +const certValidityPeriod = 180 * 24 * time.Hour + +func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { + key, cert, err := keyToCertificate(privKey) + if err != nil { + return nil, err + } + return &tls.Config{ + InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. + ClientAuth: tls.RequireAnyClientCert, + Certificates: []tls.Certificate{{ + Certificate: [][]byte{cert.Raw}, + PrivateKey: key, + }}, + }, nil +} + +func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { + if len(chain) != 1 { + return nil, errors.New("expected one certificates in the chain") + } + pool := x509.NewCertPool() + pool.AddCert(chain[0]) + if _, err := chain[0].Verify(x509.VerifyOptions{Roots: pool}); err != nil { + return nil, err + } + remotePubKey, err := x509.MarshalPKIXPublicKey(chain[0].PublicKey) + if err != nil { + return nil, err + } + return ic.UnmarshalRsaPublicKey(remotePubKey) +} + +func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { + sn, err := rand.Int(rand.Reader, big.NewInt(1<<62)) + if err != nil { + return nil, nil, err + } + tmpl := &x509.Certificate{ + SerialNumber: sn, + NotBefore: time.Now().Add(-24 * time.Hour), + NotAfter: time.Now().Add(certValidityPeriod), + } + + var publicKey, privateKey interface{} + keyBytes, err := sk.Bytes() + if err != nil { + return nil, nil, err + } + pbmes := new(pb.PrivateKey) + if err := proto.Unmarshal(keyBytes, pbmes); err != nil { + return nil, nil, err + } + switch pbmes.GetType() { + case pb.KeyType_RSA: + k, err := x509.ParsePKCS1PrivateKey(pbmes.GetData()) + if err != nil { + return nil, nil, err + } + publicKey = &k.PublicKey + privateKey = k + // TODO: add support for ECDSA + default: + return nil, nil, errors.New("unsupported key type for TLS") + } + certDER, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, publicKey, privateKey) + if err != nil { + return nil, nil, err + } + cert, err := x509.ParseCertificate(certDER) + if err != nil { + return nil, nil, err + } + return privateKey, cert, nil +} diff --git a/p2p/security/tls/libp2p_tls_suite_test.go b/p2p/security/tls/libp2p_tls_suite_test.go new file mode 100644 index 0000000000..3303ed3030 --- /dev/null +++ b/p2p/security/tls/libp2p_tls_suite_test.go @@ -0,0 +1,13 @@ +package libp2ptls + +import ( + "testing" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +func TestLibp2pTLS(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "libp2p TLS Suite") +} diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go new file mode 100644 index 0000000000..16941bc059 --- /dev/null +++ b/p2p/security/tls/transport.go @@ -0,0 +1,81 @@ +package libp2ptls + +import ( + "context" + "crypto/tls" + "net" + + cs "github.com/libp2p/go-conn-security" + ci "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" +) + +// ID is the protocol ID (used when negotiating with multistream) +const ID = "/tls/1.0.0" + +// Transport constructs secure communication sessions for a peer. +type Transport struct { + identity *Identity + + localPeer peer.ID + privKey ci.PrivKey +} + +// New creates a TLS encrypted transport +func New(key ci.PrivKey) (*Transport, error) { + id, err := peer.IDFromPrivateKey(key) + if err != nil { + return nil, err + } + identity, err := NewIdentity(key) + if err != nil { + return nil, err + } + return &Transport{ + identity: identity, + localPeer: id, + privKey: key, + }, nil +} + +var _ cs.Transport = &Transport{} + +// SecureInbound runs the TLS handshake as a server. +func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Conn, error) { + serv := tls.Server(insecure, t.identity.Config) + // TODO: use the ctx + // see https://github.com/golang/go/issues/18482 + if err := serv.Handshake(); err != nil { + return nil, err + } + return t.setupConn(serv) +} + +// SecureOutbound runs the TLS handshake as a client. +func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { + cl := tls.Client(insecure, t.identity.ConfigForPeer(p)) + // TODO: use the ctx + // see https://github.com/golang/go/issues/18482 + if err := cl.Handshake(); err != nil { + return nil, err + } + return t.setupConn(cl) +} + +func (t *Transport) setupConn(tlsConn *tls.Conn) (cs.Conn, error) { + remotePubKey, err := KeyFromChain(tlsConn.ConnectionState().PeerCertificates) + if err != nil { + return nil, err + } + remotePeerID, err := peer.IDFromPublicKey(remotePubKey) + if err != nil { + return nil, err + } + return &conn{ + Conn: tlsConn, + localPeer: t.localPeer, + privKey: t.privKey, + remotePeer: remotePeerID, + remotePubKey: remotePubKey, + }, nil +} diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go new file mode 100644 index 0000000000..1a55433e13 --- /dev/null +++ b/p2p/security/tls/transport_test.go @@ -0,0 +1,161 @@ +package libp2ptls + +import ( + "context" + "crypto/rand" + "crypto/rsa" + "crypto/x509" + "net" + + cs "github.com/libp2p/go-conn-security" + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +var _ = Describe("Transport", func() { + var ( + serverKey, clientKey ic.PrivKey + serverID, clientID peer.ID + ) + + createPeer := func() (peer.ID, ic.PrivKey) { + key, err := rsa.GenerateKey(rand.Reader, 1024) + Expect(err).ToNot(HaveOccurred()) + priv, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(key)) + Expect(err).ToNot(HaveOccurred()) + id, err := peer.IDFromPrivateKey(priv) + Expect(err).ToNot(HaveOccurred()) + return id, priv + } + + connect := func() (net.Conn, net.Conn) { + ln, err := net.Listen("tcp", "localhost:0") + Expect(err).ToNot(HaveOccurred()) + defer ln.Close() + serverConnChan := make(chan net.Conn) + go func() { + defer GinkgoRecover() + conn, err := ln.Accept() + Expect(err).ToNot(HaveOccurred()) + serverConnChan <- conn + }() + conn, err := net.Dial("tcp", ln.Addr().String()) + Expect(err).ToNot(HaveOccurred()) + return conn, <-serverConnChan + } + + // modify the cert chain such that verificiation will fail + invalidateCertChain := func(identity *Identity) { + key, err := rsa.GenerateKey(rand.Reader, 1024) + Expect(err).ToNot(HaveOccurred()) + identity.Config.Certificates[0].PrivateKey = key + } + + BeforeEach(func() { + serverID, serverKey = createPeer() + clientID, clientKey = createPeer() + }) + + It("handshakes", func() { + clientTransport, err := New(clientKey) + Expect(err).ToNot(HaveOccurred()) + serverTransport, err := New(serverKey) + Expect(err).ToNot(HaveOccurred()) + + clientInsecureConn, serverInsecureConn := connect() + + serverConnChan := make(chan cs.Conn) + go func() { + defer GinkgoRecover() + serverConn, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) + Expect(err).ToNot(HaveOccurred()) + serverConnChan <- serverConn + }() + clientConn, err := clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) + Expect(err).ToNot(HaveOccurred()) + var serverConn cs.Conn + Eventually(serverConnChan).Should(Receive(&serverConn)) + defer clientConn.Close() + defer serverConn.Close() + Expect(clientConn.LocalPeer()).To(Equal(clientID)) + Expect(serverConn.LocalPeer()).To(Equal(serverID)) + Expect(clientConn.LocalPrivateKey()).To(Equal(clientKey)) + Expect(serverConn.LocalPrivateKey()).To(Equal(serverKey)) + Expect(clientConn.RemotePeer()).To(Equal(serverID)) + Expect(serverConn.RemotePeer()).To(Equal(clientID)) + Expect(clientConn.RemotePublicKey()).To(Equal(serverKey.GetPublic())) + Expect(serverConn.RemotePublicKey()).To(Equal(clientKey.GetPublic())) + // exchange some data + _, err = serverConn.Write([]byte("foobar")) + Expect(err).ToNot(HaveOccurred()) + b := make([]byte, 6) + _, err = clientConn.Read(b) + Expect(err).ToNot(HaveOccurred()) + Expect(string(b)).To(Equal("foobar")) + }) + + It("fails if the peer ID doesn't match", func() { + thirdPartyID, _ := createPeer() + + serverTransport, err := New(serverKey) + Expect(err).ToNot(HaveOccurred()) + clientTransport, err := New(clientKey) + Expect(err).ToNot(HaveOccurred()) + + clientInsecureConn, serverInsecureConn := connect() + + go func() { + defer GinkgoRecover() + _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("tls: bad certificate")) + }() + // dial, but expect the wrong peer ID + _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, thirdPartyID) + Expect(err).To(MatchError("peer IDs don't match")) + }) + + It("fails if the client presents an invalid cert chain", func() { + serverTransport, err := New(serverKey) + Expect(err).ToNot(HaveOccurred()) + clientTransport, err := New(clientKey) + Expect(err).ToNot(HaveOccurred()) + invalidateCertChain(clientTransport.identity) + + clientInsecureConn, serverInsecureConn := connect() + + go func() { + defer GinkgoRecover() + _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("crypto/rsa: verification error")) + }() + + _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("tls: bad certificate")) + }) + + It("fails if the server presents an invalid cert chain", func() { + serverTransport, err := New(serverKey) + Expect(err).ToNot(HaveOccurred()) + invalidateCertChain(serverTransport.identity) + clientTransport, err := New(clientKey) + Expect(err).ToNot(HaveOccurred()) + + clientInsecureConn, serverInsecureConn := connect() + + go func() { + defer GinkgoRecover() + _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("tls: bad certificate")) + }() + + _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("crypto/rsa: verification error")) + }) +}) From b70779f12c0760d1718c98f7faa7af2afca5b3c1 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 24 Nov 2018 15:34:45 +0700 Subject: [PATCH 0850/3965] add a license --- p2p/security/tls/LICENSE.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 p2p/security/tls/LICENSE.md diff --git a/p2p/security/tls/LICENSE.md b/p2p/security/tls/LICENSE.md new file mode 100644 index 0000000000..a94e82cceb --- /dev/null +++ b/p2p/security/tls/LICENSE.md @@ -0,0 +1,7 @@ +Copyright 2018 Marten Seemann + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. From 09e1e2ad8c68f9433447423e4d17c5b8fea2ce08 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 24 Nov 2018 16:41:08 +0700 Subject: [PATCH 0851/3965] fix handshake tests --- p2p/security/tls/transport_test.go | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 1a55433e13..5e50868c95 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -106,15 +106,18 @@ var _ = Describe("Transport", func() { clientInsecureConn, serverInsecureConn := connect() + done := make(chan struct{}) go func() { defer GinkgoRecover() _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) Expect(err).To(HaveOccurred()) Expect(err.Error()).To(ContainSubstring("tls: bad certificate")) + close(done) }() // dial, but expect the wrong peer ID _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, thirdPartyID) Expect(err).To(MatchError("peer IDs don't match")) + Eventually(done).Should(BeClosed()) }) It("fails if the client presents an invalid cert chain", func() { @@ -126,16 +129,19 @@ var _ = Describe("Transport", func() { clientInsecureConn, serverInsecureConn := connect() + done := make(chan struct{}) go func() { defer GinkgoRecover() _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) Expect(err).To(HaveOccurred()) Expect(err.Error()).To(ContainSubstring("crypto/rsa: verification error")) + close(done) }() _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) Expect(err).To(HaveOccurred()) Expect(err.Error()).To(ContainSubstring("tls: bad certificate")) + Eventually(done).Should(BeClosed()) }) It("fails if the server presents an invalid cert chain", func() { @@ -147,15 +153,18 @@ var _ = Describe("Transport", func() { clientInsecureConn, serverInsecureConn := connect() + done := make(chan struct{}) go func() { defer GinkgoRecover() _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("tls: bad certificate")) + // TLS returns a weird error here: "remote error: tls: unexpected message" + close(done) }() _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) Expect(err).To(HaveOccurred()) Expect(err.Error()).To(ContainSubstring("crypto/rsa: verification error")) + Eventually(done).Should(BeClosed()) }) }) From a52733e7c43f925ac920e60b4f7914e7b8e7bd8d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 26 Nov 2018 15:10:44 +0000 Subject: [PATCH 0852/3965] implement two-tiered lookahead-based GC procedure. The GC procedure consists of two tiers: lookahead and purge. The lookahead tier runs less frequently (default 12 hours) than the purge tier (default 5 minutes). A lookahead round traverses the entire store and picks entries that need to be visited in the current lookahead window, pinning those entries in a separate region in the KV store, indexed by the next visit timestamp. A purge round iterates over the lookahead window only, and refreshes the entries that require a visit. It removes them from the lookahead region unless the entry requires another visit within the current window. --- p2p/host/peerstore/pstoreds/addr_book.go | 248 +++++++++++++++++++---- p2p/host/peerstore/pstoreds/ds_test.go | 4 +- p2p/host/peerstore/pstoreds/peerstore.go | 19 +- 3 files changed, 227 insertions(+), 44 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 3eae391d10..2a1caf62d5 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -4,14 +4,15 @@ import ( "context" "fmt" "sort" + "strconv" "sync" + "sync/atomic" "time" ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - pool "github.com/libp2p/go-buffer-pool" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" pb "github.com/libp2p/go-libp2p-peerstore/pb" @@ -32,9 +33,12 @@ const ( var ( log = logging.Logger("peerstore/ds") // Peer addresses are stored under the following db key pattern: - // /peers/addr// + // /peers/addrs/ addrBookBase = ds.NewKey("/peers/addrs") - arPool = &sync.Pool{ + // GC lookahead entries are stored in keys with pattern: + // /peers/gc/addrs// => nil + gcLookaheadBase = ds.NewKey("/peers/gc/addrs") + arPool = &sync.Pool{ New: func() interface{} { return &addrsRecord{ AddrBookRecord: &pb.AddrBookRecord{}, @@ -63,15 +67,12 @@ func (r *addrsRecord) FlushInTxn(txn ds.Txn) (err error) { if len(r.Addrs) == 0 { return txn.Delete(key) } - data := pool.Get(r.Size()) - defer pool.Put(data) - - // i is the number of bytes that were effectively written. - i, err := r.MarshalTo(data) + // cannot use a buffer pool because data is retained in the txn until it's committed or discarded. + data, err := r.Marshal() if err != nil { return err } - if err := txn.Put(key, data[:i]); err != nil { + if err = txn.Put(key, data); err != nil { return err } // write succeeded; record is no longer dirty. @@ -148,9 +149,9 @@ func (r *addrsRecord) Refresh() (chgd bool) { // dsAddrBook is an address book backed by a Datastore with a GC-like procedure // to purge expired entries. It uses an in-memory address stream manager. type dsAddrBook struct { - ctx context.Context - gcInterval time.Duration - gcMaxPurgePerCycle int + ctx context.Context + + opts Options cache cache ds ds.TxnDatastore @@ -159,6 +160,9 @@ type dsAddrBook struct { flushJobCh chan *addrsRecord cancelFn func() closedCh chan struct{} + + gcCurrWindowEnd int64 + gcLookaheadRunning int32 } var _ pstore.AddrBook = (*dsAddrBook)(nil) @@ -181,7 +185,7 @@ func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab * mgr := &dsAddrBook{ ctx: ctx, cancelFn: cancelFn, - gcInterval: opts.GCInterval, + opts: opts, cache: cache, ds: store, subsManager: pstoremem.NewAddrSubManager(), @@ -258,10 +262,21 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs // background runs the housekeeping process that takes care of: // -// * purging expired addresses from the datastore at regular intervals. +// * GCing expired addresses from the datastore at regular intervals. // * persisting asynchronous flushes to the datastore. func (ab *dsAddrBook) background() { - timer := time.NewTicker(ab.gcInterval) + // placeholder tickers. + pruneTimer, lookaheadTimer := new(time.Ticker), new(time.Ticker) + + // populate the tickers after the initial delay has passed. + go func() { + select { + case <-time.After(ab.opts.GCInitialDelay): + pruneTimer = time.NewTicker(ab.opts.GCPruneInterval) + lookaheadTimer = time.NewTicker(ab.opts.GCLookaheadInterval) + } + }() + for { select { case fj := <-ab.flushJobCh: @@ -277,77 +292,240 @@ func (ab *dsAddrBook) background() { } } - case <-timer.C: + case <-pruneTimer.C: ab.purgeCycle() + case <-lookaheadTimer.C: + ab.populateLookahead() + case <-ab.ctx.Done(): - timer.Stop() + pruneTimer.Stop() + lookaheadTimer.Stop() close(ab.closedCh) return } } } -var purgeQuery = query.Query{Prefix: addrBookBase.String()} +var purgeQuery = query.Query{Prefix: gcLookaheadBase.String(), KeysOnly: true} -// purgeCycle runs a GC cycle +// purgeCycle runs a single GC cycle, operating within the lookahead window. +// +// It scans the lookahead region for entries that need to be visited, and performs a refresh on them. An errors trigger +// the removal of the GC entry, in order to prevent unactionable items from accumulating. If the error happened to be +// temporary, the entry will be revisited in the next lookahead window. func (ab *dsAddrBook) purgeCycle() { + if atomic.LoadInt32(&ab.gcLookaheadRunning) > 0 { + // yield if lookahead is running. + return + } + var id peer.ID record := arPool.Get().(*addrsRecord) + record.Reset() defer arPool.Put(record) txn, err := ab.ds.NewTransaction(false) if err != nil { - log.Warningf("failed while purging entries: %v\n", err) + log.Warningf("failed while purging entries: %v", err) return } defer txn.Discard() + // This function drops an unparseable GC entry; this is for safety. It is an escape hatch in case + // we modify the format of keys going forward. If a user runs a new version against an old DB, + // if we don't clean up unparseable entries we'll end up accumulating garbage. + dropInError := func(key ds.Key, err error, msg string) { + if err != nil { + log.Warningf("failed while %s with GC key: %v, err: %v", msg, key, err) + } + if err = txn.Delete(key); err != nil { + log.Warningf("failed to delete corrupt GC lookahead entry: %v, err: %v", key, err) + } + } + + // This function drops a GC key if the entry is refreshed correctly. It may reschedule another visit + // if the next earliest expiry falls within the current window again. + dropOrReschedule := func(key ds.Key, ar *addrsRecord) { + if err = txn.Delete(key); err != nil { + log.Warningf("failed to delete lookahead entry: %v, err: %v", key, err) + } + + // re-add the record if it needs to be visited again in this window. + if len(ar.Addrs) != 0 && ar.Addrs[0].Expiry <= ab.gcCurrWindowEnd { + gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", ar.Addrs[0].Expiry, key.Name())) + if err = txn.Put(gcKey, []byte{}); err != nil { + log.Warningf("failed to add new GC key: %v, err: %v", gcKey, err) + } + } + } + results, err := txn.Query(purgeQuery) if err != nil { - log.Warningf("failed while purging entries: %v\n", err) + log.Warningf("failed while fetching entries to purge: %v", err) return } defer results.Close() + now := time.Now().Unix() + + // keys: /peers/gc/addrs// + // values: nil for result := range results.Next() { - k, err := b32.RawStdEncoding.DecodeString(ds.RawKey(result.Key).Name()) + gcKey := ds.RawKey(result.Key) + + ts, err := strconv.ParseInt(gcKey.Parent().Name(), 10, 64) + if err != nil { + dropInError(gcKey, err, "parsing timestamp") + log.Warningf("failed while parsing timestamp from key: %v, err: %v", result.Key, err) + continue + } else if ts > now { + // this is an ordered cursor; when we hit an entry with a timestamp beyond now, we can break. + break + } + + idb32, err := b32.RawStdEncoding.DecodeString(gcKey.Name()) if err != nil { - // TODO: drop the record? this will keep failing forever. - log.Warningf("failed while purging record: %v, err: %v\n", result.Key, err) + dropInError(gcKey, err, "parsing peer ID") + log.Warningf("failed while parsing b32 peer ID from key: %v, err: %v", result.Key, err) continue } - id, err = peer.IDFromBytes(k) + + id, err = peer.IDFromBytes(idb32) if err != nil { - // TODO: drop the record? this will keep failing forever. - log.Warningf("failed to get extract peer ID from bytes (hex): %x, err: %v\n", k, err) + dropInError(gcKey, err, "decoding peer ID") + log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) continue } + // if the record is in cache, we refresh it and flush it if necessary. if e, ok := ab.cache.Peek(id); ok { cached := e.(*addrsRecord) cached.Lock() if cached.Refresh() { - cached.FlushInTxn(txn) + if err = cached.FlushInTxn(txn); err != nil { + log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) + } } + dropOrReschedule(gcKey, cached) cached.Unlock() continue } - if err := record.Unmarshal(result.Value); err != nil { - // TODO: drop the record? this will keep failing forever. - log.Warningf("failed while deserializing entry with key: %v, err: %v\n", result.Key, err) + record.Reset() + + // otherwise, fetch it from the store, refresh it and flush it. + entryKey := addrBookBase.ChildString(gcKey.Name()) + val, err := txn.Get(entryKey) + if err != nil { + // captures all errors, including ErrNotFound. + dropInError(gcKey, err, "fetching entry") + continue + } + err = record.Unmarshal(val) + if err != nil { + dropInError(gcKey, err, "unmarshalling entry") continue } if record.Refresh() { - record.FlushInTxn(txn) + err = record.FlushInTxn(txn) + if err != nil { + log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) + } } + dropOrReschedule(gcKey, record) + } + + err = txn.Commit() + if err != nil { + log.Warningf("failed to commit GC prune transaction: %v", err) + } +} + +var populateLookaheadQuery = query.Query{Prefix: addrBookBase.String(), KeysOnly: true} + +// populateLookahead populates the lookahead window by scanning the entire store and picking entries whose earliest +// expiration falls within the new window. +// +// Those entries are stored in the lookahead region in the store, indexed by the timestamp when they need to be +// visited, to facilitate temporal range scans. +func (ab *dsAddrBook) populateLookahead() { + if !atomic.CompareAndSwapInt32(&ab.gcLookaheadRunning, 0, 1) { + return + } + + until := time.Now().Add(ab.opts.GCLookaheadInterval).Unix() + + var id peer.ID + record := arPool.Get().(*addrsRecord) + defer arPool.Put(record) + + txn, err := ab.ds.NewTransaction(false) + if err != nil { + log.Warningf("failed while filling lookahead GC region: %v", err) + return + } + defer txn.Discard() + + results, err := txn.Query(populateLookaheadQuery) + if err != nil { + log.Warningf("failed while filling lookahead GC region: %v", err) + return + } + defer results.Close() + + for result := range results.Next() { + idb32 := ds.RawKey(result.Key).Name() + k, err := b32.RawStdEncoding.DecodeString(idb32) + if err != nil { + log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) + continue + } + if id, err = peer.IDFromBytes(k); err != nil { + log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) + } + + // if the record is in cache, use the cached version. + if e, ok := ab.cache.Peek(id); ok { + cached := e.(*addrsRecord) + cached.RLock() + if len(cached.Addrs) == 0 || cached.Addrs[0].Expiry > until { + cached.RUnlock() + continue + } + gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", cached.Addrs[0].Expiry, idb32)) + if err = txn.Put(gcKey, []byte{}); err != nil { + log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) + } + cached.RUnlock() + continue + } + record.Reset() + + val, err := txn.Get(ds.RawKey(result.Key)) + if err != nil { + log.Warningf("failed which getting record from store for peer: %v, err: %v", id.Pretty(), err) + continue + } + if err := record.Unmarshal(val); err != nil { + log.Warningf("failed while unmarshalling record from store for peer: %v, err: %v", id.Pretty(), err) + continue + } + if len(record.Addrs) > 0 && record.Addrs[0].Expiry <= until { + gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", record.Addrs[0].Expiry, idb32)) + if err = txn.Put(gcKey, []byte{}); err != nil { + log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) + } + } } if err = txn.Commit(); err != nil { - log.Warningf("failed to commit GC transaction: %v\n", err) + log.Warningf("failed to commit GC lookahead transaction: %v", err) } + + ab.gcCurrWindowEnd = until + atomic.StoreInt32(&ab.gcLookaheadRunning, 0) } // AddAddr will add a new address if it's not already in the AddrBook. @@ -447,12 +625,12 @@ func (ab *dsAddrBook) ClearAddrs(p peer.ID) { key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(p))) txn, err := ab.ds.NewTransaction(false) if err != nil { - log.Errorf("failed to clear addresses for peer %s: %v\n", p.Pretty(), err) + log.Errorf("failed to clear addresses for peer %s: %v", p.Pretty(), err) } defer txn.Discard() if err := txn.Delete(key); err != nil { - log.Errorf("failed to clear addresses for peer %s: %v\n", p.Pretty(), err) + log.Errorf("failed to clear addresses for peer %s: %v", p.Pretty(), err) } if err = txn.Commit(); err != nil { diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index a3cc24f6cf..48e6e7fa90 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -37,7 +37,7 @@ func TestDsAddrBook(t *testing.T) { t.Parallel() opts := DefaultOpts() - opts.GCInterval = 1 * time.Second + opts.GCPruneInterval = 1 * time.Second opts.CacheSize = 1024 pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) @@ -47,7 +47,7 @@ func TestDsAddrBook(t *testing.T) { t.Parallel() opts := DefaultOpts() - opts.GCInterval = 1 * time.Second + opts.GCPruneInterval = 1 * time.Second opts.CacheSize = 0 pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 65ac325a4b..ba01ad14b1 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -19,21 +19,26 @@ type Options struct { CacheSize uint // Sweep interval to purge expired addresses from the datastore. - GCInterval time.Duration + GCPruneInterval time.Duration - // Number of times to retry transactional writes. - WriteRetries uint + // Interval to renew the GC lookahead window. + GCLookaheadInterval time.Duration + + // Initial delay before GC routines start. Intended to give the system time to initialise before starting GC. + GCInitialDelay time.Duration } // DefaultOpts returns the default options for a persistent peerstore: // * Cache size: 1024 -// * TTL sweep interval: 1 second +// * GC prune interval: 5 minutes +// * GC lookahead interval: 12 hours // * WriteRetries: 5 func DefaultOpts() Options { return Options{ - CacheSize: 1024, - GCInterval: 5 * time.Minute, - WriteRetries: 5, + CacheSize: 1024, + GCPruneInterval: 5 * time.Minute, + GCLookaheadInterval: 12 * time.Hour, + GCInitialDelay: 60 * time.Second, } } From a90dec50aec924f1e23cc913d2b17c46456c9f50 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 28 Nov 2018 00:13:13 +0000 Subject: [PATCH 0853/3965] add tests for new GC routines. --- p2p/host/peerstore/pstoreds/ds_gc_test.go | 141 ++++++++++++++++++++ p2p/host/peerstore/test/addr_book_suite.go | 84 +++++------- p2p/host/peerstore/test/benchmarks_suite.go | 2 +- p2p/host/peerstore/test/utils.go | 24 +++- 4 files changed, 198 insertions(+), 53 deletions(-) create mode 100644 p2p/host/peerstore/pstoreds/ds_gc_test.go diff --git a/p2p/host/peerstore/pstoreds/ds_gc_test.go b/p2p/host/peerstore/pstoreds/ds_gc_test.go new file mode 100644 index 0000000000..5b2a0f1614 --- /dev/null +++ b/p2p/host/peerstore/pstoreds/ds_gc_test.go @@ -0,0 +1,141 @@ +package pstoreds + +import ( + "testing" + "time" + + "github.com/ipfs/go-datastore/query" + "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-peerstore/test" +) + +var lookaheadQuery = query.Query{Prefix: gcLookaheadBase.String(), KeysOnly: true} + +type testProbe struct { + t *testing.T + ab peerstore.AddrBook +} + +func (tp *testProbe) countLookaheadEntries() (i int) { + results, err := tp.ab.(*dsAddrBook).ds.Query(lookaheadQuery) + if err != nil { + tp.t.Fatal(err) + } + + defer results.Close() + for range results.Next() { + i++ + } + return i +} +func (tp *testProbe) clearCache() { + for _, k := range tp.ab.(*dsAddrBook).cache.Keys() { + tp.ab.(*dsAddrBook).cache.Remove(k) + } +} + +func TestGCLookahead(t *testing.T) { + opts := DefaultOpts() + + // effectively disable automatic GC for this test. + opts.GCInitialDelay = 90 * time.Hour + opts.GCLookaheadInterval = 10 * time.Second + opts.GCPruneInterval = 1 * time.Minute + + factory := addressBookFactory(t, badgerStore, opts) + ab, closeFn := factory() + defer closeFn() + + tp := &testProbe{t, ab} + + ids := test.GeneratePeerIDs(10) + addrs := test.GenerateAddrs(100) + + // lookahead is 10 seconds, so these entries will be outside the lookahead window. + ab.AddAddrs(ids[0], addrs[:10], time.Hour) + ab.AddAddrs(ids[1], addrs[10:20], time.Hour) + ab.AddAddrs(ids[2], addrs[20:30], time.Hour) + ab.(*dsAddrBook).populateLookahead() + if i := tp.countLookaheadEntries(); i != 0 { + t.Errorf("expected no GC lookahead entries, got: %v", i) + } + + // Purge the cache, to exercise a different path in the lookahead cycle. + tp.clearCache() + + // change addresses of a peer to have TTL 1 second, placing them in the lookahead window. + ab.UpdateAddrs(ids[1], time.Hour, time.Second) + ab.(*dsAddrBook).populateLookahead() + if i := tp.countLookaheadEntries(); i != 1 { + t.Errorf("expected 1 GC lookahead entry, got: %v", i) + } + + // change addresses of another to have TTL 5 second, placing them in the lookahead window. + ab.UpdateAddrs(ids[2], time.Hour, 5*time.Second) + ab.(*dsAddrBook).populateLookahead() + if i := tp.countLookaheadEntries(); i != 2 { + t.Errorf("expected 2 GC lookahead entries, got: %v", i) + } +} + +func TestGCPurging(t *testing.T) { + opts := DefaultOpts() + + // effectively disable automatic GC for this test. + opts.GCInitialDelay = 90 * time.Hour + opts.GCLookaheadInterval = 12 * time.Second + opts.GCPruneInterval = 1 * time.Minute + + factory := addressBookFactory(t, badgerStore, opts) + ab, closeFn := factory() + defer closeFn() + + tp := &testProbe{t, ab} + + ids := test.GeneratePeerIDs(10) + addrs := test.GenerateAddrs(100) + + // stagger addresses within the lookahead window, but stagger them. + ab.AddAddrs(ids[0], addrs[:10], 1*time.Second) + ab.AddAddrs(ids[1], addrs[30:40], 1*time.Second) + ab.AddAddrs(ids[2], addrs[60:70], 1*time.Second) + + ab.AddAddrs(ids[0], addrs[10:20], 4*time.Second) + ab.AddAddrs(ids[1], addrs[40:50], 4*time.Second) + + ab.AddAddrs(ids[0], addrs[20:30], 10*time.Second) + ab.AddAddrs(ids[1], addrs[50:60], 10*time.Second) + + ab.(*dsAddrBook).populateLookahead() + if i := tp.countLookaheadEntries(); i != 3 { + t.Errorf("expected 3 GC lookahead entries, got: %v", i) + } + + <-time.After(2 * time.Second) + ab.(*dsAddrBook).purgeCycle() + + if i := tp.countLookaheadEntries(); i != 2 { + t.Errorf("expected 2 GC lookahead entries, got: %v", i) + } + + // Purge the cache, to exercise a different path in the purge cycle. + tp.clearCache() + + <-time.After(5 * time.Second) + ab.(*dsAddrBook).purgeCycle() + + if i := tp.countLookaheadEntries(); i != 2 { + t.Errorf("expected 2 GC lookahead entries, got: %v", i) + } + + <-time.After(5 * time.Second) + ab.(*dsAddrBook).purgeCycle() + + if i := tp.countLookaheadEntries(); i != 0 { + t.Errorf("expected 0 GC lookahead entries, got: %v", i) + } + + if i := len(ab.PeersWithAddrs()); i != 0 { + t.Errorf("expected 0 entries in database, got: %v", i) + } +} diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index 7cb4cf8deb..534829bd28 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -1,12 +1,9 @@ package test import ( - "fmt" "testing" "time" - peer "github.com/libp2p/go-libp2p-peer" - pt "github.com/libp2p/go-libp2p-peer/test" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) @@ -39,27 +36,11 @@ func TestAddrBook(t *testing.T, factory AddrBookFactory) { } } -func generateAddrs(count int) []ma.Multiaddr { - var addrs = make([]ma.Multiaddr, count) - for i := 0; i < count; i++ { - addrs[i] = multiaddr(fmt.Sprintf("/ip4/1.1.1.%d/tcp/1111", i)) - } - return addrs -} - -func generatePeerIds(count int) []peer.ID { - var ids = make([]peer.ID, count) - for i := 0; i < count; i++ { - ids[i], _ = pt.RandPeerID() - } - return ids -} - func testAddAddress(ab pstore.AddrBook) func(*testing.T) { return func(t *testing.T) { t.Run("add a single address", func(t *testing.T) { - id := generatePeerIds(1)[0] - addrs := generateAddrs(1) + id := GeneratePeerIDs(1)[0] + addrs := GenerateAddrs(1) ab.AddAddr(id, addrs[0], time.Hour) @@ -67,8 +48,8 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { }) t.Run("idempotent add single address", func(t *testing.T) { - id := generatePeerIds(1)[0] - addrs := generateAddrs(1) + id := GeneratePeerIDs(1)[0] + addrs := GenerateAddrs(1) ab.AddAddr(id, addrs[0], time.Hour) ab.AddAddr(id, addrs[0], time.Hour) @@ -77,16 +58,16 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { }) t.Run("add multiple addresses", func(t *testing.T) { - id := generatePeerIds(1)[0] - addrs := generateAddrs(3) + id := GeneratePeerIDs(1)[0] + addrs := GenerateAddrs(3) ab.AddAddrs(id, addrs, time.Hour) testHas(t, addrs, ab.Addrs(id)) }) t.Run("idempotent add multiple addresses", func(t *testing.T) { - id := generatePeerIds(1)[0] - addrs := generateAddrs(3) + id := GeneratePeerIDs(1)[0] + addrs := GenerateAddrs(3) ab.AddAddrs(id, addrs, time.Hour) ab.AddAddrs(id, addrs, time.Hour) @@ -95,8 +76,8 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { }) t.Run("adding an existing address with a later expiration extends its ttl", func(t *testing.T) { - id := generatePeerIds(1)[0] - addrs := generateAddrs(3) + id := GeneratePeerIDs(1)[0] + addrs := GenerateAddrs(3) ab.AddAddrs(id, addrs, time.Second) @@ -112,8 +93,8 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { func testClearWorks(ab pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - ids := generatePeerIds(2) - addrs := generateAddrs(5) + ids := GeneratePeerIDs(2) + addrs := GenerateAddrs(5) ab.AddAddrs(ids[0], addrs[0:3], time.Hour) ab.AddAddrs(ids[1], addrs[3:], time.Hour) @@ -133,29 +114,36 @@ func testClearWorks(ab pstore.AddrBook) func(t *testing.T) { func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id := generatePeerIds(1)[0] - addr := generateAddrs(1)[0] + id := GeneratePeerIDs(1)[0] + addrs := GenerateAddrs(100) + + m.SetAddrs(id, addrs, time.Hour) + testHas(t, addrs, m.Addrs(id)) + + // remove two addresses. + m.SetAddr(id, addrs[50], -1) + m.SetAddr(id, addrs[75], -1) - m.SetAddr(id, addr, time.Hour) - testHas(t, []ma.Multiaddr{addr}, m.Addrs(id)) + // calculate the survivors + survivors := append(addrs[0:50], addrs[51:]...) + survivors = append(survivors[0:74], survivors[75:]...) - m.SetAddr(id, addr, -1) - testHas(t, nil, m.Addrs(id)) + testHas(t, survivors, m.Addrs(id)) } } func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { t.Run("update ttl of peer with no addrs", func(t *testing.T) { - id := generatePeerIds(1)[0] + id := GeneratePeerIDs(1)[0] // Shouldn't panic. m.UpdateAddrs(id, time.Hour, time.Minute) }) t.Run("update ttls successfully", func(t *testing.T) { - ids := generatePeerIds(2) - addrs1, addrs2 := generateAddrs(2), generateAddrs(2) + ids := GeneratePeerIDs(2) + addrs1, addrs2 := GenerateAddrs(2), GenerateAddrs(2) // set two keys with different ttls for each peer. m.SetAddr(ids[0], addrs1[0], time.Hour) @@ -200,7 +188,7 @@ func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { func testNilAddrsDontBreak(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - id := generatePeerIds(1)[0] + id := GeneratePeerIDs(1)[0] m.SetAddr(id, nil, time.Hour) m.AddAddr(id, nil, time.Hour) @@ -209,9 +197,9 @@ func testNilAddrsDontBreak(m pstore.AddrBook) func(t *testing.T) { func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - ids := generatePeerIds(2) - addrs1 := generateAddrs(3) - addrs2 := generateAddrs(2) + ids := GeneratePeerIDs(2) + addrs1 := GenerateAddrs(3) + addrs2 := GenerateAddrs(2) m.AddAddrs(ids[0], addrs1, time.Hour) m.AddAddrs(ids[1], addrs2, time.Hour) @@ -254,8 +242,8 @@ func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { func testClearWithIterator(m pstore.AddrBook) func(t *testing.T) { return func(t *testing.T) { - ids := generatePeerIds(2) - addrs := generateAddrs(100) + ids := GeneratePeerIDs(2) + addrs := GenerateAddrs(100) // Add the peers with 50 addresses each. m.AddAddrs(ids[0], addrs[:50], pstore.PermanentAddrTTL) @@ -292,8 +280,8 @@ func testPeersWithAddrs(m pstore.AddrBook) func(t *testing.T) { }) t.Run("non-empty addrbook", func(t *testing.T) { - ids := generatePeerIds(2) - addrs := generateAddrs(10) + ids := GeneratePeerIDs(2) + addrs := GenerateAddrs(10) m.AddAddrs(ids[0], addrs[:5], pstore.PermanentAddrTTL) m.AddAddrs(ids[1], addrs[5:], pstore.PermanentAddrTTL) diff --git a/p2p/host/peerstore/test/benchmarks_suite.go b/p2p/host/peerstore/test/benchmarks_suite.go index f283b9b0a9..cf04eb94a9 100644 --- a/p2p/host/peerstore/test/benchmarks_suite.go +++ b/p2p/host/peerstore/test/benchmarks_suite.go @@ -36,7 +36,7 @@ func BenchmarkPeerstore(b *testing.B, factory PeerstoreFactory, variant string) // Start all test peer producing goroutines, where each produces peers with as many // multiaddrs as the n field in the param struct. for _, p := range params { - go addressProducer(ctx, b, p.ch, p.n) + go AddressProducer(ctx, b, p.ch, p.n) } // So tests are always run in the same order. diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index 6649820785..15b73c8d54 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -10,7 +10,7 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -func multiaddr(m string) ma.Multiaddr { +func Multiaddr(m string) ma.Multiaddr { maddr, err := ma.NewMultiaddr(m) if err != nil { panic(err) @@ -23,7 +23,7 @@ type peerpair struct { Addr []ma.Multiaddr } -func randomPeer(b *testing.B, addrCount int) *peerpair { +func RandomPeer(b *testing.B, addrCount int) *peerpair { var ( pid peer.ID err error @@ -44,11 +44,11 @@ func randomPeer(b *testing.B, addrCount int) *peerpair { return &peerpair{pid, addrs} } -func addressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair, addrsPerPeer int) { +func AddressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair, addrsPerPeer int) { b.Helper() defer close(addrs) for { - p := randomPeer(b, addrsPerPeer) + p := RandomPeer(b, addrsPerPeer) select { case addrs <- p: case <-ctx.Done(): @@ -56,3 +56,19 @@ func addressProducer(ctx context.Context, b *testing.B, addrs chan *peerpair, ad } } } + +func GenerateAddrs(count int) []ma.Multiaddr { + var addrs = make([]ma.Multiaddr, count) + for i := 0; i < count; i++ { + addrs[i] = Multiaddr(fmt.Sprintf("/ip4/1.1.1.%d/tcp/1111", i)) + } + return addrs +} + +func GeneratePeerIDs(count int) []peer.ID { + var ids = make([]peer.ID, count) + for i := 0; i < count; i++ { + ids[i], _ = pt.RandPeerID() + } + return ids +} From 22119931a486f7778c281a14b59e03d0aff1766e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 28 Nov 2018 00:56:51 +0000 Subject: [PATCH 0854/3965] minor cleanup. --- p2p/host/peerstore/pstoreds/addr_book.go | 2 +- p2p/host/peerstore/pstoreds/ds_gc_test.go | 4 ---- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 2a1caf62d5..c57e3ed134 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -213,7 +213,7 @@ func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { } // loadRecord is a read-through fetch. It fetches a record from cache, falling back to the -// datastore upon a miss, and returning an newly initialized record if the peer doesn't exist. +// datastore upon a miss, and returning a newly initialized record if the peer doesn't exist. // // loadRecord calls Refresh() on the record before returning it. If the record changes // as a result and `update=true`, an async flush is scheduled. diff --git a/p2p/host/peerstore/pstoreds/ds_gc_test.go b/p2p/host/peerstore/pstoreds/ds_gc_test.go index 5b2a0f1614..c6f9a6e90c 100644 --- a/p2p/host/peerstore/pstoreds/ds_gc_test.go +++ b/p2p/host/peerstore/pstoreds/ds_gc_test.go @@ -113,7 +113,6 @@ func TestGCPurging(t *testing.T) { <-time.After(2 * time.Second) ab.(*dsAddrBook).purgeCycle() - if i := tp.countLookaheadEntries(); i != 2 { t.Errorf("expected 2 GC lookahead entries, got: %v", i) } @@ -123,18 +122,15 @@ func TestGCPurging(t *testing.T) { <-time.After(5 * time.Second) ab.(*dsAddrBook).purgeCycle() - if i := tp.countLookaheadEntries(); i != 2 { t.Errorf("expected 2 GC lookahead entries, got: %v", i) } <-time.After(5 * time.Second) ab.(*dsAddrBook).purgeCycle() - if i := tp.countLookaheadEntries(); i != 0 { t.Errorf("expected 0 GC lookahead entries, got: %v", i) } - if i := len(ab.PeersWithAddrs()); i != 0 { t.Errorf("expected 0 entries in database, got: %v", i) } From 4fc69bd7e54177f412ad3add7beb10e5f01bfe61 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 28 Nov 2018 01:08:12 +0000 Subject: [PATCH 0855/3965] improve test coverage of new GC routine. --- p2p/host/peerstore/pstoreds/ds_gc_test.go | 33 ++++++++++++++--------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/ds_gc_test.go b/p2p/host/peerstore/pstoreds/ds_gc_test.go index c6f9a6e90c..f3e5ea6765 100644 --- a/p2p/host/peerstore/pstoreds/ds_gc_test.go +++ b/p2p/host/peerstore/pstoreds/ds_gc_test.go @@ -60,11 +60,12 @@ func TestGCLookahead(t *testing.T) { t.Errorf("expected no GC lookahead entries, got: %v", i) } + // change addresses of a peer to have TTL 1 second, placing them in the lookahead window. + ab.UpdateAddrs(ids[1], time.Hour, time.Second) + // Purge the cache, to exercise a different path in the lookahead cycle. tp.clearCache() - // change addresses of a peer to have TTL 1 second, placing them in the lookahead window. - ab.UpdateAddrs(ids[1], time.Hour, time.Second) ab.(*dsAddrBook).populateLookahead() if i := tp.countLookaheadEntries(); i != 1 { t.Errorf("expected 1 GC lookahead entry, got: %v", i) @@ -83,7 +84,7 @@ func TestGCPurging(t *testing.T) { // effectively disable automatic GC for this test. opts.GCInitialDelay = 90 * time.Hour - opts.GCLookaheadInterval = 12 * time.Second + opts.GCLookaheadInterval = 20 * time.Second opts.GCPruneInterval = 1 * time.Minute factory := addressBookFactory(t, badgerStore, opts) @@ -106,15 +107,18 @@ func TestGCPurging(t *testing.T) { ab.AddAddrs(ids[0], addrs[20:30], 10*time.Second) ab.AddAddrs(ids[1], addrs[50:60], 10*time.Second) + // this is inside the window, but it will survive the purges we do in the test. + ab.AddAddrs(ids[3], addrs[70:80], 15*time.Second) + ab.(*dsAddrBook).populateLookahead() - if i := tp.countLookaheadEntries(); i != 3 { - t.Errorf("expected 3 GC lookahead entries, got: %v", i) + if i := tp.countLookaheadEntries(); i != 4 { + t.Errorf("expected 4 GC lookahead entries, got: %v", i) } <-time.After(2 * time.Second) ab.(*dsAddrBook).purgeCycle() - if i := tp.countLookaheadEntries(); i != 2 { - t.Errorf("expected 2 GC lookahead entries, got: %v", i) + if i := tp.countLookaheadEntries(); i != 3 { + t.Errorf("expected 3 GC lookahead entries, got: %v", i) } // Purge the cache, to exercise a different path in the purge cycle. @@ -122,16 +126,19 @@ func TestGCPurging(t *testing.T) { <-time.After(5 * time.Second) ab.(*dsAddrBook).purgeCycle() - if i := tp.countLookaheadEntries(); i != 2 { - t.Errorf("expected 2 GC lookahead entries, got: %v", i) + if i := tp.countLookaheadEntries(); i != 3 { + t.Errorf("expected 3 GC lookahead entries, got: %v", i) } <-time.After(5 * time.Second) ab.(*dsAddrBook).purgeCycle() - if i := tp.countLookaheadEntries(); i != 0 { - t.Errorf("expected 0 GC lookahead entries, got: %v", i) + if i := tp.countLookaheadEntries(); i != 1 { + t.Errorf("expected 1 GC lookahead entries, got: %v", i) + } + if i := len(ab.PeersWithAddrs()); i != 1 { + t.Errorf("expected 1 entries in database, got: %v", i) } - if i := len(ab.PeersWithAddrs()); i != 0 { - t.Errorf("expected 0 entries in database, got: %v", i) + if p := ab.PeersWithAddrs()[0]; p != ids[3] { + t.Errorf("expected remaining peer to be #3, got: %v, expected: %v", p, ids[3]) } } From 7ffff3dd6767dd5f0c7c3a7f1c778a38d687d710 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 28 Nov 2018 01:29:09 +0000 Subject: [PATCH 0856/3965] rename GCPruneInterval to GCPurgeInterval for consistency. --- p2p/host/peerstore/pstoreds/addr_book.go | 2 +- p2p/host/peerstore/pstoreds/ds_gc_test.go | 4 ++-- p2p/host/peerstore/pstoreds/ds_test.go | 4 ++-- p2p/host/peerstore/pstoreds/peerstore.go | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index c57e3ed134..4db3d64cf9 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -272,7 +272,7 @@ func (ab *dsAddrBook) background() { go func() { select { case <-time.After(ab.opts.GCInitialDelay): - pruneTimer = time.NewTicker(ab.opts.GCPruneInterval) + pruneTimer = time.NewTicker(ab.opts.GCPurgeInterval) lookaheadTimer = time.NewTicker(ab.opts.GCLookaheadInterval) } }() diff --git a/p2p/host/peerstore/pstoreds/ds_gc_test.go b/p2p/host/peerstore/pstoreds/ds_gc_test.go index f3e5ea6765..6de9539615 100644 --- a/p2p/host/peerstore/pstoreds/ds_gc_test.go +++ b/p2p/host/peerstore/pstoreds/ds_gc_test.go @@ -40,7 +40,7 @@ func TestGCLookahead(t *testing.T) { // effectively disable automatic GC for this test. opts.GCInitialDelay = 90 * time.Hour opts.GCLookaheadInterval = 10 * time.Second - opts.GCPruneInterval = 1 * time.Minute + opts.GCPurgeInterval = 1 * time.Minute factory := addressBookFactory(t, badgerStore, opts) ab, closeFn := factory() @@ -85,7 +85,7 @@ func TestGCPurging(t *testing.T) { // effectively disable automatic GC for this test. opts.GCInitialDelay = 90 * time.Hour opts.GCLookaheadInterval = 20 * time.Second - opts.GCPruneInterval = 1 * time.Minute + opts.GCPurgeInterval = 1 * time.Minute factory := addressBookFactory(t, badgerStore, opts) ab, closeFn := factory() diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 48e6e7fa90..abb712b49e 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -37,7 +37,7 @@ func TestDsAddrBook(t *testing.T) { t.Parallel() opts := DefaultOpts() - opts.GCPruneInterval = 1 * time.Second + opts.GCPurgeInterval = 1 * time.Second opts.CacheSize = 1024 pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) @@ -47,7 +47,7 @@ func TestDsAddrBook(t *testing.T) { t.Parallel() opts := DefaultOpts() - opts.GCPruneInterval = 1 * time.Second + opts.GCPurgeInterval = 1 * time.Second opts.CacheSize = 0 pt.TestAddrBook(t, addressBookFactory(t, dsFactory, opts)) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index ba01ad14b1..a189f66d8f 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -19,7 +19,7 @@ type Options struct { CacheSize uint // Sweep interval to purge expired addresses from the datastore. - GCPruneInterval time.Duration + GCPurgeInterval time.Duration // Interval to renew the GC lookahead window. GCLookaheadInterval time.Duration @@ -36,7 +36,7 @@ type Options struct { func DefaultOpts() Options { return Options{ CacheSize: 1024, - GCPruneInterval: 5 * time.Minute, + GCPurgeInterval: 5 * time.Minute, GCLookaheadInterval: 12 * time.Hour, GCInitialDelay: 60 * time.Second, } From d76db1d65d9b070020a2a5f541c8e4914aa221b2 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 28 Nov 2018 08:41:11 +0700 Subject: [PATCH 0857/3965] fix contributing link in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index bfced51bde..1e7bda608f 100644 --- a/README.md +++ b/README.md @@ -193,7 +193,7 @@ List of packages currently in existence for libp2p: # Contribute -go-libp2p is part of [The IPFS Project](https://github.com/ipfs/ipfs), and is MIT licensed open source software. We welcome contributions big and small! Take a look at the [community contributing notes](https://github.com/ipfs/community/blob/master/contributing.md). Please make sure to check the [issues](https://github.com/ipfs/go-libp2p/issues). Search the closed ones before reporting things, and help us with the open ones. +go-libp2p is part of [The IPFS Project](https://github.com/ipfs/ipfs), and is MIT licensed open source software. We welcome contributions big and small! Take a look at the [community contributing notes](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md). Please make sure to check the [issues](https://github.com/ipfs/go-libp2p/issues). Search the closed ones before reporting things, and help us with the open ones. Guidelines: From 63843cc17e2d4d4dcb105cf8f846c657125fdae5 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 28 Nov 2018 14:06:54 +0700 Subject: [PATCH 0858/3965] close the underlying connection when the context is canceled --- p2p/security/tls/transport.go | 32 +++++++++++++++++++++--- p2p/security/tls/transport_test.go | 40 ++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 4 deletions(-) diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 16941bc059..40181f9a8a 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -43,8 +43,20 @@ var _ cs.Transport = &Transport{} // SecureInbound runs the TLS handshake as a server. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Conn, error) { serv := tls.Server(insecure, t.identity.Config) - // TODO: use the ctx - // see https://github.com/golang/go/issues/18482 + + // There's no way to pass a context to tls.Conn.Handshake(). + // See https://github.com/golang/go/issues/18482. + // Close the connection instead. + done := make(chan struct{}) + defer close(done) + go func() { + select { + case <-done: + case <-ctx.Done(): + insecure.Close() + } + }() + if err := serv.Handshake(); err != nil { return nil, err } @@ -54,8 +66,20 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Co // SecureOutbound runs the TLS handshake as a client. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { cl := tls.Client(insecure, t.identity.ConfigForPeer(p)) - // TODO: use the ctx - // see https://github.com/golang/go/issues/18482 + + // There's no way to pass a context to tls.Conn.Handshake(). + // See https://github.com/golang/go/issues/18482. + // Close the connection instead. + done := make(chan struct{}) + defer close(done) + go func() { + select { + case <-done: + case <-ctx.Done(): + insecure.Close() + } + }() + if err := cl.Handshake(); err != nil { return nil, err } diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 5e50868c95..94b61ea3a6 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -96,6 +96,46 @@ var _ = Describe("Transport", func() { Expect(string(b)).To(Equal("foobar")) }) + It("fails when the context of the outgoing connection is canceled", func() { + clientTransport, err := New(clientKey) + Expect(err).ToNot(HaveOccurred()) + serverTransport, err := New(serverKey) + Expect(err).ToNot(HaveOccurred()) + + clientInsecureConn, serverInsecureConn := connect() + + go func() { + defer GinkgoRecover() + _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) + Expect(err).To(HaveOccurred()) + }() + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err = clientTransport.SecureOutbound(ctx, clientInsecureConn, serverID) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("use of closed network connection")) + }) + + It("fails when the context of the incoming connection is canceled", func() { + clientTransport, err := New(clientKey) + Expect(err).ToNot(HaveOccurred()) + serverTransport, err := New(serverKey) + Expect(err).ToNot(HaveOccurred()) + + clientInsecureConn, serverInsecureConn := connect() + + go func() { + defer GinkgoRecover() + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err := serverTransport.SecureInbound(ctx, serverInsecureConn) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("use of closed network connection")) + }() + _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) + Expect(err).To(HaveOccurred()) + }) + It("fails if the peer ID doesn't match", func() { thirdPartyID, _ := createPeer() From 1c0f10c904dd928dc069cccd434e8dbaf8011a61 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 29 Nov 2018 09:36:22 +0700 Subject: [PATCH 0859/3965] return the context cancelation error --- p2p/security/tls/transport.go | 44 ++++++++++++++++-------------- p2p/security/tls/transport_test.go | 6 ++-- 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 40181f9a8a..6bd5ba4400 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -43,30 +43,23 @@ var _ cs.Transport = &Transport{} // SecureInbound runs the TLS handshake as a server. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Conn, error) { serv := tls.Server(insecure, t.identity.Config) - - // There's no way to pass a context to tls.Conn.Handshake(). - // See https://github.com/golang/go/issues/18482. - // Close the connection instead. - done := make(chan struct{}) - defer close(done) - go func() { - select { - case <-done: - case <-ctx.Done(): - insecure.Close() - } - }() - - if err := serv.Handshake(); err != nil { - return nil, err - } - return t.setupConn(serv) + return t.handshake(ctx, insecure, serv) } // SecureOutbound runs the TLS handshake as a client. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { cl := tls.Client(insecure, t.identity.ConfigForPeer(p)) + return t.handshake(ctx, insecure, cl) +} +func (t *Transport) handshake( + ctx context.Context, + // in Go 1.10, we need to close the underlying net.Conn + // in Go 1.11 this was fixed, and tls.Conn.Close() works as well + insecure net.Conn, + tlsConn *tls.Conn, +) (cs.Conn, error) { + errChan := make(chan error, 2) // There's no way to pass a context to tls.Conn.Handshake(). // See https://github.com/golang/go/issues/18482. // Close the connection instead. @@ -76,14 +69,23 @@ func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p pee select { case <-done: case <-ctx.Done(): + errChan <- ctx.Err() insecure.Close() } }() - if err := cl.Handshake(); err != nil { - return nil, err + if err := tlsConn.Handshake(); err != nil { + // if the context was canceled, return the context error + errChan <- err + return nil, <-errChan + } + conn, err := t.setupConn(tlsConn) + if err != nil { + // if the context was canceled, return the context error + errChan <- err + return nil, <-errChan } - return t.setupConn(cl) + return conn, nil } func (t *Transport) setupConn(tlsConn *tls.Conn) (cs.Conn, error) { diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 94b61ea3a6..bc4d5b3813 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -112,8 +112,7 @@ var _ = Describe("Transport", func() { ctx, cancel := context.WithCancel(context.Background()) cancel() _, err = clientTransport.SecureOutbound(ctx, clientInsecureConn, serverID) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("use of closed network connection")) + Expect(err).To(MatchError(context.Canceled)) }) It("fails when the context of the incoming connection is canceled", func() { @@ -129,8 +128,7 @@ var _ = Describe("Transport", func() { ctx, cancel := context.WithCancel(context.Background()) cancel() _, err := serverTransport.SecureInbound(ctx, serverInsecureConn) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("use of closed network connection")) + Expect(err).To(MatchError(context.Canceled)) }() _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) Expect(err).To(HaveOccurred()) From 31eb77d4469848cfc170673776a22567ee2435a7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 29 Nov 2018 12:24:40 +0200 Subject: [PATCH 0860/3965] add delay in initial relay advertisement to allow the dht time to bootstrap --- p2p/host/relay/autorelay_test.go | 3 ++- p2p/host/relay/relay.go | 10 +++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index ce50ea8c09..4d57d78048 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -26,9 +26,10 @@ import ( // test specific parameters func init() { - autonat.AutoNATIdentifyDelay = 100 * time.Millisecond + autonat.AutoNATIdentifyDelay = 500 * time.Millisecond autonat.AutoNATBootDelay = 1 * time.Second relay.BootDelay = 1 * time.Second + relay.AdvertiseBootDelay = 1 * time.Millisecond manet.Private4 = []*net.IPNet{} } diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index 69b6737078..3f282a13c1 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -2,6 +2,7 @@ package relay import ( "context" + "time" basic "github.com/libp2p/go-libp2p/p2p/host/basic" @@ -10,6 +11,10 @@ import ( ma "github.com/multiformats/go-multiaddr" ) +var ( + AdvertiseBootDelay = 5 * time.Second +) + // RelayHost is a Host that provides Relay services. type RelayHost struct { *basic.BasicHost @@ -25,7 +30,10 @@ func NewRelayHost(ctx context.Context, bhost *basic.BasicHost, advertise discove advertise: advertise, } bhost.AddrsFactory = h.hostAddrs - discovery.Advertise(ctx, advertise, RelayRendezvous) + go func() { + time.Sleep(AdvertiseBootDelay) + discovery.Advertise(ctx, advertise, RelayRendezvous) + }() return h } From eaf15fd98bfcbc788b0508c6ec8dcac8e1827c7e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 29 Nov 2018 20:38:57 +0700 Subject: [PATCH 0861/3965] simplify returning of context cancellation errors --- p2p/security/tls/transport.go | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 6bd5ba4400..5a75296f1a 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -59,7 +59,6 @@ func (t *Transport) handshake( insecure net.Conn, tlsConn *tls.Conn, ) (cs.Conn, error) { - errChan := make(chan error, 2) // There's no way to pass a context to tls.Conn.Handshake(). // See https://github.com/golang/go/issues/18482. // Close the connection instead. @@ -69,21 +68,24 @@ func (t *Transport) handshake( select { case <-done: case <-ctx.Done(): - errChan <- ctx.Err() insecure.Close() } }() if err := tlsConn.Handshake(); err != nil { // if the context was canceled, return the context error - errChan <- err - return nil, <-errChan + if ctxErr := ctx.Err(); ctxErr != nil { + return nil, ctxErr + } + return nil, err } conn, err := t.setupConn(tlsConn) if err != nil { // if the context was canceled, return the context error - errChan <- err - return nil, <-errChan + if ctxErr := ctx.Err(); ctxErr != nil { + return nil, ctxErr + } + return nil, err } return conn, nil } From 88d98287ff08455fa4e473684a4c842dd0428d1a Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Fri, 30 Nov 2018 16:54:25 +0530 Subject: [PATCH 0862/3965] suppressing error msg for NoSecurity option --- p2p/protocol/identify/id.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 5991fe6950..a2d52a48a9 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -186,9 +186,11 @@ func (ids *IDService) populateMessage(mes *pb.Identify, c inet.Conn) { // set our public key ownKey := ids.Host.Peerstore().PubKey(ids.Host.ID()) - if ownKey == nil { + prKey := ids.Host.Peerstore().PrivKey(ids.Host.ID()) + + if ownKey == nil && prKey != nil { log.Errorf("did not have own public key in Peerstore") - } else { + } else if ownKey != nil { if kb, err := ownKey.Bytes(); err != nil { log.Errorf("failed to convert key to bytes") } else { From 05b8d7ff49b077da08ba10cb67c9bec3383586a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 30 Nov 2018 16:09:25 +0000 Subject: [PATCH 0863/3965] document custom gogo types; reorg code. --- p2p/host/peerstore/pb/custom.go | 41 +++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/p2p/host/peerstore/pb/custom.go b/p2p/host/peerstore/pb/custom.go index e51b2ca7d3..711963c941 100644 --- a/p2p/host/peerstore/pb/custom.go +++ b/p2p/host/peerstore/pb/custom.go @@ -3,19 +3,28 @@ package pstore_pb import ( "encoding/json" + proto "github.com/gogo/protobuf/proto" peer "github.com/libp2p/go-libp2p-peer" pt "github.com/libp2p/go-libp2p-peer/test" ma "github.com/multiformats/go-multiaddr" ) +// customGogoType aggregates the interfaces that custom Gogo types need to implement. +// it is only used for type assertions. +type customGogoType interface { + proto.Marshaler + proto.Unmarshaler + json.Marshaler + json.Unmarshaler + proto.Sizer +} + +// ProtoAddr is a custom type used by gogo to serde raw peer IDs into the peer.ID type, and back. type ProtoPeerID struct { peer.ID } -func NewPopulatedProtoPeerID(r randyPstore) *ProtoPeerID { - id, _ := pt.RandPeerID() - return &ProtoPeerID{ID: id} -} +var _ customGogoType = (*ProtoPeerID)(nil) func (id ProtoPeerID) Marshal() ([]byte, error) { return []byte(id.ID), nil @@ -36,26 +45,24 @@ func (id *ProtoPeerID) Unmarshal(data []byte) (err error) { } func (id *ProtoPeerID) UnmarshalJSON(data []byte) error { - v := new([]byte) + var v []byte err := json.Unmarshal(data, v) if err != nil { return err } - return id.Unmarshal(*v) + return id.Unmarshal(v) } func (id ProtoPeerID) Size() int { return len([]byte(id.ID)) } +// ProtoAddr is a custom type used by gogo to serde raw multiaddresses into the ma.Multiaddr type, and back. type ProtoAddr struct { ma.Multiaddr } -func NewPopulatedProtoAddr(r randyPstore) *ProtoAddr { - a, _ := ma.NewMultiaddr("/ip4/123.123.123.123/tcp/7001") - return &ProtoAddr{Multiaddr: a} -} +var _ customGogoType = (*ProtoAddr)(nil) func (a ProtoAddr) Marshal() ([]byte, error) { return a.Bytes(), nil @@ -87,3 +94,17 @@ func (a *ProtoAddr) UnmarshalJSON(data []byte) error { func (a ProtoAddr) Size() int { return len(a.Bytes()) } + +// NewPopulatedProtoAddr generates a populated instance of the custom gogo type ProtoAddr. +// It is required by gogo-generated tests. +func NewPopulatedProtoAddr(r randyPstore) *ProtoAddr { + a, _ := ma.NewMultiaddr("/ip4/123.123.123.123/tcp/7001") + return &ProtoAddr{Multiaddr: a} +} + +// NewPopulatedProtoPeerID generates a populated instance of the custom gogo type ProtoPeerID. +// It is required by gogo-generated tests. +func NewPopulatedProtoPeerID(r randyPstore) *ProtoPeerID { + id, _ := pt.RandPeerID() + return &ProtoPeerID{ID: id} +} From 97374b3b853a1e5c1b6d8d68f8be363908e4d2b2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 30 Nov 2018 16:12:38 +0000 Subject: [PATCH 0864/3965] Revert "use a Pool for address record objects." --- p2p/host/peerstore/pstoreds/addr_book.go | 35 ++++-------------------- 1 file changed, 6 insertions(+), 29 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 4db3d64cf9..284cc4a511 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -18,8 +18,8 @@ import ( pb "github.com/libp2p/go-libp2p-peerstore/pb" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" + lru "github.com/hashicorp/golang-lru" ma "github.com/multiformats/go-multiaddr" - lru "github.com/raulk/golang-lru" b32 "github.com/whyrusleeping/base32" ) @@ -38,14 +38,6 @@ var ( // GC lookahead entries are stored in keys with pattern: // /peers/gc/addrs// => nil gcLookaheadBase = ds.NewKey("/peers/gc/addrs") - arPool = &sync.Pool{ - New: func() interface{} { - return &addrsRecord{ - AddrBookRecord: &pb.AddrBookRecord{}, - dirty: false, - } - }, - } ) // addrsRecord decorates the AddrBookRecord with locks and metadata. @@ -55,11 +47,6 @@ type addrsRecord struct { dirty bool } -func (r *addrsRecord) Reset() { - r.AddrBookRecord.Reset() - r.dirty = false -} - // FlushInTxn writes the record to the datastore by calling ds.Put, unless the record is // marked for deletion, in which case the deletion is executed via ds.Delete. func (r *addrsRecord) FlushInTxn(txn ds.Txn) (err error) { @@ -172,11 +159,7 @@ var _ pstore.AddrBook = (*dsAddrBook)(nil) func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab *dsAddrBook, err error) { var cache cache = new(noopCache) if opts.CacheSize > 0 { - evictCallback := func(key interface{}, value interface{}) { - value.(*addrsRecord).Reset() - arPool.Put(value) - } - if cache, err = lru.NewARCWithEvict(int(opts.CacheSize), evictCallback); err != nil { + if cache, err = lru.NewARC(int(opts.CacheSize)); err != nil { return nil, err } } @@ -242,7 +225,7 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs } if err == nil { - pr = arPool.Get().(*addrsRecord) + pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} if err = pr.Unmarshal(data); err != nil { return nil, err } @@ -250,8 +233,7 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs ab.asyncFlush(pr) } } else { - pr = arPool.Get().(*addrsRecord) - pr.Id = &pb.ProtoPeerID{ID: id} + pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{Id: &pb.ProtoPeerID{ID: id}}} } if cache { @@ -321,10 +303,7 @@ func (ab *dsAddrBook) purgeCycle() { } var id peer.ID - record := arPool.Get().(*addrsRecord) - record.Reset() - defer arPool.Put(record) - + record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} txn, err := ab.ds.NewTransaction(false) if err != nil { log.Warningf("failed while purging entries: %v", err) @@ -457,9 +436,7 @@ func (ab *dsAddrBook) populateLookahead() { until := time.Now().Add(ab.opts.GCLookaheadInterval).Unix() var id peer.ID - record := arPool.Get().(*addrsRecord) - defer arPool.Put(record) - + record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} txn, err := ab.ds.NewTransaction(false) if err != nil { log.Warningf("failed while filling lookahead GC region: %v", err) From ecfe476ed13359dd7bd9b93bf75ab06e6a6e5fc7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 30 Nov 2018 16:56:37 +0000 Subject: [PATCH 0865/3965] rename addrsRecord.Refresh() => addrsRecord.Clean(). --- p2p/host/peerstore/pstoreds/addr_book.go | 34 ++++++++++++------------ 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 284cc4a511..018726ff41 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -81,16 +81,16 @@ func (r *addrsRecord) Flush(ds ds.TxnDatastore) (err error) { return txn.Commit() } -// Refresh is called on records to perform housekeeping. The return value signals if the record was changed -// as a result of the refresh. +// Clean is called on records to perform housekeeping. The return value signals if the record was changed +// as a result of the cleaning. // -// Refresh does the following: +// Clean does the following: // * sorts the addresses by expiration (soonest expiring first). // * removes the addresses that have expired. // // It short-circuits optimistically when we know there's nothing to do. // -// Refresh is called from several points: +// Clean is called from several points: // * when accessing and loading an entry. // * when performing periodic GC. // * after an entry has been modified (e.g. addresses have been added or removed, @@ -98,7 +98,7 @@ func (r *addrsRecord) Flush(ds ds.TxnDatastore) (err error) { // // If the return value is true, the caller can perform a flush immediately, or can schedule an async // flush, depending on the context. -func (r *addrsRecord) Refresh() (chgd bool) { +func (r *addrsRecord) Clean() (chgd bool) { now := time.Now().Unix() if !r.dirty && len(r.Addrs) > 0 && r.Addrs[0].Expiry > now { // record is not dirty, and we have no expired entries to purge. @@ -198,14 +198,14 @@ func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { // loadRecord is a read-through fetch. It fetches a record from cache, falling back to the // datastore upon a miss, and returning a newly initialized record if the peer doesn't exist. // -// loadRecord calls Refresh() on the record before returning it. If the record changes +// loadRecord calls Clean() on the record before returning it. If the record changes // as a result and `update=true`, an async flush is scheduled. // // If `cache=true`, the record is inserted in the cache when loaded from the datastore. func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrsRecord, err error) { if e, ok := ab.cache.Get(id); ok { pr = e.(*addrsRecord) - if pr.Refresh() && update { + if pr.Clean() && update { ab.asyncFlush(pr) } return pr, nil @@ -229,7 +229,7 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs if err = pr.Unmarshal(data); err != nil { return nil, err } - if pr.Refresh() && update { + if pr.Clean() && update { ab.asyncFlush(pr) } } else { @@ -293,7 +293,7 @@ var purgeQuery = query.Query{Prefix: gcLookaheadBase.String(), KeysOnly: true} // purgeCycle runs a single GC cycle, operating within the lookahead window. // -// It scans the lookahead region for entries that need to be visited, and performs a refresh on them. An errors trigger +// It scans the lookahead region for entries that need to be visited, and performs a Clean() on them. An errors trigger // the removal of the GC entry, in order to prevent unactionable items from accumulating. If the error happened to be // temporary, the entry will be revisited in the next lookahead window. func (ab *dsAddrBook) purgeCycle() { @@ -323,7 +323,7 @@ func (ab *dsAddrBook) purgeCycle() { } } - // This function drops a GC key if the entry is refreshed correctly. It may reschedule another visit + // This function drops a GC key if the entry is cleaned correctly. It may reschedule another visit // if the next earliest expiry falls within the current window again. dropOrReschedule := func(key ds.Key, ar *addrsRecord) { if err = txn.Delete(key); err != nil { @@ -377,11 +377,11 @@ func (ab *dsAddrBook) purgeCycle() { continue } - // if the record is in cache, we refresh it and flush it if necessary. + // if the record is in cache, we clean it and flush it if necessary. if e, ok := ab.cache.Peek(id); ok { cached := e.(*addrsRecord) cached.Lock() - if cached.Refresh() { + if cached.Clean() { if err = cached.FlushInTxn(txn); err != nil { log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) } @@ -393,7 +393,7 @@ func (ab *dsAddrBook) purgeCycle() { record.Reset() - // otherwise, fetch it from the store, refresh it and flush it. + // otherwise, fetch it from the store, clean it and flush it. entryKey := addrBookBase.ChildString(gcKey.Name()) val, err := txn.Get(entryKey) if err != nil { @@ -406,7 +406,7 @@ func (ab *dsAddrBook) purgeCycle() { dropInError(gcKey, err, "unmarshalling entry") continue } - if record.Refresh() { + if record.Clean() { err = record.FlushInTxn(txn) if err != nil { log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) @@ -554,7 +554,7 @@ func (ab *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.D pr.dirty = true } - if pr.Refresh() { + if pr.Clean() { pr.Flush(ab.ds) } } @@ -665,7 +665,7 @@ Outer: pr.Addrs = append(pr.Addrs, added...) pr.dirty = true - pr.Refresh() + pr.Clean() return pr.Flush(ab.ds) } @@ -698,7 +698,7 @@ func (ab *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) (err error) { pr.Addrs = pr.Addrs[:survived] pr.dirty = true - pr.Refresh() + pr.Clean() return pr.Flush(ab.ds) } From c822dae627913ec8fc412db3fbd5894743158806 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 30 Nov 2018 17:32:51 +0000 Subject: [PATCH 0866/3965] fix raciness in init logic; dedicated goroutines for GC and flushing. --- p2p/host/peerstore/pstoreds/addr_book.go | 78 +++++++++++++---------- p2p/host/peerstore/pstoreds/ds_gc_test.go | 30 +++++++++ 2 files changed, 75 insertions(+), 33 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 018726ff41..f19910a13b 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -146,7 +146,7 @@ type dsAddrBook struct { flushJobCh chan *addrsRecord cancelFn func() - closedCh chan struct{} + closeDone sync.WaitGroup gcCurrWindowEnd int64 gcLookaheadRunning int32 @@ -173,7 +173,6 @@ func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab * ds: store, subsManager: pstoremem.NewAddrSubManager(), flushJobCh: make(chan *addrsRecord, 32), - closedCh: make(chan struct{}), } // kick off periodic GC. @@ -184,7 +183,7 @@ func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab * func (ab *dsAddrBook) Close() { ab.cancelFn() - <-ab.closedCh + ab.closeDone.Wait() } func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { @@ -247,46 +246,59 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs // * GCing expired addresses from the datastore at regular intervals. // * persisting asynchronous flushes to the datastore. func (ab *dsAddrBook) background() { - // placeholder tickers. - pruneTimer, lookaheadTimer := new(time.Ticker), new(time.Ticker) + // goroutine that takes care of executing flush jobs. + go func() { + ab.closeDone.Add(1) + for { + select { + case fj := <-ab.flushJobCh: + if cached, ok := ab.cache.Peek(fj.Id.ID); ok { + // Only continue flushing if the record we have in memory is the same as for which the flush + // job was requested. If it's not in memory, it has been evicted and we don't know if we hold + // the latest state or not. Similarly, if it's cached but the pointer is different, it means + // it was evicted and has been reloaded, so we're also uncertain if we hold the latest state. + if pr := cached.(*addrsRecord); pr == fj { + pr.RLock() + pr.Flush(ab.ds) + pr.RUnlock() + } + } + case <-ab.ctx.Done(): + ab.closeDone.Done() + return + } + } + }() - // populate the tickers after the initial delay has passed. + // goroutine that takes care of GC. go func() { select { case <-time.After(ab.opts.GCInitialDelay): - pruneTimer = time.NewTicker(ab.opts.GCPurgeInterval) - lookaheadTimer = time.NewTicker(ab.opts.GCLookaheadInterval) + case <-ab.ctx.Done(): + // yield if we have been cancelled/closed before the delay elapses. + return } - }() - for { - select { - case fj := <-ab.flushJobCh: - if cached, ok := ab.cache.Peek(fj.Id.ID); ok { - // Only continue flushing if the record we have in memory is the same as for which the flush - // job was requested. If it's not in memory, it has been evicted and we don't know if we hold - // the latest state or not. Similarly, if it's cached but the pointer is different, it means - // it was evicted and has been reloaded, so we're also uncertain if we hold the latest state. - if pr := cached.(*addrsRecord); pr == fj { - pr.RLock() - pr.Flush(ab.ds) - pr.RUnlock() - } - } + ab.closeDone.Add(1) + purgeTimer := time.NewTicker(ab.opts.GCPurgeInterval) + lookaheadTimer := time.NewTicker(ab.opts.GCLookaheadInterval) - case <-pruneTimer.C: - ab.purgeCycle() + for { + select { + case <-purgeTimer.C: + ab.purgeCycle() - case <-lookaheadTimer.C: - ab.populateLookahead() + case <-lookaheadTimer.C: + ab.populateLookahead() - case <-ab.ctx.Done(): - pruneTimer.Stop() - lookaheadTimer.Stop() - close(ab.closedCh) - return + case <-ab.ctx.Done(): + purgeTimer.Stop() + lookaheadTimer.Stop() + ab.closeDone.Done() + return + } } - } + }() } var purgeQuery = query.Query{Prefix: gcLookaheadBase.String(), KeysOnly: true} diff --git a/p2p/host/peerstore/pstoreds/ds_gc_test.go b/p2p/host/peerstore/pstoreds/ds_gc_test.go index 6de9539615..02b65bd658 100644 --- a/p2p/host/peerstore/pstoreds/ds_gc_test.go +++ b/p2p/host/peerstore/pstoreds/ds_gc_test.go @@ -142,3 +142,33 @@ func TestGCPurging(t *testing.T) { t.Errorf("expected remaining peer to be #3, got: %v, expected: %v", p, ids[3]) } } + +func TestGCDelay(t *testing.T) { + ids := test.GeneratePeerIDs(10) + addrs := test.GenerateAddrs(100) + + opts := DefaultOpts() + + opts.GCInitialDelay = 2 * time.Second + opts.GCLookaheadInterval = 2 * time.Second + opts.GCPurgeInterval = 6 * time.Hour + + factory := addressBookFactory(t, badgerStore, opts) + ab, closeFn := factory() + defer closeFn() + + tp := &testProbe{t, ab} + + ab.AddAddrs(ids[0], addrs, 1*time.Second) + + // immediately after we should be having no lookahead entries. + if i := tp.countLookaheadEntries(); i != 0 { + t.Errorf("expected no lookahead entries, got: %d", i) + } + + // delay + lookahead interval = 4 seconds + 2 seconds for some slack = 6 seconds. + <-time.After(6 * time.Second) + if i := tp.countLookaheadEntries(); i != 1 { + t.Errorf("expected 1 lookahead entry, got: %d", i) + } +} From 3ae2d75e46964bb069c1aeaef105529778904000 Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Sun, 2 Dec 2018 00:56:46 +0530 Subject: [PATCH 0867/3965] added comments for clarity --- p2p/protocol/identify/id.go | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index a2d52a48a9..8427a5d021 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -186,11 +186,18 @@ func (ids *IDService) populateMessage(mes *pb.Identify, c inet.Conn) { // set our public key ownKey := ids.Host.Peerstore().PubKey(ids.Host.ID()) - prKey := ids.Host.Peerstore().PrivKey(ids.Host.ID()) - if ownKey == nil && prKey != nil { - log.Errorf("did not have own public key in Peerstore") + // check if we even have a public key. + if ownKey == nil { + // public key is nil. We are either using insecure transport or something erratic happened. + // check if we're even operating in "secure mode" + if ids.Host.Peerstore().PrivKey(ids.Host.ID()) != nil { + // private key is present. But NO public key. Something bad happened. + log.Errorf("did not have own public key in Peerstore") + } + // if neither of the key is present it is safe to assume that we are using an insecure transport. } else if ownKey != nil { + // public key is present. Safe to proceed. if kb, err := ownKey.Bytes(); err != nil { log.Errorf("failed to convert key to bytes") } else { From 36a2645306a65a6f80b4f7a52cc312d61237e5d4 Mon Sep 17 00:00:00 2001 From: Abhishek Upperwal Date: Sun, 2 Dec 2018 01:00:46 +0530 Subject: [PATCH 0868/3965] removing redundant if condition --- p2p/protocol/identify/id.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 8427a5d021..d565bc5f68 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -196,7 +196,7 @@ func (ids *IDService) populateMessage(mes *pb.Identify, c inet.Conn) { log.Errorf("did not have own public key in Peerstore") } // if neither of the key is present it is safe to assume that we are using an insecure transport. - } else if ownKey != nil { + } else { // public key is present. Safe to proceed. if kb, err := ownKey.Bytes(); err != nil { log.Errorf("failed to convert key to bytes") From 6dda428253a2247786119d17631d70ed7da66943 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 3 Dec 2018 14:03:47 +0000 Subject: [PATCH 0869/3965] drop the need for a transactional datastore; introduce buffer pool. --- p2p/host/peerstore/pstoreds/addr_book.go | 100 ++++++----------------- p2p/host/peerstore/pstoreds/keybook.go | 4 +- p2p/host/peerstore/pstoreds/peerstore.go | 10 +-- 3 files changed, 29 insertions(+), 85 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index f19910a13b..1a57c07d0f 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -13,6 +13,7 @@ import ( query "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" + pool "github.com/libp2p/go-buffer-pool" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" pb "github.com/libp2p/go-libp2p-peerstore/pb" @@ -47,19 +48,21 @@ type addrsRecord struct { dirty bool } -// FlushInTxn writes the record to the datastore by calling ds.Put, unless the record is +// Flush writes the record to the datastore by calling ds.Put, unless the record is // marked for deletion, in which case the deletion is executed via ds.Delete. -func (r *addrsRecord) FlushInTxn(txn ds.Txn) (err error) { +func (r *addrsRecord) Flush(ds ds.Datastore) (err error) { key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id.ID))) if len(r.Addrs) == 0 { - return txn.Delete(key) + return ds.Delete(key) } - // cannot use a buffer pool because data is retained in the txn until it's committed or discarded. - data, err := r.Marshal() + data := pool.Get(r.Size()) + defer pool.Put(data) + + n, err := r.MarshalTo(data) if err != nil { return err } - if err = txn.Put(key, data); err != nil { + if err = ds.Put(key, data[:n]); err != nil { return err } // write succeeded; record is no longer dirty. @@ -67,20 +70,6 @@ func (r *addrsRecord) FlushInTxn(txn ds.Txn) (err error) { return nil } -// Flush creates a ds.Txn, and calls FlushInTxn with it. -func (r *addrsRecord) Flush(ds ds.TxnDatastore) (err error) { - txn, err := ds.NewTransaction(false) - if err != nil { - return err - } - defer txn.Discard() - - if err = r.FlushInTxn(txn); err != nil { - return err - } - return txn.Commit() -} - // Clean is called on records to perform housekeeping. The return value signals if the record was changed // as a result of the cleaning. // @@ -141,7 +130,7 @@ type dsAddrBook struct { opts Options cache cache - ds ds.TxnDatastore + ds ds.Datastore subsManager *pstoremem.AddrSubManager flushJobCh chan *addrsRecord @@ -210,14 +199,8 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs return pr, nil } - txn, err := ab.ds.NewTransaction(true) - if err != nil { - return nil, err - } - defer txn.Discard() - key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(id))) - data, err := txn.Get(key) + data, err := ab.ds.Get(key) if err != nil && err != ds.ErrNotFound { return nil, err @@ -316,12 +299,6 @@ func (ab *dsAddrBook) purgeCycle() { var id peer.ID record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} - txn, err := ab.ds.NewTransaction(false) - if err != nil { - log.Warningf("failed while purging entries: %v", err) - return - } - defer txn.Discard() // This function drops an unparseable GC entry; this is for safety. It is an escape hatch in case // we modify the format of keys going forward. If a user runs a new version against an old DB, @@ -330,7 +307,7 @@ func (ab *dsAddrBook) purgeCycle() { if err != nil { log.Warningf("failed while %s with GC key: %v, err: %v", msg, key, err) } - if err = txn.Delete(key); err != nil { + if err = ab.ds.Delete(key); err != nil { log.Warningf("failed to delete corrupt GC lookahead entry: %v, err: %v", key, err) } } @@ -338,20 +315,20 @@ func (ab *dsAddrBook) purgeCycle() { // This function drops a GC key if the entry is cleaned correctly. It may reschedule another visit // if the next earliest expiry falls within the current window again. dropOrReschedule := func(key ds.Key, ar *addrsRecord) { - if err = txn.Delete(key); err != nil { + if err := ab.ds.Delete(key); err != nil { log.Warningf("failed to delete lookahead entry: %v, err: %v", key, err) } // re-add the record if it needs to be visited again in this window. if len(ar.Addrs) != 0 && ar.Addrs[0].Expiry <= ab.gcCurrWindowEnd { gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", ar.Addrs[0].Expiry, key.Name())) - if err = txn.Put(gcKey, []byte{}); err != nil { + if err := ab.ds.Put(gcKey, []byte{}); err != nil { log.Warningf("failed to add new GC key: %v, err: %v", gcKey, err) } } } - results, err := txn.Query(purgeQuery) + results, err := ab.ds.Query(purgeQuery) if err != nil { log.Warningf("failed while fetching entries to purge: %v", err) return @@ -394,7 +371,7 @@ func (ab *dsAddrBook) purgeCycle() { cached := e.(*addrsRecord) cached.Lock() if cached.Clean() { - if err = cached.FlushInTxn(txn); err != nil { + if err = cached.Flush(ab.ds); err != nil { log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) } } @@ -407,7 +384,7 @@ func (ab *dsAddrBook) purgeCycle() { // otherwise, fetch it from the store, clean it and flush it. entryKey := addrBookBase.ChildString(gcKey.Name()) - val, err := txn.Get(entryKey) + val, err := ab.ds.Get(entryKey) if err != nil { // captures all errors, including ErrNotFound. dropInError(gcKey, err, "fetching entry") @@ -419,18 +396,13 @@ func (ab *dsAddrBook) purgeCycle() { continue } if record.Clean() { - err = record.FlushInTxn(txn) + err = record.Flush(ab.ds) if err != nil { log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) } } dropOrReschedule(gcKey, record) } - - err = txn.Commit() - if err != nil { - log.Warningf("failed to commit GC prune transaction: %v", err) - } } var populateLookaheadQuery = query.Query{Prefix: addrBookBase.String(), KeysOnly: true} @@ -449,14 +421,7 @@ func (ab *dsAddrBook) populateLookahead() { var id peer.ID record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} - txn, err := ab.ds.NewTransaction(false) - if err != nil { - log.Warningf("failed while filling lookahead GC region: %v", err) - return - } - defer txn.Discard() - - results, err := txn.Query(populateLookaheadQuery) + results, err := ab.ds.Query(populateLookaheadQuery) if err != nil { log.Warningf("failed while filling lookahead GC region: %v", err) return @@ -483,7 +448,7 @@ func (ab *dsAddrBook) populateLookahead() { continue } gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", cached.Addrs[0].Expiry, idb32)) - if err = txn.Put(gcKey, []byte{}); err != nil { + if err = ab.ds.Put(gcKey, []byte{}); err != nil { log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) } cached.RUnlock() @@ -492,7 +457,7 @@ func (ab *dsAddrBook) populateLookahead() { record.Reset() - val, err := txn.Get(ds.RawKey(result.Key)) + val, err := ab.ds.Get(ds.RawKey(result.Key)) if err != nil { log.Warningf("failed which getting record from store for peer: %v, err: %v", id.Pretty(), err) continue @@ -503,16 +468,12 @@ func (ab *dsAddrBook) populateLookahead() { } if len(record.Addrs) > 0 && record.Addrs[0].Expiry <= until { gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", record.Addrs[0].Expiry, idb32)) - if err = txn.Put(gcKey, []byte{}); err != nil { + if err = ab.ds.Put(gcKey, []byte{}); err != nil { log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) } } } - if err = txn.Commit(); err != nil { - log.Warningf("failed to commit GC lookahead transaction: %v", err) - } - ab.gcCurrWindowEnd = until atomic.StoreInt32(&ab.gcLookaheadRunning, 0) } @@ -612,19 +573,9 @@ func (ab *dsAddrBook) ClearAddrs(p peer.ID) { ab.cache.Remove(p) key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(p))) - txn, err := ab.ds.NewTransaction(false) - if err != nil { - log.Errorf("failed to clear addresses for peer %s: %v", p.Pretty(), err) - } - defer txn.Discard() - - if err := txn.Delete(key); err != nil { + if err := ab.ds.Delete(key); err != nil { log.Errorf("failed to clear addresses for peer %s: %v", p.Pretty(), err) } - - if err = txn.Commit(); err != nil { - log.Errorf("failed to commit transaction when deleting keys, cause: %v", err) - } } func (ab *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode) (err error) { @@ -669,9 +620,8 @@ Outer: Expiry: newExp, } added = append(added, entry) - // TODO: should we only broadcast if we updated the store successfully? - // we have no way of rolling back the state of the in-memory record, although we - // could at the expense of allocs. But is it worthwhile? + // note: there's a minor chance that writing the record will fail, in which case we would've broadcast + // the addresses without persisting them. This is very unlikely and not much of an issue. ab.subsManager.BroadcastAddr(p, addr) } diff --git a/p2p/host/peerstore/pstoreds/keybook.go b/p2p/host/peerstore/pstoreds/keybook.go index dc54b19239..8dcc82691d 100644 --- a/p2p/host/peerstore/pstoreds/keybook.go +++ b/p2p/host/peerstore/pstoreds/keybook.go @@ -23,12 +23,12 @@ var ( ) type dsKeyBook struct { - ds ds.TxnDatastore + ds ds.Datastore } var _ pstore.KeyBook = (*dsKeyBook)(nil) -func NewKeyBook(_ context.Context, store ds.TxnDatastore, _ Options) (pstore.KeyBook, error) { +func NewKeyBook(_ context.Context, store ds.Datastore, _ Options) (pstore.KeyBook, error) { return &dsKeyBook{store}, nil } diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index a189f66d8f..780797f2c9 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -64,20 +64,14 @@ func NewPeerstore(ctx context.Context, store ds.TxnDatastore, opts Options) (pst } // uniquePeerIds extracts and returns unique peer IDs from database keys. -func uniquePeerIds(ds ds.TxnDatastore, prefix ds.Key, extractor func(result query.Result) string) (peer.IDSlice, error) { +func uniquePeerIds(ds ds.Datastore, prefix ds.Key, extractor func(result query.Result) string) (peer.IDSlice, error) { var ( q = query.Query{Prefix: prefix.String(), KeysOnly: true} results query.Results err error ) - txn, err := ds.NewTransaction(true) - if err != nil { - return nil, err - } - defer txn.Discard() - - if results, err = txn.Query(q); err != nil { + if results, err = ds.Query(q); err != nil { log.Error(err) return nil, err } From 58828e8cc3023ed880854d801b734a0f8ebd726e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 3 Dec 2018 15:54:41 +0000 Subject: [PATCH 0870/3965] add a benchmark for lookahead cycles. --- p2p/host/peerstore/pstoreds/ds_gc_test.go | 31 +++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/p2p/host/peerstore/pstoreds/ds_gc_test.go b/p2p/host/peerstore/pstoreds/ds_gc_test.go index 02b65bd658..509817f0ac 100644 --- a/p2p/host/peerstore/pstoreds/ds_gc_test.go +++ b/p2p/host/peerstore/pstoreds/ds_gc_test.go @@ -172,3 +172,34 @@ func TestGCDelay(t *testing.T) { t.Errorf("expected 1 lookahead entry, got: %d", i) } } + +func BenchmarkLookaheadCycle(b *testing.B) { + ids := test.GeneratePeerIDs(100) + addrs := test.GenerateAddrs(100) + + opts := DefaultOpts() + + opts.GCInitialDelay = 2 * time.Hour + opts.GCLookaheadInterval = 2 * time.Hour + opts.GCPurgeInterval = 6 * time.Hour + + factory := addressBookFactory(b, badgerStore, opts) + ab, closeFn := factory() + defer closeFn() + + inside, outside := 1*time.Minute, 48*time.Hour + for i, id := range ids { + var ttl time.Duration + if i%2 == 0 { + ttl = inside + } else { + ttl = outside + } + ab.AddAddrs(id, addrs, ttl) + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + ab.(*dsAddrBook).populateLookahead() + } +} From 9f17c4077d131858ea0938df5acf50d82e5068ae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 3 Dec 2018 17:29:28 +0000 Subject: [PATCH 0871/3965] make gc operations run on cyclic batches. --- p2p/host/peerstore/pstoreds/addr_book.go | 217 +------------ p2p/host/peerstore/pstoreds/addr_book_gc.go | 296 ++++++++++++++++++ .../{ds_gc_test.go => addr_book_gc_test.go} | 0 p2p/host/peerstore/pstoreds/ds_test.go | 4 +- p2p/host/peerstore/pstoreds/peerstore.go | 2 +- 5 files changed, 309 insertions(+), 210 deletions(-) create mode 100644 p2p/host/peerstore/pstoreds/addr_book_gc.go rename p2p/host/peerstore/pstoreds/{ds_gc_test.go => addr_book_gc_test.go} (100%) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 1a57c07d0f..8ea007f19b 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -4,16 +4,13 @@ import ( "context" "fmt" "sort" - "strconv" "sync" - "sync/atomic" "time" ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - pool "github.com/libp2p/go-buffer-pool" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" pb "github.com/libp2p/go-libp2p-peerstore/pb" @@ -36,11 +33,13 @@ var ( // Peer addresses are stored under the following db key pattern: // /peers/addrs/ addrBookBase = ds.NewKey("/peers/addrs") - // GC lookahead entries are stored in keys with pattern: - // /peers/gc/addrs// => nil - gcLookaheadBase = ds.NewKey("/peers/gc/addrs") ) +type dsWriter interface { + Put(key ds.Key, value []byte) error + Delete(key ds.Key) error +} + // addrsRecord decorates the AddrBookRecord with locks and metadata. type addrsRecord struct { sync.RWMutex @@ -50,19 +49,17 @@ type addrsRecord struct { // Flush writes the record to the datastore by calling ds.Put, unless the record is // marked for deletion, in which case the deletion is executed via ds.Delete. -func (r *addrsRecord) Flush(ds ds.Datastore) (err error) { +func (r *addrsRecord) Flush(ds dsWriter) (err error) { key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id.ID))) if len(r.Addrs) == 0 { return ds.Delete(key) } - data := pool.Get(r.Size()) - defer pool.Put(data) - n, err := r.MarshalTo(data) + data, err := r.Marshal() if err != nil { return err } - if err = ds.Put(key, data[:n]); err != nil { + if err = ds.Put(key, data); err != nil { return err } // write succeeded; record is no longer dirty. @@ -130,7 +127,7 @@ type dsAddrBook struct { opts Options cache cache - ds ds.Datastore + ds ds.Batching subsManager *pstoremem.AddrSubManager flushJobCh chan *addrsRecord @@ -145,7 +142,7 @@ var _ pstore.AddrBook = (*dsAddrBook)(nil) // NewAddrBook initializes a new address book given a Datastore instance, a context for managing the TTL manager, // and the interval at which the TTL manager should sweep the Datastore. -func NewAddrBook(ctx context.Context, store ds.TxnDatastore, opts Options) (ab *dsAddrBook, err error) { +func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAddrBook, err error) { var cache cache = new(noopCache) if opts.CacheSize > 0 { if cache, err = lru.NewARC(int(opts.CacheSize)); err != nil { @@ -284,200 +281,6 @@ func (ab *dsAddrBook) background() { }() } -var purgeQuery = query.Query{Prefix: gcLookaheadBase.String(), KeysOnly: true} - -// purgeCycle runs a single GC cycle, operating within the lookahead window. -// -// It scans the lookahead region for entries that need to be visited, and performs a Clean() on them. An errors trigger -// the removal of the GC entry, in order to prevent unactionable items from accumulating. If the error happened to be -// temporary, the entry will be revisited in the next lookahead window. -func (ab *dsAddrBook) purgeCycle() { - if atomic.LoadInt32(&ab.gcLookaheadRunning) > 0 { - // yield if lookahead is running. - return - } - - var id peer.ID - record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} - - // This function drops an unparseable GC entry; this is for safety. It is an escape hatch in case - // we modify the format of keys going forward. If a user runs a new version against an old DB, - // if we don't clean up unparseable entries we'll end up accumulating garbage. - dropInError := func(key ds.Key, err error, msg string) { - if err != nil { - log.Warningf("failed while %s with GC key: %v, err: %v", msg, key, err) - } - if err = ab.ds.Delete(key); err != nil { - log.Warningf("failed to delete corrupt GC lookahead entry: %v, err: %v", key, err) - } - } - - // This function drops a GC key if the entry is cleaned correctly. It may reschedule another visit - // if the next earliest expiry falls within the current window again. - dropOrReschedule := func(key ds.Key, ar *addrsRecord) { - if err := ab.ds.Delete(key); err != nil { - log.Warningf("failed to delete lookahead entry: %v, err: %v", key, err) - } - - // re-add the record if it needs to be visited again in this window. - if len(ar.Addrs) != 0 && ar.Addrs[0].Expiry <= ab.gcCurrWindowEnd { - gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", ar.Addrs[0].Expiry, key.Name())) - if err := ab.ds.Put(gcKey, []byte{}); err != nil { - log.Warningf("failed to add new GC key: %v, err: %v", gcKey, err) - } - } - } - - results, err := ab.ds.Query(purgeQuery) - if err != nil { - log.Warningf("failed while fetching entries to purge: %v", err) - return - } - defer results.Close() - - now := time.Now().Unix() - - // keys: /peers/gc/addrs// - // values: nil - for result := range results.Next() { - gcKey := ds.RawKey(result.Key) - - ts, err := strconv.ParseInt(gcKey.Parent().Name(), 10, 64) - if err != nil { - dropInError(gcKey, err, "parsing timestamp") - log.Warningf("failed while parsing timestamp from key: %v, err: %v", result.Key, err) - continue - } else if ts > now { - // this is an ordered cursor; when we hit an entry with a timestamp beyond now, we can break. - break - } - - idb32, err := b32.RawStdEncoding.DecodeString(gcKey.Name()) - if err != nil { - dropInError(gcKey, err, "parsing peer ID") - log.Warningf("failed while parsing b32 peer ID from key: %v, err: %v", result.Key, err) - continue - } - - id, err = peer.IDFromBytes(idb32) - if err != nil { - dropInError(gcKey, err, "decoding peer ID") - log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) - continue - } - - // if the record is in cache, we clean it and flush it if necessary. - if e, ok := ab.cache.Peek(id); ok { - cached := e.(*addrsRecord) - cached.Lock() - if cached.Clean() { - if err = cached.Flush(ab.ds); err != nil { - log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) - } - } - dropOrReschedule(gcKey, cached) - cached.Unlock() - continue - } - - record.Reset() - - // otherwise, fetch it from the store, clean it and flush it. - entryKey := addrBookBase.ChildString(gcKey.Name()) - val, err := ab.ds.Get(entryKey) - if err != nil { - // captures all errors, including ErrNotFound. - dropInError(gcKey, err, "fetching entry") - continue - } - err = record.Unmarshal(val) - if err != nil { - dropInError(gcKey, err, "unmarshalling entry") - continue - } - if record.Clean() { - err = record.Flush(ab.ds) - if err != nil { - log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) - } - } - dropOrReschedule(gcKey, record) - } -} - -var populateLookaheadQuery = query.Query{Prefix: addrBookBase.String(), KeysOnly: true} - -// populateLookahead populates the lookahead window by scanning the entire store and picking entries whose earliest -// expiration falls within the new window. -// -// Those entries are stored in the lookahead region in the store, indexed by the timestamp when they need to be -// visited, to facilitate temporal range scans. -func (ab *dsAddrBook) populateLookahead() { - if !atomic.CompareAndSwapInt32(&ab.gcLookaheadRunning, 0, 1) { - return - } - - until := time.Now().Add(ab.opts.GCLookaheadInterval).Unix() - - var id peer.ID - record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} - results, err := ab.ds.Query(populateLookaheadQuery) - if err != nil { - log.Warningf("failed while filling lookahead GC region: %v", err) - return - } - defer results.Close() - - for result := range results.Next() { - idb32 := ds.RawKey(result.Key).Name() - k, err := b32.RawStdEncoding.DecodeString(idb32) - if err != nil { - log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) - continue - } - if id, err = peer.IDFromBytes(k); err != nil { - log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) - } - - // if the record is in cache, use the cached version. - if e, ok := ab.cache.Peek(id); ok { - cached := e.(*addrsRecord) - cached.RLock() - if len(cached.Addrs) == 0 || cached.Addrs[0].Expiry > until { - cached.RUnlock() - continue - } - gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", cached.Addrs[0].Expiry, idb32)) - if err = ab.ds.Put(gcKey, []byte{}); err != nil { - log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) - } - cached.RUnlock() - continue - } - - record.Reset() - - val, err := ab.ds.Get(ds.RawKey(result.Key)) - if err != nil { - log.Warningf("failed which getting record from store for peer: %v, err: %v", id.Pretty(), err) - continue - } - if err := record.Unmarshal(val); err != nil { - log.Warningf("failed while unmarshalling record from store for peer: %v, err: %v", id.Pretty(), err) - continue - } - if len(record.Addrs) > 0 && record.Addrs[0].Expiry <= until { - gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", record.Addrs[0].Expiry, idb32)) - if err = ab.ds.Put(gcKey, []byte{}); err != nil { - log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) - } - } - } - - ab.gcCurrWindowEnd = until - atomic.StoreInt32(&ab.gcLookaheadRunning, 0) -} - // AddAddr will add a new address if it's not already in the AddrBook. func (ab *dsAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { ab.AddAddrs(p, []ma.Multiaddr{addr}, ttl) diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go new file mode 100644 index 0000000000..17bf14d7f3 --- /dev/null +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -0,0 +1,296 @@ +package pstoreds + +import ( + "fmt" + "strconv" + "sync/atomic" + "time" + + ds "github.com/ipfs/go-datastore" + query "github.com/ipfs/go-datastore/query" + "github.com/pkg/errors" + + peer "github.com/libp2p/go-libp2p-peer" + pb "github.com/libp2p/go-libp2p-peerstore/pb" + + b32 "github.com/whyrusleeping/base32" +) + +var ( + // GC lookahead entries are stored in keys with pattern: + // /peers/gc/addrs// => nil + gcLookaheadBase = ds.NewKey("/peers/gc/addrs") + // in GC routines, how many operations do we place in a batch before it's committed. + gcOpsPerBatch = 20 + // queries + purgeQuery = query.Query{Prefix: gcLookaheadBase.String(), KeysOnly: true} + populateLookaheadQuery = query.Query{Prefix: addrBookBase.String(), KeysOnly: true} +) + +// cyclicBatch is similar to go-datastore autobatch, but it's driven by an actual Batch facility offered by the +// datastore. It populates an ongoing batch with operations and automatically flushes it after N pending operations +// have been reached. `N` is currently hardcoded to 20. An explicit `Commit()` closes this cyclic batch, erroring all +// further operations. +type cyclicBatch struct { + ds.Batch + ds ds.Batching + pending int +} + +func newCyclicBatch(ds ds.Batching) (ds.Batch, error) { + batch, err := ds.Batch() + if err != nil { + return nil, err + } + return &cyclicBatch{Batch: batch, ds: ds}, nil +} + +func (cb *cyclicBatch) cycle() (err error) { + if cb.Batch == nil { + return errors.New("cyclic batch is closed") + } + if cb.pending < gcOpsPerBatch { + // we haven't reached the threshold yet. + return nil + } + // commit and renew the batch. + if err = cb.Batch.Commit(); err != nil { + return errors.Wrap(err, "failed while committing cyclic batch") + } + if cb.Batch, err = cb.ds.Batch(); err != nil { + return errors.Wrap(err, "failed while renewing cyclic batch") + } + return nil +} + +func (cb *cyclicBatch) Put(key ds.Key, val []byte) error { + if err := cb.cycle(); err != nil { + return err + } + cb.pending++ + return cb.Batch.Put(key, val) +} + +func (cb *cyclicBatch) Delete(key ds.Key) error { + if err := cb.cycle(); err != nil { + return err + } + cb.pending++ + return cb.Batch.Delete(key) +} + +func (cb *cyclicBatch) Commit() error { + if err := cb.Batch.Commit(); err != nil { + return err + } + cb.pending = 0 + cb.Batch = nil + return nil +} + +// purgeCycle runs a single GC cycle, operating within the lookahead window. +// +// It scans the lookahead region for entries that need to be visited, and performs a Clean() on them. An errors trigger +// the removal of the GC entry, in order to prevent unactionable items from accumulating. If the error happened to be +// temporary, the entry will be revisited in the next lookahead window. +func (ab *dsAddrBook) purgeCycle() { + if atomic.LoadInt32(&ab.gcLookaheadRunning) > 0 { + // yield if lookahead is running. + return + } + + var id peer.ID + record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} + batch, err := newCyclicBatch(ab.ds) + if err != nil { + log.Warningf("failed while creating batch to purge GC entries: %v", err) + } + + // This function drops an unparseable GC entry; this is for safety. It is an escape hatch in case + // we modify the format of keys going forward. If a user runs a new version against an old DB, + // if we don't clean up unparseable entries we'll end up accumulating garbage. + dropInError := func(key ds.Key, err error, msg string) { + if err != nil { + log.Warningf("failed while %s with GC key: %v, err: %v", msg, key, err) + } + if err = batch.Delete(key); err != nil { + log.Warningf("failed to delete corrupt GC lookahead entry: %v, err: %v", key, err) + } + } + + // This function drops a GC key if the entry is cleaned correctly. It may reschedule another visit + // if the next earliest expiry falls within the current window again. + dropOrReschedule := func(key ds.Key, ar *addrsRecord) { + if err := batch.Delete(key); err != nil { + log.Warningf("failed to delete lookahead entry: %v, err: %v", key, err) + } + + // re-add the record if it needs to be visited again in this window. + if len(ar.Addrs) != 0 && ar.Addrs[0].Expiry <= ab.gcCurrWindowEnd { + gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", ar.Addrs[0].Expiry, key.Name())) + if err := batch.Put(gcKey, []byte{}); err != nil { + log.Warningf("failed to add new GC key: %v, err: %v", gcKey, err) + } + } + } + + results, err := ab.ds.Query(purgeQuery) + if err != nil { + log.Warningf("failed while fetching entries to purge: %v", err) + return + } + defer results.Close() + + now := time.Now().Unix() + + // keys: /peers/gc/addrs// + // values: nil + for result := range results.Next() { + gcKey := ds.RawKey(result.Key) + ts, err := strconv.ParseInt(gcKey.Parent().Name(), 10, 64) + if err != nil { + dropInError(gcKey, err, "parsing timestamp") + log.Warningf("failed while parsing timestamp from key: %v, err: %v", result.Key, err) + continue + } else if ts > now { + // this is an ordered cursor; when we hit an entry with a timestamp beyond now, we can break. + break + } + + idb32, err := b32.RawStdEncoding.DecodeString(gcKey.Name()) + if err != nil { + dropInError(gcKey, err, "parsing peer ID") + log.Warningf("failed while parsing b32 peer ID from key: %v, err: %v", result.Key, err) + continue + } + + id, err = peer.IDFromBytes(idb32) + if err != nil { + dropInError(gcKey, err, "decoding peer ID") + log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) + continue + } + + // if the record is in cache, we clean it and flush it if necessary. + if e, ok := ab.cache.Peek(id); ok { + cached := e.(*addrsRecord) + cached.Lock() + if cached.Clean() { + if err = cached.Flush(batch); err != nil { + log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) + } + } + dropOrReschedule(gcKey, cached) + cached.Unlock() + continue + } + + record.Reset() + + // otherwise, fetch it from the store, clean it and flush it. + entryKey := addrBookBase.ChildString(gcKey.Name()) + val, err := ab.ds.Get(entryKey) + if err != nil { + // captures all errors, including ErrNotFound. + dropInError(gcKey, err, "fetching entry") + continue + } + err = record.Unmarshal(val) + if err != nil { + dropInError(gcKey, err, "unmarshalling entry") + continue + } + if record.Clean() { + err = record.Flush(batch) + if err != nil { + log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) + } + } + dropOrReschedule(gcKey, record) + } + + if err = batch.Commit(); err != nil { + log.Warningf("failed to commit GC purge batch: %v", err) + } +} + +// populateLookahead populates the lookahead window by scanning the entire store and picking entries whose earliest +// expiration falls within the new window. +// +// Those entries are stored in the lookahead region in the store, indexed by the timestamp when they need to be +// visited, to facilitate temporal range scans. +func (ab *dsAddrBook) populateLookahead() { + if !atomic.CompareAndSwapInt32(&ab.gcLookaheadRunning, 0, 1) { + return + } + + until := time.Now().Add(ab.opts.GCLookaheadInterval).Unix() + + var id peer.ID + record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} + results, err := ab.ds.Query(populateLookaheadQuery) + if err != nil { + log.Warningf("failed while querying to populate lookahead GC window: %v", err) + return + } + defer results.Close() + + batch, err := newCyclicBatch(ab.ds) + if err != nil { + log.Warningf("failed while creating batch to populate lookahead GC window: %v", err) + return + } + + for result := range results.Next() { + idb32 := ds.RawKey(result.Key).Name() + k, err := b32.RawStdEncoding.DecodeString(idb32) + if err != nil { + log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) + continue + } + if id, err = peer.IDFromBytes(k); err != nil { + log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) + } + + // if the record is in cache, use the cached version. + if e, ok := ab.cache.Peek(id); ok { + cached := e.(*addrsRecord) + cached.RLock() + if len(cached.Addrs) == 0 || cached.Addrs[0].Expiry > until { + cached.RUnlock() + continue + } + gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", cached.Addrs[0].Expiry, idb32)) + if err = batch.Put(gcKey, []byte{}); err != nil { + log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) + } + cached.RUnlock() + continue + } + + record.Reset() + + val, err := ab.ds.Get(ds.RawKey(result.Key)) + if err != nil { + log.Warningf("failed which getting record from store for peer: %v, err: %v", id.Pretty(), err) + continue + } + if err := record.Unmarshal(val); err != nil { + log.Warningf("failed while unmarshalling record from store for peer: %v, err: %v", id.Pretty(), err) + continue + } + if len(record.Addrs) > 0 && record.Addrs[0].Expiry <= until { + gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", record.Addrs[0].Expiry, idb32)) + if err = batch.Put(gcKey, []byte{}); err != nil { + log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) + } + } + } + + if err = batch.Commit(); err != nil { + log.Warningf("failed to commit GC lookahead batch: %v", err) + } + + ab.gcCurrWindowEnd = until + atomic.StoreInt32(&ab.gcLookaheadRunning, 0) +} diff --git a/p2p/host/peerstore/pstoreds/ds_gc_test.go b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go similarity index 100% rename from p2p/host/peerstore/pstoreds/ds_gc_test.go rename to p2p/host/peerstore/pstoreds/addr_book_gc_test.go diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index abb712b49e..026fde855e 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -15,7 +15,7 @@ import ( pt "github.com/libp2p/go-libp2p-peerstore/test" ) -type datastoreFactory func(tb testing.TB) (ds.TxnDatastore, func()) +type datastoreFactory func(tb testing.TB) (ds.Batching, func()) var dstores = map[string]datastoreFactory{ "Badger": badgerStore, @@ -88,7 +88,7 @@ func BenchmarkDsPeerstore(b *testing.B) { } } -func badgerStore(tb testing.TB) (ds.TxnDatastore, func()) { +func badgerStore(tb testing.TB) (ds.Batching, func()) { dataPath, err := ioutil.TempDir(os.TempDir(), "badger") if err != nil { tb.Fatal(err) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 780797f2c9..71378d0743 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -43,7 +43,7 @@ func DefaultOpts() Options { } // NewPeerstore creates a peerstore backed by the provided persistent datastore. -func NewPeerstore(ctx context.Context, store ds.TxnDatastore, opts Options) (pstore.Peerstore, error) { +func NewPeerstore(ctx context.Context, store ds.Batching, opts Options) (pstore.Peerstore, error) { addrBook, err := NewAddrBook(ctx, store, opts) if err != nil { return nil, err From a1b5b9ed5328818452a28d40decd7abe4f20293a Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 4 Dec 2018 12:09:52 +0200 Subject: [PATCH 0872/3965] explicit option to enable autorelay --- config/config.go | 55 +++++++++++++++++++++----------- options.go | 9 ++++++ p2p/host/relay/autorelay_test.go | 4 +-- 3 files changed, 48 insertions(+), 20 deletions(-) diff --git a/config/config.go b/config/config.go index c503509465..58a34cb1ee 100644 --- a/config/config.go +++ b/config/config.go @@ -66,6 +66,8 @@ type Config struct { DisablePing bool Routing RoutingC + + EnableAutoRelay bool } // NewNode constructs a new libp2p Host from the Config. @@ -166,34 +168,51 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return nil, err } + // Configure routing and autorelay + var router routing.PeerRouting if cfg.Routing != nil { - router, err := cfg.Routing(h) + router, err = cfg.Routing(h) if err != nil { h.Close() return nil, err } + } + + if cfg.EnableAutoRelay { + if !cfg.Relay { + h.Close() + return nil, fmt.Errorf("cannot enable autorelay; relay is not enabled") + } + + if router == nil { + h.Close() + return nil, fmt.Errorf("cannot enable autorelay; no routing for discovery") + } crouter, ok := router.(routing.ContentRouting) - if ok { - if cfg.Relay { - discovery := discovery.NewRoutingDiscovery(crouter) - - hop := false - for _, opt := range cfg.RelayOpts { - if opt == circuit.OptHop { - hop = true - break - } - } - - if hop { - h = relay.NewRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) - } else { - h = relay.NewAutoRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) - } + if !ok { + h.Close() + return nil, fmt.Errorf("cannot enable autorelay; no suitable routing for discovery") + } + + discovery := discovery.NewRoutingDiscovery(crouter) + + hop := false + for _, opt := range cfg.RelayOpts { + if opt == circuit.OptHop { + hop = true + break } } + if hop { + h = relay.NewRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + } else { + h = relay.NewAutoRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + } + } + + if router != nil { h = routed.Wrap(h, router) } diff --git a/options.go b/options.go index 0015c80818..125ec8d601 100644 --- a/options.go +++ b/options.go @@ -220,6 +220,15 @@ func DisableRelay() Option { } } +// EnableAutoRelay configures libp2p to enable autorelay advertising; requires relay to +// be enabled and the Routing option to provide an instance of ContentRouting. +func EnableAutoRelay() Option { + return func(cfg *Config) error { + cfg.EnableAutoRelay = true + return nil + } +} + // FilterAddresses configures libp2p to never dial nor accept connections from // the given addresses. func FilterAddresses(addrs ...*net.IPNet) Option { diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index ce50ea8c09..82a0f8ac6f 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -142,11 +142,11 @@ func TestAutoRelay(t *testing.T) { } h1 := makeAutoNATServicePrivate(ctx, t) - _, err := libp2p.New(ctx, libp2p.EnableRelay(circuit.OptHop), libp2p.Routing(makeRouting)) + _, err := libp2p.New(ctx, libp2p.EnableRelay(circuit.OptHop), libp2p.EnableAutoRelay(), libp2p.Routing(makeRouting)) if err != nil { t.Fatal(err) } - h3, err := libp2p.New(ctx, libp2p.EnableRelay(), libp2p.Routing(makeRouting)) + h3, err := libp2p.New(ctx, libp2p.EnableRelay(), libp2p.EnableAutoRelay(), libp2p.Routing(makeRouting)) if err != nil { t.Fatal(err) } From aa4a32b94c6c2c84b3d60272d2519a5335dc410f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 4 Dec 2018 11:47:22 +0000 Subject: [PATCH 0873/3965] return error when committing a closed cyclic batch. --- p2p/host/peerstore/pstoreds/addr_book_gc.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 17bf14d7f3..a0212fa9bb 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -8,7 +8,7 @@ import ( ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" - "github.com/pkg/errors" + errors "github.com/pkg/errors" peer "github.com/libp2p/go-libp2p-peer" pb "github.com/libp2p/go-libp2p-peerstore/pb" @@ -80,6 +80,9 @@ func (cb *cyclicBatch) Delete(key ds.Key) error { } func (cb *cyclicBatch) Commit() error { + if cb.Batch == nil { + return errors.New("cyclic batch is closed") + } if err := cb.Batch.Commit(); err != nil { return err } From 5d0a6e23ce373afa83515d34bedc1493b4fac0bc Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 5 Dec 2018 10:02:56 +0200 Subject: [PATCH 0874/3965] use select for initial advertisement delay --- p2p/host/relay/relay.go | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index 3f282a13c1..b5db9ce586 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -31,8 +31,11 @@ func NewRelayHost(ctx context.Context, bhost *basic.BasicHost, advertise discove } bhost.AddrsFactory = h.hostAddrs go func() { - time.Sleep(AdvertiseBootDelay) - discovery.Advertise(ctx, advertise, RelayRendezvous) + select { + case <-time.After(AdvertiseBootDelay): + discovery.Advertise(ctx, advertise, RelayRendezvous) + case <-ctx.Done(): + } }() return h } From 98a47e949a02b45a7be72aa5fecff7a620c5533e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 7 Dec 2018 13:47:26 -0800 Subject: [PATCH 0875/3965] disable inline-peer id test --- p2p/host/peerstore/test/keybook_suite.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/peerstore/test/keybook_suite.go b/p2p/host/peerstore/test/keybook_suite.go index f4f84967e2..9803128bd6 100644 --- a/p2p/host/peerstore/test/keybook_suite.go +++ b/p2p/host/peerstore/test/keybook_suite.go @@ -140,6 +140,8 @@ func testKeyBookPeers(kb pstore.KeyBook) func(t *testing.T) { func testInlinedPubKeyAddedOnRetrieve(kb pstore.KeyBook) func(t *testing.T) { return func(t *testing.T) { + t.Skip("key inlining disabled for now: see libp2p/specs#111") + if peers := kb.PeersWithKeys(); len(peers) > 0 { t.Error("expected peers to be empty on init") } From 2ad7f703fea582f5470eb7439c6807a6123e274f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 7 Dec 2018 14:51:00 -0800 Subject: [PATCH 0876/3965] gx publish 6.0.28 --- .gx/lastpubver | 2 +- package.json | 90 +++++++++++++++++++++++++------------------------- 2 files changed, 46 insertions(+), 46 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 25c4296c72..977a9ca238 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.27: QmVvV8JQmmqPCwXAaesWJPheUiEFQJ9HWRhWhuFuxVQxpR +6.0.28: QmRBaUEQEeFWywfrZJ64QgsmvcqgLSK3VbvGMR2NM2Edpf diff --git a/package.json b/package.json index 5afdc6cac0..1db03ad909 100644 --- a/package.json +++ b/package.json @@ -50,33 +50,33 @@ }, { "author": "whyrusleeping", - "hash": "QmbP5E6C5Ks2b43vTZoyjSUHx2PPgoe78PZpJSJqCJ7LYx", + "hash": "QmQSREHX6CMoPkT5FfuA4A9cWSFwoKY8RAzjf5m7Gpdtqu", "name": "go-libp2p-loggables", - "version": "1.1.27" + "version": "1.1.28" }, { "author": "whyrusleeping", - "hash": "QmcE2awGbLPqCDH5PbqBTyqeoYpBAFNSnYK2Kw8azQkqXH", + "hash": "QmaTTgNexJg8UVJdJkPzJdNuqGtKDK9BsK31iqK9tn7m8Z", "name": "go-libp2p-secio", - "version": "2.0.22" + "version": "2.0.23" }, { "author": "whyrusleeping", - "hash": "QmQAGG1zxfePqj2t7bLxyN8AFccZ889DDR9Gn8kVLDrGZo", + "hash": "QmZ9zH2FnLcxv1xyzFeUpDUeo55xEhZQHgveZijcxr7TLj", "name": "go-libp2p-peerstore", - "version": "2.0.11" + "version": "2.0.12" }, { "author": "whyrusleeping", - "hash": "QmdQx4ZhKGdv9TvpCFpMxFzjTQFHRmFqjBxkRVwzT1JNes", + "hash": "Qmb3qartY8DSgRaBA3Go4EEjY1ZbXhCcvmc4orsBKMjgRg", "name": "go-libp2p-transport", - "version": "3.0.19" + "version": "3.0.20" }, { "author": "whyrusleeping", - "hash": "QmUmXYaTmtvefgKoVA7hY3uUN8D1CiqxufigSyWuWwQjXy", + "hash": "QmVJrp38KHSdysXZyWq3wK9RtdYW843oa9Psf9MVU6NNx3", "name": "go-tcp-transport", - "version": "2.0.20" + "version": "2.0.21" }, { "author": "whyrusleeping", @@ -92,33 +92,33 @@ }, { "author": "whyrusleeping", - "hash": "QmZXjR5X1p4KrQ967cTsy4MymMzUM8mZECF3PV8UcN4o3g", + "hash": "QmPuhRE325DR8ChNcFtgd6F1eANCHy1oohXZPpYop4xsK6", "name": "go-testutil", - "version": "1.2.13" + "version": "1.2.14" }, { "author": "whyrusleeping", - "hash": "QmenvQQy4bFGSiHJUGupVmCRHfetg5rH3vTp9Z2f6v2KXR", + "hash": "QmPtFaR7BWHLAjSwLh9kXcyrgTzDpuhcWLkx8ioa9RMYnx", "name": "go-libp2p-net", - "version": "3.0.20" + "version": "3.0.21" }, { "author": "whyrusleeping", - "hash": "QmfBAmuDFoPTMC232UQenPDYAzHQ48crKaXG9AfQqFuRpN", + "hash": "QmbYN6UmTJn5UUQdi5CTsU86TXVBSrTcRk5UmyA36Qx2J6", "name": "go-libp2p-metrics", - "version": "2.1.10" + "version": "2.1.11" }, { "author": "whyrusleeping", - "hash": "QmahxMNoNuSsgQefo9rkpcfRFmQrMN6Q99aztKXf63K7YJ", + "hash": "QmfD51tKgJiTMnW9JEiDiPwsCY4mqUoxkhKhBfyW12spTC", "name": "go-libp2p-host", - "version": "3.0.19" + "version": "3.0.20" }, { "author": "whyrusleeping", - "hash": "QmQrYHkcGprZBUFnRigeiZFkaFDBHtmRhDdPpSiiUTRNwv", + "hash": "QmQdLXW5JTSsrVb3ZpnpbASRwyM8CcE4XcM5nPbN19dWLr", "name": "go-libp2p-swarm", - "version": "3.0.26" + "version": "3.0.27" }, { "author": "whyrusleeping", @@ -128,15 +128,15 @@ }, { "author": "whyrusleeping", - "hash": "QmPphQ7HMN49UGAwX3VhxibKkNP3fjLDdkcM5SjhksZSNq", + "hash": "QmdMbtWsbi7rn8oZ3gyduKv3Df4MXtwrbn2MyVEgUPF6Mc", "name": "go-libp2p-netutil", - "version": "0.4.16" + "version": "0.4.17" }, { "author": "whyrusleeping", - "hash": "QmWnd4JYBouYJMuW9Gt5a5hFpW2whwgAaxyxBWKXZXzVgy", + "hash": "QmWHq17GUcxRuXL7duPuQPuVM8adyRoMfP8TGfy1MK9N5r", "name": "go-libp2p-blankhost", - "version": "0.3.19" + "version": "0.3.20" }, { "author": "whyrusleeping", @@ -158,15 +158,15 @@ }, { "author": "whyrusleeping", - "hash": "QmcqU6QUDSXprb1518vYDGczrTJTyGwLG9eUa5iNX4xUtS", + "hash": "QmY5Grm8pJdiSSVsYxx4uNRgweY72EmYwuSDbRnbFok3iY", "name": "go-libp2p-peer", - "version": "2.4.3" + "version": "3.0.0" }, { "author": "vyzo", - "hash": "QmWxCQ66YzAb24Py5D3qeDJcVTF4aW9AeAmppQ7FhQZxgZ", + "hash": "QmNcNWuV38HBGYtRUi3okmfXSMEmXWwNgb82N3PzqqsHhY", "name": "go-libp2p-circuit", - "version": "2.3.6" + "version": "2.3.7" }, { "author": "lgierth", @@ -176,9 +176,9 @@ }, { "author": "why", - "hash": "QmQSucBpqUVQ5Q1stDmm2Bon4Tq4KNhNXuVmLMraARoUoh", + "hash": "QmRkzmq686MdtAVHZLncm3sXCzyFzBq4eLxk2rch2r788f", "name": "go-libp2p-interface-connmgr", - "version": "0.0.25" + "version": "0.0.26" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmP2qH43GBjhUo1XPrn5k6y4fvahSrLwNPf4ddjitxo2Kz", + "hash": "QmZJpaENuYSCLnV4gWjrWhN2LMJWgTp3u2b6XZx3C12rNF", "name": "go-ws-transport", - "version": "2.0.19" + "version": "2.0.20" }, { "author": "stebalien", - "hash": "QmPcoYfH3kjZmasoVuVWrjPvdX5zwG4U8iXgEgZJD14dBf", + "hash": "QmX8fjWTb1hynPkHrssWc6YGwphXmfnikLa6VxQzQjMRKJ", "name": "go-conn-security-multistream", - "version": "0.1.19" + "version": "0.1.20" }, { "author": "Stebalien", - "hash": "QmbyjEKtxXmZdiByBiNhfayzEuEPPBvuD2dLpHky8cHUvy", + "hash": "QmXCnmY9nBCDQBhzxrn5ZKqSYYHiFzc5btFt32UpgECHnS", "name": "go-conn-security", - "version": "0.1.20" + "version": "0.1.21" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "QmPbKqnriyf7c2Kr5NHR2tw52SkWqdb1uHxLWT3h3qBmeS", + "hash": "Qmc9KUyhx1adPnHX2TBjEWvKej2Gg2kvisAFoQ74UiWYhd", "name": "go-libp2p-transport-upgrader", - "version": "0.1.20" + "version": "0.1.21" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -229,20 +229,20 @@ }, { "author": "vyzo", - "hash": "QmejQHZsodEXxdeFQazcWsi4Dkmp4mX7QEZrWXHoVR5EtK", + "hash": "QmYJtCabf3prS3HKQUGgqDLVxvbT9iDx5mfeVfhtCcJxxE", "name": "go-libp2p-discovery", - "version": "1.0.3" + "version": "1.0.4" }, { "author": "vyzo", - "hash": "QmYdJ7pcu5fbwBeiue9TPaNie3QwPaGsh6sPzZiz9gMgLQ", + "hash": "QmQXeTmQcPFnf3ZAvik2qgKxWNoZ27aKGcW8hUBwBrTxT1", "name": "go-libp2p-autonat", - "version": "1.0.3" + "version": "1.0.4" }, { - "hash": "QmZBH87CAPFHcc7cYmBqeSQ98zQ3SX9KUxiYgzPmLWNVKz", + "hash": "QmRASJXJUFygM5qU4YrH7k7jD6S4Hg8nJmgqJ4bYJvLatd", "name": "go-libp2p-routing", - "version": "2.7.5" + "version": "2.7.6" } ], "gxVersion": "0.4.0", @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.27" + "version": "6.0.28" } From 55a490b4ad12846bd710083d047c20a7a5597e40 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 12 Dec 2018 18:30:02 +0000 Subject: [PATCH 0877/3965] fix silence period between trims not being honored. --- p2p/net/connmgr/connmgr.go | 39 +++++++++++++-------- p2p/net/connmgr/connmgr_test.go | 61 ++++++++++++++++++++++++++------- 2 files changed, 74 insertions(+), 26 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 8e7d5aaafb..af88319cc0 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -4,6 +4,7 @@ import ( "context" "sort" "sync" + "sync/atomic" "time" logging "github.com/ipfs/go-log" @@ -13,6 +14,8 @@ import ( ma "github.com/multiformats/go-multiaddr" ) +const silencePeriod = 10 * time.Second + var log = logging.Logger("connmgr") // BasicConnMgr is a ConnManager that trims connections whenever the count exceeds the @@ -23,17 +26,16 @@ var log = logging.Logger("connmgr") // // See configuration parameters in NewConnManager. type BasicConnMgr struct { - highWater int - lowWater int - + lk sync.Mutex + highWater int + lowWater int + connCount int gracePeriod time.Duration + peers map[peer.ID]*peerInfo - peers map[peer.ID]*peerInfo - connCount int - - lk sync.Mutex - - lastTrim time.Time + // guarded by atomic; 0=not running, 1=running. + trimRunning int32 + lastTrim time.Time } var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) @@ -68,12 +70,25 @@ type peerInfo struct { // pruning those peers with the lowest scores first, as long as they are not within their // grace period. func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { + if !atomic.CompareAndSwapInt32(&cm.trimRunning, 0, 1) { + // a trim is running, skip this one. + return + } + + defer atomic.StoreInt32(&cm.trimRunning, 0) + if time.Since(cm.lastTrim) < silencePeriod { + // skip this attempt to trim as the last one just took place. + return + } + defer log.EventBegin(ctx, "connCleanup").Done() for _, c := range cm.getConnsToClose(ctx) { log.Info("closing conn: ", c.RemotePeer()) log.Event(ctx, "closeConn", c.RemotePeer()) c.Close() } + + cm.lastTrim = time.Now() } // getConnsToClose runs the heuristics described in TrimOpenConns and returns the @@ -87,8 +102,6 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { return nil } now := time.Now() - cm.lastTrim = now - if len(cm.peers) < cm.lowWater { log.Info("open connection count below limit") return nil @@ -263,9 +276,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { cm.connCount++ if cm.connCount > nn.highWater { - if cm.lastTrim.IsZero() || time.Since(cm.lastTrim) > time.Second*10 { - go cm.TrimOpenConns(context.Background()) - } + go cm.TrimOpenConns(context.Background()) } } diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index cf11f208db..90285c5955 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -13,12 +13,17 @@ import ( type tconn struct { inet.Conn - peer peer.ID - closed bool + + peer peer.ID + closed bool + disconnectNotify func(net inet.Network, conn inet.Conn) } func (c *tconn) Close() error { c.closed = true + if c.disconnectNotify != nil { + c.disconnectNotify(nil, c) + } return nil } @@ -34,9 +39,9 @@ func (c *tconn) RemoteMultiaddr() ma.Multiaddr { return addr } -func randConn(t *testing.T) inet.Conn { +func randConn(t *testing.T, discNotify func(inet.Network, inet.Conn)) inet.Conn { pid := tu.RandPeerIDFatal(t) - return &tconn{peer: pid} + return &tconn{peer: pid, disconnectNotify: discNotify} } func TestConnTrimming(t *testing.T) { @@ -45,7 +50,7 @@ func TestConnTrimming(t *testing.T) { var conns []inet.Conn for i := 0; i < 300; i++ { - rc := randConn(t) + rc := randConn(t, nil) conns = append(conns, rc) not.Connected(nil, rc) } @@ -98,7 +103,7 @@ func TestConnsToClose(t *testing.T) { cm = NewConnManager(1, 1, time.Duration(10*time.Minute)) not := cm.Notifee() for i := 0; i < 5; i++ { - conn := randConn(t) + conn := randConn(t, nil) not.Connected(nil, conn) } conns = cm.getConnsToClose(context.Background()) @@ -111,7 +116,7 @@ func TestGetTagInfo(t *testing.T) { start := time.Now() cm := NewConnManager(1, 1, time.Duration(10*time.Minute)) not := cm.Notifee() - conn := randConn(t) + conn := randConn(t, nil) not.Connected(nil, conn) end := time.Now() @@ -192,7 +197,7 @@ func TestTagPeerNonExistant(t *testing.T) { func TestUntagPeer(t *testing.T) { cm := NewConnManager(1, 1, time.Duration(10*time.Minute)) not := cm.Notifee() - conn := randConn(t) + conn := randConn(t, nil) not.Connected(nil, conn) rp := conn.RemotePeer() cm.TagPeer(rp, "tag", 5) @@ -223,7 +228,7 @@ func TestGetInfo(t *testing.T) { gp := time.Duration(10 * time.Minute) cm := NewConnManager(1, 5, gp) not := cm.Notifee() - conn := randConn(t) + conn := randConn(t, nil) not.Connected(nil, conn) cm.TrimOpenConns(context.Background()) end := time.Now() @@ -250,7 +255,7 @@ func TestDoubleConnection(t *testing.T) { gp := time.Duration(10 * time.Minute) cm := NewConnManager(1, 5, gp) not := cm.Notifee() - conn := randConn(t) + conn := randConn(t, nil) not.Connected(nil, conn) cm.TagPeer(conn.RemotePeer(), "foo", 10) not.Connected(nil, conn) @@ -266,11 +271,11 @@ func TestDisconnected(t *testing.T) { gp := time.Duration(10 * time.Minute) cm := NewConnManager(1, 5, gp) not := cm.Notifee() - conn := randConn(t) + conn := randConn(t, nil) not.Connected(nil, conn) cm.TagPeer(conn.RemotePeer(), "foo", 10) - not.Disconnected(nil, randConn(t)) + not.Disconnected(nil, randConn(t, nil)) if cm.connCount != 1 { t.Fatal("unexpected number of connections") } @@ -294,3 +299,35 @@ func TestDisconnected(t *testing.T) { t.Fatal("unexpected number of peers") } } + +// see https://github.com/libp2p/go-libp2p-connmgr/issues/23 +func TestQuickBurstRespectsSilencePeriod(t *testing.T) { + cm := NewConnManager(10, 20, 0) + not := cm.Notifee() + + var conns []inet.Conn + + // quickly produce 30 connections (sending us above the high watermark) + for i := 0; i < 30; i++ { + rc := randConn(t, not.Disconnected) + conns = append(conns, rc) + not.Connected(nil, rc) + } + + // wait for a few seconds + time.Sleep(time.Second * 3) + + // only the first trim is allowed in; make sure we close at most 20 connections, not all of them. + var closed int + for _, c := range conns { + if c.(*tconn).closed { + closed++ + } + } + if closed > 20 { + t.Fatalf("should have closed at most 20 connections, closed: %d", closed) + } + if total := closed + cm.connCount; total != 30 { + t.Fatalf("expected closed connections + open conn count to equal 30, value: %d", total) + } +} From 97f80a6ca59e929383849085a8822d6898c76f25 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 13 Dec 2018 12:58:51 +0000 Subject: [PATCH 0878/3965] switch to channel-based semaphore to ensure one trim in progress. --- p2p/net/connmgr/connmgr.go | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index af88319cc0..626fe76175 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -4,13 +4,12 @@ import ( "context" "sort" "sync" - "sync/atomic" "time" logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-interface-connmgr" + ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" inet "github.com/libp2p/go-libp2p-net" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) @@ -33,9 +32,9 @@ type BasicConnMgr struct { gracePeriod time.Duration peers map[peer.ID]*peerInfo - // guarded by atomic; 0=not running, 1=running. - trimRunning int32 - lastTrim time.Time + // channel-based semaphore that enforces only a single trim is in progress + trimRunningCh chan struct{} + lastTrim time.Time } var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) @@ -48,10 +47,11 @@ var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) // subject to pruning. func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { return &BasicConnMgr{ - highWater: hi, - lowWater: low, - gracePeriod: grace, - peers: make(map[peer.ID]*peerInfo), + highWater: hi, + lowWater: low, + gracePeriod: grace, + peers: make(map[peer.ID]*peerInfo), + trimRunningCh: make(chan struct{}, 1), } } @@ -69,25 +69,26 @@ type peerInfo struct { // equal the low watermark. Peers are sorted in ascending order based on their total value, // pruning those peers with the lowest scores first, as long as they are not within their // grace period. +// +// TODO: error return value so we can cleanly signal we are aborting because: +// (a) there's another trim in progress, or (b) the silence period is in effect. func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { - if !atomic.CompareAndSwapInt32(&cm.trimRunning, 0, 1) { - // a trim is running, skip this one. + select { + case cm.trimRunningCh <- struct{}{}: + default: return } - - defer atomic.StoreInt32(&cm.trimRunning, 0) + defer func() { <-cm.trimRunningCh }() if time.Since(cm.lastTrim) < silencePeriod { // skip this attempt to trim as the last one just took place. return } - defer log.EventBegin(ctx, "connCleanup").Done() for _, c := range cm.getConnsToClose(ctx) { log.Info("closing conn: ", c.RemotePeer()) log.Event(ctx, "closeConn", c.RemotePeer()) c.Close() } - cm.lastTrim = time.Now() } From c78f6942f7e2c088e10186a5aa1f0a14198f69be Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Tue, 18 Dec 2018 20:12:51 +0100 Subject: [PATCH 0879/3965] gx publish 6.0.29 License: MIT Signed-off-by: Hector Sanjuan --- .gx/lastpubver | 2 +- package.json | 102 ++++++++++++++++++++++++------------------------- 2 files changed, 52 insertions(+), 52 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 977a9ca238..9edfa6bcdf 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.28: QmRBaUEQEeFWywfrZJ64QgsmvcqgLSK3VbvGMR2NM2Edpf +6.0.29: QmdJdFQc5U3RAKgJQGmWR7SSM7TLuER5FWz5Wq6Tzs2CnS diff --git a/package.json b/package.json index 1db03ad909..58d1d6d905 100644 --- a/package.json +++ b/package.json @@ -39,50 +39,50 @@ "version": "1.5.8" }, { - "hash": "QmQVUtnrNGtCRkCMpXgpApfzQjc8FDaDVxHqWH8cnZQeh5", + "hash": "QmZcLBXKaFe8ND5YHPkJRAwmhJGrVsi1JqDZNyJ4nRK5Mj", "name": "go-multiaddr-net", - "version": "1.6.6" + "version": "1.7.1" }, { - "hash": "QmRKLtwMw131aK7ugC3G7ybpumMz78YrJe5dzneyindvG1", + "hash": "QmNTCey11oxhb1AxDnQBRHtdhap6Ctud872NjAYPYYXPuc", "name": "go-multiaddr", - "version": "1.3.6" + "version": "1.4.0" }, { "author": "whyrusleeping", - "hash": "QmQSREHX6CMoPkT5FfuA4A9cWSFwoKY8RAzjf5m7Gpdtqu", + "hash": "QmWZnkipEJbsdUDhgTGBnKAN1iHM2WDMNqsXHyAuaAiCgo", "name": "go-libp2p-loggables", - "version": "1.1.28" + "version": "1.1.29" }, { "author": "whyrusleeping", - "hash": "QmaTTgNexJg8UVJdJkPzJdNuqGtKDK9BsK31iqK9tn7m8Z", + "hash": "QmQoDgEDBPJ6RfKDgdoMpPwoK4WEZHUosHoUs6YVDGoen3", "name": "go-libp2p-secio", - "version": "2.0.23" + "version": "2.0.24" }, { "author": "whyrusleeping", - "hash": "QmZ9zH2FnLcxv1xyzFeUpDUeo55xEhZQHgveZijcxr7TLj", + "hash": "QmPiemjiKBC9VA7vZF82m4x1oygtg2c2YVqag8PX7dN1BD", "name": "go-libp2p-peerstore", - "version": "2.0.12" + "version": "2.0.13" }, { "author": "whyrusleeping", - "hash": "Qmb3qartY8DSgRaBA3Go4EEjY1ZbXhCcvmc4orsBKMjgRg", + "hash": "QmS4UBXoQ5QgTJA5pc62egqa5KrQRhsDHhaFHEoGUASsxp", "name": "go-libp2p-transport", - "version": "3.0.20" + "version": "3.0.21" }, { "author": "whyrusleeping", - "hash": "QmVJrp38KHSdysXZyWq3wK9RtdYW843oa9Psf9MVU6NNx3", + "hash": "QmNUchfAD5CpjV5QZwVBpKpgNtVD1Vk2k7arGsC1yPnPin", "name": "go-tcp-transport", - "version": "2.0.21" + "version": "2.0.22" }, { "author": "whyrusleeping", - "hash": "QmQJRvWaYAvU3Mdtk33ADXr9JAZwKMBYBGPkRQBDvyj2nn", + "hash": "QmQgSnRC74nHoXrN9CShvfWUUSrgAMJ4unjbnuBVsxk2mw", "name": "go-maddr-filter", - "version": "1.1.11" + "version": "1.1.12" }, { "author": "whyrusleeping", @@ -92,15 +92,15 @@ }, { "author": "whyrusleeping", - "hash": "QmPuhRE325DR8ChNcFtgd6F1eANCHy1oohXZPpYop4xsK6", + "hash": "QmNvHv84aH2qZafDuSdKJCQ1cvPZ1kmQmyD4YtzjUHuk9v", "name": "go-testutil", - "version": "1.2.14" + "version": "1.2.15" }, { "author": "whyrusleeping", - "hash": "QmPtFaR7BWHLAjSwLh9kXcyrgTzDpuhcWLkx8ioa9RMYnx", + "hash": "QmNgLg1NTw37iWbYPKcyK85YJ9Whs1MkPtJwhfqbNYAyKg", "name": "go-libp2p-net", - "version": "3.0.21" + "version": "3.0.22" }, { "author": "whyrusleeping", @@ -110,33 +110,33 @@ }, { "author": "whyrusleeping", - "hash": "QmfD51tKgJiTMnW9JEiDiPwsCY4mqUoxkhKhBfyW12spTC", + "hash": "QmaoXrM4Z41PD48JY36YqQGKQpLGjyLA2cKcLsES7YddAq", "name": "go-libp2p-host", - "version": "3.0.20" + "version": "3.0.21" }, { "author": "whyrusleeping", - "hash": "QmQdLXW5JTSsrVb3ZpnpbASRwyM8CcE4XcM5nPbN19dWLr", + "hash": "QmegQFxhr1J6yZ1vDQuDmJi5jntmj6BL96S11HVtXNCaHb", "name": "go-libp2p-swarm", - "version": "3.0.27" + "version": "3.0.28" }, { "author": "whyrusleeping", - "hash": "QmYVUZr4KxAD7ginKydBtJpVzy2nEe7vXyKvJYjneAX273", + "hash": "QmXFPLTyWRrWp4zkNrD5S3s8S5LRJyJx67hQwzwaTYtVzi", "name": "go-libp2p-nat", - "version": "0.8.9" + "version": "0.8.10" }, { "author": "whyrusleeping", - "hash": "QmdMbtWsbi7rn8oZ3gyduKv3Df4MXtwrbn2MyVEgUPF6Mc", + "hash": "QmcD6i5pkzKcy5AiSEU22xmj8MYkkPSaXVvbYo7Wx2hp6H", "name": "go-libp2p-netutil", - "version": "0.4.17" + "version": "0.4.18" }, { "author": "whyrusleeping", - "hash": "QmWHq17GUcxRuXL7duPuQPuVM8adyRoMfP8TGfy1MK9N5r", + "hash": "QmQLbY1oKd4eHrikizXXwYkxn6yujUNSUMimv3UCaWTSWX", "name": "go-libp2p-blankhost", - "version": "0.3.20" + "version": "0.3.21" }, { "author": "whyrusleeping", @@ -164,21 +164,21 @@ }, { "author": "vyzo", - "hash": "QmNcNWuV38HBGYtRUi3okmfXSMEmXWwNgb82N3PzqqsHhY", + "hash": "QmWuMW6UKZMJo9bFFDwnjg8tW3AtKisMHHrXEutQdmJ19N", "name": "go-libp2p-circuit", - "version": "2.3.7" + "version": "2.3.8" }, { "author": "lgierth", - "hash": "QmT4zgnKCyZBpRyxzsvZqUjzUkMWLJ2pZCw7uk6M6Kto5m", + "hash": "QmQc7jbDUsxUJZyFJzxVrnrWeECCct6fErEpMqtjyWvCX8", "name": "go-multiaddr-dns", - "version": "0.2.6" + "version": "0.3.0" }, { "author": "why", - "hash": "QmRkzmq686MdtAVHZLncm3sXCzyFzBq4eLxk2rch2r788f", + "hash": "QmSFo2QrMF4M1mKdB291ZqNtsie4NfwXCRdWgDU3inw4Ff", "name": "go-libp2p-interface-connmgr", - "version": "0.0.26" + "version": "0.0.27" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmZJpaENuYSCLnV4gWjrWhN2LMJWgTp3u2b6XZx3C12rNF", + "hash": "QmPMuvbzktV1oUfBVf6dc3t2DF5qyB2pKKqhDfAZbcbRg6", "name": "go-ws-transport", - "version": "2.0.20" + "version": "2.0.21" }, { "author": "stebalien", - "hash": "QmX8fjWTb1hynPkHrssWc6YGwphXmfnikLa6VxQzQjMRKJ", + "hash": "QmXrweWQgN62WvMcP2EWDpD2qYXiFqenqdaLjUzX7yBqLv", "name": "go-conn-security-multistream", - "version": "0.1.20" + "version": "0.1.21" }, { "author": "Stebalien", - "hash": "QmXCnmY9nBCDQBhzxrn5ZKqSYYHiFzc5btFt32UpgECHnS", + "hash": "QmVovmja8iXHy1JouPULFdKExUrGutwzgptJZEAFG9rL1t", "name": "go-conn-security", - "version": "0.1.21" + "version": "0.1.22" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "Qmc9KUyhx1adPnHX2TBjEWvKej2Gg2kvisAFoQ74UiWYhd", + "hash": "Qmc7NvDoQaeCaGMuwXz45whL3J21o4Wt7pysztcDZ1VDmn", "name": "go-libp2p-transport-upgrader", - "version": "0.1.21" + "version": "0.1.22" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -229,20 +229,20 @@ }, { "author": "vyzo", - "hash": "QmYJtCabf3prS3HKQUGgqDLVxvbT9iDx5mfeVfhtCcJxxE", + "hash": "QmemYsfqwAbyvqwFiApk1GfLKhDkMm8ZQK6fCvzDbaRNyX", "name": "go-libp2p-discovery", - "version": "1.0.4" + "version": "1.0.5" }, { "author": "vyzo", - "hash": "QmQXeTmQcPFnf3ZAvik2qgKxWNoZ27aKGcW8hUBwBrTxT1", + "hash": "QmXmZtMdQokSodDNvPdhDyaVRAjgybvR8dQtuMsNoWv4Lq", "name": "go-libp2p-autonat", - "version": "1.0.4" + "version": "1.0.5" }, { - "hash": "QmRASJXJUFygM5qU4YrH7k7jD6S4Hg8nJmgqJ4bYJvLatd", + "hash": "QmTiRqrF5zkdZyrdsL5qndG1UbeWi8k8N2pYxCtXWrahR2", "name": "go-libp2p-routing", - "version": "2.7.6" + "version": "2.7.7" } ], "gxVersion": "0.4.0", @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.28" + "version": "6.0.29" } From 9258075d6ca01fc86c6aad085163889e2f325f18 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Tue, 18 Dec 2018 18:21:48 +1100 Subject: [PATCH 0880/3965] Don't log raw binary Causes issues particularly when logging to the terminal. License: MIT Signed-off-by: Matt Joiner --- p2p/protocol/internal/circuitv1-deprecated/listen.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index 0b1885be03..35661cb8c6 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -33,7 +33,7 @@ func (l *RelayListener) Accept() (manet.Conn, error) { } // TODO: Pretty print. - log.Infof("accepted relay connection: %s", c) + log.Infof("accepted relay connection: %q", c) return c, nil case <-l.ctx.Done(): From 168488bfc3127ac2e263c55dd8962b4dab4749b0 Mon Sep 17 00:00:00 2001 From: Anton Petrov Date: Tue, 18 Dec 2018 22:58:25 +0000 Subject: [PATCH 0881/3965] readme: remove duplicate repo entries in README and package-list.json License: MIT Signed-off-by: Anton Petrov --- README.md | 2 -- package-list.json | 2 -- 2 files changed, 4 deletions(-) diff --git a/README.md b/README.md index 6d02c0764a..ebeb0e50fc 100644 --- a/README.md +++ b/README.md @@ -179,9 +179,7 @@ List of packages currently in existence for libp2p: | [`go-msgio`](//github.com/libp2p/go-msgio) | [![Travis CI](https://travis-ci.org/libp2p/go-msgio.svg?branch=master)](https://travis-ci.org/libp2p/go-msgio) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-msgio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-msgio) | length prefixed data channel | | [`go-addr-util`](//github.com/libp2p/go-addr-util) | [![Travis CI](https://travis-ci.org/libp2p/go-addr-util.svg?branch=master)](https://travis-ci.org/libp2p/go-addr-util) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-addr-util/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-addr-util) | address utilities for libp2p swarm | | [`go-buffer-pool`](//github.com/libp2p/go-buffer-pool) | [![Travis CI](https://travis-ci.org/libp2p/go-buffer-pool.svg?branch=master)](https://travis-ci.org/libp2p/go-buffer-pool) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-buffer-pool/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-buffer-pool) | a variable size buffer pool for go | -| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | | [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-routing-helpers/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-routing-helpers/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | routing helpers | -| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | a library to perform filtering of multiaddrs. | | [`go-reuseport`](//github.com/libp2p/go-reuseport) | [![Travis CI](https://travis-ci.org/libp2p/go-reuseport.svg?branch=master)](https://travis-ci.org/libp2p/go-reuseport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-reuseport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-reuseport) | enables reuse of addresses | | [`go-sockaddr`](//github.com/libp2p/go-sockaddr) | [![Travis CI](https://travis-ci.org/libp2p/go-sockaddr.svg?branch=master)](https://travis-ci.org/libp2p/go-sockaddr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-sockaddr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-sockaddr) | utils for sockaddr conversions | | [`go-flow-metrics`](//github.com/libp2p/go-flow-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-flow-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-flow-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-flow-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-flow-metrics) | metrics library | diff --git a/package-list.json b/package-list.json index 7e1f62c3c7..142eda0a23 100644 --- a/package-list.json +++ b/package-list.json @@ -84,9 +84,7 @@ ["libp2p/go-msgio", "go-msgio", "length prefixed data channel"], ["libp2p/go-addr-util", "go-addr-util", "address utilities for libp2p swarm"], ["libp2p/go-buffer-pool", "go-buffer-pool", "a variable size buffer pool for go"], - ["libp2p/go-libp2p-loggables", "go-libp2p-loggables", "logging helpers"], ["libp2p/go-libp2p-routing-helpers", "go-libp2p-routing-helpers", "routing helpers"], - ["libp2p/go-maddr-filter", "go-maddr-filter", "a library to perform filtering of multiaddrs."], ["libp2p/go-reuseport", "go-reuseport", "enables reuse of addresses"], ["libp2p/go-sockaddr", "go-sockaddr", "utils for sockaddr conversions"], ["libp2p/go-flow-metrics", "go-flow-metrics", "metrics library"], From 758b51660345cbba640ddccbd78b705d2131ec46 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 18 Dec 2018 21:15:32 -0800 Subject: [PATCH 0882/3965] mock: export ratelimiter as RateLimiter Technically a breaking change because I changed the constructor casing but nobody's using this outside of libp2p (yet), mostly because the type hasn't been exported. --- p2p/net/mock/mock_link.go | 4 ++-- p2p/net/mock/mock_test.go | 2 +- p2p/net/mock/ratelimiter.go | 16 ++++++++-------- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/p2p/net/mock/mock_link.go b/p2p/net/mock/mock_link.go index 9c6049ac88..d123666620 100644 --- a/p2p/net/mock/mock_link.go +++ b/p2p/net/mock/mock_link.go @@ -16,7 +16,7 @@ type link struct { mock *mocknet nets []*peernet opts LinkOptions - ratelimiter *ratelimiter + ratelimiter *RateLimiter // this could have addresses on both sides. sync.RWMutex @@ -25,7 +25,7 @@ type link struct { func newLink(mn *mocknet, opts LinkOptions) *link { l := &link{mock: mn, opts: opts, - ratelimiter: NewRatelimiter(opts.Bandwidth)} + ratelimiter: NewRateLimiter(opts.Bandwidth)} return l } diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index d6d3c72680..ceeea38fd5 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -487,7 +487,7 @@ func TestAdding(t *testing.T) { } func TestRateLimiting(t *testing.T) { - rl := NewRatelimiter(10) + rl := NewRateLimiter(10) if !within(rl.Limit(10), time.Duration(float32(time.Second)), time.Millisecond/10) { t.Fail() diff --git a/p2p/net/mock/ratelimiter.go b/p2p/net/mock/ratelimiter.go index 0b21c9a4d9..d4e5797b65 100644 --- a/p2p/net/mock/ratelimiter.go +++ b/p2p/net/mock/ratelimiter.go @@ -5,9 +5,9 @@ import ( "time" ) -// A ratelimiter is used by a link to determine how long to wait before sending +// A RateLimiter is used by a link to determine how long to wait before sending // data given a bandwidth cap. -type ratelimiter struct { +type RateLimiter struct { lock sync.Mutex bandwidth float64 // bytes per nanosecond allowance float64 // in bytes @@ -17,11 +17,11 @@ type ratelimiter struct { duration time.Duration // total delay introduced due to rate limiting } -// Creates a new ratelimiter with bandwidth (in bytes/sec) -func NewRatelimiter(bandwidth float64) *ratelimiter { +// Creates a new RateLimiter with bandwidth (in bytes/sec) +func NewRateLimiter(bandwidth float64) *RateLimiter { // convert bandwidth to bytes per nanosecond b := bandwidth / float64(time.Second) - return &ratelimiter{ + return &RateLimiter{ bandwidth: b, allowance: 0, maxAllowance: bandwidth, @@ -29,8 +29,8 @@ func NewRatelimiter(bandwidth float64) *ratelimiter { } } -// Changes bandwidth of a ratelimiter and resets its allowance -func (r *ratelimiter) UpdateBandwidth(bandwidth float64) { +// Changes bandwidth of a RateLimiter and resets its allowance +func (r *RateLimiter) UpdateBandwidth(bandwidth float64) { r.lock.Lock() defer r.lock.Unlock() // Convert bandwidth from bytes/second to bytes/nanosecond @@ -43,7 +43,7 @@ func (r *ratelimiter) UpdateBandwidth(bandwidth float64) { } // Returns how long to wait before sending data with length 'dataSize' bytes -func (r *ratelimiter) Limit(dataSize int) time.Duration { +func (r *RateLimiter) Limit(dataSize int) time.Duration { r.lock.Lock() defer r.lock.Unlock() // update time From 768598b353be264ce8420be6353d753a654967ee Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 19 Dec 2018 10:10:58 -0800 Subject: [PATCH 0883/3965] gx publish 6.0.30 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 9edfa6bcdf..5ffe1f8e09 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.29: QmdJdFQc5U3RAKgJQGmWR7SSM7TLuER5FWz5Wq6Tzs2CnS +6.0.30: QmYxivS34F2M2n44WQQnRHGAKS8aoRUxwGpi9wk4Cdn4Jf diff --git a/package.json b/package.json index 58d1d6d905..c58d67e70c 100644 --- a/package.json +++ b/package.json @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.29" + "version": "6.0.30" } From 179ba24cab82df352236f558a7f783dc959dbc47 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 20 Dec 2018 18:58:13 +0200 Subject: [PATCH 0884/3965] track autonat peer addresses --- p2p/host/autonat/autonat.go | 28 +++++++++++++++------------- p2p/host/autonat/autonat_test.go | 2 +- p2p/host/autonat/notify.go | 2 +- 3 files changed, 17 insertions(+), 15 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 0aa987ca63..e0418c45f8 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -10,6 +10,7 @@ import ( host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" + pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) @@ -51,7 +52,7 @@ type AmbientAutoNAT struct { getAddrs GetAddrs mx sync.Mutex - peers map[peer.ID]struct{} + peers map[peer.ID][]ma.Multiaddr status NATStatus addr ma.Multiaddr // Reflects the confidence on of the NATStatus being private, as a single @@ -73,7 +74,7 @@ func NewAutoNAT(ctx context.Context, h host.Host, getAddrs GetAddrs) AutoNAT { ctx: ctx, host: h, getAddrs: getAddrs, - peers: make(map[peer.ID]struct{}), + peers: make(map[peer.ID][]ma.Multiaddr), status: NATStatusUnknown, } @@ -134,14 +135,15 @@ func (as *AmbientAutoNAT) autodetect() { cli := NewAutoNATClient(as.host, as.getAddrs) failures := 0 - for _, p := range peers { + for _, pi := range peers { ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) - a, err := cli.DialBack(ctx, p) + as.host.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) + a, err := cli.DialBack(ctx, pi.ID) cancel() switch { case err == nil: - log.Debugf("NAT status is public; address through %s: %s", p.Pretty(), a.String()) + log.Debugf("NAT status is public; address through %s: %s", pi.ID.Pretty(), a.String()) as.mx.Lock() as.addr = a as.status = NATStatusPublic @@ -150,7 +152,7 @@ func (as *AmbientAutoNAT) autodetect() { return case IsDialError(err): - log.Debugf("dial error through %s: %s", p.Pretty(), err.Error()) + log.Debugf("dial error through %s: %s", pi.ID.Pretty(), err.Error()) failures++ if failures >= 3 || as.confidence >= 3 { // 3 times is enemy action log.Debugf("NAT status is private") @@ -162,7 +164,7 @@ func (as *AmbientAutoNAT) autodetect() { } default: - log.Debugf("Error dialing through %s: %s", p.Pretty(), err.Error()) + log.Debugf("Error dialing through %s: %s", pi.ID.Pretty(), err.Error()) } } @@ -179,7 +181,7 @@ func (as *AmbientAutoNAT) autodetect() { as.mx.Unlock() } -func (as *AmbientAutoNAT) getPeers() []peer.ID { +func (as *AmbientAutoNAT) getPeers() []pstore.PeerInfo { as.mx.Lock() defer as.mx.Unlock() @@ -187,13 +189,13 @@ func (as *AmbientAutoNAT) getPeers() []peer.ID { return nil } - var connected, others []peer.ID + var connected, others []pstore.PeerInfo - for p := range as.peers { + for p, addrs := range as.peers { if as.host.Network().Connectedness(p) == inet.Connected { - connected = append(connected, p) + connected = append(connected, pstore.PeerInfo{ID: p, Addrs: addrs}) } else { - others = append(others, p) + others = append(others, pstore.PeerInfo{ID: p, Addrs: addrs}) } } @@ -207,7 +209,7 @@ func (as *AmbientAutoNAT) getPeers() []peer.ID { } } -func shufflePeers(peers []peer.ID) { +func shufflePeers(peers []pstore.PeerInfo) { for i := range peers { j := rand.Intn(i + 1) peers[i], peers[j] = peers[j], peers[i] diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index aa8d879135..5436619846 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -73,7 +73,7 @@ func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Mes func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, AutoNAT) { h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) a := NewAutoNAT(ctx, h, nil) - a.(*AmbientAutoNAT).peers[ash.ID()] = struct{}{} + a.(*AmbientAutoNAT).peers[ash.ID()] = ash.Addrs() return h, a } diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index ebe709baf6..69f5a6457f 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -32,7 +32,7 @@ func (as *AmbientAutoNAT) Connected(net inet.Network, c inet.Conn) { if len(protos) > 0 { log.Infof("Discovered AutoNAT peer %s", p.Pretty()) as.mx.Lock() - as.peers[p] = struct{}{} + as.peers[p] = as.host.Peerstore().Addrs(p) as.mx.Unlock() } }() From c5f50f9c2d0f1dff76069bdf844d45b44fcd9f67 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 21 Dec 2018 11:20:09 +0200 Subject: [PATCH 0885/3965] don't advertise the raw public address in autorelay --- p2p/host/relay/autorelay.go | 30 +++--------------------------- 1 file changed, 3 insertions(+), 27 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index e97ce88321..0a40c0c250 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -203,36 +203,12 @@ func (h *AutoRelayHost) doUpdateAddrs() { addrs := h.baseAddrs() raddrs := make([]ma.Multiaddr, 0, len(addrs)+len(h.relays)) - // remove our public addresses from the list and replace them by just the public IP + // remove our public addresses from the list for _, addr := range addrs { if manet.IsPublicAddr(addr) { - ip, err := addr.ValueForProtocol(ma.P_IP4) - if err == nil { - pub, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s", ip)) - if err != nil { - panic(err) - } - - if !containsAddr(raddrs, pub) { - raddrs = append(raddrs, pub) - } - continue - } - - ip, err = addr.ValueForProtocol(ma.P_IP6) - if err == nil { - pub, err := ma.NewMultiaddr(fmt.Sprintf("/ip6/%s", ip)) - if err != nil { - panic(err) - } - if !containsAddr(raddrs, pub) { - raddrs = append(raddrs, pub) - } - continue - } - } else { - raddrs = append(raddrs, addr) + continue } + raddrs = append(raddrs, addr) } // add relay specific addrs to the list From 440615fe4e6edac9b01968c36365e9c69f41ebe6 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Mon, 31 Dec 2018 14:17:30 +1100 Subject: [PATCH 0886/3965] Use new go-reuseport.Control --- p2p/net/reuseport/reuseport.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/p2p/net/reuseport/reuseport.go b/p2p/net/reuseport/reuseport.go index 47ceac2278..35e8a85c21 100644 --- a/p2p/net/reuseport/reuseport.go +++ b/p2p/net/reuseport/reuseport.go @@ -44,10 +44,9 @@ func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) ( return fallbackDialer.DialContext(ctx, network, raddr) } - d := reuseport.Dialer{ - D: net.Dialer{ - LocalAddr: laddr, - }, + d := net.Dialer{ + LocalAddr: laddr, + Control: reuseport.Control, } con, err := d.DialContext(ctx, network, raddr) From 8b89db36a5082fb37aca1cbaba815c0eace85a79 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Wed, 2 Jan 2019 17:43:57 +1100 Subject: [PATCH 0887/3965] Add SetLinger(0) in reuseDial No longer implicitly provided by go-reuseport's Dialer as it was removed. --- p2p/net/reuseport/reuseport.go | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/p2p/net/reuseport/reuseport.go b/p2p/net/reuseport/reuseport.go index 35e8a85c21..dd271d11a0 100644 --- a/p2p/net/reuseport/reuseport.go +++ b/p2p/net/reuseport/reuseport.go @@ -38,8 +38,9 @@ func reuseErrShouldRetry(err error) bool { } } -// Dials using reusport and then redials normally if that fails. -func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) (net.Conn, error) { +// Dials using reuseport and then redials normally if that fails. +// TODO(anacrolix): This shouldn't fail anymore: Remove fallback. +func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) (con net.Conn, err error) { if laddr == nil { return fallbackDialer.DialContext(ctx, network, raddr) } @@ -48,8 +49,17 @@ func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) ( LocalAddr: laddr, Control: reuseport.Control, } + defer func() { + if err != nil { + return + } + // This is transplanted from go-reuseport, which once set no linger on + // dialing and may be a requirement for desired behaviour in this + // package. + con.(*net.TCPConn).SetLinger(0) + }() - con, err := d.DialContext(ctx, network, raddr) + con, err = d.DialContext(ctx, network, raddr) if err == nil { return con, nil } From 8ceb85678fac58ed84786086f9d7182f0ae652cc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 8 Jan 2019 16:05:55 -0800 Subject: [PATCH 0888/3965] annotate errors --- p2p/net/upgrader/upgrader.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index bfbfa7d017..1c901b27dc 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -74,7 +74,7 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma pconn, err := u.Protector.Protect(conn) if err != nil { conn.Close() - return nil, err + return nil, fmt.Errorf("failed to setup private network protector: %s", err) } conn = pconn } else if pnet.ForcePrivateNetwork { @@ -85,12 +85,12 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma sconn, err := u.setupSecurity(ctx, conn, p) if err != nil { conn.Close() - return nil, err + return nil, fmt.Errorf("failed to negotiate security protocol: %s", err) } smconn, err := u.setupMuxer(ctx, sconn, p) if err != nil { conn.Close() - return nil, err + return nil, fmt.Errorf("failed to negotiate security stream multiplexer: %s", err) } return &transportConn{ Conn: smconn, From 072f460a716c375254bb11bbedabd3a3c9ea53d9 Mon Sep 17 00:00:00 2001 From: Anton Petrov Date: Thu, 10 Jan 2019 07:51:46 +0000 Subject: [PATCH 0889/3965] docs: update broken links in NEWS.md --- NEWS.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/NEWS.md b/NEWS.md index 7b7e27af2d..ff497d72bc 100644 --- a/NEWS.md +++ b/NEWS.md @@ -289,16 +289,16 @@ new security framework is now as simple as: [go-libp2p-conn]: https://github.com/libp2p/go-libp2p-conn [go-libp2p-interface-conn]: https://github.com/libp2p/go-libp2p-interface-conn [go-libp2p-net]: https://github.com/libp2p/go-libp2p-net -[go-libp2p-netutil]: https://github.com/libp2p/go-libp2p/go-libp2p-netutil -[go-libp2p-swarm]: https://github.com/libp2p/go-libp2p/go-libp2p-swarm -[go-libp2p-swarm/testing]: https://github.com/libp2p/go-libp2p/go-libp2p-swarm/testing +[go-libp2p-netutil]: https://github.com/libp2p/go-libp2p-netutil +[go-libp2p-swarm]: https://github.com/libp2p/go-libp2p-swarm +[go-libp2p-swarm/testing]: https://github.com/libp2p/go-libp2p-swarm/tree/master/testing [go-libp2p-transport]: https://github.com/libp2p/go-libp2p-transport -[go-libp2p-transport/test]: https://github.com/libp2p/go-libp2p-transport/test +[go-libp2p-transport/test]: https://github.com/libp2p/go-libp2p-transport/tree/master/test [go-libp2p-transport-upgrader]: https://github.com/libp2p/go-libp2p-transport-upgrader [go-peerstream]: https://github.com/libp2p/go-peerstream [go-tcp-transport]: https://github.com/libp2p/go-tcp-transport [go-ws-transport]: https://github.com/libp2p/go-ws-transport -[example:echo]: https://github.com/libp2p/go-libp2p/tree/master/examples/echo +[example:echo]: https://github.com/libp2p/go-libp2p-examples/tree/master/echo [doc:go-libp2p-transport]: https://godoc.org/github.com/libp2p/go-libp2p-transport From 955b8056cb4ace524eac46ca500a07cf64a43445 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 11 Jan 2019 13:51:35 +0700 Subject: [PATCH 0890/3965] add support for ECDSA keys --- p2p/security/tls/crypto.go | 19 ++++++++- p2p/security/tls/libp2p_tls_suite_test.go | 5 +++ p2p/security/tls/transport_test.go | 49 ++++++++++++++++++----- 3 files changed, 61 insertions(+), 12 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 1a6812a18e..583fe39c14 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -5,6 +5,7 @@ import ( "crypto/tls" "crypto/x509" "errors" + "fmt" "math/big" "time" @@ -93,7 +94,14 @@ func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { if err != nil { return nil, err } - return ic.UnmarshalRsaPublicKey(remotePubKey) + switch chain[0].PublicKeyAlgorithm { + case x509.RSA: + return ic.UnmarshalRsaPublicKey(remotePubKey) + case x509.ECDSA: + return ic.UnmarshalECDSAPublicKey(remotePubKey) + default: + return nil, fmt.Errorf("unexpected public key algorithm: %d", chain[0].PublicKeyAlgorithm) + } } func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { @@ -124,7 +132,14 @@ func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { } publicKey = &k.PublicKey privateKey = k - // TODO: add support for ECDSA + case pb.KeyType_ECDSA: + k, err := x509.ParseECPrivateKey(pbmes.GetData()) + if err != nil { + return nil, nil, err + } + publicKey = &k.PublicKey + privateKey = k + // TODO: add support for Ed25519 default: return nil, nil, errors.New("unsupported key type for TLS") } diff --git a/p2p/security/tls/libp2p_tls_suite_test.go b/p2p/security/tls/libp2p_tls_suite_test.go index 3303ed3030..e0e6785862 100644 --- a/p2p/security/tls/libp2p_tls_suite_test.go +++ b/p2p/security/tls/libp2p_tls_suite_test.go @@ -1,6 +1,7 @@ package libp2ptls import ( + mrand "math/rand" "testing" . "github.com/onsi/ginkgo" @@ -11,3 +12,7 @@ func TestLibp2pTLS(t *testing.T) { RegisterFailHandler(Fail) RunSpecs(t, "libp2p TLS Suite") } + +var _ = BeforeSuite(func() { + mrand.Seed(GinkgoRandomSeed()) +}) diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index bc4d5b3813..d06326d90f 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -2,9 +2,12 @@ package libp2ptls import ( "context" + "crypto/ecdsa" + "crypto/elliptic" "crypto/rand" "crypto/rsa" - "crypto/x509" + "fmt" + mrand "math/rand" "net" cs "github.com/libp2p/go-conn-security" @@ -21,10 +24,18 @@ var _ = Describe("Transport", func() { ) createPeer := func() (peer.ID, ic.PrivKey) { - key, err := rsa.GenerateKey(rand.Reader, 1024) - Expect(err).ToNot(HaveOccurred()) - priv, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(key)) - Expect(err).ToNot(HaveOccurred()) + var priv ic.PrivKey + if mrand.Int()%2 == 0 { + fmt.Fprintln(GinkgoWriter, " using an ECDSA key") + var err error + priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) + Expect(err).ToNot(HaveOccurred()) + } else { + fmt.Fprintln(GinkgoWriter, " using an RSA key") + var err error + priv, _, err = ic.GenerateRSAKeyPair(1024, rand.Reader) + Expect(err).ToNot(HaveOccurred()) + } id, err := peer.IDFromPrivateKey(priv) Expect(err).ToNot(HaveOccurred()) return id, priv @@ -48,13 +59,24 @@ var _ = Describe("Transport", func() { // modify the cert chain such that verificiation will fail invalidateCertChain := func(identity *Identity) { - key, err := rsa.GenerateKey(rand.Reader, 1024) - Expect(err).ToNot(HaveOccurred()) - identity.Config.Certificates[0].PrivateKey = key + switch identity.Config.Certificates[0].PrivateKey.(type) { + case *rsa.PrivateKey: + key, err := rsa.GenerateKey(rand.Reader, 1024) + Expect(err).ToNot(HaveOccurred()) + identity.Config.Certificates[0].PrivateKey = key + case *ecdsa.PrivateKey: + key, err := ecdsa.GenerateKey(elliptic.P224(), rand.Reader) + Expect(err).ToNot(HaveOccurred()) + identity.Config.Certificates[0].PrivateKey = key + default: + Fail("unexpected private key type") + } } BeforeEach(func() { + fmt.Fprintf(GinkgoWriter, "Initializing a server") serverID, serverKey = createPeer() + fmt.Fprintf(GinkgoWriter, "Initializing a client") clientID, clientKey = createPeer() }) @@ -135,6 +157,7 @@ var _ = Describe("Transport", func() { }) It("fails if the peer ID doesn't match", func() { + fmt.Fprintf(GinkgoWriter, "Creating another peer") thirdPartyID, _ := createPeer() serverTransport, err := New(serverKey) @@ -172,7 +195,10 @@ var _ = Describe("Transport", func() { defer GinkgoRecover() _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("crypto/rsa: verification error")) + Expect(err.Error()).To(Or( + ContainSubstring("crypto/rsa: verification error"), + ContainSubstring("ECDSA verification failure"), + )) close(done) }() @@ -202,7 +228,10 @@ var _ = Describe("Transport", func() { _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("crypto/rsa: verification error")) + Expect(err.Error()).To(Or( + ContainSubstring("crypto/rsa: verification error"), + ContainSubstring("ECDSA verification failure"), + )) Eventually(done).Should(BeClosed()) }) }) From 0e9ce4e396eb92387880474424dadc305a3f4196 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 23 Jan 2019 13:13:11 +0000 Subject: [PATCH 0891/3965] add unit test to verify AddAddr doesn't shorten TTL. --- p2p/host/peerstore/test/addr_book_suite.go | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index 7cb4cf8deb..cbea8b8fab 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -107,6 +107,21 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { time.Sleep(1200 * time.Millisecond) testHas(t, addrs[2:], ab.Addrs(id)) }) + + t.Run("adding an existing address with an earlier expiration is noop", func(t *testing.T) { + id := generatePeerIds(1)[0] + addrs := generateAddrs(3) + + ab.AddAddrs(id, addrs, time.Hour) + + // same address as before but with a lower TTL + ab.AddAddrs(id, addrs[2:], time.Second) + + // after the initial TTL has expired, check that all three addresses are still present (i.e. the TTL on + // the modified one was not shortened). + time.Sleep(2100 * time.Millisecond) + testHas(t, addrs, ab.Addrs(id)) + }) } } From 70611007e5cbe410794edde1f65c2e04928f6e9a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 23 Jan 2019 07:43:12 -0800 Subject: [PATCH 0892/3965] avoid spawning goroutines for canceled dials This may speed up working through our dial queue a bit (bit probably isn't the main issue). --- p2p/net/swarm/limiter.go | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 02aed50a2e..11c20bc100 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -77,17 +77,26 @@ func newDialLimiterWithParams(df dialfunc, fdLimit, perPeerLimit int) *dialLimit func (dl *dialLimiter) freeFDToken() { dl.fdConsuming-- - if len(dl.waitingOnFd) > 0 { + for len(dl.waitingOnFd) > 0 { next := dl.waitingOnFd[0] dl.waitingOnFd[0] = nil // clear out memory dl.waitingOnFd = dl.waitingOnFd[1:] + if len(dl.waitingOnFd) == 0 { - dl.waitingOnFd = nil // clear out memory + // clear out memory. + dl.waitingOnFd = nil + } + + // Skip over canceled dials instead of queuing up a goroutine. + if next.cancelled() { + dl.freePeerToken(next) + continue } dl.fdConsuming++ // we already have activePerPeer token at this point so we can just dial go dl.executeDial(next) + return } } @@ -99,18 +108,25 @@ func (dl *dialLimiter) freePeerToken(dj *dialJob) { } waitlist := dl.waitingOnPeerLimit[dj.peer] - if len(waitlist) > 0 { + for len(waitlist) > 0 { next := waitlist[0] - if len(waitlist) == 1 { + waitlist[0] = nil // clear out memory + waitlist = waitlist[1:] + + if len(waitlist) == 0 { delete(dl.waitingOnPeerLimit, next.peer) } else { - waitlist[0] = nil // clear out memory - dl.waitingOnPeerLimit[next.peer] = waitlist[1:] + dl.waitingOnPeerLimit[next.peer] = waitlist + } + + if next.cancelled() { + continue } dl.activePerPeer[next.peer]++ // just kidding, we still want this token dl.addCheckFdLimit(next) + return } } From 8d093f29d2a00a300f4a6ca02940b8a3e8d28277 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 23 Jan 2019 09:06:37 -0800 Subject: [PATCH 0893/3965] dialer: handle dial cancel and/or completion before trying new addresses --- p2p/net/swarm/swarm_dial.go | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 6030ca79d7..37f1234cc4 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -358,14 +358,34 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. defer s.limiter.clearAllPeerDials(p) var active int - for { + for remoteAddrs != nil || active > 0 { + // Check for context cancellations and/or responses first. + select { + case <-ctx.Done(): + if exitErr == defaultDialFail { + exitErr = ctx.Err() + } + return nil, exitErr + case resp := <-respch: + active-- + if resp.Err != nil { + log.Infof("got error on dial to %s: %s", resp.Addr, resp.Err) + // Errors are normal, lots of dials will fail + exitErr = resp.Err + } else if resp.Conn != nil { + return resp.Conn, nil + } + + // We got a result, try again from the top. + continue + default: + } + + // Now, attempt to dial. select { case addr, ok := <-remoteAddrs: if !ok { remoteAddrs = nil - if active == 0 { - return nil, exitErr - } continue } @@ -382,15 +402,12 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. log.Infof("got error on dial to %s: %s", resp.Addr, resp.Err) // Errors are normal, lots of dials will fail exitErr = resp.Err - - if remoteAddrs == nil && active == 0 { - return nil, exitErr - } } else if resp.Conn != nil { return resp.Conn, nil } } } + return nil, exitErr } // limitedDial will start a dial to the given peer when From addf6f1a11b99e26b50dd46043c87d4a5a40ce88 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 29 Jan 2019 16:06:09 +0000 Subject: [PATCH 0894/3965] enhance debug logging in limiter. --- p2p/net/swarm/limiter.go | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 11c20bc100..2def638c3d 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -75,6 +75,7 @@ func newDialLimiterWithParams(df dialfunc, fdLimit, perPeerLimit int) *dialLimit // freeFDToken frees FD token and if there are any schedules another waiting dialJob // in it's place func (dl *dialLimiter) freeFDToken() { + log.Debugf("[limiter] freeing FD token; waiting: %d; consuming: %d", len(dl.waitingOnFd), dl.fdConsuming) dl.fdConsuming-- for len(dl.waitingOnFd) > 0 { @@ -101,6 +102,8 @@ func (dl *dialLimiter) freeFDToken() { } func (dl *dialLimiter) freePeerToken(dj *dialJob) { + log.Debugf("[limiter] freeing peer token; peer %s; addr: %s; active for peer: %d; waiting on peer limit: %d", + dj.peer, dj.addr, dl.activePerPeer[dj.peer], len(dl.waitingOnPeerLimit[dj.peer])) // release tokens in reverse order than we take them dl.activePerPeer[dj.peer]-- if dl.activePerPeer[dj.peer] == 0 { @@ -144,19 +147,28 @@ func (dl *dialLimiter) finishedDial(dj *dialJob) { func (dl *dialLimiter) addCheckFdLimit(dj *dialJob) { if addrutil.IsFDCostlyTransport(dj.addr) { if dl.fdConsuming >= dl.fdLimit { + log.Debugf("[limiter] blocked dial waiting on FD token; peer: %s; addr: %s; consuming: %d; "+ + "limit: %d; waiting: %d", dj.peer, dj.addr, dl.fdConsuming, dl.fdLimit, len(dl.waitingOnFd)) dl.waitingOnFd = append(dl.waitingOnFd, dj) return } + log.Debugf("[limiter] taking FD token: peer: %s; addr: %s; prev consuming: %d", + dj.peer, dj.addr, dl.fdConsuming) // take token dl.fdConsuming++ } + log.Debugf("[limiter] executing dial; peer: %s; addr: %s; FD consuming: %d; waiting: %d", + dj.peer, dj.addr, dl.fdConsuming, len(dl.waitingOnFd)) go dl.executeDial(dj) } func (dl *dialLimiter) addCheckPeerLimit(dj *dialJob) { if dl.activePerPeer[dj.peer] >= dl.perPeerLimit { + log.Debugf("[limiter] blocked dial waiting on peer limit; peer: %s; addr: %s; active: %d; "+ + "peer limit: %d; waiting: %d", dj.peer, dj.addr, dl.activePerPeer[dj.peer], dl.perPeerLimit, + len(dl.waitingOnPeerLimit[dj.peer])) wlist := dl.waitingOnPeerLimit[dj.peer] dl.waitingOnPeerLimit[dj.peer] = append(wlist, dj) return @@ -173,6 +185,7 @@ func (dl *dialLimiter) AddDialJob(dj *dialJob) { dl.lk.Lock() defer dl.lk.Unlock() + log.Debugf("[limiter] adding a dial job through limiter: %v", dj.addr) dl.addCheckPeerLimit(dj) } @@ -180,6 +193,7 @@ func (dl *dialLimiter) clearAllPeerDials(p peer.ID) { dl.lk.Lock() defer dl.lk.Unlock() delete(dl.waitingOnPeerLimit, p) + log.Debugf("[limiter] clearing all peer dials: %v", p) // NB: the waitingOnFd list doesn't need to be cleaned out here, we will // remove them as we encounter them because they are 'cancelled' at this // point @@ -189,6 +203,8 @@ func (dl *dialLimiter) clearAllPeerDials(p peer.ID) { // channel when finished. Once the response is sent it also releases all tokens // it held during the dial. func (dl *dialLimiter) executeDial(j *dialJob) { + log.Debugf("[limiter] executing dial (dialfunc); peer: %s; addr: %s; FD consuming: %d; waiting: %d", + j.peer, j.addr, dl.fdConsuming, len(dl.waitingOnFd)) defer dl.finishedDial(j) if j.cancelled() { return From 1c5f2407cb89f2bc5003a7e5cb2c224eea94d20b Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Mon, 4 Feb 2019 15:18:27 +1100 Subject: [PATCH 0895/3965] Add CircleCI config --- p2p/net/swarm/.circleci/config.yml | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 p2p/net/swarm/.circleci/config.yml diff --git a/p2p/net/swarm/.circleci/config.yml b/p2p/net/swarm/.circleci/config.yml new file mode 100644 index 0000000000..68d7e72668 --- /dev/null +++ b/p2p/net/swarm/.circleci/config.yml @@ -0,0 +1,10 @@ +version: 2 +jobs: + build: + docker: + - image: circleci/golang:1.11 + environment: + GO111MODULE: "on" + steps: + - checkout + - run: go test -v -race -bench . ./... From 27f08059c5ebf66c56e4c871a33e2d2cd632d4e6 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Mon, 4 Feb 2019 15:20:35 +1100 Subject: [PATCH 0896/3965] Fix logging race --- p2p/net/swarm/limiter.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index 2def638c3d..ce9e2448ca 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -203,8 +203,10 @@ func (dl *dialLimiter) clearAllPeerDials(p peer.ID) { // channel when finished. Once the response is sent it also releases all tokens // it held during the dial. func (dl *dialLimiter) executeDial(j *dialJob) { + dl.lk.Lock() log.Debugf("[limiter] executing dial (dialfunc); peer: %s; addr: %s; FD consuming: %d; waiting: %d", j.peer, j.addr, dl.fdConsuming, len(dl.waitingOnFd)) + dl.lk.Unlock() defer dl.finishedDial(j) if j.cancelled() { return From 9d153e8ea7414170b086b49c972abf0f92da153e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 4 Feb 2019 17:47:51 +0000 Subject: [PATCH 0897/3965] remove superfluous log statement. --- p2p/net/swarm/limiter.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index ce9e2448ca..ef78f0eeb8 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -203,10 +203,6 @@ func (dl *dialLimiter) clearAllPeerDials(p peer.ID) { // channel when finished. Once the response is sent it also releases all tokens // it held during the dial. func (dl *dialLimiter) executeDial(j *dialJob) { - dl.lk.Lock() - log.Debugf("[limiter] executing dial (dialfunc); peer: %s; addr: %s; FD consuming: %d; waiting: %d", - j.peer, j.addr, dl.fdConsuming, len(dl.waitingOnFd)) - dl.lk.Unlock() defer dl.finishedDial(j) if j.cancelled() { return From 96a640d5913618166a8c665729c341806373a17a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 31 Jan 2019 13:52:54 +0000 Subject: [PATCH 0898/3965] make FD limits configurable by env prop. --- p2p/net/swarm/limiter.go | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index ef78f0eeb8..fa9582bbe6 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -2,6 +2,8 @@ package swarm import ( "context" + "os" + "strconv" "sync" "time" @@ -59,7 +61,13 @@ type dialLimiter struct { type dialfunc func(context.Context, peer.ID, ma.Multiaddr) (transport.Conn, error) func newDialLimiter(df dialfunc) *dialLimiter { - return newDialLimiterWithParams(df, ConcurrentFdDials, DefaultPerPeerRateLimit) + fd := ConcurrentFdDials + if env := os.Getenv("LIBP2P_SWARM_FD_LIMIT"); env != "" { + if n, err := strconv.ParseInt(env, 10, 32); err == nil { + fd = int(n) + } + } + return newDialLimiterWithParams(df, fd, DefaultPerPeerRateLimit) } func newDialLimiterWithParams(df dialfunc, fdLimit, perPeerLimit int) *dialLimiter { From e7c54d6c06cd902df60e370f9664e2fb7414967a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 5 Feb 2019 15:07:30 +0000 Subject: [PATCH 0899/3965] pstore ds: enforce order in gc queries. --- p2p/host/peerstore/pstoreds/addr_book_gc.go | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index a0212fa9bb..2c45b86468 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -23,8 +23,16 @@ var ( // in GC routines, how many operations do we place in a batch before it's committed. gcOpsPerBatch = 20 // queries - purgeQuery = query.Query{Prefix: gcLookaheadBase.String(), KeysOnly: true} - populateLookaheadQuery = query.Query{Prefix: addrBookBase.String(), KeysOnly: true} + purgeQuery = query.Query{ + Prefix: gcLookaheadBase.String(), + Orders: []query.Order{query.OrderByKey{}}, + KeysOnly: true, + } + populateLookaheadQuery = query.Query{ + Prefix: addrBookBase.String(), + Orders: []query.Order{query.OrderByKey{}}, + KeysOnly: true, + } ) // cyclicBatch is similar to go-datastore autobatch, but it's driven by an actual Batch facility offered by the From 042ae937fc5c9ccf4322f357cc8de0e89154925e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 5 Feb 2019 16:52:27 +0000 Subject: [PATCH 0900/3965] improve code readability. --- p2p/host/peerstore/pstoreds/addr_book.go | 136 ++++++++++---------- p2p/host/peerstore/pstoreds/addr_book_gc.go | 22 ++-- p2p/host/peerstore/test/addr_book_suite.go | 4 +- 3 files changed, 78 insertions(+), 84 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 8ea007f19b..cff7a246d6 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -30,6 +30,7 @@ const ( var ( log = logging.Logger("peerstore/ds") + // Peer addresses are stored under the following db key pattern: // /peers/addrs/ addrBookBase = ds.NewKey("/peers/addrs") @@ -47,9 +48,9 @@ type addrsRecord struct { dirty bool } -// Flush writes the record to the datastore by calling ds.Put, unless the record is -// marked for deletion, in which case the deletion is executed via ds.Delete. -func (r *addrsRecord) Flush(ds dsWriter) (err error) { +// flush writes the record to the datastore by calling ds.Put, unless the record is +// marked for deletion, in which case we call ds.Delete. +func (r *addrsRecord) flush(ds dsWriter) (err error) { key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id.ID))) if len(r.Addrs) == 0 { return ds.Delete(key) @@ -93,7 +94,7 @@ func (r *addrsRecord) Clean() (chgd bool) { if len(r.Addrs) == 0 { // this is a ghost record; let's signal it has to be written. - // Flush() will take care of doing the deletion. + // flush() will take care of doing the deletion. return true } @@ -151,7 +152,7 @@ func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAd } ctx, cancelFn := context.WithCancel(ctx) - mgr := &dsAddrBook{ + ab = &dsAddrBook{ ctx: ctx, cancelFn: cancelFn, opts: opts, @@ -161,10 +162,11 @@ func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAd flushJobCh: make(chan *addrsRecord, 32), } - // kick off periodic GC. - go mgr.background() + // kick off background processes. + go ab.flusher() + go ab.gc() - return mgr, nil + return ab, nil } func (ab *dsAddrBook) Close() { @@ -176,7 +178,7 @@ func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { select { case ab.flushJobCh <- pr: default: - log.Warningf("flush queue is full; could not flush peer %v", pr.Id.ID.Pretty()) + log.Warningf("flush queue is full; could not flush record for peer %s", pr.Id.ID) } } @@ -184,9 +186,9 @@ func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { // datastore upon a miss, and returning a newly initialized record if the peer doesn't exist. // // loadRecord calls Clean() on the record before returning it. If the record changes -// as a result and `update=true`, an async flush is scheduled. +// as a result and the update argument is true, an async flush is queued. // -// If `cache=true`, the record is inserted in the cache when loaded from the datastore. +// If the cache argument is true, the record is inserted in the cache when loaded from the datastore. func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrsRecord, err error) { if e, ok := ab.cache.Get(id); ok { pr = e.(*addrsRecord) @@ -196,23 +198,22 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs return pr, nil } + pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(id))) data, err := ab.ds.Get(key) - if err != nil && err != ds.ErrNotFound { - return nil, err - } - - if err == nil { - pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} + switch err { + case ds.ErrNotFound: + pr.Id = &pb.ProtoPeerID{ID: id} + case nil: if err = pr.Unmarshal(data); err != nil { return nil, err } if pr.Clean() && update { ab.asyncFlush(pr) } - } else { - pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{Id: &pb.ProtoPeerID{ID: id}}} + default: + return nil, err } if cache { @@ -221,64 +222,59 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs return pr, nil } -// background runs the housekeeping process that takes care of: -// -// * GCing expired addresses from the datastore at regular intervals. -// * persisting asynchronous flushes to the datastore. -func (ab *dsAddrBook) background() { - // goroutine that takes care of executing flush jobs. - go func() { - ab.closeDone.Add(1) - for { - select { - case fj := <-ab.flushJobCh: - if cached, ok := ab.cache.Peek(fj.Id.ID); ok { - // Only continue flushing if the record we have in memory is the same as for which the flush - // job was requested. If it's not in memory, it has been evicted and we don't know if we hold - // the latest state or not. Similarly, if it's cached but the pointer is different, it means - // it was evicted and has been reloaded, so we're also uncertain if we hold the latest state. - if pr := cached.(*addrsRecord); pr == fj { - pr.RLock() - pr.Flush(ab.ds) - pr.RUnlock() - } +// flusher is a goroutine that takes care of persisting asynchronous flushes to the datastore. +func (ab *dsAddrBook) flusher() { + ab.closeDone.Add(1) + for { + select { + case fj := <-ab.flushJobCh: + if cached, ok := ab.cache.Peek(fj.Id.ID); ok { + // Only continue flushing if the record we have in memory is the same as for which the flush + // job was requested. If it's not in memory, it has been evicted and we don't know if we hold + // the latest state or not. Similarly, if it's cached but the pointer is different, it means + // it was evicted and has been reloaded, so we're also uncertain if we hold the latest state. + if pr := cached.(*addrsRecord); pr != fj { + pr.RLock() + pr.flush(ab.ds) + pr.RUnlock() } - case <-ab.ctx.Done(): - ab.closeDone.Done() - return } - } - }() - // goroutine that takes care of GC. - go func() { - select { - case <-time.After(ab.opts.GCInitialDelay): case <-ab.ctx.Done(): - // yield if we have been cancelled/closed before the delay elapses. + ab.closeDone.Done() return } + } +} + +// gc is a goroutine that prunes expired addresses from the datastore at regular intervals. +func (ab *dsAddrBook) gc() { + select { + case <-time.After(ab.opts.GCInitialDelay): + case <-ab.ctx.Done(): + // yield if we have been cancelled/closed before the delay elapses. + return + } - ab.closeDone.Add(1) - purgeTimer := time.NewTicker(ab.opts.GCPurgeInterval) - lookaheadTimer := time.NewTicker(ab.opts.GCLookaheadInterval) + ab.closeDone.Add(1) + purgeTimer := time.NewTicker(ab.opts.GCPurgeInterval) + lookaheadTimer := time.NewTicker(ab.opts.GCLookaheadInterval) - for { - select { - case <-purgeTimer.C: - ab.purgeCycle() + for { + select { + case <-purgeTimer.C: + ab.purgeCycle() - case <-lookaheadTimer.C: - ab.populateLookahead() + case <-lookaheadTimer.C: + ab.populateLookahead() - case <-ab.ctx.Done(): - purgeTimer.Stop() - lookaheadTimer.Stop() - ab.closeDone.Done() - return - } + case <-ab.ctx.Done(): + purgeTimer.Stop() + lookaheadTimer.Stop() + ab.closeDone.Done() + return } - }() + } } // AddAddr will add a new address if it's not already in the AddrBook. @@ -331,7 +327,7 @@ func (ab *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.D } if pr.Clean() { - pr.Flush(ab.ds) + pr.flush(ab.ds) } } @@ -391,7 +387,7 @@ func (ab *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duratio defer pr.Unlock() newExp := time.Now().Add(ttl).Unix() - existed := make([]bool, len(addrs)) // keeps track of which addrs we found + existed := make([]bool, len(addrs)) // keeps track of which addrs we found. Outer: for i, incoming := range addrs { @@ -431,7 +427,7 @@ Outer: pr.Addrs = append(pr.Addrs, added...) pr.dirty = true pr.Clean() - return pr.Flush(ab.ds) + return pr.flush(ab.ds) } func (ab *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) (err error) { @@ -464,7 +460,7 @@ func (ab *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) (err error) { pr.dirty = true pr.Clean() - return pr.Flush(ab.ds) + return pr.flush(ab.ds) } func cleanAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 2c45b86468..f1758317d0 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -35,10 +35,11 @@ var ( } ) -// cyclicBatch is similar to go-datastore autobatch, but it's driven by an actual Batch facility offered by the -// datastore. It populates an ongoing batch with operations and automatically flushes it after N pending operations -// have been reached. `N` is currently hardcoded to 20. An explicit `Commit()` closes this cyclic batch, erroring all -// further operations. +// cyclicBatch buffers datastore write operations and automatically flushes them after gcOpsPerBatch (20) have been +// queued. An explicit `Commit()` closes this cyclic batch, erroring all further operations. +// +// It is similar to go-datastore autobatch, but it's driven by an actual Batch facility offered by the +// datastore. type cyclicBatch struct { ds.Batch ds ds.Batching @@ -99,11 +100,8 @@ func (cb *cyclicBatch) Commit() error { return nil } -// purgeCycle runs a single GC cycle, operating within the lookahead window. -// -// It scans the lookahead region for entries that need to be visited, and performs a Clean() on them. An errors trigger -// the removal of the GC entry, in order to prevent unactionable items from accumulating. If the error happened to be -// temporary, the entry will be revisited in the next lookahead window. +// purgeCycle runs a single GC purge cycle. It operates within the lookahead window if lookahead is enabled; else it +// visits all entries in the datastore, deleting the addresses that have expired. func (ab *dsAddrBook) purgeCycle() { if atomic.LoadInt32(&ab.gcLookaheadRunning) > 0 { // yield if lookahead is running. @@ -111,7 +109,7 @@ func (ab *dsAddrBook) purgeCycle() { } var id peer.ID - record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} + record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} // empty record to reuse and avoid allocs. batch, err := newCyclicBatch(ab.ds) if err != nil { log.Warningf("failed while creating batch to purge GC entries: %v", err) @@ -187,7 +185,7 @@ func (ab *dsAddrBook) purgeCycle() { cached := e.(*addrsRecord) cached.Lock() if cached.Clean() { - if err = cached.Flush(batch); err != nil { + if err = cached.flush(batch); err != nil { log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) } } @@ -212,7 +210,7 @@ func (ab *dsAddrBook) purgeCycle() { continue } if record.Clean() { - err = record.Flush(batch) + err = record.flush(batch) if err != nil { log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) } diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index dfd720a631..94f8fb5db2 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -90,8 +90,8 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { }) t.Run("adding an existing address with an earlier expiration is noop", func(t *testing.T) { - id := generatePeerIds(1)[0] - addrs := generateAddrs(3) + id := GeneratePeerIDs(1)[0] + addrs := GenerateAddrs(3) ab.AddAddrs(id, addrs, time.Hour) From 629d11fa9281e9d1c306c16ff0a93fa1c1a302f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 5 Feb 2019 16:53:23 +0000 Subject: [PATCH 0901/3965] adopt new ds.Write interface. --- p2p/host/peerstore/pstoreds/addr_book.go | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index cff7a246d6..45c820256b 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -36,11 +36,6 @@ var ( addrBookBase = ds.NewKey("/peers/addrs") ) -type dsWriter interface { - Put(key ds.Key, value []byte) error - Delete(key ds.Key) error -} - // addrsRecord decorates the AddrBookRecord with locks and metadata. type addrsRecord struct { sync.RWMutex @@ -50,17 +45,17 @@ type addrsRecord struct { // flush writes the record to the datastore by calling ds.Put, unless the record is // marked for deletion, in which case we call ds.Delete. -func (r *addrsRecord) flush(ds dsWriter) (err error) { +func (r *addrsRecord) flush(write ds.Write) (err error) { key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id.ID))) if len(r.Addrs) == 0 { - return ds.Delete(key) + return write.Delete(key) } data, err := r.Marshal() if err != nil { return err } - if err = ds.Put(key, data); err != nil { + if err = write.Put(key, data); err != nil { return err } // write succeeded; record is no longer dirty. From 888582c1e5b66586fff27e8c136fbf2d6101e3a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 5 Feb 2019 17:53:50 +0000 Subject: [PATCH 0902/3965] factor out cyclic batch into file; and gc into type. --- p2p/host/peerstore/pstoreds/addr_book.go | 50 ++----- p2p/host/peerstore/pstoreds/addr_book_gc.go | 138 +++++++++--------- .../peerstore/pstoreds/addr_book_gc_test.go | 19 ++- p2p/host/peerstore/pstoreds/cyclic_batch.go | 72 +++++++++ 4 files changed, 160 insertions(+), 119 deletions(-) create mode 100644 p2p/host/peerstore/pstoreds/cyclic_batch.go diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 45c820256b..ff2dfa4ed7 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -118,20 +118,17 @@ func (r *addrsRecord) Clean() (chgd bool) { // dsAddrBook is an address book backed by a Datastore with a GC-like procedure // to purge expired entries. It uses an in-memory address stream manager. type dsAddrBook struct { - ctx context.Context - + ctx context.Context opts Options cache cache ds ds.Batching + gc *dsAddrBookGc subsManager *pstoremem.AddrSubManager flushJobCh chan *addrsRecord cancelFn func() - closeDone sync.WaitGroup - - gcCurrWindowEnd int64 - gcLookaheadRunning int32 + done sync.WaitGroup } var _ pstore.AddrBook = (*dsAddrBook)(nil) @@ -157,16 +154,18 @@ func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAd flushJobCh: make(chan *addrsRecord, 32), } + ab.gc = newAddressBookGc(ctx, ab) + // kick off background processes. go ab.flusher() - go ab.gc() + go ab.gc.background() return ab, nil } func (ab *dsAddrBook) Close() { ab.cancelFn() - ab.closeDone.Wait() + ab.done.Wait() } func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { @@ -219,7 +218,9 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs // flusher is a goroutine that takes care of persisting asynchronous flushes to the datastore. func (ab *dsAddrBook) flusher() { - ab.closeDone.Add(1) + ab.done.Add(1) + defer ab.done.Done() + for { select { case fj := <-ab.flushJobCh: @@ -236,37 +237,6 @@ func (ab *dsAddrBook) flusher() { } case <-ab.ctx.Done(): - ab.closeDone.Done() - return - } - } -} - -// gc is a goroutine that prunes expired addresses from the datastore at regular intervals. -func (ab *dsAddrBook) gc() { - select { - case <-time.After(ab.opts.GCInitialDelay): - case <-ab.ctx.Done(): - // yield if we have been cancelled/closed before the delay elapses. - return - } - - ab.closeDone.Add(1) - purgeTimer := time.NewTicker(ab.opts.GCPurgeInterval) - lookaheadTimer := time.NewTicker(ab.opts.GCLookaheadInterval) - - for { - select { - case <-purgeTimer.C: - ab.purgeCycle() - - case <-lookaheadTimer.C: - ab.populateLookahead() - - case <-ab.ctx.Done(): - purgeTimer.Stop() - lookaheadTimer.Stop() - ab.closeDone.Done() return } } diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index f1758317d0..80a7a61e3b 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -1,14 +1,13 @@ package pstoreds import ( + "context" "fmt" "strconv" - "sync/atomic" "time" ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" - errors "github.com/pkg/errors" peer "github.com/libp2p/go-libp2p-peer" pb "github.com/libp2p/go-libp2p-peerstore/pb" @@ -20,14 +19,17 @@ var ( // GC lookahead entries are stored in keys with pattern: // /peers/gc/addrs// => nil gcLookaheadBase = ds.NewKey("/peers/gc/addrs") + // in GC routines, how many operations do we place in a batch before it's committed. gcOpsPerBatch = 20 + // queries purgeQuery = query.Query{ Prefix: gcLookaheadBase.String(), Orders: []query.Order{query.OrderByKey{}}, KeysOnly: true, } + populateLookaheadQuery = query.Query{ Prefix: addrBookBase.String(), Orders: []query.Order{query.OrderByKey{}}, @@ -35,82 +37,73 @@ var ( } ) -// cyclicBatch buffers datastore write operations and automatically flushes them after gcOpsPerBatch (20) have been -// queued. An explicit `Commit()` closes this cyclic batch, erroring all further operations. -// -// It is similar to go-datastore autobatch, but it's driven by an actual Batch facility offered by the -// datastore. -type cyclicBatch struct { - ds.Batch - ds ds.Batching - pending int +// dsAddrBookGc encapsulates the GC behaviour to maintain a datastore-backed address book. +type dsAddrBookGc struct { + ctx context.Context + ab *dsAddrBook + running chan struct{} + currWindowEnd int64 } -func newCyclicBatch(ds ds.Batching) (ds.Batch, error) { - batch, err := ds.Batch() - if err != nil { - return nil, err +func newAddressBookGc(ctx context.Context, ab *dsAddrBook) *dsAddrBookGc { + return &dsAddrBookGc{ + ctx: ctx, + ab: ab, + running: make(chan struct{}, 1), } - return &cyclicBatch{Batch: batch, ds: ds}, nil } -func (cb *cyclicBatch) cycle() (err error) { - if cb.Batch == nil { - return errors.New("cyclic batch is closed") - } - if cb.pending < gcOpsPerBatch { - // we haven't reached the threshold yet. - return nil - } - // commit and renew the batch. - if err = cb.Batch.Commit(); err != nil { - return errors.Wrap(err, "failed while committing cyclic batch") - } - if cb.Batch, err = cb.ds.Batch(); err != nil { - return errors.Wrap(err, "failed while renewing cyclic batch") - } - return nil -} +// gc prunes expired addresses from the datastore at regular intervals. It should be spawned as a goroutine. +func (gc *dsAddrBookGc) background() { + gc.ab.done.Add(1) + defer gc.ab.done.Done() -func (cb *cyclicBatch) Put(key ds.Key, val []byte) error { - if err := cb.cycle(); err != nil { - return err + select { + case <-time.After(gc.ab.opts.GCInitialDelay): + case <-gc.ab.ctx.Done(): + // yield if we have been cancelled/closed before the delay elapses. + return } - cb.pending++ - return cb.Batch.Put(key, val) -} -func (cb *cyclicBatch) Delete(key ds.Key) error { - if err := cb.cycle(); err != nil { - return err - } - cb.pending++ - return cb.Batch.Delete(key) -} + purgeTimer := time.NewTicker(gc.ab.opts.GCPurgeInterval) + defer purgeTimer.Stop() -func (cb *cyclicBatch) Commit() error { - if cb.Batch == nil { - return errors.New("cyclic batch is closed") + var lookaheadCh <-chan time.Time + if gc.ab.opts.GCLookaheadInterval > 0 { + lookaheadTimer := time.NewTicker(gc.ab.opts.GCLookaheadInterval) + lookaheadCh = lookaheadTimer.C + defer lookaheadTimer.Stop() } - if err := cb.Batch.Commit(); err != nil { - return err + + for { + select { + case <-purgeTimer.C: + gc.purgeCycle() + + case <-lookaheadCh: + // will never trigger if lookahead is disabled (nil Duration). + gc.populateLookahead() + + case <-gc.ctx.Done(): + return + } } - cb.pending = 0 - cb.Batch = nil - return nil } // purgeCycle runs a single GC purge cycle. It operates within the lookahead window if lookahead is enabled; else it // visits all entries in the datastore, deleting the addresses that have expired. -func (ab *dsAddrBook) purgeCycle() { - if atomic.LoadInt32(&ab.gcLookaheadRunning) > 0 { +func (gc *dsAddrBookGc) purgeCycle() { + select { + case gc.running <- struct{}{}: + defer func() { <-gc.running }() + default: // yield if lookahead is running. return } var id peer.ID record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} // empty record to reuse and avoid allocs. - batch, err := newCyclicBatch(ab.ds) + batch, err := newCyclicBatch(gc.ab.ds) if err != nil { log.Warningf("failed while creating batch to purge GC entries: %v", err) } @@ -135,7 +128,7 @@ func (ab *dsAddrBook) purgeCycle() { } // re-add the record if it needs to be visited again in this window. - if len(ar.Addrs) != 0 && ar.Addrs[0].Expiry <= ab.gcCurrWindowEnd { + if len(ar.Addrs) != 0 && ar.Addrs[0].Expiry <= gc.currWindowEnd { gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", ar.Addrs[0].Expiry, key.Name())) if err := batch.Put(gcKey, []byte{}); err != nil { log.Warningf("failed to add new GC key: %v, err: %v", gcKey, err) @@ -143,7 +136,7 @@ func (ab *dsAddrBook) purgeCycle() { } } - results, err := ab.ds.Query(purgeQuery) + results, err := gc.ab.ds.Query(purgeQuery) if err != nil { log.Warningf("failed while fetching entries to purge: %v", err) return @@ -181,7 +174,7 @@ func (ab *dsAddrBook) purgeCycle() { } // if the record is in cache, we clean it and flush it if necessary. - if e, ok := ab.cache.Peek(id); ok { + if e, ok := gc.ab.cache.Peek(id); ok { cached := e.(*addrsRecord) cached.Lock() if cached.Clean() { @@ -198,7 +191,7 @@ func (ab *dsAddrBook) purgeCycle() { // otherwise, fetch it from the store, clean it and flush it. entryKey := addrBookBase.ChildString(gcKey.Name()) - val, err := ab.ds.Get(entryKey) + val, err := gc.ab.ds.Get(entryKey) if err != nil { // captures all errors, including ErrNotFound. dropInError(gcKey, err, "fetching entry") @@ -224,27 +217,31 @@ func (ab *dsAddrBook) purgeCycle() { } // populateLookahead populates the lookahead window by scanning the entire store and picking entries whose earliest -// expiration falls within the new window. +// expiration falls within the window period. // // Those entries are stored in the lookahead region in the store, indexed by the timestamp when they need to be // visited, to facilitate temporal range scans. -func (ab *dsAddrBook) populateLookahead() { - if !atomic.CompareAndSwapInt32(&ab.gcLookaheadRunning, 0, 1) { +func (gc *dsAddrBookGc) populateLookahead() { + select { + case gc.running <- struct{}{}: + defer func() { <-gc.running }() + default: + // yield if something's running. return } - until := time.Now().Add(ab.opts.GCLookaheadInterval).Unix() + until := time.Now().Add(gc.ab.opts.GCLookaheadInterval).Unix() var id peer.ID record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} - results, err := ab.ds.Query(populateLookaheadQuery) + results, err := gc.ab.ds.Query(populateLookaheadQuery) if err != nil { log.Warningf("failed while querying to populate lookahead GC window: %v", err) return } defer results.Close() - batch, err := newCyclicBatch(ab.ds) + batch, err := newCyclicBatch(gc.ab.ds) if err != nil { log.Warningf("failed while creating batch to populate lookahead GC window: %v", err) return @@ -262,7 +259,7 @@ func (ab *dsAddrBook) populateLookahead() { } // if the record is in cache, use the cached version. - if e, ok := ab.cache.Peek(id); ok { + if e, ok := gc.ab.cache.Peek(id); ok { cached := e.(*addrsRecord) cached.RLock() if len(cached.Addrs) == 0 || cached.Addrs[0].Expiry > until { @@ -279,7 +276,7 @@ func (ab *dsAddrBook) populateLookahead() { record.Reset() - val, err := ab.ds.Get(ds.RawKey(result.Key)) + val, err := gc.ab.ds.Get(ds.RawKey(result.Key)) if err != nil { log.Warningf("failed which getting record from store for peer: %v, err: %v", id.Pretty(), err) continue @@ -300,6 +297,5 @@ func (ab *dsAddrBook) populateLookahead() { log.Warningf("failed to commit GC lookahead batch: %v", err) } - ab.gcCurrWindowEnd = until - atomic.StoreInt32(&ab.gcLookaheadRunning, 0) + gc.currWindowEnd = until } diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc_test.go b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go index 509817f0ac..3d43363ea4 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc_test.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go @@ -44,6 +44,7 @@ func TestGCLookahead(t *testing.T) { factory := addressBookFactory(t, badgerStore, opts) ab, closeFn := factory() + gc := ab.(*dsAddrBook).gc defer closeFn() tp := &testProbe{t, ab} @@ -55,7 +56,8 @@ func TestGCLookahead(t *testing.T) { ab.AddAddrs(ids[0], addrs[:10], time.Hour) ab.AddAddrs(ids[1], addrs[10:20], time.Hour) ab.AddAddrs(ids[2], addrs[20:30], time.Hour) - ab.(*dsAddrBook).populateLookahead() + + gc.populateLookahead() if i := tp.countLookaheadEntries(); i != 0 { t.Errorf("expected no GC lookahead entries, got: %v", i) } @@ -66,14 +68,14 @@ func TestGCLookahead(t *testing.T) { // Purge the cache, to exercise a different path in the lookahead cycle. tp.clearCache() - ab.(*dsAddrBook).populateLookahead() + gc.populateLookahead() if i := tp.countLookaheadEntries(); i != 1 { t.Errorf("expected 1 GC lookahead entry, got: %v", i) } // change addresses of another to have TTL 5 second, placing them in the lookahead window. ab.UpdateAddrs(ids[2], time.Hour, 5*time.Second) - ab.(*dsAddrBook).populateLookahead() + gc.populateLookahead() if i := tp.countLookaheadEntries(); i != 2 { t.Errorf("expected 2 GC lookahead entries, got: %v", i) } @@ -89,6 +91,7 @@ func TestGCPurging(t *testing.T) { factory := addressBookFactory(t, badgerStore, opts) ab, closeFn := factory() + gc := ab.(*dsAddrBook).gc defer closeFn() tp := &testProbe{t, ab} @@ -110,13 +113,13 @@ func TestGCPurging(t *testing.T) { // this is inside the window, but it will survive the purges we do in the test. ab.AddAddrs(ids[3], addrs[70:80], 15*time.Second) - ab.(*dsAddrBook).populateLookahead() + gc.populateLookahead() if i := tp.countLookaheadEntries(); i != 4 { t.Errorf("expected 4 GC lookahead entries, got: %v", i) } <-time.After(2 * time.Second) - ab.(*dsAddrBook).purgeCycle() + gc.purgeCycle() if i := tp.countLookaheadEntries(); i != 3 { t.Errorf("expected 3 GC lookahead entries, got: %v", i) } @@ -125,13 +128,13 @@ func TestGCPurging(t *testing.T) { tp.clearCache() <-time.After(5 * time.Second) - ab.(*dsAddrBook).purgeCycle() + gc.purgeCycle() if i := tp.countLookaheadEntries(); i != 3 { t.Errorf("expected 3 GC lookahead entries, got: %v", i) } <-time.After(5 * time.Second) - ab.(*dsAddrBook).purgeCycle() + gc.purgeCycle() if i := tp.countLookaheadEntries(); i != 1 { t.Errorf("expected 1 GC lookahead entries, got: %v", i) } @@ -200,6 +203,6 @@ func BenchmarkLookaheadCycle(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { - ab.(*dsAddrBook).populateLookahead() + ab.(*dsAddrBook).gc.populateLookahead() } } diff --git a/p2p/host/peerstore/pstoreds/cyclic_batch.go b/p2p/host/peerstore/pstoreds/cyclic_batch.go new file mode 100644 index 0000000000..64dc103c9e --- /dev/null +++ b/p2p/host/peerstore/pstoreds/cyclic_batch.go @@ -0,0 +1,72 @@ +package pstoreds + +import ( + "github.com/pkg/errors" + + ds "github.com/ipfs/go-datastore" +) + +// cyclicBatch buffers ds write operations and automatically flushes them after gcOpsPerBatch (20) have been +// queued. An explicit `Commit()` closes this cyclic batch, erroring all further operations. +// +// It is similar to go-ds autobatch, but it's driven by an actual Batch facility offered by the +// ds. +type cyclicBatch struct { + ds.Batch + ds ds.Batching + pending int +} + +func newCyclicBatch(ds ds.Batching) (ds.Batch, error) { + batch, err := ds.Batch() + if err != nil { + return nil, err + } + return &cyclicBatch{Batch: batch, ds: ds}, nil +} + +func (cb *cyclicBatch) cycle() (err error) { + if cb.Batch == nil { + return errors.New("cyclic batch is closed") + } + if cb.pending < gcOpsPerBatch { + // we haven't reached the threshold yet. + return nil + } + // commit and renew the batch. + if err = cb.Batch.Commit(); err != nil { + return errors.Wrap(err, "failed while committing cyclic batch") + } + if cb.Batch, err = cb.ds.Batch(); err != nil { + return errors.Wrap(err, "failed while renewing cyclic batch") + } + return nil +} + +func (cb *cyclicBatch) Put(key ds.Key, val []byte) error { + if err := cb.cycle(); err != nil { + return err + } + cb.pending++ + return cb.Batch.Put(key, val) +} + +func (cb *cyclicBatch) Delete(key ds.Key) error { + if err := cb.cycle(); err != nil { + return err + } + cb.pending++ + return cb.Batch.Delete(key) +} + +func (cb *cyclicBatch) Commit() error { + if cb.Batch == nil { + return errors.New("cyclic batch is closed") + } + if err := cb.Batch.Commit(); err != nil { + return err + } + cb.pending = 0 + cb.Batch = nil + return nil +} From c0eb65bc40d48a12b35260103bb18e2334105854 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 6 Feb 2019 09:35:10 +0000 Subject: [PATCH 0903/3965] pstore ds: make gc lookahead window optional. --- p2p/host/peerstore/pstoreds/addr_book.go | 5 +- p2p/host/peerstore/pstoreds/addr_book_gc.go | 111 +++++++++++++++--- .../peerstore/pstoreds/addr_book_gc_test.go | 60 ++++++++-- p2p/host/peerstore/pstoreds/peerstore.go | 29 +++-- p2p/host/peerstore/test/addr_book_suite.go | 99 ++++++---------- p2p/host/peerstore/test/utils.go | 22 ++++ 6 files changed, 229 insertions(+), 97 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index ff2dfa4ed7..cf766ba69e 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -154,11 +154,12 @@ func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAd flushJobCh: make(chan *addrsRecord, 32), } - ab.gc = newAddressBookGc(ctx, ab) + if ab.gc, err = newAddressBookGc(ctx, ab); err != nil { + return nil, err + } // kick off background processes. go ab.flusher() - go ab.gc.background() return ab, nil } diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 80a7a61e3b..931dad2278 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -24,12 +24,18 @@ var ( gcOpsPerBatch = 20 // queries - purgeQuery = query.Query{ + purgeLookaheadQuery = query.Query{ Prefix: gcLookaheadBase.String(), Orders: []query.Order{query.OrderByKey{}}, KeysOnly: true, } + purgeStoreQuery = query.Query{ + Prefix: addrBookBase.String(), + Orders: []query.Order{query.OrderByKey{}}, + KeysOnly: false, + } + populateLookaheadQuery = query.Query{ Prefix: addrBookBase.String(), Orders: []query.Order{query.OrderByKey{}}, @@ -39,18 +45,45 @@ var ( // dsAddrBookGc encapsulates the GC behaviour to maintain a datastore-backed address book. type dsAddrBookGc struct { - ctx context.Context - ab *dsAddrBook - running chan struct{} - currWindowEnd int64 + ctx context.Context + ab *dsAddrBook + running chan struct{} + lookaheadEnabled bool + purgeFunc func() + currWindowEnd int64 } -func newAddressBookGc(ctx context.Context, ab *dsAddrBook) *dsAddrBookGc { - return &dsAddrBookGc{ - ctx: ctx, - ab: ab, - running: make(chan struct{}, 1), +func newAddressBookGc(ctx context.Context, ab *dsAddrBook) (*dsAddrBookGc, error) { + if ab.opts.GCPurgeInterval < 0 { + return nil, fmt.Errorf("negative GC purge interval provided: %s", ab.opts.GCPurgeInterval) } + if ab.opts.GCLookaheadInterval < 0 { + return nil, fmt.Errorf("negative GC lookahead interval provided: %s", ab.opts.GCLookaheadInterval) + } + if ab.opts.GCInitialDelay < 0 { + return nil, fmt.Errorf("negative GC initial delay provided: %s", ab.opts.GCInitialDelay) + } + + lookaheadEnabled := ab.opts.GCLookaheadInterval > 0 + gc := &dsAddrBookGc{ + ctx: ctx, + ab: ab, + running: make(chan struct{}, 1), + lookaheadEnabled: lookaheadEnabled, + } + + if lookaheadEnabled { + gc.purgeFunc = gc.purgeLookahead + } else { + gc.purgeFunc = gc.purgeStore + } + + // do not start GC timers if purge is disabled; this GC can only be triggered manually. + if ab.opts.GCPurgeInterval > 0 { + go gc.background() + } + + return gc, nil } // gc prunes expired addresses from the datastore at regular intervals. It should be spawned as a goroutine. @@ -69,7 +102,7 @@ func (gc *dsAddrBookGc) background() { defer purgeTimer.Stop() var lookaheadCh <-chan time.Time - if gc.ab.opts.GCLookaheadInterval > 0 { + if gc.lookaheadEnabled { lookaheadTimer := time.NewTicker(gc.ab.opts.GCLookaheadInterval) lookaheadCh = lookaheadTimer.C defer lookaheadTimer.Stop() @@ -78,7 +111,7 @@ func (gc *dsAddrBookGc) background() { for { select { case <-purgeTimer.C: - gc.purgeCycle() + gc.purgeFunc() case <-lookaheadCh: // will never trigger if lookahead is disabled (nil Duration). @@ -92,7 +125,7 @@ func (gc *dsAddrBookGc) background() { // purgeCycle runs a single GC purge cycle. It operates within the lookahead window if lookahead is enabled; else it // visits all entries in the datastore, deleting the addresses that have expired. -func (gc *dsAddrBookGc) purgeCycle() { +func (gc *dsAddrBookGc) purgeLookahead() { select { case gc.running <- struct{}{}: defer func() { <-gc.running }() @@ -136,7 +169,7 @@ func (gc *dsAddrBookGc) purgeCycle() { } } - results, err := gc.ab.ds.Query(purgeQuery) + results, err := gc.ab.ds.Query(purgeLookaheadQuery) if err != nil { log.Warningf("failed while fetching entries to purge: %v", err) return @@ -216,12 +249,62 @@ func (gc *dsAddrBookGc) purgeCycle() { } } +func (gc *dsAddrBookGc) purgeStore() { + select { + case gc.running <- struct{}{}: + defer func() { <-gc.running }() + default: + // yield if lookahead is running. + return + } + + record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} // empty record to reuse and avoid allocs. + batch, err := newCyclicBatch(gc.ab.ds) + if err != nil { + log.Warningf("failed while creating batch to purge GC entries: %v", err) + } + + results, err := gc.ab.ds.Query(purgeStoreQuery) + if err != nil { + log.Warningf("failed while opening iterator: %v", err) + return + } + defer results.Close() + + // keys: /peers/addrs/ + for result := range results.Next() { + record.Reset() + if err = record.Unmarshal(result.Value); err != nil { + // TODO log + continue + } + + id := record.Id.ID + if !record.Clean() { + continue + } + + if err := record.flush(batch); err != nil { + log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id, err) + } + gc.ab.cache.Remove(id) + } + + if err = batch.Commit(); err != nil { + log.Warningf("failed to commit GC purge batch: %v", err) + } +} + // populateLookahead populates the lookahead window by scanning the entire store and picking entries whose earliest // expiration falls within the window period. // // Those entries are stored in the lookahead region in the store, indexed by the timestamp when they need to be // visited, to facilitate temporal range scans. func (gc *dsAddrBookGc) populateLookahead() { + if gc.ab.opts.GCLookaheadInterval == 0 { + return + } + select { case gc.running <- struct{}{}: defer func() { <-gc.running }() diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc_test.go b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go index 3d43363ea4..9486543db0 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc_test.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go @@ -4,16 +4,17 @@ import ( "testing" "time" - "github.com/ipfs/go-datastore/query" - "github.com/libp2p/go-libp2p-peerstore" - "github.com/libp2p/go-libp2p-peerstore/test" + query "github.com/ipfs/go-datastore/query" + pstore "github.com/libp2p/go-libp2p-peerstore" + test "github.com/libp2p/go-libp2p-peerstore/test" + ma "github.com/multiformats/go-multiaddr" ) var lookaheadQuery = query.Query{Prefix: gcLookaheadBase.String(), KeysOnly: true} type testProbe struct { t *testing.T - ab peerstore.AddrBook + ab pstore.AddrBook } func (tp *testProbe) countLookaheadEntries() (i int) { @@ -119,7 +120,7 @@ func TestGCPurging(t *testing.T) { } <-time.After(2 * time.Second) - gc.purgeCycle() + gc.purgeLookahead() if i := tp.countLookaheadEntries(); i != 3 { t.Errorf("expected 3 GC lookahead entries, got: %v", i) } @@ -128,13 +129,13 @@ func TestGCPurging(t *testing.T) { tp.clearCache() <-time.After(5 * time.Second) - gc.purgeCycle() + gc.purgeLookahead() if i := tp.countLookaheadEntries(); i != 3 { t.Errorf("expected 3 GC lookahead entries, got: %v", i) } <-time.After(5 * time.Second) - gc.purgeCycle() + gc.purgeLookahead() if i := tp.countLookaheadEntries(); i != 1 { t.Errorf("expected 1 GC lookahead entries, got: %v", i) } @@ -176,6 +177,51 @@ func TestGCDelay(t *testing.T) { } } +func TestGCLookaheadDisabled(t *testing.T) { + ids := test.GeneratePeerIDs(10) + addrs := test.GenerateAddrs(100) + + opts := DefaultOpts() + + // effectively disable automatic GC for this test. + opts.GCInitialDelay = 90 * time.Hour + opts.GCLookaheadInterval = 0 // disable lookahead + opts.GCPurgeInterval = 9 * time.Hour + + factory := addressBookFactory(t, badgerStore, opts) + ab, closeFn := factory() + defer closeFn() + + tp := &testProbe{t, ab} + + // four peers: + // ids[0] has 10 addresses, all of which expire in 500ms. + // ids[1] has 20 addresses; 50% expire in 500ms and 50% in 10 hours. + // ids[2] has 10 addresses; all expire in 10 hours. + // ids[3] has 60 addresses; all expire in 10 hours. + ab.AddAddrs(ids[0], addrs[:10], 500*time.Millisecond) + ab.AddAddrs(ids[1], addrs[10:20], 500*time.Millisecond) + ab.AddAddrs(ids[1], addrs[20:30], 10*time.Hour) + ab.AddAddrs(ids[2], addrs[30:40], 10*time.Hour) + ab.AddAddrs(ids[3], addrs[40:], 10*time.Hour) + + time.Sleep(100 * time.Millisecond) + + if i := tp.countLookaheadEntries(); i != 0 { + t.Errorf("expected no GC lookahead entries, got: %v", i) + } + + time.Sleep(500 * time.Millisecond) + gc := ab.(*dsAddrBook).gc + gc.purgeFunc() + + var empty []ma.Multiaddr + test.AssertAddressesEqual(t, empty, ab.Addrs(ids[0])) + test.AssertAddressesEqual(t, addrs[20:30], ab.Addrs(ids[1])) + test.AssertAddressesEqual(t, addrs[30:40], ab.Addrs(ids[2])) + test.AssertAddressesEqual(t, addrs[40:], ab.Addrs(ids[3])) +} + func BenchmarkLookaheadCycle(b *testing.B) { ids := test.GeneratePeerIDs(100) addrs := test.GenerateAddrs(100) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 71378d0743..7f4a1a785c 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -18,26 +18,30 @@ type Options struct { // The size of the in-memory cache. A value of 0 or lower disables the cache. CacheSize uint - // Sweep interval to purge expired addresses from the datastore. + // Sweep interval to purge expired addresses from the datastore. If this is a zero value, GC will not run + // automatically, but it'll be available on demand via explicit calls. GCPurgeInterval time.Duration - // Interval to renew the GC lookahead window. + // Interval to renew the GC lookahead window. If this is a zero value, lookahead will be disabled and we'll + // traverse the entire datastore for every purge cycle. GCLookaheadInterval time.Duration - // Initial delay before GC routines start. Intended to give the system time to initialise before starting GC. + // Initial delay before GC processes start. Intended to give the system breathing room to fully boot + // before starting GC. GCInitialDelay time.Duration } // DefaultOpts returns the default options for a persistent peerstore: -// * Cache size: 1024 -// * GC prune interval: 5 minutes -// * GC lookahead interval: 12 hours -// * WriteRetries: 5 +// +// * Cache size: 1024. +// * GC purge interval: 10 minutes. +// * GC lookahead interval: disabled. +// * GC initial delay: 60 seconds. func DefaultOpts() Options { return Options{ CacheSize: 1024, - GCPurgeInterval: 5 * time.Minute, - GCLookaheadInterval: 12 * time.Hour, + GCPurgeInterval: 10 * time.Minute, + GCLookaheadInterval: 0, GCInitialDelay: 60 * time.Second, } } @@ -88,12 +92,11 @@ func uniquePeerIds(ds ds.Datastore, prefix ds.Key, extractor func(result query.R return peer.IDSlice{}, nil } - ids := make(peer.IDSlice, len(idset)) - i := 0 + ids := make(peer.IDSlice, 0, len(idset)) for id := range idset { pid, _ := base32.RawStdEncoding.DecodeString(id) - ids[i], _ = peer.IDFromBytes(pid) - i++ + id, _ := peer.IDFromBytes(pid) + ids = append(ids, id) } return ids, nil } diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index 94f8fb5db2..c01fe98d73 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -5,7 +5,6 @@ import ( "time" pstore "github.com/libp2p/go-libp2p-peerstore" - ma "github.com/multiformats/go-multiaddr" ) var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ @@ -44,7 +43,7 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { ab.AddAddr(id, addrs[0], time.Hour) - testHas(t, addrs, ab.Addrs(id)) + AssertAddressesEqual(t, addrs, ab.Addrs(id)) }) t.Run("idempotent add single address", func(t *testing.T) { @@ -54,7 +53,7 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { ab.AddAddr(id, addrs[0], time.Hour) ab.AddAddr(id, addrs[0], time.Hour) - testHas(t, addrs, ab.Addrs(id)) + AssertAddressesEqual(t, addrs, ab.Addrs(id)) }) t.Run("add multiple addresses", func(t *testing.T) { @@ -62,7 +61,7 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { addrs := GenerateAddrs(3) ab.AddAddrs(id, addrs, time.Hour) - testHas(t, addrs, ab.Addrs(id)) + AssertAddressesEqual(t, addrs, ab.Addrs(id)) }) t.Run("idempotent add multiple addresses", func(t *testing.T) { @@ -72,7 +71,7 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { ab.AddAddrs(id, addrs, time.Hour) ab.AddAddrs(id, addrs, time.Hour) - testHas(t, addrs, ab.Addrs(id)) + AssertAddressesEqual(t, addrs, ab.Addrs(id)) }) t.Run("adding an existing address with a later expiration extends its ttl", func(t *testing.T) { @@ -86,7 +85,7 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { // after the initial TTL has expired, check that only the third address is present. time.Sleep(1200 * time.Millisecond) - testHas(t, addrs[2:], ab.Addrs(id)) + AssertAddressesEqual(t, addrs[2:], ab.Addrs(id)) }) t.Run("adding an existing address with an earlier expiration is noop", func(t *testing.T) { @@ -101,7 +100,7 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { // after the initial TTL has expired, check that all three addresses are still present (i.e. the TTL on // the modified one was not shortened). time.Sleep(2100 * time.Millisecond) - testHas(t, addrs, ab.Addrs(id)) + AssertAddressesEqual(t, addrs, ab.Addrs(id)) }) } } @@ -114,16 +113,16 @@ func testClearWorks(ab pstore.AddrBook) func(t *testing.T) { ab.AddAddrs(ids[0], addrs[0:3], time.Hour) ab.AddAddrs(ids[1], addrs[3:], time.Hour) - testHas(t, addrs[0:3], ab.Addrs(ids[0])) - testHas(t, addrs[3:], ab.Addrs(ids[1])) + AssertAddressesEqual(t, addrs[0:3], ab.Addrs(ids[0])) + AssertAddressesEqual(t, addrs[3:], ab.Addrs(ids[1])) ab.ClearAddrs(ids[0]) - testHas(t, nil, ab.Addrs(ids[0])) - testHas(t, addrs[3:], ab.Addrs(ids[1])) + AssertAddressesEqual(t, nil, ab.Addrs(ids[0])) + AssertAddressesEqual(t, addrs[3:], ab.Addrs(ids[1])) ab.ClearAddrs(ids[1]) - testHas(t, nil, ab.Addrs(ids[0])) - testHas(t, nil, ab.Addrs(ids[1])) + AssertAddressesEqual(t, nil, ab.Addrs(ids[0])) + AssertAddressesEqual(t, nil, ab.Addrs(ids[1])) } } @@ -133,7 +132,7 @@ func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { addrs := GenerateAddrs(100) m.SetAddrs(id, addrs, time.Hour) - testHas(t, addrs, m.Addrs(id)) + AssertAddressesEqual(t, addrs, m.Addrs(id)) // remove two addresses. m.SetAddr(id, addrs[50], -1) @@ -143,7 +142,7 @@ func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { survivors := append(addrs[0:50], addrs[51:]...) survivors = append(survivors[0:74], survivors[75:]...) - testHas(t, survivors, m.Addrs(id)) + AssertAddressesEqual(t, survivors, m.Addrs(id)) } } @@ -167,8 +166,8 @@ func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { m.SetAddr(ids[1], addrs2[1], time.Minute) // Sanity check. - testHas(t, addrs1, m.Addrs(ids[0])) - testHas(t, addrs2, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1, m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2, m.Addrs(ids[1])) // Will only affect addrs1[0]. // Badger does not support subsecond TTLs. @@ -176,26 +175,26 @@ func testUpdateTTLs(m pstore.AddrBook) func(t *testing.T) { m.UpdateAddrs(ids[0], time.Hour, 1*time.Second) // No immediate effect. - testHas(t, addrs1, m.Addrs(ids[0])) - testHas(t, addrs2, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1, m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2, m.Addrs(ids[1])) // After a wait, addrs[0] is gone. time.Sleep(1500 * time.Millisecond) - testHas(t, addrs1[1:2], m.Addrs(ids[0])) - testHas(t, addrs2, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1[1:2], m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2, m.Addrs(ids[1])) // Will only affect addrs2[0]. m.UpdateAddrs(ids[1], time.Hour, 1*time.Second) // No immediate effect. - testHas(t, addrs1[1:2], m.Addrs(ids[0])) - testHas(t, addrs2, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1[1:2], m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2, m.Addrs(ids[1])) time.Sleep(1500 * time.Millisecond) // First addrs is gone in both. - testHas(t, addrs1[1:], m.Addrs(ids[0])) - testHas(t, addrs2[1:], m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1[1:], m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2[1:], m.Addrs(ids[1])) }) } @@ -219,39 +218,39 @@ func testAddressesExpire(m pstore.AddrBook) func(t *testing.T) { m.AddAddrs(ids[0], addrs1, time.Hour) m.AddAddrs(ids[1], addrs2, time.Hour) - testHas(t, addrs1, m.Addrs(ids[0])) - testHas(t, addrs2, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1, m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2, m.Addrs(ids[1])) m.AddAddrs(ids[0], addrs1, 2*time.Hour) m.AddAddrs(ids[1], addrs2, 2*time.Hour) - testHas(t, addrs1, m.Addrs(ids[0])) - testHas(t, addrs2, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1, m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2, m.Addrs(ids[1])) m.SetAddr(ids[0], addrs1[0], 100*time.Microsecond) <-time.After(100 * time.Millisecond) - testHas(t, addrs1[1:3], m.Addrs(ids[0])) - testHas(t, addrs2, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1[1:3], m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2, m.Addrs(ids[1])) m.SetAddr(ids[0], addrs1[2], 100*time.Microsecond) <-time.After(100 * time.Millisecond) - testHas(t, addrs1[1:2], m.Addrs(ids[0])) - testHas(t, addrs2, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1[1:2], m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2, m.Addrs(ids[1])) m.SetAddr(ids[1], addrs2[0], 100*time.Microsecond) <-time.After(100 * time.Millisecond) - testHas(t, addrs1[1:2], m.Addrs(ids[0])) - testHas(t, addrs2[1:], m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1[1:2], m.Addrs(ids[0])) + AssertAddressesEqual(t, addrs2[1:], m.Addrs(ids[1])) m.SetAddr(ids[1], addrs2[1], 100*time.Microsecond) <-time.After(100 * time.Millisecond) - testHas(t, addrs1[1:2], m.Addrs(ids[0])) - testHas(t, nil, m.Addrs(ids[1])) + AssertAddressesEqual(t, addrs1[1:2], m.Addrs(ids[0])) + AssertAddressesEqual(t, nil, m.Addrs(ids[1])) m.SetAddr(ids[0], addrs1[1], 100*time.Microsecond) <-time.After(100 * time.Millisecond) - testHas(t, nil, m.Addrs(ids[0])) - testHas(t, nil, m.Addrs(ids[1])) + AssertAddressesEqual(t, nil, m.Addrs(ids[0])) + AssertAddressesEqual(t, nil, m.Addrs(ids[1])) } } @@ -307,25 +306,3 @@ func testPeersWithAddrs(m pstore.AddrBook) func(t *testing.T) { }) } } - -func testHas(t *testing.T, exp, act []ma.Multiaddr) { - t.Helper() - if len(exp) != len(act) { - t.Fatalf("lengths not the same. expected %d, got %d\n", len(exp), len(act)) - } - - for _, a := range exp { - found := false - - for _, b := range act { - if a.Equal(b) { - found = true - break - } - } - - if !found { - t.Fatalf("expected address %s not found", a) - } - } -} diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index 15b73c8d54..d7af3ab81a 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -72,3 +72,25 @@ func GeneratePeerIDs(count int) []peer.ID { } return ids } + +func AssertAddressesEqual(t *testing.T, exp, act []ma.Multiaddr) { + t.Helper() + if len(exp) != len(act) { + t.Fatalf("lengths not the same. expected %d, got %d\n", len(exp), len(act)) + } + + for _, a := range exp { + found := false + + for _, b := range act { + if a.Equal(b) { + found = true + break + } + } + + if !found { + t.Fatalf("expected address %s not found", a) + } + } +} From 7bd040c51ecc83f6344750a168a050a4c37523ef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 6 Feb 2019 14:35:39 +0000 Subject: [PATCH 0904/3965] improve docs; cyclic batch arg; validations. --- p2p/host/peerstore/pstoreds/addr_book.go | 51 +++++++++++++-------- p2p/host/peerstore/pstoreds/addr_book_gc.go | 18 ++++---- p2p/host/peerstore/pstoreds/cyclic_batch.go | 10 ++-- 3 files changed, 50 insertions(+), 29 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index cf766ba69e..624f35e373 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -31,7 +31,7 @@ const ( var ( log = logging.Logger("peerstore/ds") - // Peer addresses are stored under the following db key pattern: + // Peer addresses are stored db key pattern: // /peers/addrs/ addrBookBase = ds.NewKey("/peers/addrs") ) @@ -44,7 +44,7 @@ type addrsRecord struct { } // flush writes the record to the datastore by calling ds.Put, unless the record is -// marked for deletion, in which case we call ds.Delete. +// marked for deletion, in which case we call ds.Delete. To be called within a lock. func (r *addrsRecord) flush(write ds.Write) (err error) { key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id.ID))) if len(r.Addrs) == 0 { @@ -63,20 +63,19 @@ func (r *addrsRecord) flush(write ds.Write) (err error) { return nil } -// Clean is called on records to perform housekeeping. The return value signals if the record was changed -// as a result of the cleaning. +// Clean is called on records to perform housekeeping. The return value indicates if the record was changed +// as a result of this call. // // Clean does the following: -// * sorts the addresses by expiration (soonest expiring first). -// * removes the addresses that have expired. +// * sorts addresses by expiration (soonest expiring first). +// * removes expired addresses. // -// It short-circuits optimistically when we know there's nothing to do. +// It short-circuits optimistically when there's nothing to do. // // Clean is called from several points: -// * when accessing and loading an entry. +// * when accessing an entry. // * when performing periodic GC. -// * after an entry has been modified (e.g. addresses have been added or removed, -// TTLs updated, etc.) +// * after an entry has been modified (e.g. addresses have been added or removed, TTLs updated, etc.) // // If the return value is true, the caller can perform a flush immediately, or can schedule an async // flush, depending on the context. @@ -101,8 +100,8 @@ func (r *addrsRecord) Clean() (chgd bool) { }) } - // since addresses are sorted by expiration, we find the first survivor and split the - // slice on its index. + // since addresses are sorted by expiration, we find the first + // survivor and split the slice on its index. pivot := -1 for i, addr := range r.Addrs { if addr.Expiry > now { @@ -115,8 +114,8 @@ func (r *addrsRecord) Clean() (chgd bool) { return r.dirty || pivot >= 0 } -// dsAddrBook is an address book backed by a Datastore with a GC-like procedure -// to purge expired entries. It uses an in-memory address stream manager. +// dsAddrBook is an address book backed by a Datastore with a GC procedure to purge expired entries. It uses an +// in-memory address stream manager. See the NewAddrBook for more information. type dsAddrBook struct { ctx context.Context opts Options @@ -133,8 +132,25 @@ type dsAddrBook struct { var _ pstore.AddrBook = (*dsAddrBook)(nil) -// NewAddrBook initializes a new address book given a Datastore instance, a context for managing the TTL manager, -// and the interval at which the TTL manager should sweep the Datastore. +// NewAddrBook initializes a new datastore-backed address book. It serves as a drop-in replacement for pstoremem +// (memory-backed peerstore), and works with any datastore implementing the ds.Batching interface. +// +// Addresses and peer records are serialized into protobuf, storing one datastore entry per peer, along with metadata +// to control address expiration. To alleviate disk access and serde overhead, we internally use a read/write-through +// ARC cache, the size of which is adjustable via Options.CacheSize. +// +// The user has a choice of two GC algorithms: +// +// - lookahead GC: minimises the amount of full store traversals by maintaining a time-indexed list of entries that +// need to be visited within the period specified in Options.GCLookaheadInterval. This is useful in scenarios with +// considerable TTL variance, coupled with datastores whose native iterators return entries in lexicographical key +// order. Enable this mode by passing a value Options.GCLookaheadInterval > 0. Lookahead windows are jumpy, not +// sliding. Purges operate exclusively over the lookahead window with periodicity Options.GCPurgeInterval. +// +// - full-purge GC (default): performs a full visit of the store with periodicity Options.GCPurgeInterval. Useful when +// the range of possible TTL values is small and the values themselves are also extreme, e.g. 10 minutes or +// permanent, popular values used in other libp2p modules. In this cited case, optimizing with lookahead windows +// makes little sense. func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAddrBook, err error) { var cache cache = new(noopCache) if opts.CacheSize > 0 { @@ -158,7 +174,6 @@ func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAd return nil, err } - // kick off background processes. go ab.flusher() return ab, nil @@ -180,7 +195,7 @@ func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { // loadRecord is a read-through fetch. It fetches a record from cache, falling back to the // datastore upon a miss, and returning a newly initialized record if the peer doesn't exist. // -// loadRecord calls Clean() on the record before returning it. If the record changes +// loadRecord calls Clean() on existing recordsrecord before returning it. If the record changes // as a result and the update argument is true, an async flush is queued. // // If the cache argument is true, the record is inserted in the cache when loaded from the datastore. diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 931dad2278..61444abb91 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -16,13 +16,12 @@ import ( ) var ( - // GC lookahead entries are stored in keys with pattern: + // GC lookahead entries are stored in key pattern: // /peers/gc/addrs// => nil + // in databases with lexicographical key order, this time-indexing allows us to visit + // only the timeslice we are interested in. gcLookaheadBase = ds.NewKey("/peers/gc/addrs") - // in GC routines, how many operations do we place in a batch before it's committed. - gcOpsPerBatch = 20 - // queries purgeLookaheadQuery = query.Query{ Prefix: gcLookaheadBase.String(), @@ -43,7 +42,6 @@ var ( } ) -// dsAddrBookGc encapsulates the GC behaviour to maintain a datastore-backed address book. type dsAddrBookGc struct { ctx context.Context ab *dsAddrBook @@ -63,6 +61,10 @@ func newAddressBookGc(ctx context.Context, ab *dsAddrBook) (*dsAddrBookGc, error if ab.opts.GCInitialDelay < 0 { return nil, fmt.Errorf("negative GC initial delay provided: %s", ab.opts.GCInitialDelay) } + if ab.opts.GCLookaheadInterval > 0 && ab.opts.GCLookaheadInterval < ab.opts.GCPurgeInterval { + return nil, fmt.Errorf("lookahead interval must be larger than purge interval, respectively: %s, %s", + ab.opts.GCLookaheadInterval, ab.opts.GCPurgeInterval) + } lookaheadEnabled := ab.opts.GCLookaheadInterval > 0 gc := &dsAddrBookGc{ @@ -136,7 +138,7 @@ func (gc *dsAddrBookGc) purgeLookahead() { var id peer.ID record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} // empty record to reuse and avoid allocs. - batch, err := newCyclicBatch(gc.ab.ds) + batch, err := newCyclicBatch(gc.ab.ds, defaultOpsPerCyclicBatch) if err != nil { log.Warningf("failed while creating batch to purge GC entries: %v", err) } @@ -259,7 +261,7 @@ func (gc *dsAddrBookGc) purgeStore() { } record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} // empty record to reuse and avoid allocs. - batch, err := newCyclicBatch(gc.ab.ds) + batch, err := newCyclicBatch(gc.ab.ds, defaultOpsPerCyclicBatch) if err != nil { log.Warningf("failed while creating batch to purge GC entries: %v", err) } @@ -324,7 +326,7 @@ func (gc *dsAddrBookGc) populateLookahead() { } defer results.Close() - batch, err := newCyclicBatch(gc.ab.ds) + batch, err := newCyclicBatch(gc.ab.ds, defaultOpsPerCyclicBatch) if err != nil { log.Warningf("failed while creating batch to populate lookahead GC window: %v", err) return diff --git a/p2p/host/peerstore/pstoreds/cyclic_batch.go b/p2p/host/peerstore/pstoreds/cyclic_batch.go index 64dc103c9e..1cf8579ca3 100644 --- a/p2p/host/peerstore/pstoreds/cyclic_batch.go +++ b/p2p/host/peerstore/pstoreds/cyclic_batch.go @@ -6,18 +6,22 @@ import ( ds "github.com/ipfs/go-datastore" ) -// cyclicBatch buffers ds write operations and automatically flushes them after gcOpsPerBatch (20) have been +// how many operations are queued in a cyclic batch before we flush it. +var defaultOpsPerCyclicBatch = 20 + +// cyclicBatch buffers ds write operations and automatically flushes them after defaultOpsPerCyclicBatch (20) have been // queued. An explicit `Commit()` closes this cyclic batch, erroring all further operations. // // It is similar to go-ds autobatch, but it's driven by an actual Batch facility offered by the // ds. type cyclicBatch struct { + threshold int ds.Batch ds ds.Batching pending int } -func newCyclicBatch(ds ds.Batching) (ds.Batch, error) { +func newCyclicBatch(ds ds.Batching, threshold int) (ds.Batch, error) { batch, err := ds.Batch() if err != nil { return nil, err @@ -29,7 +33,7 @@ func (cb *cyclicBatch) cycle() (err error) { if cb.Batch == nil { return errors.New("cyclic batch is closed") } - if cb.pending < gcOpsPerBatch { + if cb.pending < cb.threshold { // we haven't reached the threshold yet. return nil } From fa5c778b5de4790b5219fd323b17b1d2d92742ba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 6 Feb 2019 14:40:51 +0000 Subject: [PATCH 0905/3965] make addrsRecord.Clean() private. --- p2p/host/peerstore/pstoreds/addr_book.go | 20 ++++++++++---------- p2p/host/peerstore/pstoreds/addr_book_gc.go | 6 +++--- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 624f35e373..2f71c1e6a6 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -63,23 +63,23 @@ func (r *addrsRecord) flush(write ds.Write) (err error) { return nil } -// Clean is called on records to perform housekeeping. The return value indicates if the record was changed +// clean is called on records to perform housekeeping. The return value indicates if the record was changed // as a result of this call. // -// Clean does the following: +// clean does the following: // * sorts addresses by expiration (soonest expiring first). // * removes expired addresses. // // It short-circuits optimistically when there's nothing to do. // -// Clean is called from several points: +// clean is called from several points: // * when accessing an entry. // * when performing periodic GC. // * after an entry has been modified (e.g. addresses have been added or removed, TTLs updated, etc.) // // If the return value is true, the caller can perform a flush immediately, or can schedule an async // flush, depending on the context. -func (r *addrsRecord) Clean() (chgd bool) { +func (r *addrsRecord) clean() (chgd bool) { now := time.Now().Unix() if !r.dirty && len(r.Addrs) > 0 && r.Addrs[0].Expiry > now { // record is not dirty, and we have no expired entries to purge. @@ -195,14 +195,14 @@ func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { // loadRecord is a read-through fetch. It fetches a record from cache, falling back to the // datastore upon a miss, and returning a newly initialized record if the peer doesn't exist. // -// loadRecord calls Clean() on existing recordsrecord before returning it. If the record changes +// loadRecord calls clean() on existing recordsrecord before returning it. If the record changes // as a result and the update argument is true, an async flush is queued. // // If the cache argument is true, the record is inserted in the cache when loaded from the datastore. func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrsRecord, err error) { if e, ok := ab.cache.Get(id); ok { pr = e.(*addrsRecord) - if pr.Clean() && update { + if pr.clean() && update { ab.asyncFlush(pr) } return pr, nil @@ -219,7 +219,7 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs if err = pr.Unmarshal(data); err != nil { return nil, err } - if pr.Clean() && update { + if pr.clean() && update { ab.asyncFlush(pr) } default: @@ -307,7 +307,7 @@ func (ab *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.D pr.dirty = true } - if pr.Clean() { + if pr.clean() { pr.flush(ab.ds) } } @@ -407,7 +407,7 @@ Outer: pr.Addrs = append(pr.Addrs, added...) pr.dirty = true - pr.Clean() + pr.clean() return pr.flush(ab.ds) } @@ -440,7 +440,7 @@ func (ab *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) (err error) { pr.Addrs = pr.Addrs[:survived] pr.dirty = true - pr.Clean() + pr.clean() return pr.flush(ab.ds) } diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 61444abb91..5316cf5bef 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -212,7 +212,7 @@ func (gc *dsAddrBookGc) purgeLookahead() { if e, ok := gc.ab.cache.Peek(id); ok { cached := e.(*addrsRecord) cached.Lock() - if cached.Clean() { + if cached.clean() { if err = cached.flush(batch); err != nil { log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) } @@ -237,7 +237,7 @@ func (gc *dsAddrBookGc) purgeLookahead() { dropInError(gcKey, err, "unmarshalling entry") continue } - if record.Clean() { + if record.clean() { err = record.flush(batch) if err != nil { log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) @@ -282,7 +282,7 @@ func (gc *dsAddrBookGc) purgeStore() { } id := record.Id.ID - if !record.Clean() { + if !record.clean() { continue } From 12975960dea6f1acc70ee2484412c0b0bc36e367 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 6 Feb 2019 14:43:10 +0000 Subject: [PATCH 0906/3965] change default purge interval to 2 hours. --- p2p/host/peerstore/pstoreds/peerstore.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 7f4a1a785c..fc0263d48b 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -31,16 +31,16 @@ type Options struct { GCInitialDelay time.Duration } -// DefaultOpts returns the default options for a persistent peerstore: +// DefaultOpts returns the default options for a persistent peerstore, with the full-purge GC algorithm: // // * Cache size: 1024. -// * GC purge interval: 10 minutes. +// * GC purge interval: 2 hours. // * GC lookahead interval: disabled. // * GC initial delay: 60 seconds. func DefaultOpts() Options { return Options{ CacheSize: 1024, - GCPurgeInterval: 10 * time.Minute, + GCPurgeInterval: 2 * time.Hour, GCLookaheadInterval: 0, GCInitialDelay: 60 * time.Second, } From b5b6b71afeda4b14985ab4dfca9506e266068c1c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 6 Feb 2019 16:00:54 +0000 Subject: [PATCH 0907/3965] fix tests; run lookahead after initial delay. --- p2p/host/peerstore/pstoreds/addr_book_gc.go | 1 + p2p/host/peerstore/pstoreds/addr_book_gc_test.go | 12 ++++++------ 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 5316cf5bef..f8b656f4e9 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -107,6 +107,7 @@ func (gc *dsAddrBookGc) background() { if gc.lookaheadEnabled { lookaheadTimer := time.NewTicker(gc.ab.opts.GCLookaheadInterval) lookaheadCh = lookaheadTimer.C + gc.populateLookahead() // do a lookahead now defer lookaheadTimer.Stop() } diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc_test.go b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go index 9486543db0..2c328eebc8 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc_test.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go @@ -41,7 +41,7 @@ func TestGCLookahead(t *testing.T) { // effectively disable automatic GC for this test. opts.GCInitialDelay = 90 * time.Hour opts.GCLookaheadInterval = 10 * time.Second - opts.GCPurgeInterval = 1 * time.Minute + opts.GCPurgeInterval = 1 * time.Second factory := addressBookFactory(t, badgerStore, opts) ab, closeFn := factory() @@ -88,7 +88,7 @@ func TestGCPurging(t *testing.T) { // effectively disable automatic GC for this test. opts.GCInitialDelay = 90 * time.Hour opts.GCLookaheadInterval = 20 * time.Second - opts.GCPurgeInterval = 1 * time.Minute + opts.GCPurgeInterval = 1 * time.Second factory := addressBookFactory(t, badgerStore, opts) ab, closeFn := factory() @@ -154,8 +154,8 @@ func TestGCDelay(t *testing.T) { opts := DefaultOpts() opts.GCInitialDelay = 2 * time.Second - opts.GCLookaheadInterval = 2 * time.Second - opts.GCPurgeInterval = 6 * time.Hour + opts.GCLookaheadInterval = 1 * time.Minute + opts.GCPurgeInterval = 30 * time.Second factory := addressBookFactory(t, badgerStore, opts) ab, closeFn := factory() @@ -170,8 +170,8 @@ func TestGCDelay(t *testing.T) { t.Errorf("expected no lookahead entries, got: %d", i) } - // delay + lookahead interval = 4 seconds + 2 seconds for some slack = 6 seconds. - <-time.After(6 * time.Second) + // after the initial delay has passed. + <-time.After(3 * time.Second) if i := tp.countLookaheadEntries(); i != 1 { t.Errorf("expected 1 lookahead entry, got: %d", i) } From 3e74f0464a1423dae6f382813e954f08a15c3c6f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 7 Feb 2019 15:31:23 -0800 Subject: [PATCH 0908/3965] gx publish 6.0.31 --- .gx/lastpubver | 2 +- package.json | 90 +++++++++++++++++++++++++------------------------- 2 files changed, 46 insertions(+), 46 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 5ffe1f8e09..4df79f556d 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.30: QmYxivS34F2M2n44WQQnRHGAKS8aoRUxwGpi9wk4Cdn4Jf +6.0.31: QmSgtf5vHyugoxcwMbyNy6bZ9qPDDTJSYEED2GkWjLwitZ diff --git a/package.json b/package.json index c58d67e70c..81ea30dafb 100644 --- a/package.json +++ b/package.json @@ -50,33 +50,33 @@ }, { "author": "whyrusleeping", - "hash": "QmWZnkipEJbsdUDhgTGBnKAN1iHM2WDMNqsXHyAuaAiCgo", + "hash": "QmU7CkhdputERjy5QQr4kEUsKWzQPmkw3DZEMWxeShu6QR", "name": "go-libp2p-loggables", - "version": "1.1.29" + "version": "1.1.30" }, { "author": "whyrusleeping", - "hash": "QmQoDgEDBPJ6RfKDgdoMpPwoK4WEZHUosHoUs6YVDGoen3", + "hash": "QmQsNqBLwQbEGMJ85zAT6D7zZnLyCR57YWh4sh4g1V43qK", "name": "go-libp2p-secio", - "version": "2.0.24" + "version": "2.0.25" }, { "author": "whyrusleeping", - "hash": "QmPiemjiKBC9VA7vZF82m4x1oygtg2c2YVqag8PX7dN1BD", + "hash": "QmQFFp4ntkd4C14sP3FaH9WJyBuetuGUVo6dShNHvnoEvC", "name": "go-libp2p-peerstore", - "version": "2.0.13" + "version": "2.0.14" }, { "author": "whyrusleeping", - "hash": "QmS4UBXoQ5QgTJA5pc62egqa5KrQRhsDHhaFHEoGUASsxp", + "hash": "QmUDtgnEr7FFrtK2LQM2dFzTNWghnrApBDcU3iHEJz8eQS", "name": "go-libp2p-transport", - "version": "3.0.21" + "version": "3.0.22" }, { "author": "whyrusleeping", - "hash": "QmNUchfAD5CpjV5QZwVBpKpgNtVD1Vk2k7arGsC1yPnPin", + "hash": "QmayGfkAeV33kHs8rw78wkPq4QZBgbG6LhyZJQ9gfyYG2o", "name": "go-tcp-transport", - "version": "2.0.22" + "version": "2.0.23" }, { "author": "whyrusleeping", @@ -92,33 +92,33 @@ }, { "author": "whyrusleeping", - "hash": "QmNvHv84aH2qZafDuSdKJCQ1cvPZ1kmQmyD4YtzjUHuk9v", + "hash": "QmVnJMgafh5MBYiyqbvDtoCL8pcQvbEGD2k9o9GFpBWPzY", "name": "go-testutil", - "version": "1.2.15" + "version": "1.2.16" }, { "author": "whyrusleeping", - "hash": "QmNgLg1NTw37iWbYPKcyK85YJ9Whs1MkPtJwhfqbNYAyKg", + "hash": "QmZ7cBWUXkyWTMN4qH6NGoyMVs7JugyFChBNP4ZUp5rJHH", "name": "go-libp2p-net", - "version": "3.0.22" + "version": "3.0.23" }, { "author": "whyrusleeping", - "hash": "QmbYN6UmTJn5UUQdi5CTsU86TXVBSrTcRk5UmyA36Qx2J6", + "hash": "QmRN8cMBqfgLgrcaeK6vqUd7HuyvKbNnzaSj4TRBW9XvPQ", "name": "go-libp2p-metrics", - "version": "2.1.11" + "version": "2.1.12" }, { "author": "whyrusleeping", - "hash": "QmaoXrM4Z41PD48JY36YqQGKQpLGjyLA2cKcLsES7YddAq", + "hash": "QmfRHxh8bt4jWLKRhNvR5fn7mFACrQBFLqV4wyoymEExKV", "name": "go-libp2p-host", - "version": "3.0.21" + "version": "3.0.22" }, { "author": "whyrusleeping", - "hash": "QmegQFxhr1J6yZ1vDQuDmJi5jntmj6BL96S11HVtXNCaHb", + "hash": "QmTJCJaS8Cpjc2MkoS32iwr4zMZtbLkaF9GJsUgH1uwtN9", "name": "go-libp2p-swarm", - "version": "3.0.28" + "version": "3.0.29" }, { "author": "whyrusleeping", @@ -128,15 +128,15 @@ }, { "author": "whyrusleeping", - "hash": "QmcD6i5pkzKcy5AiSEU22xmj8MYkkPSaXVvbYo7Wx2hp6H", + "hash": "QmZtiLFhqsDFC3TmmUyWMFJXibURW362UmrXqs3zJim1yb", "name": "go-libp2p-netutil", - "version": "0.4.18" + "version": "0.4.19" }, { "author": "whyrusleeping", - "hash": "QmQLbY1oKd4eHrikizXXwYkxn6yujUNSUMimv3UCaWTSWX", + "hash": "QmYZJzRGPeRpEufmdqXPAcKrpg9gxCnRVRadTn99PH2P77", "name": "go-libp2p-blankhost", - "version": "0.3.21" + "version": "0.3.22" }, { "author": "whyrusleeping", @@ -158,15 +158,15 @@ }, { "author": "whyrusleeping", - "hash": "QmY5Grm8pJdiSSVsYxx4uNRgweY72EmYwuSDbRnbFok3iY", + "hash": "QmPJxxDsX2UbchSHobbYuvz7qnyJTFKvaKMzE2rZWJ4x5B", "name": "go-libp2p-peer", - "version": "3.0.0" + "version": "3.1.0" }, { "author": "vyzo", - "hash": "QmWuMW6UKZMJo9bFFDwnjg8tW3AtKisMHHrXEutQdmJ19N", + "hash": "QmQG8wJtY6KfsTH2tjhaThFPeYVJGm7cmRMxen73ipA4Z5", "name": "go-libp2p-circuit", - "version": "2.3.8" + "version": "2.3.9" }, { "author": "lgierth", @@ -176,9 +176,9 @@ }, { "author": "why", - "hash": "QmSFo2QrMF4M1mKdB291ZqNtsie4NfwXCRdWgDU3inw4Ff", + "hash": "QmebAt96MwXHnbJ5uns6KLm3eSVLPDaaCB4DU7phQUi9a3", "name": "go-libp2p-interface-connmgr", - "version": "0.0.27" + "version": "0.0.28" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmPMuvbzktV1oUfBVf6dc3t2DF5qyB2pKKqhDfAZbcbRg6", + "hash": "QmTPHQHPgmoJtmhMSpVhRPeb23kXGgkUEDKfn9ZndnaQmy", "name": "go-ws-transport", - "version": "2.0.21" + "version": "2.0.22" }, { "author": "stebalien", - "hash": "QmXrweWQgN62WvMcP2EWDpD2qYXiFqenqdaLjUzX7yBqLv", + "hash": "QmaMmPhkoDQBdGVMyoKw2cbLp8p46FXm1UrSr5U8tvYnJk", "name": "go-conn-security-multistream", - "version": "0.1.21" + "version": "0.1.22" }, { "author": "Stebalien", - "hash": "QmVovmja8iXHy1JouPULFdKExUrGutwzgptJZEAFG9rL1t", + "hash": "QmWmeXRTSyWvjPQZgXjXTj2aP74tMSgJwWi1SAvHsvBJVj", "name": "go-conn-security", - "version": "0.1.22" + "version": "0.1.23" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "Qmc7NvDoQaeCaGMuwXz45whL3J21o4Wt7pysztcDZ1VDmn", + "hash": "QmQwvsMzMDTW2K8ySZYgnTVCkzQXVDxmGB5upvVFwdumJV", "name": "go-libp2p-transport-upgrader", - "version": "0.1.22" + "version": "0.1.23" }, { "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", @@ -229,20 +229,20 @@ }, { "author": "vyzo", - "hash": "QmemYsfqwAbyvqwFiApk1GfLKhDkMm8ZQK6fCvzDbaRNyX", + "hash": "QmZttrZ34Aw74CwgGFmaSExBBhECHZfNGMiYpDvTQLCAhM", "name": "go-libp2p-discovery", - "version": "1.0.5" + "version": "1.0.6" }, { "author": "vyzo", - "hash": "QmXmZtMdQokSodDNvPdhDyaVRAjgybvR8dQtuMsNoWv4Lq", + "hash": "QmZgrJk2k14P3zHUAz4hdk1TnU57iaTWEk8fGmFkrafEMX", "name": "go-libp2p-autonat", - "version": "1.0.5" + "version": "1.0.6" }, { - "hash": "QmTiRqrF5zkdZyrdsL5qndG1UbeWi8k8N2pYxCtXWrahR2", + "hash": "QmRjT8Bkut84fHf9nxMQBxGsqLAkqzMdFaemDK7e61dBNZ", "name": "go-libp2p-routing", - "version": "2.7.7" + "version": "2.7.8" } ], "gxVersion": "0.4.0", @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.30" + "version": "6.0.31" } From 96c2f752227dc9a94047a55e69d82eece122d977 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 12 Feb 2019 10:44:03 +0200 Subject: [PATCH 0909/3965] use 6hrs as ttl for routing based advertisements --- p2p/discovery/routing/routing.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index f79726c930..9355ff0805 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -31,8 +31,8 @@ func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Opt return 0, err } - // this is the dht provide validity - return 24 * time.Hour, nil + // the DHT provider record validity is 24hrs, but it is recommnded to republish every 6hrs + return 6 * time.Hour, nil } func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Option) (<-chan pstore.PeerInfo, error) { From 3a993345d072c8729771fd6b2032317f5fc927ef Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 12 Feb 2019 10:56:54 +0200 Subject: [PATCH 0910/3965] use PeerRouting in autorelay to find relay peer addresses --- config/config.go | 2 +- p2p/host/relay/autorelay.go | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/config/config.go b/config/config.go index 58a34cb1ee..4c9342165b 100644 --- a/config/config.go +++ b/config/config.go @@ -208,7 +208,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { if hop { h = relay.NewRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) } else { - h = relay.NewAutoRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + h = relay.NewAutoRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery, router) } } diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 0a40c0c250..d91f31eee1 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -16,6 +16,7 @@ import ( inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" + routing "github.com/libp2p/go-libp2p-routing" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) @@ -44,6 +45,7 @@ func init() { type AutoRelayHost struct { *basic.BasicHost discover discovery.Discoverer + router routing.PeerRouting autonat autonat.AutoNAT addrsF basic.AddrsFactory @@ -54,10 +56,11 @@ type AutoRelayHost struct { addrs []ma.Multiaddr } -func NewAutoRelayHost(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer) *AutoRelayHost { +func NewAutoRelayHost(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer, router routing.PeerRouting) *AutoRelayHost { h := &AutoRelayHost{ BasicHost: bhost, discover: discover, + router: router, addrsF: bhost.AddrsFactory, relays: make(map[peer.ID]pstore.PeerInfo), disconnect: make(chan struct{}, 1), @@ -148,6 +151,16 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { h.mx.Unlock() cctx, cancel := context.WithTimeout(ctx, 60*time.Second) + + if len(pi.Addrs) == 0 { + pi, err = h.router.FindPeer(cctx, pi.ID) + if err != nil { + log.Debugf("error finding relay peer %s: %s", pi.ID, err.Error()) + cancel() + continue + } + } + err = h.Connect(cctx, pi) cancel() if err != nil { From 2d37539e73354531e1aa084b68823775bf739c6f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" Date: Tue, 12 Feb 2019 12:02:01 +0000 Subject: [PATCH 0911/3965] Bump github.com/jackpal/gateway from 1.0.4 to 1.0.5 Bumps [github.com/jackpal/gateway](https://github.com/jackpal/gateway) from 1.0.4 to 1.0.5. - [Release notes](https://github.com/jackpal/gateway/releases) - [Commits](https://github.com/jackpal/gateway/compare/v1.0.4...v1.0.5) Signed-off-by: dependabot[bot] --- go.mod | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/go.mod b/go.mod index d7f4b28787..73255a0b80 100644 --- a/go.mod +++ b/go.mod @@ -2,7 +2,7 @@ module github.com/fd/go-nat require ( github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324 - github.com/jackpal/gateway v1.0.4 + github.com/jackpal/gateway v1.0.5 github.com/jackpal/go-nat-pmp v1.0.1 golang.org/x/net v0.0.0-20180524181706-dfa909b99c79 golang.org/x/text v0.3.0 From 74ff6a1a2a0a615cb14fbceeff0672397f249cf2 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 12 Feb 2019 20:06:39 +0200 Subject: [PATCH 0912/3965] gx publish 6.0.32 --- .gx/lastpubver | 2 +- package.json | 7 +++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 4df79f556d..ed9f932ee9 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.31: QmSgtf5vHyugoxcwMbyNy6bZ9qPDDTJSYEED2GkWjLwitZ +6.0.32: QmXMxMqPgL1G8WMd9WKFdBekXaT6pjP5j8DMKWfS53YNDQ diff --git a/package.json b/package.json index 81ea30dafb..d854acc1f5 100644 --- a/package.json +++ b/package.json @@ -229,9 +229,9 @@ }, { "author": "vyzo", - "hash": "QmZttrZ34Aw74CwgGFmaSExBBhECHZfNGMiYpDvTQLCAhM", + "hash": "QmPFeZGkmZqnUTMbmx1gZ7P1LoBWtu7ozAExkJPB1RaRJe", "name": "go-libp2p-discovery", - "version": "1.0.6" + "version": "1.0.7" }, { "author": "vyzo", @@ -250,6 +250,5 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.31" + "version": "6.0.32" } - From 5f26fd8e0762e9bdad1d90384391033d544c8789 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 14 Feb 2019 12:52:56 +0200 Subject: [PATCH 0913/3965] correctly encode ns to CID --- p2p/discovery/routing/routing.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index 9355ff0805..6a45de4464 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -56,7 +56,7 @@ func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Opt } func nsToCid(ns string) (cid.Cid, error) { - h, err := mh.Encode([]byte(ns), mh.SHA2_256) + h, err := mh.Sum([]byte(ns), mh.SHA2_256, -1) if err != nil { return cid.Undef, err } From 7fa37513fd991ab537cfd536e53a51001f260db4 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 14 Feb 2019 19:26:13 +0200 Subject: [PATCH 0914/3965] increase initial relay advertisement delay to 30s --- p2p/host/relay/relay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index b5db9ce586..bf91c20ee4 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -12,7 +12,7 @@ import ( ) var ( - AdvertiseBootDelay = 5 * time.Second + AdvertiseBootDelay = 30 * time.Second ) // RelayHost is a Host that provides Relay services. From 250048d1962424dcf4713f8e57722ebcd836e8f0 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 14 Feb 2019 19:54:59 +0200 Subject: [PATCH 0915/3965] add a timeout to Provide in routing.Advertise --- p2p/discovery/routing/routing.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index 6a45de4464..0b8369bfca 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -26,7 +26,11 @@ func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Opt return 0, err } - err = d.Provide(ctx, cid, true) + // this context requires a timeout or else the DHT may never find any peers + pctx, cancel := context.WithTimeout(ctx, 60*time.Second) + defer cancel() + + err = d.Provide(pctx, cid, true) if err != nil { return 0, err } From 2662fe1311ea8dda7948a581f79ca836b1ffd205 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 14 Feb 2019 18:08:24 +0000 Subject: [PATCH 0916/3965] clarify comment. --- p2p/discovery/routing/routing.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index 0b8369bfca..3f6a9738e5 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -26,7 +26,9 @@ func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Opt return 0, err } - // this context requires a timeout or else the DHT may never find any peers + // this context requires a timeout; it determines how long the DHT looks for + // closest peers to the key/CID before it goes on to provide the record to them. + // Not setting a timeout here will make the DHT wander forever. pctx, cancel := context.WithTimeout(ctx, 60*time.Second) defer cancel() From b8e47c5dea7dded7ee27a93b28771784421c2ed2 Mon Sep 17 00:00:00 2001 From: iulianpascalau Date: Thu, 14 Feb 2019 20:19:03 +0200 Subject: [PATCH 0917/3965] mock package: - changed GetStreams function from mock_conn.go to call allStreams which has the same functionality but is concurrent safe - changed protocol field from mock_stream to be concurrent safe. Taken the implementation from swarm_stream.go --- p2p/net/mock/mock_conn.go | 6 +----- p2p/net/mock/mock_stream.go | 9 ++++++--- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/p2p/net/mock/mock_conn.go b/p2p/net/mock/mock_conn.go index f4cf10358b..3e0af7268a 100644 --- a/p2p/net/mock/mock_conn.go +++ b/p2p/net/mock/mock_conn.go @@ -121,11 +121,7 @@ func (c *conn) NewStream() (inet.Stream, error) { } func (c *conn) GetStreams() []inet.Stream { - var out []inet.Stream - for e := c.streams.Front(); e != nil; e = e.Next() { - out = append(out, e.Value.(*stream)) - } - return out + return c.allStreams() } // LocalMultiaddr is the Multiaddr on this side diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index 8d01505049..331b8960b8 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -5,6 +5,7 @@ import ( "errors" "io" "net" + "sync/atomic" "time" inet "github.com/libp2p/go-libp2p-net" @@ -24,7 +25,7 @@ type stream struct { writeErr error - protocol protocol.ID + protocol atomic.Value stat inet.Stat } @@ -70,7 +71,9 @@ func (s *stream) Write(p []byte) (n int, err error) { } func (s *stream) Protocol() protocol.ID { - return s.protocol + // Ignore type error. It means that the protocol is unset. + p, _ := s.protocol.Load().(protocol.ID) + return p } func (s *stream) Stat() inet.Stat { @@ -78,7 +81,7 @@ func (s *stream) Stat() inet.Stat { } func (s *stream) SetProtocol(proto protocol.ID) { - s.protocol = proto + s.protocol.Store(proto) } func (s *stream) Close() error { From d80a58df1de8e7590ae55143bc59538e6e4f2454 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 14 Feb 2019 20:40:41 +0200 Subject: [PATCH 0918/3965] gx publish 6.0.33 --- .gx/lastpubver | 2 +- package.json | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index ed9f932ee9..9360d2a36b 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.32: QmXMxMqPgL1G8WMd9WKFdBekXaT6pjP5j8DMKWfS53YNDQ +6.0.33: QmebEmt23jQxrwnqBkFL4qbpE8EnnQunpv5U32LS5ESus1 diff --git a/package.json b/package.json index d854acc1f5..3da2aa719d 100644 --- a/package.json +++ b/package.json @@ -229,9 +229,9 @@ }, { "author": "vyzo", - "hash": "QmPFeZGkmZqnUTMbmx1gZ7P1LoBWtu7ozAExkJPB1RaRJe", + "hash": "Qmezr13Pzko3pxYz4EYMk6mQpJem72TsTVhwR3jGBDEoo7", "name": "go-libp2p-discovery", - "version": "1.0.7" + "version": "1.0.8" }, { "author": "vyzo", @@ -250,5 +250,5 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.32" + "version": "6.0.33" } From b3789be9f912eb6ea1c65709bef9278a0b62de04 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Fri, 15 Feb 2019 15:46:50 +1100 Subject: [PATCH 0919/3965] Improve context use --- p2p/net/swarm/limiter.go | 7 +------ p2p/net/swarm/swarm_listen.go | 4 +++- 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index fa9582bbe6..bd4ab57256 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -27,12 +27,7 @@ type dialJob struct { } func (dj *dialJob) cancelled() bool { - select { - case <-dj.ctx.Done(): - return true - default: - return false - } + return dj.ctx.Err() != nil } func (dj *dialJob) dialTimeout() time.Duration { diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 31b4a04208..7b37d53497 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -74,7 +74,9 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { for { c, err := list.Accept() if err != nil { - log.Warningf("swarm listener accept error: %s", err) + if s.ctx.Err() == nil { + log.Errorf("swarm listener accept error: %s", err) + } return } log.Debugf("swarm listener accepted connection: %s", c) From a65b8ab2d149b529d1d7616565cd2affb40b23a5 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Fri, 15 Feb 2019 15:47:27 +1100 Subject: [PATCH 0920/3965] Reflow comments --- p2p/net/swarm/swarm.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index c8d6fa02d1..cc77e80ec6 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -121,8 +121,8 @@ func (s *Swarm) teardown() error { s.conns.m = nil s.conns.Unlock() - // Lots of go routines but we might as well do this in parallel. We want - // to shut down as fast as possible. + // Lots of goroutines but we might as well do this in parallel. We want to shut down as fast as + // possible. for l := range listeners { go func(l transport.Listener) { @@ -148,8 +148,8 @@ func (s *Swarm) teardown() error { return nil } -// AddAddrFilter adds a multiaddr filter to the set of filters the swarm will -// use to determine which addresses not to dial to. +// AddAddrFilter adds a multiaddr filter to the set of filters the swarm will use to determine which +// addresses not to dial to. func (s *Swarm) AddAddrFilter(f string) error { m, err := mafilter.NewMask(f) if err != nil { From 6ee2ac3e464ddf557f44d8e10584d552ae33f879 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Fri, 15 Feb 2019 15:47:55 +1100 Subject: [PATCH 0921/3965] Do notifications synchronously --- p2p/net/swarm/swarm.go | 14 +------------- p2p/net/swarm/swarm_conn.go | 15 ++++----------- 2 files changed, 5 insertions(+), 24 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index cc77e80ec6..85c5680c6b 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -208,9 +208,6 @@ func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { // * The other will be decremented when Conn.start exits. s.refs.Add(2) - // Take the notification lock before releasing the conns lock to block - // Disconnect notifications until after the Connect notifications done. - c.notifyLk.Lock() s.conns.Unlock() // We have a connection now. Cancel all other in-progress dials. @@ -220,7 +217,6 @@ func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { s.notifyAll(func(f inet.Notifiee) { f.Connected(s, c) }) - c.notifyLk.Unlock() c.start() @@ -438,18 +434,10 @@ func (s *Swarm) Backoff() *DialBackoff { // notifyAll sends a signal to all Notifiees func (s *Swarm) notifyAll(notify func(inet.Notifiee)) { - var wg sync.WaitGroup - s.notifs.RLock() - wg.Add(len(s.notifs.m)) for f := range s.notifs.m { - go func(f inet.Notifiee) { - defer wg.Done() - notify(f) - }(f) + notify(f) } - - wg.Wait() s.notifs.RUnlock() } diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 26a7794d15..4e837cd934 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -65,17 +65,10 @@ func (c *Conn) doClose() { s.Reset() } - // do this in a goroutine to avoid deadlocking if we call close in an open notification. - go func() { - // prevents us from issuing close notifications before finishing the open notifications - c.notifyLk.Lock() - defer c.notifyLk.Unlock() - - c.swarm.notifyAll(func(f inet.Notifiee) { - f.Disconnected(c.swarm, c) - }) - c.swarm.refs.Done() // taken in Swarm.addConn - }() + c.swarm.notifyAll(func(f inet.Notifiee) { + f.Disconnected(c.swarm, c) + }) + c.swarm.refs.Done() // taken in Swarm.addConn } func (c *Conn) removeStream(s *Stream) { From 76866ef9f3fc93854335b6e10be3d821606ec560 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Fri, 15 Feb 2019 15:48:11 +1100 Subject: [PATCH 0922/3965] Fix code format --- p2p/net/swarm/swarm_dial.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 37f1234cc4..56d8420e73 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -275,7 +275,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { logdial["dial"] = "failure" // start off with failure. set to "success" at the end. sk := s.peers.PrivKey(s.local) - logdial["encrypted"] = (sk != nil) // log wether this will be an encrypted dial or not. + logdial["encrypted"] = sk != nil // log whether this will be an encrypted dial or not. if sk == nil { // fine for sk to be nil, just log. log.Debug("Dial not given PrivateKey, so WILL NOT SECURE conn.") From 2327766848ef87d91d3b1a2bfbf0d7cb4e3684dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 15 Feb 2019 18:43:17 +0000 Subject: [PATCH 0923/3965] increment WaitGroups before spawning goroutines. --- p2p/host/peerstore/pstoreds/addr_book.go | 10 ++++++---- p2p/host/peerstore/pstoreds/addr_book_gc.go | 4 ++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 2f71c1e6a6..7c4c9464cf 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -127,7 +127,9 @@ type dsAddrBook struct { flushJobCh chan *addrsRecord cancelFn func() - done sync.WaitGroup + + // controls children goroutine lifetime. + childrenDone sync.WaitGroup } var _ pstore.AddrBook = (*dsAddrBook)(nil) @@ -174,6 +176,7 @@ func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAd return nil, err } + ab.childrenDone.Add(1) go ab.flusher() return ab, nil @@ -181,7 +184,7 @@ func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAd func (ab *dsAddrBook) Close() { ab.cancelFn() - ab.done.Wait() + ab.childrenDone.Wait() } func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { @@ -234,8 +237,7 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs // flusher is a goroutine that takes care of persisting asynchronous flushes to the datastore. func (ab *dsAddrBook) flusher() { - ab.done.Add(1) - defer ab.done.Done() + defer ab.childrenDone.Done() for { select { diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index f8b656f4e9..6d90b07c92 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -82,6 +82,7 @@ func newAddressBookGc(ctx context.Context, ab *dsAddrBook) (*dsAddrBookGc, error // do not start GC timers if purge is disabled; this GC can only be triggered manually. if ab.opts.GCPurgeInterval > 0 { + gc.ab.childrenDone.Add(1) go gc.background() } @@ -90,8 +91,7 @@ func newAddressBookGc(ctx context.Context, ab *dsAddrBook) (*dsAddrBookGc, error // gc prunes expired addresses from the datastore at regular intervals. It should be spawned as a goroutine. func (gc *dsAddrBookGc) background() { - gc.ab.done.Add(1) - defer gc.ab.done.Done() + defer gc.ab.childrenDone.Done() select { case <-time.After(gc.ab.opts.GCInitialDelay): From 8be0831ae03fbbb3a7c7de983723977951c21021 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 15 Feb 2019 18:44:05 +0000 Subject: [PATCH 0924/3965] gc: drain async flush queue with grace period before closing. --- p2p/host/peerstore/pstoreds/addr_book.go | 40 +++++++++++++++++------- 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 7c4c9464cf..892bdc4b6f 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -239,23 +239,39 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs func (ab *dsAddrBook) flusher() { defer ab.childrenDone.Done() + doFlush := func(fj *addrsRecord) { + if cached, ok := ab.cache.Peek(fj.Id.ID); ok { + // Only continue flushing if the record we have in memory is the same as for which the flush + // job was requested. If it's not in memory, it has been evicted and we don't know if we hold + // the latest state or not. Similarly, if it's cached but the pointer is different, it means + // it was evicted and has been reloaded, so we're also uncertain if we hold the latest state. + if pr := cached.(*addrsRecord); pr != fj { + pr.RLock() + pr.flush(ab.ds) + pr.RUnlock() + } + } + } + for { select { case fj := <-ab.flushJobCh: - if cached, ok := ab.cache.Peek(fj.Id.ID); ok { - // Only continue flushing if the record we have in memory is the same as for which the flush - // job was requested. If it's not in memory, it has been evicted and we don't know if we hold - // the latest state or not. Similarly, if it's cached but the pointer is different, it means - // it was evicted and has been reloaded, so we're also uncertain if we hold the latest state. - if pr := cached.(*addrsRecord); pr != fj { - pr.RLock() - pr.flush(ab.ds) - pr.RUnlock() - } - } + doFlush(fj) case <-ab.ctx.Done(): - return + // drain the flush queue, with a grace period of 1 second. + // async flush jobs are not important for consistency by definition (they correspond to expired addrs + // which will anyway expire on the next load). + for { + select { + case fj := <-ab.flushJobCh: + doFlush(fj) + case <-time.After(1 * time.Second): + return + default: + return + } + } } } } From 3b8d83ffdc92bbf19d442f0cc4fc47a59f05d2b6 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Sat, 16 Feb 2019 12:43:11 +1100 Subject: [PATCH 0925/3965] Resolve addrs with P2P instead of IPFS (#539) --- p2p/host/basic/basic_host.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 13452e2f9c..1a89857808 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -413,7 +413,7 @@ func (h *BasicHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { } func (h *BasicHost) resolveAddrs(ctx context.Context, pi pstore.PeerInfo) ([]ma.Multiaddr, error) { - proto := ma.ProtocolWithCode(ma.P_IPFS).Name + proto := ma.ProtocolWithCode(ma.P_P2P).Name p2paddr, err := ma.NewMultiaddr("/" + proto + "/" + pi.ID.Pretty()) if err != nil { return nil, err From 10b3e2a26565ffcc709bd22017b77b00aaab6b6a Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 10 Feb 2019 15:06:26 +0800 Subject: [PATCH 0926/3965] avoid using interface{} when generating certificates --- p2p/security/tls/crypto.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 583fe39c14..23178fa9ab 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -1,6 +1,7 @@ package libp2ptls import ( + "crypto" "crypto/rand" "crypto/tls" "crypto/x509" @@ -104,7 +105,7 @@ func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { } } -func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { +func keyToCertificate(sk ic.PrivKey) (crypto.PrivateKey, *x509.Certificate, error) { sn, err := rand.Int(rand.Reader, big.NewInt(1<<62)) if err != nil { return nil, nil, err @@ -115,7 +116,8 @@ func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { NotAfter: time.Now().Add(certValidityPeriod), } - var publicKey, privateKey interface{} + var privateKey crypto.PrivateKey + var publicKey crypto.PublicKey keyBytes, err := sk.Bytes() if err != nil { return nil, nil, err From 9f8a3248114d079de1f851ed9bbfaa3b69710a03 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 17 Feb 2019 11:15:42 +0800 Subject: [PATCH 0927/3965] remove unneeded marshaling / unmarshaling when generating cert chain --- p2p/security/tls/crypto.go | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 23178fa9ab..eb817eef35 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -10,7 +10,6 @@ import ( "math/big" "time" - "github.com/gogo/protobuf/proto" ic "github.com/libp2p/go-libp2p-crypto" pb "github.com/libp2p/go-libp2p-crypto/pb" peer "github.com/libp2p/go-libp2p-peer" @@ -118,24 +117,20 @@ func keyToCertificate(sk ic.PrivKey) (crypto.PrivateKey, *x509.Certificate, erro var privateKey crypto.PrivateKey var publicKey crypto.PublicKey - keyBytes, err := sk.Bytes() + raw, err := sk.Raw() if err != nil { return nil, nil, err } - pbmes := new(pb.PrivateKey) - if err := proto.Unmarshal(keyBytes, pbmes); err != nil { - return nil, nil, err - } - switch pbmes.GetType() { + switch sk.Type() { case pb.KeyType_RSA: - k, err := x509.ParsePKCS1PrivateKey(pbmes.GetData()) + k, err := x509.ParsePKCS1PrivateKey(raw) if err != nil { return nil, nil, err } publicKey = &k.PublicKey privateKey = k case pb.KeyType_ECDSA: - k, err := x509.ParseECPrivateKey(pbmes.GetData()) + k, err := x509.ParseECPrivateKey(raw) if err != nil { return nil, nil, err } From 5ecc2f97d3d4f75522bdcdcf12ef3778b6363ec3 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 10 Feb 2019 11:12:10 +0800 Subject: [PATCH 0928/3965] drop support for Go 1.10 TLS 1.3 will require Go >= 1.12, so we don't need any fixes that were specific to Go 1.10 any more. --- p2p/security/tls/transport.go | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 5a75296f1a..3f25989210 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -43,21 +43,16 @@ var _ cs.Transport = &Transport{} // SecureInbound runs the TLS handshake as a server. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Conn, error) { serv := tls.Server(insecure, t.identity.Config) - return t.handshake(ctx, insecure, serv) + return t.handshake(ctx, serv) } // SecureOutbound runs the TLS handshake as a client. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { cl := tls.Client(insecure, t.identity.ConfigForPeer(p)) - return t.handshake(ctx, insecure, cl) + return t.handshake(ctx, cl) } -func (t *Transport) handshake( - ctx context.Context, - // in Go 1.10, we need to close the underlying net.Conn - // in Go 1.11 this was fixed, and tls.Conn.Close() works as well - insecure net.Conn, - tlsConn *tls.Conn, +func (t *Transport) handshake(ctx context.Context, tlsConn *tls.Conn, ) (cs.Conn, error) { // There's no way to pass a context to tls.Conn.Handshake(). // See https://github.com/golang/go/issues/18482. @@ -68,7 +63,7 @@ func (t *Transport) handshake( select { case <-done: case <-ctx.Done(): - insecure.Close() + tlsConn.Close() } }() From 490981871c7882d07e589a3f466ac8dd6f2f0675 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 10 Feb 2019 15:25:29 +0800 Subject: [PATCH 0929/3965] improve logging in tests --- p2p/security/tls/transport_test.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index d06326d90f..c3d92196d2 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -26,18 +26,19 @@ var _ = Describe("Transport", func() { createPeer := func() (peer.ID, ic.PrivKey) { var priv ic.PrivKey if mrand.Int()%2 == 0 { - fmt.Fprintln(GinkgoWriter, " using an ECDSA key") + fmt.Fprintf(GinkgoWriter, " using an ECDSA key: ") var err error priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) Expect(err).ToNot(HaveOccurred()) } else { - fmt.Fprintln(GinkgoWriter, " using an RSA key") + fmt.Fprintf(GinkgoWriter, " using an RSA key: ") var err error priv, _, err = ic.GenerateRSAKeyPair(1024, rand.Reader) Expect(err).ToNot(HaveOccurred()) } id, err := peer.IDFromPrivateKey(priv) Expect(err).ToNot(HaveOccurred()) + fmt.Fprintln(GinkgoWriter, id.Pretty()) return id, priv } From 0b45a8d2fb671404be4248f55226e4eefb4e97a1 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 10 Feb 2019 16:23:43 +0800 Subject: [PATCH 0930/3965] make sure to close the connection if the context is already canceled --- p2p/security/tls/transport.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 3f25989210..835e8f61e4 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -57,6 +57,11 @@ func (t *Transport) handshake(ctx context.Context, tlsConn *tls.Conn, // There's no way to pass a context to tls.Conn.Handshake(). // See https://github.com/golang/go/issues/18482. // Close the connection instead. + select { + case <-ctx.Done(): + tlsConn.Close() + default: + } done := make(chan struct{}) defer close(done) go func() { From 1c09b025c071bc26803c041add92f8b8f1e02d78 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 10 Feb 2019 15:51:59 +0800 Subject: [PATCH 0931/3965] switch to TLS 1.3 TLS 1.3 support was recently made opt-in in Go 1.12, so we need to explicitly enable it. --- p2p/security/tls/crypto.go | 1 + p2p/security/tls/transport.go | 13 +++++++++++++ p2p/security/tls/transport_test.go | 23 ++++++++--------------- 3 files changed, 22 insertions(+), 15 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index eb817eef35..eb1b7a12e0 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -72,6 +72,7 @@ func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { return nil, err } return &tls.Config{ + MinVersion: tls.VersionTLS13, InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. ClientAuth: tls.RequireAnyClientCert, Certificates: []tls.Certificate{{ diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 835e8f61e4..db820cf0a5 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -4,12 +4,19 @@ import ( "context" "crypto/tls" "net" + "os" cs "github.com/libp2p/go-conn-security" ci "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" ) +// TLS 1.3 is opt-in in Go 1.12 +// Activate it by setting the tls13 GODEBUG flag. +func init() { + os.Setenv("GODEBUG", os.Getenv("GODEBUG")+",tls13=1") +} + // ID is the protocol ID (used when negotiating with multistream) const ID = "/tls/1.0.0" @@ -47,6 +54,12 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Co } // SecureOutbound runs the TLS handshake as a client. +// Note that SecureOutbound will not return an error if the server doesn't +// accept the certificate. This is due to the fact that in TLS 1.3, the client +// sends its certificate and the ClientFinished in the same flight, and can send +// application data immediately afterwards. +// If the handshake fails, the server will close the connection. The client will +// notice this after 1 RTT when calling Read. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { cl := tls.Client(insecure, t.identity.ConfigForPeer(p)) return t.handshake(ctx, cl) diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index c3d92196d2..3e281e3e1b 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -66,7 +66,7 @@ var _ = Describe("Transport", func() { Expect(err).ToNot(HaveOccurred()) identity.Config.Certificates[0].PrivateKey = key case *ecdsa.PrivateKey: - key, err := ecdsa.GenerateKey(elliptic.P224(), rand.Reader) + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) Expect(err).ToNot(HaveOccurred()) identity.Config.Certificates[0].PrivateKey = key default: @@ -195,17 +195,14 @@ var _ = Describe("Transport", func() { go func() { defer GinkgoRecover() _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(Or( - ContainSubstring("crypto/rsa: verification error"), - ContainSubstring("ECDSA verification failure"), - )) + Expect(err).To(MatchError("tls: invalid certificate signature")) close(done) }() - _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("tls: bad certificate")) + conn, err := clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) + Expect(err).ToNot(HaveOccurred()) + _, err = conn.Read([]byte{0}) + Expect(err).To(MatchError("remote error: tls: error decrypting message")) Eventually(done).Should(BeClosed()) }) @@ -223,16 +220,12 @@ var _ = Describe("Transport", func() { defer GinkgoRecover() _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) Expect(err).To(HaveOccurred()) - // TLS returns a weird error here: "remote error: tls: unexpected message" + Expect(err.Error()).To(ContainSubstring("remote error: tls:")) close(done) }() _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(Or( - ContainSubstring("crypto/rsa: verification error"), - ContainSubstring("ECDSA verification failure"), - )) + Expect(err).To(MatchError("tls: invalid certificate signature")) Eventually(done).Should(BeClosed()) }) }) From a9adb0fb8ef9c7337de410a1e5f98dc5f6bdb8a4 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 18 Feb 2019 15:44:01 +0100 Subject: [PATCH 0932/3965] gx publish 2.3.10 --- .../circuitv1-deprecated/pb/relay.pb.go | 163 +++++++++++------- 1 file changed, 101 insertions(+), 62 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go index 5a98753bab..ca165ee60d 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go @@ -3,13 +3,13 @@ package relay_pb -import proto "github.com/gogo/protobuf/proto" -import fmt "fmt" -import math "math" - -import github_com_gogo_protobuf_proto "github.com/gogo/protobuf/proto" - -import io "io" +import ( + fmt "fmt" + github_com_gogo_protobuf_proto "github.com/gogo/protobuf/proto" + proto "github.com/gogo/protobuf/proto" + io "io" + math "math" +) // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal @@ -61,6 +61,7 @@ var CircuitRelay_Status_name = map[int32]string{ 390: "STOP_RELAY_REFUSED", 400: "MALFORMED_MESSAGE", } + var CircuitRelay_Status_value = map[string]int32{ "SUCCESS": 100, "HOP_SRC_ADDR_TOO_LONG": 220, @@ -85,9 +86,11 @@ func (x CircuitRelay_Status) Enum() *CircuitRelay_Status { *p = x return p } + func (x CircuitRelay_Status) String() string { return proto.EnumName(CircuitRelay_Status_name, int32(x)) } + func (x *CircuitRelay_Status) UnmarshalJSON(data []byte) error { value, err := proto.UnmarshalJSONEnum(CircuitRelay_Status_value, data, "CircuitRelay_Status") if err != nil { @@ -96,8 +99,9 @@ func (x *CircuitRelay_Status) UnmarshalJSON(data []byte) error { *x = CircuitRelay_Status(value) return nil } + func (CircuitRelay_Status) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_relay_cd6009b961688bf5, []int{0, 0} + return fileDescriptor_9f69a7d5a802d584, []int{0, 0} } type CircuitRelay_Type int32 @@ -115,6 +119,7 @@ var CircuitRelay_Type_name = map[int32]string{ 3: "STATUS", 4: "CAN_HOP", } + var CircuitRelay_Type_value = map[string]int32{ "HOP": 1, "STOP": 2, @@ -127,9 +132,11 @@ func (x CircuitRelay_Type) Enum() *CircuitRelay_Type { *p = x return p } + func (x CircuitRelay_Type) String() string { return proto.EnumName(CircuitRelay_Type_name, int32(x)) } + func (x *CircuitRelay_Type) UnmarshalJSON(data []byte) error { value, err := proto.UnmarshalJSONEnum(CircuitRelay_Type_value, data, "CircuitRelay_Type") if err != nil { @@ -138,8 +145,9 @@ func (x *CircuitRelay_Type) UnmarshalJSON(data []byte) error { *x = CircuitRelay_Type(value) return nil } + func (CircuitRelay_Type) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_relay_cd6009b961688bf5, []int{0, 1} + return fileDescriptor_9f69a7d5a802d584, []int{0, 1} } type CircuitRelay struct { @@ -156,7 +164,7 @@ func (m *CircuitRelay) Reset() { *m = CircuitRelay{} } func (m *CircuitRelay) String() string { return proto.CompactTextString(m) } func (*CircuitRelay) ProtoMessage() {} func (*CircuitRelay) Descriptor() ([]byte, []int) { - return fileDescriptor_relay_cd6009b961688bf5, []int{0} + return fileDescriptor_9f69a7d5a802d584, []int{0} } func (m *CircuitRelay) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -173,8 +181,8 @@ func (m *CircuitRelay) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) return b[:n], nil } } -func (dst *CircuitRelay) XXX_Merge(src proto.Message) { - xxx_messageInfo_CircuitRelay.Merge(dst, src) +func (m *CircuitRelay) XXX_Merge(src proto.Message) { + xxx_messageInfo_CircuitRelay.Merge(m, src) } func (m *CircuitRelay) XXX_Size() int { return m.Size() @@ -225,7 +233,7 @@ func (m *CircuitRelay_Peer) Reset() { *m = CircuitRelay_Peer{} } func (m *CircuitRelay_Peer) String() string { return proto.CompactTextString(m) } func (*CircuitRelay_Peer) ProtoMessage() {} func (*CircuitRelay_Peer) Descriptor() ([]byte, []int) { - return fileDescriptor_relay_cd6009b961688bf5, []int{0, 0} + return fileDescriptor_9f69a7d5a802d584, []int{0, 0} } func (m *CircuitRelay_Peer) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -242,8 +250,8 @@ func (m *CircuitRelay_Peer) XXX_Marshal(b []byte, deterministic bool) ([]byte, e return b[:n], nil } } -func (dst *CircuitRelay_Peer) XXX_Merge(src proto.Message) { - xxx_messageInfo_CircuitRelay_Peer.Merge(dst, src) +func (m *CircuitRelay_Peer) XXX_Merge(src proto.Message) { + xxx_messageInfo_CircuitRelay_Peer.Merge(m, src) } func (m *CircuitRelay_Peer) XXX_Size() int { return m.Size() @@ -269,11 +277,48 @@ func (m *CircuitRelay_Peer) GetAddrs() [][]byte { } func init() { - proto.RegisterType((*CircuitRelay)(nil), "relay.pb.CircuitRelay") - proto.RegisterType((*CircuitRelay_Peer)(nil), "relay.pb.CircuitRelay.Peer") proto.RegisterEnum("relay.pb.CircuitRelay_Status", CircuitRelay_Status_name, CircuitRelay_Status_value) proto.RegisterEnum("relay.pb.CircuitRelay_Type", CircuitRelay_Type_name, CircuitRelay_Type_value) + proto.RegisterType((*CircuitRelay)(nil), "relay.pb.CircuitRelay") + proto.RegisterType((*CircuitRelay_Peer)(nil), "relay.pb.CircuitRelay.Peer") } + +func init() { proto.RegisterFile("relay.proto", fileDescriptor_9f69a7d5a802d584) } + +var fileDescriptor_9f69a7d5a802d584 = []byte{ + // 473 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x92, 0x4f, 0x6f, 0xd3, 0x3e, + 0x18, 0xc7, 0x65, 0x27, 0xbf, 0x76, 0x7a, 0x5a, 0x4d, 0xfe, 0x59, 0x63, 0x64, 0x9d, 0x56, 0xaa, + 0x9e, 0x7a, 0x40, 0x45, 0x4c, 0xe2, 0x05, 0x98, 0xc4, 0xdd, 0x2a, 0xd2, 0x38, 0xb2, 0x5d, 0x24, + 0x4e, 0x56, 0x69, 0x72, 0xa8, 0x84, 0xd4, 0x2a, 0xcd, 0x0e, 0xbd, 0xc3, 0xb8, 0x21, 0x8e, 0xbc, + 0x1c, 0xe0, 0xc4, 0x91, 0x17, 0xc0, 0x3f, 0xf5, 0x65, 0xc0, 0x05, 0xd9, 0x5d, 0x33, 0x44, 0x37, + 0x89, 0xa3, 0x9f, 0xef, 0xe7, 0xe3, 0x3c, 0xf9, 0x26, 0xd0, 0x28, 0xf2, 0x17, 0x93, 0x55, 0x7f, + 0x51, 0xcc, 0xcb, 0x39, 0xdd, 0xbb, 0x3a, 0x3c, 0xef, 0xbe, 0xae, 0x41, 0x33, 0x9c, 0x15, 0xd3, + 0x8b, 0x59, 0x29, 0xed, 0x8c, 0x3e, 0x00, 0xbf, 0x5c, 0x2d, 0xf2, 0x00, 0x75, 0x50, 0x6f, 0xff, + 0xf4, 0xb8, 0xbf, 0x25, 0xfb, 0x7f, 0x52, 0x7d, 0xbd, 0x5a, 0xe4, 0xd2, 0x81, 0xf4, 0x11, 0xd4, + 0x97, 0xc5, 0x34, 0xcd, 0xf3, 0x22, 0xc0, 0x1d, 0xd4, 0x6b, 0xdc, 0xea, 0x58, 0x44, 0x6e, 0x59, + 0xab, 0x65, 0xcb, 0xd2, 0x69, 0xde, 0x3f, 0x68, 0x57, 0x2c, 0x7d, 0x08, 0xfe, 0x74, 0x9e, 0xe5, + 0x81, 0xef, 0xd6, 0x3b, 0xb9, 0xc5, 0x51, 0xe5, 0xa4, 0xbc, 0x58, 0x4a, 0x87, 0xb6, 0xee, 0x83, + 0xef, 0xd4, 0x7d, 0xc0, 0xb3, 0x2c, 0x40, 0x1d, 0xdc, 0x6b, 0x4a, 0x3c, 0xcb, 0xe8, 0x01, 0xfc, + 0x37, 0xc9, 0xb2, 0x62, 0x19, 0xe0, 0x8e, 0xd7, 0x6b, 0xca, 0xcd, 0xa1, 0xfb, 0xd1, 0x83, 0xda, + 0x46, 0xa7, 0x0d, 0xa8, 0xab, 0x71, 0x18, 0x72, 0xa5, 0x48, 0x46, 0x5b, 0x70, 0xe7, 0x5c, 0xa4, + 0x46, 0xc9, 0xd0, 0xb0, 0x28, 0x92, 0x46, 0x0b, 0x61, 0x62, 0x91, 0x9c, 0x91, 0x2f, 0x68, 0x9b, + 0x45, 0x4a, 0xff, 0x95, 0x7d, 0x45, 0xb4, 0x0d, 0x47, 0x5b, 0x6f, 0x34, 0x8e, 0xf5, 0xd0, 0x01, + 0xc3, 0xe4, 0x29, 0x8b, 0x87, 0x11, 0xf9, 0x59, 0xe5, 0xd6, 0xdd, 0xcd, 0x7f, 0x21, 0x7a, 0x17, + 0xa8, 0xcd, 0x13, 0x61, 0x42, 0x91, 0x24, 0x46, 0x0b, 0x8b, 0x92, 0x97, 0x98, 0x1e, 0xc2, 0xff, + 0x36, 0x08, 0x59, 0xa2, 0x4d, 0x34, 0x64, 0xb1, 0x9b, 0xbf, 0xc2, 0xf4, 0x04, 0x82, 0x6a, 0x2e, + 0x52, 0x9e, 0xb8, 0xab, 0x95, 0x96, 0x9c, 0x8d, 0xc8, 0x25, 0xa6, 0x47, 0x70, 0x50, 0xc5, 0x2a, + 0xe5, 0xec, 0x89, 0x91, 0x3c, 0x66, 0xcf, 0xc8, 0x1b, 0x4c, 0x8f, 0xe1, 0xb0, 0x8a, 0xdc, 0xd0, + 0x3e, 0x4d, 0xf1, 0x78, 0x40, 0xde, 0xb9, 0x50, 0xe9, 0x1b, 0x0b, 0x78, 0x7f, 0x1d, 0xee, 0x36, + 0xf0, 0x01, 0xd3, 0x7b, 0xd0, 0xaa, 0xcc, 0xdd, 0x57, 0xfc, 0x76, 0x0d, 0xdc, 0xdc, 0xc1, 0x77, + 0x6c, 0x3b, 0x70, 0xc0, 0x66, 0x29, 0xc9, 0x07, 0x63, 0xc5, 0x23, 0x72, 0xe9, 0xd9, 0x0e, 0x46, + 0x2c, 0x1e, 0x08, 0x39, 0xe2, 0x91, 0x19, 0x71, 0xa5, 0xd8, 0x19, 0x27, 0x6f, 0xbd, 0xee, 0x29, + 0xf8, 0xf6, 0x0f, 0xa5, 0x75, 0xf0, 0xce, 0x45, 0x4a, 0x10, 0xdd, 0x03, 0xdf, 0xde, 0x40, 0x30, + 0x05, 0xa8, 0x29, 0xcd, 0xf4, 0x58, 0x11, 0xcf, 0x7e, 0xe0, 0x90, 0x25, 0xc6, 0x22, 0xfe, 0xe3, + 0xe6, 0xa7, 0x75, 0x1b, 0x7d, 0x5e, 0xb7, 0xd1, 0x8f, 0x75, 0x1b, 0xfd, 0x0e, 0x00, 0x00, 0xff, + 0xff, 0x6b, 0x22, 0x33, 0xbb, 0x2f, 0x03, 0x00, 0x00, +} + func (m *CircuitRelay) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) @@ -372,6 +417,9 @@ func encodeVarintRelay(dAtA []byte, offset int, v uint64) int { return offset + 1 } func (m *CircuitRelay) Size() (n int) { + if m == nil { + return 0 + } var l int _ = l if m.Type != nil { @@ -395,6 +443,9 @@ func (m *CircuitRelay) Size() (n int) { } func (m *CircuitRelay_Peer) Size() (n int) { + if m == nil { + return 0 + } var l int _ = l if m.Id != nil { @@ -441,7 +492,7 @@ func (m *CircuitRelay) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -469,7 +520,7 @@ func (m *CircuitRelay) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - v |= (CircuitRelay_Type(b) & 0x7F) << shift + v |= CircuitRelay_Type(b&0x7F) << shift if b < 0x80 { break } @@ -489,7 +540,7 @@ func (m *CircuitRelay) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - msglen |= (int(b) & 0x7F) << shift + msglen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -498,6 +549,9 @@ func (m *CircuitRelay) Unmarshal(dAtA []byte) error { return ErrInvalidLengthRelay } postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthRelay + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -522,7 +576,7 @@ func (m *CircuitRelay) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - msglen |= (int(b) & 0x7F) << shift + msglen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -531,6 +585,9 @@ func (m *CircuitRelay) Unmarshal(dAtA []byte) error { return ErrInvalidLengthRelay } postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthRelay + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -555,7 +612,7 @@ func (m *CircuitRelay) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - v |= (CircuitRelay_Status(b) & 0x7F) << shift + v |= CircuitRelay_Status(b&0x7F) << shift if b < 0x80 { break } @@ -570,6 +627,9 @@ func (m *CircuitRelay) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthRelay } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthRelay + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } @@ -599,7 +659,7 @@ func (m *CircuitRelay_Peer) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -627,7 +687,7 @@ func (m *CircuitRelay_Peer) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -636,6 +696,9 @@ func (m *CircuitRelay_Peer) Unmarshal(dAtA []byte) error { return ErrInvalidLengthRelay } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthRelay + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -659,7 +722,7 @@ func (m *CircuitRelay_Peer) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -668,6 +731,9 @@ func (m *CircuitRelay_Peer) Unmarshal(dAtA []byte) error { return ErrInvalidLengthRelay } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthRelay + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -683,6 +749,9 @@ func (m *CircuitRelay_Peer) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthRelay } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthRelay + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } @@ -753,10 +822,13 @@ func skipRelay(dAtA []byte) (n int, err error) { break } } - iNdEx += length if length < 0 { return 0, ErrInvalidLengthRelay } + iNdEx += length + if iNdEx < 0 { + return 0, ErrInvalidLengthRelay + } return iNdEx, nil case 3: for { @@ -785,6 +857,9 @@ func skipRelay(dAtA []byte) (n int, err error) { return 0, err } iNdEx = start + next + if iNdEx < 0 { + return 0, ErrInvalidLengthRelay + } } return iNdEx, nil case 4: @@ -803,39 +878,3 @@ var ( ErrInvalidLengthRelay = fmt.Errorf("proto: negative length found during unmarshaling") ErrIntOverflowRelay = fmt.Errorf("proto: integer overflow") ) - -func init() { proto.RegisterFile("relay.proto", fileDescriptor_relay_cd6009b961688bf5) } - -var fileDescriptor_relay_cd6009b961688bf5 = []byte{ - // 473 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x92, 0x4f, 0x6f, 0xd3, 0x3e, - 0x18, 0xc7, 0x65, 0x27, 0xbf, 0x76, 0x7a, 0x5a, 0x4d, 0xfe, 0x59, 0x63, 0x64, 0x9d, 0x56, 0xaa, - 0x9e, 0x7a, 0x40, 0x45, 0x4c, 0xe2, 0x05, 0x98, 0xc4, 0xdd, 0x2a, 0xd2, 0x38, 0xb2, 0x5d, 0x24, - 0x4e, 0x56, 0x69, 0x72, 0xa8, 0x84, 0xd4, 0x2a, 0xcd, 0x0e, 0xbd, 0xc3, 0xb8, 0x21, 0x8e, 0xbc, - 0x1c, 0xe0, 0xc4, 0x91, 0x17, 0xc0, 0x3f, 0xf5, 0x65, 0xc0, 0x05, 0xd9, 0x5d, 0x33, 0x44, 0x37, - 0x89, 0xa3, 0x9f, 0xef, 0xe7, 0xe3, 0x3c, 0xf9, 0x26, 0xd0, 0x28, 0xf2, 0x17, 0x93, 0x55, 0x7f, - 0x51, 0xcc, 0xcb, 0x39, 0xdd, 0xbb, 0x3a, 0x3c, 0xef, 0xbe, 0xae, 0x41, 0x33, 0x9c, 0x15, 0xd3, - 0x8b, 0x59, 0x29, 0xed, 0x8c, 0x3e, 0x00, 0xbf, 0x5c, 0x2d, 0xf2, 0x00, 0x75, 0x50, 0x6f, 0xff, - 0xf4, 0xb8, 0xbf, 0x25, 0xfb, 0x7f, 0x52, 0x7d, 0xbd, 0x5a, 0xe4, 0xd2, 0x81, 0xf4, 0x11, 0xd4, - 0x97, 0xc5, 0x34, 0xcd, 0xf3, 0x22, 0xc0, 0x1d, 0xd4, 0x6b, 0xdc, 0xea, 0x58, 0x44, 0x6e, 0x59, - 0xab, 0x65, 0xcb, 0xd2, 0x69, 0xde, 0x3f, 0x68, 0x57, 0x2c, 0x7d, 0x08, 0xfe, 0x74, 0x9e, 0xe5, - 0x81, 0xef, 0xd6, 0x3b, 0xb9, 0xc5, 0x51, 0xe5, 0xa4, 0xbc, 0x58, 0x4a, 0x87, 0xb6, 0xee, 0x83, - 0xef, 0xd4, 0x7d, 0xc0, 0xb3, 0x2c, 0x40, 0x1d, 0xdc, 0x6b, 0x4a, 0x3c, 0xcb, 0xe8, 0x01, 0xfc, - 0x37, 0xc9, 0xb2, 0x62, 0x19, 0xe0, 0x8e, 0xd7, 0x6b, 0xca, 0xcd, 0xa1, 0xfb, 0xd1, 0x83, 0xda, - 0x46, 0xa7, 0x0d, 0xa8, 0xab, 0x71, 0x18, 0x72, 0xa5, 0x48, 0x46, 0x5b, 0x70, 0xe7, 0x5c, 0xa4, - 0x46, 0xc9, 0xd0, 0xb0, 0x28, 0x92, 0x46, 0x0b, 0x61, 0x62, 0x91, 0x9c, 0x91, 0x2f, 0x68, 0x9b, - 0x45, 0x4a, 0xff, 0x95, 0x7d, 0x45, 0xb4, 0x0d, 0x47, 0x5b, 0x6f, 0x34, 0x8e, 0xf5, 0xd0, 0x01, - 0xc3, 0xe4, 0x29, 0x8b, 0x87, 0x11, 0xf9, 0x59, 0xe5, 0xd6, 0xdd, 0xcd, 0x7f, 0x21, 0x7a, 0x17, - 0xa8, 0xcd, 0x13, 0x61, 0x42, 0x91, 0x24, 0x46, 0x0b, 0x8b, 0x92, 0x97, 0x98, 0x1e, 0xc2, 0xff, - 0x36, 0x08, 0x59, 0xa2, 0x4d, 0x34, 0x64, 0xb1, 0x9b, 0xbf, 0xc2, 0xf4, 0x04, 0x82, 0x6a, 0x2e, - 0x52, 0x9e, 0xb8, 0xab, 0x95, 0x96, 0x9c, 0x8d, 0xc8, 0x25, 0xa6, 0x47, 0x70, 0x50, 0xc5, 0x2a, - 0xe5, 0xec, 0x89, 0x91, 0x3c, 0x66, 0xcf, 0xc8, 0x1b, 0x4c, 0x8f, 0xe1, 0xb0, 0x8a, 0xdc, 0xd0, - 0x3e, 0x4d, 0xf1, 0x78, 0x40, 0xde, 0xb9, 0x50, 0xe9, 0x1b, 0x0b, 0x78, 0x7f, 0x1d, 0xee, 0x36, - 0xf0, 0x01, 0xd3, 0x7b, 0xd0, 0xaa, 0xcc, 0xdd, 0x57, 0xfc, 0x76, 0x0d, 0xdc, 0xdc, 0xc1, 0x77, - 0x6c, 0x3b, 0x70, 0xc0, 0x66, 0x29, 0xc9, 0x07, 0x63, 0xc5, 0x23, 0x72, 0xe9, 0xd9, 0x0e, 0x46, - 0x2c, 0x1e, 0x08, 0x39, 0xe2, 0x91, 0x19, 0x71, 0xa5, 0xd8, 0x19, 0x27, 0x6f, 0xbd, 0xee, 0x29, - 0xf8, 0xf6, 0x0f, 0xa5, 0x75, 0xf0, 0xce, 0x45, 0x4a, 0x10, 0xdd, 0x03, 0xdf, 0xde, 0x40, 0x30, - 0x05, 0xa8, 0x29, 0xcd, 0xf4, 0x58, 0x11, 0xcf, 0x7e, 0xe0, 0x90, 0x25, 0xc6, 0x22, 0xfe, 0xe3, - 0xe6, 0xa7, 0x75, 0x1b, 0x7d, 0x5e, 0xb7, 0xd1, 0x8f, 0x75, 0x1b, 0xfd, 0x0e, 0x00, 0x00, 0xff, - 0xff, 0x6b, 0x22, 0x33, 0xbb, 0x2f, 0x03, 0x00, 0x00, -} From fb9be136d83e39b977253eda87a12db4afc01c3d Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 18 Feb 2019 15:53:06 +0100 Subject: [PATCH 0933/3965] gx publish 1.0.7 --- p2p/host/autonat/pb/Makefile | 6 + p2p/host/autonat/pb/autonat.pb.go | 191 ++++++++++++++++++++---------- 2 files changed, 132 insertions(+), 65 deletions(-) create mode 100644 p2p/host/autonat/pb/Makefile diff --git a/p2p/host/autonat/pb/Makefile b/p2p/host/autonat/pb/Makefile new file mode 100644 index 0000000000..dd21e878f8 --- /dev/null +++ b/p2p/host/autonat/pb/Makefile @@ -0,0 +1,6 @@ +pbgos := $(patsubst %.proto,%.pb.go,$(wildcard *.proto)) + +all: $(pbgos) + +%.pb.go: %.proto + protoc --gogofast_out=. --proto_path=$(GOPATH)/src:. $< diff --git a/p2p/host/autonat/pb/autonat.pb.go b/p2p/host/autonat/pb/autonat.pb.go index 3f2db2e1a1..3617b4615a 100644 --- a/p2p/host/autonat/pb/autonat.pb.go +++ b/p2p/host/autonat/pb/autonat.pb.go @@ -3,11 +3,12 @@ package autonat_pb -import proto "github.com/gogo/protobuf/proto" -import fmt "fmt" -import math "math" - -import io "io" +import ( + fmt "fmt" + proto "github.com/gogo/protobuf/proto" + io "io" + math "math" +) // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal @@ -31,6 +32,7 @@ var Message_MessageType_name = map[int32]string{ 0: "DIAL", 1: "DIAL_RESPONSE", } + var Message_MessageType_value = map[string]int32{ "DIAL": 0, "DIAL_RESPONSE": 1, @@ -41,9 +43,11 @@ func (x Message_MessageType) Enum() *Message_MessageType { *p = x return p } + func (x Message_MessageType) String() string { return proto.EnumName(Message_MessageType_name, int32(x)) } + func (x *Message_MessageType) UnmarshalJSON(data []byte) error { value, err := proto.UnmarshalJSONEnum(Message_MessageType_value, data, "Message_MessageType") if err != nil { @@ -52,8 +56,9 @@ func (x *Message_MessageType) UnmarshalJSON(data []byte) error { *x = Message_MessageType(value) return nil } + func (Message_MessageType) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 0} + return fileDescriptor_a04e278ef61ac07a, []int{0, 0} } type Message_ResponseStatus int32 @@ -73,6 +78,7 @@ var Message_ResponseStatus_name = map[int32]string{ 200: "E_BAD_REQUEST", 300: "E_INTERNAL_ERROR", } + var Message_ResponseStatus_value = map[string]int32{ "OK": 0, "E_DIAL_ERROR": 100, @@ -86,9 +92,11 @@ func (x Message_ResponseStatus) Enum() *Message_ResponseStatus { *p = x return p } + func (x Message_ResponseStatus) String() string { return proto.EnumName(Message_ResponseStatus_name, int32(x)) } + func (x *Message_ResponseStatus) UnmarshalJSON(data []byte) error { value, err := proto.UnmarshalJSONEnum(Message_ResponseStatus_value, data, "Message_ResponseStatus") if err != nil { @@ -97,8 +105,9 @@ func (x *Message_ResponseStatus) UnmarshalJSON(data []byte) error { *x = Message_ResponseStatus(value) return nil } + func (Message_ResponseStatus) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 1} + return fileDescriptor_a04e278ef61ac07a, []int{0, 1} } type Message struct { @@ -114,7 +123,7 @@ func (m *Message) Reset() { *m = Message{} } func (m *Message) String() string { return proto.CompactTextString(m) } func (*Message) ProtoMessage() {} func (*Message) Descriptor() ([]byte, []int) { - return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0} + return fileDescriptor_a04e278ef61ac07a, []int{0} } func (m *Message) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -131,8 +140,8 @@ func (m *Message) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { return b[:n], nil } } -func (dst *Message) XXX_Merge(src proto.Message) { - xxx_messageInfo_Message.Merge(dst, src) +func (m *Message) XXX_Merge(src proto.Message) { + xxx_messageInfo_Message.Merge(m, src) } func (m *Message) XXX_Size() int { return m.Size() @@ -176,7 +185,7 @@ func (m *Message_PeerInfo) Reset() { *m = Message_PeerInfo{} } func (m *Message_PeerInfo) String() string { return proto.CompactTextString(m) } func (*Message_PeerInfo) ProtoMessage() {} func (*Message_PeerInfo) Descriptor() ([]byte, []int) { - return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 0} + return fileDescriptor_a04e278ef61ac07a, []int{0, 0} } func (m *Message_PeerInfo) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -193,8 +202,8 @@ func (m *Message_PeerInfo) XXX_Marshal(b []byte, deterministic bool) ([]byte, er return b[:n], nil } } -func (dst *Message_PeerInfo) XXX_Merge(src proto.Message) { - xxx_messageInfo_Message_PeerInfo.Merge(dst, src) +func (m *Message_PeerInfo) XXX_Merge(src proto.Message) { + xxx_messageInfo_Message_PeerInfo.Merge(m, src) } func (m *Message_PeerInfo) XXX_Size() int { return m.Size() @@ -230,7 +239,7 @@ func (m *Message_Dial) Reset() { *m = Message_Dial{} } func (m *Message_Dial) String() string { return proto.CompactTextString(m) } func (*Message_Dial) ProtoMessage() {} func (*Message_Dial) Descriptor() ([]byte, []int) { - return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 1} + return fileDescriptor_a04e278ef61ac07a, []int{0, 1} } func (m *Message_Dial) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -247,8 +256,8 @@ func (m *Message_Dial) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) return b[:n], nil } } -func (dst *Message_Dial) XXX_Merge(src proto.Message) { - xxx_messageInfo_Message_Dial.Merge(dst, src) +func (m *Message_Dial) XXX_Merge(src proto.Message) { + xxx_messageInfo_Message_Dial.Merge(m, src) } func (m *Message_Dial) XXX_Size() int { return m.Size() @@ -279,7 +288,7 @@ func (m *Message_DialResponse) Reset() { *m = Message_DialResponse{} } func (m *Message_DialResponse) String() string { return proto.CompactTextString(m) } func (*Message_DialResponse) ProtoMessage() {} func (*Message_DialResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_autonat_bd0ec7a019b57e9d, []int{0, 2} + return fileDescriptor_a04e278ef61ac07a, []int{0, 2} } func (m *Message_DialResponse) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -296,8 +305,8 @@ func (m *Message_DialResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte return b[:n], nil } } -func (dst *Message_DialResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_Message_DialResponse.Merge(dst, src) +func (m *Message_DialResponse) XXX_Merge(src proto.Message) { + xxx_messageInfo_Message_DialResponse.Merge(m, src) } func (m *Message_DialResponse) XXX_Size() int { return m.Size() @@ -330,13 +339,44 @@ func (m *Message_DialResponse) GetAddr() []byte { } func init() { + proto.RegisterEnum("autonat.pb.Message_MessageType", Message_MessageType_name, Message_MessageType_value) + proto.RegisterEnum("autonat.pb.Message_ResponseStatus", Message_ResponseStatus_name, Message_ResponseStatus_value) proto.RegisterType((*Message)(nil), "autonat.pb.Message") proto.RegisterType((*Message_PeerInfo)(nil), "autonat.pb.Message.PeerInfo") proto.RegisterType((*Message_Dial)(nil), "autonat.pb.Message.Dial") proto.RegisterType((*Message_DialResponse)(nil), "autonat.pb.Message.DialResponse") - proto.RegisterEnum("autonat.pb.Message_MessageType", Message_MessageType_name, Message_MessageType_value) - proto.RegisterEnum("autonat.pb.Message_ResponseStatus", Message_ResponseStatus_name, Message_ResponseStatus_value) } + +func init() { proto.RegisterFile("autonat.proto", fileDescriptor_a04e278ef61ac07a) } + +var fileDescriptor_a04e278ef61ac07a = []byte{ + // 372 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x74, 0x90, 0xcf, 0x8a, 0xda, 0x50, + 0x14, 0xc6, 0xbd, 0x31, 0xb5, 0xf6, 0x18, 0xc3, 0xed, 0xa1, 0x85, 0x20, 0x25, 0x0d, 0x59, 0x49, + 0x29, 0x22, 0x76, 0x53, 0xba, 0x53, 0x72, 0x0b, 0xd2, 0x56, 0xed, 0x49, 0x5c, 0x87, 0x94, 0xdc, + 0x0e, 0x01, 0x31, 0x21, 0x89, 0x30, 0x6e, 0xe6, 0x89, 0x66, 0x3b, 0xef, 0xe0, 0x72, 0x1e, 0x61, + 0xf0, 0x49, 0x86, 0x5c, 0xa3, 0xa3, 0xe0, 0xac, 0xce, 0x1f, 0x7e, 0xdf, 0x39, 0x1f, 0x1f, 0x74, + 0xa3, 0x4d, 0x99, 0xae, 0xa3, 0x72, 0x90, 0xe5, 0x69, 0x99, 0x22, 0x9c, 0xc6, 0x7f, 0xee, 0x83, + 0x0e, 0x6f, 0xff, 0xc8, 0xa2, 0x88, 0x6e, 0x24, 0x7e, 0x03, 0xbd, 0xdc, 0x66, 0xd2, 0x62, 0x0e, + 0xeb, 0x9b, 0xa3, 0xcf, 0x83, 0x17, 0x6c, 0x50, 0x23, 0xc7, 0x1a, 0x6c, 0x33, 0x49, 0x0a, 0xc6, + 0xaf, 0xa0, 0xc7, 0x49, 0xb4, 0xb2, 0x34, 0x87, 0xf5, 0x3b, 0x23, 0xeb, 0x9a, 0xc8, 0x4b, 0xa2, + 0x15, 0x29, 0x0a, 0x3d, 0x30, 0xaa, 0x4a, 0xb2, 0xc8, 0xd2, 0x75, 0x21, 0xad, 0xa6, 0x52, 0x39, + 0xaf, 0xaa, 0x6a, 0x8e, 0x2e, 0x54, 0xbd, 0x21, 0xb4, 0x17, 0x52, 0xe6, 0xd3, 0xf5, 0xff, 0x14, + 0x4d, 0xd0, 0x92, 0x58, 0x59, 0x36, 0x48, 0x4b, 0x62, 0xfc, 0x00, 0x6f, 0xa2, 0x38, 0xce, 0x0b, + 0x4b, 0x73, 0x9a, 0x7d, 0x83, 0x0e, 0x43, 0xef, 0x3b, 0xe8, 0xd5, 0x3d, 0x1c, 0x82, 0x9e, 0x49, + 0x99, 0x2b, 0xbe, 0x33, 0xfa, 0x74, 0xed, 0xef, 0xf1, 0x32, 0x29, 0xb2, 0x77, 0x07, 0xc6, 0xb9, + 0x13, 0xfc, 0x01, 0xad, 0xa2, 0x8c, 0xca, 0x4d, 0x51, 0xc7, 0xe4, 0x5e, 0xbb, 0x71, 0xa4, 0x7d, + 0x45, 0x52, 0xad, 0x40, 0x1b, 0xe0, 0xd0, 0x05, 0xf2, 0xb6, 0x54, 0x89, 0xbd, 0xa3, 0xb3, 0x0d, + 0x22, 0xe8, 0x95, 0x5d, 0x95, 0x8a, 0x41, 0xaa, 0x77, 0xbf, 0x40, 0xe7, 0x2c, 0x74, 0x6c, 0x83, + 0xee, 0x4d, 0xc7, 0xbf, 0x79, 0x03, 0xdf, 0x43, 0xb7, 0xea, 0x42, 0x12, 0xfe, 0x62, 0x3e, 0xf3, + 0x05, 0x67, 0x6e, 0x02, 0xe6, 0xe5, 0x67, 0x6c, 0x81, 0x36, 0xff, 0xc5, 0x1b, 0xc8, 0xc1, 0x10, + 0xa1, 0xc2, 0x05, 0xd1, 0x9c, 0x78, 0x8c, 0x08, 0x66, 0xbd, 0x21, 0xf1, 0x73, 0xe9, 0x0b, 0x8f, + 0x4b, 0x44, 0xe8, 0x8a, 0x70, 0x32, 0xf6, 0x42, 0x12, 0x7f, 0x97, 0xc2, 0x0f, 0xf8, 0x8e, 0xe1, + 0x47, 0xe0, 0x22, 0x9c, 0xce, 0x02, 0x41, 0xb3, 0x93, 0xfa, 0x5e, 0x9b, 0x18, 0xbb, 0xbd, 0xcd, + 0x1e, 0xf7, 0x36, 0x7b, 0xda, 0xdb, 0xec, 0x39, 0x00, 0x00, 0xff, 0xff, 0x8e, 0xe2, 0x93, 0x4e, + 0x61, 0x02, 0x00, 0x00, +} + func (m *Message) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) @@ -497,6 +537,9 @@ func encodeVarintAutonat(dAtA []byte, offset int, v uint64) int { return offset + 1 } func (m *Message) Size() (n int) { + if m == nil { + return 0 + } var l int _ = l if m.Type != nil { @@ -517,6 +560,9 @@ func (m *Message) Size() (n int) { } func (m *Message_PeerInfo) Size() (n int) { + if m == nil { + return 0 + } var l int _ = l if m.Id != nil { @@ -536,6 +582,9 @@ func (m *Message_PeerInfo) Size() (n int) { } func (m *Message_Dial) Size() (n int) { + if m == nil { + return 0 + } var l int _ = l if m.Peer != nil { @@ -549,6 +598,9 @@ func (m *Message_Dial) Size() (n int) { } func (m *Message_DialResponse) Size() (n int) { + if m == nil { + return 0 + } var l int _ = l if m.Status != nil { @@ -596,7 +648,7 @@ func (m *Message) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -624,7 +676,7 @@ func (m *Message) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - v |= (Message_MessageType(b) & 0x7F) << shift + v |= Message_MessageType(b&0x7F) << shift if b < 0x80 { break } @@ -644,7 +696,7 @@ func (m *Message) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - msglen |= (int(b) & 0x7F) << shift + msglen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -653,6 +705,9 @@ func (m *Message) Unmarshal(dAtA []byte) error { return ErrInvalidLengthAutonat } postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthAutonat + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -677,7 +732,7 @@ func (m *Message) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - msglen |= (int(b) & 0x7F) << shift + msglen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -686,6 +741,9 @@ func (m *Message) Unmarshal(dAtA []byte) error { return ErrInvalidLengthAutonat } postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthAutonat + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -705,6 +763,9 @@ func (m *Message) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthAutonat } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthAutonat + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } @@ -733,7 +794,7 @@ func (m *Message_PeerInfo) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -761,7 +822,7 @@ func (m *Message_PeerInfo) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -770,6 +831,9 @@ func (m *Message_PeerInfo) Unmarshal(dAtA []byte) error { return ErrInvalidLengthAutonat } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAutonat + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -792,7 +856,7 @@ func (m *Message_PeerInfo) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -801,6 +865,9 @@ func (m *Message_PeerInfo) Unmarshal(dAtA []byte) error { return ErrInvalidLengthAutonat } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAutonat + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -816,6 +883,9 @@ func (m *Message_PeerInfo) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthAutonat } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthAutonat + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } @@ -844,7 +914,7 @@ func (m *Message_Dial) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -872,7 +942,7 @@ func (m *Message_Dial) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - msglen |= (int(b) & 0x7F) << shift + msglen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -881,6 +951,9 @@ func (m *Message_Dial) Unmarshal(dAtA []byte) error { return ErrInvalidLengthAutonat } postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthAutonat + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -900,6 +973,9 @@ func (m *Message_Dial) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthAutonat } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthAutonat + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } @@ -928,7 +1004,7 @@ func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -956,7 +1032,7 @@ func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - v |= (Message_ResponseStatus(b) & 0x7F) << shift + v |= Message_ResponseStatus(b&0x7F) << shift if b < 0x80 { break } @@ -976,7 +1052,7 @@ func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - stringLen |= (uint64(b) & 0x7F) << shift + stringLen |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -986,6 +1062,9 @@ func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { return ErrInvalidLengthAutonat } postIndex := iNdEx + intStringLen + if postIndex < 0 { + return ErrInvalidLengthAutonat + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -1006,7 +1085,7 @@ func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -1015,6 +1094,9 @@ func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { return ErrInvalidLengthAutonat } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAutonat + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -1032,6 +1114,9 @@ func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthAutonat } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthAutonat + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } @@ -1099,10 +1184,13 @@ func skipAutonat(dAtA []byte) (n int, err error) { break } } - iNdEx += length if length < 0 { return 0, ErrInvalidLengthAutonat } + iNdEx += length + if iNdEx < 0 { + return 0, ErrInvalidLengthAutonat + } return iNdEx, nil case 3: for { @@ -1131,6 +1219,9 @@ func skipAutonat(dAtA []byte) (n int, err error) { return 0, err } iNdEx = start + next + if iNdEx < 0 { + return 0, ErrInvalidLengthAutonat + } } return iNdEx, nil case 4: @@ -1149,33 +1240,3 @@ var ( ErrInvalidLengthAutonat = fmt.Errorf("proto: negative length found during unmarshaling") ErrIntOverflowAutonat = fmt.Errorf("proto: integer overflow") ) - -func init() { proto.RegisterFile("autonat.proto", fileDescriptor_autonat_bd0ec7a019b57e9d) } - -var fileDescriptor_autonat_bd0ec7a019b57e9d = []byte{ - // 372 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x74, 0x90, 0xcf, 0x8a, 0xda, 0x50, - 0x14, 0xc6, 0xbd, 0x31, 0xb5, 0xf6, 0x18, 0xc3, 0xed, 0xa1, 0x85, 0x20, 0x25, 0x0d, 0x59, 0x49, - 0x29, 0x22, 0x76, 0x53, 0xba, 0x53, 0x72, 0x0b, 0xd2, 0x56, 0xed, 0x49, 0x5c, 0x87, 0x94, 0xdc, - 0x0e, 0x01, 0x31, 0x21, 0x89, 0x30, 0x6e, 0xe6, 0x89, 0x66, 0x3b, 0xef, 0xe0, 0x72, 0x1e, 0x61, - 0xf0, 0x49, 0x86, 0x5c, 0xa3, 0xa3, 0xe0, 0xac, 0xce, 0x1f, 0x7e, 0xdf, 0x39, 0x1f, 0x1f, 0x74, - 0xa3, 0x4d, 0x99, 0xae, 0xa3, 0x72, 0x90, 0xe5, 0x69, 0x99, 0x22, 0x9c, 0xc6, 0x7f, 0xee, 0x83, - 0x0e, 0x6f, 0xff, 0xc8, 0xa2, 0x88, 0x6e, 0x24, 0x7e, 0x03, 0xbd, 0xdc, 0x66, 0xd2, 0x62, 0x0e, - 0xeb, 0x9b, 0xa3, 0xcf, 0x83, 0x17, 0x6c, 0x50, 0x23, 0xc7, 0x1a, 0x6c, 0x33, 0x49, 0x0a, 0xc6, - 0xaf, 0xa0, 0xc7, 0x49, 0xb4, 0xb2, 0x34, 0x87, 0xf5, 0x3b, 0x23, 0xeb, 0x9a, 0xc8, 0x4b, 0xa2, - 0x15, 0x29, 0x0a, 0x3d, 0x30, 0xaa, 0x4a, 0xb2, 0xc8, 0xd2, 0x75, 0x21, 0xad, 0xa6, 0x52, 0x39, - 0xaf, 0xaa, 0x6a, 0x8e, 0x2e, 0x54, 0xbd, 0x21, 0xb4, 0x17, 0x52, 0xe6, 0xd3, 0xf5, 0xff, 0x14, - 0x4d, 0xd0, 0x92, 0x58, 0x59, 0x36, 0x48, 0x4b, 0x62, 0xfc, 0x00, 0x6f, 0xa2, 0x38, 0xce, 0x0b, - 0x4b, 0x73, 0x9a, 0x7d, 0x83, 0x0e, 0x43, 0xef, 0x3b, 0xe8, 0xd5, 0x3d, 0x1c, 0x82, 0x9e, 0x49, - 0x99, 0x2b, 0xbe, 0x33, 0xfa, 0x74, 0xed, 0xef, 0xf1, 0x32, 0x29, 0xb2, 0x77, 0x07, 0xc6, 0xb9, - 0x13, 0xfc, 0x01, 0xad, 0xa2, 0x8c, 0xca, 0x4d, 0x51, 0xc7, 0xe4, 0x5e, 0xbb, 0x71, 0xa4, 0x7d, - 0x45, 0x52, 0xad, 0x40, 0x1b, 0xe0, 0xd0, 0x05, 0xf2, 0xb6, 0x54, 0x89, 0xbd, 0xa3, 0xb3, 0x0d, - 0x22, 0xe8, 0x95, 0x5d, 0x95, 0x8a, 0x41, 0xaa, 0x77, 0xbf, 0x40, 0xe7, 0x2c, 0x74, 0x6c, 0x83, - 0xee, 0x4d, 0xc7, 0xbf, 0x79, 0x03, 0xdf, 0x43, 0xb7, 0xea, 0x42, 0x12, 0xfe, 0x62, 0x3e, 0xf3, - 0x05, 0x67, 0x6e, 0x02, 0xe6, 0xe5, 0x67, 0x6c, 0x81, 0x36, 0xff, 0xc5, 0x1b, 0xc8, 0xc1, 0x10, - 0xa1, 0xc2, 0x05, 0xd1, 0x9c, 0x78, 0x8c, 0x08, 0x66, 0xbd, 0x21, 0xf1, 0x73, 0xe9, 0x0b, 0x8f, - 0x4b, 0x44, 0xe8, 0x8a, 0x70, 0x32, 0xf6, 0x42, 0x12, 0x7f, 0x97, 0xc2, 0x0f, 0xf8, 0x8e, 0xe1, - 0x47, 0xe0, 0x22, 0x9c, 0xce, 0x02, 0x41, 0xb3, 0x93, 0xfa, 0x5e, 0x9b, 0x18, 0xbb, 0xbd, 0xcd, - 0x1e, 0xf7, 0x36, 0x7b, 0xda, 0xdb, 0xec, 0x39, 0x00, 0x00, 0xff, 0xff, 0x8e, 0xe2, 0x93, 0x4e, - 0x61, 0x02, 0x00, 0x00, -} From 533d9866c687dd8dcba954cfd74c16a489cb68a7 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 18 Feb 2019 16:01:43 +0100 Subject: [PATCH 0934/3965] gx publish 6.0.34 --- .gx/lastpubver | 2 +- p2p/protocol/identify/pb/identify.pb.go | 100 +++++++++++++-------- package.json | 111 ++++++++++++------------ 3 files changed, 123 insertions(+), 90 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 9360d2a36b..634d3869d7 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.33: QmebEmt23jQxrwnqBkFL4qbpE8EnnQunpv5U32LS5ESus1 +6.0.34: QmZLcEoTHWSoe5XtkJ5o5AQYMvdQR4FZGSNBZ2KVuFadN7 diff --git a/p2p/protocol/identify/pb/identify.pb.go b/p2p/protocol/identify/pb/identify.pb.go index 774ab2b89c..de1275ba3f 100644 --- a/p2p/protocol/identify/pb/identify.pb.go +++ b/p2p/protocol/identify/pb/identify.pb.go @@ -3,11 +3,12 @@ package identify_pb -import proto "github.com/gogo/protobuf/proto" -import fmt "fmt" -import math "math" - -import io "io" +import ( + fmt "fmt" + proto "github.com/gogo/protobuf/proto" + io "io" + math "math" +) // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal @@ -47,7 +48,7 @@ func (m *Identify) Reset() { *m = Identify{} } func (m *Identify) String() string { return proto.CompactTextString(m) } func (*Identify) ProtoMessage() {} func (*Identify) Descriptor() ([]byte, []int) { - return fileDescriptor_identify_daaec8baf46eae80, []int{0} + return fileDescriptor_83f1e7e6b485409f, []int{0} } func (m *Identify) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -64,8 +65,8 @@ func (m *Identify) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { return b[:n], nil } } -func (dst *Identify) XXX_Merge(src proto.Message) { - xxx_messageInfo_Identify.Merge(dst, src) +func (m *Identify) XXX_Merge(src proto.Message) { + xxx_messageInfo_Identify.Merge(m, src) } func (m *Identify) XXX_Size() int { return m.Size() @@ -121,6 +122,25 @@ func (m *Identify) GetProtocols() []string { func init() { proto.RegisterType((*Identify)(nil), "identify.pb.Identify") } + +func init() { proto.RegisterFile("identify.proto", fileDescriptor_83f1e7e6b485409f) } + +var fileDescriptor_83f1e7e6b485409f = []byte{ + // 187 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0xcb, 0x4c, 0x49, 0xcd, + 0x2b, 0xc9, 0x4c, 0xab, 0xd4, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x46, 0xf0, 0x93, 0x94, + 0x6e, 0x31, 0x72, 0x71, 0x78, 0x42, 0xf9, 0x42, 0x1a, 0x5c, 0xfc, 0x60, 0x25, 0xc9, 0xf9, 0x39, + 0x61, 0xa9, 0x45, 0xc5, 0x99, 0xf9, 0x79, 0x12, 0xac, 0x0a, 0x8c, 0x1a, 0x9c, 0x41, 0xe8, 0xc2, + 0x42, 0x4a, 0x5c, 0x3c, 0x89, 0xe9, 0xa9, 0x79, 0x25, 0x30, 0x65, 0x6c, 0x60, 0x65, 0x28, 0x62, + 0x42, 0x32, 0x5c, 0x9c, 0x05, 0xa5, 0x49, 0x39, 0x99, 0xc9, 0xde, 0xa9, 0x95, 0x12, 0x8c, 0x0a, + 0x8c, 0x1a, 0x3c, 0x41, 0x08, 0x01, 0x21, 0x05, 0x2e, 0xee, 0x9c, 0xcc, 0xe2, 0x92, 0xd4, 0x3c, + 0xc7, 0x94, 0x94, 0xa2, 0x62, 0x09, 0x26, 0x05, 0x66, 0x0d, 0x9e, 0x20, 0x64, 0x21, 0x90, 0x1d, + 0xf9, 0x49, 0xc5, 0xa9, 0x45, 0x65, 0xa9, 0x29, 0x20, 0x01, 0x09, 0x16, 0xb0, 0x11, 0x28, 0x62, + 0x60, 0x3b, 0xa0, 0x4e, 0x2b, 0x96, 0x60, 0x56, 0x60, 0xd6, 0xe0, 0x0c, 0x42, 0x08, 0x38, 0xf1, + 0x9c, 0x78, 0x24, 0xc7, 0x78, 0xe1, 0x91, 0x1c, 0xe3, 0x83, 0x47, 0x72, 0x8c, 0x80, 0x00, 0x00, + 0x00, 0xff, 0xff, 0xc2, 0x2c, 0x19, 0x46, 0x08, 0x01, 0x00, 0x00, +} + func (m *Identify) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) @@ -199,6 +219,9 @@ func encodeVarintIdentify(dAtA []byte, offset int, v uint64) int { return offset + 1 } func (m *Identify) Size() (n int) { + if m == nil { + return 0 + } var l int _ = l if m.PublicKey != nil { @@ -263,7 +286,7 @@ func (m *Identify) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -291,7 +314,7 @@ func (m *Identify) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -300,6 +323,9 @@ func (m *Identify) Unmarshal(dAtA []byte) error { return ErrInvalidLengthIdentify } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -322,7 +348,7 @@ func (m *Identify) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -331,6 +357,9 @@ func (m *Identify) Unmarshal(dAtA []byte) error { return ErrInvalidLengthIdentify } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -351,7 +380,7 @@ func (m *Identify) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - stringLen |= (uint64(b) & 0x7F) << shift + stringLen |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -361,6 +390,9 @@ func (m *Identify) Unmarshal(dAtA []byte) error { return ErrInvalidLengthIdentify } postIndex := iNdEx + intStringLen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -380,7 +412,7 @@ func (m *Identify) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -389,6 +421,9 @@ func (m *Identify) Unmarshal(dAtA []byte) error { return ErrInvalidLengthIdentify } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -411,7 +446,7 @@ func (m *Identify) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - stringLen |= (uint64(b) & 0x7F) << shift + stringLen |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -421,6 +456,9 @@ func (m *Identify) Unmarshal(dAtA []byte) error { return ErrInvalidLengthIdentify } postIndex := iNdEx + intStringLen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -441,7 +479,7 @@ func (m *Identify) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - stringLen |= (uint64(b) & 0x7F) << shift + stringLen |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -451,6 +489,9 @@ func (m *Identify) Unmarshal(dAtA []byte) error { return ErrInvalidLengthIdentify } postIndex := iNdEx + intStringLen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -466,6 +507,9 @@ func (m *Identify) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthIdentify } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthIdentify + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } @@ -533,10 +577,13 @@ func skipIdentify(dAtA []byte) (n int, err error) { break } } - iNdEx += length if length < 0 { return 0, ErrInvalidLengthIdentify } + iNdEx += length + if iNdEx < 0 { + return 0, ErrInvalidLengthIdentify + } return iNdEx, nil case 3: for { @@ -565,6 +612,9 @@ func skipIdentify(dAtA []byte) (n int, err error) { return 0, err } iNdEx = start + next + if iNdEx < 0 { + return 0, ErrInvalidLengthIdentify + } } return iNdEx, nil case 4: @@ -583,21 +633,3 @@ var ( ErrInvalidLengthIdentify = fmt.Errorf("proto: negative length found during unmarshaling") ErrIntOverflowIdentify = fmt.Errorf("proto: integer overflow") ) - -func init() { proto.RegisterFile("identify.proto", fileDescriptor_identify_daaec8baf46eae80) } - -var fileDescriptor_identify_daaec8baf46eae80 = []byte{ - // 187 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0xcb, 0x4c, 0x49, 0xcd, - 0x2b, 0xc9, 0x4c, 0xab, 0xd4, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x46, 0xf0, 0x93, 0x94, - 0x6e, 0x31, 0x72, 0x71, 0x78, 0x42, 0xf9, 0x42, 0x32, 0x5c, 0x9c, 0x05, 0xa5, 0x49, 0x39, 0x99, - 0xc9, 0xde, 0xa9, 0x95, 0x12, 0x8c, 0x0a, 0x8c, 0x1a, 0x3c, 0x41, 0x08, 0x01, 0x21, 0x05, 0x2e, - 0xee, 0x9c, 0xcc, 0xe2, 0x92, 0xd4, 0x3c, 0xc7, 0x94, 0x94, 0xa2, 0x62, 0x09, 0x26, 0x05, 0x66, - 0x0d, 0x9e, 0x20, 0x64, 0x21, 0xb0, 0x7e, 0x90, 0x15, 0xc9, 0xf9, 0x39, 0xc5, 0x12, 0xcc, 0x0a, - 0xcc, 0x1a, 0x9c, 0x41, 0x08, 0x01, 0x21, 0x25, 0x2e, 0x9e, 0xfc, 0xa4, 0xe2, 0xd4, 0xa2, 0xb2, - 0xd4, 0x14, 0x90, 0x72, 0x09, 0x16, 0xb0, 0x05, 0x28, 0x62, 0x42, 0x1a, 0x5c, 0xfc, 0x30, 0x0d, - 0x61, 0xa9, 0x45, 0xc5, 0x99, 0xf9, 0x79, 0x12, 0xac, 0x0a, 0x8c, 0x1a, 0x9c, 0x41, 0xe8, 0xc2, - 0x20, 0xd3, 0x12, 0xd3, 0x53, 0xf3, 0x4a, 0x60, 0xca, 0xd8, 0xc0, 0xca, 0x50, 0xc4, 0x9c, 0x78, - 0x4e, 0x3c, 0x92, 0x63, 0xbc, 0xf0, 0x48, 0x8e, 0xf1, 0xc1, 0x23, 0x39, 0x46, 0x40, 0x00, 0x00, - 0x00, 0xff, 0xff, 0x4b, 0x9c, 0x90, 0x7a, 0x08, 0x01, 0x00, 0x00, -} diff --git a/package.json b/package.json index 3da2aa719d..c0da654cd1 100644 --- a/package.json +++ b/package.json @@ -34,9 +34,9 @@ "version": "1.0.0" }, { - "hash": "QmcuXC5cxs79ro2cUuHs4HQ2bkDLJUYokwL8aivcX6HW3C", + "hash": "QmbkT7eMTyXfpeyB3ZMxxcxg7XH8t6uXp49jqzz4HB7BGF", "name": "go-log", - "version": "1.5.8" + "version": "1.5.9" }, { "hash": "QmZcLBXKaFe8ND5YHPkJRAwmhJGrVsi1JqDZNyJ4nRK5Mj", @@ -50,33 +50,33 @@ }, { "author": "whyrusleeping", - "hash": "QmU7CkhdputERjy5QQr4kEUsKWzQPmkw3DZEMWxeShu6QR", + "hash": "QmcXCcD4dC51dkBWSvDQP1ySyzZUn6hbL9jkzJfYQc6VZB", "name": "go-libp2p-loggables", - "version": "1.1.30" + "version": "1.1.31" }, { "author": "whyrusleeping", - "hash": "QmQsNqBLwQbEGMJ85zAT6D7zZnLyCR57YWh4sh4g1V43qK", + "hash": "QmQ9Rt29eWd7Zvvna4Jd4i56UqPkhEJsBne3gYHUK7rtTk", "name": "go-libp2p-secio", - "version": "2.0.25" + "version": "2.0.26" }, { "author": "whyrusleeping", - "hash": "QmQFFp4ntkd4C14sP3FaH9WJyBuetuGUVo6dShNHvnoEvC", + "hash": "QmVYTrDbcoZNkHyYfjopWwUK8ebfUq1DqDwSQLbizwPjgq", "name": "go-libp2p-peerstore", - "version": "2.0.14" + "version": "2.0.16" }, { "author": "whyrusleeping", - "hash": "QmUDtgnEr7FFrtK2LQM2dFzTNWghnrApBDcU3iHEJz8eQS", + "hash": "QmNw1fY1BuZ82u1qkFHxCFSXHgWMFZrFC22V3Jn7H8GYRL", "name": "go-libp2p-transport", - "version": "3.0.22" + "version": "3.0.23" }, { "author": "whyrusleeping", - "hash": "QmayGfkAeV33kHs8rw78wkPq4QZBgbG6LhyZJQ9gfyYG2o", + "hash": "QmRgVwr2PYKgFeB1pKQcD6Q6PeoA7mF1x5UpmxEjBjKu9X", "name": "go-tcp-transport", - "version": "2.0.23" + "version": "2.0.24" }, { "author": "whyrusleeping", @@ -92,57 +92,57 @@ }, { "author": "whyrusleeping", - "hash": "QmVnJMgafh5MBYiyqbvDtoCL8pcQvbEGD2k9o9GFpBWPzY", + "hash": "QmeFVdhzY13YZPWxCiQvmLercrumFRoQZFQEYw2BtzyiQc", "name": "go-testutil", - "version": "1.2.16" + "version": "1.2.17" }, { "author": "whyrusleeping", - "hash": "QmZ7cBWUXkyWTMN4qH6NGoyMVs7JugyFChBNP4ZUp5rJHH", + "hash": "Qmf1MBFKph9DWDMsdksXUFWSpUW1cGvCsmYiN9Wsdr2Dn5", "name": "go-libp2p-net", - "version": "3.0.23" + "version": "3.0.26" }, { "author": "whyrusleeping", - "hash": "QmRN8cMBqfgLgrcaeK6vqUd7HuyvKbNnzaSj4TRBW9XvPQ", + "hash": "QmZZseAa9xcK6tT3YpaShNUAEpyRAoWmUL5ojH3uGNepAc", "name": "go-libp2p-metrics", - "version": "2.1.12" + "version": "2.1.13" }, { "author": "whyrusleeping", - "hash": "QmfRHxh8bt4jWLKRhNvR5fn7mFACrQBFLqV4wyoymEExKV", + "hash": "QmfLNahGe1CVYdxdcoKYXaSkScoDdvF68UPy8zmUB78cEG", "name": "go-libp2p-host", - "version": "3.0.22" + "version": "3.0.23" }, { "author": "whyrusleeping", - "hash": "QmTJCJaS8Cpjc2MkoS32iwr4zMZtbLkaF9GJsUgH1uwtN9", + "hash": "QmaWkV7WgViKgYNtnm2LfPCTuwKp1CMrfAvwfLwtajB9MA", "name": "go-libp2p-swarm", - "version": "3.0.29" + "version": "3.0.30" }, { "author": "whyrusleeping", - "hash": "QmXFPLTyWRrWp4zkNrD5S3s8S5LRJyJx67hQwzwaTYtVzi", + "hash": "QmQhG54sm28e44HfGjG37F68Tzu45f18qimgdyJRa5VR8a", "name": "go-libp2p-nat", - "version": "0.8.10" + "version": "0.8.11" }, { "author": "whyrusleeping", - "hash": "QmZtiLFhqsDFC3TmmUyWMFJXibURW362UmrXqs3zJim1yb", + "hash": "QmQTUGt4PugW7zESt3CdY7wqpq8dMyG8xYHMZ9wbQnkPbR", "name": "go-libp2p-netutil", - "version": "0.4.19" + "version": "0.4.20" }, { "author": "whyrusleeping", - "hash": "QmYZJzRGPeRpEufmdqXPAcKrpg9gxCnRVRadTn99PH2P77", + "hash": "QmWRQyCNQTsuAqt4AHnjFpiLwnYoCxRQQub1AjNwcgXX4C", "name": "go-libp2p-blankhost", - "version": "0.3.22" + "version": "0.3.23" }, { "author": "whyrusleeping", - "hash": "QmNiJiXwWE3kRhZrC5ej3kSjWHm337pYfhjLGSCDNKJP2s", + "hash": "QmTW4SdgBWq9GjsBsHeUx8WuGxzhgzAf88UMH2w62PC8yK", "name": "go-libp2p-crypto", - "version": "2.0.4" + "version": "2.0.7" }, { "author": "whyrusleeping", @@ -158,15 +158,15 @@ }, { "author": "whyrusleeping", - "hash": "QmPJxxDsX2UbchSHobbYuvz7qnyJTFKvaKMzE2rZWJ4x5B", + "hash": "QmTu65MVbemtUxJEWgsTtzv9Zv9P8rvmqNA4eG9TrTRGYc", "name": "go-libp2p-peer", - "version": "3.1.0" + "version": "3.1.1" }, { "author": "vyzo", - "hash": "QmQG8wJtY6KfsTH2tjhaThFPeYVJGm7cmRMxen73ipA4Z5", + "hash": "QmYJkngqrmSAGgohtNAPwqkkNzkVDEiay34f9zk4YqXcTJ", "name": "go-libp2p-circuit", - "version": "2.3.9" + "version": "2.3.10" }, { "author": "lgierth", @@ -176,33 +176,33 @@ }, { "author": "why", - "hash": "QmebAt96MwXHnbJ5uns6KLm3eSVLPDaaCB4DU7phQUi9a3", + "hash": "QmVgpHPZkbxnPyEMZWi2pr2C179SGSYf8sbS95tKji8AWS", "name": "go-libp2p-interface-connmgr", - "version": "0.0.28" + "version": "0.0.29" }, { "author": "whyrusleeping", - "hash": "QmZsejKNkeFSQe5TcmYXJ8iq6qPL1FpsP4eAA8j7RfE7xg", + "hash": "QmZEft2D8in8cNEZJtZoPCSDpS9LQ7jiWX3gPBcZ4s5m2k", "name": "go-smux-multiplex", - "version": "3.0.17" + "version": "3.0.18" }, { "author": "whyrusleeping", - "hash": "QmTPHQHPgmoJtmhMSpVhRPeb23kXGgkUEDKfn9ZndnaQmy", + "hash": "QmQgYed6sPyWktsaub62GqXHJrkZ69S2xbm7sJhx79cZV6", "name": "go-ws-transport", - "version": "2.0.22" + "version": "2.0.23" }, { "author": "stebalien", - "hash": "QmaMmPhkoDQBdGVMyoKw2cbLp8p46FXm1UrSr5U8tvYnJk", + "hash": "QmZHWsA42XjBGY9kovAACnrzsCEMB6dnw5Xe28qWXpPvSb", "name": "go-conn-security-multistream", - "version": "0.1.22" + "version": "0.1.23" }, { "author": "Stebalien", - "hash": "QmWmeXRTSyWvjPQZgXjXTj2aP74tMSgJwWi1SAvHsvBJVj", + "hash": "QmUc71oqEeRBtBVjFQmpCmShwrwQkAAq3zGr5UjbPUk5tW", "name": "go-conn-security", - "version": "0.1.23" + "version": "0.1.24" }, { "author": "libp2p", @@ -218,31 +218,31 @@ }, { "author": "steb", - "hash": "QmQwvsMzMDTW2K8ySZYgnTVCkzQXVDxmGB5upvVFwdumJV", + "hash": "QmUEH1xBHFJPdiATuSACBFogZJQv2CcXCwWHGwCaVppfgS", "name": "go-libp2p-transport-upgrader", - "version": "0.1.23" + "version": "0.1.24" }, { - "hash": "QmdxUuburamoF6zF9qjeQC4WYcWGbWuRmdLacMEsW8ioD8", + "hash": "QmddjPSGZb3ieihSseFeCfVRpZzcqczPNsD2DvarSwnjJB", "name": "gogo-protobuf", - "version": "0.0.0" + "version": "1.2.1" }, { "author": "vyzo", - "hash": "Qmezr13Pzko3pxYz4EYMk6mQpJem72TsTVhwR3jGBDEoo7", + "hash": "QmRVa9TKPakfnqER7HsuqKPCbtDrHJ5EzFufofTU1jFybg", "name": "go-libp2p-discovery", - "version": "1.0.8" + "version": "1.0.9" }, { "author": "vyzo", - "hash": "QmZgrJk2k14P3zHUAz4hdk1TnU57iaTWEk8fGmFkrafEMX", + "hash": "QmPVfkm53wSnCgrmuXcYwaeah6R9zhrRjNqa65gAyPmVoD", "name": "go-libp2p-autonat", - "version": "1.0.6" + "version": "1.0.8" }, { - "hash": "QmRjT8Bkut84fHf9nxMQBxGsqLAkqzMdFaemDK7e61dBNZ", + "hash": "QmWXyhiyAHFmuhpZs1cgTFmpcsKyys4ACqoJzZnWMDQhFg", "name": "go-libp2p-routing", - "version": "2.7.8" + "version": "2.7.9" } ], "gxVersion": "0.4.0", @@ -250,5 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.33" + "version": "6.0.34" } + From af0c6bd9ddbb68ff901e5c72fbd2a5bcdb861a4d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 18 Feb 2019 16:35:48 +0000 Subject: [PATCH 0935/3965] pstoreds: remove async flushing on read: now sync. --- p2p/host/peerstore/pstoreds/addr_book.go | 97 +++++---------------- p2p/host/peerstore/pstoreds/addr_book_gc.go | 3 +- 2 files changed, 25 insertions(+), 75 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 892bdc4b6f..955f4d616a 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -77,8 +77,7 @@ func (r *addrsRecord) flush(write ds.Write) (err error) { // * when performing periodic GC. // * after an entry has been modified (e.g. addresses have been added or removed, TTLs updated, etc.) // -// If the return value is true, the caller can perform a flush immediately, or can schedule an async -// flush, depending on the context. +// If the return value is true, the caller should perform a flush immediately to sync the record with the store. func (r *addrsRecord) clean() (chgd bool) { now := time.Now().Unix() if !r.dirty && len(r.Addrs) > 0 && r.Addrs[0].Expiry > now { @@ -125,11 +124,9 @@ type dsAddrBook struct { gc *dsAddrBookGc subsManager *pstoremem.AddrSubManager - flushJobCh chan *addrsRecord - cancelFn func() - // controls children goroutine lifetime. childrenDone sync.WaitGroup + cancelFn func() } var _ pstore.AddrBook = (*dsAddrBook)(nil) @@ -154,31 +151,27 @@ var _ pstore.AddrBook = (*dsAddrBook)(nil) // permanent, popular values used in other libp2p modules. In this cited case, optimizing with lookahead windows // makes little sense. func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAddrBook, err error) { - var cache cache = new(noopCache) - if opts.CacheSize > 0 { - if cache, err = lru.NewARC(int(opts.CacheSize)); err != nil { - return nil, err - } - } - ctx, cancelFn := context.WithCancel(ctx) ab = &dsAddrBook{ ctx: ctx, - cancelFn: cancelFn, - opts: opts, - cache: cache, ds: store, + opts: opts, + cancelFn: cancelFn, subsManager: pstoremem.NewAddrSubManager(), - flushJobCh: make(chan *addrsRecord, 32), + } + + if opts.CacheSize > 0 { + if ab.cache, err = lru.NewARC(int(opts.CacheSize)); err != nil { + return nil, err + } + } else { + ab.cache = new(noopCache) } if ab.gc, err = newAddressBookGc(ctx, ab); err != nil { return nil, err } - ab.childrenDone.Add(1) - go ab.flusher() - return ab, nil } @@ -187,28 +180,23 @@ func (ab *dsAddrBook) Close() { ab.childrenDone.Wait() } -func (ab *dsAddrBook) asyncFlush(pr *addrsRecord) { - select { - case ab.flushJobCh <- pr: - default: - log.Warningf("flush queue is full; could not flush record for peer %s", pr.Id.ID) - } -} - // loadRecord is a read-through fetch. It fetches a record from cache, falling back to the // datastore upon a miss, and returning a newly initialized record if the peer doesn't exist. // -// loadRecord calls clean() on existing recordsrecord before returning it. If the record changes -// as a result and the update argument is true, an async flush is queued. +// loadRecord calls clean() on an existing record before returning it. If the record changes +// as a result and the update argument is true, the resulting state is saved in the datastore. // // If the cache argument is true, the record is inserted in the cache when loaded from the datastore. func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrsRecord, err error) { if e, ok := ab.cache.Get(id); ok { pr = e.(*addrsRecord) + pr.Lock() + defer pr.Unlock() + if pr.clean() && update { - ab.asyncFlush(pr) + err = pr.flush(ab.ds) } - return pr, nil + return pr, err } pr = &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} @@ -217,13 +205,15 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs switch err { case ds.ErrNotFound: + err = nil pr.Id = &pb.ProtoPeerID{ID: id} case nil: if err = pr.Unmarshal(data); err != nil { return nil, err } + // this record is new and local for now (not in cache), so we don't need to lock. if pr.clean() && update { - ab.asyncFlush(pr) + err = pr.flush(ab.ds) } default: return nil, err @@ -232,48 +222,7 @@ func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrs if cache { ab.cache.Add(id, pr) } - return pr, nil -} - -// flusher is a goroutine that takes care of persisting asynchronous flushes to the datastore. -func (ab *dsAddrBook) flusher() { - defer ab.childrenDone.Done() - - doFlush := func(fj *addrsRecord) { - if cached, ok := ab.cache.Peek(fj.Id.ID); ok { - // Only continue flushing if the record we have in memory is the same as for which the flush - // job was requested. If it's not in memory, it has been evicted and we don't know if we hold - // the latest state or not. Similarly, if it's cached but the pointer is different, it means - // it was evicted and has been reloaded, so we're also uncertain if we hold the latest state. - if pr := cached.(*addrsRecord); pr != fj { - pr.RLock() - pr.flush(ab.ds) - pr.RUnlock() - } - } - } - - for { - select { - case fj := <-ab.flushJobCh: - doFlush(fj) - - case <-ab.ctx.Done(): - // drain the flush queue, with a grace period of 1 second. - // async flush jobs are not important for consistency by definition (they correspond to expired addrs - // which will anyway expire on the next load). - for { - select { - case fj := <-ab.flushJobCh: - doFlush(fj) - case <-time.After(1 * time.Second): - return - default: - return - } - } - } - } + return pr, err } // AddAddr will add a new address if it's not already in the AddrBook. diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 6d90b07c92..f40d5533d7 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -42,6 +42,7 @@ var ( } ) +// dsAddrBookGc is responsible for garbage collection in a datastore-backed address book. type dsAddrBookGc struct { ctx context.Context ab *dsAddrBook @@ -149,7 +150,7 @@ func (gc *dsAddrBookGc) purgeLookahead() { // if we don't clean up unparseable entries we'll end up accumulating garbage. dropInError := func(key ds.Key, err error, msg string) { if err != nil { - log.Warningf("failed while %s with GC key: %v, err: %v", msg, key, err) + log.Warningf("failed while %s record with GC key: %v, err: %v; deleting", msg, key, err) } if err = batch.Delete(key); err != nil { log.Warningf("failed to delete corrupt GC lookahead entry: %v, err: %v", key, err) From 617cbd50b5a5b0d9ad4eef3401cec3fd72512660 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 18 Feb 2019 19:54:53 +0100 Subject: [PATCH 0936/3965] gx publish 6.0.35 --- .gx/lastpubver | 2 +- package.json | 88 +++++++++++++++++++++++++------------------------- 2 files changed, 45 insertions(+), 45 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 634d3869d7..80cb9f97f1 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.34: QmZLcEoTHWSoe5XtkJ5o5AQYMvdQR4FZGSNBZ2KVuFadN7 +6.0.35: QmcNGX5RaxPPCYwa6yGXM1EcUbrreTTinixLcYGmMwf1sx diff --git a/package.json b/package.json index c0da654cd1..e1a84cbde9 100644 --- a/package.json +++ b/package.json @@ -56,27 +56,27 @@ }, { "author": "whyrusleeping", - "hash": "QmQ9Rt29eWd7Zvvna4Jd4i56UqPkhEJsBne3gYHUK7rtTk", + "hash": "QmV1XUxssTbBHiVMC8Nc4LyDJGNBFMB31jH3xw64m2C1nj", "name": "go-libp2p-secio", - "version": "2.0.26" + "version": "2.0.28" }, { "author": "whyrusleeping", - "hash": "QmVYTrDbcoZNkHyYfjopWwUK8ebfUq1DqDwSQLbizwPjgq", + "hash": "QmRhFARzTHcFh8wUxwN5KvyTGq73FLC65EfFAhz8Ng7aGb", "name": "go-libp2p-peerstore", - "version": "2.0.16" + "version": "2.0.17" }, { "author": "whyrusleeping", - "hash": "QmNw1fY1BuZ82u1qkFHxCFSXHgWMFZrFC22V3Jn7H8GYRL", + "hash": "QmVcZvoaUGfXHzGJ5mFvAeCouL9SET2M7U52zZaEPyk2o5", "name": "go-libp2p-transport", - "version": "3.0.23" + "version": "3.0.25" }, { "author": "whyrusleeping", - "hash": "QmRgVwr2PYKgFeB1pKQcD6Q6PeoA7mF1x5UpmxEjBjKu9X", + "hash": "QmRUAf47dQEwdpZDhHDcWr5MD6p94ii6fv5VZph9a8sc72", "name": "go-tcp-transport", - "version": "2.0.24" + "version": "2.0.25" }, { "author": "whyrusleeping", @@ -98,9 +98,9 @@ }, { "author": "whyrusleeping", - "hash": "Qmf1MBFKph9DWDMsdksXUFWSpUW1cGvCsmYiN9Wsdr2Dn5", + "hash": "QmTGxDz2CjBucFzPNTiWwzQmTWdrBnzqbqrMucDYMsjuPb", "name": "go-libp2p-net", - "version": "3.0.26" + "version": "3.0.28" }, { "author": "whyrusleeping", @@ -110,15 +110,15 @@ }, { "author": "whyrusleeping", - "hash": "QmfLNahGe1CVYdxdcoKYXaSkScoDdvF68UPy8zmUB78cEG", + "hash": "Qmd52WKRSwrBK5gUaJKawryZQ5by6UbNB8KVW2Zy6JtbyW", "name": "go-libp2p-host", - "version": "3.0.23" + "version": "3.0.24" }, { "author": "whyrusleeping", - "hash": "QmaWkV7WgViKgYNtnm2LfPCTuwKp1CMrfAvwfLwtajB9MA", + "hash": "QmU7iTrsNaJfu1Rf5DrvaJLH9wJtQwmP4Dj8oPduprAU68", "name": "go-libp2p-swarm", - "version": "3.0.30" + "version": "3.0.31" }, { "author": "whyrusleeping", @@ -128,15 +128,15 @@ }, { "author": "whyrusleeping", - "hash": "QmQTUGt4PugW7zESt3CdY7wqpq8dMyG8xYHMZ9wbQnkPbR", + "hash": "QmUD9zcPC7rkiyp3KWBnBor7GcpU671DC8F6hBnZZUnrfv", "name": "go-libp2p-netutil", - "version": "0.4.20" + "version": "0.4.21" }, { "author": "whyrusleeping", - "hash": "QmWRQyCNQTsuAqt4AHnjFpiLwnYoCxRQQub1AjNwcgXX4C", + "hash": "QmSka7Ax3a1HhyGaC3mqqrWbDhHsTmyzvED4Xk6dfvPXbY", "name": "go-libp2p-blankhost", - "version": "0.3.23" + "version": "0.3.24" }, { "author": "whyrusleeping", @@ -146,15 +146,15 @@ }, { "author": "whyrusleeping", - "hash": "Qmdps3CYh5htGQSrPvzg5PHouVexLmtpbuLCqc4vuej8PC", + "hash": "QmcVFwGS6sjfxVico2bd1gQGRu5A2ymwZunVmMdeV5zEYb", "name": "go-smux-yamux", - "version": "2.0.8" + "version": "2.0.9" }, { "author": "whyrusleeping", - "hash": "QmRYdszNNq7ykPqavVNKMVyyjX59AcTisHqzussDfhwHkK", + "hash": "QmUWkEreTZdTxUVDfpQ2fLywJh6dinfEYahEoBDTMQ2hks", "name": "go-smux-multistream", - "version": "2.0.2" + "version": "2.0.3" }, { "author": "whyrusleeping", @@ -164,9 +164,9 @@ }, { "author": "vyzo", - "hash": "QmYJkngqrmSAGgohtNAPwqkkNzkVDEiay34f9zk4YqXcTJ", + "hash": "QmNaXXRfJ93t4HicX8N2WZPhdE8KU39MPGALuH421GFgKA", "name": "go-libp2p-circuit", - "version": "2.3.10" + "version": "2.3.11" }, { "author": "lgierth", @@ -176,33 +176,33 @@ }, { "author": "why", - "hash": "QmVgpHPZkbxnPyEMZWi2pr2C179SGSYf8sbS95tKji8AWS", + "hash": "QmcCk4LZRJPAKuwY9dusFea7LckELZgo5HagErTbm39o38", "name": "go-libp2p-interface-connmgr", - "version": "0.0.29" + "version": "0.0.30" }, { "author": "whyrusleeping", - "hash": "QmZEft2D8in8cNEZJtZoPCSDpS9LQ7jiWX3gPBcZ4s5m2k", + "hash": "QmQ4TZxzazF9dfqiH52ygGGU4BHQQTR9aCGMgHt4tCRdEA", "name": "go-smux-multiplex", "version": "3.0.18" }, { "author": "whyrusleeping", - "hash": "QmQgYed6sPyWktsaub62GqXHJrkZ69S2xbm7sJhx79cZV6", + "hash": "QmdKcuSPQDK7FkG5k3TYscKMaL9BQ122eEiKLqCKC8mEZf", "name": "go-ws-transport", - "version": "2.0.23" + "version": "2.0.24" }, { "author": "stebalien", - "hash": "QmZHWsA42XjBGY9kovAACnrzsCEMB6dnw5Xe28qWXpPvSb", + "hash": "QmS81oMV7WjkXt8AdbeYTBL8Vn2tKmyoa5U7Lnk3FjnyPc", "name": "go-conn-security-multistream", - "version": "0.1.23" + "version": "0.1.24" }, { "author": "Stebalien", - "hash": "QmUc71oqEeRBtBVjFQmpCmShwrwQkAAq3zGr5UjbPUk5tW", + "hash": "QmWdufxe3UrFtycBBj1895k4xnUWCbkuGbmBxtT7N3smjm", "name": "go-conn-security", - "version": "0.1.24" + "version": "0.1.26" }, { "author": "libp2p", @@ -212,15 +212,15 @@ }, { "author": "whyrusleeping", - "hash": "QmY9JXR3FupnYAYJWK9aMr9bCpqWKcToQ1tz8DVGTrHpHw", + "hash": "QmVtV1y2e8W4eQgzsP6qfSpCCZ6zWYE4m6NzJjB7iswwrT", "name": "go-stream-muxer", - "version": "3.0.0" + "version": "3.1.0" }, { "author": "steb", - "hash": "QmUEH1xBHFJPdiATuSACBFogZJQv2CcXCwWHGwCaVppfgS", + "hash": "QmePVH87Z1oTzrAn5vkPwDTNrieSYCqq2AQ99nxZ1fWJ7m", "name": "go-libp2p-transport-upgrader", - "version": "0.1.24" + "version": "0.1.25" }, { "hash": "QmddjPSGZb3ieihSseFeCfVRpZzcqczPNsD2DvarSwnjJB", @@ -229,20 +229,20 @@ }, { "author": "vyzo", - "hash": "QmRVa9TKPakfnqER7HsuqKPCbtDrHJ5EzFufofTU1jFybg", + "hash": "QmNaJjRsgYQCfmUSZYyk7Jv3Bh78TzKxX4ZwAPKAHfKR4n", "name": "go-libp2p-discovery", - "version": "1.0.9" + "version": "1.0.10" }, { "author": "vyzo", - "hash": "QmPVfkm53wSnCgrmuXcYwaeah6R9zhrRjNqa65gAyPmVoD", + "hash": "QmckHMRXmzTnq8YUNhNtohCzKBHiaPYqnbdvp87rjCmCi6", "name": "go-libp2p-autonat", - "version": "1.0.8" + "version": "1.0.9" }, { - "hash": "QmWXyhiyAHFmuhpZs1cgTFmpcsKyys4ACqoJzZnWMDQhFg", + "hash": "QmWaDSNoSdSXU9b6udyaq9T8y6LkzMwqWxECznFqvtcTsk", "name": "go-libp2p-routing", - "version": "2.7.9" + "version": "2.7.10" } ], "gxVersion": "0.4.0", @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.34" + "version": "6.0.35" } From c8c717cc32310db4334160298d9c535a67163bf6 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Tue, 19 Feb 2019 16:31:52 +1100 Subject: [PATCH 0937/3965] Revert "Do notifications synchronously" This reverts commit 6ee2ac3e464ddf557f44d8e10584d552ae33f879. --- p2p/net/swarm/swarm.go | 14 +++++++++++++- p2p/net/swarm/swarm_conn.go | 15 +++++++++++---- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 85c5680c6b..cc77e80ec6 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -208,6 +208,9 @@ func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { // * The other will be decremented when Conn.start exits. s.refs.Add(2) + // Take the notification lock before releasing the conns lock to block + // Disconnect notifications until after the Connect notifications done. + c.notifyLk.Lock() s.conns.Unlock() // We have a connection now. Cancel all other in-progress dials. @@ -217,6 +220,7 @@ func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { s.notifyAll(func(f inet.Notifiee) { f.Connected(s, c) }) + c.notifyLk.Unlock() c.start() @@ -434,10 +438,18 @@ func (s *Swarm) Backoff() *DialBackoff { // notifyAll sends a signal to all Notifiees func (s *Swarm) notifyAll(notify func(inet.Notifiee)) { + var wg sync.WaitGroup + s.notifs.RLock() + wg.Add(len(s.notifs.m)) for f := range s.notifs.m { - notify(f) + go func(f inet.Notifiee) { + defer wg.Done() + notify(f) + }(f) } + + wg.Wait() s.notifs.RUnlock() } diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 4e837cd934..26a7794d15 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -65,10 +65,17 @@ func (c *Conn) doClose() { s.Reset() } - c.swarm.notifyAll(func(f inet.Notifiee) { - f.Disconnected(c.swarm, c) - }) - c.swarm.refs.Done() // taken in Swarm.addConn + // do this in a goroutine to avoid deadlocking if we call close in an open notification. + go func() { + // prevents us from issuing close notifications before finishing the open notifications + c.notifyLk.Lock() + defer c.notifyLk.Unlock() + + c.swarm.notifyAll(func(f inet.Notifiee) { + f.Disconnected(c.swarm, c) + }) + c.swarm.refs.Done() // taken in Swarm.addConn + }() } func (c *Conn) removeStream(s *Stream) { From e7afaa3f50995030dcd06b1583ef3fc6e1858845 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Tue, 19 Feb 2019 20:36:36 +1100 Subject: [PATCH 0938/3965] Fix failing tests? --- p2p/net/swarm/dial_test.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index dd93bd7cec..c9d0d5dda0 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -150,7 +150,7 @@ func TestSimultDials(t *testing.T) { func newSilentPeer(t *testing.T) (peer.ID, ma.Multiaddr, net.Listener) { dst := testutil.RandPeerIDFatal(t) - lst, err := net.Listen("tcp", ":0") + lst, err := net.Listen("tcp4", ":0") if err != nil { t.Fatal(err) } @@ -168,7 +168,6 @@ func newSilentPeer(t *testing.T) (peer.ID, ma.Multiaddr, net.Listener) { } func TestDialWait(t *testing.T) { - // t.Skip("skipping for another test") t.Parallel() ctx := context.Background() From 6b6b7a301f98615b864b38a27cbacfd497362d50 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 19 Feb 2019 11:45:32 +0000 Subject: [PATCH 0939/3965] add io.Closer interface and implementation to Peerstore. --- p2p/host/peerstore/interface.go | 3 +++ p2p/host/peerstore/peerstore.go | 14 ++++++++++++++ p2p/host/peerstore/pstoreds/addr_book.go | 3 ++- p2p/host/peerstore/pstoreds/ds_test.go | 22 ++++++++++------------ 4 files changed, 29 insertions(+), 13 deletions(-) diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 25d2c08578..4d166684f5 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -3,6 +3,7 @@ package peerstore import ( "context" "errors" + "io" "math" "time" @@ -49,6 +50,8 @@ const ( // Peerstore provides a threadsafe store of Peer related // information. type Peerstore interface { + io.Closer + AddrBook KeyBook PeerMetadata diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index e460b1c612..c61b63c174 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -2,6 +2,7 @@ package peerstore import ( "fmt" + "io" "sync" peer "github.com/libp2p/go-libp2p-peer" @@ -31,6 +32,19 @@ func NewPeerstore(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { } } +func (ps *peerstore) Close() (err error) { + if cl, ok := ps.KeyBook.(io.Closer); ok { + cl.Close() + } + if cl, ok := ps.AddrBook.(io.Closer); ok { + cl.Close() + } + if cl, ok := ps.PeerMetadata.(io.Closer); ok { + cl.Close() + } + return nil +} + func (ps *peerstore) Peers() peer.IDSlice { set := map[peer.ID]struct{}{} for _, p := range ps.PeersWithKeys() { diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 955f4d616a..760ff58e42 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -175,9 +175,10 @@ func NewAddrBook(ctx context.Context, store ds.Batching, opts Options) (ab *dsAd return ab, nil } -func (ab *dsAddrBook) Close() { +func (ab *dsAddrBook) Close() error { ab.cancelFn() ab.childrenDone.Wait() + return nil } // loadRecord is a read-through fetch. It fetches a record from cache, falling back to the diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 026fde855e..3ae7ef7900 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -19,7 +19,6 @@ type datastoreFactory func(tb testing.TB) (ds.Batching, func()) var dstores = map[string]datastoreFactory{ "Badger": badgerStore, - // TODO: Enable once go-ds-leveldb supports TTL via a shim. // "Leveldb": leveldbStore, } @@ -122,42 +121,41 @@ func leveldbStore(tb testing.TB) (ds.TxnDatastore, func()) { func peerstoreFactory(tb testing.TB, storeFactory datastoreFactory, opts Options) pt.PeerstoreFactory { return func() (pstore.Peerstore, func()) { - store, closeFunc := storeFactory(tb) - + store, storeCloseFn := storeFactory(tb) ps, err := NewPeerstore(context.Background(), store, opts) if err != nil { tb.Fatal(err) } - - return ps, closeFunc + closer := func() { + ps.Close() + storeCloseFn() + } + return ps, closer } } func addressBookFactory(tb testing.TB, storeFactory datastoreFactory, opts Options) pt.AddrBookFactory { return func() (pstore.AddrBook, func()) { store, closeFunc := storeFactory(tb) - ab, err := NewAddrBook(context.Background(), store, opts) if err != nil { tb.Fatal(err) } - - return ab, func() { + closer := func() { ab.Close() closeFunc() } + return ab, closer } } func keyBookFactory(tb testing.TB, storeFactory datastoreFactory, opts Options) pt.KeyBookFactory { return func() (pstore.KeyBook, func()) { - store, closeFunc := storeFactory(tb) - + store, storeCloseFn := storeFactory(tb) kb, err := NewKeyBook(context.Background(), store, opts) if err != nil { tb.Fatal(err) } - - return kb, closeFunc + return kb, storeCloseFn } } From 9a5e27a293bc6613aafd3a602304959e4f961415 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 19 Feb 2019 17:41:14 +0000 Subject: [PATCH 0940/3965] pstoreds: switch protobuf compilation to gogofaster. --- p2p/host/peerstore/pb/Makefile | 2 +- p2p/host/peerstore/pb/pstore.pb.go | 133 +++++++++++++------------ p2p/host/peerstore/pb/pstorepb_test.go | 16 +-- 3 files changed, 77 insertions(+), 74 deletions(-) diff --git a/p2p/host/peerstore/pb/Makefile b/p2p/host/peerstore/pb/Makefile index cc99b57246..51e71f89f8 100644 --- a/p2p/host/peerstore/pb/Makefile +++ b/p2p/host/peerstore/pb/Makefile @@ -4,7 +4,7 @@ GO = $(PB:.proto=.pb.go) all: $(GO) %.pb.go: %.proto - protoc --proto_path=$(GOPATH)/src:. --gogofast_out=. $< + protoc --proto_path=$(GOPATH)/src:. --gogofaster_out=. $< clean: rm -f *.pb.go diff --git a/p2p/host/peerstore/pb/pstore.pb.go b/p2p/host/peerstore/pb/pstore.pb.go index ba5a51b63d..c80958cce5 100644 --- a/p2p/host/peerstore/pb/pstore.pb.go +++ b/p2p/host/peerstore/pb/pstore.pb.go @@ -3,12 +3,13 @@ package pstore_pb -import proto "github.com/gogo/protobuf/proto" -import fmt "fmt" -import math "math" -import _ "github.com/gogo/protobuf/gogoproto" - -import io "io" +import ( + fmt "fmt" + _ "github.com/gogo/protobuf/gogoproto" + proto "github.com/gogo/protobuf/proto" + io "io" + math "math" +) // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal @@ -26,17 +27,14 @@ type AddrBookRecord struct { // The peer ID. Id *ProtoPeerID `protobuf:"bytes,1,opt,name=id,proto3,customtype=ProtoPeerID" json:"id,omitempty"` // The multiaddresses. This is a sorted list where element 0 expires the soonest. - Addrs []*AddrBookRecord_AddrEntry `protobuf:"bytes,2,rep,name=addrs" json:"addrs,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + Addrs []*AddrBookRecord_AddrEntry `protobuf:"bytes,2,rep,name=addrs,proto3" json:"addrs,omitempty"` } func (m *AddrBookRecord) Reset() { *m = AddrBookRecord{} } func (m *AddrBookRecord) String() string { return proto.CompactTextString(m) } func (*AddrBookRecord) ProtoMessage() {} func (*AddrBookRecord) Descriptor() ([]byte, []int) { - return fileDescriptor_pstore_fcc3073dbc5464a9, []int{0} + return fileDescriptor_f96873690e08a98f, []int{0} } func (m *AddrBookRecord) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -53,8 +51,8 @@ func (m *AddrBookRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, erro return b[:n], nil } } -func (dst *AddrBookRecord) XXX_Merge(src proto.Message) { - xxx_messageInfo_AddrBookRecord.Merge(dst, src) +func (m *AddrBookRecord) XXX_Merge(src proto.Message) { + xxx_messageInfo_AddrBookRecord.Merge(m, src) } func (m *AddrBookRecord) XXX_Size() int { return m.Size() @@ -78,17 +76,14 @@ type AddrBookRecord_AddrEntry struct { // The point in time when this address expires. Expiry int64 `protobuf:"varint,2,opt,name=expiry,proto3" json:"expiry,omitempty"` // The original TTL of this address. - Ttl int64 `protobuf:"varint,3,opt,name=ttl,proto3" json:"ttl,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + Ttl int64 `protobuf:"varint,3,opt,name=ttl,proto3" json:"ttl,omitempty"` } func (m *AddrBookRecord_AddrEntry) Reset() { *m = AddrBookRecord_AddrEntry{} } func (m *AddrBookRecord_AddrEntry) String() string { return proto.CompactTextString(m) } func (*AddrBookRecord_AddrEntry) ProtoMessage() {} func (*AddrBookRecord_AddrEntry) Descriptor() ([]byte, []int) { - return fileDescriptor_pstore_fcc3073dbc5464a9, []int{0, 0} + return fileDescriptor_f96873690e08a98f, []int{0, 0} } func (m *AddrBookRecord_AddrEntry) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -105,8 +100,8 @@ func (m *AddrBookRecord_AddrEntry) XXX_Marshal(b []byte, deterministic bool) ([] return b[:n], nil } } -func (dst *AddrBookRecord_AddrEntry) XXX_Merge(src proto.Message) { - xxx_messageInfo_AddrBookRecord_AddrEntry.Merge(dst, src) +func (m *AddrBookRecord_AddrEntry) XXX_Merge(src proto.Message) { + xxx_messageInfo_AddrBookRecord_AddrEntry.Merge(m, src) } func (m *AddrBookRecord_AddrEntry) XXX_Size() int { return m.Size() @@ -135,6 +130,29 @@ func init() { proto.RegisterType((*AddrBookRecord)(nil), "pstore.pb.AddrBookRecord") proto.RegisterType((*AddrBookRecord_AddrEntry)(nil), "pstore.pb.AddrBookRecord.AddrEntry") } + +func init() { proto.RegisterFile("pstore.proto", fileDescriptor_f96873690e08a98f) } + +var fileDescriptor_f96873690e08a98f = []byte{ + // 255 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x29, 0x28, 0x2e, 0xc9, + 0x2f, 0x4a, 0xd5, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x84, 0xf1, 0x92, 0xa4, 0x74, 0xd3, + 0x33, 0x4b, 0x32, 0x4a, 0x93, 0xf4, 0x92, 0xf3, 0x73, 0xf5, 0xd3, 0xf3, 0xd3, 0xf3, 0xf5, 0xc1, + 0x2a, 0x92, 0x4a, 0xd3, 0xc0, 0x3c, 0x30, 0x07, 0xcc, 0x82, 0xe8, 0x54, 0x3a, 0xc6, 0xc8, 0xc5, + 0xe7, 0x98, 0x92, 0x52, 0xe4, 0x94, 0x9f, 0x9f, 0x1d, 0x94, 0x9a, 0x9c, 0x5f, 0x94, 0x22, 0x24, + 0xcf, 0xc5, 0x94, 0x99, 0x22, 0xc1, 0xa8, 0xc0, 0xa8, 0xc1, 0xe3, 0xc4, 0x7f, 0xeb, 0x9e, 0x3c, + 0x77, 0x00, 0x48, 0x65, 0x40, 0x6a, 0x6a, 0x91, 0xa7, 0x4b, 0x10, 0x53, 0x66, 0x8a, 0x90, 0x25, + 0x17, 0x6b, 0x62, 0x4a, 0x4a, 0x51, 0xb1, 0x04, 0x93, 0x02, 0xb3, 0x06, 0xb7, 0x91, 0xb2, 0x1e, + 0xdc, 0x76, 0x3d, 0x54, 0xa3, 0xc0, 0x5c, 0xd7, 0xbc, 0x92, 0xa2, 0xca, 0x20, 0x88, 0x0e, 0xa9, + 0x08, 0x2e, 0x4e, 0xb8, 0x98, 0x90, 0x22, 0x17, 0x0b, 0x48, 0x14, 0x6a, 0x15, 0xef, 0xad, 0x7b, + 0xf2, 0x9c, 0x60, 0xab, 0x40, 0x2a, 0x82, 0xc0, 0x52, 0x42, 0x62, 0x5c, 0x6c, 0xa9, 0x15, 0x05, + 0x99, 0x45, 0x95, 0x12, 0x4c, 0x0a, 0x8c, 0x1a, 0xcc, 0x41, 0x50, 0x9e, 0x90, 0x00, 0x17, 0x73, + 0x49, 0x49, 0x8e, 0x04, 0x33, 0x58, 0x10, 0xc4, 0x74, 0x52, 0xf8, 0xf1, 0x50, 0x8e, 0xf1, 0xc0, + 0x23, 0x39, 0xc6, 0x13, 0x8f, 0xe4, 0x18, 0x2f, 0x3c, 0x92, 0x63, 0x7c, 0xf0, 0x48, 0x8e, 0x71, + 0xc2, 0x63, 0x39, 0x86, 0x0b, 0x8f, 0xe5, 0x18, 0x6e, 0x3c, 0x96, 0x63, 0x48, 0x62, 0x03, 0xfb, + 0xd8, 0x18, 0x10, 0x00, 0x00, 0xff, 0xff, 0xb1, 0x1a, 0x16, 0x43, 0x3b, 0x01, 0x00, 0x00, +} + func (m *AddrBookRecord) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) @@ -172,9 +190,6 @@ func (m *AddrBookRecord) MarshalTo(dAtA []byte) (int, error) { i += n } } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) - } return i, nil } @@ -213,9 +228,6 @@ func (m *AddrBookRecord_AddrEntry) MarshalTo(dAtA []byte) (int, error) { i++ i = encodeVarintPstore(dAtA, i, uint64(m.Ttl)) } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) - } return i, nil } @@ -239,7 +251,6 @@ func NewPopulatedAddrBookRecord(r randyPstore, easy bool) *AddrBookRecord { } } if !easy && r.Intn(10) != 0 { - this.XXX_unrecognized = randUnrecognizedPstore(r, 3) } return this } @@ -256,7 +267,6 @@ func NewPopulatedAddrBookRecord_AddrEntry(r randyPstore, easy bool) *AddrBookRec this.Ttl *= -1 } if !easy && r.Intn(10) != 0 { - this.XXX_unrecognized = randUnrecognizedPstore(r, 4) } return this } @@ -349,9 +359,6 @@ func (m *AddrBookRecord) Size() (n int) { n += 1 + l + sovPstore(uint64(l)) } } - if m.XXX_unrecognized != nil { - n += len(m.XXX_unrecognized) - } return n } @@ -371,9 +378,6 @@ func (m *AddrBookRecord_AddrEntry) Size() (n int) { if m.Ttl != 0 { n += 1 + sovPstore(uint64(m.Ttl)) } - if m.XXX_unrecognized != nil { - n += len(m.XXX_unrecognized) - } return n } @@ -405,7 +409,7 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -433,7 +437,7 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -442,6 +446,9 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { return ErrInvalidLengthPstore } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPstore + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -465,7 +472,7 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - msglen |= (int(b) & 0x7F) << shift + msglen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -474,6 +481,9 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { return ErrInvalidLengthPstore } postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthPstore + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -491,10 +501,12 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthPstore } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthPstore + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } - m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) iNdEx += skippy } } @@ -519,7 +531,7 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - wire |= (uint64(b) & 0x7F) << shift + wire |= uint64(b&0x7F) << shift if b < 0x80 { break } @@ -547,7 +559,7 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - byteLen |= (int(b) & 0x7F) << shift + byteLen |= int(b&0x7F) << shift if b < 0x80 { break } @@ -556,6 +568,9 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { return ErrInvalidLengthPstore } postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPstore + } if postIndex > l { return io.ErrUnexpectedEOF } @@ -579,7 +594,7 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - m.Expiry |= (int64(b) & 0x7F) << shift + m.Expiry |= int64(b&0x7F) << shift if b < 0x80 { break } @@ -598,7 +613,7 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { } b := dAtA[iNdEx] iNdEx++ - m.Ttl |= (int64(b) & 0x7F) << shift + m.Ttl |= int64(b&0x7F) << shift if b < 0x80 { break } @@ -612,10 +627,12 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { if skippy < 0 { return ErrInvalidLengthPstore } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthPstore + } if (iNdEx + skippy) > l { return io.ErrUnexpectedEOF } - m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) iNdEx += skippy } } @@ -679,10 +696,13 @@ func skipPstore(dAtA []byte) (n int, err error) { break } } - iNdEx += length if length < 0 { return 0, ErrInvalidLengthPstore } + iNdEx += length + if iNdEx < 0 { + return 0, ErrInvalidLengthPstore + } return iNdEx, nil case 3: for { @@ -711,6 +731,9 @@ func skipPstore(dAtA []byte) (n int, err error) { return 0, err } iNdEx = start + next + if iNdEx < 0 { + return 0, ErrInvalidLengthPstore + } } return iNdEx, nil case 4: @@ -729,25 +752,3 @@ var ( ErrInvalidLengthPstore = fmt.Errorf("proto: negative length found during unmarshaling") ErrIntOverflowPstore = fmt.Errorf("proto: integer overflow") ) - -func init() { proto.RegisterFile("pstore.proto", fileDescriptor_pstore_fcc3073dbc5464a9) } - -var fileDescriptor_pstore_fcc3073dbc5464a9 = []byte{ - // 243 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x29, 0x28, 0x2e, 0xc9, - 0x2f, 0x4a, 0xd5, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x84, 0xf1, 0x92, 0xa4, 0x74, 0xd3, - 0x33, 0x4b, 0x32, 0x4a, 0x93, 0xf4, 0x92, 0xf3, 0x73, 0xf5, 0xd3, 0xf3, 0xd3, 0xf3, 0xf5, 0xc1, - 0x2a, 0x92, 0x4a, 0xd3, 0xc0, 0x3c, 0x30, 0x07, 0xcc, 0x82, 0xe8, 0x54, 0x3a, 0xc6, 0xc8, 0xc5, - 0xe7, 0x98, 0x92, 0x52, 0xe4, 0x94, 0x9f, 0x9f, 0x1d, 0x94, 0x9a, 0x9c, 0x5f, 0x94, 0x22, 0x24, - 0xcf, 0xc5, 0x94, 0x99, 0x22, 0xc1, 0xa8, 0xc0, 0xa8, 0xc1, 0xe3, 0xc4, 0x7f, 0xeb, 0x9e, 0x3c, - 0x77, 0x00, 0x48, 0x65, 0x40, 0x6a, 0x6a, 0x91, 0xa7, 0x4b, 0x10, 0x53, 0x66, 0x8a, 0x90, 0x25, - 0x17, 0x6b, 0x62, 0x4a, 0x4a, 0x51, 0xb1, 0x04, 0x93, 0x02, 0xb3, 0x06, 0xb7, 0x91, 0xb2, 0x1e, - 0xdc, 0x76, 0x3d, 0x54, 0xa3, 0xc0, 0x5c, 0xd7, 0xbc, 0x92, 0xa2, 0xca, 0x20, 0x88, 0x0e, 0xa9, - 0x08, 0x2e, 0x4e, 0xb8, 0x98, 0x90, 0x22, 0x17, 0x0b, 0x48, 0x14, 0x6a, 0x15, 0xef, 0xad, 0x7b, - 0xf2, 0x9c, 0x60, 0xab, 0x40, 0x2a, 0x82, 0xc0, 0x52, 0x42, 0x62, 0x5c, 0x6c, 0xa9, 0x15, 0x05, - 0x99, 0x45, 0x95, 0x12, 0x4c, 0x0a, 0x8c, 0x1a, 0xcc, 0x41, 0x50, 0x9e, 0x90, 0x00, 0x17, 0x73, - 0x49, 0x49, 0x8e, 0x04, 0x33, 0x58, 0x10, 0xc4, 0x74, 0x12, 0xf9, 0xf1, 0x50, 0x8e, 0xf1, 0xc0, - 0x23, 0x39, 0xc6, 0x13, 0x8f, 0xe4, 0x18, 0x2f, 0x3c, 0x92, 0x63, 0x7c, 0xf0, 0x48, 0x8e, 0x31, - 0x89, 0x0d, 0xec, 0x4b, 0x63, 0x40, 0x00, 0x00, 0x00, 0xff, 0xff, 0x2b, 0x91, 0xbf, 0xc2, 0x2f, - 0x01, 0x00, 0x00, -} diff --git a/p2p/host/peerstore/pb/pstorepb_test.go b/p2p/host/peerstore/pb/pstorepb_test.go index 9b7ae039c5..3569d045ff 100644 --- a/p2p/host/peerstore/pb/pstorepb_test.go +++ b/p2p/host/peerstore/pb/pstorepb_test.go @@ -3,13 +3,15 @@ package pstore_pb -import testing "testing" -import math_rand "math/rand" -import github_com_gogo_protobuf_proto "github.com/gogo/protobuf/proto" -import proto "github.com/gogo/protobuf/proto" -import fmt "fmt" -import math "math" -import _ "github.com/gogo/protobuf/gogoproto" +import ( + fmt "fmt" + _ "github.com/gogo/protobuf/gogoproto" + github_com_gogo_protobuf_proto "github.com/gogo/protobuf/proto" + proto "github.com/gogo/protobuf/proto" + math "math" + math_rand "math/rand" + testing "testing" +) // Reference imports to suppress errors if they are not otherwise used. var _ = proto.Marshal From 695d0ce801951bcb81dc68664d6a2bd2a8c09e13 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 20 Feb 2019 20:16:01 -0800 Subject: [PATCH 0941/3965] gx publish 6.0.36 --- .gx/lastpubver | 2 +- package.json | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 80cb9f97f1..22f479ad04 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.35: QmcNGX5RaxPPCYwa6yGXM1EcUbrreTTinixLcYGmMwf1sx +6.0.36: QmcZUhA1xQo8meuYBFLcTHqQb2ogpDCTZUTfTrko7PUeHs diff --git a/package.json b/package.json index e1a84cbde9..de4927a04d 100644 --- a/package.json +++ b/package.json @@ -229,9 +229,9 @@ }, { "author": "vyzo", - "hash": "QmNaJjRsgYQCfmUSZYyk7Jv3Bh78TzKxX4ZwAPKAHfKR4n", + "hash": "QmRgyf3SX5gWEvGRjAgBhUhH6ago2pFRGbML2XWCZuH4mf", "name": "go-libp2p-discovery", - "version": "1.0.10" + "version": "1.0.11" }, { "author": "vyzo", @@ -240,9 +240,9 @@ "version": "1.0.9" }, { - "hash": "QmWaDSNoSdSXU9b6udyaq9T8y6LkzMwqWxECznFqvtcTsk", + "hash": "QmcxZXMqFu4vjLQRfG2tAcg6DPQNurgZ2SQ5iQVk6dXQjn", "name": "go-libp2p-routing", - "version": "2.7.10" + "version": "2.7.11" } ], "gxVersion": "0.4.0", @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.35" + "version": "6.0.36" } From f799512a9738a19b917b4fdd54e25715bacf2908 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 21 Feb 2019 16:20:12 +0800 Subject: [PATCH 0942/3965] derive and save the client's pub key in tls.Config.VerifyPeerCertificate --- p2p/security/tls/crypto.go | 65 +++++++---- p2p/security/tls/transport.go | 60 +++++++--- p2p/security/tls/transport_test.go | 182 ++++++++++++++++++++--------- 3 files changed, 218 insertions(+), 89 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index eb1b7a12e0..272bb49d60 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -8,6 +8,7 @@ import ( "errors" "fmt" "math/big" + "net" "time" ic "github.com/libp2p/go-libp2p-crypto" @@ -15,17 +16,54 @@ import ( peer "github.com/libp2p/go-libp2p-peer" ) +const certValidityPeriod = 180 * 24 * time.Hour + // Identity is used to secure connections type Identity struct { *tls.Config } // NewIdentity creates a new identity -func NewIdentity(privKey ic.PrivKey) (*Identity, error) { - conf, err := generateConfig(privKey) +func NewIdentity( + privKey ic.PrivKey, + verifiedPeerCallback func(net.Conn, ic.PubKey), +) (*Identity, error) { + key, cert, err := keyToCertificate(privKey) if err != nil { return nil, err } + conf := &tls.Config{ + MinVersion: tls.VersionTLS13, + InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. + ClientAuth: tls.RequireAnyClientCert, + Certificates: []tls.Certificate{{ + Certificate: [][]byte{cert.Raw}, + PrivateKey: key, + }}, + } + // When receiving the ClientHello, create a new tls.Config. + // This new config has a VerifyPeerCertificate set, which calls the verifiedPeerCallback + // when we derived the remote's public key from its certificate chain. + conf.GetConfigForClient = func(ch *tls.ClientHelloInfo) (*tls.Config, error) { + c := conf.Clone() + c.VerifyPeerCertificate = func(rawCerts [][]byte, _ [][]*x509.Certificate) error { + chain := make([]*x509.Certificate, len(rawCerts)) + for i := 0; i < len(rawCerts); i++ { + cert, err := x509.ParseCertificate(rawCerts[i]) + if err != nil { + return err + } + chain[i] = cert + } + pubKey, err := getRemotePubKey(chain) + if err != nil { + return err + } + verifiedPeerCallback(ch.Conn, pubKey) + return nil + } + return c, nil + } return &Identity{conf}, nil } @@ -64,24 +102,7 @@ func KeyFromChain(chain []*x509.Certificate) (ic.PubKey, error) { return getRemotePubKey(chain) } -const certValidityPeriod = 180 * 24 * time.Hour - -func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { - key, cert, err := keyToCertificate(privKey) - if err != nil { - return nil, err - } - return &tls.Config{ - MinVersion: tls.VersionTLS13, - InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. - ClientAuth: tls.RequireAnyClientCert, - Certificates: []tls.Certificate{{ - Certificate: [][]byte{cert.Raw}, - PrivateKey: key, - }}, - }, nil -} - +// getRemotePubKey derives the remote's public key from the certificate chain. func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { if len(chain) != 1 { return nil, errors.New("expected one certificates in the chain") @@ -89,7 +110,9 @@ func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { pool := x509.NewCertPool() pool.AddCert(chain[0]) if _, err := chain[0].Verify(x509.VerifyOptions{Roots: pool}); err != nil { - return nil, err + // If we return an x509 error here, it will be sent on the wire. + // Wrap the error to avoid that. + return nil, fmt.Errorf("certificate verification failed: %s", err) } remotePubKey, err := x509.MarshalPKIXPublicKey(chain[0].PublicKey) if err != nil { diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index db820cf0a5..751a461487 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -5,9 +5,11 @@ import ( "crypto/tls" "net" "os" + "sync" cs "github.com/libp2p/go-conn-security" ci "github.com/libp2p/go-libp2p-crypto" + ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" ) @@ -26,6 +28,9 @@ type Transport struct { localPeer peer.ID privKey ci.PrivKey + + incomingMutex sync.Mutex + incoming map[net.Conn]ic.PubKey } // New creates a TLS encrypted transport @@ -34,23 +39,32 @@ func New(key ci.PrivKey) (*Transport, error) { if err != nil { return nil, err } - identity, err := NewIdentity(key) + t := &Transport{ + localPeer: id, + privKey: key, + incoming: make(map[net.Conn]ic.PubKey), + } + identity, err := NewIdentity(key, t.verifiedPeer) if err != nil { return nil, err } - return &Transport{ - identity: identity, - localPeer: id, - privKey: key, - }, nil + t.identity = identity + return t, nil } var _ cs.Transport = &Transport{} // SecureInbound runs the TLS handshake as a server. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Conn, error) { + defer func() { + t.incomingMutex.Lock() + // only contains this connection if we successfully derived the client's key + delete(t.incoming, insecure) + t.incomingMutex.Unlock() + }() + serv := tls.Server(insecure, t.identity.Config) - return t.handshake(ctx, serv) + return t.handshake(ctx, insecure, serv) } // SecureOutbound runs the TLS handshake as a client. @@ -62,10 +76,13 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Co // notice this after 1 RTT when calling Read. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { cl := tls.Client(insecure, t.identity.ConfigForPeer(p)) - return t.handshake(ctx, cl) + return t.handshake(ctx, insecure, cl) } -func (t *Transport) handshake(ctx context.Context, tlsConn *tls.Conn, +func (t *Transport) handshake( + ctx context.Context, + insecure net.Conn, + tlsConn *tls.Conn, ) (cs.Conn, error) { // There's no way to pass a context to tls.Conn.Handshake(). // See https://github.com/golang/go/issues/18482. @@ -92,7 +109,7 @@ func (t *Transport) handshake(ctx context.Context, tlsConn *tls.Conn, } return nil, err } - conn, err := t.setupConn(tlsConn) + conn, err := t.setupConn(insecure, tlsConn) if err != nil { // if the context was canceled, return the context error if ctxErr := ctx.Err(); ctxErr != nil { @@ -103,10 +120,25 @@ func (t *Transport) handshake(ctx context.Context, tlsConn *tls.Conn, return conn, nil } -func (t *Transport) setupConn(tlsConn *tls.Conn) (cs.Conn, error) { - remotePubKey, err := KeyFromChain(tlsConn.ConnectionState().PeerCertificates) - if err != nil { - return nil, err +func (t *Transport) verifiedPeer(conn net.Conn, pubKey ic.PubKey) { + t.incomingMutex.Lock() + t.incoming[conn] = pubKey + t.incomingMutex.Unlock() +} + +func (t *Transport) setupConn(insecure net.Conn, tlsConn *tls.Conn) (cs.Conn, error) { + t.incomingMutex.Lock() + remotePubKey := t.incoming[insecure] + t.incomingMutex.Unlock() + + // This case only occurs for the client. + // Servers already determined the client's key in the VerifyPeerCertificate callback. + if remotePubKey == nil { + var err error + remotePubKey, err = KeyFromChain(tlsConn.ConnectionState().PeerCertificates) + if err != nil { + return nil, err + } } remotePeerID, err := peer.IDFromPublicKey(remotePubKey) if err != nil { diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 3e281e3e1b..c10cbdd187 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -6,9 +6,15 @@ import ( "crypto/elliptic" "crypto/rand" "crypto/rsa" + "crypto/tls" + "crypto/x509" "fmt" + "math/big" mrand "math/rand" "net" + "time" + + "github.com/onsi/gomega/gbytes" cs "github.com/libp2p/go-conn-security" ic "github.com/libp2p/go-libp2p-crypto" @@ -17,6 +23,12 @@ import ( . "github.com/onsi/gomega" ) +type transform struct { + name string + apply func(*Identity) + remoteErr string // the error that the side validating the chain gets +} + var _ = Describe("Transport", func() { var ( serverKey, clientKey ic.PrivKey @@ -58,22 +70,6 @@ var _ = Describe("Transport", func() { return conn, <-serverConnChan } - // modify the cert chain such that verificiation will fail - invalidateCertChain := func(identity *Identity) { - switch identity.Config.Certificates[0].PrivateKey.(type) { - case *rsa.PrivateKey: - key, err := rsa.GenerateKey(rand.Reader, 1024) - Expect(err).ToNot(HaveOccurred()) - identity.Config.Certificates[0].PrivateKey = key - case *ecdsa.PrivateKey: - key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) - Expect(err).ToNot(HaveOccurred()) - identity.Config.Certificates[0].PrivateKey = key - default: - Fail("unexpected private key type") - } - } - BeforeEach(func() { fmt.Fprintf(GinkgoWriter, "Initializing a server") serverID, serverKey = createPeer() @@ -182,50 +178,128 @@ var _ = Describe("Transport", func() { Eventually(done).Should(BeClosed()) }) - It("fails if the client presents an invalid cert chain", func() { - serverTransport, err := New(serverKey) - Expect(err).ToNot(HaveOccurred()) - clientTransport, err := New(clientKey) - Expect(err).ToNot(HaveOccurred()) - invalidateCertChain(clientTransport.identity) + Context("invalid certificates", func() { + invalidateCertChain := func(identity *Identity) { + switch identity.Config.Certificates[0].PrivateKey.(type) { + case *rsa.PrivateKey: + key, err := rsa.GenerateKey(rand.Reader, 1024) + Expect(err).ToNot(HaveOccurred()) + identity.Config.Certificates[0].PrivateKey = key + case *ecdsa.PrivateKey: + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + Expect(err).ToNot(HaveOccurred()) + identity.Config.Certificates[0].PrivateKey = key + default: + Fail("unexpected private key type") + } + } - clientInsecureConn, serverInsecureConn := connect() + twoCerts := func(identity *Identity) { + tmpl := &x509.Certificate{SerialNumber: big.NewInt(1)} + key1, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + Expect(err).ToNot(HaveOccurred()) + key2, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + Expect(err).ToNot(HaveOccurred()) + cert1DER, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, key1.Public(), key1) + Expect(err).ToNot(HaveOccurred()) + cert1, err := x509.ParseCertificate(cert1DER) + Expect(err).ToNot(HaveOccurred()) + cert2DER, err := x509.CreateCertificate(rand.Reader, tmpl, cert1, key2.Public(), key2) + Expect(err).ToNot(HaveOccurred()) + identity.Config.Certificates = []tls.Certificate{{ + Certificate: [][]byte{cert2DER, cert1DER}, + PrivateKey: key2, + }} + } - done := make(chan struct{}) - go func() { - defer GinkgoRecover() - _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) - Expect(err).To(MatchError("tls: invalid certificate signature")) - close(done) - }() + expiredCert := func(identity *Identity) { + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(-time.Minute), + } + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + Expect(err).ToNot(HaveOccurred()) + cert, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, key.Public(), key) + Expect(err).ToNot(HaveOccurred()) + identity.Config.Certificates = []tls.Certificate{{ + Certificate: [][]byte{cert}, + PrivateKey: key, + }} + } - conn, err := clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) - Expect(err).ToNot(HaveOccurred()) - _, err = conn.Read([]byte{0}) - Expect(err).To(MatchError("remote error: tls: error decrypting message")) - Eventually(done).Should(BeClosed()) - }) + transforms := []transform{ + { + name: "private key used in the TLS handshake doesn't match the public key in the cert", + apply: invalidateCertChain, + remoteErr: "tls: invalid certificate signature", + }, + { + name: "certificate chain contains 2 certs", + apply: twoCerts, + remoteErr: "expected one certificates in the chain", + }, + { + name: "cert is expired", + apply: expiredCert, + remoteErr: "certificate verification failed: x509: certificate has expired or is not yet valid", + }, + } - It("fails if the server presents an invalid cert chain", func() { - serverTransport, err := New(serverKey) - Expect(err).ToNot(HaveOccurred()) - invalidateCertChain(serverTransport.identity) - clientTransport, err := New(clientKey) - Expect(err).ToNot(HaveOccurred()) + for i := range transforms { + t := transforms[i] - clientInsecureConn, serverInsecureConn := connect() + It(fmt.Sprintf("fails if the client presents an invalid cert: %s", t.name), func() { + serverTransport, err := New(serverKey) + Expect(err).ToNot(HaveOccurred()) + clientTransport, err := New(clientKey) + Expect(err).ToNot(HaveOccurred()) + t.apply(clientTransport.identity) - done := make(chan struct{}) - go func() { - defer GinkgoRecover() - _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("remote error: tls:")) - close(done) - }() + clientInsecureConn, serverInsecureConn := connect() - _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) - Expect(err).To(MatchError("tls: invalid certificate signature")) - Eventually(done).Should(BeClosed()) + done := make(chan struct{}) + go func() { + defer GinkgoRecover() + _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) + Expect(err).To(MatchError(t.remoteErr)) + close(done) + }() + + conn, err := clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) + Expect(err).ToNot(HaveOccurred()) + _, err = gbytes.TimeoutReader(conn, time.Second).Read([]byte{0}) + Expect(err).To(Or( + // if the certificate's public key doesn't match the private key used for signing + MatchError("remote error: tls: error decrypting message"), + // all other errors + MatchError("remote error: tls: bad certificate"), + )) + Eventually(done).Should(BeClosed()) + }) + + It(fmt.Sprintf("fails if the server presents an invalid cert: %s", t.name), func() { + serverTransport, err := New(serverKey) + Expect(err).ToNot(HaveOccurred()) + t.apply(serverTransport.identity) + clientTransport, err := New(clientKey) + Expect(err).ToNot(HaveOccurred()) + + clientInsecureConn, serverInsecureConn := connect() + + done := make(chan struct{}) + go func() { + defer GinkgoRecover() + _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("remote error: tls:")) + close(done) + }() + + _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) + Expect(err).To(MatchError(t.remoteErr)) + Eventually(done).Should(BeClosed()) + }) + } }) }) From 5677418dda1692ecdbb43968242f03cbe7c6f578 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 21 Feb 2019 16:33:00 +0800 Subject: [PATCH 0943/3965] derive and save the server's pub key in tls.Config.VerifyPeerCertificate --- p2p/security/tls/crypto.go | 11 ++++---- p2p/security/tls/transport.go | 48 +++++++++++++++++------------------ 2 files changed, 29 insertions(+), 30 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 272bb49d60..3b6b425e48 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -69,7 +69,10 @@ func NewIdentity( // ConfigForPeer creates a new tls.Config that verifies the peers certificate chain. // It should be used to create a new tls.Config before dialing. -func (i *Identity) ConfigForPeer(remote peer.ID) *tls.Config { +func (i *Identity) ConfigForPeer( + remote peer.ID, + verifiedPeerCallback func(ic.PubKey), +) *tls.Config { // We need to check the peer ID in the VerifyPeerCertificate callback. // The tls.Config it is also used for listening, and we might also have concurrent dials. // Clone it so we can check for the specific peer ID we're dialing here. @@ -92,16 +95,12 @@ func (i *Identity) ConfigForPeer(remote peer.ID) *tls.Config { if !remote.MatchesPublicKey(pubKey) { return errors.New("peer IDs don't match") } + verifiedPeerCallback(pubKey) return nil } return conf } -// KeyFromChain takes a chain of x509.Certificates and returns the peer's public key. -func KeyFromChain(chain []*x509.Certificate) (ic.PubKey, error) { - return getRemotePubKey(chain) -} - // getRemotePubKey derives the remote's public key from the certificate chain. func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { if len(chain) != 1 { diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 751a461487..dd57b261e8 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -3,6 +3,7 @@ package libp2ptls import ( "context" "crypto/tls" + "errors" "net" "os" "sync" @@ -29,8 +30,8 @@ type Transport struct { localPeer peer.ID privKey ci.PrivKey - incomingMutex sync.Mutex - incoming map[net.Conn]ic.PubKey + activeMutex sync.Mutex + active map[net.Conn]ic.PubKey } // New creates a TLS encrypted transport @@ -42,9 +43,14 @@ func New(key ci.PrivKey) (*Transport, error) { t := &Transport{ localPeer: id, privKey: key, - incoming: make(map[net.Conn]ic.PubKey), + active: make(map[net.Conn]ic.PubKey), } - identity, err := NewIdentity(key, t.verifiedPeer) + + identity, err := NewIdentity(key, func(conn net.Conn, pubKey ic.PubKey) { + t.activeMutex.Lock() + t.active[conn] = pubKey + t.activeMutex.Unlock() + }) if err != nil { return nil, err } @@ -57,10 +63,10 @@ var _ cs.Transport = &Transport{} // SecureInbound runs the TLS handshake as a server. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Conn, error) { defer func() { - t.incomingMutex.Lock() + t.activeMutex.Lock() // only contains this connection if we successfully derived the client's key - delete(t.incoming, insecure) - t.incomingMutex.Unlock() + delete(t.active, insecure) + t.activeMutex.Unlock() }() serv := tls.Server(insecure, t.identity.Config) @@ -75,7 +81,12 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Co // If the handshake fails, the server will close the connection. The client will // notice this after 1 RTT when calling Read. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { - cl := tls.Client(insecure, t.identity.ConfigForPeer(p)) + verifiedCallback := func(pubKey ic.PubKey) { + t.activeMutex.Lock() + t.active[insecure] = pubKey + t.activeMutex.Unlock() + } + cl := tls.Client(insecure, t.identity.ConfigForPeer(p, verifiedCallback)) return t.handshake(ctx, insecure, cl) } @@ -120,26 +131,15 @@ func (t *Transport) handshake( return conn, nil } -func (t *Transport) verifiedPeer(conn net.Conn, pubKey ic.PubKey) { - t.incomingMutex.Lock() - t.incoming[conn] = pubKey - t.incomingMutex.Unlock() -} - func (t *Transport) setupConn(insecure net.Conn, tlsConn *tls.Conn) (cs.Conn, error) { - t.incomingMutex.Lock() - remotePubKey := t.incoming[insecure] - t.incomingMutex.Unlock() + t.activeMutex.Lock() + remotePubKey := t.active[insecure] + t.activeMutex.Unlock() - // This case only occurs for the client. - // Servers already determined the client's key in the VerifyPeerCertificate callback. if remotePubKey == nil { - var err error - remotePubKey, err = KeyFromChain(tlsConn.ConnectionState().PeerCertificates) - if err != nil { - return nil, err - } + return nil, errors.New("go-libp2p-tls BUG: expected remote pub key to be set") } + remotePeerID, err := peer.IDFromPublicKey(remotePubKey) if err != nil { return nil, err From 3a30a4ed85fd268fd674b98dca9054d124f85752 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 21 Feb 2019 19:31:19 +0000 Subject: [PATCH 0944/3965] pstoreds: remove dirty flag when deleting from db. --- p2p/host/peerstore/pstoreds/addr_book.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 760ff58e42..9690bd5a4d 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -48,7 +48,10 @@ type addrsRecord struct { func (r *addrsRecord) flush(write ds.Write) (err error) { key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id.ID))) if len(r.Addrs) == 0 { - return write.Delete(key) + if err = write.Delete(key); err == nil { + r.dirty = false + } + return err } data, err := r.Marshal() From 6a955caf4b3cc0d121afca12ad18eafb3284ceae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 22 Feb 2019 14:39:51 +0000 Subject: [PATCH 0945/3965] pstore.Close(): return errors from children. --- p2p/host/peerstore/peerstore.go | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index c61b63c174..40411fb2ce 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -33,14 +33,21 @@ func NewPeerstore(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { } func (ps *peerstore) Close() (err error) { - if cl, ok := ps.KeyBook.(io.Closer); ok { - cl.Close() - } - if cl, ok := ps.AddrBook.(io.Closer); ok { - cl.Close() + var errs []error + weakClose := func(name string, c interface{}) { + if cl, ok := c.(io.Closer); ok { + if err = cl.Close(); err != nil { + errs = append(errs, fmt.Errorf("%s error: %s", name, err)) + } + } } - if cl, ok := ps.PeerMetadata.(io.Closer); ok { - cl.Close() + + weakClose("keybook", ps.KeyBook) + weakClose("addressbook", ps.AddrBook) + weakClose("peermetadata", ps.PeerMetadata) + + if len(errs) > 0 { + return fmt.Errorf("failed while closing peerstore; err(s): %q", errs) } return nil } From 562d54ff157c5f57e2c306c634bef8a29ead2cf3 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Tue, 26 Feb 2019 17:03:57 +0000 Subject: [PATCH 0946/3965] gx: update go-libp2p-peer --- package.json | 88 ++++++++++++++++++++++++++-------------------------- 1 file changed, 44 insertions(+), 44 deletions(-) diff --git a/package.json b/package.json index de4927a04d..9589cfc5d1 100644 --- a/package.json +++ b/package.json @@ -50,33 +50,33 @@ }, { "author": "whyrusleeping", - "hash": "QmcXCcD4dC51dkBWSvDQP1ySyzZUn6hbL9jkzJfYQc6VZB", + "hash": "QmYnRuq7khXcMRxUq334ytHGAXMQhmEAi9GTZyFdPv1oU8", "name": "go-libp2p-loggables", - "version": "1.1.31" + "version": "1.1.32" }, { "author": "whyrusleeping", - "hash": "QmV1XUxssTbBHiVMC8Nc4LyDJGNBFMB31jH3xw64m2C1nj", + "hash": "QmdKtgri3kBaoipc9rsuvJkG8u3cVxThRRWm9dr36gsBnZ", "name": "go-libp2p-secio", - "version": "2.0.28" + "version": "2.0.29" }, { "author": "whyrusleeping", - "hash": "QmRhFARzTHcFh8wUxwN5KvyTGq73FLC65EfFAhz8Ng7aGb", + "hash": "QmTGp1mdZiTegNyGHgPbTHMaxgFbmQ6BMBQHUBKkaYcLGX", "name": "go-libp2p-peerstore", - "version": "2.0.17" + "version": "2.0.18" }, { "author": "whyrusleeping", - "hash": "QmVcZvoaUGfXHzGJ5mFvAeCouL9SET2M7U52zZaEPyk2o5", + "hash": "QmTJ6xe8A2oSA1kSvAbpvK1AvciBzNkuTST9sEigtm8TMh", "name": "go-libp2p-transport", - "version": "3.0.25" + "version": "3.0.26" }, { "author": "whyrusleeping", - "hash": "QmRUAf47dQEwdpZDhHDcWr5MD6p94ii6fv5VZph9a8sc72", + "hash": "QmZxnV9fQ2iw6U4AJcZXQGMQNLfvHK1HeoVFRo4sC7dUbA", "name": "go-tcp-transport", - "version": "2.0.25" + "version": "2.0.26" }, { "author": "whyrusleeping", @@ -92,33 +92,33 @@ }, { "author": "whyrusleeping", - "hash": "QmeFVdhzY13YZPWxCiQvmLercrumFRoQZFQEYw2BtzyiQc", + "hash": "QmQkRQ28FCmjByD5RvYjM7WRJvfvGSVqZyMJBQYGEWxvfc", "name": "go-testutil", - "version": "1.2.17" + "version": "1.2.18" }, { "author": "whyrusleeping", - "hash": "QmTGxDz2CjBucFzPNTiWwzQmTWdrBnzqbqrMucDYMsjuPb", + "hash": "QmTrXnjUsyc5KzeAVKU1LWZGCVsVA3mn1UcLPSXpR9K6y1", "name": "go-libp2p-net", - "version": "3.0.28" + "version": "3.0.29" }, { "author": "whyrusleeping", - "hash": "QmZZseAa9xcK6tT3YpaShNUAEpyRAoWmUL5ojH3uGNepAc", + "hash": "QmSwVwKUWzdf3ppM3FbBbpuqHUNtUFJPQQdfvKmgZoz2gR", "name": "go-libp2p-metrics", - "version": "2.1.13" + "version": "2.1.14" }, { "author": "whyrusleeping", - "hash": "Qmd52WKRSwrBK5gUaJKawryZQ5by6UbNB8KVW2Zy6JtbyW", + "hash": "QmXV4Faq9pyDJG8b7JTFy2puokMEGXb8cEAq6rS18GiYsW", "name": "go-libp2p-host", - "version": "3.0.24" + "version": "3.0.25" }, { "author": "whyrusleeping", - "hash": "QmU7iTrsNaJfu1Rf5DrvaJLH9wJtQwmP4Dj8oPduprAU68", + "hash": "QmWscwo9rCqNBnAfLd2BHEfXnrLmQmN9f5GtoVS7MBLExC", "name": "go-libp2p-swarm", - "version": "3.0.31" + "version": "3.0.32" }, { "author": "whyrusleeping", @@ -128,15 +128,15 @@ }, { "author": "whyrusleeping", - "hash": "QmUD9zcPC7rkiyp3KWBnBor7GcpU671DC8F6hBnZZUnrfv", + "hash": "QmaBYvrh9ioGCDvuV7dn9jVbLw6r7tVEBA1t2hh5ctxUpK", "name": "go-libp2p-netutil", - "version": "0.4.21" + "version": "0.4.22" }, { "author": "whyrusleeping", - "hash": "QmSka7Ax3a1HhyGaC3mqqrWbDhHsTmyzvED4Xk6dfvPXbY", + "hash": "QmPMqaBSAptBEQr5DdgxepYioXNKcE1rqaMMbUnBpDDMtV", "name": "go-libp2p-blankhost", - "version": "0.3.24" + "version": "0.3.25" }, { "author": "whyrusleeping", @@ -158,15 +158,15 @@ }, { "author": "whyrusleeping", - "hash": "QmTu65MVbemtUxJEWgsTtzv9Zv9P8rvmqNA4eG9TrTRGYc", + "hash": "QmYVXrKrKHDC9FobgmcmshCDyWwdrfwfanNQN4oxJ9Fk3h", "name": "go-libp2p-peer", - "version": "3.1.1" + "version": "3.1.2" }, { "author": "vyzo", - "hash": "QmNaXXRfJ93t4HicX8N2WZPhdE8KU39MPGALuH421GFgKA", + "hash": "QmYPqgHSFGdQZEJ62JkRyt2a9PhLJxHeQZbh4QbsnggvNr", "name": "go-libp2p-circuit", - "version": "2.3.11" + "version": "2.3.12" }, { "author": "lgierth", @@ -176,9 +176,9 @@ }, { "author": "why", - "hash": "QmcCk4LZRJPAKuwY9dusFea7LckELZgo5HagErTbm39o38", + "hash": "Qma3q582QVokfmfPCJyZKeqXdqsMKP9C3QEQVUEiaHbxsQ", "name": "go-libp2p-interface-connmgr", - "version": "0.0.30" + "version": "0.0.31" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmdKcuSPQDK7FkG5k3TYscKMaL9BQ122eEiKLqCKC8mEZf", + "hash": "QmNn8et1QAAo5sYrPoddyoSvpSf11pdXVTVJmNkUiChBwd", "name": "go-ws-transport", - "version": "2.0.24" + "version": "2.0.25" }, { "author": "stebalien", - "hash": "QmS81oMV7WjkXt8AdbeYTBL8Vn2tKmyoa5U7Lnk3FjnyPc", + "hash": "QmejRRt5BFzSFzzXaA3QH6D2osQ9UaozTJEDRLqhxSQzRt", "name": "go-conn-security-multistream", - "version": "0.1.24" + "version": "0.1.25" }, { "author": "Stebalien", - "hash": "QmWdufxe3UrFtycBBj1895k4xnUWCbkuGbmBxtT7N3smjm", + "hash": "QmdEByBKgZYNCLyFg5Wm9eBWviqZvJNXLBAWU876LfvAjn", "name": "go-conn-security", - "version": "0.1.26" + "version": "0.1.27" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "QmePVH87Z1oTzrAn5vkPwDTNrieSYCqq2AQ99nxZ1fWJ7m", + "hash": "QmVQiWpN6mNkyfaJRFryT5SQ382i7qXKhePaTXVHUuRcLK", "name": "go-libp2p-transport-upgrader", - "version": "0.1.25" + "version": "0.1.26" }, { "hash": "QmddjPSGZb3ieihSseFeCfVRpZzcqczPNsD2DvarSwnjJB", @@ -229,20 +229,20 @@ }, { "author": "vyzo", - "hash": "QmRgyf3SX5gWEvGRjAgBhUhH6ago2pFRGbML2XWCZuH4mf", + "hash": "QmeXAxhhhrmJfCqTxCwcAR7uktnoehbFHprqusongvsR9z", "name": "go-libp2p-discovery", - "version": "1.0.11" + "version": "1.0.12" }, { "author": "vyzo", - "hash": "QmckHMRXmzTnq8YUNhNtohCzKBHiaPYqnbdvp87rjCmCi6", + "hash": "QmPEHgKDk48ueyS3iEGk5tpGdi7jhgJ1H6qEZArVc81Dut", "name": "go-libp2p-autonat", - "version": "1.0.9" + "version": "1.0.10" }, { - "hash": "QmcxZXMqFu4vjLQRfG2tAcg6DPQNurgZ2SQ5iQVk6dXQjn", + "hash": "QmeJ6bdyeMyNdR1Aputc9u5q2XSdUUnQuNqmQVLu569ePx", "name": "go-libp2p-routing", - "version": "2.7.11" + "version": "2.7.12" } ], "gxVersion": "0.4.0", From 6c69405fdacb0f8453c09305a2de61e3f599da81 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Tue, 26 Feb 2019 17:52:10 +0000 Subject: [PATCH 0947/3965] gx publish 6.0.37 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 22f479ad04..5ce21438fa 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.36: QmcZUhA1xQo8meuYBFLcTHqQb2ogpDCTZUTfTrko7PUeHs +6.0.37: QmUkFGSeAoTPjN4yBnUNCAgP1Go1rJ5m9b42TQYsSQ5cy1 diff --git a/package.json b/package.json index 9589cfc5d1..a7a0f8227d 100644 --- a/package.json +++ b/package.json @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.36" + "version": "6.0.37" } From 8ba018c7285107007dc4e8588bf5eee55e04c90d Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Tue, 26 Feb 2019 20:05:37 +0000 Subject: [PATCH 0948/3965] gx: update go-multiaddr --- package.json | 100 +++++++++++++++++++++++++-------------------------- 1 file changed, 50 insertions(+), 50 deletions(-) diff --git a/package.json b/package.json index a7a0f8227d..71e0b63cd4 100644 --- a/package.json +++ b/package.json @@ -39,50 +39,50 @@ "version": "1.5.9" }, { - "hash": "QmZcLBXKaFe8ND5YHPkJRAwmhJGrVsi1JqDZNyJ4nRK5Mj", + "hash": "Qmc85NSvmSG4Frn9Vb2cBc1rMyULH6D3TNVEfCzSKoUpip", "name": "go-multiaddr-net", - "version": "1.7.1" + "version": "1.7.2" }, { - "hash": "QmNTCey11oxhb1AxDnQBRHtdhap6Ctud872NjAYPYYXPuc", + "hash": "QmTZBfrPJmjWsCvHEtX5FE6KimVJhsJg5sBbqEFYf4UZtL", "name": "go-multiaddr", - "version": "1.4.0" + "version": "1.4.1" }, { "author": "whyrusleeping", - "hash": "QmYnRuq7khXcMRxUq334ytHGAXMQhmEAi9GTZyFdPv1oU8", + "hash": "QmUbSLukzZYZvEYxynj9Dtd1WrGLxxg9R4U68vCMPWHmRU", "name": "go-libp2p-loggables", - "version": "1.1.32" + "version": "1.1.33" }, { "author": "whyrusleeping", - "hash": "QmdKtgri3kBaoipc9rsuvJkG8u3cVxThRRWm9dr36gsBnZ", + "hash": "QmSVaJe1aRjc78cZARTtf4pqvXERYwihyYhZWoVWceHnsK", "name": "go-libp2p-secio", - "version": "2.0.29" + "version": "2.0.30" }, { "author": "whyrusleeping", - "hash": "QmTGp1mdZiTegNyGHgPbTHMaxgFbmQ6BMBQHUBKkaYcLGX", + "hash": "QmaCTz9RkrU13bm9kMB54f7atgqM4qkjDZpRwRoJiWXEqs", "name": "go-libp2p-peerstore", - "version": "2.0.18" + "version": "2.0.19" }, { "author": "whyrusleeping", - "hash": "QmTJ6xe8A2oSA1kSvAbpvK1AvciBzNkuTST9sEigtm8TMh", + "hash": "QmNQWMWWBmkAcaVEspSNwYB95axzKFhYTdqZtABA2zXoPu", "name": "go-libp2p-transport", - "version": "3.0.26" + "version": "3.0.27" }, { "author": "whyrusleeping", - "hash": "QmZxnV9fQ2iw6U4AJcZXQGMQNLfvHK1HeoVFRo4sC7dUbA", + "hash": "QmUv23eH7fcpNNugwYy6MgoU4p3JoTxXtRLiZqQCdMbxeE", "name": "go-tcp-transport", - "version": "2.0.26" + "version": "2.0.27" }, { "author": "whyrusleeping", - "hash": "QmQgSnRC74nHoXrN9CShvfWUUSrgAMJ4unjbnuBVsxk2mw", + "hash": "QmT6C5ebDy92zyRzdmSNyda5q7zkNXy68X47RDJiHpvaxd", "name": "go-maddr-filter", - "version": "1.1.12" + "version": "1.1.13" }, { "author": "whyrusleeping", @@ -92,15 +92,15 @@ }, { "author": "whyrusleeping", - "hash": "QmQkRQ28FCmjByD5RvYjM7WRJvfvGSVqZyMJBQYGEWxvfc", + "hash": "QmWapVoHjtKhn4MhvKNoPTkJKADFGACfXPFnt7combwp5W", "name": "go-testutil", - "version": "1.2.18" + "version": "1.2.19" }, { "author": "whyrusleeping", - "hash": "QmTrXnjUsyc5KzeAVKU1LWZGCVsVA3mn1UcLPSXpR9K6y1", + "hash": "QmY3ArotKMKaL7YGfbQfyDrib6RVraLqZYWXZvVgZktBxp", "name": "go-libp2p-net", - "version": "3.0.29" + "version": "3.0.30" }, { "author": "whyrusleeping", @@ -110,33 +110,33 @@ }, { "author": "whyrusleeping", - "hash": "QmXV4Faq9pyDJG8b7JTFy2puokMEGXb8cEAq6rS18GiYsW", + "hash": "QmYrWiWM4qtrnCeT3R14jY3ZZyirDNJgwK57q4qFYePgbd", "name": "go-libp2p-host", - "version": "3.0.25" + "version": "3.0.26" }, { "author": "whyrusleeping", - "hash": "QmWscwo9rCqNBnAfLd2BHEfXnrLmQmN9f5GtoVS7MBLExC", + "hash": "QmX9A6whepz59nU5jU9fbVQGkZQspdWkvcpP7gCihoKnGS", "name": "go-libp2p-swarm", - "version": "3.0.32" + "version": "3.0.33" }, { "author": "whyrusleeping", - "hash": "QmQhG54sm28e44HfGjG37F68Tzu45f18qimgdyJRa5VR8a", + "hash": "QmZ1zCb95y9oHaJRMQmbXh3FgUuwz1V2mbCXBztnhehJkL", "name": "go-libp2p-nat", - "version": "0.8.11" + "version": "0.8.12" }, { "author": "whyrusleeping", - "hash": "QmaBYvrh9ioGCDvuV7dn9jVbLw6r7tVEBA1t2hh5ctxUpK", + "hash": "QmQzyPDx8rLqJ5evnHAo4Mpbsb64RKGZzLABzsbXQM6a2j", "name": "go-libp2p-netutil", - "version": "0.4.22" + "version": "0.4.23" }, { "author": "whyrusleeping", - "hash": "QmPMqaBSAptBEQr5DdgxepYioXNKcE1rqaMMbUnBpDDMtV", + "hash": "QmcBTHN7uAMBdkzRoQ3n9cE7tGu8Ubd9zmahjskjTRw4Uf", "name": "go-libp2p-blankhost", - "version": "0.3.25" + "version": "0.3.26" }, { "author": "whyrusleeping", @@ -164,21 +164,21 @@ }, { "author": "vyzo", - "hash": "QmYPqgHSFGdQZEJ62JkRyt2a9PhLJxHeQZbh4QbsnggvNr", + "hash": "QmYiV5fLw3q8reSx3WgnKAJsi8DmNz6vdnNiXaSmiRDN3m", "name": "go-libp2p-circuit", - "version": "2.3.12" + "version": "2.3.13" }, { "author": "lgierth", - "hash": "QmQc7jbDUsxUJZyFJzxVrnrWeECCct6fErEpMqtjyWvCX8", + "hash": "QmU98UaAEh4WJAcir2qjfztU77JQ14kAwHNFkjUXHZA3Vy", "name": "go-multiaddr-dns", - "version": "0.3.0" + "version": "0.3.1" }, { "author": "why", - "hash": "Qma3q582QVokfmfPCJyZKeqXdqsMKP9C3QEQVUEiaHbxsQ", + "hash": "QmXa6sgzUvP5bgF5CyyV36bZYv5VDRwttggQYUPvFybLVd", "name": "go-libp2p-interface-connmgr", - "version": "0.0.31" + "version": "0.0.32" }, { "author": "whyrusleeping", @@ -188,21 +188,21 @@ }, { "author": "whyrusleeping", - "hash": "QmNn8et1QAAo5sYrPoddyoSvpSf11pdXVTVJmNkUiChBwd", + "hash": "QmXqtBCF1fuQz9yQdVZuCBBdDkREWAkCJSZ37FCXfAY6gn", "name": "go-ws-transport", - "version": "2.0.25" + "version": "2.0.26" }, { "author": "stebalien", - "hash": "QmejRRt5BFzSFzzXaA3QH6D2osQ9UaozTJEDRLqhxSQzRt", + "hash": "QmZWmFkMm28sWeDr5Xh1LexdKBGYGp946MNCfgtLqfX73z", "name": "go-conn-security-multistream", - "version": "0.1.25" + "version": "0.1.26" }, { "author": "Stebalien", - "hash": "QmdEByBKgZYNCLyFg5Wm9eBWviqZvJNXLBAWU876LfvAjn", + "hash": "QmPRoHqULmP4MuKAN5EFaJ64MLpeMY8cny2318xDBDmmkp", "name": "go-conn-security", - "version": "0.1.27" + "version": "0.1.28" }, { "author": "libp2p", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "QmVQiWpN6mNkyfaJRFryT5SQ382i7qXKhePaTXVHUuRcLK", + "hash": "QmNjhxwzKp9uB1T2p4PBW3xi7pboJ7iXKvAHQW92ayqAm2", "name": "go-libp2p-transport-upgrader", - "version": "0.1.26" + "version": "0.1.27" }, { "hash": "QmddjPSGZb3ieihSseFeCfVRpZzcqczPNsD2DvarSwnjJB", @@ -229,20 +229,20 @@ }, { "author": "vyzo", - "hash": "QmeXAxhhhrmJfCqTxCwcAR7uktnoehbFHprqusongvsR9z", + "hash": "QmafAuXdx5RUfygWnd7PNXjTN3GJYtxQtYEnsgMzQDKNDt", "name": "go-libp2p-discovery", - "version": "1.0.12" + "version": "1.0.13" }, { "author": "vyzo", - "hash": "QmPEHgKDk48ueyS3iEGk5tpGdi7jhgJ1H6qEZArVc81Dut", + "hash": "QmYPCgEBteaGHifXGnp8TjA4hoVgiEMjvpE4D4J8hAtYnx", "name": "go-libp2p-autonat", - "version": "1.0.10" + "version": "1.0.11" }, { - "hash": "QmeJ6bdyeMyNdR1Aputc9u5q2XSdUUnQuNqmQVLu569ePx", + "hash": "QmYxUdYY9S6yg5tSPVin5GFTvtfsLauVcr7reHDD3dM8xf", "name": "go-libp2p-routing", - "version": "2.7.12" + "version": "2.7.13" } ], "gxVersion": "0.4.0", From e65ae1d2ea02546c964ca864cf786c6faac3d223 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Tue, 26 Feb 2019 20:05:59 +0000 Subject: [PATCH 0949/3965] gx publish 6.0.38 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 5ce21438fa..e90efd6660 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.37: QmUkFGSeAoTPjN4yBnUNCAgP1Go1rJ5m9b42TQYsSQ5cy1 +6.0.38: QmWRUZmLb9qEpwuHTtrzbdE5LQxm64qftncw5o8tBVPobL diff --git a/package.json b/package.json index 71e0b63cd4..c29c355c94 100644 --- a/package.json +++ b/package.json @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.37" + "version": "6.0.38" } From c4577c770392f9b24753da6e0863da2466683427 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 27 Feb 2019 11:52:36 -0800 Subject: [PATCH 0950/3965] gx publish 6.0.39 --- .gx/lastpubver | 2 +- package.json | 34 +++++++++++++++++----------------- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index e90efd6660..1f329afb6a 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.38: QmWRUZmLb9qEpwuHTtrzbdE5LQxm64qftncw5o8tBVPobL +6.0.39: QmRxk6AUaGaKCfzS1xSNRojiAPd7h2ih8GuCdjJBF3Y6GK diff --git a/package.json b/package.json index c29c355c94..4ef9c074d9 100644 --- a/package.json +++ b/package.json @@ -74,9 +74,9 @@ }, { "author": "whyrusleeping", - "hash": "QmUv23eH7fcpNNugwYy6MgoU4p3JoTxXtRLiZqQCdMbxeE", + "hash": "QmTGiDkw4eeKq31wwpQRk5GwWiReaxrcTQLuCCLWgfKo5M", "name": "go-tcp-transport", - "version": "2.0.27" + "version": "2.0.28" }, { "author": "whyrusleeping", @@ -116,9 +116,9 @@ }, { "author": "whyrusleeping", - "hash": "QmX9A6whepz59nU5jU9fbVQGkZQspdWkvcpP7gCihoKnGS", + "hash": "Qma3Xp3FXFSP4prirEiRYHJ2tgGE8EAx9i6JLziPLpAQjq", "name": "go-libp2p-swarm", - "version": "3.0.33" + "version": "3.0.34" }, { "author": "whyrusleeping", @@ -164,9 +164,9 @@ }, { "author": "vyzo", - "hash": "QmYiV5fLw3q8reSx3WgnKAJsi8DmNz6vdnNiXaSmiRDN3m", + "hash": "QmZBfqr863PYD7BKbmCFSNmzsqYmtr2DKgzubsQaiTQkMc", "name": "go-libp2p-circuit", - "version": "2.3.13" + "version": "2.3.14" }, { "author": "lgierth", @@ -182,15 +182,15 @@ }, { "author": "whyrusleeping", - "hash": "QmQ4TZxzazF9dfqiH52ygGGU4BHQQTR9aCGMgHt4tCRdEA", + "hash": "QmaJvNdDccVkTELQLCGXWrLxgaQ14aMdhzZx1EiHPXKbDc", "name": "go-smux-multiplex", - "version": "3.0.18" + "version": "3.0.19" }, { "author": "whyrusleeping", - "hash": "QmXqtBCF1fuQz9yQdVZuCBBdDkREWAkCJSZ37FCXfAY6gn", + "hash": "QmaSWc4ox6SZQF6DHZvDuM9sP1syNajkKuPXmKR1t5BAz5", "name": "go-ws-transport", - "version": "2.0.26" + "version": "2.0.27" }, { "author": "stebalien", @@ -218,9 +218,9 @@ }, { "author": "steb", - "hash": "QmNjhxwzKp9uB1T2p4PBW3xi7pboJ7iXKvAHQW92ayqAm2", + "hash": "QmeqC5shQjEBRG9B8roZqQCJ9xb7Pq6AbWxJFMyLgqBBWh", "name": "go-libp2p-transport-upgrader", - "version": "0.1.27" + "version": "0.1.28" }, { "hash": "QmddjPSGZb3ieihSseFeCfVRpZzcqczPNsD2DvarSwnjJB", @@ -229,15 +229,15 @@ }, { "author": "vyzo", - "hash": "QmafAuXdx5RUfygWnd7PNXjTN3GJYtxQtYEnsgMzQDKNDt", + "hash": "QmWXhsJTd4eTVAy9n8mDYiFmcMv1VHJ73qGkkeDHZfDhui", "name": "go-libp2p-discovery", - "version": "1.0.13" + "version": "1.0.14" }, { "author": "vyzo", - "hash": "QmYPCgEBteaGHifXGnp8TjA4hoVgiEMjvpE4D4J8hAtYnx", + "hash": "QmZjvy5eqc1T9VbFe5KBiDryUJSRNbSbHZD7JMc6CeUXuu", "name": "go-libp2p-autonat", - "version": "1.0.11" + "version": "1.0.12" }, { "hash": "QmYxUdYY9S6yg5tSPVin5GFTvtfsLauVcr7reHDD3dM8xf", @@ -250,6 +250,6 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.38" + "version": "6.0.39" } From 92fedfe742a462d368bc98d1ecf730be2833f48e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 27 Feb 2019 12:31:03 -0800 Subject: [PATCH 0951/3965] make peer verification use a channel --- p2p/security/tls/crypto.go | 83 +++++++++++++----------------- p2p/security/tls/transport.go | 49 ++++++------------ p2p/security/tls/transport_test.go | 10 ++-- 3 files changed, 57 insertions(+), 85 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 3b6b425e48..0c31bf5d52 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -8,7 +8,6 @@ import ( "errors" "fmt" "math/big" - "net" "time" ic "github.com/libp2p/go-libp2p-crypto" @@ -20,66 +19,55 @@ const certValidityPeriod = 180 * 24 * time.Hour // Identity is used to secure connections type Identity struct { - *tls.Config + config tls.Config } // NewIdentity creates a new identity -func NewIdentity( - privKey ic.PrivKey, - verifiedPeerCallback func(net.Conn, ic.PubKey), -) (*Identity, error) { +func NewIdentity(privKey ic.PrivKey) (*Identity, error) { key, cert, err := keyToCertificate(privKey) if err != nil { return nil, err } - conf := &tls.Config{ - MinVersion: tls.VersionTLS13, - InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. - ClientAuth: tls.RequireAnyClientCert, - Certificates: []tls.Certificate{{ - Certificate: [][]byte{cert.Raw}, - PrivateKey: key, - }}, - } - // When receiving the ClientHello, create a new tls.Config. - // This new config has a VerifyPeerCertificate set, which calls the verifiedPeerCallback - // when we derived the remote's public key from its certificate chain. - conf.GetConfigForClient = func(ch *tls.ClientHelloInfo) (*tls.Config, error) { - c := conf.Clone() - c.VerifyPeerCertificate = func(rawCerts [][]byte, _ [][]*x509.Certificate) error { - chain := make([]*x509.Certificate, len(rawCerts)) - for i := 0; i < len(rawCerts); i++ { - cert, err := x509.ParseCertificate(rawCerts[i]) - if err != nil { - return err - } - chain[i] = cert - } - pubKey, err := getRemotePubKey(chain) - if err != nil { - return err - } - verifiedPeerCallback(ch.Conn, pubKey) - return nil - } - return c, nil - } - return &Identity{conf}, nil + return &Identity{ + config: tls.Config{ + MinVersion: tls.VersionTLS13, + InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. + ClientAuth: tls.RequireAnyClientCert, + Certificates: []tls.Certificate{{ + Certificate: [][]byte{cert.Raw}, + PrivateKey: key, + }}, + VerifyPeerCertificate: func(_ [][]byte, _ [][]*x509.Certificate) error { + panic("tls config not specialized for peer") + }, + }, + }, nil +} + +// ConfigForAny is a short-hand for ConfigForPeer(""). +func (i *Identity) ConfigForAny() (*tls.Config, <-chan ic.PubKey) { + return i.ConfigForPeer("") } -// ConfigForPeer creates a new tls.Config that verifies the peers certificate chain. -// It should be used to create a new tls.Config before dialing. +// ConfigForPeer creates a new single-use tls.Config that verifies the peer's +// certificate chain and returns the peer's public key via the channel. If the +// peer ID is empty, the returned config will accept any peer. +// +// It should be used to create a new tls.Config before securing either an +// incoming or outgoing connection. func (i *Identity) ConfigForPeer( remote peer.ID, - verifiedPeerCallback func(ic.PubKey), -) *tls.Config { +) (*tls.Config, <-chan ic.PubKey) { + keyCh := make(chan ic.PubKey, 1) // We need to check the peer ID in the VerifyPeerCertificate callback. // The tls.Config it is also used for listening, and we might also have concurrent dials. // Clone it so we can check for the specific peer ID we're dialing here. - conf := i.Config.Clone() + conf := i.config.Clone() // We're using InsecureSkipVerify, so the verifiedChains parameter will always be empty. // We need to parse the certificates ourselves from the raw certs. conf.VerifyPeerCertificate = func(rawCerts [][]byte, _ [][]*x509.Certificate) error { + defer close(keyCh) + chain := make([]*x509.Certificate, len(rawCerts)) for i := 0; i < len(rawCerts); i++ { cert, err := x509.ParseCertificate(rawCerts[i]) @@ -88,17 +76,18 @@ func (i *Identity) ConfigForPeer( } chain[i] = cert } + pubKey, err := getRemotePubKey(chain) if err != nil { return err } - if !remote.MatchesPublicKey(pubKey) { + if remote != "" && !remote.MatchesPublicKey(pubKey) { return errors.New("peer IDs don't match") } - verifiedPeerCallback(pubKey) + keyCh <- pubKey return nil } - return conf + return conf, keyCh } // getRemotePubKey derives the remote's public key from the certificate chain. diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index dd57b261e8..6301f8897c 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -6,7 +6,6 @@ import ( "errors" "net" "os" - "sync" cs "github.com/libp2p/go-conn-security" ci "github.com/libp2p/go-libp2p-crypto" @@ -29,9 +28,6 @@ type Transport struct { localPeer peer.ID privKey ci.PrivKey - - activeMutex sync.Mutex - active map[net.Conn]ic.PubKey } // New creates a TLS encrypted transport @@ -43,14 +39,9 @@ func New(key ci.PrivKey) (*Transport, error) { t := &Transport{ localPeer: id, privKey: key, - active: make(map[net.Conn]ic.PubKey), } - identity, err := NewIdentity(key, func(conn net.Conn, pubKey ic.PubKey) { - t.activeMutex.Lock() - t.active[conn] = pubKey - t.activeMutex.Unlock() - }) + identity, err := NewIdentity(key) if err != nil { return nil, err } @@ -62,15 +53,8 @@ var _ cs.Transport = &Transport{} // SecureInbound runs the TLS handshake as a server. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Conn, error) { - defer func() { - t.activeMutex.Lock() - // only contains this connection if we successfully derived the client's key - delete(t.active, insecure) - t.activeMutex.Unlock() - }() - - serv := tls.Server(insecure, t.identity.Config) - return t.handshake(ctx, insecure, serv) + config, keyCh := t.identity.ConfigForAny() + return t.handshake(ctx, tls.Server(insecure, config), keyCh) } // SecureOutbound runs the TLS handshake as a client. @@ -81,19 +65,14 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Co // If the handshake fails, the server will close the connection. The client will // notice this after 1 RTT when calling Read. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { - verifiedCallback := func(pubKey ic.PubKey) { - t.activeMutex.Lock() - t.active[insecure] = pubKey - t.activeMutex.Unlock() - } - cl := tls.Client(insecure, t.identity.ConfigForPeer(p, verifiedCallback)) - return t.handshake(ctx, insecure, cl) + config, keyCh := t.identity.ConfigForPeer(p) + return t.handshake(ctx, tls.Client(insecure, config), keyCh) } func (t *Transport) handshake( ctx context.Context, - insecure net.Conn, tlsConn *tls.Conn, + keyCh <-chan ci.PubKey, ) (cs.Conn, error) { // There's no way to pass a context to tls.Conn.Handshake(). // See https://github.com/golang/go/issues/18482. @@ -120,7 +99,15 @@ func (t *Transport) handshake( } return nil, err } - conn, err := t.setupConn(insecure, tlsConn) + + // Should be ready by this point, don't block. + var remotePubKey ic.PubKey + select { + case remotePubKey = <-keyCh: + default: + } + + conn, err := t.setupConn(tlsConn, remotePubKey) if err != nil { // if the context was canceled, return the context error if ctxErr := ctx.Err(); ctxErr != nil { @@ -131,11 +118,7 @@ func (t *Transport) handshake( return conn, nil } -func (t *Transport) setupConn(insecure net.Conn, tlsConn *tls.Conn) (cs.Conn, error) { - t.activeMutex.Lock() - remotePubKey := t.active[insecure] - t.activeMutex.Unlock() - +func (t *Transport) setupConn(tlsConn *tls.Conn, remotePubKey ic.PubKey) (cs.Conn, error) { if remotePubKey == nil { return nil, errors.New("go-libp2p-tls BUG: expected remote pub key to be set") } diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index c10cbdd187..a02debe8c9 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -180,15 +180,15 @@ var _ = Describe("Transport", func() { Context("invalid certificates", func() { invalidateCertChain := func(identity *Identity) { - switch identity.Config.Certificates[0].PrivateKey.(type) { + switch identity.config.Certificates[0].PrivateKey.(type) { case *rsa.PrivateKey: key, err := rsa.GenerateKey(rand.Reader, 1024) Expect(err).ToNot(HaveOccurred()) - identity.Config.Certificates[0].PrivateKey = key + identity.config.Certificates[0].PrivateKey = key case *ecdsa.PrivateKey: key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) Expect(err).ToNot(HaveOccurred()) - identity.Config.Certificates[0].PrivateKey = key + identity.config.Certificates[0].PrivateKey = key default: Fail("unexpected private key type") } @@ -206,7 +206,7 @@ var _ = Describe("Transport", func() { Expect(err).ToNot(HaveOccurred()) cert2DER, err := x509.CreateCertificate(rand.Reader, tmpl, cert1, key2.Public(), key2) Expect(err).ToNot(HaveOccurred()) - identity.Config.Certificates = []tls.Certificate{{ + identity.config.Certificates = []tls.Certificate{{ Certificate: [][]byte{cert2DER, cert1DER}, PrivateKey: key2, }} @@ -222,7 +222,7 @@ var _ = Describe("Transport", func() { Expect(err).ToNot(HaveOccurred()) cert, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, key.Public(), key) Expect(err).ToNot(HaveOccurred()) - identity.Config.Certificates = []tls.Certificate{{ + identity.config.Certificates = []tls.Certificate{{ Certificate: [][]byte{cert}, PrivateKey: key, }} From ebc4872cb93b36f0834121d45043de7373e60cda Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 28 Feb 2019 09:16:27 +0900 Subject: [PATCH 0952/3965] fix duplicate import --- p2p/security/tls/conn.go | 10 +++++----- p2p/security/tls/transport.go | 5 ++--- p2p/security/tls/transport_test.go | 12 ++++++------ 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/p2p/security/tls/conn.go b/p2p/security/tls/conn.go index f4eb600089..d5450b1c6d 100644 --- a/p2p/security/tls/conn.go +++ b/p2p/security/tls/conn.go @@ -4,7 +4,7 @@ import ( "crypto/tls" cs "github.com/libp2p/go-conn-security" - ic "github.com/libp2p/go-libp2p-crypto" + ci "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" ) @@ -12,10 +12,10 @@ type conn struct { *tls.Conn localPeer peer.ID - privKey ic.PrivKey + privKey ci.PrivKey remotePeer peer.ID - remotePubKey ic.PubKey + remotePubKey ci.PubKey } var _ cs.Conn = &conn{} @@ -24,7 +24,7 @@ func (c *conn) LocalPeer() peer.ID { return c.localPeer } -func (c *conn) LocalPrivateKey() ic.PrivKey { +func (c *conn) LocalPrivateKey() ci.PrivKey { return c.privKey } @@ -32,6 +32,6 @@ func (c *conn) RemotePeer() peer.ID { return c.remotePeer } -func (c *conn) RemotePublicKey() ic.PubKey { +func (c *conn) RemotePublicKey() ci.PubKey { return c.remotePubKey } diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 6301f8897c..8b3e132f6b 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -9,7 +9,6 @@ import ( cs "github.com/libp2p/go-conn-security" ci "github.com/libp2p/go-libp2p-crypto" - ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" ) @@ -101,7 +100,7 @@ func (t *Transport) handshake( } // Should be ready by this point, don't block. - var remotePubKey ic.PubKey + var remotePubKey ci.PubKey select { case remotePubKey = <-keyCh: default: @@ -118,7 +117,7 @@ func (t *Transport) handshake( return conn, nil } -func (t *Transport) setupConn(tlsConn *tls.Conn, remotePubKey ic.PubKey) (cs.Conn, error) { +func (t *Transport) setupConn(tlsConn *tls.Conn, remotePubKey ci.PubKey) (cs.Conn, error) { if remotePubKey == nil { return nil, errors.New("go-libp2p-tls BUG: expected remote pub key to be set") } diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index a02debe8c9..02076eed92 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -17,7 +17,7 @@ import ( "github.com/onsi/gomega/gbytes" cs "github.com/libp2p/go-conn-security" - ic "github.com/libp2p/go-libp2p-crypto" + ci "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" @@ -31,21 +31,21 @@ type transform struct { var _ = Describe("Transport", func() { var ( - serverKey, clientKey ic.PrivKey + serverKey, clientKey ci.PrivKey serverID, clientID peer.ID ) - createPeer := func() (peer.ID, ic.PrivKey) { - var priv ic.PrivKey + createPeer := func() (peer.ID, ci.PrivKey) { + var priv ci.PrivKey if mrand.Int()%2 == 0 { fmt.Fprintf(GinkgoWriter, " using an ECDSA key: ") var err error - priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) + priv, _, err = ci.GenerateECDSAKeyPair(rand.Reader) Expect(err).ToNot(HaveOccurred()) } else { fmt.Fprintf(GinkgoWriter, " using an RSA key: ") var err error - priv, _, err = ic.GenerateRSAKeyPair(1024, rand.Reader) + priv, _, err = ci.GenerateRSAKeyPair(1024, rand.Reader) Expect(err).ToNot(HaveOccurred()) } id, err := peer.IDFromPrivateKey(priv) From 7d129c2a6f2766db042f79a358aee76095fd5989 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 20 Feb 2019 17:44:39 +0800 Subject: [PATCH 0953/3965] add an example server and client --- p2p/security/tls/example/README.md | 6 ++ p2p/security/tls/example/client/main.go | 68 +++++++++++++++++++++++ p2p/security/tls/example/server/main.go | 73 +++++++++++++++++++++++++ 3 files changed, 147 insertions(+) create mode 100644 p2p/security/tls/example/README.md create mode 100644 p2p/security/tls/example/client/main.go create mode 100644 p2p/security/tls/example/server/main.go diff --git a/p2p/security/tls/example/README.md b/p2p/security/tls/example/README.md new file mode 100644 index 0000000000..dff0f66885 --- /dev/null +++ b/p2p/security/tls/example/README.md @@ -0,0 +1,6 @@ +# TLS handshake example + +Run +```bash +go run example/server/main.go +``` diff --git a/p2p/security/tls/example/client/main.go b/p2p/security/tls/example/client/main.go new file mode 100644 index 0000000000..7e8ffca5da --- /dev/null +++ b/p2p/security/tls/example/client/main.go @@ -0,0 +1,68 @@ +package main + +import ( + "context" + "crypto/rand" + "flag" + "fmt" + "io/ioutil" + "net" + "time" + + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" + libp2ptls "github.com/libp2p/go-libp2p-tls" +) + +func main() { + if err := startClient(); err != nil { + panic(err) + } +} + +func startClient() error { + port := flag.Int("p", 5533, "port") + peerIDString := flag.String("id", "", "peer ID") + flag.Parse() + + peerID, err := peer.IDB58Decode(*peerIDString) + if err != nil { + return err + } + + priv, _, err := ic.GenerateECDSAKeyPair(rand.Reader) + if err != nil { + return err + } + id, err := peer.IDFromPrivateKey(priv) + if err != nil { + return err + } + fmt.Printf("Generated new peer with an ECDSA key. Peer ID: %s\n", id.Pretty()) + tp, err := libp2ptls.New(priv) + if err != nil { + return err + } + + remoteAddr := fmt.Sprintf("localhost:%d", *port) + fmt.Printf("Dialing %s\n", remoteAddr) + conn, err := net.Dial("tcp", remoteAddr) + if err != nil { + return err + } + fmt.Printf("Dialed raw connection to %s\n", conn.RemoteAddr()) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + sconn, err := tp.SecureOutbound(ctx, conn, peerID) + if err != nil { + return err + } + fmt.Printf("Authenticated server: %s\n", sconn.RemotePeer().Pretty()) + data, err := ioutil.ReadAll(sconn) + if err != nil { + return err + } + fmt.Printf("Received message from server: %s\n", string(data)) + return nil +} diff --git a/p2p/security/tls/example/server/main.go b/p2p/security/tls/example/server/main.go new file mode 100644 index 0000000000..d8371658b3 --- /dev/null +++ b/p2p/security/tls/example/server/main.go @@ -0,0 +1,73 @@ +package main + +import ( + "context" + "crypto/rand" + "flag" + "fmt" + "net" + "time" + + ic "github.com/libp2p/go-libp2p-crypto" + peer "github.com/libp2p/go-libp2p-peer" + libp2ptls "github.com/libp2p/go-libp2p-tls" +) + +func main() { + if err := startServer(); err != nil { + panic(err) + } +} + +func startServer() error { + port := flag.Int("p", 5533, "port") + flag.Parse() + + priv, _, err := ic.GenerateECDSAKeyPair(rand.Reader) + if err != nil { + return err + } + id, err := peer.IDFromPrivateKey(priv) + if err != nil { + return err + } + fmt.Printf("Generated new peer with an ECDSA key. Peer ID: %s\n", id.Pretty()) + tp, err := libp2ptls.New(priv) + if err != nil { + return err + } + + ln, err := net.Listen("tcp", fmt.Sprintf("localhost:%d", *port)) + if err != nil { + return err + } + fmt.Printf("Listening for new connections on %s\n", ln.Addr()) + fmt.Printf("Now run the following command in a separate terminal:\n") + fmt.Printf("\tgo run example/client/main.go -p %d -id %s\n", *port, id.Pretty()) + + for { + conn, err := ln.Accept() + if err != nil { + return err + } + fmt.Printf("Accepted raw connection from %s\n", conn.RemoteAddr()) + go func() { + if err := handleConn(tp, conn); err != nil { + fmt.Printf("Error handling connection from %s: %s\n", conn.RemoteAddr(), err) + } + }() + } +} + +func handleConn(tp *libp2ptls.Transport, conn net.Conn) error { + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + sconn, err := tp.SecureInbound(ctx, conn) + if err != nil { + return err + } + fmt.Printf("Authenticated client: %s\n", sconn.RemotePeer().Pretty()) + fmt.Fprintf(sconn, "Hello client!") + fmt.Printf("Closing connection to %s\n", conn.RemoteAddr()) + return sconn.Close() +} From e4b8bb72f3dba471ad8e115ce8bdf7389685890c Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 20 Feb 2019 18:10:23 +0800 Subject: [PATCH 0954/3965] add a command line flag to set the key type --- p2p/security/tls/example/client/main.go | 25 ++++++++++++++++++++++--- p2p/security/tls/example/server/main.go | 23 +++++++++++++++++++++-- 2 files changed, 43 insertions(+), 5 deletions(-) diff --git a/p2p/security/tls/example/client/main.go b/p2p/security/tls/example/client/main.go index 7e8ffca5da..678ece6ca6 100644 --- a/p2p/security/tls/example/client/main.go +++ b/p2p/security/tls/example/client/main.go @@ -23,22 +23,41 @@ func main() { func startClient() error { port := flag.Int("p", 5533, "port") peerIDString := flag.String("id", "", "peer ID") + keyType := flag.String("key", "ecdsa", "rsa, ecdsa, ed25519 or secp256k1") flag.Parse() - peerID, err := peer.IDB58Decode(*peerIDString) + var priv ic.PrivKey + var err error + switch *keyType { + case "rsa": + fmt.Printf("Generated new peer with an RSA key.") + priv, _, err = ic.GenerateRSAKeyPair(2048, rand.Reader) + case "ecdsa": + fmt.Printf("Generated new peer with an ECDSA key.") + priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) + case "ed25519": + fmt.Printf("Generated new peer with an Ed25519 key.") + priv, _, err = ic.GenerateEd25519Key(rand.Reader) + case "secp256k1": + fmt.Printf("Generated new peer with an Secp256k1 key.") + priv, _, err = ic.GenerateSecp256k1Key(rand.Reader) + default: + return fmt.Errorf("unknown key type: %s", *keyType) + } if err != nil { return err } - priv, _, err := ic.GenerateECDSAKeyPair(rand.Reader) + peerID, err := peer.IDB58Decode(*peerIDString) if err != nil { return err } + id, err := peer.IDFromPrivateKey(priv) if err != nil { return err } - fmt.Printf("Generated new peer with an ECDSA key. Peer ID: %s\n", id.Pretty()) + fmt.Printf(" Peer ID: %s\n", id.Pretty()) tp, err := libp2ptls.New(priv) if err != nil { return err diff --git a/p2p/security/tls/example/server/main.go b/p2p/security/tls/example/server/main.go index d8371658b3..9988ec08b2 100644 --- a/p2p/security/tls/example/server/main.go +++ b/p2p/security/tls/example/server/main.go @@ -21,17 +21,36 @@ func main() { func startServer() error { port := flag.Int("p", 5533, "port") + keyType := flag.String("key", "ecdsa", "rsa, ecdsa, ed25519 or secp256k1") flag.Parse() - priv, _, err := ic.GenerateECDSAKeyPair(rand.Reader) + var priv ic.PrivKey + var err error + switch *keyType { + case "rsa": + fmt.Printf("Generated new peer with an RSA key.") + priv, _, err = ic.GenerateRSAKeyPair(2048, rand.Reader) + case "ecdsa": + fmt.Printf("Generated new peer with an ECDSA key.") + priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) + case "ed25519": + fmt.Printf("Generated new peer with an Ed25519 key.") + priv, _, err = ic.GenerateEd25519Key(rand.Reader) + case "secp256k1": + fmt.Printf("Generated new peer with an Secp256k1 key.") + priv, _, err = ic.GenerateSecp256k1Key(rand.Reader) + default: + return fmt.Errorf("unknown key type: %s", *keyType) + } if err != nil { return err } + id, err := peer.IDFromPrivateKey(priv) if err != nil { return err } - fmt.Printf("Generated new peer with an ECDSA key. Peer ID: %s\n", id.Pretty()) + fmt.Printf(" Peer ID: %s\n", id.Pretty()) tp, err := libp2ptls.New(priv) if err != nil { return err From e996c4ac2a879f5cf2887c30cb220934dc6ca508 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 21 Feb 2019 17:04:59 +0800 Subject: [PATCH 0955/3965] rename example to cmd, move to a single .go file --- p2p/security/tls/{example => cmd}/README.md | 2 +- p2p/security/tls/cmd/tlsdiag.go | 33 +++++++++++++++++++ .../client/main.go => cmd/tlsdiag/client.go} | 31 ++--------------- p2p/security/tls/cmd/tlsdiag/key.go | 28 ++++++++++++++++ .../server/main.go => cmd/tlsdiag/server.go} | 33 +++---------------- 5 files changed, 69 insertions(+), 58 deletions(-) rename p2p/security/tls/{example => cmd}/README.md (57%) create mode 100644 p2p/security/tls/cmd/tlsdiag.go rename p2p/security/tls/{example/client/main.go => cmd/tlsdiag/client.go} (61%) create mode 100644 p2p/security/tls/cmd/tlsdiag/key.go rename p2p/security/tls/{example/server/main.go => cmd/tlsdiag/server.go} (62%) diff --git a/p2p/security/tls/example/README.md b/p2p/security/tls/cmd/README.md similarity index 57% rename from p2p/security/tls/example/README.md rename to p2p/security/tls/cmd/README.md index dff0f66885..d0efa12ce6 100644 --- a/p2p/security/tls/example/README.md +++ b/p2p/security/tls/cmd/README.md @@ -2,5 +2,5 @@ Run ```bash -go run example/server/main.go +go run cmd/tlsdiag.go server ``` diff --git a/p2p/security/tls/cmd/tlsdiag.go b/p2p/security/tls/cmd/tlsdiag.go new file mode 100644 index 0000000000..73aef5fe60 --- /dev/null +++ b/p2p/security/tls/cmd/tlsdiag.go @@ -0,0 +1,33 @@ +package main + +import ( + "fmt" + "os" + + "github.com/libp2p/go-libp2p-tls/cmd/cmdimpl" +) + +func main() { + if len(os.Args) <= 1 { + fmt.Println("missing argument: client / server") + return + } + + role := os.Args[1] + // remove the role argument from os.Args + os.Args = append([]string{os.Args[0]}, os.Args[2:]...) + + var err error + switch role { + case "client": + err = cmdimpl.StartClient() + case "server": + err = cmdimpl.StartServer() + default: + fmt.Println("invalid argument. Expected client / server") + return + } + if err != nil { + panic(err) + } +} diff --git a/p2p/security/tls/example/client/main.go b/p2p/security/tls/cmd/tlsdiag/client.go similarity index 61% rename from p2p/security/tls/example/client/main.go rename to p2p/security/tls/cmd/tlsdiag/client.go index 678ece6ca6..9de3347846 100644 --- a/p2p/security/tls/example/client/main.go +++ b/p2p/security/tls/cmd/tlsdiag/client.go @@ -1,49 +1,24 @@ -package main +package cmdimpl import ( "context" - "crypto/rand" "flag" "fmt" "io/ioutil" "net" "time" - ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" libp2ptls "github.com/libp2p/go-libp2p-tls" ) -func main() { - if err := startClient(); err != nil { - panic(err) - } -} - -func startClient() error { +func StartClient() error { port := flag.Int("p", 5533, "port") peerIDString := flag.String("id", "", "peer ID") keyType := flag.String("key", "ecdsa", "rsa, ecdsa, ed25519 or secp256k1") flag.Parse() - var priv ic.PrivKey - var err error - switch *keyType { - case "rsa": - fmt.Printf("Generated new peer with an RSA key.") - priv, _, err = ic.GenerateRSAKeyPair(2048, rand.Reader) - case "ecdsa": - fmt.Printf("Generated new peer with an ECDSA key.") - priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) - case "ed25519": - fmt.Printf("Generated new peer with an Ed25519 key.") - priv, _, err = ic.GenerateEd25519Key(rand.Reader) - case "secp256k1": - fmt.Printf("Generated new peer with an Secp256k1 key.") - priv, _, err = ic.GenerateSecp256k1Key(rand.Reader) - default: - return fmt.Errorf("unknown key type: %s", *keyType) - } + priv, err := generateKey(*keyType) if err != nil { return err } diff --git a/p2p/security/tls/cmd/tlsdiag/key.go b/p2p/security/tls/cmd/tlsdiag/key.go new file mode 100644 index 0000000000..fc2ad8a788 --- /dev/null +++ b/p2p/security/tls/cmd/tlsdiag/key.go @@ -0,0 +1,28 @@ +package cmdimpl + +import ( + "crypto/rand" + "fmt" + + ic "github.com/libp2p/go-libp2p-crypto" +) + +func generateKey(keyType string) (priv ic.PrivKey, err error) { + switch keyType { + case "rsa": + fmt.Printf("Generated new peer with an RSA key.") + priv, _, err = ic.GenerateRSAKeyPair(2048, rand.Reader) + case "ecdsa": + fmt.Printf("Generated new peer with an ECDSA key.") + priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) + case "ed25519": + fmt.Printf("Generated new peer with an Ed25519 key.") + priv, _, err = ic.GenerateEd25519Key(rand.Reader) + case "secp256k1": + fmt.Printf("Generated new peer with an Secp256k1 key.") + priv, _, err = ic.GenerateSecp256k1Key(rand.Reader) + default: + return nil, fmt.Errorf("unknown key type: %s", keyType) + } + return +} diff --git a/p2p/security/tls/example/server/main.go b/p2p/security/tls/cmd/tlsdiag/server.go similarity index 62% rename from p2p/security/tls/example/server/main.go rename to p2p/security/tls/cmd/tlsdiag/server.go index 9988ec08b2..b192c44bba 100644 --- a/p2p/security/tls/example/server/main.go +++ b/p2p/security/tls/cmd/tlsdiag/server.go @@ -1,47 +1,22 @@ -package main +package cmdimpl import ( "context" - "crypto/rand" "flag" "fmt" "net" "time" - ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" libp2ptls "github.com/libp2p/go-libp2p-tls" ) -func main() { - if err := startServer(); err != nil { - panic(err) - } -} - -func startServer() error { +func StartServer() error { port := flag.Int("p", 5533, "port") keyType := flag.String("key", "ecdsa", "rsa, ecdsa, ed25519 or secp256k1") flag.Parse() - var priv ic.PrivKey - var err error - switch *keyType { - case "rsa": - fmt.Printf("Generated new peer with an RSA key.") - priv, _, err = ic.GenerateRSAKeyPair(2048, rand.Reader) - case "ecdsa": - fmt.Printf("Generated new peer with an ECDSA key.") - priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) - case "ed25519": - fmt.Printf("Generated new peer with an Ed25519 key.") - priv, _, err = ic.GenerateEd25519Key(rand.Reader) - case "secp256k1": - fmt.Printf("Generated new peer with an Secp256k1 key.") - priv, _, err = ic.GenerateSecp256k1Key(rand.Reader) - default: - return fmt.Errorf("unknown key type: %s", *keyType) - } + priv, err := generateKey(*keyType) if err != nil { return err } @@ -62,7 +37,7 @@ func startServer() error { } fmt.Printf("Listening for new connections on %s\n", ln.Addr()) fmt.Printf("Now run the following command in a separate terminal:\n") - fmt.Printf("\tgo run example/client/main.go -p %d -id %s\n", *port, id.Pretty()) + fmt.Printf("\tgo run cmd/tlsdiag.go client -p %d -id %s\n", *port, id.Pretty()) for { conn, err := ln.Accept() From 20005517f5cfde5875bbbba888ec11bf667bc3a5 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 28 Feb 2019 09:44:58 +0900 Subject: [PATCH 0956/3965] fix package name of tlsdiag --- p2p/security/tls/cmd/tlsdiag.go | 6 +++--- p2p/security/tls/cmd/tlsdiag/client.go | 2 +- p2p/security/tls/cmd/tlsdiag/key.go | 2 +- p2p/security/tls/cmd/tlsdiag/server.go | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/security/tls/cmd/tlsdiag.go b/p2p/security/tls/cmd/tlsdiag.go index 73aef5fe60..4aa2f2f09c 100644 --- a/p2p/security/tls/cmd/tlsdiag.go +++ b/p2p/security/tls/cmd/tlsdiag.go @@ -4,7 +4,7 @@ import ( "fmt" "os" - "github.com/libp2p/go-libp2p-tls/cmd/cmdimpl" + "github.com/libp2p/go-libp2p-tls/cmd/tlsdiag" ) func main() { @@ -20,9 +20,9 @@ func main() { var err error switch role { case "client": - err = cmdimpl.StartClient() + err = tlsdiag.StartClient() case "server": - err = cmdimpl.StartServer() + err = tlsdiag.StartServer() default: fmt.Println("invalid argument. Expected client / server") return diff --git a/p2p/security/tls/cmd/tlsdiag/client.go b/p2p/security/tls/cmd/tlsdiag/client.go index 9de3347846..21e4d18098 100644 --- a/p2p/security/tls/cmd/tlsdiag/client.go +++ b/p2p/security/tls/cmd/tlsdiag/client.go @@ -1,4 +1,4 @@ -package cmdimpl +package tlsdiag import ( "context" diff --git a/p2p/security/tls/cmd/tlsdiag/key.go b/p2p/security/tls/cmd/tlsdiag/key.go index fc2ad8a788..038974225a 100644 --- a/p2p/security/tls/cmd/tlsdiag/key.go +++ b/p2p/security/tls/cmd/tlsdiag/key.go @@ -1,4 +1,4 @@ -package cmdimpl +package tlsdiag import ( "crypto/rand" diff --git a/p2p/security/tls/cmd/tlsdiag/server.go b/p2p/security/tls/cmd/tlsdiag/server.go index b192c44bba..0a895baf12 100644 --- a/p2p/security/tls/cmd/tlsdiag/server.go +++ b/p2p/security/tls/cmd/tlsdiag/server.go @@ -1,4 +1,4 @@ -package cmdimpl +package tlsdiag import ( "context" From f22b6c7f7339d439e8ba222b7a4af77830a0c595 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Thu, 28 Feb 2019 11:17:08 +1100 Subject: [PATCH 0957/3965] Improve dial error messages --- p2p/net/swarm/swarm_dial.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 56d8420e73..1bed042267 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -290,6 +290,9 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { that we previously had (halting a dial when we run out of addrs) */ goodAddrs := s.filterKnownUndialables(s.peers.Addrs(p)) + if len(goodAddrs) == 0 { + return nil, errors.New("no good addresses") + } goodAddrsChan := make(chan ma.Multiaddr, len(goodAddrs)) for _, a := range goodAddrs { goodAddrsChan <- a @@ -352,7 +355,7 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. // use a single response type instead of errs and conns, reduces complexity *a ton* respch := make(chan dialResult) - defaultDialFail := fmt.Errorf("failed to dial %s (default failure)", p) + defaultDialFail := inet.ErrNoRemoteAddrs exitErr := defaultDialFail defer s.limiter.clearAllPeerDials(p) From 2dd6b7c8aec53d4708f979638d6eb0b819407736 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 28 Feb 2019 18:33:47 +0000 Subject: [PATCH 0958/3965] migrate to multiformats/go-base32. --- p2p/host/peerstore/pstoreds/addr_book.go | 2 +- p2p/host/peerstore/pstoreds/addr_book_gc.go | 2 +- p2p/host/peerstore/pstoreds/keybook.go | 2 +- p2p/host/peerstore/pstoreds/metadata.go | 2 +- p2p/host/peerstore/pstoreds/peerstore.go | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 9690bd5a4d..faef34b9e3 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -17,8 +17,8 @@ import ( pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" lru "github.com/hashicorp/golang-lru" + b32 "github.com/multiformats/go-base32" ma "github.com/multiformats/go-multiaddr" - b32 "github.com/whyrusleeping/base32" ) type ttlWriteMode int diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index f40d5533d7..8c66025439 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -12,7 +12,7 @@ import ( peer "github.com/libp2p/go-libp2p-peer" pb "github.com/libp2p/go-libp2p-peerstore/pb" - b32 "github.com/whyrusleeping/base32" + b32 "github.com/multiformats/go-base32" ) var ( diff --git a/p2p/host/peerstore/pstoreds/keybook.go b/p2p/host/peerstore/pstoreds/keybook.go index 8dcc82691d..29885c5da2 100644 --- a/p2p/host/peerstore/pstoreds/keybook.go +++ b/p2p/host/peerstore/pstoreds/keybook.go @@ -4,7 +4,7 @@ import ( "context" "errors" - base32 "github.com/whyrusleeping/base32" + base32 "github.com/multiformats/go-base32" ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" diff --git a/p2p/host/peerstore/pstoreds/metadata.go b/p2p/host/peerstore/pstoreds/metadata.go index 7c1d124f73..dca550cbe3 100644 --- a/p2p/host/peerstore/pstoreds/metadata.go +++ b/p2p/host/peerstore/pstoreds/metadata.go @@ -5,7 +5,7 @@ import ( "context" "encoding/gob" - base32 "github.com/whyrusleeping/base32" + base32 "github.com/multiformats/go-base32" ds "github.com/ipfs/go-datastore" diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index fc0263d48b..be7b09e9a8 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -4,7 +4,7 @@ import ( "context" "time" - base32 "github.com/whyrusleeping/base32" + base32 "github.com/multiformats/go-base32" ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" From 06c184b582abc99b71b8c53ea783e8778759c7f5 Mon Sep 17 00:00:00 2001 From: Ryan Skidmore Date: Fri, 1 Mar 2019 15:09:57 +0000 Subject: [PATCH 0959/3965] Added method to generically discover IG Devices --- nat.go | 2 ++ upnp.go | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/nat.go b/nat.go index 3cf9648585..eb574d63c2 100644 --- a/nat.go +++ b/nat.go @@ -41,6 +41,8 @@ func DiscoverGateway() (NAT, error) { return nat, nil case nat := <-discoverUPNP_IG2(): return nat, nil + case nat := <-discoverUPNP_GenIGDev(): + return nat, nil case nat := <-discoverNATPMP(): return nat, nil case <-time.After(10 * time.Second): diff --git a/upnp.go b/upnp.go index 86d6e9b3ba..caad350114 100644 --- a/upnp.go +++ b/upnp.go @@ -2,11 +2,15 @@ package nat import ( "net" + "net/url" + "strings" "time" "github.com/huin/goupnp" "github.com/huin/goupnp/dcps/internetgateway1" "github.com/huin/goupnp/dcps/internetgateway2" + + "github.com/koron/go-ssdp" ) var ( @@ -123,6 +127,62 @@ func discoverUPNP_IG2() <-chan NAT { return res } +func discoverUPNP_GenIGDev() <-chan NAT { + res := make(chan NAT, 1) + go func() { + DeviceList, err := ssdp.Search(ssdp.All, 5, "") + if err != nil { + return + } + var gw ssdp.Service + for _, Service := range DeviceList { + if strings.Contains(Service.Type, "InternetGatewayDevice") { + gw = Service + break + } + } + + DeviceURL, err := url.Parse(gw.Location) + if err != nil { + return + } + RootDevice, err := goupnp.DeviceByURL(DeviceURL) + if err != nil { + return + } + + RootDevice.Device.VisitServices(func(srv *goupnp.Service) { + switch srv.ServiceType { + case internetgateway1.URN_WANIPConnection_1: + client := &internetgateway1.WANIPConnection1{ServiceClient: goupnp.ServiceClient{ + SOAPClient: srv.NewSOAPClient(), + RootDevice: RootDevice, + Service: srv, + }} + _, isNat, err := client.GetNATRSIPStatus() + if err == nil && isNat { + res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-IP1)", RootDevice} + return + } + + case internetgateway1.URN_WANPPPConnection_1: + client := &internetgateway1.WANPPPConnection1{ServiceClient: goupnp.ServiceClient{ + SOAPClient: srv.NewSOAPClient(), + RootDevice: RootDevice, + Service: srv, + }} + _, isNat, err := client.GetNATRSIPStatus() + if err == nil && isNat { + res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-PPP1)", RootDevice} + return + } + + } + }) + }() + return res +} + type upnp_NAT_Client interface { GetExternalIPAddress() (string, error) AddPortMapping(string, uint16, string, uint16, string, bool, string, uint32) error From 900ec35643486e5263aff91711dd2bf949945d41 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 1 Mar 2019 16:43:50 +0000 Subject: [PATCH 0960/3965] add gomod support // tag v0.0.1. --- .travis.yml | 32 ++++++++++++++++++++++++++++++++ go.mod | 4 +--- go.sum | 11 +++++++++++ 3 files changed, 44 insertions(+), 3 deletions(-) create mode 100644 .travis.yml create mode 100644 go.sum diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000000..4cfe98c242 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,32 @@ +os: + - linux + +language: go + +go: + - 1.11.x + +env: + global: + - GOTFLAGS="-race" + matrix: + - BUILD_DEPTYPE=gx + - BUILD_DEPTYPE=gomod + + +# disable travis install +install: + - true + +script: + - bash <(curl -s https://raw.githubusercontent.com/ipfs/ci-helpers/master/travis-ci/run-standard-tests.sh) + + +cache: + directories: + - $GOPATH/src/gx + - $GOPATH/pkg/mod + - $HOME/.cache/go-build + +notifications: + email: false diff --git a/go.mod b/go.mod index eb7991248d..bc8f7bcac9 100644 --- a/go.mod +++ b/go.mod @@ -1,9 +1,7 @@ module github.com/libp2p/go-nat require ( - github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324 + github.com/huin/goupnp v1.0.0 github.com/jackpal/gateway v1.0.5 github.com/jackpal/go-nat-pmp v1.0.1 - golang.org/x/net v0.0.0-20180524181706-dfa909b99c79 - golang.org/x/text v0.3.0 ) diff --git a/go.sum b/go.sum new file mode 100644 index 0000000000..cf1f0624a3 --- /dev/null +++ b/go.sum @@ -0,0 +1,11 @@ +github.com/huin/goupnp v1.0.0 h1:wg75sLpL6DZqwHQN6E1Cfk6mtfzS45z8OV+ic+DtHRo= +github.com/huin/goupnp v1.0.0/go.mod h1:n9v9KO1tAxYH82qOn+UTIFQDmx5n1Zxd/ClZDMX7Bnc= +github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150/go.mod h1:PpLOETDnJ0o3iZrZfqZzyLl6l7F3c6L1oWn7OICBi6o= +github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc= +github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= +github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= +github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= +golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1 h1:Y/KGZSOdz/2r0WJ9Mkmz6NJBusp0kiNx1Cn82lzJQ6w= +golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= From c4b498537c1845292f8fbba6fddec3587eb15698 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sat, 2 Mar 2019 00:13:40 +0000 Subject: [PATCH 0961/3965] add gomod support // tag v0.0.1. --- .travis.yml | 23 +++-- go.mod | 47 +++++++++++ go.sum | 238 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 301 insertions(+), 7 deletions(-) create mode 100644 go.mod create mode 100644 go.sum diff --git a/.travis.yml b/.travis.yml index 6f8b950982..4cfe98c242 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,23 +1,32 @@ os: - linux - - osx - -sudo: false language: go go: - - 1.11.1 + - 1.11.x + +env: + global: + - GOTFLAGS="-race" + matrix: + - BUILD_DEPTYPE=gx + - BUILD_DEPTYPE=gomod + +# disable travis install install: - - make deps + - true script: - bash <(curl -s https://raw.githubusercontent.com/ipfs/ci-helpers/master/travis-ci/run-standard-tests.sh) + cache: - directories: - - $GOPATH/src/gx + directories: + - $GOPATH/src/gx + - $GOPATH/pkg/mod + - $HOME/.cache/go-build notifications: email: false diff --git a/go.mod b/go.mod new file mode 100644 index 0000000000..ac95f4404c --- /dev/null +++ b/go.mod @@ -0,0 +1,47 @@ +module github.com/libp2p/go-libp2p + +require ( + github.com/gogo/protobuf v1.2.1 + github.com/ipfs/go-cid v0.0.1 + github.com/ipfs/go-detect-race v0.0.1 + github.com/ipfs/go-ipfs-util v0.0.1 + github.com/ipfs/go-log v0.0.1 + github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 + github.com/libp2p/go-conn-security v0.0.1 + github.com/libp2p/go-conn-security-multistream v0.0.1 + github.com/libp2p/go-libp2p-autonat v0.0.1 + github.com/libp2p/go-libp2p-blankhost v0.0.1 + github.com/libp2p/go-libp2p-circuit v0.0.1 + github.com/libp2p/go-libp2p-crypto v0.0.1 + github.com/libp2p/go-libp2p-discovery v0.0.1 + github.com/libp2p/go-libp2p-host v0.0.1 + github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 + github.com/libp2p/go-libp2p-interface-pnet v0.0.1 + github.com/libp2p/go-libp2p-loggables v0.0.1 + github.com/libp2p/go-libp2p-metrics v0.0.1 + github.com/libp2p/go-libp2p-nat v0.0.1 + github.com/libp2p/go-libp2p-net v0.0.1 + github.com/libp2p/go-libp2p-netutil v0.0.1 + github.com/libp2p/go-libp2p-peer v0.0.1 + github.com/libp2p/go-libp2p-peerstore v0.0.1 + github.com/libp2p/go-libp2p-protocol v0.0.1 + github.com/libp2p/go-libp2p-routing v0.0.1 + github.com/libp2p/go-libp2p-secio v0.0.1 + github.com/libp2p/go-libp2p-swarm v0.0.1 + github.com/libp2p/go-libp2p-transport v0.0.4 + github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 + github.com/libp2p/go-maddr-filter v0.0.1 + github.com/libp2p/go-stream-muxer v0.0.1 + github.com/libp2p/go-tcp-transport v0.0.1 + github.com/libp2p/go-testutil v0.0.1 + github.com/libp2p/go-ws-transport v0.0.1 + github.com/miekg/dns v1.1.4 // indirect + github.com/multiformats/go-multiaddr v0.0.1 + github.com/multiformats/go-multiaddr-dns v0.0.2 + github.com/multiformats/go-multiaddr-net v0.0.1 + github.com/multiformats/go-multistream v0.0.1 + github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible + github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible + github.com/whyrusleeping/go-smux-yamux v2.0.8+incompatible + github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 +) diff --git a/go.sum b/go.sum new file mode 100644 index 0000000000..137e6081cf --- /dev/null +++ b/go.sum @@ -0,0 +1,238 @@ +github.com/AndreasBriese/bbloom v0.0.0-20180913140656-343706a395b7/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= +github.com/Kubuxu/go-os-helper v0.0.1/go.mod h1:N8B+I7vPCT80IcP58r50u4+gEEcsZETFUpAzWW2ep1Y= +github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= +github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32 h1:qkOC5Gd33k54tobS36cXdAzJbeHaduLtnLQQwNoIi78= +github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32/go.mod h1:DrZx5ec/dmnfpw9KyYoQyYo7d0KEvTkk/5M/vbZjAr8= +github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA= +github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= +github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd/go.mod h1:HHNXQzUsZCxOoE+CPiyCTO6x34Zs86zZUiwtpXoGdtg= +github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVaaLLH7j4eDXPRvw78tMflu7Ie2bzYOH4Y8rRKBY= +github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc= +github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY= +github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= +github.com/coreos/go-semver v0.2.0 h1:3Jm3tLmsgAYcjC+4Up7hJrFBPr+n7rAqYeSw/SZazuY= +github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dgraph-io/badger v1.5.5-0.20190226225317-8115aed38f8f/go.mod h1:VZxzAIRPHRVNRKRo6AXrX9BJegn6il06VMTZVJYCIjQ= +github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= +github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/fd/go-nat v1.0.0 h1:DPyQ97sxA9ThrWYRPcWUz/z9TnpTIGRYODIQc/dy64M= +github.com/fd/go-nat v1.0.0/go.mod h1:BTBu/CKvMmOMUPkKVef1pngt2WFH/lg7E6yQnulfp6E= +github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I= +github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= +github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98= +github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE= +github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.0 h1:kbxbvI4Un1LUWKxufD+BiE6AEExYYgkQLQmLFqA1LFk= +github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0= +github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= +github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q= +github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= +github.com/gxed/hashland/keccakpg v0.0.1 h1:wrk3uMNaMxbXiHibbPO4S0ymqJMm41WiudyFSs7UnsU= +github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= +github.com/gxed/hashland/murmur3 v0.0.1 h1:SheiaIt0sda5K+8FLz952/1iWS9zrnKsEJaOJu4ZbSc= +github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= +github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= +github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= +github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324 h1:PV190X5/DzQ/tbFFG5YpT5mH6q+cHlfgqI5JuRnH9oE= +github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324/go.mod h1:MZ2ZmwcBpvOoJ22IJsc7va19ZwoheaBk43rKg12SKag= +github.com/ipfs/go-cid v0.0.1 h1:GBjWPktLnNyX0JiQCNFpUuUSoMw5KMyqrsejHYlILBE= +github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= +github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= +github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= +github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= +github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8= +github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc= +github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= +github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50= +github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc= +github.com/ipfs/go-log v0.0.1 h1:9XTUN/rW64BCG1YhPK9Hoy3q8nr4gOmHHBpgFdfw6Lc= +github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= +github.com/jackpal/gateway v1.0.4 h1:LS5EHkLuQ6jzaHwULi0vL+JO0mU/n4yUtK8oUjHHOlM= +github.com/jackpal/gateway v1.0.4/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= +github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= +github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= +github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec h1:DQqZhhDvrTrEQ3Qod5yfavcA064e53xlQ+xajiorXgM= +github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec/go.mod h1:rGaEvXB4uRSZMmzKNLoXvTu1sfx+1kv/DojUlPrSZGs= +github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2 h1:vhC1OXXiT9R2pczegwz6moDvuRpggaroAXhPIseh57A= +github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2/go.mod h1:8GXXJV31xl8whumTzdZsTt3RnUIiPqzkyf7mxToRCMs= +github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 h1:bspPhN+oKYFk5fcGNuQzp6IGzYQSenLEgH3s6jkXrWw= +github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsjFq/qrU3Rar62tu1gASgGw6chQbSh/XgIIXCY= +github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= +github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= +github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4= +github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/libp2p/go-addr-util v0.0.1 h1:TpTQm9cXVRVSKsYbgQ7GKc3KbbHVTnbostgGaDEP+88= +github.com/libp2p/go-addr-util v0.0.1/go.mod h1:4ac6O7n9rIAKB1dnd+s8IbbMXkt+oBpzX4/+RACcnlQ= +github.com/libp2p/go-buffer-pool v0.0.1 h1:9Rrn/H46cXjaA2HQ5Y8lyhOS1NhTkZ4yuEs2r3Eechg= +github.com/libp2p/go-buffer-pool v0.0.1/go.mod h1:xtyIz9PMobb13WaxR6Zo1Pd1zXJKYg0a8KiIvDp3TzQ= +github.com/libp2p/go-conn-security v0.0.1 h1:4kMMrqrt9EUNCNjX1xagSJC+bq16uqjMe9lk1KBMVNs= +github.com/libp2p/go-conn-security v0.0.1/go.mod h1:bGmu51N0KU9IEjX7kl2PQjgZa40JQWnayTvNMgD/vyk= +github.com/libp2p/go-conn-security-multistream v0.0.1 h1:XefjAQRHcnUaxKb26RGupToucx3uU4ecbOZ3aACXlDU= +github.com/libp2p/go-conn-security-multistream v0.0.1/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= +github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= +github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= +github.com/libp2p/go-libp2p-autonat v0.0.1 h1:d5eskFxeJ4ag1ekhMC3yLTK+z+6RTw9W1Yv8HQma75k= +github.com/libp2p/go-libp2p-autonat v0.0.1/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= +github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= +github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= +github.com/libp2p/go-libp2p-circuit v0.0.1 h1:DYbjyQ5ZY3QVAVYZWG4uzBQ6Wmcd1C82Bk8Q/pJlM1I= +github.com/libp2p/go-libp2p-circuit v0.0.1/go.mod h1:Dqm0s/BiV63j8EEAs8hr1H5HudqvCAeXxDyic59lCwE= +github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= +github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= +github.com/libp2p/go-libp2p-discovery v0.0.1 h1:VkjCKmJQMwpDUwtA8Qc1z3TQAHJgQ5nGQ6cdN0wQXOw= +github.com/libp2p/go-libp2p-discovery v0.0.1/go.mod h1:ZkkF9xIFRLA1xCc7bstYFkd80gBGK8Fc1JqGoU2i+zI= +github.com/libp2p/go-libp2p-host v0.0.1 h1:dnqusU+DheGcdxrE718kG4XgHNuL2n9eEv8Rg5zy8hQ= +github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= +github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 h1:Q9EkNSLAOF+u90L88qmE9z/fTdjLh8OsJwGw74mkwk4= +github.com/libp2p/go-libp2p-interface-connmgr v0.0.1/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= +github.com/libp2p/go-libp2p-interface-pnet v0.0.1 h1:7GnzRrBTJHEsofi1ahFdPN9Si6skwXQE9UqR2S+Pkh8= +github.com/libp2p/go-libp2p-interface-pnet v0.0.1/go.mod h1:el9jHpQAXK5dnTpKA4yfCNBZXvrzdOU75zz+C6ryp3k= +github.com/libp2p/go-libp2p-loggables v0.0.1 h1:HVww9oAnINIxbt69LJNkxD8lnbfgteXR97Xm4p3l9ps= +github.com/libp2p/go-libp2p-loggables v0.0.1/go.mod h1:lDipDlBNYbpyqyPX/KcoO+eq0sJYEVR2JgOexcivchg= +github.com/libp2p/go-libp2p-metrics v0.0.1 h1:yumdPC/P2VzINdmcKZd0pciSUCpou+s0lwYCjBbzQZU= +github.com/libp2p/go-libp2p-metrics v0.0.1/go.mod h1:jQJ95SXXA/K1VZi13h52WZMa9ja78zjyy5rspMsC/08= +github.com/libp2p/go-libp2p-nat v0.0.1 h1:on/zju7XE+JXc8gH+vTKmIh2UJFC1K8kGnJYluQrlz4= +github.com/libp2p/go-libp2p-nat v0.0.1/go.mod h1:4L6ajyUIlJvx1Cbh5pc6Ma6vMDpKXf3GgLO5u7W0oQ4= +github.com/libp2p/go-libp2p-net v0.0.1 h1:xJ4Vh4yKF/XKb8fd1Ev0ebAGzVjMxXzrxG2kjtU+F5Q= +github.com/libp2p/go-libp2p-net v0.0.1/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= +github.com/libp2p/go-libp2p-netutil v0.0.1 h1:LgD6+skofkOx8z6odD9+MZHKjupv3ng1u6KRhaADTnA= +github.com/libp2p/go-libp2p-netutil v0.0.1/go.mod h1:GdusFvujWZI9Vt0X5BKqwWWmZFxecf9Gt03cKxm2f/Q= +github.com/libp2p/go-libp2p-peer v0.0.1 h1:0qwAOljzYewINrU+Kndoc+1jAL7vzY/oY2Go4DCGfyY= +github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= +github.com/libp2p/go-libp2p-peerstore v0.0.1 h1:twKovq8YK5trLrd3nB7PD2Zu9JcyAIdm7Bz9yBWjhq8= +github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= +github.com/libp2p/go-libp2p-protocol v0.0.1 h1:+zkEmZ2yFDi5adpVE3t9dqh/N9TbpFWywowzeEzBbLM= +github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= +github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbfdvpNntjhc= +github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= +github.com/libp2p/go-libp2p-secio v0.0.1 h1:CqE/RdsizOwItdgLe632iyft/w0tshDLmZGAiKDcUAI= +github.com/libp2p/go-libp2p-secio v0.0.1/go.mod h1:IdG6iQybdcYmbTzxp4J5dwtUEDTOvZrT0opIDVNPrJs= +github.com/libp2p/go-libp2p-swarm v0.0.1 h1:Vne+hjaDwXqzgNwQ2vb2YKbnbOTyXjtS47stT66Apc4= +github.com/libp2p/go-libp2p-swarm v0.0.1/go.mod h1:mh+KZxkbd3lQnveQ3j2q60BM1Cw2mX36XXQqwfPOShs= +github.com/libp2p/go-libp2p-transport v0.0.1/go.mod h1:UzbUs9X+PHOSw7S3ZmeOxfnwaQY5vGDzZmKPod3N3tk= +github.com/libp2p/go-libp2p-transport v0.0.4 h1:/CPHQMN75/IQwkhBxxIo6p6PtL3rwFZtlzBROT3e8mw= +github.com/libp2p/go-libp2p-transport v0.0.4/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= +github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 h1:rNtXkY6dty46mxYOHHAZQchI7gQdJStF683FhVnei/k= +github.com/libp2p/go-libp2p-transport-upgrader v0.0.1/go.mod h1:NJpUAgQab/8K6K0m+JmZCe5RUXG10UMEx4kWe9Ipj5c= +github.com/libp2p/go-maddr-filter v0.0.1 h1:apvYTg0aIxxQyBX+XHKOR+0+lYhGs1Yv+JmTH9nyl5I= +github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= +github.com/libp2p/go-mplex v0.0.1 h1:dn2XGSrUxLtz3/8u85bGrwhUEKPX8MOF3lpmcWBZCWc= +github.com/libp2p/go-mplex v0.0.1/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= +github.com/libp2p/go-msgio v0.0.1 h1:znj97n5FtXGCLDwe9x8jpHmY770SW4WStBGcCDh6GJw= +github.com/libp2p/go-msgio v0.0.1/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= +github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= +github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= +github.com/libp2p/go-reuseport-transport v0.0.1 h1:UIRneNxLDmEGNjGHpIiWzSWkZ5bhxMCP9x3Vh7BSc7E= +github.com/libp2p/go-reuseport-transport v0.0.1/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= +github.com/libp2p/go-stream-muxer v0.0.1 h1:Ce6e2Pyu+b5MC1k3eeFtAax0pW4gc6MosYSLV05UeLw= +github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= +github.com/libp2p/go-tcp-transport v0.0.1 h1:WyvJVw2lYAnr6CU+GZZ4oCt06fvORlmvBlFX2+ZpZDM= +github.com/libp2p/go-tcp-transport v0.0.1/go.mod h1:mnjg0o0O5TmXUaUIanYPUqkW4+u6mK0en8rlpA6BBTs= +github.com/libp2p/go-testutil v0.0.1 h1:Xg+O0G2HIMfHqBOBDcMS1iSZJ3GEcId4qOxCQvsGZHk= +github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= +github.com/libp2p/go-ws-transport v0.0.1 h1:9ytMqq86Xvp8rcnC/1ZNuH612eXLDglvcu4ZHseJl8s= +github.com/libp2p/go-ws-transport v0.0.1/go.mod h1:p3bKjDWHEgtuKKj+2OdPYs5dAPIjtpQGHF2tJfGz7Ww= +github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= +github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= +github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= +github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= +github.com/miekg/dns v1.1.4 h1:rCMZsU2ScVSYcAsOXgmC6+AKOK+6pmQTOcw03nfwYV0= +github.com/miekg/dns v1.1.4/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= +github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g= +github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ= +github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16 h1:5W7KhL8HVF3XCFOweFD3BNESdnO8ewyYTFT2R+/b8FQ= +github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= +github.com/mr-tron/base58 v1.1.0 h1:Y51FGVJ91WBqCEabAi5OPUz38eAx8DakuAm5svLcsfQ= +github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= +github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI= +github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA= +github.com/multiformats/go-multiaddr v0.0.1 h1:/QUV3VBMDI6pi6xfiw7lr6xhDWWvQKn9udPn68kLSdY= +github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= +github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= +github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= +github.com/multiformats/go-multiaddr-net v0.0.1 h1:76O59E3FavvHqNg7jvzWzsPSW5JSi/ek0E4eiDVbg9g= +github.com/multiformats/go-multiaddr-net v0.0.1/go.mod h1:nw6HSxNmCIQH27XPGBuX+d1tnvM7ihcFwHMSstNAVUU= +github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= +github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= +github.com/multiformats/go-multihash v0.0.1 h1:HHwN1K12I+XllBCrqKnhX949Orn4oawPkegHMu2vDqQ= +github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= +github.com/multiformats/go-multistream v0.0.1 h1:JV4VfSdY9n7ECTtY59/TlSyFCzRILvYx4T4Ws8ZgihU= +github.com/multiformats/go-multistream v0.0.1/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= +github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.7.0 h1:WSHQ+IS43OoUrWtD1/bbclrwK8TTH5hzp+umCiuxHgs= +github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/gomega v1.4.3 h1:RE1xgDvH7imwFD45h+u2SgIfERHlS2yNG4DObb5BSKU= +github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/opentracing/opentracing-go v1.0.2 h1:3jA2P6O1F9UOrWVpwrIo17pu01KWvNWg4X946/Y5Zwg= +github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= +github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= +github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= +github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1/go.mod h1:8UvriyWtv5Q5EOgjHaSseUEdkQfvwFv1I/In/O2M9gc= +github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6Rz4bwmou+oE6Dt4Cb2BGMur5eR/GYptkKUVHo= +github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM= +github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f h1:M/lL30eFZTKnomXY6huvM6G0+gVquFNf6mxghaWlFUg= +github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f/go.mod h1:cZNvX9cFybI01GriPRMXDtczuvUhgbcYr9iCGaNlRv8= +github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible h1:iqksILj8STw03EJQe7Laj4ubnw+ojOyik18cd5vPL1o= +github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible/go.mod h1:34LEDbeKFZInPUrAG+bjuJmUXONGdEFW7XL0SpTY1y4= +github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible h1:BdYHctE9HJZLquG9tpTdwWcbG4FaX6tVKPGjCGgiVxo= +github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible/go.mod h1:dRWHHvc4HDQSHh9gbKEBbUZ+f2Q8iZTPG3UOGYODxSQ= +github.com/whyrusleeping/go-smux-yamux v2.0.8+incompatible h1:IGm/UP/JpEFS6D787sZnZg7RA6fZIR9c/Ms9DeAVNuk= +github.com/whyrusleeping/go-smux-yamux v2.0.8+incompatible/go.mod h1:6qHUzBXUbB9MXmw3AUdB52L8sEb/hScCqOdW2kj/wuI= +github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= +github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= +github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxfC1Y6mYncdk+kq8d5aLx0Q+/gyZGE44M= +github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= +github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= +github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= +github.com/whyrusleeping/yamux v1.1.5 h1:4CK3aUUJQu0qpKZv5gEWJjNOQtdbdDhVVS6PJ+HimdE= +github.com/whyrusleeping/yamux v1.1.5/go.mod h1:E8LnQQ8HKx5KD29HZFUwM1PxCOdPRzGwur1mcYhXcD8= +golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25 h1:jsG6UpNLt9iAsb0S2AGW28DveNzzgmbXR+ENoPjUeIU= +golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/net v0.0.0-20180524181706-dfa909b99c79/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190227160552-c95aed5357e7 h1:C2F/nMkR/9sfUTpvR3QrjBuTdvMUC/cFajkphs1YLQo= +golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f h1:wMNYb4v58l5UBM7MYRLPG6ZhfOqbKu7X5eyFl8ZhKvA= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e h1:ZytStCyV048ZqDsWHiYDdoI2Vd4msMcrDECFxS+tL9c= +golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= +gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= +gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE= +gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= From a615c186643cdd9bc9566deb325eaa57e32de3de Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 2 Mar 2019 10:18:21 +0200 Subject: [PATCH 0962/3965] mutex protect Status to pacify the race detector --- p2p/host/autonat/autonat.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index e0418c45f8..410d733edd 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -85,6 +85,8 @@ func NewAutoNAT(ctx context.Context, h host.Host, getAddrs GetAddrs) AutoNAT { } func (as *AmbientAutoNAT) Status() NATStatus { + as.mx.Lock() + defer as.mx.Unlock() return as.status } From 5ab68af1bcb044afbd836cd37e13f237c3d84d2c Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 2 Mar 2019 10:18:37 +0200 Subject: [PATCH 0963/3965] lock peer table in test to pacify the race detector --- p2p/host/autonat/autonat_test.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 5436619846..9e81453675 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -73,8 +73,9 @@ func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Mes func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, AutoNAT) { h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) a := NewAutoNAT(ctx, h, nil) + a.(*AmbientAutoNAT).mx.Lock() a.(*AmbientAutoNAT).peers[ash.ID()] = ash.Addrs() - + a.(*AmbientAutoNAT).mx.Unlock() return h, a } From 444e14b8788b4b40412f72fe71774bc199347b73 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 5 Mar 2019 12:11:48 +0200 Subject: [PATCH 0964/3965] bump window size to 16MiB --- p2p/muxer/yamux/yamux.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 12212eeb8f..05940af6d8 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -50,12 +50,12 @@ var DefaultTransport = (*Transport)(&yamux.Config{ EnableKeepAlive: true, // from yamux.DefaultConfig KeepAliveInterval: 30 * time.Second, // from yamux.DefaultConfig ConnectionWriteTimeout: 10 * time.Second, // from yamux.DefaultConfig - // We've bumped this to 1MiB as this critically limits throughput. + // We've bumped this to 16MiB as this critically limits throughput. // // 1MiB means a best case of 10MiB/s (83.89Mbps) on a connection with // 100ms latency. The default gave us 2.4MiB *best case* which was // totally unacceptable. - MaxStreamWindowSize: uint32(1024 * 1024), + MaxStreamWindowSize: uint32(16 * 1024 * 1024), LogOutput: ioutil.Discard, }) From d6a73bd9d7306f4539cfb8ee23ba7e9d23b42d14 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Mar 2019 13:58:46 -0800 Subject: [PATCH 0965/3965] remove all uses of multiaddrs NATs only care about TCP/UDP and ports. Using multiaddrs here made this library really hard to work with correctly. Furthermore, this library doesn't _actually_ support specifying the internal IP address. However, we'd still _act_ like the internal IP address mattered. This caused all sorts of mismatches. --- p2p/net/nat/mapping.go | 81 +++++++----------- p2p/net/nat/nat.go | 184 +++++++++------------------------------- p2p/net/nat/notifier.go | 2 +- 3 files changed, 71 insertions(+), 196 deletions(-) diff --git a/p2p/net/nat/mapping.go b/p2p/net/nat/mapping.go index b03b002d00..33f25f615f 100644 --- a/p2p/net/nat/mapping.go +++ b/p2p/net/nat/mapping.go @@ -2,12 +2,11 @@ package nat import ( "fmt" + "net" "sync" "time" "github.com/jbenet/goprocess" - ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" ) // Mapping represents a port mapping in a NAT. @@ -27,12 +26,9 @@ type Mapping interface { // established, port will be 0 ExternalPort() int - // InternalAddr returns the internal address. - InternalAddr() ma.Multiaddr - // ExternalAddr returns the external facing address. If the mapping is not // established, addr will be nil, and and ErrNoMapping will be returned. - ExternalAddr() (addr ma.Multiaddr, err error) + ExternalAddr() (addr net.Addr, err error) // Close closes the port mapping Close() error @@ -47,12 +43,9 @@ type mapping struct { intport int extport int permanent bool - intaddr ma.Multiaddr proc goprocess.Process - comment string - - cached ma.Multiaddr + cached net.IP cacheTime time.Time cacheLk sync.Mutex } @@ -87,55 +80,41 @@ func (m *mapping) setExternalPort(p int) { m.extport = p } -func (m *mapping) InternalAddr() ma.Multiaddr { - m.Lock() - defer m.Unlock() - return m.intaddr -} - -func (m *mapping) ExternalAddr() (ma.Multiaddr, error) { +func (m *mapping) ExternalAddr() (net.Addr, error) { m.cacheLk.Lock() - ctime := m.cacheTime - cval := m.cached - m.cacheLk.Unlock() - if time.Since(ctime) < CacheTime { - return cval, nil - } - - if m.ExternalPort() == 0 { // dont even try right now. + defer m.cacheLk.Unlock() + oport := m.ExternalPort() + if oport == 0 { + // dont even try right now. return nil, ErrNoMapping } - m.nat.natmu.Lock() - ip, err := m.nat.nat.GetExternalAddress() - m.nat.natmu.Unlock() - if err != nil { - return nil, err - } + if time.Since(m.cacheTime) >= CacheTime { + m.nat.natmu.Lock() + cval, err := m.nat.nat.GetExternalAddress() + m.nat.natmu.Unlock() - ipmaddr, err := manet.FromIP(ip) - if err != nil { - return nil, fmt.Errorf("error parsing ip") - } + if err != nil { + return nil, err + } - // call m.ExternalPort again, as mapping may have changed under our feet. (tocttou) - extport := m.ExternalPort() - if extport == 0 { - return nil, ErrNoMapping + m.cached = cval + m.cacheTime = time.Now() } - - tcp, err := ma.NewMultiaddr(fmt.Sprintf("/%s/%d", m.Protocol(), extport)) - if err != nil { - return nil, err + switch m.Protocol() { + case "tcp": + return &net.TCPAddr{ + IP: m.cached, + Port: oport, + }, nil + case "udp": + return &net.UDPAddr{ + IP: m.cached, + Port: oport, + }, nil + default: + panic(fmt.Sprintf("invalid protocol %q", m.Protocol())) } - - maddr2 := ipmaddr.Encapsulate(tcp) - - m.cacheLk.Lock() - m.cached = maddr2 - m.cacheTime = time.Now() - m.cacheLk.Unlock() - return maddr2, nil } func (m *mapping) Close() error { diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 0c47a4c2c2..938d1153b3 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -1,11 +1,9 @@ package nat import ( + "context" "errors" "fmt" - "net" - "strconv" - "strings" "sync" "time" @@ -13,8 +11,6 @@ import ( logging "github.com/ipfs/go-log" goprocess "github.com/jbenet/goprocess" periodic "github.com/jbenet/goprocess/periodic" - ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" ) var ( @@ -33,19 +29,38 @@ const CacheTime = time.Second * 15 // DiscoverNAT looks for a NAT device in the network and // returns an object that can manage port mappings. -func DiscoverNAT() *NAT { - nat, err := nat.DiscoverGateway() +func DiscoverNAT(ctx context.Context) (*NAT, error) { + var ( + natInstance nat.NAT + err error + ) + + done := make(chan struct{}) + go func() { + defer close(done) + // This will abort in 10 seconds anyways. + natInstance, err = nat.DiscoverGateway() + }() + + select { + case <-done: + case <-ctx.Done(): + return nil, ctx.Err() + } + if err != nil { - log.Debug("DiscoverGateway error:", err) - return nil + return nil, err } - addr, err := nat.GetDeviceAddress() + + // Log the device addr. + addr, err := natInstance.GetDeviceAddress() if err != nil { log.Debug("DiscoverGateway address error:", err) } else { log.Debug("DiscoverGateway address:", addr) } - return newNAT(nat) + + return newNAT(natInstance), nil } // NAT is an object that manages address port mappings in @@ -55,7 +70,7 @@ func DiscoverNAT() *NAT { type NAT struct { natmu sync.Mutex nat nat.NAT - proc goprocess.Process // manages nat mappings lifecycle + proc goprocess.Process mappingmu sync.RWMutex // guards mappings mappings map[*mapping]struct{} @@ -116,66 +131,31 @@ func (nat *NAT) rmMapping(m *mapping) { // NAT devices may not respect our port requests, and even lie. // Clients should not store the mapped results, but rather always // poll our object for the latest mappings. -func (nat *NAT) NewMapping(maddr ma.Multiaddr) (Mapping, error) { +func (nat *NAT) NewMapping(protocol string, port int) (Mapping, error) { if nat == nil { return nil, fmt.Errorf("no nat available") } - network, addr, err := manet.DialArgs(maddr) - if err != nil { - return nil, fmt.Errorf("DialArgs failed on addr: %s", maddr.String()) - } - - var ip net.IP - switch network { - case "tcp", "tcp4", "tcp6": - addr, err := net.ResolveTCPAddr(network, addr) - if err != nil { - return nil, err - } - ip = addr.IP - network = "tcp" - case "udp", "udp4", "udp6": - addr, err := net.ResolveUDPAddr(network, addr) - if err != nil { - return nil, err - } - ip = addr.IP - network = "udp" + switch protocol { + case "tcp", "udp": default: - return nil, fmt.Errorf("transport not supported by NAT: %s", network) - } - - // XXX: Known limitation: doesn't handle multiple internal addresses. - // If this applies to you, you can figure it out yourself. Ideally, the - // NAT library would allow us to handle this case but the "go way" - // appears to be to just "shrug" at edge-cases. - if !ip.IsUnspecified() { - internalAddr, err := nat.nat.GetInternalAddress() - if err != nil { - return nil, fmt.Errorf("failed to discover address on nat: %s", err) - } - if !ip.Equal(internalAddr) { - return nil, fmt.Errorf("nat address is %s, refusing to map %s", internalAddr, ip) - } - } - - intports := strings.Split(addr, ":")[1] - intport, err := strconv.Atoi(intports) - if err != nil { - return nil, err + return nil, fmt.Errorf("invalid protocol: %s", protocol) } m := &mapping{ + intport: port, nat: nat, - proto: network, - intport: intport, - intaddr: maddr, + proto: protocol, } + m.proc = goprocess.WithTeardown(func() error { nat.rmMapping(m) + nat.natmu.Lock() + defer nat.natmu.Unlock() + nat.nat.DeletePortMapping(m.Protocol(), m.InternalPort()) return nil }) + nat.addMapping(m) m.proc.AddChild(periodic.Every(MappingDuration/3, func(worker goprocess.Process) { @@ -193,9 +173,6 @@ func (nat *NAT) establishMapping(m *mapping) { log.Debugf("Attempting port map: %s/%d", m.Protocol(), m.InternalPort()) comment := "libp2p" - if m.comment != "" { - comment = "libp2p-" + m.comment - } nat.natmu.Lock() newport, err := nat.nat.AddPortMapping(m.Protocol(), m.InternalPort(), comment, MappingDuration) @@ -205,7 +182,7 @@ func (nat *NAT) establishMapping(m *mapping) { } nat.natmu.Unlock() - failure := func() { + if err != nil || newport == 0 { m.setExternalPort(0) // clear mapping // TODO: log.Event log.Warningf("failed to establish port mapping: %s", err) @@ -215,22 +192,11 @@ func (nat *NAT) establishMapping(m *mapping) { // we do not close if the mapping failed, // because it may work again next time. - } - - if err != nil || newport == 0 { - failure() return } m.setExternalPort(newport) - ext, err := m.ExternalAddr() - if err != nil { - log.Debugf("NAT Mapping addr error: %s %s", m.InternalAddr(), err) - failure() - return - } - - log.Debugf("NAT Mapping: %s --> %s", m.InternalAddr(), ext) + log.Debugf("NAT Mapping: %s --> %s (%s)", m.ExternalPort(), m.InternalPort(), m.Protocol()) if oldport != 0 && newport != oldport { log.Debugf("failed to renew same port mapping: ch %d -> %d", oldport, newport) nat.Notifier.notifyAll(func(n Notifiee) { @@ -242,73 +208,3 @@ func (nat *NAT) establishMapping(m *mapping) { n.MappingSuccess(nat, m) }) } - -// PortMapAddrs attempts to open (and continue to keep open) -// port mappings for given addrs. This function blocks until -// all addresses have been tried. This allows clients to -// retrieve results immediately after: -// -// nat.PortMapAddrs(addrs) -// mapped := nat.ExternalAddrs() -// -// Some may not succeed, and mappings may change over time; -// NAT devices may not respect our port requests, and even lie. -// Clients should not store the mapped results, but rather always -// poll our object for the latest mappings. -func (nat *NAT) PortMapAddrs(addrs []ma.Multiaddr) { - // spin off addr mappings independently. - var wg sync.WaitGroup - for _, addr := range addrs { - // do all of them concurrently - wg.Add(1) - go func(addr ma.Multiaddr) { - defer wg.Done() - nat.NewMapping(addr) - }(addr) - } - wg.Wait() -} - -// MappedAddrs returns address mappings NAT believes have been -// successfully established. Unsuccessful mappings are nil. This is: -// -// map[internalAddr]externalAddr -// -// This set of mappings _may not_ be correct, as NAT devices are finicky. -// Consider this with _best effort_ semantics. -func (nat *NAT) MappedAddrs() map[ma.Multiaddr]ma.Multiaddr { - - mappings := nat.Mappings() - addrmap := make(map[ma.Multiaddr]ma.Multiaddr, len(mappings)) - - for _, m := range mappings { - i := m.InternalAddr() - e, err := m.ExternalAddr() - if err != nil { - addrmap[i] = nil - } else { - addrmap[i] = e - } - } - return addrmap -} - -// ExternalAddrs returns a list of addresses that NAT believes have -// been successfully established. Unsuccessful mappings are omitted, -// so nat.ExternalAddrs() may return less addresses than nat.InternalAddrs(). -// To see which addresses are mapped, use nat.MappedAddrs(). -// -// This set of mappings _may not_ be correct, as NAT devices are finicky. -// Consider this with _best effort_ semantics. -func (nat *NAT) ExternalAddrs() []ma.Multiaddr { - mappings := nat.Mappings() - addrs := make([]ma.Multiaddr, 0, len(mappings)) - for _, m := range mappings { - a, err := m.ExternalAddr() - if err != nil { - continue // this mapping not currently successful. - } - addrs = append(addrs, a) - } - return addrs -} diff --git a/p2p/net/nat/notifier.go b/p2p/net/nat/notifier.go index 462a78b953..10fb6ac6bf 100644 --- a/p2p/net/nat/notifier.go +++ b/p2p/net/nat/notifier.go @@ -36,7 +36,7 @@ type Notifiee interface { // Called when mapping a port succeeds, but the mapping is // with a different port than an earlier success. - MappingChanged(nat *NAT, m Mapping, oldport int, newport int) + MappingChanged(nat *NAT, m Mapping, oldport, newport int) // Called when a port mapping fails. NAT will continue attempting after // the next period. To stop trying, use: mapping.Close(). After this failure, From 9523eec478a46869f2dfc0ce599cda28ae2cde2c Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 6 Mar 2019 13:40:29 +0200 Subject: [PATCH 0966/3965] gx publish 6.0.40 --- .gx/lastpubver | 2 +- package.json | 7 +++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 1f329afb6a..013215106f 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.39: QmRxk6AUaGaKCfzS1xSNRojiAPd7h2ih8GuCdjJBF3Y6GK +6.0.40: QmeKNtwDqFhJrq4R36QtNCemGvbMCBmzxoqKUJWj5Ueu3u diff --git a/package.json b/package.json index 4ef9c074d9..5bd187e451 100644 --- a/package.json +++ b/package.json @@ -235,9 +235,9 @@ }, { "author": "vyzo", - "hash": "QmZjvy5eqc1T9VbFe5KBiDryUJSRNbSbHZD7JMc6CeUXuu", + "hash": "QmUkgXU9YiNQG1jMjc2cscD96BcLWSocrXFXtZvyobBeGE", "name": "go-libp2p-autonat", - "version": "1.0.12" + "version": "1.0.13" }, { "hash": "QmYxUdYY9S6yg5tSPVin5GFTvtfsLauVcr7reHDD3dM8xf", @@ -250,6 +250,5 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.39" + "version": "6.0.40" } - From 943d7b398fc871282f7d14b0e16171ac0d3fbd2e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Mar 2019 09:58:32 -0800 Subject: [PATCH 0967/3965] expose all discovered NATs This change makes it possible to configure all discovered NATs, not just the first one found. --- nat.go | 65 +++++++++++++++++++++++++++++++++++++++++++++--------- natpmp.go | 40 +++++++++++++++++++++++---------- upnp.go | 66 +++++++++++++++++++++++++++++++++++++++---------------- 3 files changed, 131 insertions(+), 40 deletions(-) diff --git a/nat.go b/nat.go index eb574d63c2..4505df60bb 100644 --- a/nat.go +++ b/nat.go @@ -2,6 +2,7 @@ package nat import ( + "context" "errors" "math" "math/rand" @@ -34,20 +35,64 @@ type NAT interface { DeletePortMapping(protocol string, internalPort int) (err error) } +// DiscoverNATs returns all NATs discovered in the network. +func DiscoverNATs(ctx context.Context) <-chan NAT { + nats := make(chan NAT) + + go func() { + defer close(nats) + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + upnpIg1 := discoverUPNP_IG1(ctx) + upnpIg2 := discoverUPNP_IG2(ctx) + natpmp := discoverNATPMP(ctx) + upnpGenIGDev := discoverUPNP_GenIGDev(ctx) + for upnpIg1 != nil || upnpIg2 != nil || natpmp != nil { + var ( + nat NAT + ok bool + ) + select { + case nat, ok = <-upnpIg1: + if !ok { + upnpIg1 = nil + } + case nat, ok = <-upnpIg2: + if !ok { + upnpIg2 = nil + } + case nat, ok = <-upnpGenIGDev: + if !ok { + upnpGenIGDev = nil + } + case nat, ok = <-natpmp: + if !ok { + natpmp = nil + } + } + if ok { + select { + case nats <- nat: + case <-ctx.Done(): + return + } + } + } + }() + return nats +} + // DiscoverGateway attempts to find a gateway device. func DiscoverGateway() (NAT, error) { - select { - case nat := <-discoverUPNP_IG1(): - return nat, nil - case nat := <-discoverUPNP_IG2(): - return nat, nil - case nat := <-discoverUPNP_GenIGDev(): - return nat, nil - case nat := <-discoverNATPMP(): - return nat, nil - case <-time.After(10 * time.Second): + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + nat := <-DiscoverNATs(ctx) + if nat == nil { return nil, ErrNoNATFound } + return nat, nil } func randomPort() int { diff --git a/natpmp.go b/natpmp.go index 395a5dd578..a31a499988 100644 --- a/natpmp.go +++ b/natpmp.go @@ -1,6 +1,7 @@ package nat import ( + "context" "net" "time" @@ -12,25 +13,42 @@ var ( _ NAT = (*natpmpNAT)(nil) ) -func discoverNATPMP() <-chan NAT { +func discoverNATPMP(ctx context.Context) <-chan NAT { res := make(chan NAT, 1) ip, err := gateway.DiscoverGateway() if err == nil { - go discoverNATPMPWithAddr(res, ip) + go func() { + defer close(res) + // Unfortunately, we can't actually _stop_ the natpmp + // library. However, we can at least close _our_ channel + // and walk away. + select { + case client, ok := <-discoverNATPMPWithAddr(ip): + if ok { + res <- &natpmpNAT{client, ip, make(map[int]int)} + } + case <-ctx.Done(): + } + }() + } else { + close(res) } - return res } -func discoverNATPMPWithAddr(c chan NAT, ip net.IP) { - client := natpmp.NewClient(ip) - _, err := client.GetExternalAddress() - if err != nil { - return - } - - c <- &natpmpNAT{client, ip, make(map[int]int)} +func discoverNATPMPWithAddr(ip net.IP) <-chan *natpmp.Client { + res := make(chan *natpmp.Client, 1) + go func() { + defer close(res) + client := natpmp.NewClient(ip) + _, err := client.GetExternalAddress() + if err != nil { + return + } + res <- client + }() + return res } type natpmpNAT struct { diff --git a/upnp.go b/upnp.go index caad350114..ccfeb14a2f 100644 --- a/upnp.go +++ b/upnp.go @@ -1,6 +1,7 @@ package nat import ( + "context" "net" "net/url" "strings" @@ -17,9 +18,10 @@ var ( _ NAT = (*upnp_NAT)(nil) ) -func discoverUPNP_IG1() <-chan NAT { - res := make(chan NAT, 1) +func discoverUPNP_IG1(ctx context.Context) <-chan NAT { + res := make(chan NAT) go func() { + defer close(res) // find devices devs, err := goupnp.DiscoverDevices(internetgateway1.URN_WANConnectionDevice_1) @@ -33,6 +35,9 @@ func discoverUPNP_IG1() <-chan NAT { } dev.Root.Device.VisitServices(func(srv *goupnp.Service) { + if ctx.Err() != nil { + return + } switch srv.ServiceType { case internetgateway1.URN_WANIPConnection_1: client := &internetgateway1.WANIPConnection1{ServiceClient: goupnp.ServiceClient{ @@ -42,8 +47,10 @@ func discoverUPNP_IG1() <-chan NAT { }} _, isNat, err := client.GetNATRSIPStatus() if err == nil && isNat { - res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-IP1)", dev.Root} - return + select { + case res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-IP1)", dev.Root}: + case <-ctx.Done(): + } } case internetgateway1.URN_WANPPPConnection_1: @@ -54,8 +61,10 @@ func discoverUPNP_IG1() <-chan NAT { }} _, isNat, err := client.GetNATRSIPStatus() if err == nil && isNat { - res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-PPP1)", dev.Root} - return + select { + case res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-PPP1)", dev.Root}: + case <-ctx.Done(): + } } } @@ -66,9 +75,10 @@ func discoverUPNP_IG1() <-chan NAT { return res } -func discoverUPNP_IG2() <-chan NAT { - res := make(chan NAT, 1) +func discoverUPNP_IG2(ctx context.Context) <-chan NAT { + res := make(chan NAT) go func() { + defer close(res) // find devices devs, err := goupnp.DiscoverDevices(internetgateway2.URN_WANConnectionDevice_2) @@ -82,6 +92,9 @@ func discoverUPNP_IG2() <-chan NAT { } dev.Root.Device.VisitServices(func(srv *goupnp.Service) { + if ctx.Err() != nil { + return + } switch srv.ServiceType { case internetgateway2.URN_WANIPConnection_1: client := &internetgateway2.WANIPConnection1{ServiceClient: goupnp.ServiceClient{ @@ -91,8 +104,10 @@ func discoverUPNP_IG2() <-chan NAT { }} _, isNat, err := client.GetNATRSIPStatus() if err == nil && isNat { - res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-IP1)", dev.Root} - return + select { + case res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-IP1)", dev.Root}: + case <-ctx.Done(): + } } case internetgateway2.URN_WANIPConnection_2: @@ -103,8 +118,10 @@ func discoverUPNP_IG2() <-chan NAT { }} _, isNat, err := client.GetNATRSIPStatus() if err == nil && isNat { - res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-IP2)", dev.Root} - return + select { + case res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-IP2)", dev.Root}: + case <-ctx.Done(): + } } case internetgateway2.URN_WANPPPConnection_1: @@ -115,8 +132,10 @@ func discoverUPNP_IG2() <-chan NAT { }} _, isNat, err := client.GetNATRSIPStatus() if err == nil && isNat { - res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-PPP1)", dev.Root} - return + select { + case res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG2-PPP1)", dev.Root}: + case <-ctx.Done(): + } } } @@ -127,9 +146,11 @@ func discoverUPNP_IG2() <-chan NAT { return res } -func discoverUPNP_GenIGDev() <-chan NAT { +func discoverUPNP_GenIGDev(ctx context.Context) <-chan NAT { res := make(chan NAT, 1) go func() { + defer close(res) + DeviceList, err := ssdp.Search(ssdp.All, 5, "") if err != nil { return @@ -152,6 +173,9 @@ func discoverUPNP_GenIGDev() <-chan NAT { } RootDevice.Device.VisitServices(func(srv *goupnp.Service) { + if ctx.Err() != nil { + return + } switch srv.ServiceType { case internetgateway1.URN_WANIPConnection_1: client := &internetgateway1.WANIPConnection1{ServiceClient: goupnp.ServiceClient{ @@ -161,8 +185,10 @@ func discoverUPNP_GenIGDev() <-chan NAT { }} _, isNat, err := client.GetNATRSIPStatus() if err == nil && isNat { - res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-IP1)", RootDevice} - return + select { + case res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-IP1)", RootDevice}: + case <-ctx.Done(): + } } case internetgateway1.URN_WANPPPConnection_1: @@ -173,8 +199,10 @@ func discoverUPNP_GenIGDev() <-chan NAT { }} _, isNat, err := client.GetNATRSIPStatus() if err == nil && isNat { - res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-PPP1)", RootDevice} - return + select { + case res <- &upnp_NAT{client, make(map[int]int), "UPNP (IG1-PPP1)", RootDevice}: + case <-ctx.Done(): + } } } From c8594db014d29a36fe16880224818429f2a867c2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Mar 2019 10:44:40 -0800 Subject: [PATCH 0968/3965] try to pick the best NAT when multiple NATs are discovered --- nat.go | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/nat.go b/nat.go index 4505df60bb..727ae2a2e9 100644 --- a/nat.go +++ b/nat.go @@ -8,6 +8,8 @@ import ( "math/rand" "net" "time" + + "github.com/jackpal/gateway" ) var ErrNoExternalAddress = errors.New("no external address") @@ -88,11 +90,37 @@ func DiscoverNATs(ctx context.Context) <-chan NAT { func DiscoverGateway() (NAT, error) { ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() - nat := <-DiscoverNATs(ctx) - if nat == nil { + + var nats []NAT + for nat := range DiscoverNATs(ctx) { + nats = append(nats, nat) + } + switch len(nats) { + case 0: return nil, ErrNoNATFound + case 1: + return nats[0], nil + } + gw, _ := gateway.DiscoverGateway() + bestNAT := nats[0] + natGw, _ := bestNAT.GetDeviceAddress() + bestNATIsGw := gw != nil && natGw.Equal(gw) + // 1. Prefer gateways discovered _last_. This is an OK heuristic for + // discovering the most-upstream (furthest) NAT. + // 2. Prefer gateways that actually match our known gateway address. + // Some relays like to claim to be NATs even if they aren't. + for _, nat := range nats[1:] { + natGw, _ := nat.GetDeviceAddress() + natIsGw := gw != nil && natGw.Equal(gw) + + if bestNATIsGw && !natIsGw { + continue + } + + bestNATIsGw = natIsGw + bestNAT = nat } - return nat, nil + return bestNAT, nil } func randomPort() int { From d0ce3e1bac797582ffee08a2b55a2ba72f14e484 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Mar 2019 10:54:27 -0800 Subject: [PATCH 0969/3965] dep: update autonat --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index ac95f4404c..1909f869d6 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,7 @@ require ( github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 github.com/libp2p/go-conn-security v0.0.1 github.com/libp2p/go-conn-security-multistream v0.0.1 - github.com/libp2p/go-libp2p-autonat v0.0.1 + github.com/libp2p/go-libp2p-autonat v0.0.2 github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.1 github.com/libp2p/go-libp2p-crypto v0.0.1 diff --git a/go.sum b/go.sum index 137e6081cf..ca2ff48b15 100644 --- a/go.sum +++ b/go.sum @@ -86,6 +86,8 @@ github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atq github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-libp2p-autonat v0.0.1 h1:d5eskFxeJ4ag1ekhMC3yLTK+z+6RTw9W1Yv8HQma75k= github.com/libp2p/go-libp2p-autonat v0.0.1/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= +github.com/libp2p/go-libp2p-autonat v0.0.2 h1:ilo9QPzNPf1hMkqaPG55yzvhILf5ZtijstJhcii+l3s= +github.com/libp2p/go-libp2p-autonat v0.0.2/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= github.com/libp2p/go-libp2p-circuit v0.0.1 h1:DYbjyQ5ZY3QVAVYZWG4uzBQ6Wmcd1C82Bk8Q/pJlM1I= From 3617750071a6d1b22d8dc7a2639533e4f02eba21 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 5 Mar 2019 18:01:53 -0800 Subject: [PATCH 0970/3965] better nat mapping 1. Update to work with https://github.com/libp2p/go-libp2p-nat/pull/14. 2. Avoid observed addrs when our NAT tells us about external addrs. 3. Ignore bad addrs reported by our NAT. Substitute with observed addrs. 4. Fix https://github.com/libp2p/go-libp2p/issues/428. --- go.mod | 2 +- go.sum | 2 + p2p/host/basic/basic_host.go | 156 ++++++++++++++++++--- p2p/host/basic/natmgr.go | 223 ++++++++++++++++--------------- p2p/protocol/identify/id.go | 4 + p2p/protocol/identify/obsaddr.go | 40 +++++- package.json | 4 +- 7 files changed, 299 insertions(+), 132 deletions(-) diff --git a/go.mod b/go.mod index 1909f869d6..9e89318e5c 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/libp2p/go-libp2p-interface-pnet v0.0.1 github.com/libp2p/go-libp2p-loggables v0.0.1 github.com/libp2p/go-libp2p-metrics v0.0.1 - github.com/libp2p/go-libp2p-nat v0.0.1 + github.com/libp2p/go-libp2p-nat v0.0.2 github.com/libp2p/go-libp2p-net v0.0.1 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.0.1 diff --git a/go.sum b/go.sum index ca2ff48b15..7ae6ccbaef 100644 --- a/go.sum +++ b/go.sum @@ -108,6 +108,8 @@ github.com/libp2p/go-libp2p-metrics v0.0.1 h1:yumdPC/P2VzINdmcKZd0pciSUCpou+s0lw github.com/libp2p/go-libp2p-metrics v0.0.1/go.mod h1:jQJ95SXXA/K1VZi13h52WZMa9ja78zjyy5rspMsC/08= github.com/libp2p/go-libp2p-nat v0.0.1 h1:on/zju7XE+JXc8gH+vTKmIh2UJFC1K8kGnJYluQrlz4= github.com/libp2p/go-libp2p-nat v0.0.1/go.mod h1:4L6ajyUIlJvx1Cbh5pc6Ma6vMDpKXf3GgLO5u7W0oQ4= +github.com/libp2p/go-libp2p-nat v0.0.2 h1:sKI5hiCsGFhuEKdXMsF9mywQu2qhfoIGX6a+VG6zelE= +github.com/libp2p/go-libp2p-nat v0.0.2/go.mod h1:QrjXQSD5Dj4IJOdEcjHRkWTSomyxRo6HnUkf/TfQpLQ= github.com/libp2p/go-libp2p-net v0.0.1 h1:xJ4Vh4yKF/XKb8fd1Ev0ebAGzVjMxXzrxG2kjtU+F5Q= github.com/libp2p/go-libp2p-net v0.0.1/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= github.com/libp2p/go-libp2p-netutil v0.0.1 h1:LgD6+skofkOx8z6odD9+MZHKjupv3ng1u6KRhaADTnA= diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 1a89857808..6127d34001 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -3,12 +3,14 @@ package basichost import ( "context" "io" + "net" "time" logging "github.com/ipfs/go-log" goprocess "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" + inat "github.com/libp2p/go-libp2p-nat" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" @@ -17,6 +19,7 @@ import ( ping "github.com/libp2p/go-libp2p/p2p/protocol/ping" ma "github.com/multiformats/go-multiaddr" madns "github.com/multiformats/go-multiaddr-dns" + manet "github.com/multiformats/go-multiaddr-net" msmux "github.com/multiformats/go-multistream" ) @@ -485,17 +488,15 @@ func (h *BasicHost) Addrs() []ma.Multiaddr { } // mergeAddrs merges input address lists, leave only unique addresses -func mergeAddrs(addrLists ...[]ma.Multiaddr) (uniqueAddrs []ma.Multiaddr) { +func dedupAddrs(addrs []ma.Multiaddr) (uniqueAddrs []ma.Multiaddr) { exists := make(map[string]bool) - for _, addrList := range addrLists { - for _, addr := range addrList { - k := string(addr.Bytes()) - if exists[k] { - continue - } - exists[k] = true - uniqueAddrs = append(uniqueAddrs, addr) + for _, addr := range addrs { + k := string(addr.Bytes()) + if exists[k] { + continue } + exists[k] = true + uniqueAddrs = append(uniqueAddrs, addr) } return uniqueAddrs } @@ -507,19 +508,140 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { if err != nil { log.Debug("error retrieving network interface addrs") } - var observedAddrs []ma.Multiaddr - if h.ids != nil { - // peer observed addresses - observedAddrs = h.ids.OwnObservedAddrs() - } - var natAddrs []ma.Multiaddr + var natMappings []inat.Mapping + // natmgr is nil if we do not use nat option; // h.natmgr.NAT() is nil if not ready, or no nat is available. if h.natmgr != nil && h.natmgr.NAT() != nil { - natAddrs = h.natmgr.NAT().ExternalAddrs() + natMappings = h.natmgr.NAT().Mappings() } - return mergeAddrs(listenAddrs, observedAddrs, natAddrs) + finalAddrs := listenAddrs + if len(natMappings) > 0 { + + // We have successfully mapped ports on our NAT. Use those + // instead of observed addresses (mostly). + + // First, generate a mapping table. + // protocol -> internal port -> external addr + ports := make(map[string]map[int]net.Addr) + for _, m := range natMappings { + addr, err := m.ExternalAddr() + if err != nil { + // mapping not ready yet. + continue + } + protoPorts, ok := ports[m.Protocol()] + if !ok { + protoPorts = make(map[int]net.Addr) + ports[m.Protocol()] = protoPorts + } + protoPorts[m.InternalPort()] = addr + } + + // Next, apply this mapping to our addresses. + for _, listen := range listenAddrs { + found := false + transport, rest := ma.SplitFunc(listen, func(c ma.Component) bool { + if found { + return true + } + switch c.Protocol().Code { + case ma.P_TCP, ma.P_UDP: + found = true + } + return false + }) + if !manet.IsThinWaist(transport) { + continue + } + + naddr, err := manet.ToNetAddr(transport) + if err != nil { + log.Error("error parsing net multiaddr %q: %s", transport, err) + continue + } + + var ( + ip net.IP + iport int + protocol string + ) + switch naddr := naddr.(type) { + case *net.TCPAddr: + ip = naddr.IP + iport = naddr.Port + protocol = "tcp" + case *net.UDPAddr: + ip = naddr.IP + iport = naddr.Port + protocol = "udp" + default: + continue + } + + if !ip.IsGlobalUnicast() { + // We only map global unicast ports. + continue + } + + mappedAddr, ok := ports[protocol][iport] + if !ok { + // Not mapped. + continue + } + + mappedMaddr, err := manet.FromNetAddr(mappedAddr) + if err != nil { + log.Errorf("mapped addr can't be turned into a multiaddr %q: %s", mappedAddr, err) + continue + } + + // Did the router give us a routable public addr? + if manet.IsPublicAddr(mappedMaddr) { + // Yes, use it. + extMaddr := mappedMaddr + if rest != nil { + extMaddr = ma.Join(extMaddr, rest) + } + + // Add in the mapped addr. + finalAddrs = append(finalAddrs, extMaddr) + continue + } + + // No. Ok, let's try our observed addresses. + + // Now, check if we have any observed addresses that + // differ from the one reported by the router. Routers + // don't always give the most accurate information. + observed := h.ids.ObservedAddrsFor(listen) + + if len(observed) == 0 { + continue + } + + // Drop the IP from the external maddr + _, extMaddrNoIP := ma.SplitFirst(mappedMaddr) + + for _, obsMaddr := range observed { + // Extract a public observed addr. + ip, _ := ma.SplitFirst(obsMaddr) + if ip == nil || !manet.IsPublicAddr(ip) { + continue + } + + finalAddrs = append(finalAddrs, ma.Join(ip, extMaddrNoIP)) + } + } + } else { + var observedAddrs []ma.Multiaddr + if h.ids != nil { + observedAddrs = h.ids.OwnObservedAddrs() + } + finalAddrs = append(finalAddrs, observedAddrs...) + } + return dedupAddrs(finalAddrs) } // Close shuts down the Host's services (network, etc). diff --git a/p2p/host/basic/natmgr.go b/p2p/host/basic/natmgr.go index 349f79e789..005f9edec9 100644 --- a/p2p/host/basic/natmgr.go +++ b/p2p/host/basic/natmgr.go @@ -1,11 +1,12 @@ package basichost import ( - "context" + "net" + "strconv" "sync" goprocess "github.com/jbenet/goprocess" - lgbl "github.com/libp2p/go-libp2p-loggables" + goprocessctx "github.com/jbenet/goprocess/context" inat "github.com/libp2p/go-libp2p-nat" inet "github.com/libp2p/go-libp2p-net" ma "github.com/multiformats/go-multiaddr" @@ -37,11 +38,14 @@ func NewNATManager(net inet.Network) NATManager { // * closing the natManager closes the nat and its mappings. type natManager struct { net inet.Network - natmu sync.RWMutex // guards nat (ready could obviate this mutex, but safety first.) + natmu sync.RWMutex nat *inat.NAT - ready chan struct{} // closed once the nat is ready to process port mappings - proc goprocess.Process // natManager has a process + children. can be closed. + ready chan struct{} // closed once the nat is ready to process port mappings + + refreshMu sync.Mutex + + proc goprocess.Process // natManager has a process + children. can be closed. } func newNatManager(net inet.Network) *natManager { @@ -74,7 +78,6 @@ func (nmgr *natManager) Ready() <-chan struct{} { } func (nmgr *natManager) discoverNAT() { - nmgr.proc.Go(func(worker goprocess.Process) { // inat.DiscoverNAT blocks until the nat is found or a timeout // is reached. we unfortunately cannot specify timeouts-- the @@ -87,131 +90,137 @@ func (nmgr *natManager) discoverNAT() { // to avoid leaking resources in a non-obvious way. the only case // this affects is when the daemon is being started up and _immediately_ // asked to close. other services are also starting up, so ok to wait. - discoverdone := make(chan struct{}) - var nat *inat.NAT - go func() { - defer close(discoverdone) - nat = inat.DiscoverNAT() - }() - - // by this point -- after finding the NAT -- we may have already - // be closing. if so, just exit. - select { - case <-worker.Closing(): + + natInstance, err := inat.DiscoverNAT(goprocessctx.OnClosingContext(worker)) + if err != nil { + log.Error("DiscoverNAT error:", err) + close(nmgr.ready) return - case <-discoverdone: - if nat == nil { // no nat, or failed to get it. - return - } } - // wire up the nat to close when nmgr closes. - // nmgr.proc is our parent, and waiting for us. - nmgr.proc.AddChild(nat.Process()) - - // set the nat. nmgr.natmu.Lock() - nmgr.nat = nat + nmgr.nat = natInstance nmgr.natmu.Unlock() - - // signal that we're ready to process nat mappings: close(nmgr.ready) + // wire up the nat to close when nmgr closes. + // nmgr.proc is our parent, and waiting for us. + nmgr.proc.AddChild(nmgr.nat.Process()) + // sign natManager up for network notifications // we need to sign up here to avoid missing some notifs // before the NAT has been found. nmgr.net.Notify((*nmgrNetNotifiee)(nmgr)) - - // if any interfaces were brought up while we were setting up - // the nat, now is the time to setup port mappings for them. - // we release ready, then grab them to avoid losing any. adding - // a port mapping is idempotent, so its ok to add the same twice. - addrs := nmgr.net.ListenAddresses() - for _, addr := range addrs { - // we do it async because it's slow and we may want to close beforehand - go addPortMapping(nmgr, addr) - } + nmgr.refresh() }) } -// NAT returns the natManager's nat object. this may be nil, if -// (a) the search process is still ongoing, or (b) the search process -// found no nat. Clients must check whether the return value is nil. -func (nmgr *natManager) NAT() *inat.NAT { - nmgr.natmu.Lock() - defer nmgr.natmu.Unlock() - return nmgr.nat -} - -func addPortMapping(nmgr *natManager, intaddr ma.Multiaddr) { +func (nmgr *natManager) refresh() { nat := nmgr.NAT() if nat == nil { - panic("natManager addPortMapping called without a nat.") + // Nothing to do. + return } - // first, check if the port mapping already exists. - for _, mapping := range nat.Mappings() { - if mapping.InternalAddr().Equal(intaddr) { - return // it exists! return. + nmgr.proc.Go(func(_ goprocess.Process) { + nmgr.refreshMu.Lock() + defer nmgr.refreshMu.Unlock() + + ports := map[string]map[int]bool{ + "tcp": map[int]bool{}, + "udp": map[int]bool{}, } - } + for _, maddr := range nmgr.net.ListenAddresses() { + // Strip the IP + maIP, rest := ma.SplitFirst(maddr) + if maIP == nil || rest == nil { + continue + } - ctx := context.TODO() - lm := make(lgbl.DeferredMap) - lm["internalAddr"] = func() interface{} { return intaddr.String() } + switch maIP.Protocol().Code { + case ma.P_IP6, ma.P_IP4: + default: + continue + } - defer log.EventBegin(ctx, "natMgrAddPortMappingWait", lm).Done() + // Only bother if we're listening on a + // unicast/unspecified IP. + ip := net.IP(maIP.RawValue()) + if !(ip.IsGlobalUnicast() || ip.IsUnspecified()) { + continue + } - select { - case <-nmgr.proc.Closing(): - lm["outcome"] = "cancelled" - return // no use. - case <-nmgr.ready: // wait until it's ready. - } + // Extract the port/protocol + proto, _ := ma.SplitFirst(rest) + if proto == nil { + continue + } - // actually start the port map (sub-event because waiting may take a while) - defer log.EventBegin(ctx, "natMgrAddPortMapping", lm).Done() + var protocol string + switch proto.Protocol().Code { + case ma.P_TCP: + protocol = "tcp" + case ma.P_UDP: + protocol = "udp" + default: + continue + } - // get the nat - m, err := nat.NewMapping(intaddr) - if err != nil { - lm["outcome"] = "failure" - lm["error"] = err - return - } + port, err := strconv.ParseUint(proto.Value(), 10, 16) + if err != nil { + // bug in multiaddr + panic(err) + } + ports[protocol][int(port)] = false + } - extaddr, err := m.ExternalAddr() - if err != nil { - lm["outcome"] = "failure" - lm["error"] = err - return - } + var wg sync.WaitGroup + defer wg.Wait() + + // Close old mappings + for _, m := range nat.Mappings() { + mappedPort := m.InternalPort() + if _, ok := ports[m.Protocol()][mappedPort]; !ok { + // No longer need this mapping. + wg.Add(1) + go func(m inat.Mapping) { + defer wg.Done() + m.Close() + }(m) + } else { + // already mapped + ports[m.Protocol()][mappedPort] = true + } + } - lm["outcome"] = "success" - lm["externalAddr"] = func() interface{} { return extaddr.String() } - log.Infof("established nat port mapping: %s <--> %s", intaddr, extaddr) + // Create new mappings. + for proto, pports := range ports { + for port, mapped := range pports { + if mapped { + continue + } + wg.Add(1) + go func(proto string, port int) { + defer wg.Done() + _, err := nat.NewMapping(proto, port) + if err != nil { + log.Errorf("failed to port-map %s port %d: %s", proto, port, err) + } + }(proto, port) + } + } + }) } -func rmPortMapping(nmgr *natManager, intaddr ma.Multiaddr) { - nat := nmgr.NAT() - if nat == nil { - panic("natManager rmPortMapping called without a nat.") - } - - // list the port mappings (it may be gone on it's own, so we need to - // check this list, and not store it ourselves behind the scenes) - - // close mappings for this internal address. - for _, mapping := range nat.Mappings() { - if mapping.InternalAddr().Equal(intaddr) { - mapping.Close() - } - } +// NAT returns the natManager's nat object. this may be nil, if +// (a) the search process is still ongoing, or (b) the search process +// found no nat. Clients must check whether the return value is nil. +func (nmgr *natManager) NAT() *inat.NAT { + nmgr.natmu.Lock() + defer nmgr.natmu.Unlock() + return nmgr.nat } -// nmgrNetNotifiee implements the network notification listening part -// of the natManager. this is merely listening to Listen() and ListenClose() -// events. type nmgrNetNotifiee natManager func (nn *nmgrNetNotifiee) natManager() *natManager { @@ -219,19 +228,11 @@ func (nn *nmgrNetNotifiee) natManager() *natManager { } func (nn *nmgrNetNotifiee) Listen(n inet.Network, addr ma.Multiaddr) { - if nn.natManager().NAT() == nil { - return // not ready or doesnt exist. - } - - addPortMapping(nn.natManager(), addr) + nn.natManager().refresh() } func (nn *nmgrNetNotifiee) ListenClose(n inet.Network, addr ma.Multiaddr) { - if nn.natManager().NAT() == nil { - return // not ready or doesnt exist. - } - - rmPortMapping(nn.natManager(), addr) + nn.natManager().refresh() } func (nn *nmgrNetNotifiee) Connected(inet.Network, inet.Conn) {} diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index d565bc5f68..1ff7a568e5 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -74,6 +74,10 @@ func (ids *IDService) OwnObservedAddrs() []ma.Multiaddr { return ids.observedAddrs.Addrs() } +func (ids *IDService) ObservedAddrsFor(local ma.Multiaddr) []ma.Multiaddr { + return ids.observedAddrs.AddrsFor(local) +} + func (ids *IDService) IdentifyConn(c inet.Conn) { ids.currmu.Lock() if wait, found := ids.currid[c]; found { diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 40c6f78df2..1f6a10a489 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -30,6 +30,7 @@ type ObservedAddr struct { func (oa *ObservedAddr) activated(ttl time.Duration) bool { // cleanup SeenBy set now := time.Now() + for k, ob := range oa.SeenBy { if now.Sub(ob.seenTime) > ttl*ActivationThresh { delete(oa.SeenBy, k) @@ -51,6 +52,43 @@ type ObservedAddrSet struct { ttl time.Duration } +// AddrsFor return all activated observed addresses associated with the given +// (resolved) listen address. +func (oas *ObservedAddrSet) AddrsFor(addr ma.Multiaddr) (addrs []ma.Multiaddr) { + oas.Lock() + defer oas.Unlock() + + // for zero-value. + if len(oas.addrs) == 0 { + return nil + } + + key := string(addr.Bytes()) + observedAddrs, ok := oas.addrs[key] + if !ok { + return + } + + now := time.Now() + filteredAddrs := make([]*ObservedAddr, 0, len(observedAddrs)) + for _, a := range observedAddrs { + // leave only alive observed addresses + if now.Sub(a.LastSeen) <= oas.ttl { + filteredAddrs = append(filteredAddrs, a) + if a.activated(oas.ttl) { + addrs = append(addrs, a.Addr) + } + } + } + if len(filteredAddrs) > 0 { + oas.addrs[key] = filteredAddrs + } else { + delete(oas.addrs, key) + } + + return addrs +} + // Addrs return all activated observed addresses func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { oas.Lock() @@ -92,7 +130,7 @@ func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, now := time.Now() observerString := observerGroup(observer) - localString := local.String() + localString := string(local.Bytes()) ob := observation{ seenTime: now, connDirection: direction, diff --git a/package.json b/package.json index 5bd187e451..b21159674f 100644 --- a/package.json +++ b/package.json @@ -122,9 +122,9 @@ }, { "author": "whyrusleeping", - "hash": "QmZ1zCb95y9oHaJRMQmbXh3FgUuwz1V2mbCXBztnhehJkL", + "hash": "QmRbx7DYHgw3uNn2RuU2nv9Bdh96ZdtT65CG1CGPNRQcGZ", "name": "go-libp2p-nat", - "version": "0.8.12" + "version": "0.8.13" }, { "author": "whyrusleeping", From 8c8f08e36f8cc90bd81bd71d5848e04ea19bf7dd Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Mar 2019 08:32:04 -0800 Subject: [PATCH 0971/3965] natmgr: rename refresh to sync --- p2p/host/basic/natmgr.go | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/p2p/host/basic/natmgr.go b/p2p/host/basic/natmgr.go index 005f9edec9..6c0c663ba3 100644 --- a/p2p/host/basic/natmgr.go +++ b/p2p/host/basic/natmgr.go @@ -43,7 +43,7 @@ type natManager struct { ready chan struct{} // closed once the nat is ready to process port mappings - refreshMu sync.Mutex + syncMu sync.Mutex proc goprocess.Process // natManager has a process + children. can be closed. } @@ -111,11 +111,13 @@ func (nmgr *natManager) discoverNAT() { // we need to sign up here to avoid missing some notifs // before the NAT has been found. nmgr.net.Notify((*nmgrNetNotifiee)(nmgr)) - nmgr.refresh() + nmgr.sync() }) } -func (nmgr *natManager) refresh() { +// syncs the current NAT mappings, removing any outdated mappings and adding any +// new mappings. +func (nmgr *natManager) sync() { nat := nmgr.NAT() if nat == nil { // Nothing to do. @@ -123,8 +125,8 @@ func (nmgr *natManager) refresh() { } nmgr.proc.Go(func(_ goprocess.Process) { - nmgr.refreshMu.Lock() - defer nmgr.refreshMu.Unlock() + nmgr.syncMu.Lock() + defer nmgr.syncMu.Unlock() ports := map[string]map[int]bool{ "tcp": map[int]bool{}, @@ -228,11 +230,11 @@ func (nn *nmgrNetNotifiee) natManager() *natManager { } func (nn *nmgrNetNotifiee) Listen(n inet.Network, addr ma.Multiaddr) { - nn.natManager().refresh() + nn.natManager().sync() } func (nn *nmgrNetNotifiee) ListenClose(n inet.Network, addr ma.Multiaddr) { - nn.natManager().refresh() + nn.natManager().sync() } func (nn *nmgrNetNotifiee) Connected(inet.Network, inet.Conn) {} From 58f40b9d11276968df215e22649ad079957bc602 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Mar 2019 11:08:14 -0800 Subject: [PATCH 0972/3965] mocknet: fix data race --- p2p/net/mock/mock_link.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/p2p/net/mock/mock_link.go b/p2p/net/mock/mock_link.go index d123666620..9e31cbec2b 100644 --- a/p2p/net/mock/mock_link.go +++ b/p2p/net/mock/mock_link.go @@ -76,15 +76,21 @@ func (l *link) Peers() []peer.ID { } func (l *link) SetOptions(o LinkOptions) { + l.Lock() + defer l.Unlock() l.opts = o l.ratelimiter.UpdateBandwidth(l.opts.Bandwidth) } func (l *link) Options() LinkOptions { + l.RLock() + defer l.RUnlock() return l.opts } func (l *link) GetLatency() time.Duration { + l.RLock() + defer l.RUnlock() return l.opts.Latency } From e96aefc3b3840c7568c81520e3d1e5b5f4fbf7d0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Mar 2019 11:21:25 -0800 Subject: [PATCH 0973/3965] autorelay: disable consistantly failing auto-relay test --- p2p/host/relay/autorelay_test.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index c7bbeba7dc..379abd0ff0 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -133,6 +133,8 @@ func connect(t *testing.T, a, b host.Host) { // and the actual test! func TestAutoRelay(t *testing.T) { + t.Skip("fails 99% of the time") + ctx, cancel := context.WithCancel(context.Background()) defer cancel() From 5b643cde2de7430cd01f78c38e01df510a2c7797 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Mar 2019 11:27:28 -0800 Subject: [PATCH 0974/3965] mock: reduce peer count in FuzzManyPeers when the race detector is enabled --- p2p/net/mock/mock_test.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index ceeea38fd5..de7cc58224 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -586,7 +586,11 @@ func TestLimitedStreams(t *testing.T) { } } func TestFuzzManyPeers(t *testing.T) { - for i := 0; i < 50000; i++ { + peerCount := 50000 + if detectrace.WithRace() { + peerCount = 1000 + } + for i := 0; i < peerCount; i++ { _, err := FullMeshConnected(context.Background(), 2) if err != nil { t.Fatal(err) From 5c51323aa2fa397623ed03084cad20f62e25d635 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 6 Mar 2019 12:01:08 -0800 Subject: [PATCH 0975/3965] mock: increase tolerance for latency tests These were exceeding their tolerances when run under the race detector on CI. --- p2p/net/mock/mock_test.go | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index de7cc58224..fa53c16ec5 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -489,36 +489,36 @@ func TestAdding(t *testing.T) { func TestRateLimiting(t *testing.T) { rl := NewRateLimiter(10) - if !within(rl.Limit(10), time.Duration(float32(time.Second)), time.Millisecond/10) { - t.Fail() + if !within(rl.Limit(10), time.Duration(float32(time.Second)), time.Millisecond) { + t.Fatal() } if !within(rl.Limit(10), time.Duration(float32(time.Second*2)), time.Millisecond) { - t.Fail() + t.Fatal() } if !within(rl.Limit(10), time.Duration(float32(time.Second*3)), time.Millisecond) { - t.Fail() + t.Fatal() } if within(rl.Limit(10), time.Duration(float32(time.Second*3)), time.Millisecond) { - t.Fail() + t.Fatal() } rl.UpdateBandwidth(50) - if !within(rl.Limit(75), time.Duration(float32(time.Second)*1.5), time.Millisecond/10) { - t.Fail() + if !within(rl.Limit(75), time.Duration(float32(time.Second)*1.5), time.Millisecond) { + t.Fatal() } - if within(rl.Limit(75), time.Duration(float32(time.Second)*1.5), time.Millisecond/10) { - t.Fail() + if within(rl.Limit(75), time.Duration(float32(time.Second)*1.5), time.Millisecond) { + t.Fatal() } rl.UpdateBandwidth(100) - if !within(rl.Limit(1), time.Duration(time.Millisecond*10), time.Millisecond/10) { - t.Fail() + if !within(rl.Limit(1), time.Duration(time.Millisecond*10), time.Millisecond) { + t.Fatal() } - if within(rl.Limit(1), time.Duration(time.Millisecond*10), time.Millisecond/10) { - t.Fail() + if within(rl.Limit(1), time.Duration(time.Millisecond*10), time.Millisecond) { + t.Fatal() } } From 82ac6473304dbb81ef2ed88a048f52132da74036 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 7 Mar 2019 00:54:35 +0100 Subject: [PATCH 0976/3965] readme: update usage instructions for gomod. --- README.md | 75 ++++++++----------------------------------------------- 1 file changed, 10 insertions(+), 65 deletions(-) diff --git a/README.md b/README.md index a405e78535..504ef97da0 100644 --- a/README.md +++ b/README.md @@ -52,24 +52,23 @@ libp2p is the product of a long, and arduous quest of understanding -- a deep di > - [**The libp2p Specification**](https://github.com/libp2p/specs) > - [**go-libp2p implementation**](https://github.com/libp2p/go-libp2p) > - [**js-libp2p implementation**](https://github.com/libp2p/js-libp2p) +> - [**rust-libp2p implementation**](https://github.com/libp2p/rust-libp2p) +## Usage -## Bundles +This repository (`go-libp2p`) serves as the entrypoint to the universe of modules that compose the Go implementation of the libp2p stack. -There is currently only one bundle of `go-libp2p`, this package. This bundle is used by [`go-ipfs`](https://github.com/ipfs/go-ipfs). +We mainly use [Go modules](https://github.com/golang/go/wiki/Modules) for our dependency and release management (and thus require go >= 1.11). In order to get the best developer experience, we recommend you do too. Otherwise, you may ocassionally encounter a breaking build as you'll be running off master (which, by definition, is not guaranteed to be stable). -## Usage +You can start using go-libp2p in your Go application simply by adding imports from our repos, e.g.: -`go-libp2p` repo is a place holder for the list of Go modules that compose Go libp2p, as well as its entry point. +```go +import "github.com/libp2p/go-libp2p" +``` -### Install +The next time you run `go get` or `go build`, the Go build tools will look for [available releases](https://github.com/libp2p/go-libp2p/releases), and will pick the highest available one. -```bash -> go get -u -d github.com/libp2p/go-libp2p/... -> cd $GOPATH/src/github.com/libp2p/go-libp2p -> make -> make deps -``` +As new releases of go-libp2p are made available, you can upgrade your application by manually editing your `go.mod` file, or using the [Go tools](https://golang.org/cmd/go/#hdr-Maintaining_module_requirements) to maintain module requirements. ### API @@ -188,7 +187,6 @@ List of packages currently in existence for libp2p: | [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-examples) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-examples/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-examples/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | | [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit-progs) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | - # Contribute go-libp2p is part of [The IPFS Project](https://github.com/ipfs/ipfs), and is MIT licensed open source software. We welcome contributions big and small! Take a look at the [community contributing notes](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md). Please make sure to check the [issues](https://github.com/ipfs/go-libp2p/issues). Search the closed ones before reporting things, and help us with the open ones. @@ -208,56 +206,3 @@ There's a few things you can do right now to help out: - Go through the modules below and **check out existing issues**. This would be especially useful for modules in active development. Some knowledge of IPFS/libp2p may be required, as well as the infrasture behind it - for instance, you may need to read up on p2p and more complex operations like muxing to be able to help technically. - **Perform code reviews**. - **Add tests**. There can never be enough tests. - -## Modularizing go-libp2p - -We have currently a work in progress of modularizing go-libp2p from a repo monolith to several packages in different repos that can be reused for other projects of swapped for custom libp2p builds. - -We want to maintain history, so we'll use git-subtree for extracting packages. Find instructions below: - -```sh -# 1) create the extracted tree (has the directory specified as -P as its root) -> cd go-libp2p/ -> git subtree split -P p2p/crypto/secio/ -b libp2p-secio -62b0a5c21574bcbe06c422785cd5feff378ae5bd -# important to delete the tree now, so that outdated imports fail in step 5 -> git rm -r p2p/crypto/secio/ -> git commit -> cd ../ - -# 2) make the new repo -> mkdir go-libp2p-secio -> cd go-libp2p-secio/ -> git init && git commit --allow-empty - -# 3) fetch the extracted tree from the previous repo -> git remote add libp2p ../go-libp2p -> git fetch libp2p -> git reset --hard libp2p/libp2p-secio - -# 4) update self import paths -> sed -someflagsidontknow 'go-libp2p/p2p/crypto/secio' 'golibp2p-secio' -> git commit - -# 5) create package.json and check all imports are correct -> vim package.json -> gx --verbose install --global -> gx-go rewrite -> go test ./... -> gx-go rewrite --undo -> git commit - -# 4) make the package ready -> vim README.md LICENSE -> git commit - -# 5) bump the version separately -> vim package.json -> gx publish -> git add package.json .gx/ -> git commit -m 'Publish 1.2.3' - -# 6) clean up and push -> git remote rm libp2p -> git push origin master -``` From 2a7478131acd47f52823178c92f09b2fe8eda868 Mon Sep 17 00:00:00 2001 From: Leonhard Markert Date: Thu, 7 Mar 2019 03:23:01 +0000 Subject: [PATCH 0977/3965] Fix typos in p2p/net/README.md "Requet" -> "Request", "tls" -> "TLS" --- p2p/net/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/README.md b/p2p/net/README.md index 23e054b222..a7c9062a3c 100644 --- a/p2p/net/README.md +++ b/p2p/net/README.md @@ -4,9 +4,9 @@ The IPFS Network package handles all of the peer-to-peer networking. It connects - `Conn` - a connection to a single Peer - `MultiConn` - a set of connections to a single Peer - - `SecureConn` - an encrypted (tls-like) connection + - `SecureConn` - an encrypted (TLS-like) connection - `Swarm` - holds connections to Peers, multiplexes from/to each `MultiConn` -- `Muxer` - multiplexes between `Services` and `Swarm`. Handles `Requet/Reply`. +- `Muxer` - multiplexes between `Services` and `Swarm`. Handles `Request/Reply`. - `Service` - connects between an outside client service and Network. - `Handler` - the client service part that handles requests From eb20b61a6d2ecbe133b37a4508063e4c1e7f4266 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 7 Mar 2019 13:43:17 +0200 Subject: [PATCH 0978/3965] update go-smux-yamux --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 9e89318e5c..470c601b12 100644 --- a/go.mod +++ b/go.mod @@ -42,6 +42,6 @@ require ( github.com/multiformats/go-multistream v0.0.1 github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible - github.com/whyrusleeping/go-smux-yamux v2.0.8+incompatible + github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 ) diff --git a/go.sum b/go.sum index 7ae6ccbaef..dcacbd2146 100644 --- a/go.sum +++ b/go.sum @@ -202,6 +202,8 @@ github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible h1:BdYHctE9HJZL github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible/go.mod h1:dRWHHvc4HDQSHh9gbKEBbUZ+f2Q8iZTPG3UOGYODxSQ= github.com/whyrusleeping/go-smux-yamux v2.0.8+incompatible h1:IGm/UP/JpEFS6D787sZnZg7RA6fZIR9c/Ms9DeAVNuk= github.com/whyrusleeping/go-smux-yamux v2.0.8+incompatible/go.mod h1:6qHUzBXUbB9MXmw3AUdB52L8sEb/hScCqOdW2kj/wuI= +github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible h1:nVkExQ7pYlN9e45LcqTCOiDD0904fjtm0flnHZGbXkw= +github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible/go.mod h1:6qHUzBXUbB9MXmw3AUdB52L8sEb/hScCqOdW2kj/wuI= github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxfC1Y6mYncdk+kq8d5aLx0Q+/gyZGE44M= From e5e6e67c50098b289909b0faa82f076c2adaba69 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 7 Mar 2019 15:22:02 +0200 Subject: [PATCH 0979/3965] gx publish 6.0.41 --- .gx/lastpubver | 2 +- package.json | 22 +++++++++++----------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index 013215106f..9719aa2344 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -6.0.40: QmeKNtwDqFhJrq4R36QtNCemGvbMCBmzxoqKUJWj5Ueu3u +6.0.41: QmTRN7hRxvGkxKxDdeudty7sRet4L7ZKZCqKsXHa79wmAc diff --git a/package.json b/package.json index b21159674f..97265d44be 100644 --- a/package.json +++ b/package.json @@ -116,9 +116,9 @@ }, { "author": "whyrusleeping", - "hash": "Qma3Xp3FXFSP4prirEiRYHJ2tgGE8EAx9i6JLziPLpAQjq", + "hash": "QmQVoMEL1CxrVusTSUdYsiJXVBnvSqNUpBsGybkwSfksEF", "name": "go-libp2p-swarm", - "version": "3.0.34" + "version": "3.0.35" }, { "author": "whyrusleeping", @@ -146,9 +146,9 @@ }, { "author": "whyrusleeping", - "hash": "QmcVFwGS6sjfxVico2bd1gQGRu5A2ymwZunVmMdeV5zEYb", + "hash": "QmcqwxmCsWjNvQFzeZ4rzYVycWDY8C7Ejtiw2G3djArQzf", "name": "go-smux-yamux", - "version": "2.0.9" + "version": "2.0.10" }, { "author": "whyrusleeping", @@ -164,9 +164,9 @@ }, { "author": "vyzo", - "hash": "QmZBfqr863PYD7BKbmCFSNmzsqYmtr2DKgzubsQaiTQkMc", + "hash": "QmRTkLxADQRbgnhpt2zzQQJr8Ri764b7dujoDkZw33b3iE", "name": "go-libp2p-circuit", - "version": "2.3.14" + "version": "2.3.15" }, { "author": "lgierth", @@ -229,15 +229,15 @@ }, { "author": "vyzo", - "hash": "QmWXhsJTd4eTVAy9n8mDYiFmcMv1VHJ73qGkkeDHZfDhui", + "hash": "QmWA8k8apx6egshEjemkuxpKNJS7W7heCgzTnBhAvX9yoB", "name": "go-libp2p-discovery", - "version": "1.0.14" + "version": "1.0.15" }, { "author": "vyzo", - "hash": "QmUkgXU9YiNQG1jMjc2cscD96BcLWSocrXFXtZvyobBeGE", + "hash": "QmNnJhrc4ZtcPZg3oXDVdJ7HRtxDfLE2ccH56cZZjZ6y3p", "name": "go-libp2p-autonat", - "version": "1.0.13" + "version": "1.0.14" }, { "hash": "QmYxUdYY9S6yg5tSPVin5GFTvtfsLauVcr7reHDD3dM8xf", @@ -250,5 +250,5 @@ "license": "MIT", "name": "go-libp2p", "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.40" + "version": "6.0.41" } From 3899936deea191b565fbab337c7fea6711f10f27 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Fri, 8 Mar 2019 15:23:17 +1100 Subject: [PATCH 0980/3965] Add an error for no addresses --- p2p/net/swarm/swarm_dial.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 1bed042267..db7d88894d 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -289,7 +289,11 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { the improved rate limiter, while maintaining the outward behaviour that we previously had (halting a dial when we run out of addrs) */ - goodAddrs := s.filterKnownUndialables(s.peers.Addrs(p)) + peerAddrs := s.peers.Addrs(p) + if len(peerAddrs) == 0 { + return nil, errors.New("no addresses") + } + goodAddrs := s.filterKnownUndialables(peerAddrs) if len(goodAddrs) == 0 { return nil, errors.New("no good addresses") } From 31088db32d8daded300eed170e01f2dd0e9d7112 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Fri, 8 Mar 2019 15:24:02 +1100 Subject: [PATCH 0981/3965] Check for context failure, rather than a specific downstream error --- p2p/net/swarm/swarm_dial.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index db7d88894d..c0c5e69d74 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -84,7 +84,7 @@ const DefaultPerPeerRateLimit = 8 // DialBackoff is a type for tracking peer dial backoffs. // -// * It's safe to use it's zero value. +// * It's safe to use its zero value. // * It's thread-safe. // * It's *not* safe to move this type after using. type DialBackoff struct { @@ -248,7 +248,7 @@ func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { log.Debugf("ignoring dial error because we have a connection: %s", err) return conn, nil } - if err != context.Canceled { + if ctx.Err() == nil { log.Event(ctx, "swarmDialBackoffAdd", logdial) s.backf.AddBackoff(p) // let others know to backoff } From 4943e51c32abc487ac674c644ef5999764176252 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Fri, 8 Mar 2019 15:16:03 +1100 Subject: [PATCH 0982/3965] Implement fmt.Stringer for PeerInfo --- p2p/host/peerstore/peerinfo.go | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index f0ff3bad0e..6adaa20391 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -5,7 +5,7 @@ import ( "fmt" "strings" - "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) @@ -18,6 +18,12 @@ type PeerInfo struct { Addrs []ma.Multiaddr } +var _ fmt.Stringer = PeerInfo{} + +func (pi PeerInfo) String() string { + return fmt.Sprintf("{%v: %v}", pi.ID, pi.Addrs) +} + var ErrInvalidAddr = fmt.Errorf("invalid p2p multiaddr") func InfoFromP2pAddr(m ma.Multiaddr) (*PeerInfo, error) { From c6c1a3447084cacae3684fe14ada5b0fafc218e0 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 20 Feb 2019 18:21:46 +0800 Subject: [PATCH 0983/3965] implement the new handshake --- p2p/security/tls/crypto.go | 138 ++++++++++++++---------- p2p/security/tls/extension.go | 23 ++++ p2p/security/tls/extension_test.go | 19 ++++ p2p/security/tls/transport_test.go | 163 +++++++++++++++++++++++++---- 4 files changed, 270 insertions(+), 73 deletions(-) create mode 100644 p2p/security/tls/extension.go create mode 100644 p2p/security/tls/extension_test.go diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 0c31bf5d52..2b9cd52482 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -1,21 +1,31 @@ package libp2ptls import ( - "crypto" + "crypto/ecdsa" + "crypto/elliptic" "crypto/rand" "crypto/tls" "crypto/x509" + "crypto/x509/pkix" + "encoding/asn1" "errors" "fmt" "math/big" "time" + crypto "github.com/libp2p/go-libp2p-crypto" ic "github.com/libp2p/go-libp2p-crypto" - pb "github.com/libp2p/go-libp2p-crypto/pb" peer "github.com/libp2p/go-libp2p-peer" ) -const certValidityPeriod = 180 * 24 * time.Hour +const certValidityPeriod = 100 * 365 * 24 * time.Hour // ~100 years + +var extensionID = getPrefixedExtensionID([]int{1, 1}) + +type signedKey struct { + PubKey []byte + Signature []byte +} // Identity is used to secure connections type Identity struct { @@ -24,7 +34,7 @@ type Identity struct { // NewIdentity creates a new identity func NewIdentity(privKey ic.PrivKey) (*Identity, error) { - key, cert, err := keyToCertificate(privKey) + cert, err := keyToCertificate(privKey) if err != nil { return nil, err } @@ -33,10 +43,7 @@ func NewIdentity(privKey ic.PrivKey) (*Identity, error) { MinVersion: tls.VersionTLS13, InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. ClientAuth: tls.RequireAnyClientCert, - Certificates: []tls.Certificate{{ - Certificate: [][]byte{cert.Raw}, - PrivateKey: key, - }}, + Certificates: []tls.Certificate{*cert}, VerifyPeerCertificate: func(_ [][]byte, _ [][]*x509.Certificate) error { panic("tls config not specialized for peer") }, @@ -95,70 +102,95 @@ func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { if len(chain) != 1 { return nil, errors.New("expected one certificates in the chain") } + cert := chain[0] pool := x509.NewCertPool() - pool.AddCert(chain[0]) - if _, err := chain[0].Verify(x509.VerifyOptions{Roots: pool}); err != nil { + pool.AddCert(cert) + if _, err := cert.Verify(x509.VerifyOptions{Roots: pool}); err != nil { // If we return an x509 error here, it will be sent on the wire. // Wrap the error to avoid that. return nil, fmt.Errorf("certificate verification failed: %s", err) } - remotePubKey, err := x509.MarshalPKIXPublicKey(chain[0].PublicKey) + + var found bool + var keyExt pkix.Extension + // find the libp2p key extension, skipping all unknown extensions + for _, ext := range cert.Extensions { + if extensionIDEqual(ext.Id, extensionID) { + keyExt = ext + found = true + break + } + } + if !found { + return nil, errors.New("expected certificate to contain the key extension") + } + var sk signedKey + if _, err := asn1.Unmarshal(keyExt.Value, &sk); err != nil { + return nil, fmt.Errorf("unmarshalling signed certificate failed: %s", err) + } + pubKey, err := crypto.UnmarshalPublicKey(sk.PubKey) + if err != nil { + return nil, fmt.Errorf("unmarshalling public key failed: %s", err) + } + certKeyPub, err := x509.MarshalPKIXPublicKey(cert.PublicKey) if err != nil { return nil, err } - switch chain[0].PublicKeyAlgorithm { - case x509.RSA: - return ic.UnmarshalRsaPublicKey(remotePubKey) - case x509.ECDSA: - return ic.UnmarshalECDSAPublicKey(remotePubKey) - default: - return nil, fmt.Errorf("unexpected public key algorithm: %d", chain[0].PublicKeyAlgorithm) + valid, err := pubKey.Verify(certKeyPub, sk.Signature) + if err != nil { + return nil, fmt.Errorf("signature verification failed: %s", err) } + if !valid { + return nil, errors.New("signature invalid") + } + return pubKey, nil } -func keyToCertificate(sk ic.PrivKey) (crypto.PrivateKey, *x509.Certificate, error) { - sn, err := rand.Int(rand.Reader, big.NewInt(1<<62)) +func keyToCertificate(sk ic.PrivKey) (*tls.Certificate, error) { + certKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) if err != nil { - return nil, nil, err - } - tmpl := &x509.Certificate{ - SerialNumber: sn, - NotBefore: time.Now().Add(-24 * time.Hour), - NotAfter: time.Now().Add(certValidityPeriod), + return nil, err } - var privateKey crypto.PrivateKey - var publicKey crypto.PublicKey - raw, err := sk.Raw() + keyBytes, err := crypto.MarshalPublicKey(sk.GetPublic()) if err != nil { - return nil, nil, err + return nil, err } - switch sk.Type() { - case pb.KeyType_RSA: - k, err := x509.ParsePKCS1PrivateKey(raw) - if err != nil { - return nil, nil, err - } - publicKey = &k.PublicKey - privateKey = k - case pb.KeyType_ECDSA: - k, err := x509.ParseECPrivateKey(raw) - if err != nil { - return nil, nil, err - } - publicKey = &k.PublicKey - privateKey = k - // TODO: add support for Ed25519 - default: - return nil, nil, errors.New("unsupported key type for TLS") + certKeyPub, err := x509.MarshalPKIXPublicKey(certKey.Public()) + if err != nil { + return nil, err + } + signature, err := sk.Sign(certKeyPub) + if err != nil { + return nil, err + } + value, err := asn1.Marshal(signedKey{ + PubKey: keyBytes, + Signature: signature, + }) + if err != nil { + return nil, err } - certDER, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, publicKey, privateKey) + + sn, err := rand.Int(rand.Reader, big.NewInt(1<<62)) if err != nil { - return nil, nil, err + return nil, err + } + tmpl := &x509.Certificate{ + SerialNumber: sn, + NotBefore: time.Time{}, + NotAfter: time.Now().Add(certValidityPeriod), + // after calling CreateCertificate, these will end up in Certificate.Extensions + ExtraExtensions: []pkix.Extension{ + {Id: extensionID, Value: value}, + }, } - cert, err := x509.ParseCertificate(certDER) + certDER, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, certKey.Public(), certKey) if err != nil { - return nil, nil, err + return nil, err } - return privateKey, cert, nil + return &tls.Certificate{ + Certificate: [][]byte{certDER}, + PrivateKey: certKey, + }, nil } diff --git a/p2p/security/tls/extension.go b/p2p/security/tls/extension.go new file mode 100644 index 0000000000..33c106d928 --- /dev/null +++ b/p2p/security/tls/extension.go @@ -0,0 +1,23 @@ +package libp2ptls + +// TODO: get an assigment for a valid OID +var extensionPrefix = []int{1, 3, 6, 1, 4, 1, 123456789} + +// getPrefixedExtensionID returns an Object Identifier +// that can be used in x509 Certificates. +func getPrefixedExtensionID(suffix []int) []int { + return append(extensionPrefix, suffix...) +} + +// extensionIDEqual compares two extension IDs. +func extensionIDEqual(a, b []int) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} diff --git a/p2p/security/tls/extension_test.go b/p2p/security/tls/extension_test.go new file mode 100644 index 0000000000..425e8f4a3c --- /dev/null +++ b/p2p/security/tls/extension_test.go @@ -0,0 +1,19 @@ +package libp2ptls + +import ( + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +var _ = Describe("Extensions", func() { + It("generates a prefixed extension ID", func() { + Expect(getPrefixedExtensionID([]int{13, 37})).To(Equal([]int{1, 3, 6, 1, 4, 1, 123456789, 13, 37})) + }) + + It("compares extension IDs", func() { + Expect(extensionIDEqual([]int{1, 2, 3, 4}, []int{1, 2, 3, 4})).To(BeTrue()) + Expect(extensionIDEqual([]int{1, 2, 3, 4}, []int{1, 2, 3})).To(BeFalse()) + Expect(extensionIDEqual([]int{1, 2, 3}, []int{1, 2, 3, 4})).To(BeFalse()) + Expect(extensionIDEqual([]int{1, 2, 3, 4}, []int{4, 3, 2, 1})).To(BeFalse()) + }) +}) diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 02076eed92..62a8f1cd15 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -2,12 +2,15 @@ package libp2ptls import ( "context" + "crypto" "crypto/ecdsa" "crypto/elliptic" "crypto/rand" "crypto/rsa" "crypto/tls" "crypto/x509" + "crypto/x509/pkix" + "encoding/asn1" "fmt" "math/big" mrand "math/rand" @@ -15,6 +18,7 @@ import ( "time" "github.com/onsi/gomega/gbytes" + "github.com/onsi/gomega/types" cs "github.com/libp2p/go-conn-security" ci "github.com/libp2p/go-libp2p-crypto" @@ -26,7 +30,7 @@ import ( type transform struct { name string apply func(*Identity) - remoteErr string // the error that the side validating the chain gets + remoteErr types.GomegaMatcher // the error that the side validating the chain gets } var _ = Describe("Transport", func() { @@ -37,17 +41,22 @@ var _ = Describe("Transport", func() { createPeer := func() (peer.ID, ci.PrivKey) { var priv ci.PrivKey - if mrand.Int()%2 == 0 { + var err error + switch mrand.Int() % 4 { + case 0: fmt.Fprintf(GinkgoWriter, " using an ECDSA key: ") - var err error priv, _, err = ci.GenerateECDSAKeyPair(rand.Reader) - Expect(err).ToNot(HaveOccurred()) - } else { + case 1: fmt.Fprintf(GinkgoWriter, " using an RSA key: ") - var err error priv, _, err = ci.GenerateRSAKeyPair(1024, rand.Reader) - Expect(err).ToNot(HaveOccurred()) + case 2: + fmt.Fprintf(GinkgoWriter, " using an Ed25519 key: ") + priv, _, err = ci.GenerateEd25519Key(rand.Reader) + case 3: + fmt.Fprintf(GinkgoWriter, " using an Ed25519 key: ") + priv, _, err = ci.GenerateSecp256k1Key(rand.Reader) } + Expect(err).ToNot(HaveOccurred()) id, err := peer.IDFromPrivateKey(priv) Expect(err).ToNot(HaveOccurred()) fmt.Fprintln(GinkgoWriter, id.Pretty()) @@ -212,37 +221,149 @@ var _ = Describe("Transport", func() { }} } + getCertWithKey := func(key crypto.Signer, tmpl *x509.Certificate) tls.Certificate { + cert, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, key.Public(), key) + Expect(err).ToNot(HaveOccurred()) + return tls.Certificate{ + Certificate: [][]byte{cert}, + PrivateKey: key, + } + } + + getCert := func(tmpl *x509.Certificate) tls.Certificate { + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + Expect(err).ToNot(HaveOccurred()) + return getCertWithKey(key, tmpl) + } + expiredCert := func(identity *Identity) { - tmpl := &x509.Certificate{ + cert := getCert(&x509.Certificate{ SerialNumber: big.NewInt(1), NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(-time.Minute), - } - key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + }) + identity.config.Certificates = []tls.Certificate{cert} + } + + noKeyExtension := func(identity *Identity) { + cert := getCert(&x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + }) + identity.config.Certificates = []tls.Certificate{cert} + } + + unparseableKeyExtension := func(identity *Identity) { + cert := getCert(&x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + ExtraExtensions: []pkix.Extension{ + {Id: extensionID, Value: []byte("foobar")}, + }, + }) + identity.config.Certificates = []tls.Certificate{cert} + } + + unparseableKey := func(identity *Identity) { + data, err := asn1.Marshal(signedKey{PubKey: []byte("foobar")}) Expect(err).ToNot(HaveOccurred()) - cert, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, key.Public(), key) + cert := getCert(&x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + ExtraExtensions: []pkix.Extension{ + {Id: extensionID, Value: data}, + }, + }) + identity.config.Certificates = []tls.Certificate{cert} + } + + tooShortSignature := func(identity *Identity) { + key, _, err := ci.GenerateSecp256k1Key(rand.Reader) Expect(err).ToNot(HaveOccurred()) - identity.config.Certificates = []tls.Certificate{{ - Certificate: [][]byte{cert}, - PrivateKey: key, - }} + keyBytes, err := key.GetPublic().Bytes() + Expect(err).ToNot(HaveOccurred()) + data, err := asn1.Marshal(signedKey{ + PubKey: keyBytes, + Signature: []byte("foobar"), + }) + Expect(err).ToNot(HaveOccurred()) + cert := getCert(&x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + ExtraExtensions: []pkix.Extension{ + {Id: extensionID, Value: data}, + }, + }) + identity.config.Certificates = []tls.Certificate{cert} + } + + invalidSignature := func(identity *Identity) { + key, _, err := ci.GenerateSecp256k1Key(rand.Reader) + Expect(err).ToNot(HaveOccurred()) + keyBytes, err := key.GetPublic().Bytes() + Expect(err).ToNot(HaveOccurred()) + signature, err := key.Sign([]byte("foobar")) + Expect(err).ToNot(HaveOccurred()) + data, err := asn1.Marshal(signedKey{ + PubKey: keyBytes, + Signature: signature, + }) + Expect(err).ToNot(HaveOccurred()) + cert := getCert(&x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + ExtraExtensions: []pkix.Extension{ + {Id: extensionID, Value: data}, + }, + }) + identity.config.Certificates = []tls.Certificate{cert} } transforms := []transform{ { name: "private key used in the TLS handshake doesn't match the public key in the cert", apply: invalidateCertChain, - remoteErr: "tls: invalid certificate signature", + remoteErr: Equal("tls: invalid certificate signature"), }, { name: "certificate chain contains 2 certs", apply: twoCerts, - remoteErr: "expected one certificates in the chain", + remoteErr: Equal("expected one certificates in the chain"), }, { name: "cert is expired", apply: expiredCert, - remoteErr: "certificate verification failed: x509: certificate has expired or is not yet valid", + remoteErr: Equal("certificate verification failed: x509: certificate has expired or is not yet valid"), + }, + { + name: "cert doesn't have the key extension", + apply: noKeyExtension, + remoteErr: Equal("expected certificate to contain the key extension"), + }, + { + name: "key extension not parseable", + apply: unparseableKeyExtension, + remoteErr: ContainSubstring("asn1"), + }, + { + name: "key protobuf not parseable", + apply: unparseableKey, + remoteErr: ContainSubstring("unmarshalling public key failed: proto:"), + }, + { + name: "signature is malformed", + apply: tooShortSignature, + remoteErr: ContainSubstring("signature verification failed:"), + }, + { + name: "signature is invalid", + apply: invalidSignature, + remoteErr: Equal("signature invalid"), }, } @@ -262,7 +383,8 @@ var _ = Describe("Transport", func() { go func() { defer GinkgoRecover() _, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) - Expect(err).To(MatchError(t.remoteErr)) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(t.remoteErr) close(done) }() @@ -297,7 +419,8 @@ var _ = Describe("Transport", func() { }() _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) - Expect(err).To(MatchError(t.remoteErr)) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(t.remoteErr) Eventually(done).Should(BeClosed()) }) } From afcc2e4cffb400cbb1dfa2a9e98c03a9cc603cd0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 11 Mar 2019 23:06:32 +0900 Subject: [PATCH 0984/3965] fix logging when using secp256k1 key in tests Co-Authored-By: marten-seemann --- p2p/security/tls/transport_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 62a8f1cd15..82944560e4 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -53,7 +53,7 @@ var _ = Describe("Transport", func() { fmt.Fprintf(GinkgoWriter, " using an Ed25519 key: ") priv, _, err = ci.GenerateEd25519Key(rand.Reader) case 3: - fmt.Fprintf(GinkgoWriter, " using an Ed25519 key: ") + fmt.Fprintf(GinkgoWriter, " using an secp256k1 key: ") priv, _, err = ci.GenerateSecp256k1Key(rand.Reader) } Expect(err).ToNot(HaveOccurred()) From d42487d59b7f65380674ae1581876f89d68f1f3c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 11 Mar 2019 16:08:01 +0000 Subject: [PATCH 0985/3965] remove Jenkinsfile. --- ci/Jenkinsfile | 1 - 1 file changed, 1 deletion(-) delete mode 100644 ci/Jenkinsfile diff --git a/ci/Jenkinsfile b/ci/Jenkinsfile deleted file mode 100644 index b2067e6232..0000000000 --- a/ci/Jenkinsfile +++ /dev/null @@ -1 +0,0 @@ -golang() From 3b89938828d7baf5396f7550715e78cb55de02e1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 11 Mar 2019 16:08:01 +0000 Subject: [PATCH 0986/3965] remove Jenkinsfile. --- p2p/host/autonat/ci/Jenkinsfile | 1 - 1 file changed, 1 deletion(-) delete mode 100644 p2p/host/autonat/ci/Jenkinsfile diff --git a/p2p/host/autonat/ci/Jenkinsfile b/p2p/host/autonat/ci/Jenkinsfile deleted file mode 100644 index b2067e6232..0000000000 --- a/p2p/host/autonat/ci/Jenkinsfile +++ /dev/null @@ -1 +0,0 @@ -golang() From 53419e79397b3676c975f689beac885ad20e955b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 11 Mar 2019 16:08:01 +0000 Subject: [PATCH 0987/3965] remove Jenkinsfile. --- p2p/host/autonat/ci/Jenkinsfile | 1 - 1 file changed, 1 deletion(-) delete mode 100644 p2p/host/autonat/ci/Jenkinsfile diff --git a/p2p/host/autonat/ci/Jenkinsfile b/p2p/host/autonat/ci/Jenkinsfile deleted file mode 100644 index b2067e6232..0000000000 --- a/p2p/host/autonat/ci/Jenkinsfile +++ /dev/null @@ -1 +0,0 @@ -golang() From e426ef9d27106ae2b62c8284a54059ec4c9fd263 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 11 Mar 2019 18:53:11 -0700 Subject: [PATCH 0988/3965] gx import go-ssdp --- package.json | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/package.json b/package.json index 61cdf82719..b8fa3361bb 100644 --- a/package.json +++ b/package.json @@ -24,6 +24,12 @@ "hash": "QmYsYNh6saxUYHajdj49uiRzdxQgiFTtymrjf3d1f2Cer4", "name": "go-nat-pmp", "version": "1.0.1" + }, + { + "author": "koron", + "hash": "QmfDrLLyLVTLbDFfYvPsMqj3hgacxFCXyShZ5YjobqUAtr", + "name": "go-ssdp", + "version": "0.0.0" } ], "gxVersion": "0.9.0", From ab32e615c4b32a07ccbe9a857901c7e60a291754 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 12 Mar 2019 10:40:02 -0700 Subject: [PATCH 0989/3965] gx release 1.0.3 --- .gx/lastpubver | 2 +- package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gx/lastpubver b/.gx/lastpubver index d816451dee..0306793c53 100644 --- a/.gx/lastpubver +++ b/.gx/lastpubver @@ -1 +1 @@ -1.0.2: QmUTkdfn3e6EDb15C4XMHfooD5AoQxt5joYMeaqU56LU8h +1.0.3: QmdwkZHamNNrj7k3G29rnurmW3mFzsDhnyXppNcgYsiBVz diff --git a/package.json b/package.json index b8fa3361bb..7e176a89b2 100644 --- a/package.json +++ b/package.json @@ -36,6 +36,6 @@ "language": "go", "license": "Apache-2.0", "name": "go-nat", - "version": "1.0.2" + "version": "1.0.3" } From 8d26ea3591eae653289581ea1c0c84d036467a6e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 12 Mar 2019 10:45:40 -0700 Subject: [PATCH 0990/3965] switch to libp2p's go-nat fork --- p2p/net/nat/nat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 938d1153b3..178f7d0140 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -7,10 +7,10 @@ import ( "sync" "time" - nat "github.com/fd/go-nat" logging "github.com/ipfs/go-log" goprocess "github.com/jbenet/goprocess" periodic "github.com/jbenet/goprocess/periodic" + nat "github.com/libp2p/go-nat" ) var ( From 86b63ef35c8075dbbb6b9381ea973105660f22ea Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Wed, 13 Mar 2019 14:22:47 +1100 Subject: [PATCH 0991/3965] Don't modify the error handling --- p2p/net/swarm/swarm_dial.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index c0c5e69d74..8a3ba49172 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -248,7 +248,7 @@ func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { log.Debugf("ignoring dial error because we have a connection: %s", err) return conn, nil } - if ctx.Err() == nil { + if err != context.Canceled { log.Event(ctx, "swarmDialBackoffAdd", logdial) s.backf.AddBackoff(p) // let others know to backoff } From a2bf05d8819975bf957d7c6c5e367011f668da85 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 14 Mar 2019 09:46:12 +0900 Subject: [PATCH 0992/3965] use the new Protocol Labs PEN for the certificate extension --- p2p/security/tls/extension.go | 3 +-- p2p/security/tls/extension_test.go | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/security/tls/extension.go b/p2p/security/tls/extension.go index 33c106d928..9472c77e83 100644 --- a/p2p/security/tls/extension.go +++ b/p2p/security/tls/extension.go @@ -1,7 +1,6 @@ package libp2ptls -// TODO: get an assigment for a valid OID -var extensionPrefix = []int{1, 3, 6, 1, 4, 1, 123456789} +var extensionPrefix = []int{1, 3, 6, 1, 4, 1, 53594} // getPrefixedExtensionID returns an Object Identifier // that can be used in x509 Certificates. diff --git a/p2p/security/tls/extension_test.go b/p2p/security/tls/extension_test.go index 425e8f4a3c..5a7ef47756 100644 --- a/p2p/security/tls/extension_test.go +++ b/p2p/security/tls/extension_test.go @@ -7,7 +7,7 @@ import ( var _ = Describe("Extensions", func() { It("generates a prefixed extension ID", func() { - Expect(getPrefixedExtensionID([]int{13, 37})).To(Equal([]int{1, 3, 6, 1, 4, 1, 123456789, 13, 37})) + Expect(getPrefixedExtensionID([]int{13, 37})).To(Equal([]int{1, 3, 6, 1, 4, 1, 53594, 13, 37})) }) It("compares extension IDs", func() { From a49a4b1ccf801bbccaabf4ec410b418279050d8a Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 11 Mar 2019 15:01:47 +0900 Subject: [PATCH 0993/3965] use ChaCha if one of the peers doesn't have AES hardware support --- p2p/security/tls/crypto.go | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 2b9cd52482..c0ac657ec8 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -13,6 +13,8 @@ import ( "math/big" "time" + "golang.org/x/sys/cpu" + crypto "github.com/libp2p/go-libp2p-crypto" ic "github.com/libp2p/go-libp2p-crypto" peer "github.com/libp2p/go-libp2p-peer" @@ -40,10 +42,11 @@ func NewIdentity(privKey ic.PrivKey) (*Identity, error) { } return &Identity{ config: tls.Config{ - MinVersion: tls.VersionTLS13, - InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. - ClientAuth: tls.RequireAnyClientCert, - Certificates: []tls.Certificate{*cert}, + MinVersion: tls.VersionTLS13, + PreferServerCipherSuites: preferServerCipherSuites(), + InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. + ClientAuth: tls.RequireAnyClientCert, + Certificates: []tls.Certificate{*cert}, VerifyPeerCertificate: func(_ [][]byte, _ [][]*x509.Certificate) error { panic("tls config not specialized for peer") }, @@ -194,3 +197,25 @@ func keyToCertificate(sk ic.PrivKey) (*tls.Certificate, error) { PrivateKey: certKey, }, nil } + +// We want nodes without AES hardware (e.g. ARM) support to always use ChaCha. +// Only if both nodes have AES hardware support (e.g. x86), AES should be used. +// x86->x86: AES, ARM->x86: ChaCha, x86->ARM: ChaCha and ARM->ARM: Chacha +// This function returns true if we don't have AES hardware support, and false otherwise. +// Thus, ARM servers will always use their own cipher suite preferences (ChaCha first), +// and x86 servers will aways use the client's cipher suite preferences. +func preferServerCipherSuites() bool { + // Copied from the Go TLS implementation. + + // Check the cpu flags for each platform that has optimized GCM implementations. + // Worst case, these variables will just all be false. + var ( + hasGCMAsmAMD64 = cpu.X86.HasAES && cpu.X86.HasPCLMULQDQ + hasGCMAsmARM64 = cpu.ARM64.HasAES && cpu.ARM64.HasPMULL + // Keep in sync with crypto/aes/cipher_s390x.go. + hasGCMAsmS390X = cpu.S390X.HasAES && cpu.S390X.HasAESCBC && cpu.S390X.HasAESCTR && (cpu.S390X.HasGHASH || cpu.S390X.HasAESGCM) + + hasGCMAsm = hasGCMAsmAMD64 || hasGCMAsmARM64 || hasGCMAsmS390X + ) + return !hasGCMAsm +} From 2b073e1ebf1659955c45b29744d705b2705fc82c Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 14 Mar 2019 11:19:39 +0900 Subject: [PATCH 0994/3965] use a prefix when signing the public key --- p2p/security/tls/crypto.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index c0ac657ec8..c991e33b4b 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -21,6 +21,7 @@ import ( ) const certValidityPeriod = 100 * 365 * 24 * time.Hour // ~100 years +const certificatePrefix = "libp2p-tls-handshake:" var extensionID = getPrefixedExtensionID([]int{1, 1}) @@ -139,7 +140,7 @@ func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { if err != nil { return nil, err } - valid, err := pubKey.Verify(certKeyPub, sk.Signature) + valid, err := pubKey.Verify(append([]byte(certificatePrefix), certKeyPub...), sk.Signature) if err != nil { return nil, fmt.Errorf("signature verification failed: %s", err) } @@ -163,7 +164,7 @@ func keyToCertificate(sk ic.PrivKey) (*tls.Certificate, error) { if err != nil { return nil, err } - signature, err := sk.Sign(certKeyPub) + signature, err := sk.Sign(append([]byte(certificatePrefix), certKeyPub...)) if err != nil { return nil, err } From 5da711dc6c1ffb9f79258b53758072a62e64ddd9 Mon Sep 17 00:00:00 2001 From: lnykww Date: Thu, 14 Mar 2019 10:57:19 +0800 Subject: [PATCH 0995/3965] Fix UDP listen on a Unspecified Address and Dial from the Unspecified Address LocalMultiaddr May return the Unspecified Address, if so, we need to check if we listen on the Unspecified address and they were equal. --- p2p/protocol/identify/id.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 1ff7a568e5..e909c4071f 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -413,7 +413,7 @@ func (ids *IDService) consumeObservedAddress(observed []byte, c inet.Conn) { } log.Debugf("identify identifying observed multiaddr: %s %s", c.LocalMultiaddr(), ifaceaddrs) - if !addrInAddrs(c.LocalMultiaddr(), ifaceaddrs) { + if !addrInAddrs(c.LocalMultiaddr(), ifaceaddrs) && !addrInAddrs(c.LocalMultiaddr(), ids.Host.Network().ListenAddresses()) { // not in our list return } From 2b1f7dd23fb28d2ec3f3e54a0ed704d1b37e47b2 Mon Sep 17 00:00:00 2001 From: Andrew Nesbitt Date: Fri, 15 Mar 2019 14:51:31 +0000 Subject: [PATCH 0996/3965] Remove jenkins column from package table Also update travis-ci.org to travis-ci.com --- README.md | 116 ++++++++++++++++++++++------------------------ package-list.json | 1 - 2 files changed, 56 insertions(+), 61 deletions(-) diff --git a/README.md b/README.md index 504ef97da0..cb198624f5 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@

- +
@@ -110,82 +110,78 @@ $ gx test ./p2p/ List of packages currently in existence for libp2p: -| Name | CI/Travis | CI/Jenkins | Coverage | Description | -| ---------|---------|---------|---------|--------- | +| Name | CI | Coverage | Description | +| ---------|---------|---------|--------- | | **Libp2p** | -| [`go-libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | go-libp2p entry point | -| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-host) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | libp2p "host" interface | -| [`go-libp2p-blankhost`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-blankhost) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | minimal implementation of the "host" interface | +| [`go-libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | go-libp2p entry point | +| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-host) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | libp2p "host" interface | +| [`go-libp2p-blankhost`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-blankhost) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | minimal implementation of the "host" interface | | **Network** | -| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-net) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | libp2p connection and "network" interfaces | -| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-swarm) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-swarm/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-swarm/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | reference implementation | +| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-net) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | libp2p connection and "network" interfaces | +| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-swarm) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | reference implementation | | **Transport** | -| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | interfaces | -| [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-ws-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | WebSocket transport | -| [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-tcp-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | TCP transport | -| [`go-libp2p-quic-transport`](//github.com/libp2p/go-libp2p-quic-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-quic-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-quic-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-quic-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-quic-transport) | QUIC transport | -| [`go-udp-transport`](//github.com/libp2p/go-udp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-udp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-udp-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-udp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-udp-transport) | UDP transport | -| [`go-utp-transport`](//github.com/libp2p/go-utp-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-utp-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-utp-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-utp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-utp-transport) | uTorrent transport (UTP) | -| [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-circuit/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-circuit/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | relay transport | -| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-transport-upgrader) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | upgrades multiaddr-net connections into full libp2p transports | -| [`go-libp2p-reuseport-transport`](//github.com/libp2p/go-libp2p-reuseport-transport) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-reuseport-transport.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-reuseport-transport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport) | partial transport for building transports that reuse ports | +| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | interfaces | +| [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-ws-transport) | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | WebSocket transport | +| [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-tcp-transport) | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | TCP transport | +| [`go-libp2p-quic-transport`](//github.com/libp2p/go-libp2p-quic-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-quic-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-quic-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-quic-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-quic-transport) | QUIC transport | +| [`go-udp-transport`](//github.com/libp2p/go-udp-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-udp-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-udp-transport) | [![codecov](https://codecov.io/gh/libp2p/go-udp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-udp-transport) | UDP transport | +| [`go-utp-transport`](//github.com/libp2p/go-utp-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-utp-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-utp-transport) | [![codecov](https://codecov.io/gh/libp2p/go-utp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-utp-transport) | uTorrent transport (UTP) | +| [`go-libp2p-circuit`](//github.com/libp2p/go-libp2p-circuit) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-circuit.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-circuit) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit) | relay transport | +| [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-transport-upgrader) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | upgrades multiaddr-net connections into full libp2p transports | +| [`go-libp2p-reuseport-transport`](//github.com/libp2p/go-libp2p-reuseport-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-reuseport-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-reuseport-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport) | partial transport for building transports that reuse ports | | **Encrypted Channels** | -| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | interfaces | -| [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-secio) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | SecIO crypto channel | -| [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.org/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.org/libp2p/go-conn-security-multistream) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | multistream multiplexed meta crypto channel | +| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.com/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.com/libp2p/go-conn-security) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | interfaces | +| [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-secio) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | SecIO crypto channel | +| [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.com/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.com/libp2p/go-conn-security-multistream) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | multistream multiplexed meta crypto channel | | **Private Network** | -| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-pnet) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | interfaces | -| [`go-libp2p-pnet`](//github.com/libp2p/go-libp2p-pnet) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pnet.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pnet) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pnet) | reference implementation | +| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | interfaces | +| [`go-libp2p-pnet`](//github.com/libp2p/go-libp2p-pnet) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-pnet.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pnet) | reference implementation | | **Stream Muxers** | -| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.org/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.org/libp2p/go-stream-muxer) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | interfaces | -| [`go-smux-yamux`](//github.com/whyrusleeping/go-smux-yamux) | [![Travis CI](https://travis-ci.org/whyrusleeping/go-smux-yamux.svg?branch=master)](https://travis-ci.org/whyrusleeping/go-smux-yamux) | N/A | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-yamux/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-yamux) | YAMUX stream multiplexer | -| [`go-smux-mplex`](//github.com/whyrusleeping/go-smux-mplex) | [![Travis CI](https://travis-ci.org/whyrusleeping/go-smux-mplex.svg?branch=master)](https://travis-ci.org/whyrusleeping/go-smux-mplex) | N/A | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-mplex/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-mplex) | MPLEX stream multiplexer | +| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.com/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.com/libp2p/go-stream-muxer) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | interfaces | +| [`go-smux-yamux`](//github.com/whyrusleeping/go-smux-yamux) | [![Travis CI](https://travis-ci.com/whyrusleeping/go-smux-yamux.svg?branch=master)](https://travis-ci.com/whyrusleeping/go-smux-yamux) | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-yamux/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-yamux) | YAMUX stream multiplexer | +| [`go-smux-mplex`](//github.com/whyrusleeping/go-smux-mplex) | [![Travis CI](https://travis-ci.com/whyrusleeping/go-smux-mplex.svg?branch=master)](https://travis-ci.com/whyrusleeping/go-smux-mplex) | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-mplex/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-mplex) | MPLEX stream multiplexer | | **NAT Traversal** | -| [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-nat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | | -| [`go-libp2p-autonat`](//github.com/libp2p/go-libp2p-autonat) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat) | NAT autodetection (client) | -| [`go-libp2p-autonat-svc`](//github.com/libp2p/go-libp2p-autonat-svc) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-autonat-svc.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-autonat-svc) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-autonat-svc) | NAT autodetection (service) | +| [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-nat) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | | | **Peerstore** | -| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peerstore) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | interfaces and reference implementation | +| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-peerstore) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | interfaces and reference implementation | | **Connection Manager** | -| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-interface-connmgr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | interface | -| [`go-libp2p-connmgr`](//github.com/libp2p/go-libp2p-connmgr) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-connmgr.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-connmgr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-connmgr) | reference implementation | +| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | interface | +| [`go-libp2p-connmgr`](//github.com/libp2p/go-libp2p-connmgr) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-connmgr.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-connmgr) | reference implementation | | **Routing** | -| [`go-libp2p-routing`](//github.com/libp2p/go-libp2p-routing) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing) | routing interfaces | -| [`go-libp2p-record`](//github.com/libp2p/go-libp2p-record) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-record.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-record) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-record/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-record) | record type and validator logic | -| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-routing-helpers/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-routing-helpers/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | -| [`go-libp2p-kad-dht`](//github.com/libp2p/go-libp2p-kad-dht) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kad-dht) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-kad-dht/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-kad-dht/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kad-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kad-dht) | Kademlia-like router | -| [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub-router) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-pubsub-router/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-pubsub-router/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | -| **Discovery** | -| [`go-libp2p-discovery`](//github.com/libp2p/go-libp2p-discovery) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-discovery.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-discovery) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-discovery) | active peer disovery interface | +| [`go-libp2p-routing`](//github.com/libp2p/go-libp2p-routing) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-routing.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-routing) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing) | routing interfaces | +| [`go-libp2p-record`](//github.com/libp2p/go-libp2p-record) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-record.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-record) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-record/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-record) | record type and validator logic | +| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-routing-helpers) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | +| [`go-libp2p-kad-dht`](//github.com/libp2p/go-libp2p-kad-dht) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-kad-dht) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kad-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kad-dht) | Kademlia-like router | +| [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-pubsub-router) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | | **Consensus** | -| [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-consensus) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | -| [`go-libp2p-raft`](//github.com/libp2p/go-libp2p-raft) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-raft.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-raft) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-raft/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-raft/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-raft/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-raft) | consensus implementation over raft | +| [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-consensus) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | +| [`go-libp2p-raft`](//github.com/libp2p/go-libp2p-raft) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-raft.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-raft) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-raft/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-raft) | consensus implementation over raft | | **Pubsub** | -| [`go-libp2p-pubsub`](//github.com/libp2p/go-libp2p-pubsub) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-pubsub.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-pubsub) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-pubsub/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-pubsub/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub) | multiple pubsub over libp2p implementations | +| [`go-libp2p-pubsub`](//github.com/libp2p/go-libp2p-pubsub) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-pubsub.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-pubsub) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub) | multiple pubsub over libp2p implementations | | **RPC** | -| [`go-libp2p-gorpc`](//github.com/libp2p/go-libp2p-gorpc) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-gorpc.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-gorpc) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-gorpc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-gorpc) | a simple RPC library for libp2p | +| [`go-libp2p-gorpc`](//github.com/libp2p/go-libp2p-gorpc) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-gorpc.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-gorpc) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-gorpc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-gorpc) | a simple RPC library for libp2p | | **Metrics** | -| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | libp2p metrics interfaces/collectors | +| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-metrics) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | libp2p metrics interfaces/collectors | | **Data Types** | -| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-peer) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | libp2p peer-ID datatype | -| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-crypto) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | libp2p key types | -| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-protocol) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | libp2p protocol datatype | -| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kbucket) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademlia routing table helper types | +| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-peer) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | libp2p peer-ID datatype | +| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-crypto) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | libp2p key types | +| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-protocol) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | libp2p protocol datatype | +| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-kbucket) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademlia routing table helper types | | **Utilities/miscellaneous** | -| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-loggables) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | -| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.org/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.org/libp2p/go-maddr-filter) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | multiaddr filtering helpers | -| [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-netutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | misc utilities | -| [`go-msgio`](//github.com/libp2p/go-msgio) | [![Travis CI](https://travis-ci.org/libp2p/go-msgio.svg?branch=master)](https://travis-ci.org/libp2p/go-msgio) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-msgio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-msgio) | length prefixed data channel | -| [`go-addr-util`](//github.com/libp2p/go-addr-util) | [![Travis CI](https://travis-ci.org/libp2p/go-addr-util.svg?branch=master)](https://travis-ci.org/libp2p/go-addr-util) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-addr-util/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-addr-util) | address utilities for libp2p swarm | -| [`go-buffer-pool`](//github.com/libp2p/go-buffer-pool) | [![Travis CI](https://travis-ci.org/libp2p/go-buffer-pool.svg?branch=master)](https://travis-ci.org/libp2p/go-buffer-pool) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-buffer-pool/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-buffer-pool) | a variable size buffer pool for go | -| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-routing-helpers) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-routing-helpers/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-routing-helpers/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | routing helpers | -| [`go-reuseport`](//github.com/libp2p/go-reuseport) | [![Travis CI](https://travis-ci.org/libp2p/go-reuseport.svg?branch=master)](https://travis-ci.org/libp2p/go-reuseport) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-reuseport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-reuseport) | enables reuse of addresses | -| [`go-sockaddr`](//github.com/libp2p/go-sockaddr) | [![Travis CI](https://travis-ci.org/libp2p/go-sockaddr.svg?branch=master)](https://travis-ci.org/libp2p/go-sockaddr) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-sockaddr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-sockaddr) | utils for sockaddr conversions | -| [`go-flow-metrics`](//github.com/libp2p/go-flow-metrics) | [![Travis CI](https://travis-ci.org/libp2p/go-flow-metrics.svg?branch=master)](https://travis-ci.org/libp2p/go-flow-metrics) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-flow-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-flow-metrics) | metrics library | +| [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-loggables) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | +| [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.com/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.com/libp2p/go-maddr-filter) | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | multiaddr filtering helpers | +| [`go-libp2p-netutil`](//github.com/libp2p/go-libp2p-netutil) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-netutil.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-netutil) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-netutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-netutil) | misc utilities | +| [`go-msgio`](//github.com/libp2p/go-msgio) | [![Travis CI](https://travis-ci.com/libp2p/go-msgio.svg?branch=master)](https://travis-ci.com/libp2p/go-msgio) | [![codecov](https://codecov.io/gh/libp2p/go-msgio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-msgio) | length prefixed data channel | +| [`go-addr-util`](//github.com/libp2p/go-addr-util) | [![Travis CI](https://travis-ci.com/libp2p/go-addr-util.svg?branch=master)](https://travis-ci.com/libp2p/go-addr-util) | [![codecov](https://codecov.io/gh/libp2p/go-addr-util/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-addr-util) | address utilities for libp2p swarm | +| [`go-buffer-pool`](//github.com/libp2p/go-buffer-pool) | [![Travis CI](https://travis-ci.com/libp2p/go-buffer-pool.svg?branch=master)](https://travis-ci.com/libp2p/go-buffer-pool) | [![codecov](https://codecov.io/gh/libp2p/go-buffer-pool/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-buffer-pool) | a variable size buffer pool for go | +| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-routing-helpers) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | routing helpers | +| [`go-reuseport`](//github.com/libp2p/go-reuseport) | [![Travis CI](https://travis-ci.com/libp2p/go-reuseport.svg?branch=master)](https://travis-ci.com/libp2p/go-reuseport) | [![codecov](https://codecov.io/gh/libp2p/go-reuseport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-reuseport) | enables reuse of addresses | +| [`go-sockaddr`](//github.com/libp2p/go-sockaddr) | [![Travis CI](https://travis-ci.com/libp2p/go-sockaddr.svg?branch=master)](https://travis-ci.com/libp2p/go-sockaddr) | [![codecov](https://codecov.io/gh/libp2p/go-sockaddr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-sockaddr) | utils for sockaddr conversions | +| [`go-flow-metrics`](//github.com/libp2p/go-flow-metrics) | [![Travis CI](https://travis-ci.com/libp2p/go-flow-metrics.svg?branch=master)](https://travis-ci.com/libp2p/go-flow-metrics) | [![codecov](https://codecov.io/gh/libp2p/go-flow-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-flow-metrics) | metrics library | | **Testing and examples** | -| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.org/libp2p/go-testutil.svg?branch=master)](https://travis-ci.org/libp2p/go-testutil) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | a collection of testing utilities for ipfs and libp2p | -| [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-examples) | [![jenkins](https://ci.ipfs.team/buildStatus/icon?job=libp2p/go-libp2p-examples/master)](https://ci.ipfs.team/job/libp2p/job/go-libp2p-examples/job/master/) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | -| [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.org/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-circuit-progs) | N/A | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | +| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.com/libp2p/go-testutil.svg?branch=master)](https://travis-ci.com/libp2p/go-testutil) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | a collection of testing utilities for ipfs and libp2p | +| [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-examples) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | +| [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-circuit-progs) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | # Contribute diff --git a/package-list.json b/package-list.json index 142eda0a23..e29a741036 100644 --- a/package-list.json +++ b/package-list.json @@ -2,7 +2,6 @@ "columns": [ "Name", "CI/Travis", - "CI/Jenkins", "Coverage", "Description" ], From 7741442478a567fa9371248f87b286936f1b3067 Mon Sep 17 00:00:00 2001 From: Lars Gierth Date: Mon, 18 Mar 2019 13:29:00 +0100 Subject: [PATCH 0997/3965] relay: simplify declaration of multiaddr var --- p2p/host/relay/autorelay.go | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index d91f31eee1..b0eb1ed6ac 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -30,17 +30,9 @@ var ( BootDelay = 60 * time.Second - unspecificRelay ma.Multiaddr + unspecificRelay = ma.StringCast("/p2p-circuit") ) -func init() { - var err error - unspecificRelay, err = ma.NewMultiaddr("/p2p-circuit") - if err != nil { - panic(err) - } -} - // AutoRelayHost is a Host that uses relays for connectivity when a NAT is detected. type AutoRelayHost struct { *basic.BasicHost From 767f2b74c4b248a52e5ae2dfd5e6e7a87bc5c763 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 21 Mar 2019 18:05:11 -0700 Subject: [PATCH 0998/3965] remove dependency on go-multicodec It's unmaintained and pulls in github.com/whyrusleeping/cbor. This patch explicitly re-implements the minimally needed features. Also fixes #12. --- p2p/net/pnet/codec.go | 58 ++++++++++++++++++++++++++++---------- p2p/net/pnet/codec_test.go | 4 +++ 2 files changed, 47 insertions(+), 15 deletions(-) diff --git a/p2p/net/pnet/codec.go b/p2p/net/pnet/codec.go index ac6982834f..9dde327b15 100644 --- a/p2p/net/pnet/codec.go +++ b/p2p/net/pnet/codec.go @@ -1,35 +1,63 @@ package pnet import ( + "bufio" + "bytes" + "encoding/base64" + "encoding/hex" "fmt" "io" - - mc "github.com/multiformats/go-multicodec" - bmux "github.com/multiformats/go-multicodec/base/mux" ) var ( - pathPSKv1 = []byte("/key/swarm/psk/1.0.0/") - headerPSKv1 = mc.Header(pathPSKv1) + pathPSKv1 = []byte("/key/swarm/psk/1.0.0/") + pathBase16 = "/base16/" + pathBase64 = "/base64/" ) -func decodeV1PSK(in io.Reader) (*[32]byte, error) { - var err error - in, err = mc.WrapTransformPathToHeader(in) +func readHeader(r *bufio.Reader) ([]byte, error) { + header, err := r.ReadBytes('\n') if err != nil { return nil, err } - err = mc.ConsumeHeader(in, headerPSKv1) + + return bytes.TrimRight(header, "\r\n"), nil +} + +func expectHeader(r *bufio.Reader, expected []byte) error { + header, err := readHeader(r) if err != nil { - return nil, fmt.Errorf("psk header error: %s", err.Error()) + return err + } + if !bytes.Equal(header, expected) { + return fmt.Errorf("expected file header %s, got: %s", pathPSKv1, header) } + return nil +} - in, err = mc.WrapTransformPathToHeader(in) +func decodeV1PSK(in io.Reader) (*[32]byte, error) { + reader := bufio.NewReader(in) + if err := expectHeader(reader, pathPSKv1); err != nil { + return nil, err + } + header, err := readHeader(reader) if err != nil { - return nil, fmt.Errorf("wrapping error: %s", err.Error()) + return nil, err } - out := [32]byte{} - err = bmux.AllBasesMux().Decoder(in).Decode(out[:]) - return &out, err + var decoder io.Reader + switch string(header) { + case pathBase16: + decoder = hex.NewDecoder(reader) + case pathBase64: + decoder = base64.NewDecoder(base64.StdEncoding, reader) + default: + return nil, fmt.Errorf("unknown encoding: %s", header) + } + out := new([32]byte) + _, err = io.ReadFull(decoder, out[:]) + if err != nil { + return nil, err + } + return out, nil } diff --git a/p2p/net/pnet/codec_test.go b/p2p/net/pnet/codec_test.go index 04afe65f08..8167b34d01 100644 --- a/p2p/net/pnet/codec_test.go +++ b/p2p/net/pnet/codec_test.go @@ -46,6 +46,10 @@ func TestDecodeB64(t *testing.T) { if err != nil { t.Fatal(err) } + err = e.Close() + if err != nil { + t.Fatal(err) + } psk, err := decodeV1PSK(b) if err != nil { From e90259bca678a407ce889627cdcedc72a14f1fa2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 21 Mar 2019 18:10:44 -0700 Subject: [PATCH 0999/3965] test decoding on windows --- p2p/net/pnet/codec_test.go | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/p2p/net/pnet/codec_test.go b/p2p/net/pnet/codec_test.go index 8167b34d01..e6d8409fb5 100644 --- a/p2p/net/pnet/codec_test.go +++ b/p2p/net/pnet/codec_test.go @@ -6,18 +6,29 @@ import ( "testing" ) -func bufWithBase(base string) *bytes.Buffer { +func bufWithBase(base string, windows bool) *bytes.Buffer { b := &bytes.Buffer{} b.Write(pathPSKv1) + if windows { + b.WriteString("\r") + } b.WriteString("\n") b.WriteString(base) + if windows { + b.WriteString("\r") + } b.WriteString("\n") return b } func TestDecodeHex(t *testing.T) { - b := bufWithBase("/base16/") + testDecodeHex(t, true) + testDecodeHex(t, false) +} + +func testDecodeHex(t *testing.T, windows bool) { + b := bufWithBase("/base16/", windows) for i := 0; i < 32; i++ { b.WriteString("FF") } @@ -35,7 +46,12 @@ func TestDecodeHex(t *testing.T) { } func TestDecodeB64(t *testing.T) { - b := bufWithBase("/base64/") + testDecodeB64(t, true) + testDecodeB64(t, false) +} + +func testDecodeB64(t *testing.T, windows bool) { + b := bufWithBase("/base64/", windows) key := make([]byte, 32) for i := 0; i < 32; i++ { key[i] = byte(i) From 8c9004d36b9b3802faa20a7994f114b511e07491 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Fri, 22 Mar 2019 02:44:36 +0100 Subject: [PATCH 1000/3965] Add bad encoding test case License: MIT Signed-off-by: Jakub Sztandera --- p2p/net/pnet/codec_test.go | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/p2p/net/pnet/codec_test.go b/p2p/net/pnet/codec_test.go index e6d8409fb5..e59d9b2bd4 100644 --- a/p2p/net/pnet/codec_test.go +++ b/p2p/net/pnet/codec_test.go @@ -27,6 +27,20 @@ func TestDecodeHex(t *testing.T) { testDecodeHex(t, false) } +func TestDecodeBad(t *testing.T) { + testDecodeBad(t, true) + testDecodeBad(t, false) +} +func testDecodeBad(t *testing.T, windows bool) { + b := bufWithBase("/verybadbase/", windows) + b.WriteString("Have fun decoding that key") + + _, err := decodeV1PSK(b) + if err == nil { + t.Fatal("expected 'unknown encoding' got nil") + } +} + func testDecodeHex(t *testing.T, windows bool) { b := bufWithBase("/base16/", windows) for i := 0; i < 32; i++ { From 50f09fade6a7313034d73c720ffe06df57e2d0b7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 21 Mar 2019 18:50:51 -0700 Subject: [PATCH 1001/3965] add back support for binary encoding --- p2p/net/pnet/codec.go | 3 +++ p2p/net/pnet/codec_test.go | 28 ++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/p2p/net/pnet/codec.go b/p2p/net/pnet/codec.go index 9dde327b15..19b929f636 100644 --- a/p2p/net/pnet/codec.go +++ b/p2p/net/pnet/codec.go @@ -11,6 +11,7 @@ import ( var ( pathPSKv1 = []byte("/key/swarm/psk/1.0.0/") + pathBin = "/bin/" pathBase16 = "/base16/" pathBase64 = "/base64/" ) @@ -51,6 +52,8 @@ func decodeV1PSK(in io.Reader) (*[32]byte, error) { decoder = hex.NewDecoder(reader) case pathBase64: decoder = base64.NewDecoder(base64.StdEncoding, reader) + case pathBin: + decoder = reader default: return nil, fmt.Errorf("unknown encoding: %s", header) } diff --git a/p2p/net/pnet/codec_test.go b/p2p/net/pnet/codec_test.go index e59d9b2bd4..233f3419c6 100644 --- a/p2p/net/pnet/codec_test.go +++ b/p2p/net/pnet/codec_test.go @@ -31,6 +31,7 @@ func TestDecodeBad(t *testing.T) { testDecodeBad(t, true) testDecodeBad(t, false) } + func testDecodeBad(t *testing.T, windows bool) { b := bufWithBase("/verybadbase/", windows) b.WriteString("Have fun decoding that key") @@ -93,3 +94,30 @@ func testDecodeB64(t *testing.T, windows bool) { } } + +func TestDecodeBin(t *testing.T) { + testDecodeBin(t, true) + testDecodeBin(t, false) +} + +func testDecodeBin(t *testing.T, windows bool) { + b := bufWithBase("/bin/", windows) + key := make([]byte, 32) + for i := 0; i < 32; i++ { + key[i] = byte(i) + } + + b.Write(key) + + psk, err := decodeV1PSK(b) + if err != nil { + t.Fatal(err) + } + + for i, b := range psk { + if b != psk[i] { + t.Fatal("byte was wrong") + } + } + +} From 3f15992cdf63710fe15005bd7bfcce7dcfad0ba6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 27 Mar 2019 23:18:02 +0000 Subject: [PATCH 1002/3965] add peer protection capability to conn mgr. Users can call Protect() to save a peer from pruning, and Unprotect() to remove the protection. Protected peers don't count towards the target quantity to prune during a connection manager cycle. --- p2p/net/connmgr/connmgr.go | 30 ++++++++++++++-- p2p/net/connmgr/connmgr_test.go | 61 +++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 3 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 626fe76175..15adec9de7 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -13,7 +13,7 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -const silencePeriod = 10 * time.Second +var SilencePeriod = 10 * time.Second var log = logging.Logger("connmgr") @@ -32,9 +32,13 @@ type BasicConnMgr struct { gracePeriod time.Duration peers map[peer.ID]*peerInfo + plk sync.RWMutex + protected map[peer.ID]struct{} + // channel-based semaphore that enforces only a single trim is in progress trimRunningCh chan struct{} lastTrim time.Time + silencePeriod time.Duration } var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) @@ -52,9 +56,23 @@ func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { gracePeriod: grace, peers: make(map[peer.ID]*peerInfo), trimRunningCh: make(chan struct{}, 1), + protected: make(map[peer.ID]struct{}, 16), + silencePeriod: SilencePeriod, } } +func (cm *BasicConnMgr) Protect(id peer.ID) { + cm.plk.Lock() + defer cm.plk.Unlock() + cm.protected[id] = struct{}{} +} + +func (cm *BasicConnMgr) Unprotect(id peer.ID) { + cm.plk.Lock() + defer cm.plk.Unlock() + delete(cm.protected, id) +} + // peerInfo stores metadata for a given peer. type peerInfo struct { tags map[string]int // value for each tag @@ -79,7 +97,7 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { return } defer func() { <-cm.trimRunningCh }() - if time.Since(cm.lastTrim) < silencePeriod { + if time.Since(cm.lastTrim) < cm.silencePeriod { // skip this attempt to trim as the last one just took place. return } @@ -110,9 +128,15 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { var infos []*peerInfo - for _, inf := range cm.peers { + cm.plk.RLock() + for id, inf := range cm.peers { + if _, ok := cm.protected[id]; ok { + // skip protected peer; it's not eligible for pruning. + continue + } infos = append(infos, inf) } + cm.plk.RUnlock() // Sort peers according to their value. sort.Slice(infos, func(i, j int) bool { diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index 90285c5955..0e1e323c2d 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -331,3 +331,64 @@ func TestQuickBurstRespectsSilencePeriod(t *testing.T) { t.Fatalf("expected closed connections + open conn count to equal 30, value: %d", total) } } + +func TestPeerProtection(t *testing.T) { + SilencePeriod = 0 + cm := NewConnManager(10, 20, 0) + SilencePeriod = 10 * time.Second + + not := cm.Notifee() + + // produce 20 connections with unique peers. + var conns []inet.Conn + for i := 0; i < 20; i++ { + rc := randConn(t, not.Disconnected) + conns = append(conns, rc) + not.Connected(nil, rc) + cm.TagPeer(rc.RemotePeer(), "test", 20) + } + + // protect the first 5 peers. + var protected []inet.Conn + for _, c := range conns[0:5] { + cm.Protect(c.RemotePeer()) + protected = append(protected, c) + // remove the tag to make them even more eligible for pruning. + cm.UntagPeer(c.RemotePeer(), "test") + } + + // add one more connection, sending the connection manager overboard. + not.Connected(nil, randConn(t, not.Disconnected)) + + // the pruning happens in the background -- this timing condition is not good. + time.Sleep(1 * time.Second) + + for _, c := range protected { + if c.(*tconn).closed { + t.Error("protected connection was closed by connection manager") + } + } + + // unprotect the first peer. + cm.Unprotect(protected[0].RemotePeer()) + + // add 11 more connections, sending the connection manager overboard again. + for i := 0; i < 11; i++ { + rc := randConn(t, not.Disconnected) + conns = append(conns, rc) + not.Connected(nil, rc) + cm.TagPeer(rc.RemotePeer(), "test", 20) + } + + // the pruning happens in the background -- this timing condition is not good. + time.Sleep(1 * time.Second) + + if !protected[0].(*tconn).closed { + t.Error("unprotected connection was kept open by connection manager") + } + for _, c := range protected[1:] { + if c.(*tconn).closed { + t.Error("protected connection was closed by connection manager") + } + } +} From 16a60f9751790b2723a61efb1fceb73da5176bda Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 28 Mar 2019 02:20:06 +0000 Subject: [PATCH 1003/3965] reduce nat error level --- p2p/host/basic/natmgr.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/basic/natmgr.go b/p2p/host/basic/natmgr.go index 6c0c663ba3..25c8b029d9 100644 --- a/p2p/host/basic/natmgr.go +++ b/p2p/host/basic/natmgr.go @@ -93,7 +93,7 @@ func (nmgr *natManager) discoverNAT() { natInstance, err := inat.DiscoverNAT(goprocessctx.OnClosingContext(worker)) if err != nil { - log.Error("DiscoverNAT error:", err) + log.Info("DiscoverNAT error:", err) close(nmgr.ready) return } From c19c1f34cc1d7511cd6057eb201c654968a39005 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 29 Mar 2019 13:57:36 +0000 Subject: [PATCH 1004/3965] scope protections by tag to avoid cross-interference. Also slightly reworked the loop that selects candidates to prune to make it better behaved. Before, if we needed to prune N connections, we preselected N and ended up doing nothing if they happened to be in the grace period. Now we skip over them and prune ungraced connections until we meet our target. --- p2p/net/connmgr/connmgr.go | 57 ++++++++----- p2p/net/connmgr/connmgr_test.go | 145 ++++++++++++++++++++++++++++++-- 2 files changed, 174 insertions(+), 28 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 15adec9de7..2629dffdf0 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -33,7 +33,7 @@ type BasicConnMgr struct { peers map[peer.ID]*peerInfo plk sync.RWMutex - protected map[peer.ID]struct{} + protected map[peer.ID]map[string]struct{} // channel-based semaphore that enforces only a single trim is in progress trimRunningCh chan struct{} @@ -56,25 +56,41 @@ func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { gracePeriod: grace, peers: make(map[peer.ID]*peerInfo), trimRunningCh: make(chan struct{}, 1), - protected: make(map[peer.ID]struct{}, 16), + protected: make(map[peer.ID]map[string]struct{}, 16), silencePeriod: SilencePeriod, } } -func (cm *BasicConnMgr) Protect(id peer.ID) { +func (cm *BasicConnMgr) Protect(id peer.ID, tag string) { cm.plk.Lock() defer cm.plk.Unlock() - cm.protected[id] = struct{}{} + + tags, ok := cm.protected[id] + if !ok { + tags = make(map[string]struct{}, 2) + cm.protected[id] = tags + } + tags[tag] = struct{}{} } -func (cm *BasicConnMgr) Unprotect(id peer.ID) { +func (cm *BasicConnMgr) Unprotect(id peer.ID, tag string) (protected bool) { cm.plk.Lock() defer cm.plk.Unlock() - delete(cm.protected, id) + + tags, ok := cm.protected[id] + if !ok { + return false + } + if delete(tags, tag); len(tags) == 0 { + delete(cm.protected, id) + return false + } + return true } // peerInfo stores metadata for a given peer. type peerInfo struct { + id peer.ID tags map[string]int // value for each tag value int // cached sum of all tag values @@ -126,45 +142,46 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { return nil } - var infos []*peerInfo - + var candidates []*peerInfo cm.plk.RLock() for id, inf := range cm.peers { if _, ok := cm.protected[id]; ok { - // skip protected peer; it's not eligible for pruning. + // skip over protected peer. continue } - infos = append(infos, inf) + candidates = append(candidates, inf) } cm.plk.RUnlock() // Sort peers according to their value. - sort.Slice(infos, func(i, j int) bool { - return infos[i].value < infos[j].value + sort.Slice(candidates, func(i, j int) bool { + return candidates[i].value < candidates[j].value }) - closeCount := len(infos) - cm.lowWater - toclose := infos[:closeCount] + target := len(cm.peers) - cm.lowWater // 2x number of peers we're disconnecting from because we may have more // than one connection per peer. Slightly over allocating isn't an issue // as this is a very short-lived array. - closed := make([]inet.Conn, 0, len(toclose)*2) + selected := make([]inet.Conn, 0, target*2) - for _, inf := range toclose { + for _, inf := range candidates { // TODO: should we be using firstSeen or the time associated with the connection itself? if inf.firstSeen.Add(cm.gracePeriod).After(now) { continue } - // TODO: if a peer has more than one connection, maybe only close one? for c := range inf.conns { - // TODO: probably don't want to always do this in a goroutine - closed = append(closed, c) + selected = append(selected, c) + } + + target-- + if target == 0 { + break } } - return closed + return selected } // GetTagInfo is called to fetch the tag information associated with a given diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index 0e1e323c2d..a5057612ae 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -317,6 +317,9 @@ func TestQuickBurstRespectsSilencePeriod(t *testing.T) { // wait for a few seconds time.Sleep(time.Second * 3) + cm.lk.Lock() // pacify the race detector + defer cm.lk.Unlock() + // only the first trim is allowed in; make sure we close at most 20 connections, not all of them. var closed int for _, c := range conns { @@ -332,9 +335,9 @@ func TestQuickBurstRespectsSilencePeriod(t *testing.T) { } } -func TestPeerProtection(t *testing.T) { +func TestPeerProtectionSingleTag(t *testing.T) { SilencePeriod = 0 - cm := NewConnManager(10, 20, 0) + cm := NewConnManager(19, 20, 0) SilencePeriod = 10 * time.Second not := cm.Notifee() @@ -351,10 +354,10 @@ func TestPeerProtection(t *testing.T) { // protect the first 5 peers. var protected []inet.Conn for _, c := range conns[0:5] { - cm.Protect(c.RemotePeer()) + cm.Protect(c.RemotePeer(), "global") protected = append(protected, c) - // remove the tag to make them even more eligible for pruning. - cm.UntagPeer(c.RemotePeer(), "test") + // tag them negatively to make them preferred for pruning. + cm.TagPeer(c.RemotePeer(), "test", -100) } // add one more connection, sending the connection manager overboard. @@ -370,10 +373,79 @@ func TestPeerProtection(t *testing.T) { } // unprotect the first peer. - cm.Unprotect(protected[0].RemotePeer()) + cm.Unprotect(protected[0].RemotePeer(), "global") + + // add 2 more connections, sending the connection manager overboard again. + for i := 0; i < 2; i++ { + rc := randConn(t, not.Disconnected) + conns = append(conns, rc) + not.Connected(nil, rc) + cm.TagPeer(rc.RemotePeer(), "test", 20) + } + + // the pruning happens in the background -- this timing condition is not good. + time.Sleep(1 * time.Second) + + cm.lk.Lock() // pacify the race detector + defer cm.lk.Unlock() + + if !protected[0].(*tconn).closed { + t.Error("unprotected connection was kept open by connection manager") + } + for _, c := range protected[1:] { + if c.(*tconn).closed { + t.Error("protected connection was closed by connection manager") + } + } +} + +func TestPeerProtectionMultipleTags(t *testing.T) { + SilencePeriod = 0 + cm := NewConnManager(19, 20, 0) + SilencePeriod = 10 * time.Second + + not := cm.Notifee() + + // produce 20 connections with unique peers. + var conns []inet.Conn + for i := 0; i < 20; i++ { + rc := randConn(t, not.Disconnected) + conns = append(conns, rc) + not.Connected(nil, rc) + cm.TagPeer(rc.RemotePeer(), "test", 20) + } + + // protect the first 5 peers under two tags. + var protected []inet.Conn + for _, c := range conns[0:5] { + cm.Protect(c.RemotePeer(), "tag1") + cm.Protect(c.RemotePeer(), "tag2") + protected = append(protected, c) + // tag them negatively to make them preferred for pruning. + cm.TagPeer(c.RemotePeer(), "test", -100) + } + + // add one more connection, sending the connection manager overboard. + not.Connected(nil, randConn(t, not.Disconnected)) + + // the pruning happens in the background -- this timing condition is not good. + time.Sleep(1 * time.Second) + + for _, c := range protected { + if c.(*tconn).closed { + t.Error("protected connection was closed by connection manager") + } + } + + // remove the protection from one tag. + for _, c := range protected { + if !cm.Unprotect(c.RemotePeer(), "tag1") { + t.Error("peer should still be protected") + } + } - // add 11 more connections, sending the connection manager overboard again. - for i := 0; i < 11; i++ { + // add 2 more connections, sending the connection manager overboard again. + for i := 0; i < 2; i++ { rc := randConn(t, not.Disconnected) conns = append(conns, rc) not.Connected(nil, rc) @@ -383,6 +455,30 @@ func TestPeerProtection(t *testing.T) { // the pruning happens in the background -- this timing condition is not good. time.Sleep(1 * time.Second) + // connections should still remain open, as they were protected. + for _, c := range protected[0:] { + if c.(*tconn).closed { + t.Error("protected connection was closed by connection manager") + } + } + + // unprotect the first peer entirely. + cm.Unprotect(protected[0].RemotePeer(), "tag2") + + // add 2 more connections, sending the connection manager overboard again. + for i := 0; i < 2; i++ { + rc := randConn(t, not.Disconnected) + conns = append(conns, rc) + not.Connected(nil, rc) + cm.TagPeer(rc.RemotePeer(), "test", 20) + } + + // the pruning happens in the background -- this timing condition is not good. + time.Sleep(1 * time.Second) + + cm.lk.Lock() // pacify the race detector + defer cm.lk.Unlock() + if !protected[0].(*tconn).closed { t.Error("unprotected connection was kept open by connection manager") } @@ -391,4 +487,37 @@ func TestPeerProtection(t *testing.T) { t.Error("protected connection was closed by connection manager") } } + +} + +func TestPeerProtectionIdempotent(t *testing.T) { + SilencePeriod = 0 + cm := NewConnManager(10, 20, 0) + SilencePeriod = 10 * time.Second + + id, _ := tu.RandPeerID() + cm.Protect(id, "global") + cm.Protect(id, "global") + cm.Protect(id, "global") + cm.Protect(id, "global") + + if len(cm.protected[id]) > 1 { + t.Error("expected peer to be protected only once") + } + + if !cm.Unprotect(id, "unused") { + t.Error("expected peer to continue to be protected") + } + + if !cm.Unprotect(id, "unused2") { + t.Error("expected peer to continue to be protected") + } + + if cm.Unprotect(id, "global") { + t.Error("expected peer to be unprotected") + } + + if len(cm.protected) > 0 { + t.Error("expected no protections") + } } From 58baf56c8ef6144ac1b22c6e14cc0b23833375ec Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 31 Mar 2019 12:13:29 +0300 Subject: [PATCH 1005/3965] reduce dialback timeout to 15s --- p2p/host/autonat/svc.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index b874b7d1a6..bd704beb44 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -21,7 +21,7 @@ import ( const P_CIRCUIT = 290 var ( - AutoNATServiceDialTimeout = 42 * time.Second + AutoNATServiceDialTimeout = 15 * time.Second AutoNATServiceResetInterval = 1 * time.Minute AutoNATServiceThrottle = 3 From a79b935f6f43f4a206346772c5c2416c6954f1da Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 1 Apr 2019 12:34:01 +0300 Subject: [PATCH 1006/3965] parallelize dialbacks --- p2p/host/autonat/autonat.go | 94 +++++++++++++++++++++++-------------- 1 file changed, 59 insertions(+), 35 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 410d733edd..7897ff7741 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -32,7 +32,7 @@ var ( AutoNATBootDelay = 15 * time.Second AutoNATRetryInterval = 90 * time.Second AutoNATRefreshInterval = 15 * time.Minute - AutoNATRequestTimeout = 60 * time.Second + AutoNATRequestTimeout = 30 * time.Second ) // AutoNAT is the interface for ambient NAT autodiscovery @@ -135,50 +135,74 @@ func (as *AmbientAutoNAT) autodetect() { } cli := NewAutoNATClient(as.host, as.getAddrs) - failures := 0 - - for _, pi := range peers { - ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) - as.host.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) - a, err := cli.DialBack(ctx, pi.ID) - cancel() - - switch { - case err == nil: - log.Debugf("NAT status is public; address through %s: %s", pi.ID.Pretty(), a.String()) - as.mx.Lock() - as.addr = a - as.status = NATStatusPublic - as.confidence = 0 - as.mx.Unlock() - return + ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) + defer cancel() + + var mx sync.Mutex + var pubaddr ma.Multiaddr + private := 0 + public := 0 - case IsDialError(err): - log.Debugf("dial error through %s: %s", pi.ID.Pretty(), err.Error()) - failures++ - if failures >= 3 || as.confidence >= 3 { // 3 times is enemy action - log.Debugf("NAT status is private") - as.mx.Lock() - as.status = NATStatusPrivate - as.confidence = 3 - as.mx.Unlock() - return + probe := 3 - as.confidence + if probe == 0 { + probe = 1 + } + if probe > len(peers) { + probe = len(peers) + } + + var wg sync.WaitGroup + + for _, pi := range peers[:probe] { + wg.Add(1) + go func(pi pstore.PeerInfo) { + as.host.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) + a, err := cli.DialBack(ctx, pi.ID) + + switch { + case err == nil: + log.Debugf("Dialback through %s successful; public address is %s", pi.ID.Pretty(), a.String()) + mx.Lock() + public++ + pubaddr = a + mx.Unlock() + + case IsDialError(err): + log.Debugf("Dialback through %s failed", pi.ID.Pretty()) + mx.Lock() + private++ + mx.Unlock() + + default: + log.Debugf("Dialback error through %s: %s", pi.ID.Pretty(), err) } - default: - log.Debugf("Error dialing through %s: %s", pi.ID.Pretty(), err.Error()) - } + wg.Done() + }(pi) } + wg.Wait() + as.mx.Lock() - if failures > 0 { - as.status = NATStatusPrivate - as.confidence++ + if public > 0 { + log.Debugf("NAT status is public") + as.status = NATStatusPublic + as.addr = pubaddr + if as.confidence < 3 { + as.confidence++ + } + } else if private > 0 { log.Debugf("NAT status is private") + as.status = NATStatusPrivate + as.addr = nil + if as.confidence < 3 { + as.confidence++ + } } else { + log.Debugf("NAT status is unknown") as.status = NATStatusUnknown + as.addr = nil as.confidence = 0 - log.Debugf("NAT status is unknown") } as.mx.Unlock() } From 3c9b5301c25cbf1c335cb6f8dc76b551385949df Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 1 Apr 2019 12:39:39 +0300 Subject: [PATCH 1007/3965] reset confidence when there is a public<->private transition --- p2p/host/autonat/autonat.go | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 7897ff7741..2bc5fd06e5 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -186,18 +186,22 @@ func (as *AmbientAutoNAT) autodetect() { as.mx.Lock() if public > 0 { log.Debugf("NAT status is public") - as.status = NATStatusPublic - as.addr = pubaddr - if as.confidence < 3 { + if as.status == NATStatusPrivate { + as.confidence = 0 + } else if as.confidence < 3 { as.confidence++ } + as.status = NATStatusPublic + as.addr = pubaddr } else if private > 0 { log.Debugf("NAT status is private") - as.status = NATStatusPrivate - as.addr = nil - if as.confidence < 3 { + if as.status == NATStatusPublic { + as.confidence = 0 + } else if as.confidence < 3 { as.confidence++ } + as.status = NATStatusPrivate + as.addr = nil } else { log.Debugf("NAT status is unknown") as.status = NATStatusUnknown From 7c7e813051e6b47f877038efa4944dc1db0e2e4a Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 2 Apr 2019 20:15:25 +0300 Subject: [PATCH 1008/3965] use anonymous struct for collecting dialback results --- p2p/host/autonat/autonat.go | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 2bc5fd06e5..46d949dfe1 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -138,10 +138,12 @@ func (as *AmbientAutoNAT) autodetect() { ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) defer cancel() - var mx sync.Mutex - var pubaddr ma.Multiaddr - private := 0 - public := 0 + var result struct { + sync.Mutex + private int + public int + pubaddr ma.Multiaddr + } probe := 3 - as.confidence if probe == 0 { @@ -162,16 +164,16 @@ func (as *AmbientAutoNAT) autodetect() { switch { case err == nil: log.Debugf("Dialback through %s successful; public address is %s", pi.ID.Pretty(), a.String()) - mx.Lock() - public++ - pubaddr = a - mx.Unlock() + result.Lock() + result.public++ + result.pubaddr = a + result.Unlock() case IsDialError(err): log.Debugf("Dialback through %s failed", pi.ID.Pretty()) - mx.Lock() - private++ - mx.Unlock() + result.Lock() + result.private++ + result.Unlock() default: log.Debugf("Dialback error through %s: %s", pi.ID.Pretty(), err) @@ -184,7 +186,7 @@ func (as *AmbientAutoNAT) autodetect() { wg.Wait() as.mx.Lock() - if public > 0 { + if result.public > 0 { log.Debugf("NAT status is public") if as.status == NATStatusPrivate { as.confidence = 0 @@ -192,8 +194,8 @@ func (as *AmbientAutoNAT) autodetect() { as.confidence++ } as.status = NATStatusPublic - as.addr = pubaddr - } else if private > 0 { + as.addr = result.pubaddr + } else if result.private > 0 { log.Debugf("NAT status is private") if as.status == NATStatusPublic { as.confidence = 0 From 946f564fee058404373f8b11935866e362df0de8 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 2 Apr 2019 20:17:18 +0300 Subject: [PATCH 1009/3965] add comments for status flips dropping confidence to 0 --- p2p/host/autonat/autonat.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 46d949dfe1..fcb110aee0 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -189,6 +189,7 @@ func (as *AmbientAutoNAT) autodetect() { if result.public > 0 { log.Debugf("NAT status is public") if as.status == NATStatusPrivate { + // we are flipping our NATStatus, so confidence drops to 0 as.confidence = 0 } else if as.confidence < 3 { as.confidence++ @@ -198,6 +199,7 @@ func (as *AmbientAutoNAT) autodetect() { } else if result.private > 0 { log.Debugf("NAT status is private") if as.status == NATStatusPublic { + // we are flipping our NATStatus, so confidence drops to 0 as.confidence = 0 } else if as.confidence < 3 { as.confidence++ From 0bdfbe083e1d4afabd05d131ee28453a887e295e Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 2 Apr 2019 20:20:10 +0300 Subject: [PATCH 1010/3965] don't flip to status unknown on first faiulure, use the confidence level --- p2p/host/autonat/autonat.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index fcb110aee0..10fbc8f1d9 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -206,11 +206,13 @@ func (as *AmbientAutoNAT) autodetect() { } as.status = NATStatusPrivate as.addr = nil + } else if as.confidence > 0 { + // don't just flip to unknown, reduce confidence first + as.confidence-- } else { log.Debugf("NAT status is unknown") as.status = NATStatusUnknown as.addr = nil - as.confidence = 0 } as.mx.Unlock() } From e2412502a64adf39a2f871d08c8512aba316d0a7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Apr 2019 11:44:16 +0300 Subject: [PATCH 1011/3965] reduce autorelay boot delay to 30s --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index b0eb1ed6ac..a6adf48d8d 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -28,7 +28,7 @@ const ( var ( DesiredRelays = 3 - BootDelay = 60 * time.Second + BootDelay = 30 * time.Second unspecificRelay = ma.StringCast("/p2p-circuit") ) From c3b5ece119cddf71765782da47d7478cd4afecac Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Apr 2019 11:51:26 +0300 Subject: [PATCH 1012/3965] reduce autorelay discovery timeout to 30s --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index a6adf48d8d..3bc911d7d0 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -122,7 +122,7 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { limit = 2 * need } - dctx, cancel := context.WithTimeout(ctx, 60*time.Second) + dctx, cancel := context.WithTimeout(ctx, 30*time.Second) pis, err := discovery.FindPeers(dctx, h.discover, RelayRendezvous, limit) cancel() if err != nil { From bace7d4cebe13c0c6de9401a563470a7d8109af3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Apr 2019 12:02:49 +0300 Subject: [PATCH 1013/3965] autorelay: correctly handle private->public transition --- p2p/host/relay/autorelay.go | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 3bc911d7d0..ca8296987b 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -85,13 +85,32 @@ func (h *AutoRelayHost) background(ctx context.Context) { return } + // when true, we need to identify push + push := false + for { wait := autonat.AutoNATRefreshInterval switch h.autonat.Status() { case autonat.NATStatusUnknown: wait = autonat.AutoNATRetryInterval + case autonat.NATStatusPublic: + // invalidate addrs + h.mx.Lock() + if h.addrs != nil { + h.addrs = nil + push = true + } + h.mx.Unlock() + + // if we had previously announced relay addrs, push our public addrs + if push { + push = false + h.PushIdentify() + } + case autonat.NATStatusPrivate: + push = false // clear, findRelays pushes as needed h.findRelays(ctx) } @@ -99,7 +118,10 @@ func (h *AutoRelayHost) background(ctx context.Context) { case <-h.disconnect: // invalidate addrs h.mx.Lock() - h.addrs = nil + if h.addrs != nil { + h.addrs = nil + push = true + } h.mx.Unlock() case <-time.After(wait): case <-ctx.Done(): From 62531f9a105309bf2d4ad49657c3b8dac47549a4 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Apr 2019 17:27:03 +0300 Subject: [PATCH 1014/3965] reduce autorelay boot delay to 20s --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index ca8296987b..d5f7daba02 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -28,7 +28,7 @@ const ( var ( DesiredRelays = 3 - BootDelay = 30 * time.Second + BootDelay = 20 * time.Second unspecificRelay = ma.StringCast("/p2p-circuit") ) From 7970ca46d86696e5f540e3bdfba62a079c2c8d8b Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 3 Apr 2019 22:11:03 +0300 Subject: [PATCH 1015/3965] defer wg.Done --- p2p/host/autonat/autonat.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 10fbc8f1d9..60188c94da 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -158,6 +158,8 @@ func (as *AmbientAutoNAT) autodetect() { for _, pi := range peers[:probe] { wg.Add(1) go func(pi pstore.PeerInfo) { + defer wg.Done() + as.host.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) a, err := cli.DialBack(ctx, pi.ID) @@ -178,8 +180,6 @@ func (as *AmbientAutoNAT) autodetect() { default: log.Debugf("Dialback error through %s: %s", pi.ID.Pretty(), err) } - - wg.Done() }(pi) } From 5b0e2d3373e9428b4c0fa5f79cde078e209e7090 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 3 Apr 2019 16:34:41 -0400 Subject: [PATCH 1016/3965] link to docs & discuss in readme --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index cb198624f5..12361a0f43 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,8 @@ libp2p is the product of a long, and arduous quest of understanding -- a deep di > > We will be writing a set of docs, posts, tutorials, and talks to explain what p2p is, why it is tremendously useful, and how it can help your existing and new projects. But in the meantime, check out > +> - [**Our developing collection of docs**](https://docs.libp2p.io) +> - [**Our community discussion forums**](https://discuss.libp2p.io) > - [**The libp2p Specification**](https://github.com/libp2p/specs) > - [**go-libp2p implementation**](https://github.com/libp2p/go-libp2p) > - [**js-libp2p implementation**](https://github.com/libp2p/js-libp2p) @@ -191,7 +193,7 @@ Guidelines: - read the [libp2p spec](https://github.com/libp2p/specs) - please make branches + pull-request, even if working on the main repository -- ask questions or talk about things in [Issues](https://github.com/libp2p/go-libp2p/issues) or #ipfs on freenode. +- ask questions or talk about things in [Issues](https://github.com/libp2p/go-libp2p/issues), our [discussion forums](https://discuss.libp2p.io), or #libp2p or #ipfs on freenode. - ensure you are able to contribute (no legal issues please-- we use the DCO) - run `go fmt` before pushing any code - run `golint` and `go vet` too -- some things (like protobuf files) are expected to fail. From 5abb5b9ff06139d46e1ed28024b32cb74203f079 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 3 Apr 2019 19:26:55 -0700 Subject: [PATCH 1017/3965] don't set linger to 0 We should do this from go-tcp-transport (https://github.com/libp2p/go-tcp-transport/pull/36), not here, as setting linger to 0 will cause us to send RST packets instead of nicely closing connections. Also, document why we need the fallback dialer. --- p2p/net/reuseport/reuseport.go | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/p2p/net/reuseport/reuseport.go b/p2p/net/reuseport/reuseport.go index dd271d11a0..82f9f37c84 100644 --- a/p2p/net/reuseport/reuseport.go +++ b/p2p/net/reuseport/reuseport.go @@ -39,7 +39,6 @@ func reuseErrShouldRetry(err error) bool { } // Dials using reuseport and then redials normally if that fails. -// TODO(anacrolix): This shouldn't fail anymore: Remove fallback. func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) (con net.Conn, err error) { if laddr == nil { return fallbackDialer.DialContext(ctx, network, raddr) @@ -49,15 +48,6 @@ func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) ( LocalAddr: laddr, Control: reuseport.Control, } - defer func() { - if err != nil { - return - } - // This is transplanted from go-reuseport, which once set no linger on - // dialing and may be a requirement for desired behaviour in this - // package. - con.(*net.TCPConn).SetLinger(0) - }() con, err = d.DialContext(ctx, network, raddr) if err == nil { @@ -65,6 +55,8 @@ func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) ( } if reuseErrShouldRetry(err) && ctx.Err() == nil { + // We could have an existing socket open or we could have one + // stuck in TIME-WAIT. log.Debugf("failed to reuse port, dialing with a random port: %s", err) con, err = fallbackDialer.DialContext(ctx, network, raddr) } From 16e9eb2bb0f2916c9fa5667c663d955ef4a8886d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 3 Apr 2019 23:17:58 -0700 Subject: [PATCH 1018/3965] modernize request handling I believe this was written before request hijacking was a thing. We no longer need to hold the `ServeHTTP` function open. --- p2p/transport/websocket/conn.go | 18 ++++++-------- p2p/transport/websocket/listener.go | 30 +++-------------------- p2p/transport/websocket/websocket.go | 2 +- p2p/transport/websocket/websocket_test.go | 18 +++++++++++--- 4 files changed, 26 insertions(+), 42 deletions(-) diff --git a/p2p/transport/websocket/conn.go b/p2p/transport/websocket/conn.go index 168497bf39..b63a28a9ed 100644 --- a/p2p/transport/websocket/conn.go +++ b/p2p/transport/websocket/conn.go @@ -19,7 +19,6 @@ var _ net.Conn = (*Conn)(nil) type Conn struct { *ws.Conn DefaultMessageType int - done func() reader io.Reader closeOnce sync.Once } @@ -85,14 +84,14 @@ func (c *Conn) Write(b []byte) (n int, err error) { func (c *Conn) Close() error { var err error c.closeOnce.Do(func() { - if c.done != nil { - c.done() - // Be nice to GC - c.done = nil + err1 := c.Conn.WriteControl(ws.CloseMessage, nil, time.Now().Add(GracefulCloseTimeout)) + err2 := c.Conn.Close() + switch { + case err1 != nil: + err = err1 + case err2 != nil: + err = err2 } - - c.Conn.WriteControl(ws.CloseMessage, nil, time.Now().Add(GracefulCloseTimeout)) - err = c.Conn.Close() }) return err } @@ -122,10 +121,9 @@ func (c *Conn) SetWriteDeadline(t time.Time) error { } // NewConn creates a Conn given a regular gorilla/websocket Conn. -func NewConn(raw *ws.Conn, done func()) *Conn { +func NewConn(raw *ws.Conn) *Conn { return &Conn{ Conn: raw, DefaultMessageType: ws.BinaryMessage, - done: done, } } diff --git a/p2p/transport/websocket/listener.go b/p2p/transport/websocket/listener.go index 3d774f09dc..437be9893f 100644 --- a/p2p/transport/websocket/listener.go +++ b/p2p/transport/websocket/listener.go @@ -1,7 +1,6 @@ package websocket import ( - "context" "fmt" "net" "net/http" @@ -21,7 +20,7 @@ type listener struct { func (l *listener) serve() { defer close(l.closed) - http.Serve(l.Listener, l) + _ = http.Serve(l.Listener, l) } func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { @@ -31,35 +30,12 @@ func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { return } - ctx, cancel := context.WithCancel(context.Background()) - - var cnCh <-chan bool - if cn, ok := w.(http.CloseNotifier); ok { - cnCh = cn.CloseNotify() - } - - wscon := NewConn(c, cancel) - // Just to make sure. - defer wscon.Close() - select { - case l.incoming <- wscon: + case l.incoming <- NewConn(c): case <-l.closed: c.Close() - return - case <-cnCh: - return - } - - // wait until conn gets closed, otherwise the handler closes it early - select { - case <-ctx.Done(): - case <-l.closed: - c.Close() - return - case <-cnCh: - return } + // The connection has been hijacked, it's safe to return. } func (l *listener) Accept() (manet.Conn, error) { diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index b9ec1d6a9a..681d67640c 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -86,7 +86,7 @@ func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (ma return nil, err } - mnc, err := manet.WrapNetConn(NewConn(wscon, nil)) + mnc, err := manet.WrapNetConn(NewConn(wscon)) if err != nil { wscon.Close() return nil, err diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index 2c9ad32441..b9c688dc4b 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -75,8 +75,14 @@ func TestWebsocketListen(t *testing.T) { return } - c.Write(msg) - c.Close() + _, err = c.Write(msg) + if err != nil { + t.Error(err) + } + err = c.Close() + if err != nil { + t.Error(err) + } }() c, err := l.Accept() @@ -120,8 +126,12 @@ func TestConcurrentClose(t *testing.T) { return } - go c.Write(msg) - go c.Close() + go func() { + _, _ = c.Write(msg) + }() + go func() { + _ = c.Close() + }() } }() From 87f331a7fc4ca5aa71f695190f2e187165d00421 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 3 Apr 2019 19:29:38 -0700 Subject: [PATCH 1019/3965] set linger to 0 for both inbound and outbound connections This causes us to send RST packets instead of FIN packets when closing connections and means connections immediately enter the "reset" state instead of entering the TIME-WAIT state. Importantly, this means we can immediately reuse the 5-tuple and reconnect. --- p2p/transport/tcp/tcp.go | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 18e119ec71..5f7fec8a81 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -2,6 +2,7 @@ package tcp import ( "context" + "net" "time" logging "github.com/ipfs/go-log" @@ -20,6 +21,29 @@ var DefaultConnectTimeout = 5 * time.Second var log = logging.Logger("tcp-tpt") +// try to set linger on the connection, if possible. +func tryLinger(conn net.Conn, sec int) { + if lingerConn, ok := conn.(interface { + SetLinger(int) error + }); ok { + _ = lingerConn.SetLinger(sec) + } +} + +type lingerListener struct { + manet.Listener + sec int +} + +func (ll *lingerListener) Accept() (manet.Conn, error) { + c, err := ll.Listener.Accept() + if err != nil { + return nil, err + } + tryLinger(c, ll.sec) + return c, nil +} + // TcpTransport is the TCP transport. type TcpTransport struct { // Connection upgrader for upgrading insecure stream connections to @@ -73,6 +97,10 @@ func (t *TcpTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) if err != nil { return nil, err } + // Set linger to 0 so we never get stuck in the TIME-WAIT state. When + // linger is 0, connections are _reset_ instead of closed with a FIN. + // This means we can immediately reuse the 5-tuple and reconnect. + tryLinger(conn, 0) return t.Upgrader.UpgradeOutbound(ctx, t, conn, p) } @@ -94,6 +122,7 @@ func (t *TcpTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { if err != nil { return nil, err } + list = &lingerListener{list, 0} return t.Upgrader.UpgradeListener(t, list), nil } From b33e73168904b92c8bb946ed7ee8110c73ac31a5 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 4 Apr 2019 10:19:25 +0300 Subject: [PATCH 1020/3965] gomod: update go-libp2p-autonat --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 470c601b12..c6099cdd66 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,7 @@ require ( github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 github.com/libp2p/go-conn-security v0.0.1 github.com/libp2p/go-conn-security-multistream v0.0.1 - github.com/libp2p/go-libp2p-autonat v0.0.2 + github.com/libp2p/go-libp2p-autonat v0.0.3 github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.1 github.com/libp2p/go-libp2p-crypto v0.0.1 diff --git a/go.sum b/go.sum index dcacbd2146..3559a45764 100644 --- a/go.sum +++ b/go.sum @@ -88,6 +88,8 @@ github.com/libp2p/go-libp2p-autonat v0.0.1 h1:d5eskFxeJ4ag1ekhMC3yLTK+z+6RTw9W1Y github.com/libp2p/go-libp2p-autonat v0.0.1/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-autonat v0.0.2 h1:ilo9QPzNPf1hMkqaPG55yzvhILf5ZtijstJhcii+l3s= github.com/libp2p/go-libp2p-autonat v0.0.2/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= +github.com/libp2p/go-libp2p-autonat v0.0.3 h1:PUD+pAx8Qs9hh+Bowzxq8RCkg/Vwrz5oCFC4peixXQk= +github.com/libp2p/go-libp2p-autonat v0.0.3/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= github.com/libp2p/go-libp2p-circuit v0.0.1 h1:DYbjyQ5ZY3QVAVYZWG4uzBQ6Wmcd1C82Bk8Q/pJlM1I= From 2a8398b05f4d0bcd08c6d395087a2376930aec22 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Apr 2019 00:45:26 -0700 Subject: [PATCH 1021/3965] dep: update transports --- go.mod | 4 ++-- go.sum | 14 ++++++-------- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/go.mod b/go.mod index c6099cdd66..2ae577495a 100644 --- a/go.mod +++ b/go.mod @@ -32,9 +32,9 @@ require ( github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 github.com/libp2p/go-maddr-filter v0.0.1 github.com/libp2p/go-stream-muxer v0.0.1 - github.com/libp2p/go-tcp-transport v0.0.1 + github.com/libp2p/go-tcp-transport v0.0.2 github.com/libp2p/go-testutil v0.0.1 - github.com/libp2p/go-ws-transport v0.0.1 + github.com/libp2p/go-ws-transport v0.0.2 github.com/miekg/dns v1.1.4 // indirect github.com/multiformats/go-multiaddr v0.0.1 github.com/multiformats/go-multiaddr-dns v0.0.2 diff --git a/go.sum b/go.sum index 3559a45764..9974f7a950 100644 --- a/go.sum +++ b/go.sum @@ -84,10 +84,6 @@ github.com/libp2p/go-conn-security-multistream v0.0.1 h1:XefjAQRHcnUaxKb26RGupTo github.com/libp2p/go-conn-security-multistream v0.0.1/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= -github.com/libp2p/go-libp2p-autonat v0.0.1 h1:d5eskFxeJ4ag1ekhMC3yLTK+z+6RTw9W1Yv8HQma75k= -github.com/libp2p/go-libp2p-autonat v0.0.1/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= -github.com/libp2p/go-libp2p-autonat v0.0.2 h1:ilo9QPzNPf1hMkqaPG55yzvhILf5ZtijstJhcii+l3s= -github.com/libp2p/go-libp2p-autonat v0.0.2/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-autonat v0.0.3 h1:PUD+pAx8Qs9hh+Bowzxq8RCkg/Vwrz5oCFC4peixXQk= github.com/libp2p/go-libp2p-autonat v0.0.3/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= @@ -108,8 +104,6 @@ github.com/libp2p/go-libp2p-loggables v0.0.1 h1:HVww9oAnINIxbt69LJNkxD8lnbfgteXR github.com/libp2p/go-libp2p-loggables v0.0.1/go.mod h1:lDipDlBNYbpyqyPX/KcoO+eq0sJYEVR2JgOexcivchg= github.com/libp2p/go-libp2p-metrics v0.0.1 h1:yumdPC/P2VzINdmcKZd0pciSUCpou+s0lwYCjBbzQZU= github.com/libp2p/go-libp2p-metrics v0.0.1/go.mod h1:jQJ95SXXA/K1VZi13h52WZMa9ja78zjyy5rspMsC/08= -github.com/libp2p/go-libp2p-nat v0.0.1 h1:on/zju7XE+JXc8gH+vTKmIh2UJFC1K8kGnJYluQrlz4= -github.com/libp2p/go-libp2p-nat v0.0.1/go.mod h1:4L6ajyUIlJvx1Cbh5pc6Ma6vMDpKXf3GgLO5u7W0oQ4= github.com/libp2p/go-libp2p-nat v0.0.2 h1:sKI5hiCsGFhuEKdXMsF9mywQu2qhfoIGX6a+VG6zelE= github.com/libp2p/go-libp2p-nat v0.0.2/go.mod h1:QrjXQSD5Dj4IJOdEcjHRkWTSomyxRo6HnUkf/TfQpLQ= github.com/libp2p/go-libp2p-net v0.0.1 h1:xJ4Vh4yKF/XKb8fd1Ev0ebAGzVjMxXzrxG2kjtU+F5Q= @@ -143,14 +137,18 @@ github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FW github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= github.com/libp2p/go-reuseport-transport v0.0.1 h1:UIRneNxLDmEGNjGHpIiWzSWkZ5bhxMCP9x3Vh7BSc7E= github.com/libp2p/go-reuseport-transport v0.0.1/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= +github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0GIEjMiHTwR/KN4= +github.com/libp2p/go-reuseport-transport v0.0.2/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= github.com/libp2p/go-stream-muxer v0.0.1 h1:Ce6e2Pyu+b5MC1k3eeFtAax0pW4gc6MosYSLV05UeLw= github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= github.com/libp2p/go-tcp-transport v0.0.1 h1:WyvJVw2lYAnr6CU+GZZ4oCt06fvORlmvBlFX2+ZpZDM= github.com/libp2p/go-tcp-transport v0.0.1/go.mod h1:mnjg0o0O5TmXUaUIanYPUqkW4+u6mK0en8rlpA6BBTs= +github.com/libp2p/go-tcp-transport v0.0.2 h1:EzOSRaHpyrGpJ5qe+9SaxJM1mrWlkSLFfNTufUkq0lg= +github.com/libp2p/go-tcp-transport v0.0.2/go.mod h1:VjZFHasNJ0QiJQNNwiFDy25qyGWTXQWs8GM5DR4/L1k= github.com/libp2p/go-testutil v0.0.1 h1:Xg+O0G2HIMfHqBOBDcMS1iSZJ3GEcId4qOxCQvsGZHk= github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= -github.com/libp2p/go-ws-transport v0.0.1 h1:9ytMqq86Xvp8rcnC/1ZNuH612eXLDglvcu4ZHseJl8s= -github.com/libp2p/go-ws-transport v0.0.1/go.mod h1:p3bKjDWHEgtuKKj+2OdPYs5dAPIjtpQGHF2tJfGz7Ww= +github.com/libp2p/go-ws-transport v0.0.2 h1:PtK1AoM16nm96FwPBQoq+4T4t9LdDwOhkB+mdXuGSlg= +github.com/libp2p/go-ws-transport v0.0.2/go.mod h1:p3bKjDWHEgtuKKj+2OdPYs5dAPIjtpQGHF2tJfGz7Ww= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= From 925970409796e5dc468dd20f9dfd55015d1f7d93 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 4 Apr 2019 22:13:44 +0900 Subject: [PATCH 1022/3965] update quic-go to v0.11.0 --- p2p/transport/quic/conn_test.go | 4 ++-- p2p/transport/quic/stream.go | 7 +++---- p2p/transport/quic/transport.go | 1 - 3 files changed, 5 insertions(+), 7 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index d4744882a0..1e0863e77c 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -137,7 +137,7 @@ var _ = Describe("Connection", func() { // dial, but expect the wrong peer ID _, err = clientTransport.Dial(context.Background(), serverAddr, thirdPartyID) Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("TLS handshake error: bad certificate")) + Expect(err.Error()).To(ContainSubstring("CRYPTO_ERROR")) Consistently(serverConnChan).ShouldNot(Receive()) }) @@ -165,7 +165,7 @@ var _ = Describe("Connection", func() { Expect(err).ToNot(HaveOccurred()) _, err = clientTransport.Dial(context.Background(), serverAddr, serverID) Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("TLS handshake error: bad certificate")) + Expect(err.Error()).To(ContainSubstring("CRYPTO_ERROR")) Consistently(serverConnChan).ShouldNot(Receive()) }) diff --git a/p2p/transport/quic/stream.go b/p2p/transport/quic/stream.go index 3fcf0d7d79..108536560e 100644 --- a/p2p/transport/quic/stream.go +++ b/p2p/transport/quic/stream.go @@ -12,8 +12,7 @@ type stream struct { var _ smux.Stream = &stream{} func (s *stream) Reset() error { - if err := s.Stream.CancelRead(0); err != nil { - return err - } - return s.Stream.CancelWrite(0) + s.Stream.CancelRead(0) + s.Stream.CancelWrite(0) + return nil } diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 212ddf4e46..2351959c00 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -19,7 +19,6 @@ import ( ) var quicConfig = &quic.Config{ - Versions: []quic.VersionNumber{quic.VersionMilestone0_10_0}, MaxIncomingStreams: 1000, MaxIncomingUniStreams: -1, // disable unidirectional streams MaxReceiveStreamFlowControlWindow: 3 * (1 << 20), // 3 MB From a517f4d213c8c950b8b1d761b1158aa283306cfc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Apr 2019 12:59:57 -0700 Subject: [PATCH 1023/3965] chore: remove inline interface (@vyzo CR) --- p2p/transport/tcp/tcp.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 5f7fec8a81..84a34a4d4f 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -23,9 +23,11 @@ var log = logging.Logger("tcp-tpt") // try to set linger on the connection, if possible. func tryLinger(conn net.Conn, sec int) { - if lingerConn, ok := conn.(interface { + type canLinger interface { SetLinger(int) error - }); ok { + } + + if lingerConn, ok := conn.(canLinger); ok { _ = lingerConn.SetLinger(sec) } } From 2df36e51fcf6682634f85771c577d313486d09a8 Mon Sep 17 00:00:00 2001 From: tg Date: Tue, 26 Mar 2019 21:39:22 +0300 Subject: [PATCH 1024/3965] return all dial errors if dial has failed License: MIT Signed-off-by: Georgij Tolstov --- p2p/net/swarm/dial_test.go | 48 +++++++++++++++++++++++++++++++++++++ p2p/net/swarm/swarm_dial.go | 37 ++++++++++++++++------------ 2 files changed, 70 insertions(+), 15 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index c9d0d5dda0..adb3438ac4 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -2,7 +2,9 @@ package swarm_test import ( "context" + "fmt" "net" + "regexp" "sync" "testing" "time" @@ -480,3 +482,49 @@ func TestDialBackoffClears(t *testing.T) { t.Log("correctly cleared backoff") } } + +func TestDialPeerFailed(t *testing.T) { + t.Parallel() + ctx := context.Background() + + swarms := makeSwarms(ctx, t, 2) + defer closeSwarms(swarms) + testedSwarm, targetSwarm := swarms[0], swarms[1] + + exceptedErrorsCount := 5 + for i := 0; i < exceptedErrorsCount; i++ { + _, silentPeerAddress, silentPeerListener := newSilentPeer(t) + go acceptAndHang(silentPeerListener) + defer silentPeerListener.Close() + + testedSwarm.Peerstore().AddAddr( + targetSwarm.LocalPeer(), + silentPeerAddress, + pstore.PermanentAddrTTL) + } + + _, err := testedSwarm.DialPeer(ctx, targetSwarm.LocalPeer()) + if err == nil { + t.Fatal(err) + } + + // dial_test.go:508: correctly get a combined error: dial attempt failed: 10 errors occurred: + // * --> (/ip4/127.0.0.1/tcp/46485) dial attempt failed: failed to negotiate security protocol: context deadline exceeded + // * --> (/ip4/127.0.0.1/tcp/34881) dial attempt failed: failed to negotiate security protocol: context deadline exceeded + // ... + + errorCountRegexpString := fmt.Sprintf("%d errors occurred", exceptedErrorsCount) + errorCountRegexp := regexp.MustCompile(errorCountRegexpString) + if !errorCountRegexp.MatchString(err.Error()) { + t.Fatalf("can't find total err count: `%s' in `%s'", errorCountRegexpString, err.Error()) + } + + connectErrorsRegexpString := `\* --> \(.+?\) dial attempt failed:.+` + connectErrorsRegexp := regexp.MustCompile(connectErrorsRegexpString) + connectErrors := connectErrorsRegexp.FindAll([]byte(err.Error()), -1) + if len(connectErrors) != exceptedErrorsCount { + t.Fatalf("connectErrors must contain %d errros; "+ + "but `%s' was found in `%s' %d times", + exceptedErrorsCount, connectErrorsRegexpString, err.Error(), len(connectErrors)) + } +} diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 8a3ba49172..d4b315525f 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -7,6 +7,8 @@ import ( "sync" "time" + "github.com/hashicorp/go-multierror" + logging "github.com/ipfs/go-log" addrutil "github.com/libp2p/go-addr-util" lgbl "github.com/libp2p/go-libp2p-loggables" @@ -358,9 +360,7 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. // use a single response type instead of errs and conns, reduces complexity *a ton* respch := make(chan dialResult) - - defaultDialFail := inet.ErrNoRemoteAddrs - exitErr := defaultDialFail + var dialErrors *multierror.Error defer s.limiter.clearAllPeerDials(p) @@ -369,16 +369,17 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. // Check for context cancellations and/or responses first. select { case <-ctx.Done(): - if exitErr == defaultDialFail { - exitErr = ctx.Err() + if dialError := dialErrors.ErrorOrNil(); dialError != nil { + return nil, dialError } - return nil, exitErr + + return nil, ctx.Err() case resp := <-respch: active-- if resp.Err != nil { - log.Infof("got error on dial to %s: %s", resp.Addr, resp.Err) // Errors are normal, lots of dials will fail - exitErr = resp.Err + log.Infof("got error on dial: %s", resp.Err) + dialErrors = multierror.Append(dialErrors, resp.Err) } else if resp.Conn != nil { return resp.Conn, nil } @@ -399,22 +400,28 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. s.limitedDial(ctx, p, addr, respch) active++ case <-ctx.Done(): - if exitErr == defaultDialFail { - exitErr = ctx.Err() + if dialError := dialErrors.ErrorOrNil(); dialError != nil { + return nil, dialError } - return nil, exitErr + + return nil, ctx.Err() case resp := <-respch: active-- if resp.Err != nil { - log.Infof("got error on dial to %s: %s", resp.Addr, resp.Err) // Errors are normal, lots of dials will fail - exitErr = resp.Err + log.Infof("got error on dial: %s", resp.Err) + dialErrors = multierror.Append(dialErrors, resp.Err) } else if resp.Conn != nil { return resp.Conn, nil } } } - return nil, exitErr + + if dialError := dialErrors.ErrorOrNil(); dialError != nil { + return nil, dialError + } + + return nil, inet.ErrNoRemoteAddrs } // limitedDial will start a dial to the given peer when @@ -443,7 +450,7 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (tra connC, err := tpt.Dial(ctx, addr, p) if err != nil { - return nil, fmt.Errorf("%s --> %s dial attempt failed: %s", s.local, p, err) + return nil, fmt.Errorf("%s --> %s (%s) dial attempt failed: %s", s.local, p, addr, err) } // Trust the transport? Yeah... right. From 544c412193e7a9c24f96d1c2ae33e44ff7fa0984 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 5 Apr 2019 11:03:20 -0700 Subject: [PATCH 1025/3965] identify: only store _reported_ multiaddrs We still tell the remote host about the observed addr but we don't store it. That way, we give them a chance to decide if they want to actually use and advertise it. Ideally, we'd distinguish between local information and signed routing information but we don't do that yet. This should reduce the address explosion issue where peers learn about multiple (bad) observed addresses for the same peer. It should also give peers more control over how they can be dialed. --- p2p/protocol/identify/id.go | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index e909c4071f..bd89ca30cf 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -238,11 +238,8 @@ func (ids *IDService) consumeMessage(mes *pb.Identify, c inet.Conn) { lmaddrs = append(lmaddrs, maddr) } - // if the address reported by the connection roughly matches their annoucned - // listener addresses, its likely to be an external NAT address - if HasConsistentTransport(c.RemoteMultiaddr(), lmaddrs) { - lmaddrs = append(lmaddrs, c.RemoteMultiaddr()) - } + // NOTE: Do not add `c.RemoteMultiaddr()` to the peerstore if the remote + // peer doesn't tell us to do so. Otherwise, we'll advertise it. // Extend the TTLs on the known (probably) good addresses. // Taking the lock ensures that we don't concurrently process a disconnect. From 09d9f0cc220433f2c74821bbea78d411b9f8797e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 5 Apr 2019 12:13:06 -0700 Subject: [PATCH 1026/3965] chore: fail in the libp2p constructor if we fail to store the key --- config/config.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/config/config.go b/config/config.go index 4c9342165b..f94f7e7df6 100644 --- a/config/config.go +++ b/config/config.go @@ -99,8 +99,12 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } if !cfg.Insecure { - cfg.Peerstore.AddPrivKey(pid, cfg.PeerKey) - cfg.Peerstore.AddPubKey(pid, cfg.PeerKey.GetPublic()) + if err := cfg.Peerstore.AddPrivKey(pid, cfg.PeerKey); err != nil { + return nil, err + } + if err := cfg.Peerstore.AddPubKey(pid, cfg.PeerKey.GetPublic()); err != nil { + return nil, err + } } // TODO: Make the swarm implementation configurable. From 125e0995c8c1c6e2f7d308d4e25238bef5e406b2 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Fri, 5 Apr 2019 14:36:04 -0700 Subject: [PATCH 1027/3965] Fix broken link in README.md It looks like `github.com/whyrusleeping/go-smux-mplex` was renamed to `github.com/whyrusleeping/go-smux-multiplex`. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 12361a0f43..ee1e917917 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,7 @@ List of packages currently in existence for libp2p: | **Stream Muxers** | | [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.com/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.com/libp2p/go-stream-muxer) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | interfaces | | [`go-smux-yamux`](//github.com/whyrusleeping/go-smux-yamux) | [![Travis CI](https://travis-ci.com/whyrusleeping/go-smux-yamux.svg?branch=master)](https://travis-ci.com/whyrusleeping/go-smux-yamux) | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-yamux/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-yamux) | YAMUX stream multiplexer | -| [`go-smux-mplex`](//github.com/whyrusleeping/go-smux-mplex) | [![Travis CI](https://travis-ci.com/whyrusleeping/go-smux-mplex.svg?branch=master)](https://travis-ci.com/whyrusleeping/go-smux-mplex) | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-mplex/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-mplex) | MPLEX stream multiplexer | +| [`go-smux-multiplex`](//github.com/whyrusleeping/go-smux-multiplex) | [![Travis CI](https://travis-ci.com/whyrusleeping/go-smux-multiplex.svg?branch=master)](https://travis-ci.com/whyrusleeping/go-smux-multiplex) | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-multiplex/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-multiplex) | MPLEX stream multiplexer | | **NAT Traversal** | | [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-nat) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | | | **Peerstore** | From 80ada8a7d6ce34f1c4448d03eb924bb63d9b13a2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 5 Apr 2019 12:13:24 -0700 Subject: [PATCH 1028/3965] relay: no more wrapping We don't need the host wrappers, we can just replace the filter AddrsFactory. Also, always filter out relay addresses. --- config/config.go | 33 +++++-- p2p/host/relay/autorelay.go | 177 ++++++++++++++++-------------------- p2p/host/relay/doc.go | 5 - p2p/host/relay/relay.go | 37 +++----- 4 files changed, 115 insertions(+), 137 deletions(-) diff --git a/config/config.go b/config/config.go index 4c9342165b..695a3e2c16 100644 --- a/config/config.go +++ b/config/config.go @@ -109,18 +109,29 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { swrm.Filters = cfg.Filters } - var h host.Host - h, err = bhost.NewHost(ctx, swrm, &bhost.HostOpts{ + h, err := bhost.NewHost(ctx, swrm, &bhost.HostOpts{ ConnManager: cfg.ConnManager, AddrsFactory: cfg.AddrsFactory, NATManager: cfg.NATManager, EnablePing: !cfg.DisablePing, }) + if err != nil { swrm.Close() return nil, err } + if cfg.Relay { + // If we've enabled the relay, we should filter out relay + // addresses by default. + // + // TODO: We shouldn't be doing this here. + oldFactory := h.AddrsFactory + h.AddrsFactory = func(addrs []ma.Multiaddr) []ma.Multiaddr { + return relay.Filter(oldFactory(addrs)) + } + } + upgrader := new(tptu.Upgrader) upgrader.Protector = cfg.Protector upgrader.Filters = swrm.Filters @@ -206,18 +217,24 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } if hop { - h = relay.NewRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery) + // advertise ourselves + // TODO: Why do we only do this when EnableAutoRelay is + // set? This has absolutely _nothing_ to do with + // autorelay. + relay.Advertise(ctx, discovery) } else { - h = relay.NewAutoRelayHost(swrm.Context(), h.(*bhost.BasicHost), discovery, router) + // TODO + // 1. Stop abusing contexts like this. + // 2. Introduce a service management system (e.g., + // uber's fx) so we can actually manage the lifetime of + // this service. + _ = relay.NewAutoRelay(swrm.Context(), h, discovery, router) } } if router != nil { - h = routed.Wrap(h, router) + return routed.Wrap(h, router), nil } - - // TODO: Bootstrapping. - return h, nil } diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index d5f7daba02..3182a9f0ef 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -12,7 +12,6 @@ import ( autonat "github.com/libp2p/go-libp2p-autonat" _ "github.com/libp2p/go-libp2p-circuit" discovery "github.com/libp2p/go-libp2p-discovery" - host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" @@ -29,13 +28,11 @@ var ( DesiredRelays = 3 BootDelay = 20 * time.Second - - unspecificRelay = ma.StringCast("/p2p-circuit") ) -// AutoRelayHost is a Host that uses relays for connectivity when a NAT is detected. -type AutoRelayHost struct { - *basic.BasicHost +// AutoRelay is a Host that uses relays for connectivity when a NAT is detected. +type AutoRelay struct { + host *basic.BasicHost discover discovery.Discoverer router routing.PeerRouting autonat autonat.AutoNAT @@ -48,37 +45,37 @@ type AutoRelayHost struct { addrs []ma.Multiaddr } -func NewAutoRelayHost(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer, router routing.PeerRouting) *AutoRelayHost { - h := &AutoRelayHost{ - BasicHost: bhost, +func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer, router routing.PeerRouting) *AutoRelay { + ar := &AutoRelay{ + host: bhost, discover: discover, router: router, addrsF: bhost.AddrsFactory, relays: make(map[peer.ID]pstore.PeerInfo), disconnect: make(chan struct{}, 1), } - h.autonat = autonat.NewAutoNAT(ctx, bhost, h.baseAddrs) - bhost.AddrsFactory = h.hostAddrs - bhost.Network().Notify(h) - go h.background(ctx) - return h + ar.autonat = autonat.NewAutoNAT(ctx, bhost, ar.baseAddrs) + bhost.AddrsFactory = ar.hostAddrs + bhost.Network().Notify(ar) + go ar.background(ctx) + return ar } -func (h *AutoRelayHost) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { - h.mx.Lock() - defer h.mx.Unlock() - if h.addrs != nil && h.autonat.Status() == autonat.NATStatusPrivate { - return h.addrs +func (ar *AutoRelay) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { + ar.mx.Lock() + defer ar.mx.Unlock() + if ar.addrs != nil && ar.autonat.Status() == autonat.NATStatusPrivate { + return ar.addrs } else { - return filterUnspecificRelay(h.addrsF(addrs)) + return ar.addrsF(addrs) } } -func (h *AutoRelayHost) baseAddrs() []ma.Multiaddr { - return filterUnspecificRelay(h.addrsF(h.AllAddrs())) +func (ar *AutoRelay) baseAddrs() []ma.Multiaddr { + return ar.addrsF(ar.host.AllAddrs()) } -func (h *AutoRelayHost) background(ctx context.Context) { +func (ar *AutoRelay) background(ctx context.Context) { select { case <-time.After(autonat.AutoNATBootDelay + BootDelay): case <-ctx.Done(): @@ -90,39 +87,39 @@ func (h *AutoRelayHost) background(ctx context.Context) { for { wait := autonat.AutoNATRefreshInterval - switch h.autonat.Status() { + switch ar.autonat.Status() { case autonat.NATStatusUnknown: wait = autonat.AutoNATRetryInterval case autonat.NATStatusPublic: // invalidate addrs - h.mx.Lock() - if h.addrs != nil { - h.addrs = nil + ar.mx.Lock() + if ar.addrs != nil { + ar.addrs = nil push = true } - h.mx.Unlock() + ar.mx.Unlock() // if we had previously announced relay addrs, push our public addrs if push { push = false - h.PushIdentify() + ar.host.PushIdentify() } case autonat.NATStatusPrivate: push = false // clear, findRelays pushes as needed - h.findRelays(ctx) + ar.findRelays(ctx) } select { - case <-h.disconnect: + case <-ar.disconnect: // invalidate addrs - h.mx.Lock() - if h.addrs != nil { - h.addrs = nil + ar.mx.Lock() + if ar.addrs != nil { + ar.addrs = nil push = true } - h.mx.Unlock() + ar.mx.Unlock() case <-time.After(wait): case <-ctx.Done(): return @@ -130,14 +127,14 @@ func (h *AutoRelayHost) background(ctx context.Context) { } } -func (h *AutoRelayHost) findRelays(ctx context.Context) { - h.mx.Lock() - if len(h.relays) >= DesiredRelays { - h.mx.Unlock() +func (ar *AutoRelay) findRelays(ctx context.Context) { + ar.mx.Lock() + if len(ar.relays) >= DesiredRelays { + ar.mx.Unlock() return } - need := DesiredRelays - len(h.relays) - h.mx.Unlock() + need := DesiredRelays - len(ar.relays) + ar.mx.Unlock() limit := 20 if need > limit/2 { @@ -145,29 +142,29 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { } dctx, cancel := context.WithTimeout(ctx, 30*time.Second) - pis, err := discovery.FindPeers(dctx, h.discover, RelayRendezvous, limit) + pis, err := discovery.FindPeers(dctx, ar.discover, RelayRendezvous, limit) cancel() if err != nil { log.Debugf("error discovering relays: %s", err.Error()) return } - pis = h.selectRelays(pis) + pis = ar.selectRelays(pis) update := 0 for _, pi := range pis { - h.mx.Lock() - if _, ok := h.relays[pi.ID]; ok { - h.mx.Unlock() + ar.mx.Lock() + if _, ok := ar.relays[pi.ID]; ok { + ar.mx.Unlock() continue } - h.mx.Unlock() + ar.mx.Unlock() cctx, cancel := context.WithTimeout(ctx, 60*time.Second) if len(pi.Addrs) == 0 { - pi, err = h.router.FindPeer(cctx, pi.ID) + pi, err = ar.router.FindPeer(cctx, pi.ID) if err != nil { log.Debugf("error finding relay peer %s: %s", pi.ID, err.Error()) cancel() @@ -175,7 +172,7 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { } } - err = h.Connect(cctx, pi) + err = ar.host.Connect(cctx, pi) cancel() if err != nil { log.Debugf("error connecting to relay %s: %s", pi.ID, err.Error()) @@ -183,12 +180,12 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { } log.Debugf("connected to relay %s", pi.ID) - h.mx.Lock() - h.relays[pi.ID] = pi - h.mx.Unlock() + ar.mx.Lock() + ar.relays[pi.ID] = pi + ar.mx.Unlock() // tag the connection as very important - h.ConnManager().TagPeer(pi.ID, "relay", 42) + ar.host.ConnManager().TagPeer(pi.ID, "relay", 42) update++ need-- @@ -197,24 +194,24 @@ func (h *AutoRelayHost) findRelays(ctx context.Context) { } } - if update > 0 || h.addrs == nil { - h.updateAddrs() + if update > 0 || ar.addrs == nil { + ar.updateAddrs() } } -func (h *AutoRelayHost) selectRelays(pis []pstore.PeerInfo) []pstore.PeerInfo { +func (ar *AutoRelay) selectRelays(pis []pstore.PeerInfo) []pstore.PeerInfo { // TODO better relay selection strategy; this just selects random relays // but we should probably use ping latency as the selection metric shuffleRelays(pis) return pis } -func (h *AutoRelayHost) updateAddrs() { - h.doUpdateAddrs() - h.PushIdentify() +func (ar *AutoRelay) updateAddrs() { + ar.doUpdateAddrs() + ar.host.PushIdentify() } -// This function updates our NATed advertised addrs (h.addrs) +// This function updates our NATed advertised addrs (ar.addrs) // The public addrs are rewritten so that they only retain the public IP part; they // become undialable but are useful as a hint to the dialer to determine whether or not // to dial private addrs. @@ -223,12 +220,12 @@ func (h *AutoRelayHost) updateAddrs() { // On top of those, we add the relay-specific addrs for the relays to which we are // connected. For each non-private relay addr, we encapsulate the p2p-circuit addr // through which we can be dialed. -func (h *AutoRelayHost) doUpdateAddrs() { - h.mx.Lock() - defer h.mx.Unlock() +func (ar *AutoRelay) doUpdateAddrs() { + ar.mx.Lock() + defer ar.mx.Unlock() - addrs := h.baseAddrs() - raddrs := make([]ma.Multiaddr, 0, len(addrs)+len(h.relays)) + addrs := ar.baseAddrs() + raddrs := make([]ma.Multiaddr, 0, len(addrs)+len(ar.relays)) // remove our public addresses from the list for _, addr := range addrs { @@ -239,7 +236,7 @@ func (h *AutoRelayHost) doUpdateAddrs() { } // add relay specific addrs to the list - for _, pi := range h.relays { + for _, pi := range ar.relays { circuit, err := ma.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", pi.ID.Pretty())) if err != nil { panic(err) @@ -253,18 +250,7 @@ func (h *AutoRelayHost) doUpdateAddrs() { } } - h.addrs = raddrs -} - -func filterUnspecificRelay(addrs []ma.Multiaddr) []ma.Multiaddr { - res := make([]ma.Multiaddr, 0, len(addrs)) - for _, addr := range addrs { - if addr.Equal(unspecificRelay) { - continue - } - res = append(res, addr) - } - return res + ar.addrs = raddrs } func shuffleRelays(pis []pstore.PeerInfo) { @@ -274,34 +260,23 @@ func shuffleRelays(pis []pstore.PeerInfo) { } } -func containsAddr(lst []ma.Multiaddr, addr ma.Multiaddr) bool { - for _, xaddr := range lst { - if xaddr.Equal(addr) { - return true - } - } - return false -} +func (ar *AutoRelay) Listen(inet.Network, ma.Multiaddr) {} +func (ar *AutoRelay) ListenClose(inet.Network, ma.Multiaddr) {} +func (ar *AutoRelay) Connected(inet.Network, inet.Conn) {} -// notify -func (h *AutoRelayHost) Listen(inet.Network, ma.Multiaddr) {} -func (h *AutoRelayHost) ListenClose(inet.Network, ma.Multiaddr) {} -func (h *AutoRelayHost) Connected(inet.Network, inet.Conn) {} - -func (h *AutoRelayHost) Disconnected(_ inet.Network, c inet.Conn) { +func (ar *AutoRelay) Disconnected(net inet.Network, c inet.Conn) { p := c.RemotePeer() - h.mx.Lock() - defer h.mx.Unlock() - if _, ok := h.relays[p]; ok { - delete(h.relays, p) + + ar.mx.Lock() + defer ar.mx.Unlock() + if _, ok := ar.relays[p]; ok { + delete(ar.relays, p) select { - case h.disconnect <- struct{}{}: + case ar.disconnect <- struct{}{}: default: } } } -func (h *AutoRelayHost) OpenedStream(inet.Network, inet.Stream) {} -func (h *AutoRelayHost) ClosedStream(inet.Network, inet.Stream) {} - -var _ host.Host = (*AutoRelayHost)(nil) +func (ar *AutoRelay) OpenedStream(inet.Network, inet.Stream) {} +func (ar *AutoRelay) ClosedStream(inet.Network, inet.Stream) {} diff --git a/p2p/host/relay/doc.go b/p2p/host/relay/doc.go index c0a5878113..a9c72f6003 100644 --- a/p2p/host/relay/doc.go +++ b/p2p/host/relay/doc.go @@ -14,11 +14,6 @@ How it works: - `AutoNATService` instances are instantiated in the bootstrappers (or other well known publicly reachable hosts) -- `RelayHost`s are constructed with - `libp2p.New(libp2p.EnableRelay(circuit.OptHop), libp2p.Routing(makeDHT))`. - They provide Relay Hop services, and advertise through the DHT - in the `/libp2p/relay` namespace - - `AutoRelayHost`s are constructed with `libp2p.New(libp2p.Routing(makeDHT))` They passively discover autonat service instances and test dialability of their listen address set through them. When the presence of NAT is detected, diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index bf91c20ee4..0f38166984 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -4,10 +4,8 @@ import ( "context" "time" - basic "github.com/libp2p/go-libp2p/p2p/host/basic" - + circuit "github.com/libp2p/go-libp2p-circuit" discovery "github.com/libp2p/go-libp2p-discovery" - host "github.com/libp2p/go-libp2p-host" ma "github.com/multiformats/go-multiaddr" ) @@ -15,21 +13,8 @@ var ( AdvertiseBootDelay = 30 * time.Second ) -// RelayHost is a Host that provides Relay services. -type RelayHost struct { - *basic.BasicHost - advertise discovery.Advertiser - addrsF basic.AddrsFactory -} - -// New constructs a new RelayHost -func NewRelayHost(ctx context.Context, bhost *basic.BasicHost, advertise discovery.Advertiser) *RelayHost { - h := &RelayHost{ - BasicHost: bhost, - addrsF: bhost.AddrsFactory, - advertise: advertise, - } - bhost.AddrsFactory = h.hostAddrs +// Advertise advertises this node as a libp2p relay. +func Advertise(ctx context.Context, advertise discovery.Advertiser) { go func() { select { case <-time.After(AdvertiseBootDelay): @@ -37,11 +22,17 @@ func NewRelayHost(ctx context.Context, bhost *basic.BasicHost, advertise discove case <-ctx.Done(): } }() - return h } -func (h *RelayHost) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { - return filterUnspecificRelay(h.addrsF(addrs)) +// Filter filters out all relay addresses. +func Filter(addrs []ma.Multiaddr) []ma.Multiaddr { + raddrs := make([]ma.Multiaddr, 0, len(addrs)) + for _, addr := range addrs { + _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) + if err == nil { + continue + } + raddrs = append(raddrs, addr) + } + return raddrs } - -var _ host.Host = (*RelayHost)(nil) From 5b665643c78379131e4190b213870855e9019046 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 5 Apr 2019 19:51:41 -0700 Subject: [PATCH 1029/3965] relay: only stop advertising relay addresses if we fully disconnect --- p2p/host/relay/autorelay.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 3182a9f0ef..bd88857707 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -269,6 +269,12 @@ func (ar *AutoRelay) Disconnected(net inet.Network, c inet.Conn) { ar.mx.Lock() defer ar.mx.Unlock() + + if ar.host.Network().Connectedness(p) == inet.Connected { + // We have a second connection. + return + } + if _, ok := ar.relays[p]; ok { delete(ar.relays, p) select { From 45df977d578fd0e5d09b8ba68bb694bba4691f2e Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 6 Apr 2019 12:22:12 +0300 Subject: [PATCH 1030/3965] bump relay record limit to 50 --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index bd88857707..fbbbeef565 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -136,7 +136,7 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { need := DesiredRelays - len(ar.relays) ar.mx.Unlock() - limit := 20 + limit := 50 if need > limit/2 { limit = 2 * need } From 873082f1e3aa149f2a1e29eb79ac2b792679edad Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 6 Apr 2019 07:17:27 -0700 Subject: [PATCH 1031/3965] unexport private method --- p2p/protocol/internal/circuitv1-deprecated/notify.go | 2 +- p2p/protocol/internal/circuitv1-deprecated/relay.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/notify.go b/p2p/protocol/internal/circuitv1-deprecated/notify.go index a82ef552b4..6d9f62dc2a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/notify.go +++ b/p2p/protocol/internal/circuitv1-deprecated/notify.go @@ -13,7 +13,7 @@ var _ inet.Notifiee = (*RelayNotifiee)(nil) type RelayNotifiee Relay -func (r *Relay) Notifiee() inet.Notifiee { +func (r *Relay) notifiee() inet.Notifiee { return (*RelayNotifiee)(r) } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index a5323e7bbf..1d02bf1258 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -90,7 +90,7 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. h.SetStreamHandler(ProtoID, r.handleNewStream) if r.discovery { - h.Network().Notify(r.Notifiee()) + h.Network().Notify(r.notifiee()) } return r, nil From a9483ee3aa54ffd8cd9eefa6aba0b5a990955aed Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 6 Apr 2019 07:17:29 -0700 Subject: [PATCH 1032/3965] doc: document options --- .../internal/circuitv1-deprecated/relay.go | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 1d02bf1258..8fa10c457f 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -27,6 +27,7 @@ const maxMessageSize = 4096 var RelayAcceptTimeout = time.Minute var HopConnectTimeout = 10 * time.Second +// Relay is the relay transport and service. type Relay struct { host host.Host upgrader *tptu.Upgrader @@ -47,11 +48,22 @@ type Relay struct { lhLk sync.Mutex } +// RelayOpts are options for configuring the relay transport. type RelayOpt int var ( - OptActive = RelayOpt(0) - OptHop = RelayOpt(1) + // OptActive configures the relay transport to actively establish + // outbound connections on behalf of clients. You probably don't want to + // enable this unless you know what you're doing. + OptActive = RelayOpt(0) + // OptHop configures the relay transport to accept requests to relay + // traffic on behalf of third-parties. Unless OptActive is specified, + // this will only relay traffic between peers already connected to this + // node. + OptHop = RelayOpt(1) + // OptDiscovery configures this relay transport to discover new relays + // by probing every new peer. You almost _certainly_ don't want to + // enable this. OptDiscovery = RelayOpt(2) ) @@ -63,6 +75,7 @@ func (e RelayError) Error() string { return fmt.Sprintf("error opening relay circuit: %s (%d)", pb.CircuitRelay_Status_name[int32(e.Code)], e.Code) } +// NewRelay constructs a new relay. func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts ...RelayOpt) (*Relay, error) { r := &Relay{ upgrader: upgrader, From 1d8efc31aa0e9f65110272d772aef6eec782d477 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 6 Apr 2019 07:34:52 -0700 Subject: [PATCH 1033/3965] doc: document relay options --- options.go | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/options.go b/options.go index 125ec8d601..9ef466c3a4 100644 --- a/options.go +++ b/options.go @@ -201,7 +201,12 @@ func AddrsFactory(factory config.AddrsFactory) Option { } } -// EnableRelay configures libp2p to enable the relay transport with configuration options. +// EnableRelay configures libp2p to enable the relay transport with +// configuration options. By default, this option only configures libp2p to +// accept inbound connections from relays and make outbound connections +// _through_ relays when requested by the remote peer. (default: enabled) +// +// To _act_ as a relay, pass the circuit.OptHop option. func EnableRelay(options ...circuit.RelayOpt) Option { return func(cfg *Config) error { cfg.RelayCustom = true @@ -211,7 +216,7 @@ func EnableRelay(options ...circuit.RelayOpt) Option { } } -// DisableRelay configures libp2p to disable the relay transport +// DisableRelay configures libp2p to disable the relay transport. func DisableRelay() Option { return func(cfg *Config) error { cfg.RelayCustom = true @@ -220,8 +225,18 @@ func DisableRelay() Option { } } -// EnableAutoRelay configures libp2p to enable autorelay advertising; requires relay to -// be enabled and the Routing option to provide an instance of ContentRouting. +// EnableAutoRelay configures libp2p to enable the AutoRelay subsystem. It is an +// error to enable AutoRelay without enabling relay (enabled by default) and +// routing (not enabled by default). +// +// This subsystem performs two functions: +// +// 1. When this libp2p node is configured to act as a relay "hop" +// (circuit.OptHop is passed to EnableRelay), this node will advertise itself +// as a public relay using the provided routing system. +// 2. When this libp2p node is _not_ configured as a relay "hop", it will +// automatically if it is unreachable (e.g., behind a NAT). If so, it will +// find, configure, and announce a set of public relays. func EnableAutoRelay() Option { return func(cfg *Config) error { cfg.EnableAutoRelay = true From c580255d13d79f4ed4db0a080f655c6c0f82b5fc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 6 Apr 2019 07:41:58 -0700 Subject: [PATCH 1034/3965] relay: drop design comments These are issues for another day. --- config/config.go | 8 -------- 1 file changed, 8 deletions(-) diff --git a/config/config.go b/config/config.go index 695a3e2c16..e895eed98b 100644 --- a/config/config.go +++ b/config/config.go @@ -218,16 +218,8 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { if hop { // advertise ourselves - // TODO: Why do we only do this when EnableAutoRelay is - // set? This has absolutely _nothing_ to do with - // autorelay. relay.Advertise(ctx, discovery) } else { - // TODO - // 1. Stop abusing contexts like this. - // 2. Introduce a service management system (e.g., - // uber's fx) so we can actually manage the lifetime of - // this service. _ = relay.NewAutoRelay(swrm.Context(), h, discovery, router) } } From fee7b01e32c02c88417e146c925aefe050725155 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 6 Apr 2019 18:11:33 +0300 Subject: [PATCH 1035/3965] add missing word --- options.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/options.go b/options.go index 9ef466c3a4..e98edda945 100644 --- a/options.go +++ b/options.go @@ -235,7 +235,7 @@ func DisableRelay() Option { // (circuit.OptHop is passed to EnableRelay), this node will advertise itself // as a public relay using the provided routing system. // 2. When this libp2p node is _not_ configured as a relay "hop", it will -// automatically if it is unreachable (e.g., behind a NAT). If so, it will +// automatically detect if it is unreachable (e.g., behind a NAT). If so, it will // find, configure, and announce a set of public relays. func EnableAutoRelay() Option { return func(cfg *Config) error { From 7093262620cba98d1104bd02380cebf538d35043 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 6 Apr 2019 08:11:22 -0700 Subject: [PATCH 1036/3965] relay: update documentation --- p2p/host/relay/doc.go | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/p2p/host/relay/doc.go b/p2p/host/relay/doc.go index a9c72f6003..f7511ad97e 100644 --- a/p2p/host/relay/doc.go +++ b/p2p/host/relay/doc.go @@ -1,25 +1,28 @@ /* -The relay package contains host implementations that automatically -advertise relay addresses when the presence of NAT is detected. This -feature is dubbed `autorelay`. +The relay package contains the components necessary to implement the "autorelay" +feature. Warning: the internal interfaces are unstable. System Components: -- AutoNATService instances -- see https://github.com/libp2p/go-libp2p-autonat-svc -- One or more relays, instances of `RelayHost` -- The autorelayed hosts, instances of `AutoRelayHost`. +- A discovery service to discover public relays. +- An AutoNAT client used to determine if the node is behind a NAT/firewall. +- One or more autonat services, instances of `AutoNATServices`. These are used + by the autonat client. +- One or more relays, instances of `RelayHost`. +- The AutoRelay service. This is the service that actually: -How it works: -- `AutoNATService` instances are instantiated in the - bootstrappers (or other well known publicly reachable hosts) +AutoNATService: https://github.com/libp2p/go-libp2p-autonat-svc +AutoNAT: https://github.com/libp2p/go-libp2p-autonat -- `AutoRelayHost`s are constructed with `libp2p.New(libp2p.Routing(makeDHT))` +How it works: +- `AutoNATService` instances are instantiated in the bootstrappers (or other + well known publicly reachable hosts) +- `AutoRelay`s are constructed with `libp2p.New(libp2p.Routing(makeDHT))` They passively discover autonat service instances and test dialability of their listen address set through them. When the presence of NAT is detected, they discover relays through the DHT, connect to some of them and begin advertising relay addresses. The new set of addresses is propagated to connected peers through the `identify/push` protocol. - */ package relay From 87d57b515544e4dd41d30b2190300c84cc322486 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 6 Apr 2019 08:23:07 -0700 Subject: [PATCH 1037/3965] relay: allow users to explicitly add relay addresses --- config/config.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/config.go b/config/config.go index e895eed98b..3370f6e372 100644 --- a/config/config.go +++ b/config/config.go @@ -128,7 +128,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { // TODO: We shouldn't be doing this here. oldFactory := h.AddrsFactory h.AddrsFactory = func(addrs []ma.Multiaddr) []ma.Multiaddr { - return relay.Filter(oldFactory(addrs)) + return oldFactory(relay.Filter(addrs)) } } From 2684cc16ea673c374a00415bbc71ffb5cc2b8cc3 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 7 Apr 2019 14:36:06 +0900 Subject: [PATCH 1038/3965] disable session tickets When resuming a session using session tickets, no certificate chain is presented, and the callbacks needed to verify the peer identity would not be called. --- p2p/security/tls/crypto.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index c991e33b4b..50c9d71a43 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -51,6 +51,7 @@ func NewIdentity(privKey ic.PrivKey) (*Identity, error) { VerifyPeerCertificate: func(_ [][]byte, _ [][]*x509.Certificate) error { panic("tls config not specialized for peer") }, + SessionTicketsDisabled: true, }, }, nil } From 3891451288aca04bc759ead3cc7d83b2d9077e26 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 7 Apr 2019 15:29:27 +0300 Subject: [PATCH 1039/3965] use io.CopyBuffer with explicitly allocated buffers io.Copy uses a 32KB buffer, which pprof indicates result in significant memory usage. --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 8fa10c457f..ec9ca7321c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -376,7 +376,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { go func() { defer r.rmLiveHop(src.ID, dst.ID) - count, err := io.Copy(s, bs) + count, err := io.CopyBuffer(s, bs, make([]byte, 4096)) if err != nil { log.Debugf("relay copy error: %s", err) // Reset both. @@ -390,7 +390,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() go func() { - count, err := io.Copy(bs, s) + count, err := io.CopyBuffer(bs, s, make([]byte, 4096)) if err != nil { log.Debugf("relay copy error: %s", err) // Reset both. From 13baaaf73e7cc5c6a2f2426ca94518c5f99f6f08 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 7 Apr 2019 22:31:58 +0300 Subject: [PATCH 1040/3965] use a sync.Pool for relay buffers --- .../internal/circuitv1-deprecated/relay.go | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index ec9ca7321c..f3f578362a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -40,6 +40,8 @@ type Relay struct { incoming chan *Conn + bufPool sync.Pool + relays map[peer.ID]struct{} mx sync.Mutex @@ -85,6 +87,11 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. incoming: make(chan *Conn), relays: make(map[peer.ID]struct{}), liveHops: make(map[peer.ID]map[peer.ID]int), + bufPool: sync.Pool{ + New: func() interface{} { + return make([]byte, 4096) + }, + }, } for _, opt := range opts { @@ -376,7 +383,10 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { go func() { defer r.rmLiveHop(src.ID, dst.ID) - count, err := io.CopyBuffer(s, bs, make([]byte, 4096)) + buf := r.bufPool.Get().([]byte) + defer r.bufPool.Put(buf) + + count, err := io.CopyBuffer(s, bs, buf) if err != nil { log.Debugf("relay copy error: %s", err) // Reset both. @@ -390,7 +400,10 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() go func() { - count, err := io.CopyBuffer(bs, s, make([]byte, 4096)) + buf := r.bufPool.Get().([]byte) + defer r.bufPool.Put(buf) + + count, err := io.CopyBuffer(bs, s, buf) if err != nil { log.Debugf("relay copy error: %s", err) // Reset both. From 9cc9d459f4fc73315ff8e1160939a84a6803683e Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 8 Apr 2019 10:57:15 +0300 Subject: [PATCH 1041/3965] use go-buffer-pool --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index f3f578362a..07e49e0b9e 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -10,6 +10,7 @@ import ( pb "github.com/libp2p/go-libp2p-circuit/pb" logging "github.com/ipfs/go-log" + pool "github.com/libp2p/go-buffer-pool" host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" @@ -40,7 +41,7 @@ type Relay struct { incoming chan *Conn - bufPool sync.Pool + bufPool pool.BufferPool relays map[peer.ID]struct{} mx sync.Mutex @@ -87,11 +88,6 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. incoming: make(chan *Conn), relays: make(map[peer.ID]struct{}), liveHops: make(map[peer.ID]map[peer.ID]int), - bufPool: sync.Pool{ - New: func() interface{} { - return make([]byte, 4096) - }, - }, } for _, opt := range opts { @@ -383,7 +379,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { go func() { defer r.rmLiveHop(src.ID, dst.ID) - buf := r.bufPool.Get().([]byte) + buf := r.bufPool.Get(4096) defer r.bufPool.Put(buf) count, err := io.CopyBuffer(s, bs, buf) @@ -400,7 +396,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() go func() { - buf := r.bufPool.Get().([]byte) + buf := r.bufPool.Get(4096) defer r.bufPool.Put(buf) count, err := io.CopyBuffer(bs, s, buf) From e0ed4d05e2ac7574dbb051b61b9d14acfd1a4c3e Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 8 Apr 2019 10:59:44 +0300 Subject: [PATCH 1042/3965] use global buffer pool --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 07e49e0b9e..ddda926140 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -41,8 +41,6 @@ type Relay struct { incoming chan *Conn - bufPool pool.BufferPool - relays map[peer.ID]struct{} mx sync.Mutex @@ -379,8 +377,8 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { go func() { defer r.rmLiveHop(src.ID, dst.ID) - buf := r.bufPool.Get(4096) - defer r.bufPool.Put(buf) + buf := pool.Get(4096) + defer pool.Put(buf) count, err := io.CopyBuffer(s, bs, buf) if err != nil { @@ -396,8 +394,8 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() go func() { - buf := r.bufPool.Get(4096) - defer r.bufPool.Put(buf) + buf := pool.Get(4096) + defer pool.Put(buf) count, err := io.CopyBuffer(bs, s, buf) if err != nil { From 25d32c5c368277bd64c85c1b66d247c12ed2cc21 Mon Sep 17 00:00:00 2001 From: Matt Joiner Date: Mon, 8 Apr 2019 18:11:34 +1000 Subject: [PATCH 1043/3965] Add to comment on KeyBook.PrivKey (#64) --- p2p/host/peerstore/interface.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 4d166684f5..bdf88393a4 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -125,14 +125,15 @@ type AddrBook interface { // KeyBook tracks the keys of Peers. type KeyBook interface { - // PubKey stores the public key of a peer. PubKey(peer.ID) ic.PubKey // AddPubKey stores the public key of a peer. AddPubKey(peer.ID, ic.PubKey) error - // PrivKey returns the private key of a peer. + // PrivKey returns the private key of a peer, if known. Generally this might only be our own + // private key, see + // https://discuss.libp2p.io/t/what-is-the-purpose-of-having-map-peer-id-privatekey-in-peerstore/74. PrivKey(peer.ID) ic.PrivKey // AddPrivKey stores the private key of a peer. From 9bb06fcb2aefa07a84837cdf23273173568cd31f Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 8 Apr 2019 19:07:47 +0300 Subject: [PATCH 1044/3965] gomod: update go-libp2p-circuit --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 2ae577495a..93d39af313 100644 --- a/go.mod +++ b/go.mod @@ -11,7 +11,7 @@ require ( github.com/libp2p/go-conn-security-multistream v0.0.1 github.com/libp2p/go-libp2p-autonat v0.0.3 github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.1 + github.com/libp2p/go-libp2p-circuit v0.0.2 github.com/libp2p/go-libp2p-crypto v0.0.1 github.com/libp2p/go-libp2p-discovery v0.0.1 github.com/libp2p/go-libp2p-host v0.0.1 diff --git a/go.sum b/go.sum index 9974f7a950..3ebe0a2998 100644 --- a/go.sum +++ b/go.sum @@ -88,8 +88,8 @@ github.com/libp2p/go-libp2p-autonat v0.0.3 h1:PUD+pAx8Qs9hh+Bowzxq8RCkg/Vwrz5oCF github.com/libp2p/go-libp2p-autonat v0.0.3/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.1 h1:DYbjyQ5ZY3QVAVYZWG4uzBQ6Wmcd1C82Bk8Q/pJlM1I= -github.com/libp2p/go-libp2p-circuit v0.0.1/go.mod h1:Dqm0s/BiV63j8EEAs8hr1H5HudqvCAeXxDyic59lCwE= +github.com/libp2p/go-libp2p-circuit v0.0.2 h1:wwbYslVw1aW2DAAuJxXRS5gB2clz1s23gv9NOb8iwsE= +github.com/libp2p/go-libp2p-circuit v0.0.2/go.mod h1:zV136p4UQ76qH/Wj+X/Hivcg6sf6Yb1G7YL8o+GGj38= github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-discovery v0.0.1 h1:VkjCKmJQMwpDUwtA8Qc1z3TQAHJgQ5nGQ6cdN0wQXOw= From bcbf7a59c1c719dc731721a25236642c910caa0f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 5 Apr 2019 11:38:30 -0700 Subject: [PATCH 1045/3965] identify: only record observed addresses that match an address we're announcing This is should prevent us from, e.g., announcing relay addresses _just_ because a peer tells us we're reachable through a relay. --- p2p/protocol/identify/id.go | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index bd89ca30cf..01fc7a7b2c 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -240,6 +240,11 @@ func (ids *IDService) consumeMessage(mes *pb.Identify, c inet.Conn) { // NOTE: Do not add `c.RemoteMultiaddr()` to the peerstore if the remote // peer doesn't tell us to do so. Otherwise, we'll advertise it. + // + // This can cause an "addr-splosion" issue where the network will slowly + // gossip and collect observed but unadvertised addresses. Given a NAT + // that picks random source ports, this can cause DHT nodes to collect + // many undialable addresses for other peers. // Extend the TTLs on the known (probably) good addresses. // Taking the lock ensures that we don't concurrently process a disconnect. @@ -415,6 +420,11 @@ func (ids *IDService) consumeObservedAddress(observed []byte, c inet.Conn) { return } + if !HasConsistentTransport(maddr, ids.Host.Addrs()) { + log.Debugf("ignoring observed multiaddr that doesn't match the transports of any addresses we're announcing", c.RemoteMultiaddr()) + return + } + // ok! we have the observed version of one of our ListenAddresses! log.Debugf("added own observed listen addr: %s --> %s", c.LocalMultiaddr(), maddr) ids.observedAddrs.Add(maddr, c.LocalMultiaddr(), c.RemoteMultiaddr(), From a7eb2efe4be1ae46873240535f955687e809f2e8 Mon Sep 17 00:00:00 2001 From: whyrusleeping Date: Mon, 8 Apr 2019 16:07:31 -0700 Subject: [PATCH 1046/3965] move things outside of the lock in obsaddr --- p2p/protocol/identify/obsaddr.go | 16 +++++----- p2p/protocol/identify/obsaddr_test.go | 46 +++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 8 deletions(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 1f6a10a489..793105bd07 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -119,6 +119,14 @@ func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, direction net.Direction) { + now := time.Now() + observerString := observerGroup(observer) + localString := string(local.Bytes()) + ob := observation{ + seenTime: now, + connDirection: direction, + } + oas.Lock() defer oas.Unlock() @@ -128,14 +136,6 @@ func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, oas.ttl = pstore.OwnObservedAddrTTL } - now := time.Now() - observerString := observerGroup(observer) - localString := string(local.Bytes()) - ob := observation{ - seenTime: now, - connDirection: direction, - } - observedAddrs := oas.addrs[localString] // check if observed address seen yet, if so, update it for i, previousObserved := range observedAddrs { diff --git a/p2p/protocol/identify/obsaddr_test.go b/p2p/protocol/identify/obsaddr_test.go index 7291326218..8d2e8c0e23 100644 --- a/p2p/protocol/identify/obsaddr_test.go +++ b/p2p/protocol/identify/obsaddr_test.go @@ -1,6 +1,7 @@ package identify import ( + "sync" "testing" "time" @@ -117,3 +118,48 @@ func TestObsAddrSet(t *testing.T) { t.Error("addrs should have timed out") } } + +func TestAddAddrsProfile(b *testing.T) { + m := func(s string) ma.Multiaddr { + m, err := ma.NewMultiaddr(s) + if err != nil { + b.Fatal(err) + } + return m + } + oas := &ObservedAddrSet{} + + add := func(oas *ObservedAddrSet, observed, observer ma.Multiaddr) { + dummyLocal := m("/ip4/127.0.0.1/tcp/10086") + dummyDirection := net.DirOutbound + + oas.Add(observed, dummyLocal, observer, dummyDirection) + } + + a1 := m("/ip4/1.2.3.4/tcp/1231") + a2 := m("/ip4/1.2.3.4/tcp/1232") + a3 := m("/ip4/1.2.3.4/tcp/1233") + a4 := m("/ip4/1.2.3.4/tcp/1234") + a5 := m("/ip4/1.2.3.4/tcp/1235") + + b1 := m("/ip4/1.2.3.6/tcp/1236") + b2 := m("/ip4/1.2.3.7/tcp/1237") + b3 := m("/ip4/1.2.3.8/tcp/1237") + b4 := m("/ip4/1.2.3.9/tcp/1237") + b5 := m("/ip4/1.2.3.10/tcp/1237") + + _ = []ma.Multiaddr{a1, a2, a3, a4, a5, b1, b2, b3, b4, b5} + + var wg sync.WaitGroup + for i := 0; i < 1000; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for j := 0; j < 10000; j++ { + add(oas, a1, b1) + } + }() + } + + wg.Wait() +} From aebe405680aedbebb95240467d2c05c4f4fad279 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 8 Apr 2019 18:29:45 -0700 Subject: [PATCH 1047/3965] test: disable TestAddAddrsProfile when the race detector is enabled --- p2p/protocol/identify/obsaddr_test.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/p2p/protocol/identify/obsaddr_test.go b/p2p/protocol/identify/obsaddr_test.go index 8d2e8c0e23..12bdb54a8f 100644 --- a/p2p/protocol/identify/obsaddr_test.go +++ b/p2p/protocol/identify/obsaddr_test.go @@ -5,6 +5,7 @@ import ( "testing" "time" + detectrace "github.com/ipfs/go-detect-race" net "github.com/libp2p/go-libp2p-net" ma "github.com/multiformats/go-multiaddr" ) @@ -120,6 +121,9 @@ func TestObsAddrSet(t *testing.T) { } func TestAddAddrsProfile(b *testing.T) { + if detectrace.WithRace() { + b.Skip("test too slow when the race detector is running") + } m := func(s string) ma.Multiaddr { m, err := ma.NewMultiaddr(s) if err != nil { From 316f0867f68564ed7128bb740fdcbb42b0f46988 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 8 Apr 2019 18:41:22 -0700 Subject: [PATCH 1048/3965] identify: avoid parsing/printing multiaddrs --- p2p/protocol/identify/obsaddr.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 793105bd07..af153edc60 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -167,7 +167,8 @@ func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, // IP addresses. In practice, this is what we want. func observerGroup(m ma.Multiaddr) string { //TODO: If IPv6 rolls out we should mark /64 routing zones as one group - return ma.Split(m)[0].String() + first, _ := ma.SplitFirst(m) + return string(first.Bytes()) } func (oas *ObservedAddrSet) SetTTL(ttl time.Duration) { From e76c8a9c388c0b94a1feb126ee8d47844a2d7ebc Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 9 Apr 2019 12:13:30 +0300 Subject: [PATCH 1049/3965] don't eagerly update the observed address set in Addrs allocates and eats a lot of cpu time under the lock; let a background worker gc periodically. --- p2p/protocol/identify/obsaddr.go | 26 +++++--------------------- 1 file changed, 5 insertions(+), 21 deletions(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index af153edc60..374722b1ac 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -70,21 +70,11 @@ func (oas *ObservedAddrSet) AddrsFor(addr ma.Multiaddr) (addrs []ma.Multiaddr) { } now := time.Now() - filteredAddrs := make([]*ObservedAddr, 0, len(observedAddrs)) for _, a := range observedAddrs { - // leave only alive observed addresses - if now.Sub(a.LastSeen) <= oas.ttl { - filteredAddrs = append(filteredAddrs, a) - if a.activated(oas.ttl) { - addrs = append(addrs, a.Addr) - } + if now.Sub(a.LastSeen) <= oas.ttl && a.activated(oas.ttl) { + addrs = append(addrs, a.Addr) } } - if len(filteredAddrs) > 0 { - oas.addrs[key] = filteredAddrs - } else { - delete(oas.addrs, key) - } return addrs } @@ -100,18 +90,12 @@ func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { } now := time.Now() - for local, observedAddrs := range oas.addrs { - filteredAddrs := make([]*ObservedAddr, 0, len(observedAddrs)) + for _, observedAddrs := range oas.addrs { for _, a := range observedAddrs { - // leave only alive observed addresses - if now.Sub(a.LastSeen) <= oas.ttl { - filteredAddrs = append(filteredAddrs, a) - if a.activated(oas.ttl) { - addrs = append(addrs, a.Addr) - } + if now.Sub(a.LastSeen) <= oas.ttl && a.activated(oas.ttl) { + addrs = append(addrs, a.Addr) } } - oas.addrs[local] = filteredAddrs } return addrs } From b8a152f593afe50043c4a12607f1642284aaa8ec Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 9 Apr 2019 12:44:13 +0300 Subject: [PATCH 1050/3965] use a background worker for updating/gcing the observed address set --- p2p/host/basic/basic_host.go | 2 +- p2p/protocol/identify/id.go | 9 +++-- p2p/protocol/identify/obsaddr.go | 65 ++++++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+), 5 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 6127d34001..0725fa702d 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -132,7 +132,7 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, h.ids = opts.IdentifyService } else { // we can't set this as a default above because it depends on the *BasicHost. - h.ids = identify.NewIDService(h) + h.ids = identify.NewIDService(ctx, h) } if uint64(opts.NegotiationTimeout) != 0 { diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 01fc7a7b2c..f2e118b466 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -53,15 +53,16 @@ type IDService struct { // our own observed addresses. // TODO: instead of expiring, remove these when we disconnect - observedAddrs ObservedAddrSet + observedAddrs *ObservedAddrSet } // NewIDService constructs a new *IDService and activates it by // attaching its stream handler to the given host.Host. -func NewIDService(h host.Host) *IDService { +func NewIDService(ctx context.Context, h host.Host) *IDService { s := &IDService{ - Host: h, - currid: make(map[inet.Conn]chan struct{}), + Host: h, + currid: make(map[inet.Conn]chan struct{}), + observedAddrs: NewObservedAddrSet(ctx), } h.SetStreamHandler(ID, s.requestHandler) h.SetStreamHandler(IDPush, s.pushHandler) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 374722b1ac..f693763ab5 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -1,6 +1,7 @@ package identify import ( + "context" "sync" "time" @@ -11,6 +12,8 @@ import ( const ActivationThresh = 4 +var GCInterval = 10 * time.Minute + type observation struct { seenTime time.Time connDirection net.Direction @@ -42,6 +45,11 @@ func (oa *ObservedAddr) activated(ttl time.Duration) bool { return len(oa.SeenBy) >= ActivationThresh } +type newObservation struct { + observed, local, observer ma.Multiaddr + direction net.Direction +} + // ObservedAddrSet keeps track of a set of ObservedAddrs // the zero-value is ready to be used. type ObservedAddrSet struct { @@ -50,6 +58,18 @@ type ObservedAddrSet struct { // local(internal) address -> list of observed(external) addresses addrs map[string][]*ObservedAddr ttl time.Duration + + // this is the worker channel + wch chan newObservation +} + +func NewObservedAddrSet(ctx context.Context) *ObservedAddrSet { + oas := &ObservedAddrSet{ + addrs: make(map[string][]*ObservedAddr), + wch: make(chan newObservation, 1), + } + go oas.worker(ctx) + return oas } // AddrsFor return all activated observed addresses associated with the given @@ -102,6 +122,51 @@ func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, direction net.Direction) { + select { + case oas.wch <- newObservation{observed: observed, local: local, observer: observer, direction: direction}: + default: + log.Debugf("dropping address observation of %s; buffer full", observed) + } +} + +func (oas *ObservedAddrSet) worker(ctx context.Context) { + ticker := time.NewTicker(GCInterval) + defer ticker.Stop() + + for { + select { + case obs := <-oas.wch: + oas.doAdd(obs.observed, obs.local, obs.observer, obs.direction) + + case <-ticker.C: + oas.gc() + + case <-ctx.Done(): + return + } + } +} + +func (oas *ObservedAddrSet) gc() { + oas.Lock() + defer oas.Unlock() + + now := time.Now() + for local, observedAddrs := range oas.addrs { + // TODO we can do this without allocating by compacting the array in place + filteredAddrs := make([]*ObservedAddr, 0, len(observedAddrs)) + for _, a := range observedAddrs { + // leave only alive observed addresses + if now.Sub(a.LastSeen) <= oas.ttl { + filteredAddrs = append(filteredAddrs, a) + } + } + oas.addrs[local] = filteredAddrs + } +} + +func (oas *ObservedAddrSet) doAdd(observed, local, observer ma.Multiaddr, + direction net.Direction) { now := time.Now() observerString := observerGroup(observer) From 9b8192b7d265723e985d1f3ecdef0bee03490dd2 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 9 Apr 2019 13:19:54 +0300 Subject: [PATCH 1051/3965] initialize ttl in ObservedAddrSet --- p2p/protocol/identify/obsaddr.go | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index f693763ab5..5c7e0f99f1 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -66,6 +66,7 @@ type ObservedAddrSet struct { func NewObservedAddrSet(ctx context.Context) *ObservedAddrSet { oas := &ObservedAddrSet{ addrs: make(map[string][]*ObservedAddr), + ttl: pstore.OwnObservedAddrTTL, wch: make(chan newObservation, 1), } go oas.worker(ctx) @@ -78,7 +79,6 @@ func (oas *ObservedAddrSet) AddrsFor(addr ma.Multiaddr) (addrs []ma.Multiaddr) { oas.Lock() defer oas.Unlock() - // for zero-value. if len(oas.addrs) == 0 { return nil } @@ -104,7 +104,6 @@ func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { oas.Lock() defer oas.Unlock() - // for zero-value. if len(oas.addrs) == 0 { return nil } @@ -179,12 +178,6 @@ func (oas *ObservedAddrSet) doAdd(observed, local, observer ma.Multiaddr, oas.Lock() defer oas.Unlock() - // for zero-value. - if oas.addrs == nil { - oas.addrs = make(map[string][]*ObservedAddr) - oas.ttl = pstore.OwnObservedAddrTTL - } - observedAddrs := oas.addrs[localString] // check if observed address seen yet, if so, update it for i, previousObserved := range observedAddrs { From f4af8d493df64fdcb4f0e3d32c9a23686e9924ec Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 9 Apr 2019 13:20:38 +0300 Subject: [PATCH 1052/3965] fix identify tests --- p2p/protocol/identify/id_test.go | 4 ++-- p2p/protocol/identify/obsaddr_test.go | 12 ++++++++++-- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index 621ce1aa0c..fd56f28ca5 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -26,8 +26,8 @@ func subtestIDService(t *testing.T) { h1p := h1.ID() h2p := h2.ID() - ids1 := identify.NewIDService(h1) - ids2 := identify.NewIDService(h2) + ids1 := identify.NewIDService(ctx, h1) + ids2 := identify.NewIDService(ctx, h2) testKnowsAddrs(t, h1, h2p, []ma.Multiaddr{}) // nothing testKnowsAddrs(t, h2, h1p, []ma.Multiaddr{}) // nothing diff --git a/p2p/protocol/identify/obsaddr_test.go b/p2p/protocol/identify/obsaddr_test.go index 12bdb54a8f..440b009b46 100644 --- a/p2p/protocol/identify/obsaddr_test.go +++ b/p2p/protocol/identify/obsaddr_test.go @@ -1,6 +1,7 @@ package identify import ( + "context" "sync" "testing" "time" @@ -52,7 +53,9 @@ func TestObsAddrSet(t *testing.T) { b4 := m("/ip4/1.2.3.9/tcp/1237") b5 := m("/ip4/1.2.3.10/tcp/1237") - oas := &ObservedAddrSet{} + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + oas := NewObservedAddrSet(ctx) if !addrsMarch(oas.Addrs(), nil) { t.Error("addrs should be empty") @@ -63,6 +66,7 @@ func TestObsAddrSet(t *testing.T) { dummyDirection := net.DirOutbound oas.Add(observed, dummyLocal, observer, dummyDirection) + time.Sleep(1 * time.Millisecond) // let the worker run } add(oas, a1, a4) @@ -131,13 +135,17 @@ func TestAddAddrsProfile(b *testing.T) { } return m } - oas := &ObservedAddrSet{} + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + oas := NewObservedAddrSet(ctx) add := func(oas *ObservedAddrSet, observed, observer ma.Multiaddr) { dummyLocal := m("/ip4/127.0.0.1/tcp/10086") dummyDirection := net.DirOutbound oas.Add(observed, dummyLocal, observer, dummyDirection) + time.Sleep(1 * time.Millisecond) // let the worker run } a1 := m("/ip4/1.2.3.4/tcp/1231") From 17001b24ddaed5d4ba59abc9481cc9b1622ef0d9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 9 Apr 2019 13:40:46 +0300 Subject: [PATCH 1053/3965] use a read/write lock for observed address set --- p2p/protocol/identify/obsaddr.go | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 5c7e0f99f1..53a5f9c5b9 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -53,7 +53,7 @@ type newObservation struct { // ObservedAddrSet keeps track of a set of ObservedAddrs // the zero-value is ready to be used. type ObservedAddrSet struct { - sync.Mutex // guards whole datastruct. + sync.RWMutex // guards whole datastruct. // local(internal) address -> list of observed(external) addresses addrs map[string][]*ObservedAddr @@ -76,8 +76,8 @@ func NewObservedAddrSet(ctx context.Context) *ObservedAddrSet { // AddrsFor return all activated observed addresses associated with the given // (resolved) listen address. func (oas *ObservedAddrSet) AddrsFor(addr ma.Multiaddr) (addrs []ma.Multiaddr) { - oas.Lock() - defer oas.Unlock() + oas.RLock() + defer oas.RUnlock() if len(oas.addrs) == 0 { return nil @@ -101,8 +101,8 @@ func (oas *ObservedAddrSet) AddrsFor(addr ma.Multiaddr) (addrs []ma.Multiaddr) { // Addrs return all activated observed addresses func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { - oas.Lock() - defer oas.Unlock() + oas.RLock() + defer oas.RUnlock() if len(oas.addrs) == 0 { return nil @@ -220,11 +220,7 @@ func (oas *ObservedAddrSet) SetTTL(ttl time.Duration) { } func (oas *ObservedAddrSet) TTL() time.Duration { - oas.Lock() - defer oas.Unlock() - // for zero-value. - if oas.addrs == nil { - oas.ttl = pstore.OwnObservedAddrTTL - } + oas.RLock() + defer oas.RUnlock() return oas.ttl } From fe7ba05e38c294f2163d0a2c1ea226712e20201b Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 9 Apr 2019 14:00:03 +0300 Subject: [PATCH 1054/3965] delete empty address sets on observed address set gc --- p2p/protocol/identify/obsaddr.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 53a5f9c5b9..b0c8b19c7b 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -160,7 +160,11 @@ func (oas *ObservedAddrSet) gc() { filteredAddrs = append(filteredAddrs, a) } } - oas.addrs[local] = filteredAddrs + if len(filteredAddrs) > 0 { + oas.addrs[local] = filteredAddrs + } else { + delete(oas.addrs, local) + } } } From 2a4829423604c52680b01903611c0751ec6452d9 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Tue, 9 Apr 2019 11:12:43 -0400 Subject: [PATCH 1055/3965] add discuss badge to readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index ee1e917917..393320fc6f 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ +

From 506082e1bd0d4ab3c6943d75e6b84ced689e35f8 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Tue, 9 Apr 2019 11:58:29 -0400 Subject: [PATCH 1056/3965] update readme badges --- p2p/net/pnet/.#README.md | 1 + 1 file changed, 1 insertion(+) create mode 120000 p2p/net/pnet/.#README.md diff --git a/p2p/net/pnet/.#README.md b/p2p/net/pnet/.#README.md new file mode 120000 index 0000000000..2cb729bed9 --- /dev/null +++ b/p2p/net/pnet/.#README.md @@ -0,0 +1 @@ +yusef@Yusefs-MacBook-Pro-2.local.40096 \ No newline at end of file From 6793f6f3e0b1d7fb5b604dd24f3aac90b4cc2059 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Tue, 9 Apr 2019 11:58:55 -0400 Subject: [PATCH 1057/3965] update readme badges --- p2p/net/pnet/.#README.md | 1 - 1 file changed, 1 deletion(-) delete mode 120000 p2p/net/pnet/.#README.md diff --git a/p2p/net/pnet/.#README.md b/p2p/net/pnet/.#README.md deleted file mode 120000 index 2cb729bed9..0000000000 --- a/p2p/net/pnet/.#README.md +++ /dev/null @@ -1 +0,0 @@ -yusef@Yusefs-MacBook-Pro-2.local.40096 \ No newline at end of file From efdc140264ee2fa5deefbf572700a0dab1e07241 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 9 Apr 2019 20:35:36 +0300 Subject: [PATCH 1058/3965] raise activation channel capacity to 16 --- p2p/protocol/identify/obsaddr.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index b0c8b19c7b..b568ba5055 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -67,7 +67,7 @@ func NewObservedAddrSet(ctx context.Context) *ObservedAddrSet { oas := &ObservedAddrSet{ addrs: make(map[string][]*ObservedAddr), ttl: pstore.OwnObservedAddrTTL, - wch: make(chan newObservation, 1), + wch: make(chan newObservation, 16), } go oas.worker(ctx) return oas From a4776e6174b4c4a1d67138785be4c382aadb5406 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 9 Apr 2019 21:52:40 +0300 Subject: [PATCH 1059/3965] fix panic in observed address activation check The activated check cleans up, which leads to panics with concurrent reads. This moves the seenBy set clean up into gc, where it belongs --- p2p/protocol/identify/obsaddr.go | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index b568ba5055..685e5f10b5 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -31,15 +31,6 @@ type ObservedAddr struct { } func (oa *ObservedAddr) activated(ttl time.Duration) bool { - // cleanup SeenBy set - now := time.Now() - - for k, ob := range oa.SeenBy { - if now.Sub(ob.seenTime) > ttl*ActivationThresh { - delete(oa.SeenBy, k) - } - } - // We only activate if in the TTL other peers observed the same address // of ours at least 4 times. return len(oa.SeenBy) >= ActivationThresh @@ -154,7 +145,15 @@ func (oas *ObservedAddrSet) gc() { for local, observedAddrs := range oas.addrs { // TODO we can do this without allocating by compacting the array in place filteredAddrs := make([]*ObservedAddr, 0, len(observedAddrs)) + for _, a := range observedAddrs { + // clean up SeenBy set + for k, ob := range a.SeenBy { + if now.Sub(ob.seenTime) > oas.ttl*ActivationThresh { + delete(a.SeenBy, k) + } + } + // leave only alive observed addresses if now.Sub(a.LastSeen) <= oas.ttl { filteredAddrs = append(filteredAddrs, a) From 17bfd15452f61430f8b30e8fa1455fcef4d0dd1b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 9 Apr 2019 15:22:13 -0700 Subject: [PATCH 1060/3965] full close the autonat stream This is ridiculous but we need to fix the interfaces... --- p2p/host/autonat/svc.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index bd704beb44..febae7a407 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -58,7 +58,7 @@ func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) } func (as *AutoNATService) handleStream(s inet.Stream) { - defer s.Close() + defer inet.FullClose(s) pid := s.Conn().RemotePeer() log.Debugf("New stream from %s", pid.Pretty()) From 745284da235056b6d0efb16cce9744a2dfe90901 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 9 Apr 2019 15:23:57 -0700 Subject: [PATCH 1061/3965] fully close the autonat client stream The "correct" way to do this is to send an write, close, read, then read the EOF. However, we don't really do that _anywhere_ in our code. --- p2p/host/autonat/client.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index 0fd665d3df..aa3075e4c9 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -49,7 +49,9 @@ func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) if err != nil { return nil, err } - defer s.Close() + // Might as well just reset the stream. Once we get to this point, we + // don't care about being nice. + defer inet.FullClose(s) r := ggio.NewDelimitedReader(s, inet.MessageSizeMax) w := ggio.NewDelimitedWriter(s) @@ -57,12 +59,14 @@ func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) req := newDialMessage(pstore.PeerInfo{ID: c.h.ID(), Addrs: c.getAddrs()}) err = w.WriteMsg(req) if err != nil { + s.Reset() return nil, err } var res pb.Message err = r.ReadMsg(&res) if err != nil { + s.Reset() return nil, err } From 2bd14ad185defc626d11e2a28513e5426bd5b67f Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Apr 2019 10:42:16 +0300 Subject: [PATCH 1062/3965] update go-libp2p-autonat --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 93d39af313..67df3c17ef 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,7 @@ require ( github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 github.com/libp2p/go-conn-security v0.0.1 github.com/libp2p/go-conn-security-multistream v0.0.1 - github.com/libp2p/go-libp2p-autonat v0.0.3 + github.com/libp2p/go-libp2p-autonat v0.0.4 github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.2 github.com/libp2p/go-libp2p-crypto v0.0.1 diff --git a/go.sum b/go.sum index 3ebe0a2998..afabbc0791 100644 --- a/go.sum +++ b/go.sum @@ -84,8 +84,8 @@ github.com/libp2p/go-conn-security-multistream v0.0.1 h1:XefjAQRHcnUaxKb26RGupTo github.com/libp2p/go-conn-security-multistream v0.0.1/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= -github.com/libp2p/go-libp2p-autonat v0.0.3 h1:PUD+pAx8Qs9hh+Bowzxq8RCkg/Vwrz5oCFC4peixXQk= -github.com/libp2p/go-libp2p-autonat v0.0.3/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= +github.com/libp2p/go-libp2p-autonat v0.0.4 h1:cZzdB9KW1ZkHnSjLCB6aFNw47XS4r+SecCVMuVB1xgo= +github.com/libp2p/go-libp2p-autonat v0.0.4/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= github.com/libp2p/go-libp2p-circuit v0.0.2 h1:wwbYslVw1aW2DAAuJxXRS5gB2clz1s23gv9NOb8iwsE= From 02d01da08cad077ae1518c894d403fe32e17e409 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Apr 2019 20:24:32 +0300 Subject: [PATCH 1063/3965] context option to disable dialing when opening a new stream --- p2p/net/swarm/swarm.go | 16 ++++++++++++++++ p2p/net/swarm/swarm_test.go | 11 +++++++++++ 2 files changed, 27 insertions(+) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index cc77e80ec6..c4661d4b33 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -37,6 +37,18 @@ var ErrSwarmClosed = errors.New("swarm closed") // transport is misbehaving. var ErrAddrFiltered = errors.New("address filtered") +// ErrNoConn is returned when attempting to open a stream to a peer with the NoDial +// option and no usable connection is available. +var ErrNoConn = errors.New("no usable connection to peer") + +// ContextOption is the type of context options understood by the swarm +type ContextOption string + +// NoDial is a context option that instructs the swarm to not attempt a new +// dial when opening a stream. The value of the key should be a string indicating +// the source of the option. +var NoDial = ContextOption("swarm.NoDial") + // Swarm is a connection muxer, allowing connections to other peers to // be opened and closed, while still using the same Chan for all // communication. The Chan sends/receives Messages, which note the @@ -295,6 +307,10 @@ func (s *Swarm) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { for { c := s.bestConnToPeer(p) if c == nil { + if nodial := ctx.Value(NoDial); nodial != nil { + return nil, ErrNoConn + } + if dials >= DialAttempts { return nil, errors.New("max dial attempts exceeded") } diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 4911fc96a1..aa5abeecf5 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -50,6 +50,7 @@ func EchoStreamHandler(stream inet.Stream) { return } } + }() } @@ -335,3 +336,13 @@ func TestFilterBounds(t *testing.T) { t.Log("got connect") } } + +func TestNoDial(t *testing.T) { + ctx := context.Background() + swarms := makeSwarms(ctx, t, 2) + + _, err := swarms[0].NewStream(context.WithValue(ctx, NoDial, "swarm.test"), swarms[1].LocalPeer()) + if err != ErrNoConn { + t.Fatal("should have failed with ErrNoConn") + } +} From b355bce1630d240bd72c020b72028fce7e424bf2 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Apr 2019 21:46:51 +0300 Subject: [PATCH 1064/3965] update to use NoDial option from go-libp2p-net --- p2p/net/swarm/swarm.go | 16 ++-------------- p2p/net/swarm/swarm_test.go | 4 ++-- 2 files changed, 4 insertions(+), 16 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index c4661d4b33..1094e72e19 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -37,18 +37,6 @@ var ErrSwarmClosed = errors.New("swarm closed") // transport is misbehaving. var ErrAddrFiltered = errors.New("address filtered") -// ErrNoConn is returned when attempting to open a stream to a peer with the NoDial -// option and no usable connection is available. -var ErrNoConn = errors.New("no usable connection to peer") - -// ContextOption is the type of context options understood by the swarm -type ContextOption string - -// NoDial is a context option that instructs the swarm to not attempt a new -// dial when opening a stream. The value of the key should be a string indicating -// the source of the option. -var NoDial = ContextOption("swarm.NoDial") - // Swarm is a connection muxer, allowing connections to other peers to // be opened and closed, while still using the same Chan for all // communication. The Chan sends/receives Messages, which note the @@ -307,8 +295,8 @@ func (s *Swarm) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { for { c := s.bestConnToPeer(p) if c == nil { - if nodial := ctx.Value(NoDial); nodial != nil { - return nil, ErrNoConn + if nodial, _ := inet.GetNoDial(ctx); nodial { + return nil, inet.ErrNoConn } if dials >= DialAttempts { diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index aa5abeecf5..89bf909295 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -341,8 +341,8 @@ func TestNoDial(t *testing.T) { ctx := context.Background() swarms := makeSwarms(ctx, t, 2) - _, err := swarms[0].NewStream(context.WithValue(ctx, NoDial, "swarm.test"), swarms[1].LocalPeer()) - if err != ErrNoConn { + _, err := swarms[0].NewStream(inet.WithNoDial(ctx, "swarm test"), swarms[1].LocalPeer()) + if err != inet.ErrNoConn { t.Fatal("should have failed with ErrNoConn") } } From fb0dd0ed76b768836be2d5a8286291bc375769e9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Apr 2019 21:48:47 +0300 Subject: [PATCH 1065/3965] kill stray line --- p2p/net/swarm/swarm_test.go | 1 - 1 file changed, 1 deletion(-) diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 89bf909295..69eceb94c4 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -50,7 +50,6 @@ func EchoStreamHandler(stream inet.Stream) { return } } - }() } From f39355f319f688ab9fd09e3f8583691afa2d8acb Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Apr 2019 22:41:36 +0300 Subject: [PATCH 1066/3965] use NoDial option for opening streams in non active relays It turns out that the relays are doing dialing on NewStream in the wild, presumably because they still have dead connections. This uses the new NoDial option to instruct the swarm to not dial the hop target instead of explicitly checking for open connections and is robust to failed connections. --- .../internal/circuitv1-deprecated/relay.go | 21 +++++++++---------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index ddda926140..ec311104d2 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -300,24 +300,23 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { } // open stream - ctp := r.host.Network().ConnsToPeer(dst.ID) - - if len(ctp) == 0 && !r.active { - r.handleError(s, pb.CircuitRelay_HOP_NO_CONN_TO_DST) - return - } + ctx, cancel := context.WithTimeout(r.ctx, HopConnectTimeout) + defer cancel() - if len(dst.Addrs) > 0 { + if !r.active { + ctx = inet.WithNoDial(ctx, "relay hop") + } else if len(dst.Addrs) > 0 { r.host.Peerstore().AddAddrs(dst.ID, dst.Addrs, pstore.TempAddrTTL) } - ctx, cancel := context.WithTimeout(r.ctx, HopConnectTimeout) - defer cancel() - bs, err := r.host.NewStream(ctx, dst.ID, ProtoID) if err != nil { log.Debugf("error opening relay stream to %s: %s", dst.ID.Pretty(), err.Error()) - r.handleError(s, pb.CircuitRelay_HOP_CANT_DIAL_DST) + if err == inet.ErrNoConn { + r.handleError(s, pb.CircuitRelay_HOP_NO_CONN_TO_DST) + } else { + r.handleError(s, pb.CircuitRelay_HOP_CANT_DIAL_DST) + } return } From 25adea6d568aa59ac67c818436263ca96fc8eba7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Apr 2019 23:09:17 +0300 Subject: [PATCH 1067/3965] gomod: update go-libp2p-net, go-libp2p-swarm, and go-libp2p-circuit --- go.mod | 6 +++--- go.sum | 10 ++++++++-- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/go.mod b/go.mod index 67df3c17ef..cd4fd9ce55 100644 --- a/go.mod +++ b/go.mod @@ -11,7 +11,7 @@ require ( github.com/libp2p/go-conn-security-multistream v0.0.1 github.com/libp2p/go-libp2p-autonat v0.0.4 github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.2 + github.com/libp2p/go-libp2p-circuit v0.0.3 github.com/libp2p/go-libp2p-crypto v0.0.1 github.com/libp2p/go-libp2p-discovery v0.0.1 github.com/libp2p/go-libp2p-host v0.0.1 @@ -20,14 +20,14 @@ require ( github.com/libp2p/go-libp2p-loggables v0.0.1 github.com/libp2p/go-libp2p-metrics v0.0.1 github.com/libp2p/go-libp2p-nat v0.0.2 - github.com/libp2p/go-libp2p-net v0.0.1 + github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.0.1 github.com/libp2p/go-libp2p-peerstore v0.0.1 github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 github.com/libp2p/go-libp2p-secio v0.0.1 - github.com/libp2p/go-libp2p-swarm v0.0.1 + github.com/libp2p/go-libp2p-swarm v0.0.2 github.com/libp2p/go-libp2p-transport v0.0.4 github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 github.com/libp2p/go-maddr-filter v0.0.1 diff --git a/go.sum b/go.sum index afabbc0791..7a6ae33f09 100644 --- a/go.sum +++ b/go.sum @@ -37,6 +37,8 @@ github.com/gxed/hashland/keccakpg v0.0.1 h1:wrk3uMNaMxbXiHibbPO4S0ymqJMm41WiudyF github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= github.com/gxed/hashland/murmur3 v0.0.1 h1:SheiaIt0sda5K+8FLz952/1iWS9zrnKsEJaOJu4ZbSc= github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= +github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= @@ -88,8 +90,8 @@ github.com/libp2p/go-libp2p-autonat v0.0.4 h1:cZzdB9KW1ZkHnSjLCB6aFNw47XS4r+SecC github.com/libp2p/go-libp2p-autonat v0.0.4/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.2 h1:wwbYslVw1aW2DAAuJxXRS5gB2clz1s23gv9NOb8iwsE= -github.com/libp2p/go-libp2p-circuit v0.0.2/go.mod h1:zV136p4UQ76qH/Wj+X/Hivcg6sf6Yb1G7YL8o+GGj38= +github.com/libp2p/go-libp2p-circuit v0.0.3 h1:FKhhuMOegdNIEbXkZsClEiqXJr+NmR+REmQRiMgtSyA= +github.com/libp2p/go-libp2p-circuit v0.0.3/go.mod h1:p1cHJnB9xnX5/1vZLkXgKwmNEOQQuF/Hp+SkATXnXYk= github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-discovery v0.0.1 h1:VkjCKmJQMwpDUwtA8Qc1z3TQAHJgQ5nGQ6cdN0wQXOw= @@ -108,6 +110,8 @@ github.com/libp2p/go-libp2p-nat v0.0.2 h1:sKI5hiCsGFhuEKdXMsF9mywQu2qhfoIGX6a+VG github.com/libp2p/go-libp2p-nat v0.0.2/go.mod h1:QrjXQSD5Dj4IJOdEcjHRkWTSomyxRo6HnUkf/TfQpLQ= github.com/libp2p/go-libp2p-net v0.0.1 h1:xJ4Vh4yKF/XKb8fd1Ev0ebAGzVjMxXzrxG2kjtU+F5Q= github.com/libp2p/go-libp2p-net v0.0.1/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= +github.com/libp2p/go-libp2p-net v0.0.2 h1:qP06u4TYXfl7uW/hzqPhlVVTSA2nw1B/bHBJaUnbh6M= +github.com/libp2p/go-libp2p-net v0.0.2/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= github.com/libp2p/go-libp2p-netutil v0.0.1 h1:LgD6+skofkOx8z6odD9+MZHKjupv3ng1u6KRhaADTnA= github.com/libp2p/go-libp2p-netutil v0.0.1/go.mod h1:GdusFvujWZI9Vt0X5BKqwWWmZFxecf9Gt03cKxm2f/Q= github.com/libp2p/go-libp2p-peer v0.0.1 h1:0qwAOljzYewINrU+Kndoc+1jAL7vzY/oY2Go4DCGfyY= @@ -122,6 +126,8 @@ github.com/libp2p/go-libp2p-secio v0.0.1 h1:CqE/RdsizOwItdgLe632iyft/w0tshDLmZGA github.com/libp2p/go-libp2p-secio v0.0.1/go.mod h1:IdG6iQybdcYmbTzxp4J5dwtUEDTOvZrT0opIDVNPrJs= github.com/libp2p/go-libp2p-swarm v0.0.1 h1:Vne+hjaDwXqzgNwQ2vb2YKbnbOTyXjtS47stT66Apc4= github.com/libp2p/go-libp2p-swarm v0.0.1/go.mod h1:mh+KZxkbd3lQnveQ3j2q60BM1Cw2mX36XXQqwfPOShs= +github.com/libp2p/go-libp2p-swarm v0.0.2 h1:cpHHXTeU2IgUu8LPemF7vaLPGtVC6VxMoll2EwqlC+E= +github.com/libp2p/go-libp2p-swarm v0.0.2/go.mod h1:n0cAAcKyndIrJWctQwjqXlAdIPBZzfdpBjx1SSvz30g= github.com/libp2p/go-libp2p-transport v0.0.1/go.mod h1:UzbUs9X+PHOSw7S3ZmeOxfnwaQY5vGDzZmKPod3N3tk= github.com/libp2p/go-libp2p-transport v0.0.4 h1:/CPHQMN75/IQwkhBxxIo6p6PtL3rwFZtlzBROT3e8mw= github.com/libp2p/go-libp2p-transport v0.0.4/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= From 1e48e15da7fb91a058fadc0298239a2eae7cdfd0 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Apr 2019 23:12:14 +0300 Subject: [PATCH 1068/3965] respect NoDial option in routed host --- p2p/host/routed/routed.go | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/p2p/host/routed/routed.go b/p2p/host/routed/routed.go index d921536895..e5b2e40d14 100644 --- a/p2p/host/routed/routed.go +++ b/p2p/host/routed/routed.go @@ -169,9 +169,12 @@ func (rh *RoutedHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol // Ensure we have a connection, with peer addresses resolved by the routing system (#207) // It is not sufficient to let the underlying host connect, it will most likely not have // any addresses for the peer without any prior connections. - err := rh.Connect(ctx, pstore.PeerInfo{ID: p}) - if err != nil { - return nil, err + // If the caller wants to prevent the host from dialing, it should use the NoDial option. + if nodial, _ := inet.GetNoDial(ctx); !nodial { + err := rh.Connect(ctx, pstore.PeerInfo{ID: p}) + if err != nil { + return nil, err + } } return rh.host.NewStream(ctx, p, pids...) From e4061bbb878caa3cf928e1b608b2bdd003d62196 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 10 Apr 2019 23:12:43 +0300 Subject: [PATCH 1069/3965] travis: remove gx build --- .travis.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 4cfe98c242..5163d693fc 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,7 +10,6 @@ env: global: - GOTFLAGS="-race" matrix: - - BUILD_DEPTYPE=gx - BUILD_DEPTYPE=gomod @@ -24,7 +23,6 @@ script: cache: directories: - - $GOPATH/src/gx - $GOPATH/pkg/mod - $HOME/.cache/go-build From e38641febf66dceea39f0a8e52d49533f25a03cb Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 11 Apr 2019 11:48:03 +0300 Subject: [PATCH 1070/3965] use buffer pool in newDelimitedReader instead of allocating the buffer flat --- p2p/protocol/internal/circuitv1-deprecated/util.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index 5812dbad72..e1fa3dbe9a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -9,6 +9,7 @@ import ( ggio "github.com/gogo/protobuf/io" proto "github.com/gogo/protobuf/proto" + pool "github.com/libp2p/go-buffer-pool" peer "github.com/libp2p/go-libp2p-peer" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" @@ -64,7 +65,7 @@ type delimitedReader struct { // - messages are small (max 4k) and the length fits in a couple of bytes, // so overall we have at most three reads per message. func newDelimitedReader(r io.Reader, maxSize int) *delimitedReader { - return &delimitedReader{r: r, buf: make([]byte, maxSize)} + return &delimitedReader{r: r, buf: pool.Get(maxSize)} } func (d *delimitedReader) ReadByte() (byte, error) { From 50c453c28c02fedca2a55e5d2021e671500280b0 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 11 Apr 2019 11:59:53 +0300 Subject: [PATCH 1071/3965] put back the buffer in the pool --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 1 + p2p/protocol/internal/circuitv1-deprecated/util.go | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index ec311104d2..af6b11587c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -323,6 +323,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { // stop handshake rd := newDelimitedReader(bs, maxMessageSize) wr := newDelimitedWriter(bs) + defer rd.Close() msg.Type = pb.CircuitRelay_STOP.Enum() diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index e1fa3dbe9a..7d56b89503 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -68,6 +68,13 @@ func newDelimitedReader(r io.Reader, maxSize int) *delimitedReader { return &delimitedReader{r: r, buf: pool.Get(maxSize)} } +func (d *delimitedReader) Close() { + if d.buf != nil { + pool.Put(d.buf) + d.buf = nil + } +} + func (d *delimitedReader) ReadByte() (byte, error) { buf := d.buf[:1] _, err := d.r.Read(buf) From 92c514ce8611d3ae9969db03119375b90614eab7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 11 Apr 2019 13:01:43 +0300 Subject: [PATCH 1072/3965] close the delimited reader everywhere --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index af6b11587c..b15841b29e 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -170,6 +170,7 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore rd := newDelimitedReader(s, maxMessageSize) wr := newDelimitedWriter(s) + defer rd.Close() var msg pb.CircuitRelay @@ -218,6 +219,7 @@ func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { rd := newDelimitedReader(s, maxMessageSize) wr := newDelimitedWriter(s) + defer rd.Close() var msg pb.CircuitRelay @@ -249,6 +251,7 @@ func (r *Relay) handleNewStream(s inet.Stream) { log.Infof("new relay stream from: %s", s.Conn().RemotePeer()) rd := newDelimitedReader(s, maxMessageSize) + defer rd.Close() var msg pb.CircuitRelay From d092ebaf218cad98c087aaf24bae8a7461196295 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 11 Apr 2019 10:23:32 -0400 Subject: [PATCH 1073/3965] deprecate gx in readme, link to workspace This adds a link to the workspace-go-libp2p repo and some context on what it's used for. It also removes gx-related instructions and adds a deprecation notice about gx, encouraging users to migrate. --- README.md | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 393320fc6f..02559c98b4 100644 --- a/README.md +++ b/README.md @@ -83,29 +83,31 @@ Examples can be found in the [examples repo](https://github.com/libp2p/go-libp2p ## Development -### Dependencies +### Using the libp2p workspace -While developing, you need to use [gx to install and link your dependencies](https://github.com/whyrusleeping/gx#dependencies), to do that, run: +While developing, you may need to make changes to several modules at once, or you may want changes made locally in one module to be available for import by another. -```sh -> make deps -``` +The [go libp2p workspace](https://github.com/libp2p/workspace-go-libp2p) provides a developer-oriented view of the modules that comprise go-libp2p. -Before commiting and pushing to Github, make sure to rewind the gx'ify of dependencies. You can do that with: +Using the tooling in the workspace repository, you can checkout all of go-libp2p's module repos and enter "local mode", which adds [replace directives](https://github.com/golang/go/wiki/Modules#gomod) to the go.mod files in each local working copy. When you build locally, the libp2p depdendencies will be resolved from your local working copies. -```sh -> make publish -``` +Once you've committed your changes, you can switch back to "remote mode", which removes the replace directives and pulls imports from the main go module cache. -### Tests +See the [workspace repo](https://github.com/libp2p/workspace-go-libp2p) for more information. -Running of individual tests is done through `gx test ` +### About gx -```bash -$ cd $GOPATH/src/github.com/libp2p/go-libp2p -$ make deps -$ gx test ./p2p/ -``` +Before adopting gomod, libp2p used [gx](https://github.com/whyrusleeping/gx) to manage dependencies using [IPFS](https://ipfs.io). + +Due to the difficulties in keeping both dependency management solutions up-to-date, gx support was ended in April 2019. + +Ending gx support does not mean that existing gx builds will break. Because gx references dependencies by their immutable IPFS hash, any currently working gx builds will continue to work for as long as the dependencies are resolvable in IPFS. + +However, new changes to go-libp2p will not be published via gx, and users are encouraged to adopt gomod to stay up-to-date. + +### Tests + +`go test ./...` will run all tests in the repo. ### Packages From c331b52d71e8046abffafb948415488698b0d254 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 11 Apr 2019 10:28:02 -0400 Subject: [PATCH 1074/3965] Add link to discuss topic about gx migration --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 02559c98b4..5d04cb125c 100644 --- a/README.md +++ b/README.md @@ -105,6 +105,8 @@ Ending gx support does not mean that existing gx builds will break. Because gx r However, new changes to go-libp2p will not be published via gx, and users are encouraged to adopt gomod to stay up-to-date. +If you experience any issues migrating from gx to gomod, please [join the discussion at the libp2p forums](https://discuss.libp2p.io/t/gomod-and-go-libp2p/44). + ### Tests `go test ./...` will run all tests in the repo. From 51621c18bd3f5e186504ff3754ef8eb97ece9f60 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 11 Apr 2019 11:48:26 -0400 Subject: [PATCH 1075/3965] fix TOC links --- README.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 5d04cb125c..90e7b250be 100644 --- a/README.md +++ b/README.md @@ -31,16 +31,15 @@ # Table of Contents - [Background](#background) -- [Bundles](#bundles) - [Usage](#usage) - - [Install](#install) - [API](#api) - [Examples](#examples) - [Development](#development) + - [Using the libp2p Workspace](#using-the-libp2p-workspace) + - [About gx](#about-gx) - [Tests](#tests) - [Packages](#packages) - [Contribute](#contribute) -- [License](#license) ## Background @@ -83,7 +82,7 @@ Examples can be found in the [examples repo](https://github.com/libp2p/go-libp2p ## Development -### Using the libp2p workspace +### Using the libp2p Workspace While developing, you may need to make changes to several modules at once, or you may want changes made locally in one module to be available for import by another. From 29e5ff48bbda3898173a82cfa6ad2eb334a5945f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 11 Apr 2019 21:11:15 +0100 Subject: [PATCH 1076/3965] farewell gx; thanks for serving us well. --- .gx/lastpubver | 1 - Makefile | 10 -- README.md | 4 + package.json | 254 ------------------------------------------------- 4 files changed, 4 insertions(+), 265 deletions(-) delete mode 100644 .gx/lastpubver delete mode 100644 Makefile delete mode 100644 package.json diff --git a/.gx/lastpubver b/.gx/lastpubver deleted file mode 100644 index 9719aa2344..0000000000 --- a/.gx/lastpubver +++ /dev/null @@ -1 +0,0 @@ -6.0.41: QmTRN7hRxvGkxKxDdeudty7sRet4L7ZKZCqKsXHa79wmAc diff --git a/Makefile b/Makefile deleted file mode 100644 index f9dabde9b2..0000000000 --- a/Makefile +++ /dev/null @@ -1,10 +0,0 @@ -gx: - go get github.com/whyrusleeping/gx - go get github.com/whyrusleeping/gx-go - -deps: gx - gx --verbose install --global - gx-go rewrite - -publish: - gx-go rewrite --undo diff --git a/README.md b/README.md index 90e7b250be..47ea441d5f 100644 --- a/README.md +++ b/README.md @@ -208,3 +208,7 @@ There's a few things you can do right now to help out: - Go through the modules below and **check out existing issues**. This would be especially useful for modules in active development. Some knowledge of IPFS/libp2p may be required, as well as the infrasture behind it - for instance, you may need to read up on p2p and more complex operations like muxing to be able to help technically. - **Perform code reviews**. - **Add tests**. There can never be enough tests. + +--- + +The last gx published version of this module was: 6.0.41: QmTRN7hRxvGkxKxDdeudty7sRet4L7ZKZCqKsXHa79wmAc diff --git a/package.json b/package.json deleted file mode 100644 index 97265d44be..0000000000 --- a/package.json +++ /dev/null @@ -1,254 +0,0 @@ -{ - "author": "whyrusleeping", - "bugs": { - "url": "https://github.com/libp2p/go-libp2p" - }, - "gx": { - "dvcsimport": "github.com/libp2p/go-libp2p", - "goversion": "1.5.2" - }, - "gxDependencies": [ - { - "hash": "QmekaTKpWkYGcn4ZEC5PwJDRCQHapwugmmG86g2Xpz5GBH", - "name": "mdns", - "version": "0.2.0" - }, - { - "hash": "QmNohiVssaPw3KVLZik59DBVGTSm2dGvYT9eoXt5DQ36Yz", - "name": "go-ipfs-util", - "version": "1.2.9" - }, - { - "hash": "QmabLh8TrJ3emfAoQk5AbqbLTbMyj7XqumMFmAFxa9epo8", - "name": "go-multistream", - "version": "0.3.9" - }, - { - "hash": "Qmf7HqcW7LtCi1W8y2bdx2eJpze74jkbKqpByxgXikdbLF", - "name": "go-detect-race", - "version": "1.0.1" - }, - { - "hash": "QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP", - "name": "goprocess", - "version": "1.0.0" - }, - { - "hash": "QmbkT7eMTyXfpeyB3ZMxxcxg7XH8t6uXp49jqzz4HB7BGF", - "name": "go-log", - "version": "1.5.9" - }, - { - "hash": "Qmc85NSvmSG4Frn9Vb2cBc1rMyULH6D3TNVEfCzSKoUpip", - "name": "go-multiaddr-net", - "version": "1.7.2" - }, - { - "hash": "QmTZBfrPJmjWsCvHEtX5FE6KimVJhsJg5sBbqEFYf4UZtL", - "name": "go-multiaddr", - "version": "1.4.1" - }, - { - "author": "whyrusleeping", - "hash": "QmUbSLukzZYZvEYxynj9Dtd1WrGLxxg9R4U68vCMPWHmRU", - "name": "go-libp2p-loggables", - "version": "1.1.33" - }, - { - "author": "whyrusleeping", - "hash": "QmSVaJe1aRjc78cZARTtf4pqvXERYwihyYhZWoVWceHnsK", - "name": "go-libp2p-secio", - "version": "2.0.30" - }, - { - "author": "whyrusleeping", - "hash": "QmaCTz9RkrU13bm9kMB54f7atgqM4qkjDZpRwRoJiWXEqs", - "name": "go-libp2p-peerstore", - "version": "2.0.19" - }, - { - "author": "whyrusleeping", - "hash": "QmNQWMWWBmkAcaVEspSNwYB95axzKFhYTdqZtABA2zXoPu", - "name": "go-libp2p-transport", - "version": "3.0.27" - }, - { - "author": "whyrusleeping", - "hash": "QmTGiDkw4eeKq31wwpQRk5GwWiReaxrcTQLuCCLWgfKo5M", - "name": "go-tcp-transport", - "version": "2.0.28" - }, - { - "author": "whyrusleeping", - "hash": "QmT6C5ebDy92zyRzdmSNyda5q7zkNXy68X47RDJiHpvaxd", - "name": "go-maddr-filter", - "version": "1.1.13" - }, - { - "author": "whyrusleeping", - "hash": "QmZNkThpqfVXs9GNbexPrfBbXSLNYeKrE7jwFM2oqHbyqN", - "name": "go-libp2p-protocol", - "version": "1.0.0" - }, - { - "author": "whyrusleeping", - "hash": "QmWapVoHjtKhn4MhvKNoPTkJKADFGACfXPFnt7combwp5W", - "name": "go-testutil", - "version": "1.2.19" - }, - { - "author": "whyrusleeping", - "hash": "QmY3ArotKMKaL7YGfbQfyDrib6RVraLqZYWXZvVgZktBxp", - "name": "go-libp2p-net", - "version": "3.0.30" - }, - { - "author": "whyrusleeping", - "hash": "QmSwVwKUWzdf3ppM3FbBbpuqHUNtUFJPQQdfvKmgZoz2gR", - "name": "go-libp2p-metrics", - "version": "2.1.14" - }, - { - "author": "whyrusleeping", - "hash": "QmYrWiWM4qtrnCeT3R14jY3ZZyirDNJgwK57q4qFYePgbd", - "name": "go-libp2p-host", - "version": "3.0.26" - }, - { - "author": "whyrusleeping", - "hash": "QmQVoMEL1CxrVusTSUdYsiJXVBnvSqNUpBsGybkwSfksEF", - "name": "go-libp2p-swarm", - "version": "3.0.35" - }, - { - "author": "whyrusleeping", - "hash": "QmRbx7DYHgw3uNn2RuU2nv9Bdh96ZdtT65CG1CGPNRQcGZ", - "name": "go-libp2p-nat", - "version": "0.8.13" - }, - { - "author": "whyrusleeping", - "hash": "QmQzyPDx8rLqJ5evnHAo4Mpbsb64RKGZzLABzsbXQM6a2j", - "name": "go-libp2p-netutil", - "version": "0.4.23" - }, - { - "author": "whyrusleeping", - "hash": "QmcBTHN7uAMBdkzRoQ3n9cE7tGu8Ubd9zmahjskjTRw4Uf", - "name": "go-libp2p-blankhost", - "version": "0.3.26" - }, - { - "author": "whyrusleeping", - "hash": "QmTW4SdgBWq9GjsBsHeUx8WuGxzhgzAf88UMH2w62PC8yK", - "name": "go-libp2p-crypto", - "version": "2.0.7" - }, - { - "author": "whyrusleeping", - "hash": "QmcqwxmCsWjNvQFzeZ4rzYVycWDY8C7Ejtiw2G3djArQzf", - "name": "go-smux-yamux", - "version": "2.0.10" - }, - { - "author": "whyrusleeping", - "hash": "QmUWkEreTZdTxUVDfpQ2fLywJh6dinfEYahEoBDTMQ2hks", - "name": "go-smux-multistream", - "version": "2.0.3" - }, - { - "author": "whyrusleeping", - "hash": "QmYVXrKrKHDC9FobgmcmshCDyWwdrfwfanNQN4oxJ9Fk3h", - "name": "go-libp2p-peer", - "version": "3.1.2" - }, - { - "author": "vyzo", - "hash": "QmRTkLxADQRbgnhpt2zzQQJr8Ri764b7dujoDkZw33b3iE", - "name": "go-libp2p-circuit", - "version": "2.3.15" - }, - { - "author": "lgierth", - "hash": "QmU98UaAEh4WJAcir2qjfztU77JQ14kAwHNFkjUXHZA3Vy", - "name": "go-multiaddr-dns", - "version": "0.3.1" - }, - { - "author": "why", - "hash": "QmXa6sgzUvP5bgF5CyyV36bZYv5VDRwttggQYUPvFybLVd", - "name": "go-libp2p-interface-connmgr", - "version": "0.0.32" - }, - { - "author": "whyrusleeping", - "hash": "QmaJvNdDccVkTELQLCGXWrLxgaQ14aMdhzZx1EiHPXKbDc", - "name": "go-smux-multiplex", - "version": "3.0.19" - }, - { - "author": "whyrusleeping", - "hash": "QmaSWc4ox6SZQF6DHZvDuM9sP1syNajkKuPXmKR1t5BAz5", - "name": "go-ws-transport", - "version": "2.0.27" - }, - { - "author": "stebalien", - "hash": "QmZWmFkMm28sWeDr5Xh1LexdKBGYGp946MNCfgtLqfX73z", - "name": "go-conn-security-multistream", - "version": "0.1.26" - }, - { - "author": "Stebalien", - "hash": "QmPRoHqULmP4MuKAN5EFaJ64MLpeMY8cny2318xDBDmmkp", - "name": "go-conn-security", - "version": "0.1.28" - }, - { - "author": "libp2p", - "hash": "QmW7Ump7YyBMr712Ta3iEVh3ZYcfVvJaPryfbCnyE826b4", - "name": "go-libp2p-interface-pnet", - "version": "3.0.0" - }, - { - "author": "whyrusleeping", - "hash": "QmVtV1y2e8W4eQgzsP6qfSpCCZ6zWYE4m6NzJjB7iswwrT", - "name": "go-stream-muxer", - "version": "3.1.0" - }, - { - "author": "steb", - "hash": "QmeqC5shQjEBRG9B8roZqQCJ9xb7Pq6AbWxJFMyLgqBBWh", - "name": "go-libp2p-transport-upgrader", - "version": "0.1.28" - }, - { - "hash": "QmddjPSGZb3ieihSseFeCfVRpZzcqczPNsD2DvarSwnjJB", - "name": "gogo-protobuf", - "version": "1.2.1" - }, - { - "author": "vyzo", - "hash": "QmWA8k8apx6egshEjemkuxpKNJS7W7heCgzTnBhAvX9yoB", - "name": "go-libp2p-discovery", - "version": "1.0.15" - }, - { - "author": "vyzo", - "hash": "QmNnJhrc4ZtcPZg3oXDVdJ7HRtxDfLE2ccH56cZZjZ6y3p", - "name": "go-libp2p-autonat", - "version": "1.0.14" - }, - { - "hash": "QmYxUdYY9S6yg5tSPVin5GFTvtfsLauVcr7reHDD3dM8xf", - "name": "go-libp2p-routing", - "version": "2.7.13" - } - ], - "gxVersion": "0.4.0", - "language": "go", - "license": "MIT", - "name": "go-libp2p", - "releaseCmd": "git commit -a -m \"gx publish $VERSION\"", - "version": "6.0.41" -} From 4a84cb455f3eddc5e46927656256c56b1e5c5cfc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 11 Apr 2019 21:11:16 +0100 Subject: [PATCH 1077/3965] farewell gx; thanks for serving us well. --- .gx/lastpubver | 1 - .travis.yml | 2 -- README.md | 4 ++++ package.json | 41 ----------------------------------------- 4 files changed, 4 insertions(+), 44 deletions(-) delete mode 100644 .gx/lastpubver delete mode 100644 package.json diff --git a/.gx/lastpubver b/.gx/lastpubver deleted file mode 100644 index 0306793c53..0000000000 --- a/.gx/lastpubver +++ /dev/null @@ -1 +0,0 @@ -1.0.3: QmdwkZHamNNrj7k3G29rnurmW3mFzsDhnyXppNcgYsiBVz diff --git a/.travis.yml b/.travis.yml index 4cfe98c242..5163d693fc 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,7 +10,6 @@ env: global: - GOTFLAGS="-race" matrix: - - BUILD_DEPTYPE=gx - BUILD_DEPTYPE=gomod @@ -24,7 +23,6 @@ script: cache: directories: - - $GOPATH/src/gx - $GOPATH/pkg/mod - $HOME/.cache/go-build diff --git a/README.md b/README.md index eec79193e6..61b5cc8891 100644 --- a/README.md +++ b/README.md @@ -3,3 +3,7 @@ [![GoDoc](https://godoc.org/github.com/libp2p/go-nat?status.svg)](https://godoc.org/github.com/libp2p/go-nat) [![status](https://sourcegraph.com/api/repos/github.com/libp2p/go-nat/.badges/status.png)](https://sourcegraph.com/github.com/libp2p/go-nat) Forked from: [fd/go-nat](https://github.com/fd/go-nat). + +--- + +The last gx published version of this module was: 1.0.3: QmdwkZHamNNrj7k3G29rnurmW3mFzsDhnyXppNcgYsiBVz diff --git a/package.json b/package.json deleted file mode 100644 index 7e176a89b2..0000000000 --- a/package.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "author": "fd", - "bugs": { - "url": "https://github.com/issues/go-nat/issues" - }, - "gx": { - "dvcsimport": "github.com/libp2p/go-nat" - }, - "gxDependencies": [ - { - "author": "huin", - "hash": "QmVwfv63beSAAirq3tiLY6fkNqvjThLnCofL7363YkaScy", - "name": "goupnp", - "version": "0.1.0" - }, - { - "author": "jackpal", - "hash": "Qmf2fBLzCvFxs3vvZaoQyKSTv2rjApek4F1kzRxAfK6P4P", - "name": "gateway", - "version": "1.0.4" - }, - { - "author": "jackpal", - "hash": "QmYsYNh6saxUYHajdj49uiRzdxQgiFTtymrjf3d1f2Cer4", - "name": "go-nat-pmp", - "version": "1.0.1" - }, - { - "author": "koron", - "hash": "QmfDrLLyLVTLbDFfYvPsMqj3hgacxFCXyShZ5YjobqUAtr", - "name": "go-ssdp", - "version": "0.0.0" - } - ], - "gxVersion": "0.9.0", - "language": "go", - "license": "Apache-2.0", - "name": "go-nat", - "version": "1.0.3" -} - From 018d54f516bf5f5b7228dd59b45eff9a378ce269 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 11 Apr 2019 21:47:34 -0700 Subject: [PATCH 1078/3965] keep temp addresses for 2 minutes Unfortunately, services like the DHT are slow enough that we can forget addresses before we successfully dial. --- p2p/host/peerstore/interface.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index 4d166684f5..61b7af97e0 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -19,7 +19,7 @@ var ( AddressTTL = time.Hour // TempAddrTTL is the ttl used for a short lived address - TempAddrTTL = time.Second * 10 + TempAddrTTL = time.Minute * 2 // ProviderAddrTTL is the TTL of an address we've received from a provider. // This is also a temporary address, but lasts longer. After this expires, From 2c33dc34ac436764f65af8374db03d1fe3787118 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 11 Apr 2019 22:13:02 -0700 Subject: [PATCH 1079/3965] dep: update go-libp2p-nat Switches to libp2p/go-nat which has some improved gateway detection logic. --- go.mod | 2 +- go.sum | 23 ++++++++++++++--------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/go.mod b/go.mod index cd4fd9ce55..fff4abc21e 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/libp2p/go-libp2p-interface-pnet v0.0.1 github.com/libp2p/go-libp2p-loggables v0.0.1 github.com/libp2p/go-libp2p-metrics v0.0.1 - github.com/libp2p/go-libp2p-nat v0.0.2 + github.com/libp2p/go-libp2p-nat v0.0.3 github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.0.1 diff --git a/go.sum b/go.sum index 7a6ae33f09..e4a4351d9a 100644 --- a/go.sum +++ b/go.sum @@ -18,8 +18,6 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs github.com/dgraph-io/badger v1.5.5-0.20190226225317-8115aed38f8f/go.mod h1:VZxzAIRPHRVNRKRo6AXrX9BJegn6il06VMTZVJYCIjQ= github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= -github.com/fd/go-nat v1.0.0 h1:DPyQ97sxA9ThrWYRPcWUz/z9TnpTIGRYODIQc/dy64M= -github.com/fd/go-nat v1.0.0/go.mod h1:BTBu/CKvMmOMUPkKVef1pngt2WFH/lg7E6yQnulfp6E= github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98= @@ -37,13 +35,16 @@ github.com/gxed/hashland/keccakpg v0.0.1 h1:wrk3uMNaMxbXiHibbPO4S0ymqJMm41WiudyF github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= github.com/gxed/hashland/murmur3 v0.0.1 h1:SheiaIt0sda5K+8FLz952/1iWS9zrnKsEJaOJu4ZbSc= github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= +github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA= github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o= github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= -github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324 h1:PV190X5/DzQ/tbFFG5YpT5mH6q+cHlfgqI5JuRnH9oE= -github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324/go.mod h1:MZ2ZmwcBpvOoJ22IJsc7va19ZwoheaBk43rKg12SKag= +github.com/huin/goupnp v1.0.0 h1:wg75sLpL6DZqwHQN6E1Cfk6mtfzS45z8OV+ic+DtHRo= +github.com/huin/goupnp v1.0.0/go.mod h1:n9v9KO1tAxYH82qOn+UTIFQDmx5n1Zxd/ClZDMX7Bnc= +github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150/go.mod h1:PpLOETDnJ0o3iZrZfqZzyLl6l7F3c6L1oWn7OICBi6o= github.com/ipfs/go-cid v0.0.1 h1:GBjWPktLnNyX0JiQCNFpUuUSoMw5KMyqrsejHYlILBE= github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= @@ -56,8 +57,8 @@ github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc= github.com/ipfs/go-log v0.0.1 h1:9XTUN/rW64BCG1YhPK9Hoy3q8nr4gOmHHBpgFdfw6Lc= github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= -github.com/jackpal/gateway v1.0.4 h1:LS5EHkLuQ6jzaHwULi0vL+JO0mU/n4yUtK8oUjHHOlM= -github.com/jackpal/gateway v1.0.4/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= +github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc= +github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec h1:DQqZhhDvrTrEQ3Qod5yfavcA064e53xlQ+xajiorXgM= @@ -71,6 +72,8 @@ github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlT github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4= +github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b h1:wxtKgYHEncAU00muMD06dzLiahtGM1eouRNOzVV7tdQ= +github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= @@ -106,8 +109,8 @@ github.com/libp2p/go-libp2p-loggables v0.0.1 h1:HVww9oAnINIxbt69LJNkxD8lnbfgteXR github.com/libp2p/go-libp2p-loggables v0.0.1/go.mod h1:lDipDlBNYbpyqyPX/KcoO+eq0sJYEVR2JgOexcivchg= github.com/libp2p/go-libp2p-metrics v0.0.1 h1:yumdPC/P2VzINdmcKZd0pciSUCpou+s0lwYCjBbzQZU= github.com/libp2p/go-libp2p-metrics v0.0.1/go.mod h1:jQJ95SXXA/K1VZi13h52WZMa9ja78zjyy5rspMsC/08= -github.com/libp2p/go-libp2p-nat v0.0.2 h1:sKI5hiCsGFhuEKdXMsF9mywQu2qhfoIGX6a+VG6zelE= -github.com/libp2p/go-libp2p-nat v0.0.2/go.mod h1:QrjXQSD5Dj4IJOdEcjHRkWTSomyxRo6HnUkf/TfQpLQ= +github.com/libp2p/go-libp2p-nat v0.0.3 h1:nLLtf+Do3Cn+wJeb3ZscwMWzksZRcVyYG/q7BSUm7rA= +github.com/libp2p/go-libp2p-nat v0.0.3/go.mod h1:pZbRd59Du4UU6BCFL3bi2zp2+2YNuu5CAxvIvZ4CSqQ= github.com/libp2p/go-libp2p-net v0.0.1 h1:xJ4Vh4yKF/XKb8fd1Ev0ebAGzVjMxXzrxG2kjtU+F5Q= github.com/libp2p/go-libp2p-net v0.0.1/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= github.com/libp2p/go-libp2p-net v0.0.2 h1:qP06u4TYXfl7uW/hzqPhlVVTSA2nw1B/bHBJaUnbh6M= @@ -139,6 +142,8 @@ github.com/libp2p/go-mplex v0.0.1 h1:dn2XGSrUxLtz3/8u85bGrwhUEKPX8MOF3lpmcWBZCWc github.com/libp2p/go-mplex v0.0.1/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= github.com/libp2p/go-msgio v0.0.1 h1:znj97n5FtXGCLDwe9x8jpHmY770SW4WStBGcCDh6GJw= github.com/libp2p/go-msgio v0.0.1/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= +github.com/libp2p/go-nat v0.0.2 h1:wtMkB7RTp9d0NT9jXfMbdhQI7HOJrV8roPMg4WmahbI= +github.com/libp2p/go-nat v0.0.2/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= github.com/libp2p/go-reuseport-transport v0.0.1 h1:UIRneNxLDmEGNjGHpIiWzSWkZ5bhxMCP9x3Vh7BSc7E= @@ -223,8 +228,8 @@ golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnf golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25 h1:jsG6UpNLt9iAsb0S2AGW28DveNzzgmbXR+ENoPjUeIU= golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/net v0.0.0-20180524181706-dfa909b99c79/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7 h1:C2F/nMkR/9sfUTpvR3QrjBuTdvMUC/cFajkphs1YLQo= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f h1:wMNYb4v58l5UBM7MYRLPG6ZhfOqbKu7X5eyFl8ZhKvA= From 3f647da3badec64ef95ad393e159386e171204fb Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 11 Apr 2019 22:31:41 -0700 Subject: [PATCH 1080/3965] fix detection of GenIG NATs fixes #6 --- nat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nat.go b/nat.go index 727ae2a2e9..371fdb0354 100644 --- a/nat.go +++ b/nat.go @@ -51,7 +51,7 @@ func DiscoverNATs(ctx context.Context) <-chan NAT { upnpIg2 := discoverUPNP_IG2(ctx) natpmp := discoverNATPMP(ctx) upnpGenIGDev := discoverUPNP_GenIGDev(ctx) - for upnpIg1 != nil || upnpIg2 != nil || natpmp != nil { + for upnpIg1 != nil || upnpIg2 != nil || natpmp != nil || upnpGenIGDev != nil { var ( nat NAT ok bool From 523cb01fafd52a886eff7de2af8c2846322e761f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 11 Apr 2019 22:34:27 -0700 Subject: [PATCH 1081/3965] Revert "Try to map external port the same as internal port" This reverts commit 3fa58ced20b2ff6110ae7eca6c22e18c06c8da6f. Its safer to just pick a random port. Unfortunately, some NATs (e.g., mine) suck and will happily map a port that's already been mapped elsewhere (e.g., in the port forwarding config). While this patch does the _nice_ thing, it was mostly cosmetic. --- natpmp.go | 7 ------- upnp.go | 6 ------ 2 files changed, 13 deletions(-) diff --git a/natpmp.go b/natpmp.go index e437250c3f..a31a499988 100644 --- a/natpmp.go +++ b/natpmp.go @@ -111,13 +111,6 @@ func (n *natpmpNAT) AddPortMapping(protocol string, internalPort int, descriptio } } - // try to map external port the same as internal port - _, err = n.c.AddPortMapping(protocol, internalPort, internalPort, timeoutInSeconds) - if err == nil { - n.ports[internalPort] = internalPort - return internalPort, nil - } - for i := 0; i < 3; i++ { externalPort := randomPort() _, err = n.c.AddPortMapping(protocol, internalPort, externalPort, timeoutInSeconds) diff --git a/upnp.go b/upnp.go index dddf7e7047..ccfeb14a2f 100644 --- a/upnp.go +++ b/upnp.go @@ -264,12 +264,6 @@ func (u *upnp_NAT) AddPortMapping(protocol string, internalPort int, description } } - // try to map external port the same as internal port - err = u.c.AddPortMapping("", uint16(internalPort), mapProtocol(protocol), uint16(internalPort), ip.String(), true, description, timeoutInSeconds) - if err == nil { - return internalPort, nil - } - for i := 0; i < 3; i++ { externalPort := randomPort() err = u.c.AddPortMapping("", uint16(externalPort), mapProtocol(protocol), uint16(internalPort), ip.String(), true, description, timeoutInSeconds) From b94e2ce8f3611db9eee4d22814bb4e31378fa171 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 12 Apr 2019 10:52:17 +0300 Subject: [PATCH 1082/3965] gomod: update go-libp2p-circuit --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index fff4abc21e..af3053ee66 100644 --- a/go.mod +++ b/go.mod @@ -11,7 +11,7 @@ require ( github.com/libp2p/go-conn-security-multistream v0.0.1 github.com/libp2p/go-libp2p-autonat v0.0.4 github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.3 + github.com/libp2p/go-libp2p-circuit v0.0.4 github.com/libp2p/go-libp2p-crypto v0.0.1 github.com/libp2p/go-libp2p-discovery v0.0.1 github.com/libp2p/go-libp2p-host v0.0.1 diff --git a/go.sum b/go.sum index e4a4351d9a..3f5188aad0 100644 --- a/go.sum +++ b/go.sum @@ -93,8 +93,8 @@ github.com/libp2p/go-libp2p-autonat v0.0.4 h1:cZzdB9KW1ZkHnSjLCB6aFNw47XS4r+SecC github.com/libp2p/go-libp2p-autonat v0.0.4/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.3 h1:FKhhuMOegdNIEbXkZsClEiqXJr+NmR+REmQRiMgtSyA= -github.com/libp2p/go-libp2p-circuit v0.0.3/go.mod h1:p1cHJnB9xnX5/1vZLkXgKwmNEOQQuF/Hp+SkATXnXYk= +github.com/libp2p/go-libp2p-circuit v0.0.4 h1:yOgEadnSVFj3e9KLBuLG+edqCImeav0VXxXvcimpOUQ= +github.com/libp2p/go-libp2p-circuit v0.0.4/go.mod h1:p1cHJnB9xnX5/1vZLkXgKwmNEOQQuF/Hp+SkATXnXYk= github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-discovery v0.0.1 h1:VkjCKmJQMwpDUwtA8Qc1z3TQAHJgQ5nGQ6cdN0wQXOw= From ef92f67df618768207b14dd125d80b6e715dad97 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 12 Apr 2019 10:29:02 -0700 Subject: [PATCH 1083/3965] dep: update go-libp2p-nat again This time, don't try to map the same internal/external port. Random ports are safer as not all NAT devices can handle port conflicts... --- go.mod | 2 +- go.sum | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/go.mod b/go.mod index af3053ee66..f6fc97408f 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/libp2p/go-libp2p-interface-pnet v0.0.1 github.com/libp2p/go-libp2p-loggables v0.0.1 github.com/libp2p/go-libp2p-metrics v0.0.1 - github.com/libp2p/go-libp2p-nat v0.0.3 + github.com/libp2p/go-libp2p-nat v0.0.4 github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.0.1 diff --git a/go.sum b/go.sum index 3f5188aad0..2f17c5de7f 100644 --- a/go.sum +++ b/go.sum @@ -109,8 +109,8 @@ github.com/libp2p/go-libp2p-loggables v0.0.1 h1:HVww9oAnINIxbt69LJNkxD8lnbfgteXR github.com/libp2p/go-libp2p-loggables v0.0.1/go.mod h1:lDipDlBNYbpyqyPX/KcoO+eq0sJYEVR2JgOexcivchg= github.com/libp2p/go-libp2p-metrics v0.0.1 h1:yumdPC/P2VzINdmcKZd0pciSUCpou+s0lwYCjBbzQZU= github.com/libp2p/go-libp2p-metrics v0.0.1/go.mod h1:jQJ95SXXA/K1VZi13h52WZMa9ja78zjyy5rspMsC/08= -github.com/libp2p/go-libp2p-nat v0.0.3 h1:nLLtf+Do3Cn+wJeb3ZscwMWzksZRcVyYG/q7BSUm7rA= -github.com/libp2p/go-libp2p-nat v0.0.3/go.mod h1:pZbRd59Du4UU6BCFL3bi2zp2+2YNuu5CAxvIvZ4CSqQ= +github.com/libp2p/go-libp2p-nat v0.0.4 h1:+KXK324yaY701On8a0aGjTnw8467kW3ExKcqW2wwmyw= +github.com/libp2p/go-libp2p-nat v0.0.4/go.mod h1:N9Js/zVtAXqaeT99cXgTV9e75KpnWCvVOiGzlcHmBbY= github.com/libp2p/go-libp2p-net v0.0.1 h1:xJ4Vh4yKF/XKb8fd1Ev0ebAGzVjMxXzrxG2kjtU+F5Q= github.com/libp2p/go-libp2p-net v0.0.1/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= github.com/libp2p/go-libp2p-net v0.0.2 h1:qP06u4TYXfl7uW/hzqPhlVVTSA2nw1B/bHBJaUnbh6M= @@ -142,8 +142,8 @@ github.com/libp2p/go-mplex v0.0.1 h1:dn2XGSrUxLtz3/8u85bGrwhUEKPX8MOF3lpmcWBZCWc github.com/libp2p/go-mplex v0.0.1/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= github.com/libp2p/go-msgio v0.0.1 h1:znj97n5FtXGCLDwe9x8jpHmY770SW4WStBGcCDh6GJw= github.com/libp2p/go-msgio v0.0.1/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= -github.com/libp2p/go-nat v0.0.2 h1:wtMkB7RTp9d0NT9jXfMbdhQI7HOJrV8roPMg4WmahbI= -github.com/libp2p/go-nat v0.0.2/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= +github.com/libp2p/go-nat v0.0.3 h1:l6fKV+p0Xa354EqQOQP+d8CivdLM4kl5GxC1hSc/UeI= +github.com/libp2p/go-nat v0.0.3/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= github.com/libp2p/go-reuseport-transport v0.0.1 h1:UIRneNxLDmEGNjGHpIiWzSWkZ5bhxMCP9x3Vh7BSc7E= From e2acda1c426e826eb4d70c3233bc816a0b8cf5ab Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 12 Apr 2019 20:59:41 +0300 Subject: [PATCH 1084/3965] gomod: update go-libp2p-peerstore --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index f6fc97408f..ba871457b9 100644 --- a/go.mod +++ b/go.mod @@ -23,7 +23,7 @@ require ( github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.0.1 - github.com/libp2p/go-libp2p-peerstore v0.0.1 + github.com/libp2p/go-libp2p-peerstore v0.0.2 github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 github.com/libp2p/go-libp2p-secio v0.0.1 diff --git a/go.sum b/go.sum index 2f17c5de7f..8406344758 100644 --- a/go.sum +++ b/go.sum @@ -121,6 +121,8 @@ github.com/libp2p/go-libp2p-peer v0.0.1 h1:0qwAOljzYewINrU+Kndoc+1jAL7vzY/oY2Go4 github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= github.com/libp2p/go-libp2p-peerstore v0.0.1 h1:twKovq8YK5trLrd3nB7PD2Zu9JcyAIdm7Bz9yBWjhq8= github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= +github.com/libp2p/go-libp2p-peerstore v0.0.2 h1:Lirt3A1Oq11jszJ4SPNBo8chNv61UWXE538KUEGxTVk= +github.com/libp2p/go-libp2p-peerstore v0.0.2/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= github.com/libp2p/go-libp2p-protocol v0.0.1 h1:+zkEmZ2yFDi5adpVE3t9dqh/N9TbpFWywowzeEzBbLM= github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbfdvpNntjhc= From ffd4981247c2d229fff54a35118a278a77070e86 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 22:36:29 +0300 Subject: [PATCH 1085/3965] increase autorelay discovery limit to 1k --- p2p/host/relay/autorelay.go | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index fbbbeef565..a580a74d74 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -136,10 +136,7 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { need := DesiredRelays - len(ar.relays) ar.mx.Unlock() - limit := 50 - if need > limit/2 { - limit = 2 * need - } + limit := 1000 dctx, cancel := context.WithTimeout(ctx, 30*time.Second) pis, err := discovery.FindPeers(dctx, ar.discover, RelayRendezvous, limit) From cfdf4178ff21966589a8a1189c0d58f20b25ed04 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 23:12:56 +0300 Subject: [PATCH 1086/3965] use 3hrs as routing advertisement ttl --- p2p/discovery/routing/routing.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index 3f6a9738e5..c618ba7c63 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -37,8 +37,9 @@ func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Opt return 0, err } - // the DHT provider record validity is 24hrs, but it is recommnded to republish every 6hrs - return 6 * time.Hour, nil + // the DHT provider record validity is 24hrs, but it is recommnded to republish at least every 6hrs + // we go one step further and republish every 3hrs + return 3 * time.Hour, nil } func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Option) (<-chan pstore.PeerInfo, error) { From a1ebc4d8525f863d9580e8239df6a12c094dbacb Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 23:46:41 +0300 Subject: [PATCH 1087/3965] reduce relay find peer and connect timeout to 30s --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index a580a74d74..0d85ad5f5c 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -158,7 +158,7 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { } ar.mx.Unlock() - cctx, cancel := context.WithTimeout(ctx, 60*time.Second) + cctx, cancel := context.WithTimeout(ctx, 30*time.Second) if len(pi.Addrs) == 0 { pi, err = ar.router.FindPeer(cctx, pi.ID) From aa03a9b8395cb62680aa9843843bd419d1737e34 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 12:20:57 +0300 Subject: [PATCH 1088/3965] clean up relay address sets to curtail addrsplosion --- p2p/host/relay/autorelay.go | 58 +++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 0d85ad5f5c..3c7720728c 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "math/rand" + "strconv" "sync" "time" @@ -169,6 +170,14 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { } } + cleanupAddressSet(&pi) + + if len(pi.Addrs) == 0 { + log.Debugf("no usable addresses for relay peer %s", pi.ID) + cancel() + continue + } + err = ar.host.Connect(cctx, pi) cancel() if err != nil { @@ -250,6 +259,55 @@ func (ar *AutoRelay) doUpdateAddrs() { ar.addrs = raddrs } +// This function cleans up a relay's address set to remove private addresses and curtail +// addrsplosion. For the latter, we use the following heuristic: +// - if the address set includes a (tcp) address with the default port 4001, +// we remove all tcp addrs with a different port +// - Otherwise we remove all addrs with ephemeral ports (>= 32768) +func cleanupAddressSet(pi *pstore.PeerInfo) { + // pass-1: find default port + has4001 := false + for _, addr := range pi.Addrs { + port, err := tcpPort(addr) + if err != nil { + continue + } + if port == 4001 { + has4001 = true + break + } + } + + // pass-2: cleanup + var newAddrs []ma.Multiaddr + + for _, addr := range pi.Addrs { + if manet.IsPrivateAddr(addr) { + continue + } + + port, err := tcpPort(addr) + if err == nil { + if (has4001 && port != 4001) || port >= 32768 { + continue + } + } + + newAddrs = append(newAddrs, addr) + } + + pi.Addrs = newAddrs +} + +func tcpPort(addr ma.Multiaddr) (int, error) { + val, err := addr.ValueForProtocol(ma.P_TCP) + if err != nil { + // not tcp + return 0, err + } + return strconv.Atoi(val) +} + func shuffleRelays(pis []pstore.PeerInfo) { for i := range pis { j := rand.Intn(i + 1) From 9863e22d4a8d5f987b377e2168b84cb650e61c85 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 14:06:13 +0300 Subject: [PATCH 1089/3965] rework relay selection logic --- p2p/host/relay/autorelay.go | 67 ++++++++++++++++++++++++------------- 1 file changed, 44 insertions(+), 23 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 3c7720728c..47ec35776b 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -147,8 +147,7 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { return } - pis = ar.selectRelays(pis) - + pis = ar.selectRelays(ctx, pis, 20) update := 0 for _, pi := range pis { @@ -159,25 +158,7 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { } ar.mx.Unlock() - cctx, cancel := context.WithTimeout(ctx, 30*time.Second) - - if len(pi.Addrs) == 0 { - pi, err = ar.router.FindPeer(cctx, pi.ID) - if err != nil { - log.Debugf("error finding relay peer %s: %s", pi.ID, err.Error()) - cancel() - continue - } - } - - cleanupAddressSet(&pi) - - if len(pi.Addrs) == 0 { - log.Debugf("no usable addresses for relay peer %s", pi.ID) - cancel() - continue - } - + cctx, cancel := context.WithTimeout(ctx, 15*time.Second) err = ar.host.Connect(cctx, pi) cancel() if err != nil { @@ -205,11 +186,51 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { } } -func (ar *AutoRelay) selectRelays(pis []pstore.PeerInfo) []pstore.PeerInfo { +func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, count int) []pstore.PeerInfo { // TODO better relay selection strategy; this just selects random relays // but we should probably use ping latency as the selection metric shuffleRelays(pis) - return pis + + if len(pis) < count { + count = len(pis) + } + + // only select relays that can be found by routing + type queryResult struct { + pi pstore.PeerInfo + err error + } + result := make([]pstore.PeerInfo, 0, count) + resultCh := make(chan queryResult, len(pis)) + + qctx, cancel := context.WithTimeout(ctx, 15*time.Second) + defer cancel() + + for _, pi := range pis { + go func(p peer.ID) { + pi, err := ar.router.FindPeer(qctx, p) + if err != nil { + log.Debugf("Error finding relay peer %s: %s", p, err.Error()) + } + resultCh <- queryResult{pi: pi, err: err} + }(pi.ID) + } + + for len(result) < count { + select { + case qr := <-resultCh: + if qr.err == nil { + cleanupAddressSet(&qr.pi) + result = append(result, qr.pi) + } + + case <-qctx.Done(): + break + } + } + + shuffleRelays(result) + return result } func (ar *AutoRelay) updateAddrs() { From 765ed0e1a63b3ce8c225c8ff697afc7354fb6c68 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 14:13:36 +0300 Subject: [PATCH 1090/3965] select 50 relays --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 47ec35776b..fbb26a74e8 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -147,7 +147,7 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { return } - pis = ar.selectRelays(ctx, pis, 20) + pis = ar.selectRelays(ctx, pis, 50) update := 0 for _, pi := range pis { From 39f2b7ad456c99868dc6f177ad299f9354582c87 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 14:25:30 +0300 Subject: [PATCH 1091/3965] fix selectRelays to return promptly when there are query errors --- p2p/host/relay/autorelay.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index fbb26a74e8..24035e5361 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -189,7 +189,6 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, count int) []pstore.PeerInfo { // TODO better relay selection strategy; this just selects random relays // but we should probably use ping latency as the selection metric - shuffleRelays(pis) if len(pis) < count { count = len(pis) @@ -216,12 +215,13 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co }(pi.ID) } - for len(result) < count { + rcount := 0 + for len(result) < count && rcount < len(pis) { select { case qr := <-resultCh: + rcount++ if qr.err == nil { - cleanupAddressSet(&qr.pi) - result = append(result, qr.pi) + result = append(result, cleanupAddressSet(qr.pi)) } case <-qctx.Done(): @@ -285,7 +285,7 @@ func (ar *AutoRelay) doUpdateAddrs() { // - if the address set includes a (tcp) address with the default port 4001, // we remove all tcp addrs with a different port // - Otherwise we remove all addrs with ephemeral ports (>= 32768) -func cleanupAddressSet(pi *pstore.PeerInfo) { +func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { // pass-1: find default port has4001 := false for _, addr := range pi.Addrs { @@ -317,7 +317,7 @@ func cleanupAddressSet(pi *pstore.PeerInfo) { newAddrs = append(newAddrs, addr) } - pi.Addrs = newAddrs + return pstore.PeerInfo{ID: pi.ID, Addrs: newAddrs} } func tcpPort(addr ma.Multiaddr) (int, error) { From becb89a2451296e02b7b7cf23443c6fed44be7e3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 14:32:31 +0300 Subject: [PATCH 1092/3965] shuffle relay set before queries --- p2p/host/relay/autorelay.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 24035e5361..42e36b439e 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -205,6 +205,8 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co qctx, cancel := context.WithTimeout(ctx, 15*time.Second) defer cancel() + // shuffle to randomize the order of queries + shuffleRelays(pis) for _, pi := range pis { go func(p peer.ID) { pi, err := ar.router.FindPeer(qctx, p) From 1138fb604084ba6dab531e8b3b35fe4cf7ff5e13 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 14:37:33 +0300 Subject: [PATCH 1093/3965] only determine default port if it is in a public addr --- p2p/host/relay/autorelay.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 42e36b439e..106b6f1e20 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -291,6 +291,10 @@ func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { // pass-1: find default port has4001 := false for _, addr := range pi.Addrs { + if manet.IsPrivateAddr(addr) { + continue + } + port, err := tcpPort(addr) if err != nil { continue From 31dfaf60296b5152f009b496bb351402077d71f2 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 14:53:30 +0300 Subject: [PATCH 1094/3965] increase relay advertising boot delay to 15min --- p2p/host/relay/relay.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index 0f38166984..d0af9d227a 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -10,7 +10,8 @@ import ( ) var ( - AdvertiseBootDelay = 30 * time.Second + // this is purposefully long to require some node stability before advertising as a relay + AdvertiseBootDelay = 15 * time.Minute ) // Advertise advertises this node as a libp2p relay. From c9f627e221ad0780429169b6831924986b52da60 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 15:06:15 +0300 Subject: [PATCH 1095/3965] increase FindPeer timeout to 30s --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 106b6f1e20..55f8c6d8d1 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -202,7 +202,7 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co result := make([]pstore.PeerInfo, 0, count) resultCh := make(chan queryResult, len(pis)) - qctx, cancel := context.WithTimeout(ctx, 15*time.Second) + qctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() // shuffle to randomize the order of queries From 45d28886172bca54341c1eb0a2f0c88f3d5f9fc9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 15:37:01 +0300 Subject: [PATCH 1096/3965] don't drop ephemeral ports in address set clean up --- p2p/host/relay/autorelay.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 55f8c6d8d1..b30a300d05 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -286,7 +286,6 @@ func (ar *AutoRelay) doUpdateAddrs() { // addrsplosion. For the latter, we use the following heuristic: // - if the address set includes a (tcp) address with the default port 4001, // we remove all tcp addrs with a different port -// - Otherwise we remove all addrs with ephemeral ports (>= 32768) func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { // pass-1: find default port has4001 := false @@ -313,9 +312,9 @@ func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { continue } - port, err := tcpPort(addr) - if err == nil { - if (has4001 && port != 4001) || port >= 32768 { + if has4001 { + port, err := tcpPort(addr) + if err == nil && port != 4001 { continue } } From da65fd74a6939a697964a0df1d03855814cd888e Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 15:37:20 +0300 Subject: [PATCH 1097/3965] fix and reinstate autorelay test --- p2p/host/relay/autorelay_test.go | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index 379abd0ff0..c9da893920 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -26,8 +26,8 @@ import ( // test specific parameters func init() { - autonat.AutoNATIdentifyDelay = 500 * time.Millisecond - autonat.AutoNATBootDelay = 1 * time.Second + autonat.AutoNATIdentifyDelay = 1 * time.Second + autonat.AutoNATBootDelay = 2 * time.Second relay.BootDelay = 1 * time.Second relay.AdvertiseBootDelay = 1 * time.Millisecond manet.Private4 = []*net.IPNet{} @@ -37,6 +37,7 @@ func init() { type mockRoutingTable struct { mx sync.Mutex providers map[string]map[peer.ID]pstore.PeerInfo + peers map[peer.ID]pstore.PeerInfo } type mockRouting struct { @@ -53,7 +54,13 @@ func newMockRouting(h host.Host, tab *mockRoutingTable) *mockRouting { } func (m *mockRouting) FindPeer(ctx context.Context, p peer.ID) (pstore.PeerInfo, error) { - return pstore.PeerInfo{}, routing.ErrNotFound + m.tab.mx.Lock() + defer m.tab.mx.Unlock() + pi, ok := m.tab.peers[p] + if !ok { + return pstore.PeerInfo{}, routing.ErrNotFound + } + return pi, nil } func (m *mockRouting) Provide(ctx context.Context, cid cid.Cid, bcast bool) error { @@ -66,7 +73,12 @@ func (m *mockRouting) Provide(ctx context.Context, cid cid.Cid, bcast bool) erro m.tab.providers[cid.String()] = pmap } - pmap[m.h.ID()] = pstore.PeerInfo{ID: m.h.ID(), Addrs: m.h.Addrs()} + pi := pstore.PeerInfo{ID: m.h.ID(), Addrs: m.h.Addrs()} + pmap[m.h.ID()] = pi + if m.tab.peers == nil { + m.tab.peers = make(map[peer.ID]pstore.PeerInfo) + } + m.tab.peers[m.h.ID()] = pi return nil } @@ -133,7 +145,7 @@ func connect(t *testing.T, a, b host.Host) { // and the actual test! func TestAutoRelay(t *testing.T) { - t.Skip("fails 99% of the time") + //t.Skip("fails 99% of the time") ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -165,7 +177,7 @@ func TestAutoRelay(t *testing.T) { // connect to AutoNAT and let detection/discovery work its magic connect(t, h1, h3) - time.Sleep(3 * time.Second) + time.Sleep(5 * time.Second) // verify that we now advertise relay addrs (but not unspecific relay addrs) unspecificRelay, err := ma.NewMultiaddr("/p2p-circuit") From 8c7da83b55ee0a3289ef2d52305dcb53296d99b5 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 15:47:46 +0300 Subject: [PATCH 1098/3965] also check private addresses when looking for 4001 --- p2p/host/relay/autorelay.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index b30a300d05..a86b8e99f0 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -290,10 +290,6 @@ func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { // pass-1: find default port has4001 := false for _, addr := range pi.Addrs { - if manet.IsPrivateAddr(addr) { - continue - } - port, err := tcpPort(addr) if err != nil { continue From 081bb0f7ed4a86870c22f225440b861fa66397f7 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 15:55:45 +0300 Subject: [PATCH 1099/3965] small tweak in autorelay test --- p2p/host/relay/autorelay_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index c9da893920..cb661aa30b 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -29,7 +29,7 @@ func init() { autonat.AutoNATIdentifyDelay = 1 * time.Second autonat.AutoNATBootDelay = 2 * time.Second relay.BootDelay = 1 * time.Second - relay.AdvertiseBootDelay = 1 * time.Millisecond + relay.AdvertiseBootDelay = 100 * time.Millisecond manet.Private4 = []*net.IPNet{} } From 4a6e767da228ce9f9a2a63e4fd4d4e14e1579481 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 16:03:58 +0300 Subject: [PATCH 1100/3965] log and ignore relay peers with empty address sets --- p2p/host/relay/autorelay.go | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index a86b8e99f0..2f7f35055c 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -211,7 +211,7 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co go func(p peer.ID) { pi, err := ar.router.FindPeer(qctx, p) if err != nil { - log.Debugf("Error finding relay peer %s: %s", p, err.Error()) + log.Debugf("error finding relay peer %s: %s", p, err.Error()) } resultCh <- queryResult{pi: pi, err: err} }(pi.ID) @@ -223,7 +223,12 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co case qr := <-resultCh: rcount++ if qr.err == nil { - result = append(result, cleanupAddressSet(qr.pi)) + pi := cleanupAddressSet(qr.pi) + if len(pi.Addrs) > 0 { + result = append(result, pi) + } else { + log.Debugf("ignoring relay peer %s: cleaned up address set is empty", pi.ID) + } } case <-qctx.Done(): From 376379b5b84f423402d6fd01470933ad7f8e1fb9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 16:34:11 +0300 Subject: [PATCH 1101/3965] retry to find relays if we fail to connect to any --- p2p/host/relay/autorelay.go | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 2f7f35055c..c65d8056b4 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -129,8 +129,10 @@ func (ar *AutoRelay) background(ctx context.Context) { } func (ar *AutoRelay) findRelays(ctx context.Context) { +again: ar.mx.Lock() - if len(ar.relays) >= DesiredRelays { + haveRelays := len(ar.relays) + if haveRelays >= DesiredRelays { ar.mx.Unlock() return } @@ -169,6 +171,7 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { log.Debugf("connected to relay %s", pi.ID) ar.mx.Lock() ar.relays[pi.ID] = pi + haveRelays++ ar.mx.Unlock() // tag the connection as very important @@ -181,6 +184,13 @@ func (ar *AutoRelay) findRelays(ctx context.Context) { } } + if haveRelays == 0 { + // we failed to find any relays and we are not connected to any! + // wait a little and try again, the discovery query might have returned only dead peers + time.Sleep(30 * time.Second) + goto again + } + if update > 0 || ar.addrs == nil { ar.updateAddrs() } From c8b8014fd08c0d890b80911f9f767544e1d5d8df Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 16:36:52 +0300 Subject: [PATCH 1102/3965] gate retry wait with the context --- p2p/host/relay/autorelay.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index c65d8056b4..3a8803f2eb 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -187,8 +187,12 @@ again: if haveRelays == 0 { // we failed to find any relays and we are not connected to any! // wait a little and try again, the discovery query might have returned only dead peers - time.Sleep(30 * time.Second) - goto again + select { + case <-time.After(30 * time.Second): + goto again + case <-ctx.Done(): + return + } } if update > 0 || ar.addrs == nil { From e8e2ab1930008a3979325d1e326aaf95d8afbda1 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 17:01:17 +0300 Subject: [PATCH 1103/3965] limit relay selection to 20 --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 3a8803f2eb..e07d3fdd44 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -149,7 +149,7 @@ again: return } - pis = ar.selectRelays(ctx, pis, 50) + pis = ar.selectRelays(ctx, pis, 20) update := 0 for _, pi := range pis { From 8d5c11c072fd0615cd99b323c6460ad19b03c587 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 17:40:16 +0300 Subject: [PATCH 1104/3965] fix private->public->private transition issues with address set --- p2p/host/relay/autorelay.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index e07d3fdd44..842f2ea685 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -133,7 +133,14 @@ again: ar.mx.Lock() haveRelays := len(ar.relays) if haveRelays >= DesiredRelays { + // this dance is necessary to cover the Private->Public->Private transition + // where we were already connected to enough relays while private and dropped + // the addrs while public + needUpdate := ar.addrs == nil ar.mx.Unlock() + if needUpdate { + ar.updateAddrs() + } return } need := DesiredRelays - len(ar.relays) From 0c69da940943bdacd51507bd8b49e81233329dc5 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 17:50:22 +0300 Subject: [PATCH 1105/3965] move the address invalidation check outside the lock --- p2p/host/relay/autorelay.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 842f2ea685..c3edbad881 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -133,12 +133,11 @@ again: ar.mx.Lock() haveRelays := len(ar.relays) if haveRelays >= DesiredRelays { + ar.mx.Unlock() // this dance is necessary to cover the Private->Public->Private transition // where we were already connected to enough relays while private and dropped // the addrs while public - needUpdate := ar.addrs == nil - ar.mx.Unlock() - if needUpdate { + if ar.addrs == nil { ar.updateAddrs() } return From 08306349383187c5af5ee824d17cc46be7597c41 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 21:16:35 +0300 Subject: [PATCH 1106/3965] limit number of FindPeer queries in relay selection --- p2p/host/relay/autorelay.go | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index c3edbad881..eb10727711 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -155,7 +155,7 @@ again: return } - pis = ar.selectRelays(ctx, pis, 20) + pis = ar.selectRelays(ctx, pis, 20, 50) update := 0 for _, pi := range pis { @@ -206,28 +206,37 @@ again: } } -func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, count int) []pstore.PeerInfo { +func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, count, maxq int) []pstore.PeerInfo { // TODO better relay selection strategy; this just selects random relays // but we should probably use ping latency as the selection metric + if len(pis) == 0 { + log.Debugf("no relays discovered") + return pis + } + if len(pis) < count { count = len(pis) } + if len(pis) < maxq { + maxq = len(pis) + } + // only select relays that can be found by routing type queryResult struct { pi pstore.PeerInfo err error } result := make([]pstore.PeerInfo, 0, count) - resultCh := make(chan queryResult, len(pis)) + resultCh := make(chan queryResult, maxq) qctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() // shuffle to randomize the order of queries shuffleRelays(pis) - for _, pi := range pis { + for _, pi := range pis[:maxq] { go func(p peer.ID) { pi, err := ar.router.FindPeer(qctx, p) if err != nil { @@ -238,7 +247,7 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co } rcount := 0 - for len(result) < count && rcount < len(pis) { + for len(result) < count && rcount < maxq { select { case qr := <-resultCh: rcount++ From 27f465ee356a54256f23c8ef8d641b99791c6de9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 14 Apr 2019 21:29:48 +0300 Subject: [PATCH 1107/3965] some better logging --- p2p/host/relay/autorelay.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index eb10727711..288640eeda 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -155,6 +155,8 @@ again: return } + log.Debugf("discovered %d relays", len(pis)) + pis = ar.selectRelays(ctx, pis, 20, 50) update := 0 @@ -193,6 +195,7 @@ again: if haveRelays == 0 { // we failed to find any relays and we are not connected to any! // wait a little and try again, the discovery query might have returned only dead peers + log.Debug("no relays connected; retrying in 30s") select { case <-time.After(30 * time.Second): goto again @@ -211,7 +214,6 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co // but we should probably use ping latency as the selection metric if len(pis) == 0 { - log.Debugf("no relays discovered") return pis } From 8d2457577d898b6daa1402842da16916e6875527 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Apr 2019 10:45:57 +0300 Subject: [PATCH 1108/3965] gomod: update go-libp2p-discovery --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index ba871457b9..285ace9ca3 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,7 @@ require ( github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.4 github.com/libp2p/go-libp2p-crypto v0.0.1 - github.com/libp2p/go-libp2p-discovery v0.0.1 + github.com/libp2p/go-libp2p-discovery v0.0.2 github.com/libp2p/go-libp2p-host v0.0.1 github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 github.com/libp2p/go-libp2p-interface-pnet v0.0.1 diff --git a/go.sum b/go.sum index 8406344758..f41b5191bd 100644 --- a/go.sum +++ b/go.sum @@ -97,8 +97,8 @@ github.com/libp2p/go-libp2p-circuit v0.0.4 h1:yOgEadnSVFj3e9KLBuLG+edqCImeav0VXx github.com/libp2p/go-libp2p-circuit v0.0.4/go.mod h1:p1cHJnB9xnX5/1vZLkXgKwmNEOQQuF/Hp+SkATXnXYk= github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= -github.com/libp2p/go-libp2p-discovery v0.0.1 h1:VkjCKmJQMwpDUwtA8Qc1z3TQAHJgQ5nGQ6cdN0wQXOw= -github.com/libp2p/go-libp2p-discovery v0.0.1/go.mod h1:ZkkF9xIFRLA1xCc7bstYFkd80gBGK8Fc1JqGoU2i+zI= +github.com/libp2p/go-libp2p-discovery v0.0.2 h1:Rf+20nsFcCnHo4Kxvf8ofAft75+fW+cXy9FonNVyU/g= +github.com/libp2p/go-libp2p-discovery v0.0.2/go.mod h1:ZkkF9xIFRLA1xCc7bstYFkd80gBGK8Fc1JqGoU2i+zI= github.com/libp2p/go-libp2p-host v0.0.1 h1:dnqusU+DheGcdxrE718kG4XgHNuL2n9eEv8Rg5zy8hQ= github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 h1:Q9EkNSLAOF+u90L88qmE9z/fTdjLh8OsJwGw74mkwk4= From 06391d4f2f4af216f2b46489eeba56de3ad61e42 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 15 Apr 2019 14:16:11 +0300 Subject: [PATCH 1109/3965] replace peer addresses in identify --- p2p/protocol/identify/id.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index f2e118b466..a250c3ff39 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -252,8 +252,10 @@ func (ids *IDService) consumeMessage(mes *pb.Identify, c inet.Conn) { ids.addrMu.Lock() switch ids.Host.Network().Connectedness(p) { case inet.Connected: + ids.Host.Peerstore().UpdateAddrs(p, pstore.ConnectedAddrTTL, 0) ids.Host.Peerstore().AddAddrs(p, lmaddrs, pstore.ConnectedAddrTTL) default: + ids.Host.Peerstore().UpdateAddrs(p, pstore.ConnectedAddrTTL, 0) ids.Host.Peerstore().AddAddrs(p, lmaddrs, pstore.RecentlyConnectedAddrTTL) } ids.addrMu.Unlock() From a88ae79f6fbd239a103a8da77d46b1041d0d1fdf Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 16 Apr 2019 21:09:15 +0300 Subject: [PATCH 1110/3965] use transientTTL for invalidated addrs instead of 0 --- p2p/protocol/identify/id.go | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index a250c3ff39..9aac465bfa 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -33,6 +33,9 @@ const LibP2PVersion = "ipfs/0.1.0" var ClientVersion = "go-libp2p/3.3.4" +// transientTTL is a short ttl for invalidated previously connected addrs +const transientTTL = 10 * time.Second + // IDService is a structure that implements ProtocolIdentify. // It is a trivial service that gives the other peer some // useful information about the local peer. A sort of hello. @@ -252,10 +255,12 @@ func (ids *IDService) consumeMessage(mes *pb.Identify, c inet.Conn) { ids.addrMu.Lock() switch ids.Host.Network().Connectedness(p) { case inet.Connected: - ids.Host.Peerstore().UpdateAddrs(p, pstore.ConnectedAddrTTL, 0) + // invalidate previous addrs -- we use a transient ttl instead of 0 to ensure there + // is no period of having no good addrs whatsoever + ids.Host.Peerstore().UpdateAddrs(p, pstore.ConnectedAddrTTL, transientTTL) ids.Host.Peerstore().AddAddrs(p, lmaddrs, pstore.ConnectedAddrTTL) default: - ids.Host.Peerstore().UpdateAddrs(p, pstore.ConnectedAddrTTL, 0) + ids.Host.Peerstore().UpdateAddrs(p, pstore.ConnectedAddrTTL, transientTTL) ids.Host.Peerstore().AddAddrs(p, lmaddrs, pstore.RecentlyConnectedAddrTTL) } ids.addrMu.Unlock() From 6d47278df050589fac7fd59fd5cf634a6d7a2617 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Tue, 16 Apr 2019 22:11:14 +0100 Subject: [PATCH 1111/3965] fix error handling: return on err. --- p2p/host/peerstore/pstoreds/addr_book.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index faef34b9e3..0eacacf344 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -264,6 +264,7 @@ func (ab *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.D pr, err := ab.loadRecord(p, true, false) if err != nil { log.Errorf("failed to update ttls for peer %s: %s\n", p.Pretty(), err) + return } pr.Lock() From cff9cb577c6ee182c41ce3dbec9e43f6f712a86c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 17 Apr 2019 16:20:02 -0700 Subject: [PATCH 1112/3965] constructor: allow nil options This can make it significantly easier to configure libp2p with optional options. --- config/config.go | 3 +++ config/config_test.go | 23 +++++++++++++++++++++++ libp2p.go | 3 +++ libp2p_test.go | 25 +++++++++++++++++++++++++ 4 files changed, 54 insertions(+) create mode 100644 config/config_test.go diff --git a/config/config.go b/config/config.go index 61b3c0318a..0b8c676bd4 100644 --- a/config/config.go +++ b/config/config.go @@ -242,6 +242,9 @@ type Option func(cfg *Config) error // encountered (if any). func (cfg *Config) Apply(opts ...Option) error { for _, opt := range opts { + if opt == nil { + continue + } if err := opt(cfg); err != nil { return err } diff --git a/config/config_test.go b/config/config_test.go new file mode 100644 index 0000000000..7f99b833a7 --- /dev/null +++ b/config/config_test.go @@ -0,0 +1,23 @@ +package config + +import ( + "testing" +) + +func TestNilOption(t *testing.T) { + var cfg Config + optsRun := 0 + opt := func(c *Config) error { + optsRun++ + return nil + } + if err := cfg.Apply(nil); err != nil { + t.Fatal(err) + } + if err := cfg.Apply(opt, nil, nil, opt, opt, nil); err != nil { + t.Fatal(err) + } + if optsRun != 3 { + t.Fatalf("expected to have handled 3 options, handled %d", optsRun) + } +} diff --git a/libp2p.go b/libp2p.go index ad026ad4bd..252c9a2920 100644 --- a/libp2p.go +++ b/libp2p.go @@ -19,6 +19,9 @@ type Option = config.Option func ChainOptions(opts ...Option) Option { return func(cfg *Config) error { for _, opt := range opts { + if opt == nil { + continue + } if err := opt(cfg); err != nil { return err } diff --git a/libp2p_test.go b/libp2p_test.go index 3ecedb3b83..84aaa9e464 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -133,3 +133,28 @@ func makeRandomHost(t *testing.T, port int) (host.Host, error) { return New(ctx, opts...) } + +func TestChainOptions(t *testing.T) { + var cfg Config + var optsRun []int + optcount := 0 + newOpt := func() Option { + index := optcount + optcount++ + return func(c *Config) error { + optsRun = append(optsRun, index) + return nil + } + } + if err := cfg.Apply(newOpt(), nil, ChainOptions(newOpt(), newOpt(), ChainOptions(), ChainOptions(nil, newOpt()))); err != nil { + t.Fatal(err) + } + if optcount != len(optsRun) { + t.Errorf("expected to have handled %d options, handled %d", optcount, len(optsRun)) + } + for i, x := range optsRun { + if i != x { + t.Errorf("expected opt %d, got opt %d", i, x) + } + } +} From 262b35f27234290365edf0aae8bfb3af686eab9b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 17 Apr 2019 18:05:08 -0700 Subject: [PATCH 1113/3965] chore: improve nil option tests --- libp2p_test.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/libp2p_test.go b/libp2p_test.go index 84aaa9e464..f2a28d6b23 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -146,12 +146,17 @@ func TestChainOptions(t *testing.T) { return nil } } + if err := cfg.Apply(newOpt(), nil, ChainOptions(newOpt(), newOpt(), ChainOptions(), ChainOptions(nil, newOpt()))); err != nil { t.Fatal(err) } - if optcount != len(optsRun) { + + // Make sure we ran all options. + if optcount != 4 { t.Errorf("expected to have handled %d options, handled %d", optcount, len(optsRun)) } + + // Make sure we ran the options in-order. for i, x := range optsRun { if i != x { t.Errorf("expected opt %d, got opt %d", i, x) From 4a4b14819a5ca77312983adc7c5fd086d0cf27e0 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Apr 2019 11:42:20 +0300 Subject: [PATCH 1114/3965] add comment about eliding the lock on addrs read --- p2p/host/relay/autorelay.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 288640eeda..a0d0523ef7 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -136,7 +136,9 @@ again: ar.mx.Unlock() // this dance is necessary to cover the Private->Public->Private transition // where we were already connected to enough relays while private and dropped - // the addrs while public + // the addrs while public. + // Note tht we don't need the lock for reading addrs, as the only writer is + // the current goroutine. if ar.addrs == nil { ar.updateAddrs() } From 433a0c09086e85fae2cbd76aeaa72d2a6bcef307 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Apr 2019 14:42:06 +0300 Subject: [PATCH 1115/3965] extract cleanupAddrSet and implement better heuristic --- p2p/host/relay/addrsplosion.go | 137 +++++++++++++++++++++++++++++++++ p2p/host/relay/autorelay.go | 50 +----------- 2 files changed, 138 insertions(+), 49 deletions(-) create mode 100644 p2p/host/relay/addrsplosion.go diff --git a/p2p/host/relay/addrsplosion.go b/p2p/host/relay/addrsplosion.go new file mode 100644 index 0000000000..bdb1435412 --- /dev/null +++ b/p2p/host/relay/addrsplosion.go @@ -0,0 +1,137 @@ +package relay + +import ( + "encoding/binary" + + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +// This function cleans up a relay's address set to remove private addresses and curtail +// addrsplosion. +func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { + var public, private []ma.Multiaddr + + for _, a := range pi.Addrs { + if manet.IsPublicAddr(a) || isDNSAddr(a) { + public = append(public, a) + continue + } + + // discard unroutable addrs + if manet.IsPrivateAddr(a) { + private = append(private, a) + } + } + + if !hasAddrsplosion(public) { + return pstore.PeerInfo{ID: pi.ID, Addrs: public} + } + + addrs := sanitizeAddrsplodedSet(public, private) + return pstore.PeerInfo{ID: pi.ID, Addrs: addrs} +} + +func isDNSAddr(a ma.Multiaddr) bool { + dnsaddr := false + ma.ForEach(a, func(c ma.Component) bool { + switch c.Protocol().Code { + case 54, 55, 56: + dnsaddr = true + } + + return false + }) + + return dnsaddr +} + +// we have addrsplosion if for some protocol we advertise multiple ports on +// the same base address. +func hasAddrsplosion(addrs []ma.Multiaddr) bool { + aset := make(map[string]int) + + for _, a := range addrs { + key, port := addrKeyAndPort(a) + xport, ok := aset[key] + if ok && port != xport { + return true + } + aset[key] = port + } + + return false +} + +func addrKeyAndPort(a ma.Multiaddr) (string, int) { + var ( + key string + port int + ) + + ma.ForEach(a, func(c ma.Component) bool { + switch c.Protocol().Code { + case ma.P_TCP, ma.P_UDP: + port = int(binary.BigEndian.Uint16(c.RawValue())) + key += "/" + c.Protocol().Name + default: + val := c.Value() + if val == "" { + val = c.Protocol().Name + } + key += "/" + val + } + return true + }) + + return key, port +} + +// clean up addrsplosion +// the following heuristic is used: +// - for each base address/protocol combination, if there are multiple ports advertised then +// only accept the default port if present. +// - If the default port is not present, we check for non-standard ports by tracking +// private port bindings if present. +// - If there is no default or private port binding, then we can't infer the correct +// port and give up and return all addrs (for that base address) +func sanitizeAddrsplodedSet(public, private []ma.Multiaddr) []ma.Multiaddr { + type portAndAddr struct { + addr ma.Multiaddr + port int + } + + privports := make(map[int]struct{}) + pubaddrs := make(map[string][]portAndAddr) + + for _, a := range private { + _, port := addrKeyAndPort(a) + privports[port] = struct{}{} + } + + for _, a := range public { + key, port := addrKeyAndPort(a) + pubaddrs[key] = append(pubaddrs[key], portAndAddr{addr: a, port: port}) + } + + var result []ma.Multiaddr + for _, pas := range pubaddrs { + if len(pas) == 1 { + result = append(result, pas[0].addr) + continue + } + + for _, pa := range pas { + if pa.port == 4001 || pa.port == 4002 { + result = append(result, pa.addr) + continue + } + if _, ok := privports[pa.port]; ok { + result = append(result, pa.addr) + } + } + } + + return result +} diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index a0d0523ef7..894548c34a 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -4,7 +4,6 @@ import ( "context" "fmt" "math/rand" - "strconv" "sync" "time" @@ -320,54 +319,6 @@ func (ar *AutoRelay) doUpdateAddrs() { ar.addrs = raddrs } -// This function cleans up a relay's address set to remove private addresses and curtail -// addrsplosion. For the latter, we use the following heuristic: -// - if the address set includes a (tcp) address with the default port 4001, -// we remove all tcp addrs with a different port -func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { - // pass-1: find default port - has4001 := false - for _, addr := range pi.Addrs { - port, err := tcpPort(addr) - if err != nil { - continue - } - if port == 4001 { - has4001 = true - break - } - } - - // pass-2: cleanup - var newAddrs []ma.Multiaddr - - for _, addr := range pi.Addrs { - if manet.IsPrivateAddr(addr) { - continue - } - - if has4001 { - port, err := tcpPort(addr) - if err == nil && port != 4001 { - continue - } - } - - newAddrs = append(newAddrs, addr) - } - - return pstore.PeerInfo{ID: pi.ID, Addrs: newAddrs} -} - -func tcpPort(addr ma.Multiaddr) (int, error) { - val, err := addr.ValueForProtocol(ma.P_TCP) - if err != nil { - // not tcp - return 0, err - } - return strconv.Atoi(val) -} - func shuffleRelays(pis []pstore.PeerInfo) { for i := range pis { j := rand.Intn(i + 1) @@ -375,6 +326,7 @@ func shuffleRelays(pis []pstore.PeerInfo) { } } +// Notifee func (ar *AutoRelay) Listen(inet.Network, ma.Multiaddr) {} func (ar *AutoRelay) ListenClose(inet.Network, ma.Multiaddr) {} func (ar *AutoRelay) Connected(inet.Network, inet.Conn) {} From a331f99b6588ca4b1fd23d229677e6513130e649 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Apr 2019 14:42:31 +0300 Subject: [PATCH 1116/3965] addrsplosion test --- p2p/host/relay/addrsplosion_test.go | 98 +++++++++++++++++++++++++++++ p2p/host/relay/autorelay_test.go | 5 +- 2 files changed, 102 insertions(+), 1 deletion(-) create mode 100644 p2p/host/relay/addrsplosion_test.go diff --git a/p2p/host/relay/addrsplosion_test.go b/p2p/host/relay/addrsplosion_test.go new file mode 100644 index 0000000000..0eb0c5be6c --- /dev/null +++ b/p2p/host/relay/addrsplosion_test.go @@ -0,0 +1,98 @@ +package relay + +import ( + "fmt" + "testing" + + pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" + _ "github.com/multiformats/go-multiaddr-dns" +) + +func TestCleanupAddrs(t *testing.T) { + // test with no addrsplosion + addrs := makeAddrList( + "/ip4/127.0.0.1/tcp/4001", + "/ip4/127.0.0.1/udp/4002/quic", + "/ip4/1.2.3.4/tcp/4001", + "/ip4/1.2.3.4/udp/4002/quic", + "/dnsaddr/somedomain.com/tcp/4002/ws", + ) + clean := makeAddrList( + "/ip4/1.2.3.4/tcp/4001", + "/ip4/1.2.3.4/udp/4002/quic", + "/dnsaddr/somedomain.com/tcp/4002/ws", + ) + + pi := cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) + if !sameAddrs(clean, pi.Addrs) { + fmt.Println(pi.Addrs) + t.Fatal("cleaned up set doesn't match expected") + } + + // test with default port addrspolosion + addrs = makeAddrList( + "/ip4/127.0.0.1/tcp/4001", + "/ip4/1.2.3.4/tcp/4001", + "/ip4/1.2.3.4/tcp/33333", + "/ip4/1.2.3.4/tcp/33334", + "/ip4/1.2.3.4/tcp/33335", + "/ip4/1.2.3.4/udp/4002/quic", + ) + clean = makeAddrList( + "/ip4/1.2.3.4/tcp/4001", + "/ip4/1.2.3.4/udp/4002/quic", + ) + pi = cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) + if !sameAddrs(clean, pi.Addrs) { + t.Fatal("cleaned up set doesn't match expected") + } + + // test with non-standard port addrsplosion + addrs = makeAddrList( + "/ip4/127.0.0.1/tcp/12345", + "/ip4/1.2.3.4/tcp/12345", + "/ip4/1.2.3.4/tcp/33333", + "/ip4/1.2.3.4/tcp/33334", + "/ip4/1.2.3.4/tcp/33335", + ) + clean = makeAddrList( + "/ip4/1.2.3.4/tcp/12345", + ) + pi = cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) + if !sameAddrs(clean, pi.Addrs) { + t.Fatal("cleaned up set doesn't match expected") + } + +} + +func makeAddrList(strs ...string) []ma.Multiaddr { + result := make([]ma.Multiaddr, 0, len(strs)) + for _, s := range strs { + a := ma.StringCast(s) + result = append(result, a) + } + return result +} + +func sameAddrs(as, bs []ma.Multiaddr) bool { + if len(as) != len(bs) { + return false + } + + for _, a := range as { + if !findAddr(a, bs) { + return false + } + } + return true +} + +func findAddr(a ma.Multiaddr, bs []ma.Multiaddr) bool { + for _, b := range bs { + if a.Equal(b) { + return true + } + } + return false +} diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index cb661aa30b..d86fdf6b35 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -30,7 +30,6 @@ func init() { autonat.AutoNATBootDelay = 2 * time.Second relay.BootDelay = 1 * time.Second relay.AdvertiseBootDelay = 100 * time.Millisecond - manet.Private4 = []*net.IPNet{} } // mock routing @@ -147,6 +146,10 @@ func connect(t *testing.T, a, b host.Host) { func TestAutoRelay(t *testing.T) { //t.Skip("fails 99% of the time") + save := manet.Private4 + manet.Private4 = []*net.IPNet{} + defer func() { manet.Private4 = save }() + ctx, cancel := context.WithCancel(context.Background()) defer cancel() From e82eabe130d5cd2c545ba38efb0d905902aad6cd Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 18 Apr 2019 14:49:42 +0300 Subject: [PATCH 1117/3965] cover the case where we can't select a default port in addrsplosion clean up --- p2p/host/relay/addrsplosion.go | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/p2p/host/relay/addrsplosion.go b/p2p/host/relay/addrsplosion.go index bdb1435412..a44401b108 100644 --- a/p2p/host/relay/addrsplosion.go +++ b/p2p/host/relay/addrsplosion.go @@ -118,16 +118,30 @@ func sanitizeAddrsplodedSet(public, private []ma.Multiaddr) []ma.Multiaddr { var result []ma.Multiaddr for _, pas := range pubaddrs { if len(pas) == 1 { + // it's not addrsploded result = append(result, pas[0].addr) continue } + haveAddr := false for _, pa := range pas { if pa.port == 4001 || pa.port == 4002 { + // it's a default port, use it result = append(result, pa.addr) + haveAddr = true continue } + if _, ok := privports[pa.port]; ok { + // it matches a privately bound port, use it + result = append(result, pa.addr) + haveAddr = true + } + } + + if !haveAddr { + // we weren't able to select a port; bite the bullet and use them all + for _, pa := range pas { result = append(result, pa.addr) } } From 9d7f6b83e5566ebbe379efaedae4f7096b4ef979 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 13:48:18 +0300 Subject: [PATCH 1118/3965] rewrite isDNSAddr to use ma.SplitFirst --- p2p/host/relay/addrsplosion.go | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/p2p/host/relay/addrsplosion.go b/p2p/host/relay/addrsplosion.go index a44401b108..3eca947b0c 100644 --- a/p2p/host/relay/addrsplosion.go +++ b/p2p/host/relay/addrsplosion.go @@ -34,17 +34,13 @@ func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { } func isDNSAddr(a ma.Multiaddr) bool { - dnsaddr := false - ma.ForEach(a, func(c ma.Component) bool { - switch c.Protocol().Code { + if first, _ := ma.SplitFirst(a); first != nil { + switch first.Protocol().Code { case 54, 55, 56: - dnsaddr = true + return true } - - return false - }) - - return dnsaddr + } + return false } // we have addrsplosion if for some protocol we advertise multiple ports on From 528c473840c7f24f93302133fd133fadd02cdcfc Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 13:53:27 +0300 Subject: [PATCH 1119/3965] filter relay addrs in address set cleanup --- p2p/host/relay/addrsplosion.go | 21 +++++++++++++++++++++ p2p/host/relay/relay.go | 4 +--- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/p2p/host/relay/addrsplosion.go b/p2p/host/relay/addrsplosion.go index 3eca947b0c..51617a3b80 100644 --- a/p2p/host/relay/addrsplosion.go +++ b/p2p/host/relay/addrsplosion.go @@ -3,6 +3,7 @@ package relay import ( "encoding/binary" + circuit "github.com/libp2p/go-libp2p-circuit" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" @@ -14,6 +15,10 @@ func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { var public, private []ma.Multiaddr for _, a := range pi.Addrs { + if isRelayAddr(a) { + continue + } + if manet.IsPublicAddr(a) || isDNSAddr(a) { public = append(public, a) continue @@ -33,6 +38,22 @@ func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { return pstore.PeerInfo{ID: pi.ID, Addrs: addrs} } +func isRelayAddr(a ma.Multiaddr) bool { + isRelay := false + + ma.ForEach(a, func(c ma.Component) bool { + switch c.Protocol().Code { + case circuit.P_CIRCUIT: + isRelay = true + return false + default: + return true + } + }) + + return isRelay +} + func isDNSAddr(a ma.Multiaddr) bool { if first, _ := ma.SplitFirst(a); first != nil { switch first.Protocol().Code { diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index d0af9d227a..2adea8fcc6 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -4,7 +4,6 @@ import ( "context" "time" - circuit "github.com/libp2p/go-libp2p-circuit" discovery "github.com/libp2p/go-libp2p-discovery" ma "github.com/multiformats/go-multiaddr" ) @@ -29,8 +28,7 @@ func Advertise(ctx context.Context, advertise discovery.Advertiser) { func Filter(addrs []ma.Multiaddr) []ma.Multiaddr { raddrs := make([]ma.Multiaddr, 0, len(addrs)) for _, addr := range addrs { - _, err := addr.ValueForProtocol(circuit.P_CIRCUIT) - if err == nil { + if isRelayAddr(addr) { continue } raddrs = append(raddrs, addr) From 21c4e1d298be6601a4921b20cec963a5ea8d8adf Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 13:58:10 +0300 Subject: [PATCH 1120/3965] test for privately bound port first when cleaning up addrsplosion --- p2p/host/relay/addrsplosion.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/host/relay/addrsplosion.go b/p2p/host/relay/addrsplosion.go index 51617a3b80..3b114a312a 100644 --- a/p2p/host/relay/addrsplosion.go +++ b/p2p/host/relay/addrsplosion.go @@ -142,15 +142,15 @@ func sanitizeAddrsplodedSet(public, private []ma.Multiaddr) []ma.Multiaddr { haveAddr := false for _, pa := range pas { - if pa.port == 4001 || pa.port == 4002 { - // it's a default port, use it + if _, ok := privports[pa.port]; ok { + // it matches a privately bound port, use it result = append(result, pa.addr) haveAddr = true continue } - if _, ok := privports[pa.port]; ok { - // it matches a privately bound port, use it + if pa.port == 4001 || pa.port == 4002 { + // it's a default port, use it result = append(result, pa.addr) haveAddr = true } From ff4b98a6fc70332a64d0a654a47f7c6ae218762a Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 13:58:45 +0300 Subject: [PATCH 1121/3965] some more addrsplosion tests --- p2p/host/relay/addrsplosion_test.go | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/p2p/host/relay/addrsplosion_test.go b/p2p/host/relay/addrsplosion_test.go index 0eb0c5be6c..9104fc3dde 100644 --- a/p2p/host/relay/addrsplosion_test.go +++ b/p2p/host/relay/addrsplosion_test.go @@ -48,6 +48,23 @@ func TestCleanupAddrs(t *testing.T) { t.Fatal("cleaned up set doesn't match expected") } + // test with default port addrsplosion but no private addrs + addrs = makeAddrList( + "/ip4/1.2.3.4/tcp/4001", + "/ip4/1.2.3.4/tcp/33333", + "/ip4/1.2.3.4/tcp/33334", + "/ip4/1.2.3.4/tcp/33335", + "/ip4/1.2.3.4/udp/4002/quic", + ) + clean = makeAddrList( + "/ip4/1.2.3.4/tcp/4001", + "/ip4/1.2.3.4/udp/4002/quic", + ) + pi = cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) + if !sameAddrs(clean, pi.Addrs) { + t.Fatal("cleaned up set doesn't match expected") + } + // test with non-standard port addrsplosion addrs = makeAddrList( "/ip4/127.0.0.1/tcp/12345", @@ -64,6 +81,16 @@ func TestCleanupAddrs(t *testing.T) { t.Fatal("cleaned up set doesn't match expected") } + // test with a squeaky clean address set + addrs = makeAddrList( + "/ip4/1.2.3.4/tcp/4001", + "/ip4/1.2.3.4/udp/4001/quic", + ) + clean = addrs + pi = cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) + if !sameAddrs(clean, pi.Addrs) { + t.Fatal("cleaned up set doesn't match expected") + } } func makeAddrList(strs ...string) []ma.Multiaddr { From a8d14f9b025d3073c60938257735e60ff8e23b89 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 14:09:56 +0300 Subject: [PATCH 1122/3965] use addresses from the peerstore if available --- p2p/host/relay/autorelay.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 894548c34a..b5816683b2 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -240,6 +240,12 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co // shuffle to randomize the order of queries shuffleRelays(pis) for _, pi := range pis[:maxq] { + // first check to see if we already know this peer from a previous query + addrs := ar.host.Peerstore().Addrs(pi.ID) + if len(addrs) > 0 { + resultCh <- queryResult{pi: pstore.PeerInfo{ID: pi.ID, Addrs: addrs}, err: nil} + continue + } go func(p peer.ID) { pi, err := ar.router.FindPeer(qctx, p) if err != nil { From 9b2731e5bbb495a76eb0f1a784c40d651c5f6a3e Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 19:48:28 +0300 Subject: [PATCH 1123/3965] used named constants for dns address protocols --- p2p/host/relay/addrsplosion.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/addrsplosion.go b/p2p/host/relay/addrsplosion.go index 3b114a312a..a36a01f63b 100644 --- a/p2p/host/relay/addrsplosion.go +++ b/p2p/host/relay/addrsplosion.go @@ -6,6 +6,7 @@ import ( circuit "github.com/libp2p/go-libp2p-circuit" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" + dns "github.com/multiformats/go-multiaddr-dns" manet "github.com/multiformats/go-multiaddr-net" ) @@ -57,7 +58,7 @@ func isRelayAddr(a ma.Multiaddr) bool { func isDNSAddr(a ma.Multiaddr) bool { if first, _ := ma.SplitFirst(a); first != nil { switch first.Protocol().Code { - case 54, 55, 56: + case dns.P_DNS4, dns.P_DNS6, dns.P_DNSADDR: return true } } From bf651ca7c73c68d8a0c15685475c2b71f1c0743d Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 12:50:42 +0300 Subject: [PATCH 1124/3965] use a single, NoDial context in identify push --- p2p/protocol/identify/id.go | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 9aac465bfa..acf0aa9020 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -156,19 +156,35 @@ func (ids *IDService) pushHandler(s inet.Stream) { } func (ids *IDService) Push() { + var wg sync.WaitGroup + + // we could make this context timeout-less since we are only opening a new + // stream over an existing connection. This would avoid the need for the + // supervisory goroutine below, but timeout-less contexts in network operations + // make me nervous. + ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) + ctx = inet.WithNoDial(ctx, "identify push") + for _, p := range ids.Host.Network().Peers() { + wg.Add(1) go func(p peer.ID) { - ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) - defer cancel() + defer wg.Done() + s, err := ids.Host.NewStream(ctx, p, IDPush) if err != nil { - log.Debugf("error opening push stream: %s", err.Error()) + log.Debugf("error opening push stream to %s: %s", p, err.Error()) return } ids.requestHandler(s) }(p) } + + // this supervisory goroutine is necessary to cancel the context + go func() { + wg.Wait() + cancel() + }() } func (ids *IDService) populateMessage(mes *pb.Identify, c inet.Conn) { From 8fa257cdf01d32d89485f008cee1a7aedf5dccd0 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 12:56:29 +0300 Subject: [PATCH 1125/3965] track context given in identify constructor and use it to gate identify push --- p2p/protocol/identify/id.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index acf0aa9020..d53c99eafe 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -47,6 +47,8 @@ const transientTTL = 10 * time.Second type IDService struct { Host host.Host + ctx context.Context + // connections undergoing identification // for wait purposes currid map[inet.Conn]chan struct{} @@ -64,6 +66,7 @@ type IDService struct { func NewIDService(ctx context.Context, h host.Host) *IDService { s := &IDService{ Host: h, + ctx: ctx, currid: make(map[inet.Conn]chan struct{}), observedAddrs: NewObservedAddrSet(ctx), } @@ -162,7 +165,7 @@ func (ids *IDService) Push() { // stream over an existing connection. This would avoid the need for the // supervisory goroutine below, but timeout-less contexts in network operations // make me nervous. - ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) + ctx, cancel := context.WithTimeout(ids.ctx, 15*time.Second) ctx = inet.WithNoDial(ctx, "identify push") for _, p := range ids.Host.Network().Peers() { From 7502fc44c9a199a42029ee523786458a0d34dba1 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 13:57:52 +0300 Subject: [PATCH 1126/3965] add a background task in basic host that periodically pushes identify if needed --- p2p/host/basic/basic_host.go | 73 +++++++++++++++++++++++++++++++++++- 1 file changed, 72 insertions(+), 1 deletion(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 0725fa702d..3369554a77 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -4,6 +4,7 @@ import ( "context" "io" "net" + "sync" "time" logging "github.com/ipfs/go-log" @@ -70,6 +71,10 @@ type BasicHost struct { negtimeout time.Duration proc goprocess.Process + + bgcancel func() + mx sync.Mutex + lastAddrs []ma.Multiaddr } // HostOpts holds options that can be passed to NewHost in order to @@ -164,6 +169,11 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, net.SetConnHandler(h.newConnHandler) net.SetStreamHandler(h.newStreamHandler) + + bgctx, cancel := context.WithCancel(ctx) + h.bgcancel = cancel + go h.background(bgctx) + return h, nil } @@ -263,7 +273,67 @@ func (h *BasicHost) newStreamHandler(s inet.Stream) { // PushIdentify pushes an identify update through the identify push protocol // Warning: this interface is unstable and may disappear in the future. func (h *BasicHost) PushIdentify() { - h.ids.Push() + push := false + + h.mx.Lock() + addrs := h.Addrs() + if !sameAddrs(addrs, h.lastAddrs) { + push = true + h.lastAddrs = addrs + } + h.mx.Unlock() + + if push { + h.ids.Push() + } +} + +func (h *BasicHost) background(ctx context.Context) { + // periodically schedules an IdentifyPush to update our peers for changes + // in our address set (if needed) + ticker := time.NewTicker(1 * time.Minute) + defer ticker.Stop() + + // initialize lastAddrs + h.mx.Lock() + if h.lastAddrs == nil { + h.lastAddrs = h.Addrs() + } + h.mx.Unlock() + + for { + select { + case <-ticker.C: + h.PushIdentify() + + case <-ctx.Done(): + return + } + } +} + +func sameAddrs(a, b []ma.Multiaddr) bool { + if len(a) != len(b) { + return false + } + + // this is O(n*m), might be worth using a map to turn into O(n+m) + for _, addr := range a { + if !findAddr(addr, b) { + return false + } + } + + return true +} + +func findAddr(addr ma.Multiaddr, addrs []ma.Multiaddr) bool { + for _, xaddr := range addrs { + if addr.Equal(xaddr) { + return true + } + } + return false } // ID returns the (local) peer.ID associated with this Host @@ -646,6 +716,7 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { // Close shuts down the Host's services (network, etc). func (h *BasicHost) Close() error { + h.bgcancel() return h.proc.Close() } From 584590975466e52a9a3eb021ab83c501b25421db Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 14:08:02 +0300 Subject: [PATCH 1127/3965] add initialization delay in basic host background task --- p2p/host/basic/basic_host.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 3369554a77..62aad3e280 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -289,6 +289,9 @@ func (h *BasicHost) PushIdentify() { } func (h *BasicHost) background(ctx context.Context) { + // wait a bit for the host to initialize (avoid race with libp2p constructor) + time.Sleep(1 * time.Second) + // periodically schedules an IdentifyPush to update our peers for changes // in our address set (if needed) ticker := time.NewTicker(1 * time.Minute) From c9b2f468ba794236a8adefcc7ee5fb0554a7804a Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 14:17:07 +0300 Subject: [PATCH 1128/3965] reduce peer count in TestFuzzManyPeers when running under the race detector --- p2p/net/mock/mock_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index fa53c16ec5..473e4424df 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -588,7 +588,7 @@ func TestLimitedStreams(t *testing.T) { func TestFuzzManyPeers(t *testing.T) { peerCount := 50000 if detectrace.WithRace() { - peerCount = 1000 + peerCount = 100 } for i := 0; i < peerCount; i++ { _, err := FullMeshConnected(context.Background(), 2) From c73f68bdb4714ca330b6c92192d823efbbcae9fd Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 13 Apr 2019 19:32:45 +0300 Subject: [PATCH 1129/3965] explicit Start method for basic host --- config/config.go | 3 +++ p2p/host/basic/basic_host.go | 21 ++++++++++++--------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/config/config.go b/config/config.go index 0b8c676bd4..79b04420b3 100644 --- a/config/config.go +++ b/config/config.go @@ -228,6 +228,9 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } } + // start the host background tasks + h.Start() + if router != nil { return routed.Wrap(h, router), nil } diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 62aad3e280..3aa64e948f 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -72,7 +72,8 @@ type BasicHost struct { proc goprocess.Process - bgcancel func() + ctx context.Context + cancel func() mx sync.Mutex lastAddrs []ma.Multiaddr } @@ -171,8 +172,8 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, net.SetStreamHandler(h.newStreamHandler) bgctx, cancel := context.WithCancel(ctx) - h.bgcancel = cancel - go h.background(bgctx) + h.ctx = bgctx + h.cancel = cancel return h, nil } @@ -214,6 +215,11 @@ func New(net inet.Network, opts ...interface{}) *BasicHost { return h } +// Start starts background tasks in the host +func (h *BasicHost) Start() { + go h.background() +} + // newConnHandler is the remote-opened conn handler for inet.Network func (h *BasicHost) newConnHandler(c inet.Conn) { // Clear protocols on connecting to new peer to avoid issues caused @@ -288,10 +294,7 @@ func (h *BasicHost) PushIdentify() { } } -func (h *BasicHost) background(ctx context.Context) { - // wait a bit for the host to initialize (avoid race with libp2p constructor) - time.Sleep(1 * time.Second) - +func (h *BasicHost) background() { // periodically schedules an IdentifyPush to update our peers for changes // in our address set (if needed) ticker := time.NewTicker(1 * time.Minute) @@ -309,7 +312,7 @@ func (h *BasicHost) background(ctx context.Context) { case <-ticker.C: h.PushIdentify() - case <-ctx.Done(): + case <-h.ctx.Done(): return } } @@ -719,7 +722,7 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { // Close shuts down the Host's services (network, etc). func (h *BasicHost) Close() error { - h.bgcancel() + h.cancel() return h.proc.Close() } From 3697552406a7261757f208c95718e43f687ed4d0 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 13:03:18 +0300 Subject: [PATCH 1130/3965] handle misbehaving peers in identify push --- p2p/protocol/identify/id.go | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index d53c99eafe..cfc4d08897 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -161,11 +161,7 @@ func (ids *IDService) pushHandler(s inet.Stream) { func (ids *IDService) Push() { var wg sync.WaitGroup - // we could make this context timeout-less since we are only opening a new - // stream over an existing connection. This would avoid the need for the - // supervisory goroutine below, but timeout-less contexts in network operations - // make me nervous. - ctx, cancel := context.WithTimeout(ids.ctx, 15*time.Second) + ctx, cancel := context.WithTimeout(ids.ctx, 30*time.Second) ctx = inet.WithNoDial(ctx, "identify push") for _, p := range ids.Host.Network().Peers() { @@ -179,7 +175,18 @@ func (ids *IDService) Push() { return } - ids.requestHandler(s) + rch := make(chan struct{}, 1) + go func() { + ids.requestHandler(s) + rch <- struct{}{} + }() + + select { + case <-rch: + case <-ctx.Done(): + // this is taking too long, abort! + s.Reset() + } }(p) } From ebc5d16f5dce3dbb54901f31776d0659fbf97744 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 13:08:06 +0300 Subject: [PATCH 1131/3965] use a map in sameAddrs to avoid quadratic behaviour --- p2p/host/basic/basic_host.go | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 3aa64e948f..9fa5441fa7 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -323,9 +323,14 @@ func sameAddrs(a, b []ma.Multiaddr) bool { return false } - // this is O(n*m), might be worth using a map to turn into O(n+m) + bmap := make(map[string]struct{}) + for _, addr := range b { + bmap[string(addr.Bytes())] = struct{}{} + } + for _, addr := range a { - if !findAddr(addr, b) { + _, ok := bmap[string(addr.Bytes())] + if !ok { return false } } @@ -333,15 +338,6 @@ func sameAddrs(a, b []ma.Multiaddr) bool { return true } -func findAddr(addr ma.Multiaddr, addrs []ma.Multiaddr) bool { - for _, xaddr := range addrs { - if addr.Equal(xaddr) { - return true - } - } - return false -} - // ID returns the (local) peer.ID associated with this Host func (h *BasicHost) ID() peer.ID { return h.Network().LocalPeer() From f17a4a86677c48b1cced0b0e65b8218a280307c6 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 19:39:00 +0300 Subject: [PATCH 1132/3965] preallocate map in sameAddrs --- p2p/host/basic/basic_host.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 9fa5441fa7..1b11bbdc80 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -323,7 +323,7 @@ func sameAddrs(a, b []ma.Multiaddr) bool { return false } - bmap := make(map[string]struct{}) + bmap := make(map[string]struct{}, len(b)) for _, addr := range b { bmap[string(addr.Bytes())] = struct{}{} } From bd22c49b0d380e789369a329eb7c638f3a5375a3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 19 Apr 2019 20:49:54 +0300 Subject: [PATCH 1133/3965] remove redundant private addr check when constructing our relay address set findRelays cleans up address sets now for addrsplosion, and this removes private addrs as well. --- p2p/host/relay/autorelay.go | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index b5816683b2..127ef99b66 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -315,10 +315,8 @@ func (ar *AutoRelay) doUpdateAddrs() { } for _, addr := range pi.Addrs { - if !manet.IsPrivateAddr(addr) { - pub := addr.Encapsulate(circuit) - raddrs = append(raddrs, pub) - } + pub := addr.Encapsulate(circuit) + raddrs = append(raddrs, pub) } } From f4f924e1d4bd28007ef1ff4e07a9684af0fa8992 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 00:04:26 +0300 Subject: [PATCH 1134/3965] don't track relay addrs, use the peerstore --- p2p/host/relay/addrsplosion.go | 10 ++++------ p2p/host/relay/addrsplosion_test.go | 23 ++++++++++------------- p2p/host/relay/autorelay.go | 21 +++++++++------------ 3 files changed, 23 insertions(+), 31 deletions(-) diff --git a/p2p/host/relay/addrsplosion.go b/p2p/host/relay/addrsplosion.go index a36a01f63b..89965bca41 100644 --- a/p2p/host/relay/addrsplosion.go +++ b/p2p/host/relay/addrsplosion.go @@ -4,7 +4,6 @@ import ( "encoding/binary" circuit "github.com/libp2p/go-libp2p-circuit" - pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" dns "github.com/multiformats/go-multiaddr-dns" manet "github.com/multiformats/go-multiaddr-net" @@ -12,10 +11,10 @@ import ( // This function cleans up a relay's address set to remove private addresses and curtail // addrsplosion. -func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { +func cleanupAddressSet(addrs []ma.Multiaddr) []ma.Multiaddr { var public, private []ma.Multiaddr - for _, a := range pi.Addrs { + for _, a := range addrs { if isRelayAddr(a) { continue } @@ -32,11 +31,10 @@ func cleanupAddressSet(pi pstore.PeerInfo) pstore.PeerInfo { } if !hasAddrsplosion(public) { - return pstore.PeerInfo{ID: pi.ID, Addrs: public} + return public } - addrs := sanitizeAddrsplodedSet(public, private) - return pstore.PeerInfo{ID: pi.ID, Addrs: addrs} + return sanitizeAddrsplodedSet(public, private) } func isRelayAddr(a ma.Multiaddr) bool { diff --git a/p2p/host/relay/addrsplosion_test.go b/p2p/host/relay/addrsplosion_test.go index 9104fc3dde..cdb10def63 100644 --- a/p2p/host/relay/addrsplosion_test.go +++ b/p2p/host/relay/addrsplosion_test.go @@ -1,10 +1,8 @@ package relay import ( - "fmt" "testing" - pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" _ "github.com/multiformats/go-multiaddr-dns" ) @@ -24,9 +22,8 @@ func TestCleanupAddrs(t *testing.T) { "/dnsaddr/somedomain.com/tcp/4002/ws", ) - pi := cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) - if !sameAddrs(clean, pi.Addrs) { - fmt.Println(pi.Addrs) + r := cleanupAddressSet(addrs) + if !sameAddrs(clean, r) { t.Fatal("cleaned up set doesn't match expected") } @@ -43,8 +40,8 @@ func TestCleanupAddrs(t *testing.T) { "/ip4/1.2.3.4/tcp/4001", "/ip4/1.2.3.4/udp/4002/quic", ) - pi = cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) - if !sameAddrs(clean, pi.Addrs) { + r = cleanupAddressSet(addrs) + if !sameAddrs(clean, r) { t.Fatal("cleaned up set doesn't match expected") } @@ -60,8 +57,8 @@ func TestCleanupAddrs(t *testing.T) { "/ip4/1.2.3.4/tcp/4001", "/ip4/1.2.3.4/udp/4002/quic", ) - pi = cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) - if !sameAddrs(clean, pi.Addrs) { + r = cleanupAddressSet(addrs) + if !sameAddrs(clean, r) { t.Fatal("cleaned up set doesn't match expected") } @@ -76,8 +73,8 @@ func TestCleanupAddrs(t *testing.T) { clean = makeAddrList( "/ip4/1.2.3.4/tcp/12345", ) - pi = cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) - if !sameAddrs(clean, pi.Addrs) { + r = cleanupAddressSet(addrs) + if !sameAddrs(clean, r) { t.Fatal("cleaned up set doesn't match expected") } @@ -87,8 +84,8 @@ func TestCleanupAddrs(t *testing.T) { "/ip4/1.2.3.4/udp/4001/quic", ) clean = addrs - pi = cleanupAddressSet(pstore.PeerInfo{Addrs: addrs}) - if !sameAddrs(clean, pi.Addrs) { + r = cleanupAddressSet(addrs) + if !sameAddrs(clean, r) { t.Fatal("cleaned up set doesn't match expected") } } diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 127ef99b66..5a88f1541c 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -41,7 +41,7 @@ type AutoRelay struct { disconnect chan struct{} mx sync.Mutex - relays map[peer.ID]pstore.PeerInfo + relays map[peer.ID]struct{} addrs []ma.Multiaddr } @@ -51,7 +51,7 @@ func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discover discover: discover, router: router, addrsF: bhost.AddrsFactory, - relays: make(map[peer.ID]pstore.PeerInfo), + relays: make(map[peer.ID]struct{}), disconnect: make(chan struct{}, 1), } ar.autonat = autonat.NewAutoNAT(ctx, bhost, ar.baseAddrs) @@ -179,7 +179,7 @@ again: log.Debugf("connected to relay %s", pi.ID) ar.mx.Lock() - ar.relays[pi.ID] = pi + ar.relays[pi.ID] = struct{}{} haveRelays++ ar.mx.Unlock() @@ -261,12 +261,7 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co case qr := <-resultCh: rcount++ if qr.err == nil { - pi := cleanupAddressSet(qr.pi) - if len(pi.Addrs) > 0 { - result = append(result, pi) - } else { - log.Debugf("ignoring relay peer %s: cleaned up address set is empty", pi.ID) - } + result = append(result, qr.pi) } case <-qctx.Done(): @@ -308,13 +303,15 @@ func (ar *AutoRelay) doUpdateAddrs() { } // add relay specific addrs to the list - for _, pi := range ar.relays { - circuit, err := ma.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", pi.ID.Pretty())) + for p := range ar.relays { + addrs := cleanupAddressSet(ar.host.Peerstore().Addrs(p)) + + circuit, err := ma.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", p.Pretty())) if err != nil { panic(err) } - for _, addr := range pi.Addrs { + for _, addr := range addrs { pub := addr.Encapsulate(circuit) raddrs = append(raddrs, pub) } From c09717275bf5ea10eceef25971769233b8087d0c Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 00:54:44 +0300 Subject: [PATCH 1135/3965] compute relay address set dynamically --- p2p/host/relay/autorelay.go | 117 ++++++++++++++++-------------------- 1 file changed, 53 insertions(+), 64 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 5a88f1541c..dc811dd8cc 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -42,7 +42,7 @@ type AutoRelay struct { mx sync.Mutex relays map[peer.ID]struct{} - addrs []ma.Multiaddr + status autonat.NATStatus } func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer, router routing.PeerRouting) *AutoRelay { @@ -53,6 +53,7 @@ func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discover addrsF: bhost.AddrsFactory, relays: make(map[peer.ID]struct{}), disconnect: make(chan struct{}, 1), + status: autonat.NATStatusUnknown, } ar.autonat = autonat.NewAutoNAT(ctx, bhost, ar.baseAddrs) bhost.AddrsFactory = ar.hostAddrs @@ -61,20 +62,14 @@ func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discover return ar } -func (ar *AutoRelay) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { - ar.mx.Lock() - defer ar.mx.Unlock() - if ar.addrs != nil && ar.autonat.Status() == autonat.NATStatusPrivate { - return ar.addrs - } else { - return ar.addrsF(addrs) - } -} - func (ar *AutoRelay) baseAddrs() []ma.Multiaddr { return ar.addrsF(ar.host.AllAddrs()) } +func (ar *AutoRelay) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { + return ar.relayAddrs(ar.addrsF(addrs)) +} + func (ar *AutoRelay) background(ctx context.Context) { select { case <-time.After(autonat.AutoNATBootDelay + BootDelay): @@ -89,37 +84,37 @@ func (ar *AutoRelay) background(ctx context.Context) { wait := autonat.AutoNATRefreshInterval switch ar.autonat.Status() { case autonat.NATStatusUnknown: + ar.mx.Lock() + ar.status = autonat.NATStatusUnknown + ar.mx.Unlock() wait = autonat.AutoNATRetryInterval case autonat.NATStatusPublic: - // invalidate addrs ar.mx.Lock() - if ar.addrs != nil { - ar.addrs = nil + if ar.status != autonat.NATStatusPublic { push = true } + ar.status = autonat.NATStatusPublic ar.mx.Unlock() - // if we had previously announced relay addrs, push our public addrs - if push { - push = false - ar.host.PushIdentify() + case autonat.NATStatusPrivate: + update := ar.findRelays(ctx) + ar.mx.Lock() + if update || ar.status != autonat.NATStatusPrivate { + push = true } + ar.status = autonat.NATStatusPrivate + ar.mx.Unlock() + } - case autonat.NATStatusPrivate: - push = false // clear, findRelays pushes as needed - ar.findRelays(ctx) + if push { + push = false + ar.host.PushIdentify() } select { case <-ar.disconnect: - // invalidate addrs - ar.mx.Lock() - if ar.addrs != nil { - ar.addrs = nil - push = true - } - ar.mx.Unlock() + push = true case <-time.After(wait): case <-ctx.Done(): return @@ -127,21 +122,13 @@ func (ar *AutoRelay) background(ctx context.Context) { } } -func (ar *AutoRelay) findRelays(ctx context.Context) { +func (ar *AutoRelay) findRelays(ctx context.Context) bool { again: ar.mx.Lock() haveRelays := len(ar.relays) if haveRelays >= DesiredRelays { ar.mx.Unlock() - // this dance is necessary to cover the Private->Public->Private transition - // where we were already connected to enough relays while private and dropped - // the addrs while public. - // Note tht we don't need the lock for reading addrs, as the only writer is - // the current goroutine. - if ar.addrs == nil { - ar.updateAddrs() - } - return + return false } need := DesiredRelays - len(ar.relays) ar.mx.Unlock() @@ -153,7 +140,16 @@ again: cancel() if err != nil { log.Debugf("error discovering relays: %s", err.Error()) - return + + if haveRelays == 0 { + log.Debug("no relays connected; retrying in 30s") + select { + case <-time.After(30 * time.Second): + goto again + case <-ctx.Done(): + return false + } + } } log.Debugf("discovered %d relays", len(pis)) @@ -201,13 +197,11 @@ again: case <-time.After(30 * time.Second): goto again case <-ctx.Done(): - return + return false } } - if update > 0 || ar.addrs == nil { - ar.updateAddrs() - } + return update > 0 } func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, count, maxq int) []pstore.PeerInfo { @@ -273,33 +267,28 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co return result } -func (ar *AutoRelay) updateAddrs() { - ar.doUpdateAddrs() - ar.host.PushIdentify() -} - -// This function updates our NATed advertised addrs (ar.addrs) -// The public addrs are rewritten so that they only retain the public IP part; they -// become undialable but are useful as a hint to the dialer to determine whether or not -// to dial private addrs. -// The non-public addrs are included verbatim so that peers behind the same NAT/firewall -// can still dial us directly. -// On top of those, we add the relay-specific addrs for the relays to which we are -// connected. For each non-private relay addr, we encapsulate the p2p-circuit addr -// through which we can be dialed. -func (ar *AutoRelay) doUpdateAddrs() { +// This function is computes the NATed relay addrs when our status is private: +// - The public addrs are removed from the address set. +// - The non-public addrs are included verbatim so that peers behind the same NAT/firewall +// can still dial us directly. +// - On top of those, we add the relay-specific addrs for the relays to which we are +// connected. For each non-private relay addr, we encapsulate the p2p-circuit addr +// through which we can be dialed. +func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { ar.mx.Lock() defer ar.mx.Unlock() - addrs := ar.baseAddrs() + if ar.status != autonat.NATStatusPrivate { + return addrs + } + raddrs := make([]ma.Multiaddr, 0, len(addrs)+len(ar.relays)) - // remove our public addresses from the list + // only keep private addrs from the original addr set for _, addr := range addrs { - if manet.IsPublicAddr(addr) { - continue + if manet.IsPrivateAddr(addr) { + raddrs = append(raddrs, addr) } - raddrs = append(raddrs, addr) } // add relay specific addrs to the list @@ -317,7 +306,7 @@ func (ar *AutoRelay) doUpdateAddrs() { } } - ar.addrs = raddrs + return raddrs } func shuffleRelays(pis []pstore.PeerInfo) { From 4727d5b8490438869d9f90edb2ebdbf8acce5032 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 01:40:29 +0300 Subject: [PATCH 1136/3965] don't preallocate result array, we don't know how long it will be --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index dc811dd8cc..73e126a2d7 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -282,7 +282,7 @@ func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { return addrs } - raddrs := make([]ma.Multiaddr, 0, len(addrs)+len(ar.relays)) + var raddrs []ma.Multiaddr // only keep private addrs from the original addr set for _, addr := range addrs { From 5c9299a45cec4d334c5440273a2e048b01afb04a Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 01:47:41 +0300 Subject: [PATCH 1137/3965] pacify the race detector --- p2p/host/relay/autorelay_test.go | 2 -- 1 file changed, 2 deletions(-) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index d86fdf6b35..ff3d7601ab 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -146,9 +146,7 @@ func connect(t *testing.T, a, b host.Host) { func TestAutoRelay(t *testing.T) { //t.Skip("fails 99% of the time") - save := manet.Private4 manet.Private4 = []*net.IPNet{} - defer func() { manet.Private4 = save }() ctx, cancel := context.WithCancel(context.Background()) defer cancel() From 2af204e40d1fa10bef6bf37552bad88906a2bc1c Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 20 Apr 2019 12:06:17 +0900 Subject: [PATCH 1138/3965] when ListenUDP fails once, try again next time --- p2p/transport/quic/transport.go | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 2351959c00..d9f2c8f09c 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -31,26 +31,30 @@ var quicConfig = &quic.Config{ } type connManager struct { - connIPv4Once sync.Once - connIPv4 net.PacketConn + mutex sync.Mutex - connIPv6Once sync.Once - connIPv6 net.PacketConn + connIPv4 net.PacketConn + connIPv6 net.PacketConn } func (c *connManager) GetConnForAddr(network string) (net.PacketConn, error) { + c.mutex.Lock() + defer c.mutex.Unlock() + switch network { case "udp4": + if c.connIPv4 != nil { + return c.connIPv4, nil + } var err error - c.connIPv4Once.Do(func() { - c.connIPv4, err = c.createConn(network, "0.0.0.0:0") - }) + c.connIPv4, err = c.createConn(network, "0.0.0.0:0") return c.connIPv4, err case "udp6": + if c.connIPv6 != nil { + return c.connIPv6, nil + } var err error - c.connIPv6Once.Do(func() { - c.connIPv6, err = c.createConn(network, ":0") - }) + c.connIPv6, err = c.createConn(network, ":0") return c.connIPv6, err default: return nil, fmt.Errorf("unsupported network: %s", network) From 1a8111970b8d45b6eee8151d209735878ebcfb24 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 06:25:37 +0300 Subject: [PATCH 1139/3965] reduce scope of the lock, pre-allocate result slice in relayAddrs --- p2p/host/relay/autorelay.go | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 73e126a2d7..02f2fff8a2 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -276,13 +276,18 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co // through which we can be dialed. func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { ar.mx.Lock() - defer ar.mx.Unlock() - if ar.status != autonat.NATStatusPrivate { + ar.mx.Unlock() return addrs } - var raddrs []ma.Multiaddr + relays := make([]peer.ID, 0, len(ar.relays)) + for p := range ar.relays { + relays = append(relays, p) + } + ar.mx.Unlock() + + raddrs := make([]ma.Multiaddr, 0, 4*len(relays)+2) // only keep private addrs from the original addr set for _, addr := range addrs { @@ -292,7 +297,7 @@ func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { } // add relay specific addrs to the list - for p := range ar.relays { + for _, p := range relays { addrs := cleanupAddressSet(ar.host.Peerstore().Addrs(p)) circuit, err := ma.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", p.Pretty())) From f9e182f7477adeca2b7cb51796b5621ceabbe17a Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 06:31:51 +0300 Subject: [PATCH 1140/3965] gate max number of retries in findRelays --- p2p/host/relay/autorelay.go | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 02f2fff8a2..ad54e4d901 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -123,6 +123,8 @@ func (ar *AutoRelay) background(ctx context.Context) { } func (ar *AutoRelay) findRelays(ctx context.Context) bool { + retry := 0 + again: ar.mx.Lock() haveRelays := len(ar.relays) @@ -142,6 +144,12 @@ again: log.Debugf("error discovering relays: %s", err.Error()) if haveRelays == 0 { + retry++ + if retry > 5 { + log.Debug("no relays connected; giving up") + return false + } + log.Debug("no relays connected; retrying in 30s") select { case <-time.After(30 * time.Second): @@ -192,6 +200,12 @@ again: if haveRelays == 0 { // we failed to find any relays and we are not connected to any! // wait a little and try again, the discovery query might have returned only dead peers + retry++ + if retry > 5 { + log.Debug("no relays connected; giving up") + return false + } + log.Debug("no relays connected; retrying in 30s") select { case <-time.After(30 * time.Second): From 4629431a12fcf04e218648240fa9deb6746b4299 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 11:17:10 +0300 Subject: [PATCH 1141/3965] some tweaks - select 25 of 50 relays instead of 20 - increase connect timeout to 30s --- p2p/host/relay/autorelay.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index ad54e4d901..1de60f507a 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -162,7 +162,7 @@ again: log.Debugf("discovered %d relays", len(pis)) - pis = ar.selectRelays(ctx, pis, 20, 50) + pis = ar.selectRelays(ctx, pis, 25, 50) update := 0 for _, pi := range pis { @@ -173,7 +173,7 @@ again: } ar.mx.Unlock() - cctx, cancel := context.WithTimeout(ctx, 15*time.Second) + cctx, cancel := context.WithTimeout(ctx, 30*time.Second) err = ar.host.Connect(cctx, pi) cancel() if err != nil { @@ -254,6 +254,8 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co resultCh <- queryResult{pi: pstore.PeerInfo{ID: pi.ID, Addrs: addrs}, err: nil} continue } + + // no known addrs, do a query go func(p peer.ID) { pi, err := ar.router.FindPeer(qctx, p) if err != nil { From 35e805dc4120981d3ed9cba9a276bf559cdf36cb Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 11:50:06 +0300 Subject: [PATCH 1142/3965] add ignore list to account for connection failures --- p2p/host/relay/autorelay.go | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 1de60f507a..dcaa239b71 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -123,17 +123,17 @@ func (ar *AutoRelay) background(ctx context.Context) { } func (ar *AutoRelay) findRelays(ctx context.Context) bool { + ignore := make(map[peer.ID]struct{}) retry := 0 again: ar.mx.Lock() haveRelays := len(ar.relays) + ar.mx.Unlock() if haveRelays >= DesiredRelays { - ar.mx.Unlock() return false } - need := DesiredRelays - len(ar.relays) - ar.mx.Unlock() + need := DesiredRelays - haveRelays limit := 1000 @@ -162,7 +162,7 @@ again: log.Debugf("discovered %d relays", len(pis)) - pis = ar.selectRelays(ctx, pis, 25, 50) + pis = ar.selectRelays(ctx, ignore, pis, 25, 50) update := 0 for _, pi := range pis { @@ -178,6 +178,7 @@ again: cancel() if err != nil { log.Debugf("error connecting to relay %s: %s", pi.ID, err.Error()) + ignore[pi.ID] = struct{}{} continue } @@ -206,6 +207,12 @@ again: return false } + // if we failed to select any relays, clear the ignore list as some of the dead + // may have come back + if len(pis) == 0 { + ignore = make(map[peer.ID]struct{}) + } + log.Debug("no relays connected; retrying in 30s") select { case <-time.After(30 * time.Second): @@ -218,7 +225,7 @@ again: return update > 0 } -func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, count, maxq int) []pstore.PeerInfo { +func (ar *AutoRelay) selectRelays(ctx context.Context, ignore map[peer.ID]struct{}, pis []pstore.PeerInfo, count, maxq int) []pstore.PeerInfo { // TODO better relay selection strategy; this just selects random relays // but we should probably use ping latency as the selection metric @@ -248,7 +255,13 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo, co // shuffle to randomize the order of queries shuffleRelays(pis) for _, pi := range pis[:maxq] { - // first check to see if we already know this peer from a previous query + // check if it is in the ignore list + if _, ok := ignore[pi.ID]; ok { + maxq-- // decrement this for the result collection loop, we are doing one less query + continue + } + + // check to see if we already know this peer from a previous query addrs := ar.host.Peerstore().Addrs(pi.ID) if len(addrs) > 0 { resultCh <- queryResult{pi: pstore.PeerInfo{ID: pi.ID, Addrs: addrs}, err: nil} From 8d073cec9e0d0ac70f46ae60d109ed8b7edc23a6 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 20 Apr 2019 13:10:28 +0300 Subject: [PATCH 1143/3965] kill the parallel query logic in selectRelays; let it be random the presence of stashed query results from discovery in the peerstore _biases_ the selection towards fully DHT nodes, which penalizes dedicated relays. --- p2p/host/relay/autorelay.go | 89 ++++++------------------------------- 1 file changed, 14 insertions(+), 75 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index dcaa239b71..0c8b681256 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -123,7 +123,6 @@ func (ar *AutoRelay) background(ctx context.Context) { } func (ar *AutoRelay) findRelays(ctx context.Context) bool { - ignore := make(map[peer.ID]struct{}) retry := 0 again: @@ -162,7 +161,7 @@ again: log.Debugf("discovered %d relays", len(pis)) - pis = ar.selectRelays(ctx, ignore, pis, 25, 50) + pis = ar.selectRelays(ctx, pis) update := 0 for _, pi := range pis { @@ -173,12 +172,21 @@ again: } ar.mx.Unlock() - cctx, cancel := context.WithTimeout(ctx, 30*time.Second) + cctx, cancel := context.WithTimeout(ctx, 60*time.Second) + + if len(pi.Addrs) == 0 { + pi, err = ar.router.FindPeer(cctx, pi.ID) + if err != nil { + log.Debugf("error finding relay peer %s: %s", pi.ID, err.Error()) + cancel() + continue + } + } + err = ar.host.Connect(cctx, pi) cancel() if err != nil { log.Debugf("error connecting to relay %s: %s", pi.ID, err.Error()) - ignore[pi.ID] = struct{}{} continue } @@ -207,12 +215,6 @@ again: return false } - // if we failed to select any relays, clear the ignore list as some of the dead - // may have come back - if len(pis) == 0 { - ignore = make(map[peer.ID]struct{}) - } - log.Debug("no relays connected; retrying in 30s") select { case <-time.After(30 * time.Second): @@ -225,75 +227,12 @@ again: return update > 0 } -func (ar *AutoRelay) selectRelays(ctx context.Context, ignore map[peer.ID]struct{}, pis []pstore.PeerInfo, count, maxq int) []pstore.PeerInfo { +func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo) []pstore.PeerInfo { // TODO better relay selection strategy; this just selects random relays // but we should probably use ping latency as the selection metric - if len(pis) == 0 { - return pis - } - - if len(pis) < count { - count = len(pis) - } - - if len(pis) < maxq { - maxq = len(pis) - } - - // only select relays that can be found by routing - type queryResult struct { - pi pstore.PeerInfo - err error - } - result := make([]pstore.PeerInfo, 0, count) - resultCh := make(chan queryResult, maxq) - - qctx, cancel := context.WithTimeout(ctx, 30*time.Second) - defer cancel() - - // shuffle to randomize the order of queries shuffleRelays(pis) - for _, pi := range pis[:maxq] { - // check if it is in the ignore list - if _, ok := ignore[pi.ID]; ok { - maxq-- // decrement this for the result collection loop, we are doing one less query - continue - } - - // check to see if we already know this peer from a previous query - addrs := ar.host.Peerstore().Addrs(pi.ID) - if len(addrs) > 0 { - resultCh <- queryResult{pi: pstore.PeerInfo{ID: pi.ID, Addrs: addrs}, err: nil} - continue - } - - // no known addrs, do a query - go func(p peer.ID) { - pi, err := ar.router.FindPeer(qctx, p) - if err != nil { - log.Debugf("error finding relay peer %s: %s", p, err.Error()) - } - resultCh <- queryResult{pi: pi, err: err} - }(pi.ID) - } - - rcount := 0 - for len(result) < count && rcount < maxq { - select { - case qr := <-resultCh: - rcount++ - if qr.err == nil { - result = append(result, qr.pi) - } - - case <-qctx.Done(): - break - } - } - - shuffleRelays(result) - return result + return pis } // This function is computes the NATed relay addrs when our status is private: From 776a53a370e748a0f6d14f3cc4d572343f7164fe Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 20 Apr 2019 14:23:23 -0700 Subject: [PATCH 1144/3965] autorelay: break findRelays into multiple functions and avoid the goto (@stebalien is picky and opinionated...) --- p2p/host/relay/autorelay.go | 151 ++++++++++++++++++------------------ 1 file changed, 77 insertions(+), 74 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 0c8b681256..669f65fdc6 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -123,108 +123,111 @@ func (ar *AutoRelay) background(ctx context.Context) { } func (ar *AutoRelay) findRelays(ctx context.Context) bool { - retry := 0 - -again: - ar.mx.Lock() - haveRelays := len(ar.relays) - ar.mx.Unlock() - if haveRelays >= DesiredRelays { + if ar.numRelays() >= DesiredRelays { return false } - need := DesiredRelays - haveRelays - - limit := 1000 - - dctx, cancel := context.WithTimeout(ctx, 30*time.Second) - pis, err := discovery.FindPeers(dctx, ar.discover, RelayRendezvous, limit) - cancel() - if err != nil { - log.Debugf("error discovering relays: %s", err.Error()) - - if haveRelays == 0 { - retry++ - if retry > 5 { - log.Debug("no relays connected; giving up") - return false - } + update := false + for retry := 0; retry < 5; retry++ { + if retry > 0 { log.Debug("no relays connected; retrying in 30s") select { case <-time.After(30 * time.Second): - goto again case <-ctx.Done(): - return false + return update } } + + update = ar.findRelaysOnce(ctx) || update + if ar.numRelays() > 0 { + return update + } } + return update +} +func (ar *AutoRelay) findRelaysOnce(ctx context.Context) bool { + pis, err := ar.discoverRelays(ctx) + if err != nil { + log.Debugf("error discovering relays: %s", err) + return false + } log.Debugf("discovered %d relays", len(pis)) - pis = ar.selectRelays(ctx, pis) - update := 0 + log.Debugf("selected %d relays", len(pis)) + update := false for _, pi := range pis { - ar.mx.Lock() - if _, ok := ar.relays[pi.ID]; ok { - ar.mx.Unlock() - continue - } - ar.mx.Unlock() + update = ar.tryRelay(ctx, pi) || update + } + return update +} - cctx, cancel := context.WithTimeout(ctx, 60*time.Second) +func (ar *AutoRelay) numRelays() int { + ar.mx.Lock() + defer ar.mx.Unlock() + return len(ar.relays) +} - if len(pi.Addrs) == 0 { - pi, err = ar.router.FindPeer(cctx, pi.ID) - if err != nil { - log.Debugf("error finding relay peer %s: %s", pi.ID, err.Error()) - cancel() - continue - } - } +// usingRelay returns if we're currently using the given relay. +func (ar *AutoRelay) usingRelay(p peer.ID) bool { + ar.mx.Lock() + defer ar.mx.Unlock() + _, ok := ar.relays[p] + return ok +} - err = ar.host.Connect(cctx, pi) - cancel() - if err != nil { - log.Debugf("error connecting to relay %s: %s", pi.ID, err.Error()) - continue - } +// addRelay adds the given relay to our set of relays. +// returns true when we add a new relay +func (ar *AutoRelay) tryRelay(ctx context.Context, pi pstore.PeerInfo) bool { + if ar.usingRelay(pi.ID) { + return false + } - log.Debugf("connected to relay %s", pi.ID) - ar.mx.Lock() - ar.relays[pi.ID] = struct{}{} - haveRelays++ - ar.mx.Unlock() + if !ar.connect(ctx, pi) { + return false + } - // tag the connection as very important - ar.host.ConnManager().TagPeer(pi.ID, "relay", 42) + ar.mx.Lock() + defer ar.mx.Unlock() - update++ - need-- - if need == 0 { - break - } + // make sure we're still connected. + if ar.host.Network().Connectedness(pi.ID) != inet.Connected { + return false } + ar.relays[pi.ID] = struct{}{} - if haveRelays == 0 { - // we failed to find any relays and we are not connected to any! - // wait a little and try again, the discovery query might have returned only dead peers - retry++ - if retry > 5 { - log.Debug("no relays connected; giving up") - return false - } + return true +} - log.Debug("no relays connected; retrying in 30s") - select { - case <-time.After(30 * time.Second): - goto again - case <-ctx.Done(): +func (ar *AutoRelay) connect(ctx context.Context, pi pstore.PeerInfo) bool { + ctx, cancel := context.WithTimeout(ctx, 60*time.Second) + defer cancel() + + if len(pi.Addrs) == 0 { + var err error + pi, err = ar.router.FindPeer(ctx, pi.ID) + if err != nil { + log.Debugf("error finding relay peer %s: %s", pi.ID, err.Error()) return false } } - return update > 0 + err := ar.host.Connect(ctx, pi) + if err != nil { + log.Debugf("error connecting to relay %s: %s", pi.ID, err.Error()) + return false + } + + // tag the connection as very important + ar.host.ConnManager().TagPeer(pi.ID, "relay", 42) + return true +} + +func (ar *AutoRelay) discoverRelays(ctx context.Context) ([]pstore.PeerInfo, error) { + ctx, cancel := context.WithTimeout(ctx, 30*time.Second) + defer cancel() + return discovery.FindPeers(ctx, ar.discover, RelayRendezvous, 1000) } func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo) []pstore.PeerInfo { From abfb4c890161ca3328201de4a77e8a3e93427b16 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 21 Apr 2019 11:18:24 +0300 Subject: [PATCH 1145/3965] fix bug in findRelaysOnce: it connects to all relays --- p2p/host/relay/autorelay.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 669f65fdc6..05a1c101ac 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -159,6 +159,9 @@ func (ar *AutoRelay) findRelaysOnce(ctx context.Context) bool { update := false for _, pi := range pis { update = ar.tryRelay(ctx, pi) || update + if ar.numRelays() >= DesiredRelays { + break + } } return update } From 330481f135c9de71fc711b8f8d889053bd0fd39b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 22 Apr 2019 09:46:13 -0700 Subject: [PATCH 1146/3965] dep: update secio --- go.mod | 2 +- go.sum | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 285ace9ca3..8bd4be5b70 100644 --- a/go.mod +++ b/go.mod @@ -26,7 +26,7 @@ require ( github.com/libp2p/go-libp2p-peerstore v0.0.2 github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 - github.com/libp2p/go-libp2p-secio v0.0.1 + github.com/libp2p/go-libp2p-secio v0.0.2 github.com/libp2p/go-libp2p-swarm v0.0.2 github.com/libp2p/go-libp2p-transport v0.0.4 github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 diff --git a/go.sum b/go.sum index f41b5191bd..0526379e90 100644 --- a/go.sum +++ b/go.sum @@ -129,6 +129,8 @@ github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbf github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= github.com/libp2p/go-libp2p-secio v0.0.1 h1:CqE/RdsizOwItdgLe632iyft/w0tshDLmZGAiKDcUAI= github.com/libp2p/go-libp2p-secio v0.0.1/go.mod h1:IdG6iQybdcYmbTzxp4J5dwtUEDTOvZrT0opIDVNPrJs= +github.com/libp2p/go-libp2p-secio v0.0.2 h1:P8Sv1aAQ9bdLb58T4T7kUC1kXsPovdUFhUSmVy4z3oM= +github.com/libp2p/go-libp2p-secio v0.0.2/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= github.com/libp2p/go-libp2p-swarm v0.0.1 h1:Vne+hjaDwXqzgNwQ2vb2YKbnbOTyXjtS47stT66Apc4= github.com/libp2p/go-libp2p-swarm v0.0.1/go.mod h1:mh+KZxkbd3lQnveQ3j2q60BM1Cw2mX36XXQqwfPOShs= github.com/libp2p/go-libp2p-swarm v0.0.2 h1:cpHHXTeU2IgUu8LPemF7vaLPGtVC6VxMoll2EwqlC+E= @@ -144,6 +146,8 @@ github.com/libp2p/go-mplex v0.0.1 h1:dn2XGSrUxLtz3/8u85bGrwhUEKPX8MOF3lpmcWBZCWc github.com/libp2p/go-mplex v0.0.1/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= github.com/libp2p/go-msgio v0.0.1 h1:znj97n5FtXGCLDwe9x8jpHmY770SW4WStBGcCDh6GJw= github.com/libp2p/go-msgio v0.0.1/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= +github.com/libp2p/go-msgio v0.0.2 h1:ivPvEKHxmVkTClHzg6RXTYHqaJQ0V9cDbq+6lKb3UV0= +github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-nat v0.0.3 h1:l6fKV+p0Xa354EqQOQP+d8CivdLM4kl5GxC1hSc/UeI= github.com/libp2p/go-nat v0.0.3/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= From 97f1e71e8688f9ddae4c628a44026435e01068e3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 22 Apr 2019 19:48:01 +0300 Subject: [PATCH 1147/3965] cache relayAddrs for a short period of time --- p2p/host/relay/autorelay.go | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 05a1c101ac..d442b0f6d0 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -43,6 +43,9 @@ type AutoRelay struct { mx sync.Mutex relays map[peer.ID]struct{} status autonat.NATStatus + + cachedAddrs []ma.Multiaddr + cachedAddrsExpiry time.Time } func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer, router routing.PeerRouting) *AutoRelay { @@ -108,6 +111,9 @@ func (ar *AutoRelay) background(ctx context.Context) { } if push { + ar.mx.Lock() + ar.cachedAddrs = nil + ar.mx.Unlock() push = false ar.host.PushIdentify() } @@ -250,16 +256,20 @@ func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo) [] // through which we can be dialed. func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { ar.mx.Lock() + defer ar.mx.Unlock() + if ar.status != autonat.NATStatusPrivate { - ar.mx.Unlock() return addrs } + if ar.cachedAddrs != nil && time.Now().Before(ar.cachedAddrsExpiry) { + return ar.cachedAddrs + } + relays := make([]peer.ID, 0, len(ar.relays)) for p := range ar.relays { relays = append(relays, p) } - ar.mx.Unlock() raddrs := make([]ma.Multiaddr, 0, 4*len(relays)+2) @@ -285,6 +295,9 @@ func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { } } + ar.cachedAddrs = raddrs + ar.cachedAddrsExpiry = time.Now().Add(30 * time.Second) + return raddrs } From ba00011d347a661969e04714d820fff604180258 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 22 Apr 2019 20:26:02 +0300 Subject: [PATCH 1148/3965] avoid intermediate allocation in relayAddrs now that we have the lock for the duration of the method, we don't need the intermediate relays array. This removes it, and also extends the pre-allocation of the result array by 2 so that it can cover two localhost and two private address bindings. --- p2p/host/relay/autorelay.go | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index d442b0f6d0..11fc436f8c 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -266,12 +266,7 @@ func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { return ar.cachedAddrs } - relays := make([]peer.ID, 0, len(ar.relays)) - for p := range ar.relays { - relays = append(relays, p) - } - - raddrs := make([]ma.Multiaddr, 0, 4*len(relays)+2) + raddrs := make([]ma.Multiaddr, 0, 4*len(ar.relays)+4) // only keep private addrs from the original addr set for _, addr := range addrs { @@ -281,7 +276,7 @@ func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { } // add relay specific addrs to the list - for _, p := range relays { + for p := range ar.relays { addrs := cleanupAddressSet(ar.host.Peerstore().Addrs(p)) circuit, err := ma.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", p.Pretty())) From 60c8703403bd9759f6ccdeb2a1b6e82857384696 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 23 Apr 2019 00:10:05 -0700 Subject: [PATCH 1149/3965] optimize memory of in-memory peerstore --- p2p/host/peerstore/pstoremem/metadata.go | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index ba57dd7fbf..5562c9cfd9 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -7,10 +7,15 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" ) +type metakey struct { + id peer.ID + key string +} + type memoryPeerMetadata struct { // store other data, like versions //ds ds.ThreadSafeDatastore - ds map[string]interface{} + ds map[metakey]interface{} dslock sync.Mutex } @@ -18,26 +23,21 @@ var _ pstore.PeerMetadata = (*memoryPeerMetadata)(nil) func NewPeerMetadata() pstore.PeerMetadata { return &memoryPeerMetadata{ - ds: make(map[string]interface{}), + ds: make(map[metakey]interface{}), } } func (ps *memoryPeerMetadata) Put(p peer.ID, key string, val interface{}) error { - //dsk := ds.NewKey(string(p) + "/" + key) - //return ps.ds.Put(dsk, val) ps.dslock.Lock() defer ps.dslock.Unlock() - ps.ds[string(p)+"/"+key] = val + ps.ds[metakey{p, key}] = val return nil } func (ps *memoryPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { - //dsk := ds.NewKey(string(p) + "/" + key) - //return ps.ds.Get(dsk) - ps.dslock.Lock() defer ps.dslock.Unlock() - i, ok := ps.ds[string(p)+"/"+key] + i, ok := ps.ds[metakey{p, key}] if !ok { return nil, pstore.ErrNotFound } From 7b7825756164b3441fb2db6c297595b5d1183df8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 23 Apr 2019 00:54:18 -0700 Subject: [PATCH 1150/3965] intern protocol names Ideally, we'd store a bitmap. But we'll have to think through how we want that to play out with the datastore-backed peerstore (maybe move the `*Protocols` methods down to the backend implementation?). --- p2p/host/peerstore/peerstore.go | 44 ++++++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 40411fb2ce..29e0c6740c 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -10,6 +10,9 @@ import ( var _ Peerstore = (*peerstore)(nil) +const maxInternedProtocols = 256 +const maxInternedProtocolSize = 96 + type peerstore struct { Metrics @@ -18,17 +21,19 @@ type peerstore struct { PeerMetadata // lock for protocol information, separate from datastore lock - protolock sync.Mutex + protolock sync.Mutex + internedProtocols map[string]string } // NewPeerstore creates a data structure that stores peer data, backed by the // supplied implementations of KeyBook, AddrBook and PeerMetadata. func NewPeerstore(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { return &peerstore{ - KeyBook: kb, - AddrBook: ab, - PeerMetadata: md, - Metrics: NewMetrics(), + KeyBook: kb, + AddrBook: ab, + PeerMetadata: md, + Metrics: NewMetrics(), + internedProtocols: make(map[string]string), } } @@ -75,13 +80,30 @@ func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { } } +func (ps *peerstore) internProtocol(s string) string { + if len(s) > maxInternedProtocolSize { + return s + } + + if interned, ok := ps.internedProtocols[s]; ok { + return interned + } + + if len(ps.internedProtocols) >= maxInternedProtocols { + ps.internedProtocols = make(map[string]string, maxInternedProtocols) + } + + ps.internedProtocols[s] = s + return s +} + func (ps *peerstore) SetProtocols(p peer.ID, protos ...string) error { ps.protolock.Lock() defer ps.protolock.Unlock() - protomap := make(map[string]struct{}) + protomap := make(map[string]struct{}, len(protos)) for _, proto := range protos { - protomap[proto] = struct{}{} + protomap[ps.internProtocol(proto)] = struct{}{} } return ps.Put(p, "protocols", protomap) @@ -96,7 +118,7 @@ func (ps *peerstore) AddProtocols(p peer.ID, protos ...string) error { } for _, proto := range protos { - protomap[proto] = struct{}{} + protomap[ps.internProtocol(proto)] = struct{}{} } return ps.Put(p, "protocols", protomap) @@ -127,8 +149,8 @@ func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { return nil, err } - var out []string - for k, _ := range pmap { + out := make([]string, 0, len(pmap)) + for k := range pmap { out = append(out, k) } @@ -143,7 +165,7 @@ func (ps *peerstore) SupportsProtocols(p peer.ID, protos ...string) ([]string, e return nil, err } - var out []string + out := make([]string, 0, len(protos)) for _, proto := range protos { if _, ok := pmap[proto]; ok { out = append(out, proto) From 3a98ccd7c976893388c363325f3c32aa178e2f66 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 23 Apr 2019 00:59:28 -0700 Subject: [PATCH 1151/3965] fix unmarshalling of peer IDs --- p2p/host/peerstore/pb/custom.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pb/custom.go b/p2p/host/peerstore/pb/custom.go index 711963c941..2ffcc32bb6 100644 --- a/p2p/host/peerstore/pb/custom.go +++ b/p2p/host/peerstore/pb/custom.go @@ -46,7 +46,7 @@ func (id *ProtoPeerID) Unmarshal(data []byte) (err error) { func (id *ProtoPeerID) UnmarshalJSON(data []byte) error { var v []byte - err := json.Unmarshal(data, v) + err := json.Unmarshal(data, &v) if err != nil { return err } From 30857c1c574df333f6620760c1c383aaed6ff7c6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 23 Apr 2019 09:36:39 -0700 Subject: [PATCH 1152/3965] peerstore: bump number/size of interned protocol names 512 * 256 (max path length on many systems) = 128KiB --- p2p/host/peerstore/peerstore.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 29e0c6740c..306127e392 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -10,8 +10,8 @@ import ( var _ Peerstore = (*peerstore)(nil) -const maxInternedProtocols = 256 -const maxInternedProtocolSize = 96 +const maxInternedProtocols = 512 +const maxInternedProtocolSize = 256 type peerstore struct { Metrics From 3d252fa3062e8e7003b3784eb59bf96eab9aa1b8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Apr 2019 09:02:51 -0700 Subject: [PATCH 1153/3965] dial: return a nice custom dial error * Limits the number of dial errors we keep * Exposes the dial errors * Avoids string formatting unless we actually need to. Helps address https://github.com/libp2p/go-libp2p-swarm/issues/119 --- p2p/net/swarm/dial_error.go | 69 ++++++++++++++++++++++++++++++++++ p2p/net/swarm/dial_test.go | 28 +++++--------- p2p/net/swarm/swarm_dial.go | 74 +++++++++++++++++++++---------------- 3 files changed, 121 insertions(+), 50 deletions(-) create mode 100644 p2p/net/swarm/dial_error.go diff --git a/p2p/net/swarm/dial_error.go b/p2p/net/swarm/dial_error.go new file mode 100644 index 0000000000..6d5bb14929 --- /dev/null +++ b/p2p/net/swarm/dial_error.go @@ -0,0 +1,69 @@ +package swarm + +import ( + "fmt" + "strings" + + peer "github.com/libp2p/go-libp2p-peer" + ma "github.com/multiformats/go-multiaddr" +) + +// maxDialDialErrors is the maximum number of dial errors we record +const maxDialDialErrors = 16 + +// DialError is the error type returned when dialing. +type DialError struct { + Peer peer.ID + DialErrors []TransportError + Cause error + Skipped int +} + +func (e *DialError) recordErr(addr ma.Multiaddr, err error) { + if len(e.DialErrors) >= maxDialDialErrors { + e.Skipped++ + return + } + e.DialErrors = append(e.DialErrors, TransportError{ + Address: addr, + Cause: err, + }) +} + +func (e *DialError) Error() string { + var builder strings.Builder + fmt.Fprintf(&builder, "failed to dial %s:", e.Peer) + if e.Cause != nil { + fmt.Fprintf(&builder, " %s", e.Cause) + } + for _, te := range e.DialErrors { + fmt.Fprintf(&builder, "\n * [%s] %s", te.Address, te.Cause) + } + if e.Skipped > 0 { + fmt.Fprintf(&builder, "\n ... skipping %d errors ...", e.Skipped) + } + return builder.String() +} + +// Unwrap implements https://godoc.org/golang.org/x/xerrors#Wrapper. +func (e *DialError) Unwrap() error { + // If we have a context error, that's the "ultimate" error. + if e.Cause != nil { + return e.Cause + } + return nil +} + +var _ error = (*DialError)(nil) + +// TransportError is the error returned when dialing a specific address. +type TransportError struct { + Address ma.Multiaddr + Cause error +} + +func (e *TransportError) Error() string { + return fmt.Sprintf("failed to dial %s: %s", e.Address, e.Cause) +} + +var _ error = (*TransportError)(nil) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index adb3438ac4..99585d2ce7 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -2,9 +2,7 @@ package swarm_test import ( "context" - "fmt" "net" - "regexp" "sync" "testing" "time" @@ -491,8 +489,8 @@ func TestDialPeerFailed(t *testing.T) { defer closeSwarms(swarms) testedSwarm, targetSwarm := swarms[0], swarms[1] - exceptedErrorsCount := 5 - for i := 0; i < exceptedErrorsCount; i++ { + expectedErrorsCount := 5 + for i := 0; i < expectedErrorsCount; i++ { _, silentPeerAddress, silentPeerListener := newSilentPeer(t) go acceptAndHang(silentPeerListener) defer silentPeerListener.Close() @@ -508,23 +506,17 @@ func TestDialPeerFailed(t *testing.T) { t.Fatal(err) } - // dial_test.go:508: correctly get a combined error: dial attempt failed: 10 errors occurred: - // * --> (/ip4/127.0.0.1/tcp/46485) dial attempt failed: failed to negotiate security protocol: context deadline exceeded - // * --> (/ip4/127.0.0.1/tcp/34881) dial attempt failed: failed to negotiate security protocol: context deadline exceeded + // dial_test.go:508: correctly get a combined error: failed to dial PEER: all dials failed + // * [/ip4/127.0.0.1/tcp/46485] failed to negotiate security protocol: context deadline exceeded + // * [/ip4/127.0.0.1/tcp/34881] failed to negotiate security protocol: context deadline exceeded // ... - errorCountRegexpString := fmt.Sprintf("%d errors occurred", exceptedErrorsCount) - errorCountRegexp := regexp.MustCompile(errorCountRegexpString) - if !errorCountRegexp.MatchString(err.Error()) { - t.Fatalf("can't find total err count: `%s' in `%s'", errorCountRegexpString, err.Error()) + dialErr, ok := err.(*DialError) + if !ok { + t.Fatalf("expected *DialError, got %T", err) } - connectErrorsRegexpString := `\* --> \(.+?\) dial attempt failed:.+` - connectErrorsRegexp := regexp.MustCompile(connectErrorsRegexpString) - connectErrors := connectErrorsRegexp.FindAll([]byte(err.Error()), -1) - if len(connectErrors) != exceptedErrorsCount { - t.Fatalf("connectErrors must contain %d errros; "+ - "but `%s' was found in `%s' %d times", - exceptedErrorsCount, connectErrorsRegexpString, err.Error(), len(connectErrors)) + if len(dialErr.DialErrors) != expectedErrorsCount { + t.Errorf("expected %d errors, got %d", expectedErrorsCount, len(dialErr.DialErrors)) } } diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index d4b315525f..3feeffe3fe 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -7,8 +7,6 @@ import ( "sync" "time" - "github.com/hashicorp/go-multierror" - logging "github.com/ipfs/go-log" addrutil "github.com/libp2p/go-addr-util" lgbl "github.com/libp2p/go-libp2p-loggables" @@ -34,15 +32,23 @@ var ( // been dialed too frequently ErrDialBackoff = errors.New("dial backoff") - // ErrDialFailed is returned when connecting to a peer has ultimately failed - ErrDialFailed = errors.New("dial attempt failed") - // ErrDialToSelf is returned if we attempt to dial our own peer ErrDialToSelf = errors.New("dial to self attempted") // ErrNoTransport is returned when we don't know a transport for the // given multiaddr. ErrNoTransport = errors.New("no transport for protocol") + + // ErrAllDialsFailed is returned when connecting to a peer has ultimately failed + ErrAllDialsFailed = errors.New("all dials failed") + + // ErrNoAddresses is returned when we fail to find any addresses for a + // peer we're trying to dial. + ErrNoAddresses = errors.New("no addresses") + + // ErrNoAddresses is returned when we find addresses for a peer but + // can't use any of them. + ErrNoGoodAddresses = errors.New("no good addresses") ) // DialAttempts governs how many times a goroutine will try to dial a given peer. @@ -256,7 +262,7 @@ func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { } // ok, we failed. - return nil, fmt.Errorf("dial attempt failed: %s", err) + return nil, err } return conn, nil } @@ -293,11 +299,11 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { */ peerAddrs := s.peers.Addrs(p) if len(peerAddrs) == 0 { - return nil, errors.New("no addresses") + return nil, &DialError{Peer: p, Cause: ErrNoAddresses} } goodAddrs := s.filterKnownUndialables(peerAddrs) if len(goodAddrs) == 0 { - return nil, errors.New("no good addresses") + return nil, &DialError{Peer: p, Cause: ErrNoGoodAddresses} } goodAddrsChan := make(chan ma.Multiaddr, len(goodAddrs)) for _, a := range goodAddrs { @@ -307,10 +313,18 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { ///////// // try to get a connection to any addr - connC, err := s.dialAddrs(ctx, p, goodAddrsChan) - if err != nil { - logdial["error"] = err.Error() - return nil, err + connC, dialErr := s.dialAddrs(ctx, p, goodAddrsChan) + if dialErr != nil { + logdial["error"] = dialErr.Cause.Error() + if dialErr.Cause == context.Canceled { + // always prefer the "context canceled" error. + // we rely on behing able to check `err == context.Canceled` + // + // Removing this will BREAK backoff (causing us to + // backoff when canceling dials). + return nil, context.Canceled + } + return nil, dialErr } logdial["conn"] = logging.Metadata{ "localAddr": connC.LocalMultiaddr(), @@ -320,7 +334,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { if err != nil { logdial["error"] = err.Error() connC.Close() // close the connection. didn't work out :( - return nil, err + return nil, &DialError{Peer: p, Cause: err} } logdial["dial"] = "success" @@ -352,7 +366,7 @@ func (s *Swarm) filterKnownUndialables(addrs []ma.Multiaddr) []ma.Multiaddr { ) } -func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (transport.Conn, error) { +func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (transport.Conn, *DialError) { log.Debugf("%s swarm dialing %s", s.local, p) ctx, cancel := context.WithCancel(ctx) @@ -360,26 +374,23 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. // use a single response type instead of errs and conns, reduces complexity *a ton* respch := make(chan dialResult) - var dialErrors *multierror.Error + err := new(DialError) defer s.limiter.clearAllPeerDials(p) var active int +dialLoop: for remoteAddrs != nil || active > 0 { // Check for context cancellations and/or responses first. select { case <-ctx.Done(): - if dialError := dialErrors.ErrorOrNil(); dialError != nil { - return nil, dialError - } - - return nil, ctx.Err() + break dialLoop case resp := <-respch: active-- if resp.Err != nil { // Errors are normal, lots of dials will fail log.Infof("got error on dial: %s", resp.Err) - dialErrors = multierror.Append(dialErrors, resp.Err) + err.recordErr(resp.Addr, resp.Err) } else if resp.Conn != nil { return resp.Conn, nil } @@ -400,28 +411,27 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. s.limitedDial(ctx, p, addr, respch) active++ case <-ctx.Done(): - if dialError := dialErrors.ErrorOrNil(); dialError != nil { - return nil, dialError - } - - return nil, ctx.Err() + break dialLoop case resp := <-respch: active-- if resp.Err != nil { // Errors are normal, lots of dials will fail log.Infof("got error on dial: %s", resp.Err) - dialErrors = multierror.Append(dialErrors, resp.Err) + err.recordErr(resp.Addr, resp.Err) } else if resp.Conn != nil { return resp.Conn, nil } } } - if dialError := dialErrors.ErrorOrNil(); dialError != nil { - return nil, dialError + if ctxErr := ctx.Err(); ctxErr != nil { + err.Cause = ctxErr + } else if len(err.DialErrors) == 0 { + err.Cause = inet.ErrNoRemoteAddrs + } else { + err.Cause = ErrAllDialsFailed } - - return nil, inet.ErrNoRemoteAddrs + return nil, err } // limitedDial will start a dial to the given peer when @@ -450,7 +460,7 @@ func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (tra connC, err := tpt.Dial(ctx, addr, p) if err != nil { - return nil, fmt.Errorf("%s --> %s (%s) dial attempt failed: %s", s.local, p, addr, err) + return nil, err } // Trust the transport? Yeah... right. From 5f198db75d6bf432360ae2c4b5eec19ab5213777 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 25 Apr 2019 20:11:36 +0300 Subject: [PATCH 1154/3965] gomod: update go-libp2p-swarm, go-libp2p-peerstore --- go.mod | 4 ++-- go.sum | 6 ++++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/go.mod b/go.mod index 8bd4be5b70..3729622340 100644 --- a/go.mod +++ b/go.mod @@ -23,11 +23,11 @@ require ( github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.0.1 - github.com/libp2p/go-libp2p-peerstore v0.0.2 + github.com/libp2p/go-libp2p-peerstore v0.0.3 github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 github.com/libp2p/go-libp2p-secio v0.0.2 - github.com/libp2p/go-libp2p-swarm v0.0.2 + github.com/libp2p/go-libp2p-swarm v0.0.3 github.com/libp2p/go-libp2p-transport v0.0.4 github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 github.com/libp2p/go-maddr-filter v0.0.1 diff --git a/go.sum b/go.sum index 0526379e90..b3912d3509 100644 --- a/go.sum +++ b/go.sum @@ -121,8 +121,8 @@ github.com/libp2p/go-libp2p-peer v0.0.1 h1:0qwAOljzYewINrU+Kndoc+1jAL7vzY/oY2Go4 github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= github.com/libp2p/go-libp2p-peerstore v0.0.1 h1:twKovq8YK5trLrd3nB7PD2Zu9JcyAIdm7Bz9yBWjhq8= github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= -github.com/libp2p/go-libp2p-peerstore v0.0.2 h1:Lirt3A1Oq11jszJ4SPNBo8chNv61UWXE538KUEGxTVk= -github.com/libp2p/go-libp2p-peerstore v0.0.2/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= +github.com/libp2p/go-libp2p-peerstore v0.0.3 h1:1qUG/xeFjFSnwmOLP/iyNtHDA0kaEzlJzM8dm3lPSBo= +github.com/libp2p/go-libp2p-peerstore v0.0.3/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= github.com/libp2p/go-libp2p-protocol v0.0.1 h1:+zkEmZ2yFDi5adpVE3t9dqh/N9TbpFWywowzeEzBbLM= github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbfdvpNntjhc= @@ -135,6 +135,8 @@ github.com/libp2p/go-libp2p-swarm v0.0.1 h1:Vne+hjaDwXqzgNwQ2vb2YKbnbOTyXjtS47st github.com/libp2p/go-libp2p-swarm v0.0.1/go.mod h1:mh+KZxkbd3lQnveQ3j2q60BM1Cw2mX36XXQqwfPOShs= github.com/libp2p/go-libp2p-swarm v0.0.2 h1:cpHHXTeU2IgUu8LPemF7vaLPGtVC6VxMoll2EwqlC+E= github.com/libp2p/go-libp2p-swarm v0.0.2/go.mod h1:n0cAAcKyndIrJWctQwjqXlAdIPBZzfdpBjx1SSvz30g= +github.com/libp2p/go-libp2p-swarm v0.0.3 h1:gF11uO1WCbFtTjKLQ+gZ/UdbYScZq8PfqA53XdbiK8Q= +github.com/libp2p/go-libp2p-swarm v0.0.3/go.mod h1:/2HbOacAKDYT1g0UEZjUPlzD+SBtvqkg4TaYeoBA2TY= github.com/libp2p/go-libp2p-transport v0.0.1/go.mod h1:UzbUs9X+PHOSw7S3ZmeOxfnwaQY5vGDzZmKPod3N3tk= github.com/libp2p/go-libp2p-transport v0.0.4 h1:/CPHQMN75/IQwkhBxxIo6p6PtL3rwFZtlzBROT3e8mw= github.com/libp2p/go-libp2p-transport v0.0.4/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= From 085f98ae8fd9f2b0c38c9cf5f87d584895fa3425 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 25 Apr 2019 19:25:53 -0700 Subject: [PATCH 1155/3965] block while writing Make sure to wait until we're done writing before returning. The close should interrupt this process but we should still wait. might be related to https://github.com/ipfs/go-ipfs/issues/6197 --- p2p/net/conn-security-multistream/ssms.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/net/conn-security-multistream/ssms.go b/p2p/net/conn-security-multistream/ssms.go index 312b775cd3..b74b749938 100644 --- a/p2p/net/conn-security-multistream/ssms.go +++ b/p2p/net/conn-security-multistream/ssms.go @@ -82,6 +82,7 @@ func (sm *SSMuxer) selectProto(ctx context.Context, insecure net.Conn, server bo // We *must* do this. We have outstanding work on the connection // and it's no longer safe to use. insecure.Close() + <-done // wait to stop using the connection. return nil, ctx.Err() } } From e87a4e322b4b1c2e89fedf709d1748aec4cfb7ba Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 25 Apr 2019 19:31:56 -0700 Subject: [PATCH 1156/3965] improve correctness of closing connections on failure may be related to https://github.com/ipfs/go-ipfs/issues/6197 (but I can't find one) --- p2p/net/upgrader/upgrader.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index 1c901b27dc..eacbb3dee8 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -89,7 +89,7 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma } smconn, err := u.setupMuxer(ctx, sconn, p) if err != nil { - conn.Close() + sconn.Close() return nil, fmt.Errorf("failed to negotiate security stream multiplexer: %s", err) } return &transportConn{ @@ -122,6 +122,10 @@ func (u *Upgrader) setupMuxer(ctx context.Context, conn net.Conn, p peer.ID) (sm case <-done: return smconn, err case <-ctx.Done(): + // interrupt this process + conn.Close() + // wait to finish + <-done return nil, ctx.Err() } } From f418d2c15b38464a805da3514faab9ccab5c3c7b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 26 Apr 2019 01:32:43 -0700 Subject: [PATCH 1157/3965] test: cleanup after tests Otherwise, setting global variables races with running goroutines. --- p2p/net/upgrader/listener_test.go | 74 +++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 18 deletions(-) diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go index b216c462e3..969b98d238 100644 --- a/p2p/net/upgrader/listener_test.go +++ b/p2p/net/upgrader/listener_test.go @@ -104,6 +104,7 @@ var _ = Describe("Listener", func() { It("accepts a single connection", func() { ln := createListener(defaultUpgrader) + defer ln.Close() cconn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) Expect(err).ToNot(HaveOccurred()) sconn, err := ln.Accept() @@ -113,6 +114,7 @@ var _ = Describe("Listener", func() { It("accepts multiple connections", func() { ln := createListener(defaultUpgrader) + defer ln.Close() const num = 10 for i := 0; i < 10; i++ { cconn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) @@ -127,11 +129,15 @@ var _ = Describe("Listener", func() { const timeout = 200 * time.Millisecond tpt.AcceptTimeout = timeout ln := createListener(defaultUpgrader) + defer ln.Close() conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) - Expect(err).ToNot(HaveOccurred()) + if !Expect(err).ToNot(HaveOccurred()) { + return + } done := make(chan struct{}) go func() { defer GinkgoRecover() + defer conn.Close() str, err := conn.OpenStream() Expect(err).ToNot(HaveOccurred()) // start a Read. It will block until the connection is closed @@ -151,10 +157,16 @@ var _ = Describe("Listener", func() { done := make(chan struct{}) go func() { defer GinkgoRecover() - _, _ = ln.Accept() + conn, err := ln.Accept() + if !Expect(err).To(HaveOccurred()) { + conn.Close() + } close(done) }() - _, _ = dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + if !Expect(err).To(HaveOccurred()) { + conn.Close() + } Consistently(done).ShouldNot(BeClosed()) // make the goroutine return ln.Close() @@ -178,6 +190,7 @@ var _ = Describe("Listener", func() { if err != nil { return } + conn.Close() accepted <- conn } }() @@ -187,8 +200,14 @@ var _ = Describe("Listener", func() { wg.Add(1) go func() { defer GinkgoRecover() - _, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) - Expect(err).ToNot(HaveOccurred()) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + if Expect(err).ToNot(HaveOccurred()) { + stream, err := conn.AcceptStream() // wait for conn to be accepted. + if !Expect(err).To(HaveOccurred()) { + stream.Close() + } + conn.Close() + } wg.Done() }() } @@ -201,29 +220,40 @@ var _ = Describe("Listener", func() { It("stops setting up when the more than AcceptQueueLength connections are waiting to get accepted", func() { ln := createListener(defaultUpgrader) + defer ln.Close() + // setup AcceptQueueLength connections, but don't accept any of them - dialed := make(chan struct{}, 10*st.AcceptQueueLength) // used as a thread-safe counter + dialed := make(chan tpt.Conn, 10*st.AcceptQueueLength) // used as a thread-safe counter for i := 0; i < st.AcceptQueueLength; i++ { go func() { defer GinkgoRecover() - _, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(err).ToNot(HaveOccurred()) - dialed <- struct{}{} + dialed <- conn }() } Eventually(dialed).Should(HaveLen(st.AcceptQueueLength)) // dial a new connection. This connection should not complete setup, since the queue is full go func() { defer GinkgoRecover() - _, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(err).ToNot(HaveOccurred()) - dialed <- struct{}{} + dialed <- conn }() Consistently(dialed).Should(HaveLen(st.AcceptQueueLength)) // accept a single connection. Now the new connection should be set up, and fill the queue again - _, err := ln.Accept() - Expect(err).ToNot(HaveOccurred()) + conn, err := ln.Accept() + if Expect(err).ToNot(HaveOccurred()) { + conn.Close() + } Eventually(dialed).Should(HaveLen(st.AcceptQueueLength + 1)) + + // Cleanup + for i := 0; i < st.AcceptQueueLength+1; i++ { + if c := <-dialed; c != nil { + c.Close() + } + } }) }) @@ -233,9 +263,12 @@ var _ = Describe("Listener", func() { done := make(chan struct{}) go func() { defer GinkgoRecover() - _, err := ln.Accept() - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("use of closed network connection")) + conn, err := ln.Accept() + if Expect(err).To(HaveOccurred()) { + Expect(err.Error()).To(ContainSubstring("use of closed network connection")) + } else { + conn.Close() + } close(done) }() Consistently(done).ShouldNot(BeClosed()) @@ -246,15 +279,20 @@ var _ = Describe("Listener", func() { It("doesn't accept new connections when it is closed", func() { ln := createListener(defaultUpgrader) Expect(ln.Close()).To(Succeed()) - _, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) - Expect(err).To(HaveOccurred()) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) + if !Expect(err).To(HaveOccurred()) { + conn.Close() + } }) It("closes incoming connections that have not yet been accepted", func() { ln := createListener(defaultUpgrader) conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + if !Expect(err).ToNot(HaveOccurred()) { + ln.Close() + return + } Expect(conn.IsClosed()).To(BeFalse()) - Expect(err).ToNot(HaveOccurred()) Expect(ln.Close()).To(Succeed()) Eventually(conn.IsClosed).Should(BeTrue()) }) From eeac0bb952a9ef39fcf4b852ff912da34991f917 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 26 Apr 2019 01:40:19 -0700 Subject: [PATCH 1158/3965] update upgraders to maybe fix a concurrent connection write issue --- go.mod | 4 ++-- go.sum | 4 ++++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index 3729622340..0ee7583467 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,7 @@ require ( github.com/ipfs/go-log v0.0.1 github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 github.com/libp2p/go-conn-security v0.0.1 - github.com/libp2p/go-conn-security-multistream v0.0.1 + github.com/libp2p/go-conn-security-multistream v0.0.2 github.com/libp2p/go-libp2p-autonat v0.0.4 github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.4 @@ -29,7 +29,7 @@ require ( github.com/libp2p/go-libp2p-secio v0.0.2 github.com/libp2p/go-libp2p-swarm v0.0.3 github.com/libp2p/go-libp2p-transport v0.0.4 - github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 + github.com/libp2p/go-libp2p-transport-upgrader v0.0.2 github.com/libp2p/go-maddr-filter v0.0.1 github.com/libp2p/go-stream-muxer v0.0.1 github.com/libp2p/go-tcp-transport v0.0.2 diff --git a/go.sum b/go.sum index b3912d3509..cd6fcb53fa 100644 --- a/go.sum +++ b/go.sum @@ -87,6 +87,8 @@ github.com/libp2p/go-conn-security v0.0.1 h1:4kMMrqrt9EUNCNjX1xagSJC+bq16uqjMe9l github.com/libp2p/go-conn-security v0.0.1/go.mod h1:bGmu51N0KU9IEjX7kl2PQjgZa40JQWnayTvNMgD/vyk= github.com/libp2p/go-conn-security-multistream v0.0.1 h1:XefjAQRHcnUaxKb26RGupToucx3uU4ecbOZ3aACXlDU= github.com/libp2p/go-conn-security-multistream v0.0.1/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= +github.com/libp2p/go-conn-security-multistream v0.0.2 h1:Ykz0lnNjxk+0SdslUmlLNyrleqdpS1S/VW+dxFdt74Y= +github.com/libp2p/go-conn-security-multistream v0.0.2/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-libp2p-autonat v0.0.4 h1:cZzdB9KW1ZkHnSjLCB6aFNw47XS4r+SecCVMuVB1xgo= @@ -142,6 +144,8 @@ github.com/libp2p/go-libp2p-transport v0.0.4 h1:/CPHQMN75/IQwkhBxxIo6p6PtL3rwFZt github.com/libp2p/go-libp2p-transport v0.0.4/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 h1:rNtXkY6dty46mxYOHHAZQchI7gQdJStF683FhVnei/k= github.com/libp2p/go-libp2p-transport-upgrader v0.0.1/go.mod h1:NJpUAgQab/8K6K0m+JmZCe5RUXG10UMEx4kWe9Ipj5c= +github.com/libp2p/go-libp2p-transport-upgrader v0.0.2 h1:VygEEnAyTPZxVkEOVsmLp0vGLGBEn4W1UBHcy9cEywg= +github.com/libp2p/go-libp2p-transport-upgrader v0.0.2/go.mod h1:NJpUAgQab/8K6K0m+JmZCe5RUXG10UMEx4kWe9Ipj5c= github.com/libp2p/go-maddr-filter v0.0.1 h1:apvYTg0aIxxQyBX+XHKOR+0+lYhGs1Yv+JmTH9nyl5I= github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-mplex v0.0.1 h1:dn2XGSrUxLtz3/8u85bGrwhUEKPX8MOF3lpmcWBZCWc= From e5022827a194590d7f2978d01e4a9c4b997253b3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 27 Apr 2019 22:15:45 +0300 Subject: [PATCH 1159/3965] read/write protocol lock --- p2p/host/peerstore/peerstore.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 306127e392..f7f87b8593 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -21,7 +21,7 @@ type peerstore struct { PeerMetadata // lock for protocol information, separate from datastore lock - protolock sync.Mutex + protolock sync.RWMutex internedProtocols map[string]string } @@ -142,8 +142,8 @@ func (ps *peerstore) getProtocolMap(p peer.ID) (map[string]struct{}, error) { } func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { - ps.protolock.Lock() - defer ps.protolock.Unlock() + ps.protolock.RLock() + defer ps.protolock.RUnlock() pmap, err := ps.getProtocolMap(p) if err != nil { return nil, err @@ -158,8 +158,8 @@ func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { } func (ps *peerstore) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { - ps.protolock.Lock() - defer ps.protolock.Unlock() + ps.protolock.RLock() + defer ps.protolock.RUnlock() pmap, err := ps.getProtocolMap(p) if err != nil { return nil, err From bacd7c9b15955a1b09b72f1ec986b147ecd82729 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 27 Apr 2019 22:19:39 +0300 Subject: [PATCH 1160/3965] read/write address book lock --- p2p/host/peerstore/pstoremem/addr_book.go | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 723b4337cb..002f8b1ff9 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -30,7 +30,7 @@ var _ pstore.AddrBook = (*memoryAddrBook)(nil) // memoryAddrBook manages addresses. type memoryAddrBook struct { - addrmu sync.Mutex + addrmu sync.RWMutex // Use pointers to save memory. Maps always leave some fraction of their // space unused. storing the *values* directly in the map will // drastically increase the space waste. In our case, by 6x. @@ -68,8 +68,8 @@ func (mab *memoryAddrBook) gc() { } func (mab *memoryAddrBook) PeersWithAddrs() peer.IDSlice { - mab.addrmu.Lock() - defer mab.addrmu.Unlock() + mab.addrmu.RLock() + defer mab.addrmu.RUnlock() pids := make(peer.IDSlice, 0, len(mab.addrs)) for pid := range mab.addrs { @@ -178,8 +178,8 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t // Addresses returns all known (and valid) addresses for a given func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { - mab.addrmu.Lock() - defer mab.addrmu.Unlock() + mab.addrmu.RLock() + defer mab.addrmu.RUnlock() amap, found := mab.addrs[p] if !found { @@ -210,8 +210,8 @@ func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { // AddrStream returns a channel on which all new addresses discovered for a // given peer ID will be published. func (mab *memoryAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { - mab.addrmu.Lock() - defer mab.addrmu.Unlock() + mab.addrmu.RLock() + defer mab.addrmu.RUnlock() baseaddrslice := mab.addrs[p] initial := make([]ma.Multiaddr, 0, len(baseaddrslice)) From db8f683abf9ddc8a209223839a6771efd50f7547 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 27 Apr 2019 22:19:49 +0300 Subject: [PATCH 1161/3965] read/write metadata lock --- p2p/host/peerstore/pstoremem/metadata.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index 5562c9cfd9..162bee5087 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -16,7 +16,7 @@ type memoryPeerMetadata struct { // store other data, like versions //ds ds.ThreadSafeDatastore ds map[metakey]interface{} - dslock sync.Mutex + dslock sync.RWMutex } var _ pstore.PeerMetadata = (*memoryPeerMetadata)(nil) @@ -35,8 +35,8 @@ func (ps *memoryPeerMetadata) Put(p peer.ID, key string, val interface{}) error } func (ps *memoryPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { - ps.dslock.Lock() - defer ps.dslock.Unlock() + ps.dslock.RLock() + defer ps.dslock.RUnlock() i, ok := ps.ds[metakey{p, key}] if !ok { return nil, pstore.ErrNotFound From 8e2547048fc24c288774e9f3e8ac6589a13c36cb Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 28 Apr 2019 10:38:49 +0300 Subject: [PATCH 1162/3965] gomod: update go-libp2p-peerstore --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 0ee7583467..57a319c96b 100644 --- a/go.mod +++ b/go.mod @@ -23,7 +23,7 @@ require ( github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.0.1 - github.com/libp2p/go-libp2p-peerstore v0.0.3 + github.com/libp2p/go-libp2p-peerstore v0.0.4 github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 github.com/libp2p/go-libp2p-secio v0.0.2 diff --git a/go.sum b/go.sum index cd6fcb53fa..9f0165a692 100644 --- a/go.sum +++ b/go.sum @@ -123,8 +123,8 @@ github.com/libp2p/go-libp2p-peer v0.0.1 h1:0qwAOljzYewINrU+Kndoc+1jAL7vzY/oY2Go4 github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= github.com/libp2p/go-libp2p-peerstore v0.0.1 h1:twKovq8YK5trLrd3nB7PD2Zu9JcyAIdm7Bz9yBWjhq8= github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= -github.com/libp2p/go-libp2p-peerstore v0.0.3 h1:1qUG/xeFjFSnwmOLP/iyNtHDA0kaEzlJzM8dm3lPSBo= -github.com/libp2p/go-libp2p-peerstore v0.0.3/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= +github.com/libp2p/go-libp2p-peerstore v0.0.4 h1:ME5mB9WT3DB+4hTgSDpNx7iSlePtIdZOtY9wSvUJnis= +github.com/libp2p/go-libp2p-peerstore v0.0.4/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= github.com/libp2p/go-libp2p-protocol v0.0.1 h1:+zkEmZ2yFDi5adpVE3t9dqh/N9TbpFWywowzeEzBbLM= github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbfdvpNntjhc= From fd9ff6b20d5f11d3f95bf82d876036d4ec84167f Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 28 Apr 2019 11:35:23 +0300 Subject: [PATCH 1163/3965] don't delete under the read lock let gc clean up the addrs --- p2p/host/peerstore/pstoremem/addr_book.go | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 002f8b1ff9..5061f0ff67 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -188,11 +188,9 @@ func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { now := time.Now() good := make([]ma.Multiaddr, 0, len(amap)) - for k, m := range amap { + for _, m := range amap { if !m.ExpiredBy(now) { good = append(good, m.Addr) - } else { - delete(amap, k) } } From 21e65137a0b310f675cfb632ddc790bb1bf7ebfa Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 28 Apr 2019 12:17:48 +0300 Subject: [PATCH 1164/3965] periodically schedule gc --- p2p/host/peerstore/pstoremem/addr_book.go | 29 ++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 5061f0ff67..f2e89ec755 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -37,14 +37,41 @@ type memoryAddrBook struct { addrs map[peer.ID]map[string]*expiringAddr nextGC time.Time + ctx context.Context + cancel func() subManager *AddrSubManager } func NewAddrBook() pstore.AddrBook { - return &memoryAddrBook{ + ctx, cancel := context.WithCancel(context.Background()) + + ab := &memoryAddrBook{ addrs: make(map[peer.ID]map[string]*expiringAddr), subManager: NewAddrSubManager(), + ctx: ctx, + cancel: cancel, + } + + go ab.background() + return ab +} + +// background periodically schedules a gc +func (mab *memoryAddrBook) background() { + ticker := time.NewTicker(1 * time.Hour) + defer ticker.Stop() + + for { + select { + case <-ticker.C: + mab.addrmu.Lock() + mab.gc() + mab.addrmu.Unlock() + + case <-mab.ctx.Done(): + return + } } } From 4bdabc40c0e7311ba8635e99f7fc42c9b0a370ad Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 28 Apr 2019 12:19:26 +0300 Subject: [PATCH 1165/3965] add Close method to memory address book, which cancels the gc process --- p2p/host/peerstore/pstoremem/addr_book.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index f2e89ec755..66290bce9e 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -75,6 +75,11 @@ func (mab *memoryAddrBook) background() { } } +func (mab *memoryAddrBook) Close() error { + mab.cancel() + return nil +} + // gc garbage collects the in-memory address book. The caller *must* hold the addrmu lock. func (mab *memoryAddrBook) gc() { now := time.Now() From 3dd2b42e20b01a70f7b72382291332d1af34f26f Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 28 Apr 2019 16:21:07 +0300 Subject: [PATCH 1166/3965] don't eager gc --- p2p/host/peerstore/pstoremem/addr_book.go | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 66290bce9e..cb9576ace4 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -36,7 +36,6 @@ type memoryAddrBook struct { // drastically increase the space waste. In our case, by 6x. addrs map[peer.ID]map[string]*expiringAddr - nextGC time.Time ctx context.Context cancel func() @@ -65,9 +64,7 @@ func (mab *memoryAddrBook) background() { for { select { case <-ticker.C: - mab.addrmu.Lock() mab.gc() - mab.addrmu.Unlock() case <-mab.ctx.Done(): return @@ -80,12 +77,12 @@ func (mab *memoryAddrBook) Close() error { return nil } -// gc garbage collects the in-memory address book. The caller *must* hold the addrmu lock. +// gc garbage collects the in-memory address book. func (mab *memoryAddrBook) gc() { + mab.addrmu.Lock() + defer mab.addrmu.Unlock() + now := time.Now() - if !now.After(mab.nextGC) { - return - } for p, amap := range mab.addrs { for k, addr := range amap { if addr.ExpiredBy(now) { @@ -96,7 +93,6 @@ func (mab *memoryAddrBook) gc() { delete(mab.addrs, p) } } - mab.nextGC = time.Now().Add(pstore.AddressTTL) } func (mab *memoryAddrBook) PeersWithAddrs() peer.IDSlice { @@ -146,7 +142,6 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du mab.subManager.BroadcastAddr(p, addr) } } - mab.gc() } // SetAddr calls mgr.SetAddrs(p, addr, ttl) @@ -183,7 +178,6 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du delete(amap, addrstr) } } - mab.gc() } // UpdateAddrs updates the addresses associated with the given peer that have @@ -205,7 +199,6 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t amap[k] = addr } } - mab.gc() } // Addresses returns all known (and valid) addresses for a given From 621aa0da19ceba2186e885ee265aae334d19f4c6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 28 Apr 2019 12:44:50 -0700 Subject: [PATCH 1167/3965] dep: update peerstore and secio Pull in a panic fix and a fix for https://github.com/libp2p/go-mplex/issues/43 --- go.mod | 4 ++-- go.sum | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index 57a319c96b..0724e21e47 100644 --- a/go.mod +++ b/go.mod @@ -23,10 +23,10 @@ require ( github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.0.1 - github.com/libp2p/go-libp2p-peerstore v0.0.4 + github.com/libp2p/go-libp2p-peerstore v0.0.5 github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 - github.com/libp2p/go-libp2p-secio v0.0.2 + github.com/libp2p/go-libp2p-secio v0.0.3 github.com/libp2p/go-libp2p-swarm v0.0.3 github.com/libp2p/go-libp2p-transport v0.0.4 github.com/libp2p/go-libp2p-transport-upgrader v0.0.2 diff --git a/go.sum b/go.sum index 9f0165a692..54f6e4a917 100644 --- a/go.sum +++ b/go.sum @@ -123,16 +123,16 @@ github.com/libp2p/go-libp2p-peer v0.0.1 h1:0qwAOljzYewINrU+Kndoc+1jAL7vzY/oY2Go4 github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= github.com/libp2p/go-libp2p-peerstore v0.0.1 h1:twKovq8YK5trLrd3nB7PD2Zu9JcyAIdm7Bz9yBWjhq8= github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= -github.com/libp2p/go-libp2p-peerstore v0.0.4 h1:ME5mB9WT3DB+4hTgSDpNx7iSlePtIdZOtY9wSvUJnis= -github.com/libp2p/go-libp2p-peerstore v0.0.4/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= +github.com/libp2p/go-libp2p-peerstore v0.0.5 h1:/qc3asc93924TaqVulOnY/+c4tpzomSE/ijT7LWF09M= +github.com/libp2p/go-libp2p-peerstore v0.0.5/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= github.com/libp2p/go-libp2p-protocol v0.0.1 h1:+zkEmZ2yFDi5adpVE3t9dqh/N9TbpFWywowzeEzBbLM= github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbfdvpNntjhc= github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= github.com/libp2p/go-libp2p-secio v0.0.1 h1:CqE/RdsizOwItdgLe632iyft/w0tshDLmZGAiKDcUAI= github.com/libp2p/go-libp2p-secio v0.0.1/go.mod h1:IdG6iQybdcYmbTzxp4J5dwtUEDTOvZrT0opIDVNPrJs= -github.com/libp2p/go-libp2p-secio v0.0.2 h1:P8Sv1aAQ9bdLb58T4T7kUC1kXsPovdUFhUSmVy4z3oM= -github.com/libp2p/go-libp2p-secio v0.0.2/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= +github.com/libp2p/go-libp2p-secio v0.0.3 h1:h3fPeDrej7bvvARnC2oSjAfcLZOaS4REZKgWCRQNpE4= +github.com/libp2p/go-libp2p-secio v0.0.3/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= github.com/libp2p/go-libp2p-swarm v0.0.1 h1:Vne+hjaDwXqzgNwQ2vb2YKbnbOTyXjtS47stT66Apc4= github.com/libp2p/go-libp2p-swarm v0.0.1/go.mod h1:mh+KZxkbd3lQnveQ3j2q60BM1Cw2mX36XXQqwfPOShs= github.com/libp2p/go-libp2p-swarm v0.0.2 h1:cpHHXTeU2IgUu8LPemF7vaLPGtVC6VxMoll2EwqlC+E= From 273a5f2a9d12c548ab9a15b7661fc72aee4ef84e Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 30 Apr 2019 19:04:50 +0300 Subject: [PATCH 1168/3965] update go-multistream --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 0724e21e47..f8fe6dd81f 100644 --- a/go.mod +++ b/go.mod @@ -39,7 +39,7 @@ require ( github.com/multiformats/go-multiaddr v0.0.1 github.com/multiformats/go-multiaddr-dns v0.0.2 github.com/multiformats/go-multiaddr-net v0.0.1 - github.com/multiformats/go-multistream v0.0.1 + github.com/multiformats/go-multistream v0.0.2 github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible diff --git a/go.sum b/go.sum index 54f6e4a917..1c51b73b1e 100644 --- a/go.sum +++ b/go.sum @@ -199,6 +199,8 @@ github.com/multiformats/go-multihash v0.0.1 h1:HHwN1K12I+XllBCrqKnhX949Orn4oawPk github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= github.com/multiformats/go-multistream v0.0.1 h1:JV4VfSdY9n7ECTtY59/TlSyFCzRILvYx4T4Ws8ZgihU= github.com/multiformats/go-multistream v0.0.1/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= +github.com/multiformats/go-multistream v0.0.2 h1:uWy8f8Icfo9t/U80+CR5oqM84gSgNqh8xLB7rVslfcw= +github.com/multiformats/go-multistream v0.0.2/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0 h1:WSHQ+IS43OoUrWtD1/bbclrwK8TTH5hzp+umCiuxHgs= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= From 0c48498807ae6d7926f4d00314af0c6ce1c3a888 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 1 May 2019 22:16:23 -0700 Subject: [PATCH 1169/3965] fix: remove notifier 1. It was unused. 2. It was broken. Attempting to register a notification listener would have registered the notifier itself. 3. We need to switch to some consistent event/notification system anyways so we're going to replace this code no matter what. (found by @pornin) --- p2p/net/nat/nat.go | 13 ------------ p2p/net/nat/notifier.go | 47 ----------------------------------------- 2 files changed, 60 deletions(-) delete mode 100644 p2p/net/nat/notifier.go diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 178f7d0140..046627289d 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -74,8 +74,6 @@ type NAT struct { mappingmu sync.RWMutex // guards mappings mappings map[*mapping]struct{} - - Notifier } func newNAT(realNAT nat.NAT) *NAT { @@ -186,10 +184,6 @@ func (nat *NAT) establishMapping(m *mapping) { m.setExternalPort(0) // clear mapping // TODO: log.Event log.Warningf("failed to establish port mapping: %s", err) - nat.Notifier.notifyAll(func(n Notifiee) { - n.MappingFailed(nat, m, oldport, err) - }) - // we do not close if the mapping failed, // because it may work again next time. return @@ -199,12 +193,5 @@ func (nat *NAT) establishMapping(m *mapping) { log.Debugf("NAT Mapping: %s --> %s (%s)", m.ExternalPort(), m.InternalPort(), m.Protocol()) if oldport != 0 && newport != oldport { log.Debugf("failed to renew same port mapping: ch %d -> %d", oldport, newport) - nat.Notifier.notifyAll(func(n Notifiee) { - n.MappingChanged(nat, m, oldport, newport) - }) } - - nat.Notifier.notifyAll(func(n Notifiee) { - n.MappingSuccess(nat, m) - }) } diff --git a/p2p/net/nat/notifier.go b/p2p/net/nat/notifier.go deleted file mode 100644 index 10fb6ac6bf..0000000000 --- a/p2p/net/nat/notifier.go +++ /dev/null @@ -1,47 +0,0 @@ -package nat - -import ( - notifier "github.com/whyrusleeping/go-notifier" -) - -// Notifier is an object that assists NAT in notifying listeners. -// It is implemented using thirdparty/notifier -type Notifier struct { - n notifier.Notifier -} - -func (n *Notifier) notifyAll(notify func(n Notifiee)) { - n.n.NotifyAll(func(n notifier.Notifiee) { - notify(n.(Notifiee)) - }) -} - -// Notify signs up notifiee to listen to NAT events. -func (n *Notifier) Notify(notifiee Notifiee) { - n.n.Notify(n) -} - -// StopNotify stops signaling events to notifiee. -func (n *Notifier) StopNotify(notifiee Notifiee) { - n.n.StopNotify(notifiee) -} - -// Notifiee is an interface objects must implement to listen to NAT events. -type Notifiee interface { - - // Called every time a successful mapping happens - // Warning: the port mapping may have changed. If that is the - // case, both MappingSuccess and MappingChanged are called. - MappingSuccess(nat *NAT, m Mapping) - - // Called when mapping a port succeeds, but the mapping is - // with a different port than an earlier success. - MappingChanged(nat *NAT, m Mapping, oldport, newport int) - - // Called when a port mapping fails. NAT will continue attempting after - // the next period. To stop trying, use: mapping.Close(). After this failure, - // mapping.ExternalPort() will be zero, and nat.ExternalAddrs() will not - // return the address for this mapping. With luck, the next attempt will - // succeed, without the client needing to do anything. - MappingFailed(nat *NAT, m Mapping, oldport int, err error) -} From 4d48e11a2fe504869c3a298a00e7874c31b176b3 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 2 May 2019 11:24:00 +0300 Subject: [PATCH 1170/3965] set 1 minute deadline for stop handshake --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index b15841b29e..3c96c9b2da 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -328,6 +328,9 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { wr := newDelimitedWriter(bs) defer rd.Close() + // set handshake deadline + bs.SetDeadline(time.Now().Add(1 * time.Minute)) + msg.Type = pb.CircuitRelay_STOP.Enum() err = wr.WriteMsg(msg) @@ -373,6 +376,9 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { // relay connection log.Infof("relaying connection between %s and %s", src.ID.Pretty(), dst.ID.Pretty()) + // reset deadline + bs.SetDeadline(time.Time{}) + r.addLiveHop(src.ID, dst.ID) // Don't reset streams after finishing or the other side will get an From 1cbb82411431b1b315aedee59fcdb9754fac784c Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 2 May 2019 12:12:38 +0300 Subject: [PATCH 1171/3965] make stop handshake timeout a variable --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 3c96c9b2da..8b24ebd29d 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -25,8 +25,11 @@ const ProtoID = "/libp2p/circuit/relay/0.1.0" const maxMessageSize = 4096 -var RelayAcceptTimeout = time.Minute -var HopConnectTimeout = 10 * time.Second +var ( + RelayAcceptTimeout = time.Minute + HopConnectTimeout = 10 * time.Second + StopHandshakeTimeout = time.Minute +) // Relay is the relay transport and service. type Relay struct { @@ -329,7 +332,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { defer rd.Close() // set handshake deadline - bs.SetDeadline(time.Now().Add(1 * time.Minute)) + bs.SetDeadline(time.Now().Add(StopHandshakeTimeout)) msg.Type = pb.CircuitRelay_STOP.Enum() From 865cd4237bbc60a423057d6aad7d6e76b923e749 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 2 May 2019 12:16:11 +0300 Subject: [PATCH 1172/3965] adjust accept and connect timeouts --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 8b24ebd29d..2a5f740210 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -26,9 +26,9 @@ const ProtoID = "/libp2p/circuit/relay/0.1.0" const maxMessageSize = 4096 var ( - RelayAcceptTimeout = time.Minute - HopConnectTimeout = 10 * time.Second - StopHandshakeTimeout = time.Minute + RelayAcceptTimeout = 10 * time.Second + HopConnectTimeout = 30 * time.Second + StopHandshakeTimeout = 1 * time.Minute ) // Relay is the relay transport and service. From b481650dbc5d6cac76581acfa9e37b2bc8c6bf1b Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 2 May 2019 19:34:42 +0300 Subject: [PATCH 1173/3965] gomod: update go-libp2p-circuit --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index f8fe6dd81f..f8f820dbc1 100644 --- a/go.mod +++ b/go.mod @@ -11,7 +11,7 @@ require ( github.com/libp2p/go-conn-security-multistream v0.0.2 github.com/libp2p/go-libp2p-autonat v0.0.4 github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.4 + github.com/libp2p/go-libp2p-circuit v0.0.5 github.com/libp2p/go-libp2p-crypto v0.0.1 github.com/libp2p/go-libp2p-discovery v0.0.2 github.com/libp2p/go-libp2p-host v0.0.1 diff --git a/go.sum b/go.sum index 1c51b73b1e..038132738c 100644 --- a/go.sum +++ b/go.sum @@ -95,8 +95,8 @@ github.com/libp2p/go-libp2p-autonat v0.0.4 h1:cZzdB9KW1ZkHnSjLCB6aFNw47XS4r+SecC github.com/libp2p/go-libp2p-autonat v0.0.4/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.4 h1:yOgEadnSVFj3e9KLBuLG+edqCImeav0VXxXvcimpOUQ= -github.com/libp2p/go-libp2p-circuit v0.0.4/go.mod h1:p1cHJnB9xnX5/1vZLkXgKwmNEOQQuF/Hp+SkATXnXYk= +github.com/libp2p/go-libp2p-circuit v0.0.5 h1:gLxNF+Vqe3JfAW2lD34Gilh5kAD7eyAdCvKD6lbAvMw= +github.com/libp2p/go-libp2p-circuit v0.0.5/go.mod h1:p1cHJnB9xnX5/1vZLkXgKwmNEOQQuF/Hp+SkATXnXYk= github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-discovery v0.0.2 h1:Rf+20nsFcCnHo4Kxvf8ofAft75+fW+cXy9FonNVyU/g= From 0938e5d8a2a3ad5f5d8706159eefec54c7fe0230 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 3 May 2019 10:27:04 -0700 Subject: [PATCH 1174/3965] don't mask context --- nat.go | 3 --- 1 file changed, 3 deletions(-) diff --git a/nat.go b/nat.go index 371fdb0354..c38030075c 100644 --- a/nat.go +++ b/nat.go @@ -44,9 +44,6 @@ func DiscoverNATs(ctx context.Context) <-chan NAT { go func() { defer close(nats) - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - upnpIg1 := discoverUPNP_IG1(ctx) upnpIg2 := discoverUPNP_IG2(ctx) natpmp := discoverNATPMP(ctx) From afd88c4dceb05d5e17c504d84dfae7675f9a6ddb Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 3 May 2019 22:30:20 +0300 Subject: [PATCH 1175/3965] gomod: update go-libp2p-crypto --- go.mod | 2 +- go.sum | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index f8f820dbc1..55539fe56b 100644 --- a/go.mod +++ b/go.mod @@ -12,7 +12,7 @@ require ( github.com/libp2p/go-libp2p-autonat v0.0.4 github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.5 - github.com/libp2p/go-libp2p-crypto v0.0.1 + github.com/libp2p/go-libp2p-crypto v0.0.2 github.com/libp2p/go-libp2p-discovery v0.0.2 github.com/libp2p/go-libp2p-host v0.0.1 github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 diff --git a/go.sum b/go.sum index 038132738c..f3b37d7bc4 100644 --- a/go.sum +++ b/go.sum @@ -99,6 +99,8 @@ github.com/libp2p/go-libp2p-circuit v0.0.5 h1:gLxNF+Vqe3JfAW2lD34Gilh5kAD7eyAdCv github.com/libp2p/go-libp2p-circuit v0.0.5/go.mod h1:p1cHJnB9xnX5/1vZLkXgKwmNEOQQuF/Hp+SkATXnXYk= github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= +github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= +github.com/libp2p/go-libp2p-crypto v0.0.2/go.mod h1:eETI5OUfBnvARGOHrJz2eWNyTUxEGZnBxMcbUjfIj4I= github.com/libp2p/go-libp2p-discovery v0.0.2 h1:Rf+20nsFcCnHo4Kxvf8ofAft75+fW+cXy9FonNVyU/g= github.com/libp2p/go-libp2p-discovery v0.0.2/go.mod h1:ZkkF9xIFRLA1xCc7bstYFkd80gBGK8Fc1JqGoU2i+zI= github.com/libp2p/go-libp2p-host v0.0.1 h1:dnqusU+DheGcdxrE718kG4XgHNuL2n9eEv8Rg5zy8hQ= @@ -212,6 +214,10 @@ github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a h1:/eS3yfGjQKG+9kayBkj0ip1BGhq6zJ3eaVksphxAaek= +github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= +github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= +github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572/go.mod h1:w0SWMsp6j9O/dk4/ZpIhL+3CkG8ofA2vuv7k+ltqUMc= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= From 08cdc1931373be83bf942046f39b59ef640d8463 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 6 May 2019 14:01:32 +0300 Subject: [PATCH 1176/3965] implement UpsertTag --- p2p/net/connmgr/connmgr.go | 17 +++++++++++++++++ p2p/net/connmgr/connmgr_test.go | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 2629dffdf0..cd2173e6c6 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -244,6 +244,23 @@ func (cm *BasicConnMgr) UntagPeer(p peer.ID, tag string) { delete(pi.tags, tag) } +// UpsertTag is called to insert/update a peer tag +func (cm *BasicConnMgr) UpsertTag(p peer.ID, tag string, upsert func(int) int) { + cm.lk.Lock() + defer cm.lk.Unlock() + + pi, ok := cm.peers[p] + if !ok { + log.Info("tried to upsert tag from untracked peer: ", p) + return + } + + oldval := pi.tags[tag] + newval := upsert(oldval) + pi.value += (newval - oldval) + pi.tags[tag] = newval +} + // CMInfo holds the configuration for BasicConnMgr, as well as status data. type CMInfo struct { // The low watermark, as described in NewConnManager. diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index a5057612ae..b8ebba7e8d 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -521,3 +521,35 @@ func TestPeerProtectionIdempotent(t *testing.T) { t.Error("expected no protections") } } + +func TestUpsertTag(t *testing.T) { + cm := NewConnManager(1, 1, time.Duration(10*time.Minute)) + not := cm.Notifee() + conn := randConn(t, nil) + not.Connected(nil, conn) + rp := conn.RemotePeer() + + cm.UpsertTag(rp, "tag", func(v int) int { return v + 1 }) + if len(cm.peers[rp].tags) != 1 { + t.Fatal("expected a tag") + } + if cm.peers[rp].value != 1 { + t.Fatal("expected a tag value of 1") + } + + cm.UpsertTag(rp, "tag", func(v int) int { return v + 1 }) + if len(cm.peers[rp].tags) != 1 { + t.Fatal("expected a tag") + } + if cm.peers[rp].value != 2 { + t.Fatal("expected a tag value of 2") + } + + cm.UpsertTag(rp, "tag", func(v int) int { return v - 1 }) + if len(cm.peers[rp].tags) != 1 { + t.Fatal("expected a tag") + } + if cm.peers[rp].value != 1 { + t.Fatal("expected a tag value of 1") + } +} From a1bae0a3efceacd0e57e2990a7cec2f9fa773107 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 6 May 2019 15:16:00 +0300 Subject: [PATCH 1177/3965] gomod: update go-libp2p-host, go-libp2p-interface-connmgr --- go.mod | 4 ++-- go.sum | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index 55539fe56b..fe7f505d9f 100644 --- a/go.mod +++ b/go.mod @@ -14,8 +14,8 @@ require ( github.com/libp2p/go-libp2p-circuit v0.0.5 github.com/libp2p/go-libp2p-crypto v0.0.2 github.com/libp2p/go-libp2p-discovery v0.0.2 - github.com/libp2p/go-libp2p-host v0.0.1 - github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 + github.com/libp2p/go-libp2p-host v0.0.3 + github.com/libp2p/go-libp2p-interface-connmgr v0.0.4 github.com/libp2p/go-libp2p-interface-pnet v0.0.1 github.com/libp2p/go-libp2p-loggables v0.0.1 github.com/libp2p/go-libp2p-metrics v0.0.1 diff --git a/go.sum b/go.sum index f3b37d7bc4..003f302598 100644 --- a/go.sum +++ b/go.sum @@ -12,6 +12,8 @@ github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtE github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= github.com/coreos/go-semver v0.2.0 h1:3Jm3tLmsgAYcjC+4Up7hJrFBPr+n7rAqYeSw/SZazuY= github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/coreos/go-semver v0.2.1-0.20180108230905-e214231b295a h1:U0BbGfKnviqVBJQB4etvm+mKx53KfkumNLBt6YeF/0Q= +github.com/coreos/go-semver v0.2.1-0.20180108230905-e214231b295a/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -105,8 +107,12 @@ github.com/libp2p/go-libp2p-discovery v0.0.2 h1:Rf+20nsFcCnHo4Kxvf8ofAft75+fW+cX github.com/libp2p/go-libp2p-discovery v0.0.2/go.mod h1:ZkkF9xIFRLA1xCc7bstYFkd80gBGK8Fc1JqGoU2i+zI= github.com/libp2p/go-libp2p-host v0.0.1 h1:dnqusU+DheGcdxrE718kG4XgHNuL2n9eEv8Rg5zy8hQ= github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= +github.com/libp2p/go-libp2p-host v0.0.3 h1:BB/1Z+4X0rjKP5lbQTmjEjLbDVbrcmLOlA6QDsN5/j4= +github.com/libp2p/go-libp2p-host v0.0.3/go.mod h1:Y/qPyA6C8j2coYyos1dfRm0I8+nvd4TGrDGt4tA7JR8= github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 h1:Q9EkNSLAOF+u90L88qmE9z/fTdjLh8OsJwGw74mkwk4= github.com/libp2p/go-libp2p-interface-connmgr v0.0.1/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= +github.com/libp2p/go-libp2p-interface-connmgr v0.0.4 h1:/LngXETpII5qOD7YjAcQiIxhVtdAk/NQe5t9sC6BR0E= +github.com/libp2p/go-libp2p-interface-connmgr v0.0.4/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= github.com/libp2p/go-libp2p-interface-pnet v0.0.1 h1:7GnzRrBTJHEsofi1ahFdPN9Si6skwXQE9UqR2S+Pkh8= github.com/libp2p/go-libp2p-interface-pnet v0.0.1/go.mod h1:el9jHpQAXK5dnTpKA4yfCNBZXvrzdOU75zz+C6ryp3k= github.com/libp2p/go-libp2p-loggables v0.0.1 h1:HVww9oAnINIxbt69LJNkxD8lnbfgteXR97Xm4p3l9ps= From 1ab3f810773e12a70ef904a6c9a8df4e5d8701ed Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 4 May 2019 22:53:57 +0300 Subject: [PATCH 1178/3965] put a hard limit on the number of active hop streams --- .../internal/circuitv1-deprecated/relay.go | 66 +++++++------------ 1 file changed, 23 insertions(+), 43 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 2a5f740210..3fe6e666ef 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -5,6 +5,7 @@ import ( "fmt" "io" "sync" + "sync/atomic" "time" pb "github.com/libp2p/go-libp2p-circuit/pb" @@ -29,6 +30,9 @@ var ( RelayAcceptTimeout = 10 * time.Second HopConnectTimeout = 30 * time.Second StopHandshakeTimeout = 1 * time.Minute + + HopStreamBuffer = 4096 + HopStreamLimit = 1 << 19 // 512K hops for 1M goroutines ) // Relay is the relay transport and service. @@ -47,9 +51,9 @@ type Relay struct { relays map[peer.ID]struct{} mx sync.Mutex - liveHops map[peer.ID]map[peer.ID]int - lhCount uint64 - lhLk sync.Mutex + // atomic counters + sCount int32 + lhCount int32 } // RelayOpts are options for configuring the relay transport. @@ -88,7 +92,6 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. self: h.ID(), incoming: make(chan *Conn), relays: make(map[peer.ID]struct{}), - liveHops: make(map[peer.ID]map[peer.ID]int), } for _, opt := range opts { @@ -114,48 +117,15 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. } func (r *Relay) addLiveHop(from, to peer.ID) { - r.lhLk.Lock() - defer r.lhLk.Unlock() - - trg, ok := r.liveHops[from] - if !ok { - trg = make(map[peer.ID]int) - r.liveHops[from] = trg - } - trg[to]++ - r.lhCount++ + atomic.AddInt32(&r.lhCount, 1) } func (r *Relay) rmLiveHop(from, to peer.ID) { - r.lhLk.Lock() - defer r.lhLk.Unlock() - - trg, ok := r.liveHops[from] - if !ok { - return - } - var count int - if count, ok = trg[to]; !ok { - return - } - count-- - - r.lhCount-- - if count <= 0 { - delete(trg, to) - if len(trg) == 0 { - delete(r.liveHops, from) - } - } else { - trg[to] = count - } + atomic.AddInt32(&r.lhCount, -1) } -func (r *Relay) GetActiveHops() uint64 { - r.lhLk.Lock() - defer r.lhLk.Unlock() - - return r.lhCount +func (r *Relay) GetActiveHops() int32 { + return atomic.LoadInt32(&r.lhCount) } func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { @@ -283,6 +253,16 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { return } + sCount := atomic.AddInt32(&r.sCount, 1) + lhCount := atomic.LoadInt32(&r.lhCount) + defer atomic.AddInt32(&r.sCount, -1) + + if (sCount + lhCount) > int32(HopStreamLimit) { + log.Warning("hop stream limit exceeded; resetting stream") + s.Reset() + return + } + src, err := peerToPeerInfo(msg.GetSrcPeer()) if err != nil { r.handleError(s, pb.CircuitRelay_HOP_SRC_MULTIADDR_INVALID) @@ -389,7 +369,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { go func() { defer r.rmLiveHop(src.ID, dst.ID) - buf := pool.Get(4096) + buf := pool.Get(HopStreamBuffer) defer pool.Put(buf) count, err := io.CopyBuffer(s, bs, buf) @@ -406,7 +386,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() go func() { - buf := pool.Get(4096) + buf := pool.Get(HopStreamBuffer) defer pool.Put(buf) count, err := io.CopyBuffer(bs, s, buf) From d9338983c2849f376b7e2e6309031a73ac302e57 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 7 May 2019 14:31:36 +0300 Subject: [PATCH 1179/3965] better variable naming --- .../internal/circuitv1-deprecated/relay.go | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 3fe6e666ef..81d5d0a8b1 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -31,8 +31,8 @@ var ( HopConnectTimeout = 30 * time.Second StopHandshakeTimeout = 1 * time.Minute - HopStreamBuffer = 4096 - HopStreamLimit = 1 << 19 // 512K hops for 1M goroutines + HopStreamBufferSize = 4096 + HopStreamLimit = 1 << 19 // 512K hops for 1M goroutines ) // Relay is the relay transport and service. @@ -52,8 +52,8 @@ type Relay struct { mx sync.Mutex // atomic counters - sCount int32 - lhCount int32 + streamCount int32 + liveHopCount int32 } // RelayOpts are options for configuring the relay transport. @@ -117,15 +117,15 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. } func (r *Relay) addLiveHop(from, to peer.ID) { - atomic.AddInt32(&r.lhCount, 1) + atomic.AddInt32(&r.liveHopCount, 1) } func (r *Relay) rmLiveHop(from, to peer.ID) { - atomic.AddInt32(&r.lhCount, -1) + atomic.AddInt32(&r.liveHopCount, -1) } func (r *Relay) GetActiveHops() int32 { - return atomic.LoadInt32(&r.lhCount) + return atomic.LoadInt32(&r.liveHopCount) } func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { @@ -253,11 +253,11 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { return } - sCount := atomic.AddInt32(&r.sCount, 1) - lhCount := atomic.LoadInt32(&r.lhCount) - defer atomic.AddInt32(&r.sCount, -1) + streamCount := atomic.AddInt32(&r.streamCount, 1) + liveHopCount := atomic.LoadInt32(&r.liveHopCount) + defer atomic.AddInt32(&r.streamCount, -1) - if (sCount + lhCount) > int32(HopStreamLimit) { + if (streamCount + liveHopCount) > int32(HopStreamLimit) { log.Warning("hop stream limit exceeded; resetting stream") s.Reset() return @@ -369,7 +369,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { go func() { defer r.rmLiveHop(src.ID, dst.ID) - buf := pool.Get(HopStreamBuffer) + buf := pool.Get(HopStreamBufferSize) defer pool.Put(buf) count, err := io.CopyBuffer(s, bs, buf) @@ -386,7 +386,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() go func() { - buf := pool.Get(HopStreamBuffer) + buf := pool.Get(HopStreamBufferSize) defer pool.Put(buf) count, err := io.CopyBuffer(bs, s, buf) From 1224606dd05f9f70bb3f4944113011bb3c00f441 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 7 May 2019 21:13:59 +0300 Subject: [PATCH 1180/3965] tag peers with live hop streams --- .../internal/circuitv1-deprecated/relay.go | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 81d5d0a8b1..646df850cd 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -118,10 +118,15 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. func (r *Relay) addLiveHop(from, to peer.ID) { atomic.AddInt32(&r.liveHopCount, 1) + r.host.ConnManager().UpsertTag(from, "relay-hop-stream", func(v int) int { return v + 1 }) + r.host.ConnManager().UpsertTag(to, "relay-hop-stream", func(v int) int { return v + 1 }) } func (r *Relay) rmLiveHop(from, to peer.ID) { atomic.AddInt32(&r.liveHopCount, -1) + r.host.ConnManager().UpsertTag(from, "relay-hop-stream", func(v int) int { return v - 1 }) + r.host.ConnManager().UpsertTag(to, "relay-hop-stream", func(v int) int { return v - 1 }) + } func (r *Relay) GetActiveHops() int32 { @@ -364,10 +369,13 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { r.addLiveHop(src.ID, dst.ID) + var wg sync.WaitGroup + wg.Add(2) + // Don't reset streams after finishing or the other side will get an // error, not an EOF. go func() { - defer r.rmLiveHop(src.ID, dst.ID) + defer wg.Done() buf := pool.Get(HopStreamBufferSize) defer pool.Put(buf) @@ -386,6 +394,8 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() go func() { + defer wg.Done() + buf := pool.Get(HopStreamBufferSize) defer pool.Put(buf) @@ -401,6 +411,11 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { } log.Debugf("relayed %d bytes from %s to %s", count, src.ID.Pretty(), dst.ID.Pretty()) }() + + go func() { + wg.Wait() + r.rmLiveHop(src.ID, dst.ID) + }() } func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { From b8616c65b3dfcfdfcdf830dee364e8fbcde22a41 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 7 May 2019 21:33:19 +0300 Subject: [PATCH 1181/3965] use cancelable background context in identify --- p2p/host/basic/basic_host.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 1b11bbdc80..02ca695a63 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -115,12 +115,16 @@ type HostOpts struct { // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, error) { + bgctx, cancel := context.WithCancel(ctx) + h := &BasicHost{ network: net, mux: msmux.NewMultistreamMuxer(), negtimeout: DefaultNegotiationTimeout, AddrsFactory: DefaultAddrsFactory, maResolver: madns.DefaultResolver, + ctx: bgctx, + cancel: cancel, } h.proc = goprocessctx.WithContextAndTeardown(ctx, func() error { @@ -138,7 +142,7 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, h.ids = opts.IdentifyService } else { // we can't set this as a default above because it depends on the *BasicHost. - h.ids = identify.NewIDService(ctx, h) + h.ids = identify.NewIDService(bgctx, h) } if uint64(opts.NegotiationTimeout) != 0 { @@ -171,10 +175,6 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, net.SetConnHandler(h.newConnHandler) net.SetStreamHandler(h.newStreamHandler) - bgctx, cancel := context.WithCancel(ctx) - h.ctx = bgctx - h.cancel = cancel - return h, nil } From 66dfe163ba40cd72c3bd10bb6ddea17a3286117f Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 7 May 2019 22:19:18 +0300 Subject: [PATCH 1182/3965] don't use an extra goroutine for live hop cleanup --- .../internal/circuitv1-deprecated/relay.go | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 646df850cd..0afcf0467a 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -369,13 +369,18 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { r.addLiveHop(src.ID, dst.ID) - var wg sync.WaitGroup - wg.Add(2) + goroutines := new(int32) + *goroutines = 2 + done := func() { + if atomic.AddInt32(goroutines, -1) == 0 { + r.rmLiveHop(src.ID, dst.ID) + } + } // Don't reset streams after finishing or the other side will get an // error, not an EOF. go func() { - defer wg.Done() + defer done() buf := pool.Get(HopStreamBufferSize) defer pool.Put(buf) @@ -394,7 +399,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() go func() { - defer wg.Done() + defer done() buf := pool.Get(HopStreamBufferSize) defer pool.Put(buf) @@ -411,11 +416,6 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { } log.Debugf("relayed %d bytes from %s to %s", count, src.ID.Pretty(), dst.ID.Pretty()) }() - - go func() { - wg.Wait() - r.rmLiveHop(src.ID, dst.ID) - }() } func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { From b5b6230dedef9fbe13f57c2d4b56d32caaa6b111 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 7 May 2019 22:39:38 +0300 Subject: [PATCH 1183/3965] gomod: update go-libp2p-circuit --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index fe7f505d9f..1461bebf2d 100644 --- a/go.mod +++ b/go.mod @@ -11,7 +11,7 @@ require ( github.com/libp2p/go-conn-security-multistream v0.0.2 github.com/libp2p/go-libp2p-autonat v0.0.4 github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.5 + github.com/libp2p/go-libp2p-circuit v0.0.6 github.com/libp2p/go-libp2p-crypto v0.0.2 github.com/libp2p/go-libp2p-discovery v0.0.2 github.com/libp2p/go-libp2p-host v0.0.3 diff --git a/go.sum b/go.sum index 003f302598..bd60e00c1f 100644 --- a/go.sum +++ b/go.sum @@ -97,8 +97,8 @@ github.com/libp2p/go-libp2p-autonat v0.0.4 h1:cZzdB9KW1ZkHnSjLCB6aFNw47XS4r+SecC github.com/libp2p/go-libp2p-autonat v0.0.4/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.5 h1:gLxNF+Vqe3JfAW2lD34Gilh5kAD7eyAdCvKD6lbAvMw= -github.com/libp2p/go-libp2p-circuit v0.0.5/go.mod h1:p1cHJnB9xnX5/1vZLkXgKwmNEOQQuF/Hp+SkATXnXYk= +github.com/libp2p/go-libp2p-circuit v0.0.6 h1:egD2CKFVdqnHgIHzPkM6J7m3MKZpFqoTPDfxBqQ7kRQ= +github.com/libp2p/go-libp2p-circuit v0.0.6/go.mod h1:W34ISBRpoCPUeOR26xzTbLo+s3hDO9153hJCfvHzBlg= github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= From 26fe9dc6c989cd78976fdd86023c4dfe12be75f7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 7 May 2019 11:58:57 -0700 Subject: [PATCH 1184/3965] dep: update goprocess 1. It now supports go modules. 2. It should spawn fewer goroutines, especially when used with contexts. --- go.mod | 2 +- go.sum | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index fe7f505d9f..b0202806e2 100644 --- a/go.mod +++ b/go.mod @@ -6,7 +6,7 @@ require ( github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 github.com/ipfs/go-log v0.0.1 - github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 + github.com/jbenet/goprocess v0.1.2 github.com/libp2p/go-conn-security v0.0.1 github.com/libp2p/go-conn-security-multistream v0.0.2 github.com/libp2p/go-libp2p-autonat v0.0.4 diff --git a/go.sum b/go.sum index 003f302598..dd523351ba 100644 --- a/go.sum +++ b/go.sum @@ -65,10 +65,14 @@ github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEM github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec h1:DQqZhhDvrTrEQ3Qod5yfavcA064e53xlQ+xajiorXgM= github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec/go.mod h1:rGaEvXB4uRSZMmzKNLoXvTu1sfx+1kv/DojUlPrSZGs= +github.com/jbenet/go-cienv v0.1.0 h1:Vc/s0QbQtoxX8MwwSLWWh+xNNZvM3Lw7NsTcHrvvhMc= +github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA= github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2 h1:vhC1OXXiT9R2pczegwz6moDvuRpggaroAXhPIseh57A= github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2/go.mod h1:8GXXJV31xl8whumTzdZsTt3RnUIiPqzkyf7mxToRCMs= github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 h1:bspPhN+oKYFk5fcGNuQzp6IGzYQSenLEgH3s6jkXrWw= github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsjFq/qrU3Rar62tu1gASgGw6chQbSh/XgIIXCY= +github.com/jbenet/goprocess v0.1.2 h1:TfvRwwUfgT/umj/REz2B8IxAKzUEX84+xdRAQ+eF2B8= +github.com/jbenet/goprocess v0.1.2/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= From d3d5351d8160b6b531fc39327a8313a3b8bb6228 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 7 May 2019 15:40:45 -0700 Subject: [PATCH 1185/3965] ping: return a stream of results Otherwise, we can't return errors. This is a breaking change but unlikely to have a large impact on anyone but go-ipfs. Part of https://github.com/ipfs/go-ipfs/issues/6298 --- p2p/protocol/ping/ping.go | 47 ++++++++++++++++++++++++++-------- p2p/protocol/ping/ping_test.go | 12 ++++----- 2 files changed, 42 insertions(+), 17 deletions(-) diff --git a/p2p/protocol/ping/ping.go b/p2p/protocol/ping/ping.go index b76133cc16..8f33513b5f 100644 --- a/p2p/protocol/ping/ping.go +++ b/p2p/protocol/ping/ping.go @@ -71,42 +71,67 @@ func (p *PingService) PingHandler(s inet.Stream) { } } -func (ps *PingService) Ping(ctx context.Context, p peer.ID) (<-chan time.Duration, error) { +// Result is a result of a ping attempt, either an RTT or an error. +type Result struct { + RTT time.Duration + Error error +} + +func (ps *PingService) Ping(ctx context.Context, p peer.ID) <-chan Result { return Ping(ctx, ps.Host, p) } -func Ping(ctx context.Context, h host.Host, p peer.ID) (<-chan time.Duration, error) { +// Ping pings the remote peer until the context is canceled, returning a stream +// of RTTs or errors. +func Ping(ctx context.Context, h host.Host, p peer.ID) <-chan Result { s, err := h.NewStream(ctx, p, ID) if err != nil { - return nil, err + ch := make(chan Result, 1) + ch <- Result{Error: err} + close(ch) + return ch } - out := make(chan time.Duration) + ctx, cancel := context.WithCancel(ctx) + + out := make(chan Result) go func() { defer close(out) - defer s.Reset() + defer cancel() + for { select { case <-ctx.Done(): return default: - t, err := ping(s) - if err != nil { - log.Debugf("ping error: %s", err) + var res Result + res.RTT, res.Error = ping(s) + + // canceled, ignore everything. + if ctx.Err() != nil { return } - h.Peerstore().RecordLatency(p, t) + // No error, record the RTT. + if res.Error == nil { + h.Peerstore().RecordLatency(p, res.RTT) + } + select { - case out <- t: + case out <- res: case <-ctx.Done(): return } } } }() + go func() { + // forces the ping to abort. + <-ctx.Done() + s.Reset() + }() - return out, nil + return out } func ping(s inet.Stream) (time.Duration, error) { diff --git a/p2p/protocol/ping/ping_test.go b/p2p/protocol/ping/ping_test.go index f8ea0cc79e..043eac0fd1 100644 --- a/p2p/protocol/ping/ping_test.go +++ b/p2p/protocol/ping/ping_test.go @@ -37,15 +37,15 @@ func TestPing(t *testing.T) { func testPing(t *testing.T, ps *ping.PingService, p peer.ID) { pctx, cancel := context.WithCancel(context.Background()) defer cancel() - ts, err := ps.Ping(pctx, p) - if err != nil { - t.Fatal(err) - } + ts := ps.Ping(pctx, p) for i := 0; i < 5; i++ { select { - case took := <-ts: - t.Log("ping took: ", took) + case res := <-ts: + if res.Error != nil { + t.Fatal(res.Error) + } + t.Log("ping took: ", res.RTT) case <-time.After(time.Second * 4): t.Fatal("failed to receive ping") } From d0ab45164c3f9715db464e1f45832a8535b18ca7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 7 May 2019 15:43:48 -0700 Subject: [PATCH 1186/3965] ping: simplify ping loop --- p2p/protocol/ping/ping.go | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/p2p/protocol/ping/ping.go b/p2p/protocol/ping/ping.go index 8f33513b5f..d8e143bd07 100644 --- a/p2p/protocol/ping/ping.go +++ b/p2p/protocol/ping/ping.go @@ -99,29 +99,24 @@ func Ping(ctx context.Context, h host.Host, p peer.ID) <-chan Result { defer close(out) defer cancel() - for { + for ctx.Err() == nil { + var res Result + res.RTT, res.Error = ping(s) + + // canceled, ignore everything. + if ctx.Err() != nil { + return + } + + // No error, record the RTT. + if res.Error == nil { + h.Peerstore().RecordLatency(p, res.RTT) + } + select { + case out <- res: case <-ctx.Done(): return - default: - var res Result - res.RTT, res.Error = ping(s) - - // canceled, ignore everything. - if ctx.Err() != nil { - return - } - - // No error, record the RTT. - if res.Error == nil { - h.Peerstore().RecordLatency(p, res.RTT) - } - - select { - case out <- res: - case <-ctx.Done(): - return - } } } }() From 1b9cf3e2cba18adc0976ca9b27c36b445340416c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 7 May 2019 15:51:29 -0700 Subject: [PATCH 1187/3965] test: disable flaky autorelay test in CI This keeps breaking travis. --- go.mod | 1 + p2p/host/relay/autorelay_test.go | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 4530dd72fe..878501790d 100644 --- a/go.mod +++ b/go.mod @@ -6,6 +6,7 @@ require ( github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 github.com/ipfs/go-log v0.0.1 + github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.2 github.com/libp2p/go-conn-security v0.0.1 github.com/libp2p/go-conn-security-multistream v0.0.2 diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index ff3d7601ab..4f5485a387 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -7,6 +7,7 @@ import ( "testing" "time" + cienv "github.com/jbenet/go-cienv" libp2p "github.com/libp2p/go-libp2p" relay "github.com/libp2p/go-libp2p/p2p/host/relay" @@ -144,7 +145,9 @@ func connect(t *testing.T, a, b host.Host) { // and the actual test! func TestAutoRelay(t *testing.T) { - //t.Skip("fails 99% of the time") + if cienv.IsRunning() { + t.Skip("disabled on CI: fails 99% of the time") + } manet.Private4 = []*net.IPNet{} From d8fed21c5d130173de95b41f181cd6a9d397c6fd Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 8 May 2019 20:11:05 +0300 Subject: [PATCH 1188/3965] gomod: update go-multistream --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 878501790d..7ea9600af4 100644 --- a/go.mod +++ b/go.mod @@ -40,7 +40,7 @@ require ( github.com/multiformats/go-multiaddr v0.0.1 github.com/multiformats/go-multiaddr-dns v0.0.2 github.com/multiformats/go-multiaddr-net v0.0.1 - github.com/multiformats/go-multistream v0.0.2 + github.com/multiformats/go-multistream v0.0.3 github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible diff --git a/go.sum b/go.sum index 02d34eaedd..d031bd30ae 100644 --- a/go.sum +++ b/go.sum @@ -211,8 +211,8 @@ github.com/multiformats/go-multihash v0.0.1 h1:HHwN1K12I+XllBCrqKnhX949Orn4oawPk github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= github.com/multiformats/go-multistream v0.0.1 h1:JV4VfSdY9n7ECTtY59/TlSyFCzRILvYx4T4Ws8ZgihU= github.com/multiformats/go-multistream v0.0.1/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= -github.com/multiformats/go-multistream v0.0.2 h1:uWy8f8Icfo9t/U80+CR5oqM84gSgNqh8xLB7rVslfcw= -github.com/multiformats/go-multistream v0.0.2/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= +github.com/multiformats/go-multistream v0.0.3 h1:sRTU9QWosQWXteaM0PHCFVWSIHWxOyjdwL4MF7Lsdfg= +github.com/multiformats/go-multistream v0.0.3/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0 h1:WSHQ+IS43OoUrWtD1/bbclrwK8TTH5hzp+umCiuxHgs= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= From e8e29ddabca1c1181edf68f4b7bba57d3f73b4dd Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 May 2019 14:38:23 -0700 Subject: [PATCH 1189/3965] dep: update multihash, multiaddr, and peer Also updates our base58 library. --- go.mod | 4 ++-- go.sum | 19 +++++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index 7ea9600af4..216795b7ed 100644 --- a/go.mod +++ b/go.mod @@ -23,7 +23,7 @@ require ( github.com/libp2p/go-libp2p-nat v0.0.4 github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 - github.com/libp2p/go-libp2p-peer v0.0.1 + github.com/libp2p/go-libp2p-peer v0.1.1 github.com/libp2p/go-libp2p-peerstore v0.0.5 github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 @@ -37,7 +37,7 @@ require ( github.com/libp2p/go-testutil v0.0.1 github.com/libp2p/go-ws-transport v0.0.2 github.com/miekg/dns v1.1.4 // indirect - github.com/multiformats/go-multiaddr v0.0.1 + github.com/multiformats/go-multiaddr v0.0.2 github.com/multiformats/go-multiaddr-dns v0.0.2 github.com/multiformats/go-multiaddr-net v0.0.1 github.com/multiformats/go-multistream v0.0.3 diff --git a/go.sum b/go.sum index d031bd30ae..e067f54214 100644 --- a/go.sum +++ b/go.sum @@ -133,6 +133,8 @@ github.com/libp2p/go-libp2p-netutil v0.0.1 h1:LgD6+skofkOx8z6odD9+MZHKjupv3ng1u6 github.com/libp2p/go-libp2p-netutil v0.0.1/go.mod h1:GdusFvujWZI9Vt0X5BKqwWWmZFxecf9Gt03cKxm2f/Q= github.com/libp2p/go-libp2p-peer v0.0.1 h1:0qwAOljzYewINrU+Kndoc+1jAL7vzY/oY2Go4DCGfyY= github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= +github.com/libp2p/go-libp2p-peer v0.1.1 h1:qGCWD1a+PyZcna6htMPo26jAtqirVnJ5NvBQIKV7rRY= +github.com/libp2p/go-libp2p-peer v0.1.1/go.mod h1:jkF12jGB4Gk/IOo+yomm+7oLWxF278F7UnrYUQ1Q8es= github.com/libp2p/go-libp2p-peerstore v0.0.1 h1:twKovq8YK5trLrd3nB7PD2Zu9JcyAIdm7Bz9yBWjhq8= github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= github.com/libp2p/go-libp2p-peerstore v0.0.5 h1:/qc3asc93924TaqVulOnY/+c4tpzomSE/ijT7LWF09M= @@ -194,12 +196,18 @@ github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0 github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ= github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16 h1:5W7KhL8HVF3XCFOweFD3BNESdnO8ewyYTFT2R+/b8FQ= github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= +github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5 h1:l16XLUUJ34wIz+RIvLhSwGvLvKyy+W598b135bJN6mg= +github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/mr-tron/base58 v1.1.0 h1:Y51FGVJ91WBqCEabAi5OPUz38eAx8DakuAm5svLcsfQ= github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= +github.com/mr-tron/base58 v1.1.2 h1:ZEw4I2EgPKDJ2iEw0cNmLB3ROrEmkOtXIkaG7wZg+78= +github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI= github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA= github.com/multiformats/go-multiaddr v0.0.1 h1:/QUV3VBMDI6pi6xfiw7lr6xhDWWvQKn9udPn68kLSdY= github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.0.2 h1:RBysRCv5rv3FWlhKWKoXv8tnsCUpEpIZpCmqAGZos2s= +github.com/multiformats/go-multiaddr v0.0.2/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= @@ -209,6 +217,8 @@ github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmr github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= github.com/multiformats/go-multihash v0.0.1 h1:HHwN1K12I+XllBCrqKnhX949Orn4oawPkegHMu2vDqQ= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= +github.com/multiformats/go-multihash v0.0.5 h1:1wxmCvTXAifAepIMyF39vZinRw5sbqjPs/UIi93+uik= +github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= github.com/multiformats/go-multistream v0.0.1 h1:JV4VfSdY9n7ECTtY59/TlSyFCzRILvYx4T4Ws8ZgihU= github.com/multiformats/go-multistream v0.0.1/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/multiformats/go-multistream v0.0.3 h1:sRTU9QWosQWXteaM0PHCFVWSIHWxOyjdwL4MF7Lsdfg= @@ -228,6 +238,8 @@ github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a h1:/eS3yfGjQ github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572/go.mod h1:w0SWMsp6j9O/dk4/ZpIhL+3CkG8ofA2vuv7k+ltqUMc= +github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= +github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= @@ -258,10 +270,15 @@ golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnf golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25 h1:jsG6UpNLt9iAsb0S2AGW28DveNzzgmbXR+ENoPjUeIU= golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734 h1:p/H982KKEjUnLJkM3tt/LemDnOc1GiZL5FCVlORJ5zo= +golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7 h1:C2F/nMkR/9sfUTpvR3QrjBuTdvMUC/cFajkphs1YLQo= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f h1:wMNYb4v58l5UBM7MYRLPG6ZhfOqbKu7X5eyFl8ZhKvA= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -270,6 +287,8 @@ golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e h1:ZytStCyV048ZqDsWHiYDdoI2Vd4msMcrDECFxS+tL9c= golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= From 4f9f73108814fd4a389820fbfe2faeff928da18d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 9 May 2019 00:27:58 +0100 Subject: [PATCH 1190/3965] segment the memory peerstore; granular locks. --- p2p/host/peerstore/pstoremem/addr_book.go | 125 +++++++++++++++------- 1 file changed, 85 insertions(+), 40 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index cb9576ace4..7aa0c3f3e8 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -4,6 +4,7 @@ import ( "context" "sort" "sync" + "sync/atomic" "time" logging "github.com/ipfs/go-log" @@ -26,7 +27,19 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { return t.After(e.Expires) } -var _ pstore.AddrBook = (*memoryAddrBook)(nil) +type segments [256]*segment + +type segment struct { + size uint32 + + lk sync.RWMutex + addrs map[peer.ID]map[string]*expiringAddr +} + +func (s *segments) get(id peer.ID) *segment { + b := []byte(id) + return s[b[len(b)-1]%byte(255)] +} // memoryAddrBook manages addresses. type memoryAddrBook struct { @@ -34,7 +47,7 @@ type memoryAddrBook struct { // Use pointers to save memory. Maps always leave some fraction of their // space unused. storing the *values* directly in the map will // drastically increase the space waste. In our case, by 6x. - addrs map[peer.ID]map[string]*expiringAddr + segments segments ctx context.Context cancel func() @@ -42,11 +55,18 @@ type memoryAddrBook struct { subManager *AddrSubManager } +var _ pstore.AddrBook = (*memoryAddrBook)(nil) + func NewAddrBook() pstore.AddrBook { ctx, cancel := context.WithCancel(context.Background()) ab := &memoryAddrBook{ - addrs: make(map[peer.ID]map[string]*expiringAddr), + segments: func() (ret segments) { + for i, _ := range ret { + ret[i] = &segment{addrs: make(map[peer.ID]map[string]*expiringAddr)} + } + return ret + }(), subManager: NewAddrSubManager(), ctx: ctx, cancel: cancel, @@ -79,29 +99,37 @@ func (mab *memoryAddrBook) Close() error { // gc garbage collects the in-memory address book. func (mab *memoryAddrBook) gc() { - mab.addrmu.Lock() - defer mab.addrmu.Unlock() - now := time.Now() - for p, amap := range mab.addrs { - for k, addr := range amap { - if addr.ExpiredBy(now) { - delete(amap, k) + for _, s := range mab.segments { + s.lk.Lock() + for p, amap := range s.addrs { + for k, addr := range amap { + if addr.ExpiredBy(now) { + delete(amap, k) + } + } + if len(amap) == 0 { + delete(s.addrs, p) } } - if len(amap) == 0 { - delete(mab.addrs, p) - } + s.lk.Unlock() } + } func (mab *memoryAddrBook) PeersWithAddrs() peer.IDSlice { - mab.addrmu.RLock() - defer mab.addrmu.RUnlock() + var length uint32 + for _, s := range mab.segments { + length += atomic.LoadUint32(&s.size) + } - pids := make(peer.IDSlice, 0, len(mab.addrs)) - for pid := range mab.addrs { - pids = append(pids, pid) + pids := make(peer.IDSlice, 0, length) + for _, s := range mab.segments { + s.lk.RLock() + for pid, _ := range s.addrs { + pids = append(pids, pid) + } + s.lk.RUnlock() } return pids } @@ -115,18 +143,22 @@ func (mab *memoryAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Durati // (time-to-live), after which the address is no longer valid. // If the manager has a longer TTL, the operation is a no-op for that address func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - mab.addrmu.Lock() - defer mab.addrmu.Unlock() - // if ttl is zero, exit. nothing to do. if ttl <= 0 { return } - amap := mab.addrs[p] + s := mab.segments.get(p) + s.lk.Lock() + defer s.lk.Unlock() + + // update the segment size + defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) + + amap := s.addrs[p] if amap == nil { amap = make(map[string]*expiringAddr, len(addrs)) - mab.addrs[p] = amap + s.addrs[p] = amap } exp := time.Now().Add(ttl) for _, addr := range addrs { @@ -152,13 +184,17 @@ func (mab *memoryAddrBook) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Durati // SetAddrs sets the ttl on addresses. This clears any TTL there previously. // This is used when we receive the best estimate of the validity of an address. func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { - mab.addrmu.Lock() - defer mab.addrmu.Unlock() + s := mab.segments.get(p) + s.lk.Lock() + defer s.lk.Unlock() + + // update the segment size + defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) - amap := mab.addrs[p] + amap := s.addrs[p] if amap == nil { amap = make(map[string]*expiringAddr, len(addrs)) - mab.addrs[p] = amap + s.addrs[p] = amap } exp := time.Now().Add(ttl) @@ -172,7 +208,6 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du if ttl > 0 { amap[addrstr] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} - mab.subManager.BroadcastAddr(p, addr) } else { delete(amap, addrstr) @@ -183,10 +218,14 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du // UpdateAddrs updates the addresses associated with the given peer that have // the given oldTTL to have the given newTTL. func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { - mab.addrmu.Lock() - defer mab.addrmu.Unlock() + s := mab.segments.get(p) + s.lk.Lock() + defer s.lk.Unlock() + + // update the segment size + defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) - amap, found := mab.addrs[p] + amap, found := s.addrs[p] if !found { return } @@ -203,10 +242,11 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t // Addresses returns all known (and valid) addresses for a given func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { - mab.addrmu.RLock() - defer mab.addrmu.RUnlock() + s := mab.segments.get(p) + s.lk.RLock() + defer s.lk.RUnlock() - amap, found := mab.addrs[p] + amap, found := s.addrs[p] if !found { return nil } @@ -224,19 +264,24 @@ func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { // ClearAddrs removes all previously stored addresses func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { - mab.addrmu.Lock() - defer mab.addrmu.Unlock() + s := mab.segments.get(p) + s.lk.Lock() + defer s.lk.Unlock() + + // update the segment size + defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) - delete(mab.addrs, p) + delete(s.addrs, p) } // AddrStream returns a channel on which all new addresses discovered for a // given peer ID will be published. func (mab *memoryAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { - mab.addrmu.RLock() - defer mab.addrmu.RUnlock() + s := mab.segments.get(p) + s.lk.RLock() + defer s.lk.RUnlock() - baseaddrslice := mab.addrs[p] + baseaddrslice := s.addrs[p] initial := make([]ma.Multiaddr, 0, len(baseaddrslice)) for _, a := range baseaddrslice { initial = append(initial, a.Addr) From eff34c57a7e35a3e85cad7078fd7c8bec2eae85c Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 9 May 2019 11:20:12 +0300 Subject: [PATCH 1191/3965] update segment size on gc --- p2p/host/peerstore/pstoremem/addr_book.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 7aa0c3f3e8..cd7a920cf1 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -112,6 +112,7 @@ func (mab *memoryAddrBook) gc() { delete(s.addrs, p) } } + atomic.StoreUint32(&s.size, uint32(len(s.addrs))) s.lk.Unlock() } From 0c0b648d8ac3e003e88676107e050c3dff220f05 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 9 May 2019 12:14:03 +0300 Subject: [PATCH 1192/3965] segmented protocol info --- p2p/host/peerstore/peerstore.go | 145 ++++++++++++++++---------------- 1 file changed, 74 insertions(+), 71 deletions(-) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index f7f87b8593..fde9cd4c6d 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -10,8 +10,38 @@ import ( var _ Peerstore = (*peerstore)(nil) -const maxInternedProtocols = 512 -const maxInternedProtocolSize = 256 +const maxInternedProtocols = 64 +const maxInternedProtocolSize = 128 + +type segment struct { + lk sync.RWMutex + interned map[string]string + protocols map[peer.ID]map[string]struct{} +} + +type segments [256]*segment + +func (s *segments) get(id peer.ID) *segment { + b := []byte(id) + return s[b[len(b)-1]] +} + +func (s *segment) internProtocol(proto string) string { + if len(proto) > maxInternedProtocolSize { + return proto + } + + if interned, ok := s.interned[proto]; ok { + return interned + } + + if len(s.interned) >= maxInternedProtocols { + s.interned = make(map[string]string, maxInternedProtocols) + } + + s.interned[proto] = proto + return proto +} type peerstore struct { Metrics @@ -20,20 +50,27 @@ type peerstore struct { AddrBook PeerMetadata - // lock for protocol information, separate from datastore lock - protolock sync.RWMutex - internedProtocols map[string]string + // segments for protocol information + segments segments } // NewPeerstore creates a data structure that stores peer data, backed by the // supplied implementations of KeyBook, AddrBook and PeerMetadata. func NewPeerstore(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { return &peerstore{ - KeyBook: kb, - AddrBook: ab, - PeerMetadata: md, - Metrics: NewMetrics(), - internedProtocols: make(map[string]string), + KeyBook: kb, + AddrBook: ab, + PeerMetadata: md, + Metrics: NewMetrics(), + segments: func() (ret segments) { + for i := range ret { + ret[i] = &segment{ + interned: make(map[string]string), + protocols: make(map[peer.ID]map[string]struct{}), + } + } + return ret + }(), } } @@ -80,77 +117,46 @@ func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { } } -func (ps *peerstore) internProtocol(s string) string { - if len(s) > maxInternedProtocolSize { - return s - } - - if interned, ok := ps.internedProtocols[s]; ok { - return interned - } - - if len(ps.internedProtocols) >= maxInternedProtocols { - ps.internedProtocols = make(map[string]string, maxInternedProtocols) - } - - ps.internedProtocols[s] = s - return s -} - func (ps *peerstore) SetProtocols(p peer.ID, protos ...string) error { - ps.protolock.Lock() - defer ps.protolock.Unlock() + s := ps.segments.get(p) + s.lk.Lock() + defer s.lk.Unlock() - protomap := make(map[string]struct{}, len(protos)) + newprotos := make(map[string]struct{}, len(protos)) for _, proto := range protos { - protomap[ps.internProtocol(proto)] = struct{}{} + newprotos[s.internProtocol(proto)] = struct{}{} } - return ps.Put(p, "protocols", protomap) + s.protocols[p] = newprotos + + return nil } func (ps *peerstore) AddProtocols(p peer.ID, protos ...string) error { - ps.protolock.Lock() - defer ps.protolock.Unlock() - protomap, err := ps.getProtocolMap(p) - if err != nil { - return err + s := ps.segments.get(p) + s.lk.Lock() + defer s.lk.Unlock() + + protomap, ok := s.protocols[p] + if !ok { + protomap = make(map[string]struct{}) + s.protocols[p] = protomap } for _, proto := range protos { - protomap[ps.internProtocol(proto)] = struct{}{} + protomap[s.internProtocol(proto)] = struct{}{} } - return ps.Put(p, "protocols", protomap) -} - -func (ps *peerstore) getProtocolMap(p peer.ID) (map[string]struct{}, error) { - iprotomap, err := ps.Get(p, "protocols") - switch err { - default: - return nil, err - case ErrNotFound: - return make(map[string]struct{}), nil - case nil: - cast, ok := iprotomap.(map[string]struct{}) - if !ok { - return nil, fmt.Errorf("stored protocol set was not a map") - } - - return cast, nil - } + return nil } func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { - ps.protolock.RLock() - defer ps.protolock.RUnlock() - pmap, err := ps.getProtocolMap(p) - if err != nil { - return nil, err - } + s := ps.segments.get(p) + s.lk.RLock() + defer s.lk.RUnlock() - out := make([]string, 0, len(pmap)) - for k := range pmap { + out := make([]string, 0, len(s.protocols)) + for k := range s.protocols[p] { out = append(out, k) } @@ -158,16 +164,13 @@ func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { } func (ps *peerstore) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { - ps.protolock.RLock() - defer ps.protolock.RUnlock() - pmap, err := ps.getProtocolMap(p) - if err != nil { - return nil, err - } + s := ps.segments.get(p) + s.lk.RLock() + defer s.lk.RUnlock() out := make([]string, 0, len(protos)) for _, proto := range protos { - if _, ok := pmap[proto]; ok { + if _, ok := s.protocols[p][proto]; ok { out = append(out, proto) } } From 708d17c2d0d54f19d596c7ff28570e932758c934 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 9 May 2019 15:18:01 -0700 Subject: [PATCH 1193/3965] dep: update goprocess fixes a panic and has a few more perf improvements --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 216795b7ed..c4f65f4bf1 100644 --- a/go.mod +++ b/go.mod @@ -7,7 +7,7 @@ require ( github.com/ipfs/go-ipfs-util v0.0.1 github.com/ipfs/go-log v0.0.1 github.com/jbenet/go-cienv v0.1.0 - github.com/jbenet/goprocess v0.1.2 + github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security v0.0.1 github.com/libp2p/go-conn-security-multistream v0.0.2 github.com/libp2p/go-libp2p-autonat v0.0.4 diff --git a/go.sum b/go.sum index e067f54214..a109b2cd7e 100644 --- a/go.sum +++ b/go.sum @@ -71,8 +71,8 @@ github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2 h1:vhC1 github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2/go.mod h1:8GXXJV31xl8whumTzdZsTt3RnUIiPqzkyf7mxToRCMs= github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 h1:bspPhN+oKYFk5fcGNuQzp6IGzYQSenLEgH3s6jkXrWw= github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsjFq/qrU3Rar62tu1gASgGw6chQbSh/XgIIXCY= -github.com/jbenet/goprocess v0.1.2 h1:TfvRwwUfgT/umj/REz2B8IxAKzUEX84+xdRAQ+eF2B8= -github.com/jbenet/goprocess v0.1.2/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= +github.com/jbenet/goprocess v0.1.3 h1:YKyIEECS/XvcfHtBzxtjBBbWK+MbvA6dG8ASiqwvr10= +github.com/jbenet/goprocess v0.1.3/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= From dbd00cdb069f69e862fc8177c8b005b390bcaa9a Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 10 May 2019 13:53:52 +0300 Subject: [PATCH 1194/3965] implement segmented lock to eliminate big lock contention --- p2p/net/connmgr/connmgr.go | 123 ++++++++++++++++++++++++------------- 1 file changed, 79 insertions(+), 44 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index cd2173e6c6..3010e30579 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -4,6 +4,7 @@ import ( "context" "sort" "sync" + "sync/atomic" "time" logging "github.com/ipfs/go-log" @@ -25,12 +26,11 @@ var log = logging.Logger("connmgr") // // See configuration parameters in NewConnManager. type BasicConnMgr struct { - lk sync.Mutex highWater int lowWater int - connCount int + connCount int32 gracePeriod time.Duration - peers map[peer.ID]*peerInfo + segments segments plk sync.RWMutex protected map[peer.ID]map[string]struct{} @@ -43,6 +43,27 @@ type BasicConnMgr struct { var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) +type segment struct { + sync.Mutex + peers map[peer.ID]*peerInfo +} + +type segments [256]*segment + +func (s *segments) get(id peer.ID) *segment { + b := []byte(id) + return s[b[len(b)-1]] +} + +func (s *segments) countPeers() (count int) { + for _, seg := range s { + seg.Lock() + count += len(seg.peers) + seg.Unlock() + } + return count +} + // NewConnManager creates a new BasicConnMgr with the provided params: // * lo and hi are watermarks governing the number of connections that'll be maintained. // When the peer count exceeds the 'high watermark', as many peers will be pruned (and @@ -54,10 +75,17 @@ func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { highWater: hi, lowWater: low, gracePeriod: grace, - peers: make(map[peer.ID]*peerInfo), trimRunningCh: make(chan struct{}, 1), protected: make(map[peer.ID]map[string]struct{}, 16), silencePeriod: SilencePeriod, + segments: func() (ret segments) { + for i := range ret { + ret[i] = &segment{ + peers: make(map[peer.ID]*peerInfo), + } + } + return ret + }(), } } @@ -129,27 +157,29 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { // getConnsToClose runs the heuristics described in TrimOpenConns and returns the // connections to close. func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { - cm.lk.Lock() - defer cm.lk.Unlock() - if cm.lowWater == 0 || cm.highWater == 0 { // disabled return nil } now := time.Now() - if len(cm.peers) < cm.lowWater { + npeers := cm.segments.countPeers() + if npeers < cm.lowWater { log.Info("open connection count below limit") return nil } var candidates []*peerInfo cm.plk.RLock() - for id, inf := range cm.peers { - if _, ok := cm.protected[id]; ok { - // skip over protected peer. - continue + for _, s := range cm.segments { + s.Lock() + for id, inf := range s.peers { + if _, ok := cm.protected[id]; ok { + // skip over protected peer. + continue + } + candidates = append(candidates, inf) } - candidates = append(candidates, inf) + s.Unlock() } cm.plk.RUnlock() @@ -158,7 +188,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { return candidates[i].value < candidates[j].value }) - target := len(cm.peers) - cm.lowWater + target := npeers - cm.lowWater // 2x number of peers we're disconnecting from because we may have more // than one connection per peer. Slightly over allocating isn't an issue @@ -187,10 +217,11 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { // GetTagInfo is called to fetch the tag information associated with a given // peer, nil is returned if p refers to an unknown peer. func (cm *BasicConnMgr) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { - cm.lk.Lock() - defer cm.lk.Unlock() + s := cm.segments.get(p) + s.Lock() + defer s.Unlock() - pi, ok := cm.peers[p] + pi, ok := s.peers[p] if !ok { return nil } @@ -214,10 +245,11 @@ func (cm *BasicConnMgr) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { // TagPeer is called to associate a string and integer with a given peer. func (cm *BasicConnMgr) TagPeer(p peer.ID, tag string, val int) { - cm.lk.Lock() - defer cm.lk.Unlock() + s := cm.segments.get(p) + s.Lock() + defer s.Unlock() - pi, ok := cm.peers[p] + pi, ok := s.peers[p] if !ok { log.Info("tried to tag conn from untracked peer: ", p) return @@ -230,10 +262,11 @@ func (cm *BasicConnMgr) TagPeer(p peer.ID, tag string, val int) { // UntagPeer is called to disassociate a string and integer from a given peer. func (cm *BasicConnMgr) UntagPeer(p peer.ID, tag string) { - cm.lk.Lock() - defer cm.lk.Unlock() + s := cm.segments.get(p) + s.Lock() + defer s.Unlock() - pi, ok := cm.peers[p] + pi, ok := s.peers[p] if !ok { log.Info("tried to remove tag from untracked peer: ", p) return @@ -246,10 +279,11 @@ func (cm *BasicConnMgr) UntagPeer(p peer.ID, tag string) { // UpsertTag is called to insert/update a peer tag func (cm *BasicConnMgr) UpsertTag(p peer.ID, tag string, upsert func(int) int) { - cm.lk.Lock() - defer cm.lk.Unlock() + s := cm.segments.get(p) + s.Lock() + defer s.Unlock() - pi, ok := cm.peers[p] + pi, ok := s.peers[p] if !ok { log.Info("tried to upsert tag from untracked peer: ", p) return @@ -281,15 +315,12 @@ type CMInfo struct { // GetInfo returns the configuration and status data for this connection manager. func (cm *BasicConnMgr) GetInfo() CMInfo { - cm.lk.Lock() - defer cm.lk.Unlock() - return CMInfo{ HighWater: cm.highWater, LowWater: cm.lowWater, LastTrim: cm.lastTrim, GracePeriod: cm.gracePeriod, - ConnCount: cm.connCount, + ConnCount: int(atomic.LoadInt32(&cm.connCount)), } } @@ -312,29 +343,31 @@ func (nn *cmNotifee) cm() *BasicConnMgr { func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { cm := nn.cm() - cm.lk.Lock() - defer cm.lk.Unlock() + p := c.RemotePeer() + s := cm.segments.get(p) + s.Lock() + defer s.Unlock() - pinfo, ok := cm.peers[c.RemotePeer()] + pinfo, ok := s.peers[p] if !ok { pinfo = &peerInfo{ firstSeen: time.Now(), tags: make(map[string]int), conns: make(map[inet.Conn]time.Time), } - cm.peers[c.RemotePeer()] = pinfo + s.peers[p] = pinfo } _, ok = pinfo.conns[c] if ok { - log.Error("received connected notification for conn we are already tracking: ", c.RemotePeer()) + log.Error("received connected notification for conn we are already tracking: ", p) return } pinfo.conns[c] = time.Now() - cm.connCount++ + connCount := atomic.AddInt32(&cm.connCount, 1) - if cm.connCount > nn.highWater { + if int(connCount) > nn.highWater { go cm.TrimOpenConns(context.Background()) } } @@ -344,26 +377,28 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { func (nn *cmNotifee) Disconnected(n inet.Network, c inet.Conn) { cm := nn.cm() - cm.lk.Lock() - defer cm.lk.Unlock() + p := c.RemotePeer() + s := cm.segments.get(p) + s.Lock() + defer s.Unlock() - cinf, ok := cm.peers[c.RemotePeer()] + cinf, ok := s.peers[p] if !ok { - log.Error("received disconnected notification for peer we are not tracking: ", c.RemotePeer()) + log.Error("received disconnected notification for peer we are not tracking: ", p) return } _, ok = cinf.conns[c] if !ok { - log.Error("received disconnected notification for conn we are not tracking: ", c.RemotePeer()) + log.Error("received disconnected notification for conn we are not tracking: ", p) return } delete(cinf.conns, c) - cm.connCount-- if len(cinf.conns) == 0 { - delete(cm.peers, c.RemotePeer()) + delete(s.peers, p) } + atomic.AddInt32(&cm.connCount, -1) } // Listen is no-op in this implementation. From 30f9525c0355f06e8fd56f4fbb4d2c7f3714a5bc Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 10 May 2019 13:54:00 +0300 Subject: [PATCH 1195/3965] fix tests --- p2p/net/connmgr/connmgr_test.go | 54 ++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index b8ebba7e8d..20c792c025 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -5,6 +5,7 @@ import ( "testing" "time" + detectrace "github.com/ipfs/go-detect-race" inet "github.com/libp2p/go-libp2p-net" peer "github.com/libp2p/go-libp2p-peer" tu "github.com/libp2p/go-testutil" @@ -189,7 +190,7 @@ func TestTagPeerNonExistant(t *testing.T) { id := tu.RandPeerIDFatal(t) cm.TagPeer(id, "test", 1) - if len(cm.peers) != 0 { + if cm.segments.countPeers() != 0 { t.Fatal("expected zero peers") } } @@ -205,20 +206,20 @@ func TestUntagPeer(t *testing.T) { id := tu.RandPeerIDFatal(t) cm.UntagPeer(id, "test") - if len(cm.peers[rp].tags) != 2 { + if len(cm.segments.get(rp).peers[rp].tags) != 2 { t.Fatal("expected tags to be uneffected") } cm.UntagPeer(conn.RemotePeer(), "test") - if len(cm.peers[rp].tags) != 2 { + if len(cm.segments.get(rp).peers[rp].tags) != 2 { t.Fatal("expected tags to be uneffected") } cm.UntagPeer(conn.RemotePeer(), "tag") - if len(cm.peers[rp].tags) != 1 { + if len(cm.segments.get(rp).peers[rp].tags) != 1 { t.Fatal("expected tag to be removed") } - if cm.peers[rp].value != 5 { + if cm.segments.get(rp).peers[rp].value != 5 { t.Fatal("expected aggreagte tag value to be 5") } } @@ -262,7 +263,7 @@ func TestDoubleConnection(t *testing.T) { if cm.connCount != 1 { t.Fatal("unexpected number of connections") } - if cm.peers[conn.RemotePeer()].value != 10 { + if cm.segments.get(conn.RemotePeer()).peers[conn.RemotePeer()].value != 10 { t.Fatal("unexpected peer value") } } @@ -279,7 +280,7 @@ func TestDisconnected(t *testing.T) { if cm.connCount != 1 { t.Fatal("unexpected number of connections") } - if cm.peers[conn.RemotePeer()].value != 10 { + if cm.segments.get(conn.RemotePeer()).peers[conn.RemotePeer()].value != 10 { t.Fatal("unexpected peer value") } @@ -287,7 +288,7 @@ func TestDisconnected(t *testing.T) { if cm.connCount != 1 { t.Fatal("unexpected number of connections") } - if cm.peers[conn.RemotePeer()].value != 10 { + if cm.segments.get(conn.RemotePeer()).peers[conn.RemotePeer()].value != 10 { t.Fatal("unexpected peer value") } @@ -295,13 +296,17 @@ func TestDisconnected(t *testing.T) { if cm.connCount != 0 { t.Fatal("unexpected number of connections") } - if len(cm.peers) != 0 { + if cm.segments.countPeers() != 0 { t.Fatal("unexpected number of peers") } } // see https://github.com/libp2p/go-libp2p-connmgr/issues/23 func TestQuickBurstRespectsSilencePeriod(t *testing.T) { + if detectrace.WithRace() { + t.Skip("race detector is unhappy with this test") + } + cm := NewConnManager(10, 20, 0) not := cm.Notifee() @@ -317,9 +322,6 @@ func TestQuickBurstRespectsSilencePeriod(t *testing.T) { // wait for a few seconds time.Sleep(time.Second * 3) - cm.lk.Lock() // pacify the race detector - defer cm.lk.Unlock() - // only the first trim is allowed in; make sure we close at most 20 connections, not all of them. var closed int for _, c := range conns { @@ -330,12 +332,16 @@ func TestQuickBurstRespectsSilencePeriod(t *testing.T) { if closed > 20 { t.Fatalf("should have closed at most 20 connections, closed: %d", closed) } - if total := closed + cm.connCount; total != 30 { + if total := closed + int(cm.connCount); total != 30 { t.Fatalf("expected closed connections + open conn count to equal 30, value: %d", total) } } func TestPeerProtectionSingleTag(t *testing.T) { + if detectrace.WithRace() { + t.Skip("race detector is unhappy with this test") + } + SilencePeriod = 0 cm := NewConnManager(19, 20, 0) SilencePeriod = 10 * time.Second @@ -386,9 +392,6 @@ func TestPeerProtectionSingleTag(t *testing.T) { // the pruning happens in the background -- this timing condition is not good. time.Sleep(1 * time.Second) - cm.lk.Lock() // pacify the race detector - defer cm.lk.Unlock() - if !protected[0].(*tconn).closed { t.Error("unprotected connection was kept open by connection manager") } @@ -400,6 +403,10 @@ func TestPeerProtectionSingleTag(t *testing.T) { } func TestPeerProtectionMultipleTags(t *testing.T) { + if detectrace.WithRace() { + t.Skip("race detector is unhappy with this test") + } + SilencePeriod = 0 cm := NewConnManager(19, 20, 0) SilencePeriod = 10 * time.Second @@ -476,9 +483,6 @@ func TestPeerProtectionMultipleTags(t *testing.T) { // the pruning happens in the background -- this timing condition is not good. time.Sleep(1 * time.Second) - cm.lk.Lock() // pacify the race detector - defer cm.lk.Unlock() - if !protected[0].(*tconn).closed { t.Error("unprotected connection was kept open by connection manager") } @@ -530,26 +534,26 @@ func TestUpsertTag(t *testing.T) { rp := conn.RemotePeer() cm.UpsertTag(rp, "tag", func(v int) int { return v + 1 }) - if len(cm.peers[rp].tags) != 1 { + if len(cm.segments.get(rp).peers[rp].tags) != 1 { t.Fatal("expected a tag") } - if cm.peers[rp].value != 1 { + if cm.segments.get(rp).peers[rp].value != 1 { t.Fatal("expected a tag value of 1") } cm.UpsertTag(rp, "tag", func(v int) int { return v + 1 }) - if len(cm.peers[rp].tags) != 1 { + if len(cm.segments.get(rp).peers[rp].tags) != 1 { t.Fatal("expected a tag") } - if cm.peers[rp].value != 2 { + if cm.segments.get(rp).peers[rp].value != 2 { t.Fatal("expected a tag value of 2") } cm.UpsertTag(rp, "tag", func(v int) int { return v - 1 }) - if len(cm.peers[rp].tags) != 1 { + if len(cm.segments.get(rp).peers[rp].tags) != 1 { t.Fatal("expected a tag") } - if cm.peers[rp].value != 1 { + if cm.segments.get(rp).peers[rp].value != 1 { t.Fatal("expected a tag value of 1") } } From 96ea7fc0d2e097c5fe324935abfc22bad4299c5c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 10 May 2019 16:16:32 +0100 Subject: [PATCH 1196/3965] add benchmark for lock contention (possibly inaccurate). --- p2p/net/connmgr/bench_test.go | 51 +++++++++++++++++++++++++++++++++ p2p/net/connmgr/connmgr_test.go | 2 +- 2 files changed, 52 insertions(+), 1 deletion(-) create mode 100644 p2p/net/connmgr/bench_test.go diff --git a/p2p/net/connmgr/bench_test.go b/p2p/net/connmgr/bench_test.go new file mode 100644 index 0000000000..91124d0dd8 --- /dev/null +++ b/p2p/net/connmgr/bench_test.go @@ -0,0 +1,51 @@ +package connmgr + +import ( + "math/rand" + "sync" + "testing" + + inet "github.com/libp2p/go-libp2p-net" +) + +func randomConns(tb testing.TB) (c [5000]inet.Conn) { + for i, _ := range c { + c[i] = randConn(tb, nil) + } + return c +} + +func BenchmarkLockContention(b *testing.B) { + conns := randomConns(b) + cm := NewConnManager(1000, 1000, 0) + not := cm.Notifee() + + kill := make(chan struct{}) + var wg sync.WaitGroup + + for i := 0; i < 16; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for { + select { + case <-kill: + return + default: + _ = cm.GetTagInfo(conns[rand.Intn(3000)].RemotePeer()) + } + } + }() + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + rc := conns[rand.Intn(3000)] + not.Connected(nil, rc) + cm.TagPeer(rc.RemotePeer(), "tag", 100) + cm.UntagPeer(rc.RemotePeer(), "tag") + not.Disconnected(nil, rc) + } + close(kill) + wg.Wait() +} diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index 20c792c025..ba61fc30de 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -40,7 +40,7 @@ func (c *tconn) RemoteMultiaddr() ma.Multiaddr { return addr } -func randConn(t *testing.T, discNotify func(inet.Network, inet.Conn)) inet.Conn { +func randConn(t testing.TB, discNotify func(inet.Network, inet.Conn)) inet.Conn { pid := tu.RandPeerIDFatal(t) return &tconn{peer: pid, disconnectNotify: discNotify} } From c7917a3b24797a1f8302dafba256f8b2bcf7e646 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 10 May 2019 20:47:13 +0300 Subject: [PATCH 1197/3965] fix bench_test to use TagPeer in parallel goroutines --- p2p/net/connmgr/bench_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/connmgr/bench_test.go b/p2p/net/connmgr/bench_test.go index 91124d0dd8..aac11bb994 100644 --- a/p2p/net/connmgr/bench_test.go +++ b/p2p/net/connmgr/bench_test.go @@ -32,7 +32,7 @@ func BenchmarkLockContention(b *testing.B) { case <-kill: return default: - _ = cm.GetTagInfo(conns[rand.Intn(3000)].RemotePeer()) + cm.TagPeer(conns[rand.Intn(len(conns))].RemotePeer(), "another-tag", 1) } } }() @@ -40,7 +40,7 @@ func BenchmarkLockContention(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { - rc := conns[rand.Intn(3000)] + rc := conns[rand.Intn(len(conns))] not.Connected(nil, rc) cm.TagPeer(rc.RemotePeer(), "tag", 100) cm.UntagPeer(rc.RemotePeer(), "tag") From fb404b5c2af71a3bf5ab90313290a637cea60cdc Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 11 May 2019 13:41:27 +0300 Subject: [PATCH 1198/3965] gomod: update go-mplex --- go.mod | 1 + go.sum | 2 ++ 2 files changed, 3 insertions(+) diff --git a/go.mod b/go.mod index c4f65f4bf1..3046a45b65 100644 --- a/go.mod +++ b/go.mod @@ -32,6 +32,7 @@ require ( github.com/libp2p/go-libp2p-transport v0.0.4 github.com/libp2p/go-libp2p-transport-upgrader v0.0.2 github.com/libp2p/go-maddr-filter v0.0.1 + github.com/libp2p/go-mplex v0.0.2 // indirect github.com/libp2p/go-stream-muxer v0.0.1 github.com/libp2p/go-tcp-transport v0.0.2 github.com/libp2p/go-testutil v0.0.1 diff --git a/go.sum b/go.sum index a109b2cd7e..88c17b3712 100644 --- a/go.sum +++ b/go.sum @@ -164,6 +164,8 @@ github.com/libp2p/go-maddr-filter v0.0.1 h1:apvYTg0aIxxQyBX+XHKOR+0+lYhGs1Yv+JmT github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-mplex v0.0.1 h1:dn2XGSrUxLtz3/8u85bGrwhUEKPX8MOF3lpmcWBZCWc= github.com/libp2p/go-mplex v0.0.1/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= +github.com/libp2p/go-mplex v0.0.2 h1:tNHgiEvcAXNASiX+SOYO32dkI57rFUHdWWLSmsoO9QQ= +github.com/libp2p/go-mplex v0.0.2/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= github.com/libp2p/go-msgio v0.0.1 h1:znj97n5FtXGCLDwe9x8jpHmY770SW4WStBGcCDh6GJw= github.com/libp2p/go-msgio v0.0.1/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-msgio v0.0.2 h1:ivPvEKHxmVkTClHzg6RXTYHqaJQ0V9cDbq+6lKb3UV0= From 68c9087d5515cf612391adeb1673e7a7528053e9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 12 May 2019 14:01:38 +0300 Subject: [PATCH 1199/3965] fix some issues in connection trimming - fix potential concurrent modification of connection map from concurrent connects/disconnects by locking the segment - don't spawn goroutines on every connect during trims --- p2p/net/connmgr/connmgr.go | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 3010e30579..2b159b5581 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -145,12 +145,18 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { // skip this attempt to trim as the last one just took place. return } + + // mark to stop spawning goroutines in every connection while trimming + cm.lastTrim = time.Now() + defer log.EventBegin(ctx, "connCleanup").Done() for _, c := range cm.getConnsToClose(ctx) { log.Info("closing conn: ", c.RemotePeer()) log.Event(ctx, "closeConn", c.RemotePeer()) c.Close() } + + // we just finished, set the trim time cm.lastTrim = time.Now() } @@ -168,7 +174,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { return nil } - var candidates []*peerInfo + candidates := make([]*peerInfo, 0, npeers) cm.plk.RLock() for _, s := range cm.segments { s.Lock() @@ -201,9 +207,13 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { continue } + // lock this to protect from concurrent modifications from connect/disconnect events + s := cm.segments.get(inf.id) + s.Lock() for c := range inf.conns { selected = append(selected, c) } + s.Unlock() target-- if target == 0 { @@ -351,6 +361,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { pinfo, ok := s.peers[p] if !ok { pinfo = &peerInfo{ + id: p, firstSeen: time.Now(), tags: make(map[string]int), conns: make(map[inet.Conn]time.Time), @@ -367,7 +378,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { pinfo.conns[c] = time.Now() connCount := atomic.AddInt32(&cm.connCount, 1) - if int(connCount) > nn.highWater { + if int(connCount) > nn.highWater && time.Since(cm.lastTrim) > cm.silencePeriod { go cm.TrimOpenConns(context.Background()) } } From a696dc0095ba91b0aeb3b69af703e674048e08ff Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 12 May 2019 14:24:10 +0300 Subject: [PATCH 1200/3965] revert racey last trim check --- p2p/net/connmgr/connmgr.go | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 2b159b5581..fc51b9d248 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -146,9 +146,6 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { return } - // mark to stop spawning goroutines in every connection while trimming - cm.lastTrim = time.Now() - defer log.EventBegin(ctx, "connCleanup").Done() for _, c := range cm.getConnsToClose(ctx) { log.Info("closing conn: ", c.RemotePeer()) @@ -156,7 +153,6 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { c.Close() } - // we just finished, set the trim time cm.lastTrim = time.Now() } @@ -378,7 +374,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { pinfo.conns[c] = time.Now() connCount := atomic.AddInt32(&cm.connCount, 1) - if int(connCount) > nn.highWater && time.Since(cm.lastTrim) > cm.silencePeriod { + if int(connCount) > nn.highWater { go cm.TrimOpenConns(context.Background()) } } From 0ad5be224a0917637b9f57242f3676736f14373d Mon Sep 17 00:00:00 2001 From: vyzo Date: Sun, 12 May 2019 15:41:34 +0300 Subject: [PATCH 1201/3965] fix peer comparison to low water mark otherwise the target can be 0, which makes it negative on decrement, and it will end up dropping all connections. --- p2p/net/connmgr/connmgr.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index fc51b9d248..b860cc7108 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -165,7 +165,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { } now := time.Now() npeers := cm.segments.countPeers() - if npeers < cm.lowWater { + if npeers <= cm.lowWater { log.Info("open connection count below limit") return nil } From a5bbace17f8720827fa67dc9b44aa5141ec049b5 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 16 May 2019 18:30:16 +0300 Subject: [PATCH 1202/3965] allow user-spceified TTL in routing advertisements --- p2p/discovery/routing/routing.go | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index c618ba7c63..5d85b90627 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -21,6 +21,19 @@ func NewRoutingDiscovery(router routing.ContentRouting) *RoutingDiscovery { } func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Option) (time.Duration, error) { + var options Options + err := options.Apply(opts...) + if err != nil { + return 0, err + } + + ttl := options.Ttl + if ttl == 0 || ttl > 3*time.Hour { + // the DHT provider record validity is 24hrs, but it is recommnded to republish at least every 6hrs + // we go one step further and republish every 3hrs + ttl = 3 * time.Hour + } + cid, err := nsToCid(ns) if err != nil { return 0, err @@ -37,9 +50,7 @@ func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Opt return 0, err } - // the DHT provider record validity is 24hrs, but it is recommnded to republish at least every 6hrs - // we go one step further and republish every 3hrs - return 3 * time.Hour, nil + return ttl, nil } func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Option) (<-chan pstore.PeerInfo, error) { From c36fd0fd5b3c8e008bb1df19b088e5b86e7af55a Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 16 May 2019 18:30:53 +0300 Subject: [PATCH 1203/3965] add options to utility interface --- p2p/discovery/util/util.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/discovery/util/util.go b/p2p/discovery/util/util.go index be43a4b15c..2a267ef39e 100644 --- a/p2p/discovery/util/util.go +++ b/p2p/discovery/util/util.go @@ -11,10 +11,10 @@ import ( var log = logging.Logger("discovery") // FindPeers is a utility function that synchonously collects peers from a Discoverer -func FindPeers(ctx context.Context, d Discoverer, ns string, limit int) ([]pstore.PeerInfo, error) { - res := make([]pstore.PeerInfo, 0, limit) +func FindPeers(ctx context.Context, d Discoverer, ns string, opts ...Option) ([]pstore.PeerInfo, error) { + var res []pstore.PeerInfo - ch, err := d.FindPeers(ctx, ns, Limit(limit)) + ch, err := d.FindPeers(ctx, ns, opts...) if err != nil { return nil, err } @@ -27,10 +27,10 @@ func FindPeers(ctx context.Context, d Discoverer, ns string, limit int) ([]pstor } // Advertise is a utility function that persistently advertises a service through an Advertiser -func Advertise(ctx context.Context, a Advertiser, ns string) { +func Advertise(ctx context.Context, a Advertiser, ns string, opts ...Option) { go func() { for { - ttl, err := a.Advertise(ctx, ns) + ttl, err := a.Advertise(ctx, ns, opts...) if err != nil { log.Debugf("Error advertising %s: %s", ns, err.Error()) if ctx.Err() != nil { From 383cb2a583b348dd48626c951e4add7a92d1aa38 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 16 May 2019 18:44:07 +0300 Subject: [PATCH 1204/3965] fix test --- p2p/discovery/routing/routing_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/discovery/routing/routing_test.go b/p2p/discovery/routing/routing_test.go index 82cebf31e1..5aff567abe 100644 --- a/p2p/discovery/routing/routing_test.go +++ b/p2p/discovery/routing/routing_test.go @@ -89,7 +89,7 @@ func TestRoutingDiscovery(t *testing.T) { t.Fatal(err) } - pis, err := FindPeers(ctx, d2, "/test", 20) + pis, err := FindPeers(ctx, d2, "/test", Limit(20)) if err != nil { t.Fatal(err) } From 157f460ee3b31beb8e8d7be92cf3ef7a9601c3ec Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 12:33:33 +0300 Subject: [PATCH 1205/3965] refactor protocol functionality into ProtoBook --- p2p/host/peerstore/interface.go | 14 ++- p2p/host/peerstore/peerstore.go | 112 +------------------- p2p/host/peerstore/pstoremem/addr_book.go | 14 +-- p2p/host/peerstore/pstoremem/peerstore.go | 1 + p2p/host/peerstore/pstoremem/protobook.go | 123 ++++++++++++++++++++++ 5 files changed, 144 insertions(+), 120 deletions(-) create mode 100644 p2p/host/peerstore/pstoremem/protobook.go diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index c669ba94d3..e3d988cee7 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -56,17 +56,13 @@ type Peerstore interface { KeyBook PeerMetadata Metrics + ProtoBook // PeerInfo returns a peer.PeerInfo struct for given peer.ID. // This is a small slice of the information Peerstore has on // that peer, useful to other services. PeerInfo(peer.ID) PeerInfo - GetProtocols(peer.ID) ([]string, error) - AddProtocols(peer.ID, ...string) error - SetProtocols(peer.ID, ...string) error - SupportsProtocols(peer.ID, ...string) ([]string, error) - // Peers returns all of the peer IDs stored across all inner stores. Peers() peer.IDSlice } @@ -142,3 +138,11 @@ type KeyBook interface { // PeersWithKeys returns all the peer IDs stored in the KeyBook PeersWithKeys() peer.IDSlice } + +// ProtoBook tracks the protocols supported by peers +type ProtoBook interface { + GetProtocols(peer.ID) ([]string, error) + AddProtocols(peer.ID, ...string) error + SetProtocols(peer.ID, ...string) error + SupportsProtocols(peer.ID, ...string) ([]string, error) +} diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index fde9cd4c6d..ea29a17fbb 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -3,74 +3,30 @@ package peerstore import ( "fmt" "io" - "sync" peer "github.com/libp2p/go-libp2p-peer" ) var _ Peerstore = (*peerstore)(nil) -const maxInternedProtocols = 64 -const maxInternedProtocolSize = 128 - -type segment struct { - lk sync.RWMutex - interned map[string]string - protocols map[peer.ID]map[string]struct{} -} - -type segments [256]*segment - -func (s *segments) get(id peer.ID) *segment { - b := []byte(id) - return s[b[len(b)-1]] -} - -func (s *segment) internProtocol(proto string) string { - if len(proto) > maxInternedProtocolSize { - return proto - } - - if interned, ok := s.interned[proto]; ok { - return interned - } - - if len(s.interned) >= maxInternedProtocols { - s.interned = make(map[string]string, maxInternedProtocols) - } - - s.interned[proto] = proto - return proto -} - type peerstore struct { Metrics KeyBook AddrBook + ProtoBook PeerMetadata - - // segments for protocol information - segments segments } // NewPeerstore creates a data structure that stores peer data, backed by the // supplied implementations of KeyBook, AddrBook and PeerMetadata. -func NewPeerstore(kb KeyBook, ab AddrBook, md PeerMetadata) Peerstore { +func NewPeerstore(kb KeyBook, ab AddrBook, pb ProtoBook, md PeerMetadata) Peerstore { return &peerstore{ KeyBook: kb, AddrBook: ab, + ProtoBook: pb, PeerMetadata: md, Metrics: NewMetrics(), - segments: func() (ret segments) { - for i := range ret { - ret[i] = &segment{ - interned: make(map[string]string), - protocols: make(map[peer.ID]map[string]struct{}), - } - } - return ret - }(), } } @@ -86,6 +42,7 @@ func (ps *peerstore) Close() (err error) { weakClose("keybook", ps.KeyBook) weakClose("addressbook", ps.AddrBook) + weakClose("protobook", ps.ProtoBook) weakClose("peermetadata", ps.PeerMetadata) if len(errs) > 0 { @@ -117,67 +74,6 @@ func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { } } -func (ps *peerstore) SetProtocols(p peer.ID, protos ...string) error { - s := ps.segments.get(p) - s.lk.Lock() - defer s.lk.Unlock() - - newprotos := make(map[string]struct{}, len(protos)) - for _, proto := range protos { - newprotos[s.internProtocol(proto)] = struct{}{} - } - - s.protocols[p] = newprotos - - return nil -} - -func (ps *peerstore) AddProtocols(p peer.ID, protos ...string) error { - s := ps.segments.get(p) - s.lk.Lock() - defer s.lk.Unlock() - - protomap, ok := s.protocols[p] - if !ok { - protomap = make(map[string]struct{}) - s.protocols[p] = protomap - } - - for _, proto := range protos { - protomap[s.internProtocol(proto)] = struct{}{} - } - - return nil -} - -func (ps *peerstore) GetProtocols(p peer.ID) ([]string, error) { - s := ps.segments.get(p) - s.lk.RLock() - defer s.lk.RUnlock() - - out := make([]string, 0, len(s.protocols)) - for k := range s.protocols[p] { - out = append(out, k) - } - - return out, nil -} - -func (ps *peerstore) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { - s := ps.segments.get(p) - s.lk.RLock() - defer s.lk.RUnlock() - - out := make([]string, 0, len(protos)) - for _, proto := range protos { - if _, ok := s.protocols[p][proto]; ok { - out = append(out, proto) - } - } - - return out, nil -} - func PeerInfos(ps Peerstore, peers peer.IDSlice) []PeerInfo { pi := make([]PeerInfo, len(peers)) for i, p := range peers { diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index cd7a920cf1..6baee9e69d 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -27,18 +27,18 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { return t.After(e.Expires) } -type segments [256]*segment +type addrSegments [256]*addrSegment -type segment struct { +type addrSegment struct { size uint32 lk sync.RWMutex addrs map[peer.ID]map[string]*expiringAddr } -func (s *segments) get(id peer.ID) *segment { +func (s *addrSegments) get(id peer.ID) *addrSegment { b := []byte(id) - return s[b[len(b)-1]%byte(255)] + return s[b[len(b)-1]] } // memoryAddrBook manages addresses. @@ -47,7 +47,7 @@ type memoryAddrBook struct { // Use pointers to save memory. Maps always leave some fraction of their // space unused. storing the *values* directly in the map will // drastically increase the space waste. In our case, by 6x. - segments segments + segments addrSegments ctx context.Context cancel func() @@ -61,9 +61,9 @@ func NewAddrBook() pstore.AddrBook { ctx, cancel := context.WithCancel(context.Background()) ab := &memoryAddrBook{ - segments: func() (ret segments) { + segments: func() (ret addrSegments) { for i, _ := range ret { - ret[i] = &segment{addrs: make(map[peer.ID]map[string]*expiringAddr)} + ret[i] = &addrSegment{addrs: make(map[peer.ID]map[string]*expiringAddr)} } return ret }(), diff --git a/p2p/host/peerstore/pstoremem/peerstore.go b/p2p/host/peerstore/pstoremem/peerstore.go index 7d87313de1..c7cbd672f2 100644 --- a/p2p/host/peerstore/pstoremem/peerstore.go +++ b/p2p/host/peerstore/pstoremem/peerstore.go @@ -7,5 +7,6 @@ func NewPeerstore() pstore.Peerstore { return pstore.NewPeerstore( NewKeyBook(), NewAddrBook(), + NewProtoBook(), NewPeerMetadata()) } diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go new file mode 100644 index 0000000000..9074696c11 --- /dev/null +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -0,0 +1,123 @@ +package pstoremem + +import ( + "sync" + + peer "github.com/libp2p/go-libp2p-peer" + + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +const maxInternedProtocols = 64 +const maxInternedProtocolSize = 128 + +type protoSegment struct { + lk sync.RWMutex + interned map[string]string + protocols map[peer.ID]map[string]struct{} +} + +type protoSegments [256]*protoSegment + +func (s *protoSegments) get(id peer.ID) *protoSegment { + b := []byte(id) + return s[b[len(b)-1]] +} + +func (s *protoSegment) internProtocol(proto string) string { + if len(proto) > maxInternedProtocolSize { + return proto + } + + if interned, ok := s.interned[proto]; ok { + return interned + } + + if len(s.interned) >= maxInternedProtocols { + s.interned = make(map[string]string, maxInternedProtocols) + } + + s.interned[proto] = proto + return proto +} + +type memoryProtoBook struct { + segments protoSegments +} + +var _ pstore.ProtoBook = (*memoryProtoBook)(nil) + +func NewProtoBook() pstore.ProtoBook { + return &memoryProtoBook{ + segments: func() (ret protoSegments) { + for i := range ret { + ret[i] = &protoSegment{ + interned: make(map[string]string), + protocols: make(map[peer.ID]map[string]struct{}), + } + } + return ret + }(), + } +} + +func (pb *memoryProtoBook) SetProtocols(p peer.ID, protos ...string) error { + s := pb.segments.get(p) + s.lk.Lock() + defer s.lk.Unlock() + + newprotos := make(map[string]struct{}, len(protos)) + for _, proto := range protos { + newprotos[s.internProtocol(proto)] = struct{}{} + } + + s.protocols[p] = newprotos + + return nil +} + +func (pb *memoryProtoBook) AddProtocols(p peer.ID, protos ...string) error { + s := pb.segments.get(p) + s.lk.Lock() + defer s.lk.Unlock() + + protomap, ok := s.protocols[p] + if !ok { + protomap = make(map[string]struct{}) + s.protocols[p] = protomap + } + + for _, proto := range protos { + protomap[s.internProtocol(proto)] = struct{}{} + } + + return nil +} + +func (pb *memoryProtoBook) GetProtocols(p peer.ID) ([]string, error) { + s := pb.segments.get(p) + s.lk.RLock() + defer s.lk.RUnlock() + + out := make([]string, 0, len(s.protocols)) + for k := range s.protocols[p] { + out = append(out, k) + } + + return out, nil +} + +func (pb *memoryProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { + s := pb.segments.get(p) + s.lk.RLock() + defer s.lk.RUnlock() + + out := make([]string, 0, len(protos)) + for _, proto := range protos { + if _, ok := s.protocols[p][proto]; ok { + out = append(out, proto) + } + } + + return out, nil +} From bac273480312b3a73df69d98ad015e38647504eb Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 12:37:46 +0300 Subject: [PATCH 1206/3965] embed lock in segment --- p2p/host/peerstore/pstoremem/addr_book.go | 43 +++++++++++------------ p2p/host/peerstore/pstoremem/protobook.go | 18 +++++----- 2 files changed, 30 insertions(+), 31 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 6baee9e69d..8ea785d5f7 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -30,10 +30,13 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { type addrSegments [256]*addrSegment type addrSegment struct { - size uint32 + sync.RWMutex - lk sync.RWMutex + // Use pointers to save memory. Maps always leave some fraction of their + // space unused. storing the *values* directly in the map will + // drastically increase the space waste. In our case, by 6x. addrs map[peer.ID]map[string]*expiringAddr + size uint32 } func (s *addrSegments) get(id peer.ID) *addrSegment { @@ -43,10 +46,6 @@ func (s *addrSegments) get(id peer.ID) *addrSegment { // memoryAddrBook manages addresses. type memoryAddrBook struct { - addrmu sync.RWMutex - // Use pointers to save memory. Maps always leave some fraction of their - // space unused. storing the *values* directly in the map will - // drastically increase the space waste. In our case, by 6x. segments addrSegments ctx context.Context @@ -101,7 +100,7 @@ func (mab *memoryAddrBook) Close() error { func (mab *memoryAddrBook) gc() { now := time.Now() for _, s := range mab.segments { - s.lk.Lock() + s.Lock() for p, amap := range s.addrs { for k, addr := range amap { if addr.ExpiredBy(now) { @@ -113,7 +112,7 @@ func (mab *memoryAddrBook) gc() { } } atomic.StoreUint32(&s.size, uint32(len(s.addrs))) - s.lk.Unlock() + s.Unlock() } } @@ -126,11 +125,11 @@ func (mab *memoryAddrBook) PeersWithAddrs() peer.IDSlice { pids := make(peer.IDSlice, 0, length) for _, s := range mab.segments { - s.lk.RLock() + s.RLock() for pid, _ := range s.addrs { pids = append(pids, pid) } - s.lk.RUnlock() + s.RUnlock() } return pids } @@ -150,8 +149,8 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du } s := mab.segments.get(p) - s.lk.Lock() - defer s.lk.Unlock() + s.Lock() + defer s.Unlock() // update the segment size defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) @@ -186,8 +185,8 @@ func (mab *memoryAddrBook) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Durati // This is used when we receive the best estimate of the validity of an address. func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { s := mab.segments.get(p) - s.lk.Lock() - defer s.lk.Unlock() + s.Lock() + defer s.Unlock() // update the segment size defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) @@ -220,8 +219,8 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du // the given oldTTL to have the given newTTL. func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { s := mab.segments.get(p) - s.lk.Lock() - defer s.lk.Unlock() + s.Lock() + defer s.Unlock() // update the segment size defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) @@ -244,8 +243,8 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t // Addresses returns all known (and valid) addresses for a given func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { s := mab.segments.get(p) - s.lk.RLock() - defer s.lk.RUnlock() + s.RLock() + defer s.RUnlock() amap, found := s.addrs[p] if !found { @@ -266,8 +265,8 @@ func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { // ClearAddrs removes all previously stored addresses func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { s := mab.segments.get(p) - s.lk.Lock() - defer s.lk.Unlock() + s.Lock() + defer s.Unlock() // update the segment size defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) @@ -279,8 +278,8 @@ func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { // given peer ID will be published. func (mab *memoryAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { s := mab.segments.get(p) - s.lk.RLock() - defer s.lk.RUnlock() + s.RLock() + defer s.RUnlock() baseaddrslice := s.addrs[p] initial := make([]ma.Multiaddr, 0, len(baseaddrslice)) diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index 9074696c11..e9ca4d77dc 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -12,7 +12,7 @@ const maxInternedProtocols = 64 const maxInternedProtocolSize = 128 type protoSegment struct { - lk sync.RWMutex + sync.RWMutex interned map[string]string protocols map[peer.ID]map[string]struct{} } @@ -63,8 +63,8 @@ func NewProtoBook() pstore.ProtoBook { func (pb *memoryProtoBook) SetProtocols(p peer.ID, protos ...string) error { s := pb.segments.get(p) - s.lk.Lock() - defer s.lk.Unlock() + s.Lock() + defer s.Unlock() newprotos := make(map[string]struct{}, len(protos)) for _, proto := range protos { @@ -78,8 +78,8 @@ func (pb *memoryProtoBook) SetProtocols(p peer.ID, protos ...string) error { func (pb *memoryProtoBook) AddProtocols(p peer.ID, protos ...string) error { s := pb.segments.get(p) - s.lk.Lock() - defer s.lk.Unlock() + s.Lock() + defer s.Unlock() protomap, ok := s.protocols[p] if !ok { @@ -96,8 +96,8 @@ func (pb *memoryProtoBook) AddProtocols(p peer.ID, protos ...string) error { func (pb *memoryProtoBook) GetProtocols(p peer.ID) ([]string, error) { s := pb.segments.get(p) - s.lk.RLock() - defer s.lk.RUnlock() + s.RLock() + defer s.RUnlock() out := make([]string, 0, len(s.protocols)) for k := range s.protocols[p] { @@ -109,8 +109,8 @@ func (pb *memoryProtoBook) GetProtocols(p peer.ID) ([]string, error) { func (pb *memoryProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { s := pb.segments.get(p) - s.lk.RLock() - defer s.lk.RUnlock() + s.RLock() + defer s.RUnlock() out := make([]string, 0, len(protos)) for _, proto := range protos { From e30651c40b620741d36905434bfe10f46e51f16a Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 12:47:07 +0300 Subject: [PATCH 1207/3965] global interned protocol table --- p2p/host/peerstore/pstoremem/protobook.go | 66 +++++++++++++++-------- 1 file changed, 43 insertions(+), 23 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index e9ca4d77dc..e4017012b7 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -8,12 +8,13 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" ) -const maxInternedProtocols = 64 -const maxInternedProtocolSize = 128 +const ( + maxInternedProtocols = 512 + maxInternedProtocolSize = 256 +) type protoSegment struct { sync.RWMutex - interned map[string]string protocols map[peer.ID]map[string]struct{} } @@ -24,35 +25,21 @@ func (s *protoSegments) get(id peer.ID) *protoSegment { return s[b[len(b)-1]] } -func (s *protoSegment) internProtocol(proto string) string { - if len(proto) > maxInternedProtocolSize { - return proto - } - - if interned, ok := s.interned[proto]; ok { - return interned - } - - if len(s.interned) >= maxInternedProtocols { - s.interned = make(map[string]string, maxInternedProtocols) - } - - s.interned[proto] = proto - return proto -} - type memoryProtoBook struct { segments protoSegments + + lk sync.RWMutex + interned map[string]string } var _ pstore.ProtoBook = (*memoryProtoBook)(nil) func NewProtoBook() pstore.ProtoBook { return &memoryProtoBook{ + interned: make(map[string]string, maxInternedProtocols), segments: func() (ret protoSegments) { for i := range ret { ret[i] = &protoSegment{ - interned: make(map[string]string), protocols: make(map[peer.ID]map[string]struct{}), } } @@ -61,6 +48,39 @@ func NewProtoBook() pstore.ProtoBook { } } +func (pb *memoryProtoBook) internProtocol(proto string) string { + if len(proto) > maxInternedProtocolSize { + return proto + } + + // check if it is interned with the read lock + pb.lk.RLock() + interned, ok := pb.interned[proto] + pb.lk.RUnlock() + + if ok { + return interned + } + + // intern with the write lock + pb.lk.Lock() + defer pb.lk.Unlock() + + // check again in case it got interned in between locks + interned, ok = pb.interned[proto] + if ok { + return interned + } + + // if we've filled the table, throw it away and start over + if len(pb.interned) >= maxInternedProtocols { + pb.interned = make(map[string]string, maxInternedProtocols) + } + + pb.interned[proto] = proto + return proto +} + func (pb *memoryProtoBook) SetProtocols(p peer.ID, protos ...string) error { s := pb.segments.get(p) s.Lock() @@ -68,7 +88,7 @@ func (pb *memoryProtoBook) SetProtocols(p peer.ID, protos ...string) error { newprotos := make(map[string]struct{}, len(protos)) for _, proto := range protos { - newprotos[s.internProtocol(proto)] = struct{}{} + newprotos[pb.internProtocol(proto)] = struct{}{} } s.protocols[p] = newprotos @@ -88,7 +108,7 @@ func (pb *memoryProtoBook) AddProtocols(p peer.ID, protos ...string) error { } for _, proto := range protos { - protomap[s.internProtocol(proto)] = struct{}{} + protomap[pb.internProtocol(proto)] = struct{}{} } return nil From 12eafa0b903fcc9c7546ebcf79b42c60a0ec10f4 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 13:13:18 +0300 Subject: [PATCH 1208/3965] implement protobook for pstoreds --- p2p/host/peerstore/pstoreds/peerstore.go | 4 +- p2p/host/peerstore/pstoreds/protobook.go | 122 +++++++++++++++++++++++ 2 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 p2p/host/peerstore/pstoreds/protobook.go diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index be7b09e9a8..b4560f7363 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -63,7 +63,9 @@ func NewPeerstore(ctx context.Context, store ds.Batching, opts Options) (pstore. return nil, err } - ps := pstore.NewPeerstore(keyBook, addrBook, peerMetadata) + protoBook := NewProtoBook(peerMetadata) + + ps := pstore.NewPeerstore(keyBook, addrBook, protoBook, peerMetadata) return ps, nil } diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go new file mode 100644 index 0000000000..27184f74fd --- /dev/null +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -0,0 +1,122 @@ +package pstoreds + +import ( + "fmt" + "sync" + + peer "github.com/libp2p/go-libp2p-peer" + + pstore "github.com/libp2p/go-libp2p-peerstore" +) + +type dsProtoBook struct { + lks [256]sync.RWMutex + meta pstore.PeerMetadata +} + +var _ pstore.ProtoBook = (*dsProtoBook)(nil) + +func NewProtoBook(meta pstore.PeerMetadata) pstore.ProtoBook { + return &dsProtoBook{meta: meta} +} + +func (pb *dsProtoBook) Lock(p peer.ID) { + b := []byte(p) + pb.lks[b[len(b)-1]].Lock() +} + +func (pb *dsProtoBook) Unlock(p peer.ID) { + b := []byte(p) + pb.lks[b[len(b)-1]].Unlock() +} + +func (pb *dsProtoBook) RLock(p peer.ID) { + b := []byte(p) + pb.lks[b[len(b)-1]].RLock() +} + +func (pb *dsProtoBook) RUnlock(p peer.ID) { + b := []byte(p) + pb.lks[b[len(b)-1]].RUnlock() +} + +func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { + pb.Lock(p) + defer pb.Unlock(p) + + protomap := make(map[string]struct{}, len(protos)) + for _, proto := range protos { + protomap[proto] = struct{}{} + } + + return pb.meta.Put(p, "protocols", protomap) +} + +func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { + pb.Lock(p) + defer pb.Unlock(p) + + pmap, err := pb.getProtocolMap(p) + if err != nil { + return err + } + + for _, proto := range protos { + pmap[proto] = struct{}{} + } + + return pb.meta.Put(p, "protocols", pmap) +} + +func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { + pb.RLock(p) + defer pb.RUnlock(p) + + pmap, err := pb.getProtocolMap(p) + if err != nil { + return nil, err + } + + res := make([]string, 0, len(pmap)) + for proto := range pmap { + res = append(res, proto) + } + + return res, nil +} + +func (pb *dsProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { + pb.RLock(p) + defer pb.RUnlock(p) + + pmap, err := pb.getProtocolMap(p) + if err != nil { + return nil, err + } + + res := make([]string, 0, len(protos)) + for _, proto := range protos { + if _, ok := pmap[proto]; ok { + res = append(res, proto) + } + } + + return res, nil +} + +func (pb *dsProtoBook) getProtocolMap(p peer.ID) (map[string]struct{}, error) { + iprotomap, err := pb.meta.Get(p, "protocols") + switch err { + default: + return nil, err + case pstore.ErrNotFound: + return make(map[string]struct{}), nil + case nil: + cast, ok := iprotomap.(map[string]struct{}) + if !ok { + return nil, fmt.Errorf("stored protocol set was not a map") + } + + return cast, nil + } +} From e531639ebcacf950ef1a7d8098ac576e0b8a4f23 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 13:14:39 +0300 Subject: [PATCH 1209/3965] cosmetics --- p2p/host/peerstore/pstoremem/addr_book.go | 4 ++-- p2p/host/peerstore/pstoremem/protobook.go | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 8ea785d5f7..2bc80f185e 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -39,8 +39,8 @@ type addrSegment struct { size uint32 } -func (s *addrSegments) get(id peer.ID) *addrSegment { - b := []byte(id) +func (s *addrSegments) get(p peer.ID) *addrSegment { + b := []byte(p) return s[b[len(b)-1]] } diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index e4017012b7..a112d00bbb 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -20,8 +20,8 @@ type protoSegment struct { type protoSegments [256]*protoSegment -func (s *protoSegments) get(id peer.ID) *protoSegment { - b := []byte(id) +func (s *protoSegments) get(p peer.ID) *protoSegment { + b := []byte(p) return s[b[len(b)-1]] } From fc6f373465f915fc5672d4663391ae1c2fdde38e Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 22:04:29 +0300 Subject: [PATCH 1210/3965] gomod: update go-libp2p-discovery --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 3046a45b65..c9a9263fb9 100644 --- a/go.mod +++ b/go.mod @@ -14,7 +14,7 @@ require ( github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.6 github.com/libp2p/go-libp2p-crypto v0.0.2 - github.com/libp2p/go-libp2p-discovery v0.0.2 + github.com/libp2p/go-libp2p-discovery v0.0.3 github.com/libp2p/go-libp2p-host v0.0.3 github.com/libp2p/go-libp2p-interface-connmgr v0.0.4 github.com/libp2p/go-libp2p-interface-pnet v0.0.1 diff --git a/go.sum b/go.sum index 88c17b3712..aeb183f8a9 100644 --- a/go.sum +++ b/go.sum @@ -107,8 +107,8 @@ github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= github.com/libp2p/go-libp2p-crypto v0.0.2/go.mod h1:eETI5OUfBnvARGOHrJz2eWNyTUxEGZnBxMcbUjfIj4I= -github.com/libp2p/go-libp2p-discovery v0.0.2 h1:Rf+20nsFcCnHo4Kxvf8ofAft75+fW+cXy9FonNVyU/g= -github.com/libp2p/go-libp2p-discovery v0.0.2/go.mod h1:ZkkF9xIFRLA1xCc7bstYFkd80gBGK8Fc1JqGoU2i+zI= +github.com/libp2p/go-libp2p-discovery v0.0.3 h1:/41EY87ErU1yQLoPoTfJXV3xomDwx7Ti1Ae66DuDk9U= +github.com/libp2p/go-libp2p-discovery v0.0.3/go.mod h1:ZkkF9xIFRLA1xCc7bstYFkd80gBGK8Fc1JqGoU2i+zI= github.com/libp2p/go-libp2p-host v0.0.1 h1:dnqusU+DheGcdxrE718kG4XgHNuL2n9eEv8Rg5zy8hQ= github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= github.com/libp2p/go-libp2p-host v0.0.3 h1:BB/1Z+4X0rjKP5lbQTmjEjLbDVbrcmLOlA6QDsN5/j4= From 0a221500fd1ba681d52e6548de62ca786b482770 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 22:05:57 +0300 Subject: [PATCH 1211/3965] update use of discovery.FindPeers for new interface --- p2p/host/relay/autorelay.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 11fc436f8c..fbfb849740 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -236,7 +236,7 @@ func (ar *AutoRelay) connect(ctx context.Context, pi pstore.PeerInfo) bool { func (ar *AutoRelay) discoverRelays(ctx context.Context) ([]pstore.PeerInfo, error) { ctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() - return discovery.FindPeers(ctx, ar.discover, RelayRendezvous, 1000) + return discovery.FindPeers(ctx, ar.discover, RelayRendezvous, discovery.Limit(1000)) } func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo) []pstore.PeerInfo { From 70c4f155e75ab1230656b88e3cdaaba9eca90a29 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 22:07:30 +0300 Subject: [PATCH 1212/3965] advertise for relay services every 30min --- p2p/host/relay/relay.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index 2adea8fcc6..7c57a7cd8a 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -11,6 +11,7 @@ import ( var ( // this is purposefully long to require some node stability before advertising as a relay AdvertiseBootDelay = 15 * time.Minute + AdvertiseTTL = 30 * time.Minute ) // Advertise advertises this node as a libp2p relay. @@ -18,7 +19,7 @@ func Advertise(ctx context.Context, advertise discovery.Advertiser) { go func() { select { case <-time.After(AdvertiseBootDelay): - discovery.Advertise(ctx, advertise, RelayRendezvous) + discovery.Advertise(ctx, advertise, RelayRendezvous, discovery.TTL(AdvertiseTTL)) case <-ctx.Done(): } }() From ba7cb93032ed6a54c0dd512b41fd3ca25076c63f Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 23:36:46 +0300 Subject: [PATCH 1213/3965] index the peer ID strings directly to avoid copying the byte slice --- p2p/host/peerstore/pstoreds/protobook.go | 12 ++++-------- p2p/host/peerstore/pstoremem/addr_book.go | 3 +-- p2p/host/peerstore/pstoremem/protobook.go | 3 +-- 3 files changed, 6 insertions(+), 12 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go index 27184f74fd..88e28838fc 100644 --- a/p2p/host/peerstore/pstoreds/protobook.go +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -21,23 +21,19 @@ func NewProtoBook(meta pstore.PeerMetadata) pstore.ProtoBook { } func (pb *dsProtoBook) Lock(p peer.ID) { - b := []byte(p) - pb.lks[b[len(b)-1]].Lock() + pb.lks[byte(p[len(p)-1])].Lock() } func (pb *dsProtoBook) Unlock(p peer.ID) { - b := []byte(p) - pb.lks[b[len(b)-1]].Unlock() + pb.lks[byte(p[len(p)-1])].Unlock() } func (pb *dsProtoBook) RLock(p peer.ID) { - b := []byte(p) - pb.lks[b[len(b)-1]].RLock() + pb.lks[byte(p[len(p)-1])].RLock() } func (pb *dsProtoBook) RUnlock(p peer.ID) { - b := []byte(p) - pb.lks[b[len(b)-1]].RUnlock() + pb.lks[byte(p[len(p)-1])].RUnlock() } func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 2bc80f185e..92782766a0 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -40,8 +40,7 @@ type addrSegment struct { } func (s *addrSegments) get(p peer.ID) *addrSegment { - b := []byte(p) - return s[b[len(b)-1]] + return s[byte(p[len(p)-1])] } // memoryAddrBook manages addresses. diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index a112d00bbb..04cd14591c 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -21,8 +21,7 @@ type protoSegment struct { type protoSegments [256]*protoSegment func (s *protoSegments) get(p peer.ID) *protoSegment { - b := []byte(p) - return s[b[len(b)-1]] + return s[byte(p[len(p)-1])] } type memoryProtoBook struct { From 1a7ee918961af54251f6e035ebac7aac43650d34 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 23:39:21 +0300 Subject: [PATCH 1214/3965] don't track segment sizes, it's a fruitless optimization --- p2p/host/peerstore/pstoremem/addr_book.go | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 92782766a0..a497821d05 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -4,7 +4,6 @@ import ( "context" "sort" "sync" - "sync/atomic" "time" logging "github.com/ipfs/go-log" @@ -36,7 +35,6 @@ type addrSegment struct { // space unused. storing the *values* directly in the map will // drastically increase the space waste. In our case, by 6x. addrs map[peer.ID]map[string]*expiringAddr - size uint32 } func (s *addrSegments) get(p peer.ID) *addrSegment { @@ -110,19 +108,13 @@ func (mab *memoryAddrBook) gc() { delete(s.addrs, p) } } - atomic.StoreUint32(&s.size, uint32(len(s.addrs))) s.Unlock() } } func (mab *memoryAddrBook) PeersWithAddrs() peer.IDSlice { - var length uint32 - for _, s := range mab.segments { - length += atomic.LoadUint32(&s.size) - } - - pids := make(peer.IDSlice, 0, length) + var pids peer.IDSlice for _, s := range mab.segments { s.RLock() for pid, _ := range s.addrs { @@ -151,9 +143,6 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du s.Lock() defer s.Unlock() - // update the segment size - defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) - amap := s.addrs[p] if amap == nil { amap = make(map[string]*expiringAddr, len(addrs)) @@ -187,9 +176,6 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du s.Lock() defer s.Unlock() - // update the segment size - defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) - amap := s.addrs[p] if amap == nil { amap = make(map[string]*expiringAddr, len(addrs)) @@ -221,9 +207,6 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t s.Lock() defer s.Unlock() - // update the segment size - defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) - amap, found := s.addrs[p] if !found { return @@ -267,9 +250,6 @@ func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { s.Lock() defer s.Unlock() - // update the segment size - defer atomic.StoreUint32(&s.size, uint32(len(s.addrs))) - delete(s.addrs, p) } From 0ab596742dea755a310247a535c6957a36599dfd Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 23:41:05 +0300 Subject: [PATCH 1215/3965] make lock methods private --- p2p/host/peerstore/pstoreds/protobook.go | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go index 88e28838fc..02c865a8a9 100644 --- a/p2p/host/peerstore/pstoreds/protobook.go +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -20,25 +20,25 @@ func NewProtoBook(meta pstore.PeerMetadata) pstore.ProtoBook { return &dsProtoBook{meta: meta} } -func (pb *dsProtoBook) Lock(p peer.ID) { +func (pb *dsProtoBook) lock(p peer.ID) { pb.lks[byte(p[len(p)-1])].Lock() } -func (pb *dsProtoBook) Unlock(p peer.ID) { +func (pb *dsProtoBook) unlock(p peer.ID) { pb.lks[byte(p[len(p)-1])].Unlock() } -func (pb *dsProtoBook) RLock(p peer.ID) { +func (pb *dsProtoBook) rlock(p peer.ID) { pb.lks[byte(p[len(p)-1])].RLock() } -func (pb *dsProtoBook) RUnlock(p peer.ID) { +func (pb *dsProtoBook) runlock(p peer.ID) { pb.lks[byte(p[len(p)-1])].RUnlock() } func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { - pb.Lock(p) - defer pb.Unlock(p) + pb.lock(p) + defer pb.unlock(p) protomap := make(map[string]struct{}, len(protos)) for _, proto := range protos { @@ -49,8 +49,8 @@ func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { } func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { - pb.Lock(p) - defer pb.Unlock(p) + pb.lock(p) + defer pb.unlock(p) pmap, err := pb.getProtocolMap(p) if err != nil { @@ -65,8 +65,8 @@ func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { } func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { - pb.RLock(p) - defer pb.RUnlock(p) + pb.rlock(p) + defer pb.runlock(p) pmap, err := pb.getProtocolMap(p) if err != nil { @@ -82,8 +82,8 @@ func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { } func (pb *dsProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { - pb.RLock(p) - defer pb.RUnlock(p) + pb.rlock(p) + defer pb.runlock(p) pmap, err := pb.getProtocolMap(p) if err != nil { From c9ef3d9b9710b25ae14f682d92159db186e76129 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 17 May 2019 23:46:49 +0300 Subject: [PATCH 1216/3965] index string directly to avoid intermediate byte slice --- p2p/net/connmgr/connmgr.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index b860cc7108..af73945c1f 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -50,9 +50,8 @@ type segment struct { type segments [256]*segment -func (s *segments) get(id peer.ID) *segment { - b := []byte(id) - return s[b[len(b)-1]] +func (s *segments) get(p peer.ID) *segment { + return s[byte(p[len(p)-1])] } func (s *segments) countPeers() (count int) { From 72e862833f1b198e4ad1a3c70f9970de68cc9b77 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 13 May 2019 19:54:02 +0300 Subject: [PATCH 1217/3965] implement background trimming --- p2p/net/connmgr/connmgr.go | 39 ++++++++++++++++++++++++++++++++------ 1 file changed, 33 insertions(+), 6 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index af73945c1f..9a3912da5b 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -39,6 +39,9 @@ type BasicConnMgr struct { trimRunningCh chan struct{} lastTrim time.Time silencePeriod time.Duration + + ctx context.Context + cancel func() } var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) @@ -70,13 +73,16 @@ func (s *segments) countPeers() (count int) { // * grace is the amount of time a newly opened connection is given before it becomes // subject to pruning. func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { - return &BasicConnMgr{ + ctx, cancel := context.WithCancel(context.Background()) + cm := &BasicConnMgr{ highWater: hi, lowWater: low, gracePeriod: grace, trimRunningCh: make(chan struct{}, 1), protected: make(map[peer.ID]map[string]struct{}, 16), silencePeriod: SilencePeriod, + ctx: ctx, + cancel: cancel, segments: func() (ret segments) { for i := range ret { ret[i] = &segment{ @@ -86,6 +92,14 @@ func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { return ret }(), } + + go cm.background() + return cm +} + +func (cm *BasicConnMgr) Close() error { + cm.cancel() + return nil } func (cm *BasicConnMgr) Protect(id peer.ID, tag string) { @@ -155,6 +169,23 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { cm.lastTrim = time.Now() } +func (cm *BasicConnMgr) background() { + ticker := time.NewTicker(time.Minute) + defer ticker.Stop() + + for { + select { + case <-ticker.C: + if atomic.LoadInt32(&cm.connCount) > int32(cm.highWater) { + cm.TrimOpenConns(cm.ctx) + } + + case <-cm.ctx.Done(): + return + } + } +} + // getConnsToClose runs the heuristics described in TrimOpenConns and returns the // connections to close. func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { @@ -371,11 +402,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { } pinfo.conns[c] = time.Now() - connCount := atomic.AddInt32(&cm.connCount, 1) - - if int(connCount) > nn.highWater { - go cm.TrimOpenConns(context.Background()) - } + atomic.AddInt32(&cm.connCount, 1) } // Disconnected is called by notifiers to inform that an existing connection has been closed or terminated. From 75cc6185548a259ac3e46c122128c52ee039f548 Mon Sep 17 00:00:00 2001 From: vyzo Date: Mon, 13 May 2019 19:57:23 +0300 Subject: [PATCH 1218/3965] fix tests --- p2p/net/connmgr/connmgr_test.go | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index ba61fc30de..7f4ed10436 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -369,8 +369,7 @@ func TestPeerProtectionSingleTag(t *testing.T) { // add one more connection, sending the connection manager overboard. not.Connected(nil, randConn(t, not.Disconnected)) - // the pruning happens in the background -- this timing condition is not good. - time.Sleep(1 * time.Second) + cm.TrimOpenConns(context.Background()) for _, c := range protected { if c.(*tconn).closed { @@ -389,8 +388,7 @@ func TestPeerProtectionSingleTag(t *testing.T) { cm.TagPeer(rc.RemotePeer(), "test", 20) } - // the pruning happens in the background -- this timing condition is not good. - time.Sleep(1 * time.Second) + cm.TrimOpenConns(context.Background()) if !protected[0].(*tconn).closed { t.Error("unprotected connection was kept open by connection manager") @@ -435,8 +433,7 @@ func TestPeerProtectionMultipleTags(t *testing.T) { // add one more connection, sending the connection manager overboard. not.Connected(nil, randConn(t, not.Disconnected)) - // the pruning happens in the background -- this timing condition is not good. - time.Sleep(1 * time.Second) + cm.TrimOpenConns(context.Background()) for _, c := range protected { if c.(*tconn).closed { @@ -459,8 +456,7 @@ func TestPeerProtectionMultipleTags(t *testing.T) { cm.TagPeer(rc.RemotePeer(), "test", 20) } - // the pruning happens in the background -- this timing condition is not good. - time.Sleep(1 * time.Second) + cm.TrimOpenConns(context.Background()) // connections should still remain open, as they were protected. for _, c := range protected[0:] { @@ -480,8 +476,7 @@ func TestPeerProtectionMultipleTags(t *testing.T) { cm.TagPeer(rc.RemotePeer(), "test", 20) } - // the pruning happens in the background -- this timing condition is not good. - time.Sleep(1 * time.Second) + cm.TrimOpenConns(context.Background()) if !protected[0].(*tconn).closed { t.Error("unprotected connection was kept open by connection manager") From 234c13d26a11e88549c169ce756eef90d5113bf8 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 14 May 2019 15:21:19 +0300 Subject: [PATCH 1219/3965] fix issue #44 --- p2p/net/connmgr/connmgr.go | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 9a3912da5b..e61ad15ca9 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -194,12 +194,13 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { return nil } now := time.Now() - npeers := cm.segments.countPeers() - if npeers <= cm.lowWater { + nconns := int(atomic.LoadInt32(&cm.connCount)) + if nconns <= cm.lowWater { log.Info("open connection count below limit") return nil } + npeers := cm.segments.countPeers() candidates := make([]*peerInfo, 0, npeers) cm.plk.RLock() for _, s := range cm.segments { @@ -220,12 +221,10 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { return candidates[i].value < candidates[j].value }) - target := npeers - cm.lowWater + target := nconns - cm.lowWater - // 2x number of peers we're disconnecting from because we may have more - // than one connection per peer. Slightly over allocating isn't an issue - // as this is a very short-lived array. - selected := make([]inet.Conn, 0, target*2) + // slightly overallocate because we may have more than one conns per peer + selected := make([]inet.Conn, 0, target+10) for _, inf := range candidates { // TODO: should we be using firstSeen or the time associated with the connection itself? @@ -239,10 +238,10 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { for c := range inf.conns { selected = append(selected, c) } + target -= len(inf.conns) s.Unlock() - target-- - if target == 0 { + if target <= 0 { break } } From f4a4d804b59ad116727cef9280dcfd29188564ab Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 18 May 2019 11:42:47 +0300 Subject: [PATCH 1220/3965] use segments in ds protobook instead of a lock array --- p2p/host/peerstore/pstoreds/protobook.go | 52 ++++++++++++------------ 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go index 02c865a8a9..9d86427b6d 100644 --- a/p2p/host/peerstore/pstoreds/protobook.go +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -9,36 +9,38 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" ) -type dsProtoBook struct { - lks [256]sync.RWMutex - meta pstore.PeerMetadata +type protoSegment struct { + sync.RWMutex } -var _ pstore.ProtoBook = (*dsProtoBook)(nil) - -func NewProtoBook(meta pstore.PeerMetadata) pstore.ProtoBook { - return &dsProtoBook{meta: meta} -} +type protoSegments [256]*protoSegment -func (pb *dsProtoBook) lock(p peer.ID) { - pb.lks[byte(p[len(p)-1])].Lock() +func (s *protoSegments) get(p peer.ID) *protoSegment { + return s[byte(p[len(p)-1])] } -func (pb *dsProtoBook) unlock(p peer.ID) { - pb.lks[byte(p[len(p)-1])].Unlock() +type dsProtoBook struct { + segments protoSegments + meta pstore.PeerMetadata } -func (pb *dsProtoBook) rlock(p peer.ID) { - pb.lks[byte(p[len(p)-1])].RLock() -} +var _ pstore.ProtoBook = (*dsProtoBook)(nil) -func (pb *dsProtoBook) runlock(p peer.ID) { - pb.lks[byte(p[len(p)-1])].RUnlock() +func NewProtoBook(meta pstore.PeerMetadata) pstore.ProtoBook { + return &dsProtoBook{ + meta: meta, + segments: func() (ret protoSegments) { + for i := range ret { + ret[i] = &protoSegment{} + } + return ret + }(), + } } func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { - pb.lock(p) - defer pb.unlock(p) + pb.segments.get(p).Lock() + defer pb.segments.get(p).Unlock() protomap := make(map[string]struct{}, len(protos)) for _, proto := range protos { @@ -49,8 +51,8 @@ func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { } func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { - pb.lock(p) - defer pb.unlock(p) + pb.segments.get(p).Lock() + defer pb.segments.get(p).Unlock() pmap, err := pb.getProtocolMap(p) if err != nil { @@ -65,8 +67,8 @@ func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { } func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { - pb.rlock(p) - defer pb.runlock(p) + pb.segments.get(p).RLock() + defer pb.segments.get(p).RUnlock() pmap, err := pb.getProtocolMap(p) if err != nil { @@ -82,8 +84,8 @@ func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { } func (pb *dsProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { - pb.rlock(p) - defer pb.runlock(p) + pb.segments.get(p).RLock() + defer pb.segments.get(p).RUnlock() pmap, err := pb.getProtocolMap(p) if err != nil { From f3ed4cf7e84c5867e60abfa609edfbcc0a47eb5b Mon Sep 17 00:00:00 2001 From: vyzo Date: Sat, 18 May 2019 12:07:18 +0300 Subject: [PATCH 1221/3965] gomod: update go-libp2p-peerstore@v0.0.6 --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index c9a9263fb9..a0dd2710f0 100644 --- a/go.mod +++ b/go.mod @@ -24,7 +24,7 @@ require ( github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 github.com/libp2p/go-libp2p-peer v0.1.1 - github.com/libp2p/go-libp2p-peerstore v0.0.5 + github.com/libp2p/go-libp2p-peerstore v0.0.6 github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 github.com/libp2p/go-libp2p-secio v0.0.3 diff --git a/go.sum b/go.sum index aeb183f8a9..d9eb1a7c37 100644 --- a/go.sum +++ b/go.sum @@ -137,8 +137,8 @@ github.com/libp2p/go-libp2p-peer v0.1.1 h1:qGCWD1a+PyZcna6htMPo26jAtqirVnJ5NvBQI github.com/libp2p/go-libp2p-peer v0.1.1/go.mod h1:jkF12jGB4Gk/IOo+yomm+7oLWxF278F7UnrYUQ1Q8es= github.com/libp2p/go-libp2p-peerstore v0.0.1 h1:twKovq8YK5trLrd3nB7PD2Zu9JcyAIdm7Bz9yBWjhq8= github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= -github.com/libp2p/go-libp2p-peerstore v0.0.5 h1:/qc3asc93924TaqVulOnY/+c4tpzomSE/ijT7LWF09M= -github.com/libp2p/go-libp2p-peerstore v0.0.5/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= +github.com/libp2p/go-libp2p-peerstore v0.0.6 h1:RgX/djPFXqZGktW0j2eF4NAX0pzDsCot45jO2GewC+g= +github.com/libp2p/go-libp2p-peerstore v0.0.6/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= github.com/libp2p/go-libp2p-protocol v0.0.1 h1:+zkEmZ2yFDi5adpVE3t9dqh/N9TbpFWywowzeEzBbLM= github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbfdvpNntjhc= From 661843a33b02b4a460e74237bb3654c41354fb76 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 21 May 2019 14:02:37 +0300 Subject: [PATCH 1222/3965] gomod: update go-libp2p-interface-connmgr --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index a0dd2710f0..a07e274fb7 100644 --- a/go.mod +++ b/go.mod @@ -16,7 +16,7 @@ require ( github.com/libp2p/go-libp2p-crypto v0.0.2 github.com/libp2p/go-libp2p-discovery v0.0.3 github.com/libp2p/go-libp2p-host v0.0.3 - github.com/libp2p/go-libp2p-interface-connmgr v0.0.4 + github.com/libp2p/go-libp2p-interface-connmgr v0.0.5 github.com/libp2p/go-libp2p-interface-pnet v0.0.1 github.com/libp2p/go-libp2p-loggables v0.0.1 github.com/libp2p/go-libp2p-metrics v0.0.1 diff --git a/go.sum b/go.sum index d9eb1a7c37..7b2c3b398d 100644 --- a/go.sum +++ b/go.sum @@ -117,6 +117,8 @@ github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 h1:Q9EkNSLAOF+u90L88qmE9z/f github.com/libp2p/go-libp2p-interface-connmgr v0.0.1/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= github.com/libp2p/go-libp2p-interface-connmgr v0.0.4 h1:/LngXETpII5qOD7YjAcQiIxhVtdAk/NQe5t9sC6BR0E= github.com/libp2p/go-libp2p-interface-connmgr v0.0.4/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= +github.com/libp2p/go-libp2p-interface-connmgr v0.0.5 h1:KG/KNYL2tYzXAfMvQN5K1aAGTYSYUMJ1prgYa2/JI1E= +github.com/libp2p/go-libp2p-interface-connmgr v0.0.5/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= github.com/libp2p/go-libp2p-interface-pnet v0.0.1 h1:7GnzRrBTJHEsofi1ahFdPN9Si6skwXQE9UqR2S+Pkh8= github.com/libp2p/go-libp2p-interface-pnet v0.0.1/go.mod h1:el9jHpQAXK5dnTpKA4yfCNBZXvrzdOU75zz+C6ryp3k= github.com/libp2p/go-libp2p-loggables v0.0.1 h1:HVww9oAnINIxbt69LJNkxD8lnbfgteXR97Xm4p3l9ps= From 3437213b5f8c22a174fb7b6076d760b438954325 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 21 May 2019 14:03:28 +0300 Subject: [PATCH 1223/3965] close connection manager --- p2p/host/basic/basic_host.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 02ca695a63..00540307a1 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -719,6 +719,7 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { // Close shuts down the Host's services (network, etc). func (h *BasicHost) Close() error { h.cancel() + h.cmgr.Close() return h.proc.Close() } From 7a70e760a154bb03ec884472df05ac7e3c4182e5 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 21 May 2019 23:05:12 +0300 Subject: [PATCH 1224/3965] copy default config, initialize coalesce delay --- p2p/muxer/yamux/yamux.go | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 05940af6d8..edc95cf87f 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -44,20 +44,20 @@ func (c *conn) AcceptStream() (smux.Stream, error) { // yamux-backed connections. type Transport yamux.Config -// DefaultTransport has default settings for yamux -var DefaultTransport = (*Transport)(&yamux.Config{ - AcceptBacklog: 256, // from yamux.DefaultConfig - EnableKeepAlive: true, // from yamux.DefaultConfig - KeepAliveInterval: 30 * time.Second, // from yamux.DefaultConfig - ConnectionWriteTimeout: 10 * time.Second, // from yamux.DefaultConfig +var DefaultTransport *Transport + +func init() { + config := yamux.DefaultConfig() // We've bumped this to 16MiB as this critically limits throughput. // // 1MiB means a best case of 10MiB/s (83.89Mbps) on a connection with // 100ms latency. The default gave us 2.4MiB *best case* which was // totally unacceptable. - MaxStreamWindowSize: uint32(16 * 1024 * 1024), - LogOutput: ioutil.Discard, -}) + config.MaxStreamWindowSize = uint32(16 * 1024 * 1024) + // don't spam + config.LogOutput = ioutil.Discard + DefaultTransport = (*Transport)(config) +} func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { var s *yamux.Session From 946aa0e70d1309ec3aea23237fb6f86f444de9bc Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 22 May 2019 00:20:09 +0300 Subject: [PATCH 1225/3965] gomod: update go-mplex@v0.0.3 --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index a07e274fb7..333fd919e1 100644 --- a/go.mod +++ b/go.mod @@ -32,7 +32,7 @@ require ( github.com/libp2p/go-libp2p-transport v0.0.4 github.com/libp2p/go-libp2p-transport-upgrader v0.0.2 github.com/libp2p/go-maddr-filter v0.0.1 - github.com/libp2p/go-mplex v0.0.2 // indirect + github.com/libp2p/go-mplex v0.0.3 // indirect github.com/libp2p/go-stream-muxer v0.0.1 github.com/libp2p/go-tcp-transport v0.0.2 github.com/libp2p/go-testutil v0.0.1 diff --git a/go.sum b/go.sum index 7b2c3b398d..89af6171fa 100644 --- a/go.sum +++ b/go.sum @@ -166,8 +166,8 @@ github.com/libp2p/go-maddr-filter v0.0.1 h1:apvYTg0aIxxQyBX+XHKOR+0+lYhGs1Yv+JmT github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-mplex v0.0.1 h1:dn2XGSrUxLtz3/8u85bGrwhUEKPX8MOF3lpmcWBZCWc= github.com/libp2p/go-mplex v0.0.1/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= -github.com/libp2p/go-mplex v0.0.2 h1:tNHgiEvcAXNASiX+SOYO32dkI57rFUHdWWLSmsoO9QQ= -github.com/libp2p/go-mplex v0.0.2/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= +github.com/libp2p/go-mplex v0.0.3 h1:YiMaevQcZtFU6DmKIF8xEO0vaui5kM5HJ1V1xkWQv14= +github.com/libp2p/go-mplex v0.0.3/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= github.com/libp2p/go-msgio v0.0.1 h1:znj97n5FtXGCLDwe9x8jpHmY770SW4WStBGcCDh6GJw= github.com/libp2p/go-msgio v0.0.1/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-msgio v0.0.2 h1:ivPvEKHxmVkTClHzg6RXTYHqaJQ0V9cDbq+6lKb3UV0= From 88c1e0ecf78a5788a9b7effbb5733d0326faac9d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 14:46:20 -0700 Subject: [PATCH 1226/3965] fix: unused import --- p2p/muxer/yamux/yamux.go | 1 - 1 file changed, 1 deletion(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index edc95cf87f..03a3230814 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -3,7 +3,6 @@ package sm_yamux import ( "io/ioutil" "net" - "time" smux "github.com/libp2p/go-stream-muxer" yamux "github.com/whyrusleeping/yamux" From dea021f127e5c5325cc3cefb6cee6cab469ca445 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 16:35:00 -0700 Subject: [PATCH 1227/3965] dep: update go-smux-* And friends. Emergency release to handle some tags I deleted. --- go.mod | 31 ++++++++-------- go.sum | 109 ++++++++++++++++++--------------------------------------- 2 files changed, 49 insertions(+), 91 deletions(-) diff --git a/go.mod b/go.mod index 333fd919e1..9be52c0a4e 100644 --- a/go.mod +++ b/go.mod @@ -2,7 +2,7 @@ module github.com/libp2p/go-libp2p require ( github.com/gogo/protobuf v1.2.1 - github.com/ipfs/go-cid v0.0.1 + github.com/ipfs/go-cid v0.0.2 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 github.com/ipfs/go-log v0.0.1 @@ -10,11 +10,11 @@ require ( github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security v0.0.1 github.com/libp2p/go-conn-security-multistream v0.0.2 - github.com/libp2p/go-libp2p-autonat v0.0.4 + github.com/libp2p/go-libp2p-autonat v0.0.5 github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.6 + github.com/libp2p/go-libp2p-circuit v0.0.7 github.com/libp2p/go-libp2p-crypto v0.0.2 - github.com/libp2p/go-libp2p-discovery v0.0.3 + github.com/libp2p/go-libp2p-discovery v0.0.4 github.com/libp2p/go-libp2p-host v0.0.3 github.com/libp2p/go-libp2p-interface-connmgr v0.0.5 github.com/libp2p/go-libp2p-interface-pnet v0.0.1 @@ -28,22 +28,21 @@ require ( github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 github.com/libp2p/go-libp2p-secio v0.0.3 - github.com/libp2p/go-libp2p-swarm v0.0.3 + github.com/libp2p/go-libp2p-swarm v0.0.5 github.com/libp2p/go-libp2p-transport v0.0.4 - github.com/libp2p/go-libp2p-transport-upgrader v0.0.2 - github.com/libp2p/go-maddr-filter v0.0.1 - github.com/libp2p/go-mplex v0.0.3 // indirect + github.com/libp2p/go-libp2p-transport-upgrader v0.0.3 + github.com/libp2p/go-maddr-filter v0.0.4 github.com/libp2p/go-stream-muxer v0.0.1 - github.com/libp2p/go-tcp-transport v0.0.2 + github.com/libp2p/go-tcp-transport v0.0.3 github.com/libp2p/go-testutil v0.0.1 - github.com/libp2p/go-ws-transport v0.0.2 - github.com/miekg/dns v1.1.4 // indirect - github.com/multiformats/go-multiaddr v0.0.2 + github.com/libp2p/go-ws-transport v0.0.3 + github.com/miekg/dns v1.1.12 // indirect + github.com/multiformats/go-multiaddr v0.0.4 github.com/multiformats/go-multiaddr-dns v0.0.2 github.com/multiformats/go-multiaddr-net v0.0.1 - github.com/multiformats/go-multistream v0.0.3 - github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible - github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible - github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible + github.com/multiformats/go-multistream v0.0.4 + github.com/whyrusleeping/go-smux-multiplex v0.1.0 + github.com/whyrusleeping/go-smux-multistream v0.1.0 + github.com/whyrusleeping/go-smux-yamux v0.1.1 github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 ) diff --git a/go.sum b/go.sum index 89af6171fa..70587a7ca2 100644 --- a/go.sum +++ b/go.sum @@ -10,7 +10,6 @@ github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVa github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc= github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY= github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= -github.com/coreos/go-semver v0.2.0 h1:3Jm3tLmsgAYcjC+4Up7hJrFBPr+n7rAqYeSw/SZazuY= github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/coreos/go-semver v0.2.1-0.20180108230905-e214231b295a h1:U0BbGfKnviqVBJQB4etvm+mKx53KfkumNLBt6YeF/0Q= github.com/coreos/go-semver v0.2.1-0.20180108230905-e214231b295a/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= @@ -33,22 +32,17 @@ github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q= github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= -github.com/gxed/hashland/keccakpg v0.0.1 h1:wrk3uMNaMxbXiHibbPO4S0ymqJMm41WiudyFSs7UnsU= github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= -github.com/gxed/hashland/murmur3 v0.0.1 h1:SheiaIt0sda5K+8FLz952/1iWS9zrnKsEJaOJu4ZbSc= github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= -github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA= -github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= -github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o= -github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/huin/goupnp v1.0.0 h1:wg75sLpL6DZqwHQN6E1Cfk6mtfzS45z8OV+ic+DtHRo= github.com/huin/goupnp v1.0.0/go.mod h1:n9v9KO1tAxYH82qOn+UTIFQDmx5n1Zxd/ClZDMX7Bnc= github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150/go.mod h1:PpLOETDnJ0o3iZrZfqZzyLl6l7F3c6L1oWn7OICBi6o= -github.com/ipfs/go-cid v0.0.1 h1:GBjWPktLnNyX0JiQCNFpUuUSoMw5KMyqrsejHYlILBE= github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= +github.com/ipfs/go-cid v0.0.2 h1:tuuKaZPU1M6HcejsO3AcYWW8sZ8MTvyxfc4uqB4eFE8= +github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= @@ -63,13 +57,11 @@ github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= -github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec h1:DQqZhhDvrTrEQ3Qod5yfavcA064e53xlQ+xajiorXgM= github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec/go.mod h1:rGaEvXB4uRSZMmzKNLoXvTu1sfx+1kv/DojUlPrSZGs= github.com/jbenet/go-cienv v0.1.0 h1:Vc/s0QbQtoxX8MwwSLWWh+xNNZvM3Lw7NsTcHrvvhMc= github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA= github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2 h1:vhC1OXXiT9R2pczegwz6moDvuRpggaroAXhPIseh57A= github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2/go.mod h1:8GXXJV31xl8whumTzdZsTt3RnUIiPqzkyf7mxToRCMs= -github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8 h1:bspPhN+oKYFk5fcGNuQzp6IGzYQSenLEgH3s6jkXrWw= github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsjFq/qrU3Rar62tu1gASgGw6chQbSh/XgIIXCY= github.com/jbenet/goprocess v0.1.3 h1:YKyIEECS/XvcfHtBzxtjBBbWK+MbvA6dG8ASiqwvr10= github.com/jbenet/goprocess v0.1.3/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= @@ -87,35 +79,30 @@ github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/libp2p/go-addr-util v0.0.1 h1:TpTQm9cXVRVSKsYbgQ7GKc3KbbHVTnbostgGaDEP+88= github.com/libp2p/go-addr-util v0.0.1/go.mod h1:4ac6O7n9rIAKB1dnd+s8IbbMXkt+oBpzX4/+RACcnlQ= -github.com/libp2p/go-buffer-pool v0.0.1 h1:9Rrn/H46cXjaA2HQ5Y8lyhOS1NhTkZ4yuEs2r3Eechg= github.com/libp2p/go-buffer-pool v0.0.1/go.mod h1:xtyIz9PMobb13WaxR6Zo1Pd1zXJKYg0a8KiIvDp3TzQ= +github.com/libp2p/go-buffer-pool v0.0.2 h1:QNK2iAFa8gjAe1SPz6mHSMuCcjs+X1wlHzeOSqcmlfs= +github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM= github.com/libp2p/go-conn-security v0.0.1 h1:4kMMrqrt9EUNCNjX1xagSJC+bq16uqjMe9lk1KBMVNs= github.com/libp2p/go-conn-security v0.0.1/go.mod h1:bGmu51N0KU9IEjX7kl2PQjgZa40JQWnayTvNMgD/vyk= -github.com/libp2p/go-conn-security-multistream v0.0.1 h1:XefjAQRHcnUaxKb26RGupToucx3uU4ecbOZ3aACXlDU= -github.com/libp2p/go-conn-security-multistream v0.0.1/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= github.com/libp2p/go-conn-security-multistream v0.0.2 h1:Ykz0lnNjxk+0SdslUmlLNyrleqdpS1S/VW+dxFdt74Y= github.com/libp2p/go-conn-security-multistream v0.0.2/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= -github.com/libp2p/go-libp2p-autonat v0.0.4 h1:cZzdB9KW1ZkHnSjLCB6aFNw47XS4r+SecCVMuVB1xgo= -github.com/libp2p/go-libp2p-autonat v0.0.4/go.mod h1:fs71q5Xk+pdnKU014o2iq1RhMs9/PMaG5zXRFNnIIT4= +github.com/libp2p/go-libp2p-autonat v0.0.5 h1:knn3iuP/UZdR5XtgflR9l7Gle4tlRG9Y4frzshMl3KY= +github.com/libp2p/go-libp2p-autonat v0.0.5/go.mod h1:cKt+qOSnWAZp0dqIuUk62v0/QAPw0vnLuVZnmzkOXRk= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.6 h1:egD2CKFVdqnHgIHzPkM6J7m3MKZpFqoTPDfxBqQ7kRQ= -github.com/libp2p/go-libp2p-circuit v0.0.6/go.mod h1:W34ISBRpoCPUeOR26xzTbLo+s3hDO9153hJCfvHzBlg= -github.com/libp2p/go-libp2p-crypto v0.0.1 h1:JNQd8CmoGTohO/akqrH16ewsqZpci2CbgYH/LmYl8gw= +github.com/libp2p/go-libp2p-circuit v0.0.7 h1:2oIJp4Qud9V8Boy+31nmbNBwH5PCJisGH4SwYTPSU50= +github.com/libp2p/go-libp2p-circuit v0.0.7/go.mod h1:DFCgZ2DklFGTUIZIhSvbbWXTErUgjyNrJGfDHOrTKIA= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= github.com/libp2p/go-libp2p-crypto v0.0.2/go.mod h1:eETI5OUfBnvARGOHrJz2eWNyTUxEGZnBxMcbUjfIj4I= -github.com/libp2p/go-libp2p-discovery v0.0.3 h1:/41EY87ErU1yQLoPoTfJXV3xomDwx7Ti1Ae66DuDk9U= -github.com/libp2p/go-libp2p-discovery v0.0.3/go.mod h1:ZkkF9xIFRLA1xCc7bstYFkd80gBGK8Fc1JqGoU2i+zI= -github.com/libp2p/go-libp2p-host v0.0.1 h1:dnqusU+DheGcdxrE718kG4XgHNuL2n9eEv8Rg5zy8hQ= +github.com/libp2p/go-libp2p-discovery v0.0.4 h1:/kZwOVmcUHvB94zegSJYnUA9EvT1g8APoQJb5FHyT1c= +github.com/libp2p/go-libp2p-discovery v0.0.4/go.mod h1:ReQGiv7QTtza8FUWzewfuMmRDVOQVp+lxHlJJA8YQCM= github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= github.com/libp2p/go-libp2p-host v0.0.3 h1:BB/1Z+4X0rjKP5lbQTmjEjLbDVbrcmLOlA6QDsN5/j4= github.com/libp2p/go-libp2p-host v0.0.3/go.mod h1:Y/qPyA6C8j2coYyos1dfRm0I8+nvd4TGrDGt4tA7JR8= -github.com/libp2p/go-libp2p-interface-connmgr v0.0.1 h1:Q9EkNSLAOF+u90L88qmE9z/fTdjLh8OsJwGw74mkwk4= github.com/libp2p/go-libp2p-interface-connmgr v0.0.1/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= -github.com/libp2p/go-libp2p-interface-connmgr v0.0.4 h1:/LngXETpII5qOD7YjAcQiIxhVtdAk/NQe5t9sC6BR0E= github.com/libp2p/go-libp2p-interface-connmgr v0.0.4/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= github.com/libp2p/go-libp2p-interface-connmgr v0.0.5 h1:KG/KNYL2tYzXAfMvQN5K1aAGTYSYUMJ1prgYa2/JI1E= github.com/libp2p/go-libp2p-interface-connmgr v0.0.5/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= @@ -127,17 +114,14 @@ github.com/libp2p/go-libp2p-metrics v0.0.1 h1:yumdPC/P2VzINdmcKZd0pciSUCpou+s0lw github.com/libp2p/go-libp2p-metrics v0.0.1/go.mod h1:jQJ95SXXA/K1VZi13h52WZMa9ja78zjyy5rspMsC/08= github.com/libp2p/go-libp2p-nat v0.0.4 h1:+KXK324yaY701On8a0aGjTnw8467kW3ExKcqW2wwmyw= github.com/libp2p/go-libp2p-nat v0.0.4/go.mod h1:N9Js/zVtAXqaeT99cXgTV9e75KpnWCvVOiGzlcHmBbY= -github.com/libp2p/go-libp2p-net v0.0.1 h1:xJ4Vh4yKF/XKb8fd1Ev0ebAGzVjMxXzrxG2kjtU+F5Q= github.com/libp2p/go-libp2p-net v0.0.1/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= github.com/libp2p/go-libp2p-net v0.0.2 h1:qP06u4TYXfl7uW/hzqPhlVVTSA2nw1B/bHBJaUnbh6M= github.com/libp2p/go-libp2p-net v0.0.2/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= github.com/libp2p/go-libp2p-netutil v0.0.1 h1:LgD6+skofkOx8z6odD9+MZHKjupv3ng1u6KRhaADTnA= github.com/libp2p/go-libp2p-netutil v0.0.1/go.mod h1:GdusFvujWZI9Vt0X5BKqwWWmZFxecf9Gt03cKxm2f/Q= -github.com/libp2p/go-libp2p-peer v0.0.1 h1:0qwAOljzYewINrU+Kndoc+1jAL7vzY/oY2Go4DCGfyY= github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= github.com/libp2p/go-libp2p-peer v0.1.1 h1:qGCWD1a+PyZcna6htMPo26jAtqirVnJ5NvBQIKV7rRY= github.com/libp2p/go-libp2p-peer v0.1.1/go.mod h1:jkF12jGB4Gk/IOo+yomm+7oLWxF278F7UnrYUQ1Q8es= -github.com/libp2p/go-libp2p-peerstore v0.0.1 h1:twKovq8YK5trLrd3nB7PD2Zu9JcyAIdm7Bz9yBWjhq8= github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= github.com/libp2p/go-libp2p-peerstore v0.0.6 h1:RgX/djPFXqZGktW0j2eF4NAX0pzDsCot45jO2GewC+g= github.com/libp2p/go-libp2p-peerstore v0.0.6/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= @@ -145,73 +129,55 @@ github.com/libp2p/go-libp2p-protocol v0.0.1 h1:+zkEmZ2yFDi5adpVE3t9dqh/N9TbpFWyw github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbfdvpNntjhc= github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= -github.com/libp2p/go-libp2p-secio v0.0.1 h1:CqE/RdsizOwItdgLe632iyft/w0tshDLmZGAiKDcUAI= -github.com/libp2p/go-libp2p-secio v0.0.1/go.mod h1:IdG6iQybdcYmbTzxp4J5dwtUEDTOvZrT0opIDVNPrJs= github.com/libp2p/go-libp2p-secio v0.0.3 h1:h3fPeDrej7bvvARnC2oSjAfcLZOaS4REZKgWCRQNpE4= github.com/libp2p/go-libp2p-secio v0.0.3/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= -github.com/libp2p/go-libp2p-swarm v0.0.1 h1:Vne+hjaDwXqzgNwQ2vb2YKbnbOTyXjtS47stT66Apc4= -github.com/libp2p/go-libp2p-swarm v0.0.1/go.mod h1:mh+KZxkbd3lQnveQ3j2q60BM1Cw2mX36XXQqwfPOShs= -github.com/libp2p/go-libp2p-swarm v0.0.2 h1:cpHHXTeU2IgUu8LPemF7vaLPGtVC6VxMoll2EwqlC+E= -github.com/libp2p/go-libp2p-swarm v0.0.2/go.mod h1:n0cAAcKyndIrJWctQwjqXlAdIPBZzfdpBjx1SSvz30g= -github.com/libp2p/go-libp2p-swarm v0.0.3 h1:gF11uO1WCbFtTjKLQ+gZ/UdbYScZq8PfqA53XdbiK8Q= -github.com/libp2p/go-libp2p-swarm v0.0.3/go.mod h1:/2HbOacAKDYT1g0UEZjUPlzD+SBtvqkg4TaYeoBA2TY= +github.com/libp2p/go-libp2p-swarm v0.0.5 h1:oue1c+NbyFTuwCkWulX+ouyQoaZsdYeYcAzQ/slfxXY= +github.com/libp2p/go-libp2p-swarm v0.0.5/go.mod h1:+nkJir4feiXtWQjb/4CQHMEK8Vw+c5nVVxT8R5bs0yY= github.com/libp2p/go-libp2p-transport v0.0.1/go.mod h1:UzbUs9X+PHOSw7S3ZmeOxfnwaQY5vGDzZmKPod3N3tk= github.com/libp2p/go-libp2p-transport v0.0.4 h1:/CPHQMN75/IQwkhBxxIo6p6PtL3rwFZtlzBROT3e8mw= github.com/libp2p/go-libp2p-transport v0.0.4/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= -github.com/libp2p/go-libp2p-transport-upgrader v0.0.1 h1:rNtXkY6dty46mxYOHHAZQchI7gQdJStF683FhVnei/k= -github.com/libp2p/go-libp2p-transport-upgrader v0.0.1/go.mod h1:NJpUAgQab/8K6K0m+JmZCe5RUXG10UMEx4kWe9Ipj5c= -github.com/libp2p/go-libp2p-transport-upgrader v0.0.2 h1:VygEEnAyTPZxVkEOVsmLp0vGLGBEn4W1UBHcy9cEywg= -github.com/libp2p/go-libp2p-transport-upgrader v0.0.2/go.mod h1:NJpUAgQab/8K6K0m+JmZCe5RUXG10UMEx4kWe9Ipj5c= -github.com/libp2p/go-maddr-filter v0.0.1 h1:apvYTg0aIxxQyBX+XHKOR+0+lYhGs1Yv+JmTH9nyl5I= +github.com/libp2p/go-libp2p-transport-upgrader v0.0.3 h1:ZcXhGEMnv8f+ycqVl/9mnIe6uhUYy0aj/vEYQUoG7xU= +github.com/libp2p/go-libp2p-transport-upgrader v0.0.3/go.mod h1:Ng1HzfMIopyYscMHNFmJqiMMcpgDlj0t+NyjVWW89ws= github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= -github.com/libp2p/go-mplex v0.0.1 h1:dn2XGSrUxLtz3/8u85bGrwhUEKPX8MOF3lpmcWBZCWc= -github.com/libp2p/go-mplex v0.0.1/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= +github.com/libp2p/go-maddr-filter v0.0.4 h1:hx8HIuuwk34KePddrp2mM5ivgPkZ09JH4AvsALRbFUs= +github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-mplex v0.0.3 h1:YiMaevQcZtFU6DmKIF8xEO0vaui5kM5HJ1V1xkWQv14= github.com/libp2p/go-mplex v0.0.3/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= -github.com/libp2p/go-msgio v0.0.1 h1:znj97n5FtXGCLDwe9x8jpHmY770SW4WStBGcCDh6GJw= -github.com/libp2p/go-msgio v0.0.1/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-msgio v0.0.2 h1:ivPvEKHxmVkTClHzg6RXTYHqaJQ0V9cDbq+6lKb3UV0= github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-nat v0.0.3 h1:l6fKV+p0Xa354EqQOQP+d8CivdLM4kl5GxC1hSc/UeI= github.com/libp2p/go-nat v0.0.3/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= -github.com/libp2p/go-reuseport-transport v0.0.1 h1:UIRneNxLDmEGNjGHpIiWzSWkZ5bhxMCP9x3Vh7BSc7E= -github.com/libp2p/go-reuseport-transport v0.0.1/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0GIEjMiHTwR/KN4= github.com/libp2p/go-reuseport-transport v0.0.2/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= github.com/libp2p/go-stream-muxer v0.0.1 h1:Ce6e2Pyu+b5MC1k3eeFtAax0pW4gc6MosYSLV05UeLw= github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= -github.com/libp2p/go-tcp-transport v0.0.1 h1:WyvJVw2lYAnr6CU+GZZ4oCt06fvORlmvBlFX2+ZpZDM= -github.com/libp2p/go-tcp-transport v0.0.1/go.mod h1:mnjg0o0O5TmXUaUIanYPUqkW4+u6mK0en8rlpA6BBTs= -github.com/libp2p/go-tcp-transport v0.0.2 h1:EzOSRaHpyrGpJ5qe+9SaxJM1mrWlkSLFfNTufUkq0lg= -github.com/libp2p/go-tcp-transport v0.0.2/go.mod h1:VjZFHasNJ0QiJQNNwiFDy25qyGWTXQWs8GM5DR4/L1k= +github.com/libp2p/go-tcp-transport v0.0.3 h1:ZDMczeg+6uvvlxnShYFY0wGjQqStFhWAYT7769utxuM= +github.com/libp2p/go-tcp-transport v0.0.3/go.mod h1:f11C2zvCaGDkE8aFPUKmuYZwd3pP6HI24LeLMWhJnkQ= github.com/libp2p/go-testutil v0.0.1 h1:Xg+O0G2HIMfHqBOBDcMS1iSZJ3GEcId4qOxCQvsGZHk= github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= -github.com/libp2p/go-ws-transport v0.0.2 h1:PtK1AoM16nm96FwPBQoq+4T4t9LdDwOhkB+mdXuGSlg= -github.com/libp2p/go-ws-transport v0.0.2/go.mod h1:p3bKjDWHEgtuKKj+2OdPYs5dAPIjtpQGHF2tJfGz7Ww= +github.com/libp2p/go-ws-transport v0.0.3 h1:29hkvsmdOhniNuJ5AzFICfY5SA5nvDO9Xi3FtcGqO+4= +github.com/libp2p/go-ws-transport v0.0.3/go.mod h1:iU0kzfMcO4tBVIk3z+7srp1YG/RFLWTSuO4enpivw8g= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= -github.com/miekg/dns v1.1.4 h1:rCMZsU2ScVSYcAsOXgmC6+AKOK+6pmQTOcw03nfwYV0= -github.com/miekg/dns v1.1.4/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= +github.com/miekg/dns v1.1.12 h1:WMhc1ik4LNkTg8U9l3hI1LvxKmIL+f1+WV/SZtCbDDA= +github.com/miekg/dns v1.1.12/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ= -github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16 h1:5W7KhL8HVF3XCFOweFD3BNESdnO8ewyYTFT2R+/b8FQ= github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5 h1:l16XLUUJ34wIz+RIvLhSwGvLvKyy+W598b135bJN6mg= github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= -github.com/mr-tron/base58 v1.1.0 h1:Y51FGVJ91WBqCEabAi5OPUz38eAx8DakuAm5svLcsfQ= github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= github.com/mr-tron/base58 v1.1.2 h1:ZEw4I2EgPKDJ2iEw0cNmLB3ROrEmkOtXIkaG7wZg+78= github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI= github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA= -github.com/multiformats/go-multiaddr v0.0.1 h1:/QUV3VBMDI6pi6xfiw7lr6xhDWWvQKn9udPn68kLSdY= github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= -github.com/multiformats/go-multiaddr v0.0.2 h1:RBysRCv5rv3FWlhKWKoXv8tnsCUpEpIZpCmqAGZos2s= -github.com/multiformats/go-multiaddr v0.0.2/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.0.4 h1:WgMSI84/eRLdbptXMkMWDXPjPq7SPLIgGUVm2eroyU4= +github.com/multiformats/go-multiaddr v0.0.4/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= @@ -219,14 +185,12 @@ github.com/multiformats/go-multiaddr-net v0.0.1 h1:76O59E3FavvHqNg7jvzWzsPSW5JSi github.com/multiformats/go-multiaddr-net v0.0.1/go.mod h1:nw6HSxNmCIQH27XPGBuX+d1tnvM7ihcFwHMSstNAVUU= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= -github.com/multiformats/go-multihash v0.0.1 h1:HHwN1K12I+XllBCrqKnhX949Orn4oawPkegHMu2vDqQ= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= github.com/multiformats/go-multihash v0.0.5 h1:1wxmCvTXAifAepIMyF39vZinRw5sbqjPs/UIi93+uik= github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= -github.com/multiformats/go-multistream v0.0.1 h1:JV4VfSdY9n7ECTtY59/TlSyFCzRILvYx4T4Ws8ZgihU= github.com/multiformats/go-multistream v0.0.1/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= -github.com/multiformats/go-multistream v0.0.3 h1:sRTU9QWosQWXteaM0PHCFVWSIHWxOyjdwL4MF7Lsdfg= -github.com/multiformats/go-multistream v0.0.3/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= +github.com/multiformats/go-multistream v0.0.4 h1:rNgWgFyzRSTI9L+xISrz7kN5MdNXoEcoIeeCH05wLKA= +github.com/multiformats/go-multistream v0.0.4/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0 h1:WSHQ+IS43OoUrWtD1/bbclrwK8TTH5hzp+umCiuxHgs= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= @@ -253,33 +217,29 @@ github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6 github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM= github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f h1:M/lL30eFZTKnomXY6huvM6G0+gVquFNf6mxghaWlFUg= github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f/go.mod h1:cZNvX9cFybI01GriPRMXDtczuvUhgbcYr9iCGaNlRv8= -github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible h1:iqksILj8STw03EJQe7Laj4ubnw+ojOyik18cd5vPL1o= -github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible/go.mod h1:34LEDbeKFZInPUrAG+bjuJmUXONGdEFW7XL0SpTY1y4= -github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible h1:BdYHctE9HJZLquG9tpTdwWcbG4FaX6tVKPGjCGgiVxo= -github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible/go.mod h1:dRWHHvc4HDQSHh9gbKEBbUZ+f2Q8iZTPG3UOGYODxSQ= -github.com/whyrusleeping/go-smux-yamux v2.0.8+incompatible h1:IGm/UP/JpEFS6D787sZnZg7RA6fZIR9c/Ms9DeAVNuk= -github.com/whyrusleeping/go-smux-yamux v2.0.8+incompatible/go.mod h1:6qHUzBXUbB9MXmw3AUdB52L8sEb/hScCqOdW2kj/wuI= -github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible h1:nVkExQ7pYlN9e45LcqTCOiDD0904fjtm0flnHZGbXkw= -github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible/go.mod h1:6qHUzBXUbB9MXmw3AUdB52L8sEb/hScCqOdW2kj/wuI= +github.com/whyrusleeping/go-smux-multiplex v0.1.0 h1:23qGy3Z16UeiMh/BTiRe6pRm7MlvF5EM2873uFTQbro= +github.com/whyrusleeping/go-smux-multiplex v0.1.0/go.mod h1:OXL5hggHNZSsadXDlBJDD4eD3IQYEB3Yu6xpovd6pPw= +github.com/whyrusleeping/go-smux-multistream v0.1.0 h1:+x8twbOR2IJtc1I5PF+UT6DqTJmbog4NiB/k/hyYW7k= +github.com/whyrusleeping/go-smux-multistream v0.1.0/go.mod h1:/usW3LIBirW4h9ko1PnoF7tExBnbxPBszG0n4wylJr8= +github.com/whyrusleeping/go-smux-yamux v0.1.1 h1:g9dfTJbs/StYfi1e3id9+nQO8lYaY4sCIW2zESECcTM= +github.com/whyrusleeping/go-smux-yamux v0.1.1/go.mod h1:Yw+ayOEKERDHXLJ4GiE5AnBmldJW4QRLDzGFC9do8G0= github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxfC1Y6mYncdk+kq8d5aLx0Q+/gyZGE44M= github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= -github.com/whyrusleeping/yamux v1.1.5 h1:4CK3aUUJQu0qpKZv5gEWJjNOQtdbdDhVVS6PJ+HimdE= -github.com/whyrusleeping/yamux v1.1.5/go.mod h1:E8LnQQ8HKx5KD29HZFUwM1PxCOdPRzGwur1mcYhXcD8= +github.com/whyrusleeping/yamux v1.2.0 h1:PzUrk7/Z0g/N5V4/+DesmKXYcCToALgj+SbATgs0B34= +github.com/whyrusleeping/yamux v1.2.0/go.mod h1:Cgw3gpb4DrDZ1FrP/5pxg/cpiY54Gr5uCXwUylwi2GE= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25 h1:jsG6UpNLt9iAsb0S2AGW28DveNzzgmbXR+ENoPjUeIU= golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734 h1:p/H982KKEjUnLJkM3tt/LemDnOc1GiZL5FCVlORJ5zo= golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190227160552-c95aed5357e7 h1:C2F/nMkR/9sfUTpvR3QrjBuTdvMUC/cFajkphs1YLQo= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= @@ -289,7 +249,6 @@ golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e h1:ZytStCyV048ZqDsWHiYDdoI2Vd4msMcrDECFxS+tL9c= golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= From 97d03a46b4669b5185348f2e84d1955002b95865 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 16:57:29 -0700 Subject: [PATCH 1228/3965] test: add a small delay to deal with write coalescing --- p2p/test/backpressure/backpressure_test.go | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/p2p/test/backpressure/backpressure_test.go b/p2p/test/backpressure/backpressure_test.go index ab84ab2ab5..097c697bd0 100644 --- a/p2p/test/backpressure/backpressure_test.go +++ b/p2p/test/backpressure/backpressure_test.go @@ -211,6 +211,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { // writeStats lets us listen to all the writes and return // how many happened and how much was written writeStats := func() (int, int) { + t.Helper() writes := 0 bytes := 0 for { @@ -259,6 +260,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { // receive a number of bytes from a stream. // returns the number of bytes written. receive := func(s inet.Stream, expect int) { + t.Helper() log.Debugf("receiver to read %d bytes", expect) rbuf := make([]byte, expect) n, err := io.ReadFull(s, rbuf) @@ -294,6 +296,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { // let's make sure r/w works. testSenderWrote := func(bytesE int) { + t.Helper() bytesA, writesA := writeStats() if bytesA != bytesE { t.Errorf("numbers failed: %d =?= %d bytes, via %d writes", bytesA, bytesE, writesA) @@ -311,7 +314,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { roundsTotal := 0 for roundsTotal < (2 << 20) { // let the sender fill its buffers, it will stop sending. - <-time.After(300 * time.Millisecond) + <-time.After(400 * time.Millisecond) b, _ := writeStats() testSenderWrote(0) testSenderWrote(0) @@ -357,7 +360,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { // and a couple rounds more for good measure ;) for i := 0; i < 3; i++ { // let the sender fill its buffers, it will stop sending. - <-time.After(300 * time.Millisecond) + <-time.After(400 * time.Millisecond) b, _ := writeStats() testSenderWrote(0) testSenderWrote(0) From c35c511ccc5e410632a421f15ecfbaa9b2924973 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 18:41:01 -0700 Subject: [PATCH 1229/3965] migrate to libp2p org --- p2p/muxer/yamux/yamux.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 03a3230814..965c8fb47c 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -5,7 +5,7 @@ import ( "net" smux "github.com/libp2p/go-stream-muxer" - yamux "github.com/whyrusleeping/yamux" + yamux "github.com/libp2p/go-yamux" ) // Conn is a connection to a remote peer. From 67b7b275cfad17c45434329289ca13d56014b05b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 18:56:06 -0700 Subject: [PATCH 1230/3965] dep: import go-libp2p-mplex into the libp2p org --- p2p/net/upgrader/listener_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go index 969b98d238..6d98f0850e 100644 --- a/p2p/net/upgrader/listener_test.go +++ b/p2p/net/upgrader/listener_test.go @@ -14,7 +14,7 @@ import ( smux "github.com/libp2p/go-stream-muxer" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" - mplex "github.com/whyrusleeping/go-smux-multiplex" + mplex "github.com/libp2p/go-libp2p-mplex" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" From b06b0e6b216e33f1a39fe984acfe47c1b4a13eaa Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 19:00:11 -0700 Subject: [PATCH 1231/3965] dep: import go-libp2p-mplex into the libp2p org --- p2p/transport/tcp/tcp_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/tcp/tcp_test.go b/p2p/transport/tcp/tcp_test.go index 4467e8a2d7..4efd1cb6b2 100644 --- a/p2p/transport/tcp/tcp_test.go +++ b/p2p/transport/tcp/tcp_test.go @@ -4,10 +4,10 @@ import ( "testing" insecure "github.com/libp2p/go-conn-security/insecure" + mplex "github.com/libp2p/go-libp2p-mplex" tptu "github.com/libp2p/go-libp2p-transport-upgrader" utils "github.com/libp2p/go-libp2p-transport/test" ma "github.com/multiformats/go-multiaddr" - mplex "github.com/whyrusleeping/go-smux-multiplex" ) func TestTcpTransport(t *testing.T) { From b7f9522391acbd230f97b682c09c5de64edb3632 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 19:22:52 -0700 Subject: [PATCH 1232/3965] dep: import go-smux-* into the libp2p org --- p2p/transport/websocket/websocket_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index b9c688dc4b..ae55afe907 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -9,10 +9,10 @@ import ( "testing/iotest" insecure "github.com/libp2p/go-conn-security/insecure" + mplex "github.com/libp2p/go-libp2p-mplex" tptu "github.com/libp2p/go-libp2p-transport-upgrader" utils "github.com/libp2p/go-libp2p-transport/test" ma "github.com/multiformats/go-multiaddr" - mplex "github.com/whyrusleeping/go-smux-multiplex" ) func TestCanDial(t *testing.T) { From cf873f59f686bfe50c2e240e4ff5f3c4ac1556b6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 19:33:31 -0700 Subject: [PATCH 1233/3965] dep: import go-smux-* into the libp2p org --- p2p/net/swarm/testing/testing.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/swarm/testing/testing.go b/p2p/net/swarm/testing/testing.go index e3a62c5283..edaaf5b274 100644 --- a/p2p/net/swarm/testing/testing.go +++ b/p2p/net/swarm/testing/testing.go @@ -11,10 +11,10 @@ import ( pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" secio "github.com/libp2p/go-libp2p-secio" tptu "github.com/libp2p/go-libp2p-transport-upgrader" + yamux "github.com/libp2p/go-libp2p-yamux" + msmux "github.com/libp2p/go-stream-muxer-multistream" tcp "github.com/libp2p/go-tcp-transport" tu "github.com/libp2p/go-testutil" - msmux "github.com/whyrusleeping/go-smux-multistream" - yamux "github.com/whyrusleeping/go-smux-yamux" swarm "github.com/libp2p/go-libp2p-swarm" ) From 26f880b899dd20faa31e27ca2e7d21ef80148026 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 19:44:10 -0700 Subject: [PATCH 1234/3965] dep: import go-smux-* into the libp2p org 0. I imported whyrusleeping/go-smux-* into go mod. 1. To handle post-1.0 tags, I renamed all of our "gx" tags to "gx/v..." as we did with the rest of libp2p. 2. This broke _everything_ because `go mod` absolutely can't deal with deleted tags if any transitive go.sum file mentions the tag. 3. To fix this, I first got rid of all mentions of these 2.x+incompatible tags. 4. Unfortunately, I realized this was still going to be a nightmare for downstream users as a `go get github.com/libp2p/go-libp2p@latest` on a fresh clone (no go mod cache) would fail. I didn't want to have to explain to all of our downstream users "please delete your go.sum files!". 5. So I did what I should have done in the first place: a. I restored those tags. b. I forked all the whyrusleeping/go-smux-* repositories (and whyrusleeping/yamux) into the libp2p org where. c. I removed all tags but the _new_ go-mod tags on those new libp2p repos. What does this mean for downstream users? If they manually import any of these stream muxer repos, they'll have to fix their imports. Otherwise, it shouldn't make a difference. --- config/muxer.go | 2 +- config/muxer_test.go | 2 +- defaults.go | 4 ++-- go.mod | 14 +++++++------- go.sum | 16 ++++++++++++++++ 5 files changed, 27 insertions(+), 11 deletions(-) diff --git a/config/muxer.go b/config/muxer.go index ee2adeb08a..88a83910ad 100644 --- a/config/muxer.go +++ b/config/muxer.go @@ -5,7 +5,7 @@ import ( host "github.com/libp2p/go-libp2p-host" mux "github.com/libp2p/go-stream-muxer" - msmux "github.com/whyrusleeping/go-smux-multistream" + msmux "github.com/libp2p/go-stream-muxer-multistream" ) // MuxC is a stream multiplex transport constructor diff --git a/config/muxer_test.go b/config/muxer_test.go index d7f7baeb5d..490c95e62b 100644 --- a/config/muxer_test.go +++ b/config/muxer_test.go @@ -7,9 +7,9 @@ import ( host "github.com/libp2p/go-libp2p-host" peer "github.com/libp2p/go-libp2p-peer" swarmt "github.com/libp2p/go-libp2p-swarm/testing" + yamux "github.com/libp2p/go-libp2p-yamux" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" mux "github.com/libp2p/go-stream-muxer" - yamux "github.com/whyrusleeping/go-smux-yamux" ) func TestMuxerSimple(t *testing.T) { diff --git a/defaults.go b/defaults.go index 679fd7af5a..6cd5df42e3 100644 --- a/defaults.go +++ b/defaults.go @@ -6,13 +6,13 @@ import ( "crypto/rand" crypto "github.com/libp2p/go-libp2p-crypto" + mplex "github.com/libp2p/go-libp2p-mplex" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" secio "github.com/libp2p/go-libp2p-secio" + yamux "github.com/libp2p/go-libp2p-yamux" tcp "github.com/libp2p/go-tcp-transport" ws "github.com/libp2p/go-ws-transport" multiaddr "github.com/multiformats/go-multiaddr" - mplex "github.com/whyrusleeping/go-smux-multiplex" - yamux "github.com/whyrusleeping/go-smux-yamux" ) // DefaultSecurity is the default security option. diff --git a/go.mod b/go.mod index 9be52c0a4e..95f5f342d9 100644 --- a/go.mod +++ b/go.mod @@ -20,6 +20,7 @@ require ( github.com/libp2p/go-libp2p-interface-pnet v0.0.1 github.com/libp2p/go-libp2p-loggables v0.0.1 github.com/libp2p/go-libp2p-metrics v0.0.1 + github.com/libp2p/go-libp2p-mplex v0.1.1 github.com/libp2p/go-libp2p-nat v0.0.4 github.com/libp2p/go-libp2p-net v0.0.2 github.com/libp2p/go-libp2p-netutil v0.0.1 @@ -28,12 +29,14 @@ require ( github.com/libp2p/go-libp2p-protocol v0.0.1 github.com/libp2p/go-libp2p-routing v0.0.1 github.com/libp2p/go-libp2p-secio v0.0.3 - github.com/libp2p/go-libp2p-swarm v0.0.5 - github.com/libp2p/go-libp2p-transport v0.0.4 - github.com/libp2p/go-libp2p-transport-upgrader v0.0.3 + github.com/libp2p/go-libp2p-swarm v0.0.6 + github.com/libp2p/go-libp2p-transport v0.0.5 + github.com/libp2p/go-libp2p-transport-upgrader v0.0.4 + github.com/libp2p/go-libp2p-yamux v0.1.2 github.com/libp2p/go-maddr-filter v0.0.4 github.com/libp2p/go-stream-muxer v0.0.1 - github.com/libp2p/go-tcp-transport v0.0.3 + github.com/libp2p/go-stream-muxer-multistream v0.1.1 + github.com/libp2p/go-tcp-transport v0.0.4 github.com/libp2p/go-testutil v0.0.1 github.com/libp2p/go-ws-transport v0.0.3 github.com/miekg/dns v1.1.12 // indirect @@ -41,8 +44,5 @@ require ( github.com/multiformats/go-multiaddr-dns v0.0.2 github.com/multiformats/go-multiaddr-net v0.0.1 github.com/multiformats/go-multistream v0.0.4 - github.com/whyrusleeping/go-smux-multiplex v0.1.0 - github.com/whyrusleeping/go-smux-multistream v0.1.0 - github.com/whyrusleeping/go-smux-yamux v0.1.1 github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 ) diff --git a/go.sum b/go.sum index 70587a7ca2..c950e489a4 100644 --- a/go.sum +++ b/go.sum @@ -112,6 +112,8 @@ github.com/libp2p/go-libp2p-loggables v0.0.1 h1:HVww9oAnINIxbt69LJNkxD8lnbfgteXR github.com/libp2p/go-libp2p-loggables v0.0.1/go.mod h1:lDipDlBNYbpyqyPX/KcoO+eq0sJYEVR2JgOexcivchg= github.com/libp2p/go-libp2p-metrics v0.0.1 h1:yumdPC/P2VzINdmcKZd0pciSUCpou+s0lwYCjBbzQZU= github.com/libp2p/go-libp2p-metrics v0.0.1/go.mod h1:jQJ95SXXA/K1VZi13h52WZMa9ja78zjyy5rspMsC/08= +github.com/libp2p/go-libp2p-mplex v0.1.1 h1:lSPS1VJ36P01gGO//KgcsmSah5uoC3X9r7WY5j+iP4c= +github.com/libp2p/go-libp2p-mplex v0.1.1/go.mod h1:KUQWpGkCzfV7UIpi8SKsAVxyBgz1c9R5EvxgnwLsb/I= github.com/libp2p/go-libp2p-nat v0.0.4 h1:+KXK324yaY701On8a0aGjTnw8467kW3ExKcqW2wwmyw= github.com/libp2p/go-libp2p-nat v0.0.4/go.mod h1:N9Js/zVtAXqaeT99cXgTV9e75KpnWCvVOiGzlcHmBbY= github.com/libp2p/go-libp2p-net v0.0.1/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= @@ -133,11 +135,19 @@ github.com/libp2p/go-libp2p-secio v0.0.3 h1:h3fPeDrej7bvvARnC2oSjAfcLZOaS4REZKgW github.com/libp2p/go-libp2p-secio v0.0.3/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= github.com/libp2p/go-libp2p-swarm v0.0.5 h1:oue1c+NbyFTuwCkWulX+ouyQoaZsdYeYcAzQ/slfxXY= github.com/libp2p/go-libp2p-swarm v0.0.5/go.mod h1:+nkJir4feiXtWQjb/4CQHMEK8Vw+c5nVVxT8R5bs0yY= +github.com/libp2p/go-libp2p-swarm v0.0.6 h1:gE0P/v2h+KEXtAi9YTw2UBOSODJ4m9VuuJ+ktc2LVUo= +github.com/libp2p/go-libp2p-swarm v0.0.6/go.mod h1:s5GZvzg9xXe8sbeESuFpjt8CJPTCa8mhEusweJqyFy8= github.com/libp2p/go-libp2p-transport v0.0.1/go.mod h1:UzbUs9X+PHOSw7S3ZmeOxfnwaQY5vGDzZmKPod3N3tk= github.com/libp2p/go-libp2p-transport v0.0.4 h1:/CPHQMN75/IQwkhBxxIo6p6PtL3rwFZtlzBROT3e8mw= github.com/libp2p/go-libp2p-transport v0.0.4/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= +github.com/libp2p/go-libp2p-transport v0.0.5 h1:pV6+UlRxyDpASSGD+60vMvdifSCby6JkJDfi+yUMHac= +github.com/libp2p/go-libp2p-transport v0.0.5/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= github.com/libp2p/go-libp2p-transport-upgrader v0.0.3 h1:ZcXhGEMnv8f+ycqVl/9mnIe6uhUYy0aj/vEYQUoG7xU= github.com/libp2p/go-libp2p-transport-upgrader v0.0.3/go.mod h1:Ng1HzfMIopyYscMHNFmJqiMMcpgDlj0t+NyjVWW89ws= +github.com/libp2p/go-libp2p-transport-upgrader v0.0.4 h1:uGMOd14BL1oFlfb/cGfOxPjiTKBhzWV4aMjjoCF1Z1o= +github.com/libp2p/go-libp2p-transport-upgrader v0.0.4/go.mod h1:RGq+tupk+oj7PzL2kn/m1w6YXxcIAYJYeI90h6BGgUc= +github.com/libp2p/go-libp2p-yamux v0.1.2 h1:DgGItlrWi0j9y1OhRMC8qqL4zj2MEPWeKJTHb55R16Q= +github.com/libp2p/go-libp2p-yamux v0.1.2/go.mod h1:xUoV/RmYkg6BW/qGxA9XJyg+HzXFYkeXbnhjmnYzKp8= github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.4 h1:hx8HIuuwk34KePddrp2mM5ivgPkZ09JH4AvsALRbFUs= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= @@ -153,12 +163,18 @@ github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0 github.com/libp2p/go-reuseport-transport v0.0.2/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= github.com/libp2p/go-stream-muxer v0.0.1 h1:Ce6e2Pyu+b5MC1k3eeFtAax0pW4gc6MosYSLV05UeLw= github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= +github.com/libp2p/go-stream-muxer-multistream v0.1.1 h1:DhHqb4nu1fQv/vQKeLAaZGmhLsUA4SF77IdYJiWE1d4= +github.com/libp2p/go-stream-muxer-multistream v0.1.1/go.mod h1:zmGdfkQ1AzOECIAcccoL8L//laqawOsO03zX8Sa+eGw= github.com/libp2p/go-tcp-transport v0.0.3 h1:ZDMczeg+6uvvlxnShYFY0wGjQqStFhWAYT7769utxuM= github.com/libp2p/go-tcp-transport v0.0.3/go.mod h1:f11C2zvCaGDkE8aFPUKmuYZwd3pP6HI24LeLMWhJnkQ= +github.com/libp2p/go-tcp-transport v0.0.4 h1:2iRu994wCT/iEz62F+c60FUoSkijNEQ0q2Itc+79XlQ= +github.com/libp2p/go-tcp-transport v0.0.4/go.mod h1:+E8HvC8ezEVOxIo3V5vCK9l1y/19K427vCzQ+xHKH/o= github.com/libp2p/go-testutil v0.0.1 h1:Xg+O0G2HIMfHqBOBDcMS1iSZJ3GEcId4qOxCQvsGZHk= github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= github.com/libp2p/go-ws-transport v0.0.3 h1:29hkvsmdOhniNuJ5AzFICfY5SA5nvDO9Xi3FtcGqO+4= github.com/libp2p/go-ws-transport v0.0.3/go.mod h1:iU0kzfMcO4tBVIk3z+7srp1YG/RFLWTSuO4enpivw8g= +github.com/libp2p/go-yamux v1.2.1 h1:VumHkMhJ2iFk1lzAeoDRgekiZSylGc6NnAEihVdBCiw= +github.com/libp2p/go-yamux v1.2.1/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= From 192e96f186d130f4e3c1a15209e1ab2a06dad87c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 20:06:26 -0700 Subject: [PATCH 1235/3965] test: increase delay for backpressure CI is really slow. --- p2p/test/backpressure/backpressure_test.go | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/p2p/test/backpressure/backpressure_test.go b/p2p/test/backpressure/backpressure_test.go index 097c697bd0..3d32d30885 100644 --- a/p2p/test/backpressure/backpressure_test.go +++ b/p2p/test/backpressure/backpressure_test.go @@ -253,7 +253,11 @@ func TestStBackpressureStreamWrite(t *testing.T) { } log.Debugf("sender wrote %d bytes", n) - senderWrote <- n + select { + case senderWrote <- n: + default: + t.Error("sender wrote channel full") + } } } @@ -314,9 +318,10 @@ func TestStBackpressureStreamWrite(t *testing.T) { roundsTotal := 0 for roundsTotal < (2 << 20) { // let the sender fill its buffers, it will stop sending. - <-time.After(400 * time.Millisecond) + <-time.After(time.Second) b, _ := writeStats() testSenderWrote(0) + <-time.After(100 * time.Millisecond) testSenderWrote(0) // drain it all, wait again @@ -360,9 +365,10 @@ func TestStBackpressureStreamWrite(t *testing.T) { // and a couple rounds more for good measure ;) for i := 0; i < 3; i++ { // let the sender fill its buffers, it will stop sending. - <-time.After(400 * time.Millisecond) + <-time.After(time.Second) b, _ := writeStats() testSenderWrote(0) + <-time.After(100 * time.Millisecond) testSenderWrote(0) // drain it all, wait again From cad2c649ac551db66e3cc74c3db9a1854bcbbb76 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 22 May 2019 11:54:32 +0300 Subject: [PATCH 1236/3965] gomod: update go-libp2p-autonat --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 95f5f342d9..a61c61e929 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security v0.0.1 github.com/libp2p/go-conn-security-multistream v0.0.2 - github.com/libp2p/go-libp2p-autonat v0.0.5 + github.com/libp2p/go-libp2p-autonat v0.0.6 github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.7 github.com/libp2p/go-libp2p-crypto v0.0.2 diff --git a/go.sum b/go.sum index c950e489a4..945ad8aa26 100644 --- a/go.sum +++ b/go.sum @@ -88,8 +88,8 @@ github.com/libp2p/go-conn-security-multistream v0.0.2 h1:Ykz0lnNjxk+0SdslUmlLNyr github.com/libp2p/go-conn-security-multistream v0.0.2/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= -github.com/libp2p/go-libp2p-autonat v0.0.5 h1:knn3iuP/UZdR5XtgflR9l7Gle4tlRG9Y4frzshMl3KY= -github.com/libp2p/go-libp2p-autonat v0.0.5/go.mod h1:cKt+qOSnWAZp0dqIuUk62v0/QAPw0vnLuVZnmzkOXRk= +github.com/libp2p/go-libp2p-autonat v0.0.6 h1:OCStANLLpeyQeWFUuqZJ7aS9+Bx0/uoVb1PtLA9fGTQ= +github.com/libp2p/go-libp2p-autonat v0.0.6/go.mod h1:uZneLdOkZHro35xIhpbtTzLlgYturpu4J5+0cZK3MqE= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= github.com/libp2p/go-libp2p-circuit v0.0.7 h1:2oIJp4Qud9V8Boy+31nmbNBwH5PCJisGH4SwYTPSU50= From 58ac4c8110e465fa25b944fe3d75ae1c9caad833 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 22 May 2019 11:57:05 +0300 Subject: [PATCH 1237/3965] gomod: update go-ws-transport --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index a61c61e929..cc0c537699 100644 --- a/go.mod +++ b/go.mod @@ -38,7 +38,7 @@ require ( github.com/libp2p/go-stream-muxer-multistream v0.1.1 github.com/libp2p/go-tcp-transport v0.0.4 github.com/libp2p/go-testutil v0.0.1 - github.com/libp2p/go-ws-transport v0.0.3 + github.com/libp2p/go-ws-transport v0.0.4 github.com/miekg/dns v1.1.12 // indirect github.com/multiformats/go-multiaddr v0.0.4 github.com/multiformats/go-multiaddr-dns v0.0.2 diff --git a/go.sum b/go.sum index 945ad8aa26..c79a174ede 100644 --- a/go.sum +++ b/go.sum @@ -171,8 +171,8 @@ github.com/libp2p/go-tcp-transport v0.0.4 h1:2iRu994wCT/iEz62F+c60FUoSkijNEQ0q2I github.com/libp2p/go-tcp-transport v0.0.4/go.mod h1:+E8HvC8ezEVOxIo3V5vCK9l1y/19K427vCzQ+xHKH/o= github.com/libp2p/go-testutil v0.0.1 h1:Xg+O0G2HIMfHqBOBDcMS1iSZJ3GEcId4qOxCQvsGZHk= github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= -github.com/libp2p/go-ws-transport v0.0.3 h1:29hkvsmdOhniNuJ5AzFICfY5SA5nvDO9Xi3FtcGqO+4= -github.com/libp2p/go-ws-transport v0.0.3/go.mod h1:iU0kzfMcO4tBVIk3z+7srp1YG/RFLWTSuO4enpivw8g= +github.com/libp2p/go-ws-transport v0.0.4 h1:3wt9ed0gIUrne627XHvPMTwG4/AUpsLDy4TGQi2EyQ0= +github.com/libp2p/go-ws-transport v0.0.4/go.mod h1:X9wfEcm2LAJYMox9x2VHAMHAZZSQMFC9mIa/UF6OuZk= github.com/libp2p/go-yamux v1.2.1 h1:VumHkMhJ2iFk1lzAeoDRgekiZSylGc6NnAEihVdBCiw= github.com/libp2p/go-yamux v1.2.1/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= From 3310709eef8e001295f0f7691391efc70c2d937d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 8 May 2019 15:19:06 -0700 Subject: [PATCH 1238/3965] call Stream.Reset instead of Stream.Close May fix https://github.com/ipfs/go-ipfs/issues/6237 Basically, 1. We hang while closing a stream (because `Close` waits). 2. This blocks the connection manager because it assumes that close _doesn't_ wait. This may also fix a stream leak. --- .../internal/circuitv1-deprecated/conn.go | 35 ++++++++++++++++--- .../internal/circuitv1-deprecated/listen.go | 4 +-- .../internal/circuitv1-deprecated/relay.go | 4 +-- 3 files changed, 34 insertions(+), 9 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index df39f119f5..9c7f11e87b 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -3,6 +3,7 @@ package relay import ( "fmt" "net" + "time" inet "github.com/libp2p/go-libp2p-net" pstore "github.com/libp2p/go-libp2p-peerstore" @@ -11,7 +12,7 @@ import ( ) type Conn struct { - inet.Stream + stream inet.Stream remote pstore.PeerInfo } @@ -28,9 +29,33 @@ func (n *NetAddr) String() string { return fmt.Sprintf("relay[%s-%s]", n.Remote, n.Relay) } +func (c *Conn) Close() error { + return c.stream.Reset() +} + +func (c *Conn) Read(buf []byte) (int, error) { + return c.stream.Read(buf) +} + +func (c *Conn) Write(buf []byte) (int, error) { + return c.stream.Write(buf) +} + +func (c *Conn) SetDeadline(t time.Time) error { + return c.stream.SetDeadline(t) +} + +func (c *Conn) SetReadDeadline(t time.Time) error { + return c.stream.SetReadDeadline(t) +} + +func (c *Conn) SetWriteDeadline(t time.Time) error { + return c.stream.SetReadDeadline(t) +} + func (c *Conn) RemoteAddr() net.Addr { return &NetAddr{ - Relay: c.Conn().RemotePeer().Pretty(), + Relay: c.stream.Conn().RemotePeer().Pretty(), Remote: c.remote.ID.Pretty(), } } @@ -38,7 +63,7 @@ func (c *Conn) RemoteAddr() net.Addr { // TODO: is it okay to cast c.Conn().RemotePeer() into a multiaddr? might be "user input" func (c *Conn) RemoteMultiaddr() ma.Multiaddr { proto := ma.ProtocolWithCode(ma.P_P2P).Name - peerid := c.Conn().RemotePeer().Pretty() + peerid := c.stream.Conn().RemotePeer().Pretty() p2paddr := ma.StringCast(fmt.Sprintf("/%s/%s", proto, peerid)) circaddr := ma.Cast(ma.CodeToVarint(P_CIRCUIT)) @@ -46,11 +71,11 @@ func (c *Conn) RemoteMultiaddr() ma.Multiaddr { } func (c *Conn) LocalMultiaddr() ma.Multiaddr { - return c.Conn().LocalMultiaddr() + return c.stream.Conn().LocalMultiaddr() } func (c *Conn) LocalAddr() net.Addr { - na, err := manet.ToNetAddr(c.Conn().LocalMultiaddr()) + na, err := manet.ToNetAddr(c.stream.Conn().LocalMultiaddr()) if err != nil { log.Error("failed to convert local multiaddr to net addr:", err) return nil diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index 35661cb8c6..0de412020e 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -25,10 +25,10 @@ func (r *Relay) Listener() *RelayListener { func (l *RelayListener) Accept() (manet.Conn, error) { select { case c := <-l.incoming: - err := l.Relay().writeResponse(c.Stream, pb.CircuitRelay_SUCCESS) + err := l.Relay().writeResponse(c.stream, pb.CircuitRelay_SUCCESS) if err != nil { log.Debugf("error writing relay response: %s", err.Error()) - c.Stream.Reset() + c.stream.Reset() return nil, err } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 0afcf0467a..8ce0447dd2 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -180,7 +180,7 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore return nil, RelayError{msg.GetCode()} } - return &Conn{Stream: s, remote: dest}, nil + return &Conn{stream: s, remote: dest}, nil } func (r *Relay) Matches(addr ma.Multiaddr) bool { @@ -438,7 +438,7 @@ func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { } select { - case r.incoming <- &Conn{Stream: s, remote: src}: + case r.incoming <- &Conn{stream: s, remote: src}: case <-time.After(RelayAcceptTimeout): r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) } From 1beb4bb8f0195cf0319e6dc82db77b31771ddcb9 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 May 2019 09:31:25 -0700 Subject: [PATCH 1239/3965] fix tests for reset on close --- .../circuitv1-deprecated/relay_test.go | 118 ++++++++++++++---- 1 file changed, 96 insertions(+), 22 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 58f641f473..ff0bcad70d 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -4,7 +4,9 @@ import ( "bytes" "context" "fmt" + "io" "io/ioutil" + "net" "testing" "time" @@ -72,22 +74,38 @@ func TestBasicRelay(t *testing.T) { r3 := newTestRelay(t, ctx, hosts[2]) + var ( + conn1, conn2 net.Conn + done = make(chan struct{}) + ) + + defer func() { + <-done + if conn1 != nil { + conn1.Close() + } + if conn2 != nil { + conn2.Close() + } + }() + msg := []byte("relay works!") go func() { + defer close(done) list := r3.Listener() - con, err := list.Accept() + var err error + conn1, err = list.Accept() if err != nil { t.Error(err) return } - _, err = con.Write(msg) + _, err = conn1.Write(msg) if err != nil { t.Error(err) return } - con.Close() }() rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) @@ -96,18 +114,20 @@ func TestBasicRelay(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - con, err := r1.DialPeer(rctx, rinfo, dinfo) + var err error + conn2, err = r1.DialPeer(rctx, rinfo, dinfo) if err != nil { t.Fatal(err) } - data, err := ioutil.ReadAll(con) + result := make([]byte, len(msg)) + _, err = io.ReadFull(conn2, result) if err != nil { t.Fatal(err) } - if !bytes.Equal(data, msg) { - t.Fatal("message was incorrect:", string(data)) + if !bytes.Equal(result, msg) { + t.Fatal("message was incorrect:", string(result)) } } @@ -186,22 +206,38 @@ func TestBasicRelayDial(t *testing.T) { _ = newTestRelay(t, ctx, hosts[1], OptHop) r3 := newTestRelay(t, ctx, hosts[2]) + var ( + conn1, conn2 net.Conn + done = make(chan struct{}) + ) + + defer func() { + <-done + if conn1 != nil { + conn1.Close() + } + if conn2 != nil { + conn2.Close() + } + }() + msg := []byte("relay works!") go func() { + defer close(done) list := r3.Listener() - con, err := list.Accept() + var err error + conn1, err = list.Accept() if err != nil { t.Error(err) return } - _, err = con.Write(msg) + _, err = conn1.Write(msg) if err != nil { t.Error(err) return } - con.Close() }() addr := ma.StringCast(fmt.Sprintf("/ipfs/%s/p2p-circuit", hosts[1].ID().Pretty())) @@ -209,12 +245,14 @@ func TestBasicRelayDial(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - con, err := r1.Dial(rctx, addr, hosts[2].ID()) + var err error + conn2, err = r1.Dial(rctx, addr, hosts[2].ID()) if err != nil { t.Fatal(err) } - data, err := ioutil.ReadAll(con) + data := make([]byte, len(msg)) + _, err = io.ReadFull(conn2, data) if err != nil { t.Fatal(err) } @@ -241,22 +279,38 @@ func TestUnspecificRelayDial(t *testing.T) { time.Sleep(100 * time.Millisecond) + var ( + conn1, conn2 net.Conn + done = make(chan struct{}) + ) + + defer func() { + <-done + if conn1 != nil { + conn1.Close() + } + if conn2 != nil { + conn2.Close() + } + }() + msg := []byte("relay works!") go func() { + defer close(done) list := r3.Listener() - con, err := list.Accept() + var err error + conn1, err = list.Accept() if err != nil { t.Error(err) return } - _, err = con.Write(msg) + _, err = conn1.Write(msg) if err != nil { t.Error(err) return } - con.Close() }() addr := ma.StringCast(fmt.Sprintf("/p2p-circuit")) @@ -264,12 +318,14 @@ func TestUnspecificRelayDial(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - con, err := r1.Dial(rctx, addr, hosts[2].ID()) + var err error + conn2, err = r1.Dial(rctx, addr, hosts[2].ID()) if err != nil { t.Fatal(err) } - data, err := ioutil.ReadAll(con) + data := make([]byte, len(msg)) + _, err = io.ReadFull(conn2, data) if err != nil { t.Fatal(err) } @@ -368,22 +424,38 @@ func TestActiveRelay(t *testing.T) { r3 := newTestRelay(t, ctx, hosts[2]) + var ( + conn1, conn2 net.Conn + done = make(chan struct{}) + ) + + defer func() { + <-done + if conn1 != nil { + conn1.Close() + } + if conn2 != nil { + conn2.Close() + } + }() + msg := []byte("relay works!") go func() { + defer close(done) list := r3.Listener() - con, err := list.Accept() + var err error + conn2, err = list.Accept() if err != nil { t.Error(err) return } - _, err = con.Write(msg) + _, err = conn2.Write(msg) if err != nil { t.Error(err) return } - con.Close() }() rinfo := hosts[1].Peerstore().PeerInfo(hosts[1].ID()) @@ -392,12 +464,14 @@ func TestActiveRelay(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - con, err := r1.DialPeer(rctx, rinfo, dinfo) + var err error + conn2, err = r1.DialPeer(rctx, rinfo, dinfo) if err != nil { t.Fatal(err) } - data, err := ioutil.ReadAll(con) + data := make([]byte, len(msg)) + _, err = io.ReadFull(conn2, data) if err != nil { t.Fatal(err) } From 673ca29c3ab6e7c25573c9f0632f91e458236ca4 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 22 May 2019 18:14:49 +0300 Subject: [PATCH 1240/3965] fix bug in conn.SetWriteDeadline --- p2p/protocol/internal/circuitv1-deprecated/conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 9c7f11e87b..adf8287b0c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -50,7 +50,7 @@ func (c *Conn) SetReadDeadline(t time.Time) error { } func (c *Conn) SetWriteDeadline(t time.Time) error { - return c.stream.SetReadDeadline(t) + return c.stream.SetWriteDeadline(t) } func (c *Conn) RemoteAddr() net.Addr { From 8d6b0237ad5bb911da21c5ed12ead6bfeaaabc05 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 22 May 2019 16:48:30 +0300 Subject: [PATCH 1241/3965] tag hop relays for active stop connections --- p2p/protocol/internal/circuitv1-deprecated/conn.go | 10 ++++++++++ p2p/protocol/internal/circuitv1-deprecated/dial.go | 1 + p2p/protocol/internal/circuitv1-deprecated/listen.go | 1 + p2p/protocol/internal/circuitv1-deprecated/relay.go | 12 ++++++------ p2p/protocol/internal/circuitv1-deprecated/util.go | 12 ++++++++++++ 5 files changed, 30 insertions(+), 6 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index adf8287b0c..803f381655 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -14,6 +14,7 @@ import ( type Conn struct { stream inet.Stream remote pstore.PeerInfo + relay *Relay } type NetAddr struct { @@ -30,6 +31,7 @@ func (n *NetAddr) String() string { } func (c *Conn) Close() error { + c.untagHop() return c.stream.Reset() } @@ -60,6 +62,14 @@ func (c *Conn) RemoteAddr() net.Addr { } } +func (c *Conn) tagHop() { + c.relay.host.ConnManager().UpsertTag(c.stream.Conn().RemotePeer(), "relay-hop-stream", incrementTag) +} + +func (c *Conn) untagHop() { + c.relay.host.ConnManager().UpsertTag(c.stream.Conn().RemotePeer(), "relay-hop-stream", decrementTag) +} + // TODO: is it okay to cast c.Conn().RemotePeer() into a multiaddr? might be "user input" func (c *Conn) RemoteMultiaddr() ma.Multiaddr { proto := ma.ProtocolWithCode(ma.P_P2P).Name diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index ecfd23fe6b..9213a48c20 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -16,6 +16,7 @@ func (d *RelayTransport) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (t if err != nil { return nil, err } + c.tagHop() return d.upgrader.UpgradeOutbound(ctx, d, c, p) } diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index 0de412020e..b00015b565 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -35,6 +35,7 @@ func (l *RelayListener) Accept() (manet.Conn, error) { // TODO: Pretty print. log.Infof("accepted relay connection: %q", c) + c.tagHop() return c, nil case <-l.ctx.Done(): return nil, l.ctx.Err() diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 8ce0447dd2..c77cd59e3b 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -118,14 +118,14 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. func (r *Relay) addLiveHop(from, to peer.ID) { atomic.AddInt32(&r.liveHopCount, 1) - r.host.ConnManager().UpsertTag(from, "relay-hop-stream", func(v int) int { return v + 1 }) - r.host.ConnManager().UpsertTag(to, "relay-hop-stream", func(v int) int { return v + 1 }) + r.host.ConnManager().UpsertTag(from, "relay-hop-stream", incrementTag) + r.host.ConnManager().UpsertTag(to, "relay-hop-stream", incrementTag) } func (r *Relay) rmLiveHop(from, to peer.ID) { atomic.AddInt32(&r.liveHopCount, -1) - r.host.ConnManager().UpsertTag(from, "relay-hop-stream", func(v int) int { return v - 1 }) - r.host.ConnManager().UpsertTag(to, "relay-hop-stream", func(v int) int { return v - 1 }) + r.host.ConnManager().UpsertTag(from, "relay-hop-stream", decrementTag) + r.host.ConnManager().UpsertTag(to, "relay-hop-stream", decrementTag) } @@ -180,7 +180,7 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore return nil, RelayError{msg.GetCode()} } - return &Conn{stream: s, remote: dest}, nil + return &Conn{stream: s, remote: dest, relay: r}, nil } func (r *Relay) Matches(addr ma.Multiaddr) bool { @@ -438,7 +438,7 @@ func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { } select { - case r.incoming <- &Conn{stream: s, remote: src}: + case r.incoming <- &Conn{stream: s, remote: src, relay: r}: case <-time.After(RelayAcceptTimeout): r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) } diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index 7d56b89503..5eff3f1bd4 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -49,6 +49,18 @@ func peerInfoToPeer(pi pstore.PeerInfo) *pb.CircuitRelay_Peer { return p } +func incrementTag(v int) int { + return v + 1 +} + +func decrementTag(v int) int { + if v > 0 { + return v - 1 + } else { + return v + } +} + type delimitedReader struct { r io.Reader buf []byte From 36f27a0729d3d6a2665398f4a795ee5d7dfda750 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 22 May 2019 19:55:24 +0300 Subject: [PATCH 1242/3965] add comments for tagging logic --- p2p/protocol/internal/circuitv1-deprecated/conn.go | 6 ++++++ p2p/protocol/internal/circuitv1-deprecated/relay.go | 5 +++++ 2 files changed, 11 insertions(+) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 803f381655..f6e313c5e7 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -62,10 +62,16 @@ func (c *Conn) RemoteAddr() net.Addr { } } +// Increment the underlying relay connection tag by 1, thus increasing its protection from +// connection pruning. This ensures that connections to relays are not accidentally closed, +// by the connection manager, taking with them all the relayed connections (that may themselves +// be protected). func (c *Conn) tagHop() { c.relay.host.ConnManager().UpsertTag(c.stream.Conn().RemotePeer(), "relay-hop-stream", incrementTag) } +// Decrement the underlying relay connection tag by 1; this is performed when we close the +// relayed connection. func (c *Conn) untagHop() { c.relay.host.ConnManager().UpsertTag(c.stream.Conn().RemotePeer(), "relay-hop-stream", decrementTag) } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index c77cd59e3b..411339ae9e 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -116,12 +116,17 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. return r, nil } +// Increment the live hop count and increment the connection manager tags by 1 for the two +// sides of the hop stream. This ensures that connections with many hop streams will be protected +// from pruning, thus minimizing disruption from connection trimming in a relay node. func (r *Relay) addLiveHop(from, to peer.ID) { atomic.AddInt32(&r.liveHopCount, 1) r.host.ConnManager().UpsertTag(from, "relay-hop-stream", incrementTag) r.host.ConnManager().UpsertTag(to, "relay-hop-stream", incrementTag) } +// Decrement the live hpo count and decrement the connection manager tags for the two sides +// of the hop stream. func (r *Relay) rmLiveHop(from, to peer.ID) { atomic.AddInt32(&r.liveHopCount, -1) r.host.ConnManager().UpsertTag(from, "relay-hop-stream", decrementTag) From 7b2149dd786e8063a2f6286992419bb072ac99b4 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 22 May 2019 19:57:21 +0300 Subject: [PATCH 1243/3965] store the host instead of the Relay instance in Conn --- p2p/protocol/internal/circuitv1-deprecated/conn.go | 7 ++++--- p2p/protocol/internal/circuitv1-deprecated/relay.go | 4 ++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index f6e313c5e7..33b9d94c87 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -5,6 +5,7 @@ import ( "net" "time" + host "github.com/libp2p/go-libp2p-host" inet "github.com/libp2p/go-libp2p-net" pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" @@ -14,7 +15,7 @@ import ( type Conn struct { stream inet.Stream remote pstore.PeerInfo - relay *Relay + host host.Host } type NetAddr struct { @@ -67,13 +68,13 @@ func (c *Conn) RemoteAddr() net.Addr { // by the connection manager, taking with them all the relayed connections (that may themselves // be protected). func (c *Conn) tagHop() { - c.relay.host.ConnManager().UpsertTag(c.stream.Conn().RemotePeer(), "relay-hop-stream", incrementTag) + c.host.ConnManager().UpsertTag(c.stream.Conn().RemotePeer(), "relay-hop-stream", incrementTag) } // Decrement the underlying relay connection tag by 1; this is performed when we close the // relayed connection. func (c *Conn) untagHop() { - c.relay.host.ConnManager().UpsertTag(c.stream.Conn().RemotePeer(), "relay-hop-stream", decrementTag) + c.host.ConnManager().UpsertTag(c.stream.Conn().RemotePeer(), "relay-hop-stream", decrementTag) } // TODO: is it okay to cast c.Conn().RemotePeer() into a multiaddr? might be "user input" diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 411339ae9e..f68debcfdb 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -185,7 +185,7 @@ func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore return nil, RelayError{msg.GetCode()} } - return &Conn{stream: s, remote: dest, relay: r}, nil + return &Conn{stream: s, remote: dest, host: r.host}, nil } func (r *Relay) Matches(addr ma.Multiaddr) bool { @@ -443,7 +443,7 @@ func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { } select { - case r.incoming <- &Conn{stream: s, remote: src, relay: r}: + case r.incoming <- &Conn{stream: s, remote: src, host: r.host}: case <-time.After(RelayAcceptTimeout): r.handleError(s, pb.CircuitRelay_STOP_RELAY_REFUSED) } From aacc91fe0c12f7ac68f8312746f3f29f11b8c380 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 22 May 2019 18:28:04 +0100 Subject: [PATCH 1244/3965] initial commit. Consolidate test utilities, harnesses and suites (#1) --- p2p/transport/testsuite/stream_suite.go | 479 +++++++++++++++++++++ p2p/transport/testsuite/transport_suite.go | 287 ++++++++++++ p2p/transport/testsuite/utils_suite.go | 45 ++ 3 files changed, 811 insertions(+) create mode 100644 p2p/transport/testsuite/stream_suite.go create mode 100644 p2p/transport/testsuite/transport_suite.go create mode 100644 p2p/transport/testsuite/utils_suite.go diff --git a/p2p/transport/testsuite/stream_suite.go b/p2p/transport/testsuite/stream_suite.go new file mode 100644 index 0000000000..6f978e57be --- /dev/null +++ b/p2p/transport/testsuite/stream_suite.go @@ -0,0 +1,479 @@ +package ttransport + +import ( + "bytes" + "context" + "fmt" + "io" + "io/ioutil" + "os" + "runtime/debug" + "strconv" + "sync" + "testing" + "time" + + crand "crypto/rand" + mrand "math/rand" + + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" + "github.com/libp2p/go-libp2p-testing/race" + + ma "github.com/multiformats/go-multiaddr" +) + +// VerboseDebugging can be set to true to enable verbose debug logging in the +// stream stress tests. +var VerboseDebugging = false + +var randomness []byte + +var StressTestTimeout = 1 * time.Minute + +func init() { + // read 1MB of randomness + randomness = make([]byte, 1<<20) + if _, err := crand.Read(randomness); err != nil { + panic(err) + } + + if timeout := os.Getenv("TEST_STRESS_TIMEOUT_MS"); timeout != "" { + if v, err := strconv.ParseInt(timeout, 10, 32); err == nil { + StressTestTimeout = time.Duration(v) * time.Millisecond + } + } +} + +type Options struct { + connNum int + streamNum int + msgNum int + msgMin int + msgMax int +} + +func fullClose(t *testing.T, s mux.MuxedStream) { + if err := s.Close(); err != nil { + t.Error(err) + s.Reset() + return + } + b, err := ioutil.ReadAll(s) + if err != nil { + t.Error(err) + } + if len(b) != 0 { + t.Error("expected to be done reading") + } +} + +func randBuf(size int) []byte { + n := len(randomness) - size + if size < 1 { + panic(fmt.Errorf("requested too large buffer (%d). max is %d", size, len(randomness))) + } + + start := mrand.Intn(n) + return randomness[start : start+size] +} + +func checkErr(t *testing.T, err error) { + t.Helper() + if err != nil { + debug.PrintStack() + // TODO: not safe to call in parallel + t.Fatal(err) + } +} + +func debugLog(t *testing.T, s string, args ...interface{}) { + if VerboseDebugging { + t.Logf(s, args...) + } +} + +func echoStream(t *testing.T, s mux.MuxedStream) { + defer s.Close() + // echo everything + var err error + if VerboseDebugging { + t.Logf("accepted stream") + _, err = io.Copy(&logWriter{t, s}, s) + t.Log("closing stream") + } else { + _, err = io.Copy(s, s) // echo everything + } + if err != nil { + t.Error(err) + } +} + +type logWriter struct { + t *testing.T + W io.Writer +} + +func (lw *logWriter) Write(buf []byte) (int, error) { + lw.t.Logf("logwriter: writing %d bytes", len(buf)) + return lw.W.Write(buf) +} + +func goServe(t *testing.T, l transport.Listener) (done func()) { + closed := make(chan struct{}, 1) + + go func() { + for { + c, err := l.Accept() + if err != nil { + select { + case <-closed: + return // closed naturally. + default: + checkErr(t, err) + } + } + + debugLog(t, "accepted connection") + go func() { + for { + str, err := c.AcceptStream() + if err != nil { + break + } + go echoStream(t, str) + } + }() + } + }() + + return func() { + closed <- struct{}{} + } +} + +func SubtestStress(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID, opt Options) { + msgsize := 1 << 11 + errs := make(chan error, 0) // dont block anything. + + rateLimitN := 5000 // max of 5k funcs, because -race has 8k max. + rateLimitChan := make(chan struct{}, rateLimitN) + for i := 0; i < rateLimitN; i++ { + rateLimitChan <- struct{}{} + } + + rateLimit := func(f func()) { + <-rateLimitChan + f() + rateLimitChan <- struct{}{} + } + + writeStream := func(s mux.MuxedStream, bufs chan<- []byte) { + debugLog(t, "writeStream %p, %d msgNum", s, opt.msgNum) + + for i := 0; i < opt.msgNum; i++ { + buf := randBuf(msgsize) + bufs <- buf + debugLog(t, "%p writing %d bytes (message %d/%d #%x)", s, len(buf), i, opt.msgNum, buf[:3]) + if _, err := s.Write(buf); err != nil { + errs <- fmt.Errorf("s.Write(buf): %s", err) + continue + } + } + } + + readStream := func(s mux.MuxedStream, bufs <-chan []byte) { + debugLog(t, "readStream %p, %d msgNum", s, opt.msgNum) + + buf2 := make([]byte, msgsize) + i := 0 + for buf1 := range bufs { + i++ + debugLog(t, "%p reading %d bytes (message %d/%d #%x)", s, len(buf1), i-1, opt.msgNum, buf1[:3]) + + if _, err := io.ReadFull(s, buf2); err != nil { + errs <- fmt.Errorf("io.ReadFull(s, buf2): %s", err) + debugLog(t, "%p failed to read %d bytes (message %d/%d #%x)", s, len(buf1), i-1, opt.msgNum, buf1[:3]) + continue + } + if !bytes.Equal(buf1, buf2) { + errs <- fmt.Errorf("buffers not equal (%x != %x)", buf1[:3], buf2[:3]) + } + } + } + + openStreamAndRW := func(c mux.MuxedConn) { + debugLog(t, "openStreamAndRW %p, %d opt.msgNum", c, opt.msgNum) + + s, err := c.OpenStream() + if err != nil { + errs <- fmt.Errorf("Failed to create NewStream: %s", err) + return + } + + bufs := make(chan []byte, opt.msgNum) + go func() { + writeStream(s, bufs) + close(bufs) + }() + + readStream(s, bufs) + fullClose(t, s) + } + + openConnAndRW := func() { + debugLog(t, "openConnAndRW") + + l, err := ta.Listen(maddr) + checkErr(t, err) + + done := goServe(t, l) + defer done() + + c, err := tb.Dial(context.Background(), l.Multiaddr(), peerA) + checkErr(t, err) + + // serve the outgoing conn, because some muxers assume + // that we _always_ call serve. (this is an error?) + go func() { + debugLog(t, "serving connection") + for { + str, err := c.AcceptStream() + if err != nil { + break + } + go echoStream(t, str) + } + }() + + var wg sync.WaitGroup + for i := 0; i < opt.streamNum; i++ { + wg.Add(1) + go rateLimit(func() { + defer wg.Done() + openStreamAndRW(c) + }) + } + wg.Wait() + c.Close() + } + + openConnsAndRW := func() { + debugLog(t, "openConnsAndRW, %d conns", opt.connNum) + + var wg sync.WaitGroup + for i := 0; i < opt.connNum; i++ { + wg.Add(1) + go rateLimit(func() { + defer wg.Done() + openConnAndRW() + }) + } + wg.Wait() + } + + go func() { + openConnsAndRW() + close(errs) // done + }() + + for err := range errs { + t.Error(err) + } + +} + +func SubtestStreamOpenStress(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + l, err := ta.Listen(maddr) + checkErr(t, err) + defer l.Close() + + count := 10000 + workers := 5 + + if race.WithRace() { + // the race detector can only deal with 8128 simultaneous goroutines, so let's make sure we don't go overboard. + count = 1000 + } + + var ( + connA, connB transport.CapableConn + ) + + accepted := make(chan error, 1) + go func() { + var err error + connA, err = l.Accept() + accepted <- err + }() + connB, err = tb.Dial(context.Background(), l.Multiaddr(), peerA) + checkErr(t, err) + checkErr(t, <-accepted) + + defer func() { + if connA != nil { + connA.Close() + } + if connB != nil { + connB.Close() + } + }() + + var wg sync.WaitGroup + wg.Add(1) + go func() { + defer wg.Done() + for j := 0; j < workers; j++ { + wg.Add(1) + go func() { + defer wg.Done() + for i := 0; i < count; i++ { + s, err := connA.OpenStream() + if err != nil { + t.Error(err) + return + } + wg.Add(1) + go func() { + defer wg.Done() + fullClose(t, s) + }() + } + }() + } + }() + + wg.Add(1) + go func() { + defer wg.Done() + for i := 0; i < count*workers; i++ { + str, err := connB.AcceptStream() + if err != nil { + break + } + wg.Add(1) + go func() { + defer wg.Done() + fullClose(t, str) + }() + } + }() + + timeout := time.After(StressTestTimeout) + done := make(chan struct{}) + + go func() { + wg.Wait() + close(done) + }() + + select { + case <-timeout: + t.Fatal("timed out receiving streams") + case <-done: + } +} + +func SubtestStreamReset(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + l, err := ta.Listen(maddr) + checkErr(t, err) + + done := make(chan struct{}, 2) + go func() { + muxa, err := l.Accept() + checkErr(t, err) + + s, err := muxa.OpenStream() + if err != nil { + panic(err) + } + + // Some transports won't open the stream until we write. That's + // fine. + s.Write([]byte("foo")) + + time.Sleep(time.Millisecond * 50) + + _, err = s.Write([]byte("bar")) + if err == nil { + t.Error("should have failed to write") + } + + s.Close() + done <- struct{}{} + }() + + muxb, err := tb.Dial(context.Background(), l.Multiaddr(), peerA) + checkErr(t, err) + + go func() { + str, err := muxb.AcceptStream() + checkErr(t, err) + str.Reset() + done <- struct{}{} + }() + + <-done + <-done +} + +func SubtestStress1Conn1Stream1Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + SubtestStress(t, ta, tb, maddr, peerA, Options{ + connNum: 1, + streamNum: 1, + msgNum: 1, + msgMax: 100, + msgMin: 100, + }) +} + +func SubtestStress1Conn1Stream100Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + SubtestStress(t, ta, tb, maddr, peerA, Options{ + connNum: 1, + streamNum: 1, + msgNum: 100, + msgMax: 100, + msgMin: 100, + }) +} + +func SubtestStress1Conn100Stream100Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + SubtestStress(t, ta, tb, maddr, peerA, Options{ + connNum: 1, + streamNum: 100, + msgNum: 100, + msgMax: 100, + msgMin: 100, + }) +} + +func SubtestStress50Conn10Stream50Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + SubtestStress(t, ta, tb, maddr, peerA, Options{ + connNum: 50, + streamNum: 10, + msgNum: 50, + msgMax: 100, + msgMin: 100, + }) +} + +func SubtestStress1Conn1000Stream10Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + SubtestStress(t, ta, tb, maddr, peerA, Options{ + connNum: 1, + streamNum: 1000, + msgNum: 10, + msgMax: 100, + msgMin: 100, + }) +} + +func SubtestStress1Conn100Stream100Msg10MB(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + SubtestStress(t, ta, tb, maddr, peerA, Options{ + connNum: 1, + streamNum: 100, + msgNum: 100, + msgMax: 10000, + msgMin: 1000, + }) +} diff --git a/p2p/transport/testsuite/transport_suite.go b/p2p/transport/testsuite/transport_suite.go new file mode 100644 index 0000000000..213cfe936c --- /dev/null +++ b/p2p/transport/testsuite/transport_suite.go @@ -0,0 +1,287 @@ +package ttransport + +import ( + "bytes" + "context" + "fmt" + "io/ioutil" + "sync" + "testing" + + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" + + ma "github.com/multiformats/go-multiaddr" +) + +var testData = []byte("this is some test data") + +type streamAndConn struct { + stream mux.MuxedStream + conn transport.CapableConn +} + +func SubtestProtocols(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + rawIPAddr, _ := ma.NewMultiaddr("/ip4/1.2.3.4") + if ta.CanDial(rawIPAddr) || tb.CanDial(rawIPAddr) { + t.Error("nothing should be able to dial raw IP") + } + + tprotos := make(map[int]bool) + for _, p := range ta.Protocols() { + tprotos[p] = true + } + + if !ta.Proxy() { + protos := maddr.Protocols() + proto := protos[len(protos)-1] + if !tprotos[proto.Code] { + t.Errorf("transport should have reported that it supports protocol '%s' (%d)", proto.Name, proto.Code) + } + } else { + found := false + for _, proto := range maddr.Protocols() { + if tprotos[proto.Code] { + found = true + break + } + } + if !found { + t.Errorf("didn't find any matching proxy protocols in maddr: %s", maddr) + } + } +} + +func SubtestBasic(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + list, err := ta.Listen(maddr) + if err != nil { + t.Fatal(err) + } + defer list.Close() + + var ( + connA, connB transport.CapableConn + done = make(chan struct{}) + ) + defer func() { + <-done + if connA != nil { + connA.Close() + } + if connB != nil { + connB.Close() + } + }() + + go func() { + defer close(done) + var err error + connB, err = list.Accept() + if err != nil { + t.Error(err) + return + } + s, err := connB.AcceptStream() + if err != nil { + t.Error(err) + return + } + + buf, err := ioutil.ReadAll(s) + if err != nil { + t.Error(err) + return + } + + if !bytes.Equal(testData, buf) { + t.Errorf("expected %s, got %s", testData, buf) + } + + n, err := s.Write(testData) + if err != nil { + t.Error(err) + return + } + if n != len(testData) { + t.Error(err) + return + } + + err = s.Close() + if err != nil { + t.Error(err) + } + }() + + if !tb.CanDial(list.Multiaddr()) { + t.Error("CanDial should have returned true") + } + + connA, err = tb.Dial(ctx, list.Multiaddr(), peerA) + if err != nil { + t.Fatal(err) + } + + s, err := connA.OpenStream() + if err != nil { + t.Fatal(err) + } + + n, err := s.Write(testData) + if err != nil { + t.Fatal(err) + return + } + + if n != len(testData) { + t.Fatalf("failed to write enough data (a->b)") + return + } + err = s.Close() + if err != nil { + t.Fatal(err) + return + } + + buf, err := ioutil.ReadAll(s) + if err != nil { + t.Fatal(err) + return + } + if !bytes.Equal(testData, buf) { + t.Errorf("expected %s, got %s", testData, buf) + } +} + +func SubtestPingPong(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + streams := 100 + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + list, err := ta.Listen(maddr) + if err != nil { + t.Fatal(err) + } + defer list.Close() + + var wg sync.WaitGroup + + wg.Add(1) + go func() { + defer wg.Done() + c, err := list.Accept() + if err != nil { + t.Error(err) + return + } + defer c.Close() + + var sWg sync.WaitGroup + for i := 0; i < streams; i++ { + s, err := c.AcceptStream() + if err != nil { + t.Error(err) + return + } + + sWg.Add(1) + go func() { + defer sWg.Done() + + data, err := ioutil.ReadAll(s) + if err != nil { + s.Reset() + t.Error(err) + return + } + if !bytes.HasPrefix(data, testData) { + t.Errorf("expected %q to have prefix %q", string(data), string(testData)) + } + + n, err := s.Write(data) + if err != nil { + s.Reset() + t.Error(err) + return + } + + if n != len(data) { + s.Reset() + t.Error(err) + return + } + s.Close() + }() + } + sWg.Wait() + }() + + if !tb.CanDial(list.Multiaddr()) { + t.Error("CanDial should have returned true") + } + + c, err := tb.Dial(ctx, list.Multiaddr(), peerA) + if err != nil { + t.Fatal(err) + } + defer c.Close() + + for i := 0; i < streams; i++ { + s, err := c.OpenStream() + if err != nil { + t.Error(err) + continue + } + + wg.Add(1) + go func(i int) { + defer wg.Done() + data := []byte(fmt.Sprintf("%s - %d", testData, i)) + n, err := s.Write(data) + if err != nil { + s.Reset() + t.Error(err) + return + } + + if n != len(data) { + s.Reset() + t.Error("failed to write enough data (a->b)") + return + } + s.Close() + + ret, err := ioutil.ReadAll(s) + if err != nil { + s.Reset() + t.Error(err) + return + } + if !bytes.Equal(data, ret) { + t.Errorf("expected %q, got %q", string(data), string(ret)) + } + }(i) + } + wg.Wait() +} + +func SubtestCancel(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { + list, err := ta.Listen(maddr) + if err != nil { + t.Fatal(err) + } + defer list.Close() + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + c, err := tb.Dial(ctx, list.Multiaddr(), peerA) + if err == nil { + c.Close() + t.Fatal("dial should have failed") + } +} diff --git a/p2p/transport/testsuite/utils_suite.go b/p2p/transport/testsuite/utils_suite.go new file mode 100644 index 0000000000..1d520ff262 --- /dev/null +++ b/p2p/transport/testsuite/utils_suite.go @@ -0,0 +1,45 @@ +package ttransport + +import ( + "reflect" + "runtime" + "testing" + + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" + + ma "github.com/multiformats/go-multiaddr" +) + +var Subtests = []func(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID){ + SubtestProtocols, + SubtestBasic, + SubtestCancel, + SubtestPingPong, + + // Stolen from the stream muxer test suite. + SubtestStress1Conn1Stream1Msg, + SubtestStress1Conn1Stream100Msg, + SubtestStress1Conn100Stream100Msg, + SubtestStress50Conn10Stream50Msg, + SubtestStress1Conn1000Stream10Msg, + SubtestStress1Conn100Stream100Msg10MB, + SubtestStreamOpenStress, + SubtestStreamReset, +} + +func getFunctionName(i interface{}) string { + return runtime.FuncForPC(reflect.ValueOf(i).Pointer()).Name() +} + +func SubtestTransport(t *testing.T, ta, tb transport.Transport, addr string, peerA peer.ID) { + maddr, err := ma.NewMultiaddr(addr) + if err != nil { + t.Fatal(err) + } + for _, f := range Subtests { + t.Run(getFunctionName(f), func(t *testing.T) { + f(t, ta, tb, maddr, peerA) + }) + } +} From 0a9354a85723f689a084a2803e9af7182320dd7a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 22 May 2019 18:31:11 +0100 Subject: [PATCH 1245/3965] Absorb go-libp2p abstractions and core types into this module (#1) --- core/alias.go | 50 +++ core/connmgr/connmgr.go | 78 ++++ core/connmgr/null.go | 23 ++ core/crypto/bench_test.go | 84 +++++ core/crypto/ecdsa.go | 186 ++++++++++ core/crypto/ecdsa_test.go | 96 +++++ core/crypto/ed25519.go | 155 ++++++++ core/crypto/ed25519_test.go | 220 +++++++++++ core/crypto/fixture_test.go | 132 +++++++ core/crypto/key.go | 351 ++++++++++++++++++ core/crypto/key_test.go | 147 ++++++++ core/crypto/openssl_common.go | 98 +++++ core/crypto/pb/Makefile | 11 + core/crypto/pb/crypto.pb.go | 643 +++++++++++++++++++++++++++++++++ core/crypto/pb/crypto.proto | 20 + core/crypto/rsa_common.go | 10 + core/crypto/rsa_go.go | 125 +++++++ core/crypto/rsa_openssl.go | 62 ++++ core/crypto/rsa_test.go | 102 ++++++ core/crypto/secp256k1.go | 125 +++++++ core/crypto/secp256k1_test.go | 96 +++++ core/crypto/test_data/0.priv | Bin 0 -> 1198 bytes core/crypto/test_data/0.pub | Bin 0 -> 299 bytes core/crypto/test_data/0.sig | Bin 0 -> 256 bytes core/crypto/test_data/2.priv | 1 + core/crypto/test_data/2.pub | 1 + core/crypto/test_data/2.sig | 1 + core/crypto/test_data/3.priv | Bin 0 -> 125 bytes core/crypto/test_data/3.pub | Bin 0 -> 95 bytes core/crypto/test_data/3.sig | Bin 0 -> 72 bytes core/discovery/discovery.go | 27 ++ core/discovery/options.go | 41 +++ core/helpers/match.go | 42 +++ core/helpers/match_test.go | 33 ++ core/helpers/stream.go | 56 +++ core/helpers/stream_test.go | 151 ++++++++ core/host/helpers.go | 11 + core/host/host.go | 71 ++++ core/metrics/bandwidth.go | 153 ++++++++ core/metrics/bandwidth_test.go | 158 ++++++++ core/metrics/reporter.go | 31 ++ core/mux/mux.go | 70 ++++ core/network/conn.go | 58 +++ core/network/context.go | 48 +++ core/network/context_test.go | 40 ++ core/network/errors.go | 10 + core/network/network.go | 138 +++++++ core/network/notifee.go | 92 +++++ core/network/notifee_test.go | 123 +++++++ core/network/stream.go | 24 ++ core/peer/addrinfo.go | 80 ++++ core/peer/addrinfo_serde.go | 38 ++ core/peer/peer.go | 186 ++++++++++ core/peer/peer_serde.go | 75 ++++ core/peer/peer_serde_test.go | 83 +++++ core/peer/peer_test.go | 244 +++++++++++++ core/peer/set.go | 71 ++++ core/peerstore/peerstore.go | 162 +++++++++ core/pnet/env.go | 19 + core/pnet/error.go | 34 ++ core/pnet/error_test.go | 20 + core/pnet/protector.go | 15 + core/protocol/id.go | 9 + core/protocol/switch.go | 81 +++++ core/routing/options.go | 50 +++ core/routing/query.go | 86 +++++ core/routing/query_serde.go | 40 ++ core/routing/query_test.go | 44 +++ core/routing/routing.go | 125 +++++++ core/sec/insecure/insecure.go | 90 +++++ core/sec/security.go | 26 ++ core/transport/transport.go | 113 ++++++ 72 files changed, 5885 insertions(+) create mode 100644 core/alias.go create mode 100644 core/connmgr/connmgr.go create mode 100644 core/connmgr/null.go create mode 100644 core/crypto/bench_test.go create mode 100644 core/crypto/ecdsa.go create mode 100644 core/crypto/ecdsa_test.go create mode 100644 core/crypto/ed25519.go create mode 100644 core/crypto/ed25519_test.go create mode 100644 core/crypto/fixture_test.go create mode 100644 core/crypto/key.go create mode 100644 core/crypto/key_test.go create mode 100644 core/crypto/openssl_common.go create mode 100644 core/crypto/pb/Makefile create mode 100644 core/crypto/pb/crypto.pb.go create mode 100644 core/crypto/pb/crypto.proto create mode 100644 core/crypto/rsa_common.go create mode 100644 core/crypto/rsa_go.go create mode 100644 core/crypto/rsa_openssl.go create mode 100644 core/crypto/rsa_test.go create mode 100644 core/crypto/secp256k1.go create mode 100644 core/crypto/secp256k1_test.go create mode 100644 core/crypto/test_data/0.priv create mode 100644 core/crypto/test_data/0.pub create mode 100644 core/crypto/test_data/0.sig create mode 100644 core/crypto/test_data/2.priv create mode 100644 core/crypto/test_data/2.pub create mode 100644 core/crypto/test_data/2.sig create mode 100644 core/crypto/test_data/3.priv create mode 100644 core/crypto/test_data/3.pub create mode 100644 core/crypto/test_data/3.sig create mode 100644 core/discovery/discovery.go create mode 100644 core/discovery/options.go create mode 100644 core/helpers/match.go create mode 100644 core/helpers/match_test.go create mode 100644 core/helpers/stream.go create mode 100644 core/helpers/stream_test.go create mode 100644 core/host/helpers.go create mode 100644 core/host/host.go create mode 100644 core/metrics/bandwidth.go create mode 100644 core/metrics/bandwidth_test.go create mode 100644 core/metrics/reporter.go create mode 100644 core/mux/mux.go create mode 100644 core/network/conn.go create mode 100644 core/network/context.go create mode 100644 core/network/context_test.go create mode 100644 core/network/errors.go create mode 100644 core/network/network.go create mode 100644 core/network/notifee.go create mode 100644 core/network/notifee_test.go create mode 100644 core/network/stream.go create mode 100644 core/peer/addrinfo.go create mode 100644 core/peer/addrinfo_serde.go create mode 100644 core/peer/peer.go create mode 100644 core/peer/peer_serde.go create mode 100644 core/peer/peer_serde_test.go create mode 100644 core/peer/peer_test.go create mode 100644 core/peer/set.go create mode 100644 core/peerstore/peerstore.go create mode 100644 core/pnet/env.go create mode 100644 core/pnet/error.go create mode 100644 core/pnet/error_test.go create mode 100644 core/pnet/protector.go create mode 100644 core/protocol/id.go create mode 100644 core/protocol/switch.go create mode 100644 core/routing/options.go create mode 100644 core/routing/query.go create mode 100644 core/routing/query_serde.go create mode 100644 core/routing/query_test.go create mode 100644 core/routing/routing.go create mode 100644 core/sec/insecure/insecure.go create mode 100644 core/sec/security.go create mode 100644 core/transport/transport.go diff --git a/core/alias.go b/core/alias.go new file mode 100644 index 0000000000..1a19343867 --- /dev/null +++ b/core/alias.go @@ -0,0 +1,50 @@ +// Package core provides convenient access to foundational, central go-libp2p primitives via type aliases. +package core + +import ( + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/protocol" + "github.com/multiformats/go-multiaddr" +) + +// Multiaddr aliases the Multiaddr type from github.com/multiformats/go-multiaddr. +// +// Refer to the docs on that type for more info. +type Multiaddr = multiaddr.Multiaddr + +// PeerID aliases peer.ID. +// +// Refer to the docs on that type for more info. +type PeerID = peer.ID + +// PeerID aliases protocol.ID. +// +// Refer to the docs on that type for more info. +type ProtocolID = protocol.ID + +// PeerAddrInfo aliases peer.AddrInfo. +// +// Refer to the docs on that type for more info. +type PeerAddrInfo = peer.AddrInfo + +// Host aliases host.Host. +// +// Refer to the docs on that type for more info. +type Host = host.Host + +// Network aliases network.Network. +// +// Refer to the docs on that type for more info. +type Network = network.Network + +// Conn aliases network.Conn. +// +// Refer to the docs on that type for more info. +type Conn = network.Conn + +// Stream aliases network.Stream. +// +// Refer to the docs on that type for more info. +type Stream = network.Stream diff --git a/core/connmgr/connmgr.go b/core/connmgr/connmgr.go new file mode 100644 index 0000000000..98baf8b9a2 --- /dev/null +++ b/core/connmgr/connmgr.go @@ -0,0 +1,78 @@ +// Package connmgr provides connection tracking and management interfaces for libp2p. +// +// The ConnManager interface exported from this package allows libp2p to enforce an +// upper bound on the total number of open connections. To avoid service disruptions, +// connections can be tagged with metadata and optionally "protected" to ensure that +// essential connections are not arbitrarily cut. +package connmgr + +import ( + "context" + "time" + + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" +) + +// ConnManager tracks connections to peers, and allows consumers to associate metadata +// with each peer. +// +// It enables connections to be trimmed based on implementation-defined heuristics. +// The ConnManager allows libp2p to enforce an upper bound on the total number of +// open connections. +type ConnManager interface { + + // TagPeer tags a peer with a string, associating a weight with the tag. + TagPeer(peer.ID, string, int) + + // Untag removes the tagged value from the peer. + UntagPeer(p peer.ID, tag string) + + // UpsertTag updates an existing tag or inserts a new one. + // + // The connection manager calls the upsert function supplying the current + // value of the tag (or zero if inexistent). The return value is used as + // the new value of the tag. + UpsertTag(p peer.ID, tag string, upsert func(int) int) + + // GetTagInfo returns the metadata associated with the peer, + // or nil if no metadata has been recorded for the peer. + GetTagInfo(p peer.ID) *TagInfo + + // TrimOpenConns terminates open connections based on an implementation-defined + // heuristic. + TrimOpenConns(ctx context.Context) + + // Notifee returns an implementation that can be called back to inform of + // opened and closed connections. + Notifee() network.Notifiee + + // Protect protects a peer from having its connection(s) pruned. + // + // Tagging allows different parts of the system to manage protections without interfering with one another. + // + // Calls to Protect() with the same tag are idempotent. They are not refcounted, so after multiple calls + // to Protect() with the same tag, a single Unprotect() call bearing the same tag will revoke the protection. + Protect(id peer.ID, tag string) + + // Unprotect removes a protection that may have been placed on a peer, under the specified tag. + // + // The return value indicates whether the peer continues to be protected after this call, by way of a different tag. + // See notes on Protect() for more info. + Unprotect(id peer.ID, tag string) (protected bool) + + // Close closes the connection manager and stops background processes + Close() error +} + +// TagInfo stores metadata associated with a peer. +type TagInfo struct { + FirstSeen time.Time + Value int + + // Tags maps tag ids to the numerical values. + Tags map[string]int + + // Conns maps connection ids (such as remote multiaddr) to their creation time. + Conns map[string]time.Time +} diff --git a/core/connmgr/null.go b/core/connmgr/null.go new file mode 100644 index 0000000000..473af32895 --- /dev/null +++ b/core/connmgr/null.go @@ -0,0 +1,23 @@ +package connmgr + +import ( + "context" + + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" +) + +// NullConnMgr is a ConnMgr that provides no functionality. +type NullConnMgr struct{} + +var _ ConnManager = (*NullConnMgr)(nil) + +func (_ NullConnMgr) TagPeer(peer.ID, string, int) {} +func (_ NullConnMgr) UntagPeer(peer.ID, string) {} +func (_ NullConnMgr) UpsertTag(peer.ID, string, func(int) int) {} +func (_ NullConnMgr) GetTagInfo(peer.ID) *TagInfo { return &TagInfo{} } +func (_ NullConnMgr) TrimOpenConns(ctx context.Context) {} +func (_ NullConnMgr) Notifee() network.Notifiee { return network.GlobalNoopNotifiee } +func (_ NullConnMgr) Protect(peer.ID, string) {} +func (_ NullConnMgr) Unprotect(peer.ID, string) bool { return false } +func (_ NullConnMgr) Close() error { return nil } diff --git a/core/crypto/bench_test.go b/core/crypto/bench_test.go new file mode 100644 index 0000000000..fffc3bb7de --- /dev/null +++ b/core/crypto/bench_test.go @@ -0,0 +1,84 @@ +package crypto + +import "testing" + +func BenchmarkSignRSA1B(b *testing.B) { RunBenchmarkSignRSA(b, 1) } +func BenchmarkSignRSA10B(b *testing.B) { RunBenchmarkSignRSA(b, 10) } +func BenchmarkSignRSA100B(b *testing.B) { RunBenchmarkSignRSA(b, 100) } +func BenchmarkSignRSA1000B(b *testing.B) { RunBenchmarkSignRSA(b, 1000) } +func BenchmarkSignRSA10000B(b *testing.B) { RunBenchmarkSignRSA(b, 10000) } +func BenchmarkSignRSA100000B(b *testing.B) { RunBenchmarkSignRSA(b, 100000) } + +func BenchmarkVerifyRSA1B(b *testing.B) { RunBenchmarkVerifyRSA(b, 1) } +func BenchmarkVerifyRSA10B(b *testing.B) { RunBenchmarkVerifyRSA(b, 10) } +func BenchmarkVerifyRSA100B(b *testing.B) { RunBenchmarkVerifyRSA(b, 100) } +func BenchmarkVerifyRSA1000B(b *testing.B) { RunBenchmarkVerifyRSA(b, 1000) } +func BenchmarkVerifyRSA10000B(b *testing.B) { RunBenchmarkVerifyRSA(b, 10000) } +func BenchmarkVerifyRSA100000B(b *testing.B) { RunBenchmarkVerifyRSA(b, 100000) } + +func BenchmarkSignEd255191B(b *testing.B) { RunBenchmarkSignEd25519(b, 1) } +func BenchmarkSignEd2551910B(b *testing.B) { RunBenchmarkSignEd25519(b, 10) } +func BenchmarkSignEd25519100B(b *testing.B) { RunBenchmarkSignEd25519(b, 100) } +func BenchmarkSignEd255191000B(b *testing.B) { RunBenchmarkSignEd25519(b, 1000) } +func BenchmarkSignEd2551910000B(b *testing.B) { RunBenchmarkSignEd25519(b, 10000) } +func BenchmarkSignEd25519100000B(b *testing.B) { RunBenchmarkSignEd25519(b, 100000) } + +func BenchmarkVerifyEd255191B(b *testing.B) { RunBenchmarkVerifyEd25519(b, 1) } +func BenchmarkVerifyEd2551910B(b *testing.B) { RunBenchmarkVerifyEd25519(b, 10) } +func BenchmarkVerifyEd25519100B(b *testing.B) { RunBenchmarkVerifyEd25519(b, 100) } +func BenchmarkVerifyEd255191000B(b *testing.B) { RunBenchmarkVerifyEd25519(b, 1000) } +func BenchmarkVerifyEd2551910000B(b *testing.B) { RunBenchmarkVerifyEd25519(b, 10000) } +func BenchmarkVerifyEd25519100000B(b *testing.B) { RunBenchmarkVerifyEd25519(b, 100000) } + +func RunBenchmarkSignRSA(b *testing.B, numBytes int) { + runBenchmarkSign(b, numBytes, RSA) +} + +func RunBenchmarkSignEd25519(b *testing.B, numBytes int) { + runBenchmarkSign(b, numBytes, Ed25519) +} + +func runBenchmarkSign(b *testing.B, numBytes int, t int) { + secret, _, err := GenerateKeyPair(t, 1024) + if err != nil { + b.Fatal(err) + } + someData := make([]byte, numBytes) + b.ResetTimer() + for i := 0; i < b.N; i++ { + _, err := secret.Sign(someData) + if err != nil { + b.Fatal(err) + } + } +} + +func RunBenchmarkVerifyRSA(b *testing.B, numBytes int) { + runBenchmarkSign(b, numBytes, RSA) +} + +func RunBenchmarkVerifyEd25519(b *testing.B, numBytes int) { + runBenchmarkSign(b, numBytes, Ed25519) +} + +func runBenchmarkVerify(b *testing.B, numBytes int, t int) { + secret, public, err := GenerateKeyPair(t, 1024) + if err != nil { + b.Fatal(err) + } + someData := make([]byte, numBytes) + signature, err := secret.Sign(someData) + if err != nil { + b.Fatal(err) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + valid, err := public.Verify(someData, signature) + if err != nil { + b.Fatal(err) + } + if !valid { + b.Fatal("signature should be valid") + } + } +} diff --git a/core/crypto/ecdsa.go b/core/crypto/ecdsa.go new file mode 100644 index 0000000000..58e5d5f5ab --- /dev/null +++ b/core/crypto/ecdsa.go @@ -0,0 +1,186 @@ +package crypto + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "encoding/asn1" + "errors" + "io" + "math/big" + + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + + sha256 "github.com/minio/sha256-simd" +) + +// ECDSAPrivateKey is an implementation of an ECDSA private key +type ECDSAPrivateKey struct { + priv *ecdsa.PrivateKey +} + +// ECDSAPublicKey is an implementation of an ECDSA public key +type ECDSAPublicKey struct { + pub *ecdsa.PublicKey +} + +// ECDSASig holds the r and s values of an ECDSA signature +type ECDSASig struct { + R, S *big.Int +} + +var ( + // ErrNotECDSAPubKey is returned when the public key passed is not an ecdsa public key + ErrNotECDSAPubKey = errors.New("not an ecdsa public key") + // ErrNilSig is returned when the signature is nil + ErrNilSig = errors.New("sig is nil") + // ErrNilPrivateKey is returned when a nil private key is provided + ErrNilPrivateKey = errors.New("private key is nil") + // ECDSACurve is the default ecdsa curve used + ECDSACurve = elliptic.P256() +) + +// GenerateECDSAKeyPair generates a new ecdsa private and public key +func GenerateECDSAKeyPair(src io.Reader) (PrivKey, PubKey, error) { + return GenerateECDSAKeyPairWithCurve(ECDSACurve, src) +} + +// GenerateECDSAKeyPairWithCurve generates a new ecdsa private and public key with a speicified curve +func GenerateECDSAKeyPairWithCurve(curve elliptic.Curve, src io.Reader) (PrivKey, PubKey, error) { + priv, err := ecdsa.GenerateKey(curve, src) + if err != nil { + return nil, nil, err + } + + return &ECDSAPrivateKey{priv}, &ECDSAPublicKey{&priv.PublicKey}, nil +} + +// ECDSAKeyPairFromKey generates a new ecdsa private and public key from an input private key +func ECDSAKeyPairFromKey(priv *ecdsa.PrivateKey) (PrivKey, PubKey, error) { + if priv == nil { + return nil, nil, ErrNilPrivateKey + } + + return &ECDSAPrivateKey{priv}, &ECDSAPublicKey{&priv.PublicKey}, nil +} + +// MarshalECDSAPrivateKey returns x509 bytes from a private key +func MarshalECDSAPrivateKey(ePriv ECDSAPrivateKey) ([]byte, error) { + return x509.MarshalECPrivateKey(ePriv.priv) +} + +// MarshalECDSAPublicKey returns x509 bytes from a public key +func MarshalECDSAPublicKey(ePub ECDSAPublicKey) ([]byte, error) { + return x509.MarshalPKIXPublicKey(ePub.pub) +} + +// UnmarshalECDSAPrivateKey returns a private key from x509 bytes +func UnmarshalECDSAPrivateKey(data []byte) (PrivKey, error) { + priv, err := x509.ParseECPrivateKey(data) + if err != nil { + return nil, err + } + + return &ECDSAPrivateKey{priv}, nil +} + +// UnmarshalECDSAPublicKey returns the public key from x509 bytes +func UnmarshalECDSAPublicKey(data []byte) (PubKey, error) { + pubIfc, err := x509.ParsePKIXPublicKey(data) + if err != nil { + return nil, err + } + + pub, ok := pubIfc.(*ecdsa.PublicKey) + if !ok { + return nil, ErrNotECDSAPubKey + } + + return &ECDSAPublicKey{pub}, nil +} + +// Bytes returns the private key as protobuf bytes +func (ePriv *ECDSAPrivateKey) Bytes() ([]byte, error) { + return MarshalPrivateKey(ePriv) +} + +// Type returns the key type +func (ePriv *ECDSAPrivateKey) Type() pb.KeyType { + return pb.KeyType_ECDSA +} + +// Raw returns x509 bytes from a private key +func (ePriv *ECDSAPrivateKey) Raw() ([]byte, error) { + return x509.MarshalECPrivateKey(ePriv.priv) +} + +// Equals compares to private keys +func (ePriv *ECDSAPrivateKey) Equals(o Key) bool { + oPriv, ok := o.(*ECDSAPrivateKey) + if !ok { + return false + } + + return ePriv.priv.D.Cmp(oPriv.priv.D) == 0 +} + +// Sign returns the signature of the input data +func (ePriv *ECDSAPrivateKey) Sign(data []byte) ([]byte, error) { + hash := sha256.Sum256(data) + r, s, err := ecdsa.Sign(rand.Reader, ePriv.priv, hash[:]) + if err != nil { + return nil, err + } + + return asn1.Marshal(ECDSASig{ + R: r, + S: s, + }) +} + +// GetPublic returns a public key +func (ePriv *ECDSAPrivateKey) GetPublic() PubKey { + return &ECDSAPublicKey{&ePriv.priv.PublicKey} +} + +// Bytes returns the public key as protobuf bytes +func (ePub *ECDSAPublicKey) Bytes() ([]byte, error) { + return MarshalPublicKey(ePub) +} + +// Type returns the key type +func (ePub *ECDSAPublicKey) Type() pb.KeyType { + return pb.KeyType_ECDSA +} + +// Raw returns x509 bytes from a public key +func (ePub ECDSAPublicKey) Raw() ([]byte, error) { + return x509.MarshalPKIXPublicKey(ePub.pub) +} + +// Equals compares to public keys +func (ePub *ECDSAPublicKey) Equals(o Key) bool { + oPub, ok := o.(*ECDSAPublicKey) + if !ok { + return false + } + + return ePub.pub.X != nil && ePub.pub.Y != nil && oPub.pub.X != nil && oPub.pub.Y != nil && + 0 == ePub.pub.X.Cmp(oPub.pub.X) && 0 == ePub.pub.Y.Cmp(oPub.pub.Y) +} + +// Verify compares data to a signature +func (ePub *ECDSAPublicKey) Verify(data, sigBytes []byte) (bool, error) { + sig := new(ECDSASig) + if _, err := asn1.Unmarshal(sigBytes, sig); err != nil { + return false, err + } + if sig == nil { + return false, ErrNilSig + } + + hash := sha256.Sum256(data) + + return ecdsa.Verify(ePub.pub, hash[:], sig.R, sig.S), nil +} diff --git a/core/crypto/ecdsa_test.go b/core/crypto/ecdsa_test.go new file mode 100644 index 0000000000..328b9a7c6c --- /dev/null +++ b/core/crypto/ecdsa_test.go @@ -0,0 +1,96 @@ +package crypto + +import ( + "crypto/rand" + "testing" +) + +func TestECDSABasicSignAndVerify(t *testing.T) { + priv, pub, err := GenerateECDSAKeyPair(rand.Reader) + if err != nil { + t.Fatal(err) + } + + data := []byte("hello! and welcome to some awesome crypto primitives") + + sig, err := priv.Sign(data) + if err != nil { + t.Fatal(err) + } + + ok, err := pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + + if !ok { + t.Fatal("signature didnt match") + } + + // change data + data[0] = ^data[0] + ok, err = pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + + if ok { + t.Fatal("signature matched and shouldn't") + } +} + +func TestECDSASignZero(t *testing.T) { + priv, pub, err := GenerateECDSAKeyPair(rand.Reader) + if err != nil { + t.Fatal(err) + } + + data := make([]byte, 0) + sig, err := priv.Sign(data) + if err != nil { + t.Fatal(err) + } + + ok, err := pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + if !ok { + t.Fatal("signature didn't match") + } +} + +func TestECDSAMarshalLoop(t *testing.T) { + priv, pub, err := GenerateECDSAKeyPair(rand.Reader) + if err != nil { + t.Fatal(err) + } + + privB, err := priv.Bytes() + if err != nil { + t.Fatal(err) + } + + privNew, err := UnmarshalPrivateKey(privB) + if err != nil { + t.Fatal(err) + } + + if !priv.Equals(privNew) || !privNew.Equals(priv) { + t.Fatal("keys are not equal") + } + + pubB, err := pub.Bytes() + if err != nil { + t.Fatal(err) + } + pubNew, err := UnmarshalPublicKey(pubB) + if err != nil { + t.Fatal(err) + } + + if !pub.Equals(pubNew) || !pubNew.Equals(pub) { + t.Fatal("keys are not equal") + } + +} diff --git a/core/crypto/ed25519.go b/core/crypto/ed25519.go new file mode 100644 index 0000000000..b6e553147d --- /dev/null +++ b/core/crypto/ed25519.go @@ -0,0 +1,155 @@ +package crypto + +import ( + "bytes" + "errors" + "fmt" + "io" + + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + + "golang.org/x/crypto/ed25519" +) + +// Ed25519PrivateKey is an ed25519 private key. +type Ed25519PrivateKey struct { + k ed25519.PrivateKey +} + +// Ed25519PublicKey is an ed25519 public key. +type Ed25519PublicKey struct { + k ed25519.PublicKey +} + +// GenerateEd25519Key generates a new ed25519 private and public key pair. +func GenerateEd25519Key(src io.Reader) (PrivKey, PubKey, error) { + pub, priv, err := ed25519.GenerateKey(src) + if err != nil { + return nil, nil, err + } + + return &Ed25519PrivateKey{ + k: priv, + }, + &Ed25519PublicKey{ + k: pub, + }, + nil +} + +// Type of the private key (Ed25519). +func (k *Ed25519PrivateKey) Type() pb.KeyType { + return pb.KeyType_Ed25519 +} + +// Bytes marshals an ed25519 private key to protobuf bytes. +func (k *Ed25519PrivateKey) Bytes() ([]byte, error) { + return MarshalPrivateKey(k) +} + +// Raw private key bytes. +func (k *Ed25519PrivateKey) Raw() ([]byte, error) { + // The Ed25519 private key contains two 32-bytes curve points, the private + // key and the public key. + // It makes it more efficient to get the public key without re-computing an + // elliptic curve multiplication. + buf := make([]byte, len(k.k)) + copy(buf, k.k) + + return buf, nil +} + +func (k *Ed25519PrivateKey) pubKeyBytes() []byte { + return k.k[ed25519.PrivateKeySize-ed25519.PublicKeySize:] +} + +// Equals compares two ed25519 private keys. +func (k *Ed25519PrivateKey) Equals(o Key) bool { + edk, ok := o.(*Ed25519PrivateKey) + if !ok { + return false + } + + return bytes.Equal(k.k, edk.k) +} + +// GetPublic returns an ed25519 public key from a private key. +func (k *Ed25519PrivateKey) GetPublic() PubKey { + return &Ed25519PublicKey{k: k.pubKeyBytes()} +} + +// Sign returns a signature from an input message. +func (k *Ed25519PrivateKey) Sign(msg []byte) ([]byte, error) { + return ed25519.Sign(k.k, msg), nil +} + +// Type of the public key (Ed25519). +func (k *Ed25519PublicKey) Type() pb.KeyType { + return pb.KeyType_Ed25519 +} + +// Bytes returns a ed25519 public key as protobuf bytes. +func (k *Ed25519PublicKey) Bytes() ([]byte, error) { + return MarshalPublicKey(k) +} + +// Raw public key bytes. +func (k *Ed25519PublicKey) Raw() ([]byte, error) { + return k.k, nil +} + +// Equals compares two ed25519 public keys. +func (k *Ed25519PublicKey) Equals(o Key) bool { + edk, ok := o.(*Ed25519PublicKey) + if !ok { + return false + } + + return bytes.Equal(k.k, edk.k) +} + +// Verify checks a signature agains the input data. +func (k *Ed25519PublicKey) Verify(data []byte, sig []byte) (bool, error) { + return ed25519.Verify(k.k, data, sig), nil +} + +// UnmarshalEd25519PublicKey returns a public key from input bytes. +func UnmarshalEd25519PublicKey(data []byte) (PubKey, error) { + if len(data) != 32 { + return nil, errors.New("expect ed25519 public key data size to be 32") + } + + return &Ed25519PublicKey{ + k: ed25519.PublicKey(data), + }, nil +} + +// UnmarshalEd25519PrivateKey returns a private key from input bytes. +func UnmarshalEd25519PrivateKey(data []byte) (PrivKey, error) { + switch len(data) { + case ed25519.PrivateKeySize + ed25519.PublicKeySize: + // Remove the redundant public key. See issue #36. + redundantPk := data[ed25519.PrivateKeySize:] + pk := data[ed25519.PrivateKeySize-ed25519.PublicKeySize : ed25519.PrivateKeySize] + if !bytes.Equal(pk, redundantPk) { + return nil, errors.New("expected redundant ed25519 public key to be redundant") + } + + // No point in storing the extra data. + newKey := make([]byte, ed25519.PrivateKeySize) + copy(newKey, data[:ed25519.PrivateKeySize]) + data = newKey + case ed25519.PrivateKeySize: + default: + return nil, fmt.Errorf( + "expected ed25519 data size to be %d or %d, got %d", + ed25519.PrivateKeySize, + ed25519.PrivateKeySize+ed25519.PublicKeySize, + len(data), + ) + } + + return &Ed25519PrivateKey{ + k: ed25519.PrivateKey(data), + }, nil +} diff --git a/core/crypto/ed25519_test.go b/core/crypto/ed25519_test.go new file mode 100644 index 0000000000..222b5154b6 --- /dev/null +++ b/core/crypto/ed25519_test.go @@ -0,0 +1,220 @@ +package crypto + +import ( + "crypto/rand" + "testing" + + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + + "golang.org/x/crypto/ed25519" +) + +func TestBasicSignAndVerify(t *testing.T) { + priv, pub, err := GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + + data := []byte("hello! and welcome to some awesome crypto primitives") + + sig, err := priv.Sign(data) + if err != nil { + t.Fatal(err) + } + + ok, err := pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + + if !ok { + t.Fatal("signature didn't match") + } + + // change data + data[0] = ^data[0] + ok, err = pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + + if ok { + t.Fatal("signature matched and shouldn't") + } +} + +func TestSignZero(t *testing.T) { + priv, pub, err := GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + + data := make([]byte, 0) + sig, err := priv.Sign(data) + if err != nil { + t.Fatal(err) + } + + ok, err := pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + if !ok { + t.Fatal("signature didn't match") + } +} + +func TestMarshalLoop(t *testing.T) { + priv, pub, err := GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + + t.Run("PrivateKey", func(t *testing.T) { + for name, f := range map[string]func() ([]byte, error){ + "Bytes": priv.Bytes, + "Marshal": func() ([]byte, error) { + return MarshalPrivateKey(priv) + }, + "Redundant": func() ([]byte, error) { + // See issue #36. + // Ed25519 private keys used to contain the public key twice. + // For backwards-compatibility, we need to continue supporting + // that scenario. + pbmes := new(pb.PrivateKey) + pbmes.Type = priv.Type() + data, err := priv.Raw() + if err != nil { + t.Fatal(err) + } + + pbmes.Data = append(data, data[len(data)-ed25519.PublicKeySize:]...) + return pbmes.Marshal() + }, + } { + t.Run(name, func(t *testing.T) { + bts, err := f() + if err != nil { + t.Fatal(err) + } + + privNew, err := UnmarshalPrivateKey(bts) + if err != nil { + t.Fatal(err) + } + + if !priv.Equals(privNew) || !privNew.Equals(priv) { + t.Fatal("keys are not equal") + } + + msg := []byte("My child, my sister,\nThink of the rapture\nOf living together there!") + signed, err := privNew.Sign(msg) + if err != nil { + t.Fatal(err) + } + + ok, err := privNew.GetPublic().Verify(msg, signed) + if err != nil { + t.Fatal(err) + } + + if !ok { + t.Fatal("signature didn't match") + } + }) + } + }) + + t.Run("PublicKey", func(t *testing.T) { + for name, f := range map[string]func() ([]byte, error){ + "Bytes": pub.Bytes, + "Marshal": func() ([]byte, error) { + return MarshalPublicKey(pub) + }, + } { + t.Run(name, func(t *testing.T) { + bts, err := f() + if err != nil { + t.Fatal(err) + } + pubNew, err := UnmarshalPublicKey(bts) + if err != nil { + t.Fatal(err) + } + + if !pub.Equals(pubNew) || !pubNew.Equals(pub) { + t.Fatal("keys are not equal") + } + }) + } + }) +} + +func TestUnmarshalErrors(t *testing.T) { + t.Run("PublicKey", func(t *testing.T) { + t.Run("Invalid data length", func(t *testing.T) { + pbmes := &pb.PublicKey{ + Type: pb.KeyType_Ed25519, + Data: []byte{42}, + } + + data, err := pbmes.Marshal() + if err != nil { + t.Fatal(err) + } + + _, err = UnmarshalPublicKey(data) + if err == nil { + t.Fatal("expected an error") + } + }) + }) + + t.Run("PrivateKey", func(t *testing.T) { + t.Run("Redundant public key mismatch", func(t *testing.T) { + priv, _, err := GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + + pbmes := new(pb.PrivateKey) + pbmes.Type = priv.Type() + data, err := priv.Raw() + if err != nil { + t.Fatal(err) + } + + // Append the private key instead of the public key. + pbmes.Data = append(data, data[:ed25519.PublicKeySize]...) + b, err := pbmes.Marshal() + if err != nil { + t.Fatal(err) + } + + _, err = UnmarshalPrivateKey(b) + if err == nil { + t.Fatal("expected an error") + } + if err.Error() != "expected redundant ed25519 public key to be redundant" { + t.Fatalf("invalid error received: %s", err.Error()) + } + }) + + t.Run("Invalid data length", func(t *testing.T) { + pbmes := &pb.PrivateKey{ + Type: pb.KeyType_Ed25519, + Data: []byte{42}, + } + + data, err := pbmes.Marshal() + if err != nil { + t.Fatal(err) + } + + _, err = UnmarshalPrivateKey(data) + if err == nil { + t.Fatal("expected an error") + } + }) + }) +} diff --git a/core/crypto/fixture_test.go b/core/crypto/fixture_test.go new file mode 100644 index 0000000000..471e85472f --- /dev/null +++ b/core/crypto/fixture_test.go @@ -0,0 +1,132 @@ +package crypto_test + +import ( + "bytes" + "crypto/rand" + "fmt" + "io" + "io/ioutil" + "testing" + + crypto "github.com/libp2p/go-libp2p-core/crypto" + crypto_pb "github.com/libp2p/go-libp2p-core/crypto/pb" +) + +var message = []byte("Libp2p is the _best_!") + +type testCase struct { + keyType crypto_pb.KeyType + gen func(i io.Reader) (crypto.PrivKey, crypto.PubKey, error) + sigDeterministic bool +} + +var keyTypes = []testCase{ + { + keyType: crypto_pb.KeyType_ECDSA, + gen: crypto.GenerateECDSAKeyPair, + }, + { + keyType: crypto_pb.KeyType_Secp256k1, + sigDeterministic: true, + gen: crypto.GenerateSecp256k1Key, + }, + { + keyType: crypto_pb.KeyType_RSA, + sigDeterministic: true, + gen: func(i io.Reader) (crypto.PrivKey, crypto.PubKey, error) { + return crypto.GenerateRSAKeyPair(2048, i) + }, + }, +} + +func fname(kt crypto_pb.KeyType, ext string) string { + return fmt.Sprintf("test_data/%d.%s", kt, ext) +} + +func TestFixtures(t *testing.T) { + for _, tc := range keyTypes { + t.Run(tc.keyType.String(), func(t *testing.T) { + pubBytes, err := ioutil.ReadFile(fname(tc.keyType, "pub")) + if err != nil { + t.Fatal(err) + } + privBytes, err := ioutil.ReadFile(fname(tc.keyType, "priv")) + if err != nil { + t.Fatal(err) + } + sigBytes, err := ioutil.ReadFile(fname(tc.keyType, "sig")) + if err != nil { + t.Fatal(err) + } + pub, err := crypto.UnmarshalPublicKey(pubBytes) + if err != nil { + t.Fatal(err) + } + pubBytes2, err := pub.Bytes() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(pubBytes2, pubBytes) { + t.Fatal("encoding round-trip failed") + } + priv, err := crypto.UnmarshalPrivateKey(privBytes) + if err != nil { + t.Fatal(err) + } + privBytes2, err := priv.Bytes() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(privBytes2, privBytes) { + t.Fatal("encoding round-trip failed") + } + ok, err := pub.Verify(message, sigBytes) + if !ok || err != nil { + t.Fatal("failed to validate signature with public key") + } + + if tc.sigDeterministic { + sigBytes2, err := priv.Sign(message) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(sigBytes2, sigBytes) { + t.Fatal("signature not deterministic") + } + } + }) + } +} + +func init() { + // set to true to re-generate test data + if false { + generate() + panic("generated") + } +} + +// generate re-generates test data +func generate() { + for _, tc := range keyTypes { + priv, pub, err := tc.gen(rand.Reader) + if err != nil { + panic(err) + } + pubb, err := pub.Bytes() + if err != nil { + panic(err) + } + privb, err := priv.Bytes() + if err != nil { + panic(err) + } + sig, err := priv.Sign(message) + if err != nil { + panic(err) + } + ioutil.WriteFile(fname(tc.keyType, "pub"), pubb, 0666) + ioutil.WriteFile(fname(tc.keyType, "priv"), privb, 0666) + ioutil.WriteFile(fname(tc.keyType, "sig"), sig, 0666) + } +} diff --git a/core/crypto/key.go b/core/crypto/key.go new file mode 100644 index 0000000000..50167dbe02 --- /dev/null +++ b/core/crypto/key.go @@ -0,0 +1,351 @@ +// Package crypto implements various cryptographic utilities used by libp2p. +// This includes a Public and Private key interface and key implementations +// for supported key algorithms. +package crypto + +import ( + "bytes" + "crypto/elliptic" + "crypto/hmac" + "crypto/rand" + "crypto/sha1" + "crypto/sha512" + "encoding/base64" + "errors" + "fmt" + "hash" + "io" + + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + + "github.com/gogo/protobuf/proto" + sha256 "github.com/minio/sha256-simd" +) + +const ( + // RSA is an enum for the supported RSA key type + RSA = iota + // Ed25519 is an enum for the supported Ed25519 key type + Ed25519 + // Secp256k1 is an enum for the supported Secp256k1 key type + Secp256k1 + // ECDSA is an enum for the supported ECDSA key type + ECDSA +) + +var ( + // ErrBadKeyType is returned when a key is not supported + ErrBadKeyType = errors.New("invalid or unsupported key type") + // KeyTypes is a list of supported keys + KeyTypes = []int{ + RSA, + Ed25519, + Secp256k1, + ECDSA, + } +) + +// PubKeyUnmarshaller is a func that creates a PubKey from a given slice of bytes +type PubKeyUnmarshaller func(data []byte) (PubKey, error) + +// PrivKeyUnmarshaller is a func that creates a PrivKey from a given slice of bytes +type PrivKeyUnmarshaller func(data []byte) (PrivKey, error) + +// PubKeyUnmarshallers is a map of unmarshallers by key type +var PubKeyUnmarshallers = map[pb.KeyType]PubKeyUnmarshaller{ + pb.KeyType_RSA: UnmarshalRsaPublicKey, + pb.KeyType_Ed25519: UnmarshalEd25519PublicKey, + pb.KeyType_Secp256k1: UnmarshalSecp256k1PublicKey, + pb.KeyType_ECDSA: UnmarshalECDSAPublicKey, +} + +// PrivKeyUnmarshallers is a map of unmarshallers by key type +var PrivKeyUnmarshallers = map[pb.KeyType]PrivKeyUnmarshaller{ + pb.KeyType_RSA: UnmarshalRsaPrivateKey, + pb.KeyType_Ed25519: UnmarshalEd25519PrivateKey, + pb.KeyType_Secp256k1: UnmarshalSecp256k1PrivateKey, + pb.KeyType_ECDSA: UnmarshalECDSAPrivateKey, +} + +// Key represents a crypto key that can be compared to another key +type Key interface { + // Bytes returns a serialized, storeable representation of this key + // DEPRECATED in favor of Marshal / Unmarshal + Bytes() ([]byte, error) + + // Equals checks whether two PubKeys are the same + Equals(Key) bool + + // Raw returns the raw bytes of the key (not wrapped in the + // libp2p-crypto protobuf). + // + // This function is the inverse of {Priv,Pub}KeyUnmarshaler. + Raw() ([]byte, error) + + // Type returns the protobof key type. + Type() pb.KeyType +} + +// PrivKey represents a private key that can be used to generate a public key and sign data +type PrivKey interface { + Key + + // Cryptographically sign the given bytes + Sign([]byte) ([]byte, error) + + // Return a public key paired with this private key + GetPublic() PubKey +} + +// PubKey is a public key that can be used to verifiy data signed with the corresponding private key +type PubKey interface { + Key + + // Verify that 'sig' is the signed hash of 'data' + Verify(data []byte, sig []byte) (bool, error) +} + +// GenSharedKey generates the shared key from a given private key +type GenSharedKey func([]byte) ([]byte, error) + +// GenerateKeyPair generates a private and public key +func GenerateKeyPair(typ, bits int) (PrivKey, PubKey, error) { + return GenerateKeyPairWithReader(typ, bits, rand.Reader) +} + +// GenerateKeyPairWithReader returns a keypair of the given type and bitsize +func GenerateKeyPairWithReader(typ, bits int, src io.Reader) (PrivKey, PubKey, error) { + switch typ { + case RSA: + return GenerateRSAKeyPair(bits, src) + case Ed25519: + return GenerateEd25519Key(src) + case Secp256k1: + return GenerateSecp256k1Key(src) + case ECDSA: + return GenerateECDSAKeyPair(src) + default: + return nil, nil, ErrBadKeyType + } +} + +// GenerateEKeyPair returns an ephemeral public key and returns a function that will compute +// the shared secret key. Used in the identify module. +// +// Focuses only on ECDH now, but can be made more general in the future. +func GenerateEKeyPair(curveName string) ([]byte, GenSharedKey, error) { + var curve elliptic.Curve + + switch curveName { + case "P-256": + curve = elliptic.P256() + case "P-384": + curve = elliptic.P384() + case "P-521": + curve = elliptic.P521() + } + + priv, x, y, err := elliptic.GenerateKey(curve, rand.Reader) + if err != nil { + return nil, nil, err + } + + pubKey := elliptic.Marshal(curve, x, y) + + done := func(theirPub []byte) ([]byte, error) { + // Verify and unpack node's public key. + x, y := elliptic.Unmarshal(curve, theirPub) + if x == nil { + return nil, fmt.Errorf("malformed public key: %d %v", len(theirPub), theirPub) + } + + if !curve.IsOnCurve(x, y) { + return nil, errors.New("invalid public key") + } + + // Generate shared secret. + secret, _ := curve.ScalarMult(x, y, priv) + + return secret.Bytes(), nil + } + + return pubKey, done, nil +} + +// StretchedKeys ... +type StretchedKeys struct { + IV []byte + MacKey []byte + CipherKey []byte +} + +// KeyStretcher returns a set of keys for each party by stretching the shared key. +// (myIV, theirIV, myCipherKey, theirCipherKey, myMACKey, theirMACKey) +func KeyStretcher(cipherType string, hashType string, secret []byte) (StretchedKeys, StretchedKeys) { + var cipherKeySize int + var ivSize int + switch cipherType { + case "AES-128": + ivSize = 16 + cipherKeySize = 16 + case "AES-256": + ivSize = 16 + cipherKeySize = 32 + case "Blowfish": + ivSize = 8 + // Note: cypherKeySize arbitrarily selected, needs more thought + cipherKeySize = 32 + } + + hmacKeySize := 20 + + seed := []byte("key expansion") + + result := make([]byte, 2*(ivSize+cipherKeySize+hmacKeySize)) + + var h func() hash.Hash + + switch hashType { + case "SHA1": + h = sha1.New + case "SHA256": + h = sha256.New + case "SHA512": + h = sha512.New + default: + panic("Unrecognized hash function, programmer error?") + } + + m := hmac.New(h, secret) + // note: guaranteed to never return an error + m.Write(seed) + + a := m.Sum(nil) + + j := 0 + for j < len(result) { + m.Reset() + + // note: guaranteed to never return an error. + m.Write(a) + m.Write(seed) + + b := m.Sum(nil) + + todo := len(b) + + if j+todo > len(result) { + todo = len(result) - j + } + + copy(result[j:j+todo], b) + + j += todo + + m.Reset() + + // note: guaranteed to never return an error. + m.Write(a) + + a = m.Sum(nil) + } + + half := len(result) / 2 + r1 := result[:half] + r2 := result[half:] + + var k1 StretchedKeys + var k2 StretchedKeys + + k1.IV = r1[0:ivSize] + k1.CipherKey = r1[ivSize : ivSize+cipherKeySize] + k1.MacKey = r1[ivSize+cipherKeySize:] + + k2.IV = r2[0:ivSize] + k2.CipherKey = r2[ivSize : ivSize+cipherKeySize] + k2.MacKey = r2[ivSize+cipherKeySize:] + + return k1, k2 +} + +// UnmarshalPublicKey converts a protobuf serialized public key into its +// representative object +func UnmarshalPublicKey(data []byte) (PubKey, error) { + pmes := new(pb.PublicKey) + err := proto.Unmarshal(data, pmes) + if err != nil { + return nil, err + } + + um, ok := PubKeyUnmarshallers[pmes.GetType()] + if !ok { + return nil, ErrBadKeyType + } + + return um(pmes.GetData()) +} + +// MarshalPublicKey converts a public key object into a protobuf serialized +// public key +func MarshalPublicKey(k PubKey) ([]byte, error) { + pbmes := new(pb.PublicKey) + pbmes.Type = k.Type() + data, err := k.Raw() + if err != nil { + return nil, err + } + pbmes.Data = data + + return proto.Marshal(pbmes) +} + +// UnmarshalPrivateKey converts a protobuf serialized private key into its +// representative object +func UnmarshalPrivateKey(data []byte) (PrivKey, error) { + pmes := new(pb.PrivateKey) + err := proto.Unmarshal(data, pmes) + if err != nil { + return nil, err + } + + um, ok := PrivKeyUnmarshallers[pmes.GetType()] + if !ok { + return nil, ErrBadKeyType + } + + return um(pmes.GetData()) +} + +// MarshalPrivateKey converts a key object into its protobuf serialized form. +func MarshalPrivateKey(k PrivKey) ([]byte, error) { + pbmes := new(pb.PrivateKey) + pbmes.Type = k.Type() + data, err := k.Raw() + if err != nil { + return nil, err + } + + pbmes.Data = data + return proto.Marshal(pbmes) +} + +// ConfigDecodeKey decodes from b64 (for config file), and unmarshals. +func ConfigDecodeKey(b string) ([]byte, error) { + return base64.StdEncoding.DecodeString(b) +} + +// ConfigEncodeKey encodes to b64 (for config file), and marshals. +func ConfigEncodeKey(b []byte) string { + return base64.StdEncoding.EncodeToString(b) +} + +// KeyEqual checks whether two Keys are equivalent (have identical byte representations). +func KeyEqual(k1, k2 Key) bool { + if k1 == k2 { + return true + } + + b1, err1 := k1.Bytes() + b2, err2 := k2.Bytes() + return bytes.Equal(b1, b2) && err1 == err2 +} diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go new file mode 100644 index 0000000000..a93fbf0857 --- /dev/null +++ b/core/crypto/key_test.go @@ -0,0 +1,147 @@ +package crypto_test + +import ( + "bytes" + "crypto/rand" + "testing" + + . "github.com/libp2p/go-libp2p-core/crypto" + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + "github.com/libp2p/go-libp2p-testing/crypto" +) + +func TestKeys(t *testing.T) { + for _, typ := range KeyTypes { + testKeyType(typ, t) + } +} + +func testKeyType(typ int, t *testing.T) { + sk, pk, err := tcrypto.RandTestKeyPair(typ, 512) + if err != nil { + t.Fatal(err) + } + + testKeySignature(t, sk) + testKeyEncoding(t, sk) + testKeyEquals(t, sk) + testKeyEquals(t, pk) +} + +func testKeySignature(t *testing.T, sk PrivKey) { + pk := sk.GetPublic() + + text := make([]byte, 16) + if _, err := rand.Read(text); err != nil { + t.Fatal(err) + } + + sig, err := sk.Sign(text) + if err != nil { + t.Fatal(err) + } + + valid, err := pk.Verify(text, sig) + if err != nil { + t.Fatal(err) + } + + if !valid { + t.Fatal("Invalid signature.") + } +} + +func testKeyEncoding(t *testing.T, sk PrivKey) { + skbm, err := MarshalPrivateKey(sk) + if err != nil { + t.Fatal(err) + } + + sk2, err := UnmarshalPrivateKey(skbm) + if err != nil { + t.Fatal(err) + } + + if !sk.Equals(sk2) { + t.Error("Unmarshaled private key didn't match original.\n") + } + + skbm2, err := MarshalPrivateKey(sk2) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(skbm, skbm2) { + t.Error("skb -> marshal -> unmarshal -> skb failed.\n", skbm, "\n", skbm2) + } + + pk := sk.GetPublic() + pkbm, err := MarshalPublicKey(pk) + if err != nil { + t.Fatal(err) + } + + pk2, err := UnmarshalPublicKey(pkbm) + if err != nil { + t.Fatal(err) + } + + if !pk.Equals(pk2) { + t.Error("Unmarshaled public key didn't match original.\n") + } + + pkbm2, err := MarshalPublicKey(pk) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(pkbm, pkbm2) { + t.Error("skb -> marshal -> unmarshal -> skb failed.\n", pkbm, "\n", pkbm2) + } +} + +func testKeyEquals(t *testing.T, k Key) { + kb, err := k.Bytes() + if err != nil { + t.Fatal(err) + } + + if !KeyEqual(k, k) { + t.Fatal("Key not equal to itself.") + } + + if !KeyEqual(k, testkey(kb)) { + t.Fatal("Key not equal to key with same bytes.") + } + + sk, pk, err := tcrypto.RandTestKeyPair(RSA, 512) + if err != nil { + t.Fatal(err) + } + + if KeyEqual(k, sk) { + t.Fatal("Keys should not equal.") + } + + if KeyEqual(k, pk) { + t.Fatal("Keys should not equal.") + } +} + +type testkey []byte + +func (pk testkey) Bytes() ([]byte, error) { + return pk, nil +} + +func (pk testkey) Type() pb.KeyType { + return pb.KeyType_RSA +} + +func (pk testkey) Raw() ([]byte, error) { + return pk, nil +} + +func (pk testkey) Equals(k Key) bool { + return KeyEqual(pk, k) +} diff --git a/core/crypto/openssl_common.go b/core/crypto/openssl_common.go new file mode 100644 index 0000000000..f235e75f33 --- /dev/null +++ b/core/crypto/openssl_common.go @@ -0,0 +1,98 @@ +// +build openssl + +package crypto + +import ( + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + + openssl "github.com/spacemonkeygo/openssl" +) + +// define these as separate types so we can add more key types later and reuse +// code. + +type opensslPublicKey struct { + key openssl.PublicKey +} + +type opensslPrivateKey struct { + key openssl.PrivateKey +} + +func unmarshalOpensslPrivateKey(b []byte) (opensslPrivateKey, error) { + sk, err := openssl.LoadPrivateKeyFromDER(b) + if err != nil { + return opensslPrivateKey{}, err + } + return opensslPrivateKey{sk}, nil +} + +func unmarshalOpensslPublicKey(b []byte) (opensslPublicKey, error) { + sk, err := openssl.LoadPublicKeyFromDER(b) + if err != nil { + return opensslPublicKey{}, err + } + return opensslPublicKey{sk}, nil +} + +// Verify compares a signature against input data +func (pk *opensslPublicKey) Verify(data, sig []byte) (bool, error) { + err := pk.key.VerifyPKCS1v15(openssl.SHA256_Method, data, sig) + return err == nil, err +} + +func (pk *opensslPublicKey) Type() pb.KeyType { + switch pk.key.KeyType() { + case openssl.KeyTypeRSA: + return pb.KeyType_RSA + default: + return -1 + } +} + +// Bytes returns protobuf bytes of a public key +func (pk *opensslPublicKey) Bytes() ([]byte, error) { + return MarshalPublicKey(pk) +} + +func (pk *opensslPublicKey) Raw() ([]byte, error) { + return pk.key.MarshalPKIXPublicKeyDER() +} + +// Equals checks whether this key is equal to another +func (pk *opensslPublicKey) Equals(k Key) bool { + return KeyEqual(pk, k) +} + +// Sign returns a signature of the input data +func (sk *opensslPrivateKey) Sign(message []byte) ([]byte, error) { + return sk.key.SignPKCS1v15(openssl.SHA256_Method, message) +} + +// GetPublic returns a public key +func (sk *opensslPrivateKey) GetPublic() PubKey { + return &opensslPublicKey{sk.key} +} + +func (sk *opensslPrivateKey) Type() pb.KeyType { + switch sk.key.KeyType() { + case openssl.KeyTypeRSA: + return pb.KeyType_RSA + default: + return -1 + } +} + +// Bytes returns protobuf bytes from a private key +func (sk *opensslPrivateKey) Bytes() ([]byte, error) { + return MarshalPrivateKey(sk) +} + +func (sk *opensslPrivateKey) Raw() ([]byte, error) { + return sk.key.MarshalPKCS1PrivateKeyDER() +} + +// Equals checks whether this key is equal to another +func (sk *opensslPrivateKey) Equals(k Key) bool { + return KeyEqual(sk, k) +} diff --git a/core/crypto/pb/Makefile b/core/crypto/pb/Makefile new file mode 100644 index 0000000000..df34e54b01 --- /dev/null +++ b/core/crypto/pb/Makefile @@ -0,0 +1,11 @@ +PB = $(wildcard *.proto) +GO = $(PB:.proto=.pb.go) + +all: $(GO) + +%.pb.go: %.proto + protoc --proto_path=$(GOPATH)/src:. --gogofaster_out=. $< + +clean: + rm -f *.pb.go + rm -f *.go diff --git a/core/crypto/pb/crypto.pb.go b/core/crypto/pb/crypto.pb.go new file mode 100644 index 0000000000..5fa7aec73f --- /dev/null +++ b/core/crypto/pb/crypto.pb.go @@ -0,0 +1,643 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: crypto.proto + +package crypto_pb + +import ( + fmt "fmt" + github_com_gogo_protobuf_proto "github.com/gogo/protobuf/proto" + proto "github.com/gogo/protobuf/proto" + io "io" + math "math" +) + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + +type KeyType int32 + +const ( + KeyType_RSA KeyType = 0 + KeyType_Ed25519 KeyType = 1 + KeyType_Secp256k1 KeyType = 2 + KeyType_ECDSA KeyType = 3 +) + +var KeyType_name = map[int32]string{ + 0: "RSA", + 1: "Ed25519", + 2: "Secp256k1", + 3: "ECDSA", +} + +var KeyType_value = map[string]int32{ + "RSA": 0, + "Ed25519": 1, + "Secp256k1": 2, + "ECDSA": 3, +} + +func (x KeyType) Enum() *KeyType { + p := new(KeyType) + *p = x + return p +} + +func (x KeyType) String() string { + return proto.EnumName(KeyType_name, int32(x)) +} + +func (x *KeyType) UnmarshalJSON(data []byte) error { + value, err := proto.UnmarshalJSONEnum(KeyType_value, data, "KeyType") + if err != nil { + return err + } + *x = KeyType(value) + return nil +} + +func (KeyType) EnumDescriptor() ([]byte, []int) { + return fileDescriptor_527278fb02d03321, []int{0} +} + +type PublicKey struct { + Type KeyType `protobuf:"varint,1,req,name=Type,enum=crypto.pb.KeyType" json:"Type"` + Data []byte `protobuf:"bytes,2,req,name=Data" json:"Data"` +} + +func (m *PublicKey) Reset() { *m = PublicKey{} } +func (m *PublicKey) String() string { return proto.CompactTextString(m) } +func (*PublicKey) ProtoMessage() {} +func (*PublicKey) Descriptor() ([]byte, []int) { + return fileDescriptor_527278fb02d03321, []int{0} +} +func (m *PublicKey) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *PublicKey) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_PublicKey.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *PublicKey) XXX_Merge(src proto.Message) { + xxx_messageInfo_PublicKey.Merge(m, src) +} +func (m *PublicKey) XXX_Size() int { + return m.Size() +} +func (m *PublicKey) XXX_DiscardUnknown() { + xxx_messageInfo_PublicKey.DiscardUnknown(m) +} + +var xxx_messageInfo_PublicKey proto.InternalMessageInfo + +func (m *PublicKey) GetType() KeyType { + if m != nil { + return m.Type + } + return KeyType_RSA +} + +func (m *PublicKey) GetData() []byte { + if m != nil { + return m.Data + } + return nil +} + +type PrivateKey struct { + Type KeyType `protobuf:"varint,1,req,name=Type,enum=crypto.pb.KeyType" json:"Type"` + Data []byte `protobuf:"bytes,2,req,name=Data" json:"Data"` +} + +func (m *PrivateKey) Reset() { *m = PrivateKey{} } +func (m *PrivateKey) String() string { return proto.CompactTextString(m) } +func (*PrivateKey) ProtoMessage() {} +func (*PrivateKey) Descriptor() ([]byte, []int) { + return fileDescriptor_527278fb02d03321, []int{1} +} +func (m *PrivateKey) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *PrivateKey) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_PrivateKey.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *PrivateKey) XXX_Merge(src proto.Message) { + xxx_messageInfo_PrivateKey.Merge(m, src) +} +func (m *PrivateKey) XXX_Size() int { + return m.Size() +} +func (m *PrivateKey) XXX_DiscardUnknown() { + xxx_messageInfo_PrivateKey.DiscardUnknown(m) +} + +var xxx_messageInfo_PrivateKey proto.InternalMessageInfo + +func (m *PrivateKey) GetType() KeyType { + if m != nil { + return m.Type + } + return KeyType_RSA +} + +func (m *PrivateKey) GetData() []byte { + if m != nil { + return m.Data + } + return nil +} + +func init() { + proto.RegisterEnum("crypto.pb.KeyType", KeyType_name, KeyType_value) + proto.RegisterType((*PublicKey)(nil), "crypto.pb.PublicKey") + proto.RegisterType((*PrivateKey)(nil), "crypto.pb.PrivateKey") +} + +func init() { proto.RegisterFile("crypto.proto", fileDescriptor_527278fb02d03321) } + +var fileDescriptor_527278fb02d03321 = []byte{ + // 203 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x49, 0x2e, 0xaa, 0x2c, + 0x28, 0xc9, 0xd7, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x84, 0xf1, 0x92, 0x94, 0x82, 0xb9, + 0x38, 0x03, 0x4a, 0x93, 0x72, 0x32, 0x93, 0xbd, 0x53, 0x2b, 0x85, 0x74, 0xb8, 0x58, 0x42, 0x2a, + 0x0b, 0x52, 0x25, 0x18, 0x15, 0x98, 0x34, 0xf8, 0x8c, 0x84, 0xf4, 0xe0, 0xca, 0xf4, 0xbc, 0x53, + 0x2b, 0x41, 0x32, 0x4e, 0x2c, 0x27, 0xee, 0xc9, 0x33, 0x04, 0x81, 0x55, 0x09, 0x49, 0x70, 0xb1, + 0xb8, 0x24, 0x96, 0x24, 0x4a, 0x30, 0x29, 0x30, 0x69, 0xf0, 0xc0, 0x64, 0x40, 0x22, 0x4a, 0x21, + 0x5c, 0x5c, 0x01, 0x45, 0x99, 0x65, 0x89, 0x25, 0xa9, 0x54, 0x34, 0x55, 0xcb, 0x92, 0x8b, 0x1d, + 0xaa, 0x41, 0x88, 0x9d, 0x8b, 0x39, 0x28, 0xd8, 0x51, 0x80, 0x41, 0x88, 0x9b, 0x8b, 0xdd, 0x35, + 0xc5, 0xc8, 0xd4, 0xd4, 0xd0, 0x52, 0x80, 0x51, 0x88, 0x97, 0x8b, 0x33, 0x38, 0x35, 0xb9, 0xc0, + 0xc8, 0xd4, 0x2c, 0xdb, 0x50, 0x80, 0x49, 0x88, 0x93, 0x8b, 0xd5, 0xd5, 0xd9, 0x25, 0xd8, 0x51, + 0x80, 0xd9, 0x49, 0xe2, 0xc4, 0x23, 0x39, 0xc6, 0x0b, 0x8f, 0xe4, 0x18, 0x1f, 0x3c, 0x92, 0x63, + 0x9c, 0xf0, 0x58, 0x8e, 0xe1, 0xc2, 0x63, 0x39, 0x86, 0x1b, 0x8f, 0xe5, 0x18, 0x00, 0x01, 0x00, + 0x00, 0xff, 0xff, 0x13, 0xbe, 0xd4, 0xff, 0x19, 0x01, 0x00, 0x00, +} + +func (m *PublicKey) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *PublicKey) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + dAtA[i] = 0x8 + i++ + i = encodeVarintCrypto(dAtA, i, uint64(m.Type)) + if m.Data != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintCrypto(dAtA, i, uint64(len(m.Data))) + i += copy(dAtA[i:], m.Data) + } + return i, nil +} + +func (m *PrivateKey) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *PrivateKey) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + dAtA[i] = 0x8 + i++ + i = encodeVarintCrypto(dAtA, i, uint64(m.Type)) + if m.Data != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintCrypto(dAtA, i, uint64(len(m.Data))) + i += copy(dAtA[i:], m.Data) + } + return i, nil +} + +func encodeVarintCrypto(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func (m *PublicKey) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + n += 1 + sovCrypto(uint64(m.Type)) + if m.Data != nil { + l = len(m.Data) + n += 1 + l + sovCrypto(uint64(l)) + } + return n +} + +func (m *PrivateKey) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + n += 1 + sovCrypto(uint64(m.Type)) + if m.Data != nil { + l = len(m.Data) + n += 1 + l + sovCrypto(uint64(l)) + } + return n +} + +func sovCrypto(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozCrypto(x uint64) (n int) { + return sovCrypto(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *PublicKey) Unmarshal(dAtA []byte) error { + var hasFields [1]uint64 + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowCrypto + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: PublicKey: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: PublicKey: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Type", wireType) + } + m.Type = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowCrypto + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.Type |= KeyType(b&0x7F) << shift + if b < 0x80 { + break + } + } + hasFields[0] |= uint64(0x00000001) + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowCrypto + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthCrypto + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthCrypto + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + hasFields[0] |= uint64(0x00000002) + default: + iNdEx = preIndex + skippy, err := skipCrypto(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthCrypto + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthCrypto + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + if hasFields[0]&uint64(0x00000001) == 0 { + return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Type") + } + if hasFields[0]&uint64(0x00000002) == 0 { + return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Data") + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *PrivateKey) Unmarshal(dAtA []byte) error { + var hasFields [1]uint64 + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowCrypto + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: PrivateKey: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: PrivateKey: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Type", wireType) + } + m.Type = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowCrypto + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.Type |= KeyType(b&0x7F) << shift + if b < 0x80 { + break + } + } + hasFields[0] |= uint64(0x00000001) + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowCrypto + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthCrypto + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthCrypto + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + hasFields[0] |= uint64(0x00000002) + default: + iNdEx = preIndex + skippy, err := skipCrypto(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthCrypto + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthCrypto + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + if hasFields[0]&uint64(0x00000001) == 0 { + return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Type") + } + if hasFields[0]&uint64(0x00000002) == 0 { + return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Data") + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipCrypto(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowCrypto + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowCrypto + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowCrypto + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if length < 0 { + return 0, ErrInvalidLengthCrypto + } + iNdEx += length + if iNdEx < 0 { + return 0, ErrInvalidLengthCrypto + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowCrypto + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipCrypto(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + if iNdEx < 0 { + return 0, ErrInvalidLengthCrypto + } + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthCrypto = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowCrypto = fmt.Errorf("proto: integer overflow") +) diff --git a/core/crypto/pb/crypto.proto b/core/crypto/pb/crypto.proto new file mode 100644 index 0000000000..cb5cee8a22 --- /dev/null +++ b/core/crypto/pb/crypto.proto @@ -0,0 +1,20 @@ +syntax = "proto2"; + +package crypto.pb; + +enum KeyType { + RSA = 0; + Ed25519 = 1; + Secp256k1 = 2; + ECDSA = 3; +} + +message PublicKey { + required KeyType Type = 1; + required bytes Data = 2; +} + +message PrivateKey { + required KeyType Type = 1; + required bytes Data = 2; +} diff --git a/core/crypto/rsa_common.go b/core/crypto/rsa_common.go new file mode 100644 index 0000000000..d50651f218 --- /dev/null +++ b/core/crypto/rsa_common.go @@ -0,0 +1,10 @@ +package crypto + +import ( + "errors" +) + +// ErrRsaKeyTooSmall is returned when trying to generate or parse an RSA key +// that's smaller than 512 bits. Keys need to be larger enough to sign a 256bit +// hash so this is a reasonable absolute minimum. +var ErrRsaKeyTooSmall = errors.New("rsa keys must be >= 512 bits to be useful") diff --git a/core/crypto/rsa_go.go b/core/crypto/rsa_go.go new file mode 100644 index 0000000000..e9813779dc --- /dev/null +++ b/core/crypto/rsa_go.go @@ -0,0 +1,125 @@ +// +build !openssl + +package crypto + +import ( + "crypto" + "crypto/rand" + "crypto/rsa" + "crypto/x509" + "errors" + "io" + + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + + "github.com/minio/sha256-simd" +) + +// RsaPrivateKey is an rsa private key +type RsaPrivateKey struct { + sk rsa.PrivateKey +} + +// RsaPublicKey is an rsa public key +type RsaPublicKey struct { + k rsa.PublicKey +} + +// GenerateRSAKeyPair generates a new rsa private and public key +func GenerateRSAKeyPair(bits int, src io.Reader) (PrivKey, PubKey, error) { + if bits < 512 { + return nil, nil, ErrRsaKeyTooSmall + } + priv, err := rsa.GenerateKey(src, bits) + if err != nil { + return nil, nil, err + } + pk := priv.PublicKey + return &RsaPrivateKey{sk: *priv}, &RsaPublicKey{pk}, nil +} + +// Verify compares a signature against input data +func (pk *RsaPublicKey) Verify(data, sig []byte) (bool, error) { + hashed := sha256.Sum256(data) + err := rsa.VerifyPKCS1v15(&pk.k, crypto.SHA256, hashed[:], sig) + if err != nil { + return false, err + } + return true, nil +} + +func (pk *RsaPublicKey) Type() pb.KeyType { + return pb.KeyType_RSA +} + +// Bytes returns protobuf bytes of a public key +func (pk *RsaPublicKey) Bytes() ([]byte, error) { + return MarshalPublicKey(pk) +} + +func (pk *RsaPublicKey) Raw() ([]byte, error) { + return x509.MarshalPKIXPublicKey(&pk.k) +} + +// Equals checks whether this key is equal to another +func (pk *RsaPublicKey) Equals(k Key) bool { + return KeyEqual(pk, k) +} + +// Sign returns a signature of the input data +func (sk *RsaPrivateKey) Sign(message []byte) ([]byte, error) { + hashed := sha256.Sum256(message) + return rsa.SignPKCS1v15(rand.Reader, &sk.sk, crypto.SHA256, hashed[:]) +} + +// GetPublic returns a public key +func (sk *RsaPrivateKey) GetPublic() PubKey { + return &RsaPublicKey{sk.sk.PublicKey} +} + +func (sk *RsaPrivateKey) Type() pb.KeyType { + return pb.KeyType_RSA +} + +// Bytes returns protobuf bytes from a private key +func (sk *RsaPrivateKey) Bytes() ([]byte, error) { + return MarshalPrivateKey(sk) +} + +func (sk *RsaPrivateKey) Raw() ([]byte, error) { + b := x509.MarshalPKCS1PrivateKey(&sk.sk) + return b, nil +} + +// Equals checks whether this key is equal to another +func (sk *RsaPrivateKey) Equals(k Key) bool { + return KeyEqual(sk, k) +} + +// UnmarshalRsaPrivateKey returns a private key from the input x509 bytes +func UnmarshalRsaPrivateKey(b []byte) (PrivKey, error) { + sk, err := x509.ParsePKCS1PrivateKey(b) + if err != nil { + return nil, err + } + if sk.N.BitLen() < 512 { + return nil, ErrRsaKeyTooSmall + } + return &RsaPrivateKey{sk: *sk}, nil +} + +// UnmarshalRsaPublicKey returns a public key from the input x509 bytes +func UnmarshalRsaPublicKey(b []byte) (PubKey, error) { + pub, err := x509.ParsePKIXPublicKey(b) + if err != nil { + return nil, err + } + pk, ok := pub.(*rsa.PublicKey) + if !ok { + return nil, errors.New("not actually an rsa public key") + } + if pk.N.BitLen() < 512 { + return nil, ErrRsaKeyTooSmall + } + return &RsaPublicKey{*pk}, nil +} diff --git a/core/crypto/rsa_openssl.go b/core/crypto/rsa_openssl.go new file mode 100644 index 0000000000..96c5588615 --- /dev/null +++ b/core/crypto/rsa_openssl.go @@ -0,0 +1,62 @@ +// +build openssl + +package crypto + +import ( + "errors" + "io" + + openssl "github.com/spacemonkeygo/openssl" +) + +// RsaPrivateKey is an rsa private key +type RsaPrivateKey struct { + opensslPrivateKey +} + +// RsaPublicKey is an rsa public key +type RsaPublicKey struct { + opensslPublicKey +} + +// GenerateRSAKeyPair generates a new rsa private and public key +func GenerateRSAKeyPair(bits int, _ io.Reader) (PrivKey, PubKey, error) { + if bits < 512 { + return nil, nil, ErrRsaKeyTooSmall + } + + key, err := openssl.GenerateRSAKey(bits) + if err != nil { + return nil, nil, err + } + return &RsaPrivateKey{opensslPrivateKey{key}}, &RsaPublicKey{opensslPublicKey{key}}, nil +} + +// GetPublic returns a public key +func (sk *RsaPrivateKey) GetPublic() PubKey { + return &RsaPublicKey{opensslPublicKey{sk.opensslPrivateKey.key}} +} + +// UnmarshalRsaPrivateKey returns a private key from the input x509 bytes +func UnmarshalRsaPrivateKey(b []byte) (PrivKey, error) { + key, err := unmarshalOpensslPrivateKey(b) + if err != nil { + return nil, err + } + if key.Type() != RSA { + return nil, errors.New("not actually an rsa public key") + } + return &RsaPrivateKey{key}, nil +} + +// UnmarshalRsaPublicKey returns a public key from the input x509 bytes +func UnmarshalRsaPublicKey(b []byte) (PubKey, error) { + key, err := unmarshalOpensslPublicKey(b) + if err != nil { + return nil, err + } + if key.Type() != RSA { + return nil, errors.New("not actually an rsa public key") + } + return &RsaPublicKey{key}, nil +} diff --git a/core/crypto/rsa_test.go b/core/crypto/rsa_test.go new file mode 100644 index 0000000000..7ee520acb7 --- /dev/null +++ b/core/crypto/rsa_test.go @@ -0,0 +1,102 @@ +package crypto + +import ( + "crypto/rand" + "testing" +) + +func TestRSABasicSignAndVerify(t *testing.T) { + priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + if err != nil { + t.Fatal(err) + } + + data := []byte("hello! and welcome to some awesome crypto primitives") + + sig, err := priv.Sign(data) + if err != nil { + t.Fatal(err) + } + + ok, err := pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + + if !ok { + t.Fatal("signature didnt match") + } + + // change data + data[0] = ^data[0] + ok, err = pub.Verify(data, sig) + if err == nil { + t.Fatal("should have produced a verification error") + } + + if ok { + t.Fatal("signature matched and shouldn't") + } +} + +func TestRSASmallKey(t *testing.T) { + _, _, err := GenerateRSAKeyPair(384, rand.Reader) + if err != ErrRsaKeyTooSmall { + t.Fatal("should have refused to create small RSA key") + } +} + +func TestRSASignZero(t *testing.T) { + priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + if err != nil { + t.Fatal(err) + } + + data := make([]byte, 0) + sig, err := priv.Sign(data) + if err != nil { + t.Fatal(err) + } + + ok, err := pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + if !ok { + t.Fatal("signature didn't match") + } +} + +func TestRSAMarshalLoop(t *testing.T) { + priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + if err != nil { + t.Fatal(err) + } + + privB, err := priv.Bytes() + if err != nil { + t.Fatal(err) + } + + privNew, err := UnmarshalPrivateKey(privB) + if err != nil { + t.Fatal(err) + } + + if !priv.Equals(privNew) || !privNew.Equals(priv) { + t.Fatal("keys are not equal") + } + + pubB, err := pub.Bytes() + if err != nil { + t.Fatal(err) + } + pubNew, err := UnmarshalPublicKey(pubB) + if err != nil { + t.Fatal(err) + } + + if !pub.Equals(pubNew) || !pubNew.Equals(pub) { + t.Fatal("keys are not equal") + } +} diff --git a/core/crypto/secp256k1.go b/core/crypto/secp256k1.go new file mode 100644 index 0000000000..d2ac74b996 --- /dev/null +++ b/core/crypto/secp256k1.go @@ -0,0 +1,125 @@ +package crypto + +import ( + "fmt" + "io" + + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + + btcec "github.com/btcsuite/btcd/btcec" + sha256 "github.com/minio/sha256-simd" +) + +// Secp256k1PrivateKey is an Secp256k1 private key +type Secp256k1PrivateKey btcec.PrivateKey + +// Secp256k1PublicKey is an Secp256k1 public key +type Secp256k1PublicKey btcec.PublicKey + +// GenerateSecp256k1Key generates a new Secp256k1 private and public key pair +func GenerateSecp256k1Key(src io.Reader) (PrivKey, PubKey, error) { + privk, err := btcec.NewPrivateKey(btcec.S256()) + if err != nil { + return nil, nil, err + } + + k := (*Secp256k1PrivateKey)(privk) + return k, k.GetPublic(), nil +} + +// UnmarshalSecp256k1PrivateKey returns a private key from bytes +func UnmarshalSecp256k1PrivateKey(data []byte) (PrivKey, error) { + if len(data) != btcec.PrivKeyBytesLen { + return nil, fmt.Errorf("expected secp256k1 data size to be %d", btcec.PrivKeyBytesLen) + } + + privk, _ := btcec.PrivKeyFromBytes(btcec.S256(), data) + return (*Secp256k1PrivateKey)(privk), nil +} + +// UnmarshalSecp256k1PublicKey returns a public key from bytes +func UnmarshalSecp256k1PublicKey(data []byte) (PubKey, error) { + k, err := btcec.ParsePubKey(data, btcec.S256()) + if err != nil { + return nil, err + } + + return (*Secp256k1PublicKey)(k), nil +} + +// Bytes returns protobuf bytes from a private key +func (k *Secp256k1PrivateKey) Bytes() ([]byte, error) { + return MarshalPrivateKey(k) +} + +// Type returns the private key type +func (k *Secp256k1PrivateKey) Type() pb.KeyType { + return pb.KeyType_Secp256k1 +} + +// Raw returns the bytes of the key +func (k *Secp256k1PrivateKey) Raw() ([]byte, error) { + return (*btcec.PrivateKey)(k).Serialize(), nil +} + +// Equals compares two private keys +func (k *Secp256k1PrivateKey) Equals(o Key) bool { + sk, ok := o.(*Secp256k1PrivateKey) + if !ok { + return false + } + + return k.D.Cmp(sk.D) == 0 +} + +// Sign returns a signature from input data +func (k *Secp256k1PrivateKey) Sign(data []byte) ([]byte, error) { + hash := sha256.Sum256(data) + sig, err := (*btcec.PrivateKey)(k).Sign(hash[:]) + if err != nil { + return nil, err + } + + return sig.Serialize(), nil +} + +// GetPublic returns a public key +func (k *Secp256k1PrivateKey) GetPublic() PubKey { + return (*Secp256k1PublicKey)((*btcec.PrivateKey)(k).PubKey()) +} + +// Bytes returns protobuf bytes from a public key +func (k *Secp256k1PublicKey) Bytes() ([]byte, error) { + return MarshalPublicKey(k) +} + +// Type returns the public key type +func (k *Secp256k1PublicKey) Type() pb.KeyType { + return pb.KeyType_Secp256k1 +} + +// Raw returns the bytes of the key +func (k *Secp256k1PublicKey) Raw() ([]byte, error) { + return (*btcec.PublicKey)(k).SerializeCompressed(), nil +} + +// Equals compares two public keys +func (k *Secp256k1PublicKey) Equals(o Key) bool { + sk, ok := o.(*Secp256k1PublicKey) + if !ok { + return false + } + + return (*btcec.PublicKey)(k).IsEqual((*btcec.PublicKey)(sk)) +} + +// Verify compares a signature against the input data +func (k *Secp256k1PublicKey) Verify(data []byte, sigStr []byte) (bool, error) { + sig, err := btcec.ParseDERSignature(sigStr, btcec.S256()) + if err != nil { + return false, err + } + + hash := sha256.Sum256(data) + return sig.Verify(hash[:], (*btcec.PublicKey)(k)), nil +} diff --git a/core/crypto/secp256k1_test.go b/core/crypto/secp256k1_test.go new file mode 100644 index 0000000000..aff2bd9773 --- /dev/null +++ b/core/crypto/secp256k1_test.go @@ -0,0 +1,96 @@ +package crypto + +import ( + "crypto/rand" + "testing" +) + +func TestSecp256k1BasicSignAndVerify(t *testing.T) { + priv, pub, err := GenerateSecp256k1Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + + data := []byte("hello! and welcome to some awesome crypto primitives") + + sig, err := priv.Sign(data) + if err != nil { + t.Fatal(err) + } + + ok, err := pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + + if !ok { + t.Fatal("signature didnt match") + } + + // change data + data[0] = ^data[0] + ok, err = pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + + if ok { + t.Fatal("signature matched and shouldn't") + } +} + +func TestSecp256k1SignZero(t *testing.T) { + priv, pub, err := GenerateSecp256k1Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + + data := make([]byte, 0) + sig, err := priv.Sign(data) + if err != nil { + t.Fatal(err) + } + + ok, err := pub.Verify(data, sig) + if err != nil { + t.Fatal(err) + } + if !ok { + t.Fatal("signature didn't match") + } +} + +func TestSecp256k1MarshalLoop(t *testing.T) { + priv, pub, err := GenerateSecp256k1Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + + privB, err := priv.Bytes() + if err != nil { + t.Fatal(err) + } + + privNew, err := UnmarshalPrivateKey(privB) + if err != nil { + t.Fatal(err) + } + + if !priv.Equals(privNew) || !privNew.Equals(priv) { + t.Fatal("keys are not equal") + } + + pubB, err := pub.Bytes() + if err != nil { + t.Fatal(err) + } + pubNew, err := UnmarshalPublicKey(pubB) + if err != nil { + t.Fatal(err) + } + + if !pub.Equals(pubNew) || !pubNew.Equals(pub) { + t.Fatal("keys are not equal") + } + +} diff --git a/core/crypto/test_data/0.priv b/core/crypto/test_data/0.priv new file mode 100644 index 0000000000000000000000000000000000000000..9047d5d95da8d54887fe5983a9e9e4d9017db511 GIT binary patch literal 1198 zcmV;f1X24401~MQFoFc70s#O5f&l>l^0!+-j417TtNs`14f~V9KQ6{N9S1LlFSl3- zxoskT?;wNIA4+%zpM|YzjB|-@qlKe}Lk%3jnvkL}@Cgx7Tva&4wk?T&#%13TP(^C& zblGnC8FJR5NY;c!Jmu5l>)EGR6zp4aTeyfT1bw@%01DWOQZ`v?*Z{=L`s2qRA`{(D+=J7bt9;Dl zpJn3S=f`Z9vc=bs>`u1XRH|+irBq{Op5F>36Tqt3 zxHO+7nwwx-J#pT|J8AA~=uxLr0s{d60Rn;n08<2nA{Sn$Tb&qt^(x~r)|XY-R=F1p z8cCPjbf0&`q{aow5gk&?TKJVPi;*6)lQcUqgKEpcS>J>>^`sNb1S!5D=t6zl+7Rjk( zYND*U&2#Orgq!XHhDWt_;Vy)`iN%bETqDILV3#64n=yFRX69Il((zQS%1r(3 z)4oWbyKgSs{ei4Z!~Ts4+wm2T4T+xXbdCOFmlHiz62dqzSNXWIX=IxZZo9&52Y=nS z=;brvC%D1OT7a&--qVUd3I7ICl5l!y~fO`9SQvI zg0!T(6}2&dnA<`tw3f#7Xv^Z)CuNy-gBhV2EM8c8(H8t9u>GvwtXD2&atS%P*mt)6rro&N1R{dFHK#nb=*x z{{n%50Q|R!9!0VLp_YTv$s^t&jkINb*awC$09>!A_9$@`(foFoJ`iQ~xu^|I#UJ`3 zGvM(GL}#tQ@$6^i#37j}X?vvC`xdB(J_etSi2q0V>|$+4v23lY8z3)%C4dUMiX2c9 z>_jsMG{PR^#%&IMJ&beVhD4@r$Fv=p4uX!wtpb680OZB?3gY~S@sW&OrCjG)vcjw6 zA9WA74nA+0;>h|6qA)?-(B}%Um|{Ka+-R8Q!40$QY;a8b+FAl;W<+Y^#=1Ff0xq1w zl80+cuNpj!0_8lJtH8a%dd(P2z@IXPOdS50Z>T-7g$x1os%GPZ;M>mc?Q~TzUX$(6>} zBM$@ESv#mV9%E*+CMQQ1RsuSdGWk(THG!nd?0@KmdrcNaICJ;pFslnqP`uf?`rITK z$k`a24#2p(+~*Y&bS0si0dhd@uO;#2D)eIn_jca(xdMTK0Ib$_%b!(+j$2`MrZO8* zK0VVe#NRZ66g;0%>I1jW4sv-NuP46&Xiyr|N*J&>_rbgur@oH$9`#uJs%zkUQ%aHfT;!I2{KshA+2R2)S({e(xZI(;rHB2A_qkYK(JY;z?zVv zFz^WxQCw9x#I`Mof5v6s5l}^H>~+nuF${?E2b_ZSYI!$YdF1#&@aq*JKN)h?qDa<+ zMLgxxdIw-?GKm xknB#j*;J}-6s1&SWuD&(B@@7^*|;>HC7PRHTRm~!#5-y3Yv@s@Qvw4400F=}h;aY_ literal 0 HcmV?d00001 diff --git a/core/crypto/test_data/0.sig b/core/crypto/test_data/0.sig new file mode 100644 index 0000000000000000000000000000000000000000..2f16825274520604c4d67c5dc4a1efd11c4cb5a0 GIT binary patch literal 256 zcmV+b0ssDP5fdf%5Un{zGtbk0Mt&H@6Mwcn7%ZEPqeF&re1kDy+yM>l5^mzU^EDO! zS%)zD!_$Fo#oNXv__1;bLna{ikzc+WZd%SDZ9ln;Ka<+9=)s>16_V{NiZx5j)4 zz$b0Y=dnKD literal 0 HcmV?d00001 diff --git a/core/crypto/test_data/2.priv b/core/crypto/test_data/2.priv new file mode 100644 index 0000000000..1004ab9153 --- /dev/null +++ b/core/crypto/test_data/2.priv @@ -0,0 +1 @@ + 1A½`jPLDò4”ØóNò[¹µ-ªÐX¾ƒ¶àF±X \ No newline at end of file diff --git a/core/crypto/test_data/2.pub b/core/crypto/test_data/2.pub new file mode 100644 index 0000000000..dd984bdcc8 --- /dev/null +++ b/core/crypto/test_data/2.pub @@ -0,0 +1 @@ +!5@­„*ø¤5Q©Mƒƒ¥U©’&PkˆùSÒ磡³Ö¢ \ No newline at end of file diff --git a/core/crypto/test_data/2.sig b/core/crypto/test_data/2.sig new file mode 100644 index 0000000000..b96001bc8b --- /dev/null +++ b/core/crypto/test_data/2.sig @@ -0,0 +1 @@ +0D 1§Ö3ÂóäZCuú¨Ü›¢@’Åõ³ÊÒÈþLóˆò Iê!ŸE†ƒÒGuÕCê²pCGû5I<@;ÂÂY²ž \ No newline at end of file diff --git a/core/crypto/test_data/3.priv b/core/crypto/test_data/3.priv new file mode 100644 index 0000000000000000000000000000000000000000..7a05f359f803dfd03a776c8648e43dd501f55fe0 GIT binary patch literal 125 zcmV-@0D}Jr0}^>KcLD(c1Rx*4R7vBbLOZy%lt-jIJ|nF_m8?&nyLCl+&htxXl%Jpq z1_&yKNX|V20SBQ(13~}<;+lqZ^I=X5%W^<#{vkK%_{I0uEfaqVF5M$OH8)3h{U4g` fO)UCzamug8?*iL{xF2`V;k?+#6|C^NqIZs|4V^dR literal 0 HcmV?d00001 diff --git a/core/crypto/test_data/3.pub b/core/crypto/test_data/3.pub new file mode 100644 index 0000000000000000000000000000000000000000..f4551f8811180cad4e160e2db6dfd675c774cb69 GIT binary patch literal 95 zcmd;J7K%2AG!SNE*J|@PXUoLM#sOw9GqN)~F|a(E)mHL3(U0eJkwf-BMe~-_MYi>^A7p)^AGpjI4-*8!_GzJeJcS7%qWZi literal 0 HcmV?d00001 diff --git a/core/crypto/test_data/3.sig b/core/crypto/test_data/3.sig new file mode 100644 index 0000000000000000000000000000000000000000..253c09f6d7b2bd30be7d49abbda27ea1bfd01043 GIT binary patch literal 72 zcmV-O0Jr}zMgk!KglcAmx#ufZ46^)6b@|J~ijb=6r}Oe&=NW@7x4AlH0wDm+nJc%w eE6gaz+XR?>5Us?J{xpJcl}~n-%MJhk4g5X!9waUR literal 0 HcmV?d00001 diff --git a/core/discovery/discovery.go b/core/discovery/discovery.go new file mode 100644 index 0000000000..f463e0e852 --- /dev/null +++ b/core/discovery/discovery.go @@ -0,0 +1,27 @@ +// Package discovery provides service advertisement and peer discovery interfaces for libp2p. +package discovery + +import ( + "context" + "time" + + "github.com/libp2p/go-libp2p-core/peer" +) + +// Advertiser is an interface for advertising services +type Advertiser interface { + // Advertise advertises a service + Advertise(ctx context.Context, ns string, opts ...Option) (time.Duration, error) +} + +// Discoverer is an interface for peer discovery +type Discoverer interface { + // FindPeers discovers peers providing a service + FindPeers(ctx context.Context, ns string, opts ...Option) (<-chan peer.AddrInfo, error) +} + +// Discovery is an interface that combines service advertisement and peer discovery +type Discovery interface { + Advertiser + Discoverer +} diff --git a/core/discovery/options.go b/core/discovery/options.go new file mode 100644 index 0000000000..7b28305268 --- /dev/null +++ b/core/discovery/options.go @@ -0,0 +1,41 @@ +package discovery + +import "time" + +// DiscoveryOpt is a single discovery option. +type Option func(opts *Options) error + +// DiscoveryOpts is a set of discovery options. +type Options struct { + Ttl time.Duration + Limit int + + // Other (implementation-specific) options + Other map[interface{}]interface{} +} + +// Apply applies the given options to this DiscoveryOpts +func (opts *Options) Apply(options ...Option) error { + for _, o := range options { + if err := o(opts); err != nil { + return err + } + } + return nil +} + +// TTL is an option that provides a hint for the duration of an advertisement +func TTL(ttl time.Duration) Option { + return func(opts *Options) error { + opts.Ttl = ttl + return nil + } +} + +// Limit is an option that provides an upper bound on the peer count for discovery +func Limit(limit int) Option { + return func(opts *Options) error { + opts.Limit = limit + return nil + } +} diff --git a/core/helpers/match.go b/core/helpers/match.go new file mode 100644 index 0000000000..2e24667897 --- /dev/null +++ b/core/helpers/match.go @@ -0,0 +1,42 @@ +package helpers + +import ( + "strings" + + "github.com/coreos/go-semver/semver" + "github.com/libp2p/go-libp2p-core/protocol" +) + +// MultistreamSemverMatcher returns a matcher function for a given base protocol. +// The matcher function will return a boolean indicating whether a protocol ID +// matches the base protocol. A given protocol ID matches the base protocol if +// the IDs are the same and if the semantic version of the base protocol is the +// same or higher than that of the protocol ID provided. +// TODO +func MultistreamSemverMatcher(base protocol.ID) (func(string) bool, error) { + parts := strings.Split(string(base), "/") + vers, err := semver.NewVersion(parts[len(parts)-1]) + if err != nil { + return nil, err + } + + return func(check string) bool { + chparts := strings.Split(check, "/") + if len(chparts) != len(parts) { + return false + } + + for i, v := range chparts[:len(chparts)-1] { + if parts[i] != v { + return false + } + } + + chvers, err := semver.NewVersion(chparts[len(chparts)-1]) + if err != nil { + return false + } + + return vers.Major == chvers.Major && vers.Minor >= chvers.Minor + }, nil +} diff --git a/core/helpers/match_test.go b/core/helpers/match_test.go new file mode 100644 index 0000000000..8a6e4b678d --- /dev/null +++ b/core/helpers/match_test.go @@ -0,0 +1,33 @@ +package helpers + +import ( + "testing" +) + +func TestSemverMatching(t *testing.T) { + m, err := MultistreamSemverMatcher("/testing/4.3.5") + if err != nil { + t.Fatal(err) + } + + cases := map[string]bool{ + "/testing/4.3.0": true, + "/testing/4.3.7": true, + "/testing/4.3.5": true, + "/testing/4.2.7": true, + "/testing/4.0.0": true, + "/testing/5.0.0": false, + "/cars/dogs/4.3.5": false, + "/foo/1.0.0": false, + "": false, + "dogs": false, + "/foo": false, + "/foo/1.1.1.1": false, + } + + for p, ok := range cases { + if m(p) != ok { + t.Fatalf("expected %s to be %t", p, ok) + } + } +} diff --git a/core/helpers/stream.go b/core/helpers/stream.go new file mode 100644 index 0000000000..0b4c1f3e00 --- /dev/null +++ b/core/helpers/stream.go @@ -0,0 +1,56 @@ +package helpers + +import ( + "errors" + "io" + "time" + + "github.com/libp2p/go-libp2p-core/network" +) + +// EOFTimeout is the maximum amount of time to wait to successfully observe an +// EOF on the stream. Defaults to 60 seconds. +var EOFTimeout = time.Second * 60 + +// ErrExpectedEOF is returned when we read data while expecting an EOF. +var ErrExpectedEOF = errors.New("read data when expecting EOF") + +// FullClose closes the stream and waits to read an EOF from the other side. +// +// * If it reads any data *before* the EOF, it resets the stream. +// * If it doesn't read an EOF within EOFTimeout, it resets the stream. +// +// You'll likely want to invoke this as `go FullClose(stream)` to close the +// stream in the background. +func FullClose(s network.Stream) error { + if err := s.Close(); err != nil { + s.Reset() + return err + } + return AwaitEOF(s) +} + +// AwaitEOF waits for an EOF on the given stream, returning an error if that +// fails. It waits at most EOFTimeout (defaults to 1 minute) after which it +// resets the stream. +func AwaitEOF(s network.Stream) error { + // So we don't wait forever + s.SetDeadline(time.Now().Add(EOFTimeout)) + + // We *have* to observe the EOF. Otherwise, we leak the stream. + // Now, technically, we should do this *before* + // returning from SendMessage as the message + // hasn't really been sent yet until we see the + // EOF but we don't actually *know* what + // protocol the other side is speaking. + n, err := s.Read([]byte{0}) + if n > 0 || err == nil { + s.Reset() + return ErrExpectedEOF + } + if err != io.EOF { + s.Reset() + return err + } + return nil +} diff --git a/core/helpers/stream_test.go b/core/helpers/stream_test.go new file mode 100644 index 0000000000..d267f1ba3d --- /dev/null +++ b/core/helpers/stream_test.go @@ -0,0 +1,151 @@ +package helpers_test + +import ( + "errors" + "io" + "testing" + "time" + + "github.com/libp2p/go-libp2p-core/helpers" + network "github.com/libp2p/go-libp2p-core/network" +) + +var errCloseFailed = errors.New("close failed") +var errWriteFailed = errors.New("write failed") +var errReadFailed = errors.New("read failed") + +type stream struct { + network.Stream + + data []byte + + failRead, failWrite, failClose bool + + reset bool +} + +func (s *stream) Reset() error { + s.reset = true + return nil +} + +func (s *stream) Close() error { + if s.failClose { + return errCloseFailed + } + return nil +} + +func (s *stream) SetDeadline(t time.Time) error { + s.SetReadDeadline(t) + s.SetWriteDeadline(t) + return nil +} + +func (s *stream) SetReadDeadline(t time.Time) error { + return nil +} + +func (s *stream) SetWriteDeadline(t time.Time) error { + return nil +} + +func (s *stream) Write(b []byte) (int, error) { + if s.failWrite { + return 0, errWriteFailed + } + return len(b), nil +} + +func (s *stream) Read(b []byte) (int, error) { + var err error + if s.failRead { + err = errReadFailed + } + if len(s.data) == 0 { + if err == nil { + err = io.EOF + } + return 0, err + } + n := copy(b, s.data) + s.data = s.data[n:] + return n, err +} + +func TestNormal(t *testing.T) { + var s stream + if err := helpers.FullClose(&s); err != nil { + t.Fatal(err) + } + if s.reset { + t.Fatal("stream should not have been reset") + } +} + +func TestFailRead(t *testing.T) { + var s stream + s.failRead = true + if helpers.FullClose(&s) != errReadFailed { + t.Fatal("expected read to fail with:", errReadFailed) + } + if !s.reset { + t.Fatal("expected stream to be reset") + } +} + +func TestFailClose(t *testing.T) { + var s stream + s.failClose = true + if helpers.FullClose(&s) != errCloseFailed { + t.Fatal("expected close to fail with:", errCloseFailed) + } + if !s.reset { + t.Fatal("expected stream to be reset") + } +} + +func TestFailWrite(t *testing.T) { + var s stream + s.failWrite = true + if err := helpers.FullClose(&s); err != nil { + t.Fatal(err) + } + if s.reset { + t.Fatal("stream should not have been reset") + } +} + +func TestReadDataOne(t *testing.T) { + var s stream + s.data = []byte{0} + if err := helpers.FullClose(&s); err != helpers.ErrExpectedEOF { + t.Fatal("expected:", helpers.ErrExpectedEOF) + } + if !s.reset { + t.Fatal("stream have been reset") + } +} + +func TestReadDataMany(t *testing.T) { + var s stream + s.data = []byte{0, 1, 2, 3} + if err := helpers.FullClose(&s); err != helpers.ErrExpectedEOF { + t.Fatal("expected:", helpers.ErrExpectedEOF) + } + if !s.reset { + t.Fatal("stream have been reset") + } +} + +func TestReadDataError(t *testing.T) { + var s stream + s.data = []byte{0, 1, 2, 3} + s.failRead = true + if err := helpers.FullClose(&s); err != helpers.ErrExpectedEOF { + t.Fatal("expected:", helpers.ErrExpectedEOF) + } + if !s.reset { + t.Fatal("stream have been reset") + } +} diff --git a/core/host/helpers.go b/core/host/helpers.go new file mode 100644 index 0000000000..a24beb1bbe --- /dev/null +++ b/core/host/helpers.go @@ -0,0 +1,11 @@ +package host + +import "github.com/libp2p/go-libp2p-core/peer" + +// InfoFromHost returns a peer.AddrInfo struct with the Host's ID and all of its Addrs. +func InfoFromHost(h Host) *peer.AddrInfo { + return &peer.AddrInfo{ + ID: h.ID(), + Addrs: h.Addrs(), + } +} diff --git a/core/host/host.go b/core/host/host.go new file mode 100644 index 0000000000..d4b3c03fdc --- /dev/null +++ b/core/host/host.go @@ -0,0 +1,71 @@ +// Package host provides the core Host interface for libp2p. +// +// Host represents a single libp2p node in a peer-to-peer network. +package host + +import ( + "context" + + "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/protocol" + + ma "github.com/multiformats/go-multiaddr" +) + +// Host is an object participating in a p2p network, which +// implements protocols or provides services. It handles +// requests like a Server, and issues requests like a Client. +// It is called Host because it is both Server and Client (and Peer +// may be confusing). +type Host interface { + // ID returns the (local) peer.ID associated with this Host + ID() peer.ID + + // Peerstore returns the Host's repository of Peer Addresses and Keys. + Peerstore() peerstore.Peerstore + + // Returns the listen addresses of the Host + Addrs() []ma.Multiaddr + + // Networks returns the Network interface of the Host + Network() network.Network + + // Mux returns the Mux multiplexing incoming streams to protocol handlers + Mux() protocol.Switch + + // Connect ensures there is a connection between this host and the peer with + // given peer.ID. Connect will absorb the addresses in pi into its internal + // peerstore. If there is not an active connection, Connect will issue a + // h.Network.Dial, and block until a connection is open, or an error is + // returned. // TODO: Relay + NAT. + Connect(ctx context.Context, pi peer.AddrInfo) error + + // SetStreamHandler sets the protocol handler on the Host's Mux. + // This is equivalent to: + // host.Mux().SetHandler(proto, handler) + // (Threadsafe) + SetStreamHandler(pid protocol.ID, handler network.StreamHandler) + + // SetStreamHandlerMatch sets the protocol handler on the Host's Mux + // using a matching function for protocol selection. + SetStreamHandlerMatch(protocol.ID, func(string) bool, network.StreamHandler) + + // RemoveStreamHandler removes a handler on the mux that was set by + // SetStreamHandler + RemoveStreamHandler(pid protocol.ID) + + // NewStream opens a new stream to given peer p, and writes a p2p/protocol + // header with given ProtocolID. If there is no connection to p, attempts + // to create one. If ProtocolID is "", writes no header. + // (Threadsafe) + NewStream(ctx context.Context, p peer.ID, pids ...protocol.ID) (network.Stream, error) + + // Close shuts down the host, its Network, and services. + Close() error + + // ConnManager returns this hosts connection manager + ConnManager() connmgr.ConnManager +} diff --git a/core/metrics/bandwidth.go b/core/metrics/bandwidth.go new file mode 100644 index 0000000000..e2c8acf30d --- /dev/null +++ b/core/metrics/bandwidth.go @@ -0,0 +1,153 @@ +// Package metrics provides metrics collection and reporting interfaces for libp2p. +package metrics + +import ( + "github.com/libp2p/go-flow-metrics" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/protocol" +) + +// BandwidthCounter tracks incoming and outgoing data transferred by the local peer. +// Metrics are available for total bandwidth across all peers / protocols, as well +// as segmented by remote peer ID and protocol ID. +type BandwidthCounter struct { + totalIn flow.Meter + totalOut flow.Meter + + protocolIn flow.MeterRegistry + protocolOut flow.MeterRegistry + + peerIn flow.MeterRegistry + peerOut flow.MeterRegistry +} + +// NewBandwidthCounter creates a new BandwidthCounter. +func NewBandwidthCounter() *BandwidthCounter { + return new(BandwidthCounter) +} + +// LogSentMessage records the size of an outgoing message +// without associating the bandwidth to a specific peer or protocol. +func (bwc *BandwidthCounter) LogSentMessage(size int64) { + bwc.totalOut.Mark(uint64(size)) +} + +// LogRecvMessage records the size of an incoming message +// without associating the bandwith to a specific peer or protocol. +func (bwc *BandwidthCounter) LogRecvMessage(size int64) { + bwc.totalIn.Mark(uint64(size)) +} + +// LogSentMessageStream records the size of an outgoing message over a single logical stream. +// Bandwidth is associated with the given protocol.ID and peer.ID. +func (bwc *BandwidthCounter) LogSentMessageStream(size int64, proto protocol.ID, p peer.ID) { + bwc.protocolOut.Get(string(proto)).Mark(uint64(size)) + bwc.peerOut.Get(string(p)).Mark(uint64(size)) +} + +// LogRecvMessageStream records the size of an incoming message over a single logical stream. +// Bandwidth is associated with the given protocol.ID and peer.ID. +func (bwc *BandwidthCounter) LogRecvMessageStream(size int64, proto protocol.ID, p peer.ID) { + bwc.protocolIn.Get(string(proto)).Mark(uint64(size)) + bwc.peerIn.Get(string(p)).Mark(uint64(size)) +} + +// GetBandwidthForPeer returns a Stats struct with bandwidth metrics associated with the given peer.ID. +// The metrics returned include all traffic sent / received for the peer, regardless of protocol. +func (bwc *BandwidthCounter) GetBandwidthForPeer(p peer.ID) (out Stats) { + inSnap := bwc.peerIn.Get(string(p)).Snapshot() + outSnap := bwc.peerOut.Get(string(p)).Snapshot() + + return Stats{ + TotalIn: int64(inSnap.Total), + TotalOut: int64(outSnap.Total), + RateIn: inSnap.Rate, + RateOut: outSnap.Rate, + } +} + +// GetBandwidthForProtocol returns a Stats struct with bandwidth metrics associated with the given protocol.ID. +// The metrics returned include all traffic sent / recieved for the protocol, regardless of which peers were +// involved. +func (bwc *BandwidthCounter) GetBandwidthForProtocol(proto protocol.ID) (out Stats) { + inSnap := bwc.protocolIn.Get(string(proto)).Snapshot() + outSnap := bwc.protocolOut.Get(string(proto)).Snapshot() + + return Stats{ + TotalIn: int64(inSnap.Total), + TotalOut: int64(outSnap.Total), + RateIn: inSnap.Rate, + RateOut: outSnap.Rate, + } +} + +// GetBandwidthTotals returns a Stats struct with bandwidth metrics for all data sent / recieved by the +// local peer, regardless of protocol or remote peer IDs. +func (bwc *BandwidthCounter) GetBandwidthTotals() (out Stats) { + inSnap := bwc.totalIn.Snapshot() + outSnap := bwc.totalOut.Snapshot() + + return Stats{ + TotalIn: int64(inSnap.Total), + TotalOut: int64(outSnap.Total), + RateIn: inSnap.Rate, + RateOut: outSnap.Rate, + } +} + +// GetBandwidthByPeer returns a map of all remembered peers and the bandwidth +// metrics with respect to each. This method may be very expensive. +func (bwc *BandwidthCounter) GetBandwidthByPeer() map[peer.ID]Stats { + peers := make(map[peer.ID]Stats) + + bwc.peerIn.ForEach(func(p string, meter *flow.Meter) { + id := peer.ID(p) + snap := meter.Snapshot() + + stat := peers[id] + stat.TotalIn = int64(snap.Total) + stat.RateIn = snap.Rate + peers[id] = stat + }) + + bwc.peerOut.ForEach(func(p string, meter *flow.Meter) { + id := peer.ID(p) + snap := meter.Snapshot() + + stat := peers[id] + stat.TotalOut = int64(snap.Total) + stat.RateOut = snap.Rate + peers[id] = stat + }) + + return peers +} + +// GetBandwidthByProtocol returns a map of all remembered protocols and +// the bandwidth metrics with respect to each. This method may be moderately +// expensive. +func (bwc *BandwidthCounter) GetBandwidthByProtocol() map[protocol.ID]Stats { + protocols := make(map[protocol.ID]Stats) + + bwc.protocolIn.ForEach(func(p string, meter *flow.Meter) { + id := protocol.ID(p) + snap := meter.Snapshot() + + stat := protocols[id] + stat.TotalIn = int64(snap.Total) + stat.RateIn = snap.Rate + protocols[id] = stat + }) + + bwc.protocolOut.ForEach(func(p string, meter *flow.Meter) { + id := protocol.ID(p) + snap := meter.Snapshot() + + stat := protocols[id] + stat.TotalOut = int64(snap.Total) + stat.RateOut = snap.Rate + protocols[id] = stat + }) + + return protocols +} diff --git a/core/metrics/bandwidth_test.go b/core/metrics/bandwidth_test.go new file mode 100644 index 0000000000..de7b0fdcfd --- /dev/null +++ b/core/metrics/bandwidth_test.go @@ -0,0 +1,158 @@ +package metrics + +import ( + "fmt" + "math" + "sync" + "testing" + "time" + + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/protocol" +) + +func BenchmarkBandwidthCounter(b *testing.B) { + b.StopTimer() + b.ResetTimer() + + for i := 0; i < b.N; i++ { + bwc := NewBandwidthCounter() + round(bwc, b) + } +} + +func round(bwc *BandwidthCounter, b *testing.B) { + start := make(chan struct{}) + var wg sync.WaitGroup + wg.Add(10000) + for i := 0; i < 1000; i++ { + p := peer.ID(fmt.Sprintf("peer-%d", i)) + for j := 0; j < 10; j++ { + proto := protocol.ID(fmt.Sprintf("bitswap-%d", j)) + go func() { + defer wg.Done() + <-start + + for i := 0; i < 1000; i++ { + bwc.LogSentMessage(100) + bwc.LogSentMessageStream(100, proto, p) + time.Sleep(1 * time.Millisecond) + } + }() + } + } + + b.StartTimer() + close(start) + wg.Wait() + b.StopTimer() +} + +// Allow 7% errors for bw calculations. +const acceptableError = 0.07 + +func TestBandwidthCounter(t *testing.T) { + bwc := NewBandwidthCounter() + start := make(chan struct{}) + var wg sync.WaitGroup + wg.Add(200) + for i := 0; i < 100; i++ { + p := peer.ID(fmt.Sprintf("peer-%d", i)) + for j := 0; j < 2; j++ { + proto := protocol.ID(fmt.Sprintf("proto-%d", j)) + go func() { + defer wg.Done() + <-start + + t := time.NewTicker(100 * time.Millisecond) + defer t.Stop() + + for i := 0; i < 40; i++ { + bwc.LogSentMessage(100) + bwc.LogRecvMessage(50) + bwc.LogSentMessageStream(100, proto, p) + bwc.LogRecvMessageStream(50, proto, p) + <-t.C + } + }() + } + } + + assertProtocols := func(check func(Stats)) { + byProtocol := bwc.GetBandwidthByProtocol() + if len(byProtocol) != 2 { + t.Errorf("expected 2 protocols, got %d", len(byProtocol)) + } + for i := 0; i < 2; i++ { + p := protocol.ID(fmt.Sprintf("proto-%d", i)) + for _, stats := range [...]Stats{bwc.GetBandwidthForProtocol(p), byProtocol[p]} { + check(stats) + } + } + } + + assertPeers := func(check func(Stats)) { + byPeer := bwc.GetBandwidthByPeer() + if len(byPeer) != 100 { + t.Errorf("expected 100 peers, got %d", len(byPeer)) + } + for i := 0; i < 100; i++ { + p := peer.ID(fmt.Sprintf("peer-%d", i)) + for _, stats := range [...]Stats{bwc.GetBandwidthForPeer(p), byPeer[p]} { + check(stats) + } + } + } + + close(start) + time.Sleep(2*time.Second + 100*time.Millisecond) + + assertPeers(func(stats Stats) { + assertApproxEq(t, 2000, stats.RateOut) + assertApproxEq(t, 1000, stats.RateIn) + }) + + assertProtocols(func(stats Stats) { + assertApproxEq(t, 100000, stats.RateOut) + assertApproxEq(t, 50000, stats.RateIn) + }) + + { + stats := bwc.GetBandwidthTotals() + assertApproxEq(t, 200000, stats.RateOut) + assertApproxEq(t, 100000, stats.RateIn) + } + + wg.Wait() + time.Sleep(1 * time.Second) + + assertPeers(func(stats Stats) { + assertEq(t, 8000, stats.TotalOut) + assertEq(t, 4000, stats.TotalIn) + }) + + assertProtocols(func(stats Stats) { + assertEq(t, 400000, stats.TotalOut) + assertEq(t, 200000, stats.TotalIn) + }) + + { + stats := bwc.GetBandwidthTotals() + assertEq(t, 800000, stats.TotalOut) + assertEq(t, 400000, stats.TotalIn) + } +} + +func assertEq(t *testing.T, expected, actual int64) { + if expected != actual { + t.Errorf("expected %d, got %d", expected, actual) + } +} + +func assertApproxEq(t *testing.T, expected, actual float64) { + t.Helper() + margin := expected * acceptableError + if !(math.Abs(expected-actual) <= margin) { + t.Errorf("expected %f (±%f), got %f", expected, margin, actual) + } +} diff --git a/core/metrics/reporter.go b/core/metrics/reporter.go new file mode 100644 index 0000000000..860345d639 --- /dev/null +++ b/core/metrics/reporter.go @@ -0,0 +1,31 @@ +// Package metrics provides metrics collection and reporting interfaces for libp2p. +package metrics + +import ( + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/protocol" +) + +// Stats represents a point-in-time snapshot of bandwidth metrics. +// +// The TotalIn and TotalOut fields record cumulative bytes sent / received. +// The RateIn and RateOut fields record bytes sent / received per second. +type Stats struct { + TotalIn int64 + TotalOut int64 + RateIn float64 + RateOut float64 +} + +// Reporter provides methods for logging and retrieving metrics. +type Reporter interface { + LogSentMessage(int64) + LogRecvMessage(int64) + LogSentMessageStream(int64, protocol.ID, peer.ID) + LogRecvMessageStream(int64, protocol.ID, peer.ID) + GetBandwidthForPeer(peer.ID) Stats + GetBandwidthForProtocol(protocol.ID) Stats + GetBandwidthTotals() Stats + GetBandwidthByPeer() map[peer.ID]Stats + GetBandwidthByProtocol() map[protocol.ID]Stats +} diff --git a/core/mux/mux.go b/core/mux/mux.go new file mode 100644 index 0000000000..39f2e51c1b --- /dev/null +++ b/core/mux/mux.go @@ -0,0 +1,70 @@ +// Package mux provides stream multiplexing interfaces for libp2p. +// +// For a conceptual overview of stream multiplexing in libp2p, see +// https://docs.libp2p.io/concepts/stream-multiplexing/ +package mux + +import ( + "errors" + "io" + "net" + "time" +) + +// ErrReset is returned when reading or writing on a reset stream. +var ErrReset = errors.New("stream reset") + +// Stream is a bidirectional io pipe within a connection. +type MuxedStream interface { + io.Reader + io.Writer + + // Close closes the stream for writing. Reading will still work (that + // is, the remote side can still write). + io.Closer + + // Reset closes both ends of the stream. Use this to tell the remote + // side to hang up and go away. + Reset() error + + SetDeadline(time.Time) error + SetReadDeadline(time.Time) error + SetWriteDeadline(time.Time) error +} + +// NoopHandler do nothing. Resets streams as soon as they are opened. +var NoopHandler = func(s MuxedStream) { s.Reset() } + +// MuxedConn represents a connection to a remote peer that has been +// extended to support stream multiplexing. +// +// A MuxedConn allows a single net.Conn connection to carry many logically +// independent bidirectional streams of binary data. +// +// Together with network.ConnSecurity, MuxedConn is a component of the +// transport.CapableConn interface, which represents a "raw" network +// connection that has been "upgraded" to support the libp2p capabilities +// of secure communication and stream multiplexing. +type MuxedConn interface { + // Close closes the stream muxer and the the underlying net.Conn. + io.Closer + + // IsClosed returns whether a connection is fully closed, so it can + // be garbage collected. + IsClosed() bool + + // OpenStream creates a new stream. + OpenStream() (MuxedStream, error) + + // AcceptStream accepts a stream opened by the other side. + AcceptStream() (MuxedStream, error) +} + +// Multiplexer wraps a net.Conn with a stream multiplexing +// implementation and returns a MuxedConn that supports opening +// multiple streams over the underlying net.Conn +type Multiplexer interface { + + // NewConn constructs a new connection + NewConn(c net.Conn, isServer bool) (MuxedConn, error) +} diff --git a/core/network/conn.go b/core/network/conn.go new file mode 100644 index 0000000000..e7844a0e80 --- /dev/null +++ b/core/network/conn.go @@ -0,0 +1,58 @@ +package network + +import ( + "io" + + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + + ma "github.com/multiformats/go-multiaddr" +) + +// Conn is a connection to a remote peer. It multiplexes streams. +// Usually there is no need to use a Conn directly, but it may +// be useful to get information about the peer on the other side: +// stream.Conn().RemotePeer() +type Conn interface { + io.Closer + + ConnSecurity + ConnMultiaddrs + + // NewStream constructs a new Stream over this conn. + NewStream() (Stream, error) + + // GetStreams returns all open streams over this conn. + GetStreams() []Stream + + // Stat stores metadata pertaining to this conn. + Stat() Stat +} + +// ConnSecurity is the interface that one can mix into a connection interface to +// give it the security methods. +type ConnSecurity interface { + // LocalPeer returns our peer ID + LocalPeer() peer.ID + + // LocalPrivateKey returns our private key + LocalPrivateKey() ic.PrivKey + + // RemotePeer returns the peer ID of the remote peer. + RemotePeer() peer.ID + + // RemotePublicKey returns the public key of the remote peer. + RemotePublicKey() ic.PubKey +} + +// ConnMultiaddrs is an interface mixin for connection types that provide multiaddr +// addresses for the endpoints. +type ConnMultiaddrs interface { + // LocalMultiaddr returns the local Multiaddr associated + // with this connection + LocalMultiaddr() ma.Multiaddr + + // RemoteMultiaddr returns the remote Multiaddr associated + // with this connection + RemoteMultiaddr() ma.Multiaddr +} diff --git a/core/network/context.go b/core/network/context.go new file mode 100644 index 0000000000..9025f83a22 --- /dev/null +++ b/core/network/context.go @@ -0,0 +1,48 @@ +package network + +import ( + "context" + "time" +) + +// DialPeerTimeout is the default timeout for a single call to `DialPeer`. When +// there are multiple concurrent calls to `DialPeer`, this timeout will apply to +// each independently. +var DialPeerTimeout = 60 * time.Second + +type noDialCtxKey struct{} +type dialPeerTimeoutCtxKey struct{} + +var noDial = noDialCtxKey{} + +// WithNoDial constructs a new context with an option that instructs the network +// to not attempt a new dial when opening a stream. +func WithNoDial(ctx context.Context, reason string) context.Context { + return context.WithValue(ctx, noDial, reason) +} + +// GetNoDial returns true if the no dial option is set in the context. +func GetNoDial(ctx context.Context) (nodial bool, reason string) { + v := ctx.Value(noDial) + if v != nil { + return true, v.(string) + } + + return false, "" +} + +// GetDialPeerTimeout returns the current DialPeer timeout (or the default). +func GetDialPeerTimeout(ctx context.Context) time.Duration { + if to, ok := ctx.Value(dialPeerTimeoutCtxKey{}).(time.Duration); ok { + return to + } + return DialPeerTimeout +} + +// WithDialPeerTimeout returns a new context with the DialPeer timeout applied. +// +// This timeout overrides the default DialPeerTimeout and applies per-dial +// independently. +func WithDialPeerTimeout(ctx context.Context, timeout time.Duration) context.Context { + return context.WithValue(ctx, dialPeerTimeoutCtxKey{}, timeout) +} diff --git a/core/network/context_test.go b/core/network/context_test.go new file mode 100644 index 0000000000..09125516ad --- /dev/null +++ b/core/network/context_test.go @@ -0,0 +1,40 @@ +package network + +import ( + "context" + "testing" + "time" +) + +func TestDefaultTimeout(t *testing.T) { + ctx := context.Background() + dur := GetDialPeerTimeout(ctx) + if dur != DialPeerTimeout { + t.Fatal("expected default peer timeout") + } +} + +func TestNonDefaultTimeout(t *testing.T) { + customTimeout := time.Duration(1) + ctx := context.WithValue( + context.Background(), + dialPeerTimeoutCtxKey{}, + customTimeout, + ) + dur := GetDialPeerTimeout(ctx) + if dur != customTimeout { + t.Fatal("peer timeout doesn't match set timeout") + } +} + +func TestSettingTimeout(t *testing.T) { + customTimeout := time.Duration(1) + ctx := WithDialPeerTimeout( + context.Background(), + customTimeout, + ) + dur := GetDialPeerTimeout(ctx) + if dur != customTimeout { + t.Fatal("peer timeout doesn't match set timeout") + } +} diff --git a/core/network/errors.go b/core/network/errors.go new file mode 100644 index 0000000000..f0cf7291e8 --- /dev/null +++ b/core/network/errors.go @@ -0,0 +1,10 @@ +package network + +import "errors" + +// ErrNoRemoteAddrs is returned when there are no addresses associated with a peer during a dial. +var ErrNoRemoteAddrs = errors.New("no remote addresses") + +// ErrNoConn is returned when attempting to open a stream to a peer with the NoDial +// option and no usable connection is available. +var ErrNoConn = errors.New("no usable connection to peer") diff --git a/core/network/network.go b/core/network/network.go new file mode 100644 index 0000000000..467109c48c --- /dev/null +++ b/core/network/network.go @@ -0,0 +1,138 @@ +// Package network provides core networking abstractions for libp2p. +// +// The network package provides the high-level Network interface for interacting +// with other libp2p peers, which is the primary public API for initiating and +// accepting connections to remote peers. +package network + +import ( + "context" + "io" + + "github.com/jbenet/goprocess" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + + ma "github.com/multiformats/go-multiaddr" +) + +// MessageSizeMax is a soft (recommended) maximum for network messages. +// One can write more, as the interface is a stream. But it is useful +// to bunch it up into multiple read/writes when the whole message is +// a single, large serialized object. +const MessageSizeMax = 1 << 22 // 4 MB + +// Direction represents which peer in a stream initiated a connection. +type Direction int + +const ( + // DirUnknown is the default direction. + DirUnknown Direction = iota + // DirInbound is for when the remote peer initiated a connection. + DirInbound + // DirOutbound is for when the local peer initiated a connection. + DirOutbound +) + +// Connectedness signals the capacity for a connection with a given node. +// It is used to signal to services and other peers whether a node is reachable. +type Connectedness int + +const ( + // NotConnected means no connection to peer, and no extra information (default) + NotConnected Connectedness = iota + + // Connected means has an open, live connection to peer + Connected + + // CanConnect means recently connected to peer, terminated gracefully + CanConnect + + // CannotConnect means recently attempted connecting but failed to connect. + // (should signal "made effort, failed") + CannotConnect +) + +// Stat stores metadata pertaining to a given Stream/Conn. +type Stat struct { + Direction Direction + Extra map[interface{}]interface{} +} + +// StreamHandler is the type of function used to listen for +// streams opened by the remote side. +type StreamHandler func(Stream) + +// ConnHandler is the type of function used to listen for +// connections opened by the remote side. +type ConnHandler func(Conn) + +// Network is the interface used to connect to the outside world. +// It dials and listens for connections. it uses a Swarm to pool +// connections (see swarm pkg, and peerstream.Swarm). Connections +// are encrypted with a TLS-like protocol. +type Network interface { + Dialer + io.Closer + + // SetStreamHandler sets the handler for new streams opened by the + // remote side. This operation is threadsafe. + SetStreamHandler(StreamHandler) + + // SetConnHandler sets the handler for new connections opened by the + // remote side. This operation is threadsafe. + SetConnHandler(ConnHandler) + + // NewStream returns a new stream to given peer p. + // If there is no connection to p, attempts to create one. + NewStream(context.Context, peer.ID) (Stream, error) + + // Listen tells the network to start listening on given multiaddrs. + Listen(...ma.Multiaddr) error + + // ListenAddresses returns a list of addresses at which this network listens. + ListenAddresses() []ma.Multiaddr + + // InterfaceListenAddresses returns a list of addresses at which this network + // listens. It expands "any interface" addresses (/ip4/0.0.0.0, /ip6/::) to + // use the known local interfaces. + InterfaceListenAddresses() ([]ma.Multiaddr, error) + + // Process returns the network's Process + Process() goprocess.Process +} + +// Dialer represents a service that can dial out to peers +// (this is usually just a Network, but other services may not need the whole +// stack, and thus it becomes easier to mock) +type Dialer interface { + // Peerstore returns the internal peerstore + // This is useful to tell the dialer about a new address for a peer. + // Or use one of the public keys found out over the network. + Peerstore() peerstore.Peerstore + + // LocalPeer returns the local peer associated with this network + LocalPeer() peer.ID + + // DialPeer establishes a connection to a given peer + DialPeer(context.Context, peer.ID) (Conn, error) + + // ClosePeer closes the connection to a given peer + ClosePeer(peer.ID) error + + // Connectedness returns a state signaling connection capabilities + Connectedness(peer.ID) Connectedness + + // Peers returns the peers connected + Peers() []peer.ID + + // Conns returns the connections in this Netowrk + Conns() []Conn + + // ConnsToPeer returns the connections in this Netowrk for given peer. + ConnsToPeer(p peer.ID) []Conn + + // Notify/StopNotify register and unregister a notifiee for signals + Notify(Notifiee) + StopNotify(Notifiee) +} diff --git a/core/network/notifee.go b/core/network/notifee.go new file mode 100644 index 0000000000..10ef72f1e1 --- /dev/null +++ b/core/network/notifee.go @@ -0,0 +1,92 @@ +package network + +import ( + ma "github.com/multiformats/go-multiaddr" +) + +// Notifiee is an interface for an object wishing to receive +// notifications from a Network. +type Notifiee interface { + Listen(Network, ma.Multiaddr) // called when network starts listening on an addr + ListenClose(Network, ma.Multiaddr) // called when network stops listening on an addr + Connected(Network, Conn) // called when a connection opened + Disconnected(Network, Conn) // called when a connection closed + OpenedStream(Network, Stream) // called when a stream opened + ClosedStream(Network, Stream) // called when a stream closed + + // TODO + // PeerConnected(Network, peer.ID) // called when a peer connected + // PeerDisconnected(Network, peer.ID) // called when a peer disconnected +} + +// NotifyBundle implements Notifiee by calling any of the functions set on it, +// and nop'ing if they are unset. This is the easy way to register for +// notifications. +type NotifyBundle struct { + ListenF func(Network, ma.Multiaddr) + ListenCloseF func(Network, ma.Multiaddr) + + ConnectedF func(Network, Conn) + DisconnectedF func(Network, Conn) + + OpenedStreamF func(Network, Stream) + ClosedStreamF func(Network, Stream) +} + +var _ Notifiee = (*NotifyBundle)(nil) + +// Listen calls ListenF if it is not null. +func (nb *NotifyBundle) Listen(n Network, a ma.Multiaddr) { + if nb.ListenF != nil { + nb.ListenF(n, a) + } +} + +// ListenClose calls ListenCloseF if it is not null. +func (nb *NotifyBundle) ListenClose(n Network, a ma.Multiaddr) { + if nb.ListenCloseF != nil { + nb.ListenCloseF(n, a) + } +} + +// Connected calls ConnectedF if it is not null. +func (nb *NotifyBundle) Connected(n Network, c Conn) { + if nb.ConnectedF != nil { + nb.ConnectedF(n, c) + } +} + +// Disconnected calls DisconnectedF if it is not null. +func (nb *NotifyBundle) Disconnected(n Network, c Conn) { + if nb.DisconnectedF != nil { + nb.DisconnectedF(n, c) + } +} + +// OpenedStream calls OpenedStreamF if it is not null. +func (nb *NotifyBundle) OpenedStream(n Network, s Stream) { + if nb.OpenedStreamF != nil { + nb.OpenedStreamF(n, s) + } +} + +// ClosedStream calls ClosedStreamF if it is not null. +func (nb *NotifyBundle) ClosedStream(n Network, s Stream) { + if nb.ClosedStreamF != nil { + nb.ClosedStreamF(n, s) + } +} + +// Global noop notifiee. Do not change. +var GlobalNoopNotifiee = &NoopNotifiee{} + +type NoopNotifiee struct{} + +var _ Notifiee = (*NoopNotifiee)(nil) + +func (nn *NoopNotifiee) Connected(n Network, c Conn) {} +func (nn *NoopNotifiee) Disconnected(n Network, c Conn) {} +func (nn *NoopNotifiee) Listen(n Network, addr ma.Multiaddr) {} +func (nn *NoopNotifiee) ListenClose(n Network, addr ma.Multiaddr) {} +func (nn *NoopNotifiee) OpenedStream(Network, Stream) {} +func (nn *NoopNotifiee) ClosedStream(Network, Stream) {} diff --git a/core/network/notifee_test.go b/core/network/notifee_test.go new file mode 100644 index 0000000000..1948074a39 --- /dev/null +++ b/core/network/notifee_test.go @@ -0,0 +1,123 @@ +package network + +import ( + "testing" + + ma "github.com/multiformats/go-multiaddr" +) + +func TestListen(T *testing.T) { + var notifee NotifyBundle + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") + if err != nil { + T.Fatal("unexpected multiaddr error") + } + notifee.Listen(nil, addr) + + called := false + notifee.ListenF = func(Network, ma.Multiaddr) { + called = true + } + if called { + T.Fatal("called should be false") + } + + notifee.Listen(nil, addr) + if !called { + T.Fatal("Listen should have been called") + } +} + +func TestListenClose(T *testing.T) { + var notifee NotifyBundle + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") + if err != nil { + T.Fatal("unexpected multiaddr error") + } + notifee.ListenClose(nil, addr) + + called := false + notifee.ListenCloseF = func(Network, ma.Multiaddr) { + called = true + } + if called { + T.Fatal("called should be false") + } + + notifee.ListenClose(nil, addr) + if !called { + T.Fatal("ListenClose should have been called") + } +} + +func TestConnected(T *testing.T) { + var notifee NotifyBundle + notifee.Connected(nil, nil) + + called := false + notifee.ConnectedF = func(Network, Conn) { + called = true + } + if called { + T.Fatal("called should be false") + } + + notifee.Connected(nil, nil) + if !called { + T.Fatal("Connected should have been called") + } +} + +func TestDisconnected(T *testing.T) { + var notifee NotifyBundle + notifee.Disconnected(nil, nil) + + called := false + notifee.DisconnectedF = func(Network, Conn) { + called = true + } + if called { + T.Fatal("called should be false") + } + + notifee.Disconnected(nil, nil) + if !called { + T.Fatal("Disconnected should have been called") + } +} + +func TestOpenedStream(T *testing.T) { + var notifee NotifyBundle + notifee.OpenedStream(nil, nil) + + called := false + notifee.OpenedStreamF = func(Network, Stream) { + called = true + } + if called { + T.Fatal("called should be false") + } + + notifee.OpenedStream(nil, nil) + if !called { + T.Fatal("OpenedStream should have been called") + } +} + +func TestClosedStream(T *testing.T) { + var notifee NotifyBundle + notifee.ClosedStream(nil, nil) + + called := false + notifee.ClosedStreamF = func(Network, Stream) { + called = true + } + if called { + T.Fatal("called should be false") + } + + notifee.ClosedStream(nil, nil) + if !called { + T.Fatal("ClosedStream should have been called") + } +} diff --git a/core/network/stream.go b/core/network/stream.go new file mode 100644 index 0000000000..d13dc30519 --- /dev/null +++ b/core/network/stream.go @@ -0,0 +1,24 @@ +package network + +import ( + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/protocol" +) + +// Stream represents a bidirectional channel between two agents in +// a libp2p network. "agent" is as granular as desired, potentially +// being a "request -> reply" pair, or whole protocols. +// +// Streams are backed by a multiplexer underneath the hood. +type Stream interface { + mux.MuxedStream + + Protocol() protocol.ID + SetProtocol(id protocol.ID) + + // Stat returns metadata pertaining to this stream. + Stat() Stat + + // Conn returns the connection this stream is part of. + Conn() Conn +} diff --git a/core/peer/addrinfo.go b/core/peer/addrinfo.go new file mode 100644 index 0000000000..c7324e4f3e --- /dev/null +++ b/core/peer/addrinfo.go @@ -0,0 +1,80 @@ +package peer + +import ( + "fmt" + "strings" + + ma "github.com/multiformats/go-multiaddr" +) + +// AddrInfo is a small struct used to pass around a peer with +// a set of addresses (and later, keys?). +type AddrInfo struct { + ID ID + Addrs []ma.Multiaddr +} + +var _ fmt.Stringer = AddrInfo{} + +func (pi AddrInfo) String() string { + return fmt.Sprintf("{%v: %v}", pi.ID, pi.Addrs) +} + +var ErrInvalidAddr = fmt.Errorf("invalid p2p multiaddr") + +func AddrInfoFromP2pAddr(m ma.Multiaddr) (*AddrInfo, error) { + if m == nil { + return nil, ErrInvalidAddr + } + + // make sure it's a P2P addr + parts := ma.Split(m) + if len(parts) < 1 { + return nil, ErrInvalidAddr + } + + // TODO(lgierth): we shouldn't assume /p2p is the last part + p2ppart := parts[len(parts)-1] + if p2ppart.Protocols()[0].Code != ma.P_P2P { + return nil, ErrInvalidAddr + } + + // make sure the /p2p value parses as a peer.ID + peerIdParts := strings.Split(p2ppart.String(), "/") + peerIdStr := peerIdParts[len(peerIdParts)-1] + id, err := IDB58Decode(peerIdStr) + if err != nil { + return nil, err + } + + // we might have received just an /p2p part, which means there's no addr. + var addrs []ma.Multiaddr + if len(parts) > 1 { + addrs = append(addrs, ma.Join(parts[:len(parts)-1]...)) + } + + return &AddrInfo{ + ID: id, + Addrs: addrs, + }, nil +} + +func AddrInfoToP2pAddrs(pi *AddrInfo) ([]ma.Multiaddr, error) { + var addrs []ma.Multiaddr + tpl := "/" + ma.ProtocolWithCode(ma.P_P2P).Name + "/" + for _, addr := range pi.Addrs { + p2paddr, err := ma.NewMultiaddr(tpl + IDB58Encode(pi.ID)) + if err != nil { + return nil, err + } + addrs = append(addrs, addr.Encapsulate(p2paddr)) + } + return addrs, nil +} + +func (pi *AddrInfo) Loggable() map[string]interface{} { + return map[string]interface{}{ + "peerID": pi.ID.Pretty(), + "addrs": pi.Addrs, + } +} diff --git a/core/peer/addrinfo_serde.go b/core/peer/addrinfo_serde.go new file mode 100644 index 0000000000..1df24e2b73 --- /dev/null +++ b/core/peer/addrinfo_serde.go @@ -0,0 +1,38 @@ +package peer + +import ( + "encoding/json" + + ma "github.com/multiformats/go-multiaddr" +) + +func (pi AddrInfo) MarshalJSON() ([]byte, error) { + out := make(map[string]interface{}) + out["ID"] = pi.ID.Pretty() + var addrs []string + for _, a := range pi.Addrs { + addrs = append(addrs, a.String()) + } + out["Addrs"] = addrs + return json.Marshal(out) +} + +func (pi *AddrInfo) UnmarshalJSON(b []byte) error { + var data map[string]interface{} + err := json.Unmarshal(b, &data) + if err != nil { + return err + } + pid, err := IDB58Decode(data["ID"].(string)) + if err != nil { + return err + } + pi.ID = pid + addrs, ok := data["Addrs"].([]interface{}) + if ok { + for _, a := range addrs { + pi.Addrs = append(pi.Addrs, ma.StringCast(a.(string))) + } + } + return nil +} diff --git a/core/peer/peer.go b/core/peer/peer.go new file mode 100644 index 0000000000..db307ee1e3 --- /dev/null +++ b/core/peer/peer.go @@ -0,0 +1,186 @@ +// Package peer implements an object used to represent peers in the libp2p network. +package peer + +import ( + "encoding/hex" + "errors" + "fmt" + + ic "github.com/libp2p/go-libp2p-core/crypto" + b58 "github.com/mr-tron/base58/base58" + mh "github.com/multiformats/go-multihash" +) + +var ( + // ErrEmptyPeerID is an error for empty peer ID. + ErrEmptyPeerID = errors.New("empty peer ID") + // ErrNoPublicKey is an error for peer IDs that don't embed public keys + ErrNoPublicKey = errors.New("public key is not embedded in peer ID") +) + +// AdvancedEnableInlining enables automatically inlining keys shorter than +// 42 bytes into the peer ID (using the "identity" multihash function). +// +// WARNING: This flag will likely be set to false in the future and eventually +// be removed in favor of using a hash function specified by the key itself. +// See: https://github.com/libp2p/specs/issues/138 +// +// DO NOT change this flag unless you know what you're doing. +// +// This currently defaults to true for backwards compatibility but will likely +// be set to false by default when an upgrade path is determined. +var AdvancedEnableInlining = true + +const maxInlineKeyLength = 42 + +// ID is a libp2p peer identity. +// +// Peer IDs are derived by hashing a peer's public key and encoding the +// hash output as a multihash. See IDFromPublicKey for details. +type ID string + +// Pretty returns a base58-encoded string representation of the ID. +func (id ID) Pretty() string { + return IDB58Encode(id) +} + +// Loggable returns a pretty peer ID string in loggable JSON format. +func (id ID) Loggable() map[string]interface{} { + return map[string]interface{}{ + "peerID": id.Pretty(), + } +} + +func (id ID) String() string { + return id.Pretty() +} + +// String prints out the peer ID. +// +// TODO(brian): ensure correctness at ID generation and +// enforce this by only exposing functions that generate +// IDs safely. Then any peer.ID type found in the +// codebase is known to be correct. +func (id ID) ShortString() string { + pid := id.Pretty() + if len(pid) <= 10 { + return fmt.Sprintf("", pid) + } + return fmt.Sprintf("", pid[:2], pid[len(pid)-6:]) +} + +// MatchesPrivateKey tests whether this ID was derived from the secret key sk. +func (id ID) MatchesPrivateKey(sk ic.PrivKey) bool { + return id.MatchesPublicKey(sk.GetPublic()) +} + +// MatchesPublicKey tests whether this ID was derived from the public key pk. +func (id ID) MatchesPublicKey(pk ic.PubKey) bool { + oid, err := IDFromPublicKey(pk) + if err != nil { + return false + } + return oid == id +} + +// ExtractPublicKey attempts to extract the public key from an ID +// +// This method returns ErrNoPublicKey if the peer ID looks valid but it can't extract +// the public key. +func (id ID) ExtractPublicKey() (ic.PubKey, error) { + decoded, err := mh.Decode([]byte(id)) + if err != nil { + return nil, err + } + if decoded.Code != mh.ID { + return nil, ErrNoPublicKey + } + pk, err := ic.UnmarshalPublicKey(decoded.Digest) + if err != nil { + return nil, err + } + return pk, nil +} + +// Validate checks if ID is empty or not. +func (id ID) Validate() error { + if id == ID("") { + return ErrEmptyPeerID + } + + return nil +} + +// IDFromString casts a string to the ID type, and validates +// the value to make sure it is a multihash. +func IDFromString(s string) (ID, error) { + if _, err := mh.Cast([]byte(s)); err != nil { + return ID(""), err + } + return ID(s), nil +} + +// IDFromBytes casts a byte slice to the ID type, and validates +// the value to make sure it is a multihash. +func IDFromBytes(b []byte) (ID, error) { + if _, err := mh.Cast(b); err != nil { + return ID(""), err + } + return ID(b), nil +} + +// IDB58Decode accepts a base58-encoded multihash representing a peer ID +// and returns the decoded ID if the input is valid. +func IDB58Decode(s string) (ID, error) { + m, err := mh.FromB58String(s) + if err != nil { + return "", err + } + return ID(m), err +} + +// IDB58Encode returns the base58-encoded multihash representation of the ID. +func IDB58Encode(id ID) string { + return b58.Encode([]byte(id)) +} + +// IDHexDecode accepts a hex-encoded multihash representing a peer ID +// and returns the decoded ID if the input is valid. +func IDHexDecode(s string) (ID, error) { + m, err := mh.FromHexString(s) + if err != nil { + return "", err + } + return ID(m), err +} + +// IDHexEncode returns the hex-encoded multihash representation of the ID. +func IDHexEncode(id ID) string { + return hex.EncodeToString([]byte(id)) +} + +// IDFromPublicKey returns the Peer ID corresponding to the public key pk. +func IDFromPublicKey(pk ic.PubKey) (ID, error) { + b, err := pk.Bytes() + if err != nil { + return "", err + } + var alg uint64 = mh.SHA2_256 + if AdvancedEnableInlining && len(b) <= maxInlineKeyLength { + alg = mh.ID + } + hash, _ := mh.Sum(b, alg, -1) + return ID(hash), nil +} + +// IDFromPrivateKey returns the Peer ID corresponding to the secret key sk. +func IDFromPrivateKey(sk ic.PrivKey) (ID, error) { + return IDFromPublicKey(sk.GetPublic()) +} + +// IDSlice for sorting peers +type IDSlice []ID + +func (es IDSlice) Len() int { return len(es) } +func (es IDSlice) Swap(i, j int) { es[i], es[j] = es[j], es[i] } +func (es IDSlice) Less(i, j int) bool { return string(es[i]) < string(es[j]) } diff --git a/core/peer/peer_serde.go b/core/peer/peer_serde.go new file mode 100644 index 0000000000..7f1b3e6a65 --- /dev/null +++ b/core/peer/peer_serde.go @@ -0,0 +1,75 @@ +// This file contains Protobuf and JSON serialization/deserialization methods for peer IDs. +package peer + +import ( + "encoding" + "encoding/json" +) + +// Interface assertions commented out to avoid introducing hard dependencies to protobuf. +// var _ proto.Marshaler = (*ID)(nil) +// var _ proto.Unmarshaler = (*ID)(nil) +var _ json.Marshaler = (*ID)(nil) +var _ json.Unmarshaler = (*ID)(nil) + +var _ encoding.BinaryMarshaler = (*ID)(nil) +var _ encoding.BinaryUnmarshaler = (*ID)(nil) +var _ encoding.TextMarshaler = (*ID)(nil) +var _ encoding.TextUnmarshaler = (*ID)(nil) + +func (id ID) Marshal() ([]byte, error) { + return []byte(id), nil +} + +// BinaryMarshal returns the byte representation of the peer ID. +func (id ID) MarshalBinary() ([]byte, error) { + return id.Marshal() +} + +func (id ID) MarshalTo(data []byte) (n int, err error) { + return copy(data, []byte(id)), nil +} + +func (id *ID) Unmarshal(data []byte) (err error) { + *id, err = IDFromBytes(data) + return err +} + +// BinaryUnmarshal sets the ID from its binary representation. +func (id *ID) UnmarshalBinary(data []byte) error { + return id.Unmarshal(data) +} + +// Implements Gogo's proto.Sizer, but we omit the compile-time assertion to avoid introducing a hard +// dependency on gogo. +func (id ID) Size() int { + return len([]byte(id)) +} + +func (id ID) MarshalJSON() ([]byte, error) { + return json.Marshal(IDB58Encode(id)) +} + +func (id *ID) UnmarshalJSON(data []byte) (err error) { + var v string + if err = json.Unmarshal(data, &v); err != nil { + return err + } + *id, err = IDB58Decode(v) + return err +} + +// TextMarshal returns the text encoding of the ID. +func (id ID) MarshalText() ([]byte, error) { + return []byte(IDB58Encode(id)), nil +} + +// TextUnmarshal restores the ID from its text encoding. +func (id *ID) UnmarshalText(data []byte) error { + pid, err := IDB58Decode(string(data)) + if err != nil { + return err + } + *id = pid + return nil +} diff --git a/core/peer/peer_serde_test.go b/core/peer/peer_serde_test.go new file mode 100644 index 0000000000..62ecb0f725 --- /dev/null +++ b/core/peer/peer_serde_test.go @@ -0,0 +1,83 @@ +package peer_test + +import ( + "testing" + + "github.com/libp2p/go-libp2p-core/peer" + . "github.com/libp2p/go-libp2p-testing/peer" +) + +func TestPeerSerdePB(t *testing.T) { + id, err := RandPeerID() + if err != nil { + t.Fatal(err) + } + b, err := id.Marshal() + if err != nil { + t.Fatal(err) + } + + var id2 peer.ID + if err = id2.Unmarshal(b); err != nil { + t.Fatal(err) + } + if id != id2 { + t.Error("expected equal ids in circular serde test") + } +} + +func TestPeerSerdeJSON(t *testing.T) { + id, err := RandPeerID() + if err != nil { + t.Fatal(err) + } + b, err := id.MarshalJSON() + if err != nil { + t.Fatal(err) + } + var id2 peer.ID + if err = id2.UnmarshalJSON(b); err != nil { + t.Fatal(err) + } + if id != id2 { + t.Error("expected equal ids in circular serde test") + } +} + +func TestBinaryMarshaler(t *testing.T) { + id, err := RandPeerID() + if err != nil { + t.Fatal(err) + } + b, err := id.MarshalBinary() + if err != nil { + t.Fatal(err) + } + + var id2 peer.ID + if err = id2.UnmarshalBinary(b); err != nil { + t.Fatal(err) + } + if id != id2 { + t.Error("expected equal ids in circular serde test") + } +} + +func TestTextMarshaler(t *testing.T) { + id, err := RandPeerID() + if err != nil { + t.Fatal(err) + } + b, err := id.MarshalText() + if err != nil { + t.Fatal(err) + } + + var id2 peer.ID + if err = id2.UnmarshalText(b); err != nil { + t.Fatal(err) + } + if id != id2 { + t.Error("expected equal ids in circular serde test") + } +} diff --git a/core/peer/peer_test.go b/core/peer/peer_test.go new file mode 100644 index 0000000000..a134532140 --- /dev/null +++ b/core/peer/peer_test.go @@ -0,0 +1,244 @@ +package peer_test + +import ( + "crypto/rand" + "encoding/base64" + "fmt" + "strings" + "testing" + + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-testing/crypto" + + . "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-testing/peer" + + b58 "github.com/mr-tron/base58/base58" + mh "github.com/multiformats/go-multihash" +) + +var gen1 keyset // generated +var gen2 keyset // generated +var man keyset // manual + +func hash(b []byte) []byte { + h, _ := mh.Sum(b, mh.SHA2_256, -1) + return []byte(h) +} + +func init() { + if err := gen1.generate(); err != nil { + panic(err) + } + if err := gen2.generate(); err != nil { + panic(err) + } + + skManBytes = strings.Replace(skManBytes, "\n", "", -1) + if err := man.load(hpkpMan, skManBytes); err != nil { + panic(err) + } +} + +type keyset struct { + sk ic.PrivKey + pk ic.PubKey + hpk string + hpkp string +} + +func (ks *keyset) generate() error { + var err error + ks.sk, ks.pk, err = tcrypto.RandTestKeyPair(ic.RSA, 512) + if err != nil { + return err + } + + bpk, err := ks.pk.Bytes() + if err != nil { + return err + } + + ks.hpk = string(hash(bpk)) + ks.hpkp = b58.Encode([]byte(ks.hpk)) + return nil +} + +func (ks *keyset) load(hpkp, skBytesStr string) error { + skBytes, err := base64.StdEncoding.DecodeString(skBytesStr) + if err != nil { + return err + } + + ks.sk, err = ic.UnmarshalPrivateKey(skBytes) + if err != nil { + return err + } + + ks.pk = ks.sk.GetPublic() + bpk, err := ks.pk.Bytes() + if err != nil { + return err + } + + ks.hpk = string(hash(bpk)) + ks.hpkp = b58.Encode([]byte(ks.hpk)) + if ks.hpkp != hpkp { + return fmt.Errorf("hpkp doesn't match key. %s", hpkp) + } + return nil +} + +func TestIDMatchesPublicKey(t *testing.T) { + + test := func(ks keyset) { + p1, err := IDB58Decode(ks.hpkp) + if err != nil { + t.Fatal(err) + } + + if ks.hpk != string(p1) { + t.Error("p1 and hpk differ") + } + + if !p1.MatchesPublicKey(ks.pk) { + t.Fatal("p1 does not match pk") + } + + p2, err := IDFromPublicKey(ks.pk) + if err != nil { + t.Fatal(err) + } + + if p1 != p2 { + t.Error("p1 and p2 differ", p1.Pretty(), p2.Pretty()) + } + + if p2.Pretty() != ks.hpkp { + t.Error("hpkp and p2.Pretty differ", ks.hpkp, p2.Pretty()) + } + } + + test(gen1) + test(gen2) + test(man) +} + +func TestIDMatchesPrivateKey(t *testing.T) { + + test := func(ks keyset) { + p1, err := IDB58Decode(ks.hpkp) + if err != nil { + t.Fatal(err) + } + + if ks.hpk != string(p1) { + t.Error("p1 and hpk differ") + } + + if !p1.MatchesPrivateKey(ks.sk) { + t.Fatal("p1 does not match sk") + } + + p2, err := IDFromPrivateKey(ks.sk) + if err != nil { + t.Fatal(err) + } + + if p1 != p2 { + t.Error("p1 and p2 differ", p1.Pretty(), p2.Pretty()) + } + } + + test(gen1) + test(gen2) + test(man) +} + +func TestPublicKeyExtraction(t *testing.T) { + t.Skip("disabled until libp2p/go-libp2p-crypto#51 is fixed") + // Happy path + _, originalPub, err := ic.GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + + id, err := IDFromPublicKey(originalPub) + if err != nil { + t.Fatal(err) + } + + extractedPub, err := id.ExtractPublicKey() + if err != nil { + t.Fatal(err) + } + if extractedPub == nil { + t.Fatal("failed to extract public key") + } + if !originalPub.Equals(extractedPub) { + t.Fatal("extracted public key doesn't match") + } + + // Test invalid multihash (invariant of the type of public key) + pk, err := ID("").ExtractPublicKey() + if err == nil { + t.Fatal("expected an error") + } + if pk != nil { + t.Fatal("expected a nil public key") + } + + // Shouldn't work for, e.g. RSA keys (too large) + + _, rsaPub, err := ic.GenerateKeyPair(ic.RSA, 2048) + if err != nil { + t.Fatal(err) + } + rsaId, err := IDFromPublicKey(rsaPub) + if err != nil { + t.Fatal(err) + } + extractedRsaPub, err := rsaId.ExtractPublicKey() + if err != ErrNoPublicKey { + t.Fatal(err) + } + if extractedRsaPub != nil { + t.Fatal("expected to fail to extract public key from rsa ID") + } +} + +func TestValidate(t *testing.T) { + // Empty peer ID invalidates + err := ID("").Validate() + if err == nil { + t.Error("expected error") + } else if err != ErrEmptyPeerID { + t.Error("expected error message: " + ErrEmptyPeerID.Error()) + } + + // Non-empty peer ID validates + p, err := tpeer.RandPeerID() + if err != nil { + t.Fatal(err) + } + + err = p.Validate() + if err != nil { + t.Error("expected nil, but found " + err.Error()) + } +} + +var hpkpMan = `QmRK3JgmVEGiewxWbhpXLJyjWuGuLeSTMTndA1coMHEy5o` +var skManBytes = ` +CAAS4AQwggJcAgEAAoGBAL7w+Wc4VhZhCdM/+Hccg5Nrf4q9NXWwJylbSrXz/unFS24wyk6pEk0zi3W +7li+vSNVO+NtJQw9qGNAMtQKjVTP+3Vt/jfQRnQM3s6awojtjueEWuLYVt62z7mofOhCtj+VwIdZNBo +/EkLZ0ETfcvN5LVtLYa8JkXybnOPsLvK+PAgMBAAECgYBdk09HDM7zzL657uHfzfOVrdslrTCj6p5mo +DzvCxLkkjIzYGnlPuqfNyGjozkpSWgSUc+X+EGLLl3WqEOVdWJtbM61fewEHlRTM5JzScvwrJ39t7o6 +CCAjKA0cBWBd6UWgbN/t53RoWvh9HrA2AW5YrT0ZiAgKe9y7EMUaENVJ8QJBAPhpdmb4ZL4Fkm4OKia +NEcjzn6mGTlZtef7K/0oRC9+2JkQnCuf6HBpaRhJoCJYg7DW8ZY+AV6xClKrgjBOfERMCQQDExhnzu2 +dsQ9k8QChBlpHO0TRbZBiQfC70oU31kM1AeLseZRmrxv9Yxzdl8D693NNWS2JbKOXl0kMHHcuGQLMVA +kBZ7WvkmPV3aPL6jnwp2pXepntdVnaTiSxJ1dkXShZ/VSSDNZMYKY306EtHrIu3NZHtXhdyHKcggDXr +qkBrdgErAkAlpGPojUwemOggr4FD8sLX1ot2hDJyyV7OK2FXfajWEYJyMRL1Gm9Uk1+Un53RAkJneqp +JGAzKpyttXBTIDO51AkEA98KTiROMnnU8Y6Mgcvr68/SMIsvCYMt9/mtwSBGgl80VaTQ5Hpaktl6Xbh +VUt5Wv0tRxlXZiViCGCD1EtrrwTw== +` diff --git a/core/peer/set.go b/core/peer/set.go new file mode 100644 index 0000000000..ea82ea807e --- /dev/null +++ b/core/peer/set.go @@ -0,0 +1,71 @@ +package peer + +import ( + "sync" +) + +// PeerSet is a threadsafe set of peers +type Set struct { + lk sync.RWMutex + ps map[ID]struct{} + + size int +} + +func NewSet() *Set { + ps := new(Set) + ps.ps = make(map[ID]struct{}) + ps.size = -1 + return ps +} + +func NewLimitedSet(size int) *Set { + ps := new(Set) + ps.ps = make(map[ID]struct{}) + ps.size = size + return ps +} + +func (ps *Set) Add(p ID) { + ps.lk.Lock() + ps.ps[p] = struct{}{} + ps.lk.Unlock() +} + +func (ps *Set) Contains(p ID) bool { + ps.lk.RLock() + _, ok := ps.ps[p] + ps.lk.RUnlock() + return ok +} + +func (ps *Set) Size() int { + ps.lk.RLock() + defer ps.lk.RUnlock() + return len(ps.ps) +} + +// TryAdd Attempts to add the given peer into the set. +// This operation can fail for one of two reasons: +// 1) The given peer is already in the set +// 2) The number of peers in the set is equal to size +func (ps *Set) TryAdd(p ID) bool { + var success bool + ps.lk.Lock() + if _, ok := ps.ps[p]; !ok && (len(ps.ps) < ps.size || ps.size == -1) { + success = true + ps.ps[p] = struct{}{} + } + ps.lk.Unlock() + return success +} + +func (ps *Set) Peers() []ID { + ps.lk.Lock() + out := make([]ID, 0, len(ps.ps)) + for p, _ := range ps.ps { + out = append(out, p) + } + ps.lk.Unlock() + return out +} diff --git a/core/peerstore/peerstore.go b/core/peerstore/peerstore.go new file mode 100644 index 0000000000..2e1b644989 --- /dev/null +++ b/core/peerstore/peerstore.go @@ -0,0 +1,162 @@ +// Package peerstore provides types and interfaces for local storage of address information, +// metadata, and public key material about libp2p peers. +package peerstore + +import ( + "context" + "errors" + "io" + "math" + "time" + + "github.com/libp2p/go-libp2p-core/peer" + + ic "github.com/libp2p/go-libp2p-core/crypto" + + ma "github.com/multiformats/go-multiaddr" +) + +var ErrNotFound = errors.New("item not found") + +var ( + // AddressTTL is the expiration time of addresses. + AddressTTL = time.Hour + + // TempAddrTTL is the ttl used for a short lived address + TempAddrTTL = time.Minute * 2 + + // ProviderAddrTTL is the TTL of an address we've received from a provider. + // This is also a temporary address, but lasts longer. After this expires, + // the records we return will require an extra lookup. + ProviderAddrTTL = time.Minute * 10 + + // RecentlyConnectedAddrTTL is used when we recently connected to a peer. + // It means that we are reasonably certain of the peer's address. + RecentlyConnectedAddrTTL = time.Minute * 10 + + // OwnObservedAddrTTL is used for our own external addresses observed by peers. + OwnObservedAddrTTL = time.Minute * 10 +) + +// Permanent TTLs (distinct so we can distinguish between them, constant as they +// are, in fact, permanent) +const ( + // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes). + PermanentAddrTTL = math.MaxInt64 - iota + + // ConnectedAddrTTL is the ttl used for the addresses of a peer to whom + // we're connected directly. This is basically permanent, as we will + // clear them + re-add under a TempAddrTTL after disconnecting. + ConnectedAddrTTL +) + +// Peerstore provides a threadsafe store of Peer related +// information. +type Peerstore interface { + io.Closer + + AddrBook + KeyBook + PeerMetadata + Metrics + ProtoBook + + // PeerInfo returns a peer.PeerInfo struct for given peer.ID. + // This is a small slice of the information Peerstore has on + // that peer, useful to other services. + PeerInfo(peer.ID) peer.AddrInfo + + // Peers returns all of the peer IDs stored across all inner stores. + Peers() peer.IDSlice +} + +// PeerMetadata can handle values of any type. Serializing values is +// up to the implementation. Dynamic type introspection may not be +// supported, in which case explicitly enlisting types in the +// serializer may be required. +// +// Refer to the docs of the underlying implementation for more +// information. +type PeerMetadata interface { + // Get/Put is a simple registry for other peer-related key/value pairs. + // if we find something we use often, it should become its own set of + // methods. this is a last resort. + Get(p peer.ID, key string) (interface{}, error) + Put(p peer.ID, key string, val interface{}) error +} + +// AddrBook holds the multiaddrs of peers. +type AddrBook interface { + + // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) + AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) + + // AddAddrs gives this AddrBook addresses to use, with a given ttl + // (time-to-live), after which the address is no longer valid. + // If the manager has a longer TTL, the operation is a no-op for that address + AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) + + // SetAddr calls mgr.SetAddrs(p, addr, ttl) + SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) + + // SetAddrs sets the ttl on addresses. This clears any TTL there previously. + // This is used when we receive the best estimate of the validity of an address. + SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) + + // UpdateAddrs updates the addresses associated with the given peer that have + // the given oldTTL to have the given newTTL. + UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) + + // Addresses returns all known (and valid) addresses for a given peer + Addrs(p peer.ID) []ma.Multiaddr + + // AddrStream returns a channel that gets all addresses for a given + // peer sent on it. If new addresses are added after the call is made + // they will be sent along through the channel as well. + AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr + + // ClearAddresses removes all previously stored addresses + ClearAddrs(p peer.ID) + + // PeersWithAddrs returns all of the peer IDs stored in the AddrBook + PeersWithAddrs() peer.IDSlice +} + +// KeyBook tracks the keys of Peers. +type KeyBook interface { + // PubKey stores the public key of a peer. + PubKey(peer.ID) ic.PubKey + + // AddPubKey stores the public key of a peer. + AddPubKey(peer.ID, ic.PubKey) error + + // PrivKey returns the private key of a peer, if known. Generally this might only be our own + // private key, see + // https://discuss.libp2p.io/t/what-is-the-purpose-of-having-map-peer-id-privatekey-in-peerstore/74. + PrivKey(peer.ID) ic.PrivKey + + // AddPrivKey stores the private key of a peer. + AddPrivKey(peer.ID, ic.PrivKey) error + + // PeersWithKeys returns all the peer IDs stored in the KeyBook + PeersWithKeys() peer.IDSlice +} + +// Metrics is just an object that tracks metrics +// across a set of peers. +type Metrics interface { + // RecordLatency records a new latency measurement + RecordLatency(peer.ID, time.Duration) + + // LatencyEWMA returns an exponentially-weighted moving avg. + // of all measurements of a peer's latency. + LatencyEWMA(peer.ID) time.Duration +} + +// ProtoBook tracks the protocols supported by peers +type ProtoBook interface { + GetProtocols(peer.ID) ([]string, error) + AddProtocols(peer.ID, ...string) error + SetProtocols(peer.ID, ...string) error + SupportsProtocols(peer.ID, ...string) ([]string, error) +} diff --git a/core/pnet/env.go b/core/pnet/env.go new file mode 100644 index 0000000000..c8db5e3cbd --- /dev/null +++ b/core/pnet/env.go @@ -0,0 +1,19 @@ +package pnet + +import "os" + +// EnvKey defines environment variable name for forcing usage of PNet in libp2p +// When environment variable of this name is set to "1" the ForcePrivateNetwork +// variable will be set to true. +const EnvKey = "LIBP2P_FORCE_PNET" + +// ForcePrivateNetwork is boolean variable that forces usage of PNet in libp2p +// Setting this variable to true or setting LIBP2P_FORCE_PNET environment variable +// to true will make libp2p to require private network protector. +// If no network protector is provided and this variable is set to true libp2p will +// refuse to connect. +var ForcePrivateNetwork = false + +func init() { + ForcePrivateNetwork = os.Getenv(EnvKey) == "1" +} diff --git a/core/pnet/error.go b/core/pnet/error.go new file mode 100644 index 0000000000..184b71d6ac --- /dev/null +++ b/core/pnet/error.go @@ -0,0 +1,34 @@ +package pnet + +// ErrNotInPrivateNetwork is an error that should be returned by libp2p when it +// tries to dial with ForcePrivateNetwork set and no PNet Protector +var ErrNotInPrivateNetwork = NewError("private network was not configured but" + + " is enforced by the environment") + +// Error is error type for ease of detecting PNet errors +type Error interface { + IsPNetError() bool +} + +// NewError creates new Error +func NewError(err string) error { + return pnetErr("privnet: " + err) +} + +// IsPNetError checks if given error is PNet Error +func IsPNetError(err error) bool { + v, ok := err.(Error) + return ok && v.IsPNetError() +} + +type pnetErr string + +var _ Error = (*pnetErr)(nil) + +func (p pnetErr) Error() string { + return string(p) +} + +func (pnetErr) IsPNetError() bool { + return true +} diff --git a/core/pnet/error_test.go b/core/pnet/error_test.go new file mode 100644 index 0000000000..e1fe462b2a --- /dev/null +++ b/core/pnet/error_test.go @@ -0,0 +1,20 @@ +package pnet + +import ( + "errors" + "testing" +) + +func TestIsPnetErr(t *testing.T) { + err := NewError("test") + + if err.Error() != "privnet: test" { + t.Fatalf("expected 'privnet: test' got '%s'", err.Error()) + } + if !IsPNetError(err) { + t.Fatal("expected the pnetErr to be detected by IsPnetError") + } + if IsPNetError(errors.New("not pnet error")) { + t.Fatal("expected generic error not to be pnetError") + } +} diff --git a/core/pnet/protector.go b/core/pnet/protector.go new file mode 100644 index 0000000000..c443ee8fda --- /dev/null +++ b/core/pnet/protector.go @@ -0,0 +1,15 @@ +// Package pnet provides interfaces for private networking in libp2p. +package pnet + +import "net" + +// Protector interface is a way for private network implementation to be transparent in +// libp2p. It is created by implementation and use by libp2p-conn to secure connections +// so they can be only established with selected number of peers. +type Protector interface { + // Wraps passed connection to protect it + Protect(net.Conn) (net.Conn, error) + + // Returns key fingerprint that is safe to expose + Fingerprint() []byte +} diff --git a/core/protocol/id.go b/core/protocol/id.go new file mode 100644 index 0000000000..f7e4a32baf --- /dev/null +++ b/core/protocol/id.go @@ -0,0 +1,9 @@ +package protocol + +// ID is an identifier used to write protocol headers in streams. +type ID string + +// These are reserved protocol.IDs. +const ( + TestingID ID = "/p2p/_testing" +) diff --git a/core/protocol/switch.go b/core/protocol/switch.go new file mode 100644 index 0000000000..f3ac369bec --- /dev/null +++ b/core/protocol/switch.go @@ -0,0 +1,81 @@ +// Package protocol provides core interfaces for protocol routing and negotiation in libp2p. +package protocol + +import ( + "io" +) + +// HandlerFunc is a user-provided function used by the Router to +// handle a protocol/stream. +// +// Will be invoked with the protocol ID string as the first argument, +// which may differ from the ID used for registration if the handler +// was registered using a match function. +type HandlerFunc = func(protocol string, rwc io.ReadWriteCloser) error + +// Router is an interface that allows users to add and remove protocol handlers, +// which will be invoked when incoming stream requests for registered protocols +// are accepted. +// +// Upon receiving an incoming stream request, the Router will check all registered +// protocol handlers to determine which (if any) is capable of handling the stream. +// The handlers are checked in order of registration; if multiple handlers are +// eligible, only the first to be registered will be invoked. +type Router interface { + + // AddHandler registers the given handler to be invoked for + // an exact literal match of the given protocol ID string. + AddHandler(protocol string, handler HandlerFunc) + + // AddHandlerWithFunc registers the given handler to be invoked + // when the provided match function returns true. + // + // The match function will be invoked with an incoming protocol + // ID string, and should return true if the handler supports + // the protocol. Note that the protocol ID argument is not + // used for matching; if you want to match the protocol ID + // string exactly, you must check for it in your match function. + AddHandlerWithFunc(protocol string, match func(string) bool, handler HandlerFunc) + + // RemoveHandler removes the registered handler (if any) for the + // given protocol ID string. + RemoveHandler(protocol string) + + // Protocols returns a list of all registered protocol ID strings. + // Note that the Router may be able to handle protocol IDs not + // included in this list if handlers were added with match functions + // using AddHandlerWithFunc. + Protocols() []string +} + +// Negotiator is a component capable of reaching agreement over what protocols +// to use for inbound streams of communication. +type Negotiator interface { + + // NegotiateLazy will return the registered protocol handler to use + // for a given inbound stream, returning as soon as the protocol has been + // determined. Returns an error if negotiation fails. + // + // NegotiateLazy may return before all protocol negotiation responses have been + // written to the stream. This is in contrast to Negotiate, which will block until + // the Negotiator is finished with the stream. + NegotiateLazy(rwc io.ReadWriteCloser) (io.ReadWriteCloser, string, HandlerFunc, error) + + // Negotiate will return the registered protocol handler to use for a given + // inbound stream, returning after the protocol has been determined and the + // Negotiator has finished using the stream for negotiation. Returns an + // error if negotiation fails. + Negotiate(rwc io.ReadWriteCloser) (string, HandlerFunc, error) + + // Handle calls Negotiate to determine which protocol handler to use for an + // inbound stream, then invokes the protocol handler function, passing it + // the protocol ID and the stream. Returns an error if negotiation fails. + Handle(rwc io.ReadWriteCloser) error +} + +// Switch is the component responsible for "dispatching" incoming stream requests to +// their corresponding stream handlers. It is both a Negotiator and a Router. +type Switch interface { + Router + Negotiator +} diff --git a/core/routing/options.go b/core/routing/options.go new file mode 100644 index 0000000000..4b235cbfc0 --- /dev/null +++ b/core/routing/options.go @@ -0,0 +1,50 @@ +package routing + +// Option is a single routing option. +type Option func(opts *Options) error + +// Options is a set of routing options +type Options struct { + // Allow expired values. + Expired bool + Offline bool + // Other (ValueStore implementation specific) options. + Other map[interface{}]interface{} +} + +// Apply applies the given options to this Options +func (opts *Options) Apply(options ...Option) error { + for _, o := range options { + if err := o(opts); err != nil { + return err + } + } + return nil +} + +// ToOption converts this Options to a single Option. +func (opts *Options) ToOption() Option { + return func(nopts *Options) error { + *nopts = *opts + if opts.Other != nil { + nopts.Other = make(map[interface{}]interface{}, len(opts.Other)) + for k, v := range opts.Other { + nopts.Other[k] = v + } + } + return nil + } +} + +// Expired is an option that tells the routing system to return expired records +// when no newer records are known. +var Expired Option = func(opts *Options) error { + opts.Expired = true + return nil +} + +// Offline is an option that tells the routing system to operate offline (i.e., rely on cached/local data only). +var Offline Option = func(opts *Options) error { + opts.Offline = true + return nil +} diff --git a/core/routing/query.go b/core/routing/query.go new file mode 100644 index 0000000000..3769d1c265 --- /dev/null +++ b/core/routing/query.go @@ -0,0 +1,86 @@ +package routing + +import ( + "context" + "sync" + + "github.com/libp2p/go-libp2p-core/peer" +) + +type QueryEventType int + +// Number of events to buffer. +var QueryEventBufferSize = 16 + +const ( + SendingQuery QueryEventType = iota + PeerResponse + FinalPeer + QueryError + Provider + Value + AddingPeer + DialingPeer +) + +type QueryEvent struct { + ID peer.ID + Type QueryEventType + Responses []*peer.AddrInfo + Extra string +} + +type routingQueryKey struct{} +type eventChannel struct { + mu sync.Mutex + ctx context.Context + ch chan<- *QueryEvent +} + +// waitThenClose is spawned in a goroutine when the channel is registered. This +// safely cleans up the channel when the context has been canceled. +func (e *eventChannel) waitThenClose() { + <-e.ctx.Done() + e.mu.Lock() + close(e.ch) + // 1. Signals that we're done. + // 2. Frees memory (in case we end up hanging on to this for a while). + e.ch = nil + e.mu.Unlock() +} + +// send sends an event on the event channel, aborting if either the passed or +// the internal context expire. +func (e *eventChannel) send(ctx context.Context, ev *QueryEvent) { + e.mu.Lock() + // Closed. + if e.ch == nil { + e.mu.Unlock() + return + } + // in case the passed context is unrelated, wait on both. + select { + case e.ch <- ev: + case <-e.ctx.Done(): + case <-ctx.Done(): + } + e.mu.Unlock() +} + +func RegisterForQueryEvents(ctx context.Context) (context.Context, <-chan *QueryEvent) { + ch := make(chan *QueryEvent, QueryEventBufferSize) + ech := &eventChannel{ch: ch, ctx: ctx} + go ech.waitThenClose() + return context.WithValue(ctx, routingQueryKey{}, ech), ch +} + +func PublishQueryEvent(ctx context.Context, ev *QueryEvent) { + ich := ctx.Value(routingQueryKey{}) + if ich == nil { + return + } + + // We *want* to panic here. + ech := ich.(*eventChannel) + ech.send(ctx, ev) +} diff --git a/core/routing/query_serde.go b/core/routing/query_serde.go new file mode 100644 index 0000000000..c7423ffe4c --- /dev/null +++ b/core/routing/query_serde.go @@ -0,0 +1,40 @@ +package routing + +import ( + "encoding/json" + + "github.com/libp2p/go-libp2p-core/peer" +) + +func (qe *QueryEvent) MarshalJSON() ([]byte, error) { + return json.Marshal(map[string]interface{}{ + "ID": peer.IDB58Encode(qe.ID), + "Type": int(qe.Type), + "Responses": qe.Responses, + "Extra": qe.Extra, + }) +} + +func (qe *QueryEvent) UnmarshalJSON(b []byte) error { + temp := struct { + ID string + Type int + Responses []*peer.AddrInfo + Extra string + }{} + err := json.Unmarshal(b, &temp) + if err != nil { + return err + } + if len(temp.ID) > 0 { + pid, err := peer.IDB58Decode(temp.ID) + if err != nil { + return err + } + qe.ID = pid + } + qe.Type = QueryEventType(temp.Type) + qe.Responses = temp.Responses + qe.Extra = temp.Extra + return nil +} diff --git a/core/routing/query_test.go b/core/routing/query_test.go new file mode 100644 index 0000000000..15b4846dbb --- /dev/null +++ b/core/routing/query_test.go @@ -0,0 +1,44 @@ +package routing + +import ( + "context" + "fmt" + "sync" + "testing" +) + +func TestEventsCancel(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + ctx, events := RegisterForQueryEvents(ctx) + goch := make(chan struct{}) + var wg sync.WaitGroup + wg.Add(2) + go func() { + defer wg.Done() + for i := 0; i < 100; i++ { + PublishQueryEvent(ctx, &QueryEvent{Extra: fmt.Sprint(i)}) + } + close(goch) + for i := 100; i < 1000; i++ { + PublishQueryEvent(ctx, &QueryEvent{Extra: fmt.Sprint(i)}) + } + }() + go func() { + defer wg.Done() + i := 0 + for e := range events { + if i < 100 { + if e.Extra != fmt.Sprint(i) { + t.Errorf("expected %d, got %s", i, e.Extra) + } + } + i++ + } + if i < 100 { + t.Errorf("expected at least 100 events, got %d", i) + } + }() + <-goch + cancel() + wg.Wait() +} diff --git a/core/routing/routing.go b/core/routing/routing.go new file mode 100644 index 0000000000..c7078054a6 --- /dev/null +++ b/core/routing/routing.go @@ -0,0 +1,125 @@ +// Package routing provides interfaces for peer routing and content routing in libp2p. +package routing + +import ( + "context" + "errors" + + ci "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + + cid "github.com/ipfs/go-cid" +) + +// ErrNotFound is returned when the router fails to find the requested record. +var ErrNotFound = errors.New("routing: not found") + +// ErrNotSupported is returned when the router doesn't support the given record +// type/operation. +var ErrNotSupported = errors.New("routing: operation or key not supported") + +// ContentRouting is a value provider layer of indirection. It is used to find +// information about who has what content. +// +// Content is identified by CID (content identifier), which encodes a hash +// of the identified content in a future-proof manner. +type ContentRouting interface { + // Provide adds the given cid to the content routing system. If 'true' is + // passed, it also announces it, otherwise it is just kept in the local + // accounting of which objects are being provided. + Provide(context.Context, cid.Cid, bool) error + + // Search for peers who are able to provide a given key + FindProvidersAsync(context.Context, cid.Cid, int) <-chan peer.AddrInfo +} + +// PeerRouting is a way to find address information about certain peers. +// This can be implemented by a simple lookup table, a tracking server, +// or even a DHT. +type PeerRouting interface { + // Find specific Peer + // FindPeer searches for a peer with given ID, returns a peer.AddrInfo + // with relevant addresses. + FindPeer(context.Context, peer.ID) (peer.AddrInfo, error) +} + +// ValueStore is a basic Put/Get interface. +type ValueStore interface { + + // PutValue adds value corresponding to given Key. + PutValue(context.Context, string, []byte, ...Option) error + + // GetValue searches for the value corresponding to given Key. + GetValue(context.Context, string, ...Option) ([]byte, error) + + // SearchValue searches for better and better values from this value + // store corresponding to the given Key. By default implementations must + // stop the search after a good value is found. A 'good' value is a value + // that would be returned from GetValue. + // + // Useful when you want a result *now* but still want to hear about + // better/newer results. + // + // Implementations of this methods won't return ErrNotFound. When a value + // couldn't be found, the channel will get closed without passing any results + SearchValue(context.Context, string, ...Option) (<-chan []byte, error) +} + +// Routing is the combination of different routing types supported by libp2p. +// It can be satisfied by a single item (such as a DHT) or multiple different +// pieces that are more optimized to each task. +type Routing interface { + ContentRouting + PeerRouting + ValueStore + + // Bootstrap allows callers to hint to the routing system to get into a + // Boostrapped state and remain there. It is not a synchronous call. + Bootstrap(context.Context) error + + // TODO expose io.Closer or plain-old Close error +} + +// PubKeyFetcher is an interfaces that should be implemented by value stores +// that can optimize retrieval of public keys. +// +// TODO(steb): Consider removing, see https://github.com/libp2p/go-libp2p-routing/issues/22. +type PubKeyFetcher interface { + // GetPublicKey returns the public key for the given peer. + GetPublicKey(context.Context, peer.ID) (ci.PubKey, error) +} + +// KeyForPublicKey returns the key used to retrieve public keys +// from a value store. +func KeyForPublicKey(id peer.ID) string { + return "/pk/" + string(id) +} + +// GetPublicKey retrieves the public key associated with the given peer ID from +// the value store. +// +// If the ValueStore is also a PubKeyFetcher, this method will call GetPublicKey +// (which may be better optimized) instead of GetValue. +func GetPublicKey(r ValueStore, ctx context.Context, p peer.ID) (ci.PubKey, error) { + switch k, err := p.ExtractPublicKey(); err { + case peer.ErrNoPublicKey: + // check the datastore + case nil: + return k, nil + default: + return nil, err + } + + if dht, ok := r.(PubKeyFetcher); ok { + // If we have a DHT as our routing system, use optimized fetcher + return dht.GetPublicKey(ctx, p) + } + key := KeyForPublicKey(p) + pkval, err := r.GetValue(ctx, key) + if err != nil { + return nil, err + } + + // get PublicKey from node.Data + return ci.UnmarshalPublicKey(pkval) +} diff --git a/core/sec/insecure/insecure.go b/core/sec/insecure/insecure.go new file mode 100644 index 0000000000..801569cb0b --- /dev/null +++ b/core/sec/insecure/insecure.go @@ -0,0 +1,90 @@ +// Package insecure provides an insecure, unencrypted implementation of the the SecureConn and SecureTransport interfaces. +// +// Recommended only for testing and other non-production usage. +package insecure + +import ( + "context" + "net" + + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" + + ci "github.com/libp2p/go-libp2p-core/crypto" +) + +// ID is the multistream-select protocol ID that should be used when identifying +// this security transport. +const ID = "/plaintext/1.0.0" + +// Transport is a no-op stream security transport. It provides no +// security and simply mocks the security and identity methods to +// return peer IDs known ahead of time. +type Transport struct { + id peer.ID +} + +// New constructs a new insecure transport. +func New(id peer.ID) *Transport { + return &Transport{ + id: id, + } +} + +// LocalPeer returns the transports local peer ID. +func (t *Transport) LocalPeer() peer.ID { + return t.id +} + +// LocalPrivateKey returns nil. This transport is not secure. +func (t *Transport) LocalPrivateKey() ci.PrivKey { + return nil +} + +// SecureInbound *pretends to secure* an outbound connection to the given peer. +func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { + return &Conn{ + Conn: insecure, + local: t.id, + }, nil +} + +// SecureOutbound *pretends to secure* an outbound connection to the given peer. +func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { + return &Conn{ + Conn: insecure, + local: t.id, + remote: p, + }, nil +} + +// Conn is the connection type returned by the insecure transport. +type Conn struct { + net.Conn + local peer.ID + remote peer.ID +} + +// LocalPeer returns the local peer ID. +func (ic *Conn) LocalPeer() peer.ID { + return ic.local +} + +// RemotePeer returns the remote peer ID if we initiated the dial. Otherwise, it +// returns "" (because this connection isn't actually secure). +func (ic *Conn) RemotePeer() peer.ID { + return ic.remote +} + +// RemotePublicKey returns nil. This connection is not secure +func (ic *Conn) RemotePublicKey() ci.PubKey { + return nil +} + +// LocalPrivateKey returns nil. This connection is not secure. +func (ic *Conn) LocalPrivateKey() ci.PrivKey { + return nil +} + +var _ sec.SecureTransport = (*Transport)(nil) +var _ sec.SecureConn = (*Conn)(nil) diff --git a/core/sec/security.go b/core/sec/security.go new file mode 100644 index 0000000000..95c8e6a651 --- /dev/null +++ b/core/sec/security.go @@ -0,0 +1,26 @@ +// Package sec provides secure connection and transport interfaces for libp2p. +package sec + +import ( + "context" + "net" + + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" +) + +// SecureConn is an authenticated, encrypted connection. +type SecureConn interface { + net.Conn + network.ConnSecurity +} + +// A SecureTransport turns inbound and outbound unauthenticated, +// plain-text, native connections into authenticated, encrypted connections. +type SecureTransport interface { + // SecureInbound secures an inbound connection. + SecureInbound(ctx context.Context, insecure net.Conn) (SecureConn, error) + + // SecureOutbound secures an outbound connection. + SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (SecureConn, error) +} diff --git a/core/transport/transport.go b/core/transport/transport.go new file mode 100644 index 0000000000..774b8d824d --- /dev/null +++ b/core/transport/transport.go @@ -0,0 +1,113 @@ +// Package transport provides the Transport interface, which represents +// the devices and network protocols used to send and recieve data. +package transport + +import ( + "context" + "net" + "time" + + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + + ma "github.com/multiformats/go-multiaddr" +) + +// DialTimeout is the maximum duration a Dial is allowed to take. +// This includes the time between dialing the raw network connection, +// protocol selection as well the handshake, if applicable. +var DialTimeout = 60 * time.Second + +// AcceptTimeout is the maximum duration an Accept is allowed to take. +// This includes the time between accepting the raw network connection, +// protocol selection as well as the handshake, if applicable. +var AcceptTimeout = 60 * time.Second + +// A CapableConn represents a connection that has offers the basic +// capabilities required by libp2p: stream multiplexing, encryption and +// peer authentication. +// +// These capabilities may be natively provided by the transport, or they +// may be shimmed via the "connection upgrade" process, which converts a +// "raw" network connection into one that supports such capabilities by +// layering an encryption channel and a stream multiplexer. +// +// CapableConn provides accessors for the local and remote multiaddrs used to +// establish the connection and an accessor for the underlying Transport. +type CapableConn interface { + mux.MuxedConn + network.ConnSecurity + network.ConnMultiaddrs + + // Transport returns the transport to which this connection belongs. + Transport() Transport +} + +// Transport represents any device by which you can connect to and accept +// connections from other peers. +// +// The Transport interface allows you to open connections to other peers +// by dialing them, and also lets you listen for incoming connections. +// +// Connections returned by Dial and passed into Listeners are of type +// CapableConn, which means that they have been upgraded to support +// stream multiplexing and connection security (encryption and authentication). +// +// For a conceptual overview, see https://docs.libp2p.io/concepts/transport/ +type Transport interface { + // Dial dials a remote peer. It should try to reuse local listener + // addresses if possible but it may choose not to. + Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (CapableConn, error) + + // CanDial returns true if this transport knows how to dial the given + // multiaddr. + // + // Returning true does not guarantee that dialing this multiaddr will + // succeed. This function should *only* be used to preemptively filter + // out addresses that we can't dial. + CanDial(addr ma.Multiaddr) bool + + // Listen listens on the passed multiaddr. + Listen(laddr ma.Multiaddr) (Listener, error) + + // Protocol returns the set of protocols handled by this transport. + // + // See the Network interface for an explanation of how this is used. + Protocols() []int + + // Proxy returns true if this is a proxy transport. + // + // See the Network interface for an explanation of how this is used. + // TODO: Make this a part of the go-multiaddr protocol instead? + Proxy() bool +} + +// Listener is an interface closely resembling the net.Listener interface. The +// only real difference is that Accept() returns Conn's of the type in this +// package, and also exposes a Multiaddr method as opposed to a regular Addr +// method +type Listener interface { + Accept() (CapableConn, error) + Close() error + Addr() net.Addr + Multiaddr() ma.Multiaddr +} + +// Network is an inet.Network with methods for managing transports. +type TransportNetwork interface { + network.Network + + // AddTransport adds a transport to this Network. + // + // When dialing, this Network will iterate over the protocols in the + // remote multiaddr and pick the first protocol registered with a proxy + // transport, if any. Otherwise, it'll pick the transport registered to + // handle the last protocol in the multiaddr. + // + // When listening, this Network will iterate over the protocols in the + // local multiaddr and pick the *last* protocol registered with a proxy + // transport, if any. Otherwise, it'll pick the transport registered to + // handle the last protocol in the multiaddr. + AddTransport(t Transport) error +} From 6f25e76e56e24409a30617ae005a3b14f6b674e2 Mon Sep 17 00:00:00 2001 From: vyzo Date: Wed, 22 May 2019 20:55:20 +0300 Subject: [PATCH 1246/3965] gomod: update go-libp2p-circuit@v0.0.8 --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index cc0c537699..1965adcfa0 100644 --- a/go.mod +++ b/go.mod @@ -12,7 +12,7 @@ require ( github.com/libp2p/go-conn-security-multistream v0.0.2 github.com/libp2p/go-libp2p-autonat v0.0.6 github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.7 + github.com/libp2p/go-libp2p-circuit v0.0.8 github.com/libp2p/go-libp2p-crypto v0.0.2 github.com/libp2p/go-libp2p-discovery v0.0.4 github.com/libp2p/go-libp2p-host v0.0.3 diff --git a/go.sum b/go.sum index c79a174ede..24a32d813e 100644 --- a/go.sum +++ b/go.sum @@ -92,8 +92,8 @@ github.com/libp2p/go-libp2p-autonat v0.0.6 h1:OCStANLLpeyQeWFUuqZJ7aS9+Bx0/uoVb1 github.com/libp2p/go-libp2p-autonat v0.0.6/go.mod h1:uZneLdOkZHro35xIhpbtTzLlgYturpu4J5+0cZK3MqE= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.7 h1:2oIJp4Qud9V8Boy+31nmbNBwH5PCJisGH4SwYTPSU50= -github.com/libp2p/go-libp2p-circuit v0.0.7/go.mod h1:DFCgZ2DklFGTUIZIhSvbbWXTErUgjyNrJGfDHOrTKIA= +github.com/libp2p/go-libp2p-circuit v0.0.8 h1:vd9vZDy+LDssTvUuxIqnYUOAK2hfHoSQO2xjWhPVEmc= +github.com/libp2p/go-libp2p-circuit v0.0.8/go.mod h1:DFCgZ2DklFGTUIZIhSvbbWXTErUgjyNrJGfDHOrTKIA= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= github.com/libp2p/go-libp2p-crypto v0.0.2/go.mod h1:eETI5OUfBnvARGOHrJz2eWNyTUxEGZnBxMcbUjfIj4I= From a59b684e6d404d17529475c28efbc75ff8698a28 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 22 May 2019 13:19:06 -0700 Subject: [PATCH 1247/3965] fix: don't buffer reads This doesn't help when running over a security transport and allocates a bunch of memory. --- p2p/muxer/yamux/yamux.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 965c8fb47c..55e45bdde3 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -55,6 +55,9 @@ func init() { config.MaxStreamWindowSize = uint32(16 * 1024 * 1024) // don't spam config.LogOutput = ioutil.Discard + // We always run over a security transport that buffers internally + // (i.e., uses a block cipher). + config.ReadBufSize = 0 DefaultTransport = (*Transport)(config) } From 90574a9e8d55f2f56cdda12308c3a659b75b2577 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 23 May 2019 15:36:41 +0100 Subject: [PATCH 1248/3965] Buffer early tags in temporary entries (#39) --- p2p/net/connmgr/connmgr.go | 81 ++++++++++++++++++++++----------- p2p/net/connmgr/connmgr_test.go | 50 ++++++++++++++++++-- 2 files changed, 102 insertions(+), 29 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index e61ad15ca9..681649662c 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -53,12 +53,12 @@ type segment struct { type segments [256]*segment -func (s *segments) get(p peer.ID) *segment { - return s[byte(p[len(p)-1])] +func (ss *segments) get(p peer.ID) *segment { + return ss[byte(p[len(p)-1])] } -func (s *segments) countPeers() (count int) { - for _, seg := range s { +func (ss *segments) countPeers() (count int) { + for _, seg := range ss { seg.Lock() count += len(seg.peers) seg.Unlock() @@ -66,6 +66,23 @@ func (s *segments) countPeers() (count int) { return count } +func (s *segment) tagInfoFor(p peer.ID) *peerInfo { + pi, ok := s.peers[p] + if ok { + return pi + } + // create a temporary peer to buffer early tags before the Connected notification arrives. + pi = &peerInfo{ + id: p, + firstSeen: time.Now(), // this timestamp will be updated when the first Connected notification arrives. + temp: true, + tags: make(map[string]int), + conns: make(map[inet.Conn]time.Time), + } + s.peers[p] = pi + return pi +} + // NewConnManager creates a new BasicConnMgr with the provided params: // * lo and hi are watermarks governing the number of connections that'll be maintained. // When the peer count exceeds the 'high watermark', as many peers will be pruned (and @@ -134,6 +151,7 @@ type peerInfo struct { id peer.ID tags map[string]int // value for each tag value int // cached sum of all tag values + temp bool // this is a temporary entry holding early tags, and awaiting connections conns map[inet.Conn]time.Time // start time of each connection @@ -218,7 +236,13 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { // Sort peers according to their value. sort.Slice(candidates, func(i, j int) bool { - return candidates[i].value < candidates[j].value + left, right := candidates[i], candidates[j] + // temporary peers are preferred for pruning. + if left.temp != right.temp { + return left.temp + } + // otherwise, compare by value. + return left.value < right.value }) target := nconns - cm.lowWater @@ -227,6 +251,9 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { selected := make([]inet.Conn, 0, target+10) for _, inf := range candidates { + if target <= 0 { + break + } // TODO: should we be using firstSeen or the time associated with the connection itself? if inf.firstSeen.Add(cm.gracePeriod).After(now) { continue @@ -235,15 +262,18 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { // lock this to protect from concurrent modifications from connect/disconnect events s := cm.segments.get(inf.id) s.Lock() - for c := range inf.conns { - selected = append(selected, c) + + if len(inf.conns) == 0 && inf.temp { + // handle temporary entries for early tags -- this entry has gone past the grace period + // and still holds no connections, so prune it. + delete(s.peers, inf.id) + } else { + for c := range inf.conns { + selected = append(selected, c) + } } target -= len(inf.conns) s.Unlock() - - if target <= 0 { - break - } } return selected @@ -284,14 +314,10 @@ func (cm *BasicConnMgr) TagPeer(p peer.ID, tag string, val int) { s.Lock() defer s.Unlock() - pi, ok := s.peers[p] - if !ok { - log.Info("tried to tag conn from untracked peer: ", p) - return - } + pi := s.tagInfoFor(p) // Update the total value of the peer. - pi.value += (val - pi.tags[tag]) + pi.value += val - pi.tags[tag] pi.tags[tag] = val } @@ -318,15 +344,11 @@ func (cm *BasicConnMgr) UpsertTag(p peer.ID, tag string, upsert func(int) int) { s.Lock() defer s.Unlock() - pi, ok := s.peers[p] - if !ok { - log.Info("tried to upsert tag from untracked peer: ", p) - return - } + pi := s.tagInfoFor(p) oldval := pi.tags[tag] newval := upsert(oldval) - pi.value += (newval - oldval) + pi.value += newval - oldval pi.tags[tag] = newval } @@ -383,15 +405,22 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { s.Lock() defer s.Unlock() - pinfo, ok := s.peers[p] + id := c.RemotePeer() + pinfo, ok := s.peers[id] if !ok { pinfo = &peerInfo{ - id: p, + id: id, firstSeen: time.Now(), tags: make(map[string]int), conns: make(map[inet.Conn]time.Time), } - s.peers[p] = pinfo + s.peers[id] = pinfo + } else if pinfo.temp { + // we had created a temporary entry for this peer to buffer early tags before the + // Connected notification arrived: flip the temporary flag, and update the firstSeen + // timestamp to the real one. + pinfo.temp = false + pinfo.firstSeen = time.Now() } _, ok = pinfo.conns[c] diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index 7f4ed10436..d7933ae893 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -190,8 +190,8 @@ func TestTagPeerNonExistant(t *testing.T) { id := tu.RandPeerIDFatal(t) cm.TagPeer(id, "test", 1) - if cm.segments.countPeers() != 0 { - t.Fatal("expected zero peers") + if !cm.segments.get(id).peers[id].temp { + t.Fatal("expected 1 temporary entry") } } @@ -525,9 +525,9 @@ func TestUpsertTag(t *testing.T) { cm := NewConnManager(1, 1, time.Duration(10*time.Minute)) not := cm.Notifee() conn := randConn(t, nil) - not.Connected(nil, conn) rp := conn.RemotePeer() + // this is an early tag, before the Connected notification arrived. cm.UpsertTag(rp, "tag", func(v int) int { return v + 1 }) if len(cm.segments.get(rp).peers[rp].tags) != 1 { t.Fatal("expected a tag") @@ -536,6 +536,9 @@ func TestUpsertTag(t *testing.T) { t.Fatal("expected a tag value of 1") } + // now let's notify the connection. + not.Connected(nil, conn) + cm.UpsertTag(rp, "tag", func(v int) int { return v + 1 }) if len(cm.segments.get(rp).peers[rp].tags) != 1 { t.Fatal("expected a tag") @@ -552,3 +555,44 @@ func TestUpsertTag(t *testing.T) { t.Fatal("expected a tag value of 1") } } + +func TestTemporaryEntriesClearedFirst(t *testing.T) { + cm := NewConnManager(1, 1, 0) + + id := tu.RandPeerIDFatal(t) + cm.TagPeer(id, "test", 20) + + if cm.GetTagInfo(id).Value != 20 { + t.Fatal("expected an early tag with value 20") + } + + not := cm.Notifee() + conn1, conn2 := randConn(t, nil), randConn(t, nil) + not.Connected(nil, conn1) + not.Connected(nil, conn2) + + cm.TrimOpenConns(context.Background()) + if cm.GetTagInfo(id) != nil { + t.Fatal("expected no temporary tags after trimming") + } +} + +func TestTemporaryEntryConvertedOnConnection(t *testing.T) { + cm := NewConnManager(1, 1, 0) + + conn := randConn(t, nil) + cm.TagPeer(conn.RemotePeer(), "test", 20) + + ti := cm.segments.get(conn.RemotePeer()).peers[conn.RemotePeer()] + + if ti.value != 20 || !ti.temp { + t.Fatal("expected a temporary tag with value 20") + } + + not := cm.Notifee() + not.Connected(nil, conn) + + if ti.value != 20 || ti.temp { + t.Fatal("expected a non-temporary tag with value 20") + } +} From d090163ddaedfcba3342f4c7ea6e6db842207906 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 23 May 2019 16:07:58 +0100 Subject: [PATCH 1249/3965] absorb {crypto,peer} test utils. (#8) Avoids circular module dependency. --- core/crypto/key_test.go | 6 +++--- core/peer/peer_serde_test.go | 2 +- core/peer/peer_test.go | 8 ++++---- core/test/crypto.go | 25 +++++++++++++++++++++++++ core/test/peer.go | 30 ++++++++++++++++++++++++++++++ 5 files changed, 63 insertions(+), 8 deletions(-) create mode 100644 core/test/crypto.go create mode 100644 core/test/peer.go diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index a93fbf0857..2a5e89a8c2 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -7,7 +7,7 @@ import ( . "github.com/libp2p/go-libp2p-core/crypto" pb "github.com/libp2p/go-libp2p-core/crypto/pb" - "github.com/libp2p/go-libp2p-testing/crypto" + "github.com/libp2p/go-libp2p-core/test" ) func TestKeys(t *testing.T) { @@ -17,7 +17,7 @@ func TestKeys(t *testing.T) { } func testKeyType(typ int, t *testing.T) { - sk, pk, err := tcrypto.RandTestKeyPair(typ, 512) + sk, pk, err := test.RandTestKeyPair(typ, 512) if err != nil { t.Fatal(err) } @@ -114,7 +114,7 @@ func testKeyEquals(t *testing.T, k Key) { t.Fatal("Key not equal to key with same bytes.") } - sk, pk, err := tcrypto.RandTestKeyPair(RSA, 512) + sk, pk, err := test.RandTestKeyPair(RSA, 512) if err != nil { t.Fatal(err) } diff --git a/core/peer/peer_serde_test.go b/core/peer/peer_serde_test.go index 62ecb0f725..8a52ba9a32 100644 --- a/core/peer/peer_serde_test.go +++ b/core/peer/peer_serde_test.go @@ -4,7 +4,7 @@ import ( "testing" "github.com/libp2p/go-libp2p-core/peer" - . "github.com/libp2p/go-libp2p-testing/peer" + . "github.com/libp2p/go-libp2p-core/test" ) func TestPeerSerdePB(t *testing.T) { diff --git a/core/peer/peer_test.go b/core/peer/peer_test.go index a134532140..a44904e356 100644 --- a/core/peer/peer_test.go +++ b/core/peer/peer_test.go @@ -8,10 +8,10 @@ import ( "testing" ic "github.com/libp2p/go-libp2p-core/crypto" - "github.com/libp2p/go-libp2p-testing/crypto" + "github.com/libp2p/go-libp2p-core/test" . "github.com/libp2p/go-libp2p-core/peer" - "github.com/libp2p/go-libp2p-testing/peer" + "github.com/libp2p/go-libp2p-core/test" b58 "github.com/mr-tron/base58/base58" mh "github.com/multiformats/go-multihash" @@ -49,7 +49,7 @@ type keyset struct { func (ks *keyset) generate() error { var err error - ks.sk, ks.pk, err = tcrypto.RandTestKeyPair(ic.RSA, 512) + ks.sk, ks.pk, err = test.RandTestKeyPair(ic.RSA, 512) if err != nil { return err } @@ -217,7 +217,7 @@ func TestValidate(t *testing.T) { } // Non-empty peer ID validates - p, err := tpeer.RandPeerID() + p, err := test.RandPeerID() if err != nil { t.Fatal(err) } diff --git a/core/test/crypto.go b/core/test/crypto.go new file mode 100644 index 0000000000..931eae9b9f --- /dev/null +++ b/core/test/crypto.go @@ -0,0 +1,25 @@ +package test + +import ( + "math/rand" + "sync/atomic" + "time" + + ci "github.com/libp2p/go-libp2p-core/crypto" +) + +var generatedPairs int64 = 0 + +func RandTestKeyPair(typ, bits int) (ci.PrivKey, ci.PubKey, error) { + seed := time.Now().UnixNano() + + // workaround for low time resolution + seed += atomic.AddInt64(&generatedPairs, 1) << 32 + + return SeededTestKeyPair(typ, bits, time.Now().UnixNano()) +} + +func SeededTestKeyPair(typ, bits int, seed int64) (ci.PrivKey, ci.PubKey, error) { + r := rand.New(rand.NewSource(seed)) + return ci.GenerateKeyPairWithReader(typ, bits, r) +} diff --git a/core/test/peer.go b/core/test/peer.go new file mode 100644 index 0000000000..9d78989509 --- /dev/null +++ b/core/test/peer.go @@ -0,0 +1,30 @@ +package test + +import ( + "io" + "math/rand" + "testing" + "time" + + "github.com/libp2p/go-libp2p-core/peer" + + mh "github.com/multiformats/go-multihash" +) + +func RandPeerID() (peer.ID, error) { + r := rand.New(rand.NewSource(time.Now().UnixNano())) + buf := make([]byte, 16) + if _, err := io.ReadFull(r, buf); err != nil { + return "", err + } + h, _ := mh.Sum(buf, mh.SHA2_256, -1) + return peer.ID(h), nil +} + +func RandPeerIDFatal(t testing.TB) peer.ID { + p, err := RandPeerID() + if err != nil { + t.Fatal(err) + } + return p +} From 717e15fe601f2df93e95797062af5601eea16e2d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 23 May 2019 16:14:50 +0100 Subject: [PATCH 1250/3965] fix duplicated import. --- core/peer/peer_test.go | 2 -- 1 file changed, 2 deletions(-) diff --git a/core/peer/peer_test.go b/core/peer/peer_test.go index a44904e356..41f323d31e 100644 --- a/core/peer/peer_test.go +++ b/core/peer/peer_test.go @@ -8,8 +8,6 @@ import ( "testing" ic "github.com/libp2p/go-libp2p-core/crypto" - "github.com/libp2p/go-libp2p-core/test" - . "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/test" From 075e2a20429a4dc8f440cdfd2f4910e7759c2119 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 23 May 2019 18:11:06 +0100 Subject: [PATCH 1251/3965] Consolidate abstractions and core types into go-libp2p-core (#69) --- p2p/host/peerstore/interface.go | 156 ++++--------------- p2p/host/peerstore/metrics.go | 14 +- p2p/host/peerstore/peerinfo.go | 115 ++------------ p2p/host/peerstore/peerinfo_test.go | 232 ---------------------------- 4 files changed, 41 insertions(+), 476 deletions(-) delete mode 100644 p2p/host/peerstore/peerinfo_test.go diff --git a/p2p/host/peerstore/interface.go b/p2p/host/peerstore/interface.go index e3d988cee7..105dedc71a 100644 --- a/p2p/host/peerstore/interface.go +++ b/p2p/host/peerstore/interface.go @@ -1,148 +1,46 @@ package peerstore -import ( - "context" - "errors" - "io" - "math" - "time" +import core "github.com/libp2p/go-libp2p-core/peerstore" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" -) - -var ErrNotFound = errors.New("item not found") +// Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.ErrNotFound instead. +var ErrNotFound = core.ErrNotFound var ( - // AddressTTL is the expiration time of addresses. - AddressTTL = time.Hour + // Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.AddressTTL instead. + AddressTTL = core.AddressTTL - // TempAddrTTL is the ttl used for a short lived address - TempAddrTTL = time.Minute * 2 + // Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.TempAddrTTL instead. + TempAddrTTL = core.TempAddrTTL - // ProviderAddrTTL is the TTL of an address we've received from a provider. - // This is also a temporary address, but lasts longer. After this expires, - // the records we return will require an extra lookup. - ProviderAddrTTL = time.Minute * 10 + // Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.ProviderAddrTTL instead. + ProviderAddrTTL = core.ProviderAddrTTL - // RecentlyConnectedAddrTTL is used when we recently connected to a peer. - // It means that we are reasonably certain of the peer's address. - RecentlyConnectedAddrTTL = time.Minute * 10 + // Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.RecentlyConnectedAddrTTL instead. + RecentlyConnectedAddrTTL = core.RecentlyConnectedAddrTTL - // OwnObservedAddrTTL is used for our own external addresses observed by peers. - OwnObservedAddrTTL = time.Minute * 10 + // Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.OwnObservedAddrTTL instead. + OwnObservedAddrTTL = core.OwnObservedAddrTTL ) -// Permanent TTLs (distinct so we can distinguish between them, constant as they -// are, in fact, permanent) const ( + // Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.PermanentAddrTTL instead. + PermanentAddrTTL = core.PermanentAddrTTL - // PermanentAddrTTL is the ttl for a "permanent address" (e.g. bootstrap nodes). - PermanentAddrTTL = math.MaxInt64 - iota - - // ConnectedAddrTTL is the ttl used for the addresses of a peer to whom - // we're connected directly. This is basically permanent, as we will - // clear them + re-add under a TempAddrTTL after disconnecting. - ConnectedAddrTTL + // Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.ConnectedAddrTTL instead. + ConnectedAddrTTL = core.ConnectedAddrTTL ) -// Peerstore provides a threadsafe store of Peer related -// information. -type Peerstore interface { - io.Closer - - AddrBook - KeyBook - PeerMetadata - Metrics - ProtoBook - - // PeerInfo returns a peer.PeerInfo struct for given peer.ID. - // This is a small slice of the information Peerstore has on - // that peer, useful to other services. - PeerInfo(peer.ID) PeerInfo - - // Peers returns all of the peer IDs stored across all inner stores. - Peers() peer.IDSlice -} - -// PeerMetadata can handle values of any type. Serializing values is -// up to the implementation. Dynamic type introspection may not be -// supported, in which case explicitly enlisting types in the -// serializer may be required. -// -// Refer to the docs of the underlying implementation for more -// information. -type PeerMetadata interface { - // Get/Put is a simple registry for other peer-related key/value pairs. - // if we find something we use often, it should become its own set of - // methods. this is a last resort. - Get(p peer.ID, key string) (interface{}, error) - Put(p peer.ID, key string, val interface{}) error -} - -// AddrBook holds the multiaddrs of peers. -type AddrBook interface { - - // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) - AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) - - // AddAddrs gives this AddrBook addresses to use, with a given ttl - // (time-to-live), after which the address is no longer valid. - // If the manager has a longer TTL, the operation is a no-op for that address - AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) - - // SetAddr calls mgr.SetAddrs(p, addr, ttl) - SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) - - // SetAddrs sets the ttl on addresses. This clears any TTL there previously. - // This is used when we receive the best estimate of the validity of an address. - SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) - - // UpdateAddrs updates the addresses associated with the given peer that have - // the given oldTTL to have the given newTTL. - UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) - - // Addresses returns all known (and valid) addresses for a given peer - Addrs(p peer.ID) []ma.Multiaddr - - // AddrStream returns a channel that gets all addresses for a given - // peer sent on it. If new addresses are added after the call is made - // they will be sent along through the channel as well. - AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr - - // ClearAddresses removes all previously stored addresses - ClearAddrs(p peer.ID) - - // PeersWithAddrs returns all of the peer IDs stored in the AddrBook - PeersWithAddrs() peer.IDSlice -} - -// KeyBook tracks the keys of Peers. -type KeyBook interface { - // PubKey stores the public key of a peer. - PubKey(peer.ID) ic.PubKey - - // AddPubKey stores the public key of a peer. - AddPubKey(peer.ID, ic.PubKey) error +// Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.Peerstore instead. +type Peerstore = core.Peerstore - // PrivKey returns the private key of a peer, if known. Generally this might only be our own - // private key, see - // https://discuss.libp2p.io/t/what-is-the-purpose-of-having-map-peer-id-privatekey-in-peerstore/74. - PrivKey(peer.ID) ic.PrivKey +// Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.PeerMetadata instead. +type PeerMetadata = core.PeerMetadata - // AddPrivKey stores the private key of a peer. - AddPrivKey(peer.ID, ic.PrivKey) error +// Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.AddrBook instead. +type AddrBook = core.AddrBook - // PeersWithKeys returns all the peer IDs stored in the KeyBook - PeersWithKeys() peer.IDSlice -} +// Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.KeyBook instead. +type KeyBook = core.KeyBook -// ProtoBook tracks the protocols supported by peers -type ProtoBook interface { - GetProtocols(peer.ID) ([]string, error) - AddProtocols(peer.ID, ...string) error - SetProtocols(peer.ID, ...string) error - SupportsProtocols(peer.ID, ...string) ([]string, error) -} +// Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.ProtoBook instead. +type ProtoBook = core.ProtoBook diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index 05b867692b..f6d0ccb56e 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -4,6 +4,7 @@ import ( "sync" "time" + moved "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-peer" ) @@ -12,17 +13,8 @@ import ( // 1 is 100% change, 0 is no change. var LatencyEWMASmoothing = 0.1 -// Metrics is just an object that tracks metrics -// across a set of peers. -type Metrics interface { - - // RecordLatency records a new latency measurement - RecordLatency(peer.ID, time.Duration) - - // LatencyEWMA returns an exponentially-weighted moving avg. - // of all measurements of a peer's latency. - LatencyEWMA(peer.ID) time.Duration -} +// Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.Metrics instead. +type Metrics = moved.Metrics type metrics struct { latmap map[peer.ID]time.Duration diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index 6adaa20391..24c8c31f3a 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -1,115 +1,22 @@ package peerstore import ( - "encoding/json" - "fmt" - "strings" - - peer "github.com/libp2p/go-libp2p-peer" + core "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" ) -// PeerInfo is a small struct used to pass around a peer with -// a set of addresses (and later, keys?). This is not meant to be -// a complete view of the system, but rather to model updates to -// the peerstore. It is used by things like the routing system. -type PeerInfo struct { - ID peer.ID - Addrs []ma.Multiaddr -} - -var _ fmt.Stringer = PeerInfo{} - -func (pi PeerInfo) String() string { - return fmt.Sprintf("{%v: %v}", pi.ID, pi.Addrs) -} - -var ErrInvalidAddr = fmt.Errorf("invalid p2p multiaddr") - -func InfoFromP2pAddr(m ma.Multiaddr) (*PeerInfo, error) { - if m == nil { - return nil, ErrInvalidAddr - } - - // make sure it's an IPFS addr - parts := ma.Split(m) - if len(parts) < 1 { - return nil, ErrInvalidAddr - } +// Deprecated: use github.com/libp2p/go-libp2p-core/peer.Info instead. +type PeerInfo = core.AddrInfo - // TODO(lgierth): we shouldn't assume /ipfs is the last part - ipfspart := parts[len(parts)-1] - if ipfspart.Protocols()[0].Code != ma.P_IPFS { - return nil, ErrInvalidAddr - } - - // make sure the /ipfs value parses as a peer.ID - peerIdParts := strings.Split(ipfspart.String(), "/") - peerIdStr := peerIdParts[len(peerIdParts)-1] - id, err := peer.IDB58Decode(peerIdStr) - if err != nil { - return nil, err - } - - // we might have received just an /ipfs part, which means there's no addr. - var addrs []ma.Multiaddr - if len(parts) > 1 { - addrs = append(addrs, ma.Join(parts[:len(parts)-1]...)) - } - - return &PeerInfo{ - ID: id, - Addrs: addrs, - }, nil -} - -func InfoToP2pAddrs(pi *PeerInfo) ([]ma.Multiaddr, error) { - var addrs []ma.Multiaddr - tpl := "/" + ma.ProtocolWithCode(ma.P_IPFS).Name + "/" - for _, addr := range pi.Addrs { - p2paddr, err := ma.NewMultiaddr(tpl + peer.IDB58Encode(pi.ID)) - if err != nil { - return nil, err - } - addrs = append(addrs, addr.Encapsulate(p2paddr)) - } - return addrs, nil -} - -func (pi *PeerInfo) Loggable() map[string]interface{} { - return map[string]interface{}{ - "peerID": pi.ID.Pretty(), - "addrs": pi.Addrs, - } -} +// Deprecated: use github.com/libp2p/go-libp2p-core/peer.ErrInvalidAddr instead. +var ErrInvalidAddr = core.ErrInvalidAddr -func (pi PeerInfo) MarshalJSON() ([]byte, error) { - out := make(map[string]interface{}) - out["ID"] = pi.ID.Pretty() - var addrs []string - for _, a := range pi.Addrs { - addrs = append(addrs, a.String()) - } - out["Addrs"] = addrs - return json.Marshal(out) +// Deprecated: use github.com/libp2p/go-libp2p-core/peer.AddrInfoFromP2pAddr instead. +func InfoFromP2pAddr(m ma.Multiaddr) (*core.AddrInfo, error) { + return core.AddrInfoFromP2pAddr(m) } -func (pi *PeerInfo) UnmarshalJSON(b []byte) error { - var data map[string]interface{} - err := json.Unmarshal(b, &data) - if err != nil { - return err - } - pid, err := peer.IDB58Decode(data["ID"].(string)) - if err != nil { - return err - } - pi.ID = pid - addrs, ok := data["Addrs"].([]interface{}) - if ok { - for _, a := range addrs { - pi.Addrs = append(pi.Addrs, ma.StringCast(a.(string))) - } - } - return nil +// Deprecated: use github.com/libp2p/go-libp2p-core/peer.AddrInfoToP2pAddrs instead. +func InfoToP2pAddrs(pi *core.AddrInfo) ([]ma.Multiaddr, error) { + return core.AddrInfoToP2pAddrs(pi) } diff --git a/p2p/host/peerstore/peerinfo_test.go b/p2p/host/peerstore/peerinfo_test.go deleted file mode 100644 index 336c719b03..0000000000 --- a/p2p/host/peerstore/peerinfo_test.go +++ /dev/null @@ -1,232 +0,0 @@ -package peerstore - -import ( - "encoding/json" - "testing" - - "github.com/libp2p/go-libp2p-peer" - ma "github.com/multiformats/go-multiaddr" -) - -func mustAddr(t *testing.T, s string) ma.Multiaddr { - addr, err := ma.NewMultiaddr(s) - if err != nil { - t.Fatal(err) - } - - return addr -} - -func TestPeerInfoMarshal(t *testing.T) { - a := mustAddr(t, "/ip4/1.2.3.4/tcp/4536") - b := mustAddr(t, "/ip4/1.2.3.8/udp/7777") - id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") - if err != nil { - t.Fatal(err) - } - - pi := &PeerInfo{ - ID: id, - Addrs: []ma.Multiaddr{a, b}, - } - - data, err := pi.MarshalJSON() - if err != nil { - t.Fatal(err) - } - - pi2 := new(PeerInfo) - if err := pi2.UnmarshalJSON(data); err != nil { - t.Fatal(err) - } - - if pi2.ID != pi.ID { - t.Fatal("ids didnt match after marshal") - } - - if !pi.Addrs[0].Equal(pi2.Addrs[0]) { - t.Fatal("wrong addrs") - } - - if !pi.Addrs[1].Equal(pi2.Addrs[1]) { - t.Fatal("wrong addrs") - } - - lgbl := pi2.Loggable() - if lgbl["peerID"] != id.Pretty() { - t.Fatal("loggables gave wrong peerID output") - } -} - -func TestPeerInfoMarshalWithPointer(t *testing.T) { - a := mustAddr(t, "/ip4/1.2.3.4/tcp/4536") - b := mustAddr(t, "/ip4/1.2.3.8/udp/7777") - id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") - if err != nil { - t.Fatal(err) - } - - pi := PeerInfo{ - ID: id, - Addrs: []ma.Multiaddr{a, b}, - } - - data, err := json.Marshal(&pi) - if err != nil { - t.Fatal(err) - } - - pi2 := new(PeerInfo) - if err := json.Unmarshal(data, pi2); err != nil { - t.Fatal(err) - } - - if pi2.ID != pi.ID { - t.Fatal("ids didnt match after marshal") - } - - if !pi.Addrs[0].Equal(pi2.Addrs[0]) { - t.Fatal("wrong addrs") - } - - if !pi.Addrs[1].Equal(pi2.Addrs[1]) { - t.Fatal("wrong addrs") - } - - lgbl := pi2.Loggable() - if lgbl["peerID"] != id.Pretty() { - t.Fatal("loggables gave wrong peerID output") - } -} - -func TestPeerInfoMarshalWithValue(t *testing.T) { - a := mustAddr(t, "/ip4/1.2.3.4/tcp/4536") - b := mustAddr(t, "/ip4/1.2.3.8/udp/7777") - id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") - if err != nil { - t.Fatal(err) - } - - pi := PeerInfo{ - ID: id, - Addrs: []ma.Multiaddr{a, b}, - } - - data, err := json.Marshal(pi) - if err != nil { - t.Fatal(err) - } - - pi2 := new(PeerInfo) - if err := json.Unmarshal(data, pi2); err != nil { - t.Fatal(err) - } - - if pi2.ID != pi.ID { - t.Fatal("ids didnt match after marshal") - } - - if !pi.Addrs[0].Equal(pi2.Addrs[0]) { - t.Fatal("wrong addrs") - } - - if !pi.Addrs[1].Equal(pi2.Addrs[1]) { - t.Fatal("wrong addrs") - } - - lgbl := pi2.Loggable() - if lgbl["peerID"] != id.Pretty() { - t.Fatal("loggables gave wrong peerID output") - } -} - -func TestP2pAddrParsing(t *testing.T) { - id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") - if err != nil { - t.Error(err) - } - addr := ma.StringCast("/ip4/1.2.3.4/tcp/4536") - p2paddr := ma.Join(addr, ma.StringCast("/ipfs/"+peer.IDB58Encode(id))) - - pinfo, err := InfoFromP2pAddr(p2paddr) - if err != nil { - t.Error(err) - } - - if pinfo.ID != id { - t.Fatalf("expected PeerID [%s], got [%s]", id, pinfo.ID) - } - - if len(pinfo.Addrs) != 1 { - t.Fatalf("expected 1 addr, got %d", len(pinfo.Addrs)) - } - - if !addr.Equal(pinfo.Addrs[0]) { - t.Fatalf("expected addr [%s], got [%s]", addr, pinfo.Addrs[0]) - } - - addr = ma.StringCast("/ipfs/" + peer.IDB58Encode(id)) - pinfo, err = InfoFromP2pAddr(addr) - if err != nil { - t.Error(err) - } - - if pinfo.ID != id { - t.Fatalf("expected PeerID [%s], got [%s]", id, pinfo.ID) - } - - if len(pinfo.Addrs) > 0 { - t.Fatalf("expected 0 addrs, got %d", len(pinfo.Addrs)) - } - - addr = ma.StringCast("/ip4/1.2.3.4/tcp/4536") - pinfo, err = InfoFromP2pAddr(addr) - if err == nil { - t.Fatalf("expected error, got none") - } - - addr = ma.StringCast("/ip4/1.2.3.4/tcp/4536/http") - pinfo, err = InfoFromP2pAddr(addr) - if err == nil { - t.Fatalf("expected error, got none") - } -} - -func TestP2pAddrConstruction(t *testing.T) { - id, err := peer.IDB58Decode("QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ") - if err != nil { - t.Error(err) - } - addr := ma.StringCast("/ip4/1.2.3.4/tcp/4536") - p2paddr := ma.Join(addr, ma.StringCast("/ipfs/"+peer.IDB58Encode(id))) - - pi := &PeerInfo{ID: id, Addrs: []ma.Multiaddr{addr}} - p2paddrs, err := InfoToP2pAddrs(pi) - if err != nil { - t.Error(err) - } - - if len(p2paddrs) != 1 { - t.Fatalf("expected 1 addr, got %d", len(p2paddrs)) - } - - if !p2paddr.Equal(p2paddrs[0]) { - t.Fatalf("expected [%s], got [%s]", p2paddr, p2paddrs[0]) - } - - pi = &PeerInfo{ID: id} - p2paddrs, err = InfoToP2pAddrs(pi) - if err != nil { - t.Error(err) - } - - if len(p2paddrs) > 0 { - t.Fatalf("expected 0 addrs, got %d", len(p2paddrs)) - } - - pi = &PeerInfo{Addrs: []ma.Multiaddr{ma.StringCast("/ip4/1.2.3.4/tcp/4536")}} - _, err = InfoToP2pAddrs(pi) - if err == nil { - t.Fatalf("expected error, got none") - } -} From 2857d54b8012ccc9459baf618235fc5d18f7b7ac Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 23 May 2019 18:45:35 +0100 Subject: [PATCH 1252/3965] SubtestPingPong: ensure connections are closed. https://github.com/libp2p/go-libp2p-transport/pull/51 --- p2p/transport/testsuite/transport_suite.go | 23 ++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/p2p/transport/testsuite/transport_suite.go b/p2p/transport/testsuite/transport_suite.go index 213cfe936c..eef75bfbca 100644 --- a/p2p/transport/testsuite/transport_suite.go +++ b/p2p/transport/testsuite/transport_suite.go @@ -169,21 +169,33 @@ func SubtestPingPong(t *testing.T, ta, tb transport.Transport, maddr ma.Multiadd } defer list.Close() + var ( + connA, connB transport.CapableConn + ) + defer func() { + if connA != nil { + connA.Close() + } + if connB != nil { + connB.Close() + } + }() + var wg sync.WaitGroup wg.Add(1) go func() { defer wg.Done() - c, err := list.Accept() + var err error + connA, err = list.Accept() if err != nil { t.Error(err) return } - defer c.Close() var sWg sync.WaitGroup for i := 0; i < streams; i++ { - s, err := c.AcceptStream() + s, err := connA.AcceptStream() if err != nil { t.Error(err) return @@ -225,14 +237,13 @@ func SubtestPingPong(t *testing.T, ta, tb transport.Transport, maddr ma.Multiadd t.Error("CanDial should have returned true") } - c, err := tb.Dial(ctx, list.Multiaddr(), peerA) + connB, err = tb.Dial(ctx, list.Multiaddr(), peerA) if err != nil { t.Fatal(err) } - defer c.Close() for i := 0; i < streams; i++ { - s, err := c.OpenStream() + s, err := connB.OpenStream() if err != nil { t.Error(err) continue From d3fad322ee8e5c83e6d442b849d832de4fef97c9 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 24 May 2019 12:40:38 +0300 Subject: [PATCH 1253/3965] gomod: update go-libp2p-discovery --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 1965adcfa0..4b48ddfed3 100644 --- a/go.mod +++ b/go.mod @@ -14,7 +14,7 @@ require ( github.com/libp2p/go-libp2p-blankhost v0.0.1 github.com/libp2p/go-libp2p-circuit v0.0.8 github.com/libp2p/go-libp2p-crypto v0.0.2 - github.com/libp2p/go-libp2p-discovery v0.0.4 + github.com/libp2p/go-libp2p-discovery v0.0.5 github.com/libp2p/go-libp2p-host v0.0.3 github.com/libp2p/go-libp2p-interface-connmgr v0.0.5 github.com/libp2p/go-libp2p-interface-pnet v0.0.1 diff --git a/go.sum b/go.sum index 24a32d813e..3542bf3c7d 100644 --- a/go.sum +++ b/go.sum @@ -97,8 +97,8 @@ github.com/libp2p/go-libp2p-circuit v0.0.8/go.mod h1:DFCgZ2DklFGTUIZIhSvbbWXTErU github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= github.com/libp2p/go-libp2p-crypto v0.0.2/go.mod h1:eETI5OUfBnvARGOHrJz2eWNyTUxEGZnBxMcbUjfIj4I= -github.com/libp2p/go-libp2p-discovery v0.0.4 h1:/kZwOVmcUHvB94zegSJYnUA9EvT1g8APoQJb5FHyT1c= -github.com/libp2p/go-libp2p-discovery v0.0.4/go.mod h1:ReQGiv7QTtza8FUWzewfuMmRDVOQVp+lxHlJJA8YQCM= +github.com/libp2p/go-libp2p-discovery v0.0.5 h1:VpPd7u2odnrrRcW+gVdjLDcXsc35k0Tjxqgbzlre6Uo= +github.com/libp2p/go-libp2p-discovery v0.0.5/go.mod h1:YtF20GUxjgoKZ4zmXj8j3Nb2TUSBHFlOCetzYdbZL5I= github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= github.com/libp2p/go-libp2p-host v0.0.3 h1:BB/1Z+4X0rjKP5lbQTmjEjLbDVbrcmLOlA6QDsN5/j4= github.com/libp2p/go-libp2p-host v0.0.3/go.mod h1:Y/qPyA6C8j2coYyos1dfRm0I8+nvd4TGrDGt4tA7JR8= From 29d536740bc0da5a93fa735045c711b0a3fd6036 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 24 May 2019 13:51:22 +0300 Subject: [PATCH 1254/3965] gomod: update go-libp2p-circuit --- go.mod | 2 +- go.sum | 16 ++-------------- 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/go.mod b/go.mod index 4b48ddfed3..1efff7a00e 100644 --- a/go.mod +++ b/go.mod @@ -12,7 +12,7 @@ require ( github.com/libp2p/go-conn-security-multistream v0.0.2 github.com/libp2p/go-libp2p-autonat v0.0.6 github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.8 + github.com/libp2p/go-libp2p-circuit v0.0.9 github.com/libp2p/go-libp2p-crypto v0.0.2 github.com/libp2p/go-libp2p-discovery v0.0.5 github.com/libp2p/go-libp2p-host v0.0.3 diff --git a/go.sum b/go.sum index 3542bf3c7d..34f3b90d9f 100644 --- a/go.sum +++ b/go.sum @@ -92,8 +92,8 @@ github.com/libp2p/go-libp2p-autonat v0.0.6 h1:OCStANLLpeyQeWFUuqZJ7aS9+Bx0/uoVb1 github.com/libp2p/go-libp2p-autonat v0.0.6/go.mod h1:uZneLdOkZHro35xIhpbtTzLlgYturpu4J5+0cZK3MqE= github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.8 h1:vd9vZDy+LDssTvUuxIqnYUOAK2hfHoSQO2xjWhPVEmc= -github.com/libp2p/go-libp2p-circuit v0.0.8/go.mod h1:DFCgZ2DklFGTUIZIhSvbbWXTErUgjyNrJGfDHOrTKIA= +github.com/libp2p/go-libp2p-circuit v0.0.9 h1:tjdgP9hv8+Pa/xsprBpEFngq4t8aLvjfibBYoDjO9i4= +github.com/libp2p/go-libp2p-circuit v0.0.9/go.mod h1:uU+IBvEQzCu953/ps7bYzC/D/R0Ho2A9LfKVVCatlqU= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= github.com/libp2p/go-libp2p-crypto v0.0.2/go.mod h1:eETI5OUfBnvARGOHrJz2eWNyTUxEGZnBxMcbUjfIj4I= @@ -133,13 +133,9 @@ github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbf github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= github.com/libp2p/go-libp2p-secio v0.0.3 h1:h3fPeDrej7bvvARnC2oSjAfcLZOaS4REZKgWCRQNpE4= github.com/libp2p/go-libp2p-secio v0.0.3/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= -github.com/libp2p/go-libp2p-swarm v0.0.5 h1:oue1c+NbyFTuwCkWulX+ouyQoaZsdYeYcAzQ/slfxXY= -github.com/libp2p/go-libp2p-swarm v0.0.5/go.mod h1:+nkJir4feiXtWQjb/4CQHMEK8Vw+c5nVVxT8R5bs0yY= github.com/libp2p/go-libp2p-swarm v0.0.6 h1:gE0P/v2h+KEXtAi9YTw2UBOSODJ4m9VuuJ+ktc2LVUo= github.com/libp2p/go-libp2p-swarm v0.0.6/go.mod h1:s5GZvzg9xXe8sbeESuFpjt8CJPTCa8mhEusweJqyFy8= github.com/libp2p/go-libp2p-transport v0.0.1/go.mod h1:UzbUs9X+PHOSw7S3ZmeOxfnwaQY5vGDzZmKPod3N3tk= -github.com/libp2p/go-libp2p-transport v0.0.4 h1:/CPHQMN75/IQwkhBxxIo6p6PtL3rwFZtlzBROT3e8mw= -github.com/libp2p/go-libp2p-transport v0.0.4/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= github.com/libp2p/go-libp2p-transport v0.0.5 h1:pV6+UlRxyDpASSGD+60vMvdifSCby6JkJDfi+yUMHac= github.com/libp2p/go-libp2p-transport v0.0.5/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= github.com/libp2p/go-libp2p-transport-upgrader v0.0.3 h1:ZcXhGEMnv8f+ycqVl/9mnIe6uhUYy0aj/vEYQUoG7xU= @@ -165,8 +161,6 @@ github.com/libp2p/go-stream-muxer v0.0.1 h1:Ce6e2Pyu+b5MC1k3eeFtAax0pW4gc6MosYSL github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= github.com/libp2p/go-stream-muxer-multistream v0.1.1 h1:DhHqb4nu1fQv/vQKeLAaZGmhLsUA4SF77IdYJiWE1d4= github.com/libp2p/go-stream-muxer-multistream v0.1.1/go.mod h1:zmGdfkQ1AzOECIAcccoL8L//laqawOsO03zX8Sa+eGw= -github.com/libp2p/go-tcp-transport v0.0.3 h1:ZDMczeg+6uvvlxnShYFY0wGjQqStFhWAYT7769utxuM= -github.com/libp2p/go-tcp-transport v0.0.3/go.mod h1:f11C2zvCaGDkE8aFPUKmuYZwd3pP6HI24LeLMWhJnkQ= github.com/libp2p/go-tcp-transport v0.0.4 h1:2iRu994wCT/iEz62F+c60FUoSkijNEQ0q2Itc+79XlQ= github.com/libp2p/go-tcp-transport v0.0.4/go.mod h1:+E8HvC8ezEVOxIo3V5vCK9l1y/19K427vCzQ+xHKH/o= github.com/libp2p/go-testutil v0.0.1 h1:Xg+O0G2HIMfHqBOBDcMS1iSZJ3GEcId4qOxCQvsGZHk= @@ -235,18 +229,12 @@ github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f h1:M/lL3 github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f/go.mod h1:cZNvX9cFybI01GriPRMXDtczuvUhgbcYr9iCGaNlRv8= github.com/whyrusleeping/go-smux-multiplex v0.1.0 h1:23qGy3Z16UeiMh/BTiRe6pRm7MlvF5EM2873uFTQbro= github.com/whyrusleeping/go-smux-multiplex v0.1.0/go.mod h1:OXL5hggHNZSsadXDlBJDD4eD3IQYEB3Yu6xpovd6pPw= -github.com/whyrusleeping/go-smux-multistream v0.1.0 h1:+x8twbOR2IJtc1I5PF+UT6DqTJmbog4NiB/k/hyYW7k= -github.com/whyrusleeping/go-smux-multistream v0.1.0/go.mod h1:/usW3LIBirW4h9ko1PnoF7tExBnbxPBszG0n4wylJr8= -github.com/whyrusleeping/go-smux-yamux v0.1.1 h1:g9dfTJbs/StYfi1e3id9+nQO8lYaY4sCIW2zESECcTM= -github.com/whyrusleeping/go-smux-yamux v0.1.1/go.mod h1:Yw+ayOEKERDHXLJ4GiE5AnBmldJW4QRLDzGFC9do8G0= github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxfC1Y6mYncdk+kq8d5aLx0Q+/gyZGE44M= github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= -github.com/whyrusleeping/yamux v1.2.0 h1:PzUrk7/Z0g/N5V4/+DesmKXYcCToALgj+SbATgs0B34= -github.com/whyrusleeping/yamux v1.2.0/go.mod h1:Cgw3gpb4DrDZ1FrP/5pxg/cpiY54Gr5uCXwUylwi2GE= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= From 6813fdd0d11b725df2df6394d313fd4b9d6d5f7a Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 24 May 2019 15:09:25 +0300 Subject: [PATCH 1255/3965] gomod: update go-ws-transport --- go.mod | 2 +- go.sum | 8 ++------ 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/go.mod b/go.mod index 1efff7a00e..90f63ec2d7 100644 --- a/go.mod +++ b/go.mod @@ -38,7 +38,7 @@ require ( github.com/libp2p/go-stream-muxer-multistream v0.1.1 github.com/libp2p/go-tcp-transport v0.0.4 github.com/libp2p/go-testutil v0.0.1 - github.com/libp2p/go-ws-transport v0.0.4 + github.com/libp2p/go-ws-transport v0.0.5 github.com/miekg/dns v1.1.12 // indirect github.com/multiformats/go-multiaddr v0.0.4 github.com/multiformats/go-multiaddr-dns v0.0.2 diff --git a/go.sum b/go.sum index 34f3b90d9f..51efb3f167 100644 --- a/go.sum +++ b/go.sum @@ -138,8 +138,6 @@ github.com/libp2p/go-libp2p-swarm v0.0.6/go.mod h1:s5GZvzg9xXe8sbeESuFpjt8CJPTCa github.com/libp2p/go-libp2p-transport v0.0.1/go.mod h1:UzbUs9X+PHOSw7S3ZmeOxfnwaQY5vGDzZmKPod3N3tk= github.com/libp2p/go-libp2p-transport v0.0.5 h1:pV6+UlRxyDpASSGD+60vMvdifSCby6JkJDfi+yUMHac= github.com/libp2p/go-libp2p-transport v0.0.5/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= -github.com/libp2p/go-libp2p-transport-upgrader v0.0.3 h1:ZcXhGEMnv8f+ycqVl/9mnIe6uhUYy0aj/vEYQUoG7xU= -github.com/libp2p/go-libp2p-transport-upgrader v0.0.3/go.mod h1:Ng1HzfMIopyYscMHNFmJqiMMcpgDlj0t+NyjVWW89ws= github.com/libp2p/go-libp2p-transport-upgrader v0.0.4 h1:uGMOd14BL1oFlfb/cGfOxPjiTKBhzWV4aMjjoCF1Z1o= github.com/libp2p/go-libp2p-transport-upgrader v0.0.4/go.mod h1:RGq+tupk+oj7PzL2kn/m1w6YXxcIAYJYeI90h6BGgUc= github.com/libp2p/go-libp2p-yamux v0.1.2 h1:DgGItlrWi0j9y1OhRMC8qqL4zj2MEPWeKJTHb55R16Q= @@ -165,8 +163,8 @@ github.com/libp2p/go-tcp-transport v0.0.4 h1:2iRu994wCT/iEz62F+c60FUoSkijNEQ0q2I github.com/libp2p/go-tcp-transport v0.0.4/go.mod h1:+E8HvC8ezEVOxIo3V5vCK9l1y/19K427vCzQ+xHKH/o= github.com/libp2p/go-testutil v0.0.1 h1:Xg+O0G2HIMfHqBOBDcMS1iSZJ3GEcId4qOxCQvsGZHk= github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= -github.com/libp2p/go-ws-transport v0.0.4 h1:3wt9ed0gIUrne627XHvPMTwG4/AUpsLDy4TGQi2EyQ0= -github.com/libp2p/go-ws-transport v0.0.4/go.mod h1:X9wfEcm2LAJYMox9x2VHAMHAZZSQMFC9mIa/UF6OuZk= +github.com/libp2p/go-ws-transport v0.0.5 h1:IHeR0X9nvE5hOdOD8X/FDQ6jIapdohToQseItwvpixU= +github.com/libp2p/go-ws-transport v0.0.5/go.mod h1:Qbl4BxPfXXhhd/o0wcrgoaItHqA9tnZjoFZnxykuaXU= github.com/libp2p/go-yamux v1.2.1 h1:VumHkMhJ2iFk1lzAeoDRgekiZSylGc6NnAEihVdBCiw= github.com/libp2p/go-yamux v1.2.1/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= @@ -227,8 +225,6 @@ github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6 github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM= github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f h1:M/lL30eFZTKnomXY6huvM6G0+gVquFNf6mxghaWlFUg= github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f/go.mod h1:cZNvX9cFybI01GriPRMXDtczuvUhgbcYr9iCGaNlRv8= -github.com/whyrusleeping/go-smux-multiplex v0.1.0 h1:23qGy3Z16UeiMh/BTiRe6pRm7MlvF5EM2873uFTQbro= -github.com/whyrusleeping/go-smux-multiplex v0.1.0/go.mod h1:OXL5hggHNZSsadXDlBJDD4eD3IQYEB3Yu6xpovd6pPw= github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxfC1Y6mYncdk+kq8d5aLx0Q+/gyZGE44M= From 57d79cc31d23cc5aed7e32493d5f234300e1c260 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 24 May 2019 15:33:02 -0700 Subject: [PATCH 1256/3965] Consolidate abstractions and core types into go-libp2p-core (#28) --- p2p/net/upgrader/conn.go | 12 +++---- p2p/net/upgrader/listener.go | 11 ++----- p2p/net/upgrader/listener_test.go | 52 +++++++++++++++---------------- p2p/net/upgrader/upgrader.go | 31 +++++++++--------- 4 files changed, 51 insertions(+), 55 deletions(-) diff --git a/p2p/net/upgrader/conn.go b/p2p/net/upgrader/conn.go index 48e4814166..4551a0f915 100644 --- a/p2p/net/upgrader/conn.go +++ b/p2p/net/upgrader/conn.go @@ -3,15 +3,15 @@ package stream import ( "fmt" - inet "github.com/libp2p/go-libp2p-net" - transport "github.com/libp2p/go-libp2p-transport" - smux "github.com/libp2p/go-stream-muxer" + mux "github.com/libp2p/go-libp2p-core/mux" + network "github.com/libp2p/go-libp2p-core/network" + transport "github.com/libp2p/go-libp2p-core/transport" ) type transportConn struct { - smux.Conn - inet.ConnMultiaddrs - inet.ConnSecurity + mux.MuxedConn + network.ConnMultiaddrs + network.ConnSecurity transport transport.Transport } diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index fa26825fd5..dc45325966 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -7,24 +7,19 @@ import ( logging "github.com/ipfs/go-log" tec "github.com/jbenet/go-temp-err-catcher" - transport "github.com/libp2p/go-libp2p-transport" + transport "github.com/libp2p/go-libp2p-core/transport" manet "github.com/multiformats/go-multiaddr-net" ) var log = logging.Logger("stream-upgrader") -type connErr struct { - conn transport.Conn - err error -} - type listener struct { manet.Listener transport transport.Transport upgrader *Upgrader - incoming chan transport.Conn + incoming chan transport.CapableConn err error // Used for backpressure @@ -139,7 +134,7 @@ func (l *listener) handleIncoming() { } // Accept accepts a connection. -func (l *listener) Accept() (transport.Conn, error) { +func (l *listener) Accept() (transport.CapableConn, error) { for c := range l.incoming { // Could have been sitting there for a while. if !c.IsClosed() { diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go index 6d98f0850e..f3d89f4b3f 100644 --- a/p2p/net/upgrader/listener_test.go +++ b/p2p/net/upgrader/listener_test.go @@ -7,14 +7,14 @@ import ( "sync" "time" - insecure "github.com/libp2p/go-conn-security/insecure" - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" + core "github.com/libp2p/go-libp2p-core" + mux "github.com/libp2p/go-libp2p-core/mux" + insecure "github.com/libp2p/go-libp2p-core/sec/insecure" + tpt "github.com/libp2p/go-libp2p-core/transport" + mplex "github.com/libp2p/go-libp2p-mplex" st "github.com/libp2p/go-libp2p-transport-upgrader" - smux "github.com/libp2p/go-stream-muxer" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" - mplex "github.com/libp2p/go-libp2p-mplex" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" @@ -24,7 +24,7 @@ import ( // It makes sure that this happens at the same time for client and server. type negotiatingMuxer struct{} -func (m *negotiatingMuxer) NewConn(c net.Conn, isServer bool) (smux.Conn, error) { +func (m *negotiatingMuxer) NewConn(c net.Conn, isServer bool) (mux.MuxedConn, error) { var err error // run a fake muxer negotiation if isServer { @@ -43,10 +43,10 @@ type blockingMuxer struct { unblock chan struct{} } -var _ smux.Transport = &blockingMuxer{} +var _ mux.Multiplexer = &blockingMuxer{} func newBlockingMuxer() *blockingMuxer { return &blockingMuxer{unblock: make(chan struct{})} } -func (m *blockingMuxer) NewConn(c net.Conn, isServer bool) (smux.Conn, error) { +func (m *blockingMuxer) NewConn(c net.Conn, isServer bool) (mux.MuxedConn, error) { <-m.unblock return (&negotiatingMuxer{}).NewConn(c, isServer) } @@ -55,21 +55,21 @@ func (m *blockingMuxer) Unblock() { close(m.unblock) } // errorMuxer is a muxer that errors while setting up type errorMuxer struct{} -var _ smux.Transport = &errorMuxer{} +var _ mux.Multiplexer = &errorMuxer{} -func (m *errorMuxer) NewConn(c net.Conn, isServer bool) (smux.Conn, error) { +func (m *errorMuxer) NewConn(c net.Conn, isServer bool) (mux.MuxedConn, error) { return nil, errors.New("mux error") } var _ = Describe("Listener", func() { var ( defaultUpgrader = &st.Upgrader{ - Secure: insecure.New(peer.ID(1)), + Secure: insecure.New(core.PeerID(1)), Muxer: &negotiatingMuxer{}, } ) - testConn := func(clientConn, serverConn tpt.Conn) { + testConn := func(clientConn, serverConn tpt.CapableConn) { cstr, err := clientConn.OpenStream() ExpectWithOffset(0, err).ToNot(HaveOccurred()) _, err = cstr.Write([]byte("foobar")) @@ -90,7 +90,7 @@ var _ = Describe("Listener", func() { return upgrader.UpgradeListener(nil, ln) } - dial := func(upgrader *st.Upgrader, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { + dial := func(upgrader *st.Upgrader, raddr ma.Multiaddr, p core.PeerID) (tpt.CapableConn, error) { macon, err := manet.Dial(raddr) if err != nil { return nil, err @@ -105,7 +105,7 @@ var _ = Describe("Listener", func() { It("accepts a single connection", func() { ln := createListener(defaultUpgrader) defer ln.Close() - cconn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) + cconn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(1)) Expect(err).ToNot(HaveOccurred()) sconn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) @@ -117,7 +117,7 @@ var _ = Describe("Listener", func() { defer ln.Close() const num = 10 for i := 0; i < 10; i++ { - cconn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) + cconn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(1)) Expect(err).ToNot(HaveOccurred()) sconn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) @@ -130,7 +130,7 @@ var _ = Describe("Listener", func() { tpt.AcceptTimeout = timeout ln := createListener(defaultUpgrader) defer ln.Close() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) if !Expect(err).ToNot(HaveOccurred()) { return } @@ -150,7 +150,7 @@ var _ = Describe("Listener", func() { It("doesn't accept connections that fail to setup", func() { upgrader := &st.Upgrader{ - Secure: insecure.New(peer.ID(1)), + Secure: insecure.New(core.PeerID(1)), Muxer: &errorMuxer{}, } ln := createListener(upgrader) @@ -163,7 +163,7 @@ var _ = Describe("Listener", func() { } close(done) }() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) if !Expect(err).To(HaveOccurred()) { conn.Close() } @@ -178,11 +178,11 @@ var _ = Describe("Listener", func() { num := 3 * st.AcceptQueueLength bm := newBlockingMuxer() upgrader := &st.Upgrader{ - Secure: insecure.New(peer.ID(1)), + Secure: insecure.New(core.PeerID(1)), Muxer: bm, } ln := createListener(upgrader) - accepted := make(chan tpt.Conn, num) + accepted := make(chan tpt.CapableConn, num) go func() { defer GinkgoRecover() for { @@ -200,7 +200,7 @@ var _ = Describe("Listener", func() { wg.Add(1) go func() { defer GinkgoRecover() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) if Expect(err).ToNot(HaveOccurred()) { stream, err := conn.AcceptStream() // wait for conn to be accepted. if !Expect(err).To(HaveOccurred()) { @@ -223,11 +223,11 @@ var _ = Describe("Listener", func() { defer ln.Close() // setup AcceptQueueLength connections, but don't accept any of them - dialed := make(chan tpt.Conn, 10*st.AcceptQueueLength) // used as a thread-safe counter + dialed := make(chan tpt.CapableConn, 10*st.AcceptQueueLength) // used as a thread-safe counter for i := 0; i < st.AcceptQueueLength; i++ { go func() { defer GinkgoRecover() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) Expect(err).ToNot(HaveOccurred()) dialed <- conn }() @@ -236,7 +236,7 @@ var _ = Describe("Listener", func() { // dial a new connection. This connection should not complete setup, since the queue is full go func() { defer GinkgoRecover() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) Expect(err).ToNot(HaveOccurred()) dialed <- conn }() @@ -279,7 +279,7 @@ var _ = Describe("Listener", func() { It("doesn't accept new connections when it is closed", func() { ln := createListener(defaultUpgrader) Expect(ln.Close()).To(Succeed()) - conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(1)) if !Expect(err).To(HaveOccurred()) { conn.Close() } @@ -287,7 +287,7 @@ var _ = Describe("Listener", func() { It("closes incoming connections that have not yet been accepted", func() { ln := createListener(defaultUpgrader) - conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) if !Expect(err).ToNot(HaveOccurred()) { ln.Close() return diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index eacbb3dee8..d68ae8cd8d 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -6,12 +6,12 @@ import ( "fmt" "net" - ss "github.com/libp2p/go-conn-security" - pnet "github.com/libp2p/go-libp2p-interface-pnet" - peer "github.com/libp2p/go-libp2p-peer" - transport "github.com/libp2p/go-libp2p-transport" + core "github.com/libp2p/go-libp2p-core" + mux "github.com/libp2p/go-libp2p-core/mux" + pnet "github.com/libp2p/go-libp2p-core/pnet" + sec "github.com/libp2p/go-libp2p-core/sec" + transport "github.com/libp2p/go-libp2p-core/transport" filter "github.com/libp2p/go-maddr-filter" - smux "github.com/libp2p/go-stream-muxer" manet "github.com/multiformats/go-multiaddr-net" ) @@ -26,8 +26,8 @@ var AcceptQueueLength = 16 // to a full transport connection (secure and multiplexed). type Upgrader struct { Protector pnet.Protector - Secure ss.Transport - Muxer smux.Transport + Secure sec.SecureTransport + Muxer mux.Multiplexer Filters *filter.Filters } @@ -39,7 +39,7 @@ func (u *Upgrader) UpgradeListener(t transport.Transport, list manet.Listener) t upgrader: u, transport: t, threshold: newThreshold(AcceptQueueLength), - incoming: make(chan transport.Conn), + incoming: make(chan transport.CapableConn), cancel: cancel, ctx: ctx, } @@ -49,7 +49,7 @@ func (u *Upgrader) UpgradeListener(t transport.Transport, list manet.Listener) t // UpgradeOutbound upgrades the given outbound multiaddr-net connection into a // full libp2p-transport connection. -func (u *Upgrader) UpgradeOutbound(ctx context.Context, t transport.Transport, maconn manet.Conn, p peer.ID) (transport.Conn, error) { +func (u *Upgrader) UpgradeOutbound(ctx context.Context, t transport.Transport, maconn manet.Conn, p core.PeerID) (transport.CapableConn, error) { if p == "" { return nil, ErrNilPeer } @@ -58,11 +58,11 @@ func (u *Upgrader) UpgradeOutbound(ctx context.Context, t transport.Transport, m // UpgradeInbound upgrades the given inbound multiaddr-net connection into a // full libp2p-transport connection. -func (u *Upgrader) UpgradeInbound(ctx context.Context, t transport.Transport, maconn manet.Conn) (transport.Conn, error) { +func (u *Upgrader) UpgradeInbound(ctx context.Context, t transport.Transport, maconn manet.Conn) (transport.CapableConn, error) { return u.upgrade(ctx, t, maconn, "") } -func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn manet.Conn, p peer.ID) (transport.Conn, error) { +func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn manet.Conn, p core.PeerID) (transport.CapableConn, error) { if u.Filters != nil && u.Filters.AddrBlocked(maconn.RemoteMultiaddr()) { log.Debugf("blocked connection from %s", maconn.RemoteMultiaddr()) maconn.Close() @@ -78,6 +78,7 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma } conn = pconn } else if pnet.ForcePrivateNetwork { + conn.Close() log.Error("tried to dial with no Private Network Protector but usage" + " of Private Networks is forced by the enviroment") return nil, pnet.ErrNotInPrivateNetwork @@ -93,25 +94,25 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma return nil, fmt.Errorf("failed to negotiate security stream multiplexer: %s", err) } return &transportConn{ - Conn: smconn, + MuxedConn: smconn, ConnMultiaddrs: maconn, ConnSecurity: sconn, transport: t, }, nil } -func (u *Upgrader) setupSecurity(ctx context.Context, conn net.Conn, p peer.ID) (ss.Conn, error) { +func (u *Upgrader) setupSecurity(ctx context.Context, conn net.Conn, p core.PeerID) (sec.SecureConn, error) { if p == "" { return u.Secure.SecureInbound(ctx, conn) } return u.Secure.SecureOutbound(ctx, conn, p) } -func (u *Upgrader) setupMuxer(ctx context.Context, conn net.Conn, p peer.ID) (smux.Conn, error) { +func (u *Upgrader) setupMuxer(ctx context.Context, conn net.Conn, p core.PeerID) (mux.MuxedConn, error) { // TODO: The muxer should take a context. done := make(chan struct{}) - var smconn smux.Conn + var smconn mux.MuxedConn var err error go func() { defer close(done) From 12917c5cfee0f0865229cce233fda8991b364ed6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 24 May 2019 16:30:16 -0700 Subject: [PATCH 1257/3965] Consolidate abstractions and core types into go-libp2p-core (#21) --- p2p/host/blank/blank.go | 47 ++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 24 deletions(-) diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go index 1c896b54b1..215ba34e1c 100644 --- a/p2p/host/blank/blank.go +++ b/p2p/host/blank/blank.go @@ -5,12 +5,11 @@ import ( "io" logging "github.com/ipfs/go-log" - host "github.com/libp2p/go-libp2p-host" - ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - protocol "github.com/libp2p/go-libp2p-protocol" + core "github.com/libp2p/go-libp2p-core" + connmgr "github.com/libp2p/go-libp2p-core/connmgr" + network "github.com/libp2p/go-libp2p-core/network" + pstore "github.com/libp2p/go-libp2p-core/peerstore" + protocol "github.com/libp2p/go-libp2p-core/protocol" ma "github.com/multiformats/go-multiaddr" mstream "github.com/multiformats/go-multistream" ) @@ -19,15 +18,15 @@ var log = logging.Logger("blankhost") // BlankHost is the thinnest implementation of the host.Host interface type BlankHost struct { - n inet.Network + n core.Network mux *mstream.MultistreamMuxer - cmgr ifconnmgr.ConnManager + cmgr connmgr.ConnManager } -func NewBlankHost(n inet.Network) *BlankHost { +func NewBlankHost(n core.Network) *BlankHost { bh := &BlankHost{ n: n, - cmgr: &ifconnmgr.NullConnMgr{}, + cmgr: &connmgr.NullConnMgr{}, mux: mstream.NewMultistreamMuxer(), } @@ -35,7 +34,7 @@ func NewBlankHost(n inet.Network) *BlankHost { return bh } -var _ host.Host = (*BlankHost)(nil) +var _ core.Host = (*BlankHost)(nil) func (bh *BlankHost) Addrs() []ma.Multiaddr { addrs, err := bh.n.InterfaceListenAddresses() @@ -51,7 +50,7 @@ func (bh *BlankHost) Close() error { return bh.n.Close() } -func (bh *BlankHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { +func (bh *BlankHost) Connect(ctx context.Context, pi core.PeerAddrInfo) error { // absorb addresses into peerstore bh.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) @@ -68,11 +67,11 @@ func (bh *BlankHost) Peerstore() pstore.Peerstore { return bh.n.Peerstore() } -func (bh *BlankHost) ID() peer.ID { +func (bh *BlankHost) ID() core.PeerID { return bh.n.LocalPeer() } -func (bh *BlankHost) NewStream(ctx context.Context, p peer.ID, protos ...protocol.ID) (inet.Stream, error) { +func (bh *BlankHost) NewStream(ctx context.Context, p core.PeerID, protos ...core.ProtocolID) (core.Stream, error) { s, err := bh.n.NewStream(ctx, p) if err != nil { return nil, err @@ -96,30 +95,30 @@ func (bh *BlankHost) NewStream(ctx context.Context, p peer.ID, protos ...protoco return s, nil } -func (bh *BlankHost) RemoveStreamHandler(p protocol.ID) { +func (bh *BlankHost) RemoveStreamHandler(p core.ProtocolID) { bh.Mux().RemoveHandler(string(p)) } -func (bh *BlankHost) SetStreamHandler(pid protocol.ID, handler inet.StreamHandler) { +func (bh *BlankHost) SetStreamHandler(pid core.ProtocolID, handler network.StreamHandler) { bh.Mux().AddHandler(string(pid), func(p string, rwc io.ReadWriteCloser) error { - is := rwc.(inet.Stream) + is := rwc.(core.Stream) is.SetProtocol(protocol.ID(p)) handler(is) return nil }) } -func (bh *BlankHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler inet.StreamHandler) { +func (bh *BlankHost) SetStreamHandlerMatch(pid core.ProtocolID, m func(string) bool, handler network.StreamHandler) { bh.Mux().AddHandlerWithFunc(string(pid), m, func(p string, rwc io.ReadWriteCloser) error { - is := rwc.(inet.Stream) + is := rwc.(core.Stream) is.SetProtocol(protocol.ID(p)) handler(is) return nil }) } -// newStreamHandler is the remote-opened stream handler for inet.Network -func (h *BlankHost) newStreamHandler(s inet.Stream) { +// newStreamHandler is the remote-opened stream handler for core.Network +func (h *BlankHost) newStreamHandler(s core.Stream) { protoID, handle, err := h.Mux().Negotiate(s) if err != nil { @@ -134,15 +133,15 @@ func (h *BlankHost) newStreamHandler(s inet.Stream) { } // TODO: i'm not sure this really needs to be here -func (bh *BlankHost) Mux() *mstream.MultistreamMuxer { +func (bh *BlankHost) Mux() protocol.Switch { return bh.mux } // TODO: also not sure this fits... Might be better ways around this (leaky abstractions) -func (bh *BlankHost) Network() inet.Network { +func (bh *BlankHost) Network() core.Network { return bh.n } -func (bh *BlankHost) ConnManager() ifconnmgr.ConnManager { +func (bh *BlankHost) ConnManager() connmgr.ConnManager { return bh.cmgr } From a2d70a7084d2209f957ff33d5315b10a13dde7e1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 24 May 2019 19:01:04 -0700 Subject: [PATCH 1258/3965] fix: don't try to write an error _twice_. The websocket upgrader does this for us. --- p2p/transport/websocket/listener.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/listener.go b/p2p/transport/websocket/listener.go index 437be9893f..517c719aac 100644 --- a/p2p/transport/websocket/listener.go +++ b/p2p/transport/websocket/listener.go @@ -26,7 +26,7 @@ func (l *listener) serve() { func (l *listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { c, err := upgrader.Upgrade(w, r, nil) if err != nil { - http.Error(w, "Failed to upgrade websocket", 400) + // The upgrader writes a response for us. return } From 88e0587d8c6639a5e2cf360a3ab75ccde565202e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sat, 25 May 2019 12:28:06 +0100 Subject: [PATCH 1259/3965] use migrated types; drop import prefixes. (#22) --- p2p/host/blank/blank.go | 51 ++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go index 215ba34e1c..59189782b3 100644 --- a/p2p/host/blank/blank.go +++ b/p2p/host/blank/blank.go @@ -4,12 +4,15 @@ import ( "context" "io" + "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/protocol" + logging "github.com/ipfs/go-log" - core "github.com/libp2p/go-libp2p-core" - connmgr "github.com/libp2p/go-libp2p-core/connmgr" - network "github.com/libp2p/go-libp2p-core/network" - pstore "github.com/libp2p/go-libp2p-core/peerstore" - protocol "github.com/libp2p/go-libp2p-core/protocol" + ma "github.com/multiformats/go-multiaddr" mstream "github.com/multiformats/go-multistream" ) @@ -18,12 +21,12 @@ var log = logging.Logger("blankhost") // BlankHost is the thinnest implementation of the host.Host interface type BlankHost struct { - n core.Network + n network.Network mux *mstream.MultistreamMuxer cmgr connmgr.ConnManager } -func NewBlankHost(n core.Network) *BlankHost { +func NewBlankHost(n network.Network) *BlankHost { bh := &BlankHost{ n: n, cmgr: &connmgr.NullConnMgr{}, @@ -34,7 +37,7 @@ func NewBlankHost(n core.Network) *BlankHost { return bh } -var _ core.Host = (*BlankHost)(nil) +var _ host.Host = (*BlankHost)(nil) func (bh *BlankHost) Addrs() []ma.Multiaddr { addrs, err := bh.n.InterfaceListenAddresses() @@ -50,28 +53,28 @@ func (bh *BlankHost) Close() error { return bh.n.Close() } -func (bh *BlankHost) Connect(ctx context.Context, pi core.PeerAddrInfo) error { +func (bh *BlankHost) Connect(ctx context.Context, ai peer.AddrInfo) error { // absorb addresses into peerstore - bh.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) + bh.Peerstore().AddAddrs(ai.ID, ai.Addrs, peerstore.TempAddrTTL) - cs := bh.n.ConnsToPeer(pi.ID) + cs := bh.n.ConnsToPeer(ai.ID) if len(cs) > 0 { return nil } - _, err := bh.Network().DialPeer(ctx, pi.ID) + _, err := bh.Network().DialPeer(ctx, ai.ID) return err } -func (bh *BlankHost) Peerstore() pstore.Peerstore { +func (bh *BlankHost) Peerstore() peerstore.Peerstore { return bh.n.Peerstore() } -func (bh *BlankHost) ID() core.PeerID { +func (bh *BlankHost) ID() peer.ID { return bh.n.LocalPeer() } -func (bh *BlankHost) NewStream(ctx context.Context, p core.PeerID, protos ...core.ProtocolID) (core.Stream, error) { +func (bh *BlankHost) NewStream(ctx context.Context, p peer.ID, protos ...protocol.ID) (network.Stream, error) { s, err := bh.n.NewStream(ctx, p) if err != nil { return nil, err @@ -95,32 +98,32 @@ func (bh *BlankHost) NewStream(ctx context.Context, p core.PeerID, protos ...cor return s, nil } -func (bh *BlankHost) RemoveStreamHandler(p core.ProtocolID) { +func (bh *BlankHost) RemoveStreamHandler(p protocol.ID) { bh.Mux().RemoveHandler(string(p)) } -func (bh *BlankHost) SetStreamHandler(pid core.ProtocolID, handler network.StreamHandler) { +func (bh *BlankHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) { bh.Mux().AddHandler(string(pid), func(p string, rwc io.ReadWriteCloser) error { - is := rwc.(core.Stream) + is := rwc.(network.Stream) is.SetProtocol(protocol.ID(p)) handler(is) return nil }) } -func (bh *BlankHost) SetStreamHandlerMatch(pid core.ProtocolID, m func(string) bool, handler network.StreamHandler) { +func (bh *BlankHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler network.StreamHandler) { bh.Mux().AddHandlerWithFunc(string(pid), m, func(p string, rwc io.ReadWriteCloser) error { - is := rwc.(core.Stream) + is := rwc.(network.Stream) is.SetProtocol(protocol.ID(p)) handler(is) return nil }) } -// newStreamHandler is the remote-opened stream handler for core.Network -func (h *BlankHost) newStreamHandler(s core.Stream) { +// newStreamHandler is the remote-opened stream handler for network.Network +func (bh *BlankHost) newStreamHandler(s network.Stream) { - protoID, handle, err := h.Mux().Negotiate(s) + protoID, handle, err := bh.Mux().Negotiate(s) if err != nil { log.Warning("protocol mux failed: %s", err) s.Close() @@ -138,7 +141,7 @@ func (bh *BlankHost) Mux() protocol.Switch { } // TODO: also not sure this fits... Might be better ways around this (leaky abstractions) -func (bh *BlankHost) Network() core.Network { +func (bh *BlankHost) Network() network.Network { return bh.n } From 65aca08c7fa09f883d057d86cad2c447b1abc4b6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sat, 25 May 2019 12:57:31 +0100 Subject: [PATCH 1260/3965] migrate to consolidated types (#45) --- p2p/net/connmgr/bench_test.go | 4 ++-- p2p/net/connmgr/connmgr.go | 37 +++++++++++++++++---------------- p2p/net/connmgr/connmgr_test.go | 26 ++++++++++++----------- 3 files changed, 35 insertions(+), 32 deletions(-) diff --git a/p2p/net/connmgr/bench_test.go b/p2p/net/connmgr/bench_test.go index aac11bb994..3749e63fc4 100644 --- a/p2p/net/connmgr/bench_test.go +++ b/p2p/net/connmgr/bench_test.go @@ -5,10 +5,10 @@ import ( "sync" "testing" - inet "github.com/libp2p/go-libp2p-net" + "github.com/libp2p/go-libp2p-core/network" ) -func randomConns(tb testing.TB) (c [5000]inet.Conn) { +func randomConns(tb testing.TB) (c [5000]network.Conn) { for i, _ := range c { c[i] = randConn(tb, nil) } diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 681649662c..0ee8901742 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -7,10 +7,11 @@ import ( "sync/atomic" "time" + "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + logging "github.com/ipfs/go-log" - ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" ma "github.com/multiformats/go-multiaddr" ) @@ -44,7 +45,7 @@ type BasicConnMgr struct { cancel func() } -var _ ifconnmgr.ConnManager = (*BasicConnMgr)(nil) +var _ connmgr.ConnManager = (*BasicConnMgr)(nil) type segment struct { sync.Mutex @@ -77,7 +78,7 @@ func (s *segment) tagInfoFor(p peer.ID) *peerInfo { firstSeen: time.Now(), // this timestamp will be updated when the first Connected notification arrives. temp: true, tags: make(map[string]int), - conns: make(map[inet.Conn]time.Time), + conns: make(map[network.Conn]time.Time), } s.peers[p] = pi return pi @@ -153,7 +154,7 @@ type peerInfo struct { value int // cached sum of all tag values temp bool // this is a temporary entry holding early tags, and awaiting connections - conns map[inet.Conn]time.Time // start time of each connection + conns map[network.Conn]time.Time // start time of each connection firstSeen time.Time // timestamp when we began tracking this peer. } @@ -206,7 +207,7 @@ func (cm *BasicConnMgr) background() { // getConnsToClose runs the heuristics described in TrimOpenConns and returns the // connections to close. -func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { +func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { if cm.lowWater == 0 || cm.highWater == 0 { // disabled return nil @@ -248,7 +249,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { target := nconns - cm.lowWater // slightly overallocate because we may have more than one conns per peer - selected := make([]inet.Conn, 0, target+10) + selected := make([]network.Conn, 0, target+10) for _, inf := range candidates { if target <= 0 { @@ -281,7 +282,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []inet.Conn { // GetTagInfo is called to fetch the tag information associated with a given // peer, nil is returned if p refers to an unknown peer. -func (cm *BasicConnMgr) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { +func (cm *BasicConnMgr) GetTagInfo(p peer.ID) *connmgr.TagInfo { s := cm.segments.get(p) s.Lock() defer s.Unlock() @@ -291,7 +292,7 @@ func (cm *BasicConnMgr) GetTagInfo(p peer.ID) *ifconnmgr.TagInfo { return nil } - out := &ifconnmgr.TagInfo{ + out := &connmgr.TagInfo{ FirstSeen: pi.firstSeen, Value: pi.value, Tags: make(map[string]int), @@ -384,7 +385,7 @@ func (cm *BasicConnMgr) GetInfo() CMInfo { // Notifee returns a sink through which Notifiers can inform the BasicConnMgr when // events occur. Currently, the notifee only reacts upon connection events // {Connected, Disconnected}. -func (cm *BasicConnMgr) Notifee() inet.Notifiee { +func (cm *BasicConnMgr) Notifee() network.Notifiee { return (*cmNotifee)(cm) } @@ -397,7 +398,7 @@ func (nn *cmNotifee) cm() *BasicConnMgr { // Connected is called by notifiers to inform that a new connection has been established. // The notifee updates the BasicConnMgr to start tracking the connection. If the new connection // count exceeds the high watermark, a trim may be triggered. -func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { +func (nn *cmNotifee) Connected(n network.Network, c network.Conn) { cm := nn.cm() p := c.RemotePeer() @@ -412,7 +413,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { id: id, firstSeen: time.Now(), tags: make(map[string]int), - conns: make(map[inet.Conn]time.Time), + conns: make(map[network.Conn]time.Time), } s.peers[id] = pinfo } else if pinfo.temp { @@ -435,7 +436,7 @@ func (nn *cmNotifee) Connected(n inet.Network, c inet.Conn) { // Disconnected is called by notifiers to inform that an existing connection has been closed or terminated. // The notifee updates the BasicConnMgr accordingly to stop tracking the connection, and performs housekeeping. -func (nn *cmNotifee) Disconnected(n inet.Network, c inet.Conn) { +func (nn *cmNotifee) Disconnected(n network.Network, c network.Conn) { cm := nn.cm() p := c.RemotePeer() @@ -463,13 +464,13 @@ func (nn *cmNotifee) Disconnected(n inet.Network, c inet.Conn) { } // Listen is no-op in this implementation. -func (nn *cmNotifee) Listen(n inet.Network, addr ma.Multiaddr) {} +func (nn *cmNotifee) Listen(n network.Network, addr ma.Multiaddr) {} // ListenClose is no-op in this implementation. -func (nn *cmNotifee) ListenClose(n inet.Network, addr ma.Multiaddr) {} +func (nn *cmNotifee) ListenClose(n network.Network, addr ma.Multiaddr) {} // OpenedStream is no-op in this implementation. -func (nn *cmNotifee) OpenedStream(inet.Network, inet.Stream) {} +func (nn *cmNotifee) OpenedStream(network.Network, network.Stream) {} // ClosedStream is no-op in this implementation. -func (nn *cmNotifee) ClosedStream(inet.Network, inet.Stream) {} +func (nn *cmNotifee) ClosedStream(network.Network, network.Stream) {} diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index d7933ae893..edaff7bc6c 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -6,18 +6,20 @@ import ( "time" detectrace "github.com/ipfs/go-detect-race" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - tu "github.com/libp2p/go-testutil" + + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + + tu "github.com/libp2p/go-libp2p-core/test" ma "github.com/multiformats/go-multiaddr" ) type tconn struct { - inet.Conn + network.Conn peer peer.ID closed bool - disconnectNotify func(net inet.Network, conn inet.Conn) + disconnectNotify func(net network.Network, conn network.Conn) } func (c *tconn) Close() error { @@ -40,7 +42,7 @@ func (c *tconn) RemoteMultiaddr() ma.Multiaddr { return addr } -func randConn(t testing.TB, discNotify func(inet.Network, inet.Conn)) inet.Conn { +func randConn(t testing.TB, discNotify func(network.Network, network.Conn)) network.Conn { pid := tu.RandPeerIDFatal(t) return &tconn{peer: pid, disconnectNotify: discNotify} } @@ -49,7 +51,7 @@ func TestConnTrimming(t *testing.T) { cm := NewConnManager(200, 300, 0) not := cm.Notifee() - var conns []inet.Conn + var conns []network.Conn for i := 0; i < 300; i++ { rc := randConn(t, nil) conns = append(conns, rc) @@ -310,7 +312,7 @@ func TestQuickBurstRespectsSilencePeriod(t *testing.T) { cm := NewConnManager(10, 20, 0) not := cm.Notifee() - var conns []inet.Conn + var conns []network.Conn // quickly produce 30 connections (sending us above the high watermark) for i := 0; i < 30; i++ { @@ -349,7 +351,7 @@ func TestPeerProtectionSingleTag(t *testing.T) { not := cm.Notifee() // produce 20 connections with unique peers. - var conns []inet.Conn + var conns []network.Conn for i := 0; i < 20; i++ { rc := randConn(t, not.Disconnected) conns = append(conns, rc) @@ -358,7 +360,7 @@ func TestPeerProtectionSingleTag(t *testing.T) { } // protect the first 5 peers. - var protected []inet.Conn + var protected []network.Conn for _, c := range conns[0:5] { cm.Protect(c.RemotePeer(), "global") protected = append(protected, c) @@ -412,7 +414,7 @@ func TestPeerProtectionMultipleTags(t *testing.T) { not := cm.Notifee() // produce 20 connections with unique peers. - var conns []inet.Conn + var conns []network.Conn for i := 0; i < 20; i++ { rc := randConn(t, not.Disconnected) conns = append(conns, rc) @@ -421,7 +423,7 @@ func TestPeerProtectionMultipleTags(t *testing.T) { } // protect the first 5 peers under two tags. - var protected []inet.Conn + var protected []network.Conn for _, c := range conns[0:5] { cm.Protect(c.RemotePeer(), "tag1") cm.Protect(c.RemotePeer(), "tag2") From c86093dbed349fe347566c6ee64eefe49ccf5abb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sat, 25 May 2019 13:01:32 +0100 Subject: [PATCH 1261/3965] migrate to consolidated types (#12) --- p2p/net/conn-security-multistream/ssms.go | 18 +++++++++--------- p2p/net/conn-security-multistream/ssms_test.go | 4 ++-- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/p2p/net/conn-security-multistream/ssms.go b/p2p/net/conn-security-multistream/ssms.go index b74b749938..219ee6fcff 100644 --- a/p2p/net/conn-security-multistream/ssms.go +++ b/p2p/net/conn-security-multistream/ssms.go @@ -5,8 +5,8 @@ import ( "fmt" "net" - connsec "github.com/libp2p/go-conn-security" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" mss "github.com/multiformats/go-multistream" ) @@ -16,19 +16,19 @@ import ( // after use. type SSMuxer struct { mux mss.MultistreamMuxer - tpts map[string]connsec.Transport + tpts map[string]sec.SecureTransport OrderPreference []string } -var _ connsec.Transport = (*SSMuxer)(nil) +var _ sec.SecureTransport = (*SSMuxer)(nil) // AddTransport adds a stream security transport to this multistream muxer. // // This method is *not* thread-safe. It should be called only when initializing // the SSMuxer. -func (sm *SSMuxer) AddTransport(path string, transport connsec.Transport) { +func (sm *SSMuxer) AddTransport(path string, transport sec.SecureTransport) { if sm.tpts == nil { - sm.tpts = make(map[string]connsec.Transport, 1) + sm.tpts = make(map[string]sec.SecureTransport, 1) } sm.mux.AddHandler(path, nil) @@ -38,7 +38,7 @@ func (sm *SSMuxer) AddTransport(path string, transport connsec.Transport) { // SecureInbound secures an inbound connection using this multistream // multiplexed stream security transport. -func (sm *SSMuxer) SecureInbound(ctx context.Context, insecure net.Conn) (connsec.Conn, error) { +func (sm *SSMuxer) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { tpt, err := sm.selectProto(ctx, insecure, true) if err != nil { return nil, err @@ -48,7 +48,7 @@ func (sm *SSMuxer) SecureInbound(ctx context.Context, insecure net.Conn) (connse // SecureOutbound secures an outbound connection using this multistream // multiplexed stream security transport. -func (sm *SSMuxer) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (connsec.Conn, error) { +func (sm *SSMuxer) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { tpt, err := sm.selectProto(ctx, insecure, false) if err != nil { return nil, err @@ -56,7 +56,7 @@ func (sm *SSMuxer) SecureOutbound(ctx context.Context, insecure net.Conn, p peer return tpt.SecureOutbound(ctx, insecure, p) } -func (sm *SSMuxer) selectProto(ctx context.Context, insecure net.Conn, server bool) (connsec.Transport, error) { +func (sm *SSMuxer) selectProto(ctx context.Context, insecure net.Conn, server bool) (sec.SecureTransport, error) { var proto string var err error done := make(chan struct{}) diff --git a/p2p/net/conn-security-multistream/ssms_test.go b/p2p/net/conn-security-multistream/ssms_test.go index 457c1ce644..c10830f0f5 100644 --- a/p2p/net/conn-security-multistream/ssms_test.go +++ b/p2p/net/conn-security-multistream/ssms_test.go @@ -6,8 +6,8 @@ import ( "sync" "testing" - insecure "github.com/libp2p/go-conn-security/insecure" - sst "github.com/libp2p/go-conn-security/test" + "github.com/libp2p/go-libp2p-core/sec/insecure" + sst "github.com/libp2p/go-libp2p-testing/suites/sec" ) func TestCommonProto(t *testing.T) { From 3ca427f3bc169859acfffafa7ca53e8a9d426dcf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sat, 25 May 2019 13:27:36 +0100 Subject: [PATCH 1262/3965] Consolidate abstractions and core types into go-libp2p-core (#27) --- p2p/net/pnet/protector.go | 2 +- p2p/net/pnet/psk_conn.go | 11 ++++++----- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index 38706817ab..8b8d2dd015 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -5,7 +5,7 @@ import ( "io" "net" - ipnet "github.com/libp2p/go-libp2p-interface-pnet" + ipnet "github.com/libp2p/go-libp2p-core/pnet" ) var _ ipnet.Protector = (*protector)(nil) diff --git a/p2p/net/pnet/psk_conn.go b/p2p/net/pnet/psk_conn.go index 5b16ef7a6c..5d35d8794c 100644 --- a/p2p/net/pnet/psk_conn.go +++ b/p2p/net/pnet/psk_conn.go @@ -6,17 +6,18 @@ import ( "io" "net" - salsa20 "github.com/davidlazar/go-crypto/salsa20" + "github.com/libp2p/go-libp2p-core/pnet" + + "github.com/davidlazar/go-crypto/salsa20" pool "github.com/libp2p/go-buffer-pool" - ipnet "github.com/libp2p/go-libp2p-interface-pnet" ) // we are using buffer pool as user needs their slice back // so we can't do XOR cripter in place var ( - errShortNonce = ipnet.NewError("could not read full nonce") - errInsecureNil = ipnet.NewError("insecure is nil") - errPSKNil = ipnet.NewError("pre-shread key is nil") + errShortNonce = pnet.NewError("could not read full nonce") + errInsecureNil = pnet.NewError("insecure is nil") + errPSKNil = pnet.NewError("pre-shread key is nil") ) type pskConn struct { From aa541ddf45c27e04e2f8d7dae26d08ce2e0ca637 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sat, 25 May 2019 14:49:18 +0100 Subject: [PATCH 1263/3965] migrate to consolidated types; add travis config (#1) --- p2p/muxer/mplex/multiplex.go | 11 ++++++----- p2p/muxer/mplex/multiplex_test.go | 2 +- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go index 1cef4ef3ff..bf53c9e7fb 100644 --- a/p2p/muxer/mplex/multiplex.go +++ b/p2p/muxer/mplex/multiplex.go @@ -3,8 +3,9 @@ package peerstream_multiplex import ( "net" - mp "github.com/libp2p/go-mplex" // Conn is a connection to a remote peer. - smux "github.com/libp2p/go-stream-muxer" // Conn is a connection to a remote peer. + "github.com/libp2p/go-libp2p-core/mux" + + mp "github.com/libp2p/go-mplex" ) type conn struct { @@ -20,12 +21,12 @@ func (c *conn) IsClosed() bool { } // OpenStream creates a new stream. -func (c *conn) OpenStream() (smux.Stream, error) { +func (c *conn) OpenStream() (mux.MuxedStream, error) { return c.Multiplex.NewStream() } // AcceptStream accepts a stream opened by the other side. -func (c *conn) AcceptStream() (smux.Stream, error) { +func (c *conn) AcceptStream() (mux.MuxedStream, error) { return c.Multiplex.Accept() } @@ -36,6 +37,6 @@ type Transport struct{} // DefaultTransport has default settings for multiplex var DefaultTransport = &Transport{} -func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { +func (t *Transport) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { return &conn{mp.NewMultiplex(nc, isServer)}, nil } diff --git a/p2p/muxer/mplex/multiplex_test.go b/p2p/muxer/mplex/multiplex_test.go index 2b392e231d..5fd46a1f03 100644 --- a/p2p/muxer/mplex/multiplex_test.go +++ b/p2p/muxer/mplex/multiplex_test.go @@ -3,7 +3,7 @@ package peerstream_multiplex import ( "testing" - test "github.com/libp2p/go-stream-muxer/test" + test "github.com/libp2p/go-libp2p-testing/suites/mux" ) func TestMultiplexTransport(t *testing.T) { From 9bcb09e28a5953d22098f0146e366349a15d93fb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 12:39:19 +0100 Subject: [PATCH 1264/3965] complete the migration to consolidated types. (#23) --- p2p/net/upgrader/conn.go | 6 +++--- p2p/net/upgrader/listener.go | 8 +++++++- p2p/net/upgrader/listener_test.go | 32 +++++++++++++++---------------- p2p/net/upgrader/upgrader.go | 20 +++++++++---------- 4 files changed, 36 insertions(+), 30 deletions(-) diff --git a/p2p/net/upgrader/conn.go b/p2p/net/upgrader/conn.go index 4551a0f915..2098b151e7 100644 --- a/p2p/net/upgrader/conn.go +++ b/p2p/net/upgrader/conn.go @@ -3,9 +3,9 @@ package stream import ( "fmt" - mux "github.com/libp2p/go-libp2p-core/mux" - network "github.com/libp2p/go-libp2p-core/network" - transport "github.com/libp2p/go-libp2p-core/transport" + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/transport" ) type transportConn struct { diff --git a/p2p/net/upgrader/listener.go b/p2p/net/upgrader/listener.go index dc45325966..f792c991be 100644 --- a/p2p/net/upgrader/listener.go +++ b/p2p/net/upgrader/listener.go @@ -5,14 +5,20 @@ import ( "fmt" "sync" + "github.com/libp2p/go-libp2p-core/transport" + logging "github.com/ipfs/go-log" tec "github.com/jbenet/go-temp-err-catcher" - transport "github.com/libp2p/go-libp2p-core/transport" manet "github.com/multiformats/go-multiaddr-net" ) var log = logging.Logger("stream-upgrader") +type connErr struct { + conn transport.CapableConn + err error +} + type listener struct { manet.Listener diff --git a/p2p/net/upgrader/listener_test.go b/p2p/net/upgrader/listener_test.go index f3d89f4b3f..c5d9719cc9 100644 --- a/p2p/net/upgrader/listener_test.go +++ b/p2p/net/upgrader/listener_test.go @@ -7,9 +7,9 @@ import ( "sync" "time" - core "github.com/libp2p/go-libp2p-core" - mux "github.com/libp2p/go-libp2p-core/mux" - insecure "github.com/libp2p/go-libp2p-core/sec/insecure" + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec/insecure" tpt "github.com/libp2p/go-libp2p-core/transport" mplex "github.com/libp2p/go-libp2p-mplex" st "github.com/libp2p/go-libp2p-transport-upgrader" @@ -64,7 +64,7 @@ func (m *errorMuxer) NewConn(c net.Conn, isServer bool) (mux.MuxedConn, error) { var _ = Describe("Listener", func() { var ( defaultUpgrader = &st.Upgrader{ - Secure: insecure.New(core.PeerID(1)), + Secure: insecure.New(peer.ID(1)), Muxer: &negotiatingMuxer{}, } ) @@ -90,7 +90,7 @@ var _ = Describe("Listener", func() { return upgrader.UpgradeListener(nil, ln) } - dial := func(upgrader *st.Upgrader, raddr ma.Multiaddr, p core.PeerID) (tpt.CapableConn, error) { + dial := func(upgrader *st.Upgrader, raddr ma.Multiaddr, p peer.ID) (tpt.CapableConn, error) { macon, err := manet.Dial(raddr) if err != nil { return nil, err @@ -105,7 +105,7 @@ var _ = Describe("Listener", func() { It("accepts a single connection", func() { ln := createListener(defaultUpgrader) defer ln.Close() - cconn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(1)) + cconn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) Expect(err).ToNot(HaveOccurred()) sconn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) @@ -117,7 +117,7 @@ var _ = Describe("Listener", func() { defer ln.Close() const num = 10 for i := 0; i < 10; i++ { - cconn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(1)) + cconn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) Expect(err).ToNot(HaveOccurred()) sconn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) @@ -130,7 +130,7 @@ var _ = Describe("Listener", func() { tpt.AcceptTimeout = timeout ln := createListener(defaultUpgrader) defer ln.Close() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) if !Expect(err).ToNot(HaveOccurred()) { return } @@ -150,7 +150,7 @@ var _ = Describe("Listener", func() { It("doesn't accept connections that fail to setup", func() { upgrader := &st.Upgrader{ - Secure: insecure.New(core.PeerID(1)), + Secure: insecure.New(peer.ID(1)), Muxer: &errorMuxer{}, } ln := createListener(upgrader) @@ -163,7 +163,7 @@ var _ = Describe("Listener", func() { } close(done) }() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) if !Expect(err).To(HaveOccurred()) { conn.Close() } @@ -178,7 +178,7 @@ var _ = Describe("Listener", func() { num := 3 * st.AcceptQueueLength bm := newBlockingMuxer() upgrader := &st.Upgrader{ - Secure: insecure.New(core.PeerID(1)), + Secure: insecure.New(peer.ID(1)), Muxer: bm, } ln := createListener(upgrader) @@ -200,7 +200,7 @@ var _ = Describe("Listener", func() { wg.Add(1) go func() { defer GinkgoRecover() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) if Expect(err).ToNot(HaveOccurred()) { stream, err := conn.AcceptStream() // wait for conn to be accepted. if !Expect(err).To(HaveOccurred()) { @@ -227,7 +227,7 @@ var _ = Describe("Listener", func() { for i := 0; i < st.AcceptQueueLength; i++ { go func() { defer GinkgoRecover() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(err).ToNot(HaveOccurred()) dialed <- conn }() @@ -236,7 +236,7 @@ var _ = Describe("Listener", func() { // dial a new connection. This connection should not complete setup, since the queue is full go func() { defer GinkgoRecover() - conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) Expect(err).ToNot(HaveOccurred()) dialed <- conn }() @@ -279,7 +279,7 @@ var _ = Describe("Listener", func() { It("doesn't accept new connections when it is closed", func() { ln := createListener(defaultUpgrader) Expect(ln.Close()).To(Succeed()) - conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(1)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(1)) if !Expect(err).To(HaveOccurred()) { conn.Close() } @@ -287,7 +287,7 @@ var _ = Describe("Listener", func() { It("closes incoming connections that have not yet been accepted", func() { ln := createListener(defaultUpgrader) - conn, err := dial(defaultUpgrader, ln.Multiaddr(), core.PeerID(2)) + conn, err := dial(defaultUpgrader, ln.Multiaddr(), peer.ID(2)) if !Expect(err).ToNot(HaveOccurred()) { ln.Close() return diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index d68ae8cd8d..5e3854b0c1 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -6,11 +6,12 @@ import ( "fmt" "net" - core "github.com/libp2p/go-libp2p-core" - mux "github.com/libp2p/go-libp2p-core/mux" - pnet "github.com/libp2p/go-libp2p-core/pnet" - sec "github.com/libp2p/go-libp2p-core/sec" - transport "github.com/libp2p/go-libp2p-core/transport" + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/pnet" + "github.com/libp2p/go-libp2p-core/sec" + "github.com/libp2p/go-libp2p-core/transport" + filter "github.com/libp2p/go-maddr-filter" manet "github.com/multiformats/go-multiaddr-net" ) @@ -49,7 +50,7 @@ func (u *Upgrader) UpgradeListener(t transport.Transport, list manet.Listener) t // UpgradeOutbound upgrades the given outbound multiaddr-net connection into a // full libp2p-transport connection. -func (u *Upgrader) UpgradeOutbound(ctx context.Context, t transport.Transport, maconn manet.Conn, p core.PeerID) (transport.CapableConn, error) { +func (u *Upgrader) UpgradeOutbound(ctx context.Context, t transport.Transport, maconn manet.Conn, p peer.ID) (transport.CapableConn, error) { if p == "" { return nil, ErrNilPeer } @@ -62,7 +63,7 @@ func (u *Upgrader) UpgradeInbound(ctx context.Context, t transport.Transport, ma return u.upgrade(ctx, t, maconn, "") } -func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn manet.Conn, p core.PeerID) (transport.CapableConn, error) { +func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn manet.Conn, p peer.ID) (transport.CapableConn, error) { if u.Filters != nil && u.Filters.AddrBlocked(maconn.RemoteMultiaddr()) { log.Debugf("blocked connection from %s", maconn.RemoteMultiaddr()) maconn.Close() @@ -78,7 +79,6 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma } conn = pconn } else if pnet.ForcePrivateNetwork { - conn.Close() log.Error("tried to dial with no Private Network Protector but usage" + " of Private Networks is forced by the enviroment") return nil, pnet.ErrNotInPrivateNetwork @@ -101,14 +101,14 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma }, nil } -func (u *Upgrader) setupSecurity(ctx context.Context, conn net.Conn, p core.PeerID) (sec.SecureConn, error) { +func (u *Upgrader) setupSecurity(ctx context.Context, conn net.Conn, p peer.ID) (sec.SecureConn, error) { if p == "" { return u.Secure.SecureInbound(ctx, conn) } return u.Secure.SecureOutbound(ctx, conn, p) } -func (u *Upgrader) setupMuxer(ctx context.Context, conn net.Conn, p core.PeerID) (mux.MuxedConn, error) { +func (u *Upgrader) setupMuxer(ctx context.Context, conn net.Conn, p peer.ID) (mux.MuxedConn, error) { // TODO: The muxer should take a context. done := make(chan struct{}) From ed7828ceefc655c6120275ce2e70fb368cb04e3c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 15:11:07 +0100 Subject: [PATCH 1265/3965] migrate to consolidated types (#40) --- p2p/transport/tcp/reuseport.go | 2 +- p2p/transport/tcp/tcp.go | 13 +++++++------ p2p/transport/tcp/tcp_test.go | 8 +++++--- 3 files changed, 13 insertions(+), 10 deletions(-) diff --git a/p2p/transport/tcp/reuseport.go b/p2p/transport/tcp/reuseport.go index 04a5da3ed1..e3b7e1250f 100644 --- a/p2p/transport/tcp/reuseport.go +++ b/p2p/transport/tcp/reuseport.go @@ -4,7 +4,7 @@ import ( "os" "strings" - reuseport "github.com/libp2p/go-reuseport" + "github.com/libp2p/go-reuseport" ) // envReuseport is the env variable name used to turn off reuse port. diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 84a34a4d4f..1483d67e69 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -6,13 +6,14 @@ import ( "time" logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" tptu "github.com/libp2p/go-libp2p-transport-upgrader" rtpt "github.com/libp2p/go-reuseport-transport" + ma "github.com/multiformats/go-multiaddr" + mafmt "github.com/multiformats/go-multiaddr-fmt" manet "github.com/multiformats/go-multiaddr-net" - mafmt "github.com/whyrusleeping/mafmt" ) // DefaultConnectTimeout is the (default) maximum amount of time the TCP @@ -61,7 +62,7 @@ type TcpTransport struct { reuse rtpt.Transport } -var _ tpt.Transport = &TcpTransport{} +var _ transport.Transport = &TcpTransport{} // NewTCPTransport creates a tcp transport object that tracks dialers and listeners // created. It represents an entire tcp stack (though it might not necessarily be) @@ -94,7 +95,7 @@ func (t *TcpTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (manet.Co } // Dial dials the peer at the remote address. -func (t *TcpTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { +func (t *TcpTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (transport.CapableConn, error) { conn, err := t.maDial(ctx, raddr) if err != nil { return nil, err @@ -119,7 +120,7 @@ func (t *TcpTransport) maListen(laddr ma.Multiaddr) (manet.Listener, error) { } // Listen listens on the given multiaddr. -func (t *TcpTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { +func (t *TcpTransport) Listen(laddr ma.Multiaddr) (transport.Listener, error) { list, err := t.maListen(laddr) if err != nil { return nil, err diff --git a/p2p/transport/tcp/tcp_test.go b/p2p/transport/tcp/tcp_test.go index 4efd1cb6b2..cfceef50f3 100644 --- a/p2p/transport/tcp/tcp_test.go +++ b/p2p/transport/tcp/tcp_test.go @@ -3,10 +3,12 @@ package tcp import ( "testing" - insecure "github.com/libp2p/go-conn-security/insecure" + "github.com/libp2p/go-libp2p-core/sec/insecure" mplex "github.com/libp2p/go-libp2p-mplex" tptu "github.com/libp2p/go-libp2p-transport-upgrader" - utils "github.com/libp2p/go-libp2p-transport/test" + + ttransport "github.com/libp2p/go-libp2p-testing/suites/transport" + ma "github.com/multiformats/go-multiaddr" ) @@ -22,7 +24,7 @@ func TestTcpTransport(t *testing.T) { }) zero := "/ip4/127.0.0.1/tcp/0" - utils.SubtestTransport(t, ta, tb, zero, "peerA") + ttransport.SubtestTransport(t, ta, tb, zero, "peerA") envReuseportVal = false } From 486ba75ea3cab46ae5f8d350e72e418776bd5ec9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 15:24:17 +0100 Subject: [PATCH 1266/3965] migrate to consolidated types; add travis config. (#2) --- p2p/muxer/muxer-multistream/multistream.go | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/p2p/muxer/muxer-multistream/multistream.go b/p2p/muxer/muxer-multistream/multistream.go index ad453a9958..c7b5887983 100644 --- a/p2p/muxer/muxer-multistream/multistream.go +++ b/p2p/muxer/muxer-multistream/multistream.go @@ -7,7 +7,8 @@ import ( "net" "time" - smux "github.com/libp2p/go-stream-muxer" + "github.com/libp2p/go-libp2p-core/mux" + mss "github.com/multiformats/go-multistream" ) @@ -16,7 +17,7 @@ var DefaultNegotiateTimeout = time.Second * 60 type Transport struct { mux *mss.MultistreamMuxer - tpts map[string]smux.Transport + tpts map[string]mux.Multiplexer NegotiateTimeout time.Duration @@ -26,18 +27,18 @@ type Transport struct { func NewBlankTransport() *Transport { return &Transport{ mux: mss.NewMultistreamMuxer(), - tpts: make(map[string]smux.Transport), + tpts: make(map[string]mux.Multiplexer), NegotiateTimeout: DefaultNegotiateTimeout, } } -func (t *Transport) AddTransport(path string, tpt smux.Transport) { +func (t *Transport) AddTransport(path string, tpt mux.Multiplexer) { t.mux.AddHandler(path, nil) t.tpts[path] = tpt t.OrderPreference = append(t.OrderPreference, path) } -func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { +func (t *Transport) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { if t.NegotiateTimeout != 0 { if err := nc.SetDeadline(time.Now().Add(t.NegotiateTimeout)); err != nil { return nil, err From cb81de4268bcfea5c2cd63b3baa6144b51478c5a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 15:44:42 +0100 Subject: [PATCH 1267/3965] migrate to consolidated types; add travis config. (#2) --- p2p/muxer/yamux/yamux.go | 8 ++++---- p2p/muxer/yamux/yamux_test.go | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go index 55e45bdde3..4875761cf4 100644 --- a/p2p/muxer/yamux/yamux.go +++ b/p2p/muxer/yamux/yamux.go @@ -4,7 +4,7 @@ import ( "io/ioutil" "net" - smux "github.com/libp2p/go-stream-muxer" + mux "github.com/libp2p/go-libp2p-core/mux" yamux "github.com/libp2p/go-yamux" ) @@ -24,7 +24,7 @@ func (c *conn) IsClosed() bool { } // OpenStream creates a new stream. -func (c *conn) OpenStream() (smux.Stream, error) { +func (c *conn) OpenStream() (mux.MuxedStream, error) { s, err := c.yamuxSession().OpenStream() if err != nil { return nil, err @@ -34,7 +34,7 @@ func (c *conn) OpenStream() (smux.Stream, error) { } // AcceptStream accepts a stream opened by the other side. -func (c *conn) AcceptStream() (smux.Stream, error) { +func (c *conn) AcceptStream() (mux.MuxedStream, error) { s, err := c.yamuxSession().AcceptStream() return s, err } @@ -61,7 +61,7 @@ func init() { DefaultTransport = (*Transport)(config) } -func (t *Transport) NewConn(nc net.Conn, isServer bool) (smux.Conn, error) { +func (t *Transport) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { var s *yamux.Session var err error if isServer { diff --git a/p2p/muxer/yamux/yamux_test.go b/p2p/muxer/yamux/yamux_test.go index fbd0188b8d..409b0967c8 100644 --- a/p2p/muxer/yamux/yamux_test.go +++ b/p2p/muxer/yamux/yamux_test.go @@ -3,9 +3,9 @@ package sm_yamux import ( "testing" - test "github.com/libp2p/go-stream-muxer/test" + tmux "github.com/libp2p/go-libp2p-testing/suites/mux" ) func TestYamuxTransport(t *testing.T) { - test.SubtestAll(t, DefaultTransport) + tmux.SubtestAll(t, DefaultTransport) } From e132f6af3ffe51f271e3cd9e697ccde202e9d5e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 15:55:50 +0100 Subject: [PATCH 1268/3965] migrate to consolidated types; remove CircleCI config (#127) --- p2p/net/swarm/.circleci/config.yml | 10 ---- p2p/net/swarm/dial_error.go | 3 +- p2p/net/swarm/dial_sync.go | 2 +- p2p/net/swarm/dial_sync_test.go | 2 +- p2p/net/swarm/dial_test.go | 31 ++++++------ p2p/net/swarm/limiter.go | 9 ++-- p2p/net/swarm/limiter_test.go | 20 ++++---- p2p/net/swarm/peers_test.go | 16 +++---- p2p/net/swarm/simul_test.go | 9 ++-- p2p/net/swarm/swarm.go | 76 +++++++++++++++--------------- p2p/net/swarm/swarm_addr_test.go | 9 ++-- p2p/net/swarm/swarm_conn.go | 35 +++++++------- p2p/net/swarm/swarm_dial.go | 21 +++++---- p2p/net/swarm/swarm_listen.go | 7 +-- p2p/net/swarm/swarm_net_test.go | 30 ++++++------ p2p/net/swarm/swarm_notif_test.go | 49 +++++++++---------- p2p/net/swarm/swarm_stream.go | 20 ++++---- p2p/net/swarm/swarm_test.go | 27 ++++++----- p2p/net/swarm/swarm_transport.go | 3 +- p2p/net/swarm/testing/testing.go | 19 ++++---- p2p/net/swarm/transport_test.go | 6 +-- 21 files changed, 205 insertions(+), 199 deletions(-) delete mode 100644 p2p/net/swarm/.circleci/config.yml diff --git a/p2p/net/swarm/.circleci/config.yml b/p2p/net/swarm/.circleci/config.yml deleted file mode 100644 index 68d7e72668..0000000000 --- a/p2p/net/swarm/.circleci/config.yml +++ /dev/null @@ -1,10 +0,0 @@ -version: 2 -jobs: - build: - docker: - - image: circleci/golang:1.11 - environment: - GO111MODULE: "on" - steps: - - checkout - - run: go test -v -race -bench . ./... diff --git a/p2p/net/swarm/dial_error.go b/p2p/net/swarm/dial_error.go index 6d5bb14929..d21796172f 100644 --- a/p2p/net/swarm/dial_error.go +++ b/p2p/net/swarm/dial_error.go @@ -4,7 +4,8 @@ import ( "fmt" "strings" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" + ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index 82256eca48..f746b9a9a3 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -4,7 +4,7 @@ import ( "context" "sync" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" ) // DialFunc is the type of function expected by DialSync. diff --git a/p2p/net/swarm/dial_sync_test.go b/p2p/net/swarm/dial_sync_test.go index 53c3fc25b4..485d1a3171 100644 --- a/p2p/net/swarm/dial_sync_test.go +++ b/p2p/net/swarm/dial_sync_test.go @@ -9,7 +9,7 @@ import ( . "github.com/libp2p/go-libp2p-swarm" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" ) func getMockDialFunc() (DialFunc, func(), context.Context, <-chan struct{}) { diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 99585d2ce7..73bd02c674 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -8,12 +8,15 @@ import ( "time" addrutil "github.com/libp2p/go-addr-util" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/transport" + + testutil "github.com/libp2p/go-libp2p-core/test" swarmt "github.com/libp2p/go-libp2p-swarm/testing" - transport "github.com/libp2p/go-libp2p-transport" - testutil "github.com/libp2p/go-testutil" - ci "github.com/libp2p/go-testutil/ci" + "github.com/libp2p/go-libp2p-testing/ci" + ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" @@ -39,7 +42,7 @@ func TestBasicDialPeer(t *testing.T) { s1 := swarms[0] s2 := swarms[1] - s1.Peerstore().AddAddrs(s2.LocalPeer(), s2.ListenAddresses(), pstore.PermanentAddrTTL) + s1.Peerstore().AddAddrs(s2.LocalPeer(), s2.ListenAddresses(), peerstore.PermanentAddrTTL) c, err := s1.DialPeer(ctx, s2.LocalPeer()) if err != nil { @@ -64,7 +67,7 @@ func TestDialWithNoListeners(t *testing.T) { defer closeSwarms(swarms) s2 := swarms[0] - s1.Peerstore().AddAddrs(s2.LocalPeer(), s2.ListenAddresses(), pstore.PermanentAddrTTL) + s1.Peerstore().AddAddrs(s2.LocalPeer(), s2.ListenAddresses(), peerstore.PermanentAddrTTL) c, err := s1.DialPeer(ctx, s2.LocalPeer()) if err != nil { @@ -108,7 +111,7 @@ func TestSimultDials(t *testing.T) { connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // copy for other peer log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.LocalPeer(), dst, addr) - s.Peerstore().AddAddr(dst, addr, pstore.TempAddrTTL) + s.Peerstore().AddAddr(dst, addr, peerstore.TempAddrTTL) if _, err := s.DialPeer(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } @@ -179,7 +182,7 @@ func TestDialWait(t *testing.T) { s2p, s2addr, s2l := newSilentPeer(t) go acceptAndHang(s2l) defer s2l.Close() - s1.Peerstore().AddAddr(s2p, s2addr, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddr(s2p, s2addr, peerstore.PermanentAddrTTL) before := time.Now() if c, err := s1.DialPeer(ctx, s2p); err == nil { @@ -221,13 +224,13 @@ func TestDialBackoff(t *testing.T) { if err != nil { t.Fatal(err) } - s1.Peerstore().AddAddrs(s2.LocalPeer(), s2addrs, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddrs(s2.LocalPeer(), s2addrs, peerstore.PermanentAddrTTL) // dial to a non-existent peer. s3p, s3addr, s3l := newSilentPeer(t) go acceptAndHang(s3l) defer s3l.Close() - s1.Peerstore().AddAddr(s3p, s3addr, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddr(s3p, s3addr, peerstore.PermanentAddrTTL) // in this test we will: // 1) dial 10x to each node. @@ -430,7 +433,7 @@ func TestDialBackoffClears(t *testing.T) { defer s2l.Close() // phase 1 -- dial to non-operational addresses - s1.Peerstore().AddAddr(s2.LocalPeer(), s2bad, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddr(s2.LocalPeer(), s2bad, peerstore.PermanentAddrTTL) before := time.Now() if c, err := s1.DialPeer(ctx, s2.LocalPeer()); err == nil { @@ -459,7 +462,7 @@ func TestDialBackoffClears(t *testing.T) { if err != nil { t.Fatal(err) } - s1.Peerstore().AddAddrs(s2.LocalPeer(), ifaceAddrs1, pstore.PermanentAddrTTL) + s1.Peerstore().AddAddrs(s2.LocalPeer(), ifaceAddrs1, peerstore.PermanentAddrTTL) if _, err := s1.DialPeer(ctx, s2.LocalPeer()); err == nil { t.Fatal("should have failed to dial backed off peer") @@ -498,7 +501,7 @@ func TestDialPeerFailed(t *testing.T) { testedSwarm.Peerstore().AddAddr( targetSwarm.LocalPeer(), silentPeerAddress, - pstore.PermanentAddrTTL) + peerstore.PermanentAddrTTL) } _, err := testedSwarm.DialPeer(ctx, targetSwarm.LocalPeer()) diff --git a/p2p/net/swarm/limiter.go b/p2p/net/swarm/limiter.go index bd4ab57256..6808dd71a6 100644 --- a/p2p/net/swarm/limiter.go +++ b/p2p/net/swarm/limiter.go @@ -7,14 +7,15 @@ import ( "sync" "time" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" + addrutil "github.com/libp2p/go-addr-util" - peer "github.com/libp2p/go-libp2p-peer" - transport "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) type dialResult struct { - Conn transport.Conn + Conn transport.CapableConn Addr ma.Multiaddr Err error } @@ -53,7 +54,7 @@ type dialLimiter struct { waitingOnPeerLimit map[peer.ID][]*dialJob } -type dialfunc func(context.Context, peer.ID, ma.Multiaddr) (transport.Conn, error) +type dialfunc func(context.Context, peer.ID, ma.Multiaddr) (transport.CapableConn, error) func newDialLimiter(df dialfunc) *dialLimiter { fd := ConcurrentFdDials diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 4338cad50a..32d87fa13d 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -9,8 +9,8 @@ import ( "testing" "time" - peer "github.com/libp2p/go-libp2p-peer" - transport "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" ma "github.com/multiformats/go-multiaddr" mafmt "github.com/whyrusleeping/mafmt" ) @@ -56,13 +56,13 @@ func tryDialAddrs(ctx context.Context, l *dialLimiter, p peer.ID, addrs []ma.Mul } func hangDialFunc(hang chan struct{}) dialfunc { - return func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.Conn, error) { + return func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.CapableConn, error) { if mafmt.UTP.Matches(a) { - return transport.Conn(nil), nil + return transport.CapableConn(nil), nil } if tcpPortOver(a, 10) { - return transport.Conn(nil), nil + return transport.CapableConn(nil), nil } <-hang @@ -173,9 +173,9 @@ func TestFDLimiting(t *testing.T) { func TestTokenRedistribution(t *testing.T) { var lk sync.Mutex hangchs := make(map[peer.ID]chan struct{}) - df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.Conn, error) { + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.CapableConn, error) { if tcpPortOver(a, 10) { - return (transport.Conn)(nil), nil + return (transport.CapableConn)(nil), nil } lk.Lock() @@ -268,9 +268,9 @@ func TestTokenRedistribution(t *testing.T) { } func TestStressLimiter(t *testing.T) { - df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.Conn, error) { + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.CapableConn, error) { if tcpPortOver(a, 1000) { - return transport.Conn(nil), nil + return transport.CapableConn(nil), nil } time.Sleep(time.Millisecond * time.Duration(5+rand.Intn(100))) @@ -322,7 +322,7 @@ func TestStressLimiter(t *testing.T) { } func TestFDLimitUnderflow(t *testing.T) { - df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.Conn, error) { + df := func(ctx context.Context, p peer.ID, a ma.Multiaddr) (transport.CapableConn, error) { timeout := make(chan bool, 1) go func() { time.Sleep(time.Second * 5) diff --git a/p2p/net/swarm/peers_test.go b/p2p/net/swarm/peers_test.go index 76cc00b022..fd1984d072 100644 --- a/p2p/net/swarm/peers_test.go +++ b/p2p/net/swarm/peers_test.go @@ -1,19 +1,19 @@ package swarm_test import ( + "context" "testing" - "context" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + ma "github.com/multiformats/go-multiaddr" . "github.com/libp2p/go-libp2p-swarm" ) func TestPeers(t *testing.T) { - ctx := context.Background() swarms := makeSwarms(ctx, t, 2) s1 := swarms[0] @@ -21,7 +21,7 @@ func TestPeers(t *testing.T) { connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // TODO: make a DialAddr func. - s.Peerstore().AddAddr(dst, addr, pstore.PermanentAddrTTL) + s.Peerstore().AddAddr(dst, addr, peerstore.PermanentAddrTTL) // t.Logf("connections from %s", s.LocalPeer()) // for _, c := range s.ConnsToPeer(dst) { // t.Logf("connection from %s to %s: %v", s.LocalPeer(), dst, c) @@ -35,10 +35,10 @@ func TestPeers(t *testing.T) { s1GotConn := make(chan struct{}, 0) s2GotConn := make(chan struct{}, 0) - s1.SetConnHandler(func(c inet.Conn) { + s1.SetConnHandler(func(c network.Conn) { s1GotConn <- struct{}{} }) - s2.SetConnHandler(func(c inet.Conn) { + s2.SetConnHandler(func(c network.Conn) { s2GotConn <- struct{}{} }) diff --git a/p2p/net/swarm/simul_test.go b/p2p/net/swarm/simul_test.go index b162ab380f..0373e37dfb 100644 --- a/p2p/net/swarm/simul_test.go +++ b/p2p/net/swarm/simul_test.go @@ -7,13 +7,14 @@ import ( "testing" "time" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - ci "github.com/libp2p/go-testutil/ci" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + ma "github.com/multiformats/go-multiaddr" . "github.com/libp2p/go-libp2p-swarm" swarmt "github.com/libp2p/go-libp2p-swarm/testing" + "github.com/libp2p/go-libp2p-testing/ci" ) func TestSimultOpen(t *testing.T) { @@ -30,7 +31,7 @@ func TestSimultOpen(t *testing.T) { defer wg.Done() // copy for other peer log.Debugf("TestSimultOpen: connecting: %s --> %s (%s)", s.LocalPeer(), dst, addr) - s.Peerstore().AddAddr(dst, addr, pstore.PermanentAddrTTL) + s.Peerstore().AddAddr(dst, addr, peerstore.PermanentAddrTTL) if _, err := s.DialPeer(ctx, dst); err != nil { t.Error("error swarm dialing to peer", err) } diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 1094e72e19..b14e9cf0b9 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -9,14 +9,16 @@ import ( "sync/atomic" "time" + "github.com/libp2p/go-libp2p-core/metrics" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/transport" + logging "github.com/ipfs/go-log" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" - metrics "github.com/libp2p/go-libp2p-metrics" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - transport "github.com/libp2p/go-libp2p-transport" + filter "github.com/libp2p/go-maddr-filter" mafilter "github.com/whyrusleeping/multiaddr-filter" ) @@ -47,7 +49,7 @@ type Swarm struct { refs sync.WaitGroup local peer.ID - peers pstore.Peerstore + peers peerstore.Peerstore conns struct { sync.RWMutex @@ -61,7 +63,7 @@ type Swarm struct { notifs struct { sync.RWMutex - m map[inet.Notifiee]struct{} + m map[network.Notifiee]struct{} } transports struct { @@ -87,7 +89,7 @@ type Swarm struct { } // NewSwarm constructs a Swarm -func NewSwarm(ctx context.Context, local peer.ID, peers pstore.Peerstore, bwc metrics.Reporter) *Swarm { +func NewSwarm(ctx context.Context, local peer.ID, peers peerstore.Peerstore, bwc metrics.Reporter) *Swarm { s := &Swarm{ local: local, peers: peers, @@ -98,7 +100,7 @@ func NewSwarm(ctx context.Context, local peer.ID, peers pstore.Peerstore, bwc me s.conns.m = make(map[peer.ID][]*Conn) s.listeners.m = make(map[transport.Listener]struct{}) s.transports.m = make(map[int]transport.Transport) - s.notifs.m = make(map[inet.Notifiee]struct{}) + s.notifs.m = make(map[network.Notifiee]struct{}) s.dsync = NewDialSync(s.doDial) s.limiter = newDialLimiter(s.dialAddr) @@ -165,7 +167,7 @@ func (s *Swarm) Process() goprocess.Process { return s.proc } -func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { +func (s *Swarm) addConn(tc transport.CapableConn, dir network.Direction) (*Conn, error) { // The underlying transport (or the dialer) *should* filter it's own // connections but we should double check anyways. raddr := tc.RemoteMultiaddr() @@ -194,7 +196,7 @@ func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { } // Wrap and register the connection. - stat := inet.Stat{Direction: dir} + stat := network.Stat{Direction: dir} c := &Conn{ conn: tc, swarm: s, @@ -217,7 +219,7 @@ func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { // This should be fast, no reason to wait till later. s.dsync.CancelDial(p) - s.notifyAll(func(f inet.Notifiee) { + s.notifyAll(func(f network.Notifiee) { f.Connected(s, c) }) c.notifyLk.Unlock() @@ -235,7 +237,7 @@ func (s *Swarm) addConn(tc transport.Conn, dir inet.Direction) (*Conn, error) { } // Peerstore returns this swarms internal Peerstore. -func (s *Swarm) Peerstore() pstore.Peerstore { +func (s *Swarm) Peerstore() peerstore.Peerstore { return s.peers } @@ -253,30 +255,30 @@ func (s *Swarm) Close() error { // SetConnHandler assigns the handler for new connections. // You will rarely use this. See SetStreamHandler -func (s *Swarm) SetConnHandler(handler inet.ConnHandler) { +func (s *Swarm) SetConnHandler(handler network.ConnHandler) { s.connh.Store(handler) } // ConnHandler gets the handler for new connections. -func (s *Swarm) ConnHandler() inet.ConnHandler { - handler, _ := s.connh.Load().(inet.ConnHandler) +func (s *Swarm) ConnHandler() network.ConnHandler { + handler, _ := s.connh.Load().(network.ConnHandler) return handler } // SetStreamHandler assigns the handler for new streams. -func (s *Swarm) SetStreamHandler(handler inet.StreamHandler) { +func (s *Swarm) SetStreamHandler(handler network.StreamHandler) { s.streamh.Store(handler) } // StreamHandler gets the handler for new streams. -func (s *Swarm) StreamHandler() inet.StreamHandler { - handler, _ := s.streamh.Load().(inet.StreamHandler) +func (s *Swarm) StreamHandler() network.StreamHandler { + handler, _ := s.streamh.Load().(network.StreamHandler) return handler } // NewStream creates a new stream on any available connection to peer, dialing // if necessary. -func (s *Swarm) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { +func (s *Swarm) NewStream(ctx context.Context, p peer.ID) (network.Stream, error) { log.Debugf("[%s] opening stream to peer [%s]", s.local, p) // Algorithm: @@ -295,8 +297,8 @@ func (s *Swarm) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { for { c := s.bestConnToPeer(p) if c == nil { - if nodial, _ := inet.GetNoDial(ctx); nodial { - return nil, inet.ErrNoConn + if nodial, _ := network.GetNoDial(ctx); nodial { + return nil, network.ErrNoConn } if dials >= DialAttempts { @@ -322,13 +324,13 @@ func (s *Swarm) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { } // ConnsToPeer returns all the live connections to peer. -func (s *Swarm) ConnsToPeer(p peer.ID) []inet.Conn { +func (s *Swarm) ConnsToPeer(p peer.ID) []network.Conn { // TODO: Consider sorting the connection list best to worst. Currently, // it's sorted oldest to newest. s.conns.RLock() defer s.conns.RUnlock() conns := s.conns.m[p] - output := make([]inet.Conn, len(conns)) + output := make([]network.Conn, len(conns)) for i, c := range conns { output[i] = c } @@ -366,20 +368,20 @@ func (s *Swarm) bestConnToPeer(p peer.ID) *Conn { // Connectedness returns our "connectedness" state with the given peer. // // To check if we have an open connection, use `s.Connectedness(p) == -// inet.Connected`. -func (s *Swarm) Connectedness(p peer.ID) inet.Connectedness { +// network.Connected`. +func (s *Swarm) Connectedness(p peer.ID) network.Connectedness { if s.bestConnToPeer(p) != nil { - return inet.Connected + return network.Connected } - return inet.NotConnected + return network.NotConnected } // Conns returns a slice of all connections. -func (s *Swarm) Conns() []inet.Conn { +func (s *Swarm) Conns() []network.Conn { s.conns.RLock() defer s.conns.RUnlock() - conns := make([]inet.Conn, 0, len(s.conns.m)) + conns := make([]network.Conn, 0, len(s.conns.m)) for _, cs := range s.conns.m { for _, c := range cs { conns = append(conns, c) @@ -399,7 +401,7 @@ func (s *Swarm) ClosePeer(p peer.ID) error { default: errCh := make(chan error) for _, c := range conns { - go func(c inet.Conn) { + go func(c network.Conn) { errCh <- c.Close() }(c) } @@ -441,13 +443,13 @@ func (s *Swarm) Backoff() *DialBackoff { } // notifyAll sends a signal to all Notifiees -func (s *Swarm) notifyAll(notify func(inet.Notifiee)) { +func (s *Swarm) notifyAll(notify func(network.Notifiee)) { var wg sync.WaitGroup s.notifs.RLock() wg.Add(len(s.notifs.m)) for f := range s.notifs.m { - go func(f inet.Notifiee) { + go func(f network.Notifiee) { defer wg.Done() notify(f) }(f) @@ -458,14 +460,14 @@ func (s *Swarm) notifyAll(notify func(inet.Notifiee)) { } // Notify signs up Notifiee to receive signals when events happen -func (s *Swarm) Notify(f inet.Notifiee) { +func (s *Swarm) Notify(f network.Notifiee) { s.notifs.Lock() s.notifs.m[f] = struct{}{} s.notifs.Unlock() } // StopNotify unregisters Notifiee fromr receiving signals -func (s *Swarm) StopNotify(f inet.Notifiee) { +func (s *Swarm) StopNotify(f network.Notifiee) { s.notifs.Lock() delete(s.notifs.m, f) s.notifs.Unlock() @@ -500,5 +502,5 @@ func (s *Swarm) String() string { } // Swarm is a Network. -var _ inet.Network = (*Swarm)(nil) -var _ transport.Network = (*Swarm)(nil) +var _ network.Network = (*Swarm)(nil) +var _ transport.TransportNetwork = (*Swarm)(nil) diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index bafcfb6697..e75d660615 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -4,8 +4,9 @@ import ( "context" "testing" - pstore "github.com/libp2p/go-libp2p-peerstore" - testutil "github.com/libp2p/go-testutil" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/test" + ma "github.com/multiformats/go-multiaddr" ) @@ -23,8 +24,8 @@ func TestDialBadAddrs(t *testing.T) { s := makeSwarms(ctx, t, 1)[0] test := func(a ma.Multiaddr) { - p := testutil.RandPeerIDFatal(t) - s.Peerstore().AddAddr(p, a, pstore.PermanentAddrTTL) + p := test.RandPeerIDFatal(t) + s.Peerstore().AddAddr(p, a, peerstore.PermanentAddrTTL) if _, err := s.DialPeer(ctx, p); err == nil { t.Errorf("swarm should not dial: %s", p) } diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index 26a7794d15..cdb9866a2a 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -5,11 +5,12 @@ import ( "fmt" "sync" - ic "github.com/libp2p/go-libp2p-crypto" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - transport "github.com/libp2p/go-libp2p-transport" - smux "github.com/libp2p/go-stream-muxer" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" + ma "github.com/multiformats/go-multiaddr" ) @@ -21,7 +22,7 @@ var ErrConnClosed = errors.New("connection closed") // Conn is the connection type used by swarm. In general, you won't use this // type directly. type Conn struct { - conn transport.Conn + conn transport.CapableConn swarm *Swarm closeOnce sync.Once @@ -34,7 +35,7 @@ type Conn struct { m map[*Stream]struct{} } - stat inet.Stat + stat network.Stat } // Close closes this connection. @@ -71,7 +72,7 @@ func (c *Conn) doClose() { c.notifyLk.Lock() defer c.notifyLk.Unlock() - c.swarm.notifyAll(func(f inet.Notifiee) { + c.swarm.notifyAll(func(f network.Notifiee) { f.Disconnected(c.swarm, c) }) c.swarm.refs.Done() // taken in Swarm.addConn @@ -100,7 +101,7 @@ func (c *Conn) start() { } c.swarm.refs.Add(1) go func() { - s, err := c.addStream(ts, inet.DirInbound) + s, err := c.addStream(ts, network.DirInbound) // Don't defer this. We don't want to block // swarm shutdown on the connection handler. @@ -161,20 +162,20 @@ func (c *Conn) RemotePublicKey() ic.PubKey { } // Stat returns metadata pertaining to this connection -func (c *Conn) Stat() inet.Stat { +func (c *Conn) Stat() network.Stat { return c.stat } // NewStream returns a new Stream from this connection -func (c *Conn) NewStream() (inet.Stream, error) { +func (c *Conn) NewStream() (network.Stream, error) { ts, err := c.conn.OpenStream() if err != nil { return nil, err } - return c.addStream(ts, inet.DirOutbound) + return c.addStream(ts, network.DirOutbound) } -func (c *Conn) addStream(ts smux.Stream, dir inet.Direction) (*Stream, error) { +func (c *Conn) addStream(ts mux.MuxedStream, dir network.Direction) (*Stream, error) { c.streams.Lock() // Are we still online? if c.streams.m == nil { @@ -184,7 +185,7 @@ func (c *Conn) addStream(ts smux.Stream, dir inet.Direction) (*Stream, error) { } // Wrap and register the stream. - stat := inet.Stat{Direction: dir} + stat := network.Stat{Direction: dir} s := &Stream{ stream: ts, conn: c, @@ -202,7 +203,7 @@ func (c *Conn) addStream(ts smux.Stream, dir inet.Direction) (*Stream, error) { s.notifyLk.Lock() c.streams.Unlock() - c.swarm.notifyAll(func(f inet.Notifiee) { + c.swarm.notifyAll(func(f network.Notifiee) { f.OpenedStream(c.swarm, s) }) s.notifyLk.Unlock() @@ -211,10 +212,10 @@ func (c *Conn) addStream(ts smux.Stream, dir inet.Direction) (*Stream, error) { } // GetStreams returns the streams associated with this connection. -func (c *Conn) GetStreams() []inet.Stream { +func (c *Conn) GetStreams() []network.Stream { c.streams.Lock() defer c.streams.Unlock() - streams := make([]inet.Stream, 0, len(c.streams.m)) + streams := make([]network.Stream, 0, len(c.streams.m)) for s := range c.streams.m { streams = append(streams, s) } diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 3feeffe3fe..40faad7848 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -7,12 +7,13 @@ import ( "sync" "time" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" + lgbl "github.com/libp2p/go-libp2p-loggables" + logging "github.com/ipfs/go-log" addrutil "github.com/libp2p/go-addr-util" - lgbl "github.com/libp2p/go-libp2p-loggables" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - transport "github.com/libp2p/go-libp2p-transport" ma "github.com/multiformats/go-multiaddr" ) @@ -180,7 +181,7 @@ func (db *DialBackoff) Clear(p peer.ID) { // the connection will happen over. Swarm can use whichever it choses. // This allows us to use various transport protocols, do NAT traversal/relay, // etc. to achieve connection. -func (s *Swarm) DialPeer(ctx context.Context, p peer.ID) (inet.Conn, error) { +func (s *Swarm) DialPeer(ctx context.Context, p peer.ID) (network.Conn, error) { return s.dialPeer(ctx, p) } @@ -216,7 +217,7 @@ func (s *Swarm) dialPeer(ctx context.Context, p peer.ID) (*Conn, error) { } // apply the DialPeer timeout - ctx, cancel := context.WithTimeout(ctx, inet.GetDialPeerTimeout(ctx)) + ctx, cancel := context.WithTimeout(ctx, network.GetDialPeerTimeout(ctx)) defer cancel() conn, err = s.dsync.DialLock(ctx, p) @@ -330,7 +331,7 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { "localAddr": connC.LocalMultiaddr(), "remoteAddr": connC.RemoteMultiaddr(), } - swarmC, err := s.addConn(connC, inet.DirOutbound) + swarmC, err := s.addConn(connC, network.DirOutbound) if err != nil { logdial["error"] = err.Error() connC.Close() // close the connection. didn't work out :( @@ -366,7 +367,7 @@ func (s *Swarm) filterKnownUndialables(addrs []ma.Multiaddr) []ma.Multiaddr { ) } -func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (transport.Conn, *DialError) { +func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma.Multiaddr) (transport.CapableConn, *DialError) { log.Debugf("%s swarm dialing %s", s.local, p) ctx, cancel := context.WithCancel(ctx) @@ -427,7 +428,7 @@ dialLoop: if ctxErr := ctx.Err(); ctxErr != nil { err.Cause = ctxErr } else if len(err.DialErrors) == 0 { - err.Cause = inet.ErrNoRemoteAddrs + err.Cause = network.ErrNoRemoteAddrs } else { err.Cause = ErrAllDialsFailed } @@ -446,7 +447,7 @@ func (s *Swarm) limitedDial(ctx context.Context, p peer.ID, a ma.Multiaddr, resp }) } -func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (transport.Conn, error) { +func (s *Swarm) dialAddr(ctx context.Context, p peer.ID, addr ma.Multiaddr) (transport.CapableConn, error) { // Just to double check. Costs nothing. if s.local == p { return nil, ErrDialToSelf diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 7b37d53497..85060c4e2c 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -3,7 +3,8 @@ package swarm import ( "fmt" - inet "github.com/libp2p/go-libp2p-net" + "github.com/libp2p/go-libp2p-core/network" + ma "github.com/multiformats/go-multiaddr" ) @@ -59,7 +60,7 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { maddr := list.Multiaddr() // signal to our notifiees on successful conn. - s.notifyAll(func(n inet.Notifiee) { + s.notifyAll(func(n network.Notifiee) { n.Listen(s, maddr) }) @@ -83,7 +84,7 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { s.refs.Add(1) go func() { defer s.refs.Done() - _, err := s.addConn(c, inet.DirInbound) + _, err := s.addConn(c, network.DirInbound) if err != nil { // Probably just means that the swarm has been closed. log.Warningf("add conn failed: ", err) diff --git a/p2p/net/swarm/swarm_net_test.go b/p2p/net/swarm/swarm_net_test.go index 425a1707d1..2ba64edb96 100644 --- a/p2p/net/swarm/swarm_net_test.go +++ b/p2p/net/swarm/swarm_net_test.go @@ -6,7 +6,7 @@ import ( "testing" "time" - inet "github.com/libp2p/go-libp2p-net" + "github.com/libp2p/go-libp2p-core/network" . "github.com/libp2p/go-libp2p-swarm/testing" ) @@ -17,14 +17,14 @@ func TestConnectednessCorrect(t *testing.T) { ctx := context.Background() - nets := make([]inet.Network, 4) + nets := make([]network.Network, 4) for i := 0; i < 4; i++ { nets[i] = GenSwarm(t, ctx) } // connect 0-1, 0-2, 0-3, 1-2, 2-3 - dial := func(a, b inet.Network) { + dial := func(a, b network.Network) { DivulgeAddresses(b, a) if _, err := a.DialPeer(ctx, b.LocalPeer()); err != nil { t.Fatalf("Failed to dial: %s", err) @@ -44,14 +44,14 @@ func TestConnectednessCorrect(t *testing.T) { // test those connected show up correctly // test connected - expectConnectedness(t, nets[0], nets[1], inet.Connected) - expectConnectedness(t, nets[0], nets[3], inet.Connected) - expectConnectedness(t, nets[1], nets[2], inet.Connected) - expectConnectedness(t, nets[3], nets[2], inet.Connected) + expectConnectedness(t, nets[0], nets[1], network.Connected) + expectConnectedness(t, nets[0], nets[3], network.Connected) + expectConnectedness(t, nets[1], nets[2], network.Connected) + expectConnectedness(t, nets[3], nets[2], network.Connected) // test not connected - expectConnectedness(t, nets[0], nets[2], inet.NotConnected) - expectConnectedness(t, nets[1], nets[3], inet.NotConnected) + expectConnectedness(t, nets[0], nets[2], network.NotConnected) + expectConnectedness(t, nets[1], nets[3], network.NotConnected) if len(nets[0].Peers()) != 2 { t.Fatal("expected net 0 to have two peers") @@ -71,7 +71,7 @@ func TestConnectednessCorrect(t *testing.T) { time.Sleep(time.Millisecond * 50) - expectConnectedness(t, nets[2], nets[1], inet.NotConnected) + expectConnectedness(t, nets[2], nets[1], network.NotConnected) for _, n := range nets { n.Close() @@ -82,7 +82,7 @@ func TestConnectednessCorrect(t *testing.T) { } } -func expectConnectedness(t *testing.T, a, b inet.Network, expected inet.Connectedness) { +func expectConnectedness(t *testing.T, a, b network.Network, expected network.Connectedness) { es := "%s is connected to %s, but Connectedness incorrect. %s %s %s" atob := a.Connectedness(b.LocalPeer()) btoa := b.Connectedness(a.LocalPeer()) @@ -96,7 +96,7 @@ func expectConnectedness(t *testing.T, a, b inet.Network, expected inet.Connecte } } -func printConns(n inet.Network) string { +func printConns(n network.Network) string { s := fmt.Sprintf("Connections in %s:\n", n) for _, c := range n.Conns() { s = s + fmt.Sprintf("- %s\n", c) @@ -108,12 +108,12 @@ func TestNetworkOpenStream(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - nets := make([]inet.Network, 4) + nets := make([]network.Network, 4) for i := 0; i < 4; i++ { nets[i] = GenSwarm(t, ctx) } - dial := func(a, b inet.Network) { + dial := func(a, b network.Network) { DivulgeAddresses(b, a) if _, err := a.DialPeer(ctx, b.LocalPeer()); err != nil { t.Fatalf("Failed to dial: %s", err) @@ -125,7 +125,7 @@ func TestNetworkOpenStream(t *testing.T) { dial(nets[1], nets[2]) done := make(chan bool) - nets[1].SetStreamHandler(func(s inet.Stream) { + nets[1].SetStreamHandler(func(s network.Stream) { defer close(done) defer s.Close() diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index d05766d6c7..f2c91ebb80 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -1,12 +1,13 @@ package swarm_test import ( + "context" "testing" "time" - "context" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + ma "github.com/multiformats/go-multiaddr" . "github.com/libp2p/go-libp2p-swarm" @@ -41,7 +42,7 @@ func TestNotifications(t *testing.T) { // test everyone got the correct connection opened calls for i, s := range swarms { n := notifiees[i] - notifs := make(map[peer.ID][]inet.Conn) + notifs := make(map[peer.ID][]network.Conn) for j, s2 := range swarms { if i == j { continue @@ -81,7 +82,7 @@ func TestNotifications(t *testing.T) { } } - complement := func(c inet.Conn) (*Swarm, *netNotifiee, *Conn) { + complement := func(c network.Conn) (*Swarm, *netNotifiee, *Conn) { for i, s := range swarms { for _, c2 := range s.Conns() { if c.LocalMultiaddr().Equal(c2.RemoteMultiaddr()) && @@ -94,8 +95,8 @@ func TestNotifications(t *testing.T) { return nil, nil, nil } - testOCStream := func(n *netNotifiee, s inet.Stream) { - var s2 inet.Stream + testOCStream := func(n *netNotifiee, s network.Stream) { + var s2 network.Stream select { case s2 = <-n.openedStream: t.Log("got notif for opened stream") @@ -117,9 +118,9 @@ func TestNotifications(t *testing.T) { } } - streams := make(chan inet.Stream) + streams := make(chan network.Stream) for _, s := range swarms { - s.SetStreamHandler(func(s inet.Stream) { + s.SetStreamHandler(func(s network.Stream) { streams <- s s.Reset() }) @@ -151,7 +152,7 @@ func TestNotifications(t *testing.T) { c.Close() c2.Close() - var c3, c4 inet.Conn + var c3, c4 network.Conn select { case c3 = <-n.disconnected: case <-time.After(timeout): @@ -176,38 +177,38 @@ func TestNotifications(t *testing.T) { type netNotifiee struct { listen chan ma.Multiaddr listenClose chan ma.Multiaddr - connected chan inet.Conn - disconnected chan inet.Conn - openedStream chan inet.Stream - closedStream chan inet.Stream + connected chan network.Conn + disconnected chan network.Conn + openedStream chan network.Stream + closedStream chan network.Stream } func newNetNotifiee(buffer int) *netNotifiee { return &netNotifiee{ listen: make(chan ma.Multiaddr, buffer), listenClose: make(chan ma.Multiaddr, buffer), - connected: make(chan inet.Conn, buffer), - disconnected: make(chan inet.Conn, buffer), - openedStream: make(chan inet.Stream, buffer), - closedStream: make(chan inet.Stream, buffer), + connected: make(chan network.Conn, buffer), + disconnected: make(chan network.Conn, buffer), + openedStream: make(chan network.Stream, buffer), + closedStream: make(chan network.Stream, buffer), } } -func (nn *netNotifiee) Listen(n inet.Network, a ma.Multiaddr) { +func (nn *netNotifiee) Listen(n network.Network, a ma.Multiaddr) { nn.listen <- a } -func (nn *netNotifiee) ListenClose(n inet.Network, a ma.Multiaddr) { +func (nn *netNotifiee) ListenClose(n network.Network, a ma.Multiaddr) { nn.listenClose <- a } -func (nn *netNotifiee) Connected(n inet.Network, v inet.Conn) { +func (nn *netNotifiee) Connected(n network.Network, v network.Conn) { nn.connected <- v } -func (nn *netNotifiee) Disconnected(n inet.Network, v inet.Conn) { +func (nn *netNotifiee) Disconnected(n network.Network, v network.Conn) { nn.disconnected <- v } -func (nn *netNotifiee) OpenedStream(n inet.Network, v inet.Stream) { +func (nn *netNotifiee) OpenedStream(n network.Network, v network.Stream) { nn.openedStream <- v } -func (nn *netNotifiee) ClosedStream(n inet.Network, v inet.Stream) { +func (nn *netNotifiee) ClosedStream(n network.Network, v network.Stream) { nn.closedStream <- v } diff --git a/p2p/net/swarm/swarm_stream.go b/p2p/net/swarm/swarm_stream.go index 754dbd50e7..9dded2a9d7 100644 --- a/p2p/net/swarm/swarm_stream.go +++ b/p2p/net/swarm/swarm_stream.go @@ -7,9 +7,9 @@ import ( "sync/atomic" "time" - inet "github.com/libp2p/go-libp2p-net" - protocol "github.com/libp2p/go-libp2p-protocol" - smux "github.com/libp2p/go-stream-muxer" + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/protocol" ) type streamState int @@ -23,12 +23,12 @@ const ( ) // Validate Stream conforms to the go-libp2p-net Stream interface -var _ inet.Stream = &Stream{} +var _ network.Stream = &Stream{} // Stream is the stream type used by swarm. In general, you won't use this type // directly. type Stream struct { - stream smux.Stream + stream mux.MuxedStream conn *Conn state struct { @@ -40,7 +40,7 @@ type Stream struct { protocol atomic.Value - stat inet.Stat + stat network.Stat } func (s *Stream) String() string { @@ -54,8 +54,8 @@ func (s *Stream) String() string { ) } -// Conn returns the Conn associated with this stream, as an inet.Conn -func (s *Stream) Conn() inet.Conn { +// Conn returns the Conn associated with this stream, as an network.Conn +func (s *Stream) Conn() network.Conn { return s.conn } @@ -133,7 +133,7 @@ func (s *Stream) remove() { s.notifyLk.Lock() defer s.notifyLk.Unlock() - s.conn.swarm.notifyAll(func(f inet.Notifiee) { + s.conn.swarm.notifyAll(func(f network.Notifiee) { f.ClosedStream(s.conn.swarm, s) }) s.conn.swarm.refs.Done() @@ -172,6 +172,6 @@ func (s *Stream) SetWriteDeadline(t time.Time) error { } // Stat returns metadata information for this stream. -func (s *Stream) Stat() inet.Stat { +func (s *Stream) Stat() network.Stat { return s.stat } diff --git a/p2p/net/swarm/swarm_test.go b/p2p/net/swarm/swarm_test.go index 69eceb94c4..b155373f67 100644 --- a/p2p/net/swarm/swarm_test.go +++ b/p2p/net/swarm/swarm_test.go @@ -11,9 +11,10 @@ import ( "time" logging "github.com/ipfs/go-log" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + ma "github.com/multiformats/go-multiaddr" . "github.com/libp2p/go-libp2p-swarm" @@ -22,7 +23,7 @@ import ( var log = logging.Logger("swarm_test") -func EchoStreamHandler(stream inet.Stream) { +func EchoStreamHandler(stream network.Stream) { go func() { defer stream.Close() @@ -77,7 +78,7 @@ func connectSwarms(t *testing.T, ctx context.Context, swarms []*Swarm) { var wg sync.WaitGroup connect := func(s *Swarm, dst peer.ID, addr ma.Multiaddr) { // TODO: make a DialAddr func. - s.Peerstore().AddAddr(dst, addr, pstore.PermanentAddrTTL) + s.Peerstore().AddAddr(dst, addr, peerstore.PermanentAddrTTL) if _, err := s.DialPeer(ctx, dst); err != nil { t.Fatal("error swarm dialing to peer", err) } @@ -116,7 +117,7 @@ func SubtestSwarm(t *testing.T, SwarmNum int, MsgNum int) { _, cancel := context.WithCancel(ctx) got := map[peer.ID]int{} errChan := make(chan error, MsgNum*len(swarms)) - streamChan := make(chan inet.Stream, MsgNum) + streamChan := make(chan network.Stream, MsgNum) // send out "ping" x MsgNum to every peer go func() { @@ -252,7 +253,7 @@ func TestConnHandler(t *testing.T) { swarms := makeSwarms(ctx, t, 5) gotconn := make(chan struct{}, 10) - swarms[0].SetConnHandler(func(conn inet.Conn) { + swarms[0].SetConnHandler(func(conn network.Conn) { gotconn <- struct{}{} }) @@ -283,7 +284,7 @@ func TestAddrBlocking(t *testing.T) { ctx := context.Background() swarms := makeSwarms(ctx, t, 2) - swarms[0].SetConnHandler(func(conn inet.Conn) { + swarms[0].SetConnHandler(func(conn network.Conn) { t.Errorf("no connections should happen! -- %s", conn) }) @@ -294,13 +295,13 @@ func TestAddrBlocking(t *testing.T) { swarms[1].Filters.AddDialFilter(block) - swarms[1].Peerstore().AddAddr(swarms[0].LocalPeer(), swarms[0].ListenAddresses()[0], pstore.PermanentAddrTTL) + swarms[1].Peerstore().AddAddr(swarms[0].LocalPeer(), swarms[0].ListenAddresses()[0], peerstore.PermanentAddrTTL) _, err = swarms[1].DialPeer(ctx, swarms[0].LocalPeer()) if err == nil { t.Fatal("dial should have failed") } - swarms[0].Peerstore().AddAddr(swarms[1].LocalPeer(), swarms[1].ListenAddresses()[0], pstore.PermanentAddrTTL) + swarms[0].Peerstore().AddAddr(swarms[1].LocalPeer(), swarms[1].ListenAddresses()[0], peerstore.PermanentAddrTTL) _, err = swarms[0].DialPeer(ctx, swarms[1].LocalPeer()) if err == nil { t.Fatal("dial should have failed") @@ -312,7 +313,7 @@ func TestFilterBounds(t *testing.T) { swarms := makeSwarms(ctx, t, 2) conns := make(chan struct{}, 8) - swarms[0].SetConnHandler(func(conn inet.Conn) { + swarms[0].SetConnHandler(func(conn network.Conn) { conns <- struct{}{} }) @@ -340,8 +341,8 @@ func TestNoDial(t *testing.T) { ctx := context.Background() swarms := makeSwarms(ctx, t, 2) - _, err := swarms[0].NewStream(inet.WithNoDial(ctx, "swarm test"), swarms[1].LocalPeer()) - if err != inet.ErrNoConn { + _, err := swarms[0].NewStream(network.WithNoDial(ctx, "swarm test"), swarms[1].LocalPeer()) + if err != network.ErrNoConn { t.Fatal("should have failed with ErrNoConn") } } diff --git a/p2p/net/swarm/swarm_transport.go b/p2p/net/swarm/swarm_transport.go index 0431488777..307bfe641f 100644 --- a/p2p/net/swarm/swarm_transport.go +++ b/p2p/net/swarm/swarm_transport.go @@ -4,7 +4,8 @@ import ( "fmt" "strings" - transport "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/transport" + ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/net/swarm/testing/testing.go b/p2p/net/swarm/testing/testing.go index edaaf5b274..6ffe7d2ae4 100644 --- a/p2p/net/swarm/testing/testing.go +++ b/p2p/net/swarm/testing/testing.go @@ -4,17 +4,18 @@ import ( "context" "testing" + "github.com/libp2p/go-libp2p-core/metrics" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-testing/net" + "github.com/libp2p/go-tcp-transport" + csms "github.com/libp2p/go-conn-security-multistream" - metrics "github.com/libp2p/go-libp2p-metrics" - inet "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" secio "github.com/libp2p/go-libp2p-secio" tptu "github.com/libp2p/go-libp2p-transport-upgrader" yamux "github.com/libp2p/go-libp2p-yamux" msmux "github.com/libp2p/go-stream-muxer-multistream" - tcp "github.com/libp2p/go-tcp-transport" - tu "github.com/libp2p/go-testutil" swarm "github.com/libp2p/go-libp2p-swarm" ) @@ -65,7 +66,7 @@ func GenSwarm(t *testing.T, ctx context.Context, opts ...Option) *swarm.Swarm { o(t, &cfg) } - p := tu.RandPeerNetParamsOrFatal(t) + p := tnet.RandPeerNetParamsOrFatal(t) ps := pstoremem.NewPeerstore() ps.AddPubKey(p.ID, p.PubKey) @@ -84,15 +85,15 @@ func GenSwarm(t *testing.T, ctx context.Context, opts ...Option) *swarm.Swarm { t.Fatal(err) } - s.Peerstore().AddAddrs(p.ID, s.ListenAddresses(), pstore.PermanentAddrTTL) + s.Peerstore().AddAddrs(p.ID, s.ListenAddresses(), peerstore.PermanentAddrTTL) } return s } // DivulgeAddresses adds swarm a's addresses to swarm b's peerstore. -func DivulgeAddresses(a, b inet.Network) { +func DivulgeAddresses(a, b network.Network) { id := a.LocalPeer() addrs := a.Peerstore().Addrs(id) - b.Peerstore().AddAddrs(id, addrs, pstore.PermanentAddrTTL) + b.Peerstore().AddAddrs(id, addrs, peerstore.PermanentAddrTTL) } diff --git a/p2p/net/swarm/transport_test.go b/p2p/net/swarm/transport_test.go index 018775b261..f6090a6ef8 100644 --- a/p2p/net/swarm/transport_test.go +++ b/p2p/net/swarm/transport_test.go @@ -6,8 +6,8 @@ import ( swarmt "github.com/libp2p/go-libp2p-swarm/testing" - peer "github.com/libp2p/go-libp2p-peer" - transport "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" ma "github.com/multiformats/go-multiaddr" ) @@ -16,7 +16,7 @@ type dummyTransport struct { proxy bool } -func (dt *dummyTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (transport.Conn, error) { +func (dt *dummyTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (transport.CapableConn, error) { panic("unimplemented") } From 83826f44e3943ab8ea0ce5c66ea0c96776dfbaec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 16:04:17 +0100 Subject: [PATCH 1269/3965] Consolidate abstractions and core types into go-libp2p-core (#24) --- p2p/discovery/routing/routing.go | 8 +++++--- p2p/discovery/routing/routing_test.go | 17 ++++++++--------- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index 5d85b90627..70e70870a5 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -5,8 +5,10 @@ import ( "time" cid "github.com/ipfs/go-cid" - pstore "github.com/libp2p/go-libp2p-peerstore" - routing "github.com/libp2p/go-libp2p-routing" + + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/routing" + mh "github.com/multiformats/go-multihash" ) @@ -53,7 +55,7 @@ func (d *RoutingDiscovery) Advertise(ctx context.Context, ns string, opts ...Opt return ttl, nil } -func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Option) (<-chan pstore.PeerInfo, error) { +func (d *RoutingDiscovery) FindPeers(ctx context.Context, ns string, opts ...Option) (<-chan peer.AddrInfo, error) { var options Options err := options.Apply(opts...) if err != nil { diff --git a/p2p/discovery/routing/routing_test.go b/p2p/discovery/routing/routing_test.go index 5aff567abe..3fa39bd682 100644 --- a/p2p/discovery/routing/routing_test.go +++ b/p2p/discovery/routing/routing_test.go @@ -7,15 +7,14 @@ import ( cid "github.com/ipfs/go-cid" bhost "github.com/libp2p/go-libp2p-blankhost" - host "github.com/libp2p/go-libp2p-host" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" swarmt "github.com/libp2p/go-libp2p-swarm/testing" ) type mockRoutingTable struct { mx sync.Mutex - providers map[string]map[peer.ID]pstore.PeerInfo + providers map[string]map[peer.ID]peer.AddrInfo } type mockRouting struct { @@ -24,7 +23,7 @@ type mockRouting struct { } func NewMockRoutingTable() *mockRoutingTable { - return &mockRoutingTable{providers: make(map[string]map[peer.ID]pstore.PeerInfo)} + return &mockRoutingTable{providers: make(map[string]map[peer.ID]peer.AddrInfo)} } func NewMockRouting(h host.Host, tab *mockRoutingTable) *mockRouting { @@ -37,17 +36,17 @@ func (m *mockRouting) Provide(ctx context.Context, cid cid.Cid, bcast bool) erro pmap, ok := m.tab.providers[cid.String()] if !ok { - pmap = make(map[peer.ID]pstore.PeerInfo) + pmap = make(map[peer.ID]peer.AddrInfo) m.tab.providers[cid.String()] = pmap } - pmap[m.h.ID()] = pstore.PeerInfo{ID: m.h.ID(), Addrs: m.h.Addrs()} + pmap[m.h.ID()] = peer.AddrInfo{ID: m.h.ID(), Addrs: m.h.Addrs()} return nil } -func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit int) <-chan pstore.PeerInfo { - ch := make(chan pstore.PeerInfo) +func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit int) <-chan peer.AddrInfo { + ch := make(chan peer.AddrInfo) go func() { defer close(ch) m.tab.mx.Lock() From 47d54ffb67f962abd00d4e9a7c90280b91ff158b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 16:04:17 +0100 Subject: [PATCH 1270/3965] Consolidate abstractions and core types into go-libp2p-core (#24) --- p2p/discovery/util/util.go | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/p2p/discovery/util/util.go b/p2p/discovery/util/util.go index 2a267ef39e..f2f2d2b093 100644 --- a/p2p/discovery/util/util.go +++ b/p2p/discovery/util/util.go @@ -4,15 +4,16 @@ import ( "context" "time" + "github.com/libp2p/go-libp2p-core/peer" + logging "github.com/ipfs/go-log" - pstore "github.com/libp2p/go-libp2p-peerstore" ) var log = logging.Logger("discovery") -// FindPeers is a utility function that synchonously collects peers from a Discoverer -func FindPeers(ctx context.Context, d Discoverer, ns string, opts ...Option) ([]pstore.PeerInfo, error) { - var res []pstore.PeerInfo +// FindPeers is a utility function that synchronously collects peers from a Discoverer. +func FindPeers(ctx context.Context, d Discoverer, ns string, opts ...Option) ([]peer.AddrInfo, error) { + var res []peer.AddrInfo ch, err := d.FindPeers(ctx, ns, opts...) if err != nil { @@ -26,7 +27,7 @@ func FindPeers(ctx context.Context, d Discoverer, ns string, opts ...Option) ([] return res, nil } -// Advertise is a utility function that persistently advertises a service through an Advertiser +// Advertise is a utility function that persistently advertises a service through an Advertiser. func Advertise(ctx context.Context, a Advertiser, ns string, opts ...Option) { go func() { for { From e92179ba1d23a99f8e198fa6d209bb1f13d2ed1e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 16:23:34 +0100 Subject: [PATCH 1271/3965] migrate to consolidated types (#78) --- .../internal/circuitv1-deprecated/conn.go | 11 ++--- .../internal/circuitv1-deprecated/dial.go | 17 ++++---- .../internal/circuitv1-deprecated/notify.go | 4 +- .../internal/circuitv1-deprecated/relay.go | 43 ++++++++++--------- .../circuitv1-deprecated/relay_test.go | 3 +- .../circuitv1-deprecated/transport.go | 11 ++--- .../circuitv1-deprecated/transport_test.go | 15 ++++--- .../internal/circuitv1-deprecated/util.go | 14 +++--- 8 files changed, 62 insertions(+), 56 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 33b9d94c87..8fa53b3da7 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -5,16 +5,17 @@ import ( "net" "time" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) type Conn struct { - stream inet.Stream - remote pstore.PeerInfo + stream network.Stream + remote peer.AddrInfo host host.Host } diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 9213a48c20..6668f3970f 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -5,13 +5,12 @@ import ( "fmt" "math/rand" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - tpt "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" ma "github.com/multiformats/go-multiaddr" ) -func (d *RelayTransport) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (tpt.Conn, error) { +func (d *RelayTransport) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (transport.CapableConn, error) { c, err := d.Relay().Dial(ctx, a, p) if err != nil { return nil, err @@ -37,7 +36,7 @@ func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (*Conn, err } } - dinfo := &pstore.PeerInfo{ID: p, Addrs: []ma.Multiaddr{}} + dinfo := &peer.AddrInfo{ID: p, Addrs: []ma.Multiaddr{}} if len(destaddr.Bytes()) > 0 { dinfo.Addrs = append(dinfo.Addrs, destaddr) } @@ -47,8 +46,8 @@ func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (*Conn, err return r.tryDialRelays(ctx, *dinfo) } - var rinfo *pstore.PeerInfo - rinfo, err := pstore.InfoFromP2pAddr(relayaddr) + var rinfo *peer.AddrInfo + rinfo, err := peer.AddrInfoFromP2pAddr(relayaddr) if err != nil { return nil, fmt.Errorf("error parsing multiaddr '%s': %s", relayaddr.String(), err) } @@ -56,7 +55,7 @@ func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (*Conn, err return r.DialPeer(ctx, *rinfo, *dinfo) } -func (r *Relay) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) (*Conn, error) { +func (r *Relay) tryDialRelays(ctx context.Context, dinfo peer.AddrInfo) (*Conn, error) { var relays []peer.ID r.mx.Lock() for p := range r.relays { @@ -76,7 +75,7 @@ func (r *Relay) tryDialRelays(ctx context.Context, dinfo pstore.PeerInfo) (*Conn } rctx, cancel := context.WithTimeout(ctx, HopConnectTimeout) - c, err := r.DialPeer(rctx, pstore.PeerInfo{ID: relay}, dinfo) + c, err := r.DialPeer(rctx, peer.AddrInfo{ID: relay}, dinfo) cancel() if err == nil { diff --git a/p2p/protocol/internal/circuitv1-deprecated/notify.go b/p2p/protocol/internal/circuitv1-deprecated/notify.go index 6d9f62dc2a..813daed4d9 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/notify.go +++ b/p2p/protocol/internal/circuitv1-deprecated/notify.go @@ -4,8 +4,8 @@ import ( "context" "time" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" + inet "github.com/libp2p/go-libp2p-core/network" + peer "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index f68debcfdb..1bb7cc0d79 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -10,13 +10,16 @@ import ( pb "github.com/libp2p/go-libp2p-circuit/pb" - logging "github.com/ipfs/go-log" + "github.com/libp2p/go-libp2p-core/helpers" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + pool "github.com/libp2p/go-buffer-pool" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" tptu "github.com/libp2p/go-libp2p-transport-upgrader" + + logging "github.com/ipfs/go-log" ma "github.com/multiformats/go-multiaddr" ) @@ -138,12 +141,12 @@ func (r *Relay) GetActiveHops() int32 { return atomic.LoadInt32(&r.liveHopCount) } -func (r *Relay) DialPeer(ctx context.Context, relay pstore.PeerInfo, dest pstore.PeerInfo) (*Conn, error) { +func (r *Relay) DialPeer(ctx context.Context, relay peer.AddrInfo, dest peer.AddrInfo) (*Conn, error) { log.Debugf("dialing peer %s through relay %s", dest.ID, relay.ID) if len(relay.Addrs) > 0 { - r.host.Peerstore().AddAddrs(relay.ID, relay.Addrs, pstore.TempAddrTTL) + r.host.Peerstore().AddAddrs(relay.ID, relay.Addrs, peerstore.TempAddrTTL) } s, err := r.host.NewStream(ctx, relay.ID, ProtoID) @@ -219,7 +222,7 @@ func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { s.Reset() return false, err } - if err := inet.FullClose(s); err != nil { + if err := helpers.FullClose(s); err != nil { return false, err } @@ -230,7 +233,7 @@ func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { return msg.GetCode() == pb.CircuitRelay_SUCCESS, nil } -func (r *Relay) handleNewStream(s inet.Stream) { +func (r *Relay) handleNewStream(s network.Stream) { log.Infof("new relay stream from: %s", s.Conn().RemotePeer()) rd := newDelimitedReader(s, maxMessageSize) @@ -257,7 +260,7 @@ func (r *Relay) handleNewStream(s inet.Stream) { } } -func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { +func (r *Relay) handleHopStream(s network.Stream, msg *pb.CircuitRelay) { if !r.hop { r.handleError(s, pb.CircuitRelay_HOP_CANT_SPEAK_RELAY) return @@ -300,15 +303,15 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { defer cancel() if !r.active { - ctx = inet.WithNoDial(ctx, "relay hop") + ctx = network.WithNoDial(ctx, "relay hop") } else if len(dst.Addrs) > 0 { - r.host.Peerstore().AddAddrs(dst.ID, dst.Addrs, pstore.TempAddrTTL) + r.host.Peerstore().AddAddrs(dst.ID, dst.Addrs, peerstore.TempAddrTTL) } bs, err := r.host.NewStream(ctx, dst.ID, ProtoID) if err != nil { log.Debugf("error opening relay stream to %s: %s", dst.ID.Pretty(), err.Error()) - if err == inet.ErrNoConn { + if err == network.ErrNoConn { r.handleError(s, pb.CircuitRelay_HOP_NO_CONN_TO_DST) } else { r.handleError(s, pb.CircuitRelay_HOP_CANT_DIAL_DST) @@ -423,7 +426,7 @@ func (r *Relay) handleHopStream(s inet.Stream, msg *pb.CircuitRelay) { }() } -func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { +func (r *Relay) handleStopStream(s network.Stream, msg *pb.CircuitRelay) { src, err := peerToPeerInfo(msg.GetSrcPeer()) if err != nil { r.handleError(s, pb.CircuitRelay_STOP_SRC_MULTIADDR_INVALID) @@ -439,7 +442,7 @@ func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { log.Infof("relay connection from: %s", src.ID) if len(src.Addrs) > 0 { - r.host.Peerstore().AddAddrs(src.ID, src.Addrs, pstore.TempAddrTTL) + r.host.Peerstore().AddAddrs(src.ID, src.Addrs, peerstore.TempAddrTTL) } select { @@ -449,7 +452,7 @@ func (r *Relay) handleStopStream(s inet.Stream, msg *pb.CircuitRelay) { } } -func (r *Relay) handleCanHop(s inet.Stream, msg *pb.CircuitRelay) { +func (r *Relay) handleCanHop(s network.Stream, msg *pb.CircuitRelay) { var err error if r.hop { @@ -462,22 +465,22 @@ func (r *Relay) handleCanHop(s inet.Stream, msg *pb.CircuitRelay) { s.Reset() log.Debugf("error writing relay response: %s", err.Error()) } else { - inet.FullClose(s) + helpers.FullClose(s) } } -func (r *Relay) handleError(s inet.Stream, code pb.CircuitRelay_Status) { +func (r *Relay) handleError(s network.Stream, code pb.CircuitRelay_Status) { log.Warningf("relay error: %s (%d)", pb.CircuitRelay_Status_name[int32(code)], code) err := r.writeResponse(s, code) if err != nil { s.Reset() log.Debugf("error writing relay response: %s", err.Error()) } else { - inet.FullClose(s) + helpers.FullClose(s) } } -func (r *Relay) writeResponse(s inet.Stream, code pb.CircuitRelay_Status) error { +func (r *Relay) writeResponse(s network.Stream, code pb.CircuitRelay_Status) error { wr := newDelimitedWriter(s) var msg pb.CircuitRelay diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index ff0bcad70d..40a4829754 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -14,7 +14,8 @@ import ( pb "github.com/libp2p/go-libp2p-circuit/pb" bhost "github.com/libp2p/go-libp2p-blankhost" - host "github.com/libp2p/go-libp2p-host" + "github.com/libp2p/go-libp2p-core/host" + swarm "github.com/libp2p/go-libp2p-swarm" swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 86821d327c..57a65779f2 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -4,8 +4,9 @@ import ( "context" "fmt" - host "github.com/libp2p/go-libp2p-host" - tpt "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/transport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" ma "github.com/multiformats/go-multiaddr" ) @@ -23,7 +24,7 @@ func init() { ma.AddProtocol(Protocol) } -var _ tpt.Transport = (*RelayTransport)(nil) +var _ transport.Transport = (*RelayTransport)(nil) type RelayTransport Relay @@ -35,7 +36,7 @@ func (r *Relay) Transport() *RelayTransport { return (*RelayTransport)(r) } -func (t *RelayTransport) Listen(laddr ma.Multiaddr) (tpt.Listener, error) { +func (t *RelayTransport) Listen(laddr ma.Multiaddr) (transport.Listener, error) { // TODO: Ensure we have a connection to the relay, if specified. Also, // make sure the multiaddr makes sense. if !t.Relay().Matches(laddr) { @@ -58,7 +59,7 @@ func (t *RelayTransport) Protocols() []int { // AddRelayTransport constructs a relay and adds it as a transport to the host network. func AddRelayTransport(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts ...RelayOpt) error { - n, ok := h.Network().(tpt.Network) + n, ok := h.Network().(transport.TransportNetwork) if !ok { return fmt.Errorf("%v is not a transport network", h.Network()) } diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go index 23e428783f..960ffaf8ce 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go @@ -10,9 +10,10 @@ import ( . "github.com/libp2p/go-libp2p-circuit" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peerstore" + swarm "github.com/libp2p/go-libp2p-swarm" swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" @@ -45,7 +46,7 @@ func testSetupRelay(t *testing.T, ctx context.Context) []host.Host { time.Sleep(100 * time.Millisecond) - handler := func(s inet.Stream) { + handler := func(s network.Stream) { _, err := s.Write(msg) if err != nil { t.Error(err) @@ -72,7 +73,7 @@ func TestFullAddressTransportDial(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) + hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, peerstore.TempAddrTTL) s, err := hosts[0].NewStream(rctx, hosts[2].ID(), TestProto) if err != nil { @@ -103,7 +104,7 @@ func TestSpecificRelayTransportDial(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) + hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, peerstore.TempAddrTTL) s, err := hosts[0].NewStream(rctx, hosts[2].ID(), TestProto) if err != nil { @@ -134,7 +135,7 @@ func TestUnspecificRelayTransportDial(t *testing.T) { rctx, rcancel := context.WithTimeout(ctx, time.Second) defer rcancel() - hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, pstore.TempAddrTTL) + hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, peerstore.TempAddrTTL) s, err := hosts[0].NewStream(rctx, hosts[2].ID(), TestProto) if err != nil { diff --git a/p2p/protocol/internal/circuitv1-deprecated/util.go b/p2p/protocol/internal/circuitv1-deprecated/util.go index 5eff3f1bd4..ca7a23f328 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/util.go +++ b/p2p/protocol/internal/circuitv1-deprecated/util.go @@ -7,22 +7,22 @@ import ( pb "github.com/libp2p/go-libp2p-circuit/pb" + "github.com/libp2p/go-libp2p-core/peer" + ggio "github.com/gogo/protobuf/io" proto "github.com/gogo/protobuf/proto" pool "github.com/libp2p/go-buffer-pool" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) -func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { +func peerToPeerInfo(p *pb.CircuitRelay_Peer) (peer.AddrInfo, error) { if p == nil { - return pstore.PeerInfo{}, errors.New("nil peer") + return peer.AddrInfo{}, errors.New("nil peer") } id, err := peer.IDFromBytes(p.Id) if err != nil { - return pstore.PeerInfo{}, err + return peer.AddrInfo{}, err } addrs := make([]ma.Multiaddr, 0, len(p.Addrs)) @@ -33,10 +33,10 @@ func peerToPeerInfo(p *pb.CircuitRelay_Peer) (pstore.PeerInfo, error) { } } - return pstore.PeerInfo{ID: id, Addrs: addrs}, nil + return peer.AddrInfo{ID: id, Addrs: addrs}, nil } -func peerInfoToPeer(pi pstore.PeerInfo) *pb.CircuitRelay_Peer { +func peerInfoToPeer(pi peer.AddrInfo) *pb.CircuitRelay_Peer { addrs := make([][]byte, len(pi.Addrs)) for i, addr := range pi.Addrs { addrs[i] = addr.Bytes() From 0df8d18488856c53bace74b4614d3eda96419a53 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 16:41:32 +0100 Subject: [PATCH 1272/3965] migrate to consolidated types. (#23) --- p2p/host/autonat/autonat.go | 24 ++++++++++++------------ p2p/host/autonat/autonat_test.go | 13 +++++++------ p2p/host/autonat/client.go | 14 +++++++------- p2p/host/autonat/notify.go | 17 +++++++++-------- p2p/host/autonat/proto.go | 5 +++-- 5 files changed, 38 insertions(+), 35 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 60188c94da..7e839dddd2 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -7,10 +7,10 @@ import ( "sync" "time" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" ma "github.com/multiformats/go-multiaddr" ) @@ -157,10 +157,10 @@ func (as *AmbientAutoNAT) autodetect() { for _, pi := range peers[:probe] { wg.Add(1) - go func(pi pstore.PeerInfo) { + go func(pi peer.AddrInfo) { defer wg.Done() - as.host.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) + as.host.Peerstore().AddAddrs(pi.ID, pi.Addrs, peerstore.TempAddrTTL) a, err := cli.DialBack(ctx, pi.ID) switch { @@ -217,7 +217,7 @@ func (as *AmbientAutoNAT) autodetect() { as.mx.Unlock() } -func (as *AmbientAutoNAT) getPeers() []pstore.PeerInfo { +func (as *AmbientAutoNAT) getPeers() []peer.AddrInfo { as.mx.Lock() defer as.mx.Unlock() @@ -225,13 +225,13 @@ func (as *AmbientAutoNAT) getPeers() []pstore.PeerInfo { return nil } - var connected, others []pstore.PeerInfo + var connected, others []peer.AddrInfo for p, addrs := range as.peers { - if as.host.Network().Connectedness(p) == inet.Connected { - connected = append(connected, pstore.PeerInfo{ID: p, Addrs: addrs}) + if as.host.Network().Connectedness(p) == network.Connected { + connected = append(connected, peer.AddrInfo{ID: p, Addrs: addrs}) } else { - others = append(others, pstore.PeerInfo{ID: p, Addrs: addrs}) + others = append(others, peer.AddrInfo{ID: p, Addrs: addrs}) } } @@ -245,7 +245,7 @@ func (as *AmbientAutoNAT) getPeers() []pstore.PeerInfo { } } -func shufflePeers(peers []pstore.PeerInfo) { +func shufflePeers(peers []peer.AddrInfo) { for i := range peers { j := rand.Intn(i + 1) peers[i], peers[j] = peers[j], peers[i] diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 9e81453675..e5fc7fff37 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -6,12 +6,13 @@ import ( "time" pb "github.com/libp2p/go-libp2p-autonat/pb" + "github.com/libp2p/go-libp2p-core/peer" + + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" ggio "github.com/gogo/protobuf/io" bhost "github.com/libp2p/go-libp2p-blankhost" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" ) @@ -36,7 +37,7 @@ func makeAutoNATServicePublic(ctx context.Context, t *testing.T) host.Host { return h } -func sayAutoNATPrivate(s inet.Stream) { +func sayAutoNATPrivate(s network.Stream) { defer s.Close() w := ggio.NewDelimitedWriter(s) res := pb.Message{ @@ -46,7 +47,7 @@ func sayAutoNATPrivate(s inet.Stream) { w.WriteMsg(&res) } -func sayAutoNATPublic(s inet.Stream) { +func sayAutoNATPublic(s network.Stream) { defer s.Close() w := ggio.NewDelimitedWriter(s) res := pb.Message{ @@ -80,7 +81,7 @@ func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, A } func connect(t *testing.T, a, b host.Host) { - pinfo := pstore.PeerInfo{ID: a.ID(), Addrs: a.Addrs()} + pinfo := peer.AddrInfo{ID: a.ID(), Addrs: a.Addrs()} err := b.Connect(context.Background(), pinfo) if err != nil { t.Fatal(err) diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index aa3075e4c9..0d8549b48a 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -5,12 +5,12 @@ import ( "fmt" pb "github.com/libp2p/go-libp2p-autonat/pb" + "github.com/libp2p/go-libp2p-core/helpers" ggio "github.com/gogo/protobuf/io" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" ) @@ -51,12 +51,12 @@ func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) } // Might as well just reset the stream. Once we get to this point, we // don't care about being nice. - defer inet.FullClose(s) + defer helpers.FullClose(s) - r := ggio.NewDelimitedReader(s, inet.MessageSizeMax) + r := ggio.NewDelimitedReader(s, network.MessageSizeMax) w := ggio.NewDelimitedWriter(s) - req := newDialMessage(pstore.PeerInfo{ID: c.h.ID(), Addrs: c.getAddrs()}) + req := newDialMessage(peer.AddrInfo{ID: c.h.ID(), Addrs: c.getAddrs()}) err = w.WriteMsg(req) if err != nil { s.Reset() diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 69f5a6457f..4ea6561603 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -3,20 +3,21 @@ package autonat import ( "time" - inet "github.com/libp2p/go-libp2p-net" + "github.com/libp2p/go-libp2p-core/network" + ma "github.com/multiformats/go-multiaddr" ) -var _ inet.Notifiee = (*AmbientAutoNAT)(nil) +var _ network.Notifiee = (*AmbientAutoNAT)(nil) var AutoNATIdentifyDelay = 5 * time.Second -func (as *AmbientAutoNAT) Listen(net inet.Network, a ma.Multiaddr) {} -func (as *AmbientAutoNAT) ListenClose(net inet.Network, a ma.Multiaddr) {} -func (as *AmbientAutoNAT) OpenedStream(net inet.Network, s inet.Stream) {} -func (as *AmbientAutoNAT) ClosedStream(net inet.Network, s inet.Stream) {} +func (as *AmbientAutoNAT) Listen(net network.Network, a ma.Multiaddr) {} +func (as *AmbientAutoNAT) ListenClose(net network.Network, a ma.Multiaddr) {} +func (as *AmbientAutoNAT) OpenedStream(net network.Network, s network.Stream) {} +func (as *AmbientAutoNAT) ClosedStream(net network.Network, s network.Stream) {} -func (as *AmbientAutoNAT) Connected(net inet.Network, c inet.Conn) { +func (as *AmbientAutoNAT) Connected(net network.Network, c network.Conn) { p := c.RemotePeer() go func() { @@ -38,4 +39,4 @@ func (as *AmbientAutoNAT) Connected(net inet.Network, c inet.Conn) { }() } -func (as *AmbientAutoNAT) Disconnected(net inet.Network, c inet.Conn) {} +func (as *AmbientAutoNAT) Disconnected(net network.Network, c network.Conn) {} diff --git a/p2p/host/autonat/proto.go b/p2p/host/autonat/proto.go index 2dbb8ebaa7..61ef986d3b 100644 --- a/p2p/host/autonat/proto.go +++ b/p2p/host/autonat/proto.go @@ -3,15 +3,16 @@ package autonat import ( pb "github.com/libp2p/go-libp2p-autonat/pb" + "github.com/libp2p/go-libp2p-core/peer" + logging "github.com/ipfs/go-log" - pstore "github.com/libp2p/go-libp2p-peerstore" ) const AutoNATProto = "/libp2p/autonat/1.0.0" var log = logging.Logger("autonat") -func newDialMessage(pi pstore.PeerInfo) *pb.Message { +func newDialMessage(pi peer.AddrInfo) *pb.Message { msg := new(pb.Message) msg.Type = pb.Message_DIAL.Enum() msg.Dial = new(pb.Message_Dial) From 5a33a88caf0ba70df2866c484c9bc1b6d6c0a5b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 17:00:40 +0100 Subject: [PATCH 1273/3965] migrate to consolidated types. (#62) --- p2p/transport/quic/conn.go | 15 ++++++++------- p2p/transport/quic/conn_test.go | 18 +++++++++--------- p2p/transport/quic/crypto.go | 4 ++-- p2p/transport/quic/listener.go | 11 ++++++----- p2p/transport/quic/listener_test.go | 5 +++-- p2p/transport/quic/stream.go | 5 +++-- p2p/transport/quic/transport.go | 9 +++++---- p2p/transport/quic/transport_test.go | 2 +- 8 files changed, 37 insertions(+), 32 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index ed837835cb..bf173458a4 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -1,10 +1,11 @@ package libp2pquic import ( - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" - smux "github.com/libp2p/go-stream-muxer" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/peer" + tpt "github.com/libp2p/go-libp2p-core/transport" + quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" ) @@ -22,7 +23,7 @@ type conn struct { remoteMultiaddr ma.Multiaddr } -var _ tpt.Conn = &conn{} +var _ tpt.CapableConn = &conn{} func (c *conn) Close() error { return c.sess.Close() @@ -34,13 +35,13 @@ func (c *conn) IsClosed() bool { } // OpenStream creates a new stream. -func (c *conn) OpenStream() (smux.Stream, error) { +func (c *conn) OpenStream() (mux.MuxedStream, error) { qstr, err := c.sess.OpenStreamSync() return &stream{Stream: qstr}, err } // AcceptStream accepts a stream opened by the other side. -func (c *conn) AcceptStream() (smux.Stream, error) { +func (c *conn) AcceptStream() (mux.MuxedStream, error) { qstr, err := c.sess.AcceptStream() return &stream{Stream: qstr}, err } diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 1e0863e77c..8b86af2073 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -10,9 +10,9 @@ import ( "io/ioutil" "time" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + tpt "github.com/libp2p/go-libp2p-core/transport" ma "github.com/multiformats/go-multiaddr" . "github.com/onsi/ginkgo" @@ -35,9 +35,9 @@ var _ = Describe("Connection", func() { return id, priv } - runServer := func(tr tpt.Transport, multiaddr string) (ma.Multiaddr, <-chan tpt.Conn) { + runServer := func(tr tpt.Transport, multiaddr string) (ma.Multiaddr, <-chan tpt.CapableConn) { addrChan := make(chan ma.Multiaddr) - connChan := make(chan tpt.Conn) + connChan := make(chan tpt.CapableConn) go func() { defer GinkgoRecover() addr, err := ma.NewMultiaddr(multiaddr) @@ -204,8 +204,8 @@ var _ = Describe("Connection", func() { // wait for both servers to accept a connection // then send some data go func() { - for _, c := range []tpt.Conn{<-serverConnChan, <-serverConnChan2} { - go func(conn tpt.Conn) { + for _, c := range []tpt.CapableConn{<-serverConnChan, <-serverConnChan2} { + go func(conn tpt.CapableConn) { defer GinkgoRecover() str, err := conn.OpenStream() Expect(err).ToNot(HaveOccurred()) @@ -225,8 +225,8 @@ var _ = Describe("Connection", func() { done := make(chan struct{}, 2) // receive the data on both connections at the same time - for _, c := range []tpt.Conn{c1, c2} { - go func(conn tpt.Conn) { + for _, c := range []tpt.CapableConn{c1, c2} { + go func(conn tpt.CapableConn) { defer GinkgoRecover() str, err := conn.AcceptStream() Expect(err).ToNot(HaveOccurred()) diff --git a/p2p/transport/quic/crypto.go b/p2p/transport/quic/crypto.go index 1ce18bcb9d..68a30a629e 100644 --- a/p2p/transport/quic/crypto.go +++ b/p2p/transport/quic/crypto.go @@ -11,8 +11,8 @@ import ( "time" "github.com/gogo/protobuf/proto" - ic "github.com/libp2p/go-libp2p-crypto" - pb "github.com/libp2p/go-libp2p-crypto/pb" + ic "github.com/libp2p/go-libp2p-core/crypto" + pb "github.com/libp2p/go-libp2p-core/crypto/pb" ) // mint certificate selection is broken. diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index ef8b019048..327af2f4c1 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -4,9 +4,10 @@ import ( "crypto/tls" "net" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + tpt "github.com/libp2p/go-libp2p-core/transport" + quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" @@ -57,7 +58,7 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, } // Accept accepts new connections. -func (l *listener) Accept() (tpt.Conn, error) { +func (l *listener) Accept() (tpt.CapableConn, error) { for { sess, err := l.quicListener.Accept() if err != nil { @@ -72,7 +73,7 @@ func (l *listener) Accept() (tpt.Conn, error) { } } -func (l *listener) setupConn(sess quic.Session) (tpt.Conn, error) { +func (l *listener) setupConn(sess quic.Session) (tpt.CapableConn, error) { remotePubKey, err := getRemotePubKey(sess.ConnectionState().PeerCertificates) if err != nil { return nil, err diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index dd7974cf45..bf0435f1ac 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -7,8 +7,9 @@ import ( "fmt" "net" - ic "github.com/libp2p/go-libp2p-crypto" - tpt "github.com/libp2p/go-libp2p-transport" + ic "github.com/libp2p/go-libp2p-core/crypto" + tpt "github.com/libp2p/go-libp2p-core/transport" + ma "github.com/multiformats/go-multiaddr" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" diff --git a/p2p/transport/quic/stream.go b/p2p/transport/quic/stream.go index 108536560e..e757de20dd 100644 --- a/p2p/transport/quic/stream.go +++ b/p2p/transport/quic/stream.go @@ -1,7 +1,8 @@ package libp2pquic import ( - smux "github.com/libp2p/go-stream-muxer" + "github.com/libp2p/go-libp2p-core/mux" + quic "github.com/lucas-clemente/quic-go" ) @@ -9,7 +10,7 @@ type stream struct { quic.Stream } -var _ smux.Stream = &stream{} +var _ mux.MuxedStream = &stream{} func (s *stream) Reset() error { s.Stream.CancelRead(0) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index d9f2c8f09c..c33b241766 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -9,9 +9,10 @@ import ( "net" "sync" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + tpt "github.com/libp2p/go-libp2p-core/transport" + quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" @@ -99,7 +100,7 @@ func NewTransport(key ic.PrivKey) (tpt.Transport, error) { } // Dial dials a new QUIC connection -func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { +func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tpt.CapableConn, error) { network, host, err := manet.DialArgs(raddr) if err != nil { return nil, err diff --git a/p2p/transport/quic/transport_test.go b/p2p/transport/quic/transport_test.go index 0decd920a7..8dfece43ba 100644 --- a/p2p/transport/quic/transport_test.go +++ b/p2p/transport/quic/transport_test.go @@ -1,7 +1,7 @@ package libp2pquic import ( - tpt "github.com/libp2p/go-libp2p-transport" + tpt "github.com/libp2p/go-libp2p-core/transport" ma "github.com/multiformats/go-multiaddr" . "github.com/onsi/ginkgo" From 3255249d4ea91a5464ccb1bf358bb6273abbf8cf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 17:34:01 +0100 Subject: [PATCH 1274/3965] migrate to consolidated types. (#45) --- p2p/transport/websocket/websocket.go | 14 ++++++++------ p2p/transport/websocket/websocket_test.go | 7 ++++--- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 681d67640c..972681a648 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -8,10 +8,12 @@ import ( "net/http" "net/url" - ws "github.com/gorilla/websocket" - peer "github.com/libp2p/go-libp2p-peer" - tpt "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + + ws "github.com/gorilla/websocket" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" mafmt "github.com/whyrusleeping/mafmt" @@ -61,7 +63,7 @@ func New(u *tptu.Upgrader) *WebsocketTransport { return &WebsocketTransport{u} } -var _ tpt.Transport = (*WebsocketTransport)(nil) +var _ transport.Transport = (*WebsocketTransport)(nil) func (t *WebsocketTransport) CanDial(a ma.Multiaddr) bool { return WsFmt.Matches(a) @@ -94,7 +96,7 @@ func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (ma return mnc, nil } -func (t *WebsocketTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tpt.Conn, error) { +func (t *WebsocketTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (transport.CapableConn, error) { macon, err := t.maDial(ctx, raddr) if err != nil { return nil, err @@ -130,7 +132,7 @@ func (t *WebsocketTransport) maListen(a ma.Multiaddr) (manet.Listener, error) { return malist, nil } -func (t *WebsocketTransport) Listen(a ma.Multiaddr) (tpt.Listener, error) { +func (t *WebsocketTransport) Listen(a ma.Multiaddr) (transport.Listener, error) { malist, err := t.maListen(a) if err != nil { return nil, err diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index ae55afe907..4f397c8243 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -8,10 +8,11 @@ import ( "testing" "testing/iotest" - insecure "github.com/libp2p/go-conn-security/insecure" + "github.com/libp2p/go-libp2p-core/sec/insecure" + mplex "github.com/libp2p/go-libp2p-mplex" + ttransport "github.com/libp2p/go-libp2p-testing/suites/transport" tptu "github.com/libp2p/go-libp2p-transport-upgrader" - utils "github.com/libp2p/go-libp2p-transport/test" ma "github.com/multiformats/go-multiaddr" ) @@ -50,7 +51,7 @@ func TestWebsocketTransport(t *testing.T) { }) zero := "/ip4/127.0.0.1/tcp/0/ws" - utils.SubtestTransport(t, ta, tb, zero, "peerA") + ttransport.SubtestTransport(t, ta, tb, zero, "peerA") } func TestWebsocketListen(t *testing.T) { From d87f89314c795f87f32b0c900243728f1481ddb7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 22:55:46 +0100 Subject: [PATCH 1275/3965] Consolidate abstractions and core types into go-libp2p-core (#601) --- config/config.go | 28 ++-- config/constructor_types.go | 27 ++-- config/muxer.go | 16 +-- config/muxer_test.go | 19 +-- config/reflection_magic.go | 2 +- config/security.go | 27 ++-- config/transport.go | 5 +- defaults.go | 2 +- go.mod | 49 +++---- go.sum | 151 ++++++++++----------- libp2p.go | 2 +- libp2p_test.go | 8 +- options.go | 17 +-- p2p/discovery/mdns.go | 10 +- p2p/discovery/mdns_test.go | 11 +- p2p/host/basic/basic_host.go | 74 +++++----- p2p/host/basic/basic_host_test.go | 34 ++--- p2p/host/basic/natmgr.go | 20 +-- p2p/host/relay/autorelay.go | 36 ++--- p2p/host/relay/autorelay_test.go | 35 +++-- p2p/host/relay/relay.go | 3 +- p2p/host/routed/routed.go | 42 +++--- p2p/net/mock/interface.go | 40 +++--- p2p/net/mock/mock_conn.go | 28 ++-- p2p/net/mock/mock_link.go | 18 +-- p2p/net/mock/mock_net.go | 38 +++--- p2p/net/mock/mock_notif_test.go | 55 ++++---- p2p/net/mock/mock_peernet.go | 66 ++++----- p2p/net/mock/mock_printer.go | 6 +- p2p/net/mock/mock_stream.go | 18 +-- p2p/net/mock/mock_test.go | 45 +++--- p2p/protocol/identify/id.go | 70 +++++----- p2p/protocol/identify/id_test.go | 19 +-- p2p/protocol/identify/obsaddr.go | 15 +- p2p/protocol/identify/obsaddr_test.go | 2 +- p2p/protocol/ping/ping.go | 10 +- p2p/protocol/ping/ping_test.go | 6 +- p2p/test/backpressure/backpressure_test.go | 16 +-- p2p/test/reconnects/reconnect_test.go | 12 +- 39 files changed, 544 insertions(+), 538 deletions(-) diff --git a/config/config.go b/config/config.go index 79b04420b3..3bb51bf1f7 100644 --- a/config/config.go +++ b/config/config.go @@ -4,24 +4,26 @@ import ( "context" "fmt" + "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/metrics" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/pnet" + "github.com/libp2p/go-libp2p-core/routing" + bhost "github.com/libp2p/go-libp2p/p2p/host/basic" relay "github.com/libp2p/go-libp2p/p2p/host/relay" routed "github.com/libp2p/go-libp2p/p2p/host/routed" - logging "github.com/ipfs/go-log" circuit "github.com/libp2p/go-libp2p-circuit" - crypto "github.com/libp2p/go-libp2p-crypto" discovery "github.com/libp2p/go-libp2p-discovery" - host "github.com/libp2p/go-libp2p-host" - ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" - pnet "github.com/libp2p/go-libp2p-interface-pnet" - metrics "github.com/libp2p/go-libp2p-metrics" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - routing "github.com/libp2p/go-libp2p-routing" swarm "github.com/libp2p/go-libp2p-swarm" tptu "github.com/libp2p/go-libp2p-transport-upgrader" + + logging "github.com/ipfs/go-log" filter "github.com/libp2p/go-maddr-filter" ma "github.com/multiformats/go-multiaddr" ) @@ -33,7 +35,7 @@ var log = logging.Logger("p2p-config") type AddrsFactory = bhost.AddrsFactory // NATManagerC is a NATManager constructor. -type NATManagerC func(inet.Network) bhost.NATManager +type NATManagerC func(network.Network) bhost.NATManager type RoutingC func(host.Host) (routing.PeerRouting, error) @@ -58,9 +60,9 @@ type Config struct { AddrsFactory bhost.AddrsFactory Filters *filter.Filters - ConnManager ifconnmgr.ConnManager + ConnManager connmgr.ConnManager NATManager NATManagerC - Peerstore pstore.Peerstore + Peerstore peerstore.Peerstore Reporter metrics.Reporter DisablePing bool diff --git a/config/constructor_types.go b/config/constructor_types.go index 0cf4163fc1..6f58c266c7 100644 --- a/config/constructor_types.go +++ b/config/constructor_types.go @@ -4,30 +4,31 @@ import ( "fmt" "reflect" - security "github.com/libp2p/go-conn-security" - crypto "github.com/libp2p/go-libp2p-crypto" - host "github.com/libp2p/go-libp2p-host" - pnet "github.com/libp2p/go-libp2p-interface-pnet" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - transport "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/pnet" + "github.com/libp2p/go-libp2p-core/sec" + "github.com/libp2p/go-libp2p-core/transport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" filter "github.com/libp2p/go-maddr-filter" - mux "github.com/libp2p/go-stream-muxer" ) var ( // interfaces hostType = reflect.TypeOf((*host.Host)(nil)).Elem() - networkType = reflect.TypeOf((*inet.Network)(nil)).Elem() + networkType = reflect.TypeOf((*network.Network)(nil)).Elem() transportType = reflect.TypeOf((*transport.Transport)(nil)).Elem() - muxType = reflect.TypeOf((*mux.Transport)(nil)).Elem() - securityType = reflect.TypeOf((*security.Transport)(nil)).Elem() + muxType = reflect.TypeOf((*mux.Multiplexer)(nil)).Elem() + securityType = reflect.TypeOf((*sec.SecureTransport)(nil)).Elem() protectorType = reflect.TypeOf((*pnet.Protector)(nil)).Elem() privKeyType = reflect.TypeOf((*crypto.PrivKey)(nil)).Elem() pubKeyType = reflect.TypeOf((*crypto.PubKey)(nil)).Elem() - pstoreType = reflect.TypeOf((*pstore.Peerstore)(nil)).Elem() + pstoreType = reflect.TypeOf((*peerstore.Peerstore)(nil)).Elem() // concrete types peerIDType = reflect.TypeOf((peer.ID)("")) diff --git a/config/muxer.go b/config/muxer.go index 88a83910ad..6f99336c51 100644 --- a/config/muxer.go +++ b/config/muxer.go @@ -3,13 +3,13 @@ package config import ( "fmt" - host "github.com/libp2p/go-libp2p-host" - mux "github.com/libp2p/go-stream-muxer" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/mux" msmux "github.com/libp2p/go-stream-muxer-multistream" ) // MuxC is a stream multiplex transport constructor -type MuxC func(h host.Host) (mux.Transport, error) +type MuxC func(h host.Host) (mux.Multiplexer, error) // MsMuxC is a tuple containing a multiplex transport constructor and a protocol // ID. @@ -24,8 +24,8 @@ var muxArgTypes = newArgTypeSet(hostType, networkType, peerIDType, pstoreType) // using reflection. func MuxerConstructor(m interface{}) (MuxC, error) { // Already constructed? - if t, ok := m.(mux.Transport); ok { - return func(_ host.Host) (mux.Transport, error) { + if t, ok := m.(mux.Multiplexer); ok { + return func(_ host.Host) (mux.Multiplexer, error) { return t, nil }, nil } @@ -34,16 +34,16 @@ func MuxerConstructor(m interface{}) (MuxC, error) { if err != nil { return nil, err } - return func(h host.Host) (mux.Transport, error) { + return func(h host.Host) (mux.Multiplexer, error) { t, err := ctor(h, nil) if err != nil { return nil, err } - return t.(mux.Transport), nil + return t.(mux.Multiplexer), nil }, nil } -func makeMuxer(h host.Host, tpts []MsMuxC) (mux.Transport, error) { +func makeMuxer(h host.Host, tpts []MsMuxC) (mux.Multiplexer, error) { muxMuxer := msmux.NewBlankTransport() transportSet := make(map[string]struct{}, len(tpts)) for _, tptC := range tpts { diff --git a/config/muxer_test.go b/config/muxer_test.go index 490c95e62b..6fe5aa2c42 100644 --- a/config/muxer_test.go +++ b/config/muxer_test.go @@ -4,17 +4,18 @@ import ( "context" "testing" - host "github.com/libp2p/go-libp2p-host" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" swarmt "github.com/libp2p/go-libp2p-swarm/testing" - yamux "github.com/libp2p/go-libp2p-yamux" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" - mux "github.com/libp2p/go-stream-muxer" + + mux "github.com/libp2p/go-libp2p-core/mux" + yamux "github.com/libp2p/go-libp2p-yamux" ) func TestMuxerSimple(t *testing.T) { // single - _, err := MuxerConstructor(func(_ peer.ID) mux.Transport { return nil }) + _, err := MuxerConstructor(func(_ peer.ID) mux.Multiplexer { return nil }) if err != nil { t.Fatal(err) } @@ -27,14 +28,14 @@ func TestMuxerByValue(t *testing.T) { } } func TestMuxerDuplicate(t *testing.T) { - _, err := MuxerConstructor(func(_ peer.ID, _ peer.ID) mux.Transport { return nil }) + _, err := MuxerConstructor(func(_ peer.ID, _ peer.ID) mux.Multiplexer { return nil }) if err != nil { t.Fatal(err) } } func TestMuxerError(t *testing.T) { - _, err := MuxerConstructor(func() (mux.Transport, error) { return nil, nil }) + _, err := MuxerConstructor(func() (mux.Multiplexer, error) { return nil, nil }) if err != nil { t.Fatal(err) } @@ -45,8 +46,8 @@ func TestMuxerBadTypes(t *testing.T) { func() error { return nil }, func() string { return "" }, func() {}, - func(string) mux.Transport { return nil }, - func(string) (mux.Transport, error) { return nil, nil }, + func(string) mux.Multiplexer { return nil }, + func(string) (mux.Multiplexer, error) { return nil, nil }, nil, "testing", } { diff --git a/config/reflection_magic.go b/config/reflection_magic.go index d66a26e27c..b422cf71ad 100644 --- a/config/reflection_magic.go +++ b/config/reflection_magic.go @@ -5,7 +5,7 @@ import ( "reflect" "runtime" - host "github.com/libp2p/go-libp2p-host" + "github.com/libp2p/go-libp2p-core/host" tptu "github.com/libp2p/go-libp2p-transport-upgrader" ) diff --git a/config/security.go b/config/security.go index 2798fda85f..8e1c710cbf 100644 --- a/config/security.go +++ b/config/security.go @@ -3,15 +3,16 @@ package config import ( "fmt" - security "github.com/libp2p/go-conn-security" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" + "github.com/libp2p/go-libp2p-core/sec/insecure" + csms "github.com/libp2p/go-conn-security-multistream" - insecure "github.com/libp2p/go-conn-security/insecure" - host "github.com/libp2p/go-libp2p-host" - peer "github.com/libp2p/go-libp2p-peer" ) // SecC is a security transport constructor -type SecC func(h host.Host) (security.Transport, error) +type SecC func(h host.Host) (sec.SecureTransport, error) // MsSecC is a tuple containing a security transport constructor and a protocol // ID. @@ -27,34 +28,34 @@ var securityArgTypes = newArgTypeSet( // SecurityConstructor creates a security constructor from the passed parameter // using reflection. -func SecurityConstructor(sec interface{}) (SecC, error) { +func SecurityConstructor(security interface{}) (SecC, error) { // Already constructed? - if t, ok := sec.(security.Transport); ok { - return func(_ host.Host) (security.Transport, error) { + if t, ok := security.(sec.SecureTransport); ok { + return func(_ host.Host) (sec.SecureTransport, error) { return t, nil }, nil } - ctor, err := makeConstructor(sec, securityType, securityArgTypes) + ctor, err := makeConstructor(security, securityType, securityArgTypes) if err != nil { return nil, err } - return func(h host.Host) (security.Transport, error) { + return func(h host.Host) (sec.SecureTransport, error) { t, err := ctor(h, nil) if err != nil { return nil, err } - return t.(security.Transport), nil + return t.(sec.SecureTransport), nil }, nil } -func makeInsecureTransport(id peer.ID) security.Transport { +func makeInsecureTransport(id peer.ID) sec.SecureTransport { secMuxer := new(csms.SSMuxer) secMuxer.AddTransport(insecure.ID, insecure.New(id)) return secMuxer } -func makeSecurityTransport(h host.Host, tpts []MsSecC) (security.Transport, error) { +func makeSecurityTransport(h host.Host, tpts []MsSecC) (sec.SecureTransport, error) { secMuxer := new(csms.SSMuxer) transportSet := make(map[string]struct{}, len(tpts)) for _, tptC := range tpts { diff --git a/config/transport.go b/config/transport.go index d9b4414d38..ac8f8ee7fc 100644 --- a/config/transport.go +++ b/config/transport.go @@ -1,8 +1,9 @@ package config import ( - host "github.com/libp2p/go-libp2p-host" - transport "github.com/libp2p/go-libp2p-transport" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/transport" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" ) diff --git a/defaults.go b/defaults.go index 6cd5df42e3..18de400d95 100644 --- a/defaults.go +++ b/defaults.go @@ -5,7 +5,7 @@ package libp2p import ( "crypto/rand" - crypto "github.com/libp2p/go-libp2p-crypto" + crypto "github.com/libp2p/go-libp2p-core/crypto" mplex "github.com/libp2p/go-libp2p-mplex" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" secio "github.com/libp2p/go-libp2p-secio" diff --git a/go.mod b/go.mod index 90f63ec2d7..91fe7d2178 100644 --- a/go.mod +++ b/go.mod @@ -8,41 +8,30 @@ require ( github.com/ipfs/go-log v0.0.1 github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.3 - github.com/libp2p/go-conn-security v0.0.1 - github.com/libp2p/go-conn-security-multistream v0.0.2 - github.com/libp2p/go-libp2p-autonat v0.0.6 - github.com/libp2p/go-libp2p-blankhost v0.0.1 - github.com/libp2p/go-libp2p-circuit v0.0.9 - github.com/libp2p/go-libp2p-crypto v0.0.2 - github.com/libp2p/go-libp2p-discovery v0.0.5 - github.com/libp2p/go-libp2p-host v0.0.3 - github.com/libp2p/go-libp2p-interface-connmgr v0.0.5 - github.com/libp2p/go-libp2p-interface-pnet v0.0.1 - github.com/libp2p/go-libp2p-loggables v0.0.1 - github.com/libp2p/go-libp2p-metrics v0.0.1 - github.com/libp2p/go-libp2p-mplex v0.1.1 + github.com/libp2p/go-conn-security-multistream v0.1.0 + github.com/libp2p/go-libp2p-autonat v0.1.0 + github.com/libp2p/go-libp2p-blankhost v0.1.1 + github.com/libp2p/go-libp2p-circuit v0.1.0 + github.com/libp2p/go-libp2p-core v0.0.1 + github.com/libp2p/go-libp2p-discovery v0.1.0 + github.com/libp2p/go-libp2p-loggables v0.1.0 + github.com/libp2p/go-libp2p-mplex v0.2.1 github.com/libp2p/go-libp2p-nat v0.0.4 - github.com/libp2p/go-libp2p-net v0.0.2 - github.com/libp2p/go-libp2p-netutil v0.0.1 - github.com/libp2p/go-libp2p-peer v0.1.1 - github.com/libp2p/go-libp2p-peerstore v0.0.6 - github.com/libp2p/go-libp2p-protocol v0.0.1 - github.com/libp2p/go-libp2p-routing v0.0.1 - github.com/libp2p/go-libp2p-secio v0.0.3 - github.com/libp2p/go-libp2p-swarm v0.0.6 - github.com/libp2p/go-libp2p-transport v0.0.5 - github.com/libp2p/go-libp2p-transport-upgrader v0.0.4 - github.com/libp2p/go-libp2p-yamux v0.1.2 + github.com/libp2p/go-libp2p-netutil v0.1.0 + github.com/libp2p/go-libp2p-peerstore v0.1.0 + github.com/libp2p/go-libp2p-secio v0.1.0 + github.com/libp2p/go-libp2p-swarm v0.1.0 + github.com/libp2p/go-libp2p-testing v0.0.3 + github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 + github.com/libp2p/go-libp2p-yamux v0.2.0 github.com/libp2p/go-maddr-filter v0.0.4 - github.com/libp2p/go-stream-muxer v0.0.1 - github.com/libp2p/go-stream-muxer-multistream v0.1.1 - github.com/libp2p/go-tcp-transport v0.0.4 - github.com/libp2p/go-testutil v0.0.1 - github.com/libp2p/go-ws-transport v0.0.5 + github.com/libp2p/go-stream-muxer-multistream v0.2.0 + github.com/libp2p/go-tcp-transport v0.1.0 + github.com/libp2p/go-ws-transport v0.1.0 github.com/miekg/dns v1.1.12 // indirect github.com/multiformats/go-multiaddr v0.0.4 github.com/multiformats/go-multiaddr-dns v0.0.2 github.com/multiformats/go-multiaddr-net v0.0.1 - github.com/multiformats/go-multistream v0.0.4 + github.com/multiformats/go-multistream v0.1.0 github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 ) diff --git a/go.sum b/go.sum index 51efb3f167..0f6dbadbb6 100644 --- a/go.sum +++ b/go.sum @@ -10,9 +10,8 @@ github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVa github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc= github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY= github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= -github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= -github.com/coreos/go-semver v0.2.1-0.20180108230905-e214231b295a h1:U0BbGfKnviqVBJQB4etvm+mKx53KfkumNLBt6YeF/0Q= -github.com/coreos/go-semver v0.2.1-0.20180108230905-e214231b295a/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/coreos/go-semver v0.3.0 h1:wkHLiw0WNATZnSG7epLsujiMCgPAc9xhjJ4tgnAxmfM= +github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -32,7 +31,9 @@ github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q= github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= +github.com/gxed/hashland/keccakpg v0.0.1 h1:wrk3uMNaMxbXiHibbPO4S0ymqJMm41WiudyFSs7UnsU= github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= +github.com/gxed/hashland/murmur3 v0.0.1 h1:SheiaIt0sda5K+8FLz952/1iWS9zrnKsEJaOJu4ZbSc= github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= @@ -82,71 +83,51 @@ github.com/libp2p/go-addr-util v0.0.1/go.mod h1:4ac6O7n9rIAKB1dnd+s8IbbMXkt+oBpz github.com/libp2p/go-buffer-pool v0.0.1/go.mod h1:xtyIz9PMobb13WaxR6Zo1Pd1zXJKYg0a8KiIvDp3TzQ= github.com/libp2p/go-buffer-pool v0.0.2 h1:QNK2iAFa8gjAe1SPz6mHSMuCcjs+X1wlHzeOSqcmlfs= github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM= -github.com/libp2p/go-conn-security v0.0.1 h1:4kMMrqrt9EUNCNjX1xagSJC+bq16uqjMe9lk1KBMVNs= -github.com/libp2p/go-conn-security v0.0.1/go.mod h1:bGmu51N0KU9IEjX7kl2PQjgZa40JQWnayTvNMgD/vyk= -github.com/libp2p/go-conn-security-multistream v0.0.2 h1:Ykz0lnNjxk+0SdslUmlLNyrleqdpS1S/VW+dxFdt74Y= -github.com/libp2p/go-conn-security-multistream v0.0.2/go.mod h1:nc9vud7inQ+d6SO0I/6dSWrdMnHnzZNHeyUQqrAJulE= +github.com/libp2p/go-conn-security-multistream v0.1.0 h1:aqGmto+ttL/uJgX0JtQI0tD21CIEy5eYd1Hlp0juHY0= +github.com/libp2p/go-conn-security-multistream v0.1.0/go.mod h1:aw6eD7LOsHEX7+2hJkDxw1MteijaVcI+/eP2/x3J1xc= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= -github.com/libp2p/go-libp2p-autonat v0.0.6 h1:OCStANLLpeyQeWFUuqZJ7aS9+Bx0/uoVb1PtLA9fGTQ= -github.com/libp2p/go-libp2p-autonat v0.0.6/go.mod h1:uZneLdOkZHro35xIhpbtTzLlgYturpu4J5+0cZK3MqE= -github.com/libp2p/go-libp2p-blankhost v0.0.1 h1:/mZuuiwntNR8RywnCFlGHLKrKLYne+qciBpQXWqp5fk= -github.com/libp2p/go-libp2p-blankhost v0.0.1/go.mod h1:Ibpbw/7cPPYwFb7PACIWdvxxv0t0XCCI10t7czjAjTc= -github.com/libp2p/go-libp2p-circuit v0.0.9 h1:tjdgP9hv8+Pa/xsprBpEFngq4t8aLvjfibBYoDjO9i4= -github.com/libp2p/go-libp2p-circuit v0.0.9/go.mod h1:uU+IBvEQzCu953/ps7bYzC/D/R0Ho2A9LfKVVCatlqU= -github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= -github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= -github.com/libp2p/go-libp2p-crypto v0.0.2/go.mod h1:eETI5OUfBnvARGOHrJz2eWNyTUxEGZnBxMcbUjfIj4I= -github.com/libp2p/go-libp2p-discovery v0.0.5 h1:VpPd7u2odnrrRcW+gVdjLDcXsc35k0Tjxqgbzlre6Uo= -github.com/libp2p/go-libp2p-discovery v0.0.5/go.mod h1:YtF20GUxjgoKZ4zmXj8j3Nb2TUSBHFlOCetzYdbZL5I= -github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= -github.com/libp2p/go-libp2p-host v0.0.3 h1:BB/1Z+4X0rjKP5lbQTmjEjLbDVbrcmLOlA6QDsN5/j4= -github.com/libp2p/go-libp2p-host v0.0.3/go.mod h1:Y/qPyA6C8j2coYyos1dfRm0I8+nvd4TGrDGt4tA7JR8= -github.com/libp2p/go-libp2p-interface-connmgr v0.0.1/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= -github.com/libp2p/go-libp2p-interface-connmgr v0.0.4/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= -github.com/libp2p/go-libp2p-interface-connmgr v0.0.5 h1:KG/KNYL2tYzXAfMvQN5K1aAGTYSYUMJ1prgYa2/JI1E= -github.com/libp2p/go-libp2p-interface-connmgr v0.0.5/go.mod h1:GarlRLH0LdeWcLnYM/SaBykKFl9U5JFnbBGruAk/D5k= -github.com/libp2p/go-libp2p-interface-pnet v0.0.1 h1:7GnzRrBTJHEsofi1ahFdPN9Si6skwXQE9UqR2S+Pkh8= -github.com/libp2p/go-libp2p-interface-pnet v0.0.1/go.mod h1:el9jHpQAXK5dnTpKA4yfCNBZXvrzdOU75zz+C6ryp3k= -github.com/libp2p/go-libp2p-loggables v0.0.1 h1:HVww9oAnINIxbt69LJNkxD8lnbfgteXR97Xm4p3l9ps= -github.com/libp2p/go-libp2p-loggables v0.0.1/go.mod h1:lDipDlBNYbpyqyPX/KcoO+eq0sJYEVR2JgOexcivchg= -github.com/libp2p/go-libp2p-metrics v0.0.1 h1:yumdPC/P2VzINdmcKZd0pciSUCpou+s0lwYCjBbzQZU= -github.com/libp2p/go-libp2p-metrics v0.0.1/go.mod h1:jQJ95SXXA/K1VZi13h52WZMa9ja78zjyy5rspMsC/08= -github.com/libp2p/go-libp2p-mplex v0.1.1 h1:lSPS1VJ36P01gGO//KgcsmSah5uoC3X9r7WY5j+iP4c= -github.com/libp2p/go-libp2p-mplex v0.1.1/go.mod h1:KUQWpGkCzfV7UIpi8SKsAVxyBgz1c9R5EvxgnwLsb/I= +github.com/libp2p/go-libp2p-autonat v0.1.0 h1:aCWAu43Ri4nU0ZPO7NyLzUvvfqd0nE3dX0R/ZGYVgOU= +github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3PtvYlJmg7NeVtPRqH8= +github.com/libp2p/go-libp2p-blankhost v0.1.1 h1:X919sCh+KLqJcNRApj43xCSiQRYqOSI88Fdf55ngf78= +github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= +github.com/libp2p/go-libp2p-circuit v0.1.0 h1:eniLL3Y9aq/sryfyV1IAHj5rlvuyj3b7iz8tSiZpdhY= +github.com/libp2p/go-libp2p-circuit v0.1.0/go.mod h1:Ahq4cY3V9VJcHcn1SBXjr78AbFkZeIRmfunbA7pmFh8= +github.com/libp2p/go-libp2p-core v0.0.1 h1:HSTZtFIq/W5Ue43Zw+uWZyy2Vl5WtF0zDjKN8/DT/1I= +github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= +github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= +github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= +github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= +github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= +github.com/libp2p/go-libp2p-loggables v0.1.0 h1:h3w8QFfCt2UJl/0/NW4K829HX/0S4KD31PQ7m8UXXO8= +github.com/libp2p/go-libp2p-loggables v0.1.0/go.mod h1:EyumB2Y6PrYjr55Q3/tiJ/o3xoDasoRYM7nOzEpoa90= +github.com/libp2p/go-libp2p-mplex v0.2.0/go.mod h1:Ejl9IyjvXJ0T9iqUTE1jpYATQ9NM3g+OtR+EMMODbKo= +github.com/libp2p/go-libp2p-mplex v0.2.1 h1:E1xaJBQnbSiTHGI1gaBKmKhu1TUKkErKJnE8iGvirYI= +github.com/libp2p/go-libp2p-mplex v0.2.1/go.mod h1:SC99Rxs8Vuzrf/6WhmH41kNn13TiYdAWNYHrwImKLnE= github.com/libp2p/go-libp2p-nat v0.0.4 h1:+KXK324yaY701On8a0aGjTnw8467kW3ExKcqW2wwmyw= github.com/libp2p/go-libp2p-nat v0.0.4/go.mod h1:N9Js/zVtAXqaeT99cXgTV9e75KpnWCvVOiGzlcHmBbY= -github.com/libp2p/go-libp2p-net v0.0.1/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= -github.com/libp2p/go-libp2p-net v0.0.2 h1:qP06u4TYXfl7uW/hzqPhlVVTSA2nw1B/bHBJaUnbh6M= -github.com/libp2p/go-libp2p-net v0.0.2/go.mod h1:Yt3zgmlsHOgUWSXmt5V/Jpz9upuJBE8EgNU9DrCcR8c= -github.com/libp2p/go-libp2p-netutil v0.0.1 h1:LgD6+skofkOx8z6odD9+MZHKjupv3ng1u6KRhaADTnA= -github.com/libp2p/go-libp2p-netutil v0.0.1/go.mod h1:GdusFvujWZI9Vt0X5BKqwWWmZFxecf9Gt03cKxm2f/Q= -github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= -github.com/libp2p/go-libp2p-peer v0.1.1 h1:qGCWD1a+PyZcna6htMPo26jAtqirVnJ5NvBQIKV7rRY= -github.com/libp2p/go-libp2p-peer v0.1.1/go.mod h1:jkF12jGB4Gk/IOo+yomm+7oLWxF278F7UnrYUQ1Q8es= -github.com/libp2p/go-libp2p-peerstore v0.0.1/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= -github.com/libp2p/go-libp2p-peerstore v0.0.6 h1:RgX/djPFXqZGktW0j2eF4NAX0pzDsCot45jO2GewC+g= -github.com/libp2p/go-libp2p-peerstore v0.0.6/go.mod h1:RabLyPVJLuNQ+GFyoEkfi8H4Ti6k/HtZJ7YKgtSq+20= -github.com/libp2p/go-libp2p-protocol v0.0.1 h1:+zkEmZ2yFDi5adpVE3t9dqh/N9TbpFWywowzeEzBbLM= -github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= -github.com/libp2p/go-libp2p-routing v0.0.1 h1:hPMAWktf9rYi3ME4MG48qE7dq1ofJxiQbfdvpNntjhc= -github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= -github.com/libp2p/go-libp2p-secio v0.0.3 h1:h3fPeDrej7bvvARnC2oSjAfcLZOaS4REZKgWCRQNpE4= -github.com/libp2p/go-libp2p-secio v0.0.3/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= -github.com/libp2p/go-libp2p-swarm v0.0.6 h1:gE0P/v2h+KEXtAi9YTw2UBOSODJ4m9VuuJ+ktc2LVUo= -github.com/libp2p/go-libp2p-swarm v0.0.6/go.mod h1:s5GZvzg9xXe8sbeESuFpjt8CJPTCa8mhEusweJqyFy8= -github.com/libp2p/go-libp2p-transport v0.0.1/go.mod h1:UzbUs9X+PHOSw7S3ZmeOxfnwaQY5vGDzZmKPod3N3tk= -github.com/libp2p/go-libp2p-transport v0.0.5 h1:pV6+UlRxyDpASSGD+60vMvdifSCby6JkJDfi+yUMHac= -github.com/libp2p/go-libp2p-transport v0.0.5/go.mod h1:StoY3sx6IqsP6XKoabsPnHCwqKXWUMWU7Rfcsubee/A= -github.com/libp2p/go-libp2p-transport-upgrader v0.0.4 h1:uGMOd14BL1oFlfb/cGfOxPjiTKBhzWV4aMjjoCF1Z1o= -github.com/libp2p/go-libp2p-transport-upgrader v0.0.4/go.mod h1:RGq+tupk+oj7PzL2kn/m1w6YXxcIAYJYeI90h6BGgUc= -github.com/libp2p/go-libp2p-yamux v0.1.2 h1:DgGItlrWi0j9y1OhRMC8qqL4zj2MEPWeKJTHb55R16Q= -github.com/libp2p/go-libp2p-yamux v0.1.2/go.mod h1:xUoV/RmYkg6BW/qGxA9XJyg+HzXFYkeXbnhjmnYzKp8= -github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= +github.com/libp2p/go-libp2p-netutil v0.1.0 h1:zscYDNVEcGxyUpMd0JReUZTrpMfia8PmLKcKF72EAMQ= +github.com/libp2p/go-libp2p-netutil v0.1.0/go.mod h1:3Qv/aDqtMLTUyQeundkKsA+YCThNdbQD54k3TqjpbFU= +github.com/libp2p/go-libp2p-peer v0.2.0 h1:EQ8kMjaCUwt/Y5uLgjT8iY2qg0mGUT0N1zUjer50DsY= +github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMgxjsHm7+J5kjWY= +github.com/libp2p/go-libp2p-peerstore v0.1.0 h1:MKh7pRNPHSh1fLPj8u/M/s/napdmeNpoi9BRy9lPN0E= +github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HBb6u0MOvdJ7YIsmcwtY= +github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLMWdB7GTQ= +github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= +github.com/libp2p/go-libp2p-swarm v0.1.0 h1:HrFk2p0awrGEgch9JXK/qp/hfjqQfgNxpLWnCiWPg5s= +github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= +github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= +github.com/libp2p/go-libp2p-testing v0.0.3 h1:bdij4bKaaND7tCsaXVjRfYkMpvoOeKj9AVQGJllA6jM= +github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= +github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 h1:PZMS9lhjK9VytzMCW3tWHAXtKXmlURSc3ZdvwEcKCzw= +github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07qPW4r01GnTenLW4oi3lOPbUMGJJA= +github.com/libp2p/go-libp2p-yamux v0.2.0 h1:TSPZ5cMMz/wdoYsye/wU1TE4G3LDGMoeEN0xgnCKU/I= +github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= github.com/libp2p/go-maddr-filter v0.0.4 h1:hx8HIuuwk34KePddrp2mM5ivgPkZ09JH4AvsALRbFUs= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= -github.com/libp2p/go-mplex v0.0.3 h1:YiMaevQcZtFU6DmKIF8xEO0vaui5kM5HJ1V1xkWQv14= github.com/libp2p/go-mplex v0.0.3/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= +github.com/libp2p/go-mplex v0.1.0 h1:/nBTy5+1yRyY82YaO6HXQRnO5IAGsXTjEJaR3LdTPc0= +github.com/libp2p/go-mplex v0.1.0/go.mod h1:SXgmdki2kwCUlCCbfGLEgHjC4pFqhTp0ZoV6aiKgxDU= github.com/libp2p/go-msgio v0.0.2 h1:ivPvEKHxmVkTClHzg6RXTYHqaJQ0V9cDbq+6lKb3UV0= github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-nat v0.0.3 h1:l6fKV+p0Xa354EqQOQP+d8CivdLM4kl5GxC1hSc/UeI= @@ -155,18 +136,15 @@ github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FW github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0GIEjMiHTwR/KN4= github.com/libp2p/go-reuseport-transport v0.0.2/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= -github.com/libp2p/go-stream-muxer v0.0.1 h1:Ce6e2Pyu+b5MC1k3eeFtAax0pW4gc6MosYSLV05UeLw= github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= -github.com/libp2p/go-stream-muxer-multistream v0.1.1 h1:DhHqb4nu1fQv/vQKeLAaZGmhLsUA4SF77IdYJiWE1d4= -github.com/libp2p/go-stream-muxer-multistream v0.1.1/go.mod h1:zmGdfkQ1AzOECIAcccoL8L//laqawOsO03zX8Sa+eGw= -github.com/libp2p/go-tcp-transport v0.0.4 h1:2iRu994wCT/iEz62F+c60FUoSkijNEQ0q2Itc+79XlQ= -github.com/libp2p/go-tcp-transport v0.0.4/go.mod h1:+E8HvC8ezEVOxIo3V5vCK9l1y/19K427vCzQ+xHKH/o= -github.com/libp2p/go-testutil v0.0.1 h1:Xg+O0G2HIMfHqBOBDcMS1iSZJ3GEcId4qOxCQvsGZHk= -github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= -github.com/libp2p/go-ws-transport v0.0.5 h1:IHeR0X9nvE5hOdOD8X/FDQ6jIapdohToQseItwvpixU= -github.com/libp2p/go-ws-transport v0.0.5/go.mod h1:Qbl4BxPfXXhhd/o0wcrgoaItHqA9tnZjoFZnxykuaXU= -github.com/libp2p/go-yamux v1.2.1 h1:VumHkMhJ2iFk1lzAeoDRgekiZSylGc6NnAEihVdBCiw= -github.com/libp2p/go-yamux v1.2.1/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-stream-muxer-multistream v0.2.0 h1:714bRJ4Zy9mdhyTLJ+ZKiROmAFwUHpeRidG+q7LTQOg= +github.com/libp2p/go-stream-muxer-multistream v0.2.0/go.mod h1:j9eyPol/LLRqT+GPLSxvimPhNph4sfYfMoDPd7HkzIc= +github.com/libp2p/go-tcp-transport v0.1.0 h1:IGhowvEqyMFknOar4FWCKSWE0zL36UFKQtiRQD60/8o= +github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= +github.com/libp2p/go-ws-transport v0.1.0 h1:F+0OvvdmPTDsVc4AjPHjV7L7Pk1B7D5QwtDcKE2oag4= +github.com/libp2p/go-ws-transport v0.1.0/go.mod h1:rjw1MG1LU9YDC6gzmwObkPd/Sqwhw7yT74kj3raBFuo= +github.com/libp2p/go-yamux v1.2.2 h1:s6J6o7+ajoQMjHe7BEnq+EynOj5D2EoG8CuQgL3F2vg= +github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= @@ -175,35 +153,45 @@ github.com/miekg/dns v1.1.12 h1:WMhc1ik4LNkTg8U9l3hI1LvxKmIL+f1+WV/SZtCbDDA= github.com/miekg/dns v1.1.12/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ= +github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16 h1:5W7KhL8HVF3XCFOweFD3BNESdnO8ewyYTFT2R+/b8FQ= github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= -github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5 h1:l16XLUUJ34wIz+RIvLhSwGvLvKyy+W598b135bJN6mg= github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= +github.com/minio/sha256-simd v0.1.0 h1:U41/2erhAKcmSI14xh/ZTUdBPOzDOIfS93ibzUSl8KM= +github.com/minio/sha256-simd v0.1.0/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= +github.com/mr-tron/base58 v1.1.0 h1:Y51FGVJ91WBqCEabAi5OPUz38eAx8DakuAm5svLcsfQ= github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= +github.com/mr-tron/base58 v1.1.1/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= github.com/mr-tron/base58 v1.1.2 h1:ZEw4I2EgPKDJ2iEw0cNmLB3ROrEmkOtXIkaG7wZg+78= github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI= github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA= +github.com/multiformats/go-multiaddr v0.0.1 h1:/QUV3VBMDI6pi6xfiw7lr6xhDWWvQKn9udPn68kLSdY= github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.0.2/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr v0.0.4 h1:WgMSI84/eRLdbptXMkMWDXPjPq7SPLIgGUVm2eroyU4= github.com/multiformats/go-multiaddr v0.0.4/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= +github.com/multiformats/go-multiaddr-fmt v0.0.1 h1:5YjeOIzbX8OTKVaN72aOzGIYW7PnrZrnkDyOfAWRSMA= +github.com/multiformats/go-multiaddr-fmt v0.0.1/go.mod h1:aBYjqL4T/7j4Qx+R73XSv/8JsgnRFlf0w2KGLCmXl3Q= github.com/multiformats/go-multiaddr-net v0.0.1 h1:76O59E3FavvHqNg7jvzWzsPSW5JSi/ek0E4eiDVbg9g= github.com/multiformats/go-multiaddr-net v0.0.1/go.mod h1:nw6HSxNmCIQH27XPGBuX+d1tnvM7ihcFwHMSstNAVUU= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= +github.com/multiformats/go-multihash v0.0.1 h1:HHwN1K12I+XllBCrqKnhX949Orn4oawPkegHMu2vDqQ= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= github.com/multiformats/go-multihash v0.0.5 h1:1wxmCvTXAifAepIMyF39vZinRw5sbqjPs/UIi93+uik= github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= -github.com/multiformats/go-multistream v0.0.1/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= -github.com/multiformats/go-multistream v0.0.4 h1:rNgWgFyzRSTI9L+xISrz7kN5MdNXoEcoIeeCH05wLKA= -github.com/multiformats/go-multistream v0.0.4/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= +github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ= +github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/ginkgo v1.7.0 h1:WSHQ+IS43OoUrWtD1/bbclrwK8TTH5hzp+umCiuxHgs= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/gomega v1.4.3 h1:RE1xgDvH7imwFD45h+u2SgIfERHlS2yNG4DObb5BSKU= +github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w= +github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo= +github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/opentracing/opentracing-go v1.0.2 h1:3jA2P6O1F9UOrWVpwrIo17pu01KWvNWg4X946/Y5Zwg= github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= @@ -232,14 +220,16 @@ github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30/go.mod h1:j4l84 github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67 h1:ng3VDlRp5/DHpSWl02R4rM9I+8M2rhmsuLwAMmkLQWE= golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734 h1:p/H982KKEjUnLJkM3tt/LemDnOc1GiZL5FCVlORJ5zo= golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f h1:R423Cnkcp5JABoeemiGEPlt9tHXFfw5kvc0yqlxRPWo= +golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190227160552-c95aed5357e7 h1:C2F/nMkR/9sfUTpvR3QrjBuTdvMUC/cFajkphs1YLQo= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= @@ -248,6 +238,7 @@ golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 h1:DH4skfRX4EBpamg7iV4ZlCpblAHI6s6TDM39bFZumv8= golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI= diff --git a/libp2p.go b/libp2p.go index 252c9a2920..32fddbdb84 100644 --- a/libp2p.go +++ b/libp2p.go @@ -5,7 +5,7 @@ import ( config "github.com/libp2p/go-libp2p/config" - host "github.com/libp2p/go-libp2p-host" + "github.com/libp2p/go-libp2p-core/host" ) // Config describes a set of settings for a libp2p node diff --git a/libp2p_test.go b/libp2p_test.go index f2a28d6b23..d1099c647d 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -7,9 +7,9 @@ import ( "strings" "testing" - crypto "github.com/libp2p/go-libp2p-crypto" - host "github.com/libp2p/go-libp2p-host" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-tcp-transport" ) @@ -59,7 +59,7 @@ func TestNoTransports(t *testing.T) { } defer b.Close() - err = a.Connect(ctx, pstore.PeerInfo{ + err = a.Connect(ctx, peer.AddrInfo{ ID: b.ID(), Addrs: b.Addrs(), }) diff --git a/options.go b/options.go index e98edda945..6d310f1aa3 100644 --- a/options.go +++ b/options.go @@ -7,15 +7,16 @@ import ( "fmt" "net" + "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/metrics" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/pnet" + + circuit "github.com/libp2p/go-libp2p-circuit" config "github.com/libp2p/go-libp2p/config" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" - circuit "github.com/libp2p/go-libp2p-circuit" - crypto "github.com/libp2p/go-libp2p-crypto" - ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" - pnet "github.com/libp2p/go-libp2p-interface-pnet" - metrics "github.com/libp2p/go-libp2p-metrics" - pstore "github.com/libp2p/go-libp2p-peerstore" filter "github.com/libp2p/go-maddr-filter" ma "github.com/multiformats/go-multiaddr" ) @@ -132,7 +133,7 @@ func Transport(tpt interface{}) Option { } // Peerstore configures libp2p to use the given peerstore. -func Peerstore(ps pstore.Peerstore) Option { +func Peerstore(ps peerstore.Peerstore) Option { return func(cfg *Config) error { if cfg.Peerstore != nil { return fmt.Errorf("cannot specify multiple peerstore options") @@ -180,7 +181,7 @@ func Identity(sk crypto.PrivKey) Option { } // ConnectionManager configures libp2p to use the given connection manager. -func ConnectionManager(connman ifconnmgr.ConnManager) Option { +func ConnectionManager(connman connmgr.ConnManager) Option { return func(cfg *Config) error { if cfg.ConnManager != nil { return fmt.Errorf("cannot specify multiple connection managers") diff --git a/p2p/discovery/mdns.go b/p2p/discovery/mdns.go index e41c175caa..3476e6ebf7 100644 --- a/p2p/discovery/mdns.go +++ b/p2p/discovery/mdns.go @@ -8,10 +8,10 @@ import ( "sync" "time" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" + logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-host" - "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" "github.com/whyrusleeping/mdns" @@ -33,7 +33,7 @@ type Service interface { } type Notifee interface { - HandlePeerFound(pstore.PeerInfo) + HandlePeerFound(peer.AddrInfo) } type mdnsService struct { @@ -173,7 +173,7 @@ func (m *mdnsService) handleEntry(e *mdns.ServiceEntry) { return } - pi := pstore.PeerInfo{ + pi := peer.AddrInfo{ ID: mpeer, Addrs: []ma.Multiaddr{maddr}, } diff --git a/p2p/discovery/mdns_test.go b/p2p/discovery/mdns_test.go index bb0e84d6f1..8f1e5bec91 100644 --- a/p2p/discovery/mdns_test.go +++ b/p2p/discovery/mdns_test.go @@ -5,19 +5,18 @@ import ( "testing" "time" - bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" - host "github.com/libp2p/go-libp2p-host" swarmt "github.com/libp2p/go-libp2p-swarm/testing" - - pstore "github.com/libp2p/go-libp2p-peerstore" + bhost "github.com/libp2p/go-libp2p/p2p/host/basic" ) type DiscoveryNotifee struct { h host.Host } -func (n *DiscoveryNotifee) HandlePeerFound(pi pstore.PeerInfo) { +func (n *DiscoveryNotifee) HandlePeerFound(pi peer.AddrInfo) { n.h.Connect(context.Background(), pi) } @@ -49,7 +48,7 @@ func TestMdnsDiscovery(t *testing.T) { time.Sleep(time.Second * 2) - err = a.Connect(ctx, pstore.PeerInfo{ID: b.ID()}) + err = a.Connect(ctx, peer.AddrInfo{ID: b.ID()}) if err != nil { t.Fatal(err) } diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 00540307a1..b1d8cbcd42 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -7,15 +7,19 @@ import ( "sync" "time" + "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/protocol" + logging "github.com/ipfs/go-log" goprocess "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" - ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" + inat "github.com/libp2p/go-libp2p-nat" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - protocol "github.com/libp2p/go-libp2p-protocol" + identify "github.com/libp2p/go-libp2p/p2p/protocol/identify" ping "github.com/libp2p/go-libp2p/p2p/protocol/ping" ma "github.com/multiformats/go-multiaddr" @@ -58,13 +62,13 @@ const NATPortMap Option = iota // * uses an identity service to send + receive node information // * uses a nat service to establish NAT port mappings type BasicHost struct { - network inet.Network + network network.Network mux *msmux.MultistreamMuxer ids *identify.IDService pings *ping.PingService natmgr NATManager maResolver *madns.Resolver - cmgr ifconnmgr.ConnManager + cmgr connmgr.ConnManager AddrsFactory AddrsFactory @@ -78,6 +82,8 @@ type BasicHost struct { lastAddrs []ma.Multiaddr } +var _ host.Host = (*BasicHost)(nil) + // HostOpts holds options that can be passed to NewHost in order to // customize construction of the *BasicHost. type HostOpts struct { @@ -104,17 +110,17 @@ type HostOpts struct { // NATManager takes care of setting NAT port mappings, and discovering external addresses. // If omitted, this will simply be disabled. - NATManager func(inet.Network) NATManager + NATManager func(network.Network) NATManager // ConnManager is a libp2p connection manager - ConnManager ifconnmgr.ConnManager + ConnManager connmgr.ConnManager // EnablePing indicates whether to instantiate the ping service EnablePing bool } // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. -func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, error) { +func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHost, error) { bgctx, cancel := context.WithCancel(ctx) h := &BasicHost{ @@ -162,7 +168,7 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, } if opts.ConnManager == nil { - h.cmgr = &ifconnmgr.NullConnMgr{} + h.cmgr = &connmgr.NullConnMgr{} } else { h.cmgr = opts.ConnManager net.Notify(h.cmgr.Notifee()) @@ -182,11 +188,11 @@ func NewHost(ctx context.Context, net inet.Network, opts *HostOpts) (*BasicHost, // The following options can be passed: // * NATPortMap // * AddrsFactory -// * ifconnmgr.ConnManager +// * connmgr.ConnManager // * madns.Resolver // // This function is deprecated in favor of NewHost and HostOpts. -func New(net inet.Network, opts ...interface{}) *BasicHost { +func New(net network.Network, opts ...interface{}) *BasicHost { hostopts := &HostOpts{} for _, o := range opts { @@ -198,7 +204,7 @@ func New(net inet.Network, opts ...interface{}) *BasicHost { } case AddrsFactory: hostopts.AddrsFactory = AddrsFactory(o) - case ifconnmgr.ConnManager: + case connmgr.ConnManager: hostopts.ConnManager = o case *madns.Resolver: hostopts.MultiaddrResolver = o @@ -221,16 +227,16 @@ func (h *BasicHost) Start() { } // newConnHandler is the remote-opened conn handler for inet.Network -func (h *BasicHost) newConnHandler(c inet.Conn) { +func (h *BasicHost) newConnHandler(c network.Conn) { // Clear protocols on connecting to new peer to avoid issues caused // by misremembering protocols between reconnects h.Peerstore().SetProtocols(c.RemotePeer()) h.ids.IdentifyConn(c) } -// newStreamHandler is the remote-opened stream handler for inet.Network +// newStreamHandler is the remote-opened stream handler for network.Network // TODO: this feels a bit wonky -func (h *BasicHost) newStreamHandler(s inet.Stream) { +func (h *BasicHost) newStreamHandler(s network.Stream) { before := time.Now() if h.negtimeout > 0 { @@ -344,17 +350,17 @@ func (h *BasicHost) ID() peer.ID { } // Peerstore returns the Host's repository of Peer Addresses and Keys. -func (h *BasicHost) Peerstore() pstore.Peerstore { +func (h *BasicHost) Peerstore() peerstore.Peerstore { return h.Network().Peerstore() } // Network returns the Network interface of the Host -func (h *BasicHost) Network() inet.Network { +func (h *BasicHost) Network() network.Network { return h.network } // Mux returns the Mux multiplexing incoming streams to protocol handlers -func (h *BasicHost) Mux() *msmux.MultistreamMuxer { +func (h *BasicHost) Mux() protocol.Switch { return h.mux } @@ -367,9 +373,9 @@ func (h *BasicHost) IDService() *identify.IDService { // This is equivalent to: // host.Mux().SetHandler(proto, handler) // (Threadsafe) -func (h *BasicHost) SetStreamHandler(pid protocol.ID, handler inet.StreamHandler) { +func (h *BasicHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) { h.Mux().AddHandler(string(pid), func(p string, rwc io.ReadWriteCloser) error { - is := rwc.(inet.Stream) + is := rwc.(network.Stream) is.SetProtocol(protocol.ID(p)) handler(is) return nil @@ -378,9 +384,9 @@ func (h *BasicHost) SetStreamHandler(pid protocol.ID, handler inet.StreamHandler // SetStreamHandlerMatch sets the protocol handler on the Host's Mux // using a matching function to do protocol comparisons -func (h *BasicHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler inet.StreamHandler) { +func (h *BasicHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler network.StreamHandler) { h.Mux().AddHandlerWithFunc(string(pid), m, func(p string, rwc io.ReadWriteCloser) error { - is := rwc.(inet.Stream) + is := rwc.(network.Stream) is.SetProtocol(protocol.ID(p)) handler(is) return nil @@ -396,7 +402,7 @@ func (h *BasicHost) RemoveStreamHandler(pid protocol.ID) { // header with given protocol.ID. If there is no connection to p, attempts // to create one. If ProtocolID is "", writes no header. // (Threadsafe) -func (h *BasicHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol.ID) (inet.Stream, error) { +func (h *BasicHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol.ID) (network.Stream, error) { pref, err := h.preferredProtocol(p, pids) if err != nil { return nil, err @@ -450,7 +456,7 @@ func (h *BasicHost) preferredProtocol(p peer.ID, pids []protocol.ID) (protocol.I return out, nil } -func (h *BasicHost) newStream(ctx context.Context, p peer.ID, pid protocol.ID) (inet.Stream, error) { +func (h *BasicHost) newStream(ctx context.Context, p peer.ID, pid protocol.ID) (network.Stream, error) { s, err := h.Network().NewStream(ctx, p) if err != nil { return nil, err @@ -470,11 +476,11 @@ func (h *BasicHost) newStream(ctx context.Context, p peer.ID, pid protocol.ID) ( // h.Network.Dial, and block until a connection is open, or an error is returned. // Connect will absorb the addresses in pi into its internal peerstore. // It will also resolve any /dns4, /dns6, and /dnsaddr addresses. -func (h *BasicHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { +func (h *BasicHost) Connect(ctx context.Context, pi peer.AddrInfo) error { // absorb addresses into peerstore - h.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) + h.Peerstore().AddAddrs(pi.ID, pi.Addrs, peerstore.TempAddrTTL) - if h.Network().Connectedness(pi.ID) == inet.Connected { + if h.Network().Connectedness(pi.ID) == network.Connected { return nil } @@ -482,12 +488,12 @@ func (h *BasicHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { if err != nil { return err } - h.Peerstore().AddAddrs(pi.ID, resolved, pstore.TempAddrTTL) + h.Peerstore().AddAddrs(pi.ID, resolved, peerstore.TempAddrTTL) return h.dialPeer(ctx, pi.ID) } -func (h *BasicHost) resolveAddrs(ctx context.Context, pi pstore.PeerInfo) ([]ma.Multiaddr, error) { +func (h *BasicHost) resolveAddrs(ctx context.Context, pi peer.AddrInfo) ([]ma.Multiaddr, error) { proto := ma.ProtocolWithCode(ma.P_P2P).Name p2paddr, err := ma.NewMultiaddr("/" + proto + "/" + pi.ID.Pretty()) if err != nil { @@ -507,7 +513,7 @@ func (h *BasicHost) resolveAddrs(ctx context.Context, pi pstore.PeerInfo) ([]ma. log.Infof("error resolving %s: %s", reqaddr, err) } for _, res := range resaddrs { - pi, err := pstore.InfoFromP2pAddr(res) + pi, err := peer.AddrInfoFromP2pAddr(res) if err != nil { log.Infof("error parsing %s: %s", res, err) } @@ -549,7 +555,7 @@ func (h *BasicHost) dialPeer(ctx context.Context, p peer.ID) error { return nil } -func (h *BasicHost) ConnManager() ifconnmgr.ConnManager { +func (h *BasicHost) ConnManager() connmgr.ConnManager { return h.cmgr } @@ -724,7 +730,7 @@ func (h *BasicHost) Close() error { } type streamWrapper struct { - inet.Stream + network.Stream rw io.ReadWriter } diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index affdec7b3a..270e353e53 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -8,19 +8,19 @@ import ( "testing" "time" - testutil "github.com/libp2p/go-testutil" + "github.com/libp2p/go-libp2p-core/helpers" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/protocol" + "github.com/libp2p/go-libp2p-core/test" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" - protocol "github.com/libp2p/go-libp2p-protocol" swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" madns "github.com/multiformats/go-multiaddr-dns" ) func TestHostSimple(t *testing.T) { - ctx := context.Background() h1 := New(swarmt.GenSwarm(t, ctx)) h2 := New(swarmt.GenSwarm(t, ctx)) @@ -33,7 +33,7 @@ func TestHostSimple(t *testing.T) { } piper, pipew := io.Pipe() - h2.SetStreamHandler(protocol.TestingID, func(s inet.Stream) { + h2.SetStreamHandler(protocol.TestingID, func(s network.Stream) { defer s.Close() w := io.MultiWriter(s, pipew) io.Copy(w, s) // mirror everything @@ -125,7 +125,7 @@ func TestHostProtoPreference(t *testing.T) { connectedOn := make(chan protocol.ID) - handler := func(s inet.Stream) { + handler := func(s network.Stream) { connectedOn <- s.Protocol() s.Close() } @@ -140,7 +140,7 @@ func TestHostProtoPreference(t *testing.T) { assertWait(t, connectedOn, protoOld) s.Close() - mfunc, err := host.MultistreamSemverMatcher(protoMinor) + mfunc, err := helpers.MultistreamSemverMatcher(protoMinor) if err != nil { t.Fatal(err) } @@ -180,7 +180,7 @@ func TestHostProtoMismatch(t *testing.T) { defer h1.Close() defer h2.Close() - h1.SetStreamHandler("/super", func(s inet.Stream) { + h1.SetStreamHandler("/super", func(s network.Stream) { t.Error("shouldnt get here") s.Reset() }) @@ -199,7 +199,7 @@ func TestHostProtoPreknowledge(t *testing.T) { h2 := New(swarmt.GenSwarm(t, ctx)) conn := make(chan protocol.ID) - handler := func(s inet.Stream) { + handler := func(s network.Stream) { conn <- s.Protocol() s.Close() } @@ -258,7 +258,7 @@ func TestNewDialOld(t *testing.T) { defer h2.Close() connectedOn := make(chan protocol.ID) - h1.SetStreamHandler("/testing", func(s inet.Stream) { + h1.SetStreamHandler("/testing", func(s network.Stream) { connectedOn <- s.Protocol() s.Close() }) @@ -286,7 +286,7 @@ func TestProtoDowngrade(t *testing.T) { defer h2.Close() connectedOn := make(chan protocol.ID) - h1.SetStreamHandler("/testing/1.0.0", func(s inet.Stream) { + h1.SetStreamHandler("/testing/1.0.0", func(s network.Stream) { connectedOn <- s.Protocol() s.Close() }) @@ -307,7 +307,7 @@ func TestProtoDowngrade(t *testing.T) { time.Sleep(time.Millisecond * 50) // allow notifications to propogate h1.RemoveStreamHandler("/testing/1.0.0") - h1.SetStreamHandler("/testing", func(s inet.Stream) { + h1.SetStreamHandler("/testing", func(s network.Stream) { connectedOn <- s.Protocol() s.Close() }) @@ -339,11 +339,11 @@ func TestProtoDowngrade(t *testing.T) { func TestAddrResolution(t *testing.T) { ctx := context.Background() - p1, err := testutil.RandPeerID() + p1, err := test.RandPeerID() if err != nil { t.Error(err) } - p2, err := testutil.RandPeerID() + p2, err := test.RandPeerID() if err != nil { t.Error(err) } @@ -363,7 +363,7 @@ func TestAddrResolution(t *testing.T) { h := New(swarmt.GenSwarm(t, ctx), resolver) defer h.Close() - pi, err := pstore.InfoFromP2pAddr(p2paddr1) + pi, err := peer.AddrInfoFromP2pAddr(p2paddr1) if err != nil { t.Error(err) } diff --git a/p2p/host/basic/natmgr.go b/p2p/host/basic/natmgr.go index 25c8b029d9..c0bf96a8f1 100644 --- a/p2p/host/basic/natmgr.go +++ b/p2p/host/basic/natmgr.go @@ -7,8 +7,8 @@ import ( goprocess "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" + "github.com/libp2p/go-libp2p-core/network" inat "github.com/libp2p/go-libp2p-nat" - inet "github.com/libp2p/go-libp2p-net" ma "github.com/multiformats/go-multiaddr" ) @@ -26,7 +26,7 @@ type NATManager interface { } // Create a NAT manager. -func NewNATManager(net inet.Network) NATManager { +func NewNATManager(net network.Network) NATManager { return newNatManager(net) } @@ -37,7 +37,7 @@ func NewNATManager(net inet.Network) NATManager { // as the network signals Listen() or ListenClose(). // * closing the natManager closes the nat and its mappings. type natManager struct { - net inet.Network + net network.Network natmu sync.RWMutex nat *inat.NAT @@ -48,7 +48,7 @@ type natManager struct { proc goprocess.Process // natManager has a process + children. can be closed. } -func newNatManager(net inet.Network) *natManager { +func newNatManager(net network.Network) *natManager { nmgr := &natManager{ net: net, ready: make(chan struct{}), @@ -229,15 +229,15 @@ func (nn *nmgrNetNotifiee) natManager() *natManager { return (*natManager)(nn) } -func (nn *nmgrNetNotifiee) Listen(n inet.Network, addr ma.Multiaddr) { +func (nn *nmgrNetNotifiee) Listen(n network.Network, addr ma.Multiaddr) { nn.natManager().sync() } -func (nn *nmgrNetNotifiee) ListenClose(n inet.Network, addr ma.Multiaddr) { +func (nn *nmgrNetNotifiee) ListenClose(n network.Network, addr ma.Multiaddr) { nn.natManager().sync() } -func (nn *nmgrNetNotifiee) Connected(inet.Network, inet.Conn) {} -func (nn *nmgrNetNotifiee) Disconnected(inet.Network, inet.Conn) {} -func (nn *nmgrNetNotifiee) OpenedStream(inet.Network, inet.Stream) {} -func (nn *nmgrNetNotifiee) ClosedStream(inet.Network, inet.Stream) {} +func (nn *nmgrNetNotifiee) Connected(network.Network, network.Conn) {} +func (nn *nmgrNetNotifiee) Disconnected(network.Network, network.Conn) {} +func (nn *nmgrNetNotifiee) OpenedStream(network.Network, network.Stream) {} +func (nn *nmgrNetNotifiee) ClosedStream(network.Network, network.Stream) {} diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index fbfb849740..ac381ac62f 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -7,15 +7,15 @@ import ( "sync" "time" - basic "github.com/libp2p/go-libp2p/p2p/host/basic" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/routing" autonat "github.com/libp2p/go-libp2p-autonat" _ "github.com/libp2p/go-libp2p-circuit" discovery "github.com/libp2p/go-libp2p-discovery" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - routing "github.com/libp2p/go-libp2p-routing" + basic "github.com/libp2p/go-libp2p/p2p/host/basic" + ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) @@ -188,7 +188,7 @@ func (ar *AutoRelay) usingRelay(p peer.ID) bool { // addRelay adds the given relay to our set of relays. // returns true when we add a new relay -func (ar *AutoRelay) tryRelay(ctx context.Context, pi pstore.PeerInfo) bool { +func (ar *AutoRelay) tryRelay(ctx context.Context, pi peer.AddrInfo) bool { if ar.usingRelay(pi.ID) { return false } @@ -201,7 +201,7 @@ func (ar *AutoRelay) tryRelay(ctx context.Context, pi pstore.PeerInfo) bool { defer ar.mx.Unlock() // make sure we're still connected. - if ar.host.Network().Connectedness(pi.ID) != inet.Connected { + if ar.host.Network().Connectedness(pi.ID) != network.Connected { return false } ar.relays[pi.ID] = struct{}{} @@ -209,7 +209,7 @@ func (ar *AutoRelay) tryRelay(ctx context.Context, pi pstore.PeerInfo) bool { return true } -func (ar *AutoRelay) connect(ctx context.Context, pi pstore.PeerInfo) bool { +func (ar *AutoRelay) connect(ctx context.Context, pi peer.AddrInfo) bool { ctx, cancel := context.WithTimeout(ctx, 60*time.Second) defer cancel() @@ -233,13 +233,13 @@ func (ar *AutoRelay) connect(ctx context.Context, pi pstore.PeerInfo) bool { return true } -func (ar *AutoRelay) discoverRelays(ctx context.Context) ([]pstore.PeerInfo, error) { +func (ar *AutoRelay) discoverRelays(ctx context.Context) ([]peer.AddrInfo, error) { ctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() return discovery.FindPeers(ctx, ar.discover, RelayRendezvous, discovery.Limit(1000)) } -func (ar *AutoRelay) selectRelays(ctx context.Context, pis []pstore.PeerInfo) []pstore.PeerInfo { +func (ar *AutoRelay) selectRelays(ctx context.Context, pis []peer.AddrInfo) []peer.AddrInfo { // TODO better relay selection strategy; this just selects random relays // but we should probably use ping latency as the selection metric @@ -296,7 +296,7 @@ func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { return raddrs } -func shuffleRelays(pis []pstore.PeerInfo) { +func shuffleRelays(pis []peer.AddrInfo) { for i := range pis { j := rand.Intn(i + 1) pis[i], pis[j] = pis[j], pis[i] @@ -304,17 +304,17 @@ func shuffleRelays(pis []pstore.PeerInfo) { } // Notifee -func (ar *AutoRelay) Listen(inet.Network, ma.Multiaddr) {} -func (ar *AutoRelay) ListenClose(inet.Network, ma.Multiaddr) {} -func (ar *AutoRelay) Connected(inet.Network, inet.Conn) {} +func (ar *AutoRelay) Listen(network.Network, ma.Multiaddr) {} +func (ar *AutoRelay) ListenClose(network.Network, ma.Multiaddr) {} +func (ar *AutoRelay) Connected(network.Network, network.Conn) {} -func (ar *AutoRelay) Disconnected(net inet.Network, c inet.Conn) { +func (ar *AutoRelay) Disconnected(net network.Network, c network.Conn) { p := c.RemotePeer() ar.mx.Lock() defer ar.mx.Unlock() - if ar.host.Network().Connectedness(p) == inet.Connected { + if ar.host.Network().Connectedness(p) == network.Connected { // We have a second connection. return } @@ -328,5 +328,5 @@ func (ar *AutoRelay) Disconnected(net inet.Network, c inet.Conn) { } } -func (ar *AutoRelay) OpenedStream(inet.Network, inet.Stream) {} -func (ar *AutoRelay) ClosedStream(inet.Network, inet.Stream) {} +func (ar *AutoRelay) OpenedStream(network.Network, network.Stream) {} +func (ar *AutoRelay) ClosedStream(network.Network, network.Stream) {} diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index 4f5485a387..b6a6930c02 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -16,11 +16,10 @@ import ( autonat "github.com/libp2p/go-libp2p-autonat" autonatpb "github.com/libp2p/go-libp2p-autonat/pb" circuit "github.com/libp2p/go-libp2p-circuit" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - routing "github.com/libp2p/go-libp2p-routing" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/routing" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) @@ -36,8 +35,8 @@ func init() { // mock routing type mockRoutingTable struct { mx sync.Mutex - providers map[string]map[peer.ID]pstore.PeerInfo - peers map[peer.ID]pstore.PeerInfo + providers map[string]map[peer.ID]peer.AddrInfo + peers map[peer.ID]peer.AddrInfo } type mockRouting struct { @@ -46,19 +45,19 @@ type mockRouting struct { } func newMockRoutingTable() *mockRoutingTable { - return &mockRoutingTable{providers: make(map[string]map[peer.ID]pstore.PeerInfo)} + return &mockRoutingTable{providers: make(map[string]map[peer.ID]peer.AddrInfo)} } func newMockRouting(h host.Host, tab *mockRoutingTable) *mockRouting { return &mockRouting{h: h, tab: tab} } -func (m *mockRouting) FindPeer(ctx context.Context, p peer.ID) (pstore.PeerInfo, error) { +func (m *mockRouting) FindPeer(ctx context.Context, p peer.ID) (peer.AddrInfo, error) { m.tab.mx.Lock() defer m.tab.mx.Unlock() pi, ok := m.tab.peers[p] if !ok { - return pstore.PeerInfo{}, routing.ErrNotFound + return peer.AddrInfo{}, routing.ErrNotFound } return pi, nil } @@ -69,22 +68,22 @@ func (m *mockRouting) Provide(ctx context.Context, cid cid.Cid, bcast bool) erro pmap, ok := m.tab.providers[cid.String()] if !ok { - pmap = make(map[peer.ID]pstore.PeerInfo) + pmap = make(map[peer.ID]peer.AddrInfo) m.tab.providers[cid.String()] = pmap } - pi := pstore.PeerInfo{ID: m.h.ID(), Addrs: m.h.Addrs()} + pi := peer.AddrInfo{ID: m.h.ID(), Addrs: m.h.Addrs()} pmap[m.h.ID()] = pi if m.tab.peers == nil { - m.tab.peers = make(map[peer.ID]pstore.PeerInfo) + m.tab.peers = make(map[peer.ID]peer.AddrInfo) } m.tab.peers[m.h.ID()] = pi return nil } -func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit int) <-chan pstore.PeerInfo { - ch := make(chan pstore.PeerInfo) +func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit int) <-chan peer.AddrInfo { + ch := make(chan peer.AddrInfo) go func() { defer close(ch) m.tab.mx.Lock() @@ -117,7 +116,7 @@ func makeAutoNATServicePrivate(ctx context.Context, t *testing.T) host.Host { return h } -func sayAutoNATPrivate(s inet.Stream) { +func sayAutoNATPrivate(s network.Stream) { defer s.Close() w := ggio.NewDelimitedWriter(s) res := autonatpb.Message{ @@ -136,7 +135,7 @@ func newDialResponseError(status autonatpb.Message_ResponseStatus, text string) // connector func connect(t *testing.T, a, b host.Host) { - pinfo := pstore.PeerInfo{ID: a.ID(), Addrs: a.Addrs()} + pinfo := peer.AddrInfo{ID: a.ID(), Addrs: a.Addrs()} err := b.Connect(context.Background(), pinfo) if err != nil { t.Fatal(err) @@ -214,7 +213,7 @@ func TestAutoRelay(t *testing.T) { } } - err = h4.Connect(ctx, pstore.PeerInfo{ID: h3.ID(), Addrs: raddrs}) + err = h4.Connect(ctx, peer.AddrInfo{ID: h3.ID(), Addrs: raddrs}) if err != nil { t.Fatal(err) } diff --git a/p2p/host/relay/relay.go b/p2p/host/relay/relay.go index 7c57a7cd8a..f72b74ff72 100644 --- a/p2p/host/relay/relay.go +++ b/p2p/host/relay/relay.go @@ -4,7 +4,8 @@ import ( "context" "time" - discovery "github.com/libp2p/go-libp2p-discovery" + "github.com/libp2p/go-libp2p-discovery" + ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/host/routed/routed.go b/p2p/host/routed/routed.go index e5b2e40d14..1282020bcb 100644 --- a/p2p/host/routed/routed.go +++ b/p2p/host/routed/routed.go @@ -5,18 +5,18 @@ import ( "fmt" "time" - host "github.com/libp2p/go-libp2p-host" + "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/protocol" logging "github.com/ipfs/go-log" circuit "github.com/libp2p/go-libp2p-circuit" - ifconnmgr "github.com/libp2p/go-libp2p-interface-connmgr" lgbl "github.com/libp2p/go-libp2p-loggables" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" - protocol "github.com/libp2p/go-libp2p-protocol" + ma "github.com/multiformats/go-multiaddr" - msmux "github.com/multiformats/go-multistream" ) var log = logging.Logger("routedhost") @@ -34,7 +34,7 @@ type RoutedHost struct { } type Routing interface { - FindPeer(context.Context, peer.ID) (pstore.PeerInfo, error) + FindPeer(context.Context, peer.ID) (peer.AddrInfo, error) } func Wrap(h host.Host, r Routing) *RoutedHost { @@ -46,15 +46,15 @@ func Wrap(h host.Host, r Routing) *RoutedHost { // // RoutedHost's Connect differs in that if the host has no addresses for a // given peer, it will use its routing system to try to find some. -func (rh *RoutedHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { +func (rh *RoutedHost) Connect(ctx context.Context, pi peer.AddrInfo) error { // first, check if we're already connected. - if rh.Network().Connectedness(pi.ID) == inet.Connected { + if rh.Network().Connectedness(pi.ID) == network.Connected { return nil } // if we were given some addresses, keep + use them. if len(pi.Addrs) > 0 { - rh.Peerstore().AddAddrs(pi.ID, pi.Addrs, pstore.TempAddrTTL) + rh.Peerstore().AddAddrs(pi.ID, pi.Addrs, peerstore.TempAddrTTL) } // Check if we have some addresses in our recent memory. @@ -102,7 +102,7 @@ func (rh *RoutedHost) Connect(ctx context.Context, pi pstore.PeerInfo) error { continue } - rh.Peerstore().AddAddrs(relayID, relayAddrs, pstore.TempAddrTTL) + rh.Peerstore().AddAddrs(relayID, relayAddrs, peerstore.TempAddrTTL) } // if we're here, we got some addrs. let's use our wrapped host to connect. @@ -137,7 +137,7 @@ func (rh *RoutedHost) ID() peer.ID { return rh.host.ID() } -func (rh *RoutedHost) Peerstore() pstore.Peerstore { +func (rh *RoutedHost) Peerstore() peerstore.Peerstore { return rh.host.Peerstore() } @@ -145,19 +145,19 @@ func (rh *RoutedHost) Addrs() []ma.Multiaddr { return rh.host.Addrs() } -func (rh *RoutedHost) Network() inet.Network { +func (rh *RoutedHost) Network() network.Network { return rh.host.Network() } -func (rh *RoutedHost) Mux() *msmux.MultistreamMuxer { +func (rh *RoutedHost) Mux() protocol.Switch { return rh.host.Mux() } -func (rh *RoutedHost) SetStreamHandler(pid protocol.ID, handler inet.StreamHandler) { +func (rh *RoutedHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) { rh.host.SetStreamHandler(pid, handler) } -func (rh *RoutedHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler inet.StreamHandler) { +func (rh *RoutedHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler network.StreamHandler) { rh.host.SetStreamHandlerMatch(pid, m, handler) } @@ -165,13 +165,13 @@ func (rh *RoutedHost) RemoveStreamHandler(pid protocol.ID) { rh.host.RemoveStreamHandler(pid) } -func (rh *RoutedHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol.ID) (inet.Stream, error) { +func (rh *RoutedHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol.ID) (network.Stream, error) { // Ensure we have a connection, with peer addresses resolved by the routing system (#207) // It is not sufficient to let the underlying host connect, it will most likely not have // any addresses for the peer without any prior connections. // If the caller wants to prevent the host from dialing, it should use the NoDial option. - if nodial, _ := inet.GetNoDial(ctx); !nodial { - err := rh.Connect(ctx, pstore.PeerInfo{ID: p}) + if nodial, _ := network.GetNoDial(ctx); !nodial { + err := rh.Connect(ctx, peer.AddrInfo{ID: p}) if err != nil { return nil, err } @@ -183,7 +183,7 @@ func (rh *RoutedHost) Close() error { // no need to close IpfsRouting. we dont own it. return rh.host.Close() } -func (rh *RoutedHost) ConnManager() ifconnmgr.ConnManager { +func (rh *RoutedHost) ConnManager() connmgr.ConnManager { return rh.host.ConnManager() } diff --git a/p2p/net/mock/interface.go b/p2p/net/mock/interface.go index e62fc4d718..57861bf1cd 100644 --- a/p2p/net/mock/interface.go +++ b/p2p/net/mock/interface.go @@ -1,43 +1,43 @@ // Package mocknet provides a mock net.Network to test with. // -// - a Mocknet has many inet.Networks +// - a Mocknet has many network.Networks // - a Mocknet has many Links -// - a Link joins two inet.Networks -// - inet.Conns and inet.Streams are created by inet.Networks +// - a Link joins two network.Networks +// - network.Conns and network.Streams are created by network.Networks package mocknet import ( "io" "time" - host "github.com/libp2p/go-libp2p-host" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" - ic "github.com/libp2p/go-libp2p-crypto" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" ) type Mocknet interface { - // GenPeer generates a peer and its inet.Network in the Mocknet + // GenPeer generates a peer and its network.Network in the Mocknet GenPeer() (host.Host, error) // AddPeer adds an existing peer. we need both a privkey and addr. // ID is derived from PrivKey AddPeer(ic.PrivKey, ma.Multiaddr) (host.Host, error) - AddPeerWithPeerstore(peer.ID, pstore.Peerstore) (host.Host, error) + AddPeerWithPeerstore(peer.ID, peerstore.Peerstore) (host.Host, error) // retrieve things (with randomized iteration order) Peers() []peer.ID - Net(peer.ID) inet.Network - Nets() []inet.Network + Net(peer.ID) network.Network + Nets() []network.Network Host(peer.ID) host.Host Hosts() []host.Host Links() LinkMap LinksBetweenPeers(a, b peer.ID) []Link - LinksBetweenNets(a, b inet.Network) []Link + LinksBetweenNets(a, b network.Network) []Link // Links are the **ability to connect**. // think of Links as the physical medium. @@ -45,10 +45,10 @@ type Mocknet interface { // (this makes it possible to test dial failures, and // things like relaying traffic) LinkPeers(peer.ID, peer.ID) (Link, error) - LinkNets(inet.Network, inet.Network) (Link, error) + LinkNets(network.Network, network.Network) (Link, error) Unlink(Link) error UnlinkPeers(peer.ID, peer.ID) error - UnlinkNets(inet.Network, inet.Network) error + UnlinkNets(network.Network, network.Network) error // LinkDefaults are the default options that govern links // if they do not have thier own option set. @@ -57,10 +57,10 @@ type Mocknet interface { // Connections are the usual. Connecting means Dialing. // **to succeed, peers must be linked beforehand** - ConnectPeers(peer.ID, peer.ID) (inet.Conn, error) - ConnectNets(inet.Network, inet.Network) (inet.Conn, error) + ConnectPeers(peer.ID, peer.ID) (network.Conn, error) + ConnectNets(network.Network, network.Network) (network.Conn, error) DisconnectPeers(peer.ID, peer.ID) error - DisconnectNets(inet.Network, inet.Network) error + DisconnectNets(network.Network, network.Network) error LinkAll() error ConnectAllButSelf() error } @@ -79,7 +79,7 @@ type LinkOptions struct { // connect. This allows constructing topologies where specific // nodes cannot talk to each other directly. :) type Link interface { - Networks() []inet.Network + Networks() []network.Network Peers() []peer.ID SetOptions(LinkOptions) @@ -96,7 +96,7 @@ type LinkMap map[string]map[string]map[Link]struct{} type Printer interface { // MocknetLinks shows the entire Mocknet's link table :) MocknetLinks(mn Mocknet) - NetworkConns(ni inet.Network) + NetworkConns(ni network.Network) } // PrinterTo returns a Printer ready to write to w. diff --git a/p2p/net/mock/mock_conn.go b/p2p/net/mock/mock_conn.go index 3e0af7268a..17f56528ca 100644 --- a/p2p/net/mock/mock_conn.go +++ b/p2p/net/mock/mock_conn.go @@ -5,9 +5,9 @@ import ( "sync" process "github.com/jbenet/goprocess" - ic "github.com/libp2p/go-libp2p-crypto" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" ) @@ -29,16 +29,16 @@ type conn struct { rconn *conn // counterpart streams list.List proc process.Process - stat inet.Stat + stat network.Stat sync.RWMutex } -func newConn(ln, rn *peernet, l *link, dir inet.Direction) *conn { +func newConn(ln, rn *peernet, l *link, dir network.Direction) *conn { c := &conn{net: ln, link: l} c.local = ln.peer c.remote = rn.peer - c.stat = inet.Stat{Direction: dir} + c.stat = network.Stat{Direction: dir} c.localAddr = ln.ps.Addrs(ln.peer)[0] c.remoteAddr = rn.ps.Addrs(rn.peer)[0] @@ -59,7 +59,7 @@ func (c *conn) teardown() error { s.Reset() } c.net.removeConn(c) - c.net.notifyAll(func(n inet.Notifiee) { + c.net.notifyAll(func(n network.Notifiee) { n.Disconnected(c.net, c) }) return nil @@ -83,11 +83,11 @@ func (c *conn) removeStream(s *stream) { } } -func (c *conn) allStreams() []inet.Stream { +func (c *conn) allStreams() []network.Stream { c.RLock() defer c.RUnlock() - strs := make([]inet.Stream, 0, c.streams.Len()) + strs := make([]network.Stream, 0, c.streams.Len()) for e := c.streams.Front(); e != nil; e = e.Next() { s := e.Value.(*stream) strs = append(strs, s) @@ -98,7 +98,7 @@ func (c *conn) allStreams() []inet.Stream { func (c *conn) remoteOpenedStream(s *stream) { c.addStream(s) c.net.handleNewStream(s) - c.net.notifyAll(func(n inet.Notifiee) { + c.net.notifyAll(func(n network.Notifiee) { n.OpenedStream(c.net, s) }) } @@ -106,21 +106,21 @@ func (c *conn) remoteOpenedStream(s *stream) { func (c *conn) openStream() *stream { sl, sr := c.link.newStreamPair() c.addStream(sl) - c.net.notifyAll(func(n inet.Notifiee) { + c.net.notifyAll(func(n network.Notifiee) { n.OpenedStream(c.net, sl) }) c.rconn.remoteOpenedStream(sr) return sl } -func (c *conn) NewStream() (inet.Stream, error) { +func (c *conn) NewStream() (network.Stream, error) { log.Debugf("Conn.NewStreamWithProtocol: %s --> %s", c.local, c.remote) s := c.openStream() return s, nil } -func (c *conn) GetStreams() []inet.Stream { +func (c *conn) GetStreams() []network.Stream { return c.allStreams() } @@ -155,6 +155,6 @@ func (c *conn) RemotePublicKey() ic.PubKey { } // Stat returns metadata about the connection -func (c *conn) Stat() inet.Stat { +func (c *conn) Stat() network.Stat { return c.stat } diff --git a/p2p/net/mock/mock_link.go b/p2p/net/mock/mock_link.go index 9e31cbec2b..bd9481fb2d 100644 --- a/p2p/net/mock/mock_link.go +++ b/p2p/net/mock/mock_link.go @@ -6,12 +6,12 @@ import ( "sync" "time" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" ) // link implements mocknet.Link -// and, for simplicity, inet.Conn +// and, for simplicity, network.Conn type link struct { mock *mocknet nets []*peernet @@ -33,8 +33,8 @@ func (l *link) newConnPair(dialer *peernet) (*conn, *conn) { l.RLock() defer l.RUnlock() - c1 := newConn(l.nets[0], l.nets[1], l, inet.DirOutbound) - c2 := newConn(l.nets[1], l.nets[0], l, inet.DirInbound) + c1 := newConn(l.nets[0], l.nets[1], l, network.DirOutbound) + c2 := newConn(l.nets[1], l.nets[0], l, network.DirInbound) c1.rconn = c2 c2.rconn = c1 @@ -48,16 +48,16 @@ func (l *link) newStreamPair() (*stream, *stream) { ra, wb := io.Pipe() rb, wa := io.Pipe() - sa := NewStream(wa, ra, inet.DirOutbound) - sb := NewStream(wb, rb, inet.DirInbound) + sa := NewStream(wa, ra, network.DirOutbound) + sb := NewStream(wb, rb, network.DirInbound) return sa, sb } -func (l *link) Networks() []inet.Network { +func (l *link) Networks() []network.Network { l.RLock() defer l.RUnlock() - cp := make([]inet.Network, len(l.nets)) + cp := make([]network.Network, len(l.nets)) for i, n := range l.nets { cp[i] = n } diff --git a/p2p/net/mock/mock_net.go b/p2p/net/mock/mock_net.go index 2e34d3f74b..832d7c2556 100644 --- a/p2p/net/mock/mock_net.go +++ b/p2p/net/mock/mock_net.go @@ -7,16 +7,18 @@ import ( "sort" "sync" - host "github.com/libp2p/go-libp2p-host" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + bhost "github.com/libp2p/go-libp2p/p2p/host/basic" "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" - ic "github.com/libp2p/go-libp2p-crypto" - inet "github.com/libp2p/go-libp2p-net" + p2putil "github.com/libp2p/go-libp2p-netutil" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" ma "github.com/multiformats/go-multiaddr" ) @@ -88,14 +90,14 @@ func (mn *mocknet) AddPeer(k ic.PrivKey, a ma.Multiaddr) (host.Host, error) { } ps := pstoremem.NewPeerstore() - ps.AddAddr(p, a, pstore.PermanentAddrTTL) + ps.AddAddr(p, a, peerstore.PermanentAddrTTL) ps.AddPrivKey(p, k) ps.AddPubKey(p, k.GetPublic()) return mn.AddPeerWithPeerstore(p, ps) } -func (mn *mocknet) AddPeerWithPeerstore(p peer.ID, ps pstore.Peerstore) (host.Host, error) { +func (mn *mocknet) AddPeerWithPeerstore(p peer.ID, ps peerstore.Peerstore) (host.Host, error) { n, err := newPeernet(mn.ctx, mn, p, ps) if err != nil { return nil, err @@ -138,7 +140,7 @@ func (mn *mocknet) Host(pid peer.ID) host.Host { return host } -func (mn *mocknet) Net(pid peer.ID) inet.Network { +func (mn *mocknet) Net(pid peer.ID) network.Network { mn.Lock() n := mn.nets[pid] mn.Unlock() @@ -158,11 +160,11 @@ func (mn *mocknet) Hosts() []host.Host { return cp } -func (mn *mocknet) Nets() []inet.Network { +func (mn *mocknet) Nets() []network.Network { mn.Lock() defer mn.Unlock() - cp := make([]inet.Network, 0, len(mn.nets)) + cp := make([]network.Network, 0, len(mn.nets)) for _, n := range mn.nets { cp = append(cp, n) } @@ -220,7 +222,7 @@ func (mn *mocknet) LinkPeers(p1, p2 peer.ID) (Link, error) { return mn.LinkNets(n1, n2) } -func (mn *mocknet) validate(n inet.Network) (*peernet, error) { +func (mn *mocknet) validate(n network.Network) (*peernet, error) { // WARNING: assumes locks acquired nr, ok := n.(*peernet) @@ -235,7 +237,7 @@ func (mn *mocknet) validate(n inet.Network) (*peernet, error) { return nr, nil } -func (mn *mocknet) LinkNets(n1, n2 inet.Network) (Link, error) { +func (mn *mocknet) LinkNets(n1, n2 network.Network) (Link, error) { mn.Lock() n1r, err1 := mn.validate(n1) n2r, err2 := mn.validate(n2) @@ -280,7 +282,7 @@ func (mn *mocknet) UnlinkPeers(p1, p2 peer.ID) error { return nil } -func (mn *mocknet) UnlinkNets(n1, n2 inet.Network) error { +func (mn *mocknet) UnlinkNets(n1, n2 network.Network) error { return mn.UnlinkPeers(n1.LocalPeer(), n2.LocalPeer()) } @@ -337,11 +339,11 @@ func (mn *mocknet) ConnectAllButSelf() error { return nil } -func (mn *mocknet) ConnectPeers(a, b peer.ID) (inet.Conn, error) { +func (mn *mocknet) ConnectPeers(a, b peer.ID) (network.Conn, error) { return mn.Net(a).DialPeer(mn.ctx, b) } -func (mn *mocknet) ConnectNets(a, b inet.Network) (inet.Conn, error) { +func (mn *mocknet) ConnectNets(a, b network.Network) (network.Conn, error) { return a.DialPeer(mn.ctx, b.LocalPeer()) } @@ -349,7 +351,7 @@ func (mn *mocknet) DisconnectPeers(p1, p2 peer.ID) error { return mn.Net(p1).ClosePeer(p2) } -func (mn *mocknet) DisconnectNets(n1, n2 inet.Network) error { +func (mn *mocknet) DisconnectNets(n1, n2 network.Network) error { return n1.ClosePeer(n2.LocalPeer()) } @@ -365,7 +367,7 @@ func (mn *mocknet) LinksBetweenPeers(p1, p2 peer.ID) []Link { return cp } -func (mn *mocknet) LinksBetweenNets(n1, n2 inet.Network) []Link { +func (mn *mocknet) LinksBetweenNets(n1, n2 network.Network) []Link { return mn.LinksBetweenPeers(n1.LocalPeer(), n2.LocalPeer()) } @@ -382,7 +384,7 @@ func (mn *mocknet) LinkDefaults() LinkOptions { } // netSlice for sorting by peer -type netSlice []inet.Network +type netSlice []network.Network func (es netSlice) Len() int { return len(es) } func (es netSlice) Swap(i, j int) { es[i], es[j] = es[j], es[i] } diff --git a/p2p/net/mock/mock_notif_test.go b/p2p/net/mock/mock_notif_test.go index 13758ff756..4caecbedfa 100644 --- a/p2p/net/mock/mock_notif_test.go +++ b/p2p/net/mock/mock_notif_test.go @@ -5,8 +5,9 @@ import ( "testing" "time" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/helpers" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" ) @@ -37,7 +38,7 @@ func TestNotifications(t *testing.T) { // test everyone got the correct connection opened calls for i, s := range nets { n := notifiees[i] - notifs := make(map[peer.ID][]inet.Conn) + notifs := make(map[peer.ID][]network.Conn) for j, s2 := range nets { if i == j { continue @@ -77,7 +78,7 @@ func TestNotifications(t *testing.T) { } } - complement := func(c inet.Conn) (inet.Network, *netNotifiee, *conn) { + complement := func(c network.Conn) (network.Network, *netNotifiee, *conn) { for i, s := range nets { for _, c2 := range s.Conns() { if c2.(*conn).rconn == c { @@ -89,8 +90,8 @@ func TestNotifications(t *testing.T) { return nil, nil, nil } - testOCStream := func(n *netNotifiee, s inet.Stream) { - var s2 inet.Stream + testOCStream := func(n *netNotifiee, s network.Stream) { + var s2 network.Stream select { case s2 = <-n.openedStream: t.Log("got notif for opened stream") @@ -113,8 +114,8 @@ func TestNotifications(t *testing.T) { } for _, s := range nets { - s.SetStreamHandler(func(s inet.Stream) { - inet.FullClose(s) + s.SetStreamHandler(func(s network.Stream) { + helpers.FullClose(s) }) } @@ -127,11 +128,11 @@ func TestNotifications(t *testing.T) { } } - streams := make(chan inet.Stream) + streams := make(chan network.Stream) for _, s := range nets { - s.SetStreamHandler(func(s inet.Stream) { + s.SetStreamHandler(func(s network.Stream) { streams <- s - inet.FullClose(s) + helpers.FullClose(s) }) } @@ -146,7 +147,7 @@ func TestNotifications(t *testing.T) { } else { t.Logf("%s %s <--%p--> %s %s", c.LocalPeer(), c.LocalMultiaddr(), st1, c.RemotePeer(), c.RemoteMultiaddr()) // st1.Write([]byte("hello")) - go inet.FullClose(st1) + go helpers.FullClose(st1) st2 := <-streams t.Logf("%s %s <--%p--> %s %s", c2.LocalPeer(), c2.LocalMultiaddr(), st2, c2.RemotePeer(), c2.RemoteMultiaddr()) testOCStream(notifiees[i], st1) @@ -163,7 +164,7 @@ func TestNotifications(t *testing.T) { c.(*conn).Close() c2.Close() - var c3, c4 inet.Conn + var c3, c4 network.Conn select { case c3 = <-n.disconnected: case <-time.After(timeout): @@ -188,38 +189,38 @@ func TestNotifications(t *testing.T) { type netNotifiee struct { listen chan ma.Multiaddr listenClose chan ma.Multiaddr - connected chan inet.Conn - disconnected chan inet.Conn - openedStream chan inet.Stream - closedStream chan inet.Stream + connected chan network.Conn + disconnected chan network.Conn + openedStream chan network.Stream + closedStream chan network.Stream } func newNetNotifiee(buffer int) *netNotifiee { return &netNotifiee{ listen: make(chan ma.Multiaddr, buffer), listenClose: make(chan ma.Multiaddr, buffer), - connected: make(chan inet.Conn, buffer), - disconnected: make(chan inet.Conn, buffer), - openedStream: make(chan inet.Stream, buffer), - closedStream: make(chan inet.Stream, buffer), + connected: make(chan network.Conn, buffer), + disconnected: make(chan network.Conn, buffer), + openedStream: make(chan network.Stream, buffer), + closedStream: make(chan network.Stream, buffer), } } -func (nn *netNotifiee) Listen(n inet.Network, a ma.Multiaddr) { +func (nn *netNotifiee) Listen(n network.Network, a ma.Multiaddr) { nn.listen <- a } -func (nn *netNotifiee) ListenClose(n inet.Network, a ma.Multiaddr) { +func (nn *netNotifiee) ListenClose(n network.Network, a ma.Multiaddr) { nn.listenClose <- a } -func (nn *netNotifiee) Connected(n inet.Network, v inet.Conn) { +func (nn *netNotifiee) Connected(n network.Network, v network.Conn) { nn.connected <- v } -func (nn *netNotifiee) Disconnected(n inet.Network, v inet.Conn) { +func (nn *netNotifiee) Disconnected(n network.Network, v network.Conn) { nn.disconnected <- v } -func (nn *netNotifiee) OpenedStream(n inet.Network, v inet.Stream) { +func (nn *netNotifiee) OpenedStream(n network.Network, v network.Stream) { nn.openedStream <- v } -func (nn *netNotifiee) ClosedStream(n inet.Network, v inet.Stream) { +func (nn *netNotifiee) ClosedStream(n network.Network, v network.Stream) { nn.closedStream <- v } diff --git a/p2p/net/mock/mock_peernet.go b/p2p/net/mock/mock_peernet.go index 35c9bbe647..b8d02fe275 100644 --- a/p2p/net/mock/mock_peernet.go +++ b/p2p/net/mock/mock_peernet.go @@ -6,20 +6,22 @@ import ( "math/rand" "sync" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" ) -// peernet implements inet.Network +// peernet implements network.Network type peernet struct { mocknet *mocknet // parent peer peer.ID - ps pstore.Peerstore + ps peerstore.Peerstore // conns are actual live connections between peers. // many conns could run over each link. @@ -27,19 +29,19 @@ type peernet struct { connsByPeer map[peer.ID]map[*conn]struct{} connsByLink map[*link]map[*conn]struct{} - // implement inet.Network - streamHandler inet.StreamHandler - connHandler inet.ConnHandler + // implement network.Network + streamHandler network.StreamHandler + connHandler network.ConnHandler notifmu sync.Mutex - notifs map[inet.Notifiee]struct{} + notifs map[network.Notifiee]struct{} proc goprocess.Process sync.RWMutex } // newPeernet constructs a new peernet -func newPeernet(ctx context.Context, m *mocknet, p peer.ID, ps pstore.Peerstore) (*peernet, error) { +func newPeernet(ctx context.Context, m *mocknet, p peer.ID, ps peerstore.Peerstore) (*peernet, error) { n := &peernet{ mocknet: m, @@ -49,7 +51,7 @@ func newPeernet(ctx context.Context, m *mocknet, p peer.ID, ps pstore.Peerstore) connsByPeer: map[peer.ID]map[*conn]struct{}{}, connsByLink: map[*link]map[*conn]struct{}{}, - notifs: make(map[inet.Notifiee]struct{}), + notifs: make(map[network.Notifiee]struct{}), } n.proc = goprocessctx.WithContextAndTeardown(ctx, n.teardown) @@ -83,7 +85,7 @@ func (pn *peernet) Close() error { return pn.proc.Close() } -func (pn *peernet) Peerstore() pstore.Peerstore { +func (pn *peernet) Peerstore() peerstore.Peerstore { return pn.ps } @@ -92,7 +94,7 @@ func (pn *peernet) String() string { } // handleNewStream is an internal function to trigger the client's handler -func (pn *peernet) handleNewStream(s inet.Stream) { +func (pn *peernet) handleNewStream(s network.Stream) { pn.RLock() handler := pn.streamHandler pn.RUnlock() @@ -102,7 +104,7 @@ func (pn *peernet) handleNewStream(s inet.Stream) { } // handleNewConn is an internal function to trigger the client's handler -func (pn *peernet) handleNewConn(c inet.Conn) { +func (pn *peernet) handleNewConn(c network.Conn) { pn.RLock() handler := pn.connHandler pn.RUnlock() @@ -113,7 +115,7 @@ func (pn *peernet) handleNewConn(c inet.Conn) { // DialPeer attempts to establish a connection to a given peer. // Respects the context. -func (pn *peernet) DialPeer(ctx context.Context, p peer.ID) (inet.Conn, error) { +func (pn *peernet) DialPeer(ctx context.Context, p peer.ID) (network.Conn, error) { return pn.connect(p) } @@ -159,7 +161,7 @@ func (pn *peernet) openConn(r peer.ID, l *link) *conn { lc, rc := l.newConnPair(pn) log.Debugf("%s opening connection to %s", pn.LocalPeer(), lc.RemotePeer()) pn.addConn(lc) - pn.notifyAll(func(n inet.Notifiee) { + pn.notifyAll(func(n network.Notifiee) { n.Connected(pn, lc) }) rc.net.remoteOpenedConn(rc) @@ -170,7 +172,7 @@ func (pn *peernet) remoteOpenedConn(c *conn) { log.Debugf("%s accepting connection from %s", pn.LocalPeer(), c.RemotePeer()) pn.addConn(c) pn.handleNewConn(c) - pn.notifyAll(func(n inet.Notifiee) { + pn.notifyAll(func(n network.Notifiee) { n.Connected(pn, c) }) } @@ -240,11 +242,11 @@ func (pn *peernet) Peers() []peer.ID { } // Conns returns all the connections of this peer -func (pn *peernet) Conns() []inet.Conn { +func (pn *peernet) Conns() []network.Conn { pn.RLock() defer pn.RUnlock() - out := make([]inet.Conn, 0, len(pn.connsByPeer)) + out := make([]network.Conn, 0, len(pn.connsByPeer)) for _, cs := range pn.connsByPeer { for c := range cs { out = append(out, c) @@ -253,7 +255,7 @@ func (pn *peernet) Conns() []inet.Conn { return out } -func (pn *peernet) ConnsToPeer(p peer.ID) []inet.Conn { +func (pn *peernet) ConnsToPeer(p peer.ID) []network.Conn { pn.RLock() defer pn.RUnlock() @@ -262,7 +264,7 @@ func (pn *peernet) ConnsToPeer(p peer.ID) []inet.Conn { return nil } - var cs2 []inet.Conn + var cs2 []network.Conn for c := range cs { cs2 = append(cs2, c) } @@ -298,7 +300,7 @@ func (pn *peernet) BandwidthTotals() (in uint64, out uint64) { // Listen tells the network to start listening on given multiaddrs. func (pn *peernet) Listen(addrs ...ma.Multiaddr) error { - pn.Peerstore().AddAddrs(pn.LocalPeer(), addrs, pstore.PermanentAddrTTL) + pn.Peerstore().AddAddrs(pn.LocalPeer(), addrs, peerstore.PermanentAddrTTL) return nil } @@ -316,20 +318,20 @@ func (pn *peernet) InterfaceListenAddresses() ([]ma.Multiaddr, error) { // Connectedness returns a state signaling connection capabilities // For now only returns Connecter || NotConnected. Expand into more later. -func (pn *peernet) Connectedness(p peer.ID) inet.Connectedness { +func (pn *peernet) Connectedness(p peer.ID) network.Connectedness { pn.Lock() defer pn.Unlock() cs, found := pn.connsByPeer[p] if found && len(cs) > 0 { - return inet.Connected + return network.Connected } - return inet.NotConnected + return network.NotConnected } // NewStream returns a new stream to given peer p. // If there is no connection to p, attempts to create one. -func (pn *peernet) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error) { +func (pn *peernet) NewStream(ctx context.Context, p peer.ID) (network.Stream, error) { c, err := pn.DialPeer(ctx, p) if err != nil { return nil, err @@ -339,7 +341,7 @@ func (pn *peernet) NewStream(ctx context.Context, p peer.ID) (inet.Stream, error // SetStreamHandler sets the new stream handler on the Network. // This operation is threadsafe. -func (pn *peernet) SetStreamHandler(h inet.StreamHandler) { +func (pn *peernet) SetStreamHandler(h network.StreamHandler) { pn.Lock() pn.streamHandler = h pn.Unlock() @@ -347,35 +349,35 @@ func (pn *peernet) SetStreamHandler(h inet.StreamHandler) { // SetConnHandler sets the new conn handler on the Network. // This operation is threadsafe. -func (pn *peernet) SetConnHandler(h inet.ConnHandler) { +func (pn *peernet) SetConnHandler(h network.ConnHandler) { pn.Lock() pn.connHandler = h pn.Unlock() } // Notify signs up Notifiee to receive signals when events happen -func (pn *peernet) Notify(f inet.Notifiee) { +func (pn *peernet) Notify(f network.Notifiee) { pn.notifmu.Lock() pn.notifs[f] = struct{}{} pn.notifmu.Unlock() } // StopNotify unregisters Notifiee from receiving signals -func (pn *peernet) StopNotify(f inet.Notifiee) { +func (pn *peernet) StopNotify(f network.Notifiee) { pn.notifmu.Lock() delete(pn.notifs, f) pn.notifmu.Unlock() } // notifyAll runs the notification function on all Notifiees -func (pn *peernet) notifyAll(notification func(f inet.Notifiee)) { +func (pn *peernet) notifyAll(notification func(f network.Notifiee)) { pn.notifmu.Lock() var wg sync.WaitGroup for n := range pn.notifs { // make sure we dont block // and they dont block each other. wg.Add(1) - go func(n inet.Notifiee) { + go func(n network.Notifiee) { defer wg.Done() notification(n) }(n) diff --git a/p2p/net/mock/mock_printer.go b/p2p/net/mock/mock_printer.go index 4990811a37..344560e06b 100644 --- a/p2p/net/mock/mock_printer.go +++ b/p2p/net/mock/mock_printer.go @@ -4,8 +4,8 @@ import ( "fmt" "io" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" ) // separate object so our interfaces are separate :) @@ -26,7 +26,7 @@ func (p *printer) MocknetLinks(mn Mocknet) { fmt.Fprintf(p.w, "\n") } -func (p *printer) NetworkConns(ni inet.Network) { +func (p *printer) NetworkConns(ni network.Network) { fmt.Fprintf(p.w, "%s connected to:\n", ni.LocalPeer()) for _, c := range ni.Conns() { diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index 331b8960b8..7919054884 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -8,11 +8,11 @@ import ( "sync/atomic" "time" - inet "github.com/libp2p/go-libp2p-net" - protocol "github.com/libp2p/go-libp2p-protocol" + "github.com/libp2p/go-libp2p-core/network" + protocol "github.com/libp2p/go-libp2p-core/protocol" ) -// stream implements inet.Stream +// stream implements network.Stream type stream struct { write *io.PipeWriter read *io.PipeReader @@ -26,7 +26,7 @@ type stream struct { writeErr error protocol atomic.Value - stat inet.Stat + stat network.Stat } var ErrReset error = errors.New("stream reset") @@ -37,7 +37,7 @@ type transportObject struct { arrivalTime time.Time } -func NewStream(w *io.PipeWriter, r *io.PipeReader, dir inet.Direction) *stream { +func NewStream(w *io.PipeWriter, r *io.PipeReader, dir network.Direction) *stream { s := &stream{ read: r, write: w, @@ -45,7 +45,7 @@ func NewStream(w *io.PipeWriter, r *io.PipeReader, dir inet.Direction) *stream { close: make(chan struct{}, 1), closed: make(chan struct{}), toDeliver: make(chan *transportObject), - stat: inet.Stat{Direction: dir}, + stat: network.Stat{Direction: dir}, } go s.transport() @@ -76,7 +76,7 @@ func (s *stream) Protocol() protocol.ID { return p } -func (s *stream) Stat() inet.Stat { +func (s *stream) Stat() network.Stat { return s.stat } @@ -118,12 +118,12 @@ func (s *stream) teardown() { // Mark as closed. close(s.closed) - s.conn.net.notifyAll(func(n inet.Notifiee) { + s.conn.net.notifyAll(func(n network.Notifiee) { n.ClosedStream(s.conn.net, s) }) } -func (s *stream) Conn() inet.Conn { +func (s *stream) Conn() network.Conn { return s.conn } diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index 473e4424df..3d233ce04e 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -10,15 +10,18 @@ import ( "testing" "time" + ci "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/protocol" + "github.com/libp2p/go-libp2p-core/test" + detectrace "github.com/ipfs/go-detect-race" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - protocol "github.com/libp2p/go-libp2p-protocol" - testutil "github.com/libp2p/go-testutil" + tnet "github.com/libp2p/go-libp2p-testing/net" ) func randPeer(t *testing.T) peer.ID { - p, err := testutil.RandPeerID() + p, err := test.RandPeerID() if err != nil { t.Fatal(err) } @@ -28,15 +31,15 @@ func randPeer(t *testing.T) peer.ID { func TestNetworkSetup(t *testing.T) { ctx := context.Background() - sk1, _, err := testutil.RandTestKeyPair(512) + sk1, _, err := test.RandTestKeyPair(ci.RSA, 512) if err != nil { t.Fatal(t) } - sk2, _, err := testutil.RandTestKeyPair(512) + sk2, _, err := test.RandTestKeyPair(ci.RSA, 512) if err != nil { t.Fatal(t) } - sk3, _, err := testutil.RandTestKeyPair(512) + sk3, _, err := test.RandTestKeyPair(ci.RSA, 512) if err != nil { t.Fatal(t) } @@ -45,9 +48,9 @@ func TestNetworkSetup(t *testing.T) { // add peers to mock net - a1 := testutil.RandLocalTCPAddress() - a2 := testutil.RandLocalTCPAddress() - a3 := testutil.RandLocalTCPAddress() + a1 := tnet.RandLocalTCPAddress() + a2 := tnet.RandLocalTCPAddress() + a3 := tnet.RandLocalTCPAddress() h1, err := mn.AddPeer(sk1, a1) if err != nil { @@ -278,7 +281,7 @@ func TestStreams(t *testing.T) { t.Fatal(err) } - handler := func(s inet.Stream) { + handler := func(s network.Stream) { b := make([]byte, 4) if _, err := io.ReadFull(s, b); err != nil { panic(err) @@ -315,8 +318,8 @@ func TestStreams(t *testing.T) { } -func makePinger(st string, n int) func(inet.Stream) { - return func(s inet.Stream) { +func makePinger(st string, n int) func(network.Stream) { + return func(s network.Stream) { go func() { defer s.Close() @@ -336,8 +339,8 @@ func makePinger(st string, n int) func(inet.Stream) { } } -func makePonger(st string) func(inet.Stream) { - return func(s inet.Stream) { +func makePonger(st string) func(network.Stream) { + return func(s network.Stream) { go func() { defer s.Close() @@ -409,12 +412,12 @@ func TestAdding(t *testing.T) { peers := []peer.ID{} for i := 0; i < 3; i++ { - sk, _, err := testutil.RandTestKeyPair(512) + sk, _, err := test.RandTestKeyPair(ci.RSA, 512) if err != nil { t.Fatal(err) } - a := testutil.RandLocalTCPAddress() + a := tnet.RandLocalTCPAddress() h, err := mn.AddPeer(sk, a) if err != nil { t.Fatal(err) @@ -440,7 +443,7 @@ func TestAdding(t *testing.T) { if h2 == nil { t.Fatalf("no host for %s", p2) } - h2.SetStreamHandler(protocol.TestingID, func(s inet.Stream) { + h2.SetStreamHandler(protocol.TestingID, func(s network.Stream) { defer s.Close() b := make([]byte, 4) @@ -535,7 +538,7 @@ func TestLimitedStreams(t *testing.T) { var wg sync.WaitGroup messages := 4 messageSize := 500 - handler := func(s inet.Stream) { + handler := func(s network.Stream) { b := make([]byte, messageSize) for i := 0; i < messages; i++ { if _, err := io.ReadFull(s, b); err != nil { @@ -619,7 +622,7 @@ func TestStreamsWithLatency(t *testing.T) { // we'll write once to a single stream wg.Add(1) - handler := func(s inet.Stream) { + handler := func(s network.Stream) { b := make([]byte, mln) if _, err := io.ReadFull(s, b); err != nil { diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index cfc4d08897..a3ddddaa45 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -5,16 +5,20 @@ import ( "sync" "time" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/helpers" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + pb "github.com/libp2p/go-libp2p/p2p/protocol/identify/pb" ggio "github.com/gogo/protobuf/io" logging "github.com/ipfs/go-log" - ic "github.com/libp2p/go-libp2p-crypto" - host "github.com/libp2p/go-libp2p-host" + lgbl "github.com/libp2p/go-libp2p-loggables" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + ma "github.com/multiformats/go-multiaddr" msmux "github.com/multiformats/go-multistream" ) @@ -51,7 +55,7 @@ type IDService struct { // connections undergoing identification // for wait purposes - currid map[inet.Conn]chan struct{} + currid map[network.Conn]chan struct{} currmu sync.RWMutex addrMu sync.Mutex @@ -67,7 +71,7 @@ func NewIDService(ctx context.Context, h host.Host) *IDService { s := &IDService{ Host: h, ctx: ctx, - currid: make(map[inet.Conn]chan struct{}), + currid: make(map[network.Conn]chan struct{}), observedAddrs: NewObservedAddrSet(ctx), } h.SetStreamHandler(ID, s.requestHandler) @@ -85,7 +89,7 @@ func (ids *IDService) ObservedAddrsFor(local ma.Multiaddr) []ma.Multiaddr { return ids.observedAddrs.AddrsFor(local) } -func (ids *IDService) IdentifyConn(c inet.Conn) { +func (ids *IDService) IdentifyConn(c network.Conn) { ids.currmu.Lock() if wait, found := ids.currid[c]; found { ids.currmu.Unlock() @@ -124,8 +128,8 @@ func (ids *IDService) IdentifyConn(c inet.Conn) { ids.responseHandler(s) } -func (ids *IDService) requestHandler(s inet.Stream) { - defer inet.FullClose(s) +func (ids *IDService) requestHandler(s network.Stream) { + defer helpers.FullClose(s) c := s.Conn() w := ggio.NewDelimitedWriter(s) @@ -137,7 +141,7 @@ func (ids *IDService) requestHandler(s inet.Stream) { c.RemotePeer(), c.RemoteMultiaddr()) } -func (ids *IDService) responseHandler(s inet.Stream) { +func (ids *IDService) responseHandler(s network.Stream) { c := s.Conn() r := ggio.NewDelimitedReader(s, 2048) @@ -151,10 +155,10 @@ func (ids *IDService) responseHandler(s inet.Stream) { log.Debugf("%s received message from %s %s", ID, c.RemotePeer(), c.RemoteMultiaddr()) - go inet.FullClose(s) + go helpers.FullClose(s) } -func (ids *IDService) pushHandler(s inet.Stream) { +func (ids *IDService) pushHandler(s network.Stream) { ids.responseHandler(s) } @@ -162,7 +166,7 @@ func (ids *IDService) Push() { var wg sync.WaitGroup ctx, cancel := context.WithTimeout(ids.ctx, 30*time.Second) - ctx = inet.WithNoDial(ctx, "identify push") + ctx = network.WithNoDial(ctx, "identify push") for _, p := range ids.Host.Network().Peers() { wg.Add(1) @@ -197,7 +201,7 @@ func (ids *IDService) Push() { }() } -func (ids *IDService) populateMessage(mes *pb.Identify, c inet.Conn) { +func (ids *IDService) populateMessage(mes *pb.Identify, c network.Conn) { // set protocols this node is currently handling protos := ids.Host.Mux().Protocols() @@ -246,7 +250,7 @@ func (ids *IDService) populateMessage(mes *pb.Identify, c inet.Conn) { mes.AgentVersion = &av } -func (ids *IDService) consumeMessage(mes *pb.Identify, c inet.Conn) { +func (ids *IDService) consumeMessage(mes *pb.Identify, c network.Conn) { p := c.RemotePeer() // mes.Protocols @@ -280,14 +284,14 @@ func (ids *IDService) consumeMessage(mes *pb.Identify, c inet.Conn) { // Taking the lock ensures that we don't concurrently process a disconnect. ids.addrMu.Lock() switch ids.Host.Network().Connectedness(p) { - case inet.Connected: + case network.Connected: // invalidate previous addrs -- we use a transient ttl instead of 0 to ensure there // is no period of having no good addrs whatsoever - ids.Host.Peerstore().UpdateAddrs(p, pstore.ConnectedAddrTTL, transientTTL) - ids.Host.Peerstore().AddAddrs(p, lmaddrs, pstore.ConnectedAddrTTL) + ids.Host.Peerstore().UpdateAddrs(p, peerstore.ConnectedAddrTTL, transientTTL) + ids.Host.Peerstore().AddAddrs(p, lmaddrs, peerstore.ConnectedAddrTTL) default: - ids.Host.Peerstore().UpdateAddrs(p, pstore.ConnectedAddrTTL, transientTTL) - ids.Host.Peerstore().AddAddrs(p, lmaddrs, pstore.RecentlyConnectedAddrTTL) + ids.Host.Peerstore().UpdateAddrs(p, peerstore.ConnectedAddrTTL, transientTTL) + ids.Host.Peerstore().AddAddrs(p, lmaddrs, peerstore.RecentlyConnectedAddrTTL) } ids.addrMu.Unlock() @@ -304,7 +308,7 @@ func (ids *IDService) consumeMessage(mes *pb.Identify, c inet.Conn) { ids.consumeReceivedPubKey(c, mes.PublicKey) } -func (ids *IDService) consumeReceivedPubKey(c inet.Conn, kb []byte) { +func (ids *IDService) consumeReceivedPubKey(c network.Conn, kb []byte) { lp := c.LocalPeer() rp := c.RemotePeer() @@ -412,7 +416,7 @@ func HasConsistentTransport(a ma.Multiaddr, green []ma.Multiaddr) bool { // This happens async so the connection can start to be used // even if handshake3 knowledge is not necesary. // Users **MUST** call IdentifyWait _after_ IdentifyConn -func (ids *IDService) IdentifyWait(c inet.Conn) <-chan struct{} { +func (ids *IDService) IdentifyWait(c network.Conn) <-chan struct{} { ids.currmu.Lock() ch, found := ids.currid[c] ids.currmu.Unlock() @@ -427,7 +431,7 @@ func (ids *IDService) IdentifyWait(c inet.Conn) <-chan struct{} { return ch } -func (ids *IDService) consumeObservedAddress(observed []byte, c inet.Conn) { +func (ids *IDService) consumeObservedAddress(observed []byte, c network.Conn) { if observed == nil { return } @@ -481,30 +485,30 @@ func (nn *netNotifiee) IDService() *IDService { return (*IDService)(nn) } -func (nn *netNotifiee) Connected(n inet.Network, v inet.Conn) { +func (nn *netNotifiee) Connected(n network.Network, v network.Conn) { // TODO: deprecate the setConnHandler hook, and kick off // identification here. } -func (nn *netNotifiee) Disconnected(n inet.Network, v inet.Conn) { +func (nn *netNotifiee) Disconnected(n network.Network, v network.Conn) { // undo the setting of addresses to peer.ConnectedAddrTTL we did ids := nn.IDService() ids.addrMu.Lock() defer ids.addrMu.Unlock() - if ids.Host.Network().Connectedness(v.RemotePeer()) != inet.Connected { + if ids.Host.Network().Connectedness(v.RemotePeer()) != network.Connected { // Last disconnect. ps := ids.Host.Peerstore() - ps.UpdateAddrs(v.RemotePeer(), pstore.ConnectedAddrTTL, pstore.RecentlyConnectedAddrTTL) + ps.UpdateAddrs(v.RemotePeer(), peerstore.ConnectedAddrTTL, peerstore.RecentlyConnectedAddrTTL) } } -func (nn *netNotifiee) OpenedStream(n inet.Network, v inet.Stream) {} -func (nn *netNotifiee) ClosedStream(n inet.Network, v inet.Stream) {} -func (nn *netNotifiee) Listen(n inet.Network, a ma.Multiaddr) {} -func (nn *netNotifiee) ListenClose(n inet.Network, a ma.Multiaddr) {} +func (nn *netNotifiee) OpenedStream(n network.Network, v network.Stream) {} +func (nn *netNotifiee) ClosedStream(n network.Network, v network.Stream) {} +func (nn *netNotifiee) Listen(n network.Network, a ma.Multiaddr) {} +func (nn *netNotifiee) ListenClose(n network.Network, a ma.Multiaddr) {} -func logProtocolMismatchDisconnect(c inet.Conn, protocol, agent string) { +func logProtocolMismatchDisconnect(c network.Conn, protocol, agent string) { lm := make(lgbl.DeferredMap) lm["remotePeer"] = func() interface{} { return c.RemotePeer().Pretty() } lm["remoteAddr"] = func() interface{} { return c.RemoteMultiaddr().String() } diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index fd56f28ca5..d3ff401d41 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -5,14 +5,15 @@ import ( "testing" "time" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + + blhost "github.com/libp2p/go-libp2p-blankhost" swarmt "github.com/libp2p/go-libp2p-swarm/testing" identify "github.com/libp2p/go-libp2p/p2p/protocol/identify" - blhost "github.com/libp2p/go-libp2p-blankhost" - host "github.com/libp2p/go-libp2p-host" ma "github.com/multiformats/go-multiaddr" ) @@ -34,7 +35,7 @@ func subtestIDService(t *testing.T) { forgetMe, _ := ma.NewMultiaddr("/ip4/1.2.3.4/tcp/1234") - h2.Peerstore().AddAddr(h1p, forgetMe, pstore.RecentlyConnectedAddrTTL) + h2.Peerstore().AddAddr(h1p, forgetMe, peerstore.RecentlyConnectedAddrTTL) time.Sleep(500 * time.Millisecond) h2pi := h2.Peerstore().PeerInfo(h2p) @@ -155,10 +156,10 @@ func testHasPublicKey(t *testing.T, h host.Host, p peer.ID, shouldBe ic.PubKey) // this is becasue it used to be concurrent. Now, Dial wait till the // id service is done. func TestIDService(t *testing.T) { - oldTTL := pstore.RecentlyConnectedAddrTTL - pstore.RecentlyConnectedAddrTTL = time.Second + oldTTL := peerstore.RecentlyConnectedAddrTTL + peerstore.RecentlyConnectedAddrTTL = time.Second defer func() { - pstore.RecentlyConnectedAddrTTL = oldTTL + peerstore.RecentlyConnectedAddrTTL = oldTTL }() N := 3 diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 685e5f10b5..12e22860dc 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -5,8 +5,9 @@ import ( "sync" "time" - net "github.com/libp2p/go-libp2p-net" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peerstore" + ma "github.com/multiformats/go-multiaddr" ) @@ -16,7 +17,7 @@ var GCInterval = 10 * time.Minute type observation struct { seenTime time.Time - connDirection net.Direction + connDirection network.Direction } // ObservedAddr is an entry for an address reported by our peers. @@ -38,7 +39,7 @@ func (oa *ObservedAddr) activated(ttl time.Duration) bool { type newObservation struct { observed, local, observer ma.Multiaddr - direction net.Direction + direction network.Direction } // ObservedAddrSet keeps track of a set of ObservedAddrs @@ -57,7 +58,7 @@ type ObservedAddrSet struct { func NewObservedAddrSet(ctx context.Context) *ObservedAddrSet { oas := &ObservedAddrSet{ addrs: make(map[string][]*ObservedAddr), - ttl: pstore.OwnObservedAddrTTL, + ttl: peerstore.OwnObservedAddrTTL, wch: make(chan newObservation, 16), } go oas.worker(ctx) @@ -111,7 +112,7 @@ func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { } func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, - direction net.Direction) { + direction network.Direction) { select { case oas.wch <- newObservation{observed: observed, local: local, observer: observer, direction: direction}: default: @@ -168,7 +169,7 @@ func (oas *ObservedAddrSet) gc() { } func (oas *ObservedAddrSet) doAdd(observed, local, observer ma.Multiaddr, - direction net.Direction) { + direction network.Direction) { now := time.Now() observerString := observerGroup(observer) diff --git a/p2p/protocol/identify/obsaddr_test.go b/p2p/protocol/identify/obsaddr_test.go index 440b009b46..3a42a4c000 100644 --- a/p2p/protocol/identify/obsaddr_test.go +++ b/p2p/protocol/identify/obsaddr_test.go @@ -7,7 +7,7 @@ import ( "time" detectrace "github.com/ipfs/go-detect-race" - net "github.com/libp2p/go-libp2p-net" + net "github.com/libp2p/go-libp2p-core/network" ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/protocol/ping/ping.go b/p2p/protocol/ping/ping.go index d8e143bd07..9d9b571eae 100644 --- a/p2p/protocol/ping/ping.go +++ b/p2p/protocol/ping/ping.go @@ -9,9 +9,9 @@ import ( u "github.com/ipfs/go-ipfs-util" logging "github.com/ipfs/go-log" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" ) var log = logging.Logger("ping") @@ -32,7 +32,7 @@ func NewPingService(h host.Host) *PingService { return ps } -func (p *PingService) PingHandler(s inet.Stream) { +func (p *PingService) PingHandler(s network.Stream) { buf := make([]byte, PingSize) errCh := make(chan error, 1) @@ -129,7 +129,7 @@ func Ping(ctx context.Context, h host.Host, p peer.ID) <-chan Result { return out } -func ping(s inet.Stream) (time.Duration, error) { +func ping(s network.Stream) (time.Duration, error) { buf := make([]byte, PingSize) u.NewTimeSeededRand().Read(buf) diff --git a/p2p/protocol/ping/ping_test.go b/p2p/protocol/ping/ping_test.go index 043eac0fd1..942fe13252 100644 --- a/p2p/protocol/ping/ping_test.go +++ b/p2p/protocol/ping/ping_test.go @@ -5,8 +5,8 @@ import ( "testing" "time" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + "github.com/libp2p/go-libp2p-core/peer" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" ping "github.com/libp2p/go-libp2p/p2p/protocol/ping" @@ -18,7 +18,7 @@ func TestPing(t *testing.T) { h1 := bhost.New(swarmt.GenSwarm(t, ctx)) h2 := bhost.New(swarmt.GenSwarm(t, ctx)) - err := h1.Connect(ctx, pstore.PeerInfo{ + err := h1.Connect(ctx, peer.AddrInfo{ ID: h2.ID(), Addrs: h2.Addrs(), }) diff --git a/p2p/test/backpressure/backpressure_test.go b/p2p/test/backpressure/backpressure_test.go index 3d32d30885..92c8f20bcf 100644 --- a/p2p/test/backpressure/backpressure_test.go +++ b/p2p/test/backpressure/backpressure_test.go @@ -11,10 +11,10 @@ import ( u "github.com/ipfs/go-ipfs-util" logging "github.com/ipfs/go-log" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - protocol "github.com/libp2p/go-libp2p-protocol" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + protocol "github.com/libp2p/go-libp2p-core/protocol" swarmt "github.com/libp2p/go-libp2p-swarm/testing" ) @@ -59,7 +59,7 @@ a problem. senderDone := make(chan struct{}) // the receiver handles requests with some rate limiting - receiver := func(s inet.Stream) { + receiver := func(s network.Stream) { log.Debug("receiver received a stream") <-receiverRatelimit // acquire @@ -76,7 +76,7 @@ a problem. // the sender opens streams as fast as possible sender := func(host host.Host, remote peer.ID) { - var s inet.Stream + var s network.Stream var err error defer func() { t.Error(err) @@ -230,7 +230,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { // completion of every write. This makes it possible to see how // fast it's actually writing. We pair this with a receiver // that waits for a signal to read. - sender := func(s inet.Stream) { + sender := func(s network.Stream) { defer func() { s.Close() senderDone <- struct{}{} @@ -263,7 +263,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { // receive a number of bytes from a stream. // returns the number of bytes written. - receive := func(s inet.Stream, expect int) { + receive := func(s network.Stream, expect int) { t.Helper() log.Debugf("receiver to read %d bytes", expect) rbuf := make([]byte, expect) diff --git a/p2p/test/reconnects/reconnect_test.go b/p2p/test/reconnects/reconnect_test.go index 2f721c3609..e1d969ffaa 100644 --- a/p2p/test/reconnects/reconnect_test.go +++ b/p2p/test/reconnects/reconnect_test.go @@ -12,15 +12,15 @@ import ( u "github.com/ipfs/go-ipfs-util" logging "github.com/ipfs/go-log" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - protocol "github.com/libp2p/go-libp2p-protocol" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + protocol "github.com/libp2p/go-libp2p-core/protocol" swarmt "github.com/libp2p/go-libp2p-swarm/testing" ) var log = logging.Logger("reconnect") -func EchoStreamHandler(stream inet.Stream) { +func EchoStreamHandler(stream network.Stream) { c := stream.Conn() log.Debugf("%s echoing %s", c.LocalPeer(), c.RemotePeer()) go func() { @@ -51,9 +51,9 @@ func newSendChans() sendChans { } } -func newSender() (chan sendChans, func(s inet.Stream)) { +func newSender() (chan sendChans, func(s network.Stream)) { scc := make(chan sendChans) - return scc, func(s inet.Stream) { + return scc, func(s network.Stream) { sc := newSendChans() scc <- sc From 9ad01ee912607d192a9956b2d6f3c2f09f7bd110 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 26 May 2019 23:40:01 +0100 Subject: [PATCH 1276/3965] migrate to consolidated types. (#21) --- p2p/host/autonat/svc.go | 21 +++++++++++---------- p2p/host/autonat/svc_test.go | 11 ++++++----- 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index febae7a407..a685f3d36d 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -5,15 +5,16 @@ import ( "sync" "time" + "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p-core/helpers" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + pb "github.com/libp2p/go-libp2p-autonat/pb" ggio "github.com/gogo/protobuf/io" - libp2p "github.com/libp2p/go-libp2p" autonat "github.com/libp2p/go-libp2p-autonat" - host "github.com/libp2p/go-libp2p-host" - inet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) @@ -57,13 +58,13 @@ func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) return as, nil } -func (as *AutoNATService) handleStream(s inet.Stream) { - defer inet.FullClose(s) +func (as *AutoNATService) handleStream(s network.Stream) { + defer helpers.FullClose(s) pid := s.Conn().RemotePeer() log.Debugf("New stream from %s", pid.Pretty()) - r := ggio.NewDelimitedReader(s, inet.MessageSizeMax) + r := ggio.NewDelimitedReader(s, network.MessageSizeMax) w := ggio.NewDelimitedWriter(s) var req pb.Message @@ -146,7 +147,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me return newDialResponseError(pb.Message_E_DIAL_ERROR, "no dialable addresses") } - return as.doDial(pstore.PeerInfo{ID: p, Addrs: addrs}) + return as.doDial(peer.AddrInfo{ID: p, Addrs: addrs}) } func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { @@ -164,7 +165,7 @@ func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { return false } -func (as *AutoNATService) doDial(pi pstore.PeerInfo) *pb.Message_DialResponse { +func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { // rate limit check as.mx.Lock() count := as.reqs[pi.ID] diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index c0a676dece..9ef9ec0197 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -6,10 +6,11 @@ import ( "testing" "time" - libp2p "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" + autonat "github.com/libp2p/go-libp2p-autonat" - host "github.com/libp2p/go-libp2p-host" - pstore "github.com/libp2p/go-libp2p-peerstore" manet "github.com/multiformats/go-multiaddr-net" ) @@ -38,14 +39,14 @@ func makeAutoNATClient(ctx context.Context, t *testing.T) (host.Host, autonat.Au } func connect(t *testing.T, a, b host.Host) { - pinfo := pstore.PeerInfo{ID: a.ID(), Addrs: a.Addrs()} + pinfo := peer.AddrInfo{ID: a.ID(), Addrs: a.Addrs()} err := b.Connect(context.Background(), pinfo) if err != nil { t.Fatal(err) } } -// Note: these tests assume that the host has only private inet addresses! +// Note: these tests assume that the host has only private network addresses! func TestAutoNATServiceDialError(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() From 9e002de8e0bfd7118deb3a446e7321f381a40d3c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 27 May 2019 00:22:15 +0100 Subject: [PATCH 1277/3965] migrate to consolidated types. --- p2p/net/gostream/addr.go | 2 +- p2p/net/gostream/conn.go | 15 ++++++++------- p2p/net/gostream/gostream_test.go | 9 +++++---- p2p/net/gostream/listener.go | 12 ++++++------ 4 files changed, 20 insertions(+), 18 deletions(-) diff --git a/p2p/net/gostream/addr.go b/p2p/net/gostream/addr.go index db3b25c50f..9546c0d722 100644 --- a/p2p/net/gostream/addr.go +++ b/p2p/net/gostream/addr.go @@ -1,6 +1,6 @@ package gostream -import peer "github.com/libp2p/go-libp2p-peer" +import "github.com/libp2p/go-libp2p-core/peer" // addr implements net.Addr and holds a libp2p peer ID. type addr struct{ id peer.ID } diff --git a/p2p/net/gostream/conn.go b/p2p/net/gostream/conn.go index dc5050ffaa..8ecc24b7e8 100644 --- a/p2p/net/gostream/conn.go +++ b/p2p/net/gostream/conn.go @@ -5,20 +5,21 @@ import ( "net" "time" - host "github.com/libp2p/go-libp2p-host" - pnet "github.com/libp2p/go-libp2p-net" - peer "github.com/libp2p/go-libp2p-peer" - protocol "github.com/libp2p/go-libp2p-protocol" + "github.com/libp2p/go-libp2p-core/helpers" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/protocol" ) // conn is an implementation of net.Conn which wraps // libp2p streams. type conn struct { - s pnet.Stream + s network.Stream } // newConn creates a conn given a libp2p stream -func newConn(s pnet.Stream) net.Conn { +func newConn(s network.Stream) net.Conn { return &conn{s} } @@ -39,7 +40,7 @@ func (c *conn) Close() error { c.s.Reset() return err } - go pnet.AwaitEOF(c.s) + go helpers.AwaitEOF(c.s) return nil } diff --git a/p2p/net/gostream/gostream_test.go b/p2p/net/gostream/gostream_test.go index cdbd5be124..b74a85a474 100644 --- a/p2p/net/gostream/gostream_test.go +++ b/p2p/net/gostream/gostream_test.go @@ -6,10 +6,11 @@ import ( "testing" "time" - libp2p "github.com/libp2p/go-libp2p" - host "github.com/libp2p/go-libp2p-host" - peerstore "github.com/libp2p/go-libp2p-peerstore" - protocol "github.com/libp2p/go-libp2p-protocol" + "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/protocol" + multiaddr "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/net/gostream/listener.go b/p2p/net/gostream/listener.go index e66c262a70..48e6cffb31 100644 --- a/p2p/net/gostream/listener.go +++ b/p2p/net/gostream/listener.go @@ -4,9 +4,9 @@ import ( "context" "net" - host "github.com/libp2p/go-libp2p-host" - pnet "github.com/libp2p/go-libp2p-net" - protocol "github.com/libp2p/go-libp2p-protocol" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/protocol" ) // listener is an implementation of net.Listener which handles @@ -17,7 +17,7 @@ type listener struct { ctx context.Context tag protocol.ID cancel func() - streamCh chan pnet.Stream + streamCh chan network.Stream } // Accept returns the next a connection to this listener. @@ -56,10 +56,10 @@ func Listen(h host.Host, tag protocol.ID) (net.Listener, error) { ctx: ctx, cancel: cancel, tag: tag, - streamCh: make(chan pnet.Stream), + streamCh: make(chan network.Stream), } - h.SetStreamHandler(tag, func(s pnet.Stream) { + h.SetStreamHandler(tag, func(s network.Stream) { select { case l.streamCh <- s: case <-ctx.Done(): From c37e733d403f0261be74d9fb8d9f8fc42368397e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 27 May 2019 12:00:30 +0100 Subject: [PATCH 1278/3965] migrate to consolidated types. (#30) --- p2p/security/tls/cmd/tlsdiag/client.go | 2 +- p2p/security/tls/cmd/tlsdiag/key.go | 2 +- p2p/security/tls/cmd/tlsdiag/server.go | 2 +- p2p/security/tls/conn.go | 8 ++++---- p2p/security/tls/crypto.go | 8 ++++---- p2p/security/tls/transport.go | 16 ++++++++-------- p2p/security/tls/transport_test.go | 11 ++++++----- 7 files changed, 25 insertions(+), 24 deletions(-) diff --git a/p2p/security/tls/cmd/tlsdiag/client.go b/p2p/security/tls/cmd/tlsdiag/client.go index 21e4d18098..8b0fc46fb0 100644 --- a/p2p/security/tls/cmd/tlsdiag/client.go +++ b/p2p/security/tls/cmd/tlsdiag/client.go @@ -8,7 +8,7 @@ import ( "net" "time" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" libp2ptls "github.com/libp2p/go-libp2p-tls" ) diff --git a/p2p/security/tls/cmd/tlsdiag/key.go b/p2p/security/tls/cmd/tlsdiag/key.go index 038974225a..557a485c35 100644 --- a/p2p/security/tls/cmd/tlsdiag/key.go +++ b/p2p/security/tls/cmd/tlsdiag/key.go @@ -4,7 +4,7 @@ import ( "crypto/rand" "fmt" - ic "github.com/libp2p/go-libp2p-crypto" + ic "github.com/libp2p/go-libp2p-core/crypto" ) func generateKey(keyType string) (priv ic.PrivKey, err error) { diff --git a/p2p/security/tls/cmd/tlsdiag/server.go b/p2p/security/tls/cmd/tlsdiag/server.go index 0a895baf12..290af13d63 100644 --- a/p2p/security/tls/cmd/tlsdiag/server.go +++ b/p2p/security/tls/cmd/tlsdiag/server.go @@ -7,7 +7,7 @@ import ( "net" "time" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" libp2ptls "github.com/libp2p/go-libp2p-tls" ) diff --git a/p2p/security/tls/conn.go b/p2p/security/tls/conn.go index d5450b1c6d..cf32fa459d 100644 --- a/p2p/security/tls/conn.go +++ b/p2p/security/tls/conn.go @@ -3,9 +3,9 @@ package libp2ptls import ( "crypto/tls" - cs "github.com/libp2p/go-conn-security" - ci "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" + ci "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" ) type conn struct { @@ -18,7 +18,7 @@ type conn struct { remotePubKey ci.PubKey } -var _ cs.Conn = &conn{} +var _ sec.SecureConn = &conn{} func (c *conn) LocalPeer() peer.ID { return c.localPeer diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 50c9d71a43..917c65a666 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -13,11 +13,11 @@ import ( "math/big" "time" + crypto "github.com/libp2p/go-libp2p-crypto" "golang.org/x/sys/cpu" - crypto "github.com/libp2p/go-libp2p-crypto" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" ) const certValidityPeriod = 100 * 365 * 24 * time.Hour // ~100 years @@ -133,7 +133,7 @@ func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { if _, err := asn1.Unmarshal(keyExt.Value, &sk); err != nil { return nil, fmt.Errorf("unmarshalling signed certificate failed: %s", err) } - pubKey, err := crypto.UnmarshalPublicKey(sk.PubKey) + pubKey, err := ic.UnmarshalPublicKey(sk.PubKey) if err != nil { return nil, fmt.Errorf("unmarshalling public key failed: %s", err) } diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 8b3e132f6b..2dc4257a3e 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -7,9 +7,9 @@ import ( "net" "os" - cs "github.com/libp2p/go-conn-security" - ci "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" + ci "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" ) // TLS 1.3 is opt-in in Go 1.12 @@ -48,10 +48,10 @@ func New(key ci.PrivKey) (*Transport, error) { return t, nil } -var _ cs.Transport = &Transport{} +var _ sec.SecureTransport = &Transport{} // SecureInbound runs the TLS handshake as a server. -func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Conn, error) { +func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { config, keyCh := t.identity.ConfigForAny() return t.handshake(ctx, tls.Server(insecure, config), keyCh) } @@ -63,7 +63,7 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (cs.Co // application data immediately afterwards. // If the handshake fails, the server will close the connection. The client will // notice this after 1 RTT when calling Read. -func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (cs.Conn, error) { +func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { config, keyCh := t.identity.ConfigForPeer(p) return t.handshake(ctx, tls.Client(insecure, config), keyCh) } @@ -72,7 +72,7 @@ func (t *Transport) handshake( ctx context.Context, tlsConn *tls.Conn, keyCh <-chan ci.PubKey, -) (cs.Conn, error) { +) (sec.SecureConn, error) { // There's no way to pass a context to tls.Conn.Handshake(). // See https://github.com/golang/go/issues/18482. // Close the connection instead. @@ -117,7 +117,7 @@ func (t *Transport) handshake( return conn, nil } -func (t *Transport) setupConn(tlsConn *tls.Conn, remotePubKey ci.PubKey) (cs.Conn, error) { +func (t *Transport) setupConn(tlsConn *tls.Conn, remotePubKey ci.PubKey) (sec.SecureConn, error) { if remotePubKey == nil { return nil, errors.New("go-libp2p-tls BUG: expected remote pub key to be set") } diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 82944560e4..94f2c215f0 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -20,9 +20,10 @@ import ( "github.com/onsi/gomega/gbytes" "github.com/onsi/gomega/types" - cs "github.com/libp2p/go-conn-security" - ci "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" + ci "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" + . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) @@ -94,7 +95,7 @@ var _ = Describe("Transport", func() { clientInsecureConn, serverInsecureConn := connect() - serverConnChan := make(chan cs.Conn) + serverConnChan := make(chan sec.SecureConn) go func() { defer GinkgoRecover() serverConn, err := serverTransport.SecureInbound(context.Background(), serverInsecureConn) @@ -103,7 +104,7 @@ var _ = Describe("Transport", func() { }() clientConn, err := clientTransport.SecureOutbound(context.Background(), clientInsecureConn, serverID) Expect(err).ToNot(HaveOccurred()) - var serverConn cs.Conn + var serverConn sec.SecureConn Eventually(serverConnChan).Should(Receive(&serverConn)) defer clientConn.Close() defer serverConn.Close() From a862d91790813f186821ae5d6f80202e934c3609 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 27 May 2019 14:35:05 -0700 Subject: [PATCH 1279/3965] feat: improve protocol interning This used to specify a max size/count but that was only because this logic was shared between all peerstores (including the on-disk one). Now that: 1. This only applies to the in-memory one. 2. We never actually GC these. We can just intern indefinitely. --- p2p/host/peerstore/pstoremem/protobook.go | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index 04cd14591c..3a5550d2b6 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -8,11 +8,6 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" ) -const ( - maxInternedProtocols = 512 - maxInternedProtocolSize = 256 -) - type protoSegment struct { sync.RWMutex protocols map[peer.ID]map[string]struct{} @@ -35,7 +30,7 @@ var _ pstore.ProtoBook = (*memoryProtoBook)(nil) func NewProtoBook() pstore.ProtoBook { return &memoryProtoBook{ - interned: make(map[string]string, maxInternedProtocols), + interned: make(map[string]string, 256), segments: func() (ret protoSegments) { for i := range ret { ret[i] = &protoSegment{ @@ -48,10 +43,6 @@ func NewProtoBook() pstore.ProtoBook { } func (pb *memoryProtoBook) internProtocol(proto string) string { - if len(proto) > maxInternedProtocolSize { - return proto - } - // check if it is interned with the read lock pb.lk.RLock() interned, ok := pb.interned[proto] @@ -71,11 +62,6 @@ func (pb *memoryProtoBook) internProtocol(proto string) string { return interned } - // if we've filled the table, throw it away and start over - if len(pb.interned) >= maxInternedProtocols { - pb.interned = make(map[string]string, maxInternedProtocols) - } - pb.interned[proto] = proto return proto } From 5197d8f344f14ead84c85b4ffc3a6c4bb7d520af Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 27 May 2019 14:41:29 -0700 Subject: [PATCH 1280/3965] feat: intern agent versions This also starts taking up a bunch of memory after a while. --- p2p/host/peerstore/pstoremem/metadata.go | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index 162bee5087..9adebd8ae9 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -7,6 +7,11 @@ import ( pstore "github.com/libp2p/go-libp2p-peerstore" ) +var internKeys = map[string]bool{ + "AgentVersion": true, + "ProtocolVersion": true, +} + type metakey struct { id peer.ID key string @@ -15,8 +20,9 @@ type metakey struct { type memoryPeerMetadata struct { // store other data, like versions //ds ds.ThreadSafeDatastore - ds map[metakey]interface{} - dslock sync.RWMutex + ds map[metakey]interface{} + dslock sync.RWMutex + interned map[string]interface{} } var _ pstore.PeerMetadata = (*memoryPeerMetadata)(nil) @@ -30,6 +36,13 @@ func NewPeerMetadata() pstore.PeerMetadata { func (ps *memoryPeerMetadata) Put(p peer.ID, key string, val interface{}) error { ps.dslock.Lock() defer ps.dslock.Unlock() + if vals, ok := val.(string); ok && internKeys[key] { + if interned, ok := ps.interned[vals]; ok { + val = interned + } else { + ps.interned[vals] = val + } + } ps.ds[metakey{p, key}] = val return nil } From 6d1358157427c89d53eed795a1cabde7c6ef922c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 30 May 2019 21:57:09 -0700 Subject: [PATCH 1281/3965] chore: cleanup address parsing * Use SplitLast to avoid allocating too much. * Avoid converting to/from strings when working with multiaddrs. --- core/peer/addrinfo.go | 43 ++++++++++++------------------------------- 1 file changed, 12 insertions(+), 31 deletions(-) diff --git a/core/peer/addrinfo.go b/core/peer/addrinfo.go index c7324e4f3e..56888787e2 100644 --- a/core/peer/addrinfo.go +++ b/core/peer/addrinfo.go @@ -2,7 +2,6 @@ package peer import ( "fmt" - "strings" ma "github.com/multiformats/go-multiaddr" ) @@ -27,47 +26,29 @@ func AddrInfoFromP2pAddr(m ma.Multiaddr) (*AddrInfo, error) { return nil, ErrInvalidAddr } - // make sure it's a P2P addr - parts := ma.Split(m) - if len(parts) < 1 { + transport, p2ppart := ma.SplitLast(m) + if p2ppart == nil || p2ppart.Protocol().Code != ma.P_P2P { return nil, ErrInvalidAddr } - - // TODO(lgierth): we shouldn't assume /p2p is the last part - p2ppart := parts[len(parts)-1] - if p2ppart.Protocols()[0].Code != ma.P_P2P { - return nil, ErrInvalidAddr - } - - // make sure the /p2p value parses as a peer.ID - peerIdParts := strings.Split(p2ppart.String(), "/") - peerIdStr := peerIdParts[len(peerIdParts)-1] - id, err := IDB58Decode(peerIdStr) + id, err := IDFromBytes(p2ppart.RawValue()) if err != nil { return nil, err } - - // we might have received just an /p2p part, which means there's no addr. - var addrs []ma.Multiaddr - if len(parts) > 1 { - addrs = append(addrs, ma.Join(parts[:len(parts)-1]...)) + info := &AddrInfo{ID: id} + if transport != nil { + info.Addrs = []ma.Multiaddr{transport} } - - return &AddrInfo{ - ID: id, - Addrs: addrs, - }, nil + return info, nil } func AddrInfoToP2pAddrs(pi *AddrInfo) ([]ma.Multiaddr, error) { var addrs []ma.Multiaddr - tpl := "/" + ma.ProtocolWithCode(ma.P_P2P).Name + "/" + p2ppart, err := ma.NewComponent("p2p", IDB58Encode(pi.ID)) + if err != nil { + return nil, err + } for _, addr := range pi.Addrs { - p2paddr, err := ma.NewMultiaddr(tpl + IDB58Encode(pi.ID)) - if err != nil { - return nil, err - } - addrs = append(addrs, addr.Encapsulate(p2paddr)) + addrs = append(addrs, addr.Encapsulate(p2ppart)) } return addrs, nil } From f914c757b1b354dfc618101c5efccb938c63c15b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 30 May 2019 22:25:37 -0700 Subject: [PATCH 1282/3965] fix: handle empty addrs case --- core/peer/addrinfo.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/core/peer/addrinfo.go b/core/peer/addrinfo.go index 56888787e2..cc9633b7b0 100644 --- a/core/peer/addrinfo.go +++ b/core/peer/addrinfo.go @@ -47,6 +47,9 @@ func AddrInfoToP2pAddrs(pi *AddrInfo) ([]ma.Multiaddr, error) { if err != nil { return nil, err } + if len(pi.Addrs) == 0 { + return []ma.Multiaddr{p2ppart}, nil + } for _, addr := range pi.Addrs { addrs = append(addrs, addr.Encapsulate(p2ppart)) } From 73b10b9adfdd0027111c101a7cff18c90d7a0d6e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 31 May 2019 06:51:16 -0700 Subject: [PATCH 1283/3965] dep: switch to core (#80) * dep: switch to core Specifically, remove go-libp2p-{crypto,peer} deps. * catch a few more usages of deprecated packages. --- p2p/host/peerstore/metrics.go | 6 ++--- p2p/host/peerstore/metrics_test.go | 6 ++--- p2p/host/peerstore/pb/custom.go | 4 +-- p2p/host/peerstore/peerstore.go | 27 ++++++++++--------- p2p/host/peerstore/pstoreds/addr_book.go | 4 +-- p2p/host/peerstore/pstoreds/addr_book_gc.go | 2 +- .../peerstore/pstoreds/addr_book_gc_test.go | 2 +- p2p/host/peerstore/pstoreds/ds_test.go | 2 +- p2p/host/peerstore/pstoreds/keybook.go | 6 ++--- p2p/host/peerstore/pstoreds/metadata.go | 4 +-- p2p/host/peerstore/pstoreds/peerstore.go | 5 ++-- p2p/host/peerstore/pstoreds/protobook.go | 4 +-- p2p/host/peerstore/pstoremem/addr_book.go | 4 +-- p2p/host/peerstore/pstoremem/inmem_test.go | 2 +- p2p/host/peerstore/pstoremem/keybook.go | 6 ++--- p2p/host/peerstore/pstoremem/metadata.go | 4 +-- p2p/host/peerstore/pstoremem/protobook.go | 4 +-- p2p/host/peerstore/queue/distance.go | 2 +- p2p/host/peerstore/queue/interface.go | 2 +- p2p/host/peerstore/queue/queue_test.go | 2 +- p2p/host/peerstore/queue/sync.go | 2 +- p2p/host/peerstore/test/addr_book_suite.go | 2 +- p2p/host/peerstore/test/benchmarks_suite.go | 2 +- p2p/host/peerstore/test/keybook_suite.go | 26 +++++++++--------- p2p/host/peerstore/test/peerstore_suite.go | 6 ++--- p2p/host/peerstore/test/utils.go | 4 +-- 26 files changed, 71 insertions(+), 69 deletions(-) diff --git a/p2p/host/peerstore/metrics.go b/p2p/host/peerstore/metrics.go index f6d0ccb56e..5edfbd48db 100644 --- a/p2p/host/peerstore/metrics.go +++ b/p2p/host/peerstore/metrics.go @@ -4,8 +4,8 @@ import ( "sync" "time" - moved "github.com/libp2p/go-libp2p-core/peerstore" - "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" + core "github.com/libp2p/go-libp2p-core/peerstore" ) // LatencyEWMASmooting governs the decay of the EWMA (the speed @@ -14,7 +14,7 @@ import ( var LatencyEWMASmoothing = 0.1 // Deprecated: use github.com/libp2p/go-libp2p-core/peerstore.Metrics instead. -type Metrics = moved.Metrics +type Metrics = core.Metrics type metrics struct { latmap map[peer.ID]time.Duration diff --git a/p2p/host/peerstore/metrics_test.go b/p2p/host/peerstore/metrics_test.go index 2010db3089..8686d06715 100644 --- a/p2p/host/peerstore/metrics_test.go +++ b/p2p/host/peerstore/metrics_test.go @@ -7,14 +7,14 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-peer/test" + "github.com/libp2p/go-libp2p-core/test" ) func TestLatencyEWMAFun(t *testing.T) { t.Skip("run it for fun") m := NewMetrics() - id, err := testutil.RandPeerID() + id, err := test.RandPeerID() if err != nil { t.Fatal(err) } @@ -41,7 +41,7 @@ func TestLatencyEWMAFun(t *testing.T) { func TestLatencyEWMA(t *testing.T) { m := NewMetrics() - id, err := testutil.RandPeerID() + id, err := test.RandPeerID() if err != nil { t.Fatal(err) } diff --git a/p2p/host/peerstore/pb/custom.go b/p2p/host/peerstore/pb/custom.go index 2ffcc32bb6..41784a4f6d 100644 --- a/p2p/host/peerstore/pb/custom.go +++ b/p2p/host/peerstore/pb/custom.go @@ -4,8 +4,8 @@ import ( "encoding/json" proto "github.com/gogo/protobuf/proto" - peer "github.com/libp2p/go-libp2p-peer" - pt "github.com/libp2p/go-libp2p-peer/test" + peer "github.com/libp2p/go-libp2p-core/peer" + pt "github.com/libp2p/go-libp2p-core/test" ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index ea29a17fbb..3331b4a796 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -4,23 +4,24 @@ import ( "fmt" "io" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) -var _ Peerstore = (*peerstore)(nil) +var _ pstore.Peerstore = (*peerstore)(nil) type peerstore struct { - Metrics + pstore.Metrics - KeyBook - AddrBook - ProtoBook - PeerMetadata + pstore.KeyBook + pstore.AddrBook + pstore.ProtoBook + pstore.PeerMetadata } // NewPeerstore creates a data structure that stores peer data, backed by the // supplied implementations of KeyBook, AddrBook and PeerMetadata. -func NewPeerstore(kb KeyBook, ab AddrBook, pb ProtoBook, md PeerMetadata) Peerstore { +func NewPeerstore(kb pstore.KeyBook, ab pstore.AddrBook, pb pstore.ProtoBook, md pstore.PeerMetadata) pstore.Peerstore { return &peerstore{ KeyBook: kb, AddrBook: ab, @@ -67,22 +68,22 @@ func (ps *peerstore) Peers() peer.IDSlice { return pps } -func (ps *peerstore) PeerInfo(p peer.ID) PeerInfo { - return PeerInfo{ +func (ps *peerstore) PeerInfo(p peer.ID) peer.AddrInfo { + return peer.AddrInfo{ ID: p, Addrs: ps.AddrBook.Addrs(p), } } -func PeerInfos(ps Peerstore, peers peer.IDSlice) []PeerInfo { - pi := make([]PeerInfo, len(peers)) +func PeerInfos(ps pstore.Peerstore, peers peer.IDSlice) []peer.AddrInfo { + pi := make([]peer.AddrInfo, len(peers)) for i, p := range peers { pi[i] = ps.PeerInfo(p) } return pi } -func PeerInfoIDs(pis []PeerInfo) peer.IDSlice { +func PeerInfoIDs(pis []peer.AddrInfo) peer.IDSlice { ps := make(peer.IDSlice, len(pis)) for i, pi := range pis { ps[i] = pi.ID diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 0eacacf344..bba648e845 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -11,8 +11,8 @@ import ( query "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + peer "github.com/libp2p/go-libp2p-core/peer" + pstore "github.com/libp2p/go-libp2p-core/peerstore" pb "github.com/libp2p/go-libp2p-peerstore/pb" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 8c66025439..6ff1a0f05e 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -9,7 +9,7 @@ import ( ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" - peer "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-core/peer" pb "github.com/libp2p/go-libp2p-peerstore/pb" b32 "github.com/multiformats/go-base32" diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc_test.go b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go index 2c328eebc8..e21f43f23a 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc_test.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc_test.go @@ -5,7 +5,7 @@ import ( "time" query "github.com/ipfs/go-datastore/query" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" test "github.com/libp2p/go-libp2p-peerstore/test" ma "github.com/multiformats/go-multiaddr" ) diff --git a/p2p/host/peerstore/pstoreds/ds_test.go b/p2p/host/peerstore/pstoreds/ds_test.go index 3ae7ef7900..1264ee7813 100644 --- a/p2p/host/peerstore/pstoreds/ds_test.go +++ b/p2p/host/peerstore/pstoreds/ds_test.go @@ -11,7 +11,7 @@ import ( badger "github.com/ipfs/go-ds-badger" leveldb "github.com/ipfs/go-ds-leveldb" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" pt "github.com/libp2p/go-libp2p-peerstore/test" ) diff --git a/p2p/host/peerstore/pstoreds/keybook.go b/p2p/host/peerstore/pstoreds/keybook.go index 29885c5da2..64b73a3024 100644 --- a/p2p/host/peerstore/pstoreds/keybook.go +++ b/p2p/host/peerstore/pstoreds/keybook.go @@ -9,9 +9,9 @@ import ( ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + ic "github.com/libp2p/go-libp2p-core/crypto" + peer "github.com/libp2p/go-libp2p-core/peer" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) // Public and private keys are stored under the following db key pattern: diff --git a/p2p/host/peerstore/pstoreds/metadata.go b/p2p/host/peerstore/pstoreds/metadata.go index dca550cbe3..f54f382ba7 100644 --- a/p2p/host/peerstore/pstoreds/metadata.go +++ b/p2p/host/peerstore/pstoreds/metadata.go @@ -10,8 +10,8 @@ import ( ds "github.com/ipfs/go-datastore" pool "github.com/libp2p/go-buffer-pool" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + peer "github.com/libp2p/go-libp2p-core/peer" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) // Metadata is stored under the following db key pattern: diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index b4560f7363..99b381db2f 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -9,7 +9,8 @@ import ( ds "github.com/ipfs/go-datastore" query "github.com/ipfs/go-datastore/query" - peer "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-core/peer" + peerstore "github.com/libp2p/go-libp2p-core/peerstore" pstore "github.com/libp2p/go-libp2p-peerstore" ) @@ -47,7 +48,7 @@ func DefaultOpts() Options { } // NewPeerstore creates a peerstore backed by the provided persistent datastore. -func NewPeerstore(ctx context.Context, store ds.Batching, opts Options) (pstore.Peerstore, error) { +func NewPeerstore(ctx context.Context, store ds.Batching, opts Options) (peerstore.Peerstore, error) { addrBook, err := NewAddrBook(ctx, store, opts) if err != nil { return nil, err diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go index 9d86427b6d..f4349b6503 100644 --- a/p2p/host/peerstore/pstoreds/protobook.go +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -4,9 +4,9 @@ import ( "fmt" "sync" - peer "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-core/peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) type protoSegment struct { diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index a497821d05..1a4bccace3 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -7,10 +7,10 @@ import ( "time" logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" addr "github.com/libp2p/go-libp2p-peerstore/addr" ) diff --git a/p2p/host/peerstore/pstoremem/inmem_test.go b/p2p/host/peerstore/pstoremem/inmem_test.go index 97e24e4288..4e34756349 100644 --- a/p2p/host/peerstore/pstoremem/inmem_test.go +++ b/p2p/host/peerstore/pstoremem/inmem_test.go @@ -3,7 +3,7 @@ package pstoremem import ( "testing" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" pt "github.com/libp2p/go-libp2p-peerstore/test" ) diff --git a/p2p/host/peerstore/pstoremem/keybook.go b/p2p/host/peerstore/pstoremem/keybook.go index 87baea24f7..f6ab84d236 100644 --- a/p2p/host/peerstore/pstoremem/keybook.go +++ b/p2p/host/peerstore/pstoremem/keybook.go @@ -4,10 +4,10 @@ import ( "errors" "sync" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" + ic "github.com/libp2p/go-libp2p-core/crypto" + peer "github.com/libp2p/go-libp2p-core/peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) type memoryKeyBook struct { diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index 9adebd8ae9..0e816fe798 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -3,8 +3,8 @@ package pstoremem import ( "sync" - peer "github.com/libp2p/go-libp2p-peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + peer "github.com/libp2p/go-libp2p-core/peer" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) var internKeys = map[string]bool{ diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index 3a5550d2b6..a1ce2311da 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -3,9 +3,9 @@ package pstoremem import ( "sync" - peer "github.com/libp2p/go-libp2p-peer" + peer "github.com/libp2p/go-libp2p-core/peer" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) type protoSegment struct { diff --git a/p2p/host/peerstore/queue/distance.go b/p2p/host/peerstore/queue/distance.go index 9c3e026e55..d7e79890ea 100644 --- a/p2p/host/peerstore/queue/distance.go +++ b/p2p/host/peerstore/queue/distance.go @@ -5,7 +5,7 @@ import ( "math/big" "sync" - "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" ks "github.com/whyrusleeping/go-keyspace" ) diff --git a/p2p/host/peerstore/queue/interface.go b/p2p/host/peerstore/queue/interface.go index 26e1d48b7d..7cdea71ed5 100644 --- a/p2p/host/peerstore/queue/interface.go +++ b/p2p/host/peerstore/queue/interface.go @@ -1,6 +1,6 @@ package queue -import "github.com/libp2p/go-libp2p-peer" +import "github.com/libp2p/go-libp2p-core/peer" // PeerQueue maintains a set of peers ordered according to a metric. // Implementations of PeerQueue could order peers based on distances along diff --git a/p2p/host/peerstore/queue/queue_test.go b/p2p/host/peerstore/queue/queue_test.go index 93b0e9f570..0c9afc7fc5 100644 --- a/p2p/host/peerstore/queue/queue_test.go +++ b/p2p/host/peerstore/queue/queue_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" mh "github.com/multiformats/go-multihash" ) diff --git a/p2p/host/peerstore/queue/sync.go b/p2p/host/peerstore/queue/sync.go index 3815ec782c..a17a759668 100644 --- a/p2p/host/peerstore/queue/sync.go +++ b/p2p/host/peerstore/queue/sync.go @@ -4,7 +4,7 @@ import ( "context" logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/peer" ) var log = logging.Logger("peerqueue") diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index c01fe98d73..72a899cd78 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -4,7 +4,7 @@ import ( "testing" "time" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ diff --git a/p2p/host/peerstore/test/benchmarks_suite.go b/p2p/host/peerstore/test/benchmarks_suite.go index cf04eb94a9..fb440f9d18 100644 --- a/p2p/host/peerstore/test/benchmarks_suite.go +++ b/p2p/host/peerstore/test/benchmarks_suite.go @@ -6,7 +6,7 @@ import ( "sort" "testing" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) var peerstoreBenchmarks = map[string]func(pstore.Peerstore, chan *peerpair) func(*testing.B){ diff --git a/p2p/host/peerstore/test/keybook_suite.go b/p2p/host/peerstore/test/keybook_suite.go index 9803128bd6..c8dbc268bb 100644 --- a/p2p/host/peerstore/test/keybook_suite.go +++ b/p2p/host/peerstore/test/keybook_suite.go @@ -4,11 +4,11 @@ import ( "sort" "testing" - ic "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" - pt "github.com/libp2p/go-libp2p-peer/test" + ic "github.com/libp2p/go-libp2p-core/crypto" + peer "github.com/libp2p/go-libp2p-core/peer" + pt "github.com/libp2p/go-libp2p-core/test" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) var keyBookSuite = map[string]func(kb pstore.KeyBook) func(*testing.T){ @@ -41,7 +41,7 @@ func testKeybookPrivKey(kb pstore.KeyBook) func(t *testing.T) { t.Error("expected peers to be empty on init") } - priv, _, err := pt.RandTestKeyPair(512) + priv, _, err := pt.RandTestKeyPair(ic.RSA, 512) if err != nil { t.Error(err) } @@ -76,7 +76,7 @@ func testKeyBookPubKey(kb pstore.KeyBook) func(t *testing.T) { t.Error("expected peers to be empty on init") } - _, pub, err := pt.RandTestKeyPair(512) + _, pub, err := pt.RandTestKeyPair(ic.RSA, 512) if err != nil { t.Error(err) } @@ -114,12 +114,12 @@ func testKeyBookPeers(kb pstore.KeyBook) func(t *testing.T) { var peers peer.IDSlice for i := 0; i < 10; i++ { // Add a public key. - _, pub, _ := pt.RandTestKeyPair(512) + _, pub, _ := pt.RandTestKeyPair(ic.RSA, 512) p1, _ := peer.IDFromPublicKey(pub) kb.AddPubKey(p1, pub) // Add a private key. - priv, _, _ := pt.RandTestKeyPair(512) + priv, _, _ := pt.RandTestKeyPair(ic.RSA, 512) p2, _ := peer.IDFromPrivateKey(priv) kb.AddPrivKey(p2, priv) @@ -192,7 +192,7 @@ func BenchmarkKeyBook(b *testing.B, factory KeyBookFactory) { func benchmarkPubKey(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { - _, pub, err := pt.RandTestKeyPair(512) + _, pub, err := pt.RandTestKeyPair(ic.RSA, 512) if err != nil { b.Error(err) } @@ -216,7 +216,7 @@ func benchmarkPubKey(kb pstore.KeyBook) func(*testing.B) { func benchmarkAddPubKey(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { - _, pub, err := pt.RandTestKeyPair(512) + _, pub, err := pt.RandTestKeyPair(ic.RSA, 512) if err != nil { b.Error(err) } @@ -235,7 +235,7 @@ func benchmarkAddPubKey(kb pstore.KeyBook) func(*testing.B) { func benchmarkPrivKey(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { - priv, _, err := pt.RandTestKeyPair(512) + priv, _, err := pt.RandTestKeyPair(ic.RSA, 512) if err != nil { b.Error(err) } @@ -259,7 +259,7 @@ func benchmarkPrivKey(kb pstore.KeyBook) func(*testing.B) { func benchmarkAddPrivKey(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { - priv, _, err := pt.RandTestKeyPair(512) + priv, _, err := pt.RandTestKeyPair(ic.RSA, 512) if err != nil { b.Error(err) } @@ -279,7 +279,7 @@ func benchmarkAddPrivKey(kb pstore.KeyBook) func(*testing.B) { func benchmarkPeersWithKeys(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { for i := 0; i < 10; i++ { - priv, pub, err := pt.RandTestKeyPair(512) + priv, pub, err := pt.RandTestKeyPair(ic.RSA, 512) if err != nil { b.Error(err) } diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index cd4bb634cc..f10478bb33 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -8,11 +8,11 @@ import ( "testing" "time" - crypto "github.com/libp2p/go-libp2p-crypto" - peer "github.com/libp2p/go-libp2p-peer" + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" - pstore "github.com/libp2p/go-libp2p-peerstore" + pstore "github.com/libp2p/go-libp2p-core/peerstore" ) var peerstoreSuite = map[string]func(pstore.Peerstore) func(*testing.T){ diff --git a/p2p/host/peerstore/test/utils.go b/p2p/host/peerstore/test/utils.go index d7af3ab81a..c169894b21 100644 --- a/p2p/host/peerstore/test/utils.go +++ b/p2p/host/peerstore/test/utils.go @@ -5,8 +5,8 @@ import ( "fmt" "testing" - peer "github.com/libp2p/go-libp2p-peer" - pt "github.com/libp2p/go-libp2p-peer/test" + peer "github.com/libp2p/go-libp2p-core/peer" + pt "github.com/libp2p/go-libp2p-core/test" ma "github.com/multiformats/go-multiaddr" ) From 5a976ee3d8cc2119742d0895c37e4fbce385ab32 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 31 May 2019 10:55:48 -0700 Subject: [PATCH 1284/3965] feat(peer): implement AddrInfosFromP2pAddrs and SplitAddr * SplitAddr is a simpler way to split an address into a multiaddr and an ID. * AddrInfosFromP2pAddrs converts a set of multiaddrs into a set of AddrInfos. --- core/peer/addrinfo.go | 46 +++++++++++-- core/peer/addrinfo_test.go | 135 +++++++++++++++++++++++++++++++++++++ 2 files changed, 175 insertions(+), 6 deletions(-) create mode 100644 core/peer/addrinfo_test.go diff --git a/core/peer/addrinfo.go b/core/peer/addrinfo.go index cc9633b7b0..6423d846b1 100644 --- a/core/peer/addrinfo.go +++ b/core/peer/addrinfo.go @@ -21,18 +21,51 @@ func (pi AddrInfo) String() string { var ErrInvalidAddr = fmt.Errorf("invalid p2p multiaddr") -func AddrInfoFromP2pAddr(m ma.Multiaddr) (*AddrInfo, error) { +// AddrInfosFromP2pAddrs converts a set of Multiaddrs to a set of AddrInfos. +func AddrInfosFromP2pAddrs(maddrs ...ma.Multiaddr) ([]AddrInfo, error) { + m := make(map[ID][]ma.Multiaddr) + for _, maddr := range maddrs { + transport, id := SplitAddr(maddr) + if id == "" { + return nil, ErrInvalidAddr + } + if transport == nil { + if _, ok := m[id]; !ok { + m[id] = nil + } + } else { + m[id] = append(m[id], transport) + } + } + ais := make([]AddrInfo, 0, len(m)) + for id, maddrs := range m { + ais = append(ais, AddrInfo{ID: id, Addrs: maddrs}) + } + return ais, nil +} + +// SplitAddr splits a p2p Multiaddr into a transport multiaddr and a peer ID. +// +// * Returns a nil transport if the address only contains a /p2p part. +// * Returns a empty peer ID if the address doesn't contain a /p2p part. +func SplitAddr(m ma.Multiaddr) (transport ma.Multiaddr, id ID) { if m == nil { - return nil, ErrInvalidAddr + return nil, "" } transport, p2ppart := ma.SplitLast(m) if p2ppart == nil || p2ppart.Protocol().Code != ma.P_P2P { - return nil, ErrInvalidAddr + return m, "" } - id, err := IDFromBytes(p2ppart.RawValue()) - if err != nil { - return nil, err + id = ID(p2ppart.RawValue()) // already validated by the multiaddr library. + return transport, id +} + +// AddrInfoFromP2pAddr converts a Multiaddr to an AddrInfo. +func AddrInfoFromP2pAddr(m ma.Multiaddr) (*AddrInfo, error) { + transport, id := SplitAddr(m) + if id == "" { + return nil, ErrInvalidAddr } info := &AddrInfo{ID: id} if transport != nil { @@ -41,6 +74,7 @@ func AddrInfoFromP2pAddr(m ma.Multiaddr) (*AddrInfo, error) { return info, nil } +// AddrInfoToP2pAddr converts an AddrInfo to a list of Multiaddrs. func AddrInfoToP2pAddrs(pi *AddrInfo) ([]ma.Multiaddr, error) { var addrs []ma.Multiaddr p2ppart, err := ma.NewComponent("p2p", IDB58Encode(pi.ID)) diff --git a/core/peer/addrinfo_test.go b/core/peer/addrinfo_test.go new file mode 100644 index 0000000000..b902fdf350 --- /dev/null +++ b/core/peer/addrinfo_test.go @@ -0,0 +1,135 @@ +package peer_test + +import ( + "testing" + + ma "github.com/multiformats/go-multiaddr" + + . "github.com/libp2p/go-libp2p-core/peer" +) + +var ( + testID ID + maddrFull, maddrTpt, maddrPeer ma.Multiaddr +) + +func init() { + var err error + testID, err = IDB58Decode("QmS3zcG7LhYZYSJMhyRZvTddvbNUqtt8BJpaSs6mi1K5Va") + if err != nil { + panic(err) + } + maddrPeer = ma.StringCast("/p2p/" + IDB58Encode(testID)) + maddrTpt = ma.StringCast("/ip4/127.0.0.1/tcp/1234") + maddrFull = maddrTpt.Encapsulate(maddrPeer) +} + +func TestSplitAddr(t *testing.T) { + tpt, id := SplitAddr(maddrFull) + if !tpt.Equal(maddrTpt) { + t.Fatal("expected transport") + } + if id != testID { + t.Fatalf("%s != %s", id, testID) + } + + tpt, id = SplitAddr(maddrPeer) + if tpt != nil { + t.Fatal("expected no transport") + } + if id != testID { + t.Fatalf("%s != %s", id, testID) + } + + tpt, id = SplitAddr(maddrTpt) + if !tpt.Equal(maddrTpt) { + t.Fatal("expected a transport") + } + if id != "" { + t.Fatal("expected no peer ID") + } +} + +func TestAddrInfoFromP2pAddr(t *testing.T) { + ai, err := AddrInfoFromP2pAddr(maddrFull) + if err != nil { + t.Fatal(err) + } + if len(ai.Addrs) != 1 || !ai.Addrs[0].Equal(maddrTpt) { + t.Fatal("expected transport") + } + if ai.ID != testID { + t.Fatalf("%s != %s", ai.ID, testID) + } + + ai, err = AddrInfoFromP2pAddr(maddrPeer) + if err != nil { + t.Fatal(err) + } + if len(ai.Addrs) != 0 { + t.Fatal("expected transport") + } + if ai.ID != testID { + t.Fatalf("%s != %s", ai.ID, testID) + } + + _, err = AddrInfoFromP2pAddr(maddrTpt) + if err != ErrInvalidAddr { + t.Fatalf("wrong error: %s", err) + } +} + +func TestAddrInfosFromP2pAddrs(t *testing.T) { + infos, err := AddrInfosFromP2pAddrs() + if err != nil { + t.Fatal(err) + } + if len(infos) != 0 { + t.Fatal("expected no addrs") + } + infos, err = AddrInfosFromP2pAddrs(nil) + if err == nil { + t.Fatal("expected nil multiaddr to fail") + } + + addrs := []ma.Multiaddr{ + ma.StringCast("/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64"), + ma.StringCast("/ip4/104.236.76.40/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64"), + + ma.StringCast("/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd"), + ma.StringCast("/ip4/178.62.158.247/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd"), + + ma.StringCast("/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM"), + } + expected := map[string][]ma.Multiaddr{ + "QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64": { + ma.StringCast("/ip4/128.199.219.111/tcp/4001"), + ma.StringCast("/ip4/104.236.76.40/tcp/4001"), + }, + "QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd": { + ma.StringCast("/ip4/178.62.158.247/tcp/4001"), + }, + "QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM": nil, + } + infos, err = AddrInfosFromP2pAddrs(addrs...) + if err != nil { + t.Fatal(err) + } + for _, info := range infos { + exaddrs, ok := expected[info.ID.Pretty()] + if !ok { + t.Fatalf("didn't expect peer %s", info.ID) + } + if len(info.Addrs) != len(exaddrs) { + t.Fatalf("got %d addrs, expected %d", len(info.Addrs), len(exaddrs)) + } + // AddrInfosFromP2pAddrs preserves order. I'd like to keep this + // guarantee for now. + for i, addr := range info.Addrs { + if !exaddrs[i].Equal(addr) { + t.Fatalf("expected %s, got %s", exaddrs[i], addr) + } + } + delete(expected, info.ID.Pretty()) + } +} From 74ed3396d706b091c8efb6f08971070afa340bfb Mon Sep 17 00:00:00 2001 From: whyrusleeping Date: Fri, 31 May 2019 18:31:04 -0700 Subject: [PATCH 1285/3965] set map in constructor --- p2p/host/peerstore/pstoremem/metadata.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index 0e816fe798..821c2b6a51 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -29,7 +29,8 @@ var _ pstore.PeerMetadata = (*memoryPeerMetadata)(nil) func NewPeerMetadata() pstore.PeerMetadata { return &memoryPeerMetadata{ - ds: make(map[metakey]interface{}), + ds: make(map[metakey]interface{}), + interned: make(map[string]interface{}), } } From 31ec77d6dca4fa0bb7a6289e003869b0d4f835bd Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 31 May 2019 19:52:40 -0700 Subject: [PATCH 1286/3965] test: add metadata test --- p2p/host/peerstore/test/peerstore_suite.go | 41 ++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index f10478bb33..020da4caa0 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -21,6 +21,7 @@ var peerstoreSuite = map[string]func(pstore.Peerstore) func(*testing.T){ "AddStreamDuplicates": testAddrStreamDuplicates, "PeerstoreProtoStore": testPeerstoreProtoStore, "BasicPeerstore": testBasicPeerstore, + "Metadata": testMetadata, } type PeerstoreFactory func() (pstore.Peerstore, func()) @@ -279,6 +280,46 @@ func testBasicPeerstore(ps pstore.Peerstore) func(t *testing.T) { } } +func testMetadata(ps pstore.Peerstore) func(t *testing.T) { + return func(t *testing.T) { + pids := make([]peer.ID, 10) + for i := range pids { + priv, _, _ := crypto.GenerateKeyPair(crypto.RSA, 512) + p, _ := peer.IDFromPrivateKey(priv) + pids[i] = p + } + for _, p := range pids { + if err := ps.Put(p, "AgentVersion", "string"); err != nil { + t.Errorf("failed to put %q: %s", "AgentVersion", err) + } + if err := ps.Put(p, "bar", 1); err != nil { + t.Errorf("failed to put %q: %s", "bar", err) + } + } + for _, p := range pids { + v, err := ps.Get(p, "AgentVersion") + if err != nil { + t.Errorf("failed to find %q: %s", "AgentVersion", err) + continue + } + if v != "string" { + t.Errorf("expected %q, got %q", "string", p) + continue + } + + v, err = ps.Get(p, "bar") + if err != nil { + t.Errorf("failed to find %q: %s", "bar", err) + continue + } + if v != 1 { + t.Errorf("expected %q, got %v", 1, v) + continue + } + } + } +} + func getAddrs(t *testing.T, n int) []ma.Multiaddr { var addrs []ma.Multiaddr for i := 0; i < n; i++ { From 0635f49abb5c9db08573b0a00c31a922400961f7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 31 May 2019 20:36:29 -0700 Subject: [PATCH 1287/3965] fix: update to non-coalescing yamux --- go.mod | 4 ++-- go.sum | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index 91fe7d2178..d951ad3a1b 100644 --- a/go.mod +++ b/go.mod @@ -21,9 +21,9 @@ require ( github.com/libp2p/go-libp2p-peerstore v0.1.0 github.com/libp2p/go-libp2p-secio v0.1.0 github.com/libp2p/go-libp2p-swarm v0.1.0 - github.com/libp2p/go-libp2p-testing v0.0.3 + github.com/libp2p/go-libp2p-testing v0.0.4 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 - github.com/libp2p/go-libp2p-yamux v0.2.0 + github.com/libp2p/go-libp2p-yamux v0.2.1 github.com/libp2p/go-maddr-filter v0.0.4 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.0 diff --git a/go.sum b/go.sum index 0f6dbadbb6..47c9026caa 100644 --- a/go.sum +++ b/go.sum @@ -119,10 +119,14 @@ github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evl github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.3 h1:bdij4bKaaND7tCsaXVjRfYkMpvoOeKj9AVQGJllA6jM= github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= +github.com/libp2p/go-libp2p-testing v0.0.4 h1:Qev57UR47GcLPXWjrunv5aLIQGO4n9mhI/8/EIrEEFc= +github.com/libp2p/go-libp2p-testing v0.0.4/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 h1:PZMS9lhjK9VytzMCW3tWHAXtKXmlURSc3ZdvwEcKCzw= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07qPW4r01GnTenLW4oi3lOPbUMGJJA= github.com/libp2p/go-libp2p-yamux v0.2.0 h1:TSPZ5cMMz/wdoYsye/wU1TE4G3LDGMoeEN0xgnCKU/I= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= +github.com/libp2p/go-libp2p-yamux v0.2.1 h1:Q3XYNiKCC2vIxrvUJL+Jg1kiyeEaIDNKLjgEjo3VQdI= +github.com/libp2p/go-libp2p-yamux v0.2.1/go.mod h1:1FBXiHDk1VyRM1C0aez2bCfHQ4vMZKkAQzZbkSQt5fI= github.com/libp2p/go-maddr-filter v0.0.4 h1:hx8HIuuwk34KePddrp2mM5ivgPkZ09JH4AvsALRbFUs= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-mplex v0.0.3/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= @@ -145,6 +149,8 @@ github.com/libp2p/go-ws-transport v0.1.0 h1:F+0OvvdmPTDsVc4AjPHjV7L7Pk1B7D5QwtDc github.com/libp2p/go-ws-transport v0.1.0/go.mod h1:rjw1MG1LU9YDC6gzmwObkPd/Sqwhw7yT74kj3raBFuo= github.com/libp2p/go-yamux v1.2.2 h1:s6J6o7+ajoQMjHe7BEnq+EynOj5D2EoG8CuQgL3F2vg= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= +github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= From 10d45bd76c399038233b7c8fb60613acff106075 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 3 Jun 2019 23:14:41 +0200 Subject: [PATCH 1288/3965] feat: cache interface addresses for 1 minute This can be quite an overhead in cases of high connection rates. The main overhead is thread blocking syscall causing a lot of context switching. License: MIT Signed-off-by: Jakub Sztandera --- p2p/net/swarm/swarm.go | 5 ++++ p2p/net/swarm/swarm_addr.go | 46 ++++++++++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index b14e9cf0b9..642faea856 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -20,6 +20,7 @@ import ( goprocessctx "github.com/jbenet/goprocess/context" filter "github.com/libp2p/go-maddr-filter" + ma "github.com/multiformats/go-multiaddr" mafilter "github.com/whyrusleeping/multiaddr-filter" ) @@ -58,6 +59,10 @@ type Swarm struct { listeners struct { sync.RWMutex + + ifaceAddresses []ma.Multiaddr + cacheEOL time.Time + m map[transport.Listener]struct{} } diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index c86d58bb4d..10179fa3b9 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -1,6 +1,8 @@ package swarm import ( + "time" + addrutil "github.com/libp2p/go-addr-util" ma "github.com/multiformats/go-multiaddr" ) @@ -9,6 +11,10 @@ import ( func (s *Swarm) ListenAddresses() []ma.Multiaddr { s.listeners.RLock() defer s.listeners.RUnlock() + return s.listenAddressesNoLock() +} + +func (s *Swarm) listenAddressesNoLock() []ma.Multiaddr { addrs := make([]ma.Multiaddr, 0, len(s.listeners.m)) for l := range s.listeners.m { addrs = append(addrs, l.Multiaddr()) @@ -16,9 +22,47 @@ func (s *Swarm) ListenAddresses() []ma.Multiaddr { return addrs } +const ifaceAddrsCacheDuration = 1 * time.Minute + // InterfaceListenAddresses returns a list of addresses at which this swarm // listens. It expands "any interface" addresses (/ip4/0.0.0.0, /ip6/::) to // use the known local interfaces. func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { - return addrutil.ResolveUnspecifiedAddresses(s.ListenAddresses(), nil) + s.listeners.RLock() // RLock start + listenAddrs := s.listenAddressesNoLock() + + ifaceAddrs := s.listeners.ifaceAddresses + isEOL := time.Now().After(s.listeners.cacheEOL) + s.listeners.RUnlock() // RLock end + + if listenAddrs != nil && !isEOL { + // Cache is valid + return addrutil.ResolveUnspecifiedAddresses(listenAddrs, ifaceAddrs) + } + + // Cache is not valid + // Perfrom double checked locking + + s.listeners.Lock() // Lock start + + listenAddrs = s.listenAddressesNoLock() + + ifaceAddrs = s.listeners.ifaceAddresses + isEOL = time.Now().After(s.listeners.cacheEOL) + if listenAddrs == nil || isEOL { + // Cache is still invalid + var err error + ifaceAddrs, err = addrutil.InterfaceAddresses() + if err != nil { + s.listeners.Unlock() // Lock early exit + return nil, err + } + + s.listeners.ifaceAddresses = ifaceAddrs + s.listeners.cacheEOL = time.Now().Add(ifaceAddrsCacheDuration) + } + + s.listeners.Unlock() // Lock end + + return addrutil.ResolveUnspecifiedAddresses(listenAddrs, ifaceAddrs) } From c7245b745f049911ac170c853bd13ba55e2bfea6 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Tue, 4 Jun 2019 21:21:52 +0200 Subject: [PATCH 1289/3965] feat: cache full InterfaceListenAddresses License: MIT Signed-off-by: Jakub Sztandera --- p2p/net/swarm/swarm.go | 4 ++-- p2p/net/swarm/swarm_addr.go | 22 +++++++++++----------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 642faea856..cc7eb978a6 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -60,8 +60,8 @@ type Swarm struct { listeners struct { sync.RWMutex - ifaceAddresses []ma.Multiaddr - cacheEOL time.Time + ifaceListenAddres []ma.Multiaddr + cacheEOL time.Time m map[transport.Listener]struct{} } diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 10179fa3b9..6210a4f809 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -29,15 +29,14 @@ const ifaceAddrsCacheDuration = 1 * time.Minute // use the known local interfaces. func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { s.listeners.RLock() // RLock start - listenAddrs := s.listenAddressesNoLock() - ifaceAddrs := s.listeners.ifaceAddresses + ifaceListenAddres := s.listeners.ifaceListenAddres isEOL := time.Now().After(s.listeners.cacheEOL) s.listeners.RUnlock() // RLock end - if listenAddrs != nil && !isEOL { + if ifaceListenAddres != nil && !isEOL { // Cache is valid - return addrutil.ResolveUnspecifiedAddresses(listenAddrs, ifaceAddrs) + return ifaceListenAddres, nil } // Cache is not valid @@ -45,24 +44,25 @@ func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { s.listeners.Lock() // Lock start - listenAddrs = s.listenAddressesNoLock() - - ifaceAddrs = s.listeners.ifaceAddresses + ifaceListenAddres = s.listeners.ifaceListenAddres isEOL = time.Now().After(s.listeners.cacheEOL) - if listenAddrs == nil || isEOL { + if ifaceListenAddres == nil || isEOL { // Cache is still invalid + var err error - ifaceAddrs, err = addrutil.InterfaceAddresses() + ifaceListenAddres, err = addrutil.ResolveUnspecifiedAddresses( + s.listenAddressesNoLock(), nil) + if err != nil { s.listeners.Unlock() // Lock early exit return nil, err } - s.listeners.ifaceAddresses = ifaceAddrs + s.listeners.ifaceListenAddres = ifaceListenAddres s.listeners.cacheEOL = time.Now().Add(ifaceAddrsCacheDuration) } s.listeners.Unlock() // Lock end - return addrutil.ResolveUnspecifiedAddresses(listenAddrs, ifaceAddrs) + return ifaceListenAddres, nil } From b80c6891d771fe207061e4a1d72ef2571df98d44 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Tue, 4 Jun 2019 23:10:02 +0200 Subject: [PATCH 1290/3965] invalidate cache for InterfaceListenAddresses on Listen License: MIT Signed-off-by: Jakub Sztandera --- p2p/net/swarm/swarm_listen.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 85060c4e2c..570acfedae 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -2,6 +2,7 @@ package swarm import ( "fmt" + "time" "github.com/libp2p/go-libp2p-core/network" @@ -55,6 +56,7 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { } s.refs.Add(1) s.listeners.m[list] = struct{}{} + s.listeners.cacheEOL = time.Time{} s.listeners.Unlock() maddr := list.Multiaddr() @@ -69,6 +71,7 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { list.Close() s.listeners.Lock() delete(s.listeners.m, list) + s.listeners.cacheEOL = time.Time{} s.listeners.Unlock() s.refs.Done() }() From c12ef6a31836b9fc1a49bb42338f6dd4a8017c3b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 6 Jun 2019 00:23:04 -0700 Subject: [PATCH 1291/3965] logging: make the swarm less noisy Avoid logging about closed listeners, etc., when shutting down. --- p2p/net/swarm/swarm.go | 5 +++++ p2p/net/swarm/swarm_listen.go | 11 ++++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index cc7eb978a6..5366bdcc6f 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -116,6 +116,11 @@ func NewSwarm(ctx context.Context, local peer.ID, peers peerstore.Peerstore, bwc } func (s *Swarm) teardown() error { + // Wait for the context to be canceled. + // This allows other parts of the swarm to detect that we're shutting + // down. + <-s.ctx.Done() + // Prevents new connections and/or listeners from being added to the swarm. s.listeners.Lock() diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index 570acfedae..f1cfa9e7f1 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -79,6 +79,7 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { c, err := list.Accept() if err != nil { if s.ctx.Err() == nil { + // only log if the swarm is still running. log.Errorf("swarm listener accept error: %s", err) } return @@ -88,9 +89,13 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { go func() { defer s.refs.Done() _, err := s.addConn(c, network.DirInbound) - if err != nil { - // Probably just means that the swarm has been closed. - log.Warningf("add conn failed: ", err) + switch err { + case nil: + case ErrSwarmClosed: + // ignore. + return + default: + log.Warningf("add conn %s failed: ", err) return } }() From 85e37a9201daa58ba7a3a17a9576a2eb659ddcbe Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Thu, 6 Jun 2019 19:41:40 +0200 Subject: [PATCH 1292/3965] Update README links and copyright --- p2p/net/gostream/coverage.out | 37 ----------------------------------- 1 file changed, 37 deletions(-) delete mode 100644 p2p/net/gostream/coverage.out diff --git a/p2p/net/gostream/coverage.out b/p2p/net/gostream/coverage.out deleted file mode 100644 index 77b128ca5a..0000000000 --- a/p2p/net/gostream/coverage.out +++ /dev/null @@ -1,37 +0,0 @@ -mode: count -github.com/hsanjuan/go-libp2p-http/addr.go:8.33,8.52 1 0 -github.com/hsanjuan/go-libp2p-http/addr.go:9.33,9.57 1 1 -github.com/hsanjuan/go-libp2p-http/conn.go:17.38,19.2 1 2 -github.com/hsanjuan/go-libp2p-http/conn.go:22.50,24.2 1 3 -github.com/hsanjuan/go-libp2p-http/conn.go:27.51,29.2 1 2 -github.com/hsanjuan/go-libp2p-http/conn.go:33.30,35.2 1 2 -github.com/hsanjuan/go-libp2p-http/conn.go:38.37,40.2 1 0 -github.com/hsanjuan/go-libp2p-http/conn.go:43.38,45.2 1 1 -github.com/hsanjuan/go-libp2p-http/conn.go:51.47,53.2 1 0 -github.com/hsanjuan/go-libp2p-http/conn.go:57.51,59.2 1 0 -github.com/hsanjuan/go-libp2p-http/conn.go:65.52,67.2 1 0 -github.com/hsanjuan/go-libp2p-http/listener.go:25.47,26.9 1 2 -github.com/hsanjuan/go-libp2p-http/listener.go:27.2,28.25 1 1 -github.com/hsanjuan/go-libp2p-http/listener.go:29.2,30.26 1 1 -github.com/hsanjuan/go-libp2p-http/listener.go:36.34,40.2 3 2 -github.com/hsanjuan/go-libp2p-http/listener.go:43.36,45.2 1 1 -github.com/hsanjuan/go-libp2p-http/listener.go:49.48,59.54 3 1 -github.com/hsanjuan/go-libp2p-http/listener.go:67.2,67.15 1 1 -github.com/hsanjuan/go-libp2p-http/listener.go:59.54,60.10 1 1 -github.com/hsanjuan/go-libp2p-http/listener.go:61.3,61.24 0 1 -github.com/hsanjuan/go-libp2p-http/listener.go:62.3,63.13 1 0 -github.com/hsanjuan/go-libp2p-http/listener.go:73.58,78.16 4 1 -github.com/hsanjuan/go-libp2p-http/listener.go:82.2,83.16 2 1 -github.com/hsanjuan/go-libp2p-http/listener.go:86.2,86.24 1 1 -github.com/hsanjuan/go-libp2p-http/listener.go:78.16,80.3 1 0 -github.com/hsanjuan/go-libp2p-http/listener.go:83.16,85.3 1 0 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:16.76,17.19 1 1 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:21.2,22.16 2 1 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:26.2,27.16 2 1 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:30.2,33.16 3 1 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:37.2,38.37 2 1 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:17.19,19.3 1 0 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:22.16,24.3 1 0 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:27.16,29.3 1 0 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:33.16,35.3 1 0 -github.com/hsanjuan/go-libp2p-http/roundtripper.go:51.46,53.2 1 1 From 4a3c8bf495fd9ba0dc22f6924394ebb122327752 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Sat, 8 Jun 2019 00:42:17 +0200 Subject: [PATCH 1293/3965] Add go-libp2p-gostream and go-libp2p-http to readme License: MIT Signed-off-by: Hector Sanjuan --- README.md | 5 ++++- package-list.json | 4 +++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 47ea441d5f..17f482f75b 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ If you experience any issues migrating from gx to gomod, please [join the discus List of packages currently in existence for libp2p: -| Name | CI | Coverage | Description | +| Name | CI/Travis | Coverage | Description | | ---------|---------|---------|--------- | | **Libp2p** | | [`go-libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | go-libp2p entry point | @@ -184,11 +184,14 @@ List of packages currently in existence for libp2p: | [`go-reuseport`](//github.com/libp2p/go-reuseport) | [![Travis CI](https://travis-ci.com/libp2p/go-reuseport.svg?branch=master)](https://travis-ci.com/libp2p/go-reuseport) | [![codecov](https://codecov.io/gh/libp2p/go-reuseport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-reuseport) | enables reuse of addresses | | [`go-sockaddr`](//github.com/libp2p/go-sockaddr) | [![Travis CI](https://travis-ci.com/libp2p/go-sockaddr.svg?branch=master)](https://travis-ci.com/libp2p/go-sockaddr) | [![codecov](https://codecov.io/gh/libp2p/go-sockaddr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-sockaddr) | utils for sockaddr conversions | | [`go-flow-metrics`](//github.com/libp2p/go-flow-metrics) | [![Travis CI](https://travis-ci.com/libp2p/go-flow-metrics.svg?branch=master)](https://travis-ci.com/libp2p/go-flow-metrics) | [![codecov](https://codecov.io/gh/libp2p/go-flow-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-flow-metrics) | metrics library | +| [`go-libp2p-gostream`](//github.com/libp2p/go-libp2p-gostream) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-gostream.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-gostream) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-gostream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-gostream) | Go 'net' wrappers for libp2p | +| [`go-libp2p-http`](//github.com/libp2p/go-libp2p-http) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-http.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-http) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-http/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-http) | HTTP on top of libp2p streams | | **Testing and examples** | | [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.com/libp2p/go-testutil.svg?branch=master)](https://travis-ci.com/libp2p/go-testutil) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | a collection of testing utilities for ipfs and libp2p | | [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-examples) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | | [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-circuit-progs) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | + # Contribute go-libp2p is part of [The IPFS Project](https://github.com/ipfs/ipfs), and is MIT licensed open source software. We welcome contributions big and small! Take a look at the [community contributing notes](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md). Please make sure to check the [issues](https://github.com/ipfs/go-libp2p/issues). Search the closed ones before reporting things, and help us with the open ones. diff --git a/package-list.json b/package-list.json index e29a741036..7679240600 100644 --- a/package-list.json +++ b/package-list.json @@ -38,7 +38,7 @@ "Stream Muxers", ["libp2p/go-stream-muxer", "go-stream-muxer", "interfaces"], ["whyrusleeping/go-smux-yamux", "go-smux-yamux", "YAMUX stream multiplexer"], - ["whyrusleeping/go-smux-mplex", "go-smux-mplex", "MPLEX stream multiplexer"], + ["whyrusleeping/go-smux-multiplex", "go-smux-multiplex", "MPLEX stream multiplexer"], "NAT Traversal", ["libp2p/go-libp2p-nat", "go-libp2p-nat"], @@ -87,6 +87,8 @@ ["libp2p/go-reuseport", "go-reuseport", "enables reuse of addresses"], ["libp2p/go-sockaddr", "go-sockaddr", "utils for sockaddr conversions"], ["libp2p/go-flow-metrics", "go-flow-metrics", "metrics library"], + ["libp2p/go-libp2p-gostream", "go-libp2p-gostream", "Go 'net' wrappers for libp2p"], + ["libp2p/go-libp2p-http", "go-libp2p-http", "HTTP on top of libp2p streams"], "Testing and examples", ["libp2p/go-testutil", "go-testutil", "a collection of testing utilities for ipfs and libp2p"], From e92294228f82eadfb04fe12ddf3d70ee1742c1fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Thu, 13 Jun 2019 04:22:28 +0200 Subject: [PATCH 1294/3965] Initial commit --- p2p/host/eventbus/LICENSE-APACHE | 5 +++++ p2p/host/eventbus/LICENSE-MIT | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+) create mode 100644 p2p/host/eventbus/LICENSE-APACHE create mode 100644 p2p/host/eventbus/LICENSE-MIT diff --git a/p2p/host/eventbus/LICENSE-APACHE b/p2p/host/eventbus/LICENSE-APACHE new file mode 100644 index 0000000000..14478a3b60 --- /dev/null +++ b/p2p/host/eventbus/LICENSE-APACHE @@ -0,0 +1,5 @@ +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. diff --git a/p2p/host/eventbus/LICENSE-MIT b/p2p/host/eventbus/LICENSE-MIT new file mode 100644 index 0000000000..72dc60d84b --- /dev/null +++ b/p2p/host/eventbus/LICENSE-MIT @@ -0,0 +1,19 @@ +The MIT License (MIT) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. From 4e8b8db0e8201a38e44ea690fa9d73e4d3491e5f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Thu, 13 Jun 2019 04:23:03 +0200 Subject: [PATCH 1295/3965] Initial implementation --- p2p/host/eventbus/basic.go | 144 ++++++++++++++++++++++++++++++++ p2p/host/eventbus/basic_test.go | 13 +++ p2p/host/eventbus/interface.go | 25 ++++++ 3 files changed, 182 insertions(+) create mode 100644 p2p/host/eventbus/basic.go create mode 100644 p2p/host/eventbus/basic_test.go create mode 100644 p2p/host/eventbus/interface.go diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go new file mode 100644 index 0000000000..1abd3da629 --- /dev/null +++ b/p2p/host/eventbus/basic.go @@ -0,0 +1,144 @@ +package event + +import ( + "errors" + "fmt" + "io" + "reflect" + "sync" +) + +/////////////////////// +// BUS + +type bus struct { + lk sync.Mutex + nodes map[string]*node +} + +func NewBus() Bus { + return &bus{ + nodes: map[string]*node{}, + } +} + +func (b *bus) withNode(evtType interface{}, cb func(*node)) error { + typ := reflect.TypeOf(evtType) + if typ.Kind() != reflect.Ptr { + return errors.New("subscribe called with non-pointer type") + } + typ = typ.Elem() + path := typePath(typ) + + b.lk.Lock() + + n, ok := b.nodes[path] + if !ok { + n = newNode(typ) + b.nodes[path] = n + } + + n.lk.Lock() + b.lk.Unlock() + cb(n) + n.lk.Unlock() + return nil +} + +func (b *bus) Sub(evtType interface{}) (s Subscription, err error) { + err = b.withNode(evtType, func(n *node) { + s = n.sub(0) + }) + return +} + +func (b *bus) Emitter(evtType interface{}) (e Emitter, err error) { + err = b.withNode(evtType, func(n *node) { + e = &emitter{ + Closer: closer(func(){}), //TODO: actually do something here + node: n, + } + }) + return +} + +/////////////////////// +// NODE + +type node struct { + // Note: make sure to NEVER lock bus.lk when this lock is held + lk sync.RWMutex + + typ reflect.Type + + n int + sinks map[int]chan interface{} +} + +func newNode(typ reflect.Type) *node { + return &node{ + typ: typ, + + sinks: map[int]chan interface{}{}, + } +} + +func (n *node) sub(buf int) Subscription { + out := make(chan interface{}, buf) + n.n++ + n.sinks[n.n] = out + return &sub{ + out: out, + } +} + +func (n *node) Emit(event interface{}) { + etype := reflect.TypeOf(event) + if etype != n.typ { + panic(fmt.Sprintf("Emit called with wrong type. expected: %s, got: %s", n.typ, etype)) + } + + n.lk.RLock() + for _, ch := range n.sinks { + ch <- event + } +} + +/////////////////////// +// SUB + +type sub struct { + io.Closer + out <-chan interface{} +} + +func (s *sub) Events() <-chan interface{} { + return s.out +} + +/////////////////////// +// EMITTERS + +type emitter struct { + io.Closer + *node +} + +/////////////////////// +// UTILS + +func typePath(t reflect.Type) string { + return t.PkgPath() + "/" + t.String() +} + +type closer func() + +func (c closer) Close() error { + c() + return nil +} + +var _ Bus = &bus{} +var _ Subscription = &sub{} +var _ Emitter = &emitter{} +var _ io.Closer = closer(nil) \ No newline at end of file diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go new file mode 100644 index 0000000000..4b25381131 --- /dev/null +++ b/p2p/host/eventbus/basic_test.go @@ -0,0 +1,13 @@ +package event + +import ( + "testing" +) + +type EventA struct{} + +func TestSimple(t *testing.T) { + bus := NewBus() + _, _ = bus.Sub(new(EventA)) + +} diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go new file mode 100644 index 0000000000..92d862f0df --- /dev/null +++ b/p2p/host/eventbus/interface.go @@ -0,0 +1,25 @@ +package event + +import ( + "io" +) + +type Bus interface { + //Sub(...Opt) func(evtType interface{}, error) + Sub(evtTypes interface{}) (Subscription, error) + Emitter(evtTypes interface{}) (Emitter, error) +} + +type Subscription interface { + io.Closer + + Events() <-chan interface{} +} + +type Emitter interface { + io.Closer + + Emit(event interface{}) +} + + From 3f45d5bab0a5a575b98a84d65a77e2de5fa17a61 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Thu, 13 Jun 2019 08:51:54 +0200 Subject: [PATCH 1296/3965] MVP --- p2p/host/eventbus/basic.go | 108 +++++++++++++++----------- p2p/host/eventbus/basic_test.go | 131 +++++++++++++++++++++++++++++++- p2p/host/eventbus/interface.go | 39 +++++----- 3 files changed, 212 insertions(+), 66 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 1abd3da629..10372b8e8e 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -3,9 +3,9 @@ package event import ( "errors" "fmt" - "io" "reflect" "sync" + "sync/atomic" ) /////////////////////// @@ -45,18 +45,63 @@ func (b *bus) withNode(evtType interface{}, cb func(*node)) error { return nil } -func (b *bus) Sub(evtType interface{}) (s Subscription, err error) { +func (b *bus) tryDropNode(evtType interface{}) { + path := typePath(reflect.TypeOf(evtType).Elem()) + + b.lk.Lock() + n, ok := b.nodes[path] + if !ok { // already dropped + b.lk.Unlock() + return + } + + n.lk.Lock() + if n.nEmitters > 0 || len(n.sinks) > 0 { + n.lk.Unlock() + b.lk.Unlock() + return // still in use + } + n.lk.Unlock() + + delete(b.nodes, path) + b.lk.Unlock() +} + +func (b *bus) Subscribe(evtType interface{}) (s <-chan interface{}, c CancelFunc, err error) { err = b.withNode(evtType, func(n *node) { - s = n.sub(0) + out, i := n.sub(0) + s = out + c = func() { + n.lk.Lock() + delete(n.sinks, i) + close(out) + tryDrop := len(n.sinks) == 0 && n.nEmitters == 0 + n.lk.Unlock() + if tryDrop { + b.tryDropNode(evtType) + } + } }) return } -func (b *bus) Emitter(evtType interface{}) (e Emitter, err error) { +func (b *bus) Emitter(evtType interface{}) (e EmitFunc, c CancelFunc, err error) { err = b.withNode(evtType, func(n *node) { - e = &emitter{ - Closer: closer(func(){}), //TODO: actually do something here - node: n, + atomic.AddInt32(&n.nEmitters, 1) + closed := false + + e = func(event interface{}) { + if closed { + panic("emitter is closed") + } + n.emit(event) + } + + c = func() { + closed = true + if atomic.AddInt32(&n.nEmitters, -1) == 0 { + b.tryDropNode(evtType) + } } }) return @@ -71,8 +116,9 @@ type node struct { typ reflect.Type - n int - sinks map[int]chan interface{} + nEmitters int32 + nSinks int + sinks map[int]chan interface{} } func newNode(typ reflect.Type) *node { @@ -83,16 +129,15 @@ func newNode(typ reflect.Type) *node { } } -func (n *node) sub(buf int) Subscription { +func (n *node) sub(buf int) (chan interface{}, int) { out := make(chan interface{}, buf) - n.n++ - n.sinks[n.n] = out - return &sub{ - out: out, - } + i := n.nSinks + n.nSinks++ + n.sinks[i] = out + return out, i } -func (n *node) Emit(event interface{}) { +func (n *node) emit(event interface{}) { etype := reflect.TypeOf(event) if etype != n.typ { panic(fmt.Sprintf("Emit called with wrong type. expected: %s, got: %s", n.typ, etype)) @@ -102,26 +147,7 @@ func (n *node) Emit(event interface{}) { for _, ch := range n.sinks { ch <- event } -} - -/////////////////////// -// SUB - -type sub struct { - io.Closer - out <-chan interface{} -} - -func (s *sub) Events() <-chan interface{} { - return s.out -} - -/////////////////////// -// EMITTERS - -type emitter struct { - io.Closer - *node + n.lk.RUnlock() } /////////////////////// @@ -131,14 +157,4 @@ func typePath(t reflect.Type) string { return t.PkgPath() + "/" + t.String() } -type closer func() - -func (c closer) Close() error { - c() - return nil -} - var _ Bus = &bus{} -var _ Subscription = &sub{} -var _ Emitter = &emitter{} -var _ io.Closer = closer(nil) \ No newline at end of file diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 4b25381131..4895005d67 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -1,13 +1,140 @@ package event import ( + "sync" "testing" + "time" ) type EventA struct{} +type EventB int -func TestSimple(t *testing.T) { +func TestEmit(t *testing.T) { bus := NewBus() - _, _ = bus.Sub(new(EventA)) + events, cancel, err := bus.Subscribe(new(EventA)) + if err != nil { + t.Fatal(err) + } + go func() { + defer cancel() + <-events + }() + + emit, cancel, err := bus.Emitter(new(EventA)) + if err != nil { + t.Fatal(err) + } + defer cancel() + + emit(EventA{}) +} + +func TestSub(t *testing.T) { + bus := NewBus() + events, cancel, err := bus.Subscribe(new(EventB)) + if err != nil { + t.Fatal(err) + } + + var event EventB + + var wait sync.WaitGroup + wait.Add(1) + + go func() { + defer cancel() + event = (<-events).(EventB) + wait.Done() + }() + + emit, cancel, err := bus.Emitter(new(EventB)) + if err != nil { + t.Fatal(err) + } + defer cancel() + + emit(EventB(7)) + wait.Wait() + + if event != 7 { + t.Error("got wrong event") + } +} + +func TestEmitNoSubNoBlock(t *testing.T) { + bus := NewBus() + + emit, cancel, err := bus.Emitter(new(EventA)) + if err != nil { + t.Fatal(err) + } + defer cancel() + + emit(EventA{}) +} + +func TestEmitOnClosed(t *testing.T) { + bus := NewBus() + + emit, cancel, err := bus.Emitter(new(EventA)) + if err != nil { + t.Fatal(err) + } + cancel() + + defer func() { + r := recover() + if r == nil { + t.Errorf("expected panic") + } + if r.(string) != "emitter is closed" { + t.Error("unexpected message") + } + }() + + emit(EventA{}) +} + +func TestClosingRaces(t *testing.T) { + subs := 50000 + emits := 50000 + + var wg sync.WaitGroup + var lk sync.RWMutex + lk.Lock() + + wg.Add(subs + emits) + + bus := NewBus() + + for i := 0; i < subs; i++ { + go func() { + lk.RLock() + defer lk.RUnlock() + + _, cancel, _ := bus.Subscribe(new(EventA)) + time.Sleep(10 * time.Millisecond) + cancel() + + wg.Done() + }() + } + for i := 0; i < emits; i++ { + go func() { + lk.RLock() + defer lk.RUnlock() + + _, cancel, _ := bus.Emitter(new(EventA)) + time.Sleep(10 * time.Millisecond) + cancel() + + wg.Done() + }() + } + + time.Sleep(10 * time.Millisecond) + lk.Unlock() // start everything + + wg.Wait() } diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 92d862f0df..81150e4d57 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -1,25 +1,28 @@ package event -import ( - "io" -) - type Bus interface { - //Sub(...Opt) func(evtType interface{}, error) - Sub(evtTypes interface{}) (Subscription, error) - Emitter(evtTypes interface{}) (Emitter, error) -} - -type Subscription interface { - io.Closer - - Events() <-chan interface{} + // Subscribe creates new subscription. Failing to drain the incoming channel + // will cause publishers to get blocked + // + // evtTypes only accepts typed nil pointers, and uses the type information to + // select output type + // + // Example: + // sub, cancel, err := eventbus.Subscribe(new(os.Signal)) + // defer cancel() + // + // evt := (<-sub).(os.Signal) // guaranteed to be safe + Subscribe(eventType interface{}) (<-chan interface{}, CancelFunc, error) + + + Emitter(eventType interface{}) (EmitFunc, CancelFunc, error) } -type Emitter interface { - io.Closer - - Emit(event interface{}) -} +// EmitFunc emits events. If any channel subscribed to the topic is blocked, +// calls to EmitFunc will block +// +// Calling this function with wrong event type will cause a panic +type EmitFunc func(event interface{}) +type CancelFunc func() From c56ac377a369580423b77415c1c23a05687c32b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Thu, 13 Jun 2019 10:02:48 +0200 Subject: [PATCH 1297/3965] More tests, benchmarks --- p2p/host/eventbus/basic_test.go | 156 +++++++++++++++++++++++++++++++- 1 file changed, 153 insertions(+), 3 deletions(-) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 4895005d67..7a95c12b7a 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -2,6 +2,7 @@ package event import ( "sync" + "sync/atomic" "testing" "time" ) @@ -106,14 +107,14 @@ func TestClosingRaces(t *testing.T) { wg.Add(subs + emits) - bus := NewBus() + b := NewBus() for i := 0; i < subs; i++ { go func() { lk.RLock() defer lk.RUnlock() - _, cancel, _ := bus.Subscribe(new(EventA)) + _, cancel, _ := b.Subscribe(new(EventA)) time.Sleep(10 * time.Millisecond) cancel() @@ -125,7 +126,7 @@ func TestClosingRaces(t *testing.T) { lk.RLock() defer lk.RUnlock() - _, cancel, _ := bus.Emitter(new(EventA)) + _, cancel, _ := b.Emitter(new(EventA)) time.Sleep(10 * time.Millisecond) cancel() @@ -137,4 +138,153 @@ func TestClosingRaces(t *testing.T) { lk.Unlock() // start everything wg.Wait() + + if len(b.(*bus).nodes) != 0 { + t.Error("expected no nodes") + } +} + +func TestSubMany(t *testing.T) { + bus := NewBus() + + var r int32 + + n := 50000 + var wait sync.WaitGroup + var ready sync.WaitGroup + wait.Add(n) + ready.Add(n) + + for i := 0; i < n; i++ { + go func() { + events, cancel, err := bus.Subscribe(new(EventB)) + if err != nil { + panic(err) + } + defer cancel() + + ready.Done() + atomic.AddInt32(&r, int32((<-events).(EventB))) + wait.Done() + }() + } + + emit, cancel, err := bus.Emitter(new(EventB)) + if err != nil { + t.Fatal(err) + } + defer cancel() + + ready.Wait() + + emit(EventB(7)) + wait.Wait() + + if int(r) != 7 * n { + t.Error("got wrong result") + } +} + +func testMany(t testing.TB, subs, emits, msgs int) { + bus := NewBus() + + var r int64 + + var wait sync.WaitGroup + var ready sync.WaitGroup + wait.Add(subs + emits) + ready.Add(subs) + + for i := 0; i < subs; i++ { + go func() { + events, cancel, err := bus.Subscribe(new(EventB)) + if err != nil { + panic(err) + } + defer cancel() + + ready.Done() + for i := 0; i < emits * msgs; i++ { + atomic.AddInt64(&r, int64((<-events).(EventB))) + } + wait.Done() + }() + } + + for i := 0; i < emits; i++ { + go func() { + emit, cancel, err := bus.Emitter(new(EventB)) + if err != nil { + panic(err) + } + defer cancel() + + ready.Wait() + + for i := 0; i < msgs; i++ { + emit(EventB(97)) + } + + wait.Done() + }() + } + + wait.Wait() + + if int(r) != 97 * subs * emits * msgs { + t.Fatal("got wrong result") + } +} + +func TestBothMany(t *testing.T) { + testMany(t, 10000, 100, 10) +} + +func BenchmarkSubs(b *testing.B) { + b.ReportAllocs() + testMany(b, b.N, 100, 100) +} + +func BenchmarkEmits(b *testing.B) { + b.ReportAllocs() + testMany(b, 100, b.N, 100) +} + +func BenchmarkMsgs(b *testing.B) { + b.ReportAllocs() + testMany(b, 100, 100, b.N) +} + +func BenchmarkOneToMany(b *testing.B) { + b.ReportAllocs() + testMany(b, b.N, 1, 100) +} + +func BenchmarkManyToOne(b *testing.B) { + b.ReportAllocs() + testMany(b, 1, b.N, 100) +} + +func BenchmarkMs1e2m4(b *testing.B) { + b.N = 1000000 + b.ReportAllocs() + testMany(b, 10, 100, 10000) +} + +func BenchmarkMs1e0m6(b *testing.B) { + b.N = 1000000 + b.ReportAllocs() + testMany(b, 10, 1, 1000000) +} + +func BenchmarkMs0e6m0(b *testing.B) { + b.N = 1000000 + b.ReportAllocs() + testMany(b, 1, 1000000, 1) +} + +func BenchmarkMs6e0m0(b *testing.B) { + b.N = 1000000 + b.ReportAllocs() + testMany(b, 1000000, 1, 1) } From fae8970e4951b49438c3e7ae253617136164bf16 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Thu, 13 Jun 2019 10:04:12 +0200 Subject: [PATCH 1298/3965] Placeholder for options in the interface --- p2p/host/eventbus/basic.go | 7 +++++-- p2p/host/eventbus/interface.go | 10 ++++++++-- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 10372b8e8e..9554e734b3 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -67,7 +67,7 @@ func (b *bus) tryDropNode(evtType interface{}) { b.lk.Unlock() } -func (b *bus) Subscribe(evtType interface{}) (s <-chan interface{}, c CancelFunc, err error) { +func (b *bus) Subscribe(evtType interface{}, _ ...SubOption) (s <-chan interface{}, c CancelFunc, err error) { err = b.withNode(evtType, func(n *node) { out, i := n.sub(0) s = out @@ -85,7 +85,7 @@ func (b *bus) Subscribe(evtType interface{}) (s <-chan interface{}, c CancelFunc return } -func (b *bus) Emitter(evtType interface{}) (e EmitFunc, c CancelFunc, err error) { +func (b *bus) Emitter(evtType interface{}, _ ...EmitterOption) (e EmitFunc, c CancelFunc, err error) { err = b.withNode(evtType, func(n *node) { atomic.AddInt32(&n.nEmitters, 1) closed := false @@ -118,6 +118,9 @@ type node struct { nEmitters int32 nSinks int + + // TODO: we could make emit a bit faster by making this into an array, but + // it doesn't seem needed for now sinks map[int]chan interface{} } diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 81150e4d57..80e9585d4c 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -1,5 +1,11 @@ package event +type SubSettings struct {} +type SubOption func(*SubSettings) + +type EmitterSettings struct {} +type EmitterOption func(*EmitterSettings) + type Bus interface { // Subscribe creates new subscription. Failing to drain the incoming channel // will cause publishers to get blocked @@ -12,10 +18,10 @@ type Bus interface { // defer cancel() // // evt := (<-sub).(os.Signal) // guaranteed to be safe - Subscribe(eventType interface{}) (<-chan interface{}, CancelFunc, error) + Subscribe(eventType interface{}, opts ...SubOption) (<-chan interface{}, CancelFunc, error) - Emitter(eventType interface{}) (EmitFunc, CancelFunc, error) + Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, CancelFunc, error) } // EmitFunc emits events. If any channel subscribed to the topic is blocked, From 14497659808e2a77680c9fdd34aeb328d87dd197 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 13 Jun 2019 11:14:44 -0700 Subject: [PATCH 1299/3965] basic_host: ensure we close correctly when the context is canceled As long as we _accept_ a context, we need to obey it. --- p2p/host/basic/basic_host.go | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index b1d8cbcd42..f0eb4ea22d 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -76,8 +76,6 @@ type BasicHost struct { proc goprocess.Process - ctx context.Context - cancel func() mx sync.Mutex lastAddrs []ma.Multiaddr } @@ -121,22 +119,21 @@ type HostOpts struct { // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHost, error) { - bgctx, cancel := context.WithCancel(ctx) - h := &BasicHost{ network: net, mux: msmux.NewMultistreamMuxer(), negtimeout: DefaultNegotiationTimeout, AddrsFactory: DefaultAddrsFactory, maResolver: madns.DefaultResolver, - ctx: bgctx, - cancel: cancel, } h.proc = goprocessctx.WithContextAndTeardown(ctx, func() error { if h.natmgr != nil { h.natmgr.Close() } + if h.cmgr != nil { + h.cmgr.Close() + } return h.Network().Close() }) @@ -148,7 +145,7 @@ func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHo h.ids = opts.IdentifyService } else { // we can't set this as a default above because it depends on the *BasicHost. - h.ids = identify.NewIDService(bgctx, h) + h.ids = identify.NewIDService(goprocessctx.WithProcessClosing(ctx, h.proc), h) } if uint64(opts.NegotiationTimeout) != 0 { @@ -223,7 +220,7 @@ func New(net network.Network, opts ...interface{}) *BasicHost { // Start starts background tasks in the host func (h *BasicHost) Start() { - go h.background() + h.proc.Go(h.background) } // newConnHandler is the remote-opened conn handler for inet.Network @@ -300,7 +297,7 @@ func (h *BasicHost) PushIdentify() { } } -func (h *BasicHost) background() { +func (h *BasicHost) background(p goprocess.Process) { // periodically schedules an IdentifyPush to update our peers for changes // in our address set (if needed) ticker := time.NewTicker(1 * time.Minute) @@ -318,7 +315,7 @@ func (h *BasicHost) background() { case <-ticker.C: h.PushIdentify() - case <-h.ctx.Done(): + case <-p.Closing(): return } } @@ -724,8 +721,6 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { // Close shuts down the Host's services (network, etc). func (h *BasicHost) Close() error { - h.cancel() - h.cmgr.Close() return h.proc.Close() } From b7ad393cac4515b2a3f00b1e86ff9ad81e77f366 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Thu, 13 Jun 2019 22:25:53 +0200 Subject: [PATCH 1300/3965] Address @bigs review --- p2p/host/eventbus/basic.go | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 9554e734b3..7ddf9e1d84 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -40,8 +40,8 @@ func (b *bus) withNode(evtType interface{}, cb func(*node)) error { n.lk.Lock() b.lk.Unlock() + defer n.lk.Unlock() cb(n) - n.lk.Unlock() return nil } @@ -116,8 +116,11 @@ type node struct { typ reflect.Type + // emitter ref count nEmitters int32 - nSinks int + + // sink index counter + sinkC int // TODO: we could make emit a bit faster by making this into an array, but // it doesn't seem needed for now @@ -134,8 +137,8 @@ func newNode(typ reflect.Type) *node { func (n *node) sub(buf int) (chan interface{}, int) { out := make(chan interface{}, buf) - i := n.nSinks - n.nSinks++ + i := n.sinkC + n.sinkC++ n.sinks[i] = out return out, i } From eba14f59e28112cdb1ca481a4bcb5cccee94de5b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Fri, 14 Jun 2019 18:57:21 +0200 Subject: [PATCH 1301/3965] POC SendTo --- p2p/host/eventbus/basic.go | 26 +++++++++++++++++ p2p/host/eventbus/basic_test.go | 50 ++++++++++++++++++++++++++++++++- p2p/host/eventbus/interface.go | 1 + 3 files changed, 76 insertions(+), 1 deletion(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 7ddf9e1d84..f755679979 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -69,6 +69,8 @@ func (b *bus) tryDropNode(evtType interface{}) { func (b *bus) Subscribe(evtType interface{}, _ ...SubOption) (s <-chan interface{}, c CancelFunc, err error) { err = b.withNode(evtType, func(n *node) { + // when all subs are waiting on this channel, setting this to 1 doesn't + // really affect benchmarks out, i := n.sub(0) s = out c = func() { @@ -107,6 +109,30 @@ func (b *bus) Emitter(evtType interface{}, _ ...EmitterOption) (e EmitFunc, c Ca return } +func (b *bus) SendTo(typedChan interface{}) (CancelFunc, error) { + typ := reflect.TypeOf(typedChan) + if typ.Kind() != reflect.Chan { + return nil, errors.New("expected a channel") + } + if typ.ChanDir() & reflect.SendDir == 0 { + return nil, errors.New("channel doesn't allow send") + } + etype := reflect.New(typ.Elem()) + sub, cf, err := b.Subscribe(etype.Interface()) + if err != nil { + return nil, err + } + + go func() { + tcv := reflect.ValueOf(typedChan) + for event := range sub { + tcv.Send(reflect.ValueOf(event)) + } + }() + + return cf, nil +} + /////////////////////// // NODE diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 7a95c12b7a..c8b3de5f97 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -185,6 +185,42 @@ func TestSubMany(t *testing.T) { } } +func TestSendTo(t *testing.T) { + testSendTo(t, 1000) +} + +func testSendTo(t testing.TB, msgs int) { + bus := NewBus() + + go func() { + emit, cancel, err := bus.Emitter(new(EventB)) + if err != nil { + panic(err) + } + defer cancel() + + for i := 0; i < msgs; i++ { + emit(EventB(97)) + } + }() + + ch := make(chan EventB) + cancel, err := bus.SendTo(ch) + if err != nil { + return + } + defer cancel() + + r := 0 + for i := 0; i < msgs; i++ { + r += int(<-ch) + } + + if int(r) != 97 * msgs { + t.Fatal("got wrong result") + } +} + func testMany(t testing.TB, subs, emits, msgs int) { bus := NewBus() @@ -272,11 +308,17 @@ func BenchmarkMs1e2m4(b *testing.B) { } func BenchmarkMs1e0m6(b *testing.B) { - b.N = 1000000 + b.N = 10000000 b.ReportAllocs() testMany(b, 10, 1, 1000000) } +func BenchmarkMs0e0m6(b *testing.B) { + b.N = 1000000 + b.ReportAllocs() + testMany(b, 1, 1, 1000000) +} + func BenchmarkMs0e6m0(b *testing.B) { b.N = 1000000 b.ReportAllocs() @@ -288,3 +330,9 @@ func BenchmarkMs6e0m0(b *testing.B) { b.ReportAllocs() testMany(b, 1000000, 1, 1) } + +func BenchmarkSendTo(b *testing.B) { + b.N = 1000000 + b.ReportAllocs() + testSendTo(b, b.N) +} diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 80e9585d4c..3b505296b4 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -20,6 +20,7 @@ type Bus interface { // evt := (<-sub).(os.Signal) // guaranteed to be safe Subscribe(eventType interface{}, opts ...SubOption) (<-chan interface{}, CancelFunc, error) + SendTo(typedChan interface{}) (CancelFunc, error) Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, CancelFunc, error) } From 056412681f91d96b6c3623c4945d15d0aada2998 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Sun, 16 Jun 2019 17:20:33 +0200 Subject: [PATCH 1302/3965] Subscribe with user provided typed channels --- p2p/host/eventbus/basic.go | 83 +++++++++++++-------------------- p2p/host/eventbus/basic_test.go | 30 ++++++------ p2p/host/eventbus/interface.go | 5 +- 3 files changed, 48 insertions(+), 70 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index f755679979..5d67edf533 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -22,12 +22,7 @@ func NewBus() Bus { } } -func (b *bus) withNode(evtType interface{}, cb func(*node)) error { - typ := reflect.TypeOf(evtType) - if typ.Kind() != reflect.Ptr { - return errors.New("subscribe called with non-pointer type") - } - typ = typ.Elem() +func (b *bus) withNode(typ reflect.Type, cb func(*node)) error { path := typePath(typ) b.lk.Lock() @@ -45,8 +40,8 @@ func (b *bus) withNode(evtType interface{}, cb func(*node)) error { return nil } -func (b *bus) tryDropNode(evtType interface{}) { - path := typePath(reflect.TypeOf(evtType).Elem()) +func (b *bus) tryDropNode(typ reflect.Type) { + path := typePath(typ) b.lk.Lock() n, ok := b.nodes[path] @@ -67,20 +62,27 @@ func (b *bus) tryDropNode(evtType interface{}) { b.lk.Unlock() } -func (b *bus) Subscribe(evtType interface{}, _ ...SubOption) (s <-chan interface{}, c CancelFunc, err error) { - err = b.withNode(evtType, func(n *node) { +func (b *bus) Subscribe(typedChan interface{}, _ ...SubOption) (c CancelFunc, err error) { + refCh := reflect.ValueOf(typedChan) + typ := refCh.Type() + if typ.Kind() != reflect.Chan { + return nil, errors.New("expected a channel") + } + if typ.ChanDir() & reflect.SendDir == 0 { + return nil, errors.New("channel doesn't allow send") + } + + err = b.withNode(typ.Elem(), func(n *node) { // when all subs are waiting on this channel, setting this to 1 doesn't // really affect benchmarks - out, i := n.sub(0) - s = out + i := n.sub(refCh) c = func() { n.lk.Lock() delete(n.sinks, i) - close(out) tryDrop := len(n.sinks) == 0 && n.nEmitters == 0 n.lk.Unlock() if tryDrop { - b.tryDropNode(evtType) + b.tryDropNode(typ.Elem()) } } }) @@ -88,7 +90,13 @@ func (b *bus) Subscribe(evtType interface{}, _ ...SubOption) (s <-chan interface } func (b *bus) Emitter(evtType interface{}, _ ...EmitterOption) (e EmitFunc, c CancelFunc, err error) { - err = b.withNode(evtType, func(n *node) { + typ := reflect.TypeOf(evtType) + if typ.Kind() != reflect.Ptr { + return nil, nil, errors.New("emitter called with non-pointer type") + } + typ = typ.Elem() + + err = b.withNode(typ, func(n *node) { atomic.AddInt32(&n.nEmitters, 1) closed := false @@ -102,37 +110,13 @@ func (b *bus) Emitter(evtType interface{}, _ ...EmitterOption) (e EmitFunc, c Ca c = func() { closed = true if atomic.AddInt32(&n.nEmitters, -1) == 0 { - b.tryDropNode(evtType) + b.tryDropNode(typ) } } }) return } -func (b *bus) SendTo(typedChan interface{}) (CancelFunc, error) { - typ := reflect.TypeOf(typedChan) - if typ.Kind() != reflect.Chan { - return nil, errors.New("expected a channel") - } - if typ.ChanDir() & reflect.SendDir == 0 { - return nil, errors.New("channel doesn't allow send") - } - etype := reflect.New(typ.Elem()) - sub, cf, err := b.Subscribe(etype.Interface()) - if err != nil { - return nil, err - } - - go func() { - tcv := reflect.ValueOf(typedChan) - for event := range sub { - tcv.Send(reflect.ValueOf(event)) - } - }() - - return cf, nil -} - /////////////////////// // NODE @@ -150,34 +134,33 @@ type node struct { // TODO: we could make emit a bit faster by making this into an array, but // it doesn't seem needed for now - sinks map[int]chan interface{} + sinks map[int]reflect.Value } func newNode(typ reflect.Type) *node { return &node{ typ: typ, - sinks: map[int]chan interface{}{}, + sinks: map[int]reflect.Value{}, } } -func (n *node) sub(buf int) (chan interface{}, int) { - out := make(chan interface{}, buf) +func (n *node) sub(outChan reflect.Value) int { i := n.sinkC n.sinkC++ - n.sinks[i] = out - return out, i + n.sinks[i] = outChan + return i } func (n *node) emit(event interface{}) { - etype := reflect.TypeOf(event) - if etype != n.typ { - panic(fmt.Sprintf("Emit called with wrong type. expected: %s, got: %s", n.typ, etype)) + eval := reflect.ValueOf(event) + if eval.Type() != n.typ { + panic(fmt.Sprintf("Emit called with wrong type. expected: %s, got: %s", n.typ, eval.Type())) } n.lk.RLock() for _, ch := range n.sinks { - ch <- event + ch.Send(eval) } n.lk.RUnlock() } diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index c8b3de5f97..f77e5d746e 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -12,7 +12,8 @@ type EventB int func TestEmit(t *testing.T) { bus := NewBus() - events, cancel, err := bus.Subscribe(new(EventA)) + events := make(chan EventA) + cancel, err := bus.Subscribe(events) if err != nil { t.Fatal(err) } @@ -33,7 +34,8 @@ func TestEmit(t *testing.T) { func TestSub(t *testing.T) { bus := NewBus() - events, cancel, err := bus.Subscribe(new(EventB)) + events := make(chan EventB) + cancel, err := bus.Subscribe(events) if err != nil { t.Fatal(err) } @@ -45,7 +47,7 @@ func TestSub(t *testing.T) { go func() { defer cancel() - event = (<-events).(EventB) + event = <-events wait.Done() }() @@ -114,7 +116,7 @@ func TestClosingRaces(t *testing.T) { lk.RLock() defer lk.RUnlock() - _, cancel, _ := b.Subscribe(new(EventA)) + cancel, _ := b.Subscribe(make(chan EventA)) time.Sleep(10 * time.Millisecond) cancel() @@ -157,14 +159,15 @@ func TestSubMany(t *testing.T) { for i := 0; i < n; i++ { go func() { - events, cancel, err := bus.Subscribe(new(EventB)) + events := make(chan EventB) + cancel, err := bus.Subscribe(events) if err != nil { panic(err) } defer cancel() ready.Done() - atomic.AddInt32(&r, int32((<-events).(EventB))) + atomic.AddInt32(&r, int32(<-events)) wait.Done() }() } @@ -185,7 +188,7 @@ func TestSubMany(t *testing.T) { } } -func TestSendTo(t *testing.T) { +/*func TestSendTo(t *testing.T) { testSendTo(t, 1000) } @@ -219,7 +222,7 @@ func testSendTo(t testing.TB, msgs int) { if int(r) != 97 * msgs { t.Fatal("got wrong result") } -} +}*/ func testMany(t testing.TB, subs, emits, msgs int) { bus := NewBus() @@ -233,7 +236,8 @@ func testMany(t testing.TB, subs, emits, msgs int) { for i := 0; i < subs; i++ { go func() { - events, cancel, err := bus.Subscribe(new(EventB)) + events := make(chan EventB) + cancel, err := bus.Subscribe(events) if err != nil { panic(err) } @@ -241,7 +245,7 @@ func testMany(t testing.TB, subs, emits, msgs int) { ready.Done() for i := 0; i < emits * msgs; i++ { - atomic.AddInt64(&r, int64((<-events).(EventB))) + atomic.AddInt64(&r, int64(<-events)) } wait.Done() }() @@ -330,9 +334,3 @@ func BenchmarkMs6e0m0(b *testing.B) { b.ReportAllocs() testMany(b, 1000000, 1, 1) } - -func BenchmarkSendTo(b *testing.B) { - b.N = 1000000 - b.ReportAllocs() - testSendTo(b, b.N) -} diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 3b505296b4..d8b78fe335 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -18,9 +18,7 @@ type Bus interface { // defer cancel() // // evt := (<-sub).(os.Signal) // guaranteed to be safe - Subscribe(eventType interface{}, opts ...SubOption) (<-chan interface{}, CancelFunc, error) - - SendTo(typedChan interface{}) (CancelFunc, error) + Subscribe(typedChan interface{}, opts ...SubOption) (CancelFunc, error) Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, CancelFunc, error) } @@ -32,4 +30,3 @@ type Bus interface { type EmitFunc func(event interface{}) type CancelFunc func() - From e78382b2818285a2d4c868e0d8c578fa036130fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Sun, 16 Jun 2019 17:25:13 +0200 Subject: [PATCH 1303/3965] Set correct B.N on BenchmarkMs1e0m6 --- p2p/host/eventbus/basic_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 7a95c12b7a..55625407f5 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -272,7 +272,7 @@ func BenchmarkMs1e2m4(b *testing.B) { } func BenchmarkMs1e0m6(b *testing.B) { - b.N = 1000000 + b.N = 10000000 b.ReportAllocs() testMany(b, 10, 1, 1000000) } From 25d453370e7663a505a47f277ef33b41436e4297 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Sun, 16 Jun 2019 18:15:35 +0200 Subject: [PATCH 1304/3965] ForceSubType --- p2p/host/eventbus/basic.go | 17 +++++++++++- p2p/host/eventbus/basic_test.go | 48 +++++++++++++++++---------------- p2p/host/eventbus/interface.go | 22 +++++++++++++-- 3 files changed, 61 insertions(+), 26 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 5d67edf533..a2db391bad 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -62,7 +62,14 @@ func (b *bus) tryDropNode(typ reflect.Type) { b.lk.Unlock() } -func (b *bus) Subscribe(typedChan interface{}, _ ...SubOption) (c CancelFunc, err error) { +func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err error) { + var settings SubSettings + for _, opt := range opts { + if err := opt(&settings); err != nil { + return nil, err + } + } + refCh := reflect.ValueOf(typedChan) typ := refCh.Type() if typ.Kind() != reflect.Chan { @@ -72,6 +79,13 @@ func (b *bus) Subscribe(typedChan interface{}, _ ...SubOption) (c CancelFunc, er return nil, errors.New("channel doesn't allow send") } + if settings.forcedType != nil { + if settings.forcedType.Elem().AssignableTo(typ) { + return nil, fmt.Errorf("forced type %s cannot be sent to chan %s", settings.forcedType, typ) + } + typ = settings.forcedType + } + err = b.withNode(typ.Elem(), func(n *node) { // when all subs are waiting on this channel, setting this to 1 doesn't // really affect benchmarks @@ -159,6 +173,7 @@ func (n *node) emit(event interface{}) { } n.lk.RLock() + // TODO: try using reflect.Select for _, ch := range n.sinks { ch.Send(eval) } diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index f77e5d746e..cd4512056e 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -1,6 +1,7 @@ package event import ( + "fmt" "sync" "sync/atomic" "testing" @@ -10,6 +11,10 @@ import ( type EventA struct{} type EventB int +func (EventA) String() string { + return "Oh, Hello" +} + func TestEmit(t *testing.T) { bus := NewBus() events := make(chan EventA) @@ -188,41 +193,38 @@ func TestSubMany(t *testing.T) { } } -/*func TestSendTo(t *testing.T) { - testSendTo(t, 1000) -} - -func testSendTo(t testing.TB, msgs int) { +func TestSubType(t *testing.T) { bus := NewBus() + events := make(chan fmt.Stringer) + cancel, err := bus.Subscribe(events, ForceSubType(new(EventA))) + if err != nil { + t.Fatal(err) + } + + var event fmt.Stringer + + var wait sync.WaitGroup + wait.Add(1) go func() { - emit, cancel, err := bus.Emitter(new(EventB)) - if err != nil { - panic(err) - } defer cancel() - - for i := 0; i < msgs; i++ { - emit(EventB(97)) - } + event = <-events + wait.Done() }() - ch := make(chan EventB) - cancel, err := bus.SendTo(ch) + emit, cancel, err := bus.Emitter(new(EventA)) if err != nil { - return + t.Fatal(err) } defer cancel() - r := 0 - for i := 0; i < msgs; i++ { - r += int(<-ch) - } + emit(EventA{}) + wait.Wait() - if int(r) != 97 * msgs { - t.Fatal("got wrong result") + if event.String() != "Oh, Hello" { + t.Error("didn't get the correct message") } -}*/ +} func testMany(t testing.TB, subs, emits, msgs int) { bus := NewBus() diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index d8b78fe335..fef8f817af 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -1,7 +1,25 @@ package event -type SubSettings struct {} -type SubOption func(*SubSettings) +import ( + "errors" + "reflect" +) + +type SubSettings struct { + forcedType reflect.Type +} +type SubOption func(*SubSettings) error + +func ForceSubType(evtType interface{}) SubOption { + return func(s *SubSettings) error { + typ := reflect.TypeOf(evtType) + if typ.Kind() != reflect.Ptr { + return errors.New("ForceSubType called with non-pointer type") + } + s.forcedType = typ + return nil + } +} type EmitterSettings struct {} type EmitterOption func(*EmitterSettings) From af4305fbb426223c309066bfe9a8e3041b4bee7c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Sun, 16 Jun 2019 19:06:49 +0200 Subject: [PATCH 1305/3965] Gofmt --- p2p/host/eventbus/basic.go | 11 +++++++---- p2p/host/eventbus/basic_test.go | 6 +++--- p2p/host/eventbus/interface.go | 15 +++++++-------- 3 files changed, 17 insertions(+), 15 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index a2db391bad..c5133ed7bc 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -12,7 +12,7 @@ import ( // BUS type bus struct { - lk sync.Mutex + lk sync.Mutex nodes map[string]*node } @@ -75,7 +75,7 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, if typ.Kind() != reflect.Chan { return nil, errors.New("expected a channel") } - if typ.ChanDir() & reflect.SendDir == 0 { + if typ.ChanDir()&reflect.SendDir == 0 { return nil, errors.New("channel doesn't allow send") } @@ -144,11 +144,14 @@ type node struct { nEmitters int32 // sink index counter - sinkC int + sinkC int // TODO: we could make emit a bit faster by making this into an array, but // it doesn't seem needed for now - sinks map[int]reflect.Value + sinks map[int]reflect.Value + + keepLast bool + last reflect.Value } func newNode(typ reflect.Type) *node { diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index cd4512056e..c3014c5fc0 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -188,7 +188,7 @@ func TestSubMany(t *testing.T) { emit(EventB(7)) wait.Wait() - if int(r) != 7 * n { + if int(r) != 7*n { t.Error("got wrong result") } } @@ -246,7 +246,7 @@ func testMany(t testing.TB, subs, emits, msgs int) { defer cancel() ready.Done() - for i := 0; i < emits * msgs; i++ { + for i := 0; i < emits*msgs; i++ { atomic.AddInt64(&r, int64(<-events)) } wait.Done() @@ -273,7 +273,7 @@ func testMany(t testing.TB, subs, emits, msgs int) { wait.Wait() - if int(r) != 97 * subs * emits * msgs { + if int(r) != 97*subs*emits*msgs { t.Fatal("got wrong result") } } diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index fef8f817af..c514e38175 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -21,23 +21,22 @@ func ForceSubType(evtType interface{}) SubOption { } } -type EmitterSettings struct {} +type EmitterSettings struct{} type EmitterOption func(*EmitterSettings) type Bus interface { - // Subscribe creates new subscription. Failing to drain the incoming channel - // will cause publishers to get blocked + // Subscribe creates new subscription. Failing to drain the channel will cause + // publishers to get blocked + Subscribe(typedChan interface{}, opts ...SubOption) (CancelFunc, error) + + // Emitter creates new emitter // - // evtTypes only accepts typed nil pointers, and uses the type information to + // eventType accepts typed nil pointers, and uses the type information to // select output type // // Example: // sub, cancel, err := eventbus.Subscribe(new(os.Signal)) // defer cancel() - // - // evt := (<-sub).(os.Signal) // guaranteed to be safe - Subscribe(typedChan interface{}, opts ...SubOption) (CancelFunc, error) - Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, CancelFunc, error) } From 1e198f4825286ab16d0daf8e4d72e3186c7cb762 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Sun, 16 Jun 2019 19:06:16 +0200 Subject: [PATCH 1306/3965] Use slice instead of map License: MIT Signed-off-by: Jakub Sztandera --- p2p/host/eventbus/basic.go | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index c5133ed7bc..9ebde0d2f0 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -89,10 +89,17 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err = b.withNode(typ.Elem(), func(n *node) { // when all subs are waiting on this channel, setting this to 1 doesn't // really affect benchmarks - i := n.sub(refCh) + n.sub(refCh) + c = func() { n.lk.Lock() - delete(n.sinks, i) + for i := 0; i < len(n.sinks); i++ { + if n.sinks[i] == refCh { + n.sinks[i] = n.sinks[len(n.sinks)-1] + n.sinks = n.sinks[:len(n.sinks)-1] + break + } + } tryDrop := len(n.sinks) == 0 && n.nEmitters == 0 n.lk.Unlock() if tryDrop { @@ -143,30 +150,20 @@ type node struct { // emitter ref count nEmitters int32 - // sink index counter - sinkC int - - // TODO: we could make emit a bit faster by making this into an array, but - // it doesn't seem needed for now - sinks map[int]reflect.Value - keepLast bool last reflect.Value + + sinks []reflect.Value } func newNode(typ reflect.Type) *node { return &node{ typ: typ, - - sinks: map[int]reflect.Value{}, } } -func (n *node) sub(outChan reflect.Value) int { - i := n.sinkC - n.sinkC++ - n.sinks[i] = outChan - return i +func (n *node) sub(outChan reflect.Value) { + n.sinks = append(n.sinks, outChan) } func (n *node) emit(event interface{}) { From db5502986dc2eff7784f4e31ba56ee57f8cae52c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Sun, 16 Jun 2019 21:42:47 +0200 Subject: [PATCH 1307/3965] Stateful emits --- p2p/host/eventbus/basic.go | 51 +++++++++++---- p2p/host/eventbus/basic_test.go | 108 ++++++++++++++++++++++++++++---- p2p/host/eventbus/interface.go | 8 ++- 3 files changed, 140 insertions(+), 27 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 9ebde0d2f0..7b6a7b4bf2 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -22,7 +22,7 @@ func NewBus() Bus { } } -func (b *bus) withNode(typ reflect.Type, cb func(*node)) error { +func (b *bus) withNode(typ reflect.Type, cb func(*node), async func(*node)) error { path := typePath(typ) b.lk.Lock() @@ -35,8 +35,21 @@ func (b *bus) withNode(typ reflect.Type, cb func(*node)) error { n.lk.Lock() b.lk.Unlock() - defer n.lk.Unlock() + + ok = false + defer func() { + if !ok { + n.lk.Unlock() + } + go func() { + defer n.lk.Unlock() + async(n) + }() + }() + cb(n) + ok = true + return nil } @@ -87,10 +100,7 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, } err = b.withNode(typ.Elem(), func(n *node) { - // when all subs are waiting on this channel, setting this to 1 doesn't - // really affect benchmarks - n.sub(refCh) - + n.sinks = append(n.sinks, refCh) c = func() { n.lk.Lock() for i := 0; i < len(n.sinks); i++ { @@ -106,11 +116,25 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, b.tryDropNode(typ.Elem()) } } + }, func(n *node) { + if n.keepLast { + lastVal, ok := n.last.Load().(reflect.Value) + if !ok { + return + } + + refCh.Send(lastVal) + } }) return } -func (b *bus) Emitter(evtType interface{}, _ ...EmitterOption) (e EmitFunc, c CancelFunc, err error) { +func (b *bus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, c CancelFunc, err error) { + var settings EmitterSettings + for _, opt := range opts { + opt(&settings) + } + typ := reflect.TypeOf(evtType) if typ.Kind() != reflect.Ptr { return nil, nil, errors.New("emitter called with non-pointer type") @@ -120,6 +144,7 @@ func (b *bus) Emitter(evtType interface{}, _ ...EmitterOption) (e EmitFunc, c Ca err = b.withNode(typ, func(n *node) { atomic.AddInt32(&n.nEmitters, 1) closed := false + n.keepLast = n.keepLast || settings.makeStateful e = func(event interface{}) { if closed { @@ -134,7 +159,7 @@ func (b *bus) Emitter(evtType interface{}, _ ...EmitterOption) (e EmitFunc, c Ca b.tryDropNode(typ) } } - }) + }, func(_ *node) {}) return } @@ -151,7 +176,7 @@ type node struct { nEmitters int32 keepLast bool - last reflect.Value + last atomic.Value sinks []reflect.Value } @@ -162,10 +187,6 @@ func newNode(typ reflect.Type) *node { } } -func (n *node) sub(outChan reflect.Value) { - n.sinks = append(n.sinks, outChan) -} - func (n *node) emit(event interface{}) { eval := reflect.ValueOf(event) if eval.Type() != n.typ { @@ -173,6 +194,10 @@ func (n *node) emit(event interface{}) { } n.lk.RLock() + if n.keepLast { + n.last.Store(eval) + } + // TODO: try using reflect.Select for _, ch := range n.sinks { ch.Send(eval) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index c3014c5fc0..77e9c9343d 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -226,7 +226,75 @@ func TestSubType(t *testing.T) { } } -func testMany(t testing.TB, subs, emits, msgs int) { +func TestNonStateful(t *testing.T) { + bus := NewBus() + emit, cancelE1, err := bus.Emitter(new(EventB)) + if err != nil { + t.Fatal(err) + } + defer cancelE1() + + eventsA := make(chan EventB, 1) + cancelS, err := bus.Subscribe(eventsA) + if err != nil { + t.Fatal(err) + } + defer cancelS() + + select { + case <-eventsA: + t.Fatal("didn't expect to get an event") + default: + } + + emit(EventB(1)) + + select { + case e := <-eventsA: + if e != 1 { + t.Fatal("got wrong event") + } + default: + t.Fatal("expected to get an event") + } + + eventsB := make(chan EventB, 1) + cancelS2, err := bus.Subscribe(eventsB) + if err != nil { + t.Fatal(err) + } + defer cancelS2() + + select { + case <-eventsA: + t.Fatal("didn't expect to get an event") + default: + } +} + +func TestStateful(t *testing.T) { + bus := NewBus() + emit, cancelE1, err := bus.Emitter(new(EventB), Stateful) + if err != nil { + t.Fatal(err) + } + defer cancelE1() + + emit(EventB(2)) + + eventsA := make(chan EventB, 1) + cancelS, err := bus.Subscribe(eventsA) + if err != nil { + t.Fatal(err) + } + defer cancelS() + + if <-eventsA != 2 { + t.Fatal("got wrong event") + } +} + +func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { bus := NewBus() var r int64 @@ -255,7 +323,9 @@ func testMany(t testing.TB, subs, emits, msgs int) { for i := 0; i < emits; i++ { go func() { - emit, cancel, err := bus.Emitter(new(EventB)) + emit, cancel, err := bus.Emitter(new(EventB), func(settings *EmitterSettings) { + settings.makeStateful = stateful + }) if err != nil { panic(err) } @@ -279,60 +349,72 @@ func testMany(t testing.TB, subs, emits, msgs int) { } func TestBothMany(t *testing.T) { - testMany(t, 10000, 100, 10) + testMany(t, 10000, 100, 10, false) } func BenchmarkSubs(b *testing.B) { b.ReportAllocs() - testMany(b, b.N, 100, 100) + testMany(b, b.N, 100, 100, false) } func BenchmarkEmits(b *testing.B) { b.ReportAllocs() - testMany(b, 100, b.N, 100) + testMany(b, 100, b.N, 100, false) } func BenchmarkMsgs(b *testing.B) { b.ReportAllocs() - testMany(b, 100, 100, b.N) + testMany(b, 100, 100, b.N, false) } func BenchmarkOneToMany(b *testing.B) { b.ReportAllocs() - testMany(b, b.N, 1, 100) + testMany(b, b.N, 1, 100, false) } func BenchmarkManyToOne(b *testing.B) { b.ReportAllocs() - testMany(b, 1, b.N, 100) + testMany(b, 1, b.N, 100, false) } func BenchmarkMs1e2m4(b *testing.B) { b.N = 1000000 b.ReportAllocs() - testMany(b, 10, 100, 10000) + testMany(b, 10, 100, 10000, false) } func BenchmarkMs1e0m6(b *testing.B) { b.N = 10000000 b.ReportAllocs() - testMany(b, 10, 1, 1000000) + testMany(b, 10, 1, 1000000, false) } func BenchmarkMs0e0m6(b *testing.B) { b.N = 1000000 b.ReportAllocs() - testMany(b, 1, 1, 1000000) + testMany(b, 1, 1, 1000000, false) +} + +func BenchmarkStatefulMs1e0m6(b *testing.B) { + b.N = 10000000 + b.ReportAllocs() + testMany(b, 10, 1, 1000000, true) +} + +func BenchmarkStatefulMs0e0m6(b *testing.B) { + b.N = 1000000 + b.ReportAllocs() + testMany(b, 1, 1, 1000000, true) } func BenchmarkMs0e6m0(b *testing.B) { b.N = 1000000 b.ReportAllocs() - testMany(b, 1, 1000000, 1) + testMany(b, 1, 1000000, 1, false) } func BenchmarkMs6e0m0(b *testing.B) { b.N = 1000000 b.ReportAllocs() - testMany(b, 1000000, 1, 1) + testMany(b, 1000000, 1, 1, false) } diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index c514e38175..c316756ea3 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -21,9 +21,15 @@ func ForceSubType(evtType interface{}) SubOption { } } -type EmitterSettings struct{} +type EmitterSettings struct{ + makeStateful bool +} type EmitterOption func(*EmitterSettings) +func Stateful(s *EmitterSettings) { + s.makeStateful = true +} + type Bus interface { // Subscribe creates new subscription. Failing to drain the channel will cause // publishers to get blocked From c653488de9ada31faff8a7a67924b9fba273a6bf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 12:13:18 +0200 Subject: [PATCH 1308/3965] Fix data races --- p2p/host/eventbus/basic.go | 4 ++-- p2p/host/eventbus/basic_test.go | 32 +++++++++++++++++++++++--------- 2 files changed, 25 insertions(+), 11 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 7b6a7b4bf2..6dc2b2250e 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -64,7 +64,7 @@ func (b *bus) tryDropNode(typ reflect.Type) { } n.lk.Lock() - if n.nEmitters > 0 || len(n.sinks) > 0 { + if atomic.LoadInt32(&n.nEmitters) > 0 || len(n.sinks) > 0 { n.lk.Unlock() b.lk.Unlock() return // still in use @@ -110,7 +110,7 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, break } } - tryDrop := len(n.sinks) == 0 && n.nEmitters == 0 + tryDrop := len(n.sinks) == 0 && atomic.LoadInt32(&n.nEmitters) == 0 n.lk.Unlock() if tryDrop { b.tryDropNode(typ.Elem()) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 77e9c9343d..72a68713c4 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -6,11 +6,21 @@ import ( "sync/atomic" "testing" "time" + + "github.com/jbenet/go-detect-race" ) type EventA struct{} type EventB int +func getN() int { + n := 50000 + if detectrace.WithRace() { + n = 1000 + } + return n +} + func (EventA) String() string { return "Oh, Hello" } @@ -28,11 +38,11 @@ func TestEmit(t *testing.T) { <-events }() - emit, cancel, err := bus.Emitter(new(EventA)) + emit, cancel2, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - defer cancel() + defer cancel2() emit(EventA{}) } @@ -56,11 +66,11 @@ func TestSub(t *testing.T) { wait.Done() }() - emit, cancel, err := bus.Emitter(new(EventB)) + emit, cancel2, err := bus.Emitter(new(EventB)) if err != nil { t.Fatal(err) } - defer cancel() + defer cancel2() emit(EventB(7)) wait.Wait() @@ -105,8 +115,8 @@ func TestEmitOnClosed(t *testing.T) { } func TestClosingRaces(t *testing.T) { - subs := 50000 - emits := 50000 + subs := getN() + emits := getN() var wg sync.WaitGroup var lk sync.RWMutex @@ -156,7 +166,7 @@ func TestSubMany(t *testing.T) { var r int32 - n := 50000 + n := getN() var wait sync.WaitGroup var ready sync.WaitGroup wait.Add(n) @@ -212,11 +222,11 @@ func TestSubType(t *testing.T) { wait.Done() }() - emit, cancel, err := bus.Emitter(new(EventA)) + emit, cancel2, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - defer cancel() + defer cancel2() emit(EventA{}) wait.Wait() @@ -295,6 +305,10 @@ func TestStateful(t *testing.T) { } func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { + if detectrace.WithRace() && subs + emits > 5000 { + t.SkipNow() + } + bus := NewBus() var r int64 From e8cf02cfdfb6ed20c128962d6ded5ca723dbf137 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 12:31:36 +0200 Subject: [PATCH 1309/3965] Simplify withNode a bit --- p2p/host/eventbus/basic.go | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 6dc2b2250e..12e746f3c4 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -36,19 +36,12 @@ func (b *bus) withNode(typ reflect.Type, cb func(*node), async func(*node)) erro n.lk.Lock() b.lk.Unlock() - ok = false - defer func() { - if !ok { - n.lk.Unlock() - } - go func() { - defer n.lk.Unlock() - async(n) - }() - }() - cb(n) - ok = true + + go func() { + defer n.lk.Unlock() + async(n) + }() return nil } From c476ef625712f1564f2e6a1415a6c713522d4546 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 12:46:59 +0200 Subject: [PATCH 1310/3965] Some more docs --- p2p/host/eventbus/interface.go | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index c316756ea3..477d84b9a9 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -33,6 +33,12 @@ func Stateful(s *EmitterSettings) { type Bus interface { // Subscribe creates new subscription. Failing to drain the channel will cause // publishers to get blocked + // + // Example: + // ch := make(chan EventT, 10) + // defer close(ch) + // cancel, err := eventbus.Subscribe(ch) + // defer cancel() Subscribe(typedChan interface{}, opts ...SubOption) (CancelFunc, error) // Emitter creates new emitter @@ -41,8 +47,10 @@ type Bus interface { // select output type // // Example: - // sub, cancel, err := eventbus.Subscribe(new(os.Signal)) + // emit, cancel, err := eventbus.Emitter(new(EventT)) // defer cancel() + // + // emit(EventT{}) Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, CancelFunc, error) } From 21e43859b6f95b55da6381d80e92c5d15e3d1b5e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 13:51:25 +0200 Subject: [PATCH 1311/3965] Use reflect.Type in node map --- p2p/host/eventbus/basic.go | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 12e746f3c4..8a5c374ba4 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -13,24 +13,22 @@ import ( type bus struct { lk sync.Mutex - nodes map[string]*node + nodes map[reflect.Type]*node } func NewBus() Bus { return &bus{ - nodes: map[string]*node{}, + nodes: map[reflect.Type]*node{}, } } func (b *bus) withNode(typ reflect.Type, cb func(*node), async func(*node)) error { - path := typePath(typ) - b.lk.Lock() - n, ok := b.nodes[path] + n, ok := b.nodes[typ] if !ok { n = newNode(typ) - b.nodes[path] = n + b.nodes[typ] = n } n.lk.Lock() @@ -47,10 +45,8 @@ func (b *bus) withNode(typ reflect.Type, cb func(*node), async func(*node)) erro } func (b *bus) tryDropNode(typ reflect.Type) { - path := typePath(typ) - b.lk.Lock() - n, ok := b.nodes[path] + n, ok := b.nodes[typ] if !ok { // already dropped b.lk.Unlock() return @@ -64,7 +60,7 @@ func (b *bus) tryDropNode(typ reflect.Type) { } n.lk.Unlock() - delete(b.nodes, path) + delete(b.nodes, typ) b.lk.Unlock() } @@ -201,8 +197,4 @@ func (n *node) emit(event interface{}) { /////////////////////// // UTILS -func typePath(t reflect.Type) string { - return t.PkgPath() + "/" + t.String() -} - var _ Bus = &bus{} From 3ed7fb3bd296278717ffe4f695cc52d62b62bbd3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 14:13:48 +0200 Subject: [PATCH 1312/3965] Document options --- p2p/host/eventbus/interface.go | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 477d84b9a9..5953836399 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -10,6 +10,22 @@ type SubSettings struct { } type SubOption func(*SubSettings) error +// ForceSubType is a Subscribe option which overrides the type to which +// the subscription will be done. Note that the evtType must be assignable +// to channel type. +// +// This also allows for subscribing to multiple eventbus channels with one +// Go channel to get better ordering guarantees. +// +// Example: +// type Event struct{} +// func (Event) String() string { +// return "event" +// } +// +// eventCh := make(chan fmt.Stringer) // interface { String() string } +// cancel, err := eventbus.Subscribe(eventCh, event.ForceSubType(new(Event))) +// [...] func ForceSubType(evtType interface{}) SubOption { return func(s *SubSettings) error { typ := reflect.TypeOf(evtType) @@ -26,10 +42,18 @@ type EmitterSettings struct{ } type EmitterOption func(*EmitterSettings) +// Stateful is an Emitter option which makes makes the eventbus channel +// 'remember' last event sent, and when a new subscriber joins the +// bus, the remembered event is immediately sent to the subscription +// channel. +// +// This allows to provide state tracking for dynamic systems, and/or +// allows new subscribers to verify that there are Emitters on the channel func Stateful(s *EmitterSettings) { s.makeStateful = true } +// Bus is an interface to type-based event delivery system type Bus interface { // Subscribe creates new subscription. Failing to drain the channel will cause // publishers to get blocked From beb94094e90a3cb25977ba91c16b92ed2a683930 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 14:22:10 +0200 Subject: [PATCH 1313/3965] Address @stebalien's review --- p2p/host/eventbus/basic.go | 7 +++---- p2p/host/eventbus/basic_test.go | 4 ++-- p2p/host/eventbus/interface.go | 15 ++++++++------- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 8a5c374ba4..5b06da6788 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -65,7 +65,7 @@ func (b *bus) tryDropNode(typ reflect.Type) { } func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err error) { - var settings SubSettings + var settings subSettings for _, opt := range opts { if err := opt(&settings); err != nil { return nil, err @@ -94,7 +94,7 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, n.lk.Lock() for i := 0; i < len(n.sinks); i++ { if n.sinks[i] == refCh { - n.sinks[i] = n.sinks[len(n.sinks)-1] + n.sinks[i], n.sinks[len(n.sinks)-1] = n.sinks[len(n.sinks)-1], reflect.Value{} n.sinks = n.sinks[:len(n.sinks)-1] break } @@ -119,7 +119,7 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, } func (b *bus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, c CancelFunc, err error) { - var settings EmitterSettings + var settings emitterSettings for _, opt := range opts { opt(&settings) } @@ -187,7 +187,6 @@ func (n *node) emit(event interface{}) { n.last.Store(eval) } - // TODO: try using reflect.Select for _, ch := range n.sinks { ch.Send(eval) } diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 72a68713c4..14ca4083bd 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -305,7 +305,7 @@ func TestStateful(t *testing.T) { } func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { - if detectrace.WithRace() && subs + emits > 5000 { + if detectrace.WithRace() && subs+emits > 5000 { t.SkipNow() } @@ -337,7 +337,7 @@ func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { for i := 0; i < emits; i++ { go func() { - emit, cancel, err := bus.Emitter(new(EventB), func(settings *EmitterSettings) { + emit, cancel, err := bus.Emitter(new(EventB), func(settings *emitterSettings) { settings.makeStateful = stateful }) if err != nil { diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 5953836399..96fb06034f 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -5,10 +5,10 @@ import ( "reflect" ) -type SubSettings struct { +type subSettings struct { forcedType reflect.Type } -type SubOption func(*SubSettings) error +type SubOption func(*subSettings) error // ForceSubType is a Subscribe option which overrides the type to which // the subscription will be done. Note that the evtType must be assignable @@ -27,7 +27,7 @@ type SubOption func(*SubSettings) error // cancel, err := eventbus.Subscribe(eventCh, event.ForceSubType(new(Event))) // [...] func ForceSubType(evtType interface{}) SubOption { - return func(s *SubSettings) error { + return func(s *subSettings) error { typ := reflect.TypeOf(evtType) if typ.Kind() != reflect.Ptr { return errors.New("ForceSubType called with non-pointer type") @@ -37,10 +37,10 @@ func ForceSubType(evtType interface{}) SubOption { } } -type EmitterSettings struct{ +type emitterSettings struct { makeStateful bool } -type EmitterOption func(*EmitterSettings) +type EmitterOption func(*emitterSettings) // Stateful is an Emitter option which makes makes the eventbus channel // 'remember' last event sent, and when a new subscriber joins the @@ -49,14 +49,15 @@ type EmitterOption func(*EmitterSettings) // // This allows to provide state tracking for dynamic systems, and/or // allows new subscribers to verify that there are Emitters on the channel -func Stateful(s *EmitterSettings) { +func Stateful(s *emitterSettings) { s.makeStateful = true } // Bus is an interface to type-based event delivery system type Bus interface { // Subscribe creates new subscription. Failing to drain the channel will cause - // publishers to get blocked + // publishers to get blocked. CancelFunc is guaranteed to return after last send + // to the channel // // Example: // ch := make(chan EventT, 10) From 33ccec5392f85b71d07d68b4ef4ee1e1ee32c0d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 14:27:37 +0200 Subject: [PATCH 1314/3965] Move close to EmitFunc --- p2p/host/eventbus/basic.go | 18 +++++++-------- p2p/host/eventbus/basic_test.go | 40 ++++++++++++++++----------------- p2p/host/eventbus/interface.go | 8 ++++++- 3 files changed, 36 insertions(+), 30 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 5b06da6788..8ff9deef98 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -118,7 +118,7 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, return } -func (b *bus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, c CancelFunc, err error) { +func (b *bus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, err error) { var settings emitterSettings for _, opt := range opts { opt(&settings) @@ -126,7 +126,7 @@ func (b *bus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, c typ := reflect.TypeOf(evtType) if typ.Kind() != reflect.Ptr { - return nil, nil, errors.New("emitter called with non-pointer type") + return nil, errors.New("emitter called with non-pointer type") } typ = typ.Elem() @@ -139,14 +139,14 @@ func (b *bus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, c if closed { panic("emitter is closed") } - n.emit(event) - } - - c = func() { - closed = true - if atomic.AddInt32(&n.nEmitters, -1) == 0 { - b.tryDropNode(typ) + if event == closeEmit { + closed = true + if atomic.AddInt32(&n.nEmitters, -1) == 0 { + b.tryDropNode(typ) + } + return } + n.emit(event) } }, func(_ *node) {}) return diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 14ca4083bd..941c574064 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -38,11 +38,11 @@ func TestEmit(t *testing.T) { <-events }() - emit, cancel2, err := bus.Emitter(new(EventA)) + emit, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - defer cancel2() + defer emit.Close() emit(EventA{}) } @@ -66,11 +66,11 @@ func TestSub(t *testing.T) { wait.Done() }() - emit, cancel2, err := bus.Emitter(new(EventB)) + emit, err := bus.Emitter(new(EventB)) if err != nil { t.Fatal(err) } - defer cancel2() + defer emit.Close() emit(EventB(7)) wait.Wait() @@ -83,11 +83,11 @@ func TestSub(t *testing.T) { func TestEmitNoSubNoBlock(t *testing.T) { bus := NewBus() - emit, cancel, err := bus.Emitter(new(EventA)) + emit, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - defer cancel() + defer emit.Close() emit(EventA{}) } @@ -95,11 +95,11 @@ func TestEmitNoSubNoBlock(t *testing.T) { func TestEmitOnClosed(t *testing.T) { bus := NewBus() - emit, cancel, err := bus.Emitter(new(EventA)) + emit, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - cancel() + emit.Close() defer func() { r := recover() @@ -143,9 +143,9 @@ func TestClosingRaces(t *testing.T) { lk.RLock() defer lk.RUnlock() - _, cancel, _ := b.Emitter(new(EventA)) + emit, _ := b.Emitter(new(EventA)) time.Sleep(10 * time.Millisecond) - cancel() + emit.Close() wg.Done() }() @@ -187,11 +187,11 @@ func TestSubMany(t *testing.T) { }() } - emit, cancel, err := bus.Emitter(new(EventB)) + emit, err := bus.Emitter(new(EventB)) if err != nil { t.Fatal(err) } - defer cancel() + defer emit.Close() ready.Wait() @@ -222,11 +222,11 @@ func TestSubType(t *testing.T) { wait.Done() }() - emit, cancel2, err := bus.Emitter(new(EventA)) + emit, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - defer cancel2() + defer emit.Close() emit(EventA{}) wait.Wait() @@ -238,11 +238,11 @@ func TestSubType(t *testing.T) { func TestNonStateful(t *testing.T) { bus := NewBus() - emit, cancelE1, err := bus.Emitter(new(EventB)) + emit, err := bus.Emitter(new(EventB)) if err != nil { t.Fatal(err) } - defer cancelE1() + defer emit.Close() eventsA := make(chan EventB, 1) cancelS, err := bus.Subscribe(eventsA) @@ -284,11 +284,11 @@ func TestNonStateful(t *testing.T) { func TestStateful(t *testing.T) { bus := NewBus() - emit, cancelE1, err := bus.Emitter(new(EventB), Stateful) + emit, err := bus.Emitter(new(EventB), Stateful) if err != nil { t.Fatal(err) } - defer cancelE1() + defer emit.Close() emit(EventB(2)) @@ -337,13 +337,13 @@ func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { for i := 0; i < emits; i++ { go func() { - emit, cancel, err := bus.Emitter(new(EventB), func(settings *emitterSettings) { + emit, err := bus.Emitter(new(EventB), func(settings *emitterSettings) { settings.makeStateful = stateful }) if err != nil { panic(err) } - defer cancel() + defer emit.Close() ready.Wait() diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 96fb06034f..27386bef4b 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -5,6 +5,8 @@ import ( "reflect" ) +var closeEmit struct{} + type subSettings struct { forcedType reflect.Type } @@ -76,7 +78,7 @@ type Bus interface { // defer cancel() // // emit(EventT{}) - Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, CancelFunc, error) + Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, error) } // EmitFunc emits events. If any channel subscribed to the topic is blocked, @@ -85,4 +87,8 @@ type Bus interface { // Calling this function with wrong event type will cause a panic type EmitFunc func(event interface{}) +func (f EmitFunc) Close() { + f(closeEmit) +} + type CancelFunc func() From e7eaa60b4509e25f411181b45abbf310dcf191b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 14:32:55 +0200 Subject: [PATCH 1315/3965] Update Emitter docs --- p2p/host/eventbus/interface.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 27386bef4b..8f3c42ba3a 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -74,8 +74,8 @@ type Bus interface { // select output type // // Example: - // emit, cancel, err := eventbus.Emitter(new(EventT)) - // defer cancel() + // emit, err := eventbus.Emitter(new(EventT)) + // defer emit.Close() // MUST call this after being done with the emitter // // emit(EventT{}) Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, error) From 8a6f8f9f8820645b92d29218e3bf4a081afe92a3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 14:48:41 +0200 Subject: [PATCH 1316/3965] return struct from NewBus --- p2p/host/eventbus/basic.go | 16 ++++++++-------- p2p/host/eventbus/basic_test.go | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 8ff9deef98..4ebd8e8a2f 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -11,18 +11,18 @@ import ( /////////////////////// // BUS -type bus struct { +type BasicBus struct { lk sync.Mutex nodes map[reflect.Type]*node } func NewBus() Bus { - return &bus{ + return &BasicBus{ nodes: map[reflect.Type]*node{}, } } -func (b *bus) withNode(typ reflect.Type, cb func(*node), async func(*node)) error { +func (b *BasicBus) withNode(typ reflect.Type, cb func(*node), async func(*node)) error { b.lk.Lock() n, ok := b.nodes[typ] @@ -44,7 +44,7 @@ func (b *bus) withNode(typ reflect.Type, cb func(*node), async func(*node)) erro return nil } -func (b *bus) tryDropNode(typ reflect.Type) { +func (b *BasicBus) tryDropNode(typ reflect.Type) { b.lk.Lock() n, ok := b.nodes[typ] if !ok { // already dropped @@ -64,7 +64,7 @@ func (b *bus) tryDropNode(typ reflect.Type) { b.lk.Unlock() } -func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err error) { +func (b *BasicBus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err error) { var settings subSettings for _, opt := range opts { if err := opt(&settings); err != nil { @@ -118,7 +118,7 @@ func (b *bus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, return } -func (b *bus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, err error) { +func (b *BasicBus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, err error) { var settings emitterSettings for _, opt := range opts { opt(&settings) @@ -156,7 +156,7 @@ func (b *bus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, e // NODE type node struct { - // Note: make sure to NEVER lock bus.lk when this lock is held + // Note: make sure to NEVER lock BasicBus.lk when this lock is held lk sync.RWMutex typ reflect.Type @@ -196,4 +196,4 @@ func (n *node) emit(event interface{}) { /////////////////////// // UTILS -var _ Bus = &bus{} +var _ Bus = &BasicBus{} diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 941c574064..7144006019 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -156,7 +156,7 @@ func TestClosingRaces(t *testing.T) { wg.Wait() - if len(b.(*bus).nodes) != 0 { + if len(b.(*BasicBus).nodes) != 0 { t.Error("expected no nodes") } } From e4cf01bb288ac0a5d53c81e4de06e39937a9be6b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 15:00:22 +0200 Subject: [PATCH 1317/3965] BasicBus to basicBus --- p2p/host/eventbus/basic.go | 18 +++++++++--------- p2p/host/eventbus/basic_test.go | 2 +- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 4ebd8e8a2f..ae5af16866 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -11,18 +11,18 @@ import ( /////////////////////// // BUS -type BasicBus struct { +type basicBus struct { lk sync.Mutex nodes map[reflect.Type]*node } -func NewBus() Bus { - return &BasicBus{ +func NewBus() *basicBus { + return &basicBus{ nodes: map[reflect.Type]*node{}, } } -func (b *BasicBus) withNode(typ reflect.Type, cb func(*node), async func(*node)) error { +func (b *basicBus) withNode(typ reflect.Type, cb func(*node), async func(*node)) error { b.lk.Lock() n, ok := b.nodes[typ] @@ -44,7 +44,7 @@ func (b *BasicBus) withNode(typ reflect.Type, cb func(*node), async func(*node)) return nil } -func (b *BasicBus) tryDropNode(typ reflect.Type) { +func (b *basicBus) tryDropNode(typ reflect.Type) { b.lk.Lock() n, ok := b.nodes[typ] if !ok { // already dropped @@ -64,7 +64,7 @@ func (b *BasicBus) tryDropNode(typ reflect.Type) { b.lk.Unlock() } -func (b *BasicBus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err error) { +func (b *basicBus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err error) { var settings subSettings for _, opt := range opts { if err := opt(&settings); err != nil { @@ -118,7 +118,7 @@ func (b *BasicBus) Subscribe(typedChan interface{}, opts ...SubOption) (c Cancel return } -func (b *BasicBus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, err error) { +func (b *basicBus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, err error) { var settings emitterSettings for _, opt := range opts { opt(&settings) @@ -156,7 +156,7 @@ func (b *BasicBus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFu // NODE type node struct { - // Note: make sure to NEVER lock BasicBus.lk when this lock is held + // Note: make sure to NEVER lock basicBus.lk when this lock is held lk sync.RWMutex typ reflect.Type @@ -196,4 +196,4 @@ func (n *node) emit(event interface{}) { /////////////////////// // UTILS -var _ Bus = &BasicBus{} +var _ Bus = &basicBus{} diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 7144006019..af05a5cbe0 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -156,7 +156,7 @@ func TestClosingRaces(t *testing.T) { wg.Wait() - if len(b.(*BasicBus).nodes) != 0 { + if len(b.nodes) != 0 { t.Error("expected no nodes") } } From 1831efdf6c57fd69cc028075f1b47b64a79f2702 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 19 Jun 2019 14:08:16 +0100 Subject: [PATCH 1318/3965] generify options. --- p2p/host/eventbus/interface.go | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/interface.go index 8f3c42ba3a..bff5d46f47 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/interface.go @@ -10,7 +10,8 @@ var closeEmit struct{} type subSettings struct { forcedType reflect.Type } -type SubOption func(*subSettings) error + +type SubOption func(interface{}) error // ForceSubType is a Subscribe option which overrides the type to which // the subscription will be done. Note that the evtType must be assignable @@ -29,7 +30,8 @@ type SubOption func(*subSettings) error // cancel, err := eventbus.Subscribe(eventCh, event.ForceSubType(new(Event))) // [...] func ForceSubType(evtType interface{}) SubOption { - return func(s *subSettings) error { + return func(settings interface{}) error { + s := settings.(*subSettings) typ := reflect.TypeOf(evtType) if typ.Kind() != reflect.Ptr { return errors.New("ForceSubType called with non-pointer type") @@ -42,7 +44,7 @@ func ForceSubType(evtType interface{}) SubOption { type emitterSettings struct { makeStateful bool } -type EmitterOption func(*emitterSettings) +type EmitterOption func(interface{}) error // Stateful is an Emitter option which makes makes the eventbus channel // 'remember' last event sent, and when a new subscriber joins the From ee0cf463ce13d6a3a44c154cfabf3e8d27b5e939 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Wed, 19 Jun 2019 15:36:31 +0200 Subject: [PATCH 1319/3965] Drop the interface --- p2p/host/eventbus/basic.go | 36 ++++++++++++++++- p2p/host/eventbus/basic_test.go | 5 ++- p2p/host/eventbus/{interface.go => opts.go} | 45 ++------------------- 3 files changed, 40 insertions(+), 46 deletions(-) rename p2p/host/eventbus/{interface.go => opts.go} (54%) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index ae5af16866..9f075aaf55 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -11,6 +11,7 @@ import ( /////////////////////// // BUS +// basicBus is a type-based event delivery system type basicBus struct { lk sync.Mutex nodes map[reflect.Type]*node @@ -64,6 +65,15 @@ func (b *basicBus) tryDropNode(typ reflect.Type) { b.lk.Unlock() } +// Subscribe creates new subscription. Failing to drain the channel will cause +// publishers to get blocked. CancelFunc is guaranteed to return after last send +// to the channel +// +// Example: +// ch := make(chan EventT, 10) +// defer close(ch) +// cancel, err := eventbus.Subscribe(ch) +// defer cancel() func (b *basicBus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err error) { var settings subSettings for _, opt := range opts { @@ -118,6 +128,16 @@ func (b *basicBus) Subscribe(typedChan interface{}, opts ...SubOption) (c Cancel return } +// Emitter creates new emitter +// +// eventType accepts typed nil pointers, and uses the type information to +// select output type +// +// Example: +// emit, err := eventbus.Emitter(new(EventT)) +// defer emit.Close() // MUST call this after being done with the emitter +// +// emit(EventT{}) func (b *basicBus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, err error) { var settings emitterSettings for _, opt := range opts { @@ -194,6 +214,18 @@ func (n *node) emit(event interface{}) { } /////////////////////// -// UTILS +// TYPES -var _ Bus = &basicBus{} +var closeEmit struct{} + +// EmitFunc emits events. If any channel subscribed to the topic is blocked, +// calls to EmitFunc will block +// +// Calling this function with wrong event type will cause a panic +type EmitFunc func(event interface{}) + +func (f EmitFunc) Close() { + f(closeEmit) +} + +type CancelFunc func() diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index af05a5cbe0..e088b6d23e 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -337,8 +337,9 @@ func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { for i := 0; i < emits; i++ { go func() { - emit, err := bus.Emitter(new(EventB), func(settings *emitterSettings) { - settings.makeStateful = stateful + emit, err := bus.Emitter(new(EventB), func(settings interface{}) error { + settings.(*emitterSettings).makeStateful = stateful + return nil }) if err != nil { panic(err) diff --git a/p2p/host/eventbus/interface.go b/p2p/host/eventbus/opts.go similarity index 54% rename from p2p/host/eventbus/interface.go rename to p2p/host/eventbus/opts.go index bff5d46f47..cde6b7474f 100644 --- a/p2p/host/eventbus/interface.go +++ b/p2p/host/eventbus/opts.go @@ -5,8 +5,6 @@ import ( "reflect" ) -var closeEmit struct{} - type subSettings struct { forcedType reflect.Type } @@ -53,44 +51,7 @@ type EmitterOption func(interface{}) error // // This allows to provide state tracking for dynamic systems, and/or // allows new subscribers to verify that there are Emitters on the channel -func Stateful(s *emitterSettings) { - s.makeStateful = true +func Stateful(s interface{}) error { + s.(*emitterSettings).makeStateful = true + return nil } - -// Bus is an interface to type-based event delivery system -type Bus interface { - // Subscribe creates new subscription. Failing to drain the channel will cause - // publishers to get blocked. CancelFunc is guaranteed to return after last send - // to the channel - // - // Example: - // ch := make(chan EventT, 10) - // defer close(ch) - // cancel, err := eventbus.Subscribe(ch) - // defer cancel() - Subscribe(typedChan interface{}, opts ...SubOption) (CancelFunc, error) - - // Emitter creates new emitter - // - // eventType accepts typed nil pointers, and uses the type information to - // select output type - // - // Example: - // emit, err := eventbus.Emitter(new(EventT)) - // defer emit.Close() // MUST call this after being done with the emitter - // - // emit(EventT{}) - Emitter(eventType interface{}, opts ...EmitterOption) (EmitFunc, error) -} - -// EmitFunc emits events. If any channel subscribed to the topic is blocked, -// calls to EmitFunc will block -// -// Calling this function with wrong event type will cause a panic -type EmitFunc func(event interface{}) - -func (f EmitFunc) Close() { - f(closeEmit) -} - -type CancelFunc func() From b9fe91677a073ac43e6392d4d7af2f029cc241da Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 19 Jun 2019 16:09:42 +0100 Subject: [PATCH 1320/3965] make Emitter an interface. --- p2p/host/eventbus/basic.go | 73 +++++++++++++++++---------------- p2p/host/eventbus/basic_test.go | 54 ++++++++++++------------ p2p/host/eventbus/opts.go | 9 ++-- 3 files changed, 69 insertions(+), 67 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 9f075aaf55..ae7509eb9d 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -6,18 +6,48 @@ import ( "reflect" "sync" "sync/atomic" + + "github.com/libp2p/go-libp2p-core/event" ) /////////////////////// // BUS +type CancelFunc = func() + // basicBus is a type-based event delivery system type basicBus struct { lk sync.Mutex nodes map[reflect.Type]*node } -func NewBus() *basicBus { +var _ event.Bus = (*basicBus)(nil) + +type Emitter struct { + n *node + typ reflect.Type + closed int32 + dropper func(reflect.Type) +} + +func (e *Emitter) Emit(evt interface{}) { + if atomic.LoadInt32(&e.closed) != 0 { + panic("emitter is closed") + } + e.n.emit(evt) +} + +func (e *Emitter) Close() error { + if !atomic.CompareAndSwapInt32(&e.closed, 0, 1) { + panic("closed an emitter more than once") + } + if atomic.AddInt32(&e.n.nEmitters, -1) == 0 { + e.dropper(e.typ) + } + return nil +} + +func NewBus() event.Bus { return &basicBus{ nodes: map[reflect.Type]*node{}, } @@ -74,7 +104,7 @@ func (b *basicBus) tryDropNode(typ reflect.Type) { // defer close(ch) // cancel, err := eventbus.Subscribe(ch) // defer cancel() -func (b *basicBus) Subscribe(typedChan interface{}, opts ...SubOption) (c CancelFunc, err error) { +func (b *basicBus) Subscribe(typedChan interface{}, opts ...SubscriptionOpt) (c CancelFunc, err error) { var settings subSettings for _, opt := range opts { if err := opt(&settings); err != nil { @@ -138,10 +168,12 @@ func (b *basicBus) Subscribe(typedChan interface{}, opts ...SubOption) (c Cancel // defer emit.Close() // MUST call this after being done with the emitter // // emit(EventT{}) -func (b *basicBus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFunc, err error) { +func (b *basicBus) Emitter(evtType interface{}, opts ...EmitterOpt) (e event.Emitter, err error) { var settings emitterSettings for _, opt := range opts { - opt(&settings) + if err := opt(&settings); err != nil { + return nil, err + } } typ := reflect.TypeOf(evtType) @@ -152,22 +184,8 @@ func (b *basicBus) Emitter(evtType interface{}, opts ...EmitterOption) (e EmitFu err = b.withNode(typ, func(n *node) { atomic.AddInt32(&n.nEmitters, 1) - closed := false n.keepLast = n.keepLast || settings.makeStateful - - e = func(event interface{}) { - if closed { - panic("emitter is closed") - } - if event == closeEmit { - closed = true - if atomic.AddInt32(&n.nEmitters, -1) == 0 { - b.tryDropNode(typ) - } - return - } - n.emit(event) - } + e = &Emitter{n: n, typ: typ, dropper: b.tryDropNode} }, func(_ *node) {}) return } @@ -212,20 +230,3 @@ func (n *node) emit(event interface{}) { } n.lk.RUnlock() } - -/////////////////////// -// TYPES - -var closeEmit struct{} - -// EmitFunc emits events. If any channel subscribed to the topic is blocked, -// calls to EmitFunc will block -// -// Calling this function with wrong event type will cause a panic -type EmitFunc func(event interface{}) - -func (f EmitFunc) Close() { - f(closeEmit) -} - -type CancelFunc func() diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index e088b6d23e..d9d599c4fa 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -38,13 +38,13 @@ func TestEmit(t *testing.T) { <-events }() - emit, err := bus.Emitter(new(EventA)) + em, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - defer emit.Close() + defer em.Close() - emit(EventA{}) + em.Emit(EventA{}) } func TestSub(t *testing.T) { @@ -66,13 +66,13 @@ func TestSub(t *testing.T) { wait.Done() }() - emit, err := bus.Emitter(new(EventB)) + em, err := bus.Emitter(new(EventB)) if err != nil { t.Fatal(err) } - defer emit.Close() + defer em.Close() - emit(EventB(7)) + em.Emit(EventB(7)) wait.Wait() if event != 7 { @@ -83,23 +83,23 @@ func TestSub(t *testing.T) { func TestEmitNoSubNoBlock(t *testing.T) { bus := NewBus() - emit, err := bus.Emitter(new(EventA)) + em, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - defer emit.Close() + defer em.Close() - emit(EventA{}) + em.Emit(EventA{}) } func TestEmitOnClosed(t *testing.T) { bus := NewBus() - emit, err := bus.Emitter(new(EventA)) + em, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - emit.Close() + em.Close() defer func() { r := recover() @@ -111,7 +111,7 @@ func TestEmitOnClosed(t *testing.T) { } }() - emit(EventA{}) + em.Emit(EventA{}) } func TestClosingRaces(t *testing.T) { @@ -187,15 +187,15 @@ func TestSubMany(t *testing.T) { }() } - emit, err := bus.Emitter(new(EventB)) + em, err := bus.Emitter(new(EventB)) if err != nil { t.Fatal(err) } - defer emit.Close() + defer em.Close() ready.Wait() - emit(EventB(7)) + em.Emit(EventB(7)) wait.Wait() if int(r) != 7*n { @@ -222,13 +222,13 @@ func TestSubType(t *testing.T) { wait.Done() }() - emit, err := bus.Emitter(new(EventA)) + em, err := bus.Emitter(new(EventA)) if err != nil { t.Fatal(err) } - defer emit.Close() + defer em.Close() - emit(EventA{}) + em.Emit(EventA{}) wait.Wait() if event.String() != "Oh, Hello" { @@ -238,11 +238,11 @@ func TestSubType(t *testing.T) { func TestNonStateful(t *testing.T) { bus := NewBus() - emit, err := bus.Emitter(new(EventB)) + em, err := bus.Emitter(new(EventB)) if err != nil { t.Fatal(err) } - defer emit.Close() + defer em.Close() eventsA := make(chan EventB, 1) cancelS, err := bus.Subscribe(eventsA) @@ -257,7 +257,7 @@ func TestNonStateful(t *testing.T) { default: } - emit(EventB(1)) + em.Emit(EventB(1)) select { case e := <-eventsA: @@ -284,13 +284,13 @@ func TestNonStateful(t *testing.T) { func TestStateful(t *testing.T) { bus := NewBus() - emit, err := bus.Emitter(new(EventB), Stateful) + em, err := bus.Emitter(new(EventB), Stateful) if err != nil { t.Fatal(err) } - defer emit.Close() + defer em.Close() - emit(EventB(2)) + em.Emit(EventB(2)) eventsA := make(chan EventB, 1) cancelS, err := bus.Subscribe(eventsA) @@ -337,19 +337,19 @@ func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { for i := 0; i < emits; i++ { go func() { - emit, err := bus.Emitter(new(EventB), func(settings interface{}) error { + em, err := bus.Emitter(new(EventB), func(settings interface{}) error { settings.(*emitterSettings).makeStateful = stateful return nil }) if err != nil { panic(err) } - defer emit.Close() + defer em.Close() ready.Wait() for i := 0; i < msgs; i++ { - emit(EventB(97)) + em.Emit(EventB(97)) } wait.Done() diff --git a/p2p/host/eventbus/opts.go b/p2p/host/eventbus/opts.go index cde6b7474f..567391254b 100644 --- a/p2p/host/eventbus/opts.go +++ b/p2p/host/eventbus/opts.go @@ -5,12 +5,14 @@ import ( "reflect" ) +type SubscriptionOpt = func(interface{}) error + +type EmitterOpt = func(interface{}) error + type subSettings struct { forcedType reflect.Type } -type SubOption func(interface{}) error - // ForceSubType is a Subscribe option which overrides the type to which // the subscription will be done. Note that the evtType must be assignable // to channel type. @@ -27,7 +29,7 @@ type SubOption func(interface{}) error // eventCh := make(chan fmt.Stringer) // interface { String() string } // cancel, err := eventbus.Subscribe(eventCh, event.ForceSubType(new(Event))) // [...] -func ForceSubType(evtType interface{}) SubOption { +func ForceSubType(evtType interface{}) SubscriptionOpt { return func(settings interface{}) error { s := settings.(*subSettings) typ := reflect.TypeOf(evtType) @@ -42,7 +44,6 @@ func ForceSubType(evtType interface{}) SubOption { type emitterSettings struct { makeStateful bool } -type EmitterOption func(interface{}) error // Stateful is an Emitter option which makes makes the eventbus channel // 'remember' last event sent, and when a new subscriber joins the From a214a1be4fde7e3398edf8baf43aaf993aa849e1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 19 Jun 2019 16:12:29 +0100 Subject: [PATCH 1321/3965] remove superfluous types. --- p2p/host/eventbus/basic.go | 6 ++---- p2p/host/eventbus/opts.go | 8 +++----- 2 files changed, 5 insertions(+), 9 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index ae7509eb9d..39ce88519a 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -13,8 +13,6 @@ import ( /////////////////////// // BUS -type CancelFunc = func() - // basicBus is a type-based event delivery system type basicBus struct { lk sync.Mutex @@ -104,7 +102,7 @@ func (b *basicBus) tryDropNode(typ reflect.Type) { // defer close(ch) // cancel, err := eventbus.Subscribe(ch) // defer cancel() -func (b *basicBus) Subscribe(typedChan interface{}, opts ...SubscriptionOpt) (c CancelFunc, err error) { +func (b *basicBus) Subscribe(typedChan interface{}, opts ...event.SubscriptionOpt) (c event.CancelFunc, err error) { var settings subSettings for _, opt := range opts { if err := opt(&settings); err != nil { @@ -168,7 +166,7 @@ func (b *basicBus) Subscribe(typedChan interface{}, opts ...SubscriptionOpt) (c // defer emit.Close() // MUST call this after being done with the emitter // // emit(EventT{}) -func (b *basicBus) Emitter(evtType interface{}, opts ...EmitterOpt) (e event.Emitter, err error) { +func (b *basicBus) Emitter(evtType interface{}, opts ...event.EmitterOpt) (e event.Emitter, err error) { var settings emitterSettings for _, opt := range opts { if err := opt(&settings); err != nil { diff --git a/p2p/host/eventbus/opts.go b/p2p/host/eventbus/opts.go index 567391254b..78b70033a7 100644 --- a/p2p/host/eventbus/opts.go +++ b/p2p/host/eventbus/opts.go @@ -3,11 +3,9 @@ package event import ( "errors" "reflect" -) - -type SubscriptionOpt = func(interface{}) error -type EmitterOpt = func(interface{}) error + "github.com/libp2p/go-libp2p-core/event" +) type subSettings struct { forcedType reflect.Type @@ -29,7 +27,7 @@ type subSettings struct { // eventCh := make(chan fmt.Stringer) // interface { String() string } // cancel, err := eventbus.Subscribe(eventCh, event.ForceSubType(new(Event))) // [...] -func ForceSubType(evtType interface{}) SubscriptionOpt { +func ForceSubType(evtType interface{}) event.SubscriptionOpt { return func(settings interface{}) error { s := settings.(*subSettings) typ := reflect.TypeOf(evtType) From c10f6bdd74526ceaca637c75d4fd2b3384ab8fb2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 19 Jun 2019 16:21:55 +0100 Subject: [PATCH 1322/3965] fix compile error in tests. --- p2p/host/eventbus/basic_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index d9d599c4fa..d657dbbaa0 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -156,7 +156,7 @@ func TestClosingRaces(t *testing.T) { wg.Wait() - if len(b.nodes) != 0 { + if len(b.(*basicBus).nodes) != 0 { t.Error("expected no nodes") } } From c6ab7baea5eec1dbf651fa87dd39d59309e46389 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 19 Jun 2019 16:55:25 +0100 Subject: [PATCH 1323/3965] rename eventbus package. (#4) --- p2p/host/eventbus/basic.go | 2 +- p2p/host/eventbus/basic_test.go | 2 +- p2p/host/eventbus/opts.go | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 39ce88519a..8cb030d4f2 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -1,4 +1,4 @@ -package event +package eventbus import ( "errors" diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index d657dbbaa0..4a0edc9c9a 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -1,4 +1,4 @@ -package event +package eventbus import ( "fmt" diff --git a/p2p/host/eventbus/opts.go b/p2p/host/eventbus/opts.go index 78b70033a7..38265557b3 100644 --- a/p2p/host/eventbus/opts.go +++ b/p2p/host/eventbus/opts.go @@ -1,4 +1,4 @@ -package event +package eventbus import ( "errors" From ebfb75774480e3491c20f2d9b6ce9be485abe5a1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 19 Jun 2019 21:58:27 +0100 Subject: [PATCH 1324/3965] peerstore/protobook: support removing protocols. (#20) --- core/peerstore/peerstore.go | 1 + 1 file changed, 1 insertion(+) diff --git a/core/peerstore/peerstore.go b/core/peerstore/peerstore.go index 2e1b644989..7f2c01c9dd 100644 --- a/core/peerstore/peerstore.go +++ b/core/peerstore/peerstore.go @@ -158,5 +158,6 @@ type ProtoBook interface { GetProtocols(peer.ID) ([]string, error) AddProtocols(peer.ID, ...string) error SetProtocols(peer.ID, ...string) error + RemoveProtocols(peer.ID, ...string) error SupportsProtocols(peer.ID, ...string) ([]string, error) } From d297b443d33777e868dd98f3be6524bc394e29d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 19 Jun 2019 23:53:07 +0100 Subject: [PATCH 1325/3965] protobook: support removing protocols. (#84) --- p2p/host/peerstore/pstoreds/protobook.go | 22 ++++++++++++++++++ p2p/host/peerstore/pstoremem/protobook.go | 17 ++++++++++++++ p2p/host/peerstore/test/peerstore_suite.go | 27 +++++++++++++++++++++- 3 files changed, 65 insertions(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go index f4349b6503..fd5f54cb20 100644 --- a/p2p/host/peerstore/pstoreds/protobook.go +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -102,6 +102,28 @@ func (pb *dsProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, return res, nil } +func (pb *dsProtoBook) RemoveProtocols(p peer.ID, protos ...string) error { + s := pb.segments.get(p) + s.RLock() + defer s.RUnlock() + + pmap, err := pb.getProtocolMap(p) + if err != nil { + return err + } + + if len(pmap) == 0 { + // nothing to do. + return nil + } + + for _, proto := range protos { + delete(pmap, proto) + } + + return pb.meta.Put(p, "protocols", pmap) +} + func (pb *dsProtoBook) getProtocolMap(p peer.ID) (map[string]struct{}, error) { iprotomap, err := pb.meta.Get(p, "protocols") switch err { diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index a1ce2311da..7a0c86fe1f 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -112,6 +112,23 @@ func (pb *memoryProtoBook) GetProtocols(p peer.ID) ([]string, error) { return out, nil } +func (pb *memoryProtoBook) RemoveProtocols(p peer.ID, protos ...string) error { + s := pb.segments.get(p) + s.RLock() + defer s.RUnlock() + + protomap, ok := s.protocols[p] + if !ok { + // nothing to remove. + return nil + } + + for _, proto := range protos { + delete(protomap, pb.internProtocol(proto)) + } + return nil +} + func (pb *memoryProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { s := pb.segments.get(p) s.RLock() diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index 020da4caa0..fb4a813769 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "math/rand" + "reflect" "sort" "testing" "time" @@ -240,7 +241,8 @@ func testPeerstoreProtoStore(ps pstore.Peerstore) func(t *testing.T) { t.Fatal("got wrong supported array: ", supported) } - err = ps.SetProtocols(p1, "other") + protos = []string{"other", "yet another", "one more"} + err = ps.SetProtocols(p1, protos...) if err != nil { t.Fatal(err) } @@ -253,6 +255,29 @@ func testPeerstoreProtoStore(ps pstore.Peerstore) func(t *testing.T) { if len(supported) != 0 { t.Fatal("none of those protocols should have been supported") } + + supported, err = ps.GetProtocols(p1) + if err != nil { + t.Fatal(err) + } + sort.Strings(supported) + sort.Strings(protos) + if !reflect.DeepEqual(supported, protos) { + t.Fatalf("expected previously set protos; expected: %v, have: %v", protos, supported) + } + + err = ps.RemoveProtocols(p1, protos[:2]...) + if err != nil { + t.Fatal(err) + } + + supported, err = ps.GetProtocols(p1) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(supported, protos[2:]) { + t.Fatal("expected only one protocol to remain") + } } } From d962316f81cca0892a320d779620b7a7a37d9496 Mon Sep 17 00:00:00 2001 From: Whyrusleeping Date: Thu, 20 Jun 2019 16:59:53 +0200 Subject: [PATCH 1326/3965] eventbus abstraction + initial events (#17) --- core/event/bus.go | 48 ++++++++++++++++++++++++++++++++++++++++++ core/event/doc.go | 11 ++++++++++ core/event/protocol.go | 26 +++++++++++++++++++++++ core/host/host.go | 4 ++++ core/protocol/id.go | 20 ++++++++++++++++++ 5 files changed, 109 insertions(+) create mode 100644 core/event/bus.go create mode 100644 core/event/doc.go create mode 100644 core/event/protocol.go diff --git a/core/event/bus.go b/core/event/bus.go new file mode 100644 index 0000000000..25388397fe --- /dev/null +++ b/core/event/bus.go @@ -0,0 +1,48 @@ +package event + +import "io" + +// SubscriptionOpt represents a subscriber option. Use the options exposed by the implementation of choice. +type SubscriptionOpt = func(interface{}) error + +// EmitterOpt represents an emitter option. Use the options exposed by the implementation of choice. +type EmitterOpt = func(interface{}) error + +// CancelFunc closes a subscriber. +type CancelFunc = func() + +// Emitter represents an actor that emits events onto the eventbus. +type Emitter interface { + io.Closer + + // Emit emits an event onto the eventbus. If any channel subscribed to the topic is blocked, + // calls to Emit will block. + // + // Calling this function with wrong event type will cause a panic. + Emit(evt interface{}) +} + +// Bus is an interface for a type-based event delivery system. +type Bus interface { + // Subscribe creates a new subscription. + // + // Failing to drain the channel may cause publishers to block. CancelFunc must return after + // last send to the channel. + // + // Example: + // ch := make(chan EventT, 10) + // defer close(ch) + // cancel, err := eventbus.Subscribe(ch) + // defer cancel() + Subscribe(typedChan interface{}, opts ...SubscriptionOpt) (CancelFunc, error) + + // Emitter creates a new event emitter. + // + // eventType accepts typed nil pointers, and uses the type information for wiring purposes. + // + // Example: + // em, err := eventbus.Emitter(new(EventT)) + // defer em.Close() // MUST call this after being done with the emitter + // em.Emit(EventT{}) + Emitter(eventType interface{}, opts ...EmitterOpt) (Emitter, error) +} diff --git a/core/event/doc.go b/core/event/doc.go new file mode 100644 index 0000000000..3a070f3afb --- /dev/null +++ b/core/event/doc.go @@ -0,0 +1,11 @@ +// Package event contains the abstractions for a local event bus, along with the standard events +// that libp2p subsystems may emit. +// +// Source code is arranged as follows: +// * doc.go: this file. +// * bus.go: abstractions for the event bus. +// * rest: event structs, sensibly categorised in files by entity, and following this naming convention: +// Evt[Entity (noun)][Event (verb past tense / gerund)] +// The past tense is used to convey that something happened, whereas the gerund form of the verb (-ing) +// expresses that a process is in progress. Examples: EvtConnEstablishing, EvtConnEstablished. +package event diff --git a/core/event/protocol.go b/core/event/protocol.go new file mode 100644 index 0000000000..87b4312da8 --- /dev/null +++ b/core/event/protocol.go @@ -0,0 +1,26 @@ +package event + +import ( + peer "github.com/libp2p/go-libp2p-core/peer" + protocol "github.com/libp2p/go-libp2p-core/protocol" +) + +// EvtPeerProtocolsUpdated should be emitted when a peer we're connected to adds or removes protocols from their stack. +type EvtPeerProtocolsUpdated struct { + // Peer is the peer whose protocols were updated. + Peer peer.ID + // Added enumerates the protocols that were added by this peer. + Added []protocol.ID + // Removed enumerates the protocols that were removed by this peer. + Removed []protocol.ID +} + +// EvtLocalProtocolsUpdated should be emitted when stream handlers are attached or detached from the local host. +// For handlers attached with a matcher predicate (host.SetStreamHandlerMatch()), only the protocol ID will be +// included in this event. +type EvtLocalProtocolsUpdated struct { + // Added enumerates the protocols that were added locally. + Added []protocol.ID + // Removed enumerates the protocols that were removed locally. + Removed []protocol.ID +} diff --git a/core/host/host.go b/core/host/host.go index d4b3c03fdc..9aad5e12cb 100644 --- a/core/host/host.go +++ b/core/host/host.go @@ -7,6 +7,7 @@ import ( "context" "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" @@ -68,4 +69,7 @@ type Host interface { // ConnManager returns this hosts connection manager ConnManager() connmgr.ConnManager + + // EventBus returns the hosts eventbus + EventBus() event.Bus } diff --git a/core/protocol/id.go b/core/protocol/id.go index f7e4a32baf..9df3b5bcf1 100644 --- a/core/protocol/id.go +++ b/core/protocol/id.go @@ -7,3 +7,23 @@ type ID string const ( TestingID ID = "/p2p/_testing" ) + +// ConvertFromStrings is a convenience function that takes a slice of strings and +// converts it to a slice of protocol.ID. +func ConvertFromStrings(ids []string) (res []ID) { + res = make([]ID, 0, len(ids)) + for _, id := range ids { + res = append(res, ID(id)) + } + return res +} + +// ConvertToStrings is a convenience function that takes a slice of protocol.ID and +// converts it to a slice of strings. +func ConvertToStrings(ids []ID) (res []string) { + res = make([]string, 0, len(ids)) + for _, id := range ids { + res = append(res, string(id)) + } + return res +} From 7f5c37c09a9f5a03b534d97adca2cccb28f3716a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 20 Jun 2019 16:17:57 +0100 Subject: [PATCH 1327/3965] =?UTF-8?q?Refinements=20before=20first=20releas?= =?UTF-8?q?e=20=F0=9F=8E=89=20(#7)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- p2p/host/eventbus/basic_test.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 4a0edc9c9a..8d2bfae820 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - "github.com/jbenet/go-detect-race" + "github.com/libp2p/go-libp2p-testing/race" ) type EventA struct{} @@ -15,7 +15,7 @@ type EventB int func getN() int { n := 50000 - if detectrace.WithRace() { + if race.WithRace() { n = 1000 } return n @@ -305,7 +305,7 @@ func TestStateful(t *testing.T) { } func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { - if detectrace.WithRace() && subs+emits > 5000 { + if race.WithRace() && subs+emits > 5000 { t.SkipNow() } From 29bca348a98459c8820f9875e82cfd5422ed530b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 20 Jun 2019 16:27:16 +0100 Subject: [PATCH 1328/3965] integrate the event bus into host implementations (#23) --- p2p/host/blank/blank.go | 43 ++++++++++++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 9 deletions(-) diff --git a/p2p/host/blank/blank.go b/p2p/host/blank/blank.go index 59189782b3..b98b2e5247 100644 --- a/p2p/host/blank/blank.go +++ b/p2p/host/blank/blank.go @@ -5,12 +5,15 @@ import ( "io" "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/protocol" + "github.com/libp2p/go-eventbus" + logging "github.com/ipfs/go-log" ma "github.com/multiformats/go-multiaddr" @@ -21,16 +24,26 @@ var log = logging.Logger("blankhost") // BlankHost is the thinnest implementation of the host.Host interface type BlankHost struct { - n network.Network - mux *mstream.MultistreamMuxer - cmgr connmgr.ConnManager + n network.Network + mux *mstream.MultistreamMuxer + cmgr connmgr.ConnManager + eventbus event.Bus + emitters struct { + evtLocalProtocolsUpdated event.Emitter + } } func NewBlankHost(n network.Network) *BlankHost { bh := &BlankHost{ - n: n, - cmgr: &connmgr.NullConnMgr{}, - mux: mstream.NewMultistreamMuxer(), + n: n, + cmgr: &connmgr.NullConnMgr{}, + mux: mstream.NewMultistreamMuxer(), + eventbus: eventbus.NewBus(), + } + + var err error + if bh.emitters.evtLocalProtocolsUpdated, err = bh.eventbus.Emitter(&event.EvtLocalProtocolsUpdated{}); err != nil { + return nil } n.SetStreamHandler(bh.newStreamHandler) @@ -98,8 +111,11 @@ func (bh *BlankHost) NewStream(ctx context.Context, p peer.ID, protos ...protoco return s, nil } -func (bh *BlankHost) RemoveStreamHandler(p protocol.ID) { - bh.Mux().RemoveHandler(string(p)) +func (bh *BlankHost) RemoveStreamHandler(pid protocol.ID) { + bh.Mux().RemoveHandler(string(pid)) + bh.emitters.evtLocalProtocolsUpdated.Emit(event.EvtLocalProtocolsUpdated{ + Removed: []protocol.ID{pid}, + }) } func (bh *BlankHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) { @@ -109,6 +125,9 @@ func (bh *BlankHost) SetStreamHandler(pid protocol.ID, handler network.StreamHan handler(is) return nil }) + bh.emitters.evtLocalProtocolsUpdated.Emit(event.EvtLocalProtocolsUpdated{ + Added: []protocol.ID{pid}, + }) } func (bh *BlankHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler network.StreamHandler) { @@ -118,11 +137,13 @@ func (bh *BlankHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler(is) return nil }) + bh.emitters.evtLocalProtocolsUpdated.Emit(event.EvtLocalProtocolsUpdated{ + Added: []protocol.ID{pid}, + }) } // newStreamHandler is the remote-opened stream handler for network.Network func (bh *BlankHost) newStreamHandler(s network.Stream) { - protoID, handle, err := bh.Mux().Negotiate(s) if err != nil { log.Warning("protocol mux failed: %s", err) @@ -148,3 +169,7 @@ func (bh *BlankHost) Network() network.Network { func (bh *BlankHost) ConnManager() connmgr.ConnManager { return bh.cmgr } + +func (bh *BlankHost) EventBus() event.Bus { + return bh.eventbus +} From 464d1e2220f10e07bcbac716547e6d241d16c4f4 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 19 Jun 2019 11:24:12 -0400 Subject: [PATCH 1329/3965] Add OpenCensus metrics registration functionality to core --- core/metrics/register.go | 70 ++++++++++++++++++++++++++ core/metrics/register_test.go | 94 +++++++++++++++++++++++++++++++++++ 2 files changed, 164 insertions(+) create mode 100644 core/metrics/register.go create mode 100644 core/metrics/register_test.go diff --git a/core/metrics/register.go b/core/metrics/register.go new file mode 100644 index 0000000000..2fbb74bb29 --- /dev/null +++ b/core/metrics/register.go @@ -0,0 +1,70 @@ +package metrics + +import ( + "fmt" + "sync" + + "go.opencensus.io/stats/view" +) + +var registeredViews = map[string][]*view.View{} +var mu = new(sync.Mutex) + +type ErrNamespace struct { + Namespace string +} + +// ErrUnregisteredNamespace is an error for lookup of requested unregistered Namespace +type ErrUnregisteredNamespace ErrNamespace + +func (e ErrUnregisteredNamespace) Error() string { + return fmt.Sprintf("no views found registered under Namespace %s", e.Namespace) +} + +// ErrDuplicateNamespaceRegistration is an error for a Namespace that has already +// registered views +type ErrDuplicateNamespaceRegistration ErrNamespace + +func (e ErrDuplicateNamespaceRegistration) Error() string { + return fmt.Sprintf("duplicate registration of views by Namespace %s", e.Namespace) +} + +// RegisterViews accepts a namespace and a slice of Views, which will be registered +// with opencensus and maintained in the global registered views map +func RegisterViews(namespace string, views ...*view.View) error { + mu.Lock() + defer mu.Unlock() + _, ok := registeredViews[namespace] + if ok { + return ErrDuplicateNamespaceRegistration{Namespace: namespace} + } else { + registeredViews[namespace] = views + } + + return nil +} + +// LookupViews returns all views for a Namespace name. Returns an error if the +// Namespace has not been registered. +func LookupViews(name string) ([]*view.View, error) { + mu.Lock() + defer mu.Unlock() + views, ok := registeredViews[name] + if !ok { + return nil, ErrUnregisteredNamespace{Namespace: name} + } + response := make([]*view.View, len(views)) + copy(response, views) + return response, nil +} + +// AllViews returns all registered views as a single slice +func AllViews() []*view.View { + var views []*view.View + mu.Lock() + defer mu.Unlock() + for _, vs := range registeredViews { + views = append(views, vs...) + } + return views +} diff --git a/core/metrics/register_test.go b/core/metrics/register_test.go new file mode 100644 index 0000000000..0f4e57ebc1 --- /dev/null +++ b/core/metrics/register_test.go @@ -0,0 +1,94 @@ +package metrics + +import ( + "fmt" + "go.opencensus.io/stats" + "testing" + + "go.opencensus.io/stats/view" +) + +func newTestMeasure(name string) stats.Measure { + return stats.Int64(fmt.Sprintf("test/measure/%s", name), + fmt.Sprintf("Test measurement %s", name), + stats.UnitDimensionless, + ) +} + +func newTestView(name string) *view.View { + return &view.View{ + Name: fmt.Sprintf("test/%s", name), + Description: fmt.Sprintf("Test view %s", name), + Measure: newTestMeasure(name), + Aggregation: view.LastValue(), + } +} + +func TestRegisteringViews(t *testing.T) { + registeredViews = make(map[string][]*view.View) + + t.Run("test registering first views", func(t *testing.T) { + testView := newTestView("empty-map-0") + + if err := RegisterViews("test", testView); err != nil { + t.Fatal("unable to register view in empty map", err) + } + }) + + t.Run("test registering with existing views", func(t *testing.T) { + testView := newTestView("empty-map-1") + testView2 := newTestView("existing-entity-0") + + if err := RegisterViews("test2", testView); err != nil { + t.Fatal("unable to register view in empty map", err) + } + if err := RegisterViews("test3", testView2); err != nil { + t.Fatal("unable to register view in map", err) + } + }) + + t.Run("test registering duplicate views", func(t *testing.T) { + testView := newTestView("empty-map-2") + testView2 := newTestView("existing-entity-1") + + if err := RegisterViews("test4", testView); err != nil { + t.Fatal("unable to register view in empty map", err) + } + if err := RegisterViews("test4", testView2); err == nil { + t.Fatal("allowed duplicate view registration") + } + }) + + t.Run("test looking up views", func(t *testing.T) { + testView := newTestView("empty-map-3") + + if err := RegisterViews("test5", testView); err != nil { + t.Fatal("unable to register view in empty map", err) + } + + views, err := LookupViews("test5") + if err != nil { + t.Fatal("error looking up views", err) + } + + if views[0].Name != testView.Name { + t.Fatal("incorrect view lookup, received name:", views[0].Name) + } + }) +} + +func TestAllViews(t *testing.T) { + registeredViews = make(map[string][]*view.View) + t.Run("test retrieving all views", func(t *testing.T) { + views := []*view.View{newTestView("all-views-0"), newTestView("all-views-1"), newTestView("all-views-2")} + + if err := RegisterViews("test6", views...); err != nil { + t.Fatal("unable to register multiple views at once", err) + } + + allViews := AllViews() + if len(allViews) != len(views) { + t.Fatalf("didn't receive equal number of views: %d %d", len(views), len(allViews)) + } + }) +} From d3e5c0cc8514a3f372b54c8aa972c67a9cd6c742 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Fri, 21 Jun 2019 18:46:11 +0200 Subject: [PATCH 1330/3965] eventbus interface changes (#22) --- core/event/bus.go | 50 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 12 deletions(-) diff --git a/core/event/bus.go b/core/event/bus.go index 25388397fe..028bde7b9c 100644 --- a/core/event/bus.go +++ b/core/event/bus.go @@ -22,27 +22,53 @@ type Emitter interface { Emit(evt interface{}) } +// Subscription represents a subscription to one or multiple event types. +type Subscription interface { + io.Closer + + // Out returns the channel from which to consume events. + Out() <-chan interface{} +} + // Bus is an interface for a type-based event delivery system. type Bus interface { - // Subscribe creates a new subscription. + // Subscribe creates a new Subscription. // - // Failing to drain the channel may cause publishers to block. CancelFunc must return after - // last send to the channel. + // eventType can be either a pointer to a single event type, or a slice of pointers to + // subscribe to multiple event types at once, under a single subscription (and channel). // - // Example: - // ch := make(chan EventT, 10) - // defer close(ch) - // cancel, err := eventbus.Subscribe(ch) - // defer cancel() - Subscribe(typedChan interface{}, opts ...SubscriptionOpt) (CancelFunc, error) + // Failing to drain the channel may cause publishers to block. + // + // Simple example + // + // sub, err := eventbus.Subscribe(new(EventType)) + // defer sub.Close() + // for e := range sub.Out() { + // event := e.(EventType) // guaranteed safe + // [...] + // } + // + // Multi-type example + // + // sub, err := eventbus.Subscribe([]interface{}{new(EventA), new(EventB)}) + // defer sub.Close() + // for e := range sub.Out() { + // select e.(type): + // case EventA: + // [...] + // case EventB: + // [...] + // } + // } + Subscribe(eventType interface{}, opts ...SubscriptionOpt) (Subscription, error) // Emitter creates a new event emitter. // // eventType accepts typed nil pointers, and uses the type information for wiring purposes. // // Example: - // em, err := eventbus.Emitter(new(EventT)) - // defer em.Close() // MUST call this after being done with the emitter - // em.Emit(EventT{}) + // em, err := eventbus.Emitter(new(EventT)) + // defer em.Close() // MUST call this after being done with the emitter + // em.Emit(EventT{}) Emitter(eventType interface{}, opts ...EmitterOpt) (Emitter, error) } From a3f161b18e6089ab52dbbd17479b43acd6c31a93 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Fri, 21 Jun 2019 18:50:36 +0200 Subject: [PATCH 1331/3965] refactor interfaces (#9) --- p2p/host/eventbus/basic.go | 118 ++++++++++++++++++-------------- p2p/host/eventbus/basic_test.go | 68 +++++++++--------- p2p/host/eventbus/opts.go | 36 ++-------- 3 files changed, 103 insertions(+), 119 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 8cb030d4f2..ca102b5637 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -21,21 +21,21 @@ type basicBus struct { var _ event.Bus = (*basicBus)(nil) -type Emitter struct { +type emitter struct { n *node typ reflect.Type closed int32 dropper func(reflect.Type) } -func (e *Emitter) Emit(evt interface{}) { +func (e *emitter) Emit(evt interface{}) { if atomic.LoadInt32(&e.closed) != 0 { panic("emitter is closed") } e.n.emit(evt) } -func (e *Emitter) Close() error { +func (e *emitter) Close() error { if !atomic.CompareAndSwapInt32(&e.closed, 0, 1) { panic("closed an emitter more than once") } @@ -93,16 +93,42 @@ func (b *basicBus) tryDropNode(typ reflect.Type) { b.lk.Unlock() } +type sub struct { + ch chan interface{} + nodes []*node + dropper func(reflect.Type) +} + +func (s *sub) Out() <-chan interface{} { + return s.ch +} + +func (s *sub) Close() error { + close(s.ch) + for _, n := range s.nodes { + n.lk.Lock() + for i := 0; i < len(n.sinks); i++ { + if n.sinks[i] == s.ch { + n.sinks[i], n.sinks[len(n.sinks)-1] = n.sinks[len(n.sinks)-1], nil + n.sinks = n.sinks[:len(n.sinks)-1] + break + } + } + tryDrop := len(n.sinks) == 0 && atomic.LoadInt32(&n.nEmitters) == 0 + n.lk.Unlock() + if tryDrop { + s.dropper(n.typ) + } + } + return nil +} + +var _ event.Subscription = (*sub)(nil) + // Subscribe creates new subscription. Failing to drain the channel will cause // publishers to get blocked. CancelFunc is guaranteed to return after last send // to the channel -// -// Example: -// ch := make(chan EventT, 10) -// defer close(ch) -// cancel, err := eventbus.Subscribe(ch) -// defer cancel() -func (b *basicBus) Subscribe(typedChan interface{}, opts ...event.SubscriptionOpt) (c event.CancelFunc, err error) { +func (b *basicBus) Subscribe(evtTypes interface{}, opts ...event.SubscriptionOpt) (_ event.Subscription, err error) { var settings subSettings for _, opt := range opts { if err := opt(&settings); err != nil { @@ -110,50 +136,40 @@ func (b *basicBus) Subscribe(typedChan interface{}, opts ...event.SubscriptionOp } } - refCh := reflect.ValueOf(typedChan) - typ := refCh.Type() - if typ.Kind() != reflect.Chan { - return nil, errors.New("expected a channel") + types, ok := evtTypes.([]interface{}) + if !ok { + types = []interface{}{evtTypes} } - if typ.ChanDir()&reflect.SendDir == 0 { - return nil, errors.New("channel doesn't allow send") + + out := &sub{ + ch: make(chan interface{}, settings.buffer), + nodes: make([]*node, len(types)), + + dropper: b.tryDropNode, } - if settings.forcedType != nil { - if settings.forcedType.Elem().AssignableTo(typ) { - return nil, fmt.Errorf("forced type %s cannot be sent to chan %s", settings.forcedType, typ) + for i, etyp := range types { + typ := reflect.TypeOf(etyp) + + if typ.Kind() != reflect.Ptr { + return nil, errors.New("subscribe called with non-pointer type") } - typ = settings.forcedType - } - - err = b.withNode(typ.Elem(), func(n *node) { - n.sinks = append(n.sinks, refCh) - c = func() { - n.lk.Lock() - for i := 0; i < len(n.sinks); i++ { - if n.sinks[i] == refCh { - n.sinks[i], n.sinks[len(n.sinks)-1] = n.sinks[len(n.sinks)-1], reflect.Value{} - n.sinks = n.sinks[:len(n.sinks)-1] - break + + err = b.withNode(typ.Elem(), func(n *node) { + n.sinks = append(n.sinks, out.ch) + out.nodes[i] = n + }, func(n *node) { + if n.keepLast { + l := n.last.Load() + if l == nil { + return } + out.ch <- l } - tryDrop := len(n.sinks) == 0 && atomic.LoadInt32(&n.nEmitters) == 0 - n.lk.Unlock() - if tryDrop { - b.tryDropNode(typ.Elem()) - } - } - }, func(n *node) { - if n.keepLast { - lastVal, ok := n.last.Load().(reflect.Value) - if !ok { - return - } + }) + } - refCh.Send(lastVal) - } - }) - return + return out, nil } // Emitter creates new emitter @@ -183,7 +199,7 @@ func (b *basicBus) Emitter(evtType interface{}, opts ...event.EmitterOpt) (e eve err = b.withNode(typ, func(n *node) { atomic.AddInt32(&n.nEmitters, 1) n.keepLast = n.keepLast || settings.makeStateful - e = &Emitter{n: n, typ: typ, dropper: b.tryDropNode} + e = &emitter{n: n, typ: typ, dropper: b.tryDropNode} }, func(_ *node) {}) return } @@ -203,7 +219,7 @@ type node struct { keepLast bool last atomic.Value - sinks []reflect.Value + sinks []chan interface{} } func newNode(typ reflect.Type) *node { @@ -220,11 +236,11 @@ func (n *node) emit(event interface{}) { n.lk.RLock() if n.keepLast { - n.last.Store(eval) + n.last.Store(event) } for _, ch := range n.sinks { - ch.Send(eval) + ch <- event } n.lk.RUnlock() } diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 8d2bfae820..fc23e61290 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -27,15 +27,14 @@ func (EventA) String() string { func TestEmit(t *testing.T) { bus := NewBus() - events := make(chan EventA) - cancel, err := bus.Subscribe(events) + sub, err := bus.Subscribe(new(EventA)) if err != nil { t.Fatal(err) } go func() { - defer cancel() - <-events + defer sub.Close() + <-sub.Out() }() em, err := bus.Emitter(new(EventA)) @@ -49,8 +48,7 @@ func TestEmit(t *testing.T) { func TestSub(t *testing.T) { bus := NewBus() - events := make(chan EventB) - cancel, err := bus.Subscribe(events) + sub, err := bus.Subscribe(new(EventB)) if err != nil { t.Fatal(err) } @@ -61,8 +59,8 @@ func TestSub(t *testing.T) { wait.Add(1) go func() { - defer cancel() - event = <-events + defer sub.Close() + event = (<-sub.Out()).(EventB) wait.Done() }() @@ -131,9 +129,9 @@ func TestClosingRaces(t *testing.T) { lk.RLock() defer lk.RUnlock() - cancel, _ := b.Subscribe(make(chan EventA)) + sub, _ := b.Subscribe(new(EventA)) time.Sleep(10 * time.Millisecond) - cancel() + sub.Close() wg.Done() }() @@ -174,15 +172,14 @@ func TestSubMany(t *testing.T) { for i := 0; i < n; i++ { go func() { - events := make(chan EventB) - cancel, err := bus.Subscribe(events) + sub, err := bus.Subscribe(new(EventB)) if err != nil { panic(err) } - defer cancel() + defer sub.Close() ready.Done() - atomic.AddInt32(&r, int32(<-events)) + atomic.AddInt32(&r, int32((<-sub.Out()).(EventB))) wait.Done() }() } @@ -205,8 +202,7 @@ func TestSubMany(t *testing.T) { func TestSubType(t *testing.T) { bus := NewBus() - events := make(chan fmt.Stringer) - cancel, err := bus.Subscribe(events, ForceSubType(new(EventA))) + sub, err := bus.Subscribe([]interface{}{new(EventA), new(EventB)}) if err != nil { t.Fatal(err) } @@ -217,8 +213,8 @@ func TestSubType(t *testing.T) { wait.Add(1) go func() { - defer cancel() - event = <-events + defer sub.Close() + event = (<-sub.Out()).(EventA) wait.Done() }() @@ -244,15 +240,14 @@ func TestNonStateful(t *testing.T) { } defer em.Close() - eventsA := make(chan EventB, 1) - cancelS, err := bus.Subscribe(eventsA) + sub1, err := bus.Subscribe(new(EventB), BufSize(1)) if err != nil { t.Fatal(err) } - defer cancelS() + defer sub1.Close() select { - case <-eventsA: + case <-sub1.Out(): t.Fatal("didn't expect to get an event") default: } @@ -260,23 +255,22 @@ func TestNonStateful(t *testing.T) { em.Emit(EventB(1)) select { - case e := <-eventsA: - if e != 1 { + case e := <-sub1.Out(): + if e.(EventB) != 1 { t.Fatal("got wrong event") } default: t.Fatal("expected to get an event") } - eventsB := make(chan EventB, 1) - cancelS2, err := bus.Subscribe(eventsB) + sub2, err := bus.Subscribe(new(EventB), BufSize(1)) if err != nil { t.Fatal(err) } - defer cancelS2() + defer sub2.Close() select { - case <-eventsA: + case <-sub2.Out(): t.Fatal("didn't expect to get an event") default: } @@ -292,14 +286,13 @@ func TestStateful(t *testing.T) { em.Emit(EventB(2)) - eventsA := make(chan EventB, 1) - cancelS, err := bus.Subscribe(eventsA) + sub, err := bus.Subscribe(new(EventB), BufSize(1)) if err != nil { t.Fatal(err) } - defer cancelS() + defer sub.Close() - if <-eventsA != 2 { + if (<-sub.Out()).(EventB) != 2 { t.Fatal("got wrong event") } } @@ -320,16 +313,19 @@ func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { for i := 0; i < subs; i++ { go func() { - events := make(chan EventB) - cancel, err := bus.Subscribe(events) + sub, err := bus.Subscribe(new(EventB)) if err != nil { panic(err) } - defer cancel() + defer sub.Close() ready.Done() for i := 0; i < emits*msgs; i++ { - atomic.AddInt64(&r, int64(<-events)) + e, ok := <-sub.Out() + if !ok { + panic("wat") + } + atomic.AddInt64(&r, int64(e.(EventB))) } wait.Done() }() diff --git a/p2p/host/eventbus/opts.go b/p2p/host/eventbus/opts.go index 38265557b3..50c673685b 100644 --- a/p2p/host/eventbus/opts.go +++ b/p2p/host/eventbus/opts.go @@ -1,40 +1,12 @@ package eventbus -import ( - "errors" - "reflect" - - "github.com/libp2p/go-libp2p-core/event" -) - type subSettings struct { - forcedType reflect.Type + buffer int } -// ForceSubType is a Subscribe option which overrides the type to which -// the subscription will be done. Note that the evtType must be assignable -// to channel type. -// -// This also allows for subscribing to multiple eventbus channels with one -// Go channel to get better ordering guarantees. -// -// Example: -// type Event struct{} -// func (Event) String() string { -// return "event" -// } -// -// eventCh := make(chan fmt.Stringer) // interface { String() string } -// cancel, err := eventbus.Subscribe(eventCh, event.ForceSubType(new(Event))) -// [...] -func ForceSubType(evtType interface{}) event.SubscriptionOpt { - return func(settings interface{}) error { - s := settings.(*subSettings) - typ := reflect.TypeOf(evtType) - if typ.Kind() != reflect.Ptr { - return errors.New("ForceSubType called with non-pointer type") - } - s.forcedType = typ +func BufSize(n int) func(interface{}) error { + return func(s interface{}) error { + s.(*subSettings).buffer = n return nil } } From de37ea9063136aead391e3e7659402032f3abb4a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Sat, 22 Jun 2019 12:05:03 +0200 Subject: [PATCH 1332/3965] fix close deadlock and Sub type error --- p2p/host/eventbus/basic.go | 28 +++++++++++++++++++++++----- p2p/host/eventbus/basic_test.go | 21 +++++++++++++++++++++ 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index ca102b5637..2fe37983a3 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -104,9 +104,21 @@ func (s *sub) Out() <-chan interface{} { } func (s *sub) Close() error { - close(s.ch) + stop := make(chan struct{}) + go func() { + for { + select { + case <-s.ch: + case <-stop: + close(s.ch) + return + } + } + }() + for _, n := range s.nodes { n.lk.Lock() + for i := 0; i < len(n.sinks); i++ { if n.sinks[i] == s.ch { n.sinks[i], n.sinks[len(n.sinks)-1] = n.sinks[len(n.sinks)-1], nil @@ -114,12 +126,16 @@ func (s *sub) Close() error { break } } + tryDrop := len(n.sinks) == 0 && atomic.LoadInt32(&n.nEmitters) == 0 + n.lk.Unlock() + if tryDrop { s.dropper(n.typ) } } + close(stop) return nil } @@ -148,12 +164,14 @@ func (b *basicBus) Subscribe(evtTypes interface{}, opts ...event.SubscriptionOpt dropper: b.tryDropNode, } - for i, etyp := range types { - typ := reflect.TypeOf(etyp) - - if typ.Kind() != reflect.Ptr { + for _, etyp := range types { + if reflect.TypeOf(etyp).Kind() != reflect.Ptr { return nil, errors.New("subscribe called with non-pointer type") } + } + + for i, etyp := range types { + typ := reflect.TypeOf(etyp) err = b.withNode(typ.Elem(), func(n *node) { n.sinks = append(n.sinks, out.ch) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index fc23e61290..62b84eb1cf 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -297,6 +297,27 @@ func TestStateful(t *testing.T) { } } +func TestCloseBlocking(t *testing.T) { + bus := NewBus() + em, err := bus.Emitter(new(EventB)) + if err != nil { + t.Fatal(err) + } + + sub, err := bus.Subscribe(new(EventB)) + if err != nil { + t.Fatal(err) + } + + go func() { + em.Emit(EventB(159)) + }() + + time.Sleep(10 * time.Millisecond) // make sure that emit is blocked + + sub.Close() // cancel sub +} + func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { if race.WithRace() && subs+emits > 5000 { t.SkipNow() From 16976a34d4c919f478966a3068f0f95f02da3897 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Sat, 22 Jun 2019 15:14:04 +0200 Subject: [PATCH 1333/3965] Add test for #10 --- p2p/host/eventbus/basic_test.go | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index 62b84eb1cf..aaad88afb6 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -318,6 +318,21 @@ func TestCloseBlocking(t *testing.T) { sub.Close() // cancel sub } +func TestSubFailFully(t *testing.T) { + bus := NewBus() + em, err := bus.Emitter(new(EventB)) + if err != nil { + t.Fatal(err) + } + + _, err = bus.Subscribe([]interface{}{new(EventB), 5}) + if err == nil || err.Error() != "subscribe called with non-pointer type" { + t.Fatal(err) + } + + em.Emit(EventB(159)) // will hang if sub doesn't fail properly +} + func testMany(t testing.TB, subs, emits, msgs int, stateful bool) { if race.WithRace() && subs+emits > 5000 { t.SkipNow() From 6ef6b01ed3ee396a6c69419a6afcfb8e2c28fad5 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Sat, 22 Jun 2019 15:28:00 +0200 Subject: [PATCH 1334/3965] Improve benchmarks License: MIT Signed-off-by: Jakub Sztandera --- p2p/host/eventbus/basic_test.go | 126 ++++++++++++++++++-------------- 1 file changed, 73 insertions(+), 53 deletions(-) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index fc23e61290..c186724692 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -363,69 +363,89 @@ func TestBothMany(t *testing.T) { testMany(t, 10000, 100, 10, false) } -func BenchmarkSubs(b *testing.B) { - b.ReportAllocs() - testMany(b, b.N, 100, 100, false) +type benchCase struct { + subs int + emits int + stateful bool } -func BenchmarkEmits(b *testing.B) { - b.ReportAllocs() - testMany(b, 100, b.N, 100, false) +func (bc benchCase) name() string { + return fmt.Sprintf("subs-%d/emits-%d/stateful-%t", bc.subs, bc.emits, bc.stateful) } -func BenchmarkMsgs(b *testing.B) { - b.ReportAllocs() - testMany(b, 100, 100, b.N, false) -} - -func BenchmarkOneToMany(b *testing.B) { - b.ReportAllocs() - testMany(b, b.N, 1, 100, false) -} - -func BenchmarkManyToOne(b *testing.B) { - b.ReportAllocs() - testMany(b, 1, b.N, 100, false) -} - -func BenchmarkMs1e2m4(b *testing.B) { - b.N = 1000000 - b.ReportAllocs() - testMany(b, 10, 100, 10000, false) +func genTestCases() []benchCase { + ret := make([]benchCase, 0, 200) + for stateful := 0; stateful < 2; stateful++ { + for subs := uint(0); subs <= 8; subs = subs + 2 { + for emits := uint(0); emits <= 8; emits = emits + 2 { + ret = append(ret, benchCase{1 << subs, 1 << emits, stateful == 1}) + } + } + } + return ret } -func BenchmarkMs1e0m6(b *testing.B) { - b.N = 10000000 - b.ReportAllocs() - testMany(b, 10, 1, 1000000, false) +func BenchmarkEvents(b *testing.B) { + for _, bc := range genTestCases() { + b.Run(bc.name(), benchMany(bc)) + } } -func BenchmarkMs0e0m6(b *testing.B) { - b.N = 1000000 - b.ReportAllocs() - testMany(b, 1, 1, 1000000, false) -} +func benchMany(bc benchCase) func(*testing.B) { + return func(b *testing.B) { + b.ReportAllocs() + subs := bc.subs + emits := bc.emits + stateful := bc.stateful + bus := NewBus() + var wait sync.WaitGroup + var ready sync.WaitGroup + wait.Add(subs + emits) + ready.Add(subs + emits) + + for i := 0; i < subs; i++ { + go func() { + sub, err := bus.Subscribe(new(EventB)) + if err != nil { + panic(err) + } + defer sub.Close() + + ready.Done() + ready.Wait() + for i := 0; i < (b.N/emits)*emits; i++ { + _, ok := <-sub.Out() + if !ok { + panic("wat") + } + } + wait.Done() + }() + } -func BenchmarkStatefulMs1e0m6(b *testing.B) { - b.N = 10000000 - b.ReportAllocs() - testMany(b, 10, 1, 1000000, true) -} + for i := 0; i < emits; i++ { + go func() { + em, err := bus.Emitter(new(EventB), func(settings interface{}) error { + settings.(*emitterSettings).makeStateful = stateful + return nil + }) + if err != nil { + panic(err) + } + defer em.Close() -func BenchmarkStatefulMs0e0m6(b *testing.B) { - b.N = 1000000 - b.ReportAllocs() - testMany(b, 1, 1, 1000000, true) -} + ready.Done() + ready.Wait() -func BenchmarkMs0e6m0(b *testing.B) { - b.N = 1000000 - b.ReportAllocs() - testMany(b, 1, 1000000, 1, false) -} + for i := 0; i < b.N/emits; i++ { + em.Emit(EventB(97)) + } -func BenchmarkMs6e0m0(b *testing.B) { - b.N = 1000000 - b.ReportAllocs() - testMany(b, 1000000, 1, 1, false) + wait.Done() + }() + } + ready.Wait() + b.ResetTimer() + wait.Wait() + } } From a5a2fbb4d0ac7f293d7491bebc6129523af31d5c Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Sat, 22 Jun 2019 15:45:09 +0200 Subject: [PATCH 1335/3965] Make the test fail in reasonable time License: MIT Signed-off-by: Jakub Sztandera --- p2p/host/eventbus/basic_test.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index aaad88afb6..49cc4d77b0 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -318,6 +318,11 @@ func TestCloseBlocking(t *testing.T) { sub.Close() // cancel sub } +func panicOnTimeout(d time.Duration) { + <-time.After(d) + panic("timeout reached") +} + func TestSubFailFully(t *testing.T) { bus := NewBus() em, err := bus.Emitter(new(EventB)) @@ -330,6 +335,8 @@ func TestSubFailFully(t *testing.T) { t.Fatal(err) } + go panicOnTimeout(5 * time.Second) + em.Emit(EventB(159)) // will hang if sub doesn't fail properly } From 12258ae84244d7a015769b937d3da28e2561d72f Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Sat, 22 Jun 2019 15:48:06 +0200 Subject: [PATCH 1336/3965] Reduce number of samples License: MIT Signed-off-by: Jakub Sztandera --- p2p/host/eventbus/basic_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index c186724692..ccd6bfd61e 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -376,8 +376,8 @@ func (bc benchCase) name() string { func genTestCases() []benchCase { ret := make([]benchCase, 0, 200) for stateful := 0; stateful < 2; stateful++ { - for subs := uint(0); subs <= 8; subs = subs + 2 { - for emits := uint(0); emits <= 8; emits = emits + 2 { + for subs := uint(0); subs <= 8; subs = subs + 4 { + for emits := uint(0); emits <= 8; emits = emits + 4 { ret = append(ret, benchCase{1 << subs, 1 << emits, stateful == 1}) } } From 6759096209555cbf86f3bf80e4a0945a7814ce31 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Sat, 22 Jun 2019 15:50:30 +0200 Subject: [PATCH 1337/3965] Clean up format License: MIT Signed-off-by: Jakub Sztandera --- p2p/host/eventbus/basic_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index ccd6bfd61e..b6b74cff00 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -370,7 +370,7 @@ type benchCase struct { } func (bc benchCase) name() string { - return fmt.Sprintf("subs-%d/emits-%d/stateful-%t", bc.subs, bc.emits, bc.stateful) + return fmt.Sprintf("subs-%03d/emits-%03d/stateful-%t", bc.subs, bc.emits, bc.stateful) } func genTestCases() []benchCase { From 6500e55492f785879cde767831f88663cec297a0 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Sat, 22 Jun 2019 17:25:05 +0200 Subject: [PATCH 1338/3965] Add benchmarks for subscribe and emitter License: MIT Signed-off-by: Jakub Sztandera --- p2p/host/eventbus/basic_test.go | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index b6b74cff00..deddc6b558 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -449,3 +449,36 @@ func benchMany(bc benchCase) func(*testing.B) { wait.Wait() } } + +var div = 100 + +func BenchmarkSubscribe(b *testing.B) { + b.ReportAllocs() + for i := 0; i < b.N/div; i++ { + bus := NewBus() + for j := 0; j < div; j++ { + bus.Subscribe(new(EventA)) + } + } +} + +func BenchmarkEmitter(b *testing.B) { + b.ReportAllocs() + for i := 0; i < b.N/div; i++ { + bus := NewBus() + for j := 0; j < div; j++ { + bus.Emitter(new(EventA)) + } + } +} + +func BenchmarkSubscribeAndEmitter(b *testing.B) { + b.ReportAllocs() + for i := 0; i < b.N/div; i++ { + bus := NewBus() + for j := 0; j < div; j++ { + bus.Subscribe(new(EventA)) + bus.Emitter(new(EventA)) + } + } +} From e69d17141ca58ba6afbf13098e90c9377938e590 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 24 Jun 2019 14:44:47 +0100 Subject: [PATCH 1339/3965] integrate the event bus, handle protocol update events, make identify emit deltas (#659) --- go.mod | 7 +- go.sum | 46 ++++ p2p/host/basic/basic_host.go | 38 ++- p2p/host/basic/basic_host_test.go | 40 +++ p2p/host/routed/routed.go | 5 + p2p/protocol/identify/id.go | 113 +++++--- p2p/protocol/identify/id_delta.go | 83 ++++++ p2p/protocol/identify/id_push.go | 20 ++ p2p/protocol/identify/id_test.go | 187 ++++++++++++- p2p/protocol/identify/pb/identify.pb.go | 344 ++++++++++++++++++++++-- p2p/protocol/identify/pb/identify.proto | 10 + 11 files changed, 837 insertions(+), 56 deletions(-) create mode 100644 p2p/protocol/identify/id_delta.go create mode 100644 p2p/protocol/identify/id_push.go diff --git a/go.mod b/go.mod index d951ad3a1b..edc7e01179 100644 --- a/go.mod +++ b/go.mod @@ -9,16 +9,17 @@ require ( github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 + github.com/libp2p/go-eventbus v0.0.2 github.com/libp2p/go-libp2p-autonat v0.1.0 - github.com/libp2p/go-libp2p-blankhost v0.1.1 + github.com/libp2p/go-libp2p-blankhost v0.1.3 github.com/libp2p/go-libp2p-circuit v0.1.0 - github.com/libp2p/go-libp2p-core v0.0.1 + github.com/libp2p/go-libp2p-core v0.0.6 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 github.com/libp2p/go-libp2p-nat v0.0.4 github.com/libp2p/go-libp2p-netutil v0.1.0 - github.com/libp2p/go-libp2p-peerstore v0.1.0 + github.com/libp2p/go-libp2p-peerstore v0.1.1 github.com/libp2p/go-libp2p-secio v0.1.0 github.com/libp2p/go-libp2p-swarm v0.1.0 github.com/libp2p/go-libp2p-testing v0.0.4 diff --git a/go.sum b/go.sum index 47c9026caa..d55c44d57e 100644 --- a/go.sum +++ b/go.sum @@ -1,15 +1,21 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= github.com/AndreasBriese/bbloom v0.0.0-20180913140656-343706a395b7/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/Kubuxu/go-os-helper v0.0.1/go.mod h1:N8B+I7vPCT80IcP58r50u4+gEEcsZETFUpAzWW2ep1Y= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32 h1:qkOC5Gd33k54tobS36cXdAzJbeHaduLtnLQQwNoIi78= github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32/go.mod h1:DrZx5ec/dmnfpw9KyYoQyYo7d0KEvTkk/5M/vbZjAr8= +github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c h1:aEbSeNALREWXk0G7UdNhR3ayBV7tZ4M2PNmnrCAph6Q= +github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI= github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA= github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= +github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd/go.mod h1:HHNXQzUsZCxOoE+CPiyCTO6x34Zs86zZUiwtpXoGdtg= github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVaaLLH7j4eDXPRvw78tMflu7Ie2bzYOH4Y8rRKBY= github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc= github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY= github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/coreos/go-semver v0.3.0 h1:wkHLiw0WNATZnSG7epLsujiMCgPAc9xhjJ4tgnAxmfM= github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -23,10 +29,14 @@ github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMo github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98= github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE= github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.0 h1:kbxbvI4Un1LUWKxufD+BiE6AEExYYgkQLQmLFqA1LFk= github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0= github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q= @@ -35,6 +45,8 @@ github.com/gxed/hashland/keccakpg v0.0.1 h1:wrk3uMNaMxbXiHibbPO4S0ymqJMm41WiudyF github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= github.com/gxed/hashland/murmur3 v0.0.1 h1:SheiaIt0sda5K+8FLz952/1iWS9zrnKsEJaOJu4ZbSc= github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= +github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= @@ -85,16 +97,24 @@ github.com/libp2p/go-buffer-pool v0.0.2 h1:QNK2iAFa8gjAe1SPz6mHSMuCcjs+X1wlHzeOS github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM= github.com/libp2p/go-conn-security-multistream v0.1.0 h1:aqGmto+ttL/uJgX0JtQI0tD21CIEy5eYd1Hlp0juHY0= github.com/libp2p/go-conn-security-multistream v0.1.0/go.mod h1:aw6eD7LOsHEX7+2hJkDxw1MteijaVcI+/eP2/x3J1xc= +github.com/libp2p/go-eventbus v0.0.2 h1:L9eslON8FjFBJlyUs9fyEZKnxSqZd2AMDUNldPrqmZI= +github.com/libp2p/go-eventbus v0.0.2/go.mod h1:Hr/yGlwxA/stuLnpMiu82lpNKpvRy3EaJxPu40XYOwk= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-libp2p-autonat v0.1.0 h1:aCWAu43Ri4nU0ZPO7NyLzUvvfqd0nE3dX0R/ZGYVgOU= github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3PtvYlJmg7NeVtPRqH8= github.com/libp2p/go-libp2p-blankhost v0.1.1 h1:X919sCh+KLqJcNRApj43xCSiQRYqOSI88Fdf55ngf78= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= +github.com/libp2p/go-libp2p-blankhost v0.1.3 h1:0KycuXvPDhmehw0ASsg+s1o3IfXgCUDqfzAl94KEBOg= +github.com/libp2p/go-libp2p-blankhost v0.1.3/go.mod h1:KML1//wiKR8vuuJO0y3LUd1uLv+tlkGTAr3jC0S5cLg= github.com/libp2p/go-libp2p-circuit v0.1.0 h1:eniLL3Y9aq/sryfyV1IAHj5rlvuyj3b7iz8tSiZpdhY= github.com/libp2p/go-libp2p-circuit v0.1.0/go.mod h1:Ahq4cY3V9VJcHcn1SBXjr78AbFkZeIRmfunbA7pmFh8= github.com/libp2p/go-libp2p-core v0.0.1 h1:HSTZtFIq/W5Ue43Zw+uWZyy2Vl5WtF0zDjKN8/DT/1I= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= +github.com/libp2p/go-libp2p-core v0.0.4 h1:wNg7b2tKrZlSNibP+j7A1v8eatjf4+9+YRw0L3DUeFE= +github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= +github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyLCcvo1w= +github.com/libp2p/go-libp2p-core v0.0.6/go.mod h1:0d9xmaYAVY5qmbp/fcgxHT3ZJsLjYeYPMJAUKpaCHrE= github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= @@ -112,6 +132,8 @@ github.com/libp2p/go-libp2p-peer v0.2.0 h1:EQ8kMjaCUwt/Y5uLgjT8iY2qg0mGUT0N1zUje github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMgxjsHm7+J5kjWY= github.com/libp2p/go-libp2p-peerstore v0.1.0 h1:MKh7pRNPHSh1fLPj8u/M/s/napdmeNpoi9BRy9lPN0E= github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HBb6u0MOvdJ7YIsmcwtY= +github.com/libp2p/go-libp2p-peerstore v0.1.1 h1:AJZF2sPpVo+0aAr3IrRiGVsPjJm1INlUQ9EGClgXJ4M= +github.com/libp2p/go-libp2p-peerstore v0.1.1/go.mod h1:ojEWnwG7JpJLkJ9REWYXQslyu9ZLrPWPEcCdiZzEbSM= github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLMWdB7GTQ= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-swarm v0.1.0 h1:HrFk2p0awrGEgch9JXK/qp/hfjqQfgNxpLWnCiWPg5s= @@ -225,6 +247,8 @@ github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxf github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= +go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= +go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67 h1:ng3VDlRp5/DHpSWl02R4rM9I+8M2rhmsuLwAMmkLQWE= golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= @@ -233,14 +257,28 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f h1:R423Cnkcp5JABoeemiGEPlt9tHXFfw5kvc0yqlxRPWo= golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 h1:IcSOAf4PyMp3U3XbIEj1/xJ2BjNN2jWv7JoyOsMxXUU= +golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7 h1:C2F/nMkR/9sfUTpvR3QrjBuTdvMUC/cFajkphs1YLQo= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f h1:wMNYb4v58l5UBM7MYRLPG6ZhfOqbKu7X5eyFl8ZhKvA= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6 h1:bjcUS9ztw9kFmmIxJInhon/0Is3p+EHBKNgquIzo1OI= +golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -252,7 +290,14 @@ golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -262,3 +307,4 @@ gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkep gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index f0eb4ea22d..1f877f4a79 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -7,21 +7,24 @@ import ( "sync" "time" + "github.com/libp2p/go-libp2p/p2p/protocol/identify" + "github.com/libp2p/go-libp2p/p2p/protocol/ping" + "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/protocol" + "github.com/libp2p/go-eventbus" + inat "github.com/libp2p/go-libp2p-nat" + logging "github.com/ipfs/go-log" - goprocess "github.com/jbenet/goprocess" + "github.com/jbenet/goprocess" goprocessctx "github.com/jbenet/goprocess/context" - inat "github.com/libp2p/go-libp2p-nat" - - identify "github.com/libp2p/go-libp2p/p2p/protocol/identify" - ping "github.com/libp2p/go-libp2p/p2p/protocol/ping" ma "github.com/multiformats/go-multiaddr" madns "github.com/multiformats/go-multiaddr-dns" manet "github.com/multiformats/go-multiaddr-net" @@ -69,6 +72,7 @@ type BasicHost struct { natmgr NATManager maResolver *madns.Resolver cmgr connmgr.ConnManager + eventbus event.Bus AddrsFactory AddrsFactory @@ -78,6 +82,9 @@ type BasicHost struct { mx sync.Mutex lastAddrs []ma.Multiaddr + emitters struct { + evtLocalProtocolsUpdated event.Emitter + } } var _ host.Host = (*BasicHost)(nil) @@ -85,7 +92,6 @@ var _ host.Host = (*BasicHost)(nil) // HostOpts holds options that can be passed to NewHost in order to // customize construction of the *BasicHost. type HostOpts struct { - // MultistreamMuxer is essential for the *BasicHost and will use a sensible default value if omitted. MultistreamMuxer *msmux.MultistreamMuxer @@ -125,6 +131,12 @@ func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHo negtimeout: DefaultNegotiationTimeout, AddrsFactory: DefaultAddrsFactory, maResolver: madns.DefaultResolver, + eventbus: eventbus.NewBus(), + } + + var err error + if h.emitters.evtLocalProtocolsUpdated, err = h.eventbus.Emitter(&event.EvtLocalProtocolsUpdated{}); err != nil { + return nil, err } h.proc = goprocessctx.WithContextAndTeardown(ctx, func() error { @@ -366,6 +378,10 @@ func (h *BasicHost) IDService() *identify.IDService { return h.ids } +func (h *BasicHost) EventBus() event.Bus { + return h.eventbus +} + // SetStreamHandler sets the protocol handler on the Host's Mux. // This is equivalent to: // host.Mux().SetHandler(proto, handler) @@ -377,6 +393,9 @@ func (h *BasicHost) SetStreamHandler(pid protocol.ID, handler network.StreamHand handler(is) return nil }) + h.emitters.evtLocalProtocolsUpdated.Emit(event.EvtLocalProtocolsUpdated{ + Added: []protocol.ID{pid}, + }) } // SetStreamHandlerMatch sets the protocol handler on the Host's Mux @@ -388,11 +407,17 @@ func (h *BasicHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler(is) return nil }) + h.emitters.evtLocalProtocolsUpdated.Emit(event.EvtLocalProtocolsUpdated{ + Added: []protocol.ID{pid}, + }) } // RemoveStreamHandler returns .. func (h *BasicHost) RemoveStreamHandler(pid protocol.ID) { h.Mux().RemoveHandler(string(pid)) + h.emitters.evtLocalProtocolsUpdated.Emit(event.EvtLocalProtocolsUpdated{ + Removed: []protocol.ID{pid}, + }) } // NewStream opens a new stream to given peer p, and writes a p2p/protocol @@ -721,6 +746,7 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { // Close shuts down the Host's services (network, etc). func (h *BasicHost) Close() error { + _ = h.emitters.evtLocalProtocolsUpdated.Close() return h.proc.Close() } diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index 270e353e53..b80c702c99 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -4,10 +4,13 @@ import ( "bytes" "context" "io" + "reflect" "sort" "testing" "time" + "github.com/libp2p/go-eventbus" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/helpers" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" @@ -69,6 +72,43 @@ func TestHostSimple(t *testing.T) { } } +func TestProtocolHandlerEvents(t *testing.T) { + ctx := context.Background() + h := New(swarmt.GenSwarm(t, ctx)) + defer h.Close() + + sub, err := h.EventBus().Subscribe(&event.EvtLocalProtocolsUpdated{}, eventbus.BufSize(16)) + if err != nil { + t.Fatal(err) + } + defer sub.Close() + + assert := func(added, removed []protocol.ID) { + var next event.EvtLocalProtocolsUpdated + select { + case evt := <-sub.Out(): + next = evt.(event.EvtLocalProtocolsUpdated) + break + case <-time.After(5 * time.Second): + t.Fatal("event not received in 5 seconds") + } + + if !reflect.DeepEqual(added, next.Added) { + t.Errorf("expected added: %v; received: %v", added, next.Added) + } + if !reflect.DeepEqual(removed, next.Removed) { + t.Errorf("expected removed: %v; received: %v", removed, next.Removed) + } + } + + h.SetStreamHandler(protocol.TestingID, func(s network.Stream) {}) + assert([]protocol.ID{protocol.TestingID}, nil) + h.SetStreamHandler(protocol.ID("foo"), func(s network.Stream) {}) + assert([]protocol.ID{protocol.ID("foo")}, nil) + h.RemoveStreamHandler(protocol.TestingID) + assert(nil, []protocol.ID{protocol.TestingID}) +} + func TestHostAddrsFactory(t *testing.T) { maddr := ma.StringCast("/ip4/1.2.3.4/tcp/1234") addrsFactory := func(addrs []ma.Multiaddr) []ma.Multiaddr { diff --git a/p2p/host/routed/routed.go b/p2p/host/routed/routed.go index 1282020bcb..aecab09568 100644 --- a/p2p/host/routed/routed.go +++ b/p2p/host/routed/routed.go @@ -6,6 +6,7 @@ import ( "time" "github.com/libp2p/go-libp2p-core/connmgr" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" @@ -153,6 +154,10 @@ func (rh *RoutedHost) Mux() protocol.Switch { return rh.host.Mux() } +func (rh *RoutedHost) EventBus() event.Bus { + return rh.host.EventBus() +} + func (rh *RoutedHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) { rh.host.SetStreamHandler(pid, handler) } diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index a3ddddaa45..f44716b25f 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -5,20 +5,21 @@ import ( "sync" "time" + "github.com/libp2p/go-eventbus" ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/helpers" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/protocol" pb "github.com/libp2p/go-libp2p/p2p/protocol/identify/pb" ggio "github.com/gogo/protobuf/io" logging "github.com/ipfs/go-log" - lgbl "github.com/libp2p/go-libp2p-loggables" - ma "github.com/multiformats/go-multiaddr" msmux "github.com/multiformats/go-multistream" ) @@ -28,9 +29,6 @@ var log = logging.Logger("net/identify") // ID is the protocol.ID of the Identify Service. const ID = "/ipfs/id/1.0.0" -// IDPush is the protocol.ID of the Identify push protocol -const IDPush = "/ipfs/id/push/1.0.0" - // LibP2PVersion holds the current protocol version for a client running this code // TODO(jbenet): fix the versioning mess. const LibP2PVersion = "ipfs/0.1.0" @@ -63,6 +61,11 @@ type IDService struct { // our own observed addresses. // TODO: instead of expiring, remove these when we disconnect observedAddrs *ObservedAddrSet + + subscription event.Subscription + emitters struct { + evtPeerProtocolsUpdated event.Emitter + } } // NewIDService constructs a new *IDService and activates it by @@ -74,12 +77,50 @@ func NewIDService(ctx context.Context, h host.Host) *IDService { currid: make(map[network.Conn]chan struct{}), observedAddrs: NewObservedAddrSet(ctx), } + + // handle local protocol handler updates, and push deltas to peers. + var err error + s.subscription, err = h.EventBus().Subscribe(&event.EvtLocalProtocolsUpdated{}, eventbus.BufSize(128)) + if err != nil { + log.Warningf("identify service not subscribed to local protocol handlers updates; err: %s", err) + } else { + go s.handleEvents() + } + + s.emitters.evtPeerProtocolsUpdated, err = h.EventBus().Emitter(&event.EvtPeerProtocolsUpdated{}) + if err != nil { + log.Warningf("identify service not emitting peer protocol updates; err: %s", err) + } + h.SetStreamHandler(ID, s.requestHandler) h.SetStreamHandler(IDPush, s.pushHandler) + h.SetStreamHandler(IDDelta, s.deltaHandler) h.Network().Notify((*netNotifiee)(s)) return s } +func (ids *IDService) handleEvents() { + sub := ids.subscription + defer func() { + _ = sub.Close() + // drain the channel. + for range sub.Out() { + } + }() + + for { + select { + case evt, more := <-sub.Out(): + if !more { + return + } + ids.fireProtocolDelta(evt.(event.EvtLocalProtocolsUpdated)) + case <-ids.ctx.Done(): + return + } + } +} + // OwnObservedAddrs returns the addresses peers have reported we've dialed from func (ids *IDService) OwnObservedAddrs() []ma.Multiaddr { return ids.observedAddrs.Addrs() @@ -137,8 +178,7 @@ func (ids *IDService) requestHandler(s network.Stream) { ids.populateMessage(&mes, s.Conn()) w.WriteMsg(&mes) - log.Debugf("%s sent message to %s %s", ID, - c.RemotePeer(), c.RemoteMultiaddr()) + log.Debugf("%s sent message to %s %s", ID, c.RemotePeer(), c.RemoteMultiaddr()) } func (ids *IDService) responseHandler(s network.Stream) { @@ -151,29 +191,51 @@ func (ids *IDService) responseHandler(s network.Stream) { s.Reset() return } - ids.consumeMessage(&mes, c) - log.Debugf("%s received message from %s %s", ID, - c.RemotePeer(), c.RemoteMultiaddr()) - go helpers.FullClose(s) -} + defer func() { go helpers.FullClose(s) }() -func (ids *IDService) pushHandler(s network.Stream) { - ids.responseHandler(s) + log.Debugf("%s received message from %s %s", s.Protocol(), c.RemotePeer(), c.RemoteMultiaddr()) + ids.consumeMessage(&mes, c) } -func (ids *IDService) Push() { +func (ids *IDService) broadcast(proto protocol.ID, payloadWriter func(s network.Stream)) { var wg sync.WaitGroup ctx, cancel := context.WithTimeout(ids.ctx, 30*time.Second) - ctx = network.WithNoDial(ctx, "identify push") + ctx = network.WithNoDial(ctx, string(proto)) + pstore := ids.Host.Peerstore() for _, p := range ids.Host.Network().Peers() { wg.Add(1) - go func(p peer.ID) { + + go func(p peer.ID, conns []network.Conn) { defer wg.Done() - s, err := ids.Host.NewStream(ctx, p, IDPush) + // if we're in the process of identifying the connection, let's wait. + // we don't use ids.IdentifyWait() to avoid unnecessary channel creation. + Loop: + for _, c := range conns { + ids.currmu.RLock() + if wait, ok := ids.currid[c]; ok { + ids.currmu.RUnlock() + select { + case <-wait: + break Loop + case <-ctx.Done(): + return + } + } + ids.currmu.RUnlock() + } + + // avoid the unnecessary stream if the peer does not support the protocol. + if sup, err := pstore.SupportsProtocols(p, string(proto)); err != nil && len(sup) == 0 { + // the peer does not support the required protocol. + return + } + // if the peerstore query errors, we go ahead anyway. + + s, err := ids.Host.NewStream(ctx, p, proto) if err != nil { log.Debugf("error opening push stream to %s: %s", p, err.Error()) return @@ -181,7 +243,7 @@ func (ids *IDService) Push() { rch := make(chan struct{}, 1) go func() { - ids.requestHandler(s) + payloadWriter(s) rch <- struct{}{} }() @@ -191,7 +253,7 @@ func (ids *IDService) Push() { // this is taking too long, abort! s.Reset() } - }(p) + }(p, ids.Host.Network().ConnsToPeer(p)) } // this supervisory goroutine is necessary to cancel the context @@ -202,7 +264,6 @@ func (ids *IDService) Push() { } func (ids *IDService) populateMessage(mes *pb.Identify, c network.Conn) { - // set protocols this node is currently handling protos := ids.Host.Mux().Protocols() mes.Protocols = make([]string, len(protos)) @@ -507,13 +568,3 @@ func (nn *netNotifiee) OpenedStream(n network.Network, v network.Stream) {} func (nn *netNotifiee) ClosedStream(n network.Network, v network.Stream) {} func (nn *netNotifiee) Listen(n network.Network, a ma.Multiaddr) {} func (nn *netNotifiee) ListenClose(n network.Network, a ma.Multiaddr) {} - -func logProtocolMismatchDisconnect(c network.Conn, protocol, agent string) { - lm := make(lgbl.DeferredMap) - lm["remotePeer"] = func() interface{} { return c.RemotePeer().Pretty() } - lm["remoteAddr"] = func() interface{} { return c.RemoteMultiaddr().String() } - lm["protocolVersion"] = protocol - lm["agentVersion"] = agent - log.Event(context.TODO(), "IdentifyProtocolMismatch", lm) - log.Debugf("IdentifyProtocolMismatch %s %s %s (disconnected)", c.RemotePeer(), protocol, agent) -} diff --git a/p2p/protocol/identify/id_delta.go b/p2p/protocol/identify/id_delta.go new file mode 100644 index 0000000000..2cc95b7290 --- /dev/null +++ b/p2p/protocol/identify/id_delta.go @@ -0,0 +1,83 @@ +package identify + +import ( + "github.com/libp2p/go-libp2p-core/event" + "github.com/libp2p/go-libp2p-core/helpers" + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/protocol" + + ggio "github.com/gogo/protobuf/io" + pb "github.com/libp2p/go-libp2p/p2p/protocol/identify/pb" +) + +const IDDelta = "/p2p/id/delta/1.0.0" + +// deltaHandler handles incoming delta updates from peers. +func (ids *IDService) deltaHandler(s network.Stream) { + c := s.Conn() + + r := ggio.NewDelimitedReader(s, 2048) + mes := pb.Identify{} + if err := r.ReadMsg(&mes); err != nil { + log.Warning("error reading identify message: ", err) + s.Reset() + return + } + + defer func() { go helpers.FullClose(s) }() + log.Debugf("%s received message from %s %s", s.Protocol(), c.RemotePeer(), c.RemoteMultiaddr()) + + delta := mes.GetDelta() + if delta == nil { + return + } + + p := s.Conn().RemotePeer() + if err := ids.consumeDelta(p, delta); err != nil { + log.Warningf("delta update from peer %s failed: %s", p, err) + } +} + +// fireProtocolDelta fires a delta message to all connected peers to signal a local protocol table update. +func (ids *IDService) fireProtocolDelta(evt event.EvtLocalProtocolsUpdated) { + mes := pb.Identify{ + Delta: &pb.Delta{ + AddedProtocols: protocol.ConvertToStrings(evt.Added), + RmProtocols: protocol.ConvertToStrings(evt.Removed), + }, + } + deltaWriter := func(s network.Stream) { + defer helpers.FullClose(s) + c := s.Conn() + err := ggio.NewDelimitedWriter(s).WriteMsg(&mes) + if err != nil { + log.Warningf("%s error while sending delta update to %s: %s", IDDelta, c.RemotePeer(), c.RemoteMultiaddr()) + return + } + log.Debugf("%s sent delta update to %s: %s", IDDelta, c.RemotePeer(), c.RemoteMultiaddr()) + } + ids.broadcast(IDDelta, deltaWriter) +} + +// consumeDelta processes an incoming delta from a peer, updating the peerstore +// and emitting the appropriate events. +func (ids *IDService) consumeDelta(id peer.ID, delta *pb.Delta) error { + err := ids.Host.Peerstore().AddProtocols(id, delta.GetAddedProtocols()...) + if err != nil { + return err + } + + err = ids.Host.Peerstore().RemoveProtocols(id, delta.GetRmProtocols()...) + if err != nil { + return err + } + + evt := event.EvtPeerProtocolsUpdated{ + Peer: id, + Added: protocol.ConvertFromStrings(delta.GetAddedProtocols()), + Removed: protocol.ConvertFromStrings(delta.GetRmProtocols()), + } + ids.emitters.evtPeerProtocolsUpdated.Emit(evt) + return nil +} diff --git a/p2p/protocol/identify/id_push.go b/p2p/protocol/identify/id_push.go new file mode 100644 index 0000000000..7f0104ce73 --- /dev/null +++ b/p2p/protocol/identify/id_push.go @@ -0,0 +1,20 @@ +package identify + +import "github.com/libp2p/go-libp2p-core/network" + +// IDPush is the protocol.ID of the Identify push protocol. It sends full identify messages containing +// the current state of the peer. +// +// It is in the process of being replaced by identify delta, which sends only diffs for better +// resource utilisation. +const IDPush = "/ipfs/id/push/1.0.0" + +// Push pushes a full identify message to all peers containing the current state. +func (ids *IDService) Push() { + ids.broadcast(IDPush, ids.requestHandler) +} + +// pushHandler handles incoming identify push streams. The behaviour is identical to the ordinary identify protocol. +func (ids *IDService) pushHandler(s network.Stream) { + ids.responseHandler(s) +} diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index d3ff401d41..4bbf91b959 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -2,17 +2,24 @@ package identify_test import ( "context" + "reflect" + "sort" "testing" "time" + "github.com/libp2p/go-eventbus" ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/event" + "github.com/libp2p/go-libp2p-core/helpers" "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/protocol" blhost "github.com/libp2p/go-libp2p-blankhost" swarmt "github.com/libp2p/go-libp2p-swarm/testing" - identify "github.com/libp2p/go-libp2p/p2p/protocol/identify" + "github.com/libp2p/go-libp2p/p2p/protocol/identify" ma "github.com/multiformats/go-multiaddr" ) @@ -182,3 +189,181 @@ func TestProtoMatching(t *testing.T) { t.Fatal("expected mismatch") } } + +func TestIdentifyDeltaOnProtocolChange(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + h1 := blhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h2 := blhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + defer h2.Close() + defer h1.Close() + + h2.SetStreamHandler(protocol.TestingID, func(_ network.Stream) {}) + + ids1 := identify.NewIDService(ctx, h1) + _ = identify.NewIDService(ctx, h2) + + if err := h1.Connect(ctx, peer.AddrInfo{ID: h2.ID(), Addrs: h2.Addrs()}); err != nil { + t.Fatal(err) + } + + conn := h1.Network().ConnsToPeer(h2.ID())[0] + ids1.IdentifyConn(conn) + select { + case <-ids1.IdentifyWait(conn): + case <-time.After(5 * time.Second): + t.Fatal("took over 5 seconds to identify") + } + + protos, err := h1.Peerstore().GetProtocols(h2.ID()) + if err != nil { + t.Fatal(err) + } + sort.Strings(protos) + if sort.SearchStrings(protos, string(protocol.TestingID)) == len(protos) { + t.Fatalf("expected peer 1 to know that peer 2 speaks the Test protocol amongst others") + } + + // set up a subscriber to listen to peer protocol updated events in h1. We expect to receive events from h2 + // as protocols are added and removed. + sub, err := h1.EventBus().Subscribe(&event.EvtPeerProtocolsUpdated{}, eventbus.BufSize(16)) + if err != nil { + t.Fatal(err) + } + defer sub.Close() + + // add two new protocols in h2 and wait for identify to send deltas. + h2.SetStreamHandler(protocol.ID("foo"), func(_ network.Stream) {}) + h2.SetStreamHandler(protocol.ID("bar"), func(_ network.Stream) {}) + <-time.After(500 * time.Millisecond) + + // check that h1 now knows about h2's new protocols. + protos, err = h1.Peerstore().GetProtocols(h2.ID()) + if err != nil { + t.Fatal(err) + } + have := make(map[string]struct{}, len(protos)) + for _, p := range protos { + have[p] = struct{}{} + } + + if _, ok := have["foo"]; !ok { + t.Fatalf("expected peer 1 to know that peer 2 now speaks protocol 'foo', known: %v", protos) + } + if _, ok := have["bar"]; !ok { + t.Fatalf("expected peer 1 to know that peer 2 now speaks protocol 'bar', known: %v", protos) + } + + // remove one of the newly added protocols from h2, and wait for identify to send the delta. + h2.RemoveStreamHandler(protocol.ID("bar")) + <-time.After(500 * time.Millisecond) + + // check that h1 now has forgotten about h2's bar protocol. + protos, err = h1.Peerstore().GetProtocols(h2.ID()) + if err != nil { + t.Fatal(err) + } + have = make(map[string]struct{}, len(protos)) + for _, p := range protos { + have[p] = struct{}{} + } + if _, ok := have["foo"]; !ok { + t.Fatalf("expected peer 1 to know that peer 2 now speaks protocol 'foo', known: %v", protos) + } + if _, ok := have["bar"]; ok { + t.Fatalf("expected peer 1 to have forgotten that peer 2 spoke protocol 'bar', known: %v", protos) + } + + // make sure that h1 emitted events in the eventbus for h2's protocol updates. + evts := make([]event.EvtPeerProtocolsUpdated, 3) + done := make(chan struct{}) + go func() { + evts[0] = (<-sub.Out()).(event.EvtPeerProtocolsUpdated) + evts[1] = (<-sub.Out()).(event.EvtPeerProtocolsUpdated) + evts[2] = (<-sub.Out()).(event.EvtPeerProtocolsUpdated) + close(done) + }() + + select { + case <-done: + case <-time.After(1 * time.Second): + t.Fatalf("timed out while consuming events from subscription") + } + + added := protocol.ConvertToStrings(append(evts[0].Added, append(evts[1].Added, evts[2].Added...)...)) + removed := protocol.ConvertToStrings(append(evts[0].Removed, append(evts[1].Removed, evts[2].Removed...)...)) + sort.Strings(added) + sort.Strings(removed) + + if !reflect.DeepEqual(added, []string{"bar", "foo"}) { + t.Fatalf("expected to have received updates for added protos") + } + if !reflect.DeepEqual(removed, []string{"bar"}) { + t.Fatalf("expected to have received updates for removed protos") + } +} + +// TestIdentifyDeltaWhileIdentifyingConn tests that the host waits to push delta updates if an identify is ongoing. +func TestIdentifyDeltaWhileIdentifyingConn(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + h1 := blhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h2 := blhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + defer h2.Close() + defer h1.Close() + + _ = identify.NewIDService(ctx, h1) + ids2 := identify.NewIDService(ctx, h2) + + // replace the original identify handler by one that blocks until we close the block channel. + // this allows us to control how long identify runs. + block := make(chan struct{}) + h1.RemoveStreamHandler(identify.ID) + h1.SetStreamHandler(identify.ID, func(s network.Stream) { + <-block + go helpers.FullClose(s) + }) + + // from h2 connect to h1. + if err := h2.Connect(ctx, peer.AddrInfo{ID: h1.ID(), Addrs: h1.Addrs()}); err != nil { + t.Fatal(err) + } + + // from h2, identify h1. + conn := h2.Network().ConnsToPeer(h1.ID())[0] + go func() { + ids2.IdentifyConn(conn) + <-ids2.IdentifyWait(conn) + }() + + <-time.After(500 * time.Millisecond) + + // subscribe to events in h1; after identify h1 should receive the delta from h2 and publish an event in the bus. + sub, err := h1.EventBus().Subscribe(&event.EvtPeerProtocolsUpdated{}, eventbus.BufSize(16)) + if err != nil { + t.Fatal(err) + } + defer sub.Close() + + // add a handler in h2; the delta to h1 will queue until we're done identifying h1. + h2.SetStreamHandler(protocol.TestingID, func(_ network.Stream) {}) + <-time.After(500 * time.Millisecond) + + // make sure we haven't received any events yet. + if q := len(sub.Out()); q > 0 { + t.Fatalf("expected no events yet; queued: %d", q) + } + + close(block) + select { + case evt := <-sub.Out(): + e := evt.(event.EvtPeerProtocolsUpdated) + if e.Peer != h2.ID() || len(e.Added) != 1 || e.Added[0] != protocol.TestingID { + t.Fatalf("expected an event for protocol changes in h2, with the testing protocol added; instead got: %v", evt) + } + case <-time.After(2 * time.Second): + t.Fatalf("timed out while waiting for an event for the protocol changes in h2") + } +} diff --git a/p2p/protocol/identify/pb/identify.pb.go b/p2p/protocol/identify/pb/identify.pb.go index de1275ba3f..8480e5ba54 100644 --- a/p2p/protocol/identify/pb/identify.pb.go +++ b/p2p/protocol/identify/pb/identify.pb.go @@ -21,6 +21,63 @@ var _ = math.Inf // proto package needs to be updated. const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package +type Delta struct { + // new protocols now serviced by the peer. + AddedProtocols []string `protobuf:"bytes,1,rep,name=added_protocols,json=addedProtocols" json:"added_protocols,omitempty"` + // protocols dropped by the peer. + RmProtocols []string `protobuf:"bytes,2,rep,name=rm_protocols,json=rmProtocols" json:"rm_protocols,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *Delta) Reset() { *m = Delta{} } +func (m *Delta) String() string { return proto.CompactTextString(m) } +func (*Delta) ProtoMessage() {} +func (*Delta) Descriptor() ([]byte, []int) { + return fileDescriptor_83f1e7e6b485409f, []int{0} +} +func (m *Delta) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Delta) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Delta.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *Delta) XXX_Merge(src proto.Message) { + xxx_messageInfo_Delta.Merge(m, src) +} +func (m *Delta) XXX_Size() int { + return m.Size() +} +func (m *Delta) XXX_DiscardUnknown() { + xxx_messageInfo_Delta.DiscardUnknown(m) +} + +var xxx_messageInfo_Delta proto.InternalMessageInfo + +func (m *Delta) GetAddedProtocols() []string { + if m != nil { + return m.AddedProtocols + } + return nil +} + +func (m *Delta) GetRmProtocols() []string { + if m != nil { + return m.RmProtocols + } + return nil +} + type Identify struct { // protocolVersion determines compatibility between peers ProtocolVersion *string `protobuf:"bytes,5,opt,name=protocolVersion" json:"protocolVersion,omitempty"` @@ -38,7 +95,9 @@ type Identify struct { // determine whether its connection to the local peer goes through NAT. ObservedAddr []byte `protobuf:"bytes,4,opt,name=observedAddr" json:"observedAddr,omitempty"` // protocols are the services this node is running - Protocols []string `protobuf:"bytes,3,rep,name=protocols" json:"protocols,omitempty"` + Protocols []string `protobuf:"bytes,3,rep,name=protocols" json:"protocols,omitempty"` + // a delta update is incompatible with everything else. If this field is included, none of the others can appear. + Delta *Delta `protobuf:"bytes,7,opt,name=delta" json:"delta,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_unrecognized []byte `json:"-"` XXX_sizecache int32 `json:"-"` @@ -48,7 +107,7 @@ func (m *Identify) Reset() { *m = Identify{} } func (m *Identify) String() string { return proto.CompactTextString(m) } func (*Identify) ProtoMessage() {} func (*Identify) Descriptor() ([]byte, []int) { - return fileDescriptor_83f1e7e6b485409f, []int{0} + return fileDescriptor_83f1e7e6b485409f, []int{1} } func (m *Identify) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -119,26 +178,89 @@ func (m *Identify) GetProtocols() []string { return nil } +func (m *Identify) GetDelta() *Delta { + if m != nil { + return m.Delta + } + return nil +} + func init() { + proto.RegisterType((*Delta)(nil), "identify.pb.Delta") proto.RegisterType((*Identify)(nil), "identify.pb.Identify") } func init() { proto.RegisterFile("identify.proto", fileDescriptor_83f1e7e6b485409f) } var fileDescriptor_83f1e7e6b485409f = []byte{ - // 187 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0xcb, 0x4c, 0x49, 0xcd, - 0x2b, 0xc9, 0x4c, 0xab, 0xd4, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x46, 0xf0, 0x93, 0x94, - 0x6e, 0x31, 0x72, 0x71, 0x78, 0x42, 0xf9, 0x42, 0x1a, 0x5c, 0xfc, 0x60, 0x25, 0xc9, 0xf9, 0x39, - 0x61, 0xa9, 0x45, 0xc5, 0x99, 0xf9, 0x79, 0x12, 0xac, 0x0a, 0x8c, 0x1a, 0x9c, 0x41, 0xe8, 0xc2, - 0x42, 0x4a, 0x5c, 0x3c, 0x89, 0xe9, 0xa9, 0x79, 0x25, 0x30, 0x65, 0x6c, 0x60, 0x65, 0x28, 0x62, - 0x42, 0x32, 0x5c, 0x9c, 0x05, 0xa5, 0x49, 0x39, 0x99, 0xc9, 0xde, 0xa9, 0x95, 0x12, 0x8c, 0x0a, - 0x8c, 0x1a, 0x3c, 0x41, 0x08, 0x01, 0x21, 0x05, 0x2e, 0xee, 0x9c, 0xcc, 0xe2, 0x92, 0xd4, 0x3c, - 0xc7, 0x94, 0x94, 0xa2, 0x62, 0x09, 0x26, 0x05, 0x66, 0x0d, 0x9e, 0x20, 0x64, 0x21, 0x90, 0x1d, - 0xf9, 0x49, 0xc5, 0xa9, 0x45, 0x65, 0xa9, 0x29, 0x20, 0x01, 0x09, 0x16, 0xb0, 0x11, 0x28, 0x62, - 0x60, 0x3b, 0xa0, 0x4e, 0x2b, 0x96, 0x60, 0x56, 0x60, 0xd6, 0xe0, 0x0c, 0x42, 0x08, 0x38, 0xf1, - 0x9c, 0x78, 0x24, 0xc7, 0x78, 0xe1, 0x91, 0x1c, 0xe3, 0x83, 0x47, 0x72, 0x8c, 0x80, 0x00, 0x00, - 0x00, 0xff, 0xff, 0xc2, 0x2c, 0x19, 0x46, 0x08, 0x01, 0x00, 0x00, + // 251 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x5c, 0x90, 0xb1, 0x4e, 0xc3, 0x30, + 0x14, 0x45, 0xe5, 0x96, 0x02, 0x79, 0xb1, 0x5a, 0xe9, 0x4d, 0x1e, 0x50, 0x64, 0xb2, 0xe0, 0x29, + 0x03, 0x7f, 0x00, 0x62, 0x41, 0x2c, 0xc8, 0x48, 0xac, 0x28, 0xa9, 0x1f, 0xc8, 0x52, 0x1a, 0x57, + 0x8e, 0x41, 0xea, 0xce, 0xc7, 0x31, 0xf2, 0x09, 0x28, 0x5f, 0x82, 0xe2, 0x92, 0x26, 0x65, 0xf4, + 0xd1, 0x91, 0xef, 0xbb, 0x17, 0x96, 0xd6, 0x50, 0x13, 0xec, 0xeb, 0xae, 0xd8, 0x7a, 0x17, 0x1c, + 0xa6, 0xe3, 0xbb, 0xca, 0x9f, 0x60, 0x71, 0x47, 0x75, 0x28, 0xf1, 0x0a, 0x56, 0xa5, 0x31, 0x64, + 0x5e, 0xa2, 0xb4, 0x76, 0x75, 0x2b, 0x98, 0x9c, 0xab, 0x44, 0x2f, 0x23, 0x7e, 0x1c, 0x28, 0x5e, + 0x02, 0xf7, 0x9b, 0x89, 0x35, 0x8b, 0x56, 0xea, 0x37, 0x07, 0x25, 0xff, 0x9c, 0xc1, 0xf9, 0xfd, + 0x5f, 0x08, 0x2a, 0x58, 0x0d, 0xf2, 0x33, 0xf9, 0xd6, 0xba, 0x46, 0x2c, 0x24, 0x53, 0x89, 0xfe, + 0x8f, 0x31, 0x07, 0x5e, 0xbe, 0x51, 0x13, 0x06, 0xed, 0x34, 0x6a, 0x47, 0x0c, 0x2f, 0x20, 0xd9, + 0xbe, 0x57, 0xb5, 0x5d, 0x3f, 0xd0, 0x4e, 0x30, 0xc9, 0x14, 0xd7, 0x23, 0x40, 0x09, 0x69, 0x6d, + 0xdb, 0x40, 0xcd, 0x8d, 0x31, 0x7e, 0x7f, 0x1a, 0xd7, 0x53, 0xd4, 0x67, 0xb8, 0xaa, 0x25, 0xff, + 0x41, 0xa6, 0x07, 0xe2, 0x24, 0x7e, 0x71, 0xc4, 0x62, 0xc6, 0xa1, 0xde, 0x3c, 0xd6, 0x1b, 0x01, + 0x2a, 0x58, 0x98, 0x7e, 0x31, 0x71, 0x26, 0x99, 0x4a, 0xaf, 0xb1, 0x98, 0xcc, 0x59, 0xc4, 0x2d, + 0xf5, 0x5e, 0xb8, 0xe5, 0x5f, 0x5d, 0xc6, 0xbe, 0xbb, 0x8c, 0xfd, 0x74, 0x19, 0xfb, 0x0d, 0x00, + 0x00, 0xff, 0xff, 0xa1, 0x77, 0x03, 0x42, 0x87, 0x01, 0x00, 0x00, +} + +func (m *Delta) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Delta) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.AddedProtocols) > 0 { + for _, s := range m.AddedProtocols { + dAtA[i] = 0xa + i++ + l = len(s) + for l >= 1<<7 { + dAtA[i] = uint8(uint64(l)&0x7f | 0x80) + l >>= 7 + i++ + } + dAtA[i] = uint8(l) + i++ + i += copy(dAtA[i:], s) + } + } + if len(m.RmProtocols) > 0 { + for _, s := range m.RmProtocols { + dAtA[i] = 0x12 + i++ + l = len(s) + for l >= 1<<7 { + dAtA[i] = uint8(uint64(l)&0x7f | 0x80) + l >>= 7 + i++ + } + dAtA[i] = uint8(l) + i++ + i += copy(dAtA[i:], s) + } + } + if m.XXX_unrecognized != nil { + i += copy(dAtA[i:], m.XXX_unrecognized) + } + return i, nil } func (m *Identify) Marshal() (dAtA []byte, err error) { @@ -203,6 +325,16 @@ func (m *Identify) MarshalTo(dAtA []byte) (int, error) { i = encodeVarintIdentify(dAtA, i, uint64(len(*m.AgentVersion))) i += copy(dAtA[i:], *m.AgentVersion) } + if m.Delta != nil { + dAtA[i] = 0x3a + i++ + i = encodeVarintIdentify(dAtA, i, uint64(m.Delta.Size())) + n1, err := m.Delta.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n1 + } if m.XXX_unrecognized != nil { i += copy(dAtA[i:], m.XXX_unrecognized) } @@ -218,6 +350,30 @@ func encodeVarintIdentify(dAtA []byte, offset int, v uint64) int { dAtA[offset] = uint8(v) return offset + 1 } +func (m *Delta) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if len(m.AddedProtocols) > 0 { + for _, s := range m.AddedProtocols { + l = len(s) + n += 1 + l + sovIdentify(uint64(l)) + } + } + if len(m.RmProtocols) > 0 { + for _, s := range m.RmProtocols { + l = len(s) + n += 1 + l + sovIdentify(uint64(l)) + } + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + func (m *Identify) Size() (n int) { if m == nil { return 0 @@ -252,6 +408,10 @@ func (m *Identify) Size() (n int) { l = len(*m.AgentVersion) n += 1 + l + sovIdentify(uint64(l)) } + if m.Delta != nil { + l = m.Delta.Size() + n += 1 + l + sovIdentify(uint64(l)) + } if m.XXX_unrecognized != nil { n += len(m.XXX_unrecognized) } @@ -271,6 +431,124 @@ func sovIdentify(x uint64) (n int) { func sozIdentify(x uint64) (n int) { return sovIdentify(uint64((x << 1) ^ uint64((int64(x) >> 63)))) } +func (m *Delta) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Delta: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Delta: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field AddedProtocols", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + intStringLen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.AddedProtocols = append(m.AddedProtocols, string(dAtA[iNdEx:postIndex])) + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field RmProtocols", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + intStringLen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.RmProtocols = append(m.RmProtocols, string(dAtA[iNdEx:postIndex])) + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipIdentify(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthIdentify + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthIdentify + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} func (m *Identify) Unmarshal(dAtA []byte) error { l := len(dAtA) iNdEx := 0 @@ -498,6 +776,42 @@ func (m *Identify) Unmarshal(dAtA []byte) error { s := string(dAtA[iNdEx:postIndex]) m.AgentVersion = &s iNdEx = postIndex + case 7: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Delta", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowIdentify + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthIdentify + } + postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthIdentify + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.Delta == nil { + m.Delta = &Delta{} + } + if err := m.Delta.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex default: iNdEx = preIndex skippy, err := skipIdentify(dAtA[iNdEx:]) diff --git a/p2p/protocol/identify/pb/identify.proto b/p2p/protocol/identify/pb/identify.proto index 5270c4cf58..44283870f1 100644 --- a/p2p/protocol/identify/pb/identify.proto +++ b/p2p/protocol/identify/pb/identify.proto @@ -2,6 +2,13 @@ syntax = "proto2"; package identify.pb; +message Delta { + // new protocols now serviced by the peer. + repeated string added_protocols = 1; + // protocols dropped by the peer. + repeated string rm_protocols = 2; +} + message Identify { // protocolVersion determines compatibility between peers @@ -26,4 +33,7 @@ message Identify { // protocols are the services this node is running repeated string protocols = 3; + + // a delta update is incompatible with everything else. If this field is included, none of the others can appear. + optional Delta delta = 7; } From 8483b62ecdf8356615e3fd7e468c02c90bbbfbd2 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Mon, 24 Jun 2019 19:25:52 +0200 Subject: [PATCH 1340/3965] Add license files (#24) License: MIT Signed-off-by: Jakub Sztandera --- core/LICENSE-APACHE | 13 +++++++++++++ core/LICENSE-MIT | 19 +++++++++++++++++++ 2 files changed, 32 insertions(+) create mode 100644 core/LICENSE-APACHE create mode 100644 core/LICENSE-MIT diff --git a/core/LICENSE-APACHE b/core/LICENSE-APACHE new file mode 100644 index 0000000000..546514363d --- /dev/null +++ b/core/LICENSE-APACHE @@ -0,0 +1,13 @@ +Copyright 2019. Protocol Labs, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/core/LICENSE-MIT b/core/LICENSE-MIT new file mode 100644 index 0000000000..ea532a8305 --- /dev/null +++ b/core/LICENSE-MIT @@ -0,0 +1,19 @@ +Copyright 2019. Protocol Labs, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. From 7750ab73c5c0109ed3893d55285a95c6755fbe74 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Tue, 25 Jun 2019 20:47:02 +0200 Subject: [PATCH 1341/3965] Enable buffering by default License: MIT Signed-off-by: Jakub Sztandera --- p2p/host/eventbus/basic.go | 3 ++- p2p/host/eventbus/basic_test.go | 11 +++++++++++ p2p/host/eventbus/opts.go | 4 ++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index ca102b5637..28df1d5c5e 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -129,7 +129,7 @@ var _ event.Subscription = (*sub)(nil) // publishers to get blocked. CancelFunc is guaranteed to return after last send // to the channel func (b *basicBus) Subscribe(evtTypes interface{}, opts ...event.SubscriptionOpt) (_ event.Subscription, err error) { - var settings subSettings + settings := subSettings(subSettingsDefault) for _, opt := range opts { if err := opt(&settings); err != nil { return nil, err @@ -184,6 +184,7 @@ func (b *basicBus) Subscribe(evtTypes interface{}, opts ...event.SubscriptionOpt // emit(EventT{}) func (b *basicBus) Emitter(evtType interface{}, opts ...event.EmitterOpt) (e event.Emitter, err error) { var settings emitterSettings + for _, opt := range opts { if err := opt(&settings); err != nil { return nil, err diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index deddc6b558..0f0ff6feea 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -25,6 +25,17 @@ func (EventA) String() string { return "Oh, Hello" } +func TestDefaultSubIsBuffered(t *testing.T) { + bus := NewBus() + s, err := bus.Subscribe(new(EventA)) + if err != nil { + t.Fatal(err) + } + if cap(s.(*sub).ch) == 0 { + t.Fatalf("without any options subscribe should be buffered. was %d", cap(s.(*sub).ch)) + } +} + func TestEmit(t *testing.T) { bus := NewBus() sub, err := bus.Subscribe(new(EventA)) diff --git a/p2p/host/eventbus/opts.go b/p2p/host/eventbus/opts.go index 50c673685b..921923500e 100644 --- a/p2p/host/eventbus/opts.go +++ b/p2p/host/eventbus/opts.go @@ -4,6 +4,10 @@ type subSettings struct { buffer int } +var subSettingsDefault = subSettings{ + buffer: 16, +} + func BufSize(n int) func(interface{}) error { return func(s interface{}) error { s.(*subSettings).buffer = n From 81697d5dbd71e7ca87c93726b5ca4ad3e5d2b4d2 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 26 Jun 2019 10:05:45 -0400 Subject: [PATCH 1342/3965] update package table after -core refactor --- README.md | 54 ++++++++++++++++++++++++------------------- package-list.json | 58 ++++++++++++++++++++++++++--------------------- 2 files changed, 63 insertions(+), 49 deletions(-) diff --git a/README.md b/README.md index 17f482f75b..dfbb9f4139 100644 --- a/README.md +++ b/README.md @@ -120,13 +120,11 @@ List of packages currently in existence for libp2p: | ---------|---------|---------|--------- | | **Libp2p** | | [`go-libp2p`](//github.com/libp2p/go-libp2p) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p) | go-libp2p entry point | -| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-host) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | libp2p "host" interface | +| [`go-libp2p-core`](//github.com/libp2p/go-libp2p-core) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-core.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-core) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-core/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-core) | core interfaces, types, and abstractions | | [`go-libp2p-blankhost`](//github.com/libp2p/go-libp2p-blankhost) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-blankhost.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-blankhost) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-blankhost/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-blankhost) | minimal implementation of the "host" interface | | **Network** | -| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-net) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | libp2p connection and "network" interfaces | -| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-swarm) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | reference implementation | +| [`go-libp2p-swarm`](//github.com/libp2p/go-libp2p-swarm) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-swarm.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-swarm) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-swarm/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-swarm) | reference implementation of network state machine | | **Transport** | -| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | interfaces | | [`go-ws-transport`](//github.com/libp2p/go-ws-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-ws-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-ws-transport) | [![codecov](https://codecov.io/gh/libp2p/go-ws-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-ws-transport) | WebSocket transport | | [`go-tcp-transport`](//github.com/libp2p/go-tcp-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-tcp-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-tcp-transport) | [![codecov](https://codecov.io/gh/libp2p/go-tcp-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-tcp-transport) | TCP transport | | [`go-libp2p-quic-transport`](//github.com/libp2p/go-libp2p-quic-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-quic-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-quic-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-quic-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-quic-transport) | QUIC transport | @@ -136,28 +134,25 @@ List of packages currently in existence for libp2p: | [`go-libp2p-transport-upgrader`](//github.com/libp2p/go-libp2p-transport-upgrader) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-transport-upgrader.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-transport-upgrader) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport-upgrader) | upgrades multiaddr-net connections into full libp2p transports | | [`go-libp2p-reuseport-transport`](//github.com/libp2p/go-libp2p-reuseport-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-reuseport-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-reuseport-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport) | partial transport for building transports that reuse ports | | **Encrypted Channels** | -| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.com/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.com/libp2p/go-conn-security) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | interfaces | | [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-secio) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | SecIO crypto channel | +| [`go-libp2p-tls-transport`](//github.com/libp2p/go-libp2p-tls-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-tls-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-tls-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-tls-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-tls-transport) | TLS 1.3+ crypto channel | | [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.com/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.com/libp2p/go-conn-security-multistream) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | multistream multiplexed meta crypto channel | | **Private Network** | -| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | interfaces | -| [`go-libp2p-pnet`](//github.com/libp2p/go-libp2p-pnet) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-pnet.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pnet) | reference implementation | +| [`go-libp2p-pnet`](//github.com/libp2p/go-libp2p-pnet) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-pnet.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pnet) | reference private networking implementation | | **Stream Muxers** | -| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.com/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.com/libp2p/go-stream-muxer) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | interfaces | -| [`go-smux-yamux`](//github.com/whyrusleeping/go-smux-yamux) | [![Travis CI](https://travis-ci.com/whyrusleeping/go-smux-yamux.svg?branch=master)](https://travis-ci.com/whyrusleeping/go-smux-yamux) | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-yamux/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-yamux) | YAMUX stream multiplexer | -| [`go-smux-multiplex`](//github.com/whyrusleeping/go-smux-multiplex) | [![Travis CI](https://travis-ci.com/whyrusleeping/go-smux-multiplex.svg?branch=master)](https://travis-ci.com/whyrusleeping/go-smux-multiplex) | [![codecov](https://codecov.io/gh/whyrusleeping/go-smux-multiplex/branch/master/graph/badge.svg)](https://codecov.io/gh/whyrusleeping/go-smux-multiplex) | MPLEX stream multiplexer | +| [`go-libp2p-yamux`](//github.com/libp2p/go-libp2p-yamux) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-yamux.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-yamux) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-yamux/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-yamux) | YAMUX stream multiplexer | +| [`go-libp2p-mplex`](//github.com/libp2p/go-libp2p-mplex) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-mplex.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-mplex) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-mplex/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-mplex) | MPLEX stream multiplexer | | **NAT Traversal** | | [`go-libp2p-nat`](//github.com/libp2p/go-libp2p-nat) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-nat.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-nat) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-nat/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-nat) | | | **Peerstore** | -| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-peerstore) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | interfaces and reference implementation | +| [`go-libp2p-peerstore`](//github.com/libp2p/go-libp2p-peerstore) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-peerstore.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-peerstore) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peerstore/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peerstore) | reference implementation of peer metadata storage component | | **Connection Manager** | -| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | interface | -| [`go-libp2p-connmgr`](//github.com/libp2p/go-libp2p-connmgr) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-connmgr.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-connmgr) | reference implementation | +| [`go-libp2p-connmgr`](//github.com/libp2p/go-libp2p-connmgr) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-connmgr.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-connmgr) | reference implementation of connection manager | | **Routing** | -| [`go-libp2p-routing`](//github.com/libp2p/go-libp2p-routing) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-routing.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-routing) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing) | routing interfaces | | [`go-libp2p-record`](//github.com/libp2p/go-libp2p-record) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-record.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-record) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-record/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-record) | record type and validator logic | -| [`go-libp2p-routing-helpers`](//github.com/libp2p/go-libp2p-routing-helpers) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-routing-helpers.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-routing-helpers) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-routing-helpers) | helpers for composing routers | | [`go-libp2p-kad-dht`](//github.com/libp2p/go-libp2p-kad-dht) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-kad-dht) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kad-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kad-dht) | Kademlia-like router | +| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-kbucket) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademlia routing table helper types | +| [`go-libp2p-coral-dht`](//github.com/libp2p/go-libp2p-coral-dht) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-coral-dht.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-coral-dht) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-coral-dht/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-coral-dht) | Router based on Coral DHT | | [`go-libp2p-pubsub-router`](//github.com/libp2p/go-libp2p-pubsub-router) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-pubsub-router.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-pubsub-router) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub-router) | record-store over pubsub adapter | | **Consensus** | | [`go-libp2p-consensus`](//github.com/libp2p/go-libp2p-consensus) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-consensus.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-consensus) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-consensus/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-consensus) | consensus protocols interfaces | @@ -166,13 +161,6 @@ List of packages currently in existence for libp2p: | [`go-libp2p-pubsub`](//github.com/libp2p/go-libp2p-pubsub) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-pubsub.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-pubsub) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pubsub/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pubsub) | multiple pubsub over libp2p implementations | | **RPC** | | [`go-libp2p-gorpc`](//github.com/libp2p/go-libp2p-gorpc) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-gorpc.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-gorpc) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-gorpc/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-gorpc) | a simple RPC library for libp2p | -| **Metrics** | -| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-metrics) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | libp2p metrics interfaces/collectors | -| **Data Types** | -| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-peer) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | libp2p peer-ID datatype | -| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-crypto) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | libp2p key types | -| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-protocol) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | libp2p protocol datatype | -| [`go-libp2p-kbucket`](//github.com/libp2p/go-libp2p-kbucket) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-kbucket.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-kbucket) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-kbucket/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-kbucket) | Kademlia routing table helper types | | **Utilities/miscellaneous** | | [`go-libp2p-loggables`](//github.com/libp2p/go-libp2p-loggables) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-loggables.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-loggables) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-loggables/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-loggables) | logging helpers | | [`go-maddr-filter`](//github.com/libp2p/go-maddr-filter) | [![Travis CI](https://travis-ci.com/libp2p/go-maddr-filter.svg?branch=master)](https://travis-ci.com/libp2p/go-maddr-filter) | [![codecov](https://codecov.io/gh/libp2p/go-maddr-filter/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-maddr-filter) | multiaddr filtering helpers | @@ -187,9 +175,29 @@ List of packages currently in existence for libp2p: | [`go-libp2p-gostream`](//github.com/libp2p/go-libp2p-gostream) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-gostream.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-gostream) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-gostream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-gostream) | Go 'net' wrappers for libp2p | | [`go-libp2p-http`](//github.com/libp2p/go-libp2p-http) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-http.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-http) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-http/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-http) | HTTP on top of libp2p streams | | **Testing and examples** | -| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.com/libp2p/go-testutil.svg?branch=master)](https://travis-ci.com/libp2p/go-testutil) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | a collection of testing utilities for ipfs and libp2p | +| [`go-libp2p-testing`](//github.com/libp2p/go-libp2p-testing) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-testing.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-testing) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-testing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-testing) | a collection of testing utilities for libp2p | | [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-examples) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | +| **Deprecated** | +| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.com/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.com/libp2p/go-conn-security) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | absorbed into go-libp2p-core | +| [`go-peerstream`](//github.com/libp2p/go-peerstream) | [![Travis CI](https://travis-ci.com/libp2p/go-peerstream.svg?branch=master)](https://travis-ci.com/libp2p/go-peerstream) | [![codecov](https://codecov.io/gh/libp2p/go-peerstream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-peerstream) | early work that informed libp2p design | +| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.com/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.com/libp2p/go-stream-muxer) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | absorbed into go-libp2p-core | +| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.com/libp2p/go-testutil.svg?branch=master)](https://travis-ci.com/libp2p/go-testutil) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | replaced by go-libp2p-testing | | [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-circuit-progs) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | +| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-crypto) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | absorbed into go-libp2p-core | +| [`go-libp2p-conn`](//github.com/libp2p/go-libp2p-conn) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-conn.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-conn) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-conn/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-conn) | replaced by go-libp2p-swarm and go-libp2p-transport-upgrader | +| [`go-libp2p-discovery`](//github.com/libp2p/go-libp2p-discovery) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-discovery.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-discovery) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-discovery) | absorbed into go-libp2p-core | +| [`go-libp2p-dummy-conn`](//github.com/libp2p/go-libp2p-dummy-conn) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-dummy-conn.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-dummy-conn) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-dummy-conn/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-dummy-conn) | implempentation of deprecated interface | +| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-host) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | absorbed into go-libp2p-core | +| [`go-libp2p-identify`](//github.com/libp2p/go-libp2p-identify) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-identify.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-identify) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-identify/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-identify) | moved to go-libp2p | +| [`go-libp2p-interface-conn`](//github.com/libp2p/go-libp2p-interface-conn) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-conn.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-conn) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-conn/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-conn) | absorbed into go-libp2p-core | +| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | absorbed into go-libp2p-core | +| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | absorbed into go-libp2p-core | +| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-metrics) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | absorbed into go-libp2p-core | +| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-net) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | absorbed into go-libp2p-core | +| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-peer) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | absorbed into go-libp2p-core | +| [`go-libp2p-ping`](//github.com/libp2p/go-libp2p-ping) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-ping.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-ping) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-ping/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-ping) | moved to go-libp2p | +| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-protocol) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | absorbed into go-libp2p-core | +| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | absorbed into go-libp2p-core | # Contribute diff --git a/package-list.json b/package-list.json index 7679240600..21971bba11 100644 --- a/package-list.json +++ b/package-list.json @@ -8,15 +8,13 @@ "rows": [ "Libp2p", ["libp2p/go-libp2p", "go-libp2p", "go-libp2p entry point"], - ["libp2p/go-libp2p-host", "go-libp2p-host", "libp2p \"host\" interface"], + ["libp2p/go-libp2p-core", "go-libp2p-core", "core interfaces, types, and abstractions"], ["libp2p/go-libp2p-blankhost", "go-libp2p-blankhost", "minimal implementation of the \"host\" interface"], "Network", - ["libp2p/go-libp2p-net", "go-libp2p-net", "libp2p connection and \"network\" interfaces"], - ["libp2p/go-libp2p-swarm", "go-libp2p-swarm", "reference implementation"], + ["libp2p/go-libp2p-swarm", "go-libp2p-swarm", "reference implementation of network state machine"], "Transport", - ["libp2p/go-libp2p-transport", "go-libp2p-transport", "interfaces"], ["libp2p/go-ws-transport", "go-ws-transport", "WebSocket transport"], ["libp2p/go-tcp-transport", "go-tcp-transport", "TCP transport"], ["libp2p/go-libp2p-quic-transport", "go-libp2p-quic-transport", "QUIC transport"], @@ -27,34 +25,31 @@ ["libp2p/go-libp2p-reuseport-transport", "go-libp2p-reuseport-transport", "partial transport for building transports that reuse ports"], "Encrypted Channels", - ["libp2p/go-conn-security", "go-conn-security", "interfaces"], ["libp2p/go-libp2p-secio", "go-libp2p-secio", "SecIO crypto channel"], + ["libp2p/go-libp2p-tls-transport", "go-libp2p-tls-transport", "TLS 1.3+ crypto channel"], ["libp2p/go-conn-security-multistream", "go-conn-security-multistream", "multistream multiplexed meta crypto channel"], "Private Network", - ["libp2p/go-libp2p-interface-pnet", "go-libp2p-interface-pnet", "interfaces"], - ["libp2p/go-libp2p-pnet", "go-libp2p-pnet", "reference implementation"], + ["libp2p/go-libp2p-pnet", "go-libp2p-pnet", "reference private networking implementation"], "Stream Muxers", - ["libp2p/go-stream-muxer", "go-stream-muxer", "interfaces"], - ["whyrusleeping/go-smux-yamux", "go-smux-yamux", "YAMUX stream multiplexer"], - ["whyrusleeping/go-smux-multiplex", "go-smux-multiplex", "MPLEX stream multiplexer"], + ["libp2p/go-libp2p-yamux", "go-libp2p-yamux", "YAMUX stream multiplexer"], + ["libp2p/go-libp2p-mplex", "go-libp2p-mplex", "MPLEX stream multiplexer"], "NAT Traversal", ["libp2p/go-libp2p-nat", "go-libp2p-nat"], "Peerstore", - ["libp2p/go-libp2p-peerstore", "go-libp2p-peerstore", "interfaces and reference implementation"], + ["libp2p/go-libp2p-peerstore", "go-libp2p-peerstore", "reference implementation of peer metadata storage component"], "Connection Manager", - ["libp2p/go-libp2p-interface-connmgr", "go-libp2p-interface-connmgr", "interface"], - ["libp2p/go-libp2p-connmgr", "go-libp2p-connmgr", "reference implementation"], + ["libp2p/go-libp2p-connmgr", "go-libp2p-connmgr", "reference implementation of connection manager"], "Routing", - ["libp2p/go-libp2p-routing", "go-libp2p-routing", "routing interfaces"], ["libp2p/go-libp2p-record", "go-libp2p-record", "record type and validator logic"], - ["libp2p/go-libp2p-routing-helpers", "go-libp2p-routing-helpers", "helpers for composing routers"], ["libp2p/go-libp2p-kad-dht", "go-libp2p-kad-dht", "Kademlia-like router"], + ["libp2p/go-libp2p-kbucket", "go-libp2p-kbucket", "Kademlia routing table helper types"], + ["libp2p/go-libp2p-coral-dht", "go-libp2p-coral-dht", "Router based on Coral DHT"], ["libp2p/go-libp2p-pubsub-router", "go-libp2p-pubsub-router", "record-store over pubsub adapter"], "Consensus", @@ -67,15 +62,6 @@ "RPC", ["libp2p/go-libp2p-gorpc", "go-libp2p-gorpc", "a simple RPC library for libp2p"], - "Metrics", - ["libp2p/go-libp2p-metrics", "go-libp2p-metrics", "libp2p metrics interfaces/collectors"], - - "Data Types", - ["libp2p/go-libp2p-peer", "go-libp2p-peer", "libp2p peer-ID datatype"], - ["libp2p/go-libp2p-crypto", "go-libp2p-crypto", "libp2p key types"], - ["libp2p/go-libp2p-protocol", "go-libp2p-protocol", "libp2p protocol datatype"], - ["libp2p/go-libp2p-kbucket", "go-libp2p-kbucket", "Kademlia routing table helper types"], - "Utilities/miscellaneous", ["libp2p/go-libp2p-loggables", "go-libp2p-loggables", "logging helpers"], ["libp2p/go-maddr-filter", "go-maddr-filter", "multiaddr filtering helpers"], @@ -91,9 +77,29 @@ ["libp2p/go-libp2p-http", "go-libp2p-http", "HTTP on top of libp2p streams"], "Testing and examples", - ["libp2p/go-testutil", "go-testutil", "a collection of testing utilities for ipfs and libp2p"], + ["libp2p/go-libp2p-testing", "go-libp2p-testing", "a collection of testing utilities for libp2p"], ["libp2p/go-libp2p-examples", "go-libp2p-examples", "go-libp2p examples and tutorials"], - ["libp2p/go-libp2p-circuit-progs", "go-libp2p-circuit-progs", "testing programs for go-libp2p-circuit"] + "Deprecated", + ["libp2p/go-conn-security", "go-conn-security", "absorbed into go-libp2p-core"], + ["libp2p/go-peerstream", "go-peerstream", "early work that informed libp2p design"], + ["libp2p/go-stream-muxer", "go-stream-muxer", "absorbed into go-libp2p-core"], + ["libp2p/go-testutil", "go-testutil", "replaced by go-libp2p-testing"], + ["libp2p/go-libp2p-circuit-progs", "go-libp2p-circuit-progs", "testing programs for go-libp2p-circuit"], + ["libp2p/go-libp2p-crypto", "go-libp2p-crypto", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-conn", "go-libp2p-conn", "replaced by go-libp2p-swarm and go-libp2p-transport-upgrader"], + ["libp2p/go-libp2p-discovery", "go-libp2p-discovery", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-dummy-conn", "go-libp2p-dummy-conn", "implempentation of deprecated interface"], + ["libp2p/go-libp2p-host", "go-libp2p-host", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-identify", "go-libp2p-identify", "moved to go-libp2p"], + ["libp2p/go-libp2p-interface-conn", "go-libp2p-interface-conn", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-interface-connmgr", "go-libp2p-interface-connmgr", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-interface-pnet", "go-libp2p-interface-pnet", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-metrics", "go-libp2p-metrics", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-net", "go-libp2p-net", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-peer", "go-libp2p-peer", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-ping", "go-libp2p-ping", "moved to go-libp2p"], + ["libp2p/go-libp2p-protocol", "go-libp2p-protocol", "absorbed into go-libp2p-core"], + ["libp2p/go-libp2p-transport", "go-libp2p-transport", "absorbed into go-libp2p-core"] ] } From 1c05f12d971847c6292c069a31bf98a245b4a4e1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 27 Jun 2019 19:20:43 +0200 Subject: [PATCH 1343/3965] nit: avoid ValueOf We only need TypeOf. --- p2p/host/eventbus/basic.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 28df1d5c5e..a257873b6c 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -230,9 +230,9 @@ func newNode(typ reflect.Type) *node { } func (n *node) emit(event interface{}) { - eval := reflect.ValueOf(event) - if eval.Type() != n.typ { - panic(fmt.Sprintf("Emit called with wrong type. expected: %s, got: %s", n.typ, eval.Type())) + typ := reflect.TypeOf(event) + if typ != n.typ { + panic(fmt.Sprintf("Emit called with wrong type. expected: %s, got: %s", n.typ, typ)) } n.lk.RLock() From edaaa65b972252e72d0687a10c7b17290e3e9b1a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 27 Jun 2019 20:01:51 +0200 Subject: [PATCH 1344/3965] nit: fix with-node 1. It doesn't return an error and we weren't checking it anyways. 2. Avoid a goroutine unless we need it. --- p2p/host/eventbus/basic.go | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 2663fc4594..686949e2d2 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -51,7 +51,7 @@ func NewBus() event.Bus { } } -func (b *basicBus) withNode(typ reflect.Type, cb func(*node), async func(*node)) error { +func (b *basicBus) withNode(typ reflect.Type, cb func(*node), async func(*node)) { b.lk.Lock() n, ok := b.nodes[typ] @@ -65,12 +65,14 @@ func (b *basicBus) withNode(typ reflect.Type, cb func(*node), async func(*node)) cb(n) - go func() { - defer n.lk.Unlock() - async(n) - }() - - return nil + if async == nil { + n.lk.Unlock() + } else { + go func() { + defer n.lk.Unlock() + async(n) + }() + } } func (b *basicBus) tryDropNode(typ reflect.Type) { @@ -173,7 +175,7 @@ func (b *basicBus) Subscribe(evtTypes interface{}, opts ...event.SubscriptionOpt for i, etyp := range types { typ := reflect.TypeOf(etyp) - err = b.withNode(typ.Elem(), func(n *node) { + b.withNode(typ.Elem(), func(n *node) { n.sinks = append(n.sinks, out.ch) out.nodes[i] = n }, func(n *node) { @@ -215,11 +217,11 @@ func (b *basicBus) Emitter(evtType interface{}, opts ...event.EmitterOpt) (e eve } typ = typ.Elem() - err = b.withNode(typ, func(n *node) { + b.withNode(typ, func(n *node) { atomic.AddInt32(&n.nEmitters, 1) n.keepLast = n.keepLast || settings.makeStateful e = &emitter{n: n, typ: typ, dropper: b.tryDropNode} - }, func(_ *node) {}) + }, nil) return } From ee68e8c4fab5250f69dd144406a0179170e864d5 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 27 Jun 2019 20:17:04 +0200 Subject: [PATCH 1345/3965] fix: completely drain on close Issue: We could partially drain and end up with, e.g., a close event missing a matching open event. --- p2p/host/eventbus/basic.go | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 2663fc4594..c123f2a98e 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -104,15 +104,10 @@ func (s *sub) Out() <-chan interface{} { } func (s *sub) Close() error { - stop := make(chan struct{}) go func() { - for { - select { - case <-s.ch: - case <-stop: - close(s.ch) - return - } + // drain the event channel, will return when closed and drained. + // this is necessary to unblock publishes to this channel. + for range s.ch { } }() @@ -135,7 +130,7 @@ func (s *sub) Close() error { s.dropper(n.typ) } } - close(stop) + close(s.ch) return nil } From 02effd281992ff362dd683471d5ac88b2c5747ce Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 27 Jun 2019 22:33:53 +0200 Subject: [PATCH 1346/3965] fix: serialize publishing Ensure that all subscribers see events in the same order. This also ensures that the subscribers never see the initial "latest" event after some other event. fixes #16 --- p2p/host/eventbus/basic.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index c123f2a98e..6c9d640a47 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -173,7 +173,7 @@ func (b *basicBus) Subscribe(evtTypes interface{}, opts ...event.SubscriptionOpt out.nodes[i] = n }, func(n *node) { if n.keepLast { - l := n.last.Load() + l := n.last if l == nil { return } @@ -223,7 +223,7 @@ func (b *basicBus) Emitter(evtType interface{}, opts ...event.EmitterOpt) (e eve type node struct { // Note: make sure to NEVER lock basicBus.lk when this lock is held - lk sync.RWMutex + lk sync.Mutex typ reflect.Type @@ -231,7 +231,7 @@ type node struct { nEmitters int32 keepLast bool - last atomic.Value + last interface{} sinks []chan interface{} } @@ -248,13 +248,13 @@ func (n *node) emit(event interface{}) { panic(fmt.Sprintf("Emit called with wrong type. expected: %s, got: %s", n.typ, typ)) } - n.lk.RLock() + n.lk.Lock() if n.keepLast { - n.last.Store(event) + n.last = event } for _, ch := range n.sinks { ch <- event } - n.lk.RUnlock() + n.lk.Unlock() } From 4f10263bebd455c61c0d02e0cc5480dc6558f007 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 28 Jun 2019 13:26:17 +0200 Subject: [PATCH 1347/3965] reduce allocations when adding addrs --- p2p/host/peerstore/pstoremem/addr_book.go | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 1a4bccace3..19b60f9b8a 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -154,10 +154,10 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du log.Warningf("was passed nil multiaddr for %s", p) continue } - addrstr := string(addr.Bytes()) - a, found := amap[addrstr] + asBytes := addr.Bytes() + a, found := amap[string(asBytes)] // won't allocate. if !found || exp.After(a.Expires) { - amap[addrstr] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} + amap[string(asBytes)] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} mab.subManager.BroadcastAddr(p, addr) } @@ -188,14 +188,14 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du log.Warningf("was passed nil multiaddr for %s", p) continue } - // re-set all of them for new ttl. - addrstr := string(addr.Bytes()) + // re-set all of them for new ttl. + aBytes := addr.Bytes() if ttl > 0 { - amap[addrstr] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} + amap[string(aBytes)] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} mab.subManager.BroadcastAddr(p, addr) } else { - delete(amap, addrstr) + delete(amap, string(aBytes)) } } } From 2f06021e0cb4e4a77b137ba014971b81d288001a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jul 2019 12:32:01 -0700 Subject: [PATCH 1348/3965] fix: use the goprocess for closing fixes #668 --- p2p/host/basic/basic_host.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 1f877f4a79..30b5d99619 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -146,6 +146,7 @@ func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHo if h.cmgr != nil { h.cmgr.Close() } + _ = h.emitters.evtLocalProtocolsUpdated.Close() return h.Network().Close() }) @@ -746,7 +747,6 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { // Close shuts down the Host's services (network, etc). func (h *BasicHost) Close() error { - _ = h.emitters.evtLocalProtocolsUpdated.Close() return h.proc.Close() } From 428d7135a55ab158006a2627eefb934e21f51adb Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jul 2019 12:46:17 -0700 Subject: [PATCH 1349/3965] doc: ensure nobody tries to add teardown logic to the host close function again --- p2p/host/basic/basic_host.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 30b5d99619..ea938c5486 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -747,6 +747,13 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { // Close shuts down the Host's services (network, etc). func (h *BasicHost) Close() error { + // You're thinking of adding some teardown logic here, right? Well + // don't! Add any process teardown logic to the teardown function in the + // constructor. + // + // This: + // 1. May be called multiple times. + // 2. May _never_ be called if the host is stopped by the context. return h.proc.Close() } From 97f639055e820dd2f9a7775e1570ff60b3ef9c03 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jul 2019 12:47:33 -0700 Subject: [PATCH 1350/3965] test: ensure double-closing the host doesn't cause any issues --- p2p/host/basic/basic_host_test.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index b80c702c99..5d582d5e12 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -23,6 +23,13 @@ import ( madns "github.com/multiformats/go-multiaddr-dns" ) +func TestHostDoubleClose(t *testing.T) { + ctx := context.Background() + h1 := New(swarmt.GenSwarm(t, ctx)) + h1.Close() + h1.Close() +} + func TestHostSimple(t *testing.T) { ctx := context.Background() h1 := New(swarmt.GenSwarm(t, ctx)) From 58fa778c37a15fa3c2f028da49282ce2e6af6143 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jul 2019 12:55:10 -0700 Subject: [PATCH 1351/3965] dep: update go-libp2p-swarm to be less noisy The noisy tests are getting annoying. --- go.mod | 4 ++-- go.sum | 4 ++++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index edc7e01179..b44916c468 100644 --- a/go.mod +++ b/go.mod @@ -21,11 +21,11 @@ require ( github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.1.1 github.com/libp2p/go-libp2p-secio v0.1.0 - github.com/libp2p/go-libp2p-swarm v0.1.0 + github.com/libp2p/go-libp2p-swarm v0.1.1 github.com/libp2p/go-libp2p-testing v0.0.4 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 github.com/libp2p/go-libp2p-yamux v0.2.1 - github.com/libp2p/go-maddr-filter v0.0.4 + github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.0 github.com/libp2p/go-ws-transport v0.1.0 diff --git a/go.sum b/go.sum index d55c44d57e..0799a5df90 100644 --- a/go.sum +++ b/go.sum @@ -138,6 +138,8 @@ github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLM github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-swarm v0.1.0 h1:HrFk2p0awrGEgch9JXK/qp/hfjqQfgNxpLWnCiWPg5s= github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= +github.com/libp2p/go-libp2p-swarm v0.1.1 h1:QW7pjyTRIxt9yyBid52YmMRGtkFXUE/rbEVWsQ0ae+w= +github.com/libp2p/go-libp2p-swarm v0.1.1/go.mod h1:4NVJaLwq/dr5kEq79Jo6pMin7ZFwLx73ln1FTefR91Q= github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.3 h1:bdij4bKaaND7tCsaXVjRfYkMpvoOeKj9AVQGJllA6jM= github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= @@ -151,6 +153,8 @@ github.com/libp2p/go-libp2p-yamux v0.2.1 h1:Q3XYNiKCC2vIxrvUJL+Jg1kiyeEaIDNKLjgE github.com/libp2p/go-libp2p-yamux v0.2.1/go.mod h1:1FBXiHDk1VyRM1C0aez2bCfHQ4vMZKkAQzZbkSQt5fI= github.com/libp2p/go-maddr-filter v0.0.4 h1:hx8HIuuwk34KePddrp2mM5ivgPkZ09JH4AvsALRbFUs= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= +github.com/libp2p/go-maddr-filter v0.0.5 h1:CW3AgbMO6vUvT4kf87y4N+0P8KUl2aqLYhrGyDUbLSg= +github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= github.com/libp2p/go-mplex v0.0.3/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= github.com/libp2p/go-mplex v0.1.0 h1:/nBTy5+1yRyY82YaO6HXQRnO5IAGsXTjEJaR3LdTPc0= github.com/libp2p/go-mplex v0.1.0/go.mod h1:SXgmdki2kwCUlCCbfGLEgHjC4pFqhTp0ZoV6aiKgxDU= From 3a41aa03ac5361f10c86141e72e2e0f760fd0ae0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jul 2019 12:56:20 -0700 Subject: [PATCH 1352/3965] test: mark helpers --- p2p/host/basic/basic_host_test.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index 5d582d5e12..56120db846 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -136,6 +136,8 @@ func TestHostAddrsFactory(t *testing.T) { } func getHostPair(ctx context.Context, t *testing.T) (host.Host, host.Host) { + t.Helper() + h1 := New(swarmt.GenSwarm(t, ctx)) h2 := New(swarmt.GenSwarm(t, ctx)) @@ -148,6 +150,7 @@ func getHostPair(ctx context.Context, t *testing.T) (host.Host, host.Host) { } func assertWait(t *testing.T, c chan protocol.ID, exp protocol.ID) { + t.Helper() select { case proto := <-c: if proto != exp { From 4c28cc75df5cc8584b0c741d4f5ac72f4b8f76e8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jul 2019 13:08:28 -0700 Subject: [PATCH 1353/3965] test: fix test fallout from notify push --- p2p/host/basic/basic_host_test.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index 56120db846..acc1db85f6 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -209,6 +209,8 @@ func TestHostProtoPreference(t *testing.T) { t.Fatal(err) } + // XXX: This is racy now that we push protocol updates. If this tests + // fails, try allowing both protoOld and protoMinor. assertWait(t, connectedOn, protoOld) s2.Close() @@ -218,6 +220,12 @@ func TestHostProtoPreference(t *testing.T) { t.Fatal(err) } + // Force a lazy handshake as we may have received a protocol update by this point. + _, err = s3.Write([]byte("hello")) + if err != nil { + t.Fatal(err) + } + assertWait(t, connectedOn, protoMinor) s3.Close() } From 3755b9afd688280a04ffb3aa8da8e354fe588f6c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 4 Jul 2019 13:18:27 -0700 Subject: [PATCH 1354/3965] test: full-close streams in stream stress test --- p2p/net/mock/mock_test.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index 3d233ce04e..967f346b93 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -11,6 +11,7 @@ import ( "time" ci "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/helpers" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/protocol" @@ -321,7 +322,7 @@ func TestStreams(t *testing.T) { func makePinger(st string, n int) func(network.Stream) { return func(s network.Stream) { go func() { - defer s.Close() + defer helpers.FullClose(s) for i := 0; i < n; i++ { b := make([]byte, 4+len(st)) @@ -342,7 +343,7 @@ func makePinger(st string, n int) func(network.Stream) { func makePonger(st string) func(network.Stream) { return func(s network.Stream) { go func() { - defer s.Close() + defer helpers.FullClose(s) for { b := make([]byte, 4+len(st)) From c0a856ce8146329b485e49a44b050ef1715e6bcb Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 10 Jul 2019 03:19:22 -0700 Subject: [PATCH 1355/3965] fix: don't assume that transports implement stringer (#134) If they don't, this could end up reading a bunch of internal data, causing a race (and yes, this has happened). --- p2p/net/swarm/swarm_conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_conn.go b/p2p/net/swarm/swarm_conn.go index cdb9866a2a..c09957c130 100644 --- a/p2p/net/swarm/swarm_conn.go +++ b/p2p/net/swarm/swarm_conn.go @@ -122,7 +122,7 @@ func (c *Conn) start() { func (c *Conn) String() string { return fmt.Sprintf( - " %s (%s)>", + " %s (%s)>", c.conn.Transport(), c.conn.LocalMultiaddr(), c.conn.LocalPeer().Pretty(), From 9797b07de486a1337e07522b1523e5b3650c1d96 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 10 Jul 2019 14:52:26 +0100 Subject: [PATCH 1356/3965] TestStreamsStress: reduce node count, increase stream density. --- p2p/net/mock/mock_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index 967f346b93..94b39655cf 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -368,7 +368,7 @@ func TestStreamsStress(t *testing.T) { ctx := context.Background() nnodes := 100 if detectrace.WithRace() { - nnodes = 50 + nnodes = 30 } mn, err := FullMeshConnected(context.Background(), nnodes) From 633b771e9a823b71e56b8bad53de1c00b08f2d8b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 10 Jul 2019 15:45:49 +0100 Subject: [PATCH 1357/3965] make RemoveProtocols take the write lock. (#90) --- p2p/host/peerstore/pstoreds/protobook.go | 24 +++++++++++++---------- p2p/host/peerstore/pstoremem/protobook.go | 4 ++-- 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go index fd5f54cb20..fc084d6619 100644 --- a/p2p/host/peerstore/pstoreds/protobook.go +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -39,8 +39,9 @@ func NewProtoBook(meta pstore.PeerMetadata) pstore.ProtoBook { } func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { - pb.segments.get(p).Lock() - defer pb.segments.get(p).Unlock() + s := pb.segments.get(p) + s.Lock() + defer s.Unlock() protomap := make(map[string]struct{}, len(protos)) for _, proto := range protos { @@ -51,8 +52,9 @@ func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { } func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { - pb.segments.get(p).Lock() - defer pb.segments.get(p).Unlock() + s := pb.segments.get(p) + s.Lock() + defer s.Unlock() pmap, err := pb.getProtocolMap(p) if err != nil { @@ -67,8 +69,9 @@ func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { } func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { - pb.segments.get(p).RLock() - defer pb.segments.get(p).RUnlock() + s := pb.segments.get(p) + s.RLock() + defer s.RUnlock() pmap, err := pb.getProtocolMap(p) if err != nil { @@ -84,8 +87,9 @@ func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { } func (pb *dsProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { - pb.segments.get(p).RLock() - defer pb.segments.get(p).RUnlock() + s := pb.segments.get(p) + s.RLock() + defer s.RUnlock() pmap, err := pb.getProtocolMap(p) if err != nil { @@ -104,8 +108,8 @@ func (pb *dsProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, func (pb *dsProtoBook) RemoveProtocols(p peer.ID, protos ...string) error { s := pb.segments.get(p) - s.RLock() - defer s.RUnlock() + s.Lock() + defer s.Unlock() pmap, err := pb.getProtocolMap(p) if err != nil { diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index 7a0c86fe1f..d627c6c7bd 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -114,8 +114,8 @@ func (pb *memoryProtoBook) GetProtocols(p peer.ID) ([]string, error) { func (pb *memoryProtoBook) RemoveProtocols(p peer.ID, protos ...string) error { s := pb.segments.get(p) - s.RLock() - defer s.RUnlock() + s.Lock() + defer s.Unlock() protomap, ok := s.protocols[p] if !ok { From 4a8d3557c00038a642bbaaff51c290a25d576dc2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 10 Jul 2019 14:53:53 +0100 Subject: [PATCH 1358/3965] upgrade peerstore. --- go.mod | 2 +- go.sum | 70 ++++++++++++++++++++++++++++++++-------------------------- 2 files changed, 40 insertions(+), 32 deletions(-) diff --git a/go.mod b/go.mod index b44916c468..05c8ace8f5 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/libp2p/go-libp2p-mplex v0.2.1 github.com/libp2p/go-libp2p-nat v0.0.4 github.com/libp2p/go-libp2p-netutil v0.1.0 - github.com/libp2p/go-libp2p-peerstore v0.1.1 + github.com/libp2p/go-libp2p-peerstore v0.1.2 github.com/libp2p/go-libp2p-secio v0.1.0 github.com/libp2p/go-libp2p-swarm v0.1.1 github.com/libp2p/go-libp2p-testing v0.0.4 diff --git a/go.sum b/go.sum index 0799a5df90..cf468647ec 100644 --- a/go.sum +++ b/go.sum @@ -1,9 +1,10 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= github.com/AndreasBriese/bbloom v0.0.0-20180913140656-343706a395b7/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= +github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/Kubuxu/go-os-helper v0.0.1/go.mod h1:N8B+I7vPCT80IcP58r50u4+gEEcsZETFUpAzWW2ep1Y= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= -github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32 h1:qkOC5Gd33k54tobS36cXdAzJbeHaduLtnLQQwNoIi78= +github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32/go.mod h1:DrZx5ec/dmnfpw9KyYoQyYo7d0KEvTkk/5M/vbZjAr8= github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c h1:aEbSeNALREWXk0G7UdNhR3ayBV7tZ4M2PNmnrCAph6Q= github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI= @@ -16,13 +17,20 @@ github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY= github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= +github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk= +github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/coreos/go-semver v0.3.0 h1:wkHLiw0WNATZnSG7epLsujiMCgPAc9xhjJ4tgnAxmfM= github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE= github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dgraph-io/badger v1.5.5-0.20190226225317-8115aed38f8f/go.mod h1:VZxzAIRPHRVNRKRo6AXrX9BJegn6il06VMTZVJYCIjQ= +github.com/dgraph-io/badger v1.6.0-rc1/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= +github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= @@ -32,8 +40,9 @@ github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zV github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.3.0 h1:kbxbvI4Un1LUWKxufD+BiE6AEExYYgkQLQmLFqA1LFk= github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0= +github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg= +github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= @@ -41,18 +50,18 @@ github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q= github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= -github.com/gxed/hashland/keccakpg v0.0.1 h1:wrk3uMNaMxbXiHibbPO4S0ymqJMm41WiudyFSs7UnsU= github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= -github.com/gxed/hashland/murmur3 v0.0.1 h1:SheiaIt0sda5K+8FLz952/1iWS9zrnKsEJaOJu4ZbSc= github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/huin/goupnp v1.0.0 h1:wg75sLpL6DZqwHQN6E1Cfk6mtfzS45z8OV+ic+DtHRo= github.com/huin/goupnp v1.0.0/go.mod h1:n9v9KO1tAxYH82qOn+UTIFQDmx5n1Zxd/ClZDMX7Bnc= github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150/go.mod h1:PpLOETDnJ0o3iZrZfqZzyLl6l7F3c6L1oWn7OICBi6o= +github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-cid v0.0.2 h1:tuuKaZPU1M6HcejsO3AcYWW8sZ8MTvyxfc4uqB4eFE8= github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= @@ -60,6 +69,7 @@ github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAK github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8= +github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaHzfGTzuE3s= github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50= @@ -103,19 +113,14 @@ github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atq github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-libp2p-autonat v0.1.0 h1:aCWAu43Ri4nU0ZPO7NyLzUvvfqd0nE3dX0R/ZGYVgOU= github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3PtvYlJmg7NeVtPRqH8= -github.com/libp2p/go-libp2p-blankhost v0.1.1 h1:X919sCh+KLqJcNRApj43xCSiQRYqOSI88Fdf55ngf78= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.3 h1:0KycuXvPDhmehw0ASsg+s1o3IfXgCUDqfzAl94KEBOg= github.com/libp2p/go-libp2p-blankhost v0.1.3/go.mod h1:KML1//wiKR8vuuJO0y3LUd1uLv+tlkGTAr3jC0S5cLg= github.com/libp2p/go-libp2p-circuit v0.1.0 h1:eniLL3Y9aq/sryfyV1IAHj5rlvuyj3b7iz8tSiZpdhY= github.com/libp2p/go-libp2p-circuit v0.1.0/go.mod h1:Ahq4cY3V9VJcHcn1SBXjr78AbFkZeIRmfunbA7pmFh8= -github.com/libp2p/go-libp2p-core v0.0.1 h1:HSTZtFIq/W5Ue43Zw+uWZyy2Vl5WtF0zDjKN8/DT/1I= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= -github.com/libp2p/go-libp2p-core v0.0.4 h1:wNg7b2tKrZlSNibP+j7A1v8eatjf4+9+YRw0L3DUeFE= github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= -github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyLCcvo1w= github.com/libp2p/go-libp2p-core v0.0.6/go.mod h1:0d9xmaYAVY5qmbp/fcgxHT3ZJsLjYeYPMJAUKpaCHrE= -github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= @@ -128,30 +133,24 @@ github.com/libp2p/go-libp2p-nat v0.0.4 h1:+KXK324yaY701On8a0aGjTnw8467kW3ExKcqW2 github.com/libp2p/go-libp2p-nat v0.0.4/go.mod h1:N9Js/zVtAXqaeT99cXgTV9e75KpnWCvVOiGzlcHmBbY= github.com/libp2p/go-libp2p-netutil v0.1.0 h1:zscYDNVEcGxyUpMd0JReUZTrpMfia8PmLKcKF72EAMQ= github.com/libp2p/go-libp2p-netutil v0.1.0/go.mod h1:3Qv/aDqtMLTUyQeundkKsA+YCThNdbQD54k3TqjpbFU= -github.com/libp2p/go-libp2p-peer v0.2.0 h1:EQ8kMjaCUwt/Y5uLgjT8iY2qg0mGUT0N1zUjer50DsY= github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMgxjsHm7+J5kjWY= -github.com/libp2p/go-libp2p-peerstore v0.1.0 h1:MKh7pRNPHSh1fLPj8u/M/s/napdmeNpoi9BRy9lPN0E= github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HBb6u0MOvdJ7YIsmcwtY= -github.com/libp2p/go-libp2p-peerstore v0.1.1 h1:AJZF2sPpVo+0aAr3IrRiGVsPjJm1INlUQ9EGClgXJ4M= -github.com/libp2p/go-libp2p-peerstore v0.1.1/go.mod h1:ojEWnwG7JpJLkJ9REWYXQslyu9ZLrPWPEcCdiZzEbSM= +github.com/libp2p/go-libp2p-peerstore v0.1.2 h1:MamqRA9OU9U/hbpeiowG3q3QNrWI+omKX8n9erT6aeE= +github.com/libp2p/go-libp2p-peerstore v0.1.2/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLMWdB7GTQ= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= -github.com/libp2p/go-libp2p-swarm v0.1.0 h1:HrFk2p0awrGEgch9JXK/qp/hfjqQfgNxpLWnCiWPg5s= github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= github.com/libp2p/go-libp2p-swarm v0.1.1 h1:QW7pjyTRIxt9yyBid52YmMRGtkFXUE/rbEVWsQ0ae+w= github.com/libp2p/go-libp2p-swarm v0.1.1/go.mod h1:4NVJaLwq/dr5kEq79Jo6pMin7ZFwLx73ln1FTefR91Q= github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= -github.com/libp2p/go-libp2p-testing v0.0.3 h1:bdij4bKaaND7tCsaXVjRfYkMpvoOeKj9AVQGJllA6jM= github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.4 h1:Qev57UR47GcLPXWjrunv5aLIQGO4n9mhI/8/EIrEEFc= github.com/libp2p/go-libp2p-testing v0.0.4/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 h1:PZMS9lhjK9VytzMCW3tWHAXtKXmlURSc3ZdvwEcKCzw= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07qPW4r01GnTenLW4oi3lOPbUMGJJA= -github.com/libp2p/go-libp2p-yamux v0.2.0 h1:TSPZ5cMMz/wdoYsye/wU1TE4G3LDGMoeEN0xgnCKU/I= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= github.com/libp2p/go-libp2p-yamux v0.2.1 h1:Q3XYNiKCC2vIxrvUJL+Jg1kiyeEaIDNKLjgEjo3VQdI= github.com/libp2p/go-libp2p-yamux v0.2.1/go.mod h1:1FBXiHDk1VyRM1C0aez2bCfHQ4vMZKkAQzZbkSQt5fI= -github.com/libp2p/go-maddr-filter v0.0.4 h1:hx8HIuuwk34KePddrp2mM5ivgPkZ09JH4AvsALRbFUs= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.5 h1:CW3AgbMO6vUvT4kf87y4N+0P8KUl2aqLYhrGyDUbLSg= github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= @@ -173,10 +172,10 @@ github.com/libp2p/go-tcp-transport v0.1.0 h1:IGhowvEqyMFknOar4FWCKSWE0zL36UFKQti github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= github.com/libp2p/go-ws-transport v0.1.0 h1:F+0OvvdmPTDsVc4AjPHjV7L7Pk1B7D5QwtDcKE2oag4= github.com/libp2p/go-ws-transport v0.1.0/go.mod h1:rjw1MG1LU9YDC6gzmwObkPd/Sqwhw7yT74kj3raBFuo= -github.com/libp2p/go-yamux v1.2.2 h1:s6J6o7+ajoQMjHe7BEnq+EynOj5D2EoG8CuQgL3F2vg= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= @@ -185,19 +184,18 @@ github.com/miekg/dns v1.1.12 h1:WMhc1ik4LNkTg8U9l3hI1LvxKmIL+f1+WV/SZtCbDDA= github.com/miekg/dns v1.1.12/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ= -github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16 h1:5W7KhL8HVF3XCFOweFD3BNESdnO8ewyYTFT2R+/b8FQ= github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/minio/sha256-simd v0.1.0 h1:U41/2erhAKcmSI14xh/ZTUdBPOzDOIfS93ibzUSl8KM= github.com/minio/sha256-simd v0.1.0/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= -github.com/mr-tron/base58 v1.1.0 h1:Y51FGVJ91WBqCEabAi5OPUz38eAx8DakuAm5svLcsfQ= +github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= +github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= github.com/mr-tron/base58 v1.1.1/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= github.com/mr-tron/base58 v1.1.2 h1:ZEw4I2EgPKDJ2iEw0cNmLB3ROrEmkOtXIkaG7wZg+78= github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI= github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA= -github.com/multiformats/go-multiaddr v0.0.1 h1:/QUV3VBMDI6pi6xfiw7lr6xhDWWvQKn9udPn68kLSdY= github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr v0.0.2/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr v0.0.4 h1:WgMSI84/eRLdbptXMkMWDXPjPq7SPLIgGUVm2eroyU4= @@ -211,7 +209,6 @@ github.com/multiformats/go-multiaddr-net v0.0.1 h1:76O59E3FavvHqNg7jvzWzsPSW5JSi github.com/multiformats/go-multiaddr-net v0.0.1/go.mod h1:nw6HSxNmCIQH27XPGBuX+d1tnvM7ihcFwHMSstNAVUU= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= -github.com/multiformats/go-multihash v0.0.1 h1:HHwN1K12I+XllBCrqKnhX949Orn4oawPkegHMu2vDqQ= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= github.com/multiformats/go-multihash v0.0.5 h1:1wxmCvTXAifAepIMyF39vZinRw5sbqjPs/UIi93+uik= github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= @@ -226,20 +223,30 @@ github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo= github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/opentracing/opentracing-go v1.0.2 h1:3jA2P6O1F9UOrWVpwrIo17pu01KWvNWg4X946/Y5Zwg= github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= +github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a h1:/eS3yfGjQKG+9kayBkj0ip1BGhq6zJ3eaVksphxAaek= github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572/go.mod h1:w0SWMsp6j9O/dk4/ZpIhL+3CkG8ofA2vuv7k+ltqUMc= github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= +github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= +github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= +github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= +github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= +github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= +github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1/go.mod h1:8UvriyWtv5Q5EOgjHaSseUEdkQfvwFv1I/In/O2M9gc= github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6Rz4bwmou+oE6Dt4Cb2BGMur5eR/GYptkKUVHo= github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM= @@ -251,15 +258,15 @@ github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxf github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= +github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67 h1:ng3VDlRp5/DHpSWl02R4rM9I+8M2rhmsuLwAMmkLQWE= +golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f h1:R423Cnkcp5JABoeemiGEPlt9tHXFfw5kvc0yqlxRPWo= golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 h1:IcSOAf4PyMp3U3XbIEj1/xJ2BjNN2jWv7JoyOsMxXUU= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= @@ -271,26 +278,26 @@ golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190227160552-c95aed5357e7 h1:C2F/nMkR/9sfUTpvR3QrjBuTdvMUC/cFajkphs1YLQo= golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= -golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f h1:wMNYb4v58l5UBM7MYRLPG6ZhfOqbKu7X5eyFl8ZhKvA= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6 h1:bjcUS9ztw9kFmmIxJInhon/0Is3p+EHBKNgquIzo1OI= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 h1:DH4skfRX4EBpamg7iV4ZlCpblAHI6s6TDM39bFZumv8= golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k= +golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -309,6 +316,7 @@ gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= -gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= From 8e4bc2ee2cca95d80196bff05a29f0366b716aff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 10 Jul 2019 15:26:08 +0100 Subject: [PATCH 1359/3965] fix #672: wait until done; fix ponger response. --- p2p/net/mock/mock_test.go | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index 94b39655cf..62863c1b57 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -30,7 +30,6 @@ func randPeer(t *testing.T) peer.ID { } func TestNetworkSetup(t *testing.T) { - ctx := context.Background() sk1, _, err := test.RandTestKeyPair(ci.RSA, 512) if err != nil { @@ -319,28 +318,35 @@ func TestStreams(t *testing.T) { } -func makePinger(st string, n int) func(network.Stream) { - return func(s network.Stream) { +func makePinger(t *testing.T, st string, n int) func(network.Stream) chan struct{} { + t.Helper() + + return func(s network.Stream) chan struct{} { + done := make(chan struct{}, 1) go func() { + defer func() { done <- struct{}{} }() defer helpers.FullClose(s) for i := 0; i < n; i++ { b := make([]byte, 4+len(st)) if _, err := s.Write([]byte("ping" + st)); err != nil { - panic(err) + t.Fatal(err) } if _, err := io.ReadFull(s, b); err != nil { - panic(err) + t.Fatal(err) } if !bytes.Equal(b, []byte("pong"+st)) { - panic("bytes mismatch") + t.Fatal("bytes mismatch") } } }() + return done } } -func makePonger(st string) func(network.Stream) { +func makePonger(t *testing.T, st string) func(network.Stream) { + t.Helper() + return func(s network.Stream) { go func() { defer helpers.FullClose(s) @@ -351,13 +357,13 @@ func makePonger(st string) func(network.Stream) { if err == io.EOF { return } - panic(err) + t.Fatal(err) } if !bytes.Equal(b, []byte("ping"+st)) { - panic("bytes mismatch") + t.Fatal("bytes mismatch") } if _, err := s.Write([]byte("pong" + st)); err != nil { - panic(err) + t.Fatal(err) } } }() @@ -378,7 +384,7 @@ func TestStreamsStress(t *testing.T) { hosts := mn.Hosts() for _, h := range hosts { - ponger := makePonger(string(protocol.TestingID)) + ponger := makePonger(t, "pingpong") h.SetStreamHandler(protocol.TestingID, ponger) } @@ -399,7 +405,7 @@ func TestStreamsStress(t *testing.T) { } log.Infof("%d start pinging", i) - makePinger("pingpong", rand.Intn(100))(s) + <-makePinger(t, "pingpong", rand.Intn(100))(s) log.Infof("%d done pinging", i) }(i) } @@ -408,7 +414,6 @@ func TestStreamsStress(t *testing.T) { } func TestAdding(t *testing.T) { - mn := New(context.Background()) peers := []peer.ID{} From 1cec1b91914b8dc087cbca68b720d020dd1495b9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 10 Jul 2019 18:29:46 +0100 Subject: [PATCH 1360/3965] remove superfluous goroutine; avoid t.Fatal from goroutines. --- p2p/net/mock/mock_test.go | 63 ++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index 62863c1b57..d0c6526833 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -3,6 +3,7 @@ package mocknet import ( "bytes" "context" + "errors" "io" "math" "math/rand" @@ -318,33 +319,27 @@ func TestStreams(t *testing.T) { } -func makePinger(t *testing.T, st string, n int) func(network.Stream) chan struct{} { +func performPing(t *testing.T, st string, n int, s network.Stream) error { t.Helper() - return func(s network.Stream) chan struct{} { - done := make(chan struct{}, 1) - go func() { - defer func() { done <- struct{}{} }() - defer helpers.FullClose(s) + defer helpers.FullClose(s) - for i := 0; i < n; i++ { - b := make([]byte, 4+len(st)) - if _, err := s.Write([]byte("ping" + st)); err != nil { - t.Fatal(err) - } - if _, err := io.ReadFull(s, b); err != nil { - t.Fatal(err) - } - if !bytes.Equal(b, []byte("pong"+st)) { - t.Fatal("bytes mismatch") - } - } - }() - return done + for i := 0; i < n; i++ { + b := make([]byte, 4+len(st)) + if _, err := s.Write([]byte("ping" + st)); err != nil { + return err + } + if _, err := io.ReadFull(s, b); err != nil { + return err + } + if !bytes.Equal(b, []byte("pong"+st)) { + return errors.New("bytes mismatch") + } } + return nil } -func makePonger(t *testing.T, st string) func(network.Stream) { +func makePonger(t *testing.T, st string, errs chan<- error) func(network.Stream) { t.Helper() return func(s network.Stream) { @@ -357,13 +352,13 @@ func makePonger(t *testing.T, st string) func(network.Stream) { if err == io.EOF { return } - t.Fatal(err) + errs <- err } if !bytes.Equal(b, []byte("ping"+st)) { - t.Fatal("bytes mismatch") + errs <- errors.New("bytes mismatch") } if _, err := s.Write([]byte("pong" + st)); err != nil { - t.Fatal(err) + errs <- err } } }() @@ -382,9 +377,11 @@ func TestStreamsStress(t *testing.T) { t.Fatal(err) } + errs := make(chan error) + hosts := mn.Hosts() for _, h := range hosts { - ponger := makePonger(t, "pingpong") + ponger := makePonger(t, "pingpong", errs) h.SetStreamHandler(protocol.TestingID, ponger) } @@ -405,18 +402,28 @@ func TestStreamsStress(t *testing.T) { } log.Infof("%d start pinging", i) - <-makePinger(t, "pingpong", rand.Intn(100))(s) + errs <- performPing(t, "pingpong", rand.Intn(100), s) log.Infof("%d done pinging", i) }(i) } - wg.Wait() + go func() { + wg.Wait() + close(errs) + }() + + for err := range errs { + if err == nil { + continue + } + t.Fatal(err) + } } func TestAdding(t *testing.T) { mn := New(context.Background()) - peers := []peer.ID{} + var peers []peer.ID for i := 0; i < 3; i++ { sk, _, err := test.RandTestKeyPair(ci.RSA, 512) if err != nil { From 7888bb008d171b4afdff8427ff0b512c56320dd6 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 10 Jul 2019 15:30:22 -0400 Subject: [PATCH 1361/3965] Replace bytes.Equal -> subtle.ConstantTimeCompare --- core/crypto/ecdsa.go | 2 +- core/crypto/ed25519.go | 8 ++++---- core/crypto/key.go | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/core/crypto/ecdsa.go b/core/crypto/ecdsa.go index 58e5d5f5ab..c069d0cb52 100644 --- a/core/crypto/ecdsa.go +++ b/core/crypto/ecdsa.go @@ -115,7 +115,7 @@ func (ePriv *ECDSAPrivateKey) Raw() ([]byte, error) { return x509.MarshalECPrivateKey(ePriv.priv) } -// Equals compares to private keys +// Equals compares two private keys func (ePriv *ECDSAPrivateKey) Equals(o Key) bool { oPriv, ok := o.(*ECDSAPrivateKey) if !ok { diff --git a/core/crypto/ed25519.go b/core/crypto/ed25519.go index b6e553147d..598746bc06 100644 --- a/core/crypto/ed25519.go +++ b/core/crypto/ed25519.go @@ -1,7 +1,7 @@ package crypto import ( - "bytes" + "crypto/subtle" "errors" "fmt" "io" @@ -70,7 +70,7 @@ func (k *Ed25519PrivateKey) Equals(o Key) bool { return false } - return bytes.Equal(k.k, edk.k) + return subtle.ConstantTimeCompare(k.k, edk.k) == 1 } // GetPublic returns an ed25519 public key from a private key. @@ -105,7 +105,7 @@ func (k *Ed25519PublicKey) Equals(o Key) bool { return false } - return bytes.Equal(k.k, edk.k) + return subtle.ConstantTimeCompare(k.k, edk.k) == 1 } // Verify checks a signature agains the input data. @@ -131,7 +131,7 @@ func UnmarshalEd25519PrivateKey(data []byte) (PrivKey, error) { // Remove the redundant public key. See issue #36. redundantPk := data[ed25519.PrivateKeySize:] pk := data[ed25519.PrivateKeySize-ed25519.PublicKeySize : ed25519.PrivateKeySize] - if !bytes.Equal(pk, redundantPk) { + if subtle.ConstantTimeCompare(pk, redundantPk) != 1 { return nil, errors.New("expected redundant ed25519 public key to be redundant") } diff --git a/core/crypto/key.go b/core/crypto/key.go index 50167dbe02..bef2dcf63a 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -4,12 +4,12 @@ package crypto import ( - "bytes" "crypto/elliptic" "crypto/hmac" "crypto/rand" "crypto/sha1" "crypto/sha512" + "crypto/subtle" "encoding/base64" "errors" "fmt" @@ -347,5 +347,5 @@ func KeyEqual(k1, k2 Key) bool { b1, err1 := k1.Bytes() b2, err2 := k2.Bytes() - return bytes.Equal(b1, b2) && err1 == err2 + return subtle.ConstantTimeCompare(b1, b2) == 1 && err1 == err2 } From 3257399c9704774659d39ef943cbf3b5275a880c Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 10 Jul 2019 15:42:51 -0400 Subject: [PATCH 1362/3965] Return error rather than panic in GenerateEKeyPair --- core/crypto/key.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/crypto/key.go b/core/crypto/key.go index 50167dbe02..de63a6dfc5 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -143,6 +143,8 @@ func GenerateEKeyPair(curveName string) ([]byte, GenSharedKey, error) { curve = elliptic.P384() case "P-521": curve = elliptic.P521() + default: + return nil, nil, fmt.Errorf("unknown curve name") } priv, x, y, err := elliptic.GenerateKey(curve, rand.Reader) From 7fd5c8611ace27f8d87b30c743722697113d7ae2 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 10 Jul 2019 17:30:45 -0400 Subject: [PATCH 1363/3965] Remove support for blowfish --- core/crypto/key.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/core/crypto/key.go b/core/crypto/key.go index 50167dbe02..b939b8323e 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -191,10 +191,6 @@ func KeyStretcher(cipherType string, hashType string, secret []byte) (StretchedK case "AES-256": ivSize = 16 cipherKeySize = 32 - case "Blowfish": - ivSize = 8 - // Note: cypherKeySize arbitrarily selected, needs more thought - cipherKeySize = 32 } hmacKeySize := 20 From 9b5e158caf7839331d3d678594a5ed3a7820935d Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 10 Jul 2019 17:54:14 -0400 Subject: [PATCH 1364/3965] Raise minimum bits required for RSA key to 2048 --- core/crypto/key_test.go | 4 ++-- core/crypto/rsa_common.go | 6 ++++-- core/crypto/rsa_go.go | 6 +++--- core/crypto/rsa_openssl.go | 2 +- core/crypto/rsa_test.go | 6 +++--- 5 files changed, 13 insertions(+), 11 deletions(-) diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index 2a5e89a8c2..dffc844b5f 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -17,7 +17,7 @@ func TestKeys(t *testing.T) { } func testKeyType(typ int, t *testing.T) { - sk, pk, err := test.RandTestKeyPair(typ, 512) + sk, pk, err := test.RandTestKeyPair(typ, 2048) if err != nil { t.Fatal(err) } @@ -114,7 +114,7 @@ func testKeyEquals(t *testing.T, k Key) { t.Fatal("Key not equal to key with same bytes.") } - sk, pk, err := test.RandTestKeyPair(RSA, 512) + sk, pk, err := test.RandTestKeyPair(RSA, 2048) if err != nil { t.Fatal(err) } diff --git a/core/crypto/rsa_common.go b/core/crypto/rsa_common.go index d50651f218..fe55cbdada 100644 --- a/core/crypto/rsa_common.go +++ b/core/crypto/rsa_common.go @@ -1,10 +1,12 @@ package crypto import ( - "errors" + "fmt" ) +const MinRsaKeyBits = 2048 + // ErrRsaKeyTooSmall is returned when trying to generate or parse an RSA key // that's smaller than 512 bits. Keys need to be larger enough to sign a 256bit // hash so this is a reasonable absolute minimum. -var ErrRsaKeyTooSmall = errors.New("rsa keys must be >= 512 bits to be useful") +var ErrRsaKeyTooSmall = fmt.Errorf("rsa keys must be >= %d bits to be useful", MinRsaKeyBits) diff --git a/core/crypto/rsa_go.go b/core/crypto/rsa_go.go index e9813779dc..023588efbb 100644 --- a/core/crypto/rsa_go.go +++ b/core/crypto/rsa_go.go @@ -27,7 +27,7 @@ type RsaPublicKey struct { // GenerateRSAKeyPair generates a new rsa private and public key func GenerateRSAKeyPair(bits int, src io.Reader) (PrivKey, PubKey, error) { - if bits < 512 { + if bits < MinRsaKeyBits { return nil, nil, ErrRsaKeyTooSmall } priv, err := rsa.GenerateKey(src, bits) @@ -102,7 +102,7 @@ func UnmarshalRsaPrivateKey(b []byte) (PrivKey, error) { if err != nil { return nil, err } - if sk.N.BitLen() < 512 { + if sk.N.BitLen() < MinRsaKeyBits { return nil, ErrRsaKeyTooSmall } return &RsaPrivateKey{sk: *sk}, nil @@ -118,7 +118,7 @@ func UnmarshalRsaPublicKey(b []byte) (PubKey, error) { if !ok { return nil, errors.New("not actually an rsa public key") } - if pk.N.BitLen() < 512 { + if pk.N.BitLen() < MinRsaKeyBits { return nil, ErrRsaKeyTooSmall } return &RsaPublicKey{*pk}, nil diff --git a/core/crypto/rsa_openssl.go b/core/crypto/rsa_openssl.go index 96c5588615..913dead6af 100644 --- a/core/crypto/rsa_openssl.go +++ b/core/crypto/rsa_openssl.go @@ -21,7 +21,7 @@ type RsaPublicKey struct { // GenerateRSAKeyPair generates a new rsa private and public key func GenerateRSAKeyPair(bits int, _ io.Reader) (PrivKey, PubKey, error) { - if bits < 512 { + if bits < MinRsaKeyBits { return nil, nil, ErrRsaKeyTooSmall } diff --git a/core/crypto/rsa_test.go b/core/crypto/rsa_test.go index 7ee520acb7..08db1366ad 100644 --- a/core/crypto/rsa_test.go +++ b/core/crypto/rsa_test.go @@ -6,7 +6,7 @@ import ( ) func TestRSABasicSignAndVerify(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) if err != nil { t.Fatal(err) } @@ -47,7 +47,7 @@ func TestRSASmallKey(t *testing.T) { } func TestRSASignZero(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) if err != nil { t.Fatal(err) } @@ -68,7 +68,7 @@ func TestRSASignZero(t *testing.T) { } func TestRSAMarshalLoop(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) if err != nil { t.Fatal(err) } From 4ff4dbef600ebc8fde684b9eacae8f287e8ab34f Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Fri, 12 Jul 2019 13:24:30 -0400 Subject: [PATCH 1365/3965] update insecure transport to plaintext/2.0.0 (#37) * add plaintext/2.0.0 (with ugly protoc hack) * gofmt * gofmt (for real this time) * add `go_package` option to proto files * Revert "add `go_package` option to proto files" 5a543a79bd89d75c9697f852638b8fe56da862f4 * less hacky protobuf imports * add doc comment for PublicKeyFromProto * clean up handshake * go fmt, lol * use network.MessageSizeMax for ggio reader --- core/crypto/key.go | 29 +- core/sec/insecure/insecure.go | 153 ++++++++-- core/sec/insecure/insecure_test.go | 160 +++++++++++ core/sec/insecure/pb/Makefile | 11 + core/sec/insecure/pb/plaintext.pb.go | 404 +++++++++++++++++++++++++++ core/sec/insecure/pb/plaintext.proto | 10 + 6 files changed, 739 insertions(+), 28 deletions(-) create mode 100644 core/sec/insecure/insecure_test.go create mode 100644 core/sec/insecure/pb/Makefile create mode 100644 core/sec/insecure/pb/plaintext.pb.go create mode 100644 core/sec/insecure/pb/plaintext.proto diff --git a/core/crypto/key.go b/core/crypto/key.go index 50167dbe02..3bec907418 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -276,29 +276,46 @@ func UnmarshalPublicKey(data []byte) (PubKey, error) { if err != nil { return nil, err } + return PublicKeyFromProto(*pmes) +} - um, ok := PubKeyUnmarshallers[pmes.GetType()] +// PublicKeyFromProto converts an unserialized protobuf PublicKey message +// into its representative object. To convert a serialized public key, +// see UnmarshalPublicKey. +func PublicKeyFromProto(keyMessage pb.PublicKey) (PubKey, error) { + um, ok := PubKeyUnmarshallers[keyMessage.GetType()] if !ok { return nil, ErrBadKeyType } - return um(pmes.GetData()) + return um(keyMessage.GetData()) } // MarshalPublicKey converts a public key object into a protobuf serialized // public key func MarshalPublicKey(k PubKey) ([]byte, error) { - pbmes := new(pb.PublicKey) - pbmes.Type = k.Type() - data, err := k.Raw() + pbmes, err := PublicKeyToProto(k) if err != nil { return nil, err } - pbmes.Data = data return proto.Marshal(pbmes) } +// PublicKeyToProto converts a public key object into an unserialized protobuf +// PublicKey message. +func PublicKeyToProto(k PubKey) (*pb.PublicKey, error) { + data, err := k.Raw() + if err != nil { + return nil, err + } + + pbmes := new(pb.PublicKey) + pbmes.Type = k.Type() + pbmes.Data = data + return pbmes, nil +} + // UnmarshalPrivateKey converts a protobuf serialized private key into its // representative object func UnmarshalPrivateKey(data []byte) (PrivKey, error) { diff --git a/core/sec/insecure/insecure.go b/core/sec/insecure/insecure.go index 801569cb0b..5f6a1bb7fc 100644 --- a/core/sec/insecure/insecure.go +++ b/core/sec/insecure/insecure.go @@ -5,64 +5,172 @@ package insecure import ( "context" + "fmt" + "github.com/libp2p/go-libp2p-core/network" "net" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" + ggio "github.com/gogo/protobuf/io" ci "github.com/libp2p/go-libp2p-core/crypto" + pb "github.com/libp2p/go-libp2p-core/sec/insecure/pb" ) // ID is the multistream-select protocol ID that should be used when identifying // this security transport. -const ID = "/plaintext/1.0.0" +const ID = "/plaintext/2.0.0" // Transport is a no-op stream security transport. It provides no -// security and simply mocks the security and identity methods to -// return peer IDs known ahead of time. +// security and simply mocks the security methods. Identity methods +// return the local peer's ID and private key, and whatever the remote +// peer presents as their ID and public key. +// No authentication of the remote identity is performed. type Transport struct { - id peer.ID + id peer.ID + key ci.PrivKey } // New constructs a new insecure transport. -func New(id peer.ID) *Transport { +func New(id peer.ID, key ci.PrivKey) *Transport { return &Transport{ - id: id, + id: id, + key: key, } } -// LocalPeer returns the transports local peer ID. +// LocalPeer returns the transport's local peer ID. func (t *Transport) LocalPeer() peer.ID { return t.id } -// LocalPrivateKey returns nil. This transport is not secure. +// LocalPrivateKey returns the local private key. +// This key is used only for identity generation and provides no security. func (t *Transport) LocalPrivateKey() ci.PrivKey { - return nil + return t.key } // SecureInbound *pretends to secure* an outbound connection to the given peer. +// It sends the local peer's ID and public key, and receives the same from the remote peer. +// No validation is performed as to the authenticity or ownership of the provided public key, +// and the key exchange provides no security. +// +// SecureInbound may fail if the remote peer sends an ID and public key that are inconsistent +// with each other, or if a network error occurs during the ID exchange. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { - return &Conn{ - Conn: insecure, - local: t.id, - }, nil + conn := &Conn{ + Conn: insecure, + local: t.id, + localPrivKey: t.key, + } + + err := conn.runHandshakeSync(ctx) + if err != nil { + return nil, err + } + + return conn, nil } // SecureOutbound *pretends to secure* an outbound connection to the given peer. +// It sends the local peer's ID and public key, and receives the same from the remote peer. +// No validation is performed as to the authenticity or ownership of the provided public key, +// and the key exchange provides no security. +// +// SecureOutbound may fail if the remote peer sends an ID and public key that are inconsistent +// with each other, or if the ID sent by the remote peer does not match the one dialed. It may +// also fail if a network error occurs during the ID exchange. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { - return &Conn{ - Conn: insecure, - local: t.id, - remote: p, - }, nil + conn := &Conn{ + Conn: insecure, + local: t.id, + localPrivKey: t.key, + } + + err := conn.runHandshakeSync(ctx) + if err != nil { + return nil, err + } + + if p != conn.remote { + return nil, fmt.Errorf("remote peer sent unexpected peer ID. expected=%s received=%s", + p, conn.remote) + } + + return conn, nil } // Conn is the connection type returned by the insecure transport. type Conn struct { net.Conn + local peer.ID remote peer.ID + + localPrivKey ci.PrivKey + remotePubKey ci.PubKey +} + +func makeExchangeMessage(privkey ci.PrivKey) (*pb.Exchange, error) { + pubkey, err := ci.PublicKeyToProto(privkey.GetPublic()) + if err != nil { + return nil, err + } + id, err := peer.IDFromPrivateKey(privkey) + if err != nil { + return nil, err + } + + return &pb.Exchange{ + Id: []byte(id), + Pubkey: pubkey, + }, nil +} + +func (ic *Conn) runHandshakeSync(ctx context.Context) error { + reader := ggio.NewDelimitedReader(ic.Conn, network.MessageSizeMax) + writer := ggio.NewDelimitedWriter(ic.Conn) + + // Generate an Exchange message + msg, err := makeExchangeMessage(ic.localPrivKey) + if err != nil { + return err + } + + // Send our Exchange and read theirs + err = writer.WriteMsg(msg) + if err != nil { + return err + } + + remoteMsg := new(pb.Exchange) + err = reader.ReadMsg(remoteMsg) + if err != nil { + return err + } + + // Pull remote ID and public key from message + remotePubkey, err := ci.PublicKeyFromProto(*remoteMsg.Pubkey) + if err != nil { + return err + } + + remoteID, err := peer.IDFromPublicKey(remotePubkey) + if err != nil { + return err + } + + // Validate that ID matches public key + if !remoteID.MatchesPublicKey(remotePubkey) { + calculatedID, _ := peer.IDFromPublicKey(remotePubkey) + return fmt.Errorf("remote peer id does not match public key. id=%s calculated_id=%s", + remoteID, calculatedID) + } + + // Add remote ID and key to conn state + ic.remotePubKey = remotePubkey + ic.remote = remoteID + return nil } // LocalPeer returns the local peer ID. @@ -76,14 +184,15 @@ func (ic *Conn) RemotePeer() peer.ID { return ic.remote } -// RemotePublicKey returns nil. This connection is not secure +// RemotePublicKey returns whatever public key was given by the remote peer. +// Note that no verification of ownership is done, as this connection is not secure. func (ic *Conn) RemotePublicKey() ci.PubKey { - return nil + return ic.remotePubKey } -// LocalPrivateKey returns nil. This connection is not secure. +// LocalPrivateKey returns the private key for the local peer. func (ic *Conn) LocalPrivateKey() ci.PrivKey { - return nil + return ic.localPrivKey } var _ sec.SecureTransport = (*Transport)(nil) diff --git a/core/sec/insecure/insecure_test.go b/core/sec/insecure/insecure_test.go new file mode 100644 index 0000000000..5c26f5d4e9 --- /dev/null +++ b/core/sec/insecure/insecure_test.go @@ -0,0 +1,160 @@ +package insecure + +import ( + "bytes" + "context" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" + "io" + "net" + "testing" + + ci "github.com/libp2p/go-libp2p-core/crypto" +) + +// Run a set of sessions through the session setup and verification. +func TestConnections(t *testing.T) { + clientTpt := newTestTransport(t, ci.RSA, 1024) + serverTpt := newTestTransport(t, ci.Ed25519, 1024) + + testConnection(t, clientTpt, serverTpt) +} + +func newTestTransport(t *testing.T, typ, bits int) *Transport { + priv, pub, err := ci.GenerateKeyPair(typ, bits) + if err != nil { + t.Fatal(err) + } + id, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Fatal(err) + } + + return &Transport{ + id: id, + key: priv, + } +} + +// Create a new pair of connected TCP sockets. +func newConnPair(t *testing.T) (net.Conn, net.Conn) { + lstnr, err := net.Listen("tcp", "localhost:0") + if err != nil { + t.Fatalf("Failed to listen: %v", err) + return nil, nil + } + + var clientErr error + var client net.Conn + addr := lstnr.Addr() + done := make(chan struct{}) + + go func() { + defer close(done) + client, clientErr = net.Dial(addr.Network(), addr.String()) + }() + + server, err := lstnr.Accept() + <-done + + lstnr.Close() + + if err != nil { + t.Fatalf("Failed to accept: %v", err) + } + + if clientErr != nil { + t.Fatalf("Failed to connect: %v", clientErr) + } + + return client, server +} + +// Create a new pair of connected sessions based off of the provided +// session generators. +func connect(t *testing.T, clientTpt, serverTpt *Transport) (sec.SecureConn, sec.SecureConn) { + client, server := newConnPair(t) + + // Connect the client and server sessions + done := make(chan struct{}) + + var clientConn sec.SecureConn + var clientErr error + go func() { + defer close(done) + clientConn, clientErr = clientTpt.SecureOutbound(context.TODO(), client, serverTpt.LocalPeer()) + }() + + serverConn, serverErr := serverTpt.SecureInbound(context.TODO(), server) + <-done + + if serverErr != nil { + t.Fatal(serverErr) + } + + if clientErr != nil { + t.Fatal(clientErr) + } + + return clientConn, serverConn +} + +// Check the peer IDs +func testIDs(t *testing.T, clientTpt, serverTpt *Transport, clientConn, serverConn sec.SecureConn) { + if clientConn.LocalPeer() != clientTpt.LocalPeer() { + t.Fatal("Client Local Peer ID mismatch.") + } + + if clientConn.RemotePeer() != serverTpt.LocalPeer() { + t.Fatal("Client Remote Peer ID mismatch.") + } + + if clientConn.LocalPeer() != serverConn.RemotePeer() { + t.Fatal("Server Local Peer ID mismatch.") + } +} + +// Check the keys +func testKeys(t *testing.T, clientTpt, serverTpt *Transport, clientConn, serverConn sec.SecureConn) { + sk := serverConn.LocalPrivateKey() + pk := sk.GetPublic() + + if !sk.Equals(serverTpt.LocalPrivateKey()) { + t.Error("Private key Mismatch.") + } + + if !pk.Equals(clientConn.RemotePublicKey()) { + t.Error("Public key mismatch.") + } +} + +// Check sending and receiving messages +func testReadWrite(t *testing.T, clientConn, serverConn sec.SecureConn) { + before := []byte("hello world") + _, err := clientConn.Write(before) + if err != nil { + t.Fatal(err) + } + + after := make([]byte, len(before)) + _, err = io.ReadFull(serverConn, after) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(before, after) { + t.Errorf("Message mismatch. %v != %v", before, after) + } +} + +// Setup a new session with a pair of locally connected sockets +func testConnection(t *testing.T, clientTpt, serverTpt *Transport) { + clientConn, serverConn := connect(t, clientTpt, serverTpt) + + testIDs(t, clientTpt, serverTpt, clientConn, serverConn) + testKeys(t, clientTpt, serverTpt, clientConn, serverConn) + testReadWrite(t, clientConn, serverConn) + + clientConn.Close() + serverConn.Close() +} diff --git a/core/sec/insecure/pb/Makefile b/core/sec/insecure/pb/Makefile new file mode 100644 index 0000000000..4fb825a4bb --- /dev/null +++ b/core/sec/insecure/pb/Makefile @@ -0,0 +1,11 @@ +PB = $(wildcard *.proto) +GO = $(PB:.proto=.pb.go) + +all: $(GO) + +%.pb.go: %.proto + protoc --proto_path=$(GOPATH)/src:../../../crypto/pb:. --gogofaster_out=. $< + +clean: + rm -f *.pb.go + rm -f *.go diff --git a/core/sec/insecure/pb/plaintext.pb.go b/core/sec/insecure/pb/plaintext.pb.go new file mode 100644 index 0000000000..68b12f0c82 --- /dev/null +++ b/core/sec/insecure/pb/plaintext.pb.go @@ -0,0 +1,404 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: plaintext.proto + +package plaintext_pb + +import ( + fmt "fmt" + proto "github.com/gogo/protobuf/proto" + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + io "io" + math "math" +) + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + +type Exchange struct { + Id []byte `protobuf:"bytes,1,opt,name=id" json:"id"` + Pubkey *pb.PublicKey `protobuf:"bytes,2,opt,name=pubkey" json:"pubkey,omitempty"` +} + +func (m *Exchange) Reset() { *m = Exchange{} } +func (m *Exchange) String() string { return proto.CompactTextString(m) } +func (*Exchange) ProtoMessage() {} +func (*Exchange) Descriptor() ([]byte, []int) { + return fileDescriptor_aba144f73931b711, []int{0} +} +func (m *Exchange) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Exchange) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Exchange.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *Exchange) XXX_Merge(src proto.Message) { + xxx_messageInfo_Exchange.Merge(m, src) +} +func (m *Exchange) XXX_Size() int { + return m.Size() +} +func (m *Exchange) XXX_DiscardUnknown() { + xxx_messageInfo_Exchange.DiscardUnknown(m) +} + +var xxx_messageInfo_Exchange proto.InternalMessageInfo + +func (m *Exchange) GetId() []byte { + if m != nil { + return m.Id + } + return nil +} + +func (m *Exchange) GetPubkey() *pb.PublicKey { + if m != nil { + return m.Pubkey + } + return nil +} + +func init() { + proto.RegisterType((*Exchange)(nil), "plaintext.pb.Exchange") +} + +func init() { proto.RegisterFile("plaintext.proto", fileDescriptor_aba144f73931b711) } + +var fileDescriptor_aba144f73931b711 = []byte{ + // 187 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x2f, 0xc8, 0x49, 0xcc, + 0xcc, 0x2b, 0x49, 0xad, 0x28, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x41, 0x12, 0x48, + 0x92, 0x32, 0x4f, 0xcf, 0x2c, 0xc9, 0x28, 0x4d, 0xd2, 0x4b, 0xce, 0xcf, 0xd5, 0xcf, 0xc9, 0x4c, + 0x2a, 0x30, 0x2a, 0xd0, 0x4f, 0xcf, 0xd7, 0x85, 0xb0, 0x74, 0x93, 0xf3, 0x8b, 0x52, 0xf5, 0x93, + 0x8b, 0x2a, 0x0b, 0x4a, 0xf2, 0xf5, 0x0b, 0x92, 0xa0, 0x2c, 0x88, 0x31, 0x4a, 0x7e, 0x5c, 0x1c, + 0xae, 0x15, 0xc9, 0x19, 0x89, 0x79, 0xe9, 0xa9, 0x42, 0x22, 0x5c, 0x4c, 0x99, 0x29, 0x12, 0x8c, + 0x0a, 0x8c, 0x1a, 0x3c, 0x4e, 0x2c, 0x27, 0xee, 0xc9, 0x33, 0x04, 0x31, 0x65, 0xa6, 0x08, 0xe9, + 0x70, 0xb1, 0x15, 0x94, 0x26, 0x65, 0xa7, 0x56, 0x4a, 0x30, 0x29, 0x30, 0x6a, 0x70, 0x1b, 0x89, + 0xe8, 0xc1, 0x0c, 0x48, 0xd2, 0x0b, 0x28, 0x4d, 0xca, 0xc9, 0x4c, 0xf6, 0x4e, 0xad, 0x0c, 0x82, + 0xaa, 0x71, 0x92, 0x38, 0xf1, 0x48, 0x8e, 0xf1, 0xc2, 0x23, 0x39, 0xc6, 0x07, 0x8f, 0xe4, 0x18, + 0x27, 0x3c, 0x96, 0x63, 0xb8, 0xf0, 0x58, 0x8e, 0xe1, 0xc6, 0x63, 0x39, 0x06, 0x40, 0x00, 0x00, + 0x00, 0xff, 0xff, 0x40, 0xde, 0x90, 0x0b, 0xc2, 0x00, 0x00, 0x00, +} + +func (m *Exchange) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Exchange) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Id != nil { + dAtA[i] = 0xa + i++ + i = encodeVarintPlaintext(dAtA, i, uint64(len(m.Id))) + i += copy(dAtA[i:], m.Id) + } + if m.Pubkey != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintPlaintext(dAtA, i, uint64(m.Pubkey.Size())) + n1, err := m.Pubkey.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n1 + } + return i, nil +} + +func encodeVarintPlaintext(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func (m *Exchange) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.Id != nil { + l = len(m.Id) + n += 1 + l + sovPlaintext(uint64(l)) + } + if m.Pubkey != nil { + l = m.Pubkey.Size() + n += 1 + l + sovPlaintext(uint64(l)) + } + return n +} + +func sovPlaintext(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozPlaintext(x uint64) (n int) { + return sovPlaintext(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *Exchange) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPlaintext + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Exchange: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Exchange: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Id", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPlaintext + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPlaintext + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPlaintext + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Id = append(m.Id[:0], dAtA[iNdEx:postIndex]...) + if m.Id == nil { + m.Id = []byte{} + } + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Pubkey", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPlaintext + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthPlaintext + } + postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthPlaintext + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.Pubkey == nil { + m.Pubkey = &pb.PublicKey{} + } + if err := m.Pubkey.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipPlaintext(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPlaintext + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthPlaintext + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipPlaintext(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPlaintext + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPlaintext + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPlaintext + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if length < 0 { + return 0, ErrInvalidLengthPlaintext + } + iNdEx += length + if iNdEx < 0 { + return 0, ErrInvalidLengthPlaintext + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPlaintext + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipPlaintext(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + if iNdEx < 0 { + return 0, ErrInvalidLengthPlaintext + } + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthPlaintext = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowPlaintext = fmt.Errorf("proto: integer overflow") +) diff --git a/core/sec/insecure/pb/plaintext.proto b/core/sec/insecure/pb/plaintext.proto new file mode 100644 index 0000000000..1e299df585 --- /dev/null +++ b/core/sec/insecure/pb/plaintext.proto @@ -0,0 +1,10 @@ +syntax = "proto2"; + +package plaintext.pb; + +import "github.com/libp2p/go-libp2p-core/crypto/pb/crypto.proto"; + +message Exchange { + optional bytes id = 1; + optional crypto.pb.PublicKey pubkey = 2; +} From cc9f17d667c628dc85734ed1038df38ae73bcbdb Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 12 Jul 2019 16:04:56 -0400 Subject: [PATCH 1366/3965] Add env flag for allowing unsafe rsa keys in tests --- core/crypto/key_test.go | 4 ++-- core/crypto/rsa_common.go | 21 +++++++++++++++++---- core/crypto/rsa_test.go | 6 +++--- 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index dffc844b5f..2a5e89a8c2 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -17,7 +17,7 @@ func TestKeys(t *testing.T) { } func testKeyType(typ int, t *testing.T) { - sk, pk, err := test.RandTestKeyPair(typ, 2048) + sk, pk, err := test.RandTestKeyPair(typ, 512) if err != nil { t.Fatal(err) } @@ -114,7 +114,7 @@ func testKeyEquals(t *testing.T, k Key) { t.Fatal("Key not equal to key with same bytes.") } - sk, pk, err := test.RandTestKeyPair(RSA, 2048) + sk, pk, err := test.RandTestKeyPair(RSA, 512) if err != nil { t.Fatal(err) } diff --git a/core/crypto/rsa_common.go b/core/crypto/rsa_common.go index fe55cbdada..89ae51c788 100644 --- a/core/crypto/rsa_common.go +++ b/core/crypto/rsa_common.go @@ -2,11 +2,24 @@ package crypto import ( "fmt" + "os" ) -const MinRsaKeyBits = 2048 +// UnsafeRsaKeyEnv is an environment variable which, when set, lowers the +// minimum required bits of RSA keys to 512. This should be used exclusively in +// test situations. +const UnsafeRsaKeyEnv = "LIBP2P_ALLOW_UNSAFE_RSA_KEYS" + +var MinRsaKeyBits = 2048 // ErrRsaKeyTooSmall is returned when trying to generate or parse an RSA key -// that's smaller than 512 bits. Keys need to be larger enough to sign a 256bit -// hash so this is a reasonable absolute minimum. -var ErrRsaKeyTooSmall = fmt.Errorf("rsa keys must be >= %d bits to be useful", MinRsaKeyBits) +// that's smaller than MinRsaKeyBits bits. In test +var ErrRsaKeyTooSmall error + +func init() { + if _, ok := os.LookupEnv(UnsafeRsaKeyEnv); ok { + MinRsaKeyBits = 512 + } + + ErrRsaKeyTooSmall = fmt.Errorf("rsa keys must be >= %d bits to be useful", MinRsaKeyBits) +} diff --git a/core/crypto/rsa_test.go b/core/crypto/rsa_test.go index 08db1366ad..7ee520acb7 100644 --- a/core/crypto/rsa_test.go +++ b/core/crypto/rsa_test.go @@ -6,7 +6,7 @@ import ( ) func TestRSABasicSignAndVerify(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) if err != nil { t.Fatal(err) } @@ -47,7 +47,7 @@ func TestRSASmallKey(t *testing.T) { } func TestRSASignZero(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) if err != nil { t.Fatal(err) } @@ -68,7 +68,7 @@ func TestRSASignZero(t *testing.T) { } func TestRSAMarshalLoop(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) if err != nil { t.Fatal(err) } From d1866f89ae8dd0fd0bea96fff33b7bb12d8f8121 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 12 Jul 2019 16:33:45 -0400 Subject: [PATCH 1367/3965] Use short-circuiting comparisons for public keys --- core/crypto/ed25519.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/core/crypto/ed25519.go b/core/crypto/ed25519.go index 598746bc06..e8834d4818 100644 --- a/core/crypto/ed25519.go +++ b/core/crypto/ed25519.go @@ -1,6 +1,7 @@ package crypto import ( + "bytes" "crypto/subtle" "errors" "fmt" @@ -105,7 +106,7 @@ func (k *Ed25519PublicKey) Equals(o Key) bool { return false } - return subtle.ConstantTimeCompare(k.k, edk.k) == 1 + return bytes.Equal(k.k, edk.k) } // Verify checks a signature agains the input data. @@ -131,7 +132,7 @@ func UnmarshalEd25519PrivateKey(data []byte) (PrivKey, error) { // Remove the redundant public key. See issue #36. redundantPk := data[ed25519.PrivateKeySize:] pk := data[ed25519.PrivateKeySize-ed25519.PublicKeySize : ed25519.PrivateKeySize] - if subtle.ConstantTimeCompare(pk, redundantPk) != 1 { + if !bytes.Equal(pk, redundantPk) { return nil, errors.New("expected redundant ed25519 public key to be redundant") } From a9b8c568af8894793be35fb7472b5d4df0545d75 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 12 Jul 2019 17:09:27 -0400 Subject: [PATCH 1368/3965] Add test for unknown elliptic curves --- core/crypto/key_test.go | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index 2a5e89a8c2..0b25432d7c 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -145,3 +145,15 @@ func (pk testkey) Raw() ([]byte, error) { func (pk testkey) Equals(k Key) bool { return KeyEqual(pk, k) } + +func TestUnknownCurveErrors(t *testing.T) { + _, _, err := GenerateEKeyPair("P-256") + if err != nil { + t.Fatal(err) + } + + _, _, err = GenerateEKeyPair("error-please") + if err == nil { + t.Fatal("expected invalid key type to error") + } +} From cfafedbcdbba36414ff1bb1bb3e7ef595fbc4679 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Mon, 15 Jul 2019 00:10:22 +0100 Subject: [PATCH 1369/3965] Revert "update insecure transport to plaintext/2.0.0 (#37)" (#38) This reverts commit 4ff4dbef600ebc8fde684b9eacae8f287e8ab34f. --- core/crypto/key.go | 29 +- core/sec/insecure/insecure.go | 153 ++-------- core/sec/insecure/insecure_test.go | 160 ----------- core/sec/insecure/pb/Makefile | 11 - core/sec/insecure/pb/plaintext.pb.go | 404 --------------------------- core/sec/insecure/pb/plaintext.proto | 10 - 6 files changed, 28 insertions(+), 739 deletions(-) delete mode 100644 core/sec/insecure/insecure_test.go delete mode 100644 core/sec/insecure/pb/Makefile delete mode 100644 core/sec/insecure/pb/plaintext.pb.go delete mode 100644 core/sec/insecure/pb/plaintext.proto diff --git a/core/crypto/key.go b/core/crypto/key.go index cad23dd47d..cb8e77858e 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -278,44 +278,27 @@ func UnmarshalPublicKey(data []byte) (PubKey, error) { if err != nil { return nil, err } - return PublicKeyFromProto(*pmes) -} -// PublicKeyFromProto converts an unserialized protobuf PublicKey message -// into its representative object. To convert a serialized public key, -// see UnmarshalPublicKey. -func PublicKeyFromProto(keyMessage pb.PublicKey) (PubKey, error) { - um, ok := PubKeyUnmarshallers[keyMessage.GetType()] + um, ok := PubKeyUnmarshallers[pmes.GetType()] if !ok { return nil, ErrBadKeyType } - return um(keyMessage.GetData()) + return um(pmes.GetData()) } // MarshalPublicKey converts a public key object into a protobuf serialized // public key func MarshalPublicKey(k PubKey) ([]byte, error) { - pbmes, err := PublicKeyToProto(k) - if err != nil { - return nil, err - } - - return proto.Marshal(pbmes) -} - -// PublicKeyToProto converts a public key object into an unserialized protobuf -// PublicKey message. -func PublicKeyToProto(k PubKey) (*pb.PublicKey, error) { + pbmes := new(pb.PublicKey) + pbmes.Type = k.Type() data, err := k.Raw() if err != nil { return nil, err } - - pbmes := new(pb.PublicKey) - pbmes.Type = k.Type() pbmes.Data = data - return pbmes, nil + + return proto.Marshal(pbmes) } // UnmarshalPrivateKey converts a protobuf serialized private key into its diff --git a/core/sec/insecure/insecure.go b/core/sec/insecure/insecure.go index 5f6a1bb7fc..801569cb0b 100644 --- a/core/sec/insecure/insecure.go +++ b/core/sec/insecure/insecure.go @@ -5,172 +5,64 @@ package insecure import ( "context" - "fmt" - "github.com/libp2p/go-libp2p-core/network" "net" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" - ggio "github.com/gogo/protobuf/io" ci "github.com/libp2p/go-libp2p-core/crypto" - pb "github.com/libp2p/go-libp2p-core/sec/insecure/pb" ) // ID is the multistream-select protocol ID that should be used when identifying // this security transport. -const ID = "/plaintext/2.0.0" +const ID = "/plaintext/1.0.0" // Transport is a no-op stream security transport. It provides no -// security and simply mocks the security methods. Identity methods -// return the local peer's ID and private key, and whatever the remote -// peer presents as their ID and public key. -// No authentication of the remote identity is performed. +// security and simply mocks the security and identity methods to +// return peer IDs known ahead of time. type Transport struct { - id peer.ID - key ci.PrivKey + id peer.ID } // New constructs a new insecure transport. -func New(id peer.ID, key ci.PrivKey) *Transport { +func New(id peer.ID) *Transport { return &Transport{ - id: id, - key: key, + id: id, } } -// LocalPeer returns the transport's local peer ID. +// LocalPeer returns the transports local peer ID. func (t *Transport) LocalPeer() peer.ID { return t.id } -// LocalPrivateKey returns the local private key. -// This key is used only for identity generation and provides no security. +// LocalPrivateKey returns nil. This transport is not secure. func (t *Transport) LocalPrivateKey() ci.PrivKey { - return t.key + return nil } // SecureInbound *pretends to secure* an outbound connection to the given peer. -// It sends the local peer's ID and public key, and receives the same from the remote peer. -// No validation is performed as to the authenticity or ownership of the provided public key, -// and the key exchange provides no security. -// -// SecureInbound may fail if the remote peer sends an ID and public key that are inconsistent -// with each other, or if a network error occurs during the ID exchange. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { - conn := &Conn{ - Conn: insecure, - local: t.id, - localPrivKey: t.key, - } - - err := conn.runHandshakeSync(ctx) - if err != nil { - return nil, err - } - - return conn, nil + return &Conn{ + Conn: insecure, + local: t.id, + }, nil } // SecureOutbound *pretends to secure* an outbound connection to the given peer. -// It sends the local peer's ID and public key, and receives the same from the remote peer. -// No validation is performed as to the authenticity or ownership of the provided public key, -// and the key exchange provides no security. -// -// SecureOutbound may fail if the remote peer sends an ID and public key that are inconsistent -// with each other, or if the ID sent by the remote peer does not match the one dialed. It may -// also fail if a network error occurs during the ID exchange. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { - conn := &Conn{ - Conn: insecure, - local: t.id, - localPrivKey: t.key, - } - - err := conn.runHandshakeSync(ctx) - if err != nil { - return nil, err - } - - if p != conn.remote { - return nil, fmt.Errorf("remote peer sent unexpected peer ID. expected=%s received=%s", - p, conn.remote) - } - - return conn, nil + return &Conn{ + Conn: insecure, + local: t.id, + remote: p, + }, nil } // Conn is the connection type returned by the insecure transport. type Conn struct { net.Conn - local peer.ID remote peer.ID - - localPrivKey ci.PrivKey - remotePubKey ci.PubKey -} - -func makeExchangeMessage(privkey ci.PrivKey) (*pb.Exchange, error) { - pubkey, err := ci.PublicKeyToProto(privkey.GetPublic()) - if err != nil { - return nil, err - } - id, err := peer.IDFromPrivateKey(privkey) - if err != nil { - return nil, err - } - - return &pb.Exchange{ - Id: []byte(id), - Pubkey: pubkey, - }, nil -} - -func (ic *Conn) runHandshakeSync(ctx context.Context) error { - reader := ggio.NewDelimitedReader(ic.Conn, network.MessageSizeMax) - writer := ggio.NewDelimitedWriter(ic.Conn) - - // Generate an Exchange message - msg, err := makeExchangeMessage(ic.localPrivKey) - if err != nil { - return err - } - - // Send our Exchange and read theirs - err = writer.WriteMsg(msg) - if err != nil { - return err - } - - remoteMsg := new(pb.Exchange) - err = reader.ReadMsg(remoteMsg) - if err != nil { - return err - } - - // Pull remote ID and public key from message - remotePubkey, err := ci.PublicKeyFromProto(*remoteMsg.Pubkey) - if err != nil { - return err - } - - remoteID, err := peer.IDFromPublicKey(remotePubkey) - if err != nil { - return err - } - - // Validate that ID matches public key - if !remoteID.MatchesPublicKey(remotePubkey) { - calculatedID, _ := peer.IDFromPublicKey(remotePubkey) - return fmt.Errorf("remote peer id does not match public key. id=%s calculated_id=%s", - remoteID, calculatedID) - } - - // Add remote ID and key to conn state - ic.remotePubKey = remotePubkey - ic.remote = remoteID - return nil } // LocalPeer returns the local peer ID. @@ -184,15 +76,14 @@ func (ic *Conn) RemotePeer() peer.ID { return ic.remote } -// RemotePublicKey returns whatever public key was given by the remote peer. -// Note that no verification of ownership is done, as this connection is not secure. +// RemotePublicKey returns nil. This connection is not secure func (ic *Conn) RemotePublicKey() ci.PubKey { - return ic.remotePubKey + return nil } -// LocalPrivateKey returns the private key for the local peer. +// LocalPrivateKey returns nil. This connection is not secure. func (ic *Conn) LocalPrivateKey() ci.PrivKey { - return ic.localPrivKey + return nil } var _ sec.SecureTransport = (*Transport)(nil) diff --git a/core/sec/insecure/insecure_test.go b/core/sec/insecure/insecure_test.go deleted file mode 100644 index 5c26f5d4e9..0000000000 --- a/core/sec/insecure/insecure_test.go +++ /dev/null @@ -1,160 +0,0 @@ -package insecure - -import ( - "bytes" - "context" - "github.com/libp2p/go-libp2p-core/peer" - "github.com/libp2p/go-libp2p-core/sec" - "io" - "net" - "testing" - - ci "github.com/libp2p/go-libp2p-core/crypto" -) - -// Run a set of sessions through the session setup and verification. -func TestConnections(t *testing.T) { - clientTpt := newTestTransport(t, ci.RSA, 1024) - serverTpt := newTestTransport(t, ci.Ed25519, 1024) - - testConnection(t, clientTpt, serverTpt) -} - -func newTestTransport(t *testing.T, typ, bits int) *Transport { - priv, pub, err := ci.GenerateKeyPair(typ, bits) - if err != nil { - t.Fatal(err) - } - id, err := peer.IDFromPublicKey(pub) - if err != nil { - t.Fatal(err) - } - - return &Transport{ - id: id, - key: priv, - } -} - -// Create a new pair of connected TCP sockets. -func newConnPair(t *testing.T) (net.Conn, net.Conn) { - lstnr, err := net.Listen("tcp", "localhost:0") - if err != nil { - t.Fatalf("Failed to listen: %v", err) - return nil, nil - } - - var clientErr error - var client net.Conn - addr := lstnr.Addr() - done := make(chan struct{}) - - go func() { - defer close(done) - client, clientErr = net.Dial(addr.Network(), addr.String()) - }() - - server, err := lstnr.Accept() - <-done - - lstnr.Close() - - if err != nil { - t.Fatalf("Failed to accept: %v", err) - } - - if clientErr != nil { - t.Fatalf("Failed to connect: %v", clientErr) - } - - return client, server -} - -// Create a new pair of connected sessions based off of the provided -// session generators. -func connect(t *testing.T, clientTpt, serverTpt *Transport) (sec.SecureConn, sec.SecureConn) { - client, server := newConnPair(t) - - // Connect the client and server sessions - done := make(chan struct{}) - - var clientConn sec.SecureConn - var clientErr error - go func() { - defer close(done) - clientConn, clientErr = clientTpt.SecureOutbound(context.TODO(), client, serverTpt.LocalPeer()) - }() - - serverConn, serverErr := serverTpt.SecureInbound(context.TODO(), server) - <-done - - if serverErr != nil { - t.Fatal(serverErr) - } - - if clientErr != nil { - t.Fatal(clientErr) - } - - return clientConn, serverConn -} - -// Check the peer IDs -func testIDs(t *testing.T, clientTpt, serverTpt *Transport, clientConn, serverConn sec.SecureConn) { - if clientConn.LocalPeer() != clientTpt.LocalPeer() { - t.Fatal("Client Local Peer ID mismatch.") - } - - if clientConn.RemotePeer() != serverTpt.LocalPeer() { - t.Fatal("Client Remote Peer ID mismatch.") - } - - if clientConn.LocalPeer() != serverConn.RemotePeer() { - t.Fatal("Server Local Peer ID mismatch.") - } -} - -// Check the keys -func testKeys(t *testing.T, clientTpt, serverTpt *Transport, clientConn, serverConn sec.SecureConn) { - sk := serverConn.LocalPrivateKey() - pk := sk.GetPublic() - - if !sk.Equals(serverTpt.LocalPrivateKey()) { - t.Error("Private key Mismatch.") - } - - if !pk.Equals(clientConn.RemotePublicKey()) { - t.Error("Public key mismatch.") - } -} - -// Check sending and receiving messages -func testReadWrite(t *testing.T, clientConn, serverConn sec.SecureConn) { - before := []byte("hello world") - _, err := clientConn.Write(before) - if err != nil { - t.Fatal(err) - } - - after := make([]byte, len(before)) - _, err = io.ReadFull(serverConn, after) - if err != nil { - t.Fatal(err) - } - - if !bytes.Equal(before, after) { - t.Errorf("Message mismatch. %v != %v", before, after) - } -} - -// Setup a new session with a pair of locally connected sockets -func testConnection(t *testing.T, clientTpt, serverTpt *Transport) { - clientConn, serverConn := connect(t, clientTpt, serverTpt) - - testIDs(t, clientTpt, serverTpt, clientConn, serverConn) - testKeys(t, clientTpt, serverTpt, clientConn, serverConn) - testReadWrite(t, clientConn, serverConn) - - clientConn.Close() - serverConn.Close() -} diff --git a/core/sec/insecure/pb/Makefile b/core/sec/insecure/pb/Makefile deleted file mode 100644 index 4fb825a4bb..0000000000 --- a/core/sec/insecure/pb/Makefile +++ /dev/null @@ -1,11 +0,0 @@ -PB = $(wildcard *.proto) -GO = $(PB:.proto=.pb.go) - -all: $(GO) - -%.pb.go: %.proto - protoc --proto_path=$(GOPATH)/src:../../../crypto/pb:. --gogofaster_out=. $< - -clean: - rm -f *.pb.go - rm -f *.go diff --git a/core/sec/insecure/pb/plaintext.pb.go b/core/sec/insecure/pb/plaintext.pb.go deleted file mode 100644 index 68b12f0c82..0000000000 --- a/core/sec/insecure/pb/plaintext.pb.go +++ /dev/null @@ -1,404 +0,0 @@ -// Code generated by protoc-gen-gogo. DO NOT EDIT. -// source: plaintext.proto - -package plaintext_pb - -import ( - fmt "fmt" - proto "github.com/gogo/protobuf/proto" - pb "github.com/libp2p/go-libp2p-core/crypto/pb" - io "io" - math "math" -) - -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package - -type Exchange struct { - Id []byte `protobuf:"bytes,1,opt,name=id" json:"id"` - Pubkey *pb.PublicKey `protobuf:"bytes,2,opt,name=pubkey" json:"pubkey,omitempty"` -} - -func (m *Exchange) Reset() { *m = Exchange{} } -func (m *Exchange) String() string { return proto.CompactTextString(m) } -func (*Exchange) ProtoMessage() {} -func (*Exchange) Descriptor() ([]byte, []int) { - return fileDescriptor_aba144f73931b711, []int{0} -} -func (m *Exchange) XXX_Unmarshal(b []byte) error { - return m.Unmarshal(b) -} -func (m *Exchange) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - if deterministic { - return xxx_messageInfo_Exchange.Marshal(b, m, deterministic) - } else { - b = b[:cap(b)] - n, err := m.MarshalTo(b) - if err != nil { - return nil, err - } - return b[:n], nil - } -} -func (m *Exchange) XXX_Merge(src proto.Message) { - xxx_messageInfo_Exchange.Merge(m, src) -} -func (m *Exchange) XXX_Size() int { - return m.Size() -} -func (m *Exchange) XXX_DiscardUnknown() { - xxx_messageInfo_Exchange.DiscardUnknown(m) -} - -var xxx_messageInfo_Exchange proto.InternalMessageInfo - -func (m *Exchange) GetId() []byte { - if m != nil { - return m.Id - } - return nil -} - -func (m *Exchange) GetPubkey() *pb.PublicKey { - if m != nil { - return m.Pubkey - } - return nil -} - -func init() { - proto.RegisterType((*Exchange)(nil), "plaintext.pb.Exchange") -} - -func init() { proto.RegisterFile("plaintext.proto", fileDescriptor_aba144f73931b711) } - -var fileDescriptor_aba144f73931b711 = []byte{ - // 187 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x2f, 0xc8, 0x49, 0xcc, - 0xcc, 0x2b, 0x49, 0xad, 0x28, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x41, 0x12, 0x48, - 0x92, 0x32, 0x4f, 0xcf, 0x2c, 0xc9, 0x28, 0x4d, 0xd2, 0x4b, 0xce, 0xcf, 0xd5, 0xcf, 0xc9, 0x4c, - 0x2a, 0x30, 0x2a, 0xd0, 0x4f, 0xcf, 0xd7, 0x85, 0xb0, 0x74, 0x93, 0xf3, 0x8b, 0x52, 0xf5, 0x93, - 0x8b, 0x2a, 0x0b, 0x4a, 0xf2, 0xf5, 0x0b, 0x92, 0xa0, 0x2c, 0x88, 0x31, 0x4a, 0x7e, 0x5c, 0x1c, - 0xae, 0x15, 0xc9, 0x19, 0x89, 0x79, 0xe9, 0xa9, 0x42, 0x22, 0x5c, 0x4c, 0x99, 0x29, 0x12, 0x8c, - 0x0a, 0x8c, 0x1a, 0x3c, 0x4e, 0x2c, 0x27, 0xee, 0xc9, 0x33, 0x04, 0x31, 0x65, 0xa6, 0x08, 0xe9, - 0x70, 0xb1, 0x15, 0x94, 0x26, 0x65, 0xa7, 0x56, 0x4a, 0x30, 0x29, 0x30, 0x6a, 0x70, 0x1b, 0x89, - 0xe8, 0xc1, 0x0c, 0x48, 0xd2, 0x0b, 0x28, 0x4d, 0xca, 0xc9, 0x4c, 0xf6, 0x4e, 0xad, 0x0c, 0x82, - 0xaa, 0x71, 0x92, 0x38, 0xf1, 0x48, 0x8e, 0xf1, 0xc2, 0x23, 0x39, 0xc6, 0x07, 0x8f, 0xe4, 0x18, - 0x27, 0x3c, 0x96, 0x63, 0xb8, 0xf0, 0x58, 0x8e, 0xe1, 0xc6, 0x63, 0x39, 0x06, 0x40, 0x00, 0x00, - 0x00, 0xff, 0xff, 0x40, 0xde, 0x90, 0x0b, 0xc2, 0x00, 0x00, 0x00, -} - -func (m *Exchange) Marshal() (dAtA []byte, err error) { - size := m.Size() - dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) - if err != nil { - return nil, err - } - return dAtA[:n], nil -} - -func (m *Exchange) MarshalTo(dAtA []byte) (int, error) { - var i int - _ = i - var l int - _ = l - if m.Id != nil { - dAtA[i] = 0xa - i++ - i = encodeVarintPlaintext(dAtA, i, uint64(len(m.Id))) - i += copy(dAtA[i:], m.Id) - } - if m.Pubkey != nil { - dAtA[i] = 0x12 - i++ - i = encodeVarintPlaintext(dAtA, i, uint64(m.Pubkey.Size())) - n1, err := m.Pubkey.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err - } - i += n1 - } - return i, nil -} - -func encodeVarintPlaintext(dAtA []byte, offset int, v uint64) int { - for v >= 1<<7 { - dAtA[offset] = uint8(v&0x7f | 0x80) - v >>= 7 - offset++ - } - dAtA[offset] = uint8(v) - return offset + 1 -} -func (m *Exchange) Size() (n int) { - if m == nil { - return 0 - } - var l int - _ = l - if m.Id != nil { - l = len(m.Id) - n += 1 + l + sovPlaintext(uint64(l)) - } - if m.Pubkey != nil { - l = m.Pubkey.Size() - n += 1 + l + sovPlaintext(uint64(l)) - } - return n -} - -func sovPlaintext(x uint64) (n int) { - for { - n++ - x >>= 7 - if x == 0 { - break - } - } - return n -} -func sozPlaintext(x uint64) (n int) { - return sovPlaintext(uint64((x << 1) ^ uint64((int64(x) >> 63)))) -} -func (m *Exchange) Unmarshal(dAtA []byte) error { - l := len(dAtA) - iNdEx := 0 - for iNdEx < l { - preIndex := iNdEx - var wire uint64 - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return ErrIntOverflowPlaintext - } - if iNdEx >= l { - return io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - wire |= uint64(b&0x7F) << shift - if b < 0x80 { - break - } - } - fieldNum := int32(wire >> 3) - wireType := int(wire & 0x7) - if wireType == 4 { - return fmt.Errorf("proto: Exchange: wiretype end group for non-group") - } - if fieldNum <= 0 { - return fmt.Errorf("proto: Exchange: illegal tag %d (wire type %d)", fieldNum, wire) - } - switch fieldNum { - case 1: - if wireType != 2 { - return fmt.Errorf("proto: wrong wireType = %d for field Id", wireType) - } - var byteLen int - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return ErrIntOverflowPlaintext - } - if iNdEx >= l { - return io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - byteLen |= int(b&0x7F) << shift - if b < 0x80 { - break - } - } - if byteLen < 0 { - return ErrInvalidLengthPlaintext - } - postIndex := iNdEx + byteLen - if postIndex < 0 { - return ErrInvalidLengthPlaintext - } - if postIndex > l { - return io.ErrUnexpectedEOF - } - m.Id = append(m.Id[:0], dAtA[iNdEx:postIndex]...) - if m.Id == nil { - m.Id = []byte{} - } - iNdEx = postIndex - case 2: - if wireType != 2 { - return fmt.Errorf("proto: wrong wireType = %d for field Pubkey", wireType) - } - var msglen int - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return ErrIntOverflowPlaintext - } - if iNdEx >= l { - return io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - msglen |= int(b&0x7F) << shift - if b < 0x80 { - break - } - } - if msglen < 0 { - return ErrInvalidLengthPlaintext - } - postIndex := iNdEx + msglen - if postIndex < 0 { - return ErrInvalidLengthPlaintext - } - if postIndex > l { - return io.ErrUnexpectedEOF - } - if m.Pubkey == nil { - m.Pubkey = &pb.PublicKey{} - } - if err := m.Pubkey.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { - return err - } - iNdEx = postIndex - default: - iNdEx = preIndex - skippy, err := skipPlaintext(dAtA[iNdEx:]) - if err != nil { - return err - } - if skippy < 0 { - return ErrInvalidLengthPlaintext - } - if (iNdEx + skippy) < 0 { - return ErrInvalidLengthPlaintext - } - if (iNdEx + skippy) > l { - return io.ErrUnexpectedEOF - } - iNdEx += skippy - } - } - - if iNdEx > l { - return io.ErrUnexpectedEOF - } - return nil -} -func skipPlaintext(dAtA []byte) (n int, err error) { - l := len(dAtA) - iNdEx := 0 - for iNdEx < l { - var wire uint64 - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowPlaintext - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - wire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - wireType := int(wire & 0x7) - switch wireType { - case 0: - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowPlaintext - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - iNdEx++ - if dAtA[iNdEx-1] < 0x80 { - break - } - } - return iNdEx, nil - case 1: - iNdEx += 8 - return iNdEx, nil - case 2: - var length int - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowPlaintext - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - length |= (int(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - if length < 0 { - return 0, ErrInvalidLengthPlaintext - } - iNdEx += length - if iNdEx < 0 { - return 0, ErrInvalidLengthPlaintext - } - return iNdEx, nil - case 3: - for { - var innerWire uint64 - var start int = iNdEx - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowPlaintext - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - innerWire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - innerWireType := int(innerWire & 0x7) - if innerWireType == 4 { - break - } - next, err := skipPlaintext(dAtA[start:]) - if err != nil { - return 0, err - } - iNdEx = start + next - if iNdEx < 0 { - return 0, ErrInvalidLengthPlaintext - } - } - return iNdEx, nil - case 4: - return iNdEx, nil - case 5: - iNdEx += 4 - return iNdEx, nil - default: - return 0, fmt.Errorf("proto: illegal wireType %d", wireType) - } - } - panic("unreachable") -} - -var ( - ErrInvalidLengthPlaintext = fmt.Errorf("proto: negative length found during unmarshaling") - ErrIntOverflowPlaintext = fmt.Errorf("proto: integer overflow") -) diff --git a/core/sec/insecure/pb/plaintext.proto b/core/sec/insecure/pb/plaintext.proto deleted file mode 100644 index 1e299df585..0000000000 --- a/core/sec/insecure/pb/plaintext.proto +++ /dev/null @@ -1,10 +0,0 @@ -syntax = "proto2"; - -package plaintext.pb; - -import "github.com/libp2p/go-libp2p-core/crypto/pb/crypto.proto"; - -message Exchange { - optional bytes id = 1; - optional crypto.pb.PublicKey pubkey = 2; -} From 554a8e7bed5de55c4346a736f9ea1ed889d72802 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 2 Jul 2019 12:25:47 -0700 Subject: [PATCH 1370/3965] mdns: always use interface addresses We don't want to use the transformed/munged host addresses for local announcements. Ideally, we'd take a more scientific approach to this (i.e., "host, please give me addresses relative to X") but we can't do that yet. --- go.sum | 1 + p2p/discovery/mdns.go | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/go.sum b/go.sum index cf468647ec..b417df666f 100644 --- a/go.sum +++ b/go.sum @@ -120,6 +120,7 @@ github.com/libp2p/go-libp2p-circuit v0.1.0 h1:eniLL3Y9aq/sryfyV1IAHj5rlvuyj3b7iz github.com/libp2p/go-libp2p-circuit v0.1.0/go.mod h1:Ahq4cY3V9VJcHcn1SBXjr78AbFkZeIRmfunbA7pmFh8= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= +github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyLCcvo1w= github.com/libp2p/go-libp2p-core v0.0.6/go.mod h1:0d9xmaYAVY5qmbp/fcgxHT3ZJsLjYeYPMJAUKpaCHrE= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= diff --git a/p2p/discovery/mdns.go b/p2p/discovery/mdns.go index 3476e6ebf7..2da6618910 100644 --- a/p2p/discovery/mdns.go +++ b/p2p/discovery/mdns.go @@ -49,7 +49,11 @@ type mdnsService struct { func getDialableListenAddrs(ph host.Host) ([]*net.TCPAddr, error) { var out []*net.TCPAddr - for _, addr := range ph.Addrs() { + addrs, err := ph.Network().InterfaceListenAddresses() + if err != nil { + return nil, err + } + for _, addr := range addrs { na, err := manet.ToNetAddr(addr) if err != nil { continue From 21142a6fb68885ac0afb70f3689178850d4762bd Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 17 Jul 2019 17:34:22 -0400 Subject: [PATCH 1371/3965] Add defualt case with meaningful panic --- core/crypto/key.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/crypto/key.go b/core/crypto/key.go index b939b8323e..2706a7a3e7 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -191,6 +191,8 @@ func KeyStretcher(cipherType string, hashType string, secret []byte) (StretchedK case "AES-256": ivSize = 16 cipherKeySize = 32 + default: + panic("Unrecognized cipher, programmer error?") } hmacKeySize := 20 From a7ce9cc59346c811dce274e4b08b669cf3ca0778 Mon Sep 17 00:00:00 2001 From: Christian Muehlhaeuser Date: Mon, 22 Jul 2019 02:38:39 +0200 Subject: [PATCH 1372/3965] Removed ineffectual assignments We don't use `cs` here, drop it. --- p2p/net/mock/mock_peernet.go | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/p2p/net/mock/mock_peernet.go b/p2p/net/mock/mock_peernet.go index b8d02fe275..bcf0644656 100644 --- a/p2p/net/mock/mock_peernet.go +++ b/p2p/net/mock/mock_peernet.go @@ -183,17 +183,15 @@ func (pn *peernet) addConn(c *conn) { pn.Lock() defer pn.Unlock() - cs, found := pn.connsByPeer[c.RemotePeer()] + _, found := pn.connsByPeer[c.RemotePeer()] if !found { - cs = map[*conn]struct{}{} - pn.connsByPeer[c.RemotePeer()] = cs + pn.connsByPeer[c.RemotePeer()] = map[*conn]struct{}{} } pn.connsByPeer[c.RemotePeer()][c] = struct{}{} - cs, found = pn.connsByLink[c.link] + _, found = pn.connsByLink[c.link] if !found { - cs = map[*conn]struct{}{} - pn.connsByLink[c.link] = cs + pn.connsByLink[c.link] = map[*conn]struct{}{} } pn.connsByLink[c.link][c] = struct{}{} } From 40cb489097972a1e4631684f901cc51667ff0437 Mon Sep 17 00:00:00 2001 From: Christian Muehlhaeuser Date: Mon, 22 Jul 2019 14:44:24 +0200 Subject: [PATCH 1373/3965] Fixed typos (#680) Just a few nit-picky typo fixes. --- defaults.go | 2 +- p2p/host/basic/basic_host_test.go | 2 +- p2p/net/mock/interface.go | 2 +- p2p/net/mock/mock_net.go | 2 +- p2p/protocol/identify/id.go | 2 +- p2p/protocol/identify/id_test.go | 2 +- p2p/test/backpressure/backpressure_test.go | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/defaults.go b/defaults.go index 18de400d95..cdcd8288f9 100644 --- a/defaults.go +++ b/defaults.go @@ -125,7 +125,7 @@ var Defaults Option = func(cfg *Config) error { } // FallbackDefaults applies default options to the libp2p node if and only if no -// other relevent options have been applied. will be appended to the options +// other relevant options have been applied. will be appended to the options // passed into New. var FallbackDefaults Option = func(cfg *Config) error { for _, def := range defaults { diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index acc1db85f6..5e980f8967 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -363,7 +363,7 @@ func TestProtoDowngrade(t *testing.T) { h1.Network().ConnsToPeer(h2.ID())[0].Close() - time.Sleep(time.Millisecond * 50) // allow notifications to propogate + time.Sleep(time.Millisecond * 50) // allow notifications to propagate h1.RemoveStreamHandler("/testing/1.0.0") h1.SetStreamHandler("/testing", func(s network.Stream) { connectedOn <- s.Protocol() diff --git a/p2p/net/mock/interface.go b/p2p/net/mock/interface.go index 57861bf1cd..16dcfcfe56 100644 --- a/p2p/net/mock/interface.go +++ b/p2p/net/mock/interface.go @@ -51,7 +51,7 @@ type Mocknet interface { UnlinkNets(network.Network, network.Network) error // LinkDefaults are the default options that govern links - // if they do not have thier own option set. + // if they do not have their own option set. SetLinkDefaults(LinkOptions) LinkDefaults() LinkOptions diff --git a/p2p/net/mock/mock_net.go b/p2p/net/mock/mock_net.go index 832d7c2556..5b75c399f4 100644 --- a/p2p/net/mock/mock_net.go +++ b/p2p/net/mock/mock_net.go @@ -286,7 +286,7 @@ func (mn *mocknet) UnlinkNets(n1, n2 network.Network) error { return mn.UnlinkPeers(n1.LocalPeer(), n2.LocalPeer()) } -// get from the links map. and lazily contruct. +// get from the links map. and lazily construct. func (mn *mocknet) linksMapGet(p1, p2 peer.ID) map[*link]struct{} { l1, found := mn.links[p1] diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index f44716b25f..1cfffa57b1 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -475,7 +475,7 @@ func HasConsistentTransport(a ma.Multiaddr, green []ma.Multiaddr) bool { // IdentifyWait returns a channel which will be closed once // "ProtocolIdentify" (handshake3) finishes on given conn. // This happens async so the connection can start to be used -// even if handshake3 knowledge is not necesary. +// even if handshake3 knowledge is not necessary. // Users **MUST** call IdentifyWait _after_ IdentifyConn func (ids *IDService) IdentifyWait(c network.Conn) <-chan struct{} { ids.currmu.Lock() diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index 4bbf91b959..dedf9a094e 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -160,7 +160,7 @@ func testHasPublicKey(t *testing.T, h host.Host, p peer.ID, shouldBe ic.PubKey) } // TestIDServiceWait gives the ID service 1s to finish after dialing -// this is becasue it used to be concurrent. Now, Dial wait till the +// this is because it used to be concurrent. Now, Dial wait till the // id service is done. func TestIDService(t *testing.T) { oldTTL := peerstore.RecentlyConnectedAddrTTL diff --git a/p2p/test/backpressure/backpressure_test.go b/p2p/test/backpressure/backpressure_test.go index 92c8f20bcf..e102f4d1ae 100644 --- a/p2p/test/backpressure/backpressure_test.go +++ b/p2p/test/backpressure/backpressure_test.go @@ -330,7 +330,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { } roundsTime := time.Since(roundsStart) - // now read continously, while we measure stats. + // now read continuously, while we measure stats. stop := make(chan struct{}) contStart := time.Now() From 0603b300569542516231da533dc3b89c64edd880 Mon Sep 17 00:00:00 2001 From: Christian Muehlhaeuser Date: Mon, 22 Jul 2019 14:44:47 +0200 Subject: [PATCH 1374/3965] Make use of time.Until & time.Since (#681) No need to manually subtract timestamps. --- p2p/host/basic/basic_host.go | 2 +- p2p/net/mock/mock_stream.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index ea938c5486..19d45542a7 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -258,7 +258,7 @@ func (h *BasicHost) newStreamHandler(s network.Stream) { } lzc, protoID, handle, err := h.Mux().NegotiateLazy(s) - took := time.Now().Sub(before) + took := time.Since(before) if err != nil { if err == io.EOF { logf := log.Debugf diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index 7919054884..2bf8929686 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -191,7 +191,7 @@ func (s *stream) transport() { default: } } - delay := o.arrivalTime.Sub(time.Now()) + delay := time.Until(o.arrivalTime) if delay >= 0 { timer.Reset(delay) } else { From 038cff7946be0835c1d49f88b4ce054c2ff8c77b Mon Sep 17 00:00:00 2001 From: Christian Muehlhaeuser Date: Mon, 22 Jul 2019 14:45:37 +0200 Subject: [PATCH 1375/3965] Simplify increments (#684) --- p2p/test/backpressure/backpressure_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/test/backpressure/backpressure_test.go b/p2p/test/backpressure/backpressure_test.go index e102f4d1ae..72525b53f8 100644 --- a/p2p/test/backpressure/backpressure_test.go +++ b/p2p/test/backpressure/backpressure_test.go @@ -218,7 +218,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { select { case n := <-senderWrote: writes++ - bytes = bytes + n + bytes += n default: log.Debugf("stats: sender wrote %d bytes, %d writes", bytes, writes) return bytes, writes @@ -326,7 +326,7 @@ func TestStBackpressureStreamWrite(t *testing.T) { // drain it all, wait again receive(s, b) - roundsTotal = roundsTotal + b + roundsTotal += b } roundsTime := time.Since(roundsStart) From 169f2d4139f8e72a6c4497583e62209e35fef35d Mon Sep 17 00:00:00 2001 From: Christian Muehlhaeuser Date: Mon, 22 Jul 2019 14:46:15 +0200 Subject: [PATCH 1376/3965] Avoid unnecessary conversions (#685) No need to convert these types. --- p2p/host/basic/basic_host.go | 2 +- p2p/protocol/identify/id.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 19d45542a7..2d965bdae1 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -213,7 +213,7 @@ func New(net network.Network, opts ...interface{}) *BasicHost { hostopts.NATManager = NewNATManager } case AddrsFactory: - hostopts.AddrsFactory = AddrsFactory(o) + hostopts.AddrsFactory = o case connmgr.ConnManager: hostopts.ConnManager = o case *madns.Resolver: diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 1cfffa57b1..1fa2bcf1f6 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -268,7 +268,7 @@ func (ids *IDService) populateMessage(mes *pb.Identify, c network.Conn) { protos := ids.Host.Mux().Protocols() mes.Protocols = make([]string, len(protos)) for i, p := range protos { - mes.Protocols[i] = string(p) + mes.Protocols[i] = p } // observed address so other side is informed of their From c73d2b9c8ee65326fe6c1b76a05a6b6cdcf223b9 Mon Sep 17 00:00:00 2001 From: Christian Muehlhaeuser Date: Mon, 22 Jul 2019 14:46:45 +0200 Subject: [PATCH 1377/3965] Added missing error check in test (#683) --- p2p/host/relay/autorelay_test.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index b6a6930c02..25653d8c1a 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -169,6 +169,9 @@ func TestAutoRelay(t *testing.T) { t.Fatal(err) } h4, err := libp2p.New(ctx, libp2p.EnableRelay()) + if err != nil { + t.Fatal(err) + } // verify that we don't advertise relay addrs initially for _, addr := range h3.Addrs() { From 1df1ff755d4e3b460f973eb2027bb07603f11dd8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 22 Jul 2019 15:10:43 -0700 Subject: [PATCH 1378/3965] avoid duplicate randomly generated keys/peer-ids This implements #4 from #43. fixes #43 --- core/test/crypto.go | 10 +++------- core/test/peer.go | 7 +------ 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/core/test/crypto.go b/core/test/crypto.go index 931eae9b9f..8733cf9242 100644 --- a/core/test/crypto.go +++ b/core/test/crypto.go @@ -3,20 +3,16 @@ package test import ( "math/rand" "sync/atomic" - "time" ci "github.com/libp2p/go-libp2p-core/crypto" ) -var generatedPairs int64 = 0 +var globalSeed int64 func RandTestKeyPair(typ, bits int) (ci.PrivKey, ci.PubKey, error) { - seed := time.Now().UnixNano() - // workaround for low time resolution - seed += atomic.AddInt64(&generatedPairs, 1) << 32 - - return SeededTestKeyPair(typ, bits, time.Now().UnixNano()) + seed := atomic.AddInt64(&globalSeed, 1) + return SeededTestKeyPair(typ, bits, seed) } func SeededTestKeyPair(typ, bits int, seed int64) (ci.PrivKey, ci.PubKey, error) { diff --git a/core/test/peer.go b/core/test/peer.go index 9d78989509..b95639fe23 100644 --- a/core/test/peer.go +++ b/core/test/peer.go @@ -1,10 +1,8 @@ package test import ( - "io" "math/rand" "testing" - "time" "github.com/libp2p/go-libp2p-core/peer" @@ -12,11 +10,8 @@ import ( ) func RandPeerID() (peer.ID, error) { - r := rand.New(rand.NewSource(time.Now().UnixNano())) buf := make([]byte, 16) - if _, err := io.ReadFull(r, buf); err != nil { - return "", err - } + rand.Read(buf) h, _ := mh.Sum(buf, mh.SHA2_256, -1) return peer.ID(h), nil } From 4d3147400f2fe10a9e7d0d16b7837e1afcdf079c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 24 Jul 2019 16:29:51 -0700 Subject: [PATCH 1379/3965] fix multiple TTL bugs The first fix independently extends the address expiration time and the address TTL: By example: * We have an address with a TTL of 4s that will expire in 1s. * We update it with a TTL of 3s. Before this change: * We end up with an address with a TTL of 3s that will expire in 3s. After this change: * We end up with an address with a TTL of 4s that will expire in 3s. --- The second fix prevents the in-memory addressbook from announcing existing addresses every time their TTLs get updated. --- The third fix correctly updates TTLs for existing addresses in the on-disk addressbook. This fixes https://github.com/libp2p/go-libp2p-identify/issues/2 --- p2p/host/peerstore/pstoreds/addr_book.go | 18 +++++++++++---- p2p/host/peerstore/pstoremem/addr_book.go | 15 +++++++++--- p2p/host/peerstore/test/addr_book_suite.go | 27 +++++++++++++++++++++- 3 files changed, 52 insertions(+), 8 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index bba648e845..2f75265c5b 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -347,11 +347,21 @@ Outer: for _, have := range pr.Addrs { if incoming.Equal(have.Addr) { existed[i] = true - if mode == ttlExtend && have.Expiry > newExp { - // if we're only extending TTLs but the addr already has a longer one, we skip it. - continue Outer + switch mode { + case ttlOverride: + have.Ttl = int64(ttl) + have.Expiry = newExp + case ttlExtend: + if int64(ttl) > have.Ttl { + have.Ttl = int64(ttl) + } + if newExp > have.Expiry { + have.Expiry = newExp + } + default: + panic("BUG: unimplemented ttl mode") } - have.Expiry = newExp + // we found the address, and addresses cannot be duplicate, // so let's move on to the next. continue Outer diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 19b60f9b8a..90ffb175bf 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -132,7 +132,7 @@ func (mab *memoryAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Durati // AddAddrs gives memoryAddrBook addresses to use, with a given ttl // (time-to-live), after which the address is no longer valid. -// If the manager has a longer TTL, the operation is a no-op for that address +// This function never reduces the TTL or expiration of an address. func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { // if ttl is zero, exit. nothing to do. if ttl <= 0 { @@ -156,10 +156,19 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du } asBytes := addr.Bytes() a, found := amap[string(asBytes)] // won't allocate. - if !found || exp.After(a.Expires) { + if !found { + // not found, save and announce it. amap[string(asBytes)] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} - mab.subManager.BroadcastAddr(p, addr) + } else { + // Update expiration/TTL independently. + // We never want to reduce either. + if ttl > a.TTL { + a.TTL = ttl + } + if exp.After(a.Expires) { + a.Expires = exp + } } } } diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index 72a899cd78..ea6078cf61 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -86,9 +86,13 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { // after the initial TTL has expired, check that only the third address is present. time.Sleep(1200 * time.Millisecond) AssertAddressesEqual(t, addrs[2:], ab.Addrs(id)) + + // make sure we actually set the TTL + ab.UpdateAddrs(id, time.Hour, 0) + AssertAddressesEqual(t, nil, ab.Addrs(id)) }) - t.Run("adding an existing address with an earlier expiration is noop", func(t *testing.T) { + t.Run("adding an existing address with an earlier expiration never reduces the expiration", func(t *testing.T) { id := GeneratePeerIDs(1)[0] addrs := GenerateAddrs(3) @@ -102,6 +106,27 @@ func testAddAddress(ab pstore.AddrBook) func(*testing.T) { time.Sleep(2100 * time.Millisecond) AssertAddressesEqual(t, addrs, ab.Addrs(id)) }) + + t.Run("adding an existing address with an earlier expiration never reduces the TTL", func(t *testing.T) { + id := GeneratePeerIDs(1)[0] + addrs := GenerateAddrs(1) + + ab.AddAddrs(id, addrs, 4*time.Second) + // 4 seconds left + time.Sleep(3 * time.Second) + // 1 second left + ab.AddAddrs(id, addrs, 3*time.Second) + // 3 seconds left + time.Sleep(2) + // 1 seconds left. + + // We still have the address. + AssertAddressesEqual(t, addrs, ab.Addrs(id)) + + // The TTL wasn't reduced + ab.UpdateAddrs(id, 4*time.Second, 0) + AssertAddressesEqual(t, nil, ab.Addrs(id)) + }) } } From 34c4b5bde06251469255d41d773f05b9d49c4138 Mon Sep 17 00:00:00 2001 From: Jakub Sztandera Date: Thu, 25 Jul 2019 16:51:06 +0200 Subject: [PATCH 1380/3965] Add compat check (#40) License: MIT Signed-off-by: Jakub Sztandera --- core/compat-check | 18 ++++++++++++++++++ core/tools.go | 7 +++++++ 2 files changed, 25 insertions(+) create mode 100755 core/compat-check create mode 100644 core/tools.go diff --git a/core/compat-check b/core/compat-check new file mode 100755 index 0000000000..51899f5f37 --- /dev/null +++ b/core/compat-check @@ -0,0 +1,18 @@ +#!/usr/bin/env bash + +check-compat() ( + set -e + go run github.com/smola/gocompat/cmd/gocompat compare --git-refs="remotes/origin/master..$(git rev-parse HEAD)" --go1compat ./... +) + +export GO111MODULE=on +if [[ "$TRAVIS" == "true" ]]; then + git fetch origin master:remotes/origin/master > /dev/null 2>&1 +fi + +if ! check-compat; then + echo ">> API Compatibility broken!!" + exit 1 +fi + + diff --git a/core/tools.go b/core/tools.go new file mode 100644 index 0000000000..c9a7140e05 --- /dev/null +++ b/core/tools.go @@ -0,0 +1,7 @@ +// +build tools + +package core + +import ( + _ "github.com/smola/gocompat" +) From cea811c943b7e6370322d1cf6fa4e0edf5b52f16 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 25 Jul 2019 12:20:15 -0400 Subject: [PATCH 1381/3965] use new insecure transport constructor --- p2p/transport/tcp/tcp_test.go | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/p2p/transport/tcp/tcp_test.go b/p2p/transport/tcp/tcp_test.go index cfceef50f3..a4a85bdf04 100644 --- a/p2p/transport/tcp/tcp_test.go +++ b/p2p/transport/tcp/tcp_test.go @@ -1,6 +1,8 @@ package tcp import ( + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" "testing" "github.com/libp2p/go-libp2p-core/sec/insecure" @@ -14,17 +16,20 @@ import ( func TestTcpTransport(t *testing.T) { for i := 0; i < 2; i++ { + ia := makeInsecureTransport(t) + ib := makeInsecureTransport(t) + ta := NewTCPTransport(&tptu.Upgrader{ - Secure: insecure.New("peerA"), + Secure: ia, Muxer: new(mplex.Transport), }) tb := NewTCPTransport(&tptu.Upgrader{ - Secure: insecure.New("peerB"), + Secure: ib, Muxer: new(mplex.Transport), }) zero := "/ip4/127.0.0.1/tcp/0" - ttransport.SubtestTransport(t, ta, tb, zero, "peerA") + ttransport.SubtestTransport(t, ta, tb, zero, ia.LocalPeer()) envReuseportVal = false } @@ -39,7 +44,7 @@ func TestTcpTransportCantListenUtp(t *testing.T) { } tpt := NewTCPTransport(&tptu.Upgrader{ - Secure: insecure.New("peerB"), + Secure: makeInsecureTransport(t), Muxer: new(mplex.Transport), }) @@ -52,3 +57,15 @@ func TestTcpTransportCantListenUtp(t *testing.T) { } envReuseportVal = true } + +func makeInsecureTransport(t *testing.T) *insecure.Transport { + priv, pub, err := crypto.GenerateKeyPair(crypto.Ed25519, 256) + if err != nil { + t.Fatal(err) + } + id, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Fatal(err) + } + return insecure.NewWithIdentity(id, priv) +} From f72759323472fc9775225f2217c1a056a22edcb3 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 25 Jul 2019 15:11:55 -0700 Subject: [PATCH 1382/3965] don't count connections in the grace period against the limit fixes #46 --- p2p/net/connmgr/connmgr.go | 21 +++++-- p2p/net/connmgr/connmgr_test.go | 99 ++++++++++++++++++++++++++++++--- 2 files changed, 106 insertions(+), 14 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 0ee8901742..e430a80cfc 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -221,6 +221,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { npeers := cm.segments.countPeers() candidates := make([]*peerInfo, 0, npeers) + ncandidates := 0 cm.plk.RLock() for _, s := range cm.segments { s.Lock() @@ -229,12 +230,26 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { // skip over protected peer. continue } + if inf.firstSeen.Add(cm.gracePeriod).After(now) { + // skip peers in the grace period. + continue + } candidates = append(candidates, inf) + ncandidates += len(inf.conns) } s.Unlock() } cm.plk.RUnlock() + if ncandidates < cm.lowWater { + log.Info("open connection count above limit but too many are in the grace period") + // We have too many connections but fewer than lowWater + // connections out of the grace period. + // + // If we trimmed now, we'd kill potentially useful connections. + return nil + } + // Sort peers according to their value. sort.Slice(candidates, func(i, j int) bool { left, right := candidates[i], candidates[j] @@ -246,7 +261,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { return left.value < right.value }) - target := nconns - cm.lowWater + target := ncandidates - cm.lowWater // slightly overallocate because we may have more than one conns per peer selected := make([]network.Conn, 0, target+10) @@ -255,10 +270,6 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { if target <= 0 { break } - // TODO: should we be using firstSeen or the time associated with the connection itself? - if inf.firstSeen.Add(cm.gracePeriod).After(now) { - continue - } // lock this to protect from concurrent modifications from connect/disconnect events s := cm.segments.get(inf.id) diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index edaff7bc6c..ad787864c7 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -303,6 +303,63 @@ func TestDisconnected(t *testing.T) { } } +func TestGracePeriod(t *testing.T) { + if detectrace.WithRace() { + t.Skip("race detector is unhappy with this test") + } + + SilencePeriod = 0 + cm := NewConnManager(10, 20, 100*time.Millisecond) + SilencePeriod = 10 * time.Second + + not := cm.Notifee() + + var conns []network.Conn + + // Add a connection and wait the grace period. + { + rc := randConn(t, not.Disconnected) + conns = append(conns, rc) + not.Connected(nil, rc) + + time.Sleep(200 * time.Millisecond) + + if rc.(*tconn).closed { + t.Fatal("expected conn to remain open") + } + } + + // quickly add 30 connections (sending us above the high watermark) + for i := 0; i < 30; i++ { + rc := randConn(t, not.Disconnected) + conns = append(conns, rc) + not.Connected(nil, rc) + } + + cm.TrimOpenConns(context.Background()) + + for _, c := range conns { + if c.(*tconn).closed { + t.Fatal("expected no conns to be closed") + } + } + + time.Sleep(200 * time.Millisecond) + + cm.TrimOpenConns(context.Background()) + + closed := 0 + for _, c := range conns { + if c.(*tconn).closed { + closed++ + } + } + + if closed != 21 { + t.Fatal("expected to have closed 21 connections") + } +} + // see https://github.com/libp2p/go-libp2p-connmgr/issues/23 func TestQuickBurstRespectsSilencePeriod(t *testing.T) { if detectrace.WithRace() { @@ -350,13 +407,17 @@ func TestPeerProtectionSingleTag(t *testing.T) { not := cm.Notifee() - // produce 20 connections with unique peers. var conns []network.Conn - for i := 0; i < 20; i++ { + addConn := func(value int) { rc := randConn(t, not.Disconnected) conns = append(conns, rc) not.Connected(nil, rc) - cm.TagPeer(rc.RemotePeer(), "test", 20) + cm.TagPeer(rc.RemotePeer(), "test", value) + } + + // produce 20 connections with unique peers. + for i := 0; i < 20; i++ { + addConn(20) } // protect the first 5 peers. @@ -368,8 +429,21 @@ func TestPeerProtectionSingleTag(t *testing.T) { cm.TagPeer(c.RemotePeer(), "test", -100) } - // add one more connection, sending the connection manager overboard. - not.Connected(nil, randConn(t, not.Disconnected)) + // add 1 more conn, this shouldn't send us over the limit as protected conns don't count + addConn(20) + + cm.TrimOpenConns(context.Background()) + + for _, c := range conns { + if c.(*tconn).closed { + t.Error("connection was closed by connection manager") + } + } + + // add 5 more connection, sending the connection manager overboard. + for i := 0; i < 5; i++ { + addConn(20) + } cm.TrimOpenConns(context.Background()) @@ -379,15 +453,22 @@ func TestPeerProtectionSingleTag(t *testing.T) { } } + closed := 0 + for _, c := range conns { + if c.(*tconn).closed { + closed++ + } + } + if closed != 2 { + t.Errorf("expected 2 connection to be closed, found %d", closed) + } + // unprotect the first peer. cm.Unprotect(protected[0].RemotePeer(), "global") // add 2 more connections, sending the connection manager overboard again. for i := 0; i < 2; i++ { - rc := randConn(t, not.Disconnected) - conns = append(conns, rc) - not.Connected(nil, rc) - cm.TagPeer(rc.RemotePeer(), "test", 20) + addConn(20) } cm.TrimOpenConns(context.Background()) From 0ae766d031a9dd8d518728205096a04b253fd9e4 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 26 Jul 2019 09:35:08 -0700 Subject: [PATCH 1383/3965] chore: port issue templates from go-ipfs --- .github/ISSUE_TEMPLATE/bug-report.md | 20 ++++++++++++++++++++ .github/ISSUE_TEMPLATE/doc.md | 13 +++++++++++++ .github/ISSUE_TEMPLATE/enhancement.md | 11 +++++++++++ .github/ISSUE_TEMPLATE/feature.md | 15 +++++++++++++++ .github/ISSUE_TEMPLATE/question.md | 9 +++++++++ 5 files changed, 68 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/bug-report.md create mode 100644 .github/ISSUE_TEMPLATE/doc.md create mode 100644 .github/ISSUE_TEMPLATE/enhancement.md create mode 100644 .github/ISSUE_TEMPLATE/feature.md create mode 100644 .github/ISSUE_TEMPLATE/question.md diff --git a/.github/ISSUE_TEMPLATE/bug-report.md b/.github/ISSUE_TEMPLATE/bug-report.md new file mode 100644 index 0000000000..47facff2c3 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug-report.md @@ -0,0 +1,20 @@ +--- +name: 'Bug Report' +about: 'Report a bug in go-libp2p.' +labels: bug +--- + + + +

+Version Information +
+
+
+
diff --git a/.github/ISSUE_TEMPLATE/doc.md b/.github/ISSUE_TEMPLATE/doc.md new file mode 100644 index 0000000000..5a0d198218 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/doc.md @@ -0,0 +1,13 @@ +--- +name: 'Documentation Issue' +about: 'Report missing/erroneous documentation, propose new documentation, report broken links, etc.' +labels: documentation +--- + +#### Location + + + +#### Description + + diff --git a/.github/ISSUE_TEMPLATE/enhancement.md b/.github/ISSUE_TEMPLATE/enhancement.md new file mode 100644 index 0000000000..72d04f70a5 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/enhancement.md @@ -0,0 +1,11 @@ +--- +name: 'Enhancement' +about: 'Suggest an improvement to an existing go-libp2p feature.' +labels: enhancement +--- + + diff --git a/.github/ISSUE_TEMPLATE/feature.md b/.github/ISSUE_TEMPLATE/feature.md new file mode 100644 index 0000000000..7c3ed9d010 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature.md @@ -0,0 +1,15 @@ +--- +name: 'Feature' +about: 'Suggest a new feature in go-libp2p.' +labels: feature +--- + + diff --git a/.github/ISSUE_TEMPLATE/question.md b/.github/ISSUE_TEMPLATE/question.md new file mode 100644 index 0000000000..97d8ca5faf --- /dev/null +++ b/.github/ISSUE_TEMPLATE/question.md @@ -0,0 +1,9 @@ +--- +name: 'Question/Support' +about: 'Ask a question about go-libp2p or request support.' +labels: question, invalid +--- + +This bug tracker is only for actionable bug reports and feature requests. Please direct any questions to https://discuss.libp2p.io or to our Matrix (#libp2p:matrix.org) or IRC (#libp2p on freenode) channels. + +If you don't get an immediate response, please keep trying. From 250af2033927b8a0c299f793b71c2c3470d5049e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 28 Jul 2019 06:40:50 +0900 Subject: [PATCH 1384/3965] don't use deprecated go-libp2p-crypto.MarshalPublicKey --- p2p/security/tls/crypto.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 917c65a666..b3693b6af7 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -13,7 +13,6 @@ import ( "math/big" "time" - crypto "github.com/libp2p/go-libp2p-crypto" "golang.org/x/sys/cpu" ic "github.com/libp2p/go-libp2p-core/crypto" @@ -157,7 +156,7 @@ func keyToCertificate(sk ic.PrivKey) (*tls.Certificate, error) { return nil, err } - keyBytes, err := crypto.MarshalPublicKey(sk.GetPublic()) + keyBytes, err := ic.MarshalPublicKey(sk.GetPublic()) if err != nil { return nil, err } From 08736ac1f3ea870124ee33dbc6452324c22e5993 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 28 Jul 2019 06:49:08 +0900 Subject: [PATCH 1385/3965] set an ALPN value in the tls.Config --- p2p/security/tls/crypto.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index b3693b6af7..a0a52cea72 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -21,6 +21,7 @@ import ( const certValidityPeriod = 100 * 365 * 24 * time.Hour // ~100 years const certificatePrefix = "libp2p-tls-handshake:" +const alpn string = "libp2p" var extensionID = getPrefixedExtensionID([]int{1, 1}) @@ -50,6 +51,7 @@ func NewIdentity(privKey ic.PrivKey) (*Identity, error) { VerifyPeerCertificate: func(_ [][]byte, _ [][]*x509.Certificate) error { panic("tls config not specialized for peer") }, + NextProtos: []string{alpn}, SessionTicketsDisabled: true, }, }, nil From ac5292eb4b2685639c76a42ef24c868d7540ec39 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 27 Jul 2019 16:45:37 -0700 Subject: [PATCH 1386/3965] pre-compute grace period --- p2p/net/connmgr/connmgr.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index e430a80cfc..804730dea4 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -212,7 +212,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { // disabled return nil } - now := time.Now() + nconns := int(atomic.LoadInt32(&cm.connCount)) if nconns <= cm.lowWater { log.Info("open connection count below limit") @@ -222,6 +222,8 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { npeers := cm.segments.countPeers() candidates := make([]*peerInfo, 0, npeers) ncandidates := 0 + gracePeriodStart := time.Now().Add(-cm.gracePeriod) + cm.plk.RLock() for _, s := range cm.segments { s.Lock() @@ -230,7 +232,7 @@ func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { // skip over protected peer. continue } - if inf.firstSeen.Add(cm.gracePeriod).After(now) { + if inf.firstSeen.After(gracePeriodStart) { // skip peers in the grace period. continue } From 798a70679bb11e071be35b5f8641c6da77ec1bb4 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 30 Jul 2019 15:51:57 -0400 Subject: [PATCH 1387/3965] Return error rather than panic in Emit --- p2p/host/eventbus/basic.go | 5 +++-- p2p/host/eventbus/basic_test.go | 19 +++++++------------ 2 files changed, 10 insertions(+), 14 deletions(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index b81868e86f..0e02bf0769 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -28,11 +28,12 @@ type emitter struct { dropper func(reflect.Type) } -func (e *emitter) Emit(evt interface{}) { +func (e *emitter) Emit(evt interface{}) error { if atomic.LoadInt32(&e.closed) != 0 { - panic("emitter is closed") + return fmt.Errorf("emitter is closed") } e.n.emit(evt) + return nil } func (e *emitter) Close() error { diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index b3a8dcd20d..c2aebe2d2f 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -109,18 +109,13 @@ func TestEmitOnClosed(t *testing.T) { t.Fatal(err) } em.Close() - - defer func() { - r := recover() - if r == nil { - t.Errorf("expected panic") - } - if r.(string) != "emitter is closed" { - t.Error("unexpected message") - } - }() - - em.Emit(EventA{}) + err = em.Emit(EventA{}) + if err == nil { + t.Errorf("expected panic") + } + if err.Error() != "emitter is closed" { + t.Error("unexpected message") + } } func TestClosingRaces(t *testing.T) { From 8dbcc5a9f5ba6c863347920447d67af4157bd9ce Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Tue, 30 Jul 2019 16:46:24 -0400 Subject: [PATCH 1388/3965] bring back plaintext 2.0.0 with new constructor (#42) * bring back plaintext 2.0.0 with new constructor * fix deprecation comment * rm unused context argument * only check remote id validity if we actually have keys * bring back msgio & simultaneous read/write --- core/crypto/key.go | 20 +- core/sec/insecure/insecure.go | 184 ++++++++++-- core/sec/insecure/insecure_test.go | 157 +++++++++++ core/sec/insecure/pb/Makefile | 11 + core/sec/insecure/pb/plaintext.pb.go | 404 +++++++++++++++++++++++++++ core/sec/insecure/pb/plaintext.proto | 10 + 6 files changed, 764 insertions(+), 22 deletions(-) create mode 100644 core/sec/insecure/insecure_test.go create mode 100644 core/sec/insecure/pb/Makefile create mode 100644 core/sec/insecure/pb/plaintext.pb.go create mode 100644 core/sec/insecure/pb/plaintext.proto diff --git a/core/crypto/key.go b/core/crypto/key.go index cb8e77858e..bc0026eb27 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -279,6 +279,12 @@ func UnmarshalPublicKey(data []byte) (PubKey, error) { return nil, err } + return PublicKeyFromProto(pmes) +} + +// PublicKeyFromProto converts an unserialized protobuf PublicKey message +// into its representative object. +func PublicKeyFromProto(pmes *pb.PublicKey) (PubKey, error) { um, ok := PubKeyUnmarshallers[pmes.GetType()] if !ok { return nil, ErrBadKeyType @@ -290,6 +296,17 @@ func UnmarshalPublicKey(data []byte) (PubKey, error) { // MarshalPublicKey converts a public key object into a protobuf serialized // public key func MarshalPublicKey(k PubKey) ([]byte, error) { + pbmes, err := PublicKeyToProto(k) + if err != nil { + return nil, err + } + + return proto.Marshal(pbmes) +} + +// PublicKeyToProto converts a public key object into an unserialized +// protobuf PublicKey message. +func PublicKeyToProto(k PubKey) (*pb.PublicKey, error) { pbmes := new(pb.PublicKey) pbmes.Type = k.Type() data, err := k.Raw() @@ -297,8 +314,7 @@ func MarshalPublicKey(k PubKey) ([]byte, error) { return nil, err } pbmes.Data = data - - return proto.Marshal(pbmes) + return pbmes, nil } // UnmarshalPrivateKey converts a protobuf serialized private key into its diff --git a/core/sec/insecure/insecure.go b/core/sec/insecure/insecure.go index 801569cb0b..e3670c222a 100644 --- a/core/sec/insecure/insecure.go +++ b/core/sec/insecure/insecure.go @@ -5,64 +5,207 @@ package insecure import ( "context" + "fmt" "net" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" + "github.com/libp2p/go-msgio" ci "github.com/libp2p/go-libp2p-core/crypto" + pb "github.com/libp2p/go-libp2p-core/sec/insecure/pb" ) // ID is the multistream-select protocol ID that should be used when identifying // this security transport. -const ID = "/plaintext/1.0.0" +const ID = "/plaintext/2.0.0" // Transport is a no-op stream security transport. It provides no -// security and simply mocks the security and identity methods to -// return peer IDs known ahead of time. +// security and simply mocks the security methods. Identity methods +// return the local peer's ID and private key, and whatever the remote +// peer presents as their ID and public key. +// No authentication of the remote identity is performed. type Transport struct { - id peer.ID + id peer.ID + key ci.PrivKey } // New constructs a new insecure transport. +// Deprecated: use NewWithIdentity instead. func New(id peer.ID) *Transport { return &Transport{ id: id, } } -// LocalPeer returns the transports local peer ID. +// New constructs a new insecure transport. The provided private key +// is stored and returned from LocalPrivateKey to satisfy the +// SecureTransport interface, and the public key is sent to +// remote peers. No security is provided. +func NewWithIdentity(id peer.ID, key ci.PrivKey) *Transport { + return &Transport{ + id: id, + key: key, + } +} + +// LocalPeer returns the transport's local peer ID. func (t *Transport) LocalPeer() peer.ID { return t.id } -// LocalPrivateKey returns nil. This transport is not secure. +// LocalPrivateKey returns the local private key. +// This key is used only for identity generation and provides no security. func (t *Transport) LocalPrivateKey() ci.PrivKey { - return nil + return t.key } // SecureInbound *pretends to secure* an outbound connection to the given peer. +// It sends the local peer's ID and public key, and receives the same from the remote peer. +// No validation is performed as to the authenticity or ownership of the provided public key, +// and the key exchange provides no security. +// +// SecureInbound may fail if the remote peer sends an ID and public key that are inconsistent +// with each other, or if a network error occurs during the ID exchange. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { - return &Conn{ - Conn: insecure, - local: t.id, - }, nil + conn := &Conn{ + Conn: insecure, + local: t.id, + localPrivKey: t.key, + } + + err := conn.runHandshakeSync() + if err != nil { + return nil, err + } + + return conn, nil } // SecureOutbound *pretends to secure* an outbound connection to the given peer. +// It sends the local peer's ID and public key, and receives the same from the remote peer. +// No validation is performed as to the authenticity or ownership of the provided public key, +// and the key exchange provides no security. +// +// SecureOutbound may fail if the remote peer sends an ID and public key that are inconsistent +// with each other, or if the ID sent by the remote peer does not match the one dialed. It may +// also fail if a network error occurs during the ID exchange. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { - return &Conn{ - Conn: insecure, - local: t.id, - remote: p, - }, nil + conn := &Conn{ + Conn: insecure, + local: t.id, + localPrivKey: t.key, + } + + err := conn.runHandshakeSync() + if err != nil { + return nil, err + } + + if t.key != nil && p != conn.remote { + return nil, fmt.Errorf("remote peer sent unexpected peer ID. expected=%s received=%s", + p, conn.remote) + } + + return conn, nil } // Conn is the connection type returned by the insecure transport. type Conn struct { net.Conn + local peer.ID remote peer.ID + + localPrivKey ci.PrivKey + remotePubKey ci.PubKey +} + +func makeExchangeMessage(pubkey ci.PubKey) (*pb.Exchange, error) { + keyMsg, err := ci.PublicKeyToProto(pubkey) + if err != nil { + return nil, err + } + id, err := peer.IDFromPublicKey(pubkey) + if err != nil { + return nil, err + } + + return &pb.Exchange{ + Id: []byte(id), + Pubkey: keyMsg, + }, nil +} + +func (ic *Conn) runHandshakeSync() error { + // If we were initialized without keys, behave as in plaintext/1.0.0 (do nothing) + if ic.localPrivKey == nil { + return nil + } + + rw := msgio.NewReadWriter(ic.Conn) + // Generate an Exchange message + msg, err := makeExchangeMessage(ic.localPrivKey.GetPublic()) + if err != nil { + return err + } + + // Send our Exchange and read theirs + remoteMsg, err := readWriteMsg(rw, msg) + if err != nil { + return err + } + + // Pull remote ID and public key from message + remotePubkey, err := ci.PublicKeyFromProto(remoteMsg.Pubkey) + if err != nil { + return err + } + + remoteID, err := peer.IDFromPublicKey(remotePubkey) + if err != nil { + return err + } + + // Validate that ID matches public key + if !remoteID.MatchesPublicKey(remotePubkey) { + calculatedID, _ := peer.IDFromPublicKey(remotePubkey) + return fmt.Errorf("remote peer id does not match public key. id=%s calculated_id=%s", + remoteID, calculatedID) + } + + // Add remote ID and key to conn state + ic.remotePubKey = remotePubkey + ic.remote = remoteID + return nil +} + +// read and write a message at the same time. +func readWriteMsg(c msgio.ReadWriter, out *pb.Exchange) (*pb.Exchange, error) { + outBytes, err := out.Marshal() + if err != nil { + return nil, err + } + wresult := make(chan error) + go func() { + wresult <- c.WriteMsg(outBytes) + }() + + msg, err1 := c.ReadMsg() + + // Always wait for the read to finish. + err2 := <-wresult + + if err1 != nil { + return nil, err1 + } + if err2 != nil { + c.ReleaseMsg(msg) + return nil, err2 + } + inMsg := new(pb.Exchange) + err = inMsg.Unmarshal(msg) + return inMsg, err } // LocalPeer returns the local peer ID. @@ -76,14 +219,15 @@ func (ic *Conn) RemotePeer() peer.ID { return ic.remote } -// RemotePublicKey returns nil. This connection is not secure +// RemotePublicKey returns whatever public key was given by the remote peer. +// Note that no verification of ownership is done, as this connection is not secure. func (ic *Conn) RemotePublicKey() ci.PubKey { - return nil + return ic.remotePubKey } -// LocalPrivateKey returns nil. This connection is not secure. +// LocalPrivateKey returns the private key for the local peer. func (ic *Conn) LocalPrivateKey() ci.PrivKey { - return nil + return ic.localPrivKey } var _ sec.SecureTransport = (*Transport)(nil) diff --git a/core/sec/insecure/insecure_test.go b/core/sec/insecure/insecure_test.go new file mode 100644 index 0000000000..a8b86f16c1 --- /dev/null +++ b/core/sec/insecure/insecure_test.go @@ -0,0 +1,157 @@ +package insecure + +import ( + "bytes" + "context" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" + "io" + "net" + "testing" + + ci "github.com/libp2p/go-libp2p-core/crypto" +) + +// Run a set of sessions through the session setup and verification. +func TestConnections(t *testing.T) { + clientTpt := newTestTransport(t, ci.RSA, 1024) + serverTpt := newTestTransport(t, ci.Ed25519, 1024) + + testConnection(t, clientTpt, serverTpt) +} + +func newTestTransport(t *testing.T, typ, bits int) *Transport { + priv, pub, err := ci.GenerateKeyPair(typ, bits) + if err != nil { + t.Fatal(err) + } + id, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Fatal(err) + } + + return NewWithIdentity(id, priv) +} + +// Create a new pair of connected TCP sockets. +func newConnPair(t *testing.T) (net.Conn, net.Conn) { + lstnr, err := net.Listen("tcp", "localhost:0") + if err != nil { + t.Fatalf("Failed to listen: %v", err) + return nil, nil + } + + var clientErr error + var client net.Conn + addr := lstnr.Addr() + done := make(chan struct{}) + + go func() { + defer close(done) + client, clientErr = net.Dial(addr.Network(), addr.String()) + }() + + server, err := lstnr.Accept() + <-done + + lstnr.Close() + + if err != nil { + t.Fatalf("Failed to accept: %v", err) + } + + if clientErr != nil { + t.Fatalf("Failed to connect: %v", clientErr) + } + + return client, server +} + +// Create a new pair of connected sessions based off of the provided +// session generators. +func connect(t *testing.T, clientTpt, serverTpt *Transport) (sec.SecureConn, sec.SecureConn) { + client, server := newConnPair(t) + + // Connect the client and server sessions + done := make(chan struct{}) + + var clientConn sec.SecureConn + var clientErr error + go func() { + defer close(done) + clientConn, clientErr = clientTpt.SecureOutbound(context.TODO(), client, serverTpt.LocalPeer()) + }() + + serverConn, serverErr := serverTpt.SecureInbound(context.TODO(), server) + <-done + + if serverErr != nil { + t.Fatal(serverErr) + } + + if clientErr != nil { + t.Fatal(clientErr) + } + + return clientConn, serverConn +} + +// Check the peer IDs +func testIDs(t *testing.T, clientTpt, serverTpt *Transport, clientConn, serverConn sec.SecureConn) { + if clientConn.LocalPeer() != clientTpt.LocalPeer() { + t.Fatal("Client Local Peer ID mismatch.") + } + + if clientConn.RemotePeer() != serverTpt.LocalPeer() { + t.Fatal("Client Remote Peer ID mismatch.") + } + + if clientConn.LocalPeer() != serverConn.RemotePeer() { + t.Fatal("Server Local Peer ID mismatch.") + } +} + +// Check the keys +func testKeys(t *testing.T, clientTpt, serverTpt *Transport, clientConn, serverConn sec.SecureConn) { + sk := serverConn.LocalPrivateKey() + pk := sk.GetPublic() + + if !sk.Equals(serverTpt.LocalPrivateKey()) { + t.Error("Private key Mismatch.") + } + + if !pk.Equals(clientConn.RemotePublicKey()) { + t.Error("Public key mismatch.") + } +} + +// Check sending and receiving messages +func testReadWrite(t *testing.T, clientConn, serverConn sec.SecureConn) { + before := []byte("hello world") + _, err := clientConn.Write(before) + if err != nil { + t.Fatal(err) + } + + after := make([]byte, len(before)) + _, err = io.ReadFull(serverConn, after) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(before, after) { + t.Errorf("Message mismatch. %v != %v", before, after) + } +} + +// Setup a new session with a pair of locally connected sockets +func testConnection(t *testing.T, clientTpt, serverTpt *Transport) { + clientConn, serverConn := connect(t, clientTpt, serverTpt) + + testIDs(t, clientTpt, serverTpt, clientConn, serverConn) + testKeys(t, clientTpt, serverTpt, clientConn, serverConn) + testReadWrite(t, clientConn, serverConn) + + clientConn.Close() + serverConn.Close() +} diff --git a/core/sec/insecure/pb/Makefile b/core/sec/insecure/pb/Makefile new file mode 100644 index 0000000000..4fb825a4bb --- /dev/null +++ b/core/sec/insecure/pb/Makefile @@ -0,0 +1,11 @@ +PB = $(wildcard *.proto) +GO = $(PB:.proto=.pb.go) + +all: $(GO) + +%.pb.go: %.proto + protoc --proto_path=$(GOPATH)/src:../../../crypto/pb:. --gogofaster_out=. $< + +clean: + rm -f *.pb.go + rm -f *.go diff --git a/core/sec/insecure/pb/plaintext.pb.go b/core/sec/insecure/pb/plaintext.pb.go new file mode 100644 index 0000000000..68b12f0c82 --- /dev/null +++ b/core/sec/insecure/pb/plaintext.pb.go @@ -0,0 +1,404 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: plaintext.proto + +package plaintext_pb + +import ( + fmt "fmt" + proto "github.com/gogo/protobuf/proto" + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + io "io" + math "math" +) + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + +type Exchange struct { + Id []byte `protobuf:"bytes,1,opt,name=id" json:"id"` + Pubkey *pb.PublicKey `protobuf:"bytes,2,opt,name=pubkey" json:"pubkey,omitempty"` +} + +func (m *Exchange) Reset() { *m = Exchange{} } +func (m *Exchange) String() string { return proto.CompactTextString(m) } +func (*Exchange) ProtoMessage() {} +func (*Exchange) Descriptor() ([]byte, []int) { + return fileDescriptor_aba144f73931b711, []int{0} +} +func (m *Exchange) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Exchange) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Exchange.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalTo(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *Exchange) XXX_Merge(src proto.Message) { + xxx_messageInfo_Exchange.Merge(m, src) +} +func (m *Exchange) XXX_Size() int { + return m.Size() +} +func (m *Exchange) XXX_DiscardUnknown() { + xxx_messageInfo_Exchange.DiscardUnknown(m) +} + +var xxx_messageInfo_Exchange proto.InternalMessageInfo + +func (m *Exchange) GetId() []byte { + if m != nil { + return m.Id + } + return nil +} + +func (m *Exchange) GetPubkey() *pb.PublicKey { + if m != nil { + return m.Pubkey + } + return nil +} + +func init() { + proto.RegisterType((*Exchange)(nil), "plaintext.pb.Exchange") +} + +func init() { proto.RegisterFile("plaintext.proto", fileDescriptor_aba144f73931b711) } + +var fileDescriptor_aba144f73931b711 = []byte{ + // 187 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x2f, 0xc8, 0x49, 0xcc, + 0xcc, 0x2b, 0x49, 0xad, 0x28, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x41, 0x12, 0x48, + 0x92, 0x32, 0x4f, 0xcf, 0x2c, 0xc9, 0x28, 0x4d, 0xd2, 0x4b, 0xce, 0xcf, 0xd5, 0xcf, 0xc9, 0x4c, + 0x2a, 0x30, 0x2a, 0xd0, 0x4f, 0xcf, 0xd7, 0x85, 0xb0, 0x74, 0x93, 0xf3, 0x8b, 0x52, 0xf5, 0x93, + 0x8b, 0x2a, 0x0b, 0x4a, 0xf2, 0xf5, 0x0b, 0x92, 0xa0, 0x2c, 0x88, 0x31, 0x4a, 0x7e, 0x5c, 0x1c, + 0xae, 0x15, 0xc9, 0x19, 0x89, 0x79, 0xe9, 0xa9, 0x42, 0x22, 0x5c, 0x4c, 0x99, 0x29, 0x12, 0x8c, + 0x0a, 0x8c, 0x1a, 0x3c, 0x4e, 0x2c, 0x27, 0xee, 0xc9, 0x33, 0x04, 0x31, 0x65, 0xa6, 0x08, 0xe9, + 0x70, 0xb1, 0x15, 0x94, 0x26, 0x65, 0xa7, 0x56, 0x4a, 0x30, 0x29, 0x30, 0x6a, 0x70, 0x1b, 0x89, + 0xe8, 0xc1, 0x0c, 0x48, 0xd2, 0x0b, 0x28, 0x4d, 0xca, 0xc9, 0x4c, 0xf6, 0x4e, 0xad, 0x0c, 0x82, + 0xaa, 0x71, 0x92, 0x38, 0xf1, 0x48, 0x8e, 0xf1, 0xc2, 0x23, 0x39, 0xc6, 0x07, 0x8f, 0xe4, 0x18, + 0x27, 0x3c, 0x96, 0x63, 0xb8, 0xf0, 0x58, 0x8e, 0xe1, 0xc6, 0x63, 0x39, 0x06, 0x40, 0x00, 0x00, + 0x00, 0xff, 0xff, 0x40, 0xde, 0x90, 0x0b, 0xc2, 0x00, 0x00, 0x00, +} + +func (m *Exchange) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Exchange) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Id != nil { + dAtA[i] = 0xa + i++ + i = encodeVarintPlaintext(dAtA, i, uint64(len(m.Id))) + i += copy(dAtA[i:], m.Id) + } + if m.Pubkey != nil { + dAtA[i] = 0x12 + i++ + i = encodeVarintPlaintext(dAtA, i, uint64(m.Pubkey.Size())) + n1, err := m.Pubkey.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n1 + } + return i, nil +} + +func encodeVarintPlaintext(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func (m *Exchange) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.Id != nil { + l = len(m.Id) + n += 1 + l + sovPlaintext(uint64(l)) + } + if m.Pubkey != nil { + l = m.Pubkey.Size() + n += 1 + l + sovPlaintext(uint64(l)) + } + return n +} + +func sovPlaintext(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozPlaintext(x uint64) (n int) { + return sovPlaintext(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *Exchange) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPlaintext + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Exchange: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Exchange: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Id", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPlaintext + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPlaintext + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPlaintext + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Id = append(m.Id[:0], dAtA[iNdEx:postIndex]...) + if m.Id == nil { + m.Id = []byte{} + } + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Pubkey", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPlaintext + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthPlaintext + } + postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthPlaintext + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.Pubkey == nil { + m.Pubkey = &pb.PublicKey{} + } + if err := m.Pubkey.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipPlaintext(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPlaintext + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthPlaintext + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipPlaintext(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPlaintext + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPlaintext + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPlaintext + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if length < 0 { + return 0, ErrInvalidLengthPlaintext + } + iNdEx += length + if iNdEx < 0 { + return 0, ErrInvalidLengthPlaintext + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPlaintext + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipPlaintext(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + if iNdEx < 0 { + return 0, ErrInvalidLengthPlaintext + } + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthPlaintext = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowPlaintext = fmt.Errorf("proto: integer overflow") +) diff --git a/core/sec/insecure/pb/plaintext.proto b/core/sec/insecure/pb/plaintext.proto new file mode 100644 index 0000000000..1e299df585 --- /dev/null +++ b/core/sec/insecure/pb/plaintext.proto @@ -0,0 +1,10 @@ +syntax = "proto2"; + +package plaintext.pb; + +import "github.com/libp2p/go-libp2p-core/crypto/pb/crypto.proto"; + +message Exchange { + optional bytes id = 1; + optional crypto.pb.PublicKey pubkey = 2; +} From b200369cf1dfd267c71351dabb10c7fa85a88545 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 30 Jul 2019 13:52:08 -0700 Subject: [PATCH 1389/3965] fix: trim connections every gracePeriod/2 We're still rate-limited to once every 10 seconds. However, this means that setting the grace period to something below one minute actually _does_ something useful. fixes #51 --- p2p/net/connmgr/connmgr.go | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 804730dea4..563c1fbb94 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -189,7 +189,15 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { } func (cm *BasicConnMgr) background() { - ticker := time.NewTicker(time.Minute) + interval := cm.gracePeriod / 2 + if interval < cm.silencePeriod { + interval = cm.silencePeriod + } + if interval < 10*time.Second { + interval = 10 * time.Second + } + + ticker := time.NewTicker(interval) defer ticker.Stop() for { From 902962718cb4eabb6b07afa2c856a493d8652a37 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 31 Jul 2019 13:39:19 -0700 Subject: [PATCH 1390/3965] nit: fix log format --- p2p/net/nat/nat.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index 046627289d..cdddaeaec5 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -190,7 +190,7 @@ func (nat *NAT) establishMapping(m *mapping) { } m.setExternalPort(newport) - log.Debugf("NAT Mapping: %s --> %s (%s)", m.ExternalPort(), m.InternalPort(), m.Protocol()) + log.Debugf("NAT Mapping: %d --> %d (%s)", m.ExternalPort(), m.InternalPort(), m.Protocol()) if oldport != 0 && newport != oldport { log.Debugf("failed to renew same port mapping: ch %d -> %d", oldport, newport) } From 435eaa2872786d644440c8c1e88d9d9df954b238 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 31 Jul 2019 19:46:51 -0400 Subject: [PATCH 1391/3965] Return error when closing already closed emitter --- p2p/host/eventbus/basic.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/eventbus/basic.go b/p2p/host/eventbus/basic.go index 0e02bf0769..0e928832b8 100644 --- a/p2p/host/eventbus/basic.go +++ b/p2p/host/eventbus/basic.go @@ -38,7 +38,7 @@ func (e *emitter) Emit(evt interface{}) error { func (e *emitter) Close() error { if !atomic.CompareAndSwapInt32(&e.closed, 0, 1) { - panic("closed an emitter more than once") + return fmt.Errorf("closed an emitter more than once") } if atomic.AddInt32(&e.n.nEmitters, -1) == 0 { e.dropper(e.typ) From f7ede3724980911b3480c65da94fca88f0200fdd Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 29 Jul 2019 08:17:03 +0700 Subject: [PATCH 1392/3965] expose the function to derive the peer's public key from the cert chain --- p2p/security/tls/crypto.go | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index b3693b6af7..42f0b2e0e5 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -66,9 +66,7 @@ func (i *Identity) ConfigForAny() (*tls.Config, <-chan ic.PubKey) { // // It should be used to create a new tls.Config before securing either an // incoming or outgoing connection. -func (i *Identity) ConfigForPeer( - remote peer.ID, -) (*tls.Config, <-chan ic.PubKey) { +func (i *Identity) ConfigForPeer(remote peer.ID) (*tls.Config, <-chan ic.PubKey) { keyCh := make(chan ic.PubKey, 1) // We need to check the peer ID in the VerifyPeerCertificate callback. // The tls.Config it is also used for listening, and we might also have concurrent dials. @@ -88,7 +86,7 @@ func (i *Identity) ConfigForPeer( chain[i] = cert } - pubKey, err := getRemotePubKey(chain) + pubKey, err := PubKeyFromCertChain(chain) if err != nil { return err } @@ -101,8 +99,8 @@ func (i *Identity) ConfigForPeer( return conf, keyCh } -// getRemotePubKey derives the remote's public key from the certificate chain. -func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { +// PubKeyFromCertChain verifies the certificate chain and extract the remote's public key. +func PubKeyFromCertChain(chain []*x509.Certificate) (ic.PubKey, error) { if len(chain) != 1 { return nil, errors.New("expected one certificates in the chain") } From ea13d7a1e9fdd0828d6fe086323f2e7e5e78369c Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 1 Aug 2019 09:18:41 +0700 Subject: [PATCH 1393/3965] make the error check for not receiving a public key more explicit --- p2p/security/tls/transport.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 2dc4257a3e..c563d0937b 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -105,6 +105,9 @@ func (t *Transport) handshake( case remotePubKey = <-keyCh: default: } + if remotePubKey == nil { + return nil, errors.New("go-libp2p-tls BUG: expected remote pub key to be set") + } conn, err := t.setupConn(tlsConn, remotePubKey) if err != nil { @@ -118,10 +121,6 @@ func (t *Transport) handshake( } func (t *Transport) setupConn(tlsConn *tls.Conn, remotePubKey ci.PubKey) (sec.SecureConn, error) { - if remotePubKey == nil { - return nil, errors.New("go-libp2p-tls BUG: expected remote pub key to be set") - } - remotePeerID, err := peer.IDFromPublicKey(remotePubKey) if err != nil { return nil, err From 89ba38c02a7d721957ca2ff106d9692baa1d3c1a Mon Sep 17 00:00:00 2001 From: bigs Date: Thu, 1 Aug 2019 09:38:57 -0800 Subject: [PATCH 1394/3965] Update test error message MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Åukasz Magiera --- p2p/host/eventbus/basic_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/eventbus/basic_test.go b/p2p/host/eventbus/basic_test.go index c2aebe2d2f..8be3d83195 100644 --- a/p2p/host/eventbus/basic_test.go +++ b/p2p/host/eventbus/basic_test.go @@ -111,7 +111,7 @@ func TestEmitOnClosed(t *testing.T) { em.Close() err = em.Emit(EventA{}) if err == nil { - t.Errorf("expected panic") + t.Errorf("expected error") } if err.Error() != "emitter is closed" { t.Error("unexpected message") From 3cfa7eebba6355a796bac10c543ac91400683ee4 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Thu, 1 Aug 2019 13:57:48 -0400 Subject: [PATCH 1395/3965] UNSAFE -> WEAK in RSA key environment variable --- core/crypto/rsa_common.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/crypto/rsa_common.go b/core/crypto/rsa_common.go index 89ae51c788..9293159021 100644 --- a/core/crypto/rsa_common.go +++ b/core/crypto/rsa_common.go @@ -8,7 +8,7 @@ import ( // UnsafeRsaKeyEnv is an environment variable which, when set, lowers the // minimum required bits of RSA keys to 512. This should be used exclusively in // test situations. -const UnsafeRsaKeyEnv = "LIBP2P_ALLOW_UNSAFE_RSA_KEYS" +const UnsafeRsaKeyEnv = "LIBP2P_ALLOW_WEAK_RSA_KEYS" var MinRsaKeyBits = 2048 From b07ce71d048dfdc8f7bc26cd9cb584bff7fd6363 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Thu, 1 Aug 2019 19:05:47 -0400 Subject: [PATCH 1396/3965] Rename constant for weak RSA key environment variable --- core/crypto/rsa_common.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/crypto/rsa_common.go b/core/crypto/rsa_common.go index 9293159021..c7e305439a 100644 --- a/core/crypto/rsa_common.go +++ b/core/crypto/rsa_common.go @@ -5,10 +5,10 @@ import ( "os" ) -// UnsafeRsaKeyEnv is an environment variable which, when set, lowers the +// WeakRsaKeyEnv is an environment variable which, when set, lowers the // minimum required bits of RSA keys to 512. This should be used exclusively in // test situations. -const UnsafeRsaKeyEnv = "LIBP2P_ALLOW_WEAK_RSA_KEYS" +const WeakRsaKeyEnv = "LIBP2P_ALLOW_WEAK_RSA_KEYS" var MinRsaKeyBits = 2048 @@ -17,7 +17,7 @@ var MinRsaKeyBits = 2048 var ErrRsaKeyTooSmall error func init() { - if _, ok := os.LookupEnv(UnsafeRsaKeyEnv); ok { + if _, ok := os.LookupEnv(WeakRsaKeyEnv); ok { MinRsaKeyBits = 512 } From 5915b5397430b20b78ec47c94bb88d1a3a99f191 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Fri, 28 Jun 2019 12:05:46 +0200 Subject: [PATCH 1397/3965] fix: do not allocate when comparing keys --- core/crypto/key.go | 8 ++++--- core/crypto/key_test.go | 30 +++++++++++++++++++------- core/crypto/rsa_go.go | 47 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 13 deletions(-) diff --git a/core/crypto/key.go b/core/crypto/key.go index bc0026eb27..663334fedf 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -363,7 +363,9 @@ func KeyEqual(k1, k2 Key) bool { return true } - b1, err1 := k1.Bytes() - b2, err2 := k2.Bytes() - return subtle.ConstantTimeCompare(b1, b2) == 1 && err1 == err2 + if k1.Type() != k2.Type() { + return false + } + + return k1.Equals(k2) } diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index 0b25432d7c..dcb962f099 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -101,18 +101,19 @@ func testKeyEncoding(t *testing.T, sk PrivKey) { } func testKeyEquals(t *testing.T, k Key) { - kb, err := k.Bytes() - if err != nil { - t.Fatal(err) - } + // kb, err := k.Raw() + // if err != nil { + // t.Fatal(err) + // } if !KeyEqual(k, k) { t.Fatal("Key not equal to itself.") } - if !KeyEqual(k, testkey(kb)) { - t.Fatal("Key not equal to key with same bytes.") - } + // bad test, relies on deep internals.. + // if !KeyEqual(k, testkey(kb)) { + // t.Fatal("Key not equal to key with same bytes.") + // } sk, pk, err := test.RandTestKeyPair(RSA, 512) if err != nil { @@ -143,7 +144,20 @@ func (pk testkey) Raw() ([]byte, error) { } func (pk testkey) Equals(k Key) bool { - return KeyEqual(pk, k) + if pk.Type() != k.Type() { + return false + } + a, err := pk.Raw() + if err != nil { + return false + } + + b, err := k.Raw() + if err != nil { + return false + } + + return bytes.Equal(a, b) } func TestUnknownCurveErrors(t *testing.T) { diff --git a/core/crypto/rsa_go.go b/core/crypto/rsa_go.go index 023588efbb..64534d5f43 100644 --- a/core/crypto/rsa_go.go +++ b/core/crypto/rsa_go.go @@ -3,6 +3,7 @@ package crypto import ( + "bytes" "crypto" "crypto/rand" "crypto/rsa" @@ -63,7 +64,21 @@ func (pk *RsaPublicKey) Raw() ([]byte, error) { // Equals checks whether this key is equal to another func (pk *RsaPublicKey) Equals(k Key) bool { - return KeyEqual(pk, k) + // make sure this is an rsa public key + other, ok := (k).(*RsaPublicKey) + if !ok { + a, err := pk.Raw() + if err != nil { + return false + } + b, err := k.Raw() + if err != nil { + return false + } + return bytes.Equal(a, b) + } + + return pk.k.N.Cmp(other.k.N) == 0 && pk.k.E == other.k.E } // Sign returns a signature of the input data @@ -93,7 +108,35 @@ func (sk *RsaPrivateKey) Raw() ([]byte, error) { // Equals checks whether this key is equal to another func (sk *RsaPrivateKey) Equals(k Key) bool { - return KeyEqual(sk, k) + // make sure this is an rsa public key + other, ok := (k).(*RsaPrivateKey) + if !ok { + a, err := sk.Raw() + if err != nil { + return false + } + b, err := k.Raw() + if err != nil { + return false + } + return bytes.Equal(a, b) + } + + a := sk.sk + b := other.sk + + if a.PublicKey.N.Cmp(b.PublicKey.N) != 0 { + return false + } + if a.PublicKey.E != b.PublicKey.E { + return false + } + + if a.D.Cmp(b.D) != 0 { + return false + } + + return true } // UnmarshalRsaPrivateKey returns a private key from the input x509 bytes From e64c9568ba0f39140d2516f3e471940bdc72659e Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Fri, 28 Jun 2019 12:35:29 +0200 Subject: [PATCH 1398/3965] fixup: openssl --- core/crypto/openssl_common.go | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/core/crypto/openssl_common.go b/core/crypto/openssl_common.go index f235e75f33..06c458afc8 100644 --- a/core/crypto/openssl_common.go +++ b/core/crypto/openssl_common.go @@ -3,6 +3,8 @@ package crypto import ( + "bytes" + pb "github.com/libp2p/go-libp2p-core/crypto/pb" openssl "github.com/spacemonkeygo/openssl" @@ -61,7 +63,15 @@ func (pk *opensslPublicKey) Raw() ([]byte, error) { // Equals checks whether this key is equal to another func (pk *opensslPublicKey) Equals(k Key) bool { - return KeyEqual(pk, k) + a, err := pk.Raw() + if err != nil { + return false + } + b, err := k.Raw() + if err != nil { + return false + } + return bytes.Equal(a, b) } // Sign returns a signature of the input data @@ -94,5 +104,13 @@ func (sk *opensslPrivateKey) Raw() ([]byte, error) { // Equals checks whether this key is equal to another func (sk *opensslPrivateKey) Equals(k Key) bool { - return KeyEqual(sk, k) + a, err := sk.Raw() + if err != nil { + return false + } + b, err := k.Raw() + if err != nil { + return false + } + return bytes.Equal(a, b) } From c79f0a088a6af5df492d326e6594d85da44d0fb8 Mon Sep 17 00:00:00 2001 From: dignifiedquire Date: Fri, 28 Jun 2019 14:04:56 +0200 Subject: [PATCH 1399/3965] use equal method from openssl uses https://github.com/spacemonkeygo/openssl/pull/126 --- core/crypto/openssl_common.go | 46 ++++++++++++++++++++++------------- core/crypto/rsa_openssl.go | 2 +- 2 files changed, 30 insertions(+), 18 deletions(-) diff --git a/core/crypto/openssl_common.go b/core/crypto/openssl_common.go index 06c458afc8..164ade8762 100644 --- a/core/crypto/openssl_common.go +++ b/core/crypto/openssl_common.go @@ -7,7 +7,7 @@ import ( pb "github.com/libp2p/go-libp2p-core/crypto/pb" - openssl "github.com/spacemonkeygo/openssl" + openssl "github.com/libp2p/go-openssl" ) // define these as separate types so we can add more key types later and reuse @@ -63,15 +63,21 @@ func (pk *opensslPublicKey) Raw() ([]byte, error) { // Equals checks whether this key is equal to another func (pk *opensslPublicKey) Equals(k Key) bool { - a, err := pk.Raw() - if err != nil { - return false - } - b, err := k.Raw() - if err != nil { - return false + k0, ok := k.(*RsaPublicKey) + if !ok { + a, err := pk.Raw() + if err != nil { + return false + } + b, err := k.Raw() + if err != nil { + return false + } + + return bytes.Equal(a, b) } - return bytes.Equal(a, b) + + return pk.key.Equal(k0.opensslPublicKey.key) } // Sign returns a signature of the input data @@ -104,13 +110,19 @@ func (sk *opensslPrivateKey) Raw() ([]byte, error) { // Equals checks whether this key is equal to another func (sk *opensslPrivateKey) Equals(k Key) bool { - a, err := sk.Raw() - if err != nil { - return false - } - b, err := k.Raw() - if err != nil { - return false + k0, ok := k.(*RsaPrivateKey) + if !ok { + a, err := sk.Raw() + if err != nil { + return false + } + b, err := k.Raw() + if err != nil { + return false + } + + return bytes.Equal(a, b) } - return bytes.Equal(a, b) + + return sk.key.Equal(k0.opensslPrivateKey.key) } diff --git a/core/crypto/rsa_openssl.go b/core/crypto/rsa_openssl.go index 913dead6af..fd9d451043 100644 --- a/core/crypto/rsa_openssl.go +++ b/core/crypto/rsa_openssl.go @@ -6,7 +6,7 @@ import ( "errors" "io" - openssl "github.com/spacemonkeygo/openssl" + openssl "github.com/libp2p/go-openssl" ) // RsaPrivateKey is an rsa private key From c8185ac62a651926087852163a8b69e224567bce Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 28 Jun 2019 17:23:16 +0200 Subject: [PATCH 1400/3965] use a fallback basicEquals function everywhere This also ensures we check that the types are equal, even if we're comparing directly with `k1.Equals(k2)` instead of `KeyEquals(k1, k2)`. --- core/crypto/ecdsa.go | 4 ++-- core/crypto/key.go | 15 ++++++++++++++- core/crypto/openssl_common.go | 24 ++---------------------- core/crypto/rsa_go.go | 21 ++------------------- core/crypto/secp256k1.go | 4 ++-- 5 files changed, 22 insertions(+), 46 deletions(-) diff --git a/core/crypto/ecdsa.go b/core/crypto/ecdsa.go index c069d0cb52..42ce5d816b 100644 --- a/core/crypto/ecdsa.go +++ b/core/crypto/ecdsa.go @@ -119,7 +119,7 @@ func (ePriv *ECDSAPrivateKey) Raw() ([]byte, error) { func (ePriv *ECDSAPrivateKey) Equals(o Key) bool { oPriv, ok := o.(*ECDSAPrivateKey) if !ok { - return false + return basicEquals(ePriv, o) } return ePriv.priv.D.Cmp(oPriv.priv.D) == 0 @@ -163,7 +163,7 @@ func (ePub ECDSAPublicKey) Raw() ([]byte, error) { func (ePub *ECDSAPublicKey) Equals(o Key) bool { oPub, ok := o.(*ECDSAPublicKey) if !ok { - return false + return basicEquals(ePub, o) } return ePub.pub.X != nil && ePub.pub.Y != nil && oPub.pub.X != nil && oPub.pub.Y != nil && diff --git a/core/crypto/key.go b/core/crypto/key.go index 663334fedf..ddd238272c 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -4,6 +4,7 @@ package crypto import ( + "bytes" "crypto/elliptic" "crypto/hmac" "crypto/rand" @@ -363,9 +364,21 @@ func KeyEqual(k1, k2 Key) bool { return true } + return k1.Equals(k2) +} + +func basicEquals(k1, k2 Key) bool { if k1.Type() != k2.Type() { return false } - return k1.Equals(k2) + a, err := k1.Raw() + if err != nil { + return false + } + b, err := k2.Raw() + if err != nil { + return false + } + return bytes.Equal(a, b) } diff --git a/core/crypto/openssl_common.go b/core/crypto/openssl_common.go index 164ade8762..2466521c7a 100644 --- a/core/crypto/openssl_common.go +++ b/core/crypto/openssl_common.go @@ -3,8 +3,6 @@ package crypto import ( - "bytes" - pb "github.com/libp2p/go-libp2p-core/crypto/pb" openssl "github.com/libp2p/go-openssl" @@ -65,16 +63,7 @@ func (pk *opensslPublicKey) Raw() ([]byte, error) { func (pk *opensslPublicKey) Equals(k Key) bool { k0, ok := k.(*RsaPublicKey) if !ok { - a, err := pk.Raw() - if err != nil { - return false - } - b, err := k.Raw() - if err != nil { - return false - } - - return bytes.Equal(a, b) + return basicEquals(pk, k) } return pk.key.Equal(k0.opensslPublicKey.key) @@ -112,16 +101,7 @@ func (sk *opensslPrivateKey) Raw() ([]byte, error) { func (sk *opensslPrivateKey) Equals(k Key) bool { k0, ok := k.(*RsaPrivateKey) if !ok { - a, err := sk.Raw() - if err != nil { - return false - } - b, err := k.Raw() - if err != nil { - return false - } - - return bytes.Equal(a, b) + return basicEquals(sk, k) } return sk.key.Equal(k0.opensslPrivateKey.key) diff --git a/core/crypto/rsa_go.go b/core/crypto/rsa_go.go index 64534d5f43..4fd2021970 100644 --- a/core/crypto/rsa_go.go +++ b/core/crypto/rsa_go.go @@ -3,7 +3,6 @@ package crypto import ( - "bytes" "crypto" "crypto/rand" "crypto/rsa" @@ -67,15 +66,7 @@ func (pk *RsaPublicKey) Equals(k Key) bool { // make sure this is an rsa public key other, ok := (k).(*RsaPublicKey) if !ok { - a, err := pk.Raw() - if err != nil { - return false - } - b, err := k.Raw() - if err != nil { - return false - } - return bytes.Equal(a, b) + return basicEquals(pk, k) } return pk.k.N.Cmp(other.k.N) == 0 && pk.k.E == other.k.E @@ -111,15 +102,7 @@ func (sk *RsaPrivateKey) Equals(k Key) bool { // make sure this is an rsa public key other, ok := (k).(*RsaPrivateKey) if !ok { - a, err := sk.Raw() - if err != nil { - return false - } - b, err := k.Raw() - if err != nil { - return false - } - return bytes.Equal(a, b) + return basicEquals(sk, k) } a := sk.sk diff --git a/core/crypto/secp256k1.go b/core/crypto/secp256k1.go index d2ac74b996..e2205df88e 100644 --- a/core/crypto/secp256k1.go +++ b/core/crypto/secp256k1.go @@ -66,7 +66,7 @@ func (k *Secp256k1PrivateKey) Raw() ([]byte, error) { func (k *Secp256k1PrivateKey) Equals(o Key) bool { sk, ok := o.(*Secp256k1PrivateKey) if !ok { - return false + return basicEquals(k, o) } return k.D.Cmp(sk.D) == 0 @@ -107,7 +107,7 @@ func (k *Secp256k1PublicKey) Raw() ([]byte, error) { func (k *Secp256k1PublicKey) Equals(o Key) bool { sk, ok := o.(*Secp256k1PublicKey) if !ok { - return false + return basicEquals(k, o) } return (*btcec.PublicKey)(k).IsEqual((*btcec.PublicKey)(sk)) From 707efc9b1b75ddeec19a5b94551f3a30e35b7ca5 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 29 Jun 2019 12:08:33 +0200 Subject: [PATCH 1401/3965] remove non-constant-time private key comparison --- core/crypto/key.go | 3 +-- core/crypto/rsa_go.go | 5 +---- core/crypto/secp256k1.go | 2 +- 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/core/crypto/key.go b/core/crypto/key.go index ddd238272c..fea3e31590 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -4,7 +4,6 @@ package crypto import ( - "bytes" "crypto/elliptic" "crypto/hmac" "crypto/rand" @@ -380,5 +379,5 @@ func basicEquals(k1, k2 Key) bool { if err != nil { return false } - return bytes.Equal(a, b) + return subtle.ConstantTimeCompare(a, b) == 1 } diff --git a/core/crypto/rsa_go.go b/core/crypto/rsa_go.go index 4fd2021970..f8b5999ac3 100644 --- a/core/crypto/rsa_go.go +++ b/core/crypto/rsa_go.go @@ -108,6 +108,7 @@ func (sk *RsaPrivateKey) Equals(k Key) bool { a := sk.sk b := other.sk + // Don't care about constant time. We're only comparing the public half. if a.PublicKey.N.Cmp(b.PublicKey.N) != 0 { return false } @@ -115,10 +116,6 @@ func (sk *RsaPrivateKey) Equals(k Key) bool { return false } - if a.D.Cmp(b.D) != 0 { - return false - } - return true } diff --git a/core/crypto/secp256k1.go b/core/crypto/secp256k1.go index e2205df88e..6e98ea6bf7 100644 --- a/core/crypto/secp256k1.go +++ b/core/crypto/secp256k1.go @@ -69,7 +69,7 @@ func (k *Secp256k1PrivateKey) Equals(o Key) bool { return basicEquals(k, o) } - return k.D.Cmp(sk.D) == 0 + return k.GetPublic().Equals(sk.GetPublic()) } // Sign returns a signature from input data From 9e141011c2159f285de7ef2cfc01dc4cda65a6e2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 1 Aug 2019 16:09:09 -0700 Subject: [PATCH 1402/3965] nit: simplify equality check --- core/crypto/rsa_go.go | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/core/crypto/rsa_go.go b/core/crypto/rsa_go.go index f8b5999ac3..d774991941 100644 --- a/core/crypto/rsa_go.go +++ b/core/crypto/rsa_go.go @@ -109,14 +109,7 @@ func (sk *RsaPrivateKey) Equals(k Key) bool { b := other.sk // Don't care about constant time. We're only comparing the public half. - if a.PublicKey.N.Cmp(b.PublicKey.N) != 0 { - return false - } - if a.PublicKey.E != b.PublicKey.E { - return false - } - - return true + return a.PublicKey.N.Cmp(b.PublicKey.N) == 0 && a.PublicKey.E == b.PublicKey.E } // UnmarshalRsaPrivateKey returns a private key from the input x509 bytes From 13e58e682299a6f8b488a1ecc839edcd4d1bc642 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 1 Aug 2019 17:45:55 -0700 Subject: [PATCH 1403/3965] fix: use fallback comparison for ed25519 keys --- core/crypto/ed25519.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/crypto/ed25519.go b/core/crypto/ed25519.go index e8834d4818..81a2094b8e 100644 --- a/core/crypto/ed25519.go +++ b/core/crypto/ed25519.go @@ -68,7 +68,7 @@ func (k *Ed25519PrivateKey) pubKeyBytes() []byte { func (k *Ed25519PrivateKey) Equals(o Key) bool { edk, ok := o.(*Ed25519PrivateKey) if !ok { - return false + return basicEquals(k, o) } return subtle.ConstantTimeCompare(k.k, edk.k) == 1 @@ -103,7 +103,7 @@ func (k *Ed25519PublicKey) Raw() ([]byte, error) { func (k *Ed25519PublicKey) Equals(o Key) bool { edk, ok := o.(*Ed25519PublicKey) if !ok { - return false + return basicEquals(k, o) } return bytes.Equal(k.k, edk.k) From b906ca563bd0044c7e14b25c8292d7e40046a779 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 2 Aug 2019 11:34:23 +0300 Subject: [PATCH 1404/3965] expose CanHop as a module function --- p2p/protocol/internal/circuitv1-deprecated/relay.go | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 1bb7cc0d79..fcb1b5e262 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -197,8 +197,9 @@ func (r *Relay) Matches(addr ma.Multiaddr) bool { return err == nil } -func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { - s, err := r.host.NewStream(ctx, id, ProtoID) +// Queries a peer for support of hop relay +func CanHop(ctx context.Context, host host.Host, id peer.ID) (bool, error) { + s, err := host.NewStream(ctx, id, ProtoID) if err != nil { return false, err } @@ -233,6 +234,10 @@ func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { return msg.GetCode() == pb.CircuitRelay_SUCCESS, nil } +func (r *Relay) CanHop(ctx context.Context, id peer.ID) (bool, error) { + return CanHop(ctx, r.host, id) +} + func (r *Relay) handleNewStream(s network.Stream) { log.Infof("new relay stream from: %s", s.Conn().RemotePeer()) From 46b5513add96c6474d167d1943f577a578869f22 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 2 Aug 2019 21:12:51 +0300 Subject: [PATCH 1405/3965] gomod: update go-libp2p-circuit --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 05c8ace8f5..3129807584 100644 --- a/go.mod +++ b/go.mod @@ -12,7 +12,7 @@ require ( github.com/libp2p/go-eventbus v0.0.2 github.com/libp2p/go-libp2p-autonat v0.1.0 github.com/libp2p/go-libp2p-blankhost v0.1.3 - github.com/libp2p/go-libp2p-circuit v0.1.0 + github.com/libp2p/go-libp2p-circuit v0.1.1 github.com/libp2p/go-libp2p-core v0.0.6 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 diff --git a/go.sum b/go.sum index b417df666f..de79255fd6 100644 --- a/go.sum +++ b/go.sum @@ -116,8 +116,8 @@ github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3Pt github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.3 h1:0KycuXvPDhmehw0ASsg+s1o3IfXgCUDqfzAl94KEBOg= github.com/libp2p/go-libp2p-blankhost v0.1.3/go.mod h1:KML1//wiKR8vuuJO0y3LUd1uLv+tlkGTAr3jC0S5cLg= -github.com/libp2p/go-libp2p-circuit v0.1.0 h1:eniLL3Y9aq/sryfyV1IAHj5rlvuyj3b7iz8tSiZpdhY= -github.com/libp2p/go-libp2p-circuit v0.1.0/go.mod h1:Ahq4cY3V9VJcHcn1SBXjr78AbFkZeIRmfunbA7pmFh8= +github.com/libp2p/go-libp2p-circuit v0.1.1 h1:eopfG9fAg6rEHWQO1TSrLosXDgYbbbu/RTva/tBANus= +github.com/libp2p/go-libp2p-circuit v0.1.1/go.mod h1:Ahq4cY3V9VJcHcn1SBXjr78AbFkZeIRmfunbA7pmFh8= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyLCcvo1w= From b3085753dc41d2c3481b0b34c8a287abd49f545d Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 2 Aug 2019 21:14:01 +0300 Subject: [PATCH 1406/3965] autorelay: ensure candidate relays can hop --- p2p/host/relay/autorelay.go | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index ac381ac62f..abab272e36 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -12,7 +12,7 @@ import ( "github.com/libp2p/go-libp2p-core/routing" autonat "github.com/libp2p/go-libp2p-autonat" - _ "github.com/libp2p/go-libp2p-circuit" + circuit "github.com/libp2p/go-libp2p-circuit" discovery "github.com/libp2p/go-libp2p-discovery" basic "github.com/libp2p/go-libp2p/p2p/host/basic" @@ -197,6 +197,17 @@ func (ar *AutoRelay) tryRelay(ctx context.Context, pi peer.AddrInfo) bool { return false } + ok, err := circuit.CanHop(ctx, ar.host, pi.ID) + if err != nil { + log.Debugf("error querying relay: %s", err.Error()) + return false + } + + if !ok { + // not a hop relay + return false + } + ar.mx.Lock() defer ar.mx.Unlock() From a80fc34f3ae9c5308ad49adf0ada34da2ef5ef5c Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 2 Aug 2019 15:01:41 -0400 Subject: [PATCH 1407/3965] Update deps, mocknet tests --- go.mod | 4 +++- go.sum | 21 +++++++++++++++++++++ p2p/net/mock/mock_test.go | 32 +++++++++----------------------- 3 files changed, 33 insertions(+), 24 deletions(-) diff --git a/go.mod b/go.mod index 3129807584..ad17ca3dd4 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,7 @@ require ( github.com/libp2p/go-libp2p-autonat v0.1.0 github.com/libp2p/go-libp2p-blankhost v0.1.3 github.com/libp2p/go-libp2p-circuit v0.1.1 - github.com/libp2p/go-libp2p-core v0.0.6 + github.com/libp2p/go-libp2p-core v0.2.0 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 @@ -36,3 +36,5 @@ require ( github.com/multiformats/go-multistream v0.1.0 github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 ) + +replace github.com/libp2p/go-libp2p-testing => ../../go-libp2p-testing diff --git a/go.sum b/go.sum index de79255fd6..c3a4452926 100644 --- a/go.sum +++ b/go.sum @@ -89,10 +89,13 @@ github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsj github.com/jbenet/goprocess v0.1.3 h1:YKyIEECS/XvcfHtBzxtjBBbWK+MbvA6dG8ASiqwvr10= github.com/jbenet/goprocess v0.1.3/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= +github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= +github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d/go.mod h1:P2viExyCEfeWGU259JnaQ34Inuec4R38JCyBx2edgD0= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4= +github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b h1:wxtKgYHEncAU00muMD06dzLiahtGM1eouRNOzVV7tdQ= github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= @@ -122,6 +125,8 @@ github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGc github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyLCcvo1w= github.com/libp2p/go-libp2p-core v0.0.6/go.mod h1:0d9xmaYAVY5qmbp/fcgxHT3ZJsLjYeYPMJAUKpaCHrE= +github.com/libp2p/go-libp2p-core v0.2.0 h1:ycFtuNwtZBAJSxzaHbyv6NjG3Yj5Nmra1csHaQ3zwaw= +github.com/libp2p/go-libp2p-core v0.2.0/go.mod h1:X0eyB0Gy93v0DZtSYbEM7RnMChm9Uv3j7yRXjO77xSI= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= @@ -160,6 +165,8 @@ github.com/libp2p/go-mplex v0.1.0 h1:/nBTy5+1yRyY82YaO6HXQRnO5IAGsXTjEJaR3LdTPc0 github.com/libp2p/go-mplex v0.1.0/go.mod h1:SXgmdki2kwCUlCCbfGLEgHjC4pFqhTp0ZoV6aiKgxDU= github.com/libp2p/go-msgio v0.0.2 h1:ivPvEKHxmVkTClHzg6RXTYHqaJQ0V9cDbq+6lKb3UV0= github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= +github.com/libp2p/go-msgio v0.0.4 h1:agEFehY3zWJFUHK6SEMR7UYmk2z6kC3oeCM7ybLhguA= +github.com/libp2p/go-msgio v0.0.4/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-nat v0.0.3 h1:l6fKV+p0Xa354EqQOQP+d8CivdLM4kl5GxC1hSc/UeI= github.com/libp2p/go-nat v0.0.3/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= @@ -177,10 +184,14 @@ github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZ github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= +github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= +github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= +github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE= github.com/miekg/dns v1.1.12 h1:WMhc1ik4LNkTg8U9l3hI1LvxKmIL+f1+WV/SZtCbDDA= github.com/miekg/dns v1.1.12/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g= @@ -230,6 +241,8 @@ github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= +github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= +github.com/smola/gocompat v0.2.0/go.mod h1:1B0MlxbmoZNo3h8guHp8HztB3BSYR5itql9qtVc0ypY= github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a h1:/eS3yfGjQKG+9kayBkj0ip1BGhq6zJ3eaVksphxAaek= github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= @@ -242,7 +255,9 @@ github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tL github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= +github.com/src-d/envconfig v1.0.0/go.mod h1:Q9YQZ7BKITldTBnoxsE5gOeB5y66RyPXeue/R4aaNBc= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= @@ -259,10 +274,12 @@ github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxf github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= +github.com/x-cray/logrus-prefixed-formatter v0.5.2/go.mod h1:2duySbKsL6M18s5GU7VPsoEPHyzalCE06qoARUCeBBE= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= @@ -290,6 +307,7 @@ golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6 h1:bjcUS9ztw9kFmmIxJInhon/0Is3p+EHBKNgquIzo1OI= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -302,6 +320,7 @@ golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= @@ -315,6 +334,8 @@ gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d/go.mod h1:z+K8VcOYVYcSwSjGebuDL6176A1XskgbtNl64NSg+n8= +gopkg.in/src-d/go-log.v1 v1.0.1/go.mod h1:GN34hKP0g305ysm2/hctJ0Y8nWP3zxXXJ8GFabTyABE= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index d0c6526833..2ed983839d 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -11,14 +11,12 @@ import ( "testing" "time" - ci "github.com/libp2p/go-libp2p-core/crypto" + detectrace "github.com/ipfs/go-detect-race" "github.com/libp2p/go-libp2p-core/helpers" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/protocol" "github.com/libp2p/go-libp2p-core/test" - - detectrace "github.com/ipfs/go-detect-race" tnet "github.com/libp2p/go-libp2p-testing/net" ) @@ -32,18 +30,9 @@ func randPeer(t *testing.T) peer.ID { func TestNetworkSetup(t *testing.T) { ctx := context.Background() - sk1, _, err := test.RandTestKeyPair(ci.RSA, 512) - if err != nil { - t.Fatal(t) - } - sk2, _, err := test.RandTestKeyPair(ci.RSA, 512) - if err != nil { - t.Fatal(t) - } - sk3, _, err := test.RandTestKeyPair(ci.RSA, 512) - if err != nil { - t.Fatal(t) - } + id1 := tnet.RandIdentityOrFatal(t) + id2 := tnet.RandIdentityOrFatal(t) + id3 := tnet.RandIdentityOrFatal(t) mn := New(ctx) // peers := []peer.ID{p1, p2, p3} @@ -53,19 +42,19 @@ func TestNetworkSetup(t *testing.T) { a2 := tnet.RandLocalTCPAddress() a3 := tnet.RandLocalTCPAddress() - h1, err := mn.AddPeer(sk1, a1) + h1, err := mn.AddPeer(id1.PrivateKey(), a1) if err != nil { t.Fatal(err) } p1 := h1.ID() - h2, err := mn.AddPeer(sk2, a2) + h2, err := mn.AddPeer(id2.PrivateKey(), a2) if err != nil { t.Fatal(err) } p2 := h2.ID() - h3, err := mn.AddPeer(sk3, a3) + h3, err := mn.AddPeer(id3.PrivateKey(), a3) if err != nil { t.Fatal(err) } @@ -425,13 +414,10 @@ func TestAdding(t *testing.T) { var peers []peer.ID for i := 0; i < 3; i++ { - sk, _, err := test.RandTestKeyPair(ci.RSA, 512) - if err != nil { - t.Fatal(err) - } + id := tnet.RandIdentityOrFatal(t) a := tnet.RandLocalTCPAddress() - h, err := mn.AddPeer(sk, a) + h, err := mn.AddPeer(id.PrivateKey(), a) if err != nil { t.Fatal(err) } From c261182d5066ef077c2e0c367fe1e729bb01ca72 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 2 Aug 2019 17:37:26 -0400 Subject: [PATCH 1408/3965] Adjust test timings for Ed25519 --- go.mod | 4 +--- go.sum | 2 ++ p2p/net/mock/mock_test.go | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/go.mod b/go.mod index ad17ca3dd4..43bd9fbc7a 100644 --- a/go.mod +++ b/go.mod @@ -22,7 +22,7 @@ require ( github.com/libp2p/go-libp2p-peerstore v0.1.2 github.com/libp2p/go-libp2p-secio v0.1.0 github.com/libp2p/go-libp2p-swarm v0.1.1 - github.com/libp2p/go-libp2p-testing v0.0.4 + github.com/libp2p/go-libp2p-testing v0.1.0 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 github.com/libp2p/go-libp2p-yamux v0.2.1 github.com/libp2p/go-maddr-filter v0.0.5 @@ -36,5 +36,3 @@ require ( github.com/multiformats/go-multistream v0.1.0 github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 ) - -replace github.com/libp2p/go-libp2p-testing => ../../go-libp2p-testing diff --git a/go.sum b/go.sum index c3a4452926..047818be82 100644 --- a/go.sum +++ b/go.sum @@ -152,6 +152,8 @@ github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MB github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.4 h1:Qev57UR47GcLPXWjrunv5aLIQGO4n9mhI/8/EIrEEFc= github.com/libp2p/go-libp2p-testing v0.0.4/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= +github.com/libp2p/go-libp2p-testing v0.1.0 h1:WaFRj/t3HdMZGNZqnU2pS7pDRBmMeoDx7/HDNpeyT9U= +github.com/libp2p/go-libp2p-testing v0.1.0/go.mod h1:xaZWMJrPUM5GlDBxCeGUi7kI4eqnjVyavGroI2nxEM0= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 h1:PZMS9lhjK9VytzMCW3tWHAXtKXmlURSc3ZdvwEcKCzw= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07qPW4r01GnTenLW4oi3lOPbUMGJJA= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index 2ed983839d..f78830f001 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -583,7 +583,7 @@ func TestLimitedStreams(t *testing.T) { } wg.Wait() - if !within(time.Since(before), time.Duration(time.Second*2), time.Second/3) { + if !within(time.Since(before), time.Second*2, time.Second) { t.Fatal("Expected 2ish seconds but got ", time.Since(before)) } } @@ -648,7 +648,7 @@ func TestStreamsWithLatency(t *testing.T) { wg.Wait() delta := time.Since(checkpoint) - tolerance := time.Millisecond * 100 + tolerance := time.Second if !within(delta, latency, tolerance) { t.Fatalf("Expected write to take ~%s (+/- %s), but took %s", latency.String(), tolerance.String(), delta.String()) } From 3e4400cc6847d3ddfd28d47f74492fe4b6bab256 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 2 Aug 2019 19:02:59 -0400 Subject: [PATCH 1409/3965] Bump secio --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 43bd9fbc7a..4abdb3bbee 100644 --- a/go.mod +++ b/go.mod @@ -20,7 +20,7 @@ require ( github.com/libp2p/go-libp2p-nat v0.0.4 github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.1.2 - github.com/libp2p/go-libp2p-secio v0.1.0 + github.com/libp2p/go-libp2p-secio v0.2.0 github.com/libp2p/go-libp2p-swarm v0.1.1 github.com/libp2p/go-libp2p-testing v0.1.0 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 diff --git a/go.sum b/go.sum index 047818be82..6ad6ab8ab6 100644 --- a/go.sum +++ b/go.sum @@ -145,6 +145,8 @@ github.com/libp2p/go-libp2p-peerstore v0.1.2 h1:MamqRA9OU9U/hbpeiowG3q3QNrWI+omK github.com/libp2p/go-libp2p-peerstore v0.1.2/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLMWdB7GTQ= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= +github.com/libp2p/go-libp2p-secio v0.2.0 h1:ywzZBsWEEz2KNTn5RtzauEDq5RFEefPsttXYwAWqHng= +github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= github.com/libp2p/go-libp2p-swarm v0.1.1 h1:QW7pjyTRIxt9yyBid52YmMRGtkFXUE/rbEVWsQ0ae+w= github.com/libp2p/go-libp2p-swarm v0.1.1/go.mod h1:4NVJaLwq/dr5kEq79Jo6pMin7ZFwLx73ln1FTefR91Q= From c385e71b52bd8847aa99fe7c7e1cb4498a94351f Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Fri, 2 Aug 2019 19:24:00 -0400 Subject: [PATCH 1410/3965] Bump swarm version --- go.mod | 4 ++-- go.sum | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index 4abdb3bbee..c11ed08abd 100644 --- a/go.mod +++ b/go.mod @@ -19,9 +19,9 @@ require ( github.com/libp2p/go-libp2p-mplex v0.2.1 github.com/libp2p/go-libp2p-nat v0.0.4 github.com/libp2p/go-libp2p-netutil v0.1.0 - github.com/libp2p/go-libp2p-peerstore v0.1.2 + github.com/libp2p/go-libp2p-peerstore v0.1.3 github.com/libp2p/go-libp2p-secio v0.2.0 - github.com/libp2p/go-libp2p-swarm v0.1.1 + github.com/libp2p/go-libp2p-swarm v0.2.0 github.com/libp2p/go-libp2p-testing v0.1.0 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 github.com/libp2p/go-libp2p-yamux v0.2.1 diff --git a/go.sum b/go.sum index 6ad6ab8ab6..0d05b332f1 100644 --- a/go.sum +++ b/go.sum @@ -141,15 +141,15 @@ github.com/libp2p/go-libp2p-netutil v0.1.0 h1:zscYDNVEcGxyUpMd0JReUZTrpMfia8PmLK github.com/libp2p/go-libp2p-netutil v0.1.0/go.mod h1:3Qv/aDqtMLTUyQeundkKsA+YCThNdbQD54k3TqjpbFU= github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMgxjsHm7+J5kjWY= github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HBb6u0MOvdJ7YIsmcwtY= -github.com/libp2p/go-libp2p-peerstore v0.1.2 h1:MamqRA9OU9U/hbpeiowG3q3QNrWI+omKX8n9erT6aeE= -github.com/libp2p/go-libp2p-peerstore v0.1.2/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= +github.com/libp2p/go-libp2p-peerstore v0.1.3 h1:wMgajt1uM2tMiqf4M+4qWKVyyFc8SfA+84VV9glZq1M= +github.com/libp2p/go-libp2p-peerstore v0.1.3/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLMWdB7GTQ= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0 h1:ywzZBsWEEz2KNTn5RtzauEDq5RFEefPsttXYwAWqHng= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= -github.com/libp2p/go-libp2p-swarm v0.1.1 h1:QW7pjyTRIxt9yyBid52YmMRGtkFXUE/rbEVWsQ0ae+w= -github.com/libp2p/go-libp2p-swarm v0.1.1/go.mod h1:4NVJaLwq/dr5kEq79Jo6pMin7ZFwLx73ln1FTefR91Q= +github.com/libp2p/go-libp2p-swarm v0.2.0 h1:gUqj9WYFC9mQ6khEtO5EjfaAtevwBMxsXRARbi6lAiI= +github.com/libp2p/go-libp2p-swarm v0.2.0/go.mod h1:x07b4zkMFo2EvgPV2bMTlNmdQc8i+74Jjio7xGvsTgU= github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.4 h1:Qev57UR47GcLPXWjrunv5aLIQGO4n9mhI/8/EIrEEFc= From 744a6addae2717776e785d45adcab97fdafad9f0 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 5 Aug 2019 08:12:37 +0700 Subject: [PATCH 1411/3965] update quic-go to v0.12.0 (supporting QUIC draft-22) --- p2p/transport/quic/conn.go | 6 ++++-- p2p/transport/quic/crypto.go | 3 +++ p2p/transport/quic/listener.go | 5 +++-- p2p/transport/quic/transport.go | 2 +- 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index bf173458a4..75e2193692 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -1,6 +1,8 @@ package libp2pquic import ( + "context" + ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/mux" "github.com/libp2p/go-libp2p-core/peer" @@ -36,13 +38,13 @@ func (c *conn) IsClosed() bool { // OpenStream creates a new stream. func (c *conn) OpenStream() (mux.MuxedStream, error) { - qstr, err := c.sess.OpenStreamSync() + qstr, err := c.sess.OpenStreamSync(context.Background()) return &stream{Stream: qstr}, err } // AcceptStream accepts a stream opened by the other side. func (c *conn) AcceptStream() (mux.MuxedStream, error) { - qstr, err := c.sess.AcceptStream() + qstr, err := c.sess.AcceptStream(context.Background()) return &stream{Stream: qstr}, err } diff --git a/p2p/transport/quic/crypto.go b/p2p/transport/quic/crypto.go index 68a30a629e..2ce698680f 100644 --- a/p2p/transport/quic/crypto.go +++ b/p2p/transport/quic/crypto.go @@ -18,6 +18,8 @@ import ( // mint certificate selection is broken. const hostname = "quic.ipfs" +const alpn string = "libp2p" + const certValidityPeriod = 180 * 24 * time.Hour func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { @@ -55,6 +57,7 @@ func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { Certificate: [][]byte{cert.Raw, hostCert.Raw}, PrivateKey: ephemeralKey, }}, + NextProtos: []string{alpn}, }, nil } diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 327af2f4c1..2e39854dd2 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -1,6 +1,7 @@ package libp2pquic import ( + "context" "crypto/tls" "net" @@ -60,13 +61,13 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, // Accept accepts new connections. func (l *listener) Accept() (tpt.CapableConn, error) { for { - sess, err := l.quicListener.Accept() + sess, err := l.quicListener.Accept(context.Background()) if err != nil { return nil, err } conn, err := l.setupConn(sess) if err != nil { - sess.CloseWithError(0, err) + sess.CloseWithError(0, err.Error()) continue } return conn, nil diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index c33b241766..ebf5712005 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -24,7 +24,7 @@ var quicConfig = &quic.Config{ MaxIncomingUniStreams: -1, // disable unidirectional streams MaxReceiveStreamFlowControlWindow: 3 * (1 << 20), // 3 MB MaxReceiveConnectionFlowControlWindow: 4.5 * (1 << 20), // 4.5 MB - AcceptCookie: func(clientAddr net.Addr, cookie *quic.Cookie) bool { + AcceptToken: func(clientAddr net.Addr, _ *quic.Token) bool { // TODO(#6): require source address validation when under load return true }, From cacd2a4016b0146c246902164f316a9cf115ddcd Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 29 Jul 2019 08:26:25 +0700 Subject: [PATCH 1412/3965] use the handshake logic from go-libp2p-tls --- p2p/transport/quic/conn_test.go | 79 +++---------- p2p/transport/quic/crypto.go | 123 -------------------- p2p/transport/quic/libp2pquic_suite_test.go | 6 + p2p/transport/quic/listener.go | 22 +++- p2p/transport/quic/listener_test.go | 1 + p2p/transport/quic/transport.go | 45 +++---- 6 files changed, 57 insertions(+), 219 deletions(-) delete mode 100644 p2p/transport/quic/crypto.go diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 8b86af2073..ef2430783b 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -4,10 +4,9 @@ import ( "bytes" "context" "crypto/rand" - "crypto/rsa" - "crypto/tls" - "crypto/x509" + "fmt" "io/ioutil" + mrand "math/rand" "time" ic "github.com/libp2p/go-libp2p-core/crypto" @@ -26,12 +25,26 @@ var _ = Describe("Connection", func() { ) createPeer := func() (peer.ID, ic.PrivKey) { - key, err := rsa.GenerateKey(rand.Reader, 1024) - Expect(err).ToNot(HaveOccurred()) - priv, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(key)) + var priv ic.PrivKey + var err error + switch mrand.Int() % 4 { + case 0: + fmt.Fprintf(GinkgoWriter, " using an ECDSA key: ") + priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) + case 1: + fmt.Fprintf(GinkgoWriter, " using an RSA key: ") + priv, _, err = ic.GenerateRSAKeyPair(1024, rand.Reader) + case 2: + fmt.Fprintf(GinkgoWriter, " using an Ed25519 key: ") + priv, _, err = ic.GenerateEd25519Key(rand.Reader) + case 3: + fmt.Fprintf(GinkgoWriter, " using an secp256k1 key: ") + priv, _, err = ic.GenerateSecp256k1Key(rand.Reader) + } Expect(err).ToNot(HaveOccurred()) id, err := peer.IDFromPrivateKey(priv) Expect(err).ToNot(HaveOccurred()) + fmt.Fprintln(GinkgoWriter, id.Pretty()) return id, priv } @@ -52,11 +65,6 @@ var _ = Describe("Connection", func() { return <-addrChan, connChan } - // modify the cert chain such that verificiation will fail - invalidateCertChain := func(tlsConf *tls.Config) { - tlsConf.Certificates[0].Certificate = [][]byte{tlsConf.Certificates[0].Certificate[0]} - } - BeforeEach(func() { serverID, serverKey = createPeer() clientID, clientKey = createPeer() @@ -141,55 +149,6 @@ var _ = Describe("Connection", func() { Consistently(serverConnChan).ShouldNot(Receive()) }) - It("fails if the client presents an invalid cert chain", func() { - serverTransport, err := NewTransport(serverKey) - Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") - - clientTransport, err := NewTransport(clientKey) - invalidateCertChain(clientTransport.(*transport).tlsConf) - Expect(err).ToNot(HaveOccurred()) - conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) - Expect(err).ToNot(HaveOccurred()) - Eventually(func() bool { return conn.IsClosed() }).Should(BeTrue()) - Consistently(serverConnChan).ShouldNot(Receive()) - }) - - It("fails if the server presents an invalid cert chain", func() { - serverTransport, err := NewTransport(serverKey) - invalidateCertChain(serverTransport.(*transport).tlsConf) - Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") - - clientTransport, err := NewTransport(clientKey) - Expect(err).ToNot(HaveOccurred()) - _, err = clientTransport.Dial(context.Background(), serverAddr, serverID) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("CRYPTO_ERROR")) - Consistently(serverConnChan).ShouldNot(Receive()) - }) - - It("keeps accepting connections after a failed connection attempt", func() { - serverTransport, err := NewTransport(serverKey) - Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") - - // first dial with an invalid cert chain - clientTransport1, err := NewTransport(clientKey) - invalidateCertChain(clientTransport1.(*transport).tlsConf) - Expect(err).ToNot(HaveOccurred()) - _, err = clientTransport1.Dial(context.Background(), serverAddr, serverID) - Expect(err).ToNot(HaveOccurred()) - Consistently(serverConnChan).ShouldNot(Receive()) - - // then dial with a valid client - clientTransport2, err := NewTransport(clientKey) - Expect(err).ToNot(HaveOccurred()) - _, err = clientTransport2.Dial(context.Background(), serverAddr, serverID) - Expect(err).ToNot(HaveOccurred()) - Eventually(serverConnChan).Should(Receive()) - }) - It("dials to two servers at the same time", func() { serverID2, serverKey2 := createPeer() diff --git a/p2p/transport/quic/crypto.go b/p2p/transport/quic/crypto.go deleted file mode 100644 index 2ce698680f..0000000000 --- a/p2p/transport/quic/crypto.go +++ /dev/null @@ -1,123 +0,0 @@ -package libp2pquic - -import ( - "crypto/ecdsa" - "crypto/elliptic" - "crypto/rand" - "crypto/tls" - "crypto/x509" - "errors" - "math/big" - "time" - - "github.com/gogo/protobuf/proto" - ic "github.com/libp2p/go-libp2p-core/crypto" - pb "github.com/libp2p/go-libp2p-core/crypto/pb" -) - -// mint certificate selection is broken. -const hostname = "quic.ipfs" - -const alpn string = "libp2p" - -const certValidityPeriod = 180 * 24 * time.Hour - -func generateConfig(privKey ic.PrivKey) (*tls.Config, error) { - key, hostCert, err := keyToCertificate(privKey) - if err != nil { - return nil, err - } - // The ephemeral key used just for a couple of connections (or a limited time). - ephemeralKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) - if err != nil { - return nil, err - } - // Sign the ephemeral key using the host key. - // This is the only time that the host's private key of the peer is needed. - // Note that this step could be done asynchronously, such that a running node doesn't need access its private key at all. - certTemplate := &x509.Certificate{ - DNSNames: []string{hostname}, - SerialNumber: big.NewInt(1), - NotBefore: time.Now().Add(-24 * time.Hour), - NotAfter: time.Now().Add(certValidityPeriod), - } - certDER, err := x509.CreateCertificate(rand.Reader, certTemplate, hostCert, ephemeralKey.Public(), key) - if err != nil { - return nil, err - } - cert, err := x509.ParseCertificate(certDER) - if err != nil { - return nil, err - } - return &tls.Config{ - ServerName: hostname, - InsecureSkipVerify: true, // This is not insecure here. We will verify the cert chain ourselves. - ClientAuth: tls.RequireAnyClientCert, - Certificates: []tls.Certificate{{ - Certificate: [][]byte{cert.Raw, hostCert.Raw}, - PrivateKey: ephemeralKey, - }}, - NextProtos: []string{alpn}, - }, nil -} - -func getRemotePubKey(chain []*x509.Certificate) (ic.PubKey, error) { - if len(chain) != 2 { - return nil, errors.New("expected 2 certificates in the chain") - } - pool := x509.NewCertPool() - pool.AddCert(chain[1]) - if _, err := chain[0].Verify(x509.VerifyOptions{Roots: pool}); err != nil { - return nil, err - } - remotePubKey, err := x509.MarshalPKIXPublicKey(chain[1].PublicKey) - if err != nil { - return nil, err - } - return ic.UnmarshalRsaPublicKey(remotePubKey) -} - -func keyToCertificate(sk ic.PrivKey) (interface{}, *x509.Certificate, error) { - sn, err := rand.Int(rand.Reader, big.NewInt(1<<62)) - if err != nil { - return nil, nil, err - } - tmpl := &x509.Certificate{ - SerialNumber: sn, - NotBefore: time.Now().Add(-24 * time.Hour), - NotAfter: time.Now().Add(certValidityPeriod), - IsCA: true, - BasicConstraintsValid: true, - } - - var publicKey, privateKey interface{} - keyBytes, err := sk.Bytes() - if err != nil { - return nil, nil, err - } - pbmes := new(pb.PrivateKey) - if err := proto.Unmarshal(keyBytes, pbmes); err != nil { - return nil, nil, err - } - switch pbmes.GetType() { - case pb.KeyType_RSA: - k, err := x509.ParsePKCS1PrivateKey(pbmes.GetData()) - if err != nil { - return nil, nil, err - } - publicKey = &k.PublicKey - privateKey = k - // TODO: add support for ECDSA - default: - return nil, nil, errors.New("unsupported key type for TLS") - } - certDER, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, publicKey, privateKey) - if err != nil { - return nil, nil, err - } - cert, err := x509.ParseCertificate(certDER) - if err != nil { - return nil, nil, err - } - return privateKey, cert, nil -} diff --git a/p2p/transport/quic/libp2pquic_suite_test.go b/p2p/transport/quic/libp2pquic_suite_test.go index 9675f74995..2bb5d416c6 100644 --- a/p2p/transport/quic/libp2pquic_suite_test.go +++ b/p2p/transport/quic/libp2pquic_suite_test.go @@ -1,6 +1,8 @@ package libp2pquic import ( + mrand "math/rand" + . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" @@ -11,3 +13,7 @@ func TestLibp2pQuicTransport(t *testing.T) { RegisterFailHandler(Fail) RunSpecs(t, "libp2p QUIC Transport Suite") } + +var _ = BeforeSuite(func() { + mrand.Seed(GinkgoRandomSeed()) +}) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 2e39854dd2..0d1b835326 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -8,14 +8,13 @@ import ( ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" tpt "github.com/libp2p/go-libp2p-core/transport" + p2ptls "github.com/libp2p/go-libp2p-tls" quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) -var quicListenAddr = quic.ListenAddr - // A listener listens for QUIC connections. type listener struct { quicListener quic.Listener @@ -28,7 +27,16 @@ type listener struct { var _ tpt.Listener = &listener{} -func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, key ic.PrivKey, tlsConf *tls.Config) (tpt.Listener, error) { +func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, key ic.PrivKey, identity *p2ptls.Identity) (tpt.Listener, error) { + var tlsConf tls.Config + tlsConf.GetConfigForClient = func(_ *tls.ClientHelloInfo) (*tls.Config, error) { + // return a tls.Config that verifies the peer's certificate chain. + // Note that since we have no way of associating an incoming QUIC connection with + // the peer ID calculated here, we don't actually receive the peer's public key + // from the key chan. + conf, _ := identity.ConfigForAny() + return conf, nil + } lnet, host, err := manet.DialArgs(addr) if err != nil { return nil, err @@ -41,7 +49,7 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, if err != nil { return nil, err } - ln, err := quic.Listen(conn, tlsConf, quicConfig) + ln, err := quic.Listen(conn, &tlsConf, quicConfig) if err != nil { return nil, err } @@ -75,7 +83,11 @@ func (l *listener) Accept() (tpt.CapableConn, error) { } func (l *listener) setupConn(sess quic.Session) (tpt.CapableConn, error) { - remotePubKey, err := getRemotePubKey(sess.ConnectionState().PeerCertificates) + // The tls.Config used to establish this connection already verified the certificate chain. + // Since we don't have any way of knowing which tls.Config was used though, + // we have to re-determine the peer's identity here. + // Therefore, this is expected to never fail. + remotePubKey, err := p2ptls.PubKeyFromCertChain(sess.ConnectionState().PeerCertificates) if err != nil { return nil, err } diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index bf0435f1ac..44d1c4ab17 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -30,6 +30,7 @@ var _ = Describe("Listener", func() { Context("listening on the right address", func() { It("returns the address it is listening on", func() { localAddr, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/quic") + Expect(err).ToNot(HaveOccurred()) ln, err := t.Listen(localAddr) Expect(err).ToNot(HaveOccurred()) netAddr := ln.Addr() diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index ebf5712005..e6fc143e2a 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -2,9 +2,6 @@ package libp2pquic import ( "context" - "crypto/tls" - "crypto/x509" - "errors" "fmt" "net" "sync" @@ -12,6 +9,7 @@ import ( ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" tpt "github.com/libp2p/go-libp2p-core/transport" + p2ptls "github.com/libp2p/go-libp2p-tls" quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" @@ -74,7 +72,7 @@ func (c *connManager) createConn(network, host string) (net.PacketConn, error) { type transport struct { privKey ic.PrivKey localPeer peer.ID - tlsConf *tls.Config + identity *p2ptls.Identity connManager *connManager } @@ -86,7 +84,7 @@ func NewTransport(key ic.PrivKey) (tpt.Transport, error) { if err != nil { return nil, err } - tlsConf, err := generateConfig(key) + identity, err := p2ptls.NewIdentity(key) if err != nil { return nil, err } @@ -94,7 +92,7 @@ func NewTransport(key ic.PrivKey) (tpt.Transport, error) { return &transport{ privKey: key, localPeer: localPeer, - tlsConf: tlsConf, + identity: identity, connManager: &connManager{}, }, nil } @@ -113,30 +111,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - var remotePubKey ic.PubKey - tlsConf := t.tlsConf.Clone() - // We need to check the peer ID in the VerifyPeerCertificate callback. - // The tls.Config it is also used for listening, and we might also have concurrent dials. - // Clone it so we can check for the specific peer ID we're dialing here. - tlsConf.VerifyPeerCertificate = func(rawCerts [][]byte, _ [][]*x509.Certificate) error { - chain := make([]*x509.Certificate, len(rawCerts)) - for i := 0; i < len(rawCerts); i++ { - cert, err := x509.ParseCertificate(rawCerts[i]) - if err != nil { - return err - } - chain[i] = cert - } - var err error - remotePubKey, err = getRemotePubKey(chain) - if err != nil { - return err - } - if !p.MatchesPublicKey(remotePubKey) { - return errors.New("peer IDs don't match") - } - return nil - } + tlsConf, keyCh := t.identity.ConfigForPeer(p) sess, err := quic.DialContext(ctx, pconn, addr, host, tlsConf, quicConfig) if err != nil { return nil, err @@ -145,6 +120,14 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } + + // Should be ready by this point, don't block. + var remotePubKey ic.PubKey + select { + case remotePubKey = <-keyCh: + default: + } + return &conn{ sess: sess, transport: t, @@ -164,7 +147,7 @@ func (t *transport) CanDial(addr ma.Multiaddr) bool { // Listen listens for new QUIC connections on the passed multiaddr. func (t *transport) Listen(addr ma.Multiaddr) (tpt.Listener, error) { - return newListener(addr, t, t.localPeer, t.privKey, t.tlsConf) + return newListener(addr, t, t.localPeer, t.privKey, t.identity) } // Proxy returns true if this transport proxies. From 5d493b46b352c6927077ddc21cdc8b2d316d89da Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 1 Aug 2019 09:20:19 +0700 Subject: [PATCH 1413/3965] add an error check that we actually received the peer's public key --- p2p/transport/quic/transport.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index e6fc143e2a..5b217087bb 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -2,6 +2,7 @@ package libp2pquic import ( "context" + "errors" "fmt" "net" "sync" @@ -127,6 +128,9 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp case remotePubKey = <-keyCh: default: } + if remotePubKey == nil { + return nil, errors.New("go-libp2p-quic-transport BUG: expected remote pub key to be set") + } return &conn{ sess: sess, From d1193da2f8ddcf39cff50104310b519090c06b36 Mon Sep 17 00:00:00 2001 From: lnykww Date: Thu, 21 Mar 2019 20:13:28 +0800 Subject: [PATCH 1414/3965] reuse listening connections for dialing --- p2p/transport/quic/listener.go | 27 ++---- p2p/transport/quic/reuse.go | 137 +++++++++++++++++++++++++++++++ p2p/transport/quic/reuse_test.go | 79 ++++++++++++++++++ p2p/transport/quic/transport.go | 82 +++++++++++------- 4 files changed, 275 insertions(+), 50 deletions(-) create mode 100644 p2p/transport/quic/reuse.go create mode 100644 p2p/transport/quic/reuse_test.go diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 0d1b835326..acc76f1b86 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -12,14 +12,13 @@ import ( quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" ) // A listener listens for QUIC connections. type listener struct { - quicListener quic.Listener - transport tpt.Transport - + quicListener quic.Listener + conn *reuseConn + transport *transport privKey ic.PrivKey localPeer peer.ID localMultiaddr ma.Multiaddr @@ -27,7 +26,7 @@ type listener struct { var _ tpt.Listener = &listener{} -func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, key ic.PrivKey, identity *p2ptls.Identity) (tpt.Listener, error) { +func newListener(rconn *reuseConn, t *transport, localPeer peer.ID, key ic.PrivKey, identity *p2ptls.Identity) (tpt.Listener, error) { var tlsConf tls.Config tlsConf.GetConfigForClient = func(_ *tls.ClientHelloInfo) (*tls.Config, error) { // return a tls.Config that verifies the peer's certificate chain. @@ -37,19 +36,7 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, conf, _ := identity.ConfigForAny() return conf, nil } - lnet, host, err := manet.DialArgs(addr) - if err != nil { - return nil, err - } - laddr, err := net.ResolveUDPAddr(lnet, host) - if err != nil { - return nil, err - } - conn, err := net.ListenUDP(lnet, laddr) - if err != nil { - return nil, err - } - ln, err := quic.Listen(conn, &tlsConf, quicConfig) + ln, err := quic.Listen(rconn, &tlsConf, quicConfig) if err != nil { return nil, err } @@ -58,8 +45,9 @@ func newListener(addr ma.Multiaddr, transport tpt.Transport, localPeer peer.ID, return nil, err } return &listener{ + conn: rconn, quicListener: ln, - transport: transport, + transport: t, privKey: key, localPeer: localPeer, localMultiaddr: localMultiaddr, @@ -113,6 +101,7 @@ func (l *listener) setupConn(sess quic.Session) (tpt.CapableConn, error) { // Close closes the listener. func (l *listener) Close() error { + l.conn.DecreaseCount() return l.quicListener.Close() } diff --git a/p2p/transport/quic/reuse.go b/p2p/transport/quic/reuse.go new file mode 100644 index 0000000000..56aba1b491 --- /dev/null +++ b/p2p/transport/quic/reuse.go @@ -0,0 +1,137 @@ +package libp2pquic + +import ( + "net" + "sync" + "sync/atomic" + + "github.com/vishvananda/netlink" +) + +type reuseConn struct { + net.PacketConn + refCount int32 // to be used as an atomic +} + +func newReuseConn(conn net.PacketConn) *reuseConn { + return &reuseConn{PacketConn: conn} +} + +func (c *reuseConn) IncreaseCount() { atomic.AddInt32(&c.refCount, 1) } +func (c *reuseConn) DecreaseCount() { atomic.AddInt32(&c.refCount, -1) } +func (c *reuseConn) GetCount() int { return int(atomic.LoadInt32(&c.refCount)) } + +type reuse struct { + mutex sync.Mutex + + unicast map[string] /* IP.String() */ map[int] /* port */ *reuseConn + // global contains connections that are listening on 0.0.0.0 / :: + global map[int]*reuseConn +} + +func newReuse() *reuse { + return &reuse{ + unicast: make(map[string]map[int]*reuseConn), + global: make(map[int]*reuseConn), + } +} + +func (r *reuse) getSourceIPs(network string, raddr *net.UDPAddr) ([]net.IP, error) { + // Determine the source address that the kernel would use for this IP address. + // Note: This only works on Linux. + // On other OSes, this will return a netlink.ErrNotImplemetned. + routes, err := (&netlink.Handle{}).RouteGet(raddr.IP) + if err != nil { + return nil, err + } + + ips := make([]net.IP, 0, len(routes)) + for _, route := range routes { + ips = append(ips, route.Src) + } + return ips, nil +} + +func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { + ips, err := r.getSourceIPs(network, raddr) + if err != nil && err != netlink.ErrNotImplemented { + return nil, err + } + + r.mutex.Lock() + defer r.mutex.Unlock() + + conn, err := r.dialLocked(network, raddr, ips) + if err != nil { + return nil, err + } + conn.IncreaseCount() + return conn, nil +} + +func (r *reuse) dialLocked(network string, raddr *net.UDPAddr, ips []net.IP) (*reuseConn, error) { + for _, ip := range ips { + // We already have at least one suitable connection... + if conns, ok := r.unicast[ip.String()]; ok { + // ... we don't care which port we're dialing from. Just use the first. + for _, c := range conns { + return c, nil + } + } + } + + // Use a connection listening on 0.0.0.0 (or ::). + // Again, we don't care about the port number. + for _, conn := range r.global { + return conn, nil + } + + // We don't have a connection that we can use for dialing. + // Dial a new connection from a random port. + var addr *net.UDPAddr + switch network { + case "udp4": + addr = &net.UDPAddr{IP: net.IPv4zero, Port: 0} + case "udp6": + addr = &net.UDPAddr{IP: net.IPv6zero, Port: 0} + } + conn, err := net.ListenUDP(network, addr) + if err != nil { + return nil, err + } + rconn := newReuseConn(conn) + r.global[conn.LocalAddr().(*net.UDPAddr).Port] = rconn + return rconn, nil +} + +func (r *reuse) Listen(network string, laddr *net.UDPAddr) (*reuseConn, error) { + conn, err := net.ListenUDP(network, laddr) + if err != nil { + return nil, err + } + localAddr := conn.LocalAddr().(*net.UDPAddr) + + rconn := newReuseConn(conn) + rconn.IncreaseCount() + + r.mutex.Lock() + defer r.mutex.Unlock() + + // Deal with listen on a global address + if laddr.IP.IsUnspecified() { + // The kernel already checked that the laddr is not already listen + // so we need not check here (when we create ListenUDP). + r.global[laddr.Port] = rconn + return rconn, err + } + + // Deal with listen on a unicast address + if _, ok := r.unicast[localAddr.IP.String()]; !ok { + r.unicast[laddr.IP.String()] = make(map[int]*reuseConn) + } + + // The kernel already checked that the laddr is not already listen + // so we need not check here (when we create ListenUDP). + r.unicast[laddr.IP.String()][localAddr.Port] = rconn + return rconn, err +} diff --git a/p2p/transport/quic/reuse_test.go b/p2p/transport/quic/reuse_test.go new file mode 100644 index 0000000000..2259a39195 --- /dev/null +++ b/p2p/transport/quic/reuse_test.go @@ -0,0 +1,79 @@ +package libp2pquic + +import ( + "net" + "runtime" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +var _ = Describe("Reuse", func() { + var reuse *reuse + + BeforeEach(func() { + reuse = newReuse() + }) + + It("creates a new global connection when listening on 0.0.0.0", func() { + addr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") + Expect(err).ToNot(HaveOccurred()) + conn, err := reuse.Listen("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(1)) + }) + + It("creates a new global connection when listening on [::]", func() { + addr, err := net.ResolveUDPAddr("udp6", "[::]:1234") + Expect(err).ToNot(HaveOccurred()) + conn, err := reuse.Listen("udp6", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(1)) + }) + + It("creates a new global connection when dialing", func() { + addr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") + Expect(err).ToNot(HaveOccurred()) + conn, err := reuse.Dial("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(1)) + laddr := conn.LocalAddr().(*net.UDPAddr) + Expect(laddr.IP.String()).To(Equal("0.0.0.0")) + Expect(laddr.Port).ToNot(BeZero()) + }) + + It("reuses a connection it created for listening when dialing", func() { + // listen + addr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") + Expect(err).ToNot(HaveOccurred()) + lconn, err := reuse.Listen("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(lconn.GetCount()).To(Equal(1)) + // dial + raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") + Expect(err).ToNot(HaveOccurred()) + conn, err := reuse.Dial("udp4", raddr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(2)) + }) + + if runtime.GOOS == "linux" { + It("reuses a connection it created for listening on a specific interface", func() { + raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") + Expect(err).ToNot(HaveOccurred()) + ips, err := reuse.getSourceIPs("udp4", raddr) + Expect(err).ToNot(HaveOccurred()) + Expect(ips).ToNot(BeEmpty()) + // listen + addr, err := net.ResolveUDPAddr("udp4", ips[0].String()+":0") + Expect(err).ToNot(HaveOccurred()) + lconn, err := reuse.Listen("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(lconn.GetCount()).To(Equal(1)) + // dial + conn, err := reuse.Dial("udp4", raddr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(2)) + }) + } +}) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 5b217087bb..bd0adfc714 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -3,9 +3,7 @@ package libp2pquic import ( "context" "errors" - "fmt" "net" - "sync" ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" @@ -31,42 +29,42 @@ var quicConfig = &quic.Config{ } type connManager struct { - mutex sync.Mutex - - connIPv4 net.PacketConn - connIPv6 net.PacketConn + reuseUDP4 *reuse + reuseUDP6 *reuse } -func (c *connManager) GetConnForAddr(network string) (net.PacketConn, error) { - c.mutex.Lock() - defer c.mutex.Unlock() +func newConnManager() *connManager { + return &connManager{ + reuseUDP4: newReuse(), + reuseUDP6: newReuse(), + } +} +func (c *connManager) getReuse(network string) (*reuse, error) { switch network { case "udp4": - if c.connIPv4 != nil { - return c.connIPv4, nil - } - var err error - c.connIPv4, err = c.createConn(network, "0.0.0.0:0") - return c.connIPv4, err + return c.reuseUDP4, nil case "udp6": - if c.connIPv6 != nil { - return c.connIPv6, nil - } - var err error - c.connIPv6, err = c.createConn(network, ":0") - return c.connIPv6, err + return c.reuseUDP6, nil default: - return nil, fmt.Errorf("unsupported network: %s", network) + return nil, errors.New("invalid network: must be either udp4 or udp6") + } +} + +func (c *connManager) Listen(network string, laddr *net.UDPAddr) (*reuseConn, error) { + reuse, err := c.getReuse(network) + if err != nil { + return nil, err } + return reuse.Listen(network, laddr) } -func (c *connManager) createConn(network, host string) (net.PacketConn, error) { - addr, err := net.ResolveUDPAddr(network, host) +func (c *connManager) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { + reuse, err := c.getReuse(network) if err != nil { return nil, err } - return net.ListenUDP(network, addr) + return reuse.Dial(network, raddr) } // The Transport implements the tpt.Transport interface for QUIC connections. @@ -94,7 +92,7 @@ func NewTransport(key ic.PrivKey) (tpt.Transport, error) { privKey: key, localPeer: localPeer, identity: identity, - connManager: &connManager{}, + connManager: newConnManager(), }, nil } @@ -104,7 +102,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - pconn, err := t.connManager.GetConnForAddr(network) + udpAddr, err := net.ResolveUDPAddr(network, host) if err != nil { return nil, err } @@ -113,15 +111,15 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp return nil, err } tlsConf, keyCh := t.identity.ConfigForPeer(p) - sess, err := quic.DialContext(ctx, pconn, addr, host, tlsConf, quicConfig) + pconn, err := t.connManager.Dial(network, udpAddr) if err != nil { return nil, err } - localMultiaddr, err := toQuicMultiaddr(sess.LocalAddr()) + sess, err := quic.DialContext(ctx, pconn, addr, host, tlsConf, quicConfig) if err != nil { + pconn.DecreaseCount() return nil, err } - // Should be ready by this point, don't block. var remotePubKey ic.PubKey select { @@ -129,9 +127,19 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp default: } if remotePubKey == nil { + pconn.DecreaseCount() return nil, errors.New("go-libp2p-quic-transport BUG: expected remote pub key to be set") } + go func() { + <-sess.Context().Done() + pconn.DecreaseCount() + }() + localMultiaddr, err := toQuicMultiaddr(pconn.LocalAddr()) + if err != nil { + pconn.DecreaseCount() + return nil, err + } return &conn{ sess: sess, transport: t, @@ -151,7 +159,19 @@ func (t *transport) CanDial(addr ma.Multiaddr) bool { // Listen listens for new QUIC connections on the passed multiaddr. func (t *transport) Listen(addr ma.Multiaddr) (tpt.Listener, error) { - return newListener(addr, t, t.localPeer, t.privKey, t.identity) + lnet, host, err := manet.DialArgs(addr) + if err != nil { + return nil, err + } + laddr, err := net.ResolveUDPAddr(lnet, host) + if err != nil { + return nil, err + } + conn, err := t.connManager.Listen(lnet, laddr) + if err != nil { + return nil, err + } + return newListener(conn, t, t.localPeer, t.privKey, t.identity) } // Proxy returns true if this transport proxies. From 46775f0070554d7619a87ca3d6bdf75e91a4521d Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 5 Aug 2019 18:12:51 +0700 Subject: [PATCH 1415/3965] use a single handle for each reuse --- p2p/transport/quic/reuse.go | 28 +++++++++++++++++++++------- p2p/transport/quic/reuse_test.go | 4 +++- p2p/transport/quic/transport.go | 22 +++++++++++++++++----- 3 files changed, 41 insertions(+), 13 deletions(-) diff --git a/p2p/transport/quic/reuse.go b/p2p/transport/quic/reuse.go index 56aba1b491..08fc14f3d7 100644 --- a/p2p/transport/quic/reuse.go +++ b/p2p/transport/quic/reuse.go @@ -24,23 +24,37 @@ func (c *reuseConn) GetCount() int { return int(atomic.LoadInt32(&c.refCount)) type reuse struct { mutex sync.Mutex + handle *netlink.Handle // Only set on Linux. nil on other systems. + unicast map[string] /* IP.String() */ map[int] /* port */ *reuseConn // global contains connections that are listening on 0.0.0.0 / :: global map[int]*reuseConn } -func newReuse() *reuse { +func newReuse() (*reuse, error) { + // On non-Linux systems, this will return ErrNotImplemented. + handle, err := netlink.NewHandle() + if err == netlink.ErrNotImplemented { + handle = nil + } else if err != nil { + return nil, err + } return &reuse{ unicast: make(map[string]map[int]*reuseConn), global: make(map[int]*reuseConn), - } + handle: handle, + }, nil } +// Get the source IP that the kernel would use for dialing. +// This only works on Linux. +// On other systems, this returns an empty slice of IP addresses. func (r *reuse) getSourceIPs(network string, raddr *net.UDPAddr) ([]net.IP, error) { - // Determine the source address that the kernel would use for this IP address. - // Note: This only works on Linux. - // On other OSes, this will return a netlink.ErrNotImplemetned. - routes, err := (&netlink.Handle{}).RouteGet(raddr.IP) + if r.handle == nil { + return nil, nil + } + + routes, err := r.handle.RouteGet(raddr.IP) if err != nil { return nil, err } @@ -54,7 +68,7 @@ func (r *reuse) getSourceIPs(network string, raddr *net.UDPAddr) ([]net.IP, erro func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { ips, err := r.getSourceIPs(network, raddr) - if err != nil && err != netlink.ErrNotImplemented { + if err != nil { return nil, err } diff --git a/p2p/transport/quic/reuse_test.go b/p2p/transport/quic/reuse_test.go index 2259a39195..91668b0f3b 100644 --- a/p2p/transport/quic/reuse_test.go +++ b/p2p/transport/quic/reuse_test.go @@ -12,7 +12,9 @@ var _ = Describe("Reuse", func() { var reuse *reuse BeforeEach(func() { - reuse = newReuse() + var err error + reuse, err = newReuse() + Expect(err).ToNot(HaveOccurred()) }) It("creates a new global connection when listening on 0.0.0.0", func() { diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index bd0adfc714..a9ddd41355 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -33,11 +33,19 @@ type connManager struct { reuseUDP6 *reuse } -func newConnManager() *connManager { - return &connManager{ - reuseUDP4: newReuse(), - reuseUDP6: newReuse(), +func newConnManager() (*connManager, error) { + reuseUDP4, err := newReuse() + if err != nil { + return nil, err + } + reuseUDP6, err := newReuse() + if err != nil { + return nil, err } + return &connManager{ + reuseUDP4: reuseUDP4, + reuseUDP6: reuseUDP6, + }, nil } func (c *connManager) getReuse(network string) (*reuse, error) { @@ -87,12 +95,16 @@ func NewTransport(key ic.PrivKey) (tpt.Transport, error) { if err != nil { return nil, err } + connManager, err := newConnManager() + if err != nil { + return nil, err + } return &transport{ privKey: key, localPeer: localPeer, identity: identity, - connManager: newConnManager(), + connManager: connManager, }, nil } From 1be6b00e11edb70756e08e89371d3b7c79054476 Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 6 Aug 2019 11:18:41 -0400 Subject: [PATCH 1416/3965] Add note about panic to KeyStretcher comments --- core/crypto/key.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/core/crypto/key.go b/core/crypto/key.go index 2706a7a3e7..310fd475c7 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -180,7 +180,11 @@ type StretchedKeys struct { } // KeyStretcher returns a set of keys for each party by stretching the shared key. -// (myIV, theirIV, myCipherKey, theirCipherKey, myMACKey, theirMACKey) +// (myIV, theirIV, myCipherKey, theirCipherKey, myMACKey, theirMACKey). +// This function accepts the following cipher types: +// - AES-128 +// - AES-256 +// The function will panic upon receiving an unknown cipherType func KeyStretcher(cipherType string, hashType string, secret []byte) (StretchedKeys, StretchedKeys) { var cipherKeySize int var ivSize int From 876a0fc72f101286bc96824fe3a725ce5f79a22d Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 6 Aug 2019 12:15:40 -0400 Subject: [PATCH 1417/3965] Add test for panic on unknown cipher type --- core/crypto/key_test.go | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index 2a5e89a8c2..882e4be210 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -145,3 +145,23 @@ func (pk testkey) Raw() ([]byte, error) { func (pk testkey) Equals(k Key) bool { return KeyEqual(pk, k) } + +func TestPanicOnUnknownCipherType(t *testing.T) { + passed := false + defer func() { + if !passed { + t.Fatal("expected known cipher and hash to succeed") + } + err := recover() + errStr, ok := err.(string) + if !ok { + t.Fatal("expected string in panic") + } + if errStr != "Unrecognized cipher, programmer error?" { + t.Fatal("expected \"Unrecognized cipher, programmer error?\"") + } + }() + KeyStretcher("AES-256", "SHA1", []byte("foo")) + passed = true + KeyStretcher("Fooba", "SHA1", []byte("foo")) +} From 4527fe046d9d57feca4da0e6001ce17e832e5a0c Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Tue, 30 Jul 2019 17:00:04 -0400 Subject: [PATCH 1418/3965] Return error in Emit --- core/event/bus.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/event/bus.go b/core/event/bus.go index 028bde7b9c..9dcf937822 100644 --- a/core/event/bus.go +++ b/core/event/bus.go @@ -19,7 +19,7 @@ type Emitter interface { // calls to Emit will block. // // Calling this function with wrong event type will cause a panic. - Emit(evt interface{}) + Emit(evt interface{}) error } // Subscription represents a subscription to one or multiple event types. From ff46be45996a42de47c009999874ddea0d0de6d5 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 6 Aug 2019 18:20:09 +0700 Subject: [PATCH 1419/3965] fix saving of listening connections when listening on IP:0 --- p2p/transport/quic/reuse.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/transport/quic/reuse.go b/p2p/transport/quic/reuse.go index 08fc14f3d7..47a1ffb2c9 100644 --- a/p2p/transport/quic/reuse.go +++ b/p2p/transport/quic/reuse.go @@ -132,20 +132,20 @@ func (r *reuse) Listen(network string, laddr *net.UDPAddr) (*reuseConn, error) { defer r.mutex.Unlock() // Deal with listen on a global address - if laddr.IP.IsUnspecified() { + if localAddr.IP.IsUnspecified() { // The kernel already checked that the laddr is not already listen // so we need not check here (when we create ListenUDP). - r.global[laddr.Port] = rconn + r.global[localAddr.Port] = rconn return rconn, err } // Deal with listen on a unicast address if _, ok := r.unicast[localAddr.IP.String()]; !ok { - r.unicast[laddr.IP.String()] = make(map[int]*reuseConn) + r.unicast[localAddr.IP.String()] = make(map[int]*reuseConn) } // The kernel already checked that the laddr is not already listen // so we need not check here (when we create ListenUDP). - r.unicast[laddr.IP.String()][localAddr.Port] = rconn + r.unicast[localAddr.IP.String()][localAddr.Port] = rconn return rconn, err } From d73a51776e95c501a42166471225ccccdbad7e30 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 12 Aug 2019 18:45:12 -0700 Subject: [PATCH 1420/3965] new: allow overriding the user-agent with an option Instead of using a global variable. This also: * Adds an option to the identify service to set the user agent. * Removes the ability to pass an identify service to NewHost as any reasonable Identify service already needs to be constructed with an instance of the host. --- config/config.go | 3 +++ options.go | 8 +++++++ p2p/host/basic/basic_host.go | 19 ++++++++-------- p2p/protocol/identify/id.go | 21 ++++++++++++++---- p2p/protocol/identify/id_test.go | 38 ++++++++++++++++++++++++++++++++ p2p/protocol/identify/opts.go | 13 +++++++++++ 6 files changed, 88 insertions(+), 14 deletions(-) create mode 100644 p2p/protocol/identify/opts.go diff --git a/config/config.go b/config/config.go index 3bb51bf1f7..c37dbc61c6 100644 --- a/config/config.go +++ b/config/config.go @@ -44,6 +44,8 @@ type RoutingC func(host.Host) (routing.PeerRouting, error) // This is *not* a stable interface. Use the options defined in the root // package. type Config struct { + UserAgent string + PeerKey crypto.PrivKey Transports []TptC @@ -120,6 +122,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { AddrsFactory: cfg.AddrsFactory, NATManager: cfg.NATManager, EnablePing: !cfg.DisablePing, + UserAgent: cfg.UserAgent, }) if err != nil { diff --git a/options.go b/options.go index 6d310f1aa3..c5ef639ebe 100644 --- a/options.go +++ b/options.go @@ -319,3 +319,11 @@ var NoTransports = func(cfg *Config) error { cfg.Transports = []config.TptC{} return nil } + +// UserAgent sets the libp2p user-agent sent along with the identify protocol +func UserAgent(userAgent string) Option { + return func(cfg *Config) error { + cfg.UserAgent = userAgent + return nil + } +} diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 2d965bdae1..92946c5d7c 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -100,10 +100,6 @@ type HostOpts struct { // If below 0, timeouts on streams will be deactivated. NegotiationTimeout time.Duration - // IdentifyService holds an implementation of the /ipfs/id/ protocol. - // If omitted, a new *identify.IDService will be used. - IdentifyService *identify.IDService - // AddrsFactory holds a function which can be used to override or filter the result of Addrs. // If omitted, there's no override or filtering, and the results of Addrs and AllAddrs are the same. AddrsFactory AddrsFactory @@ -121,6 +117,9 @@ type HostOpts struct { // EnablePing indicates whether to instantiate the ping service EnablePing bool + + // UserAgent sets the user-agent for the host. Defaults to ClientVersion. + UserAgent string } // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. @@ -154,12 +153,12 @@ func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHo h.mux = opts.MultistreamMuxer } - if opts.IdentifyService != nil { - h.ids = opts.IdentifyService - } else { - // we can't set this as a default above because it depends on the *BasicHost. - h.ids = identify.NewIDService(goprocessctx.WithProcessClosing(ctx, h.proc), h) - } + // we can't set this as a default above because it depends on the *BasicHost. + h.ids = identify.NewIDService( + goprocessctx.WithProcessClosing(ctx, h.proc), + h, + identify.UserAgent(opts.UserAgent), + ) if uint64(opts.NegotiationTimeout) != 0 { h.negtimeout = opts.NegotiationTimeout diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 1fa2bcf1f6..39390d1f9d 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -47,7 +47,8 @@ const transientTTL = 10 * time.Second // * Our IPFS Agent Version // * Our public Listen Addresses type IDService struct { - Host host.Host + Host host.Host + UserAgent string ctx context.Context @@ -70,9 +71,21 @@ type IDService struct { // NewIDService constructs a new *IDService and activates it by // attaching its stream handler to the given host.Host. -func NewIDService(ctx context.Context, h host.Host) *IDService { +func NewIDService(ctx context.Context, h host.Host, opts ...Option) *IDService { + var cfg config + for _, opt := range opts { + opt(&cfg) + } + + userAgent := ClientVersion + if cfg.userAgent != "" { + userAgent = cfg.userAgent + } + s := &IDService{ - Host: h, + Host: h, + UserAgent: userAgent, + ctx: ctx, currid: make(map[network.Conn]chan struct{}), observedAddrs: NewObservedAddrSet(ctx), @@ -306,7 +319,7 @@ func (ids *IDService) populateMessage(mes *pb.Identify, c network.Conn) { // set protocol versions pv := LibP2PVersion - av := ClientVersion + av := ids.UserAgent mes.ProtocolVersion = &pv mes.AgentVersion = &av } diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index dedf9a094e..5c422a9092 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -8,6 +8,7 @@ import ( "time" "github.com/libp2p/go-eventbus" + libp2p "github.com/libp2p/go-libp2p" ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/helpers" @@ -367,3 +368,40 @@ func TestIdentifyDeltaWhileIdentifyingConn(t *testing.T) { t.Fatalf("timed out while waiting for an event for the protocol changes in h2") } } + +func TestUserAgent(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + h1, err := libp2p.New( + ctx, + libp2p.UserAgent("foo"), + libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"), + ) + if err != nil { + t.Fatal(err) + } + defer h1.Close() + + h2, err := libp2p.New( + ctx, + libp2p.UserAgent("bar"), + libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"), + ) + if err != nil { + t.Fatal(err) + } + defer h2.Close() + + err = h1.Connect(ctx, peer.AddrInfo{ID: h2.ID(), Addrs: h2.Addrs()}) + if err != nil { + t.Fatal(err) + } + av, err := h1.Peerstore().Get(h2.ID(), "AgentVersion") + if err != nil { + t.Fatal(err) + } + if ver, ok := av.(string); !ok || ver != "bar" { + t.Errorf("expected agent version %q, got %q", "bar", av) + } +} diff --git a/p2p/protocol/identify/opts.go b/p2p/protocol/identify/opts.go new file mode 100644 index 0000000000..94be5b786c --- /dev/null +++ b/p2p/protocol/identify/opts.go @@ -0,0 +1,13 @@ +package identify + +type config struct { + userAgent string +} + +type Option func(*config) + +func UserAgent(ua string) Option { + return func(cfg *config) { + cfg.userAgent = ua + } +} From e337633824fc54dae3bb4b208a89de85c79e9b4f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 12 Aug 2019 18:46:51 -0700 Subject: [PATCH 1421/3965] set a default client-version using go modules This should help us improve network stats on who's using libp2p. --- p2p/protocol/identify/id.go | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 39390d1f9d..35e2e57ad8 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -2,6 +2,8 @@ package identify import ( "context" + "fmt" + "runtime/debug" "sync" "time" @@ -31,9 +33,27 @@ const ID = "/ipfs/id/1.0.0" // LibP2PVersion holds the current protocol version for a client running this code // TODO(jbenet): fix the versioning mess. +// XXX: Don't change this till 2020. You'll break all go-ipfs versions prior to +// 0.4.17 which asserted an exact version match. const LibP2PVersion = "ipfs/0.1.0" -var ClientVersion = "go-libp2p/3.3.4" +// ClientVersion is the default user agent. +// +// Deprecated: Set this with the UserAgent option. +var ClientVersion = "github.com/libp2p/go-libp2p" + +func init() { + bi, ok := debug.ReadBuildInfo() + if !ok { + return + } + version := bi.Main.Version + if version == "(devel)" { + ClientVersion = bi.Main.Path + } else { + ClientVersion = fmt.Sprintf("%s@%s", bi.Main.Path, bi.Main.Version) + } +} // transientTTL is a short ttl for invalidated previously connected addrs const transientTTL = 10 * time.Second From 19dcc590372622c3128d5f6424e88b2a7a7ec626 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 6 Aug 2019 18:48:51 +0700 Subject: [PATCH 1422/3965] delete reuse-connection if they aren't used for more than 10 seconds --- p2p/transport/quic/conn_test.go | 86 ++++++---- p2p/transport/quic/libp2pquic_suite_test.go | 27 ++++ p2p/transport/quic/listener.go | 2 +- p2p/transport/quic/listener_test.go | 3 + p2p/transport/quic/reuse.go | 86 +++++++++- p2p/transport/quic/reuse_test.go | 170 ++++++++++++++------ 6 files changed, 289 insertions(+), 85 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index ef2430783b..bf8d681149 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -48,21 +48,12 @@ var _ = Describe("Connection", func() { return id, priv } - runServer := func(tr tpt.Transport, multiaddr string) (ma.Multiaddr, <-chan tpt.CapableConn) { - addrChan := make(chan ma.Multiaddr) - connChan := make(chan tpt.CapableConn) - go func() { - defer GinkgoRecover() - addr, err := ma.NewMultiaddr(multiaddr) - Expect(err).ToNot(HaveOccurred()) - ln, err := tr.Listen(addr) - Expect(err).ToNot(HaveOccurred()) - addrChan <- ln.Multiaddr() - conn, err := ln.Accept() - Expect(err).ToNot(HaveOccurred()) - connChan <- conn - }() - return <-addrChan, connChan + runServer := func(tr tpt.Transport, multiaddr string) tpt.Listener { + addr, err := ma.NewMultiaddr(multiaddr) + Expect(err).ToNot(HaveOccurred()) + ln, err := tr.Listen(addr) + Expect(err).ToNot(HaveOccurred()) + return ln } BeforeEach(func() { @@ -73,13 +64,17 @@ var _ = Describe("Connection", func() { It("handshakes on IPv4", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + defer ln.Close() clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) - conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) + conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) + Expect(err).ToNot(HaveOccurred()) + defer conn.Close() + serverConn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) - serverConn := <-serverConnChan + defer serverConn.Close() Expect(conn.LocalPeer()).To(Equal(clientID)) Expect(conn.LocalPrivateKey()).To(Equal(clientKey)) Expect(conn.RemotePeer()).To(Equal(serverID)) @@ -93,13 +88,17 @@ var _ = Describe("Connection", func() { It("handshakes on IPv6", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport, "/ip6/::1/udp/0/quic") + ln := runServer(serverTransport, "/ip6/::1/udp/0/quic") + defer ln.Close() clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) - conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) + conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) - serverConn := <-serverConnChan + defer conn.Close() + serverConn, err := ln.Accept() + Expect(err).ToNot(HaveOccurred()) + defer serverConn.Close() Expect(conn.LocalPeer()).To(Equal(clientID)) Expect(conn.LocalPrivateKey()).To(Equal(clientKey)) Expect(conn.RemotePeer()).To(Equal(serverID)) @@ -113,13 +112,17 @@ var _ = Describe("Connection", func() { It("opens and accepts streams", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + defer ln.Close() clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) - conn, err := clientTransport.Dial(context.Background(), serverAddr, serverID) + conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) + Expect(err).ToNot(HaveOccurred()) + defer conn.Close() + serverConn, err := ln.Accept() Expect(err).ToNot(HaveOccurred()) - serverConn := <-serverConnChan + defer serverConn.Close() str, err := conn.OpenStream() Expect(err).ToNot(HaveOccurred()) @@ -138,15 +141,24 @@ var _ = Describe("Connection", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) // dial, but expect the wrong peer ID - _, err = clientTransport.Dial(context.Background(), serverAddr, thirdPartyID) + _, err = clientTransport.Dial(context.Background(), ln.Multiaddr(), thirdPartyID) Expect(err).To(HaveOccurred()) Expect(err.Error()).To(ContainSubstring("CRYPTO_ERROR")) - Consistently(serverConnChan).ShouldNot(Receive()) + + done := make(chan struct{}) + go func() { + defer GinkgoRecover() + defer close(done) + ln.Accept() + }() + Consistently(done).ShouldNot(BeClosed()) + ln.Close() + Eventually(done).Should(BeClosed()) }) It("dials to two servers at the same time", func() { @@ -154,16 +166,23 @@ var _ = Describe("Connection", func() { serverTransport, err := NewTransport(serverKey) Expect(err).ToNot(HaveOccurred()) - serverAddr, serverConnChan := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + ln1 := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") serverTransport2, err := NewTransport(serverKey2) + defer ln1.Close() Expect(err).ToNot(HaveOccurred()) - serverAddr2, serverConnChan2 := runServer(serverTransport2, "/ip4/127.0.0.1/udp/0/quic") + ln2 := runServer(serverTransport2, "/ip4/127.0.0.1/udp/0/quic") + defer ln2.Close() data := bytes.Repeat([]byte{'a'}, 5*1<<20) // 5 MB // wait for both servers to accept a connection // then send some data go func() { - for _, c := range []tpt.CapableConn{<-serverConnChan, <-serverConnChan2} { + serverConn1, err := ln1.Accept() + Expect(err).ToNot(HaveOccurred()) + serverConn2, err := ln2.Accept() + Expect(err).ToNot(HaveOccurred()) + + for _, c := range []tpt.CapableConn{serverConn1, serverConn2} { go func(conn tpt.CapableConn) { defer GinkgoRecover() str, err := conn.OpenStream() @@ -177,10 +196,12 @@ var _ = Describe("Connection", func() { clientTransport, err := NewTransport(clientKey) Expect(err).ToNot(HaveOccurred()) - c1, err := clientTransport.Dial(context.Background(), serverAddr, serverID) + c1, err := clientTransport.Dial(context.Background(), ln1.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) - c2, err := clientTransport.Dial(context.Background(), serverAddr2, serverID2) + defer c1.Close() + c2, err := clientTransport.Dial(context.Background(), ln2.Multiaddr(), serverID2) Expect(err).ToNot(HaveOccurred()) + defer c2.Close() done := make(chan struct{}, 2) // receive the data on both connections at the same time @@ -193,7 +214,6 @@ var _ = Describe("Connection", func() { d, err := ioutil.ReadAll(str) Expect(err).ToNot(HaveOccurred()) Expect(d).To(Equal(data)) - conn.Close() done <- struct{}{} }(c) } diff --git a/p2p/transport/quic/libp2pquic_suite_test.go b/p2p/transport/quic/libp2pquic_suite_test.go index 2bb5d416c6..ce48c3f619 100644 --- a/p2p/transport/quic/libp2pquic_suite_test.go +++ b/p2p/transport/quic/libp2pquic_suite_test.go @@ -1,7 +1,11 @@ package libp2pquic import ( + "bytes" mrand "math/rand" + "runtime/pprof" + "strings" + "time" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" @@ -17,3 +21,26 @@ func TestLibp2pQuicTransport(t *testing.T) { var _ = BeforeSuite(func() { mrand.Seed(GinkgoRandomSeed()) }) + +var garbageCollectIntervalOrig time.Duration +var maxUnusedDurationOrig time.Duration + +func isGarbageCollectorRunning() bool { + var b bytes.Buffer + pprof.Lookup("goroutine").WriteTo(&b, 1) + return strings.Contains(b.String(), "go-libp2p-quic-transport.(*reuse).runGarbageCollector") +} + +var _ = BeforeEach(func() { + Expect(isGarbageCollectorRunning()).To(BeFalse()) + garbageCollectIntervalOrig = garbageCollectInterval + maxUnusedDurationOrig = maxUnusedDuration + garbageCollectInterval = 50 * time.Millisecond + maxUnusedDuration = 0 +}) + +var _ = AfterEach(func() { + Eventually(isGarbageCollectorRunning).Should(BeFalse()) + garbageCollectInterval = garbageCollectIntervalOrig + maxUnusedDuration = maxUnusedDurationOrig +}) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index acc76f1b86..ed2fb731c3 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -101,7 +101,7 @@ func (l *listener) setupConn(sess quic.Session) (tpt.CapableConn, error) { // Close closes the listener. func (l *listener) Close() error { - l.conn.DecreaseCount() + defer l.conn.DecreaseCount() return l.quicListener.Close() } diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index 44d1c4ab17..e90debc810 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -33,6 +33,7 @@ var _ = Describe("Listener", func() { Expect(err).ToNot(HaveOccurred()) ln, err := t.Listen(localAddr) Expect(err).ToNot(HaveOccurred()) + defer ln.Close() netAddr := ln.Addr() Expect(netAddr).To(BeAssignableToTypeOf(&net.UDPAddr{})) port := netAddr.(*net.UDPAddr).Port @@ -45,6 +46,7 @@ var _ = Describe("Listener", func() { Expect(err).ToNot(HaveOccurred()) ln, err := t.Listen(localAddr) Expect(err).ToNot(HaveOccurred()) + defer ln.Close() netAddr := ln.Addr() Expect(netAddr).To(BeAssignableToTypeOf(&net.UDPAddr{})) port := netAddr.(*net.UDPAddr).Port @@ -57,6 +59,7 @@ var _ = Describe("Listener", func() { Expect(err).ToNot(HaveOccurred()) ln, err := t.Listen(localAddr) Expect(err).ToNot(HaveOccurred()) + defer ln.Close() netAddr := ln.Addr() Expect(netAddr).To(BeAssignableToTypeOf(&net.UDPAddr{})) port := netAddr.(*net.UDPAddr).Port diff --git a/p2p/transport/quic/reuse.go b/p2p/transport/quic/reuse.go index 47a1ffb2c9..2903b68384 100644 --- a/p2p/transport/quic/reuse.go +++ b/p2p/transport/quic/reuse.go @@ -3,27 +3,56 @@ package libp2pquic import ( "net" "sync" - "sync/atomic" + "time" "github.com/vishvananda/netlink" ) +// Constants. Defined as variables to simplify testing. +var ( + garbageCollectInterval = 30 * time.Second + maxUnusedDuration = 10 * time.Second +) + type reuseConn struct { net.PacketConn - refCount int32 // to be used as an atomic + + mutex sync.Mutex + refCount int + unusedSince time.Time } func newReuseConn(conn net.PacketConn) *reuseConn { return &reuseConn{PacketConn: conn} } -func (c *reuseConn) IncreaseCount() { atomic.AddInt32(&c.refCount, 1) } -func (c *reuseConn) DecreaseCount() { atomic.AddInt32(&c.refCount, -1) } -func (c *reuseConn) GetCount() int { return int(atomic.LoadInt32(&c.refCount)) } +func (c *reuseConn) IncreaseCount() { + c.mutex.Lock() + c.refCount++ + c.unusedSince = time.Time{} + c.mutex.Unlock() +} + +func (c *reuseConn) DecreaseCount() { + c.mutex.Lock() + c.refCount-- + if c.refCount == 0 { + c.unusedSince = time.Now() + } + c.mutex.Unlock() +} + +func (c *reuseConn) ShouldGarbageCollect(now time.Time) bool { + c.mutex.Lock() + defer c.mutex.Unlock() + return !c.unusedSince.IsZero() && c.unusedSince.Add(maxUnusedDuration).Before(now) +} type reuse struct { mutex sync.Mutex + garbageCollectorRunning bool + handle *netlink.Handle // Only set on Linux. nil on other systems. unicast map[string] /* IP.String() */ map[int] /* port */ *reuseConn @@ -46,6 +75,50 @@ func newReuse() (*reuse, error) { }, nil } +func (r *reuse) runGarbageCollector() { + ticker := time.NewTicker(garbageCollectInterval) + defer ticker.Stop() + + for now := range ticker.C { + var shouldExit bool + r.mutex.Lock() + for key, conn := range r.global { + if conn.ShouldGarbageCollect(now) { + delete(r.global, key) + } + } + for ukey, conns := range r.unicast { + for key, conn := range conns { + if conn.ShouldGarbageCollect(now) { + delete(conns, key) + } + } + if len(conns) == 0 { + delete(r.unicast, ukey) + } + } + + // stop the garbage collector if we're not tracking any connections + if len(r.global) == 0 && len(r.unicast) == 0 { + r.garbageCollectorRunning = false + shouldExit = true + } + r.mutex.Unlock() + + if shouldExit { + return + } + } +} + +// must be called while holding the mutex +func (r *reuse) maybeStartGarbageCollector() { + if !r.garbageCollectorRunning { + r.garbageCollectorRunning = true + go r.runGarbageCollector() + } +} + // Get the source IP that the kernel would use for dialing. // This only works on Linux. // On other systems, this returns an empty slice of IP addresses. @@ -80,6 +153,7 @@ func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { return nil, err } conn.IncreaseCount() + r.maybeStartGarbageCollector() return conn, nil } @@ -131,6 +205,8 @@ func (r *reuse) Listen(network string, laddr *net.UDPAddr) (*reuseConn, error) { r.mutex.Lock() defer r.mutex.Unlock() + r.maybeStartGarbageCollector() + // Deal with listen on a global address if localAddr.IP.IsUnspecified() { // The kernel already checked that the laddr is not already listen diff --git a/p2p/transport/quic/reuse_test.go b/p2p/transport/quic/reuse_test.go index 91668b0f3b..276fd5bd95 100644 --- a/p2p/transport/quic/reuse_test.go +++ b/p2p/transport/quic/reuse_test.go @@ -3,11 +3,18 @@ package libp2pquic import ( "net" "runtime" + "time" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) +func (c *reuseConn) GetCount() int { + c.mutex.Lock() + defer c.mutex.Unlock() + return c.refCount +} + var _ = Describe("Reuse", func() { var reuse *reuse @@ -17,65 +24,136 @@ var _ = Describe("Reuse", func() { Expect(err).ToNot(HaveOccurred()) }) - It("creates a new global connection when listening on 0.0.0.0", func() { - addr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") - Expect(err).ToNot(HaveOccurred()) - conn, err := reuse.Listen("udp4", addr) - Expect(err).ToNot(HaveOccurred()) - Expect(conn.GetCount()).To(Equal(1)) - }) - - It("creates a new global connection when listening on [::]", func() { - addr, err := net.ResolveUDPAddr("udp6", "[::]:1234") - Expect(err).ToNot(HaveOccurred()) - conn, err := reuse.Listen("udp6", addr) - Expect(err).ToNot(HaveOccurred()) - Expect(conn.GetCount()).To(Equal(1)) - }) + Context("creating and reusing connections", func() { + AfterEach(func() { + reuse.mutex.Lock() + for _, conn := range reuse.global { + for conn.GetCount() > 0 { + conn.DecreaseCount() + } + } + for _, conns := range reuse.unicast { + for _, conn := range conns { + for conn.GetCount() > 0 { + conn.DecreaseCount() + } + } + } + reuse.mutex.Unlock() + Eventually(isGarbageCollectorRunning).Should(BeFalse()) + }) - It("creates a new global connection when dialing", func() { - addr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") - Expect(err).ToNot(HaveOccurred()) - conn, err := reuse.Dial("udp4", addr) - Expect(err).ToNot(HaveOccurred()) - Expect(conn.GetCount()).To(Equal(1)) - laddr := conn.LocalAddr().(*net.UDPAddr) - Expect(laddr.IP.String()).To(Equal("0.0.0.0")) - Expect(laddr.Port).ToNot(BeZero()) - }) + It("creates a new global connection when listening on 0.0.0.0", func() { + addr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") + Expect(err).ToNot(HaveOccurred()) + conn, err := reuse.Listen("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(1)) + }) - It("reuses a connection it created for listening when dialing", func() { - // listen - addr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") - Expect(err).ToNot(HaveOccurred()) - lconn, err := reuse.Listen("udp4", addr) - Expect(err).ToNot(HaveOccurred()) - Expect(lconn.GetCount()).To(Equal(1)) - // dial - raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") - Expect(err).ToNot(HaveOccurred()) - conn, err := reuse.Dial("udp4", raddr) - Expect(err).ToNot(HaveOccurred()) - Expect(conn.GetCount()).To(Equal(2)) - }) + It("creates a new global connection when listening on [::]", func() { + addr, err := net.ResolveUDPAddr("udp6", "[::]:1234") + Expect(err).ToNot(HaveOccurred()) + conn, err := reuse.Listen("udp6", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(1)) + }) - if runtime.GOOS == "linux" { - It("reuses a connection it created for listening on a specific interface", func() { - raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") + It("creates a new global connection when dialing", func() { + addr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") Expect(err).ToNot(HaveOccurred()) - ips, err := reuse.getSourceIPs("udp4", raddr) + conn, err := reuse.Dial("udp4", addr) Expect(err).ToNot(HaveOccurred()) - Expect(ips).ToNot(BeEmpty()) + Expect(conn.GetCount()).To(Equal(1)) + laddr := conn.LocalAddr().(*net.UDPAddr) + Expect(laddr.IP.String()).To(Equal("0.0.0.0")) + Expect(laddr.Port).ToNot(BeZero()) + }) + + It("reuses a connection it created for listening when dialing", func() { // listen - addr, err := net.ResolveUDPAddr("udp4", ips[0].String()+":0") + addr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") Expect(err).ToNot(HaveOccurred()) lconn, err := reuse.Listen("udp4", addr) Expect(err).ToNot(HaveOccurred()) Expect(lconn.GetCount()).To(Equal(1)) // dial + raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") + Expect(err).ToNot(HaveOccurred()) conn, err := reuse.Dial("udp4", raddr) Expect(err).ToNot(HaveOccurred()) Expect(conn.GetCount()).To(Equal(2)) }) - } + + if runtime.GOOS == "linux" { + It("reuses a connection it created for listening on a specific interface", func() { + raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") + Expect(err).ToNot(HaveOccurred()) + ips, err := reuse.getSourceIPs("udp4", raddr) + Expect(err).ToNot(HaveOccurred()) + Expect(ips).ToNot(BeEmpty()) + // listen + addr, err := net.ResolveUDPAddr("udp4", ips[0].String()+":0") + Expect(err).ToNot(HaveOccurred()) + lconn, err := reuse.Listen("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(lconn.GetCount()).To(Equal(1)) + // dial + conn, err := reuse.Dial("udp4", raddr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(2)) + }) + } + }) + + Context("garbage-collecting connections", func() { + numGlobals := func() int { + reuse.mutex.Lock() + defer reuse.mutex.Unlock() + return len(reuse.global) + } + + BeforeEach(func() { + maxUnusedDuration = 100 * time.Millisecond + }) + + It("garbage collects connections once they're not used any more for a certain time", func() { + addr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") + Expect(err).ToNot(HaveOccurred()) + lconn, err := reuse.Listen("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(lconn.GetCount()).To(Equal(1)) + + closeTime := time.Now() + lconn.DecreaseCount() + + for { + num := numGlobals() + if closeTime.Add(maxUnusedDuration).Before(time.Now()) { + break + } + Expect(num).To(Equal(1)) + time.Sleep(2 * time.Millisecond) + } + Eventually(numGlobals).Should(BeZero()) + }) + + It("only stops the garbage collector when there are no more connections", func() { + addr1, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") + Expect(err).ToNot(HaveOccurred()) + conn1, err := reuse.Listen("udp4", addr1) + Expect(err).ToNot(HaveOccurred()) + + addr2, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") + Expect(err).ToNot(HaveOccurred()) + conn2, err := reuse.Listen("udp4", addr2) + Expect(err).ToNot(HaveOccurred()) + + Eventually(isGarbageCollectorRunning).Should(BeTrue()) + conn1.DecreaseCount() + Consistently(isGarbageCollectorRunning, 2*maxUnusedDuration).Should(BeTrue()) + conn2.DecreaseCount() + Eventually(isGarbageCollectorRunning, 2*maxUnusedDuration).Should(BeFalse()) + }) + }) }) From 2fe3c88eb8656c7e673c5ba6ffc01f5c515c5a29 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 11 Aug 2019 15:33:53 +0700 Subject: [PATCH 1423/3965] close reuse-connections before deleting them --- p2p/transport/quic/reuse.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/transport/quic/reuse.go b/p2p/transport/quic/reuse.go index 2903b68384..cb45aa2307 100644 --- a/p2p/transport/quic/reuse.go +++ b/p2p/transport/quic/reuse.go @@ -84,12 +84,14 @@ func (r *reuse) runGarbageCollector() { r.mutex.Lock() for key, conn := range r.global { if conn.ShouldGarbageCollect(now) { + conn.Close() delete(r.global, key) } } for ukey, conns := range r.unicast { for key, conn := range conns { if conn.ShouldGarbageCollect(now) { + conn.Close() delete(conns, key) } } From e0f4801172f62b355e5df0a4b4dae667dac9c8f7 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 14 Aug 2019 14:27:04 -0700 Subject: [PATCH 1424/3965] Compare multihash `Code` to `IDENTITY` when extracting public key This PR is a non-substantive change with the goal of making this method `ExtractPublicKey` from a `peer.ID` clearer to the reader. This goal is accomplished by using the full name of the identity hash code `IDENTITY` instead of the deprecated `ID` code. This change makes its clearer to the reader that there is a trivial way to compute the public key given the digest. Without this change it is easy to confuse `ID` with a concept relating to peer IDs (and thus implying this function somehow computes a hash pre-image to get the corrresponding public key. --- core/peer/peer.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/peer/peer.go b/core/peer/peer.go index db307ee1e3..edef635c36 100644 --- a/core/peer/peer.go +++ b/core/peer/peer.go @@ -92,7 +92,7 @@ func (id ID) ExtractPublicKey() (ic.PubKey, error) { if err != nil { return nil, err } - if decoded.Code != mh.ID { + if decoded.Code != mh.IDENTITY { return nil, ErrNoPublicKey } pk, err := ic.UnmarshalPublicKey(decoded.Digest) From 235848850eb9b64e326f85f0ca04d32b21ef2738 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 16 Aug 2019 13:02:03 +0300 Subject: [PATCH 1425/3965] options to configure known relays for autorelay --- config/config.go | 3 ++- options.go | 34 ++++++++++++++++++++++++++++++++++ p2p/host/relay/autorelay.go | 11 +++++++++-- 3 files changed, 45 insertions(+), 3 deletions(-) diff --git a/config/config.go b/config/config.go index 3bb51bf1f7..46e99aaefc 100644 --- a/config/config.go +++ b/config/config.go @@ -70,6 +70,7 @@ type Config struct { Routing RoutingC EnableAutoRelay bool + StaticRelays []peer.AddrInfo } // NewNode constructs a new libp2p Host from the Config. @@ -226,7 +227,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { // advertise ourselves relay.Advertise(ctx, discovery) } else { - _ = relay.NewAutoRelay(swrm.Context(), h, discovery, router) + _ = relay.NewAutoRelay(swrm.Context(), h, discovery, router, cfg.StaticRelays) } } diff --git a/options.go b/options.go index 6d310f1aa3..a78a3043a5 100644 --- a/options.go +++ b/options.go @@ -10,6 +10,7 @@ import ( "github.com/libp2p/go-libp2p-core/connmgr" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/metrics" + "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/pnet" @@ -245,6 +246,39 @@ func EnableAutoRelay() Option { } } +// StaticRelays configures known relays for autorelay; when this option is enabled +// then the system will use the configured relays instead of querying the DHT to +// discover relays +func StaticRelays(relays []peer.AddrInfo) Option { + return func(cfg *Config) error { + cfg.StaticRelays = relays + return nil + } +} + +// DefaultRelays configures the static relays to use the known PL-operated relays +func DefaultRelays() Option { + return func(cfg *Config) error { + for _, addr := range []string{ + "/ip4/147.75.80.110/tcp/4001/p2p/QmbFgm5zan8P6eWWmeyfncR5feYEMPbht5b1FW1C37aQ7y", + "/ip4/147.75.195.153/tcp/4001/p2p/QmW9m57aiBDHAkKj9nmFSEn7ZqrcF1fZS4bipsTCHburei", + "/ip4/147.75.70.221/tcp/4001/p2p/Qme8g49gm3q4Acp7xWBKg3nAa9fxZ1YmyDJdyGgoG6LsXh", + } { + a, err := ma.NewMultiaddr(addr) + if err != nil { + return err + } + pi, err := peer.AddrInfoFromP2pAddr(a) + if err != nil { + return err + } + cfg.StaticRelays = append(cfg.StaticRelays, *pi) + } + + return nil + } +} + // FilterAddresses configures libp2p to never dial nor accept connections from // the given addresses. func FilterAddresses(addrs ...*net.IPNet) Option { diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index abab272e36..0dfda1d5f5 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -25,7 +25,7 @@ const ( ) var ( - DesiredRelays = 3 + DesiredRelays = 1 BootDelay = 20 * time.Second ) @@ -38,6 +38,8 @@ type AutoRelay struct { autonat autonat.AutoNAT addrsF basic.AddrsFactory + static []peer.AddrInfo + disconnect chan struct{} mx sync.Mutex @@ -48,12 +50,13 @@ type AutoRelay struct { cachedAddrsExpiry time.Time } -func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer, router routing.PeerRouting) *AutoRelay { +func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discovery.Discoverer, router routing.PeerRouting, static []peer.AddrInfo) *AutoRelay { ar := &AutoRelay{ host: bhost, discover: discover, router: router, addrsF: bhost.AddrsFactory, + static: static, relays: make(map[peer.ID]struct{}), disconnect: make(chan struct{}, 1), status: autonat.NATStatusUnknown, @@ -245,6 +248,10 @@ func (ar *AutoRelay) connect(ctx context.Context, pi peer.AddrInfo) bool { } func (ar *AutoRelay) discoverRelays(ctx context.Context) ([]peer.AddrInfo, error) { + if len(ar.static) > 0 { + return ar.static, nil + } + ctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() return discovery.FindPeers(ctx, ar.discover, RelayRendezvous, discovery.Limit(1000)) From 3bfc2cd83935301a507eeb1f0718ad4722bcabb4 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 16 Aug 2019 13:44:45 -0700 Subject: [PATCH 1426/3965] ci: bump go to 1.12 --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 5163d693fc..ba87d9a80a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,7 +4,7 @@ os: language: go go: - - 1.11.x + - 1.12.x env: global: From 99c5d3c62d752c29446a6491a26310b2a6de1527 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 16 Aug 2019 13:46:36 -0700 Subject: [PATCH 1427/3965] README: bump to go 1.12 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index dfbb9f4139..9df42c292e 100644 --- a/README.md +++ b/README.md @@ -58,9 +58,9 @@ libp2p is the product of a long, and arduous quest of understanding -- a deep di ## Usage -This repository (`go-libp2p`) serves as the entrypoint to the universe of modules that compose the Go implementation of the libp2p stack. +This repository (`go-libp2p`) serves as the entrypoint to the universe of modules that compose the Go implementation of the libp2p stack. Libp2p requires go 1.12+. -We mainly use [Go modules](https://github.com/golang/go/wiki/Modules) for our dependency and release management (and thus require go >= 1.11). In order to get the best developer experience, we recommend you do too. Otherwise, you may ocassionally encounter a breaking build as you'll be running off master (which, by definition, is not guaranteed to be stable). +We mainly use [Go modules](https://github.com/golang/go/wiki/Modules) for our dependency and release management (and thus require go >= 1.12+). In order to get the best developer experience, we recommend you do too. Otherwise, you may ocassionally encounter a breaking build as you'll be running off master (which, by definition, is not guaranteed to be stable). You can start using go-libp2p in your Go application simply by adding imports from our repos, e.g.: From 85cc7d8c92d03c5f9ad9feff64908bf7e030bd9f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 21 Aug 2019 10:47:31 -0700 Subject: [PATCH 1428/3965] fix listen addrs race Copy the listen address list before returning it. If you're wondering about the syntax: https://github.com/go101/go101/wiki/How-to-perfectly-clone-a-slice%3F --- p2p/net/swarm/swarm_addr.go | 6 +++--- p2p/net/swarm/swarm_addr_test.go | 18 ++++++++++++++++++ 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 6210a4f809..650fb79d13 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -35,8 +35,8 @@ func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { s.listeners.RUnlock() // RLock end if ifaceListenAddres != nil && !isEOL { - // Cache is valid - return ifaceListenAddres, nil + // Cache is valid, clone the slice + return append(ifaceListenAddres[:0:0], ifaceListenAddres...), nil } // Cache is not valid @@ -64,5 +64,5 @@ func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { s.listeners.Unlock() // Lock end - return ifaceListenAddres, nil + return append(ifaceListenAddres[:0:0], ifaceListenAddres...), nil } diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index e75d660615..5009946c6d 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -35,3 +35,21 @@ func TestDialBadAddrs(t *testing.T) { test(m("/ip6/fe80::100")) // link local test(m("/ip4/127.0.0.1/udp/1234/utp")) // utp } + +func TestAddrRace(t *testing.T) { + ctx := context.Background() + s := makeSwarms(ctx, t, 1)[0] + + a1, err := s.InterfaceListenAddresses() + if err != nil { + t.Fatal(err) + } + a2, err := s.InterfaceListenAddresses() + if err != nil { + t.Fatal(err) + } + + if len(a1) > 0 && len(a2) > 0 && &a1[0] == &a2[0] { + t.Fatal("got the exact same address set twice; this could lead to data races") + } +} From 1edd9dd300f1f4e61ae04ff56820fc5f40db17f7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 21 Aug 2019 11:03:23 -0700 Subject: [PATCH 1429/3965] dep: update go-libp2p-swarm Fixes a race condition when a user modifies the address list returned by InterfaceListenAddresses. --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index c11ed08abd..422a39d01d 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.1.3 github.com/libp2p/go-libp2p-secio v0.2.0 - github.com/libp2p/go-libp2p-swarm v0.2.0 + github.com/libp2p/go-libp2p-swarm v0.2.1 github.com/libp2p/go-libp2p-testing v0.1.0 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 github.com/libp2p/go-libp2p-yamux v0.2.1 diff --git a/go.sum b/go.sum index 0d05b332f1..3ab31cc07b 100644 --- a/go.sum +++ b/go.sum @@ -148,8 +148,8 @@ github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBg github.com/libp2p/go-libp2p-secio v0.2.0 h1:ywzZBsWEEz2KNTn5RtzauEDq5RFEefPsttXYwAWqHng= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= -github.com/libp2p/go-libp2p-swarm v0.2.0 h1:gUqj9WYFC9mQ6khEtO5EjfaAtevwBMxsXRARbi6lAiI= -github.com/libp2p/go-libp2p-swarm v0.2.0/go.mod h1:x07b4zkMFo2EvgPV2bMTlNmdQc8i+74Jjio7xGvsTgU= +github.com/libp2p/go-libp2p-swarm v0.2.1 h1:9A8oQqPIZvbaRyrjViHeDYS7fE7fNtP7BRWdJrBHbe8= +github.com/libp2p/go-libp2p-swarm v0.2.1/go.mod h1:x07b4zkMFo2EvgPV2bMTlNmdQc8i+74Jjio7xGvsTgU= github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.4 h1:Qev57UR47GcLPXWjrunv5aLIQGO4n9mhI/8/EIrEEFc= From fa4a3a09b31c274bbd9688182d7961d4c37fd006 Mon Sep 17 00:00:00 2001 From: Kevin Mai-Husan Chia Date: Fri, 23 Aug 2019 12:31:04 +0800 Subject: [PATCH 1430/3965] Get `remoteID` from pb message --- core/sec/insecure/insecure.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/sec/insecure/insecure.go b/core/sec/insecure/insecure.go index e3670c222a..5d1c783a6a 100644 --- a/core/sec/insecure/insecure.go +++ b/core/sec/insecure/insecure.go @@ -162,7 +162,7 @@ func (ic *Conn) runHandshakeSync() error { return err } - remoteID, err := peer.IDFromPublicKey(remotePubkey) + remoteID, err := peer.IDFromBytes(remoteMsg.Id) if err != nil { return err } From 27919d22af7209395210b1c896aa1cc5f3c00fbb Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 17:30:09 -0400 Subject: [PATCH 1431/3965] begin protocol, handshake, change XX.noise.go to export stuff --- p2p/security/noise/ik/IK.noise.go | 434 +++++++++++++++++++++++++++ p2p/security/noise/info.go | 10 + p2p/security/noise/payload.pb.go | 105 +++++++ p2p/security/noise/payload.proto | 9 + p2p/security/noise/protocol.go | 147 +++++++++ p2p/security/noise/transport.go | 29 ++ p2p/security/noise/xx/XX.noise.go | 481 ++++++++++++++++++++++++++++++ 7 files changed, 1215 insertions(+) create mode 100644 p2p/security/noise/ik/IK.noise.go create mode 100644 p2p/security/noise/info.go create mode 100644 p2p/security/noise/payload.pb.go create mode 100644 p2p/security/noise/payload.proto create mode 100644 p2p/security/noise/protocol.go create mode 100644 p2p/security/noise/transport.go create mode 100644 p2p/security/noise/xx/XX.noise.go diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go new file mode 100644 index 0000000000..97071e80bd --- /dev/null +++ b/p2p/security/noise/ik/IK.noise.go @@ -0,0 +1,434 @@ +/* +IK: + <- s + ... + -> e, es, s, ss + <- e, ee, se +*/ + +// Implementation Version: 1.0.0 + +/* ---------------------------------------------------------------- * + * PARAMETERS * + * ---------------------------------------------------------------- */ + +package main + +import ( + "crypto/rand" + "crypto/subtle" + "encoding/binary" + "golang.org/x/crypto/blake2s" + "golang.org/x/crypto/chacha20poly1305" + "golang.org/x/crypto/curve25519" + "golang.org/x/crypto/hkdf" + "hash" + "io" + "math" +) + +/* ---------------------------------------------------------------- * + * TYPES * + * ---------------------------------------------------------------- */ + +type keypair struct { + public_key [32]byte + private_key [32]byte +} + +type messagebuffer struct { + ne [32]byte + ns []byte + ciphertext []byte +} + +type cipherstate struct { + k [32]byte + n uint32 +} + +type symmetricstate struct { + cs cipherstate + ck [32]byte + h [32]byte +} + +type handshakestate struct { + ss symmetricstate + s keypair + e keypair + rs [32]byte + re [32]byte + psk [32]byte +} + +type noisesession struct { + hs handshakestate + h [32]byte + cs1 cipherstate + cs2 cipherstate + mc uint64 + i bool +} + +/* ---------------------------------------------------------------- * + * CONSTANTS * + * ---------------------------------------------------------------- */ + +var emptyKey = [32]byte{ + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, +} + +var minNonce = uint32(0) + +/* ---------------------------------------------------------------- * + * UTILITY FUNCTIONS * + * ---------------------------------------------------------------- */ + +func getPublicKey(kp *keypair) [32]byte { + return kp.public_key +} + +func isEmptyKey(k [32]byte) bool { + return subtle.ConstantTimeCompare(k[:], emptyKey[:]) == 1 +} + +func validatePublicKey(k []byte) bool { + forbiddenCurveValues := [12][]byte{ + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + {224, 235, 122, 124, 59, 65, 184, 174, 22, 86, 227, 250, 241, 159, 196, 106, 218, 9, 141, 235, 156, 50, 177, 253, 134, 98, 5, 22, 95, 73, 184, 0}, + {95, 156, 149, 188, 163, 80, 140, 36, 177, 208, 177, 85, 156, 131, 239, 91, 4, 68, 92, 196, 88, 28, 142, 134, 216, 34, 78, 221, 208, 159, 17, 87}, + {236, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, + {237, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, + {238, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, + {205, 235, 122, 124, 59, 65, 184, 174, 22, 86, 227, 250, 241, 159, 196, 106, 218, 9, 141, 235, 156, 50, 177, 253, 134, 98, 5, 22, 95, 73, 184, 128}, + {76, 156, 149, 188, 163, 80, 140, 36, 177, 208, 177, 85, 156, 131, 239, 91, 4, 68, 92, 196, 88, 28, 142, 134, 216, 34, 78, 221, 208, 159, 17, 215}, + {217, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}, + {218, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}, + {219, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 25}, + } + + for _, testValue := range forbiddenCurveValues { + if subtle.ConstantTimeCompare(k[:], testValue[:]) == 1 { + panic("Invalid public key") + } + } + return true +} +/* ---------------------------------------------------------------- * + * PRIMITIVES * + * ---------------------------------------------------------------- */ + +func incrementNonce(n uint32) uint32 { + return n + 1 +} + +func dh(private_key [32]byte, public_key [32]byte) [32]byte { + var ss [32]byte + curve25519.ScalarMult(&ss, &private_key, &public_key) + return ss +} + +func generateKeypair() keypair { + var public_key [32]byte + var private_key [32]byte + _, _ = rand.Read(private_key[:]) + curve25519.ScalarBaseMult(&public_key, &private_key) + if validatePublicKey(public_key[:]) { + return keypair{public_key, private_key} + } + return generateKeypair() +} + +func generatePublicKey(private_key [32]byte) [32]byte { + var public_key [32]byte + curve25519.ScalarBaseMult(&public_key, &private_key) + return public_key +} + +func encrypt(k [32]byte, n uint32, ad []byte, plaintext []byte) []byte { + var nonce [12]byte + var ciphertext []byte + enc, _ := chacha20poly1305.New(k[:]) + binary.LittleEndian.PutUint32(nonce[4:], n) + ciphertext = enc.Seal(nil, nonce[:], plaintext, ad) + return ciphertext +} + +func decrypt(k [32]byte, n uint32, ad []byte, ciphertext []byte) (bool, []byte, []byte) { + var nonce [12]byte + var plaintext []byte + enc, err := chacha20poly1305.New(k[:]) + binary.LittleEndian.PutUint32(nonce[4:], n) + plaintext, err = enc.Open(nil, nonce[:], ciphertext, ad) + return (err == nil), ad, plaintext +} + +func getHash(a []byte, b []byte) [32]byte { + return blake2s.Sum256(append(a, b...)) +} + +func hashProtocolName(protocolName []byte) [32]byte { + var h [32]byte + if len(protocolName) <= 32 { + copy(h[:], protocolName) + } else { + h = getHash(protocolName, []byte{}) + } + return h +} + +func blake2HkdfInterface() hash.Hash { + h, _ := blake2s.New256([]byte{}) + return h +} + +func getHkdf(ck [32]byte, ikm []byte) ([32]byte, [32]byte, [32]byte) { + var k1 [32]byte + var k2 [32]byte + var k3 [32]byte + output := hkdf.New(blake2HkdfInterface, ikm[:], ck[:], []byte{}) + io.ReadFull(output, k1[:]) + io.ReadFull(output, k2[:]) + io.ReadFull(output, k3[:]) + return k1, k2, k3 +} + +/* ---------------------------------------------------------------- * + * STATE MANAGEMENT * + * ---------------------------------------------------------------- */ + +/* CipherState */ +func initializeKey(k [32]byte) cipherstate { + return cipherstate{k, minNonce} +} + +func hasKey(cs *cipherstate) bool { + return !isEmptyKey(cs.k) +} + +func setNonce(cs *cipherstate, newNonce uint32) *cipherstate { + cs.n = newNonce + return cs +} + +func encryptWithAd(cs *cipherstate, ad []byte, plaintext []byte) (*cipherstate, []byte) { + e := encrypt(cs.k, cs.n, ad, plaintext) + cs = setNonce(cs, incrementNonce(cs.n)) + return cs, e +} + +func decryptWithAd(cs *cipherstate, ad []byte, ciphertext []byte) (*cipherstate, []byte, bool) { + valid, ad, plaintext := decrypt(cs.k, cs.n, ad, ciphertext) + cs = setNonce(cs, incrementNonce(cs.n)) + return cs, plaintext, valid +} + +func reKey(cs *cipherstate) *cipherstate { + e := encrypt(cs.k, math.MaxUint32, []byte{}, emptyKey[:]) + copy(cs.k[:], e) + return cs +} + +/* SymmetricState */ + +func initializeSymmetric(protocolName []byte) symmetricstate { + h := hashProtocolName(protocolName) + ck := h + cs := initializeKey(emptyKey) + return symmetricstate{cs, ck, h} +} + +func mixKey(ss *symmetricstate, ikm [32]byte) *symmetricstate { + ck, tempK, _ := getHkdf(ss.ck, ikm[:]) + ss.cs = initializeKey(tempK) + ss.ck = ck + return ss +} + +func mixHash(ss *symmetricstate, data []byte) *symmetricstate { + ss.h = getHash(ss.h[:], data) + return ss +} + +func mixKeyAndHash(ss *symmetricstate, ikm [32]byte) *symmetricstate { + var tempH [32]byte + var tempK [32]byte + ss.ck, tempH, tempK = getHkdf(ss.ck, ikm[:]) + ss = mixHash(ss, tempH[:]) + ss.cs = initializeKey(tempK) + return ss +} + +func getHandshakeHash(ss *symmetricstate) [32]byte { + return ss.h +} + +func encryptAndHash(ss *symmetricstate, plaintext []byte) (*symmetricstate, []byte) { + var ciphertext []byte + if hasKey(&ss.cs) { + _, ciphertext = encryptWithAd(&ss.cs, ss.h[:], plaintext) + } else { + ciphertext = plaintext + } + ss = mixHash(ss, ciphertext) + return ss, ciphertext +} + +func decryptAndHash(ss *symmetricstate, ciphertext []byte) (*symmetricstate, []byte, bool) { + var plaintext []byte + var valid bool + if hasKey(&ss.cs) { + _, plaintext, valid = decryptWithAd(&ss.cs, ss.h[:], ciphertext) + } else { + plaintext, valid = ciphertext, true + } + ss = mixHash(ss, ciphertext) + return ss, plaintext, valid +} + +func split(ss *symmetricstate) (cipherstate, cipherstate) { + tempK1, tempK2, _ := getHkdf(ss.ck, []byte{}) + cs1 := initializeKey(tempK1) + cs2 := initializeKey(tempK2) + return cs1, cs2 +} + +/* HandshakeState */ + +func initializeInitiator(prologue []byte, s keypair, rs [32]byte, psk [32]byte) handshakestate { + var ss symmetricstate + var e keypair + var re [32]byte + name := []byte("Noise_IK_25519_ChaChaPoly_BLAKE2s") + ss = initializeSymmetric(name) + mixHash(&ss, prologue) + mixHash(&ss, rs[:]) + return handshakestate{ss, s, e, rs, re, psk} +} + +func initializeResponder(prologue []byte, s keypair, rs [32]byte, psk [32]byte) handshakestate { + var ss symmetricstate + var e keypair + var re [32]byte + name := []byte("Noise_IK_25519_ChaChaPoly_BLAKE2s") + ss = initializeSymmetric(name) + mixHash(&ss, prologue) + mixHash(&ss, s.public_key[:]) + return handshakestate{ss, s, e, rs, re, psk} +} + +func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, messagebuffer) { + ne, ns, ciphertext := emptyKey, []byte{}, []byte{} + hs.e = generateKeypair() + ne = hs.e.public_key + mixHash(&hs.ss, ne[:]) + /* No PSK, so skipping mixKey */ + mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) + spk := make([]byte, len(hs.s.public_key)) + copy(spk[:], hs.s.public_key[:]) + _, ns = encryptAndHash(&hs.ss, spk) + mixKey(&hs.ss, dh(hs.s.private_key, hs.rs)) + _, ciphertext = encryptAndHash(&hs.ss, payload) + messageBuffer := messagebuffer{ne, ns, ciphertext} + return hs, messageBuffer +} + +func writeMessageB(hs *handshakestate, payload []byte) ([32]byte, messagebuffer, cipherstate, cipherstate) { + ne, ns, ciphertext := emptyKey, []byte{}, []byte{} + hs.e = generateKeypair() + ne = hs.e.public_key + mixHash(&hs.ss, ne[:]) + /* No PSK, so skipping mixKey */ + mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) + mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) + _, ciphertext = encryptAndHash(&hs.ss, payload) + messageBuffer := messagebuffer{ne, ns, ciphertext} + cs1, cs2 := split(&hs.ss) + return hs.ss.h, messageBuffer, cs1, cs2 +} + +func readMessageA(hs *handshakestate, message *messagebuffer) (*handshakestate, []byte, bool) { + valid1 := true + if validatePublicKey(message.ne[:]) { + hs.re = message.ne + } + mixHash(&hs.ss, hs.re[:]) + /* No PSK, so skipping mixKey */ + mixKey(&hs.ss, dh(hs.s.private_key, hs.re)) + _, ns, valid1 := decryptAndHash(&hs.ss, message.ns) + if valid1 && len(ns) == 32 && validatePublicKey(message.ns[:]) { + copy(hs.rs[:], ns) + } + mixKey(&hs.ss, dh(hs.s.private_key, hs.rs)) + _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) + return hs, plaintext, (valid1 && valid2) +} + +func readMessageB(hs *handshakestate, message *messagebuffer) ([32]byte, []byte, bool, cipherstate, cipherstate) { + valid1 := true + if validatePublicKey(message.ne[:]) { + hs.re = message.ne + } + mixHash(&hs.ss, hs.re[:]) + /* No PSK, so skipping mixKey */ + mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) + mixKey(&hs.ss, dh(hs.s.private_key, hs.re)) + _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) + cs1, cs2 := split(&hs.ss) + return hs.ss.h, plaintext, (valid1 && valid2), cs1, cs2 +} + +/* ---------------------------------------------------------------- * + * PROCESSES * + * ---------------------------------------------------------------- */ + +func InitSession(initiator bool, prologue []byte, s keypair, rs [32]byte) noisesession { + var session noisesession + psk := emptyKey + if initiator { + session.hs = initializeInitiator(prologue, s, rs, psk) + } else { + session.hs = initializeResponder(prologue, s, rs, psk) + } + session.i = initiator + session.mc = 0 + return session +} + +func SendMessage(session *noisesession, message []byte) (*noisesession, messagebuffer) { + var messageBuffer messagebuffer + if session.mc == 0 { + _, messageBuffer = writeMessageA(&session.hs, message) + } + if session.mc == 1 { + session.h, messageBuffer, session.cs1, session.cs2 = writeMessageB(&session.hs, message) + session.hs = handshakestate{} + } + session.mc = session.mc + 1 + return session, messageBuffer +} + +func RecvMessage(session *noisesession, message *messagebuffer) (*noisesession, []byte, bool) { + var plaintext []byte + var valid bool + if session.mc == 0 { + _, plaintext, valid = readMessageA(&session.hs, message) + } + if session.mc == 1 { + session.h, plaintext, valid, session.cs1, session.cs2 = readMessageB(&session.hs, message) + session.hs = handshakestate{} + } + session.mc = session.mc + 1 + return session, plaintext, valid +} + +func main() {} \ No newline at end of file diff --git a/p2p/security/noise/info.go b/p2p/security/noise/info.go new file mode 100644 index 0000000000..9f1625cc7d --- /dev/null +++ b/p2p/security/noise/info.go @@ -0,0 +1,10 @@ +package noise + +import ( + "github.com/libp2p/go-libp2p-core/crypto" +) + +type peerInfo struct { + staticKey crypto.PubKey + ephemeralKey crypto.PubKey +} \ No newline at end of file diff --git a/p2p/security/noise/payload.pb.go b/p2p/security/noise/payload.pb.go new file mode 100644 index 0000000000..15fafac6a0 --- /dev/null +++ b/p2p/security/noise/payload.pb.go @@ -0,0 +1,105 @@ +// Code generated by protoc-gen-go. DO NOT EDIT. +// source: payload.proto + +package noise + +import ( + fmt "fmt" + proto "github.com/golang/protobuf/proto" + math "math" +) + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package + +type NoiseHandshakePayload struct { + Libp2PKey []byte `protobuf:"bytes,1,opt,name=libp2p_key,json=libp2pKey,proto3" json:"libp2p_key,omitempty"` + NoiseStaticKeySignature []byte `protobuf:"bytes,2,opt,name=noise_static_key_signature,json=noiseStaticKeySignature,proto3" json:"noise_static_key_signature,omitempty"` + Libp2PData []byte `protobuf:"bytes,3,opt,name=libp2p_data,json=libp2pData,proto3" json:"libp2p_data,omitempty"` + Libp2PDataSignature []byte `protobuf:"bytes,4,opt,name=libp2p_data_signature,json=libp2pDataSignature,proto3" json:"libp2p_data_signature,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *NoiseHandshakePayload) Reset() { *m = NoiseHandshakePayload{} } +func (m *NoiseHandshakePayload) String() string { return proto.CompactTextString(m) } +func (*NoiseHandshakePayload) ProtoMessage() {} +func (*NoiseHandshakePayload) Descriptor() ([]byte, []int) { + return fileDescriptor_678c914f1bee6d56, []int{0} +} + +func (m *NoiseHandshakePayload) XXX_Unmarshal(b []byte) error { + return xxx_messageInfo_NoiseHandshakePayload.Unmarshal(m, b) +} +func (m *NoiseHandshakePayload) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + return xxx_messageInfo_NoiseHandshakePayload.Marshal(b, m, deterministic) +} +func (m *NoiseHandshakePayload) XXX_Merge(src proto.Message) { + xxx_messageInfo_NoiseHandshakePayload.Merge(m, src) +} +func (m *NoiseHandshakePayload) XXX_Size() int { + return xxx_messageInfo_NoiseHandshakePayload.Size(m) +} +func (m *NoiseHandshakePayload) XXX_DiscardUnknown() { + xxx_messageInfo_NoiseHandshakePayload.DiscardUnknown(m) +} + +var xxx_messageInfo_NoiseHandshakePayload proto.InternalMessageInfo + +func (m *NoiseHandshakePayload) GetLibp2PKey() []byte { + if m != nil { + return m.Libp2PKey + } + return nil +} + +func (m *NoiseHandshakePayload) GetNoiseStaticKeySignature() []byte { + if m != nil { + return m.NoiseStaticKeySignature + } + return nil +} + +func (m *NoiseHandshakePayload) GetLibp2PData() []byte { + if m != nil { + return m.Libp2PData + } + return nil +} + +func (m *NoiseHandshakePayload) GetLibp2PDataSignature() []byte { + if m != nil { + return m.Libp2PDataSignature + } + return nil +} + +func init() { + proto.RegisterType((*NoiseHandshakePayload)(nil), "noise.NoiseHandshakePayload") +} + +func init() { proto.RegisterFile("payload.proto", fileDescriptor_678c914f1bee6d56) } + +var fileDescriptor_678c914f1bee6d56 = []byte{ + // 176 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x2d, 0x48, 0xac, 0xcc, + 0xc9, 0x4f, 0x4c, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0xcd, 0xcb, 0xcf, 0x2c, 0x4e, + 0x55, 0x3a, 0xc1, 0xc8, 0x25, 0xea, 0x07, 0x62, 0x79, 0x24, 0xe6, 0xa5, 0x14, 0x67, 0x24, 0x66, + 0xa7, 0x06, 0x40, 0x94, 0x09, 0xc9, 0x72, 0x71, 0xe5, 0x64, 0x26, 0x15, 0x18, 0x15, 0xc4, 0x67, + 0xa7, 0x56, 0x4a, 0x30, 0x2a, 0x30, 0x6a, 0xf0, 0x04, 0x71, 0x42, 0x44, 0xbc, 0x53, 0x2b, 0x85, + 0xac, 0xb9, 0xa4, 0xc0, 0x26, 0xc4, 0x17, 0x97, 0x24, 0x96, 0x64, 0x26, 0x83, 0x14, 0xc5, 0x17, + 0x67, 0xa6, 0xe7, 0x25, 0x96, 0x94, 0x16, 0xa5, 0x4a, 0x30, 0x81, 0x95, 0x8b, 0x83, 0x55, 0x04, + 0x83, 0x15, 0x78, 0xa7, 0x56, 0x06, 0xc3, 0xa4, 0x85, 0xe4, 0xb9, 0xb8, 0xa1, 0x66, 0xa7, 0x24, + 0x96, 0x24, 0x4a, 0x30, 0x83, 0x55, 0x43, 0xad, 0x73, 0x49, 0x2c, 0x49, 0x14, 0x32, 0xe2, 0x12, + 0x45, 0x52, 0x80, 0x64, 0x30, 0x0b, 0x58, 0xa9, 0x30, 0x42, 0x29, 0xdc, 0xd0, 0x24, 0x36, 0xb0, + 0xc7, 0x8c, 0x01, 0x01, 0x00, 0x00, 0xff, 0xff, 0x0f, 0xcc, 0x4e, 0xb9, 0xe9, 0x00, 0x00, 0x00, +} diff --git a/p2p/security/noise/payload.proto b/p2p/security/noise/payload.proto new file mode 100644 index 0000000000..dbaacb5196 --- /dev/null +++ b/p2p/security/noise/payload.proto @@ -0,0 +1,9 @@ +syntax = "proto3"; +package noise; + +message NoiseHandshakePayload { + bytes libp2p_key = 1; + bytes noise_static_key_signature = 2; + bytes libp2p_data = 3; + bytes libp2p_data_signature = 4; +} \ No newline at end of file diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go new file mode 100644 index 0000000000..1b55aea7de --- /dev/null +++ b/p2p/security/noise/protocol.go @@ -0,0 +1,147 @@ +package noise + +import ( + "context" + "fmt" + "net" + "time" + + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" + proto "github.com/gogo/protobuf/proto" + + xx "github.com/noot/noise/xx" +) + +type secureSession struct { + insecure net.Conn + + initiator bool + prologue []byte + + localKey crypto.PrivKey + localPeer peer.ID + remotePeer peer.ID + + local peerInfo + remote peerInfo +} + +func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, insecure net.Conn, remote peer.ID, initiator bool) (sec.SecureConn, error) { + + s := &secureSession{ + insecure: insecure, + initiator: initiator, + prologue: []byte(ID), + localKey: privKey, + localPeer: local, + remotePeer: remote, + } + + return s, nil +} + +func (s *secureSession) runHandshake(ctx context.Context) error { + + // TODO: check if static key for peer exists + // if so, do XX; otherwise do IK + + // PHASE 1: TRY XX + + // get remote static key + remotePub := [32]byte{} + remotePubRaw, err := s.RemotePublicKey().Raw() + if err != nil { + return fmt.Errorf("remote pubkey fail: %s", err) + } + copy(remotePub[:], remotePubRaw) + + // generate local static noise key + kp := xx.GenerateKeypair() + //pub := xx.GeneratePublicKey(kp.PrivKey()) + + // new XX noise session + ns := xx.InitSession(s.initiator, s.prologue, kp, remotePub) + + // send initial payload message + if s.initiator { + // create payload + payload := new(NoiseHandshakePayload) + msg, err := proto.Marshal(payload) + if err != nil { + return fmt.Errorf("proto marshal payload fail: %s", err) + } + + var msgbuf xx.MessageBuffer + ns, msgbuf = xx.SendMessage(ns, msg) + + encMsgBuf := msgbuf.Encode0() + if len(encMsgBuf) != 56 { + return fmt.Errorf("enc msg buf: len does not equal 56") + } + + _, err = s.insecure.Write(encMsgBuf) + if err != nil { + return fmt.Errorf("write to conn fail: %s", err) + } + + buf := make([]byte, 144) + _, err = s.insecure.Read(buf) + if err != nil { + return fmt.Errorf("read from conn fail: %s", err) + } + + + } + + return nil +} + +func (s *secureSession) LocalAddr() net.Addr { + return s.insecure.LocalAddr() +} + +func (s *secureSession) LocalPeer() peer.ID { + return s.localPeer +} + +func (s *secureSession) LocalPrivateKey() crypto.PrivKey { + return s.localKey +} + +func (s *secureSession) Read(in []byte) (int, error) { + return s.insecure.Read(in) +} + +func (s *secureSession) RemoteAddr() net.Addr { + return s.insecure.RemoteAddr() +} + +func (s *secureSession) RemotePeer() peer.ID { + return s.remotePeer +} + +func (s *secureSession) RemotePublicKey() crypto.PubKey { + return s.remote.staticKey +} + +func (s *secureSession) SetDeadline(t time.Time) error { + return s.insecure.SetDeadline(t) +} + +func (s *secureSession) SetReadDeadline(t time.Time) error { + return s.insecure.SetReadDeadline(t) +} + +func (s *secureSession) SetWriteDeadline(t time.Time) error { + return s.insecure.SetWriteDeadline(t) +} + +func (s *secureSession) Write(in []byte) (int, error) { + return s.insecure.Write(in) +} + +func (s *secureSession) Close() error { + return s.insecure.Close() +} \ No newline at end of file diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go new file mode 100644 index 0000000000..6baac2d605 --- /dev/null +++ b/p2p/security/noise/transport.go @@ -0,0 +1,29 @@ +package noise + +import ( + "context" + "net" + + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" +) + +const ID = "/noise/0.0.0" + +type Transport struct { + LocalID peer.ID + PrivateKey crypto.PrivKey + + //Payload NoiseHandshakePayload +} + +// SecureInbound runs noise handshake as a server +func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { + return newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, "", false) +} + +// SecureOutbound runs noise handshake as a client +func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { + return newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, p, true) +} \ No newline at end of file diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go new file mode 100644 index 0000000000..06171ceafd --- /dev/null +++ b/p2p/security/noise/xx/XX.noise.go @@ -0,0 +1,481 @@ +/* +XX: + -> e + <- e, ee, s, es + -> s, se +*/ + +// Implementation Version: 1.0.0 + +/* ---------------------------------------------------------------- * + * PARAMETERS * + * ---------------------------------------------------------------- */ + +package xx + +import ( + "crypto/rand" + "crypto/subtle" + "crypto/sha256" + + "encoding/binary" + //"golang.org/x/crypto/blake2s" + "golang.org/x/crypto/chacha20poly1305" + "golang.org/x/crypto/curve25519" + "golang.org/x/crypto/hkdf" + "hash" + "io" + "math" +) + +/* ---------------------------------------------------------------- * + * TYPES * + * ---------------------------------------------------------------- */ + +type Keypair struct { + public_key [32]byte + private_key [32]byte +} + +type MessageBuffer struct { + ne [32]byte + ns []byte + ciphertext []byte +} + +type cipherstate struct { + k [32]byte + n uint32 +} + +type symmetricstate struct { + cs cipherstate + ck [32]byte + h [32]byte +} + +type handshakestate struct { + ss symmetricstate + s Keypair + e Keypair + rs [32]byte + re [32]byte + psk [32]byte +} + +type noisesession struct { + hs handshakestate + h [32]byte + cs1 cipherstate + cs2 cipherstate + mc uint64 + i bool +} + +func NewKeypair(pub [32]byte, priv [32]byte) Keypair { + return Keypair{ + public_key: pub, + private_key: priv, + } +} + +func (kp Keypair) PubKey() [32]byte{ + return kp.public_key +} + +func (kp Keypair) PrivKey() [32]byte{ + return kp.private_key +} + +/* ---------------------------------------------------------------- * + * CONSTANTS * + * ---------------------------------------------------------------- */ + +var emptyKey = [32]byte{ + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, +} + +var minNonce = uint32(0) + +/* ---------------------------------------------------------------- * + * UTILITY FUNCTIONS * + * ---------------------------------------------------------------- */ + +func getPublicKey(kp *Keypair) [32]byte { + return kp.public_key +} + +func isEmptyKey(k [32]byte) bool { + return subtle.ConstantTimeCompare(k[:], emptyKey[:]) == 1 +} + +func (mb *MessageBuffer) Encode0() []byte { + enc := []byte{} + + enc = append(enc, mb.ne[:]...) + enc = append(enc, mb.ciphertext...) + + return enc +} + +func validatePublicKey(k []byte) bool { + forbiddenCurveValues := [12][]byte{ + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + {224, 235, 122, 124, 59, 65, 184, 174, 22, 86, 227, 250, 241, 159, 196, 106, 218, 9, 141, 235, 156, 50, 177, 253, 134, 98, 5, 22, 95, 73, 184, 0}, + {95, 156, 149, 188, 163, 80, 140, 36, 177, 208, 177, 85, 156, 131, 239, 91, 4, 68, 92, 196, 88, 28, 142, 134, 216, 34, 78, 221, 208, 159, 17, 87}, + {236, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, + {237, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, + {238, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, + {205, 235, 122, 124, 59, 65, 184, 174, 22, 86, 227, 250, 241, 159, 196, 106, 218, 9, 141, 235, 156, 50, 177, 253, 134, 98, 5, 22, 95, 73, 184, 128}, + {76, 156, 149, 188, 163, 80, 140, 36, 177, 208, 177, 85, 156, 131, 239, 91, 4, 68, 92, 196, 88, 28, 142, 134, 216, 34, 78, 221, 208, 159, 17, 215}, + {217, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}, + {218, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}, + {219, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 25}, + } + + for _, testValue := range forbiddenCurveValues { + if subtle.ConstantTimeCompare(k[:], testValue[:]) == 1 { + panic("Invalid public key") + } + } + return true +} +/* ---------------------------------------------------------------- * + * PRIMITIVES * + * ---------------------------------------------------------------- */ + +func incrementNonce(n uint32) uint32 { + return n + 1 +} + +func dh(private_key [32]byte, public_key [32]byte) [32]byte { + var ss [32]byte + curve25519.ScalarMult(&ss, &private_key, &public_key) + return ss +} + +func GenerateKeypair() Keypair { + var public_key [32]byte + var private_key [32]byte + _, _ = rand.Read(private_key[:]) + curve25519.ScalarBaseMult(&public_key, &private_key) + if validatePublicKey(public_key[:]) { + return Keypair{public_key, private_key} + } + return GenerateKeypair() +} + +func GeneratePublicKey(private_key [32]byte) [32]byte { + var public_key [32]byte + curve25519.ScalarBaseMult(&public_key, &private_key) + return public_key +} + +func encrypt(k [32]byte, n uint32, ad []byte, plaintext []byte) []byte { + var nonce [12]byte + var ciphertext []byte + enc, _ := chacha20poly1305.New(k[:]) + binary.LittleEndian.PutUint32(nonce[4:], n) + ciphertext = enc.Seal(nil, nonce[:], plaintext, ad) + return ciphertext +} + +func decrypt(k [32]byte, n uint32, ad []byte, ciphertext []byte) (bool, []byte, []byte) { + var nonce [12]byte + var plaintext []byte + enc, err := chacha20poly1305.New(k[:]) + binary.LittleEndian.PutUint32(nonce[4:], n) + plaintext, err = enc.Open(nil, nonce[:], ciphertext, ad) + return (err == nil), ad, plaintext +} + +func getHash(a []byte, b []byte) [32]byte { + //return blake2s.Sum256(append(a, b...)) + return sha256.Sum256(append(a, b...)) +} + +func hashProtocolName(protocolName []byte) [32]byte { + var h [32]byte + if len(protocolName) <= 32 { + copy(h[:], protocolName) + } else { + h = getHash(protocolName, []byte{}) + } + return h +} + +func sha256HkdfInterface() hash.Hash { + //h, _ := blake2s.New256([]byte{}) + h := sha256.New() + return h +} + +func getHkdf(ck [32]byte, ikm []byte) ([32]byte, [32]byte, [32]byte) { + var k1 [32]byte + var k2 [32]byte + var k3 [32]byte + output := hkdf.New(sha256HkdfInterface, ikm[:], ck[:], []byte{}) + io.ReadFull(output, k1[:]) + io.ReadFull(output, k2[:]) + io.ReadFull(output, k3[:]) + return k1, k2, k3 +} + +/* ---------------------------------------------------------------- * + * STATE MANAGEMENT * + * ---------------------------------------------------------------- */ + +/* CipherState */ +func initializeKey(k [32]byte) cipherstate { + return cipherstate{k, minNonce} +} + +func hasKey(cs *cipherstate) bool { + return !isEmptyKey(cs.k) +} + +func setNonce(cs *cipherstate, newNonce uint32) *cipherstate { + cs.n = newNonce + return cs +} + +func encryptWithAd(cs *cipherstate, ad []byte, plaintext []byte) (*cipherstate, []byte) { + e := encrypt(cs.k, cs.n, ad, plaintext) + cs = setNonce(cs, incrementNonce(cs.n)) + return cs, e +} + +func decryptWithAd(cs *cipherstate, ad []byte, ciphertext []byte) (*cipherstate, []byte, bool) { + valid, ad, plaintext := decrypt(cs.k, cs.n, ad, ciphertext) + cs = setNonce(cs, incrementNonce(cs.n)) + return cs, plaintext, valid +} + +func reKey(cs *cipherstate) *cipherstate { + e := encrypt(cs.k, math.MaxUint32, []byte{}, emptyKey[:]) + copy(cs.k[:], e) + return cs +} + +/* SymmetricState */ + +func initializeSymmetric(protocolName []byte) symmetricstate { + h := hashProtocolName(protocolName) + ck := h + cs := initializeKey(emptyKey) + return symmetricstate{cs, ck, h} +} + +func mixKey(ss *symmetricstate, ikm [32]byte) *symmetricstate { + ck, tempK, _ := getHkdf(ss.ck, ikm[:]) + ss.cs = initializeKey(tempK) + ss.ck = ck + return ss +} + +func mixHash(ss *symmetricstate, data []byte) *symmetricstate { + ss.h = getHash(ss.h[:], data) + return ss +} + +func mixKeyAndHash(ss *symmetricstate, ikm [32]byte) *symmetricstate { + var tempH [32]byte + var tempK [32]byte + ss.ck, tempH, tempK = getHkdf(ss.ck, ikm[:]) + ss = mixHash(ss, tempH[:]) + ss.cs = initializeKey(tempK) + return ss +} + +func getHandshakeHash(ss *symmetricstate) [32]byte { + return ss.h +} + +func encryptAndHash(ss *symmetricstate, plaintext []byte) (*symmetricstate, []byte) { + var ciphertext []byte + if hasKey(&ss.cs) { + _, ciphertext = encryptWithAd(&ss.cs, ss.h[:], plaintext) + } else { + ciphertext = plaintext + } + ss = mixHash(ss, ciphertext) + return ss, ciphertext +} + +func decryptAndHash(ss *symmetricstate, ciphertext []byte) (*symmetricstate, []byte, bool) { + var plaintext []byte + var valid bool + if hasKey(&ss.cs) { + _, plaintext, valid = decryptWithAd(&ss.cs, ss.h[:], ciphertext) + } else { + plaintext, valid = ciphertext, true + } + ss = mixHash(ss, ciphertext) + return ss, plaintext, valid +} + +func split(ss *symmetricstate) (cipherstate, cipherstate) { + tempK1, tempK2, _ := getHkdf(ss.ck, []byte{}) + cs1 := initializeKey(tempK1) + cs2 := initializeKey(tempK2) + return cs1, cs2 +} + +/* HandshakeState */ + +func initializeInitiator(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) handshakestate { + var ss symmetricstate + var e Keypair + var re [32]byte + name := []byte("Noise_XX_25519_ChaChaPoly_BLAKE2s") + ss = initializeSymmetric(name) + mixHash(&ss, prologue) + return handshakestate{ss, s, e, rs, re, psk} +} + +func initializeResponder(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) handshakestate { + var ss symmetricstate + var e Keypair + var re [32]byte + name := []byte("Noise_XX_25519_ChaChaPoly_BLAKE2s") + ss = initializeSymmetric(name) + mixHash(&ss, prologue) + return handshakestate{ss, s, e, rs, re, psk} +} + +func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { + ne, ns, ciphertext := emptyKey, []byte{}, []byte{} + hs.e = GenerateKeypair() + ne = hs.e.public_key + mixHash(&hs.ss, ne[:]) + /* No PSK, so skipping mixKey */ + _, ciphertext = encryptAndHash(&hs.ss, payload) + messageBuffer := MessageBuffer{ne, ns, ciphertext} + return hs, messageBuffer +} + +func writeMessageB(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { + ne, ns, ciphertext := emptyKey, []byte{}, []byte{} + hs.e = GenerateKeypair() + ne = hs.e.public_key + mixHash(&hs.ss, ne[:]) + /* No PSK, so skipping mixKey */ + mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) + spk := make([]byte, len(hs.s.public_key)) + copy(spk[:], hs.s.public_key[:]) + _, ns = encryptAndHash(&hs.ss, spk) + mixKey(&hs.ss, dh(hs.s.private_key, hs.re)) + _, ciphertext = encryptAndHash(&hs.ss, payload) + messageBuffer := MessageBuffer{ne, ns, ciphertext} + return hs, messageBuffer +} + +func writeMessageC(hs *handshakestate, payload []byte) ([32]byte, MessageBuffer, cipherstate, cipherstate) { + ne, ns, ciphertext := emptyKey, []byte{}, []byte{} + spk := make([]byte, len(hs.s.public_key)) + copy(spk[:], hs.s.public_key[:]) + _, ns = encryptAndHash(&hs.ss, spk) + mixKey(&hs.ss, dh(hs.s.private_key, hs.re)) + _, ciphertext = encryptAndHash(&hs.ss, payload) + messageBuffer := MessageBuffer{ne, ns, ciphertext} + cs1, cs2 := split(&hs.ss) + return hs.ss.h, messageBuffer, cs1, cs2 +} + +func readMessageA(hs *handshakestate, message *MessageBuffer) (*handshakestate, []byte, bool) { + valid1 := true + if validatePublicKey(message.ne[:]) { + hs.re = message.ne + } + mixHash(&hs.ss, hs.re[:]) + /* No PSK, so skipping mixKey */ + _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) + return hs, plaintext, (valid1 && valid2) +} + +func readMessageB(hs *handshakestate, message *MessageBuffer) (*handshakestate, []byte, bool) { + valid1 := true + if validatePublicKey(message.ne[:]) { + hs.re = message.ne + } + mixHash(&hs.ss, hs.re[:]) + /* No PSK, so skipping mixKey */ + mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) + _, ns, valid1 := decryptAndHash(&hs.ss, message.ns) + if valid1 && len(ns) == 32 && validatePublicKey(message.ns[:]) { + copy(hs.rs[:], ns) + } + mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) + _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) + return hs, plaintext, (valid1 && valid2) +} + +func readMessageC(hs *handshakestate, message *MessageBuffer) ([32]byte, []byte, bool, cipherstate, cipherstate) { + valid1 := true + _, ns, valid1 := decryptAndHash(&hs.ss, message.ns) + if valid1 && len(ns) == 32 && validatePublicKey(message.ns[:]) { + copy(hs.rs[:], ns) + } + mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) + _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) + cs1, cs2 := split(&hs.ss) + return hs.ss.h, plaintext, (valid1 && valid2), cs1, cs2 +} + +/* ---------------------------------------------------------------- * + * PROCESSES * + * ---------------------------------------------------------------- */ + +func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *noisesession { + var session noisesession + psk := emptyKey + if initiator { + session.hs = initializeInitiator(prologue, s, rs, psk) + } else { + session.hs = initializeResponder(prologue, s, rs, psk) + } + session.i = initiator + session.mc = 0 + return &session +} + +func SendMessage(session *noisesession, message []byte) (*noisesession, MessageBuffer) { + var messageBuffer MessageBuffer + if session.mc == 0 { + _, messageBuffer = writeMessageA(&session.hs, message) + } + if session.mc == 1 { + _, messageBuffer = writeMessageB(&session.hs, message) + } + if session.mc == 2 { + session.h, messageBuffer, session.cs1, session.cs2 = writeMessageC(&session.hs, message) + session.hs = handshakestate{} + } + session.mc = session.mc + 1 + return session, messageBuffer +} + +func RecvMessage(session *noisesession, message *MessageBuffer) (*noisesession, []byte, bool) { + var plaintext []byte + var valid bool + if session.mc == 0 { + _, plaintext, valid = readMessageA(&session.hs, message) + } + if session.mc == 1 { + _, plaintext, valid = readMessageB(&session.hs, message) + } + if session.mc == 2 { + session.h, plaintext, valid, session.cs1, session.cs2 = readMessageC(&session.hs, message) + session.hs = handshakestate{} + } + session.mc = session.mc + 1 + return session, plaintext, valid +} \ No newline at end of file From dd1d543b4dc60afd196ea2b452d2528385b937b4 Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 17:46:12 -0400 Subject: [PATCH 1432/3965] add XX test --- p2p/security/noise/protocol.go | 6 ++- p2p/security/noise/xx/XX_test.go | 68 ++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 p2p/security/noise/xx/XX_test.go diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 1b55aea7de..3ff33043fe 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -11,7 +11,7 @@ import ( "github.com/libp2p/go-libp2p-core/sec" proto "github.com/gogo/protobuf/proto" - xx "github.com/noot/noise/xx" + xx "github.com/ChainSafe/go-libp2p-noise/xx" ) type secureSession struct { @@ -92,7 +92,9 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("read from conn fail: %s", err) } - + var plaintext []byte + var valid bool + ns, plaintext, valid = xx.RecvMessage() } return nil diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go new file mode 100644 index 0000000000..42552b2c13 --- /dev/null +++ b/p2p/security/noise/xx/XX_test.go @@ -0,0 +1,68 @@ +package xx + +import ( + "encoding/hex" + "testing" + + "github.com/ChainSafe/go-libp2p-noise" +) + +func TestGetHkdf(t *testing.T) { + ck := [32]byte{} + ckBytes, err := hex.DecodeString("4e6f6973655f58585f32353531395f58436861436861506f6c795f53484132353600000000000000000000000000000000000000000000000000000000000000") + if err != nil { + t.Fatal(err) + } + copy(ck[:], ckBytes) + + ikm, err := hex.DecodeString("a3eae50ea37a47e8a7aa0c7cd8e16528670536dcd538cebfd724fb68ce44f1910ad898860666227d4e8dd50d22a9a64d1c0a6f47ace092510161e9e442953da3") + if err != nil { + t.Fatal(err) + } + + a, b, c := getHkdf(ck, ikm) + t.Logf("%x", a) + t.Logf("%x", b) + t.Logf("%x", c) +} + +func TestHandshake(t *testing.T) { + // generate local static noise key + kp_init := GenerateKeypair() + kp_remote := GenerateKeypair() + + prologue := []byte("/noise/0.0.0") + + // new XX noise session + ns_init := InitSession(true, prologue, kp_init, kp_remote.PublicKey()) + + // create payload + payload := new(noise.NoiseHandshakePayload) + msg, err := proto.Marshal(payload) + if err != nil { + return fmt.Errorf("proto marshal payload fail: %s", err) + } + + var msgbuf MessageBuffer + ns, msgbuf = SendMessage(ns, msg) + + encMsgBuf := msgbuf.Encode0() + if len(encMsgBuf) != 56 { + return fmt.Errorf("enc msg buf: len does not equal 56") + } + + _, err = s.insecure.Write(encMsgBuf) + if err != nil { + return fmt.Errorf("write to conn fail: %s", err) + } + + buf := make([]byte, 144) + _, err = s.insecure.Read(buf) + if err != nil { + return fmt.Errorf("read from conn fail: %s", err) + } + + var plaintext []byte + var valid bool + ns, plaintext, valid = RecvMessage() +} \ No newline at end of file From 49e1855de02f2f74b5022b383d423d1908253359 Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 18:11:30 -0400 Subject: [PATCH 1433/3965] begin xx handshake test --- p2p/security/noise/{ => pb}/payload.pb.go | 0 p2p/security/noise/{ => pb}/payload.proto | 2 +- p2p/security/noise/protocol.go | 4 +- p2p/security/noise/transport.go | 2 - p2p/security/noise/xx/XX.noise.go | 18 +++++ p2p/security/noise/xx/XX_test.go | 90 ++++++++++++++++++----- 6 files changed, 91 insertions(+), 25 deletions(-) rename p2p/security/noise/{ => pb}/payload.pb.go (100%) rename p2p/security/noise/{ => pb}/payload.proto (92%) diff --git a/p2p/security/noise/payload.pb.go b/p2p/security/noise/pb/payload.pb.go similarity index 100% rename from p2p/security/noise/payload.pb.go rename to p2p/security/noise/pb/payload.pb.go diff --git a/p2p/security/noise/payload.proto b/p2p/security/noise/pb/payload.proto similarity index 92% rename from p2p/security/noise/payload.proto rename to p2p/security/noise/pb/payload.proto index dbaacb5196..5e3e764ae5 100644 --- a/p2p/security/noise/payload.proto +++ b/p2p/security/noise/pb/payload.proto @@ -1,5 +1,5 @@ syntax = "proto3"; -package noise; +package pb; message NoiseHandshakePayload { bytes libp2p_key = 1; diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 3ff33043fe..009fdb41a2 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -12,6 +12,7 @@ import ( proto "github.com/gogo/protobuf/proto" xx "github.com/ChainSafe/go-libp2p-noise/xx" + pb "github.com/ChainSafe/go-libp2p-noise/pb" ) type secureSession struct { @@ -59,7 +60,6 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // generate local static noise key kp := xx.GenerateKeypair() - //pub := xx.GeneratePublicKey(kp.PrivKey()) // new XX noise session ns := xx.InitSession(s.initiator, s.prologue, kp, remotePub) @@ -67,7 +67,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // send initial payload message if s.initiator { // create payload - payload := new(NoiseHandshakePayload) + payload := new(pb.NoiseHandshakePayload) msg, err := proto.Marshal(payload) if err != nil { return fmt.Errorf("proto marshal payload fail: %s", err) diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 6baac2d605..305cd8af96 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -14,8 +14,6 @@ const ID = "/noise/0.0.0" type Transport struct { LocalID peer.ID PrivateKey crypto.PrivKey - - //Payload NoiseHandshakePayload } // SecureInbound runs noise handshake as a server diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 06171ceafd..3373fbc1d6 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -87,6 +87,14 @@ func (kp Keypair) PrivKey() [32]byte{ return kp.private_key } +func NewMessageBuffer(ne [32]byte, ns []byte, ciphertext []byte) MessageBuffer { + return MessageBuffer{ + ne: ne, + ns: ns, + ciphertext: ciphertext, + } +} + /* ---------------------------------------------------------------- * * CONSTANTS * * ---------------------------------------------------------------- */ @@ -125,6 +133,16 @@ func (mb *MessageBuffer) Encode0() []byte { return enc } +func (mb *MessageBuffer) Encode1() []byte { + enc := []byte{} + + enc = append(enc, mb.ne[:]...) + enc = append(enc, mb.ciphertext...) + + return enc +} + + func validatePublicKey(k []byte) bool { forbiddenCurveValues := [12][]byte{ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 42552b2c13..897deb62ed 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -1,10 +1,12 @@ package xx import ( + "crypto/rand" "encoding/hex" "testing" - - "github.com/ChainSafe/go-libp2p-noise" + proto "github.com/gogo/protobuf/proto" + pb "github.com/ChainSafe/go-libp2p-noise/pb" + "github.com/libp2p/go-libp2p-core/crypto" ) func TestGetHkdf(t *testing.T) { @@ -29,40 +31,88 @@ func TestGetHkdf(t *testing.T) { func TestHandshake(t *testing.T) { // generate local static noise key kp_init := GenerateKeypair() - kp_remote := GenerateKeypair() + kp_resp := GenerateKeypair() + libp2p_pub_init,libp2p_priv_init,err := crypto.GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } prologue := []byte("/noise/0.0.0") - // new XX noise session - ns_init := InitSession(true, prologue, kp_init, kp_remote.PublicKey()) + // initiator: new XX noise session + ns_init := InitSession(true, prologue, kp_init, kp_resp.PubKey()) + + // responder: new XX noise session + ns_resp := InitSession(false, prologue, kp_resp, kp_init.PubKey()) + // stage 0: initiator // create payload - payload := new(noise.NoiseHandshakePayload) - msg, err := proto.Marshal(payload) + payload_init := new(pb.NoiseHandshakePayload) + payload_init.Libp2PKey = libp2p_pub_init.Raw() + payload_init_enc, err := proto.Marshal(payload_init) if err != nil { - return fmt.Errorf("proto marshal payload fail: %s", err) + t.Fatalf("proto marshal payload fail: %s", err) } + // send message var msgbuf MessageBuffer - ns, msgbuf = SendMessage(ns, msg) + msg := []byte{} + msg = append(msg, payload_init_enc[:]...) + ns_init, msgbuf = SendMessage(ns_init, msg) - encMsgBuf := msgbuf.Encode0() - if len(encMsgBuf) != 56 { - return fmt.Errorf("enc msg buf: len does not equal 56") + t.Logf("stage 0 msgbuf: %v", msgbuf) + + // stage 0: responder + var plaintext []byte + var valid bool + ns_resp, plaintext, valid = RecvMessage(ns_resp, &msgbuf) + if !valid { + t.Fatalf("stage 0 receive not valid") } - _, err = s.insecure.Write(encMsgBuf) + t.Logf("stage 0 resp payload: %x", plaintext) + + // stage 1: responder + // create payload + payload_resp := new(pb.NoiseHandshakePayload) + //payload.Libp2PKey() + payload_resp_enc, err := proto.Marshal(payload_resp) if err != nil { - return fmt.Errorf("write to conn fail: %s", err) + t.Fatalf("proto marshal payload fail: %s", err) } + msg = append(msg, payload_resp_enc[:]...) + ns_resp, msgbuf = SendMessage(ns_resp, msg) - buf := make([]byte, 144) - _, err = s.insecure.Read(buf) + t.Logf("stage 1 msgbuf: %v", msgbuf) + + // stage 1: initiator + ns_init, plaintext, valid = RecvMessage(ns_init, &msgbuf) + if !valid { + t.Fatalf("stage 1 receive not valid") + } + + t.Logf("stage 1 resp payload: %x", plaintext) + + // stage 2: initiator + payload_init = new(pb.NoiseHandshakePayload) + //payload.Libp2PKey() + payload_init_enc, err = proto.Marshal(payload_init) if err != nil { - return fmt.Errorf("read from conn fail: %s", err) + t.Fatalf("proto marshal payload fail: %s", err) } + + // send message + msg = append(msg, payload_init_enc[:]...) + ns_init, msgbuf = SendMessage(ns_init, msg) + + t.Logf("stage 2 msgbuf: %v", msgbuf) + + // stage 2: responder + ns_resp, plaintext, valid = RecvMessage(ns_resp, &msgbuf) + if !valid { + t.Fatalf("stage 2 receive not valid") + } + + t.Logf("stage 2 resp payload: %x", plaintext) - var plaintext []byte - var valid bool - ns, plaintext, valid = RecvMessage() } \ No newline at end of file From 7d68b940d4b1b5185e95d8b45be35666c8aceeef Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 18:23:40 -0400 Subject: [PATCH 1434/3965] finish xx handshake test --- p2p/security/noise/ik/IK.noise.go | 3 +- p2p/security/noise/info.go | 4 +-- p2p/security/noise/protocol.go | 44 +++++++++++++-------------- p2p/security/noise/transport.go | 4 +-- p2p/security/noise/xx/XX.noise.go | 16 +++++----- p2p/security/noise/xx/XX_test.go | 49 ++++++++++++++++++++++++------- 6 files changed, 74 insertions(+), 46 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index 97071e80bd..aeb8be0c39 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -123,6 +123,7 @@ func validatePublicKey(k []byte) bool { } return true } + /* ---------------------------------------------------------------- * * PRIMITIVES * * ---------------------------------------------------------------- */ @@ -431,4 +432,4 @@ func RecvMessage(session *noisesession, message *messagebuffer) (*noisesession, return session, plaintext, valid } -func main() {} \ No newline at end of file +func main() {} diff --git a/p2p/security/noise/info.go b/p2p/security/noise/info.go index 9f1625cc7d..80cc545e12 100644 --- a/p2p/security/noise/info.go +++ b/p2p/security/noise/info.go @@ -5,6 +5,6 @@ import ( ) type peerInfo struct { - staticKey crypto.PubKey + staticKey crypto.PubKey ephemeralKey crypto.PubKey -} \ No newline at end of file +} diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 009fdb41a2..4ca00928a5 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -6,37 +6,37 @@ import ( "net" "time" + proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" - proto "github.com/gogo/protobuf/proto" - xx "github.com/ChainSafe/go-libp2p-noise/xx" pb "github.com/ChainSafe/go-libp2p-noise/pb" + xx "github.com/ChainSafe/go-libp2p-noise/xx" ) type secureSession struct { insecure net.Conn initiator bool - prologue []byte + prologue []byte - localKey crypto.PrivKey - localPeer peer.ID + localKey crypto.PrivKey + localPeer peer.ID remotePeer peer.ID - local peerInfo + local peerInfo remote peerInfo } func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, insecure net.Conn, remote peer.ID, initiator bool) (sec.SecureConn, error) { s := &secureSession{ - insecure: insecure, - initiator: initiator, - prologue: []byte(ID), - localKey: privKey, - localPeer: local, + insecure: insecure, + initiator: initiator, + prologue: []byte(ID), + localKey: privKey, + localPeer: local, remotePeer: remote, } @@ -48,7 +48,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // TODO: check if static key for peer exists // if so, do XX; otherwise do IK - // PHASE 1: TRY XX + // PHASE 1: TRY XX // get remote static key remotePub := [32]byte{} @@ -72,7 +72,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { if err != nil { return fmt.Errorf("proto marshal payload fail: %s", err) } - + var msgbuf xx.MessageBuffer ns, msgbuf = xx.SendMessage(ns, msg) @@ -86,15 +86,15 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("write to conn fail: %s", err) } - buf := make([]byte, 144) - _, err = s.insecure.Read(buf) - if err != nil { - return fmt.Errorf("read from conn fail: %s", err) - } + // buf := make([]byte, 144) + // _, err = s.insecure.Read(buf) + // if err != nil { + // return fmt.Errorf("read from conn fail: %s", err) + // } - var plaintext []byte - var valid bool - ns, plaintext, valid = xx.RecvMessage() + // var plaintext []byte + // var valid bool + // ns, plaintext, valid = xx.RecvMessage(ns, ) } return nil @@ -146,4 +146,4 @@ func (s *secureSession) Write(in []byte) (int, error) { func (s *secureSession) Close() error { return s.insecure.Close() -} \ No newline at end of file +} diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 305cd8af96..650e449b16 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -12,7 +12,7 @@ import ( const ID = "/noise/0.0.0" type Transport struct { - LocalID peer.ID + LocalID peer.ID PrivateKey crypto.PrivKey } @@ -24,4 +24,4 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.S // SecureOutbound runs noise handshake as a client func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { return newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, p, true) -} \ No newline at end of file +} diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 3373fbc1d6..db8c3b1520 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -15,8 +15,8 @@ package xx import ( "crypto/rand" - "crypto/subtle" "crypto/sha256" + "crypto/subtle" "encoding/binary" //"golang.org/x/crypto/blake2s" @@ -74,23 +74,23 @@ type noisesession struct { func NewKeypair(pub [32]byte, priv [32]byte) Keypair { return Keypair{ - public_key: pub, + public_key: pub, private_key: priv, } } -func (kp Keypair) PubKey() [32]byte{ +func (kp Keypair) PubKey() [32]byte { return kp.public_key } -func (kp Keypair) PrivKey() [32]byte{ +func (kp Keypair) PrivKey() [32]byte { return kp.private_key } func NewMessageBuffer(ne [32]byte, ns []byte, ciphertext []byte) MessageBuffer { return MessageBuffer{ - ne: ne, - ns: ns, + ne: ne, + ns: ns, ciphertext: ciphertext, } } @@ -142,7 +142,6 @@ func (mb *MessageBuffer) Encode1() []byte { return enc } - func validatePublicKey(k []byte) bool { forbiddenCurveValues := [12][]byte{ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, @@ -166,6 +165,7 @@ func validatePublicKey(k []byte) bool { } return true } + /* ---------------------------------------------------------------- * * PRIMITIVES * * ---------------------------------------------------------------- */ @@ -496,4 +496,4 @@ func RecvMessage(session *noisesession, message *MessageBuffer) (*noisesession, } session.mc = session.mc + 1 return session, plaintext, valid -} \ No newline at end of file +} diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 897deb62ed..0684fdfedb 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -3,15 +3,15 @@ package xx import ( "crypto/rand" "encoding/hex" - "testing" - proto "github.com/gogo/protobuf/proto" pb "github.com/ChainSafe/go-libp2p-noise/pb" + proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" + "testing" ) func TestGetHkdf(t *testing.T) { ck := [32]byte{} - ckBytes, err := hex.DecodeString("4e6f6973655f58585f32353531395f58436861436861506f6c795f53484132353600000000000000000000000000000000000000000000000000000000000000") + ckBytes, err := hex.DecodeString("4e6f6973655f58585f32353531395f58436861436861506f6c795f53484132353600000000000000000000000000000000000000000000000000000000000000") if err != nil { t.Fatal(err) } @@ -33,11 +33,36 @@ func TestHandshake(t *testing.T) { kp_init := GenerateKeypair() kp_resp := GenerateKeypair() - libp2p_pub_init,libp2p_priv_init,err := crypto.GenerateEd25519Key(rand.Reader) + payload_string := []byte("noise-libp2p-static-key:") + prologue := []byte("/noise/0.0.0") + + // initiator setup + libp2p_priv_init, libp2p_pub_init, err := crypto.GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + libp2p_pub_init_raw, err := libp2p_pub_init.Raw() + if err != nil { + t.Fatal(err) + } + libp2p_init_signed_payload, err := libp2p_priv_init.Sign(payload_string) + if err != nil { + t.Fatal(err) + } + + // respoonder setup + libp2p_priv_resp, libp2p_pub_resp, err := crypto.GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + libp2p_pub_resp_raw, err := libp2p_pub_resp.Raw() + if err != nil { + t.Fatal(err) + } + libp2p_resp_signed_payload, err := libp2p_priv_resp.Sign(payload_string) if err != nil { t.Fatal(err) } - prologue := []byte("/noise/0.0.0") // initiator: new XX noise session ns_init := InitSession(true, prologue, kp_init, kp_resp.PubKey()) @@ -48,12 +73,13 @@ func TestHandshake(t *testing.T) { // stage 0: initiator // create payload payload_init := new(pb.NoiseHandshakePayload) - payload_init.Libp2PKey = libp2p_pub_init.Raw() + payload_init.Libp2PKey = libp2p_pub_init_raw + payload_init.NoiseStaticKeySignature = libp2p_init_signed_payload payload_init_enc, err := proto.Marshal(payload_init) if err != nil { t.Fatalf("proto marshal payload fail: %s", err) } - + // send message var msgbuf MessageBuffer msg := []byte{} @@ -75,7 +101,8 @@ func TestHandshake(t *testing.T) { // stage 1: responder // create payload payload_resp := new(pb.NoiseHandshakePayload) - //payload.Libp2PKey() + payload_init.Libp2PKey = libp2p_pub_resp_raw + payload_init.NoiseStaticKeySignature = libp2p_resp_signed_payload payload_resp_enc, err := proto.Marshal(payload_resp) if err != nil { t.Fatalf("proto marshal payload fail: %s", err) @@ -89,7 +116,7 @@ func TestHandshake(t *testing.T) { ns_init, plaintext, valid = RecvMessage(ns_init, &msgbuf) if !valid { t.Fatalf("stage 1 receive not valid") - } + } t.Logf("stage 1 resp payload: %x", plaintext) @@ -100,7 +127,7 @@ func TestHandshake(t *testing.T) { if err != nil { t.Fatalf("proto marshal payload fail: %s", err) } - + // send message msg = append(msg, payload_init_enc[:]...) ns_init, msgbuf = SendMessage(ns_init, msg) @@ -115,4 +142,4 @@ func TestHandshake(t *testing.T) { t.Logf("stage 2 resp payload: %x", plaintext) -} \ No newline at end of file +} From 9389119ed78bde09916a81d9ed7d2293ca6484ba Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 18:37:00 -0400 Subject: [PATCH 1435/3965] append noise key to signed payload --- p2p/security/noise/xx/XX_test.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 0684fdfedb..1f7b95d1ec 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -37,6 +37,7 @@ func TestHandshake(t *testing.T) { prologue := []byte("/noise/0.0.0") // initiator setup + init_pub := kp_init.PubKey() libp2p_priv_init, libp2p_pub_init, err := crypto.GenerateEd25519Key(rand.Reader) if err != nil { t.Fatal(err) @@ -45,12 +46,13 @@ func TestHandshake(t *testing.T) { if err != nil { t.Fatal(err) } - libp2p_init_signed_payload, err := libp2p_priv_init.Sign(payload_string) + libp2p_init_signed_payload, err := libp2p_priv_init.Sign(append(payload_string, init_pub[:]...)) if err != nil { t.Fatal(err) } // respoonder setup + resp_pub := kp_resp.PubKey() libp2p_priv_resp, libp2p_pub_resp, err := crypto.GenerateEd25519Key(rand.Reader) if err != nil { t.Fatal(err) @@ -59,7 +61,7 @@ func TestHandshake(t *testing.T) { if err != nil { t.Fatal(err) } - libp2p_resp_signed_payload, err := libp2p_priv_resp.Sign(payload_string) + libp2p_resp_signed_payload, err := libp2p_priv_resp.Sign(append(payload_string, resp_pub[:]...)) if err != nil { t.Fatal(err) } From 71e20a651386083e7d26139aabf8830305c6483f Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 18:43:26 -0400 Subject: [PATCH 1436/3965] remove payload in stage 2, add todo --- p2p/security/noise/xx/XX_test.go | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 1f7b95d1ec..f9784273e3 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -103,6 +103,7 @@ func TestHandshake(t *testing.T) { // stage 1: responder // create payload payload_resp := new(pb.NoiseHandshakePayload) + // TODO: an encrypted payload is sent in the response message payload_init.Libp2PKey = libp2p_pub_resp_raw payload_init.NoiseStaticKeySignature = libp2p_resp_signed_payload payload_resp_enc, err := proto.Marshal(payload_resp) @@ -123,13 +124,6 @@ func TestHandshake(t *testing.T) { t.Logf("stage 1 resp payload: %x", plaintext) // stage 2: initiator - payload_init = new(pb.NoiseHandshakePayload) - //payload.Libp2PKey() - payload_init_enc, err = proto.Marshal(payload_init) - if err != nil { - t.Fatalf("proto marshal payload fail: %s", err) - } - // send message msg = append(msg, payload_init_enc[:]...) ns_init, msgbuf = SendMessage(ns_init, msg) From 83c3fcb8ff47850032e045dcd1b3668cfa449e6e Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 18:44:19 -0400 Subject: [PATCH 1437/3965] fix spacing --- p2p/security/noise/pb/payload.proto | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/security/noise/pb/payload.proto b/p2p/security/noise/pb/payload.proto index 5e3e764ae5..42243b247f 100644 --- a/p2p/security/noise/pb/payload.proto +++ b/p2p/security/noise/pb/payload.proto @@ -4,6 +4,6 @@ package pb; message NoiseHandshakePayload { bytes libp2p_key = 1; bytes noise_static_key_signature = 2; - bytes libp2p_data = 3; - bytes libp2p_data_signature = 4; + bytes libp2p_data = 3; + bytes libp2p_data_signature = 4; } \ No newline at end of file From 58b207ffd60cf8f13ea0974c706cffd7e1ed0945 Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 19:00:32 -0400 Subject: [PATCH 1438/3965] add encode and decoding funcs for MessageBuffer --- p2p/security/noise/protocol.go | 30 +++++++++++++++++- p2p/security/noise/xx/XX.noise.go | 52 ++++++++++++++++++++++++++++++- p2p/security/noise/xx/XX_test.go | 10 ++++-- 3 files changed, 87 insertions(+), 5 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 4ca00928a5..89a0f41f23 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -15,6 +15,8 @@ import ( xx "github.com/ChainSafe/go-libp2p-noise/xx" ) +const payload_string = "noise-libp2p-static-key:" + type secureSession struct { insecure net.Conn @@ -48,7 +50,9 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // TODO: check if static key for peer exists // if so, do XX; otherwise do IK - // PHASE 1: TRY XX + // ******************************************** // + // ************** PHASE 1: TRY XX ************* // + // ******************************************** // // get remote static key remotePub := [32]byte{} @@ -61,18 +65,34 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // generate local static noise key kp := xx.GenerateKeypair() + // setup libp2p keys + localKeyRaw, err := s.LocalPublicKey().Raw() + if err != nil { + return fmt.Errorf("err getting raw pubkey: %s", err) + } + noise_pub := kp.PubKey() + signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) + if err != nil { + return fmt.Errorf("err signing payload: %s", err) + } + // new XX noise session ns := xx.InitSession(s.initiator, s.prologue, kp, remotePub) // send initial payload message if s.initiator { + // stage 0 // + // create payload payload := new(pb.NoiseHandshakePayload) + payload.Libp2PKey = localKeyRaw + payload.NoiseStaticKeySignature = signedPayload msg, err := proto.Marshal(payload) if err != nil { return fmt.Errorf("proto marshal payload fail: %s", err) } + // create message buffer to send var msgbuf xx.MessageBuffer ns, msgbuf = xx.SendMessage(ns, msg) @@ -81,11 +101,15 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("enc msg buf: len does not equal 56") } + // send message _, err = s.insecure.Write(encMsgBuf) if err != nil { return fmt.Errorf("write to conn fail: %s", err) } + // stage 1 // + + // read reply // buf := make([]byte, 144) // _, err = s.insecure.Read(buf) // if err != nil { @@ -112,6 +136,10 @@ func (s *secureSession) LocalPrivateKey() crypto.PrivKey { return s.localKey } +func (s *secureSession) LocalPublicKey() crypto.PubKey { + return s.localKey.GetPublic() +} + func (s *secureSession) Read(in []byte) (int, error) { return s.insecure.Read(in) } diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index db8c3b1520..23262bda0d 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -19,7 +19,7 @@ import ( "crypto/subtle" "encoding/binary" - //"golang.org/x/crypto/blake2s" + "errors" "golang.org/x/crypto/chacha20poly1305" "golang.org/x/crypto/curve25519" "golang.org/x/crypto/hkdf" @@ -124,6 +124,18 @@ func isEmptyKey(k [32]byte) bool { return subtle.ConstantTimeCompare(k[:], emptyKey[:]) == 1 } +func (mb *MessageBuffer) NE() [32]byte { + return mb.ne +} + +func (mb *MessageBuffer) NS() []byte { + return mb.ns +} + +func (mb *MessageBuffer) Ciphertext() []byte { + return mb.ciphertext +} + func (mb *MessageBuffer) Encode0() []byte { enc := []byte{} @@ -137,11 +149,49 @@ func (mb *MessageBuffer) Encode1() []byte { enc := []byte{} enc = append(enc, mb.ne[:]...) + enc = append(enc, mb.ns[:]...) + enc = append(enc, mb.ciphertext...) + + return enc +} + +func (mb *MessageBuffer) Encode2() []byte { + enc := []byte{} + + enc = append(enc, mb.ns[:]...) enc = append(enc, mb.ciphertext...) return enc } +// Decodes initial message into MessageBuffer +func Decode0(in []byte) (*MessageBuffer, error) { + if len(in) < 32 { + return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") + } + + mb := new(MessageBuffer) + copy(mb.ne[:], in[:32]) + copy(mb.ciphertext, in[32:]) + + return mb, nil +} + +// Decodes messages at stage 1 or 2 into MessageBuffer +func Decode1(in []byte) (*MessageBuffer, error) { + if len(in) < 80 { + return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") + } + + mb := new(MessageBuffer) + copy(mb.ne[:], in[:32]) + copy(mb.ns, in[32:80]) + copy(mb.ciphertext, in[80:]) + + return mb, nil +} + + func validatePublicKey(k []byte) bool { forbiddenCurveValues := [12][]byte{ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index f9784273e3..4c054b8764 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -89,6 +89,7 @@ func TestHandshake(t *testing.T) { ns_init, msgbuf = SendMessage(ns_init, msg) t.Logf("stage 0 msgbuf: %v", msgbuf) + t.Logf("stage 0 msgbuf ne len: %d", len(msgbuf.NE())) // stage 0: responder var plaintext []byte @@ -103,9 +104,8 @@ func TestHandshake(t *testing.T) { // stage 1: responder // create payload payload_resp := new(pb.NoiseHandshakePayload) - // TODO: an encrypted payload is sent in the response message - payload_init.Libp2PKey = libp2p_pub_resp_raw - payload_init.NoiseStaticKeySignature = libp2p_resp_signed_payload + payload_resp.Libp2PKey = libp2p_pub_resp_raw + payload_resp.NoiseStaticKeySignature = libp2p_resp_signed_payload payload_resp_enc, err := proto.Marshal(payload_resp) if err != nil { t.Fatalf("proto marshal payload fail: %s", err) @@ -114,6 +114,8 @@ func TestHandshake(t *testing.T) { ns_resp, msgbuf = SendMessage(ns_resp, msg) t.Logf("stage 1 msgbuf: %v", msgbuf) + t.Logf("stage 1 msgbuf ne len: %d", len(msgbuf.NE())) + t.Logf("stage 1 msgbuf ns len: %d", len(msgbuf.NS())) // stage 1: initiator ns_init, plaintext, valid = RecvMessage(ns_init, &msgbuf) @@ -129,6 +131,8 @@ func TestHandshake(t *testing.T) { ns_init, msgbuf = SendMessage(ns_init, msg) t.Logf("stage 2 msgbuf: %v", msgbuf) + t.Logf("stage 2 msgbuf ne len: %d", len(msgbuf.NE())) + t.Logf("stage 2 msgbuf ns len: %d", len(msgbuf.NS())) // stage 2: responder ns_resp, plaintext, valid = RecvMessage(ns_resp, &msgbuf) From 64590b093dc087b5d1e1ef73e298539bc8d89ac0 Mon Sep 17 00:00:00 2001 From: noot Date: Fri, 23 Aug 2019 19:44:53 -0400 Subject: [PATCH 1439/3965] initial XX handshake done --- p2p/security/noise/protocol.go | 145 ++++++++++++++++++++++++------ p2p/security/noise/transport.go | 2 + p2p/security/noise/xx/XX.noise.go | 24 ++--- 3 files changed, 127 insertions(+), 44 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 89a0f41f23..cdc81af25e 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -29,6 +29,8 @@ type secureSession struct { local peerInfo remote peerInfo + + ns *xx.NoiseSession } func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, insecure net.Conn, remote peer.ID, initiator bool) (sec.SecureConn, error) { @@ -45,6 +47,51 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey return s, nil } +func (s *secureSession) sendHandshakeMessage(payload []byte, initial_stage bool) error { + // create send message w payload + var msgbuf xx.MessageBuffer + s.ns, msgbuf = xx.SendMessage(s.ns, payload) + var encMsgBuf []byte + if initial_stage { + encMsgBuf = msgbuf.Encode0() + } else { + encMsgBuf = msgbuf.Encode1() + } + + // send message + _, err := s.insecure.Write(encMsgBuf) + if err != nil { + return fmt.Errorf("write to conn fail: %s", err) + } + + return nil +} + +func (s *secureSession) recvHandshakeMessage(buf []byte, initial_stage bool) (plaintext []byte, valid bool, err error) { + _, err = s.insecure.Read(buf) + if err != nil { + return nil, false, fmt.Errorf("read from conn fail: %s", err) + } + + var msgbuf *xx.MessageBuffer + if initial_stage { + msgbuf, err = xx.Decode0(buf) + } else { + msgbuf, err = xx.Decode1(buf) + } + + if err != nil { + return nil, false, fmt.Errorf("decode msg fail: %s", err) + } + + s.ns, plaintext, valid = xx.RecvMessage(s.ns, msgbuf) + if !valid { + return nil, false, fmt.Errorf("validation fail") + } + + return plaintext, valid, nil +} + func (s *secureSession) runHandshake(ctx context.Context) error { // TODO: check if static key for peer exists @@ -70,55 +117,97 @@ func (s *secureSession) runHandshake(ctx context.Context) error { if err != nil { return fmt.Errorf("err getting raw pubkey: %s", err) } + + // sign noise data for payload noise_pub := kp.PubKey() signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) if err != nil { return fmt.Errorf("err signing payload: %s", err) } + // create payload + payload := new(pb.NoiseHandshakePayload) + payload.Libp2PKey = localKeyRaw + payload.NoiseStaticKeySignature = signedPayload + payloadEnc, err := proto.Marshal(payload) + if err != nil { + return fmt.Errorf("proto marshal payload fail: %s", err) + } + // new XX noise session - ns := xx.InitSession(s.initiator, s.prologue, kp, remotePub) + s.ns = xx.InitSession(s.initiator, s.prologue, kp, remotePub) - // send initial payload message if s.initiator { // stage 0 // - // create payload - payload := new(pb.NoiseHandshakePayload) - payload.Libp2PKey = localKeyRaw - payload.NoiseStaticKeySignature = signedPayload - msg, err := proto.Marshal(payload) + err = s.sendHandshakeMessage(payloadEnc, true) + if err != nil { + return fmt.Errorf("stage 0 intiator fail: %s", err) + } + + // stage 1 // + + // read reply + buf := make([]byte, 80+(2*len(payloadEnc))) + plaintext, valid, err := s.recvHandshakeMessage(buf, false) if err != nil { - return fmt.Errorf("proto marshal payload fail: %s", err) + return fmt.Errorf("intiator stage 1 fail: %s", err) + } + + if !valid { + return fmt.Errorf("stage 1 initiator validation fail") } - // create message buffer to send - var msgbuf xx.MessageBuffer - ns, msgbuf = xx.SendMessage(ns, msg) + // TODO: check payload + fmt.Printf("%x", plaintext) - encMsgBuf := msgbuf.Encode0() - if len(encMsgBuf) != 56 { - return fmt.Errorf("enc msg buf: len does not equal 56") + // stage 2 // + + err = s.sendHandshakeMessage(nil, false) + if err != nil { + return fmt.Errorf("stage 2 intiator fail: %s", err) } - // send message - _, err = s.insecure.Write(encMsgBuf) + } else { + + // stage 0 // + + // read message + buf := make([]byte, 32+len(payloadEnc)) + plaintext, valid, err := s.recvHandshakeMessage(buf, false) if err != nil { - return fmt.Errorf("write to conn fail: %s", err) + return fmt.Errorf("stage 0 responder fail: %s", err) + } + + if !valid { + return fmt.Errorf("stage 0 responder validation fail") } + // TODO: check payload + fmt.Printf("%x", plaintext) + // stage 1 // - - // read reply - // buf := make([]byte, 144) - // _, err = s.insecure.Read(buf) - // if err != nil { - // return fmt.Errorf("read from conn fail: %s", err) - // } - - // var plaintext []byte - // var valid bool - // ns, plaintext, valid = xx.RecvMessage(ns, ) + + err = s.sendHandshakeMessage(payloadEnc, true) + if err != nil { + return fmt.Errorf("stage 1 responder fail: %s", err) + } + + // stage 2 // + + // read message + buf = make([]byte, 80+(3*len(payloadEnc))) + plaintext, valid, err = s.recvHandshakeMessage(buf, false) + if err != nil { + return fmt.Errorf("stage 2 responder fail: %s", err) + } + + if !valid { + return fmt.Errorf("stage 2 responder validation fail") + } + + // TODO: check payload + fmt.Printf("%x", plaintext) } return nil diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 650e449b16..b2ef88d8a2 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -11,6 +11,8 @@ import ( const ID = "/noise/0.0.0" +var _ sec.SecureTransport = &Transport{} + type Transport struct { LocalID peer.ID PrivateKey crypto.PrivKey diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 23262bda0d..ab80152007 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -63,7 +63,7 @@ type handshakestate struct { psk [32]byte } -type noisesession struct { +type NoiseSession struct { hs handshakestate h [32]byte cs1 cipherstate @@ -136,6 +136,7 @@ func (mb *MessageBuffer) Ciphertext() []byte { return mb.ciphertext } +// Encodes a MessageBuffer from stage 0 func (mb *MessageBuffer) Encode0() []byte { enc := []byte{} @@ -145,6 +146,7 @@ func (mb *MessageBuffer) Encode0() []byte { return enc } +// Encodes a MessageBuffer from stage 1 and 2 func (mb *MessageBuffer) Encode1() []byte { enc := []byte{} @@ -155,16 +157,7 @@ func (mb *MessageBuffer) Encode1() []byte { return enc } -func (mb *MessageBuffer) Encode2() []byte { - enc := []byte{} - - enc = append(enc, mb.ns[:]...) - enc = append(enc, mb.ciphertext...) - - return enc -} - -// Decodes initial message into MessageBuffer +// Decodes initial message (stage 0) into MessageBuffer func Decode0(in []byte) (*MessageBuffer, error) { if len(in) < 32 { return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") @@ -191,7 +184,6 @@ func Decode1(in []byte) (*MessageBuffer, error) { return mb, nil } - func validatePublicKey(k []byte) bool { forbiddenCurveValues := [12][]byte{ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, @@ -502,8 +494,8 @@ func readMessageC(hs *handshakestate, message *MessageBuffer) ([32]byte, []byte, * PROCESSES * * ---------------------------------------------------------------- */ -func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *noisesession { - var session noisesession +func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *NoiseSession { + var session NoiseSession psk := emptyKey if initiator { session.hs = initializeInitiator(prologue, s, rs, psk) @@ -515,7 +507,7 @@ func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *noise return &session } -func SendMessage(session *noisesession, message []byte) (*noisesession, MessageBuffer) { +func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageBuffer) { var messageBuffer MessageBuffer if session.mc == 0 { _, messageBuffer = writeMessageA(&session.hs, message) @@ -531,7 +523,7 @@ func SendMessage(session *noisesession, message []byte) (*noisesession, MessageB return session, messageBuffer } -func RecvMessage(session *noisesession, message *MessageBuffer) (*noisesession, []byte, bool) { +func RecvMessage(session *NoiseSession, message *MessageBuffer) (*NoiseSession, []byte, bool) { var plaintext []byte var valid bool if session.mc == 0 { From 304f2029aab3e49d9edb45468877f8e961326f77 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 05:18:43 -0400 Subject: [PATCH 1440/3965] handshake test for xx passes --- p2p/security/noise/info.go | 10 -- p2p/security/noise/protocol.go | 84 +++++++++--- p2p/security/noise/transport.go | 4 +- p2p/security/noise/transport_test.go | 185 +++++++++++++++++++++++++++ p2p/security/noise/xx/XX.noise.go | 19 ++- 5 files changed, 268 insertions(+), 34 deletions(-) delete mode 100644 p2p/security/noise/info.go create mode 100644 p2p/security/noise/transport_test.go diff --git a/p2p/security/noise/info.go b/p2p/security/noise/info.go deleted file mode 100644 index 80cc545e12..0000000000 --- a/p2p/security/noise/info.go +++ /dev/null @@ -1,10 +0,0 @@ -package noise - -import ( - "github.com/libp2p/go-libp2p-core/crypto" -) - -type peerInfo struct { - staticKey crypto.PubKey - ephemeralKey crypto.PubKey -} diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index cdc81af25e..ada6013561 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -3,9 +3,11 @@ package noise import ( "context" "fmt" + //"io" "net" "time" + log "github.com/ChainSafe/log15" proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" @@ -33,8 +35,13 @@ type secureSession struct { ns *xx.NoiseSession } -func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, insecure net.Conn, remote peer.ID, initiator bool) (sec.SecureConn, error) { +// TODO: after reading payloads in initial message, fill in the peerInfo +type peerInfo struct { + noiseKey crypto.PubKey // static noise key + libp2pKey crypto.PubKey +} +func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, insecure net.Conn, remote peer.ID, initiator bool) (sec.SecureConn, error) { s := &secureSession{ insecure: insecure, initiator: initiator, @@ -44,10 +51,14 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey remotePeer: remote, } - return s, nil + err := s.runHandshake(ctx) + + return s, err } func (s *secureSession) sendHandshakeMessage(payload []byte, initial_stage bool) error { + log.Debug("send handshake message", "initiator", s.initiator, "payload", payload, "payload len", len(payload), "initial_stage", initial_stage) + // create send message w payload var msgbuf xx.MessageBuffer s.ns, msgbuf = xx.SendMessage(s.ns, payload) @@ -58,9 +69,13 @@ func (s *secureSession) sendHandshakeMessage(payload []byte, initial_stage bool) encMsgBuf = msgbuf.Encode1() } + log.Debug("send handshake message", "intiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) + log.Debug("send handshake message", "intiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) + // send message _, err := s.insecure.Write(encMsgBuf) if err != nil { + log.Debug("send handshake message", "initiator", s.initiator, "error", err) return fmt.Errorf("write to conn fail: %s", err) } @@ -80,20 +95,33 @@ func (s *secureSession) recvHandshakeMessage(buf []byte, initial_stage bool) (pl msgbuf, err = xx.Decode1(buf) } + log.Debug("recv handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) + if err != nil { + log.Debug("recv handshake message decode", "initiator", s.initiator, "error", err) return nil, false, fmt.Errorf("decode msg fail: %s", err) } s.ns, plaintext, valid = xx.RecvMessage(s.ns, msgbuf) if !valid { + log.Debug("recv handshake message xx", "initiator", s.initiator, "error", "validation fail") return nil, false, fmt.Errorf("validation fail") } + log.Debug("recv handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + return plaintext, valid, nil } +func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { + s.remote.libp2pKey, err = crypto.UnmarshalEd25519PublicKey(key) + return err +} + func (s *secureSession) runHandshake(ctx context.Context) error { + // s.remote = remotePeerInfo + // TODO: check if static key for peer exists // if so, do XX; otherwise do IK @@ -102,16 +130,18 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // ******************************************** // // get remote static key - remotePub := [32]byte{} - remotePubRaw, err := s.RemotePublicKey().Raw() - if err != nil { - return fmt.Errorf("remote pubkey fail: %s", err) - } - copy(remotePub[:], remotePubRaw) + // remotePub := [32]byte{} + // remotePubRaw, err := s.RemotePublicKey().Raw() + // if err != nil { + // return fmt.Errorf("remote pubkey fail: %s", err) + // } + // copy(remotePub[:], remotePubRaw) // generate local static noise key kp := xx.GenerateKeypair() + log.Debug("xx handshake", "pubkey", kp.PubKey()) + // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Raw() if err != nil { @@ -135,7 +165,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } // new XX noise session - s.ns = xx.InitSession(s.initiator, s.prologue, kp, remotePub) + s.ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) if s.initiator { // stage 0 // @@ -148,7 +178,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // stage 1 // // read reply - buf := make([]byte, 80+(2*len(payloadEnc))) + buf := make([]byte, 96+(2*len(payloadEnc))) plaintext, valid, err := s.recvHandshakeMessage(buf, false) if err != nil { return fmt.Errorf("intiator stage 1 fail: %s", err) @@ -159,11 +189,24 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } // TODO: check payload - fmt.Printf("%x", plaintext) + log.Debug("stage 1 initiator", "payload", plaintext) + + nsp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nsp) + if err != nil { + return fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") + } + + log.Debug("stage 1 initiator", "libp2pkey", fmt.Sprintf("%x", nsp.GetLibp2PKey())) + + err = s.setRemotePeerInfo(nsp.GetLibp2PKey()) + if err != nil { + return fmt.Errorf("stage 1 initiator read remote libp2p key fail") + } // stage 2 // - err = s.sendHandshakeMessage(nil, false) + err = s.sendHandshakeMessage(append(plaintext, payloadEnc...), false) if err != nil { return fmt.Errorf("stage 2 intiator fail: %s", err) } @@ -174,7 +217,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // read message buf := make([]byte, 32+len(payloadEnc)) - plaintext, valid, err := s.recvHandshakeMessage(buf, false) + plaintext, valid, err := s.recvHandshakeMessage(buf, true) if err != nil { return fmt.Errorf("stage 0 responder fail: %s", err) } @@ -184,11 +227,16 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } // TODO: check payload - fmt.Printf("%x", plaintext) + log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) + + err = s.setRemotePeerInfo(plaintext[:32]) + if err != nil { + return fmt.Errorf("stage 0 responder read remote libp2p key fail") + } // stage 1 // - err = s.sendHandshakeMessage(payloadEnc, true) + err = s.sendHandshakeMessage(append(plaintext, payloadEnc...), false) if err != nil { return fmt.Errorf("stage 1 responder fail: %s", err) } @@ -196,7 +244,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // stage 2 // // read message - buf = make([]byte, 80+(3*len(payloadEnc))) + buf = make([]byte, 96+(3*len(payloadEnc))) plaintext, valid, err = s.recvHandshakeMessage(buf, false) if err != nil { return fmt.Errorf("stage 2 responder fail: %s", err) @@ -207,7 +255,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } // TODO: check payload - fmt.Printf("%x", plaintext) + log.Debug("stage 2 responder", "payload", plaintext) } return nil @@ -242,7 +290,7 @@ func (s *secureSession) RemotePeer() peer.ID { } func (s *secureSession) RemotePublicKey() crypto.PubKey { - return s.remote.staticKey + return s.remote.libp2pKey } func (s *secureSession) SetDeadline(t time.Time) error { diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index b2ef88d8a2..b2b3e1e9b6 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -18,12 +18,12 @@ type Transport struct { PrivateKey crypto.PrivKey } -// SecureInbound runs noise handshake as a server +// SecureInbound runs noise handshake as the responder func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { return newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, "", false) } -// SecureOutbound runs noise handshake as a client +// SecureOutbound runs noise handshake as the initiator func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { return newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, p, true) } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go new file mode 100644 index 0000000000..b39503eeb6 --- /dev/null +++ b/p2p/security/noise/transport_test.go @@ -0,0 +1,185 @@ +package noise + +import ( + "bytes" + "context" + "io" + "net" + "testing" + + crypto "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/sec" +) + +func newTestTransport(t *testing.T, typ, bits int) *Transport { + priv, pub, err := crypto.GenerateKeyPair(typ, bits) + if err != nil { + t.Fatal(err) + } + id, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Fatal(err) + } + return &Transport{ + LocalID: id, + PrivateKey: priv, + } +} + +// Create a new pair of connected TCP sockets. +func newConnPair(t *testing.T) (net.Conn, net.Conn) { + lstnr, err := net.Listen("tcp", "localhost:0") + if err != nil { + t.Fatalf("Failed to listen: %v", err) + return nil, nil + } + + var clientErr error + var client net.Conn + addr := lstnr.Addr() + done := make(chan struct{}) + + go func() { + defer close(done) + client, clientErr = net.Dial(addr.Network(), addr.String()) + }() + + server, err := lstnr.Accept() + <-done + + lstnr.Close() + + if err != nil { + t.Fatalf("Failed to accept: %v", err) + } + + if clientErr != nil { + t.Fatalf("Failed to connect: %v", clientErr) + } + + return client, server +} + +func connect(t *testing.T, initTransport, respTransport *Transport) (sec.SecureConn, sec.SecureConn) { + init, resp := newConnPair(t) + + var respConn sec.SecureConn + var respErr error + done := make(chan struct{}) + go func() { + defer close(done) + respConn, respErr = respTransport.SecureOutbound(context.TODO(), resp, initTransport.LocalID) + }() + + initConn, initErr := initTransport.SecureInbound(context.TODO(), init) + <-done + + if initErr != nil { + t.Fatal(initErr) + } + + if respErr != nil { + t.Fatal(respErr) + } + + return initConn, respConn +} + +func TestIDs(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + if initConn.LocalPeer() != initTransport.LocalID { + t.Fatal("Initiator Local Peer ID mismatch.") + } + + if respConn.RemotePeer() != initTransport.LocalID { + t.Fatal("Responder Remote Peer ID mismatch.") + } + + if initConn.LocalPeer() != respConn.RemotePeer() { + t.Fatal("Responder Local Peer ID mismatch.") + } + + // TODO: check after stage 0 of handshake if updated + // if initConn.RemotePeer() != respTransport.LocalID { + // t.Error("Initiator Remote Peer ID mismatch.") + // t.Log(initConn.RemotePeer()) + // t.Log(respTransport.LocalID) + // } +} + +func TestKeys(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + sk := respConn.LocalPrivateKey() + pk := sk.GetPublic() + + if !sk.Equals(respTransport.PrivateKey) { + t.Error("Private key Mismatch.") + } + + if !pk.Equals(initConn.RemotePublicKey()) { + t.Errorf("Public key mismatch. expected %x got %x", pk, initConn.RemotePublicKey()) + } +} + +func TestRW(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + before := []byte("hello world") + _, err := initConn.Write(before) + if err != nil { + t.Fatal(err) + } + + after := make([]byte, len(before)) + _, err = io.ReadFull(respConn, after) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(before, after) { + t.Errorf("Message mismatch. %v != %v", before, after) + } +} + +func TestHandshake(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + before := []byte("hello world") + _, err := initConn.Write(before) + if err != nil { + t.Fatal(err) + } + + after := make([]byte, len(before)) + _, err = io.ReadFull(respConn, after) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(before, after) { + t.Errorf("Message mismatch. %v != %v", before, after) + } +} diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index ab80152007..d4ad9e1c9a 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -26,6 +26,7 @@ import ( "hash" "io" "math" + //log "github.com/ChainSafe/log15" ) /* ---------------------------------------------------------------- * @@ -151,9 +152,12 @@ func (mb *MessageBuffer) Encode1() []byte { enc := []byte{} enc = append(enc, mb.ne[:]...) - enc = append(enc, mb.ns[:]...) + enc = append(enc, mb.ns...) enc = append(enc, mb.ciphertext...) + // log.Debug("XX_Encode1", "ne", mb.ne) + // log.Debug("XX_Encode1", "ns", mb.ns) + return enc } @@ -163,9 +167,11 @@ func Decode0(in []byte) (*MessageBuffer, error) { return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") } + //log.Debug("XX_Decode0", "in", in) mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) - copy(mb.ciphertext, in[32:]) + mb.ciphertext = in[32:] + //log.Debug("XX_Decode0", "mb", mb) return mb, nil } @@ -176,10 +182,15 @@ func Decode1(in []byte) (*MessageBuffer, error) { return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") } + // log.Debug("XX_Decode1", "in", in) + // log.Debug("XX_Decode1", "ns", in[32:80]) + mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) - copy(mb.ns, in[32:80]) - copy(mb.ciphertext, in[80:]) + mb.ns = in[32:80] + mb.ciphertext = in[80:] + // copy(mb.ns,) + // copy(mb.ciphertext,) return mb, nil } From 763a7efc3db1185c91470431a08afcd3d44071bc Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 05:36:32 -0400 Subject: [PATCH 1441/3965] marshal and unmarshal remote pubkey from payload ok --- p2p/security/noise/protocol.go | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index ada6013561..7336bc0ebf 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -69,7 +69,7 @@ func (s *secureSession) sendHandshakeMessage(payload []byte, initial_stage bool) encMsgBuf = msgbuf.Encode1() } - log.Debug("send handshake message", "intiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) + log.Debug("send handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) log.Debug("send handshake message", "intiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) // send message @@ -114,7 +114,7 @@ func (s *secureSession) recvHandshakeMessage(buf []byte, initial_stage bool) (pl } func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { - s.remote.libp2pKey, err = crypto.UnmarshalEd25519PublicKey(key) + s.remote.libp2pKey, err = crypto.UnmarshalPublicKey(key) return err } @@ -143,11 +143,13 @@ func (s *secureSession) runHandshake(ctx context.Context) error { log.Debug("xx handshake", "pubkey", kp.PubKey()) // setup libp2p keys - localKeyRaw, err := s.LocalPublicKey().Raw() + localKeyRaw, err := s.LocalPublicKey().Bytes() if err != nil { return fmt.Errorf("err getting raw pubkey: %s", err) } + log.Debug("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) + // sign noise data for payload noise_pub := kp.PubKey() signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) @@ -201,6 +203,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { err = s.setRemotePeerInfo(nsp.GetLibp2PKey()) if err != nil { + log.Error("stage 1 initiator set remote peer info", "err", err) return fmt.Errorf("stage 1 initiator read remote libp2p key fail") } @@ -229,8 +232,17 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // TODO: check payload log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) - err = s.setRemotePeerInfo(plaintext[:32]) + nsp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nsp) + if err != nil { + return fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + } + + log.Debug("stage 0 responder", "libp2pkey", fmt.Sprintf("%x", nsp.GetLibp2PKey())) + + err = s.setRemotePeerInfo(nsp.GetLibp2PKey()) if err != nil { + log.Error("stage 0 responder set remote peer info", "err", err) return fmt.Errorf("stage 0 responder read remote libp2p key fail") } From 0239517a98fa8f92be1a6eaedd9039cfdce704af Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 05:52:13 -0400 Subject: [PATCH 1442/3965] remove unneeded payload data, check and set remote pubkeys --- p2p/security/noise/protocol.go | 42 +++++++++++++++------------- p2p/security/noise/transport_test.go | 8 ++---- p2p/security/noise/xx/XX_test.go | 4 +-- 3 files changed, 28 insertions(+), 26 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 7336bc0ebf..0e98de962b 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -70,7 +70,7 @@ func (s *secureSession) sendHandshakeMessage(payload []byte, initial_stage bool) } log.Debug("send handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - log.Debug("send handshake message", "intiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) + log.Debug("send handshake message", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) // send message _, err := s.insecure.Write(encMsgBuf) @@ -118,9 +118,12 @@ func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { return err } -func (s *secureSession) runHandshake(ctx context.Context) error { +func (s *secureSession) setRemotePeerID(key crypto.PubKey) (err error) { + s.remotePeer, err = peer.IDFromPublicKey(key) + return err +} - // s.remote = remotePeerInfo +func (s *secureSession) runHandshake(ctx context.Context) error { // TODO: check if static key for peer exists // if so, do XX; otherwise do IK @@ -129,14 +132,6 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // ************** PHASE 1: TRY XX ************* // // ******************************************** // - // get remote static key - // remotePub := [32]byte{} - // remotePubRaw, err := s.RemotePublicKey().Raw() - // if err != nil { - // return fmt.Errorf("remote pubkey fail: %s", err) - // } - // copy(remotePub[:], remotePubRaw) - // generate local static noise key kp := xx.GenerateKeypair() @@ -180,7 +175,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // stage 1 // // read reply - buf := make([]byte, 96+(2*len(payloadEnc))) + buf := make([]byte, 96+len(payloadEnc)) plaintext, valid, err := s.recvHandshakeMessage(buf, false) if err != nil { return fmt.Errorf("intiator stage 1 fail: %s", err) @@ -199,17 +194,23 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") } - log.Debug("stage 1 initiator", "libp2pkey", fmt.Sprintf("%x", nsp.GetLibp2PKey())) - err = s.setRemotePeerInfo(nsp.GetLibp2PKey()) if err != nil { log.Error("stage 1 initiator set remote peer info", "err", err) return fmt.Errorf("stage 1 initiator read remote libp2p key fail") } + pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) + if pid != s.remotePeer { + log.Error("stage 1 initiator check remote peer id err", "expected", s.remotePeer, "got", pid) + } else if err != nil { + log.Error("stage 1 initiator check remote peer id", "err", err) + } + // stage 2 // - err = s.sendHandshakeMessage(append(plaintext, payloadEnc...), false) + //err = s.sendHandshakeMessage(append(plaintext, payloadEnc...), false) + err = s.sendHandshakeMessage(nil, false) if err != nil { return fmt.Errorf("stage 2 intiator fail: %s", err) } @@ -238,17 +239,20 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") } - log.Debug("stage 0 responder", "libp2pkey", fmt.Sprintf("%x", nsp.GetLibp2PKey())) - err = s.setRemotePeerInfo(nsp.GetLibp2PKey()) if err != nil { log.Error("stage 0 responder set remote peer info", "err", err) return fmt.Errorf("stage 0 responder read remote libp2p key fail") } + err = s.setRemotePeerID(s.RemotePublicKey()) + if err != nil { + log.Error("stage 0 responder set remote peer id", "err", err) + } + // stage 1 // - err = s.sendHandshakeMessage(append(plaintext, payloadEnc...), false) + err = s.sendHandshakeMessage(payloadEnc, false) if err != nil { return fmt.Errorf("stage 1 responder fail: %s", err) } @@ -256,7 +260,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // stage 2 // // read message - buf = make([]byte, 96+(3*len(payloadEnc))) + buf = make([]byte, 96) plaintext, valid, err = s.recvHandshakeMessage(buf, false) if err != nil { return fmt.Errorf("stage 2 responder fail: %s", err) diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index b39503eeb6..7ff71c4726 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -107,11 +107,9 @@ func TestIDs(t *testing.T) { } // TODO: check after stage 0 of handshake if updated - // if initConn.RemotePeer() != respTransport.LocalID { - // t.Error("Initiator Remote Peer ID mismatch.") - // t.Log(initConn.RemotePeer()) - // t.Log(respTransport.LocalID) - // } + if initConn.RemotePeer() != respTransport.LocalID { + t.Errorf("Initiator Remote Peer ID mismatch. expected %x got %x", respTransport.LocalID, initConn.RemotePeer()) + } } func TestKeys(t *testing.T) { diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 4c054b8764..ad127facd1 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -127,8 +127,8 @@ func TestHandshake(t *testing.T) { // stage 2: initiator // send message - msg = append(msg, payload_init_enc[:]...) - ns_init, msgbuf = SendMessage(ns_init, msg) + //msg = append(msg, payload_init_enc[:]...) + ns_init, msgbuf = SendMessage(ns_init, nil) t.Logf("stage 2 msgbuf: %v", msgbuf) t.Logf("stage 2 msgbuf ne len: %d", len(msgbuf.NE())) From 7d29cd149b7f29912367a18c3b2feda831652a7e Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 06:01:57 -0400 Subject: [PATCH 1443/3965] add verification of payload --- p2p/security/noise/protocol.go | 46 +++++++++++++++++++++++++------ p2p/security/noise/xx/XX.noise.go | 4 +++ 2 files changed, 42 insertions(+), 8 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 0e98de962b..76038f65ba 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -123,6 +123,20 @@ func (s *secureSession) setRemotePeerID(key crypto.PubKey) (err error) { return err } +func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKey [32]byte) (err error) { + sig := payload.GetNoiseStaticKeySignature() + msg := append([]byte(payload_string), noiseKey[:]...) + + ok, err := s.RemotePublicKey().Verify(msg, sig) + if err != nil { + return err + } else if !ok { + return fmt.Errorf("did not verify payload") + } + + return nil +} + func (s *secureSession) runHandshake(ctx context.Context) error { // TODO: check if static key for peer exists @@ -185,21 +199,23 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("stage 1 initiator validation fail") } - // TODO: check payload log.Debug("stage 1 initiator", "payload", plaintext) - nsp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nsp) + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nhp) if err != nil { return fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") } - err = s.setRemotePeerInfo(nsp.GetLibp2PKey()) + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { log.Error("stage 1 initiator set remote peer info", "err", err) return fmt.Errorf("stage 1 initiator read remote libp2p key fail") } + // assert that remote peer ID matches libp2p public key pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) if pid != s.remotePeer { log.Error("stage 1 initiator check remote peer id err", "expected", s.remotePeer, "got", pid) @@ -207,6 +223,12 @@ func (s *secureSession) runHandshake(ctx context.Context) error { log.Error("stage 1 initiator check remote peer id", "err", err) } + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.ns.RemoteKey()) + if err != nil { + log.Error("stage 1 initiator verify payload", "err", err) + } + // stage 2 // //err = s.sendHandshakeMessage(append(plaintext, payloadEnc...), false) @@ -230,26 +252,34 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("stage 0 responder validation fail") } - // TODO: check payload log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) - nsp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nsp) + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nhp) if err != nil { return fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") } - err = s.setRemotePeerInfo(nsp.GetLibp2PKey()) + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { log.Error("stage 0 responder set remote peer info", "err", err) return fmt.Errorf("stage 0 responder read remote libp2p key fail") } + // assert that remote peer ID matches libp2p key err = s.setRemotePeerID(s.RemotePublicKey()) if err != nil { log.Error("stage 0 responder set remote peer id", "err", err) } + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.ns.RemoteKey()) + if err != nil { + log.Error("stage 1 initiator verify payload", "err", err) + } + // stage 1 // err = s.sendHandshakeMessage(payloadEnc, false) diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index d4ad9e1c9a..3c4351cf07 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -88,6 +88,10 @@ func (kp Keypair) PrivKey() [32]byte { return kp.private_key } +func (ns *NoiseSession) RemoteKey() [32]byte { + return ns.hs.rs +} + func NewMessageBuffer(ne [32]byte, ns []byte, ciphertext []byte) MessageBuffer { return MessageBuffer{ ne: ne, From dd04220b1de1411b7aaa2952551b8c5798fb02b8 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 06:09:26 -0400 Subject: [PATCH 1444/3965] update IK to export needed types --- p2p/security/noise/ik/IK.noise.go | 101 +++++++++++++++++++++++++----- 1 file changed, 86 insertions(+), 15 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index aeb8be0c39..5b7c48136a 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -12,12 +12,13 @@ IK: * PARAMETERS * * ---------------------------------------------------------------- */ -package main +package ik import ( "crypto/rand" "crypto/subtle" "encoding/binary" + "errors" "golang.org/x/crypto/blake2s" "golang.org/x/crypto/chacha20poly1305" "golang.org/x/crypto/curve25519" @@ -36,7 +37,7 @@ type keypair struct { private_key [32]byte } -type messagebuffer struct { +type MessageBuffer struct { ne [32]byte ns []byte ciphertext []byte @@ -62,7 +63,7 @@ type handshakestate struct { psk [32]byte } -type noisesession struct { +type NoiseSession struct { hs handshakestate h [32]byte cs1 cipherstate @@ -100,6 +101,76 @@ func isEmptyKey(k [32]byte) bool { return subtle.ConstantTimeCompare(k[:], emptyKey[:]) == 1 } +func (mb *MessageBuffer) NE() [32]byte { + return mb.ne +} + +func (mb *MessageBuffer) NS() []byte { + return mb.ns +} + +func (mb *MessageBuffer) Ciphertext() []byte { + return mb.ciphertext +} + +// Encodes a MessageBuffer from stage 0 +func (mb *MessageBuffer) Encode0() []byte { + enc := []byte{} + + enc = append(enc, mb.ne[:]...) + enc = append(enc, mb.ciphertext...) + + return enc +} + +// Encodes a MessageBuffer from stage 1 and 2 +func (mb *MessageBuffer) Encode1() []byte { + enc := []byte{} + + enc = append(enc, mb.ne[:]...) + enc = append(enc, mb.ns...) + enc = append(enc, mb.ciphertext...) + + // log.Debug("XX_Encode1", "ne", mb.ne) + // log.Debug("XX_Encode1", "ns", mb.ns) + + return enc +} + +// Decodes initial message (stage 0) into MessageBuffer +func Decode0(in []byte) (*MessageBuffer, error) { + if len(in) < 32 { + return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") + } + + //log.Debug("XX_Decode0", "in", in) + mb := new(MessageBuffer) + copy(mb.ne[:], in[:32]) + mb.ciphertext = in[32:] + //log.Debug("XX_Decode0", "mb", mb) + + return mb, nil +} + +// Decodes messages at stage 1 or 2 into MessageBuffer +func Decode1(in []byte) (*MessageBuffer, error) { + if len(in) < 80 { + return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") + } + + // log.Debug("XX_Decode1", "in", in) + // log.Debug("XX_Decode1", "ns", in[32:80]) + + mb := new(MessageBuffer) + copy(mb.ne[:], in[:32]) + mb.ns = in[32:80] + mb.ciphertext = in[80:] + // copy(mb.ns,) + // copy(mb.ciphertext,) + + return mb, nil +} + func validatePublicKey(k []byte) bool { forbiddenCurveValues := [12][]byte{ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, @@ -327,7 +398,7 @@ func initializeResponder(prologue []byte, s keypair, rs [32]byte, psk [32]byte) return handshakestate{ss, s, e, rs, re, psk} } -func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, messagebuffer) { +func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { ne, ns, ciphertext := emptyKey, []byte{}, []byte{} hs.e = generateKeypair() ne = hs.e.public_key @@ -339,11 +410,11 @@ func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, message _, ns = encryptAndHash(&hs.ss, spk) mixKey(&hs.ss, dh(hs.s.private_key, hs.rs)) _, ciphertext = encryptAndHash(&hs.ss, payload) - messageBuffer := messagebuffer{ne, ns, ciphertext} + messageBuffer := MessageBuffer{ne, ns, ciphertext} return hs, messageBuffer } -func writeMessageB(hs *handshakestate, payload []byte) ([32]byte, messagebuffer, cipherstate, cipherstate) { +func writeMessageB(hs *handshakestate, payload []byte) ([32]byte, MessageBuffer, cipherstate, cipherstate) { ne, ns, ciphertext := emptyKey, []byte{}, []byte{} hs.e = generateKeypair() ne = hs.e.public_key @@ -352,12 +423,12 @@ func writeMessageB(hs *handshakestate, payload []byte) ([32]byte, messagebuffer, mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) _, ciphertext = encryptAndHash(&hs.ss, payload) - messageBuffer := messagebuffer{ne, ns, ciphertext} + messageBuffer := MessageBuffer{ne, ns, ciphertext} cs1, cs2 := split(&hs.ss) return hs.ss.h, messageBuffer, cs1, cs2 } -func readMessageA(hs *handshakestate, message *messagebuffer) (*handshakestate, []byte, bool) { +func readMessageA(hs *handshakestate, message *MessageBuffer) (*handshakestate, []byte, bool) { valid1 := true if validatePublicKey(message.ne[:]) { hs.re = message.ne @@ -374,7 +445,7 @@ func readMessageA(hs *handshakestate, message *messagebuffer) (*handshakestate, return hs, plaintext, (valid1 && valid2) } -func readMessageB(hs *handshakestate, message *messagebuffer) ([32]byte, []byte, bool, cipherstate, cipherstate) { +func readMessageB(hs *handshakestate, message *MessageBuffer) ([32]byte, []byte, bool, cipherstate, cipherstate) { valid1 := true if validatePublicKey(message.ne[:]) { hs.re = message.ne @@ -392,8 +463,8 @@ func readMessageB(hs *handshakestate, message *messagebuffer) ([32]byte, []byte, * PROCESSES * * ---------------------------------------------------------------- */ -func InitSession(initiator bool, prologue []byte, s keypair, rs [32]byte) noisesession { - var session noisesession +func InitSession(initiator bool, prologue []byte, s keypair, rs [32]byte) *NoiseSession { + var session NoiseSession psk := emptyKey if initiator { session.hs = initializeInitiator(prologue, s, rs, psk) @@ -402,11 +473,11 @@ func InitSession(initiator bool, prologue []byte, s keypair, rs [32]byte) noises } session.i = initiator session.mc = 0 - return session + return &session } -func SendMessage(session *noisesession, message []byte) (*noisesession, messagebuffer) { - var messageBuffer messagebuffer +func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageBuffer) { + var messageBuffer MessageBuffer if session.mc == 0 { _, messageBuffer = writeMessageA(&session.hs, message) } @@ -418,7 +489,7 @@ func SendMessage(session *noisesession, message []byte) (*noisesession, messageb return session, messageBuffer } -func RecvMessage(session *noisesession, message *messagebuffer) (*noisesession, []byte, bool) { +func RecvMessage(session *NoiseSession, message *MessageBuffer) (*NoiseSession, []byte, bool) { var plaintext []byte var valid bool if session.mc == 0 { From 4fc0d08b325ae4004b1a7708079967907932c698 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 08:23:30 -0400 Subject: [PATCH 1445/3965] add length rw to XX messages --- p2p/security/noise/ik/IK.noise.go | 105 ++++--- p2p/security/noise/ik/IK_test.go | 108 +++++++ p2p/security/noise/protocol.go | 436 +++++++++++++++++++++------ p2p/security/noise/transport.go | 24 +- p2p/security/noise/transport_test.go | 52 +++- 5 files changed, 588 insertions(+), 137 deletions(-) create mode 100644 p2p/security/noise/ik/IK_test.go diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index 5b7c48136a..c90ab1e767 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -32,7 +32,7 @@ import ( * TYPES * * ---------------------------------------------------------------- */ -type keypair struct { +type Keypair struct { public_key [32]byte private_key [32]byte } @@ -56,8 +56,8 @@ type symmetricstate struct { type handshakestate struct { ss symmetricstate - s keypair - e keypair + s Keypair + e Keypair rs [32]byte re [32]byte psk [32]byte @@ -72,6 +72,33 @@ type NoiseSession struct { i bool } +func NewKeypair(pub [32]byte, priv [32]byte) Keypair { + return Keypair{ + public_key: pub, + private_key: priv, + } +} + +func (kp Keypair) PubKey() [32]byte { + return kp.public_key +} + +func (kp Keypair) PrivKey() [32]byte { + return kp.private_key +} + +func (ns *NoiseSession) RemoteKey() [32]byte { + return ns.hs.rs +} + +func NewMessageBuffer(ne [32]byte, ns []byte, ciphertext []byte) MessageBuffer { + return MessageBuffer{ + ne: ne, + ns: ns, + ciphertext: ciphertext, + } +} + /* ---------------------------------------------------------------- * * CONSTANTS * * ---------------------------------------------------------------- */ @@ -93,7 +120,7 @@ var minNonce = uint32(0) * UTILITY FUNCTIONS * * ---------------------------------------------------------------- */ -func getPublicKey(kp *keypair) [32]byte { +func getPublicKey(kp *Keypair) [32]byte { return kp.public_key } @@ -118,24 +145,25 @@ func (mb *MessageBuffer) Encode0() []byte { enc := []byte{} enc = append(enc, mb.ne[:]...) + enc = append(enc, mb.ns...) enc = append(enc, mb.ciphertext...) return enc } // Encodes a MessageBuffer from stage 1 and 2 -func (mb *MessageBuffer) Encode1() []byte { - enc := []byte{} +// func (mb *MessageBuffer) Encode1() []byte { +// enc := []byte{} - enc = append(enc, mb.ne[:]...) - enc = append(enc, mb.ns...) - enc = append(enc, mb.ciphertext...) +// enc = append(enc, mb.ne[:]...) +// enc = append(enc, mb.ns...) +// enc = append(enc, mb.ciphertext...) - // log.Debug("XX_Encode1", "ne", mb.ne) - // log.Debug("XX_Encode1", "ns", mb.ns) +// // log.Debug("XX_Encode1", "ne", mb.ne) +// // log.Debug("XX_Encode1", "ns", mb.ns) - return enc -} +// return enc +// } // Decodes initial message (stage 0) into MessageBuffer func Decode0(in []byte) (*MessageBuffer, error) { @@ -146,30 +174,31 @@ func Decode0(in []byte) (*MessageBuffer, error) { //log.Debug("XX_Decode0", "in", in) mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) - mb.ciphertext = in[32:] + mb.ns = in[32:80] + mb.ciphertext = in[80:] //log.Debug("XX_Decode0", "mb", mb) return mb, nil } // Decodes messages at stage 1 or 2 into MessageBuffer -func Decode1(in []byte) (*MessageBuffer, error) { - if len(in) < 80 { - return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") - } +// func Decode1(in []byte) (*MessageBuffer, error) { +// if len(in) < 80 { +// return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") +// } - // log.Debug("XX_Decode1", "in", in) - // log.Debug("XX_Decode1", "ns", in[32:80]) +// // log.Debug("XX_Decode1", "in", in) +// // log.Debug("XX_Decode1", "ns", in[32:80]) - mb := new(MessageBuffer) - copy(mb.ne[:], in[:32]) - mb.ns = in[32:80] - mb.ciphertext = in[80:] - // copy(mb.ns,) - // copy(mb.ciphertext,) +// mb := new(MessageBuffer) +// copy(mb.ne[:], in[:32]) +// mb.ns = in[32:80] +// mb.ciphertext = in[80:] +// // copy(mb.ns,) +// // copy(mb.ciphertext,) - return mb, nil -} +// return mb, nil +// } func validatePublicKey(k []byte) bool { forbiddenCurveValues := [12][]byte{ @@ -209,15 +238,15 @@ func dh(private_key [32]byte, public_key [32]byte) [32]byte { return ss } -func generateKeypair() keypair { +func GenerateKeypair() Keypair { var public_key [32]byte var private_key [32]byte _, _ = rand.Read(private_key[:]) curve25519.ScalarBaseMult(&public_key, &private_key) if validatePublicKey(public_key[:]) { - return keypair{public_key, private_key} + return Keypair{public_key, private_key} } - return generateKeypair() + return GenerateKeypair() } func generatePublicKey(private_key [32]byte) [32]byte { @@ -376,9 +405,9 @@ func split(ss *symmetricstate) (cipherstate, cipherstate) { /* HandshakeState */ -func initializeInitiator(prologue []byte, s keypair, rs [32]byte, psk [32]byte) handshakestate { +func initializeInitiator(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) handshakestate { var ss symmetricstate - var e keypair + var e Keypair var re [32]byte name := []byte("Noise_IK_25519_ChaChaPoly_BLAKE2s") ss = initializeSymmetric(name) @@ -387,9 +416,9 @@ func initializeInitiator(prologue []byte, s keypair, rs [32]byte, psk [32]byte) return handshakestate{ss, s, e, rs, re, psk} } -func initializeResponder(prologue []byte, s keypair, rs [32]byte, psk [32]byte) handshakestate { +func initializeResponder(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) handshakestate { var ss symmetricstate - var e keypair + var e Keypair var re [32]byte name := []byte("Noise_IK_25519_ChaChaPoly_BLAKE2s") ss = initializeSymmetric(name) @@ -400,7 +429,7 @@ func initializeResponder(prologue []byte, s keypair, rs [32]byte, psk [32]byte) func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - hs.e = generateKeypair() + hs.e = GenerateKeypair() ne = hs.e.public_key mixHash(&hs.ss, ne[:]) /* No PSK, so skipping mixKey */ @@ -416,7 +445,7 @@ func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, Message func writeMessageB(hs *handshakestate, payload []byte) ([32]byte, MessageBuffer, cipherstate, cipherstate) { ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - hs.e = generateKeypair() + hs.e = GenerateKeypair() ne = hs.e.public_key mixHash(&hs.ss, ne[:]) /* No PSK, so skipping mixKey */ @@ -463,7 +492,7 @@ func readMessageB(hs *handshakestate, message *MessageBuffer) ([32]byte, []byte, * PROCESSES * * ---------------------------------------------------------------- */ -func InitSession(initiator bool, prologue []byte, s keypair, rs [32]byte) *NoiseSession { +func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *NoiseSession { var session NoiseSession psk := emptyKey if initiator { diff --git a/p2p/security/noise/ik/IK_test.go b/p2p/security/noise/ik/IK_test.go new file mode 100644 index 0000000000..cf6c0ab394 --- /dev/null +++ b/p2p/security/noise/ik/IK_test.go @@ -0,0 +1,108 @@ +package ik + +import ( + "crypto/rand" + pb "github.com/ChainSafe/go-libp2p-noise/pb" + proto "github.com/gogo/protobuf/proto" + "github.com/libp2p/go-libp2p-core/crypto" + "testing" +) + +func TestHandshake(t *testing.T) { + // generate local static noise key + kp_init := GenerateKeypair() + kp_resp := GenerateKeypair() + + payload_string := []byte("noise-libp2p-static-key:") + prologue := []byte("/noise/0.0.0") + + // initiator setup + init_pub := kp_init.PubKey() + libp2p_priv_init, libp2p_pub_init, err := crypto.GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + libp2p_pub_init_raw, err := libp2p_pub_init.Raw() + if err != nil { + t.Fatal(err) + } + libp2p_init_signed_payload, err := libp2p_priv_init.Sign(append(payload_string, init_pub[:]...)) + if err != nil { + t.Fatal(err) + } + + // respoonder setup + resp_pub := kp_resp.PubKey() + libp2p_priv_resp, libp2p_pub_resp, err := crypto.GenerateEd25519Key(rand.Reader) + if err != nil { + t.Fatal(err) + } + libp2p_pub_resp_raw, err := libp2p_pub_resp.Raw() + if err != nil { + t.Fatal(err) + } + libp2p_resp_signed_payload, err := libp2p_priv_resp.Sign(append(payload_string, resp_pub[:]...)) + if err != nil { + t.Fatal(err) + } + + // initiator: new IK noise session + ns_init := InitSession(true, prologue, kp_init, kp_resp.PubKey()) + + // responder: new IK noise session + ns_resp := InitSession(false, prologue, kp_resp, kp_init.PubKey()) + + // stage 0: initiator + // create payload + payload_init := new(pb.NoiseHandshakePayload) + payload_init.Libp2PKey = libp2p_pub_init_raw + payload_init.NoiseStaticKeySignature = libp2p_init_signed_payload + payload_init_enc, err := proto.Marshal(payload_init) + if err != nil { + t.Fatalf("proto marshal payload fail: %s", err) + } + + // send message + var msgbuf MessageBuffer + msg := []byte{} + msg = append(msg, payload_init_enc[:]...) + ns_init, msgbuf = SendMessage(ns_init, msg) + + t.Logf("stage 0 msgbuf: %v", msgbuf) + t.Logf("stage 0 msgbuf ne len: %d", len(msgbuf.NE())) + + // stage 0: responder + var plaintext []byte + var valid bool + ns_resp, plaintext, valid = RecvMessage(ns_resp, &msgbuf) + if !valid { + t.Fatalf("stage 0 receive not valid") + } + + t.Logf("stage 0 resp payload: %x", plaintext) + + // stage 1: responder + // create payload + payload_resp := new(pb.NoiseHandshakePayload) + payload_resp.Libp2PKey = libp2p_pub_resp_raw + payload_resp.NoiseStaticKeySignature = libp2p_resp_signed_payload + payload_resp_enc, err := proto.Marshal(payload_resp) + if err != nil { + t.Fatalf("proto marshal payload fail: %s", err) + } + msg = append(msg, payload_resp_enc[:]...) + ns_resp, msgbuf = SendMessage(ns_resp, msg) + + t.Logf("stage 1 msgbuf: %v", msgbuf) + t.Logf("stage 1 msgbuf ne len: %d", len(msgbuf.NE())) + t.Logf("stage 1 msgbuf ns len: %d", len(msgbuf.NS())) + + // stage 1: initiator + ns_init, plaintext, valid = RecvMessage(ns_init, &msgbuf) + if !valid { + t.Fatalf("stage 1 receive not valid") + } + + t.Logf("stage 1 resp payload: %x", plaintext) + +} diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 76038f65ba..d626b55ace 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -2,6 +2,7 @@ package noise import ( "context" + "encoding/binary" "fmt" //"io" "net" @@ -11,8 +12,9 @@ import ( proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" - "github.com/libp2p/go-libp2p-core/sec" + //"github.com/libp2p/go-libp2p-core/sec" + ik "github.com/ChainSafe/go-libp2p-noise/ik" pb "github.com/ChainSafe/go-libp2p-noise/pb" xx "github.com/ChainSafe/go-libp2p-noise/xx" ) @@ -32,23 +34,34 @@ type secureSession struct { local peerInfo remote peerInfo - ns *xx.NoiseSession + xx_ns *xx.NoiseSession + ik_ns *ik.NoiseSession + + noisePipesSupport bool + noiseStaticKeyCache map[peer.ID]([32]byte) } -// TODO: after reading payloads in initial message, fill in the peerInfo type peerInfo struct { - noiseKey crypto.PubKey // static noise key + noiseKey [32]byte // static noise key libp2pKey crypto.PubKey } -func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, insecure net.Conn, remote peer.ID, initiator bool) (sec.SecureConn, error) { +func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, + insecure net.Conn, remote peer.ID, noiseStaticKeyCache map[peer.ID]([32]byte), + noisePipesSupport bool, initiator bool) (*secureSession, error) { + + if noiseStaticKeyCache == nil { + noiseStaticKeyCache = make(map[peer.ID]([32]byte)) + } s := &secureSession{ - insecure: insecure, - initiator: initiator, - prologue: []byte(ID), - localKey: privKey, - localPeer: local, - remotePeer: remote, + insecure: insecure, + initiator: initiator, + prologue: []byte(ID), + localKey: privKey, + localPeer: local, + remotePeer: remote, + noisePipesSupport: noisePipesSupport, + noiseStaticKeyCache: noiseStaticKeyCache, } err := s.runHandshake(ctx) @@ -56,12 +69,16 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey return s, err } -func (s *secureSession) sendHandshakeMessage(payload []byte, initial_stage bool) error { - log.Debug("send handshake message", "initiator", s.initiator, "payload", payload, "payload len", len(payload), "initial_stage", initial_stage) +func (s *secureSession) NoiseStaticKeyCache() map[peer.ID]([32]byte) { + return s.noiseStaticKeyCache +} + +func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { + log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload), "initial_stage", initial_stage) // create send message w payload var msgbuf xx.MessageBuffer - s.ns, msgbuf = xx.SendMessage(s.ns, payload) + s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, payload) var encMsgBuf []byte if initial_stage { encMsgBuf = msgbuf.Encode0() @@ -69,23 +86,36 @@ func (s *secureSession) sendHandshakeMessage(payload []byte, initial_stage bool) encMsgBuf = msgbuf.Encode1() } - log.Debug("send handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - log.Debug("send handshake message", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) + log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) + log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) + + err := s.WriteLength(len(encMsgBuf)) + if err != nil { + log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) + return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) + } // send message - _, err := s.insecure.Write(encMsgBuf) + _, err = s.insecure.Write(encMsgBuf) if err != nil { - log.Debug("send handshake message", "initiator", s.initiator, "error", err) - return fmt.Errorf("write to conn fail: %s", err) + log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) + return fmt.Errorf("xx_sendHandshakeMessage write to conn fail: %s", err) } return nil } -func (s *secureSession) recvHandshakeMessage(buf []byte, initial_stage bool) (plaintext []byte, valid bool, err error) { +func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { + l, err := s.ReadLength() + if err != nil { + return nil, nil, false, fmt.Errorf("read length fail: %s", err) + } + + buf = make([]byte, l) + _, err = s.insecure.Read(buf) if err != nil { - return nil, false, fmt.Errorf("read from conn fail: %s", err) + return buf, nil, false, fmt.Errorf("read from conn fail: %s", err) } var msgbuf *xx.MessageBuffer @@ -95,16 +125,73 @@ func (s *secureSession) recvHandshakeMessage(buf []byte, initial_stage bool) (pl msgbuf, err = xx.Decode1(buf) } - log.Debug("recv handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) + log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) + + if err != nil { + log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) + return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) + } + + s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) + if !valid { + log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") + return buf, nil, false, fmt.Errorf("validation fail") + } + + log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + + return buf, plaintext, valid, nil +} + +func (s *secureSession) ik_sendHandshakeMessage(payload []byte) error { + log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload)) + + // create send message w payload + var msgbuf ik.MessageBuffer + s.ik_ns, msgbuf = ik.SendMessage(s.ik_ns, payload) + var encMsgBuf []byte + //if initial_stage { + encMsgBuf = msgbuf.Encode0() + // } else { + // encMsgBuf = msgbuf.Encode1() + // } + + log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) + log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) + + // send message + _, err := s.insecure.Write(encMsgBuf) + if err != nil { + log.Error("ik_sendHandshakeMessage", "initiator", s.initiator, "error", err) + return fmt.Errorf("write to conn fail: %s", err) + } + + return nil +} + +func (s *secureSession) ik_recvHandshakeMessage(buf []byte) (plaintext []byte, valid bool, err error) { + _, err = s.insecure.Read(buf) + if err != nil { + return nil, false, fmt.Errorf("read from conn fail: %s", err) + } + + var msgbuf *ik.MessageBuffer + //if initial_stage { + msgbuf, err = ik.Decode0(buf) + // } else { + // msgbuf, err = ik.Decode1(buf) + // } + + log.Debug("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) if err != nil { - log.Debug("recv handshake message decode", "initiator", s.initiator, "error", err) + log.Error("ik_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) return nil, false, fmt.Errorf("decode msg fail: %s", err) } - s.ns, plaintext, valid = xx.RecvMessage(s.ns, msgbuf) + s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) if !valid { - log.Debug("recv handshake message xx", "initiator", s.initiator, "error", "validation fail") + log.Error("ik_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") return nil, false, fmt.Errorf("validation fail") } @@ -137,66 +224,38 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe return nil } -func (s *secureSession) runHandshake(ctx context.Context) error { - - // TODO: check if static key for peer exists - // if so, do XX; otherwise do IK - - // ******************************************** // - // ************** PHASE 1: TRY XX ************* // - // ******************************************** // - - // generate local static noise key - kp := xx.GenerateKeypair() - - log.Debug("xx handshake", "pubkey", kp.PubKey()) - - // setup libp2p keys - localKeyRaw, err := s.LocalPublicKey().Bytes() - if err != nil { - return fmt.Errorf("err getting raw pubkey: %s", err) - } - - log.Debug("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) - - // sign noise data for payload - noise_pub := kp.PubKey() - signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) - if err != nil { - return fmt.Errorf("err signing payload: %s", err) - } - - // create payload - payload := new(pb.NoiseHandshakePayload) - payload.Libp2PKey = localKeyRaw - payload.NoiseStaticKeySignature = signedPayload - payloadEnc, err := proto.Marshal(payload) - if err != nil { - return fmt.Errorf("proto marshal payload fail: %s", err) - } +func (s *secureSession) ReadLength() (int, error) { + buf := make([]byte, 2) + _, err := s.insecure.Read(buf) + return int(binary.BigEndian.Uint16(buf)), err +} - // new XX noise session - s.ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) +func (s *secureSession) WriteLength(length int) error { + buf := make([]byte, 2) + binary.BigEndian.PutUint16(buf, uint16(length)) + _, err := s.insecure.Write(buf) + return err +} +func (s *secureSession) runHandshake_xx(ctx context.Context, payloadEnc []byte) (handshakeData []byte, err error) { if s.initiator { // stage 0 // - err = s.sendHandshakeMessage(payloadEnc, true) + err = s.xx_sendHandshakeMessage(payloadEnc, true) if err != nil { - return fmt.Errorf("stage 0 intiator fail: %s", err) + return nil, fmt.Errorf("stage 0 initiator fail: %s", err) } // stage 1 // // read reply - buf := make([]byte, 96+len(payloadEnc)) - plaintext, valid, err := s.recvHandshakeMessage(buf, false) + buf, plaintext, valid, err := s.xx_recvHandshakeMessage(false) if err != nil { - return fmt.Errorf("intiator stage 1 fail: %s", err) + return buf, fmt.Errorf("initiator stage 1 fail: %s", err) } if !valid { - return fmt.Errorf("stage 1 initiator validation fail") + return buf, fmt.Errorf("stage 1 initiator validation fail") } log.Debug("stage 1 initiator", "payload", plaintext) @@ -205,14 +264,14 @@ func (s *secureSession) runHandshake(ctx context.Context) error { nhp := new(pb.NoiseHandshakePayload) err = proto.Unmarshal(plaintext, nhp) if err != nil { - return fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") + return buf, fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { log.Error("stage 1 initiator set remote peer info", "err", err) - return fmt.Errorf("stage 1 initiator read remote libp2p key fail") + return buf, fmt.Errorf("stage 1 initiator read remote libp2p key fail") } // assert that remote peer ID matches libp2p public key @@ -224,17 +283,19 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ns.RemoteKey()) + err = s.verifyPayload(nhp, s.xx_ns.RemoteKey()) if err != nil { log.Error("stage 1 initiator verify payload", "err", err) } + s.noiseStaticKeyCache[s.remotePeer] = s.xx_ns.RemoteKey() + log.Debug("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) + // stage 2 // - //err = s.sendHandshakeMessage(append(plaintext, payloadEnc...), false) - err = s.sendHandshakeMessage(nil, false) + err = s.xx_sendHandshakeMessage(s.local.noiseKey[:], false) if err != nil { - return fmt.Errorf("stage 2 intiator fail: %s", err) + return buf, fmt.Errorf("stage 2 intiator fail: %s", err) } } else { @@ -242,10 +303,143 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // stage 0 // // read message - buf := make([]byte, 32+len(payloadEnc)) - plaintext, valid, err := s.recvHandshakeMessage(buf, true) + buf, plaintext, valid, err := s.xx_recvHandshakeMessage(true) + if err != nil { + return buf, fmt.Errorf("stage 0 responder fail: %s", err) + } + + if !valid { + return buf, fmt.Errorf("stage 0 responder validation fail") + } + + log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) + + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + if err != nil { + log.Error("stage 0 responder set remote peer info", "err", err) + return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") + } + + // assert that remote peer ID matches libp2p key + err = s.setRemotePeerID(s.RemotePublicKey()) + if err != nil { + log.Error("stage 0 responder set remote peer id", "err", err) + } + + // stage 1 // + + err = s.xx_sendHandshakeMessage(payloadEnc, false) + if err != nil { + return buf, fmt.Errorf("stage 1 responder fail: %s", err) + } + + // stage 2 // + + // read message + buf, plaintext, valid, err = s.xx_recvHandshakeMessage(false) + if err != nil { + return buf, fmt.Errorf("stage 2 responder fail: %s", err) + } + + if !valid { + return buf, fmt.Errorf("stage 2 responder validation fail") + } + + log.Debug("stage 2 responder", "plaintext", plaintext, "remote key", s.xx_ns.RemoteKey()) + + copy(s.remote.noiseKey[:], plaintext) + + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.remote.noiseKey) + if err != nil { + log.Error("stage 2 responder verify payload", "err", err) + return buf, fmt.Errorf("stage 2 responder fail: %s", err) + } + + s.noiseStaticKeyCache[s.remotePeer] = s.remote.noiseKey + log.Debug("stage 2 responder", "remote key", s.remote.noiseKey) + } + + return nil, nil +} + +func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byte) error { + // generate local static noise key + kp := ik.GenerateKeypair() + + log.Debug("ik handshake", "pubkey", kp.PubKey()) + + // setup libp2p keys + localKeyRaw, err := s.LocalPublicKey().Bytes() + if err != nil { + return fmt.Errorf("err getting raw pubkey: %s", err) + } + + log.Debug("ik handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) + + // sign noise data for payload + noise_pub := kp.PubKey() + signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) + if err != nil { + return fmt.Errorf("err signing payload: %s", err) + } + + // create payload + payload := new(pb.NoiseHandshakePayload) + payload.Libp2PKey = localKeyRaw + payload.NoiseStaticKeySignature = signedPayload + payloadEnc, err := proto.Marshal(payload) + if err != nil { + return fmt.Errorf("proto marshal payload fail: %s", err) + } + + // new XX noise session + s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, [32]byte{}) + + if s.initiator { + // stage 0 // + err := s.ik_sendHandshakeMessage(payloadEnc) if err != nil { - return fmt.Errorf("stage 0 responder fail: %s", err) + log.Error("stage 0 initiator verify payload", "err", err) + return fmt.Errorf("stage 0 initiator fail: %s", err) + } + + // stage 1 // + + } else { + + // stage 0 // + + var plaintext []byte + var valid bool + if handshakeData != nil { + var msgbuf *ik.MessageBuffer + msgbuf, err = ik.Decode0(handshakeData) + + log.Debug("ik_recvHandshakeMessage", "responder", s.initiator, "msgbuf", msgbuf, "buf len", len(handshakeData)) + + if err != nil { + return fmt.Errorf("stage 0 responder fail: %s", err) + } + + s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) + log.Debug("recv handshake message", "responder", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + } else { + // read message + buf := make([]byte, 32+len(payloadEnc)) + plaintext, valid, err = s.ik_recvHandshakeMessage(buf) + if err != nil { + return fmt.Errorf("stage 0 responder fail: %s", err) + } + } if !valid { @@ -275,33 +469,87 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ns.RemoteKey()) + err = s.verifyPayload(nhp, s.ik_ns.RemoteKey()) if err != nil { - log.Error("stage 1 initiator verify payload", "err", err) + log.Error("stage 1 responder verify payload", "err", err) } - // stage 1 // + } - err = s.sendHandshakeMessage(payloadEnc, false) + return nil +} + +func (s *secureSession) runHandshake(ctx context.Context) error { + + // TODO: check if static key for peer exists + // if not, do XX; otherwise do IK + + log.Debug("runHandshake", "cache", s.noiseStaticKeyCache) + + // try + if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} && s.noisePipesSupport { + //if s.noisePipesSupport { + log.Debug("runHandshake_ik") + // ******************************************** // + // known static key for peer, try IK // + // ******************************************** // + + err := s.runHandshake_ik(ctx, nil) if err != nil { - return fmt.Errorf("stage 1 responder fail: %s", err) + log.Error("runHandshake_ik", "err", err) + + // PIPE TO XX } - // stage 2 // + } else { + // ******************************************** // + // unknown static key for peer, try XX // + // ******************************************** // - // read message - buf = make([]byte, 96) - plaintext, valid, err = s.recvHandshakeMessage(buf, false) + // generate local static noise key + kp := xx.GenerateKeypair() + + log.Debug("xx handshake", "pubkey", kp.PubKey()) + + // setup libp2p keys + localKeyRaw, err := s.LocalPublicKey().Bytes() if err != nil { - return fmt.Errorf("stage 2 responder fail: %s", err) + return fmt.Errorf("err getting raw pubkey: %s", err) } - if !valid { - return fmt.Errorf("stage 2 responder validation fail") + log.Debug("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) + + // sign noise data for payload + noise_pub := kp.PubKey() + signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) + if err != nil { + return fmt.Errorf("err signing payload: %s", err) + } + + s.local.noiseKey = noise_pub + + // create payload + payload := new(pb.NoiseHandshakePayload) + payload.Libp2PKey = localKeyRaw + payload.NoiseStaticKeySignature = signedPayload + payloadEnc, err := proto.Marshal(payload) + if err != nil { + return fmt.Errorf("proto marshal payload fail: %s", err) } - // TODO: check payload - log.Debug("stage 2 responder", "payload", plaintext) + // new XX noise session + s.xx_ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) + + handshakeData, err := s.runHandshake_xx(ctx, payloadEnc) + if err != nil { + log.Error("runHandshake_xx", "err", err) + log.Debug("try runHandshake_ik...") + err := s.runHandshake_ik(ctx, handshakeData) + if err != nil { + return fmt.Errorf("runHandshake_ik err %s", err) + } + //return fmt.Errorf("runHandshake_xx err %s", err) + } } return nil @@ -324,6 +572,7 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { } func (s *secureSession) Read(in []byte) (int, error) { + // TODO: use noise symmetric keys return s.insecure.Read(in) } @@ -352,6 +601,7 @@ func (s *secureSession) SetWriteDeadline(t time.Time) error { } func (s *secureSession) Write(in []byte) (int, error) { + // TODO: use noise symmetric keys return s.insecure.Write(in) } diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index b2b3e1e9b6..23d3c2064d 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -9,21 +9,35 @@ import ( "github.com/libp2p/go-libp2p-core/sec" ) -const ID = "/noise/0.0.0" +const ID = "/noise/0.0.1" var _ sec.SecureTransport = &Transport{} type Transport struct { - LocalID peer.ID - PrivateKey crypto.PrivKey + LocalID peer.ID + PrivateKey crypto.PrivKey + NoisePipesSupport bool + NoiseStaticKeyCache map[peer.ID]([32]byte) } // SecureInbound runs noise handshake as the responder func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { - return newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, "", false) + s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, "", t.NoiseStaticKeyCache, t.NoisePipesSupport, false) + if err != nil { + return s, err + } + + t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() + return s, nil } // SecureOutbound runs noise handshake as the initiator func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { - return newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, p, true) + s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, p, t.NoiseStaticKeyCache, t.NoisePipesSupport, true) + if err != nil { + return s, err + } + + t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() + return s, nil } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 7ff71c4726..c21f7f24d1 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -27,6 +27,22 @@ func newTestTransport(t *testing.T, typ, bits int) *Transport { } } +func newTestTransportPipes(t *testing.T, typ, bits int) *Transport { + priv, pub, err := crypto.GenerateKeyPair(typ, bits) + if err != nil { + t.Fatal(err) + } + id, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Fatal(err) + } + return &Transport{ + LocalID: id, + PrivateKey: priv, + NoisePipesSupport: true, + } +} + // Create a new pair of connected TCP sockets. func newConnPair(t *testing.T) (net.Conn, net.Conn) { lstnr, err := net.Listen("tcp", "localhost:0") @@ -157,7 +173,8 @@ func TestRW(t *testing.T) { } } -func TestHandshake(t *testing.T) { +// Tests XX handshake +func TestHandshakeXX(t *testing.T) { initTransport := newTestTransport(t, crypto.Ed25519, 2048) respTransport := newTestTransport(t, crypto.Ed25519, 2048) @@ -181,3 +198,36 @@ func TestHandshake(t *testing.T) { t.Errorf("Message mismatch. %v != %v", before, after) } } + +// Test noise pipes +func TestHandshakeIK(t *testing.T) { + initTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) + respTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + initConn.Close() + respConn.Close() + + initConn, respConn = connect(t, initTransport, respTransport) + t.Log(initTransport) + t.Log(respTransport) + defer initConn.Close() + defer respConn.Close() + + before := []byte("hello world") + _, err := initConn.Write(before) + if err != nil { + t.Fatal(err) + } + + after := make([]byte, len(before)) + _, err = io.ReadFull(respConn, after) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(before, after) { + t.Errorf("Message mismatch. %v != %v", before, after) + } + +} From 736a9fa03d0805ff1952a792c9cf915ccac592dc Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 08:55:04 -0400 Subject: [PATCH 1446/3965] separate handshakes into files --- p2p/security/noise/ik/IK_test.go | 2 +- p2p/security/noise/ik_handshake.go | 189 +++++++++++++++ p2p/security/noise/protocol.go | 373 +---------------------------- p2p/security/noise/xx_handshake.go | 216 +++++++++++++++++ 4 files changed, 409 insertions(+), 371 deletions(-) create mode 100644 p2p/security/noise/ik_handshake.go create mode 100644 p2p/security/noise/xx_handshake.go diff --git a/p2p/security/noise/ik/IK_test.go b/p2p/security/noise/ik/IK_test.go index cf6c0ab394..13190a61e7 100644 --- a/p2p/security/noise/ik/IK_test.go +++ b/p2p/security/noise/ik/IK_test.go @@ -50,7 +50,7 @@ func TestHandshake(t *testing.T) { ns_init := InitSession(true, prologue, kp_init, kp_resp.PubKey()) // responder: new IK noise session - ns_resp := InitSession(false, prologue, kp_resp, kp_init.PubKey()) + ns_resp := InitSession(false, prologue, kp_resp, [32]byte{}) // stage 0: initiator // create payload diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go new file mode 100644 index 0000000000..970acc4eff --- /dev/null +++ b/p2p/security/noise/ik_handshake.go @@ -0,0 +1,189 @@ +package noise + +import ( + "context" + "fmt" + log "github.com/ChainSafe/log15" + proto "github.com/gogo/protobuf/proto" + //"github.com/libp2p/go-libp2p-core/peer" + + ik "github.com/ChainSafe/go-libp2p-noise/ik" + pb "github.com/ChainSafe/go-libp2p-noise/pb" +) + +func (s *secureSession) ik_sendHandshakeMessage(payload []byte) error { + log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload)) + + // create send message w payload + var msgbuf ik.MessageBuffer + s.ik_ns, msgbuf = ik.SendMessage(s.ik_ns, payload) + var encMsgBuf []byte + //if initial_stage { + encMsgBuf = msgbuf.Encode0() + // } else { + // encMsgBuf = msgbuf.Encode1() + // } + + log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) + log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) + + err := s.WriteLength(len(encMsgBuf)) + if err != nil { + log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) + return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) + } + + // send message + _, err = s.insecure.Write(encMsgBuf) + if err != nil { + log.Error("ik_sendHandshakeMessage", "initiator", s.initiator, "error", err) + return fmt.Errorf("write to conn fail: %s", err) + } + + return nil +} + +func (s *secureSession) ik_recvHandshakeMessage() (buf []byte, plaintext []byte, valid bool, err error) { + l, err := s.ReadLength() + if err != nil { + return nil, nil, false, fmt.Errorf("read length fail: %s", err) + } + + buf = make([]byte, l) + + _, err = s.insecure.Read(buf) + if err != nil { + return buf, nil, false, fmt.Errorf("read from conn fail: %s", err) + } + + var msgbuf *ik.MessageBuffer + //if initial_stage { + msgbuf, err = ik.Decode0(buf) + // } else { + // msgbuf, err = ik.Decode1(buf) + // } + + log.Debug("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) + + if err != nil { + log.Error("ik_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) + return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) + } + + s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) + if !valid { + log.Error("ik_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") + return buf, nil, false, fmt.Errorf("validation fail") + } + + log.Debug("recv handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + + return buf, plaintext, valid, nil +} + +func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byte) error { + // generate local static noise key + kp := ik.GenerateKeypair() + + log.Debug("ik handshake", "pubkey", kp.PubKey()) + + // setup libp2p keys + localKeyRaw, err := s.LocalPublicKey().Bytes() + if err != nil { + return fmt.Errorf("err getting raw pubkey: %s", err) + } + + log.Debug("ik handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) + + // sign noise data for payload + noise_pub := kp.PubKey() + signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) + if err != nil { + return fmt.Errorf("err signing payload: %s", err) + } + + // create payload + payload := new(pb.NoiseHandshakePayload) + payload.Libp2PKey = localKeyRaw + payload.NoiseStaticKeySignature = signedPayload + payloadEnc, err := proto.Marshal(payload) + if err != nil { + return fmt.Errorf("proto marshal payload fail: %s", err) + } + + // new XX noise session + s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, s.noiseStaticKeyCache[s.remotePeer]) + log.Debug("ik initiator init session", "remotePeer", s.noiseStaticKeyCache[s.remotePeer]) + + if s.initiator { + // stage 0 // + err := s.ik_sendHandshakeMessage(payloadEnc) + if err != nil { + log.Error("stage 0 initiator send", "err", err) + return fmt.Errorf("stage 0 initiator fail: %s", err) + } + + // stage 1 // + + } else { + // stage 0 // + + var plaintext []byte + var valid bool + + if handshakeData != nil { + var msgbuf *ik.MessageBuffer + msgbuf, err = ik.Decode0(handshakeData) + + log.Debug("stage 0 ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(handshakeData)) + + if err != nil { + return fmt.Errorf("stage 0 responder fail: %s", err) + } + + s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) + } else { + // read message + _, plaintext, valid, err = s.ik_recvHandshakeMessage() + if err != nil { + return fmt.Errorf("stage 0 responder fail: %s", err) + } + + } + + if !valid { + return fmt.Errorf("stage 0 responder validation fail") + } + + log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) + + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + if err != nil { + log.Error("stage 0 responder set remote peer info", "err", err) + return fmt.Errorf("stage 0 responder read remote libp2p key fail") + } + + // assert that remote peer ID matches libp2p key + err = s.setRemotePeerID(s.RemotePublicKey()) + if err != nil { + log.Error("stage 0 responder set remote peer id", "err", err) + } + + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.ik_ns.RemoteKey()) + if err != nil { + log.Error("stage 1 responder verify payload", "err", err) + } + + } + + return nil +} \ No newline at end of file diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index d626b55ace..04578262c6 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -53,6 +53,7 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey if noiseStaticKeyCache == nil { noiseStaticKeyCache = make(map[peer.ID]([32]byte)) } + s := &secureSession{ insecure: insecure, initiator: initiator, @@ -73,133 +74,6 @@ func (s *secureSession) NoiseStaticKeyCache() map[peer.ID]([32]byte) { return s.noiseStaticKeyCache } -func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { - log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload), "initial_stage", initial_stage) - - // create send message w payload - var msgbuf xx.MessageBuffer - s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, payload) - var encMsgBuf []byte - if initial_stage { - encMsgBuf = msgbuf.Encode0() - } else { - encMsgBuf = msgbuf.Encode1() - } - - log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) - - err := s.WriteLength(len(encMsgBuf)) - if err != nil { - log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) - return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) - } - - // send message - _, err = s.insecure.Write(encMsgBuf) - if err != nil { - log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) - return fmt.Errorf("xx_sendHandshakeMessage write to conn fail: %s", err) - } - - return nil -} - -func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { - l, err := s.ReadLength() - if err != nil { - return nil, nil, false, fmt.Errorf("read length fail: %s", err) - } - - buf = make([]byte, l) - - _, err = s.insecure.Read(buf) - if err != nil { - return buf, nil, false, fmt.Errorf("read from conn fail: %s", err) - } - - var msgbuf *xx.MessageBuffer - if initial_stage { - msgbuf, err = xx.Decode0(buf) - } else { - msgbuf, err = xx.Decode1(buf) - } - - log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) - - if err != nil { - log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) - return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) - } - - s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) - if !valid { - log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") - return buf, nil, false, fmt.Errorf("validation fail") - } - - log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) - - return buf, plaintext, valid, nil -} - -func (s *secureSession) ik_sendHandshakeMessage(payload []byte) error { - log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload)) - - // create send message w payload - var msgbuf ik.MessageBuffer - s.ik_ns, msgbuf = ik.SendMessage(s.ik_ns, payload) - var encMsgBuf []byte - //if initial_stage { - encMsgBuf = msgbuf.Encode0() - // } else { - // encMsgBuf = msgbuf.Encode1() - // } - - log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) - log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) - - // send message - _, err := s.insecure.Write(encMsgBuf) - if err != nil { - log.Error("ik_sendHandshakeMessage", "initiator", s.initiator, "error", err) - return fmt.Errorf("write to conn fail: %s", err) - } - - return nil -} - -func (s *secureSession) ik_recvHandshakeMessage(buf []byte) (plaintext []byte, valid bool, err error) { - _, err = s.insecure.Read(buf) - if err != nil { - return nil, false, fmt.Errorf("read from conn fail: %s", err) - } - - var msgbuf *ik.MessageBuffer - //if initial_stage { - msgbuf, err = ik.Decode0(buf) - // } else { - // msgbuf, err = ik.Decode1(buf) - // } - - log.Debug("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) - - if err != nil { - log.Error("ik_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) - return nil, false, fmt.Errorf("decode msg fail: %s", err) - } - - s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) - if !valid { - log.Error("ik_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") - return nil, false, fmt.Errorf("validation fail") - } - - log.Debug("recv handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) - - return plaintext, valid, nil -} - func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { s.remote.libp2pKey, err = crypto.UnmarshalPublicKey(key) return err @@ -237,248 +111,6 @@ func (s *secureSession) WriteLength(length int) error { return err } -func (s *secureSession) runHandshake_xx(ctx context.Context, payloadEnc []byte) (handshakeData []byte, err error) { - if s.initiator { - // stage 0 // - - err = s.xx_sendHandshakeMessage(payloadEnc, true) - if err != nil { - return nil, fmt.Errorf("stage 0 initiator fail: %s", err) - } - - // stage 1 // - - // read reply - buf, plaintext, valid, err := s.xx_recvHandshakeMessage(false) - if err != nil { - return buf, fmt.Errorf("initiator stage 1 fail: %s", err) - } - - if !valid { - return buf, fmt.Errorf("stage 1 initiator validation fail") - } - - log.Debug("stage 1 initiator", "payload", plaintext) - - // unmarshal payload - nhp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return buf, fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) - if err != nil { - log.Error("stage 1 initiator set remote peer info", "err", err) - return buf, fmt.Errorf("stage 1 initiator read remote libp2p key fail") - } - - // assert that remote peer ID matches libp2p public key - pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) - if pid != s.remotePeer { - log.Error("stage 1 initiator check remote peer id err", "expected", s.remotePeer, "got", pid) - } else if err != nil { - log.Error("stage 1 initiator check remote peer id", "err", err) - } - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.xx_ns.RemoteKey()) - if err != nil { - log.Error("stage 1 initiator verify payload", "err", err) - } - - s.noiseStaticKeyCache[s.remotePeer] = s.xx_ns.RemoteKey() - log.Debug("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) - - // stage 2 // - - err = s.xx_sendHandshakeMessage(s.local.noiseKey[:], false) - if err != nil { - return buf, fmt.Errorf("stage 2 intiator fail: %s", err) - } - - } else { - - // stage 0 // - - // read message - buf, plaintext, valid, err := s.xx_recvHandshakeMessage(true) - if err != nil { - return buf, fmt.Errorf("stage 0 responder fail: %s", err) - } - - if !valid { - return buf, fmt.Errorf("stage 0 responder validation fail") - } - - log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) - - // unmarshal payload - nhp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) - if err != nil { - log.Error("stage 0 responder set remote peer info", "err", err) - return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") - } - - // assert that remote peer ID matches libp2p key - err = s.setRemotePeerID(s.RemotePublicKey()) - if err != nil { - log.Error("stage 0 responder set remote peer id", "err", err) - } - - // stage 1 // - - err = s.xx_sendHandshakeMessage(payloadEnc, false) - if err != nil { - return buf, fmt.Errorf("stage 1 responder fail: %s", err) - } - - // stage 2 // - - // read message - buf, plaintext, valid, err = s.xx_recvHandshakeMessage(false) - if err != nil { - return buf, fmt.Errorf("stage 2 responder fail: %s", err) - } - - if !valid { - return buf, fmt.Errorf("stage 2 responder validation fail") - } - - log.Debug("stage 2 responder", "plaintext", plaintext, "remote key", s.xx_ns.RemoteKey()) - - copy(s.remote.noiseKey[:], plaintext) - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.remote.noiseKey) - if err != nil { - log.Error("stage 2 responder verify payload", "err", err) - return buf, fmt.Errorf("stage 2 responder fail: %s", err) - } - - s.noiseStaticKeyCache[s.remotePeer] = s.remote.noiseKey - log.Debug("stage 2 responder", "remote key", s.remote.noiseKey) - } - - return nil, nil -} - -func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byte) error { - // generate local static noise key - kp := ik.GenerateKeypair() - - log.Debug("ik handshake", "pubkey", kp.PubKey()) - - // setup libp2p keys - localKeyRaw, err := s.LocalPublicKey().Bytes() - if err != nil { - return fmt.Errorf("err getting raw pubkey: %s", err) - } - - log.Debug("ik handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) - - // sign noise data for payload - noise_pub := kp.PubKey() - signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) - if err != nil { - return fmt.Errorf("err signing payload: %s", err) - } - - // create payload - payload := new(pb.NoiseHandshakePayload) - payload.Libp2PKey = localKeyRaw - payload.NoiseStaticKeySignature = signedPayload - payloadEnc, err := proto.Marshal(payload) - if err != nil { - return fmt.Errorf("proto marshal payload fail: %s", err) - } - - // new XX noise session - s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, [32]byte{}) - - if s.initiator { - // stage 0 // - err := s.ik_sendHandshakeMessage(payloadEnc) - if err != nil { - log.Error("stage 0 initiator verify payload", "err", err) - return fmt.Errorf("stage 0 initiator fail: %s", err) - } - - // stage 1 // - - } else { - - // stage 0 // - - var plaintext []byte - var valid bool - if handshakeData != nil { - var msgbuf *ik.MessageBuffer - msgbuf, err = ik.Decode0(handshakeData) - - log.Debug("ik_recvHandshakeMessage", "responder", s.initiator, "msgbuf", msgbuf, "buf len", len(handshakeData)) - - if err != nil { - return fmt.Errorf("stage 0 responder fail: %s", err) - } - - s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) - log.Debug("recv handshake message", "responder", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) - } else { - // read message - buf := make([]byte, 32+len(payloadEnc)) - plaintext, valid, err = s.ik_recvHandshakeMessage(buf) - if err != nil { - return fmt.Errorf("stage 0 responder fail: %s", err) - } - - } - - if !valid { - return fmt.Errorf("stage 0 responder validation fail") - } - - log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) - - // unmarshal payload - nhp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) - if err != nil { - log.Error("stage 0 responder set remote peer info", "err", err) - return fmt.Errorf("stage 0 responder read remote libp2p key fail") - } - - // assert that remote peer ID matches libp2p key - err = s.setRemotePeerID(s.RemotePublicKey()) - if err != nil { - log.Error("stage 0 responder set remote peer id", "err", err) - } - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ik_ns.RemoteKey()) - if err != nil { - log.Error("stage 1 responder verify payload", "err", err) - } - - } - - return nil -} - func (s *secureSession) runHandshake(ctx context.Context) error { // TODO: check if static key for peer exists @@ -498,7 +130,8 @@ func (s *secureSession) runHandshake(ctx context.Context) error { if err != nil { log.Error("runHandshake_ik", "err", err) - // PIPE TO XX + // TODO: PIPE TO XX + } } else { diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go new file mode 100644 index 0000000000..43fdfa52ff --- /dev/null +++ b/p2p/security/noise/xx_handshake.go @@ -0,0 +1,216 @@ +package noise + +import ( + "context" + "fmt" + log "github.com/ChainSafe/log15" + proto "github.com/gogo/protobuf/proto" + "github.com/libp2p/go-libp2p-core/peer" + + xx "github.com/ChainSafe/go-libp2p-noise/xx" + pb "github.com/ChainSafe/go-libp2p-noise/pb" +) + +func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { + log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload), "initial_stage", initial_stage) + + // create send message w payload + var msgbuf xx.MessageBuffer + s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, payload) + var encMsgBuf []byte + if initial_stage { + encMsgBuf = msgbuf.Encode0() + } else { + encMsgBuf = msgbuf.Encode1() + } + + log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) + log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) + + err := s.WriteLength(len(encMsgBuf)) + if err != nil { + log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) + return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) + } + + // send message + _, err = s.insecure.Write(encMsgBuf) + if err != nil { + log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) + return fmt.Errorf("xx_sendHandshakeMessage write to conn fail: %s", err) + } + + return nil +} + +func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { + l, err := s.ReadLength() + if err != nil { + return nil, nil, false, fmt.Errorf("read length fail: %s", err) + } + + buf = make([]byte, l) + + _, err = s.insecure.Read(buf) + if err != nil { + return buf, nil, false, fmt.Errorf("read from conn fail: %s", err) + } + + var msgbuf *xx.MessageBuffer + if initial_stage { + msgbuf, err = xx.Decode0(buf) + } else { + msgbuf, err = xx.Decode1(buf) + } + + log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) + + if err != nil { + log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) + return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) + } + + s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) + if !valid { + log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") + return buf, nil, false, fmt.Errorf("validation fail") + } + + log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + + return buf, plaintext, valid, nil +} + +func (s *secureSession) runHandshake_xx(ctx context.Context, payloadEnc []byte) (handshakeData []byte, err error) { + if s.initiator { + // stage 0 // + + err = s.xx_sendHandshakeMessage(payloadEnc, true) + if err != nil { + return nil, fmt.Errorf("stage 0 initiator fail: %s", err) + } + + // stage 1 // + + // read reply + buf, plaintext, valid, err := s.xx_recvHandshakeMessage(false) + if err != nil { + return buf, fmt.Errorf("initiator stage 1 fail: %s", err) + } + + if !valid { + return buf, fmt.Errorf("stage 1 initiator validation fail") + } + + log.Debug("stage 1 initiator", "payload", plaintext) + + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return buf, fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + if err != nil { + log.Error("stage 1 initiator set remote peer info", "err", err) + return buf, fmt.Errorf("stage 1 initiator read remote libp2p key fail") + } + + // assert that remote peer ID matches libp2p public key + pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) + if pid != s.remotePeer { + log.Error("stage 1 initiator check remote peer id err", "expected", s.remotePeer, "got", pid) + } else if err != nil { + log.Error("stage 1 initiator check remote peer id", "err", err) + } + + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.xx_ns.RemoteKey()) + if err != nil { + log.Error("stage 1 initiator verify payload", "err", err) + } + + s.noiseStaticKeyCache[s.remotePeer] = s.xx_ns.RemoteKey() + log.Debug("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) + + // stage 2 // + + err = s.xx_sendHandshakeMessage(s.local.noiseKey[:], false) + if err != nil { + return buf, fmt.Errorf("stage 2 intiator fail: %s", err) + } + + } else { + + // stage 0 // + + // read message + buf, plaintext, valid, err := s.xx_recvHandshakeMessage(true) + if err != nil { + return buf, fmt.Errorf("stage 0 responder fail: %s", err) + } + + if !valid { + return buf, fmt.Errorf("stage 0 responder validation fail") + } + + log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) + + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + if err != nil { + log.Error("stage 0 responder set remote peer info", "err", err) + return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") + } + + // assert that remote peer ID matches libp2p key + err = s.setRemotePeerID(s.RemotePublicKey()) + if err != nil { + log.Error("stage 0 responder set remote peer id", "err", err) + } + + // stage 1 // + + err = s.xx_sendHandshakeMessage(payloadEnc, false) + if err != nil { + return buf, fmt.Errorf("stage 1 responder fail: %s", err) + } + + // stage 2 // + + // read message + buf, plaintext, valid, err = s.xx_recvHandshakeMessage(false) + if err != nil { + return buf, fmt.Errorf("stage 2 responder fail: %s", err) + } + + if !valid { + return buf, fmt.Errorf("stage 2 responder validation fail") + } + + log.Debug("stage 2 responder", "plaintext", plaintext, "remote key", s.xx_ns.RemoteKey()) + + copy(s.remote.noiseKey[:], plaintext) + + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.remote.noiseKey) + if err != nil { + log.Error("stage 2 responder verify payload", "err", err) + return buf, fmt.Errorf("stage 2 responder fail: %s", err) + } + + s.noiseStaticKeyCache[s.remotePeer] = s.remote.noiseKey + log.Debug("stage 2 responder", "remote key", s.remote.noiseKey) + } + + return nil, nil +} \ No newline at end of file From 7823ac99c9a7dd712773559aaebbeb012ce4c5d2 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 09:24:43 -0400 Subject: [PATCH 1447/3965] add static key to Transport --- p2p/security/noise/ik/IK.noise.go | 2 +- p2p/security/noise/ik_handshake.go | 45 +++++--- p2p/security/noise/protocol.go | 73 +++++------- p2p/security/noise/transport.go | 7 +- p2p/security/noise/xx_handshake.go | 171 +++++++++++++++++++++++------ 5 files changed, 194 insertions(+), 104 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index c90ab1e767..c40719eaf0 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -249,7 +249,7 @@ func GenerateKeypair() Keypair { return GenerateKeypair() } -func generatePublicKey(private_key [32]byte) [32]byte { +func GeneratePublicKey(private_key [32]byte) [32]byte { var public_key [32]byte curve25519.ScalarBaseMult(&public_key, &private_key) return public_key diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 970acc4eff..f697e4ee07 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -81,16 +81,25 @@ func (s *secureSession) ik_recvHandshakeMessage() (buf []byte, plaintext []byte, return buf, plaintext, valid, nil } -func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byte) error { - // generate local static noise key - kp := ik.GenerateKeypair() +// returns last successful message upon error +func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byte) ([]byte, error) { + var kp ik.Keypair + + if s.noisePrivateKey == [32]byte{} { + // generate local static noise key + kp = ik.GenerateKeypair() + s.noisePrivateKey = kp.PrivKey() + } else { + pub := ik.GeneratePublicKey(s.noisePrivateKey) + kp = ik.NewKeypair(pub, s.noisePrivateKey) + } log.Debug("ik handshake", "pubkey", kp.PubKey()) // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() if err != nil { - return fmt.Errorf("err getting raw pubkey: %s", err) + return nil, fmt.Errorf("err getting raw pubkey: %s", err) } log.Debug("ik handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) @@ -99,7 +108,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt noise_pub := kp.PubKey() signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) if err != nil { - return fmt.Errorf("err signing payload: %s", err) + return nil, fmt.Errorf("err signing payload: %s", err) } // create payload @@ -108,19 +117,19 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt payload.NoiseStaticKeySignature = signedPayload payloadEnc, err := proto.Marshal(payload) if err != nil { - return fmt.Errorf("proto marshal payload fail: %s", err) + return nil, fmt.Errorf("proto marshal payload fail: %s", err) } // new XX noise session s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, s.noiseStaticKeyCache[s.remotePeer]) log.Debug("ik initiator init session", "remotePeer", s.noiseStaticKeyCache[s.remotePeer]) - + if s.initiator { // stage 0 // err := s.ik_sendHandshakeMessage(payloadEnc) if err != nil { log.Error("stage 0 initiator send", "err", err) - return fmt.Errorf("stage 0 initiator fail: %s", err) + return nil, fmt.Errorf("stage 0 initiator fail: %s", err) } // stage 1 // @@ -128,31 +137,33 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt } else { // stage 0 // - var plaintext []byte + log.Debug("ik responder", "noiseKey", kp.PubKey()) + var buf, plaintext []byte var valid bool if handshakeData != nil { + buf = handshakeData var msgbuf *ik.MessageBuffer msgbuf, err = ik.Decode0(handshakeData) log.Debug("stage 0 ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(handshakeData)) if err != nil { - return fmt.Errorf("stage 0 responder fail: %s", err) + return buf, fmt.Errorf("stage 0 responder fail: %s", err) } s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) } else { // read message - _, plaintext, valid, err = s.ik_recvHandshakeMessage() + buf, plaintext, valid, err = s.ik_recvHandshakeMessage() if err != nil { - return fmt.Errorf("stage 0 responder fail: %s", err) + return buf, fmt.Errorf("stage 0 responder fail: %s", err) } } if !valid { - return fmt.Errorf("stage 0 responder validation fail") + return buf, fmt.Errorf("stage 0 responder validation fail") } log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) @@ -161,14 +172,14 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt nhp := new(pb.NoiseHandshakePayload) err = proto.Unmarshal(plaintext, nhp) if err != nil { - return fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { log.Error("stage 0 responder set remote peer info", "err", err) - return fmt.Errorf("stage 0 responder read remote libp2p key fail") + return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") } // assert that remote peer ID matches libp2p key @@ -185,5 +196,5 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt } - return nil -} \ No newline at end of file + return nil, nil +} diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 04578262c6..45698f34da 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -9,7 +9,7 @@ import ( "time" log "github.com/ChainSafe/log15" - proto "github.com/gogo/protobuf/proto" + //proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" //"github.com/libp2p/go-libp2p-core/sec" @@ -39,6 +39,8 @@ type secureSession struct { noisePipesSupport bool noiseStaticKeyCache map[peer.ID]([32]byte) + + noisePrivateKey [32]byte } type peerInfo struct { @@ -46,7 +48,7 @@ type peerInfo struct { libp2pKey crypto.PubKey } -func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, +func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, noisePrivateKey [32]byte, insecure net.Conn, remote peer.ID, noiseStaticKeyCache map[peer.ID]([32]byte), noisePipesSupport bool, initiator bool) (*secureSession, error) { @@ -63,6 +65,7 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey remotePeer: remote, noisePipesSupport: noisePipesSupport, noiseStaticKeyCache: noiseStaticKeyCache, + noisePrivateKey: noisePrivateKey, } err := s.runHandshake(ctx) @@ -74,6 +77,10 @@ func (s *secureSession) NoiseStaticKeyCache() map[peer.ID]([32]byte) { return s.noiseStaticKeyCache } +func (s *secureSession) NoisePrivateKey() [32]byte { + return s.noisePrivateKey +} + func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { s.remote.libp2pKey, err = crypto.UnmarshalPublicKey(key) return err @@ -113,25 +120,25 @@ func (s *secureSession) WriteLength(length int) error { func (s *secureSession) runHandshake(ctx context.Context) error { - // TODO: check if static key for peer exists - // if not, do XX; otherwise do IK - log.Debug("runHandshake", "cache", s.noiseStaticKeyCache) - // try + // if we have the peer's noise static key and we support noise pipes, we can try IK if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} && s.noisePipesSupport { - //if s.noisePipesSupport { log.Debug("runHandshake_ik") + // ******************************************** // // known static key for peer, try IK // // ******************************************** // - err := s.runHandshake_ik(ctx, nil) + _, err := s.runHandshake_ik(ctx, nil) if err != nil { log.Error("runHandshake_ik", "err", err) - + return err // TODO: PIPE TO XX - + // _, err := s.runHandshake_xx(ctx, true, buf) + // if err != nil { + // return fmt.Errorf("runHandshake_xx err %s", err) + // } } } else { @@ -139,49 +146,19 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // unknown static key for peer, try XX // // ******************************************** // - // generate local static noise key - kp := xx.GenerateKeypair() - - log.Debug("xx handshake", "pubkey", kp.PubKey()) - - // setup libp2p keys - localKeyRaw, err := s.LocalPublicKey().Bytes() - if err != nil { - return fmt.Errorf("err getting raw pubkey: %s", err) - } - - log.Debug("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) - - // sign noise data for payload - noise_pub := kp.PubKey() - signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) - if err != nil { - return fmt.Errorf("err signing payload: %s", err) - } - - s.local.noiseKey = noise_pub - - // create payload - payload := new(pb.NoiseHandshakePayload) - payload.Libp2PKey = localKeyRaw - payload.NoiseStaticKeySignature = signedPayload - payloadEnc, err := proto.Marshal(payload) - if err != nil { - return fmt.Errorf("proto marshal payload fail: %s", err) - } - - // new XX noise session - s.xx_ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) - - handshakeData, err := s.runHandshake_xx(ctx, payloadEnc) + handshakeData, err := s.runHandshake_xx(ctx, false, nil) if err != nil { log.Error("runHandshake_xx", "err", err) log.Debug("try runHandshake_ik...") - err := s.runHandshake_ik(ctx, handshakeData) + _, err := s.runHandshake_ik(ctx, handshakeData) if err != nil { - return fmt.Errorf("runHandshake_ik err %s", err) + log.Error("runHandshake_ik", "err", err) + return err + // _, err = s.runHandshake_xx(ctx, true, msg[:32]) + // if err != nil { + // return fmt.Errorf("runHandshake_xx err %s", err) + // } } - //return fmt.Errorf("runHandshake_xx err %s", err) } } diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 23d3c2064d..ccc00b0e50 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -18,26 +18,29 @@ type Transport struct { PrivateKey crypto.PrivKey NoisePipesSupport bool NoiseStaticKeyCache map[peer.ID]([32]byte) + NoisePrivateKey [32]byte } // SecureInbound runs noise handshake as the responder func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { - s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, "", t.NoiseStaticKeyCache, t.NoisePipesSupport, false) + s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, t.NoisePrivateKey, insecure, "", t.NoiseStaticKeyCache, t.NoisePipesSupport, false) if err != nil { return s, err } t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() + t.NoisePrivateKey = s.NoisePrivateKey() return s, nil } // SecureOutbound runs noise handshake as the initiator func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { - s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, insecure, p, t.NoiseStaticKeyCache, t.NoisePipesSupport, true) + s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, t.NoisePrivateKey, insecure, p, t.NoiseStaticKeyCache, t.NoisePipesSupport, true) if err != nil { return s, err } t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() + t.NoisePrivateKey = s.NoisePrivateKey() return s, nil } diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 43fdfa52ff..6b1a86c7d4 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -7,8 +7,8 @@ import ( proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/peer" - xx "github.com/ChainSafe/go-libp2p-noise/xx" pb "github.com/ChainSafe/go-libp2p-noise/pb" + xx "github.com/ChainSafe/go-libp2p-noise/xx" ) func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { @@ -81,13 +81,58 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, return buf, plaintext, valid, nil } -func (s *secureSession) runHandshake_xx(ctx context.Context, payloadEnc []byte) (handshakeData []byte, err error) { +// if fallback = true, use msg as initial message in stage 0 +func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg []byte) (handshakeData []byte, err error) { + var kp xx.Keypair + + if s.noisePrivateKey == [32]byte{} { + // generate local static noise key + kp = xx.GenerateKeypair() + s.noisePrivateKey = kp.PrivKey() + } else { + pub := xx.GeneratePublicKey(s.noisePrivateKey) + kp = xx.NewKeypair(pub, s.noisePrivateKey) + } + + log.Debug("xx handshake", "pubkey", kp.PubKey()) + + // setup libp2p keys + localKeyRaw, err := s.LocalPublicKey().Bytes() + if err != nil { + return nil, fmt.Errorf("err getting raw pubkey: %s", err) + } + + log.Debug("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) + + // sign noise data for payload + noise_pub := kp.PubKey() + signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) + if err != nil { + return nil, fmt.Errorf("err signing payload: %s", err) + } + + s.local.noiseKey = noise_pub + + // create payload + payload := new(pb.NoiseHandshakePayload) + payload.Libp2PKey = localKeyRaw + payload.NoiseStaticKeySignature = signedPayload + payloadEnc, err := proto.Marshal(payload) + if err != nil { + return nil, fmt.Errorf("proto marshal payload fail: %s", err) + } + + // new XX noise session + s.xx_ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) + if s.initiator { // stage 0 // - err = s.xx_sendHandshakeMessage(payloadEnc, true) - if err != nil { - return nil, fmt.Errorf("stage 0 initiator fail: %s", err) + if !fallback { + err = s.xx_sendHandshakeMessage(payloadEnc, true) + if err != nil { + return nil, fmt.Errorf("stage 0 initiator fail: %s", err) + } } // stage 1 // @@ -137,46 +182,77 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, payloadEnc []byte) // stage 2 // - err = s.xx_sendHandshakeMessage(s.local.noiseKey[:], false) - if err != nil { - return buf, fmt.Errorf("stage 2 intiator fail: %s", err) + if !fallback { + err = s.xx_sendHandshakeMessage(s.local.noiseKey[:], false) + if err != nil { + return buf, fmt.Errorf("stage 2 intiator fail: %s", err) + } + } else { + err = s.xx_sendHandshakeMessage(payloadEnc, false) + if err != nil { + return buf, fmt.Errorf("stage 2 intiator fail: %s", err) + } } } else { // stage 0 // - // read message - buf, plaintext, valid, err := s.xx_recvHandshakeMessage(true) - if err != nil { - return buf, fmt.Errorf("stage 0 responder fail: %s", err) - } - - if !valid { - return buf, fmt.Errorf("stage 0 responder validation fail") - } - - log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) - - // unmarshal payload + var buf, plaintext []byte + var valid bool nhp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") - } - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) - if err != nil { - log.Error("stage 0 responder set remote peer info", "err", err) - return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") + if !fallback { + // read message + buf, plaintext, valid, err = s.xx_recvHandshakeMessage(true) + if err != nil { + return buf, fmt.Errorf("stage 0 responder fail: %s", err) + } + + if !valid { + return buf, fmt.Errorf("stage 0 responder validation fail") + } + + // unmarshal payload + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + if err != nil { + log.Error("stage 0 responder set remote peer info", "err", err) + return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") + } + + // assert that remote peer ID matches libp2p key + err = s.setRemotePeerID(s.RemotePublicKey()) + if err != nil { + log.Error("stage 0 responder set remote peer id", "err", err) + } + + } else { + var msgbuf *xx.MessageBuffer + msgbuf, err = xx.Decode0(msg) + + log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msg, "buf len", len(buf)) + + if err != nil { + log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) + return buf, fmt.Errorf("decode msg fail: %s", err) + } + + s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) + if !valid { + log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") + return buf, fmt.Errorf("validation fail") + } + + log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) } - // assert that remote peer ID matches libp2p key - err = s.setRemotePeerID(s.RemotePublicKey()) - if err != nil { - log.Error("stage 0 responder set remote peer id", "err", err) - } + log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) // stage 1 // @@ -201,6 +277,29 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, payloadEnc []byte) copy(s.remote.noiseKey[:], plaintext) + if fallback { + + // unmarshal payload + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + if err != nil { + log.Error("stage 0 responder set remote peer info", "err", err) + return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") + } + + // assert that remote peer ID matches libp2p key + err = s.setRemotePeerID(s.RemotePublicKey()) + if err != nil { + log.Error("stage 0 responder set remote peer id", "err", err) + } + + } + // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.remote.noiseKey) if err != nil { @@ -213,4 +312,4 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, payloadEnc []byte) } return nil, nil -} \ No newline at end of file +} From 5c2c2439987beca4ffa0bbc14fae24bc0392af27 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 09:32:50 -0400 Subject: [PATCH 1448/3965] ik functional w/ fallback from xx to ik --- p2p/security/noise/ik/IK.noise.go | 49 ++++++++++---------- p2p/security/noise/ik_handshake.go | 74 ++++++++++++++++++++++++------ p2p/security/noise/xx_handshake.go | 2 +- 3 files changed, 84 insertions(+), 41 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index c40719eaf0..1f9093bcde 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -151,19 +151,18 @@ func (mb *MessageBuffer) Encode0() []byte { return enc } -// Encodes a MessageBuffer from stage 1 and 2 -// func (mb *MessageBuffer) Encode1() []byte { -// enc := []byte{} +// Encodes a MessageBuffer from stage 1 +func (mb *MessageBuffer) Encode1() []byte { + enc := []byte{} -// enc = append(enc, mb.ne[:]...) -// enc = append(enc, mb.ns...) -// enc = append(enc, mb.ciphertext...) + enc = append(enc, mb.ne[:]...) + enc = append(enc, mb.ciphertext...) -// // log.Debug("XX_Encode1", "ne", mb.ne) -// // log.Debug("XX_Encode1", "ns", mb.ns) + // log.Debug("XX_Encode1", "ne", mb.ne) + // log.Debug("XX_Encode1", "ns", mb.ns) -// return enc -// } + return enc +} // Decodes initial message (stage 0) into MessageBuffer func Decode0(in []byte) (*MessageBuffer, error) { @@ -181,24 +180,24 @@ func Decode0(in []byte) (*MessageBuffer, error) { return mb, nil } -// Decodes messages at stage 1 or 2 into MessageBuffer -// func Decode1(in []byte) (*MessageBuffer, error) { -// if len(in) < 80 { -// return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") -// } +// Decodes messages at stage 1 into MessageBuffer +func Decode1(in []byte) (*MessageBuffer, error) { + if len(in) < 80 { + return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") + } -// // log.Debug("XX_Decode1", "in", in) -// // log.Debug("XX_Decode1", "ns", in[32:80]) + // log.Debug("XX_Decode1", "in", in) + // log.Debug("XX_Decode1", "ns", in[32:80]) -// mb := new(MessageBuffer) -// copy(mb.ne[:], in[:32]) -// mb.ns = in[32:80] -// mb.ciphertext = in[80:] -// // copy(mb.ns,) -// // copy(mb.ciphertext,) + mb := new(MessageBuffer) + copy(mb.ne[:], in[:32]) + //mb.ns = in[32:80] + mb.ciphertext = in[32:] + // copy(mb.ns,) + // copy(mb.ciphertext,) -// return mb, nil -// } + return mb, nil +} func validatePublicKey(k []byte) bool { forbiddenCurveValues := [12][]byte{ diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index f697e4ee07..39dda54a08 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -11,18 +11,18 @@ import ( pb "github.com/ChainSafe/go-libp2p-noise/pb" ) -func (s *secureSession) ik_sendHandshakeMessage(payload []byte) error { +func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bool) error { log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload)) // create send message w payload var msgbuf ik.MessageBuffer s.ik_ns, msgbuf = ik.SendMessage(s.ik_ns, payload) var encMsgBuf []byte - //if initial_stage { - encMsgBuf = msgbuf.Encode0() - // } else { - // encMsgBuf = msgbuf.Encode1() - // } + if initial_stage { + encMsgBuf = msgbuf.Encode0() + } else { + encMsgBuf = msgbuf.Encode1() + } log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) @@ -43,7 +43,7 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte) error { return nil } -func (s *secureSession) ik_recvHandshakeMessage() (buf []byte, plaintext []byte, valid bool, err error) { +func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { l, err := s.ReadLength() if err != nil { return nil, nil, false, fmt.Errorf("read length fail: %s", err) @@ -57,11 +57,11 @@ func (s *secureSession) ik_recvHandshakeMessage() (buf []byte, plaintext []byte, } var msgbuf *ik.MessageBuffer - //if initial_stage { - msgbuf, err = ik.Decode0(buf) - // } else { - // msgbuf, err = ik.Decode1(buf) - // } + if initial_stage { + msgbuf, err = ik.Decode0(buf) + } else { + msgbuf, err = ik.Decode1(buf) + } log.Debug("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) @@ -87,7 +87,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt if s.noisePrivateKey == [32]byte{} { // generate local static noise key - kp = ik.GenerateKeypair() + kp = ik.GenerateKeypair() s.noisePrivateKey = kp.PrivKey() } else { pub := ik.GeneratePublicKey(s.noisePrivateKey) @@ -126,7 +126,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt if s.initiator { // stage 0 // - err := s.ik_sendHandshakeMessage(payloadEnc) + err := s.ik_sendHandshakeMessage(payloadEnc, true) if err != nil { log.Error("stage 0 initiator send", "err", err) return nil, fmt.Errorf("stage 0 initiator fail: %s", err) @@ -134,6 +134,42 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt // stage 1 // + // read message + buf, plaintext, valid, err := s.ik_recvHandshakeMessage(false) + if err != nil { + return buf, fmt.Errorf("stage 1 initiator fail: %s", err) + } + + if !valid { + return buf, fmt.Errorf("stage 1 initiator validation fail") + } + + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return buf, fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + if err != nil { + log.Error("stage 1 initiator set remote peer info", "err", err) + return buf, fmt.Errorf("stage 1 initiator read remote libp2p key fail") + } + + // assert that remote peer ID matches libp2p key + err = s.setRemotePeerID(s.RemotePublicKey()) + if err != nil { + log.Error("stage 1 initiator set remote peer id", "err", err) + } + + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.noiseStaticKeyCache[s.remotePeer]) + if err != nil { + log.Error("stage 1 initiator verify payload", "err", err) + } + } else { // stage 0 // @@ -155,7 +191,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) } else { // read message - buf, plaintext, valid, err = s.ik_recvHandshakeMessage() + buf, plaintext, valid, err = s.ik_recvHandshakeMessage(true) if err != nil { return buf, fmt.Errorf("stage 0 responder fail: %s", err) } @@ -194,6 +230,14 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt log.Error("stage 1 responder verify payload", "err", err) } + // stage 1 // + + err := s.ik_sendHandshakeMessage(payloadEnc, false) + if err != nil { + log.Error("stage 1 responder send", "err", err) + return nil, fmt.Errorf("stage 1 responder fail: %s", err) + } + } return nil, nil diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 6b1a86c7d4..c5a256fe6a 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -88,7 +88,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg if s.noisePrivateKey == [32]byte{} { // generate local static noise key kp = xx.GenerateKeypair() - s.noisePrivateKey = kp.PrivKey() + s.noisePrivateKey = kp.PrivKey() } else { pub := xx.GeneratePublicKey(s.noisePrivateKey) kp = xx.NewKeypair(pub, s.noisePrivateKey) From 42481bec470cc8ca4345192e3195a1a14d496e88 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 09:54:39 -0400 Subject: [PATCH 1449/3965] add encrypt/decrypt message tests to XX --- p2p/security/noise/ik_handshake.go | 1 - p2p/security/noise/protocol.go | 5 +-- p2p/security/noise/xx/XX.noise.go | 12 +++++-- p2p/security/noise/xx/XX_test.go | 50 +++++++++++++++++++++++++++++- 4 files changed, 60 insertions(+), 8 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 39dda54a08..7b634efa9f 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -5,7 +5,6 @@ import ( "fmt" log "github.com/ChainSafe/log15" proto "github.com/gogo/protobuf/proto" - //"github.com/libp2p/go-libp2p-core/peer" ik "github.com/ChainSafe/go-libp2p-noise/ik" pb "github.com/ChainSafe/go-libp2p-noise/pb" diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 45698f34da..2bbbb628a4 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -126,9 +126,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} && s.noisePipesSupport { log.Debug("runHandshake_ik") - // ******************************************** // // known static key for peer, try IK // - // ******************************************** // _, err := s.runHandshake_ik(ctx, nil) if err != nil { @@ -142,9 +140,8 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } } else { - // ******************************************** // + // unknown static key for peer, try XX // - // ******************************************** // handshakeData, err := s.runHandshake_xx(ctx, false, nil) if err != nil { diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 3c4351cf07..5dd585926a 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -73,6 +73,14 @@ type NoiseSession struct { i bool } +func (ns *NoiseSession) CS1() *cipherstate { + return &ns.cs1 +} + +func (ns *NoiseSession) CS2() *cipherstate { + return &ns.cs2 +} + func NewKeypair(pub [32]byte, priv [32]byte) Keypair { return Keypair{ public_key: pub, @@ -532,7 +540,7 @@ func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageB } if session.mc == 2 { session.h, messageBuffer, session.cs1, session.cs2 = writeMessageC(&session.hs, message) - session.hs = handshakestate{} + //session.hs = handshakestate{} } session.mc = session.mc + 1 return session, messageBuffer @@ -549,7 +557,7 @@ func RecvMessage(session *NoiseSession, message *MessageBuffer) (*NoiseSession, } if session.mc == 2 { session.h, plaintext, valid, session.cs1, session.cs2 = readMessageC(&session.hs, message) - session.hs = handshakestate{} + //session.hs = handshakestate{} } session.mc = session.mc + 1 return session, plaintext, valid diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index ad127facd1..585efa4bd9 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -28,7 +28,7 @@ func TestGetHkdf(t *testing.T) { t.Logf("%x", c) } -func TestHandshake(t *testing.T) { +func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { // generate local static noise key kp_init := GenerateKeypair() kp_resp := GenerateKeypair() @@ -142,4 +142,52 @@ func TestHandshake(t *testing.T) { t.Logf("stage 2 resp payload: %x", plaintext) + return ns_init, ns_resp +} + +func TestHandshake(t *testing.T) { + _, _ = doHandshake(t) +} + +func TestSymmetricEncryptAndDecrypt(t *testing.T) { + ns_init, ns_resp := doHandshake(t) + cs1_init := ns_init.CS1() + cs1_resp := ns_resp.CS1() + + var out []byte + var ok bool + ad := []byte("authenticated") + msg := []byte("helloworld") + cs1_init, out = encryptWithAd(cs1_init, ad, msg) + + cs1_resp, out, ok = decryptWithAd(cs1_resp, ad, out) + if !ok { + t.Fatal("could not decrypt") + } +} + +func TestSymmetricEncryptAndDecryptAgain(t *testing.T) { + ns_init, ns_resp := doHandshake(t) + cs2_init := ns_init.CS2() + cs2_resp := ns_resp.CS2() + + var out []byte + var ok bool + ad := []byte("authenticated") + msg := []byte("helloworld") + cs2_resp, out = encryptWithAd(cs2_resp, ad, msg) + + cs2_init, out, ok = decryptWithAd(cs2_init, ad, out) + if !ok { + t.Fatal("could not decrypt") + } + + ad = []byte("authenticatedagain") + msg = []byte("helloworld2") + cs2_resp, out = encryptWithAd(cs2_resp, ad, msg) + + cs2_init, out, ok = decryptWithAd(cs2_init, ad, out) + if !ok { + t.Fatal("could not decrypt") + } } From fe54e1c0ead846cc488cd312395d1678a895436a Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 10:31:30 -0400 Subject: [PATCH 1450/3965] add encrypt and decrypt funcs --- p2p/security/noise/crypto.go | 60 +++++++++++++++++++++++++++++++ p2p/security/noise/ik/IK.noise.go | 16 ++++++--- p2p/security/noise/protocol.go | 13 +++++-- p2p/security/noise/xx/XX.noise.go | 8 ++--- p2p/security/noise/xx/XX_test.go | 12 +++---- 5 files changed, 93 insertions(+), 16 deletions(-) create mode 100644 p2p/security/noise/crypto.go diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go new file mode 100644 index 0000000000..0b5830d38f --- /dev/null +++ b/p2p/security/noise/crypto.go @@ -0,0 +1,60 @@ +package noise + +import ( + "errors" + ik "github.com/ChainSafe/go-libp2p-noise/ik" + xx "github.com/ChainSafe/go-libp2p-noise/xx" +) + +func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) { + if s.xx_complete { + if s.initiator { + cs := s.xx_ns.CS1() + cs, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) + } else { + cs := s.xx_ns.CS2() + cs, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) + } + } else if s.ik_complete { + if s.initiator { + cs := s.ik_ns.CS1() + cs, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) + } else { + cs := s.ik_ns.CS2() + cs, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) + } + } else { + return nil, errors.New("encrypt err: haven't completed handshake") + } + + return ciphertext, nil +} + +func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) { + var ok bool + if s.xx_complete { + if s.initiator { + cs := s.xx_ns.CS1() + cs, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) + } else { + cs := s.xx_ns.CS2() + cs, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) + } + } else if s.ik_complete { + if s.initiator { + cs := s.ik_ns.CS1() + cs, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) + } else { + cs := s.ik_ns.CS2() + cs, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) + } + } else { + return nil, errors.New("decrypt err: haven't completed handshake") + } + + if !ok { + return nil, errors.New("decrypt err: could not decrypt") + } + + return plaintext, nil +} diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index 1f9093bcde..01fa38e33c 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -72,6 +72,14 @@ type NoiseSession struct { i bool } +func (ns *NoiseSession) CS1() *cipherstate { + return &ns.cs1 +} + +func (ns *NoiseSession) CS2() *cipherstate { + return &ns.cs2 +} + func NewKeypair(pub [32]byte, priv [32]byte) Keypair { return Keypair{ public_key: pub, @@ -320,13 +328,13 @@ func setNonce(cs *cipherstate, newNonce uint32) *cipherstate { return cs } -func encryptWithAd(cs *cipherstate, ad []byte, plaintext []byte) (*cipherstate, []byte) { +func EncryptWithAd(cs *cipherstate, ad []byte, plaintext []byte) (*cipherstate, []byte) { e := encrypt(cs.k, cs.n, ad, plaintext) cs = setNonce(cs, incrementNonce(cs.n)) return cs, e } -func decryptWithAd(cs *cipherstate, ad []byte, ciphertext []byte) (*cipherstate, []byte, bool) { +func DecryptWithAd(cs *cipherstate, ad []byte, ciphertext []byte) (*cipherstate, []byte, bool) { valid, ad, plaintext := decrypt(cs.k, cs.n, ad, ciphertext) cs = setNonce(cs, incrementNonce(cs.n)) return cs, plaintext, valid @@ -375,7 +383,7 @@ func getHandshakeHash(ss *symmetricstate) [32]byte { func encryptAndHash(ss *symmetricstate, plaintext []byte) (*symmetricstate, []byte) { var ciphertext []byte if hasKey(&ss.cs) { - _, ciphertext = encryptWithAd(&ss.cs, ss.h[:], plaintext) + _, ciphertext = EncryptWithAd(&ss.cs, ss.h[:], plaintext) } else { ciphertext = plaintext } @@ -387,7 +395,7 @@ func decryptAndHash(ss *symmetricstate, ciphertext []byte) (*symmetricstate, []b var plaintext []byte var valid bool if hasKey(&ss.cs) { - _, plaintext, valid = decryptWithAd(&ss.cs, ss.h[:], ciphertext) + _, plaintext, valid = DecryptWithAd(&ss.cs, ss.h[:], ciphertext) } else { plaintext, valid = ciphertext, true } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 2bbbb628a4..23bf49ae6c 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -37,6 +37,9 @@ type secureSession struct { xx_ns *xx.NoiseSession ik_ns *ik.NoiseSession + xx_complete bool + ik_complete bool + noisePipesSupport bool noiseStaticKeyCache map[peer.ID]([32]byte) @@ -139,6 +142,8 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // } } + s.ik_complete = true + } else { // unknown static key for peer, try XX // @@ -156,7 +161,11 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // return fmt.Errorf("runHandshake_xx err %s", err) // } } + + s.ik_complete = true } + + s.xx_complete = true } return nil @@ -178,9 +187,9 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { return s.localKey.GetPublic() } -func (s *secureSession) Read(in []byte) (int, error) { +func (s *secureSession) Read(buf []byte) (int, error) { // TODO: use noise symmetric keys - return s.insecure.Read(in) + return s.insecure.Read(buf) } func (s *secureSession) RemoteAddr() net.Addr { diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 5dd585926a..fc35d282d7 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -330,13 +330,13 @@ func setNonce(cs *cipherstate, newNonce uint32) *cipherstate { return cs } -func encryptWithAd(cs *cipherstate, ad []byte, plaintext []byte) (*cipherstate, []byte) { +func EncryptWithAd(cs *cipherstate, ad []byte, plaintext []byte) (*cipherstate, []byte) { e := encrypt(cs.k, cs.n, ad, plaintext) cs = setNonce(cs, incrementNonce(cs.n)) return cs, e } -func decryptWithAd(cs *cipherstate, ad []byte, ciphertext []byte) (*cipherstate, []byte, bool) { +func DecryptWithAd(cs *cipherstate, ad []byte, ciphertext []byte) (*cipherstate, []byte, bool) { valid, ad, plaintext := decrypt(cs.k, cs.n, ad, ciphertext) cs = setNonce(cs, incrementNonce(cs.n)) return cs, plaintext, valid @@ -385,7 +385,7 @@ func getHandshakeHash(ss *symmetricstate) [32]byte { func encryptAndHash(ss *symmetricstate, plaintext []byte) (*symmetricstate, []byte) { var ciphertext []byte if hasKey(&ss.cs) { - _, ciphertext = encryptWithAd(&ss.cs, ss.h[:], plaintext) + _, ciphertext = EncryptWithAd(&ss.cs, ss.h[:], plaintext) } else { ciphertext = plaintext } @@ -397,7 +397,7 @@ func decryptAndHash(ss *symmetricstate, ciphertext []byte) (*symmetricstate, []b var plaintext []byte var valid bool if hasKey(&ss.cs) { - _, plaintext, valid = decryptWithAd(&ss.cs, ss.h[:], ciphertext) + _, plaintext, valid = DecryptWithAd(&ss.cs, ss.h[:], ciphertext) } else { plaintext, valid = ciphertext, true } diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 585efa4bd9..4a6065d464 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -158,9 +158,9 @@ func TestSymmetricEncryptAndDecrypt(t *testing.T) { var ok bool ad := []byte("authenticated") msg := []byte("helloworld") - cs1_init, out = encryptWithAd(cs1_init, ad, msg) + cs1_init, out = EncryptWithAd(cs1_init, ad, msg) - cs1_resp, out, ok = decryptWithAd(cs1_resp, ad, out) + cs1_resp, out, ok = DecryptWithAd(cs1_resp, ad, out) if !ok { t.Fatal("could not decrypt") } @@ -175,18 +175,18 @@ func TestSymmetricEncryptAndDecryptAgain(t *testing.T) { var ok bool ad := []byte("authenticated") msg := []byte("helloworld") - cs2_resp, out = encryptWithAd(cs2_resp, ad, msg) + cs2_resp, out = EncryptWithAd(cs2_resp, ad, msg) - cs2_init, out, ok = decryptWithAd(cs2_init, ad, out) + cs2_init, out, ok = DecryptWithAd(cs2_init, ad, out) if !ok { t.Fatal("could not decrypt") } ad = []byte("authenticatedagain") msg = []byte("helloworld2") - cs2_resp, out = encryptWithAd(cs2_resp, ad, msg) + cs2_resp, out = EncryptWithAd(cs2_resp, ad, msg) - cs2_init, out, ok = decryptWithAd(cs2_init, ad, out) + cs2_init, out, ok = DecryptWithAd(cs2_init, ad, out) if !ok { t.Fatal("could not decrypt") } From 963da41454f8d3efd5677113eec6ea36d60aaf4c Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Sat, 24 Aug 2019 10:42:51 -0400 Subject: [PATCH 1451/3965] use new constructor for insecure transport (#706) --- config/config.go | 14 ++++++-------- config/security.go | 5 +++-- 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/config/config.go b/config/config.go index 3bb51bf1f7..39d1d2f1aa 100644 --- a/config/config.go +++ b/config/config.go @@ -100,13 +100,11 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return nil, fmt.Errorf("no peerstore specified") } - if !cfg.Insecure { - if err := cfg.Peerstore.AddPrivKey(pid, cfg.PeerKey); err != nil { - return nil, err - } - if err := cfg.Peerstore.AddPubKey(pid, cfg.PeerKey.GetPublic()); err != nil { - return nil, err - } + if err := cfg.Peerstore.AddPrivKey(pid, cfg.PeerKey); err != nil { + return nil, err + } + if err := cfg.Peerstore.AddPubKey(pid, cfg.PeerKey.GetPublic()); err != nil { + return nil, err } // TODO: Make the swarm implementation configurable. @@ -142,7 +140,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { upgrader.Protector = cfg.Protector upgrader.Filters = swrm.Filters if cfg.Insecure { - upgrader.Secure = makeInsecureTransport(pid) + upgrader.Secure = makeInsecureTransport(pid, cfg.PeerKey) } else { upgrader.Secure, err = makeSecurityTransport(h, cfg.SecurityTransports) if err != nil { diff --git a/config/security.go b/config/security.go index 8e1c710cbf..72d4b09487 100644 --- a/config/security.go +++ b/config/security.go @@ -3,6 +3,7 @@ package config import ( "fmt" + "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" @@ -49,9 +50,9 @@ func SecurityConstructor(security interface{}) (SecC, error) { }, nil } -func makeInsecureTransport(id peer.ID) sec.SecureTransport { +func makeInsecureTransport(id peer.ID, privKey crypto.PrivKey) sec.SecureTransport { secMuxer := new(csms.SSMuxer) - secMuxer.AddTransport(insecure.ID, insecure.New(id)) + secMuxer.AddTransport(insecure.ID, insecure.NewWithIdentity(id, privKey)) return secMuxer } From a3c39ad62a5d215b78fd95947dc304784faa98f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sat, 24 Aug 2019 17:02:40 +0200 Subject: [PATCH 1452/3965] dep: upgrade eventbus to v0.1.0 (#709) --- go.mod | 4 ++-- go.sum | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index 422a39d01d..98d1e1f269 100644 --- a/go.mod +++ b/go.mod @@ -9,11 +9,11 @@ require ( github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 - github.com/libp2p/go-eventbus v0.0.2 + github.com/libp2p/go-eventbus v0.1.0 github.com/libp2p/go-libp2p-autonat v0.1.0 github.com/libp2p/go-libp2p-blankhost v0.1.3 github.com/libp2p/go-libp2p-circuit v0.1.1 - github.com/libp2p/go-libp2p-core v0.2.0 + github.com/libp2p/go-libp2p-core v0.2.2 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 diff --git a/go.sum b/go.sum index 3ab31cc07b..b5666cbff3 100644 --- a/go.sum +++ b/go.sum @@ -112,6 +112,8 @@ github.com/libp2p/go-conn-security-multistream v0.1.0 h1:aqGmto+ttL/uJgX0JtQI0tD github.com/libp2p/go-conn-security-multistream v0.1.0/go.mod h1:aw6eD7LOsHEX7+2hJkDxw1MteijaVcI+/eP2/x3J1xc= github.com/libp2p/go-eventbus v0.0.2 h1:L9eslON8FjFBJlyUs9fyEZKnxSqZd2AMDUNldPrqmZI= github.com/libp2p/go-eventbus v0.0.2/go.mod h1:Hr/yGlwxA/stuLnpMiu82lpNKpvRy3EaJxPu40XYOwk= +github.com/libp2p/go-eventbus v0.1.0 h1:mlawomSAjjkk97QnYiEmHsLu7E136+2oCWSHRUvMfzQ= +github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UGQf8A2eEmx4= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-libp2p-autonat v0.1.0 h1:aCWAu43Ri4nU0ZPO7NyLzUvvfqd0nE3dX0R/ZGYVgOU= @@ -127,6 +129,8 @@ github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyL github.com/libp2p/go-libp2p-core v0.0.6/go.mod h1:0d9xmaYAVY5qmbp/fcgxHT3ZJsLjYeYPMJAUKpaCHrE= github.com/libp2p/go-libp2p-core v0.2.0 h1:ycFtuNwtZBAJSxzaHbyv6NjG3Yj5Nmra1csHaQ3zwaw= github.com/libp2p/go-libp2p-core v0.2.0/go.mod h1:X0eyB0Gy93v0DZtSYbEM7RnMChm9Uv3j7yRXjO77xSI= +github.com/libp2p/go-libp2p-core v0.2.2 h1:Sv1ggdoMx9c7v7FOFkR7agraHCnAgqYsXrU1ARSRUMs= +github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZVoVrLaDRvIRng0= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= @@ -173,6 +177,8 @@ github.com/libp2p/go-msgio v0.0.4 h1:agEFehY3zWJFUHK6SEMR7UYmk2z6kC3oeCM7ybLhguA github.com/libp2p/go-msgio v0.0.4/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-nat v0.0.3 h1:l6fKV+p0Xa354EqQOQP+d8CivdLM4kl5GxC1hSc/UeI= github.com/libp2p/go-nat v0.0.3/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= +github.com/libp2p/go-openssl v0.0.2 h1:9pP2d3Ubaxkv7ZisLjx9BFwgOGnQdQYnfcH29HNY3ls= +github.com/libp2p/go-openssl v0.0.2/go.mod h1:v8Zw2ijCSWBQi8Pq5GAixw6DbFfa9u6VIYDXnvOXkc0= github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0GIEjMiHTwR/KN4= From 443240dc363bc1689c204476b1c539d7d72df6a2 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 12:29:27 -0400 Subject: [PATCH 1453/3965] add secure read/writes --- p2p/security/noise/crypto.go | 8 ++--- p2p/security/noise/crypto_test.go | 30 +++++++++++++++++++ p2p/security/noise/protocol.go | 45 ++++++++++++++++++++++++++-- p2p/security/noise/transport_test.go | 37 ++++------------------- 4 files changed, 82 insertions(+), 38 deletions(-) create mode 100644 p2p/security/noise/crypto_test.go diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index 0b5830d38f..912fcca59e 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -34,18 +34,18 @@ func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) var ok bool if s.xx_complete { if s.initiator { - cs := s.xx_ns.CS1() + cs := s.xx_ns.CS2() cs, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) } else { - cs := s.xx_ns.CS2() + cs := s.xx_ns.CS1() cs, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) } } else if s.ik_complete { if s.initiator { - cs := s.ik_ns.CS1() + cs := s.ik_ns.CS2() cs, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) } else { - cs := s.ik_ns.CS2() + cs := s.ik_ns.CS1() cs, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) } } else { diff --git a/p2p/security/noise/crypto_test.go b/p2p/security/noise/crypto_test.go new file mode 100644 index 0000000000..f023ec8965 --- /dev/null +++ b/p2p/security/noise/crypto_test.go @@ -0,0 +1,30 @@ +package noise + +import ( + "bytes" + "testing" + + "github.com/libp2p/go-libp2p-core/crypto" +) + +func TestEncryptAndDecrypt(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + plaintext := []byte("helloworld") + ciphertext, err := initConn.Encrypt(plaintext) + if err != nil { + t.Fatal(err) + } + + result, err := respConn.Decrypt(ciphertext) + if !bytes.Equal(plaintext, result) { + t.Fatalf("got %x expected %x", result, plaintext) + } else if err != nil { + t.Fatal(err) + } +} \ No newline at end of file diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 23bf49ae6c..92874141e0 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -189,7 +189,30 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { func (s *secureSession) Read(buf []byte) (int, error) { // TODO: use noise symmetric keys - return s.insecure.Read(buf) + //return s.insecure.Read(buf) + + plaintext, err := s.ReadSecure() + if err != nil { + return 0, nil + } + + copy(buf, plaintext) + return len(buf), nil +} + +func (s *secureSession) ReadSecure() ([]byte, error) { + l, err := s.ReadLength() + if err != nil { + return nil, err + } + + ciphertext := make([]byte, l) + _, err = s.Read(ciphertext) + if err != nil { + return nil, err + } + + return s.Decrypt(ciphertext) } func (s *secureSession) RemoteAddr() net.Addr { @@ -218,7 +241,25 @@ func (s *secureSession) SetWriteDeadline(t time.Time) error { func (s *secureSession) Write(in []byte) (int, error) { // TODO: use noise symmetric keys - return s.insecure.Write(in) + //return s.insecure.Write(in) + + err := s.WriteSecure(in) + return len(in), err +} + +func (s *secureSession) WriteSecure(in []byte) error { + ciphertext, err := s.Encrypt(in) + if err != nil { + return err + } + + err = s.WriteLength(len(ciphertext)) + if err != nil { + return err + } + + _, err = s.Write(ciphertext) + return err } func (s *secureSession) Close() error { diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index c21f7f24d1..30528db149 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -3,7 +3,6 @@ package noise import ( "bytes" "context" - "io" "net" "testing" @@ -77,7 +76,7 @@ func newConnPair(t *testing.T) (net.Conn, net.Conn) { return client, server } -func connect(t *testing.T, initTransport, respTransport *Transport) (sec.SecureConn, sec.SecureConn) { +func connect(t *testing.T, initTransport, respTransport *Transport) (*secureSession, *secureSession) { init, resp := newConnPair(t) var respConn sec.SecureConn @@ -99,7 +98,7 @@ func connect(t *testing.T, initTransport, respTransport *Transport) (sec.SecureC t.Fatal(respErr) } - return initConn, respConn + return initConn.(*secureSession), respConn.(*secureSession) } func TestIDs(t *testing.T) { @@ -148,31 +147,6 @@ func TestKeys(t *testing.T) { } } -func TestRW(t *testing.T) { - initTransport := newTestTransport(t, crypto.Ed25519, 2048) - respTransport := newTestTransport(t, crypto.Ed25519, 2048) - - initConn, respConn := connect(t, initTransport, respTransport) - defer initConn.Close() - defer respConn.Close() - - before := []byte("hello world") - _, err := initConn.Write(before) - if err != nil { - t.Fatal(err) - } - - after := make([]byte, len(before)) - _, err = io.ReadFull(respConn, after) - if err != nil { - t.Fatal(err) - } - - if !bytes.Equal(before, after) { - t.Errorf("Message mismatch. %v != %v", before, after) - } -} - // Tests XX handshake func TestHandshakeXX(t *testing.T) { initTransport := newTestTransport(t, crypto.Ed25519, 2048) @@ -189,7 +163,7 @@ func TestHandshakeXX(t *testing.T) { } after := make([]byte, len(before)) - _, err = io.ReadFull(respConn, after) + _, err = respConn.Read(after) if err != nil { t.Fatal(err) } @@ -221,7 +195,7 @@ func TestHandshakeIK(t *testing.T) { } after := make([]byte, len(before)) - _, err = io.ReadFull(respConn, after) + _, err = respConn.Read(after) if err != nil { t.Fatal(err) } @@ -229,5 +203,4 @@ func TestHandshakeIK(t *testing.T) { if !bytes.Equal(before, after) { t.Errorf("Message mismatch. %v != %v", before, after) } - -} +} \ No newline at end of file From 50b18700deafade42dfa8a9d174163ec32411652 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 13:50:00 -0400 Subject: [PATCH 1454/3965] wip XXfallback --- p2p/security/noise/crypto_test.go | 4 +- p2p/security/noise/ik_handshake.go | 46 +++------ p2p/security/noise/protocol.go | 37 +++---- p2p/security/noise/transport_test.go | 4 +- p2p/security/noise/xx_handshake.go | 141 ++++++++++++++++----------- 5 files changed, 120 insertions(+), 112 deletions(-) diff --git a/p2p/security/noise/crypto_test.go b/p2p/security/noise/crypto_test.go index f023ec8965..807edb8434 100644 --- a/p2p/security/noise/crypto_test.go +++ b/p2p/security/noise/crypto_test.go @@ -1,4 +1,4 @@ -package noise +package noise import ( "bytes" @@ -27,4 +27,4 @@ func TestEncryptAndDecrypt(t *testing.T) { } else if err != nil { t.Fatal(err) } -} \ No newline at end of file +} diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 7b634efa9f..f37f836623 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -17,11 +17,11 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo var msgbuf ik.MessageBuffer s.ik_ns, msgbuf = ik.SendMessage(s.ik_ns, payload) var encMsgBuf []byte - if initial_stage { - encMsgBuf = msgbuf.Encode0() - } else { - encMsgBuf = msgbuf.Encode1() - } + //if initial_stage { + encMsgBuf = msgbuf.Encode0() + // } else { + // encMsgBuf = msgbuf.Encode1() + // } log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) @@ -56,11 +56,11 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, } var msgbuf *ik.MessageBuffer - if initial_stage { - msgbuf, err = ik.Decode0(buf) - } else { - msgbuf, err = ik.Decode1(buf) - } + //if initial_stage { + msgbuf, err = ik.Decode0(buf) + // } else { + // msgbuf, err = ik.Decode1(buf) + // } log.Debug("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) @@ -81,7 +81,7 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, } // returns last successful message upon error -func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byte) ([]byte, error) { +func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { var kp ik.Keypair if s.noisePrivateKey == [32]byte{} { @@ -176,25 +176,10 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt var buf, plaintext []byte var valid bool - if handshakeData != nil { - buf = handshakeData - var msgbuf *ik.MessageBuffer - msgbuf, err = ik.Decode0(handshakeData) - - log.Debug("stage 0 ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(handshakeData)) - - if err != nil { - return buf, fmt.Errorf("stage 0 responder fail: %s", err) - } - - s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) - } else { - // read message - buf, plaintext, valid, err = s.ik_recvHandshakeMessage(true) - if err != nil { - return buf, fmt.Errorf("stage 0 responder fail: %s", err) - } - + // read message + buf, plaintext, valid, err = s.ik_recvHandshakeMessage(true) + if err != nil { + return buf, fmt.Errorf("stage 0 responder fail: %s", err) } if !valid { @@ -239,5 +224,6 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, handshakeData []byt } + log.Debug("ik_handshake done", "initiator", s.initiator) return nil, nil } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 92874141e0..0b3aeeffec 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -126,20 +126,24 @@ func (s *secureSession) runHandshake(ctx context.Context) error { log.Debug("runHandshake", "cache", s.noiseStaticKeyCache) // if we have the peer's noise static key and we support noise pipes, we can try IK - if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} && s.noisePipesSupport { + if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} || s.noisePipesSupport { log.Debug("runHandshake_ik") // known static key for peer, try IK // - _, err := s.runHandshake_ik(ctx, nil) + buf, err := s.runHandshake_ik(ctx) if err != nil { log.Error("runHandshake_ik", "err", err) - return err + // TODO: PIPE TO XX - // _, err := s.runHandshake_xx(ctx, true, buf) - // if err != nil { - // return fmt.Errorf("runHandshake_xx err %s", err) - // } + + err = s.runHandshake_xx(ctx, true, buf) + if err != nil { + log.Error("runHandshake_xx", "err", err) + return fmt.Errorf("runHandshake_xx err %s", err) + } + + s.xx_complete = true } s.ik_complete = true @@ -148,21 +152,10 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // unknown static key for peer, try XX // - handshakeData, err := s.runHandshake_xx(ctx, false, nil) + err := s.runHandshake_xx(ctx, false, nil) if err != nil { log.Error("runHandshake_xx", "err", err) - log.Debug("try runHandshake_ik...") - _, err := s.runHandshake_ik(ctx, handshakeData) - if err != nil { - log.Error("runHandshake_ik", "err", err) - return err - // _, err = s.runHandshake_xx(ctx, true, msg[:32]) - // if err != nil { - // return fmt.Errorf("runHandshake_xx err %s", err) - // } - } - - s.ik_complete = true + return err } s.xx_complete = true @@ -189,7 +182,7 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { func (s *secureSession) Read(buf []byte) (int, error) { // TODO: use noise symmetric keys - //return s.insecure.Read(buf) + return s.insecure.Read(buf) plaintext, err := s.ReadSecure() if err != nil { @@ -241,7 +234,7 @@ func (s *secureSession) SetWriteDeadline(t time.Time) error { func (s *secureSession) Write(in []byte) (int, error) { // TODO: use noise symmetric keys - //return s.insecure.Write(in) + return s.insecure.Write(in) err := s.WriteSecure(in) return len(in), err diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 30528db149..7478b2eeed 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -183,8 +183,6 @@ func TestHandshakeIK(t *testing.T) { respConn.Close() initConn, respConn = connect(t, initTransport, respTransport) - t.Log(initTransport) - t.Log(respTransport) defer initConn.Close() defer respConn.Close() @@ -203,4 +201,4 @@ func TestHandshakeIK(t *testing.T) { if !bytes.Equal(before, after) { t.Errorf("Message mismatch. %v != %v", before, after) } -} \ No newline at end of file +} diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index c5a256fe6a..6e1eeb3448 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -25,7 +25,7 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo } log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) + //log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) err := s.WriteLength(len(encMsgBuf)) if err != nil { @@ -63,7 +63,7 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, msgbuf, err = xx.Decode1(buf) } - log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) + //log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) if err != nil { log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) @@ -82,7 +82,7 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, } // if fallback = true, use msg as initial message in stage 0 -func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg []byte) (handshakeData []byte, err error) { +func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, initialMsg []byte) (err error) { var kp xx.Keypair if s.noisePrivateKey == [32]byte{} { @@ -94,12 +94,12 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg kp = xx.NewKeypair(pub, s.noisePrivateKey) } - log.Debug("xx handshake", "pubkey", kp.PubKey()) + log.Debug("xx handshake", "pubkey", kp.PubKey(), "initiator", s.initiator) // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() if err != nil { - return nil, fmt.Errorf("err getting raw pubkey: %s", err) + return fmt.Errorf("err getting raw pubkey: %s", err) } log.Debug("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) @@ -108,7 +108,8 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg noise_pub := kp.PubKey() signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) if err != nil { - return nil, fmt.Errorf("err signing payload: %s", err) + log.Error("xx handshake signing payload", "err", err) + return fmt.Errorf("err signing payload: %s", err) } s.local.noiseKey = noise_pub @@ -119,7 +120,8 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg payload.NoiseStaticKeySignature = signedPayload payloadEnc, err := proto.Marshal(payload) if err != nil { - return nil, fmt.Errorf("proto marshal payload fail: %s", err) + log.Error("xx handshake marshal payload", "err", err) + return fmt.Errorf("proto marshal payload fail: %s", err) } // new XX noise session @@ -131,68 +133,95 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg if !fallback { err = s.xx_sendHandshakeMessage(payloadEnc, true) if err != nil { - return nil, fmt.Errorf("stage 0 initiator fail: %s", err) + return fmt.Errorf("stage 0 initiator fail: %s", err) } + } else { + // initialize state as if we sent the first message + s.xx_ns, _ = xx.SendMessage(s.xx_ns, nil) } // stage 1 // - // read reply - buf, plaintext, valid, err := s.xx_recvHandshakeMessage(false) - if err != nil { - return buf, fmt.Errorf("initiator stage 1 fail: %s", err) - } + var plaintext []byte + var valid bool + if !fallback { + // read reply + _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) + if err != nil { + return fmt.Errorf("initiator stage 1 fail: %s", err) + } + + if !valid { + return fmt.Errorf("stage 1 initiator validation fail") + } + } else { + var msgbuf *xx.MessageBuffer + msgbuf, err = xx.Decode1(initialMsg) + + log.Debug("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(initialMsg)) + + if err != nil { + log.Debug("stage 1 xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) + return fmt.Errorf("decode msg fail: %s", err) + } + + s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) + if !valid { + log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") + return fmt.Errorf("validation fail") + } + + //log.Debug("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) - if !valid { - return buf, fmt.Errorf("stage 1 initiator validation fail") } log.Debug("stage 1 initiator", "payload", plaintext) + log.Debug("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) + + // stage 2 // + + if !fallback { + err = s.xx_sendHandshakeMessage(s.local.noiseKey[:], false) + if err != nil { + return fmt.Errorf("stage 2 intiator fail: %s", err) + } + } else { + err = s.xx_sendHandshakeMessage(payloadEnc, false) + if err != nil { + return fmt.Errorf("stage 2 intiator fail: %s", err) + } + } + // unmarshal payload nhp := new(pb.NoiseHandshakePayload) err = proto.Unmarshal(plaintext, nhp) if err != nil { - return buf, fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") + return fmt.Errorf("stage 2 initiator validation fail: cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { - log.Error("stage 1 initiator set remote peer info", "err", err) - return buf, fmt.Errorf("stage 1 initiator read remote libp2p key fail") + log.Error("stage 2 initiator set remote peer info", "err", err) + return fmt.Errorf("stage 2 initiator read remote libp2p key fail") } // assert that remote peer ID matches libp2p public key pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) if pid != s.remotePeer { - log.Error("stage 1 initiator check remote peer id err", "expected", s.remotePeer, "got", pid) + log.Error("stage 2 initiator check remote peer id err", "expected", s.remotePeer, "got", pid) } else if err != nil { - log.Error("stage 1 initiator check remote peer id", "err", err) + log.Error("stage 2 initiator check remote peer id", "err", err) } // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.xx_ns.RemoteKey()) if err != nil { - log.Error("stage 1 initiator verify payload", "err", err) + log.Error("stage 2 initiator verify payload", "err", err) } s.noiseStaticKeyCache[s.remotePeer] = s.xx_ns.RemoteKey() - log.Debug("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) - - // stage 2 // - - if !fallback { - err = s.xx_sendHandshakeMessage(s.local.noiseKey[:], false) - if err != nil { - return buf, fmt.Errorf("stage 2 intiator fail: %s", err) - } - } else { - err = s.xx_sendHandshakeMessage(payloadEnc, false) - if err != nil { - return buf, fmt.Errorf("stage 2 intiator fail: %s", err) - } - } } else { @@ -206,24 +235,24 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg // read message buf, plaintext, valid, err = s.xx_recvHandshakeMessage(true) if err != nil { - return buf, fmt.Errorf("stage 0 responder fail: %s", err) + return fmt.Errorf("stage 0 responder fail: %s", err) } if !valid { - return buf, fmt.Errorf("stage 0 responder validation fail") + return fmt.Errorf("stage 0 responder validation fail") } // unmarshal payload err = proto.Unmarshal(plaintext, nhp) if err != nil { - return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + return fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { log.Error("stage 0 responder set remote peer info", "err", err) - return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") + return fmt.Errorf("stage 0 responder read remote libp2p key fail") } // assert that remote peer ID matches libp2p key @@ -234,22 +263,23 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg } else { var msgbuf *xx.MessageBuffer - msgbuf, err = xx.Decode0(msg) + msgbuf, err = xx.Decode1(initialMsg) + xx_msgbuf := xx.NewMessageBuffer(msgbuf.NE(), nil, nil) - log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msg, "buf len", len(buf)) + log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", initialMsg, "buf len", len(buf)) if err != nil { log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) - return buf, fmt.Errorf("decode msg fail: %s", err) + return fmt.Errorf("decode msg fail: %s", err) } - s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) + s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, &xx_msgbuf) if !valid { log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") - return buf, fmt.Errorf("validation fail") + return fmt.Errorf("validation fail") } - log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + //log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) } log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) @@ -258,7 +288,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg err = s.xx_sendHandshakeMessage(payloadEnc, false) if err != nil { - return buf, fmt.Errorf("stage 1 responder fail: %s", err) + return fmt.Errorf("stage 1 responder fail: %s", err) } // stage 2 // @@ -266,11 +296,11 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg // read message buf, plaintext, valid, err = s.xx_recvHandshakeMessage(false) if err != nil { - return buf, fmt.Errorf("stage 2 responder fail: %s", err) + return fmt.Errorf("stage 2 responder fail: %s", err) } if !valid { - return buf, fmt.Errorf("stage 2 responder validation fail") + return fmt.Errorf("stage 2 responder validation fail") } log.Debug("stage 2 responder", "plaintext", plaintext, "remote key", s.xx_ns.RemoteKey()) @@ -282,20 +312,20 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg // unmarshal payload err = proto.Unmarshal(plaintext, nhp) if err != nil { - return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + return fmt.Errorf("stage 2 responder validation fail: cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { - log.Error("stage 0 responder set remote peer info", "err", err) - return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") + log.Error("stage 2 responder set remote peer info", "err", err) + return fmt.Errorf("stage 2 responder read remote libp2p key fail") } // assert that remote peer ID matches libp2p key err = s.setRemotePeerID(s.RemotePublicKey()) if err != nil { - log.Error("stage 0 responder set remote peer id", "err", err) + log.Error("stage 2 responder set remote peer id", "err", err) } } @@ -304,12 +334,13 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, msg err = s.verifyPayload(nhp, s.remote.noiseKey) if err != nil { log.Error("stage 2 responder verify payload", "err", err) - return buf, fmt.Errorf("stage 2 responder fail: %s", err) + return fmt.Errorf("stage 2 responder fail: %s", err) } s.noiseStaticKeyCache[s.remotePeer] = s.remote.noiseKey log.Debug("stage 2 responder", "remote key", s.remote.noiseKey) } - return nil, nil + log.Debug("xx_handshake done", "initiator", s.initiator) + return nil } From f7e9a08e1582145583fd13927e6e4aa963f1cfc3 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 14:11:26 -0400 Subject: [PATCH 1455/3965] add IK test --- p2p/security/noise/ik_handshake.go | 26 ++++++++-------- p2p/security/noise/transport.go | 3 ++ p2p/security/noise/transport_test.go | 46 ++++++++++++++++++++++++++-- p2p/security/noise/xx_handshake.go | 2 ++ 4 files changed, 62 insertions(+), 15 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index f37f836623..2b4fe4efe8 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -17,11 +17,11 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo var msgbuf ik.MessageBuffer s.ik_ns, msgbuf = ik.SendMessage(s.ik_ns, payload) var encMsgBuf []byte - //if initial_stage { - encMsgBuf = msgbuf.Encode0() - // } else { - // encMsgBuf = msgbuf.Encode1() - // } + if initial_stage { + encMsgBuf = msgbuf.Encode0() + } else { + encMsgBuf = msgbuf.Encode1() + } log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) @@ -56,11 +56,11 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, } var msgbuf *ik.MessageBuffer - //if initial_stage { - msgbuf, err = ik.Decode0(buf) - // } else { - // msgbuf, err = ik.Decode1(buf) - // } + if initial_stage { + msgbuf, err = ik.Decode0(buf) + } else { + msgbuf, err = ik.Decode1(buf) + } log.Debug("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) @@ -93,7 +93,9 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { kp = ik.NewKeypair(pub, s.noisePrivateKey) } - log.Debug("ik handshake", "pubkey", kp.PubKey()) + s.local.noiseKey = kp.PubKey() + + log.Debug("ik handshake", "noise pubkey", kp.PubKey()) // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() @@ -101,7 +103,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { return nil, fmt.Errorf("err getting raw pubkey: %s", err) } - log.Debug("ik handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) + log.Debug("ik handshake", "local libp2p key", localKeyRaw, "len", len(localKeyRaw)) // sign noise data for payload noise_pub := kp.PubKey() diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index ccc00b0e50..6d0e216b6c 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -19,6 +19,7 @@ type Transport struct { NoisePipesSupport bool NoiseStaticKeyCache map[peer.ID]([32]byte) NoisePrivateKey [32]byte + NoisePublicKey [32]byte } // SecureInbound runs noise handshake as the responder @@ -30,6 +31,7 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.S t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() t.NoisePrivateKey = s.NoisePrivateKey() + t.NoisePublicKey = s.local.noiseKey return s, nil } @@ -42,5 +44,6 @@ func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p pee t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() t.NoisePrivateKey = s.NoisePrivateKey() + t.NoisePublicKey = s.local.noiseKey return s, nil } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 7478b2eeed..b185ed978c 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -6,6 +6,7 @@ import ( "net" "testing" + //ik "github.com/ChainSafe/go-libp2p-noise/ik" crypto "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" @@ -173,15 +174,26 @@ func TestHandshakeXX(t *testing.T) { } } -// Test noise pipes +// Test IK handshake func TestHandshakeIK(t *testing.T) { - initTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) - respTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + // do initial XX handshake initConn, respConn := connect(t, initTransport, respTransport) initConn.Close() respConn.Close() + // turn on pipes, this will turn on IK + initTransport.NoisePipesSupport = true + respTransport.NoisePipesSupport = true + + // add responder's static key to initiator's key cache + keycache := make(map[peer.ID]([32]byte)) + keycache[respTransport.LocalID] = respTransport.NoisePublicKey + initTransport.NoiseStaticKeyCache = keycache + + // do IK handshake initConn, respConn = connect(t, initTransport, respTransport) defer initConn.Close() defer respConn.Close() @@ -202,3 +214,31 @@ func TestHandshakeIK(t *testing.T) { t.Errorf("Message mismatch. %v != %v", before, after) } } + +// Test noise pipes +func TestHandshakeXXfallback(t *testing.T) { + initTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) + respTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) + + // turning on pipes causes it to default to IK, but since we haven't already + // done a handshake, it'll fallback to XX + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + before := []byte("hello world") + _, err := initConn.Write(before) + if err != nil { + t.Fatal(err) + } + + after := make([]byte, len(before)) + _, err = respConn.Read(after) + if err != nil { + t.Fatal(err) + } + + if !bytes.Equal(before, after) { + t.Errorf("Message mismatch. %v != %v", before, after) + } +} diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 6e1eeb3448..8facc70f51 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -94,6 +94,8 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init kp = xx.NewKeypair(pub, s.noisePrivateKey) } + s.local.noiseKey = kp.PubKey() + log.Debug("xx handshake", "pubkey", kp.PubKey(), "initiator", s.initiator) // setup libp2p keys From 2de16d09a8c7fa27a2d98cfb1928e485ae1d75cf Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 15:01:05 -0400 Subject: [PATCH 1456/3965] begin work on moving ephemeral key from ik->xx --- p2p/security/noise/ik/IK.noise.go | 5 +++++ p2p/security/noise/ik_handshake.go | 1 + p2p/security/noise/transport_test.go | 2 +- p2p/security/noise/xx/XX.noise.go | 18 ++++++++++++++---- p2p/security/noise/xx_handshake.go | 13 ++++++++++--- 5 files changed, 31 insertions(+), 8 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index 01fa38e33c..750d37d877 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -80,6 +80,10 @@ func (ns *NoiseSession) CS2() *cipherstate { return &ns.cs2 } +func (ns *NoiseSession) Ephemeral() *Keypair { + return &ns.hs.e +} + func NewKeypair(pub [32]byte, priv [32]byte) Keypair { return Keypair{ public_key: pub, @@ -87,6 +91,7 @@ func NewKeypair(pub [32]byte, priv [32]byte) Keypair { } } + func (kp Keypair) PubKey() [32]byte { return kp.public_key } diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 2b4fe4efe8..c66bda1e8e 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -88,6 +88,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { // generate local static noise key kp = ik.GenerateKeypair() s.noisePrivateKey = kp.PrivKey() + log.Debug("ik generate new noise kp", "initiator", s.initiator) } else { pub := ik.GeneratePublicKey(s.noisePrivateKey) kp = ik.NewKeypair(pub, s.noisePrivateKey) diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index b185ed978c..e6fa9be48b 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -149,7 +149,7 @@ func TestKeys(t *testing.T) { } // Tests XX handshake -func TestHandshakeXX(t *testing.T) { +func TestHandshake_XX(t *testing.T) { initTransport := newTestTransport(t, crypto.Ed25519, 2048) respTransport := newTestTransport(t, crypto.Ed25519, 2048) diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index fc35d282d7..5e807bb757 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -81,6 +81,10 @@ func (ns *NoiseSession) CS2() *cipherstate { return &ns.cs2 } +func (ns *NoiseSession) Ephemeral() *Keypair { + return &ns.hs.e +} + func NewKeypair(pub [32]byte, priv [32]byte) Keypair { return Keypair{ public_key: pub, @@ -434,9 +438,15 @@ func initializeResponder(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) return handshakestate{ss, s, e, rs, re, psk} } -func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { +func writeMessageA(hs *handshakestate, payload []byte, e *Keypair) (*handshakestate, MessageBuffer) { ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - hs.e = GenerateKeypair() + + if e == nil { + hs.e = GenerateKeypair() + } else { + hs.e = *e + } + ne = hs.e.public_key mixHash(&hs.ss, ne[:]) /* No PSK, so skipping mixKey */ @@ -530,10 +540,10 @@ func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *Noise return &session } -func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageBuffer) { +func SendMessage(session *NoiseSession, message []byte, ephemeral *Keypair) (*NoiseSession, MessageBuffer) { var messageBuffer MessageBuffer if session.mc == 0 { - _, messageBuffer = writeMessageA(&session.hs, message) + _, messageBuffer = writeMessageA(&session.hs, message, ephemeral) } if session.mc == 1 { _, messageBuffer = writeMessageB(&session.hs, message) diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 8facc70f51..e64331ef5d 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -16,7 +16,7 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo // create send message w payload var msgbuf xx.MessageBuffer - s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, payload) + s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, payload, nil) var encMsgBuf []byte if initial_stage { encMsgBuf = msgbuf.Encode0() @@ -89,6 +89,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init // generate local static noise key kp = xx.GenerateKeypair() s.noisePrivateKey = kp.PrivKey() + log.Debug("xx generate new noise kp", "initiator", s.initiator) } else { pub := xx.GeneratePublicKey(s.noisePrivateKey) kp = xx.NewKeypair(pub, s.noisePrivateKey) @@ -138,8 +139,14 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init return fmt.Errorf("stage 0 initiator fail: %s", err) } } else { + e_ik := s.ik_ns.Ephemeral() + e_xx := xx.NewKeypair(e_ik.PubKey(), e_ik.PrivKey()) + // initialize state as if we sent the first message - s.xx_ns, _ = xx.SendMessage(s.xx_ns, nil) + var msgbuf xx.MessageBuffer + s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, nil, &e_xx) + + log.Debug("stage 0 initiator xx", "msgbuf", msgbuf) } // stage 1 // @@ -268,7 +275,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init msgbuf, err = xx.Decode1(initialMsg) xx_msgbuf := xx.NewMessageBuffer(msgbuf.NE(), nil, nil) - log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", initialMsg, "buf len", len(buf)) + log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "modified_msgbuf", xx_msgbuf, "buf len", len(buf)) if err != nil { log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) From 7c6510e7ce244b82aeef3e326ab5a93bc7437bdf Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 15:11:35 -0400 Subject: [PATCH 1457/3965] ephemeral keys passed from ik->xxfallback --- p2p/security/noise/ik/IK.noise.go | 4 ++-- p2p/security/noise/xx_handshake.go | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index 750d37d877..4090c09537 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -524,7 +524,7 @@ func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageB } if session.mc == 1 { session.h, messageBuffer, session.cs1, session.cs2 = writeMessageB(&session.hs, message) - session.hs = handshakestate{} + //session.hs = handshakestate{} } session.mc = session.mc + 1 return session, messageBuffer @@ -538,7 +538,7 @@ func RecvMessage(session *NoiseSession, message *MessageBuffer) (*NoiseSession, } if session.mc == 1 { session.h, plaintext, valid, session.cs1, session.cs2 = readMessageB(&session.hs, message) - session.hs = handshakestate{} + //session.hs = handshakestate{} } session.mc = session.mc + 1 return session, plaintext, valid diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index e64331ef5d..d01d2156fb 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -140,6 +140,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init } } else { e_ik := s.ik_ns.Ephemeral() + log.Debug("xxfallback stage 0 initiator", "ephemeral keys from ik", e_ik ) e_xx := xx.NewKeypair(e_ik.PubKey(), e_ik.PrivKey()) // initialize state as if we sent the first message @@ -317,7 +318,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init copy(s.remote.noiseKey[:], plaintext) if fallback { - // unmarshal payload err = proto.Unmarshal(plaintext, nhp) if err != nil { @@ -335,8 +335,8 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init err = s.setRemotePeerID(s.RemotePublicKey()) if err != nil { log.Error("stage 2 responder set remote peer id", "err", err) + return fmt.Errorf("stage 2 responder: remote libp2p key does not match remote peer.ID") } - } // verify payload is signed by libp2p key From f3c6167915cc27db651ea661b7e8287278fb010e Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 15:17:58 -0400 Subject: [PATCH 1458/3965] ik->kkfallback complete --- p2p/security/noise/ik/IK.noise.go | 1 - p2p/security/noise/protocol.go | 28 +++++++++++++++------------- p2p/security/noise/xx/XX.noise.go | 2 +- p2p/security/noise/xx/XX_test.go | 6 +++--- p2p/security/noise/xx_handshake.go | 4 +++- 5 files changed, 22 insertions(+), 19 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index 4090c09537..a36ac09e42 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -91,7 +91,6 @@ func NewKeypair(pub [32]byte, priv [32]byte) Keypair { } } - func (kp Keypair) PubKey() [32]byte { return kp.public_key } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 0b3aeeffec..a88c42dd23 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -84,6 +84,19 @@ func (s *secureSession) NoisePrivateKey() [32]byte { return s.noisePrivateKey } +func (s *secureSession) ReadLength() (int, error) { + buf := make([]byte, 2) + _, err := s.insecure.Read(buf) + return int(binary.BigEndian.Uint16(buf)), err +} + +func (s *secureSession) WriteLength(length int) error { + buf := make([]byte, 2) + binary.BigEndian.PutUint16(buf, uint16(length)) + _, err := s.insecure.Write(buf) + return err +} + func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { s.remote.libp2pKey, err = crypto.UnmarshalPublicKey(key) return err @@ -98,6 +111,8 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe sig := payload.GetNoiseStaticKeySignature() msg := append([]byte(payload_string), noiseKey[:]...) + log.Debug("verifyPayload", "msg", msg) + ok, err := s.RemotePublicKey().Verify(msg, sig) if err != nil { return err @@ -108,19 +123,6 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe return nil } -func (s *secureSession) ReadLength() (int, error) { - buf := make([]byte, 2) - _, err := s.insecure.Read(buf) - return int(binary.BigEndian.Uint16(buf)), err -} - -func (s *secureSession) WriteLength(length int) error { - buf := make([]byte, 2) - binary.BigEndian.PutUint16(buf, uint16(length)) - _, err := s.insecure.Write(buf) - return err -} - func (s *secureSession) runHandshake(ctx context.Context) error { log.Debug("runHandshake", "cache", s.noiseStaticKeyCache) diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 5e807bb757..5fa6dac983 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -442,7 +442,7 @@ func writeMessageA(hs *handshakestate, payload []byte, e *Keypair) (*handshakest ne, ns, ciphertext := emptyKey, []byte{}, []byte{} if e == nil { - hs.e = GenerateKeypair() + hs.e = GenerateKeypair() } else { hs.e = *e } diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 4a6065d464..3f25e6853e 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -86,7 +86,7 @@ func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { var msgbuf MessageBuffer msg := []byte{} msg = append(msg, payload_init_enc[:]...) - ns_init, msgbuf = SendMessage(ns_init, msg) + ns_init, msgbuf = SendMessage(ns_init, msg, nil) t.Logf("stage 0 msgbuf: %v", msgbuf) t.Logf("stage 0 msgbuf ne len: %d", len(msgbuf.NE())) @@ -111,7 +111,7 @@ func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { t.Fatalf("proto marshal payload fail: %s", err) } msg = append(msg, payload_resp_enc[:]...) - ns_resp, msgbuf = SendMessage(ns_resp, msg) + ns_resp, msgbuf = SendMessage(ns_resp, msg, nil) t.Logf("stage 1 msgbuf: %v", msgbuf) t.Logf("stage 1 msgbuf ne len: %d", len(msgbuf.NE())) @@ -128,7 +128,7 @@ func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { // stage 2: initiator // send message //msg = append(msg, payload_init_enc[:]...) - ns_init, msgbuf = SendMessage(ns_init, nil) + ns_init, msgbuf = SendMessage(ns_init, nil, nil) t.Logf("stage 2 msgbuf: %v", msgbuf) t.Logf("stage 2 msgbuf ne len: %d", len(msgbuf.NE())) diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index d01d2156fb..8eab4abf24 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -140,7 +140,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init } } else { e_ik := s.ik_ns.Ephemeral() - log.Debug("xxfallback stage 0 initiator", "ephemeral keys from ik", e_ik ) + log.Debug("xxfallback stage 0 initiator", "ephemeral keys from ik", e_ik) e_xx := xx.NewKeypair(e_ik.PubKey(), e_ik.PrivKey()) // initialize state as if we sent the first message @@ -339,6 +339,8 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init } } + s.remote.noiseKey = s.xx_ns.RemoteKey() + // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.remote.noiseKey) if err != nil { From e40f6b283eaf835d5c6a44fe1c87c45db2bd54ef Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 15:31:57 -0400 Subject: [PATCH 1459/3965] use ipfs/go-log --- p2p/security/noise/c.out | 228 +++++++++++++++++++++++++++++ p2p/security/noise/crypto_test.go | 24 ++- p2p/security/noise/ik_handshake.go | 25 ++-- p2p/security/noise/protocol.go | 10 +- p2p/security/noise/xx_handshake.go | 48 +++--- 5 files changed, 293 insertions(+), 42 deletions(-) create mode 100644 p2p/security/noise/c.out diff --git a/p2p/security/noise/c.out b/p2p/security/noise/c.out new file mode 100644 index 0000000000..bf84a4ac91 --- /dev/null +++ b/p2p/security/noise/c.out @@ -0,0 +1,228 @@ +mode: set +github.com/ChainSafe/go-libp2p-noise/crypto.go:9.82,10.19 1 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:30.2,30.24 1 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:10.19,11.18 1 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:11.18,14.4 2 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:14.9,17.4 2 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:18.8,18.26 1 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:18.26,19.18 1 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:19.18,22.4 2 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:22.9,25.4 2 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:26.8,28.3 1 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:33.82,35.19 2 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:55.2,55.9 1 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:59.2,59.23 1 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:35.19,36.18 1 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:36.18,39.4 2 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:39.9,42.4 2 1 +github.com/ChainSafe/go-libp2p-noise/crypto.go:43.8,43.26 1 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:43.26,44.18 1 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:44.18,47.4 2 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:47.9,50.4 2 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:51.8,53.3 1 0 +github.com/ChainSafe/go-libp2p-noise/crypto.go:55.9,57.3 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:13.91,20.19 5 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:26.2,30.16 4 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:36.2,37.16 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:42.2,42.12 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:20.19,22.3 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:22.8,24.3 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:30.16,33.3 2 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:37.16,40.3 2 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:45.123,47.16 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:51.2,54.16 3 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:58.2,59.19 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:65.2,67.16 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:72.2,73.12 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:78.2,80.35 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:47.16,49.3 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:54.16,56.3 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:59.19,61.3 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:61.8,63.3 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:67.16,70.3 2 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:73.12,76.3 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:84.78,87.37 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:97.2,103.16 4 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:107.2,112.16 4 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:117.2,121.16 5 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:126.2,129.17 3 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:230.2,231.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:87.37,92.3 3 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:92.8,95.3 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:103.16,105.3 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:112.16,114.3 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:121.16,123.3 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:129.17,132.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:140.3,141.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:145.3,145.13 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:150.3,152.17 3 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:157.3,158.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:164.3,165.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:170.3,171.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:132.17,135.4 2 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:141.17,143.4 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:145.13,147.4 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:152.17,154.4 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:158.17,161.4 2 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:165.17,167.4 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:171.17,173.4 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:175.8,184.17 5 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:188.3,188.13 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:192.3,197.17 4 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:202.3,203.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:209.3,210.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:215.3,216.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:222.3,223.17 2 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:184.17,186.4 1 1 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:188.13,190.4 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:197.17,199.4 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:203.17,206.4 2 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:210.17,212.4 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:216.17,218.4 1 0 +github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:223.17,226.4 2 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:58.66,60.32 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:64.2,78.15 3 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:60.32,62.3 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:81.70,83.2 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:85.52,87.2 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:89.51,93.2 3 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:95.55,100.2 4 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:102.67,105.2 2 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:107.72,110.2 2 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:112.105,119.16 5 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:125.2,125.12 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:119.16,121.3 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:121.8,121.16 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:121.16,123.3 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:128.65,133.78 2 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:168.2,168.12 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:133.78,139.17 3 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:153.3,153.23 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:139.17,145.18 3 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:150.4,150.24 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:145.18,148.5 2 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:155.8,160.17 2 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:165.3,165.23 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:160.17,163.4 2 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:171.46,173.2 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:175.45,177.2 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:179.58,181.2 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:183.56,185.2 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:187.55,192.16 3 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:196.2,197.22 2 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:192.16,194.3 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:200.54,202.16 2 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:206.2,208.16 3 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:212.2,212.30 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:202.16,204.3 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:208.16,210.3 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:215.47,217.2 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:219.46,221.2 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:223.57,225.2 1 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:227.56,229.2 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:231.60,233.2 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:235.61,237.2 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:239.55,245.2 3 1 +github.com/ChainSafe/go-libp2p-noise/protocol.go:247.54,249.16 2 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:253.2,254.16 2 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:258.2,259.12 2 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:249.16,251.3 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:254.16,256.3 1 0 +github.com/ChainSafe/go-libp2p-noise/protocol.go:262.39,264.2 1 1 +github.com/ChainSafe/go-libp2p-noise/transport.go:26.99,28.16 2 1 +github.com/ChainSafe/go-libp2p-noise/transport.go:32.2,35.15 4 1 +github.com/ChainSafe/go-libp2p-noise/transport.go:28.16,30.3 1 0 +github.com/ChainSafe/go-libp2p-noise/transport.go:39.111,41.16 2 1 +github.com/ChainSafe/go-libp2p-noise/transport.go:45.2,48.15 4 1 +github.com/ChainSafe/go-libp2p-noise/transport.go:41.16,43.3 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:14.91,21.19 5 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:27.2,31.16 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:37.2,38.16 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:43.2,43.12 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:21.19,23.3 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:23.8,25.3 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:31.16,34.3 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:38.16,41.3 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:46.123,48.16 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:52.2,55.16 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:59.2,60.19 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:68.2,68.16 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:73.2,74.12 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:79.2,81.35 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:48.16,50.3 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:55.16,57.3 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:60.19,62.3 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:62.8,64.3 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:68.16,71.3 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:74.12,77.3 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:85.108,88.37 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:98.2,104.16 4 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:108.2,113.16 4 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:118.2,125.16 6 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:131.2,133.17 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:355.2,356.12 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:88.37,93.3 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:93.8,96.3 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:104.16,106.3 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:113.16,116.3 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:125.16,128.3 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:133.17,136.16 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:155.3,157.16 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:188.3,194.16 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:207.3,209.17 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:214.3,215.17 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:221.3,222.26 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:229.3,230.17 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:234.3,234.60 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:136.16,138.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:138.18,140.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:141.9,151.4 6 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:157.16,160.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:164.4,164.14 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:160.18,162.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:164.14,166.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:167.9,173.18 4 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:178.4,179.14 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:173.18,176.5 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:179.14,182.5 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:194.16,196.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:196.18,198.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:199.9,201.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:201.18,203.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:209.17,211.4 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:215.17,218.4 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:222.26,224.4 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:224.9,224.24 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:224.24,226.4 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:230.17,232.4 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:236.8,244.16 4 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:295.3,300.17 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:307.3,308.17 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:312.3,312.13 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:316.3,320.15 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:342.3,346.17 3 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:351.3,352.66 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:244.16,247.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:251.4,251.14 1 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:256.4,257.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:262.4,263.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:269.4,270.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:247.18,249.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:251.14,253.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:257.18,259.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:263.18,266.5 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:270.18,272.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:274.9,281.18 5 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:286.4,287.14 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:281.18,284.5 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:287.14,290.5 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:300.17,302.4 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:308.17,310.4 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:312.13,314.4 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:320.15,323.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:328.4,329.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:335.4,336.18 2 1 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:323.18,325.5 1 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:329.18,332.5 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:336.18,339.5 2 0 +github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:346.17,349.4 2 0 diff --git a/p2p/security/noise/crypto_test.go b/p2p/security/noise/crypto_test.go index 807edb8434..09fae902ae 100644 --- a/p2p/security/noise/crypto_test.go +++ b/p2p/security/noise/crypto_test.go @@ -7,7 +7,7 @@ import ( "github.com/libp2p/go-libp2p-core/crypto" ) -func TestEncryptAndDecrypt(t *testing.T) { +func TestEncryptAndDecrypt_InitToResp(t *testing.T) { initTransport := newTestTransport(t, crypto.Ed25519, 2048) respTransport := newTestTransport(t, crypto.Ed25519, 2048) @@ -28,3 +28,25 @@ func TestEncryptAndDecrypt(t *testing.T) { t.Fatal(err) } } + +func TestEncryptAndDecrypt_RespToInit(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + plaintext := []byte("helloworld") + ciphertext, err := respConn.Encrypt(plaintext) + if err != nil { + t.Fatal(err) + } + + result, err := initConn.Decrypt(ciphertext) + if !bytes.Equal(plaintext, result) { + t.Fatalf("got %x expected %x", result, plaintext) + } else if err != nil { + t.Fatal(err) + } +} diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index c66bda1e8e..b1fc7eec20 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -3,7 +3,6 @@ package noise import ( "context" "fmt" - log "github.com/ChainSafe/log15" proto "github.com/gogo/protobuf/proto" ik "github.com/ChainSafe/go-libp2p-noise/ik" @@ -11,7 +10,7 @@ import ( ) func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bool) error { - log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload)) + log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload)) // create send message w payload var msgbuf ik.MessageBuffer @@ -23,8 +22,8 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo encMsgBuf = msgbuf.Encode1() } - log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) - log.Debug("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) + log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) + log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) err := s.WriteLength(len(encMsgBuf)) if err != nil { @@ -62,7 +61,7 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, msgbuf, err = ik.Decode1(buf) } - log.Debug("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) + log.Debugf("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) if err != nil { log.Error("ik_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) @@ -75,7 +74,7 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, return buf, nil, false, fmt.Errorf("validation fail") } - log.Debug("recv handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + log.Debugf("recv handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) return buf, plaintext, valid, nil } @@ -88,7 +87,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { // generate local static noise key kp = ik.GenerateKeypair() s.noisePrivateKey = kp.PrivKey() - log.Debug("ik generate new noise kp", "initiator", s.initiator) + log.Debugf("ik generate new noise kp", "initiator", s.initiator) } else { pub := ik.GeneratePublicKey(s.noisePrivateKey) kp = ik.NewKeypair(pub, s.noisePrivateKey) @@ -96,7 +95,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { s.local.noiseKey = kp.PubKey() - log.Debug("ik handshake", "noise pubkey", kp.PubKey()) + log.Debugf("ik handshake", "noise pubkey", kp.PubKey()) // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() @@ -104,7 +103,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { return nil, fmt.Errorf("err getting raw pubkey: %s", err) } - log.Debug("ik handshake", "local libp2p key", localKeyRaw, "len", len(localKeyRaw)) + log.Debugf("ik handshake", "local libp2p key", localKeyRaw, "len", len(localKeyRaw)) // sign noise data for payload noise_pub := kp.PubKey() @@ -124,7 +123,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { // new XX noise session s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, s.noiseStaticKeyCache[s.remotePeer]) - log.Debug("ik initiator init session", "remotePeer", s.noiseStaticKeyCache[s.remotePeer]) + log.Debugf("ik initiator init session", "remotePeer", s.noiseStaticKeyCache[s.remotePeer]) if s.initiator { // stage 0 // @@ -175,7 +174,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { } else { // stage 0 // - log.Debug("ik responder", "noiseKey", kp.PubKey()) + log.Debugf("ik responder", "noiseKey", kp.PubKey()) var buf, plaintext []byte var valid bool @@ -189,7 +188,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { return buf, fmt.Errorf("stage 0 responder validation fail") } - log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) + log.Debugf("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) // unmarshal payload nhp := new(pb.NoiseHandshakePayload) @@ -227,6 +226,6 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { } - log.Debug("ik_handshake done", "initiator", s.initiator) + log.Debugf("ik_handshake done", "initiator", s.initiator) return nil, nil } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index a88c42dd23..4504e7f705 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -8,7 +8,7 @@ import ( "net" "time" - log "github.com/ChainSafe/log15" + logging "github.com/ipfs/go-log" //proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" @@ -19,6 +19,8 @@ import ( xx "github.com/ChainSafe/go-libp2p-noise/xx" ) +var log = logging.Logger("noise") + const payload_string = "noise-libp2p-static-key:" type secureSession struct { @@ -111,7 +113,7 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe sig := payload.GetNoiseStaticKeySignature() msg := append([]byte(payload_string), noiseKey[:]...) - log.Debug("verifyPayload", "msg", msg) + log.Debugf("verifyPayload", "msg", fmt.Sprintf("%x", msg)) ok, err := s.RemotePublicKey().Verify(msg, sig) if err != nil { @@ -125,11 +127,11 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe func (s *secureSession) runHandshake(ctx context.Context) error { - log.Debug("runHandshake", "cache", s.noiseStaticKeyCache) + log.Debugf("runHandshake", "cache", s.noiseStaticKeyCache) // if we have the peer's noise static key and we support noise pipes, we can try IK if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} || s.noisePipesSupport { - log.Debug("runHandshake_ik") + log.Debugf("runHandshake_ik") // known static key for peer, try IK // diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 8eab4abf24..cd4f76ab81 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -3,7 +3,7 @@ package noise import ( "context" "fmt" - log "github.com/ChainSafe/log15" + //log "github.com/ChainSafe/log15" proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/peer" @@ -12,7 +12,7 @@ import ( ) func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { - log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload), "initial_stage", initial_stage) + log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload), "initial_stage", initial_stage) // create send message w payload var msgbuf xx.MessageBuffer @@ -24,8 +24,8 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo encMsgBuf = msgbuf.Encode1() } - log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - //log.Debug("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) + log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) + //log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) err := s.WriteLength(len(encMsgBuf)) if err != nil { @@ -63,10 +63,10 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, msgbuf, err = xx.Decode1(buf) } - //log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) + //log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) if err != nil { - log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) + log.Debugf("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) } @@ -76,7 +76,7 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, return buf, nil, false, fmt.Errorf("validation fail") } - log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) return buf, plaintext, valid, nil } @@ -89,7 +89,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init // generate local static noise key kp = xx.GenerateKeypair() s.noisePrivateKey = kp.PrivKey() - log.Debug("xx generate new noise kp", "initiator", s.initiator) + log.Debugf("xx generate new noise kp", "initiator", s.initiator) } else { pub := xx.GeneratePublicKey(s.noisePrivateKey) kp = xx.NewKeypair(pub, s.noisePrivateKey) @@ -97,7 +97,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init s.local.noiseKey = kp.PubKey() - log.Debug("xx handshake", "pubkey", kp.PubKey(), "initiator", s.initiator) + log.Debugf("xx handshake", "pubkey", kp.PubKey(), "initiator", s.initiator) // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() @@ -105,7 +105,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init return fmt.Errorf("err getting raw pubkey: %s", err) } - log.Debug("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) + log.Debugf("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) // sign noise data for payload noise_pub := kp.PubKey() @@ -140,14 +140,14 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init } } else { e_ik := s.ik_ns.Ephemeral() - log.Debug("xxfallback stage 0 initiator", "ephemeral keys from ik", e_ik) + log.Debugf("xxfallback stage 0 initiator", "ephemeral keys from ik", e_ik) e_xx := xx.NewKeypair(e_ik.PubKey(), e_ik.PrivKey()) // initialize state as if we sent the first message var msgbuf xx.MessageBuffer s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, nil, &e_xx) - log.Debug("stage 0 initiator xx", "msgbuf", msgbuf) + log.Debugf("stage 0 initiator xx", "msgbuf", msgbuf) } // stage 1 // @@ -168,10 +168,10 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init var msgbuf *xx.MessageBuffer msgbuf, err = xx.Decode1(initialMsg) - log.Debug("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(initialMsg)) + log.Debugf("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(initialMsg)) if err != nil { - log.Debug("stage 1 xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) + log.Debugf("stage 1 xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) return fmt.Errorf("decode msg fail: %s", err) } @@ -181,13 +181,13 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init return fmt.Errorf("validation fail") } - //log.Debug("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + //log.Debugf("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) } - log.Debug("stage 1 initiator", "payload", plaintext) + log.Debugf("stage 1 initiator", "payload", plaintext) - log.Debug("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) + log.Debugf("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) // stage 2 // @@ -276,10 +276,10 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init msgbuf, err = xx.Decode1(initialMsg) xx_msgbuf := xx.NewMessageBuffer(msgbuf.NE(), nil, nil) - log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "modified_msgbuf", xx_msgbuf, "buf len", len(buf)) + log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "modified_msgbuf", xx_msgbuf, "buf len", len(buf)) if err != nil { - log.Debug("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) + log.Debugf("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) return fmt.Errorf("decode msg fail: %s", err) } @@ -289,10 +289,10 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init return fmt.Errorf("validation fail") } - //log.Debug("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + //log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) } - log.Debug("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) + log.Debugf("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) // stage 1 // @@ -313,7 +313,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init return fmt.Errorf("stage 2 responder validation fail") } - log.Debug("stage 2 responder", "plaintext", plaintext, "remote key", s.xx_ns.RemoteKey()) + log.Debugf("stage 2 responder", "plaintext", plaintext, "remote key", s.xx_ns.RemoteKey()) copy(s.remote.noiseKey[:], plaintext) @@ -349,9 +349,9 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init } s.noiseStaticKeyCache[s.remotePeer] = s.remote.noiseKey - log.Debug("stage 2 responder", "remote key", s.remote.noiseKey) + log.Debugf("stage 2 responder", "remote key", s.remote.noiseKey) } - log.Debug("xx_handshake done", "initiator", s.initiator) + log.Debugf("xx_handshake done", "initiator", s.initiator) return nil } From c01c8520454bb8497b34e82507779aa8d6834853 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 15:43:27 -0400 Subject: [PATCH 1460/3965] cleanup code --- p2p/security/noise/crypto.go | 16 ++++++++-------- p2p/security/noise/crypto_test.go | 13 +++++++++++++ p2p/security/noise/ik/IK.noise.go | 15 +-------------- p2p/security/noise/ik_handshake.go | 1 - p2p/security/noise/protocol.go | 12 +----------- p2p/security/noise/transport.go | 3 +++ p2p/security/noise/xx/XX.noise.go | 11 ----------- p2p/security/noise/xx_handshake.go | 17 +++-------------- 8 files changed, 29 insertions(+), 59 deletions(-) diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index 912fcca59e..e2f9fb858f 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -10,18 +10,18 @@ func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) if s.xx_complete { if s.initiator { cs := s.xx_ns.CS1() - cs, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) + _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) } else { cs := s.xx_ns.CS2() - cs, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) + _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) } } else if s.ik_complete { if s.initiator { cs := s.ik_ns.CS1() - cs, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) + _, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) } else { cs := s.ik_ns.CS2() - cs, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) + _, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) } } else { return nil, errors.New("encrypt err: haven't completed handshake") @@ -35,18 +35,18 @@ func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) if s.xx_complete { if s.initiator { cs := s.xx_ns.CS2() - cs, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) + _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) } else { cs := s.xx_ns.CS1() - cs, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) + _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) } } else if s.ik_complete { if s.initiator { cs := s.ik_ns.CS2() - cs, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) + _, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) } else { cs := s.ik_ns.CS1() - cs, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) + _, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) } } else { return nil, errors.New("decrypt err: haven't completed handshake") diff --git a/p2p/security/noise/crypto_test.go b/p2p/security/noise/crypto_test.go index 09fae902ae..dff7ab7c01 100644 --- a/p2p/security/noise/crypto_test.go +++ b/p2p/security/noise/crypto_test.go @@ -27,6 +27,19 @@ func TestEncryptAndDecrypt_InitToResp(t *testing.T) { } else if err != nil { t.Fatal(err) } + + plaintext = []byte("goodbye") + ciphertext, err = initConn.Encrypt(plaintext) + if err != nil { + t.Fatal(err) + } + + result, err = respConn.Decrypt(ciphertext) + if !bytes.Equal(plaintext, result) { + t.Fatalf("got %x expected %x", result, plaintext) + } else if err != nil { + t.Fatal(err) + } } func TestEncryptAndDecrypt_RespToInit(t *testing.T) { diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index a36ac09e42..a68d721784 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -170,9 +170,6 @@ func (mb *MessageBuffer) Encode1() []byte { enc = append(enc, mb.ne[:]...) enc = append(enc, mb.ciphertext...) - // log.Debug("XX_Encode1", "ne", mb.ne) - // log.Debug("XX_Encode1", "ns", mb.ns) - return enc } @@ -182,12 +179,10 @@ func Decode0(in []byte) (*MessageBuffer, error) { return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") } - //log.Debug("XX_Decode0", "in", in) mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) mb.ns = in[32:80] mb.ciphertext = in[80:] - //log.Debug("XX_Decode0", "mb", mb) return mb, nil } @@ -195,18 +190,12 @@ func Decode0(in []byte) (*MessageBuffer, error) { // Decodes messages at stage 1 into MessageBuffer func Decode1(in []byte) (*MessageBuffer, error) { if len(in) < 80 { - return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") + return nil, errors.New("cannot decode stage 1 MessageBuffer: length less than 96 bytes") } - // log.Debug("XX_Decode1", "in", in) - // log.Debug("XX_Decode1", "ns", in[32:80]) - mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) - //mb.ns = in[32:80] mb.ciphertext = in[32:] - // copy(mb.ns,) - // copy(mb.ciphertext,) return mb, nil } @@ -523,7 +512,6 @@ func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageB } if session.mc == 1 { session.h, messageBuffer, session.cs1, session.cs2 = writeMessageB(&session.hs, message) - //session.hs = handshakestate{} } session.mc = session.mc + 1 return session, messageBuffer @@ -537,7 +525,6 @@ func RecvMessage(session *NoiseSession, message *MessageBuffer) (*NoiseSession, } if session.mc == 1 { session.h, plaintext, valid, session.cs1, session.cs2 = readMessageB(&session.hs, message) - //session.hs = handshakestate{} } session.mc = session.mc + 1 return session, plaintext, valid diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index b1fc7eec20..d2eb9dbe47 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -23,7 +23,6 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo } log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) - log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) err := s.WriteLength(len(encMsgBuf)) if err != nil { diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 4504e7f705..de15ed25e8 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -4,15 +4,12 @@ import ( "context" "encoding/binary" "fmt" - //"io" "net" "time" logging "github.com/ipfs/go-log" - //proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" - //"github.com/libp2p/go-libp2p-core/sec" ik "github.com/ChainSafe/go-libp2p-noise/ik" pb "github.com/ChainSafe/go-libp2p-noise/pb" @@ -126,21 +123,15 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe } func (s *secureSession) runHandshake(ctx context.Context) error { - - log.Debugf("runHandshake", "cache", s.noiseStaticKeyCache) - // if we have the peer's noise static key and we support noise pipes, we can try IK if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} || s.noisePipesSupport { - log.Debugf("runHandshake_ik") - // known static key for peer, try IK // buf, err := s.runHandshake_ik(ctx) if err != nil { log.Error("runHandshake_ik", "err", err) - // TODO: PIPE TO XX - + // IK failed, pipe to XXfallback err = s.runHandshake_xx(ctx, true, buf) if err != nil { log.Error("runHandshake_xx", "err", err) @@ -153,7 +144,6 @@ func (s *secureSession) runHandshake(ctx context.Context) error { s.ik_complete = true } else { - // unknown static key for peer, try XX // err := s.runHandshake_xx(ctx, false, nil) diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 6d0e216b6c..c784fafe87 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -9,10 +9,13 @@ import ( "github.com/libp2p/go-libp2p-core/sec" ) +// ID is the protocol ID for noise const ID = "/noise/0.0.1" var _ sec.SecureTransport = &Transport{} +// Transport implements the interface sec.SecureTransport +// https://godoc.org/github.com/libp2p/go-libp2p-core/sec#SecureConn type Transport struct { LocalID peer.ID PrivateKey crypto.PrivKey diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 5fa6dac983..bfd743ee23 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -26,7 +26,6 @@ import ( "hash" "io" "math" - //log "github.com/ChainSafe/log15" ) /* ---------------------------------------------------------------- * @@ -171,9 +170,6 @@ func (mb *MessageBuffer) Encode1() []byte { enc = append(enc, mb.ns...) enc = append(enc, mb.ciphertext...) - // log.Debug("XX_Encode1", "ne", mb.ne) - // log.Debug("XX_Encode1", "ns", mb.ns) - return enc } @@ -183,11 +179,9 @@ func Decode0(in []byte) (*MessageBuffer, error) { return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") } - //log.Debug("XX_Decode0", "in", in) mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) mb.ciphertext = in[32:] - //log.Debug("XX_Decode0", "mb", mb) return mb, nil } @@ -198,15 +192,10 @@ func Decode1(in []byte) (*MessageBuffer, error) { return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") } - // log.Debug("XX_Decode1", "in", in) - // log.Debug("XX_Decode1", "ns", in[32:80]) - mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) mb.ns = in[32:80] mb.ciphertext = in[80:] - // copy(mb.ns,) - // copy(mb.ciphertext,) return mb, nil } diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index cd4f76ab81..391acd62e9 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -3,7 +3,7 @@ package noise import ( "context" "fmt" - //log "github.com/ChainSafe/log15" + proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/peer" @@ -25,7 +25,6 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo } log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - //log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) err := s.WriteLength(len(encMsgBuf)) if err != nil { @@ -63,8 +62,6 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, msgbuf, err = xx.Decode1(buf) } - //log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) - if err != nil { log.Debugf("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) @@ -181,14 +178,8 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init return fmt.Errorf("validation fail") } - //log.Debugf("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) - } - log.Debugf("stage 1 initiator", "payload", plaintext) - - log.Debugf("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) - // stage 2 // if !fallback { @@ -243,7 +234,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init if !fallback { // read message - buf, plaintext, valid, err = s.xx_recvHandshakeMessage(true) + _, plaintext, valid, err = s.xx_recvHandshakeMessage(true) if err != nil { return fmt.Errorf("stage 0 responder fail: %s", err) } @@ -288,8 +279,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") return fmt.Errorf("validation fail") } - - //log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) } log.Debugf("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) @@ -304,7 +293,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init // stage 2 // // read message - buf, plaintext, valid, err = s.xx_recvHandshakeMessage(false) + _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) if err != nil { return fmt.Errorf("stage 2 responder fail: %s", err) } From a1ca411f53312a23a843d3d2a373fb12c343d2f3 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 15:56:31 -0400 Subject: [PATCH 1461/3965] fix ReadSecure and WriteSecure --- p2p/security/noise/protocol.go | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index de15ed25e8..b7e0387902 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -50,6 +50,8 @@ type peerInfo struct { libp2pKey crypto.PubKey } +// newSecureSession creates a noise session that can be configured to be initialized with a static +// noise key `noisePrivateKey`, a cache of previous func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, noisePrivateKey [32]byte, insecure net.Conn, remote peer.ID, noiseStaticKeyCache map[peer.ID]([32]byte), noisePipesSupport bool, initiator bool) (*secureSession, error) { @@ -175,9 +177,6 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { } func (s *secureSession) Read(buf []byte) (int, error) { - // TODO: use noise symmetric keys - return s.insecure.Read(buf) - plaintext, err := s.ReadSecure() if err != nil { return 0, nil @@ -194,7 +193,7 @@ func (s *secureSession) ReadSecure() ([]byte, error) { } ciphertext := make([]byte, l) - _, err = s.Read(ciphertext) + _, err = s.insecure.Read(ciphertext) if err != nil { return nil, err } @@ -227,9 +226,6 @@ func (s *secureSession) SetWriteDeadline(t time.Time) error { } func (s *secureSession) Write(in []byte) (int, error) { - // TODO: use noise symmetric keys - return s.insecure.Write(in) - err := s.WriteSecure(in) return len(in), err } @@ -245,7 +241,7 @@ func (s *secureSession) WriteSecure(in []byte) error { return err } - _, err = s.Write(ciphertext) + _, err = s.insecure.Write(ciphertext) return err } From 2a8a2cb67e102bbe56ebd7016531863722090804 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 16:03:17 -0400 Subject: [PATCH 1462/3965] clean up --- p2p/security/noise/ik_handshake.go | 4 ++-- p2p/security/noise/protocol.go | 14 +++++++++----- p2p/security/noise/xx_handshake.go | 4 ++-- 3 files changed, 13 insertions(+), 9 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index d2eb9dbe47..da3fcf83b4 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -24,7 +24,7 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) - err := s.WriteLength(len(encMsgBuf)) + err := s.writeLength(len(encMsgBuf)) if err != nil { log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) @@ -41,7 +41,7 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo } func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { - l, err := s.ReadLength() + l, err := s.readLength() if err != nil { return nil, nil, false, fmt.Errorf("read length fail: %s", err) } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index b7e0387902..d033d1c32d 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -51,7 +51,11 @@ type peerInfo struct { } // newSecureSession creates a noise session that can be configured to be initialized with a static -// noise key `noisePrivateKey`, a cache of previous +// noise key `noisePrivateKey`, a cache of previous peer noise keys `noiseStaticKeyCache`, an +// option `noisePipesSupport` to turn on or off noise pipes +// +// With noise pipes off, we always do XX +// With noise pipes on, we first try IK, if that fails, move to XXfallback func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, noisePrivateKey [32]byte, insecure net.Conn, remote peer.ID, noiseStaticKeyCache map[peer.ID]([32]byte), noisePipesSupport bool, initiator bool) (*secureSession, error) { @@ -85,13 +89,13 @@ func (s *secureSession) NoisePrivateKey() [32]byte { return s.noisePrivateKey } -func (s *secureSession) ReadLength() (int, error) { +func (s *secureSession) readLength() (int, error) { buf := make([]byte, 2) _, err := s.insecure.Read(buf) return int(binary.BigEndian.Uint16(buf)), err } -func (s *secureSession) WriteLength(length int) error { +func (s *secureSession) writeLength(length int) error { buf := make([]byte, 2) binary.BigEndian.PutUint16(buf, uint16(length)) _, err := s.insecure.Write(buf) @@ -187,7 +191,7 @@ func (s *secureSession) Read(buf []byte) (int, error) { } func (s *secureSession) ReadSecure() ([]byte, error) { - l, err := s.ReadLength() + l, err := s.readLength() if err != nil { return nil, err } @@ -236,7 +240,7 @@ func (s *secureSession) WriteSecure(in []byte) error { return err } - err = s.WriteLength(len(ciphertext)) + err = s.writeLength(len(ciphertext)) if err != nil { return err } diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 391acd62e9..bd7943e166 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -26,7 +26,7 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - err := s.WriteLength(len(encMsgBuf)) + err := s.writeLength(len(encMsgBuf)) if err != nil { log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) @@ -43,7 +43,7 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo } func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { - l, err := s.ReadLength() + l, err := s.readLength() if err != nil { return nil, nil, false, fmt.Errorf("read length fail: %s", err) } From b7d0e5bc52140c8416b236ecfd6ec550b7d02164 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 16:05:46 -0400 Subject: [PATCH 1463/3965] remove c.out --- p2p/security/noise/c.out | 228 --------------------------------------- 1 file changed, 228 deletions(-) delete mode 100644 p2p/security/noise/c.out diff --git a/p2p/security/noise/c.out b/p2p/security/noise/c.out deleted file mode 100644 index bf84a4ac91..0000000000 --- a/p2p/security/noise/c.out +++ /dev/null @@ -1,228 +0,0 @@ -mode: set -github.com/ChainSafe/go-libp2p-noise/crypto.go:9.82,10.19 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:30.2,30.24 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:10.19,11.18 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:11.18,14.4 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:14.9,17.4 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:18.8,18.26 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:18.26,19.18 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:19.18,22.4 2 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:22.9,25.4 2 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:26.8,28.3 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:33.82,35.19 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:55.2,55.9 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:59.2,59.23 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:35.19,36.18 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:36.18,39.4 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:39.9,42.4 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:43.8,43.26 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:43.26,44.18 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:44.18,47.4 2 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:47.9,50.4 2 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:51.8,53.3 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:55.9,57.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:13.91,20.19 5 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:26.2,30.16 4 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:36.2,37.16 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:42.2,42.12 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:20.19,22.3 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:22.8,24.3 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:30.16,33.3 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:37.16,40.3 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:45.123,47.16 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:51.2,54.16 3 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:58.2,59.19 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:65.2,67.16 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:72.2,73.12 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:78.2,80.35 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:47.16,49.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:54.16,56.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:59.19,61.3 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:61.8,63.3 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:67.16,70.3 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:73.12,76.3 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:84.78,87.37 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:97.2,103.16 4 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:107.2,112.16 4 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:117.2,121.16 5 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:126.2,129.17 3 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:230.2,231.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:87.37,92.3 3 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:92.8,95.3 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:103.16,105.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:112.16,114.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:121.16,123.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:129.17,132.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:140.3,141.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:145.3,145.13 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:150.3,152.17 3 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:157.3,158.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:164.3,165.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:170.3,171.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:132.17,135.4 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:141.17,143.4 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:145.13,147.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:152.17,154.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:158.17,161.4 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:165.17,167.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:171.17,173.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:175.8,184.17 5 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:188.3,188.13 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:192.3,197.17 4 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:202.3,203.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:209.3,210.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:215.3,216.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:222.3,223.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:184.17,186.4 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:188.13,190.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:197.17,199.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:203.17,206.4 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:210.17,212.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:216.17,218.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:223.17,226.4 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:58.66,60.32 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:64.2,78.15 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:60.32,62.3 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:81.70,83.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:85.52,87.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:89.51,93.2 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:95.55,100.2 4 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:102.67,105.2 2 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:107.72,110.2 2 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:112.105,119.16 5 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:125.2,125.12 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:119.16,121.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:121.8,121.16 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:121.16,123.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:128.65,133.78 2 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:168.2,168.12 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:133.78,139.17 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:153.3,153.23 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:139.17,145.18 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:150.4,150.24 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:145.18,148.5 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:155.8,160.17 2 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:165.3,165.23 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:160.17,163.4 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:171.46,173.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:175.45,177.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:179.58,181.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:183.56,185.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:187.55,192.16 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:196.2,197.22 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:192.16,194.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:200.54,202.16 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:206.2,208.16 3 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:212.2,212.30 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:202.16,204.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:208.16,210.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:215.47,217.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:219.46,221.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:223.57,225.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:227.56,229.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:231.60,233.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:235.61,237.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:239.55,245.2 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:247.54,249.16 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:253.2,254.16 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:258.2,259.12 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:249.16,251.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:254.16,256.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:262.39,264.2 1 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:26.99,28.16 2 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:32.2,35.15 4 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:28.16,30.3 1 0 -github.com/ChainSafe/go-libp2p-noise/transport.go:39.111,41.16 2 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:45.2,48.15 4 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:41.16,43.3 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:14.91,21.19 5 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:27.2,31.16 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:37.2,38.16 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:43.2,43.12 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:21.19,23.3 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:23.8,25.3 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:31.16,34.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:38.16,41.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:46.123,48.16 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:52.2,55.16 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:59.2,60.19 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:68.2,68.16 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:73.2,74.12 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:79.2,81.35 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:48.16,50.3 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:55.16,57.3 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:60.19,62.3 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:62.8,64.3 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:68.16,71.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:74.12,77.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:85.108,88.37 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:98.2,104.16 4 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:108.2,113.16 4 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:118.2,125.16 6 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:131.2,133.17 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:355.2,356.12 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:88.37,93.3 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:93.8,96.3 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:104.16,106.3 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:113.16,116.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:125.16,128.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:133.17,136.16 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:155.3,157.16 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:188.3,194.16 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:207.3,209.17 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:214.3,215.17 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:221.3,222.26 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:229.3,230.17 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:234.3,234.60 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:136.16,138.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:138.18,140.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:141.9,151.4 6 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:157.16,160.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:164.4,164.14 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:160.18,162.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:164.14,166.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:167.9,173.18 4 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:178.4,179.14 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:173.18,176.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:179.14,182.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:194.16,196.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:196.18,198.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:199.9,201.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:201.18,203.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:209.17,211.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:215.17,218.4 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:222.26,224.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:224.9,224.24 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:224.24,226.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:230.17,232.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:236.8,244.16 4 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:295.3,300.17 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:307.3,308.17 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:312.3,312.13 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:316.3,320.15 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:342.3,346.17 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:351.3,352.66 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:244.16,247.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:251.4,251.14 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:256.4,257.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:262.4,263.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:269.4,270.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:247.18,249.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:251.14,253.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:257.18,259.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:263.18,266.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:270.18,272.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:274.9,281.18 5 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:286.4,287.14 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:281.18,284.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:287.14,290.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:300.17,302.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:308.17,310.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:312.13,314.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:320.15,323.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:328.4,329.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:335.4,336.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:323.18,325.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:329.18,332.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:336.18,339.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:346.17,349.4 2 0 From 45182a49e80d923e5e13cc7be7cc08a301ae76ec Mon Sep 17 00:00:00 2001 From: noot <36753753+noot@users.noreply.github.com> Date: Sat, 24 Aug 2019 16:48:33 -0400 Subject: [PATCH 1464/3965] cleanup code (#3) cleanup code, fix ReadSecure and WriteSecure --- p2p/security/noise/c.out | 228 ----------------------------- p2p/security/noise/crypto.go | 16 +- p2p/security/noise/crypto_test.go | 13 ++ p2p/security/noise/ik/IK.noise.go | 15 +- p2p/security/noise/ik_handshake.go | 5 +- p2p/security/noise/protocol.go | 36 ++--- p2p/security/noise/transport.go | 3 + p2p/security/noise/xx/XX.noise.go | 11 -- p2p/security/noise/xx_handshake.go | 21 +-- 9 files changed, 45 insertions(+), 303 deletions(-) delete mode 100644 p2p/security/noise/c.out diff --git a/p2p/security/noise/c.out b/p2p/security/noise/c.out deleted file mode 100644 index bf84a4ac91..0000000000 --- a/p2p/security/noise/c.out +++ /dev/null @@ -1,228 +0,0 @@ -mode: set -github.com/ChainSafe/go-libp2p-noise/crypto.go:9.82,10.19 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:30.2,30.24 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:10.19,11.18 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:11.18,14.4 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:14.9,17.4 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:18.8,18.26 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:18.26,19.18 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:19.18,22.4 2 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:22.9,25.4 2 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:26.8,28.3 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:33.82,35.19 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:55.2,55.9 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:59.2,59.23 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:35.19,36.18 1 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:36.18,39.4 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:39.9,42.4 2 1 -github.com/ChainSafe/go-libp2p-noise/crypto.go:43.8,43.26 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:43.26,44.18 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:44.18,47.4 2 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:47.9,50.4 2 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:51.8,53.3 1 0 -github.com/ChainSafe/go-libp2p-noise/crypto.go:55.9,57.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:13.91,20.19 5 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:26.2,30.16 4 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:36.2,37.16 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:42.2,42.12 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:20.19,22.3 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:22.8,24.3 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:30.16,33.3 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:37.16,40.3 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:45.123,47.16 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:51.2,54.16 3 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:58.2,59.19 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:65.2,67.16 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:72.2,73.12 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:78.2,80.35 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:47.16,49.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:54.16,56.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:59.19,61.3 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:61.8,63.3 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:67.16,70.3 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:73.12,76.3 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:84.78,87.37 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:97.2,103.16 4 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:107.2,112.16 4 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:117.2,121.16 5 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:126.2,129.17 3 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:230.2,231.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:87.37,92.3 3 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:92.8,95.3 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:103.16,105.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:112.16,114.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:121.16,123.3 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:129.17,132.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:140.3,141.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:145.3,145.13 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:150.3,152.17 3 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:157.3,158.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:164.3,165.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:170.3,171.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:132.17,135.4 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:141.17,143.4 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:145.13,147.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:152.17,154.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:158.17,161.4 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:165.17,167.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:171.17,173.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:175.8,184.17 5 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:188.3,188.13 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:192.3,197.17 4 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:202.3,203.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:209.3,210.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:215.3,216.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:222.3,223.17 2 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:184.17,186.4 1 1 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:188.13,190.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:197.17,199.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:203.17,206.4 2 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:210.17,212.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:216.17,218.4 1 0 -github.com/ChainSafe/go-libp2p-noise/ik_handshake.go:223.17,226.4 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:58.66,60.32 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:64.2,78.15 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:60.32,62.3 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:81.70,83.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:85.52,87.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:89.51,93.2 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:95.55,100.2 4 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:102.67,105.2 2 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:107.72,110.2 2 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:112.105,119.16 5 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:125.2,125.12 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:119.16,121.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:121.8,121.16 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:121.16,123.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:128.65,133.78 2 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:168.2,168.12 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:133.78,139.17 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:153.3,153.23 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:139.17,145.18 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:150.4,150.24 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:145.18,148.5 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:155.8,160.17 2 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:165.3,165.23 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:160.17,163.4 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:171.46,173.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:175.45,177.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:179.58,181.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:183.56,185.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:187.55,192.16 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:196.2,197.22 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:192.16,194.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:200.54,202.16 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:206.2,208.16 3 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:212.2,212.30 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:202.16,204.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:208.16,210.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:215.47,217.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:219.46,221.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:223.57,225.2 1 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:227.56,229.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:231.60,233.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:235.61,237.2 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:239.55,245.2 3 1 -github.com/ChainSafe/go-libp2p-noise/protocol.go:247.54,249.16 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:253.2,254.16 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:258.2,259.12 2 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:249.16,251.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:254.16,256.3 1 0 -github.com/ChainSafe/go-libp2p-noise/protocol.go:262.39,264.2 1 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:26.99,28.16 2 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:32.2,35.15 4 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:28.16,30.3 1 0 -github.com/ChainSafe/go-libp2p-noise/transport.go:39.111,41.16 2 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:45.2,48.15 4 1 -github.com/ChainSafe/go-libp2p-noise/transport.go:41.16,43.3 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:14.91,21.19 5 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:27.2,31.16 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:37.2,38.16 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:43.2,43.12 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:21.19,23.3 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:23.8,25.3 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:31.16,34.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:38.16,41.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:46.123,48.16 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:52.2,55.16 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:59.2,60.19 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:68.2,68.16 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:73.2,74.12 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:79.2,81.35 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:48.16,50.3 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:55.16,57.3 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:60.19,62.3 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:62.8,64.3 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:68.16,71.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:74.12,77.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:85.108,88.37 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:98.2,104.16 4 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:108.2,113.16 4 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:118.2,125.16 6 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:131.2,133.17 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:355.2,356.12 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:88.37,93.3 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:93.8,96.3 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:104.16,106.3 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:113.16,116.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:125.16,128.3 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:133.17,136.16 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:155.3,157.16 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:188.3,194.16 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:207.3,209.17 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:214.3,215.17 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:221.3,222.26 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:229.3,230.17 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:234.3,234.60 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:136.16,138.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:138.18,140.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:141.9,151.4 6 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:157.16,160.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:164.4,164.14 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:160.18,162.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:164.14,166.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:167.9,173.18 4 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:178.4,179.14 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:173.18,176.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:179.14,182.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:194.16,196.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:196.18,198.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:199.9,201.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:201.18,203.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:209.17,211.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:215.17,218.4 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:222.26,224.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:224.9,224.24 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:224.24,226.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:230.17,232.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:236.8,244.16 4 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:295.3,300.17 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:307.3,308.17 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:312.3,312.13 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:316.3,320.15 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:342.3,346.17 3 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:351.3,352.66 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:244.16,247.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:251.4,251.14 1 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:256.4,257.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:262.4,263.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:269.4,270.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:247.18,249.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:251.14,253.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:257.18,259.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:263.18,266.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:270.18,272.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:274.9,281.18 5 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:286.4,287.14 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:281.18,284.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:287.14,290.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:300.17,302.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:308.17,310.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:312.13,314.4 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:320.15,323.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:328.4,329.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:335.4,336.18 2 1 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:323.18,325.5 1 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:329.18,332.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:336.18,339.5 2 0 -github.com/ChainSafe/go-libp2p-noise/xx_handshake.go:346.17,349.4 2 0 diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index 912fcca59e..e2f9fb858f 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -10,18 +10,18 @@ func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) if s.xx_complete { if s.initiator { cs := s.xx_ns.CS1() - cs, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) + _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) } else { cs := s.xx_ns.CS2() - cs, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) + _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) } } else if s.ik_complete { if s.initiator { cs := s.ik_ns.CS1() - cs, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) + _, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) } else { cs := s.ik_ns.CS2() - cs, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) + _, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) } } else { return nil, errors.New("encrypt err: haven't completed handshake") @@ -35,18 +35,18 @@ func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) if s.xx_complete { if s.initiator { cs := s.xx_ns.CS2() - cs, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) + _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) } else { cs := s.xx_ns.CS1() - cs, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) + _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) } } else if s.ik_complete { if s.initiator { cs := s.ik_ns.CS2() - cs, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) + _, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) } else { cs := s.ik_ns.CS1() - cs, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) + _, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) } } else { return nil, errors.New("decrypt err: haven't completed handshake") diff --git a/p2p/security/noise/crypto_test.go b/p2p/security/noise/crypto_test.go index 09fae902ae..dff7ab7c01 100644 --- a/p2p/security/noise/crypto_test.go +++ b/p2p/security/noise/crypto_test.go @@ -27,6 +27,19 @@ func TestEncryptAndDecrypt_InitToResp(t *testing.T) { } else if err != nil { t.Fatal(err) } + + plaintext = []byte("goodbye") + ciphertext, err = initConn.Encrypt(plaintext) + if err != nil { + t.Fatal(err) + } + + result, err = respConn.Decrypt(ciphertext) + if !bytes.Equal(plaintext, result) { + t.Fatalf("got %x expected %x", result, plaintext) + } else if err != nil { + t.Fatal(err) + } } func TestEncryptAndDecrypt_RespToInit(t *testing.T) { diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index a36ac09e42..a68d721784 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -170,9 +170,6 @@ func (mb *MessageBuffer) Encode1() []byte { enc = append(enc, mb.ne[:]...) enc = append(enc, mb.ciphertext...) - // log.Debug("XX_Encode1", "ne", mb.ne) - // log.Debug("XX_Encode1", "ns", mb.ns) - return enc } @@ -182,12 +179,10 @@ func Decode0(in []byte) (*MessageBuffer, error) { return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") } - //log.Debug("XX_Decode0", "in", in) mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) mb.ns = in[32:80] mb.ciphertext = in[80:] - //log.Debug("XX_Decode0", "mb", mb) return mb, nil } @@ -195,18 +190,12 @@ func Decode0(in []byte) (*MessageBuffer, error) { // Decodes messages at stage 1 into MessageBuffer func Decode1(in []byte) (*MessageBuffer, error) { if len(in) < 80 { - return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") + return nil, errors.New("cannot decode stage 1 MessageBuffer: length less than 96 bytes") } - // log.Debug("XX_Decode1", "in", in) - // log.Debug("XX_Decode1", "ns", in[32:80]) - mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) - //mb.ns = in[32:80] mb.ciphertext = in[32:] - // copy(mb.ns,) - // copy(mb.ciphertext,) return mb, nil } @@ -523,7 +512,6 @@ func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageB } if session.mc == 1 { session.h, messageBuffer, session.cs1, session.cs2 = writeMessageB(&session.hs, message) - //session.hs = handshakestate{} } session.mc = session.mc + 1 return session, messageBuffer @@ -537,7 +525,6 @@ func RecvMessage(session *NoiseSession, message *MessageBuffer) (*NoiseSession, } if session.mc == 1 { session.h, plaintext, valid, session.cs1, session.cs2 = readMessageB(&session.hs, message) - //session.hs = handshakestate{} } session.mc = session.mc + 1 return session, plaintext, valid diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index b1fc7eec20..da3fcf83b4 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -23,9 +23,8 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo } log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) - log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf)) - err := s.WriteLength(len(encMsgBuf)) + err := s.writeLength(len(encMsgBuf)) if err != nil { log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) @@ -42,7 +41,7 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo } func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { - l, err := s.ReadLength() + l, err := s.readLength() if err != nil { return nil, nil, false, fmt.Errorf("read length fail: %s", err) } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 4504e7f705..d033d1c32d 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -4,15 +4,12 @@ import ( "context" "encoding/binary" "fmt" - //"io" "net" "time" logging "github.com/ipfs/go-log" - //proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" - //"github.com/libp2p/go-libp2p-core/sec" ik "github.com/ChainSafe/go-libp2p-noise/ik" pb "github.com/ChainSafe/go-libp2p-noise/pb" @@ -53,6 +50,12 @@ type peerInfo struct { libp2pKey crypto.PubKey } +// newSecureSession creates a noise session that can be configured to be initialized with a static +// noise key `noisePrivateKey`, a cache of previous peer noise keys `noiseStaticKeyCache`, an +// option `noisePipesSupport` to turn on or off noise pipes +// +// With noise pipes off, we always do XX +// With noise pipes on, we first try IK, if that fails, move to XXfallback func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, noisePrivateKey [32]byte, insecure net.Conn, remote peer.ID, noiseStaticKeyCache map[peer.ID]([32]byte), noisePipesSupport bool, initiator bool) (*secureSession, error) { @@ -86,13 +89,13 @@ func (s *secureSession) NoisePrivateKey() [32]byte { return s.noisePrivateKey } -func (s *secureSession) ReadLength() (int, error) { +func (s *secureSession) readLength() (int, error) { buf := make([]byte, 2) _, err := s.insecure.Read(buf) return int(binary.BigEndian.Uint16(buf)), err } -func (s *secureSession) WriteLength(length int) error { +func (s *secureSession) writeLength(length int) error { buf := make([]byte, 2) binary.BigEndian.PutUint16(buf, uint16(length)) _, err := s.insecure.Write(buf) @@ -126,21 +129,15 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe } func (s *secureSession) runHandshake(ctx context.Context) error { - - log.Debugf("runHandshake", "cache", s.noiseStaticKeyCache) - // if we have the peer's noise static key and we support noise pipes, we can try IK if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} || s.noisePipesSupport { - log.Debugf("runHandshake_ik") - // known static key for peer, try IK // buf, err := s.runHandshake_ik(ctx) if err != nil { log.Error("runHandshake_ik", "err", err) - // TODO: PIPE TO XX - + // IK failed, pipe to XXfallback err = s.runHandshake_xx(ctx, true, buf) if err != nil { log.Error("runHandshake_xx", "err", err) @@ -153,7 +150,6 @@ func (s *secureSession) runHandshake(ctx context.Context) error { s.ik_complete = true } else { - // unknown static key for peer, try XX // err := s.runHandshake_xx(ctx, false, nil) @@ -185,9 +181,6 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { } func (s *secureSession) Read(buf []byte) (int, error) { - // TODO: use noise symmetric keys - return s.insecure.Read(buf) - plaintext, err := s.ReadSecure() if err != nil { return 0, nil @@ -198,13 +191,13 @@ func (s *secureSession) Read(buf []byte) (int, error) { } func (s *secureSession) ReadSecure() ([]byte, error) { - l, err := s.ReadLength() + l, err := s.readLength() if err != nil { return nil, err } ciphertext := make([]byte, l) - _, err = s.Read(ciphertext) + _, err = s.insecure.Read(ciphertext) if err != nil { return nil, err } @@ -237,9 +230,6 @@ func (s *secureSession) SetWriteDeadline(t time.Time) error { } func (s *secureSession) Write(in []byte) (int, error) { - // TODO: use noise symmetric keys - return s.insecure.Write(in) - err := s.WriteSecure(in) return len(in), err } @@ -250,12 +240,12 @@ func (s *secureSession) WriteSecure(in []byte) error { return err } - err = s.WriteLength(len(ciphertext)) + err = s.writeLength(len(ciphertext)) if err != nil { return err } - _, err = s.Write(ciphertext) + _, err = s.insecure.Write(ciphertext) return err } diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 6d0e216b6c..c784fafe87 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -9,10 +9,13 @@ import ( "github.com/libp2p/go-libp2p-core/sec" ) +// ID is the protocol ID for noise const ID = "/noise/0.0.1" var _ sec.SecureTransport = &Transport{} +// Transport implements the interface sec.SecureTransport +// https://godoc.org/github.com/libp2p/go-libp2p-core/sec#SecureConn type Transport struct { LocalID peer.ID PrivateKey crypto.PrivKey diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 5fa6dac983..bfd743ee23 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -26,7 +26,6 @@ import ( "hash" "io" "math" - //log "github.com/ChainSafe/log15" ) /* ---------------------------------------------------------------- * @@ -171,9 +170,6 @@ func (mb *MessageBuffer) Encode1() []byte { enc = append(enc, mb.ns...) enc = append(enc, mb.ciphertext...) - // log.Debug("XX_Encode1", "ne", mb.ne) - // log.Debug("XX_Encode1", "ns", mb.ns) - return enc } @@ -183,11 +179,9 @@ func Decode0(in []byte) (*MessageBuffer, error) { return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") } - //log.Debug("XX_Decode0", "in", in) mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) mb.ciphertext = in[32:] - //log.Debug("XX_Decode0", "mb", mb) return mb, nil } @@ -198,15 +192,10 @@ func Decode1(in []byte) (*MessageBuffer, error) { return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") } - // log.Debug("XX_Decode1", "in", in) - // log.Debug("XX_Decode1", "ns", in[32:80]) - mb := new(MessageBuffer) copy(mb.ne[:], in[:32]) mb.ns = in[32:80] mb.ciphertext = in[80:] - // copy(mb.ns,) - // copy(mb.ciphertext,) return mb, nil } diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index cd4f76ab81..bd7943e166 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -3,7 +3,7 @@ package noise import ( "context" "fmt" - //log "github.com/ChainSafe/log15" + proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/peer" @@ -25,9 +25,8 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo } log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - //log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "encMsgBuf", encMsgBuf, "ns_len", len(msgbuf.NS()), "enc_len", len(encMsgBuf), "initial_stage", initial_stage) - err := s.WriteLength(len(encMsgBuf)) + err := s.writeLength(len(encMsgBuf)) if err != nil { log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) @@ -44,7 +43,7 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo } func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { - l, err := s.ReadLength() + l, err := s.readLength() if err != nil { return nil, nil, false, fmt.Errorf("read length fail: %s", err) } @@ -63,8 +62,6 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, msgbuf, err = xx.Decode1(buf) } - //log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf), "initial_stage", initial_stage) - if err != nil { log.Debugf("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) @@ -181,14 +178,8 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init return fmt.Errorf("validation fail") } - //log.Debugf("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) - } - log.Debugf("stage 1 initiator", "payload", plaintext) - - log.Debugf("stage 1 initiator", "remote key", s.xx_ns.RemoteKey()) - // stage 2 // if !fallback { @@ -243,7 +234,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init if !fallback { // read message - buf, plaintext, valid, err = s.xx_recvHandshakeMessage(true) + _, plaintext, valid, err = s.xx_recvHandshakeMessage(true) if err != nil { return fmt.Errorf("stage 0 responder fail: %s", err) } @@ -288,8 +279,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") return fmt.Errorf("validation fail") } - - //log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) } log.Debugf("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) @@ -304,7 +293,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init // stage 2 // // read message - buf, plaintext, valid, err = s.xx_recvHandshakeMessage(false) + _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) if err != nil { return fmt.Errorf("stage 2 responder fail: %s", err) } From 85a35c04f035003947e908aeacea04b8ca2a6f4f Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 18:03:22 -0400 Subject: [PATCH 1465/3965] send payload from initiator in XX last message --- p2p/security/noise/.idea/go-libp2p-noise.iml | 8 + p2p/security/noise/.idea/misc.xml | 6 + p2p/security/noise/.idea/modules.xml | 8 + p2p/security/noise/.idea/vcs.xml | 6 + p2p/security/noise/.idea/workspace.xml | 509 +++++++++++++++++++ p2p/security/noise/integration_test.go | 96 ++++ p2p/security/noise/transport.go | 12 +- p2p/security/noise/xx_handshake.go | 69 +-- 8 files changed, 664 insertions(+), 50 deletions(-) create mode 100644 p2p/security/noise/.idea/go-libp2p-noise.iml create mode 100644 p2p/security/noise/.idea/misc.xml create mode 100644 p2p/security/noise/.idea/modules.xml create mode 100644 p2p/security/noise/.idea/vcs.xml create mode 100644 p2p/security/noise/.idea/workspace.xml create mode 100644 p2p/security/noise/integration_test.go diff --git a/p2p/security/noise/.idea/go-libp2p-noise.iml b/p2p/security/noise/.idea/go-libp2p-noise.iml new file mode 100644 index 0000000000..c956989b29 --- /dev/null +++ b/p2p/security/noise/.idea/go-libp2p-noise.iml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/p2p/security/noise/.idea/misc.xml b/p2p/security/noise/.idea/misc.xml new file mode 100644 index 0000000000..28a804d893 --- /dev/null +++ b/p2p/security/noise/.idea/misc.xml @@ -0,0 +1,6 @@ + + + + + \ No newline at end of file diff --git a/p2p/security/noise/.idea/modules.xml b/p2p/security/noise/.idea/modules.xml new file mode 100644 index 0000000000..fde53be50f --- /dev/null +++ b/p2p/security/noise/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/p2p/security/noise/.idea/vcs.xml b/p2p/security/noise/.idea/vcs.xml new file mode 100644 index 0000000000..94a25f7f4c --- /dev/null +++ b/p2p/security/noise/.idea/vcs.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/p2p/security/noise/.idea/workspace.xml b/p2p/security/noise/.idea/workspace.xml new file mode 100644 index 0000000000..40cc06a228 --- /dev/null +++ b/p2p/security/noise/.idea/workspace.xml @@ -0,0 +1,509 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + true + DEFINITION_ORDER + + + + + + + + + + Control flow issuesJavaScript + + + DOM issuesJavaScript + + + JavaScript + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - true - - - - - - file://$PROJECT_DIR$/integration_test.go - 89 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go - 62 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-conn-security-multistream@v0.1.0/ssms.go - 45 - - - - file://$PROJECT_DIR$/transport.go - 49 - - - - file://$PROJECT_DIR$/transport.go - 60 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/limiter.go - 220 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go - 85 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/listener.go - 103 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file From 048274e4452116853df64870febb8003e4301cf1 Mon Sep 17 00:00:00 2001 From: noot Date: Sat, 24 Aug 2019 20:34:12 -0400 Subject: [PATCH 1467/3965] improve secureSession Read --- p2p/security/noise/.idea/go-libp2p-noise.iml | 8 + p2p/security/noise/.idea/misc.xml | 6 + p2p/security/noise/.idea/modules.xml | 8 + p2p/security/noise/.idea/vcs.xml | 6 + p2p/security/noise/.idea/workspace.xml | 562 +++++++++++++++++++ p2p/security/noise/integration_test.go | 87 ++- p2p/security/noise/protocol.go | 65 ++- p2p/security/noise/transport.go | 4 +- p2p/security/noise/transport_test.go | 2 +- 9 files changed, 699 insertions(+), 49 deletions(-) create mode 100644 p2p/security/noise/.idea/go-libp2p-noise.iml create mode 100644 p2p/security/noise/.idea/misc.xml create mode 100644 p2p/security/noise/.idea/modules.xml create mode 100644 p2p/security/noise/.idea/vcs.xml create mode 100644 p2p/security/noise/.idea/workspace.xml diff --git a/p2p/security/noise/.idea/go-libp2p-noise.iml b/p2p/security/noise/.idea/go-libp2p-noise.iml new file mode 100644 index 0000000000..c956989b29 --- /dev/null +++ b/p2p/security/noise/.idea/go-libp2p-noise.iml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/p2p/security/noise/.idea/misc.xml b/p2p/security/noise/.idea/misc.xml new file mode 100644 index 0000000000..28a804d893 --- /dev/null +++ b/p2p/security/noise/.idea/misc.xml @@ -0,0 +1,6 @@ + + + + + \ No newline at end of file diff --git a/p2p/security/noise/.idea/modules.xml b/p2p/security/noise/.idea/modules.xml new file mode 100644 index 0000000000..fde53be50f --- /dev/null +++ b/p2p/security/noise/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/p2p/security/noise/.idea/vcs.xml b/p2p/security/noise/.idea/vcs.xml new file mode 100644 index 0000000000..94a25f7f4c --- /dev/null +++ b/p2p/security/noise/.idea/vcs.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/p2p/security/noise/.idea/workspace.xml b/p2p/security/noise/.idea/workspace.xml new file mode 100644 index 0000000000..e39f554ded --- /dev/null +++ b/p2p/security/noise/.idea/workspace.xml @@ -0,0 +1,562 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + true + DEFINITION_ORDER + + + + + + + + + + Control flow issuesJavaScript + + + DOM issuesJavaScript + + + JavaScript + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - true - - - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go - 91 - - - - file://$PROJECT_DIR$/protocol.go - 211 - - - - file://$PROJECT_DIR$/protocol.go - 2 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/multiformats/go-multistream@v0.1.0/multistream.go - 424 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go - 90 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go - 118 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/libp2p/go-stream-muxer-multistream@v0.2.0/multistream.go - 49 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/multiformats/go-multistream@v0.1.0/multistream.go - 282 - - - - file://$PROJECT_DIR$/../../../../pkg/mod/github.com/multiformats/go-multistream@v0.1.0/multistream.go - 66 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file From 382b7589573389c97856bc5c1e916b1f632f102d Mon Sep 17 00:00:00 2001 From: noot Date: Mon, 26 Aug 2019 08:33:22 -0400 Subject: [PATCH 1471/3965] add rw lock, streams functioning --- p2p/security/noise/integration_test.go | 22 ++++++++++--------- p2p/security/noise/protocol.go | 29 ++++++++++---------------- 2 files changed, 23 insertions(+), 28 deletions(-) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index 2822dbeab0..b4e887f358 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -14,7 +14,7 @@ import ( "io" mrand "math/rand" "testing" - //"time" + "time" ) func generateKey(seed int64) (crypto.PrivKey, error) { @@ -102,17 +102,19 @@ func TestLibp2pIntegration(t *testing.T) { t.Fatal(err) } - // stream, err := ha.NewStream(ctx, hb.ID(), ID) - // if err != nil { - // t.Fatal(err) - // } + stream, err := ha.NewStream(ctx, hb.ID(), ID) + if err != nil { + t.Fatal(err) + } + + _, err = stream.Write([]byte("hello\n")) + if err != nil { + t.Fatal(err) + } - // _, err = stream.Write([]byte("hello\n")) - // if err != nil { - // t.Fatal(err) - // } + fmt.Println("fin") - // fmt.Println("fin") + time.Sleep(time.Second) } func handleStream(stream net.Stream) { diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 9b57b117c0..2c3cef67d0 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -1,11 +1,11 @@ package noise import ( - "bufio" "context" "encoding/binary" "fmt" "net" + "sync" "time" logging "github.com/ipfs/go-log" @@ -45,12 +45,12 @@ type secureSession struct { noisePrivateKey [32]byte - rw bufio.ReadWriter msgBuffer []byte + rwLock sync.Mutex } type peerInfo struct { - noiseKey [32]byte // static noise key + noiseKey [32]byte // static noise public key libp2pKey crypto.PubKey } @@ -78,12 +78,9 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey noisePipesSupport: noisePipesSupport, noiseStaticKeyCache: noiseStaticKeyCache, noisePrivateKey: noisePrivateKey, - rw: *bufio.NewReadWriter(bufio.NewReader(insecure), bufio.NewWriter(insecure)), msgBuffer: []byte{}, } - s.rw.Writer.Flush() - err := s.runHandshake(ctx) return s, err @@ -189,13 +186,10 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { } func (s *secureSession) Read(buf []byte) (int, error) { - //return s.insecure.Read(buf) l := len(buf) - //log.Debug("length", l) - if l <= len(s.msgBuffer) { - //log.Debug("length < msgbuffer") + log.Debug("length < msgbuffer") copy(buf, s.msgBuffer) s.msgBuffer = s.msgBuffer[l:] @@ -205,23 +199,22 @@ func (s *secureSession) Read(buf []byte) (int, error) { l, err := s.readLength() if err != nil { - //log.Error("read length err", err) return 0, err } ciphertext := make([]byte, l) - _, err = s.rw.Read(ciphertext) + _, err = s.insecure.Read(ciphertext) if err != nil { - //log.Error("read ciphertext err", err) + log.Error("read ciphertext err", err) return 0, err } plaintext, err := s.Decrypt(ciphertext) if err != nil { - //log.Error("decrypt err", err) + log.Error("decrypt err", err) return 0, err } @@ -231,8 +224,6 @@ func (s *secureSession) Read(buf []byte) (int, error) { s.msgBuffer = append(s.msgBuffer, plaintext[len(buf):]...) } - //log.Debug("read", "plaintext", plaintext, len(plaintext)) - return c, nil } @@ -261,16 +252,18 @@ func (s *secureSession) SetWriteDeadline(t time.Time) error { } func (s *secureSession) Write(in []byte) (int, error) { + s.rwLock.Lock() + defer s.rwLock.Unlock() ciphertext, err := s.Encrypt(in) if err != nil { - //log.Error("encrypt error", err) + log.Error("encrypt error", err) return 0, err } err = s.writeLength(len(ciphertext)) if err != nil { - //log.Error("write length err", err) + log.Error("write length err", err) return 0, err } From 062c1b55e636242a99721ed476518589bdefe16b Mon Sep 17 00:00:00 2001 From: noot Date: Mon, 26 Aug 2019 16:48:20 -0400 Subject: [PATCH 1472/3965] move keypair generation to constructors, add xxfallback integration test --- p2p/security/noise/ik/IK.noise.go | 8 +- p2p/security/noise/ik_handshake.go | 19 ++-- p2p/security/noise/integration_test.go | 147 ++++++++++++++++++++++++- p2p/security/noise/protocol.go | 19 ++-- p2p/security/noise/transport.go | 36 ++++-- p2p/security/noise/transport_test.go | 2 +- p2p/security/noise/xx_handshake.go | 22 ++-- 7 files changed, 209 insertions(+), 44 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index a68d721784..61a11ed038 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -175,8 +175,8 @@ func (mb *MessageBuffer) Encode1() []byte { // Decodes initial message (stage 0) into MessageBuffer func Decode0(in []byte) (*MessageBuffer, error) { - if len(in) < 32 { - return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") + if len(in) < 80 { + return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 80 bytes") } mb := new(MessageBuffer) @@ -189,8 +189,8 @@ func Decode0(in []byte) (*MessageBuffer, error) { // Decodes messages at stage 1 into MessageBuffer func Decode1(in []byte) (*MessageBuffer, error) { - if len(in) < 80 { - return nil, errors.New("cannot decode stage 1 MessageBuffer: length less than 96 bytes") + if len(in) < 32 { + return nil, errors.New("cannot decode stage 1 MessageBuffer: length less than 32 bytes") } mb := new(MessageBuffer) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index da3fcf83b4..01624f2e17 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -80,17 +80,14 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, // returns last successful message upon error func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { - var kp ik.Keypair - - if s.noisePrivateKey == [32]byte{} { - // generate local static noise key - kp = ik.GenerateKeypair() - s.noisePrivateKey = kp.PrivKey() - log.Debugf("ik generate new noise kp", "initiator", s.initiator) - } else { - pub := ik.GeneratePublicKey(s.noisePrivateKey) - kp = ik.NewKeypair(pub, s.noisePrivateKey) - } + kp := ik.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) + + // if s.noiseKeypair == nil { + // // generate local static noise key + // s.noiseKeypair = GenerateKeypair() + // } else { + // kp = GenerateKeypair().(ik.Keypair) + // } s.local.noiseKey = kp.PubKey() diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index b4e887f358..a53712a1e5 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -44,7 +44,7 @@ func makeNode(t *testing.T, seed int64, port int) (host.Host, error) { t.Fatal(err) } - tpt := NewTransport(pid, priv, false) + tpt := NewTransport(pid, priv, false, nil) ip := "0.0.0.0" addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", ip, port)) @@ -63,7 +63,40 @@ func makeNode(t *testing.T, seed int64, port int) (host.Host, error) { return libp2p.New(ctx, options...) } -func TestLibp2pIntegration(t *testing.T) { +func makeNodePipes(t *testing.T, seed int64, port int, rpid peer.ID, rpubkey [32]byte, kp *Keypair) (host.Host, error) { + priv, err := generateKey(seed) + if err != nil { + t.Fatal(err) + } + + pid, err := peer.IDFromPrivateKey(priv) + if err != nil { + t.Fatal(err) + } + + tpt := NewTransport(pid, priv, true, kp) + tpt.NoiseStaticKeyCache = make(map[peer.ID]([32]byte)) + tpt.NoiseStaticKeyCache[rpid] = rpubkey + + ip := "0.0.0.0" + addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", ip, port)) + if err != nil { + t.Fatal(err) + } + + options := []libp2p.Option{ + libp2p.Identity(priv), + libp2p.Security(ID, tpt), + libp2p.ListenAddrs(addr), + } + + ctx := context.Background() + + h, err := libp2p.New(ctx, options...) + return h, err +} + +func TestLibp2pIntegration_NoPipes(t *testing.T) { ctx := context.Background() ha, err := makeNode(t, 1, 33333) @@ -117,6 +150,116 @@ func TestLibp2pIntegration(t *testing.T) { time.Sleep(time.Second) } +func TestLibp2pIntegration_WithPipes(t *testing.T) { + ctx := context.Background() + + kpa := GenerateKeypair() + + ha, err := makeNodePipes(t, 1, 33333, "", [32]byte{}, kpa) + if err != nil { + t.Fatal(err) + } + + defer ha.Close() + + //fmt.Printf("ha: %s/p2p/%s\n", ha.Addrs()[1].String(), ha.ID()) + + hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.public_key, nil) + if err != nil { + t.Fatal(err) + } + + defer hb.Close() + + ha.SetStreamHandler(ID, handleStream) + hb.SetStreamHandler(ID, handleStream) + + addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", ha.Addrs()[0].String(), ha.ID())) + if err != nil { + t.Fatal(err) + } + + fmt.Printf("ha: %s\n", addr) + + addrInfo, err := peer.AddrInfoFromP2pAddr(addr) + if err != nil { + t.Fatal(err) + } + + err = hb.Connect(ctx, *addrInfo) + if err != nil { + t.Fatal(err) + } + + stream, err := hb.NewStream(ctx, ha.ID(), ID) + if err != nil { + t.Fatal(err) + } + + _, err = stream.Write([]byte("hello\n")) + if err != nil { + t.Fatal(err) + } + + fmt.Println("fin") + + time.Sleep(time.Second) +} + +func TestLibp2pIntegration_XXFallback(t *testing.T) { + ctx := context.Background() + + ha, err := makeNodePipes(t, 1, 33333, "", [32]byte{}, nil) + if err != nil { + t.Fatal(err) + } + + defer ha.Close() + + //fmt.Printf("ha: %s/p2p/%s\n", ha.Addrs()[1].String(), ha.ID()) + + hb, err := makeNode(t, 2, 34343) + if err != nil { + t.Fatal(err) + } + + defer hb.Close() + + ha.SetStreamHandler(ID, handleStream) + hb.SetStreamHandler(ID, handleStream) + + addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", ha.Addrs()[0].String(), ha.ID())) + if err != nil { + t.Fatal(err) + } + + fmt.Printf("ha: %s\n", addr) + + addrInfo, err := peer.AddrInfoFromP2pAddr(addr) + if err != nil { + t.Fatal(err) + } + + err = hb.Connect(ctx, *addrInfo) + if err != nil { + t.Fatal(err) + } + + stream, err := hb.NewStream(ctx, ha.ID(), ID) + if err != nil { + t.Fatal(err) + } + + _, err = stream.Write([]byte("hello\n")) + if err != nil { + t.Fatal(err) + } + + fmt.Println("fin") + + time.Sleep(time.Second) +} + func handleStream(stream net.Stream) { defer func() { if err := stream.Close(); err != nil { diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 2c3cef67d0..6ba67ac17b 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -43,7 +43,7 @@ type secureSession struct { noisePipesSupport bool noiseStaticKeyCache map[peer.ID]([32]byte) - noisePrivateKey [32]byte + noiseKeypair *Keypair msgBuffer []byte rwLock sync.Mutex @@ -60,7 +60,7 @@ type peerInfo struct { // // With noise pipes off, we always do XX // With noise pipes on, we first try IK, if that fails, move to XXfallback -func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, noisePrivateKey [32]byte, +func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, kp *Keypair, insecure net.Conn, remote peer.ID, noiseStaticKeyCache map[peer.ID]([32]byte), noisePipesSupport bool, initiator bool) (*secureSession, error) { @@ -68,6 +68,10 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey noiseStaticKeyCache = make(map[peer.ID]([32]byte)) } + if kp == nil { + kp = GenerateKeypair() + } + s := &secureSession{ insecure: insecure, initiator: initiator, @@ -77,8 +81,8 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey remotePeer: remote, noisePipesSupport: noisePipesSupport, noiseStaticKeyCache: noiseStaticKeyCache, - noisePrivateKey: noisePrivateKey, msgBuffer: []byte{}, + noiseKeypair: kp, } err := s.runHandshake(ctx) @@ -91,7 +95,7 @@ func (s *secureSession) NoiseStaticKeyCache() map[peer.ID]([32]byte) { } func (s *secureSession) NoisePrivateKey() [32]byte { - return s.noisePrivateKey + return s.noiseKeypair.private_key } func (s *secureSession) readLength() (int, error) { @@ -135,7 +139,7 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe func (s *secureSession) runHandshake(ctx context.Context) error { // if we have the peer's noise static key and we support noise pipes, we can try IK - if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} && s.noisePipesSupport { + if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} || s.noisePipesSupport { // known static key for peer, try IK // buf, err := s.runHandshake_ik(ctx) @@ -188,9 +192,8 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { func (s *secureSession) Read(buf []byte) (int, error) { l := len(buf) + // if the session has previously if l <= len(s.msgBuffer) { - log.Debug("length < msgbuffer") - copy(buf, s.msgBuffer) s.msgBuffer = s.msgBuffer[l:] return l, nil @@ -220,6 +223,8 @@ func (s *secureSession) Read(buf []byte) (int, error) { c := copy(buf, plaintext) + // if buffer isn't large enough to store the entire message, save the extra message data + // into the session's message buffer if c < len(plaintext) { s.msgBuffer = append(s.msgBuffer, plaintext[len(buf):]...) } diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 1890dcf916..a5e1444f6e 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -2,8 +2,11 @@ package noise import ( "context" + "crypto/rand" "net" + "golang.org/x/crypto/curve25519" + "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" @@ -14,6 +17,19 @@ const ID = "/noise/0.0.1" var _ sec.SecureTransport = &Transport{} +type Keypair struct { + public_key [32]byte + private_key [32]byte +} + +func GenerateKeypair() *Keypair { + var public_key [32]byte + var private_key [32]byte + _, _ = rand.Read(private_key[:]) + curve25519.ScalarBaseMult(&public_key, &private_key) + return &Keypair{public_key, private_key} +} + // Transport implements the interface sec.SecureTransport // https://godoc.org/github.com/libp2p/go-libp2p-core/sec#SecureConn type Transport struct { @@ -21,40 +37,42 @@ type Transport struct { PrivateKey crypto.PrivKey NoisePipesSupport bool NoiseStaticKeyCache map[peer.ID]([32]byte) - NoisePrivateKey [32]byte - NoisePublicKey [32]byte + NoiseKeypair *Keypair } -func NewTransport(localID peer.ID, privkey crypto.PrivKey, noisePipesSupport bool) *Transport { +func NewTransport(localID peer.ID, privkey crypto.PrivKey, noisePipesSupport bool, kp *Keypair) *Transport { + if kp == nil { + kp = GenerateKeypair() + } + return &Transport{ LocalID: localID, PrivateKey: privkey, NoisePipesSupport: noisePipesSupport, + NoiseKeypair: kp, } } // SecureInbound runs noise handshake as the responder func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { - s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, t.NoisePrivateKey, insecure, "", t.NoiseStaticKeyCache, t.NoisePipesSupport, false) + s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, t.NoiseKeypair, insecure, "", t.NoiseStaticKeyCache, t.NoisePipesSupport, false) if err != nil { return s, err } t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() - t.NoisePrivateKey = s.NoisePrivateKey() - t.NoisePublicKey = s.local.noiseKey + t.NoiseKeypair = s.noiseKeypair return s, nil } // SecureOutbound runs noise handshake as the initiator func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { - s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, t.NoisePrivateKey, insecure, p, t.NoiseStaticKeyCache, t.NoisePipesSupport, true) + s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, t.NoiseKeypair, insecure, p, t.NoiseStaticKeyCache, t.NoisePipesSupport, true) if err != nil { return s, err } t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() - t.NoisePrivateKey = s.NoisePrivateKey() - t.NoisePublicKey = s.local.noiseKey + t.NoiseKeypair = s.noiseKeypair return s, nil } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index b185ed978c..35c69fd53a 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -190,7 +190,7 @@ func TestHandshakeIK(t *testing.T) { // add responder's static key to initiator's key cache keycache := make(map[peer.ID]([32]byte)) - keycache[respTransport.LocalID] = respTransport.NoisePublicKey + keycache[respTransport.LocalID] = respTransport.NoiseKeypair.public_key initTransport.NoiseStaticKeyCache = keycache // do IK handshake diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index ea89877bda..3d5adabf31 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -80,17 +80,15 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, // if fallback = true, use msg as initial message in stage 0 func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, initialMsg []byte) (err error) { - var kp xx.Keypair + kp := xx.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) - if s.noisePrivateKey == [32]byte{} { - // generate local static noise key - kp = xx.GenerateKeypair() - s.noisePrivateKey = kp.PrivKey() - log.Debugf("xx generate new noise kp", "initiator", s.initiator) - } else { - pub := xx.GeneratePublicKey(s.noisePrivateKey) - kp = xx.NewKeypair(pub, s.noisePrivateKey) - } + // if s.noiseKeypair == nil { + // // generate local static noise key + // s.noiseKeypair = GenerateKeypair() + // } else { + // pub := xx.GeneratePublicKey(s.noisePrivateKey) + // kp = xx.NewKeypair(pub, s.noisePrivateKey) + // } s.local.noiseKey = kp.PubKey() @@ -241,6 +239,10 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init } else { var msgbuf *xx.MessageBuffer msgbuf, err = xx.Decode1(initialMsg) + if err != nil { + log.Error("xx_recvHandshakeMessage err", err) + return err + } xx_msgbuf := xx.NewMessageBuffer(msgbuf.NE(), nil, nil) log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "modified_msgbuf", xx_msgbuf, "buf len", len(buf)) From 618ebb896a099f1ef789a5dc97bae39754382755 Mon Sep 17 00:00:00 2001 From: noot Date: Mon, 26 Aug 2019 17:27:39 -0400 Subject: [PATCH 1473/3965] fix ik/xx logic, add more tests for ik --- p2p/security/noise/ik_handshake.go | 61 +++++++++++--------------- p2p/security/noise/integration_test.go | 20 +++++---- p2p/security/noise/protocol.go | 46 ++++++++++++++++--- p2p/security/noise/transport.go | 6 +-- p2p/security/noise/xx_handshake.go | 44 ++----------------- 5 files changed, 82 insertions(+), 95 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 01624f2e17..d4a5ffeb6b 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -3,10 +3,10 @@ package noise import ( "context" "fmt" - proto "github.com/gogo/protobuf/proto" ik "github.com/ChainSafe/go-libp2p-noise/ik" pb "github.com/ChainSafe/go-libp2p-noise/pb" + proto "github.com/gogo/protobuf/proto" ) func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bool) error { @@ -79,43 +79,34 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, } // returns last successful message upon error -func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { +func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([]byte, error) { kp := ik.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) - // if s.noiseKeypair == nil { - // // generate local static noise key - // s.noiseKeypair = GenerateKeypair() - // } else { - // kp = GenerateKeypair().(ik.Keypair) - // } - - s.local.noiseKey = kp.PubKey() - log.Debugf("ik handshake", "noise pubkey", kp.PubKey()) - // setup libp2p keys - localKeyRaw, err := s.LocalPublicKey().Bytes() - if err != nil { - return nil, fmt.Errorf("err getting raw pubkey: %s", err) - } + // // setup libp2p keys + // localKeyRaw, err := s.LocalPublicKey().Bytes() + // if err != nil { + // return nil, fmt.Errorf("err getting raw pubkey: %s", err) + // } - log.Debugf("ik handshake", "local libp2p key", localKeyRaw, "len", len(localKeyRaw)) + // log.Debugf("ik handshake", "local libp2p key", localKeyRaw, "len", len(localKeyRaw)) - // sign noise data for payload - noise_pub := kp.PubKey() - signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) - if err != nil { - return nil, fmt.Errorf("err signing payload: %s", err) - } + // // sign noise data for payload + // noise_pub := kp.PubKey() + // signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) + // if err != nil { + // return nil, fmt.Errorf("err signing payload: %s", err) + // } - // create payload - payload := new(pb.NoiseHandshakePayload) - payload.Libp2PKey = localKeyRaw - payload.NoiseStaticKeySignature = signedPayload - payloadEnc, err := proto.Marshal(payload) - if err != nil { - return nil, fmt.Errorf("proto marshal payload fail: %s", err) - } + // // create payload + // payload := new(pb.NoiseHandshakePayload) + // payload.Libp2PKey = localKeyRaw + // payload.NoiseStaticKeySignature = signedPayload + // payloadEnc, err := proto.Marshal(payload) + // if err != nil { + // return nil, fmt.Errorf("proto marshal payload fail: %s", err) + // } // new XX noise session s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, s.noiseStaticKeyCache[s.remotePeer]) @@ -123,7 +114,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { if s.initiator { // stage 0 // - err := s.ik_sendHandshakeMessage(payloadEnc, true) + err := s.ik_sendHandshakeMessage(payload, true) if err != nil { log.Error("stage 0 initiator send", "err", err) return nil, fmt.Errorf("stage 0 initiator fail: %s", err) @@ -171,11 +162,9 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { // stage 0 // log.Debugf("ik responder", "noiseKey", kp.PubKey()) - var buf, plaintext []byte - var valid bool // read message - buf, plaintext, valid, err = s.ik_recvHandshakeMessage(true) + buf, plaintext, valid, err := s.ik_recvHandshakeMessage(true) if err != nil { return buf, fmt.Errorf("stage 0 responder fail: %s", err) } @@ -214,7 +203,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context) ([]byte, error) { // stage 1 // - err := s.ik_sendHandshakeMessage(payloadEnc, false) + err = s.ik_sendHandshakeMessage(payload, false) if err != nil { log.Error("stage 1 responder send", "err", err) return nil, fmt.Errorf("stage 1 responder fail: %s", err) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index a53712a1e5..01698a484a 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -33,7 +33,7 @@ func generateKey(seed int64) (crypto.PrivKey, error) { return priv, nil } -func makeNode(t *testing.T, seed int64, port int) (host.Host, error) { +func makeNode(t *testing.T, seed int64, port int, kp *Keypair) (host.Host, error) { priv, err := generateKey(seed) if err != nil { t.Fatal(err) @@ -44,7 +44,7 @@ func makeNode(t *testing.T, seed int64, port int) (host.Host, error) { t.Fatal(err) } - tpt := NewTransport(pid, priv, false, nil) + tpt := NewTransport(pid, priv, false, kp) ip := "0.0.0.0" addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", ip, port)) @@ -99,7 +99,7 @@ func makeNodePipes(t *testing.T, seed int64, port int, rpid peer.ID, rpubkey [32 func TestLibp2pIntegration_NoPipes(t *testing.T) { ctx := context.Background() - ha, err := makeNode(t, 1, 33333) + ha, err := makeNode(t, 1, 33333, nil) if err != nil { t.Fatal(err) } @@ -108,7 +108,7 @@ func TestLibp2pIntegration_NoPipes(t *testing.T) { //fmt.Printf("ha: %s/p2p/%s\n", ha.Addrs()[1].String(), ha.ID()) - hb, err := makeNode(t, 2, 34343) + hb, err := makeNode(t, 2, 34343, nil) if err != nil { t.Fatal(err) } @@ -209,7 +209,9 @@ func TestLibp2pIntegration_WithPipes(t *testing.T) { func TestLibp2pIntegration_XXFallback(t *testing.T) { ctx := context.Background() - ha, err := makeNodePipes(t, 1, 33333, "", [32]byte{}, nil) + kpa := GenerateKeypair() + + ha, err := makeNode(t, 1, 33333, kpa) if err != nil { t.Fatal(err) } @@ -218,7 +220,7 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { //fmt.Printf("ha: %s/p2p/%s\n", ha.Addrs()[1].String(), ha.ID()) - hb, err := makeNode(t, 2, 34343) + hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.public_key, nil) if err != nil { t.Fatal(err) } @@ -228,7 +230,7 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { ha.SetStreamHandler(ID, handleStream) hb.SetStreamHandler(ID, handleStream) - addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", ha.Addrs()[0].String(), ha.ID())) + addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", hb.Addrs()[0].String(), hb.ID())) if err != nil { t.Fatal(err) } @@ -240,7 +242,7 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { t.Fatal(err) } - err = hb.Connect(ctx, *addrInfo) + err = ha.Connect(ctx, *addrInfo) if err != nil { t.Fatal(err) } @@ -273,5 +275,5 @@ func handleStream(stream net.Stream) { fmt.Println("stream err", err) return } - fmt.Println("got msg:", msg) + fmt.Printf("got msg: %s", msg) } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 6ba67ac17b..01a328997f 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -8,6 +8,7 @@ import ( "sync" "time" + proto "github.com/gogo/protobuf/proto" logging "github.com/ipfs/go-log" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" @@ -72,6 +73,11 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey kp = GenerateKeypair() } + localPeerInfo := peerInfo{ + noiseKey: kp.public_key, + libp2pKey: privKey.GetPublic(), + } + s := &secureSession{ insecure: insecure, initiator: initiator, @@ -79,10 +85,11 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey localKey: privKey, localPeer: local, remotePeer: remote, + local: localPeerInfo, noisePipesSupport: noisePipesSupport, noiseStaticKeyCache: noiseStaticKeyCache, msgBuffer: []byte{}, - noiseKeypair: kp, + noiseKeypair: kp, } err := s.runHandshake(ctx) @@ -138,16 +145,43 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe } func (s *secureSession) runHandshake(ctx context.Context) error { + + // setup libp2p keys + localKeyRaw, err := s.LocalPublicKey().Bytes() + if err != nil { + return fmt.Errorf("err getting raw pubkey: %s", err) + } + + log.Debugf("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) + + // sign noise data for payload + noise_pub := s.noiseKeypair.public_key + signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) + if err != nil { + log.Error("xx handshake signing payload", "err", err) + return fmt.Errorf("err signing payload: %s", err) + } + + // create payload + payload := new(pb.NoiseHandshakePayload) + payload.Libp2PKey = localKeyRaw + payload.NoiseStaticKeySignature = signedPayload + payloadEnc, err := proto.Marshal(payload) + if err != nil { + log.Error("xx handshake marshal payload", "err", err) + return fmt.Errorf("proto marshal payload fail: %s", err) + } + // if we have the peer's noise static key and we support noise pipes, we can try IK - if s.noiseStaticKeyCache[s.remotePeer] != [32]byte{} || s.noisePipesSupport { + if (!s.initiator && s.noiseStaticKeyCache[s.remotePeer] != [32]byte{}) && s.noisePipesSupport { // known static key for peer, try IK // - buf, err := s.runHandshake_ik(ctx) + buf, err := s.runHandshake_ik(ctx, payloadEnc) if err != nil { log.Error("runHandshake_ik", "err", err) // IK failed, pipe to XXfallback - err = s.runHandshake_xx(ctx, true, buf) + err = s.runHandshake_xx(ctx, true, payloadEnc, buf) if err != nil { log.Error("runHandshake_xx", "err", err) return fmt.Errorf("runHandshake_xx err %s", err) @@ -161,7 +195,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } else { // unknown static key for peer, try XX // - err := s.runHandshake_xx(ctx, false, nil) + err := s.runHandshake_xx(ctx, false, payloadEnc, nil) if err != nil { log.Error("runHandshake_xx", "err", err) return err @@ -192,7 +226,7 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { func (s *secureSession) Read(buf []byte) (int, error) { l := len(buf) - // if the session has previously + // if the session has previously if l <= len(s.msgBuffer) { copy(buf, s.msgBuffer) s.msgBuffer = s.msgBuffer[l:] diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index a5e1444f6e..2f0dbb94a2 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -18,7 +18,7 @@ const ID = "/noise/0.0.1" var _ sec.SecureTransport = &Transport{} type Keypair struct { - public_key [32]byte + public_key [32]byte private_key [32]byte } @@ -37,7 +37,7 @@ type Transport struct { PrivateKey crypto.PrivKey NoisePipesSupport bool NoiseStaticKeyCache map[peer.ID]([32]byte) - NoiseKeypair *Keypair + NoiseKeypair *Keypair } func NewTransport(localID peer.ID, privkey crypto.PrivKey, noisePipesSupport bool, kp *Keypair) *Transport { @@ -49,7 +49,7 @@ func NewTransport(localID peer.ID, privkey crypto.PrivKey, noisePipesSupport boo LocalID: localID, PrivateKey: privkey, NoisePipesSupport: noisePipesSupport, - NoiseKeypair: kp, + NoiseKeypair: kp, } } diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 3d5adabf31..75d2975fb7 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -79,49 +79,11 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, } // if fallback = true, use msg as initial message in stage 0 -func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, initialMsg []byte) (err error) { +func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payload []byte, initialMsg []byte) (err error) { kp := xx.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) - // if s.noiseKeypair == nil { - // // generate local static noise key - // s.noiseKeypair = GenerateKeypair() - // } else { - // pub := xx.GeneratePublicKey(s.noisePrivateKey) - // kp = xx.NewKeypair(pub, s.noisePrivateKey) - // } - - s.local.noiseKey = kp.PubKey() - log.Debugf("xx handshake", "pubkey", kp.PubKey(), "initiator", s.initiator) - // setup libp2p keys - localKeyRaw, err := s.LocalPublicKey().Bytes() - if err != nil { - return fmt.Errorf("err getting raw pubkey: %s", err) - } - - log.Debugf("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) - - // sign noise data for payload - noise_pub := kp.PubKey() - signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) - if err != nil { - log.Error("xx handshake signing payload", "err", err) - return fmt.Errorf("err signing payload: %s", err) - } - - s.local.noiseKey = noise_pub - - // create payload - payload := new(pb.NoiseHandshakePayload) - payload.Libp2PKey = localKeyRaw - payload.NoiseStaticKeySignature = signedPayload - payloadEnc, err := proto.Marshal(payload) - if err != nil { - log.Error("xx handshake marshal payload", "err", err) - return fmt.Errorf("proto marshal payload fail: %s", err) - } - // new XX noise session s.xx_ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) @@ -180,7 +142,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init // stage 2 // - err = s.xx_sendHandshakeMessage(payloadEnc, false) + err = s.xx_sendHandshakeMessage(payload, false) if err != nil { return fmt.Errorf("stage 2 intiator fail: %s", err) } @@ -263,7 +225,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, init // stage 1 // - err = s.xx_sendHandshakeMessage(payloadEnc, false) + err = s.xx_sendHandshakeMessage(payload, false) if err != nil { return fmt.Errorf("stage 1 responder fail: %s", err) } From edc89c22107b85558da7fa87922eab031226f1ce Mon Sep 17 00:00:00 2001 From: noot Date: Mon, 26 Aug 2019 18:18:07 -0400 Subject: [PATCH 1474/3965] improve log formatting --- p2p/security/noise/ik_handshake.go | 101 +++++++++--------------- p2p/security/noise/protocol.go | 61 +++++++-------- p2p/security/noise/transport.go | 4 + p2p/security/noise/xx_handshake.go | 120 ++++++++++++++--------------- 4 files changed, 125 insertions(+), 161 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index d4a5ffeb6b..bcac908584 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -10,9 +10,6 @@ import ( ) func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bool) error { - log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload)) - - // create send message w payload var msgbuf ik.MessageBuffer s.ik_ns, msgbuf = ik.SendMessage(s.ik_ns, payload) var encMsgBuf []byte @@ -22,19 +19,19 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo encMsgBuf = msgbuf.Encode1() } - log.Debugf("ik_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf) + log.Debugf("ik_sendHandshakeMessage initiator=%v msgbuf=%v", s.initiator, msgbuf) err := s.writeLength(len(encMsgBuf)) if err != nil { - log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) - return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) + log.Error("ik_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) + return fmt.Errorf("ik_sendHandshakeMessage write length err=%s", err) } // send message _, err = s.insecure.Write(encMsgBuf) if err != nil { - log.Error("ik_sendHandshakeMessage", "initiator", s.initiator, "error", err) - return fmt.Errorf("write to conn fail: %s", err) + log.Error("ik_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) + return fmt.Errorf("ik_sendHandshakeMessage write to conn err=%s", err) } return nil @@ -43,14 +40,14 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { l, err := s.readLength() if err != nil { - return nil, nil, false, fmt.Errorf("read length fail: %s", err) + return nil, nil, false, fmt.Errorf("ik_recvHandshakeMessage read length err=%s", err) } buf = make([]byte, l) _, err = s.insecure.Read(buf) if err != nil { - return buf, nil, false, fmt.Errorf("read from conn fail: %s", err) + return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage read from conn err=%s", err) } var msgbuf *ik.MessageBuffer @@ -60,64 +57,42 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, msgbuf, err = ik.Decode1(buf) } - log.Debugf("ik_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(buf)) + log.Debugf("ik_recvHandshakeMessage initiator=%v msgbuf=%v", s.initiator, msgbuf) if err != nil { - log.Error("ik_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) - return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) + log.Error("ik_recvHandshakeMessage initiator=%v decode err=%s", s.initiator, err) + return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage decode msg fail: %s", err) } s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) if !valid { - log.Error("ik_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") - return buf, nil, false, fmt.Errorf("validation fail") + log.Error("ik_recvHandshakeMessage initiator=%v err=%s", s.initiator, "validation fail") + return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage validation fail") } - log.Debugf("recv handshake message", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) - return buf, plaintext, valid, nil } +// IK: +// <- s +// ... +// -> e, es, s, ss +// <- e, ee, se // returns last successful message upon error func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([]byte, error) { kp := ik.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) - log.Debugf("ik handshake", "noise pubkey", kp.PubKey()) - - // // setup libp2p keys - // localKeyRaw, err := s.LocalPublicKey().Bytes() - // if err != nil { - // return nil, fmt.Errorf("err getting raw pubkey: %s", err) - // } - - // log.Debugf("ik handshake", "local libp2p key", localKeyRaw, "len", len(localKeyRaw)) - - // // sign noise data for payload - // noise_pub := kp.PubKey() - // signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) - // if err != nil { - // return nil, fmt.Errorf("err signing payload: %s", err) - // } - - // // create payload - // payload := new(pb.NoiseHandshakePayload) - // payload.Libp2PKey = localKeyRaw - // payload.NoiseStaticKeySignature = signedPayload - // payloadEnc, err := proto.Marshal(payload) - // if err != nil { - // return nil, fmt.Errorf("proto marshal payload fail: %s", err) - // } + log.Debugf("runHandshake_ik initiator=%s pubkey=%x", kp.PubKey(), s.initiator) // new XX noise session s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, s.noiseStaticKeyCache[s.remotePeer]) - log.Debugf("ik initiator init session", "remotePeer", s.noiseStaticKeyCache[s.remotePeer]) if s.initiator { // stage 0 // err := s.ik_sendHandshakeMessage(payload, true) if err != nil { - log.Error("stage 0 initiator send", "err", err) - return nil, fmt.Errorf("stage 0 initiator fail: %s", err) + log.Errorf("runHandshake_ik stage=0 initiator=true send err=%s", err) + return nil, fmt.Errorf("runHandshake_ik stage=0 initiator=true err=%s", err) } // stage 1 // @@ -125,92 +100,86 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] // read message buf, plaintext, valid, err := s.ik_recvHandshakeMessage(false) if err != nil { - return buf, fmt.Errorf("stage 1 initiator fail: %s", err) + return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=%s", err) } if !valid { - return buf, fmt.Errorf("stage 1 initiator validation fail") + return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=validation fail") } // unmarshal payload nhp := new(pb.NoiseHandshakePayload) err = proto.Unmarshal(plaintext, nhp) if err != nil { - return buf, fmt.Errorf("stage 1 initiator validation fail: cannot unmarshal payload") + return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=validation fail: cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { - log.Error("stage 1 initiator set remote peer info", "err", err) - return buf, fmt.Errorf("stage 1 initiator read remote libp2p key fail") + log.Errorf("runHandshake_ik stage=1 initiator=true set remote peer info err=%s", err) + return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=read remote libp2p key fail") } // assert that remote peer ID matches libp2p key err = s.setRemotePeerID(s.RemotePublicKey()) if err != nil { - log.Error("stage 1 initiator set remote peer id", "err", err) + log.Errorf("runHandshake_ik stage=1 initiator=true set remote peer id err=%s", err) } // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.noiseStaticKeyCache[s.remotePeer]) if err != nil { - log.Error("stage 1 initiator verify payload", "err", err) + log.Errorf("runHandshake_ik stage=1 initiator=true verify payload err=%s", err) } } else { // stage 0 // - log.Debugf("ik responder", "noiseKey", kp.PubKey()) - // read message buf, plaintext, valid, err := s.ik_recvHandshakeMessage(true) if err != nil { - return buf, fmt.Errorf("stage 0 responder fail: %s", err) + return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err=%s", err) } if !valid { - return buf, fmt.Errorf("stage 0 responder validation fail") + return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err: validation fail") } - log.Debugf("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) - // unmarshal payload nhp := new(pb.NoiseHandshakePayload) err = proto.Unmarshal(plaintext, nhp) if err != nil { - return buf, fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err=validation fail: cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { - log.Error("stage 0 responder set remote peer info", "err", err) - return buf, fmt.Errorf("stage 0 responder read remote libp2p key fail") + return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err=read remote libp2p key fail") } // assert that remote peer ID matches libp2p key err = s.setRemotePeerID(s.RemotePublicKey()) if err != nil { - log.Error("stage 0 responder set remote peer id", "err", err) + return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false set remote peer id err=%s:", err) } // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.ik_ns.RemoteKey()) if err != nil { - log.Error("stage 1 responder verify payload", "err", err) + return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false verify payload err=%s", err) } // stage 1 // err = s.ik_sendHandshakeMessage(payload, false) if err != nil { - log.Error("stage 1 responder send", "err", err) - return nil, fmt.Errorf("stage 1 responder fail: %s", err) + return nil, fmt.Errorf("runHandshake_ik stage=1 initiator=false send err=%s", err) } } - log.Debugf("ik_handshake done", "initiator", s.initiator) + log.Debugf("runHandshake_ik done initiator=%v", s.initiator) return nil, nil } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 01a328997f..259016a083 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -18,10 +18,10 @@ import ( xx "github.com/ChainSafe/go-libp2p-noise/xx" ) -var log = logging.Logger("noise") - const payload_string = "noise-libp2p-static-key:" +var log = logging.Logger("noise") + type secureSession struct { insecure net.Conn @@ -101,6 +101,10 @@ func (s *secureSession) NoiseStaticKeyCache() map[peer.ID]([32]byte) { return s.noiseStaticKeyCache } +func (s *secureSession) NoisePublicKey() [32]byte { + return s.noiseKeypair.public_key +} + func (s *secureSession) NoisePrivateKey() [32]byte { return s.noiseKeypair.private_key } @@ -132,7 +136,7 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe sig := payload.GetNoiseStaticKeySignature() msg := append([]byte(payload_string), noiseKey[:]...) - log.Debugf("verifyPayload", "msg", fmt.Sprintf("%x", msg)) + log.Debugf("verifyPayload msg=%x", msg) ok, err := s.RemotePublicKey().Verify(msg, sig) if err != nil { @@ -145,21 +149,18 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe } func (s *secureSession) runHandshake(ctx context.Context) error { - // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() if err != nil { - return fmt.Errorf("err getting raw pubkey: %s", err) + return fmt.Errorf("runHandshake err getting raw pubkey: %s", err) } - log.Debugf("xx handshake", "local key", localKeyRaw, "len", len(localKeyRaw)) - // sign noise data for payload noise_pub := s.noiseKeypair.public_key signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) if err != nil { - log.Error("xx handshake signing payload", "err", err) - return fmt.Errorf("err signing payload: %s", err) + log.Errorf("runHandshake signing payload err=%s", err) + return fmt.Errorf("runHandshake signing payload err=%s", err) } // create payload @@ -168,23 +169,24 @@ func (s *secureSession) runHandshake(ctx context.Context) error { payload.NoiseStaticKeySignature = signedPayload payloadEnc, err := proto.Marshal(payload) if err != nil { - log.Error("xx handshake marshal payload", "err", err) - return fmt.Errorf("proto marshal payload fail: %s", err) + log.Errorf("runHandshake marshal payload err=%s", err) + return fmt.Errorf("runHandshake proto marshal payload err=%s", err) } - // if we have the peer's noise static key and we support noise pipes, we can try IK + // if we have the peer's noise static key (and we're the initiator,) and we support noise pipes, + // we can try IK first. if we support noise pipes and we're not the initiator, try IK first + // otherwise, default to XX if (!s.initiator && s.noiseStaticKeyCache[s.remotePeer] != [32]byte{}) && s.noisePipesSupport { - // known static key for peer, try IK // - + // known static key for peer, try IK buf, err := s.runHandshake_ik(ctx, payloadEnc) if err != nil { - log.Error("runHandshake_ik", "err", err) + log.Error("runHandshake ik err=%s", err) // IK failed, pipe to XXfallback err = s.runHandshake_xx(ctx, true, payloadEnc, buf) if err != nil { - log.Error("runHandshake_xx", "err", err) - return fmt.Errorf("runHandshake_xx err %s", err) + log.Error("runHandshake xx err=err", err) + return fmt.Errorf("runHandshake xx err=%s", err) } s.xx_complete = true @@ -193,11 +195,10 @@ func (s *secureSession) runHandshake(ctx context.Context) error { s.ik_complete = true } else { - // unknown static key for peer, try XX // - + // unknown static key for peer, try XX err := s.runHandshake_xx(ctx, false, payloadEnc, nil) if err != nil { - log.Error("runHandshake_xx", "err", err) + log.Error("runHandshake xx err=%s", err) return err } @@ -226,42 +227,38 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { func (s *secureSession) Read(buf []byte) (int, error) { l := len(buf) - // if the session has previously + // if we have previously unread bytes, and they fit into the buf, copy them over and return if l <= len(s.msgBuffer) { copy(buf, s.msgBuffer) s.msgBuffer = s.msgBuffer[l:] return l, nil } + // read length of encrypted message l, err := s.readLength() - if err != nil { return 0, err } + // read and decrypt ciphertext ciphertext := make([]byte, l) - _, err = s.insecure.Read(ciphertext) - if err != nil { log.Error("read ciphertext err", err) return 0, err } plaintext, err := s.Decrypt(ciphertext) - if err != nil { log.Error("decrypt err", err) return 0, err } - c := copy(buf, plaintext) - - // if buffer isn't large enough to store the entire message, save the extra message data - // into the session's message buffer - if c < len(plaintext) { - s.msgBuffer = append(s.msgBuffer, plaintext[len(buf):]...) - } + // append plaintext to message buffer, copy over what can fit in the buf + // then advance message buffer to remove what was copied + s.msgBuffer = append(s.msgBuffer, plaintext...) + c := copy(buf, s.msgBuffer) + s.msgBuffer = s.msgBuffer[c:] return c, nil } diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 2f0dbb94a2..730395f16e 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -17,11 +17,13 @@ const ID = "/noise/0.0.1" var _ sec.SecureTransport = &Transport{} +// Keypair is a noise ed25519 public-private keypair type Keypair struct { public_key [32]byte private_key [32]byte } +// GenerateKeypair creates a new ed25519 keypair func GenerateKeypair() *Keypair { var public_key [32]byte var private_key [32]byte @@ -40,6 +42,8 @@ type Transport struct { NoiseKeypair *Keypair } +// NewTransport creates a new noise transport and can be configured to use noise pipes and a given +// noise ed25519 keypair func NewTransport(localID peer.ID, privkey crypto.PrivKey, noisePipesSupport bool, kp *Keypair) *Transport { if kp == nil { kp = GenerateKeypair() diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 75d2975fb7..662e31b388 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -12,9 +12,6 @@ import ( ) func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { - log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "payload", payload, "payload len", len(payload), "initial_stage", initial_stage) - - // create send message w payload var msgbuf xx.MessageBuffer s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, payload, nil) var encMsgBuf []byte @@ -24,35 +21,34 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo encMsgBuf = msgbuf.Encode1() } - log.Debugf("xx_sendHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "initial_stage", initial_stage) - err := s.writeLength(len(encMsgBuf)) if err != nil { - log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) - return fmt.Errorf("xx_sendHandshakeMessage write length fail: %s", err) + log.Error("xx_sendHandshakeMessage initiator=%sverr=%s", s.initiator, err) + return fmt.Errorf("xx_sendHandshakeMessage write length err=%s", err) } - // send message _, err = s.insecure.Write(encMsgBuf) if err != nil { - log.Error("xx_sendHandshakeMessage", "initiator", s.initiator, "error", err) - return fmt.Errorf("xx_sendHandshakeMessage write to conn fail: %s", err) + log.Error("xx_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) + return fmt.Errorf("xx_sendHandshakeMessage write to conn err=%s", err) } + log.Debugf("xx_sendHandshakeMessage initiator=%v msgbuf=%v initial_stage=%v", s.initiator, msgbuf, initial_stage) + return nil } func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { l, err := s.readLength() if err != nil { - return nil, nil, false, fmt.Errorf("read length fail: %s", err) + return nil, nil, false, fmt.Errorf("xx_recvHandshakeMessage read length err=%s", err) } buf = make([]byte, l) _, err = s.insecure.Read(buf) if err != nil { - return buf, nil, false, fmt.Errorf("read from conn fail: %s", err) + return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage read from conn err=%s", err) } var msgbuf *xx.MessageBuffer @@ -63,26 +59,32 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, } if err != nil { - log.Debugf("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) - return buf, nil, false, fmt.Errorf("decode msg fail: %s", err) + log.Debugf("xx_recvHandshakeMessage initiator=%v decode err=%s", s.initiator, err) + return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage decode msg err=%s", err) } s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) if !valid { - log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") - return buf, nil, false, fmt.Errorf("validation fail") + log.Error("xx_recvHandshakeMessage initiator=%v err=validation fail", s.initiator) + return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage validation fail") } - log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "payload len", len(plaintext)) + log.Debugf("xx_recvHandshakeMessage initiator=%v msgbuf=%v initial_stage=%v", s.initiator, msgbuf, initial_stage) return buf, plaintext, valid, nil } -// if fallback = true, use msg as initial message in stage 0 +// Runs the XX handshake +// XX: +// -> e +// <- e, ee, s, es +// -> s, se +// if fallback = true, initialMsg is used as the message in stage 1 of the initiator and stage 0 +// of the responder func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payload []byte, initialMsg []byte) (err error) { kp := xx.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) - log.Debugf("xx handshake", "pubkey", kp.PubKey(), "initiator", s.initiator) + log.Debugf("runHandshake_xx initiator=%s fallback=%s pubkey=%x", s.initiator, fallback, kp.PubKey()) // new XX noise session s.xx_ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) @@ -93,18 +95,18 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl if !fallback { err = s.xx_sendHandshakeMessage(nil, true) if err != nil { - return fmt.Errorf("stage 0 initiator fail: %s", err) + return fmt.Errorf("runHandshake_xx stage 0 initiator fail: %s", err) } } else { e_ik := s.ik_ns.Ephemeral() - log.Debugf("xxfallback stage 0 initiator", "ephemeral keys from ik", e_ik) + log.Debugf("runHandshake_xx stage=0 initiator=true fallback=true ephemeralkeys=%x", e_ik) e_xx := xx.NewKeypair(e_ik.PubKey(), e_ik.PrivKey()) // initialize state as if we sent the first message var msgbuf xx.MessageBuffer s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, nil, &e_xx) - log.Debugf("stage 0 initiator xx", "msgbuf", msgbuf) + log.Debugf("runHandshake_xx stage=0 initiator=true fallback=true msgbuf=%v", msgbuf) } // stage 1 // @@ -115,27 +117,27 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl // read reply _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) if err != nil { - return fmt.Errorf("initiator stage 1 fail: %s", err) + return fmt.Errorf("runHandshake_xx initiator stage 1 fail: %s", err) } if !valid { - return fmt.Errorf("stage 1 initiator validation fail") + return fmt.Errorf("runHandshake_xx stage 1 initiator validation fail") } } else { var msgbuf *xx.MessageBuffer msgbuf, err = xx.Decode1(initialMsg) - log.Debugf("stage 1 xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "buf len", len(initialMsg)) + log.Debugf("xx_recvHandshakeMessage stage=1 initiator=%s msgbuf=%v", s.initiator, msgbuf) if err != nil { - log.Debugf("stage 1 xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) - return fmt.Errorf("decode msg fail: %s", err) + log.Debugf("xx_recvHandshakeMessage stage=1 initiator=%s decode_err=%s", s.initiator, err) + return fmt.Errorf("runHandshake_xx decode msg fail: %s", err) } s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) if !valid { - log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") - return fmt.Errorf("validation fail") + log.Errorf("xx_recvHandshakeMessage initiator=%s", s.initiator, "error", "validation fail") + return fmt.Errorf("runHandshake_xx validation fail") } } @@ -144,35 +146,35 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl err = s.xx_sendHandshakeMessage(payload, false) if err != nil { - return fmt.Errorf("stage 2 intiator fail: %s", err) + return fmt.Errorf("runHandshake_xx stage=2 initiator=true err=%s", err) } // unmarshal payload nhp := new(pb.NoiseHandshakePayload) err = proto.Unmarshal(plaintext, nhp) if err != nil { - return fmt.Errorf("stage 2 initiator validation fail: cannot unmarshal payload") + return fmt.Errorf("runHandshake_xx stage=2 initiator=true err=cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { - log.Error("stage 2 initiator set remote peer info", "err", err) - return fmt.Errorf("stage 2 initiator read remote libp2p key fail") + log.Errorf("runHandshake_xx stage=2 initiator=true set remote peer info err=%s", err) + return fmt.Errorf("runHandshake_xx stage=2 initiator=true read remote libp2p key fail") } // assert that remote peer ID matches libp2p public key pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) if pid != s.remotePeer { - log.Error("stage 2 initiator check remote peer id err", "expected", s.remotePeer, "got", pid) + log.Errorf("runHandshake_xx stage=2 initiator=true check remote peer id err: expected %x got %x", s.remotePeer, pid) } else if err != nil { - log.Error("stage 2 initiator check remote peer id", "err", err) + log.Errorf("runHandshake_xx stage 2 initiator check remote peer id err %s", err) } // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.xx_ns.RemoteKey()) if err != nil { - log.Error("stage 2 initiator verify payload", "err", err) + log.Errorf("runHandshake_xx stage=2 initiator=true verify payload err=%s", err) } if s.noisePipesSupport { @@ -183,7 +185,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl // stage 0 // - var buf, plaintext []byte + var plaintext []byte var valid bool nhp := new(pb.NoiseHandshakePayload) @@ -191,43 +193,36 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl // read message _, plaintext, valid, err = s.xx_recvHandshakeMessage(true) if err != nil { - return fmt.Errorf("stage 0 responder fail: %s", err) + return fmt.Errorf("runHandshake_xx stage=0 initiator=false err=%s", err) } if !valid { - return fmt.Errorf("stage 0 responder validation fail") + return fmt.Errorf("runHandshake_xx stage=0 initiator=false err=validation fail") } } else { var msgbuf *xx.MessageBuffer msgbuf, err = xx.Decode1(initialMsg) if err != nil { - log.Error("xx_recvHandshakeMessage err", err) + log.Errorf("runHandshake_xx recv msg err", err) return err } - xx_msgbuf := xx.NewMessageBuffer(msgbuf.NE(), nil, nil) - log.Debugf("xx_recvHandshakeMessage", "initiator", s.initiator, "msgbuf", msgbuf, "modified_msgbuf", xx_msgbuf, "buf len", len(buf)) - - if err != nil { - log.Debugf("xx_recvHandshakeMessage decode", "initiator", s.initiator, "error", err) - return fmt.Errorf("decode msg fail: %s", err) - } + xx_msgbuf := xx.NewMessageBuffer(msgbuf.NE(), nil, nil) + log.Debugf("runHandshake_xx initiator=false msgbuf=%v modified_msgbuf=%v", msgbuf, xx_msgbuf) s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, &xx_msgbuf) if !valid { - log.Error("xx_recvHandshakeMessage", "initiator", s.initiator, "error", "validation fail") - return fmt.Errorf("validation fail") + log.Errorf("runHandshake_xx initiator=false recv msg err=%s", "validation fail") + return fmt.Errorf("runHandshake_xx validation fail") } } - log.Debugf("stage 0 responder", "plaintext", plaintext, "plaintext len", len(plaintext)) - // stage 1 // err = s.xx_sendHandshakeMessage(payload, false) if err != nil { - return fmt.Errorf("stage 1 responder fail: %s", err) + return fmt.Errorf("runHandshake_xx stage=1 initiator=false err=%s", err) } // stage 2 // @@ -235,32 +230,32 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl // read message _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) if err != nil { - return fmt.Errorf("stage 2 responder fail: %s", err) + return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=%s", err) } if !valid { - return fmt.Errorf("stage 2 responder validation fail") + return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=validation fail") } - log.Debugf("stage 2 responder", "plaintext", plaintext, "remote key", s.xx_ns.RemoteKey()) + log.Debugf("runHandshake_xx stage=2 initiator=false remote key=%x", s.xx_ns.RemoteKey()) // unmarshal payload err = proto.Unmarshal(plaintext, nhp) if err != nil { - return fmt.Errorf("stage 0 responder validation fail: cannot unmarshal payload") + return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=cannot unmarshal payload") } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) if err != nil { - log.Error("stage 0 responder set remote peer info", "err", err) - return fmt.Errorf("stage 0 responder read remote libp2p key fail") + log.Errorf("runHandshake_xx stage=2 initiator=false set remote peer info err=%s", err) + return fmt.Errorf("runHandshake_xx stage=2 initiator=false read remote libp2p key fail") } // assert that remote peer ID matches libp2p key err = s.setRemotePeerID(s.RemotePublicKey()) if err != nil { - log.Error("stage 0 responder set remote peer id", "err", err) + log.Errorf("runHandshake_xx stage=2 initiator=false set remote peer id err=%s", err) } s.remote.noiseKey = s.xx_ns.RemoteKey() @@ -268,16 +263,15 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.remote.noiseKey) if err != nil { - log.Error("stage 2 responder verify payload", "err", err) - return fmt.Errorf("stage 2 responder fail: %s", err) + log.Errorf("runHandshake_xx stage=2 initiator=false verify payload err=%s", err) + return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=%s", err) } if s.noisePipesSupport { s.noiseStaticKeyCache[s.remotePeer] = s.remote.noiseKey } - log.Debugf("stage 2 responder", "remote key", s.remote.noiseKey) } - log.Debugf("xx_handshake done", "initiator", s.initiator) + log.Debugf("runHandshake_xx done initiator=%v", s.initiator) return nil } From ba7b4e547c82e597e5a5c690149016a9c1bd11d8 Mon Sep 17 00:00:00 2001 From: noot Date: Tue, 27 Aug 2019 03:47:07 -0400 Subject: [PATCH 1475/3965] improve log formatting --- p2p/security/noise/ik_handshake.go | 2 +- p2p/security/noise/integration_test.go | 6 ------ p2p/security/noise/xx_handshake.go | 10 +++++----- 3 files changed, 6 insertions(+), 12 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index bcac908584..d05105e5a0 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -82,7 +82,7 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([]byte, error) { kp := ik.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) - log.Debugf("runHandshake_ik initiator=%s pubkey=%x", kp.PubKey(), s.initiator) + log.Debugf("runHandshake_ik initiator=%v pubkey=%x", kp.PubKey(), s.initiator) // new XX noise session s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, s.noiseStaticKeyCache[s.remotePeer]) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index 01698a484a..21ffb4fabd 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -106,8 +106,6 @@ func TestLibp2pIntegration_NoPipes(t *testing.T) { defer ha.Close() - //fmt.Printf("ha: %s/p2p/%s\n", ha.Addrs()[1].String(), ha.ID()) - hb, err := makeNode(t, 2, 34343, nil) if err != nil { t.Fatal(err) @@ -162,8 +160,6 @@ func TestLibp2pIntegration_WithPipes(t *testing.T) { defer ha.Close() - //fmt.Printf("ha: %s/p2p/%s\n", ha.Addrs()[1].String(), ha.ID()) - hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.public_key, nil) if err != nil { t.Fatal(err) @@ -218,8 +214,6 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { defer ha.Close() - //fmt.Printf("ha: %s/p2p/%s\n", ha.Addrs()[1].String(), ha.ID()) - hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.public_key, nil) if err != nil { t.Fatal(err) diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 662e31b388..5eff33fa44 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -23,7 +23,7 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo err := s.writeLength(len(encMsgBuf)) if err != nil { - log.Error("xx_sendHandshakeMessage initiator=%sverr=%s", s.initiator, err) + log.Error("xx_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("xx_sendHandshakeMessage write length err=%s", err) } @@ -84,7 +84,7 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payload []byte, initialMsg []byte) (err error) { kp := xx.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) - log.Debugf("runHandshake_xx initiator=%s fallback=%s pubkey=%x", s.initiator, fallback, kp.PubKey()) + log.Debugf("runHandshake_xx initiator=%v fallback=%v pubkey=%x", s.initiator, fallback, kp.PubKey()) // new XX noise session s.xx_ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) @@ -127,16 +127,16 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl var msgbuf *xx.MessageBuffer msgbuf, err = xx.Decode1(initialMsg) - log.Debugf("xx_recvHandshakeMessage stage=1 initiator=%s msgbuf=%v", s.initiator, msgbuf) + log.Debugf("xx_recvHandshakeMessage stage=1 initiator=%v msgbuf=%v", s.initiator, msgbuf) if err != nil { - log.Debugf("xx_recvHandshakeMessage stage=1 initiator=%s decode_err=%s", s.initiator, err) + log.Debugf("xx_recvHandshakeMessage stage=1 initiator=%v decode_err=%s", s.initiator, err) return fmt.Errorf("runHandshake_xx decode msg fail: %s", err) } s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) if !valid { - log.Errorf("xx_recvHandshakeMessage initiator=%s", s.initiator, "error", "validation fail") + log.Errorf("xx_recvHandshakeMessage initiator=%v", s.initiator, "error", "validation fail") return fmt.Errorf("runHandshake_xx validation fail") } From 9cf6af474bc2aee09efcc37b39e59a58b4c597e5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 28 Aug 2019 11:50:26 +0100 Subject: [PATCH 1476/3965] add godocs. --- config/config.go | 4 ++++ p2p/protocol/identify/opts.go | 2 ++ 2 files changed, 6 insertions(+) diff --git a/config/config.go b/config/config.go index c37dbc61c6..782ac10c74 100644 --- a/config/config.go +++ b/config/config.go @@ -44,6 +44,10 @@ type RoutingC func(host.Host) (routing.PeerRouting, error) // This is *not* a stable interface. Use the options defined in the root // package. type Config struct { + // UserAgent is the identifier this node will send to other peers when + // identifying itself, e.g. via the identify protocol. + // + // Set it via the UserAgent option function. UserAgent string PeerKey crypto.PrivKey diff --git a/p2p/protocol/identify/opts.go b/p2p/protocol/identify/opts.go index 94be5b786c..670b213b71 100644 --- a/p2p/protocol/identify/opts.go +++ b/p2p/protocol/identify/opts.go @@ -4,8 +4,10 @@ type config struct { userAgent string } +// Option is an option function for identify. type Option func(*config) +// UserAgent sets the user agent this node will identify itself with to peers. func UserAgent(ua string) Option { return func(cfg *config) { cfg.userAgent = ua From 8c4b2e69115f54e904857c52a3f729b371e7080e Mon Sep 17 00:00:00 2001 From: Cole Brown Date: Wed, 28 Aug 2019 11:34:22 -0400 Subject: [PATCH 1477/3965] Ensure all tests pass without weak RSA key flag --- core/crypto/key_test.go | 8 ++++-- core/crypto/rsa_test.go | 6 ++--- core/peer/peer_test.go | 40 ++++++++++++++++++++---------- core/sec/insecure/insecure_test.go | 2 +- 4 files changed, 37 insertions(+), 19 deletions(-) diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index 013427456f..05e66ed301 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -17,7 +17,11 @@ func TestKeys(t *testing.T) { } func testKeyType(typ int, t *testing.T) { - sk, pk, err := test.RandTestKeyPair(typ, 512) + bits := 512 + if typ == RSA { + bits = 2048 + } + sk, pk, err := test.RandTestKeyPair(typ, bits) if err != nil { t.Fatal(err) } @@ -115,7 +119,7 @@ func testKeyEquals(t *testing.T, k Key) { // t.Fatal("Key not equal to key with same bytes.") // } - sk, pk, err := test.RandTestKeyPair(RSA, 512) + sk, pk, err := test.RandTestKeyPair(RSA, 2048) if err != nil { t.Fatal(err) } diff --git a/core/crypto/rsa_test.go b/core/crypto/rsa_test.go index 7ee520acb7..08db1366ad 100644 --- a/core/crypto/rsa_test.go +++ b/core/crypto/rsa_test.go @@ -6,7 +6,7 @@ import ( ) func TestRSABasicSignAndVerify(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) if err != nil { t.Fatal(err) } @@ -47,7 +47,7 @@ func TestRSASmallKey(t *testing.T) { } func TestRSASignZero(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) if err != nil { t.Fatal(err) } @@ -68,7 +68,7 @@ func TestRSASignZero(t *testing.T) { } func TestRSAMarshalLoop(t *testing.T) { - priv, pub, err := GenerateRSAKeyPair(512, rand.Reader) + priv, pub, err := GenerateRSAKeyPair(2048, rand.Reader) if err != nil { t.Fatal(err) } diff --git a/core/peer/peer_test.go b/core/peer/peer_test.go index 41f323d31e..757f085c15 100644 --- a/core/peer/peer_test.go +++ b/core/peer/peer_test.go @@ -47,7 +47,7 @@ type keyset struct { func (ks *keyset) generate() error { var err error - ks.sk, ks.pk, err = test.RandTestKeyPair(ic.RSA, 512) + ks.sk, ks.pk, err = test.RandTestKeyPair(ic.RSA, 2048) if err != nil { return err } @@ -226,17 +226,31 @@ func TestValidate(t *testing.T) { } } -var hpkpMan = `QmRK3JgmVEGiewxWbhpXLJyjWuGuLeSTMTndA1coMHEy5o` +var hpkpMan = `QmcJeseojbPW9hSejUM1sQ1a2QmbrryPK4Z8pWbRUPaYEn` var skManBytes = ` -CAAS4AQwggJcAgEAAoGBAL7w+Wc4VhZhCdM/+Hccg5Nrf4q9NXWwJylbSrXz/unFS24wyk6pEk0zi3W -7li+vSNVO+NtJQw9qGNAMtQKjVTP+3Vt/jfQRnQM3s6awojtjueEWuLYVt62z7mofOhCtj+VwIdZNBo -/EkLZ0ETfcvN5LVtLYa8JkXybnOPsLvK+PAgMBAAECgYBdk09HDM7zzL657uHfzfOVrdslrTCj6p5mo -DzvCxLkkjIzYGnlPuqfNyGjozkpSWgSUc+X+EGLLl3WqEOVdWJtbM61fewEHlRTM5JzScvwrJ39t7o6 -CCAjKA0cBWBd6UWgbN/t53RoWvh9HrA2AW5YrT0ZiAgKe9y7EMUaENVJ8QJBAPhpdmb4ZL4Fkm4OKia -NEcjzn6mGTlZtef7K/0oRC9+2JkQnCuf6HBpaRhJoCJYg7DW8ZY+AV6xClKrgjBOfERMCQQDExhnzu2 -dsQ9k8QChBlpHO0TRbZBiQfC70oU31kM1AeLseZRmrxv9Yxzdl8D693NNWS2JbKOXl0kMHHcuGQLMVA -kBZ7WvkmPV3aPL6jnwp2pXepntdVnaTiSxJ1dkXShZ/VSSDNZMYKY306EtHrIu3NZHtXhdyHKcggDXr -qkBrdgErAkAlpGPojUwemOggr4FD8sLX1ot2hDJyyV7OK2FXfajWEYJyMRL1Gm9Uk1+Un53RAkJneqp -JGAzKpyttXBTIDO51AkEA98KTiROMnnU8Y6Mgcvr68/SMIsvCYMt9/mtwSBGgl80VaTQ5Hpaktl6Xbh -VUt5Wv0tRxlXZiViCGCD1EtrrwTw== +CAASqAkwggSkAgEAAoIBAQC3hjPtPli71gFNzGJ6rUhYdb65BDwW7IrniEaZKi6z +tW4Iz0MouEJY8GPG1iQfqZKp5w9H2ENh4I1bk2dsezrJ7Nneg4Eqd78CmeHTAgaP +3PKsxohdMo/TOFNxwl8SkEF8FyVbio2TCoijYNHUuprZuq7MPEAJYr3Z1eEkM/xR +pMp3YI9S2SYsZQxbmmQ0/GfHOEvYajdow1qttreVTQkvmCppKtNLEU5InpX/W5fe +aQCj0pd7l74daZgM2WWz3juEUCVG7tdRUPg7ix1TYosbN96CKC3q2MJxe/wJ9gR5 +Jvjnaaaoon+mci5vrKzxdKBDmZ/ZbLiHDfVljMkbdOQLAgMBAAECggEAEULaF3JJ +vkD+lmamzIsHxuosKhKv5CgTWHuEyFsjUVu7IbD8zBOoidzyRX1WoHO+i6Rj14oL +rGUGZpqSm61rdhqE01zjBS+GE6SNjN8f5uANIxr5MGrVBDTEBGsXrhNLVXSH2vhJ +II9ZEqTEl5GFhvz7+9Ge5EMZQCfRqSoKjVMdrs+Rueuusr9p0wNg9PH1myA+cXGt +iNZA17Rj2IiWVZLDgYNo4DVQUt4mFb+wTJW4NSspGKaFebpn0hf4z21laoGoJqTC +cNETJw+QwQ0uDaRoYotTLT2/55e8XBFTdcTg5cmbZoKgMyGqZEHfRyD9reVDAZlM +EZwKtrm41kz94QKBgQDmPp5zVtFXQNONmje1NE0IjCaUKcqURXk4ZiILztfT9XLC +OXAUCs3TCq21jirCkZZ6gLfo12Wx0xJYmsKlaUOGNTa8FI5Xa7OyheYKixUvV6FW +J95P/sNuWscTjh7oZHgZk/L3yKrNzNBz7awComwV6qciXW7EP1uACHf5fS/RdQKB +gQDMDa38W9OeegRDrhCeYGsniJK7btOCzhNooruQKPPXxk+O4dyJm7VBbC/3Ch55 +a83W66T4k0Q7ysLVRT5Vqd5z3AM0sEM3ZoxUKCinG3NwPxVeXcoLasyEiq1vOFK6 +GqZKCMThCj7ZpbkWy0DPJagnYfZGC62lammuj+XQx7mvfwKBgQCTKhka/bXmgD/3 +9UeAIcLPIM2TzDZ4mQNHIjjGtVnMV8kXDaFung06xEuNjSYVoPq+qEFkqTCN/axv +R9P76BFJ2f93LehhRizggacsvAM5dFhh+i+lj+AYTBuMiz2EKpt9NcyJxhAuZKgk +QRi9wlU1mPtlArVG6HwylLcil3qV9QKBgQDJHtaU/KEY+2TGnIMuxxP2lEsjyLla +nOlOYc8C6Qpma8UwrHelfj5p7Eteb6/Xt6Tbp8kjZGuFj3T3plcpMdPbWEgkn3Kw +4TeBH0/qXUkrolHagBDLrglEvjbxf48ydV/fasM6l9GYzhofWFhZk+EoaArHwWz2 +tGrTrmsynBjt2wKBgErdYe+zZ2Wo+wXQGAoZi4pfcwiw4a97Kdh0dx+WZz7acHms +h+V20VRmEHm5h8WnJ/Wv5uK94t6NY17wzjQ7y2BN5mY5cA2cZAcpeqtv/N06tH4S +cn1UEuRB8VpwkjaPUNZhqtYK40qff2OTdJy8taFtQiN7fz9euWTC78zjph2s ` diff --git a/core/sec/insecure/insecure_test.go b/core/sec/insecure/insecure_test.go index a8b86f16c1..311cf4ef2e 100644 --- a/core/sec/insecure/insecure_test.go +++ b/core/sec/insecure/insecure_test.go @@ -14,7 +14,7 @@ import ( // Run a set of sessions through the session setup and verification. func TestConnections(t *testing.T) { - clientTpt := newTestTransport(t, ci.RSA, 1024) + clientTpt := newTestTransport(t, ci.RSA, 2048) serverTpt := newTestTransport(t, ci.Ed25519, 1024) testConnection(t, clientTpt, serverTpt) From 70edc02c26b41f49261d3b8c37c4c3df8ddc7c02 Mon Sep 17 00:00:00 2001 From: Guilhem Fanton Date: Wed, 28 Aug 2019 17:29:10 +0200 Subject: [PATCH 1478/3965] fix(android): use specific netlink families for android Android doesn't allow netlink_xfrm & netlink_nflog in his base policy in enforce mode. (see [here](https://android.googlesource.com/platform/system/sepolicy/+/3aa1c1725ea2b6fd452c5771629dcfc50a351538/public/app.te#396)) this cause a _permission denied_ when NewTransport method is called on android, however it's look like that only *netlink route* is needed, so we just have to limit netlink families support on android build to NETLINK_ROUTE only when netlink.NewHandle is called. --- p2p/transport/quic/netlink_android.go | 8 ++++++++ p2p/transport/quic/netlink_other.go | 8 ++++++++ p2p/transport/quic/reuse.go | 2 +- 3 files changed, 17 insertions(+), 1 deletion(-) create mode 100644 p2p/transport/quic/netlink_android.go create mode 100644 p2p/transport/quic/netlink_other.go diff --git a/p2p/transport/quic/netlink_android.go b/p2p/transport/quic/netlink_android.go new file mode 100644 index 0000000000..07625263c7 --- /dev/null +++ b/p2p/transport/quic/netlink_android.go @@ -0,0 +1,8 @@ +// +build android + +package libp2pquic + +import "golang.org/x/sys/unix" + +// Android doesn't allow netlink_xfrm and netlink_netfilter in his base policy +var SupportedNlFamilies = []int{unix.NETLINK_ROUTE} diff --git a/p2p/transport/quic/netlink_other.go b/p2p/transport/quic/netlink_other.go new file mode 100644 index 0000000000..47cc9d69bc --- /dev/null +++ b/p2p/transport/quic/netlink_other.go @@ -0,0 +1,8 @@ +// +build !android + +package libp2pquic + +import "github.com/vishvananda/netlink/nl" + +// nl.SupportedNlFamilies is the default netlink families used by the netlink package +var SupportedNlFamilies = nl.SupportedNlFamilies diff --git a/p2p/transport/quic/reuse.go b/p2p/transport/quic/reuse.go index cb45aa2307..c6415b4d8f 100644 --- a/p2p/transport/quic/reuse.go +++ b/p2p/transport/quic/reuse.go @@ -62,7 +62,7 @@ type reuse struct { func newReuse() (*reuse, error) { // On non-Linux systems, this will return ErrNotImplemented. - handle, err := netlink.NewHandle() + handle, err := netlink.NewHandle(SupportedNlFamilies...) if err == netlink.ErrNotImplemented { handle = nil } else if err != nil { From 7820ddc8dd2062c5521dc4730ed3cf7d8190cd95 Mon Sep 17 00:00:00 2001 From: Guilhem Fanton Date: Fri, 30 Aug 2019 15:43:22 +0200 Subject: [PATCH 1479/3965] fix(netlink): limit netlink families to NETLINK_ROUTE on linux instead of android only. --- p2p/transport/quic/netlink_android.go | 8 -------- p2p/transport/quic/netlink_linux.go | 10 ++++++++++ p2p/transport/quic/netlink_other.go | 2 +- 3 files changed, 11 insertions(+), 9 deletions(-) delete mode 100644 p2p/transport/quic/netlink_android.go create mode 100644 p2p/transport/quic/netlink_linux.go diff --git a/p2p/transport/quic/netlink_android.go b/p2p/transport/quic/netlink_android.go deleted file mode 100644 index 07625263c7..0000000000 --- a/p2p/transport/quic/netlink_android.go +++ /dev/null @@ -1,8 +0,0 @@ -// +build android - -package libp2pquic - -import "golang.org/x/sys/unix" - -// Android doesn't allow netlink_xfrm and netlink_netfilter in his base policy -var SupportedNlFamilies = []int{unix.NETLINK_ROUTE} diff --git a/p2p/transport/quic/netlink_linux.go b/p2p/transport/quic/netlink_linux.go new file mode 100644 index 0000000000..a5f6ffe6e1 --- /dev/null +++ b/p2p/transport/quic/netlink_linux.go @@ -0,0 +1,10 @@ +// +build linux + +package libp2pquic + +import "golang.org/x/sys/unix" + +// We just need netlink_route here. +// note: We should avoid the use of netlink_xfrm or netlink_netfilter has it is +// not allowed by Android in his base policy. +var SupportedNlFamilies = []int{unix.NETLINK_ROUTE} diff --git a/p2p/transport/quic/netlink_other.go b/p2p/transport/quic/netlink_other.go index 47cc9d69bc..58ad3ca124 100644 --- a/p2p/transport/quic/netlink_other.go +++ b/p2p/transport/quic/netlink_other.go @@ -1,4 +1,4 @@ -// +build !android +// +build !linux package libp2pquic From 0601ec2e9a8f7ff6bad9929c94ba7afe6b8b4cdb Mon Sep 17 00:00:00 2001 From: Jorik Schellekens Date: Wed, 4 Sep 2019 13:53:57 +0100 Subject: [PATCH 1480/3965] The docs seem to lie The docs seem to lie about what these methods do. This is the naive solution since it's my first day looking at go but would it be possible to get these methods to marshal to public or private key correctly by passing a type to them? --- core/crypto/key.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/crypto/key.go b/core/crypto/key.go index 6cd9dc0beb..6076f5296b 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -349,12 +349,12 @@ func MarshalPrivateKey(k PrivKey) ([]byte, error) { return proto.Marshal(pbmes) } -// ConfigDecodeKey decodes from b64 (for config file), and unmarshals. +// ConfigDecodeKey decodes from b64 (for config file) to a byte array that can be unmarshalled. func ConfigDecodeKey(b string) ([]byte, error) { return base64.StdEncoding.DecodeString(b) } -// ConfigEncodeKey encodes to b64 (for config file), and marshals. +// ConfigEncodeKey encodes a marshalled key to b64 (for config file). func ConfigEncodeKey(b []byte) string { return base64.StdEncoding.EncodeToString(b) } From 1b9b84b8100d3c54a8cf75e2db1d3d265a553191 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Thu, 5 Sep 2019 04:08:16 -0700 Subject: [PATCH 1481/3965] Add Filters option (#717) The new `Filters` option allows users to take ownership of the `Filters` struct, and is suitable for cases where the addresses you want to accept/deny change dynamically. --- options.go | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/options.go b/options.go index c5ef639ebe..6463ea712e 100644 --- a/options.go +++ b/options.go @@ -246,7 +246,8 @@ func EnableAutoRelay() Option { } // FilterAddresses configures libp2p to never dial nor accept connections from -// the given addresses. +// the given addresses. FilterAddresses should be used for cases where the +// addresses you want to deny are known ahead of time. func FilterAddresses(addrs ...*net.IPNet) Option { return func(cfg *Config) error { if cfg.Filters == nil { @@ -259,6 +260,17 @@ func FilterAddresses(addrs ...*net.IPNet) Option { } } +// Filters configures libp2p to use the given filters for accepting/denying +// certain addresses. Filters offers more control and should be use when the +// addresses you want to accept/deny are not known ahead of time and can +// dynamically change. +func Filters(filters *filter.Filters) Option { + return func(cfg *Config) error { + cfg.Filters = filters + return nil + } +} + // NATPortMap configures libp2p to use the default NATManager. The default // NATManager will attempt to open a port in your network's firewall using UPnP. func NATPortMap() Option { From adb766dff5e951719477d94037b852ab111aea35 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 6 Sep 2019 16:43:45 -0400 Subject: [PATCH 1482/3965] added initial backoff cache discovery --- p2p/discovery/backoff/backoff.go | 181 +++++++++++++ p2p/discovery/backoff/backoff_test.go | 125 +++++++++ p2p/discovery/backoff/backoffcache.go | 257 ++++++++++++++++++ p2p/discovery/backoff/backoffcache_test.go | 297 +++++++++++++++++++++ 4 files changed, 860 insertions(+) create mode 100644 p2p/discovery/backoff/backoff.go create mode 100644 p2p/discovery/backoff/backoff_test.go create mode 100644 p2p/discovery/backoff/backoffcache.go create mode 100644 p2p/discovery/backoff/backoffcache_test.go diff --git a/p2p/discovery/backoff/backoff.go b/p2p/discovery/backoff/backoff.go new file mode 100644 index 0000000000..15dd7805e1 --- /dev/null +++ b/p2p/discovery/backoff/backoff.go @@ -0,0 +1,181 @@ +package discovery + +import ( + "math" + "math/rand" + "time" +) + +type BackoffFactory func() BackoffStrategy + +type BackoffStrategy interface { + Delay() time.Duration + Reset() +} + +// Jitter implementations taken roughly from https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ + +// Jitter must return a duration between min and max. Min must be lower than, or equal to, max. +type Jitter func(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration + +func FullJitter(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration { + if duration <= min { + return min + } + + normalizedDur := boundedDuration(duration, min, max) - min + + return boundedDuration(time.Duration(rng.Int63n(int64(normalizedDur)))+min, min, max) +} + +func NoJitter(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration { + return boundedDuration(duration, min, max) +} + +type randomizedBackoff struct { + min time.Duration + max time.Duration + rng *rand.Rand +} + +func (b *randomizedBackoff) BoundedDelay(duration time.Duration) time.Duration { + return boundedDuration(duration, b.min, b.max) +} + +func boundedDuration(d time.Duration, min time.Duration, max time.Duration) time.Duration { + if d < min { + return min + } + if d > max { + return max + } + return d +} + +type attemptBackoff struct { + attempt int + jitter Jitter + randomizedBackoff +} + +func (b *attemptBackoff) Reset() { + b.attempt = 0 +} + +func NewFixedBackoffFactory(delay time.Duration) BackoffFactory { + return func() BackoffStrategy { + return &fixedBackoff{delay: delay} + } +} + +type fixedBackoff struct { + delay time.Duration +} + +func (b *fixedBackoff) Delay() time.Duration { + return b.delay +} + +func (b *fixedBackoff) Reset() {} + +func NewPolynomialBackoffFactory(min, max time.Duration, jitter Jitter, + timeUnits time.Duration, polyCoefs []float64, rng *rand.Rand) BackoffFactory { + return func() BackoffStrategy { + return &polynomialBackoff{ + attemptBackoff: attemptBackoff{ + randomizedBackoff: randomizedBackoff{ + min: min, + max: max, + rng: rng, + }, + jitter: jitter, + }, + timeUnits: timeUnits, + poly: polyCoefs, + } + } +} + +type polynomialBackoff struct { + attemptBackoff + timeUnits time.Duration + poly []float64 +} + +func (b *polynomialBackoff) Delay() time.Duration { + polySum := b.poly[0] + exp := 1 + attempt := b.attempt + b.attempt++ + + for _, c := range b.poly[1:] { + exp *= attempt + polySum += float64(exp) * c + } + return b.jitter(time.Duration(float64(b.timeUnits)*polySum), b.min, b.max, b.rng) +} + +func NewExponentialBackoffFactory(min, max time.Duration, jitter Jitter, + timeUnits time.Duration, base float64, offset time.Duration, rng *rand.Rand) BackoffFactory { + return func() BackoffStrategy { + return &exponentialBackoff{ + attemptBackoff: attemptBackoff{ + randomizedBackoff: randomizedBackoff{ + min: min, + max: max, + rng: rng, + }, + jitter: jitter, + }, + timeUnits: timeUnits, + base: base, + offset: offset, + } + } +} + +type exponentialBackoff struct { + attemptBackoff + timeUnits time.Duration + base float64 + offset time.Duration +} + +func (b *exponentialBackoff) Delay() time.Duration { + attempt := b.attempt + b.attempt++ + return b.jitter( + time.Duration(math.Pow(b.base, float64(attempt))*float64(b.timeUnits))+b.offset, b.min, b.max, b.rng) +} + +func NewExponentialDecorrelatedJitterFactory(min, max time.Duration, base float64, rng *rand.Rand) BackoffFactory { + return func() BackoffStrategy { + return &exponentialDecorrelatedJitter{ + randomizedBackoff: randomizedBackoff{ + min: min, + max: max, + rng: rng, + }, + base: base, + } + } +} + +type exponentialDecorrelatedJitter struct { + randomizedBackoff + base float64 + lastDelay time.Duration +} + +func (b *exponentialDecorrelatedJitter) Delay() time.Duration { + if b.lastDelay < b.min { + b.lastDelay = b.min + return b.lastDelay + } + + nextMax := int64(float64(b.lastDelay) * b.base) + b.lastDelay = boundedDuration(time.Duration(b.rng.Int63n(nextMax-int64(b.min)))+b.min, b.min, b.max) + return b.lastDelay +} + +func (b *exponentialDecorrelatedJitter) Reset() { b.lastDelay = 0 } diff --git a/p2p/discovery/backoff/backoff_test.go b/p2p/discovery/backoff/backoff_test.go new file mode 100644 index 0000000000..45cbb529a8 --- /dev/null +++ b/p2p/discovery/backoff/backoff_test.go @@ -0,0 +1,125 @@ +package discovery + +import ( + "math/rand" + "testing" + "time" +) + +func checkDelay(bkf BackoffStrategy, expected time.Duration, t *testing.T) { + t.Helper() + if calculated := bkf.Delay(); calculated != expected { + t.Fatalf("expected %v, got %v", expected, calculated) + } +} + +func TestFixedBackoff(t *testing.T) { + startDelay := time.Second + delay := startDelay + + bkf := NewFixedBackoffFactory(delay) + delay *= 2 + b1 := bkf() + delay *= 2 + b2 := bkf() + + if b1.Delay() != startDelay || b2.Delay() != startDelay { + t.Fatal("incorrect delay time") + } + + if b1.Delay() != startDelay { + t.Fatal("backoff is stateful") + } + + if b1.Reset(); b1.Delay() != startDelay { + t.Fatalf("Reset does something") + } +} + +func TestPolynomialBackoff(t *testing.T) { + rng := rand.New(rand.NewSource(0)) + bkf := NewPolynomialBackoffFactory(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, rng) + b1 := bkf() + b2 := bkf() + + if b1.Delay() != time.Second || b2.Delay() != time.Second { + t.Fatal("incorrect delay time") + } + + checkDelay(b1, time.Millisecond*5500, t) + checkDelay(b1, time.Millisecond*16500, t) + checkDelay(b1, time.Millisecond*33000, t) + checkDelay(b2, time.Millisecond*5500, t) + + b1.Reset() + b1.Delay() + checkDelay(b1, time.Millisecond*5500, t) +} + +func TestExponentialBackoff(t *testing.T) { + rng := rand.New(rand.NewSource(0)) + bkf := NewExponentialBackoffFactory(time.Millisecond*650, time.Second*7, NoJitter, time.Second, 1.5, -time.Millisecond*400, rng) + b1 := bkf() + b2 := bkf() + + if b1.Delay() != time.Millisecond*650 || b2.Delay() != time.Millisecond*650 { + t.Fatal("incorrect delay time") + } + + checkDelay(b1, time.Millisecond*1100, t) + checkDelay(b1, time.Millisecond*1850, t) + checkDelay(b1, time.Millisecond*2975, t) + checkDelay(b1, time.Microsecond*4662500, t) + checkDelay(b1, time.Second*7, t) + checkDelay(b2, time.Millisecond*1100, t) + + b1.Reset() + b1.Delay() + checkDelay(b1, time.Millisecond*1100, t) +} + +func minMaxJitterTest(jitter Jitter, t *testing.T) { + rng := rand.New(rand.NewSource(0)) + if jitter(time.Nanosecond, time.Hour*10, time.Hour*20, rng) < time.Hour*10 { + t.Fatal("Min not working") + } + if jitter(time.Hour, time.Nanosecond, time.Nanosecond*10, rng) > time.Nanosecond*10 { + t.Fatal("Max not working") + } +} + +func TestNoJitter(t *testing.T) { + minMaxJitterTest(NoJitter, t) + for i := 0; i < 10; i++ { + expected := time.Second * time.Duration(i) + if calculated := NoJitter(expected, time.Duration(0), time.Second*100, nil); calculated != expected { + t.Fatalf("expected %v, got %v", expected, calculated) + } + } +} + +func TestFullJitter(t *testing.T) { + rng := rand.New(rand.NewSource(0)) + minMaxJitterTest(FullJitter, t) + const numBuckets = 51 + const multiplier = 10 + const threshold = 20 + + histogram := make([]int, numBuckets) + + for i := 0; i < (numBuckets-1)*multiplier; i++ { + started := time.Nanosecond * 50 + calculated := FullJitter(started, 0, 100, rng) + histogram[calculated]++ + } + + for _, count := range histogram { + if count > threshold { + t.Fatal("jitter is not close to evenly spread") + } + } + + if histogram[numBuckets-1] > 0 { + t.Fatal("jitter increased overall time") + } +} diff --git a/p2p/discovery/backoff/backoffcache.go b/p2p/discovery/backoff/backoffcache.go new file mode 100644 index 0000000000..89651d39d5 --- /dev/null +++ b/p2p/discovery/backoff/backoffcache.go @@ -0,0 +1,257 @@ +package discovery + +import ( + "context" + "sync" + "time" + + "github.com/libp2p/go-libp2p-core/discovery" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-peerstore/addr" +) + +// BackoffDiscovery is an implementation of discovery that caches peer data and attenuates repeated queries +type BackoffDiscovery struct { + disc discovery.Discovery + strat BackoffFactory + peerCache map[string]*backoffCache + peerCacheMux sync.RWMutex +} + +func NewBackoffDiscovery(disc discovery.Discovery, strat BackoffFactory) (discovery.Discovery, error) { + return &BackoffDiscovery{ + disc: disc, + strat: strat, + peerCache: make(map[string]*backoffCache), + }, nil +} + +type backoffCache struct { + nextDiscover time.Time + prevPeers map[peer.ID]peer.AddrInfo + + peers map[peer.ID]peer.AddrInfo + sendingChs map[chan peer.AddrInfo]int + + ongoing bool + strat BackoffStrategy + mux sync.Mutex +} + +func (d *BackoffDiscovery) Advertise(ctx context.Context, ns string, opts ...discovery.Option) (time.Duration, error) { + return d.disc.Advertise(ctx, ns, opts...) +} + +func (d *BackoffDiscovery) FindPeers(ctx context.Context, ns string, opts ...discovery.Option) (<-chan peer.AddrInfo, error) { + // Get options + var options discovery.Options + err := options.Apply(opts...) + if err != nil { + return nil, err + } + + // Get cached peers + d.peerCacheMux.RLock() + c, ok := d.peerCache[ns] + d.peerCacheMux.RUnlock() + + /* + Overall plan: + If it's time to look for peers, look for peers, then return them + If it's not time then return cache + If it's time to look for peers, but we have already started looking. Get up to speed with ongoing request + */ + + // Setup cache if we don't have one yet + if !ok { + pc := &backoffCache{ + nextDiscover: time.Time{}, + prevPeers: make(map[peer.ID]peer.AddrInfo), + peers: make(map[peer.ID]peer.AddrInfo), + sendingChs: make(map[chan peer.AddrInfo]int), + strat: d.strat(), + } + d.peerCacheMux.Lock() + c, ok = d.peerCache[ns] + + if !ok { + d.peerCache[ns] = pc + c = pc + } + + d.peerCacheMux.Unlock() + } + + c.mux.Lock() + defer c.mux.Unlock() + + findPeers := !ok + timeExpired := false + if !findPeers { + timeExpired = time.Now().After(c.nextDiscover) + findPeers = timeExpired && !c.ongoing + } + + // If we should find peers then setup a dispatcher channel for dispatching incoming peers + if findPeers { + pch, err := d.disc.FindPeers(ctx, ns, opts...) + if err != nil { + return nil, err + } + + c.ongoing = true + + go func() { + defer func() { + c.mux.Lock() + + for ch := range c.sendingChs { + close(ch) + } + + // If the peer addresses have changed reset the backoff + if checkUpdates(c.prevPeers, c.peers) { + c.strat.Reset() + c.prevPeers = c.peers + } + c.nextDiscover = time.Now().Add(c.strat.Delay()) + + c.ongoing = false + c.peers = make(map[peer.ID]peer.AddrInfo) + c.sendingChs = make(map[chan peer.AddrInfo]int) + c.mux.Unlock() + }() + + for { + select { + case ai, ok := <-pch: + if !ok { + return + } + c.mux.Lock() + + // If we receive the same peer multiple times return the address union + var sendAi peer.AddrInfo + if prevAi, ok := c.peers[ai.ID]; ok { + if combinedAi := mergeAddrInfos(prevAi, ai); combinedAi != nil { + sendAi = *combinedAi + } else { + c.mux.Unlock() + continue + } + } else { + sendAi = ai + } + + c.peers[ai.ID] = sendAi + + for ch, rem := range c.sendingChs { + ch <- sendAi + if rem == 1 { + close(ch) + delete(c.sendingChs, ch) + break + } else if rem > 0 { + rem-- + } + } + + c.mux.Unlock() + case <-ctx.Done(): + return + } + } + }() + // If it's not yet time to search again then return cached peers + } else if !timeExpired { + chLen := options.Limit + + if chLen == 0 { + chLen = len(c.prevPeers) + } else if chLen > len(c.prevPeers) { + chLen = len(c.prevPeers) + } + pch := make(chan peer.AddrInfo, chLen) + for _, ai := range c.prevPeers { + pch <- ai + } + close(pch) + return pch, nil + } + + // Setup receiver channel for receiving peers from ongoing requests + + evtCh := make(chan peer.AddrInfo, 32) + pch := make(chan peer.AddrInfo, 8) + rcvPeers := make([]peer.AddrInfo, 0, 32) + for _, ai := range c.peers { + rcvPeers = append(rcvPeers, ai) + } + c.sendingChs[evtCh] = options.Limit + + go func() { + defer close(pch) + + for { + select { + case ai, ok := <-evtCh: + if ok { + rcvPeers = append(rcvPeers, ai) + + sentAll := true + sendPeers: + for i, p := range rcvPeers { + select { + case pch <- p: + default: + rcvPeers = rcvPeers[i:] + sentAll = false + break sendPeers + } + } + if sentAll { + rcvPeers = []peer.AddrInfo{} + } + } else { + for _, p := range rcvPeers { + select { + case pch <- p: + case <-ctx.Done(): + return + } + } + return + } + case <-ctx.Done(): + return + } + } + }() + + return pch, nil +} + +func mergeAddrInfos(prevAi, newAi peer.AddrInfo) *peer.AddrInfo { + combinedAddrs := addr.UniqueSource(addr.Slice(prevAi.Addrs), addr.Slice(newAi.Addrs)).Addrs() + if len(combinedAddrs) > len(prevAi.Addrs) { + combinedAi := &peer.AddrInfo{ID: prevAi.ID, Addrs: combinedAddrs} + return combinedAi + } + return nil +} + +func checkUpdates(orig, update map[peer.ID]peer.AddrInfo) bool { + if len(orig) != len(update) { + return true + } + for p, ai := range update { + if prevAi, ok := orig[p]; ok { + if combinedAi := mergeAddrInfos(prevAi, ai); combinedAi != nil { + return true + } + } else { + return true + } + } + return false +} diff --git a/p2p/discovery/backoff/backoffcache_test.go b/p2p/discovery/backoff/backoffcache_test.go new file mode 100644 index 0000000000..096e971c4c --- /dev/null +++ b/p2p/discovery/backoff/backoffcache_test.go @@ -0,0 +1,297 @@ +package discovery + +import ( + "context" + "sync" + "testing" + "time" + + bhost "github.com/libp2p/go-libp2p-blankhost" + "github.com/libp2p/go-libp2p-core/discovery" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" +) + +type mockDiscoveryServer struct { + mx sync.Mutex + db map[string]map[peer.ID]*discoveryRegistration +} + +type discoveryRegistration struct { + info peer.AddrInfo + expiration time.Time +} + +func newDiscoveryServer() *mockDiscoveryServer { + return &mockDiscoveryServer{ + db: make(map[string]map[peer.ID]*discoveryRegistration), + } +} + +func (s *mockDiscoveryServer) Advertise(ns string, info peer.AddrInfo, ttl time.Duration) (time.Duration, error) { + s.mx.Lock() + defer s.mx.Unlock() + + peers, ok := s.db[ns] + if !ok { + peers = make(map[peer.ID]*discoveryRegistration) + s.db[ns] = peers + } + peers[info.ID] = &discoveryRegistration{info, time.Now().Add(ttl)} + return ttl, nil +} + +func (s *mockDiscoveryServer) FindPeers(ns string, limit int) (<-chan peer.AddrInfo, error) { + s.mx.Lock() + defer s.mx.Unlock() + + peers, ok := s.db[ns] + if !ok || len(peers) == 0 { + emptyCh := make(chan peer.AddrInfo) + close(emptyCh) + return emptyCh, nil + } + + count := len(peers) + if limit != 0 && count > limit { + count = limit + } + + iterTime := time.Now() + ch := make(chan peer.AddrInfo, count) + numSent := 0 + for p, reg := range peers { + if numSent == count { + break + } + if iterTime.After(reg.expiration) { + delete(peers, p) + continue + } + + numSent++ + ch <- reg.info + } + close(ch) + + return ch, nil +} + +func (s *mockDiscoveryServer) hasPeerRecord(ns string, pid peer.ID) bool { + s.mx.Lock() + defer s.mx.Unlock() + + if peers, ok := s.db[ns]; ok { + _, ok := peers[pid] + return ok + } + return false +} + +type mockDiscoveryClient struct { + host host.Host + server *mockDiscoveryServer +} + +func (d *mockDiscoveryClient) Advertise(ctx context.Context, ns string, opts ...discovery.Option) (time.Duration, error) { + var options discovery.Options + err := options.Apply(opts...) + if err != nil { + return 0, err + } + + return d.server.Advertise(ns, *host.InfoFromHost(d.host), options.Ttl) +} + +func (d *mockDiscoveryClient) FindPeers(ctx context.Context, ns string, opts ...discovery.Option) (<-chan peer.AddrInfo, error) { + var options discovery.Options + err := options.Apply(opts...) + if err != nil { + return nil, err + } + + return d.server.FindPeers(ns, options.Limit) +} + +type delayedDiscovery struct { + disc discovery.Discovery + delay time.Duration +} + +func (d *delayedDiscovery) Advertise(ctx context.Context, ns string, opts ...discovery.Option) (time.Duration, error) { + return d.disc.Advertise(ctx, ns, opts...) +} + +func (d *delayedDiscovery) FindPeers(ctx context.Context, ns string, opts ...discovery.Option) (<-chan peer.AddrInfo, error) { + dch, err := d.disc.FindPeers(ctx, ns, opts...) + if err != nil { + return nil, err + } + + ch := make(chan peer.AddrInfo, 32) + go func() { + defer close(ch) + for ai := range dch { + ch <- ai + time.Sleep(d.delay) + } + }() + + return ch, nil +} + +func assertNumPeers(t *testing.T, ctx context.Context, d discovery.Discovery, ns string, count int) { + t.Helper() + peerCh, err := d.FindPeers(ctx, ns, discovery.Limit(10)) + if err != nil { + t.Fatal(err) + } + + peerset := make(map[peer.ID]struct{}) + for p := range peerCh { + peerset[p.ID] = struct{}{} + } + + if len(peerset) != count { + t.Fatalf("Was supposed to find %d, found %d instead", count, len(peerset)) + } +} + +func TestBackoffDiscoverySingleBackoff(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + discServer := newDiscoveryServer() + + h1 := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h2 := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + d1 := &mockDiscoveryClient{h1, discServer} + d2 := &mockDiscoveryClient{h2, discServer} + + bkf := NewExponentialBackoffFactory(time.Millisecond*100, time.Second*10, NoJitter, + time.Millisecond*100, 2.5, 0, nil) + dCache, err := NewBackoffDiscovery(d1, bkf) + if err != nil { + t.Fatal(err) + } + + const ns = "test" + + // try adding a peer then find it + d1.Advertise(ctx, ns, discovery.TTL(time.Hour)) + assertNumPeers(t, ctx, dCache, ns, 1) + + // add a new peer and make sure it is still hidden by the caching layer + d2.Advertise(ctx, ns, discovery.TTL(time.Hour)) + assertNumPeers(t, ctx, dCache, ns, 1) + + // wait for cache to expire and check for the new peer + time.Sleep(time.Millisecond * 110) + assertNumPeers(t, ctx, dCache, ns, 2) +} + +func TestBackoffDiscoveryMultipleBackoff(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + discServer := newDiscoveryServer() + + h1 := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h2 := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + d1 := &mockDiscoveryClient{h1, discServer} + d2 := &mockDiscoveryClient{h2, discServer} + + // Startup delay is 0ms. First backoff after finding data is 100ms, second backoff is 250ms. + bkf := NewExponentialBackoffFactory(time.Millisecond*100, time.Second*10, NoJitter, + time.Millisecond*100, 2.5, 0, nil) + dCache, err := NewBackoffDiscovery(d1, bkf) + if err != nil { + t.Fatal(err) + } + + const ns = "test" + + // try adding a peer then find it + d1.Advertise(ctx, ns, discovery.TTL(time.Hour)) + assertNumPeers(t, ctx, dCache, ns, 1) + + // wait a little to make sure the extra request doesn't modify the backoff + time.Sleep(time.Millisecond * 50) //50 < 100 + assertNumPeers(t, ctx, dCache, ns, 1) + + // wait for backoff to expire and check if we increase it + time.Sleep(time.Millisecond * 60) // 50+60 > 100 + assertNumPeers(t, ctx, dCache, ns, 1) + + d2.Advertise(ctx, ns, discovery.TTL(time.Millisecond*400)) + + time.Sleep(time.Millisecond * 150) //150 < 250 + assertNumPeers(t, ctx, dCache, ns, 1) + + time.Sleep(time.Millisecond * 150) //150 + 150 > 250 + assertNumPeers(t, ctx, dCache, ns, 2) + + // check that the backoff has been reset + // also checks that we can decrease our peer count (i.e. not just growing a set) + time.Sleep(time.Millisecond * 110) //110 > 100, also 150+150+110>400 + assertNumPeers(t, ctx, dCache, ns, 1) +} + +func TestBackoffDiscoverySimultaneousQuery(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + discServer := newDiscoveryServer() + + // Testing with n larger than most internal buffer sizes (32) + n := 40 + advertisers := make([]discovery.Discovery, n) + + for i := 0; i < n; i++ { + h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + advertisers[i] = &mockDiscoveryClient{h, discServer} + } + + d1 := &delayedDiscovery{advertisers[0], time.Millisecond * 10} + + bkf := NewFixedBackoffFactory(time.Millisecond * 200) + dCache, err := NewBackoffDiscovery(d1, bkf) + if err != nil { + t.Fatal(err) + } + + const ns = "test" + + for _, a := range advertisers { + if _, err := a.Advertise(ctx, ns, discovery.TTL(time.Hour)); err != nil { + t.Fatal(err) + } + } + + ch1, err := dCache.FindPeers(ctx, ns) + if err != nil { + t.Fatal(err) + } + + _ = <-ch1 + ch2, err := dCache.FindPeers(ctx, ns) + if err != nil { + t.Fatal(err) + } + + szCh2 := 0 + for ai := range ch2 { + _ = ai + szCh2++ + } + + szCh1 := 1 + for _ = range ch1 { + szCh1++ + } + + if szCh1 != n && szCh2 != n { + t.Fatalf("Channels returned %d, %d elements instead of %d", szCh1, szCh2, n) + } +} From df6fbe10082cb37f8ba4aa1ef24d11062b935ca0 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 6 Sep 2019 17:09:42 -0400 Subject: [PATCH 1483/3965] added discovery based content routing --- p2p/discovery/routing/routing.go | 27 +++++ p2p/discovery/routing/routing_test.go | 145 +++++++++++++++++++++++++- 2 files changed, 171 insertions(+), 1 deletion(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index 70e70870a5..f5ebad021b 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -2,6 +2,7 @@ package discovery import ( "context" + "github.com/libp2p/go-libp2p-core/discovery" "time" cid "github.com/ipfs/go-cid" @@ -83,3 +84,29 @@ func nsToCid(ns string) (cid.Cid, error) { return cid.NewCidV1(cid.Raw, h), nil } + +func NewDiscoveryRouting(disc discovery.Discovery) *DiscoveryRouting { + return &DiscoveryRouting{disc} +} + +type DiscoveryRouting struct { + discovery.Discovery +} + +func (r *DiscoveryRouting) Provide(ctx context.Context, c cid.Cid, bcast bool) error { + if !bcast { + return nil + } + + _, err := r.Advertise(ctx, cidToNs(c)) + return err +} + +func (r *DiscoveryRouting) FindProvidersAsync(ctx context.Context, c cid.Cid, limit int) <-chan peer.AddrInfo { + ch, _ := r.FindPeers(ctx, cidToNs(c), discovery.Limit(limit)) + return ch +} + +func cidToNs(c cid.Cid) string { + return "/provider/" + c.String() +} diff --git a/p2p/discovery/routing/routing_test.go b/p2p/discovery/routing/routing_test.go index 3fa39bd682..1534a5b68b 100644 --- a/p2p/discovery/routing/routing_test.go +++ b/p2p/discovery/routing/routing_test.go @@ -4,9 +4,11 @@ import ( "context" "sync" "testing" + "time" - cid "github.com/ipfs/go-cid" + "github.com/ipfs/go-cid" bhost "github.com/libp2p/go-libp2p-blankhost" + "github.com/libp2p/go-libp2p-core/discovery" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/peer" swarmt "github.com/libp2p/go-libp2p-swarm/testing" @@ -69,6 +71,107 @@ func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit return ch } +type mockDiscoveryServer struct { + mx sync.Mutex + db map[string]map[peer.ID]*discoveryRegistration +} + +type discoveryRegistration struct { + info peer.AddrInfo + expiration time.Time +} + +func newDiscoveryServer() *mockDiscoveryServer { + return &mockDiscoveryServer{ + db: make(map[string]map[peer.ID]*discoveryRegistration), + } +} + +func (s *mockDiscoveryServer) Advertise(ns string, info peer.AddrInfo, ttl time.Duration) (time.Duration, error) { + s.mx.Lock() + defer s.mx.Unlock() + + peers, ok := s.db[ns] + if !ok { + peers = make(map[peer.ID]*discoveryRegistration) + s.db[ns] = peers + } + peers[info.ID] = &discoveryRegistration{info, time.Now().Add(ttl)} + return ttl, nil +} + +func (s *mockDiscoveryServer) FindPeers(ns string, limit int) (<-chan peer.AddrInfo, error) { + s.mx.Lock() + defer s.mx.Unlock() + + peers, ok := s.db[ns] + if !ok || len(peers) == 0 { + emptyCh := make(chan peer.AddrInfo) + close(emptyCh) + return emptyCh, nil + } + + count := len(peers) + if limit != 0 && count > limit { + count = limit + } + + iterTime := time.Now() + ch := make(chan peer.AddrInfo, count) + numSent := 0 + for p, reg := range peers { + if numSent == count { + break + } + if iterTime.After(reg.expiration) { + delete(peers, p) + continue + } + + numSent++ + ch <- reg.info + } + close(ch) + + return ch, nil +} + +func (s *mockDiscoveryServer) hasPeerRecord(ns string, pid peer.ID) bool { + s.mx.Lock() + defer s.mx.Unlock() + + if peers, ok := s.db[ns]; ok { + _, ok := peers[pid] + return ok + } + return false +} + +type mockDiscoveryClient struct { + host host.Host + server *mockDiscoveryServer +} + +func (d *mockDiscoveryClient) Advertise(ctx context.Context, ns string, opts ...discovery.Option) (time.Duration, error) { + var options discovery.Options + err := options.Apply(opts...) + if err != nil { + return 0, err + } + + return d.server.Advertise(ns, *host.InfoFromHost(d.host), options.Ttl) +} + +func (d *mockDiscoveryClient) FindPeers(ctx context.Context, ns string, opts ...discovery.Option) (<-chan peer.AddrInfo, error) { + var options discovery.Options + err := options.Apply(opts...) + if err != nil { + return nil, err + } + + return d.server.FindPeers(ns, options.Limit) +} + func TestRoutingDiscovery(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -102,3 +205,43 @@ func TestRoutingDiscovery(t *testing.T) { t.Fatalf("Unexpected peer: %s", pi.ID) } } + +func TestDiscoveryRouting(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + h1 := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h2 := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + + dserver := newDiscoveryServer() + d1 := &mockDiscoveryClient{h1, dserver} + d2 := &mockDiscoveryClient{h2, dserver} + + r1 := NewDiscoveryRouting(d1) + r2 := NewDiscoveryRouting(d2) + + c, err := nsToCid("/test") + if err != nil { + t.Fatal(err) + } + + if err := r1.Provide(ctx, c, true); err != nil { + t.Fatal(err) + } + + pch := r2.FindProvidersAsync(ctx, c, 20) + + var allAIs []peer.AddrInfo + for ai := range pch { + allAIs = append(allAIs, ai) + } + + if len(allAIs) != 1 { + t.Fatalf("Expected 1 peer, got %d", len(allAIs)) + } + + ai := allAIs[0] + if ai.ID != h1.ID() { + t.Fatalf("Unexpected peer: %s", ai.ID) + } +} From 702512dd33e26b256b64fcdd046566e7bb775b29 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 6 Sep 2019 17:29:08 -0400 Subject: [PATCH 1484/3965] added discovery options to discovery based content routing --- p2p/discovery/routing/routing.go | 9 +++++---- p2p/discovery/routing/routing_test.go | 4 ++-- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/p2p/discovery/routing/routing.go b/p2p/discovery/routing/routing.go index f5ebad021b..91f3ecae15 100644 --- a/p2p/discovery/routing/routing.go +++ b/p2p/discovery/routing/routing.go @@ -85,12 +85,13 @@ func nsToCid(ns string) (cid.Cid, error) { return cid.NewCidV1(cid.Raw, h), nil } -func NewDiscoveryRouting(disc discovery.Discovery) *DiscoveryRouting { - return &DiscoveryRouting{disc} +func NewDiscoveryRouting(disc discovery.Discovery, opts ...discovery.Option) *DiscoveryRouting { + return &DiscoveryRouting{disc, opts} } type DiscoveryRouting struct { discovery.Discovery + opts []discovery.Option } func (r *DiscoveryRouting) Provide(ctx context.Context, c cid.Cid, bcast bool) error { @@ -98,12 +99,12 @@ func (r *DiscoveryRouting) Provide(ctx context.Context, c cid.Cid, bcast bool) e return nil } - _, err := r.Advertise(ctx, cidToNs(c)) + _, err := r.Advertise(ctx, cidToNs(c), r.opts...) return err } func (r *DiscoveryRouting) FindProvidersAsync(ctx context.Context, c cid.Cid, limit int) <-chan peer.AddrInfo { - ch, _ := r.FindPeers(ctx, cidToNs(c), discovery.Limit(limit)) + ch, _ := r.FindPeers(ctx, cidToNs(c), append([]discovery.Option{discovery.Limit(limit)}, r.opts...)...) return ch } diff --git a/p2p/discovery/routing/routing_test.go b/p2p/discovery/routing/routing_test.go index 1534a5b68b..c07a5d679c 100644 --- a/p2p/discovery/routing/routing_test.go +++ b/p2p/discovery/routing/routing_test.go @@ -217,8 +217,8 @@ func TestDiscoveryRouting(t *testing.T) { d1 := &mockDiscoveryClient{h1, dserver} d2 := &mockDiscoveryClient{h2, dserver} - r1 := NewDiscoveryRouting(d1) - r2 := NewDiscoveryRouting(d2) + r1 := NewDiscoveryRouting(d1, discovery.TTL(time.Hour)) + r2 := NewDiscoveryRouting(d2, discovery.TTL(time.Hour)) c, err := nsToCid("/test") if err != nil { From f0810711a1696ec33979427f8f425e96d34540b2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 6 Sep 2019 22:37:05 -0700 Subject: [PATCH 1485/3965] fix an incorrect error message "security stream multiplexer" is the worst sort of confusing. --- p2p/net/upgrader/upgrader.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index 5e3854b0c1..69fb90bb22 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -91,7 +91,7 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma smconn, err := u.setupMuxer(ctx, sconn, p) if err != nil { sconn.Close() - return nil, fmt.Errorf("failed to negotiate security stream multiplexer: %s", err) + return nil, fmt.Errorf("failed to negotiate stream multiplexer: %s", err) } return &transportConn{ MuxedConn: smconn, From 7f58d8fb38fda0431c42090c8b48cc10cfdbbff9 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Sun, 8 Sep 2019 08:21:28 -0400 Subject: [PATCH 1486/3965] refactored backoffcache and increased an internal channel size. --- p2p/discovery/backoff/backoffcache.go | 198 +++++++++++++------------- 1 file changed, 100 insertions(+), 98 deletions(-) diff --git a/p2p/discovery/backoff/backoffcache.go b/p2p/discovery/backoff/backoffcache.go index 89651d39d5..3376f2eb81 100644 --- a/p2p/discovery/backoff/backoffcache.go +++ b/p2p/discovery/backoff/backoffcache.go @@ -100,68 +100,7 @@ func (d *BackoffDiscovery) FindPeers(ctx context.Context, ns string, opts ...dis } c.ongoing = true - - go func() { - defer func() { - c.mux.Lock() - - for ch := range c.sendingChs { - close(ch) - } - - // If the peer addresses have changed reset the backoff - if checkUpdates(c.prevPeers, c.peers) { - c.strat.Reset() - c.prevPeers = c.peers - } - c.nextDiscover = time.Now().Add(c.strat.Delay()) - - c.ongoing = false - c.peers = make(map[peer.ID]peer.AddrInfo) - c.sendingChs = make(map[chan peer.AddrInfo]int) - c.mux.Unlock() - }() - - for { - select { - case ai, ok := <-pch: - if !ok { - return - } - c.mux.Lock() - - // If we receive the same peer multiple times return the address union - var sendAi peer.AddrInfo - if prevAi, ok := c.peers[ai.ID]; ok { - if combinedAi := mergeAddrInfos(prevAi, ai); combinedAi != nil { - sendAi = *combinedAi - } else { - c.mux.Unlock() - continue - } - } else { - sendAi = ai - } - - c.peers[ai.ID] = sendAi - - for ch, rem := range c.sendingChs { - ch <- sendAi - if rem == 1 { - close(ch) - delete(c.sendingChs, ch) - break - } else if rem > 0 { - rem-- - } - } - - c.mux.Unlock() - case <-ctx.Done(): - return - } - } - }() + go findPeerDispatcher(ctx, c, pch) // If it's not yet time to search again then return cached peers } else if !timeExpired { chLen := options.Limit @@ -180,55 +119,118 @@ func (d *BackoffDiscovery) FindPeers(ctx context.Context, ns string, opts ...dis } // Setup receiver channel for receiving peers from ongoing requests - evtCh := make(chan peer.AddrInfo, 32) - pch := make(chan peer.AddrInfo, 8) + pch := make(chan peer.AddrInfo, 32) rcvPeers := make([]peer.AddrInfo, 0, 32) for _, ai := range c.peers { rcvPeers = append(rcvPeers, ai) } c.sendingChs[evtCh] = options.Limit - go func() { - defer close(pch) - - for { - select { - case ai, ok := <-evtCh: - if ok { - rcvPeers = append(rcvPeers, ai) - - sentAll := true - sendPeers: - for i, p := range rcvPeers { - select { - case pch <- p: - default: - rcvPeers = rcvPeers[i:] - sentAll = false - break sendPeers - } - } - if sentAll { - rcvPeers = []peer.AddrInfo{} - } + go findPeerReceiver(ctx, pch, evtCh, rcvPeers) + + return pch, nil +} + +func findPeerDispatcher(ctx context.Context, c *backoffCache, pch <-chan peer.AddrInfo) { + defer func() { + c.mux.Lock() + + for ch := range c.sendingChs { + close(ch) + } + + // If the peer addresses have changed reset the backoff + if checkUpdates(c.prevPeers, c.peers) { + c.strat.Reset() + c.prevPeers = c.peers + } + c.nextDiscover = time.Now().Add(c.strat.Delay()) + + c.ongoing = false + c.peers = make(map[peer.ID]peer.AddrInfo) + c.sendingChs = make(map[chan peer.AddrInfo]int) + c.mux.Unlock() + }() + + for { + select { + case ai, ok := <-pch: + if !ok { + return + } + c.mux.Lock() + + // If we receive the same peer multiple times return the address union + var sendAi peer.AddrInfo + if prevAi, ok := c.peers[ai.ID]; ok { + if combinedAi := mergeAddrInfos(prevAi, ai); combinedAi != nil { + sendAi = *combinedAi } else { - for _, p := range rcvPeers { - select { - case pch <- p: - case <-ctx.Done(): - return - } + c.mux.Unlock() + continue + } + } else { + sendAi = ai + } + + c.peers[ai.ID] = sendAi + + for ch, rem := range c.sendingChs { + ch <- sendAi + if rem == 1 { + close(ch) + delete(c.sendingChs, ch) + break + } else if rem > 0 { + rem-- + } + } + + c.mux.Unlock() + case <-ctx.Done(): + return + } + } +} + +func findPeerReceiver(ctx context.Context, pch, evtCh chan peer.AddrInfo, rcvPeers []peer.AddrInfo) { + defer close(pch) + + for { + select { + case ai, ok := <-evtCh: + if ok { + rcvPeers = append(rcvPeers, ai) + + sentAll := true + sendPeers: + for i, p := range rcvPeers { + select { + case pch <- p: + default: + rcvPeers = rcvPeers[i:] + sentAll = false + break sendPeers + } + } + if sentAll { + rcvPeers = []peer.AddrInfo{} + } + } else { + for _, p := range rcvPeers { + select { + case pch <- p: + case <-ctx.Done(): + return } - return } - case <-ctx.Done(): return } + case <-ctx.Done(): + return } - }() - - return pch, nil + } } func mergeAddrInfos(prevAi, newAi peer.AddrInfo) *peer.AddrInfo { From 54366d06e72d6f95e14f93c5710284a27b593865 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Sun, 8 Sep 2019 08:30:33 -0400 Subject: [PATCH 1487/3965] refactor constructor functions for BackoffFactory implementations. refactored BackoffDiscovery to better differentiate the BackoffStrategy from BackoffFactory. --- p2p/discovery/backoff/backoff.go | 8 ++++---- p2p/discovery/backoff/backoff_test.go | 6 +++--- p2p/discovery/backoff/backoffcache.go | 12 ++++++------ p2p/discovery/backoff/backoffcache_test.go | 6 +++--- 4 files changed, 16 insertions(+), 16 deletions(-) diff --git a/p2p/discovery/backoff/backoff.go b/p2p/discovery/backoff/backoff.go index 15dd7805e1..e27f1a628c 100644 --- a/p2p/discovery/backoff/backoff.go +++ b/p2p/discovery/backoff/backoff.go @@ -62,7 +62,7 @@ func (b *attemptBackoff) Reset() { b.attempt = 0 } -func NewFixedBackoffFactory(delay time.Duration) BackoffFactory { +func NewFixedBackoff(delay time.Duration) BackoffFactory { return func() BackoffStrategy { return &fixedBackoff{delay: delay} } @@ -78,7 +78,7 @@ func (b *fixedBackoff) Delay() time.Duration { func (b *fixedBackoff) Reset() {} -func NewPolynomialBackoffFactory(min, max time.Duration, jitter Jitter, +func NewPolynomialBackoff(min, max time.Duration, jitter Jitter, timeUnits time.Duration, polyCoefs []float64, rng *rand.Rand) BackoffFactory { return func() BackoffStrategy { return &polynomialBackoff{ @@ -115,7 +115,7 @@ func (b *polynomialBackoff) Delay() time.Duration { return b.jitter(time.Duration(float64(b.timeUnits)*polySum), b.min, b.max, b.rng) } -func NewExponentialBackoffFactory(min, max time.Duration, jitter Jitter, +func NewExponentialBackoff(min, max time.Duration, jitter Jitter, timeUnits time.Duration, base float64, offset time.Duration, rng *rand.Rand) BackoffFactory { return func() BackoffStrategy { return &exponentialBackoff{ @@ -148,7 +148,7 @@ func (b *exponentialBackoff) Delay() time.Duration { time.Duration(math.Pow(b.base, float64(attempt))*float64(b.timeUnits))+b.offset, b.min, b.max, b.rng) } -func NewExponentialDecorrelatedJitterFactory(min, max time.Duration, base float64, rng *rand.Rand) BackoffFactory { +func NewExponentialDecorrelatedJitter(min, max time.Duration, base float64, rng *rand.Rand) BackoffFactory { return func() BackoffStrategy { return &exponentialDecorrelatedJitter{ randomizedBackoff: randomizedBackoff{ diff --git a/p2p/discovery/backoff/backoff_test.go b/p2p/discovery/backoff/backoff_test.go index 45cbb529a8..655d720622 100644 --- a/p2p/discovery/backoff/backoff_test.go +++ b/p2p/discovery/backoff/backoff_test.go @@ -17,7 +17,7 @@ func TestFixedBackoff(t *testing.T) { startDelay := time.Second delay := startDelay - bkf := NewFixedBackoffFactory(delay) + bkf := NewFixedBackoff(delay) delay *= 2 b1 := bkf() delay *= 2 @@ -38,7 +38,7 @@ func TestFixedBackoff(t *testing.T) { func TestPolynomialBackoff(t *testing.T) { rng := rand.New(rand.NewSource(0)) - bkf := NewPolynomialBackoffFactory(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, rng) + bkf := NewPolynomialBackoff(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, rng) b1 := bkf() b2 := bkf() @@ -58,7 +58,7 @@ func TestPolynomialBackoff(t *testing.T) { func TestExponentialBackoff(t *testing.T) { rng := rand.New(rand.NewSource(0)) - bkf := NewExponentialBackoffFactory(time.Millisecond*650, time.Second*7, NoJitter, time.Second, 1.5, -time.Millisecond*400, rng) + bkf := NewExponentialBackoff(time.Millisecond*650, time.Second*7, NoJitter, time.Second, 1.5, -time.Millisecond*400, rng) b1 := bkf() b2 := bkf() diff --git a/p2p/discovery/backoff/backoffcache.go b/p2p/discovery/backoff/backoffcache.go index 3376f2eb81..1c2d1d0dea 100644 --- a/p2p/discovery/backoff/backoffcache.go +++ b/p2p/discovery/backoff/backoffcache.go @@ -13,16 +13,16 @@ import ( // BackoffDiscovery is an implementation of discovery that caches peer data and attenuates repeated queries type BackoffDiscovery struct { disc discovery.Discovery - strat BackoffFactory + stratFactory BackoffFactory peerCache map[string]*backoffCache peerCacheMux sync.RWMutex } -func NewBackoffDiscovery(disc discovery.Discovery, strat BackoffFactory) (discovery.Discovery, error) { +func NewBackoffDiscovery(disc discovery.Discovery, stratFactory BackoffFactory) (discovery.Discovery, error) { return &BackoffDiscovery{ - disc: disc, - strat: strat, - peerCache: make(map[string]*backoffCache), + disc: disc, + stratFactory: stratFactory, + peerCache: make(map[string]*backoffCache), }, nil } @@ -69,7 +69,7 @@ func (d *BackoffDiscovery) FindPeers(ctx context.Context, ns string, opts ...dis prevPeers: make(map[peer.ID]peer.AddrInfo), peers: make(map[peer.ID]peer.AddrInfo), sendingChs: make(map[chan peer.AddrInfo]int), - strat: d.strat(), + strat: d.stratFactory(), } d.peerCacheMux.Lock() c, ok = d.peerCache[ns] diff --git a/p2p/discovery/backoff/backoffcache_test.go b/p2p/discovery/backoff/backoffcache_test.go index 096e971c4c..636e410477 100644 --- a/p2p/discovery/backoff/backoffcache_test.go +++ b/p2p/discovery/backoff/backoffcache_test.go @@ -169,7 +169,7 @@ func TestBackoffDiscoverySingleBackoff(t *testing.T) { d1 := &mockDiscoveryClient{h1, discServer} d2 := &mockDiscoveryClient{h2, discServer} - bkf := NewExponentialBackoffFactory(time.Millisecond*100, time.Second*10, NoJitter, + bkf := NewExponentialBackoff(time.Millisecond*100, time.Second*10, NoJitter, time.Millisecond*100, 2.5, 0, nil) dCache, err := NewBackoffDiscovery(d1, bkf) if err != nil { @@ -203,7 +203,7 @@ func TestBackoffDiscoveryMultipleBackoff(t *testing.T) { d2 := &mockDiscoveryClient{h2, discServer} // Startup delay is 0ms. First backoff after finding data is 100ms, second backoff is 250ms. - bkf := NewExponentialBackoffFactory(time.Millisecond*100, time.Second*10, NoJitter, + bkf := NewExponentialBackoff(time.Millisecond*100, time.Second*10, NoJitter, time.Millisecond*100, 2.5, 0, nil) dCache, err := NewBackoffDiscovery(d1, bkf) if err != nil { @@ -255,7 +255,7 @@ func TestBackoffDiscoverySimultaneousQuery(t *testing.T) { d1 := &delayedDiscovery{advertisers[0], time.Millisecond * 10} - bkf := NewFixedBackoffFactory(time.Millisecond * 200) + bkf := NewFixedBackoff(time.Millisecond * 200) dCache, err := NewBackoffDiscovery(d1, bkf) if err != nil { t.Fatal(err) From 529bf100cc280cd352e1d6c06b5f91b1849532de Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 10 Sep 2019 18:46:07 -0400 Subject: [PATCH 1488/3965] Update link to go-libp2p-tls in README.md (#719) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9df42c292e..545b9e0a1b 100644 --- a/README.md +++ b/README.md @@ -135,7 +135,7 @@ List of packages currently in existence for libp2p: | [`go-libp2p-reuseport-transport`](//github.com/libp2p/go-libp2p-reuseport-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-reuseport-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-reuseport-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-reuseport-transport) | partial transport for building transports that reuse ports | | **Encrypted Channels** | | [`go-libp2p-secio`](//github.com/libp2p/go-libp2p-secio) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-secio.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-secio) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-secio/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-secio) | SecIO crypto channel | -| [`go-libp2p-tls-transport`](//github.com/libp2p/go-libp2p-tls-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-tls-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-tls-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-tls-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-tls-transport) | TLS 1.3+ crypto channel | +| [`go-libp2p-tls`](//github.com/libp2p/go-libp2p-tls) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-tls.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-tls) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-tls/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-tls) | TLS 1.3+ crypto channel | | [`go-conn-security-multistream`](//github.com/libp2p/go-conn-security-multistream) | [![Travis CI](https://travis-ci.com/libp2p/go-conn-security-multistream.svg?branch=master)](https://travis-ci.com/libp2p/go-conn-security-multistream) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security-multistream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security-multistream) | multistream multiplexed meta crypto channel | | **Private Network** | | [`go-libp2p-pnet`](//github.com/libp2p/go-libp2p-pnet) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-pnet.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-pnet) | reference private networking implementation | From 02e6218268f8b3aa6452790c8f698e51bfa00d6a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Fri, 13 Sep 2019 11:55:33 +0100 Subject: [PATCH 1489/3965] readme: remove waffle link. (#720) --- README.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/README.md b/README.md index 545b9e0a1b..a066cd5794 100644 --- a/README.md +++ b/README.md @@ -22,12 +22,6 @@

-# Project status - -[![Throughput Graph](https://graphs.waffle.io/libp2p/go-libp2p/throughput.svg)](https://waffle.io/libp2p/go-libp2p/metrics/throughput) - -[**`Weekly Core Dev Calls`**](https://github.com/ipfs/pm/issues/674) - # Table of Contents - [Background](#background) @@ -40,6 +34,7 @@ - [Tests](#tests) - [Packages](#packages) - [Contribute](#contribute) +- [Weekly Core Dev Calls](https://github.com/ipfs/pm/issues/674) ## Background From 11ed2c35c62d450242847f74304ca56750130aeb Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 13 Sep 2019 13:37:04 -0700 Subject: [PATCH 1490/3965] test: close peerstore when closing the test swarm --- p2p/net/swarm/testing/testing.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/net/swarm/testing/testing.go b/p2p/net/swarm/testing/testing.go index 6ffe7d2ae4..5e39602426 100644 --- a/p2p/net/swarm/testing/testing.go +++ b/p2p/net/swarm/testing/testing.go @@ -10,6 +10,7 @@ import ( "github.com/libp2p/go-libp2p-testing/net" "github.com/libp2p/go-tcp-transport" + goprocess "github.com/jbenet/goprocess" csms "github.com/libp2p/go-conn-security-multistream" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" secio "github.com/libp2p/go-libp2p-secio" @@ -72,6 +73,7 @@ func GenSwarm(t *testing.T, ctx context.Context, opts ...Option) *swarm.Swarm { ps.AddPubKey(p.ID, p.PubKey) ps.AddPrivKey(p.ID, p.PrivKey) s := swarm.NewSwarm(ctx, p.ID, ps, metrics.NewBandwidthCounter()) + s.Process().AddChild(goprocess.WithTeardown(ps.Close)) tcpTransport := tcp.NewTCPTransport(GenUpgrader(s)) tcpTransport.DisableReuseport = cfg.disableReuseport From 4b3044e3d08c5a1d5b4aba4e71127c6ed721a526 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 00:25:25 -0700 Subject: [PATCH 1491/3965] chore: better address parsing Use ma.SplitFunc instead of splitting the entire multiaddr and comparing bytes. This avoids a bunch of unnecessary allocation and copying. --- .../internal/circuitv1-deprecated/dial.go | 25 ++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 6668f3970f..1b6e9db0f4 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -20,28 +20,25 @@ func (d *RelayTransport) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (t } func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (*Conn, error) { - if !r.Matches(a) { + // split /a/p2p-circuit/b into (/a, /p2p-circuit/b) + relayaddr, destaddr := ma.SplitFunc(a, func(c ma.Component) bool { + return c.Protocol().Code == P_CIRCUIT + }) + + // If the address contained no /p2p-circuit part, the second part is nil. + if destaddr == nil { return nil, fmt.Errorf("%s is not a relay address", a) } - parts := ma.Split(a) - - spl := ma.Cast(ma.CodeToVarint(P_CIRCUIT)) - var relayaddr, destaddr ma.Multiaddr - for i, p := range parts { - if p.Equal(spl) { - relayaddr = ma.Join(parts[:i]...) - destaddr = ma.Join(parts[i+1:]...) - break - } - } + // Strip the /p2p-circuit prefix from the destaddr. + _, destaddr = ma.SplitFirst(destaddr) dinfo := &peer.AddrInfo{ID: p, Addrs: []ma.Multiaddr{}} - if len(destaddr.Bytes()) > 0 { + if destaddr != nil { dinfo.Addrs = append(dinfo.Addrs, destaddr) } - if len(relayaddr.Bytes()) == 0 { + if relayaddr == nil { // unspecific relay address, try dialing using known hop relays return r.tryDialRelays(ctx, *dinfo) } From 3554c95e09fdde5b9354ec1fe1bbe2d3661da693 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 00:28:28 -0700 Subject: [PATCH 1492/3965] fix: expose full multiaddrs from conn.RemoteMultiaddr There's no reason _not_ to do this. --- .../internal/circuitv1-deprecated/conn.go | 15 +++++++++------ .../internal/circuitv1-deprecated/listen.go | 2 +- .../internal/circuitv1-deprecated/transport.go | 3 +++ 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/conn.go b/p2p/protocol/internal/circuitv1-deprecated/conn.go index 8fa53b3da7..11f947633e 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/conn.go +++ b/p2p/protocol/internal/circuitv1-deprecated/conn.go @@ -80,12 +80,15 @@ func (c *Conn) untagHop() { // TODO: is it okay to cast c.Conn().RemotePeer() into a multiaddr? might be "user input" func (c *Conn) RemoteMultiaddr() ma.Multiaddr { - proto := ma.ProtocolWithCode(ma.P_P2P).Name - peerid := c.stream.Conn().RemotePeer().Pretty() - p2paddr := ma.StringCast(fmt.Sprintf("/%s/%s", proto, peerid)) - - circaddr := ma.Cast(ma.CodeToVarint(P_CIRCUIT)) - return p2paddr.Encapsulate(circaddr) + // TODO: We should be able to do this directly without converting to/from a string. + relayAddr, err := ma.NewComponent( + ma.ProtocolWithCode(ma.P_P2P).Name, + c.stream.Conn().RemotePeer().Pretty(), + ) + if err != nil { + panic(err) + } + return ma.Join(c.stream.Conn().RemoteMultiaddr(), relayAddr, circuitAddr) } func (c *Conn) LocalMultiaddr() ma.Multiaddr { diff --git a/p2p/protocol/internal/circuitv1-deprecated/listen.go b/p2p/protocol/internal/circuitv1-deprecated/listen.go index b00015b565..2f0e60c635 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/listen.go +++ b/p2p/protocol/internal/circuitv1-deprecated/listen.go @@ -50,7 +50,7 @@ func (l *RelayListener) Addr() net.Addr { } func (l *RelayListener) Multiaddr() ma.Multiaddr { - return ma.Cast(ma.CodeToVarint(P_CIRCUIT)) + return circuitAddr } func (l *RelayListener) Close() error { diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 57a65779f2..89ade492ac 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -20,8 +20,11 @@ var Protocol = ma.Protocol{ VCode: ma.CodeToVarint(P_CIRCUIT), } +var circuitAddr ma.Multiaddr + func init() { ma.AddProtocol(Protocol) + circuitAddr = ma.Cast(Protocol.VCode) } var _ transport.Transport = (*RelayTransport)(nil) From b43f7b88b4bf613441be38d807effa40a4e0d673 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 10:02:44 -0700 Subject: [PATCH 1493/3965] test: don't hang on fail --- p2p/protocol/internal/circuitv1-deprecated/relay_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index 40a4829754..b23c2e0d44 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -193,7 +193,6 @@ func TestRelayReset(t *testing.T) { func TestBasicRelayDial(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) - defer cancel() hosts := getNetHosts(t, ctx, 3) @@ -213,6 +212,7 @@ func TestBasicRelayDial(t *testing.T) { ) defer func() { + cancel() <-done if conn1 != nil { conn1.Close() From 2fc0af4c10476a575699bc54fa21105ee6ab1541 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 12:08:20 -0700 Subject: [PATCH 1494/3965] dep: update go-libp2p-circuit --- go.mod | 4 +++- go.sum | 7 +++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 98d1e1f269..6907a3711a 100644 --- a/go.mod +++ b/go.mod @@ -12,7 +12,7 @@ require ( github.com/libp2p/go-eventbus v0.1.0 github.com/libp2p/go-libp2p-autonat v0.1.0 github.com/libp2p/go-libp2p-blankhost v0.1.3 - github.com/libp2p/go-libp2p-circuit v0.1.1 + github.com/libp2p/go-libp2p-circuit v0.1.2 github.com/libp2p/go-libp2p-core v0.2.2 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 @@ -36,3 +36,5 @@ require ( github.com/multiformats/go-multistream v0.1.0 github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 ) + +go 1.12 diff --git a/go.sum b/go.sum index b5666cbff3..0f4afa8328 100644 --- a/go.sum +++ b/go.sum @@ -121,8 +121,8 @@ github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3Pt github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.3 h1:0KycuXvPDhmehw0ASsg+s1o3IfXgCUDqfzAl94KEBOg= github.com/libp2p/go-libp2p-blankhost v0.1.3/go.mod h1:KML1//wiKR8vuuJO0y3LUd1uLv+tlkGTAr3jC0S5cLg= -github.com/libp2p/go-libp2p-circuit v0.1.1 h1:eopfG9fAg6rEHWQO1TSrLosXDgYbbbu/RTva/tBANus= -github.com/libp2p/go-libp2p-circuit v0.1.1/go.mod h1:Ahq4cY3V9VJcHcn1SBXjr78AbFkZeIRmfunbA7pmFh8= +github.com/libp2p/go-libp2p-circuit v0.1.2 h1:m7FkqcbveAyKJZESttlf4+b+6lZqVTYe0v2Csw7ZErc= +github.com/libp2p/go-libp2p-circuit v0.1.2/go.mod h1:nKVMY/Vp+zcuEGuEoFzRhzb/dMqs+1g3Jxd9qssM5RM= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyLCcvo1w= @@ -194,6 +194,7 @@ github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZ github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= +github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= @@ -252,6 +253,7 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= +github.com/smola/gocompat v0.2.0 h1:6b1oIMlUXIpz//VKEDzPVBK8KG7beVwmHIUEBIs/Pns= github.com/smola/gocompat v0.2.0/go.mod h1:1B0MlxbmoZNo3h8guHp8HztB3BSYR5itql9qtVc0ypY= github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a h1:/eS3yfGjQKG+9kayBkj0ip1BGhq6zJ3eaVksphxAaek= github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= @@ -332,6 +334,7 @@ golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c h1:vamGzbGri8IKo20MQncCuljcQ5uAO6kaCeawQPVblAI= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= From 7b972a034cfc4cbd5ebac3794d76e9c7ded8e228 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 18:12:23 -0700 Subject: [PATCH 1495/3965] move multiaddr protocol definitions to go-multiaddr That way, users don't need to import the transport just to parse a websocket multiaddr. --- p2p/transport/websocket/websocket.go | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 972681a648..75c87f46d8 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -3,7 +3,6 @@ package websocket import ( "context" - "fmt" "net" "net/http" "net/url" @@ -20,11 +19,9 @@ import ( ) // WsProtocol is the multiaddr protocol definition for this transport. -var WsProtocol = ma.Protocol{ - Code: 477, - Name: "ws", - VCode: ma.CodeToVarint(477), -} +// +// Deprecated: use `ma.ProtocolWithCode(ma.P_WS) +var WsProtocol = ma.ProtocolWithCode(ma.P_WS) // WsFmt is multiaddr formatter for WsProtocol var WsFmt = mafmt.And(mafmt.TCP, mafmt.Base(WsProtocol.Code)) @@ -46,11 +43,6 @@ var upgrader = ws.Upgrader{ } func init() { - err := ma.AddProtocol(WsProtocol) - if err != nil { - panic(fmt.Errorf("error registering websocket protocol: %s", err)) - } - manet.RegisterNetCodec(WsCodec) } From 6de8743b2b88565b989cf620e185b10cb563c144 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 18:41:46 -0700 Subject: [PATCH 1496/3965] dep: update and switch to go-multiaddr-fmt --- p2p/transport/websocket/websocket.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 75c87f46d8..941edac6b8 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -14,8 +14,8 @@ import ( ws "github.com/gorilla/websocket" ma "github.com/multiformats/go-multiaddr" + mafmt "github.com/multiformats/go-multiaddr-fmt" manet "github.com/multiformats/go-multiaddr-net" - mafmt "github.com/whyrusleeping/mafmt" ) // WsProtocol is the multiaddr protocol definition for this transport. From b8ac63238e27381b944a02df61fba0c04460347e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 22:42:53 -0700 Subject: [PATCH 1497/3965] dep: update multiaddr for protocol migration Protocol definitions have been migrated to the main go-multiaddr package. --- p2p/net/swarm/limiter_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/limiter_test.go b/p2p/net/swarm/limiter_test.go index 32d87fa13d..7373ddbb3e 100644 --- a/p2p/net/swarm/limiter_test.go +++ b/p2p/net/swarm/limiter_test.go @@ -12,7 +12,7 @@ import ( "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/transport" ma "github.com/multiformats/go-multiaddr" - mafmt "github.com/whyrusleeping/mafmt" + mafmt "github.com/multiformats/go-multiaddr-fmt" ) func mustAddr(t *testing.T, s string) ma.Multiaddr { From 3b3e910fb354a7f24f26f0075ebd14b3e5f92ef0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 22:52:43 -0700 Subject: [PATCH 1498/3965] move protocol definitions to go-multiaddr see https://github.com/multiformats/go-multiaddr/issues/108 --- .../internal/circuitv1-deprecated/transport.go | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport.go b/p2p/protocol/internal/circuitv1-deprecated/transport.go index 89ade492ac..082a61c21c 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport.go @@ -11,21 +11,13 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -const P_CIRCUIT = 290 +// Deprecated: use ma.P_CIRCUIT +const P_CIRCUIT = ma.P_CIRCUIT -var Protocol = ma.Protocol{ - Code: P_CIRCUIT, - Size: 0, - Name: "p2p-circuit", - VCode: ma.CodeToVarint(P_CIRCUIT), -} - -var circuitAddr ma.Multiaddr +// Deprecated: use ma.ProtocolWithCode(P_CIRCUIT) +var Protocol = ma.ProtocolWithCode(P_CIRCUIT) -func init() { - ma.AddProtocol(Protocol) - circuitAddr = ma.Cast(Protocol.VCode) -} +var circuitAddr = ma.Cast(Protocol.VCode) var _ transport.Transport = (*RelayTransport)(nil) From 1938505f5a722d1c5759158b462b58d140dadbb5 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Sep 2019 23:02:37 -0700 Subject: [PATCH 1499/3965] dep: update go-multiaddr for https://github.com/multiformats/go-multiaddr/issues/108 --- go.mod | 16 ++++++++-------- go.sum | 28 ++++++++++++++++++++++++---- p2p/host/relay/addrsplosion.go | 3 +-- p2p/host/relay/addrsplosion_test.go | 1 - 4 files changed, 33 insertions(+), 15 deletions(-) diff --git a/go.mod b/go.mod index 6907a3711a..47126c3c08 100644 --- a/go.mod +++ b/go.mod @@ -11,8 +11,8 @@ require ( github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 github.com/libp2p/go-libp2p-autonat v0.1.0 - github.com/libp2p/go-libp2p-blankhost v0.1.3 - github.com/libp2p/go-libp2p-circuit v0.1.2 + github.com/libp2p/go-libp2p-blankhost v0.1.4 + github.com/libp2p/go-libp2p-circuit v0.1.3 github.com/libp2p/go-libp2p-core v0.2.2 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 @@ -21,18 +21,18 @@ require ( github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.1.3 github.com/libp2p/go-libp2p-secio v0.2.0 - github.com/libp2p/go-libp2p-swarm v0.2.1 + github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.0 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 github.com/libp2p/go-libp2p-yamux v0.2.1 github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 - github.com/libp2p/go-tcp-transport v0.1.0 - github.com/libp2p/go-ws-transport v0.1.0 + github.com/libp2p/go-tcp-transport v0.1.1 + github.com/libp2p/go-ws-transport v0.1.1 github.com/miekg/dns v1.1.12 // indirect - github.com/multiformats/go-multiaddr v0.0.4 - github.com/multiformats/go-multiaddr-dns v0.0.2 - github.com/multiformats/go-multiaddr-net v0.0.1 + github.com/multiformats/go-multiaddr v0.1.1 + github.com/multiformats/go-multiaddr-dns v0.1.0 + github.com/multiformats/go-multiaddr-net v0.1.0 github.com/multiformats/go-multistream v0.1.0 github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 ) diff --git a/go.sum b/go.sum index 0f4afa8328..0362dd67c1 100644 --- a/go.sum +++ b/go.sum @@ -121,8 +121,12 @@ github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3Pt github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.3 h1:0KycuXvPDhmehw0ASsg+s1o3IfXgCUDqfzAl94KEBOg= github.com/libp2p/go-libp2p-blankhost v0.1.3/go.mod h1:KML1//wiKR8vuuJO0y3LUd1uLv+tlkGTAr3jC0S5cLg= +github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= +github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= github.com/libp2p/go-libp2p-circuit v0.1.2 h1:m7FkqcbveAyKJZESttlf4+b+6lZqVTYe0v2Csw7ZErc= github.com/libp2p/go-libp2p-circuit v0.1.2/go.mod h1:nKVMY/Vp+zcuEGuEoFzRhzb/dMqs+1g3Jxd9qssM5RM= +github.com/libp2p/go-libp2p-circuit v0.1.3 h1:WsMYYaA0PwdpgJSQu12EzPYf5ypkLSTgcOsWr7DYrgI= +github.com/libp2p/go-libp2p-circuit v0.1.3/go.mod h1:Xqh2TjSy8DD5iV2cCOMzdynd6h8OTBGoV1AWbWor3qM= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyLCcvo1w= @@ -152,8 +156,8 @@ github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBg github.com/libp2p/go-libp2p-secio v0.2.0 h1:ywzZBsWEEz2KNTn5RtzauEDq5RFEefPsttXYwAWqHng= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= -github.com/libp2p/go-libp2p-swarm v0.2.1 h1:9A8oQqPIZvbaRyrjViHeDYS7fE7fNtP7BRWdJrBHbe8= -github.com/libp2p/go-libp2p-swarm v0.2.1/go.mod h1:x07b4zkMFo2EvgPV2bMTlNmdQc8i+74Jjio7xGvsTgU= +github.com/libp2p/go-libp2p-swarm v0.2.2 h1:T4hUpgEs2r371PweU3DuH7EOmBIdTBCwWs+FLcgx3bQ= +github.com/libp2p/go-libp2p-swarm v0.2.2/go.mod h1:fvmtQ0T1nErXym1/aa1uJEyN7JzaTNyBcHImCxRpPKU= github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.4 h1:Qev57UR47GcLPXWjrunv5aLIQGO4n9mhI/8/EIrEEFc= @@ -188,8 +192,10 @@ github.com/libp2p/go-stream-muxer-multistream v0.2.0 h1:714bRJ4Zy9mdhyTLJ+ZKiROm github.com/libp2p/go-stream-muxer-multistream v0.2.0/go.mod h1:j9eyPol/LLRqT+GPLSxvimPhNph4sfYfMoDPd7HkzIc= github.com/libp2p/go-tcp-transport v0.1.0 h1:IGhowvEqyMFknOar4FWCKSWE0zL36UFKQtiRQD60/8o= github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= -github.com/libp2p/go-ws-transport v0.1.0 h1:F+0OvvdmPTDsVc4AjPHjV7L7Pk1B7D5QwtDcKE2oag4= -github.com/libp2p/go-ws-transport v0.1.0/go.mod h1:rjw1MG1LU9YDC6gzmwObkPd/Sqwhw7yT74kj3raBFuo= +github.com/libp2p/go-tcp-transport v0.1.1 h1:yGlqURmqgNA2fvzjSgZNlHcsd/IulAnKM8Ncu+vlqnw= +github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfjg2pWP97dFZworkY= +github.com/libp2p/go-ws-transport v0.1.1 h1:uYIT3tWc5mIJB2P+9kJ1rgOVCmsO1oW1VXoTmCiDRa0= +github.com/libp2p/go-ws-transport v0.1.1/go.mod h1:eoxG1EQUq9NC6TipLQwbOhyP80mg4gIbiAU3Atbml00= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= @@ -211,6 +217,8 @@ github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+ github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/minio/sha256-simd v0.1.0 h1:U41/2erhAKcmSI14xh/ZTUdBPOzDOIfS93ibzUSl8KM= github.com/minio/sha256-simd v0.1.0/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= +github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771 h1:MHkK1uRtFbVqvAgvWxafZe54+5uBxLluGylDiKgdhwo= +github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= @@ -223,18 +231,29 @@ github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lg github.com/multiformats/go-multiaddr v0.0.2/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr v0.0.4 h1:WgMSI84/eRLdbptXMkMWDXPjPq7SPLIgGUVm2eroyU4= github.com/multiformats/go-multiaddr v0.0.4/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.1.0/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.1.1 h1:rVAztJYMhCQ7vEFr8FvxW3mS+HF2eY/oPbOMeS0ZDnE= +github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= +github.com/multiformats/go-multiaddr-dns v0.1.0 h1:gsPeMvo91XvcsNlQXgJgfjYjbsVV99bvveguUvDBpyQ= +github.com/multiformats/go-multiaddr-dns v0.1.0/go.mod h1:01k2RAqtoXIuPa3DCavAE9/6jc6nM0H3EgZyfUhN2oY= github.com/multiformats/go-multiaddr-fmt v0.0.1 h1:5YjeOIzbX8OTKVaN72aOzGIYW7PnrZrnkDyOfAWRSMA= github.com/multiformats/go-multiaddr-fmt v0.0.1/go.mod h1:aBYjqL4T/7j4Qx+R73XSv/8JsgnRFlf0w2KGLCmXl3Q= +github.com/multiformats/go-multiaddr-fmt v0.1.0 h1:WLEFClPycPkp4fnIzoFoV9FVd49/eQsuaL3/CWe167E= +github.com/multiformats/go-multiaddr-fmt v0.1.0/go.mod h1:hGtDIW4PU4BqJ50gW2quDuPVjyWNZxToGUh/HwTZYJo= github.com/multiformats/go-multiaddr-net v0.0.1 h1:76O59E3FavvHqNg7jvzWzsPSW5JSi/ek0E4eiDVbg9g= github.com/multiformats/go-multiaddr-net v0.0.1/go.mod h1:nw6HSxNmCIQH27XPGBuX+d1tnvM7ihcFwHMSstNAVUU= +github.com/multiformats/go-multiaddr-net v0.1.0 h1:ZepO8Ezwovd+7b5XPPDhQhayk1yt0AJpzQBpq9fejx4= +github.com/multiformats/go-multiaddr-net v0.1.0/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= github.com/multiformats/go-multihash v0.0.5 h1:1wxmCvTXAifAepIMyF39vZinRw5sbqjPs/UIi93+uik= github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= +github.com/multiformats/go-multihash v0.0.8 h1:wrYcW5yxSi3dU07n5jnuS5PrNwyHy0zRHGVoUugWvXg= +github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ= github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= @@ -298,6 +317,7 @@ golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnf golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 h1:IcSOAf4PyMp3U3XbIEj1/xJ2BjNN2jWv7JoyOsMxXUU= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= diff --git a/p2p/host/relay/addrsplosion.go b/p2p/host/relay/addrsplosion.go index 89965bca41..3b0e6355dc 100644 --- a/p2p/host/relay/addrsplosion.go +++ b/p2p/host/relay/addrsplosion.go @@ -5,7 +5,6 @@ import ( circuit "github.com/libp2p/go-libp2p-circuit" ma "github.com/multiformats/go-multiaddr" - dns "github.com/multiformats/go-multiaddr-dns" manet "github.com/multiformats/go-multiaddr-net" ) @@ -56,7 +55,7 @@ func isRelayAddr(a ma.Multiaddr) bool { func isDNSAddr(a ma.Multiaddr) bool { if first, _ := ma.SplitFirst(a); first != nil { switch first.Protocol().Code { - case dns.P_DNS4, dns.P_DNS6, dns.P_DNSADDR: + case ma.P_DNS4, ma.P_DNS6, ma.P_DNSADDR: return true } } diff --git a/p2p/host/relay/addrsplosion_test.go b/p2p/host/relay/addrsplosion_test.go index cdb10def63..394f26e59f 100644 --- a/p2p/host/relay/addrsplosion_test.go +++ b/p2p/host/relay/addrsplosion_test.go @@ -4,7 +4,6 @@ import ( "testing" ma "github.com/multiformats/go-multiaddr" - _ "github.com/multiformats/go-multiaddr-dns" ) func TestCleanupAddrs(t *testing.T) { From d5faed8b70e6b28fcbf9fee28d7418ee4e1209e5 Mon Sep 17 00:00:00 2001 From: adam Date: Thu, 19 Sep 2019 15:16:52 -0700 Subject: [PATCH 1500/3965] adds ability to create keys from crypto.PrivateKey types --- core/crypto/key.go | 42 +++++++++++++++++ core/crypto/key_test.go | 99 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+) diff --git a/core/crypto/key.go b/core/crypto/key.go index 6cd9dc0beb..5a0058f3cf 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -4,9 +4,12 @@ package crypto import ( + "crypto" + "crypto/ecdsa" "crypto/elliptic" "crypto/hmac" "crypto/rand" + "crypto/rsa" "crypto/sha1" "crypto/sha512" "crypto/subtle" @@ -15,11 +18,14 @@ import ( "fmt" "hash" "io" + "log" pb "github.com/libp2p/go-libp2p-core/crypto/pb" + btcec "github.com/btcsuite/btcd/btcec" "github.com/gogo/protobuf/proto" sha256 "github.com/minio/sha256-simd" + "golang.org/x/crypto/ed25519" ) const ( @@ -174,6 +180,42 @@ func GenerateEKeyPair(curveName string) ([]byte, GenSharedKey, error) { return pubKey, done, nil } +// KeyPairFromKey generates a new private and public key from an input private key +func KeyPairFromKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { + if priv == nil { + return nil, nil, ErrNilPrivateKey + } + + switch priv.(type) { + case *rsa.PrivateKey: + log.Println("1") + rPriv, _ := priv.(*rsa.PrivateKey) + return &RsaPrivateKey{*rPriv}, &RsaPublicKey{rPriv.PublicKey}, nil + + case *ecdsa.PrivateKey: + log.Println("2") + ePriv, _ := priv.(*ecdsa.PrivateKey) + return &ECDSAPrivateKey{ePriv}, &ECDSAPublicKey{&ePriv.PublicKey}, nil + + case *ed25519.PrivateKey: + log.Println("3") + ePriv, _ := priv.(*ed25519.PrivateKey) + pubIfc := ePriv.Public() + pub, _ := pubIfc.(ed25519.PublicKey) + return &Ed25519PrivateKey{*ePriv}, &Ed25519PublicKey{pub}, nil + + case *btcec.PrivateKey: + log.Println("4") + bPriv, _ := priv.(*btcec.PrivateKey) + sPriv := Secp256k1PrivateKey(*bPriv) + sPub := Secp256k1PublicKey(*bPriv.PubKey()) + return &sPriv, &sPub, nil + + default: + return nil, nil, ErrBadKeyType + } +} + // StretchedKeys ... type StretchedKeys struct { IV []byte diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index 05e66ed301..470d067b84 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -2,12 +2,20 @@ package crypto_test import ( "bytes" + "crypto" + "crypto/ecdsa" + "crypto/elliptic" "crypto/rand" + "crypto/rsa" + "fmt" "testing" + btcec "github.com/btcsuite/btcd/btcec" . "github.com/libp2p/go-libp2p-core/crypto" pb "github.com/libp2p/go-libp2p-core/crypto/pb" "github.com/libp2p/go-libp2p-core/test" + sha256 "github.com/minio/sha256-simd" + "golang.org/x/crypto/ed25519" ) func TestKeys(t *testing.T) { @@ -16,6 +24,97 @@ func TestKeys(t *testing.T) { } } +func TestKeyPairFromKey(t *testing.T) { + var ( + data = []byte(`hello world`) + hashed = sha256.Sum256(data) + ) + + privk, err := btcec.NewPrivateKey(btcec.S256()) + if err != nil { + t.Fatalf("err generating btcec priv key:\n%v", err) + } + sigK, err := privk.Sign(hashed[:]) + if err != nil { + t.Fatalf("err generating btcec sig:\n%v", err) + } + + eKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatalf("err generating ecdsa priv key:\n%v", err) + } + sigE, err := eKey.Sign(rand.Reader, hashed[:], crypto.SHA256) + if err != nil { + t.Fatalf("err generating ecdsa sig:\n%v", err) + } + + rKey, err := rsa.GenerateKey(rand.Reader, 2048) + if err != nil { + t.Fatalf("err generating rsa priv key:\n%v", err) + } + sigR, err := rKey.Sign(rand.Reader, hashed[:], crypto.SHA256) + if err != nil { + t.Fatalf("err generating rsa sig:\n%v", err) + } + + _, edKey, err := ed25519.GenerateKey(rand.Reader) + sigEd := ed25519.Sign(edKey, data[:]) + if err != nil { + t.Fatalf("err generating ed25519 sig:\n%v", err) + } + + for i, tt := range []struct { + in crypto.PrivateKey + typ pb.KeyType + sig []byte + }{ + { + eKey, + ECDSA, + sigE, + }, + { + privk, + Secp256k1, + sigK.Serialize(), + }, + { + rKey, + RSA, + sigR, + }, + { + &edKey, + Ed25519, + sigEd, + }, + } { + t.Run(fmt.Sprintf("%v", i), func(t *testing.T) { + priv, pub, err := KeyPairFromKey(tt.in) + if err != nil { + t.Fatal(err) + } + + if priv == nil || pub == nil { + t.Errorf("received nil private key or public key: %v, %v", priv, pub) + } + + if priv == nil || priv.Type() != tt.typ { + t.Errorf("want %v; got %v", tt.typ, priv.Type()) + } + + v, err := pub.Verify(data[:], tt.sig) + if err != nil { + t.Error(err) + } + + if !v { + t.Error("signature was not verified") + } + }) + } +} + func testKeyType(typ int, t *testing.T) { bits := 512 if typ == RSA { From 9f0b201ce13132e54401eb213370fba7b926741b Mon Sep 17 00:00:00 2001 From: adam Date: Thu, 19 Sep 2019 16:21:08 -0700 Subject: [PATCH 1501/3965] incorporates code review from @Stebalien --- core/crypto/key.go | 44 +------------------------------ core/crypto/key_not_openssl.go | 40 +++++++++++++++++++++++++++++ core/crypto/key_openssl.go | 47 ++++++++++++++++++++++++++++++++++ 3 files changed, 88 insertions(+), 43 deletions(-) create mode 100644 core/crypto/key_not_openssl.go create mode 100644 core/crypto/key_openssl.go diff --git a/core/crypto/key.go b/core/crypto/key.go index 5a0058f3cf..04ac48bf56 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -1,15 +1,12 @@ -// Package crypto implements various cryptographic utilities used by libp2p. +// Cackage crypto implements various cryptographic utilities used by libp2p. // This includes a Public and Private key interface and key implementations // for supported key algorithms. package crypto import ( - "crypto" - "crypto/ecdsa" "crypto/elliptic" "crypto/hmac" "crypto/rand" - "crypto/rsa" "crypto/sha1" "crypto/sha512" "crypto/subtle" @@ -18,14 +15,11 @@ import ( "fmt" "hash" "io" - "log" pb "github.com/libp2p/go-libp2p-core/crypto/pb" - btcec "github.com/btcsuite/btcd/btcec" "github.com/gogo/protobuf/proto" sha256 "github.com/minio/sha256-simd" - "golang.org/x/crypto/ed25519" ) const ( @@ -180,42 +174,6 @@ func GenerateEKeyPair(curveName string) ([]byte, GenSharedKey, error) { return pubKey, done, nil } -// KeyPairFromKey generates a new private and public key from an input private key -func KeyPairFromKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { - if priv == nil { - return nil, nil, ErrNilPrivateKey - } - - switch priv.(type) { - case *rsa.PrivateKey: - log.Println("1") - rPriv, _ := priv.(*rsa.PrivateKey) - return &RsaPrivateKey{*rPriv}, &RsaPublicKey{rPriv.PublicKey}, nil - - case *ecdsa.PrivateKey: - log.Println("2") - ePriv, _ := priv.(*ecdsa.PrivateKey) - return &ECDSAPrivateKey{ePriv}, &ECDSAPublicKey{&ePriv.PublicKey}, nil - - case *ed25519.PrivateKey: - log.Println("3") - ePriv, _ := priv.(*ed25519.PrivateKey) - pubIfc := ePriv.Public() - pub, _ := pubIfc.(ed25519.PublicKey) - return &Ed25519PrivateKey{*ePriv}, &Ed25519PublicKey{pub}, nil - - case *btcec.PrivateKey: - log.Println("4") - bPriv, _ := priv.(*btcec.PrivateKey) - sPriv := Secp256k1PrivateKey(*bPriv) - sPub := Secp256k1PublicKey(*bPriv.PubKey()) - return &sPriv, &sPub, nil - - default: - return nil, nil, ErrBadKeyType - } -} - // StretchedKeys ... type StretchedKeys struct { IV []byte diff --git a/core/crypto/key_not_openssl.go b/core/crypto/key_not_openssl.go new file mode 100644 index 0000000000..9b85aefcd8 --- /dev/null +++ b/core/crypto/key_not_openssl.go @@ -0,0 +1,40 @@ +// +build !openssl + +package crypto + +import ( + "crypto" + "crypto/ecdsa" + "crypto/rsa" + + btcec "github.com/btcsuite/btcd/btcec" + "golang.org/x/crypto/ed25519" +) + +// KeyPairFromKey generates a new private and public key from an input private key +func KeyPairFromKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { + if priv == nil { + return nil, nil, ErrNilPrivateKey + } + + switch p := priv.(type) { + case *rsa.PrivateKey: + return &RsaPrivateKey{*p}, &RsaPublicKey{p.PublicKey}, nil + + case *ecdsa.PrivateKey: + return &ECDSAPrivateKey{p}, &ECDSAPublicKey{&p.PublicKey}, nil + + case *ed25519.PrivateKey: + pubIfc := p.Public() + pub, _ := pubIfc.(ed25519.PublicKey) + return &Ed25519PrivateKey{*p}, &Ed25519PublicKey{pub}, nil + + case *btcec.PrivateKey: + sPriv := Secp256k1PrivateKey(*p) + sPub := Secp256k1PublicKey(*p.PubKey()) + return &sPriv, &sPub, nil + + default: + return nil, nil, ErrBadKeyType + } +} diff --git a/core/crypto/key_openssl.go b/core/crypto/key_openssl.go new file mode 100644 index 0000000000..3eae98b957 --- /dev/null +++ b/core/crypto/key_openssl.go @@ -0,0 +1,47 @@ +// +build openssl + +package crypto + +import ( + "crypto" + "crypto/ecdsa" + "crypto/rsa" + "crypto/x509" + + btcec "github.com/btcsuite/btcd/btcec" + openssl "github.com/libp2p/go-openssl" + "golang.org/x/crypto/ed25519" +) + +// KeyPairFromKey generates a new private and public key from an input private key +func KeyPairFromKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { + if priv == nil { + return nil, nil, ErrNilPrivateKey + } + + switch p := priv.(type) { + case *rsa.PrivateKey: + pk, err := openssl.LoadPrivateKeyFromDER(x509.MarshalPKCS1PrivateKey(p)) + if err != nil { + return nil, nil, err + } + + return &opensslPrivateKey{pk}, &opensslPublicKey{pk}, nil + + case *ecdsa.PrivateKey: + return &ECDSAPrivateKey{p}, &ECDSAPublicKey{&p.PublicKey}, nil + + case *ed25519.PrivateKey: + pubIfc := p.Public() + pub, _ := pubIfc.(ed25519.PublicKey) + return &Ed25519PrivateKey{*p}, &Ed25519PublicKey{pub}, nil + + case *btcec.PrivateKey: + sPriv := Secp256k1PrivateKey(*p) + sPub := Secp256k1PublicKey(*p.PubKey()) + return &sPriv, &sPub, nil + + default: + return nil, nil, ErrBadKeyType + } +} From 4eb22d0411bf0279774d75c8a5a869a84931db07 Mon Sep 17 00:00:00 2001 From: adam Date: Thu, 19 Sep 2019 16:24:24 -0700 Subject: [PATCH 1502/3965] fixes typo --- core/crypto/key.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/crypto/key.go b/core/crypto/key.go index 04ac48bf56..6cd9dc0beb 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -1,4 +1,4 @@ -// Cackage crypto implements various cryptographic utilities used by libp2p. +// Package crypto implements various cryptographic utilities used by libp2p. // This includes a Public and Private key interface and key implementations // for supported key algorithms. package crypto From 3082b657a762cb8f35094a5ed5f03413e5e02274 Mon Sep 17 00:00:00 2001 From: adam Date: Fri, 20 Sep 2019 23:37:38 -0700 Subject: [PATCH 1503/3965] incorporates code review comments from @raulk; changes KeyPairFromKey to KeyPairFromStdKey and improves godoc. --- core/crypto/key_not_openssl.go | 4 ++-- core/crypto/key_openssl.go | 4 ++-- core/crypto/key_test.go | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/core/crypto/key_not_openssl.go b/core/crypto/key_not_openssl.go index 9b85aefcd8..ae8b01f00b 100644 --- a/core/crypto/key_not_openssl.go +++ b/core/crypto/key_not_openssl.go @@ -11,8 +11,8 @@ import ( "golang.org/x/crypto/ed25519" ) -// KeyPairFromKey generates a new private and public key from an input private key -func KeyPairFromKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { +// KeyPairFromStdKey wraps standard library (and secp256k1) private keys in libp2p/go-libp2p-core/crypto keys +func KeyPairFromStdKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { if priv == nil { return nil, nil, ErrNilPrivateKey } diff --git a/core/crypto/key_openssl.go b/core/crypto/key_openssl.go index 3eae98b957..4545492cf3 100644 --- a/core/crypto/key_openssl.go +++ b/core/crypto/key_openssl.go @@ -13,8 +13,8 @@ import ( "golang.org/x/crypto/ed25519" ) -// KeyPairFromKey generates a new private and public key from an input private key -func KeyPairFromKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { +// KeyPairFromStdKey wraps standard library (and secp256k1) private keys in libp2p/go-libp2p-core/crypto keys +func KeyPairFromStdKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { if priv == nil { return nil, nil, ErrNilPrivateKey } diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index 470d067b84..aff2badd4c 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -90,7 +90,7 @@ func TestKeyPairFromKey(t *testing.T) { }, } { t.Run(fmt.Sprintf("%v", i), func(t *testing.T) { - priv, pub, err := KeyPairFromKey(tt.in) + priv, pub, err := KeyPairFromStdKey(tt.in) if err != nil { t.Fatal(err) } From a7bc619a1a9fc97819ee821e263cbc3e3c8aaa57 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 24 Sep 2019 10:31:52 -0700 Subject: [PATCH 1504/3965] crypto: use constant time compare when decoding private keys In practice, this is impossible to exploit without being able to corrupt the private key which would allow a much simpler guess-and-check attack. However, it's still a bad practice to compare private key material like this. --- core/crypto/ed25519.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/crypto/ed25519.go b/core/crypto/ed25519.go index e8834d4818..231dadb035 100644 --- a/core/crypto/ed25519.go +++ b/core/crypto/ed25519.go @@ -132,7 +132,7 @@ func UnmarshalEd25519PrivateKey(data []byte) (PrivKey, error) { // Remove the redundant public key. See issue #36. redundantPk := data[ed25519.PrivateKeySize:] pk := data[ed25519.PrivateKeySize-ed25519.PublicKeySize : ed25519.PrivateKeySize] - if !bytes.Equal(pk, redundantPk) { + if subtle.ConstantTimeCompare(pk, redundantPk) == 0 { return nil, errors.New("expected redundant ed25519 public key to be redundant") } From b2f4537e5faaa8013280584e4b96152efc1d3ec6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 24 Sep 2019 10:34:34 -0700 Subject: [PATCH 1505/3965] crypto: make the ECDSA receiver consistent --- core/crypto/ecdsa.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/crypto/ecdsa.go b/core/crypto/ecdsa.go index 42ce5d816b..0767274644 100644 --- a/core/crypto/ecdsa.go +++ b/core/crypto/ecdsa.go @@ -155,7 +155,7 @@ func (ePub *ECDSAPublicKey) Type() pb.KeyType { } // Raw returns x509 bytes from a public key -func (ePub ECDSAPublicKey) Raw() ([]byte, error) { +func (ePub *ECDSAPublicKey) Raw() ([]byte, error) { return x509.MarshalPKIXPublicKey(ePub.pub) } From 4f475ba73f72a734a67287cb5b4d898d1e2aeb87 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 24 Sep 2019 10:34:39 -0700 Subject: [PATCH 1506/3965] crypto: use basicEquals for ECDSA The check was incomplete as it didn't test the curve. This switches us to use basicEquals (which is also constant-time). Note: This key type isn't used by anyone in-practice (to the best of my knowledge). --- core/crypto/ecdsa.go | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-) diff --git a/core/crypto/ecdsa.go b/core/crypto/ecdsa.go index 0767274644..9ea769513a 100644 --- a/core/crypto/ecdsa.go +++ b/core/crypto/ecdsa.go @@ -117,12 +117,7 @@ func (ePriv *ECDSAPrivateKey) Raw() ([]byte, error) { // Equals compares two private keys func (ePriv *ECDSAPrivateKey) Equals(o Key) bool { - oPriv, ok := o.(*ECDSAPrivateKey) - if !ok { - return basicEquals(ePriv, o) - } - - return ePriv.priv.D.Cmp(oPriv.priv.D) == 0 + return basicEquals(ePriv, o) } // Sign returns the signature of the input data @@ -161,13 +156,7 @@ func (ePub *ECDSAPublicKey) Raw() ([]byte, error) { // Equals compares to public keys func (ePub *ECDSAPublicKey) Equals(o Key) bool { - oPub, ok := o.(*ECDSAPublicKey) - if !ok { - return basicEquals(ePub, o) - } - - return ePub.pub.X != nil && ePub.pub.Y != nil && oPub.pub.X != nil && oPub.pub.Y != nil && - 0 == ePub.pub.X.Cmp(oPub.pub.X) && 0 == ePub.pub.Y.Cmp(oPub.pub.Y) + return basicEquals(ePub, o) } // Verify compares data to a signature From afa8538210d0c6ef38a8b425737df1474769ec16 Mon Sep 17 00:00:00 2001 From: Preston Van Loon Date: Tue, 24 Sep 2019 15:37:22 -0700 Subject: [PATCH 1507/3965] Update autonat.proto --- p2p/host/autonat/pb/autonat.proto | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/host/autonat/pb/autonat.proto b/p2p/host/autonat/pb/autonat.proto index 7107e1c07f..777270a139 100644 --- a/p2p/host/autonat/pb/autonat.proto +++ b/p2p/host/autonat/pb/autonat.proto @@ -1,3 +1,5 @@ +syntax = "proto2"; + package autonat.pb; message Message { From 1e889e1867d967da88ccc16420896fdd39b1d4de Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 25 Sep 2019 11:31:45 -0700 Subject: [PATCH 1508/3965] dep: update final dep updates before a release --- go.mod | 10 +++++----- go.sum | 51 +++++++++++++++++++++++++++++++++++++-------------- 2 files changed, 42 insertions(+), 19 deletions(-) diff --git a/go.mod b/go.mod index 47126c3c08..5f909db544 100644 --- a/go.mod +++ b/go.mod @@ -1,8 +1,8 @@ module github.com/libp2p/go-libp2p require ( - github.com/gogo/protobuf v1.2.1 - github.com/ipfs/go-cid v0.0.2 + github.com/gogo/protobuf v1.3.0 + github.com/ipfs/go-cid v0.0.3 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 github.com/ipfs/go-log v0.0.1 @@ -13,7 +13,7 @@ require ( github.com/libp2p/go-libp2p-autonat v0.1.0 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.3 - github.com/libp2p/go-libp2p-core v0.2.2 + github.com/libp2p/go-libp2p-core v0.2.3 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 @@ -28,13 +28,13 @@ require ( github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 - github.com/libp2p/go-ws-transport v0.1.1 + github.com/libp2p/go-ws-transport v0.1.2 github.com/miekg/dns v1.1.12 // indirect github.com/multiformats/go-multiaddr v0.1.1 github.com/multiformats/go-multiaddr-dns v0.1.0 github.com/multiformats/go-multiaddr-net v0.1.0 github.com/multiformats/go-multistream v0.1.0 - github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 + github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 ) go 1.12 diff --git a/go.sum b/go.sum index 0362dd67c1..90200587e2 100644 --- a/go.sum +++ b/go.sum @@ -8,6 +8,8 @@ github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5 github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32/go.mod h1:DrZx5ec/dmnfpw9KyYoQyYo7d0KEvTkk/5M/vbZjAr8= github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c h1:aEbSeNALREWXk0G7UdNhR3ayBV7tZ4M2PNmnrCAph6Q= github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI= +github.com/btcsuite/btcd v0.0.0-20190824003749-130ea5bddde3 h1:A/EVblehb75cUgXA5njHPn0kLAsykn6mJGz7rnmW5W0= +github.com/btcsuite/btcd v0.0.0-20190824003749-130ea5bddde3/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI= github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA= github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= @@ -37,7 +39,11 @@ github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMo github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98= github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE= github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= +github.com/gogo/protobuf v1.3.0 h1:G8O7TerXerS4F6sx9OV7/nRfJdnXgHZu/S/7F2SN+UE= +github.com/gogo/protobuf v1.3.0/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0= @@ -46,10 +52,12 @@ github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5y github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q= -github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= +github.com/gorilla/websocket v1.4.1 h1:q7AeDBpnBk8AogcD4DSag/Ukw/KV+YhzLj2bP5HvKCM= +github.com/gorilla/websocket v1.4.1/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= @@ -65,6 +73,8 @@ github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANyt github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-cid v0.0.2 h1:tuuKaZPU1M6HcejsO3AcYWW8sZ8MTvyxfc4uqB4eFE8= github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= +github.com/ipfs/go-cid v0.0.3 h1:UIAh32wymBpStoe83YCzwVQQ5Oy/H0FdxvUS6DJDzms= +github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= @@ -93,6 +103,7 @@ github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJS github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d/go.mod h1:P2viExyCEfeWGU259JnaQ34Inuec4R38JCyBx2edgD0= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= +github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= @@ -110,8 +121,6 @@ github.com/libp2p/go-buffer-pool v0.0.2 h1:QNK2iAFa8gjAe1SPz6mHSMuCcjs+X1wlHzeOS github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM= github.com/libp2p/go-conn-security-multistream v0.1.0 h1:aqGmto+ttL/uJgX0JtQI0tD21CIEy5eYd1Hlp0juHY0= github.com/libp2p/go-conn-security-multistream v0.1.0/go.mod h1:aw6eD7LOsHEX7+2hJkDxw1MteijaVcI+/eP2/x3J1xc= -github.com/libp2p/go-eventbus v0.0.2 h1:L9eslON8FjFBJlyUs9fyEZKnxSqZd2AMDUNldPrqmZI= -github.com/libp2p/go-eventbus v0.0.2/go.mod h1:Hr/yGlwxA/stuLnpMiu82lpNKpvRy3EaJxPu40XYOwk= github.com/libp2p/go-eventbus v0.1.0 h1:mlawomSAjjkk97QnYiEmHsLu7E136+2oCWSHRUvMfzQ= github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UGQf8A2eEmx4= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= @@ -119,22 +128,18 @@ github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZ github.com/libp2p/go-libp2p-autonat v0.1.0 h1:aCWAu43Ri4nU0ZPO7NyLzUvvfqd0nE3dX0R/ZGYVgOU= github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3PtvYlJmg7NeVtPRqH8= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= -github.com/libp2p/go-libp2p-blankhost v0.1.3 h1:0KycuXvPDhmehw0ASsg+s1o3IfXgCUDqfzAl94KEBOg= -github.com/libp2p/go-libp2p-blankhost v0.1.3/go.mod h1:KML1//wiKR8vuuJO0y3LUd1uLv+tlkGTAr3jC0S5cLg= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= -github.com/libp2p/go-libp2p-circuit v0.1.2 h1:m7FkqcbveAyKJZESttlf4+b+6lZqVTYe0v2Csw7ZErc= -github.com/libp2p/go-libp2p-circuit v0.1.2/go.mod h1:nKVMY/Vp+zcuEGuEoFzRhzb/dMqs+1g3Jxd9qssM5RM= github.com/libp2p/go-libp2p-circuit v0.1.3 h1:WsMYYaA0PwdpgJSQu12EzPYf5ypkLSTgcOsWr7DYrgI= github.com/libp2p/go-libp2p-circuit v0.1.3/go.mod h1:Xqh2TjSy8DD5iV2cCOMzdynd6h8OTBGoV1AWbWor3qM= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= -github.com/libp2p/go-libp2p-core v0.0.6 h1:SsYhfWJ47vLP1Rd9/0hqEm/W/PlFbC/3YLZyLCcvo1w= -github.com/libp2p/go-libp2p-core v0.0.6/go.mod h1:0d9xmaYAVY5qmbp/fcgxHT3ZJsLjYeYPMJAUKpaCHrE= github.com/libp2p/go-libp2p-core v0.2.0 h1:ycFtuNwtZBAJSxzaHbyv6NjG3Yj5Nmra1csHaQ3zwaw= github.com/libp2p/go-libp2p-core v0.2.0/go.mod h1:X0eyB0Gy93v0DZtSYbEM7RnMChm9Uv3j7yRXjO77xSI= github.com/libp2p/go-libp2p-core v0.2.2 h1:Sv1ggdoMx9c7v7FOFkR7agraHCnAgqYsXrU1ARSRUMs= github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZVoVrLaDRvIRng0= +github.com/libp2p/go-libp2p-core v0.2.3 h1:zXikZ5pLfebtTMeIYfcwVQ2Pae77O0FIwDquwM6AGNM= +github.com/libp2p/go-libp2p-core v0.2.3/go.mod h1:GqhyQqyIAPsxFYXHMjfXgMv03lxsvM0mFzuYA9Ib42A= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= @@ -194,8 +199,8 @@ github.com/libp2p/go-tcp-transport v0.1.0 h1:IGhowvEqyMFknOar4FWCKSWE0zL36UFKQti github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= github.com/libp2p/go-tcp-transport v0.1.1 h1:yGlqURmqgNA2fvzjSgZNlHcsd/IulAnKM8Ncu+vlqnw= github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfjg2pWP97dFZworkY= -github.com/libp2p/go-ws-transport v0.1.1 h1:uYIT3tWc5mIJB2P+9kJ1rgOVCmsO1oW1VXoTmCiDRa0= -github.com/libp2p/go-ws-transport v0.1.1/go.mod h1:eoxG1EQUq9NC6TipLQwbOhyP80mg4gIbiAU3Atbml00= +github.com/libp2p/go-ws-transport v0.1.2 h1:VnxQcLfSGtqupqPpBNu8fUiCv+IN1RJ2BcVqQEM+z8E= +github.com/libp2p/go-ws-transport v0.1.2/go.mod h1:dsh2Ld8F+XNmzpkaAijmg5Is+e9l6/1tK/6VFOdN69Y= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= @@ -219,6 +224,8 @@ github.com/minio/sha256-simd v0.1.0 h1:U41/2erhAKcmSI14xh/ZTUdBPOzDOIfS93ibzUSl8 github.com/minio/sha256-simd v0.1.0/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771 h1:MHkK1uRtFbVqvAgvWxafZe54+5uBxLluGylDiKgdhwo= github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= +github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU= +github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= @@ -301,14 +308,16 @@ github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f h1:M/lL3 github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f/go.mod h1:cZNvX9cFybI01GriPRMXDtczuvUhgbcYr9iCGaNlRv8= github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= -github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30 h1:nMCC9Pwz1pxfC1Y6mYncdk+kq8d5aLx0Q+/gyZGE44M= -github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= +github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 h1:Y1/FEOpaCpD21WxrmfeIYCFPuVPRCY2XZTWzTNHGw30= +github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= github.com/x-cray/logrus-prefixed-formatter v0.5.2/go.mod h1:2duySbKsL6M18s5GU7VPsoEPHyzalCE06qoARUCeBBE= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= +go.opencensus.io v0.22.1 h1:8dP3SGL7MPB94crU3bEPplMPe83FI4EouesJUeFHv50= +go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= @@ -320,9 +329,12 @@ golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 h1:IcSOAf4PyMp3U3XbIEj1/xJ2BjNN2jWv7JoyOsMxXUU= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190923035154-9ee001bba392 h1:ACG4HJsFiNMf47Y4PeRoebLNy/2lXT9EtprMuTFWt1M= +golang.org/x/crypto v0.0.0-20190923035154-9ee001bba392/go.mod h1:/lpIB1dKB+9EgE3H3cr1v9wB50oz8l4C4h62xy7jSTY= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -347,21 +359,32 @@ golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 h1:rOhMmluY6kLMhdnrivzec6lLgaVbMHMn2ISQXJeJ5EM= +golang.org/x/sys v0.0.0-20190922100055-0a153f010e69/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c h1:vamGzbGri8IKo20MQncCuljcQ5uAO6kaCeawQPVblAI= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= From 7f87ef83e4234117e95479ee510fdbbe1852940a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 26 Sep 2019 10:07:42 -0700 Subject: [PATCH 1509/3965] fix: close gracefully If we send no message, the client gets a 1005. This will send a 1000. --- p2p/transport/websocket/conn.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/transport/websocket/conn.go b/p2p/transport/websocket/conn.go index b63a28a9ed..d7ae0c194c 100644 --- a/p2p/transport/websocket/conn.go +++ b/p2p/transport/websocket/conn.go @@ -84,7 +84,11 @@ func (c *Conn) Write(b []byte) (n int, err error) { func (c *Conn) Close() error { var err error c.closeOnce.Do(func() { - err1 := c.Conn.WriteControl(ws.CloseMessage, nil, time.Now().Add(GracefulCloseTimeout)) + err1 := c.Conn.WriteControl( + ws.CloseMessage, + ws.FormatCloseMessage(ws.CloseNormalClosure, "closed"), + time.Now().Add(GracefulCloseTimeout), + ) err2 := c.Conn.Close() switch { case err1 != nil: From 32a893742b494545ecbdfb9bcaa9aa4b795efc22 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Fri, 27 Sep 2019 17:14:16 +0200 Subject: [PATCH 1510/3965] Pass context to dial and do not hardcode it --- p2p/net/gostream/conn.go | 5 +---- p2p/net/gostream/gostream_test.go | 2 +- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/p2p/net/gostream/conn.go b/p2p/net/gostream/conn.go index 8ecc24b7e8..84f27c23bd 100644 --- a/p2p/net/gostream/conn.go +++ b/p2p/net/gostream/conn.go @@ -79,10 +79,7 @@ func (c *conn) SetWriteDeadline(t time.Time) error { // Dial opens a stream to the destination address // (which should parseable to a peer ID) using the given // host and returns it as a standard net.Conn. -func Dial(h host.Host, pid peer.ID, tag protocol.ID) (net.Conn, error) { - ctx, cancel := context.WithTimeout(context.Background(), time.Second*30) - defer cancel() - +func Dial(ctx context.Context, h host.Host, pid peer.ID, tag protocol.ID) (net.Conn, error) { s, err := h.NewStream(ctx, pid, tag) if err != nil { return nil, err diff --git a/p2p/net/gostream/gostream_test.go b/p2p/net/gostream/gostream_test.go index b74a85a474..e25d4e1982 100644 --- a/p2p/net/gostream/gostream_test.go +++ b/p2p/net/gostream/gostream_test.go @@ -80,7 +80,7 @@ func TestServerClient(t *testing.T) { } }(ctx) - clientConn, err := Dial(clientHost, srvHost.ID(), tag) + clientConn, err := Dial(ctx, clientHost, srvHost.ID(), tag) if err != nil { t.Fatal(err) } From 5e9fc8e39e4be302786f4891d4d2000edd49f78d Mon Sep 17 00:00:00 2001 From: Mike Goelzer Date: Mon, 30 Sep 2019 09:58:09 -0700 Subject: [PATCH 1511/3965] Comment that KeyStretcher is pre-deprecation --- core/crypto/key.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/crypto/key.go b/core/crypto/key.go index 6cd9dc0beb..bdb6f089b5 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -181,6 +181,8 @@ type StretchedKeys struct { CipherKey []byte } +// PENDING DEPRECATION: KeyStretcher() will be deprecated with secio; for new +// code, please use PBKDF2 (golang.org/x/crypto/pbkdf2) instead. // KeyStretcher returns a set of keys for each party by stretching the shared key. // (myIV, theirIV, myCipherKey, theirCipherKey, myMACKey, theirMACKey). // This function accepts the following cipher types: From a7676729b15f27dc42b281d611ea668e40dcc9fc Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Wed, 7 Aug 2019 18:34:15 -0700 Subject: [PATCH 1512/3965] Implement bare bones Wasm support (needs to be cleaned up) --- .../websocket/browser_integration_go_test.go | 80 +++++++ .../websocket/browser_integration_js_test.go | 62 +++++ p2p/transport/websocket/conn.go | 121 ---------- p2p/transport/websocket/conn_go.go | 129 +++++++++++ p2p/transport/websocket/conn_js.go | 212 ++++++++++++++++++ p2p/transport/websocket/listener.go | 2 + p2p/transport/websocket/websocket.go | 92 +------- p2p/transport/websocket/websocket_go.go | 97 ++++++++ p2p/transport/websocket/websocket_js.go | 43 ++++ p2p/transport/websocket/websocket_test.go | 2 + 10 files changed, 629 insertions(+), 211 deletions(-) create mode 100644 p2p/transport/websocket/browser_integration_go_test.go create mode 100644 p2p/transport/websocket/browser_integration_js_test.go create mode 100644 p2p/transport/websocket/conn_go.go create mode 100644 p2p/transport/websocket/conn_js.go create mode 100644 p2p/transport/websocket/websocket_go.go create mode 100644 p2p/transport/websocket/websocket_js.go diff --git a/p2p/transport/websocket/browser_integration_go_test.go b/p2p/transport/websocket/browser_integration_go_test.go new file mode 100644 index 0000000000..1b82b4da3d --- /dev/null +++ b/p2p/transport/websocket/browser_integration_go_test.go @@ -0,0 +1,80 @@ +// +build !js + +package websocket + +import ( + "bufio" + "fmt" + "sync" + "testing" + + "github.com/libp2p/go-libp2p-core/sec/insecure" + mplex "github.com/libp2p/go-libp2p-mplex" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + ma "github.com/multiformats/go-multiaddr" +) + +const ( + testServerPort = ":9714" +) + +// TestInBrowser is a harness that allows us to use `go test` in order to run +// WebAssembly tests in a headless browser. +func TestInBrowser(t *testing.T) { + // Start a transport which the browser peer will dial. + wg := &sync.WaitGroup{} + wg.Add(1) + go func() { + defer wg.Done() + tpt := New(&tptu.Upgrader{ + Secure: insecure.New("serverPeer"), + Muxer: new(mplex.Transport), + }) + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") + if err != nil { + t.Fatal(err) + } + listener, err := tpt.Listen(addr) + if err != nil { + t.Fatal(err) + } + fmt.Println("server started listener") + conn, err := listener.Accept() + if err != nil { + t.Fatal(err) + } + defer conn.Close() + fmt.Println("server received conn") + stream, err := conn.OpenStream() + if err != nil { + fmt.Println("Could not open stream:", err.Error()) + t.Fatal(err) + } + defer stream.Close() + fmt.Println("server opened stream") + buf := bufio.NewReader(stream) + if _, err := stream.Write([]byte("ping\n")); err != nil { + t.Fatal(err) + } + fmt.Println("server wrote message") + msg, err := buf.ReadString('\n') + if err != nil { + t.Fatal("could not read pong message:" + err.Error()) + } + fmt.Println("server received message") + expected := "pong\n" + if msg != expected { + t.Fatalf("Received wrong message. Expected %q but got %q", expected, msg) + } + }() + + // fmt.Println("Starting headless browser tests...") + // cmd := exec.Command("go", "test", "-exec", `"$GOPATH/bin/wasmbrowsertest"`, "-run", "TestInBrowser", ".") + // cmd.Env = append(os.Environ(), []string{"GOOS=js", "GOARCH=wasm"}...) + // if output, err := cmd.CombinedOutput(); err != nil { + // t.Log(string(output)) + // t.Fatal(err) + // } + + wg.Wait() +} diff --git a/p2p/transport/websocket/browser_integration_js_test.go b/p2p/transport/websocket/browser_integration_js_test.go new file mode 100644 index 0000000000..5842e305df --- /dev/null +++ b/p2p/transport/websocket/browser_integration_js_test.go @@ -0,0 +1,62 @@ +// +build js,wasm + +package websocket + +import ( + "bufio" + "context" + "fmt" + "testing" + "time" + + "github.com/libp2p/go-libp2p-core/sec/insecure" + mplex "github.com/libp2p/go-libp2p-mplex" + tptu "github.com/libp2p/go-libp2p-transport-upgrader" + ma "github.com/multiformats/go-multiaddr" +) + +func TestInBrowser(t *testing.T) { + tpt := New(&tptu.Upgrader{ + Secure: insecure.New("browserPeer"), + Muxer: new(mplex.Transport), + }) + addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") + if err != nil { + t.Fatal("could not parse multiaddress:" + err.Error()) + } + fmt.Println("parsed addr") + conn, err := tpt.Dial(context.Background(), addr, "serverPeer") + if err != nil { + t.Fatal("could not dial server:" + err.Error()) + } + fmt.Println("dialed server") + defer conn.Close() + + stream, err := conn.AcceptStream() + if err != nil { + t.Fatal("could not accept stream:" + err.Error()) + } + fmt.Println("accepted stream") + defer stream.Close() + + buf := bufio.NewReader(stream) + msg, err := buf.ReadString('\n') + if err != nil { + t.Fatal("could not read ping message:" + err.Error()) + } + fmt.Println("read ping") + expected := "ping\n" + if msg != expected { + t.Fatalf("Received wrong message. Expected %q but got %q", expected, msg) + } + + _, err = stream.Write([]byte("pong\n")) + if err != nil { + t.Fatal("could not write pong message:" + err.Error()) + } + fmt.Println("wrote pong") + + // TODO(albrow): This hack is necessary in order to give the reader time to + // finish. We should find some way to remove it. + time.Sleep(1 * time.Second) +} diff --git a/p2p/transport/websocket/conn.go b/p2p/transport/websocket/conn.go index d7ae0c194c..b100b44598 100644 --- a/p2p/transport/websocket/conn.go +++ b/p2p/transport/websocket/conn.go @@ -1,12 +1,8 @@ package websocket import ( - "io" "net" - "sync" "time" - - ws "github.com/gorilla/websocket" ) // GracefulCloseTimeout is the time to wait trying to gracefully close a @@ -14,120 +10,3 @@ import ( var GracefulCloseTimeout = 100 * time.Millisecond var _ net.Conn = (*Conn)(nil) - -// Conn implements net.Conn interface for gorilla/websocket. -type Conn struct { - *ws.Conn - DefaultMessageType int - reader io.Reader - closeOnce sync.Once -} - -func (c *Conn) Read(b []byte) (int, error) { - if c.reader == nil { - if err := c.prepNextReader(); err != nil { - return 0, err - } - } - - for { - n, err := c.reader.Read(b) - switch err { - case io.EOF: - c.reader = nil - - if n > 0 { - return n, nil - } - - if err := c.prepNextReader(); err != nil { - return 0, err - } - - // explicitly looping - default: - return n, err - } - } -} - -func (c *Conn) prepNextReader() error { - t, r, err := c.Conn.NextReader() - if err != nil { - if wserr, ok := err.(*ws.CloseError); ok { - if wserr.Code == 1000 || wserr.Code == 1005 { - return io.EOF - } - } - return err - } - - if t == ws.CloseMessage { - return io.EOF - } - - c.reader = r - return nil -} - -func (c *Conn) Write(b []byte) (n int, err error) { - if err := c.Conn.WriteMessage(c.DefaultMessageType, b); err != nil { - return 0, err - } - - return len(b), nil -} - -// Close closes the connection. Only the first call to Close will receive the -// close error, subsequent and concurrent calls will return nil. -// This method is thread-safe. -func (c *Conn) Close() error { - var err error - c.closeOnce.Do(func() { - err1 := c.Conn.WriteControl( - ws.CloseMessage, - ws.FormatCloseMessage(ws.CloseNormalClosure, "closed"), - time.Now().Add(GracefulCloseTimeout), - ) - err2 := c.Conn.Close() - switch { - case err1 != nil: - err = err1 - case err2 != nil: - err = err2 - } - }) - return err -} - -func (c *Conn) LocalAddr() net.Addr { - return NewAddr(c.Conn.LocalAddr().String()) -} - -func (c *Conn) RemoteAddr() net.Addr { - return NewAddr(c.Conn.RemoteAddr().String()) -} - -func (c *Conn) SetDeadline(t time.Time) error { - if err := c.SetReadDeadline(t); err != nil { - return err - } - - return c.SetWriteDeadline(t) -} - -func (c *Conn) SetReadDeadline(t time.Time) error { - return c.Conn.SetReadDeadline(t) -} - -func (c *Conn) SetWriteDeadline(t time.Time) error { - return c.Conn.SetWriteDeadline(t) -} - -// NewConn creates a Conn given a regular gorilla/websocket Conn. -func NewConn(raw *ws.Conn) *Conn { - return &Conn{ - Conn: raw, - DefaultMessageType: ws.BinaryMessage, - } -} diff --git a/p2p/transport/websocket/conn_go.go b/p2p/transport/websocket/conn_go.go new file mode 100644 index 0000000000..6be4697ec6 --- /dev/null +++ b/p2p/transport/websocket/conn_go.go @@ -0,0 +1,129 @@ +// +build !js + +package websocket + +import ( + "io" + "net" + "sync" + "time" + + ws "github.com/gorilla/websocket" +) + +// Conn implements net.Conn interface for gorilla/websocket. +type Conn struct { + *ws.Conn + DefaultMessageType int + reader io.Reader + closeOnce sync.Once +} + +func (c *Conn) Read(b []byte) (int, error) { + if c.reader == nil { + if err := c.prepNextReader(); err != nil { + return 0, err + } + } + + for { + n, err := c.reader.Read(b) + switch err { + case io.EOF: + c.reader = nil + + if n > 0 { + return n, nil + } + + if err := c.prepNextReader(); err != nil { + return 0, err + } + + // explicitly looping + default: + return n, err + } + } +} + +func (c *Conn) prepNextReader() error { + t, r, err := c.Conn.NextReader() + if err != nil { + if wserr, ok := err.(*ws.CloseError); ok { + if wserr.Code == 1000 || wserr.Code == 1005 { + return io.EOF + } + } + return err + } + + if t == ws.CloseMessage { + return io.EOF + } + + c.reader = r + return nil +} + +func (c *Conn) Write(b []byte) (n int, err error) { + if err := c.Conn.WriteMessage(c.DefaultMessageType, b); err != nil { + return 0, err + } + + return len(b), nil +} + +// Close closes the connection. Only the first call to Close will receive the +// close error, subsequent and concurrent calls will return nil. +// This method is thread-safe. +func (c *Conn) Close() error { + var err error + c.closeOnce.Do(func() { + err1 := c.Conn.WriteControl( + ws.CloseMessage, + ws.FormatCloseMessage(ws.CloseNormalClosure, "closed"), + time.Now().Add(GracefulCloseTimeout), + ) + err2 := c.Conn.Close() + switch { + case err1 != nil: + err = err1 + case err2 != nil: + err = err2 + } + }) + return err +} + +func (c *Conn) LocalAddr() net.Addr { + return NewAddr(c.Conn.LocalAddr().String()) +} + +func (c *Conn) RemoteAddr() net.Addr { + return NewAddr(c.Conn.RemoteAddr().String()) +} + +func (c *Conn) SetDeadline(t time.Time) error { + if err := c.SetReadDeadline(t); err != nil { + return err + } + + return c.SetWriteDeadline(t) +} + +func (c *Conn) SetReadDeadline(t time.Time) error { + return c.Conn.SetReadDeadline(t) +} + +func (c *Conn) SetWriteDeadline(t time.Time) error { + return c.Conn.SetWriteDeadline(t) +} + +// NewConn creates a Conn given a regular gorilla/websocket Conn. +func NewConn(raw *ws.Conn) *Conn { + return &Conn{ + Conn: raw, + DefaultMessageType: ws.BinaryMessage, + } +} diff --git a/p2p/transport/websocket/conn_js.go b/p2p/transport/websocket/conn_js.go new file mode 100644 index 0000000000..cb8b3248d3 --- /dev/null +++ b/p2p/transport/websocket/conn_js.go @@ -0,0 +1,212 @@ +package websocket + +import ( + "bytes" + "errors" + "fmt" + "io" + "net" + "strings" + "sync" + "syscall/js" + "time" +) + +const ( + webSocketStateConnecting = 0 + webSocketStateOpen = 1 + webSocketStateClosing = 2 + webSocketStateClosed = 3 +) + +var errIsClosed = errors.New("connection is closed") + +// Conn implements net.Conn interface for WebSockets in js/wasm. +type Conn struct { + js.Value + messageHandler *js.Func + closeHandler *js.Func + mut sync.Mutex + incomingData chan []byte + currData *bytes.Buffer + closeSignal chan struct{} +} + +func (c *Conn) Read(b []byte) (int, error) { + c.mut.Lock() + defer c.mut.Unlock() + if err := c.checkOpen(); err != nil { + return 0, io.EOF + } + + if c.currData == nil { + fmt.Println("Waiting for incoming data") + select { + case data := <-c.incomingData: + fmt.Println("Received new incoming data") + fmt.Println(string(data)) + c.currData = bytes.NewBuffer(data) + case <-c.closeSignal: + return 0, io.EOF + } + } + + n, err := c.currData.Read(b) + if err == io.EOF { + c.currData = nil + return n, nil + } + return n, err +} + +func (c *Conn) checkOpen() error { + state := c.Get("readyState").Int() + switch state { + case webSocketStateOpen: + fmt.Println("readyState is open") + case webSocketStateClosed, webSocketStateClosing: + fmt.Println("readyState is closed or closing") + return errIsClosed + default: + fmt.Printf("readyState is %d\n", state) + } + return nil +} + +func (c *Conn) Write(b []byte) (n int, err error) { + fmt.Println("Write was called") + // c.mut.Lock() + // defer c.mut.Unlock() + if err := c.checkOpen(); err != nil { + return 0, err + } + c.Call("send", string(b)) + return len(b), nil +} + +// Close closes the connection. Only the first call to Close will receive the +// close error, subsequent and concurrent calls will return nil. +// This method is thread-safe. +func (c *Conn) Close() error { + // c.mut.Lock() + // defer c.mut.Unlock() + c.Call("close") + if c.messageHandler != nil { + c.Call("removeEventListener", "message", *c.messageHandler) + c.messageHandler.Release() + } + if c.closeHandler != nil { + c.Call("removeEventListener", "close", *c.closeHandler) + c.closeHandler.Release() + } + return nil +} + +func (c *Conn) LocalAddr() net.Addr { + // TODO(albrow): is there some way to get the local address? + return NewAddr("0.0.0.0:0") +} + +func (c *Conn) RemoteAddr() net.Addr { + rawURL := c.Get("url").String() + withoutPrefix := strings.TrimPrefix(rawURL, "ws://") + withoutSuffix := strings.TrimSuffix(withoutPrefix, "/") + return NewAddr(withoutSuffix) +} + +func (c *Conn) SetDeadline(t time.Time) error { + // Not supported + return nil +} + +func (c *Conn) SetReadDeadline(t time.Time) error { + // Not supported + return nil +} + +func (c *Conn) SetWriteDeadline(t time.Time) error { + // Not supported + return nil +} + +// NewConn creates a Conn given a regular js/wasm WebSocket Conn. +func NewConn(raw js.Value) *Conn { + conn := &Conn{ + Value: raw, + incomingData: make(chan []byte), + closeSignal: make(chan struct{}), + } + conn.setUpHandlers() + return conn +} + +func (c *Conn) setUpHandlers() { + // c.mut.Lock() + // defer c.mut.Unlock() + if c.messageHandler != nil { + // Message handlers already created. Nothing to do. + return + } + messageHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { + go func() { + // TODO(albrow): Currently we assume data is of type Blob. Really we + // should check binaryType and then decode accordingly. + blob := args[0].Get("data") + text := readBlob(blob) + fmt.Println("onmessage was triggered") + fmt.Println("received:", text) + fmt.Println("Sending to incomingData") + c.incomingData <- []byte(text) + fmt.Println("Sent to incomingData") + }() + return nil + }) + c.messageHandler = &messageHandler + c.Call("addEventListener", "message", messageHandler) + + closeHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { + fmt.Println("onclose was triggered") + close(c.closeSignal) + return nil + }) + c.closeHandler = &closeHandler + c.Call("addEventListener", "close", closeHandler) +} + +func (c *Conn) waitForOpen() error { + openSignal := make(chan struct{}) + handler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { + fmt.Println("onopen was triggered") + close(openSignal) + return nil + }) + defer func() { + c.Call("removeEventListener", "open", handler) + }() + defer handler.Release() + c.Call("addEventListener", "open", handler) + <-openSignal + return nil +} + +func readBlob(blob js.Value) string { + // const reader = new FileReader(); + // reader.addEventListener('loadend', (e) => { + // const text = e.srcElement.result; + // console.log(text); + // }); + // reader.readAsText(blb); + reader := js.Global().Get("FileReader").New() + textChan := make(chan string) + loadEndFunc := js.FuncOf(func(this js.Value, args []js.Value) interface{} { + go func() { + text := args[0].Get("srcElement").Get("result").String() + textChan <- text + }() + return nil + }) + defer loadEndFunc.Release() + reader.Call("addEventListener", "loadend", loadEndFunc) + reader.Call("readAsText", blob) + return <-textChan +} diff --git a/p2p/transport/websocket/listener.go b/p2p/transport/websocket/listener.go index 517c719aac..47fb8e7129 100644 --- a/p2p/transport/websocket/listener.go +++ b/p2p/transport/websocket/listener.go @@ -1,3 +1,5 @@ +// +build !js + package websocket import ( diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 941edac6b8..7e1c900f27 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -3,16 +3,10 @@ package websocket import ( "context" - "net" - "net/http" - "net/url" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/transport" - tptu "github.com/libp2p/go-libp2p-transport-upgrader" - - ws "github.com/gorilla/websocket" ma "github.com/multiformats/go-multiaddr" mafmt "github.com/multiformats/go-multiaddr-fmt" manet "github.com/multiformats/go-multiaddr-net" @@ -34,18 +28,12 @@ var WsCodec = &manet.NetCodec{ ParseNetAddr: ParseWebsocketNetAddr, } -// Default gorilla upgrader -var upgrader = ws.Upgrader{ - // Allow requests from *all* origins. - CheckOrigin: func(r *http.Request) bool { - return true - }, -} - func init() { manet.RegisterNetCodec(WsCodec) } +var _ transport.Transport = (*WebsocketTransport)(nil) + // WebsocketTransport is the actual go-libp2p transport type WebsocketTransport struct { Upgrader *tptu.Upgrader @@ -55,8 +43,6 @@ func New(u *tptu.Upgrader) *WebsocketTransport { return &WebsocketTransport{u} } -var _ transport.Transport = (*WebsocketTransport)(nil) - func (t *WebsocketTransport) CanDial(a ma.Multiaddr) bool { return WsFmt.Matches(a) } @@ -69,25 +55,6 @@ func (t *WebsocketTransport) Proxy() bool { return false } -func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { - wsurl, err := parseMultiaddr(raddr) - if err != nil { - return nil, err - } - - wscon, _, err := ws.DefaultDialer.Dial(wsurl, nil) - if err != nil { - return nil, err - } - - mnc, err := manet.WrapNetConn(NewConn(wscon)) - if err != nil { - wscon.Close() - return nil, err - } - return mnc, nil -} - func (t *WebsocketTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (transport.CapableConn, error) { macon, err := t.maDial(ctx, raddr) if err != nil { @@ -95,58 +62,3 @@ func (t *WebsocketTransport) Dial(ctx context.Context, raddr ma.Multiaddr, p pee } return t.Upgrader.UpgradeOutbound(ctx, t, macon, p) } - -func (t *WebsocketTransport) maListen(a ma.Multiaddr) (manet.Listener, error) { - lnet, lnaddr, err := manet.DialArgs(a) - if err != nil { - return nil, err - } - - nl, err := net.Listen(lnet, lnaddr) - if err != nil { - return nil, err - } - - u, err := url.Parse("http://" + nl.Addr().String()) - if err != nil { - nl.Close() - return nil, err - } - - malist, err := t.wrapListener(nl, u) - if err != nil { - nl.Close() - return nil, err - } - - go malist.serve() - - return malist, nil -} - -func (t *WebsocketTransport) Listen(a ma.Multiaddr) (transport.Listener, error) { - malist, err := t.maListen(a) - if err != nil { - return nil, err - } - return t.Upgrader.UpgradeListener(t, malist), nil -} - -func (t *WebsocketTransport) wrapListener(l net.Listener, origin *url.URL) (*listener, error) { - laddr, err := manet.FromNetAddr(l.Addr()) - if err != nil { - return nil, err - } - wsma, err := ma.NewMultiaddr("/ws") - if err != nil { - return nil, err - } - laddr = laddr.Encapsulate(wsma) - - return &listener{ - laddr: laddr, - Listener: l, - incoming: make(chan *Conn), - closed: make(chan struct{}), - }, nil -} diff --git a/p2p/transport/websocket/websocket_go.go b/p2p/transport/websocket/websocket_go.go new file mode 100644 index 0000000000..86b872c02f --- /dev/null +++ b/p2p/transport/websocket/websocket_go.go @@ -0,0 +1,97 @@ +// +build !js + +package websocket + +import ( + "context" + "net" + "net/http" + "net/url" + + ws "github.com/gorilla/websocket" + "github.com/libp2p/go-libp2p-core/transport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +// Default gorilla upgrader +var upgrader = ws.Upgrader{ + // Allow requests from *all* origins. + CheckOrigin: func(r *http.Request) bool { + return true + }, +} + +func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { + wsurl, err := parseMultiaddr(raddr) + if err != nil { + return nil, err + } + + wscon, _, err := ws.DefaultDialer.Dial(wsurl, nil) + if err != nil { + return nil, err + } + + mnc, err := manet.WrapNetConn(NewConn(wscon)) + if err != nil { + wscon.Close() + return nil, err + } + return mnc, nil +} + +func (t *WebsocketTransport) maListen(a ma.Multiaddr) (manet.Listener, error) { + lnet, lnaddr, err := manet.DialArgs(a) + if err != nil { + return nil, err + } + + nl, err := net.Listen(lnet, lnaddr) + if err != nil { + return nil, err + } + + u, err := url.Parse("http://" + nl.Addr().String()) + if err != nil { + nl.Close() + return nil, err + } + + malist, err := t.wrapListener(nl, u) + if err != nil { + nl.Close() + return nil, err + } + + go malist.serve() + + return malist, nil +} + +func (t *WebsocketTransport) Listen(a ma.Multiaddr) (transport.Listener, error) { + malist, err := t.maListen(a) + if err != nil { + return nil, err + } + return t.Upgrader.UpgradeListener(t, malist), nil +} + +func (t *WebsocketTransport) wrapListener(l net.Listener, origin *url.URL) (*listener, error) { + laddr, err := manet.FromNetAddr(l.Addr()) + if err != nil { + return nil, err + } + wsma, err := ma.NewMultiaddr("/ws") + if err != nil { + return nil, err + } + laddr = laddr.Encapsulate(wsma) + + return &listener{ + laddr: laddr, + Listener: l, + incoming: make(chan *Conn), + closed: make(chan struct{}), + }, nil +} diff --git a/p2p/transport/websocket/websocket_js.go b/p2p/transport/websocket/websocket_js.go new file mode 100644 index 0000000000..fd3c812490 --- /dev/null +++ b/p2p/transport/websocket/websocket_js.go @@ -0,0 +1,43 @@ +// +build js,wasm + +package websocket + +import ( + "context" + "errors" + "fmt" + "syscall/js" + + "github.com/libp2p/go-libp2p-core/transport" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { + wsurl, err := parseMultiaddr(raddr) + if err != nil { + return nil, err + } + fmt.Println("wsurl:", wsurl) + + rawConn := js.Global().Get("WebSocket").New(wsurl) + rawConn.Call("addEventListener", "error", js.FuncOf(func(this js.Value, args []js.Value) interface{} { + fmt.Println("onerror was triggered") + js.Global().Get("console").Call("log", args[0]) + return nil + })) + conn := NewConn(rawConn) + conn.waitForOpen() + mnc, err := manet.WrapNetConn(conn) + if err != nil { + conn.Close() + return nil, err + } + + fmt.Println("Returning from maDial") + return mnc, nil +} + +func (t *WebsocketTransport) Listen(a ma.Multiaddr) (transport.Listener, error) { + return nil, errors.New("Listen not implemented on js/wasm") +} diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index 4f397c8243..30872e94cd 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -1,3 +1,5 @@ +// +build !js + package websocket import ( From 486168bb29acf3e15eb0b7522b2b1b6e7351dea4 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Thu, 8 Aug 2019 15:04:58 -0700 Subject: [PATCH 1513/3965] Make it possible to run browser tests with a single go test command --- .../websocket/browser_integration_go_test.go | 50 +++++++++---------- .../websocket/browser_integration_js_test.go | 6 --- p2p/transport/websocket/conn_js.go | 16 ------ p2p/transport/websocket/websocket_js.go | 4 -- p2p/transport/websocket/websocket_test.go | 1 - 5 files changed, 25 insertions(+), 52 deletions(-) diff --git a/p2p/transport/websocket/browser_integration_go_test.go b/p2p/transport/websocket/browser_integration_go_test.go index 1b82b4da3d..464fca73a8 100644 --- a/p2p/transport/websocket/browser_integration_go_test.go +++ b/p2p/transport/websocket/browser_integration_go_test.go @@ -5,7 +5,10 @@ package websocket import ( "bufio" "fmt" - "sync" + "os" + "os/exec" + "path/filepath" + "strings" "testing" "github.com/libp2p/go-libp2p-core/sec/insecure" @@ -22,59 +25,56 @@ const ( // WebAssembly tests in a headless browser. func TestInBrowser(t *testing.T) { // Start a transport which the browser peer will dial. - wg := &sync.WaitGroup{} - wg.Add(1) + serverDoneSignal := make(chan struct{}) go func() { - defer wg.Done() + defer func() { + close(serverDoneSignal) + }() tpt := New(&tptu.Upgrader{ Secure: insecure.New("serverPeer"), Muxer: new(mplex.Transport), }) addr, err := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/5555/ws") if err != nil { - t.Fatal(err) + t.Fatal("SERVER:", err) } listener, err := tpt.Listen(addr) if err != nil { - t.Fatal(err) + t.Fatal("SERVER:", err) } - fmt.Println("server started listener") conn, err := listener.Accept() if err != nil { - t.Fatal(err) + t.Fatal("SERVER:", err) } defer conn.Close() - fmt.Println("server received conn") stream, err := conn.OpenStream() if err != nil { - fmt.Println("Could not open stream:", err.Error()) - t.Fatal(err) + t.Fatal("SERVER: could not open stream:", err) } defer stream.Close() - fmt.Println("server opened stream") buf := bufio.NewReader(stream) if _, err := stream.Write([]byte("ping\n")); err != nil { - t.Fatal(err) + t.Fatal("SERVER:", err) } - fmt.Println("server wrote message") msg, err := buf.ReadString('\n') if err != nil { - t.Fatal("could not read pong message:" + err.Error()) + t.Fatal("SERVER: could not read pong message:" + err.Error()) } - fmt.Println("server received message") expected := "pong\n" if msg != expected { - t.Fatalf("Received wrong message. Expected %q but got %q", expected, msg) + t.Fatalf("SERVER: Received wrong message. Expected %q but got %q", expected, msg) } }() - // fmt.Println("Starting headless browser tests...") - // cmd := exec.Command("go", "test", "-exec", `"$GOPATH/bin/wasmbrowsertest"`, "-run", "TestInBrowser", ".") - // cmd.Env = append(os.Environ(), []string{"GOOS=js", "GOARCH=wasm"}...) - // if output, err := cmd.CombinedOutput(); err != nil { - // t.Log(string(output)) - // t.Fatal(err) - // } + testExecPath := filepath.Join(os.Getenv("GOPATH"), "bin", "wasmbrowsertest") + cmd := exec.Command("go", "test", "-exec", testExecPath, "-run", "TestInBrowser", ".", "-v") + cmd.Env = append(os.Environ(), []string{"GOOS=js", "GOARCH=wasm"}...) + output, err := cmd.CombinedOutput() + if err != nil { + formattedOutput := "\t" + strings.Join(strings.Split(string(output), "\n"), "\n\t") + fmt.Println("BROWSER OUTPUT:\n", formattedOutput) + t.Fatal("BROWSER:", err) + } - wg.Wait() + <-serverDoneSignal } diff --git a/p2p/transport/websocket/browser_integration_js_test.go b/p2p/transport/websocket/browser_integration_js_test.go index 5842e305df..656db447ba 100644 --- a/p2p/transport/websocket/browser_integration_js_test.go +++ b/p2p/transport/websocket/browser_integration_js_test.go @@ -5,7 +5,6 @@ package websocket import ( "bufio" "context" - "fmt" "testing" "time" @@ -24,19 +23,16 @@ func TestInBrowser(t *testing.T) { if err != nil { t.Fatal("could not parse multiaddress:" + err.Error()) } - fmt.Println("parsed addr") conn, err := tpt.Dial(context.Background(), addr, "serverPeer") if err != nil { t.Fatal("could not dial server:" + err.Error()) } - fmt.Println("dialed server") defer conn.Close() stream, err := conn.AcceptStream() if err != nil { t.Fatal("could not accept stream:" + err.Error()) } - fmt.Println("accepted stream") defer stream.Close() buf := bufio.NewReader(stream) @@ -44,7 +40,6 @@ func TestInBrowser(t *testing.T) { if err != nil { t.Fatal("could not read ping message:" + err.Error()) } - fmt.Println("read ping") expected := "ping\n" if msg != expected { t.Fatalf("Received wrong message. Expected %q but got %q", expected, msg) @@ -54,7 +49,6 @@ func TestInBrowser(t *testing.T) { if err != nil { t.Fatal("could not write pong message:" + err.Error()) } - fmt.Println("wrote pong") // TODO(albrow): This hack is necessary in order to give the reader time to // finish. We should find some way to remove it. diff --git a/p2p/transport/websocket/conn_js.go b/p2p/transport/websocket/conn_js.go index cb8b3248d3..bd04f83772 100644 --- a/p2p/transport/websocket/conn_js.go +++ b/p2p/transport/websocket/conn_js.go @@ -3,7 +3,6 @@ package websocket import ( "bytes" "errors" - "fmt" "io" "net" "strings" @@ -40,11 +39,8 @@ func (c *Conn) Read(b []byte) (int, error) { } if c.currData == nil { - fmt.Println("Waiting for incoming data") select { case data := <-c.incomingData: - fmt.Println("Received new incoming data") - fmt.Println(string(data)) c.currData = bytes.NewBuffer(data) case <-c.closeSignal: return 0, io.EOF @@ -62,19 +58,13 @@ func (c *Conn) Read(b []byte) (int, error) { func (c *Conn) checkOpen() error { state := c.Get("readyState").Int() switch state { - case webSocketStateOpen: - fmt.Println("readyState is open") case webSocketStateClosed, webSocketStateClosing: - fmt.Println("readyState is closed or closing") return errIsClosed - default: - fmt.Printf("readyState is %d\n", state) } return nil } func (c *Conn) Write(b []byte) (n int, err error) { - fmt.Println("Write was called") // c.mut.Lock() // defer c.mut.Unlock() if err := c.checkOpen(); err != nil { @@ -153,11 +143,7 @@ func (c *Conn) setUpHandlers() { // should check binaryType and then decode accordingly. blob := args[0].Get("data") text := readBlob(blob) - fmt.Println("onmessage was triggered") - fmt.Println("received:", text) - fmt.Println("Sending to incomingData") c.incomingData <- []byte(text) - fmt.Println("Sent to incomingData") }() return nil }) @@ -165,7 +151,6 @@ func (c *Conn) setUpHandlers() { c.Call("addEventListener", "message", messageHandler) closeHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - fmt.Println("onclose was triggered") close(c.closeSignal) return nil }) @@ -176,7 +161,6 @@ func (c *Conn) setUpHandlers() { func (c *Conn) waitForOpen() error { openSignal := make(chan struct{}) handler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - fmt.Println("onopen was triggered") close(openSignal) return nil }) diff --git a/p2p/transport/websocket/websocket_js.go b/p2p/transport/websocket/websocket_js.go index fd3c812490..9002533e0c 100644 --- a/p2p/transport/websocket/websocket_js.go +++ b/p2p/transport/websocket/websocket_js.go @@ -5,7 +5,6 @@ package websocket import ( "context" "errors" - "fmt" "syscall/js" "github.com/libp2p/go-libp2p-core/transport" @@ -18,11 +17,9 @@ func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (ma if err != nil { return nil, err } - fmt.Println("wsurl:", wsurl) rawConn := js.Global().Get("WebSocket").New(wsurl) rawConn.Call("addEventListener", "error", js.FuncOf(func(this js.Value, args []js.Value) interface{} { - fmt.Println("onerror was triggered") js.Global().Get("console").Call("log", args[0]) return nil })) @@ -34,7 +31,6 @@ func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (ma return nil, err } - fmt.Println("Returning from maDial") return mnc, nil } diff --git a/p2p/transport/websocket/websocket_test.go b/p2p/transport/websocket/websocket_test.go index 30872e94cd..bada345232 100644 --- a/p2p/transport/websocket/websocket_test.go +++ b/p2p/transport/websocket/websocket_test.go @@ -11,7 +11,6 @@ import ( "testing/iotest" "github.com/libp2p/go-libp2p-core/sec/insecure" - mplex "github.com/libp2p/go-libp2p-mplex" ttransport "github.com/libp2p/go-libp2p-testing/suites/transport" tptu "github.com/libp2p/go-libp2p-transport-upgrader" From 4979f94d4e97b06f51c8c86e1bd8f5bddd0f0807 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Wed, 14 Aug 2019 15:42:56 -0700 Subject: [PATCH 1514/3965] Guarantee reading happens in order and add some comments --- p2p/transport/websocket/conn_js.go | 146 ++++++++++++++++++----------- 1 file changed, 90 insertions(+), 56 deletions(-) diff --git a/p2p/transport/websocket/conn_js.go b/p2p/transport/websocket/conn_js.go index bd04f83772..9bca3f3518 100644 --- a/p2p/transport/websocket/conn_js.go +++ b/p2p/transport/websocket/conn_js.go @@ -18,7 +18,9 @@ const ( webSocketStateClosed = 3 ) -var errIsClosed = errors.New("connection is closed") +const incomingDataBufferSize = 100 + +var errConnectionClosed = errors.New("connection is closed") // Conn implements net.Conn interface for WebSockets in js/wasm. type Conn struct { @@ -27,50 +29,78 @@ type Conn struct { closeHandler *js.Func mut sync.Mutex incomingData chan []byte - currData *bytes.Buffer + currDataMut sync.RWMutex + currData bytes.Buffer closeSignal chan struct{} + dataSignal chan struct{} +} + +// NewConn creates a Conn given a regular js/wasm WebSocket Conn. +func NewConn(raw js.Value) *Conn { + conn := &Conn{ + Value: raw, + incomingData: make(chan []byte, incomingDataBufferSize), + closeSignal: make(chan struct{}), + dataSignal: make(chan struct{}), + } + conn.setUpHandlers() + go func() { + // TODO(albrow): Handle error appropriately + err := conn.readLoop() + if err != nil { + panic(err) + } + }() + return conn } func (c *Conn) Read(b []byte) (int, error) { - c.mut.Lock() - defer c.mut.Unlock() if err := c.checkOpen(); err != nil { return 0, io.EOF } - if c.currData == nil { - select { - case data := <-c.incomingData: - c.currData = bytes.NewBuffer(data) - case <-c.closeSignal: - return 0, io.EOF + for { + c.currDataMut.RLock() + n, err := c.currData.Read(b) + c.currDataMut.RUnlock() + if err != nil && err != io.EOF { + // Return any unexpected errors immediately. + return n, err + } else if n == 0 || err == io.EOF { + // There is no data ready to be read. Wait for more data or for the + // connection to be closed. + select { + case <-c.dataSignal: + continue + case <-c.closeSignal: + return 0, io.EOF + } + } else { + return n, err } } - - n, err := c.currData.Read(b) - if err == io.EOF { - c.currData = nil - return n, nil - } - return n, err } +// checkOpen returns an error if the connection is not open. Otherwise, it +// returns nil. func (c *Conn) checkOpen() error { state := c.Get("readyState").Int() switch state { case webSocketStateClosed, webSocketStateClosing: - return errIsClosed + return errConnectionClosed } return nil } func (c *Conn) Write(b []byte) (n int, err error) { - // c.mut.Lock() - // defer c.mut.Unlock() if err := c.checkOpen(); err != nil { return 0, err } - c.Call("send", string(b)) + uint8Array := js.Global().Get("Uint8Array").New(len(b)) + for i := 0; i < len(b); i++ { + uint8Array.SetIndex(i, b[i]) + } + c.Call("send", uint8Array.Get("buffer")) return len(b), nil } @@ -78,8 +108,8 @@ func (c *Conn) Write(b []byte) (n int, err error) { // close error, subsequent and concurrent calls will return nil. // This method is thread-safe. func (c *Conn) Close() error { - // c.mut.Lock() - // defer c.mut.Unlock() + c.mut.Lock() + defer c.mut.Unlock() c.Call("close") if c.messageHandler != nil { c.Call("removeEventListener", "message", *c.messageHandler) @@ -93,7 +123,6 @@ func (c *Conn) Close() error { } func (c *Conn) LocalAddr() net.Addr { - // TODO(albrow): is there some way to get the local address? return NewAddr("0.0.0.0:0") } @@ -119,20 +148,9 @@ func (c *Conn) SetWriteDeadline(t time.Time) error { return nil } -// NewConn creates a Conn given a regular js/wasm WebSocket Conn. -func NewConn(raw js.Value) *Conn { - conn := &Conn{ - Value: raw, - incomingData: make(chan []byte), - closeSignal: make(chan struct{}), - } - conn.setUpHandlers() - return conn -} - func (c *Conn) setUpHandlers() { - // c.mut.Lock() - // defer c.mut.Unlock() + c.mut.Lock() + defer c.mut.Unlock() if c.messageHandler != nil { // Message handlers already created. Nothing to do. return @@ -142,8 +160,8 @@ func (c *Conn) setUpHandlers() { // TODO(albrow): Currently we assume data is of type Blob. Really we // should check binaryType and then decode accordingly. blob := args[0].Get("data") - text := readBlob(blob) - c.incomingData <- []byte(text) + data := readBlob(blob) + c.incomingData <- data }() return nil }) @@ -158,39 +176,55 @@ func (c *Conn) setUpHandlers() { c.Call("addEventListener", "close", closeHandler) } +// readLoop continuosly reads from the c.incoming data channel and writes to the +// current data buffer. +func (c *Conn) readLoop() error { + for data := range c.incomingData { + c.currDataMut.Lock() + _, err := c.currData.Write(data) + if err != nil { + c.currDataMut.Unlock() + return err + } + c.currDataMut.Unlock() + c.dataSignal <- struct{}{} + } + return nil +} + func (c *Conn) waitForOpen() error { openSignal := make(chan struct{}) handler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { close(openSignal) return nil }) - defer func() { - c.Call("removeEventListener", "open", handler) - }() + defer c.Call("removeEventListener", "open", handler) defer handler.Release() c.Call("addEventListener", "open", handler) <-openSignal return nil } -func readBlob(blob js.Value) string { - // const reader = new FileReader(); - // reader.addEventListener('loadend', (e) => { - // const text = e.srcElement.result; - // console.log(text); - // }); - // reader.readAsText(blb); +// readBlob converts a JavaScript Blob into a slice of bytes. It uses the +// FileReader API under the hood. +func readBlob(blob js.Value) []byte { reader := js.Global().Get("FileReader").New() - textChan := make(chan string) + dataChan := make(chan []byte) loadEndFunc := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - go func() { - text := args[0].Get("srcElement").Get("result").String() - textChan <- text - }() + // event.result is of type ArrayBuffer. We can convert it to a JavaScript + // Uint8Array, which is directly translatable to the Go type []byte. + buffer := reader.Get("result") + view := js.Global().Get("Uint8Array").New(buffer) + dataLen := view.Get("length").Int() + data := make([]byte, dataLen) + for i := 0; i < dataLen; i++ { + data[i] = byte(view.Index(i).Int()) + } + dataChan <- data return nil }) defer loadEndFunc.Release() reader.Call("addEventListener", "loadend", loadEndFunc) - reader.Call("readAsText", blob) - return <-textChan + reader.Call("readAsArrayBuffer", blob) + return <-dataChan } From ee5c8299d28ecfd7f8b9aff86e26469b082f7642 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Wed, 14 Aug 2019 15:49:47 -0700 Subject: [PATCH 1515/3965] return os.ErrNoDeadline where appropriate --- p2p/transport/websocket/conn_js.go | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/p2p/transport/websocket/conn_js.go b/p2p/transport/websocket/conn_js.go index 9bca3f3518..d8c8a9cdfb 100644 --- a/p2p/transport/websocket/conn_js.go +++ b/p2p/transport/websocket/conn_js.go @@ -5,6 +5,7 @@ import ( "errors" "io" "net" + "os" "strings" "sync" "syscall/js" @@ -134,18 +135,15 @@ func (c *Conn) RemoteAddr() net.Addr { } func (c *Conn) SetDeadline(t time.Time) error { - // Not supported - return nil + return os.ErrNoDeadline } func (c *Conn) SetReadDeadline(t time.Time) error { - // Not supported - return nil + return os.ErrNoDeadline } func (c *Conn) SetWriteDeadline(t time.Time) error { - // Not supported - return nil + return os.ErrNoDeadline } func (c *Conn) setUpHandlers() { From 31e5ba48ed86523337fb0da924391d5c2be45b8e Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Wed, 14 Aug 2019 15:57:36 -0700 Subject: [PATCH 1516/3965] Change go and js suffixes to native and browser --- ...tegration_js_test.go => browser_integration_browser_test.go} | 0 ...ntegration_go_test.go => browser_integration_native_test.go} | 0 p2p/transport/websocket/{conn_js.go => conn_browser.go} | 2 ++ p2p/transport/websocket/{conn_go.go => conn_native.go} | 0 .../websocket/{websocket_js.go => websocket_browser.go} | 0 .../websocket/{websocket_go.go => websocket_native.go} | 0 6 files changed, 2 insertions(+) rename p2p/transport/websocket/{browser_integration_js_test.go => browser_integration_browser_test.go} (100%) rename p2p/transport/websocket/{browser_integration_go_test.go => browser_integration_native_test.go} (100%) rename p2p/transport/websocket/{conn_js.go => conn_browser.go} (99%) rename p2p/transport/websocket/{conn_go.go => conn_native.go} (100%) rename p2p/transport/websocket/{websocket_js.go => websocket_browser.go} (100%) rename p2p/transport/websocket/{websocket_go.go => websocket_native.go} (100%) diff --git a/p2p/transport/websocket/browser_integration_js_test.go b/p2p/transport/websocket/browser_integration_browser_test.go similarity index 100% rename from p2p/transport/websocket/browser_integration_js_test.go rename to p2p/transport/websocket/browser_integration_browser_test.go diff --git a/p2p/transport/websocket/browser_integration_go_test.go b/p2p/transport/websocket/browser_integration_native_test.go similarity index 100% rename from p2p/transport/websocket/browser_integration_go_test.go rename to p2p/transport/websocket/browser_integration_native_test.go diff --git a/p2p/transport/websocket/conn_js.go b/p2p/transport/websocket/conn_browser.go similarity index 99% rename from p2p/transport/websocket/conn_js.go rename to p2p/transport/websocket/conn_browser.go index d8c8a9cdfb..76a51f6c39 100644 --- a/p2p/transport/websocket/conn_js.go +++ b/p2p/transport/websocket/conn_browser.go @@ -1,3 +1,5 @@ +// +build js,wasm + package websocket import ( diff --git a/p2p/transport/websocket/conn_go.go b/p2p/transport/websocket/conn_native.go similarity index 100% rename from p2p/transport/websocket/conn_go.go rename to p2p/transport/websocket/conn_native.go diff --git a/p2p/transport/websocket/websocket_js.go b/p2p/transport/websocket/websocket_browser.go similarity index 100% rename from p2p/transport/websocket/websocket_js.go rename to p2p/transport/websocket/websocket_browser.go diff --git a/p2p/transport/websocket/websocket_go.go b/p2p/transport/websocket/websocket_native.go similarity index 100% rename from p2p/transport/websocket/websocket_go.go rename to p2p/transport/websocket/websocket_native.go From a52e415ef70b8a6c4eed3a59c0d95f9747e398e5 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Fri, 16 Aug 2019 17:40:31 -0700 Subject: [PATCH 1517/3965] Revert to returning nil instead of os.ErrNoDeadline --- p2p/transport/websocket/conn_browser.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 76a51f6c39..5eca1fbac6 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -7,7 +7,6 @@ import ( "errors" "io" "net" - "os" "strings" "sync" "syscall/js" @@ -137,15 +136,15 @@ func (c *Conn) RemoteAddr() net.Addr { } func (c *Conn) SetDeadline(t time.Time) error { - return os.ErrNoDeadline + return nil } func (c *Conn) SetReadDeadline(t time.Time) error { - return os.ErrNoDeadline + return nil } func (c *Conn) SetWriteDeadline(t time.Time) error { - return os.ErrNoDeadline + return nil } func (c *Conn) setUpHandlers() { From 7fcd4492cadf36e26efce9149030cb09678a7e2a Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 13:08:18 -0700 Subject: [PATCH 1518/3965] Cache local and remote address --- p2p/transport/websocket/conn_browser.go | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 5eca1fbac6..2c34291bcf 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -35,6 +35,8 @@ type Conn struct { currData bytes.Buffer closeSignal chan struct{} dataSignal chan struct{} + localAddr net.Addr + remoteAddr net.Addr } // NewConn creates a Conn given a regular js/wasm WebSocket Conn. @@ -44,6 +46,8 @@ func NewConn(raw js.Value) *Conn { incomingData: make(chan []byte, incomingDataBufferSize), closeSignal: make(chan struct{}), dataSignal: make(chan struct{}), + localAddr: NewAddr("0.0.0.0:0"), + remoteAddr: getRemoteAddr(raw), } conn.setUpHandlers() go func() { @@ -125,16 +129,20 @@ func (c *Conn) Close() error { } func (c *Conn) LocalAddr() net.Addr { - return NewAddr("0.0.0.0:0") + return c.localAddr } -func (c *Conn) RemoteAddr() net.Addr { - rawURL := c.Get("url").String() +func getRemoteAddr(val js.Value) net.Addr { + rawURL := val.Get("url").String() withoutPrefix := strings.TrimPrefix(rawURL, "ws://") withoutSuffix := strings.TrimSuffix(withoutPrefix, "/") return NewAddr(withoutSuffix) } +func (c *Conn) RemoteAddr() net.Addr { + return c.remoteAddr +} + func (c *Conn) SetDeadline(t time.Time) error { return nil } From 9f099798d124aba4d426e2f3375c4722e3f7cfd2 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 13:30:55 -0700 Subject: [PATCH 1519/3965] Optimize readBlob by using the readAsBinaryString method --- p2p/transport/websocket/conn_browser.go | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 2c34291bcf..25857ce91b 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -218,20 +218,12 @@ func readBlob(blob js.Value) []byte { reader := js.Global().Get("FileReader").New() dataChan := make(chan []byte) loadEndFunc := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - // event.result is of type ArrayBuffer. We can convert it to a JavaScript - // Uint8Array, which is directly translatable to the Go type []byte. - buffer := reader.Get("result") - view := js.Global().Get("Uint8Array").New(buffer) - dataLen := view.Get("length").Int() - data := make([]byte, dataLen) - for i := 0; i < dataLen; i++ { - data[i] = byte(view.Index(i).Int()) - } + data := []byte(reader.Get("result").String()) dataChan <- data return nil }) defer loadEndFunc.Release() reader.Call("addEventListener", "loadend", loadEndFunc) - reader.Call("readAsArrayBuffer", blob) + reader.Call("readAsBinaryString", blob) return <-dataChan } From 49b017b696e418900c46effb3690980288b0b2a8 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 13:59:36 -0700 Subject: [PATCH 1520/3965] Add better error handling to readBlob --- p2p/transport/websocket/conn_browser.go | 46 +++++++++++++++++++++---- 1 file changed, 39 insertions(+), 7 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 25857ce91b..8f40053308 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -5,6 +5,7 @@ package websocket import ( "bytes" "errors" + "fmt" "io" "net" "strings" @@ -167,7 +168,11 @@ func (c *Conn) setUpHandlers() { // TODO(albrow): Currently we assume data is of type Blob. Really we // should check binaryType and then decode accordingly. blob := args[0].Get("data") - data := readBlob(blob) + data, err := readBlob(blob) + if err != nil { + // TODO(albrow): store and return error on next read. + panic(err) + } c.incomingData <- data }() return nil @@ -214,16 +219,43 @@ func (c *Conn) waitForOpen() error { // readBlob converts a JavaScript Blob into a slice of bytes. It uses the // FileReader API under the hood. -func readBlob(blob js.Value) []byte { +func readBlob(blob js.Value) ([]byte, error) { reader := js.Global().Get("FileReader").New() - dataChan := make(chan []byte) - loadEndFunc := js.FuncOf(func(this js.Value, args []js.Value) interface{} { + + // Set up two handlers, one for loadend and one for error. Each handler will + // send results through a channel. + dataChan := make(chan []byte, 1) + loadEndHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { data := []byte(reader.Get("result").String()) dataChan <- data return nil }) - defer loadEndFunc.Release() - reader.Call("addEventListener", "loadend", loadEndFunc) + defer loadEndHandler.Release() + reader.Call("addEventListener", "loadend", loadEndHandler) + errChan := make(chan error, 1) + errorHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { + errChan <- convertJSError(args[0]) + return nil + }) + defer errorHandler.Release() + reader.Call("addEventListener", "error", errorHandler) + + // Call readAsBinaryString and wait to receive from either channel. reader.Call("readAsBinaryString", blob) - return <-dataChan + select { + case data := <-dataChan: + return data, nil + case err := <-errChan: + return nil, err + } +} + +func convertJSError(val js.Value) error { + var typ string + if gotType := val.Get("type"); gotType != js.Undefined() { + typ = gotType.String() + } else { + typ = val.Type().String() + } + return fmt.Errorf("JavaScript error: %s %s", typ, val.Get("message").String()) } From f85474b0afeaa348432c377b370252b16a0e84bf Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 14:41:08 -0700 Subject: [PATCH 1521/3965] Revert back to using readAsArrayBuffer --- p2p/transport/websocket/conn_browser.go | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 8f40053308..17651117d2 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -226,7 +226,15 @@ func readBlob(blob js.Value) ([]byte, error) { // send results through a channel. dataChan := make(chan []byte, 1) loadEndHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - data := []byte(reader.Get("result").String()) + // event.result is of type ArrayBuffer. We can convert it to a JavaScript + // Uint8Array, which is directly translatable to the Go type []byte. + buffer := reader.Get("result") + view := js.Global().Get("Uint8Array").New(buffer) + dataLen := view.Get("length").Int() + data := make([]byte, dataLen) + for i := 0; i < dataLen; i++ { + data[i] = byte(view.Index(i).Int()) + } dataChan <- data return nil }) @@ -240,8 +248,8 @@ func readBlob(blob js.Value) ([]byte, error) { defer errorHandler.Release() reader.Call("addEventListener", "error", errorHandler) - // Call readAsBinaryString and wait to receive from either channel. - reader.Call("readAsBinaryString", blob) + // Call readAsArrayBuffer and wait to receive from either channel. + reader.Call("readAsArrayBuffer", blob) select { case data := <-dataChan: return data, nil From 42732897d527628b85b7185f9c3d4865682cd7e1 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 15:30:06 -0700 Subject: [PATCH 1522/3965] Preserve message order by not using a goroutine in messageHandler --- p2p/transport/websocket/conn_browser.go | 29 +++++++++++-------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 17651117d2..be009e1070 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -31,7 +31,7 @@ type Conn struct { messageHandler *js.Func closeHandler *js.Func mut sync.Mutex - incomingData chan []byte + incomingData chan js.Value currDataMut sync.RWMutex currData bytes.Buffer closeSignal chan struct{} @@ -44,7 +44,7 @@ type Conn struct { func NewConn(raw js.Value) *Conn { conn := &Conn{ Value: raw, - incomingData: make(chan []byte, incomingDataBufferSize), + incomingData: make(chan js.Value, incomingDataBufferSize), closeSignal: make(chan struct{}), dataSignal: make(chan struct{}), localAddr: NewAddr("0.0.0.0:0"), @@ -164,17 +164,10 @@ func (c *Conn) setUpHandlers() { return } messageHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - go func() { - // TODO(albrow): Currently we assume data is of type Blob. Really we - // should check binaryType and then decode accordingly. - blob := args[0].Get("data") - data, err := readBlob(blob) - if err != nil { - // TODO(albrow): store and return error on next read. - panic(err) - } - c.incomingData <- data - }() + // TODO(albrow): Currently we assume data is of type Blob. Really we + // should check binaryType and then decode accordingly. + blob := args[0].Get("data") + c.incomingData <- blob return nil }) c.messageHandler = &messageHandler @@ -191,10 +184,13 @@ func (c *Conn) setUpHandlers() { // readLoop continuosly reads from the c.incoming data channel and writes to the // current data buffer. func (c *Conn) readLoop() error { - for data := range c.incomingData { - c.currDataMut.Lock() - _, err := c.currData.Write(data) + for blob := range c.incomingData { + data, err := readBlob(blob) if err != nil { + return err + } + c.currDataMut.Lock() + if _, err := c.currData.Write(data); err != nil { c.currDataMut.Unlock() return err } @@ -250,6 +246,7 @@ func readBlob(blob js.Value) ([]byte, error) { // Call readAsArrayBuffer and wait to receive from either channel. reader.Call("readAsArrayBuffer", blob) + select { case data := <-dataChan: return data, nil From 2eb6cb2d8de0336a9e86ce9bb57ae14d35001ec9 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 17:11:41 -0700 Subject: [PATCH 1523/3965] Use ArrayBuffer type for binary data This allows us to avoid the rather obtuse FileReader API and makes it easier to read incoming data syncronously. --- p2p/transport/websocket/conn_browser.go | 110 +++++++----------------- 1 file changed, 32 insertions(+), 78 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index be009e1070..388769f32c 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -21,8 +21,6 @@ const ( webSocketStateClosed = 3 ) -const incomingDataBufferSize = 100 - var errConnectionClosed = errors.New("connection is closed") // Conn implements net.Conn interface for WebSockets in js/wasm. @@ -31,7 +29,6 @@ type Conn struct { messageHandler *js.Func closeHandler *js.Func mut sync.Mutex - incomingData chan js.Value currDataMut sync.RWMutex currData bytes.Buffer closeSignal chan struct{} @@ -43,21 +40,17 @@ type Conn struct { // NewConn creates a Conn given a regular js/wasm WebSocket Conn. func NewConn(raw js.Value) *Conn { conn := &Conn{ - Value: raw, - incomingData: make(chan js.Value, incomingDataBufferSize), - closeSignal: make(chan struct{}), - dataSignal: make(chan struct{}), - localAddr: NewAddr("0.0.0.0:0"), - remoteAddr: getRemoteAddr(raw), + Value: raw, + closeSignal: make(chan struct{}), + dataSignal: make(chan struct{}), + localAddr: NewAddr("0.0.0.0:0"), + remoteAddr: getRemoteAddr(raw), } + // Force the JavaScript WebSockets API to use the ArrayBuffer type for + // incoming messages instead of the Blob type. This is better for us because + // ArrayBuffer can be converted to []byte synchronously but Blob cannot. + conn.Set("binaryType", "arraybuffer") conn.setUpHandlers() - go func() { - // TODO(albrow): Handle error appropriately - err := conn.readLoop() - if err != nil { - panic(err) - } - }() return conn } @@ -164,10 +157,21 @@ func (c *Conn) setUpHandlers() { return } messageHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - // TODO(albrow): Currently we assume data is of type Blob. Really we - // should check binaryType and then decode accordingly. - blob := args[0].Get("data") - c.incomingData <- blob + arrayBuffer := args[0].Get("data") + data := arrayBufferToBytes(arrayBuffer) + c.currDataMut.Lock() + if _, err := c.currData.Write(data); err != nil { + c.currDataMut.Unlock() + return err + } + c.currDataMut.Unlock() + + // Non-blocking signal + select { + case c.dataSignal <- struct{}{}: + default: + } + return nil }) c.messageHandler = &messageHandler @@ -181,25 +185,6 @@ func (c *Conn) setUpHandlers() { c.Call("addEventListener", "close", closeHandler) } -// readLoop continuosly reads from the c.incoming data channel and writes to the -// current data buffer. -func (c *Conn) readLoop() error { - for blob := range c.incomingData { - data, err := readBlob(blob) - if err != nil { - return err - } - c.currDataMut.Lock() - if _, err := c.currData.Write(data); err != nil { - c.currDataMut.Unlock() - return err - } - c.currDataMut.Unlock() - c.dataSignal <- struct{}{} - } - return nil -} - func (c *Conn) waitForOpen() error { openSignal := make(chan struct{}) handler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { @@ -213,46 +198,15 @@ func (c *Conn) waitForOpen() error { return nil } -// readBlob converts a JavaScript Blob into a slice of bytes. It uses the -// FileReader API under the hood. -func readBlob(blob js.Value) ([]byte, error) { - reader := js.Global().Get("FileReader").New() - - // Set up two handlers, one for loadend and one for error. Each handler will - // send results through a channel. - dataChan := make(chan []byte, 1) - loadEndHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - // event.result is of type ArrayBuffer. We can convert it to a JavaScript - // Uint8Array, which is directly translatable to the Go type []byte. - buffer := reader.Get("result") - view := js.Global().Get("Uint8Array").New(buffer) - dataLen := view.Get("length").Int() - data := make([]byte, dataLen) - for i := 0; i < dataLen; i++ { - data[i] = byte(view.Index(i).Int()) - } - dataChan <- data - return nil - }) - defer loadEndHandler.Release() - reader.Call("addEventListener", "loadend", loadEndHandler) - errChan := make(chan error, 1) - errorHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - errChan <- convertJSError(args[0]) - return nil - }) - defer errorHandler.Release() - reader.Call("addEventListener", "error", errorHandler) - - // Call readAsArrayBuffer and wait to receive from either channel. - reader.Call("readAsArrayBuffer", blob) - - select { - case data := <-dataChan: - return data, nil - case err := <-errChan: - return nil, err +// arrayBufferToBytes converts a JavaScript ArrayBuffer to a slice of bytes. +func arrayBufferToBytes(buffer js.Value) []byte { + view := js.Global().Get("Uint8Array").New(buffer) + dataLen := view.Get("length").Int() + data := make([]byte, dataLen) + for i := 0; i < dataLen; i++ { + data[i] = byte(view.Index(i).Int()) } + return data } func convertJSError(val js.Value) error { From e327fad1620c9aa6169a74cd1b34035d62013853 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 17:15:57 -0700 Subject: [PATCH 1524/3965] Use a buffer of 1 for dataSignal channel --- p2p/transport/websocket/conn_browser.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 388769f32c..7da0027bdb 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -42,7 +42,7 @@ func NewConn(raw js.Value) *Conn { conn := &Conn{ Value: raw, closeSignal: make(chan struct{}), - dataSignal: make(chan struct{}), + dataSignal: make(chan struct{}, 1), localAddr: NewAddr("0.0.0.0:0"), remoteAddr: getRemoteAddr(raw), } From eda7468e8b924736289346969c2882e04639739d Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 17:38:26 -0700 Subject: [PATCH 1525/3965] Add TODO for returning os.ErrNoDeadline --- p2p/transport/websocket/conn_browser.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 7da0027bdb..ffd38c5932 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -137,6 +137,8 @@ func (c *Conn) RemoteAddr() net.Addr { return c.remoteAddr } +// TODO: Return os.ErrNoDeadline. For now we return nil because multiplexers +// don't handle the error correctly. func (c *Conn) SetDeadline(t time.Time) error { return nil } From 38b0b7fbbed72fd362535fdd775e17bbf1607673 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Mon, 19 Aug 2019 18:41:58 -0700 Subject: [PATCH 1526/3965] Add proper error handling --- p2p/transport/websocket/conn_browser.go | 88 ++++++++++++++++++-- p2p/transport/websocket/websocket_browser.go | 8 +- 2 files changed, 82 insertions(+), 14 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index ffd38c5932..07b88daca1 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -28,13 +28,16 @@ type Conn struct { js.Value messageHandler *js.Func closeHandler *js.Func + errorHandler *js.Func mut sync.Mutex currDataMut sync.RWMutex currData bytes.Buffer + closeOnce sync.Once closeSignal chan struct{} dataSignal chan struct{} localAddr net.Addr remoteAddr net.Addr + firstErr error } // NewConn creates a Conn given a regular js/wasm WebSocket Conn. @@ -56,7 +59,7 @@ func NewConn(raw js.Value) *Conn { func (c *Conn) Read(b []byte) (int, error) { if err := c.checkOpen(); err != nil { - return 0, io.EOF + return c.readAfterErr(b) } for { @@ -73,7 +76,7 @@ func (c *Conn) Read(b []byte) (int, error) { case <-c.dataSignal: continue case <-c.closeSignal: - return 0, io.EOF + return c.readAfterErr(b) } } else { return n, err @@ -81,6 +84,22 @@ func (c *Conn) Read(b []byte) (int, error) { } } +// readAfterError reads from c.currData. If there is no more data left it +// returns c.firstErr if non-nil and otherwise returns io.EOF. +func (c *Conn) readAfterErr(b []byte) (int, error) { + c.currDataMut.RLock() + n, err := c.currData.Read(b) + c.currDataMut.RUnlock() + if n == 0 { + if c.firstErr != nil { + return 0, c.firstErr + } else { + return 0, io.EOF + } + } + return n, err +} + // checkOpen returns an error if the connection is not open. Otherwise, it // returns nil. func (c *Conn) checkOpen() error { @@ -108,9 +127,21 @@ func (c *Conn) Write(b []byte) (n int, err error) { // close error, subsequent and concurrent calls will return nil. // This method is thread-safe. func (c *Conn) Close() error { + c.signalClose() + c.Call("close") + c.releaseHandlers() + return nil +} + +func (c *Conn) signalClose() { + c.closeOnce.Do(func() { + close(c.closeSignal) + }) +} + +func (c *Conn) releaseHandlers() { c.mut.Lock() defer c.mut.Unlock() - c.Call("close") if c.messageHandler != nil { c.Call("removeEventListener", "message", *c.messageHandler) c.messageHandler.Release() @@ -119,7 +150,10 @@ func (c *Conn) Close() error { c.Call("removeEventListener", "close", *c.closeHandler) c.closeHandler.Release() } - return nil + if c.errorHandler != nil { + c.Call("removeEventListener", "error", *c.errorHandler) + c.errorHandler.Release() + } } func (c *Conn) LocalAddr() net.Addr { @@ -180,11 +214,27 @@ func (c *Conn) setUpHandlers() { c.Call("addEventListener", "message", messageHandler) closeHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { - close(c.closeSignal) + go func() { + c.signalClose() + c.mut.Lock() + // Store the error in c.firstErr. It will be returned by Read later on. + c.firstErr = errorEventToError(args[0]) + c.mut.Unlock() + c.releaseHandlers() + }() return nil }) c.closeHandler = &closeHandler c.Call("addEventListener", "close", closeHandler) + + errorHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { + // Unfortunately, the "error" event doesn't appear to give us any useful + // information. All we can do is close the connection. + c.Close() + return nil + }) + c.errorHandler = &errorHandler + c.Call("addEventListener", "error", errorHandler) } func (c *Conn) waitForOpen() error { @@ -196,8 +246,14 @@ func (c *Conn) waitForOpen() error { defer c.Call("removeEventListener", "open", handler) defer handler.Release() c.Call("addEventListener", "open", handler) - <-openSignal - return nil + select { + case <-openSignal: + return nil + case <-c.closeSignal: + // c.closeSignal means there was an error when trying to open the + // connection. + return c.firstErr + } } // arrayBufferToBytes converts a JavaScript ArrayBuffer to a slice of bytes. @@ -211,12 +267,26 @@ func arrayBufferToBytes(buffer js.Value) []byte { return data } -func convertJSError(val js.Value) error { +func errorEventToError(val js.Value) error { var typ string if gotType := val.Get("type"); gotType != js.Undefined() { typ = gotType.String() } else { typ = val.Type().String() } - return fmt.Errorf("JavaScript error: %s %s", typ, val.Get("message").String()) + var reason string + if gotReason := val.Get("reason"); gotReason != js.Undefined() && gotReason.String() != "" { + reason = gotReason.String() + } else { + code := val.Get("code") + if code != js.Undefined() { + switch code := code.Int(); code { + case 1006: + reason = "code 1006: connection unexpectedly closed" + default: + reason = fmt.Sprintf("unexpected code: %d", code) + } + } + } + return fmt.Errorf("JavaScript error: (%s) %s", typ, reason) } diff --git a/p2p/transport/websocket/websocket_browser.go b/p2p/transport/websocket/websocket_browser.go index 9002533e0c..acf306f5cd 100644 --- a/p2p/transport/websocket/websocket_browser.go +++ b/p2p/transport/websocket/websocket_browser.go @@ -19,12 +19,10 @@ func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (ma } rawConn := js.Global().Get("WebSocket").New(wsurl) - rawConn.Call("addEventListener", "error", js.FuncOf(func(this js.Value, args []js.Value) interface{} { - js.Global().Get("console").Call("log", args[0]) - return nil - })) conn := NewConn(rawConn) - conn.waitForOpen() + if err := conn.waitForOpen(); err != nil { + return nil, err + } mnc, err := manet.WrapNetConn(conn) if err != nil { conn.Close() From b8d9682b7bc797774d565aa920b0df175f81cbbd Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 12:39:29 -0700 Subject: [PATCH 1527/3965] Expand comment about time.Sleep hack --- p2p/transport/websocket/browser_integration_browser_test.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/transport/websocket/browser_integration_browser_test.go b/p2p/transport/websocket/browser_integration_browser_test.go index 656db447ba..6a55c30275 100644 --- a/p2p/transport/websocket/browser_integration_browser_test.go +++ b/p2p/transport/websocket/browser_integration_browser_test.go @@ -51,6 +51,9 @@ func TestInBrowser(t *testing.T) { } // TODO(albrow): This hack is necessary in order to give the reader time to - // finish. We should find some way to remove it. + // finish. As soon as this test function returns, the browser window is + // closed, which means there is no time for the other end of the connection to + // read the "pong" message. We should find some way to remove this hack if + // possible. time.Sleep(1 * time.Second) } From e08426cc8d11465fbfdd4f71b97baf50e2a02ee4 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 12:51:09 -0700 Subject: [PATCH 1528/3965] Call conn.Close if waitForOpen returns an error --- p2p/transport/websocket/websocket_browser.go | 1 + 1 file changed, 1 insertion(+) diff --git a/p2p/transport/websocket/websocket_browser.go b/p2p/transport/websocket/websocket_browser.go index acf306f5cd..258a82917c 100644 --- a/p2p/transport/websocket/websocket_browser.go +++ b/p2p/transport/websocket/websocket_browser.go @@ -21,6 +21,7 @@ func (t *WebsocketTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (ma rawConn := js.Global().Get("WebSocket").New(wsurl) conn := NewConn(rawConn) if err := conn.waitForOpen(); err != nil { + conn.Close() return nil, err } mnc, err := manet.WrapNetConn(conn) From fe9ccdf21a8e6af72fd5e2eb1ff36e6d8003ee32 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 12:57:46 -0700 Subject: [PATCH 1529/3965] Simplify Conn.Read logic --- p2p/transport/websocket/conn_browser.go | 29 ++++++++++++------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 07b88daca1..73ee2aee4d 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -64,22 +64,21 @@ func (c *Conn) Read(b []byte) (int, error) { for { c.currDataMut.RLock() - n, err := c.currData.Read(b) + n, _ := c.currData.Read(b) c.currDataMut.RUnlock() - if err != nil && err != io.EOF { - // Return any unexpected errors immediately. - return n, err - } else if n == 0 || err == io.EOF { - // There is no data ready to be read. Wait for more data or for the - // connection to be closed. - select { - case <-c.dataSignal: - continue - case <-c.closeSignal: - return c.readAfterErr(b) - } - } else { - return n, err + + if n != 0 { + // Data was ready. Return the number of bytes read. + return n, nil + } + + // There is no data ready to be read. Wait for more data or for the + // connection to be closed. + select { + case <-c.dataSignal: + continue + case <-c.closeSignal: + return c.readAfterErr(b) } } } From 169c516ab7deb78dcc4a98d610adf3fd1f58c796 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 13:12:55 -0700 Subject: [PATCH 1530/3965] Simplify iteration over Uint8Array --- p2p/transport/websocket/conn_browser.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 73ee2aee4d..3e4d28aa58 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -115,8 +115,8 @@ func (c *Conn) Write(b []byte) (n int, err error) { return 0, err } uint8Array := js.Global().Get("Uint8Array").New(len(b)) - for i := 0; i < len(b); i++ { - uint8Array.SetIndex(i, b[i]) + for i, bi := range b { + uint8Array.SetIndex(i, bi) } c.Call("send", uint8Array.Get("buffer")) return len(b), nil From b4e7083efe474cd32cdf1ad02136245930f50d71 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 13:33:28 -0700 Subject: [PATCH 1531/3965] Set handlers to nil after releasing them --- p2p/transport/websocket/conn_browser.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 3e4d28aa58..733ca2b828 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -144,14 +144,17 @@ func (c *Conn) releaseHandlers() { if c.messageHandler != nil { c.Call("removeEventListener", "message", *c.messageHandler) c.messageHandler.Release() + c.messageHandler = nil } if c.closeHandler != nil { c.Call("removeEventListener", "close", *c.closeHandler) c.closeHandler.Release() + c.closeHandler = nil } if c.errorHandler != nil { c.Call("removeEventListener", "error", *c.errorHandler) c.errorHandler.Release() + c.errorHandler = nil } } From a4d3db5f8575d3e599fc1d7e73bcdcda5cc77baa Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 14:45:34 -0700 Subject: [PATCH 1532/3965] Use a sync.Once in Conn.Close --- p2p/transport/websocket/conn_browser.go | 35 ++++++++++++++----------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 733ca2b828..f676b303bc 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -26,18 +26,19 @@ var errConnectionClosed = errors.New("connection is closed") // Conn implements net.Conn interface for WebSockets in js/wasm. type Conn struct { js.Value - messageHandler *js.Func - closeHandler *js.Func - errorHandler *js.Func - mut sync.Mutex - currDataMut sync.RWMutex - currData bytes.Buffer - closeOnce sync.Once - closeSignal chan struct{} - dataSignal chan struct{} - localAddr net.Addr - remoteAddr net.Addr - firstErr error + messageHandler *js.Func + closeHandler *js.Func + errorHandler *js.Func + mut sync.Mutex + currDataMut sync.RWMutex + currData bytes.Buffer + closeOnce sync.Once + closeSignalOnce sync.Once + closeSignal chan struct{} + dataSignal chan struct{} + localAddr net.Addr + remoteAddr net.Addr + firstErr error } // NewConn creates a Conn given a regular js/wasm WebSocket Conn. @@ -126,14 +127,16 @@ func (c *Conn) Write(b []byte) (n int, err error) { // close error, subsequent and concurrent calls will return nil. // This method is thread-safe. func (c *Conn) Close() error { - c.signalClose() - c.Call("close") - c.releaseHandlers() + c.closeOnce.Do(func() { + c.signalClose() + c.Call("close") + c.releaseHandlers() + }) return nil } func (c *Conn) signalClose() { - c.closeOnce.Do(func() { + c.closeSignalOnce.Do(func() { close(c.closeSignal) }) } From 9ec49775da86263190782a4bad060ad4e8b1b870 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 15:20:54 -0700 Subject: [PATCH 1533/3965] Recover from uncaught JavaScript exceptions in Conn.Write --- p2p/transport/websocket/conn_browser.go | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index f676b303bc..203fcdabd8 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -112,6 +112,11 @@ func (c *Conn) checkOpen() error { } func (c *Conn) Write(b []byte) (n int, err error) { + defer func() { + if e := recover(); e != nil { + err = recoveredValueToError(e) + } + }() if err := c.checkOpen(); err != nil { return 0, err } @@ -295,3 +300,12 @@ func errorEventToError(val js.Value) error { } return fmt.Errorf("JavaScript error: (%s) %s", typ, reason) } + +func recoveredValueToError(e interface{}) error { + switch e := e.(type) { + case error: + return e + default: + return fmt.Errorf("recovered from unexpected panic: %T %s", e, e) + } +} From badfe1f1308af05d210f9e2186f98f793ac795ce Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 15:44:36 -0700 Subject: [PATCH 1534/3965] Switch order of Call("close") and signalClose() --- p2p/transport/websocket/conn_browser.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 203fcdabd8..b603ebbe48 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -133,8 +133,8 @@ func (c *Conn) Write(b []byte) (n int, err error) { // This method is thread-safe. func (c *Conn) Close() error { c.closeOnce.Do(func() { - c.signalClose() c.Call("close") + c.signalClose() c.releaseHandlers() }) return nil From 391b41eef53ef12e3e526120fe3964bedbbbf691 Mon Sep 17 00:00:00 2001 From: Alex Browne Date: Tue, 17 Sep 2019 16:00:36 -0700 Subject: [PATCH 1535/3965] Remove unused constant --- p2p/transport/websocket/browser_integration_native_test.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/p2p/transport/websocket/browser_integration_native_test.go b/p2p/transport/websocket/browser_integration_native_test.go index 464fca73a8..f960822fc6 100644 --- a/p2p/transport/websocket/browser_integration_native_test.go +++ b/p2p/transport/websocket/browser_integration_native_test.go @@ -17,10 +17,6 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -const ( - testServerPort = ":9714" -) - // TestInBrowser is a harness that allows us to use `go test` in order to run // WebAssembly tests in a headless browser. func TestInBrowser(t *testing.T) { From 9da7d27a46b353e7eeebada29a102ca0baafed1f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 30 Sep 2019 15:09:36 -0700 Subject: [PATCH 1536/3965] Set firstErr before signaling the connection close --- p2p/transport/websocket/conn_browser.go | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index b603ebbe48..73d087ccfc 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -38,7 +38,7 @@ type Conn struct { dataSignal chan struct{} localAddr net.Addr remoteAddr net.Addr - firstErr error + firstErr error // only read this _after_ observing that closeSignal has been closed. } // NewConn creates a Conn given a regular js/wasm WebSocket Conn. @@ -134,14 +134,15 @@ func (c *Conn) Write(b []byte) (n int, err error) { func (c *Conn) Close() error { c.closeOnce.Do(func() { c.Call("close") - c.signalClose() + c.signalClose(nil) c.releaseHandlers() }) return nil } -func (c *Conn) signalClose() { +func (c *Conn) signalClose(err error) { c.closeSignalOnce.Do(func() { + c.firstErr = err close(c.closeSignal) }) } @@ -225,11 +226,7 @@ func (c *Conn) setUpHandlers() { closeHandler := js.FuncOf(func(this js.Value, args []js.Value) interface{} { go func() { - c.signalClose() - c.mut.Lock() - // Store the error in c.firstErr. It will be returned by Read later on. - c.firstErr = errorEventToError(args[0]) - c.mut.Unlock() + c.signalClose(errorEventToError(args[0])) c.releaseHandlers() }() return nil From 908e0da96a83f716dbe7f2d098844f601e9762f0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 30 Sep 2019 15:18:58 -0700 Subject: [PATCH 1537/3965] Handle connection errors before handling data. --- p2p/transport/websocket/conn_browser.go | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 73d087ccfc..1f021529d1 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -6,7 +6,6 @@ import ( "bytes" "errors" "fmt" - "io" "net" "strings" "sync" @@ -59,8 +58,10 @@ func NewConn(raw js.Value) *Conn { } func (c *Conn) Read(b []byte) (int, error) { - if err := c.checkOpen(); err != nil { - return c.readAfterErr(b) + select { + case <-c.closeSignal: + c.readAfterErr(b) + default: } for { @@ -77,7 +78,6 @@ func (c *Conn) Read(b []byte) (int, error) { // connection to be closed. select { case <-c.dataSignal: - continue case <-c.closeSignal: return c.readAfterErr(b) } @@ -87,16 +87,12 @@ func (c *Conn) Read(b []byte) (int, error) { // readAfterError reads from c.currData. If there is no more data left it // returns c.firstErr if non-nil and otherwise returns io.EOF. func (c *Conn) readAfterErr(b []byte) (int, error) { + if c.firstErr != nil { + return 0, c.firstErr + } c.currDataMut.RLock() n, err := c.currData.Read(b) c.currDataMut.RUnlock() - if n == 0 { - if c.firstErr != nil { - return 0, c.firstErr - } else { - return 0, io.EOF - } - } return n, err } From 88b65ce76553b6846eeb1de93b40129d1826257a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 30 Sep 2019 15:56:32 -0700 Subject: [PATCH 1538/3965] test: auto-install wasm test dependency --- .../browser_integration_native_test.go | 39 +++++++++++++++++-- p2p/transport/websocket/tools/go.mod | 5 +++ p2p/transport/websocket/tools/go.sum | 31 +++++++++++++++ p2p/transport/websocket/tools/tools.go | 5 +++ 4 files changed, 76 insertions(+), 4 deletions(-) create mode 100644 p2p/transport/websocket/tools/go.mod create mode 100644 p2p/transport/websocket/tools/go.sum create mode 100644 p2p/transport/websocket/tools/tools.go diff --git a/p2p/transport/websocket/browser_integration_native_test.go b/p2p/transport/websocket/browser_integration_native_test.go index f960822fc6..fa940b5c53 100644 --- a/p2p/transport/websocket/browser_integration_native_test.go +++ b/p2p/transport/websocket/browser_integration_native_test.go @@ -4,7 +4,6 @@ package websocket import ( "bufio" - "fmt" "os" "os/exec" "path/filepath" @@ -17,9 +16,35 @@ import ( ma "github.com/multiformats/go-multiaddr" ) +var ( + wasmBrowserTestBin = "wasmbrowsertest" + wasmBrowserTestDir = filepath.Join("tools", "bin") + wasmBrowserTestPackage = "github.com/agnivade/wasmbrowsertest" +) + // TestInBrowser is a harness that allows us to use `go test` in order to run // WebAssembly tests in a headless browser. func TestInBrowser(t *testing.T) { + // ensure we have the right tools. + err := os.MkdirAll(wasmBrowserTestDir, 0755) + + t.Logf("building %s", wasmBrowserTestPackage) + if err != nil && !os.IsExist(err) { + t.Fatal(err) + } + + cmd := exec.Command( + "go", "build", + "-o", wasmBrowserTestBin, + "github.com/agnivade/wasmbrowsertest", + ) + cmd.Dir = wasmBrowserTestDir + err = cmd.Run() + if err != nil { + t.Fatal(err) + } + t.Log("starting server") + // Start a transport which the browser peer will dial. serverDoneSignal := make(chan struct{}) go func() { @@ -62,13 +87,19 @@ func TestInBrowser(t *testing.T) { } }() - testExecPath := filepath.Join(os.Getenv("GOPATH"), "bin", "wasmbrowsertest") - cmd := exec.Command("go", "test", "-exec", testExecPath, "-run", "TestInBrowser", ".", "-v") + t.Log("starting browser") + + cmd = exec.Command( + "go", "test", "-v", + "-exec", filepath.Join(wasmBrowserTestDir, wasmBrowserTestBin), + "-run", "TestInBrowser", + ".", + ) cmd.Env = append(os.Environ(), []string{"GOOS=js", "GOARCH=wasm"}...) output, err := cmd.CombinedOutput() if err != nil { formattedOutput := "\t" + strings.Join(strings.Split(string(output), "\n"), "\n\t") - fmt.Println("BROWSER OUTPUT:\n", formattedOutput) + t.Log("BROWSER OUTPUT:\n", formattedOutput) t.Fatal("BROWSER:", err) } diff --git a/p2p/transport/websocket/tools/go.mod b/p2p/transport/websocket/tools/go.mod new file mode 100644 index 0000000000..d34d7385b2 --- /dev/null +++ b/p2p/transport/websocket/tools/go.mod @@ -0,0 +1,5 @@ +module github.com/libp2p/go-ws-transport/tools + +go 1.13 + +require github.com/agnivade/wasmbrowsertest v0.3.1 diff --git a/p2p/transport/websocket/tools/go.sum b/p2p/transport/websocket/tools/go.sum new file mode 100644 index 0000000000..94b9c35ab1 --- /dev/null +++ b/p2p/transport/websocket/tools/go.sum @@ -0,0 +1,31 @@ +github.com/agnivade/wasmbrowsertest v0.3.1 h1:bA9aA+bcp7KuqGvmCuBdnMqy6PXxFjYP7FxsaT+JSqc= +github.com/agnivade/wasmbrowsertest v0.3.1/go.mod h1:zQt6ZTdl338xxRaMW395qccVE2eQm0SjC/SDz0mPWQI= +github.com/chromedp/cdproto v0.0.0-20190614062957-d6d2f92b486d/go.mod h1:S8mB5wY3vV+vRIzf39xDXsw3XKYewW9X6rW2aEmkrSw= +github.com/chromedp/cdproto v0.0.0-20190621002710-8cbd498dd7a0 h1:4Wocv9f+KWF4GtZudyrn8JSBTgHQbGp86mcsoH7j1iQ= +github.com/chromedp/cdproto v0.0.0-20190621002710-8cbd498dd7a0/go.mod h1:S8mB5wY3vV+vRIzf39xDXsw3XKYewW9X6rW2aEmkrSw= +github.com/chromedp/chromedp v0.3.1-0.20190619195644-fd957a4d2901 h1:tg66ykM8VYqP9k4DFQwSMnYv84HNTruF+GR6kefFNg4= +github.com/chromedp/chromedp v0.3.1-0.20190619195644-fd957a4d2901/go.mod h1:mJdvfrVn594N9tfiPecUidF6W5jPRKHymqHfzbobPsM= +github.com/edsrzf/mmap-go v1.0.0 h1:CEBF7HpRnUCSJgGUb5h1Gm7e3VkmVDrR8lvWVLtrOFw= +github.com/edsrzf/mmap-go v1.0.0/go.mod h1:YO35OhQPt3KJa3ryjFM5Bs14WD66h8eGKpfaBNrHW5M= +github.com/go-interpreter/wagon v0.5.1-0.20190713202023-55a163980b6c h1:DLLAPVFrk9iNzljMKF512CUmrFImQ6WU3sDiUS4IRqk= +github.com/go-interpreter/wagon v0.5.1-0.20190713202023-55a163980b6c/go.mod h1:5+b/MBYkclRZngKF5s6qrgWxSLgE9F5dFdO1hAueZLc= +github.com/gobwas/httphead v0.0.0-20180130184737-2c6c146eadee h1:s+21KNqlpePfkah2I+gwHF8xmJWRjooY+5248k6m4A0= +github.com/gobwas/httphead v0.0.0-20180130184737-2c6c146eadee/go.mod h1:L0fX3K22YWvt/FAX9NnzrNzcI4wNYi9Yku4O0LKYflo= +github.com/gobwas/pool v0.2.0 h1:QEmUOlnSjWtnpRGHF3SauEiOsy82Cup83Vf2LcMlnc8= +github.com/gobwas/pool v0.2.0/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw= +github.com/gobwas/ws v1.0.2 h1:CoAavW/wd/kulfZmSIBt6p24n4j7tHgNVCjsfHVNUbo= +github.com/gobwas/ws v1.0.2/go.mod h1:szmBTxLgaFppYjEmNtny/v3w89xOydFnnZMcgRRu/EM= +github.com/google/pprof v0.0.0-20190515194954-54271f7e092f h1:Jnx61latede7zDD3DiiP4gmNz33uK0U5HDUaF0a/HVQ= +github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= +github.com/knq/sysutil v0.0.0-20181215143952-f05b59f0f307 h1:vl4eIlySbjertFaNwiMjXsGrFVK25aOWLq7n+3gh2ls= +github.com/knq/sysutil v0.0.0-20181215143952-f05b59f0f307/go.mod h1:BjPj+aVjl9FW/cCGiF3nGh5v+9Gd3VCgBQbod/GlMaQ= +github.com/mailru/easyjson v0.0.0-20190403194419-1ea4449da983/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.0.0-20190620125010-da37f6c1e481 h1:IaSjLMT6WvkoZZjspGxy3rdaTEmWLoRm49WbtVUi9sA= +github.com/mailru/easyjson v0.0.0-20190620125010-da37f6c1e481/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/twitchyliquid64/golang-asm v0.0.0-20190126203739-365674df15fc h1:RTUQlKzoZZVG3umWNzOYeFecQLIh+dbxXvJp1zPQJTI= +github.com/twitchyliquid64/golang-asm v0.0.0-20190126203739-365674df15fc/go.mod h1:NoCfSFWosfqMqmmD7hApkirIK9ozpHjxRnRxs1l413A= +golang.org/x/sys v0.0.0-20190306220234-b354f8bf4d9e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190618155005-516e3c20635f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7 h1:LepdCS8Gf/MVejFIt8lsiexZATdoGVyp5bcyS+rYoUI= +golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= diff --git a/p2p/transport/websocket/tools/tools.go b/p2p/transport/websocket/tools/tools.go new file mode 100644 index 0000000000..d051c510b9 --- /dev/null +++ b/p2p/transport/websocket/tools/tools.go @@ -0,0 +1,5 @@ +// +build tools + +package tools + +import _ "github.com/agnivade/wasmbrowsertest" From c285b8a1d14db52b21e62341103b1e81e5b2f1db Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 15 Jul 2019 14:25:48 -0700 Subject: [PATCH 1539/3965] feat: support encoding/decoding peer IDs as CIDs _in text_ --- core/peer/peer.go | 73 ++++++++++++++++++++++++++++++++++++++---- core/peer/peer_test.go | 45 ++++++++++++++++++++++++++ 2 files changed, 111 insertions(+), 7 deletions(-) diff --git a/core/peer/peer.go b/core/peer/peer.go index edef635c36..9ae0a84406 100644 --- a/core/peer/peer.go +++ b/core/peer/peer.go @@ -5,7 +5,9 @@ import ( "encoding/hex" "errors" "fmt" + "strings" + cid "github.com/ipfs/go-cid" ic "github.com/libp2p/go-libp2p-core/crypto" b58 "github.com/mr-tron/base58/base58" mh "github.com/multiformats/go-multihash" @@ -129,23 +131,24 @@ func IDFromBytes(b []byte) (ID, error) { return ID(b), nil } -// IDB58Decode accepts a base58-encoded multihash representing a peer ID -// and returns the decoded ID if the input is valid. +// IDB58Decode decodes a peer ID. +// +// Deprecated: Use Decode. func IDB58Decode(s string) (ID, error) { - m, err := mh.FromB58String(s) - if err != nil { - return "", err - } - return ID(m), err + return Decode(s) } // IDB58Encode returns the base58-encoded multihash representation of the ID. +// +// Deprecated: Use Encode. func IDB58Encode(id ID) string { return b58.Encode([]byte(id)) } // IDHexDecode accepts a hex-encoded multihash representing a peer ID // and returns the decoded ID if the input is valid. +// +// Deprecated: Don't raw-hex encode peer IDs, use base16 CIDs. func IDHexDecode(s string) (ID, error) { m, err := mh.FromHexString(s) if err != nil { @@ -155,10 +158,66 @@ func IDHexDecode(s string) (ID, error) { } // IDHexEncode returns the hex-encoded multihash representation of the ID. +// +// Deprecated: Don't raw-hex encode peer IDs, use base16 CIDs. func IDHexEncode(id ID) string { return hex.EncodeToString([]byte(id)) } +// Decode accepts an encoded peer ID and returns the decoded ID if the input is +// valid. +// +// The encoded peer ID can either be a CID of a key or a raw multihash (identity +// or sha256-256). +func Decode(s string) (ID, error) { + if strings.HasPrefix(s, "Qm") || strings.HasPrefix(s, "1") { + // base58 encoded sha256 or identity multihash + m, err := mh.FromB58String(s) + if err != nil { + return "", fmt.Errorf("failed to parse peer ID: %s", err) + } + return ID(m), nil + } + + c, err := cid.Decode(s) + if err != nil { + return "", fmt.Errorf("failed to parse peer ID: %s", err) + } + return FromCid(c) +} + +// Encode encodes a peer ID as a string. +// +// At the moment, it base58 encodes the peer ID but, in the future, it will +// switch to encoding it as a CID by default. +func Encode(id ID) string { + return IDB58Encode(id) +} + +// FromCid converts a CID to a peer ID, if possible. +func FromCid(c cid.Cid) (ID, error) { + ty := c.Type() + if ty != cid.Libp2pKey { + s := cid.CodecToStr[ty] + if s == "" { + s = fmt.Sprintf("[unknown multicodec %d]", ty) + } + return "", fmt.Errorf("can't convert CID of type %s to a peer ID", s) + } + return ID(c.Hash()), nil +} + +// ToCid encodes a peer ID as a CID of the public key. +// +// If the peer ID is invalid (e.g., empty), this will return the empty CID. +func ToCid(id ID) cid.Cid { + m, err := mh.Cast([]byte(id)) + if err != nil { + return cid.Cid{} + } + return cid.NewCidV1(cid.Libp2pKey, m) +} + // IDFromPublicKey returns the Peer ID corresponding to the public key pk. func IDFromPublicKey(pk ic.PubKey) (ID, error) { b, err := pk.Bytes() diff --git a/core/peer/peer_test.go b/core/peer/peer_test.go index 757f085c15..b203a7ad2f 100644 --- a/core/peer/peer_test.go +++ b/core/peer/peer_test.go @@ -153,6 +153,51 @@ func TestIDMatchesPrivateKey(t *testing.T) { test(man) } +func TestIDEncoding(t *testing.T) { + test := func(ks keyset) { + p1, err := IDB58Decode(ks.hpkp) + if err != nil { + t.Fatal(err) + } + + if ks.hpk != string(p1) { + t.Error("p1 and hpk differ") + } + + c := ToCid(p1) + p2, err := FromCid(c) + if err != nil || p1 != p2 { + t.Fatal("failed to round-trip through CID:", err) + } + p3, err := Decode(c.String()) + if err != nil { + t.Fatal(err) + } + if p3 != p1 { + t.Fatal("failed to round trip through CID string") + } + + if ks.hpkp != Encode(p1) { + t.Fatal("should always encode peer IDs as base58 by default") + } + } + + test(gen1) + test(gen2) + test(man) + + exampleCid := "bafkreifoybygix7fh3r3g5rqle3wcnhqldgdg4shzf4k3ulyw3gn7mabt4" + _, err := Decode(exampleCid) + if err == nil { + t.Fatal("should refuse to decode a non-peer ID CID") + } + + c := ToCid("") + if c.Defined() { + t.Fatal("cid of empty peer ID should have been undefined") + } +} + func TestPublicKeyExtraction(t *testing.T) { t.Skip("disabled until libp2p/go-libp2p-crypto#51 is fixed") // Happy path From 5f60501a04d5bdf97feda565f2cba45d6a1706ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Sun, 6 Oct 2019 23:02:50 +0900 Subject: [PATCH 1540/3965] readme: add instructions for bypassing default gomod proxy. (#731) We do not want to contribute to informing Google of every single user that uses go-libp2p, thanks. Also, the default proxy (proxy.golang.org) contains old and deprecated `+incompatible` versions that the Go toolchain selects over the more recent go-modded versions. See https://github.com/golang/go/issues/34189 and https://github.com/golang/go/issues/34217. --- README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a066cd5794..3c2862f326 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,13 @@ You can start using go-libp2p in your Go application simply by adding imports fr import "github.com/libp2p/go-libp2p" ``` -The next time you run `go get` or `go build`, the Go build tools will look for [available releases](https://github.com/libp2p/go-libp2p/releases), and will pick the highest available one. +Run `go get` or `go build`, excluding the libp2p repos from Go modules proxy usage. You only need to do this the first time you import go-libp2p to make sure you latch onto the correct version lineage (see [golang/go#34189](https://github.com/golang/go/issues/34189) for context): + +```sh +$ GOPRIVATE='github.com/libp2p/*' go get ./... +``` + +The Go build tools will look for [available releases](https://github.com/libp2p/go-libp2p/releases), and will pick the highest available one. As new releases of go-libp2p are made available, you can upgrade your application by manually editing your `go.mod` file, or using the [Go tools](https://golang.org/cmd/go/#hdr-Maintaining_module_requirements) to maintain module requirements. From 9cd56c0babc918852643f146112f2e7782aa1418 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 8 Oct 2019 13:01:29 +0300 Subject: [PATCH 1541/3965] use a global variable for default relays and rename DefaultRelays option to DefaultStaticRelays. --- options.go | 13 +++++-------- p2p/host/relay/autorelay.go | 7 +++++++ 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/options.go b/options.go index a78a3043a5..899542ea85 100644 --- a/options.go +++ b/options.go @@ -17,6 +17,7 @@ import ( circuit "github.com/libp2p/go-libp2p-circuit" config "github.com/libp2p/go-libp2p/config" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" + autorelay "github.com/libp2p/go-libp2p/p2p/host/relay" filter "github.com/libp2p/go-maddr-filter" ma "github.com/multiformats/go-multiaddr" @@ -251,19 +252,15 @@ func EnableAutoRelay() Option { // discover relays func StaticRelays(relays []peer.AddrInfo) Option { return func(cfg *Config) error { - cfg.StaticRelays = relays + cfg.StaticRelays = append(cfg.StaticRelays, relays...) return nil } } -// DefaultRelays configures the static relays to use the known PL-operated relays -func DefaultRelays() Option { +// DefaultStaticRelays configures the static relays to use the known PL-operated relays +func DefaultStaticRelays() Option { return func(cfg *Config) error { - for _, addr := range []string{ - "/ip4/147.75.80.110/tcp/4001/p2p/QmbFgm5zan8P6eWWmeyfncR5feYEMPbht5b1FW1C37aQ7y", - "/ip4/147.75.195.153/tcp/4001/p2p/QmW9m57aiBDHAkKj9nmFSEn7ZqrcF1fZS4bipsTCHburei", - "/ip4/147.75.70.221/tcp/4001/p2p/Qme8g49gm3q4Acp7xWBKg3nAa9fxZ1YmyDJdyGgoG6LsXh", - } { + for _, addr := range autorelay.DefaultRelays { a, err := ma.NewMultiaddr(addr) if err != nil { return err diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 0dfda1d5f5..0c3f40b53f 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -30,6 +30,13 @@ var ( BootDelay = 20 * time.Second ) +// These are the known PL-operated relays +var DefaultRelays = []string{ + "/ip4/147.75.80.110/tcp/4001/p2p/QmbFgm5zan8P6eWWmeyfncR5feYEMPbht5b1FW1C37aQ7y", + "/ip4/147.75.195.153/tcp/4001/p2p/QmW9m57aiBDHAkKj9nmFSEn7ZqrcF1fZS4bipsTCHburei", + "/ip4/147.75.70.221/tcp/4001/p2p/Qme8g49gm3q4Acp7xWBKg3nAa9fxZ1YmyDJdyGgoG6LsXh", +} + // AutoRelay is a Host that uses relays for connectivity when a NAT is detected. type AutoRelay struct { host *basic.BasicHost From be38134b3cf73bd9d2b1f8606f897720b08c838b Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 11 Oct 2019 18:00:47 -0400 Subject: [PATCH 1542/3965] polynomialbackoff: added checks for small degree polynomials --- p2p/discovery/backoff/backoff.go | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/p2p/discovery/backoff/backoff.go b/p2p/discovery/backoff/backoff.go index 15dd7805e1..141e218a5f 100644 --- a/p2p/discovery/backoff/backoff.go +++ b/p2p/discovery/backoff/backoff.go @@ -103,14 +103,22 @@ type polynomialBackoff struct { } func (b *polynomialBackoff) Delay() time.Duration { - polySum := b.poly[0] - exp := 1 - attempt := b.attempt - b.attempt++ - - for _, c := range b.poly[1:] { - exp *= attempt - polySum += float64(exp) * c + var polySum float64 + switch len(b.poly) { + case 0: + return 0 + case 1: + polySum = b.poly[0] + default: + polySum = b.poly[0] + exp := 1 + attempt := b.attempt + b.attempt++ + + for _, c := range b.poly[1:] { + exp *= attempt + polySum += float64(exp) * c + } } return b.jitter(time.Duration(float64(b.timeUnits)*polySum), b.min, b.max, b.rng) } From 66d546e7e874ed0557c61c2306e275941fb2c817 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 11 Oct 2019 18:01:19 -0400 Subject: [PATCH 1543/3965] add backoff connector --- p2p/discovery/backoff/backoffconnector.go | 82 +++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 p2p/discovery/backoff/backoffconnector.go diff --git a/p2p/discovery/backoff/backoffconnector.go b/p2p/discovery/backoff/backoffconnector.go new file mode 100644 index 0000000000..f184d7ee5b --- /dev/null +++ b/p2p/discovery/backoff/backoffconnector.go @@ -0,0 +1,82 @@ +package discovery + +import ( + "context" + "sync" + "time" + + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" + + "github.com/hashicorp/golang-lru/simplelru" +) + +type BackoffConnector struct { + cache simplelru.LRUCache + host host.Host + connTryDur time.Duration + backoff BackoffFactory + mux sync.Mutex +} + +func NewBackoffConnector(h host.Host, cache simplelru.LRUCache, connectionTryDuration time.Duration, backoff BackoffFactory) *BackoffConnector { + return &BackoffConnector{ + cache: cache, + host: h, + connTryDur: connectionTryDuration, + backoff: backoff, + } +} + +type connCacheData struct { + nextTry time.Time + strat BackoffStrategy +} + +func (c *BackoffConnector) Connect(ctx context.Context, peerCh <-chan peer.AddrInfo) { + for { + select { + case pi, ok := <-peerCh: + if !ok { + return + } + + if pi.ID == c.host.ID() || pi.ID == "" { + continue + } + + c.mux.Lock() + val, ok := c.cache.Get(pi.ID) + var cachedPeer *connCacheData + if ok { + tv := val.(*connCacheData) + now := time.Now() + if now.Before(tv.nextTry) { + c.mux.Unlock() + continue + } + + tv.nextTry = now.Add(tv.strat.Delay()) + } else { + cachedPeer = &connCacheData{strat: c.backoff()} + cachedPeer.nextTry = time.Now().Add(cachedPeer.strat.Delay()) + c.cache.Add(pi.ID, cachedPeer) + } + + go func(pi peer.AddrInfo) { + ctx, cancel := context.WithTimeout(ctx, c.connTryDur) + defer cancel() + + err := c.host.Connect(ctx, pi) + if err != nil { + log.Debugf("Error connecting to pubsub peer %s: %s", pi.ID, err.Error()) + return + } + }(pi) + + case <-ctx.Done(): + log.Infof("discovery: backoff connector context error %v", ctx.Err()) + return + } + } +} From d5a0348682b93163fb9c520430cd7ce26f719f12 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Sun, 13 Oct 2019 18:00:49 -0400 Subject: [PATCH 1544/3965] fix and tests for backoff connector --- p2p/discovery/backoff/backoffconnector.go | 15 ++- .../backoff/backoffconnector_test.go | 108 ++++++++++++++++++ 2 files changed, 118 insertions(+), 5 deletions(-) create mode 100644 p2p/discovery/backoff/backoffconnector_test.go diff --git a/p2p/discovery/backoff/backoffconnector.go b/p2p/discovery/backoff/backoffconnector.go index f184d7ee5b..5f6c01d328 100644 --- a/p2p/discovery/backoff/backoffconnector.go +++ b/p2p/discovery/backoff/backoffconnector.go @@ -2,30 +2,34 @@ package discovery import ( "context" + lru "github.com/hashicorp/golang-lru" "sync" "time" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/peer" - - "github.com/hashicorp/golang-lru/simplelru" ) type BackoffConnector struct { - cache simplelru.LRUCache + cache *lru.TwoQueueCache host host.Host connTryDur time.Duration backoff BackoffFactory mux sync.Mutex } -func NewBackoffConnector(h host.Host, cache simplelru.LRUCache, connectionTryDuration time.Duration, backoff BackoffFactory) *BackoffConnector { +func NewBackoffConnector(h host.Host, cacheSize int, connectionTryDuration time.Duration, backoff BackoffFactory) (*BackoffConnector, error) { + cache, err := lru.New2Q(cacheSize) + if err != nil { + return nil, err + } + return &BackoffConnector{ cache: cache, host: h, connTryDur: connectionTryDuration, backoff: backoff, - } + }, nil } type connCacheData struct { @@ -62,6 +66,7 @@ func (c *BackoffConnector) Connect(ctx context.Context, peerCh <-chan peer.AddrI cachedPeer.nextTry = time.Now().Add(cachedPeer.strat.Delay()) c.cache.Add(pi.ID, cachedPeer) } + c.mux.Unlock() go func(pi peer.AddrInfo) { ctx, cancel := context.WithTimeout(ctx, c.connTryDur) diff --git a/p2p/discovery/backoff/backoffconnector_test.go b/p2p/discovery/backoff/backoffconnector_test.go new file mode 100644 index 0000000000..4ff5a647c0 --- /dev/null +++ b/p2p/discovery/backoff/backoffconnector_test.go @@ -0,0 +1,108 @@ +package discovery + +import ( + "context" + "fmt" + "sync" + "testing" + "time" + + bhost "github.com/libp2p/go-libp2p-blankhost" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" +) + +type maxDialHost struct { + host.Host + + mux sync.Mutex + timesDialed map[peer.ID]int + maxTimesToDial map[peer.ID]int +} + +func (h *maxDialHost) Connect(ctx context.Context, ai peer.AddrInfo) error { + pid := ai.ID + + h.mux.Lock() + defer h.mux.Unlock() + numDials := h.timesDialed[pid] + numDials += 1 + h.timesDialed[pid] = numDials + + if maxDials, ok := h.maxTimesToDial[pid]; ok && numDials > maxDials { + return fmt.Errorf("should not be dialing peer %s", pid.String()) + } + + return h.Host.Connect(ctx, ai) +} + +func getNetHosts(t *testing.T, ctx context.Context, n int) []host.Host { + var out []host.Host + + for i := 0; i < n; i++ { + netw := swarmt.GenSwarm(t, ctx) + h := bhost.NewBlankHost(netw) + out = append(out, h) + } + + return out +} + +func loadCh(peers []host.Host) <-chan peer.AddrInfo { + ch := make(chan peer.AddrInfo, len(peers)) + for _, p := range peers { + ch <- p.Peerstore().PeerInfo(p.ID()) + } + close(ch) + return ch +} + +func TestBackoffConnector(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hosts := getNetHosts(t, ctx, 5) + primary := &maxDialHost{ + Host: hosts[0], + mux: sync.Mutex{}, + timesDialed: make(map[peer.ID]int), + maxTimesToDial: map[peer.ID]int{ + hosts[1].ID(): 1, + hosts[2].ID(): 2, + }, + } + + bc, err := NewBackoffConnector(primary, 10, time.Minute, NewFixedBackoff(time.Millisecond*1500)) + if err != nil { + t.Fatal(err) + } + + bc.Connect(ctx, loadCh(hosts)) + + time.Sleep(time.Millisecond * 100) + if expected, actual := len(hosts) - 1, len(primary.Network().Conns()); actual != expected { + t.Fatalf("wrong number of connections. expected %d, actual %d", expected, actual) + } + + for _, c := range primary.Network().Conns() { + c.Close() + } + + for len(primary.Network().Conns()) > 0 { + time.Sleep(time.Millisecond * 100) + } + + bc.Connect(ctx, loadCh(hosts)) + if numConns := len(primary.Network().Conns()); numConns != 0 { + t.Fatal("shouldn't be connected to any peers") + } + + time.Sleep(time.Millisecond * 1600) + bc.Connect(ctx, loadCh(hosts)) + + time.Sleep(time.Millisecond * 100) + if expected, actual := len(hosts) - 2, len(primary.Network().Conns()); actual != expected { + t.Fatalf("wrong number of connections. expected %d, actual %d", expected, actual) + } +} From 204664acc1c283be847e778f2182822b87ecf1e4 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Sun, 13 Oct 2019 18:01:38 -0400 Subject: [PATCH 1545/3965] added configurable channel buffer sizes to the backoff cache. --- p2p/discovery/backoff/backoffcache.go | 47 +++++++++++++++++-- .../backoff/backoffconnector_test.go | 4 +- 2 files changed, 44 insertions(+), 7 deletions(-) diff --git a/p2p/discovery/backoff/backoffcache.go b/p2p/discovery/backoff/backoffcache.go index 1c2d1d0dea..b4f9a4056a 100644 --- a/p2p/discovery/backoff/backoffcache.go +++ b/p2p/discovery/backoff/backoffcache.go @@ -2,6 +2,7 @@ package discovery import ( "context" + "fmt" "sync" "time" @@ -16,14 +17,50 @@ type BackoffDiscovery struct { stratFactory BackoffFactory peerCache map[string]*backoffCache peerCacheMux sync.RWMutex + + parallelBufSz int + returnedBufSz int } -func NewBackoffDiscovery(disc discovery.Discovery, stratFactory BackoffFactory) (discovery.Discovery, error) { - return &BackoffDiscovery{ +type BackoffDiscoveryOption func(*BackoffDiscovery) error + +func NewBackoffDiscovery(disc discovery.Discovery, stratFactory BackoffFactory, opts ...BackoffDiscoveryOption) (discovery.Discovery, error) { + b := &BackoffDiscovery{ disc: disc, stratFactory: stratFactory, peerCache: make(map[string]*backoffCache), - }, nil + + parallelBufSz: 32, + returnedBufSz: 32, + } + + for _, opt := range opts { + if err := opt(b); err != nil { + return nil, err + } + } + + return b, nil +} + +func WithBackoffDiscoverySimultaneousQueryBufferSize(size int) BackoffDiscoveryOption { + return func(b *BackoffDiscovery) error { + if size < 0 { + return fmt.Errorf("cannot set size to be smaller than 0") + } + b.parallelBufSz = size + return nil + } +} + +func WithBackoffDiscoveryReturnedChannelSize(size int) BackoffDiscoveryOption { + return func(b *BackoffDiscovery) error { + if size < 0 { + return fmt.Errorf("cannot set size to be smaller than 0") + } + b.returnedBufSz = size + return nil + } } type backoffCache struct { @@ -119,8 +156,8 @@ func (d *BackoffDiscovery) FindPeers(ctx context.Context, ns string, opts ...dis } // Setup receiver channel for receiving peers from ongoing requests - evtCh := make(chan peer.AddrInfo, 32) - pch := make(chan peer.AddrInfo, 32) + evtCh := make(chan peer.AddrInfo, d.parallelBufSz) + pch := make(chan peer.AddrInfo, d.returnedBufSz) rcvPeers := make([]peer.AddrInfo, 0, 32) for _, ai := range c.peers { rcvPeers = append(rcvPeers, ai) diff --git a/p2p/discovery/backoff/backoffconnector_test.go b/p2p/discovery/backoff/backoffconnector_test.go index 4ff5a647c0..e88a9c2b1e 100644 --- a/p2p/discovery/backoff/backoffconnector_test.go +++ b/p2p/discovery/backoff/backoffconnector_test.go @@ -81,7 +81,7 @@ func TestBackoffConnector(t *testing.T) { bc.Connect(ctx, loadCh(hosts)) time.Sleep(time.Millisecond * 100) - if expected, actual := len(hosts) - 1, len(primary.Network().Conns()); actual != expected { + if expected, actual := len(hosts)-1, len(primary.Network().Conns()); actual != expected { t.Fatalf("wrong number of connections. expected %d, actual %d", expected, actual) } @@ -102,7 +102,7 @@ func TestBackoffConnector(t *testing.T) { bc.Connect(ctx, loadCh(hosts)) time.Sleep(time.Millisecond * 100) - if expected, actual := len(hosts) - 2, len(primary.Network().Conns()); actual != expected { + if expected, actual := len(hosts)-2, len(primary.Network().Conns()); actual != expected { t.Fatalf("wrong number of connections. expected %d, actual %d", expected, actual) } } From eba1a6b4b3b7ceed47b4025cb4536066f5848085 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 21 Oct 2019 15:52:27 -0700 Subject: [PATCH 1546/3965] fix(key size): forbid small openssl RSA keys Also, add a test. --- core/crypto/rsa_openssl.go | 6 ++++++ core/crypto/rsa_test.go | 24 +++++++++++++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/core/crypto/rsa_openssl.go b/core/crypto/rsa_openssl.go index fd9d451043..9b7a342345 100644 --- a/core/crypto/rsa_openssl.go +++ b/core/crypto/rsa_openssl.go @@ -43,6 +43,9 @@ func UnmarshalRsaPrivateKey(b []byte) (PrivKey, error) { if err != nil { return nil, err } + if 8*key.key.Size() < MinRsaKeyBits { + return nil, ErrRsaKeyTooSmall + } if key.Type() != RSA { return nil, errors.New("not actually an rsa public key") } @@ -55,6 +58,9 @@ func UnmarshalRsaPublicKey(b []byte) (PubKey, error) { if err != nil { return nil, err } + if 8*key.key.Size() < MinRsaKeyBits { + return nil, ErrRsaKeyTooSmall + } if key.Type() != RSA { return nil, errors.New("not actually an rsa public key") } diff --git a/core/crypto/rsa_test.go b/core/crypto/rsa_test.go index 08db1366ad..9de03a1850 100644 --- a/core/crypto/rsa_test.go +++ b/core/crypto/rsa_test.go @@ -40,10 +40,32 @@ func TestRSABasicSignAndVerify(t *testing.T) { } func TestRSASmallKey(t *testing.T) { - _, _, err := GenerateRSAKeyPair(384, rand.Reader) + _, _, err := GenerateRSAKeyPair(MinRsaKeyBits/2, rand.Reader) if err != ErrRsaKeyTooSmall { t.Fatal("should have refused to create small RSA key") } + MinRsaKeyBits /= 2 + badPriv, badPub, err := GenerateRSAKeyPair(MinRsaKeyBits, rand.Reader) + if err != nil { + t.Fatalf("should have succeeded, got: %s", err) + } + pubBytes, err := MarshalPublicKey(badPub) + if err != nil { + t.Fatal(err) + } + privBytes, err := MarshalPrivateKey(badPriv) + if err != nil { + t.Fatal(err) + } + MinRsaKeyBits *= 2 + _, err = UnmarshalPublicKey(pubBytes) + if err != ErrRsaKeyTooSmall { + t.Fatal("should have refused to unmarshal a weak key") + } + _, err = UnmarshalPrivateKey(privBytes) + if err != ErrRsaKeyTooSmall { + t.Fatal("should have refused to unmarshal a weak key") + } } func TestRSASignZero(t *testing.T) { From 50b20ffe5b87b28c57cd6b09d2bcec4267d80012 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 25 Oct 2019 11:24:05 -0400 Subject: [PATCH 1547/3965] added backoff docstrings --- p2p/discovery/backoff/backoff.go | 17 +++++++++++++++++ p2p/discovery/backoff/backoffcache.go | 4 ++++ p2p/discovery/backoff/backoffconnector.go | 6 ++++++ 3 files changed, 27 insertions(+) diff --git a/p2p/discovery/backoff/backoff.go b/p2p/discovery/backoff/backoff.go index 37fd6c3c68..944d952b4f 100644 --- a/p2p/discovery/backoff/backoff.go +++ b/p2p/discovery/backoff/backoff.go @@ -8,8 +8,11 @@ import ( type BackoffFactory func() BackoffStrategy +// BackoffStrategy describes how backoff will be implemented. BackoffStratgies are stateful. type BackoffStrategy interface { + // Delay calculates how long the next backoff duration should be, given the prior calls to Delay Delay() time.Duration + // Reset clears the internal state of the BackoffStrategy Reset() } @@ -18,6 +21,8 @@ type BackoffStrategy interface { // Jitter must return a duration between min and max. Min must be lower than, or equal to, max. type Jitter func(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration +// FullJitter returns a random number uniformly chose from the range [min, boundedDur]. +// boundedDur is the duration bounded between min and max. func FullJitter(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration { if duration <= min { return min @@ -28,6 +33,7 @@ func FullJitter(duration time.Duration, min time.Duration, max time.Duration, rn return boundedDuration(time.Duration(rng.Int63n(int64(normalizedDur)))+min, min, max) } +// NoJitter returns the duration bounded between min and max func NoJitter(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration { return boundedDuration(duration, min, max) } @@ -62,6 +68,7 @@ func (b *attemptBackoff) Reset() { b.attempt = 0 } +// NewFixedBackoff creates a BackoffFactory with a constant backoff duration func NewFixedBackoff(delay time.Duration) BackoffFactory { return func() BackoffStrategy { return &fixedBackoff{delay: delay} @@ -78,6 +85,10 @@ func (b *fixedBackoff) Delay() time.Duration { func (b *fixedBackoff) Reset() {} +// NewPolynomialBackoff creates a BackoffFactory with backoff of the form c0*x^0, c1*x^1, ...cn*x^n where x is the attempt number +// jitter is the function for adding randomness around the backoff +// timeUnits are the units of time the polynomial is evaluated in +// polyCoefs is the array of polynomial coefficients from [c0, c1, ... cn] func NewPolynomialBackoff(min, max time.Duration, jitter Jitter, timeUnits time.Duration, polyCoefs []float64, rng *rand.Rand) BackoffFactory { return func() BackoffStrategy { @@ -123,6 +134,9 @@ func (b *polynomialBackoff) Delay() time.Duration { return b.jitter(time.Duration(float64(b.timeUnits)*polySum), b.min, b.max, b.rng) } +// NewExponentialBackoff creates a BackoffFactory with backoff of the form base^x + offset where x is the attempt number +// jitter is the function for adding randomness around the backoff +// timeUnits are the units of time the base^x is evaluated in func NewExponentialBackoff(min, max time.Duration, jitter Jitter, timeUnits time.Duration, base float64, offset time.Duration, rng *rand.Rand) BackoffFactory { return func() BackoffStrategy { @@ -156,6 +170,9 @@ func (b *exponentialBackoff) Delay() time.Duration { time.Duration(math.Pow(b.base, float64(attempt))*float64(b.timeUnits))+b.offset, b.min, b.max, b.rng) } +// NewExponentialDecorrelatedJitter creates a BackoffFactory with backoff of the roughly of the form base^x where x is the attempt number. +// Delays start at the minimum duration and after each attempt delay = rand(min, delay * base), bounded by the max +// See https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ for more information func NewExponentialDecorrelatedJitter(min, max time.Duration, base float64, rng *rand.Rand) BackoffFactory { return func() BackoffStrategy { return &exponentialDecorrelatedJitter{ diff --git a/p2p/discovery/backoff/backoffcache.go b/p2p/discovery/backoff/backoffcache.go index b4f9a4056a..620012eced 100644 --- a/p2p/discovery/backoff/backoffcache.go +++ b/p2p/discovery/backoff/backoffcache.go @@ -43,6 +43,8 @@ func NewBackoffDiscovery(disc discovery.Discovery, stratFactory BackoffFactory, return b, nil } +// WithBackoffDiscoverySimultaneousQueryBufferSize sets the buffer size for the channels between the main FindPeers query +// for a given namespace and all simultaneous FindPeers queries for the namespace func WithBackoffDiscoverySimultaneousQueryBufferSize(size int) BackoffDiscoveryOption { return func(b *BackoffDiscovery) error { if size < 0 { @@ -53,6 +55,8 @@ func WithBackoffDiscoverySimultaneousQueryBufferSize(size int) BackoffDiscoveryO } } +// WithBackoffDiscoveryReturnedChannelSize sets the size of the buffer to be used during a FindPeer query. +// Note: This does not apply if the query occurs during the backoff time func WithBackoffDiscoveryReturnedChannelSize(size int) BackoffDiscoveryOption { return func(b *BackoffDiscovery) error { if size < 0 { diff --git a/p2p/discovery/backoff/backoffconnector.go b/p2p/discovery/backoff/backoffconnector.go index 5f6c01d328..8fe3c2de6a 100644 --- a/p2p/discovery/backoff/backoffconnector.go +++ b/p2p/discovery/backoff/backoffconnector.go @@ -10,6 +10,7 @@ import ( "github.com/libp2p/go-libp2p-core/peer" ) +// BackoffConnector is a utility to connect to peers, but only if we have not recently tried connecting to them already type BackoffConnector struct { cache *lru.TwoQueueCache host host.Host @@ -18,6 +19,10 @@ type BackoffConnector struct { mux sync.Mutex } +// NewBackoffConnector creates a utility to connect to peers, but only if we have not recently tried connecting to them already +// cacheSize is the size of a TwoQueueCache +// connectionTryDuration is how long we attempt to connect to a peer before giving up +// backoff describes the strategy used to decide how long to backoff after previously attempting to connect to a peer func NewBackoffConnector(h host.Host, cacheSize int, connectionTryDuration time.Duration, backoff BackoffFactory) (*BackoffConnector, error) { cache, err := lru.New2Q(cacheSize) if err != nil { @@ -37,6 +42,7 @@ type connCacheData struct { strat BackoffStrategy } +// Connect attemps to connect to the peers passed in by peerCh. Will not connect to peers if they are within the backoff period. func (c *BackoffConnector) Connect(ctx context.Context, peerCh <-chan peer.AddrInfo) { for { select { From b151fef4fed5ed7eb87bfd923af0b275af812768 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 25 Oct 2019 11:51:38 -0400 Subject: [PATCH 1548/3965] merge in discovery based content routing changes. moved mock discovery to a new file. --- p2p/discovery/backoff/backoffcache_test.go | 103 ------------------- p2p/discovery/backoff/mocks_test.go | 112 +++++++++++++++++++++ 2 files changed, 112 insertions(+), 103 deletions(-) create mode 100644 p2p/discovery/backoff/mocks_test.go diff --git a/p2p/discovery/backoff/backoffcache_test.go b/p2p/discovery/backoff/backoffcache_test.go index 636e410477..5d7e5dd6ea 100644 --- a/p2p/discovery/backoff/backoffcache_test.go +++ b/p2p/discovery/backoff/backoffcache_test.go @@ -2,118 +2,15 @@ package discovery import ( "context" - "sync" "testing" "time" bhost "github.com/libp2p/go-libp2p-blankhost" "github.com/libp2p/go-libp2p-core/discovery" - "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/peer" swarmt "github.com/libp2p/go-libp2p-swarm/testing" ) -type mockDiscoveryServer struct { - mx sync.Mutex - db map[string]map[peer.ID]*discoveryRegistration -} - -type discoveryRegistration struct { - info peer.AddrInfo - expiration time.Time -} - -func newDiscoveryServer() *mockDiscoveryServer { - return &mockDiscoveryServer{ - db: make(map[string]map[peer.ID]*discoveryRegistration), - } -} - -func (s *mockDiscoveryServer) Advertise(ns string, info peer.AddrInfo, ttl time.Duration) (time.Duration, error) { - s.mx.Lock() - defer s.mx.Unlock() - - peers, ok := s.db[ns] - if !ok { - peers = make(map[peer.ID]*discoveryRegistration) - s.db[ns] = peers - } - peers[info.ID] = &discoveryRegistration{info, time.Now().Add(ttl)} - return ttl, nil -} - -func (s *mockDiscoveryServer) FindPeers(ns string, limit int) (<-chan peer.AddrInfo, error) { - s.mx.Lock() - defer s.mx.Unlock() - - peers, ok := s.db[ns] - if !ok || len(peers) == 0 { - emptyCh := make(chan peer.AddrInfo) - close(emptyCh) - return emptyCh, nil - } - - count := len(peers) - if limit != 0 && count > limit { - count = limit - } - - iterTime := time.Now() - ch := make(chan peer.AddrInfo, count) - numSent := 0 - for p, reg := range peers { - if numSent == count { - break - } - if iterTime.After(reg.expiration) { - delete(peers, p) - continue - } - - numSent++ - ch <- reg.info - } - close(ch) - - return ch, nil -} - -func (s *mockDiscoveryServer) hasPeerRecord(ns string, pid peer.ID) bool { - s.mx.Lock() - defer s.mx.Unlock() - - if peers, ok := s.db[ns]; ok { - _, ok := peers[pid] - return ok - } - return false -} - -type mockDiscoveryClient struct { - host host.Host - server *mockDiscoveryServer -} - -func (d *mockDiscoveryClient) Advertise(ctx context.Context, ns string, opts ...discovery.Option) (time.Duration, error) { - var options discovery.Options - err := options.Apply(opts...) - if err != nil { - return 0, err - } - - return d.server.Advertise(ns, *host.InfoFromHost(d.host), options.Ttl) -} - -func (d *mockDiscoveryClient) FindPeers(ctx context.Context, ns string, opts ...discovery.Option) (<-chan peer.AddrInfo, error) { - var options discovery.Options - err := options.Apply(opts...) - if err != nil { - return nil, err - } - - return d.server.FindPeers(ns, options.Limit) -} - type delayedDiscovery struct { disc discovery.Discovery delay time.Duration diff --git a/p2p/discovery/backoff/mocks_test.go b/p2p/discovery/backoff/mocks_test.go new file mode 100644 index 0000000000..11f62b09fa --- /dev/null +++ b/p2p/discovery/backoff/mocks_test.go @@ -0,0 +1,112 @@ +package discovery + +import ( + "context" + "sync" + "time" + + "github.com/libp2p/go-libp2p-core/discovery" + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/peer" +) + +type mockDiscoveryServer struct { + mx sync.Mutex + db map[string]map[peer.ID]*discoveryRegistration +} + +type discoveryRegistration struct { + info peer.AddrInfo + expiration time.Time +} + +func newDiscoveryServer() *mockDiscoveryServer { + return &mockDiscoveryServer{ + db: make(map[string]map[peer.ID]*discoveryRegistration), + } +} + +func (s *mockDiscoveryServer) Advertise(ns string, info peer.AddrInfo, ttl time.Duration) (time.Duration, error) { + s.mx.Lock() + defer s.mx.Unlock() + + peers, ok := s.db[ns] + if !ok { + peers = make(map[peer.ID]*discoveryRegistration) + s.db[ns] = peers + } + peers[info.ID] = &discoveryRegistration{info, time.Now().Add(ttl)} + return ttl, nil +} + +func (s *mockDiscoveryServer) FindPeers(ns string, limit int) (<-chan peer.AddrInfo, error) { + s.mx.Lock() + defer s.mx.Unlock() + + peers, ok := s.db[ns] + if !ok || len(peers) == 0 { + emptyCh := make(chan peer.AddrInfo) + close(emptyCh) + return emptyCh, nil + } + + count := len(peers) + if limit != 0 && count > limit { + count = limit + } + + iterTime := time.Now() + ch := make(chan peer.AddrInfo, count) + numSent := 0 + for p, reg := range peers { + if numSent == count { + break + } + if iterTime.After(reg.expiration) { + delete(peers, p) + continue + } + + numSent++ + ch <- reg.info + } + close(ch) + + return ch, nil +} + +func (s *mockDiscoveryServer) hasPeerRecord(ns string, pid peer.ID) bool { + s.mx.Lock() + defer s.mx.Unlock() + + if peers, ok := s.db[ns]; ok { + _, ok := peers[pid] + return ok + } + return false +} + +type mockDiscoveryClient struct { + host host.Host + server *mockDiscoveryServer +} + +func (d *mockDiscoveryClient) Advertise(ctx context.Context, ns string, opts ...discovery.Option) (time.Duration, error) { + var options discovery.Options + err := options.Apply(opts...) + if err != nil { + return 0, err + } + + return d.server.Advertise(ns, *host.InfoFromHost(d.host), options.Ttl) +} + +func (d *mockDiscoveryClient) FindPeers(ctx context.Context, ns string, opts ...discovery.Option) (<-chan peer.AddrInfo, error) { + var options discovery.Options + err := options.Apply(opts...) + if err != nil { + return nil, err + } + + return d.server.FindPeers(ns, options.Limit) +} From bd1a26b08f5521b99329fd25c933d0a0c961e9cb Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 25 Oct 2019 11:51:38 -0400 Subject: [PATCH 1549/3965] merge in discovery based content routing changes. moved mock discovery to a new file. --- p2p/discovery/routing/routing_test.go | 101 -------------------------- 1 file changed, 101 deletions(-) diff --git a/p2p/discovery/routing/routing_test.go b/p2p/discovery/routing/routing_test.go index c07a5d679c..1a956822a4 100644 --- a/p2p/discovery/routing/routing_test.go +++ b/p2p/discovery/routing/routing_test.go @@ -71,107 +71,6 @@ func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit return ch } -type mockDiscoveryServer struct { - mx sync.Mutex - db map[string]map[peer.ID]*discoveryRegistration -} - -type discoveryRegistration struct { - info peer.AddrInfo - expiration time.Time -} - -func newDiscoveryServer() *mockDiscoveryServer { - return &mockDiscoveryServer{ - db: make(map[string]map[peer.ID]*discoveryRegistration), - } -} - -func (s *mockDiscoveryServer) Advertise(ns string, info peer.AddrInfo, ttl time.Duration) (time.Duration, error) { - s.mx.Lock() - defer s.mx.Unlock() - - peers, ok := s.db[ns] - if !ok { - peers = make(map[peer.ID]*discoveryRegistration) - s.db[ns] = peers - } - peers[info.ID] = &discoveryRegistration{info, time.Now().Add(ttl)} - return ttl, nil -} - -func (s *mockDiscoveryServer) FindPeers(ns string, limit int) (<-chan peer.AddrInfo, error) { - s.mx.Lock() - defer s.mx.Unlock() - - peers, ok := s.db[ns] - if !ok || len(peers) == 0 { - emptyCh := make(chan peer.AddrInfo) - close(emptyCh) - return emptyCh, nil - } - - count := len(peers) - if limit != 0 && count > limit { - count = limit - } - - iterTime := time.Now() - ch := make(chan peer.AddrInfo, count) - numSent := 0 - for p, reg := range peers { - if numSent == count { - break - } - if iterTime.After(reg.expiration) { - delete(peers, p) - continue - } - - numSent++ - ch <- reg.info - } - close(ch) - - return ch, nil -} - -func (s *mockDiscoveryServer) hasPeerRecord(ns string, pid peer.ID) bool { - s.mx.Lock() - defer s.mx.Unlock() - - if peers, ok := s.db[ns]; ok { - _, ok := peers[pid] - return ok - } - return false -} - -type mockDiscoveryClient struct { - host host.Host - server *mockDiscoveryServer -} - -func (d *mockDiscoveryClient) Advertise(ctx context.Context, ns string, opts ...discovery.Option) (time.Duration, error) { - var options discovery.Options - err := options.Apply(opts...) - if err != nil { - return 0, err - } - - return d.server.Advertise(ns, *host.InfoFromHost(d.host), options.Ttl) -} - -func (d *mockDiscoveryClient) FindPeers(ctx context.Context, ns string, opts ...discovery.Option) (<-chan peer.AddrInfo, error) { - var options discovery.Options - err := options.Apply(opts...) - if err != nil { - return nil, err - } - - return d.server.FindPeers(ns, options.Limit) -} - func TestRoutingDiscovery(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() From c96acd8c503f671109a75d2e6ec5f14c66b09d7d Mon Sep 17 00:00:00 2001 From: Aliabbas Merchant Date: Sun, 27 Oct 2019 16:39:24 +0530 Subject: [PATCH 1550/3965] Minor Docstring correction --- p2p/net/swarm/swarm_dial.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 40faad7848..054902e7f7 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -47,7 +47,7 @@ var ( // peer we're trying to dial. ErrNoAddresses = errors.New("no addresses") - // ErrNoAddresses is returned when we find addresses for a peer but + // ErrNoGoodAddresses is returned when we find addresses for a peer but // can't use any of them. ErrNoGoodAddresses = errors.New("no good addresses") ) From 29b557438e0a7549b56215ec594d8d02abe34a17 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 28 Oct 2019 13:07:54 -0700 Subject: [PATCH 1551/3965] chore(dep): update protobuf --- core/crypto/pb/crypto.pb.go | 126 ++++++++++++++++-------------------- 1 file changed, 54 insertions(+), 72 deletions(-) diff --git a/core/crypto/pb/crypto.pb.go b/core/crypto/pb/crypto.pb.go index 5fa7aec73f..072fad9c93 100644 --- a/core/crypto/pb/crypto.pb.go +++ b/core/crypto/pb/crypto.pb.go @@ -9,6 +9,7 @@ import ( proto "github.com/gogo/protobuf/proto" io "io" math "math" + math_bits "math/bits" ) // Reference imports to suppress errors if they are not otherwise used. @@ -20,7 +21,7 @@ var _ = math.Inf // is compatible with the proto package it is being compiled against. // A compilation error at this line likely means your copy of the // proto package needs to be updated. -const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package type KeyType int32 @@ -87,7 +88,7 @@ func (m *PublicKey) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { return xxx_messageInfo_PublicKey.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -139,7 +140,7 @@ func (m *PrivateKey) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { return xxx_messageInfo_PrivateKey.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -200,7 +201,7 @@ var fileDescriptor_527278fb02d03321 = []byte{ func (m *PublicKey) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -208,26 +209,32 @@ func (m *PublicKey) Marshal() (dAtA []byte, err error) { } func (m *PublicKey) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *PublicKey) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - dAtA[i] = 0x8 - i++ - i = encodeVarintCrypto(dAtA, i, uint64(m.Type)) if m.Data != nil { - dAtA[i] = 0x12 - i++ + i -= len(m.Data) + copy(dAtA[i:], m.Data) i = encodeVarintCrypto(dAtA, i, uint64(len(m.Data))) - i += copy(dAtA[i:], m.Data) + i-- + dAtA[i] = 0x12 } - return i, nil + i = encodeVarintCrypto(dAtA, i, uint64(m.Type)) + i-- + dAtA[i] = 0x8 + return len(dAtA) - i, nil } func (m *PrivateKey) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -235,30 +242,38 @@ func (m *PrivateKey) Marshal() (dAtA []byte, err error) { } func (m *PrivateKey) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *PrivateKey) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - dAtA[i] = 0x8 - i++ - i = encodeVarintCrypto(dAtA, i, uint64(m.Type)) if m.Data != nil { - dAtA[i] = 0x12 - i++ + i -= len(m.Data) + copy(dAtA[i:], m.Data) i = encodeVarintCrypto(dAtA, i, uint64(len(m.Data))) - i += copy(dAtA[i:], m.Data) + i-- + dAtA[i] = 0x12 } - return i, nil + i = encodeVarintCrypto(dAtA, i, uint64(m.Type)) + i-- + dAtA[i] = 0x8 + return len(dAtA) - i, nil } func encodeVarintCrypto(dAtA []byte, offset int, v uint64) int { + offset -= sovCrypto(v) + base := offset for v >= 1<<7 { dAtA[offset] = uint8(v&0x7f | 0x80) v >>= 7 offset++ } dAtA[offset] = uint8(v) - return offset + 1 + return base } func (m *PublicKey) Size() (n int) { if m == nil { @@ -289,14 +304,7 @@ func (m *PrivateKey) Size() (n int) { } func sovCrypto(x uint64) (n int) { - for { - n++ - x >>= 7 - if x == 0 { - break - } - } - return n + return (math_bits.Len64(x|1) + 6) / 7 } func sozCrypto(x uint64) (n int) { return sovCrypto(uint64((x << 1) ^ uint64((int64(x) >> 63)))) @@ -534,6 +542,7 @@ func (m *PrivateKey) Unmarshal(dAtA []byte) error { func skipCrypto(dAtA []byte) (n int, err error) { l := len(dAtA) iNdEx := 0 + depth := 0 for iNdEx < l { var wire uint64 for shift := uint(0); ; shift += 7 { @@ -565,10 +574,8 @@ func skipCrypto(dAtA []byte) (n int, err error) { break } } - return iNdEx, nil case 1: iNdEx += 8 - return iNdEx, nil case 2: var length int for shift := uint(0); ; shift += 7 { @@ -589,55 +596,30 @@ func skipCrypto(dAtA []byte) (n int, err error) { return 0, ErrInvalidLengthCrypto } iNdEx += length - if iNdEx < 0 { - return 0, ErrInvalidLengthCrypto - } - return iNdEx, nil case 3: - for { - var innerWire uint64 - var start int = iNdEx - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowCrypto - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - innerWire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - innerWireType := int(innerWire & 0x7) - if innerWireType == 4 { - break - } - next, err := skipCrypto(dAtA[start:]) - if err != nil { - return 0, err - } - iNdEx = start + next - if iNdEx < 0 { - return 0, ErrInvalidLengthCrypto - } - } - return iNdEx, nil + depth++ case 4: - return iNdEx, nil + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupCrypto + } + depth-- case 5: iNdEx += 4 - return iNdEx, nil default: return 0, fmt.Errorf("proto: illegal wireType %d", wireType) } + if iNdEx < 0 { + return 0, ErrInvalidLengthCrypto + } + if depth == 0 { + return iNdEx, nil + } } - panic("unreachable") + return 0, io.ErrUnexpectedEOF } var ( - ErrInvalidLengthCrypto = fmt.Errorf("proto: negative length found during unmarshaling") - ErrIntOverflowCrypto = fmt.Errorf("proto: integer overflow") + ErrInvalidLengthCrypto = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowCrypto = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupCrypto = fmt.Errorf("proto: unexpected end of group") ) From 62b9a38323c8d237a2eeb46d0cf2f24596f63f1e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 28 Oct 2019 15:32:22 -0700 Subject: [PATCH 1552/3965] chore(dep): update deps * update go-libp2p-core and fix tests for new minimum key size. This slows down the test but terribly. * switch to go-multiaddr-fmt * update the datastore * update gogo/protobuf --- p2p/host/peerstore/addr/sorting.go | 2 +- p2p/host/peerstore/pb/pstore.pb.go | 170 ++++++++++----------- p2p/host/peerstore/test/keybook_suite.go | 57 ++++--- p2p/host/peerstore/test/peerstore_suite.go | 20 ++- 4 files changed, 128 insertions(+), 121 deletions(-) diff --git a/p2p/host/peerstore/addr/sorting.go b/p2p/host/peerstore/addr/sorting.go index f8a89150a2..5c7c6c3a3d 100644 --- a/p2p/host/peerstore/addr/sorting.go +++ b/p2p/host/peerstore/addr/sorting.go @@ -4,8 +4,8 @@ import ( "bytes" ma "github.com/multiformats/go-multiaddr" + mafmt "github.com/multiformats/go-multiaddr-fmt" manet "github.com/multiformats/go-multiaddr-net" - mafmt "github.com/whyrusleeping/mafmt" ) func isFDCostlyTransport(a ma.Multiaddr) bool { diff --git a/p2p/host/peerstore/pb/pstore.pb.go b/p2p/host/peerstore/pb/pstore.pb.go index c80958cce5..c3dbaeb344 100644 --- a/p2p/host/peerstore/pb/pstore.pb.go +++ b/p2p/host/peerstore/pb/pstore.pb.go @@ -9,6 +9,7 @@ import ( proto "github.com/gogo/protobuf/proto" io "io" math "math" + math_bits "math/bits" ) // Reference imports to suppress errors if they are not otherwise used. @@ -20,7 +21,7 @@ var _ = math.Inf // is compatible with the proto package it is being compiled against. // A compilation error at this line likely means your copy of the // proto package needs to be updated. -const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package // AddrBookRecord represents a record for a peer in the address book. type AddrBookRecord struct { @@ -44,7 +45,7 @@ func (m *AddrBookRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, erro return xxx_messageInfo_AddrBookRecord.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -93,7 +94,7 @@ func (m *AddrBookRecord_AddrEntry) XXX_Marshal(b []byte, deterministic bool) ([] return xxx_messageInfo_AddrBookRecord_AddrEntry.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -156,7 +157,7 @@ var fileDescriptor_f96873690e08a98f = []byte{ func (m *AddrBookRecord) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -164,39 +165,48 @@ func (m *AddrBookRecord) Marshal() (dAtA []byte, err error) { } func (m *AddrBookRecord) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *AddrBookRecord) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.Id != nil { - dAtA[i] = 0xa - i++ - i = encodeVarintPstore(dAtA, i, uint64(m.Id.Size())) - n1, err := m.Id.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err - } - i += n1 - } if len(m.Addrs) > 0 { - for _, msg := range m.Addrs { + for iNdEx := len(m.Addrs) - 1; iNdEx >= 0; iNdEx-- { + { + size, err := m.Addrs[iNdEx].MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintPstore(dAtA, i, uint64(size)) + } + i-- dAtA[i] = 0x12 - i++ - i = encodeVarintPstore(dAtA, i, uint64(msg.Size())) - n, err := msg.MarshalTo(dAtA[i:]) - if err != nil { + } + } + if m.Id != nil { + { + size := m.Id.Size() + i -= size + if _, err := m.Id.MarshalTo(dAtA[i:]); err != nil { return 0, err } - i += n + i = encodeVarintPstore(dAtA, i, uint64(size)) } + i-- + dAtA[i] = 0xa } - return i, nil + return len(dAtA) - i, nil } func (m *AddrBookRecord_AddrEntry) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -204,46 +214,55 @@ func (m *AddrBookRecord_AddrEntry) Marshal() (dAtA []byte, err error) { } func (m *AddrBookRecord_AddrEntry) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *AddrBookRecord_AddrEntry) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.Addr != nil { - dAtA[i] = 0xa - i++ - i = encodeVarintPstore(dAtA, i, uint64(m.Addr.Size())) - n2, err := m.Addr.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err - } - i += n2 + if m.Ttl != 0 { + i = encodeVarintPstore(dAtA, i, uint64(m.Ttl)) + i-- + dAtA[i] = 0x18 } if m.Expiry != 0 { - dAtA[i] = 0x10 - i++ i = encodeVarintPstore(dAtA, i, uint64(m.Expiry)) + i-- + dAtA[i] = 0x10 } - if m.Ttl != 0 { - dAtA[i] = 0x18 - i++ - i = encodeVarintPstore(dAtA, i, uint64(m.Ttl)) + if m.Addr != nil { + { + size := m.Addr.Size() + i -= size + if _, err := m.Addr.MarshalTo(dAtA[i:]); err != nil { + return 0, err + } + i = encodeVarintPstore(dAtA, i, uint64(size)) + } + i-- + dAtA[i] = 0xa } - return i, nil + return len(dAtA) - i, nil } func encodeVarintPstore(dAtA []byte, offset int, v uint64) int { + offset -= sovPstore(v) + base := offset for v >= 1<<7 { dAtA[offset] = uint8(v&0x7f | 0x80) v >>= 7 offset++ } dAtA[offset] = uint8(v) - return offset + 1 + return base } func NewPopulatedAddrBookRecord(r randyPstore, easy bool) *AddrBookRecord { this := &AddrBookRecord{} this.Id = NewPopulatedProtoPeerID(r) - if r.Intn(10) != 0 { + if r.Intn(5) != 0 { v1 := r.Intn(5) this.Addrs = make([]*AddrBookRecord_AddrEntry, v1) for i := 0; i < v1; i++ { @@ -382,14 +401,7 @@ func (m *AddrBookRecord_AddrEntry) Size() (n int) { } func sovPstore(x uint64) (n int) { - for { - n++ - x >>= 7 - if x == 0 { - break - } - } - return n + return (math_bits.Len64(x|1) + 6) / 7 } func sozPstore(x uint64) (n int) { return sovPstore(uint64((x << 1) ^ uint64((int64(x) >> 63)))) @@ -645,6 +657,7 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { func skipPstore(dAtA []byte) (n int, err error) { l := len(dAtA) iNdEx := 0 + depth := 0 for iNdEx < l { var wire uint64 for shift := uint(0); ; shift += 7 { @@ -676,10 +689,8 @@ func skipPstore(dAtA []byte) (n int, err error) { break } } - return iNdEx, nil case 1: iNdEx += 8 - return iNdEx, nil case 2: var length int for shift := uint(0); ; shift += 7 { @@ -700,55 +711,30 @@ func skipPstore(dAtA []byte) (n int, err error) { return 0, ErrInvalidLengthPstore } iNdEx += length - if iNdEx < 0 { - return 0, ErrInvalidLengthPstore - } - return iNdEx, nil case 3: - for { - var innerWire uint64 - var start int = iNdEx - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowPstore - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - innerWire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - innerWireType := int(innerWire & 0x7) - if innerWireType == 4 { - break - } - next, err := skipPstore(dAtA[start:]) - if err != nil { - return 0, err - } - iNdEx = start + next - if iNdEx < 0 { - return 0, ErrInvalidLengthPstore - } - } - return iNdEx, nil + depth++ case 4: - return iNdEx, nil + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupPstore + } + depth-- case 5: iNdEx += 4 - return iNdEx, nil default: return 0, fmt.Errorf("proto: illegal wireType %d", wireType) } + if iNdEx < 0 { + return 0, ErrInvalidLengthPstore + } + if depth == 0 { + return iNdEx, nil + } } - panic("unreachable") + return 0, io.ErrUnexpectedEOF } var ( - ErrInvalidLengthPstore = fmt.Errorf("proto: negative length found during unmarshaling") - ErrIntOverflowPstore = fmt.Errorf("proto: integer overflow") + ErrInvalidLengthPstore = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowPstore = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupPstore = fmt.Errorf("proto: unexpected end of group") ) diff --git a/p2p/host/peerstore/test/keybook_suite.go b/p2p/host/peerstore/test/keybook_suite.go index c8dbc268bb..ec6d9362cb 100644 --- a/p2p/host/peerstore/test/keybook_suite.go +++ b/p2p/host/peerstore/test/keybook_suite.go @@ -41,9 +41,9 @@ func testKeybookPrivKey(kb pstore.KeyBook) func(t *testing.T) { t.Error("expected peers to be empty on init") } - priv, _, err := pt.RandTestKeyPair(ic.RSA, 512) + priv, _, err := pt.RandTestKeyPair(ic.RSA, 2048) if err != nil { - t.Error(err) + t.Fatal(err) } id, err := peer.IDFromPrivateKey(priv) @@ -76,14 +76,14 @@ func testKeyBookPubKey(kb pstore.KeyBook) func(t *testing.T) { t.Error("expected peers to be empty on init") } - _, pub, err := pt.RandTestKeyPair(ic.RSA, 512) + _, pub, err := pt.RandTestKeyPair(ic.RSA, 2048) if err != nil { - t.Error(err) + t.Fatal(err) } id, err := peer.IDFromPublicKey(pub) if err != nil { - t.Error(err) + t.Fatal(err) } if res := kb.PubKey(id); res != nil { @@ -114,13 +114,22 @@ func testKeyBookPeers(kb pstore.KeyBook) func(t *testing.T) { var peers peer.IDSlice for i := 0; i < 10; i++ { // Add a public key. - _, pub, _ := pt.RandTestKeyPair(ic.RSA, 512) - p1, _ := peer.IDFromPublicKey(pub) + _, pub, err := pt.RandTestKeyPair(ic.RSA, 2048) + if err != nil { + t.Fatal(err) + } + p1, err := peer.IDFromPublicKey(pub) + if err != nil { + t.Fatal(err) + } kb.AddPubKey(p1, pub) // Add a private key. - priv, _, _ := pt.RandTestKeyPair(ic.RSA, 512) - p2, _ := peer.IDFromPrivateKey(priv) + priv, _, err := pt.RandTestKeyPair(ic.RSA, 2048) + p2, err := peer.IDFromPrivateKey(priv) + if err != nil { + t.Fatal(err) + } kb.AddPrivKey(p2, priv) peers = append(peers, []peer.ID{p1, p2}...) @@ -192,14 +201,14 @@ func BenchmarkKeyBook(b *testing.B, factory KeyBookFactory) { func benchmarkPubKey(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { - _, pub, err := pt.RandTestKeyPair(ic.RSA, 512) + _, pub, err := pt.RandTestKeyPair(ic.RSA, 2048) if err != nil { - b.Error(err) + b.Fatal(err) } id, err := peer.IDFromPublicKey(pub) if err != nil { - b.Error(err) + b.Fatal(err) } err = kb.AddPubKey(id, pub) @@ -216,14 +225,14 @@ func benchmarkPubKey(kb pstore.KeyBook) func(*testing.B) { func benchmarkAddPubKey(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { - _, pub, err := pt.RandTestKeyPair(ic.RSA, 512) + _, pub, err := pt.RandTestKeyPair(ic.RSA, 2048) if err != nil { - b.Error(err) + b.Fatal(err) } id, err := peer.IDFromPublicKey(pub) if err != nil { - b.Error(err) + b.Fatal(err) } b.ResetTimer() @@ -235,14 +244,14 @@ func benchmarkAddPubKey(kb pstore.KeyBook) func(*testing.B) { func benchmarkPrivKey(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { - priv, _, err := pt.RandTestKeyPair(ic.RSA, 512) + priv, _, err := pt.RandTestKeyPair(ic.RSA, 2048) if err != nil { - b.Error(err) + b.Fatal(err) } id, err := peer.IDFromPrivateKey(priv) if err != nil { - b.Error(err) + b.Fatal(err) } err = kb.AddPrivKey(id, priv) @@ -259,14 +268,14 @@ func benchmarkPrivKey(kb pstore.KeyBook) func(*testing.B) { func benchmarkAddPrivKey(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { - priv, _, err := pt.RandTestKeyPair(ic.RSA, 512) + priv, _, err := pt.RandTestKeyPair(ic.RSA, 2048) if err != nil { - b.Error(err) + b.Fatal(err) } id, err := peer.IDFromPrivateKey(priv) if err != nil { - b.Error(err) + b.Fatal(err) } b.ResetTimer() @@ -279,14 +288,14 @@ func benchmarkAddPrivKey(kb pstore.KeyBook) func(*testing.B) { func benchmarkPeersWithKeys(kb pstore.KeyBook) func(*testing.B) { return func(b *testing.B) { for i := 0; i < 10; i++ { - priv, pub, err := pt.RandTestKeyPair(ic.RSA, 512) + priv, pub, err := pt.RandTestKeyPair(ic.RSA, 2048) if err != nil { - b.Error(err) + b.Fatal(err) } id, err := peer.IDFromPublicKey(pub) if err != nil { - b.Error(err) + b.Fatal(err) } err = kb.AddPubKey(id, pub) diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index fb4a813769..a0ae326c9a 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -287,8 +287,14 @@ func testBasicPeerstore(ps pstore.Peerstore) func(t *testing.T) { addrs := getAddrs(t, 10) for _, a := range addrs { - priv, _, _ := crypto.GenerateKeyPair(crypto.RSA, 512) - p, _ := peer.IDFromPrivateKey(priv) + priv, _, err := crypto.GenerateKeyPair(crypto.RSA, 2048) + if err != nil { + t.Fatal(err) + } + p, err := peer.IDFromPrivateKey(priv) + if err != nil { + t.Fatal(err) + } pids = append(pids, p) ps.AddAddr(p, a, pstore.PermanentAddrTTL) } @@ -309,8 +315,14 @@ func testMetadata(ps pstore.Peerstore) func(t *testing.T) { return func(t *testing.T) { pids := make([]peer.ID, 10) for i := range pids { - priv, _, _ := crypto.GenerateKeyPair(crypto.RSA, 512) - p, _ := peer.IDFromPrivateKey(priv) + priv, _, err := crypto.GenerateKeyPair(crypto.RSA, 2048) + if err != nil { + t.Fatal(err) + } + p, err := peer.IDFromPrivateKey(priv) + if err != nil { + t.Fatal(err) + } pids[i] = p } for _, p := range pids { From 05f1051a69bea62b082396426e2327ce459fd128 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 28 Oct 2019 15:47:17 -0700 Subject: [PATCH 1553/3965] chore(dep): update deps * gogo protobuf * libp2p core --- .../circuitv1-deprecated/pb/relay.pb.go | 180 ++++++++---------- 1 file changed, 84 insertions(+), 96 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go index ca165ee60d..66703f15aa 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go +++ b/p2p/protocol/internal/circuitv1-deprecated/pb/relay.pb.go @@ -9,6 +9,7 @@ import ( proto "github.com/gogo/protobuf/proto" io "io" math "math" + math_bits "math/bits" ) // Reference imports to suppress errors if they are not otherwise used. @@ -20,7 +21,7 @@ var _ = math.Inf // is compatible with the proto package it is being compiled against. // A compilation error at this line likely means your copy of the // proto package needs to be updated. -const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package type CircuitRelay_Status int32 @@ -174,7 +175,7 @@ func (m *CircuitRelay) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) return xxx_messageInfo_CircuitRelay.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -243,7 +244,7 @@ func (m *CircuitRelay_Peer) XXX_Marshal(b []byte, deterministic bool) ([]byte, e return xxx_messageInfo_CircuitRelay_Peer.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -322,7 +323,7 @@ var fileDescriptor_9f69a7d5a802d584 = []byte{ func (m *CircuitRelay) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -330,50 +331,60 @@ func (m *CircuitRelay) Marshal() (dAtA []byte, err error) { } func (m *CircuitRelay) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *CircuitRelay) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.Type != nil { - dAtA[i] = 0x8 - i++ - i = encodeVarintRelay(dAtA, i, uint64(*m.Type)) + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) } - if m.SrcPeer != nil { - dAtA[i] = 0x12 - i++ - i = encodeVarintRelay(dAtA, i, uint64(m.SrcPeer.Size())) - n1, err := m.SrcPeer.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err - } - i += n1 + if m.Code != nil { + i = encodeVarintRelay(dAtA, i, uint64(*m.Code)) + i-- + dAtA[i] = 0x20 } if m.DstPeer != nil { - dAtA[i] = 0x1a - i++ - i = encodeVarintRelay(dAtA, i, uint64(m.DstPeer.Size())) - n2, err := m.DstPeer.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err + { + size, err := m.DstPeer.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintRelay(dAtA, i, uint64(size)) } - i += n2 + i-- + dAtA[i] = 0x1a } - if m.Code != nil { - dAtA[i] = 0x20 - i++ - i = encodeVarintRelay(dAtA, i, uint64(*m.Code)) + if m.SrcPeer != nil { + { + size, err := m.SrcPeer.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintRelay(dAtA, i, uint64(size)) + } + i-- + dAtA[i] = 0x12 } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) + if m.Type != nil { + i = encodeVarintRelay(dAtA, i, uint64(*m.Type)) + i-- + dAtA[i] = 0x8 } - return i, nil + return len(dAtA) - i, nil } func (m *CircuitRelay_Peer) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -381,40 +392,50 @@ func (m *CircuitRelay_Peer) Marshal() (dAtA []byte, err error) { } func (m *CircuitRelay_Peer) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *CircuitRelay_Peer) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.Id == nil { - return 0, github_com_gogo_protobuf_proto.NewRequiredNotSetError("id") - } else { - dAtA[i] = 0xa - i++ - i = encodeVarintRelay(dAtA, i, uint64(len(m.Id))) - i += copy(dAtA[i:], m.Id) + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) } if len(m.Addrs) > 0 { - for _, b := range m.Addrs { + for iNdEx := len(m.Addrs) - 1; iNdEx >= 0; iNdEx-- { + i -= len(m.Addrs[iNdEx]) + copy(dAtA[i:], m.Addrs[iNdEx]) + i = encodeVarintRelay(dAtA, i, uint64(len(m.Addrs[iNdEx]))) + i-- dAtA[i] = 0x12 - i++ - i = encodeVarintRelay(dAtA, i, uint64(len(b))) - i += copy(dAtA[i:], b) } } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) + if m.Id == nil { + return 0, github_com_gogo_protobuf_proto.NewRequiredNotSetError("id") + } else { + i -= len(m.Id) + copy(dAtA[i:], m.Id) + i = encodeVarintRelay(dAtA, i, uint64(len(m.Id))) + i-- + dAtA[i] = 0xa } - return i, nil + return len(dAtA) - i, nil } func encodeVarintRelay(dAtA []byte, offset int, v uint64) int { + offset -= sovRelay(v) + base := offset for v >= 1<<7 { dAtA[offset] = uint8(v&0x7f | 0x80) v >>= 7 offset++ } dAtA[offset] = uint8(v) - return offset + 1 + return base } func (m *CircuitRelay) Size() (n int) { if m == nil { @@ -465,14 +486,7 @@ func (m *CircuitRelay_Peer) Size() (n int) { } func sovRelay(x uint64) (n int) { - for { - n++ - x >>= 7 - if x == 0 { - break - } - } - return n + return (math_bits.Len64(x|1) + 6) / 7 } func sozRelay(x uint64) (n int) { return sovRelay(uint64((x << 1) ^ uint64((int64(x) >> 63)))) @@ -771,6 +785,7 @@ func (m *CircuitRelay_Peer) Unmarshal(dAtA []byte) error { func skipRelay(dAtA []byte) (n int, err error) { l := len(dAtA) iNdEx := 0 + depth := 0 for iNdEx < l { var wire uint64 for shift := uint(0); ; shift += 7 { @@ -802,10 +817,8 @@ func skipRelay(dAtA []byte) (n int, err error) { break } } - return iNdEx, nil case 1: iNdEx += 8 - return iNdEx, nil case 2: var length int for shift := uint(0); ; shift += 7 { @@ -829,52 +842,27 @@ func skipRelay(dAtA []byte) (n int, err error) { if iNdEx < 0 { return 0, ErrInvalidLengthRelay } - return iNdEx, nil case 3: - for { - var innerWire uint64 - var start int = iNdEx - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowRelay - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - innerWire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - innerWireType := int(innerWire & 0x7) - if innerWireType == 4 { - break - } - next, err := skipRelay(dAtA[start:]) - if err != nil { - return 0, err - } - iNdEx = start + next - if iNdEx < 0 { - return 0, ErrInvalidLengthRelay - } - } - return iNdEx, nil + depth++ case 4: - return iNdEx, nil + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupRelay + } + depth-- case 5: iNdEx += 4 - return iNdEx, nil default: return 0, fmt.Errorf("proto: illegal wireType %d", wireType) } + if depth == 0 { + return iNdEx, nil + } } - panic("unreachable") + return 0, io.ErrUnexpectedEOF } var ( - ErrInvalidLengthRelay = fmt.Errorf("proto: negative length found during unmarshaling") - ErrIntOverflowRelay = fmt.Errorf("proto: integer overflow") + ErrInvalidLengthRelay = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowRelay = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupRelay = fmt.Errorf("proto: unexpected end of group") ) From 2a821c45480945ed131000bf1564ad430e0b5f35 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 28 Oct 2019 16:07:28 -0700 Subject: [PATCH 1554/3965] chore(dep): update * update gogo/protobuf and regenerate protobuf files * update go-libp2p --- p2p/host/autonat/pb/autonat.pb.go | 250 +++++++++++++++--------------- 1 file changed, 127 insertions(+), 123 deletions(-) diff --git a/p2p/host/autonat/pb/autonat.pb.go b/p2p/host/autonat/pb/autonat.pb.go index 3617b4615a..b176cb99e0 100644 --- a/p2p/host/autonat/pb/autonat.pb.go +++ b/p2p/host/autonat/pb/autonat.pb.go @@ -8,6 +8,7 @@ import ( proto "github.com/gogo/protobuf/proto" io "io" math "math" + math_bits "math/bits" ) // Reference imports to suppress errors if they are not otherwise used. @@ -19,7 +20,7 @@ var _ = math.Inf // is compatible with the proto package it is being compiled against. // A compilation error at this line likely means your copy of the // proto package needs to be updated. -const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package type Message_MessageType int32 @@ -133,7 +134,7 @@ func (m *Message) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { return xxx_messageInfo_Message.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -195,7 +196,7 @@ func (m *Message_PeerInfo) XXX_Marshal(b []byte, deterministic bool) ([]byte, er return xxx_messageInfo_Message_PeerInfo.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -249,7 +250,7 @@ func (m *Message_Dial) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) return xxx_messageInfo_Message_Dial.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -298,7 +299,7 @@ func (m *Message_DialResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte return xxx_messageInfo_Message_DialResponse.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -380,7 +381,7 @@ var fileDescriptor_a04e278ef61ac07a = []byte{ func (m *Message) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -388,45 +389,55 @@ func (m *Message) Marshal() (dAtA []byte, err error) { } func (m *Message) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *Message) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.Type != nil { - dAtA[i] = 0x8 - i++ - i = encodeVarintAutonat(dAtA, i, uint64(*m.Type)) - } - if m.Dial != nil { - dAtA[i] = 0x12 - i++ - i = encodeVarintAutonat(dAtA, i, uint64(m.Dial.Size())) - n1, err := m.Dial.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err - } - i += n1 + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) } if m.DialResponse != nil { + { + size, err := m.DialResponse.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintAutonat(dAtA, i, uint64(size)) + } + i-- dAtA[i] = 0x1a - i++ - i = encodeVarintAutonat(dAtA, i, uint64(m.DialResponse.Size())) - n2, err := m.DialResponse.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err + } + if m.Dial != nil { + { + size, err := m.Dial.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintAutonat(dAtA, i, uint64(size)) } - i += n2 + i-- + dAtA[i] = 0x12 } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) + if m.Type != nil { + i = encodeVarintAutonat(dAtA, i, uint64(*m.Type)) + i-- + dAtA[i] = 0x8 } - return i, nil + return len(dAtA) - i, nil } func (m *Message_PeerInfo) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -434,34 +445,42 @@ func (m *Message_PeerInfo) Marshal() (dAtA []byte, err error) { } func (m *Message_PeerInfo) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *Message_PeerInfo) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.Id != nil { - dAtA[i] = 0xa - i++ - i = encodeVarintAutonat(dAtA, i, uint64(len(m.Id))) - i += copy(dAtA[i:], m.Id) + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) } if len(m.Addrs) > 0 { - for _, b := range m.Addrs { + for iNdEx := len(m.Addrs) - 1; iNdEx >= 0; iNdEx-- { + i -= len(m.Addrs[iNdEx]) + copy(dAtA[i:], m.Addrs[iNdEx]) + i = encodeVarintAutonat(dAtA, i, uint64(len(m.Addrs[iNdEx]))) + i-- dAtA[i] = 0x12 - i++ - i = encodeVarintAutonat(dAtA, i, uint64(len(b))) - i += copy(dAtA[i:], b) } } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) + if m.Id != nil { + i -= len(m.Id) + copy(dAtA[i:], m.Id) + i = encodeVarintAutonat(dAtA, i, uint64(len(m.Id))) + i-- + dAtA[i] = 0xa } - return i, nil + return len(dAtA) - i, nil } func (m *Message_Dial) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -469,30 +488,38 @@ func (m *Message_Dial) Marshal() (dAtA []byte, err error) { } func (m *Message_Dial) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *Message_Dial) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } if m.Peer != nil { - dAtA[i] = 0xa - i++ - i = encodeVarintAutonat(dAtA, i, uint64(m.Peer.Size())) - n3, err := m.Peer.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err + { + size, err := m.Peer.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintAutonat(dAtA, i, uint64(size)) } - i += n3 - } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) + i-- + dAtA[i] = 0xa } - return i, nil + return len(dAtA) - i, nil } func (m *Message_DialResponse) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -500,41 +527,51 @@ func (m *Message_DialResponse) Marshal() (dAtA []byte, err error) { } func (m *Message_DialResponse) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *Message_DialResponse) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.Status != nil { - dAtA[i] = 0x8 - i++ - i = encodeVarintAutonat(dAtA, i, uint64(*m.Status)) - } - if m.StatusText != nil { - dAtA[i] = 0x12 - i++ - i = encodeVarintAutonat(dAtA, i, uint64(len(*m.StatusText))) - i += copy(dAtA[i:], *m.StatusText) + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) } if m.Addr != nil { - dAtA[i] = 0x1a - i++ + i -= len(m.Addr) + copy(dAtA[i:], m.Addr) i = encodeVarintAutonat(dAtA, i, uint64(len(m.Addr))) - i += copy(dAtA[i:], m.Addr) + i-- + dAtA[i] = 0x1a } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) + if m.StatusText != nil { + i -= len(*m.StatusText) + copy(dAtA[i:], *m.StatusText) + i = encodeVarintAutonat(dAtA, i, uint64(len(*m.StatusText))) + i-- + dAtA[i] = 0x12 } - return i, nil + if m.Status != nil { + i = encodeVarintAutonat(dAtA, i, uint64(*m.Status)) + i-- + dAtA[i] = 0x8 + } + return len(dAtA) - i, nil } func encodeVarintAutonat(dAtA []byte, offset int, v uint64) int { + offset -= sovAutonat(v) + base := offset for v >= 1<<7 { dAtA[offset] = uint8(v&0x7f | 0x80) v >>= 7 offset++ } dAtA[offset] = uint8(v) - return offset + 1 + return base } func (m *Message) Size() (n int) { if m == nil { @@ -621,14 +658,7 @@ func (m *Message_DialResponse) Size() (n int) { } func sovAutonat(x uint64) (n int) { - for { - n++ - x >>= 7 - if x == 0 { - break - } - } - return n + return (math_bits.Len64(x|1) + 6) / 7 } func sozAutonat(x uint64) (n int) { return sovAutonat(uint64((x << 1) ^ uint64((int64(x) >> 63)))) @@ -1133,6 +1163,7 @@ func (m *Message_DialResponse) Unmarshal(dAtA []byte) error { func skipAutonat(dAtA []byte) (n int, err error) { l := len(dAtA) iNdEx := 0 + depth := 0 for iNdEx < l { var wire uint64 for shift := uint(0); ; shift += 7 { @@ -1164,10 +1195,8 @@ func skipAutonat(dAtA []byte) (n int, err error) { break } } - return iNdEx, nil case 1: iNdEx += 8 - return iNdEx, nil case 2: var length int for shift := uint(0); ; shift += 7 { @@ -1191,52 +1220,27 @@ func skipAutonat(dAtA []byte) (n int, err error) { if iNdEx < 0 { return 0, ErrInvalidLengthAutonat } - return iNdEx, nil case 3: - for { - var innerWire uint64 - var start int = iNdEx - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowAutonat - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - innerWire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - innerWireType := int(innerWire & 0x7) - if innerWireType == 4 { - break - } - next, err := skipAutonat(dAtA[start:]) - if err != nil { - return 0, err - } - iNdEx = start + next - if iNdEx < 0 { - return 0, ErrInvalidLengthAutonat - } - } - return iNdEx, nil + depth++ case 4: - return iNdEx, nil + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupAutonat + } + depth-- case 5: iNdEx += 4 - return iNdEx, nil default: return 0, fmt.Errorf("proto: illegal wireType %d", wireType) } + if depth == 0 { + return iNdEx, nil + } } - panic("unreachable") + return 0, io.ErrUnexpectedEOF } var ( - ErrInvalidLengthAutonat = fmt.Errorf("proto: negative length found during unmarshaling") - ErrIntOverflowAutonat = fmt.Errorf("proto: integer overflow") + ErrInvalidLengthAutonat = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowAutonat = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupAutonat = fmt.Errorf("proto: unexpected end of group") ) From 94d7efc58ef542caa9447e03e38d89718ba36d62 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 28 Oct 2019 16:41:54 -0700 Subject: [PATCH 1555/3965] chore(dep): update libp2p-core and protobuf * Enforces 2048 bit RSA keys when using OpenSSL * Updates go-multiaddr-dns for /dns support. * Update gogo/protobuf and regenerate all protobufs. --- go.mod | 17 +- go.sum | 36 +++- p2p/protocol/identify/pb/identify.pb.go | 237 +++++++++++------------- 3 files changed, 139 insertions(+), 151 deletions(-) diff --git a/go.mod b/go.mod index 5f909db544..14464f2703 100644 --- a/go.mod +++ b/go.mod @@ -1,7 +1,7 @@ module github.com/libp2p/go-libp2p require ( - github.com/gogo/protobuf v1.3.0 + github.com/gogo/protobuf v1.3.1 github.com/ipfs/go-cid v0.0.3 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 @@ -10,17 +10,17 @@ require ( github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 - github.com/libp2p/go-libp2p-autonat v0.1.0 + github.com/libp2p/go-libp2p-autonat v0.1.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 - github.com/libp2p/go-libp2p-circuit v0.1.3 - github.com/libp2p/go-libp2p-core v0.2.3 + github.com/libp2p/go-libp2p-circuit v0.1.4 + github.com/libp2p/go-libp2p-core v0.2.4 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 github.com/libp2p/go-libp2p-nat v0.0.4 github.com/libp2p/go-libp2p-netutil v0.1.0 - github.com/libp2p/go-libp2p-peerstore v0.1.3 - github.com/libp2p/go-libp2p-secio v0.2.0 + github.com/libp2p/go-libp2p-peerstore v0.1.4 + github.com/libp2p/go-libp2p-secio v0.2.1 github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.0 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 @@ -31,10 +31,11 @@ require ( github.com/libp2p/go-ws-transport v0.1.2 github.com/miekg/dns v1.1.12 // indirect github.com/multiformats/go-multiaddr v0.1.1 - github.com/multiformats/go-multiaddr-dns v0.1.0 - github.com/multiformats/go-multiaddr-net v0.1.0 + github.com/multiformats/go-multiaddr-dns v0.2.0 + github.com/multiformats/go-multiaddr-net v0.1.1 github.com/multiformats/go-multistream v0.1.0 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 + golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 // indirect ) go 1.12 diff --git a/go.sum b/go.sum index 90200587e2..17a54e72e6 100644 --- a/go.sum +++ b/go.sum @@ -31,6 +31,7 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dgraph-io/badger v1.5.5-0.20190226225317-8115aed38f8f/go.mod h1:VZxzAIRPHRVNRKRo6AXrX9BJegn6il06VMTZVJYCIjQ= github.com/dgraph-io/badger v1.6.0-rc1/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= +github.com/dgraph-io/badger v1.6.0/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= @@ -41,6 +42,8 @@ github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE= github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= github.com/gogo/protobuf v1.3.0 h1:G8O7TerXerS4F6sx9OV7/nRfJdnXgHZu/S/7F2SN+UE= github.com/gogo/protobuf v1.3.0/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= +github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= @@ -76,11 +79,15 @@ github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUP github.com/ipfs/go-cid v0.0.3 h1:UIAh32wymBpStoe83YCzwVQQ5Oy/H0FdxvUS6DJDzms= github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= +github.com/ipfs/go-datastore v0.1.0/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= +github.com/ipfs/go-datastore v0.1.1/go.mod h1:w38XXW9kVFNp57Zj5knbKWM2T+KOZCGDRVNdgPHtbHw= github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8= github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaHzfGTzuE3s= +github.com/ipfs/go-ds-badger v0.0.7/go.mod h1:qt0/fWzZDoPW6jpQeqUjR5kBfhDNB65jd9YlmAvpQBk= github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc= +github.com/ipfs/go-ds-leveldb v0.1.0/go.mod h1:hqAW8y4bwX5LWcCtku2rFNX3vjDZCy5LZCg+cSZvYb8= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50= github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc= @@ -125,21 +132,21 @@ github.com/libp2p/go-eventbus v0.1.0 h1:mlawomSAjjkk97QnYiEmHsLu7E136+2oCWSHRUvM github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UGQf8A2eEmx4= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= -github.com/libp2p/go-libp2p-autonat v0.1.0 h1:aCWAu43Ri4nU0ZPO7NyLzUvvfqd0nE3dX0R/ZGYVgOU= -github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3PtvYlJmg7NeVtPRqH8= +github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= +github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= -github.com/libp2p/go-libp2p-circuit v0.1.3 h1:WsMYYaA0PwdpgJSQu12EzPYf5ypkLSTgcOsWr7DYrgI= -github.com/libp2p/go-libp2p-circuit v0.1.3/go.mod h1:Xqh2TjSy8DD5iV2cCOMzdynd6h8OTBGoV1AWbWor3qM= +github.com/libp2p/go-libp2p-circuit v0.1.4 h1:Phzbmrg3BkVzbqd4ZZ149JxCuUWu2wZcXf/Kr6hZJj8= +github.com/libp2p/go-libp2p-circuit v0.1.4/go.mod h1:CY67BrEjKNDhdTk8UgBX1Y/H5c3xkAcs3gnksxY7osU= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= github.com/libp2p/go-libp2p-core v0.2.0 h1:ycFtuNwtZBAJSxzaHbyv6NjG3Yj5Nmra1csHaQ3zwaw= github.com/libp2p/go-libp2p-core v0.2.0/go.mod h1:X0eyB0Gy93v0DZtSYbEM7RnMChm9Uv3j7yRXjO77xSI= github.com/libp2p/go-libp2p-core v0.2.2 h1:Sv1ggdoMx9c7v7FOFkR7agraHCnAgqYsXrU1ARSRUMs= github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZVoVrLaDRvIRng0= -github.com/libp2p/go-libp2p-core v0.2.3 h1:zXikZ5pLfebtTMeIYfcwVQ2Pae77O0FIwDquwM6AGNM= -github.com/libp2p/go-libp2p-core v0.2.3/go.mod h1:GqhyQqyIAPsxFYXHMjfXgMv03lxsvM0mFzuYA9Ib42A= +github.com/libp2p/go-libp2p-core v0.2.4 h1:Et6ykkTwI6PU44tr8qUF9k43vP0aduMNniShAbUJJw8= +github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= @@ -156,10 +163,14 @@ github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMg github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HBb6u0MOvdJ7YIsmcwtY= github.com/libp2p/go-libp2p-peerstore v0.1.3 h1:wMgajt1uM2tMiqf4M+4qWKVyyFc8SfA+84VV9glZq1M= github.com/libp2p/go-libp2p-peerstore v0.1.3/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= +github.com/libp2p/go-libp2p-peerstore v0.1.4 h1:d23fvq5oYMJ/lkkbO4oTwBp/JP+I/1m5gZJobNXCE/k= +github.com/libp2p/go-libp2p-peerstore v0.1.4/go.mod h1:+4BDbDiiKf4PzpANZDAT+knVdLxvqh7hXOujessqdzs= github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLMWdB7GTQ= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0 h1:ywzZBsWEEz2KNTn5RtzauEDq5RFEefPsttXYwAWqHng= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= +github.com/libp2p/go-libp2p-secio v0.2.1 h1:eNWbJTdyPA7NxhP7J3c5lT97DC5d+u+IldkgCYFTPVA= +github.com/libp2p/go-libp2p-secio v0.2.1/go.mod h1:cWtZpILJqkqrSkiYcDBh5lA3wbT2Q+hz3rJQq3iftD8= github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= github.com/libp2p/go-libp2p-swarm v0.2.2 h1:T4hUpgEs2r371PweU3DuH7EOmBIdTBCwWs+FLcgx3bQ= github.com/libp2p/go-libp2p-swarm v0.2.2/go.mod h1:fvmtQ0T1nErXym1/aa1uJEyN7JzaTNyBcHImCxRpPKU= @@ -188,6 +199,8 @@ github.com/libp2p/go-nat v0.0.3 h1:l6fKV+p0Xa354EqQOQP+d8CivdLM4kl5GxC1hSc/UeI= github.com/libp2p/go-nat v0.0.3/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= github.com/libp2p/go-openssl v0.0.2 h1:9pP2d3Ubaxkv7ZisLjx9BFwgOGnQdQYnfcH29HNY3ls= github.com/libp2p/go-openssl v0.0.2/go.mod h1:v8Zw2ijCSWBQi8Pq5GAixw6DbFfa9u6VIYDXnvOXkc0= +github.com/libp2p/go-openssl v0.0.3 h1:wjlG7HvQkt4Fq4cfH33Ivpwp0omaElYEi9z26qaIkIk= +github.com/libp2p/go-openssl v0.0.3/go.mod h1:unDrJpgy3oFr+rqXsarWifmJuNnJR4chtO1HmaZjggc= github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0GIEjMiHTwR/KN4= @@ -244,8 +257,8 @@ github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= -github.com/multiformats/go-multiaddr-dns v0.1.0 h1:gsPeMvo91XvcsNlQXgJgfjYjbsVV99bvveguUvDBpyQ= -github.com/multiformats/go-multiaddr-dns v0.1.0/go.mod h1:01k2RAqtoXIuPa3DCavAE9/6jc6nM0H3EgZyfUhN2oY= +github.com/multiformats/go-multiaddr-dns v0.2.0 h1:YWJoIDwLePniH7OU5hBnDZV6SWuvJqJ0YtN6pLeH9zA= +github.com/multiformats/go-multiaddr-dns v0.2.0/go.mod h1:TJ5pr5bBO7Y1B18djPuRsVkduhQH2YqYSbxWJzYGdK0= github.com/multiformats/go-multiaddr-fmt v0.0.1 h1:5YjeOIzbX8OTKVaN72aOzGIYW7PnrZrnkDyOfAWRSMA= github.com/multiformats/go-multiaddr-fmt v0.0.1/go.mod h1:aBYjqL4T/7j4Qx+R73XSv/8JsgnRFlf0w2KGLCmXl3Q= github.com/multiformats/go-multiaddr-fmt v0.1.0 h1:WLEFClPycPkp4fnIzoFoV9FVd49/eQsuaL3/CWe167E= @@ -254,6 +267,8 @@ github.com/multiformats/go-multiaddr-net v0.0.1 h1:76O59E3FavvHqNg7jvzWzsPSW5JSi github.com/multiformats/go-multiaddr-net v0.0.1/go.mod h1:nw6HSxNmCIQH27XPGBuX+d1tnvM7ihcFwHMSstNAVUU= github.com/multiformats/go-multiaddr-net v0.1.0 h1:ZepO8Ezwovd+7b5XPPDhQhayk1yt0AJpzQBpq9fejx4= github.com/multiformats/go-multiaddr-net v0.1.0/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= +github.com/multiformats/go-multiaddr-net v0.1.1 h1:jFFKUuXTXv+3ARyHZi3XUqQO+YWMKgBdhEvuGRfnL6s= +github.com/multiformats/go-multiaddr-net v0.1.1/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= @@ -329,8 +344,8 @@ golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 h1:IcSOAf4PyMp3U3XbIEj1/xJ2BjNN2jWv7JoyOsMxXUU= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20190923035154-9ee001bba392 h1:ACG4HJsFiNMf47Y4PeRoebLNy/2lXT9EtprMuTFWt1M= -golang.org/x/crypto v0.0.0-20190923035154-9ee001bba392/go.mod h1:/lpIB1dKB+9EgE3H3cr1v9wB50oz8l4C4h62xy7jSTY= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= @@ -377,6 +392,7 @@ golang.org/x/tools v0.0.0-20190226205152-f727befe758c h1:vamGzbGri8IKo20MQncCulj golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= diff --git a/p2p/protocol/identify/pb/identify.pb.go b/p2p/protocol/identify/pb/identify.pb.go index 8480e5ba54..5175425292 100644 --- a/p2p/protocol/identify/pb/identify.pb.go +++ b/p2p/protocol/identify/pb/identify.pb.go @@ -8,6 +8,7 @@ import ( proto "github.com/gogo/protobuf/proto" io "io" math "math" + math_bits "math/bits" ) // Reference imports to suppress errors if they are not otherwise used. @@ -19,7 +20,7 @@ var _ = math.Inf // is compatible with the proto package it is being compiled against. // A compilation error at this line likely means your copy of the // proto package needs to be updated. -const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package type Delta struct { // new protocols now serviced by the peer. @@ -45,7 +46,7 @@ func (m *Delta) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { return xxx_messageInfo_Delta.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -117,7 +118,7 @@ func (m *Identify) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { return xxx_messageInfo_Identify.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -215,7 +216,7 @@ var fileDescriptor_83f1e7e6b485409f = []byte{ func (m *Delta) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -223,50 +224,44 @@ func (m *Delta) Marshal() (dAtA []byte, err error) { } func (m *Delta) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *Delta) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if len(m.AddedProtocols) > 0 { - for _, s := range m.AddedProtocols { - dAtA[i] = 0xa - i++ - l = len(s) - for l >= 1<<7 { - dAtA[i] = uint8(uint64(l)&0x7f | 0x80) - l >>= 7 - i++ - } - dAtA[i] = uint8(l) - i++ - i += copy(dAtA[i:], s) - } + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) } if len(m.RmProtocols) > 0 { - for _, s := range m.RmProtocols { + for iNdEx := len(m.RmProtocols) - 1; iNdEx >= 0; iNdEx-- { + i -= len(m.RmProtocols[iNdEx]) + copy(dAtA[i:], m.RmProtocols[iNdEx]) + i = encodeVarintIdentify(dAtA, i, uint64(len(m.RmProtocols[iNdEx]))) + i-- dAtA[i] = 0x12 - i++ - l = len(s) - for l >= 1<<7 { - dAtA[i] = uint8(uint64(l)&0x7f | 0x80) - l >>= 7 - i++ - } - dAtA[i] = uint8(l) - i++ - i += copy(dAtA[i:], s) } } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) + if len(m.AddedProtocols) > 0 { + for iNdEx := len(m.AddedProtocols) - 1; iNdEx >= 0; iNdEx-- { + i -= len(m.AddedProtocols[iNdEx]) + copy(dAtA[i:], m.AddedProtocols[iNdEx]) + i = encodeVarintIdentify(dAtA, i, uint64(len(m.AddedProtocols[iNdEx]))) + i-- + dAtA[i] = 0xa + } } - return i, nil + return len(dAtA) - i, nil } func (m *Identify) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -274,81 +269,90 @@ func (m *Identify) Marshal() (dAtA []byte, err error) { } func (m *Identify) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *Identify) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.PublicKey != nil { - dAtA[i] = 0xa - i++ - i = encodeVarintIdentify(dAtA, i, uint64(len(m.PublicKey))) - i += copy(dAtA[i:], m.PublicKey) - } - if len(m.ListenAddrs) > 0 { - for _, b := range m.ListenAddrs { - dAtA[i] = 0x12 - i++ - i = encodeVarintIdentify(dAtA, i, uint64(len(b))) - i += copy(dAtA[i:], b) - } + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) } - if len(m.Protocols) > 0 { - for _, s := range m.Protocols { - dAtA[i] = 0x1a - i++ - l = len(s) - for l >= 1<<7 { - dAtA[i] = uint8(uint64(l)&0x7f | 0x80) - l >>= 7 - i++ - } - dAtA[i] = uint8(l) - i++ - i += copy(dAtA[i:], s) + if m.Delta != nil { + { + size, err := m.Delta.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintIdentify(dAtA, i, uint64(size)) } + i-- + dAtA[i] = 0x3a } - if m.ObservedAddr != nil { - dAtA[i] = 0x22 - i++ - i = encodeVarintIdentify(dAtA, i, uint64(len(m.ObservedAddr))) - i += copy(dAtA[i:], m.ObservedAddr) + if m.AgentVersion != nil { + i -= len(*m.AgentVersion) + copy(dAtA[i:], *m.AgentVersion) + i = encodeVarintIdentify(dAtA, i, uint64(len(*m.AgentVersion))) + i-- + dAtA[i] = 0x32 } if m.ProtocolVersion != nil { - dAtA[i] = 0x2a - i++ + i -= len(*m.ProtocolVersion) + copy(dAtA[i:], *m.ProtocolVersion) i = encodeVarintIdentify(dAtA, i, uint64(len(*m.ProtocolVersion))) - i += copy(dAtA[i:], *m.ProtocolVersion) + i-- + dAtA[i] = 0x2a } - if m.AgentVersion != nil { - dAtA[i] = 0x32 - i++ - i = encodeVarintIdentify(dAtA, i, uint64(len(*m.AgentVersion))) - i += copy(dAtA[i:], *m.AgentVersion) + if m.ObservedAddr != nil { + i -= len(m.ObservedAddr) + copy(dAtA[i:], m.ObservedAddr) + i = encodeVarintIdentify(dAtA, i, uint64(len(m.ObservedAddr))) + i-- + dAtA[i] = 0x22 } - if m.Delta != nil { - dAtA[i] = 0x3a - i++ - i = encodeVarintIdentify(dAtA, i, uint64(m.Delta.Size())) - n1, err := m.Delta.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err + if len(m.Protocols) > 0 { + for iNdEx := len(m.Protocols) - 1; iNdEx >= 0; iNdEx-- { + i -= len(m.Protocols[iNdEx]) + copy(dAtA[i:], m.Protocols[iNdEx]) + i = encodeVarintIdentify(dAtA, i, uint64(len(m.Protocols[iNdEx]))) + i-- + dAtA[i] = 0x1a } - i += n1 } - if m.XXX_unrecognized != nil { - i += copy(dAtA[i:], m.XXX_unrecognized) + if len(m.ListenAddrs) > 0 { + for iNdEx := len(m.ListenAddrs) - 1; iNdEx >= 0; iNdEx-- { + i -= len(m.ListenAddrs[iNdEx]) + copy(dAtA[i:], m.ListenAddrs[iNdEx]) + i = encodeVarintIdentify(dAtA, i, uint64(len(m.ListenAddrs[iNdEx]))) + i-- + dAtA[i] = 0x12 + } } - return i, nil + if m.PublicKey != nil { + i -= len(m.PublicKey) + copy(dAtA[i:], m.PublicKey) + i = encodeVarintIdentify(dAtA, i, uint64(len(m.PublicKey))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil } func encodeVarintIdentify(dAtA []byte, offset int, v uint64) int { + offset -= sovIdentify(v) + base := offset for v >= 1<<7 { dAtA[offset] = uint8(v&0x7f | 0x80) v >>= 7 offset++ } dAtA[offset] = uint8(v) - return offset + 1 + return base } func (m *Delta) Size() (n int) { if m == nil { @@ -419,14 +423,7 @@ func (m *Identify) Size() (n int) { } func sovIdentify(x uint64) (n int) { - for { - n++ - x >>= 7 - if x == 0 { - break - } - } - return n + return (math_bits.Len64(x|1) + 6) / 7 } func sozIdentify(x uint64) (n int) { return sovIdentify(uint64((x << 1) ^ uint64((int64(x) >> 63)))) @@ -840,6 +837,7 @@ func (m *Identify) Unmarshal(dAtA []byte) error { func skipIdentify(dAtA []byte) (n int, err error) { l := len(dAtA) iNdEx := 0 + depth := 0 for iNdEx < l { var wire uint64 for shift := uint(0); ; shift += 7 { @@ -871,10 +869,8 @@ func skipIdentify(dAtA []byte) (n int, err error) { break } } - return iNdEx, nil case 1: iNdEx += 8 - return iNdEx, nil case 2: var length int for shift := uint(0); ; shift += 7 { @@ -898,52 +894,27 @@ func skipIdentify(dAtA []byte) (n int, err error) { if iNdEx < 0 { return 0, ErrInvalidLengthIdentify } - return iNdEx, nil case 3: - for { - var innerWire uint64 - var start int = iNdEx - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowIdentify - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - innerWire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - innerWireType := int(innerWire & 0x7) - if innerWireType == 4 { - break - } - next, err := skipIdentify(dAtA[start:]) - if err != nil { - return 0, err - } - iNdEx = start + next - if iNdEx < 0 { - return 0, ErrInvalidLengthIdentify - } - } - return iNdEx, nil + depth++ case 4: - return iNdEx, nil + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupIdentify + } + depth-- case 5: iNdEx += 4 - return iNdEx, nil default: return 0, fmt.Errorf("proto: illegal wireType %d", wireType) } + if depth == 0 { + return iNdEx, nil + } } - panic("unreachable") + return 0, io.ErrUnexpectedEOF } var ( - ErrInvalidLengthIdentify = fmt.Errorf("proto: negative length found during unmarshaling") - ErrIntOverflowIdentify = fmt.Errorf("proto: integer overflow") + ErrInvalidLengthIdentify = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowIdentify = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupIdentify = fmt.Errorf("proto: unexpected end of group") ) From c18bd50d82e835a7e8cf0d9e9b9c242ffe516441 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Tue, 29 Oct 2019 11:38:35 +0000 Subject: [PATCH 1556/3965] Allow custom SubtestStress. (#7) SubtestStress is exposed but it was not possible to build a working opt parameters. --- p2p/transport/testsuite/stream_suite.go | 92 ++++++++++++------------- 1 file changed, 46 insertions(+), 46 deletions(-) diff --git a/p2p/transport/testsuite/stream_suite.go b/p2p/transport/testsuite/stream_suite.go index 6f978e57be..e04deaeb62 100644 --- a/p2p/transport/testsuite/stream_suite.go +++ b/p2p/transport/testsuite/stream_suite.go @@ -47,11 +47,11 @@ func init() { } type Options struct { - connNum int - streamNum int - msgNum int - msgMin int - msgMax int + ConnNum int + StreamNum int + MsgNum int + MsgMin int + MsgMax int } func fullClose(t *testing.T, s mux.MuxedStream) { @@ -170,12 +170,12 @@ func SubtestStress(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, } writeStream := func(s mux.MuxedStream, bufs chan<- []byte) { - debugLog(t, "writeStream %p, %d msgNum", s, opt.msgNum) + debugLog(t, "writeStream %p, %d MsgNum", s, opt.MsgNum) - for i := 0; i < opt.msgNum; i++ { + for i := 0; i < opt.MsgNum; i++ { buf := randBuf(msgsize) bufs <- buf - debugLog(t, "%p writing %d bytes (message %d/%d #%x)", s, len(buf), i, opt.msgNum, buf[:3]) + debugLog(t, "%p writing %d bytes (message %d/%d #%x)", s, len(buf), i, opt.MsgNum, buf[:3]) if _, err := s.Write(buf); err != nil { errs <- fmt.Errorf("s.Write(buf): %s", err) continue @@ -184,17 +184,17 @@ func SubtestStress(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, } readStream := func(s mux.MuxedStream, bufs <-chan []byte) { - debugLog(t, "readStream %p, %d msgNum", s, opt.msgNum) + debugLog(t, "readStream %p, %d MsgNum", s, opt.MsgNum) buf2 := make([]byte, msgsize) i := 0 for buf1 := range bufs { i++ - debugLog(t, "%p reading %d bytes (message %d/%d #%x)", s, len(buf1), i-1, opt.msgNum, buf1[:3]) + debugLog(t, "%p reading %d bytes (message %d/%d #%x)", s, len(buf1), i-1, opt.MsgNum, buf1[:3]) if _, err := io.ReadFull(s, buf2); err != nil { errs <- fmt.Errorf("io.ReadFull(s, buf2): %s", err) - debugLog(t, "%p failed to read %d bytes (message %d/%d #%x)", s, len(buf1), i-1, opt.msgNum, buf1[:3]) + debugLog(t, "%p failed to read %d bytes (message %d/%d #%x)", s, len(buf1), i-1, opt.MsgNum, buf1[:3]) continue } if !bytes.Equal(buf1, buf2) { @@ -204,7 +204,7 @@ func SubtestStress(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, } openStreamAndRW := func(c mux.MuxedConn) { - debugLog(t, "openStreamAndRW %p, %d opt.msgNum", c, opt.msgNum) + debugLog(t, "openStreamAndRW %p, %d opt.MsgNum", c, opt.MsgNum) s, err := c.OpenStream() if err != nil { @@ -212,7 +212,7 @@ func SubtestStress(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, return } - bufs := make(chan []byte, opt.msgNum) + bufs := make(chan []byte, opt.MsgNum) go func() { writeStream(s, bufs) close(bufs) @@ -248,7 +248,7 @@ func SubtestStress(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, }() var wg sync.WaitGroup - for i := 0; i < opt.streamNum; i++ { + for i := 0; i < opt.StreamNum; i++ { wg.Add(1) go rateLimit(func() { defer wg.Done() @@ -260,10 +260,10 @@ func SubtestStress(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, } openConnsAndRW := func() { - debugLog(t, "openConnsAndRW, %d conns", opt.connNum) + debugLog(t, "openConnsAndRW, %d conns", opt.ConnNum) var wg sync.WaitGroup - for i := 0; i < opt.connNum; i++ { + for i := 0; i < opt.ConnNum; i++ { wg.Add(1) go rateLimit(func() { defer wg.Done() @@ -420,60 +420,60 @@ func SubtestStreamReset(t *testing.T, ta, tb transport.Transport, maddr ma.Multi func SubtestStress1Conn1Stream1Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { SubtestStress(t, ta, tb, maddr, peerA, Options{ - connNum: 1, - streamNum: 1, - msgNum: 1, - msgMax: 100, - msgMin: 100, + ConnNum: 1, + StreamNum: 1, + MsgNum: 1, + MsgMax: 100, + MsgMin: 100, }) } func SubtestStress1Conn1Stream100Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { SubtestStress(t, ta, tb, maddr, peerA, Options{ - connNum: 1, - streamNum: 1, - msgNum: 100, - msgMax: 100, - msgMin: 100, + ConnNum: 1, + StreamNum: 1, + MsgNum: 100, + MsgMax: 100, + MsgMin: 100, }) } func SubtestStress1Conn100Stream100Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { SubtestStress(t, ta, tb, maddr, peerA, Options{ - connNum: 1, - streamNum: 100, - msgNum: 100, - msgMax: 100, - msgMin: 100, + ConnNum: 1, + StreamNum: 100, + MsgNum: 100, + MsgMax: 100, + MsgMin: 100, }) } func SubtestStress50Conn10Stream50Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { SubtestStress(t, ta, tb, maddr, peerA, Options{ - connNum: 50, - streamNum: 10, - msgNum: 50, - msgMax: 100, - msgMin: 100, + ConnNum: 50, + StreamNum: 10, + MsgNum: 50, + MsgMax: 100, + MsgMin: 100, }) } func SubtestStress1Conn1000Stream10Msg(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { SubtestStress(t, ta, tb, maddr, peerA, Options{ - connNum: 1, - streamNum: 1000, - msgNum: 10, - msgMax: 100, - msgMin: 100, + ConnNum: 1, + StreamNum: 1000, + MsgNum: 10, + MsgMax: 100, + MsgMin: 100, }) } func SubtestStress1Conn100Stream100Msg10MB(t *testing.T, ta, tb transport.Transport, maddr ma.Multiaddr, peerA peer.ID) { SubtestStress(t, ta, tb, maddr, peerA, Options{ - connNum: 1, - streamNum: 100, - msgNum: 100, - msgMax: 10000, - msgMin: 1000, + ConnNum: 1, + StreamNum: 100, + MsgNum: 100, + MsgMax: 10000, + MsgMin: 1000, }) } From fa543cf383ba16f187d1cb105ec105a73959d71c Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Tue, 29 Oct 2019 10:51:18 -0400 Subject: [PATCH 1557/3965] cleanup some function signatures --- p2p/discovery/backoff/backoff.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/discovery/backoff/backoff.go b/p2p/discovery/backoff/backoff.go index 944d952b4f..e31b7a4b95 100644 --- a/p2p/discovery/backoff/backoff.go +++ b/p2p/discovery/backoff/backoff.go @@ -19,11 +19,11 @@ type BackoffStrategy interface { // Jitter implementations taken roughly from https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ // Jitter must return a duration between min and max. Min must be lower than, or equal to, max. -type Jitter func(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration +type Jitter func(duration time.Duration, min, max time.Duration, rng *rand.Rand) time.Duration // FullJitter returns a random number uniformly chose from the range [min, boundedDur]. // boundedDur is the duration bounded between min and max. -func FullJitter(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration { +func FullJitter(duration time.Duration, min, max time.Duration, rng *rand.Rand) time.Duration { if duration <= min { return min } @@ -34,7 +34,7 @@ func FullJitter(duration time.Duration, min time.Duration, max time.Duration, rn } // NoJitter returns the duration bounded between min and max -func NoJitter(duration time.Duration, min time.Duration, max time.Duration, rng *rand.Rand) time.Duration { +func NoJitter(duration time.Duration, min, max time.Duration, rng *rand.Rand) time.Duration { return boundedDuration(duration, min, max) } @@ -48,7 +48,7 @@ func (b *randomizedBackoff) BoundedDelay(duration time.Duration) time.Duration { return boundedDuration(duration, b.min, b.max) } -func boundedDuration(d time.Duration, min time.Duration, max time.Duration) time.Duration { +func boundedDuration(d time.Duration, min, max time.Duration) time.Duration { if d < min { return min } From c635b3327e885003b6029acbf3aa2c65b8152965 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Tue, 29 Oct 2019 11:19:24 -0400 Subject: [PATCH 1558/3965] cleanup backoffcache event dispatching logic --- p2p/discovery/backoff/backoffcache.go | 31 ++++++++++++--------------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/p2p/discovery/backoff/backoffcache.go b/p2p/discovery/backoff/backoffcache.go index 620012eced..93b3db71fb 100644 --- a/p2p/discovery/backoff/backoffcache.go +++ b/p2p/discovery/backoff/backoffcache.go @@ -126,24 +126,10 @@ func (d *BackoffDiscovery) FindPeers(ctx context.Context, ns string, opts ...dis c.mux.Lock() defer c.mux.Unlock() - findPeers := !ok - timeExpired := false - if !findPeers { - timeExpired = time.Now().After(c.nextDiscover) - findPeers = timeExpired && !c.ongoing - } + timeExpired := time.Now().After(c.nextDiscover) - // If we should find peers then setup a dispatcher channel for dispatching incoming peers - if findPeers { - pch, err := d.disc.FindPeers(ctx, ns, opts...) - if err != nil { - return nil, err - } - - c.ongoing = true - go findPeerDispatcher(ctx, c, pch) - // If it's not yet time to search again then return cached peers - } else if !timeExpired { + // If it's not yet time to search again and no searches are in progress then return cached peers + if !(timeExpired || c.ongoing){ chLen := options.Limit if chLen == 0 { @@ -159,6 +145,17 @@ func (d *BackoffDiscovery) FindPeers(ctx context.Context, ns string, opts ...dis return pch, nil } + // If a request is not already in progress setup a dispatcher channel for dispatching incoming peers + if !c.ongoing { + pch, err := d.disc.FindPeers(ctx, ns, opts...) + if err != nil { + return nil, err + } + + c.ongoing = true + go findPeerDispatcher(ctx, c, pch) + } + // Setup receiver channel for receiving peers from ongoing requests evtCh := make(chan peer.AddrInfo, d.parallelBufSz) pch := make(chan peer.AddrInfo, d.returnedBufSz) From d4ccc21ee01bacca143dbc44a0a006f36146041a Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Tue, 29 Oct 2019 17:18:06 -0400 Subject: [PATCH 1559/3965] more documentation and function argument coalescing --- p2p/discovery/backoff/backoff.go | 8 ++++---- p2p/discovery/backoff/backoffcache.go | 2 +- p2p/discovery/backoff/backoffconnector.go | 4 +++- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/p2p/discovery/backoff/backoff.go b/p2p/discovery/backoff/backoff.go index e31b7a4b95..ecd9470017 100644 --- a/p2p/discovery/backoff/backoff.go +++ b/p2p/discovery/backoff/backoff.go @@ -19,11 +19,11 @@ type BackoffStrategy interface { // Jitter implementations taken roughly from https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ // Jitter must return a duration between min and max. Min must be lower than, or equal to, max. -type Jitter func(duration time.Duration, min, max time.Duration, rng *rand.Rand) time.Duration +type Jitter func(duration, min, max time.Duration, rng *rand.Rand) time.Duration // FullJitter returns a random number uniformly chose from the range [min, boundedDur]. // boundedDur is the duration bounded between min and max. -func FullJitter(duration time.Duration, min, max time.Duration, rng *rand.Rand) time.Duration { +func FullJitter(duration, min, max time.Duration, rng *rand.Rand) time.Duration { if duration <= min { return min } @@ -34,7 +34,7 @@ func FullJitter(duration time.Duration, min, max time.Duration, rng *rand.Rand) } // NoJitter returns the duration bounded between min and max -func NoJitter(duration time.Duration, min, max time.Duration, rng *rand.Rand) time.Duration { +func NoJitter(duration, min, max time.Duration, rng *rand.Rand) time.Duration { return boundedDuration(duration, min, max) } @@ -48,7 +48,7 @@ func (b *randomizedBackoff) BoundedDelay(duration time.Duration) time.Duration { return boundedDuration(duration, b.min, b.max) } -func boundedDuration(d time.Duration, min, max time.Duration) time.Duration { +func boundedDuration(d, min, max time.Duration) time.Duration { if d < min { return min } diff --git a/p2p/discovery/backoff/backoffcache.go b/p2p/discovery/backoff/backoffcache.go index 93b3db71fb..0e255e23db 100644 --- a/p2p/discovery/backoff/backoffcache.go +++ b/p2p/discovery/backoff/backoffcache.go @@ -129,7 +129,7 @@ func (d *BackoffDiscovery) FindPeers(ctx context.Context, ns string, opts ...dis timeExpired := time.Now().After(c.nextDiscover) // If it's not yet time to search again and no searches are in progress then return cached peers - if !(timeExpired || c.ongoing){ + if !(timeExpired || c.ongoing) { chLen := options.Limit if chLen == 0 { diff --git a/p2p/discovery/backoff/backoffconnector.go b/p2p/discovery/backoff/backoffconnector.go index 8fe3c2de6a..6f6c58f637 100644 --- a/p2p/discovery/backoff/backoffconnector.go +++ b/p2p/discovery/backoff/backoffconnector.go @@ -42,7 +42,9 @@ type connCacheData struct { strat BackoffStrategy } -// Connect attemps to connect to the peers passed in by peerCh. Will not connect to peers if they are within the backoff period. +// Connect attempts to connect to the peers passed in by peerCh. Will not connect to peers if they are within the backoff period. +// As Connect will attempt to dial peers as soon as it learns about them, the caller should try to keep the number, +// and rate, of inbound peers manageable. func (c *BackoffConnector) Connect(ctx context.Context, peerCh <-chan peer.AddrInfo) { for { select { From d7abb917c6f90225778cf59ea9af5d3db88f5ae6 Mon Sep 17 00:00:00 2001 From: Roman Proskuryakov Date: Fri, 1 Nov 2019 04:25:27 +0300 Subject: [PATCH 1560/3965] Add API to reset bandwidth counters --- core/metrics/bandwidth.go | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/core/metrics/bandwidth.go b/core/metrics/bandwidth.go index e2c8acf30d..aa758e8eaa 100644 --- a/core/metrics/bandwidth.go +++ b/core/metrics/bandwidth.go @@ -151,3 +151,15 @@ func (bwc *BandwidthCounter) GetBandwidthByProtocol() map[protocol.ID]Stats { return protocols } + +// Reset clears all stats. +func (bwc *BandwidthCounter) Reset() { + bwc.totalIn.Reset() + bwc.totalOut.Reset() + + bwc.protocolIn.Clear() + bwc.protocolOut.Clear() + + bwc.peerIn.Clear() + bwc.peerOut.Clear() +} From 611f243e23616042004d0f7e9d663e26aec9b721 Mon Sep 17 00:00:00 2001 From: Roman Proskuryakov Date: Fri, 1 Nov 2019 04:46:48 +0300 Subject: [PATCH 1561/3965] Add a test for BandwidthCounter::Reset --- core/metrics/bandwidth_test.go | 57 ++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/core/metrics/bandwidth_test.go b/core/metrics/bandwidth_test.go index de7b0fdcfd..49309b074d 100644 --- a/core/metrics/bandwidth_test.go +++ b/core/metrics/bandwidth_test.go @@ -143,6 +143,63 @@ func TestBandwidthCounter(t *testing.T) { } } +func TestResetBandwidthCounter(t *testing.T) { + bwc := NewBandwidthCounter() + + p := peer.ID("peer-0") + proto := protocol.ID("proto-0") + + bwc.LogSentMessage(42) + bwc.LogRecvMessage(24) + bwc.LogSentMessageStream(100, proto, p) + bwc.LogRecvMessageStream(50, proto, p) + + time.Sleep(1 * time.Second) + + { + stats := bwc.GetBandwidthTotals() + assertEq(t, 42, stats.TotalOut) + assertEq(t, 24, stats.TotalIn) + } + + { + stats := bwc.GetBandwidthByProtocol() + assertApproxEq(t, 1, float64(len(stats))) + stat := stats[proto] + assertApproxEq(t, 100, stat.RateOut) + assertApproxEq(t, 50, stat.RateIn) + } + + { + stats := bwc.GetBandwidthByPeer() + assertApproxEq(t, 1, float64(len(stats))) + stat := stats[p] + assertApproxEq(t, 100, stat.RateOut) + assertApproxEq(t, 50, stat.RateIn) + } + + bwc.Reset() + { + stats := bwc.GetBandwidthTotals() + assertEq(t, 0, stats.TotalOut) + assertEq(t, 0, stats.TotalIn) + } + + { + byProtocol := bwc.GetBandwidthByProtocol() + if len(byProtocol) != 0 { + t.Errorf("expected 0 protocols, got %d", len(byProtocol)) + } + } + + { + byPeer := bwc.GetBandwidthByPeer() + if len(byPeer) != 0 { + t.Errorf("expected 0 peers, got %d", len(byPeer)) + } + } +} + func assertEq(t *testing.T, expected, actual int64) { if expected != actual { t.Errorf("expected %d, got %d", expected, actual) From 6e18148d8ed3dd70b74d502cd5bb7282d24f1495 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 31 Oct 2019 23:47:34 -0700 Subject: [PATCH 1562/3965] fix(metrics): update go-flow-metrics and fix tests for upstream changes --- core/metrics/bandwidth_test.go | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/core/metrics/bandwidth_test.go b/core/metrics/bandwidth_test.go index 49309b074d..a3232e1527 100644 --- a/core/metrics/bandwidth_test.go +++ b/core/metrics/bandwidth_test.go @@ -48,8 +48,8 @@ func round(bwc *BandwidthCounter, b *testing.B) { b.StopTimer() } -// Allow 7% errors for bw calculations. -const acceptableError = 0.07 +// Allow 1% errors for bw calculations. +const acceptableError = 0.01 func TestBandwidthCounter(t *testing.T) { bwc := NewBandwidthCounter() @@ -62,12 +62,19 @@ func TestBandwidthCounter(t *testing.T) { proto := protocol.ID(fmt.Sprintf("proto-%d", j)) go func() { defer wg.Done() + + // make sure the bandwidth counters are active + bwc.LogSentMessage(100) + bwc.LogRecvMessage(50) + bwc.LogSentMessageStream(100, proto, p) + bwc.LogRecvMessageStream(50, proto, p) + <-start t := time.NewTicker(100 * time.Millisecond) defer t.Stop() - for i := 0; i < 40; i++ { + for i := 0; i < 39; i++ { bwc.LogSentMessage(100) bwc.LogRecvMessage(50) bwc.LogSentMessageStream(100, proto, p) @@ -104,6 +111,7 @@ func TestBandwidthCounter(t *testing.T) { } } + time.Sleep(time.Second) close(start) time.Sleep(2*time.Second + 100*time.Millisecond) @@ -149,17 +157,25 @@ func TestResetBandwidthCounter(t *testing.T) { p := peer.ID("peer-0") proto := protocol.ID("proto-0") + // We don't calculate bandwidth till we've been active for a second. bwc.LogSentMessage(42) bwc.LogRecvMessage(24) bwc.LogSentMessageStream(100, proto, p) bwc.LogRecvMessageStream(50, proto, p) - time.Sleep(1 * time.Second) + time.Sleep(1*time.Second + time.Millisecond) + + bwc.LogSentMessage(42) + bwc.LogRecvMessage(24) + bwc.LogSentMessageStream(100, proto, p) + bwc.LogRecvMessageStream(50, proto, p) + + time.Sleep(1*time.Second + time.Millisecond) { stats := bwc.GetBandwidthTotals() - assertEq(t, 42, stats.TotalOut) - assertEq(t, 24, stats.TotalIn) + assertEq(t, 84, stats.TotalOut) + assertEq(t, 48, stats.TotalIn) } { From 0328485c9f9d652c560c8500f3a5e09676d38918 Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Fri, 1 Nov 2019 21:52:28 +0100 Subject: [PATCH 1563/3965] Fix: Connection Closed after handshake The context-cancelled watchdog goroutine may start running way after the handshake has finished and the associated context has been cancelled (by the executeDial() function in go-libp2p-swarm usuaully). This results in the connection being closed right after being stablished. --- p2p/security/tls/transport.go | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 2dc4257a3e..e081becbf0 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -6,6 +6,7 @@ import ( "errors" "net" "os" + "sync" ci "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" @@ -81,9 +82,19 @@ func (t *Transport) handshake( tlsConn.Close() default: } + done := make(chan struct{}) + var wg sync.WaitGroup + + // Ensure that we do not return before + // either being done or having a context + // cancellation. + defer wg.Wait() defer close(done) + + wg.Add(1) go func() { + defer wg.Done() select { case <-done: case <-ctx.Done(): From ad8e59543bb49b25d155822c1ed80e2dab280edd Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 4 Nov 2019 19:43:46 +0000 Subject: [PATCH 1564/3965] feat(dial): implement the Timeout interface on dial errors --- p2p/net/swarm/dial_error.go | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/p2p/net/swarm/dial_error.go b/p2p/net/swarm/dial_error.go index d21796172f..f2986348bf 100644 --- a/p2p/net/swarm/dial_error.go +++ b/p2p/net/swarm/dial_error.go @@ -2,6 +2,7 @@ package swarm import ( "fmt" + "os" "strings" "github.com/libp2p/go-libp2p-core/peer" @@ -20,6 +21,10 @@ type DialError struct { Skipped int } +func (e *DialError) Timeout() bool { + return os.IsTimeout(e.Cause) +} + func (e *DialError) recordErr(addr ma.Multiaddr, err error) { if len(e.DialErrors) >= maxDialDialErrors { e.Skipped++ @@ -48,11 +53,7 @@ func (e *DialError) Error() string { // Unwrap implements https://godoc.org/golang.org/x/xerrors#Wrapper. func (e *DialError) Unwrap() error { - // If we have a context error, that's the "ultimate" error. - if e.Cause != nil { - return e.Cause - } - return nil + return e.Cause } var _ error = (*DialError)(nil) From dbf68e09d2f4cb73fda6fe21d28430b65f1fc1fe Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 4 Nov 2019 19:45:09 +0000 Subject: [PATCH 1565/3965] feat(swarm): return unwrapped context deadline errors --- p2p/net/swarm/swarm_dial.go | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 054902e7f7..ffb7ab5e2c 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -317,13 +317,14 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { connC, dialErr := s.dialAddrs(ctx, p, goodAddrsChan) if dialErr != nil { logdial["error"] = dialErr.Cause.Error() - if dialErr.Cause == context.Canceled { - // always prefer the "context canceled" error. - // we rely on behing able to check `err == context.Canceled` + switch dialErr.Cause { + case context.Canceled, context.DeadlineExceeded: + // Always prefer the context errors as we rely on being + // able to check them. // // Removing this will BREAK backoff (causing us to // backoff when canceling dials). - return nil, context.Canceled + return nil, dialErr.Cause } return nil, dialErr } From 1263fb7faa4047aa6a8b9f16df38e69346a1e3e2 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 4 Nov 2019 20:02:44 +0000 Subject: [PATCH 1566/3965] feat(swarm): improve dial context 1. Always return the caller's context error if relevant. 2. Don't return "context canceled" when we're just shutting down. 3. Don't claim that the context deadline has been exceeded when the dial timeout is canceled. --- p2p/net/swarm/dial_sync.go | 14 ++++++++++++++ p2p/net/swarm/swarm.go | 3 +++ p2p/net/swarm/swarm_dial.go | 17 ++++++++++++++--- 3 files changed, 31 insertions(+), 3 deletions(-) diff --git a/p2p/net/swarm/dial_sync.go b/p2p/net/swarm/dial_sync.go index f746b9a9a3..4c1230f5fc 100644 --- a/p2p/net/swarm/dial_sync.go +++ b/p2p/net/swarm/dial_sync.go @@ -2,11 +2,15 @@ package swarm import ( "context" + "errors" "sync" "github.com/libp2p/go-libp2p-core/peer" ) +// TODO: change this text when we fix the bug +var errDialCanceled = errors.New("dial was aborted internally, likely due to https://git.io/Je2wW") + // DialFunc is the type of function expected by DialSync. type DialFunc func(context.Context, peer.ID) (*Conn, error) @@ -78,6 +82,16 @@ func (ad *activeDial) decref() { func (ad *activeDial) start(ctx context.Context) { ad.conn, ad.err = ad.ds.dialFunc(ctx, ad.id) + + // This isn't the user's context so we should fix the error. + switch ad.err { + case context.Canceled: + // The dial was canceled with `CancelDial`. + ad.err = errDialCanceled + case context.DeadlineExceeded: + // We hit an internal timeout, not a context timeout. + ad.err = ErrDialTimeout + } close(ad.waitch) ad.cancel() } diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index 5366bdcc6f..f7c57717fa 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -40,6 +40,9 @@ var ErrSwarmClosed = errors.New("swarm closed") // transport is misbehaving. var ErrAddrFiltered = errors.New("address filtered") +// ErrDialTimeout is returned when one a dial times out due to the global timeout +var ErrDialTimeout = errors.New("dial timed out") + // Swarm is a connection muxer, allowing connections to other peers to // be opened and closed, while still using the same Chan for all // communication. The Chan sends/receives Messages, which note the diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index ffb7ab5e2c..475a7e250e 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -221,12 +221,23 @@ func (s *Swarm) dialPeer(ctx context.Context, p peer.ID) (*Conn, error) { defer cancel() conn, err = s.dsync.DialLock(ctx, p) - if err != nil { - return nil, err + if err == nil { + return conn, nil } log.Debugf("network for %s finished dialing %s", s.local, p) - return conn, err + + if ctx.Err() != nil { + // Context error trumps any dial errors as it was likely the ultimate cause. + return nil, ctx.Err() + } + + if s.ctx.Err() != nil { + // Ok, so the swarm is shutting down. + return nil, ErrSwarmClosed + } + + return nil, err } // doDial is an ugly shim method to retain all the logging and backoff logic From ace277ee7c4353569d6574aa3439ec5e9c23be3b Mon Sep 17 00:00:00 2001 From: noot Date: Mon, 4 Nov 2019 15:58:37 -0500 Subject: [PATCH 1567/3965] change xx name string from blake2s to sha256 --- p2p/security/noise/xx/XX.noise.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index bfd743ee23..9bf37b6daa 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -411,7 +411,7 @@ func initializeInitiator(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) var ss symmetricstate var e Keypair var re [32]byte - name := []byte("Noise_XX_25519_ChaChaPoly_BLAKE2s") + name := []byte("Noise_XX_25519_ChaChaPoly_SHA256") ss = initializeSymmetric(name) mixHash(&ss, prologue) return handshakestate{ss, s, e, rs, re, psk} @@ -421,7 +421,7 @@ func initializeResponder(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) var ss symmetricstate var e Keypair var re [32]byte - name := []byte("Noise_XX_25519_ChaChaPoly_BLAKE2s") + name := []byte("Noise_XX_25519_ChaChaPoly_SHA256") ss = initializeSymmetric(name) mixHash(&ss, prologue) return handshakestate{ss, s, e, rs, re, psk} From 7e49c66219cb56bc72aa7658b1607aadaedb04dc Mon Sep 17 00:00:00 2001 From: Hector Sanjuan Date: Tue, 5 Nov 2019 12:03:33 +0100 Subject: [PATCH 1568/3965] Identify: Make activation threshold configurable Also: fix the comments and try to explain how the activation/aliveness of addresses happen (and write the correct defaults). --- p2p/protocol/identify/obsaddr.go | 35 ++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/p2p/protocol/identify/obsaddr.go b/p2p/protocol/identify/obsaddr.go index 12e22860dc..d9e3318dd7 100644 --- a/p2p/protocol/identify/obsaddr.go +++ b/p2p/protocol/identify/obsaddr.go @@ -11,10 +11,23 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -const ActivationThresh = 4 - +// ActivationThresh sets how many times an address must be seen as "activated" +// and therefore advertised to other peers as an address that the local peer +// can be contacted on. The "seen" events expire by default after 40 minutes +// (OwnObservedAddressTTL * ActivationThreshold). The are cleaned up during +// the GC rounds set by GCInterval. +var ActivationThresh = 4 + +// GCInterval specicies how often to make a round cleaning seen events and +// observed addresses. An address will be cleaned if it has not been seen in +// OwnObservedAddressTTL (10 minutes). A "seen" event will be cleaned up if +// it is older than OwnObservedAddressTTL * ActivationThresh (40 minutes). var GCInterval = 10 * time.Minute +// observedAddrSetWorkerChannelSize defines how many addresses can be enqueued +// for adding to an ObservedAddrSet. +var observedAddrSetWorkerChannelSize = 16 + type observation struct { seenTime time.Time connDirection network.Direction @@ -22,8 +35,8 @@ type observation struct { // ObservedAddr is an entry for an address reported by our peers. // We only use addresses that: -// - have been observed at least 4 times in last 1h. (counter symmetric nats) -// - have been observed at least once recently (1h), because our position in the +// - have been observed at least 4 times in last 40 minutes. (counter symmetric nats) +// - have been observed at least once recently (10 minutes), because our position in the // network, or network port mapppings, may have changed. type ObservedAddr struct { Addr ma.Multiaddr @@ -32,8 +45,9 @@ type ObservedAddr struct { } func (oa *ObservedAddr) activated(ttl time.Duration) bool { - // We only activate if in the TTL other peers observed the same address - // of ours at least 4 times. + // We only activate if other peers observed the same address + // of ours at least 4 times. SeenBy peers are removed by GC if + // they say the address more than ttl*ActivationThresh return len(oa.SeenBy) >= ActivationThresh } @@ -55,11 +69,13 @@ type ObservedAddrSet struct { wch chan newObservation } +// NewObservedAddrSet returns a new set using peerstore.OwnObservedAddressTTL +// as the TTL. func NewObservedAddrSet(ctx context.Context) *ObservedAddrSet { oas := &ObservedAddrSet{ addrs: make(map[string][]*ObservedAddr), ttl: peerstore.OwnObservedAddrTTL, - wch: make(chan newObservation, 16), + wch: make(chan newObservation, observedAddrSetWorkerChannelSize), } go oas.worker(ctx) return oas @@ -111,6 +127,7 @@ func (oas *ObservedAddrSet) Addrs() (addrs []ma.Multiaddr) { return addrs } +// Add attemps to queue a new observed address to be added to the set. func (oas *ObservedAddrSet) Add(observed, local, observer ma.Multiaddr, direction network.Direction) { select { @@ -150,7 +167,7 @@ func (oas *ObservedAddrSet) gc() { for _, a := range observedAddrs { // clean up SeenBy set for k, ob := range a.SeenBy { - if now.Sub(ob.seenTime) > oas.ttl*ActivationThresh { + if now.Sub(ob.seenTime) > oas.ttl*time.Duration(ActivationThresh) { delete(a.SeenBy, k) } } @@ -217,12 +234,14 @@ func observerGroup(m ma.Multiaddr) string { return string(first.Bytes()) } +// SetTTL sets the TTL of an observed address-set. func (oas *ObservedAddrSet) SetTTL(ttl time.Duration) { oas.Lock() defer oas.Unlock() oas.ttl = ttl } +// TTL gets the TTL of an observed address-set. func (oas *ObservedAddrSet) TTL() time.Duration { oas.RLock() defer oas.RUnlock() From 34319768aae01bba0627c1ad65755448db4cb8c3 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Tue, 12 Nov 2019 11:52:55 -0500 Subject: [PATCH 1569/3965] use varints for delimiting plaintext 2.0 msgs (#74) * use varints for delimiting plaintext 2.0 msgs * lower size limit, fix comment * go mod tidy - rm unused msgio dependency --- core/sec/insecure/insecure.go | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/core/sec/insecure/insecure.go b/core/sec/insecure/insecure.go index e3670c222a..259b4f2879 100644 --- a/core/sec/insecure/insecure.go +++ b/core/sec/insecure/insecure.go @@ -6,11 +6,13 @@ package insecure import ( "context" "fmt" + "io" "net" + gogoio "github.com/gogo/protobuf/io" + "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" - "github.com/libp2p/go-msgio" ci "github.com/libp2p/go-libp2p-core/crypto" pb "github.com/libp2p/go-libp2p-core/sec/insecure/pb" @@ -143,7 +145,6 @@ func (ic *Conn) runHandshakeSync() error { return nil } - rw := msgio.NewReadWriter(ic.Conn) // Generate an Exchange message msg, err := makeExchangeMessage(ic.localPrivKey.GetPublic()) if err != nil { @@ -151,7 +152,7 @@ func (ic *Conn) runHandshakeSync() error { } // Send our Exchange and read theirs - remoteMsg, err := readWriteMsg(rw, msg) + remoteMsg, err := readWriteMsg(ic.Conn, msg) if err != nil { return err } @@ -181,31 +182,28 @@ func (ic *Conn) runHandshakeSync() error { } // read and write a message at the same time. -func readWriteMsg(c msgio.ReadWriter, out *pb.Exchange) (*pb.Exchange, error) { - outBytes, err := out.Marshal() - if err != nil { - return nil, err - } +func readWriteMsg(rw io.ReadWriter, out *pb.Exchange) (*pb.Exchange, error) { + const maxMsgSize = 1 << 16 + r := gogoio.NewDelimitedReader(rw, maxMsgSize) + w := gogoio.NewDelimitedWriter(rw) wresult := make(chan error) go func() { - wresult <- c.WriteMsg(outBytes) + wresult <- w.WriteMsg(out) }() - msg, err1 := c.ReadMsg() + inMsg := pb.Exchange{} + err := r.ReadMsg(&inMsg) - // Always wait for the read to finish. + // Always wait for the write to finish. err2 := <-wresult - if err1 != nil { - return nil, err1 + if err != nil { + return nil, err } if err2 != nil { - c.ReleaseMsg(msg) return nil, err2 } - inMsg := new(pb.Exchange) - err = inMsg.Unmarshal(msg) - return inMsg, err + return &inMsg, err } // LocalPeer returns the local peer ID. From 9fd36745d027dfcda138bfff1d58b048e519c449 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 15 Nov 2019 14:39:26 +0800 Subject: [PATCH 1570/3965] make reuse work on Windows --- p2p/transport/quic/libp2pquic_suite_test.go | 2 +- p2p/transport/quic/netlink_other.go | 3 +- .../quic/{reuse.go => reuse_base.go} | 68 +++---------------- p2p/transport/quic/reuse_linux_test.go | 42 ++++++++++++ p2p/transport/quic/reuse_not_win.go | 66 ++++++++++++++++++ p2p/transport/quic/reuse_test.go | 57 ++++++---------- p2p/transport/quic/reuse_win.go | 26 +++++++ 7 files changed, 165 insertions(+), 99 deletions(-) rename p2p/transport/quic/{reuse.go => reuse_base.go} (70%) create mode 100644 p2p/transport/quic/reuse_linux_test.go create mode 100644 p2p/transport/quic/reuse_not_win.go create mode 100644 p2p/transport/quic/reuse_win.go diff --git a/p2p/transport/quic/libp2pquic_suite_test.go b/p2p/transport/quic/libp2pquic_suite_test.go index ce48c3f619..a2e1df400d 100644 --- a/p2p/transport/quic/libp2pquic_suite_test.go +++ b/p2p/transport/quic/libp2pquic_suite_test.go @@ -28,7 +28,7 @@ var maxUnusedDurationOrig time.Duration func isGarbageCollectorRunning() bool { var b bytes.Buffer pprof.Lookup("goroutine").WriteTo(&b, 1) - return strings.Contains(b.String(), "go-libp2p-quic-transport.(*reuse).runGarbageCollector") + return strings.Contains(b.String(), "go-libp2p-quic-transport.(*reuseBase).runGarbageCollector") } var _ = BeforeEach(func() { diff --git a/p2p/transport/quic/netlink_other.go b/p2p/transport/quic/netlink_other.go index 58ad3ca124..ccc073da0b 100644 --- a/p2p/transport/quic/netlink_other.go +++ b/p2p/transport/quic/netlink_other.go @@ -1,8 +1,9 @@ // +build !linux +// +build !windows package libp2pquic import "github.com/vishvananda/netlink/nl" -// nl.SupportedNlFamilies is the default netlink families used by the netlink package +// SupportedNlFamilies is the default netlink families used by the netlink package var SupportedNlFamilies = nl.SupportedNlFamilies diff --git a/p2p/transport/quic/reuse.go b/p2p/transport/quic/reuse_base.go similarity index 70% rename from p2p/transport/quic/reuse.go rename to p2p/transport/quic/reuse_base.go index c6415b4d8f..347a9a6cf7 100644 --- a/p2p/transport/quic/reuse.go +++ b/p2p/transport/quic/reuse_base.go @@ -4,11 +4,9 @@ import ( "net" "sync" "time" - - "github.com/vishvananda/netlink" ) -// Constants. Defined as variables to simplify testing. +// Constant. Defined as variables to simplify testing. var ( garbageCollectInterval = 30 * time.Second maxUnusedDuration = 10 * time.Second @@ -48,34 +46,24 @@ func (c *reuseConn) ShouldGarbageCollect(now time.Time) bool { return !c.unusedSince.IsZero() && c.unusedSince.Add(maxUnusedDuration).Before(now) } -type reuse struct { +type reuseBase struct { mutex sync.Mutex garbageCollectorRunning bool - handle *netlink.Handle // Only set on Linux. nil on other systems. - unicast map[string] /* IP.String() */ map[int] /* port */ *reuseConn // global contains connections that are listening on 0.0.0.0 / :: global map[int]*reuseConn } -func newReuse() (*reuse, error) { - // On non-Linux systems, this will return ErrNotImplemented. - handle, err := netlink.NewHandle(SupportedNlFamilies...) - if err == netlink.ErrNotImplemented { - handle = nil - } else if err != nil { - return nil, err - } - return &reuse{ +func newReuseBase() reuseBase { + return reuseBase{ unicast: make(map[string]map[int]*reuseConn), global: make(map[int]*reuseConn), - handle: handle, - }, nil + } } -func (r *reuse) runGarbageCollector() { +func (r *reuseBase) runGarbageCollector() { ticker := time.NewTicker(garbageCollectInterval) defer ticker.Stop() @@ -114,52 +102,14 @@ func (r *reuse) runGarbageCollector() { } // must be called while holding the mutex -func (r *reuse) maybeStartGarbageCollector() { +func (r *reuseBase) maybeStartGarbageCollector() { if !r.garbageCollectorRunning { r.garbageCollectorRunning = true go r.runGarbageCollector() } } -// Get the source IP that the kernel would use for dialing. -// This only works on Linux. -// On other systems, this returns an empty slice of IP addresses. -func (r *reuse) getSourceIPs(network string, raddr *net.UDPAddr) ([]net.IP, error) { - if r.handle == nil { - return nil, nil - } - - routes, err := r.handle.RouteGet(raddr.IP) - if err != nil { - return nil, err - } - - ips := make([]net.IP, 0, len(routes)) - for _, route := range routes { - ips = append(ips, route.Src) - } - return ips, nil -} - -func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { - ips, err := r.getSourceIPs(network, raddr) - if err != nil { - return nil, err - } - - r.mutex.Lock() - defer r.mutex.Unlock() - - conn, err := r.dialLocked(network, raddr, ips) - if err != nil { - return nil, err - } - conn.IncreaseCount() - r.maybeStartGarbageCollector() - return conn, nil -} - -func (r *reuse) dialLocked(network string, raddr *net.UDPAddr, ips []net.IP) (*reuseConn, error) { +func (r *reuseBase) dialLocked(network string, raddr *net.UDPAddr, ips []net.IP) (*reuseConn, error) { for _, ip := range ips { // We already have at least one suitable connection... if conns, ok := r.unicast[ip.String()]; ok { @@ -194,7 +144,7 @@ func (r *reuse) dialLocked(network string, raddr *net.UDPAddr, ips []net.IP) (*r return rconn, nil } -func (r *reuse) Listen(network string, laddr *net.UDPAddr) (*reuseConn, error) { +func (r *reuseBase) Listen(network string, laddr *net.UDPAddr) (*reuseConn, error) { conn, err := net.ListenUDP(network, laddr) if err != nil { return nil, err diff --git a/p2p/transport/quic/reuse_linux_test.go b/p2p/transport/quic/reuse_linux_test.go new file mode 100644 index 0000000000..8bc401a0c8 --- /dev/null +++ b/p2p/transport/quic/reuse_linux_test.go @@ -0,0 +1,42 @@ +// +build linux + +package libp2pquic + +import ( + "net" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +var _ = Describe("Reuse (on Linux)", func() { + var reuse *reuse + + BeforeEach(func() { + var err error + reuse, err = newReuse() + Expect(err).ToNot(HaveOccurred()) + }) + + Context("creating and reusing connections", func() { + AfterEach(func() { closeAllConns(reuse) }) + + It("reuses a connection it created for listening on a specific interface", func() { + raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") + Expect(err).ToNot(HaveOccurred()) + ips, err := reuse.getSourceIPs("udp4", raddr) + Expect(err).ToNot(HaveOccurred()) + Expect(ips).ToNot(BeEmpty()) + // listen + addr, err := net.ResolveUDPAddr("udp4", ips[0].String()+":0") + Expect(err).ToNot(HaveOccurred()) + lconn, err := reuse.Listen("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(lconn.GetCount()).To(Equal(1)) + // dial + conn, err := reuse.Dial("udp4", raddr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(2)) + }) + }) +}) diff --git a/p2p/transport/quic/reuse_not_win.go b/p2p/transport/quic/reuse_not_win.go new file mode 100644 index 0000000000..fb36b83e89 --- /dev/null +++ b/p2p/transport/quic/reuse_not_win.go @@ -0,0 +1,66 @@ +// +build !windows + +package libp2pquic + +import ( + "net" + + "github.com/vishvananda/netlink" +) + +type reuse struct { + reuseBase + + handle *netlink.Handle // Only set on Linux. nil on other systems. +} + +func newReuse() (*reuse, error) { + handle, err := netlink.NewHandle(SupportedNlFamilies...) + if err == netlink.ErrNotImplemented { + handle = nil + } else if err != nil { + return nil, err + } + return &reuse{ + reuseBase: newReuseBase(), + handle: handle, + }, nil +} + +// Get the source IP that the kernel would use for dialing. +// This only works on Linux. +// On other systems, this returns an empty slice of IP addresses. +func (r *reuse) getSourceIPs(network string, raddr *net.UDPAddr) ([]net.IP, error) { + if r.handle == nil { + return nil, nil + } + + routes, err := r.handle.RouteGet(raddr.IP) + if err != nil { + return nil, err + } + + ips := make([]net.IP, 0, len(routes)) + for _, route := range routes { + ips = append(ips, route.Src) + } + return ips, nil +} + +func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { + ips, err := r.getSourceIPs(network, raddr) + if err != nil { + return nil, err + } + + r.mutex.Lock() + defer r.mutex.Unlock() + + conn, err := r.dialLocked(network, raddr, ips) + if err != nil { + return nil, err + } + conn.IncreaseCount() + r.maybeStartGarbageCollector() + return conn, nil +} diff --git a/p2p/transport/quic/reuse_test.go b/p2p/transport/quic/reuse_test.go index 276fd5bd95..e7739156e7 100644 --- a/p2p/transport/quic/reuse_test.go +++ b/p2p/transport/quic/reuse_test.go @@ -2,7 +2,6 @@ package libp2pquic import ( "net" - "runtime" "time" . "github.com/onsi/ginkgo" @@ -15,6 +14,24 @@ func (c *reuseConn) GetCount() int { return c.refCount } +func closeAllConns(reuse *reuse) { + reuse.mutex.Lock() + for _, conn := range reuse.global { + for conn.GetCount() > 0 { + conn.DecreaseCount() + } + } + for _, conns := range reuse.unicast { + for _, conn := range conns { + for conn.GetCount() > 0 { + conn.DecreaseCount() + } + } + } + reuse.mutex.Unlock() + Eventually(isGarbageCollectorRunning).Should(BeFalse()) +} + var _ = Describe("Reuse", func() { var reuse *reuse @@ -25,23 +42,7 @@ var _ = Describe("Reuse", func() { }) Context("creating and reusing connections", func() { - AfterEach(func() { - reuse.mutex.Lock() - for _, conn := range reuse.global { - for conn.GetCount() > 0 { - conn.DecreaseCount() - } - } - for _, conns := range reuse.unicast { - for _, conn := range conns { - for conn.GetCount() > 0 { - conn.DecreaseCount() - } - } - } - reuse.mutex.Unlock() - Eventually(isGarbageCollectorRunning).Should(BeFalse()) - }) + AfterEach(func() { closeAllConns(reuse) }) It("creates a new global connection when listening on 0.0.0.0", func() { addr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:0") @@ -84,26 +85,6 @@ var _ = Describe("Reuse", func() { Expect(err).ToNot(HaveOccurred()) Expect(conn.GetCount()).To(Equal(2)) }) - - if runtime.GOOS == "linux" { - It("reuses a connection it created for listening on a specific interface", func() { - raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") - Expect(err).ToNot(HaveOccurred()) - ips, err := reuse.getSourceIPs("udp4", raddr) - Expect(err).ToNot(HaveOccurred()) - Expect(ips).ToNot(BeEmpty()) - // listen - addr, err := net.ResolveUDPAddr("udp4", ips[0].String()+":0") - Expect(err).ToNot(HaveOccurred()) - lconn, err := reuse.Listen("udp4", addr) - Expect(err).ToNot(HaveOccurred()) - Expect(lconn.GetCount()).To(Equal(1)) - // dial - conn, err := reuse.Dial("udp4", raddr) - Expect(err).ToNot(HaveOccurred()) - Expect(conn.GetCount()).To(Equal(2)) - }) - } }) Context("garbage-collecting connections", func() { diff --git a/p2p/transport/quic/reuse_win.go b/p2p/transport/quic/reuse_win.go new file mode 100644 index 0000000000..0f57c8e0ea --- /dev/null +++ b/p2p/transport/quic/reuse_win.go @@ -0,0 +1,26 @@ +// +build windows + +package libp2pquic + +import "net" + +type reuse struct { + reuseBase +} + +func newReuse() (*reuse, error) { + return &reuse{reuseBase: newReuseBase()}, nil +} + +func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { + r.mutex.Lock() + defer r.mutex.Unlock() + + conn, err := r.dialLocked(network, raddr, nil) + if err != nil { + return nil, err + } + conn.IncreaseCount() + r.maybeStartGarbageCollector() + return conn, nil +} From 9a31a773db42cbaefb0716accaf8b1b207825d70 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 15 Nov 2019 05:18:40 -0800 Subject: [PATCH 1571/3965] chore(dep): update deps --- p2p/transport/quic/conn_test.go | 2 +- p2p/transport/quic/listener_test.go | 2 +- p2p/transport/quic/transport.go | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index bf8d681149..882f2f0138 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -33,7 +33,7 @@ var _ = Describe("Connection", func() { priv, _, err = ic.GenerateECDSAKeyPair(rand.Reader) case 1: fmt.Fprintf(GinkgoWriter, " using an RSA key: ") - priv, _, err = ic.GenerateRSAKeyPair(1024, rand.Reader) + priv, _, err = ic.GenerateRSAKeyPair(2048, rand.Reader) case 2: fmt.Fprintf(GinkgoWriter, " using an Ed25519 key: ") priv, _, err = ic.GenerateEd25519Key(rand.Reader) diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index e90debc810..85c74fa7d7 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -19,7 +19,7 @@ var _ = Describe("Listener", func() { var t tpt.Transport BeforeEach(func() { - rsaKey, err := rsa.GenerateKey(rand.Reader, 1024) + rsaKey, err := rsa.GenerateKey(rand.Reader, 2048) Expect(err).ToNot(HaveOccurred()) key, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(rsaKey)) Expect(err).ToNot(HaveOccurred()) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index a9ddd41355..03b55ca2bc 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -12,8 +12,8 @@ import ( quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" + "github.com/multiformats/go-multiaddr-fmt" manet "github.com/multiformats/go-multiaddr-net" - "github.com/whyrusleeping/mafmt" ) var quicConfig = &quic.Config{ From a47189bb9bf1fcddfa39312a3242efb11f52bcbc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 15 Nov 2019 18:40:37 -0800 Subject: [PATCH 1572/3965] feat(metrics): add function to trim idle meters --- core/metrics/bandwidth.go | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/core/metrics/bandwidth.go b/core/metrics/bandwidth.go index aa758e8eaa..2d94b054d3 100644 --- a/core/metrics/bandwidth.go +++ b/core/metrics/bandwidth.go @@ -2,6 +2,8 @@ package metrics import ( + "time" + "github.com/libp2p/go-flow-metrics" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/protocol" @@ -163,3 +165,11 @@ func (bwc *BandwidthCounter) Reset() { bwc.peerIn.Clear() bwc.peerOut.Clear() } + +// TrimIdle trims all timers idle since the given time. +func (bwc *BandwidthCounter) TrimIdle(since time.Time) { + bwc.peerIn.TrimIdle(since) + bwc.peerOut.TrimIdle(since) + bwc.protocolIn.TrimIdle(since) + bwc.protocolOut.TrimIdle(since) +} From 1b8f1b69d0355cd7f5cd663a8694fe84ca61e3db Mon Sep 17 00:00:00 2001 From: Henri S Date: Mon, 18 Nov 2019 08:35:37 -0800 Subject: [PATCH 1573/3965] small grammar fixes and updates to readme --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 3c2862f326..318b75230e 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ [libp2p](https://github.com/libp2p/specs) is a networking stack and library modularized out of [The IPFS Project](https://github.com/ipfs/ipfs), and bundled separately for other tools to use. > -libp2p is the product of a long, and arduous quest of understanding -- a deep dive into the internet's network stack, and plentiful peer-to-peer protocols from the past. Building large scale peer-to-peer systems has been complex and difficult in the last 15 years, and libp2p is a way to fix that. It is a "network stack" -- a protocol suite -- that cleanly separates concerns, and enables sophisticated applications to only use the protocols they absolutely need, without giving up interoperability and upgradeability. libp2p grew out of IPFS, but it is built so that lots of people can use it, for lots of different projects. +libp2p is the product of a long, and arduous quest of understanding -- a deep dive into the internet's network stack, and plentiful peer-to-peer protocols from the past. Building large-scale peer-to-peer systems has been complex and difficult in the last 15 years, and libp2p is a way to fix that. It is a "network stack" -- a protocol suite -- that cleanly separates concerns, and enables sophisticated applications to only use the protocols they absolutely need, without giving up interoperability and upgradeability. libp2p grew out of IPFS, but it is built so that lots of people can use it, for lots of different projects. > > We will be writing a set of docs, posts, tutorials, and talks to explain what p2p is, why it is tremendously useful, and how it can help your existing and new projects. But in the meantime, check out > @@ -87,7 +87,7 @@ Examples can be found in the [examples repo](https://github.com/libp2p/go-libp2p While developing, you may need to make changes to several modules at once, or you may want changes made locally in one module to be available for import by another. -The [go libp2p workspace](https://github.com/libp2p/workspace-go-libp2p) provides a developer-oriented view of the modules that comprise go-libp2p. +The [go-libp2p workspace](https://github.com/libp2p/workspace-go-libp2p) provides a developer-oriented view of the modules that comprise go-libp2p. Using the tooling in the workspace repository, you can checkout all of go-libp2p's module repos and enter "local mode", which adds [replace directives](https://github.com/golang/go/wiki/Modules#gomod) to the go.mod files in each local working copy. When you build locally, the libp2p depdendencies will be resolved from your local working copies. @@ -203,17 +203,17 @@ List of packages currently in existence for libp2p: # Contribute -go-libp2p is part of [The IPFS Project](https://github.com/ipfs/ipfs), and is MIT licensed open source software. We welcome contributions big and small! Take a look at the [community contributing notes](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md). Please make sure to check the [issues](https://github.com/ipfs/go-libp2p/issues). Search the closed ones before reporting things, and help us with the open ones. +go-libp2p is part of [The IPFS Project](https://github.com/ipfs/ipfs), and is MIT-licensed open source software. We welcome contributions big and small! Take a look at the [community contributing notes](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md). Please make sure to check the [issues](https://github.com/ipfs/go-libp2p/issues). Search the closed ones before reporting things, and help us with the open ones. Guidelines: - read the [libp2p spec](https://github.com/libp2p/specs) - please make branches + pull-request, even if working on the main repository -- ask questions or talk about things in [Issues](https://github.com/libp2p/go-libp2p/issues), our [discussion forums](https://discuss.libp2p.io), or #libp2p or #ipfs on freenode. -- ensure you are able to contribute (no legal issues please-- we use the DCO) +- ask questions or talk about things in [issues](https://github.com/libp2p/go-libp2p/issues), our [discussion forums](https://discuss.libp2p.io), or #libp2p or #ipfs on freenode. +- ensure you are able to contribute (no legal issues please -- we use the DCO) - run `go fmt` before pushing any code - run `golint` and `go vet` too -- some things (like protobuf files) are expected to fail. -- get in touch with @jbenet and @diasdavid about how best to contribute +- get in touch with @raulk and @mgoelzer about how best to contribute - have fun! There's a few things you can do right now to help out: From 93d0ace99efa5d8be6a0ddf83f601d304ddb0ee7 Mon Sep 17 00:00:00 2001 From: iulianpascalau Date: Thu, 21 Nov 2019 20:16:09 +0200 Subject: [PATCH 1574/3965] updated go.mod and go.sum to the latest version of github.com/koron/go-ssdp library that has the race condition fixed --- go.mod | 4 +++- go.sum | 4 ++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 718fdbdff7..a6dd7e1e41 100644 --- a/go.mod +++ b/go.mod @@ -4,5 +4,7 @@ require ( github.com/huin/goupnp v1.0.0 github.com/jackpal/gateway v1.0.5 github.com/jackpal/go-nat-pmp v1.0.1 - github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b + github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d ) + +go 1.12 diff --git a/go.sum b/go.sum index 1a3a280ed3..399175cc00 100644 --- a/go.sum +++ b/go.sum @@ -5,8 +5,8 @@ github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= -github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b h1:wxtKgYHEncAU00muMD06dzLiahtGM1eouRNOzVV7tdQ= -github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= +github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d h1:68u9r4wEvL3gYg2jvAOgROwZ3H+Y3hIDk4tbbmIjcYQ= +github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1 h1:Y/KGZSOdz/2r0WJ9Mkmz6NJBusp0kiNx1Cn82lzJQ6w= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= From d65d0c4bf83f7a210f2a1fb15e0537d7a5cd7f33 Mon Sep 17 00:00:00 2001 From: vyzo Date: Thu, 21 Nov 2019 20:36:57 +0200 Subject: [PATCH 1575/3965] gomod: update go-libp2p-nat to v0.0.5 --- go.mod | 2 +- go.sum | 14 ++++++-------- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index 14464f2703..159a13fdcd 100644 --- a/go.mod +++ b/go.mod @@ -17,7 +17,7 @@ require ( github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 - github.com/libp2p/go-libp2p-nat v0.0.4 + github.com/libp2p/go-libp2p-nat v0.0.5 github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.1.4 github.com/libp2p/go-libp2p-secio v0.2.1 diff --git a/go.sum b/go.sum index 17a54e72e6..bbb95c31bb 100644 --- a/go.sum +++ b/go.sum @@ -114,8 +114,8 @@ github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQL github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= -github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b h1:wxtKgYHEncAU00muMD06dzLiahtGM1eouRNOzVV7tdQ= -github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= +github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d h1:68u9r4wEvL3gYg2jvAOgROwZ3H+Y3hIDk4tbbmIjcYQ= +github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= @@ -155,8 +155,8 @@ github.com/libp2p/go-libp2p-loggables v0.1.0/go.mod h1:EyumB2Y6PrYjr55Q3/tiJ/o3x github.com/libp2p/go-libp2p-mplex v0.2.0/go.mod h1:Ejl9IyjvXJ0T9iqUTE1jpYATQ9NM3g+OtR+EMMODbKo= github.com/libp2p/go-libp2p-mplex v0.2.1 h1:E1xaJBQnbSiTHGI1gaBKmKhu1TUKkErKJnE8iGvirYI= github.com/libp2p/go-libp2p-mplex v0.2.1/go.mod h1:SC99Rxs8Vuzrf/6WhmH41kNn13TiYdAWNYHrwImKLnE= -github.com/libp2p/go-libp2p-nat v0.0.4 h1:+KXK324yaY701On8a0aGjTnw8467kW3ExKcqW2wwmyw= -github.com/libp2p/go-libp2p-nat v0.0.4/go.mod h1:N9Js/zVtAXqaeT99cXgTV9e75KpnWCvVOiGzlcHmBbY= +github.com/libp2p/go-libp2p-nat v0.0.5 h1:/mH8pXFVKleflDL1YwqMg27W9GD8kjEx7NY0P6eGc98= +github.com/libp2p/go-libp2p-nat v0.0.5/go.mod h1:1qubaE5bTZMJE+E/uu2URroMbzdubFz1ChgiN79yKPE= github.com/libp2p/go-libp2p-netutil v0.1.0 h1:zscYDNVEcGxyUpMd0JReUZTrpMfia8PmLKcKF72EAMQ= github.com/libp2p/go-libp2p-netutil v0.1.0/go.mod h1:3Qv/aDqtMLTUyQeundkKsA+YCThNdbQD54k3TqjpbFU= github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMgxjsHm7+J5kjWY= @@ -195,8 +195,8 @@ github.com/libp2p/go-msgio v0.0.2 h1:ivPvEKHxmVkTClHzg6RXTYHqaJQ0V9cDbq+6lKb3UV0 github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-msgio v0.0.4 h1:agEFehY3zWJFUHK6SEMR7UYmk2z6kC3oeCM7ybLhguA= github.com/libp2p/go-msgio v0.0.4/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= -github.com/libp2p/go-nat v0.0.3 h1:l6fKV+p0Xa354EqQOQP+d8CivdLM4kl5GxC1hSc/UeI= -github.com/libp2p/go-nat v0.0.3/go.mod h1:88nUEt0k0JD45Bk93NIwDqjlhiOwOoV36GchpcVc1yI= +github.com/libp2p/go-nat v0.0.4 h1:KbizNnq8YIf7+Hn7+VFL/xE0eDrkPru2zIO9NMwL8UQ= +github.com/libp2p/go-nat v0.0.4/go.mod h1:Nmw50VAvKuk38jUBcmNh6p9lUJLoODbJRvYAa/+KSDo= github.com/libp2p/go-openssl v0.0.2 h1:9pP2d3Ubaxkv7ZisLjx9BFwgOGnQdQYnfcH29HNY3ls= github.com/libp2p/go-openssl v0.0.2/go.mod h1:v8Zw2ijCSWBQi8Pq5GAixw6DbFfa9u6VIYDXnvOXkc0= github.com/libp2p/go-openssl v0.0.3 h1:wjlG7HvQkt4Fq4cfH33Ivpwp0omaElYEi9z26qaIkIk= @@ -319,8 +319,6 @@ github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljT github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1/go.mod h1:8UvriyWtv5Q5EOgjHaSseUEdkQfvwFv1I/In/O2M9gc= github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6Rz4bwmou+oE6Dt4Cb2BGMur5eR/GYptkKUVHo= github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM= -github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f h1:M/lL30eFZTKnomXY6huvM6G0+gVquFNf6mxghaWlFUg= -github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f/go.mod h1:cZNvX9cFybI01GriPRMXDtczuvUhgbcYr9iCGaNlRv8= github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 h1:Y1/FEOpaCpD21WxrmfeIYCFPuVPRCY2XZTWzTNHGw30= From 199acda704ad6efe375ad8dcd39eb44f8017a6f7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 22 Nov 2019 08:20:25 -0800 Subject: [PATCH 1576/3965] fix: really obey the context The internal nat discovery logic doesn't properly propagate the context. --- nat.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/nat.go b/nat.go index c38030075c..d38ad52c12 100644 --- a/nat.go +++ b/nat.go @@ -70,6 +70,9 @@ func DiscoverNATs(ctx context.Context) <-chan NAT { if !ok { natpmp = nil } + case <-ctx.Done(): + // timeout. + return } if ok { select { From 3b381d806f9ca6abda37e0668a5bcc67e9078018 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 24 Nov 2019 18:07:58 +0700 Subject: [PATCH 1577/3965] close the underlying connection when the handshake fails --- p2p/security/tls/transport.go | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/p2p/security/tls/transport.go b/p2p/security/tls/transport.go index 453f7451e7..214853cb0f 100644 --- a/p2p/security/tls/transport.go +++ b/p2p/security/tls/transport.go @@ -54,7 +54,11 @@ var _ sec.SecureTransport = &Transport{} // SecureInbound runs the TLS handshake as a server. func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { config, keyCh := t.identity.ConfigForAny() - return t.handshake(ctx, tls.Server(insecure, config), keyCh) + cs, err := t.handshake(ctx, tls.Server(insecure, config), keyCh) + if err != nil { + insecure.Close() + } + return cs, err } // SecureOutbound runs the TLS handshake as a client. @@ -66,7 +70,11 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.S // notice this after 1 RTT when calling Read. func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { config, keyCh := t.identity.ConfigForPeer(p) - return t.handshake(ctx, tls.Client(insecure, config), keyCh) + cs, err := t.handshake(ctx, tls.Client(insecure, config), keyCh) + if err != nil { + insecure.Close() + } + return cs, err } func (t *Transport) handshake( From 5956c76d279ccfe70f12415a7e7116125dd899d6 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 2 Dec 2019 12:17:27 -0500 Subject: [PATCH 1578/3965] fix(plaintext): regenerate protobuf code --- core/sec/insecure/pb/plaintext.pb.go | 116 +++++++++++---------------- 1 file changed, 47 insertions(+), 69 deletions(-) diff --git a/core/sec/insecure/pb/plaintext.pb.go b/core/sec/insecure/pb/plaintext.pb.go index 68b12f0c82..4c5af18376 100644 --- a/core/sec/insecure/pb/plaintext.pb.go +++ b/core/sec/insecure/pb/plaintext.pb.go @@ -9,6 +9,7 @@ import ( pb "github.com/libp2p/go-libp2p-core/crypto/pb" io "io" math "math" + math_bits "math/bits" ) // Reference imports to suppress errors if they are not otherwise used. @@ -20,7 +21,7 @@ var _ = math.Inf // is compatible with the proto package it is being compiled against. // A compilation error at this line likely means your copy of the // proto package needs to be updated. -const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package type Exchange struct { Id []byte `protobuf:"bytes,1,opt,name=id" json:"id"` @@ -41,7 +42,7 @@ func (m *Exchange) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { return xxx_messageInfo_Exchange.Marshal(b, m, deterministic) } else { b = b[:cap(b)] - n, err := m.MarshalTo(b) + n, err := m.MarshalToSizedBuffer(b) if err != nil { return nil, err } @@ -99,7 +100,7 @@ var fileDescriptor_aba144f73931b711 = []byte{ func (m *Exchange) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) - n, err := m.MarshalTo(dAtA) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) if err != nil { return nil, err } @@ -107,37 +108,47 @@ func (m *Exchange) Marshal() (dAtA []byte, err error) { } func (m *Exchange) MarshalTo(dAtA []byte) (int, error) { - var i int + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *Exchange) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) _ = i var l int _ = l - if m.Id != nil { - dAtA[i] = 0xa - i++ - i = encodeVarintPlaintext(dAtA, i, uint64(len(m.Id))) - i += copy(dAtA[i:], m.Id) - } if m.Pubkey != nil { - dAtA[i] = 0x12 - i++ - i = encodeVarintPlaintext(dAtA, i, uint64(m.Pubkey.Size())) - n1, err := m.Pubkey.MarshalTo(dAtA[i:]) - if err != nil { - return 0, err + { + size, err := m.Pubkey.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintPlaintext(dAtA, i, uint64(size)) } - i += n1 + i-- + dAtA[i] = 0x12 } - return i, nil + if m.Id != nil { + i -= len(m.Id) + copy(dAtA[i:], m.Id) + i = encodeVarintPlaintext(dAtA, i, uint64(len(m.Id))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil } func encodeVarintPlaintext(dAtA []byte, offset int, v uint64) int { + offset -= sovPlaintext(v) + base := offset for v >= 1<<7 { dAtA[offset] = uint8(v&0x7f | 0x80) v >>= 7 offset++ } dAtA[offset] = uint8(v) - return offset + 1 + return base } func (m *Exchange) Size() (n int) { if m == nil { @@ -157,14 +168,7 @@ func (m *Exchange) Size() (n int) { } func sovPlaintext(x uint64) (n int) { - for { - n++ - x >>= 7 - if x == 0 { - break - } - } - return n + return (math_bits.Len64(x|1) + 6) / 7 } func sozPlaintext(x uint64) (n int) { return sovPlaintext(uint64((x << 1) ^ uint64((int64(x) >> 63)))) @@ -295,6 +299,7 @@ func (m *Exchange) Unmarshal(dAtA []byte) error { func skipPlaintext(dAtA []byte) (n int, err error) { l := len(dAtA) iNdEx := 0 + depth := 0 for iNdEx < l { var wire uint64 for shift := uint(0); ; shift += 7 { @@ -326,10 +331,8 @@ func skipPlaintext(dAtA []byte) (n int, err error) { break } } - return iNdEx, nil case 1: iNdEx += 8 - return iNdEx, nil case 2: var length int for shift := uint(0); ; shift += 7 { @@ -350,55 +353,30 @@ func skipPlaintext(dAtA []byte) (n int, err error) { return 0, ErrInvalidLengthPlaintext } iNdEx += length - if iNdEx < 0 { - return 0, ErrInvalidLengthPlaintext - } - return iNdEx, nil case 3: - for { - var innerWire uint64 - var start int = iNdEx - for shift := uint(0); ; shift += 7 { - if shift >= 64 { - return 0, ErrIntOverflowPlaintext - } - if iNdEx >= l { - return 0, io.ErrUnexpectedEOF - } - b := dAtA[iNdEx] - iNdEx++ - innerWire |= (uint64(b) & 0x7F) << shift - if b < 0x80 { - break - } - } - innerWireType := int(innerWire & 0x7) - if innerWireType == 4 { - break - } - next, err := skipPlaintext(dAtA[start:]) - if err != nil { - return 0, err - } - iNdEx = start + next - if iNdEx < 0 { - return 0, ErrInvalidLengthPlaintext - } - } - return iNdEx, nil + depth++ case 4: - return iNdEx, nil + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupPlaintext + } + depth-- case 5: iNdEx += 4 - return iNdEx, nil default: return 0, fmt.Errorf("proto: illegal wireType %d", wireType) } + if iNdEx < 0 { + return 0, ErrInvalidLengthPlaintext + } + if depth == 0 { + return iNdEx, nil + } } - panic("unreachable") + return 0, io.ErrUnexpectedEOF } var ( - ErrInvalidLengthPlaintext = fmt.Errorf("proto: negative length found during unmarshaling") - ErrIntOverflowPlaintext = fmt.Errorf("proto: integer overflow") + ErrInvalidLengthPlaintext = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowPlaintext = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupPlaintext = fmt.Errorf("proto: unexpected end of group") ) From c64dd2c19dac8656242a17bbcccfddf2368d14b6 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 4 Dec 2019 13:16:27 -0500 Subject: [PATCH 1579/3965] fix initiator / responder roles in test setup --- p2p/security/noise/transport_test.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 35c69fd53a..63b0048ed5 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -80,15 +80,15 @@ func newConnPair(t *testing.T) (net.Conn, net.Conn) { func connect(t *testing.T, initTransport, respTransport *Transport) (*secureSession, *secureSession) { init, resp := newConnPair(t) - var respConn sec.SecureConn - var respErr error + var initConn sec.SecureConn + var initErr error done := make(chan struct{}) go func() { defer close(done) - respConn, respErr = respTransport.SecureOutbound(context.TODO(), resp, initTransport.LocalID) + initConn, initErr = initTransport.SecureOutbound(context.TODO(), init, respTransport.LocalID) }() - initConn, initErr := initTransport.SecureInbound(context.TODO(), init) + respConn, respErr := respTransport.SecureInbound(context.TODO(), resp) <-done if initErr != nil { From a7d5094883e1608642605f52b89fe3048ce8b868 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 4 Dec 2019 13:17:09 -0500 Subject: [PATCH 1580/3965] check right handshake is used in noise pipes tests --- p2p/security/noise/transport_test.go | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 63b0048ed5..585f64a3f1 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -176,25 +176,17 @@ func TestHandshakeXX(t *testing.T) { // Test IK handshake func TestHandshakeIK(t *testing.T) { - initTransport := newTestTransport(t, crypto.Ed25519, 2048) - respTransport := newTestTransport(t, crypto.Ed25519, 2048) - - // do initial XX handshake - initConn, respConn := connect(t, initTransport, respTransport) - initConn.Close() - respConn.Close() - - // turn on pipes, this will turn on IK - initTransport.NoisePipesSupport = true - respTransport.NoisePipesSupport = true + initTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) + respTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) // add responder's static key to initiator's key cache + respTransport.NoiseKeypair = GenerateKeypair() keycache := make(map[peer.ID]([32]byte)) keycache[respTransport.LocalID] = respTransport.NoiseKeypair.public_key initTransport.NoiseStaticKeyCache = keycache // do IK handshake - initConn, respConn = connect(t, initTransport, respTransport) + initConn, respConn := connect(t, initTransport, respTransport) defer initConn.Close() defer respConn.Close() @@ -213,6 +205,11 @@ func TestHandshakeIK(t *testing.T) { if !bytes.Equal(before, after) { t.Errorf("Message mismatch. %v != %v", before, after) } + + // make sure IK was actually used + if !(initConn.ik_complete && respConn.ik_complete) { + t.Error("Expected IK handshake to be used") + } } // Test noise pipes @@ -241,4 +238,9 @@ func TestHandshakeXXfallback(t *testing.T) { if !bytes.Equal(before, after) { t.Errorf("Message mismatch. %v != %v", before, after) } + + // make sure XX was actually used + if !(initConn.xx_complete && respConn.xx_complete) { + t.Error("Expected XXfallback handshake to be used") + } } From ca9fa6c563aa1dd2932738e74c671311a21e4505 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 4 Dec 2019 13:17:57 -0500 Subject: [PATCH 1581/3965] use correct message decoder in XXfallback --- p2p/security/noise/xx_handshake.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 5eff33fa44..7a84564e83 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -202,7 +202,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } else { var msgbuf *xx.MessageBuffer - msgbuf, err = xx.Decode1(initialMsg) + msgbuf, err = xx.Decode0(initialMsg) if err != nil { log.Errorf("runHandshake_xx recv msg err", err) return err From fdf4c4428c894cd39f236451b24c7640a76d7fc6 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 4 Dec 2019 13:18:28 -0500 Subject: [PATCH 1582/3965] use Errorf instead of Error --- p2p/security/noise/ik_handshake.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index d05105e5a0..4231969d91 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -60,13 +60,13 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, log.Debugf("ik_recvHandshakeMessage initiator=%v msgbuf=%v", s.initiator, msgbuf) if err != nil { - log.Error("ik_recvHandshakeMessage initiator=%v decode err=%s", s.initiator, err) + log.Errorf("ik_recvHandshakeMessage initiator=%v decode err=%s", s.initiator, err) return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage decode msg fail: %s", err) } s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) if !valid { - log.Error("ik_recvHandshakeMessage initiator=%v err=%s", s.initiator, "validation fail") + log.Errorf("ik_recvHandshakeMessage initiator=%v err=%s", s.initiator, "validation fail") return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage validation fail") } From 0a7571794415d3193b9018e10f8b3a76fabe1381 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 4 Dec 2019 13:18:52 -0500 Subject: [PATCH 1583/3965] fix hash function in Noise protocol name --- p2p/security/noise/ik/IK.noise.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go index 61a11ed038..724a98e572 100644 --- a/p2p/security/noise/ik/IK.noise.go +++ b/p2p/security/noise/ik/IK.noise.go @@ -409,7 +409,7 @@ func initializeInitiator(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) var ss symmetricstate var e Keypair var re [32]byte - name := []byte("Noise_IK_25519_ChaChaPoly_BLAKE2s") + name := []byte("Noise_IK_25519_ChaChaPoly_SHA256") ss = initializeSymmetric(name) mixHash(&ss, prologue) mixHash(&ss, rs[:]) @@ -420,7 +420,7 @@ func initializeResponder(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) var ss symmetricstate var e Keypair var re [32]byte - name := []byte("Noise_IK_25519_ChaChaPoly_BLAKE2s") + name := []byte("Noise_IK_25519_ChaChaPoly_SHA256") ss = initializeSymmetric(name) mixHash(&ss, prologue) mixHash(&ss, s.public_key[:]) From 423803f233b7f1b39abf1f58ab1d133e80c3a9e8 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 4 Dec 2019 13:25:11 -0500 Subject: [PATCH 1584/3965] fix conditional for whether to use IK --- p2p/security/noise/protocol.go | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 259016a083..766ed565de 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -173,11 +173,15 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("runHandshake proto marshal payload err=%s", err) } - // if we have the peer's noise static key (and we're the initiator,) and we support noise pipes, - // we can try IK first. if we support noise pipes and we're not the initiator, try IK first - // otherwise, default to XX - if (!s.initiator && s.noiseStaticKeyCache[s.remotePeer] != [32]byte{}) && s.noisePipesSupport { - // known static key for peer, try IK + // If we support Noise pipes, we try IK first, falling back to XX if IK fails. + // The exception is when we're the initiator and don't know the other party's + // static Noise key. Then IK will always fail, so we go straight to XX. + tryIK := s.noisePipesSupport + if s.initiator && s.noiseStaticKeyCache[s.remotePeer] == [32]byte{} { + tryIK = false + } + if tryIK { + // we're either a responder or an initiator with a known static key for the remote peer, try IK buf, err := s.runHandshake_ik(ctx, payloadEnc) if err != nil { log.Error("runHandshake ik err=%s", err) From c3157aaaa13adc7ca9659269624e1eb23bc2b08d Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 4 Dec 2019 13:25:34 -0500 Subject: [PATCH 1585/3965] only set ik_complete = true if IK succeeds --- p2p/security/noise/protocol.go | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 766ed565de..77889c649c 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -194,10 +194,9 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } s.xx_complete = true + } else { + s.ik_complete = true } - - s.ik_complete = true - } else { // unknown static key for peer, try XX err := s.runHandshake_xx(ctx, false, payloadEnc, nil) From 84d5b558b50342cf97d07578243082b8822260cd Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 5 Dec 2019 12:14:30 -0500 Subject: [PATCH 1586/3965] add test for large plaintext payloads --- p2p/security/noise/transport_test.go | 39 ++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 35c69fd53a..19ff856c0f 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -3,6 +3,7 @@ package noise import ( "bytes" "context" + "math/rand" "net" "testing" @@ -148,6 +149,44 @@ func TestKeys(t *testing.T) { } } +func makeLargePlaintext(size int) []byte { + buf := make([]byte, size) + rand.Read(buf) + return buf +} + +func TestLargePayloads(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + // enough to require a couple Noise messages, with a size that + // isn't a neat multiple of Noise message size, just in case + size := 100000 + + before := makeLargePlaintext(size) + _, err := initConn.Write(before) + if err != nil { + t.Fatal(err) + } + + after := make([]byte, len(before)) + afterLen, err := respConn.Read(after) + if err != nil { + t.Fatal(err) + } + + if len(before) != afterLen { + t.Errorf("expected to read same amount of data as written. written=%d read=%d", len(before), afterLen) + } + if !bytes.Equal(before, after) { + t.Error("Message mismatch.") + } +} + // Tests XX handshake func TestHandshakeXX(t *testing.T) { initTransport := newTestTransport(t, crypto.Ed25519, 2048) From e9237a8341dd2500184c4d11278f28ab881ce9ae Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 5 Dec 2019 12:15:57 -0500 Subject: [PATCH 1587/3965] split large plaintexts into chunks --- p2p/security/noise/protocol.go | 105 ++++++++++++++++++++++----------- 1 file changed, 72 insertions(+), 33 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 259016a083..9e44d23a83 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -20,6 +20,11 @@ import ( const payload_string = "noise-libp2p-static-key:" +// Each encrypted transport message must be <= 65,535 bytes, including 16 +// bytes of authentication data. To write larger plaintexts, we split them +// into fragments of maxPlaintextLength before encrypting. +const maxPlaintextLength = 65519 + var log = logging.Logger("noise") type secureSession struct { @@ -234,33 +239,50 @@ func (s *secureSession) Read(buf []byte) (int, error) { return l, nil } - // read length of encrypted message - l, err := s.readLength() - if err != nil { - return 0, err - } + readChunk := func(buf []byte) (int, error) { + // read length of encrypted message + l, err := s.readLength() + if err != nil { + return 0, err + } - // read and decrypt ciphertext - ciphertext := make([]byte, l) - _, err = s.insecure.Read(ciphertext) - if err != nil { - log.Error("read ciphertext err", err) - return 0, err - } + // read and decrypt ciphertext + ciphertext := make([]byte, l) + _, err = s.insecure.Read(ciphertext) + if err != nil { + log.Error("read ciphertext err", err) + return 0, err + } - plaintext, err := s.Decrypt(ciphertext) - if err != nil { - log.Error("decrypt err", err) - return 0, err + plaintext, err := s.Decrypt(ciphertext) + if err != nil { + log.Error("decrypt err", err) + return 0, err + } + + // append plaintext to message buffer, copy over what can fit in the buf + // then advance message buffer to remove what was copied + s.msgBuffer = append(s.msgBuffer, plaintext...) + c := copy(buf, s.msgBuffer) + s.msgBuffer = s.msgBuffer[c:] + return c, nil } - // append plaintext to message buffer, copy over what can fit in the buf - // then advance message buffer to remove what was copied - s.msgBuffer = append(s.msgBuffer, plaintext...) - c := copy(buf, s.msgBuffer) - s.msgBuffer = s.msgBuffer[c:] + total := 0 + for i := 0; i < len(buf); i += maxPlaintextLength { + end := i + maxPlaintextLength + if end > len(buf) { + end = len(buf) + } - return c, nil + c, err := readChunk(buf[i:end]) + total += c + if err != nil { + return total, err + } + } + + return total, nil } func (s *secureSession) RemoteAddr() net.Addr { @@ -291,20 +313,37 @@ func (s *secureSession) Write(in []byte) (int, error) { s.rwLock.Lock() defer s.rwLock.Unlock() - ciphertext, err := s.Encrypt(in) - if err != nil { - log.Error("encrypt error", err) - return 0, err - } + writeChunk := func(in []byte) (int, error) { + ciphertext, err := s.Encrypt(in) + if err != nil { + log.Error("encrypt error", err) + return 0, err + } - err = s.writeLength(len(ciphertext)) - if err != nil { - log.Error("write length err", err) - return 0, err + err = s.writeLength(len(ciphertext)) + if err != nil { + log.Error("write length err", err) + return 0, err + } + + _, err = s.insecure.Write(ciphertext) + return len(in), err } - _, err = s.insecure.Write(ciphertext) - return len(in), err + written := 0 + for i := 0; i < len(in); i += maxPlaintextLength { + end := i + maxPlaintextLength + if end > len(in) { + end = len(in) + } + + l, err := writeChunk(in[i:end]) + written += l + if err != nil { + return written, err + } + } + return written, nil } func (s *secureSession) Close() error { From c437f7a097d61f1a7cabac8c4ef4a32e0f35961f Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Fri, 6 Dec 2019 17:39:48 +0000 Subject: [PATCH 1588/3965] Bump github.com/jackpal/go-nat-pmp from 1.0.1 to 1.0.2 Bumps [github.com/jackpal/go-nat-pmp](https://github.com/jackpal/go-nat-pmp) from 1.0.1 to 1.0.2. - [Release notes](https://github.com/jackpal/go-nat-pmp/releases) - [Commits](https://github.com/jackpal/go-nat-pmp/compare/v1.0.1...v1.0.2) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index a6dd7e1e41..3b3c622235 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/libp2p/go-nat require ( github.com/huin/goupnp v1.0.0 github.com/jackpal/gateway v1.0.5 - github.com/jackpal/go-nat-pmp v1.0.1 + github.com/jackpal/go-nat-pmp v1.0.2 github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d ) diff --git a/go.sum b/go.sum index 399175cc00..547800b2b7 100644 --- a/go.sum +++ b/go.sum @@ -5,6 +5,8 @@ github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= +github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= +github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d h1:68u9r4wEvL3gYg2jvAOgROwZ3H+Y3hIDk4tbbmIjcYQ= github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1 h1:Y/KGZSOdz/2r0WJ9Mkmz6NJBusp0kiNx1Cn82lzJQ6w= From 3998fd0696f885e1d48a664a52d467eb7eab4b36 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Fri, 6 Dec 2019 17:42:05 +0000 Subject: [PATCH 1589/3965] build(deps): bump github.com/libp2p/go-libp2p-core from 0.2.4 to 0.2.5 Bumps [github.com/libp2p/go-libp2p-core](https://github.com/libp2p/go-libp2p-core) from 0.2.4 to 0.2.5. - [Release notes](https://github.com/libp2p/go-libp2p-core/releases) - [Commits](https://github.com/libp2p/go-libp2p-core/compare/v0.2.4...v0.2.5) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 159a13fdcd..6435b2e604 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,7 @@ require ( github.com/libp2p/go-libp2p-autonat v0.1.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 - github.com/libp2p/go-libp2p-core v0.2.4 + github.com/libp2p/go-libp2p-core v0.2.5 github.com/libp2p/go-libp2p-discovery v0.1.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 diff --git a/go.sum b/go.sum index bbb95c31bb..882d407170 100644 --- a/go.sum +++ b/go.sum @@ -132,6 +132,7 @@ github.com/libp2p/go-eventbus v0.1.0 h1:mlawomSAjjkk97QnYiEmHsLu7E136+2oCWSHRUvM github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UGQf8A2eEmx4= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= +github.com/libp2p/go-flow-metrics v0.0.2/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= @@ -147,6 +148,8 @@ github.com/libp2p/go-libp2p-core v0.2.2 h1:Sv1ggdoMx9c7v7FOFkR7agraHCnAgqYsXrU1A github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZVoVrLaDRvIRng0= github.com/libp2p/go-libp2p-core v0.2.4 h1:Et6ykkTwI6PU44tr8qUF9k43vP0aduMNniShAbUJJw8= github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= +github.com/libp2p/go-libp2p-core v0.2.5 h1:iP1PIiIrlRrGbE1fYq2918yBc5NlCH3pFuIPSWU9hds= +github.com/libp2p/go-libp2p-core v0.2.5/go.mod h1:6+5zJmKhsf7yHn1RbmYDu08qDUpIUxGdqHuEZckmZOA= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= From 8e6e4dfe4aa5b96486afd0718e78af11d9ab673b Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Fri, 6 Dec 2019 17:42:29 +0000 Subject: [PATCH 1590/3965] build(deps): bump github.com/multiformats/go-multiaddr Bumps [github.com/multiformats/go-multiaddr](https://github.com/multiformats/go-multiaddr) from 0.1.1 to 0.1.2. - [Release notes](https://github.com/multiformats/go-multiaddr/releases) - [Commits](https://github.com/multiformats/go-multiaddr/compare/v0.1.1...v0.1.2) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 159a13fdcd..7441c881c5 100644 --- a/go.mod +++ b/go.mod @@ -30,7 +30,7 @@ require ( github.com/libp2p/go-tcp-transport v0.1.1 github.com/libp2p/go-ws-transport v0.1.2 github.com/miekg/dns v1.1.12 // indirect - github.com/multiformats/go-multiaddr v0.1.1 + github.com/multiformats/go-multiaddr v0.1.2 github.com/multiformats/go-multiaddr-dns v0.2.0 github.com/multiformats/go-multiaddr-net v0.1.1 github.com/multiformats/go-multistream v0.1.0 diff --git a/go.sum b/go.sum index bbb95c31bb..60cc9f7551 100644 --- a/go.sum +++ b/go.sum @@ -254,6 +254,8 @@ github.com/multiformats/go-multiaddr v0.0.4/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lg github.com/multiformats/go-multiaddr v0.1.0/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr v0.1.1 h1:rVAztJYMhCQ7vEFr8FvxW3mS+HF2eY/oPbOMeS0ZDnE= github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo= +github.com/multiformats/go-multiaddr v0.1.2 h1:HWYHNSyyllbQopmVIF5K7JKJugiah+L9/kuZKHbmNdQ= +github.com/multiformats/go-multiaddr v0.1.2/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= @@ -278,6 +280,8 @@ github.com/multiformats/go-multihash v0.0.8 h1:wrYcW5yxSi3dU07n5jnuS5PrNwyHy0zRH github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ= github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= +github.com/multiformats/go-varint v0.0.1 h1:TR/0rdQtnNxuN2IhiB639xC3tWM4IUi7DkTBVTdGW/M= +github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w= From 2262669b6b3c7b6ac576004299ef18128a9cfa63 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Fri, 6 Dec 2019 17:42:51 +0000 Subject: [PATCH 1591/3965] build(deps): bump github.com/libp2p/go-libp2p-testing Bumps [github.com/libp2p/go-libp2p-testing](https://github.com/libp2p/go-libp2p-testing) from 0.1.0 to 0.1.1. - [Release notes](https://github.com/libp2p/go-libp2p-testing/releases) - [Commits](https://github.com/libp2p/go-libp2p-testing/compare/v0.1.0...v0.1.1) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 159a13fdcd..bc52f295fd 100644 --- a/go.mod +++ b/go.mod @@ -22,7 +22,7 @@ require ( github.com/libp2p/go-libp2p-peerstore v0.1.4 github.com/libp2p/go-libp2p-secio v0.2.1 github.com/libp2p/go-libp2p-swarm v0.2.2 - github.com/libp2p/go-libp2p-testing v0.1.0 + github.com/libp2p/go-libp2p-testing v0.1.1 github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 github.com/libp2p/go-libp2p-yamux v0.2.1 github.com/libp2p/go-maddr-filter v0.0.5 diff --git a/go.sum b/go.sum index bbb95c31bb..93c0efe37e 100644 --- a/go.sum +++ b/go.sum @@ -180,6 +180,8 @@ github.com/libp2p/go-libp2p-testing v0.0.4 h1:Qev57UR47GcLPXWjrunv5aLIQGO4n9mhI/ github.com/libp2p/go-libp2p-testing v0.0.4/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.1.0 h1:WaFRj/t3HdMZGNZqnU2pS7pDRBmMeoDx7/HDNpeyT9U= github.com/libp2p/go-libp2p-testing v0.1.0/go.mod h1:xaZWMJrPUM5GlDBxCeGUi7kI4eqnjVyavGroI2nxEM0= +github.com/libp2p/go-libp2p-testing v0.1.1 h1:U03z3HnGI7Ni8Xx6ONVZvUFOAzWYmolWf5W5jAOPNmU= +github.com/libp2p/go-libp2p-testing v0.1.1/go.mod h1:xaZWMJrPUM5GlDBxCeGUi7kI4eqnjVyavGroI2nxEM0= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 h1:PZMS9lhjK9VytzMCW3tWHAXtKXmlURSc3ZdvwEcKCzw= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07qPW4r01GnTenLW4oi3lOPbUMGJJA= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= From 2b468e9e4ebb3d5db9216806f1084bfcedf4086b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 6 Dec 2019 12:59:52 -0500 Subject: [PATCH 1592/3965] docs(routing/query): document public query event interfaces fixes https://github.com/libp2p/go-libp2p-routing/issues/41 --- core/routing/query.go | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/core/routing/query.go b/core/routing/query.go index 3769d1c265..42acdb1fbb 100644 --- a/core/routing/query.go +++ b/core/routing/query.go @@ -7,22 +7,32 @@ import ( "github.com/libp2p/go-libp2p-core/peer" ) +// QueryEventType indicates the query event's type. type QueryEventType int // Number of events to buffer. var QueryEventBufferSize = 16 const ( + // Sending a query to a peer. SendingQuery QueryEventType = iota + // Got a response from a peer. PeerResponse + // Found a "closest" peer (not currently used). FinalPeer + // Got an error when querying. QueryError + // Found a provider. Provider + // Found a value. Value + // Adding a peer to the query. AddingPeer + // Dialing a peer. DialingPeer ) +// QueryEvent is emitted for every notable event that happens during a DHT query. type QueryEvent struct { ID peer.ID Type QueryEventType @@ -67,6 +77,12 @@ func (e *eventChannel) send(ctx context.Context, ev *QueryEvent) { e.mu.Unlock() } +// RegisterForQueryEvents registers a query event channel with the given +// context. The returned context can be passed to DHT queries to receive query +// events on the returned channels. +// +// The passed context MUST be canceled when the caller is no longer interested +// in query events. func RegisterForQueryEvents(ctx context.Context) (context.Context, <-chan *QueryEvent) { ch := make(chan *QueryEvent, QueryEventBufferSize) ech := &eventChannel{ch: ch, ctx: ctx} @@ -74,6 +90,8 @@ func RegisterForQueryEvents(ctx context.Context) (context.Context, <-chan *Query return context.WithValue(ctx, routingQueryKey{}, ech), ch } +// PublishQueryEvent publishes a query event to the query event channel +// associated with the given context, if any. func PublishQueryEvent(ctx context.Context, ev *QueryEvent) { ich := ctx.Value(routingQueryKey{}) if ich == nil { From 1d0d036ac0928e8b140220dfaeab9a64281f68f7 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Fri, 6 Dec 2019 13:28:22 -0500 Subject: [PATCH 1593/3965] rename imports from ChainSafe to libp2p org --- p2p/security/noise/crypto.go | 4 ++-- p2p/security/noise/ik/IK_test.go | 2 +- p2p/security/noise/ik_handshake.go | 4 ++-- p2p/security/noise/protocol.go | 6 +++--- p2p/security/noise/transport_test.go | 2 +- p2p/security/noise/xx/XX_test.go | 2 +- p2p/security/noise/xx_handshake.go | 4 ++-- 7 files changed, 12 insertions(+), 12 deletions(-) diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index e2f9fb858f..9d53d4e9bc 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -2,8 +2,8 @@ package noise import ( "errors" - ik "github.com/ChainSafe/go-libp2p-noise/ik" - xx "github.com/ChainSafe/go-libp2p-noise/xx" + ik "github.com/libp2p/go-libp2p-noise/ik" + xx "github.com/libp2p/go-libp2p-noise/xx" ) func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) { diff --git a/p2p/security/noise/ik/IK_test.go b/p2p/security/noise/ik/IK_test.go index 13190a61e7..d0beb52c56 100644 --- a/p2p/security/noise/ik/IK_test.go +++ b/p2p/security/noise/ik/IK_test.go @@ -2,9 +2,9 @@ package ik import ( "crypto/rand" - pb "github.com/ChainSafe/go-libp2p-noise/pb" proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" + pb "github.com/libp2p/go-libp2p-noise/pb" "testing" ) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 4231969d91..2bdfcbb8b8 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -4,9 +4,9 @@ import ( "context" "fmt" - ik "github.com/ChainSafe/go-libp2p-noise/ik" - pb "github.com/ChainSafe/go-libp2p-noise/pb" proto "github.com/gogo/protobuf/proto" + ik "github.com/libp2p/go-libp2p-noise/ik" + pb "github.com/libp2p/go-libp2p-noise/pb" ) func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bool) error { diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index bf71908323..2d97b81eee 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -13,9 +13,9 @@ import ( "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" - ik "github.com/ChainSafe/go-libp2p-noise/ik" - pb "github.com/ChainSafe/go-libp2p-noise/pb" - xx "github.com/ChainSafe/go-libp2p-noise/xx" + ik "github.com/libp2p/go-libp2p-noise/ik" + pb "github.com/libp2p/go-libp2p-noise/pb" + xx "github.com/libp2p/go-libp2p-noise/xx" ) const payload_string = "noise-libp2p-static-key:" diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 9ddbd3efc5..ac964deef4 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -7,7 +7,7 @@ import ( "net" "testing" - //ik "github.com/ChainSafe/go-libp2p-noise/ik" + //ik "github.com/libp2p/go-libp2p-noise/ik" crypto "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 3f25e6853e..3445a4bc24 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -3,9 +3,9 @@ package xx import ( "crypto/rand" "encoding/hex" - pb "github.com/ChainSafe/go-libp2p-noise/pb" proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" + pb "github.com/libp2p/go-libp2p-noise/pb" "testing" ) diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 7a84564e83..f1f507c151 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -7,8 +7,8 @@ import ( proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/peer" - pb "github.com/ChainSafe/go-libp2p-noise/pb" - xx "github.com/ChainSafe/go-libp2p-noise/xx" + pb "github.com/libp2p/go-libp2p-noise/pb" + xx "github.com/libp2p/go-libp2p-noise/xx" ) func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { From b72caed3da34deb3622f5f576e61d239b90995dd Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Fri, 6 Dec 2019 15:13:51 -0500 Subject: [PATCH 1594/3965] guard static key cache with mutex --- p2p/security/noise/ik_handshake.go | 13 ++++++++--- p2p/security/noise/integration_test.go | 4 ++-- p2p/security/noise/keycache.go | 30 ++++++++++++++++++++++++++ p2p/security/noise/protocol.go | 10 ++++----- p2p/security/noise/transport.go | 19 ++++++++++------ p2p/security/noise/transport_test.go | 4 ++-- p2p/security/noise/xx_handshake.go | 4 ++-- 7 files changed, 63 insertions(+), 21 deletions(-) create mode 100644 p2p/security/noise/keycache.go diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 2bdfcbb8b8..df9567d912 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -84,10 +84,17 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] log.Debugf("runHandshake_ik initiator=%v pubkey=%x", kp.PubKey(), s.initiator) - // new XX noise session - s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, s.noiseStaticKeyCache[s.remotePeer]) + remoteNoiseKey := s.noiseStaticKeyCache.Load(s.remotePeer) + + // new IK noise session + s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, remoteNoiseKey) if s.initiator { + // bail out early if we don't know the remote Noise key + if remoteNoiseKey == [32]byte{} { + return nil, fmt.Errorf("runHandshake_ik aborting - unknown static key for peer %s", s.remotePeer.Pretty()) + } + // stage 0 // err := s.ik_sendHandshakeMessage(payload, true) if err != nil { @@ -128,7 +135,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] } // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.noiseStaticKeyCache[s.remotePeer]) + err = s.verifyPayload(nhp, remoteNoiseKey) if err != nil { log.Errorf("runHandshake_ik stage=1 initiator=true verify payload err=%s", err) } diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index 21ffb4fabd..f43a28355d 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -75,8 +75,8 @@ func makeNodePipes(t *testing.T, seed int64, port int, rpid peer.ID, rpubkey [32 } tpt := NewTransport(pid, priv, true, kp) - tpt.NoiseStaticKeyCache = make(map[peer.ID]([32]byte)) - tpt.NoiseStaticKeyCache[rpid] = rpubkey + tpt.NoiseStaticKeyCache = NewKeyCache() + tpt.NoiseStaticKeyCache.Store(rpid, rpubkey) ip := "0.0.0.0" addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", ip, port)) diff --git a/p2p/security/noise/keycache.go b/p2p/security/noise/keycache.go new file mode 100644 index 0000000000..f43ce78a2a --- /dev/null +++ b/p2p/security/noise/keycache.go @@ -0,0 +1,30 @@ +package noise + +import ( + "github.com/libp2p/go-libp2p-core/peer" + "sync" +) + +type KeyCache struct { + lock sync.RWMutex + m map[peer.ID][32]byte +} + +func NewKeyCache() *KeyCache { + return &KeyCache{ + lock: sync.RWMutex{}, + m: make(map[peer.ID][32]byte), + } +} + +func (kc *KeyCache) Store(p peer.ID, key [32]byte) { + kc.lock.Lock() + kc.m[p] = key + kc.lock.Unlock() +} + +func (kc *KeyCache) Load(p peer.ID) [32]byte { + kc.lock.RLock() + defer kc.lock.RUnlock() + return kc.m[p] +} diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 2d97b81eee..351d42ed31 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -47,7 +47,7 @@ type secureSession struct { ik_complete bool noisePipesSupport bool - noiseStaticKeyCache map[peer.ID]([32]byte) + noiseStaticKeyCache *KeyCache noiseKeypair *Keypair @@ -67,11 +67,11 @@ type peerInfo struct { // With noise pipes off, we always do XX // With noise pipes on, we first try IK, if that fails, move to XXfallback func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, kp *Keypair, - insecure net.Conn, remote peer.ID, noiseStaticKeyCache map[peer.ID]([32]byte), + insecure net.Conn, remote peer.ID, noiseStaticKeyCache *KeyCache, noisePipesSupport bool, initiator bool) (*secureSession, error) { if noiseStaticKeyCache == nil { - noiseStaticKeyCache = make(map[peer.ID]([32]byte)) + noiseStaticKeyCache = NewKeyCache() } if kp == nil { @@ -102,7 +102,7 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey return s, err } -func (s *secureSession) NoiseStaticKeyCache() map[peer.ID]([32]byte) { +func (s *secureSession) NoiseStaticKeyCache() *KeyCache { return s.noiseStaticKeyCache } @@ -182,7 +182,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // The exception is when we're the initiator and don't know the other party's // static Noise key. Then IK will always fail, so we go straight to XX. tryIK := s.noisePipesSupport - if s.initiator && s.noiseStaticKeyCache[s.remotePeer] == [32]byte{} { + if s.initiator && s.noiseStaticKeyCache.Load(s.remotePeer) == [32]byte{} { tryIK = false } if tryIK { diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 730395f16e..3bb6ee13b1 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -38,7 +38,7 @@ type Transport struct { LocalID peer.ID PrivateKey crypto.PrivKey NoisePipesSupport bool - NoiseStaticKeyCache map[peer.ID]([32]byte) + NoiseStaticKeyCache *KeyCache NoiseKeypair *Keypair } @@ -49,11 +49,18 @@ func NewTransport(localID peer.ID, privkey crypto.PrivKey, noisePipesSupport boo kp = GenerateKeypair() } + // the static key cache is only useful if Noise Pipes is enabled + var keyCache *KeyCache + if noisePipesSupport { + keyCache = NewKeyCache() + } + return &Transport{ - LocalID: localID, - PrivateKey: privkey, - NoisePipesSupport: noisePipesSupport, - NoiseKeypair: kp, + LocalID: localID, + PrivateKey: privkey, + NoisePipesSupport: noisePipesSupport, + NoiseKeypair: kp, + NoiseStaticKeyCache: keyCache, } } @@ -64,7 +71,6 @@ func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.S return s, err } - t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() t.NoiseKeypair = s.noiseKeypair return s, nil } @@ -76,7 +82,6 @@ func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p pee return s, err } - t.NoiseStaticKeyCache = s.NoiseStaticKeyCache() t.NoiseKeypair = s.noiseKeypair return s, nil } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index ac964deef4..b8c8957643 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -220,8 +220,8 @@ func TestHandshakeIK(t *testing.T) { // add responder's static key to initiator's key cache respTransport.NoiseKeypair = GenerateKeypair() - keycache := make(map[peer.ID]([32]byte)) - keycache[respTransport.LocalID] = respTransport.NoiseKeypair.public_key + keycache := NewKeyCache() + keycache.Store(respTransport.LocalID, respTransport.NoiseKeypair.public_key) initTransport.NoiseStaticKeyCache = keycache // do IK handshake diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index f1f507c151..fa597c6014 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -178,7 +178,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } if s.noisePipesSupport { - s.noiseStaticKeyCache[s.remotePeer] = s.xx_ns.RemoteKey() + s.noiseStaticKeyCache.Store(s.remotePeer, s.xx_ns.RemoteKey()) } } else { @@ -268,7 +268,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } if s.noisePipesSupport { - s.noiseStaticKeyCache[s.remotePeer] = s.remote.noiseKey + s.noiseStaticKeyCache.Store(s.remotePeer, s.remote.noiseKey) } } From 9db625ec8d9f49711dd760dd80b2cbc2f99d5c8b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 6 Dec 2019 16:30:25 -0500 Subject: [PATCH 1595/3965] ci: bump to go 1.13 --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index ba87d9a80a..a156d3eb5e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,7 +4,7 @@ os: language: go go: - - 1.12.x + - 1.13.x env: global: From 4b6e89e7d7f1616b924bd44b8fc8bb8b432957a8 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 6 Dec 2019 16:33:55 -0500 Subject: [PATCH 1596/3965] ci: bump to go 1.13 --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 5163d693fc..a156d3eb5e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,7 +4,7 @@ os: language: go go: - - 1.11.x + - 1.13.x env: global: From bea3a811a5b69e40094a28832144de255a9f56da Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Fri, 6 Dec 2019 21:56:09 +0000 Subject: [PATCH 1597/3965] build(deps): bump github.com/libp2p/go-libp2p-discovery Bumps [github.com/libp2p/go-libp2p-discovery](https://github.com/libp2p/go-libp2p-discovery) from 0.1.0 to 0.2.0. - [Release notes](https://github.com/libp2p/go-libp2p-discovery/releases) - [Commits](https://github.com/libp2p/go-libp2p-discovery/compare/v0.1.0...v0.2.0) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index e268ab13e1..880d9ec5d5 100644 --- a/go.mod +++ b/go.mod @@ -14,7 +14,7 @@ require ( github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 github.com/libp2p/go-libp2p-core v0.2.5 - github.com/libp2p/go-libp2p-discovery v0.1.0 + github.com/libp2p/go-libp2p-discovery v0.2.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 github.com/libp2p/go-libp2p-nat v0.0.5 diff --git a/go.sum b/go.sum index cd7811f8da..2e32148802 100644 --- a/go.sum +++ b/go.sum @@ -153,6 +153,8 @@ github.com/libp2p/go-libp2p-core v0.2.5/go.mod h1:6+5zJmKhsf7yHn1RbmYDu08qDUpIUx github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= +github.com/libp2p/go-libp2p-discovery v0.2.0 h1:1p3YSOq7VsgaL+xVHPi8XAmtGyas6D2J6rWBEfz/aiY= +github.com/libp2p/go-libp2p-discovery v0.2.0/go.mod h1:s4VGaxYMbw4+4+tsoQTqh7wfxg97AEdo4GYBt6BadWg= github.com/libp2p/go-libp2p-loggables v0.1.0 h1:h3w8QFfCt2UJl/0/NW4K829HX/0S4KD31PQ7m8UXXO8= github.com/libp2p/go-libp2p-loggables v0.1.0/go.mod h1:EyumB2Y6PrYjr55Q3/tiJ/o3xoDasoRYM7nOzEpoa90= github.com/libp2p/go-libp2p-mplex v0.2.0/go.mod h1:Ejl9IyjvXJ0T9iqUTE1jpYATQ9NM3g+OtR+EMMODbKo= From 3dade04a277560949ec1bb49a3181187240891cc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 6 Dec 2019 20:24:16 -0500 Subject: [PATCH 1598/3965] chore(dep): update --- .../conn-security-multistream/ssms_test.go | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/p2p/net/conn-security-multistream/ssms_test.go b/p2p/net/conn-security-multistream/ssms_test.go index c10830f0f5..43bf7be738 100644 --- a/p2p/net/conn-security-multistream/ssms_test.go +++ b/p2p/net/conn-security-multistream/ssms_test.go @@ -7,23 +7,31 @@ import ( "testing" "github.com/libp2p/go-libp2p-core/sec/insecure" + tnet "github.com/libp2p/go-libp2p-testing/net" sst "github.com/libp2p/go-libp2p-testing/suites/sec" ) func TestCommonProto(t *testing.T) { + idA := tnet.RandIdentityOrFatal(t) + idB := tnet.RandIdentityOrFatal(t) + var at, bt SSMuxer - atInsecure := insecure.New("peerA") - btInsecure := insecure.New("peerB") + + atInsecure := insecure.NewWithIdentity(idA.ID(), idA.PrivateKey()) + btInsecure := insecure.NewWithIdentity(idB.ID(), idB.PrivateKey()) at.AddTransport("/plaintext/1.0.0", atInsecure) bt.AddTransport("/plaintext/1.1.0", btInsecure) bt.AddTransport("/plaintext/1.0.0", btInsecure) - sst.SubtestRW(t, &at, &bt, "peerA", "peerB") + sst.SubtestRW(t, &at, &bt, idA.ID(), idB.ID()) } func TestNoCommonProto(t *testing.T) { + idA := tnet.RandIdentityOrFatal(t) + idB := tnet.RandIdentityOrFatal(t) + var at, bt SSMuxer - atInsecure := insecure.New("peerA") - btInsecure := insecure.New("peerB") + atInsecure := insecure.NewWithIdentity(idA.ID(), idA.PrivateKey()) + btInsecure := insecure.NewWithIdentity(idB.ID(), idB.PrivateKey()) at.AddTransport("/plaintext/1.0.0", atInsecure) bt.AddTransport("/plaintext/1.1.0", btInsecure) @@ -46,7 +54,7 @@ func TestNoCommonProto(t *testing.T) { go func() { defer wg.Done() defer b.Close() - _, err := bt.SecureOutbound(ctx, b, "peerA") + _, err := bt.SecureOutbound(ctx, b, idA.ID()) if err == nil { t.Fatal("connection should have failed") } From 04be62c1c8cb67e85cfc89947be906f8a7d11657 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 6 Dec 2019 21:22:27 -0500 Subject: [PATCH 1599/3965] chore: update deps And bump minimum key to 2048. --- p2p/security/tls/transport_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 94f2c215f0..62e9b0e73a 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -49,7 +49,7 @@ var _ = Describe("Transport", func() { priv, _, err = ci.GenerateECDSAKeyPair(rand.Reader) case 1: fmt.Fprintf(GinkgoWriter, " using an RSA key: ") - priv, _, err = ci.GenerateRSAKeyPair(1024, rand.Reader) + priv, _, err = ci.GenerateRSAKeyPair(2048, rand.Reader) case 2: fmt.Fprintf(GinkgoWriter, " using an Ed25519 key: ") priv, _, err = ci.GenerateEd25519Key(rand.Reader) @@ -192,7 +192,7 @@ var _ = Describe("Transport", func() { invalidateCertChain := func(identity *Identity) { switch identity.config.Certificates[0].PrivateKey.(type) { case *rsa.PrivateKey: - key, err := rsa.GenerateKey(rand.Reader, 1024) + key, err := rsa.GenerateKey(rand.Reader, 2048) Expect(err).ToNot(HaveOccurred()) identity.config.Certificates[0].PrivateKey = key case *ecdsa.PrivateKey: From 4fe79248eb89383a2328ebc3d96f04d67f5693aa Mon Sep 17 00:00:00 2001 From: Dmitry Yu Okunev Date: Tue, 2 Jul 2019 08:31:43 +0300 Subject: [PATCH 1600/3965] Use standard library "crypto/ed25519" --- core/crypto/ed25519.go | 3 +-- core/crypto/ed25519_test.go | 3 +-- core/crypto/key_not_openssl.go | 2 +- core/crypto/key_openssl.go | 2 +- core/crypto/key_test.go | 2 +- 5 files changed, 5 insertions(+), 7 deletions(-) diff --git a/core/crypto/ed25519.go b/core/crypto/ed25519.go index ceb3483798..1707e7509f 100644 --- a/core/crypto/ed25519.go +++ b/core/crypto/ed25519.go @@ -2,14 +2,13 @@ package crypto import ( "bytes" + "crypto/ed25519" "crypto/subtle" "errors" "fmt" "io" pb "github.com/libp2p/go-libp2p-core/crypto/pb" - - "golang.org/x/crypto/ed25519" ) // Ed25519PrivateKey is an ed25519 private key. diff --git a/core/crypto/ed25519_test.go b/core/crypto/ed25519_test.go index 222b5154b6..ae1bebe76e 100644 --- a/core/crypto/ed25519_test.go +++ b/core/crypto/ed25519_test.go @@ -1,12 +1,11 @@ package crypto import ( + "crypto/ed25519" "crypto/rand" "testing" pb "github.com/libp2p/go-libp2p-core/crypto/pb" - - "golang.org/x/crypto/ed25519" ) func TestBasicSignAndVerify(t *testing.T) { diff --git a/core/crypto/key_not_openssl.go b/core/crypto/key_not_openssl.go index ae8b01f00b..6b5b6fc2ce 100644 --- a/core/crypto/key_not_openssl.go +++ b/core/crypto/key_not_openssl.go @@ -5,10 +5,10 @@ package crypto import ( "crypto" "crypto/ecdsa" + "crypto/ed25519" "crypto/rsa" btcec "github.com/btcsuite/btcd/btcec" - "golang.org/x/crypto/ed25519" ) // KeyPairFromStdKey wraps standard library (and secp256k1) private keys in libp2p/go-libp2p-core/crypto keys diff --git a/core/crypto/key_openssl.go b/core/crypto/key_openssl.go index 4545492cf3..b948f3d081 100644 --- a/core/crypto/key_openssl.go +++ b/core/crypto/key_openssl.go @@ -5,12 +5,12 @@ package crypto import ( "crypto" "crypto/ecdsa" + "crypto/ed25519" "crypto/rsa" "crypto/x509" btcec "github.com/btcsuite/btcd/btcec" openssl "github.com/libp2p/go-openssl" - "golang.org/x/crypto/ed25519" ) // KeyPairFromStdKey wraps standard library (and secp256k1) private keys in libp2p/go-libp2p-core/crypto keys diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index aff2badd4c..984eaa1195 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -4,6 +4,7 @@ import ( "bytes" "crypto" "crypto/ecdsa" + "crypto/ed25519" "crypto/elliptic" "crypto/rand" "crypto/rsa" @@ -15,7 +16,6 @@ import ( pb "github.com/libp2p/go-libp2p-core/crypto/pb" "github.com/libp2p/go-libp2p-core/test" sha256 "github.com/minio/sha256-simd" - "golang.org/x/crypto/ed25519" ) func TestKeys(t *testing.T) { From f1d6c943086897ce319303f386a347900ca1d506 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 11 Dec 2019 10:38:57 +0100 Subject: [PATCH 1601/3965] feat: block on trim 1. Perform all trims in the background goroutine. 2. Make `TrimOpenConns` return early when canceled. 3. Make multiple concurrent calls to `TrimOpenConns` block until the trim finishes. Note: It already _may_ block one caller, this just ensures that it always behaves the same way. Returning a signal channel may be a nicer solution but this is less breaking. --- p2p/net/connmgr/connmgr.go | 90 +++++++++++++++++++++++++-------- p2p/net/connmgr/connmgr_test.go | 85 +++++++++++++++++++++++++++++-- 2 files changed, 149 insertions(+), 26 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 804730dea4..fb94fd8730 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -36,8 +36,9 @@ type BasicConnMgr struct { plk sync.RWMutex protected map[peer.ID]map[string]struct{} - // channel-based semaphore that enforces only a single trim is in progress - trimRunningCh chan struct{} + trimMu sync.RWMutex + trimTrigger chan struct{} + trimSignal chan struct{} lastTrim time.Time silencePeriod time.Duration @@ -96,7 +97,8 @@ func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { highWater: hi, lowWater: low, gracePeriod: grace, - trimRunningCh: make(chan struct{}, 1), + trimTrigger: make(chan struct{}), + trimSignal: make(chan struct{}), protected: make(map[peer.ID]map[string]struct{}, 16), silencePeriod: SilencePeriod, ctx: ctx, @@ -167,25 +169,28 @@ type peerInfo struct { // TODO: error return value so we can cleanly signal we are aborting because: // (a) there's another trim in progress, or (b) the silence period is in effect. func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { + cm.trimMu.RLock() + trimSignal := cm.trimSignal + cm.trimMu.RUnlock() + + // Trigger a trim. select { - case cm.trimRunningCh <- struct{}{}: - default: - return - } - defer func() { <-cm.trimRunningCh }() - if time.Since(cm.lastTrim) < cm.silencePeriod { - // skip this attempt to trim as the last one just took place. + case cm.trimTrigger <- struct{}{}: + case <-trimSignal: + // Someone else just trimmed. return + case <-cm.ctx.Done(): + case <-ctx.Done(): + // TODO: return an error? } - defer log.EventBegin(ctx, "connCleanup").Done() - for _, c := range cm.getConnsToClose(ctx) { - log.Info("closing conn: ", c.RemotePeer()) - log.Event(ctx, "closeConn", c.RemotePeer()) - c.Close() + // Wait for the trim. + select { + case <-trimSignal: + case <-cm.ctx.Done(): + case <-ctx.Done(): + // TODO: return an error? } - - cm.lastTrim = time.Now() } func (cm *BasicConnMgr) background() { @@ -195,19 +200,56 @@ func (cm *BasicConnMgr) background() { for { select { case <-ticker.C: - if atomic.LoadInt32(&cm.connCount) > int32(cm.highWater) { - cm.TrimOpenConns(cm.ctx) + if atomic.LoadInt32(&cm.connCount) < int32(cm.highWater) { + // Below high water, skip. + continue } - + case <-cm.trimTrigger: case <-cm.ctx.Done(): return } + cm.trim() + } +} + +func (cm *BasicConnMgr) trim() { + cm.trimMu.Lock() + + // read the last trim time under the lock + lastTrim := cm.lastTrim + + // swap out the trim signal + trimSignal := cm.trimSignal + cm.trimSignal = make(chan struct{}) + + cm.trimMu.Unlock() + + // always signal a trim, even if we _don't_ trim, to unblock anyone + // waiting. + defer close(trimSignal) + + // skip this attempt to trim if the last one just took place. + if time.Since(lastTrim) < cm.silencePeriod { + return } + + // do the actual trim. + defer log.EventBegin(cm.ctx, "connCleanup").Done() + for _, c := range cm.getConnsToClose() { + log.Info("closing conn: ", c.RemotePeer()) + log.Event(cm.ctx, "closeConn", c.RemotePeer()) + c.Close() + } + + // finally, update the last trim time. + cm.trimMu.Lock() + cm.lastTrim = time.Now() + cm.trimMu.Unlock() } // getConnsToClose runs the heuristics described in TrimOpenConns and returns the // connections to close. -func (cm *BasicConnMgr) getConnsToClose(ctx context.Context) []network.Conn { +func (cm *BasicConnMgr) getConnsToClose() []network.Conn { if cm.lowWater == 0 || cm.highWater == 0 { // disabled return nil @@ -386,10 +428,14 @@ type CMInfo struct { // GetInfo returns the configuration and status data for this connection manager. func (cm *BasicConnMgr) GetInfo() CMInfo { + cm.trimMu.RLock() + lastTrim := cm.lastTrim + cm.trimMu.RUnlock() + return CMInfo{ HighWater: cm.highWater, LowWater: cm.lowWater, - LastTrim: cm.lastTrim, + LastTrim: lastTrim, GracePeriod: cm.gracePeriod, ConnCount: int(atomic.LoadInt32(&cm.connCount)), } diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index ad787864c7..705fa7721c 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -2,6 +2,7 @@ package connmgr import ( "context" + "sync" "testing" "time" @@ -47,6 +48,82 @@ func randConn(t testing.TB, discNotify func(network.Network, network.Conn)) netw return &tconn{peer: pid, disconnectNotify: discNotify} } +// Make sure multiple trim calls block. +func TestTrimBlocks(t *testing.T) { + cm := NewConnManager(200, 300, 0) + + cm.trimMu.RLock() + + doneCh := make(chan struct{}, 2) + go func() { + cm.TrimOpenConns(context.Background()) + doneCh <- struct{}{} + }() + go func() { + cm.TrimOpenConns(context.Background()) + doneCh <- struct{}{} + }() + time.Sleep(time.Millisecond) + select { + case <-doneCh: + cm.trimMu.RUnlock() + t.Fatal("expected trim to block") + default: + cm.trimMu.RUnlock() + } + <-doneCh + <-doneCh +} + +// Make sure we return from trim when the context is canceled. +func TestTrimCancels(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cm := NewConnManager(200, 300, 0) + + cm.trimMu.RLock() + defer cm.trimMu.RUnlock() + + doneCh := make(chan struct{}) + go func() { + defer close(doneCh) + cm.TrimOpenConns(ctx) + }() + time.Sleep(time.Millisecond) + cancel() + <-doneCh +} + +// Make sure trim returns when closed. +func TestTrimClosed(t *testing.T) { + cm := NewConnManager(200, 300, 0) + cm.Close() + cm.TrimOpenConns(context.Background()) +} + +// Make sure joining an existing trim works. +func TestTrimJoin(t *testing.T) { + cm := NewConnManager(200, 300, 0) + cm.trimMu.RLock() + var wg sync.WaitGroup + wg.Add(3) + go func() { + defer wg.Done() + cm.TrimOpenConns(context.Background()) + }() + time.Sleep(time.Millisecond) + go func() { + defer wg.Done() + cm.TrimOpenConns(context.Background()) + }() + go func() { + defer wg.Done() + cm.TrimOpenConns(context.Background()) + }() + time.Sleep(time.Millisecond) + cm.trimMu.RUnlock() + wg.Wait() +} + func TestConnTrimming(t *testing.T) { cm := NewConnManager(200, 300, 0) not := cm.Notifee() @@ -86,19 +163,19 @@ func TestConnTrimming(t *testing.T) { func TestConnsToClose(t *testing.T) { cm := NewConnManager(0, 10, 0) - conns := cm.getConnsToClose(context.Background()) + conns := cm.getConnsToClose() if conns != nil { t.Fatal("expected no connections") } cm = NewConnManager(10, 0, 0) - conns = cm.getConnsToClose(context.Background()) + conns = cm.getConnsToClose() if conns != nil { t.Fatal("expected no connections") } cm = NewConnManager(1, 1, 0) - conns = cm.getConnsToClose(context.Background()) + conns = cm.getConnsToClose() if conns != nil { t.Fatal("expected no connections") } @@ -109,7 +186,7 @@ func TestConnsToClose(t *testing.T) { conn := randConn(t, nil) not.Connected(nil, conn) } - conns = cm.getConnsToClose(context.Background()) + conns = cm.getConnsToClose() if len(conns) != 0 { t.Fatal("expected no connections") } From d347a08e6a833bbd2642fa24fdd1c2225eb3933e Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Wed, 11 Dec 2019 11:35:57 +0000 Subject: [PATCH 1602/3965] build(deps): bump github.com/multiformats/go-multiaddr Bumps [github.com/multiformats/go-multiaddr](https://github.com/multiformats/go-multiaddr) from 0.1.2 to 0.2.0. - [Release notes](https://github.com/multiformats/go-multiaddr/releases) - [Commits](https://github.com/multiformats/go-multiaddr/compare/v0.1.2...v0.2.0) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 880d9ec5d5..7a6a5a501a 100644 --- a/go.mod +++ b/go.mod @@ -30,7 +30,7 @@ require ( github.com/libp2p/go-tcp-transport v0.1.1 github.com/libp2p/go-ws-transport v0.1.2 github.com/miekg/dns v1.1.12 // indirect - github.com/multiformats/go-multiaddr v0.1.2 + github.com/multiformats/go-multiaddr v0.2.0 github.com/multiformats/go-multiaddr-dns v0.2.0 github.com/multiformats/go-multiaddr-net v0.1.1 github.com/multiformats/go-multistream v0.1.0 diff --git a/go.sum b/go.sum index 2e32148802..b4de423660 100644 --- a/go.sum +++ b/go.sum @@ -263,6 +263,8 @@ github.com/multiformats/go-multiaddr v0.1.1 h1:rVAztJYMhCQ7vEFr8FvxW3mS+HF2eY/oP github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo= github.com/multiformats/go-multiaddr v0.1.2 h1:HWYHNSyyllbQopmVIF5K7JKJugiah+L9/kuZKHbmNdQ= github.com/multiformats/go-multiaddr v0.1.2/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4= +github.com/multiformats/go-multiaddr v0.2.0 h1:lR52sFwcTCuQb6bTfnXF6zA2XfyYvyd+5a9qECv/J90= +github.com/multiformats/go-multiaddr v0.2.0/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= From 99ddcd1293d7ba2a7e46edc2a8424ac92741a662 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 11 Dec 2019 14:50:07 +0100 Subject: [PATCH 1603/3965] fix: simplify trim logic Pro: It's a lot simpler. Con: We lose the ordering guarantee. Before, TrimOpenConns would only return once a trim that started _after_ calling TrimOpenConns finished. Now, it returns as soon as it observes _any_ trim finishing (even if the trim started earlier). --- p2p/net/connmgr/connmgr.go | 66 +++++++++++++++++---------------- p2p/net/connmgr/connmgr_test.go | 14 +++---- 2 files changed, 41 insertions(+), 39 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index fb94fd8730..3de578934e 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -36,10 +36,11 @@ type BasicConnMgr struct { plk sync.RWMutex protected map[peer.ID]map[string]struct{} - trimMu sync.RWMutex - trimTrigger chan struct{} - trimSignal chan struct{} - lastTrim time.Time + trimTrigger chan chan<- struct{} + + lastTrimMu sync.RWMutex + lastTrim time.Time + silencePeriod time.Duration ctx context.Context @@ -97,8 +98,7 @@ func NewConnManager(low, hi int, grace time.Duration) *BasicConnMgr { highWater: hi, lowWater: low, gracePeriod: grace, - trimTrigger: make(chan struct{}), - trimSignal: make(chan struct{}), + trimTrigger: make(chan chan<- struct{}), protected: make(map[peer.ID]map[string]struct{}, 16), silencePeriod: SilencePeriod, ctx: ctx, @@ -169,16 +169,10 @@ type peerInfo struct { // TODO: error return value so we can cleanly signal we are aborting because: // (a) there's another trim in progress, or (b) the silence period is in effect. func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { - cm.trimMu.RLock() - trimSignal := cm.trimSignal - cm.trimMu.RUnlock() - // Trigger a trim. + ch := make(chan struct{}) select { - case cm.trimTrigger <- struct{}{}: - case <-trimSignal: - // Someone else just trimmed. - return + case cm.trimTrigger <- ch: case <-cm.ctx.Done(): case <-ctx.Done(): // TODO: return an error? @@ -186,7 +180,7 @@ func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { // Wait for the trim. select { - case <-trimSignal: + case <-ch: case <-cm.ctx.Done(): case <-ctx.Done(): // TODO: return an error? @@ -198,35 +192,43 @@ func (cm *BasicConnMgr) background() { defer ticker.Stop() for { + var waiting chan<- struct{} select { case <-ticker.C: if atomic.LoadInt32(&cm.connCount) < int32(cm.highWater) { // Below high water, skip. continue } - case <-cm.trimTrigger: + case waiting = <-cm.trimTrigger: case <-cm.ctx.Done(): return } cm.trim() + + // Notify anyone waiting on this trim. + if waiting != nil { + close(waiting) + } + + for { + select { + case waiting = <-cm.trimTrigger: + if waiting != nil { + close(waiting) + } + continue + default: + } + break + } } } func (cm *BasicConnMgr) trim() { - cm.trimMu.Lock() - + cm.lastTrimMu.RLock() // read the last trim time under the lock lastTrim := cm.lastTrim - - // swap out the trim signal - trimSignal := cm.trimSignal - cm.trimSignal = make(chan struct{}) - - cm.trimMu.Unlock() - - // always signal a trim, even if we _don't_ trim, to unblock anyone - // waiting. - defer close(trimSignal) + cm.lastTrimMu.RUnlock() // skip this attempt to trim if the last one just took place. if time.Since(lastTrim) < cm.silencePeriod { @@ -242,9 +244,9 @@ func (cm *BasicConnMgr) trim() { } // finally, update the last trim time. - cm.trimMu.Lock() + cm.lastTrimMu.Lock() cm.lastTrim = time.Now() - cm.trimMu.Unlock() + cm.lastTrimMu.Unlock() } // getConnsToClose runs the heuristics described in TrimOpenConns and returns the @@ -428,9 +430,9 @@ type CMInfo struct { // GetInfo returns the configuration and status data for this connection manager. func (cm *BasicConnMgr) GetInfo() CMInfo { - cm.trimMu.RLock() + cm.lastTrimMu.RLock() lastTrim := cm.lastTrim - cm.trimMu.RUnlock() + cm.lastTrimMu.RUnlock() return CMInfo{ HighWater: cm.highWater, diff --git a/p2p/net/connmgr/connmgr_test.go b/p2p/net/connmgr/connmgr_test.go index 705fa7721c..8e1b4b15d7 100644 --- a/p2p/net/connmgr/connmgr_test.go +++ b/p2p/net/connmgr/connmgr_test.go @@ -52,7 +52,7 @@ func randConn(t testing.TB, discNotify func(network.Network, network.Conn)) netw func TestTrimBlocks(t *testing.T) { cm := NewConnManager(200, 300, 0) - cm.trimMu.RLock() + cm.lastTrimMu.RLock() doneCh := make(chan struct{}, 2) go func() { @@ -66,10 +66,10 @@ func TestTrimBlocks(t *testing.T) { time.Sleep(time.Millisecond) select { case <-doneCh: - cm.trimMu.RUnlock() + cm.lastTrimMu.RUnlock() t.Fatal("expected trim to block") default: - cm.trimMu.RUnlock() + cm.lastTrimMu.RUnlock() } <-doneCh <-doneCh @@ -80,8 +80,8 @@ func TestTrimCancels(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) cm := NewConnManager(200, 300, 0) - cm.trimMu.RLock() - defer cm.trimMu.RUnlock() + cm.lastTrimMu.RLock() + defer cm.lastTrimMu.RUnlock() doneCh := make(chan struct{}) go func() { @@ -103,7 +103,7 @@ func TestTrimClosed(t *testing.T) { // Make sure joining an existing trim works. func TestTrimJoin(t *testing.T) { cm := NewConnManager(200, 300, 0) - cm.trimMu.RLock() + cm.lastTrimMu.RLock() var wg sync.WaitGroup wg.Add(3) go func() { @@ -120,7 +120,7 @@ func TestTrimJoin(t *testing.T) { cm.TrimOpenConns(context.Background()) }() time.Sleep(time.Millisecond) - cm.trimMu.RUnlock() + cm.lastTrimMu.RUnlock() wg.Wait() } From 936762820afc0fd2252e758a515aa7c0ca187981 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 11 Dec 2019 15:26:38 +0100 Subject: [PATCH 1604/3965] docs: document TrimOpenConns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Raúl Kripalani --- p2p/net/connmgr/connmgr.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/net/connmgr/connmgr.go b/p2p/net/connmgr/connmgr.go index 3de578934e..b385533a88 100644 --- a/p2p/net/connmgr/connmgr.go +++ b/p2p/net/connmgr/connmgr.go @@ -166,9 +166,13 @@ type peerInfo struct { // pruning those peers with the lowest scores first, as long as they are not within their // grace period. // -// TODO: error return value so we can cleanly signal we are aborting because: -// (a) there's another trim in progress, or (b) the silence period is in effect. +// This function blocks until a trim is completed. If a trim is underway, a new +// one won't be started, and instead it'll wait until that one is completed before +// returning. func (cm *BasicConnMgr) TrimOpenConns(ctx context.Context) { + // TODO: error return value so we can cleanly signal we are aborting because: + // (a) there's another trim in progress, or (b) the silence period is in effect. + // Trigger a trim. ch := make(chan struct{}) select { From 8349e509431822bc40423452589dbcdedd08f1c0 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 11 Dec 2019 16:18:19 -0500 Subject: [PATCH 1605/3965] fix protocol id string --- p2p/security/noise/transport.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 3bb6ee13b1..0a97e546bc 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -13,7 +13,7 @@ import ( ) // ID is the protocol ID for noise -const ID = "/noise/0.0.1" +const ID = "/noise" var _ sec.SecureTransport = &Transport{} From ca63ecd6bc59c27067341c659267cbb2542a8526 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 11 Dec 2019 17:25:40 -0500 Subject: [PATCH 1606/3965] use Options to configure transport also, adds NewTransportGenerator for cases when you don't know the libp2p host key when configuring the Noise transport --- p2p/security/noise/integration_test.go | 24 ++++++--- p2p/security/noise/options.go | 44 +++++++++++++++++ p2p/security/noise/transport.go | 67 +++++++++++++++++++++++--- 3 files changed, 122 insertions(+), 13 deletions(-) create mode 100644 p2p/security/noise/options.go diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index f43a28355d..c67b6d01a4 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -5,7 +5,7 @@ import ( "context" "crypto/rand" "fmt" - libp2p "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/host" net "github.com/libp2p/go-libp2p-core/network" @@ -39,13 +39,10 @@ func makeNode(t *testing.T, seed int64, port int, kp *Keypair) (host.Host, error t.Fatal(err) } - pid, err := peer.IDFromPrivateKey(priv) + tpt, err := NewTransport(priv, NoiseKeyPair(kp)) if err != nil { t.Fatal(err) } - - tpt := NewTransport(pid, priv, false, kp) - ip := "0.0.0.0" addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", ip, port)) if err != nil { @@ -69,12 +66,11 @@ func makeNodePipes(t *testing.T, seed int64, port int, rpid peer.ID, rpubkey [32 t.Fatal(err) } - pid, err := peer.IDFromPrivateKey(priv) + tpt, err := NewTransport(priv, UseNoisePipes, NoiseKeyPair(kp)) if err != nil { t.Fatal(err) } - tpt := NewTransport(pid, priv, true, kp) tpt.NoiseStaticKeyCache = NewKeyCache() tpt.NoiseStaticKeyCache.Store(rpid, rpubkey) @@ -256,6 +252,20 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { time.Sleep(time.Second) } +func TestNewTransportGenerator(t *testing.T) { + kp := GenerateKeypair() + + ctx := context.Background() + h, err := libp2p.New(ctx, + libp2p.Security(ID, + NewTransportConstructor(NoiseKeyPair(kp), UseNoisePipes))) + + if err != nil { + t.Fatalf("unable to create libp2p host with NewTransportConstructor: %v", err) + } + _ = h.Close() +} + func handleStream(stream net.Stream) { defer func() { if err := stream.Close(); err != nil { diff --git a/p2p/security/noise/options.go b/p2p/security/noise/options.go new file mode 100644 index 0000000000..e4a0c155a9 --- /dev/null +++ b/p2p/security/noise/options.go @@ -0,0 +1,44 @@ +package noise + +// UseNoisePipes configures the Noise transport to use the Noise Pipes pattern. +// Noise Pipes attempts to use the more efficient IK handshake pattern when +// dialing a remote peer, if that peer's static Noise key is known. If this +// is unsuccessful, the transport will fallback to using the default XX pattern. +// +// Note that the fallback does not add any additional round-trips vs. simply +// using XX in the first place, however there is a slight processing overhead +// due to the initial decryption attempt of the IK message. +func UseNoisePipes(cfg *config) { + cfg.NoisePipesSupport = true +} + +// NoiseKeyPair configures the Noise transport to use the given Noise static +// keypair. This is distinct from the libp2p Host's identity keypair and is +// used only for Noise. If this option is not provided, a new Noise static +// keypair will be generated when the transport is initialized. +// +// This option is most useful when Noise Pipes is enabled, as longer static +// key lifetimes may lead to more successful IK handshake attempts. +// +// If you do use this option with a key that's been saved to disk, you must +// take care to store the key securely! +func NoiseKeyPair(kp *Keypair) Option { + return func(cfg *config) { + cfg.NoiseKeypair = kp + } +} + +type config struct { + NoiseKeypair *Keypair + NoisePipesSupport bool +} + +type Option func(cfg *config) + +func (cfg *config) applyOptions(opts ...Option) { + for _, opt := range opts { + if opt != nil { + opt(cfg) + } + } +} diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 0a97e546bc..abbce66b75 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -42,26 +42,81 @@ type Transport struct { NoiseKeypair *Keypair } -// NewTransport creates a new noise transport and can be configured to use noise pipes and a given -// noise ed25519 keypair -func NewTransport(localID peer.ID, privkey crypto.PrivKey, noisePipesSupport bool, kp *Keypair) *Transport { +type transportConstructor func(crypto.PrivKey) (*Transport, error) + +// NewTransportConstructor returns a function that will construct a new Noise transport +// using the given Options. The returned function may be provided as a libp2p.Security +// option when configuring a libp2p Host using libp2p.New: +// +// host := libp2p.New( +// libp2p.Security(noise.ID, noise.NewTransportConstructor())) +// +// The transport can be configured by passing in Options. +// +// To enable the Noise Pipes pattern (which can be more efficient when reconnecting +// to a known peer), pass in the UseNoisePipes Option: +// +// NewTransportConstructor(UseNoisePipes) +// +// To use a specific Noise keypair, pass in the NoiseKeyPair(kp) option, where +// kp is a noise.Keypair struct. This is most useful when using Noise Pipes, whose +// efficiency gains rely on the static Noise key being known in advance. Persisting +// the Noise keypair across process restarts makes it more likely that other peers +// will be able to use the more efficient IK handshake pattern. +// +// NewTransportConstructor(UseNoisePipes, NoiseKeypair(keypairLoadedFromDisk)) +func NewTransportConstructor(options ...Option) transportConstructor { + return func(privKey crypto.PrivKey) (*Transport, error) { + return NewTransport(privKey, options...) + } +} + +// NewTransport creates a new Noise transport using the given private key as its +// libp2p identity key. This function may be used when you want a transport +// instance and know the libp2p Host's identity key before the Host is initialized. +// When configuring a go-libp2p Host using libp2p.New, it's simpler to use +// NewTransportConstructor instead, which will receive the identity key when the Host +// is initialized. +// +// NewTransport supports all the same Options as NewTransportConstructor. +// +// To configure a go-libp2p Host to use the newly created transport, pass it into +// libp2p.New wrapped in a libp2p.Security Option. You will also need to +// make sure to set the libp2p.Identity option so that the Host uses the same +// identity key: +// +// privkey := loadPrivateKeyFromSomewhere() +// noiseTpt := noise.NewTransport(privkey) +// host := libp2p.New( +// libp2p.Identity(privkey), +// libp2p.Security(noise.ID, noiseTpt)) +func NewTransport(privkey crypto.PrivKey, options ...Option) (*Transport, error) { + localID, err := peer.IDFromPrivateKey(privkey) + if err != nil { + return nil, err + } + + cfg := config{} + cfg.applyOptions(options...) + + kp := cfg.NoiseKeypair if kp == nil { kp = GenerateKeypair() } // the static key cache is only useful if Noise Pipes is enabled var keyCache *KeyCache - if noisePipesSupport { + if cfg.NoisePipesSupport { keyCache = NewKeyCache() } return &Transport{ LocalID: localID, PrivateKey: privkey, - NoisePipesSupport: noisePipesSupport, + NoisePipesSupport: cfg.NoisePipesSupport, NoiseKeypair: kp, NoiseStaticKeyCache: keyCache, - } + }, nil } // SecureInbound runs noise handshake as the responder From 8ab00e7201b2e26340e1d8772a610e7a2bb200cb Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 11 Dec 2019 17:21:58 -0500 Subject: [PATCH 1607/3965] rename constructors --- p2p/security/noise/integration_test.go | 10 +++++----- p2p/security/noise/transport.go | 25 +++++++++++++------------ 2 files changed, 18 insertions(+), 17 deletions(-) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index c67b6d01a4..87b43e0de4 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -39,7 +39,7 @@ func makeNode(t *testing.T, seed int64, port int, kp *Keypair) (host.Host, error t.Fatal(err) } - tpt, err := NewTransport(priv, NoiseKeyPair(kp)) + tpt, err := New(priv, NoiseKeyPair(kp)) if err != nil { t.Fatal(err) } @@ -66,7 +66,7 @@ func makeNodePipes(t *testing.T, seed int64, port int, rpid peer.ID, rpubkey [32 t.Fatal(err) } - tpt, err := NewTransport(priv, UseNoisePipes, NoiseKeyPair(kp)) + tpt, err := New(priv, UseNoisePipes, NoiseKeyPair(kp)) if err != nil { t.Fatal(err) } @@ -252,16 +252,16 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { time.Sleep(time.Second) } -func TestNewTransportGenerator(t *testing.T) { +func TestConstrucingWithMaker(t *testing.T) { kp := GenerateKeypair() ctx := context.Background() h, err := libp2p.New(ctx, libp2p.Security(ID, - NewTransportConstructor(NoiseKeyPair(kp), UseNoisePipes))) + Maker(NoiseKeyPair(kp), UseNoisePipes))) if err != nil { - t.Fatalf("unable to create libp2p host with NewTransportConstructor: %v", err) + t.Fatalf("unable to create libp2p host with Maker: %v", err) } _ = h.Close() } diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index abbce66b75..38656aa471 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -44,19 +44,20 @@ type Transport struct { type transportConstructor func(crypto.PrivKey) (*Transport, error) -// NewTransportConstructor returns a function that will construct a new Noise transport +// Maker returns a function that will construct a new Noise transport // using the given Options. The returned function may be provided as a libp2p.Security -// option when configuring a libp2p Host using libp2p.New: +// option when configuring a libp2p Host using libp2p.New, and is compatible with the +// "reflection magic" that libp2p.New uses to inject the private identity key: // // host := libp2p.New( -// libp2p.Security(noise.ID, noise.NewTransportConstructor())) +// libp2p.Security(noise.ID, noise.Maker())) // // The transport can be configured by passing in Options. // // To enable the Noise Pipes pattern (which can be more efficient when reconnecting // to a known peer), pass in the UseNoisePipes Option: // -// NewTransportConstructor(UseNoisePipes) +// Maker(UseNoisePipes) // // To use a specific Noise keypair, pass in the NoiseKeyPair(kp) option, where // kp is a noise.Keypair struct. This is most useful when using Noise Pipes, whose @@ -64,21 +65,21 @@ type transportConstructor func(crypto.PrivKey) (*Transport, error) // the Noise keypair across process restarts makes it more likely that other peers // will be able to use the more efficient IK handshake pattern. // -// NewTransportConstructor(UseNoisePipes, NoiseKeypair(keypairLoadedFromDisk)) -func NewTransportConstructor(options ...Option) transportConstructor { +// Maker(UseNoisePipes, NoiseKeypair(keypairLoadedFromDisk)) +func Maker(options ...Option) transportConstructor { return func(privKey crypto.PrivKey) (*Transport, error) { - return NewTransport(privKey, options...) + return New(privKey, options...) } } -// NewTransport creates a new Noise transport using the given private key as its +// New creates a new Noise transport using the given private key as its // libp2p identity key. This function may be used when you want a transport // instance and know the libp2p Host's identity key before the Host is initialized. // When configuring a go-libp2p Host using libp2p.New, it's simpler to use -// NewTransportConstructor instead, which will receive the identity key when the Host +// Maker instead, which will receive the identity key when the Host // is initialized. // -// NewTransport supports all the same Options as NewTransportConstructor. +// New supports all the same Options as noise.Maker. // // To configure a go-libp2p Host to use the newly created transport, pass it into // libp2p.New wrapped in a libp2p.Security Option. You will also need to @@ -86,11 +87,11 @@ func NewTransportConstructor(options ...Option) transportConstructor { // identity key: // // privkey := loadPrivateKeyFromSomewhere() -// noiseTpt := noise.NewTransport(privkey) +// noiseTpt := noise.New(privkey) // host := libp2p.New( // libp2p.Identity(privkey), // libp2p.Security(noise.ID, noiseTpt)) -func NewTransport(privkey crypto.PrivKey, options ...Option) (*Transport, error) { +func New(privkey crypto.PrivKey, options ...Option) (*Transport, error) { localID, err := peer.IDFromPrivateKey(privkey) if err != nil { return nil, err From c67695a72e29166968bc010cd32ea400e04ace60 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 11 Dec 2019 19:29:39 -0500 Subject: [PATCH 1608/3965] don't swallow errors from rand.Read --- p2p/security/noise/integration_test.go | 15 ++++++++++++--- p2p/security/noise/protocol.go | 6 +++++- p2p/security/noise/transport.go | 18 ++++++++++++++---- p2p/security/noise/transport_test.go | 8 ++++++-- 4 files changed, 37 insertions(+), 10 deletions(-) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index 87b43e0de4..c2a9d034aa 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -147,7 +147,10 @@ func TestLibp2pIntegration_NoPipes(t *testing.T) { func TestLibp2pIntegration_WithPipes(t *testing.T) { ctx := context.Background() - kpa := GenerateKeypair() + kpa, err := GenerateKeypair() + if err != nil { + t.Fatal(err) + } ha, err := makeNodePipes(t, 1, 33333, "", [32]byte{}, kpa) if err != nil { @@ -201,7 +204,10 @@ func TestLibp2pIntegration_WithPipes(t *testing.T) { func TestLibp2pIntegration_XXFallback(t *testing.T) { ctx := context.Background() - kpa := GenerateKeypair() + kpa, err := GenerateKeypair() + if err != nil { + t.Fatal(err) + } ha, err := makeNode(t, 1, 33333, kpa) if err != nil { @@ -253,7 +259,10 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { } func TestConstrucingWithMaker(t *testing.T) { - kp := GenerateKeypair() + kp, err := GenerateKeypair() + if err != nil { + t.Fatal(err) + } ctx := context.Background() h, err := libp2p.New(ctx, diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 351d42ed31..d0773371e0 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -75,7 +75,11 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey } if kp == nil { - kp = GenerateKeypair() + var err error + kp, err = GenerateKeypair() + if err != nil { + return nil, err + } } localPeerInfo := peerInfo{ diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 38656aa471..770a18df93 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -3,6 +3,7 @@ package noise import ( "context" "crypto/rand" + "fmt" "net" "golang.org/x/crypto/curve25519" @@ -24,12 +25,18 @@ type Keypair struct { } // GenerateKeypair creates a new ed25519 keypair -func GenerateKeypair() *Keypair { +func GenerateKeypair() (*Keypair, error) { var public_key [32]byte var private_key [32]byte - _, _ = rand.Read(private_key[:]) + c, err := rand.Read(private_key[:]) + if err != nil { + return nil, err + } + if c != 32 { + return nil, fmt.Errorf("unable to generate 32 bytes of random data, got %d", c) + } curve25519.ScalarBaseMult(&public_key, &private_key) - return &Keypair{public_key, private_key} + return &Keypair{public_key, private_key}, nil } // Transport implements the interface sec.SecureTransport @@ -102,7 +109,10 @@ func New(privkey crypto.PrivKey, options ...Option) (*Transport, error) { kp := cfg.NoiseKeypair if kp == nil { - kp = GenerateKeypair() + kp, err = GenerateKeypair() + if err != nil { + return nil, err + } } // the static key cache is only useful if Noise Pipes is enabled diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index b8c8957643..8dba93b0d1 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -219,7 +219,11 @@ func TestHandshakeIK(t *testing.T) { respTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) // add responder's static key to initiator's key cache - respTransport.NoiseKeypair = GenerateKeypair() + kp, err := GenerateKeypair() + if err != nil { + t.Fatal(err) + } + respTransport.NoiseKeypair = kp keycache := NewKeyCache() keycache.Store(respTransport.LocalID, respTransport.NoiseKeypair.public_key) initTransport.NoiseStaticKeyCache = keycache @@ -230,7 +234,7 @@ func TestHandshakeIK(t *testing.T) { defer respConn.Close() before := []byte("hello world") - _, err := initConn.Write(before) + _, err = initConn.Write(before) if err != nil { t.Fatal(err) } From 8a5063a99d29d5ae999bb3c386126d838ce057de Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 12 Dec 2019 15:51:51 -0500 Subject: [PATCH 1609/3965] rm redundant check for num bytes read --- p2p/security/noise/transport.go | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 770a18df93..275d114b2c 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -3,7 +3,6 @@ package noise import ( "context" "crypto/rand" - "fmt" "net" "golang.org/x/crypto/curve25519" @@ -28,13 +27,10 @@ type Keypair struct { func GenerateKeypair() (*Keypair, error) { var public_key [32]byte var private_key [32]byte - c, err := rand.Read(private_key[:]) + _, err := rand.Read(private_key[:]) if err != nil { return nil, err } - if c != 32 { - return nil, fmt.Errorf("unable to generate 32 bytes of random data, got %d", c) - } curve25519.ScalarBaseMult(&public_key, &private_key) return &Keypair{public_key, private_key}, nil } From f360c666bd4018ac150c1479eace62115c74378c Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 12 Dec 2019 16:23:17 -0500 Subject: [PATCH 1610/3965] move Keypair to its own file --- p2p/security/noise/keys.go | 24 ++++++++++++++++++++++++ p2p/security/noise/transport.go | 21 --------------------- 2 files changed, 24 insertions(+), 21 deletions(-) create mode 100644 p2p/security/noise/keys.go diff --git a/p2p/security/noise/keys.go b/p2p/security/noise/keys.go new file mode 100644 index 0000000000..f9a01f36b1 --- /dev/null +++ b/p2p/security/noise/keys.go @@ -0,0 +1,24 @@ +package noise + +import ( + "crypto/rand" + "golang.org/x/crypto/curve25519" +) + +// Keypair is a noise ed25519 public-private keypair +type Keypair struct { + public_key [32]byte + private_key [32]byte +} + +// GenerateKeypair creates a new ed25519 keypair +func GenerateKeypair() (*Keypair, error) { + var public_key [32]byte + var private_key [32]byte + _, err := rand.Read(private_key[:]) + if err != nil { + return nil, err + } + curve25519.ScalarBaseMult(&public_key, &private_key) + return &Keypair{public_key, private_key}, nil +} diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 275d114b2c..834f307200 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -2,11 +2,8 @@ package noise import ( "context" - "crypto/rand" "net" - "golang.org/x/crypto/curve25519" - "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" @@ -17,24 +14,6 @@ const ID = "/noise" var _ sec.SecureTransport = &Transport{} -// Keypair is a noise ed25519 public-private keypair -type Keypair struct { - public_key [32]byte - private_key [32]byte -} - -// GenerateKeypair creates a new ed25519 keypair -func GenerateKeypair() (*Keypair, error) { - var public_key [32]byte - var private_key [32]byte - _, err := rand.Read(private_key[:]) - if err != nil { - return nil, err - } - curve25519.ScalarBaseMult(&public_key, &private_key) - return &Keypair{public_key, private_key}, nil -} - // Transport implements the interface sec.SecureTransport // https://godoc.org/github.com/libp2p/go-libp2p-core/sec#SecureConn type Transport struct { From ff835a9cd9586a93573a67ca83243845b570ae2b Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Thu, 12 Dec 2019 16:23:49 -0500 Subject: [PATCH 1611/3965] camelCase field names --- p2p/security/noise/ik_handshake.go | 2 +- p2p/security/noise/integration_test.go | 4 ++-- p2p/security/noise/keys.go | 14 +++++++------- p2p/security/noise/protocol.go | 8 ++++---- p2p/security/noise/transport_test.go | 2 +- p2p/security/noise/xx_handshake.go | 2 +- 6 files changed, 16 insertions(+), 16 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index df9567d912..337b37d8b5 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -80,7 +80,7 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, // <- e, ee, se // returns last successful message upon error func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([]byte, error) { - kp := ik.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) + kp := ik.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) log.Debugf("runHandshake_ik initiator=%v pubkey=%x", kp.PubKey(), s.initiator) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index c2a9d034aa..3f3f52c4b0 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -159,7 +159,7 @@ func TestLibp2pIntegration_WithPipes(t *testing.T) { defer ha.Close() - hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.public_key, nil) + hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.publicKey, nil) if err != nil { t.Fatal(err) } @@ -216,7 +216,7 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { defer ha.Close() - hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.public_key, nil) + hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.publicKey, nil) if err != nil { t.Fatal(err) } diff --git a/p2p/security/noise/keys.go b/p2p/security/noise/keys.go index f9a01f36b1..e14e216a01 100644 --- a/p2p/security/noise/keys.go +++ b/p2p/security/noise/keys.go @@ -7,18 +7,18 @@ import ( // Keypair is a noise ed25519 public-private keypair type Keypair struct { - public_key [32]byte - private_key [32]byte + publicKey [32]byte + privateKey [32]byte } // GenerateKeypair creates a new ed25519 keypair func GenerateKeypair() (*Keypair, error) { - var public_key [32]byte - var private_key [32]byte - _, err := rand.Read(private_key[:]) + var publicKey [32]byte + var privateKey [32]byte + _, err := rand.Read(privateKey[:]) if err != nil { return nil, err } - curve25519.ScalarBaseMult(&public_key, &private_key) - return &Keypair{public_key, private_key}, nil + curve25519.ScalarBaseMult(&publicKey, &privateKey) + return &Keypair{publicKey, privateKey}, nil } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index d0773371e0..073e506b15 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -83,7 +83,7 @@ func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey } localPeerInfo := peerInfo{ - noiseKey: kp.public_key, + noiseKey: kp.publicKey, libp2pKey: privKey.GetPublic(), } @@ -111,11 +111,11 @@ func (s *secureSession) NoiseStaticKeyCache() *KeyCache { } func (s *secureSession) NoisePublicKey() [32]byte { - return s.noiseKeypair.public_key + return s.noiseKeypair.publicKey } func (s *secureSession) NoisePrivateKey() [32]byte { - return s.noiseKeypair.private_key + return s.noiseKeypair.privateKey } func (s *secureSession) readLength() (int, error) { @@ -165,7 +165,7 @@ func (s *secureSession) runHandshake(ctx context.Context) error { } // sign noise data for payload - noise_pub := s.noiseKeypair.public_key + noise_pub := s.noiseKeypair.publicKey signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) if err != nil { log.Errorf("runHandshake signing payload err=%s", err) diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 8dba93b0d1..f3c8491ec2 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -225,7 +225,7 @@ func TestHandshakeIK(t *testing.T) { } respTransport.NoiseKeypair = kp keycache := NewKeyCache() - keycache.Store(respTransport.LocalID, respTransport.NoiseKeypair.public_key) + keycache.Store(respTransport.LocalID, respTransport.NoiseKeypair.publicKey) initTransport.NoiseStaticKeyCache = keycache // do IK handshake diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index fa597c6014..92f9002fcc 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -82,7 +82,7 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, // if fallback = true, initialMsg is used as the message in stage 1 of the initiator and stage 0 // of the responder func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payload []byte, initialMsg []byte) (err error) { - kp := xx.NewKeypair(s.noiseKeypair.public_key, s.noiseKeypair.private_key) + kp := xx.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) log.Debugf("runHandshake_xx initiator=%v fallback=%v pubkey=%x", s.initiator, fallback, kp.PubKey()) From eb5f403c33cb8ac3194f60b572cd646af7dafec9 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 11 Dec 2019 22:36:54 -0500 Subject: [PATCH 1612/3965] make Transport and config fields private --- p2p/security/noise/integration_test.go | 4 ++-- p2p/security/noise/options.go | 8 +++---- p2p/security/noise/transport.go | 32 +++++++++++++------------- p2p/security/noise/transport_test.go | 28 +++++++++++----------- 4 files changed, 36 insertions(+), 36 deletions(-) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index 3f3f52c4b0..616ab3b523 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -71,8 +71,8 @@ func makeNodePipes(t *testing.T, seed int64, port int, rpid peer.ID, rpubkey [32 t.Fatal(err) } - tpt.NoiseStaticKeyCache = NewKeyCache() - tpt.NoiseStaticKeyCache.Store(rpid, rpubkey) + tpt.noiseStaticKeyCache = NewKeyCache() + tpt.noiseStaticKeyCache.Store(rpid, rpubkey) ip := "0.0.0.0" addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", ip, port)) diff --git a/p2p/security/noise/options.go b/p2p/security/noise/options.go index e4a0c155a9..8ea20514e3 100644 --- a/p2p/security/noise/options.go +++ b/p2p/security/noise/options.go @@ -9,7 +9,7 @@ package noise // using XX in the first place, however there is a slight processing overhead // due to the initial decryption attempt of the IK message. func UseNoisePipes(cfg *config) { - cfg.NoisePipesSupport = true + cfg.noisePipesSupport = true } // NoiseKeyPair configures the Noise transport to use the given Noise static @@ -24,13 +24,13 @@ func UseNoisePipes(cfg *config) { // take care to store the key securely! func NoiseKeyPair(kp *Keypair) Option { return func(cfg *config) { - cfg.NoiseKeypair = kp + cfg.noiseKeypair = kp } } type config struct { - NoiseKeypair *Keypair - NoisePipesSupport bool + noiseKeypair *Keypair + noisePipesSupport bool } type Option func(cfg *config) diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 834f307200..d2bafa08d3 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -17,11 +17,11 @@ var _ sec.SecureTransport = &Transport{} // Transport implements the interface sec.SecureTransport // https://godoc.org/github.com/libp2p/go-libp2p-core/sec#SecureConn type Transport struct { - LocalID peer.ID - PrivateKey crypto.PrivKey - NoisePipesSupport bool - NoiseStaticKeyCache *KeyCache - NoiseKeypair *Keypair + localID peer.ID + privateKey crypto.PrivKey + noisePipesSupport bool + noiseStaticKeyCache *KeyCache + noiseKeypair *Keypair } type transportConstructor func(crypto.PrivKey) (*Transport, error) @@ -82,7 +82,7 @@ func New(privkey crypto.PrivKey, options ...Option) (*Transport, error) { cfg := config{} cfg.applyOptions(options...) - kp := cfg.NoiseKeypair + kp := cfg.noiseKeypair if kp == nil { kp, err = GenerateKeypair() if err != nil { @@ -92,37 +92,37 @@ func New(privkey crypto.PrivKey, options ...Option) (*Transport, error) { // the static key cache is only useful if Noise Pipes is enabled var keyCache *KeyCache - if cfg.NoisePipesSupport { + if cfg.noisePipesSupport { keyCache = NewKeyCache() } return &Transport{ - LocalID: localID, - PrivateKey: privkey, - NoisePipesSupport: cfg.NoisePipesSupport, - NoiseKeypair: kp, - NoiseStaticKeyCache: keyCache, + localID: localID, + privateKey: privkey, + noisePipesSupport: cfg.noisePipesSupport, + noiseKeypair: kp, + noiseStaticKeyCache: keyCache, }, nil } // SecureInbound runs noise handshake as the responder func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { - s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, t.NoiseKeypair, insecure, "", t.NoiseStaticKeyCache, t.NoisePipesSupport, false) + s, err := newSecureSession(ctx, t.localID, t.privateKey, t.noiseKeypair, insecure, "", t.noiseStaticKeyCache, t.noisePipesSupport, false) if err != nil { return s, err } - t.NoiseKeypair = s.noiseKeypair + t.noiseKeypair = s.noiseKeypair return s, nil } // SecureOutbound runs noise handshake as the initiator func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { - s, err := newSecureSession(ctx, t.LocalID, t.PrivateKey, t.NoiseKeypair, insecure, p, t.NoiseStaticKeyCache, t.NoisePipesSupport, true) + s, err := newSecureSession(ctx, t.localID, t.privateKey, t.noiseKeypair, insecure, p, t.noiseStaticKeyCache, t.noisePipesSupport, true) if err != nil { return s, err } - t.NoiseKeypair = s.noiseKeypair + t.noiseKeypair = s.noiseKeypair return s, nil } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index f3c8491ec2..67b72b9fc8 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -23,8 +23,8 @@ func newTestTransport(t *testing.T, typ, bits int) *Transport { t.Fatal(err) } return &Transport{ - LocalID: id, - PrivateKey: priv, + localID: id, + privateKey: priv, } } @@ -38,9 +38,9 @@ func newTestTransportPipes(t *testing.T, typ, bits int) *Transport { t.Fatal(err) } return &Transport{ - LocalID: id, - PrivateKey: priv, - NoisePipesSupport: true, + localID: id, + privateKey: priv, + noisePipesSupport: true, } } @@ -86,7 +86,7 @@ func connect(t *testing.T, initTransport, respTransport *Transport) (*secureSess done := make(chan struct{}) go func() { defer close(done) - initConn, initErr = initTransport.SecureOutbound(context.TODO(), init, respTransport.LocalID) + initConn, initErr = initTransport.SecureOutbound(context.TODO(), init, respTransport.localID) }() respConn, respErr := respTransport.SecureInbound(context.TODO(), resp) @@ -111,11 +111,11 @@ func TestIDs(t *testing.T) { defer initConn.Close() defer respConn.Close() - if initConn.LocalPeer() != initTransport.LocalID { + if initConn.LocalPeer() != initTransport.localID { t.Fatal("Initiator Local Peer ID mismatch.") } - if respConn.RemotePeer() != initTransport.LocalID { + if respConn.RemotePeer() != initTransport.localID { t.Fatal("Responder Remote Peer ID mismatch.") } @@ -124,8 +124,8 @@ func TestIDs(t *testing.T) { } // TODO: check after stage 0 of handshake if updated - if initConn.RemotePeer() != respTransport.LocalID { - t.Errorf("Initiator Remote Peer ID mismatch. expected %x got %x", respTransport.LocalID, initConn.RemotePeer()) + if initConn.RemotePeer() != respTransport.localID { + t.Errorf("Initiator Remote Peer ID mismatch. expected %x got %x", respTransport.localID, initConn.RemotePeer()) } } @@ -140,7 +140,7 @@ func TestKeys(t *testing.T) { sk := respConn.LocalPrivateKey() pk := sk.GetPublic() - if !sk.Equals(respTransport.PrivateKey) { + if !sk.Equals(respTransport.privateKey) { t.Error("Private key Mismatch.") } @@ -223,10 +223,10 @@ func TestHandshakeIK(t *testing.T) { if err != nil { t.Fatal(err) } - respTransport.NoiseKeypair = kp + respTransport.noiseKeypair = kp keycache := NewKeyCache() - keycache.Store(respTransport.LocalID, respTransport.NoiseKeypair.publicKey) - initTransport.NoiseStaticKeyCache = keycache + keycache.Store(respTransport.localID, respTransport.noiseKeypair.publicKey) + initTransport.noiseStaticKeyCache = keycache // do IK handshake initConn, respConn := connect(t, initTransport, respTransport) From 192f8eb23666f04821d02cb591888fe2378859de Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 11 Dec 2019 22:53:54 -0500 Subject: [PATCH 1613/3965] derive session params from Transport --- p2p/security/noise/protocol.go | 48 ++++++++++++++-------------- p2p/security/noise/transport.go | 16 ++-------- p2p/security/noise/transport_test.go | 32 +++++++------------ 3 files changed, 37 insertions(+), 59 deletions(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 073e506b15..ccb9e6f982 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -3,6 +3,7 @@ package noise import ( "context" "encoding/binary" + "errors" "fmt" "net" "sync" @@ -27,6 +28,8 @@ const maxPlaintextLength = 65519 var log = logging.Logger("noise") +var errNoKeypair = errors.New("cannot initiate secureSession - transport has no noise keypair") + type secureSession struct { insecure net.Conn @@ -60,45 +63,42 @@ type peerInfo struct { libp2pKey crypto.PubKey } -// newSecureSession creates a noise session that can be configured to be initialized with a static -// noise key `noisePrivateKey`, a cache of previous peer noise keys `noiseStaticKeyCache`, an -// option `noisePipesSupport` to turn on or off noise pipes +// newSecureSession creates a noise session over the given insecure Conn, using the static +// Noise keypair and libp2p identity keypair from the given Transport. // -// With noise pipes off, we always do XX -// With noise pipes on, we first try IK, if that fails, move to XXfallback -func newSecureSession(ctx context.Context, local peer.ID, privKey crypto.PrivKey, kp *Keypair, - insecure net.Conn, remote peer.ID, noiseStaticKeyCache *KeyCache, - noisePipesSupport bool, initiator bool) (*secureSession, error) { - - if noiseStaticKeyCache == nil { - noiseStaticKeyCache = NewKeyCache() +// If tpt.noisePipesSupport == true, the Noise Pipes handshake protocol will be used, +// which consists of the IK and XXfallback handshake patterns. With Noise Pipes on, we first try IK, +// if that fails, move to XXfallback. With Noise Pipes off, we always do XX. +func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, remote peer.ID, initiator bool) (*secureSession, error) { + if tpt.noiseKeypair == nil { + return nil, errNoKeypair } - if kp == nil { - var err error - kp, err = GenerateKeypair() - if err != nil { - return nil, err - } + // if the transport doesn't have a key cache, we make a new one just for + // this session. it's a bit of a waste, but saves us having to check if + // it's nil later + keyCache := tpt.noiseStaticKeyCache + if keyCache == nil { + keyCache = NewKeyCache() } localPeerInfo := peerInfo{ - noiseKey: kp.publicKey, - libp2pKey: privKey.GetPublic(), + noiseKey: tpt.noiseKeypair.publicKey, + libp2pKey: tpt.privateKey.GetPublic(), } s := &secureSession{ insecure: insecure, initiator: initiator, prologue: []byte(ID), - localKey: privKey, - localPeer: local, + localKey: tpt.privateKey, + localPeer: tpt.localID, remotePeer: remote, local: localPeerInfo, - noisePipesSupport: noisePipesSupport, - noiseStaticKeyCache: noiseStaticKeyCache, + noisePipesSupport: tpt.noisePipesSupport, + noiseStaticKeyCache: keyCache, msgBuffer: []byte{}, - noiseKeypair: kp, + noiseKeypair: tpt.noiseKeypair, } err := s.runHandshake(ctx) diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index d2bafa08d3..d6c33d7a42 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -107,22 +107,10 @@ func New(privkey crypto.PrivKey, options ...Option) (*Transport, error) { // SecureInbound runs noise handshake as the responder func (t *Transport) SecureInbound(ctx context.Context, insecure net.Conn) (sec.SecureConn, error) { - s, err := newSecureSession(ctx, t.localID, t.privateKey, t.noiseKeypair, insecure, "", t.noiseStaticKeyCache, t.noisePipesSupport, false) - if err != nil { - return s, err - } - - t.noiseKeypair = s.noiseKeypair - return s, nil + return newSecureSession(t, ctx, insecure, "", false) } // SecureOutbound runs noise handshake as the initiator func (t *Transport) SecureOutbound(ctx context.Context, insecure net.Conn, p peer.ID) (sec.SecureConn, error) { - s, err := newSecureSession(ctx, t.localID, t.privateKey, t.noiseKeypair, insecure, p, t.noiseStaticKeyCache, t.noisePipesSupport, true) - if err != nil { - return s, err - } - - t.noiseKeypair = s.noiseKeypair - return s, nil + return newSecureSession(t, ctx, insecure, p, true) } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 67b72b9fc8..0a2cbaf941 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -22,28 +22,23 @@ func newTestTransport(t *testing.T, typ, bits int) *Transport { if err != nil { t.Fatal(err) } - return &Transport{ - localID: id, - privateKey: priv, - } -} - -func newTestTransportPipes(t *testing.T, typ, bits int) *Transport { - priv, pub, err := crypto.GenerateKeyPair(typ, bits) - if err != nil { - t.Fatal(err) - } - id, err := peer.IDFromPublicKey(pub) + kp, err := GenerateKeypair() if err != nil { t.Fatal(err) } return &Transport{ - localID: id, - privateKey: priv, - noisePipesSupport: true, + localID: id, + privateKey: priv, + noiseKeypair: kp, } } +func newTestTransportPipes(t *testing.T, typ, bits int) *Transport { + tpt := newTestTransport(t, typ, bits) + tpt.noisePipesSupport = true + return tpt +} + // Create a new pair of connected TCP sockets. func newConnPair(t *testing.T) (net.Conn, net.Conn) { lstnr, err := net.Listen("tcp", "localhost:0") @@ -219,11 +214,6 @@ func TestHandshakeIK(t *testing.T) { respTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) // add responder's static key to initiator's key cache - kp, err := GenerateKeypair() - if err != nil { - t.Fatal(err) - } - respTransport.noiseKeypair = kp keycache := NewKeyCache() keycache.Store(respTransport.localID, respTransport.noiseKeypair.publicKey) initTransport.noiseStaticKeyCache = keycache @@ -234,7 +224,7 @@ func TestHandshakeIK(t *testing.T) { defer respConn.Close() before := []byte("hello world") - _, err = initConn.Write(before) + _, err := initConn.Write(before) if err != nil { t.Fatal(err) } From f18983a45575279651ffd2082ba0efbd6346f13d Mon Sep 17 00:00:00 2001 From: Mikerah Date: Fri, 13 Dec 2019 15:24:15 -0500 Subject: [PATCH 1614/3965] typo fix --- core/transport/transport.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/transport/transport.go b/core/transport/transport.go index 774b8d824d..39a8fd38eb 100644 --- a/core/transport/transport.go +++ b/core/transport/transport.go @@ -1,5 +1,5 @@ // Package transport provides the Transport interface, which represents -// the devices and network protocols used to send and recieve data. +// the devices and network protocols used to send and receive data. package transport import ( From 7d77f3ed7bbe1d8a3abe66366021c4d658aaf7e5 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 16 Dec 2019 11:22:18 +0000 Subject: [PATCH 1615/3965] build(deps): bump github.com/ipfs/go-cid from 0.0.3 to 0.0.4 Bumps [github.com/ipfs/go-cid](https://github.com/ipfs/go-cid) from 0.0.3 to 0.0.4. - [Release notes](https://github.com/ipfs/go-cid/releases) - [Commits](https://github.com/ipfs/go-cid/compare/v0.0.3...v0.0.4) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 7a6a5a501a..0a9d17afa3 100644 --- a/go.mod +++ b/go.mod @@ -2,7 +2,7 @@ module github.com/libp2p/go-libp2p require ( github.com/gogo/protobuf v1.3.1 - github.com/ipfs/go-cid v0.0.3 + github.com/ipfs/go-cid v0.0.4 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 github.com/ipfs/go-log v0.0.1 diff --git a/go.sum b/go.sum index b4de423660..5d1bc609d9 100644 --- a/go.sum +++ b/go.sum @@ -78,6 +78,8 @@ github.com/ipfs/go-cid v0.0.2 h1:tuuKaZPU1M6HcejsO3AcYWW8sZ8MTvyxfc4uqB4eFE8= github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-cid v0.0.3 h1:UIAh32wymBpStoe83YCzwVQQ5Oy/H0FdxvUS6DJDzms= github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= +github.com/ipfs/go-cid v0.0.4 h1:UlfXKrZx1DjZoBhQHmNHLC1fK1dUJDN20Y28A7s+gJ8= +github.com/ipfs/go-cid v0.0.4/go.mod h1:4LLaPOQwmk5z9LBgQnpkivrx8BJjUyGwTXCd5Xfj6+M= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= github.com/ipfs/go-datastore v0.1.0/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= github.com/ipfs/go-datastore v0.1.1/go.mod h1:w38XXW9kVFNp57Zj5knbKWM2T+KOZCGDRVNdgPHtbHw= @@ -287,6 +289,8 @@ github.com/multiformats/go-multihash v0.0.5 h1:1wxmCvTXAifAepIMyF39vZinRw5sbqjPs github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= github.com/multiformats/go-multihash v0.0.8 h1:wrYcW5yxSi3dU07n5jnuS5PrNwyHy0zRHGVoUugWvXg= github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= +github.com/multiformats/go-multihash v0.0.10 h1:lMoNbh2Ssd9PUF74Nz008KGzGPlfeV6wH3rit5IIGCM= +github.com/multiformats/go-multihash v0.0.10/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ= github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/multiformats/go-varint v0.0.1 h1:TR/0rdQtnNxuN2IhiB639xC3tWM4IUi7DkTBVTdGW/M= From e5d28696f575b4a8424943f2a90f2e17de359e0f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 16 Dec 2019 15:51:21 +0100 Subject: [PATCH 1616/3965] feat(host): recursively resolve addresses We currently only resolve /dnsaddr addresses once. This patch resolves up to 32 dnsaddr addresses per dial. fixes https://github.com/libp2p/go-libp2p/issues/744 --- go.sum | 3 +- p2p/host/basic/basic_host.go | 45 ++++++++++++++++--- p2p/host/basic/basic_host_test.go | 75 +++++++++++++++++++++++++++++-- 3 files changed, 113 insertions(+), 10 deletions(-) diff --git a/go.sum b/go.sum index 2e32148802..615a533646 100644 --- a/go.sum +++ b/go.sum @@ -132,6 +132,7 @@ github.com/libp2p/go-eventbus v0.1.0 h1:mlawomSAjjkk97QnYiEmHsLu7E136+2oCWSHRUvM github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UGQf8A2eEmx4= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= +github.com/libp2p/go-flow-metrics v0.0.2 h1:U5TvqfoyR6GVRM+bC15Ux1ltar1kbj6Zw6xOVR02CZs= github.com/libp2p/go-flow-metrics v0.0.2/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= @@ -151,8 +152,6 @@ github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV github.com/libp2p/go-libp2p-core v0.2.5 h1:iP1PIiIrlRrGbE1fYq2918yBc5NlCH3pFuIPSWU9hds= github.com/libp2p/go-libp2p-core v0.2.5/go.mod h1:6+5zJmKhsf7yHn1RbmYDu08qDUpIUxGdqHuEZckmZOA= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= -github.com/libp2p/go-libp2p-discovery v0.1.0 h1:j+R6cokKcGbnZLf4kcNwpx6mDEUPF3N6SrqMymQhmvs= -github.com/libp2p/go-libp2p-discovery v0.1.0/go.mod h1:4F/x+aldVHjHDHuX85x1zWoFTGElt8HnoDzwkFZm29g= github.com/libp2p/go-libp2p-discovery v0.2.0 h1:1p3YSOq7VsgaL+xVHPi8XAmtGyas6D2J6rWBEfz/aiY= github.com/libp2p/go-libp2p-discovery v0.2.0/go.mod h1:s4VGaxYMbw4+4+tsoQTqh7wfxg97AEdo4GYBt6BadWg= github.com/libp2p/go-libp2p-loggables v0.1.0 h1:h3w8QFfCt2UJl/0/NW4K829HX/0S4KD31PQ7m8UXXO8= diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 92946c5d7c..56bc9c222c 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -31,6 +31,10 @@ import ( msmux "github.com/multiformats/go-multistream" ) +// The maximum number of address resolution steps we'll perform for a single +// peer (for all addresses). +const maxAddressResolution = 32 + var log = logging.Logger("basichost") var ( @@ -522,28 +526,59 @@ func (h *BasicHost) resolveAddrs(ctx context.Context, pi peer.AddrInfo) ([]ma.Mu return nil, err } - var addrs []ma.Multiaddr - for _, addr := range pi.Addrs { - addrs = append(addrs, addr) + resolveSteps := 0 + + // Recursively resolve all addrs. + // + // While the toResolve list is non-empty: + // * Pop an address off. + // * If the address is fully resolved, add it to the resolved list. + // * Otherwise, resolve it and add the results to the "to resolve" list. + toResolve := append(([]ma.Multiaddr)(nil), pi.Addrs...) + resolved := make([]ma.Multiaddr, 0, len(pi.Addrs)) + for len(toResolve) > 0 { + // pop the last addr off. + addr := toResolve[len(toResolve)-1] + toResolve = toResolve[:len(toResolve)-1] + + // if it's resolved, add it to the resolved list. if !madns.Matches(addr) { + resolved = append(resolved, addr) continue } + resolveSteps++ + + // We've resolved too many addresses. We can keep all the fully + // resolved addresses but we'll need to skip the rest. + if resolveSteps >= maxAddressResolution { + log.Warningf( + "peer %s asked us to resolve too many addresses: %s/%s", + pi.ID, + resolveSteps, + maxAddressResolution, + ) + continue + } + + // otherwise, resolve it reqaddr := addr.Encapsulate(p2paddr) resaddrs, err := h.maResolver.Resolve(ctx, reqaddr) if err != nil { log.Infof("error resolving %s: %s", reqaddr, err) } + + // add the results to the toResolve list. for _, res := range resaddrs { pi, err := peer.AddrInfoFromP2pAddr(res) if err != nil { log.Infof("error parsing %s: %s", res, err) } - addrs = append(addrs, pi.Addrs...) + toResolve = append(toResolve, pi.Addrs...) } } - return addrs, nil + return resolved, nil } // dialPeer opens a connection to peer, and makes sure to identify diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index 5e980f8967..ed0370b8a4 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -407,9 +407,9 @@ func TestAddrResolution(t *testing.T) { } addr1 := ma.StringCast("/dnsaddr/example.com") addr2 := ma.StringCast("/ip4/192.0.2.1/tcp/123") - p2paddr1 := ma.StringCast("/dnsaddr/example.com/ipfs/" + p1.Pretty()) - p2paddr2 := ma.StringCast("/ip4/192.0.2.1/tcp/123/ipfs/" + p1.Pretty()) - p2paddr3 := ma.StringCast("/ip4/192.0.2.1/tcp/123/ipfs/" + p2.Pretty()) + p2paddr1 := ma.StringCast("/dnsaddr/example.com/p2p/" + p1.Pretty()) + p2paddr2 := ma.StringCast("/ip4/192.0.2.1/tcp/123/p2p/" + p1.Pretty()) + p2paddr3 := ma.StringCast("/ip4/192.0.2.1/tcp/123/p2p/" + p2.Pretty()) backend := &madns.MockBackend{ TXT: map[string][]string{"_dnsaddr.example.com": []string{ @@ -438,6 +438,75 @@ func TestAddrResolution(t *testing.T) { } } +func TestAddrResolutionRecursive(t *testing.T) { + ctx := context.Background() + + p1, err := test.RandPeerID() + if err != nil { + t.Error(err) + } + p2, err := test.RandPeerID() + if err != nil { + t.Error(err) + } + addr1 := ma.StringCast("/dnsaddr/example.com") + addr2 := ma.StringCast("/ip4/192.0.2.1/tcp/123") + p2paddr1 := ma.StringCast("/dnsaddr/example.com/p2p/" + p1.Pretty()) + p2paddr2 := ma.StringCast("/dnsaddr/example.com/p2p/" + p2.Pretty()) + p2paddr1i := ma.StringCast("/dnsaddr/foo.example.com/p2p/" + p1.Pretty()) + p2paddr2i := ma.StringCast("/dnsaddr/bar.example.com/p2p/" + p2.Pretty()) + p2paddr1f := ma.StringCast("/ip4/192.0.2.1/tcp/123/p2p/" + p1.Pretty()) + + backend := &madns.MockBackend{ + TXT: map[string][]string{ + "_dnsaddr.example.com": []string{ + "dnsaddr=" + p2paddr1i.String(), + "dnsaddr=" + p2paddr2i.String(), + }, + "_dnsaddr.foo.example.com": []string{ + "dnsaddr=" + p2paddr1f.String(), + }, + "_dnsaddr.bar.example.com": []string{ + "dnsaddr=" + p2paddr2i.String(), + }, + }, + } + resolver := &madns.Resolver{Backend: backend} + + h := New(swarmt.GenSwarm(t, ctx), resolver) + defer h.Close() + + pi1, err := peer.AddrInfoFromP2pAddr(p2paddr1) + if err != nil { + t.Error(err) + } + + tctx, cancel := context.WithTimeout(ctx, time.Millisecond*100) + defer cancel() + _ = h.Connect(tctx, *pi1) + + addrs1 := h.Peerstore().Addrs(pi1.ID) + sort.Sort(sortedMultiaddrs(addrs1)) + + if len(addrs1) != 2 || !addrs1[0].Equal(addr1) || !addrs1[1].Equal(addr2) { + t.Fatalf("expected [%s %s], got %+v", addr1, addr2, addrs1) + } + + pi2, err := peer.AddrInfoFromP2pAddr(p2paddr2) + if err != nil { + t.Error(err) + } + + _ = h.Connect(tctx, *pi2) + + addrs2 := h.Peerstore().Addrs(pi2.ID) + sort.Sort(sortedMultiaddrs(addrs2)) + + if len(addrs2) != 1 || !addrs2[0].Equal(addr1) { + t.Fatalf("expected [%s], got %+v", addr1, addrs2) + } +} + type sortedMultiaddrs []ma.Multiaddr func (sma sortedMultiaddrs) Len() int { return len(sma) } From ff532fae93258af014606b8237ebeacc81c22c03 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 16 Dec 2019 18:40:47 +0000 Subject: [PATCH 1617/3965] build(deps): bump github.com/libp2p/go-libp2p-core from 0.2.5 to 0.3.0 Bumps [github.com/libp2p/go-libp2p-core](https://github.com/libp2p/go-libp2p-core) from 0.2.5 to 0.3.0. - [Release notes](https://github.com/libp2p/go-libp2p-core/releases) - [Commits](https://github.com/libp2p/go-libp2p-core/compare/v0.2.5...v0.3.0) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 7a6a5a501a..5808fa36cd 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,7 @@ require ( github.com/libp2p/go-libp2p-autonat v0.1.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 - github.com/libp2p/go-libp2p-core v0.2.5 + github.com/libp2p/go-libp2p-core v0.3.0 github.com/libp2p/go-libp2p-discovery v0.2.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 diff --git a/go.sum b/go.sum index 3878083688..7b0daed767 100644 --- a/go.sum +++ b/go.sum @@ -10,6 +10,8 @@ github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c h1:aEbSeNALREWXk0G7U github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI= github.com/btcsuite/btcd v0.0.0-20190824003749-130ea5bddde3 h1:A/EVblehb75cUgXA5njHPn0kLAsykn6mJGz7rnmW5W0= github.com/btcsuite/btcd v0.0.0-20190824003749-130ea5bddde3/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI= +github.com/btcsuite/btcd v0.20.1-beta h1:Ik4hyJqN8Jfyv3S4AGBOmyouMsYE3EdYODkMbQjwPGw= +github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ= github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA= github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= @@ -134,6 +136,7 @@ github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atq github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-flow-metrics v0.0.2 h1:U5TvqfoyR6GVRM+bC15Ux1ltar1kbj6Zw6xOVR02CZs= github.com/libp2p/go-flow-metrics v0.0.2/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= +github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= @@ -151,6 +154,8 @@ github.com/libp2p/go-libp2p-core v0.2.4 h1:Et6ykkTwI6PU44tr8qUF9k43vP0aduMNniShA github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= github.com/libp2p/go-libp2p-core v0.2.5 h1:iP1PIiIrlRrGbE1fYq2918yBc5NlCH3pFuIPSWU9hds= github.com/libp2p/go-libp2p-core v0.2.5/go.mod h1:6+5zJmKhsf7yHn1RbmYDu08qDUpIUxGdqHuEZckmZOA= +github.com/libp2p/go-libp2p-core v0.3.0 h1:F7PqduvrztDtFsAa/bcheQ3azmNo+Nq7m8hQY5GiUW8= +github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.2.0 h1:1p3YSOq7VsgaL+xVHPi8XAmtGyas6D2J6rWBEfz/aiY= github.com/libp2p/go-libp2p-discovery v0.2.0/go.mod h1:s4VGaxYMbw4+4+tsoQTqh7wfxg97AEdo4GYBt6BadWg= @@ -207,6 +212,8 @@ github.com/libp2p/go-openssl v0.0.2 h1:9pP2d3Ubaxkv7ZisLjx9BFwgOGnQdQYnfcH29HNY3 github.com/libp2p/go-openssl v0.0.2/go.mod h1:v8Zw2ijCSWBQi8Pq5GAixw6DbFfa9u6VIYDXnvOXkc0= github.com/libp2p/go-openssl v0.0.3 h1:wjlG7HvQkt4Fq4cfH33Ivpwp0omaElYEi9z26qaIkIk= github.com/libp2p/go-openssl v0.0.3/go.mod h1:unDrJpgy3oFr+rqXsarWifmJuNnJR4chtO1HmaZjggc= +github.com/libp2p/go-openssl v0.0.4 h1:d27YZvLoTyMhIN4njrkr8zMDOM4lfpHIp6A+TK9fovg= +github.com/libp2p/go-openssl v0.0.4/go.mod h1:unDrJpgy3oFr+rqXsarWifmJuNnJR4chtO1HmaZjggc= github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0GIEjMiHTwR/KN4= @@ -251,6 +258,8 @@ github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVq github.com/mr-tron/base58 v1.1.1/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= github.com/mr-tron/base58 v1.1.2 h1:ZEw4I2EgPKDJ2iEw0cNmLB3ROrEmkOtXIkaG7wZg+78= github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= +github.com/mr-tron/base58 v1.1.3 h1:v+sk57XuaCKGXpWtVBX8YJzO7hMGx4Aajh4TQbdEFdc= +github.com/mr-tron/base58 v1.1.3/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI= github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA= github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= @@ -286,6 +295,8 @@ github.com/multiformats/go-multihash v0.0.5 h1:1wxmCvTXAifAepIMyF39vZinRw5sbqjPs github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= github.com/multiformats/go-multihash v0.0.8 h1:wrYcW5yxSi3dU07n5jnuS5PrNwyHy0zRHGVoUugWvXg= github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= +github.com/multiformats/go-multihash v0.0.10 h1:lMoNbh2Ssd9PUF74Nz008KGzGPlfeV6wH3rit5IIGCM= +github.com/multiformats/go-multihash v0.0.10/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ= github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/multiformats/go-varint v0.0.1 h1:TR/0rdQtnNxuN2IhiB639xC3tWM4IUi7DkTBVTdGW/M= @@ -326,6 +337,7 @@ github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1/go.mod h1:8UvriyWtv5Q5EOgjHaSseUEdkQfvwFv1I/In/O2M9gc= @@ -343,6 +355,7 @@ go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= go.opencensus.io v0.22.1 h1:8dP3SGL7MPB94crU3bEPplMPe83FI4EouesJUeFHv50= go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA= +go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= From 3d337a9a107c347913444b6acdcd587b79445b2a Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 17 Dec 2019 13:49:35 +0200 Subject: [PATCH 1618/3965] add mutex for write/close --- p2p/transport/websocket/conn_native.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/p2p/transport/websocket/conn_native.go b/p2p/transport/websocket/conn_native.go index 6be4697ec6..bd84b7dce9 100644 --- a/p2p/transport/websocket/conn_native.go +++ b/p2p/transport/websocket/conn_native.go @@ -17,6 +17,7 @@ type Conn struct { DefaultMessageType int reader io.Reader closeOnce sync.Once + mx sync.Mutex } func (c *Conn) Read(b []byte) (int, error) { @@ -67,6 +68,9 @@ func (c *Conn) prepNextReader() error { } func (c *Conn) Write(b []byte) (n int, err error) { + c.mx.Lock() + defer c.mx.Unlock() + if err := c.Conn.WriteMessage(c.DefaultMessageType, b); err != nil { return 0, err } @@ -78,6 +82,9 @@ func (c *Conn) Write(b []byte) (n int, err error) { // close error, subsequent and concurrent calls will return nil. // This method is thread-safe. func (c *Conn) Close() error { + c.mx.Lock() + defer c.mx.Unlock() + var err error c.closeOnce.Do(func() { err1 := c.Conn.WriteControl( From 8d90b859ff6adae434f4709f1cdef23ce1240607 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 17 Dec 2019 13:09:33 +0100 Subject: [PATCH 1619/3965] chore(dep): update go-ws-transport * Fixes a concurrent write/panic bug. * Adds WASM support. --- go.mod | 2 +- go.sum | 15 +++++++-------- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index 232e0e9a5c..1c12af8207 100644 --- a/go.mod +++ b/go.mod @@ -28,7 +28,7 @@ require ( github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 - github.com/libp2p/go-ws-transport v0.1.2 + github.com/libp2p/go-ws-transport v0.2.0 github.com/miekg/dns v1.1.12 // indirect github.com/multiformats/go-multiaddr v0.2.0 github.com/multiformats/go-multiaddr-dns v0.2.0 diff --git a/go.sum b/go.sum index dcd61b8fed..e1b2cb45de 100644 --- a/go.sum +++ b/go.sum @@ -136,8 +136,7 @@ github.com/libp2p/go-eventbus v0.1.0 h1:mlawomSAjjkk97QnYiEmHsLu7E136+2oCWSHRUvM github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UGQf8A2eEmx4= github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atqi3INF5s= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= -github.com/libp2p/go-flow-metrics v0.0.2 h1:U5TvqfoyR6GVRM+bC15Ux1ltar1kbj6Zw6xOVR02CZs= -github.com/libp2p/go-flow-metrics v0.0.2/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= +github.com/libp2p/go-flow-metrics v0.0.3 h1:8tAs/hSdNvUiLgtlSy3mxwxWP4I9y/jlkPFT7epKdeM= github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= @@ -154,8 +153,6 @@ github.com/libp2p/go-libp2p-core v0.2.2 h1:Sv1ggdoMx9c7v7FOFkR7agraHCnAgqYsXrU1A github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZVoVrLaDRvIRng0= github.com/libp2p/go-libp2p-core v0.2.4 h1:Et6ykkTwI6PU44tr8qUF9k43vP0aduMNniShAbUJJw8= github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= -github.com/libp2p/go-libp2p-core v0.2.5 h1:iP1PIiIrlRrGbE1fYq2918yBc5NlCH3pFuIPSWU9hds= -github.com/libp2p/go-libp2p-core v0.2.5/go.mod h1:6+5zJmKhsf7yHn1RbmYDu08qDUpIUxGdqHuEZckmZOA= github.com/libp2p/go-libp2p-core v0.3.0 h1:F7PqduvrztDtFsAa/bcheQ3azmNo+Nq7m8hQY5GiUW8= github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= @@ -227,8 +224,8 @@ github.com/libp2p/go-tcp-transport v0.1.0 h1:IGhowvEqyMFknOar4FWCKSWE0zL36UFKQti github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= github.com/libp2p/go-tcp-transport v0.1.1 h1:yGlqURmqgNA2fvzjSgZNlHcsd/IulAnKM8Ncu+vlqnw= github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfjg2pWP97dFZworkY= -github.com/libp2p/go-ws-transport v0.1.2 h1:VnxQcLfSGtqupqPpBNu8fUiCv+IN1RJ2BcVqQEM+z8E= -github.com/libp2p/go-ws-transport v0.1.2/go.mod h1:dsh2Ld8F+XNmzpkaAijmg5Is+e9l6/1tK/6VFOdN69Y= +github.com/libp2p/go-ws-transport v0.2.0 h1:MJCw2OrPA9+76YNRvdo1wMnSOxb9Bivj6sVFY1Xrj6w= +github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzlHoKzu0yY9p/klM= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= @@ -271,8 +268,6 @@ github.com/multiformats/go-multiaddr v0.0.4/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lg github.com/multiformats/go-multiaddr v0.1.0/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr v0.1.1 h1:rVAztJYMhCQ7vEFr8FvxW3mS+HF2eY/oPbOMeS0ZDnE= github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo= -github.com/multiformats/go-multiaddr v0.1.2 h1:HWYHNSyyllbQopmVIF5K7JKJugiah+L9/kuZKHbmNdQ= -github.com/multiformats/go-multiaddr v0.1.2/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4= github.com/multiformats/go-multiaddr v0.2.0 h1:lR52sFwcTCuQb6bTfnXF6zA2XfyYvyd+5a9qECv/J90= github.com/multiformats/go-multiaddr v0.2.0/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= @@ -339,12 +334,15 @@ github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1/go.mod h1:8UvriyWtv5Q5EOgjHaSseUEdkQfvwFv1I/In/O2M9gc= github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6Rz4bwmou+oE6Dt4Cb2BGMur5eR/GYptkKUVHo= github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM= +github.com/whyrusleeping/go-logging v0.0.1 h1:fwpzlmT0kRC/Fmd0MdmGgJG/CXIZ6gFq46FQZjprUcc= +github.com/whyrusleeping/go-logging v0.0.1/go.mod h1:lDPYj54zutzG1XYfHAhcc7oNXEburHQBn+Iqd4yS4vE= github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 h1:Y1/FEOpaCpD21WxrmfeIYCFPuVPRCY2XZTWzTNHGw30= @@ -357,6 +355,7 @@ go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= go.opencensus.io v0.22.1 h1:8dP3SGL7MPB94crU3bEPplMPe83FI4EouesJUeFHv50= go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA= +go.opencensus.io v0.22.2 h1:75k/FF0Q2YM8QYo07VPddOLBslDt1MZOdEslOHvmzAs= go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= From 9073eaaf8f086dc2eafcd76c9e3480a3759901d9 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 3 Jan 2020 09:14:43 -0800 Subject: [PATCH 1620/3965] fix: demote stream deadline errors to debug logs Otherwise, we spam an error every time we try to negotiate a protocol on a closed stream (e.g., when the underlying connection closes on us). --- p2p/host/basic/basic_host.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 56bc9c222c..2f408759ce 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -254,7 +254,7 @@ func (h *BasicHost) newStreamHandler(s network.Stream) { if h.negtimeout > 0 { if err := s.SetDeadline(time.Now().Add(h.negtimeout)); err != nil { - log.Error("setting stream deadline: ", err) + log.Debug("setting stream deadline: ", err) s.Reset() return } @@ -283,7 +283,7 @@ func (h *BasicHost) newStreamHandler(s network.Stream) { if h.negtimeout > 0 { if err := s.SetDeadline(time.Time{}); err != nil { - log.Error("resetting stream deadline: ", err) + log.Debugf("resetting stream deadline: ", err) s.Reset() return } From 4102c54de012bcd6cbbab687d11ada22b0838db4 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 9 Jan 2020 09:36:15 +0100 Subject: [PATCH 1621/3965] feat: faster copy in wasm Use the new CopyBytesTo* functions to avoid copying byte by byte. --- p2p/transport/websocket/conn_browser.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/p2p/transport/websocket/conn_browser.go b/p2p/transport/websocket/conn_browser.go index 1f021529d1..83f07a2f52 100644 --- a/p2p/transport/websocket/conn_browser.go +++ b/p2p/transport/websocket/conn_browser.go @@ -117,8 +117,8 @@ func (c *Conn) Write(b []byte) (n int, err error) { return 0, err } uint8Array := js.Global().Get("Uint8Array").New(len(b)) - for i, bi := range b { - uint8Array.SetIndex(i, bi) + if js.CopyBytesToJS(uint8Array, b) != len(b) { + panic("expected to copy all bytes") } c.Call("send", uint8Array.Get("buffer")) return len(b), nil @@ -262,10 +262,10 @@ func (c *Conn) waitForOpen() error { // arrayBufferToBytes converts a JavaScript ArrayBuffer to a slice of bytes. func arrayBufferToBytes(buffer js.Value) []byte { view := js.Global().Get("Uint8Array").New(buffer) - dataLen := view.Get("length").Int() + dataLen := view.Length() data := make([]byte, dataLen) - for i := 0; i < dataLen; i++ { - data[i] = byte(view.Index(i).Int()) + if js.CopyBytesToGo(data, view) != dataLen { + panic("expected to copy all bytes") } return data } From e616482ec8fab5b749db072a3a287ff6722272ad Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Sat, 11 Jan 2020 09:26:45 -0500 Subject: [PATCH 1622/3965] use empty prologue --- p2p/security/noise/protocol.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index ccb9e6f982..591961c615 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -90,7 +90,7 @@ func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, re s := &secureSession{ insecure: insecure, initiator: initiator, - prologue: []byte(ID), + prologue: []byte{}, localKey: tpt.privateKey, localPeer: tpt.localID, remotePeer: remote, From 33f098383220c5579b8dfd836a1366bf06df2655 Mon Sep 17 00:00:00 2001 From: Arber Avdullahu Date: Sun, 12 Jan 2020 23:15:31 +0100 Subject: [PATCH 1623/3965] Update nat.go --- p2p/net/nat/nat.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/net/nat/nat.go b/p2p/net/nat/nat.go index cdddaeaec5..9601bf2c88 100644 --- a/p2p/net/nat/nat.go +++ b/p2p/net/nat/nat.go @@ -121,7 +121,7 @@ func (nat *NAT) rmMapping(m *mapping) { nat.mappingmu.Unlock() } -// NewMapping attemps to construct a mapping on protocol and internal port +// NewMapping attempts to construct a mapping on protocol and internal port // It will also periodically renew the mapping until the returned Mapping // -- or its parent NAT -- is Closed. // @@ -183,7 +183,7 @@ func (nat *NAT) establishMapping(m *mapping) { if err != nil || newport == 0 { m.setExternalPort(0) // clear mapping // TODO: log.Event - log.Warningf("failed to establish port mapping: %s", err) + log.Warnf("failed to establish port mapping: %s", err) // we do not close if the mapping failed, // because it may work again next time. return From 3f43dfe646394c80a972743909d3b06484919047 Mon Sep 17 00:00:00 2001 From: optman Date: Wed, 15 Jan 2020 18:38:39 +0800 Subject: [PATCH 1624/3965] enable non-public address port mapping announcement --- p2p/host/basic/basic_host.go | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 2f408759ce..f8bf279cff 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -732,20 +732,23 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { continue } + extMaddr := mappedMaddr + if rest != nil { + extMaddr = ma.Join(extMaddr, rest) + } + + // Add in the mapped addr. + finalAddrs = append(finalAddrs, extMaddr) + // Did the router give us a routable public addr? if manet.IsPublicAddr(mappedMaddr) { - // Yes, use it. - extMaddr := mappedMaddr - if rest != nil { - extMaddr = ma.Join(extMaddr, rest) - } - - // Add in the mapped addr. - finalAddrs = append(finalAddrs, extMaddr) + //well done continue } - // No. Ok, let's try our observed addresses. + // No. + // in case router give us a wrong address. + // also add observed addresses // Now, check if we have any observed addresses that // differ from the one reported by the router. Routers @@ -776,6 +779,7 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { } finalAddrs = append(finalAddrs, observedAddrs...) } + return dedupAddrs(finalAddrs) } From 628240fb522c70c34f17734b69ae03a0a2374d72 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Thu, 16 Jan 2020 01:29:26 +0000 Subject: [PATCH 1625/3965] Introduce routability and protocol events; cache unmarshalled RSA keys (#105) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * event: Add autonat events (#25) * add events for identify (#26) * implement caching for rsaKey.Bytes() * store marshalled protobuf in cache for RsaPublicKey.Bytes() * fix(crypto): fix build when openssl is enabled * add godocs to routability events. Co-authored-by: Åukasz Magiera Co-authored-by: Whyrusleeping Co-authored-by: Adin Schmahmann Co-authored-by: Steven Allen --- core/crypto/key.go | 14 +++++++++++++- core/crypto/key_not_openssl.go | 2 +- core/crypto/key_openssl.go | 2 +- core/crypto/openssl_common.go | 17 ++++++++++++++--- core/crypto/rsa_go.go | 19 +++++++++++++++---- core/crypto/rsa_openssl.go | 4 ++-- core/event/identify.go | 17 +++++++++++++++++ core/event/routability.go | 21 +++++++++++++++++++++ 8 files changed, 84 insertions(+), 12 deletions(-) create mode 100644 core/event/identify.go create mode 100644 core/event/routability.go diff --git a/core/crypto/key.go b/core/crypto/key.go index 3a5263665b..090338810f 100644 --- a/core/crypto/key.go +++ b/core/crypto/key.go @@ -294,7 +294,19 @@ func PublicKeyFromProto(pmes *pb.PublicKey) (PubKey, error) { return nil, ErrBadKeyType } - return um(pmes.GetData()) + data := pmes.GetData() + + pk, err := um(data) + if err != nil { + return nil, err + } + + switch tpk := pk.(type) { + case *RsaPublicKey: + tpk.cached, _ = pmes.Marshal() + } + + return pk, nil } // MarshalPublicKey converts a public key object into a protobuf serialized diff --git a/core/crypto/key_not_openssl.go b/core/crypto/key_not_openssl.go index 6b5b6fc2ce..fb1e36adf3 100644 --- a/core/crypto/key_not_openssl.go +++ b/core/crypto/key_not_openssl.go @@ -19,7 +19,7 @@ func KeyPairFromStdKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { switch p := priv.(type) { case *rsa.PrivateKey: - return &RsaPrivateKey{*p}, &RsaPublicKey{p.PublicKey}, nil + return &RsaPrivateKey{*p}, &RsaPublicKey{k: p.PublicKey}, nil case *ecdsa.PrivateKey: return &ECDSAPrivateKey{p}, &ECDSAPublicKey{&p.PublicKey}, nil diff --git a/core/crypto/key_openssl.go b/core/crypto/key_openssl.go index b948f3d081..5b1e3607ab 100644 --- a/core/crypto/key_openssl.go +++ b/core/crypto/key_openssl.go @@ -26,7 +26,7 @@ func KeyPairFromStdKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { return nil, nil, err } - return &opensslPrivateKey{pk}, &opensslPublicKey{pk}, nil + return &opensslPrivateKey{pk}, &opensslPublicKey{key: pk}, nil case *ecdsa.PrivateKey: return &ECDSAPrivateKey{p}, &ECDSAPublicKey{&p.PublicKey}, nil diff --git a/core/crypto/openssl_common.go b/core/crypto/openssl_common.go index 2466521c7a..88807cafd7 100644 --- a/core/crypto/openssl_common.go +++ b/core/crypto/openssl_common.go @@ -3,6 +3,8 @@ package crypto import ( + "sync" + pb "github.com/libp2p/go-libp2p-core/crypto/pb" openssl "github.com/libp2p/go-openssl" @@ -13,6 +15,9 @@ import ( type opensslPublicKey struct { key openssl.PublicKey + + cacheLk sync.Mutex + cached []byte } type opensslPrivateKey struct { @@ -32,7 +37,7 @@ func unmarshalOpensslPublicKey(b []byte) (opensslPublicKey, error) { if err != nil { return opensslPublicKey{}, err } - return opensslPublicKey{sk}, nil + return opensslPublicKey{key: sk, cached: b}, nil } // Verify compares a signature against input data @@ -52,7 +57,13 @@ func (pk *opensslPublicKey) Type() pb.KeyType { // Bytes returns protobuf bytes of a public key func (pk *opensslPublicKey) Bytes() ([]byte, error) { - return MarshalPublicKey(pk) + pk.cacheLk.Lock() + var err error + if pk.cached == nil { + pk.cached, err = MarshalPublicKey(pk) + } + pk.cacheLk.Unlock() + return pk.cached, err } func (pk *opensslPublicKey) Raw() ([]byte, error) { @@ -76,7 +87,7 @@ func (sk *opensslPrivateKey) Sign(message []byte) ([]byte, error) { // GetPublic returns a public key func (sk *opensslPrivateKey) GetPublic() PubKey { - return &opensslPublicKey{sk.key} + return &opensslPublicKey{key: sk.key} } func (sk *opensslPrivateKey) Type() pb.KeyType { diff --git a/core/crypto/rsa_go.go b/core/crypto/rsa_go.go index d774991941..f28a327b83 100644 --- a/core/crypto/rsa_go.go +++ b/core/crypto/rsa_go.go @@ -9,6 +9,7 @@ import ( "crypto/x509" "errors" "io" + "sync" pb "github.com/libp2p/go-libp2p-core/crypto/pb" @@ -23,6 +24,9 @@ type RsaPrivateKey struct { // RsaPublicKey is an rsa public key type RsaPublicKey struct { k rsa.PublicKey + + cacheLk sync.Mutex + cached []byte } // GenerateRSAKeyPair generates a new rsa private and public key @@ -35,7 +39,7 @@ func GenerateRSAKeyPair(bits int, src io.Reader) (PrivKey, PubKey, error) { return nil, nil, err } pk := priv.PublicKey - return &RsaPrivateKey{sk: *priv}, &RsaPublicKey{pk}, nil + return &RsaPrivateKey{sk: *priv}, &RsaPublicKey{k: pk}, nil } // Verify compares a signature against input data @@ -54,7 +58,13 @@ func (pk *RsaPublicKey) Type() pb.KeyType { // Bytes returns protobuf bytes of a public key func (pk *RsaPublicKey) Bytes() ([]byte, error) { - return MarshalPublicKey(pk) + pk.cacheLk.Lock() + var err error + if pk.cached == nil { + pk.cached, err = MarshalPublicKey(pk) + } + pk.cacheLk.Unlock() + return pk.cached, err } func (pk *RsaPublicKey) Raw() ([]byte, error) { @@ -80,7 +90,7 @@ func (sk *RsaPrivateKey) Sign(message []byte) ([]byte, error) { // GetPublic returns a public key func (sk *RsaPrivateKey) GetPublic() PubKey { - return &RsaPublicKey{sk.sk.PublicKey} + return &RsaPublicKey{k: sk.sk.PublicKey} } func (sk *RsaPrivateKey) Type() pb.KeyType { @@ -137,5 +147,6 @@ func UnmarshalRsaPublicKey(b []byte) (PubKey, error) { if pk.N.BitLen() < MinRsaKeyBits { return nil, ErrRsaKeyTooSmall } - return &RsaPublicKey{*pk}, nil + + return &RsaPublicKey{k: *pk}, nil } diff --git a/core/crypto/rsa_openssl.go b/core/crypto/rsa_openssl.go index 9b7a342345..8e7fb74315 100644 --- a/core/crypto/rsa_openssl.go +++ b/core/crypto/rsa_openssl.go @@ -29,12 +29,12 @@ func GenerateRSAKeyPair(bits int, _ io.Reader) (PrivKey, PubKey, error) { if err != nil { return nil, nil, err } - return &RsaPrivateKey{opensslPrivateKey{key}}, &RsaPublicKey{opensslPublicKey{key}}, nil + return &RsaPrivateKey{opensslPrivateKey{key}}, &RsaPublicKey{opensslPublicKey{key: key}}, nil } // GetPublic returns a public key func (sk *RsaPrivateKey) GetPublic() PubKey { - return &RsaPublicKey{opensslPublicKey{sk.opensslPrivateKey.key}} + return &RsaPublicKey{opensslPublicKey{key: sk.opensslPrivateKey.key}} } // UnmarshalRsaPrivateKey returns a private key from the input x509 bytes diff --git a/core/event/identify.go b/core/event/identify.go new file mode 100644 index 0000000000..7c4d1895fa --- /dev/null +++ b/core/event/identify.go @@ -0,0 +1,17 @@ +package event + +import "github.com/libp2p/go-libp2p-core/peer" + +// EvtPeerIdentificationCompleted is emitted when the initial identification round for a peer is completed. +type EvtPeerIdentificationCompleted struct { + // Peer is the ID of the peer whose identification succeeded. + Peer peer.ID +} + +// EvtPeerIdentificationFailed is emitted when the initial identification round for a peer failed. +type EvtPeerIdentificationFailed struct { + // Peer is the ID of the peer whose identification failed. + Peer peer.ID + // Reason is the reason why identification failed. + Reason error +} diff --git a/core/event/routability.go b/core/event/routability.go new file mode 100644 index 0000000000..7a25d69cc2 --- /dev/null +++ b/core/event/routability.go @@ -0,0 +1,21 @@ +package event + +// EvtLocalRoutabilityPrivate is an event struct to be emitted with the local's +// node routability changes to PRIVATE (i.e. not routable from the Internet). +// +// This event is usually emitted by the AutoNAT subsystem. +type EvtLocalRoutabilityPrivate struct{} + +// EvtLocalRoutabilityPublic is an event struct to be emitted with the local's +// node routability changes to PUBLIC (i.e. appear to routable from the +// Internet). +// +// This event is usually emitted by the AutoNAT subsystem. +type EvtLocalRoutabilityPublic struct{} + +// EvtLocalRoutabilityUnknown is an event struct to be emitted with the local's +// node routability changes to UNKNOWN (i.e. we were unable to make a +// determination about our NAT status with enough confidence). +// +// This event is usually emitted by the AutoNAT subsystem. +type EvtLocalRoutabilityUnknown struct{} From d0ed06d10be149f71124b66bedef8dbacb0ec515 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 20 Jan 2020 10:30:55 -0500 Subject: [PATCH 1626/3965] add Makefile for generating protobuf code --- p2p/security/noise/pb/Makefile | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 p2p/security/noise/pb/Makefile diff --git a/p2p/security/noise/pb/Makefile b/p2p/security/noise/pb/Makefile new file mode 100644 index 0000000000..7cf8222f89 --- /dev/null +++ b/p2p/security/noise/pb/Makefile @@ -0,0 +1,11 @@ +PB = $(wildcard *.proto) +GO = $(PB:.proto=.pb.go) + +all: $(GO) + +%.pb.go: %.proto + protoc --proto_path=$(PWD):$(PWD)/../.. --gogofaster_out=. $< + +clean: + rm -f *.pb.go + rm -f *.go From a82be49036096c93f44287780a8d32142b591976 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 20 Jan 2020 10:37:23 -0500 Subject: [PATCH 1627/3965] rename protobuf fields --- p2p/security/noise/ik/IK_test.go | 8 +- p2p/security/noise/ik_handshake.go | 4 +- p2p/security/noise/pb/payload.pb.go | 400 +++++++++++++++++++++++++--- p2p/security/noise/pb/payload.proto | 9 +- p2p/security/noise/protocol.go | 6 +- p2p/security/noise/xx/XX_test.go | 8 +- p2p/security/noise/xx_handshake.go | 4 +- 7 files changed, 379 insertions(+), 60 deletions(-) diff --git a/p2p/security/noise/ik/IK_test.go b/p2p/security/noise/ik/IK_test.go index d0beb52c56..f657769a33 100644 --- a/p2p/security/noise/ik/IK_test.go +++ b/p2p/security/noise/ik/IK_test.go @@ -55,8 +55,8 @@ func TestHandshake(t *testing.T) { // stage 0: initiator // create payload payload_init := new(pb.NoiseHandshakePayload) - payload_init.Libp2PKey = libp2p_pub_init_raw - payload_init.NoiseStaticKeySignature = libp2p_init_signed_payload + payload_init.IdentityKey = libp2p_pub_init_raw + payload_init.IdentitySig = libp2p_init_signed_payload payload_init_enc, err := proto.Marshal(payload_init) if err != nil { t.Fatalf("proto marshal payload fail: %s", err) @@ -84,8 +84,8 @@ func TestHandshake(t *testing.T) { // stage 1: responder // create payload payload_resp := new(pb.NoiseHandshakePayload) - payload_resp.Libp2PKey = libp2p_pub_resp_raw - payload_resp.NoiseStaticKeySignature = libp2p_resp_signed_payload + payload_resp.IdentityKey = libp2p_pub_resp_raw + payload_resp.IdentitySig = libp2p_resp_signed_payload payload_resp_enc, err := proto.Marshal(payload_resp) if err != nil { t.Fatalf("proto marshal payload fail: %s", err) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 337b37d8b5..d672e30a37 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -122,7 +122,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] } // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { log.Errorf("runHandshake_ik stage=1 initiator=true set remote peer info err=%s", err) return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=read remote libp2p key fail") @@ -161,7 +161,7 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] } // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err=read remote libp2p key fail") } diff --git a/p2p/security/noise/pb/payload.pb.go b/p2p/security/noise/pb/payload.pb.go index 15fafac6a0..7b912a7da0 100644 --- a/p2p/security/noise/pb/payload.pb.go +++ b/p2p/security/noise/pb/payload.pb.go @@ -1,12 +1,14 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. +// Code generated by protoc-gen-gogo. DO NOT EDIT. // source: payload.proto -package noise +package pb import ( fmt "fmt" - proto "github.com/golang/protobuf/proto" + proto "github.com/gogo/protobuf/proto" + io "io" math "math" + math_bits "math/bits" ) // Reference imports to suppress errors if they are not otherwise used. @@ -18,16 +20,12 @@ var _ = math.Inf // is compatible with the proto package it is being compiled against. // A compilation error at this line likely means your copy of the // proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package type NoiseHandshakePayload struct { - Libp2PKey []byte `protobuf:"bytes,1,opt,name=libp2p_key,json=libp2pKey,proto3" json:"libp2p_key,omitempty"` - NoiseStaticKeySignature []byte `protobuf:"bytes,2,opt,name=noise_static_key_signature,json=noiseStaticKeySignature,proto3" json:"noise_static_key_signature,omitempty"` - Libp2PData []byte `protobuf:"bytes,3,opt,name=libp2p_data,json=libp2pData,proto3" json:"libp2p_data,omitempty"` - Libp2PDataSignature []byte `protobuf:"bytes,4,opt,name=libp2p_data_signature,json=libp2pDataSignature,proto3" json:"libp2p_data_signature,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + IdentityKey []byte `protobuf:"bytes,1,opt,name=identity_key,json=identityKey,proto3" json:"identity_key,omitempty"` + IdentitySig []byte `protobuf:"bytes,2,opt,name=identity_sig,json=identitySig,proto3" json:"identity_sig,omitempty"` + Data []byte `protobuf:"bytes,3,opt,name=data,proto3" json:"data,omitempty"` } func (m *NoiseHandshakePayload) Reset() { *m = NoiseHandshakePayload{} } @@ -36,18 +34,26 @@ func (*NoiseHandshakePayload) ProtoMessage() {} func (*NoiseHandshakePayload) Descriptor() ([]byte, []int) { return fileDescriptor_678c914f1bee6d56, []int{0} } - func (m *NoiseHandshakePayload) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_NoiseHandshakePayload.Unmarshal(m, b) + return m.Unmarshal(b) } func (m *NoiseHandshakePayload) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_NoiseHandshakePayload.Marshal(b, m, deterministic) + if deterministic { + return xxx_messageInfo_NoiseHandshakePayload.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } } func (m *NoiseHandshakePayload) XXX_Merge(src proto.Message) { xxx_messageInfo_NoiseHandshakePayload.Merge(m, src) } func (m *NoiseHandshakePayload) XXX_Size() int { - return xxx_messageInfo_NoiseHandshakePayload.Size(m) + return m.Size() } func (m *NoiseHandshakePayload) XXX_DiscardUnknown() { xxx_messageInfo_NoiseHandshakePayload.DiscardUnknown(m) @@ -55,51 +61,365 @@ func (m *NoiseHandshakePayload) XXX_DiscardUnknown() { var xxx_messageInfo_NoiseHandshakePayload proto.InternalMessageInfo -func (m *NoiseHandshakePayload) GetLibp2PKey() []byte { - if m != nil { - return m.Libp2PKey - } - return nil -} - -func (m *NoiseHandshakePayload) GetNoiseStaticKeySignature() []byte { +func (m *NoiseHandshakePayload) GetIdentityKey() []byte { if m != nil { - return m.NoiseStaticKeySignature + return m.IdentityKey } return nil } -func (m *NoiseHandshakePayload) GetLibp2PData() []byte { +func (m *NoiseHandshakePayload) GetIdentitySig() []byte { if m != nil { - return m.Libp2PData + return m.IdentitySig } return nil } -func (m *NoiseHandshakePayload) GetLibp2PDataSignature() []byte { +func (m *NoiseHandshakePayload) GetData() []byte { if m != nil { - return m.Libp2PDataSignature + return m.Data } return nil } func init() { - proto.RegisterType((*NoiseHandshakePayload)(nil), "noise.NoiseHandshakePayload") + proto.RegisterType((*NoiseHandshakePayload)(nil), "pb.NoiseHandshakePayload") } func init() { proto.RegisterFile("payload.proto", fileDescriptor_678c914f1bee6d56) } var fileDescriptor_678c914f1bee6d56 = []byte{ - // 176 bytes of a gzipped FileDescriptorProto + // 152 bytes of a gzipped FileDescriptorProto 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x2d, 0x48, 0xac, 0xcc, - 0xc9, 0x4f, 0x4c, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0xcd, 0xcb, 0xcf, 0x2c, 0x4e, - 0x55, 0x3a, 0xc1, 0xc8, 0x25, 0xea, 0x07, 0x62, 0x79, 0x24, 0xe6, 0xa5, 0x14, 0x67, 0x24, 0x66, - 0xa7, 0x06, 0x40, 0x94, 0x09, 0xc9, 0x72, 0x71, 0xe5, 0x64, 0x26, 0x15, 0x18, 0x15, 0xc4, 0x67, - 0xa7, 0x56, 0x4a, 0x30, 0x2a, 0x30, 0x6a, 0xf0, 0x04, 0x71, 0x42, 0x44, 0xbc, 0x53, 0x2b, 0x85, - 0xac, 0xb9, 0xa4, 0xc0, 0x26, 0xc4, 0x17, 0x97, 0x24, 0x96, 0x64, 0x26, 0x83, 0x14, 0xc5, 0x17, - 0x67, 0xa6, 0xe7, 0x25, 0x96, 0x94, 0x16, 0xa5, 0x4a, 0x30, 0x81, 0x95, 0x8b, 0x83, 0x55, 0x04, - 0x83, 0x15, 0x78, 0xa7, 0x56, 0x06, 0xc3, 0xa4, 0x85, 0xe4, 0xb9, 0xb8, 0xa1, 0x66, 0xa7, 0x24, - 0x96, 0x24, 0x4a, 0x30, 0x83, 0x55, 0x43, 0xad, 0x73, 0x49, 0x2c, 0x49, 0x14, 0x32, 0xe2, 0x12, - 0x45, 0x52, 0x80, 0x64, 0x30, 0x0b, 0x58, 0xa9, 0x30, 0x42, 0x29, 0xdc, 0xd0, 0x24, 0x36, 0xb0, - 0xc7, 0x8c, 0x01, 0x01, 0x00, 0x00, 0xff, 0xff, 0x0f, 0xcc, 0x4e, 0xb9, 0xe9, 0x00, 0x00, 0x00, + 0xc9, 0x4f, 0x4c, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0x2a, 0x48, 0x52, 0x2a, 0xe4, + 0x12, 0xf5, 0xcb, 0xcf, 0x2c, 0x4e, 0xf5, 0x48, 0xcc, 0x4b, 0x29, 0xce, 0x48, 0xcc, 0x4e, 0x0d, + 0x80, 0x28, 0x11, 0x52, 0xe4, 0xe2, 0xc9, 0x4c, 0x49, 0xcd, 0x2b, 0xc9, 0x2c, 0xa9, 0x8c, 0xcf, + 0x4e, 0xad, 0x94, 0x60, 0x54, 0x60, 0xd4, 0xe0, 0x09, 0xe2, 0x86, 0x89, 0x79, 0xa7, 0x56, 0xa2, + 0x28, 0x29, 0xce, 0x4c, 0x97, 0x60, 0x42, 0x55, 0x12, 0x9c, 0x99, 0x2e, 0x24, 0xc4, 0xc5, 0x92, + 0x92, 0x58, 0x92, 0x28, 0xc1, 0x0c, 0x96, 0x02, 0xb3, 0x9d, 0x24, 0x4e, 0x3c, 0x92, 0x63, 0xbc, + 0xf0, 0x48, 0x8e, 0xf1, 0xc1, 0x23, 0x39, 0xc6, 0x09, 0x8f, 0xe5, 0x18, 0x2e, 0x3c, 0x96, 0x63, + 0xb8, 0xf1, 0x58, 0x8e, 0x21, 0x89, 0x0d, 0xec, 0x2e, 0x63, 0x40, 0x00, 0x00, 0x00, 0xff, 0xff, + 0x51, 0x37, 0xd7, 0x40, 0xa8, 0x00, 0x00, 0x00, +} + +func (m *NoiseHandshakePayload) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil } + +func (m *NoiseHandshakePayload) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *NoiseHandshakePayload) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if len(m.Data) > 0 { + i -= len(m.Data) + copy(dAtA[i:], m.Data) + i = encodeVarintPayload(dAtA, i, uint64(len(m.Data))) + i-- + dAtA[i] = 0x1a + } + if len(m.IdentitySig) > 0 { + i -= len(m.IdentitySig) + copy(dAtA[i:], m.IdentitySig) + i = encodeVarintPayload(dAtA, i, uint64(len(m.IdentitySig))) + i-- + dAtA[i] = 0x12 + } + if len(m.IdentityKey) > 0 { + i -= len(m.IdentityKey) + copy(dAtA[i:], m.IdentityKey) + i = encodeVarintPayload(dAtA, i, uint64(len(m.IdentityKey))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil +} + +func encodeVarintPayload(dAtA []byte, offset int, v uint64) int { + offset -= sovPayload(v) + base := offset + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return base +} +func (m *NoiseHandshakePayload) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + l = len(m.IdentityKey) + if l > 0 { + n += 1 + l + sovPayload(uint64(l)) + } + l = len(m.IdentitySig) + if l > 0 { + n += 1 + l + sovPayload(uint64(l)) + } + l = len(m.Data) + if l > 0 { + n += 1 + l + sovPayload(uint64(l)) + } + return n +} + +func sovPayload(x uint64) (n int) { + return (math_bits.Len64(x|1) + 6) / 7 +} +func sozPayload(x uint64) (n int) { + return sovPayload(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *NoiseHandshakePayload) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPayload + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: NoiseHandshakePayload: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: NoiseHandshakePayload: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field IdentityKey", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPayload + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPayload + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPayload + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.IdentityKey = append(m.IdentityKey[:0], dAtA[iNdEx:postIndex]...) + if m.IdentityKey == nil { + m.IdentityKey = []byte{} + } + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field IdentitySig", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPayload + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPayload + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPayload + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.IdentitySig = append(m.IdentitySig[:0], dAtA[iNdEx:postIndex]...) + if m.IdentitySig == nil { + m.IdentitySig = []byte{} + } + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPayload + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPayload + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPayload + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipPayload(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPayload + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthPayload + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipPayload(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + depth := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPayload + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPayload + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + case 1: + iNdEx += 8 + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPayload + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if length < 0 { + return 0, ErrInvalidLengthPayload + } + iNdEx += length + case 3: + depth++ + case 4: + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupPayload + } + depth-- + case 5: + iNdEx += 4 + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + if iNdEx < 0 { + return 0, ErrInvalidLengthPayload + } + if depth == 0 { + return iNdEx, nil + } + } + return 0, io.ErrUnexpectedEOF +} + +var ( + ErrInvalidLengthPayload = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowPayload = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupPayload = fmt.Errorf("proto: unexpected end of group") +) diff --git a/p2p/security/noise/pb/payload.proto b/p2p/security/noise/pb/payload.proto index 42243b247f..05a78c6f39 100644 --- a/p2p/security/noise/pb/payload.proto +++ b/p2p/security/noise/pb/payload.proto @@ -2,8 +2,7 @@ syntax = "proto3"; package pb; message NoiseHandshakePayload { - bytes libp2p_key = 1; - bytes noise_static_key_signature = 2; - bytes libp2p_data = 3; - bytes libp2p_data_signature = 4; -} \ No newline at end of file + bytes identity_key = 1; + bytes identity_sig = 2; + bytes data = 3; +} diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 591961c615..48b4c41ee5 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -142,7 +142,7 @@ func (s *secureSession) setRemotePeerID(key crypto.PubKey) (err error) { } func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKey [32]byte) (err error) { - sig := payload.GetNoiseStaticKeySignature() + sig := payload.GetIdentitySig() msg := append([]byte(payload_string), noiseKey[:]...) log.Debugf("verifyPayload msg=%x", msg) @@ -174,8 +174,8 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // create payload payload := new(pb.NoiseHandshakePayload) - payload.Libp2PKey = localKeyRaw - payload.NoiseStaticKeySignature = signedPayload + payload.IdentityKey = localKeyRaw + payload.IdentitySig = signedPayload payloadEnc, err := proto.Marshal(payload) if err != nil { log.Errorf("runHandshake marshal payload err=%s", err) diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 3445a4bc24..4949978e75 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -75,8 +75,8 @@ func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { // stage 0: initiator // create payload payload_init := new(pb.NoiseHandshakePayload) - payload_init.Libp2PKey = libp2p_pub_init_raw - payload_init.NoiseStaticKeySignature = libp2p_init_signed_payload + payload_init.IdentityKey = libp2p_pub_init_raw + payload_init.IdentitySig = libp2p_init_signed_payload payload_init_enc, err := proto.Marshal(payload_init) if err != nil { t.Fatalf("proto marshal payload fail: %s", err) @@ -104,8 +104,8 @@ func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { // stage 1: responder // create payload payload_resp := new(pb.NoiseHandshakePayload) - payload_resp.Libp2PKey = libp2p_pub_resp_raw - payload_resp.NoiseStaticKeySignature = libp2p_resp_signed_payload + payload_resp.IdentityKey = libp2p_pub_resp_raw + payload_resp.IdentitySig = libp2p_resp_signed_payload payload_resp_enc, err := proto.Marshal(payload_resp) if err != nil { t.Fatalf("proto marshal payload fail: %s", err) diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 92f9002fcc..b32a6c126e 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -157,7 +157,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { log.Errorf("runHandshake_xx stage=2 initiator=true set remote peer info err=%s", err) return fmt.Errorf("runHandshake_xx stage=2 initiator=true read remote libp2p key fail") @@ -246,7 +246,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetLibp2PKey()) + err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { log.Errorf("runHandshake_xx stage=2 initiator=false set remote peer info err=%s", err) return fmt.Errorf("runHandshake_xx stage=2 initiator=false read remote libp2p key fail") From 6463b1cf6194716c459845467e3bb91540eadc9b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 22 Jan 2020 17:51:44 +0000 Subject: [PATCH 1628/3965] add licenses. (#69) --- p2p/transport/websocket/LICENSE-APACHE | 5 +++++ p2p/transport/websocket/LICENSE-MIT | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+) create mode 100644 p2p/transport/websocket/LICENSE-APACHE create mode 100644 p2p/transport/websocket/LICENSE-MIT diff --git a/p2p/transport/websocket/LICENSE-APACHE b/p2p/transport/websocket/LICENSE-APACHE new file mode 100644 index 0000000000..14478a3b60 --- /dev/null +++ b/p2p/transport/websocket/LICENSE-APACHE @@ -0,0 +1,5 @@ +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. diff --git a/p2p/transport/websocket/LICENSE-MIT b/p2p/transport/websocket/LICENSE-MIT new file mode 100644 index 0000000000..72dc60d84b --- /dev/null +++ b/p2p/transport/websocket/LICENSE-MIT @@ -0,0 +1,19 @@ +The MIT License (MIT) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. From 8c0011194e4efbc9d38b0b2c7f2d3685bc01e43a Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Tue, 28 Jan 2020 13:53:25 -0500 Subject: [PATCH 1629/3965] fix protocol id in integration test --- p2p/security/noise/integration_test.go | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index 616ab3b523..279ad9d671 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -17,6 +17,8 @@ import ( "time" ) +const testProtocolID = "/test/noise/integration" + func generateKey(seed int64) (crypto.PrivKey, error) { var r io.Reader if seed == 0 { @@ -109,8 +111,8 @@ func TestLibp2pIntegration_NoPipes(t *testing.T) { defer hb.Close() - ha.SetStreamHandler(ID, handleStream) - hb.SetStreamHandler(ID, handleStream) + ha.SetStreamHandler(testProtocolID, handleStream) + hb.SetStreamHandler(testProtocolID, handleStream) addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", hb.Addrs()[0].String(), hb.ID())) if err != nil { @@ -129,7 +131,7 @@ func TestLibp2pIntegration_NoPipes(t *testing.T) { t.Fatal(err) } - stream, err := ha.NewStream(ctx, hb.ID(), ID) + stream, err := ha.NewStream(ctx, hb.ID(), testProtocolID) if err != nil { t.Fatal(err) } @@ -166,8 +168,8 @@ func TestLibp2pIntegration_WithPipes(t *testing.T) { defer hb.Close() - ha.SetStreamHandler(ID, handleStream) - hb.SetStreamHandler(ID, handleStream) + ha.SetStreamHandler(testProtocolID, handleStream) + hb.SetStreamHandler(testProtocolID, handleStream) addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", ha.Addrs()[0].String(), ha.ID())) if err != nil { @@ -186,7 +188,7 @@ func TestLibp2pIntegration_WithPipes(t *testing.T) { t.Fatal(err) } - stream, err := hb.NewStream(ctx, ha.ID(), ID) + stream, err := hb.NewStream(ctx, ha.ID(), testProtocolID) if err != nil { t.Fatal(err) } @@ -223,8 +225,8 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { defer hb.Close() - ha.SetStreamHandler(ID, handleStream) - hb.SetStreamHandler(ID, handleStream) + ha.SetStreamHandler(testProtocolID, handleStream) + hb.SetStreamHandler(testProtocolID, handleStream) addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", hb.Addrs()[0].String(), hb.ID())) if err != nil { @@ -243,7 +245,7 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { t.Fatal(err) } - stream, err := hb.NewStream(ctx, ha.ID(), ID) + stream, err := hb.NewStream(ctx, ha.ID(), testProtocolID) if err != nil { t.Fatal(err) } From 297dd7fae97bbd2cf4c169b96114f5cca192b7a2 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 29 Jan 2020 09:05:19 -0500 Subject: [PATCH 1630/3965] send much more data in integration test --- p2p/security/noise/integration_test.go | 62 +++++++++++++++++--------- 1 file changed, 41 insertions(+), 21 deletions(-) diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index 279ad9d671..0cb2a93019 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -1,7 +1,6 @@ package noise import ( - "bufio" "context" "crypto/rand" "fmt" @@ -12,6 +11,7 @@ import ( "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" "io" + "io/ioutil" mrand "math/rand" "testing" "time" @@ -111,8 +111,8 @@ func TestLibp2pIntegration_NoPipes(t *testing.T) { defer hb.Close() - ha.SetStreamHandler(testProtocolID, handleStream) - hb.SetStreamHandler(testProtocolID, handleStream) + ha.SetStreamHandler(testProtocolID, streamHandler(t)) + hb.SetStreamHandler(testProtocolID, streamHandler(t)) addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", hb.Addrs()[0].String(), hb.ID())) if err != nil { @@ -136,7 +136,7 @@ func TestLibp2pIntegration_NoPipes(t *testing.T) { t.Fatal(err) } - _, err = stream.Write([]byte("hello\n")) + err = writeRandomPayloadAndClose(t, stream) if err != nil { t.Fatal(err) } @@ -168,8 +168,8 @@ func TestLibp2pIntegration_WithPipes(t *testing.T) { defer hb.Close() - ha.SetStreamHandler(testProtocolID, handleStream) - hb.SetStreamHandler(testProtocolID, handleStream) + ha.SetStreamHandler(testProtocolID, streamHandler(t)) + hb.SetStreamHandler(testProtocolID, streamHandler(t)) addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", ha.Addrs()[0].String(), ha.ID())) if err != nil { @@ -193,7 +193,7 @@ func TestLibp2pIntegration_WithPipes(t *testing.T) { t.Fatal(err) } - _, err = stream.Write([]byte("hello\n")) + err = writeRandomPayloadAndClose(t, stream) if err != nil { t.Fatal(err) } @@ -225,8 +225,8 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { defer hb.Close() - ha.SetStreamHandler(testProtocolID, handleStream) - hb.SetStreamHandler(testProtocolID, handleStream) + ha.SetStreamHandler(testProtocolID, streamHandler(t)) + hb.SetStreamHandler(testProtocolID, streamHandler(t)) addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", hb.Addrs()[0].String(), hb.ID())) if err != nil { @@ -250,7 +250,7 @@ func TestLibp2pIntegration_XXFallback(t *testing.T) { t.Fatal(err) } - _, err = stream.Write([]byte("hello\n")) + err = writeRandomPayloadAndClose(t, stream) if err != nil { t.Fatal(err) } @@ -277,18 +277,38 @@ func TestConstrucingWithMaker(t *testing.T) { _ = h.Close() } -func handleStream(stream net.Stream) { - defer func() { - if err := stream.Close(); err != nil { - log.Error("error closing stream", "err", err) - } - }() +func writeRandomPayloadAndClose(t *testing.T, stream net.Stream) error { + t.Helper() + size := 1 << 24 + r := mrand.New(mrand.NewSource(42)) + start := time.Now() + lr := io.LimitReader(r, int64(size)) - rw := bufio.NewReadWriter(bufio.NewReader(stream), bufio.NewWriter(stream)) - msg, err := rw.Reader.ReadString('\n') + c, err := io.Copy(stream, lr) + elapsed := time.Since(start) if err != nil { - fmt.Println("stream err", err) - return + return fmt.Errorf("failed to write out bytes: %v", err) + } + t.Logf("wrote %d bytes in %dms", c, elapsed.Milliseconds()) + return stream.Close() +} + +func streamHandler(t *testing.T) func(net.Stream) { + return func(stream net.Stream) { + t.Helper() + defer func() { + if err := stream.Close(); err != nil { + t.Error("error closing stream: ", err) + } + }() + + start := time.Now() + c, err := io.Copy(ioutil.Discard, stream) + elapsed := time.Since(start) + if err != nil { + t.Error("error reading from stream: ", err) + return + } + t.Logf("read %d bytes in %dms", c, elapsed.Milliseconds()) } - fmt.Printf("got msg: %s", msg) } From 7df3fe54e90988d808dc3ce518d0eda2d035bfd7 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 29 Jan 2020 09:07:06 -0500 Subject: [PATCH 1631/3965] make sure we fill buffer when reading from conn adds fillBuffer and writeAll helpers to make sure that we're actually filling our input buffers when reading from the insecure conn, and that we're writing the entire output buffer, even if it takes multiple calls to insecure.Read or insecure.Write --- p2p/security/noise/ik_handshake.go | 4 ++-- p2p/security/noise/protocol.go | 28 +++++++++++++++------------ p2p/security/noise/util.go | 31 ++++++++++++++++++++++++++++++ p2p/security/noise/xx_handshake.go | 4 ++-- 4 files changed, 51 insertions(+), 16 deletions(-) create mode 100644 p2p/security/noise/util.go diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index d672e30a37..428c4cd565 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -28,7 +28,7 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo } // send message - _, err = s.insecure.Write(encMsgBuf) + _, err = writeAll(s.insecure, encMsgBuf) if err != nil { log.Error("ik_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("ik_sendHandshakeMessage write to conn err=%s", err) @@ -45,7 +45,7 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, buf = make([]byte, l) - _, err = s.insecure.Read(buf) + _, err = fillBuffer(buf, s.insecure) if err != nil { return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage read from conn err=%s", err) } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 48b4c41ee5..3d980cf52d 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -9,14 +9,14 @@ import ( "sync" "time" - proto "github.com/gogo/protobuf/proto" + "github.com/gogo/protobuf/proto" logging "github.com/ipfs/go-log" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" - ik "github.com/libp2p/go-libp2p-noise/ik" - pb "github.com/libp2p/go-libp2p-noise/pb" - xx "github.com/libp2p/go-libp2p-noise/xx" + "github.com/libp2p/go-libp2p-noise/ik" + "github.com/libp2p/go-libp2p-noise/pb" + "github.com/libp2p/go-libp2p-noise/xx" ) const payload_string = "noise-libp2p-static-key:" @@ -55,7 +55,8 @@ type secureSession struct { noiseKeypair *Keypair msgBuffer []byte - rwLock sync.Mutex + readLock sync.Mutex + writeLock sync.Mutex } type peerInfo struct { @@ -120,14 +121,14 @@ func (s *secureSession) NoisePrivateKey() [32]byte { func (s *secureSession) readLength() (int, error) { buf := make([]byte, 2) - _, err := s.insecure.Read(buf) + _, err := fillBuffer(buf, s.insecure) return int(binary.BigEndian.Uint16(buf)), err } func (s *secureSession) writeLength(length int) error { buf := make([]byte, 2) binary.BigEndian.PutUint16(buf, uint16(length)) - _, err := s.insecure.Write(buf) + _, err := writeAll(s.insecure, buf) return err } @@ -237,6 +238,9 @@ func (s *secureSession) LocalPublicKey() crypto.PubKey { } func (s *secureSession) Read(buf []byte) (int, error) { + s.readLock.Lock() + defer s.readLock.Unlock() + l := len(buf) // if we have previously unread bytes, and they fit into the buf, copy them over and return @@ -255,7 +259,7 @@ func (s *secureSession) Read(buf []byte) (int, error) { // read and decrypt ciphertext ciphertext := make([]byte, l) - _, err = s.insecure.Read(ciphertext) + _, err = fillBuffer(ciphertext, s.insecure) if err != nil { log.Error("read ciphertext err", err) return 0, err @@ -317,8 +321,8 @@ func (s *secureSession) SetWriteDeadline(t time.Time) error { } func (s *secureSession) Write(in []byte) (int, error) { - s.rwLock.Lock() - defer s.rwLock.Unlock() + s.writeLock.Lock() + defer s.writeLock.Unlock() writeChunk := func(in []byte) (int, error) { ciphertext, err := s.Encrypt(in) @@ -329,11 +333,11 @@ func (s *secureSession) Write(in []byte) (int, error) { err = s.writeLength(len(ciphertext)) if err != nil { - log.Error("write length err", err) + log.Error("write length err: ", err) return 0, err } - _, err = s.insecure.Write(ciphertext) + _, err = writeAll(s.insecure, ciphertext) return len(in), err } diff --git a/p2p/security/noise/util.go b/p2p/security/noise/util.go new file mode 100644 index 0000000000..a20af67383 --- /dev/null +++ b/p2p/security/noise/util.go @@ -0,0 +1,31 @@ +package noise + +import "io" + +// fillBuffer reads from the given reader until the given buffer +// is full +func fillBuffer(buf []byte, reader io.Reader) (int, error) { + total := 0 + for total < len(buf) { + c, err := reader.Read(buf[total:]) + if err != nil { + return total, err + } + total += c + } + return total, nil +} + +// writeAll is a helper that writes to the given io.Writer until all input data +// has been written +func writeAll(writer io.Writer, data []byte) (int, error) { + total := 0 + for total < len(data) { + c, err := writer.Write(data[total:]) + if err != nil { + return total, err + } + total += c + } + return total, nil +} diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index b32a6c126e..f3c0ef1ab6 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -27,7 +27,7 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo return fmt.Errorf("xx_sendHandshakeMessage write length err=%s", err) } - _, err = s.insecure.Write(encMsgBuf) + _, err = writeAll(s.insecure, encMsgBuf) if err != nil { log.Error("xx_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("xx_sendHandshakeMessage write to conn err=%s", err) @@ -46,7 +46,7 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, buf = make([]byte, l) - _, err = s.insecure.Read(buf) + _, err = fillBuffer(buf, s.insecure) if err != nil { return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage read from conn err=%s", err) } From 268e9e2936b73ae4f05332709a57cf1b74ba1e09 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Wed, 29 Jan 2020 09:55:17 -0500 Subject: [PATCH 1632/3965] use io.ReadFull instead of custom helper --- p2p/security/noise/ik_handshake.go | 5 +++-- p2p/security/noise/protocol.go | 9 +++++---- p2p/security/noise/util.go | 31 ------------------------------ p2p/security/noise/xx_handshake.go | 5 +++-- 4 files changed, 11 insertions(+), 39 deletions(-) delete mode 100644 p2p/security/noise/util.go diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 428c4cd565..2978e957f3 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -3,6 +3,7 @@ package noise import ( "context" "fmt" + "io" proto "github.com/gogo/protobuf/proto" ik "github.com/libp2p/go-libp2p-noise/ik" @@ -28,7 +29,7 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo } // send message - _, err = writeAll(s.insecure, encMsgBuf) + _, err = s.insecure.Write(encMsgBuf) if err != nil { log.Error("ik_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("ik_sendHandshakeMessage write to conn err=%s", err) @@ -45,7 +46,7 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, buf = make([]byte, l) - _, err = fillBuffer(buf, s.insecure) + _, err = io.ReadFull(s.insecure, buf) if err != nil { return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage read from conn err=%s", err) } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 3d980cf52d..628d0cfe50 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -5,6 +5,7 @@ import ( "encoding/binary" "errors" "fmt" + "io" "net" "sync" "time" @@ -121,14 +122,14 @@ func (s *secureSession) NoisePrivateKey() [32]byte { func (s *secureSession) readLength() (int, error) { buf := make([]byte, 2) - _, err := fillBuffer(buf, s.insecure) + _, err := io.ReadFull(s.insecure, buf) return int(binary.BigEndian.Uint16(buf)), err } func (s *secureSession) writeLength(length int) error { buf := make([]byte, 2) binary.BigEndian.PutUint16(buf, uint16(length)) - _, err := writeAll(s.insecure, buf) + _, err := s.insecure.Write(buf) return err } @@ -259,7 +260,7 @@ func (s *secureSession) Read(buf []byte) (int, error) { // read and decrypt ciphertext ciphertext := make([]byte, l) - _, err = fillBuffer(ciphertext, s.insecure) + _, err = io.ReadFull(s.insecure, ciphertext) if err != nil { log.Error("read ciphertext err", err) return 0, err @@ -337,7 +338,7 @@ func (s *secureSession) Write(in []byte) (int, error) { return 0, err } - _, err = writeAll(s.insecure, ciphertext) + _, err = s.insecure.Write(ciphertext) return len(in), err } diff --git a/p2p/security/noise/util.go b/p2p/security/noise/util.go deleted file mode 100644 index a20af67383..0000000000 --- a/p2p/security/noise/util.go +++ /dev/null @@ -1,31 +0,0 @@ -package noise - -import "io" - -// fillBuffer reads from the given reader until the given buffer -// is full -func fillBuffer(buf []byte, reader io.Reader) (int, error) { - total := 0 - for total < len(buf) { - c, err := reader.Read(buf[total:]) - if err != nil { - return total, err - } - total += c - } - return total, nil -} - -// writeAll is a helper that writes to the given io.Writer until all input data -// has been written -func writeAll(writer io.Writer, data []byte) (int, error) { - total := 0 - for total < len(data) { - c, err := writer.Write(data[total:]) - if err != nil { - return total, err - } - total += c - } - return total, nil -} diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index f3c0ef1ab6..a0a32bdd1d 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -3,6 +3,7 @@ package noise import ( "context" "fmt" + "io" proto "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/peer" @@ -27,7 +28,7 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo return fmt.Errorf("xx_sendHandshakeMessage write length err=%s", err) } - _, err = writeAll(s.insecure, encMsgBuf) + _, err = s.insecure.Write(encMsgBuf) if err != nil { log.Error("xx_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("xx_sendHandshakeMessage write to conn err=%s", err) @@ -46,7 +47,7 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, buf = make([]byte, l) - _, err = fillBuffer(buf, s.insecure) + _, err = io.ReadFull(s.insecure, buf) if err != nil { return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage read from conn err=%s", err) } From 8b7f580cbf529aa8c015d02b44d98493ace74c65 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 3 Feb 2020 10:15:20 -0500 Subject: [PATCH 1633/3965] don't log all errors Also, return errors when validating handshake payloads and remote peer IDs. These were being logged but not returned, so the handshake would complete successfully even if the payload signature was invalid --- p2p/security/noise/ik_handshake.go | 10 ++-------- p2p/security/noise/protocol.go | 10 ---------- p2p/security/noise/xx_handshake.go | 19 +++++-------------- 3 files changed, 7 insertions(+), 32 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 2978e957f3..f7859ffa6c 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -24,14 +24,12 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo err := s.writeLength(len(encMsgBuf)) if err != nil { - log.Error("ik_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("ik_sendHandshakeMessage write length err=%s", err) } // send message _, err = s.insecure.Write(encMsgBuf) if err != nil { - log.Error("ik_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("ik_sendHandshakeMessage write to conn err=%s", err) } @@ -61,13 +59,11 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, log.Debugf("ik_recvHandshakeMessage initiator=%v msgbuf=%v", s.initiator, msgbuf) if err != nil { - log.Errorf("ik_recvHandshakeMessage initiator=%v decode err=%s", s.initiator, err) return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage decode msg fail: %s", err) } s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) if !valid { - log.Errorf("ik_recvHandshakeMessage initiator=%v err=%s", s.initiator, "validation fail") return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage validation fail") } @@ -99,7 +95,6 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] // stage 0 // err := s.ik_sendHandshakeMessage(payload, true) if err != nil { - log.Errorf("runHandshake_ik stage=0 initiator=true send err=%s", err) return nil, fmt.Errorf("runHandshake_ik stage=0 initiator=true err=%s", err) } @@ -125,20 +120,19 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { - log.Errorf("runHandshake_ik stage=1 initiator=true set remote peer info err=%s", err) return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=read remote libp2p key fail") } // assert that remote peer ID matches libp2p key err = s.setRemotePeerID(s.RemotePublicKey()) if err != nil { - log.Errorf("runHandshake_ik stage=1 initiator=true set remote peer id err=%s", err) + return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=%s", err) } // verify payload is signed by libp2p key err = s.verifyPayload(nhp, remoteNoiseKey) if err != nil { - log.Errorf("runHandshake_ik stage=1 initiator=true verify payload err=%s", err) + return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true verify payload err=%s", err) } } else { diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 628d0cfe50..78c24b6638 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -170,7 +170,6 @@ func (s *secureSession) runHandshake(ctx context.Context) error { noise_pub := s.noiseKeypair.publicKey signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) if err != nil { - log.Errorf("runHandshake signing payload err=%s", err) return fmt.Errorf("runHandshake signing payload err=%s", err) } @@ -180,7 +179,6 @@ func (s *secureSession) runHandshake(ctx context.Context) error { payload.IdentitySig = signedPayload payloadEnc, err := proto.Marshal(payload) if err != nil { - log.Errorf("runHandshake marshal payload err=%s", err) return fmt.Errorf("runHandshake proto marshal payload err=%s", err) } @@ -195,12 +193,9 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // we're either a responder or an initiator with a known static key for the remote peer, try IK buf, err := s.runHandshake_ik(ctx, payloadEnc) if err != nil { - log.Error("runHandshake ik err=%s", err) - // IK failed, pipe to XXfallback err = s.runHandshake_xx(ctx, true, payloadEnc, buf) if err != nil { - log.Error("runHandshake xx err=err", err) return fmt.Errorf("runHandshake xx err=%s", err) } @@ -212,7 +207,6 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // unknown static key for peer, try XX err := s.runHandshake_xx(ctx, false, payloadEnc, nil) if err != nil { - log.Error("runHandshake xx err=%s", err) return err } @@ -262,13 +256,11 @@ func (s *secureSession) Read(buf []byte) (int, error) { ciphertext := make([]byte, l) _, err = io.ReadFull(s.insecure, ciphertext) if err != nil { - log.Error("read ciphertext err", err) return 0, err } plaintext, err := s.Decrypt(ciphertext) if err != nil { - log.Error("decrypt err", err) return 0, err } @@ -328,13 +320,11 @@ func (s *secureSession) Write(in []byte) (int, error) { writeChunk := func(in []byte) (int, error) { ciphertext, err := s.Encrypt(in) if err != nil { - log.Error("encrypt error", err) return 0, err } err = s.writeLength(len(ciphertext)) if err != nil { - log.Error("write length err: ", err) return 0, err } diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index a0a32bdd1d..dccbb33df9 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -24,13 +24,11 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo err := s.writeLength(len(encMsgBuf)) if err != nil { - log.Error("xx_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("xx_sendHandshakeMessage write length err=%s", err) } _, err = s.insecure.Write(encMsgBuf) if err != nil { - log.Error("xx_sendHandshakeMessage initiator=%v err=%s", s.initiator, err) return fmt.Errorf("xx_sendHandshakeMessage write to conn err=%s", err) } @@ -66,7 +64,6 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) if !valid { - log.Error("xx_recvHandshakeMessage initiator=%v err=validation fail", s.initiator) return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage validation fail") } @@ -137,7 +134,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) if !valid { - log.Errorf("xx_recvHandshakeMessage initiator=%v", s.initiator, "error", "validation fail") return fmt.Errorf("runHandshake_xx validation fail") } @@ -160,22 +156,21 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { - log.Errorf("runHandshake_xx stage=2 initiator=true set remote peer info err=%s", err) return fmt.Errorf("runHandshake_xx stage=2 initiator=true read remote libp2p key fail") } // assert that remote peer ID matches libp2p public key pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) if pid != s.remotePeer { - log.Errorf("runHandshake_xx stage=2 initiator=true check remote peer id err: expected %x got %x", s.remotePeer, pid) + return fmt.Errorf("runHandshake_xx stage=2 initiator=true check remote peer id err: expected %x got %x", s.remotePeer, pid) } else if err != nil { - log.Errorf("runHandshake_xx stage 2 initiator check remote peer id err %s", err) + return fmt.Errorf("runHandshake_xx stage 2 initiator check remote peer id err %s", err) } // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.xx_ns.RemoteKey()) if err != nil { - log.Errorf("runHandshake_xx stage=2 initiator=true verify payload err=%s", err) + return fmt.Errorf("runHandshake_xx stage=2 initiator=true verify payload err=%s", err) } if s.noisePipesSupport { @@ -205,7 +200,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl var msgbuf *xx.MessageBuffer msgbuf, err = xx.Decode0(initialMsg) if err != nil { - log.Errorf("runHandshake_xx recv msg err", err) return err } @@ -214,7 +208,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, &xx_msgbuf) if !valid { - log.Errorf("runHandshake_xx initiator=false recv msg err=%s", "validation fail") return fmt.Errorf("runHandshake_xx validation fail") } } @@ -249,14 +242,13 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { - log.Errorf("runHandshake_xx stage=2 initiator=false set remote peer info err=%s", err) return fmt.Errorf("runHandshake_xx stage=2 initiator=false read remote libp2p key fail") } - // assert that remote peer ID matches libp2p key + // set remote libp2p public key from payload err = s.setRemotePeerID(s.RemotePublicKey()) if err != nil { - log.Errorf("runHandshake_xx stage=2 initiator=false set remote peer id err=%s", err) + return fmt.Errorf("runHandshake_xx stage=2 initiator=false set remote peer id err=%s", err) } s.remote.noiseKey = s.xx_ns.RemoteKey() @@ -264,7 +256,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.remote.noiseKey) if err != nil { - log.Errorf("runHandshake_xx stage=2 initiator=false verify payload err=%s", err) return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=%s", err) } From 4d60c61b7cbe0e653028b7b966e0f4db7a67a3fb Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 3 Feb 2020 10:18:02 -0500 Subject: [PATCH 1634/3965] rm debug logs --- p2p/security/noise/ik_handshake.go | 8 -------- p2p/security/noise/protocol.go | 2 -- p2p/security/noise/xx_handshake.go | 21 +-------------------- 3 files changed, 1 insertion(+), 30 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index f7859ffa6c..4aaaab274e 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -20,8 +20,6 @@ func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bo encMsgBuf = msgbuf.Encode1() } - log.Debugf("ik_sendHandshakeMessage initiator=%v msgbuf=%v", s.initiator, msgbuf) - err := s.writeLength(len(encMsgBuf)) if err != nil { return fmt.Errorf("ik_sendHandshakeMessage write length err=%s", err) @@ -56,8 +54,6 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, msgbuf, err = ik.Decode1(buf) } - log.Debugf("ik_recvHandshakeMessage initiator=%v msgbuf=%v", s.initiator, msgbuf) - if err != nil { return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage decode msg fail: %s", err) } @@ -79,8 +75,6 @@ func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([]byte, error) { kp := ik.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) - log.Debugf("runHandshake_ik initiator=%v pubkey=%x", kp.PubKey(), s.initiator) - remoteNoiseKey := s.noiseStaticKeyCache.Load(s.remotePeer) // new IK noise session @@ -181,7 +175,5 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] } } - - log.Debugf("runHandshake_ik done initiator=%v", s.initiator) return nil, nil } diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 78c24b6638..4f5b77833b 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -147,8 +147,6 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe sig := payload.GetIdentitySig() msg := append([]byte(payload_string), noiseKey[:]...) - log.Debugf("verifyPayload msg=%x", msg) - ok, err := s.RemotePublicKey().Verify(msg, sig) if err != nil { return err diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index dccbb33df9..8bc16d0ce2 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -32,8 +32,6 @@ func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bo return fmt.Errorf("xx_sendHandshakeMessage write to conn err=%s", err) } - log.Debugf("xx_sendHandshakeMessage initiator=%v msgbuf=%v initial_stage=%v", s.initiator, msgbuf, initial_stage) - return nil } @@ -58,7 +56,6 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, } if err != nil { - log.Debugf("xx_recvHandshakeMessage initiator=%v decode err=%s", s.initiator, err) return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage decode msg err=%s", err) } @@ -67,8 +64,6 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage validation fail") } - log.Debugf("xx_recvHandshakeMessage initiator=%v msgbuf=%v initial_stage=%v", s.initiator, msgbuf, initial_stage) - return buf, plaintext, valid, nil } @@ -82,8 +77,6 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payload []byte, initialMsg []byte) (err error) { kp := xx.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) - log.Debugf("runHandshake_xx initiator=%v fallback=%v pubkey=%x", s.initiator, fallback, kp.PubKey()) - // new XX noise session s.xx_ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) @@ -97,14 +90,10 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } } else { e_ik := s.ik_ns.Ephemeral() - log.Debugf("runHandshake_xx stage=0 initiator=true fallback=true ephemeralkeys=%x", e_ik) e_xx := xx.NewKeypair(e_ik.PubKey(), e_ik.PrivKey()) // initialize state as if we sent the first message - var msgbuf xx.MessageBuffer - s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, nil, &e_xx) - - log.Debugf("runHandshake_xx stage=0 initiator=true fallback=true msgbuf=%v", msgbuf) + s.xx_ns, _ = xx.SendMessage(s.xx_ns, nil, &e_xx) } // stage 1 // @@ -125,10 +114,7 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl var msgbuf *xx.MessageBuffer msgbuf, err = xx.Decode1(initialMsg) - log.Debugf("xx_recvHandshakeMessage stage=1 initiator=%v msgbuf=%v", s.initiator, msgbuf) - if err != nil { - log.Debugf("xx_recvHandshakeMessage stage=1 initiator=%v decode_err=%s", s.initiator, err) return fmt.Errorf("runHandshake_xx decode msg fail: %s", err) } @@ -204,8 +190,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } xx_msgbuf := xx.NewMessageBuffer(msgbuf.NE(), nil, nil) - log.Debugf("runHandshake_xx initiator=false msgbuf=%v modified_msgbuf=%v", msgbuf, xx_msgbuf) - s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, &xx_msgbuf) if !valid { return fmt.Errorf("runHandshake_xx validation fail") @@ -231,8 +215,6 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=validation fail") } - log.Debugf("runHandshake_xx stage=2 initiator=false remote key=%x", s.xx_ns.RemoteKey()) - // unmarshal payload err = proto.Unmarshal(plaintext, nhp) if err != nil { @@ -264,6 +246,5 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } } - log.Debugf("runHandshake_xx done initiator=%v", s.initiator) return nil } From 9f6c7d6cdea93a86b9be62271ebb4afba4f050d9 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 3 Feb 2020 10:24:50 -0500 Subject: [PATCH 1635/3965] fix peer ID check in IK handshake --- p2p/security/noise/ik_handshake.go | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go index 4aaaab274e..2ee67979fe 100644 --- a/p2p/security/noise/ik_handshake.go +++ b/p2p/security/noise/ik_handshake.go @@ -3,6 +3,7 @@ package noise import ( "context" "fmt" + "github.com/libp2p/go-libp2p-core/peer" "io" proto "github.com/gogo/protobuf/proto" @@ -117,10 +118,12 @@ func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([] return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=read remote libp2p key fail") } - // assert that remote peer ID matches libp2p key - err = s.setRemotePeerID(s.RemotePublicKey()) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=%s", err) + // assert that remote peer ID matches libp2p public key + pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) + if pid != s.remotePeer { + return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true check remote peer id err: expected %x got %x", s.remotePeer, pid) + } else if err != nil { + return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true check remote peer id err %s", err) } // verify payload is signed by libp2p key From 7aa706618cc0ef5a9a007d844697a2ae00fc0243 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 3 Feb 2020 10:49:15 -0500 Subject: [PATCH 1636/3965] add test for peer id mismatch --- p2p/security/noise/transport_test.go | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 0a2cbaf941..a8194fcaff 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -144,6 +144,26 @@ func TestKeys(t *testing.T) { } } +func TestPeerIDMismatchFailsHandshake(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + init, resp := newConnPair(t) + + var initErr error + done := make(chan struct{}) + go func() { + defer close(done) + _, initErr = initTransport.SecureOutbound(context.TODO(), init, "a-random-peer-id") + }() + + _, _ = respTransport.SecureInbound(context.TODO(), resp) + <-done + + if initErr == nil { + t.Fatal("expected initiator to fail with peer ID mismatch error") + } +} + func makeLargePlaintext(size int) []byte { buf := make([]byte, size) rand.Read(buf) From df4b7d636eb9b24042bb71b9143ec00de7d12b9a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 4 Feb 2020 10:08:27 -0800 Subject: [PATCH 1637/3965] Revert "add mutex for write/close" Actually fixed in https://github.com/multiformats/go-multistream/pull/50 This reverts commit 3d337a9a107c347913444b6acdcd587b79445b2a. --- p2p/transport/websocket/conn_native.go | 7 ------- 1 file changed, 7 deletions(-) diff --git a/p2p/transport/websocket/conn_native.go b/p2p/transport/websocket/conn_native.go index bd84b7dce9..6be4697ec6 100644 --- a/p2p/transport/websocket/conn_native.go +++ b/p2p/transport/websocket/conn_native.go @@ -17,7 +17,6 @@ type Conn struct { DefaultMessageType int reader io.Reader closeOnce sync.Once - mx sync.Mutex } func (c *Conn) Read(b []byte) (int, error) { @@ -68,9 +67,6 @@ func (c *Conn) prepNextReader() error { } func (c *Conn) Write(b []byte) (n int, err error) { - c.mx.Lock() - defer c.mx.Unlock() - if err := c.Conn.WriteMessage(c.DefaultMessageType, b); err != nil { return 0, err } @@ -82,9 +78,6 @@ func (c *Conn) Write(b []byte) (n int, err error) { // close error, subsequent and concurrent calls will return nil. // This method is thread-safe. func (c *Conn) Close() error { - c.mx.Lock() - defer c.mx.Unlock() - var err error c.closeOnce.Do(func() { err1 := c.Conn.WriteControl( From 373245d54a9d61d5503890aaa68ba3be490c2bf7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Magiera?= Date: Tue, 25 Jun 2019 23:43:16 +0200 Subject: [PATCH 1638/3965] Emit events when NAT status changes (#25) --- p2p/host/autonat/autonat.go | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 7e839dddd2..351efe3b22 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -7,6 +7,7 @@ import ( "sync" "time" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" @@ -61,6 +62,10 @@ type AmbientAutoNAT struct { // If only a single autoNAT peer is known, then the confidence increases // for each failure until it reaches 3. confidence int + + emitUnknown event.Emitter + emitPublic event.Emitter + emitPrivate event.Emitter } // NewAutoNAT creates a new ambient NAT autodiscovery instance attached to a host @@ -70,12 +75,20 @@ func NewAutoNAT(ctx context.Context, h host.Host, getAddrs GetAddrs) AutoNAT { getAddrs = h.Addrs } + emitUnknown, _ := h.EventBus().Emitter(new(event.EvtLocalRoutabilityUnknown)) + emitPublic, _ := h.EventBus().Emitter(new(event.EvtLocalRoutabilityPublic)) + emitPrivate, _ := h.EventBus().Emitter(new(event.EvtLocalRoutabilityPrivate)) + as := &AmbientAutoNAT{ ctx: ctx, host: h, getAddrs: getAddrs, peers: make(map[peer.ID][]ma.Multiaddr), status: NATStatusUnknown, + + emitUnknown: emitUnknown, + emitPublic: emitPublic, + emitPrivate: emitPrivate, } h.Network().Notify(as) @@ -90,6 +103,18 @@ func (as *AmbientAutoNAT) Status() NATStatus { return as.status } +func (as *AmbientAutoNAT) updateStatus(s NATStatus) { + as.status = s + switch s { + case NATStatusUnknown: + as.emitUnknown.Emit(event.EvtLocalRoutabilityUnknown{}) + case NATStatusPublic: + as.emitPublic.Emit(event.EvtLocalRoutabilityPublic{}) + case NATStatusPrivate: + as.emitPrivate.Emit(event.EvtLocalRoutabilityPrivate{}) + } +} + func (as *AmbientAutoNAT) PublicAddr() (ma.Multiaddr, error) { as.mx.Lock() defer as.mx.Unlock() @@ -194,7 +219,7 @@ func (as *AmbientAutoNAT) autodetect() { } else if as.confidence < 3 { as.confidence++ } - as.status = NATStatusPublic + as.updateStatus(NATStatusPublic) as.addr = result.pubaddr } else if result.private > 0 { log.Debugf("NAT status is private") @@ -204,14 +229,14 @@ func (as *AmbientAutoNAT) autodetect() { } else if as.confidence < 3 { as.confidence++ } - as.status = NATStatusPrivate + as.updateStatus(NATStatusPrivate) as.addr = nil } else if as.confidence > 0 { // don't just flip to unknown, reduce confidence first as.confidence-- } else { log.Debugf("NAT status is unknown") - as.status = NATStatusUnknown + as.updateStatus(NATStatusUnknown) as.addr = nil } as.mx.Unlock() From 67b82c6b3cca871c339883359696022003085962 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Wed, 5 Feb 2020 11:19:58 +0000 Subject: [PATCH 1639/3965] build(deps): bump github.com/multiformats/go-multistream Bumps [github.com/multiformats/go-multistream](https://github.com/multiformats/go-multistream) from 0.1.0 to 0.1.1. - [Release notes](https://github.com/multiformats/go-multistream/releases) - [Commits](https://github.com/multiformats/go-multistream/compare/v0.1.0...v0.1.1) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 1c12af8207..31c902b152 100644 --- a/go.mod +++ b/go.mod @@ -33,7 +33,7 @@ require ( github.com/multiformats/go-multiaddr v0.2.0 github.com/multiformats/go-multiaddr-dns v0.2.0 github.com/multiformats/go-multiaddr-net v0.1.1 - github.com/multiformats/go-multistream v0.1.0 + github.com/multiformats/go-multistream v0.1.1 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 // indirect ) diff --git a/go.sum b/go.sum index e1b2cb45de..72479f569f 100644 --- a/go.sum +++ b/go.sum @@ -296,6 +296,8 @@ github.com/multiformats/go-multihash v0.0.10 h1:lMoNbh2Ssd9PUF74Nz008KGzGPlfeV6w github.com/multiformats/go-multihash v0.0.10/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ= github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= +github.com/multiformats/go-multistream v0.1.1 h1:JlAdpIFhBhGRLxe9W6Om0w++Gd6KMWoFPZL/dEnm9nI= +github.com/multiformats/go-multistream v0.1.1/go.mod h1:KmHZ40hzVxiaiwlj3MEbYgK9JFk2/9UktWZAF54Du38= github.com/multiformats/go-varint v0.0.1 h1:TR/0rdQtnNxuN2IhiB639xC3tWM4IUi7DkTBVTdGW/M= github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= From f72dbb610d98756b361f91d5a72d663b7942400d Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Wed, 5 Feb 2020 11:20:21 +0000 Subject: [PATCH 1640/3965] build(deps): bump github.com/ipfs/go-cid from 0.0.4 to 0.0.5 Bumps [github.com/ipfs/go-cid](https://github.com/ipfs/go-cid) from 0.0.4 to 0.0.5. - [Release notes](https://github.com/ipfs/go-cid/releases) - [Commits](https://github.com/ipfs/go-cid/compare/v0.0.4...v0.0.5) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 1c12af8207..394fc3c2a4 100644 --- a/go.mod +++ b/go.mod @@ -2,7 +2,7 @@ module github.com/libp2p/go-libp2p require ( github.com/gogo/protobuf v1.3.1 - github.com/ipfs/go-cid v0.0.4 + github.com/ipfs/go-cid v0.0.5 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 github.com/ipfs/go-log v0.0.1 diff --git a/go.sum b/go.sum index e1b2cb45de..22f27c04fb 100644 --- a/go.sum +++ b/go.sum @@ -82,6 +82,8 @@ github.com/ipfs/go-cid v0.0.3 h1:UIAh32wymBpStoe83YCzwVQQ5Oy/H0FdxvUS6DJDzms= github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-cid v0.0.4 h1:UlfXKrZx1DjZoBhQHmNHLC1fK1dUJDN20Y28A7s+gJ8= github.com/ipfs/go-cid v0.0.4/go.mod h1:4LLaPOQwmk5z9LBgQnpkivrx8BJjUyGwTXCd5Xfj6+M= +github.com/ipfs/go-cid v0.0.5 h1:o0Ix8e/ql7Zb5UVUJEUfjsWCIY8t48++9lR8qi6oiJU= +github.com/ipfs/go-cid v0.0.5/go.mod h1:plgt+Y5MnOey4vO4UlUazGqdbEXuFYitED67FexhXog= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= github.com/ipfs/go-datastore v0.1.0/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= github.com/ipfs/go-datastore v0.1.1/go.mod h1:w38XXW9kVFNp57Zj5knbKWM2T+KOZCGDRVNdgPHtbHw= @@ -294,10 +296,14 @@ github.com/multiformats/go-multihash v0.0.8 h1:wrYcW5yxSi3dU07n5jnuS5PrNwyHy0zRH github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= github.com/multiformats/go-multihash v0.0.10 h1:lMoNbh2Ssd9PUF74Nz008KGzGPlfeV6wH3rit5IIGCM= github.com/multiformats/go-multihash v0.0.10/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= +github.com/multiformats/go-multihash v0.0.13 h1:06x+mk/zj1FoMsgNejLpy6QTvJqlSt/BhLEy87zidlc= +github.com/multiformats/go-multihash v0.0.13/go.mod h1:VdAWLKTwram9oKAatUcLxBNUjdtcVwxObEQBtRfuyjc= github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ= github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= github.com/multiformats/go-varint v0.0.1 h1:TR/0rdQtnNxuN2IhiB639xC3tWM4IUi7DkTBVTdGW/M= github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= +github.com/multiformats/go-varint v0.0.5 h1:XVZwSo04Cs3j/jS0uAEPpT3JY6DzMcVLLoWOSnCxOjg= +github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w= From 9ee9d94ea28edc50be9752fb1a5a65c0513edec3 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 5 Feb 2020 10:05:37 -0800 Subject: [PATCH 1641/3965] chore(ci): fix a flaky test (#787) Timing tests never work well in CI. --- p2p/net/mock/mock_test.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index f78830f001..ce207a78ab 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -17,6 +17,7 @@ import ( "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/protocol" "github.com/libp2p/go-libp2p-core/test" + "github.com/libp2p/go-libp2p-testing/ci" tnet "github.com/libp2p/go-libp2p-testing/net" ) @@ -489,6 +490,10 @@ func TestAdding(t *testing.T) { } func TestRateLimiting(t *testing.T) { + if ci.IsRunning() { + t.Skip("buggy in CI") + } + rl := NewRateLimiter(10) if !within(rl.Limit(10), time.Duration(float32(time.Second)), time.Millisecond) { From 44def816bfe8036443b8a4dcb541a6c9319314c6 Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Thu, 6 Feb 2020 13:46:10 +0530 Subject: [PATCH 1642/3965] unit tests for event updation --- p2p/host/autonat/autonat.go | 6 ++--- p2p/host/autonat/autonat_test.go | 38 ++++++++++++++++++++++++++++++-- 2 files changed, 39 insertions(+), 5 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 351efe3b22..2d6c9786f9 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -219,8 +219,8 @@ func (as *AmbientAutoNAT) autodetect() { } else if as.confidence < 3 { as.confidence++ } - as.updateStatus(NATStatusPublic) as.addr = result.pubaddr + as.updateStatus(NATStatusPublic) } else if result.private > 0 { log.Debugf("NAT status is private") if as.status == NATStatusPublic { @@ -229,15 +229,15 @@ func (as *AmbientAutoNAT) autodetect() { } else if as.confidence < 3 { as.confidence++ } - as.updateStatus(NATStatusPrivate) as.addr = nil + as.updateStatus(NATStatusPrivate) } else if as.confidence > 0 { // don't just flip to unknown, reduce confidence first as.confidence-- } else { log.Debugf("NAT status is unknown") - as.updateStatus(NATStatusUnknown) as.addr = nil + as.updateStatus(NATStatusUnknown) } as.mx.Unlock() } diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index e5fc7fff37..74a429ca4f 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -6,10 +6,10 @@ import ( "time" pb "github.com/libp2p/go-libp2p-autonat/pb" - "github.com/libp2p/go-libp2p-core/peer" - + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" ggio "github.com/gogo/protobuf/io" bhost "github.com/libp2p/go-libp2p-blankhost" @@ -96,6 +96,12 @@ func TestAutoNATPrivate(t *testing.T) { hs := makeAutoNATServicePrivate(ctx, t) hc, an := makeAutoNAT(ctx, t, hs) + // subscribe to AutoNat events + s, err := hc.EventBus().Subscribe(&event.EvtLocalRoutabilityPrivate{}) + if err != nil { + t.Fatalf("failed to subscribe to event EvtLocalRoutabilityPrivate, err=%s", err) + } + status := an.Status() if status != NATStatusUnknown { t.Fatalf("unexpected NAT status: %d", status) @@ -108,6 +114,17 @@ func TestAutoNATPrivate(t *testing.T) { if status != NATStatusPrivate { t.Fatalf("unexpected NAT status: %d", status) } + + select { + case e := <-s.Out(): + _, ok := e.(event.EvtLocalRoutabilityPrivate) + if !ok { + t.Fatal("got wrong event type from the bus") + } + + case <-time.After(1 * time.Second): + t.Fatal("failed to get the EvtLocalRoutabilityPrivate event from the bus") + } } func TestAutoNATPublic(t *testing.T) { @@ -117,6 +134,12 @@ func TestAutoNATPublic(t *testing.T) { hs := makeAutoNATServicePublic(ctx, t) hc, an := makeAutoNAT(ctx, t, hs) + // subscribe to AutoNat events + s, err := hc.EventBus().Subscribe(&event.EvtLocalRoutabilityPublic{}) + if err != nil { + t.Fatalf("failed to subscribe to event EvtLocalRoutabilityPublic, err=%s", err) + } + status := an.Status() if status != NATStatusUnknown { t.Fatalf("unexpected NAT status: %d", status) @@ -129,4 +152,15 @@ func TestAutoNATPublic(t *testing.T) { if status != NATStatusPublic { t.Fatalf("unexpected NAT status: %d", status) } + + select { + case e := <-s.Out(): + _, ok := e.(event.EvtLocalRoutabilityPublic) + if !ok { + t.Fatal("got wrong event type from the bus") + } + + case <-time.After(1 * time.Second): + t.Fatal("failed to get the EvtLocalRoutabilityPublic event from the bus") + } } From a8e8e9cdf8356b10878f786cef1688d4c0c93561 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Thu, 6 Feb 2020 11:19:17 +0000 Subject: [PATCH 1643/3965] build(deps): bump github.com/multiformats/go-multiaddr-net Bumps [github.com/multiformats/go-multiaddr-net](https://github.com/multiformats/go-multiaddr-net) from 0.1.1 to 0.1.2. - [Release notes](https://github.com/multiformats/go-multiaddr-net/releases) - [Commits](https://github.com/multiformats/go-multiaddr-net/compare/v0.1.1...v0.1.2) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 52f992f80d..7cb953c71b 100644 --- a/go.mod +++ b/go.mod @@ -32,7 +32,7 @@ require ( github.com/miekg/dns v1.1.12 // indirect github.com/multiformats/go-multiaddr v0.2.0 github.com/multiformats/go-multiaddr-dns v0.2.0 - github.com/multiformats/go-multiaddr-net v0.1.1 + github.com/multiformats/go-multiaddr-net v0.1.2 github.com/multiformats/go-multistream v0.1.1 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 // indirect diff --git a/go.sum b/go.sum index d4607118ac..09d410c936 100644 --- a/go.sum +++ b/go.sum @@ -287,6 +287,8 @@ github.com/multiformats/go-multiaddr-net v0.1.0 h1:ZepO8Ezwovd+7b5XPPDhQhayk1yt0 github.com/multiformats/go-multiaddr-net v0.1.0/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= github.com/multiformats/go-multiaddr-net v0.1.1 h1:jFFKUuXTXv+3ARyHZi3XUqQO+YWMKgBdhEvuGRfnL6s= github.com/multiformats/go-multiaddr-net v0.1.1/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= +github.com/multiformats/go-multiaddr-net v0.1.2 h1:P7zcBH9FRETdPkDrylcXVjQLQ2t1JQtNItZULWNWgeg= +github.com/multiformats/go-multiaddr-net v0.1.2/go.mod h1:QsWt3XK/3hwvNxZJp92iMQKME1qHfpYmyIjFVsSOY6Y= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= From 9d0e54866cc47c051e0403c9d43de48ac8919140 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Mon, 10 Feb 2020 10:13:53 -0500 Subject: [PATCH 1644/3965] only cancel dials when an outbound connection succeeds. this may result in duplicate connections, but it's better to have two connections than dropping both of them and ending up with zero connections. --- p2p/net/swarm/swarm.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index f7c57717fa..f85ef5234b 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -230,7 +230,9 @@ func (s *Swarm) addConn(tc transport.CapableConn, dir network.Direction) (*Conn, // We have a connection now. Cancel all other in-progress dials. // This should be fast, no reason to wait till later. - s.dsync.CancelDial(p) + if dir == network.DirOutbound { + s.dsync.CancelDial(p) + } s.notifyAll(func(f network.Notifiee) { f.Connected(s, c) From d1feb90b17a2869e0e575a79acf0ab2de6397170 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 10 Feb 2020 14:53:24 -0500 Subject: [PATCH 1645/3965] Signed envelopes & routing records (#73) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * add SignedEnvelope type * use struct for SignedEnvelope instead of exposing protobuf directly * doc comments for envelopes * tests for SignedEnvelopes * add helpers to make routing records for Host * fix doc comment * go fmt * add method to peerstore to retrieve signed routing records * update to match spec changes * just use nanoseconds * use proto3 & rename fields to match spec changes * use proto3 for routing records * make envelope fields private & validate on unmarshal * use buffer pool for envelope signatures * tests for RoutingState * go fmt * rename Equals -> Equal, add some comments * use test helpers * get rid of unsigned RoutingState struct, only expose SignedRoutingState * rm batching SignedRoutingStates accessor in peerstore the datastore peerstore implementation doesn't support batched reads, so it's no more efficient to get a bunch of states at once than it is to call SignedRoutingState multiple times. * whitespace * expose struct fields & remove accessors * use camelCase in protos for consistency * use multiformats uvarint for length-prefixes * remove payloadType check when unmarhaling * rm stray ref to golang/protobuf * define CertifiedAddrBook to avoid breaking API change * add events for updated addresses and routing state * remove SignedRoutingStateFromHost helper moving this to go-libp2p * add routing state records, extend peerstore API * fix: rebuild protos with new gogofaster generator * filter private addrs from signed routing records * envelope: use byte slices from pool; adjust interface. * move envelope to record package. * move protobuf files; adjust imports everywhere. * rename RoutingStateRecord -> PeerRecord also removes embedded reference to Envelope from the record, as that was confusing. as a result, the CertifiedAddrBook now accepts/returns record.SignedEnvelope instead of a specialized type. * hoist Seq from PeerRecord to SignedEnvelope * test that PeerRecords can't be signed by wrong key * commit go.sum * add Seq field to envelope signature * fix proto_path in Makefile * fix import ordering * comments for PeerRecord proto message also removes the seq field from PeerMessage proto, since it was moved to the SignedEnvelope * use Record type for envelope payloads * rename SignedEnvelope -> Envelope, unmarshal payload in ConsumeEnvelope * return buffer to pool before early return * doc comments * rename CertifiedAddrBook methods, update comments * cache unmarshalled Record payload inside Envelope * doc comments * store reflect.Type when registering Record * Revert "return buffer to pool before early return" 8d8da386f26482e06dc21989a6b5ade69f0a46d9 misread this - unsigned will be nil if there's an error, so it was right the way it was * use a DefaultRecord for unregistered PayloadTypes instead of returning an error if we don't have a registered Record for a given PayloadType, we can have a catch-all DefaultRecord type that just preserves the original payload as a []byte * cleanup DefaultRecord code a bit - removes unused error return from blankRecordForPayloadType - just references instead of copying in DefaultRecord.UnmarshalRecord I figure this is likely safe, since we'll be unmarshalling from the payload of an Envelope, which shouldn't get altered after it's created. * use explicit payloadType in MakeEnvelopeWithRecord * Revert DefaultRecord commits ae3bc7bdfb657c232229229706854a56effca80b a26c845a766b45ceabd87c17c0801d191650f0d4 * doc comments * move Seq field back to PeerRecord * make diffs optional in EvtLocalAddressesUpdated * more envelope tests * replace MakeEnvelope with record.Seal also: - add Domain and Codec fields to Record interface * fix import * add interface check * rename ProcessPeerRecord -> ConsumePeerRecord also, adds bool `accepted` return value * rename event field, add doc comment * peer record protobuf: fix field casing. * record protobuf: add docs and fix casing. * cleanup: group imports. * nit: split test/utils.go => test/{addrs,errors}.go. Co-authored-by: Raúl Kripalani --- core/crypto/pb/Makefile | 2 +- core/crypto/pb/crypto.proto | 2 + core/event/addrs.go | 82 +++++ core/peer/pb/Makefile | 11 + core/peer/pb/peer_record.pb.go | 606 +++++++++++++++++++++++++++++++++ core/peer/pb/peer_record.proto | 29 ++ core/peer/record.go | 208 +++++++++++ core/peer/record_test.go | 54 +++ core/peerstore/peerstore.go | 80 ++++- core/record/envelope.go | 297 ++++++++++++++++ core/record/envelope_test.go | 313 +++++++++++++++++ core/record/pb/Makefile | 11 + core/record/pb/envelope.pb.go | 504 +++++++++++++++++++++++++++ core/record/pb/envelope.proto | 30 ++ core/record/record.go | 102 ++++++ core/record/record_test.go | 51 +++ core/test/addrs.go | 42 +++ core/test/errors.go | 19 ++ 18 files changed, 2439 insertions(+), 4 deletions(-) create mode 100644 core/event/addrs.go create mode 100644 core/peer/pb/Makefile create mode 100644 core/peer/pb/peer_record.pb.go create mode 100644 core/peer/pb/peer_record.proto create mode 100644 core/peer/record.go create mode 100644 core/peer/record_test.go create mode 100644 core/record/envelope.go create mode 100644 core/record/envelope_test.go create mode 100644 core/record/pb/Makefile create mode 100644 core/record/pb/envelope.pb.go create mode 100644 core/record/pb/envelope.proto create mode 100644 core/record/record.go create mode 100644 core/record/record_test.go create mode 100644 core/test/addrs.go create mode 100644 core/test/errors.go diff --git a/core/crypto/pb/Makefile b/core/crypto/pb/Makefile index df34e54b01..8af2dd8177 100644 --- a/core/crypto/pb/Makefile +++ b/core/crypto/pb/Makefile @@ -4,7 +4,7 @@ GO = $(PB:.proto=.pb.go) all: $(GO) %.pb.go: %.proto - protoc --proto_path=$(GOPATH)/src:. --gogofaster_out=. $< + protoc --proto_path=$(PWD)/../..:. --gogofaster_out=. $< clean: rm -f *.pb.go diff --git a/core/crypto/pb/crypto.proto b/core/crypto/pb/crypto.proto index cb5cee8a22..182b0d4847 100644 --- a/core/crypto/pb/crypto.proto +++ b/core/crypto/pb/crypto.proto @@ -2,6 +2,8 @@ syntax = "proto2"; package crypto.pb; +option go_package = "github.com/libp2p/go-libp2p-core/crypto/pb"; + enum KeyType { RSA = 0; Ed25519 = 1; diff --git a/core/event/addrs.go b/core/event/addrs.go new file mode 100644 index 0000000000..f260d149ba --- /dev/null +++ b/core/event/addrs.go @@ -0,0 +1,82 @@ +package event + +import ( + "github.com/libp2p/go-libp2p-core/record" + ma "github.com/multiformats/go-multiaddr" +) + +// AddrAction represents an action taken on one of a Host's listen addresses. +// It is used to add context to address change events in EvtLocalAddressesUpdated. +type AddrAction int + +const ( + // Unknown means that the event producer was unable to determine why the address + // is in the current state. + Unknown AddrAction = iota + + // Added means that the address is new and was not present prior to the event. + Added + + // Maintained means that the address was not altered between the current and + // previous states. + Maintained + + // Removed means that the address was removed from the Host. + Removed +) + +// UpdatedAddress is used in the EvtLocalAddressesUpdated event to convey +// address change information. +type UpdatedAddress struct { + // Address contains the address that was updated. + Address ma.Multiaddr + + // Action indicates what action was taken on the address during the + // event. May be Unknown if the event producer cannot produce diffs. + Action AddrAction +} + +// EvtLocalAddressesUpdated should be emitted when the set of listen addresses for +// the local host changes. This may happen for a number of reasons. For example, +// we may have opened a new relay connection, established a new NAT mapping via +// UPnP, or been informed of our observed address by another peer. +// +// EvtLocalAddressesUpdated contains a snapshot of the current listen addresses, +// and may also contain a diff between the current state and the previous state. +// If the event producer is capable of creating a diff, the Diffs field will be +// true, and event consumers can inspect the Action field of each UpdatedAddress +// to see how each address was modified. +// +// For example, the Action will tell you whether an address in +// the Current list was Added by the event producer, or was Maintained without +// changes. Addresses that were removed from the Host will have the AddrAction +// of Removed, and will be in the Removed list. +// +// If the event producer is not capable or producing diffs, the Diffs field will +// be false, the Removed list will always be empty, and the Action for each +// UpdatedAddress in the Current list will be Unknown. +type EvtLocalAddressesUpdated struct { + + // Diffs indicates whether this event contains a diff of the Host's previous + // address set. + Diffs bool + + // Current contains all current listen addresses for the Host. + // If Diffs == true, the Action field of each UpdatedAddress will tell + // you whether an address was Added, or was Maintained from the previous + // state. + Current []UpdatedAddress + + // Removed contains addresses that were removed from the Host. + // This field is only set when Diffs == true. + Removed []UpdatedAddress +} + +// EvtLocalPeerRoutingStateUpdated should be emitted when a new signed PeerRecord +// for the local peer has been produced. This will happen whenever the set of listen +// addresses changes. +type EvtLocalPeerRecordUpdated struct { + // Record contains the updated peer.PeerRecord, wrapped in a record.Envelope and + // signed by the Host's private key. + Record *record.Envelope +} diff --git a/core/peer/pb/Makefile b/core/peer/pb/Makefile new file mode 100644 index 0000000000..7cf8222f89 --- /dev/null +++ b/core/peer/pb/Makefile @@ -0,0 +1,11 @@ +PB = $(wildcard *.proto) +GO = $(PB:.proto=.pb.go) + +all: $(GO) + +%.pb.go: %.proto + protoc --proto_path=$(PWD):$(PWD)/../.. --gogofaster_out=. $< + +clean: + rm -f *.pb.go + rm -f *.go diff --git a/core/peer/pb/peer_record.pb.go b/core/peer/pb/peer_record.pb.go new file mode 100644 index 0000000000..dd0755ef72 --- /dev/null +++ b/core/peer/pb/peer_record.pb.go @@ -0,0 +1,606 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: peer_record.proto + +package peer_pb + +import ( + fmt "fmt" + proto "github.com/gogo/protobuf/proto" + io "io" + math "math" + math_bits "math/bits" +) + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package + +// PeerRecord messages contain information that is useful to share with other peers. +// Currently, a PeerRecord contains the public listen addresses for a peer, but this +// is expected to expand to include other information in the future. +// +// PeerRecords are designed to be serialized to bytes and placed inside of +// SignedEnvelopes before sharing with other peers. +// See https://github.com/libp2p/go-libp2p-core/record/pb/envelope.proto for +// the SignedEnvelope definition. +type PeerRecord struct { + // peer_id contains a libp2p peer id in its binary representation. + PeerId []byte `protobuf:"bytes,1,opt,name=peer_id,json=peerId,proto3" json:"peer_id,omitempty"` + // seq contains a monotonically-increasing sequence counter to order PeerRecords in time. + Seq uint64 `protobuf:"varint,2,opt,name=seq,proto3" json:"seq,omitempty"` + // addresses is a list of public listen addresses for the peer. + Addresses []*PeerRecord_AddressInfo `protobuf:"bytes,3,rep,name=addresses,proto3" json:"addresses,omitempty"` +} + +func (m *PeerRecord) Reset() { *m = PeerRecord{} } +func (m *PeerRecord) String() string { return proto.CompactTextString(m) } +func (*PeerRecord) ProtoMessage() {} +func (*PeerRecord) Descriptor() ([]byte, []int) { + return fileDescriptor_dc0d8059ab0ad14d, []int{0} +} +func (m *PeerRecord) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *PeerRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_PeerRecord.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *PeerRecord) XXX_Merge(src proto.Message) { + xxx_messageInfo_PeerRecord.Merge(m, src) +} +func (m *PeerRecord) XXX_Size() int { + return m.Size() +} +func (m *PeerRecord) XXX_DiscardUnknown() { + xxx_messageInfo_PeerRecord.DiscardUnknown(m) +} + +var xxx_messageInfo_PeerRecord proto.InternalMessageInfo + +func (m *PeerRecord) GetPeerId() []byte { + if m != nil { + return m.PeerId + } + return nil +} + +func (m *PeerRecord) GetSeq() uint64 { + if m != nil { + return m.Seq + } + return 0 +} + +func (m *PeerRecord) GetAddresses() []*PeerRecord_AddressInfo { + if m != nil { + return m.Addresses + } + return nil +} + +// AddressInfo is a wrapper around a binary multiaddr. It is defined as a +// separate message to allow us to add per-address metadata in the future. +type PeerRecord_AddressInfo struct { + Multiaddr []byte `protobuf:"bytes,1,opt,name=multiaddr,proto3" json:"multiaddr,omitempty"` +} + +func (m *PeerRecord_AddressInfo) Reset() { *m = PeerRecord_AddressInfo{} } +func (m *PeerRecord_AddressInfo) String() string { return proto.CompactTextString(m) } +func (*PeerRecord_AddressInfo) ProtoMessage() {} +func (*PeerRecord_AddressInfo) Descriptor() ([]byte, []int) { + return fileDescriptor_dc0d8059ab0ad14d, []int{0, 0} +} +func (m *PeerRecord_AddressInfo) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *PeerRecord_AddressInfo) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_PeerRecord_AddressInfo.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *PeerRecord_AddressInfo) XXX_Merge(src proto.Message) { + xxx_messageInfo_PeerRecord_AddressInfo.Merge(m, src) +} +func (m *PeerRecord_AddressInfo) XXX_Size() int { + return m.Size() +} +func (m *PeerRecord_AddressInfo) XXX_DiscardUnknown() { + xxx_messageInfo_PeerRecord_AddressInfo.DiscardUnknown(m) +} + +var xxx_messageInfo_PeerRecord_AddressInfo proto.InternalMessageInfo + +func (m *PeerRecord_AddressInfo) GetMultiaddr() []byte { + if m != nil { + return m.Multiaddr + } + return nil +} + +func init() { + proto.RegisterType((*PeerRecord)(nil), "peer.pb.PeerRecord") + proto.RegisterType((*PeerRecord_AddressInfo)(nil), "peer.pb.PeerRecord.AddressInfo") +} + +func init() { proto.RegisterFile("peer_record.proto", fileDescriptor_dc0d8059ab0ad14d) } + +var fileDescriptor_dc0d8059ab0ad14d = []byte{ + // 189 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0x12, 0x2c, 0x48, 0x4d, 0x2d, + 0x8a, 0x2f, 0x4a, 0x4d, 0xce, 0x2f, 0x4a, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0x07, + 0x09, 0xe9, 0x15, 0x24, 0x29, 0x2d, 0x66, 0xe4, 0xe2, 0x0a, 0x48, 0x4d, 0x2d, 0x0a, 0x02, 0xcb, + 0x0a, 0x89, 0x73, 0x81, 0x65, 0xe2, 0x33, 0x53, 0x24, 0x18, 0x15, 0x18, 0x35, 0x78, 0x82, 0xd8, + 0x40, 0x5c, 0xcf, 0x14, 0x21, 0x01, 0x2e, 0xe6, 0xe2, 0xd4, 0x42, 0x09, 0x26, 0x05, 0x46, 0x0d, + 0x96, 0x20, 0x10, 0x53, 0xc8, 0x96, 0x8b, 0x33, 0x31, 0x25, 0xa5, 0x28, 0xb5, 0xb8, 0x38, 0xb5, + 0x58, 0x82, 0x59, 0x81, 0x59, 0x83, 0xdb, 0x48, 0x5e, 0x0f, 0x6a, 0xac, 0x1e, 0xc2, 0x48, 0x3d, + 0x47, 0x88, 0x22, 0xcf, 0xbc, 0xb4, 0xfc, 0x20, 0x84, 0x0e, 0x29, 0x6d, 0x2e, 0x6e, 0x24, 0x19, + 0x21, 0x19, 0x2e, 0xce, 0xdc, 0xd2, 0x9c, 0x92, 0x4c, 0x90, 0x02, 0xa8, 0xd5, 0x08, 0x01, 0x27, + 0x89, 0x13, 0x8f, 0xe4, 0x18, 0x2f, 0x3c, 0x92, 0x63, 0x7c, 0xf0, 0x48, 0x8e, 0x71, 0xc2, 0x63, + 0x39, 0x86, 0x0b, 0x8f, 0xe5, 0x18, 0x6e, 0x3c, 0x96, 0x63, 0x48, 0x62, 0x03, 0xfb, 0xc7, 0x18, + 0x10, 0x00, 0x00, 0xff, 0xff, 0xcb, 0x99, 0x56, 0x19, 0xe4, 0x00, 0x00, 0x00, +} + +func (m *PeerRecord) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *PeerRecord) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *PeerRecord) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if len(m.Addresses) > 0 { + for iNdEx := len(m.Addresses) - 1; iNdEx >= 0; iNdEx-- { + { + size, err := m.Addresses[iNdEx].MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintPeerRecord(dAtA, i, uint64(size)) + } + i-- + dAtA[i] = 0x1a + } + } + if m.Seq != 0 { + i = encodeVarintPeerRecord(dAtA, i, uint64(m.Seq)) + i-- + dAtA[i] = 0x10 + } + if len(m.PeerId) > 0 { + i -= len(m.PeerId) + copy(dAtA[i:], m.PeerId) + i = encodeVarintPeerRecord(dAtA, i, uint64(len(m.PeerId))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil +} + +func (m *PeerRecord_AddressInfo) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *PeerRecord_AddressInfo) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *PeerRecord_AddressInfo) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if len(m.Multiaddr) > 0 { + i -= len(m.Multiaddr) + copy(dAtA[i:], m.Multiaddr) + i = encodeVarintPeerRecord(dAtA, i, uint64(len(m.Multiaddr))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil +} + +func encodeVarintPeerRecord(dAtA []byte, offset int, v uint64) int { + offset -= sovPeerRecord(v) + base := offset + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return base +} +func (m *PeerRecord) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + l = len(m.PeerId) + if l > 0 { + n += 1 + l + sovPeerRecord(uint64(l)) + } + if m.Seq != 0 { + n += 1 + sovPeerRecord(uint64(m.Seq)) + } + if len(m.Addresses) > 0 { + for _, e := range m.Addresses { + l = e.Size() + n += 1 + l + sovPeerRecord(uint64(l)) + } + } + return n +} + +func (m *PeerRecord_AddressInfo) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + l = len(m.Multiaddr) + if l > 0 { + n += 1 + l + sovPeerRecord(uint64(l)) + } + return n +} + +func sovPeerRecord(x uint64) (n int) { + return (math_bits.Len64(x|1) + 6) / 7 +} +func sozPeerRecord(x uint64) (n int) { + return sovPeerRecord(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *PeerRecord) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: PeerRecord: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: PeerRecord: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field PeerId", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPeerRecord + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPeerRecord + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.PeerId = append(m.PeerId[:0], dAtA[iNdEx:postIndex]...) + if m.PeerId == nil { + m.PeerId = []byte{} + } + iNdEx = postIndex + case 2: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Seq", wireType) + } + m.Seq = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.Seq |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Addresses", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthPeerRecord + } + postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthPeerRecord + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Addresses = append(m.Addresses, &PeerRecord_AddressInfo{}) + if err := m.Addresses[len(m.Addresses)-1].Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipPeerRecord(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPeerRecord + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthPeerRecord + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *PeerRecord_AddressInfo) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: AddressInfo: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: AddressInfo: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Multiaddr", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPeerRecord + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPeerRecord + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Multiaddr = append(m.Multiaddr[:0], dAtA[iNdEx:postIndex]...) + if m.Multiaddr == nil { + m.Multiaddr = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipPeerRecord(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPeerRecord + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthPeerRecord + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipPeerRecord(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + depth := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + case 1: + iNdEx += 8 + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowPeerRecord + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if length < 0 { + return 0, ErrInvalidLengthPeerRecord + } + iNdEx += length + case 3: + depth++ + case 4: + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupPeerRecord + } + depth-- + case 5: + iNdEx += 4 + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + if iNdEx < 0 { + return 0, ErrInvalidLengthPeerRecord + } + if depth == 0 { + return iNdEx, nil + } + } + return 0, io.ErrUnexpectedEOF +} + +var ( + ErrInvalidLengthPeerRecord = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowPeerRecord = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupPeerRecord = fmt.Errorf("proto: unexpected end of group") +) diff --git a/core/peer/pb/peer_record.proto b/core/peer/pb/peer_record.proto new file mode 100644 index 0000000000..fb2835d8e6 --- /dev/null +++ b/core/peer/pb/peer_record.proto @@ -0,0 +1,29 @@ +syntax = "proto3"; + +package peer.pb; + +// PeerRecord messages contain information that is useful to share with other peers. +// Currently, a PeerRecord contains the public listen addresses for a peer, but this +// is expected to expand to include other information in the future. +// +// PeerRecords are designed to be serialized to bytes and placed inside of +// SignedEnvelopes before sharing with other peers. +// See https://github.com/libp2p/go-libp2p-core/record/pb/envelope.proto for +// the SignedEnvelope definition. +message PeerRecord { + + // AddressInfo is a wrapper around a binary multiaddr. It is defined as a + // separate message to allow us to add per-address metadata in the future. + message AddressInfo { + bytes multiaddr = 1; + } + + // peer_id contains a libp2p peer id in its binary representation. + bytes peer_id = 1; + + // seq contains a monotonically-increasing sequence counter to order PeerRecords in time. + uint64 seq = 2; + + // addresses is a list of public listen addresses for the peer. + repeated AddressInfo addresses = 3; +} diff --git a/core/peer/record.go b/core/peer/record.go new file mode 100644 index 0000000000..76c4e672d6 --- /dev/null +++ b/core/peer/record.go @@ -0,0 +1,208 @@ +package peer + +import ( + "fmt" + "time" + + pb "github.com/libp2p/go-libp2p-core/peer/pb" + "github.com/libp2p/go-libp2p-core/record" + + ma "github.com/multiformats/go-multiaddr" + + "github.com/gogo/protobuf/proto" +) + +var _ record.Record = (*PeerRecord)(nil) + +func init() { + record.RegisterType(&PeerRecord{}) +} + +// The domain string used for peer records contained in a Envelope. +const PeerRecordEnvelopeDomain = "libp2p-peer-record" + +// The type hint used to identify peer records in a Envelope. +// Defined in https://github.com/multiformats/multicodec/blob/master/table.csv +// with name "libp2p-peer-record" +var PeerRecordEnvelopePayloadType = []byte{0x03, 0x01} + +// PeerRecord contains information that is broadly useful to share with other peers, +// either through a direct exchange (as in the libp2p identify protocol), or through +// a Peer Routing provider, such as a DHT. +// +// Currently, a PeerRecord contains the public listen addresses for a peer, but this +// is expected to expand to include other information in the future. +// +// PeerRecords are ordered in time by their Seq field. Newer PeerRecords must have +// greater Seq values than older records. The NewPeerRecord function will create +// a PeerRecord with a timestamp-based Seq value. The other PeerRecord fields should +// be set by the caller: +// +// rec := peer.NewPeerRecord() +// rec.PeerID = aPeerID +// rec.Addrs = someAddrs +// +// Alternatively, you can construct a PeerRecord struct directly and use the TimestampSeq +// helper to set the Seq field: +// +// rec := peer.PeerRecord{PeerID: aPeerID, Addrs: someAddrs, Seq: peer.TimestampSeq()} +// +// Failing to set the Seq field will not result in an error, however, a PeerRecord with a +// Seq value of zero may be ignored or rejected by other peers. +// +// PeerRecords are intended to be shared with other peers inside a signed +// routing.Envelope, and PeerRecord implements the routing.Record interface +// to facilitate this. +// +// To share a PeerRecord, first call Sign to wrap the record in a Envelope +// and sign it with the local peer's private key: +// +// rec := &PeerRecord{PeerID: myPeerId, Addrs: myAddrs} +// envelope, err := rec.Sign(myPrivateKey) +// +// The resulting record.Envelope can be marshalled to a []byte and shared +// publicly. As a convenience, the MarshalSigned method will produce the +// Envelope and marshal it to a []byte in one go: +// +// rec := &PeerRecord{PeerID: myPeerId, Addrs: myAddrs} +// recordBytes, err := rec.MarshalSigned(myPrivateKey) +// +// To validate and unmarshal a signed PeerRecord from a remote peer, +// "consume" the containing envelope, which will return both the +// routing.Envelope and the inner Record. The Record must be cast to +// a PeerRecord pointer before use: +// +// envelope, untypedRecord, err := ConsumeEnvelope(envelopeBytes, PeerRecordEnvelopeDomain) +// if err != nil { +// handleError(err) +// return +// } +// peerRec := untypedRecord.(*PeerRecord) +// +type PeerRecord struct { + // PeerID is the ID of the peer this record pertains to. + PeerID ID + + // Addrs contains the public addresses of the peer this record pertains to. + Addrs []ma.Multiaddr + + // Seq is a monotonically-increasing sequence counter that's used to order + // PeerRecords in time. The interval between Seq values is unspecified, + // but newer PeerRecords MUST have a greater Seq value than older records + // for the same peer. + Seq uint64 +} + +// NewPeerRecord returns a PeerRecord with a timestamp-based sequence number. +// The returned record is otherwise empty and should be populated by the caller. +func NewPeerRecord() *PeerRecord { + return &PeerRecord{Seq: TimestampSeq()} +} + +// PeerRecordFromAddrInfo creates a PeerRecord from an AddrInfo struct. +// The returned record will have a timestamp-based sequence number. +func PeerRecordFromAddrInfo(info AddrInfo) *PeerRecord { + rec := NewPeerRecord() + rec.PeerID = info.ID + rec.Addrs = info.Addrs + return rec +} + +// TimestampSeq is a helper to generate a timestamp-based sequence number for a PeerRecord. +func TimestampSeq() uint64 { + return uint64(time.Now().UnixNano()) +} + +// Domain is used when signing and validating PeerRecords contained in Envelopes. +// It is constant for all PeerRecord instances. +func (r *PeerRecord) Domain() string { + return PeerRecordEnvelopeDomain +} + +// Codec is a binary identifier for the PeerRecord type. It is constant for all PeerRecord instances. +func (r *PeerRecord) Codec() []byte { + return PeerRecordEnvelopePayloadType +} + +// UnmarshalRecord parses a PeerRecord from a byte slice. +// This method is called automatically when consuming a record.Envelope +// whose PayloadType indicates that it contains a PeerRecord. +// It is generally not necessary or recommended to call this method directly. +func (r *PeerRecord) UnmarshalRecord(bytes []byte) error { + if r == nil { + return fmt.Errorf("cannot unmarshal PeerRecord to nil receiver") + } + + var msg pb.PeerRecord + err := proto.Unmarshal(bytes, &msg) + if err != nil { + return err + } + var id ID + err = id.UnmarshalBinary(msg.PeerId) + if err != nil { + return err + } + r.PeerID = id + r.Addrs = addrsFromProtobuf(msg.Addresses) + r.Seq = msg.Seq + return nil +} + +// MarshalRecord serializes a PeerRecord to a byte slice. +// This method is called automatically when constructing a routing.Envelope +// using Seal or PeerRecord.Sign. +func (r *PeerRecord) MarshalRecord() ([]byte, error) { + idBytes, err := r.PeerID.MarshalBinary() + if err != nil { + return nil, err + } + msg := pb.PeerRecord{ + PeerId: idBytes, + Addresses: addrsToProtobuf(r.Addrs), + Seq: r.Seq, + } + return proto.Marshal(&msg) +} + +// Equal returns true if the other PeerRecord is identical to this one. +func (r *PeerRecord) Equal(other *PeerRecord) bool { + if other == nil { + return r == nil + } + if r.PeerID != other.PeerID { + return false + } + if r.Seq != other.Seq { + return false + } + if len(r.Addrs) != len(other.Addrs) { + return false + } + for i, _ := range r.Addrs { + if !r.Addrs[i].Equal(other.Addrs[i]) { + return false + } + } + return true +} + +func addrsFromProtobuf(addrs []*pb.PeerRecord_AddressInfo) []ma.Multiaddr { + var out []ma.Multiaddr + for _, addr := range addrs { + a, err := ma.NewMultiaddrBytes(addr.Multiaddr) + if err != nil { + continue + } + out = append(out, a) + } + return out +} + +func addrsToProtobuf(addrs []ma.Multiaddr) []*pb.PeerRecord_AddressInfo { + var out []*pb.PeerRecord_AddressInfo + for _, addr := range addrs { + out = append(out, &pb.PeerRecord_AddressInfo{Multiaddr: addr.Bytes()}) + } + return out +} diff --git a/core/peer/record_test.go b/core/peer/record_test.go new file mode 100644 index 0000000000..adbe03ae17 --- /dev/null +++ b/core/peer/record_test.go @@ -0,0 +1,54 @@ +package peer_test + +import ( + "bytes" + "testing" + + "github.com/libp2p/go-libp2p-core/crypto" + . "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/record" + "github.com/libp2p/go-libp2p-core/test" +) + +func TestPeerRecordConstants(t *testing.T) { + msgf := "Changing the %s may cause peer records to be incompatible with older versions. " + + "If you've already thought that through, please update this test so that it passes with the new values." + rec := PeerRecord{} + if rec.Domain() != "libp2p-peer-record" { + t.Errorf(msgf, "signing domain") + } + if !bytes.Equal(rec.Codec(), []byte{0x03, 0x01}) { + t.Errorf(msgf, "codec value") + } +} + +func TestSignedPeerRecordFromEnvelope(t *testing.T) { + priv, _, err := test.RandTestKeyPair(crypto.Ed25519, 256) + test.AssertNilError(t, err) + + addrs := test.GenerateTestAddrs(10) + id, err := IDFromPrivateKey(priv) + test.AssertNilError(t, err) + + rec := &PeerRecord{PeerID: id, Addrs: addrs, Seq: TimestampSeq()} + envelope, err := record.Seal(rec, priv) + test.AssertNilError(t, err) + + t.Run("is unaltered after round-trip serde", func(t *testing.T) { + envBytes, err := envelope.Marshal() + test.AssertNilError(t, err) + + env2, untypedRecord, err := record.ConsumeEnvelope(envBytes, PeerRecordEnvelopeDomain) + test.AssertNilError(t, err) + rec2, ok := untypedRecord.(*PeerRecord) + if !ok { + t.Error("unmarshaled record is not a *PeerRecord") + } + if !rec.Equal(rec2) { + t.Error("expected peer record to be unaltered after round-trip serde") + } + if !envelope.Equal(env2) { + t.Error("expected signed envelope to be unchanged after round-trip serde") + } + }) +} diff --git a/core/peerstore/peerstore.go b/core/peerstore/peerstore.go index 7f2c01c9dd..e77b3c1326 100644 --- a/core/peerstore/peerstore.go +++ b/core/peerstore/peerstore.go @@ -9,9 +9,9 @@ import ( "math" "time" - "github.com/libp2p/go-libp2p-core/peer" - ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/record" ma "github.com/multiformats/go-multiaddr" ) @@ -107,7 +107,7 @@ type AddrBook interface { // the given oldTTL to have the given newTTL. UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) - // Addresses returns all known (and valid) addresses for a given peer + // Addrs returns all known (and valid) addresses for a given peer. Addrs(p peer.ID) []ma.Multiaddr // AddrStream returns a channel that gets all addresses for a given @@ -122,6 +122,80 @@ type AddrBook interface { PeersWithAddrs() peer.IDSlice } +// CertifiedAddrBook manages "self-certified" addresses for remote peers. +// Self-certified addresses are contained in peer.PeerRecords +// which are wrapped in a record.Envelope and signed by the peer +// to whom they belong. +// +// Certified addresses (CA) are generally more secure than uncertified +// addresses (UA). Consequently, CAs beat and displace UAs. When the +// peerstore learns CAs for a peer, it will reject UAs for the same peer +// (as long as the former haven't expired). +// Furthermore, peer records act like sequenced snapshots of CAs. Therefore, +// processing a peer record that's newer than the last one seen overwrites +// all addresses with the incoming ones. +// +// This interface is most useful when combined with AddrBook. +// To test whether a given AddrBook / Peerstore implementation supports +// certified addresses, callers should use the GetCertifiedAddrBook helper or +// type-assert on the CertifiedAddrBook interface: +// +// if cab, ok := aPeerstore.(CertifiedAddrBook); ok { +// cab.ConsumePeerRecord(signedPeerRecord, aTTL) +// } +// +type CertifiedAddrBook interface { + // ConsumePeerRecord adds addresses from a signed peer.PeerRecord (contained in + // a record.Envelope), which will expire after the given TTL. + // + // The 'accepted' return value indicates that the record was successfully processed + // and integrated into the CertifiedAddrBook state. If 'accepted' is false but no + // error is returned, it means that the record was ignored, most likely because + // a newer record exists for the same peer. + // + // Signed records added via this method will be stored without + // alteration as long as the address TTLs remain valid. The Envelopes + // containing the PeerRecords can be retrieved by calling GetPeerRecord(peerID). + // + // If the signed PeerRecord belongs to a peer that already has certified + // addresses in the CertifiedAddrBook, the new addresses will replace the + // older ones, iff the new record has a higher sequence number than the + // existing record. Attempting to add a peer record with a + // sequence number that's <= an existing record for the same peer will not + // result in an error, but the record will be ignored, and the 'accepted' + // bool return value will be false. + // + // If the CertifiedAddrBook is also an AddrBook (which is most likely the case), + // adding certified addresses for a peer will *replace* any + // existing non-certified addresses for that peer, and only the certified + // addresses will be returned from AddrBook.Addrs thereafter. + // + // Likewise, once certified addresses have been added for a given peer, + // any non-certified addresses added via AddrBook.AddAddrs or + // AddrBook.SetAddrs will be ignored. AddrBook.SetAddrs may still be used + // to update the TTL of certified addresses that have previously been + // added via ConsumePeerRecord. + ConsumePeerRecord(s *record.Envelope, ttl time.Duration) (accepted bool, err error) + + // GetPeerRecord returns a Envelope containing a PeerRecord for the + // given peer id, if one exists. + // Returns nil if no signed PeerRecord exists for the peer. + GetPeerRecord(p peer.ID) *record.Envelope +} + +// GetCertifiedAddrBook is a helper to "upcast" an AddrBook to a +// CertifiedAddrBook by using type assertion. If the given AddrBook +// is also a CertifiedAddrBook, it will be returned, and the ok return +// value will be true. Returns (nil, false) if the AddrBook is not a +// CertifiedAddrBook. +// +// Note that since Peerstore embeds the AddrBook interface, you can also +// call GetCertifiedAddrBook(myPeerstore). +func GetCertifiedAddrBook(ab AddrBook) (cab CertifiedAddrBook, ok bool) { + cab, ok = ab.(CertifiedAddrBook) + return cab, ok +} + // KeyBook tracks the keys of Peers. type KeyBook interface { // PubKey stores the public key of a peer. diff --git a/core/record/envelope.go b/core/record/envelope.go new file mode 100644 index 0000000000..bdc33abdf1 --- /dev/null +++ b/core/record/envelope.go @@ -0,0 +1,297 @@ +package record + +import ( + "bytes" + "errors" + "fmt" + "sync" + + "github.com/libp2p/go-libp2p-core/crypto" + pb "github.com/libp2p/go-libp2p-core/record/pb" + + pool "github.com/libp2p/go-buffer-pool" + + "github.com/gogo/protobuf/proto" + "github.com/multiformats/go-varint" +) + +// Envelope contains an arbitrary []byte payload, signed by a libp2p peer. +// +// Envelopes are signed in the context of a particular "domain", which is a +// string specified when creating and verifying the envelope. You must know the +// domain string used to produce the envelope in order to verify the signature +// and access the payload. +type Envelope struct { + // The public key that can be used to verify the signature and derive the peer id of the signer. + PublicKey crypto.PubKey + + // A binary identifier that indicates what kind of data is contained in the payload. + // TODO(yusef): enforce multicodec prefix + PayloadType []byte + + // The envelope payload. + RawPayload []byte + + // The signature of the domain string :: type hint :: payload. + signature []byte + + // the unmarshalled payload as a Record, cached on first access via the Record accessor method + cached Record + unmarshalError error + unmarshalOnce sync.Once +} + +var ErrEmptyDomain = errors.New("envelope domain must not be empty") +var ErrEmptyPayloadType = errors.New("payloadType must not be empty") +var ErrInvalidSignature = errors.New("invalid signature or incorrect domain") + +// Seal marshals the given Record, places the marshaled bytes inside an Envelope, +// and signs with the given private key. +func Seal(rec Record, privateKey crypto.PrivKey) (*Envelope, error) { + payload, err := rec.MarshalRecord() + if err != nil { + return nil, fmt.Errorf("error marshaling record: %v", err) + } + + domain := rec.Domain() + payloadType := rec.Codec() + if domain == "" { + return nil, ErrEmptyDomain + } + + if len(payloadType) == 0 { + return nil, ErrEmptyPayloadType + } + + unsigned, err := makeUnsigned(domain, payloadType, payload) + if err != nil { + return nil, err + } + defer pool.Put(unsigned) + + sig, err := privateKey.Sign(unsigned) + if err != nil { + return nil, err + } + + return &Envelope{ + PublicKey: privateKey.GetPublic(), + PayloadType: payloadType, + RawPayload: payload, + signature: sig, + }, nil +} + +// ConsumeEnvelope unmarshals a serialized Envelope and validates its +// signature using the provided 'domain' string. If validation fails, an error +// is returned, along with the unmarshalled envelope so it can be inspected. +// +// On success, ConsumeEnvelope returns the Envelope itself, as well as the inner payload, +// unmarshalled into a concrete Record type. The actual type of the returned Record depends +// on what has been registered for the Envelope's PayloadType (see RegisterType for details). +// +// You can type assert on the returned Record to convert it to an instance of the concrete +// Record type: +// +// envelope, rec, err := ConsumeEnvelope(envelopeBytes, peer.PeerRecordEnvelopeDomain) +// if err != nil { +// handleError(envelope, err) // envelope may be non-nil, even if errors occur! +// return +// } +// peerRec, ok := rec.(*peer.PeerRecord) +// if ok { +// doSomethingWithPeerRecord(peerRec) +// } +// +// Important: you MUST check the error value before using the returned Envelope. In some error +// cases, including when the envelope signature is invalid, both the Envelope and an error will +// be returned. This allows you to inspect the unmarshalled but invalid Envelope. As a result, +// you must not assume that any non-nil Envelope returned from this function is valid. +// +// If the Envelope signature is valid, but no Record type is registered for the Envelope's +// PayloadType, ErrPayloadTypeNotRegistered will be returned, along with the Envelope and +// a nil Record. +func ConsumeEnvelope(data []byte, domain string) (envelope *Envelope, rec Record, err error) { + e, err := UnmarshalEnvelope(data) + if err != nil { + return nil, nil, fmt.Errorf("failed when unmarshalling the envelope: %w", err) + } + + err = e.validate(domain) + if err != nil { + return e, nil, fmt.Errorf("failed to validate envelope: %w", err) + } + + rec, err = e.Record() + if err != nil { + return e, nil, fmt.Errorf("failed to unmarshal envelope payload: %w", err) + } + return e, rec, nil +} + +// ConsumeTypedEnvelope unmarshals a serialized Envelope and validates its +// signature. If validation fails, an error is returned, along with the unmarshalled +// envelope so it can be inspected. +// +// Unlike ConsumeEnvelope, ConsumeTypedEnvelope does not try to automatically determine +// the type of Record to unmarshal the Envelope's payload into. Instead, the caller provides +// a destination Record instance, which will unmarshal the Envelope payload. It is the caller's +// responsibility to determine whether the given Record type is able to unmarshal the payload +// correctly. +// +// rec := &MyRecordType{} +// envelope, err := ConsumeTypedEnvelope(envelopeBytes, rec) +// if err != nil { +// handleError(envelope, err) +// } +// doSomethingWithRecord(rec) +// +// Important: you MUST check the error value before using the returned Envelope. In some error +// cases, including when the envelope signature is invalid, both the Envelope and an error will +// be returned. This allows you to inspect the unmarshalled but invalid Envelope. As a result, +// you must not assume that any non-nil Envelope returned from this function is valid. +func ConsumeTypedEnvelope(data []byte, destRecord Record) (envelope *Envelope, err error) { + e, err := UnmarshalEnvelope(data) + if err != nil { + return nil, fmt.Errorf("failed when unmarshalling the envelope: %w", err) + } + + err = e.validate(destRecord.Domain()) + if err != nil { + return e, fmt.Errorf("failed to validate envelope: %w", err) + } + + err = destRecord.UnmarshalRecord(e.RawPayload) + if err != nil { + return e, fmt.Errorf("failed to unmarshal envelope payload: %w", err) + } + e.cached = destRecord + return e, nil +} + +// UnmarshalEnvelope unmarshals a serialized Envelope protobuf message, +// without validating its contents. Most users should use ConsumeEnvelope. +func UnmarshalEnvelope(data []byte) (*Envelope, error) { + var e pb.Envelope + if err := proto.Unmarshal(data, &e); err != nil { + return nil, err + } + + key, err := crypto.PublicKeyFromProto(e.PublicKey) + if err != nil { + return nil, err + } + + return &Envelope{ + PublicKey: key, + PayloadType: e.PayloadType, + RawPayload: e.Payload, + signature: e.Signature, + }, nil +} + +// Marshal returns a byte slice containing a serialized protobuf representation +// of a Envelope. +func (e *Envelope) Marshal() ([]byte, error) { + key, err := crypto.PublicKeyToProto(e.PublicKey) + if err != nil { + return nil, err + } + + msg := pb.Envelope{ + PublicKey: key, + PayloadType: e.PayloadType, + Payload: e.RawPayload, + Signature: e.signature, + } + return proto.Marshal(&msg) +} + +// Equal returns true if the other Envelope has the same public key, +// payload, payload type, and signature. This implies that they were also +// created with the same domain string. +func (e *Envelope) Equal(other *Envelope) bool { + if other == nil { + return e == nil + } + return e.PublicKey.Equals(other.PublicKey) && + bytes.Compare(e.PayloadType, other.PayloadType) == 0 && + bytes.Compare(e.signature, other.signature) == 0 && + bytes.Compare(e.RawPayload, other.RawPayload) == 0 +} + +// Record returns the Envelope's payload unmarshalled as a Record. +// The concrete type of the returned Record depends on which Record +// type was registered for the Envelope's PayloadType - see record.RegisterType. +// +// Once unmarshalled, the Record is cached for future access. +func (e *Envelope) Record() (Record, error) { + e.unmarshalOnce.Do(func() { + if e.cached != nil { + return + } + e.cached, e.unmarshalError = unmarshalRecordPayload(e.PayloadType, e.RawPayload) + }) + return e.cached, e.unmarshalError +} + +// TypedRecord unmarshals the Envelope's payload to the given Record instance. +// It is the caller's responsibility to ensure that the Record type is capable +// of unmarshalling the Envelope payload. Callers can inspect the Envelope's +// PayloadType field to determine the correct type of Record to use. +// +// This method will always unmarshal the Envelope payload even if a cached record +// exists. +func (e *Envelope) TypedRecord(dest Record) error { + return dest.UnmarshalRecord(e.RawPayload) +} + +// validate returns nil if the envelope signature is valid for the given 'domain', +// or an error if signature validation fails. +func (e *Envelope) validate(domain string) error { + unsigned, err := makeUnsigned(domain, e.PayloadType, e.RawPayload) + if err != nil { + return err + } + defer pool.Put(unsigned) + + valid, err := e.PublicKey.Verify(unsigned, e.signature) + if err != nil { + return fmt.Errorf("failed while verifying signature: %w", err) + } + if !valid { + return ErrInvalidSignature + } + return nil +} + +// makeUnsigned is a helper function that prepares a buffer to sign or verify. +// It returns a byte slice from a pool. The caller MUST return this slice to the +// pool. +func makeUnsigned(domain string, payloadType []byte, payload []byte) ([]byte, error) { + var ( + fields = [][]byte{[]byte(domain), payloadType, payload} + + // fields are prefixed with their length as an unsigned varint. we + // compute the lengths before allocating the sig buffer so we know how + // much space to add for the lengths + flen = make([][]byte, len(fields)) + size = 0 + ) + + for i, f := range fields { + l := len(f) + flen[i] = varint.ToUvarint(uint64(l)) + size += l + len(flen[i]) + } + + b := pool.Get(size) + + var s int + for i, f := range fields { + s += copy(b[s:], flen[i]) + s += copy(b[s:], f) + } + + return b[:s], nil +} diff --git a/core/record/envelope_test.go b/core/record/envelope_test.go new file mode 100644 index 0000000000..3946a572b8 --- /dev/null +++ b/core/record/envelope_test.go @@ -0,0 +1,313 @@ +package record_test + +import ( + "bytes" + "errors" + "testing" + + crypto "github.com/libp2p/go-libp2p-core/crypto" + . "github.com/libp2p/go-libp2p-core/record" + pb "github.com/libp2p/go-libp2p-core/record/pb" + "github.com/libp2p/go-libp2p-core/test" + + "github.com/gogo/protobuf/proto" +) + +type simpleRecord struct { + testDomain *string + testCodec []byte + message string +} + +func (r *simpleRecord) Domain() string { + if r.testDomain != nil { + return *r.testDomain + } + return "libp2p-testing" +} + +func (r *simpleRecord) Codec() []byte { + if r.testCodec != nil { + return r.testCodec + } + return []byte("/libp2p/testdata") +} + +func (r *simpleRecord) MarshalRecord() ([]byte, error) { + return []byte(r.message), nil +} + +func (r *simpleRecord) UnmarshalRecord(buf []byte) error { + r.message = string(buf) + return nil +} + +// Make an envelope, verify & open it, marshal & unmarshal it +func TestEnvelopeHappyPath(t *testing.T) { + var ( + rec = &simpleRecord{message: "hello world!"} + priv, pub, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + test.AssertNilError(t, err) + + payload, err := rec.MarshalRecord() + test.AssertNilError(t, err) + + envelope, err := Seal(rec, priv) + test.AssertNilError(t, err) + + if !envelope.PublicKey.Equals(pub) { + t.Error("envelope has unexpected public key") + } + + if bytes.Compare(rec.Codec(), envelope.PayloadType) != 0 { + t.Error("PayloadType does not match record Codec") + } + + serialized, err := envelope.Marshal() + test.AssertNilError(t, err) + + RegisterType(&simpleRecord{}) + deserialized, rec2, err := ConsumeEnvelope(serialized, rec.Domain()) + test.AssertNilError(t, err) + + if bytes.Compare(deserialized.RawPayload, payload) != 0 { + t.Error("payload of envelope does not match input") + } + + if !envelope.Equal(deserialized) { + t.Error("round-trip serde results in unequal envelope structures") + } + + typedRec, ok := rec2.(*simpleRecord) + if !ok { + t.Error("expected ConsumeEnvelope to return record with type registered for payloadType") + } + if typedRec.message != "hello world!" { + t.Error("unexpected alteration of record") + } +} + +func TestConsumeTypedEnvelope(t *testing.T) { + var ( + rec = simpleRecord{message: "hello world!"} + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + envelope, err := Seal(&rec, priv) + test.AssertNilError(t, err) + + envelopeBytes, err := envelope.Marshal() + test.AssertNilError(t, err) + + rec2 := &simpleRecord{} + _, err = ConsumeTypedEnvelope(envelopeBytes, rec2) + test.AssertNilError(t, err) + + if rec2.message != "hello world!" { + t.Error("unexpected alteration of record") + } +} + +func TestMakeEnvelopeFailsWithEmptyDomain(t *testing.T) { + var ( + rec = simpleRecord{message: "hello world!"} + domain = "" + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + if err != nil { + t.Fatal(err) + } + + // override domain with empty string + rec.testDomain = &domain + + _, err = Seal(&rec, priv) + test.ExpectError(t, err, "making an envelope with an empty domain should fail") +} + +func TestMakeEnvelopeFailsWithEmptyPayloadType(t *testing.T) { + var ( + rec = simpleRecord{message: "hello world!"} + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + if err != nil { + t.Fatal(err) + } + + // override payload with empty slice + rec.testCodec = []byte{} + + _, err = Seal(&rec, priv) + test.ExpectError(t, err, "making an envelope with an empty payloadType should fail") +} + +type failingRecord struct { + allowMarshal bool + allowUnmarshal bool +} + +func (r failingRecord) Domain() string { + return "testing" +} + +func (r failingRecord) Codec() []byte { + return []byte("doesn't matter") +} + +func (r failingRecord) MarshalRecord() ([]byte, error) { + if r.allowMarshal { + return []byte{}, nil + } + return nil, errors.New("marshal failed") +} +func (r failingRecord) UnmarshalRecord(data []byte) error { + if r.allowUnmarshal { + return nil + } + return errors.New("unmarshal failed") +} + +func TestSealFailsIfRecordMarshalFails(t *testing.T) { + var ( + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + if err != nil { + t.Fatal(err) + } + rec := failingRecord{} + _, err = Seal(rec, priv) + test.ExpectError(t, err, "Seal should fail if Record fails to marshal") +} + +func TestConsumeEnvelopeFailsIfEnvelopeUnmarshalFails(t *testing.T) { + _, _, err := ConsumeEnvelope([]byte("not an Envelope protobuf"), "doesn't-matter") + test.ExpectError(t, err, "ConsumeEnvelope should fail if Envelope fails to unmarshal") +} + +func TestConsumeEnvelopeFailsIfRecordUnmarshalFails(t *testing.T) { + var ( + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + if err != nil { + t.Fatal(err) + } + + RegisterType(failingRecord{}) + rec := failingRecord{allowMarshal: true} + env, err := Seal(rec, priv) + test.AssertNilError(t, err) + envBytes, err := env.Marshal() + test.AssertNilError(t, err) + + _, _, err = ConsumeEnvelope(envBytes, rec.Domain()) + test.ExpectError(t, err, "ConsumeEnvelope should fail if Record fails to unmarshal") +} + +func TestConsumeTypedEnvelopeFailsIfRecordUnmarshalFails(t *testing.T) { + var ( + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + if err != nil { + t.Fatal(err) + } + + RegisterType(failingRecord{}) + rec := failingRecord{allowMarshal: true} + env, err := Seal(rec, priv) + test.AssertNilError(t, err) + envBytes, err := env.Marshal() + test.AssertNilError(t, err) + + rec2 := failingRecord{} + _, err = ConsumeTypedEnvelope(envBytes, rec2) + test.ExpectError(t, err, "ConsumeTypedEnvelope should fail if Record fails to unmarshal") +} + +func TestEnvelopeValidateFailsForDifferentDomain(t *testing.T) { + var ( + rec = &simpleRecord{message: "hello world"} + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + test.AssertNilError(t, err) + + envelope, err := Seal(rec, priv) + test.AssertNilError(t, err) + + serialized, err := envelope.Marshal() + + // try to open our modified envelope + _, _, err = ConsumeEnvelope(serialized, "wrong-domain") + test.ExpectError(t, err, "should not be able to open envelope with incorrect domain") +} + +func TestEnvelopeValidateFailsIfPayloadTypeIsAltered(t *testing.T) { + var ( + rec = &simpleRecord{message: "hello world!"} + domain = "libp2p-testing" + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + test.AssertNilError(t, err) + + envelope, err := Seal(rec, priv) + test.AssertNilError(t, err) + + serialized := alterMessageAndMarshal(t, envelope, func(msg *pb.Envelope) { + msg.PayloadType = []byte("foo") + }) + + // try to open our modified envelope + _, _, err = ConsumeEnvelope(serialized, domain) + test.ExpectError(t, err, "should not be able to open envelope with modified PayloadType") +} + +func TestEnvelopeValidateFailsIfContentsAreAltered(t *testing.T) { + var ( + rec = &simpleRecord{message: "hello world!"} + domain = "libp2p-testing" + priv, _, err = test.RandTestKeyPair(crypto.Ed25519, 256) + ) + + test.AssertNilError(t, err) + + envelope, err := Seal(rec, priv) + test.AssertNilError(t, err) + + serialized := alterMessageAndMarshal(t, envelope, func(msg *pb.Envelope) { + msg.Payload = []byte("totally legit, trust me") + }) + + // try to open our modified envelope + _, _, err = ConsumeEnvelope(serialized, domain) + test.ExpectError(t, err, "should not be able to open envelope with modified payload") +} + +// Since we're outside of the crypto package (to avoid import cycles with test package), +// we can't alter the fields in a Envelope directly. This helper marshals +// the envelope to a protobuf and calls the alterMsg function, which should +// alter the protobuf message. +// Returns the serialized altered protobuf message. +func alterMessageAndMarshal(t *testing.T, envelope *Envelope, alterMsg func(*pb.Envelope)) []byte { + t.Helper() + + serialized, err := envelope.Marshal() + test.AssertNilError(t, err) + + msg := pb.Envelope{} + err = proto.Unmarshal(serialized, &msg) + test.AssertNilError(t, err) + + alterMsg(&msg) + serialized, err = msg.Marshal() + test.AssertNilError(t, err) + + return serialized +} diff --git a/core/record/pb/Makefile b/core/record/pb/Makefile new file mode 100644 index 0000000000..7cf8222f89 --- /dev/null +++ b/core/record/pb/Makefile @@ -0,0 +1,11 @@ +PB = $(wildcard *.proto) +GO = $(PB:.proto=.pb.go) + +all: $(GO) + +%.pb.go: %.proto + protoc --proto_path=$(PWD):$(PWD)/../.. --gogofaster_out=. $< + +clean: + rm -f *.pb.go + rm -f *.go diff --git a/core/record/pb/envelope.pb.go b/core/record/pb/envelope.pb.go new file mode 100644 index 0000000000..412809f13a --- /dev/null +++ b/core/record/pb/envelope.pb.go @@ -0,0 +1,504 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: envelope.proto + +package record_pb + +import ( + fmt "fmt" + proto "github.com/gogo/protobuf/proto" + pb "github.com/libp2p/go-libp2p-core/crypto/pb" + io "io" + math "math" + math_bits "math/bits" +) + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package + +// Envelope encloses a signed payload produced by a peer, along with the public +// key of the keypair it was signed with so that it can be statelessly validated +// by the receiver. +// +// The payload is prefixed with a byte string that determines the type, so it +// can be deserialized deterministically. Often, this byte string is a +// multicodec. +type Envelope struct { + // public_key is the public key of the keypair the enclosed payload was + // signed with. + PublicKey *pb.PublicKey `protobuf:"bytes,1,opt,name=public_key,json=publicKey,proto3" json:"public_key,omitempty"` + // payload_type encodes the type of payload, so that it can be deserialized + // deterministically. + PayloadType []byte `protobuf:"bytes,2,opt,name=payload_type,json=payloadType,proto3" json:"payload_type,omitempty"` + // payload is the actual payload carried inside this envelope. + Payload []byte `protobuf:"bytes,3,opt,name=payload,proto3" json:"payload,omitempty"` + // signature is the signature produced by the private key corresponding to + // the enclosed public key, over the payload, prefixing a domain string for + // additional security. + Signature []byte `protobuf:"bytes,5,opt,name=signature,proto3" json:"signature,omitempty"` +} + +func (m *Envelope) Reset() { *m = Envelope{} } +func (m *Envelope) String() string { return proto.CompactTextString(m) } +func (*Envelope) ProtoMessage() {} +func (*Envelope) Descriptor() ([]byte, []int) { + return fileDescriptor_ee266e8c558e9dc5, []int{0} +} +func (m *Envelope) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *Envelope) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_Envelope.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *Envelope) XXX_Merge(src proto.Message) { + xxx_messageInfo_Envelope.Merge(m, src) +} +func (m *Envelope) XXX_Size() int { + return m.Size() +} +func (m *Envelope) XXX_DiscardUnknown() { + xxx_messageInfo_Envelope.DiscardUnknown(m) +} + +var xxx_messageInfo_Envelope proto.InternalMessageInfo + +func (m *Envelope) GetPublicKey() *pb.PublicKey { + if m != nil { + return m.PublicKey + } + return nil +} + +func (m *Envelope) GetPayloadType() []byte { + if m != nil { + return m.PayloadType + } + return nil +} + +func (m *Envelope) GetPayload() []byte { + if m != nil { + return m.Payload + } + return nil +} + +func (m *Envelope) GetSignature() []byte { + if m != nil { + return m.Signature + } + return nil +} + +func init() { + proto.RegisterType((*Envelope)(nil), "record.pb.Envelope") +} + +func init() { proto.RegisterFile("envelope.proto", fileDescriptor_ee266e8c558e9dc5) } + +var fileDescriptor_ee266e8c558e9dc5 = []byte{ + // 205 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x4b, 0xcd, 0x2b, 0x4b, + 0xcd, 0xc9, 0x2f, 0x48, 0xd5, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x2c, 0x4a, 0x4d, 0xce, + 0x2f, 0x4a, 0xd1, 0x2b, 0x48, 0x92, 0x12, 0x4b, 0x2e, 0xaa, 0x2c, 0x28, 0xc9, 0xd7, 0x2f, 0x48, + 0xd2, 0x87, 0xb0, 0x20, 0x4a, 0x94, 0x66, 0x31, 0x72, 0x71, 0xb8, 0x42, 0x75, 0x09, 0x19, 0x73, + 0x71, 0x15, 0x94, 0x26, 0xe5, 0x64, 0x26, 0xc7, 0x67, 0xa7, 0x56, 0x4a, 0x30, 0x2a, 0x30, 0x6a, + 0x70, 0x1b, 0x89, 0xe8, 0xc1, 0xd4, 0x27, 0xe9, 0x05, 0x80, 0x25, 0xbd, 0x53, 0x2b, 0x83, 0x38, + 0x0b, 0x60, 0x4c, 0x21, 0x45, 0x2e, 0x9e, 0x82, 0xc4, 0xca, 0x9c, 0xfc, 0xc4, 0x94, 0xf8, 0x92, + 0xca, 0x82, 0x54, 0x09, 0x26, 0x05, 0x46, 0x0d, 0x9e, 0x20, 0x6e, 0xa8, 0x58, 0x48, 0x65, 0x41, + 0xaa, 0x90, 0x04, 0x17, 0x3b, 0x94, 0x2b, 0xc1, 0x0c, 0x96, 0x85, 0x71, 0x85, 0x64, 0xb8, 0x38, + 0x8b, 0x33, 0xd3, 0xf3, 0x12, 0x4b, 0x4a, 0x8b, 0x52, 0x25, 0x58, 0xc1, 0x72, 0x08, 0x01, 0x27, + 0x89, 0x13, 0x8f, 0xe4, 0x18, 0x2f, 0x3c, 0x92, 0x63, 0x7c, 0xf0, 0x48, 0x8e, 0x71, 0xc2, 0x63, + 0x39, 0x86, 0x0b, 0x8f, 0xe5, 0x18, 0x6e, 0x3c, 0x96, 0x63, 0x48, 0x62, 0x03, 0xbb, 0xde, 0x18, + 0x10, 0x00, 0x00, 0xff, 0xff, 0xaa, 0x0b, 0xd9, 0x6d, 0xf2, 0x00, 0x00, 0x00, +} + +func (m *Envelope) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Envelope) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *Envelope) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if len(m.Signature) > 0 { + i -= len(m.Signature) + copy(dAtA[i:], m.Signature) + i = encodeVarintEnvelope(dAtA, i, uint64(len(m.Signature))) + i-- + dAtA[i] = 0x2a + } + if len(m.Payload) > 0 { + i -= len(m.Payload) + copy(dAtA[i:], m.Payload) + i = encodeVarintEnvelope(dAtA, i, uint64(len(m.Payload))) + i-- + dAtA[i] = 0x1a + } + if len(m.PayloadType) > 0 { + i -= len(m.PayloadType) + copy(dAtA[i:], m.PayloadType) + i = encodeVarintEnvelope(dAtA, i, uint64(len(m.PayloadType))) + i-- + dAtA[i] = 0x12 + } + if m.PublicKey != nil { + { + size, err := m.PublicKey.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintEnvelope(dAtA, i, uint64(size)) + } + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil +} + +func encodeVarintEnvelope(dAtA []byte, offset int, v uint64) int { + offset -= sovEnvelope(v) + base := offset + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return base +} +func (m *Envelope) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.PublicKey != nil { + l = m.PublicKey.Size() + n += 1 + l + sovEnvelope(uint64(l)) + } + l = len(m.PayloadType) + if l > 0 { + n += 1 + l + sovEnvelope(uint64(l)) + } + l = len(m.Payload) + if l > 0 { + n += 1 + l + sovEnvelope(uint64(l)) + } + l = len(m.Signature) + if l > 0 { + n += 1 + l + sovEnvelope(uint64(l)) + } + return n +} + +func sovEnvelope(x uint64) (n int) { + return (math_bits.Len64(x|1) + 6) / 7 +} +func sozEnvelope(x uint64) (n int) { + return sovEnvelope(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *Envelope) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowEnvelope + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Envelope: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Envelope: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field PublicKey", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowEnvelope + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthEnvelope + } + postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthEnvelope + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.PublicKey == nil { + m.PublicKey = &pb.PublicKey{} + } + if err := m.PublicKey.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field PayloadType", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowEnvelope + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthEnvelope + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthEnvelope + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.PayloadType = append(m.PayloadType[:0], dAtA[iNdEx:postIndex]...) + if m.PayloadType == nil { + m.PayloadType = []byte{} + } + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Payload", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowEnvelope + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthEnvelope + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthEnvelope + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Payload = append(m.Payload[:0], dAtA[iNdEx:postIndex]...) + if m.Payload == nil { + m.Payload = []byte{} + } + iNdEx = postIndex + case 5: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Signature", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowEnvelope + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthEnvelope + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthEnvelope + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Signature = append(m.Signature[:0], dAtA[iNdEx:postIndex]...) + if m.Signature == nil { + m.Signature = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipEnvelope(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthEnvelope + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthEnvelope + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipEnvelope(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + depth := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowEnvelope + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowEnvelope + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + case 1: + iNdEx += 8 + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowEnvelope + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if length < 0 { + return 0, ErrInvalidLengthEnvelope + } + iNdEx += length + case 3: + depth++ + case 4: + if depth == 0 { + return 0, ErrUnexpectedEndOfGroupEnvelope + } + depth-- + case 5: + iNdEx += 4 + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + if iNdEx < 0 { + return 0, ErrInvalidLengthEnvelope + } + if depth == 0 { + return iNdEx, nil + } + } + return 0, io.ErrUnexpectedEOF +} + +var ( + ErrInvalidLengthEnvelope = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowEnvelope = fmt.Errorf("proto: integer overflow") + ErrUnexpectedEndOfGroupEnvelope = fmt.Errorf("proto: unexpected end of group") +) diff --git a/core/record/pb/envelope.proto b/core/record/pb/envelope.proto new file mode 100644 index 0000000000..ca3555fbf7 --- /dev/null +++ b/core/record/pb/envelope.proto @@ -0,0 +1,30 @@ +syntax = "proto3"; + +package record.pb; + +import "crypto/pb/crypto.proto"; + +// Envelope encloses a signed payload produced by a peer, along with the public +// key of the keypair it was signed with so that it can be statelessly validated +// by the receiver. +// +// The payload is prefixed with a byte string that determines the type, so it +// can be deserialized deterministically. Often, this byte string is a +// multicodec. +message Envelope { + // public_key is the public key of the keypair the enclosed payload was + // signed with. + crypto.pb.PublicKey public_key = 1; + + // payload_type encodes the type of payload, so that it can be deserialized + // deterministically. + bytes payload_type = 2; + + // payload is the actual payload carried inside this envelope. + bytes payload = 3; + + // signature is the signature produced by the private key corresponding to + // the enclosed public key, over the payload, prefixing a domain string for + // additional security. + bytes signature = 5; +} diff --git a/core/record/record.go b/core/record/record.go new file mode 100644 index 0000000000..212005780e --- /dev/null +++ b/core/record/record.go @@ -0,0 +1,102 @@ +package record + +import ( + "errors" + "reflect" +) + +var ( + // ErrPayloadTypeNotRegistered is returned from ConsumeEnvelope when the Envelope's + // PayloadType does not match any registered Record types. + ErrPayloadTypeNotRegistered = errors.New("payload type is not registered") + + payloadTypeRegistry = make(map[string]reflect.Type) +) + +// Record represents a data type that can be used as the payload of an Envelope. +// The Record interface defines the methods used to marshal and unmarshal a Record +// type to a byte slice. +// +// Record types may be "registered" as the default for a given Envelope.PayloadType +// using the RegisterType function. Once a Record type has been registered, +// an instance of that type will be created and used to unmarshal the payload of +// any Envelope with the registered PayloadType when the Envelope is opened using +// the ConsumeEnvelope function. +// +// To use an unregistered Record type instead, use ConsumeTypedEnvelope and pass in +// an instance of the Record type that you'd like the Envelope's payload to be +// unmarshaled into. +type Record interface { + + // Domain is the "signature domain" used when signing and verifying a particular + // Record type. The Domain string should be unique to your Record type, and all + // instances of the Record type must have the same Domain string. + Domain() string + + // Codec is a binary identifier for this type of record, ideally a registered multicodec + // (see https://github.com/multiformats/multicodec). + // When a Record is put into an Envelope (see record.Seal), the Codec value will be used + // as the Envelope's PayloadType. When the Envelope is later unsealed, the PayloadType + // will be used to lookup the correct Record type to unmarshal the Envelope payload into. + Codec() []byte + + // MarshalRecord converts a Record instance to a []byte, so that it can be used as an + // Envelope payload. + MarshalRecord() ([]byte, error) + + // UnmarshalRecord unmarshals a []byte payload into an instance of a particular Record type. + UnmarshalRecord([]byte) error +} + +// RegisterType associates a binary payload type identifier with a concrete +// Record type. This is used to automatically unmarshal Record payloads from Envelopes +// when using ConsumeEnvelope, and to automatically marshal Records and determine the +// correct PayloadType when calling Seal. +// +// Callers must provide an instance of the record type to be registered, which must be +// a pointer type. Registration should be done in the init function of the package +// where the Record type is defined: +// +// package hello_record +// import record "github.com/libp2p/go-libp2p-core/record" +// +// func init() { +// record.RegisterType(&HelloRecord{}) +// } +// +// type HelloRecord struct { } // etc.. +// +func RegisterType(prototype Record) { + payloadTypeRegistry[string(prototype.Codec())] = getValueType(prototype) +} + +func unmarshalRecordPayload(payloadType []byte, payloadBytes []byte) (Record, error) { + rec, err := blankRecordForPayloadType(payloadType) + if err != nil { + return nil, err + } + err = rec.UnmarshalRecord(payloadBytes) + if err != nil { + return nil, err + } + return rec, nil +} + +func blankRecordForPayloadType(payloadType []byte) (Record, error) { + valueType, ok := payloadTypeRegistry[string(payloadType)] + if !ok { + return nil, ErrPayloadTypeNotRegistered + } + + val := reflect.New(valueType) + asRecord := val.Interface().(Record) + return asRecord, nil +} + +func getValueType(i interface{}) reflect.Type { + valueType := reflect.TypeOf(i) + if valueType.Kind() == reflect.Ptr { + valueType = valueType.Elem() + } + return valueType +} diff --git a/core/record/record_test.go b/core/record/record_test.go new file mode 100644 index 0000000000..637964020c --- /dev/null +++ b/core/record/record_test.go @@ -0,0 +1,51 @@ +package record + +import "testing" + +var testPayloadType = []byte("/libp2p/test/record/payload-type") + +type testPayload struct { + unmarshalPayloadCalled bool +} + +func (p *testPayload) Domain() string { + return "testing" +} + +func (p *testPayload) Codec() []byte { + return testPayloadType +} + +func (p *testPayload) MarshalRecord() ([]byte, error) { + return []byte("hello"), nil +} + +func (p *testPayload) UnmarshalRecord(bytes []byte) error { + p.unmarshalPayloadCalled = true + return nil +} + +func TestUnmarshalPayload(t *testing.T) { + t.Run("fails if payload type is unregistered", func(t *testing.T) { + _, err := unmarshalRecordPayload([]byte("unknown type"), []byte{}) + if err != ErrPayloadTypeNotRegistered { + t.Error("Expected error when unmarshalling payload with unregistered payload type") + } + }) + + t.Run("calls UnmarshalRecord on concrete Record type", func(t *testing.T) { + RegisterType(&testPayload{}) + + payload, err := unmarshalRecordPayload(testPayloadType, []byte{}) + if err != nil { + t.Errorf("unexpected error unmarshalling registered payload type: %v", err) + } + typedPayload, ok := payload.(*testPayload) + if !ok { + t.Error("expected unmarshalled payload to be of the correct type") + } + if !typedPayload.unmarshalPayloadCalled { + t.Error("expected UnmarshalRecord to be called on concrete Record instance") + } + }) +} diff --git a/core/test/addrs.go b/core/test/addrs.go new file mode 100644 index 0000000000..e18849c487 --- /dev/null +++ b/core/test/addrs.go @@ -0,0 +1,42 @@ +package test + +import ( + "fmt" + "testing" + + ma "github.com/multiformats/go-multiaddr" +) + +func GenerateTestAddrs(n int) []ma.Multiaddr { + out := make([]ma.Multiaddr, n) + for i := 0; i < n; i++ { + a, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/1.2.3.4/tcp/%d", i)) + if err != nil { + continue + } + out[i] = a + } + return out +} + +func AssertAddressesEqual(t *testing.T, exp, act []ma.Multiaddr) { + t.Helper() + if len(exp) != len(act) { + t.Fatalf("lengths not the same. expected %d, got %d\n", len(exp), len(act)) + } + + for _, a := range exp { + found := false + + for _, b := range act { + if a.Equal(b) { + found = true + break + } + } + + if !found { + t.Fatalf("expected address %s not found", a) + } + } +} diff --git a/core/test/errors.go b/core/test/errors.go new file mode 100644 index 0000000000..82a3e696f2 --- /dev/null +++ b/core/test/errors.go @@ -0,0 +1,19 @@ +package test + +import ( + "testing" +) + +func AssertNilError(t *testing.T, err error) { + t.Helper() + if err != nil { + t.Errorf("unexpected error: %v", err) + } +} + +func ExpectError(t *testing.T, err error, msg string) { + t.Helper() + if err == nil { + t.Error(msg) + } +} From c713c0ce22b03c9807336d31193285e693b02943 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 10 Feb 2020 15:18:15 -0500 Subject: [PATCH 1646/3965] Use `PeerRecord` value instead of pointer in `EvtLocalPeerRecordUpdated` (#113) --- core/event/addrs.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/event/addrs.go b/core/event/addrs.go index f260d149ba..42d3c7d878 100644 --- a/core/event/addrs.go +++ b/core/event/addrs.go @@ -78,5 +78,5 @@ type EvtLocalAddressesUpdated struct { type EvtLocalPeerRecordUpdated struct { // Record contains the updated peer.PeerRecord, wrapped in a record.Envelope and // signed by the Host's private key. - Record *record.Envelope + Record record.Envelope } From b952c16927e04a2aed5c4d69692a01e717d3db65 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 12 Feb 2020 17:04:29 -0800 Subject: [PATCH 1647/3965] rate limiting and selectivity of autonat svc * limits addresses for a peer (at most 4 chosen) - fix #39 * clears addresses before dialing back - fix #38 * global rate limit of 30 responses per (1 - 1.25 min) - fix #36 * only dial back on the source IP - fix #32 --- p2p/host/autonat/svc.go | 59 ++++++++++++++++++++++++++++++------ p2p/host/autonat/svc_test.go | 30 ++++++++++++++++++ 2 files changed, 80 insertions(+), 9 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index a685f3d36d..98af2941da 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -1,7 +1,10 @@ package autonat import ( + "bytes" "context" + "math/rand" + "net" "sync" "time" @@ -24,8 +27,11 @@ const P_CIRCUIT = 290 var ( AutoNATServiceDialTimeout = 15 * time.Second AutoNATServiceResetInterval = 1 * time.Minute + AutoNATServiceResetJitter = 15 * time.Second - AutoNATServiceThrottle = 3 + AutoNATServiceThrottle = 3 + AutoNATGlobalThrottle = 30 + AutoNATMaxPeerAddresses = 4 ) // AutoNATService provides NAT autodetection services to other peers @@ -34,8 +40,9 @@ type AutoNATService struct { dialer host.Host // rate limiter - mx sync.Mutex - reqs map[peer.ID]int + mx sync.Mutex + reqs map[peer.ID]int + globalReqs int } // NewAutoNATService creates a new AutoNATService instance attached to a host @@ -96,6 +103,21 @@ func (as *AutoNATService) handleStream(s network.Stream) { } } +// Optimistically extract the net.IP host from a multiaddress. +func addrToIP(addr ma.Multiaddr) net.IP { + if v4, err := addr.ValueForProtocol(ma.P_IP4); err == nil { + if c, err := ma.NewComponent(ma.ProtocolWithCode(ma.P_IP4).Name, v4); err == nil { + return net.IP(c.RawValue()) + } + } + if v6, err := addr.ValueForProtocol(ma.P_IP6); err == nil { + if c, err := ma.NewComponent(ma.ProtocolWithCode(ma.P_IP6).Name, v6); err == nil { + return net.IP(c.RawValue()) + } + } + return nil +} + func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { if mpi == nil { return newDialResponseError(pb.Message_E_BAD_REQUEST, "missing peer info") @@ -113,13 +135,15 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me } } - addrs := make([]ma.Multiaddr, 0) + addrs := make([]ma.Multiaddr, 0, AutoNATMaxPeerAddresses) seen := make(map[string]struct{}) // add observed addr to the list of addresses to dial + var obsHost net.IP if !as.skipDial(obsaddr) { addrs = append(addrs, obsaddr) seen[obsaddr.String()] = struct{}{} + obsHost = addrToIP(obsaddr) } for _, maddr := range mpi.GetAddrs() { @@ -129,10 +153,22 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me continue } + if len(addrs) >= AutoNATMaxPeerAddresses { + continue + } + if as.skipDial(addr) { continue } + if err != nil { + log.Debugf("Unexpected public, non-IP multiaddr: %s", err) + continue + } + if !bytes.Equal(obsHost, addrToIP(addr)) { + continue + } + str := addr.String() _, ok := seen[str] if ok { @@ -169,16 +205,19 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { // rate limit check as.mx.Lock() count := as.reqs[pi.ID] - if count >= AutoNATServiceThrottle { + if count >= AutoNATServiceThrottle || as.globalReqs >= AutoNATGlobalThrottle { as.mx.Unlock() return newDialResponseError(pb.Message_E_DIAL_REFUSED, "too many dials") } as.reqs[pi.ID] = count + 1 + as.globalReqs++ as.mx.Unlock() ctx, cancel := context.WithTimeout(as.ctx, AutoNATServiceDialTimeout) defer cancel() + as.dialer.Peerstore().ClearAddrs(pi.ID) + err := as.dialer.Connect(ctx, pi) if err != nil { log.Debugf("error dialing %s: %s", pi.ID.Pretty(), err.Error()) @@ -200,16 +239,18 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { } func (as *AutoNATService) resetRateLimiter() { - ticker := time.NewTicker(AutoNATServiceResetInterval) - defer ticker.Stop() + timer := time.NewTimer(AutoNATServiceResetInterval) + defer timer.Stop() for { select { - case <-ticker.C: + case <-timer.C: as.mx.Lock() as.reqs = make(map[peer.ID]int) + as.globalReqs = 0 as.mx.Unlock() - + jitter := rand.Float32() * float32(AutoNATServiceResetJitter) + timer.Reset(AutoNATServiceResetInterval + time.Duration(int64(jitter))) case <-as.ctx.Done(): return } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 9ef9ec0197..f2aa91a6b3 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -11,6 +11,7 @@ import ( "github.com/libp2p/go-libp2p-core/peer" autonat "github.com/libp2p/go-libp2p-autonat" + ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) @@ -101,6 +102,8 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { AutoNATServiceThrottle = 1 save4 := manet.Private4 manet.Private4 = []*net.IPNet{} + save5 := AutoNATServiceResetJitter + AutoNATServiceResetJitter = 0 * time.Second hs, _ := makeAutoNATService(ctx, t) hc, ac := makeAutoNATClient(ctx, t) @@ -131,4 +134,31 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { AutoNATServiceResetInterval = save2 AutoNATServiceThrottle = save3 manet.Private4 = save4 + AutoNATServiceResetJitter = save5 +} + +func TestAddrToIP(t *testing.T) { + addr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") + if !addrToIP(addr).Equal(net.IPv4(127, 0, 0, 1)) { + t.Fatal("addrToIP of ipv4 localhost incorrect!") + } + + addr, _ = ma.NewMultiaddr("/ip4/192.168.0.1/tcp/6") + if !addrToIP(addr).Equal(net.IPv4(192, 168, 0, 1)) { + t.Fatal("addrToIP of ipv4 incorrect!") + } + + addr, _ = ma.NewMultiaddr("/ip6/::ffff:127.0.0.1/tcp/111") + if !addrToIP(addr).Equal(net.ParseIP("::ffff:127.0.0.1")) { + t.Fatal("addrToIP of ipv6 incorrect!") + } + addr, _ = ma.NewMultiaddr("/ip6zone/eth0/ip6/fe80::1") + if !addrToIP(addr).Equal(net.ParseIP("fe80::1")) { + t.Fatal("addrToIP of ip6zone incorrect!") + } + + addr, _ = ma.NewMultiaddr("/unix/a/b/c/d") + if addrToIP(addr) != nil { + t.Fatal("invalid addrToIP populates") + } } From 81c4f0ad249b25fd692386ae4440cfad5a737bc7 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 17 Feb 2020 10:34:32 +0700 Subject: [PATCH 1648/3965] increase the stream and connection receive windows --- p2p/transport/quic/transport.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 03b55ca2bc..3445a638f4 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -12,15 +12,15 @@ import ( quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" - "github.com/multiformats/go-multiaddr-fmt" + mafmt "github.com/multiformats/go-multiaddr-fmt" manet "github.com/multiformats/go-multiaddr-net" ) var quicConfig = &quic.Config{ MaxIncomingStreams: 1000, - MaxIncomingUniStreams: -1, // disable unidirectional streams - MaxReceiveStreamFlowControlWindow: 3 * (1 << 20), // 3 MB - MaxReceiveConnectionFlowControlWindow: 4.5 * (1 << 20), // 4.5 MB + MaxIncomingUniStreams: -1, // disable unidirectional streams + MaxReceiveStreamFlowControlWindow: 10 * (1 << 20), // 10 MB + MaxReceiveConnectionFlowControlWindow: 15 * (1 << 20), // 15 MB AcceptToken: func(clientAddr net.Addr, _ *quic.Token) bool { // TODO(#6): require source address validation when under load return true From 67313c2c8adae9a2fe663d2778d6d2ae4cf94e5b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 17 Feb 2020 11:32:02 -0500 Subject: [PATCH 1649/3965] chore(dep): update go-log --- p2p/host/peerstore/pstoreds/addr_book.go | 2 +- p2p/host/peerstore/pstoreds/addr_book_gc.go | 50 ++++++++++----------- p2p/host/peerstore/pstoremem/addr_book.go | 4 +- 3 files changed, 28 insertions(+), 28 deletions(-) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 2f75265c5b..9e04283463 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -288,7 +288,7 @@ func (ab *dsAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.D func (ab *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { pr, err := ab.loadRecord(p, true, true) if err != nil { - log.Warning("failed to load peerstore entry for peer %v while querying addrs, err: %v", p, err) + log.Warn("failed to load peerstore entry for peer %v while querying addrs, err: %v", p, err) return nil } diff --git a/p2p/host/peerstore/pstoreds/addr_book_gc.go b/p2p/host/peerstore/pstoreds/addr_book_gc.go index 6ff1a0f05e..808a5aa392 100644 --- a/p2p/host/peerstore/pstoreds/addr_book_gc.go +++ b/p2p/host/peerstore/pstoreds/addr_book_gc.go @@ -142,7 +142,7 @@ func (gc *dsAddrBookGc) purgeLookahead() { record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} // empty record to reuse and avoid allocs. batch, err := newCyclicBatch(gc.ab.ds, defaultOpsPerCyclicBatch) if err != nil { - log.Warningf("failed while creating batch to purge GC entries: %v", err) + log.Warnf("failed while creating batch to purge GC entries: %v", err) } // This function drops an unparseable GC entry; this is for safety. It is an escape hatch in case @@ -150,10 +150,10 @@ func (gc *dsAddrBookGc) purgeLookahead() { // if we don't clean up unparseable entries we'll end up accumulating garbage. dropInError := func(key ds.Key, err error, msg string) { if err != nil { - log.Warningf("failed while %s record with GC key: %v, err: %v; deleting", msg, key, err) + log.Warnf("failed while %s record with GC key: %v, err: %v; deleting", msg, key, err) } if err = batch.Delete(key); err != nil { - log.Warningf("failed to delete corrupt GC lookahead entry: %v, err: %v", key, err) + log.Warnf("failed to delete corrupt GC lookahead entry: %v, err: %v", key, err) } } @@ -161,21 +161,21 @@ func (gc *dsAddrBookGc) purgeLookahead() { // if the next earliest expiry falls within the current window again. dropOrReschedule := func(key ds.Key, ar *addrsRecord) { if err := batch.Delete(key); err != nil { - log.Warningf("failed to delete lookahead entry: %v, err: %v", key, err) + log.Warnf("failed to delete lookahead entry: %v, err: %v", key, err) } // re-add the record if it needs to be visited again in this window. if len(ar.Addrs) != 0 && ar.Addrs[0].Expiry <= gc.currWindowEnd { gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", ar.Addrs[0].Expiry, key.Name())) if err := batch.Put(gcKey, []byte{}); err != nil { - log.Warningf("failed to add new GC key: %v, err: %v", gcKey, err) + log.Warnf("failed to add new GC key: %v, err: %v", gcKey, err) } } } results, err := gc.ab.ds.Query(purgeLookaheadQuery) if err != nil { - log.Warningf("failed while fetching entries to purge: %v", err) + log.Warnf("failed while fetching entries to purge: %v", err) return } defer results.Close() @@ -189,7 +189,7 @@ func (gc *dsAddrBookGc) purgeLookahead() { ts, err := strconv.ParseInt(gcKey.Parent().Name(), 10, 64) if err != nil { dropInError(gcKey, err, "parsing timestamp") - log.Warningf("failed while parsing timestamp from key: %v, err: %v", result.Key, err) + log.Warnf("failed while parsing timestamp from key: %v, err: %v", result.Key, err) continue } else if ts > now { // this is an ordered cursor; when we hit an entry with a timestamp beyond now, we can break. @@ -199,14 +199,14 @@ func (gc *dsAddrBookGc) purgeLookahead() { idb32, err := b32.RawStdEncoding.DecodeString(gcKey.Name()) if err != nil { dropInError(gcKey, err, "parsing peer ID") - log.Warningf("failed while parsing b32 peer ID from key: %v, err: %v", result.Key, err) + log.Warnf("failed while parsing b32 peer ID from key: %v, err: %v", result.Key, err) continue } id, err = peer.IDFromBytes(idb32) if err != nil { dropInError(gcKey, err, "decoding peer ID") - log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) + log.Warnf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) continue } @@ -216,7 +216,7 @@ func (gc *dsAddrBookGc) purgeLookahead() { cached.Lock() if cached.clean() { if err = cached.flush(batch); err != nil { - log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) + log.Warnf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) } } dropOrReschedule(gcKey, cached) @@ -242,14 +242,14 @@ func (gc *dsAddrBookGc) purgeLookahead() { if record.clean() { err = record.flush(batch) if err != nil { - log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) + log.Warnf("failed to flush entry modified by GC for peer: &v, err: %v", id.Pretty(), err) } } dropOrReschedule(gcKey, record) } if err = batch.Commit(); err != nil { - log.Warningf("failed to commit GC purge batch: %v", err) + log.Warnf("failed to commit GC purge batch: %v", err) } } @@ -265,12 +265,12 @@ func (gc *dsAddrBookGc) purgeStore() { record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} // empty record to reuse and avoid allocs. batch, err := newCyclicBatch(gc.ab.ds, defaultOpsPerCyclicBatch) if err != nil { - log.Warningf("failed while creating batch to purge GC entries: %v", err) + log.Warnf("failed while creating batch to purge GC entries: %v", err) } results, err := gc.ab.ds.Query(purgeStoreQuery) if err != nil { - log.Warningf("failed while opening iterator: %v", err) + log.Warnf("failed while opening iterator: %v", err) return } defer results.Close() @@ -289,13 +289,13 @@ func (gc *dsAddrBookGc) purgeStore() { } if err := record.flush(batch); err != nil { - log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id, err) + log.Warnf("failed to flush entry modified by GC for peer: &v, err: %v", id, err) } gc.ab.cache.Remove(id) } if err = batch.Commit(); err != nil { - log.Warningf("failed to commit GC purge batch: %v", err) + log.Warnf("failed to commit GC purge batch: %v", err) } } @@ -323,14 +323,14 @@ func (gc *dsAddrBookGc) populateLookahead() { record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} results, err := gc.ab.ds.Query(populateLookaheadQuery) if err != nil { - log.Warningf("failed while querying to populate lookahead GC window: %v", err) + log.Warnf("failed while querying to populate lookahead GC window: %v", err) return } defer results.Close() batch, err := newCyclicBatch(gc.ab.ds, defaultOpsPerCyclicBatch) if err != nil { - log.Warningf("failed while creating batch to populate lookahead GC window: %v", err) + log.Warnf("failed while creating batch to populate lookahead GC window: %v", err) return } @@ -338,11 +338,11 @@ func (gc *dsAddrBookGc) populateLookahead() { idb32 := ds.RawKey(result.Key).Name() k, err := b32.RawStdEncoding.DecodeString(idb32) if err != nil { - log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) + log.Warnf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) continue } if id, err = peer.IDFromBytes(k); err != nil { - log.Warningf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) + log.Warnf("failed while decoding peer ID from key: %v, err: %v", result.Key, err) } // if the record is in cache, use the cached version. @@ -355,7 +355,7 @@ func (gc *dsAddrBookGc) populateLookahead() { } gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", cached.Addrs[0].Expiry, idb32)) if err = batch.Put(gcKey, []byte{}); err != nil { - log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) + log.Warnf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) } cached.RUnlock() continue @@ -365,23 +365,23 @@ func (gc *dsAddrBookGc) populateLookahead() { val, err := gc.ab.ds.Get(ds.RawKey(result.Key)) if err != nil { - log.Warningf("failed which getting record from store for peer: %v, err: %v", id.Pretty(), err) + log.Warnf("failed which getting record from store for peer: %v, err: %v", id.Pretty(), err) continue } if err := record.Unmarshal(val); err != nil { - log.Warningf("failed while unmarshalling record from store for peer: %v, err: %v", id.Pretty(), err) + log.Warnf("failed while unmarshalling record from store for peer: %v, err: %v", id.Pretty(), err) continue } if len(record.Addrs) > 0 && record.Addrs[0].Expiry <= until { gcKey := gcLookaheadBase.ChildString(fmt.Sprintf("%d/%s", record.Addrs[0].Expiry, idb32)) if err = batch.Put(gcKey, []byte{}); err != nil { - log.Warningf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) + log.Warnf("failed while inserting GC entry for peer: %v, err: %v", id.Pretty(), err) } } } if err = batch.Commit(); err != nil { - log.Warningf("failed to commit GC lookahead batch: %v", err) + log.Warnf("failed to commit GC lookahead batch: %v", err) } gc.currWindowEnd = until diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index 90ffb175bf..ec0b1c92c4 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -151,7 +151,7 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du exp := time.Now().Add(ttl) for _, addr := range addrs { if addr == nil { - log.Warningf("was passed nil multiaddr for %s", p) + log.Warnf("was passed nil multiaddr for %s", p) continue } asBytes := addr.Bytes() @@ -194,7 +194,7 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du exp := time.Now().Add(ttl) for _, addr := range addrs { if addr == nil { - log.Warningf("was passed nil multiaddr for %s", p) + log.Warnf("was passed nil multiaddr for %s", p) continue } From 9cdba6a4cb910f21368cd1051c32740b4a18c97b Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 19 Feb 2020 18:25:26 +0700 Subject: [PATCH 1650/3965] fix key comparisons in tests --- p2p/transport/quic/conn_test.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 882f2f0138..40a6bca89a 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -78,11 +78,11 @@ var _ = Describe("Connection", func() { Expect(conn.LocalPeer()).To(Equal(clientID)) Expect(conn.LocalPrivateKey()).To(Equal(clientKey)) Expect(conn.RemotePeer()).To(Equal(serverID)) - Expect(conn.RemotePublicKey()).To(Equal(serverKey.GetPublic())) + Expect(conn.RemotePublicKey().Equals(serverKey.GetPublic())).To(BeTrue()) Expect(serverConn.LocalPeer()).To(Equal(serverID)) Expect(serverConn.LocalPrivateKey()).To(Equal(serverKey)) Expect(serverConn.RemotePeer()).To(Equal(clientID)) - Expect(serverConn.RemotePublicKey()).To(Equal(clientKey.GetPublic())) + Expect(serverConn.RemotePublicKey().Equals(clientKey.GetPublic())).To(BeTrue()) }) It("handshakes on IPv6", func() { @@ -102,11 +102,11 @@ var _ = Describe("Connection", func() { Expect(conn.LocalPeer()).To(Equal(clientID)) Expect(conn.LocalPrivateKey()).To(Equal(clientKey)) Expect(conn.RemotePeer()).To(Equal(serverID)) - Expect(conn.RemotePublicKey()).To(Equal(serverKey.GetPublic())) + Expect(conn.RemotePublicKey().Equals(serverKey.GetPublic())).To(BeTrue()) Expect(serverConn.LocalPeer()).To(Equal(serverID)) Expect(serverConn.LocalPrivateKey()).To(Equal(serverKey)) Expect(serverConn.RemotePeer()).To(Equal(clientID)) - Expect(serverConn.RemotePublicKey()).To(Equal(clientKey.GetPublic())) + Expect(serverConn.RemotePublicKey().Equals(clientKey.GetPublic())).To(BeTrue()) }) It("opens and accepts streams", func() { From 7ba322244e0af7da4817fe33e6039fac7fa10098 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 21 Feb 2020 12:35:36 -0500 Subject: [PATCH 1651/3965] emit identify events and avoid advertising localhost MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(identify): announce localhost as long as one side of the connection is local (#742) * identify: emit events on completion/failure. (#660) * identify: Update addr advertise logic to exclude localhost addrs… (#657) This is a rollup of the stabilize fixes. Co-authored-by: bigs Co-authored-by: Raúl Kripalani --- go.mod | 2 +- go.sum | 11 ++-- libp2p.go | 2 +- p2p/protocol/identify/id.go | 40 +++++++++++--- p2p/protocol/identify/id_test.go | 89 ++++++++++++++++++++++++++++++++ 5 files changed, 133 insertions(+), 11 deletions(-) diff --git a/go.mod b/go.mod index 7cb953c71b..a215127b59 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,7 @@ require ( github.com/libp2p/go-libp2p-autonat v0.1.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 - github.com/libp2p/go-libp2p-core v0.3.0 + github.com/libp2p/go-libp2p-core v0.3.1 github.com/libp2p/go-libp2p-discovery v0.2.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 diff --git a/go.sum b/go.sum index 09d410c936..3cfebf64c4 100644 --- a/go.sum +++ b/go.sum @@ -153,10 +153,15 @@ github.com/libp2p/go-libp2p-core v0.2.0 h1:ycFtuNwtZBAJSxzaHbyv6NjG3Yj5Nmra1csHa github.com/libp2p/go-libp2p-core v0.2.0/go.mod h1:X0eyB0Gy93v0DZtSYbEM7RnMChm9Uv3j7yRXjO77xSI= github.com/libp2p/go-libp2p-core v0.2.2 h1:Sv1ggdoMx9c7v7FOFkR7agraHCnAgqYsXrU1ARSRUMs= github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZVoVrLaDRvIRng0= -github.com/libp2p/go-libp2p-core v0.2.4 h1:Et6ykkTwI6PU44tr8qUF9k43vP0aduMNniShAbUJJw8= github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= github.com/libp2p/go-libp2p-core v0.3.0 h1:F7PqduvrztDtFsAa/bcheQ3azmNo+Nq7m8hQY5GiUW8= github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw= +github.com/libp2p/go-libp2p-core v0.3.1-0.20191230184106-204a57d1afe1 h1:3WfZnKXp/5B9EIwbV9ts0W681Q+K1/ZQdlPH2dBXAHg= +github.com/libp2p/go-libp2p-core v0.3.1-0.20191230184106-204a57d1afe1/go.mod h1:LRlbRHNBEfe8yypXy9+tfroNCfmGmYGjFGe5mU7bmWY= +github.com/libp2p/go-libp2p-core v0.3.1 h1:hEnSDjScfjYvPHoTgZhC4F62M8K1x1Oco/BY0RZ1N3s= +github.com/libp2p/go-libp2p-core v0.3.1/go.mod h1:thvWy0hvaSBhnVBaW37BvzgVV68OUhgJJLAa6almrII= +github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= +github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.2.0 h1:1p3YSOq7VsgaL+xVHPi8XAmtGyas6D2J6rWBEfz/aiY= github.com/libp2p/go-libp2p-discovery v0.2.0/go.mod h1:s4VGaxYMbw4+4+tsoQTqh7wfxg97AEdo4GYBt6BadWg= @@ -249,7 +254,6 @@ github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+ github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/minio/sha256-simd v0.1.0 h1:U41/2erhAKcmSI14xh/ZTUdBPOzDOIfS93ibzUSl8KM= github.com/minio/sha256-simd v0.1.0/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= -github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771 h1:MHkK1uRtFbVqvAgvWxafZe54+5uBxLluGylDiKgdhwo= github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU= github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= @@ -337,11 +341,13 @@ github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkU github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= github.com/src-d/envconfig v1.0.0/go.mod h1:Q9YQZ7BKITldTBnoxsE5gOeB5y66RyPXeue/R4aaNBc= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= @@ -422,7 +428,6 @@ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190226205152-f727befe758c h1:vamGzbGri8IKo20MQncCuljcQ5uAO6kaCeawQPVblAI= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= diff --git a/libp2p.go b/libp2p.go index 32fddbdb84..dbdd09cf1a 100644 --- a/libp2p.go +++ b/libp2p.go @@ -46,7 +46,7 @@ func ChainOptions(opts ...Option) Option { // - If no security transport is provided, the host uses the go-libp2p's secio // encrypted transport to encrypt all traffic; // -// - If no peer identity is provided, it generates a random RSA 2048 key-par +// - If no peer identity is provided, it generates a random RSA 2048 key-pair // and derives a new identity from it; // // - If no peerstore is provided, the host is initialized with an empty diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index 35e2e57ad8..a0e1ae25c3 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -23,6 +23,7 @@ import ( logging "github.com/ipfs/go-log" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" msmux "github.com/multiformats/go-multistream" ) @@ -85,7 +86,9 @@ type IDService struct { subscription event.Subscription emitters struct { - evtPeerProtocolsUpdated event.Emitter + evtPeerProtocolsUpdated event.Emitter + evtPeerIdentificationCompleted event.Emitter + evtPeerIdentificationFailed event.Emitter } } @@ -124,6 +127,14 @@ func NewIDService(ctx context.Context, h host.Host, opts ...Option) *IDService { if err != nil { log.Warningf("identify service not emitting peer protocol updates; err: %s", err) } + s.emitters.evtPeerIdentificationCompleted, err = h.EventBus().Emitter(&event.EvtPeerIdentificationCompleted{}) + if err != nil { + log.Warningf("identify service not emitting identification completed events; err: %s", err) + } + s.emitters.evtPeerIdentificationFailed, err = h.EventBus().Emitter(&event.EvtPeerIdentificationFailed{}) + if err != nil { + log.Warningf("identify service not emitting identification failed events; err: %s", err) + } h.SetStreamHandler(ID, s.requestHandler) h.SetStreamHandler(IDPush, s.pushHandler) @@ -164,6 +175,11 @@ func (ids *IDService) ObservedAddrsFor(local ma.Multiaddr) []ma.Multiaddr { } func (ids *IDService) IdentifyConn(c network.Conn) { + var ( + s network.Stream + err error + ) + ids.currmu.Lock() if wait, found := ids.currid[c]; found { ids.currmu.Unlock() @@ -180,9 +196,16 @@ func (ids *IDService) IdentifyConn(c network.Conn) { ids.currmu.Lock() delete(ids.currid, c) ids.currmu.Unlock() + + // emit the appropriate event. + if p := c.RemotePeer(); err == nil { + ids.emitters.evtPeerIdentificationCompleted.Emit(event.EvtPeerIdentificationCompleted{Peer: p}) + } else { + ids.emitters.evtPeerIdentificationFailed.Emit(event.EvtPeerIdentificationFailed{Peer: p, Reason: err}) + } }() - s, err := c.NewStream() + s, err = c.NewStream() if err != nil { log.Debugf("error opening initial stream for %s: %s", ID, err) log.Event(context.TODO(), "IdentifyOpenFailed", c.RemotePeer()) @@ -193,7 +216,7 @@ func (ids *IDService) IdentifyConn(c network.Conn) { s.SetProtocol(ID) // ok give the response to our handler. - if err := msmux.SelectProtoOrFail(ID, s); err != nil { + if err = msmux.SelectProtoOrFail(ID, s); err != nil { log.Event(context.TODO(), "IdentifyOpenFailed", c.RemotePeer(), logging.Metadata{"error": err}) s.Reset() return @@ -310,9 +333,14 @@ func (ids *IDService) populateMessage(mes *pb.Identify, c network.Conn) { // set listen addrs, get our latest addrs from Host. laddrs := ids.Host.Addrs() - mes.ListenAddrs = make([][]byte, len(laddrs)) - for i, addr := range laddrs { - mes.ListenAddrs[i] = addr.Bytes() + // Note: LocalMultiaddr is sometimes 0.0.0.0 + viaLoopback := manet.IsIPLoopback(c.LocalMultiaddr()) || manet.IsIPLoopback(c.RemoteMultiaddr()) + mes.ListenAddrs = make([][]byte, 0, len(laddrs)) + for _, addr := range laddrs { + if !viaLoopback && manet.IsIPLoopback(addr) { + continue + } + mes.ListenAddrs = append(mes.ListenAddrs, addr.Bytes()) } log.Debugf("%s sent listen addrs to %s: %s", c.LocalPeer(), c.RemotePeer(), laddrs) diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index 5c422a9092..234dbe4891 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -17,11 +17,14 @@ import ( "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/protocol" + coretest "github.com/libp2p/go-libp2p-core/test" blhost "github.com/libp2p/go-libp2p-blankhost" swarmt "github.com/libp2p/go-libp2p-swarm/testing" "github.com/libp2p/go-libp2p/p2p/protocol/identify" + "github.com/libp2p/go-libp2p-peerstore/pstoremem" + mocknet "github.com/libp2p/go-libp2p/p2p/net/mock" ma "github.com/multiformats/go-multiaddr" ) @@ -56,6 +59,11 @@ func subtestIDService(t *testing.T) { t.Fatal("should have a conn here") } + sub, err := ids1.Host.EventBus().Subscribe(new(event.EvtPeerIdentificationCompleted), eventbus.BufSize(16)) + if err != nil { + t.Fatal(err) + } + ids1.IdentifyConn(h1t2c[0]) // the IDService should be opened automatically, by the network. @@ -102,6 +110,13 @@ func subtestIDService(t *testing.T) { // Forget the rest. testKnowsAddrs(t, h1, h2p, []ma.Multiaddr{}) testKnowsAddrs(t, h2, h1p, []ma.Multiaddr{}) + + // test that we received the "identify completed" event. + select { + case <-sub.Out(): + case <-time.After(5 * time.Second): + t.Fatalf("expected EvtPeerIdentificationCompleted event within 5 seconds; none received") + } } func testKnowsAddrs(t *testing.T, h host.Host, p peer.ID, expected []ma.Multiaddr) { @@ -191,6 +206,80 @@ func TestProtoMatching(t *testing.T) { } } +func TestLocalhostAddrFiltering(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + mn := mocknet.New(ctx) + id1 := coretest.RandPeerIDFatal(t) + ps1 := pstoremem.NewPeerstore() + p1addr1, _ := ma.NewMultiaddr("/ip4/1.2.3.4/tcp/1234") + p1addr2, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/2345") + ps1.AddAddrs(id1, []ma.Multiaddr{p1addr1, p1addr2}, peerstore.PermanentAddrTTL) + p1, err := mn.AddPeerWithPeerstore(id1, ps1) + if err != nil { + t.Fatal(err) + } + + id2 := coretest.RandPeerIDFatal(t) + ps2 := pstoremem.NewPeerstore() + p2addr1, _ := ma.NewMultiaddr("/ip4/1.2.3.5/tcp/1234") + p2addr2, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/3456") + p2addrs := []ma.Multiaddr{p2addr1, p2addr2} + ps2.AddAddrs(id2, p2addrs, peerstore.PermanentAddrTTL) + p2, err := mn.AddPeerWithPeerstore(id2, ps2) + if err != nil { + t.Fatal(err) + } + + id3 := coretest.RandPeerIDFatal(t) + ps3 := pstoremem.NewPeerstore() + p3addr1, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/4567") + ps3.AddAddrs(id3, []ma.Multiaddr{p3addr1}, peerstore.PermanentAddrTTL) + p3, err := mn.AddPeerWithPeerstore(id3, ps3) + if err != nil { + t.Fatal(err) + } + + err = mn.LinkAll() + if err != nil { + t.Fatal(err) + } + p1.Connect(ctx, peer.AddrInfo{ + ID: id2, + Addrs: p2addrs[0:1], + }) + p3.Connect(ctx, peer.AddrInfo{ + ID: id2, + Addrs: p2addrs[1:], + }) + + _ = identify.NewIDService(ctx, p1) + ids2 := identify.NewIDService(ctx, p2) + ids3 := identify.NewIDService(ctx, p3) + + conns := p2.Network().ConnsToPeer(id1) + if len(conns) == 0 { + t.Fatal("no conns") + } + conn := conns[0] + ids2.IdentifyConn(conn) + addrs := p2.Peerstore().Addrs(id1) + if len(addrs) != 1 { + t.Fatalf("expected one addr, found %s", addrs) + } + + conns = p3.Network().ConnsToPeer(id2) + if len(conns) == 0 { + t.Fatal("no conns") + } + conn = conns[0] + ids3.IdentifyConn(conn) + addrs = p3.Peerstore().Addrs(id2) + if len(addrs) != 2 { + t.Fatalf("expected 2 addrs for %s, found %d: %s", id2, len(addrs), addrs) + } +} + func TestIdentifyDeltaOnProtocolChange(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() From 10b0a942b847c0797acf674d5c481e288ccf4530 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Fri, 21 Feb 2020 13:54:33 -0800 Subject: [PATCH 1652/3965] cleaner addrToIP implementation --- p2p/host/autonat/svc.go | 45 +++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 98af2941da..141f4ee776 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -3,8 +3,10 @@ package autonat import ( "bytes" "context" + "fmt" "math/rand" "net" + "strings" "sync" "time" @@ -31,7 +33,7 @@ var ( AutoNATServiceThrottle = 3 AutoNATGlobalThrottle = 30 - AutoNATMaxPeerAddresses = 4 + AutoNATMaxPeerAddresses = 16 ) // AutoNATService provides NAT autodetection services to other peers @@ -105,17 +107,30 @@ func (as *AutoNATService) handleStream(s network.Stream) { // Optimistically extract the net.IP host from a multiaddress. func addrToIP(addr ma.Multiaddr) net.IP { - if v4, err := addr.ValueForProtocol(ma.P_IP4); err == nil { - if c, err := ma.NewComponent(ma.ProtocolWithCode(ma.P_IP4).Name, v4); err == nil { - return net.IP(c.RawValue()) - } + n, ip, err := manet.DialArgs(addr) + if err != nil { + return nil } - if v6, err := addr.ValueForProtocol(ma.P_IP6); err == nil { - if c, err := ma.NewComponent(ma.ProtocolWithCode(ma.P_IP6).Name, v6); err == nil { - return net.IP(c.RawValue()) + + // if no port: + if n == "ip" || n == "ip4" || n == "ip6" { + // Strip v6 zone if it's there. + if strings.Contains(ip, "%") { + ip = ip[:strings.Index(ip, "%")] } + return net.ParseIP(ip) } - return nil + + ip, _, err = net.SplitHostPort(ip) + if err != nil { + fmt.Printf("failed to split: %v", err) + return nil + } + // Strip v6 zone if it's there. + if strings.Contains(ip, "%") { + ip = ip[:strings.Index(ip, "%")] + } + return net.ParseIP(ip) } func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { @@ -153,18 +168,10 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me continue } - if len(addrs) >= AutoNATMaxPeerAddresses { - continue - } - if as.skipDial(addr) { continue } - if err != nil { - log.Debugf("Unexpected public, non-IP multiaddr: %s", err) - continue - } if !bytes.Equal(obsHost, addrToIP(addr)) { continue } @@ -177,6 +184,10 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me addrs = append(addrs, addr) seen[str] = struct{}{} + + if len(addrs) >= AutoNATMaxPeerAddresses { + break + } } if len(addrs) == 0 { From ba298b7231e6f08176fcca5fb381efec1879fddd Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 21 Feb 2020 18:30:15 -0500 Subject: [PATCH 1653/3965] fix: less confusing log message fixes https://github.com/ipfs/go-ipfs/issues/6922 --- p2p/net/reuseport/reuseport.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/reuseport/reuseport.go b/p2p/net/reuseport/reuseport.go index 82f9f37c84..eb14068f33 100644 --- a/p2p/net/reuseport/reuseport.go +++ b/p2p/net/reuseport/reuseport.go @@ -57,7 +57,7 @@ func reuseDial(ctx context.Context, laddr *net.TCPAddr, network, raddr string) ( if reuseErrShouldRetry(err) && ctx.Err() == nil { // We could have an existing socket open or we could have one // stuck in TIME-WAIT. - log.Debugf("failed to reuse port, dialing with a random port: %s", err) + log.Debugf("failed to reuse port, will try again with a random port: %s", err) con, err = fallbackDialer.DialContext(ctx, network, raddr) } return con, err From 00b792cf6c38a5a62c9b5964957675b8257113f6 Mon Sep 17 00:00:00 2001 From: John B Nelson Date: Tue, 25 Feb 2020 09:02:52 -0800 Subject: [PATCH 1654/3965] Correct path to peer.AddrInfo in deprecation --- p2p/host/peerstore/peerinfo.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/peerinfo.go b/p2p/host/peerstore/peerinfo.go index 24c8c31f3a..d414b827ac 100644 --- a/p2p/host/peerstore/peerinfo.go +++ b/p2p/host/peerstore/peerinfo.go @@ -5,7 +5,7 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -// Deprecated: use github.com/libp2p/go-libp2p-core/peer.Info instead. +// Deprecated: use github.com/libp2p/go-libp2p-core/peer.AddrInfo instead. type PeerInfo = core.AddrInfo // Deprecated: use github.com/libp2p/go-libp2p-core/peer.ErrInvalidAddr instead. From 426d729c513205073e4fdc71c6d2dd5f87fa65b2 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 25 Feb 2020 10:24:47 -0800 Subject: [PATCH 1655/3965] cleanup addrToIP logic --- p2p/host/autonat/svc.go | 28 +++++++++++----------------- p2p/host/autonat/svc_test.go | 10 +++++----- 2 files changed, 16 insertions(+), 22 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 141f4ee776..1073e19022 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -3,7 +3,7 @@ package autonat import ( "bytes" "context" - "fmt" + "errors" "math/rand" "net" "strings" @@ -106,31 +106,25 @@ func (as *AutoNATService) handleStream(s network.Stream) { } // Optimistically extract the net.IP host from a multiaddress. -func addrToIP(addr ma.Multiaddr) net.IP { +func addrToIP(addr ma.Multiaddr) (net.IP, error) { n, ip, err := manet.DialArgs(addr) if err != nil { - return nil + return nil, err } - // if no port: - if n == "ip" || n == "ip4" || n == "ip6" { - // Strip v6 zone if it's there. - if strings.Contains(ip, "%") { - ip = ip[:strings.Index(ip, "%")] - } - return net.ParseIP(ip) + if strings.HasPrefix(n, "tcp") || strings.HasPrefix(n, "udp") { + ip, _, err = net.SplitHostPort(ip) + } else if !strings.HasPrefix(n, "ip") { + return nil, errors.New("non-ip multiaddr") } - - ip, _, err = net.SplitHostPort(ip) if err != nil { - fmt.Printf("failed to split: %v", err) - return nil + return nil, err } // Strip v6 zone if it's there. if strings.Contains(ip, "%") { ip = ip[:strings.Index(ip, "%")] } - return net.ParseIP(ip) + return net.ParseIP(ip), nil } func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { @@ -158,7 +152,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me if !as.skipDial(obsaddr) { addrs = append(addrs, obsaddr) seen[obsaddr.String()] = struct{}{} - obsHost = addrToIP(obsaddr) + obsHost, _ = addrToIP(obsaddr) } for _, maddr := range mpi.GetAddrs() { @@ -172,7 +166,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me continue } - if !bytes.Equal(obsHost, addrToIP(addr)) { + if ip, err := addrToIP(addr); err != nil || !bytes.Equal(obsHost, ip) { continue } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index f2aa91a6b3..9be52fbf61 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -139,26 +139,26 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { func TestAddrToIP(t *testing.T) { addr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") - if !addrToIP(addr).Equal(net.IPv4(127, 0, 0, 1)) { + if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.IPv4(127, 0, 0, 1)) { t.Fatal("addrToIP of ipv4 localhost incorrect!") } addr, _ = ma.NewMultiaddr("/ip4/192.168.0.1/tcp/6") - if !addrToIP(addr).Equal(net.IPv4(192, 168, 0, 1)) { + if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.IPv4(192, 168, 0, 1)) { t.Fatal("addrToIP of ipv4 incorrect!") } addr, _ = ma.NewMultiaddr("/ip6/::ffff:127.0.0.1/tcp/111") - if !addrToIP(addr).Equal(net.ParseIP("::ffff:127.0.0.1")) { + if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.ParseIP("::ffff:127.0.0.1")) { t.Fatal("addrToIP of ipv6 incorrect!") } addr, _ = ma.NewMultiaddr("/ip6zone/eth0/ip6/fe80::1") - if !addrToIP(addr).Equal(net.ParseIP("fe80::1")) { + if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.ParseIP("fe80::1")) { t.Fatal("addrToIP of ip6zone incorrect!") } addr, _ = ma.NewMultiaddr("/unix/a/b/c/d") - if addrToIP(addr) != nil { + if _, err := addrToIP(addr); err == nil { t.Fatal("invalid addrToIP populates") } } From 9dcbc44a26bc79e0d6f78dd8a150946d055d5db3 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 25 Feb 2020 12:24:33 -0800 Subject: [PATCH 1656/3965] limit autonat-svc to LocalRoutabilityPublic nodes per #43 --- p2p/host/autonat/svc.go | 17 +++++++++++++++-- p2p/host/autonat/svc_test.go | 3 +++ 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index a685f3d36d..c1f9e53ccd 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -6,6 +6,7 @@ import ( "time" "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/helpers" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" @@ -51,9 +52,21 @@ func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) dialer: dialer, reqs: make(map[peer.ID]int), } - h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) - go as.resetRateLimiter() + s, err := h.EventBus().Subscribe(&event.EvtLocalRoutabilityPublic{}) + if err != nil { + return nil, err + } + + go func() { + defer s.Close() + select { + case <-s.Out(): + h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) + go as.resetRateLimiter() + case <-ctx.Done(): + } + }() return as, nil } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 9ef9ec0197..eec48f07c2 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -7,6 +7,7 @@ import ( "time" "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/peer" @@ -24,6 +25,8 @@ func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATS if err != nil { t.Fatal(err) } + emitPublic, _ := h.EventBus().Emitter(new(event.EvtLocalRoutabilityPublic)) + emitPublic.Emit(event.EvtLocalRoutabilityPublic{}) return h, as } From a7319194056da86ba1f3656e173330a1fcff98ee Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 25 Feb 2020 19:41:48 -0800 Subject: [PATCH 1657/3965] fix: fire a listen close event when closing the listener fixes https://github.com/libp2p/go-libp2p-swarm/issues/163 --- p2p/net/swarm/swarm_listen.go | 7 ++++++- p2p/net/swarm/swarm_notif_test.go | 20 +++++++++++++++++--- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/p2p/net/swarm/swarm_listen.go b/p2p/net/swarm/swarm_listen.go index f1cfa9e7f1..09d411dfd8 100644 --- a/p2p/net/swarm/swarm_listen.go +++ b/p2p/net/swarm/swarm_listen.go @@ -61,7 +61,7 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { maddr := list.Multiaddr() - // signal to our notifiees on successful conn. + // signal to our notifiees on listen. s.notifyAll(func(n network.Notifiee) { n.Listen(s, maddr) }) @@ -73,6 +73,11 @@ func (s *Swarm) AddListenAddr(a ma.Multiaddr) error { delete(s.listeners.m, list) s.listeners.cacheEOL = time.Time{} s.listeners.Unlock() + + // signal to our notifiees on listen close. + s.notifyAll(func(n network.Notifiee) { + n.ListenClose(s, maddr) + }) s.refs.Done() }() for { diff --git a/p2p/net/swarm/swarm_notif_test.go b/p2p/net/swarm/swarm_notif_test.go index f2c91ebb80..60cf461f5f 100644 --- a/p2p/net/swarm/swarm_notif_test.go +++ b/p2p/net/swarm/swarm_notif_test.go @@ -16,18 +16,32 @@ import ( func TestNotifications(t *testing.T) { const swarmSize = 5 + notifiees := make([]*netNotifiee, swarmSize) + ctx := context.Background() swarms := makeSwarms(ctx, t, swarmSize) defer func() { - for _, s := range swarms { - s.Close() + for i, s := range swarms { + select { + case <-notifiees[i].listenClose: + t.Error("should not have been closed") + default: + } + err := s.Close() + if err != nil { + t.Error(err) + } + select { + case <-notifiees[i].listenClose: + default: + t.Error("expected a listen close notification") + } } }() timeout := 5 * time.Second // signup notifs - notifiees := make([]*netNotifiee, len(swarms)) for i, swarm := range swarms { n := newNetNotifiee(swarmSize) swarm.Notify(n) From 44e9abed05726d212064bd4b61d62b7e6f2aa17b Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 26 Feb 2020 12:27:44 -0800 Subject: [PATCH 1658/3965] address review comments use ip.Equal for comparison add test on jitter simplify addrToIP --- p2p/host/autonat/svc.go | 30 +++++++++++++----------------- p2p/host/autonat/svc_test.go | 24 ++++++++++++++++++++---- 2 files changed, 33 insertions(+), 21 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 1073e19022..1bf37683c7 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -1,12 +1,10 @@ package autonat import ( - "bytes" "context" - "errors" + "fmt" "math/rand" "net" - "strings" "sync" "time" @@ -106,25 +104,23 @@ func (as *AutoNATService) handleStream(s network.Stream) { } // Optimistically extract the net.IP host from a multiaddress. +// TODO: use upstream manet.ToIP func addrToIP(addr ma.Multiaddr) (net.IP, error) { - n, ip, err := manet.DialArgs(addr) + n, err := manet.ToNetAddr(addr) if err != nil { return nil, err } - if strings.HasPrefix(n, "tcp") || strings.HasPrefix(n, "udp") { - ip, _, err = net.SplitHostPort(ip) - } else if !strings.HasPrefix(n, "ip") { - return nil, errors.New("non-ip multiaddr") + switch netAddr := n.(type) { + case *net.UDPAddr: + return netAddr.IP, nil + case *net.TCPAddr: + return netAddr.IP, nil + case *net.IPAddr: + return netAddr.IP, nil + default: + return nil, fmt.Errorf("non IP Multiaddr: %T", netAddr) } - if err != nil { - return nil, err - } - // Strip v6 zone if it's there. - if strings.Contains(ip, "%") { - ip = ip[:strings.Index(ip, "%")] - } - return net.ParseIP(ip), nil } func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { @@ -166,7 +162,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me continue } - if ip, err := addrToIP(addr); err != nil || !bytes.Equal(obsHost, ip) { + if ip, err := addrToIP(addr); err != nil || !obsHost.Equal(ip) { continue } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 9be52fbf61..b93e6e28ac 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -137,6 +137,26 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { AutoNATServiceResetJitter = save5 } +func TestAutoNATServiceRateLimitJitter(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save1 := AutoNATServiceResetInterval + AutoNATServiceResetInterval = 100 * time.Millisecond + save2 := AutoNATServiceResetJitter + AutoNATServiceResetJitter = 100 * time.Millisecond + + _, svc := makeAutoNATService(ctx, t) + svc.globalReqs = 1 + time.Sleep(200 * time.Millisecond) + if svc.globalReqs != 0 { + t.Fatal("reset of rate limitter occured slower than expected") + } + + AutoNATServiceResetInterval = save1 + AutoNATServiceResetJitter = save2 +} + func TestAddrToIP(t *testing.T) { addr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.IPv4(127, 0, 0, 1)) { @@ -148,10 +168,6 @@ func TestAddrToIP(t *testing.T) { t.Fatal("addrToIP of ipv4 incorrect!") } - addr, _ = ma.NewMultiaddr("/ip6/::ffff:127.0.0.1/tcp/111") - if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.ParseIP("::ffff:127.0.0.1")) { - t.Fatal("addrToIP of ipv6 incorrect!") - } addr, _ = ma.NewMultiaddr("/ip6zone/eth0/ip6/fe80::1") if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.ParseIP("fe80::1")) { t.Fatal("addrToIP of ip6zone incorrect!") From 370d2efb54c338ce39c211fcdf1ec1d22a2d8391 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 26 Feb 2020 16:39:17 -0800 Subject: [PATCH 1659/3965] Add option for forcing service startup --- p2p/host/autonat/svc.go | 15 +++++++++------ p2p/host/autonat/svc_test.go | 5 +---- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index c1f9e53ccd..8c1191e3c3 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -40,7 +40,7 @@ type AutoNATService struct { } // NewAutoNATService creates a new AutoNATService instance attached to a host -func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) (*AutoNATService, error) { +func NewAutoNATService(ctx context.Context, h host.Host, forceEnabled bool, opts ...libp2p.Option) (*AutoNATService, error) { opts = append(opts, libp2p.NoListenAddrs) dialer, err := libp2p.New(ctx, opts...) if err != nil { @@ -60,12 +60,15 @@ func NewAutoNATService(ctx context.Context, h host.Host, opts ...libp2p.Option) go func() { defer s.Close() - select { - case <-s.Out(): - h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) - go as.resetRateLimiter() - case <-ctx.Done(): + if !forceEnabled { + select { + case <-ctx.Done(): + return + case <-s.Out(): + } } + h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) + go as.resetRateLimiter() }() return as, nil diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index eec48f07c2..b666e7ef03 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -7,7 +7,6 @@ import ( "time" "github.com/libp2p/go-libp2p" - "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/peer" @@ -21,12 +20,10 @@ func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATS t.Fatal(err) } - as, err := NewAutoNATService(ctx, h) + as, err := NewAutoNATService(ctx, h, true) if err != nil { t.Fatal(err) } - emitPublic, _ := h.EventBus().Emitter(new(event.EvtLocalRoutabilityPublic)) - emitPublic.Emit(event.EvtLocalRoutabilityPublic{}) return h, as } From 3a21c6dea31ead2c24df6a21e9867e8cc387d06e Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 26 Feb 2020 17:29:10 -0800 Subject: [PATCH 1660/3965] skip locally held addresses fix #44 --- p2p/host/autonat/svc.go | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 8c1191e3c3..6835e4d486 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -32,6 +32,7 @@ var ( // AutoNATService provides NAT autodetection services to other peers type AutoNATService struct { ctx context.Context + h host.Host dialer host.Host // rate limiter @@ -49,6 +50,7 @@ func NewAutoNATService(ctx context.Context, h host.Host, forceEnabled bool, opts as := &AutoNATService{ ctx: ctx, + h: h, dialer: dialer, reqs: make(map[peer.ID]int), } @@ -178,6 +180,13 @@ func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { return true } + // Skip dialing addresses we believe are the local node's + for _, localAddr := range as.h.Addrs() { + if localAddr.Equal(addr) { + return true + } + } + return false } From f5386bdf7952a8aee6500ebced8b9c4413976455 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 26 Feb 2020 19:38:50 -0800 Subject: [PATCH 1661/3965] autorelay triggers on NAT events better use of the event bus between these two --- p2p/host/relay/autorelay.go | 44 +++++++++++++++----------------- p2p/host/relay/autorelay_test.go | 9 +++++-- 2 files changed, 27 insertions(+), 26 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 0c3f40b53f..419d54c38c 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -7,6 +7,7 @@ import ( "sync" "time" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/routing" @@ -84,33 +85,28 @@ func (ar *AutoRelay) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { } func (ar *AutoRelay) background(ctx context.Context) { - select { - case <-time.After(autonat.AutoNATBootDelay + BootDelay): - case <-ctx.Done(): - return - } + subNatPublic, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityPublic)) + pubChan := subNatPublic.Out() + subNatPrivate, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityPrivate)) + priChan := subNatPrivate.Out() + subNatUnknown, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityUnknown)) + unkChan := subNatUnknown.Out() // when true, we need to identify push push := false for { - wait := autonat.AutoNATRefreshInterval - switch ar.autonat.Status() { - case autonat.NATStatusUnknown: - ar.mx.Lock() - ar.status = autonat.NATStatusUnknown - ar.mx.Unlock() - wait = autonat.AutoNATRetryInterval - - case autonat.NATStatusPublic: + select { + case <-pubChan: ar.mx.Lock() if ar.status != autonat.NATStatusPublic { push = true } ar.status = autonat.NATStatusPublic ar.mx.Unlock() - - case autonat.NATStatusPrivate: + case <-priChan: + // TODO: this is a long-lived (2.5min task) that should get spun up in a separate thread + // and canceled if the relay learns the nat is now public. update := ar.findRelays(ctx) ar.mx.Lock() if update || ar.status != autonat.NATStatusPrivate { @@ -118,6 +114,14 @@ func (ar *AutoRelay) background(ctx context.Context) { } ar.status = autonat.NATStatusPrivate ar.mx.Unlock() + case <-unkChan: + ar.mx.Lock() + ar.status = autonat.NATStatusUnknown + ar.mx.Unlock() + case <-ar.disconnect: + push = true + case <-ctx.Done(): + return } if push { @@ -127,14 +131,6 @@ func (ar *AutoRelay) background(ctx context.Context) { push = false ar.host.PushIdentify() } - - select { - case <-ar.disconnect: - push = true - case <-time.After(wait): - case <-ctx.Done(): - return - } } } diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index 25653d8c1a..dac45dbc97 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -16,6 +16,7 @@ import ( autonat "github.com/libp2p/go-libp2p-autonat" autonatpb "github.com/libp2p/go-libp2p-autonat/pb" circuit "github.com/libp2p/go-libp2p-circuit" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" @@ -181,9 +182,13 @@ func TestAutoRelay(t *testing.T) { } } - // connect to AutoNAT and let detection/discovery work its magic + // connect to AutoNAT, have it resolve to private. connect(t, h1, h3) - time.Sleep(5 * time.Second) + time.Sleep(300 * time.Millisecond) + privEmitter, _ := h3.EventBus().Emitter(new(event.EvtLocalRoutabilityPrivate)) + privEmitter.Emit(event.EvtLocalRoutabilityPrivate{}) + // Wait for detection to do its magic + time.Sleep(3000 * time.Millisecond) // verify that we now advertise relay addrs (but not unspecific relay addrs) unspecificRelay, err := ma.NewMultiaddr("/p2p-circuit") From 85a83edf8055cba24bd94a2010553529ccff63a1 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 27 Feb 2020 08:18:32 -0800 Subject: [PATCH 1662/3965] defer subscription closes --- p2p/host/relay/autorelay.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 419d54c38c..a0b9291ccd 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -86,25 +86,25 @@ func (ar *AutoRelay) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { func (ar *AutoRelay) background(ctx context.Context) { subNatPublic, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityPublic)) - pubChan := subNatPublic.Out() + defer subNatPublic.Close() subNatPrivate, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityPrivate)) - priChan := subNatPrivate.Out() + defer subNatPrivate.Close() subNatUnknown, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityUnknown)) - unkChan := subNatUnknown.Out() + defer subNatUnknown.Close() // when true, we need to identify push push := false for { select { - case <-pubChan: + case <-subNatPublic.Out(): ar.mx.Lock() if ar.status != autonat.NATStatusPublic { push = true } ar.status = autonat.NATStatusPublic ar.mx.Unlock() - case <-priChan: + case <-subNatPrivate.Out(): // TODO: this is a long-lived (2.5min task) that should get spun up in a separate thread // and canceled if the relay learns the nat is now public. update := ar.findRelays(ctx) @@ -114,7 +114,7 @@ func (ar *AutoRelay) background(ctx context.Context) { } ar.status = autonat.NATStatusPrivate ar.mx.Unlock() - case <-unkChan: + case <-subNatUnknown.Out(): ar.mx.Lock() ar.status = autonat.NATStatusUnknown ar.mx.Unlock() From 78ca1bf57c421b79ce278075c7a311f7775f72b8 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Fri, 28 Feb 2020 00:40:52 -0500 Subject: [PATCH 1663/3965] fix: can enable autorelay without content routing as long as there are static relays configured --- config/config.go | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/config/config.go b/config/config.go index 2ace430a49..65ed52a3ba 100644 --- a/config/config.go +++ b/config/config.go @@ -207,19 +207,6 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return nil, fmt.Errorf("cannot enable autorelay; relay is not enabled") } - if router == nil { - h.Close() - return nil, fmt.Errorf("cannot enable autorelay; no routing for discovery") - } - - crouter, ok := router.(routing.ContentRouting) - if !ok { - h.Close() - return nil, fmt.Errorf("cannot enable autorelay; no suitable routing for discovery") - } - - discovery := discovery.NewRoutingDiscovery(crouter) - hop := false for _, opt := range cfg.RelayOpts { if opt == circuit.OptHop { @@ -228,11 +215,28 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } } - if hop { - // advertise ourselves - relay.Advertise(ctx, discovery) + if !hop && len(cfg.StaticRelays) > 0 { + _ = relay.NewAutoRelay(swrm.Context(), h, nil, router, cfg.StaticRelays) } else { - _ = relay.NewAutoRelay(swrm.Context(), h, discovery, router, cfg.StaticRelays) + if router == nil { + h.Close() + return nil, fmt.Errorf("cannot enable autorelay; no routing for discovery") + } + + crouter, ok := router.(routing.ContentRouting) + if !ok { + h.Close() + return nil, fmt.Errorf("cannot enable autorelay; no suitable routing for discovery") + } + + discovery := discovery.NewRoutingDiscovery(crouter) + + if hop { + // advertise ourselves + relay.Advertise(ctx, discovery) + } else { + _ = relay.NewAutoRelay(swrm.Context(), h, discovery, router, cfg.StaticRelays) + } } } From e4de2063d7dddbb7c80f60cce307c6432698859b Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 13 Sep 2019 19:14:17 +0530 Subject: [PATCH 1664/3965] enable network to emit events on a bus --- core/network/events.go | 15 +++++++++++++++ core/network/network.go | 6 ++++++ 2 files changed, 21 insertions(+) create mode 100644 core/network/events.go diff --git a/core/network/events.go b/core/network/events.go new file mode 100644 index 0000000000..cbcc1a0fc1 --- /dev/null +++ b/core/network/events.go @@ -0,0 +1,15 @@ +package network + +// EvtPeerConnectionStateChange should be emitted when we connect/disconnect from a peer +type EvtPeerConnectionStateChange struct { + Network Network + Connection Conn + NewState Connectedness +} + +// EvtStreamStateChange is emitted when we open/close a stream with a peer +type EvtStreamStateChange struct { + Network Network + Stream Stream + NewState Connectedness +} diff --git a/core/network/network.go b/core/network/network.go index 467109c48c..f8e458d390 100644 --- a/core/network/network.go +++ b/core/network/network.go @@ -10,6 +10,7 @@ import ( "io" "github.com/jbenet/goprocess" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" @@ -100,6 +101,11 @@ type Network interface { // Process returns the network's Process Process() goprocess.Process + + // EventBus returns the network's Event Bus + // we can subscribe to this bus to listen for connection/disconnection of peers, + // opening/closing of streams etc etc + EventBus() event.Bus } // Dialer represents a service that can dial out to peers From 3e40e557bc7ac72e78bd812bd87f2535b307e5ac Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 28 Feb 2020 16:06:03 +0530 Subject: [PATCH 1665/3965] changes as per review --- core/event/network.go | 21 +++++++++++++++++++++ core/network/events.go | 15 --------------- core/network/network.go | 6 ------ 3 files changed, 21 insertions(+), 21 deletions(-) create mode 100644 core/event/network.go delete mode 100644 core/network/events.go diff --git a/core/event/network.go b/core/event/network.go new file mode 100644 index 0000000000..e7428e3c1f --- /dev/null +++ b/core/event/network.go @@ -0,0 +1,21 @@ +package event + +import "github.com/libp2p/go-libp2p-core/network" + +// EvtPeerConnectionStateChange should be emitted when we connect to a peer or disconnect +// from a peer. It contains the network interface for the connection, +// the connection handle & the new state of the connection. +type EvtPeerConnectionStateChange struct { + Network network.Network + Connection network.Conn + NewState network.Connectedness +} + +// EvtStreamStateChange should be emitted when we open a new stream with a peer or close an existing stream. +// It contains the network interface for the connection, the stream handle & +// the new state of the stream. +type EvtStreamStateChange struct { + Network network.Network + Stream network.Stream + NewState network.Connectedness +} diff --git a/core/network/events.go b/core/network/events.go deleted file mode 100644 index cbcc1a0fc1..0000000000 --- a/core/network/events.go +++ /dev/null @@ -1,15 +0,0 @@ -package network - -// EvtPeerConnectionStateChange should be emitted when we connect/disconnect from a peer -type EvtPeerConnectionStateChange struct { - Network Network - Connection Conn - NewState Connectedness -} - -// EvtStreamStateChange is emitted when we open/close a stream with a peer -type EvtStreamStateChange struct { - Network Network - Stream Stream - NewState Connectedness -} diff --git a/core/network/network.go b/core/network/network.go index f8e458d390..467109c48c 100644 --- a/core/network/network.go +++ b/core/network/network.go @@ -10,7 +10,6 @@ import ( "io" "github.com/jbenet/goprocess" - "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" @@ -101,11 +100,6 @@ type Network interface { // Process returns the network's Process Process() goprocess.Process - - // EventBus returns the network's Event Bus - // we can subscribe to this bus to listen for connection/disconnection of peers, - // opening/closing of streams etc etc - EventBus() event.Bus } // Dialer represents a service that can dial out to peers From 70837b0706a184bd8ca61f6b4e316ec6f06d239e Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 28 Feb 2020 16:18:42 +0530 Subject: [PATCH 1666/3965] remove network from the events --- core/event/network.go | 2 -- 1 file changed, 2 deletions(-) diff --git a/core/event/network.go b/core/event/network.go index e7428e3c1f..8fcc1eacec 100644 --- a/core/event/network.go +++ b/core/event/network.go @@ -6,7 +6,6 @@ import "github.com/libp2p/go-libp2p-core/network" // from a peer. It contains the network interface for the connection, // the connection handle & the new state of the connection. type EvtPeerConnectionStateChange struct { - Network network.Network Connection network.Conn NewState network.Connectedness } @@ -15,7 +14,6 @@ type EvtPeerConnectionStateChange struct { // It contains the network interface for the connection, the stream handle & // the new state of the stream. type EvtStreamStateChange struct { - Network network.Network Stream network.Stream NewState network.Connectedness } From 5e30cdeaacdfdb1394048e6aa9668a4bfe78cb36 Mon Sep 17 00:00:00 2001 From: Hlib Date: Fri, 28 Feb 2020 17:21:51 +0200 Subject: [PATCH 1667/3965] use of mux.ErrReset --- p2p/net/mock/mock_stream.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index 2bf8929686..d71e53c115 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -8,6 +8,7 @@ import ( "sync/atomic" "time" + "github.com/libp2p/go-libp2p-core/mux" "github.com/libp2p/go-libp2p-core/network" protocol "github.com/libp2p/go-libp2p-core/protocol" ) @@ -29,7 +30,6 @@ type stream struct { stat network.Stat } -var ErrReset error = errors.New("stream reset") var ErrClosed error = errors.New("stream closed") type transportObject struct { @@ -98,8 +98,8 @@ func (s *stream) Close() error { func (s *stream) Reset() error { // Cancel any pending reads/writes with an error. - s.write.CloseWithError(ErrReset) - s.read.CloseWithError(ErrReset) + s.write.CloseWithError(mux.ErrReset) + s.read.CloseWithError(mux.ErrReset) select { case s.reset <- struct{}{}: @@ -206,7 +206,7 @@ func (s *stream) transport() { case s.reset <- struct{}{}: default: } - return ErrReset + return mux.ErrReset } if err := drainBuf(); err != nil { return err @@ -226,14 +226,14 @@ func (s *stream) transport() { // Reset takes precedent. select { case <-s.reset: - s.writeErr = ErrReset + s.writeErr = mux.ErrReset return default: } select { case <-s.reset: - s.writeErr = ErrReset + s.writeErr = mux.ErrReset return case <-s.close: if err := drainBuf(); err != nil { From 2a0c987b076b9728765161b9acccbdcbc570b4cc Mon Sep 17 00:00:00 2001 From: Hlib Date: Fri, 28 Feb 2020 17:55:09 +0200 Subject: [PATCH 1668/3965] restructure files layout --- p2p/muxer/yamux/conn.go | 41 ++++++++++ p2p/muxer/yamux/mux.go | 48 ++++++++++++ .../yamux/{yamux_test.go => mux_test.go} | 2 +- p2p/muxer/yamux/yamux.go | 77 ------------------- 4 files changed, 90 insertions(+), 78 deletions(-) create mode 100644 p2p/muxer/yamux/conn.go create mode 100644 p2p/muxer/yamux/mux.go rename p2p/muxer/yamux/{yamux_test.go => mux_test.go} (75%) delete mode 100644 p2p/muxer/yamux/yamux.go diff --git a/p2p/muxer/yamux/conn.go b/p2p/muxer/yamux/conn.go new file mode 100644 index 0000000000..6c39cc2035 --- /dev/null +++ b/p2p/muxer/yamux/conn.go @@ -0,0 +1,41 @@ +package sm_yamux + +import ( + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-yamux" +) + +// conn implements mux.MuxedConn over yamux.Session. +type conn yamux.Session + +// Close closes underlying yamux +func (c *conn) Close() error { + return c.yamux().Close() +} + +// IsClosed checks if yamux.Session is in closed state. +func (c *conn) IsClosed() bool { + return c.yamux().IsClosed() +} + +// OpenStream creates a new stream. +func (c *conn) OpenStream() (mux.MuxedStream, error) { + s, err := c.yamux().OpenStream() + if err != nil { + return nil, err + } + + return (*stream)(s), nil +} + +// AcceptStream accepts a stream opened by the other side. +func (c *conn) AcceptStream() (mux.MuxedStream, error) { + s, err := c.yamux().AcceptStream() + return (*stream)(s), err +} + +func (c *conn) yamux() *yamux.Session { + return (*yamux.Session)(c) +} + +var _ mux.MuxedConn = &conn{} diff --git a/p2p/muxer/yamux/mux.go b/p2p/muxer/yamux/mux.go new file mode 100644 index 0000000000..f0c1af2995 --- /dev/null +++ b/p2p/muxer/yamux/mux.go @@ -0,0 +1,48 @@ +package sm_yamux + +import ( + "io/ioutil" + "net" + + mux "github.com/libp2p/go-libp2p-core/mux" + yamux "github.com/libp2p/go-yamux" +) + +var DefaultTransport *Multiplexer + +func init() { + config := yamux.DefaultConfig() + // We've bumped this to 16MiB as this critically limits throughput. + // + // 1MiB means a best case of 10MiB/s (83.89Mbps) on a connection with + // 100ms latency. The default gave us 2.4MiB *best case* which was + // totally unacceptable. + config.MaxStreamWindowSize = uint32(16 * 1024 * 1024) + // don't spam + config.LogOutput = ioutil.Discard + // We always run over a security transport that buffers internally + // (i.e., uses a block cipher). + config.ReadBufSize = 0 + DefaultTransport = (*Multiplexer)(config) +} + +// Multiplexer implements mux.Multiplexer that constructs +// yamux-backed muxed connections. +type Multiplexer yamux.Config + +func (t *Multiplexer) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { + var s *yamux.Session + var err error + if isServer { + s, err = yamux.Server(nc, t.Config()) + } else { + s, err = yamux.Client(nc, t.Config()) + } + return (*conn)(s), err +} + +func (t *Multiplexer) Config() *yamux.Config { + return (*yamux.Config)(t) +} + +var _ mux.Multiplexer = &Multiplexer{} diff --git a/p2p/muxer/yamux/yamux_test.go b/p2p/muxer/yamux/mux_test.go similarity index 75% rename from p2p/muxer/yamux/yamux_test.go rename to p2p/muxer/yamux/mux_test.go index 409b0967c8..d3a9b4ff79 100644 --- a/p2p/muxer/yamux/yamux_test.go +++ b/p2p/muxer/yamux/mux_test.go @@ -6,6 +6,6 @@ import ( tmux "github.com/libp2p/go-libp2p-testing/suites/mux" ) -func TestYamuxTransport(t *testing.T) { +func TestDefaultMultiplexer(t *testing.T) { tmux.SubtestAll(t, DefaultTransport) } diff --git a/p2p/muxer/yamux/yamux.go b/p2p/muxer/yamux/yamux.go deleted file mode 100644 index 4875761cf4..0000000000 --- a/p2p/muxer/yamux/yamux.go +++ /dev/null @@ -1,77 +0,0 @@ -package sm_yamux - -import ( - "io/ioutil" - "net" - - mux "github.com/libp2p/go-libp2p-core/mux" - yamux "github.com/libp2p/go-yamux" -) - -// Conn is a connection to a remote peer. -type conn yamux.Session - -func (c *conn) yamuxSession() *yamux.Session { - return (*yamux.Session)(c) -} - -func (c *conn) Close() error { - return c.yamuxSession().Close() -} - -func (c *conn) IsClosed() bool { - return c.yamuxSession().IsClosed() -} - -// OpenStream creates a new stream. -func (c *conn) OpenStream() (mux.MuxedStream, error) { - s, err := c.yamuxSession().OpenStream() - if err != nil { - return nil, err - } - - return s, nil -} - -// AcceptStream accepts a stream opened by the other side. -func (c *conn) AcceptStream() (mux.MuxedStream, error) { - s, err := c.yamuxSession().AcceptStream() - return s, err -} - -// Transport is a go-peerstream transport that constructs -// yamux-backed connections. -type Transport yamux.Config - -var DefaultTransport *Transport - -func init() { - config := yamux.DefaultConfig() - // We've bumped this to 16MiB as this critically limits throughput. - // - // 1MiB means a best case of 10MiB/s (83.89Mbps) on a connection with - // 100ms latency. The default gave us 2.4MiB *best case* which was - // totally unacceptable. - config.MaxStreamWindowSize = uint32(16 * 1024 * 1024) - // don't spam - config.LogOutput = ioutil.Discard - // We always run over a security transport that buffers internally - // (i.e., uses a block cipher). - config.ReadBufSize = 0 - DefaultTransport = (*Transport)(config) -} - -func (t *Transport) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { - var s *yamux.Session - var err error - if isServer { - s, err = yamux.Server(nc, t.Config()) - } else { - s, err = yamux.Client(nc, t.Config()) - } - return (*conn)(s), err -} - -func (t *Transport) Config() *yamux.Config { - return (*yamux.Config)(t) -} From df4c879cb3372f80bae83ab0feed371f62aa4007 Mon Sep 17 00:00:00 2001 From: Hlib Date: Fri, 28 Feb 2020 17:56:44 +0200 Subject: [PATCH 1669/3965] redefine yamux stream to implement mux.MuxedStream and to respect mux.ErrReset --- p2p/muxer/yamux/stream.go | 55 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 p2p/muxer/yamux/stream.go diff --git a/p2p/muxer/yamux/stream.go b/p2p/muxer/yamux/stream.go new file mode 100644 index 0000000000..866b781fbe --- /dev/null +++ b/p2p/muxer/yamux/stream.go @@ -0,0 +1,55 @@ +package sm_yamux + +import ( + "time" + + "github.com/libp2p/go-libp2p-core/mux" + "github.com/libp2p/go-yamux" +) + +// stream implements mux.MuxedStream over yamux.Stream. +type stream yamux.Stream + +func (s *stream) Read(b []byte) (n int, err error) { + n, err = s.yamux().Read(b) + if err == yamux.ErrConnectionReset { + err = mux.ErrReset + } + + return +} + +func (s *stream) Write(b []byte) (n int, err error) { + n, err = s.yamux().Write(b) + if err == yamux.ErrConnectionReset { + err = mux.ErrReset + } + + return +} + +func (s *stream) Close() error { + return s.yamux().Close() +} + +func (s *stream) Reset() error { + return s.yamux().Reset() +} + +func (s *stream) SetDeadline(t time.Time) error { + return s.yamux().SetDeadline(t) +} + +func (s *stream) SetReadDeadline(t time.Time) error { + return s.yamux().SetReadDeadline(t) +} + +func (s *stream) SetWriteDeadline(t time.Time) error { + return s.yamux().SetWriteDeadline(t) +} + +func (s *stream) yamux() *yamux.Stream { + return (*yamux.Stream)(s) +} + +var _ mux.MuxedStream = &stream{} From 2f38053ad59346e34563a80673fe1d5517c0d846 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Fri, 28 Feb 2020 10:01:45 -0500 Subject: [PATCH 1670/3965] remove noise pipes & associated code --- p2p/security/noise/crypto.go | 53 +-- p2p/security/noise/ik/IK.noise.go | 533 ------------------------- p2p/security/noise/ik/IK_test.go | 108 ----- p2p/security/noise/ik_handshake.go | 182 --------- p2p/security/noise/integration_test.go | 173 +------- p2p/security/noise/keycache.go | 30 -- p2p/security/noise/options.go | 44 -- p2p/security/noise/protocol.go | 74 +--- p2p/security/noise/transport.go | 85 +--- p2p/security/noise/transport_test.go | 77 ---- p2p/security/noise/xx/XX.noise.go | 13 +- p2p/security/noise/xx/XX_test.go | 6 +- p2p/security/noise/xx_handshake.go | 93 ++--- 13 files changed, 78 insertions(+), 1393 deletions(-) delete mode 100644 p2p/security/noise/ik/IK.noise.go delete mode 100644 p2p/security/noise/ik/IK_test.go delete mode 100644 p2p/security/noise/ik_handshake.go delete mode 100644 p2p/security/noise/keycache.go delete mode 100644 p2p/security/noise/options.go diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index 9d53d4e9bc..71230ae43d 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -2,56 +2,39 @@ package noise import ( "errors" - ik "github.com/libp2p/go-libp2p-noise/ik" xx "github.com/libp2p/go-libp2p-noise/xx" ) func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) { - if s.xx_complete { - if s.initiator { - cs := s.xx_ns.CS1() - _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) - } else { - cs := s.xx_ns.CS2() - _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) - } - } else if s.ik_complete { - if s.initiator { - cs := s.ik_ns.CS1() - _, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) - } else { - cs := s.ik_ns.CS2() - _, ciphertext = ik.EncryptWithAd(cs, nil, plaintext) - } - } else { + if !s.xx_complete { return nil, errors.New("encrypt err: haven't completed handshake") } + if s.initiator { + cs := s.ns.CS1() + _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) + } else { + cs := s.ns.CS2() + _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) + } + return ciphertext, nil } func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) { var ok bool - if s.xx_complete { - if s.initiator { - cs := s.xx_ns.CS2() - _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) - } else { - cs := s.xx_ns.CS1() - _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) - } - } else if s.ik_complete { - if s.initiator { - cs := s.ik_ns.CS2() - _, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) - } else { - cs := s.ik_ns.CS1() - _, plaintext, ok = ik.DecryptWithAd(cs, nil, ciphertext) - } - } else { + if !s.xx_complete { return nil, errors.New("decrypt err: haven't completed handshake") } + if s.initiator { + cs := s.ns.CS2() + _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) + } else { + cs := s.ns.CS1() + _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) + } + if !ok { return nil, errors.New("decrypt err: could not decrypt") } diff --git a/p2p/security/noise/ik/IK.noise.go b/p2p/security/noise/ik/IK.noise.go deleted file mode 100644 index 724a98e572..0000000000 --- a/p2p/security/noise/ik/IK.noise.go +++ /dev/null @@ -1,533 +0,0 @@ -/* -IK: - <- s - ... - -> e, es, s, ss - <- e, ee, se -*/ - -// Implementation Version: 1.0.0 - -/* ---------------------------------------------------------------- * - * PARAMETERS * - * ---------------------------------------------------------------- */ - -package ik - -import ( - "crypto/rand" - "crypto/subtle" - "encoding/binary" - "errors" - "golang.org/x/crypto/blake2s" - "golang.org/x/crypto/chacha20poly1305" - "golang.org/x/crypto/curve25519" - "golang.org/x/crypto/hkdf" - "hash" - "io" - "math" -) - -/* ---------------------------------------------------------------- * - * TYPES * - * ---------------------------------------------------------------- */ - -type Keypair struct { - public_key [32]byte - private_key [32]byte -} - -type MessageBuffer struct { - ne [32]byte - ns []byte - ciphertext []byte -} - -type cipherstate struct { - k [32]byte - n uint32 -} - -type symmetricstate struct { - cs cipherstate - ck [32]byte - h [32]byte -} - -type handshakestate struct { - ss symmetricstate - s Keypair - e Keypair - rs [32]byte - re [32]byte - psk [32]byte -} - -type NoiseSession struct { - hs handshakestate - h [32]byte - cs1 cipherstate - cs2 cipherstate - mc uint64 - i bool -} - -func (ns *NoiseSession) CS1() *cipherstate { - return &ns.cs1 -} - -func (ns *NoiseSession) CS2() *cipherstate { - return &ns.cs2 -} - -func (ns *NoiseSession) Ephemeral() *Keypair { - return &ns.hs.e -} - -func NewKeypair(pub [32]byte, priv [32]byte) Keypair { - return Keypair{ - public_key: pub, - private_key: priv, - } -} - -func (kp Keypair) PubKey() [32]byte { - return kp.public_key -} - -func (kp Keypair) PrivKey() [32]byte { - return kp.private_key -} - -func (ns *NoiseSession) RemoteKey() [32]byte { - return ns.hs.rs -} - -func NewMessageBuffer(ne [32]byte, ns []byte, ciphertext []byte) MessageBuffer { - return MessageBuffer{ - ne: ne, - ns: ns, - ciphertext: ciphertext, - } -} - -/* ---------------------------------------------------------------- * - * CONSTANTS * - * ---------------------------------------------------------------- */ - -var emptyKey = [32]byte{ - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, -} - -var minNonce = uint32(0) - -/* ---------------------------------------------------------------- * - * UTILITY FUNCTIONS * - * ---------------------------------------------------------------- */ - -func getPublicKey(kp *Keypair) [32]byte { - return kp.public_key -} - -func isEmptyKey(k [32]byte) bool { - return subtle.ConstantTimeCompare(k[:], emptyKey[:]) == 1 -} - -func (mb *MessageBuffer) NE() [32]byte { - return mb.ne -} - -func (mb *MessageBuffer) NS() []byte { - return mb.ns -} - -func (mb *MessageBuffer) Ciphertext() []byte { - return mb.ciphertext -} - -// Encodes a MessageBuffer from stage 0 -func (mb *MessageBuffer) Encode0() []byte { - enc := []byte{} - - enc = append(enc, mb.ne[:]...) - enc = append(enc, mb.ns...) - enc = append(enc, mb.ciphertext...) - - return enc -} - -// Encodes a MessageBuffer from stage 1 -func (mb *MessageBuffer) Encode1() []byte { - enc := []byte{} - - enc = append(enc, mb.ne[:]...) - enc = append(enc, mb.ciphertext...) - - return enc -} - -// Decodes initial message (stage 0) into MessageBuffer -func Decode0(in []byte) (*MessageBuffer, error) { - if len(in) < 80 { - return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 80 bytes") - } - - mb := new(MessageBuffer) - copy(mb.ne[:], in[:32]) - mb.ns = in[32:80] - mb.ciphertext = in[80:] - - return mb, nil -} - -// Decodes messages at stage 1 into MessageBuffer -func Decode1(in []byte) (*MessageBuffer, error) { - if len(in) < 32 { - return nil, errors.New("cannot decode stage 1 MessageBuffer: length less than 32 bytes") - } - - mb := new(MessageBuffer) - copy(mb.ne[:], in[:32]) - mb.ciphertext = in[32:] - - return mb, nil -} - -func validatePublicKey(k []byte) bool { - forbiddenCurveValues := [12][]byte{ - {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, - {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, - {224, 235, 122, 124, 59, 65, 184, 174, 22, 86, 227, 250, 241, 159, 196, 106, 218, 9, 141, 235, 156, 50, 177, 253, 134, 98, 5, 22, 95, 73, 184, 0}, - {95, 156, 149, 188, 163, 80, 140, 36, 177, 208, 177, 85, 156, 131, 239, 91, 4, 68, 92, 196, 88, 28, 142, 134, 216, 34, 78, 221, 208, 159, 17, 87}, - {236, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, - {237, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, - {238, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, - {205, 235, 122, 124, 59, 65, 184, 174, 22, 86, 227, 250, 241, 159, 196, 106, 218, 9, 141, 235, 156, 50, 177, 253, 134, 98, 5, 22, 95, 73, 184, 128}, - {76, 156, 149, 188, 163, 80, 140, 36, 177, 208, 177, 85, 156, 131, 239, 91, 4, 68, 92, 196, 88, 28, 142, 134, 216, 34, 78, 221, 208, 159, 17, 215}, - {217, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}, - {218, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}, - {219, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 25}, - } - - for _, testValue := range forbiddenCurveValues { - if subtle.ConstantTimeCompare(k[:], testValue[:]) == 1 { - panic("Invalid public key") - } - } - return true -} - -/* ---------------------------------------------------------------- * - * PRIMITIVES * - * ---------------------------------------------------------------- */ - -func incrementNonce(n uint32) uint32 { - return n + 1 -} - -func dh(private_key [32]byte, public_key [32]byte) [32]byte { - var ss [32]byte - curve25519.ScalarMult(&ss, &private_key, &public_key) - return ss -} - -func GenerateKeypair() Keypair { - var public_key [32]byte - var private_key [32]byte - _, _ = rand.Read(private_key[:]) - curve25519.ScalarBaseMult(&public_key, &private_key) - if validatePublicKey(public_key[:]) { - return Keypair{public_key, private_key} - } - return GenerateKeypair() -} - -func GeneratePublicKey(private_key [32]byte) [32]byte { - var public_key [32]byte - curve25519.ScalarBaseMult(&public_key, &private_key) - return public_key -} - -func encrypt(k [32]byte, n uint32, ad []byte, plaintext []byte) []byte { - var nonce [12]byte - var ciphertext []byte - enc, _ := chacha20poly1305.New(k[:]) - binary.LittleEndian.PutUint32(nonce[4:], n) - ciphertext = enc.Seal(nil, nonce[:], plaintext, ad) - return ciphertext -} - -func decrypt(k [32]byte, n uint32, ad []byte, ciphertext []byte) (bool, []byte, []byte) { - var nonce [12]byte - var plaintext []byte - enc, err := chacha20poly1305.New(k[:]) - binary.LittleEndian.PutUint32(nonce[4:], n) - plaintext, err = enc.Open(nil, nonce[:], ciphertext, ad) - return (err == nil), ad, plaintext -} - -func getHash(a []byte, b []byte) [32]byte { - return blake2s.Sum256(append(a, b...)) -} - -func hashProtocolName(protocolName []byte) [32]byte { - var h [32]byte - if len(protocolName) <= 32 { - copy(h[:], protocolName) - } else { - h = getHash(protocolName, []byte{}) - } - return h -} - -func blake2HkdfInterface() hash.Hash { - h, _ := blake2s.New256([]byte{}) - return h -} - -func getHkdf(ck [32]byte, ikm []byte) ([32]byte, [32]byte, [32]byte) { - var k1 [32]byte - var k2 [32]byte - var k3 [32]byte - output := hkdf.New(blake2HkdfInterface, ikm[:], ck[:], []byte{}) - io.ReadFull(output, k1[:]) - io.ReadFull(output, k2[:]) - io.ReadFull(output, k3[:]) - return k1, k2, k3 -} - -/* ---------------------------------------------------------------- * - * STATE MANAGEMENT * - * ---------------------------------------------------------------- */ - -/* CipherState */ -func initializeKey(k [32]byte) cipherstate { - return cipherstate{k, minNonce} -} - -func hasKey(cs *cipherstate) bool { - return !isEmptyKey(cs.k) -} - -func setNonce(cs *cipherstate, newNonce uint32) *cipherstate { - cs.n = newNonce - return cs -} - -func EncryptWithAd(cs *cipherstate, ad []byte, plaintext []byte) (*cipherstate, []byte) { - e := encrypt(cs.k, cs.n, ad, plaintext) - cs = setNonce(cs, incrementNonce(cs.n)) - return cs, e -} - -func DecryptWithAd(cs *cipherstate, ad []byte, ciphertext []byte) (*cipherstate, []byte, bool) { - valid, ad, plaintext := decrypt(cs.k, cs.n, ad, ciphertext) - cs = setNonce(cs, incrementNonce(cs.n)) - return cs, plaintext, valid -} - -func reKey(cs *cipherstate) *cipherstate { - e := encrypt(cs.k, math.MaxUint32, []byte{}, emptyKey[:]) - copy(cs.k[:], e) - return cs -} - -/* SymmetricState */ - -func initializeSymmetric(protocolName []byte) symmetricstate { - h := hashProtocolName(protocolName) - ck := h - cs := initializeKey(emptyKey) - return symmetricstate{cs, ck, h} -} - -func mixKey(ss *symmetricstate, ikm [32]byte) *symmetricstate { - ck, tempK, _ := getHkdf(ss.ck, ikm[:]) - ss.cs = initializeKey(tempK) - ss.ck = ck - return ss -} - -func mixHash(ss *symmetricstate, data []byte) *symmetricstate { - ss.h = getHash(ss.h[:], data) - return ss -} - -func mixKeyAndHash(ss *symmetricstate, ikm [32]byte) *symmetricstate { - var tempH [32]byte - var tempK [32]byte - ss.ck, tempH, tempK = getHkdf(ss.ck, ikm[:]) - ss = mixHash(ss, tempH[:]) - ss.cs = initializeKey(tempK) - return ss -} - -func getHandshakeHash(ss *symmetricstate) [32]byte { - return ss.h -} - -func encryptAndHash(ss *symmetricstate, plaintext []byte) (*symmetricstate, []byte) { - var ciphertext []byte - if hasKey(&ss.cs) { - _, ciphertext = EncryptWithAd(&ss.cs, ss.h[:], plaintext) - } else { - ciphertext = plaintext - } - ss = mixHash(ss, ciphertext) - return ss, ciphertext -} - -func decryptAndHash(ss *symmetricstate, ciphertext []byte) (*symmetricstate, []byte, bool) { - var plaintext []byte - var valid bool - if hasKey(&ss.cs) { - _, plaintext, valid = DecryptWithAd(&ss.cs, ss.h[:], ciphertext) - } else { - plaintext, valid = ciphertext, true - } - ss = mixHash(ss, ciphertext) - return ss, plaintext, valid -} - -func split(ss *symmetricstate) (cipherstate, cipherstate) { - tempK1, tempK2, _ := getHkdf(ss.ck, []byte{}) - cs1 := initializeKey(tempK1) - cs2 := initializeKey(tempK2) - return cs1, cs2 -} - -/* HandshakeState */ - -func initializeInitiator(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) handshakestate { - var ss symmetricstate - var e Keypair - var re [32]byte - name := []byte("Noise_IK_25519_ChaChaPoly_SHA256") - ss = initializeSymmetric(name) - mixHash(&ss, prologue) - mixHash(&ss, rs[:]) - return handshakestate{ss, s, e, rs, re, psk} -} - -func initializeResponder(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) handshakestate { - var ss symmetricstate - var e Keypair - var re [32]byte - name := []byte("Noise_IK_25519_ChaChaPoly_SHA256") - ss = initializeSymmetric(name) - mixHash(&ss, prologue) - mixHash(&ss, s.public_key[:]) - return handshakestate{ss, s, e, rs, re, psk} -} - -func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { - ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - hs.e = GenerateKeypair() - ne = hs.e.public_key - mixHash(&hs.ss, ne[:]) - /* No PSK, so skipping mixKey */ - mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) - spk := make([]byte, len(hs.s.public_key)) - copy(spk[:], hs.s.public_key[:]) - _, ns = encryptAndHash(&hs.ss, spk) - mixKey(&hs.ss, dh(hs.s.private_key, hs.rs)) - _, ciphertext = encryptAndHash(&hs.ss, payload) - messageBuffer := MessageBuffer{ne, ns, ciphertext} - return hs, messageBuffer -} - -func writeMessageB(hs *handshakestate, payload []byte) ([32]byte, MessageBuffer, cipherstate, cipherstate) { - ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - hs.e = GenerateKeypair() - ne = hs.e.public_key - mixHash(&hs.ss, ne[:]) - /* No PSK, so skipping mixKey */ - mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) - mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) - _, ciphertext = encryptAndHash(&hs.ss, payload) - messageBuffer := MessageBuffer{ne, ns, ciphertext} - cs1, cs2 := split(&hs.ss) - return hs.ss.h, messageBuffer, cs1, cs2 -} - -func readMessageA(hs *handshakestate, message *MessageBuffer) (*handshakestate, []byte, bool) { - valid1 := true - if validatePublicKey(message.ne[:]) { - hs.re = message.ne - } - mixHash(&hs.ss, hs.re[:]) - /* No PSK, so skipping mixKey */ - mixKey(&hs.ss, dh(hs.s.private_key, hs.re)) - _, ns, valid1 := decryptAndHash(&hs.ss, message.ns) - if valid1 && len(ns) == 32 && validatePublicKey(message.ns[:]) { - copy(hs.rs[:], ns) - } - mixKey(&hs.ss, dh(hs.s.private_key, hs.rs)) - _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) - return hs, plaintext, (valid1 && valid2) -} - -func readMessageB(hs *handshakestate, message *MessageBuffer) ([32]byte, []byte, bool, cipherstate, cipherstate) { - valid1 := true - if validatePublicKey(message.ne[:]) { - hs.re = message.ne - } - mixHash(&hs.ss, hs.re[:]) - /* No PSK, so skipping mixKey */ - mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) - mixKey(&hs.ss, dh(hs.s.private_key, hs.re)) - _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) - cs1, cs2 := split(&hs.ss) - return hs.ss.h, plaintext, (valid1 && valid2), cs1, cs2 -} - -/* ---------------------------------------------------------------- * - * PROCESSES * - * ---------------------------------------------------------------- */ - -func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *NoiseSession { - var session NoiseSession - psk := emptyKey - if initiator { - session.hs = initializeInitiator(prologue, s, rs, psk) - } else { - session.hs = initializeResponder(prologue, s, rs, psk) - } - session.i = initiator - session.mc = 0 - return &session -} - -func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageBuffer) { - var messageBuffer MessageBuffer - if session.mc == 0 { - _, messageBuffer = writeMessageA(&session.hs, message) - } - if session.mc == 1 { - session.h, messageBuffer, session.cs1, session.cs2 = writeMessageB(&session.hs, message) - } - session.mc = session.mc + 1 - return session, messageBuffer -} - -func RecvMessage(session *NoiseSession, message *MessageBuffer) (*NoiseSession, []byte, bool) { - var plaintext []byte - var valid bool - if session.mc == 0 { - _, plaintext, valid = readMessageA(&session.hs, message) - } - if session.mc == 1 { - session.h, plaintext, valid, session.cs1, session.cs2 = readMessageB(&session.hs, message) - } - session.mc = session.mc + 1 - return session, plaintext, valid -} - -func main() {} diff --git a/p2p/security/noise/ik/IK_test.go b/p2p/security/noise/ik/IK_test.go deleted file mode 100644 index f657769a33..0000000000 --- a/p2p/security/noise/ik/IK_test.go +++ /dev/null @@ -1,108 +0,0 @@ -package ik - -import ( - "crypto/rand" - proto "github.com/gogo/protobuf/proto" - "github.com/libp2p/go-libp2p-core/crypto" - pb "github.com/libp2p/go-libp2p-noise/pb" - "testing" -) - -func TestHandshake(t *testing.T) { - // generate local static noise key - kp_init := GenerateKeypair() - kp_resp := GenerateKeypair() - - payload_string := []byte("noise-libp2p-static-key:") - prologue := []byte("/noise/0.0.0") - - // initiator setup - init_pub := kp_init.PubKey() - libp2p_priv_init, libp2p_pub_init, err := crypto.GenerateEd25519Key(rand.Reader) - if err != nil { - t.Fatal(err) - } - libp2p_pub_init_raw, err := libp2p_pub_init.Raw() - if err != nil { - t.Fatal(err) - } - libp2p_init_signed_payload, err := libp2p_priv_init.Sign(append(payload_string, init_pub[:]...)) - if err != nil { - t.Fatal(err) - } - - // respoonder setup - resp_pub := kp_resp.PubKey() - libp2p_priv_resp, libp2p_pub_resp, err := crypto.GenerateEd25519Key(rand.Reader) - if err != nil { - t.Fatal(err) - } - libp2p_pub_resp_raw, err := libp2p_pub_resp.Raw() - if err != nil { - t.Fatal(err) - } - libp2p_resp_signed_payload, err := libp2p_priv_resp.Sign(append(payload_string, resp_pub[:]...)) - if err != nil { - t.Fatal(err) - } - - // initiator: new IK noise session - ns_init := InitSession(true, prologue, kp_init, kp_resp.PubKey()) - - // responder: new IK noise session - ns_resp := InitSession(false, prologue, kp_resp, [32]byte{}) - - // stage 0: initiator - // create payload - payload_init := new(pb.NoiseHandshakePayload) - payload_init.IdentityKey = libp2p_pub_init_raw - payload_init.IdentitySig = libp2p_init_signed_payload - payload_init_enc, err := proto.Marshal(payload_init) - if err != nil { - t.Fatalf("proto marshal payload fail: %s", err) - } - - // send message - var msgbuf MessageBuffer - msg := []byte{} - msg = append(msg, payload_init_enc[:]...) - ns_init, msgbuf = SendMessage(ns_init, msg) - - t.Logf("stage 0 msgbuf: %v", msgbuf) - t.Logf("stage 0 msgbuf ne len: %d", len(msgbuf.NE())) - - // stage 0: responder - var plaintext []byte - var valid bool - ns_resp, plaintext, valid = RecvMessage(ns_resp, &msgbuf) - if !valid { - t.Fatalf("stage 0 receive not valid") - } - - t.Logf("stage 0 resp payload: %x", plaintext) - - // stage 1: responder - // create payload - payload_resp := new(pb.NoiseHandshakePayload) - payload_resp.IdentityKey = libp2p_pub_resp_raw - payload_resp.IdentitySig = libp2p_resp_signed_payload - payload_resp_enc, err := proto.Marshal(payload_resp) - if err != nil { - t.Fatalf("proto marshal payload fail: %s", err) - } - msg = append(msg, payload_resp_enc[:]...) - ns_resp, msgbuf = SendMessage(ns_resp, msg) - - t.Logf("stage 1 msgbuf: %v", msgbuf) - t.Logf("stage 1 msgbuf ne len: %d", len(msgbuf.NE())) - t.Logf("stage 1 msgbuf ns len: %d", len(msgbuf.NS())) - - // stage 1: initiator - ns_init, plaintext, valid = RecvMessage(ns_init, &msgbuf) - if !valid { - t.Fatalf("stage 1 receive not valid") - } - - t.Logf("stage 1 resp payload: %x", plaintext) - -} diff --git a/p2p/security/noise/ik_handshake.go b/p2p/security/noise/ik_handshake.go deleted file mode 100644 index 2ee67979fe..0000000000 --- a/p2p/security/noise/ik_handshake.go +++ /dev/null @@ -1,182 +0,0 @@ -package noise - -import ( - "context" - "fmt" - "github.com/libp2p/go-libp2p-core/peer" - "io" - - proto "github.com/gogo/protobuf/proto" - ik "github.com/libp2p/go-libp2p-noise/ik" - pb "github.com/libp2p/go-libp2p-noise/pb" -) - -func (s *secureSession) ik_sendHandshakeMessage(payload []byte, initial_stage bool) error { - var msgbuf ik.MessageBuffer - s.ik_ns, msgbuf = ik.SendMessage(s.ik_ns, payload) - var encMsgBuf []byte - if initial_stage { - encMsgBuf = msgbuf.Encode0() - } else { - encMsgBuf = msgbuf.Encode1() - } - - err := s.writeLength(len(encMsgBuf)) - if err != nil { - return fmt.Errorf("ik_sendHandshakeMessage write length err=%s", err) - } - - // send message - _, err = s.insecure.Write(encMsgBuf) - if err != nil { - return fmt.Errorf("ik_sendHandshakeMessage write to conn err=%s", err) - } - - return nil -} - -func (s *secureSession) ik_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { - l, err := s.readLength() - if err != nil { - return nil, nil, false, fmt.Errorf("ik_recvHandshakeMessage read length err=%s", err) - } - - buf = make([]byte, l) - - _, err = io.ReadFull(s.insecure, buf) - if err != nil { - return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage read from conn err=%s", err) - } - - var msgbuf *ik.MessageBuffer - if initial_stage { - msgbuf, err = ik.Decode0(buf) - } else { - msgbuf, err = ik.Decode1(buf) - } - - if err != nil { - return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage decode msg fail: %s", err) - } - - s.ik_ns, plaintext, valid = ik.RecvMessage(s.ik_ns, msgbuf) - if !valid { - return buf, nil, false, fmt.Errorf("ik_recvHandshakeMessage validation fail") - } - - return buf, plaintext, valid, nil -} - -// IK: -// <- s -// ... -// -> e, es, s, ss -// <- e, ee, se -// returns last successful message upon error -func (s *secureSession) runHandshake_ik(ctx context.Context, payload []byte) ([]byte, error) { - kp := ik.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) - - remoteNoiseKey := s.noiseStaticKeyCache.Load(s.remotePeer) - - // new IK noise session - s.ik_ns = ik.InitSession(s.initiator, s.prologue, kp, remoteNoiseKey) - - if s.initiator { - // bail out early if we don't know the remote Noise key - if remoteNoiseKey == [32]byte{} { - return nil, fmt.Errorf("runHandshake_ik aborting - unknown static key for peer %s", s.remotePeer.Pretty()) - } - - // stage 0 // - err := s.ik_sendHandshakeMessage(payload, true) - if err != nil { - return nil, fmt.Errorf("runHandshake_ik stage=0 initiator=true err=%s", err) - } - - // stage 1 // - - // read message - buf, plaintext, valid, err := s.ik_recvHandshakeMessage(false) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=%s", err) - } - - if !valid { - return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=validation fail") - } - - // unmarshal payload - nhp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=validation fail: cannot unmarshal payload") - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetIdentityKey()) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true err=read remote libp2p key fail") - } - - // assert that remote peer ID matches libp2p public key - pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) - if pid != s.remotePeer { - return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true check remote peer id err: expected %x got %x", s.remotePeer, pid) - } else if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true check remote peer id err %s", err) - } - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, remoteNoiseKey) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=1 initiator=true verify payload err=%s", err) - } - - } else { - // stage 0 // - - // read message - buf, plaintext, valid, err := s.ik_recvHandshakeMessage(true) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err=%s", err) - } - - if !valid { - return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err: validation fail") - } - - // unmarshal payload - nhp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err=validation fail: cannot unmarshal payload") - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetIdentityKey()) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false err=read remote libp2p key fail") - } - - // assert that remote peer ID matches libp2p key - err = s.setRemotePeerID(s.RemotePublicKey()) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false set remote peer id err=%s:", err) - } - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ik_ns.RemoteKey()) - if err != nil { - return buf, fmt.Errorf("runHandshake_ik stage=0 initiator=false verify payload err=%s", err) - } - - // stage 1 // - - err = s.ik_sendHandshakeMessage(payload, false) - if err != nil { - return nil, fmt.Errorf("runHandshake_ik stage=1 initiator=false send err=%s", err) - } - - } - return nil, nil -} diff --git a/p2p/security/noise/integration_test.go b/p2p/security/noise/integration_test.go index 0cb2a93019..bd793b3264 100644 --- a/p2p/security/noise/integration_test.go +++ b/p2p/security/noise/integration_test.go @@ -35,13 +35,13 @@ func generateKey(seed int64) (crypto.PrivKey, error) { return priv, nil } -func makeNode(t *testing.T, seed int64, port int, kp *Keypair) (host.Host, error) { +func makeNode(t *testing.T, seed int64, port int) (host.Host, error) { priv, err := generateKey(seed) if err != nil { t.Fatal(err) } - tpt, err := New(priv, NoiseKeyPair(kp)) + tpt, err := New(priv) if err != nil { t.Fatal(err) } @@ -62,49 +62,17 @@ func makeNode(t *testing.T, seed int64, port int, kp *Keypair) (host.Host, error return libp2p.New(ctx, options...) } -func makeNodePipes(t *testing.T, seed int64, port int, rpid peer.ID, rpubkey [32]byte, kp *Keypair) (host.Host, error) { - priv, err := generateKey(seed) - if err != nil { - t.Fatal(err) - } - - tpt, err := New(priv, UseNoisePipes, NoiseKeyPair(kp)) - if err != nil { - t.Fatal(err) - } - - tpt.noiseStaticKeyCache = NewKeyCache() - tpt.noiseStaticKeyCache.Store(rpid, rpubkey) - - ip := "0.0.0.0" - addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", ip, port)) - if err != nil { - t.Fatal(err) - } - - options := []libp2p.Option{ - libp2p.Identity(priv), - libp2p.Security(ID, tpt), - libp2p.ListenAddrs(addr), - } - - ctx := context.Background() - - h, err := libp2p.New(ctx, options...) - return h, err -} - -func TestLibp2pIntegration_NoPipes(t *testing.T) { +func TestLibp2pIntegration(t *testing.T) { ctx := context.Background() - ha, err := makeNode(t, 1, 33333, nil) + ha, err := makeNode(t, 1, 33333) if err != nil { t.Fatal(err) } defer ha.Close() - hb, err := makeNode(t, 2, 34343, nil) + hb, err := makeNode(t, 2, 34343) if err != nil { t.Fatal(err) } @@ -146,137 +114,6 @@ func TestLibp2pIntegration_NoPipes(t *testing.T) { time.Sleep(time.Second) } -func TestLibp2pIntegration_WithPipes(t *testing.T) { - ctx := context.Background() - - kpa, err := GenerateKeypair() - if err != nil { - t.Fatal(err) - } - - ha, err := makeNodePipes(t, 1, 33333, "", [32]byte{}, kpa) - if err != nil { - t.Fatal(err) - } - - defer ha.Close() - - hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.publicKey, nil) - if err != nil { - t.Fatal(err) - } - - defer hb.Close() - - ha.SetStreamHandler(testProtocolID, streamHandler(t)) - hb.SetStreamHandler(testProtocolID, streamHandler(t)) - - addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", ha.Addrs()[0].String(), ha.ID())) - if err != nil { - t.Fatal(err) - } - - fmt.Printf("ha: %s\n", addr) - - addrInfo, err := peer.AddrInfoFromP2pAddr(addr) - if err != nil { - t.Fatal(err) - } - - err = hb.Connect(ctx, *addrInfo) - if err != nil { - t.Fatal(err) - } - - stream, err := hb.NewStream(ctx, ha.ID(), testProtocolID) - if err != nil { - t.Fatal(err) - } - - err = writeRandomPayloadAndClose(t, stream) - if err != nil { - t.Fatal(err) - } - - fmt.Println("fin") - - time.Sleep(time.Second) -} - -func TestLibp2pIntegration_XXFallback(t *testing.T) { - ctx := context.Background() - - kpa, err := GenerateKeypair() - if err != nil { - t.Fatal(err) - } - - ha, err := makeNode(t, 1, 33333, kpa) - if err != nil { - t.Fatal(err) - } - - defer ha.Close() - - hb, err := makeNodePipes(t, 2, 34343, ha.ID(), kpa.publicKey, nil) - if err != nil { - t.Fatal(err) - } - - defer hb.Close() - - ha.SetStreamHandler(testProtocolID, streamHandler(t)) - hb.SetStreamHandler(testProtocolID, streamHandler(t)) - - addr, err := ma.NewMultiaddr(fmt.Sprintf("%s/p2p/%s", hb.Addrs()[0].String(), hb.ID())) - if err != nil { - t.Fatal(err) - } - - fmt.Printf("ha: %s\n", addr) - - addrInfo, err := peer.AddrInfoFromP2pAddr(addr) - if err != nil { - t.Fatal(err) - } - - err = ha.Connect(ctx, *addrInfo) - if err != nil { - t.Fatal(err) - } - - stream, err := hb.NewStream(ctx, ha.ID(), testProtocolID) - if err != nil { - t.Fatal(err) - } - - err = writeRandomPayloadAndClose(t, stream) - if err != nil { - t.Fatal(err) - } - - fmt.Println("fin") - - time.Sleep(time.Second) -} - -func TestConstrucingWithMaker(t *testing.T) { - kp, err := GenerateKeypair() - if err != nil { - t.Fatal(err) - } - - ctx := context.Background() - h, err := libp2p.New(ctx, - libp2p.Security(ID, - Maker(NoiseKeyPair(kp), UseNoisePipes))) - - if err != nil { - t.Fatalf("unable to create libp2p host with Maker: %v", err) - } - _ = h.Close() -} - func writeRandomPayloadAndClose(t *testing.T, stream net.Stream) error { t.Helper() size := 1 << 24 diff --git a/p2p/security/noise/keycache.go b/p2p/security/noise/keycache.go deleted file mode 100644 index f43ce78a2a..0000000000 --- a/p2p/security/noise/keycache.go +++ /dev/null @@ -1,30 +0,0 @@ -package noise - -import ( - "github.com/libp2p/go-libp2p-core/peer" - "sync" -) - -type KeyCache struct { - lock sync.RWMutex - m map[peer.ID][32]byte -} - -func NewKeyCache() *KeyCache { - return &KeyCache{ - lock: sync.RWMutex{}, - m: make(map[peer.ID][32]byte), - } -} - -func (kc *KeyCache) Store(p peer.ID, key [32]byte) { - kc.lock.Lock() - kc.m[p] = key - kc.lock.Unlock() -} - -func (kc *KeyCache) Load(p peer.ID) [32]byte { - kc.lock.RLock() - defer kc.lock.RUnlock() - return kc.m[p] -} diff --git a/p2p/security/noise/options.go b/p2p/security/noise/options.go deleted file mode 100644 index 8ea20514e3..0000000000 --- a/p2p/security/noise/options.go +++ /dev/null @@ -1,44 +0,0 @@ -package noise - -// UseNoisePipes configures the Noise transport to use the Noise Pipes pattern. -// Noise Pipes attempts to use the more efficient IK handshake pattern when -// dialing a remote peer, if that peer's static Noise key is known. If this -// is unsuccessful, the transport will fallback to using the default XX pattern. -// -// Note that the fallback does not add any additional round-trips vs. simply -// using XX in the first place, however there is a slight processing overhead -// due to the initial decryption attempt of the IK message. -func UseNoisePipes(cfg *config) { - cfg.noisePipesSupport = true -} - -// NoiseKeyPair configures the Noise transport to use the given Noise static -// keypair. This is distinct from the libp2p Host's identity keypair and is -// used only for Noise. If this option is not provided, a new Noise static -// keypair will be generated when the transport is initialized. -// -// This option is most useful when Noise Pipes is enabled, as longer static -// key lifetimes may lead to more successful IK handshake attempts. -// -// If you do use this option with a key that's been saved to disk, you must -// take care to store the key securely! -func NoiseKeyPair(kp *Keypair) Option { - return func(cfg *config) { - cfg.noiseKeypair = kp - } -} - -type config struct { - noiseKeypair *Keypair - noisePipesSupport bool -} - -type Option func(cfg *config) - -func (cfg *config) applyOptions(opts ...Option) { - for _, opt := range opts { - if opt != nil { - opt(cfg) - } - } -} diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go index 4f5b77833b..421d3ff4e5 100644 --- a/p2p/security/noise/protocol.go +++ b/p2p/security/noise/protocol.go @@ -15,7 +15,6 @@ import ( "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" - "github.com/libp2p/go-libp2p-noise/ik" "github.com/libp2p/go-libp2p-noise/pb" "github.com/libp2p/go-libp2p-noise/xx" ) @@ -44,14 +43,9 @@ type secureSession struct { local peerInfo remote peerInfo - xx_ns *xx.NoiseSession - ik_ns *ik.NoiseSession + ns *xx.NoiseSession xx_complete bool - ik_complete bool - - noisePipesSupport bool - noiseStaticKeyCache *KeyCache noiseKeypair *Keypair @@ -76,31 +70,21 @@ func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, re return nil, errNoKeypair } - // if the transport doesn't have a key cache, we make a new one just for - // this session. it's a bit of a waste, but saves us having to check if - // it's nil later - keyCache := tpt.noiseStaticKeyCache - if keyCache == nil { - keyCache = NewKeyCache() - } - localPeerInfo := peerInfo{ noiseKey: tpt.noiseKeypair.publicKey, libp2pKey: tpt.privateKey.GetPublic(), } s := &secureSession{ - insecure: insecure, - initiator: initiator, - prologue: []byte{}, - localKey: tpt.privateKey, - localPeer: tpt.localID, - remotePeer: remote, - local: localPeerInfo, - noisePipesSupport: tpt.noisePipesSupport, - noiseStaticKeyCache: keyCache, - msgBuffer: []byte{}, - noiseKeypair: tpt.noiseKeypair, + insecure: insecure, + initiator: initiator, + prologue: []byte{}, + localKey: tpt.privateKey, + localPeer: tpt.localID, + remotePeer: remote, + local: localPeerInfo, + msgBuffer: []byte{}, + noiseKeypair: tpt.noiseKeypair, } err := s.runHandshake(ctx) @@ -108,10 +92,6 @@ func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, re return s, err } -func (s *secureSession) NoiseStaticKeyCache() *KeyCache { - return s.noiseStaticKeyCache -} - func (s *secureSession) NoisePublicKey() [32]byte { return s.noiseKeypair.publicKey } @@ -180,37 +160,11 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("runHandshake proto marshal payload err=%s", err) } - // If we support Noise pipes, we try IK first, falling back to XX if IK fails. - // The exception is when we're the initiator and don't know the other party's - // static Noise key. Then IK will always fail, so we go straight to XX. - tryIK := s.noisePipesSupport - if s.initiator && s.noiseStaticKeyCache.Load(s.remotePeer) == [32]byte{} { - tryIK = false - } - if tryIK { - // we're either a responder or an initiator with a known static key for the remote peer, try IK - buf, err := s.runHandshake_ik(ctx, payloadEnc) - if err != nil { - // IK failed, pipe to XXfallback - err = s.runHandshake_xx(ctx, true, payloadEnc, buf) - if err != nil { - return fmt.Errorf("runHandshake xx err=%s", err) - } - - s.xx_complete = true - } else { - s.ik_complete = true - } - } else { - // unknown static key for peer, try XX - err := s.runHandshake_xx(ctx, false, payloadEnc, nil) - if err != nil { - return err - } - - s.xx_complete = true + err = s.runHandshake_xx(ctx, payloadEnc) + if err != nil { + return err } - + s.xx_complete = true return nil } diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index d6c33d7a42..9ad3f37743 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -17,91 +17,28 @@ var _ sec.SecureTransport = &Transport{} // Transport implements the interface sec.SecureTransport // https://godoc.org/github.com/libp2p/go-libp2p-core/sec#SecureConn type Transport struct { - localID peer.ID - privateKey crypto.PrivKey - noisePipesSupport bool - noiseStaticKeyCache *KeyCache - noiseKeypair *Keypair -} - -type transportConstructor func(crypto.PrivKey) (*Transport, error) - -// Maker returns a function that will construct a new Noise transport -// using the given Options. The returned function may be provided as a libp2p.Security -// option when configuring a libp2p Host using libp2p.New, and is compatible with the -// "reflection magic" that libp2p.New uses to inject the private identity key: -// -// host := libp2p.New( -// libp2p.Security(noise.ID, noise.Maker())) -// -// The transport can be configured by passing in Options. -// -// To enable the Noise Pipes pattern (which can be more efficient when reconnecting -// to a known peer), pass in the UseNoisePipes Option: -// -// Maker(UseNoisePipes) -// -// To use a specific Noise keypair, pass in the NoiseKeyPair(kp) option, where -// kp is a noise.Keypair struct. This is most useful when using Noise Pipes, whose -// efficiency gains rely on the static Noise key being known in advance. Persisting -// the Noise keypair across process restarts makes it more likely that other peers -// will be able to use the more efficient IK handshake pattern. -// -// Maker(UseNoisePipes, NoiseKeypair(keypairLoadedFromDisk)) -func Maker(options ...Option) transportConstructor { - return func(privKey crypto.PrivKey) (*Transport, error) { - return New(privKey, options...) - } + localID peer.ID + privateKey crypto.PrivKey + noiseKeypair *Keypair } // New creates a new Noise transport using the given private key as its -// libp2p identity key. This function may be used when you want a transport -// instance and know the libp2p Host's identity key before the Host is initialized. -// When configuring a go-libp2p Host using libp2p.New, it's simpler to use -// Maker instead, which will receive the identity key when the Host -// is initialized. -// -// New supports all the same Options as noise.Maker. -// -// To configure a go-libp2p Host to use the newly created transport, pass it into -// libp2p.New wrapped in a libp2p.Security Option. You will also need to -// make sure to set the libp2p.Identity option so that the Host uses the same -// identity key: -// -// privkey := loadPrivateKeyFromSomewhere() -// noiseTpt := noise.New(privkey) -// host := libp2p.New( -// libp2p.Identity(privkey), -// libp2p.Security(noise.ID, noiseTpt)) -func New(privkey crypto.PrivKey, options ...Option) (*Transport, error) { +// libp2p identity key. +func New(privkey crypto.PrivKey) (*Transport, error) { localID, err := peer.IDFromPrivateKey(privkey) if err != nil { return nil, err } - cfg := config{} - cfg.applyOptions(options...) - - kp := cfg.noiseKeypair - if kp == nil { - kp, err = GenerateKeypair() - if err != nil { - return nil, err - } - } - - // the static key cache is only useful if Noise Pipes is enabled - var keyCache *KeyCache - if cfg.noisePipesSupport { - keyCache = NewKeyCache() + kp, err := GenerateKeypair() + if err != nil { + return nil, err } return &Transport{ - localID: localID, - privateKey: privkey, - noisePipesSupport: cfg.noisePipesSupport, - noiseKeypair: kp, - noiseStaticKeyCache: keyCache, + localID: localID, + privateKey: privkey, + noiseKeypair: kp, }, nil } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index a8194fcaff..4c4b05031a 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -7,7 +7,6 @@ import ( "net" "testing" - //ik "github.com/libp2p/go-libp2p-noise/ik" crypto "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" @@ -33,12 +32,6 @@ func newTestTransport(t *testing.T, typ, bits int) *Transport { } } -func newTestTransportPipes(t *testing.T, typ, bits int) *Transport { - tpt := newTestTransport(t, typ, bits) - tpt.noisePipesSupport = true - return tpt -} - // Create a new pair of connected TCP sockets. func newConnPair(t *testing.T) (net.Conn, net.Conn) { lstnr, err := net.Listen("tcp", "localhost:0") @@ -227,73 +220,3 @@ func TestHandshakeXX(t *testing.T) { t.Errorf("Message mismatch. %v != %v", before, after) } } - -// Test IK handshake -func TestHandshakeIK(t *testing.T) { - initTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) - respTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) - - // add responder's static key to initiator's key cache - keycache := NewKeyCache() - keycache.Store(respTransport.localID, respTransport.noiseKeypair.publicKey) - initTransport.noiseStaticKeyCache = keycache - - // do IK handshake - initConn, respConn := connect(t, initTransport, respTransport) - defer initConn.Close() - defer respConn.Close() - - before := []byte("hello world") - _, err := initConn.Write(before) - if err != nil { - t.Fatal(err) - } - - after := make([]byte, len(before)) - _, err = respConn.Read(after) - if err != nil { - t.Fatal(err) - } - - if !bytes.Equal(before, after) { - t.Errorf("Message mismatch. %v != %v", before, after) - } - - // make sure IK was actually used - if !(initConn.ik_complete && respConn.ik_complete) { - t.Error("Expected IK handshake to be used") - } -} - -// Test noise pipes -func TestHandshakeXXfallback(t *testing.T) { - initTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) - respTransport := newTestTransportPipes(t, crypto.Ed25519, 2048) - - // turning on pipes causes it to default to IK, but since we haven't already - // done a handshake, it'll fallback to XX - initConn, respConn := connect(t, initTransport, respTransport) - defer initConn.Close() - defer respConn.Close() - - before := []byte("hello world") - _, err := initConn.Write(before) - if err != nil { - t.Fatal(err) - } - - after := make([]byte, len(before)) - _, err = respConn.Read(after) - if err != nil { - t.Fatal(err) - } - - if !bytes.Equal(before, after) { - t.Errorf("Message mismatch. %v != %v", before, after) - } - - // make sure XX was actually used - if !(initConn.xx_complete && respConn.xx_complete) { - t.Error("Expected XXfallback handshake to be used") - } -} diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go index 9bf37b6daa..f046eae1c0 100644 --- a/p2p/security/noise/xx/XX.noise.go +++ b/p2p/security/noise/xx/XX.noise.go @@ -427,15 +427,10 @@ func initializeResponder(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) return handshakestate{ss, s, e, rs, re, psk} } -func writeMessageA(hs *handshakestate, payload []byte, e *Keypair) (*handshakestate, MessageBuffer) { +func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - if e == nil { - hs.e = GenerateKeypair() - } else { - hs.e = *e - } - + hs.e = GenerateKeypair() ne = hs.e.public_key mixHash(&hs.ss, ne[:]) /* No PSK, so skipping mixKey */ @@ -529,10 +524,10 @@ func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *Noise return &session } -func SendMessage(session *NoiseSession, message []byte, ephemeral *Keypair) (*NoiseSession, MessageBuffer) { +func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageBuffer) { var messageBuffer MessageBuffer if session.mc == 0 { - _, messageBuffer = writeMessageA(&session.hs, message, ephemeral) + _, messageBuffer = writeMessageA(&session.hs, message) } if session.mc == 1 { _, messageBuffer = writeMessageB(&session.hs, message) diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go index 4949978e75..63f6d71cce 100644 --- a/p2p/security/noise/xx/XX_test.go +++ b/p2p/security/noise/xx/XX_test.go @@ -86,7 +86,7 @@ func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { var msgbuf MessageBuffer msg := []byte{} msg = append(msg, payload_init_enc[:]...) - ns_init, msgbuf = SendMessage(ns_init, msg, nil) + ns_init, msgbuf = SendMessage(ns_init, msg) t.Logf("stage 0 msgbuf: %v", msgbuf) t.Logf("stage 0 msgbuf ne len: %d", len(msgbuf.NE())) @@ -111,7 +111,7 @@ func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { t.Fatalf("proto marshal payload fail: %s", err) } msg = append(msg, payload_resp_enc[:]...) - ns_resp, msgbuf = SendMessage(ns_resp, msg, nil) + ns_resp, msgbuf = SendMessage(ns_resp, msg) t.Logf("stage 1 msgbuf: %v", msgbuf) t.Logf("stage 1 msgbuf ne len: %d", len(msgbuf.NE())) @@ -128,7 +128,7 @@ func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { // stage 2: initiator // send message //msg = append(msg, payload_init_enc[:]...) - ns_init, msgbuf = SendMessage(ns_init, nil, nil) + ns_init, msgbuf = SendMessage(ns_init, nil) t.Logf("stage 2 msgbuf: %v", msgbuf) t.Logf("stage 2 msgbuf ne len: %d", len(msgbuf.NE())) diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go index 8bc16d0ce2..d265e92f92 100644 --- a/p2p/security/noise/xx_handshake.go +++ b/p2p/security/noise/xx_handshake.go @@ -14,7 +14,7 @@ import ( func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { var msgbuf xx.MessageBuffer - s.xx_ns, msgbuf = xx.SendMessage(s.xx_ns, payload, nil) + s.ns, msgbuf = xx.SendMessage(s.ns, payload) var encMsgBuf []byte if initial_stage { encMsgBuf = msgbuf.Encode0() @@ -59,7 +59,7 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage decode msg err=%s", err) } - s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) + s.ns, plaintext, valid = xx.RecvMessage(s.ns, msgbuf) if !valid { return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage validation fail") } @@ -74,55 +74,31 @@ func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, // -> s, se // if fallback = true, initialMsg is used as the message in stage 1 of the initiator and stage 0 // of the responder -func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payload []byte, initialMsg []byte) (err error) { +func (s *secureSession) runHandshake_xx(ctx context.Context, payload []byte) (err error) { kp := xx.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) // new XX noise session - s.xx_ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) + s.ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) if s.initiator { // stage 0 // - if !fallback { - err = s.xx_sendHandshakeMessage(nil, true) - if err != nil { - return fmt.Errorf("runHandshake_xx stage 0 initiator fail: %s", err) - } - } else { - e_ik := s.ik_ns.Ephemeral() - e_xx := xx.NewKeypair(e_ik.PubKey(), e_ik.PrivKey()) - - // initialize state as if we sent the first message - s.xx_ns, _ = xx.SendMessage(s.xx_ns, nil, &e_xx) + err = s.xx_sendHandshakeMessage(nil, true) + if err != nil { + return fmt.Errorf("runHandshake_xx stage 0 initiator fail: %s", err) } // stage 1 // var plaintext []byte var valid bool - if !fallback { - // read reply - _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) - if err != nil { - return fmt.Errorf("runHandshake_xx initiator stage 1 fail: %s", err) - } - - if !valid { - return fmt.Errorf("runHandshake_xx stage 1 initiator validation fail") - } - } else { - var msgbuf *xx.MessageBuffer - msgbuf, err = xx.Decode1(initialMsg) - - if err != nil { - return fmt.Errorf("runHandshake_xx decode msg fail: %s", err) - } - - s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, msgbuf) - if !valid { - return fmt.Errorf("runHandshake_xx validation fail") - } - + // read reply + _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) + if err != nil { + return fmt.Errorf("runHandshake_xx initiator stage 1 fail: %s", err) + } + if !valid { + return fmt.Errorf("runHandshake_xx stage 1 initiator validation fail") } // stage 2 // @@ -154,15 +130,11 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl } // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.xx_ns.RemoteKey()) + err = s.verifyPayload(nhp, s.ns.RemoteKey()) if err != nil { return fmt.Errorf("runHandshake_xx stage=2 initiator=true verify payload err=%s", err) } - if s.noisePipesSupport { - s.noiseStaticKeyCache.Store(s.remotePeer, s.xx_ns.RemoteKey()) - } - } else { // stage 0 // @@ -171,29 +143,14 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl var valid bool nhp := new(pb.NoiseHandshakePayload) - if !fallback { - // read message - _, plaintext, valid, err = s.xx_recvHandshakeMessage(true) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=0 initiator=false err=%s", err) - } - - if !valid { - return fmt.Errorf("runHandshake_xx stage=0 initiator=false err=validation fail") - } - - } else { - var msgbuf *xx.MessageBuffer - msgbuf, err = xx.Decode0(initialMsg) - if err != nil { - return err - } + // read message + _, plaintext, valid, err = s.xx_recvHandshakeMessage(true) + if err != nil { + return fmt.Errorf("runHandshake_xx stage=0 initiator=false err=%s", err) + } - xx_msgbuf := xx.NewMessageBuffer(msgbuf.NE(), nil, nil) - s.xx_ns, plaintext, valid = xx.RecvMessage(s.xx_ns, &xx_msgbuf) - if !valid { - return fmt.Errorf("runHandshake_xx validation fail") - } + if !valid { + return fmt.Errorf("runHandshake_xx stage=0 initiator=false err=validation fail") } // stage 1 // @@ -233,17 +190,13 @@ func (s *secureSession) runHandshake_xx(ctx context.Context, fallback bool, payl return fmt.Errorf("runHandshake_xx stage=2 initiator=false set remote peer id err=%s", err) } - s.remote.noiseKey = s.xx_ns.RemoteKey() + s.remote.noiseKey = s.ns.RemoteKey() // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.remote.noiseKey) if err != nil { return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=%s", err) } - - if s.noisePipesSupport { - s.noiseStaticKeyCache.Store(s.remotePeer, s.remote.noiseKey) - } } return nil From 9df4fe637ce2d91d9c460287ef7986b1195aa6aa Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Fri, 28 Feb 2020 10:21:59 -0500 Subject: [PATCH 1671/3965] cleanup --- p2p/security/noise/crypto.go | 4 +- p2p/security/noise/handshake.go | 240 +++++++++++++++++++++++ p2p/security/noise/protocol.go | 305 ----------------------------- p2p/security/noise/rw.go | 127 ++++++++++++ p2p/security/noise/session.go | 127 ++++++++++++ p2p/security/noise/xx_handshake.go | 203 ------------------- 6 files changed, 496 insertions(+), 510 deletions(-) create mode 100644 p2p/security/noise/handshake.go delete mode 100644 p2p/security/noise/protocol.go create mode 100644 p2p/security/noise/rw.go create mode 100644 p2p/security/noise/session.go delete mode 100644 p2p/security/noise/xx_handshake.go diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index 71230ae43d..e837c7adad 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -6,7 +6,7 @@ import ( ) func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) { - if !s.xx_complete { + if !s.handshakeComplete { return nil, errors.New("encrypt err: haven't completed handshake") } @@ -23,7 +23,7 @@ func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) { var ok bool - if !s.xx_complete { + if !s.handshakeComplete { return nil, errors.New("decrypt err: haven't completed handshake") } diff --git a/p2p/security/noise/handshake.go b/p2p/security/noise/handshake.go new file mode 100644 index 0000000000..7fcd8570b5 --- /dev/null +++ b/p2p/security/noise/handshake.go @@ -0,0 +1,240 @@ +package noise + +import ( + "context" + "fmt" + "github.com/gogo/protobuf/proto" + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + + "github.com/libp2p/go-libp2p-noise/pb" + "github.com/libp2p/go-libp2p-noise/xx" +) + +// payloadSigPrefix is prepended to our Noise static key before signing with +// our libp2p identity key. +const payloadSigPrefix = "noise-libp2p-static-key:" + +func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { + s.remote.libp2pKey, err = crypto.UnmarshalPublicKey(key) + return err +} + +func (s *secureSession) setRemotePeerID(key crypto.PubKey) (err error) { + s.remotePeer, err = peer.IDFromPublicKey(key) + return err +} + +func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKey [32]byte) (err error) { + sig := payload.GetIdentitySig() + msg := append([]byte(payloadSigPrefix), noiseKey[:]...) + + ok, err := s.RemotePublicKey().Verify(msg, sig) + if err != nil { + return err + } else if !ok { + return fmt.Errorf("did not verify payload") + } + + return nil +} + +func (s *secureSession) sendHandshakeMessage(payload []byte, initialStage bool) error { + var msgbuf xx.MessageBuffer + s.ns, msgbuf = xx.SendMessage(s.ns, payload) + var encMsgBuf []byte + if initialStage { + encMsgBuf = msgbuf.Encode0() + } else { + encMsgBuf = msgbuf.Encode1() + } + + err := s.writeMsgInsecure(encMsgBuf) + if err != nil { + return fmt.Errorf("sendHandshakeMessage write to conn err=%s", err) + } + + return nil +} + +func (s *secureSession) recvHandshakeMessage(initialStage bool) (buf []byte, plaintext []byte, valid bool, err error) { + buf, err = s.readMsgInsecure() + if err != nil { + return nil, nil, false, fmt.Errorf("recvHandshakeMessage read length err=%s", err) + } + + var msgbuf *xx.MessageBuffer + if initialStage { + msgbuf, err = xx.Decode0(buf) + } else { + msgbuf, err = xx.Decode1(buf) + } + + if err != nil { + return buf, nil, false, fmt.Errorf("recvHandshakeMessage decode msg err=%s", err) + } + + s.ns, plaintext, valid = xx.RecvMessage(s.ns, msgbuf) + if !valid { + return buf, nil, false, fmt.Errorf("recvHandshakeMessage validation fail") + } + + return buf, plaintext, valid, nil +} + +// Runs the XX handshake +// XX: +// -> e +// <- e, ee, s, es +// -> s, se +func (s *secureSession) runHandshake(ctx context.Context) (err error) { + + // setup libp2p keys + localKeyRaw, err := s.LocalPublicKey().Bytes() + if err != nil { + return fmt.Errorf("runHandshake err getting raw pubkey: %s", err) + } + + // sign noise data for payload + noise_pub := s.noiseKeypair.publicKey + signedPayload, err := s.localKey.Sign(append([]byte(payloadSigPrefix), noise_pub[:]...)) + if err != nil { + return fmt.Errorf("runHandshake signing payload err=%s", err) + } + + // create payload + payload := new(pb.NoiseHandshakePayload) + payload.IdentityKey = localKeyRaw + payload.IdentitySig = signedPayload + payloadEnc, err := proto.Marshal(payload) + if err != nil { + return fmt.Errorf("runHandshake proto marshal payload err=%s", err) + } + + kp := xx.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) + + // new XX noise session + s.ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) + + if s.initiator { + // stage 0 // + + err = s.sendHandshakeMessage(nil, true) + if err != nil { + return fmt.Errorf("runHandshake stage 0 initiator fail: %s", err) + } + + // stage 1 // + + var plaintext []byte + var valid bool + // read reply + _, plaintext, valid, err = s.recvHandshakeMessage(false) + if err != nil { + return fmt.Errorf("runHandshake initiator stage 1 fail: %s", err) + } + if !valid { + return fmt.Errorf("runHandshake stage 1 initiator validation fail") + } + + // stage 2 // + + err = s.sendHandshakeMessage(payloadEnc, false) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=true err=%s", err) + } + + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=true err=cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetIdentityKey()) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=true read remote libp2p key fail") + } + + // assert that remote peer ID matches libp2p public key + pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) + if pid != s.remotePeer { + return fmt.Errorf("runHandshake stage=2 initiator=true check remote peer id err: expected %x got %x", s.remotePeer, pid) + } else if err != nil { + return fmt.Errorf("runHandshake stage 2 initiator check remote peer id err %s", err) + } + + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.ns.RemoteKey()) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=true verify payload err=%s", err) + } + + } else { + + // stage 0 // + + var plaintext []byte + var valid bool + nhp := new(pb.NoiseHandshakePayload) + + // read message + _, plaintext, valid, err = s.recvHandshakeMessage(true) + if err != nil { + return fmt.Errorf("runHandshake stage=0 initiator=false err=%s", err) + } + + if !valid { + return fmt.Errorf("runHandshake stage=0 initiator=false err=validation fail") + } + + // stage 1 // + + err = s.sendHandshakeMessage(payloadEnc, false) + if err != nil { + return fmt.Errorf("runHandshake stage=1 initiator=false err=%s", err) + } + + // stage 2 // + + // read message + _, plaintext, valid, err = s.recvHandshakeMessage(false) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=false err=%s", err) + } + + if !valid { + return fmt.Errorf("runHandshake stage=2 initiator=false err=validation fail") + } + + // unmarshal payload + err = proto.Unmarshal(plaintext, nhp) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=false err=cannot unmarshal payload") + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetIdentityKey()) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=false read remote libp2p key fail") + } + + // set remote libp2p public key from payload + err = s.setRemotePeerID(s.RemotePublicKey()) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=false set remote peer id err=%s", err) + } + + s.remote.noiseKey = s.ns.RemoteKey() + + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.remote.noiseKey) + if err != nil { + return fmt.Errorf("runHandshake stage=2 initiator=false err=%s", err) + } + } + + s.handshakeComplete = true + return nil +} diff --git a/p2p/security/noise/protocol.go b/p2p/security/noise/protocol.go deleted file mode 100644 index 421d3ff4e5..0000000000 --- a/p2p/security/noise/protocol.go +++ /dev/null @@ -1,305 +0,0 @@ -package noise - -import ( - "context" - "encoding/binary" - "errors" - "fmt" - "io" - "net" - "sync" - "time" - - "github.com/gogo/protobuf/proto" - logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-core/crypto" - "github.com/libp2p/go-libp2p-core/peer" - - "github.com/libp2p/go-libp2p-noise/pb" - "github.com/libp2p/go-libp2p-noise/xx" -) - -const payload_string = "noise-libp2p-static-key:" - -// Each encrypted transport message must be <= 65,535 bytes, including 16 -// bytes of authentication data. To write larger plaintexts, we split them -// into fragments of maxPlaintextLength before encrypting. -const maxPlaintextLength = 65519 - -var log = logging.Logger("noise") - -var errNoKeypair = errors.New("cannot initiate secureSession - transport has no noise keypair") - -type secureSession struct { - insecure net.Conn - - initiator bool - prologue []byte - - localKey crypto.PrivKey - localPeer peer.ID - remotePeer peer.ID - - local peerInfo - remote peerInfo - - ns *xx.NoiseSession - - xx_complete bool - - noiseKeypair *Keypair - - msgBuffer []byte - readLock sync.Mutex - writeLock sync.Mutex -} - -type peerInfo struct { - noiseKey [32]byte // static noise public key - libp2pKey crypto.PubKey -} - -// newSecureSession creates a noise session over the given insecure Conn, using the static -// Noise keypair and libp2p identity keypair from the given Transport. -// -// If tpt.noisePipesSupport == true, the Noise Pipes handshake protocol will be used, -// which consists of the IK and XXfallback handshake patterns. With Noise Pipes on, we first try IK, -// if that fails, move to XXfallback. With Noise Pipes off, we always do XX. -func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, remote peer.ID, initiator bool) (*secureSession, error) { - if tpt.noiseKeypair == nil { - return nil, errNoKeypair - } - - localPeerInfo := peerInfo{ - noiseKey: tpt.noiseKeypair.publicKey, - libp2pKey: tpt.privateKey.GetPublic(), - } - - s := &secureSession{ - insecure: insecure, - initiator: initiator, - prologue: []byte{}, - localKey: tpt.privateKey, - localPeer: tpt.localID, - remotePeer: remote, - local: localPeerInfo, - msgBuffer: []byte{}, - noiseKeypair: tpt.noiseKeypair, - } - - err := s.runHandshake(ctx) - - return s, err -} - -func (s *secureSession) NoisePublicKey() [32]byte { - return s.noiseKeypair.publicKey -} - -func (s *secureSession) NoisePrivateKey() [32]byte { - return s.noiseKeypair.privateKey -} - -func (s *secureSession) readLength() (int, error) { - buf := make([]byte, 2) - _, err := io.ReadFull(s.insecure, buf) - return int(binary.BigEndian.Uint16(buf)), err -} - -func (s *secureSession) writeLength(length int) error { - buf := make([]byte, 2) - binary.BigEndian.PutUint16(buf, uint16(length)) - _, err := s.insecure.Write(buf) - return err -} - -func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { - s.remote.libp2pKey, err = crypto.UnmarshalPublicKey(key) - return err -} - -func (s *secureSession) setRemotePeerID(key crypto.PubKey) (err error) { - s.remotePeer, err = peer.IDFromPublicKey(key) - return err -} - -func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKey [32]byte) (err error) { - sig := payload.GetIdentitySig() - msg := append([]byte(payload_string), noiseKey[:]...) - - ok, err := s.RemotePublicKey().Verify(msg, sig) - if err != nil { - return err - } else if !ok { - return fmt.Errorf("did not verify payload") - } - - return nil -} - -func (s *secureSession) runHandshake(ctx context.Context) error { - // setup libp2p keys - localKeyRaw, err := s.LocalPublicKey().Bytes() - if err != nil { - return fmt.Errorf("runHandshake err getting raw pubkey: %s", err) - } - - // sign noise data for payload - noise_pub := s.noiseKeypair.publicKey - signedPayload, err := s.localKey.Sign(append([]byte(payload_string), noise_pub[:]...)) - if err != nil { - return fmt.Errorf("runHandshake signing payload err=%s", err) - } - - // create payload - payload := new(pb.NoiseHandshakePayload) - payload.IdentityKey = localKeyRaw - payload.IdentitySig = signedPayload - payloadEnc, err := proto.Marshal(payload) - if err != nil { - return fmt.Errorf("runHandshake proto marshal payload err=%s", err) - } - - err = s.runHandshake_xx(ctx, payloadEnc) - if err != nil { - return err - } - s.xx_complete = true - return nil -} - -func (s *secureSession) LocalAddr() net.Addr { - return s.insecure.LocalAddr() -} - -func (s *secureSession) LocalPeer() peer.ID { - return s.localPeer -} - -func (s *secureSession) LocalPrivateKey() crypto.PrivKey { - return s.localKey -} - -func (s *secureSession) LocalPublicKey() crypto.PubKey { - return s.localKey.GetPublic() -} - -func (s *secureSession) Read(buf []byte) (int, error) { - s.readLock.Lock() - defer s.readLock.Unlock() - - l := len(buf) - - // if we have previously unread bytes, and they fit into the buf, copy them over and return - if l <= len(s.msgBuffer) { - copy(buf, s.msgBuffer) - s.msgBuffer = s.msgBuffer[l:] - return l, nil - } - - readChunk := func(buf []byte) (int, error) { - // read length of encrypted message - l, err := s.readLength() - if err != nil { - return 0, err - } - - // read and decrypt ciphertext - ciphertext := make([]byte, l) - _, err = io.ReadFull(s.insecure, ciphertext) - if err != nil { - return 0, err - } - - plaintext, err := s.Decrypt(ciphertext) - if err != nil { - return 0, err - } - - // append plaintext to message buffer, copy over what can fit in the buf - // then advance message buffer to remove what was copied - s.msgBuffer = append(s.msgBuffer, plaintext...) - c := copy(buf, s.msgBuffer) - s.msgBuffer = s.msgBuffer[c:] - return c, nil - } - - total := 0 - for i := 0; i < len(buf); i += maxPlaintextLength { - end := i + maxPlaintextLength - if end > len(buf) { - end = len(buf) - } - - c, err := readChunk(buf[i:end]) - total += c - if err != nil { - return total, err - } - } - - return total, nil -} - -func (s *secureSession) RemoteAddr() net.Addr { - return s.insecure.RemoteAddr() -} - -func (s *secureSession) RemotePeer() peer.ID { - return s.remotePeer -} - -func (s *secureSession) RemotePublicKey() crypto.PubKey { - return s.remote.libp2pKey -} - -func (s *secureSession) SetDeadline(t time.Time) error { - return s.insecure.SetDeadline(t) -} - -func (s *secureSession) SetReadDeadline(t time.Time) error { - return s.insecure.SetReadDeadline(t) -} - -func (s *secureSession) SetWriteDeadline(t time.Time) error { - return s.insecure.SetWriteDeadline(t) -} - -func (s *secureSession) Write(in []byte) (int, error) { - s.writeLock.Lock() - defer s.writeLock.Unlock() - - writeChunk := func(in []byte) (int, error) { - ciphertext, err := s.Encrypt(in) - if err != nil { - return 0, err - } - - err = s.writeLength(len(ciphertext)) - if err != nil { - return 0, err - } - - _, err = s.insecure.Write(ciphertext) - return len(in), err - } - - written := 0 - for i := 0; i < len(in); i += maxPlaintextLength { - end := i + maxPlaintextLength - if end > len(in) { - end = len(in) - } - - l, err := writeChunk(in[i:end]) - written += l - if err != nil { - return written, err - } - } - return written, nil -} - -func (s *secureSession) Close() error { - return s.insecure.Close() -} diff --git a/p2p/security/noise/rw.go b/p2p/security/noise/rw.go new file mode 100644 index 0000000000..fa39cff4b7 --- /dev/null +++ b/p2p/security/noise/rw.go @@ -0,0 +1,127 @@ +package noise + +import ( + "encoding/binary" + "fmt" + "io" +) + +// Each encrypted transport message must be <= 65,535 bytes, including 16 +// bytes of authentication data. To write larger plaintexts, we split them +// into fragments of maxPlaintextLength before encrypting. +const maxPlaintextLength = 65519 + +// Read reads from the secure connection, filling `buf` with plaintext data. +// May read less than len(buf) if data is available immediately. +func (s *secureSession) Read(buf []byte) (int, error) { + s.readLock.Lock() + defer s.readLock.Unlock() + + l := len(buf) + + // if we have previously unread bytes, and they fit into the buf, copy them over and return + if l <= len(s.msgBuffer) { + copy(buf, s.msgBuffer) + s.msgBuffer = s.msgBuffer[l:] + return l, nil + } + + readChunk := func(buf []byte) (int, error) { + // read length of encrypted message + ciphertext, err := s.readMsgInsecure() + if err != nil { + return 0, err + } + + plaintext, err := s.Decrypt(ciphertext) + if err != nil { + return 0, err + } + + // append plaintext to message buffer, copy over what can fit in the buf + // then advance message buffer to remove what was copied + s.msgBuffer = append(s.msgBuffer, plaintext...) + c := copy(buf, s.msgBuffer) + s.msgBuffer = s.msgBuffer[c:] + return c, nil + } + + total := 0 + for i := 0; i < len(buf); i += maxPlaintextLength { + end := i + maxPlaintextLength + if end > len(buf) { + end = len(buf) + } + + c, err := readChunk(buf[i:end]) + total += c + if err != nil { + return total, err + } + } + + return total, nil +} + +// Write encrypts the plaintext `in` data and sends it on the +// secure connection. +func (s *secureSession) Write(in []byte) (int, error) { + s.writeLock.Lock() + defer s.writeLock.Unlock() + + writeChunk := func(in []byte) (int, error) { + ciphertext, err := s.Encrypt(in) + if err != nil { + return 0, err + } + + err = s.writeMsgInsecure(ciphertext) + if err != nil { + return 0, err + } + return len(in), err + } + + written := 0 + for i := 0; i < len(in); i += maxPlaintextLength { + end := i + maxPlaintextLength + if end > len(in) { + end = len(in) + } + + l, err := writeChunk(in[i:end]) + written += l + if err != nil { + return written, err + } + } + return written, nil +} + +// readMsgInsecure reads a message from the insecure channel. +// it first reads the message length, then consumes that many bytes +// from the insecure conn. +func (s *secureSession) readMsgInsecure() ([]byte, error) { + buf := make([]byte, 2) + _, err := io.ReadFull(s.insecure, buf) + if err != nil { + return nil, fmt.Errorf("error reading length prefix: %s", err) + } + size := int(binary.BigEndian.Uint16(buf)) + buf = make([]byte, size) + _, err = io.ReadFull(s.insecure, buf) + return buf, err +} + +// writeMsgInsecure writes to the insecure conn. +// data will be prefixed with its length in bytes, written as a 16-bit uint in network order. +func (s *secureSession) writeMsgInsecure(data []byte) error { + buf := make([]byte, 2) + binary.BigEndian.PutUint16(buf, uint16(len(data))) + _, err := s.insecure.Write(buf) + if err != nil { + return fmt.Errorf("error writing length prefix: %s", err) + } + _, err = s.insecure.Write(data) + return err +} diff --git a/p2p/security/noise/session.go b/p2p/security/noise/session.go new file mode 100644 index 0000000000..0afb27f635 --- /dev/null +++ b/p2p/security/noise/session.go @@ -0,0 +1,127 @@ +package noise + +import ( + "context" + "errors" + "net" + "sync" + "time" + + logging "github.com/ipfs/go-log" + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + + "github.com/libp2p/go-libp2p-noise/xx" +) + +var log = logging.Logger("noise") + +var errNoKeypair = errors.New("cannot initiate secureSession - transport has no noise keypair") + +type secureSession struct { + insecure net.Conn + + initiator bool + prologue []byte + + localKey crypto.PrivKey + localPeer peer.ID + remotePeer peer.ID + + local peerInfo + remote peerInfo + + ns *xx.NoiseSession + + handshakeComplete bool + noiseKeypair *Keypair + + msgBuffer []byte + readLock sync.Mutex + writeLock sync.Mutex +} + +type peerInfo struct { + noiseKey [32]byte // static noise public key + libp2pKey crypto.PubKey +} + +// newSecureSession creates a noise session over the given insecure Conn, using the static +// Noise keypair and libp2p identity keypair from the given Transport. +func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, remote peer.ID, initiator bool) (*secureSession, error) { + if tpt.noiseKeypair == nil { + return nil, errNoKeypair + } + + localPeerInfo := peerInfo{ + noiseKey: tpt.noiseKeypair.publicKey, + libp2pKey: tpt.privateKey.GetPublic(), + } + + s := &secureSession{ + insecure: insecure, + initiator: initiator, + prologue: []byte{}, + localKey: tpt.privateKey, + localPeer: tpt.localID, + remotePeer: remote, + local: localPeerInfo, + msgBuffer: []byte{}, + noiseKeypair: tpt.noiseKeypair, + } + + err := s.runHandshake(ctx) + return s, err +} + +func (s *secureSession) NoisePublicKey() [32]byte { + return s.noiseKeypair.publicKey +} + +func (s *secureSession) NoisePrivateKey() [32]byte { + return s.noiseKeypair.privateKey +} + +func (s *secureSession) LocalAddr() net.Addr { + return s.insecure.LocalAddr() +} + +func (s *secureSession) LocalPeer() peer.ID { + return s.localPeer +} + +func (s *secureSession) LocalPrivateKey() crypto.PrivKey { + return s.localKey +} + +func (s *secureSession) LocalPublicKey() crypto.PubKey { + return s.localKey.GetPublic() +} + +func (s *secureSession) RemoteAddr() net.Addr { + return s.insecure.RemoteAddr() +} + +func (s *secureSession) RemotePeer() peer.ID { + return s.remotePeer +} + +func (s *secureSession) RemotePublicKey() crypto.PubKey { + return s.remote.libp2pKey +} + +func (s *secureSession) SetDeadline(t time.Time) error { + return s.insecure.SetDeadline(t) +} + +func (s *secureSession) SetReadDeadline(t time.Time) error { + return s.insecure.SetReadDeadline(t) +} + +func (s *secureSession) SetWriteDeadline(t time.Time) error { + return s.insecure.SetWriteDeadline(t) +} + +func (s *secureSession) Close() error { + return s.insecure.Close() +} diff --git a/p2p/security/noise/xx_handshake.go b/p2p/security/noise/xx_handshake.go deleted file mode 100644 index d265e92f92..0000000000 --- a/p2p/security/noise/xx_handshake.go +++ /dev/null @@ -1,203 +0,0 @@ -package noise - -import ( - "context" - "fmt" - "io" - - proto "github.com/gogo/protobuf/proto" - "github.com/libp2p/go-libp2p-core/peer" - - pb "github.com/libp2p/go-libp2p-noise/pb" - xx "github.com/libp2p/go-libp2p-noise/xx" -) - -func (s *secureSession) xx_sendHandshakeMessage(payload []byte, initial_stage bool) error { - var msgbuf xx.MessageBuffer - s.ns, msgbuf = xx.SendMessage(s.ns, payload) - var encMsgBuf []byte - if initial_stage { - encMsgBuf = msgbuf.Encode0() - } else { - encMsgBuf = msgbuf.Encode1() - } - - err := s.writeLength(len(encMsgBuf)) - if err != nil { - return fmt.Errorf("xx_sendHandshakeMessage write length err=%s", err) - } - - _, err = s.insecure.Write(encMsgBuf) - if err != nil { - return fmt.Errorf("xx_sendHandshakeMessage write to conn err=%s", err) - } - - return nil -} - -func (s *secureSession) xx_recvHandshakeMessage(initial_stage bool) (buf []byte, plaintext []byte, valid bool, err error) { - l, err := s.readLength() - if err != nil { - return nil, nil, false, fmt.Errorf("xx_recvHandshakeMessage read length err=%s", err) - } - - buf = make([]byte, l) - - _, err = io.ReadFull(s.insecure, buf) - if err != nil { - return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage read from conn err=%s", err) - } - - var msgbuf *xx.MessageBuffer - if initial_stage { - msgbuf, err = xx.Decode0(buf) - } else { - msgbuf, err = xx.Decode1(buf) - } - - if err != nil { - return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage decode msg err=%s", err) - } - - s.ns, plaintext, valid = xx.RecvMessage(s.ns, msgbuf) - if !valid { - return buf, nil, false, fmt.Errorf("xx_recvHandshakeMessage validation fail") - } - - return buf, plaintext, valid, nil -} - -// Runs the XX handshake -// XX: -// -> e -// <- e, ee, s, es -// -> s, se -// if fallback = true, initialMsg is used as the message in stage 1 of the initiator and stage 0 -// of the responder -func (s *secureSession) runHandshake_xx(ctx context.Context, payload []byte) (err error) { - kp := xx.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) - - // new XX noise session - s.ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) - - if s.initiator { - // stage 0 // - - err = s.xx_sendHandshakeMessage(nil, true) - if err != nil { - return fmt.Errorf("runHandshake_xx stage 0 initiator fail: %s", err) - } - - // stage 1 // - - var plaintext []byte - var valid bool - // read reply - _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) - if err != nil { - return fmt.Errorf("runHandshake_xx initiator stage 1 fail: %s", err) - } - if !valid { - return fmt.Errorf("runHandshake_xx stage 1 initiator validation fail") - } - - // stage 2 // - - err = s.xx_sendHandshakeMessage(payload, false) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=true err=%s", err) - } - - // unmarshal payload - nhp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=true err=cannot unmarshal payload") - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetIdentityKey()) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=true read remote libp2p key fail") - } - - // assert that remote peer ID matches libp2p public key - pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) - if pid != s.remotePeer { - return fmt.Errorf("runHandshake_xx stage=2 initiator=true check remote peer id err: expected %x got %x", s.remotePeer, pid) - } else if err != nil { - return fmt.Errorf("runHandshake_xx stage 2 initiator check remote peer id err %s", err) - } - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ns.RemoteKey()) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=true verify payload err=%s", err) - } - - } else { - - // stage 0 // - - var plaintext []byte - var valid bool - nhp := new(pb.NoiseHandshakePayload) - - // read message - _, plaintext, valid, err = s.xx_recvHandshakeMessage(true) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=0 initiator=false err=%s", err) - } - - if !valid { - return fmt.Errorf("runHandshake_xx stage=0 initiator=false err=validation fail") - } - - // stage 1 // - - err = s.xx_sendHandshakeMessage(payload, false) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=1 initiator=false err=%s", err) - } - - // stage 2 // - - // read message - _, plaintext, valid, err = s.xx_recvHandshakeMessage(false) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=%s", err) - } - - if !valid { - return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=validation fail") - } - - // unmarshal payload - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=cannot unmarshal payload") - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetIdentityKey()) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=false read remote libp2p key fail") - } - - // set remote libp2p public key from payload - err = s.setRemotePeerID(s.RemotePublicKey()) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=false set remote peer id err=%s", err) - } - - s.remote.noiseKey = s.ns.RemoteKey() - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.remote.noiseKey) - if err != nil { - return fmt.Errorf("runHandshake_xx stage=2 initiator=false err=%s", err) - } - } - - return nil -} From 1bdb5d3916a74e97b2e9875ee4c594ffe9fa454d Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Fri, 28 Feb 2020 10:51:13 -0500 Subject: [PATCH 1672/3965] test crypto error cases --- p2p/security/noise/crypto_test.go | 40 +++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/p2p/security/noise/crypto_test.go b/p2p/security/noise/crypto_test.go index dff7ab7c01..efc1a8f31d 100644 --- a/p2p/security/noise/crypto_test.go +++ b/p2p/security/noise/crypto_test.go @@ -2,6 +2,8 @@ package noise import ( "bytes" + "context" + "net" "testing" "github.com/libp2p/go-libp2p-core/crypto" @@ -63,3 +65,41 @@ func TestEncryptAndDecrypt_RespToInit(t *testing.T) { t.Fatal(err) } } + +func TestCryptoFailsIfCiphertextIsAltered(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + respTransport := newTestTransport(t, crypto.Ed25519, 2048) + + initConn, respConn := connect(t, initTransport, respTransport) + defer initConn.Close() + defer respConn.Close() + + plaintext := []byte("helloworld") + ciphertext, err := respConn.Encrypt(plaintext) + if err != nil { + t.Fatal(err) + } + + ciphertext[0] = ^ciphertext[0] + + _, err = initConn.Decrypt(ciphertext) + if err == nil { + t.Fatal("expected decryption to fail when ciphertext altered") + } +} + +func TestCryptoFailsIfHandshakeIncomplete(t *testing.T) { + initTransport := newTestTransport(t, crypto.Ed25519, 2048) + init, resp := net.Pipe() + _ = resp.Close() + + session, _ := newSecureSession(initTransport, context.TODO(), init, "remote-peer", true) + _, err := session.Encrypt([]byte("hi")) + if err == nil { + t.Error("expected encryption error when handshake incomplete") + } + _, err = session.Decrypt([]byte("it's a secret")) + if err == nil { + t.Error("expected decryption error when handshake incomplete") + } +} From f5d58efcaee5afd8f5baceb37af9b563d932e2c4 Mon Sep 17 00:00:00 2001 From: Hlib Date: Fri, 28 Feb 2020 20:12:16 +0200 Subject: [PATCH 1673/3965] revert Multiplexer name back to Transport --- p2p/muxer/yamux/{mux.go => transport.go} | 14 +++++++------- p2p/muxer/yamux/{mux_test.go => transport_test.go} | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) rename p2p/muxer/yamux/{mux.go => transport.go} (72%) rename p2p/muxer/yamux/{mux_test.go => transport_test.go} (75%) diff --git a/p2p/muxer/yamux/mux.go b/p2p/muxer/yamux/transport.go similarity index 72% rename from p2p/muxer/yamux/mux.go rename to p2p/muxer/yamux/transport.go index f0c1af2995..9b33f81440 100644 --- a/p2p/muxer/yamux/mux.go +++ b/p2p/muxer/yamux/transport.go @@ -8,7 +8,7 @@ import ( yamux "github.com/libp2p/go-yamux" ) -var DefaultTransport *Multiplexer +var DefaultTransport *Transport func init() { config := yamux.DefaultConfig() @@ -23,14 +23,14 @@ func init() { // We always run over a security transport that buffers internally // (i.e., uses a block cipher). config.ReadBufSize = 0 - DefaultTransport = (*Multiplexer)(config) + DefaultTransport = (*Transport)(config) } -// Multiplexer implements mux.Multiplexer that constructs +// Transport implements mux.Multiplexer that constructs // yamux-backed muxed connections. -type Multiplexer yamux.Config +type Transport yamux.Config -func (t *Multiplexer) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { +func (t *Transport) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { var s *yamux.Session var err error if isServer { @@ -41,8 +41,8 @@ func (t *Multiplexer) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) return (*conn)(s), err } -func (t *Multiplexer) Config() *yamux.Config { +func (t *Transport) Config() *yamux.Config { return (*yamux.Config)(t) } -var _ mux.Multiplexer = &Multiplexer{} +var _ mux.Multiplexer = &Transport{} diff --git a/p2p/muxer/yamux/mux_test.go b/p2p/muxer/yamux/transport_test.go similarity index 75% rename from p2p/muxer/yamux/mux_test.go rename to p2p/muxer/yamux/transport_test.go index d3a9b4ff79..0499971a64 100644 --- a/p2p/muxer/yamux/mux_test.go +++ b/p2p/muxer/yamux/transport_test.go @@ -6,6 +6,6 @@ import ( tmux "github.com/libp2p/go-libp2p-testing/suites/mux" ) -func TestDefaultMultiplexer(t *testing.T) { +func TestDefaultTransport(t *testing.T) { tmux.SubtestAll(t, DefaultTransport) } From 143622db35785bb73193b368f577aa736f92c8d8 Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Sat, 29 Feb 2020 11:46:17 +0530 Subject: [PATCH 1674/3965] peer connectedness event --- core/event/network.go | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/core/event/network.go b/core/event/network.go index 8fcc1eacec..c03c8d94d8 100644 --- a/core/event/network.go +++ b/core/event/network.go @@ -2,18 +2,13 @@ package event import "github.com/libp2p/go-libp2p-core/network" -// EvtPeerConnectionStateChange should be emitted when we connect to a peer or disconnect -// from a peer. It contains the network interface for the connection, -// the connection handle & the new state of the connection. -type EvtPeerConnectionStateChange struct { +// EvtPeerStateChange should be emitted when we form our first connection with a peer or drop our last +// connection with the peer. Essentially, it is emitted in two cases: +// a) We go from having no connection with a peer to having a connection with a peer. +// b) We go from having a connection/s with a peer to having no connection with the peer. +// It contains the network interface for the connection, the connection handle for the first/last connection and +// the new connection state. +type EvtPeerStateChange struct { Connection network.Conn NewState network.Connectedness } - -// EvtStreamStateChange should be emitted when we open a new stream with a peer or close an existing stream. -// It contains the network interface for the connection, the stream handle & -// the new state of the stream. -type EvtStreamStateChange struct { - Stream network.Stream - NewState network.Connectedness -} From f40e281c775c70cf5eb720ded7ef1c8506be287e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 1 Mar 2020 13:01:29 +0700 Subject: [PATCH 1675/3965] update quic-go to v0.15.0 --- p2p/transport/quic/conn.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/quic/conn.go b/p2p/transport/quic/conn.go index 75e2193692..9ba8593b72 100644 --- a/p2p/transport/quic/conn.go +++ b/p2p/transport/quic/conn.go @@ -28,7 +28,7 @@ type conn struct { var _ tpt.CapableConn = &conn{} func (c *conn) Close() error { - return c.sess.Close() + return c.sess.CloseWithError(0, "") } // IsClosed returns whether a connection is fully closed. From 3416b29bcd544443127861ab411645f06ca446c8 Mon Sep 17 00:00:00 2001 From: Hlib Date: Mon, 2 Mar 2020 12:32:40 +0200 Subject: [PATCH 1676/3965] use explicit return values --- p2p/muxer/yamux/stream.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/yamux/stream.go b/p2p/muxer/yamux/stream.go index 866b781fbe..9f6e4f9a7a 100644 --- a/p2p/muxer/yamux/stream.go +++ b/p2p/muxer/yamux/stream.go @@ -16,7 +16,7 @@ func (s *stream) Read(b []byte) (n int, err error) { err = mux.ErrReset } - return + return n, err } func (s *stream) Write(b []byte) (n int, err error) { @@ -25,7 +25,7 @@ func (s *stream) Write(b []byte) (n int, err error) { err = mux.ErrReset } - return + return n, err } func (s *stream) Close() error { From 6b1179649ff1276f10a8a6fe8b384959a281d743 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 2 Mar 2020 11:21:18 +0000 Subject: [PATCH 1677/3965] build(deps): bump github.com/multiformats/go-multiaddr Bumps [github.com/multiformats/go-multiaddr](https://github.com/multiformats/go-multiaddr) from 0.2.0 to 0.2.1. - [Release notes](https://github.com/multiformats/go-multiaddr/releases) - [Commits](https://github.com/multiformats/go-multiaddr/compare/v0.2.0...v0.2.1) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index a215127b59..097dd12de8 100644 --- a/go.mod +++ b/go.mod @@ -30,7 +30,7 @@ require ( github.com/libp2p/go-tcp-transport v0.1.1 github.com/libp2p/go-ws-transport v0.2.0 github.com/miekg/dns v1.1.12 // indirect - github.com/multiformats/go-multiaddr v0.2.0 + github.com/multiformats/go-multiaddr v0.2.1 github.com/multiformats/go-multiaddr-dns v0.2.0 github.com/multiformats/go-multiaddr-net v0.1.2 github.com/multiformats/go-multistream v0.1.1 diff --git a/go.sum b/go.sum index 3cfebf64c4..2c743856d7 100644 --- a/go.sum +++ b/go.sum @@ -276,6 +276,8 @@ github.com/multiformats/go-multiaddr v0.1.1 h1:rVAztJYMhCQ7vEFr8FvxW3mS+HF2eY/oP github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo= github.com/multiformats/go-multiaddr v0.2.0 h1:lR52sFwcTCuQb6bTfnXF6zA2XfyYvyd+5a9qECv/J90= github.com/multiformats/go-multiaddr v0.2.0/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4= +github.com/multiformats/go-multiaddr v0.2.1 h1:SgG/cw5vqyB5QQe5FPe2TqggU9WtrA9X4nZw7LlVqOI= +github.com/multiformats/go-multiaddr v0.2.1/go.mod h1:s/Apk6IyxfvMjDafnhJgJ3/46z7tZ04iMk5wP4QMGGE= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= @@ -310,6 +312,7 @@ github.com/multiformats/go-multistream v0.1.1 h1:JlAdpIFhBhGRLxe9W6Om0w++Gd6KMWo github.com/multiformats/go-multistream v0.1.1/go.mod h1:KmHZ40hzVxiaiwlj3MEbYgK9JFk2/9UktWZAF54Du38= github.com/multiformats/go-varint v0.0.1 h1:TR/0rdQtnNxuN2IhiB639xC3tWM4IUi7DkTBVTdGW/M= github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= +github.com/multiformats/go-varint v0.0.2/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/multiformats/go-varint v0.0.5 h1:XVZwSo04Cs3j/jS0uAEPpT3JY6DzMcVLLoWOSnCxOjg= github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= From 7773c57d1d99075da7fb0589216a9a1906378e61 Mon Sep 17 00:00:00 2001 From: Hlib Date: Mon, 2 Mar 2020 13:47:11 +0200 Subject: [PATCH 1678/3965] restructure files layout --- p2p/muxer/mplex/conn.go | 32 ++++++++++++++ p2p/muxer/mplex/multiplex.go | 42 ------------------- p2p/muxer/mplex/transport.go | 22 ++++++++++ .../{multiplex_test.go => transport_test.go} | 2 +- 4 files changed, 55 insertions(+), 43 deletions(-) create mode 100644 p2p/muxer/mplex/conn.go delete mode 100644 p2p/muxer/mplex/multiplex.go create mode 100644 p2p/muxer/mplex/transport.go rename p2p/muxer/mplex/{multiplex_test.go => transport_test.go} (77%) diff --git a/p2p/muxer/mplex/conn.go b/p2p/muxer/mplex/conn.go new file mode 100644 index 0000000000..00f89a04d1 --- /dev/null +++ b/p2p/muxer/mplex/conn.go @@ -0,0 +1,32 @@ +package peerstream_multiplex + +import ( + "github.com/libp2p/go-libp2p-core/mux" + mp "github.com/libp2p/go-mplex" +) + +type conn mp.Multiplex + +func (c *conn) Close() error { + return c.mplex().Close() +} + +func (c *conn) IsClosed() bool { + return c.mplex().IsClosed() +} + +// OpenStream creates a new stream. +func (c *conn) OpenStream() (mux.MuxedStream, error) { + return c.mplex().NewStream() +} + +// AcceptStream accepts a stream opened by the other side. +func (c *conn) AcceptStream() (mux.MuxedStream, error) { + return c.mplex().Accept() +} + +func (c *conn) mplex() *mp.Multiplex { + return (*mp.Multiplex)(c) +} + +var _ mux.MuxedConn = &conn{} diff --git a/p2p/muxer/mplex/multiplex.go b/p2p/muxer/mplex/multiplex.go deleted file mode 100644 index bf53c9e7fb..0000000000 --- a/p2p/muxer/mplex/multiplex.go +++ /dev/null @@ -1,42 +0,0 @@ -package peerstream_multiplex - -import ( - "net" - - "github.com/libp2p/go-libp2p-core/mux" - - mp "github.com/libp2p/go-mplex" -) - -type conn struct { - *mp.Multiplex -} - -func (c *conn) Close() error { - return c.Multiplex.Close() -} - -func (c *conn) IsClosed() bool { - return c.Multiplex.IsClosed() -} - -// OpenStream creates a new stream. -func (c *conn) OpenStream() (mux.MuxedStream, error) { - return c.Multiplex.NewStream() -} - -// AcceptStream accepts a stream opened by the other side. -func (c *conn) AcceptStream() (mux.MuxedStream, error) { - return c.Multiplex.Accept() -} - -// Transport is a go-peerstream transport that constructs -// multiplex-backed connections. -type Transport struct{} - -// DefaultTransport has default settings for multiplex -var DefaultTransport = &Transport{} - -func (t *Transport) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { - return &conn{mp.NewMultiplex(nc, isServer)}, nil -} diff --git a/p2p/muxer/mplex/transport.go b/p2p/muxer/mplex/transport.go new file mode 100644 index 0000000000..395b855383 --- /dev/null +++ b/p2p/muxer/mplex/transport.go @@ -0,0 +1,22 @@ +package peerstream_multiplex + +import ( + "net" + + "github.com/libp2p/go-libp2p-core/mux" + + mp "github.com/libp2p/go-mplex" +) + +// DefaultTransport has default settings for Transport +var DefaultTransport = &Transport{} + +// Transport implements mux.Multiplexer that constructs +// mplex-backed muxed connections. +type Transport struct{} + +func (t *Transport) NewConn(nc net.Conn, isServer bool) (mux.MuxedConn, error) { + return (*conn)(mp.NewMultiplex(nc, isServer)), nil +} + +var _ mux.Multiplexer = &Transport{} diff --git a/p2p/muxer/mplex/multiplex_test.go b/p2p/muxer/mplex/transport_test.go similarity index 77% rename from p2p/muxer/mplex/multiplex_test.go rename to p2p/muxer/mplex/transport_test.go index 5fd46a1f03..134fd44454 100644 --- a/p2p/muxer/mplex/multiplex_test.go +++ b/p2p/muxer/mplex/transport_test.go @@ -6,6 +6,6 @@ import ( test "github.com/libp2p/go-libp2p-testing/suites/mux" ) -func TestMultiplexTransport(t *testing.T) { +func TestDefaultTransport(t *testing.T) { test.SubtestAll(t, DefaultTransport) } From 166a2cc8c598749f19af398a5c25741d887274b0 Mon Sep 17 00:00:00 2001 From: Hlib Date: Mon, 2 Mar 2020 13:52:06 +0200 Subject: [PATCH 1679/3965] define stream type for mp.Stream and update go-mplex --- p2p/muxer/mplex/stream.go | 55 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 p2p/muxer/mplex/stream.go diff --git a/p2p/muxer/mplex/stream.go b/p2p/muxer/mplex/stream.go new file mode 100644 index 0000000000..382183b132 --- /dev/null +++ b/p2p/muxer/mplex/stream.go @@ -0,0 +1,55 @@ +package peerstream_multiplex + +import ( + "time" + + "github.com/libp2p/go-libp2p-core/mux" + mp "github.com/libp2p/go-mplex" +) + +// stream implements mux.MuxedStream over mplex.Stream. +type stream mp.Stream + +func (s *stream) Read(b []byte) (n int, err error) { + n, err = s.mplex().Read(b) + if err == mp.ErrStreamReset { + err = mux.ErrReset + } + + return +} + +func (s *stream) Write(b []byte) (n int, err error) { + n, err = s.mplex().Write(b) + if err == mp.ErrStreamReset { + err = mux.ErrReset + } + + return +} + +func (s *stream) Close() error { + return s.mplex().Close() +} + +func (s *stream) Reset() error { + return s.mplex().Reset() +} + +func (s *stream) SetDeadline(t time.Time) error { + return s.mplex().SetDeadline(t) +} + +func (s *stream) SetReadDeadline(t time.Time) error { + return s.mplex().SetReadDeadline(t) +} + +func (s *stream) SetWriteDeadline(t time.Time) error { + return s.mplex().SetWriteDeadline(t) +} + +func (s *stream) mplex() *mp.Stream { + return (*mp.Stream)(s) +} + +var _ mux.MuxedStream = &stream{} From 8a4d7ec4260e1dee45c460cced2c868fd7f28729 Mon Sep 17 00:00:00 2001 From: Hlib Date: Mon, 2 Mar 2020 14:03:01 +0200 Subject: [PATCH 1680/3965] use explicit return values --- p2p/muxer/mplex/stream.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/mplex/stream.go b/p2p/muxer/mplex/stream.go index 382183b132..2708ed3a36 100644 --- a/p2p/muxer/mplex/stream.go +++ b/p2p/muxer/mplex/stream.go @@ -16,7 +16,7 @@ func (s *stream) Read(b []byte) (n int, err error) { err = mux.ErrReset } - return + return n, err } func (s *stream) Write(b []byte) (n int, err error) { @@ -25,7 +25,7 @@ func (s *stream) Write(b []byte) (n int, err error) { err = mux.ErrReset } - return + return n, err } func (s *stream) Close() error { From 4d642f648ca29872b4da9b85e7df577e2059357c Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 2 Mar 2020 13:06:16 -0500 Subject: [PATCH 1681/3965] fix EOF handling in Read --- p2p/security/noise/rw.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/security/noise/rw.go b/p2p/security/noise/rw.go index fa39cff4b7..3d324346a5 100644 --- a/p2p/security/noise/rw.go +++ b/p2p/security/noise/rw.go @@ -105,7 +105,7 @@ func (s *secureSession) readMsgInsecure() ([]byte, error) { buf := make([]byte, 2) _, err := io.ReadFull(s.insecure, buf) if err != nil { - return nil, fmt.Errorf("error reading length prefix: %s", err) + return nil, err } size := int(binary.BigEndian.Uint16(buf)) buf = make([]byte, size) From 5d2c4148f67b02364fa5d2b2967dfeb362303349 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 2 Mar 2020 11:58:47 -0500 Subject: [PATCH 1682/3965] add benchmark tests --- p2p/security/noise/benchmark_test.go | 174 +++++++++++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 p2p/security/noise/benchmark_test.go diff --git a/p2p/security/noise/benchmark_test.go b/p2p/security/noise/benchmark_test.go new file mode 100644 index 0000000000..180471f646 --- /dev/null +++ b/p2p/security/noise/benchmark_test.go @@ -0,0 +1,174 @@ +package noise + +import ( + "context" + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/sec" + "io" + "io/ioutil" + "math/rand" + "net" + "testing" + "time" +) + +func makeTransport(b *testing.B) *Transport { + b.Helper() + + priv, _, err := crypto.GenerateKeyPair(crypto.Ed25519, 256) + if err != nil { + b.Fatal(err) + } + tpt, err := New(priv) + if err != nil { + b.Fatalf("error constructing transport: %v", err) + } + return tpt +} + +type benchenv struct { + *testing.B + + initTpt *Transport + respTpt *Transport + rndSrc rand.Source +} + +func setupEnv(b *testing.B) *benchenv { + b.StopTimer() + defer b.StartTimer() + initTpt := makeTransport(b) + respTpt := makeTransport(b) + + return &benchenv{ + B: b, + initTpt: initTpt, + respTpt: respTpt, + rndSrc: rand.NewSource(42), + } +} + +func (b benchenv) connect(stopTimer bool) (*secureSession, *secureSession) { + initConn, respConn := net.Pipe() + + if stopTimer { + b.StopTimer() + defer b.StartTimer() + } + + var initSession sec.SecureConn + var initErr error + done := make(chan struct{}) + go func() { + defer close(done) + initSession, initErr = b.initTpt.SecureOutbound(context.TODO(), initConn, b.respTpt.localID) + }() + + respSession, respErr := b.respTpt.SecureInbound(context.TODO(), respConn) + <-done + + if initErr != nil { + b.Fatal(initErr) + } + + if respErr != nil { + b.Fatal(respErr) + } + + return initSession.(*secureSession), respSession.(*secureSession) +} + +func drain(r io.Reader, done chan<- error) { + _, err := io.Copy(ioutil.Discard, r) + done <- err +} + +func sink(dst io.WriteCloser, src io.Reader, done chan<- error) { + _, err := io.Copy(dst, src) + if err != nil { + done <- err + } + done <- dst.Close() +} + +func pipeRandom(src rand.Source, w io.WriteCloser, r io.Reader, n int64) error { + rnd := rand.New(src) + lr := io.LimitReader(rnd, n) + + writeCh := make(chan error, 1) + readCh := make(chan error, 1) + + go sink(w, lr, writeCh) + go drain(r, readCh) + + writeDone := false + readDone := false + for !(readDone && writeDone) { + select { + case err := <-readCh: + if err != nil && err != io.EOF { + return err + } + readDone = true + case err := <-writeCh: + if err != nil && err != io.EOF { + return err + } + writeDone = true + } + } + + return nil +} + +func benchDataTransfer(b *benchenv, size int64) { + var totalBytes int64 + var totalTime time.Duration + for i := 0; i < b.N; i++ { + initSession, respSession := b.connect(true) + + start := time.Now() + err := pipeRandom(b.rndSrc, initSession, respSession, size) + if err != nil { + b.Fatalf("error sending random data: %s", err) + } + elapsed := time.Since(start) + totalTime += elapsed + totalBytes += size + } + bytesPerSec := float64(totalBytes) / totalTime.Seconds() + b.ReportMetric(bytesPerSec, "bytes/sec") +} + +func BenchmarkTransfer1MB(b *testing.B) { + benchDataTransfer(setupEnv(b), 1024*1024) +} + +func BenchmarkTransfer100MB(b *testing.B) { + benchDataTransfer(setupEnv(b), 1024*1024*100) +} + +func BenchmarkTransfer500Mb(b *testing.B) { + benchDataTransfer(setupEnv(b), 1024*1024*500) +} + +func (b benchenv) benchHandshake() { + for i := 0; i < b.N; i++ { + i, r := b.connect(false) + b.StopTimer() + err := i.Close() + if err != nil { + b.Errorf("error closing session: %s", err) + } + err = r.Close() + if err != nil { + b.Errorf("error closing session: %s", err) + } + b.StartTimer() + } +} + +func BenchmarkHandshakeXX(b *testing.B) { + env := setupEnv(b) + env.benchHandshake() +} From b152f9d6287833a4cd08aea22fd5e16e562b17f0 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 2 Mar 2020 11:43:29 -0500 Subject: [PATCH 1683/3965] replace noise explorer with flynn/noise --- p2p/security/noise/crypto.go | 31 +- p2p/security/noise/handshake.go | 119 ++++--- p2p/security/noise/session.go | 15 +- p2p/security/noise/xx/XX.noise.go | 558 ------------------------------ p2p/security/noise/xx/XX_test.go | 193 ----------- 5 files changed, 79 insertions(+), 837 deletions(-) delete mode 100644 p2p/security/noise/xx/XX.noise.go delete mode 100644 p2p/security/noise/xx/XX_test.go diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index e837c7adad..6efec72e7a 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -1,43 +1,22 @@ package noise -import ( - "errors" - xx "github.com/libp2p/go-libp2p-noise/xx" -) +import "errors" func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) { if !s.handshakeComplete { return nil, errors.New("encrypt err: haven't completed handshake") } - if s.initiator { - cs := s.ns.CS1() - _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) - } else { - cs := s.ns.CS2() - _, ciphertext = xx.EncryptWithAd(cs, nil, plaintext) - } - + // TODO: use pre-allocated buffers + ciphertext = s.enc.Encrypt(nil, nil, plaintext) return ciphertext, nil } func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) { - var ok bool if !s.handshakeComplete { return nil, errors.New("decrypt err: haven't completed handshake") } - if s.initiator { - cs := s.ns.CS2() - _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) - } else { - cs := s.ns.CS1() - _, plaintext, ok = xx.DecryptWithAd(cs, nil, ciphertext) - } - - if !ok { - return nil, errors.New("decrypt err: could not decrypt") - } - - return plaintext, nil + // TODO: use pre-allocated buffers + return s.dec.Decrypt(nil, nil, ciphertext) } diff --git a/p2p/security/noise/handshake.go b/p2p/security/noise/handshake.go index 7fcd8570b5..1e3fe1543a 100644 --- a/p2p/security/noise/handshake.go +++ b/p2p/security/noise/handshake.go @@ -4,17 +4,21 @@ import ( "context" "fmt" "github.com/gogo/protobuf/proto" + + "github.com/flynn/noise" + "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-noise/pb" - "github.com/libp2p/go-libp2p-noise/xx" ) // payloadSigPrefix is prepended to our Noise static key before signing with // our libp2p identity key. const payloadSigPrefix = "noise-libp2p-static-key:" +var cipherSuite = noise.NewCipherSuite(noise.DH25519, noise.CipherChaChaPoly, noise.HashSHA256) + func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { s.remote.libp2pKey, err = crypto.UnmarshalPublicKey(key) return err @@ -25,7 +29,7 @@ func (s *secureSession) setRemotePeerID(key crypto.PubKey) (err error) { return err } -func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKey [32]byte) (err error) { +func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKey []byte) (err error) { sig := payload.GetIdentitySig() msg := append([]byte(payloadSigPrefix), noiseKey[:]...) @@ -39,47 +43,54 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe return nil } -func (s *secureSession) sendHandshakeMessage(payload []byte, initialStage bool) error { - var msgbuf xx.MessageBuffer - s.ns, msgbuf = xx.SendMessage(s.ns, payload) - var encMsgBuf []byte - if initialStage { - encMsgBuf = msgbuf.Encode0() +func (s *secureSession) completeHandshake(cs1, cs2 *noise.CipherState) { + if s.initiator { + s.enc = cs1 + s.dec = cs2 } else { - encMsgBuf = msgbuf.Encode1() + s.enc = cs2 + s.dec = cs1 } + s.handshakeComplete = true +} - err := s.writeMsgInsecure(encMsgBuf) +func (s *secureSession) sendHandshakeMessage(payload []byte) error { + buf, cs1, cs2, err := s.hs.WriteMessage(nil, payload) if err != nil { - return fmt.Errorf("sendHandshakeMessage write to conn err=%s", err) + return err } - return nil -} - -func (s *secureSession) recvHandshakeMessage(initialStage bool) (buf []byte, plaintext []byte, valid bool, err error) { - buf, err = s.readMsgInsecure() + err = s.writeMsgInsecure(buf) if err != nil { - return nil, nil, false, fmt.Errorf("recvHandshakeMessage read length err=%s", err) + return err } - var msgbuf *xx.MessageBuffer - if initialStage { - msgbuf, err = xx.Decode0(buf) - } else { - msgbuf, err = xx.Decode1(buf) + if cs1 != nil && cs2 != nil { + s.completeHandshake(cs1, cs2) } + return nil +} +func (s *secureSession) readHandshakeMessage() ([]byte, error) { + raw, err := s.readMsgInsecure() if err != nil { - return buf, nil, false, fmt.Errorf("recvHandshakeMessage decode msg err=%s", err) + return nil, err } - - s.ns, plaintext, valid = xx.RecvMessage(s.ns, msgbuf) - if !valid { - return buf, nil, false, fmt.Errorf("recvHandshakeMessage validation fail") + msg, cs1, cs2, err := s.hs.ReadMessage(nil, raw) + if err != nil { + return nil, err + } + if cs1 != nil && cs2 != nil { + s.completeHandshake(cs1, cs2) } + return msg, nil +} - return buf, plaintext, valid, nil +func keypairToDH(kp Keypair) noise.DHKey { + return noise.DHKey{ + Private: kp.privateKey[:], + Public: kp.publicKey[:], + } } // Runs the XX handshake @@ -89,6 +100,21 @@ func (s *secureSession) recvHandshakeMessage(initialStage bool) (buf []byte, pla // -> s, se func (s *secureSession) runHandshake(ctx context.Context) (err error) { + cfg := noise.Config{ + CipherSuite: cipherSuite, + Pattern: noise.HandshakeXX, + Initiator: s.initiator, + Prologue: s.prologue, + StaticKeypair: keypairToDH(*s.noiseKeypair), + EphemeralKeypair: noise.DHKey{}, + } + + hs, err := noise.NewHandshakeState(cfg) + if err != nil { + return fmt.Errorf("runHandshake err initializing handshake state: %s", err) + } + s.hs = hs + // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() if err != nil { @@ -111,15 +137,10 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { return fmt.Errorf("runHandshake proto marshal payload err=%s", err) } - kp := xx.NewKeypair(s.noiseKeypair.publicKey, s.noiseKeypair.privateKey) - - // new XX noise session - s.ns = xx.InitSession(s.initiator, s.prologue, kp, [32]byte{}) - if s.initiator { // stage 0 // - err = s.sendHandshakeMessage(nil, true) + err = s.sendHandshakeMessage(nil) if err != nil { return fmt.Errorf("runHandshake stage 0 initiator fail: %s", err) } @@ -127,19 +148,17 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { // stage 1 // var plaintext []byte - var valid bool // read reply - _, plaintext, valid, err = s.recvHandshakeMessage(false) + plaintext, err = s.readHandshakeMessage() if err != nil { return fmt.Errorf("runHandshake initiator stage 1 fail: %s", err) } - if !valid { - return fmt.Errorf("runHandshake stage 1 initiator validation fail") - } + + s.remote.noiseKey = s.hs.PeerStatic() // stage 2 // - err = s.sendHandshakeMessage(payloadEnc, false) + err = s.sendHandshakeMessage(payloadEnc) if err != nil { return fmt.Errorf("runHandshake stage=2 initiator=true err=%s", err) } @@ -166,7 +185,7 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { } // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ns.RemoteKey()) + err = s.verifyPayload(nhp, s.remote.noiseKey) if err != nil { return fmt.Errorf("runHandshake stage=2 initiator=true verify payload err=%s", err) } @@ -176,22 +195,17 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { // stage 0 // var plaintext []byte - var valid bool nhp := new(pb.NoiseHandshakePayload) // read message - _, plaintext, valid, err = s.recvHandshakeMessage(true) + plaintext, err = s.readHandshakeMessage() if err != nil { return fmt.Errorf("runHandshake stage=0 initiator=false err=%s", err) } - if !valid { - return fmt.Errorf("runHandshake stage=0 initiator=false err=validation fail") - } - // stage 1 // - err = s.sendHandshakeMessage(payloadEnc, false) + err = s.sendHandshakeMessage(payloadEnc) if err != nil { return fmt.Errorf("runHandshake stage=1 initiator=false err=%s", err) } @@ -199,15 +213,11 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { // stage 2 // // read message - _, plaintext, valid, err = s.recvHandshakeMessage(false) + plaintext, err = s.readHandshakeMessage() if err != nil { return fmt.Errorf("runHandshake stage=2 initiator=false err=%s", err) } - if !valid { - return fmt.Errorf("runHandshake stage=2 initiator=false err=validation fail") - } - // unmarshal payload err = proto.Unmarshal(plaintext, nhp) if err != nil { @@ -226,7 +236,7 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { return fmt.Errorf("runHandshake stage=2 initiator=false set remote peer id err=%s", err) } - s.remote.noiseKey = s.ns.RemoteKey() + s.remote.noiseKey = s.hs.PeerStatic() // verify payload is signed by libp2p key err = s.verifyPayload(nhp, s.remote.noiseKey) @@ -235,6 +245,5 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { } } - s.handshakeComplete = true return nil } diff --git a/p2p/security/noise/session.go b/p2p/security/noise/session.go index 0afb27f635..1aea130d70 100644 --- a/p2p/security/noise/session.go +++ b/p2p/security/noise/session.go @@ -7,11 +7,11 @@ import ( "sync" "time" + "github.com/flynn/noise" logging "github.com/ipfs/go-log" + "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" - - "github.com/libp2p/go-libp2p-noise/xx" ) var log = logging.Logger("noise") @@ -31,7 +31,9 @@ type secureSession struct { local peerInfo remote peerInfo - ns *xx.NoiseSession + hs *noise.HandshakeState + enc *noise.CipherState + dec *noise.CipherState handshakeComplete bool noiseKeypair *Keypair @@ -42,7 +44,7 @@ type secureSession struct { } type peerInfo struct { - noiseKey [32]byte // static noise public key + noiseKey []byte // static noise public key libp2pKey crypto.PubKey } @@ -54,7 +56,7 @@ func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, re } localPeerInfo := peerInfo{ - noiseKey: tpt.noiseKeypair.publicKey, + noiseKey: tpt.noiseKeypair.publicKey[:], libp2pKey: tpt.privateKey.GetPublic(), } @@ -71,6 +73,9 @@ func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, re } err := s.runHandshake(ctx) + if err != nil { + _ = s.insecure.Close() + } return s, err } diff --git a/p2p/security/noise/xx/XX.noise.go b/p2p/security/noise/xx/XX.noise.go deleted file mode 100644 index f046eae1c0..0000000000 --- a/p2p/security/noise/xx/XX.noise.go +++ /dev/null @@ -1,558 +0,0 @@ -/* -XX: - -> e - <- e, ee, s, es - -> s, se -*/ - -// Implementation Version: 1.0.0 - -/* ---------------------------------------------------------------- * - * PARAMETERS * - * ---------------------------------------------------------------- */ - -package xx - -import ( - "crypto/rand" - "crypto/sha256" - "crypto/subtle" - - "encoding/binary" - "errors" - "golang.org/x/crypto/chacha20poly1305" - "golang.org/x/crypto/curve25519" - "golang.org/x/crypto/hkdf" - "hash" - "io" - "math" -) - -/* ---------------------------------------------------------------- * - * TYPES * - * ---------------------------------------------------------------- */ - -type Keypair struct { - public_key [32]byte - private_key [32]byte -} - -type MessageBuffer struct { - ne [32]byte - ns []byte - ciphertext []byte -} - -type cipherstate struct { - k [32]byte - n uint32 -} - -type symmetricstate struct { - cs cipherstate - ck [32]byte - h [32]byte -} - -type handshakestate struct { - ss symmetricstate - s Keypair - e Keypair - rs [32]byte - re [32]byte - psk [32]byte -} - -type NoiseSession struct { - hs handshakestate - h [32]byte - cs1 cipherstate - cs2 cipherstate - mc uint64 - i bool -} - -func (ns *NoiseSession) CS1() *cipherstate { - return &ns.cs1 -} - -func (ns *NoiseSession) CS2() *cipherstate { - return &ns.cs2 -} - -func (ns *NoiseSession) Ephemeral() *Keypair { - return &ns.hs.e -} - -func NewKeypair(pub [32]byte, priv [32]byte) Keypair { - return Keypair{ - public_key: pub, - private_key: priv, - } -} - -func (kp Keypair) PubKey() [32]byte { - return kp.public_key -} - -func (kp Keypair) PrivKey() [32]byte { - return kp.private_key -} - -func (ns *NoiseSession) RemoteKey() [32]byte { - return ns.hs.rs -} - -func NewMessageBuffer(ne [32]byte, ns []byte, ciphertext []byte) MessageBuffer { - return MessageBuffer{ - ne: ne, - ns: ns, - ciphertext: ciphertext, - } -} - -/* ---------------------------------------------------------------- * - * CONSTANTS * - * ---------------------------------------------------------------- */ - -var emptyKey = [32]byte{ - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, -} - -var minNonce = uint32(0) - -/* ---------------------------------------------------------------- * - * UTILITY FUNCTIONS * - * ---------------------------------------------------------------- */ - -func getPublicKey(kp *Keypair) [32]byte { - return kp.public_key -} - -func isEmptyKey(k [32]byte) bool { - return subtle.ConstantTimeCompare(k[:], emptyKey[:]) == 1 -} - -func (mb *MessageBuffer) NE() [32]byte { - return mb.ne -} - -func (mb *MessageBuffer) NS() []byte { - return mb.ns -} - -func (mb *MessageBuffer) Ciphertext() []byte { - return mb.ciphertext -} - -// Encodes a MessageBuffer from stage 0 -func (mb *MessageBuffer) Encode0() []byte { - enc := []byte{} - - enc = append(enc, mb.ne[:]...) - enc = append(enc, mb.ciphertext...) - - return enc -} - -// Encodes a MessageBuffer from stage 1 and 2 -func (mb *MessageBuffer) Encode1() []byte { - enc := []byte{} - - enc = append(enc, mb.ne[:]...) - enc = append(enc, mb.ns...) - enc = append(enc, mb.ciphertext...) - - return enc -} - -// Decodes initial message (stage 0) into MessageBuffer -func Decode0(in []byte) (*MessageBuffer, error) { - if len(in) < 32 { - return nil, errors.New("cannot decode stage 0 MessageBuffer: length less than 32 bytes") - } - - mb := new(MessageBuffer) - copy(mb.ne[:], in[:32]) - mb.ciphertext = in[32:] - - return mb, nil -} - -// Decodes messages at stage 1 or 2 into MessageBuffer -func Decode1(in []byte) (*MessageBuffer, error) { - if len(in) < 80 { - return nil, errors.New("cannot decode stage 1/2 MessageBuffer: length less than 96 bytes") - } - - mb := new(MessageBuffer) - copy(mb.ne[:], in[:32]) - mb.ns = in[32:80] - mb.ciphertext = in[80:] - - return mb, nil -} - -func validatePublicKey(k []byte) bool { - forbiddenCurveValues := [12][]byte{ - {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, - {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, - {224, 235, 122, 124, 59, 65, 184, 174, 22, 86, 227, 250, 241, 159, 196, 106, 218, 9, 141, 235, 156, 50, 177, 253, 134, 98, 5, 22, 95, 73, 184, 0}, - {95, 156, 149, 188, 163, 80, 140, 36, 177, 208, 177, 85, 156, 131, 239, 91, 4, 68, 92, 196, 88, 28, 142, 134, 216, 34, 78, 221, 208, 159, 17, 87}, - {236, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, - {237, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, - {238, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127}, - {205, 235, 122, 124, 59, 65, 184, 174, 22, 86, 227, 250, 241, 159, 196, 106, 218, 9, 141, 235, 156, 50, 177, 253, 134, 98, 5, 22, 95, 73, 184, 128}, - {76, 156, 149, 188, 163, 80, 140, 36, 177, 208, 177, 85, 156, 131, 239, 91, 4, 68, 92, 196, 88, 28, 142, 134, 216, 34, 78, 221, 208, 159, 17, 215}, - {217, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}, - {218, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}, - {219, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 25}, - } - - for _, testValue := range forbiddenCurveValues { - if subtle.ConstantTimeCompare(k[:], testValue[:]) == 1 { - panic("Invalid public key") - } - } - return true -} - -/* ---------------------------------------------------------------- * - * PRIMITIVES * - * ---------------------------------------------------------------- */ - -func incrementNonce(n uint32) uint32 { - return n + 1 -} - -func dh(private_key [32]byte, public_key [32]byte) [32]byte { - var ss [32]byte - curve25519.ScalarMult(&ss, &private_key, &public_key) - return ss -} - -func GenerateKeypair() Keypair { - var public_key [32]byte - var private_key [32]byte - _, _ = rand.Read(private_key[:]) - curve25519.ScalarBaseMult(&public_key, &private_key) - if validatePublicKey(public_key[:]) { - return Keypair{public_key, private_key} - } - return GenerateKeypair() -} - -func GeneratePublicKey(private_key [32]byte) [32]byte { - var public_key [32]byte - curve25519.ScalarBaseMult(&public_key, &private_key) - return public_key -} - -func encrypt(k [32]byte, n uint32, ad []byte, plaintext []byte) []byte { - var nonce [12]byte - var ciphertext []byte - enc, _ := chacha20poly1305.New(k[:]) - binary.LittleEndian.PutUint32(nonce[4:], n) - ciphertext = enc.Seal(nil, nonce[:], plaintext, ad) - return ciphertext -} - -func decrypt(k [32]byte, n uint32, ad []byte, ciphertext []byte) (bool, []byte, []byte) { - var nonce [12]byte - var plaintext []byte - enc, err := chacha20poly1305.New(k[:]) - binary.LittleEndian.PutUint32(nonce[4:], n) - plaintext, err = enc.Open(nil, nonce[:], ciphertext, ad) - return (err == nil), ad, plaintext -} - -func getHash(a []byte, b []byte) [32]byte { - //return blake2s.Sum256(append(a, b...)) - return sha256.Sum256(append(a, b...)) -} - -func hashProtocolName(protocolName []byte) [32]byte { - var h [32]byte - if len(protocolName) <= 32 { - copy(h[:], protocolName) - } else { - h = getHash(protocolName, []byte{}) - } - return h -} - -func sha256HkdfInterface() hash.Hash { - //h, _ := blake2s.New256([]byte{}) - h := sha256.New() - return h -} - -func getHkdf(ck [32]byte, ikm []byte) ([32]byte, [32]byte, [32]byte) { - var k1 [32]byte - var k2 [32]byte - var k3 [32]byte - output := hkdf.New(sha256HkdfInterface, ikm[:], ck[:], []byte{}) - io.ReadFull(output, k1[:]) - io.ReadFull(output, k2[:]) - io.ReadFull(output, k3[:]) - return k1, k2, k3 -} - -/* ---------------------------------------------------------------- * - * STATE MANAGEMENT * - * ---------------------------------------------------------------- */ - -/* CipherState */ -func initializeKey(k [32]byte) cipherstate { - return cipherstate{k, minNonce} -} - -func hasKey(cs *cipherstate) bool { - return !isEmptyKey(cs.k) -} - -func setNonce(cs *cipherstate, newNonce uint32) *cipherstate { - cs.n = newNonce - return cs -} - -func EncryptWithAd(cs *cipherstate, ad []byte, plaintext []byte) (*cipherstate, []byte) { - e := encrypt(cs.k, cs.n, ad, plaintext) - cs = setNonce(cs, incrementNonce(cs.n)) - return cs, e -} - -func DecryptWithAd(cs *cipherstate, ad []byte, ciphertext []byte) (*cipherstate, []byte, bool) { - valid, ad, plaintext := decrypt(cs.k, cs.n, ad, ciphertext) - cs = setNonce(cs, incrementNonce(cs.n)) - return cs, plaintext, valid -} - -func reKey(cs *cipherstate) *cipherstate { - e := encrypt(cs.k, math.MaxUint32, []byte{}, emptyKey[:]) - copy(cs.k[:], e) - return cs -} - -/* SymmetricState */ - -func initializeSymmetric(protocolName []byte) symmetricstate { - h := hashProtocolName(protocolName) - ck := h - cs := initializeKey(emptyKey) - return symmetricstate{cs, ck, h} -} - -func mixKey(ss *symmetricstate, ikm [32]byte) *symmetricstate { - ck, tempK, _ := getHkdf(ss.ck, ikm[:]) - ss.cs = initializeKey(tempK) - ss.ck = ck - return ss -} - -func mixHash(ss *symmetricstate, data []byte) *symmetricstate { - ss.h = getHash(ss.h[:], data) - return ss -} - -func mixKeyAndHash(ss *symmetricstate, ikm [32]byte) *symmetricstate { - var tempH [32]byte - var tempK [32]byte - ss.ck, tempH, tempK = getHkdf(ss.ck, ikm[:]) - ss = mixHash(ss, tempH[:]) - ss.cs = initializeKey(tempK) - return ss -} - -func getHandshakeHash(ss *symmetricstate) [32]byte { - return ss.h -} - -func encryptAndHash(ss *symmetricstate, plaintext []byte) (*symmetricstate, []byte) { - var ciphertext []byte - if hasKey(&ss.cs) { - _, ciphertext = EncryptWithAd(&ss.cs, ss.h[:], plaintext) - } else { - ciphertext = plaintext - } - ss = mixHash(ss, ciphertext) - return ss, ciphertext -} - -func decryptAndHash(ss *symmetricstate, ciphertext []byte) (*symmetricstate, []byte, bool) { - var plaintext []byte - var valid bool - if hasKey(&ss.cs) { - _, plaintext, valid = DecryptWithAd(&ss.cs, ss.h[:], ciphertext) - } else { - plaintext, valid = ciphertext, true - } - ss = mixHash(ss, ciphertext) - return ss, plaintext, valid -} - -func split(ss *symmetricstate) (cipherstate, cipherstate) { - tempK1, tempK2, _ := getHkdf(ss.ck, []byte{}) - cs1 := initializeKey(tempK1) - cs2 := initializeKey(tempK2) - return cs1, cs2 -} - -/* HandshakeState */ - -func initializeInitiator(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) handshakestate { - var ss symmetricstate - var e Keypair - var re [32]byte - name := []byte("Noise_XX_25519_ChaChaPoly_SHA256") - ss = initializeSymmetric(name) - mixHash(&ss, prologue) - return handshakestate{ss, s, e, rs, re, psk} -} - -func initializeResponder(prologue []byte, s Keypair, rs [32]byte, psk [32]byte) handshakestate { - var ss symmetricstate - var e Keypair - var re [32]byte - name := []byte("Noise_XX_25519_ChaChaPoly_SHA256") - ss = initializeSymmetric(name) - mixHash(&ss, prologue) - return handshakestate{ss, s, e, rs, re, psk} -} - -func writeMessageA(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { - ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - - hs.e = GenerateKeypair() - ne = hs.e.public_key - mixHash(&hs.ss, ne[:]) - /* No PSK, so skipping mixKey */ - _, ciphertext = encryptAndHash(&hs.ss, payload) - messageBuffer := MessageBuffer{ne, ns, ciphertext} - return hs, messageBuffer -} - -func writeMessageB(hs *handshakestate, payload []byte) (*handshakestate, MessageBuffer) { - ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - hs.e = GenerateKeypair() - ne = hs.e.public_key - mixHash(&hs.ss, ne[:]) - /* No PSK, so skipping mixKey */ - mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) - spk := make([]byte, len(hs.s.public_key)) - copy(spk[:], hs.s.public_key[:]) - _, ns = encryptAndHash(&hs.ss, spk) - mixKey(&hs.ss, dh(hs.s.private_key, hs.re)) - _, ciphertext = encryptAndHash(&hs.ss, payload) - messageBuffer := MessageBuffer{ne, ns, ciphertext} - return hs, messageBuffer -} - -func writeMessageC(hs *handshakestate, payload []byte) ([32]byte, MessageBuffer, cipherstate, cipherstate) { - ne, ns, ciphertext := emptyKey, []byte{}, []byte{} - spk := make([]byte, len(hs.s.public_key)) - copy(spk[:], hs.s.public_key[:]) - _, ns = encryptAndHash(&hs.ss, spk) - mixKey(&hs.ss, dh(hs.s.private_key, hs.re)) - _, ciphertext = encryptAndHash(&hs.ss, payload) - messageBuffer := MessageBuffer{ne, ns, ciphertext} - cs1, cs2 := split(&hs.ss) - return hs.ss.h, messageBuffer, cs1, cs2 -} - -func readMessageA(hs *handshakestate, message *MessageBuffer) (*handshakestate, []byte, bool) { - valid1 := true - if validatePublicKey(message.ne[:]) { - hs.re = message.ne - } - mixHash(&hs.ss, hs.re[:]) - /* No PSK, so skipping mixKey */ - _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) - return hs, plaintext, (valid1 && valid2) -} - -func readMessageB(hs *handshakestate, message *MessageBuffer) (*handshakestate, []byte, bool) { - valid1 := true - if validatePublicKey(message.ne[:]) { - hs.re = message.ne - } - mixHash(&hs.ss, hs.re[:]) - /* No PSK, so skipping mixKey */ - mixKey(&hs.ss, dh(hs.e.private_key, hs.re)) - _, ns, valid1 := decryptAndHash(&hs.ss, message.ns) - if valid1 && len(ns) == 32 && validatePublicKey(message.ns[:]) { - copy(hs.rs[:], ns) - } - mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) - _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) - return hs, plaintext, (valid1 && valid2) -} - -func readMessageC(hs *handshakestate, message *MessageBuffer) ([32]byte, []byte, bool, cipherstate, cipherstate) { - valid1 := true - _, ns, valid1 := decryptAndHash(&hs.ss, message.ns) - if valid1 && len(ns) == 32 && validatePublicKey(message.ns[:]) { - copy(hs.rs[:], ns) - } - mixKey(&hs.ss, dh(hs.e.private_key, hs.rs)) - _, plaintext, valid2 := decryptAndHash(&hs.ss, message.ciphertext) - cs1, cs2 := split(&hs.ss) - return hs.ss.h, plaintext, (valid1 && valid2), cs1, cs2 -} - -/* ---------------------------------------------------------------- * - * PROCESSES * - * ---------------------------------------------------------------- */ - -func InitSession(initiator bool, prologue []byte, s Keypair, rs [32]byte) *NoiseSession { - var session NoiseSession - psk := emptyKey - if initiator { - session.hs = initializeInitiator(prologue, s, rs, psk) - } else { - session.hs = initializeResponder(prologue, s, rs, psk) - } - session.i = initiator - session.mc = 0 - return &session -} - -func SendMessage(session *NoiseSession, message []byte) (*NoiseSession, MessageBuffer) { - var messageBuffer MessageBuffer - if session.mc == 0 { - _, messageBuffer = writeMessageA(&session.hs, message) - } - if session.mc == 1 { - _, messageBuffer = writeMessageB(&session.hs, message) - } - if session.mc == 2 { - session.h, messageBuffer, session.cs1, session.cs2 = writeMessageC(&session.hs, message) - //session.hs = handshakestate{} - } - session.mc = session.mc + 1 - return session, messageBuffer -} - -func RecvMessage(session *NoiseSession, message *MessageBuffer) (*NoiseSession, []byte, bool) { - var plaintext []byte - var valid bool - if session.mc == 0 { - _, plaintext, valid = readMessageA(&session.hs, message) - } - if session.mc == 1 { - _, plaintext, valid = readMessageB(&session.hs, message) - } - if session.mc == 2 { - session.h, plaintext, valid, session.cs1, session.cs2 = readMessageC(&session.hs, message) - //session.hs = handshakestate{} - } - session.mc = session.mc + 1 - return session, plaintext, valid -} diff --git a/p2p/security/noise/xx/XX_test.go b/p2p/security/noise/xx/XX_test.go deleted file mode 100644 index 63f6d71cce..0000000000 --- a/p2p/security/noise/xx/XX_test.go +++ /dev/null @@ -1,193 +0,0 @@ -package xx - -import ( - "crypto/rand" - "encoding/hex" - proto "github.com/gogo/protobuf/proto" - "github.com/libp2p/go-libp2p-core/crypto" - pb "github.com/libp2p/go-libp2p-noise/pb" - "testing" -) - -func TestGetHkdf(t *testing.T) { - ck := [32]byte{} - ckBytes, err := hex.DecodeString("4e6f6973655f58585f32353531395f58436861436861506f6c795f53484132353600000000000000000000000000000000000000000000000000000000000000") - if err != nil { - t.Fatal(err) - } - copy(ck[:], ckBytes) - - ikm, err := hex.DecodeString("a3eae50ea37a47e8a7aa0c7cd8e16528670536dcd538cebfd724fb68ce44f1910ad898860666227d4e8dd50d22a9a64d1c0a6f47ace092510161e9e442953da3") - if err != nil { - t.Fatal(err) - } - - a, b, c := getHkdf(ck, ikm) - t.Logf("%x", a) - t.Logf("%x", b) - t.Logf("%x", c) -} - -func doHandshake(t *testing.T) (*NoiseSession, *NoiseSession) { - // generate local static noise key - kp_init := GenerateKeypair() - kp_resp := GenerateKeypair() - - payload_string := []byte("noise-libp2p-static-key:") - prologue := []byte("/noise/0.0.0") - - // initiator setup - init_pub := kp_init.PubKey() - libp2p_priv_init, libp2p_pub_init, err := crypto.GenerateEd25519Key(rand.Reader) - if err != nil { - t.Fatal(err) - } - libp2p_pub_init_raw, err := libp2p_pub_init.Raw() - if err != nil { - t.Fatal(err) - } - libp2p_init_signed_payload, err := libp2p_priv_init.Sign(append(payload_string, init_pub[:]...)) - if err != nil { - t.Fatal(err) - } - - // respoonder setup - resp_pub := kp_resp.PubKey() - libp2p_priv_resp, libp2p_pub_resp, err := crypto.GenerateEd25519Key(rand.Reader) - if err != nil { - t.Fatal(err) - } - libp2p_pub_resp_raw, err := libp2p_pub_resp.Raw() - if err != nil { - t.Fatal(err) - } - libp2p_resp_signed_payload, err := libp2p_priv_resp.Sign(append(payload_string, resp_pub[:]...)) - if err != nil { - t.Fatal(err) - } - - // initiator: new XX noise session - ns_init := InitSession(true, prologue, kp_init, kp_resp.PubKey()) - - // responder: new XX noise session - ns_resp := InitSession(false, prologue, kp_resp, kp_init.PubKey()) - - // stage 0: initiator - // create payload - payload_init := new(pb.NoiseHandshakePayload) - payload_init.IdentityKey = libp2p_pub_init_raw - payload_init.IdentitySig = libp2p_init_signed_payload - payload_init_enc, err := proto.Marshal(payload_init) - if err != nil { - t.Fatalf("proto marshal payload fail: %s", err) - } - - // send message - var msgbuf MessageBuffer - msg := []byte{} - msg = append(msg, payload_init_enc[:]...) - ns_init, msgbuf = SendMessage(ns_init, msg) - - t.Logf("stage 0 msgbuf: %v", msgbuf) - t.Logf("stage 0 msgbuf ne len: %d", len(msgbuf.NE())) - - // stage 0: responder - var plaintext []byte - var valid bool - ns_resp, plaintext, valid = RecvMessage(ns_resp, &msgbuf) - if !valid { - t.Fatalf("stage 0 receive not valid") - } - - t.Logf("stage 0 resp payload: %x", plaintext) - - // stage 1: responder - // create payload - payload_resp := new(pb.NoiseHandshakePayload) - payload_resp.IdentityKey = libp2p_pub_resp_raw - payload_resp.IdentitySig = libp2p_resp_signed_payload - payload_resp_enc, err := proto.Marshal(payload_resp) - if err != nil { - t.Fatalf("proto marshal payload fail: %s", err) - } - msg = append(msg, payload_resp_enc[:]...) - ns_resp, msgbuf = SendMessage(ns_resp, msg) - - t.Logf("stage 1 msgbuf: %v", msgbuf) - t.Logf("stage 1 msgbuf ne len: %d", len(msgbuf.NE())) - t.Logf("stage 1 msgbuf ns len: %d", len(msgbuf.NS())) - - // stage 1: initiator - ns_init, plaintext, valid = RecvMessage(ns_init, &msgbuf) - if !valid { - t.Fatalf("stage 1 receive not valid") - } - - t.Logf("stage 1 resp payload: %x", plaintext) - - // stage 2: initiator - // send message - //msg = append(msg, payload_init_enc[:]...) - ns_init, msgbuf = SendMessage(ns_init, nil) - - t.Logf("stage 2 msgbuf: %v", msgbuf) - t.Logf("stage 2 msgbuf ne len: %d", len(msgbuf.NE())) - t.Logf("stage 2 msgbuf ns len: %d", len(msgbuf.NS())) - - // stage 2: responder - ns_resp, plaintext, valid = RecvMessage(ns_resp, &msgbuf) - if !valid { - t.Fatalf("stage 2 receive not valid") - } - - t.Logf("stage 2 resp payload: %x", plaintext) - - return ns_init, ns_resp -} - -func TestHandshake(t *testing.T) { - _, _ = doHandshake(t) -} - -func TestSymmetricEncryptAndDecrypt(t *testing.T) { - ns_init, ns_resp := doHandshake(t) - cs1_init := ns_init.CS1() - cs1_resp := ns_resp.CS1() - - var out []byte - var ok bool - ad := []byte("authenticated") - msg := []byte("helloworld") - cs1_init, out = EncryptWithAd(cs1_init, ad, msg) - - cs1_resp, out, ok = DecryptWithAd(cs1_resp, ad, out) - if !ok { - t.Fatal("could not decrypt") - } -} - -func TestSymmetricEncryptAndDecryptAgain(t *testing.T) { - ns_init, ns_resp := doHandshake(t) - cs2_init := ns_init.CS2() - cs2_resp := ns_resp.CS2() - - var out []byte - var ok bool - ad := []byte("authenticated") - msg := []byte("helloworld") - cs2_resp, out = EncryptWithAd(cs2_resp, ad, msg) - - cs2_init, out, ok = DecryptWithAd(cs2_init, ad, out) - if !ok { - t.Fatal("could not decrypt") - } - - ad = []byte("authenticatedagain") - msg = []byte("helloworld2") - cs2_resp, out = EncryptWithAd(cs2_resp, ad, msg) - - cs2_init, out, ok = DecryptWithAd(cs2_init, ad, out) - if !ok { - t.Fatal("could not decrypt") - } -} From 6e4c6b8ea0f623a21d6f83fffba1e735c8cc1df5 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 2 Mar 2020 15:19:55 -0500 Subject: [PATCH 1684/3965] move static keygen to session --- p2p/security/noise/session.go | 11 ++++++----- p2p/security/noise/transport.go | 15 ++++----------- p2p/security/noise/transport_test.go | 9 ++------- 3 files changed, 12 insertions(+), 23 deletions(-) diff --git a/p2p/security/noise/session.go b/p2p/security/noise/session.go index 1aea130d70..7e40b551fa 100644 --- a/p2p/security/noise/session.go +++ b/p2p/security/noise/session.go @@ -51,12 +51,13 @@ type peerInfo struct { // newSecureSession creates a noise session over the given insecure Conn, using the static // Noise keypair and libp2p identity keypair from the given Transport. func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, remote peer.ID, initiator bool) (*secureSession, error) { - if tpt.noiseKeypair == nil { - return nil, errNoKeypair + kp, err := GenerateKeypair() + if err != nil { + return nil, err } localPeerInfo := peerInfo{ - noiseKey: tpt.noiseKeypair.publicKey[:], + noiseKey: kp.publicKey[:], libp2pKey: tpt.privateKey.GetPublic(), } @@ -69,10 +70,10 @@ func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, re remotePeer: remote, local: localPeerInfo, msgBuffer: []byte{}, - noiseKeypair: tpt.noiseKeypair, + noiseKeypair: kp, } - err := s.runHandshake(ctx) + err = s.runHandshake(ctx) if err != nil { _ = s.insecure.Close() } diff --git a/p2p/security/noise/transport.go b/p2p/security/noise/transport.go index 9ad3f37743..8bcdf9586a 100644 --- a/p2p/security/noise/transport.go +++ b/p2p/security/noise/transport.go @@ -17,9 +17,8 @@ var _ sec.SecureTransport = &Transport{} // Transport implements the interface sec.SecureTransport // https://godoc.org/github.com/libp2p/go-libp2p-core/sec#SecureConn type Transport struct { - localID peer.ID - privateKey crypto.PrivKey - noiseKeypair *Keypair + localID peer.ID + privateKey crypto.PrivKey } // New creates a new Noise transport using the given private key as its @@ -30,15 +29,9 @@ func New(privkey crypto.PrivKey) (*Transport, error) { return nil, err } - kp, err := GenerateKeypair() - if err != nil { - return nil, err - } - return &Transport{ - localID: localID, - privateKey: privkey, - noiseKeypair: kp, + localID: localID, + privateKey: privkey, }, nil } diff --git a/p2p/security/noise/transport_test.go b/p2p/security/noise/transport_test.go index 4c4b05031a..6e51e2a018 100644 --- a/p2p/security/noise/transport_test.go +++ b/p2p/security/noise/transport_test.go @@ -21,14 +21,9 @@ func newTestTransport(t *testing.T, typ, bits int) *Transport { if err != nil { t.Fatal(err) } - kp, err := GenerateKeypair() - if err != nil { - t.Fatal(err) - } return &Transport{ - localID: id, - privateKey: priv, - noiseKeypair: kp, + localID: id, + privateKey: priv, } } From 70efae2bedc62e5420e317076f91e4050ee8ccf6 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 2 Mar 2020 16:05:03 -0500 Subject: [PATCH 1685/3965] refactor internal session state --- p2p/security/noise/crypto.go | 12 +-- p2p/security/noise/handshake.go | 158 +++++++++++++++----------------- p2p/security/noise/session.go | 78 ++++++---------- 3 files changed, 106 insertions(+), 142 deletions(-) diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index 6efec72e7a..5f4694f085 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -3,20 +3,20 @@ package noise import "errors" func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) { - if !s.handshakeComplete { - return nil, errors.New("encrypt err: haven't completed handshake") + if s.ns.enc == nil { + return nil, errors.New("cannot encrypt, handshake incomplete") } // TODO: use pre-allocated buffers - ciphertext = s.enc.Encrypt(nil, nil, plaintext) + ciphertext = s.ns.enc.Encrypt(nil, nil, plaintext) return ciphertext, nil } func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) { - if !s.handshakeComplete { - return nil, errors.New("decrypt err: haven't completed handshake") + if s.ns.dec == nil { + return nil, errors.New("cannot decrypt, handshake incomplete") } // TODO: use pre-allocated buffers - return s.dec.Decrypt(nil, nil, ciphertext) + return s.ns.dec.Decrypt(nil, nil, ciphertext) } diff --git a/p2p/security/noise/handshake.go b/p2p/security/noise/handshake.go index 1e3fe1543a..682bcf2c47 100644 --- a/p2p/security/noise/handshake.go +++ b/p2p/security/noise/handshake.go @@ -19,19 +19,28 @@ const payloadSigPrefix = "noise-libp2p-static-key:" var cipherSuite = noise.NewCipherSuite(noise.DH25519, noise.CipherChaChaPoly, noise.HashSHA256) -func (s *secureSession) setRemotePeerInfo(key []byte) (err error) { - s.remote.libp2pKey, err = crypto.UnmarshalPublicKey(key) - return err -} +func (s *secureSession) setRemotePeerInfo(keyBytes []byte) (err error) { + key, err := crypto.UnmarshalPublicKey(keyBytes) + if err != nil { + return err + } + id, err := peer.IDFromPublicKey(key) + if err != nil { + return err + } -func (s *secureSession) setRemotePeerID(key crypto.PubKey) (err error) { - s.remotePeer, err = peer.IDFromPublicKey(key) - return err + if s.remoteID != "" && s.remoteID != id { + return fmt.Errorf("peer id mismatch: expected %s, but remote key matches %s", s.remoteID, id) + } + + s.remoteID = id + s.remoteKey = key + return nil } func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKey []byte) (err error) { sig := payload.GetIdentitySig() - msg := append([]byte(payloadSigPrefix), noiseKey[:]...) + msg := append([]byte(payloadSigPrefix), noiseKey...) ok, err := s.RemotePublicKey().Verify(msg, sig) if err != nil { @@ -44,18 +53,17 @@ func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKe } func (s *secureSession) completeHandshake(cs1, cs2 *noise.CipherState) { - if s.initiator { - s.enc = cs1 - s.dec = cs2 + if s.ns.initiator { + s.ns.enc = cs1 + s.ns.dec = cs2 } else { - s.enc = cs2 - s.dec = cs1 + s.ns.enc = cs2 + s.ns.dec = cs1 } - s.handshakeComplete = true } func (s *secureSession) sendHandshakeMessage(payload []byte) error { - buf, cs1, cs2, err := s.hs.WriteMessage(nil, payload) + buf, cs1, cs2, err := s.ns.hs.WriteMessage(nil, payload) if err != nil { return err } @@ -76,7 +84,7 @@ func (s *secureSession) readHandshakeMessage() ([]byte, error) { if err != nil { return nil, err } - msg, cs1, cs2, err := s.hs.ReadMessage(nil, raw) + msg, cs1, cs2, err := s.ns.hs.ReadMessage(nil, raw) if err != nil { return nil, err } @@ -86,11 +94,29 @@ func (s *secureSession) readHandshakeMessage() ([]byte, error) { return msg, nil } -func keypairToDH(kp Keypair) noise.DHKey { - return noise.DHKey{ - Private: kp.privateKey[:], - Public: kp.publicKey[:], +func (s *secureSession) generateHandshakePayload() ([]byte, error) { + // setup libp2p keys + localKeyRaw, err := s.LocalPublicKey().Bytes() + if err != nil { + return nil, fmt.Errorf("error serializing libp2p identity key: %s", err) + } + + // sign noise data for payload + toSign := append([]byte(payloadSigPrefix), s.ns.localStatic.Public...) + signedPayload, err := s.localKey.Sign(toSign) + if err != nil { + return nil, fmt.Errorf("error sigining handshake payload: %s", err) } + + // create payload + payload := new(pb.NoiseHandshakePayload) + payload.IdentityKey = localKeyRaw + payload.IdentitySig = signedPayload + payloadEnc, err := proto.Marshal(payload) + if err != nil { + return nil, fmt.Errorf("error marshaling handshake payload: %s", err) + } + return payloadEnc, nil } // Runs the XX handshake @@ -101,113 +127,81 @@ func keypairToDH(kp Keypair) noise.DHKey { func (s *secureSession) runHandshake(ctx context.Context) (err error) { cfg := noise.Config{ - CipherSuite: cipherSuite, - Pattern: noise.HandshakeXX, - Initiator: s.initiator, - Prologue: s.prologue, - StaticKeypair: keypairToDH(*s.noiseKeypair), - EphemeralKeypair: noise.DHKey{}, + CipherSuite: cipherSuite, + Pattern: noise.HandshakeXX, + Initiator: s.ns.initiator, + StaticKeypair: s.ns.localStatic, } hs, err := noise.NewHandshakeState(cfg) if err != nil { - return fmt.Errorf("runHandshake err initializing handshake state: %s", err) + return fmt.Errorf("error initializing handshake state: %s", err) } - s.hs = hs + s.ns.hs = hs - // setup libp2p keys - localKeyRaw, err := s.LocalPublicKey().Bytes() - if err != nil { - return fmt.Errorf("runHandshake err getting raw pubkey: %s", err) - } - - // sign noise data for payload - noise_pub := s.noiseKeypair.publicKey - signedPayload, err := s.localKey.Sign(append([]byte(payloadSigPrefix), noise_pub[:]...)) - if err != nil { - return fmt.Errorf("runHandshake signing payload err=%s", err) - } - - // create payload - payload := new(pb.NoiseHandshakePayload) - payload.IdentityKey = localKeyRaw - payload.IdentitySig = signedPayload - payloadEnc, err := proto.Marshal(payload) + payload, err := s.generateHandshakePayload() if err != nil { - return fmt.Errorf("runHandshake proto marshal payload err=%s", err) + return err } - if s.initiator { + if s.ns.initiator { // stage 0 // err = s.sendHandshakeMessage(nil) if err != nil { - return fmt.Errorf("runHandshake stage 0 initiator fail: %s", err) + return fmt.Errorf("error sending handshake message: %s", err) } // stage 1 // - var plaintext []byte // read reply - plaintext, err = s.readHandshakeMessage() + plaintext, err := s.readHandshakeMessage() if err != nil { - return fmt.Errorf("runHandshake initiator stage 1 fail: %s", err) + return fmt.Errorf("error reading handshake message: %s", err) } - s.remote.noiseKey = s.hs.PeerStatic() - // stage 2 // - - err = s.sendHandshakeMessage(payloadEnc) + err = s.sendHandshakeMessage(payload) if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=true err=%s", err) + return fmt.Errorf("error sending handshake message: %s", err) } // unmarshal payload nhp := new(pb.NoiseHandshakePayload) err = proto.Unmarshal(plaintext, nhp) if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=true err=cannot unmarshal payload") + return fmt.Errorf("error unmarshaling remote handshake payload: %s", err) } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=true read remote libp2p key fail") - } - - // assert that remote peer ID matches libp2p public key - pid, err := peer.IDFromPublicKey(s.RemotePublicKey()) - if pid != s.remotePeer { - return fmt.Errorf("runHandshake stage=2 initiator=true check remote peer id err: expected %x got %x", s.remotePeer, pid) - } else if err != nil { - return fmt.Errorf("runHandshake stage 2 initiator check remote peer id err %s", err) + return fmt.Errorf("error processing remote libp2p key: %s", err) } // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.remote.noiseKey) + err = s.verifyPayload(nhp, s.ns.hs.PeerStatic()) if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=true verify payload err=%s", err) + return fmt.Errorf("error validating handshake signature: %s", err) } } else { // stage 0 // - var plaintext []byte nhp := new(pb.NoiseHandshakePayload) // read message plaintext, err = s.readHandshakeMessage() if err != nil { - return fmt.Errorf("runHandshake stage=0 initiator=false err=%s", err) + return fmt.Errorf("error reading handshake message: %s", err) } // stage 1 // - err = s.sendHandshakeMessage(payloadEnc) + err = s.sendHandshakeMessage(payload) if err != nil { - return fmt.Errorf("runHandshake stage=1 initiator=false err=%s", err) + return fmt.Errorf("error sending handshake message: %s", err) } // stage 2 // @@ -215,33 +209,25 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { // read message plaintext, err = s.readHandshakeMessage() if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=false err=%s", err) + return fmt.Errorf("error reading handshake message: %s", err) } // unmarshal payload err = proto.Unmarshal(plaintext, nhp) if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=false err=cannot unmarshal payload") + return fmt.Errorf("error unmarshaling remote handshake payload: %s", err) } // set remote libp2p public key err = s.setRemotePeerInfo(nhp.GetIdentityKey()) if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=false read remote libp2p key fail") - } - - // set remote libp2p public key from payload - err = s.setRemotePeerID(s.RemotePublicKey()) - if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=false set remote peer id err=%s", err) + return fmt.Errorf("error processing remote libp2p key: %s", err) } - s.remote.noiseKey = s.hs.PeerStatic() - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.remote.noiseKey) + err = s.verifyPayload(nhp, s.ns.hs.PeerStatic()) if err != nil { - return fmt.Errorf("runHandshake stage=2 initiator=false err=%s", err) + return fmt.Errorf("error validating handshake signature: %s", err) } } diff --git a/p2p/security/noise/session.go b/p2p/security/noise/session.go index 7e40b551fa..8714e00ef6 100644 --- a/p2p/security/noise/session.go +++ b/p2p/security/noise/session.go @@ -2,7 +2,7 @@ package noise import ( "context" - "errors" + "crypto/rand" "net" "sync" "time" @@ -16,61 +16,47 @@ import ( var log = logging.Logger("noise") -var errNoKeypair = errors.New("cannot initiate secureSession - transport has no noise keypair") - -type secureSession struct { - insecure net.Conn - - initiator bool - prologue []byte - - localKey crypto.PrivKey - localPeer peer.ID - remotePeer peer.ID - - local peerInfo - remote peerInfo +type noiseState struct { + initiator bool + localStatic noise.DHKey hs *noise.HandshakeState enc *noise.CipherState dec *noise.CipherState +} + +type secureSession struct { + ns noiseState - handshakeComplete bool - noiseKeypair *Keypair + localID peer.ID + localKey crypto.PrivKey + remoteID peer.ID + remoteKey crypto.PubKey + insecure net.Conn msgBuffer []byte readLock sync.Mutex writeLock sync.Mutex } -type peerInfo struct { - noiseKey []byte // static noise public key - libp2pKey crypto.PubKey -} - -// newSecureSession creates a noise session over the given insecure Conn, using the static -// Noise keypair and libp2p identity keypair from the given Transport. +// newSecureSession creates a noise session over the given insecure Conn, using the +// libp2p identity keypair from the given Transport. func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, remote peer.ID, initiator bool) (*secureSession, error) { - kp, err := GenerateKeypair() + kp, err := noise.DH25519.GenerateKeypair(rand.Reader) if err != nil { return nil, err } - localPeerInfo := peerInfo{ - noiseKey: kp.publicKey[:], - libp2pKey: tpt.privateKey.GetPublic(), - } - s := &secureSession{ - insecure: insecure, - initiator: initiator, - prologue: []byte{}, - localKey: tpt.privateKey, - localPeer: tpt.localID, - remotePeer: remote, - local: localPeerInfo, - msgBuffer: []byte{}, - noiseKeypair: kp, + insecure: insecure, + ns: noiseState{ + initiator: initiator, + localStatic: kp, + }, + localID: tpt.localID, + localKey: tpt.privateKey, + remoteID: remote, + msgBuffer: []byte{}, } err = s.runHandshake(ctx) @@ -80,20 +66,12 @@ func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, re return s, err } -func (s *secureSession) NoisePublicKey() [32]byte { - return s.noiseKeypair.publicKey -} - -func (s *secureSession) NoisePrivateKey() [32]byte { - return s.noiseKeypair.privateKey -} - func (s *secureSession) LocalAddr() net.Addr { return s.insecure.LocalAddr() } func (s *secureSession) LocalPeer() peer.ID { - return s.localPeer + return s.localID } func (s *secureSession) LocalPrivateKey() crypto.PrivKey { @@ -109,11 +87,11 @@ func (s *secureSession) RemoteAddr() net.Addr { } func (s *secureSession) RemotePeer() peer.ID { - return s.remotePeer + return s.remoteID } func (s *secureSession) RemotePublicKey() crypto.PubKey { - return s.remote.libp2pKey + return s.remoteKey } func (s *secureSession) SetDeadline(t time.Time) error { From 5974ecc85295885ad61b1a912c9d2d0586b29392 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 2 Mar 2020 16:33:28 -0500 Subject: [PATCH 1686/3965] factor out processing remote handshake payload --- p2p/security/noise/handshake.go | 78 ++++++++++++--------------------- 1 file changed, 29 insertions(+), 49 deletions(-) diff --git a/p2p/security/noise/handshake.go b/p2p/security/noise/handshake.go index 682bcf2c47..a2502345dd 100644 --- a/p2p/security/noise/handshake.go +++ b/p2p/security/noise/handshake.go @@ -119,13 +119,34 @@ func (s *secureSession) generateHandshakePayload() ([]byte, error) { return payloadEnc, nil } +func (s *secureSession) handleRemoteHandshakePayload(payload []byte) error { + // unmarshal payload + nhp := new(pb.NoiseHandshakePayload) + err := proto.Unmarshal(payload, nhp) + if err != nil { + return fmt.Errorf("error unmarshaling remote handshake payload: %s", err) + } + + // set remote libp2p public key + err = s.setRemotePeerInfo(nhp.GetIdentityKey()) + if err != nil { + return fmt.Errorf("error processing remote libp2p key: %s", err) + } + + // verify payload is signed by libp2p key + err = s.verifyPayload(nhp, s.ns.hs.PeerStatic()) + if err != nil { + return fmt.Errorf("error validating handshake signature: %s", err) + } + return nil +} + // Runs the XX handshake // XX: // -> e // <- e, ee, s, es // -> s, se func (s *secureSession) runHandshake(ctx context.Context) (err error) { - cfg := noise.Config{ CipherSuite: cipherSuite, Pattern: noise.HandshakeXX, @@ -146,88 +167,47 @@ func (s *secureSession) runHandshake(ctx context.Context) (err error) { if s.ns.initiator { // stage 0 // - err = s.sendHandshakeMessage(nil) if err != nil { return fmt.Errorf("error sending handshake message: %s", err) } // stage 1 // - - // read reply plaintext, err := s.readHandshakeMessage() if err != nil { return fmt.Errorf("error reading handshake message: %s", err) } + err = s.handleRemoteHandshakePayload(plaintext) + if err != nil { + return err + } // stage 2 // err = s.sendHandshakeMessage(payload) if err != nil { return fmt.Errorf("error sending handshake message: %s", err) } - - // unmarshal payload - nhp := new(pb.NoiseHandshakePayload) - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return fmt.Errorf("error unmarshaling remote handshake payload: %s", err) - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetIdentityKey()) - if err != nil { - return fmt.Errorf("error processing remote libp2p key: %s", err) - } - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ns.hs.PeerStatic()) - if err != nil { - return fmt.Errorf("error validating handshake signature: %s", err) - } - } else { - // stage 0 // - var plaintext []byte - nhp := new(pb.NoiseHandshakePayload) - - // read message - plaintext, err = s.readHandshakeMessage() + plaintext, err := s.readHandshakeMessage() if err != nil { return fmt.Errorf("error reading handshake message: %s", err) } // stage 1 // - err = s.sendHandshakeMessage(payload) if err != nil { return fmt.Errorf("error sending handshake message: %s", err) } // stage 2 // - - // read message plaintext, err = s.readHandshakeMessage() if err != nil { return fmt.Errorf("error reading handshake message: %s", err) } - - // unmarshal payload - err = proto.Unmarshal(plaintext, nhp) - if err != nil { - return fmt.Errorf("error unmarshaling remote handshake payload: %s", err) - } - - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetIdentityKey()) - if err != nil { - return fmt.Errorf("error processing remote libp2p key: %s", err) - } - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ns.hs.PeerStatic()) + err = s.handleRemoteHandshakePayload(plaintext) if err != nil { - return fmt.Errorf("error validating handshake signature: %s", err) + return err } } From 6cdc40418044dc96a9a90aefb0cc24faff647ffe Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 2 Mar 2020 15:51:36 -0800 Subject: [PATCH 1687/3965] fix: remove an unnecessary goroutine The stream handler runs in a goroutine anyways. We might as well block it. --- p2p/protocol/identify/id_delta.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/protocol/identify/id_delta.go b/p2p/protocol/identify/id_delta.go index 2cc95b7290..947cc8c858 100644 --- a/p2p/protocol/identify/id_delta.go +++ b/p2p/protocol/identify/id_delta.go @@ -25,7 +25,8 @@ func (ids *IDService) deltaHandler(s network.Stream) { return } - defer func() { go helpers.FullClose(s) }() + defer helpers.FullClose(s) + log.Debugf("%s received message from %s %s", s.Protocol(), c.RemotePeer(), c.RemoteMultiaddr()) delta := mes.GetDelta() From 4adb7a6be4fe996f8b55a17c450090152fda3a5e Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Tue, 3 Mar 2020 13:53:03 +0530 Subject: [PATCH 1688/3965] docs change --- core/event/network.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/core/event/network.go b/core/event/network.go index c03c8d94d8..1a8af306a2 100644 --- a/core/event/network.go +++ b/core/event/network.go @@ -2,12 +2,11 @@ package event import "github.com/libp2p/go-libp2p-core/network" -// EvtPeerStateChange should be emitted when we form our first connection with a peer or drop our last +// EvtPeerStateChange should be emitted everytime we form a connection with a peer or drop our last // connection with the peer. Essentially, it is emitted in two cases: -// a) We go from having no connection with a peer to having a connection with a peer. +// a) We form a/any connection with a peer. // b) We go from having a connection/s with a peer to having no connection with the peer. -// It contains the network interface for the connection, the connection handle for the first/last connection and -// the new connection state. +// It contains the connection handle and the new connection state. type EvtPeerStateChange struct { Connection network.Conn NewState network.Connectedness From 846e4efb4bbbd6774238b0825f7e954a665599eb Mon Sep 17 00:00:00 2001 From: Hlib Date: Tue, 3 Mar 2020 12:19:56 +0200 Subject: [PATCH 1689/3965] update go-yamux --- p2p/muxer/yamux/stream.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/muxer/yamux/stream.go b/p2p/muxer/yamux/stream.go index 9f6e4f9a7a..d5b865b777 100644 --- a/p2p/muxer/yamux/stream.go +++ b/p2p/muxer/yamux/stream.go @@ -12,7 +12,7 @@ type stream yamux.Stream func (s *stream) Read(b []byte) (n int, err error) { n, err = s.yamux().Read(b) - if err == yamux.ErrConnectionReset { + if err == yamux.ErrStreamReset { err = mux.ErrReset } @@ -21,7 +21,7 @@ func (s *stream) Read(b []byte) (n int, err error) { func (s *stream) Write(b []byte) (n int, err error) { n, err = s.yamux().Write(b) - if err == yamux.ErrConnectionReset { + if err == yamux.ErrStreamReset { err = mux.ErrReset } From fe2c13c71b1c091293c712d6d0fef4bdfdf985f0 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 3 Mar 2020 15:33:32 +0100 Subject: [PATCH 1690/3965] docs: uniform comment sentences --- config/muxer.go | 2 +- config/security.go | 2 +- defaults.go | 6 +++--- libp2p.go | 2 +- options.go | 8 ++++---- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/config/muxer.go b/config/muxer.go index 6f99336c51..0babdc3e0e 100644 --- a/config/muxer.go +++ b/config/muxer.go @@ -8,7 +8,7 @@ import ( msmux "github.com/libp2p/go-stream-muxer-multistream" ) -// MuxC is a stream multiplex transport constructor +// MuxC is a stream multiplex transport constructor. type MuxC func(h host.Host) (mux.Multiplexer, error) // MsMuxC is a tuple containing a multiplex transport constructor and a protocol diff --git a/config/security.go b/config/security.go index 72d4b09487..c920e22376 100644 --- a/config/security.go +++ b/config/security.go @@ -12,7 +12,7 @@ import ( csms "github.com/libp2p/go-conn-security-multistream" ) -// SecC is a security transport constructor +// SecC is a security transport constructor. type SecC func(h host.Host) (sec.SecureTransport, error) // MsSecC is a tuple containing a security transport constructor and a protocol diff --git a/defaults.go b/defaults.go index cdcd8288f9..aff515c5bc 100644 --- a/defaults.go +++ b/defaults.go @@ -44,7 +44,7 @@ var DefaultPeerstore Option = func(cfg *Config) error { return cfg.Apply(Peerstore(pstoremem.NewPeerstore())) } -// RandomIdentity generates a random identity (default behaviour) +// RandomIdentity generates a random identity. (default behaviour) var RandomIdentity = func(cfg *Config) error { priv, _, err := crypto.GenerateKeyPairWithReader(crypto.RSA, 2048, rand.Reader) if err != nil { @@ -53,7 +53,7 @@ var RandomIdentity = func(cfg *Config) error { return cfg.Apply(Identity(priv)) } -// DefaultListenAddrs configures libp2p to use default listen address +// DefaultListenAddrs configures libp2p to use default listen address. var DefaultListenAddrs = func(cfg *Config) error { defaultIP4ListenAddr, err := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/0") if err != nil { @@ -70,7 +70,7 @@ var DefaultListenAddrs = func(cfg *Config) error { )) } -// DefaultEnableRelay enables relay dialing and listening by default +// DefaultEnableRelay enables relay dialing and listening by default. var DefaultEnableRelay = func(cfg *Config) error { return cfg.Apply(EnableRelay()) } diff --git a/libp2p.go b/libp2p.go index dbdd09cf1a..bafaff88a8 100644 --- a/libp2p.go +++ b/libp2p.go @@ -8,7 +8,7 @@ import ( "github.com/libp2p/go-libp2p-core/host" ) -// Config describes a set of settings for a libp2p node +// Config describes a set of settings for a libp2p node. type Config = config.Config // Option is a libp2p config option that can be given to the libp2p constructor diff --git a/options.go b/options.go index b46c466d4c..d12ed256f5 100644 --- a/options.go +++ b/options.go @@ -1,7 +1,7 @@ package libp2p // This file contains all libp2p configuration options (except the defaults, -// those are in defaults.go) +// those are in defaults.go). import ( "fmt" @@ -249,7 +249,7 @@ func EnableAutoRelay() Option { // StaticRelays configures known relays for autorelay; when this option is enabled // then the system will use the configured relays instead of querying the DHT to -// discover relays +// discover relays. func StaticRelays(relays []peer.AddrInfo) Option { return func(cfg *Config) error { cfg.StaticRelays = append(cfg.StaticRelays, relays...) @@ -257,7 +257,7 @@ func StaticRelays(relays []peer.AddrInfo) Option { } } -// DefaultStaticRelays configures the static relays to use the known PL-operated relays +// DefaultStaticRelays configures the static relays to use the known PL-operated relays. func DefaultStaticRelays() Option { return func(cfg *Config) error { for _, addr := range autorelay.DefaultRelays { @@ -292,7 +292,7 @@ func FilterAddresses(addrs ...*net.IPNet) Option { } // Filters configures libp2p to use the given filters for accepting/denying -// certain addresses. Filters offers more control and should be use when the +// certain addresses. Filters offers more control and should be used when the // addresses you want to accept/deny are not known ahead of time and can // dynamically change. func Filters(filters *filter.Filters) Option { From db01d6a88138edf0256bdcfc53d3d299fc661f51 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 2 Mar 2020 17:46:16 -0500 Subject: [PATCH 1691/3965] cleanup --- p2p/security/noise/handshake.go | 209 +++++++++++++++++--------------- p2p/security/noise/rw.go | 1 - p2p/security/noise/session.go | 24 +--- 3 files changed, 116 insertions(+), 118 deletions(-) diff --git a/p2p/security/noise/handshake.go b/p2p/security/noise/handshake.go index a2502345dd..cbe5c94f23 100644 --- a/p2p/security/noise/handshake.go +++ b/p2p/security/noise/handshake.go @@ -2,10 +2,11 @@ package noise import ( "context" + "crypto/rand" "fmt" - "github.com/gogo/protobuf/proto" "github.com/flynn/noise" + "github.com/gogo/protobuf/proto" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" @@ -17,43 +18,93 @@ import ( // our libp2p identity key. const payloadSigPrefix = "noise-libp2p-static-key:" +// All noise session share a fixed cipher suite var cipherSuite = noise.NewCipherSuite(noise.DH25519, noise.CipherChaChaPoly, noise.HashSHA256) -func (s *secureSession) setRemotePeerInfo(keyBytes []byte) (err error) { - key, err := crypto.UnmarshalPublicKey(keyBytes) +// runHandshake exchanges handshake messages with the remote peer to establish +// a noise-libp2p session. It blocks until the handshake completes or fails. +func (s *secureSession) runHandshake(ctx context.Context) error { + kp, err := noise.DH25519.GenerateKeypair(rand.Reader) if err != nil { - return err - } - id, err := peer.IDFromPublicKey(key) - if err != nil { - return err + return fmt.Errorf("error generating static keypair: %s", err) } - if s.remoteID != "" && s.remoteID != id { - return fmt.Errorf("peer id mismatch: expected %s, but remote key matches %s", s.remoteID, id) + cfg := noise.Config{ + CipherSuite: cipherSuite, + Pattern: noise.HandshakeXX, + Initiator: s.initiator, + StaticKeypair: kp, } - s.remoteID = id - s.remoteKey = key - return nil -} + hs, err := noise.NewHandshakeState(cfg) + if err != nil { + return fmt.Errorf("error initializing handshake state: %s", err) + } -func (s *secureSession) verifyPayload(payload *pb.NoiseHandshakePayload, noiseKey []byte) (err error) { - sig := payload.GetIdentitySig() - msg := append([]byte(payloadSigPrefix), noiseKey...) + s.ns.hs = hs + s.ns.localStatic = kp - ok, err := s.RemotePublicKey().Verify(msg, sig) + payload, err := s.generateHandshakePayload() if err != nil { return err - } else if !ok { - return fmt.Errorf("did not verify payload") } + if s.initiator { + // stage 0 // + err = s.sendHandshakeMessage(nil) + if err != nil { + return fmt.Errorf("error sending handshake message: %s", err) + } + + // stage 1 // + plaintext, err := s.readHandshakeMessage() + if err != nil { + return fmt.Errorf("error reading handshake message: %s", err) + } + err = s.handleRemoteHandshakePayload(plaintext) + if err != nil { + return err + } + + // stage 2 // + err = s.sendHandshakeMessage(payload) + if err != nil { + return fmt.Errorf("error sending handshake message: %s", err) + } + } else { + // stage 0 // + plaintext, err := s.readHandshakeMessage() + if err != nil { + return fmt.Errorf("error reading handshake message: %s", err) + } + + // stage 1 // + err = s.sendHandshakeMessage(payload) + if err != nil { + return fmt.Errorf("error sending handshake message: %s", err) + } + + // stage 2 // + plaintext, err = s.readHandshakeMessage() + if err != nil { + return fmt.Errorf("error reading handshake message: %s", err) + } + err = s.handleRemoteHandshakePayload(plaintext) + if err != nil { + return err + } + } + + // we can discard the handshake state once the handshake completes + s.ns.hs = nil return nil } -func (s *secureSession) completeHandshake(cs1, cs2 *noise.CipherState) { - if s.ns.initiator { +// setCipherStates is called when the final handshake message is processed by +// either sendHandshakeMessage or readHandshakeMessage. +// It sets the initial cipher states that will be used to protect traffic after the handshake. +func (s *secureSession) setCipherStates(cs1, cs2 *noise.CipherState) { + if s.initiator { s.ns.enc = cs1 s.ns.dec = cs2 } else { @@ -62,6 +113,11 @@ func (s *secureSession) completeHandshake(cs1, cs2 *noise.CipherState) { } } +// sendHandshakeMessage sends the next handshake message in the sequence. +// Only safe to call from runHandshake, as it depends on handshake state. +// If payload is non-empty, it will be included in the handshake message. +// If this is the final message in the sequence, calls setCipherStates +// to initialize cipher states. func (s *secureSession) sendHandshakeMessage(payload []byte) error { buf, cs1, cs2, err := s.ns.hs.WriteMessage(nil, payload) if err != nil { @@ -74,11 +130,17 @@ func (s *secureSession) sendHandshakeMessage(payload []byte) error { } if cs1 != nil && cs2 != nil { - s.completeHandshake(cs1, cs2) + s.setCipherStates(cs1, cs2) } return nil } +// readHandshakeMessage reads a message from the insecure conn and tries to +// process it as the expected next message in the handshake sequence. +// Only safe to call from runHandshake, as it depends on handshake state. +// If the message contains a payload, it will be decrypted and returned. +// If this is the final message in the sequence, calls setCipherStates +// to initialize cipher states. func (s *secureSession) readHandshakeMessage() ([]byte, error) { raw, err := s.readMsgInsecure() if err != nil { @@ -89,11 +151,14 @@ func (s *secureSession) readHandshakeMessage() ([]byte, error) { return nil, err } if cs1 != nil && cs2 != nil { - s.completeHandshake(cs1, cs2) + s.setCipherStates(cs1, cs2) } return msg, nil } +// generateHandshakePayload creates a libp2p handshake payload with a +// signature of our static noise key. +// Must be called after the static key for the session has been generated. func (s *secureSession) generateHandshakePayload() ([]byte, error) { // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() @@ -119,6 +184,9 @@ func (s *secureSession) generateHandshakePayload() ([]byte, error) { return payloadEnc, nil } +// handleRemoteHandshakePayload unmarshals the handshake payload object sent +// by the remote peer and validates the signature against the peer's static Noise key. +// Only safe to call from runHandshake, as it depends on handshake state. func (s *secureSession) handleRemoteHandshakePayload(payload []byte) error { // unmarshal payload nhp := new(pb.NoiseHandshakePayload) @@ -127,89 +195,34 @@ func (s *secureSession) handleRemoteHandshakePayload(payload []byte) error { return fmt.Errorf("error unmarshaling remote handshake payload: %s", err) } - // set remote libp2p public key - err = s.setRemotePeerInfo(nhp.GetIdentityKey()) + // unpack remote peer's public libp2p key + remotePubKey, err := crypto.UnmarshalPublicKey(nhp.GetIdentityKey()) if err != nil { - return fmt.Errorf("error processing remote libp2p key: %s", err) + return err } - - // verify payload is signed by libp2p key - err = s.verifyPayload(nhp, s.ns.hs.PeerStatic()) + id, err := peer.IDFromPublicKey(remotePubKey) if err != nil { - return fmt.Errorf("error validating handshake signature: %s", err) - } - return nil -} - -// Runs the XX handshake -// XX: -// -> e -// <- e, ee, s, es -// -> s, se -func (s *secureSession) runHandshake(ctx context.Context) (err error) { - cfg := noise.Config{ - CipherSuite: cipherSuite, - Pattern: noise.HandshakeXX, - Initiator: s.ns.initiator, - StaticKeypair: s.ns.localStatic, + return err } - hs, err := noise.NewHandshakeState(cfg) - if err != nil { - return fmt.Errorf("error initializing handshake state: %s", err) + // if we know who we're trying to reach, make sure we have the right peer + if s.initiator && s.remoteID != id { + return fmt.Errorf("peer id mismatch: expected %s, but remote key matches %s", s.remoteID, id) } - s.ns.hs = hs - payload, err := s.generateHandshakePayload() + // verify payload is signed by libp2p key + sig := nhp.GetIdentitySig() + remoteStatic := s.ns.hs.PeerStatic() + msg := append([]byte(payloadSigPrefix), remoteStatic...) + ok, err := remotePubKey.Verify(msg, sig) if err != nil { - return err - } - - if s.ns.initiator { - // stage 0 // - err = s.sendHandshakeMessage(nil) - if err != nil { - return fmt.Errorf("error sending handshake message: %s", err) - } - - // stage 1 // - plaintext, err := s.readHandshakeMessage() - if err != nil { - return fmt.Errorf("error reading handshake message: %s", err) - } - err = s.handleRemoteHandshakePayload(plaintext) - if err != nil { - return err - } - - // stage 2 // - err = s.sendHandshakeMessage(payload) - if err != nil { - return fmt.Errorf("error sending handshake message: %s", err) - } - } else { - // stage 0 // - plaintext, err := s.readHandshakeMessage() - if err != nil { - return fmt.Errorf("error reading handshake message: %s", err) - } - - // stage 1 // - err = s.sendHandshakeMessage(payload) - if err != nil { - return fmt.Errorf("error sending handshake message: %s", err) - } - - // stage 2 // - plaintext, err = s.readHandshakeMessage() - if err != nil { - return fmt.Errorf("error reading handshake message: %s", err) - } - err = s.handleRemoteHandshakePayload(plaintext) - if err != nil { - return err - } + return fmt.Errorf("error verifying signature: %s", err) + } else if !ok { + return fmt.Errorf("handshake signature invalid") } + // set remote peer key and id + s.remoteID = id + s.remoteKey = remotePubKey return nil } diff --git a/p2p/security/noise/rw.go b/p2p/security/noise/rw.go index 3d324346a5..32068dfcb8 100644 --- a/p2p/security/noise/rw.go +++ b/p2p/security/noise/rw.go @@ -27,7 +27,6 @@ func (s *secureSession) Read(buf []byte) (int, error) { } readChunk := func(buf []byte) (int, error) { - // read length of encrypted message ciphertext, err := s.readMsgInsecure() if err != nil { return 0, err diff --git a/p2p/security/noise/session.go b/p2p/security/noise/session.go index 8714e00ef6..fe176551e6 100644 --- a/p2p/security/noise/session.go +++ b/p2p/security/noise/session.go @@ -2,22 +2,16 @@ package noise import ( "context" - "crypto/rand" "net" "sync" "time" "github.com/flynn/noise" - logging "github.com/ipfs/go-log" - "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" ) -var log = logging.Logger("noise") - type noiseState struct { - initiator bool localStatic noise.DHKey hs *noise.HandshakeState @@ -26,7 +20,8 @@ type noiseState struct { } type secureSession struct { - ns noiseState + initiator bool + ns noiseState localID peer.ID localKey crypto.PrivKey @@ -42,24 +37,15 @@ type secureSession struct { // newSecureSession creates a noise session over the given insecure Conn, using the // libp2p identity keypair from the given Transport. func newSecureSession(tpt *Transport, ctx context.Context, insecure net.Conn, remote peer.ID, initiator bool) (*secureSession, error) { - kp, err := noise.DH25519.GenerateKeypair(rand.Reader) - if err != nil { - return nil, err - } - s := &secureSession{ - insecure: insecure, - ns: noiseState{ - initiator: initiator, - localStatic: kp, - }, + insecure: insecure, + initiator: initiator, localID: tpt.localID, localKey: tpt.privateKey, remoteID: remote, - msgBuffer: []byte{}, } - err = s.runHandshake(ctx) + err := s.runHandshake(ctx) if err != nil { _ = s.insecure.Close() } From 1edb96a9e1ffa9a8a92923a85bfa9d2b891acfbd Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Mon, 2 Mar 2020 17:48:42 -0500 Subject: [PATCH 1692/3965] make crypto methods private --- p2p/security/noise/crypto.go | 4 ++-- p2p/security/noise/crypto_test.go | 20 ++++++++++---------- p2p/security/noise/rw.go | 4 ++-- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index 5f4694f085..3048f8c332 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -2,7 +2,7 @@ package noise import "errors" -func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) { +func (s *secureSession) encrypt(plaintext []byte) (ciphertext []byte, err error) { if s.ns.enc == nil { return nil, errors.New("cannot encrypt, handshake incomplete") } @@ -12,7 +12,7 @@ func (s *secureSession) Encrypt(plaintext []byte) (ciphertext []byte, err error) return ciphertext, nil } -func (s *secureSession) Decrypt(ciphertext []byte) (plaintext []byte, err error) { +func (s *secureSession) decrypt(ciphertext []byte) (plaintext []byte, err error) { if s.ns.dec == nil { return nil, errors.New("cannot decrypt, handshake incomplete") } diff --git a/p2p/security/noise/crypto_test.go b/p2p/security/noise/crypto_test.go index efc1a8f31d..c321a26aff 100644 --- a/p2p/security/noise/crypto_test.go +++ b/p2p/security/noise/crypto_test.go @@ -18,12 +18,12 @@ func TestEncryptAndDecrypt_InitToResp(t *testing.T) { defer respConn.Close() plaintext := []byte("helloworld") - ciphertext, err := initConn.Encrypt(plaintext) + ciphertext, err := initConn.encrypt(plaintext) if err != nil { t.Fatal(err) } - result, err := respConn.Decrypt(ciphertext) + result, err := respConn.decrypt(ciphertext) if !bytes.Equal(plaintext, result) { t.Fatalf("got %x expected %x", result, plaintext) } else if err != nil { @@ -31,12 +31,12 @@ func TestEncryptAndDecrypt_InitToResp(t *testing.T) { } plaintext = []byte("goodbye") - ciphertext, err = initConn.Encrypt(plaintext) + ciphertext, err = initConn.encrypt(plaintext) if err != nil { t.Fatal(err) } - result, err = respConn.Decrypt(ciphertext) + result, err = respConn.decrypt(ciphertext) if !bytes.Equal(plaintext, result) { t.Fatalf("got %x expected %x", result, plaintext) } else if err != nil { @@ -53,12 +53,12 @@ func TestEncryptAndDecrypt_RespToInit(t *testing.T) { defer respConn.Close() plaintext := []byte("helloworld") - ciphertext, err := respConn.Encrypt(plaintext) + ciphertext, err := respConn.encrypt(plaintext) if err != nil { t.Fatal(err) } - result, err := initConn.Decrypt(ciphertext) + result, err := initConn.decrypt(ciphertext) if !bytes.Equal(plaintext, result) { t.Fatalf("got %x expected %x", result, plaintext) } else if err != nil { @@ -75,14 +75,14 @@ func TestCryptoFailsIfCiphertextIsAltered(t *testing.T) { defer respConn.Close() plaintext := []byte("helloworld") - ciphertext, err := respConn.Encrypt(plaintext) + ciphertext, err := respConn.encrypt(plaintext) if err != nil { t.Fatal(err) } ciphertext[0] = ^ciphertext[0] - _, err = initConn.Decrypt(ciphertext) + _, err = initConn.decrypt(ciphertext) if err == nil { t.Fatal("expected decryption to fail when ciphertext altered") } @@ -94,11 +94,11 @@ func TestCryptoFailsIfHandshakeIncomplete(t *testing.T) { _ = resp.Close() session, _ := newSecureSession(initTransport, context.TODO(), init, "remote-peer", true) - _, err := session.Encrypt([]byte("hi")) + _, err := session.encrypt([]byte("hi")) if err == nil { t.Error("expected encryption error when handshake incomplete") } - _, err = session.Decrypt([]byte("it's a secret")) + _, err = session.decrypt([]byte("it's a secret")) if err == nil { t.Error("expected decryption error when handshake incomplete") } diff --git a/p2p/security/noise/rw.go b/p2p/security/noise/rw.go index 32068dfcb8..cf7d583305 100644 --- a/p2p/security/noise/rw.go +++ b/p2p/security/noise/rw.go @@ -32,7 +32,7 @@ func (s *secureSession) Read(buf []byte) (int, error) { return 0, err } - plaintext, err := s.Decrypt(ciphertext) + plaintext, err := s.decrypt(ciphertext) if err != nil { return 0, err } @@ -69,7 +69,7 @@ func (s *secureSession) Write(in []byte) (int, error) { defer s.writeLock.Unlock() writeChunk := func(in []byte) (int, error) { - ciphertext, err := s.Encrypt(in) + ciphertext, err := s.encrypt(in) if err != nil { return 0, err } From dd7ccf8247aff185c634a1d0c52595b1b178fed0 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Tue, 3 Mar 2020 09:31:36 -0500 Subject: [PATCH 1693/3965] make handshake state local to runHandshake --- p2p/security/noise/crypto.go | 8 +++--- p2p/security/noise/handshake.go | 50 +++++++++++++-------------------- p2p/security/noise/session.go | 13 +++------ 3 files changed, 28 insertions(+), 43 deletions(-) diff --git a/p2p/security/noise/crypto.go b/p2p/security/noise/crypto.go index 3048f8c332..86e206aa76 100644 --- a/p2p/security/noise/crypto.go +++ b/p2p/security/noise/crypto.go @@ -3,20 +3,20 @@ package noise import "errors" func (s *secureSession) encrypt(plaintext []byte) (ciphertext []byte, err error) { - if s.ns.enc == nil { + if s.enc == nil { return nil, errors.New("cannot encrypt, handshake incomplete") } // TODO: use pre-allocated buffers - ciphertext = s.ns.enc.Encrypt(nil, nil, plaintext) + ciphertext = s.enc.Encrypt(nil, nil, plaintext) return ciphertext, nil } func (s *secureSession) decrypt(ciphertext []byte) (plaintext []byte, err error) { - if s.ns.dec == nil { + if s.dec == nil { return nil, errors.New("cannot decrypt, handshake incomplete") } // TODO: use pre-allocated buffers - return s.ns.dec.Decrypt(nil, nil, ciphertext) + return s.dec.Decrypt(nil, nil, ciphertext) } diff --git a/p2p/security/noise/handshake.go b/p2p/security/noise/handshake.go index cbe5c94f23..23c6d63779 100644 --- a/p2p/security/noise/handshake.go +++ b/p2p/security/noise/handshake.go @@ -41,62 +41,57 @@ func (s *secureSession) runHandshake(ctx context.Context) error { return fmt.Errorf("error initializing handshake state: %s", err) } - s.ns.hs = hs - s.ns.localStatic = kp - - payload, err := s.generateHandshakePayload() + payload, err := s.generateHandshakePayload(kp) if err != nil { return err } if s.initiator { // stage 0 // - err = s.sendHandshakeMessage(nil) + err = s.sendHandshakeMessage(hs, nil) if err != nil { return fmt.Errorf("error sending handshake message: %s", err) } // stage 1 // - plaintext, err := s.readHandshakeMessage() + plaintext, err := s.readHandshakeMessage(hs) if err != nil { return fmt.Errorf("error reading handshake message: %s", err) } - err = s.handleRemoteHandshakePayload(plaintext) + err = s.handleRemoteHandshakePayload(plaintext, hs.PeerStatic()) if err != nil { return err } // stage 2 // - err = s.sendHandshakeMessage(payload) + err = s.sendHandshakeMessage(hs, payload) if err != nil { return fmt.Errorf("error sending handshake message: %s", err) } } else { // stage 0 // - plaintext, err := s.readHandshakeMessage() + plaintext, err := s.readHandshakeMessage(hs) if err != nil { return fmt.Errorf("error reading handshake message: %s", err) } // stage 1 // - err = s.sendHandshakeMessage(payload) + err = s.sendHandshakeMessage(hs, payload) if err != nil { return fmt.Errorf("error sending handshake message: %s", err) } // stage 2 // - plaintext, err = s.readHandshakeMessage() + plaintext, err = s.readHandshakeMessage(hs) if err != nil { return fmt.Errorf("error reading handshake message: %s", err) } - err = s.handleRemoteHandshakePayload(plaintext) + err = s.handleRemoteHandshakePayload(plaintext, hs.PeerStatic()) if err != nil { return err } } - // we can discard the handshake state once the handshake completes - s.ns.hs = nil return nil } @@ -105,21 +100,20 @@ func (s *secureSession) runHandshake(ctx context.Context) error { // It sets the initial cipher states that will be used to protect traffic after the handshake. func (s *secureSession) setCipherStates(cs1, cs2 *noise.CipherState) { if s.initiator { - s.ns.enc = cs1 - s.ns.dec = cs2 + s.enc = cs1 + s.dec = cs2 } else { - s.ns.enc = cs2 - s.ns.dec = cs1 + s.enc = cs2 + s.dec = cs1 } } // sendHandshakeMessage sends the next handshake message in the sequence. -// Only safe to call from runHandshake, as it depends on handshake state. // If payload is non-empty, it will be included in the handshake message. // If this is the final message in the sequence, calls setCipherStates // to initialize cipher states. -func (s *secureSession) sendHandshakeMessage(payload []byte) error { - buf, cs1, cs2, err := s.ns.hs.WriteMessage(nil, payload) +func (s *secureSession) sendHandshakeMessage(hs *noise.HandshakeState, payload []byte) error { + buf, cs1, cs2, err := hs.WriteMessage(nil, payload) if err != nil { return err } @@ -137,16 +131,15 @@ func (s *secureSession) sendHandshakeMessage(payload []byte) error { // readHandshakeMessage reads a message from the insecure conn and tries to // process it as the expected next message in the handshake sequence. -// Only safe to call from runHandshake, as it depends on handshake state. // If the message contains a payload, it will be decrypted and returned. // If this is the final message in the sequence, calls setCipherStates // to initialize cipher states. -func (s *secureSession) readHandshakeMessage() ([]byte, error) { +func (s *secureSession) readHandshakeMessage(hs *noise.HandshakeState) ([]byte, error) { raw, err := s.readMsgInsecure() if err != nil { return nil, err } - msg, cs1, cs2, err := s.ns.hs.ReadMessage(nil, raw) + msg, cs1, cs2, err := hs.ReadMessage(nil, raw) if err != nil { return nil, err } @@ -158,8 +151,7 @@ func (s *secureSession) readHandshakeMessage() ([]byte, error) { // generateHandshakePayload creates a libp2p handshake payload with a // signature of our static noise key. -// Must be called after the static key for the session has been generated. -func (s *secureSession) generateHandshakePayload() ([]byte, error) { +func (s *secureSession) generateHandshakePayload(localStatic noise.DHKey) ([]byte, error) { // setup libp2p keys localKeyRaw, err := s.LocalPublicKey().Bytes() if err != nil { @@ -167,7 +159,7 @@ func (s *secureSession) generateHandshakePayload() ([]byte, error) { } // sign noise data for payload - toSign := append([]byte(payloadSigPrefix), s.ns.localStatic.Public...) + toSign := append([]byte(payloadSigPrefix), localStatic.Public...) signedPayload, err := s.localKey.Sign(toSign) if err != nil { return nil, fmt.Errorf("error sigining handshake payload: %s", err) @@ -186,8 +178,7 @@ func (s *secureSession) generateHandshakePayload() ([]byte, error) { // handleRemoteHandshakePayload unmarshals the handshake payload object sent // by the remote peer and validates the signature against the peer's static Noise key. -// Only safe to call from runHandshake, as it depends on handshake state. -func (s *secureSession) handleRemoteHandshakePayload(payload []byte) error { +func (s *secureSession) handleRemoteHandshakePayload(payload []byte, remoteStatic []byte) error { // unmarshal payload nhp := new(pb.NoiseHandshakePayload) err := proto.Unmarshal(payload, nhp) @@ -212,7 +203,6 @@ func (s *secureSession) handleRemoteHandshakePayload(payload []byte) error { // verify payload is signed by libp2p key sig := nhp.GetIdentitySig() - remoteStatic := s.ns.hs.PeerStatic() msg := append([]byte(payloadSigPrefix), remoteStatic...) ok, err := remotePubKey.Verify(msg, sig) if err != nil { diff --git a/p2p/security/noise/session.go b/p2p/security/noise/session.go index fe176551e6..a88619afb3 100644 --- a/p2p/security/noise/session.go +++ b/p2p/security/noise/session.go @@ -7,21 +7,13 @@ import ( "time" "github.com/flynn/noise" + "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" ) -type noiseState struct { - localStatic noise.DHKey - - hs *noise.HandshakeState - enc *noise.CipherState - dec *noise.CipherState -} - type secureSession struct { initiator bool - ns noiseState localID peer.ID localKey crypto.PrivKey @@ -32,6 +24,9 @@ type secureSession struct { msgBuffer []byte readLock sync.Mutex writeLock sync.Mutex + + enc *noise.CipherState + dec *noise.CipherState } // newSecureSession creates a noise session over the given insecure Conn, using the From 370abe02446cc73a520594b65ce62e13795de5c4 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Tue, 3 Mar 2020 09:45:14 -0500 Subject: [PATCH 1694/3965] rm dead code --- p2p/security/noise/keys.go | 24 ------------------------ 1 file changed, 24 deletions(-) delete mode 100644 p2p/security/noise/keys.go diff --git a/p2p/security/noise/keys.go b/p2p/security/noise/keys.go deleted file mode 100644 index e14e216a01..0000000000 --- a/p2p/security/noise/keys.go +++ /dev/null @@ -1,24 +0,0 @@ -package noise - -import ( - "crypto/rand" - "golang.org/x/crypto/curve25519" -) - -// Keypair is a noise ed25519 public-private keypair -type Keypair struct { - publicKey [32]byte - privateKey [32]byte -} - -// GenerateKeypair creates a new ed25519 keypair -func GenerateKeypair() (*Keypair, error) { - var publicKey [32]byte - var privateKey [32]byte - _, err := rand.Read(privateKey[:]) - if err != nil { - return nil, err - } - curve25519.ScalarBaseMult(&publicKey, &privateKey) - return &Keypair{publicKey, privateKey}, nil -} From b55baf18e967d8dd509c3b430b4217fa50a6a1e1 Mon Sep 17 00:00:00 2001 From: David Dias Date: Tue, 3 Mar 2020 15:56:02 +0100 Subject: [PATCH 1695/3965] docs: remove deprecated packages --- README.md | 26 ++------------------------ package-list.json | 24 +----------------------- 2 files changed, 3 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index 318b75230e..e5514a4c00 100644 --- a/README.md +++ b/README.md @@ -87,7 +87,7 @@ Examples can be found in the [examples repo](https://github.com/libp2p/go-libp2p While developing, you may need to make changes to several modules at once, or you may want changes made locally in one module to be available for import by another. -The [go-libp2p workspace](https://github.com/libp2p/workspace-go-libp2p) provides a developer-oriented view of the modules that comprise go-libp2p. +The [go-libp2p workspace](https://github.com/libp2p/workspace-go-libp2p) provides a developer-oriented view of the modules that comprise go-libp2p. Using the tooling in the workspace repository, you can checkout all of go-libp2p's module repos and enter "local mode", which adds [replace directives](https://github.com/golang/go/wiki/Modules#gomod) to the go.mod files in each local working copy. When you build locally, the libp2p depdendencies will be resolved from your local working copies. @@ -109,7 +109,7 @@ If you experience any issues migrating from gx to gomod, please [join the discus ### Tests -`go test ./...` will run all tests in the repo. +`go test ./...` will run all tests in the repo. ### Packages @@ -178,28 +178,6 @@ List of packages currently in existence for libp2p: | **Testing and examples** | | [`go-libp2p-testing`](//github.com/libp2p/go-libp2p-testing) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-testing.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-testing) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-testing/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-testing) | a collection of testing utilities for libp2p | | [`go-libp2p-examples`](//github.com/libp2p/go-libp2p-examples) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-examples.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-examples) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-examples/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-examples) | go-libp2p examples and tutorials | -| **Deprecated** | -| [`go-conn-security`](//github.com/libp2p/go-conn-security) | [![Travis CI](https://travis-ci.com/libp2p/go-conn-security.svg?branch=master)](https://travis-ci.com/libp2p/go-conn-security) | [![codecov](https://codecov.io/gh/libp2p/go-conn-security/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-conn-security) | absorbed into go-libp2p-core | -| [`go-peerstream`](//github.com/libp2p/go-peerstream) | [![Travis CI](https://travis-ci.com/libp2p/go-peerstream.svg?branch=master)](https://travis-ci.com/libp2p/go-peerstream) | [![codecov](https://codecov.io/gh/libp2p/go-peerstream/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-peerstream) | early work that informed libp2p design | -| [`go-stream-muxer`](//github.com/libp2p/go-stream-muxer) | [![Travis CI](https://travis-ci.com/libp2p/go-stream-muxer.svg?branch=master)](https://travis-ci.com/libp2p/go-stream-muxer) | [![codecov](https://codecov.io/gh/libp2p/go-stream-muxer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-stream-muxer) | absorbed into go-libp2p-core | -| [`go-testutil`](//github.com/libp2p/go-testutil) | [![Travis CI](https://travis-ci.com/libp2p/go-testutil.svg?branch=master)](https://travis-ci.com/libp2p/go-testutil) | [![codecov](https://codecov.io/gh/libp2p/go-testutil/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-testutil) | replaced by go-libp2p-testing | -| [`go-libp2p-circuit-progs`](//github.com/libp2p/go-libp2p-circuit-progs) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-circuit-progs.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-circuit-progs) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-circuit-progs) | testing programs for go-libp2p-circuit | -| [`go-libp2p-crypto`](//github.com/libp2p/go-libp2p-crypto) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-crypto.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-crypto) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-crypto/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-crypto) | absorbed into go-libp2p-core | -| [`go-libp2p-conn`](//github.com/libp2p/go-libp2p-conn) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-conn.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-conn) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-conn/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-conn) | replaced by go-libp2p-swarm and go-libp2p-transport-upgrader | -| [`go-libp2p-discovery`](//github.com/libp2p/go-libp2p-discovery) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-discovery.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-discovery) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-discovery/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-discovery) | absorbed into go-libp2p-core | -| [`go-libp2p-dummy-conn`](//github.com/libp2p/go-libp2p-dummy-conn) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-dummy-conn.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-dummy-conn) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-dummy-conn/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-dummy-conn) | implempentation of deprecated interface | -| [`go-libp2p-host`](//github.com/libp2p/go-libp2p-host) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-host.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-host) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-host/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-host) | absorbed into go-libp2p-core | -| [`go-libp2p-identify`](//github.com/libp2p/go-libp2p-identify) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-identify.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-identify) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-identify/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-identify) | moved to go-libp2p | -| [`go-libp2p-interface-conn`](//github.com/libp2p/go-libp2p-interface-conn) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-conn.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-conn) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-conn/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-conn) | absorbed into go-libp2p-core | -| [`go-libp2p-interface-connmgr`](//github.com/libp2p/go-libp2p-interface-connmgr) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-connmgr.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-connmgr) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-connmgr) | absorbed into go-libp2p-core | -| [`go-libp2p-interface-pnet`](//github.com/libp2p/go-libp2p-interface-pnet) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-interface-pnet.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-interface-pnet) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-interface-pnet) | absorbed into go-libp2p-core | -| [`go-libp2p-metrics`](//github.com/libp2p/go-libp2p-metrics) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-metrics.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-metrics) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-metrics) | absorbed into go-libp2p-core | -| [`go-libp2p-net`](//github.com/libp2p/go-libp2p-net) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-net.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-net) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-net/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-net) | absorbed into go-libp2p-core | -| [`go-libp2p-peer`](//github.com/libp2p/go-libp2p-peer) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-peer.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-peer) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-peer/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-peer) | absorbed into go-libp2p-core | -| [`go-libp2p-ping`](//github.com/libp2p/go-libp2p-ping) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-ping.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-ping) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-ping/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-ping) | moved to go-libp2p | -| [`go-libp2p-protocol`](//github.com/libp2p/go-libp2p-protocol) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-protocol.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-protocol) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-protocol/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-protocol) | absorbed into go-libp2p-core | -| [`go-libp2p-transport`](//github.com/libp2p/go-libp2p-transport) | [![Travis CI](https://travis-ci.com/libp2p/go-libp2p-transport.svg?branch=master)](https://travis-ci.com/libp2p/go-libp2p-transport) | [![codecov](https://codecov.io/gh/libp2p/go-libp2p-transport/branch/master/graph/badge.svg)](https://codecov.io/gh/libp2p/go-libp2p-transport) | absorbed into go-libp2p-core | - # Contribute diff --git a/package-list.json b/package-list.json index 21971bba11..00f29f28a4 100644 --- a/package-list.json +++ b/package-list.json @@ -78,28 +78,6 @@ "Testing and examples", ["libp2p/go-libp2p-testing", "go-libp2p-testing", "a collection of testing utilities for libp2p"], - ["libp2p/go-libp2p-examples", "go-libp2p-examples", "go-libp2p examples and tutorials"], - - "Deprecated", - ["libp2p/go-conn-security", "go-conn-security", "absorbed into go-libp2p-core"], - ["libp2p/go-peerstream", "go-peerstream", "early work that informed libp2p design"], - ["libp2p/go-stream-muxer", "go-stream-muxer", "absorbed into go-libp2p-core"], - ["libp2p/go-testutil", "go-testutil", "replaced by go-libp2p-testing"], - ["libp2p/go-libp2p-circuit-progs", "go-libp2p-circuit-progs", "testing programs for go-libp2p-circuit"], - ["libp2p/go-libp2p-crypto", "go-libp2p-crypto", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-conn", "go-libp2p-conn", "replaced by go-libp2p-swarm and go-libp2p-transport-upgrader"], - ["libp2p/go-libp2p-discovery", "go-libp2p-discovery", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-dummy-conn", "go-libp2p-dummy-conn", "implempentation of deprecated interface"], - ["libp2p/go-libp2p-host", "go-libp2p-host", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-identify", "go-libp2p-identify", "moved to go-libp2p"], - ["libp2p/go-libp2p-interface-conn", "go-libp2p-interface-conn", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-interface-connmgr", "go-libp2p-interface-connmgr", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-interface-pnet", "go-libp2p-interface-pnet", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-metrics", "go-libp2p-metrics", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-net", "go-libp2p-net", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-peer", "go-libp2p-peer", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-ping", "go-libp2p-ping", "moved to go-libp2p"], - ["libp2p/go-libp2p-protocol", "go-libp2p-protocol", "absorbed into go-libp2p-core"], - ["libp2p/go-libp2p-transport", "go-libp2p-transport", "absorbed into go-libp2p-core"] + ["libp2p/go-libp2p-examples", "go-libp2p-examples", "go-libp2p examples and tutorials"] ] } From 969eedc98f44ff9c362e6452dc8a842ed2ea0dc9 Mon Sep 17 00:00:00 2001 From: David Dias Date: Tue, 3 Mar 2020 15:57:32 +0100 Subject: [PATCH 1696/3965] docs: remove note to non cancelled dev calls --- README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e5514a4c00..0308277107 100644 --- a/README.md +++ b/README.md @@ -29,12 +29,11 @@ - [API](#api) - [Examples](#examples) - [Development](#development) - - [Using the libp2p Workspace](#using-the-libp2p-workspace) + - [Using the go-libp2p Workspace](#using-the-go-libp2p-workspace) - [About gx](#about-gx) - [Tests](#tests) - [Packages](#packages) - [Contribute](#contribute) -- [Weekly Core Dev Calls](https://github.com/ipfs/pm/issues/674) ## Background @@ -83,7 +82,7 @@ Examples can be found in the [examples repo](https://github.com/libp2p/go-libp2p ## Development -### Using the libp2p Workspace +### Using the go-libp2p Workspace While developing, you may need to make changes to several modules at once, or you may want changes made locally in one module to be available for import by another. From 648a5c51b5f7184b449a8a49aefd18e0be35bb4d Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 3 Mar 2020 18:16:38 +0100 Subject: [PATCH 1697/3965] docs: uniform comment sentences --- core/connmgr/connmgr.go | 2 +- core/peer/addrinfo.go | 2 +- core/peer/peer.go | 6 +++--- core/peer/peer_serde.go | 12 ++++++------ core/peer/record.go | 6 +++--- core/peer/set.go | 2 +- core/peerstore/peerstore.go | 10 +++++----- 7 files changed, 20 insertions(+), 20 deletions(-) diff --git a/core/connmgr/connmgr.go b/core/connmgr/connmgr.go index 98baf8b9a2..4e3186ff12 100644 --- a/core/connmgr/connmgr.go +++ b/core/connmgr/connmgr.go @@ -61,7 +61,7 @@ type ConnManager interface { // See notes on Protect() for more info. Unprotect(id peer.ID, tag string) (protected bool) - // Close closes the connection manager and stops background processes + // Close closes the connection manager and stops background processes. Close() error } diff --git a/core/peer/addrinfo.go b/core/peer/addrinfo.go index 6423d846b1..5cbfe8c3ae 100644 --- a/core/peer/addrinfo.go +++ b/core/peer/addrinfo.go @@ -74,7 +74,7 @@ func AddrInfoFromP2pAddr(m ma.Multiaddr) (*AddrInfo, error) { return info, nil } -// AddrInfoToP2pAddr converts an AddrInfo to a list of Multiaddrs. +// AddrInfoToP2pAddrs converts an AddrInfo to a list of Multiaddrs. func AddrInfoToP2pAddrs(pi *AddrInfo) ([]ma.Multiaddr, error) { var addrs []ma.Multiaddr p2ppart, err := ma.NewComponent("p2p", IDB58Encode(pi.ID)) diff --git a/core/peer/peer.go b/core/peer/peer.go index 9ae0a84406..176b9dffa7 100644 --- a/core/peer/peer.go +++ b/core/peer/peer.go @@ -57,7 +57,7 @@ func (id ID) String() string { return id.Pretty() } -// String prints out the peer ID. +// ShortString prints out the peer ID. // // TODO(brian): ensure correctness at ID generation and // enforce this by only exposing functions that generate @@ -85,7 +85,7 @@ func (id ID) MatchesPublicKey(pk ic.PubKey) bool { return oid == id } -// ExtractPublicKey attempts to extract the public key from an ID +// ExtractPublicKey attempts to extract the public key from an ID. // // This method returns ErrNoPublicKey if the peer ID looks valid but it can't extract // the public key. @@ -237,7 +237,7 @@ func IDFromPrivateKey(sk ic.PrivKey) (ID, error) { return IDFromPublicKey(sk.GetPublic()) } -// IDSlice for sorting peers +// IDSlice for sorting peers. type IDSlice []ID func (es IDSlice) Len() int { return len(es) } diff --git a/core/peer/peer_serde.go b/core/peer/peer_serde.go index 7f1b3e6a65..e401fbbb79 100644 --- a/core/peer/peer_serde.go +++ b/core/peer/peer_serde.go @@ -1,4 +1,4 @@ -// This file contains Protobuf and JSON serialization/deserialization methods for peer IDs. +// Package peer contains Protobuf and JSON serialization/deserialization methods for peer IDs. package peer import ( @@ -21,7 +21,7 @@ func (id ID) Marshal() ([]byte, error) { return []byte(id), nil } -// BinaryMarshal returns the byte representation of the peer ID. +// MarshalBinary returns the byte representation of the peer ID. func (id ID) MarshalBinary() ([]byte, error) { return id.Marshal() } @@ -35,12 +35,12 @@ func (id *ID) Unmarshal(data []byte) (err error) { return err } -// BinaryUnmarshal sets the ID from its binary representation. +// UnmarshalBinary sets the ID from its binary representation. func (id *ID) UnmarshalBinary(data []byte) error { return id.Unmarshal(data) } -// Implements Gogo's proto.Sizer, but we omit the compile-time assertion to avoid introducing a hard +// Size implements Gogo's proto.Sizer, but we omit the compile-time assertion to avoid introducing a hard // dependency on gogo. func (id ID) Size() int { return len([]byte(id)) @@ -59,12 +59,12 @@ func (id *ID) UnmarshalJSON(data []byte) (err error) { return err } -// TextMarshal returns the text encoding of the ID. +// MarshalText returns the text encoding of the ID. func (id ID) MarshalText() ([]byte, error) { return []byte(IDB58Encode(id)), nil } -// TextUnmarshal restores the ID from its text encoding. +// UnmarshalText restores the ID from its text encoding. func (id *ID) UnmarshalText(data []byte) error { pid, err := IDB58Decode(string(data)) if err != nil { diff --git a/core/peer/record.go b/core/peer/record.go index 76c4e672d6..c45efbea7e 100644 --- a/core/peer/record.go +++ b/core/peer/record.go @@ -18,12 +18,12 @@ func init() { record.RegisterType(&PeerRecord{}) } -// The domain string used for peer records contained in a Envelope. +// PeerRecordEnvelopeDomain is the domain string used for peer records contained in a Envelope. const PeerRecordEnvelopeDomain = "libp2p-peer-record" -// The type hint used to identify peer records in a Envelope. +// PeerRecordEnvelopePayloadType is the type hint used to identify peer records in a Envelope. // Defined in https://github.com/multiformats/multicodec/blob/master/table.csv -// with name "libp2p-peer-record" +// with name "libp2p-peer-record". var PeerRecordEnvelopePayloadType = []byte{0x03, 0x01} // PeerRecord contains information that is broadly useful to share with other peers, diff --git a/core/peer/set.go b/core/peer/set.go index ea82ea807e..2251a677e2 100644 --- a/core/peer/set.go +++ b/core/peer/set.go @@ -4,7 +4,7 @@ import ( "sync" ) -// PeerSet is a threadsafe set of peers +// PeerSet is a threadsafe set of peers. type Set struct { lk sync.RWMutex ps map[ID]struct{} diff --git a/core/peerstore/peerstore.go b/core/peerstore/peerstore.go index e77b3c1326..496d34a1c9 100644 --- a/core/peerstore/peerstore.go +++ b/core/peerstore/peerstore.go @@ -22,7 +22,7 @@ var ( // AddressTTL is the expiration time of addresses. AddressTTL = time.Hour - // TempAddrTTL is the ttl used for a short lived address + // TempAddrTTL is the ttl used for a short lived address. TempAddrTTL = time.Minute * 2 // ProviderAddrTTL is the TTL of an address we've received from a provider. @@ -115,10 +115,10 @@ type AddrBook interface { // they will be sent along through the channel as well. AddrStream(context.Context, peer.ID) <-chan ma.Multiaddr - // ClearAddresses removes all previously stored addresses + // ClearAddresses removes all previously stored addresses. ClearAddrs(p peer.ID) - // PeersWithAddrs returns all of the peer IDs stored in the AddrBook + // PeersWithAddrs returns all of the peer IDs stored in the AddrBook. PeersWithAddrs() peer.IDSlice } @@ -159,7 +159,7 @@ type CertifiedAddrBook interface { // // If the signed PeerRecord belongs to a peer that already has certified // addresses in the CertifiedAddrBook, the new addresses will replace the - // older ones, iff the new record has a higher sequence number than the + // older ones, if the new record has a higher sequence number than the // existing record. Attempting to add a peer record with a // sequence number that's <= an existing record for the same peer will not // result in an error, but the record will be ignored, and the 'accepted' @@ -227,7 +227,7 @@ type Metrics interface { LatencyEWMA(peer.ID) time.Duration } -// ProtoBook tracks the protocols supported by peers +// ProtoBook tracks the protocols supported by peers. type ProtoBook interface { GetProtocols(peer.ID) ([]string, error) AddProtocols(peer.ID, ...string) error From 3069cfec014b6d761b4a7255bd4e5df9465ca06f Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 3 Mar 2020 09:56:36 -0800 Subject: [PATCH 1698/3965] unadvertise autonat service when private. remove global request limit when forced on. --- p2p/host/autonat/svc.go | 62 ++++++++++++++++++++++++++--------------- 1 file changed, 39 insertions(+), 23 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 669c0eff11..3beda641e5 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -42,9 +42,10 @@ type AutoNATService struct { dialer host.Host // rate limiter - mx sync.Mutex - reqs map[peer.ID]int - globalReqs int + mx sync.Mutex + reqs map[peer.ID]int + globalReqMax int + globalReqs int } // NewAutoNATService creates a new AutoNATService instance attached to a host @@ -56,29 +57,20 @@ func NewAutoNATService(ctx context.Context, h host.Host, forceEnabled bool, opts } as := &AutoNATService{ - ctx: ctx, - h: h, - dialer: dialer, - reqs: make(map[peer.ID]int), + ctx: ctx, + h: h, + dialer: dialer, + globalReqMax: AutoNATGlobalThrottle, + reqs: make(map[peer.ID]int), } - s, err := h.EventBus().Subscribe(&event.EvtLocalRoutabilityPublic{}) - if err != nil { - return nil, err - } - - go func() { - defer s.Close() - if !forceEnabled { - select { - case <-ctx.Done(): - return - case <-s.Out(): - } - } + if forceEnabled { + as.globalReqMax = 0 h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) go as.resetRateLimiter() - }() + } else { + go as.enableWhenPublic() + } return as, nil } @@ -231,7 +223,7 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { // rate limit check as.mx.Lock() count := as.reqs[pi.ID] - if count >= AutoNATServiceThrottle || as.globalReqs >= AutoNATGlobalThrottle { + if count >= AutoNATServiceThrottle || (as.globalReqMax > 0 && as.globalReqs >= as.globalReqMax) { as.mx.Unlock() return newDialResponseError(pb.Message_E_DIAL_REFUSED, "too many dials") } @@ -264,6 +256,30 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { return newDialResponseOK(ra) } +func (as *AutoNATService) enableWhenPublic() { + pubSub, _ := as.h.EventBus().Subscribe(&event.EvtLocalRoutabilityPublic{}) + priSub, _ := as.h.EventBus().Subscribe(&event.EvtLocalRoutabilityPrivate{}) + defer pubSub.Close() + defer priSub.Close() + + running := false + + for { + select { + case <-pubSub.Out(): + as.h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) + if !running { + go as.resetRateLimiter() + running = true + } + case <-priSub.Out(): + as.h.RemoveStreamHandler(autonat.AutoNATProto) + case <-as.ctx.Done(): + return + } + } +} + func (as *AutoNATService) resetRateLimiter() { timer := time.NewTimer(AutoNATServiceResetInterval) defer timer.Stop() From b7f53da72e340c9ba4743ffab345e7e46edfde8e Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 3 Mar 2020 18:20:55 -0800 Subject: [PATCH 1699/3965] force net connection in autonat svc --- p2p/host/autonat/svc.go | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 3beda641e5..46fd301d08 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -14,6 +14,7 @@ import ( "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" pb "github.com/libp2p/go-libp2p-autonat/pb" @@ -236,7 +237,8 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { as.dialer.Peerstore().ClearAddrs(pi.ID) - err := as.dialer.Connect(ctx, pi) + as.dialer.Peerstore().AddAddrs(pi.ID, pi.Addrs, peerstore.TempAddrTTL) + conn, err := as.dialer.Network().DialPeer(ctx, pi.ID) if err != nil { log.Debugf("error dialing %s: %s", pi.ID.Pretty(), err.Error()) // wait for the context to timeout to avoid leaking timing information @@ -245,13 +247,7 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { return newDialResponseError(pb.Message_E_DIAL_ERROR, "dial failed") } - conns := as.dialer.Network().ConnsToPeer(pi.ID) - if len(conns) == 0 { - log.Errorf("supposedly connected to %s, but no connection to peer", pi.ID.Pretty()) - return newDialResponseError(pb.Message_E_INTERNAL_ERROR, "internal service error") - } - - ra := conns[0].RemoteMultiaddr() + ra := conn.RemoteMultiaddr() as.dialer.Network().ClosePeer(pi.ID) return newDialResponseOK(ra) } From 105e5f22bf9a2f41274e3fe8cd619cff331d941a Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 4 Mar 2020 11:29:42 -0800 Subject: [PATCH 1700/3965] feat: switch to a single routability event This means we can make the event _stateful_ so subscribing always gives us the last state. --- core/event/routability.go | 24 ++++++++---------------- core/network/network.go | 18 ++++++++++++++++++ 2 files changed, 26 insertions(+), 16 deletions(-) diff --git a/core/event/routability.go b/core/event/routability.go index 7a25d69cc2..d595d278bf 100644 --- a/core/event/routability.go +++ b/core/event/routability.go @@ -1,21 +1,13 @@ package event -// EvtLocalRoutabilityPrivate is an event struct to be emitted with the local's -// node routability changes to PRIVATE (i.e. not routable from the Internet). -// -// This event is usually emitted by the AutoNAT subsystem. -type EvtLocalRoutabilityPrivate struct{} - -// EvtLocalRoutabilityPublic is an event struct to be emitted with the local's -// node routability changes to PUBLIC (i.e. appear to routable from the -// Internet). -// -// This event is usually emitted by the AutoNAT subsystem. -type EvtLocalRoutabilityPublic struct{} +import ( + "github.com/libp2p/go-libp2p-core/network" +) -// EvtLocalRoutabilityUnknown is an event struct to be emitted with the local's -// node routability changes to UNKNOWN (i.e. we were unable to make a -// determination about our NAT status with enough confidence). +// EvtLocalRoutability is an event struct to be emitted with the local's node +// routability changes state. // // This event is usually emitted by the AutoNAT subsystem. -type EvtLocalRoutabilityUnknown struct{} +type EvtLocalRoutability struct { + Routability network.Routability +} diff --git a/core/network/network.go b/core/network/network.go index 467109c48c..101d378869 100644 --- a/core/network/network.go +++ b/core/network/network.go @@ -53,6 +53,24 @@ const ( CannotConnect ) +// Routability indicates how reachable a node is. +type Routability int + +const ( + // RoutabilityUnknown indicates that the routability status is unknown. + RoutabilityUnknown = iota + + // RoutabilityPublic indicates that the node is reachable from the + // public internet. + RoutabilityPublic + + // RoutabilityPrivate indicates that the node is not reachable from the + // public internet. + // + // NOTE: This node may _still_ be reachable via relays. + RoutabilityPrivate +) + // Stat stores metadata pertaining to a given Stream/Conn. type Stat struct { Direction Direction From b1eeefc89491fd678257166ce726b366f6322d66 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 4 Mar 2020 12:43:11 -0800 Subject: [PATCH 1701/3965] rename routability -> reachability --- core/event/reachability.go | 13 +++++++++++++ core/event/routability.go | 13 ------------- core/network/network.go | 17 +++++++++-------- 3 files changed, 22 insertions(+), 21 deletions(-) create mode 100644 core/event/reachability.go delete mode 100644 core/event/routability.go diff --git a/core/event/reachability.go b/core/event/reachability.go new file mode 100644 index 0000000000..6bd7db49c8 --- /dev/null +++ b/core/event/reachability.go @@ -0,0 +1,13 @@ +package event + +import ( + "github.com/libp2p/go-libp2p-core/network" +) + +// EvtLocalReachabilityChanged is an event struct to be emitted when the local's +// node reachability changes state. +// +// This event is usually emitted by the AutoNAT subsystem. +type EvtLocalReachabilityChanged struct { + Reachability network.Reachability +} diff --git a/core/event/routability.go b/core/event/routability.go deleted file mode 100644 index d595d278bf..0000000000 --- a/core/event/routability.go +++ /dev/null @@ -1,13 +0,0 @@ -package event - -import ( - "github.com/libp2p/go-libp2p-core/network" -) - -// EvtLocalRoutability is an event struct to be emitted with the local's node -// routability changes state. -// -// This event is usually emitted by the AutoNAT subsystem. -type EvtLocalRoutability struct { - Routability network.Routability -} diff --git a/core/network/network.go b/core/network/network.go index 101d378869..7b61c50b9e 100644 --- a/core/network/network.go +++ b/core/network/network.go @@ -53,22 +53,23 @@ const ( CannotConnect ) -// Routability indicates how reachable a node is. -type Routability int +// Reachability indicates how reachable a node is. +type Reachability int const ( - // RoutabilityUnknown indicates that the routability status is unknown. - RoutabilityUnknown = iota + // ReachabilityUnknown indicates that the reachability status of the + // node is unknown. + ReachabilityUnknown = iota - // RoutabilityPublic indicates that the node is reachable from the + // ReachabilityPublic indicates that the node is reachable from the // public internet. - RoutabilityPublic + ReachabilityPublic - // RoutabilityPrivate indicates that the node is not reachable from the + // ReachabilityPrivate indicates that the node is not reachable from the // public internet. // // NOTE: This node may _still_ be reachable via relays. - RoutabilityPrivate + ReachabilityPrivate ) // Stat stores metadata pertaining to a given Stream/Conn. From edaee9d512bc8cd66a32822b36b3768f725cdbf4 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 5 Mar 2020 00:15:56 -0800 Subject: [PATCH 1702/3965] update to single event type --- p2p/host/autonat/svc.go | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 46fd301d08..2475f03cb9 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -253,23 +253,27 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { } func (as *AutoNATService) enableWhenPublic() { - pubSub, _ := as.h.EventBus().Subscribe(&event.EvtLocalRoutabilityPublic{}) - priSub, _ := as.h.EventBus().Subscribe(&event.EvtLocalRoutabilityPrivate{}) - defer pubSub.Close() - defer priSub.Close() + sub, _ := as.h.EventBus().Subscribe(&event.EvtLocalReachabilityChanged{}) + defer sub.Close() running := false for { select { - case <-pubSub.Out(): - as.h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) - if !running { - go as.resetRateLimiter() - running = true + case ev, ok := <-sub.Out(): + if !ok { + return + } + state := ev.(event.EvtLocalReachabilityChanged).Reachability + if state == network.ReachabilityPublic { + as.h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) + if !running { + go as.resetRateLimiter() + running = true + } + } else { + as.h.RemoveStreamHandler(autonat.AutoNATProto) } - case <-priSub.Out(): - as.h.RemoveStreamHandler(autonat.AutoNATProto) case <-as.ctx.Done(): return } From 0458b76abb31813e050c35f622290b1b92acec07 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 5 Mar 2020 09:12:57 -0800 Subject: [PATCH 1703/3965] update to changed event structure --- p2p/host/relay/autorelay.go | 49 ++++++++++++++++---------------- p2p/host/relay/autorelay_test.go | 4 +-- 2 files changed, 26 insertions(+), 27 deletions(-) diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index a0b9291ccd..09dc50d950 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -52,7 +52,7 @@ type AutoRelay struct { mx sync.Mutex relays map[peer.ID]struct{} - status autonat.NATStatus + status network.Reachability cachedAddrs []ma.Multiaddr cachedAddrsExpiry time.Time @@ -67,7 +67,7 @@ func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discover static: static, relays: make(map[peer.ID]struct{}), disconnect: make(chan struct{}, 1), - status: autonat.NATStatusUnknown, + status: network.ReachabilityUnknown, } ar.autonat = autonat.NewAutoNAT(ctx, bhost, ar.baseAddrs) bhost.AddrsFactory = ar.hostAddrs @@ -85,38 +85,37 @@ func (ar *AutoRelay) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { } func (ar *AutoRelay) background(ctx context.Context) { - subNatPublic, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityPublic)) - defer subNatPublic.Close() - subNatPrivate, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityPrivate)) - defer subNatPrivate.Close() - subNatUnknown, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalRoutabilityUnknown)) - defer subNatUnknown.Close() + subReachability, _ := ar.host.EventBus().Subscribe(new(event.EvtLocalReachabilityChanged)) + defer subReachability.Close() // when true, we need to identify push push := false for { select { - case <-subNatPublic.Out(): - ar.mx.Lock() - if ar.status != autonat.NATStatusPublic { - push = true + case ev, ok := <-subReachability.Out(): + if !ok { + return } - ar.status = autonat.NATStatusPublic - ar.mx.Unlock() - case <-subNatPrivate.Out(): - // TODO: this is a long-lived (2.5min task) that should get spun up in a separate thread - // and canceled if the relay learns the nat is now public. - update := ar.findRelays(ctx) + evt, ok := ev.(event.EvtLocalReachabilityChanged) + if !ok { + return + } + + var update bool + if evt.Reachability == network.ReachabilityPrivate { + // TODO: this is a long-lived (2.5min task) that should get spun up in a separate thread + // and canceled if the relay learns the nat is now public. + update = ar.findRelays(ctx) + } + ar.mx.Lock() - if update || ar.status != autonat.NATStatusPrivate { + if ar.status != evt.Reachability && evt.Reachability != network.ReachabilityUnknown { + push = true + } else if update { push = true } - ar.status = autonat.NATStatusPrivate - ar.mx.Unlock() - case <-subNatUnknown.Out(): - ar.mx.Lock() - ar.status = autonat.NATStatusUnknown + ar.status = evt.Reachability ar.mx.Unlock() case <-ar.disconnect: push = true @@ -279,7 +278,7 @@ func (ar *AutoRelay) relayAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { ar.mx.Lock() defer ar.mx.Unlock() - if ar.status != autonat.NATStatusPrivate { + if ar.status != network.ReachabilityPrivate { return addrs } diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index dac45dbc97..9abd9899ec 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -185,8 +185,8 @@ func TestAutoRelay(t *testing.T) { // connect to AutoNAT, have it resolve to private. connect(t, h1, h3) time.Sleep(300 * time.Millisecond) - privEmitter, _ := h3.EventBus().Emitter(new(event.EvtLocalRoutabilityPrivate)) - privEmitter.Emit(event.EvtLocalRoutabilityPrivate{}) + privEmitter, _ := h3.EventBus().Emitter(new(event.EvtLocalReachabilityChanged)) + privEmitter.Emit(event.EvtLocalReachabilityChanged{Reachability: network.ReachabilityPrivate}) // Wait for detection to do its magic time.Sleep(3000 * time.Millisecond) From 878b6518d9b8f6f40aa25d4a1c3721108d0a529c Mon Sep 17 00:00:00 2001 From: Vibhav Pant Date: Fri, 6 Mar 2020 15:16:59 +0530 Subject: [PATCH 1704/3965] add functions for converting libp2p keys to stdlib variants (#125) --- core/crypto/ecdsa.go | 2 ++ core/crypto/key_not_openssl.go | 40 ++++++++++++++++++++++ core/crypto/key_openssl.go | 47 ++++++++++++++++++++++++++ core/crypto/key_test.go | 61 ++++++++++++++++++++++++++++++++++ 4 files changed, 150 insertions(+) diff --git a/core/crypto/ecdsa.go b/core/crypto/ecdsa.go index 9ea769513a..3b7a425a5d 100644 --- a/core/crypto/ecdsa.go +++ b/core/crypto/ecdsa.go @@ -37,6 +37,8 @@ var ( ErrNilSig = errors.New("sig is nil") // ErrNilPrivateKey is returned when a nil private key is provided ErrNilPrivateKey = errors.New("private key is nil") + // ErrNilPublicKey is returned when a nil public key is provided + ErrNilPublicKey = errors.New("public key is nil") // ECDSACurve is the default ecdsa curve used ECDSACurve = elliptic.P256() ) diff --git a/core/crypto/key_not_openssl.go b/core/crypto/key_not_openssl.go index fb1e36adf3..1499feaab7 100644 --- a/core/crypto/key_not_openssl.go +++ b/core/crypto/key_not_openssl.go @@ -38,3 +38,43 @@ func KeyPairFromStdKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { return nil, nil, ErrBadKeyType } } + +// PrivKeyToStdKey converts libp2p/go-libp2p-core/crypto private keys to standard library (and secp256k1) private keys +func PrivKeyToStdKey(priv PrivKey) (crypto.PrivateKey, error) { + if priv == nil { + return nil, ErrNilPrivateKey + } + + switch p := priv.(type) { + case *RsaPrivateKey: + return &p.sk, nil + case *ECDSAPrivateKey: + return p.priv, nil + case *Ed25519PrivateKey: + return &p.k, nil + case *Secp256k1PrivateKey: + return p, nil + default: + return nil, ErrBadKeyType + } +} + +// PubKeyToStdKey converts libp2p/go-libp2p-core/crypto private keys to standard library (and secp256k1) public keys +func PubKeyToStdKey(pub PubKey) (crypto.PublicKey, error) { + if pub == nil { + return nil, ErrNilPublicKey + } + + switch p := pub.(type) { + case *RsaPublicKey: + return &p.k, nil + case *ECDSAPublicKey: + return p.pub, nil + case *Ed25519PublicKey: + return p.k, nil + case *Secp256k1PublicKey: + return p, nil + default: + return nil, ErrBadKeyType + } +} diff --git a/core/crypto/key_openssl.go b/core/crypto/key_openssl.go index 5b1e3607ab..8d7810ce6c 100644 --- a/core/crypto/key_openssl.go +++ b/core/crypto/key_openssl.go @@ -45,3 +45,50 @@ func KeyPairFromStdKey(priv crypto.PrivateKey) (PrivKey, PubKey, error) { return nil, nil, ErrBadKeyType } } + +// PrivKeyToStdKey converts libp2p/go-libp2p-core/crypto private keys to standard library (and secp256k1) private keys +func PrivKeyToStdKey(priv PrivKey) (crypto.PrivateKey, error) { + if priv == nil { + return nil, ErrNilPrivateKey + } + switch p := priv.(type) { + case *opensslPrivateKey: + raw, err := p.Raw() + if err != nil { + return nil, err + } + return x509.ParsePKCS1PrivateKey(raw) + case *ECDSAPrivateKey: + return p.priv, nil + case *Ed25519PrivateKey: + return &p.k, nil + case *Secp256k1PrivateKey: + return p, nil + default: + return nil, ErrBadKeyType + } +} + +// PubKeyToStdKey converts libp2p/go-libp2p-core/crypto private keys to standard library (and secp256k1) public keys +func PubKeyToStdKey(pub PubKey) (crypto.PublicKey, error) { + if pub == nil { + return nil, ErrNilPublicKey + } + + switch p := pub.(type) { + case *opensslPublicKey: + raw, err := p.Raw() + if err != nil { + return nil, err + } + return x509.ParsePKIXPublicKey(raw) + case *ECDSAPublicKey: + return p.pub, nil + case *Ed25519PublicKey: + return p.k, nil + case *Secp256k1PublicKey: + return p, nil + default: + return nil, ErrBadKeyType + } +} diff --git a/core/crypto/key_test.go b/core/crypto/key_test.go index 984eaa1195..97ab02432c 100644 --- a/core/crypto/key_test.go +++ b/core/crypto/key_test.go @@ -8,7 +8,9 @@ import ( "crypto/elliptic" "crypto/rand" "crypto/rsa" + "crypto/x509" "fmt" + "reflect" "testing" btcec "github.com/btcsuite/btcd/btcec" @@ -111,6 +113,65 @@ func TestKeyPairFromKey(t *testing.T) { if !v { t.Error("signature was not verified") } + + stdPub, err := PubKeyToStdKey(pub) + if stdPub == nil { + t.Errorf("err getting std public key from key: %v", err) + } + + var stdPubBytes []byte + + switch p := stdPub.(type) { + case *Secp256k1PublicKey: + stdPubBytes, err = p.Raw() + case ed25519.PublicKey: + stdPubBytes = []byte(p) + default: + stdPubBytes, err = x509.MarshalPKIXPublicKey(stdPub) + } + + if err != nil { + t.Errorf("Error while marshaling %v key: %v", reflect.TypeOf(stdPub), err) + } + + pubBytes, err := pub.Raw() + if err != nil { + t.Errorf("err getting raw bytes for %v key: %v", reflect.TypeOf(pub), err) + } + if !bytes.Equal(stdPubBytes, pubBytes) { + t.Errorf("err roundtripping %v key", reflect.TypeOf(pub)) + } + + stdPriv, err := PrivKeyToStdKey(priv) + if stdPub == nil { + t.Errorf("err getting std private key from key: %v", err) + } + + var stdPrivBytes []byte + + switch p := stdPriv.(type) { + case *Secp256k1PrivateKey: + stdPrivBytes, err = p.Raw() + case *ecdsa.PrivateKey: + stdPrivBytes, err = x509.MarshalECPrivateKey(p) + case *ed25519.PrivateKey: + stdPrivBytes = *p + case *rsa.PrivateKey: + stdPrivBytes = x509.MarshalPKCS1PrivateKey(p) + } + + if err != nil { + t.Errorf("err marshaling %v key: %v", reflect.TypeOf(stdPriv), err) + } + + privBytes, err := priv.Raw() + if err != nil { + t.Errorf("err getting raw bytes for %v key: %v", reflect.TypeOf(priv), err) + } + + if !bytes.Equal(stdPrivBytes, privBytes) { + t.Errorf("err roundtripping %v key", reflect.TypeOf(priv)) + } }) } } From 57708de2720958a8034abb097f34919c598261be Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 6 Mar 2020 15:58:32 +0530 Subject: [PATCH 1705/3965] changes as per review --- core/event/network.go | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/core/event/network.go b/core/event/network.go index 1a8af306a2..8786fcf181 100644 --- a/core/event/network.go +++ b/core/event/network.go @@ -1,13 +1,16 @@ package event -import "github.com/libp2p/go-libp2p-core/network" +import ( + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" +) -// EvtPeerStateChange should be emitted everytime we form a connection with a peer or drop our last +// EvtPeerConnectednessChange should be emitted every time we form a connection with a peer or drop our last // connection with the peer. Essentially, it is emitted in two cases: // a) We form a/any connection with a peer. // b) We go from having a connection/s with a peer to having no connection with the peer. -// It contains the connection handle and the new connection state. -type EvtPeerStateChange struct { - Connection network.Conn - NewState network.Connectedness +// It contains the Id of the remote peer and the new connectedness state. +type EvtPeerConnectednessChange struct { + RemotePeerId peer.ID + Connectedness network.Connectedness } From defb41516e860cc3e5ca3077dd0dbe9760e282c9 Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 6 Mar 2020 16:08:03 +0530 Subject: [PATCH 1706/3965] changes as per raul's review --- core/event/network.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/event/network.go b/core/event/network.go index 8786fcf181..05b435d287 100644 --- a/core/event/network.go +++ b/core/event/network.go @@ -5,12 +5,12 @@ import ( "github.com/libp2p/go-libp2p-core/peer" ) -// EvtPeerConnectednessChange should be emitted every time we form a connection with a peer or drop our last +// EvtPeerConnectednessChanged should be emitted every time we form a connection with a peer or drop our last // connection with the peer. Essentially, it is emitted in two cases: // a) We form a/any connection with a peer. // b) We go from having a connection/s with a peer to having no connection with the peer. // It contains the Id of the remote peer and the new connectedness state. -type EvtPeerConnectednessChange struct { - RemotePeerId peer.ID +type EvtPeerConnectednessChanged struct { + Peer peer.ID Connectedness network.Connectedness } From 75bd6d58cdeab1015b78f97ffda3a6512a3306a3 Mon Sep 17 00:00:00 2001 From: Yusef Napora Date: Fri, 6 Mar 2020 07:07:36 -0500 Subject: [PATCH 1707/3965] Certified addresses (#98) --- p2p/host/peerstore/pb/pstore.pb.go | 328 ++++++++++++++++++++- p2p/host/peerstore/pb/pstore.proto | 13 + p2p/host/peerstore/pb/pstorepb_test.go | 54 ++++ p2p/host/peerstore/peerstore.go | 1 + p2p/host/peerstore/pstoreds/addr_book.go | 242 +++++++++++---- p2p/host/peerstore/pstoreds/keybook.go | 2 +- p2p/host/peerstore/pstoreds/metadata.go | 2 +- p2p/host/peerstore/pstoreds/peerstore.go | 65 +++- p2p/host/peerstore/pstoreds/protobook.go | 2 +- p2p/host/peerstore/pstoremem/addr_book.go | 201 ++++++++++--- p2p/host/peerstore/pstoremem/keybook.go | 2 +- p2p/host/peerstore/pstoremem/metadata.go | 2 +- p2p/host/peerstore/pstoremem/peerstore.go | 75 ++++- p2p/host/peerstore/pstoremem/protobook.go | 2 +- p2p/host/peerstore/test/addr_book_suite.go | 140 +++++++++ p2p/host/peerstore/test/peerstore_suite.go | 10 + 16 files changed, 1010 insertions(+), 131 deletions(-) diff --git a/p2p/host/peerstore/pb/pstore.pb.go b/p2p/host/peerstore/pb/pstore.pb.go index c3dbaeb344..c7d99807a1 100644 --- a/p2p/host/peerstore/pb/pstore.pb.go +++ b/p2p/host/peerstore/pb/pstore.pb.go @@ -29,6 +29,8 @@ type AddrBookRecord struct { Id *ProtoPeerID `protobuf:"bytes,1,opt,name=id,proto3,customtype=ProtoPeerID" json:"id,omitempty"` // The multiaddresses. This is a sorted list where element 0 expires the soonest. Addrs []*AddrBookRecord_AddrEntry `protobuf:"bytes,2,rep,name=addrs,proto3" json:"addrs,omitempty"` + // The most recently received signed PeerRecord. + CertifiedRecord *AddrBookRecord_CertifiedRecord `protobuf:"bytes,3,opt,name=certified_record,json=certifiedRecord,proto3" json:"certified_record,omitempty"` } func (m *AddrBookRecord) Reset() { *m = AddrBookRecord{} } @@ -71,6 +73,13 @@ func (m *AddrBookRecord) GetAddrs() []*AddrBookRecord_AddrEntry { return nil } +func (m *AddrBookRecord) GetCertifiedRecord() *AddrBookRecord_CertifiedRecord { + if m != nil { + return m.CertifiedRecord + } + return nil +} + // AddrEntry represents a single multiaddress. type AddrBookRecord_AddrEntry struct { Addr *ProtoAddr `protobuf:"bytes,1,opt,name=addr,proto3,customtype=ProtoAddr" json:"addr,omitempty"` @@ -127,31 +136,93 @@ func (m *AddrBookRecord_AddrEntry) GetTtl() int64 { return 0 } +// CertifiedRecord contains a serialized signed PeerRecord used to +// populate the signedAddrs list. +type AddrBookRecord_CertifiedRecord struct { + // The Seq counter from the signed PeerRecord envelope + Seq uint64 `protobuf:"varint,1,opt,name=seq,proto3" json:"seq,omitempty"` + // The serialized bytes of the SignedEnvelope containing the PeerRecord. + Raw []byte `protobuf:"bytes,2,opt,name=raw,proto3" json:"raw,omitempty"` +} + +func (m *AddrBookRecord_CertifiedRecord) Reset() { *m = AddrBookRecord_CertifiedRecord{} } +func (m *AddrBookRecord_CertifiedRecord) String() string { return proto.CompactTextString(m) } +func (*AddrBookRecord_CertifiedRecord) ProtoMessage() {} +func (*AddrBookRecord_CertifiedRecord) Descriptor() ([]byte, []int) { + return fileDescriptor_f96873690e08a98f, []int{0, 1} +} +func (m *AddrBookRecord_CertifiedRecord) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *AddrBookRecord_CertifiedRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_AddrBookRecord_CertifiedRecord.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *AddrBookRecord_CertifiedRecord) XXX_Merge(src proto.Message) { + xxx_messageInfo_AddrBookRecord_CertifiedRecord.Merge(m, src) +} +func (m *AddrBookRecord_CertifiedRecord) XXX_Size() int { + return m.Size() +} +func (m *AddrBookRecord_CertifiedRecord) XXX_DiscardUnknown() { + xxx_messageInfo_AddrBookRecord_CertifiedRecord.DiscardUnknown(m) +} + +var xxx_messageInfo_AddrBookRecord_CertifiedRecord proto.InternalMessageInfo + +func (m *AddrBookRecord_CertifiedRecord) GetSeq() uint64 { + if m != nil { + return m.Seq + } + return 0 +} + +func (m *AddrBookRecord_CertifiedRecord) GetRaw() []byte { + if m != nil { + return m.Raw + } + return nil +} + func init() { proto.RegisterType((*AddrBookRecord)(nil), "pstore.pb.AddrBookRecord") proto.RegisterType((*AddrBookRecord_AddrEntry)(nil), "pstore.pb.AddrBookRecord.AddrEntry") + proto.RegisterType((*AddrBookRecord_CertifiedRecord)(nil), "pstore.pb.AddrBookRecord.CertifiedRecord") } func init() { proto.RegisterFile("pstore.proto", fileDescriptor_f96873690e08a98f) } var fileDescriptor_f96873690e08a98f = []byte{ - // 255 bytes of a gzipped FileDescriptorProto + // 322 bytes of a gzipped FileDescriptorProto 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x29, 0x28, 0x2e, 0xc9, 0x2f, 0x4a, 0xd5, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x84, 0xf1, 0x92, 0xa4, 0x74, 0xd3, 0x33, 0x4b, 0x32, 0x4a, 0x93, 0xf4, 0x92, 0xf3, 0x73, 0xf5, 0xd3, 0xf3, 0xd3, 0xf3, 0xf5, 0xc1, - 0x2a, 0x92, 0x4a, 0xd3, 0xc0, 0x3c, 0x30, 0x07, 0xcc, 0x82, 0xe8, 0x54, 0x3a, 0xc6, 0xc8, 0xc5, + 0x2a, 0x92, 0x4a, 0xd3, 0xc0, 0x3c, 0x30, 0x07, 0xcc, 0x82, 0xe8, 0x54, 0xba, 0xcc, 0xc4, 0xc5, 0xe7, 0x98, 0x92, 0x52, 0xe4, 0x94, 0x9f, 0x9f, 0x1d, 0x94, 0x9a, 0x9c, 0x5f, 0x94, 0x22, 0x24, 0xcf, 0xc5, 0x94, 0x99, 0x22, 0xc1, 0xa8, 0xc0, 0xa8, 0xc1, 0xe3, 0xc4, 0x7f, 0xeb, 0x9e, 0x3c, 0x77, 0x00, 0x48, 0x65, 0x40, 0x6a, 0x6a, 0x91, 0xa7, 0x4b, 0x10, 0x53, 0x66, 0x8a, 0x90, 0x25, 0x17, 0x6b, 0x62, 0x4a, 0x4a, 0x51, 0xb1, 0x04, 0x93, 0x02, 0xb3, 0x06, 0xb7, 0x91, 0xb2, 0x1e, - 0xdc, 0x76, 0x3d, 0x54, 0xa3, 0xc0, 0x5c, 0xd7, 0xbc, 0x92, 0xa2, 0xca, 0x20, 0x88, 0x0e, 0xa9, - 0x08, 0x2e, 0x4e, 0xb8, 0x98, 0x90, 0x22, 0x17, 0x0b, 0x48, 0x14, 0x6a, 0x15, 0xef, 0xad, 0x7b, - 0xf2, 0x9c, 0x60, 0xab, 0x40, 0x2a, 0x82, 0xc0, 0x52, 0x42, 0x62, 0x5c, 0x6c, 0xa9, 0x15, 0x05, - 0x99, 0x45, 0x95, 0x12, 0x4c, 0x0a, 0x8c, 0x1a, 0xcc, 0x41, 0x50, 0x9e, 0x90, 0x00, 0x17, 0x73, - 0x49, 0x49, 0x8e, 0x04, 0x33, 0x58, 0x10, 0xc4, 0x74, 0x52, 0xf8, 0xf1, 0x50, 0x8e, 0xf1, 0xc0, - 0x23, 0x39, 0xc6, 0x13, 0x8f, 0xe4, 0x18, 0x2f, 0x3c, 0x92, 0x63, 0x7c, 0xf0, 0x48, 0x8e, 0x71, - 0xc2, 0x63, 0x39, 0x86, 0x0b, 0x8f, 0xe5, 0x18, 0x6e, 0x3c, 0x96, 0x63, 0x48, 0x62, 0x03, 0xfb, - 0xd8, 0x18, 0x10, 0x00, 0x00, 0xff, 0xff, 0xb1, 0x1a, 0x16, 0x43, 0x3b, 0x01, 0x00, 0x00, + 0xdc, 0x76, 0x3d, 0x54, 0xa3, 0xc0, 0x5c, 0xd7, 0xbc, 0x92, 0xa2, 0xca, 0x20, 0x88, 0x0e, 0xa1, + 0x10, 0x2e, 0x81, 0xe4, 0xd4, 0xa2, 0x92, 0xcc, 0xb4, 0xcc, 0xd4, 0x94, 0xf8, 0x22, 0xb0, 0x22, + 0x09, 0x66, 0x05, 0x46, 0x0d, 0x6e, 0x23, 0x4d, 0xdc, 0xa6, 0x38, 0xc3, 0x74, 0x40, 0xf8, 0x41, + 0xfc, 0xc9, 0xa8, 0x02, 0x52, 0x11, 0x5c, 0x9c, 0x70, 0x9b, 0x84, 0x14, 0xb9, 0x58, 0x40, 0x76, + 0x41, 0x3d, 0xc0, 0x7b, 0xeb, 0x9e, 0x3c, 0x27, 0xd8, 0x03, 0x20, 0x15, 0x41, 0x60, 0x29, 0x21, + 0x31, 0x2e, 0xb6, 0xd4, 0x8a, 0x82, 0xcc, 0xa2, 0x4a, 0x09, 0x26, 0x05, 0x46, 0x0d, 0xe6, 0x20, + 0x28, 0x4f, 0x48, 0x80, 0x8b, 0xb9, 0xa4, 0x24, 0x07, 0xec, 0x20, 0xe6, 0x20, 0x10, 0x53, 0xca, + 0x94, 0x8b, 0x1f, 0xcd, 0x76, 0x90, 0xa2, 0xe2, 0xd4, 0x42, 0xb0, 0xf1, 0x2c, 0x41, 0x20, 0x26, + 0x48, 0xa4, 0x28, 0xb1, 0x1c, 0x6c, 0x16, 0x4f, 0x10, 0x88, 0xe9, 0xa4, 0xf0, 0xe3, 0xa1, 0x1c, + 0xe3, 0x81, 0x47, 0x72, 0x8c, 0x27, 0x1e, 0xc9, 0x31, 0x5e, 0x78, 0x24, 0xc7, 0xf8, 0xe0, 0x91, + 0x1c, 0xe3, 0x84, 0xc7, 0x72, 0x0c, 0x17, 0x1e, 0xcb, 0x31, 0xdc, 0x78, 0x2c, 0xc7, 0x90, 0xc4, + 0x06, 0x0e, 0x7e, 0x63, 0x40, 0x00, 0x00, 0x00, 0xff, 0xff, 0x29, 0xcd, 0xe8, 0xd4, 0xc8, 0x01, + 0x00, 0x00, } func (m *AddrBookRecord) Marshal() (dAtA []byte, err error) { @@ -174,6 +245,18 @@ func (m *AddrBookRecord) MarshalToSizedBuffer(dAtA []byte) (int, error) { _ = i var l int _ = l + if m.CertifiedRecord != nil { + { + size, err := m.CertifiedRecord.MarshalToSizedBuffer(dAtA[:i]) + if err != nil { + return 0, err + } + i -= size + i = encodeVarintPstore(dAtA, i, uint64(size)) + } + i-- + dAtA[i] = 0x1a + } if len(m.Addrs) > 0 { for iNdEx := len(m.Addrs) - 1; iNdEx >= 0; iNdEx-- { { @@ -248,6 +331,41 @@ func (m *AddrBookRecord_AddrEntry) MarshalToSizedBuffer(dAtA []byte) (int, error return len(dAtA) - i, nil } +func (m *AddrBookRecord_CertifiedRecord) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *AddrBookRecord_CertifiedRecord) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *AddrBookRecord_CertifiedRecord) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if len(m.Raw) > 0 { + i -= len(m.Raw) + copy(dAtA[i:], m.Raw) + i = encodeVarintPstore(dAtA, i, uint64(len(m.Raw))) + i-- + dAtA[i] = 0x12 + } + if m.Seq != 0 { + i = encodeVarintPstore(dAtA, i, uint64(m.Seq)) + i-- + dAtA[i] = 0x8 + } + return len(dAtA) - i, nil +} + func encodeVarintPstore(dAtA []byte, offset int, v uint64) int { offset -= sovPstore(v) base := offset @@ -269,6 +387,9 @@ func NewPopulatedAddrBookRecord(r randyPstore, easy bool) *AddrBookRecord { this.Addrs[i] = NewPopulatedAddrBookRecord_AddrEntry(r, easy) } } + if r.Intn(5) != 0 { + this.CertifiedRecord = NewPopulatedAddrBookRecord_CertifiedRecord(r, easy) + } if !easy && r.Intn(10) != 0 { } return this @@ -290,6 +411,19 @@ func NewPopulatedAddrBookRecord_AddrEntry(r randyPstore, easy bool) *AddrBookRec return this } +func NewPopulatedAddrBookRecord_CertifiedRecord(r randyPstore, easy bool) *AddrBookRecord_CertifiedRecord { + this := &AddrBookRecord_CertifiedRecord{} + this.Seq = uint64(uint64(r.Uint32())) + v2 := r.Intn(100) + this.Raw = make([]byte, v2) + for i := 0; i < v2; i++ { + this.Raw[i] = byte(r.Intn(256)) + } + if !easy && r.Intn(10) != 0 { + } + return this +} + type randyPstore interface { Float32() float32 Float64() float64 @@ -309,9 +443,9 @@ func randUTF8RunePstore(r randyPstore) rune { return rune(ru + 61) } func randStringPstore(r randyPstore) string { - v2 := r.Intn(100) - tmps := make([]rune, v2) - for i := 0; i < v2; i++ { + v3 := r.Intn(100) + tmps := make([]rune, v3) + for i := 0; i < v3; i++ { tmps[i] = randUTF8RunePstore(r) } return string(tmps) @@ -333,11 +467,11 @@ func randFieldPstore(dAtA []byte, r randyPstore, fieldNumber int, wire int) []by switch wire { case 0: dAtA = encodeVarintPopulatePstore(dAtA, uint64(key)) - v3 := r.Int63() + v4 := r.Int63() if r.Intn(2) == 0 { - v3 *= -1 + v4 *= -1 } - dAtA = encodeVarintPopulatePstore(dAtA, uint64(v3)) + dAtA = encodeVarintPopulatePstore(dAtA, uint64(v4)) case 1: dAtA = encodeVarintPopulatePstore(dAtA, uint64(key)) dAtA = append(dAtA, byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256)), byte(r.Intn(256))) @@ -378,6 +512,10 @@ func (m *AddrBookRecord) Size() (n int) { n += 1 + l + sovPstore(uint64(l)) } } + if m.CertifiedRecord != nil { + l = m.CertifiedRecord.Size() + n += 1 + l + sovPstore(uint64(l)) + } return n } @@ -400,6 +538,22 @@ func (m *AddrBookRecord_AddrEntry) Size() (n int) { return n } +func (m *AddrBookRecord_CertifiedRecord) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.Seq != 0 { + n += 1 + sovPstore(uint64(m.Seq)) + } + l = len(m.Raw) + if l > 0 { + n += 1 + l + sovPstore(uint64(l)) + } + return n +} + func sovPstore(x uint64) (n int) { return (math_bits.Len64(x|1) + 6) / 7 } @@ -504,6 +658,42 @@ func (m *AddrBookRecord) Unmarshal(dAtA []byte) error { return err } iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field CertifiedRecord", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthPstore + } + postIndex := iNdEx + msglen + if postIndex < 0 { + return ErrInvalidLengthPstore + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + if m.CertifiedRecord == nil { + m.CertifiedRecord = &AddrBookRecord_CertifiedRecord{} + } + if err := m.CertifiedRecord.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex default: iNdEx = preIndex skippy, err := skipPstore(dAtA[iNdEx:]) @@ -654,6 +844,112 @@ func (m *AddrBookRecord_AddrEntry) Unmarshal(dAtA []byte) error { } return nil } +func (m *AddrBookRecord_CertifiedRecord) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: CertifiedRecord: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: CertifiedRecord: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Seq", wireType) + } + m.Seq = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.Seq |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Raw", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowPstore + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthPstore + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthPstore + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Raw = append(m.Raw[:0], dAtA[iNdEx:postIndex]...) + if m.Raw == nil { + m.Raw = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipPstore(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthPstore + } + if (iNdEx + skippy) < 0 { + return ErrInvalidLengthPstore + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} func skipPstore(dAtA []byte) (n int, err error) { l := len(dAtA) iNdEx := 0 diff --git a/p2p/host/peerstore/pb/pstore.proto b/p2p/host/peerstore/pb/pstore.proto index 3894534745..0d1abd1da4 100644 --- a/p2p/host/peerstore/pb/pstore.proto +++ b/p2p/host/peerstore/pb/pstore.proto @@ -14,6 +14,9 @@ message AddrBookRecord { // The multiaddresses. This is a sorted list where element 0 expires the soonest. repeated AddrEntry addrs = 2; + // The most recently received signed PeerRecord. + CertifiedRecord certified_record = 3; + // AddrEntry represents a single multiaddress. message AddrEntry { bytes addr = 1 [(gogoproto.customtype) = "ProtoAddr"]; @@ -24,4 +27,14 @@ message AddrBookRecord { // The original TTL of this address. int64 ttl = 3; } + + // CertifiedRecord contains a serialized signed PeerRecord used to + // populate the signedAddrs list. + message CertifiedRecord { + // The Seq counter from the signed PeerRecord envelope + uint64 seq = 1; + + // The serialized bytes of the SignedEnvelope containing the PeerRecord. + bytes raw = 2; + } } diff --git a/p2p/host/peerstore/pb/pstorepb_test.go b/p2p/host/peerstore/pb/pstorepb_test.go index 3569d045ff..12d298069f 100644 --- a/p2p/host/peerstore/pb/pstorepb_test.go +++ b/p2p/host/peerstore/pb/pstorepb_test.go @@ -98,6 +98,46 @@ func BenchmarkAddrBookRecord_AddrEntryProtoUnmarshal(b *testing.B) { b.SetBytes(int64(total / b.N)) } +func BenchmarkAddrBookRecord_CertifiedRecordProtoMarshal(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + pops := make([]*AddrBookRecord_CertifiedRecord, 10000) + for i := 0; i < 10000; i++ { + pops[i] = NewPopulatedAddrBookRecord_CertifiedRecord(popr, false) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + dAtA, err := github_com_gogo_protobuf_proto.Marshal(pops[i%10000]) + if err != nil { + panic(err) + } + total += len(dAtA) + } + b.SetBytes(int64(total / b.N)) +} + +func BenchmarkAddrBookRecord_CertifiedRecordProtoUnmarshal(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + datas := make([][]byte, 10000) + for i := 0; i < 10000; i++ { + dAtA, err := github_com_gogo_protobuf_proto.Marshal(NewPopulatedAddrBookRecord_CertifiedRecord(popr, false)) + if err != nil { + panic(err) + } + datas[i] = dAtA + } + msg := &AddrBookRecord_CertifiedRecord{} + b.ResetTimer() + for i := 0; i < b.N; i++ { + total += len(datas[i%10000]) + if err := github_com_gogo_protobuf_proto.Unmarshal(datas[i%10000], msg); err != nil { + panic(err) + } + } + b.SetBytes(int64(total / b.N)) +} + func BenchmarkAddrBookRecordSize(b *testing.B) { popr := math_rand.New(math_rand.NewSource(616)) total := 0 @@ -126,4 +166,18 @@ func BenchmarkAddrBookRecord_AddrEntrySize(b *testing.B) { b.SetBytes(int64(total / b.N)) } +func BenchmarkAddrBookRecord_CertifiedRecordSize(b *testing.B) { + popr := math_rand.New(math_rand.NewSource(616)) + total := 0 + pops := make([]*AddrBookRecord_CertifiedRecord, 1000) + for i := 0; i < 1000; i++ { + pops[i] = NewPopulatedAddrBookRecord_CertifiedRecord(popr, false) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + total += pops[i%1000].Size() + } + b.SetBytes(int64(total / b.N)) +} + //These tests are generated by github.com/gogo/protobuf/plugin/testgen diff --git a/p2p/host/peerstore/peerstore.go b/p2p/host/peerstore/peerstore.go index 3331b4a796..278a5f865c 100644 --- a/p2p/host/peerstore/peerstore.go +++ b/p2p/host/peerstore/peerstore.go @@ -21,6 +21,7 @@ type peerstore struct { // NewPeerstore creates a data structure that stores peer data, backed by the // supplied implementations of KeyBook, AddrBook and PeerMetadata. +// Deprecated: use pstoreds.NewPeerstore or peerstoremem.NewPeerstore instead. func NewPeerstore(kb pstore.KeyBook, ab pstore.AddrBook, pb pstore.ProtoBook, md pstore.PeerMetadata) pstore.Peerstore { return &peerstore{ KeyBook: kb, diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 9e04283463..12cb0814e8 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -3,20 +3,21 @@ package pstoreds import ( "context" "fmt" + "github.com/libp2p/go-libp2p-core/record" "sort" "sync" "time" ds "github.com/ipfs/go-datastore" - query "github.com/ipfs/go-datastore/query" + "github.com/ipfs/go-datastore/query" logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peer" pstore "github.com/libp2p/go-libp2p-core/peerstore" pb "github.com/libp2p/go-libp2p-peerstore/pb" - pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" + "github.com/libp2p/go-libp2p-peerstore/pstoremem" - lru "github.com/hashicorp/golang-lru" + "github.com/hashicorp/golang-lru" b32 "github.com/multiformats/go-base32" ma "github.com/multiformats/go-multiaddr" ) @@ -47,6 +48,7 @@ type addrsRecord struct { // marked for deletion, in which case we call ds.Delete. To be called within a lock. func (r *addrsRecord) flush(write ds.Write) (err error) { key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(r.Id.ID))) + if len(r.Addrs) == 0 { if err = write.Delete(key); err == nil { r.dirty = false @@ -83,37 +85,49 @@ func (r *addrsRecord) flush(write ds.Write) (err error) { // If the return value is true, the caller should perform a flush immediately to sync the record with the store. func (r *addrsRecord) clean() (chgd bool) { now := time.Now().Unix() - if !r.dirty && len(r.Addrs) > 0 && r.Addrs[0].Expiry > now { + addrsLen := len(r.Addrs) + + if !r.dirty && !r.hasExpiredAddrs(now) { // record is not dirty, and we have no expired entries to purge. return false } - if len(r.Addrs) == 0 { + if addrsLen == 0 { // this is a ghost record; let's signal it has to be written. // flush() will take care of doing the deletion. return true } - if r.dirty && len(r.Addrs) > 1 { - // the record has been modified, so it may need resorting. - // we keep addresses sorted by expiration, where 0 is the soonest expiring. + if r.dirty && addrsLen > 1 { sort.Slice(r.Addrs, func(i, j int) bool { return r.Addrs[i].Expiry < r.Addrs[j].Expiry }) } + r.Addrs = removeExpired(r.Addrs, now) + + return r.dirty || len(r.Addrs) != addrsLen +} + +func (r *addrsRecord) hasExpiredAddrs(now int64) bool { + if len(r.Addrs) > 0 && r.Addrs[0].Expiry <= now { + return true + } + return false +} + +func removeExpired(entries []*pb.AddrBookRecord_AddrEntry, now int64) []*pb.AddrBookRecord_AddrEntry { // since addresses are sorted by expiration, we find the first // survivor and split the slice on its index. pivot := -1 - for i, addr := range r.Addrs { + for i, addr := range entries { if addr.Expiry > now { break } pivot = i } - r.Addrs = r.Addrs[pivot+1:] - return r.dirty || pivot >= 0 + return entries[pivot+1:] } // dsAddrBook is an address book backed by a Datastore with a GC procedure to purge expired entries. It uses an @@ -133,6 +147,7 @@ type dsAddrBook struct { } var _ pstore.AddrBook = (*dsAddrBook)(nil) +var _ pstore.CertifiedAddrBook = (*dsAddrBook)(nil) // NewAddrBook initializes a new datastore-backed address book. It serves as a drop-in replacement for pstoremem // (memory-backed peerstore), and works with any datastore implementing the ds.Batching interface. @@ -240,7 +255,91 @@ func (ab *dsAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duratio return } addrs = cleanAddrs(addrs) - ab.setAddrs(p, addrs, ttl, ttlExtend) + ab.setAddrs(p, addrs, ttl, ttlExtend, false) +} + +// ConsumePeerRecord adds addresses from a signed peer.PeerRecord (contained in +// a record.Envelope), which will expire after the given TTL. +// See https://godoc.org/github.com/libp2p/go-libp2p-core/peerstore#CertifiedAddrBook for more details. +func (ab *dsAddrBook) ConsumePeerRecord(recordEnvelope *record.Envelope, ttl time.Duration) (bool, error) { + r, err := recordEnvelope.Record() + if err != nil { + return false, err + } + rec, ok := r.(*peer.PeerRecord) + if !ok { + return false, fmt.Errorf("envelope did not contain PeerRecord") + } + if !rec.PeerID.MatchesPublicKey(recordEnvelope.PublicKey) { + return false, fmt.Errorf("signing key does not match PeerID in PeerRecord") + } + + // ensure that the seq number from envelope is > any previously received seq no + if ab.latestPeerRecordSeq(rec.PeerID) >= rec.Seq { + return false, nil + } + + addrs := cleanAddrs(rec.Addrs) + err = ab.setAddrs(rec.PeerID, addrs, ttl, ttlExtend, true) + if err != nil { + return false, err + } + + err = ab.storeSignedPeerRecord(rec.PeerID, recordEnvelope, rec) + if err != nil { + return false, err + } + return true, nil +} + +func (ab *dsAddrBook) latestPeerRecordSeq(p peer.ID) uint64 { + pr, err := ab.loadRecord(p, true, false) + if err != nil || len(pr.Addrs) == 0 || pr.CertifiedRecord == nil || len(pr.CertifiedRecord.Raw) == 0 { + return 0 + } + return pr.CertifiedRecord.Seq +} + +func (ab *dsAddrBook) storeSignedPeerRecord(p peer.ID, envelope *record.Envelope, rec *peer.PeerRecord) error { + envelopeBytes, err := envelope.Marshal() + if err != nil { + return err + } + // reload record and add routing state + // this has to be done after we add the addresses, since if + // we try to flush a datastore record with no addresses, + // it will just get deleted + pr, err := ab.loadRecord(p, true, false) + if err != nil { + return err + } + pr.CertifiedRecord = &pb.AddrBookRecord_CertifiedRecord{ + Seq: rec.Seq, + Raw: envelopeBytes, + } + pr.dirty = true + err = pr.flush(ab.ds) + return err +} + +// GetPeerRecord returns a record.Envelope containing a peer.PeerRecord for the +// given peer id, if one exists. +// Returns nil if no signed PeerRecord exists for the peer. +func (ab *dsAddrBook) GetPeerRecord(p peer.ID) *record.Envelope { + pr, err := ab.loadRecord(p, true, false) + if err != nil { + log.Errorf("unable to load record for peer %s: %v", p.Pretty(), err) + return nil + } + if pr.CertifiedRecord == nil || len(pr.CertifiedRecord.Raw) == 0 || len(pr.Addrs) == 0 { + return nil + } + state, _, err := record.ConsumeEnvelope(pr.CertifiedRecord.Raw, peer.PeerRecordEnvelopeDomain) + if err != nil { + log.Errorf("error unmarshaling stored signed peer record for peer %s: %v", p.Pretty(), err) + return nil + } + return state } // SetAddr will add or update the TTL of an address in the AddrBook. @@ -255,7 +354,7 @@ func (ab *dsAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duratio ab.deleteAddrs(p, addrs) return } - ab.setAddrs(p, addrs, ttl, ttlOverride) + ab.setAddrs(p, addrs, ttl, ttlOverride, false) } // UpdateAddrs will update any addresses for a given peer and TTL combination to @@ -295,9 +394,9 @@ func (ab *dsAddrBook) Addrs(p peer.ID) []ma.Multiaddr { pr.RLock() defer pr.RUnlock() - addrs := make([]ma.Multiaddr, 0, len(pr.Addrs)) - for _, a := range pr.Addrs { - addrs = append(addrs, a.Addr) + addrs := make([]ma.Multiaddr, len(pr.Addrs)) + for i, a := range pr.Addrs { + addrs[i] = a.Addr } return addrs } @@ -330,7 +429,7 @@ func (ab *dsAddrBook) ClearAddrs(p peer.ID) { } } -func (ab *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode) (err error) { +func (ab *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, mode ttlWriteMode, signed bool) (err error) { pr, err := ab.loadRecord(p, true, false) if err != nil { return fmt.Errorf("failed to load peerstore entry for peer %v while setting addrs, err: %v", p, err) @@ -339,14 +438,19 @@ func (ab *dsAddrBook) setAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duratio pr.Lock() defer pr.Unlock() - newExp := time.Now().Add(ttl).Unix() - existed := make([]bool, len(addrs)) // keeps track of which addrs we found. + // // if we have a signed PeerRecord, ignore attempts to add unsigned addrs + // if !signed && pr.CertifiedRecord != nil { + // return nil + // } -Outer: - for i, incoming := range addrs { - for _, have := range pr.Addrs { + newExp := time.Now().Add(ttl).Unix() + // TODO this is very inefficient O(m*n); we could build a map to use as an + // index, and test against it. That would turn it into O(m+n). This code + // will be refactored entirely anyway, and it's not being used by users + // (that we know of); so OK to keep it for now. + updateExisting := func(entryList []*pb.AddrBookRecord_AddrEntry, incoming ma.Multiaddr) *pb.AddrBookRecord_AddrEntry { + for _, have := range entryList { if incoming.Equal(have.Addr) { - existed[i] = true switch mode { case ttlOverride: have.Ttl = int64(ttl) @@ -361,38 +465,73 @@ Outer: default: panic("BUG: unimplemented ttl mode") } - - // we found the address, and addresses cannot be duplicate, - // so let's move on to the next. - continue Outer + return have } } + return nil } - // add addresses we didn't hold. - var added []*pb.AddrBookRecord_AddrEntry - for i, e := range existed { - if e { - continue - } - addr := addrs[i] - entry := &pb.AddrBookRecord_AddrEntry{ - Addr: &pb.ProtoAddr{Multiaddr: addr}, - Ttl: int64(ttl), - Expiry: newExp, + var entries []*pb.AddrBookRecord_AddrEntry + for _, incoming := range addrs { + existingEntry := updateExisting(pr.Addrs, incoming) + + if existingEntry == nil { + // if signed { + // entries = append(entries, existingEntry) + // } + // } else { + // new addr, add & broadcast + entry := &pb.AddrBookRecord_AddrEntry{ + Addr: &pb.ProtoAddr{Multiaddr: incoming}, + Ttl: int64(ttl), + Expiry: newExp, + } + entries = append(entries, entry) + + // note: there's a minor chance that writing the record will fail, in which case we would've broadcast + // the addresses without persisting them. This is very unlikely and not much of an issue. + ab.subsManager.BroadcastAddr(p, incoming) } - added = append(added, entry) - // note: there's a minor chance that writing the record will fail, in which case we would've broadcast - // the addresses without persisting them. This is very unlikely and not much of an issue. - ab.subsManager.BroadcastAddr(p, addr) } - pr.Addrs = append(pr.Addrs, added...) + // if signed { + // // when adding signed addrs, we want to keep _only_ the incoming addrs + // pr.Addrs = entries + // } else { + pr.Addrs = append(pr.Addrs, entries...) + // } + pr.dirty = true pr.clean() return pr.flush(ab.ds) } +// deletes addresses in place, avoiding copies until we encounter the first deletion. +// does not preserve order, but entries are re-sorted before flushing to disk anyway. +func deleteInPlace(s []*pb.AddrBookRecord_AddrEntry, addrs []ma.Multiaddr) []*pb.AddrBookRecord_AddrEntry { + if s == nil || len(addrs) == 0 { + return s + } + survived := len(s) +Outer: + for i, addr := range s { + for _, del := range addrs { + if !addr.Addr.Equal(del) { + continue + } + survived-- + // if there are no survivors, bail out + if survived == 0 { + break Outer + } + s[i] = s[survived] + // we've already dealt with s[i], move to the next + continue Outer + } + } + return s[:survived] +} + func (ab *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) (err error) { pr, err := ab.loadRecord(p, false, false) if err != nil { @@ -406,20 +545,7 @@ func (ab *dsAddrBook) deleteAddrs(p peer.ID, addrs []ma.Multiaddr) (err error) { pr.Lock() defer pr.Unlock() - // deletes addresses in place, and avoiding copies until we encounter the first deletion. - survived := 0 - for i, addr := range pr.Addrs { - for _, del := range addrs { - if addr.Addr.Equal(del) { - continue - } - if i != survived { - pr.Addrs[survived] = pr.Addrs[i] - } - survived++ - } - } - pr.Addrs = pr.Addrs[:survived] + pr.Addrs = deleteInPlace(pr.Addrs, addrs) pr.dirty = true pr.clean() diff --git a/p2p/host/peerstore/pstoreds/keybook.go b/p2p/host/peerstore/pstoreds/keybook.go index 64b73a3024..35021b2d04 100644 --- a/p2p/host/peerstore/pstoreds/keybook.go +++ b/p2p/host/peerstore/pstoreds/keybook.go @@ -28,7 +28,7 @@ type dsKeyBook struct { var _ pstore.KeyBook = (*dsKeyBook)(nil) -func NewKeyBook(_ context.Context, store ds.Datastore, _ Options) (pstore.KeyBook, error) { +func NewKeyBook(_ context.Context, store ds.Datastore, _ Options) (*dsKeyBook, error) { return &dsKeyBook{store}, nil } diff --git a/p2p/host/peerstore/pstoreds/metadata.go b/p2p/host/peerstore/pstoreds/metadata.go index f54f382ba7..4d285af789 100644 --- a/p2p/host/peerstore/pstoreds/metadata.go +++ b/p2p/host/peerstore/pstoreds/metadata.go @@ -36,7 +36,7 @@ func init() { // See `init()` to learn which types are registered by default. Modules wishing to store // values of other types will need to `gob.Register()` them explicitly, or else callers // will receive runtime errors. -func NewPeerMetadata(_ context.Context, store ds.Datastore, _ Options) (pstore.PeerMetadata, error) { +func NewPeerMetadata(_ context.Context, store ds.Datastore, _ Options) (*dsPeerMetadata, error) { return &dsPeerMetadata{store}, nil } diff --git a/p2p/host/peerstore/pstoreds/peerstore.go b/p2p/host/peerstore/pstoreds/peerstore.go index 99b381db2f..3c8aa0f84d 100644 --- a/p2p/host/peerstore/pstoreds/peerstore.go +++ b/p2p/host/peerstore/pstoreds/peerstore.go @@ -2,6 +2,8 @@ package pstoreds import ( "context" + "fmt" + "io" "time" base32 "github.com/multiformats/go-base32" @@ -47,8 +49,17 @@ func DefaultOpts() Options { } } +type pstoreds struct { + peerstore.Metrics + + dsKeyBook + dsAddrBook + dsProtoBook + dsPeerMetadata +} + // NewPeerstore creates a peerstore backed by the provided persistent datastore. -func NewPeerstore(ctx context.Context, store ds.Batching, opts Options) (peerstore.Peerstore, error) { +func NewPeerstore(ctx context.Context, store ds.Batching, opts Options) (*pstoreds, error) { addrBook, err := NewAddrBook(ctx, store, opts) if err != nil { return nil, err @@ -66,7 +77,13 @@ func NewPeerstore(ctx context.Context, store ds.Batching, opts Options) (peersto protoBook := NewProtoBook(peerMetadata) - ps := pstore.NewPeerstore(keyBook, addrBook, protoBook, peerMetadata) + ps := &pstoreds{ + Metrics: pstore.NewMetrics(), + dsKeyBook: *keyBook, + dsAddrBook: *addrBook, + dsPeerMetadata: *peerMetadata, + dsProtoBook: *protoBook, + } return ps, nil } @@ -103,3 +120,47 @@ func uniquePeerIds(ds ds.Datastore, prefix ds.Key, extractor func(result query.R } return ids, nil } + +func (ps *pstoreds) Close() (err error) { + var errs []error + weakClose := func(name string, c interface{}) { + if cl, ok := c.(io.Closer); ok { + if err = cl.Close(); err != nil { + errs = append(errs, fmt.Errorf("%s error: %s", name, err)) + } + } + } + + weakClose("keybook", ps.dsKeyBook) + weakClose("addressbook", ps.dsAddrBook) + weakClose("protobook", ps.dsProtoBook) + weakClose("peermetadata", ps.dsPeerMetadata) + + if len(errs) > 0 { + return fmt.Errorf("failed while closing peerstore; err(s): %q", errs) + } + return nil +} + +func (ps *pstoreds) Peers() peer.IDSlice { + set := map[peer.ID]struct{}{} + for _, p := range ps.PeersWithKeys() { + set[p] = struct{}{} + } + for _, p := range ps.PeersWithAddrs() { + set[p] = struct{}{} + } + + pps := make(peer.IDSlice, 0, len(set)) + for p := range set { + pps = append(pps, p) + } + return pps +} + +func (ps *pstoreds) PeerInfo(p peer.ID) peer.AddrInfo { + return peer.AddrInfo{ + ID: p, + Addrs: ps.dsAddrBook.Addrs(p), + } +} diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go index fc084d6619..27e70f556d 100644 --- a/p2p/host/peerstore/pstoreds/protobook.go +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -26,7 +26,7 @@ type dsProtoBook struct { var _ pstore.ProtoBook = (*dsProtoBook)(nil) -func NewProtoBook(meta pstore.PeerMetadata) pstore.ProtoBook { +func NewProtoBook(meta pstore.PeerMetadata) *dsProtoBook { return &dsProtoBook{ meta: meta, segments: func() (ret protoSegments) { diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index ec0b1c92c4..c0db4f5d8e 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -2,16 +2,18 @@ package pstoremem import ( "context" + "fmt" "sort" "sync" "time" logging "github.com/ipfs/go-log" - peer "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peer" + pstore "github.com/libp2p/go-libp2p-core/peerstore" + "github.com/libp2p/go-libp2p-core/record" ma "github.com/multiformats/go-multiaddr" - pstore "github.com/libp2p/go-libp2p-core/peerstore" - addr "github.com/libp2p/go-libp2p-peerstore/addr" + "github.com/libp2p/go-libp2p-peerstore/addr" ) var log = logging.Logger("peerstore") @@ -26,6 +28,11 @@ func (e *expiringAddr) ExpiredBy(t time.Time) bool { return t.After(e.Expires) } +type peerRecordState struct { + Envelope *record.Envelope + Seq uint64 +} + type addrSegments [256]*addrSegment type addrSegment struct { @@ -35,10 +42,12 @@ type addrSegment struct { // space unused. storing the *values* directly in the map will // drastically increase the space waste. In our case, by 6x. addrs map[peer.ID]map[string]*expiringAddr + + signedPeerRecords map[peer.ID]*peerRecordState } -func (s *addrSegments) get(p peer.ID) *addrSegment { - return s[byte(p[len(p)-1])] +func (segments *addrSegments) get(p peer.ID) *addrSegment { + return segments[byte(p[len(p)-1])] } // memoryAddrBook manages addresses. @@ -52,14 +61,17 @@ type memoryAddrBook struct { } var _ pstore.AddrBook = (*memoryAddrBook)(nil) +var _ pstore.CertifiedAddrBook = (*memoryAddrBook)(nil) -func NewAddrBook() pstore.AddrBook { +func NewAddrBook() *memoryAddrBook { ctx, cancel := context.WithCancel(context.Background()) ab := &memoryAddrBook{ segments: func() (ret addrSegments) { for i, _ := range ret { - ret[i] = &addrSegment{addrs: make(map[peer.ID]map[string]*expiringAddr)} + ret[i] = &addrSegment{ + addrs: make(map[peer.ID]map[string]*expiringAddr), + signedPeerRecords: make(map[peer.ID]*peerRecordState)} } return ret }(), @@ -98,6 +110,7 @@ func (mab *memoryAddrBook) gc() { now := time.Now() for _, s := range mab.segments { s.Lock() + var collectedPeers []peer.ID for p, amap := range s.addrs { for k, addr := range amap { if addr.ExpiredBy(now) { @@ -106,23 +119,30 @@ func (mab *memoryAddrBook) gc() { } if len(amap) == 0 { delete(s.addrs, p) + collectedPeers = append(collectedPeers, p) } } + // remove signed records for peers whose signed addrs have all been removed + for _, p := range collectedPeers { + delete(s.signedPeerRecords, p) + } s.Unlock() } - } func (mab *memoryAddrBook) PeersWithAddrs() peer.IDSlice { - var pids peer.IDSlice + // deduplicate, since the same peer could have both signed & unsigned addrs + pidSet := peer.NewSet() for _, s := range mab.segments { s.RLock() - for pid, _ := range s.addrs { - pids = append(pids, pid) + for pid, amap := range s.addrs { + if amap != nil && len(amap) > 0 { + pidSet.Add(pid) + } } s.RUnlock() } - return pids + return pidSet.Peers() } // AddAddr calls AddAddrs(p, []ma.Multiaddr{addr}, ttl) @@ -134,6 +154,48 @@ func (mab *memoryAddrBook) AddAddr(p peer.ID, addr ma.Multiaddr, ttl time.Durati // (time-to-live), after which the address is no longer valid. // This function never reduces the TTL or expiration of an address. func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + // if we have a valid peer record, ignore unsigned addrs + // peerRec := mab.GetPeerRecord(p) + // if peerRec != nil { + // return + // } + mab.addAddrs(p, addrs, ttl, false) +} + +// ConsumePeerRecord adds addresses from a signed peer.PeerRecord (contained in +// a record.Envelope), which will expire after the given TTL. +// See https://godoc.org/github.com/libp2p/go-libp2p-core/peerstore#CertifiedAddrBook for more details. +func (mab *memoryAddrBook) ConsumePeerRecord(recordEnvelope *record.Envelope, ttl time.Duration) (bool, error) { + r, err := recordEnvelope.Record() + if err != nil { + return false, err + } + rec, ok := r.(*peer.PeerRecord) + if !ok { + return false, fmt.Errorf("unable to process envelope: not a PeerRecord") + } + if !rec.PeerID.MatchesPublicKey(recordEnvelope.PublicKey) { + return false, fmt.Errorf("signing key does not match PeerID in PeerRecord") + } + + // ensure seq is greater than last received + s := mab.segments.get(rec.PeerID) + s.Lock() + lastState, found := s.signedPeerRecords[rec.PeerID] + if found && lastState.Seq >= rec.Seq { + s.Unlock() + return false, nil + } + s.signedPeerRecords[rec.PeerID] = &peerRecordState{ + Envelope: recordEnvelope, + Seq: rec.Seq, + } + s.Unlock() // need to release the lock, since addAddrs will try to take it + mab.addAddrs(rec.PeerID, rec.Addrs, ttl, true) + return true, nil +} + +func (mab *memoryAddrBook) addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, signed bool) { // if ttl is zero, exit. nothing to do. if ttl <= 0 { return @@ -143,26 +205,33 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du s.Lock() defer s.Unlock() - amap := s.addrs[p] - if amap == nil { - amap = make(map[string]*expiringAddr, len(addrs)) + amap, ok := s.addrs[p] + if !ok { + amap = make(map[string]*expiringAddr) s.addrs[p] = amap } + exp := time.Now().Add(ttl) + addrSet := make(map[string]struct{}, len(addrs)) for _, addr := range addrs { if addr == nil { log.Warnf("was passed nil multiaddr for %s", p) continue } - asBytes := addr.Bytes() - a, found := amap[string(asBytes)] // won't allocate. + k := string(addr.Bytes()) + addrSet[k] = struct{}{} + + // find the highest TTL and Expiry time between + // existing records and function args + a, found := amap[k] // won't allocate. + if !found { - // not found, save and announce it. - amap[string(asBytes)] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} + // not found, announce it. + entry := &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} + amap[k] = entry mab.subManager.BroadcastAddr(p, addr) } else { - // Update expiration/TTL independently. - // We never want to reduce either. + // update ttl & exp to whichever is greater between new and existing entry if ttl > a.TTL { a.TTL = ttl } @@ -171,6 +240,21 @@ func (mab *memoryAddrBook) AddAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du } } } + + // // when adding signed addrs, make sure only the addrs from the input list remain. + // if signed { + // for k := range amap { + // _, ok := addrSet[k] + // if !ok { + // delete(amap, k) + // } + // } + // } + + // if we've expired all the signed addresses for a peer, remove their signed routing state record + if len(addrs) == 0 { + delete(s.signedPeerRecords, p) + } } // SetAddr calls mgr.SetAddrs(p, addr, ttl) @@ -185,9 +269,9 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du s.Lock() defer s.Unlock() - amap := s.addrs[p] - if amap == nil { - amap = make(map[string]*expiringAddr, len(addrs)) + amap, ok := s.addrs[p] + if !ok { + amap = make(map[string]*expiringAddr) s.addrs[p] = amap } @@ -197,16 +281,22 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du log.Warnf("was passed nil multiaddr for %s", p) continue } + aBytes := addr.Bytes() + key := string(aBytes) // re-set all of them for new ttl. - aBytes := addr.Bytes() if ttl > 0 { - amap[string(aBytes)] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} + amap[key] = &expiringAddr{Addr: addr, Expires: exp, TTL: ttl} mab.subManager.BroadcastAddr(p, addr) } else { - delete(amap, string(aBytes)) + delete(amap, key) } } + + // if we've expired all the signed addresses for a peer, remove their signed routing state record + if len(amap) == 0 { + delete(s.signedPeerRecords, p) + } } // UpdateAddrs updates the addresses associated with the given peer that have @@ -215,35 +305,39 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t s := mab.segments.get(p) s.Lock() defer s.Unlock() - + exp := time.Now().Add(newTTL) amap, found := s.addrs[p] - if !found { - return + if found { + for k, a := range amap { + if oldTTL == a.TTL { + a.TTL = newTTL + a.Expires = exp + amap[k] = a + } + } } - exp := time.Now().Add(newTTL) - for k, addr := range amap { - if oldTTL == addr.TTL { - addr.TTL = newTTL - addr.Expires = exp - amap[k] = addr - } + // if we've expired all the signed addresses for a peer, remove their signed routing state record + if len(amap) == 0 { + delete(s.signedPeerRecords, p) } } -// Addresses returns all known (and valid) addresses for a given +// Addrs returns all known (and valid) addresses for a given peer func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { s := mab.segments.get(p) s.RLock() defer s.RUnlock() - amap, found := s.addrs[p] - if !found { - return nil - } + return validAddrs(s.addrs[p]) +} +func validAddrs(amap map[string]*expiringAddr) []ma.Multiaddr { now := time.Now() good := make([]ma.Multiaddr, 0, len(amap)) + if amap == nil { + return good + } for _, m := range amap { if !m.ExpiredBy(now) { good = append(good, m.Addr) @@ -253,6 +347,28 @@ func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { return good } +// GetPeerRecord returns a Envelope containing a PeerRecord for the +// given peer id, if one exists. +// Returns nil if no signed PeerRecord exists for the peer. +func (mab *memoryAddrBook) GetPeerRecord(p peer.ID) *record.Envelope { + s := mab.segments.get(p) + s.RLock() + defer s.RUnlock() + + // although the signed record gets garbage collected when all addrs inside it are expired, + // we may be in between the expiration time and the GC interval + // so, we check to see if we have any valid signed addrs before returning the record + if len(validAddrs(s.addrs[p])) == 0 { + return nil + } + + state := s.signedPeerRecords[p] + if state == nil { + return nil + } + return state.Envelope +} + // ClearAddrs removes all previously stored addresses func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { s := mab.segments.get(p) @@ -260,6 +376,7 @@ func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { defer s.Unlock() delete(s.addrs, p) + delete(s.signedPeerRecords, p) } // AddrStream returns a channel on which all new addresses discovered for a diff --git a/p2p/host/peerstore/pstoremem/keybook.go b/p2p/host/peerstore/pstoremem/keybook.go index f6ab84d236..79f2550b92 100644 --- a/p2p/host/peerstore/pstoremem/keybook.go +++ b/p2p/host/peerstore/pstoremem/keybook.go @@ -19,7 +19,7 @@ type memoryKeyBook struct { var _ pstore.KeyBook = (*memoryKeyBook)(nil) // noop new, but in the future we may want to do some init work. -func NewKeyBook() pstore.KeyBook { +func NewKeyBook() *memoryKeyBook { return &memoryKeyBook{ pks: map[peer.ID]ic.PubKey{}, sks: map[peer.ID]ic.PrivKey{}, diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index 821c2b6a51..7ded769192 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -27,7 +27,7 @@ type memoryPeerMetadata struct { var _ pstore.PeerMetadata = (*memoryPeerMetadata)(nil) -func NewPeerMetadata() pstore.PeerMetadata { +func NewPeerMetadata() *memoryPeerMetadata { return &memoryPeerMetadata{ ds: make(map[metakey]interface{}), interned: make(map[string]interface{}), diff --git a/p2p/host/peerstore/pstoremem/peerstore.go b/p2p/host/peerstore/pstoremem/peerstore.go index c7cbd672f2..f16887de22 100644 --- a/p2p/host/peerstore/pstoremem/peerstore.go +++ b/p2p/host/peerstore/pstoremem/peerstore.go @@ -1,12 +1,73 @@ package pstoremem -import pstore "github.com/libp2p/go-libp2p-peerstore" +import ( + "fmt" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/peerstore" + pstore "github.com/libp2p/go-libp2p-peerstore" + "io" +) + +type pstoremem struct { + peerstore.Metrics + + memoryKeyBook + memoryAddrBook + memoryProtoBook + memoryPeerMetadata +} // NewPeerstore creates an in-memory threadsafe collection of peers. -func NewPeerstore() pstore.Peerstore { - return pstore.NewPeerstore( - NewKeyBook(), - NewAddrBook(), - NewProtoBook(), - NewPeerMetadata()) +func NewPeerstore() *pstoremem { + return &pstoremem{ + Metrics: pstore.NewMetrics(), + memoryKeyBook: *NewKeyBook(), + memoryAddrBook: *NewAddrBook(), + memoryProtoBook: *NewProtoBook(), + memoryPeerMetadata: *NewPeerMetadata(), + } +} + +func (ps *pstoremem) Close() (err error) { + var errs []error + weakClose := func(name string, c interface{}) { + if cl, ok := c.(io.Closer); ok { + if err = cl.Close(); err != nil { + errs = append(errs, fmt.Errorf("%s error: %s", name, err)) + } + } + } + + weakClose("keybook", ps.memoryKeyBook) + weakClose("addressbook", ps.memoryAddrBook) + weakClose("protobook", ps.memoryProtoBook) + weakClose("peermetadata", ps.memoryPeerMetadata) + + if len(errs) > 0 { + return fmt.Errorf("failed while closing peerstore; err(s): %q", errs) + } + return nil +} + +func (ps *pstoremem) Peers() peer.IDSlice { + set := map[peer.ID]struct{}{} + for _, p := range ps.PeersWithKeys() { + set[p] = struct{}{} + } + for _, p := range ps.PeersWithAddrs() { + set[p] = struct{}{} + } + + pps := make(peer.IDSlice, 0, len(set)) + for p := range set { + pps = append(pps, p) + } + return pps +} + +func (ps *pstoremem) PeerInfo(p peer.ID) peer.AddrInfo { + return peer.AddrInfo{ + ID: p, + Addrs: ps.memoryAddrBook.Addrs(p), + } } diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index d627c6c7bd..04d8ec47ae 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -28,7 +28,7 @@ type memoryProtoBook struct { var _ pstore.ProtoBook = (*memoryProtoBook)(nil) -func NewProtoBook() pstore.ProtoBook { +func NewProtoBook() *memoryProtoBook { return &memoryProtoBook{ interned: make(map[string]string, 256), segments: func() (ret protoSegments) { diff --git a/p2p/host/peerstore/test/addr_book_suite.go b/p2p/host/peerstore/test/addr_book_suite.go index ea6078cf61..f05a4186be 100644 --- a/p2p/host/peerstore/test/addr_book_suite.go +++ b/p2p/host/peerstore/test/addr_book_suite.go @@ -1,6 +1,11 @@ package test import ( + "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/record" + "github.com/libp2p/go-libp2p-core/test" + "github.com/multiformats/go-multiaddr" "testing" "time" @@ -16,6 +21,7 @@ var addressBookSuite = map[string]func(book pstore.AddrBook) func(*testing.T){ "AddressesExpire": testAddressesExpire, "ClearWithIter": testClearWithIterator, "PeersWithAddresses": testPeersWithAddrs, + "CertifiedAddresses": testCertifiedAddresses, } type AddrBookFactory func() (pstore.AddrBook, func()) @@ -168,6 +174,30 @@ func testSetNegativeTTLClears(m pstore.AddrBook) func(t *testing.T) { survivors = append(survivors[0:74], survivors[75:]...) AssertAddressesEqual(t, survivors, m.Addrs(id)) + + // remove _all_ the addresses + m.SetAddrs(id, survivors, -1) + if len(m.Addrs(id)) != 0 { + t.Error("expected empty address list after clearing all addresses") + } + + // add half, but try to remove more than we added + m.SetAddrs(id, addrs[:50], time.Hour) + m.SetAddrs(id, addrs, -1) + if len(m.Addrs(id)) != 0 { + t.Error("expected empty address list after clearing all addresses") + } + + // try to remove the same addr multiple times + m.SetAddrs(id, addrs[:5], time.Hour) + repeated := make([]multiaddr.Multiaddr, 10) + for i := 0; i < len(repeated); i++ { + repeated[i] = addrs[0] + } + m.SetAddrs(id, repeated, -1) + if len(m.Addrs(id)) != 4 { + t.Errorf("expected 4 addrs after removing one, got %d", len(m.Addrs(id))) + } } } @@ -331,3 +361,113 @@ func testPeersWithAddrs(m pstore.AddrBook) func(t *testing.T) { }) } } + +func testCertifiedAddresses(m pstore.AddrBook) func(*testing.T) { + return func(t *testing.T) { + cab := m.(pstore.CertifiedAddrBook) + + priv, _, err := test.RandTestKeyPair(crypto.Ed25519, 256) + if err != nil { + t.Errorf("error generating testing keys: %v", err) + } + + id, _ := peer.IDFromPrivateKey(priv) + allAddrs := GenerateAddrs(10) + certifiedAddrs := allAddrs[:5] + uncertifiedAddrs := allAddrs[5:] + rec := peer.NewPeerRecord() + rec.PeerID = id + rec.Addrs = certifiedAddrs + signedRec, err := record.Seal(rec, priv) + if err != nil { + t.Errorf("error creating signed routing record: %v", err) + } + + // add a few non-certified addrs + m.AddAddrs(id, uncertifiedAddrs, time.Hour) + + // make sure they're present + AssertAddressesEqual(t, uncertifiedAddrs, m.Addrs(id)) + + // add the signed record to addr book + _, err = cab.ConsumePeerRecord(signedRec, time.Hour) + if err != nil { + t.Errorf("error adding signed routing record to addrbook: %v", err) + } + + // the non-certified addrs should be gone & we should get only certified addrs back from Addrs + // AssertAddressesEqual(t, certifiedAddrs, m.Addrs(id)) + AssertAddressesEqual(t, allAddrs, m.Addrs(id)) + + // PeersWithAddrs should return a single peer + if len(m.PeersWithAddrs()) != 1 { + t.Errorf("expected PeersWithAddrs to return 1, got %d", len(m.PeersWithAddrs())) + } + + // adding the same peer record again should result in the record being ignored + accepted, err := cab.ConsumePeerRecord(signedRec, time.Hour) + if accepted { + t.Error("Expected record with duplicate sequence number to be ignored") + } + if err != nil { + t.Errorf("Expected record with duplicate sequence number to be ignored without error, got err: %s", err) + } + + // once certified addrs exist, trying to add non-certified addrs should have no effect + // m.AddAddrs(id, uncertifiedAddrs, time.Hour) + // AssertAddressesEqual(t, certifiedAddrs, m.Addrs(id)) + m.AddAddrs(id, uncertifiedAddrs, time.Hour) + AssertAddressesEqual(t, allAddrs, m.Addrs(id)) + + // we should be able to retrieve the signed peer record + rec2 := cab.GetPeerRecord(id) + if rec2 == nil || !signedRec.Equal(rec2) { + t.Error("unable to retrieve signed routing record from addrbook") + } + + // Adding a new envelope should clear existing certified addresses. + // Only the newly-added ones should remain + certifiedAddrs = certifiedAddrs[:3] + rec = peer.NewPeerRecord() + rec.PeerID = id + rec.Addrs = certifiedAddrs + signedRec, err = record.Seal(rec, priv) + test.AssertNilError(t, err) + _, err = cab.ConsumePeerRecord(signedRec, time.Hour) + test.AssertNilError(t, err) + // AssertAddressesEqual(t, certifiedAddrs, m.Addrs(id)) + AssertAddressesEqual(t, allAddrs, m.Addrs(id)) + + // update TTL on signed addrs to -1 to remove them. + // the signed routing record should be deleted + // m.SetAddrs(id, certifiedAddrs, -1) + m.SetAddrs(id, allAddrs, -1) + if len(m.Addrs(id)) != 0 { + t.Error("expected zero certified addrs after setting TTL to -1") + } + if cab.GetPeerRecord(id) != nil { + t.Error("expected signed peer record to be removed when addresses expire") + } + + // Test that natural TTL expiration clears signed peer records + _, err = cab.ConsumePeerRecord(signedRec, time.Second) + test.AssertNilError(t, err) + AssertAddressesEqual(t, certifiedAddrs, m.Addrs(id)) + + time.Sleep(2 * time.Second) + if cab.GetPeerRecord(id) != nil { + t.Error("expected signed peer record to be removed when addresses expire") + } + + // adding a peer record that's signed with the wrong key should fail + priv2, _, err := test.RandTestKeyPair(crypto.Ed25519, 256) + test.AssertNilError(t, err) + env, err := record.Seal(rec, priv2) + test.AssertNilError(t, err) + + accepted, err = cab.ConsumePeerRecord(env, time.Second) + if accepted || err == nil { + t.Error("expected adding a PeerRecord that's signed with the wrong key to fail") + } + } +} diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index a0ae326c9a..8a629624be 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -23,6 +23,7 @@ var peerstoreSuite = map[string]func(pstore.Peerstore) func(*testing.T){ "PeerstoreProtoStore": testPeerstoreProtoStore, "BasicPeerstore": testBasicPeerstore, "Metadata": testMetadata, + "CertifiedAddrBook": testCertifiedAddrBook, } type PeerstoreFactory func() (pstore.Peerstore, func()) @@ -357,6 +358,15 @@ func testMetadata(ps pstore.Peerstore) func(t *testing.T) { } } +func testCertifiedAddrBook(ps pstore.Peerstore) func(*testing.T) { + return func(t *testing.T) { + _, ok := ps.(pstore.CertifiedAddrBook) + if !ok { + t.Error("expected peerstore to implement CertifiedAddrBook interface") + } + } +} + func getAddrs(t *testing.T, n int) []ma.Multiaddr { var addrs []ma.Multiaddr for i := 0; i < n; i++ { From 41ac79c8352801dace673dc8479fdd5bee39ebd0 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Fri, 6 Mar 2020 11:27:34 -0800 Subject: [PATCH 1708/3965] mitigate race in test --- p2p/host/autonat/svc_test.go | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 979854d430..a36340913c 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -139,7 +139,6 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { func TestAutoNATServiceRateLimitJitter(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) - defer cancel() save1 := AutoNATServiceResetInterval AutoNATServiceResetInterval = 100 * time.Millisecond @@ -147,12 +146,19 @@ func TestAutoNATServiceRateLimitJitter(t *testing.T) { AutoNATServiceResetJitter = 100 * time.Millisecond _, svc := makeAutoNATService(ctx, t) + svc.mx.Lock() svc.globalReqs = 1 + svc.mx.Unlock() time.Sleep(200 * time.Millisecond) + + svc.mx.Lock() + defer svc.mx.Unlock() if svc.globalReqs != 0 { t.Fatal("reset of rate limitter occured slower than expected") } + cancel() + AutoNATServiceResetInterval = save1 AutoNATServiceResetJitter = save2 } From b8f591a4bec903217c96dad57e0389bef5ed6fbb Mon Sep 17 00:00:00 2001 From: Hlib Kanunnikov Date: Sat, 7 Mar 2020 03:13:03 +0200 Subject: [PATCH 1709/3965] Respect mux.ErrReset (#113) --- p2p/transport/quic/stream.go | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/p2p/transport/quic/stream.go b/p2p/transport/quic/stream.go index e757de20dd..313ea227ec 100644 --- a/p2p/transport/quic/stream.go +++ b/p2p/transport/quic/stream.go @@ -6,14 +6,36 @@ import ( quic "github.com/lucas-clemente/quic-go" ) +const ( + reset quic.ErrorCode = 0 +) + type stream struct { quic.Stream } -var _ mux.MuxedStream = &stream{} +func (s *stream) Read(b []byte) (n int, err error) { + n, err = s.Stream.Read(b) + if serr, ok := err.(quic.StreamError); ok && serr.Canceled() { + err = mux.ErrReset + } + + return n, err +} + +func (s *stream) Write(b []byte) (n int, err error) { + n, err = s.Stream.Write(b) + if serr, ok := err.(quic.StreamError); ok && serr.Canceled() { + err = mux.ErrReset + } + + return n, err +} func (s *stream) Reset() error { - s.Stream.CancelRead(0) - s.Stream.CancelWrite(0) + s.Stream.CancelRead(reset) + s.Stream.CancelWrite(reset) return nil } + +var _ mux.MuxedStream = &stream{} From 9b5e9872b6954ff5dbe17da4606947b184a80566 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 7 Mar 2020 08:18:20 +0700 Subject: [PATCH 1710/3965] remove the Protector interface, introduce a PSK type (#118) * remove the Protector interface, introduce a PSK type * move decoding of the v1 PSK here --- core/pnet/codec.go | 66 ++++++++++++++++++++++ core/pnet/codec_test.go | 122 ++++++++++++++++++++++++++++++++++++++++ core/pnet/protector.go | 16 ++---- 3 files changed, 192 insertions(+), 12 deletions(-) create mode 100644 core/pnet/codec.go create mode 100644 core/pnet/codec_test.go diff --git a/core/pnet/codec.go b/core/pnet/codec.go new file mode 100644 index 0000000000..e741aba995 --- /dev/null +++ b/core/pnet/codec.go @@ -0,0 +1,66 @@ +package pnet + +import ( + "bufio" + "bytes" + "encoding/base64" + "encoding/hex" + "fmt" + "io" +) + +var ( + pathPSKv1 = []byte("/key/swarm/psk/1.0.0/") + pathBin = "/bin/" + pathBase16 = "/base16/" + pathBase64 = "/base64/" +) + +func readHeader(r *bufio.Reader) ([]byte, error) { + header, err := r.ReadBytes('\n') + if err != nil { + return nil, err + } + + return bytes.TrimRight(header, "\r\n"), nil +} + +func expectHeader(r *bufio.Reader, expected []byte) error { + header, err := readHeader(r) + if err != nil { + return err + } + if !bytes.Equal(header, expected) { + return fmt.Errorf("expected file header %s, got: %s", pathPSKv1, header) + } + return nil +} + +// DecodeV1PSK reads a Multicodec encoded V1 PSK. +func DecodeV1PSK(in io.Reader) (PSK, error) { + reader := bufio.NewReader(in) + if err := expectHeader(reader, pathPSKv1); err != nil { + return nil, err + } + header, err := readHeader(reader) + if err != nil { + return nil, err + } + + var decoder io.Reader + switch string(header) { + case pathBase16: + decoder = hex.NewDecoder(reader) + case pathBase64: + decoder = base64.NewDecoder(base64.StdEncoding, reader) + case pathBin: + decoder = reader + default: + return nil, fmt.Errorf("unknown encoding: %s", header) + } + out := make([]byte, 32) + if _, err = io.ReadFull(decoder, out[:]); err != nil { + return nil, err + } + return out, nil +} diff --git a/core/pnet/codec_test.go b/core/pnet/codec_test.go new file mode 100644 index 0000000000..b4b9272d09 --- /dev/null +++ b/core/pnet/codec_test.go @@ -0,0 +1,122 @@ +package pnet + +import ( + "bytes" + "encoding/base64" + "testing" +) + +func bufWithBase(base string, windows bool) *bytes.Buffer { + b := &bytes.Buffer{} + b.Write(pathPSKv1) + if windows { + b.WriteString("\r") + } + b.WriteString("\n") + b.WriteString(base) + if windows { + b.WriteString("\r") + } + b.WriteString("\n") + return b +} + +func TestDecodeHex(t *testing.T) { + testDecodeHex(t, true) + testDecodeHex(t, false) +} + +func TestDecodeBad(t *testing.T) { + testDecodeBad(t, true) + testDecodeBad(t, false) +} + +func testDecodeBad(t *testing.T, windows bool) { + b := bufWithBase("/verybadbase/", windows) + b.WriteString("Have fun decoding that key") + + _, err := DecodeV1PSK(b) + if err == nil { + t.Fatal("expected 'unknown encoding' got nil") + } +} + +func testDecodeHex(t *testing.T, windows bool) { + b := bufWithBase("/base16/", windows) + for i := 0; i < 32; i++ { + b.WriteString("FF") + } + + psk, err := DecodeV1PSK(b) + if err != nil { + t.Fatal(err) + } + + for _, b := range psk { + if b != 255 { + t.Fatal("byte was wrong") + } + } +} + +func TestDecodeB64(t *testing.T) { + testDecodeB64(t, true) + testDecodeB64(t, false) +} + +func testDecodeB64(t *testing.T, windows bool) { + b := bufWithBase("/base64/", windows) + key := make([]byte, 32) + for i := 0; i < 32; i++ { + key[i] = byte(i) + } + + e := base64.NewEncoder(base64.StdEncoding, b) + _, err := e.Write(key) + if err != nil { + t.Fatal(err) + } + err = e.Close() + if err != nil { + t.Fatal(err) + } + + psk, err := DecodeV1PSK(b) + if err != nil { + t.Fatal(err) + } + + for i, b := range psk { + if b != psk[i] { + t.Fatal("byte was wrong") + } + } + +} + +func TestDecodeBin(t *testing.T) { + testDecodeBin(t, true) + testDecodeBin(t, false) +} + +func testDecodeBin(t *testing.T, windows bool) { + b := bufWithBase("/bin/", windows) + key := make([]byte, 32) + for i := 0; i < 32; i++ { + key[i] = byte(i) + } + + b.Write(key) + + psk, err := DecodeV1PSK(b) + if err != nil { + t.Fatal(err) + } + + for i, b := range psk { + if b != psk[i] { + t.Fatal("byte was wrong") + } + } + +} diff --git a/core/pnet/protector.go b/core/pnet/protector.go index c443ee8fda..9d9dce9265 100644 --- a/core/pnet/protector.go +++ b/core/pnet/protector.go @@ -1,15 +1,7 @@ // Package pnet provides interfaces for private networking in libp2p. package pnet -import "net" - -// Protector interface is a way for private network implementation to be transparent in -// libp2p. It is created by implementation and use by libp2p-conn to secure connections -// so they can be only established with selected number of peers. -type Protector interface { - // Wraps passed connection to protect it - Protect(net.Conn) (net.Conn, error) - - // Returns key fingerprint that is safe to expose - Fingerprint() []byte -} +// A PSK enables private network implementation to be transparent in libp2p. +// It is used to ensure that peers can only establish connections to other peers +// that are using the same PSK. +type PSK []byte From 650cf9f0b314677a247010512b5b56465a40ffe7 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 6 Mar 2020 17:18:54 -0800 Subject: [PATCH 1711/3965] doc(event): document network events (#129) Add extensive documentation to network events to explain the edge-cases. Co-authored-by: Will --- core/event/network.go | 51 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 45 insertions(+), 6 deletions(-) diff --git a/core/event/network.go b/core/event/network.go index 05b435d287..15f9f6c898 100644 --- a/core/event/network.go +++ b/core/event/network.go @@ -5,12 +5,51 @@ import ( "github.com/libp2p/go-libp2p-core/peer" ) -// EvtPeerConnectednessChanged should be emitted every time we form a connection with a peer or drop our last -// connection with the peer. Essentially, it is emitted in two cases: -// a) We form a/any connection with a peer. -// b) We go from having a connection/s with a peer to having no connection with the peer. -// It contains the Id of the remote peer and the new connectedness state. +// EvtPeerConnectednessChanged should be emitted every time the "connectedness" to a +// given peer changes. Specifically, this event is emitted in the following +// cases: +// +// * Connectedness = Connected: Every time we transition from having no +// connections to a peer to having at least one connection to the peer. +// * Connectedness = NotConnected: Every time we transition from having at least +// one connection to a peer to having no connections to the peer. +// +// Additional connectedness states may be added in the future. This list should +// not be considered exhaustive. +// +// Take note: +// +// * It's possible to have _multiple_ connections to a given peer. +// * Both libp2p and networks are asynchronous. +// +// This means that all of the following situations are possible: +// +// A connection is cut and is re-established: +// +// * Peer A observes a transition from Connected -> NotConnected -> Connected +// * Peer B observes a transition from Connected -> NotConnected -> Connected +// +// Explanation: Both peers observe the connection die. This is the "nice" case. +// +// A connection is cut and is re-established. +// +// * Peer A observes a transition from Connected -> NotConnected -> Connected. +// * Peer B observes no transition. +// +// Explanation: Peer A re-establishes the dead connection. Peer B observes the +// new connection form before it observes the old connection die. +// +// A connection is cut: +// +// * Peer A observes no transition. +// * Peer B observes no transition. +// +// Explanation: There were two connections and one was cut. This connection +// might have been in active use but neither peer will observe a change in +// "connectedness". Peers should always make sure to re-try network requests. type EvtPeerConnectednessChanged struct { - Peer peer.ID + // Peer is the remote peer who's connectedness has changed. + Peer peer.ID + // Connectedness is the new connectedness state. Connectedness network.Connectedness } From eea1dd74b53de8e9031f9a33c26907ac7a9e7996 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 20 Feb 2020 12:45:35 +0700 Subject: [PATCH 1712/3965] accept a PSK in the transport constructor (and reject it) --- p2p/transport/quic/conn_test.go | 22 +++++++++++----------- p2p/transport/quic/listener_test.go | 2 +- p2p/transport/quic/transport.go | 11 +++++++++-- 3 files changed, 21 insertions(+), 14 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index 40a6bca89a..ef234bc5bc 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -62,12 +62,12 @@ var _ = Describe("Connection", func() { }) It("handshakes on IPv4", func() { - serverTransport, err := NewTransport(serverKey) + serverTransport, err := NewTransport(serverKey, nil) Expect(err).ToNot(HaveOccurred()) ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") defer ln.Close() - clientTransport, err := NewTransport(clientKey) + clientTransport, err := NewTransport(clientKey, nil) Expect(err).ToNot(HaveOccurred()) conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) @@ -86,12 +86,12 @@ var _ = Describe("Connection", func() { }) It("handshakes on IPv6", func() { - serverTransport, err := NewTransport(serverKey) + serverTransport, err := NewTransport(serverKey, nil) Expect(err).ToNot(HaveOccurred()) ln := runServer(serverTransport, "/ip6/::1/udp/0/quic") defer ln.Close() - clientTransport, err := NewTransport(clientKey) + clientTransport, err := NewTransport(clientKey, nil) Expect(err).ToNot(HaveOccurred()) conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) @@ -110,12 +110,12 @@ var _ = Describe("Connection", func() { }) It("opens and accepts streams", func() { - serverTransport, err := NewTransport(serverKey) + serverTransport, err := NewTransport(serverKey, nil) Expect(err).ToNot(HaveOccurred()) ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") defer ln.Close() - clientTransport, err := NewTransport(clientKey) + clientTransport, err := NewTransport(clientKey, nil) Expect(err).ToNot(HaveOccurred()) conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) @@ -139,11 +139,11 @@ var _ = Describe("Connection", func() { It("fails if the peer ID doesn't match", func() { thirdPartyID, _ := createPeer() - serverTransport, err := NewTransport(serverKey) + serverTransport, err := NewTransport(serverKey, nil) Expect(err).ToNot(HaveOccurred()) ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") - clientTransport, err := NewTransport(clientKey) + clientTransport, err := NewTransport(clientKey, nil) Expect(err).ToNot(HaveOccurred()) // dial, but expect the wrong peer ID _, err = clientTransport.Dial(context.Background(), ln.Multiaddr(), thirdPartyID) @@ -164,10 +164,10 @@ var _ = Describe("Connection", func() { It("dials to two servers at the same time", func() { serverID2, serverKey2 := createPeer() - serverTransport, err := NewTransport(serverKey) + serverTransport, err := NewTransport(serverKey, nil) Expect(err).ToNot(HaveOccurred()) ln1 := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") - serverTransport2, err := NewTransport(serverKey2) + serverTransport2, err := NewTransport(serverKey2, nil) defer ln1.Close() Expect(err).ToNot(HaveOccurred()) ln2 := runServer(serverTransport2, "/ip4/127.0.0.1/udp/0/quic") @@ -194,7 +194,7 @@ var _ = Describe("Connection", func() { } }() - clientTransport, err := NewTransport(clientKey) + clientTransport, err := NewTransport(clientKey, nil) Expect(err).ToNot(HaveOccurred()) c1, err := clientTransport.Dial(context.Background(), ln1.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index 85c74fa7d7..ac7716f353 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -23,7 +23,7 @@ var _ = Describe("Listener", func() { Expect(err).ToNot(HaveOccurred()) key, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(rsaKey)) Expect(err).ToNot(HaveOccurred()) - t, err = NewTransport(key) + t, err = NewTransport(key, nil) Expect(err).ToNot(HaveOccurred()) }) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 3445a638f4..6a61eae61e 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -5,17 +5,20 @@ import ( "errors" "net" + logging "github.com/ipfs/go-log" ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/pnet" tpt "github.com/libp2p/go-libp2p-core/transport" p2ptls "github.com/libp2p/go-libp2p-tls" - quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" mafmt "github.com/multiformats/go-multiaddr-fmt" manet "github.com/multiformats/go-multiaddr-net" ) +var log = logging.Logger("quic-transport") + var quicConfig = &quic.Config{ MaxIncomingStreams: 1000, MaxIncomingUniStreams: -1, // disable unidirectional streams @@ -86,7 +89,11 @@ type transport struct { var _ tpt.Transport = &transport{} // NewTransport creates a new QUIC transport -func NewTransport(key ic.PrivKey) (tpt.Transport, error) { +func NewTransport(key ic.PrivKey, psk pnet.PSK) (tpt.Transport, error) { + if len(psk) > 0 { + log.Error("QUIC doesn't support private networks yet.") + return nil, errors.New("QUIC doesn't support private networks yet") + } localPeer, err := peer.IDFromPrivateKey(key) if err != nil { return nil, err From a7b7b60b5d9a3c01607ab9915c4591ff9c95db43 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 20 Feb 2020 12:06:32 +0700 Subject: [PATCH 1713/3965] remove key serialization, construct conn from ipnet.PSK --- p2p/net/pnet/codec.go | 66 ----------------- p2p/net/pnet/codec_test.go | 123 ------------------------------- p2p/net/pnet/fingerprint.go | 24 ------ p2p/net/pnet/fingerprint_test.go | 18 ----- p2p/net/pnet/generate.go | 38 ---------- p2p/net/pnet/generate_test.go | 55 -------------- p2p/net/pnet/protector.go | 39 ++-------- p2p/net/pnet/psk_conn_test.go | 7 +- 8 files changed, 11 insertions(+), 359 deletions(-) delete mode 100644 p2p/net/pnet/codec.go delete mode 100644 p2p/net/pnet/codec_test.go delete mode 100644 p2p/net/pnet/fingerprint.go delete mode 100644 p2p/net/pnet/fingerprint_test.go delete mode 100644 p2p/net/pnet/generate.go delete mode 100644 p2p/net/pnet/generate_test.go diff --git a/p2p/net/pnet/codec.go b/p2p/net/pnet/codec.go deleted file mode 100644 index 19b929f636..0000000000 --- a/p2p/net/pnet/codec.go +++ /dev/null @@ -1,66 +0,0 @@ -package pnet - -import ( - "bufio" - "bytes" - "encoding/base64" - "encoding/hex" - "fmt" - "io" -) - -var ( - pathPSKv1 = []byte("/key/swarm/psk/1.0.0/") - pathBin = "/bin/" - pathBase16 = "/base16/" - pathBase64 = "/base64/" -) - -func readHeader(r *bufio.Reader) ([]byte, error) { - header, err := r.ReadBytes('\n') - if err != nil { - return nil, err - } - - return bytes.TrimRight(header, "\r\n"), nil -} - -func expectHeader(r *bufio.Reader, expected []byte) error { - header, err := readHeader(r) - if err != nil { - return err - } - if !bytes.Equal(header, expected) { - return fmt.Errorf("expected file header %s, got: %s", pathPSKv1, header) - } - return nil -} - -func decodeV1PSK(in io.Reader) (*[32]byte, error) { - reader := bufio.NewReader(in) - if err := expectHeader(reader, pathPSKv1); err != nil { - return nil, err - } - header, err := readHeader(reader) - if err != nil { - return nil, err - } - - var decoder io.Reader - switch string(header) { - case pathBase16: - decoder = hex.NewDecoder(reader) - case pathBase64: - decoder = base64.NewDecoder(base64.StdEncoding, reader) - case pathBin: - decoder = reader - default: - return nil, fmt.Errorf("unknown encoding: %s", header) - } - out := new([32]byte) - _, err = io.ReadFull(decoder, out[:]) - if err != nil { - return nil, err - } - return out, nil -} diff --git a/p2p/net/pnet/codec_test.go b/p2p/net/pnet/codec_test.go deleted file mode 100644 index 233f3419c6..0000000000 --- a/p2p/net/pnet/codec_test.go +++ /dev/null @@ -1,123 +0,0 @@ -package pnet - -import ( - "bytes" - "encoding/base64" - "testing" -) - -func bufWithBase(base string, windows bool) *bytes.Buffer { - - b := &bytes.Buffer{} - b.Write(pathPSKv1) - if windows { - b.WriteString("\r") - } - b.WriteString("\n") - b.WriteString(base) - if windows { - b.WriteString("\r") - } - b.WriteString("\n") - return b -} - -func TestDecodeHex(t *testing.T) { - testDecodeHex(t, true) - testDecodeHex(t, false) -} - -func TestDecodeBad(t *testing.T) { - testDecodeBad(t, true) - testDecodeBad(t, false) -} - -func testDecodeBad(t *testing.T, windows bool) { - b := bufWithBase("/verybadbase/", windows) - b.WriteString("Have fun decoding that key") - - _, err := decodeV1PSK(b) - if err == nil { - t.Fatal("expected 'unknown encoding' got nil") - } -} - -func testDecodeHex(t *testing.T, windows bool) { - b := bufWithBase("/base16/", windows) - for i := 0; i < 32; i++ { - b.WriteString("FF") - } - - psk, err := decodeV1PSK(b) - if err != nil { - t.Fatal(err) - } - - for _, b := range psk { - if b != 255 { - t.Fatal("byte was wrong") - } - } -} - -func TestDecodeB64(t *testing.T) { - testDecodeB64(t, true) - testDecodeB64(t, false) -} - -func testDecodeB64(t *testing.T, windows bool) { - b := bufWithBase("/base64/", windows) - key := make([]byte, 32) - for i := 0; i < 32; i++ { - key[i] = byte(i) - } - - e := base64.NewEncoder(base64.StdEncoding, b) - _, err := e.Write(key) - if err != nil { - t.Fatal(err) - } - err = e.Close() - if err != nil { - t.Fatal(err) - } - - psk, err := decodeV1PSK(b) - if err != nil { - t.Fatal(err) - } - - for i, b := range psk { - if b != psk[i] { - t.Fatal("byte was wrong") - } - } - -} - -func TestDecodeBin(t *testing.T) { - testDecodeBin(t, true) - testDecodeBin(t, false) -} - -func testDecodeBin(t *testing.T, windows bool) { - b := bufWithBase("/bin/", windows) - key := make([]byte, 32) - for i := 0; i < 32; i++ { - key[i] = byte(i) - } - - b.Write(key) - - psk, err := decodeV1PSK(b) - if err != nil { - t.Fatal(err) - } - - for i, b := range psk { - if b != psk[i] { - t.Fatal("byte was wrong") - } - } - -} diff --git a/p2p/net/pnet/fingerprint.go b/p2p/net/pnet/fingerprint.go deleted file mode 100644 index 978a46b625..0000000000 --- a/p2p/net/pnet/fingerprint.go +++ /dev/null @@ -1,24 +0,0 @@ -package pnet - -import ( - "golang.org/x/crypto/salsa20" - "golang.org/x/crypto/sha3" -) - -var zero64 = make([]byte, 64) - -func fingerprint(psk *[32]byte) []byte { - enc := make([]byte, 64) - - // We encrypt data first so we don't feed PSK to hash function. - // Salsa20 function is not reversible thus increasing our security margin. - salsa20.XORKeyStream(enc, zero64, []byte("finprint"), psk) - - out := make([]byte, 16) - // Then do Shake-128 hash to reduce its length. - // This way if for some reason Shake is broken and Salsa20 preimage is possible, - // attacker has only half of the bytes necessary to recreate psk. - sha3.ShakeSum128(out, enc) - - return out -} diff --git a/p2p/net/pnet/fingerprint_test.go b/p2p/net/pnet/fingerprint_test.go deleted file mode 100644 index 04b238e6b4..0000000000 --- a/p2p/net/pnet/fingerprint_test.go +++ /dev/null @@ -1,18 +0,0 @@ -package pnet - -import ( - "bytes" - "testing" -) - -var tpsk *[32]byte = &[32]byte{} - -func TestFingerprintGen(t *testing.T) { - f := fingerprint(tpsk) - exp := []byte{0x70, 0x8a, 0x75, 0xaf, 0xd0, 0x5a, 0xff, 0xb0, 0x87, 0x36, 0xcb, 0xf1, 0x7c, 0x73, 0x77, 0x3e} - - if !bytes.Equal(f, exp) { - t.Fatal("fingerprint different than expected") - } - -} diff --git a/p2p/net/pnet/generate.go b/p2p/net/pnet/generate.go deleted file mode 100644 index cf220ced22..0000000000 --- a/p2p/net/pnet/generate.go +++ /dev/null @@ -1,38 +0,0 @@ -package pnet - -import ( - "bytes" - "crypto/rand" - "encoding/hex" - "io" -) - -func newLine() io.Reader { - return bytes.NewReader([]byte("\n")) -} - -// GenerateV1PSK generates new PSK key that can be used with NewProtector -func GenerateV1PSK() (io.Reader, error) { - psk, err := GenerateV1Bytes() - if err != nil { - return nil, err - } - - hexPsk := make([]byte, len(psk)*2) - hex.Encode(hexPsk, psk[:]) - - // just a shortcut to NewReader - nr := func(b []byte) io.Reader { - return bytes.NewReader(b) - } - return io.MultiReader(nr(pathPSKv1), newLine(), nr([]byte("/base16/")), newLine(), nr(hexPsk)), nil -} - -func GenerateV1Bytes() (*[32]byte, error) { - psk := [32]byte{} - _, err := rand.Read(psk[:]) - if err != nil { - return nil, err - } - return &psk, nil -} diff --git a/p2p/net/pnet/generate_test.go b/p2p/net/pnet/generate_test.go deleted file mode 100644 index e06e4c1e7b..0000000000 --- a/p2p/net/pnet/generate_test.go +++ /dev/null @@ -1,55 +0,0 @@ -package pnet - -import ( - "bytes" - "io/ioutil" - "testing" -) - -func TestGeneratedPSKCanBeUsed(t *testing.T) { - psk, err := GenerateV1PSK() - if err != nil { - t.Fatal(err) - } - - _, err = NewProtector(psk) - if err != nil { - t.Fatal(err) - } -} - -func TestGeneratedKeysAreDifferent(t *testing.T) { - psk1, err := GenerateV1PSK() - if err != nil { - t.Fatal(err) - } - psk2, err := GenerateV1PSK() - if err != nil { - t.Fatal(err) - } - bpsk1, err := ioutil.ReadAll(psk1) - if err != nil { - t.Fatal(err) - } - bpsk2, err := ioutil.ReadAll(psk2) - if err != nil { - t.Fatal(err) - } - if bytes.Equal(bpsk1, bpsk2) { - t.Fatal("generated keys are the same") - } -} - -func TestGeneratedV1BytesAreDifferent(t *testing.T) { - b1, err := GenerateV1Bytes() - if err != nil { - t.Fatal(err) - } - b2, err := GenerateV1Bytes() - if err != nil { - t.Fatal(err) - } - if bytes.Equal(b1[:], b2[:]) { - t.Fatal("generated keys are the same") - } -} diff --git a/p2p/net/pnet/protector.go b/p2p/net/pnet/protector.go index 8b8d2dd015..ed63a690be 100644 --- a/p2p/net/pnet/protector.go +++ b/p2p/net/pnet/protector.go @@ -1,41 +1,18 @@ package pnet import ( - "fmt" - "io" + "errors" "net" ipnet "github.com/libp2p/go-libp2p-core/pnet" ) -var _ ipnet.Protector = (*protector)(nil) - -// NewProtector creates ipnet.Protector instance from a io.Reader stream -// that should include Multicodec encoded V1 PSK. -func NewProtector(input io.Reader) (ipnet.Protector, error) { - psk, err := decodeV1PSK(input) - if err != nil { - return nil, fmt.Errorf("malformed private network key: %s", err) +// NewProtectedConn creates a new protected connection +func NewProtectedConn(psk ipnet.PSK, conn net.Conn) (net.Conn, error) { + if len(psk) != 32 { + return nil, errors.New("expected 32 byte PSK") } - return NewV1ProtectorFromBytes(psk) -} - -// NewV1ProtectorFromBytes creates ipnet.Protector of the V1 version. -func NewV1ProtectorFromBytes(psk *[32]byte) (ipnet.Protector, error) { - return &protector{ - psk: psk, - fingerprint: fingerprint(psk), - }, nil -} - -type protector struct { - psk *[32]byte - fingerprint []byte -} - -func (p protector) Protect(in net.Conn) (net.Conn, error) { - return newPSKConn(p.psk, in) -} -func (p protector) Fingerprint() []byte { - return p.fingerprint + var p [32]byte + copy(p[:], psk) + return newPSKConn(&p, conn) } diff --git a/p2p/net/pnet/psk_conn_test.go b/p2p/net/pnet/psk_conn_test.go index 23108f4072..f807b1837c 100644 --- a/p2p/net/pnet/psk_conn_test.go +++ b/p2p/net/pnet/psk_conn_test.go @@ -8,16 +8,15 @@ import ( "testing" ) -var testPSK = [32]byte{} // null bytes are as good test key as any other key - func setupPSKConns(ctx context.Context, t *testing.T) (net.Conn, net.Conn) { + testPSK := make([]byte, 32) // null bytes are as good test key as any other key conn1, conn2 := net.Pipe() - psk1, err := newPSKConn(&testPSK, conn1) + psk1, err := NewProtectedConn(testPSK, conn1) if err != nil { t.Fatal(err) } - psk2, err := newPSKConn(&testPSK, conn2) + psk2, err := NewProtectedConn(testPSK, conn2) if err != nil { t.Fatal(err) } From 1c3506645167e6885bc81630dfc8ce714d4275d3 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 20 Feb 2020 12:20:42 +0700 Subject: [PATCH 1714/3965] use the ipnet.PSK instead of the ipnet.Protector for private networks --- p2p/net/upgrader/upgrader.go | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/p2p/net/upgrader/upgrader.go b/p2p/net/upgrader/upgrader.go index 69fb90bb22..a464df5c48 100644 --- a/p2p/net/upgrader/upgrader.go +++ b/p2p/net/upgrader/upgrader.go @@ -8,9 +8,10 @@ import ( "github.com/libp2p/go-libp2p-core/mux" "github.com/libp2p/go-libp2p-core/peer" - "github.com/libp2p/go-libp2p-core/pnet" + ipnet "github.com/libp2p/go-libp2p-core/pnet" "github.com/libp2p/go-libp2p-core/sec" "github.com/libp2p/go-libp2p-core/transport" + "github.com/libp2p/go-libp2p-pnet" filter "github.com/libp2p/go-maddr-filter" manet "github.com/multiformats/go-multiaddr-net" @@ -26,10 +27,10 @@ var AcceptQueueLength = 16 // Upgrader is a multistream upgrader that can upgrade an underlying connection // to a full transport connection (secure and multiplexed). type Upgrader struct { - Protector pnet.Protector - Secure sec.SecureTransport - Muxer mux.Multiplexer - Filters *filter.Filters + PSK ipnet.PSK + Secure sec.SecureTransport + Muxer mux.Multiplexer + Filters *filter.Filters } // UpgradeListener upgrades the passed multiaddr-net listener into a full libp2p-transport listener. @@ -71,17 +72,17 @@ func (u *Upgrader) upgrade(ctx context.Context, t transport.Transport, maconn ma } var conn net.Conn = maconn - if u.Protector != nil { - pconn, err := u.Protector.Protect(conn) + if u.PSK != nil { + pconn, err := pnet.NewProtectedConn(u.PSK, conn) if err != nil { conn.Close() return nil, fmt.Errorf("failed to setup private network protector: %s", err) } conn = pconn - } else if pnet.ForcePrivateNetwork { + } else if ipnet.ForcePrivateNetwork { log.Error("tried to dial with no Private Network Protector but usage" + " of Private Networks is forced by the enviroment") - return nil, pnet.ErrNotInPrivateNetwork + return nil, ipnet.ErrNotInPrivateNetwork } sconn, err := u.setupSecurity(ctx, conn, p) if err != nil { From 68f9fd0e13bce74a8743fc394d54cca860b6b03c Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 20 Feb 2020 12:25:38 +0700 Subject: [PATCH 1715/3965] change PrivateNetwork to accept a PSK, update constructor magic --- config/config.go | 6 ++--- config/constructor_types.go | 24 +++++++++--------- config/transport.go | 2 +- go.mod | 12 +++++---- go.sum | 49 +++++++++++++++++++++++++++++++------ options.go | 6 ++--- 6 files changed, 67 insertions(+), 32 deletions(-) diff --git a/config/config.go b/config/config.go index 2ace430a49..e7188b7d7b 100644 --- a/config/config.go +++ b/config/config.go @@ -56,7 +56,7 @@ type Config struct { Muxers []MsMuxC SecurityTransports []MsSecC Insecure bool - Protector pnet.Protector + PSK pnet.PSK RelayCustom bool Relay bool @@ -84,7 +84,7 @@ type Config struct { // This function consumes the config. Do not reuse it (really!). func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { // Check this early. Prevents us from even *starting* without verifying this. - if pnet.ForcePrivateNetwork && cfg.Protector == nil { + if pnet.ForcePrivateNetwork && len(cfg.PSK) == 0 { log.Error("tried to create a libp2p node with no Private" + " Network Protector but usage of Private Networks" + " is forced by the enviroment") @@ -145,7 +145,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } upgrader := new(tptu.Upgrader) - upgrader.Protector = cfg.Protector + upgrader.PSK = cfg.PSK upgrader.Filters = swrm.Filters if cfg.Insecure { upgrader.Secure = makeInsecureTransport(pid, cfg.PeerKey) diff --git a/config/constructor_types.go b/config/constructor_types.go index 6f58c266c7..ea2b71b468 100644 --- a/config/constructor_types.go +++ b/config/constructor_types.go @@ -25,7 +25,6 @@ var ( transportType = reflect.TypeOf((*transport.Transport)(nil)).Elem() muxType = reflect.TypeOf((*mux.Multiplexer)(nil)).Elem() securityType = reflect.TypeOf((*sec.SecureTransport)(nil)).Elem() - protectorType = reflect.TypeOf((*pnet.Protector)(nil)).Elem() privKeyType = reflect.TypeOf((*crypto.PrivKey)(nil)).Elem() pubKeyType = reflect.TypeOf((*crypto.PubKey)(nil)).Elem() pstoreType = reflect.TypeOf((*peerstore.Peerstore)(nil)).Elem() @@ -34,20 +33,21 @@ var ( peerIDType = reflect.TypeOf((peer.ID)("")) filtersType = reflect.TypeOf((*filter.Filters)(nil)) upgraderType = reflect.TypeOf((*tptu.Upgrader)(nil)) + pskType = reflect.TypeOf((pnet.PSK)(nil)) ) var argTypes = map[reflect.Type]constructor{ - upgraderType: func(h host.Host, u *tptu.Upgrader) interface{} { return u }, - hostType: func(h host.Host, u *tptu.Upgrader) interface{} { return h }, - networkType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Network() }, - muxType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Muxer }, - securityType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Secure }, - protectorType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Protector }, - filtersType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Filters }, - peerIDType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.ID() }, - privKeyType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore().PrivKey(h.ID()) }, - pubKeyType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore().PubKey(h.ID()) }, - pstoreType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore() }, + upgraderType: func(h host.Host, u *tptu.Upgrader) interface{} { return u }, + hostType: func(h host.Host, u *tptu.Upgrader) interface{} { return h }, + networkType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Network() }, + muxType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Muxer }, + securityType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Secure }, + pskType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.PSK }, + filtersType: func(h host.Host, u *tptu.Upgrader) interface{} { return u.Filters }, + peerIDType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.ID() }, + privKeyType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore().PrivKey(h.ID()) }, + pubKeyType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore().PubKey(h.ID()) }, + pstoreType: func(h host.Host, u *tptu.Upgrader) interface{} { return h.Peerstore() }, } func newArgTypeSet(types ...reflect.Type) map[reflect.Type]constructor { diff --git a/config/transport.go b/config/transport.go index ac8f8ee7fc..e9802525e5 100644 --- a/config/transport.go +++ b/config/transport.go @@ -30,7 +30,7 @@ var transportArgTypes = argTypes // * An address filter. // * A security transport. // * A stream multiplexer transport. -// * A private network protector. +// * A private network protection key. // // And returns a type implementing transport.Transport and, optionally, an error // (as the second argument). diff --git a/go.mod b/go.mod index a215127b59..d68d7d5cdc 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ require ( github.com/ipfs/go-cid v0.0.5 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 - github.com/ipfs/go-log v0.0.1 + github.com/ipfs/go-log v1.0.2 github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 @@ -13,7 +13,7 @@ require ( github.com/libp2p/go-libp2p-autonat v0.1.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 - github.com/libp2p/go-libp2p-core v0.3.1 + github.com/libp2p/go-libp2p-core v0.5.0 github.com/libp2p/go-libp2p-discovery v0.2.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.1 @@ -23,19 +23,21 @@ require ( github.com/libp2p/go-libp2p-secio v0.2.1 github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 - github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 + github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 github.com/libp2p/go-libp2p-yamux v0.2.1 github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 github.com/libp2p/go-ws-transport v0.2.0 github.com/miekg/dns v1.1.12 // indirect - github.com/multiformats/go-multiaddr v0.2.0 + github.com/multiformats/go-multiaddr v0.2.1 github.com/multiformats/go-multiaddr-dns v0.2.0 github.com/multiformats/go-multiaddr-net v0.1.2 github.com/multiformats/go-multistream v0.1.1 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 - golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 // indirect + golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d // indirect + golang.org/x/sync v0.0.0-20190423024810-112230192c58 // indirect + golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae // indirect ) go 1.12 diff --git a/go.sum b/go.sum index 3cfebf64c4..60823a0347 100644 --- a/go.sum +++ b/go.sum @@ -31,6 +31,8 @@ github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davidlazar/go-crypto v0.0.0-20170701192655-dcfb0a7ac018 h1:6xT9KW8zLC5IlbaIF5Q7JNieBoACT7iW0YTxQHR0in0= +github.com/davidlazar/go-crypto v0.0.0-20170701192655-dcfb0a7ac018/go.mod h1:rQYf4tfk5sSwFsnDg3qYaBxSjsD9S8+59vW0dKUgme4= github.com/dgraph-io/badger v1.5.5-0.20190226225317-8115aed38f8f/go.mod h1:VZxzAIRPHRVNRKRo6AXrX9BJegn6il06VMTZVJYCIjQ= github.com/dgraph-io/badger v1.6.0-rc1/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= github.com/dgraph-io/badger v1.6.0/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= @@ -80,8 +82,6 @@ github.com/ipfs/go-cid v0.0.2 h1:tuuKaZPU1M6HcejsO3AcYWW8sZ8MTvyxfc4uqB4eFE8= github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-cid v0.0.3 h1:UIAh32wymBpStoe83YCzwVQQ5Oy/H0FdxvUS6DJDzms= github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= -github.com/ipfs/go-cid v0.0.4 h1:UlfXKrZx1DjZoBhQHmNHLC1fK1dUJDN20Y28A7s+gJ8= -github.com/ipfs/go-cid v0.0.4/go.mod h1:4LLaPOQwmk5z9LBgQnpkivrx8BJjUyGwTXCd5Xfj6+M= github.com/ipfs/go-cid v0.0.5 h1:o0Ix8e/ql7Zb5UVUJEUfjsWCIY8t48++9lR8qi6oiJU= github.com/ipfs/go-cid v0.0.5/go.mod h1:plgt+Y5MnOey4vO4UlUazGqdbEXuFYitED67FexhXog= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= @@ -99,6 +99,10 @@ github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc= github.com/ipfs/go-log v0.0.1 h1:9XTUN/rW64BCG1YhPK9Hoy3q8nr4gOmHHBpgFdfw6Lc= github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= +github.com/ipfs/go-log v1.0.2 h1:s19ZwJxH8rPWzypjcDpqPLIyV7BnbLqvpli3iZoqYK0= +github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= +github.com/ipfs/go-log/v2 v2.0.2 h1:xguurydRdfKMJjKyxNXNU8lYP0VZH1NUwJRwUorjuEw= +github.com/ipfs/go-log/v2 v2.0.2/go.mod h1:O7P1lJt27vWHhOwQmcFEvlmo49ry2VY2+JfBWFaa9+0= github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc= github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= @@ -156,10 +160,8 @@ github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZV github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= github.com/libp2p/go-libp2p-core v0.3.0 h1:F7PqduvrztDtFsAa/bcheQ3azmNo+Nq7m8hQY5GiUW8= github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw= -github.com/libp2p/go-libp2p-core v0.3.1-0.20191230184106-204a57d1afe1 h1:3WfZnKXp/5B9EIwbV9ts0W681Q+K1/ZQdlPH2dBXAHg= -github.com/libp2p/go-libp2p-core v0.3.1-0.20191230184106-204a57d1afe1/go.mod h1:LRlbRHNBEfe8yypXy9+tfroNCfmGmYGjFGe5mU7bmWY= -github.com/libp2p/go-libp2p-core v0.3.1 h1:hEnSDjScfjYvPHoTgZhC4F62M8K1x1Oco/BY0RZ1N3s= -github.com/libp2p/go-libp2p-core v0.3.1/go.mod h1:thvWy0hvaSBhnVBaW37BvzgVV68OUhgJJLAa6almrII= +github.com/libp2p/go-libp2p-core v0.5.0 h1:FBQ1fpq2Fo/ClyjojVJ5AKXlKhvNc/B6U0O+7AN1ffE= +github.com/libp2p/go-libp2p-core v0.5.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= @@ -180,6 +182,8 @@ github.com/libp2p/go-libp2p-peerstore v0.1.3 h1:wMgajt1uM2tMiqf4M+4qWKVyyFc8SfA+ github.com/libp2p/go-libp2p-peerstore v0.1.3/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= github.com/libp2p/go-libp2p-peerstore v0.1.4 h1:d23fvq5oYMJ/lkkbO4oTwBp/JP+I/1m5gZJobNXCE/k= github.com/libp2p/go-libp2p-peerstore v0.1.4/go.mod h1:+4BDbDiiKf4PzpANZDAT+knVdLxvqh7hXOujessqdzs= +github.com/libp2p/go-libp2p-pnet v0.2.0 h1:J6htxttBipJujEjz1y0a5+eYoiPcFHhSYHH6na5f0/k= +github.com/libp2p/go-libp2p-pnet v0.2.0/go.mod h1:Qqvq6JH/oMZGwqs3N1Fqhv8NVhrdYcO0BW4wssv21LA= github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLMWdB7GTQ= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0 h1:ywzZBsWEEz2KNTn5RtzauEDq5RFEefPsttXYwAWqHng= @@ -199,6 +203,8 @@ github.com/libp2p/go-libp2p-testing v0.1.1 h1:U03z3HnGI7Ni8Xx6ONVZvUFOAzWYmolWf5 github.com/libp2p/go-libp2p-testing v0.1.1/go.mod h1:xaZWMJrPUM5GlDBxCeGUi7kI4eqnjVyavGroI2nxEM0= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 h1:PZMS9lhjK9VytzMCW3tWHAXtKXmlURSc3ZdvwEcKCzw= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07qPW4r01GnTenLW4oi3lOPbUMGJJA= +github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 h1:5EhPgQhXZNyfL22ERZTUoVp9UVVbNowWNVtELQaKCHk= +github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSoMuyojOdjx73Q47cYD7s5+gZOlns= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= github.com/libp2p/go-libp2p-yamux v0.2.1 h1:Q3XYNiKCC2vIxrvUJL+Jg1kiyeEaIDNKLjgEjo3VQdI= github.com/libp2p/go-libp2p-yamux v0.2.1/go.mod h1:1FBXiHDk1VyRM1C0aez2bCfHQ4vMZKkAQzZbkSQt5fI= @@ -276,6 +282,8 @@ github.com/multiformats/go-multiaddr v0.1.1 h1:rVAztJYMhCQ7vEFr8FvxW3mS+HF2eY/oP github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo= github.com/multiformats/go-multiaddr v0.2.0 h1:lR52sFwcTCuQb6bTfnXF6zA2XfyYvyd+5a9qECv/J90= github.com/multiformats/go-multiaddr v0.2.0/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4= +github.com/multiformats/go-multiaddr v0.2.1 h1:SgG/cw5vqyB5QQe5FPe2TqggU9WtrA9X4nZw7LlVqOI= +github.com/multiformats/go-multiaddr v0.2.1/go.mod h1:s/Apk6IyxfvMjDafnhJgJ3/46z7tZ04iMk5wP4QMGGE= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2 h1:/Bbsgsy3R6e3jf2qBahzNHzww6usYaZ0NhNH3sqdFS8= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= @@ -310,17 +318,25 @@ github.com/multiformats/go-multistream v0.1.1 h1:JlAdpIFhBhGRLxe9W6Om0w++Gd6KMWo github.com/multiformats/go-multistream v0.1.1/go.mod h1:KmHZ40hzVxiaiwlj3MEbYgK9JFk2/9UktWZAF54Du38= github.com/multiformats/go-varint v0.0.1 h1:TR/0rdQtnNxuN2IhiB639xC3tWM4IUi7DkTBVTdGW/M= github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= +github.com/multiformats/go-varint v0.0.2/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/multiformats/go-varint v0.0.5 h1:XVZwSo04Cs3j/jS0uAEPpT3JY6DzMcVLLoWOSnCxOjg= github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w= github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.12.0 h1:Iw5WCbBcaAAd0fpRb1c9r5YCylv4XDoCSigm1zLevwU= +github.com/onsi/ginkgo v1.12.0/go.mod h1:oUhWkIvk5aDxtKvDDuw8gItl8pKl42LzjC9KZE0HfGg= github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo= github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= +github.com/onsi/gomega v1.9.0 h1:R1uwffexN6Pr340GtYRIdZmAiN4J+iw6WG4wog1DUXg= +github.com/onsi/gomega v1.9.0/go.mod h1:Ho0h+IUsWyvy1OpqCwxlQ/21gkhVunqlU8fDGcoTdcA= github.com/opentracing/opentracing-go v1.0.2 h1:3jA2P6O1F9UOrWVpwrIo17pu01KWvNWg4X946/Y5Zwg= github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= +github.com/opentracing/opentracing-go v1.1.0 h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU= +github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= @@ -373,6 +389,14 @@ go.opencensus.io v0.22.1 h1:8dP3SGL7MPB94crU3bEPplMPe83FI4EouesJUeFHv50= go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA= go.opencensus.io v0.22.2 h1:75k/FF0Q2YM8QYo07VPddOLBslDt1MZOdEslOHvmzAs= go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8= +go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.uber.org/atomic v1.4.0 h1:cxzIVoETapQEqDhQu3QfnvXAV4AlzcvUCxkVUFw3+EU= +go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= +go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI= +go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= +go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM= +go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= @@ -386,6 +410,8 @@ golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 h1:IcSOAf4PyMp3U3XbIEj1/x golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d h1:1ZiEyfaQIg3Qh0EoqpwAakHVhecoE5wlSg5GjnafJGw= +golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= @@ -405,6 +431,8 @@ golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6 h1:bjcUS9ztw9kFmmIxJInhon/0Is3p+EHBKNgquIzo1OI= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -417,8 +445,10 @@ golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 h1:rOhMmluY6kLMhdnrivzec6lLgaVbMHMn2ISQXJeJ5EM= -golang.org/x/sys v0.0.0-20190922100055-0a153f010e69/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e h1:N7DeIrjYszNmSW409R3frPPwglRwMkXSBzwVbkOjLLA= +golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae h1:/WDfKMnPU+m5M4xB+6x4kaepxRw6jWvR5iDRdvjHgy8= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= @@ -431,6 +461,7 @@ golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= @@ -452,4 +483,6 @@ gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWD gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I= +gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/options.go b/options.go index d12ed256f5..4b198d18e1 100644 --- a/options.go +++ b/options.go @@ -147,13 +147,13 @@ func Peerstore(ps peerstore.Peerstore) Option { } // PrivateNetwork configures libp2p to use the given private network protector. -func PrivateNetwork(prot pnet.Protector) Option { +func PrivateNetwork(psk pnet.PSK) Option { return func(cfg *Config) error { - if cfg.Protector != nil { + if cfg.PSK != nil { return fmt.Errorf("cannot specify multiple private network options") } - cfg.Protector = prot + cfg.PSK = psk return nil } } From a3bbeef22e33e1fefefdb52bc73e9c1a03a93618 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 9 Mar 2020 11:19:43 +0000 Subject: [PATCH 1716/3965] build(deps): bump github.com/libp2p/go-libp2p-yamux from 0.2.1 to 0.2.2 Bumps [github.com/libp2p/go-libp2p-yamux](https://github.com/libp2p/go-libp2p-yamux) from 0.2.1 to 0.2.2. - [Release notes](https://github.com/libp2p/go-libp2p-yamux/releases) - [Commits](https://github.com/libp2p/go-libp2p-yamux/compare/v0.2.1...v0.2.2) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index d68d7d5cdc..a0be8ebc67 100644 --- a/go.mod +++ b/go.mod @@ -24,7 +24,7 @@ require ( github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 - github.com/libp2p/go-libp2p-yamux v0.2.1 + github.com/libp2p/go-libp2p-yamux v0.2.2 github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 diff --git a/go.sum b/go.sum index 60823a0347..f7384d65b1 100644 --- a/go.sum +++ b/go.sum @@ -160,6 +160,7 @@ github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZV github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= github.com/libp2p/go-libp2p-core v0.3.0 h1:F7PqduvrztDtFsAa/bcheQ3azmNo+Nq7m8hQY5GiUW8= github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw= +github.com/libp2p/go-libp2p-core v0.4.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= github.com/libp2p/go-libp2p-core v0.5.0 h1:FBQ1fpq2Fo/ClyjojVJ5AKXlKhvNc/B6U0O+7AN1ffE= github.com/libp2p/go-libp2p-core v0.5.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= @@ -208,6 +209,8 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSo github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= github.com/libp2p/go-libp2p-yamux v0.2.1 h1:Q3XYNiKCC2vIxrvUJL+Jg1kiyeEaIDNKLjgEjo3VQdI= github.com/libp2p/go-libp2p-yamux v0.2.1/go.mod h1:1FBXiHDk1VyRM1C0aez2bCfHQ4vMZKkAQzZbkSQt5fI= +github.com/libp2p/go-libp2p-yamux v0.2.2 h1:eGvbqWqWY9S5lrpe2gA0UCOLCdzCgYSAR3vo/xCsNQg= +github.com/libp2p/go-libp2p-yamux v0.2.2/go.mod h1:lIohaR0pT6mOt0AZ0L2dFze9hds9Req3OfS+B+dv4qw= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.5 h1:CW3AgbMO6vUvT4kf87y4N+0P8KUl2aqLYhrGyDUbLSg= github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= @@ -242,6 +245,8 @@ github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzl github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-yamux v1.3.0 h1:FsYzT16Wq2XqUGJsBbOxoz9g+dFklvNi7jN6YFPfl7U= +github.com/libp2p/go-yamux v1.3.0/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= From b5853020f816dad92542a0a03d33258ffdc0d0b5 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 9 Mar 2020 11:20:44 +0000 Subject: [PATCH 1717/3965] build(deps): bump github.com/libp2p/go-libp2p-peerstore Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.1.4 to 0.2.0. - [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases) - [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.1.4...v0.2.0) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index d68d7d5cdc..b5932ea111 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/libp2p/go-libp2p-mplex v0.2.1 github.com/libp2p/go-libp2p-nat v0.0.5 github.com/libp2p/go-libp2p-netutil v0.1.0 - github.com/libp2p/go-libp2p-peerstore v0.1.4 + github.com/libp2p/go-libp2p-peerstore v0.2.0 github.com/libp2p/go-libp2p-secio v0.2.1 github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 diff --git a/go.sum b/go.sum index 60823a0347..a5fad12926 100644 --- a/go.sum +++ b/go.sum @@ -70,6 +70,7 @@ github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmv github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= @@ -87,13 +88,18 @@ github.com/ipfs/go-cid v0.0.5/go.mod h1:plgt+Y5MnOey4vO4UlUazGqdbEXuFYitED67Fexh github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= github.com/ipfs/go-datastore v0.1.0/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= github.com/ipfs/go-datastore v0.1.1/go.mod h1:w38XXW9kVFNp57Zj5knbKWM2T+KOZCGDRVNdgPHtbHw= +github.com/ipfs/go-datastore v0.4.0/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= +github.com/ipfs/go-datastore v0.4.1/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= +github.com/ipfs/go-datastore v0.4.4/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8= github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaHzfGTzuE3s= github.com/ipfs/go-ds-badger v0.0.7/go.mod h1:qt0/fWzZDoPW6jpQeqUjR5kBfhDNB65jd9YlmAvpQBk= +github.com/ipfs/go-ds-badger v0.2.1/go.mod h1:Tx7l3aTph3FMFrRS838dcSJh+jjA7cX9DrGVwx/NOwE= github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc= github.com/ipfs/go-ds-leveldb v0.1.0/go.mod h1:hqAW8y4bwX5LWcCtku2rFNX3vjDZCy5LZCg+cSZvYb8= +github.com/ipfs/go-ds-leveldb v0.4.1/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50= github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc= @@ -160,6 +166,7 @@ github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZV github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= github.com/libp2p/go-libp2p-core v0.3.0 h1:F7PqduvrztDtFsAa/bcheQ3azmNo+Nq7m8hQY5GiUW8= github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw= +github.com/libp2p/go-libp2p-core v0.4.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= github.com/libp2p/go-libp2p-core v0.5.0 h1:FBQ1fpq2Fo/ClyjojVJ5AKXlKhvNc/B6U0O+7AN1ffE= github.com/libp2p/go-libp2p-core v0.5.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= @@ -182,6 +189,8 @@ github.com/libp2p/go-libp2p-peerstore v0.1.3 h1:wMgajt1uM2tMiqf4M+4qWKVyyFc8SfA+ github.com/libp2p/go-libp2p-peerstore v0.1.3/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= github.com/libp2p/go-libp2p-peerstore v0.1.4 h1:d23fvq5oYMJ/lkkbO4oTwBp/JP+I/1m5gZJobNXCE/k= github.com/libp2p/go-libp2p-peerstore v0.1.4/go.mod h1:+4BDbDiiKf4PzpANZDAT+knVdLxvqh7hXOujessqdzs= +github.com/libp2p/go-libp2p-peerstore v0.2.0 h1:XcgJhI8WyUOCbHyRLNEX5542YNj8hnLSJ2G1InRjDhk= +github.com/libp2p/go-libp2p-peerstore v0.2.0/go.mod h1:N2l3eVIeAitSg3Pi2ipSrJYnqhVnMNQZo9nkSCuAbnQ= github.com/libp2p/go-libp2p-pnet v0.2.0 h1:J6htxttBipJujEjz1y0a5+eYoiPcFHhSYHH6na5f0/k= github.com/libp2p/go-libp2p-pnet v0.2.0/go.mod h1:Qqvq6JH/oMZGwqs3N1Fqhv8NVhrdYcO0BW4wssv21LA= github.com/libp2p/go-libp2p-secio v0.1.0 h1:NNP5KLxuP97sE5Bu3iuwOWyT/dKEGMN5zSLMWdB7GTQ= @@ -340,6 +349,7 @@ github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFSt github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= @@ -447,6 +457,7 @@ golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSF golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e h1:N7DeIrjYszNmSW409R3frPPwglRwMkXSBzwVbkOjLLA= golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae h1:/WDfKMnPU+m5M4xB+6x4kaepxRw6jWvR5iDRdvjHgy8= golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= @@ -474,6 +485,7 @@ google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiq gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d/go.mod h1:z+K8VcOYVYcSwSjGebuDL6176A1XskgbtNl64NSg+n8= From 241a61a246fab62af3341f022d1788829074fe12 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 9 Mar 2020 14:21:49 +0000 Subject: [PATCH 1718/3965] build(deps): bump github.com/libp2p/go-libp2p-mplex from 0.2.1 to 0.2.2 Bumps [github.com/libp2p/go-libp2p-mplex](https://github.com/libp2p/go-libp2p-mplex) from 0.2.1 to 0.2.2. - [Release notes](https://github.com/libp2p/go-libp2p-mplex/releases) - [Commits](https://github.com/libp2p/go-libp2p-mplex/compare/v0.2.1...v0.2.2) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index b5932ea111..13e597177c 100644 --- a/go.mod +++ b/go.mod @@ -16,7 +16,7 @@ require ( github.com/libp2p/go-libp2p-core v0.5.0 github.com/libp2p/go-libp2p-discovery v0.2.0 github.com/libp2p/go-libp2p-loggables v0.1.0 - github.com/libp2p/go-libp2p-mplex v0.2.1 + github.com/libp2p/go-libp2p-mplex v0.2.2 github.com/libp2p/go-libp2p-nat v0.0.5 github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.2.0 diff --git a/go.sum b/go.sum index a5fad12926..203c0dc04d 100644 --- a/go.sum +++ b/go.sum @@ -83,6 +83,7 @@ github.com/ipfs/go-cid v0.0.2 h1:tuuKaZPU1M6HcejsO3AcYWW8sZ8MTvyxfc4uqB4eFE8= github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-cid v0.0.3 h1:UIAh32wymBpStoe83YCzwVQQ5Oy/H0FdxvUS6DJDzms= github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= +github.com/ipfs/go-cid v0.0.4/go.mod h1:4LLaPOQwmk5z9LBgQnpkivrx8BJjUyGwTXCd5Xfj6+M= github.com/ipfs/go-cid v0.0.5 h1:o0Ix8e/ql7Zb5UVUJEUfjsWCIY8t48++9lR8qi6oiJU= github.com/ipfs/go-cid v0.0.5/go.mod h1:plgt+Y5MnOey4vO4UlUazGqdbEXuFYitED67FexhXog= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= @@ -166,6 +167,7 @@ github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZV github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= github.com/libp2p/go-libp2p-core v0.3.0 h1:F7PqduvrztDtFsAa/bcheQ3azmNo+Nq7m8hQY5GiUW8= github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw= +github.com/libp2p/go-libp2p-core v0.3.1/go.mod h1:thvWy0hvaSBhnVBaW37BvzgVV68OUhgJJLAa6almrII= github.com/libp2p/go-libp2p-core v0.4.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= github.com/libp2p/go-libp2p-core v0.5.0 h1:FBQ1fpq2Fo/ClyjojVJ5AKXlKhvNc/B6U0O+7AN1ffE= github.com/libp2p/go-libp2p-core v0.5.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= @@ -179,6 +181,8 @@ github.com/libp2p/go-libp2p-loggables v0.1.0/go.mod h1:EyumB2Y6PrYjr55Q3/tiJ/o3x github.com/libp2p/go-libp2p-mplex v0.2.0/go.mod h1:Ejl9IyjvXJ0T9iqUTE1jpYATQ9NM3g+OtR+EMMODbKo= github.com/libp2p/go-libp2p-mplex v0.2.1 h1:E1xaJBQnbSiTHGI1gaBKmKhu1TUKkErKJnE8iGvirYI= github.com/libp2p/go-libp2p-mplex v0.2.1/go.mod h1:SC99Rxs8Vuzrf/6WhmH41kNn13TiYdAWNYHrwImKLnE= +github.com/libp2p/go-libp2p-mplex v0.2.2 h1:+Ld7YDAfVERQ0E+qqjE7o6fHwKuM0SqTzYiwN1lVVSA= +github.com/libp2p/go-libp2p-mplex v0.2.2/go.mod h1:74S9eum0tVQdAfFiKxAyKzNdSuLqw5oadDq7+L/FELo= github.com/libp2p/go-libp2p-nat v0.0.5 h1:/mH8pXFVKleflDL1YwqMg27W9GD8kjEx7NY0P6eGc98= github.com/libp2p/go-libp2p-nat v0.0.5/go.mod h1:1qubaE5bTZMJE+E/uu2URroMbzdubFz1ChgiN79yKPE= github.com/libp2p/go-libp2p-netutil v0.1.0 h1:zscYDNVEcGxyUpMd0JReUZTrpMfia8PmLKcKF72EAMQ= @@ -223,6 +227,8 @@ github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDR github.com/libp2p/go-mplex v0.0.3/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= github.com/libp2p/go-mplex v0.1.0 h1:/nBTy5+1yRyY82YaO6HXQRnO5IAGsXTjEJaR3LdTPc0= github.com/libp2p/go-mplex v0.1.0/go.mod h1:SXgmdki2kwCUlCCbfGLEgHjC4pFqhTp0ZoV6aiKgxDU= +github.com/libp2p/go-mplex v0.1.1 h1:huPH/GGRJzmsHR9IZJJsrSwIM5YE2gL4ssgl1YWb/ps= +github.com/libp2p/go-mplex v0.1.1/go.mod h1:Xgz2RDCi3co0LeZfgjm4OgUF15+sVR8SRcu3SFXI1lk= github.com/libp2p/go-msgio v0.0.2 h1:ivPvEKHxmVkTClHzg6RXTYHqaJQ0V9cDbq+6lKb3UV0= github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-msgio v0.0.4 h1:agEFehY3zWJFUHK6SEMR7UYmk2z6kC3oeCM7ybLhguA= From 8248da96b0a406501d72a8442a545eff6997e280 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 9 Mar 2020 07:40:38 -0700 Subject: [PATCH 1719/3965] doc(options): fix autorelay documentation Routing is only required for autorelay when no static relays are configured. --- options.go | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/options.go b/options.go index 4b198d18e1..d63afbba7f 100644 --- a/options.go +++ b/options.go @@ -228,9 +228,11 @@ func DisableRelay() Option { } } -// EnableAutoRelay configures libp2p to enable the AutoRelay subsystem. It is an -// error to enable AutoRelay without enabling relay (enabled by default) and -// routing (not enabled by default). +// EnableAutoRelay configures libp2p to enable the AutoRelay subsystem. +// +// Dependencies: +// * Relay (enabled by default) +// * Routing (to find relays), or StaticRelays/DefaultStaticRelays. // // This subsystem performs two functions: // From 43e6a96bdcd1b09ed6adf488a368c0f76f20367d Mon Sep 17 00:00:00 2001 From: Will Scott Date: Mon, 9 Mar 2020 16:25:55 -0700 Subject: [PATCH 1720/3965] additional tests --- p2p/host/autonat/svc.go | 21 ++++++-- p2p/host/autonat/svc_test.go | 93 ++++++++++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+), 5 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 2475f03cb9..3f857b3058 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -27,12 +27,23 @@ import ( const P_CIRCUIT = 290 var ( - AutoNATServiceDialTimeout = 15 * time.Second + // AutoNATServiceDialTimeout defines how long to wait for connection + // attempts before failing. + AutoNATServiceDialTimeout = 15 * time.Second + // AutoNATServiceResetInterval defines how often to reset throttling. AutoNATServiceResetInterval = 1 * time.Minute - AutoNATServiceResetJitter = 15 * time.Second - - AutoNATServiceThrottle = 3 - AutoNATGlobalThrottle = 30 + // AutoNATServiceResetJitter defines the amplitude of randomness in throttle + // reset timing. + AutoNATServiceResetJitter = 15 * time.Second + + // AutoNATServiceThrottle defines how many times each ResetInterval a peer + // can ask for its autonat address. + AutoNATServiceThrottle = 3 + // AutoNATGlobalThrottle defines how many total autonat requests this + // service will answer each ResetInterval. + AutoNATGlobalThrottle = 30 + // AutoNATMaxPeerAddresses defines maximum number of addreses the autonat + // service will consider when attempting to connect to the peer. AutoNATMaxPeerAddresses = 16 ) diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index a36340913c..9a19a7350c 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -7,7 +7,9 @@ import ( "time" "github.com/libp2p/go-libp2p" + "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" autonat "github.com/libp2p/go-libp2p-autonat" @@ -137,6 +139,55 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { AutoNATServiceResetJitter = save5 } +func TestAutoNATServiceGlobalLimiter(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save1 := AutoNATServiceDialTimeout + AutoNATServiceDialTimeout = 1 * time.Second + save2 := AutoNATServiceResetInterval + AutoNATServiceResetInterval = 10 * time.Second + save3 := AutoNATServiceThrottle + AutoNATServiceThrottle = 1 + save4 := manet.Private4 + manet.Private4 = []*net.IPNet{} + save5 := AutoNATServiceResetJitter + AutoNATServiceResetJitter = 0 * time.Second + save6 := AutoNATGlobalThrottle + AutoNATGlobalThrottle = 5 + + hs, as := makeAutoNATService(ctx, t) + as.globalReqMax = 5 + + for i := 0; i < 5; i++ { + hc, ac := makeAutoNATClient(ctx, t) + connect(t, hs, hc) + + _, err := ac.DialBack(ctx, hs.ID()) + if err != nil { + t.Fatal(err) + } + } + + hc, ac := makeAutoNATClient(ctx, t) + connect(t, hs, hc) + _, err := ac.DialBack(ctx, hs.ID()) + if err == nil { + t.Fatal("Dial back succeeded unexpectedly!") + } + + if !autonat.IsDialRefused(err) { + t.Fatal(err) + } + + AutoNATServiceDialTimeout = save1 + AutoNATServiceResetInterval = save2 + AutoNATServiceThrottle = save3 + manet.Private4 = save4 + AutoNATServiceResetJitter = save5 + AutoNATGlobalThrottle = save6 +} + func TestAutoNATServiceRateLimitJitter(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) @@ -163,6 +214,48 @@ func TestAutoNATServiceRateLimitJitter(t *testing.T) { AutoNATServiceResetJitter = save2 } +func TestAutoNATServiceStartup(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + save := manet.Private4 + manet.Private4 = []*net.IPNet{} + + h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) + if err != nil { + t.Fatal(err) + } + + _, err = NewAutoNATService(ctx, h, false) + if err != nil { + t.Fatal(err) + } + + eb, _ := h.EventBus().Emitter(new(event.EvtLocalReachabilityChanged)) + + hc, ac := makeAutoNATClient(ctx, t) + connect(t, h, hc) + + _, err = ac.DialBack(ctx, h.ID()) + if err == nil { + t.Fatal("autonat should not be started / advertising.") + } + + eb.Emit(event.EvtLocalReachabilityChanged{Reachability: network.ReachabilityPublic}) + _, err = ac.DialBack(ctx, h.ID()) + if err != nil { + t.Fatalf("autonat should be active, was %v", err) + } + + eb.Emit(event.EvtLocalReachabilityChanged{Reachability: network.ReachabilityPrivate}) + _, err = ac.DialBack(ctx, h.ID()) + if err == nil { + t.Fatal("autonat should not be started / advertising.") + } + + manet.Private4 = save +} + func TestAddrToIP(t *testing.T) { addr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.IPv4(127, 0, 0, 1)) { From 5f75aa2068f276cc5140879cda7fed1f0b131afe Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 9 Mar 2020 17:04:37 -0700 Subject: [PATCH 1721/3965] feat(mock): reliable notifications * Export StreamComplement/ConnComplement convenience functions. * Make the TestNotifications test pass reliably, even when we have a bunch of streams (identify, etc.). * Make the mock net order disconnect events after connect events. * Make closing one side of a connection actually close both sides. * Make it possible to extract a mock stream's complement. * Fire remote events at the same time as the local events. --- p2p/net/mock/complement.go | 17 +++ p2p/net/mock/mock_conn.go | 36 ++++--- p2p/net/mock/mock_link.go | 18 +--- p2p/net/mock/mock_notif_test.go | 180 ++++++++++++++++---------------- p2p/net/mock/mock_peernet.go | 19 ++-- p2p/net/mock/mock_stream.go | 24 +++-- 6 files changed, 161 insertions(+), 133 deletions(-) create mode 100644 p2p/net/mock/complement.go diff --git a/p2p/net/mock/complement.go b/p2p/net/mock/complement.go new file mode 100644 index 0000000000..4a8b251da0 --- /dev/null +++ b/p2p/net/mock/complement.go @@ -0,0 +1,17 @@ +package mocknet + +import ( + "github.com/libp2p/go-libp2p-core/network" +) + +// StreamComplement returns the other end of the given stream. This function +// panics when passed a non-mocknet stream. +func StreamComplement(s network.Stream) network.Stream { + return s.(*stream).rstream +} + +// ConnComplement returns the other end of the given connection. This function +// panics when passed a non-mocknet connection. +func ConnComplement(c network.Conn) network.Conn { + return c.(*conn).rconn +} diff --git a/p2p/net/mock/mock_conn.go b/p2p/net/mock/mock_conn.go index 17f56528ca..882522b2ea 100644 --- a/p2p/net/mock/mock_conn.go +++ b/p2p/net/mock/mock_conn.go @@ -15,6 +15,8 @@ import ( // live connection between two peers. // it goes over a particular link. type conn struct { + notifLk sync.Mutex + local peer.ID remote peer.ID @@ -34,8 +36,8 @@ type conn struct { sync.RWMutex } -func newConn(ln, rn *peernet, l *link, dir network.Direction) *conn { - c := &conn{net: ln, link: l} +func newConn(p process.Process, ln, rn *peernet, l *link, dir network.Direction) *conn { + c := &conn{net: ln, link: l, proc: p} c.local = ln.peer c.remote = rn.peer c.stat = network.Stat{Direction: dir} @@ -46,7 +48,7 @@ func newConn(ln, rn *peernet, l *link, dir network.Direction) *conn { c.localPrivKey = ln.ps.PrivKey(ln.peer) c.remotePubKey = rn.ps.PubKey(rn.peer) - c.proc = process.WithTeardown(c.teardown) + c.proc.AddChild(process.WithTeardown(c.teardown)) return c } @@ -59,6 +61,9 @@ func (c *conn) teardown() error { s.Reset() } c.net.removeConn(c) + + c.notifLk.Lock() + defer c.notifLk.Unlock() c.net.notifyAll(func(n network.Notifiee) { n.Disconnected(c.net, c) }) @@ -69,18 +74,29 @@ func (c *conn) addStream(s *stream) { c.Lock() s.conn = c c.streams.PushBack(s) + s.notifLk.Lock() + defer s.notifLk.Unlock() c.Unlock() + c.net.notifyAll(func(n network.Notifiee) { + n.OpenedStream(c.net, s) + }) } func (c *conn) removeStream(s *stream) { c.Lock() - defer c.Unlock() for e := c.streams.Front(); e != nil; e = e.Next() { if s == e.Value { c.streams.Remove(e) - return + break } } + c.Unlock() + + s.notifLk.Lock() + defer s.notifLk.Unlock() + s.conn.net.notifyAll(func(n network.Notifiee) { + n.ClosedStream(s.conn.net, s) + }) } func (c *conn) allStreams() []network.Stream { @@ -98,18 +114,12 @@ func (c *conn) allStreams() []network.Stream { func (c *conn) remoteOpenedStream(s *stream) { c.addStream(s) c.net.handleNewStream(s) - c.net.notifyAll(func(n network.Notifiee) { - n.OpenedStream(c.net, s) - }) } func (c *conn) openStream() *stream { - sl, sr := c.link.newStreamPair() + sl, sr := newStreamPair() + go c.rconn.remoteOpenedStream(sr) c.addStream(sl) - c.net.notifyAll(func(n network.Notifiee) { - n.OpenedStream(c.net, sl) - }) - c.rconn.remoteOpenedStream(sr) return sl } diff --git a/p2p/net/mock/mock_link.go b/p2p/net/mock/mock_link.go index bd9481fb2d..294b2bc627 100644 --- a/p2p/net/mock/mock_link.go +++ b/p2p/net/mock/mock_link.go @@ -1,13 +1,13 @@ package mocknet import ( - // "fmt" - "io" "sync" "time" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" + + process "github.com/jbenet/goprocess" ) // link implements mocknet.Link @@ -33,8 +33,9 @@ func (l *link) newConnPair(dialer *peernet) (*conn, *conn) { l.RLock() defer l.RUnlock() - c1 := newConn(l.nets[0], l.nets[1], l, network.DirOutbound) - c2 := newConn(l.nets[1], l.nets[0], l, network.DirInbound) + parent := process.WithTeardown(func() error { return nil }) + c1 := newConn(parent, l.nets[0], l.nets[1], l, network.DirOutbound) + c2 := newConn(parent, l.nets[1], l.nets[0], l, network.DirInbound) c1.rconn = c2 c2.rconn = c1 @@ -44,15 +45,6 @@ func (l *link) newConnPair(dialer *peernet) (*conn, *conn) { return c2, c1 } -func (l *link) newStreamPair() (*stream, *stream) { - ra, wb := io.Pipe() - rb, wa := io.Pipe() - - sa := NewStream(wa, ra, network.DirOutbound) - sb := NewStream(wb, rb, network.DirInbound) - return sa, sb -} - func (l *link) Networks() []network.Network { l.RLock() defer l.RUnlock() diff --git a/p2p/net/mock/mock_notif_test.go b/p2p/net/mock/mock_notif_test.go index 4caecbedfa..f443d4f6f0 100644 --- a/p2p/net/mock/mock_notif_test.go +++ b/p2p/net/mock/mock_notif_test.go @@ -2,6 +2,7 @@ package mocknet import ( "context" + "sync" "testing" "time" @@ -13,8 +14,10 @@ import ( func TestNotifications(t *testing.T) { const swarmSize = 5 + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() - mn, err := FullMeshLinked(context.Background(), swarmSize) + mn, err := FullMeshLinked(ctx, swarmSize) if err != nil { t.Fatal(err) } @@ -23,11 +26,13 @@ func TestNotifications(t *testing.T) { // signup notifs nets := mn.Nets() - notifiees := make([]*netNotifiee, len(nets)) - for i, pn := range nets { - n := newNetNotifiee(swarmSize) + notifiees := make(map[peer.ID]*netNotifiee, len(nets)) + for _, pn := range nets { + defer pn.Close() + + n := newNetNotifiee(t, swarmSize) pn.Notify(n) - notifiees[i] = n + notifiees[pn.LocalPeer()] = n } // connect all but self @@ -36,16 +41,16 @@ func TestNotifications(t *testing.T) { } // test everyone got the correct connection opened calls - for i, s := range nets { - n := notifiees[i] + for _, s1 := range nets { + n := notifiees[s1.LocalPeer()] notifs := make(map[peer.ID][]network.Conn) - for j, s2 := range nets { - if i == j { + for _, s2 := range nets { + if s2 == s1 { continue } // this feels a little sketchy, but its probably okay - for len(s.ConnsToPeer(s2.LocalPeer())) != len(notifs[s2.LocalPeer()]) { + for len(s1.ConnsToPeer(s2.LocalPeer())) != len(notifs[s2.LocalPeer()]) { select { case c := <-n.connected: nfp := notifs[c.RemotePeer()] @@ -57,7 +62,7 @@ func TestNotifications(t *testing.T) { } for p, cons := range notifs { - expect := s.ConnsToPeer(p) + expect := s1.ConnsToPeer(p) if len(expect) != len(cons) { t.Fatal("got different number of connections") } @@ -78,100 +83,49 @@ func TestNotifications(t *testing.T) { } } - complement := func(c network.Conn) (network.Network, *netNotifiee, *conn) { - for i, s := range nets { - for _, c2 := range s.Conns() { - if c2.(*conn).rconn == c { - return s, notifiees[i], c2.(*conn) - } - } - } - t.Fatal("complementary conn not found", c) - return nil, nil, nil - } - - testOCStream := func(n *netNotifiee, s network.Stream) { - var s2 network.Stream - select { - case s2 = <-n.openedStream: - t.Log("got notif for opened stream") - case <-time.After(timeout): - t.Fatal("timeout") - } - if s != nil && s != s2 { - t.Fatalf("got incorrect stream %p %p", s, s2) - } - - select { - case s2 = <-n.closedStream: - t.Log("got notif for closed stream") - case <-time.After(timeout): - t.Fatal("timeout") - } - if s != nil && s != s2 { - t.Fatalf("got incorrect stream %p %p", s, s2) - } - } - for _, s := range nets { s.SetStreamHandler(func(s network.Stream) { helpers.FullClose(s) }) } - // there's one stream per conn that we need to drain.... - // unsure where these are coming from - for i := range nets { - n := notifiees[i] - for j := 0; j < len(nets)-1; j++ { - testOCStream(n, nil) - } - } - - streams := make(chan network.Stream) for _, s := range nets { s.SetStreamHandler(func(s network.Stream) { - streams <- s helpers.FullClose(s) }) } - // open a streams in each conn - for i, s := range nets { + // Make sure we've received at last one stream per conn. + for _, s := range nets { conns := s.Conns() for _, c := range conns { - _, n2, c2 := complement(c) st1, err := c.NewStream() if err != nil { t.Error(err) - } else { - t.Logf("%s %s <--%p--> %s %s", c.LocalPeer(), c.LocalMultiaddr(), st1, c.RemotePeer(), c.RemoteMultiaddr()) - // st1.Write([]byte("hello")) - go helpers.FullClose(st1) - st2 := <-streams - t.Logf("%s %s <--%p--> %s %s", c2.LocalPeer(), c2.LocalMultiaddr(), st2, c2.RemotePeer(), c2.RemoteMultiaddr()) - testOCStream(notifiees[i], st1) - testOCStream(n2, st2) + continue } + t.Logf("%s %s <--%p--> %s %s", c.LocalPeer(), c.LocalMultiaddr(), st1, c.RemotePeer(), c.RemoteMultiaddr()) + helpers.FullClose(st1) } } // close conns - for i, s := range nets { - n := notifiees[i] - for _, c := range s.Conns() { - _, n2, c2 := complement(c) - c.(*conn).Close() - c2.Close() + for _, s1 := range nets { + n1 := notifiees[s1.LocalPeer()] + for _, c1 := range s1.Conns() { + c2 := ConnComplement(c1) + + n2 := notifiees[c2.LocalPeer()] + c1.Close() var c3, c4 network.Conn select { - case c3 = <-n.disconnected: + case c3 = <-n1.disconnected: case <-time.After(timeout): t.Fatal("timeout") } - if c != c3 { - t.Fatal("got incorrect conn", c, c3) + if c1 != c3 { + t.Fatal("got incorrect conn", c1, c3) } select { @@ -180,30 +134,53 @@ func TestNotifications(t *testing.T) { t.Fatal("timeout") } if c2 != c4 { - t.Fatal("got incorrect conn", c, c2) + t.Fatal("got incorrect conn", c1, c2) } } } + + for _, n1 := range notifiees { + n1.streamState.Lock() + for str1, ch1 := range n1.streamState.m { + <-ch1 + str2 := StreamComplement(str1) + n2 := notifiees[str1.Conn().RemotePeer()] + + n2.streamState.Lock() + ch2 := n2.streamState.m[str2] + n2.streamState.Unlock() + + <-ch2 + } + + n1.streamState.Unlock() + } } type netNotifiee struct { + t *testing.T + listen chan ma.Multiaddr listenClose chan ma.Multiaddr connected chan network.Conn disconnected chan network.Conn - openedStream chan network.Stream - closedStream chan network.Stream + + streamState struct { + sync.Mutex + m map[network.Stream]chan struct{} + } } -func newNetNotifiee(buffer int) *netNotifiee { - return &netNotifiee{ - listen: make(chan ma.Multiaddr, buffer), - listenClose: make(chan ma.Multiaddr, buffer), - connected: make(chan network.Conn, buffer), - disconnected: make(chan network.Conn, buffer), - openedStream: make(chan network.Stream, buffer), - closedStream: make(chan network.Stream, buffer), +func newNetNotifiee(t *testing.T, buffer int) *netNotifiee { + nn := &netNotifiee{ + t: t, + listen: make(chan ma.Multiaddr, 1), + listenClose: make(chan ma.Multiaddr, 1), + connected: make(chan network.Conn, buffer*2), + disconnected: make(chan network.Conn, buffer*2), } + nn.streamState.m = make(map[network.Stream]chan struct{}) + return nn } func (nn *netNotifiee) Listen(n network.Network, a ma.Multiaddr) { @@ -218,9 +195,28 @@ func (nn *netNotifiee) Connected(n network.Network, v network.Conn) { func (nn *netNotifiee) Disconnected(n network.Network, v network.Conn) { nn.disconnected <- v } -func (nn *netNotifiee) OpenedStream(n network.Network, v network.Stream) { - nn.openedStream <- v +func (nn *netNotifiee) OpenedStream(n network.Network, s network.Stream) { + nn.streamState.Lock() + defer nn.streamState.Unlock() + _, ok := nn.streamState.m[s] + if ok { + nn.t.Error("duplicate stream open") + return + } + nn.streamState.m[s] = make(chan struct{}) } -func (nn *netNotifiee) ClosedStream(n network.Network, v network.Stream) { - nn.closedStream <- v +func (nn *netNotifiee) ClosedStream(n network.Network, s network.Stream) { + nn.streamState.Lock() + defer nn.streamState.Unlock() + ch, ok := nn.streamState.m[s] + if !ok { + nn.t.Error("saw close event but no open event") + return + } + select { + case <-ch: + nn.t.Error("duplicate close event") + default: + close(ch) + } } diff --git a/p2p/net/mock/mock_peernet.go b/p2p/net/mock/mock_peernet.go index bcf0644656..ff549873e0 100644 --- a/p2p/net/mock/mock_peernet.go +++ b/p2p/net/mock/mock_peernet.go @@ -160,11 +160,8 @@ func (pn *peernet) connect(p peer.ID) (*conn, error) { func (pn *peernet) openConn(r peer.ID, l *link) *conn { lc, rc := l.newConnPair(pn) log.Debugf("%s opening connection to %s", pn.LocalPeer(), lc.RemotePeer()) + go rc.net.remoteOpenedConn(rc) pn.addConn(lc) - pn.notifyAll(func(n network.Notifiee) { - n.Connected(pn, lc) - }) - rc.net.remoteOpenedConn(rc) return lc } @@ -172,16 +169,12 @@ func (pn *peernet) remoteOpenedConn(c *conn) { log.Debugf("%s accepting connection from %s", pn.LocalPeer(), c.RemotePeer()) pn.addConn(c) pn.handleNewConn(c) - pn.notifyAll(func(n network.Notifiee) { - n.Connected(pn, c) - }) } // addConn constructs and adds a connection // to given remote peer over given link func (pn *peernet) addConn(c *conn) { pn.Lock() - defer pn.Unlock() _, found := pn.connsByPeer[c.RemotePeer()] if !found { @@ -194,6 +187,14 @@ func (pn *peernet) addConn(c *conn) { pn.connsByLink[c.link] = map[*conn]struct{}{} } pn.connsByLink[c.link][c] = struct{}{} + + c.notifLk.Lock() + defer c.notifLk.Unlock() + pn.Unlock() + + pn.notifyAll(func(n network.Notifiee) { + n.Connected(pn, c) + }) } // removeConn removes a given conn @@ -380,6 +381,6 @@ func (pn *peernet) notifyAll(notification func(f network.Notifiee)) { notification(n) }(n) } - wg.Wait() pn.notifmu.Unlock() + wg.Wait() } diff --git a/p2p/net/mock/mock_stream.go b/p2p/net/mock/mock_stream.go index d71e53c115..ecb32ddbbf 100644 --- a/p2p/net/mock/mock_stream.go +++ b/p2p/net/mock/mock_stream.go @@ -5,6 +5,7 @@ import ( "errors" "io" "net" + "sync" "sync/atomic" "time" @@ -15,9 +16,13 @@ import ( // stream implements network.Stream type stream struct { + notifLk sync.Mutex + + rstream *stream + conn *conn + write *io.PipeWriter read *io.PipeReader - conn *conn toDeliver chan *transportObject reset chan struct{} @@ -37,7 +42,18 @@ type transportObject struct { arrivalTime time.Time } -func NewStream(w *io.PipeWriter, r *io.PipeReader, dir network.Direction) *stream { +func newStreamPair() (*stream, *stream) { + ra, wb := io.Pipe() + rb, wa := io.Pipe() + + sa := newStream(wa, ra, network.DirOutbound) + sb := newStream(wb, rb, network.DirInbound) + sa.rstream = sb + sb.rstream = sa + return sa, sb +} + +func newStream(w *io.PipeWriter, r *io.PipeReader, dir network.Direction) *stream { s := &stream{ read: r, write: w, @@ -117,10 +133,6 @@ func (s *stream) teardown() { // Mark as closed. close(s.closed) - - s.conn.net.notifyAll(func(n network.Notifiee) { - n.ClosedStream(s.conn.net, s) - }) } func (s *stream) Conn() network.Conn { From af6b54ba13efa746f6f5d450811da3d3075e4d69 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 9 Mar 2020 19:57:19 -0700 Subject: [PATCH 1722/3965] deps: move tools to a sub-module (#130) That way, the root module doesn't depend on gocompat and all its dependencies. --- core/compat-check | 18 ------------- core/tools/compat-check | 21 +++++++++++++++ core/tools/go.mod | 5 ++++ core/tools/go.sum | 55 +++++++++++++++++++++++++++++++++++++++ core/{ => tools}/tools.go | 2 +- 5 files changed, 82 insertions(+), 19 deletions(-) delete mode 100755 core/compat-check create mode 100755 core/tools/compat-check create mode 100644 core/tools/go.mod create mode 100644 core/tools/go.sum rename core/{ => tools}/tools.go (81%) diff --git a/core/compat-check b/core/compat-check deleted file mode 100755 index 51899f5f37..0000000000 --- a/core/compat-check +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/env bash - -check-compat() ( - set -e - go run github.com/smola/gocompat/cmd/gocompat compare --git-refs="remotes/origin/master..$(git rev-parse HEAD)" --go1compat ./... -) - -export GO111MODULE=on -if [[ "$TRAVIS" == "true" ]]; then - git fetch origin master:remotes/origin/master > /dev/null 2>&1 -fi - -if ! check-compat; then - echo ">> API Compatibility broken!!" - exit 1 -fi - - diff --git a/core/tools/compat-check b/core/tools/compat-check new file mode 100755 index 0000000000..58bfec8eca --- /dev/null +++ b/core/tools/compat-check @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +check-compat() { + set -e + ( + cd tools && + mkdir -p bin && + go build -o bin/gocompat github.com/smola/gocompat/cmd/gocompat + ) + ./tools/bin/gocompat compare --git-refs="remotes/origin/master..$(git rev-parse HEAD)" --go1compat ./... +} + +export GO111MODULE=on +if [[ "$TRAVIS" == "true" ]]; then + git fetch origin master:remotes/origin/master > /dev/null 2>&1 +fi + +if ! check-compat; then + echo ">> API Compatibility broken!!" + exit 1 +fi diff --git a/core/tools/go.mod b/core/tools/go.mod new file mode 100644 index 0000000000..43e87675ba --- /dev/null +++ b/core/tools/go.mod @@ -0,0 +1,5 @@ +module github.com/libp2p/go-libp2p-core/tools + +go 1.14 + +require github.com/smola/gocompat v0.2.0 diff --git a/core/tools/go.sum b/core/tools/go.sum new file mode 100644 index 0000000000..672231fdce --- /dev/null +++ b/core/tools/go.sum @@ -0,0 +1,55 @@ +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= +github.com/jessevdk/go-flags v1.4.0 h1:4IU2WS7AumrZ/40jfhf4QVDMsQwqA7VEHozFRrGARJA= +github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= +github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d/go.mod h1:P2viExyCEfeWGU259JnaQ34Inuec4R38JCyBx2edgD0= +github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic= +github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mattn/go-colorable v0.0.9 h1:UVL0vNpWh04HeJXV0KLcaT7r06gOH2l4OW6ddYRUIY4= +github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= +github.com/mattn/go-isatty v0.0.4 h1:bnP0vzxcAdeI1zdubAl5PjU6zsERjGZb7raWodagDYs= +github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= +github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b h1:j7+1HpAFS1zy5+Q4qx1fWh90gTKwiN4QCGoY9TWyyO4= +github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE= +github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/sirupsen/logrus v1.2.0 h1:juTguoYk5qI21pwyTXY3B3Y5cOTH3ZUyZCg1v/mihuo= +github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= +github.com/smola/gocompat v0.2.0 h1:6b1oIMlUXIpz//VKEDzPVBK8KG7beVwmHIUEBIs/Pns= +github.com/smola/gocompat v0.2.0/go.mod h1:1B0MlxbmoZNo3h8guHp8HztB3BSYR5itql9qtVc0ypY= +github.com/src-d/envconfig v1.0.0 h1:/AJi6DtjFhZKNx3OB2qMsq7y4yT5//AeSZIe7rk+PX8= +github.com/src-d/envconfig v1.0.0/go.mod h1:Q9YQZ7BKITldTBnoxsE5gOeB5y66RyPXeue/R4aaNBc= +github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= +github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/x-cray/logrus-prefixed-formatter v0.5.2 h1:00txxvfBM9muc0jiLIEAkAcIMJzfthRT6usrui8uGmg= +github.com/x-cray/logrus-prefixed-formatter v0.5.2/go.mod h1:2duySbKsL6M18s5GU7VPsoEPHyzalCE06qoARUCeBBE= +golang.org/x/crypto v0.0.0-20180904163835-0709b304e793 h1:u+LnwYTOOW7Ukr/fppxEb1Nwz0AtPflrblfvUudpo+I= +golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e h1:o3PsSEY8E4eXWkXrIP9YJALUkVZqzHJT5DOasTyn8Vs= +golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce h1:Gi26mRaGtAreZ9IadlBiwSJT1EDsfk4BSHBD9oxXEFY= +golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d h1:mXa4inJUuWOoA4uEROxtJ3VMELMlVkIxIfcR0HBekAM= +gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d/go.mod h1:z+K8VcOYVYcSwSjGebuDL6176A1XskgbtNl64NSg+n8= +gopkg.in/src-d/go-log.v1 v1.0.1 h1:heWvX7J6qbGWbeFS/aRmiy1eYaT+QMV6wNvHDyMjQV4= +gopkg.in/src-d/go-log.v1 v1.0.1/go.mod h1:GN34hKP0g305ysm2/hctJ0Y8nWP3zxXXJ8GFabTyABE= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= +gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= diff --git a/core/tools.go b/core/tools/tools.go similarity index 81% rename from core/tools.go rename to core/tools/tools.go index c9a7140e05..f9922713ef 100644 --- a/core/tools.go +++ b/core/tools/tools.go @@ -1,6 +1,6 @@ // +build tools -package core +package tools import ( _ "github.com/smola/gocompat" From 58172272ae26801a425a3045f249b596714d2300 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 10 Mar 2020 11:47:56 -0700 Subject: [PATCH 1723/3965] bump to new go-multiaddr-net --- p2p/host/autonat/svc.go | 25 ++----------------------- p2p/host/autonat/svc_test.go | 23 ----------------------- 2 files changed, 2 insertions(+), 46 deletions(-) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 3f857b3058..c492b1bd8d 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -2,7 +2,6 @@ package autonat import ( "context" - "fmt" "math/rand" "net" "sync" @@ -125,26 +124,6 @@ func (as *AutoNATService) handleStream(s network.Stream) { } } -// Optimistically extract the net.IP host from a multiaddress. -// TODO: use upstream manet.ToIP -func addrToIP(addr ma.Multiaddr) (net.IP, error) { - n, err := manet.ToNetAddr(addr) - if err != nil { - return nil, err - } - - switch netAddr := n.(type) { - case *net.UDPAddr: - return netAddr.IP, nil - case *net.TCPAddr: - return netAddr.IP, nil - case *net.IPAddr: - return netAddr.IP, nil - default: - return nil, fmt.Errorf("non IP Multiaddr: %T", netAddr) - } -} - func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { if mpi == nil { return newDialResponseError(pb.Message_E_BAD_REQUEST, "missing peer info") @@ -170,7 +149,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me if !as.skipDial(obsaddr) { addrs = append(addrs, obsaddr) seen[obsaddr.String()] = struct{}{} - obsHost, _ = addrToIP(obsaddr) + obsHost, _ = manet.ToIP(obsaddr) } for _, maddr := range mpi.GetAddrs() { @@ -184,7 +163,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me continue } - if ip, err := addrToIP(addr); err != nil || !obsHost.Equal(ip) { + if ip, err := manet.ToIP(addr); err != nil || !obsHost.Equal(ip) { continue } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 9a19a7350c..5e845c737f 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -13,7 +13,6 @@ import ( "github.com/libp2p/go-libp2p-core/peer" autonat "github.com/libp2p/go-libp2p-autonat" - ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) @@ -255,25 +254,3 @@ func TestAutoNATServiceStartup(t *testing.T) { manet.Private4 = save } - -func TestAddrToIP(t *testing.T) { - addr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/tcp/0") - if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.IPv4(127, 0, 0, 1)) { - t.Fatal("addrToIP of ipv4 localhost incorrect!") - } - - addr, _ = ma.NewMultiaddr("/ip4/192.168.0.1/tcp/6") - if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.IPv4(192, 168, 0, 1)) { - t.Fatal("addrToIP of ipv4 incorrect!") - } - - addr, _ = ma.NewMultiaddr("/ip6zone/eth0/ip6/fe80::1") - if ip, err := addrToIP(addr); err != nil || !ip.Equal(net.ParseIP("fe80::1")) { - t.Fatal("addrToIP of ip6zone incorrect!") - } - - addr, _ = ma.NewMultiaddr("/unix/a/b/c/d") - if _, err := addrToIP(addr); err == nil { - t.Fatal("invalid addrToIP populates") - } -} From ca57cf904e462e0977b41c7cb382c75d673778d9 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Wed, 11 Mar 2020 11:18:38 +0000 Subject: [PATCH 1724/3965] build(deps): bump github.com/multiformats/go-multiaddr-net Bumps [github.com/multiformats/go-multiaddr-net](https://github.com/multiformats/go-multiaddr-net) from 0.1.2 to 0.1.3. - [Release notes](https://github.com/multiformats/go-multiaddr-net/releases) - [Commits](https://github.com/multiformats/go-multiaddr-net/compare/v0.1.2...v0.1.3) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index eb80cd3aa8..de901d135b 100644 --- a/go.mod +++ b/go.mod @@ -32,7 +32,7 @@ require ( github.com/miekg/dns v1.1.12 // indirect github.com/multiformats/go-multiaddr v0.2.1 github.com/multiformats/go-multiaddr-dns v0.2.0 - github.com/multiformats/go-multiaddr-net v0.1.2 + github.com/multiformats/go-multiaddr-net v0.1.3 github.com/multiformats/go-multistream v0.1.1 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d // indirect diff --git a/go.sum b/go.sum index 4e37d730ed..51dbd658e1 100644 --- a/go.sum +++ b/go.sum @@ -320,6 +320,8 @@ github.com/multiformats/go-multiaddr-net v0.1.1 h1:jFFKUuXTXv+3ARyHZi3XUqQO+YWMK github.com/multiformats/go-multiaddr-net v0.1.1/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= github.com/multiformats/go-multiaddr-net v0.1.2 h1:P7zcBH9FRETdPkDrylcXVjQLQ2t1JQtNItZULWNWgeg= github.com/multiformats/go-multiaddr-net v0.1.2/go.mod h1:QsWt3XK/3hwvNxZJp92iMQKME1qHfpYmyIjFVsSOY6Y= +github.com/multiformats/go-multiaddr-net v0.1.3 h1:q/IYAvoPKuRzGeERn3uacWgm0LIWkLZBAvO5DxSzq3g= +github.com/multiformats/go-multiaddr-net v0.1.3/go.mod h1:ilNnaM9HbmVFqsb/qcNysjCu4PVONlrBZpHIrw/qQuA= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= From 96fc42d3d16da9ffc2129336e51adc1fa4c6cead Mon Sep 17 00:00:00 2001 From: Will Date: Wed, 11 Mar 2020 16:21:01 -0700 Subject: [PATCH 1725/3965] Simplify Autonat Structure * Single goroutine managing autonat-relevent events. * Watching incoming connections and local interface changes as signals. * Emit a single 'rechabilitychanged' persistent event fix #40 fix #36 fix #35 fix #34 fix #11 obsolete #28 obsolete #9 Co-authored-by: Aarsh Shah Co-authored-by: Adin Schmahmann --- p2p/host/autonat/autonat.go | 353 +++++++++++++++++-------------- p2p/host/autonat/autonat_test.go | 145 ++++++++++--- p2p/host/autonat/notify.go | 25 +-- 3 files changed, 321 insertions(+), 202 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 2d6c9786f9..951d12c06f 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -4,29 +4,17 @@ import ( "context" "errors" "math/rand" - "sync" + "sync/atomic" "time" "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" - "github.com/libp2p/go-libp2p-core/peerstore" - ma "github.com/multiformats/go-multiaddr" -) -// NATStatus is the state of NAT as detected by the ambient service. -type NATStatus int - -const ( - // NAT status is unknown; this means that the ambient service has not been - // able to decide the presence of NAT in the most recent attempt to test - // dial through known autonat peers. initial state. - NATStatusUnknown NATStatus = iota - // NAT status is publicly dialable - NATStatusPublic - // NAT status is private network - NATStatusPrivate + "github.com/libp2p/go-eventbus" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) var ( @@ -39,7 +27,7 @@ var ( // AutoNAT is the interface for ambient NAT autodiscovery type AutoNAT interface { // Status returns the current NAT status - Status() NATStatus + Status() network.Reachability // PublicAddr returns the public dial address when NAT status is public and an // error otherwise PublicAddr() (ma.Multiaddr, error) @@ -52,20 +40,27 @@ type AmbientAutoNAT struct { getAddrs GetAddrs - mx sync.Mutex - peers map[peer.ID][]ma.Multiaddr - status NATStatus - addr ma.Multiaddr + inboundConn chan network.Conn + observations chan autoNATResult + // status is an autoNATResult reflecting current status. + status atomic.Value // Reflects the confidence on of the NATStatus being private, as a single // dialback may fail for reasons unrelated to NAT. // If it is <3, then multiple autoNAT peers may be contacted for dialback // If only a single autoNAT peer is known, then the confidence increases // for each failure until it reaches 3. - confidence int + confidence int + lastInbound time.Time + lastProbe time.Time + + subAddrUpdated event.Subscription + + emitReachabilityChanged event.Emitter +} - emitUnknown event.Emitter - emitPublic event.Emitter - emitPrivate event.Emitter +type autoNATResult struct { + network.Reachability + address ma.Multiaddr } // NewAutoNAT creates a new ambient NAT autodiscovery instance attached to a host @@ -75,21 +70,22 @@ func NewAutoNAT(ctx context.Context, h host.Host, getAddrs GetAddrs) AutoNAT { getAddrs = h.Addrs } - emitUnknown, _ := h.EventBus().Emitter(new(event.EvtLocalRoutabilityUnknown)) - emitPublic, _ := h.EventBus().Emitter(new(event.EvtLocalRoutabilityPublic)) - emitPrivate, _ := h.EventBus().Emitter(new(event.EvtLocalRoutabilityPrivate)) + subAddrUpdated, _ := h.EventBus().Subscribe(new(event.EvtLocalAddressesUpdated)) + + emitReachabilityChanged, _ := h.EventBus().Emitter(new(event.EvtLocalReachabilityChanged), eventbus.Stateful) as := &AmbientAutoNAT{ - ctx: ctx, - host: h, - getAddrs: getAddrs, - peers: make(map[peer.ID][]ma.Multiaddr), - status: NATStatusUnknown, - - emitUnknown: emitUnknown, - emitPublic: emitPublic, - emitPrivate: emitPrivate, + ctx: ctx, + host: h, + getAddrs: getAddrs, + inboundConn: make(chan network.Conn, 5), + observations: make(chan autoNATResult, 1), + + subAddrUpdated: subAddrUpdated, + + emitReachabilityChanged: emitReachabilityChanged, } + as.status.Store(autoNATResult{network.ReachabilityUnknown, nil}) h.Network().Notify(as) go as.background() @@ -97,177 +93,222 @@ func NewAutoNAT(ctx context.Context, h host.Host, getAddrs GetAddrs) AutoNAT { return as } -func (as *AmbientAutoNAT) Status() NATStatus { - as.mx.Lock() - defer as.mx.Unlock() - return as.status +// Status returns the AutoNAT observed reachability status. +func (as *AmbientAutoNAT) Status() network.Reachability { + s := as.status.Load().(autoNATResult) + return s.Reachability } -func (as *AmbientAutoNAT) updateStatus(s NATStatus) { - as.status = s - switch s { - case NATStatusUnknown: - as.emitUnknown.Emit(event.EvtLocalRoutabilityUnknown{}) - case NATStatusPublic: - as.emitPublic.Emit(event.EvtLocalRoutabilityPublic{}) - case NATStatusPrivate: - as.emitPrivate.Emit(event.EvtLocalRoutabilityPrivate{}) - } +func (as *AmbientAutoNAT) emitStatus() { + status := as.status.Load().(autoNATResult) + as.emitReachabilityChanged.Emit(event.EvtLocalReachabilityChanged{Reachability: status.Reachability}) } +// PublicAddr returns the publicly connectable Multiaddr of this node if one is known. func (as *AmbientAutoNAT) PublicAddr() (ma.Multiaddr, error) { - as.mx.Lock() - defer as.mx.Unlock() - - if as.status != NATStatusPublic { + s := as.status.Load().(autoNATResult) + if s.Reachability != network.ReachabilityPublic { return nil, errors.New("NAT Status is not public") } - return as.addr, nil + return s.address, nil +} + +func ipInList(candidate ma.Multiaddr, list []ma.Multiaddr) bool { + candidateIP, _ := manet.ToIP(candidate) + for _, i := range list { + if ip, err := manet.ToIP(i); err == nil && ip.Equal(candidateIP) { + return true + } + } + return false } func (as *AmbientAutoNAT) background() { // wait a bit for the node to come online and establish some connections // before starting autodetection - select { - case <-time.After(AutoNATBootDelay): - case <-as.ctx.Done(): - return - } + delay := AutoNATBootDelay - for { - as.autodetect() + var lastAddrUpdated time.Time + addrUpdatedChan := as.subAddrUpdated.Out() + defer as.subAddrUpdated.Close() + defer as.emitReachabilityChanged.Close() - delay := AutoNATRefreshInterval - if as.status == NATStatusUnknown { - delay = AutoNATRetryInterval - } + timer := time.NewTimer(delay) + defer timer.Stop() + timerRunning := true + for { select { - case <-time.After(delay): + // new connection occured. + case conn := <-as.inboundConn: + localAddrs := as.host.Addrs() + ca := as.status.Load().(autoNATResult) + if ca.address != nil { + localAddrs = append(localAddrs, ca.address) + } + if !ipInList(conn.RemoteMultiaddr(), localAddrs) { + as.lastInbound = time.Now() + } + + case <-addrUpdatedChan: + if !lastAddrUpdated.Add(time.Second).After(time.Now()) { + lastAddrUpdated = time.Now() + if as.confidence > 1 { + as.confidence-- + } + } + + // probe finished. + case result, ok := <-as.observations: + if !ok { + return + } + as.recordObservation(result) + case <-timer.C: + timerRunning = false case <-as.ctx.Done(): return } - } -} - -func (as *AmbientAutoNAT) autodetect() { - peers := as.getPeers() - - if len(peers) == 0 { - log.Debugf("skipping NAT auto detection; no autonat peers") - return - } - - cli := NewAutoNATClient(as.host, as.getAddrs) - ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) - defer cancel() - var result struct { - sync.Mutex - private int - public int - pubaddr ma.Multiaddr + // Drain the timer channel if it hasn't fired in preparation for Resetting it. + if timerRunning && !timer.Stop() { + <-timer.C + } + timer.Reset(as.scheduleProbe()) + timerRunning = true } +} - probe := 3 - as.confidence - if probe == 0 { - probe = 1 - } - if probe > len(peers) { - probe = len(peers) +// scheduleProbe calculates when the next probe should be scheduled for, +// and launches it if that time is now. +func (as *AmbientAutoNAT) scheduleProbe() time.Duration { + // Our baseline is a probe every 'AutoNATRefreshInterval' + // This is modulated by: + // * recent inbound connections make us willing to wait up to 2x longer between probes. + // * low confidence makes us speed up between probes. + fixedNow := time.Now() + currentStatus := as.status.Load().(autoNATResult) + + nextProbe := fixedNow + if !as.lastProbe.IsZero() { + untilNext := AutoNATRefreshInterval + if currentStatus.Reachability == network.ReachabilityUnknown { + untilNext = AutoNATRetryInterval + } else if as.confidence < 3 { + untilNext = AutoNATRetryInterval + } else if currentStatus.Reachability == network.ReachabilityPublic && as.lastInbound.After(as.lastProbe) { + untilNext *= 2 + } + nextProbe = as.lastProbe.Add(untilNext) } - - var wg sync.WaitGroup - - for _, pi := range peers[:probe] { - wg.Add(1) - go func(pi peer.AddrInfo) { - defer wg.Done() - - as.host.Peerstore().AddAddrs(pi.ID, pi.Addrs, peerstore.TempAddrTTL) - a, err := cli.DialBack(ctx, pi.ID) - - switch { - case err == nil: - log.Debugf("Dialback through %s successful; public address is %s", pi.ID.Pretty(), a.String()) - result.Lock() - result.public++ - result.pubaddr = a - result.Unlock() - - case IsDialError(err): - log.Debugf("Dialback through %s failed", pi.ID.Pretty()) - result.Lock() - result.private++ - result.Unlock() - - default: - log.Debugf("Dialback error through %s: %s", pi.ID.Pretty(), err) - } - }(pi) + if fixedNow.After(nextProbe) || fixedNow == nextProbe { + go as.probeNextPeer() + return AutoNATRetryInterval } + return nextProbe.Sub(fixedNow) +} - wg.Wait() - - as.mx.Lock() - if result.public > 0 { +// Update the current status based on an observed result. +func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { + currentStatus := as.status.Load().(autoNATResult) + if observation.Reachability == network.ReachabilityPublic { log.Debugf("NAT status is public") - if as.status == NATStatusPrivate { + changed := false + if currentStatus.Reachability != network.ReachabilityPublic { // we are flipping our NATStatus, so confidence drops to 0 as.confidence = 0 + changed = true } else if as.confidence < 3 { as.confidence++ } - as.addr = result.pubaddr - as.updateStatus(NATStatusPublic) - } else if result.private > 0 { + if observation.address != nil { + if !changed && currentStatus.address != nil && !observation.address.Equal(currentStatus.address) { + as.confidence-- + } + if currentStatus.address == nil || !observation.address.Equal(currentStatus.address) { + changed = true + } + as.status.Store(observation) + } + if observation.address != nil && changed { + as.emitStatus() + } + } else if observation.Reachability == network.ReachabilityPrivate { log.Debugf("NAT status is private") - if as.status == NATStatusPublic { - // we are flipping our NATStatus, so confidence drops to 0 - as.confidence = 0 + if currentStatus.Reachability == network.ReachabilityPublic { + if as.confidence > 0 { + as.confidence-- + } else { + // we are flipping our NATStatus, so confidence drops to 0 + as.confidence = 0 + as.status.Store(observation) + as.emitStatus() + } } else if as.confidence < 3 { as.confidence++ + as.status.Store(observation) + if currentStatus.Reachability != network.ReachabilityPrivate { + as.emitStatus() + } } - as.addr = nil - as.updateStatus(NATStatusPrivate) } else if as.confidence > 0 { // don't just flip to unknown, reduce confidence first as.confidence-- } else { log.Debugf("NAT status is unknown") - as.addr = nil - as.updateStatus(NATStatusUnknown) + as.status.Store(autoNATResult{network.ReachabilityUnknown, nil}) + if currentStatus.Reachability != network.ReachabilityUnknown { + as.emitStatus() + } } - as.mx.Unlock() } -func (as *AmbientAutoNAT) getPeers() []peer.AddrInfo { - as.mx.Lock() - defer as.mx.Unlock() +func (as *AmbientAutoNAT) probe(pi *peer.AddrInfo) { + cli := NewAutoNATClient(as.host, as.getAddrs) + ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) + defer cancel() + + a, err := cli.DialBack(ctx, pi.ID) + + switch { + case err == nil: + log.Debugf("Dialback through %s successful; public address is %s", pi.ID.Pretty(), a.String()) + as.observations <- autoNATResult{network.ReachabilityPublic, a} + case IsDialError(err): + log.Debugf("Dialback through %s failed", pi.ID.Pretty()) + as.observations <- autoNATResult{network.ReachabilityPrivate, nil} + default: + as.observations <- autoNATResult{network.ReachabilityUnknown, nil} + } +} - if len(as.peers) == 0 { - return nil +func (as *AmbientAutoNAT) probeNextPeer() { + peers := as.host.Network().Peers() + if len(peers) == 0 { + return } - var connected, others []peer.AddrInfo + addrs := make([]peer.AddrInfo, 0, len(peers)) - for p, addrs := range as.peers { - if as.host.Network().Connectedness(p) == network.Connected { - connected = append(connected, peer.AddrInfo{ID: p, Addrs: addrs}) - } else { - others = append(others, peer.AddrInfo{ID: p, Addrs: addrs}) + for _, p := range peers { + info := as.host.Peerstore().PeerInfo(p) + // Exclude peers which don't support the autonat protocol. + if proto, err := as.host.Peerstore().SupportsProtocols(p, AutoNATProto); len(proto) == 0 || err != nil { + continue } + addrs = append(addrs, info) } + // TODO: track and exclude recently probed peers. - shufflePeers(connected) - - if len(connected) < 3 { - shufflePeers(others) - return append(connected, others...) - } else { - return connected + if len(addrs) == 0 { + return } + + shufflePeers(addrs) + + as.lastProbe = time.Now() + as.probe(&addrs[0]) } func shufflePeers(peers []peer.AddrInfo) { diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 74a429ca4f..ddf6d5cde2 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -18,7 +18,7 @@ import ( ) func init() { - AutoNATBootDelay = 1 * time.Second + AutoNATBootDelay = 100 * time.Millisecond AutoNATRefreshInterval = 1 * time.Second AutoNATRetryInterval = 1 * time.Second AutoNATIdentifyDelay = 100 * time.Millisecond @@ -73,10 +73,9 @@ func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Mes func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, AutoNAT) { h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + h.Peerstore().AddAddrs(ash.ID(), ash.Addrs(), time.Minute) + h.Peerstore().AddProtocols(ash.ID(), AutoNATProto) a := NewAutoNAT(ctx, h, nil) - a.(*AmbientAutoNAT).mx.Lock() - a.(*AmbientAutoNAT).peers[ash.ID()] = ash.Addrs() - a.(*AmbientAutoNAT).mx.Unlock() return h, a } @@ -88,6 +87,19 @@ func connect(t *testing.T, a, b host.Host) { } } +func expectEvent(t *testing.T, s event.Subscription, expected network.Reachability) { + select { + case e := <-s.Out(): + ev, ok := e.(event.EvtLocalReachabilityChanged) + if !ok || ev.Reachability != expected { + t.Fatal("got wrong event type from the bus") + } + + case <-time.After(100 * time.Millisecond): + t.Fatal("failed to get the reachability event from the bus") + } +} + // tests func TestAutoNATPrivate(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) @@ -97,37 +109,57 @@ func TestAutoNATPrivate(t *testing.T) { hc, an := makeAutoNAT(ctx, t, hs) // subscribe to AutoNat events - s, err := hc.EventBus().Subscribe(&event.EvtLocalRoutabilityPrivate{}) + s, err := hc.EventBus().Subscribe(&event.EvtLocalReachabilityChanged{}) if err != nil { - t.Fatalf("failed to subscribe to event EvtLocalRoutabilityPrivate, err=%s", err) + t.Fatalf("failed to subscribe to event EvtLocalReachabilityChanged, err=%s", err) } status := an.Status() - if status != NATStatusUnknown { + if status != network.ReachabilityUnknown { t.Fatalf("unexpected NAT status: %d", status) } connect(t, hs, hc) - time.Sleep(2 * time.Second) + time.Sleep(1 * time.Second) status = an.Status() - if status != NATStatusPrivate { + if status != network.ReachabilityPrivate { t.Fatalf("unexpected NAT status: %d", status) } - select { - case e := <-s.Out(): - _, ok := e.(event.EvtLocalRoutabilityPrivate) - if !ok { - t.Fatal("got wrong event type from the bus") - } + expectEvent(t, s, network.ReachabilityPrivate) +} + +func TestAutoNATPublic(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() - case <-time.After(1 * time.Second): - t.Fatal("failed to get the EvtLocalRoutabilityPrivate event from the bus") + hs := makeAutoNATServicePublic(ctx, t) + hc, an := makeAutoNAT(ctx, t, hs) + + // subscribe to AutoNat events + s, err := hc.EventBus().Subscribe(&event.EvtLocalReachabilityChanged{}) + if err != nil { + t.Fatalf("failed to subscribe to event EvtLocalReachabilityChanged, err=%s", err) } + + status := an.Status() + if status != network.ReachabilityUnknown { + t.Fatalf("unexpected NAT status: %d", status) + } + + connect(t, hs, hc) + time.Sleep(200 * time.Millisecond) + + status = an.Status() + if status != network.ReachabilityPublic { + t.Fatalf("unexpected NAT status: %d", status) + } + + expectEvent(t, s, network.ReachabilityPublic) } -func TestAutoNATPublic(t *testing.T) { +func TestAutoNATPublictoPrivate(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -135,32 +167,89 @@ func TestAutoNATPublic(t *testing.T) { hc, an := makeAutoNAT(ctx, t, hs) // subscribe to AutoNat events - s, err := hc.EventBus().Subscribe(&event.EvtLocalRoutabilityPublic{}) + s, err := hc.EventBus().Subscribe(&event.EvtLocalReachabilityChanged{}) if err != nil { t.Fatalf("failed to subscribe to event EvtLocalRoutabilityPublic, err=%s", err) } status := an.Status() - if status != NATStatusUnknown { + if status != network.ReachabilityUnknown { t.Fatalf("unexpected NAT status: %d", status) } connect(t, hs, hc) + time.Sleep(200 * time.Millisecond) + + status = an.Status() + if status != network.ReachabilityPublic { + t.Fatalf("unexpected NAT status: %d", status) + } + + expectEvent(t, s, network.ReachabilityPublic) + + hs.SetStreamHandler(AutoNATProto, sayAutoNATPrivate) time.Sleep(2 * time.Second) status = an.Status() - if status != NATStatusPublic { + if status != network.ReachabilityPrivate { t.Fatalf("unexpected NAT status: %d", status) } +} + +func TestAutoNATObservationRecording(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hs := makeAutoNATServicePublic(ctx, t) + hc, ani := makeAutoNAT(ctx, t, hs) + an := ani.(*AmbientAutoNAT) + + s, err := hc.EventBus().Subscribe(&event.EvtLocalReachabilityChanged{}) + if err != nil { + t.Fatalf("failed to subscribe to event EvtLocalRoutabilityPublic, err=%s", err) + } + + // pubic observation without address should be ignored. + an.recordObservation(autoNATResult{network.ReachabilityPublic, nil}) + if an.Status() != network.ReachabilityUnknown { + t.Fatalf("unexpected transition") + } select { - case e := <-s.Out(): - _, ok := e.(event.EvtLocalRoutabilityPublic) - if !ok { - t.Fatal("got wrong event type from the bus") - } + case _ = <-s.Out(): + t.Fatal("not expecting a public reachability event") + default: + //expected + } + + addr, _ := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234") + an.recordObservation(autoNATResult{network.ReachabilityPublic, addr}) + if an.Status() != network.ReachabilityPublic { + t.Fatalf("failed to transition to public.") + } - case <-time.After(1 * time.Second): - t.Fatal("failed to get the EvtLocalRoutabilityPublic event from the bus") + expectEvent(t, s, network.ReachabilityPublic) + + // a single recording should have confidence still at 0, and transition to private quickly. + an.recordObservation(autoNATResult{network.ReachabilityPrivate, nil}) + if an.Status() != network.ReachabilityPrivate { + t.Fatalf("failed to transition to private.") + } + + expectEvent(t, s, network.ReachabilityPrivate) + + // stronger public confidence should be harder to undo. + an.recordObservation(autoNATResult{network.ReachabilityPublic, addr}) + an.recordObservation(autoNATResult{network.ReachabilityPublic, addr}) + if an.Status() != network.ReachabilityPublic { + t.Fatalf("failed to transition to public.") } + + expectEvent(t, s, network.ReachabilityPublic) + + an.recordObservation(autoNATResult{network.ReachabilityPrivate, nil}) + if an.Status() != network.ReachabilityPublic { + t.Fatalf("too-extreme private transition.") + } + } diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index 4ea6561603..ed8d8702dc 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -6,6 +6,7 @@ import ( "github.com/libp2p/go-libp2p-core/network" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) var _ network.Notifiee = (*AmbientAutoNAT)(nil) @@ -18,25 +19,13 @@ func (as *AmbientAutoNAT) OpenedStream(net network.Network, s network.Stream) {} func (as *AmbientAutoNAT) ClosedStream(net network.Network, s network.Stream) {} func (as *AmbientAutoNAT) Connected(net network.Network, c network.Conn) { - p := c.RemotePeer() - - go func() { - // add some delay for identify - time.Sleep(AutoNATIdentifyDelay) - - protos, err := as.host.Peerstore().SupportsProtocols(p, AutoNATProto) - if err != nil { - log.Debugf("error retrieving supported protocols for peer %s: %s", p, err) - return - } - - if len(protos) > 0 { - log.Infof("Discovered AutoNAT peer %s", p.Pretty()) - as.mx.Lock() - as.peers[p] = as.host.Peerstore().Addrs(p) - as.mx.Unlock() + if c.Stat().Direction == network.DirInbound && + manet.IsPublicAddr(c.RemoteMultiaddr()) { + select { + case as.inboundConn <- c: + default: } - }() + } } func (as *AmbientAutoNAT) Disconnected(net network.Network, c network.Conn) {} From eee717b1d836b4ae272c593f2c4732154d5dd59d Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 12 Mar 2020 08:39:56 -0700 Subject: [PATCH 1726/3965] change autonat interface to use functional options --- p2p/host/autonat/autonat.go | 57 ++++++++++++++------------------ p2p/host/autonat/autonat_test.go | 9 +---- p2p/host/autonat/client.go | 12 +------ p2p/host/autonat/interface.go | 32 ++++++++++++++++++ p2p/host/autonat/notify.go | 4 --- p2p/host/autonat/options.go | 57 ++++++++++++++++++++++++++++++++ 6 files changed, 116 insertions(+), 55 deletions(-) create mode 100644 p2p/host/autonat/interface.go create mode 100644 p2p/host/autonat/options.go diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 951d12c06f..a159f9b696 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -17,28 +17,12 @@ import ( manet "github.com/multiformats/go-multiaddr-net" ) -var ( - AutoNATBootDelay = 15 * time.Second - AutoNATRetryInterval = 90 * time.Second - AutoNATRefreshInterval = 15 * time.Minute - AutoNATRequestTimeout = 30 * time.Second -) - -// AutoNAT is the interface for ambient NAT autodiscovery -type AutoNAT interface { - // Status returns the current NAT status - Status() network.Reachability - // PublicAddr returns the public dial address when NAT status is public and an - // error otherwise - PublicAddr() (ma.Multiaddr, error) -} - // AmbientAutoNAT is the implementation of ambient NAT autodiscovery type AmbientAutoNAT struct { ctx context.Context host host.Host - getAddrs GetAddrs + *config inboundConn chan network.Conn observations chan autoNATResult @@ -63,21 +47,30 @@ type autoNATResult struct { address ma.Multiaddr } -// NewAutoNAT creates a new ambient NAT autodiscovery instance attached to a host -// If getAddrs is nil, h.Addrs will be used -func NewAutoNAT(ctx context.Context, h host.Host, getAddrs GetAddrs) AutoNAT { - if getAddrs == nil { - getAddrs = h.Addrs +// New creates a new NAT autodiscovery system attached to a host +func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { + conf := new(config) + + if err := defaults(conf); err != nil { + return nil, err + } + if conf.getAddressFunc == nil { + conf.getAddressFunc = h.Addrs } - subAddrUpdated, _ := h.EventBus().Subscribe(new(event.EvtLocalAddressesUpdated)) + for _, o := range options { + if err := o(conf); err != nil { + return nil, err + } + } + subAddrUpdated, _ := h.EventBus().Subscribe(new(event.EvtLocalAddressesUpdated)) emitReachabilityChanged, _ := h.EventBus().Emitter(new(event.EvtLocalReachabilityChanged), eventbus.Stateful) as := &AmbientAutoNAT{ ctx: ctx, host: h, - getAddrs: getAddrs, + config: conf, inboundConn: make(chan network.Conn, 5), observations: make(chan autoNATResult, 1), @@ -90,7 +83,7 @@ func NewAutoNAT(ctx context.Context, h host.Host, getAddrs GetAddrs) AutoNAT { h.Network().Notify(as) go as.background() - return as + return as, nil } // Status returns the AutoNAT observed reachability status. @@ -127,7 +120,7 @@ func ipInList(candidate ma.Multiaddr, list []ma.Multiaddr) bool { func (as *AmbientAutoNAT) background() { // wait a bit for the node to come online and establish some connections // before starting autodetection - delay := AutoNATBootDelay + delay := as.config.bootDelay var lastAddrUpdated time.Time addrUpdatedChan := as.subAddrUpdated.Out() @@ -192,11 +185,11 @@ func (as *AmbientAutoNAT) scheduleProbe() time.Duration { nextProbe := fixedNow if !as.lastProbe.IsZero() { - untilNext := AutoNATRefreshInterval + untilNext := as.config.refreshInterval if currentStatus.Reachability == network.ReachabilityUnknown { - untilNext = AutoNATRetryInterval + untilNext = as.config.retryInterval } else if as.confidence < 3 { - untilNext = AutoNATRetryInterval + untilNext = as.config.retryInterval } else if currentStatus.Reachability == network.ReachabilityPublic && as.lastInbound.After(as.lastProbe) { untilNext *= 2 } @@ -204,7 +197,7 @@ func (as *AmbientAutoNAT) scheduleProbe() time.Duration { } if fixedNow.After(nextProbe) || fixedNow == nextProbe { go as.probeNextPeer() - return AutoNATRetryInterval + return as.config.retryInterval } return nextProbe.Sub(fixedNow) } @@ -265,8 +258,8 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { } func (as *AmbientAutoNAT) probe(pi *peer.AddrInfo) { - cli := NewAutoNATClient(as.host, as.getAddrs) - ctx, cancel := context.WithTimeout(as.ctx, AutoNATRequestTimeout) + cli := NewAutoNATClient(as.host, as.config.getAddressFunc) + ctx, cancel := context.WithTimeout(as.ctx, as.config.requestTimeout) defer cancel() a, err := cli.DialBack(ctx, pi.ID) diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index ddf6d5cde2..0304cf324f 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -17,13 +17,6 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -func init() { - AutoNATBootDelay = 100 * time.Millisecond - AutoNATRefreshInterval = 1 * time.Second - AutoNATRetryInterval = 1 * time.Second - AutoNATIdentifyDelay = 100 * time.Millisecond -} - // these are mock service implementations for testing func makeAutoNATServicePrivate(ctx context.Context, t *testing.T) host.Host { h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) @@ -75,7 +68,7 @@ func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, A h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) h.Peerstore().AddAddrs(ash.ID(), ash.Addrs(), time.Minute) h.Peerstore().AddProtocols(ash.ID(), AutoNATProto) - a := NewAutoNAT(ctx, h, nil) + a, _ := New(ctx, h, WithSchedule(100*time.Millisecond, time.Second), WithoutStartupDelay()) return h, a } diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index 0d8549b48a..e2acab178e 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -14,25 +14,15 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -// AutoNATClient is a stateless client interface to AutoNAT peers -type AutoNATClient interface { - // DialBack requests from a peer providing AutoNAT services to test dial back - // and report the address on a successful connection. - DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) -} - // AutoNATError is the class of errors signalled by AutoNAT services type AutoNATError struct { Status pb.Message_ResponseStatus Text string } -// GetAddrs is a function that returns the addresses to dial back -type GetAddrs func() []ma.Multiaddr - // NewAutoNATClient creates a fresh instance of an AutoNATClient // If getAddrs is nil, h.Addrs will be used -func NewAutoNATClient(h host.Host, getAddrs GetAddrs) AutoNATClient { +func NewAutoNATClient(h host.Host, getAddrs GetAddrs) Client { if getAddrs == nil { getAddrs = h.Addrs } diff --git a/p2p/host/autonat/interface.go b/p2p/host/autonat/interface.go new file mode 100644 index 0000000000..af010b98e6 --- /dev/null +++ b/p2p/host/autonat/interface.go @@ -0,0 +1,32 @@ +package autonat + +import ( + "context" + + "github.com/libp2p/go-libp2p-core/network" + "github.com/libp2p/go-libp2p-core/peer" + + ma "github.com/multiformats/go-multiaddr" +) + +// AutoNAT is the interface for NAT autodiscovery +type AutoNAT interface { + // Status returns the current NAT status + Status() network.Reachability + // PublicAddr returns the public dial address when NAT status is public and an + // error otherwise + PublicAddr() (ma.Multiaddr, error) +} + +// Client is a stateless client interface to AutoNAT peers +type Client interface { + // DialBack requests from a peer providing AutoNAT services to test dial back + // and report the address on a successful connection. + DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) +} + +// GetAddrs is a function returning the candidate addresses for the local host. +type GetAddrs func() []ma.Multiaddr + +// Option is an Autonat option for configuration +type Option func(*config) error diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index ed8d8702dc..a444df62db 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -1,8 +1,6 @@ package autonat import ( - "time" - "github.com/libp2p/go-libp2p-core/network" ma "github.com/multiformats/go-multiaddr" @@ -11,8 +9,6 @@ import ( var _ network.Notifiee = (*AmbientAutoNAT)(nil) -var AutoNATIdentifyDelay = 5 * time.Second - func (as *AmbientAutoNAT) Listen(net network.Network, a ma.Multiaddr) {} func (as *AmbientAutoNAT) ListenClose(net network.Network, a ma.Multiaddr) {} func (as *AmbientAutoNAT) OpenedStream(net network.Network, s network.Stream) {} diff --git a/p2p/host/autonat/options.go b/p2p/host/autonat/options.go new file mode 100644 index 0000000000..4d753495d8 --- /dev/null +++ b/p2p/host/autonat/options.go @@ -0,0 +1,57 @@ +package autonat + +import ( + "time" +) + +// config holds configurable options for the autonat subsystem. +type config struct { + getAddressFunc GetAddrs + bootDelay time.Duration + retryInterval time.Duration + refreshInterval time.Duration + requestTimeout time.Duration +} + +var defaults = func(c *config) error { + c.bootDelay = 15 * time.Second + c.retryInterval = 90 * time.Second + c.refreshInterval = 15 * time.Minute + c.requestTimeout = 30 * time.Second + + return nil +} + +// WithAddresses allows overriding which Addresses the AutoNAT client beliieves +// are "its own". Useful for testing, or for more exotic port-forwarding +// scenarios where the host may be listening on different ports than it wants +// to externally advertise or verify connectability on. +func WithAddresses(addrFunc GetAddrs) Option { + return func(c *config) error { + c.getAddressFunc = addrFunc + return nil + } +} + +// WithSchedule configures how agressively probes will be made to verify the +// address of the host. retryInterval indicates how often probes should be made +// when the host lacks confident about its address, while refresh interval +// is the schedule of periodic probes when the host believes it knows its +// steady-state reachability. +func WithSchedule(retryInterval, refreshInterval time.Duration) Option { + return func(c *config) error { + c.retryInterval = retryInterval + c.refreshInterval = refreshInterval + return nil + } +} + +// WithoutStartupDelay removes the initial delay the NAT subsystem typically +// uses as a buffer for ensuring that connectivity and guesses as to the hosts +// local interfaces have settled down during startup. +func WithoutStartupDelay() Option { + return func(c *config) error { + c.bootDelay = 1 + return nil + } +} From 4548d1e922716ee7e47f4f645f009e286df39061 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 12 Mar 2020 11:45:22 -0700 Subject: [PATCH 1727/3965] remove a foot-gun --- p2p/host/autonat/options.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/p2p/host/autonat/options.go b/p2p/host/autonat/options.go index 4d753495d8..6987d3aba6 100644 --- a/p2p/host/autonat/options.go +++ b/p2p/host/autonat/options.go @@ -1,6 +1,7 @@ package autonat import ( + "errors" "time" ) @@ -28,6 +29,9 @@ var defaults = func(c *config) error { // to externally advertise or verify connectability on. func WithAddresses(addrFunc GetAddrs) Option { return func(c *config) error { + if addrFunc == nil { + return errors.New("invalid address function supplied") + } c.getAddressFunc = addrFunc return nil } From 1ca135881c276ac4919d585c78ec074fe75f6155 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Fri, 13 Mar 2020 09:40:51 +0700 Subject: [PATCH 1728/3965] update to Go 1.14 --- p2p/security/tls/transport_test.go | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 62e9b0e73a..901c85e1ac 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -327,9 +327,12 @@ var _ = Describe("Transport", func() { transforms := []transform{ { - name: "private key used in the TLS handshake doesn't match the public key in the cert", - apply: invalidateCertChain, - remoteErr: Equal("tls: invalid certificate signature"), + name: "private key used in the TLS handshake doesn't match the public key in the cert", + apply: invalidateCertChain, + remoteErr: Or( + Equal("tls: invalid signature by the client certificate: ECDSA verification failure"), + Equal("tls: invalid signature by the server certificate: ECDSA verification failure"), + ), }, { name: "certificate chain contains 2 certs", @@ -339,7 +342,7 @@ var _ = Describe("Transport", func() { { name: "cert is expired", apply: expiredCert, - remoteErr: Equal("certificate verification failed: x509: certificate has expired or is not yet valid"), + remoteErr: ContainSubstring("certificate has expired or is not yet valid"), }, { name: "cert doesn't have the key extension", From 29c56d8b005c84145761b5ad2b96ecc6bce3cb19 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sun, 15 Mar 2020 09:33:40 +0700 Subject: [PATCH 1729/3965] add support for multiaddr filtering --- p2p/transport/quic/conn_test.go | 57 +++++++++++++++++---- p2p/transport/quic/filtered_conn.go | 34 ++++++++++++ p2p/transport/quic/libp2pquic_suite_test.go | 15 ++++-- p2p/transport/quic/listener_test.go | 2 +- p2p/transport/quic/reuse_base.go | 16 ++++-- p2p/transport/quic/reuse_linux_test.go | 2 +- p2p/transport/quic/reuse_not_win.go | 6 ++- p2p/transport/quic/reuse_test.go | 2 +- p2p/transport/quic/reuse_win.go | 10 ++-- p2p/transport/quic/transport.go | 11 ++-- 10 files changed, 123 insertions(+), 32 deletions(-) create mode 100644 p2p/transport/quic/filtered_conn.go diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index ef234bc5bc..f7d7253c9c 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -7,11 +7,13 @@ import ( "fmt" "io/ioutil" mrand "math/rand" + "net" "time" ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" tpt "github.com/libp2p/go-libp2p-core/transport" + filter "github.com/libp2p/go-maddr-filter" ma "github.com/multiformats/go-multiaddr" . "github.com/onsi/ginkgo" @@ -62,12 +64,12 @@ var _ = Describe("Connection", func() { }) It("handshakes on IPv4", func() { - serverTransport, err := NewTransport(serverKey, nil) + serverTransport, err := NewTransport(serverKey, nil, nil) Expect(err).ToNot(HaveOccurred()) ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") defer ln.Close() - clientTransport, err := NewTransport(clientKey, nil) + clientTransport, err := NewTransport(clientKey, nil, nil) Expect(err).ToNot(HaveOccurred()) conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) @@ -86,12 +88,12 @@ var _ = Describe("Connection", func() { }) It("handshakes on IPv6", func() { - serverTransport, err := NewTransport(serverKey, nil) + serverTransport, err := NewTransport(serverKey, nil, nil) Expect(err).ToNot(HaveOccurred()) ln := runServer(serverTransport, "/ip6/::1/udp/0/quic") defer ln.Close() - clientTransport, err := NewTransport(clientKey, nil) + clientTransport, err := NewTransport(clientKey, nil, nil) Expect(err).ToNot(HaveOccurred()) conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) @@ -110,12 +112,12 @@ var _ = Describe("Connection", func() { }) It("opens and accepts streams", func() { - serverTransport, err := NewTransport(serverKey, nil) + serverTransport, err := NewTransport(serverKey, nil, nil) Expect(err).ToNot(HaveOccurred()) ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") defer ln.Close() - clientTransport, err := NewTransport(clientKey, nil) + clientTransport, err := NewTransport(clientKey, nil, nil) Expect(err).ToNot(HaveOccurred()) conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) @@ -139,11 +141,11 @@ var _ = Describe("Connection", func() { It("fails if the peer ID doesn't match", func() { thirdPartyID, _ := createPeer() - serverTransport, err := NewTransport(serverKey, nil) + serverTransport, err := NewTransport(serverKey, nil, nil) Expect(err).ToNot(HaveOccurred()) ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") - clientTransport, err := NewTransport(clientKey, nil) + clientTransport, err := NewTransport(clientKey, nil, nil) Expect(err).ToNot(HaveOccurred()) // dial, but expect the wrong peer ID _, err = clientTransport.Dial(context.Background(), ln.Multiaddr(), thirdPartyID) @@ -161,14 +163,47 @@ var _ = Describe("Connection", func() { Eventually(done).Should(BeClosed()) }) + It("filters addresses", func() { + filters := filter.NewFilters() + ipNet := net.IPNet{ + IP: net.IPv4(127, 0, 0, 1), + Mask: net.IPv4Mask(255, 255, 255, 255), + } + filters.AddFilter(ipNet, filter.ActionDeny) + testMA, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/1234/quic") + Expect(err).ToNot(HaveOccurred()) + Expect(filters.AddrBlocked(testMA)).To(BeTrue()) + + serverTransport, err := NewTransport(serverKey, nil, filters) + Expect(err).ToNot(HaveOccurred()) + ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + defer ln.Close() + + clientTransport, err := NewTransport(clientKey, nil, nil) + Expect(err).ToNot(HaveOccurred()) + + // make sure that connection attempts fails + quicConfig.HandshakeTimeout = 250 * time.Millisecond + _, err = clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) + Expect(err).To(HaveOccurred()) + Expect(err.(net.Error).Timeout()).To(BeTrue()) + + // now allow the address and make sure the connection goes through + quicConfig.HandshakeTimeout = 2 * time.Second + filters.AddFilter(ipNet, filter.ActionAccept) + conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) + Expect(err).ToNot(HaveOccurred()) + conn.Close() + }) + It("dials to two servers at the same time", func() { serverID2, serverKey2 := createPeer() - serverTransport, err := NewTransport(serverKey, nil) + serverTransport, err := NewTransport(serverKey, nil, nil) Expect(err).ToNot(HaveOccurred()) ln1 := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") - serverTransport2, err := NewTransport(serverKey2, nil) defer ln1.Close() + serverTransport2, err := NewTransport(serverKey2, nil, nil) Expect(err).ToNot(HaveOccurred()) ln2 := runServer(serverTransport2, "/ip4/127.0.0.1/udp/0/quic") defer ln2.Close() @@ -194,7 +229,7 @@ var _ = Describe("Connection", func() { } }() - clientTransport, err := NewTransport(clientKey, nil) + clientTransport, err := NewTransport(clientKey, nil, nil) Expect(err).ToNot(HaveOccurred()) c1, err := clientTransport.Dial(context.Background(), ln1.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) diff --git a/p2p/transport/quic/filtered_conn.go b/p2p/transport/quic/filtered_conn.go new file mode 100644 index 0000000000..dc60bb08e2 --- /dev/null +++ b/p2p/transport/quic/filtered_conn.go @@ -0,0 +1,34 @@ +package libp2pquic + +import ( + "net" + + filter "github.com/libp2p/go-maddr-filter" +) + +type filteredConn struct { + net.PacketConn + + filters *filter.Filters +} + +func newFilteredConn(c net.PacketConn, filters *filter.Filters) net.PacketConn { + return &filteredConn{PacketConn: c, filters: filters} +} + +func (c *filteredConn) ReadFrom(b []byte) (n int, addr net.Addr, rerr error) { + for { + n, addr, rerr = c.PacketConn.ReadFrom(b) + // Short Header packet, see https://tools.ietf.org/html/draft-ietf-quic-invariants-07#section-4.2. + if n < 1 || b[0]&0x80 == 0 { + return + } + maddr, err := toQuicMultiaddr(addr) + if err != nil { + panic(err) + } + if !c.filters.AddrBlocked(maddr) { + return + } + } +} diff --git a/p2p/transport/quic/libp2pquic_suite_test.go b/p2p/transport/quic/libp2pquic_suite_test.go index a2e1df400d..d6ae1510fd 100644 --- a/p2p/transport/quic/libp2pquic_suite_test.go +++ b/p2p/transport/quic/libp2pquic_suite_test.go @@ -5,12 +5,13 @@ import ( mrand "math/rand" "runtime/pprof" "strings" + "testing" "time" + "github.com/lucas-clemente/quic-go" + . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" - - "testing" ) func TestLibp2pQuicTransport(t *testing.T) { @@ -22,8 +23,11 @@ var _ = BeforeSuite(func() { mrand.Seed(GinkgoRandomSeed()) }) -var garbageCollectIntervalOrig time.Duration -var maxUnusedDurationOrig time.Duration +var ( + garbageCollectIntervalOrig time.Duration + maxUnusedDurationOrig time.Duration + origQuicConfig *quic.Config +) func isGarbageCollectorRunning() bool { var b bytes.Buffer @@ -37,10 +41,13 @@ var _ = BeforeEach(func() { maxUnusedDurationOrig = maxUnusedDuration garbageCollectInterval = 50 * time.Millisecond maxUnusedDuration = 0 + origQuicConfig = quicConfig + quicConfig = quicConfig.Clone() }) var _ = AfterEach(func() { Eventually(isGarbageCollectorRunning).Should(BeFalse()) garbageCollectInterval = garbageCollectIntervalOrig maxUnusedDuration = maxUnusedDurationOrig + quicConfig = origQuicConfig }) diff --git a/p2p/transport/quic/listener_test.go b/p2p/transport/quic/listener_test.go index ac7716f353..3388f1ac37 100644 --- a/p2p/transport/quic/listener_test.go +++ b/p2p/transport/quic/listener_test.go @@ -23,7 +23,7 @@ var _ = Describe("Listener", func() { Expect(err).ToNot(HaveOccurred()) key, err := ic.UnmarshalRsaPrivateKey(x509.MarshalPKCS1PrivateKey(rsaKey)) Expect(err).ToNot(HaveOccurred()) - t, err = NewTransport(key, nil) + t, err = NewTransport(key, nil, nil) Expect(err).ToNot(HaveOccurred()) }) diff --git a/p2p/transport/quic/reuse_base.go b/p2p/transport/quic/reuse_base.go index 347a9a6cf7..053b777395 100644 --- a/p2p/transport/quic/reuse_base.go +++ b/p2p/transport/quic/reuse_base.go @@ -4,6 +4,8 @@ import ( "net" "sync" "time" + + filter "github.com/libp2p/go-maddr-filter" ) // Constant. Defined as variables to simplify testing. @@ -20,7 +22,10 @@ type reuseConn struct { unusedSince time.Time } -func newReuseConn(conn net.PacketConn) *reuseConn { +func newReuseConn(conn net.PacketConn, filters *filter.Filters) *reuseConn { + if filters != nil { + conn = newFilteredConn(conn, filters) + } return &reuseConn{PacketConn: conn} } @@ -49,6 +54,8 @@ func (c *reuseConn) ShouldGarbageCollect(now time.Time) bool { type reuseBase struct { mutex sync.Mutex + filters *filter.Filters + garbageCollectorRunning bool unicast map[string] /* IP.String() */ map[int] /* port */ *reuseConn @@ -56,8 +63,9 @@ type reuseBase struct { global map[int]*reuseConn } -func newReuseBase() reuseBase { +func newReuseBase(filters *filter.Filters) reuseBase { return reuseBase{ + filters: filters, unicast: make(map[string]map[int]*reuseConn), global: make(map[int]*reuseConn), } @@ -139,7 +147,7 @@ func (r *reuseBase) dialLocked(network string, raddr *net.UDPAddr, ips []net.IP) if err != nil { return nil, err } - rconn := newReuseConn(conn) + rconn := newReuseConn(conn, r.filters) r.global[conn.LocalAddr().(*net.UDPAddr).Port] = rconn return rconn, nil } @@ -151,7 +159,7 @@ func (r *reuseBase) Listen(network string, laddr *net.UDPAddr) (*reuseConn, erro } localAddr := conn.LocalAddr().(*net.UDPAddr) - rconn := newReuseConn(conn) + rconn := newReuseConn(conn, r.filters) rconn.IncreaseCount() r.mutex.Lock() diff --git a/p2p/transport/quic/reuse_linux_test.go b/p2p/transport/quic/reuse_linux_test.go index 8bc401a0c8..6fe77dbbf6 100644 --- a/p2p/transport/quic/reuse_linux_test.go +++ b/p2p/transport/quic/reuse_linux_test.go @@ -14,7 +14,7 @@ var _ = Describe("Reuse (on Linux)", func() { BeforeEach(func() { var err error - reuse, err = newReuse() + reuse, err = newReuse(nil) Expect(err).ToNot(HaveOccurred()) }) diff --git a/p2p/transport/quic/reuse_not_win.go b/p2p/transport/quic/reuse_not_win.go index fb36b83e89..57097a3098 100644 --- a/p2p/transport/quic/reuse_not_win.go +++ b/p2p/transport/quic/reuse_not_win.go @@ -5,6 +5,8 @@ package libp2pquic import ( "net" + filter "github.com/libp2p/go-maddr-filter" + "github.com/vishvananda/netlink" ) @@ -14,7 +16,7 @@ type reuse struct { handle *netlink.Handle // Only set on Linux. nil on other systems. } -func newReuse() (*reuse, error) { +func newReuse(filters *filter.Filters) (*reuse, error) { handle, err := netlink.NewHandle(SupportedNlFamilies...) if err == netlink.ErrNotImplemented { handle = nil @@ -22,7 +24,7 @@ func newReuse() (*reuse, error) { return nil, err } return &reuse{ - reuseBase: newReuseBase(), + reuseBase: newReuseBase(filters), handle: handle, }, nil } diff --git a/p2p/transport/quic/reuse_test.go b/p2p/transport/quic/reuse_test.go index e7739156e7..6a24d5d575 100644 --- a/p2p/transport/quic/reuse_test.go +++ b/p2p/transport/quic/reuse_test.go @@ -37,7 +37,7 @@ var _ = Describe("Reuse", func() { BeforeEach(func() { var err error - reuse, err = newReuse() + reuse, err = newReuse(nil) Expect(err).ToNot(HaveOccurred()) }) diff --git a/p2p/transport/quic/reuse_win.go b/p2p/transport/quic/reuse_win.go index 0f57c8e0ea..14ea1babfa 100644 --- a/p2p/transport/quic/reuse_win.go +++ b/p2p/transport/quic/reuse_win.go @@ -2,14 +2,18 @@ package libp2pquic -import "net" +import ( + "net" + + filter "github.com/libp2p/go-maddr-filter" +) type reuse struct { reuseBase } -func newReuse() (*reuse, error) { - return &reuse{reuseBase: newReuseBase()}, nil +func newReuse(filters *filter.Filters) (*reuse, error) { + return &reuse{reuseBase: newReuseBase(filters)}, nil } func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 6a61eae61e..1754c795d3 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -11,6 +11,7 @@ import ( "github.com/libp2p/go-libp2p-core/pnet" tpt "github.com/libp2p/go-libp2p-core/transport" p2ptls "github.com/libp2p/go-libp2p-tls" + filter "github.com/libp2p/go-maddr-filter" quic "github.com/lucas-clemente/quic-go" ma "github.com/multiformats/go-multiaddr" mafmt "github.com/multiformats/go-multiaddr-fmt" @@ -36,12 +37,12 @@ type connManager struct { reuseUDP6 *reuse } -func newConnManager() (*connManager, error) { - reuseUDP4, err := newReuse() +func newConnManager(filters *filter.Filters) (*connManager, error) { + reuseUDP4, err := newReuse(filters) if err != nil { return nil, err } - reuseUDP6, err := newReuse() + reuseUDP6, err := newReuse(filters) if err != nil { return nil, err } @@ -89,7 +90,7 @@ type transport struct { var _ tpt.Transport = &transport{} // NewTransport creates a new QUIC transport -func NewTransport(key ic.PrivKey, psk pnet.PSK) (tpt.Transport, error) { +func NewTransport(key ic.PrivKey, psk pnet.PSK, filters *filter.Filters) (tpt.Transport, error) { if len(psk) > 0 { log.Error("QUIC doesn't support private networks yet.") return nil, errors.New("QUIC doesn't support private networks yet") @@ -102,7 +103,7 @@ func NewTransport(key ic.PrivKey, psk pnet.PSK) (tpt.Transport, error) { if err != nil { return nil, err } - connManager, err := newConnManager() + connManager, err := newConnManager(filters) if err != nil { return nil, err } From bbd82c07899d57911fe152e47e80f9bc850ee7a0 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 16 Mar 2020 14:22:48 +0700 Subject: [PATCH 1730/3965] use the resolved address for RemoteMultiaddr() --- p2p/transport/quic/transport.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 6a61eae61e..5e3a8b5522 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -121,16 +121,16 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - udpAddr, err := net.ResolveUDPAddr(network, host) + addr, err := net.ResolveUDPAddr(network, host) if err != nil { return nil, err } - addr, err := fromQuicMultiaddr(raddr) + remoteMultiaddr, err := toQuicMultiaddr(addr) if err != nil { return nil, err } tlsConf, keyCh := t.identity.ConfigForPeer(p) - pconn, err := t.connManager.Dial(network, udpAddr) + pconn, err := t.connManager.Dial(network, addr) if err != nil { return nil, err } @@ -167,7 +167,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp localMultiaddr: localMultiaddr, remotePubKey: remotePubKey, remotePeerID: p, - remoteMultiaddr: raddr, + remoteMultiaddr: remoteMultiaddr, }, nil } From 8008b5d58f269c5994097e6c6543fe40efbacf65 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Mon, 16 Mar 2020 16:11:23 -0700 Subject: [PATCH 1731/3965] update autorelay to remove dependence on autonat instead, learns nat status through event bus --- go.mod | 8 ++----- go.sum | 33 ++++++++++++--------------- p2p/host/relay/autorelay.go | 7 ------ p2p/host/relay/autorelay_test.go | 39 ++++---------------------------- 4 files changed, 22 insertions(+), 65 deletions(-) diff --git a/go.mod b/go.mod index eb80cd3aa8..9f5ae34a26 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,6 @@ require ( github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 - github.com/libp2p/go-libp2p-autonat v0.1.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 github.com/libp2p/go-libp2p-core v0.5.0 @@ -29,15 +28,12 @@ require ( github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 github.com/libp2p/go-ws-transport v0.2.0 - github.com/miekg/dns v1.1.12 // indirect + github.com/miekg/dns v1.1.28 // indirect github.com/multiformats/go-multiaddr v0.2.1 github.com/multiformats/go-multiaddr-dns v0.2.0 - github.com/multiformats/go-multiaddr-net v0.1.2 + github.com/multiformats/go-multiaddr-net v0.1.3 github.com/multiformats/go-multistream v0.1.1 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 - golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d // indirect - golang.org/x/sync v0.0.0-20190423024810-112230192c58 // indirect - golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae // indirect ) go 1.12 diff --git a/go.sum b/go.sum index 4e37d730ed..89505f1690 100644 --- a/go.sum +++ b/go.sum @@ -70,6 +70,7 @@ github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmv github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= @@ -87,8 +88,6 @@ github.com/ipfs/go-cid v0.0.4/go.mod h1:4LLaPOQwmk5z9LBgQnpkivrx8BJjUyGwTXCd5Xfj github.com/ipfs/go-cid v0.0.5 h1:o0Ix8e/ql7Zb5UVUJEUfjsWCIY8t48++9lR8qi6oiJU= github.com/ipfs/go-cid v0.0.5/go.mod h1:plgt+Y5MnOey4vO4UlUazGqdbEXuFYitED67FexhXog= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= -github.com/ipfs/go-datastore v0.1.0/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= -github.com/ipfs/go-datastore v0.1.1/go.mod h1:w38XXW9kVFNp57Zj5knbKWM2T+KOZCGDRVNdgPHtbHw= github.com/ipfs/go-datastore v0.4.0/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= github.com/ipfs/go-datastore v0.4.1/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= github.com/ipfs/go-datastore v0.4.4/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= @@ -96,10 +95,8 @@ github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8= github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaHzfGTzuE3s= -github.com/ipfs/go-ds-badger v0.0.7/go.mod h1:qt0/fWzZDoPW6jpQeqUjR5kBfhDNB65jd9YlmAvpQBk= github.com/ipfs/go-ds-badger v0.2.1/go.mod h1:Tx7l3aTph3FMFrRS838dcSJh+jjA7cX9DrGVwx/NOwE= github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc= -github.com/ipfs/go-ds-leveldb v0.1.0/go.mod h1:hqAW8y4bwX5LWcCtku2rFNX3vjDZCy5LZCg+cSZvYb8= github.com/ipfs/go-ds-leveldb v0.4.1/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50= @@ -151,8 +148,6 @@ github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atq github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-flow-metrics v0.0.3 h1:8tAs/hSdNvUiLgtlSy3mxwxWP4I9y/jlkPFT7epKdeM= github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= -github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= -github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= @@ -191,8 +186,6 @@ github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMg github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HBb6u0MOvdJ7YIsmcwtY= github.com/libp2p/go-libp2p-peerstore v0.1.3 h1:wMgajt1uM2tMiqf4M+4qWKVyyFc8SfA+84VV9glZq1M= github.com/libp2p/go-libp2p-peerstore v0.1.3/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= -github.com/libp2p/go-libp2p-peerstore v0.1.4 h1:d23fvq5oYMJ/lkkbO4oTwBp/JP+I/1m5gZJobNXCE/k= -github.com/libp2p/go-libp2p-peerstore v0.1.4/go.mod h1:+4BDbDiiKf4PzpANZDAT+knVdLxvqh7hXOujessqdzs= github.com/libp2p/go-libp2p-peerstore v0.2.0 h1:XcgJhI8WyUOCbHyRLNEX5542YNj8hnLSJ2G1InRjDhk= github.com/libp2p/go-libp2p-peerstore v0.2.0/go.mod h1:N2l3eVIeAitSg3Pi2ipSrJYnqhVnMNQZo9nkSCuAbnQ= github.com/libp2p/go-libp2p-pnet v0.2.0 h1:J6htxttBipJujEjz1y0a5+eYoiPcFHhSYHH6na5f0/k= @@ -219,8 +212,6 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07q github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 h1:5EhPgQhXZNyfL22ERZTUoVp9UVVbNowWNVtELQaKCHk= github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSoMuyojOdjx73Q47cYD7s5+gZOlns= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= -github.com/libp2p/go-libp2p-yamux v0.2.1 h1:Q3XYNiKCC2vIxrvUJL+Jg1kiyeEaIDNKLjgEjo3VQdI= -github.com/libp2p/go-libp2p-yamux v0.2.1/go.mod h1:1FBXiHDk1VyRM1C0aez2bCfHQ4vMZKkAQzZbkSQt5fI= github.com/libp2p/go-libp2p-yamux v0.2.2 h1:eGvbqWqWY9S5lrpe2gA0UCOLCdzCgYSAR3vo/xCsNQg= github.com/libp2p/go-libp2p-yamux v0.2.2/go.mod h1:lIohaR0pT6mOt0AZ0L2dFze9hds9Req3OfS+B+dv4qw= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= @@ -257,8 +248,6 @@ github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfj github.com/libp2p/go-ws-transport v0.2.0 h1:MJCw2OrPA9+76YNRvdo1wMnSOxb9Bivj6sVFY1Xrj6w= github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzlHoKzu0yY9p/klM= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= -github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= -github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.0 h1:FsYzT16Wq2XqUGJsBbOxoz9g+dFklvNi7jN6YFPfl7U= github.com/libp2p/go-yamux v1.3.0/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= @@ -271,8 +260,8 @@ github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNx github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE= -github.com/miekg/dns v1.1.12 h1:WMhc1ik4LNkTg8U9l3hI1LvxKmIL+f1+WV/SZtCbDDA= -github.com/miekg/dns v1.1.12/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= +github.com/miekg/dns v1.1.28 h1:gQhy5bsJa8zTlVI8lywCTZp1lguor+xevFoYlzeCTQY= +github.com/miekg/dns v1.1.28/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ= github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= @@ -320,6 +309,8 @@ github.com/multiformats/go-multiaddr-net v0.1.1 h1:jFFKUuXTXv+3ARyHZi3XUqQO+YWMK github.com/multiformats/go-multiaddr-net v0.1.1/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= github.com/multiformats/go-multiaddr-net v0.1.2 h1:P7zcBH9FRETdPkDrylcXVjQLQ2t1JQtNItZULWNWgeg= github.com/multiformats/go-multiaddr-net v0.1.2/go.mod h1:QsWt3XK/3hwvNxZJp92iMQKME1qHfpYmyIjFVsSOY6Y= +github.com/multiformats/go-multiaddr-net v0.1.3 h1:q/IYAvoPKuRzGeERn3uacWgm0LIWkLZBAvO5DxSzq3g= +github.com/multiformats/go-multiaddr-net v0.1.3/go.mod h1:ilNnaM9HbmVFqsb/qcNysjCu4PVONlrBZpHIrw/qQuA= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= @@ -359,6 +350,7 @@ github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFSt github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= @@ -430,12 +422,11 @@ golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 h1:IcSOAf4PyMp3U3XbIEj1/x golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d h1:1ZiEyfaQIg3Qh0EoqpwAakHVhecoE5wlSg5GjnafJGw= -golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -446,6 +437,8 @@ golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190923162816-aa69164e4478 h1:l5EDrHhldLYb3ZRHDUhXF7Om7MvYXnkV9/iQNo1lX6g= +golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -465,11 +458,11 @@ golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e h1:N7DeIrjYszNmSW409R3frPPwglRwMkXSBzwVbkOjLLA= golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5 h1:LfCXLvNmTYH9kEmVgqbnsWfruoXZIrh4YBgqVHtDvw0= golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae h1:/WDfKMnPU+m5M4xB+6x4kaepxRw6jWvR5iDRdvjHgy8= -golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= @@ -482,8 +475,11 @@ golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= @@ -495,6 +491,7 @@ google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiq gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 09dc50d950..e5187109d1 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -12,7 +12,6 @@ import ( "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/routing" - autonat "github.com/libp2p/go-libp2p-autonat" circuit "github.com/libp2p/go-libp2p-circuit" discovery "github.com/libp2p/go-libp2p-discovery" basic "github.com/libp2p/go-libp2p/p2p/host/basic" @@ -43,7 +42,6 @@ type AutoRelay struct { host *basic.BasicHost discover discovery.Discoverer router routing.PeerRouting - autonat autonat.AutoNAT addrsF basic.AddrsFactory static []peer.AddrInfo @@ -69,17 +67,12 @@ func NewAutoRelay(ctx context.Context, bhost *basic.BasicHost, discover discover disconnect: make(chan struct{}, 1), status: network.ReachabilityUnknown, } - ar.autonat = autonat.NewAutoNAT(ctx, bhost, ar.baseAddrs) bhost.AddrsFactory = ar.hostAddrs bhost.Network().Notify(ar) go ar.background(ctx) return ar } -func (ar *AutoRelay) baseAddrs() []ma.Multiaddr { - return ar.addrsF(ar.host.AllAddrs()) -} - func (ar *AutoRelay) hostAddrs(addrs []ma.Multiaddr) []ma.Multiaddr { return ar.relayAddrs(ar.addrsF(addrs)) } diff --git a/p2p/host/relay/autorelay_test.go b/p2p/host/relay/autorelay_test.go index 9abd9899ec..83b910325c 100644 --- a/p2p/host/relay/autorelay_test.go +++ b/p2p/host/relay/autorelay_test.go @@ -11,10 +11,7 @@ import ( libp2p "github.com/libp2p/go-libp2p" relay "github.com/libp2p/go-libp2p/p2p/host/relay" - ggio "github.com/gogo/protobuf/io" cid "github.com/ipfs/go-cid" - autonat "github.com/libp2p/go-libp2p-autonat" - autonatpb "github.com/libp2p/go-libp2p-autonat/pb" circuit "github.com/libp2p/go-libp2p-circuit" "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" @@ -27,8 +24,6 @@ import ( // test specific parameters func init() { - autonat.AutoNATIdentifyDelay = 1 * time.Second - autonat.AutoNATBootDelay = 2 * time.Second relay.BootDelay = 1 * time.Second relay.AdvertiseBootDelay = 100 * time.Millisecond } @@ -107,33 +102,6 @@ func (m *mockRouting) FindProvidersAsync(ctx context.Context, cid cid.Cid, limit return ch } -// mock autonat -func makeAutoNATServicePrivate(ctx context.Context, t *testing.T) host.Host { - h, err := libp2p.New(ctx) - if err != nil { - t.Fatal(err) - } - h.SetStreamHandler(autonat.AutoNATProto, sayAutoNATPrivate) - return h -} - -func sayAutoNATPrivate(s network.Stream) { - defer s.Close() - w := ggio.NewDelimitedWriter(s) - res := autonatpb.Message{ - Type: autonatpb.Message_DIAL_RESPONSE.Enum(), - DialResponse: newDialResponseError(autonatpb.Message_E_DIAL_ERROR, "no dialable addresses"), - } - w.WriteMsg(&res) -} - -func newDialResponseError(status autonatpb.Message_ResponseStatus, text string) *autonatpb.Message_DialResponse { - dr := new(autonatpb.Message_DialResponse) - dr.Status = status.Enum() - dr.StatusText = &text - return dr -} - // connector func connect(t *testing.T, a, b host.Host) { pinfo := peer.AddrInfo{ID: a.ID(), Addrs: a.Addrs()} @@ -160,8 +128,11 @@ func TestAutoRelay(t *testing.T) { return mr, nil } - h1 := makeAutoNATServicePrivate(ctx, t) - _, err := libp2p.New(ctx, libp2p.EnableRelay(circuit.OptHop), libp2p.EnableAutoRelay(), libp2p.Routing(makeRouting)) + h1, err := libp2p.New(ctx, libp2p.EnableRelay()) + if err != nil { + t.Fatal(err) + } + _, err = libp2p.New(ctx, libp2p.EnableRelay(circuit.OptHop), libp2p.EnableAutoRelay(), libp2p.Routing(makeRouting)) if err != nil { t.Fatal(err) } From 1921bc20ae37549563384f8a6a3138a1a129695f Mon Sep 17 00:00:00 2001 From: Will Scott Date: Mon, 16 Mar 2020 17:53:26 -0700 Subject: [PATCH 1732/3965] autonat startup within libp2p config --- config/config.go | 9 +++++++++ go.mod | 1 + go.sum | 2 ++ 3 files changed, 12 insertions(+) diff --git a/config/config.go b/config/config.go index fd36db753c..33a2ba4b32 100644 --- a/config/config.go +++ b/config/config.go @@ -18,6 +18,7 @@ import ( relay "github.com/libp2p/go-libp2p/p2p/host/relay" routed "github.com/libp2p/go-libp2p/p2p/host/routed" + autonat "github.com/libp2p/go-libp2p-autonat" circuit "github.com/libp2p/go-libp2p-circuit" discovery "github.com/libp2p/go-libp2p-discovery" swarm "github.com/libp2p/go-libp2p-swarm" @@ -76,6 +77,7 @@ type Config struct { Routing RoutingC EnableAutoRelay bool + AutoNATOpts []autonat.Option StaticRelays []peer.AddrInfo } @@ -201,6 +203,13 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } } + if cfg.EnableAutoRelay || cfg.AutoNATOpts != nil { + if _, err = autonat.New(ctx, h, cfg.AutoNATOpts...); err != nil { + h.Close() + return nil, fmt.Errorf("cannot enable autorelay; autonat failed to start: %v", err) + } + } + if cfg.EnableAutoRelay { if !cfg.Relay { h.Close() diff --git a/go.mod b/go.mod index 9f5ae34a26..2b3cd8b793 100644 --- a/go.mod +++ b/go.mod @@ -10,6 +10,7 @@ require ( github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 + github.com/libp2p/go-libp2p-autonat v0.1.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 github.com/libp2p/go-libp2p-core v0.5.0 diff --git a/go.sum b/go.sum index 89505f1690..9ebc2f9aa8 100644 --- a/go.sum +++ b/go.sum @@ -148,6 +148,8 @@ github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atq github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-flow-metrics v0.0.3 h1:8tAs/hSdNvUiLgtlSy3mxwxWP4I9y/jlkPFT7epKdeM= github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= +github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= +github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= From 71e2d4ed5ba1f0b8247e668a17995bb4d2b42ddb Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Tue, 17 Mar 2020 11:18:56 +0000 Subject: [PATCH 1733/3965] build(deps): bump github.com/libp2p/go-libp2p-yamux from 0.2.2 to 0.2.4 Bumps [github.com/libp2p/go-libp2p-yamux](https://github.com/libp2p/go-libp2p-yamux) from 0.2.2 to 0.2.4. - [Release notes](https://github.com/libp2p/go-libp2p-yamux/releases) - [Commits](https://github.com/libp2p/go-libp2p-yamux/compare/v0.2.2...v0.2.4) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index de901d135b..423fa4303a 100644 --- a/go.mod +++ b/go.mod @@ -24,7 +24,7 @@ require ( github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 - github.com/libp2p/go-libp2p-yamux v0.2.2 + github.com/libp2p/go-libp2p-yamux v0.2.4 github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 diff --git a/go.sum b/go.sum index 51dbd658e1..a47f524de7 100644 --- a/go.sum +++ b/go.sum @@ -223,6 +223,8 @@ github.com/libp2p/go-libp2p-yamux v0.2.1 h1:Q3XYNiKCC2vIxrvUJL+Jg1kiyeEaIDNKLjgE github.com/libp2p/go-libp2p-yamux v0.2.1/go.mod h1:1FBXiHDk1VyRM1C0aez2bCfHQ4vMZKkAQzZbkSQt5fI= github.com/libp2p/go-libp2p-yamux v0.2.2 h1:eGvbqWqWY9S5lrpe2gA0UCOLCdzCgYSAR3vo/xCsNQg= github.com/libp2p/go-libp2p-yamux v0.2.2/go.mod h1:lIohaR0pT6mOt0AZ0L2dFze9hds9Req3OfS+B+dv4qw= +github.com/libp2p/go-libp2p-yamux v0.2.4 h1:SoIR1z7kjjncCPliuchPptd4n/xTxePzWtmR2vIQLwM= +github.com/libp2p/go-libp2p-yamux v0.2.4/go.mod h1:Qq6yW6EfI9vNeL4J8H1l8yCcBoZnJiljJGsbZgfC/08= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.5 h1:CW3AgbMO6vUvT4kf87y4N+0P8KUl2aqLYhrGyDUbLSg= github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= @@ -261,6 +263,8 @@ github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.0 h1:FsYzT16Wq2XqUGJsBbOxoz9g+dFklvNi7jN6YFPfl7U= github.com/libp2p/go-yamux v1.3.0/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-yamux v1.3.2 h1:9ABGfqAAQB5hlBHWk0bAbGnS4dDaAJLSF0C7TPYia+0= +github.com/libp2p/go-yamux v1.3.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= From 8037ce2dcab43340c1c4762590b2a7bc3ee0e397 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 17 Mar 2020 11:14:21 -0700 Subject: [PATCH 1734/3965] generation of dialer host when needed --- config/config.go | 65 ++++++++++++++++++++++++++++++++---------------- go.mod | 2 +- go.sum | 8 ++++++ options.go | 9 +++++++ 4 files changed, 61 insertions(+), 23 deletions(-) diff --git a/config/config.go b/config/config.go index 33a2ba4b32..791c8cf2c0 100644 --- a/config/config.go +++ b/config/config.go @@ -13,6 +13,7 @@ import ( "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/pnet" "github.com/libp2p/go-libp2p-core/routing" + "github.com/libp2p/go-libp2p-peerstore/pstoremem" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" relay "github.com/libp2p/go-libp2p/p2p/host/relay" @@ -77,14 +78,11 @@ type Config struct { Routing RoutingC EnableAutoRelay bool - AutoNATOpts []autonat.Option + EnableAutoNAT bool StaticRelays []peer.AddrInfo } -// NewNode constructs a new libp2p Host from the Config. -// -// This function consumes the config. Do not reuse it (really!). -func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { +func (cfg *Config) newHost(ctx context.Context, store peerstore.Peerstore) (*bhost.BasicHost, error) { // Check this early. Prevents us from even *starting* without verifying this. if pnet.ForcePrivateNetwork && len(cfg.PSK) == 0 { log.Error("tried to create a libp2p node with no Private" + @@ -105,19 +103,15 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return nil, err } - if cfg.Peerstore == nil { - return nil, fmt.Errorf("no peerstore specified") - } - - if err := cfg.Peerstore.AddPrivKey(pid, cfg.PeerKey); err != nil { + if err := store.AddPrivKey(pid, cfg.PeerKey); err != nil { return nil, err } - if err := cfg.Peerstore.AddPubKey(pid, cfg.PeerKey.GetPublic()); err != nil { + if err := store.AddPubKey(pid, cfg.PeerKey.GetPublic()); err != nil { return nil, err } // TODO: Make the swarm implementation configurable. - swrm := swarm.NewSwarm(ctx, pid, cfg.Peerstore, cfg.Reporter) + swrm := swarm.NewSwarm(ctx, pid, store, cfg.Reporter) if cfg.Filters != nil { swrm.Filters = cfg.Filters } @@ -185,6 +179,21 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return nil, err } } + return h, nil +} + +// NewNode constructs a new libp2p Host from the Config. +// +// This function consumes the config. Do not reuse it (really!). +func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { + if cfg.Peerstore == nil { + return nil, fmt.Errorf("no peerstore specified") + } + + h, err := cfg.newHost(ctx, cfg.Peerstore) + if err != nil { + return nil, err + } // TODO: This method succeeds if listening on one address succeeds. We // should probably fail if listening on *any* addr fails. @@ -203,20 +212,13 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } } - if cfg.EnableAutoRelay || cfg.AutoNATOpts != nil { - if _, err = autonat.New(ctx, h, cfg.AutoNATOpts...); err != nil { - h.Close() - return nil, fmt.Errorf("cannot enable autorelay; autonat failed to start: %v", err) - } - } - + hop := false if cfg.EnableAutoRelay { if !cfg.Relay { h.Close() return nil, fmt.Errorf("cannot enable autorelay; relay is not enabled") } - hop := false for _, opt := range cfg.RelayOpts { if opt == circuit.OptHop { hop = true @@ -225,7 +227,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } if !hop && len(cfg.StaticRelays) > 0 { - _ = relay.NewAutoRelay(swrm.Context(), h, nil, router, cfg.StaticRelays) + _ = relay.NewAutoRelay(ctx, h, nil, router, cfg.StaticRelays) } else { if router == nil { h.Close() @@ -244,8 +246,27 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { // advertise ourselves relay.Advertise(ctx, discovery) } else { - _ = relay.NewAutoRelay(swrm.Context(), h, discovery, router, cfg.StaticRelays) + _ = relay.NewAutoRelay(ctx, h, discovery, router, cfg.StaticRelays) + } + } + } + + if cfg.EnableAutoRelay || cfg.EnableAutoNAT { + var autonatOpts []autonat.Option + if hop { + dialerStore := pstoremem.NewPeerstore() + dialerHost, err := cfg.newHost(ctx, dialerStore) + if err != nil { + h.Close() + return nil, err } + + autonatOpts = append(autonatOpts, autonat.EnableService(dialerHost.Network(), false)) + } + + if _, err = autonat.New(ctx, h, autonatOpts...); err != nil { + h.Close() + return nil, fmt.Errorf("cannot enable autorelay; autonat failed to start: %v", err) } } diff --git a/go.mod b/go.mod index 2b3cd8b793..1b0b51dc7b 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 - github.com/libp2p/go-libp2p-autonat v0.1.1 + github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200316221154-f1cd6ba0fd67 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 github.com/libp2p/go-libp2p-core v0.5.0 diff --git a/go.sum b/go.sum index 9ebc2f9aa8..e3ef4b77f0 100644 --- a/go.sum +++ b/go.sum @@ -148,8 +148,11 @@ github.com/libp2p/go-flow-metrics v0.0.1 h1:0gxuFd2GuK7IIP5pKljLwps6TvcuYgvG7Atq github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-flow-metrics v0.0.3 h1:8tAs/hSdNvUiLgtlSy3mxwxWP4I9y/jlkPFT7epKdeM= github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= +github.com/libp2p/go-libp2p v0.6.0/go.mod h1:mfKWI7Soz3ABX+XEBR61lGbg+ewyMtJHVt043oWeqwg= github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= +github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200316221154-f1cd6ba0fd67 h1:3UM/mimkSSSQiosUHiauy7w/7JwAxMgEY4StoeMHR4A= +github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200316221154-f1cd6ba0fd67/go.mod h1:yoF5B6yi+Wjxci55xdLlJo7gfyHwG80df4B9hiO2A88= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= @@ -262,6 +265,7 @@ github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNx github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE= +github.com/miekg/dns v1.1.12/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/miekg/dns v1.1.28 h1:gQhy5bsJa8zTlVI8lywCTZp1lguor+xevFoYlzeCTQY= github.com/miekg/dns v1.1.28/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g= @@ -424,6 +428,8 @@ golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 h1:IcSOAf4PyMp3U3XbIEj1/x golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d h1:1ZiEyfaQIg3Qh0EoqpwAakHVhecoE5wlSg5GjnafJGw= +golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= @@ -465,6 +471,8 @@ golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e h1:N7DeIrjYszNmSW409R3frPPwg golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5 h1:LfCXLvNmTYH9kEmVgqbnsWfruoXZIrh4YBgqVHtDvw0= golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae h1:/WDfKMnPU+m5M4xB+6x4kaepxRw6jWvR5iDRdvjHgy8= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= diff --git a/options.go b/options.go index d63afbba7f..12b34ad6e1 100644 --- a/options.go +++ b/options.go @@ -278,6 +278,15 @@ func DefaultStaticRelays() Option { } } +// EnableAutoNAT configures libp2p to monitor its NAT status and track when +// the local host is externally reachable. +func EnableAutoNAT() Option { + return func(cfg *Config) error { + cfg.EnableAutoNAT = true + return nil + } +} + // FilterAddresses configures libp2p to never dial nor accept connections from // the given addresses. FilterAddresses should be used for cases where the // addresses you want to deny are known ahead of time. From 69b16483fd3c0a88fb3c4b53e1cae92738db861c Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 17 Mar 2020 12:27:08 -0700 Subject: [PATCH 1735/3965] properly filter nat addresses --- config/config.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/config/config.go b/config/config.go index 791c8cf2c0..0b840358ab 100644 --- a/config/config.go +++ b/config/config.go @@ -213,6 +213,9 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } hop := false + // Note: h.AddrsFactory may be changed by AutoRelay, but non-relay version is + // used by AutoNAT below. + addrF := h.AddrsFactory if cfg.EnableAutoRelay { if !cfg.Relay { h.Close() @@ -252,7 +255,9 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } if cfg.EnableAutoRelay || cfg.EnableAutoNAT { - var autonatOpts []autonat.Option + autonatOpts := []autonat.Option{autonat.AddressGuesser(func() []ma.Multiaddr { + return addrF(h.AllAddrs()) + })} if hop { dialerStore := pstoremem.NewPeerstore() dialerHost, err := cfg.newHost(ctx, dialerStore) From dd87382dd43670bdf1f0c1dfb8e90fb94ff4e736 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 17 Mar 2020 13:10:52 -0700 Subject: [PATCH 1736/3965] update to latest autonat --- config/config.go | 2 +- go.mod | 2 +- go.sum | 2 ++ 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/config/config.go b/config/config.go index 0b840358ab..380086db47 100644 --- a/config/config.go +++ b/config/config.go @@ -255,7 +255,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } if cfg.EnableAutoRelay || cfg.EnableAutoNAT { - autonatOpts := []autonat.Option{autonat.AddressGuesser(func() []ma.Multiaddr { + autonatOpts := []autonat.Option{autonat.UsingAddresses(func() []ma.Multiaddr { return addrF(h.AllAddrs()) })} if hop { diff --git a/go.mod b/go.mod index 1b0b51dc7b..a8ed941a9f 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 - github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200316221154-f1cd6ba0fd67 + github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200317183318-4b2cc5830d44 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 github.com/libp2p/go-libp2p-core v0.5.0 diff --git a/go.sum b/go.sum index e3ef4b77f0..f76c2aa36f 100644 --- a/go.sum +++ b/go.sum @@ -153,6 +153,8 @@ github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Z github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200316221154-f1cd6ba0fd67 h1:3UM/mimkSSSQiosUHiauy7w/7JwAxMgEY4StoeMHR4A= github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200316221154-f1cd6ba0fd67/go.mod h1:yoF5B6yi+Wjxci55xdLlJo7gfyHwG80df4B9hiO2A88= +github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200317183318-4b2cc5830d44 h1:xfmItLrKVdinHJJouT4ocZFWK+Y0WaEONku6aYN92+s= +github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200317183318-4b2cc5830d44/go.mod h1:yoF5B6yi+Wjxci55xdLlJo7gfyHwG80df4B9hiO2A88= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= From e1841d877ef1f03207f8d3a5736fe55a7ec52f4a Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 12 Mar 2020 16:57:21 -0700 Subject: [PATCH 1737/3965] Integrate service into autonat --- p2p/host/autonat/autonat.go | 35 +- p2p/host/autonat/autonat_test.go | 14 - p2p/host/autonat/client.go | 14 +- p2p/host/autonat/interface.go | 4 +- p2p/host/autonat/options.go | 78 ++++- p2p/host/autonat/proto.go | 5 +- p2p/host/autonat/svc.go | 134 +++----- p2p/host/autonat/svc_test.go | 171 ++++------ p2p/host/autonat/test/autonat_test.go | 60 ++++ p2p/host/autonat/test/go.mod | 23 ++ p2p/host/autonat/test/go.sum | 471 ++++++++++++++++++++++++++ 11 files changed, 779 insertions(+), 230 deletions(-) create mode 100644 p2p/host/autonat/test/autonat_test.go create mode 100644 p2p/host/autonat/test/go.mod create mode 100644 p2p/host/autonat/test/go.sum diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index a159f9b696..6156d0cbc5 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -7,16 +7,19 @@ import ( "sync/atomic" "time" + "github.com/libp2p/go-eventbus" "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" - "github.com/libp2p/go-eventbus" + logging "github.com/ipfs/go-log" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) +var log = logging.Logger("autonat") + // AmbientAutoNAT is the implementation of ambient NAT autodiscovery type AmbientAutoNAT struct { ctx context.Context @@ -38,6 +41,8 @@ type AmbientAutoNAT struct { lastProbe time.Time subAddrUpdated event.Subscription + service *autoNATService + serviceCancel context.CancelFunc emitReachabilityChanged event.Emitter } @@ -50,12 +55,13 @@ type autoNATResult struct { // New creates a new NAT autodiscovery system attached to a host func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { conf := new(config) + conf.host = h if err := defaults(conf); err != nil { return nil, err } - if conf.getAddressFunc == nil { - conf.getAddressFunc = h.Addrs + if conf.addressFunc == nil { + conf.addressFunc = h.Addrs } for _, o := range options { @@ -83,6 +89,14 @@ func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { h.Network().Notify(as) go as.background() + if conf.dialer != nil { + var err error + as.service, err = newAutoNATService(ctx, conf) + if err != nil { + return nil, err + } + } + return as, nil } @@ -211,6 +225,11 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { if currentStatus.Reachability != network.ReachabilityPublic { // we are flipping our NATStatus, so confidence drops to 0 as.confidence = 0 + if as.service != nil && !as.config.forceServer { + ctx, cancel := context.WithCancel(as.ctx) + go as.service.Enable(ctx) + as.serviceCancel = cancel + } changed = true } else if as.confidence < 3 { as.confidence++ @@ -236,6 +255,10 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { // we are flipping our NATStatus, so confidence drops to 0 as.confidence = 0 as.status.Store(observation) + if as.serviceCancel != nil { + as.serviceCancel() + as.serviceCancel = nil + } as.emitStatus() } } else if as.confidence < 3 { @@ -252,13 +275,17 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { log.Debugf("NAT status is unknown") as.status.Store(autoNATResult{network.ReachabilityUnknown, nil}) if currentStatus.Reachability != network.ReachabilityUnknown { + if as.serviceCancel != nil { + as.serviceCancel() + as.serviceCancel = nil + } as.emitStatus() } } } func (as *AmbientAutoNAT) probe(pi *peer.AddrInfo) { - cli := NewAutoNATClient(as.host, as.config.getAddressFunc) + cli := NewAutoNATClient(as.host, as.config.addressFunc) ctx, cancel := context.WithTimeout(as.ctx, as.config.requestTimeout) defer cancel() diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 0304cf324f..c6bca8dd34 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -50,20 +50,6 @@ func sayAutoNATPublic(s network.Stream) { w.WriteMsg(&res) } -func newDialResponseOK(addr ma.Multiaddr) *pb.Message_DialResponse { - dr := new(pb.Message_DialResponse) - dr.Status = pb.Message_OK.Enum() - dr.Addr = addr.Bytes() - return dr -} - -func newDialResponseError(status pb.Message_ResponseStatus, text string) *pb.Message_DialResponse { - dr := new(pb.Message_DialResponse) - dr.Status = status.Enum() - dr.StatusText = &text - return dr -} - func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, AutoNAT) { h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) h.Peerstore().AddAddrs(ash.ID(), ash.Addrs(), time.Minute) diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index e2acab178e..b86f17a874 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -21,17 +21,17 @@ type AutoNATError struct { } // NewAutoNATClient creates a fresh instance of an AutoNATClient -// If getAddrs is nil, h.Addrs will be used -func NewAutoNATClient(h host.Host, getAddrs GetAddrs) Client { - if getAddrs == nil { - getAddrs = h.Addrs +// If addrFunc is nil, h.Addrs will be used +func NewAutoNATClient(h host.Host, addrFunc AddrFunc) Client { + if addrFunc == nil { + addrFunc = h.Addrs } - return &client{h: h, getAddrs: getAddrs} + return &client{h: h, addrFunc: addrFunc} } type client struct { h host.Host - getAddrs GetAddrs + addrFunc AddrFunc } func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) { @@ -46,7 +46,7 @@ func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) r := ggio.NewDelimitedReader(s, network.MessageSizeMax) w := ggio.NewDelimitedWriter(s) - req := newDialMessage(peer.AddrInfo{ID: c.h.ID(), Addrs: c.getAddrs()}) + req := newDialMessage(peer.AddrInfo{ID: c.h.ID(), Addrs: c.addrFunc()}) err = w.WriteMsg(req) if err != nil { s.Reset() diff --git a/p2p/host/autonat/interface.go b/p2p/host/autonat/interface.go index af010b98e6..0d84f96284 100644 --- a/p2p/host/autonat/interface.go +++ b/p2p/host/autonat/interface.go @@ -25,8 +25,8 @@ type Client interface { DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) } -// GetAddrs is a function returning the candidate addresses for the local host. -type GetAddrs func() []ma.Multiaddr +// AddrFunc is a function returning the candidate addresses for the local host. +type AddrFunc func() []ma.Multiaddr // Option is an Autonat option for configuration type Option func(*config) error diff --git a/p2p/host/autonat/options.go b/p2p/host/autonat/options.go index 6987d3aba6..5ed6ee7578 100644 --- a/p2p/host/autonat/options.go +++ b/p2p/host/autonat/options.go @@ -3,15 +3,32 @@ package autonat import ( "errors" "time" + + "github.com/libp2p/go-libp2p-core/host" + "github.com/libp2p/go-libp2p-core/network" ) // config holds configurable options for the autonat subsystem. type config struct { - getAddressFunc GetAddrs + host host.Host + + addressFunc AddrFunc + dialer network.Network + forceServer bool + + // client bootDelay time.Duration retryInterval time.Duration refreshInterval time.Duration requestTimeout time.Duration + + // server + dialTimeout time.Duration + maxPeerAddresses int + throttleGlobalMax int + throttlePeerMax int + throttleResetPeriod time.Duration + throttleResetJitter time.Duration } var defaults = func(c *config) error { @@ -20,19 +37,42 @@ var defaults = func(c *config) error { c.refreshInterval = 15 * time.Minute c.requestTimeout = 30 * time.Second + c.dialTimeout = 15 * time.Second + c.maxPeerAddresses = 16 + c.throttleGlobalMax = 30 + c.throttlePeerMax = 3 + c.throttleResetPeriod = 1 * time.Minute + c.throttleResetJitter = 15 * time.Second return nil } -// WithAddresses allows overriding which Addresses the AutoNAT client beliieves +// EnableService specifies that AutoNAT should be allowed to run a NAT service to help +// other peers determine their own NAT status. The provided Network should not be the +// default network/dialer of the host passed to `New`, as the NAT system will need to +// make parallel connections, and as such will modify both the associated peerstore +// and terminate connections of this dialer. The dialer provided +// should be compatible (TCP/UDP) however with the transports of the libp2p network. +func EnableService(dialer network.Network, forceServer bool) Option { + return func(c *config) error { + if dialer == c.host.Network() || dialer.Peerstore() == c.host.Peerstore() { + return errors.New("dialer should not be that of the host") + } + c.dialer = dialer + c.forceServer = forceServer + return nil + } +} + +// UsingAddresses allows overriding which Addresses the AutoNAT client believes // are "its own". Useful for testing, or for more exotic port-forwarding // scenarios where the host may be listening on different ports than it wants // to externally advertise or verify connectability on. -func WithAddresses(addrFunc GetAddrs) Option { +func UsingAddresses(addrFunc AddrFunc) Option { return func(c *config) error { if addrFunc == nil { return errors.New("invalid address function supplied") } - c.getAddressFunc = addrFunc + c.addressFunc = addrFunc return nil } } @@ -59,3 +99,33 @@ func WithoutStartupDelay() Option { return nil } } + +// WithoutThrottling indicates that this autonat service should not place +// restrictions on how many peers it is willing to help when acting as +// a server. +func WithoutThrottling() Option { + return func(c *config) error { + c.throttleGlobalMax = 0 + return nil + } +} + +// WithThrottling specifies how many peers (`amount`) it is willing to help +// ever `interval` amount of time when acting as a server. +func WithThrottling(amount int, interval time.Duration) Option { + return func(c *config) error { + c.throttleGlobalMax = amount + c.throttleResetPeriod = interval + c.throttleResetJitter = interval / 4 + return nil + } +} + +// WithPeerThrottling specifies a limit for the maximum number of IP checks +// this node will provide to an individual peer in each `interval`. +func WithPeerThrottling(amount int) Option { + return func(c *config) error { + c.throttlePeerMax = amount + return nil + } +} diff --git a/p2p/host/autonat/proto.go b/p2p/host/autonat/proto.go index 4699cd17a1..cf5be81ad9 100644 --- a/p2p/host/autonat/proto.go +++ b/p2p/host/autonat/proto.go @@ -5,14 +5,11 @@ import ( "github.com/libp2p/go-libp2p-core/peer" - logging "github.com/ipfs/go-log" ma "github.com/multiformats/go-multiaddr" ) const AutoNATProto = "/libp2p/autonat/1.0.0" -var log = logging.Logger("autonat") - func newDialMessage(pi peer.AddrInfo) *pb.Message { msg := new(pb.Message) msg.Type = pb.Message_DIAL.Enum() @@ -25,7 +22,7 @@ func newDialMessage(pi peer.AddrInfo) *pb.Message { } return msg -) +} func newDialResponseOK(addr ma.Multiaddr) *pb.Message_DialResponse { dr := new(pb.Message_DialResponse) diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index c492b1bd8d..d55beacc37 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -2,15 +2,13 @@ package autonat import ( "context" + "errors" "math/rand" "net" "sync" "time" - "github.com/libp2p/go-libp2p" - "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/helpers" - "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" @@ -18,75 +16,44 @@ import ( pb "github.com/libp2p/go-libp2p-autonat/pb" ggio "github.com/gogo/protobuf/io" - autonat "github.com/libp2p/go-libp2p-autonat" ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) const P_CIRCUIT = 290 -var ( - // AutoNATServiceDialTimeout defines how long to wait for connection - // attempts before failing. - AutoNATServiceDialTimeout = 15 * time.Second - // AutoNATServiceResetInterval defines how often to reset throttling. - AutoNATServiceResetInterval = 1 * time.Minute - // AutoNATServiceResetJitter defines the amplitude of randomness in throttle - // reset timing. - AutoNATServiceResetJitter = 15 * time.Second - - // AutoNATServiceThrottle defines how many times each ResetInterval a peer - // can ask for its autonat address. - AutoNATServiceThrottle = 3 - // AutoNATGlobalThrottle defines how many total autonat requests this - // service will answer each ResetInterval. - AutoNATGlobalThrottle = 30 - // AutoNATMaxPeerAddresses defines maximum number of addreses the autonat - // service will consider when attempting to connect to the peer. - AutoNATMaxPeerAddresses = 16 -) - // AutoNATService provides NAT autodetection services to other peers -type AutoNATService struct { - ctx context.Context - h host.Host - dialer host.Host +type autoNATService struct { + ctx context.Context + + config *config // rate limiter - mx sync.Mutex - reqs map[peer.ID]int - globalReqMax int - globalReqs int + mx sync.Mutex + reqs map[peer.ID]int + globalReqs int } // NewAutoNATService creates a new AutoNATService instance attached to a host -func NewAutoNATService(ctx context.Context, h host.Host, forceEnabled bool, opts ...libp2p.Option) (*AutoNATService, error) { - opts = append(opts, libp2p.NoListenAddrs) - dialer, err := libp2p.New(ctx, opts...) - if err != nil { - return nil, err +func newAutoNATService(ctx context.Context, c *config) (*autoNATService, error) { + if c.dialer == nil { + return nil, errors.New("Cannot create NAT service without a network") } - as := &AutoNATService{ - ctx: ctx, - h: h, - dialer: dialer, - globalReqMax: AutoNATGlobalThrottle, - reqs: make(map[peer.ID]int), + as := &autoNATService{ + ctx: ctx, + config: c, + reqs: make(map[peer.ID]int), } - if forceEnabled { - as.globalReqMax = 0 - h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) - go as.resetRateLimiter() - } else { - go as.enableWhenPublic() + if c.forceServer { + go as.Enable(ctx) } return as, nil } -func (as *AutoNATService) handleStream(s network.Stream) { +func (as *autoNATService) handleStream(s network.Stream) { defer helpers.FullClose(s) pid := s.Conn().RemotePeer() @@ -124,7 +91,7 @@ func (as *AutoNATService) handleStream(s network.Stream) { } } -func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { +func (as *autoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Message_PeerInfo) *pb.Message_DialResponse { if mpi == nil { return newDialResponseError(pb.Message_E_BAD_REQUEST, "missing peer info") } @@ -141,7 +108,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me } } - addrs := make([]ma.Multiaddr, 0, AutoNATMaxPeerAddresses) + addrs := make([]ma.Multiaddr, 0, as.config.maxPeerAddresses) seen := make(map[string]struct{}) // add observed addr to the list of addresses to dial @@ -176,7 +143,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me addrs = append(addrs, addr) seen[str] = struct{}{} - if len(addrs) >= AutoNATMaxPeerAddresses { + if len(addrs) >= as.config.maxPeerAddresses { break } } @@ -188,7 +155,7 @@ func (as *AutoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me return as.doDial(peer.AddrInfo{ID: p, Addrs: addrs}) } -func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { +func (as *autoNATService) skipDial(addr ma.Multiaddr) bool { // skip relay addresses _, err := addr.ValueForProtocol(P_CIRCUIT) if err == nil { @@ -201,7 +168,7 @@ func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { } // Skip dialing addresses we believe are the local node's - for _, localAddr := range as.h.Addrs() { + for _, localAddr := range as.config.host.Addrs() { if localAddr.Equal(addr) { return true } @@ -210,11 +177,12 @@ func (as *AutoNATService) skipDial(addr ma.Multiaddr) bool { return false } -func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { +func (as *autoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { // rate limit check as.mx.Lock() count := as.reqs[pi.ID] - if count >= AutoNATServiceThrottle || (as.globalReqMax > 0 && as.globalReqs >= as.globalReqMax) { + if count >= as.config.throttlePeerMax || (as.config.throttleGlobalMax > 0 && + as.globalReqs >= as.config.throttleGlobalMax) { as.mx.Unlock() return newDialResponseError(pb.Message_E_DIAL_REFUSED, "too many dials") } @@ -222,13 +190,13 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { as.globalReqs++ as.mx.Unlock() - ctx, cancel := context.WithTimeout(as.ctx, AutoNATServiceDialTimeout) + ctx, cancel := context.WithTimeout(as.ctx, as.config.dialTimeout) defer cancel() - as.dialer.Peerstore().ClearAddrs(pi.ID) + as.config.dialer.Peerstore().ClearAddrs(pi.ID) - as.dialer.Peerstore().AddAddrs(pi.ID, pi.Addrs, peerstore.TempAddrTTL) - conn, err := as.dialer.Network().DialPeer(ctx, pi.ID) + as.config.dialer.Peerstore().AddAddrs(pi.ID, pi.Addrs, peerstore.TempAddrTTL) + conn, err := as.config.dialer.DialPeer(ctx, pi.ID) if err != nil { log.Debugf("error dialing %s: %s", pi.ID.Pretty(), err.Error()) // wait for the context to timeout to avoid leaking timing information @@ -238,40 +206,15 @@ func (as *AutoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { } ra := conn.RemoteMultiaddr() - as.dialer.Network().ClosePeer(pi.ID) + as.config.dialer.ClosePeer(pi.ID) return newDialResponseOK(ra) } -func (as *AutoNATService) enableWhenPublic() { - sub, _ := as.h.EventBus().Subscribe(&event.EvtLocalReachabilityChanged{}) - defer sub.Close() - - running := false - - for { - select { - case ev, ok := <-sub.Out(): - if !ok { - return - } - state := ev.(event.EvtLocalReachabilityChanged).Reachability - if state == network.ReachabilityPublic { - as.h.SetStreamHandler(autonat.AutoNATProto, as.handleStream) - if !running { - go as.resetRateLimiter() - running = true - } - } else { - as.h.RemoveStreamHandler(autonat.AutoNATProto) - } - case <-as.ctx.Done(): - return - } - } -} +// Enable the autoNAT service temporarily until the associated context is canceled. +func (as *autoNATService) Enable(ctx context.Context) { + as.config.host.SetStreamHandler(AutoNATProto, as.handleStream) -func (as *AutoNATService) resetRateLimiter() { - timer := time.NewTimer(AutoNATServiceResetInterval) + timer := time.NewTimer(as.config.throttleResetPeriod) defer timer.Stop() for { @@ -281,9 +224,10 @@ func (as *AutoNATService) resetRateLimiter() { as.reqs = make(map[peer.ID]int) as.globalReqs = 0 as.mx.Unlock() - jitter := rand.Float32() * float32(AutoNATServiceResetJitter) - timer.Reset(AutoNATServiceResetInterval + time.Duration(int64(jitter))) - case <-as.ctx.Done(): + jitter := rand.Float32() * float32(as.config.throttleResetJitter) + timer.Reset(as.config.throttleResetPeriod + time.Duration(int64(jitter))) + case <-ctx.Done(): + as.config.host.RemoveStreamHandler(AutoNATProto) return } } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 5e845c737f..260ce64d87 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -6,46 +6,38 @@ import ( "testing" "time" - "github.com/libp2p/go-libp2p" + bhost "github.com/libp2p/go-libp2p-blankhost" "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/network" - "github.com/libp2p/go-libp2p-core/peer" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" - autonat "github.com/libp2p/go-libp2p-autonat" + ma "github.com/multiformats/go-multiaddr" manet "github.com/multiformats/go-multiaddr-net" ) -func makeAutoNATService(ctx context.Context, t *testing.T) (host.Host, *AutoNATService) { - h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) - if err != nil { - t.Fatal(err) - } - - as, err := NewAutoNATService(ctx, h, true) - if err != nil { - t.Fatal(err) - } - - return h, as +func makeAutoNATConfig(ctx context.Context, t *testing.T) *config { + h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + dh := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + c := config{host: h, dialer: dh.Network()} + _ = defaults(&c) + c.forceServer = true + return &c } -func makeAutoNATClient(ctx context.Context, t *testing.T) (host.Host, autonat.AutoNATClient) { - h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) +func makeAutoNATService(ctx context.Context, t *testing.T, c *config) *autoNATService { + as, err := newAutoNATService(ctx, c) if err != nil { t.Fatal(err) } - cli := autonat.NewAutoNATClient(h, nil) - return h, cli + return as } -func connect(t *testing.T, a, b host.Host) { - pinfo := peer.AddrInfo{ID: a.ID(), Addrs: a.Addrs()} - err := b.Connect(context.Background(), pinfo) - if err != nil { - t.Fatal(err) - } +func makeAutoNATClient(ctx context.Context, t *testing.T) (host.Host, Client) { + h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + cli := NewAutoNATClient(h, nil) + return h, cli } // Note: these tests assume that the host has only private network addresses! @@ -53,23 +45,20 @@ func TestAutoNATServiceDialError(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save := AutoNATServiceDialTimeout - AutoNATServiceDialTimeout = 1 * time.Second - - hs, _ := makeAutoNATService(ctx, t) + c := makeAutoNATConfig(ctx, t) + c.dialTimeout = 1 * time.Second + _ = makeAutoNATService(ctx, t, c) hc, ac := makeAutoNATClient(ctx, t) - connect(t, hs, hc) + connect(t, c.host, hc) - _, err := ac.DialBack(ctx, hs.ID()) + _, err := ac.DialBack(ctx, c.host.ID()) if err == nil { t.Fatal("Dial back succeeded unexpectedly!") } - if !autonat.IsDialError(err) { + if !IsDialError(err) { t.Fatal(err) } - - AutoNATServiceDialTimeout = save } func TestAutoNATServiceDialSuccess(t *testing.T) { @@ -79,11 +68,13 @@ func TestAutoNATServiceDialSuccess(t *testing.T) { save := manet.Private4 manet.Private4 = []*net.IPNet{} - hs, _ := makeAutoNATService(ctx, t) + c := makeAutoNATConfig(ctx, t) + _ = makeAutoNATService(ctx, t, c) + hc, ac := makeAutoNATClient(ctx, t) - connect(t, hs, hc) + connect(t, c.host, hc) - _, err := ac.DialBack(ctx, hs.ID()) + _, err := ac.DialBack(ctx, c.host.ID()) if err != nil { t.Fatalf("Dial back failed: %s", err.Error()) } @@ -95,68 +86,59 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save1 := AutoNATServiceDialTimeout - AutoNATServiceDialTimeout = 1 * time.Second - save2 := AutoNATServiceResetInterval - AutoNATServiceResetInterval = 1 * time.Second - save3 := AutoNATServiceThrottle - AutoNATServiceThrottle = 1 - save4 := manet.Private4 + save1 := manet.Private4 manet.Private4 = []*net.IPNet{} - save5 := AutoNATServiceResetJitter - AutoNATServiceResetJitter = 0 * time.Second - hs, _ := makeAutoNATService(ctx, t) + c := makeAutoNATConfig(ctx, t) + c.dialTimeout = 1 * time.Second + c.throttleResetPeriod = time.Second + c.throttleResetJitter = 0 + c.throttlePeerMax = 1 + _ = makeAutoNATService(ctx, t, c) + hc, ac := makeAutoNATClient(ctx, t) - connect(t, hs, hc) + connect(t, c.host, hc) - _, err := ac.DialBack(ctx, hs.ID()) + _, err := ac.DialBack(ctx, c.host.ID()) if err != nil { t.Fatal(err) } - _, err = ac.DialBack(ctx, hs.ID()) + _, err = ac.DialBack(ctx, c.host.ID()) if err == nil { t.Fatal("Dial back succeeded unexpectedly!") } - if !autonat.IsDialRefused(err) { + if !IsDialRefused(err) { t.Fatal(err) } time.Sleep(2 * time.Second) - _, err = ac.DialBack(ctx, hs.ID()) + _, err = ac.DialBack(ctx, c.host.ID()) if err != nil { t.Fatal(err) } - AutoNATServiceDialTimeout = save1 - AutoNATServiceResetInterval = save2 - AutoNATServiceThrottle = save3 - manet.Private4 = save4 - AutoNATServiceResetJitter = save5 + manet.Private4 = save1 } func TestAutoNATServiceGlobalLimiter(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save1 := AutoNATServiceDialTimeout - AutoNATServiceDialTimeout = 1 * time.Second - save2 := AutoNATServiceResetInterval - AutoNATServiceResetInterval = 10 * time.Second - save3 := AutoNATServiceThrottle - AutoNATServiceThrottle = 1 - save4 := manet.Private4 + save1 := manet.Private4 manet.Private4 = []*net.IPNet{} - save5 := AutoNATServiceResetJitter - AutoNATServiceResetJitter = 0 * time.Second - save6 := AutoNATGlobalThrottle - AutoNATGlobalThrottle = 5 - hs, as := makeAutoNATService(ctx, t) - as.globalReqMax = 5 + c := makeAutoNATConfig(ctx, t) + c.dialTimeout = time.Second + c.throttleResetPeriod = 10 * time.Second + c.throttleResetJitter = 0 + c.throttlePeerMax = 1 + c.throttleGlobalMax = 5 + _ = makeAutoNATService(ctx, t, c) + + hs := c.host for i := 0; i < 5; i++ { hc, ac := makeAutoNATClient(ctx, t) @@ -175,27 +157,21 @@ func TestAutoNATServiceGlobalLimiter(t *testing.T) { t.Fatal("Dial back succeeded unexpectedly!") } - if !autonat.IsDialRefused(err) { + if !IsDialRefused(err) { t.Fatal(err) } - AutoNATServiceDialTimeout = save1 - AutoNATServiceResetInterval = save2 - AutoNATServiceThrottle = save3 - manet.Private4 = save4 - AutoNATServiceResetJitter = save5 - AutoNATGlobalThrottle = save6 + manet.Private4 = save1 } func TestAutoNATServiceRateLimitJitter(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) - save1 := AutoNATServiceResetInterval - AutoNATServiceResetInterval = 100 * time.Millisecond - save2 := AutoNATServiceResetJitter - AutoNATServiceResetJitter = 100 * time.Millisecond - - _, svc := makeAutoNATService(ctx, t) + c := makeAutoNATConfig(ctx, t) + c.throttleResetPeriod = 100 * time.Millisecond + c.throttleResetJitter = 100 * time.Millisecond + c.throttleGlobalMax = 1 + svc := makeAutoNATService(ctx, t, c) svc.mx.Lock() svc.globalReqs = 1 svc.mx.Unlock() @@ -208,9 +184,6 @@ func TestAutoNATServiceRateLimitJitter(t *testing.T) { } cancel() - - AutoNATServiceResetInterval = save1 - AutoNATServiceResetJitter = save2 } func TestAutoNATServiceStartup(t *testing.T) { @@ -220,18 +193,13 @@ func TestAutoNATServiceStartup(t *testing.T) { save := manet.Private4 manet.Private4 = []*net.IPNet{} - h, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) + h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + dh := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + an, err := New(ctx, h, EnableService(dh.Network(), false)) if err != nil { t.Fatal(err) } - _, err = NewAutoNATService(ctx, h, false) - if err != nil { - t.Fatal(err) - } - - eb, _ := h.EventBus().Emitter(new(event.EvtLocalReachabilityChanged)) - hc, ac := makeAutoNATClient(ctx, t) connect(t, h, hc) @@ -240,16 +208,19 @@ func TestAutoNATServiceStartup(t *testing.T) { t.Fatal("autonat should not be started / advertising.") } - eb.Emit(event.EvtLocalReachabilityChanged{Reachability: network.ReachabilityPublic}) + sub, _ := h.EventBus().Subscribe(new(event.EvtLocalReachabilityChanged)) + + anc := an.(*AmbientAutoNAT) + anc.recordObservation(autoNATResult{Reachability: network.ReachabilityPublic, address: ma.StringCast("/ip4/127.0.0.1/tcp/1234")}) + + <-sub.Out() + _, err = ac.DialBack(ctx, h.ID()) if err != nil { t.Fatalf("autonat should be active, was %v", err) } - - eb.Emit(event.EvtLocalReachabilityChanged{Reachability: network.ReachabilityPrivate}) - _, err = ac.DialBack(ctx, h.ID()) - if err == nil { - t.Fatal("autonat should not be started / advertising.") + if an.Status() != network.ReachabilityPublic { + t.Fatalf("autonat should report public, but didn't") } manet.Private4 = save diff --git a/p2p/host/autonat/test/autonat_test.go b/p2p/host/autonat/test/autonat_test.go new file mode 100644 index 0000000000..ad037f6b1f --- /dev/null +++ b/p2p/host/autonat/test/autonat_test.go @@ -0,0 +1,60 @@ +package autonat_test + +import ( + "context" + "testing" + "time" + + "github.com/libp2p/go-libp2p" + autonat "github.com/libp2p/go-libp2p-autonat" + "github.com/libp2p/go-libp2p-core/event" + "github.com/libp2p/go-libp2p-core/network" +) + +// This separate testing package helps to resolve a circular dependency potentially +// being created between libp2p and libp2p-autonat + +func TestAutonatRoundtrip(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // 3 hosts are used: [client] and [service + dialback dialer] + client, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"), libp2p.EnableAutoNAT()) + if err != nil { + t.Fatal(err) + } + + service, err := libp2p.New(ctx, libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0")) + if err != nil { + t.Fatal(err) + } + dialback, err := libp2p.New(ctx, libp2p.NoListenAddrs) + if err != nil { + t.Fatal(err) + } + _, err = autonat.New(ctx, service, autonat.EnableService(dialback.Network(), true)) + if err != nil { + t.Fatal(err) + } + + client.Peerstore().AddAddrs(service.ID(), service.Addrs(), time.Hour) + err = client.Connect(ctx, service.Peerstore().PeerInfo(service.ID())) + if err != nil { + t.Fatal(err) + } + + cSub, err := client.EventBus().Subscribe(new(event.EvtLocalReachabilityChanged)) + if err != nil { + t.Fatal(err) + } + defer cSub.Close() + + select { + case stat := <-cSub.Out(): + if stat == network.ReachabilityUnknown { + t.Fatalf("After status update, client did not know its status") + } + case <-time.After(30 * time.Second): + t.Fatal("sub timed out.") + } +} diff --git a/p2p/host/autonat/test/go.mod b/p2p/host/autonat/test/go.mod new file mode 100644 index 0000000000..17ba0a0142 --- /dev/null +++ b/p2p/host/autonat/test/go.mod @@ -0,0 +1,23 @@ +module github.com/libp2p/go-libp2p-autonat/test + +replace github.com/libp2p/go-libp2p-autonat => ../ + +require ( + github.com/libp2p/go-conn-security v0.1.0 // indirect + github.com/libp2p/go-libp2p v0.6.1-0.20200317201052-dd87382dd436 + github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200317183318-4b2cc5830d44 + github.com/libp2p/go-libp2p-core v0.5.0 + github.com/libp2p/go-libp2p-host v0.1.0 // indirect + github.com/libp2p/go-libp2p-interface-connmgr v0.1.0 // indirect + github.com/libp2p/go-libp2p-interface-pnet v0.1.0 // indirect + github.com/libp2p/go-libp2p-metrics v0.1.0 // indirect + github.com/libp2p/go-libp2p-net v0.1.0 // indirect + github.com/libp2p/go-libp2p-protocol v0.1.0 // indirect + github.com/libp2p/go-libp2p-transport v0.1.0 // indirect + github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible // indirect + github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible // indirect + github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible // indirect + github.com/whyrusleeping/yamux v1.2.0 // indirect +) + +go 1.13 diff --git a/p2p/host/autonat/test/go.sum b/p2p/host/autonat/test/go.sum new file mode 100644 index 0000000000..ad52e804ee --- /dev/null +++ b/p2p/host/autonat/test/go.sum @@ -0,0 +1,471 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +github.com/AndreasBriese/bbloom v0.0.0-20180913140656-343706a395b7/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= +github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/Kubuxu/go-os-helper v0.0.1/go.mod h1:N8B+I7vPCT80IcP58r50u4+gEEcsZETFUpAzWW2ep1Y= +github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= +github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= +github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32/go.mod h1:DrZx5ec/dmnfpw9KyYoQyYo7d0KEvTkk/5M/vbZjAr8= +github.com/btcsuite/btcd v0.0.0-20190523000118-16327141da8c/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI= +github.com/btcsuite/btcd v0.0.0-20190824003749-130ea5bddde3/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI= +github.com/btcsuite/btcd v0.20.1-beta h1:Ik4hyJqN8Jfyv3S4AGBOmyouMsYE3EdYODkMbQjwPGw= +github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ= +github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA= +github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= +github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg= +github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd/go.mod h1:HHNXQzUsZCxOoE+CPiyCTO6x34Zs86zZUiwtpXoGdtg= +github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVaaLLH7j4eDXPRvw78tMflu7Ie2bzYOH4Y8rRKBY= +github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc= +github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY= +github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= +github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk= +github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/coreos/go-semver v0.3.0 h1:wkHLiw0WNATZnSG7epLsujiMCgPAc9xhjJ4tgnAxmfM= +github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE= +github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davidlazar/go-crypto v0.0.0-20170701192655-dcfb0a7ac018 h1:6xT9KW8zLC5IlbaIF5Q7JNieBoACT7iW0YTxQHR0in0= +github.com/davidlazar/go-crypto v0.0.0-20170701192655-dcfb0a7ac018/go.mod h1:rQYf4tfk5sSwFsnDg3qYaBxSjsD9S8+59vW0dKUgme4= +github.com/dgraph-io/badger v1.5.5-0.20190226225317-8115aed38f8f/go.mod h1:VZxzAIRPHRVNRKRo6AXrX9BJegn6il06VMTZVJYCIjQ= +github.com/dgraph-io/badger v1.6.0-rc1/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= +github.com/dgraph-io/badger v1.6.0/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= +github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= +github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= +github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= +github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98= +github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= +github.com/gogo/protobuf v1.3.0/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= +github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0= +github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= +github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/gorilla/websocket v1.4.1 h1:q7AeDBpnBk8AogcD4DSag/Ukw/KV+YhzLj2bP5HvKCM= +github.com/gorilla/websocket v1.4.1/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= +github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= +github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= +github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= +github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= +github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= +github.com/huin/goupnp v1.0.0 h1:wg75sLpL6DZqwHQN6E1Cfk6mtfzS45z8OV+ic+DtHRo= +github.com/huin/goupnp v1.0.0/go.mod h1:n9v9KO1tAxYH82qOn+UTIFQDmx5n1Zxd/ClZDMX7Bnc= +github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150/go.mod h1:PpLOETDnJ0o3iZrZfqZzyLl6l7F3c6L1oWn7OICBi6o= +github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= +github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= +github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= +github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= +github.com/ipfs/go-cid v0.0.4/go.mod h1:4LLaPOQwmk5z9LBgQnpkivrx8BJjUyGwTXCd5Xfj6+M= +github.com/ipfs/go-cid v0.0.5 h1:o0Ix8e/ql7Zb5UVUJEUfjsWCIY8t48++9lR8qi6oiJU= +github.com/ipfs/go-cid v0.0.5/go.mod h1:plgt+Y5MnOey4vO4UlUazGqdbEXuFYitED67FexhXog= +github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= +github.com/ipfs/go-datastore v0.4.0/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= +github.com/ipfs/go-datastore v0.4.1/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= +github.com/ipfs/go-datastore v0.4.4/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= +github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= +github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8= +github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaHzfGTzuE3s= +github.com/ipfs/go-ds-badger v0.2.1/go.mod h1:Tx7l3aTph3FMFrRS838dcSJh+jjA7cX9DrGVwx/NOwE= +github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc= +github.com/ipfs/go-ds-leveldb v0.4.1/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s= +github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= +github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50= +github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc= +github.com/ipfs/go-log v0.0.1 h1:9XTUN/rW64BCG1YhPK9Hoy3q8nr4gOmHHBpgFdfw6Lc= +github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= +github.com/ipfs/go-log v1.0.2 h1:s19ZwJxH8rPWzypjcDpqPLIyV7BnbLqvpli3iZoqYK0= +github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= +github.com/ipfs/go-log/v2 v2.0.2 h1:xguurydRdfKMJjKyxNXNU8lYP0VZH1NUwJRwUorjuEw= +github.com/ipfs/go-log/v2 v2.0.2/go.mod h1:O7P1lJt27vWHhOwQmcFEvlmo49ry2VY2+JfBWFaa9+0= +github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc= +github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= +github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= +github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= +github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec/go.mod h1:rGaEvXB4uRSZMmzKNLoXvTu1sfx+1kv/DojUlPrSZGs= +github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA= +github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2 h1:vhC1OXXiT9R2pczegwz6moDvuRpggaroAXhPIseh57A= +github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2/go.mod h1:8GXXJV31xl8whumTzdZsTt3RnUIiPqzkyf7mxToRCMs= +github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsjFq/qrU3Rar62tu1gASgGw6chQbSh/XgIIXCY= +github.com/jbenet/goprocess v0.1.3 h1:YKyIEECS/XvcfHtBzxtjBBbWK+MbvA6dG8ASiqwvr10= +github.com/jbenet/goprocess v0.1.3/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= +github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= +github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= +github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= +github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d/go.mod h1:P2viExyCEfeWGU259JnaQ34Inuec4R38JCyBx2edgD0= +github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= +github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4= +github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= +github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d h1:68u9r4wEvL3gYg2jvAOgROwZ3H+Y3hIDk4tbbmIjcYQ= +github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/libp2p/go-addr-util v0.0.1 h1:TpTQm9cXVRVSKsYbgQ7GKc3KbbHVTnbostgGaDEP+88= +github.com/libp2p/go-addr-util v0.0.1/go.mod h1:4ac6O7n9rIAKB1dnd+s8IbbMXkt+oBpzX4/+RACcnlQ= +github.com/libp2p/go-buffer-pool v0.0.1/go.mod h1:xtyIz9PMobb13WaxR6Zo1Pd1zXJKYg0a8KiIvDp3TzQ= +github.com/libp2p/go-buffer-pool v0.0.2 h1:QNK2iAFa8gjAe1SPz6mHSMuCcjs+X1wlHzeOSqcmlfs= +github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM= +github.com/libp2p/go-conn-security v0.1.0 h1:q8ii9TUOtSBD1gIoKTSOZIzPFP/agPM28amrCCoeIIA= +github.com/libp2p/go-conn-security v0.1.0/go.mod h1:NQdPF4opCZ5twtEUadzPL0tNSdkrbFc/HmLO7eWqEzY= +github.com/libp2p/go-conn-security-multistream v0.1.0 h1:aqGmto+ttL/uJgX0JtQI0tD21CIEy5eYd1Hlp0juHY0= +github.com/libp2p/go-conn-security-multistream v0.1.0/go.mod h1:aw6eD7LOsHEX7+2hJkDxw1MteijaVcI+/eP2/x3J1xc= +github.com/libp2p/go-eventbus v0.1.0 h1:mlawomSAjjkk97QnYiEmHsLu7E136+2oCWSHRUvMfzQ= +github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UGQf8A2eEmx4= +github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= +github.com/libp2p/go-flow-metrics v0.0.3 h1:8tAs/hSdNvUiLgtlSy3mxwxWP4I9y/jlkPFT7epKdeM= +github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= +github.com/libp2p/go-libp2p v0.6.0/go.mod h1:mfKWI7Soz3ABX+XEBR61lGbg+ewyMtJHVt043oWeqwg= +github.com/libp2p/go-libp2p v0.6.1-0.20200317192708-69b16483fd3c h1:gFOGoy0nvca3V13ZYGfCK9uqD548RLD70/CFKDwqSQ8= +github.com/libp2p/go-libp2p v0.6.1-0.20200317192708-69b16483fd3c/go.mod h1:AFitagZ+1JiUV1TFqkaFNCd6I8YHrnVjzNG+Hou45wc= +github.com/libp2p/go-libp2p v0.6.1-0.20200317201052-dd87382dd436 h1:h5s4ZISlgAvfDjD64z96DcsstCId4C+TKEgVq7w2SiI= +github.com/libp2p/go-libp2p v0.6.1-0.20200317201052-dd87382dd436/go.mod h1:36vxFf5QwnZWcXHDqBF5Wk2evbtZZphUXA4oHaVplPc= +github.com/libp2p/go-libp2p v6.0.23+incompatible h1:J/h9LNTeQwMhJeg3M96r/UOPLGxJn1vqJBb3LeKufpM= +github.com/libp2p/go-libp2p v6.0.23+incompatible/go.mod h1:CyUlFa6Mw04PkmMg8gBIlHUl8j3TrEiA6oFiF4SgD8w= +github.com/libp2p/go-libp2p-autonat v0.1.1 h1:WLBZcIRsjZlWdAZj9CiBSvU2wQXoUOiS1Zk1tM7DTJI= +github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= +github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= +github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= +github.com/libp2p/go-libp2p-circuit v0.1.4 h1:Phzbmrg3BkVzbqd4ZZ149JxCuUWu2wZcXf/Kr6hZJj8= +github.com/libp2p/go-libp2p-circuit v0.1.4/go.mod h1:CY67BrEjKNDhdTk8UgBX1Y/H5c3xkAcs3gnksxY7osU= +github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= +github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= +github.com/libp2p/go-libp2p-core v0.2.0/go.mod h1:X0eyB0Gy93v0DZtSYbEM7RnMChm9Uv3j7yRXjO77xSI= +github.com/libp2p/go-libp2p-core v0.2.2/go.mod h1:8fcwTbsG2B+lTgRJ1ICZtiM5GWCWZVoVrLaDRvIRng0= +github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g= +github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw= +github.com/libp2p/go-libp2p-core v0.3.1/go.mod h1:thvWy0hvaSBhnVBaW37BvzgVV68OUhgJJLAa6almrII= +github.com/libp2p/go-libp2p-core v0.4.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= +github.com/libp2p/go-libp2p-core v0.5.0 h1:FBQ1fpq2Fo/ClyjojVJ5AKXlKhvNc/B6U0O+7AN1ffE= +github.com/libp2p/go-libp2p-core v0.5.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= +github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= +github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= +github.com/libp2p/go-libp2p-discovery v0.2.0 h1:1p3YSOq7VsgaL+xVHPi8XAmtGyas6D2J6rWBEfz/aiY= +github.com/libp2p/go-libp2p-discovery v0.2.0/go.mod h1:s4VGaxYMbw4+4+tsoQTqh7wfxg97AEdo4GYBt6BadWg= +github.com/libp2p/go-libp2p-host v0.1.0 h1:OZwENiFm6JOK3YR5PZJxkXlJE8a5u8g4YvAUrEV2MjM= +github.com/libp2p/go-libp2p-host v0.1.0/go.mod h1:5+fWuLbDn8OxoxPN3CV0vsLe1hAKScSMbT84qRfxum8= +github.com/libp2p/go-libp2p-interface-connmgr v0.1.0 h1:dFYeUNi5NjKIAVE+yQJULS99CovMUx9p/IgxI+2e+uc= +github.com/libp2p/go-libp2p-interface-connmgr v0.1.0/go.mod h1:bmmppYG/Bc6FTdLYEdpuSfifDa5Nr+5Ia1Mm6lE2+Eg= +github.com/libp2p/go-libp2p-interface-pnet v0.1.0 h1:PaofJtuDcrGBukgTymiGyuI313nxARRQFmE/oxZXlog= +github.com/libp2p/go-libp2p-interface-pnet v0.1.0/go.mod h1:8+FQ08+xMxR6BjG0tUZoQzKxPAV2W7ck6IxjCWqZ6ek= +github.com/libp2p/go-libp2p-loggables v0.1.0 h1:h3w8QFfCt2UJl/0/NW4K829HX/0S4KD31PQ7m8UXXO8= +github.com/libp2p/go-libp2p-loggables v0.1.0/go.mod h1:EyumB2Y6PrYjr55Q3/tiJ/o3xoDasoRYM7nOzEpoa90= +github.com/libp2p/go-libp2p-metrics v0.1.0 h1:v7YMUTHNobFaQeqaMfJJMbnK3EPlZeb6/KFm4gE9dks= +github.com/libp2p/go-libp2p-metrics v0.1.0/go.mod h1:rpoJmXWFxnj7qs5sJ02sxSzrhaZvpqBn8GCG6Sx6E1k= +github.com/libp2p/go-libp2p-mplex v0.2.0/go.mod h1:Ejl9IyjvXJ0T9iqUTE1jpYATQ9NM3g+OtR+EMMODbKo= +github.com/libp2p/go-libp2p-mplex v0.2.1/go.mod h1:SC99Rxs8Vuzrf/6WhmH41kNn13TiYdAWNYHrwImKLnE= +github.com/libp2p/go-libp2p-mplex v0.2.2 h1:+Ld7YDAfVERQ0E+qqjE7o6fHwKuM0SqTzYiwN1lVVSA= +github.com/libp2p/go-libp2p-mplex v0.2.2/go.mod h1:74S9eum0tVQdAfFiKxAyKzNdSuLqw5oadDq7+L/FELo= +github.com/libp2p/go-libp2p-nat v0.0.5 h1:/mH8pXFVKleflDL1YwqMg27W9GD8kjEx7NY0P6eGc98= +github.com/libp2p/go-libp2p-nat v0.0.5/go.mod h1:1qubaE5bTZMJE+E/uu2URroMbzdubFz1ChgiN79yKPE= +github.com/libp2p/go-libp2p-net v0.1.0 h1:3t23V5cR4GXcNoFriNoZKFdUZEUDZgUkvfwkD2INvQE= +github.com/libp2p/go-libp2p-net v0.1.0/go.mod h1:R5VZbutk75tkC5YJJS61OCO1NWoajxYjCEV2RoHh3FY= +github.com/libp2p/go-libp2p-netutil v0.1.0/go.mod h1:3Qv/aDqtMLTUyQeundkKsA+YCThNdbQD54k3TqjpbFU= +github.com/libp2p/go-libp2p-peer v0.2.0 h1:EQ8kMjaCUwt/Y5uLgjT8iY2qg0mGUT0N1zUjer50DsY= +github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMgxjsHm7+J5kjWY= +github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HBb6u0MOvdJ7YIsmcwtY= +github.com/libp2p/go-libp2p-peerstore v0.1.3 h1:wMgajt1uM2tMiqf4M+4qWKVyyFc8SfA+84VV9glZq1M= +github.com/libp2p/go-libp2p-peerstore v0.1.3/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= +github.com/libp2p/go-libp2p-peerstore v0.2.0 h1:XcgJhI8WyUOCbHyRLNEX5542YNj8hnLSJ2G1InRjDhk= +github.com/libp2p/go-libp2p-peerstore v0.2.0/go.mod h1:N2l3eVIeAitSg3Pi2ipSrJYnqhVnMNQZo9nkSCuAbnQ= +github.com/libp2p/go-libp2p-pnet v0.2.0 h1:J6htxttBipJujEjz1y0a5+eYoiPcFHhSYHH6na5f0/k= +github.com/libp2p/go-libp2p-pnet v0.2.0/go.mod h1:Qqvq6JH/oMZGwqs3N1Fqhv8NVhrdYcO0BW4wssv21LA= +github.com/libp2p/go-libp2p-protocol v0.1.0 h1:HdqhEyhg0ToCaxgMhnOmUO8snQtt/kQlcjVk3UoJU3c= +github.com/libp2p/go-libp2p-protocol v0.1.0/go.mod h1:KQPHpAabB57XQxGrXCNvbL6UEXfQqUgC/1adR2Xtflk= +github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= +github.com/libp2p/go-libp2p-secio v0.2.0 h1:ywzZBsWEEz2KNTn5RtzauEDq5RFEefPsttXYwAWqHng= +github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= +github.com/libp2p/go-libp2p-secio v0.2.1 h1:eNWbJTdyPA7NxhP7J3c5lT97DC5d+u+IldkgCYFTPVA= +github.com/libp2p/go-libp2p-secio v0.2.1/go.mod h1:cWtZpILJqkqrSkiYcDBh5lA3wbT2Q+hz3rJQq3iftD8= +github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= +github.com/libp2p/go-libp2p-swarm v0.2.2 h1:T4hUpgEs2r371PweU3DuH7EOmBIdTBCwWs+FLcgx3bQ= +github.com/libp2p/go-libp2p-swarm v0.2.2/go.mod h1:fvmtQ0T1nErXym1/aa1uJEyN7JzaTNyBcHImCxRpPKU= +github.com/libp2p/go-libp2p-testing v0.0.1/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= +github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= +github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= +github.com/libp2p/go-libp2p-testing v0.0.4/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= +github.com/libp2p/go-libp2p-testing v0.1.0/go.mod h1:xaZWMJrPUM5GlDBxCeGUi7kI4eqnjVyavGroI2nxEM0= +github.com/libp2p/go-libp2p-testing v0.1.1/go.mod h1:xaZWMJrPUM5GlDBxCeGUi7kI4eqnjVyavGroI2nxEM0= +github.com/libp2p/go-libp2p-transport v0.1.0 h1:q68SOTvX+71mk+n5eE3+FnUEPY5UL1CSFImH0bq0Vg8= +github.com/libp2p/go-libp2p-transport v0.1.0/go.mod h1:iL3c2tV3OVldqSwJrds8pmIWf4t/TwiF+eI/mhw/jjQ= +github.com/libp2p/go-libp2p-transport-upgrader v0.1.1 h1:PZMS9lhjK9VytzMCW3tWHAXtKXmlURSc3ZdvwEcKCzw= +github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07qPW4r01GnTenLW4oi3lOPbUMGJJA= +github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 h1:5EhPgQhXZNyfL22ERZTUoVp9UVVbNowWNVtELQaKCHk= +github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSoMuyojOdjx73Q47cYD7s5+gZOlns= +github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= +github.com/libp2p/go-libp2p-yamux v0.2.2 h1:eGvbqWqWY9S5lrpe2gA0UCOLCdzCgYSAR3vo/xCsNQg= +github.com/libp2p/go-libp2p-yamux v0.2.2/go.mod h1:lIohaR0pT6mOt0AZ0L2dFze9hds9Req3OfS+B+dv4qw= +github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= +github.com/libp2p/go-maddr-filter v0.0.5 h1:CW3AgbMO6vUvT4kf87y4N+0P8KUl2aqLYhrGyDUbLSg= +github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= +github.com/libp2p/go-mplex v0.0.3/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTWe7l4Yd0= +github.com/libp2p/go-mplex v0.1.0 h1:/nBTy5+1yRyY82YaO6HXQRnO5IAGsXTjEJaR3LdTPc0= +github.com/libp2p/go-mplex v0.1.0/go.mod h1:SXgmdki2kwCUlCCbfGLEgHjC4pFqhTp0ZoV6aiKgxDU= +github.com/libp2p/go-mplex v0.1.1 h1:huPH/GGRJzmsHR9IZJJsrSwIM5YE2gL4ssgl1YWb/ps= +github.com/libp2p/go-mplex v0.1.1/go.mod h1:Xgz2RDCi3co0LeZfgjm4OgUF15+sVR8SRcu3SFXI1lk= +github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= +github.com/libp2p/go-msgio v0.0.4 h1:agEFehY3zWJFUHK6SEMR7UYmk2z6kC3oeCM7ybLhguA= +github.com/libp2p/go-msgio v0.0.4/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= +github.com/libp2p/go-nat v0.0.4 h1:KbizNnq8YIf7+Hn7+VFL/xE0eDrkPru2zIO9NMwL8UQ= +github.com/libp2p/go-nat v0.0.4/go.mod h1:Nmw50VAvKuk38jUBcmNh6p9lUJLoODbJRvYAa/+KSDo= +github.com/libp2p/go-openssl v0.0.2/go.mod h1:v8Zw2ijCSWBQi8Pq5GAixw6DbFfa9u6VIYDXnvOXkc0= +github.com/libp2p/go-openssl v0.0.3/go.mod h1:unDrJpgy3oFr+rqXsarWifmJuNnJR4chtO1HmaZjggc= +github.com/libp2p/go-openssl v0.0.4 h1:d27YZvLoTyMhIN4njrkr8zMDOM4lfpHIp6A+TK9fovg= +github.com/libp2p/go-openssl v0.0.4/go.mod h1:unDrJpgy3oFr+rqXsarWifmJuNnJR4chtO1HmaZjggc= +github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FWpFLw= +github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= +github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0GIEjMiHTwR/KN4= +github.com/libp2p/go-reuseport-transport v0.0.2/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= +github.com/libp2p/go-stream-muxer v0.0.1 h1:Ce6e2Pyu+b5MC1k3eeFtAax0pW4gc6MosYSLV05UeLw= +github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= +github.com/libp2p/go-stream-muxer-multistream v0.2.0 h1:714bRJ4Zy9mdhyTLJ+ZKiROmAFwUHpeRidG+q7LTQOg= +github.com/libp2p/go-stream-muxer-multistream v0.2.0/go.mod h1:j9eyPol/LLRqT+GPLSxvimPhNph4sfYfMoDPd7HkzIc= +github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= +github.com/libp2p/go-tcp-transport v0.1.1 h1:yGlqURmqgNA2fvzjSgZNlHcsd/IulAnKM8Ncu+vlqnw= +github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfjg2pWP97dFZworkY= +github.com/libp2p/go-ws-transport v0.2.0 h1:MJCw2OrPA9+76YNRvdo1wMnSOxb9Bivj6sVFY1Xrj6w= +github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzlHoKzu0yY9p/klM= +github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-yamux v1.3.0 h1:FsYzT16Wq2XqUGJsBbOxoz9g+dFklvNi7jN6YFPfl7U= +github.com/libp2p/go-yamux v1.3.0/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= +github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= +github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= +github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= +github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= +github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= +github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= +github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE= +github.com/miekg/dns v1.1.12/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= +github.com/miekg/dns v1.1.28/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= +github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g= +github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ= +github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= +github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= +github.com/minio/sha256-simd v0.1.0/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= +github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= +github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU= +github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= +github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= +github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= +github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= +github.com/mr-tron/base58 v1.1.1/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= +github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= +github.com/mr-tron/base58 v1.1.3 h1:v+sk57XuaCKGXpWtVBX8YJzO7hMGx4Aajh4TQbdEFdc= +github.com/mr-tron/base58 v1.1.3/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= +github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI= +github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA= +github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.0.2/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.0.4/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.1.0/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= +github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo= +github.com/multiformats/go-multiaddr v0.2.0/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4= +github.com/multiformats/go-multiaddr v0.2.1 h1:SgG/cw5vqyB5QQe5FPe2TqggU9WtrA9X4nZw7LlVqOI= +github.com/multiformats/go-multiaddr v0.2.1/go.mod h1:s/Apk6IyxfvMjDafnhJgJ3/46z7tZ04iMk5wP4QMGGE= +github.com/multiformats/go-multiaddr-dns v0.0.1 h1:jQt9c6tDSdQLIlBo4tXYx7QUHCPjxsB1zXcag/2S7zc= +github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= +github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= +github.com/multiformats/go-multiaddr-dns v0.2.0 h1:YWJoIDwLePniH7OU5hBnDZV6SWuvJqJ0YtN6pLeH9zA= +github.com/multiformats/go-multiaddr-dns v0.2.0/go.mod h1:TJ5pr5bBO7Y1B18djPuRsVkduhQH2YqYSbxWJzYGdK0= +github.com/multiformats/go-multiaddr-fmt v0.0.1/go.mod h1:aBYjqL4T/7j4Qx+R73XSv/8JsgnRFlf0w2KGLCmXl3Q= +github.com/multiformats/go-multiaddr-fmt v0.1.0 h1:WLEFClPycPkp4fnIzoFoV9FVd49/eQsuaL3/CWe167E= +github.com/multiformats/go-multiaddr-fmt v0.1.0/go.mod h1:hGtDIW4PU4BqJ50gW2quDuPVjyWNZxToGUh/HwTZYJo= +github.com/multiformats/go-multiaddr-net v0.0.1/go.mod h1:nw6HSxNmCIQH27XPGBuX+d1tnvM7ihcFwHMSstNAVUU= +github.com/multiformats/go-multiaddr-net v0.1.0 h1:ZepO8Ezwovd+7b5XPPDhQhayk1yt0AJpzQBpq9fejx4= +github.com/multiformats/go-multiaddr-net v0.1.0/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= +github.com/multiformats/go-multiaddr-net v0.1.1 h1:jFFKUuXTXv+3ARyHZi3XUqQO+YWMKgBdhEvuGRfnL6s= +github.com/multiformats/go-multiaddr-net v0.1.1/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ= +github.com/multiformats/go-multiaddr-net v0.1.2/go.mod h1:QsWt3XK/3hwvNxZJp92iMQKME1qHfpYmyIjFVsSOY6Y= +github.com/multiformats/go-multiaddr-net v0.1.3 h1:q/IYAvoPKuRzGeERn3uacWgm0LIWkLZBAvO5DxSzq3g= +github.com/multiformats/go-multiaddr-net v0.1.3/go.mod h1:ilNnaM9HbmVFqsb/qcNysjCu4PVONlrBZpHIrw/qQuA= +github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= +github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= +github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= +github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= +github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= +github.com/multiformats/go-multihash v0.0.10/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= +github.com/multiformats/go-multihash v0.0.13 h1:06x+mk/zj1FoMsgNejLpy6QTvJqlSt/BhLEy87zidlc= +github.com/multiformats/go-multihash v0.0.13/go.mod h1:VdAWLKTwram9oKAatUcLxBNUjdtcVwxObEQBtRfuyjc= +github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ= +github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg= +github.com/multiformats/go-multistream v0.1.1 h1:JlAdpIFhBhGRLxe9W6Om0w++Gd6KMWoFPZL/dEnm9nI= +github.com/multiformats/go-multistream v0.1.1/go.mod h1:KmHZ40hzVxiaiwlj3MEbYgK9JFk2/9UktWZAF54Du38= +github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= +github.com/multiformats/go-varint v0.0.2/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= +github.com/multiformats/go-varint v0.0.5 h1:XVZwSo04Cs3j/jS0uAEPpT3JY6DzMcVLLoWOSnCxOjg= +github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= +github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.12.0/go.mod h1:oUhWkIvk5aDxtKvDDuw8gItl8pKl42LzjC9KZE0HfGg= +github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= +github.com/onsi/gomega v1.9.0/go.mod h1:Ho0h+IUsWyvy1OpqCwxlQ/21gkhVunqlU8fDGcoTdcA= +github.com/opentracing/opentracing-go v1.0.2 h1:3jA2P6O1F9UOrWVpwrIo17pu01KWvNWg4X946/Y5Zwg= +github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= +github.com/opentracing/opentracing-go v1.1.0 h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU= +github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= +github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= +github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= +github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= +github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= +github.com/smola/gocompat v0.2.0/go.mod h1:1B0MlxbmoZNo3h8guHp8HztB3BSYR5itql9qtVc0ypY= +github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= +github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= +github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572/go.mod h1:w0SWMsp6j9O/dk4/ZpIhL+3CkG8ofA2vuv7k+ltqUMc= +github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= +github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= +github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= +github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= +github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= +github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= +github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= +github.com/src-d/envconfig v1.0.0/go.mod h1:Q9YQZ7BKITldTBnoxsE5gOeB5y66RyPXeue/R4aaNBc= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= +github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= +github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1/go.mod h1:8UvriyWtv5Q5EOgjHaSseUEdkQfvwFv1I/In/O2M9gc= +github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6Rz4bwmou+oE6Dt4Cb2BGMur5eR/GYptkKUVHo= +github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM= +github.com/whyrusleeping/go-logging v0.0.1 h1:fwpzlmT0kRC/Fmd0MdmGgJG/CXIZ6gFq46FQZjprUcc= +github.com/whyrusleeping/go-logging v0.0.1/go.mod h1:lDPYj54zutzG1XYfHAhcc7oNXEburHQBn+Iqd4yS4vE= +github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible h1:iqksILj8STw03EJQe7Laj4ubnw+ojOyik18cd5vPL1o= +github.com/whyrusleeping/go-smux-multiplex v3.0.16+incompatible/go.mod h1:34LEDbeKFZInPUrAG+bjuJmUXONGdEFW7XL0SpTY1y4= +github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible h1:BdYHctE9HJZLquG9tpTdwWcbG4FaX6tVKPGjCGgiVxo= +github.com/whyrusleeping/go-smux-multistream v2.0.2+incompatible/go.mod h1:dRWHHvc4HDQSHh9gbKEBbUZ+f2Q8iZTPG3UOGYODxSQ= +github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible h1:nVkExQ7pYlN9e45LcqTCOiDD0904fjtm0flnHZGbXkw= +github.com/whyrusleeping/go-smux-yamux v2.0.9+incompatible/go.mod h1:6qHUzBXUbB9MXmw3AUdB52L8sEb/hScCqOdW2kj/wuI= +github.com/whyrusleeping/mafmt v1.2.8 h1:TCghSl5kkwEE0j+sU/gudyhVMRlpBin8fMBBHg59EbA= +github.com/whyrusleeping/mafmt v1.2.8/go.mod h1:faQJFPbLSxzD9xpA02ttW/tS9vZykNvXwGvqIpk20FA= +github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9/go.mod h1:j4l84WPFclQPj320J9gp0XwNKBb3U0zt5CBqjPp22G4= +github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 h1:E9S12nwJwEOXe2d6gT6qxdvqMnNq+VnSsKPgm2ZZNds= +github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7/go.mod h1:X2c0RVCI1eSUFI8eLcY3c0423ykwiUdxLJtkDvruhjI= +github.com/whyrusleeping/yamux v1.2.0 h1:PzUrk7/Z0g/N5V4/+DesmKXYcCToALgj+SbATgs0B34= +github.com/whyrusleeping/yamux v1.2.0/go.mod h1:Cgw3gpb4DrDZ1FrP/5pxg/cpiY54Gr5uCXwUylwi2GE= +github.com/x-cray/logrus-prefixed-formatter v0.5.2/go.mod h1:2duySbKsL6M18s5GU7VPsoEPHyzalCE06qoARUCeBBE= +github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= +go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= +go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA= +go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8= +go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.uber.org/atomic v1.4.0 h1:cxzIVoETapQEqDhQu3QfnvXAV4AlzcvUCxkVUFw3+EU= +go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= +go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI= +go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= +go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM= +go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= +golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d h1:1ZiEyfaQIg3Qh0EoqpwAakHVhecoE5wlSg5GjnafJGw= +golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190923162816-aa69164e4478 h1:l5EDrHhldLYb3ZRHDUhXF7Om7MvYXnkV9/iQNo1lX6g= +golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k= +golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae h1:/WDfKMnPU+m5M4xB+6x4kaepxRw6jWvR5iDRdvjHgy8= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d/go.mod h1:z+K8VcOYVYcSwSjGebuDL6176A1XskgbtNl64NSg+n8= +gopkg.in/src-d/go-log.v1 v1.0.1/go.mod h1:GN34hKP0g305ysm2/hctJ0Y8nWP3zxXXJ8GFabTyABE= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= +gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= From 791cb9dd289be2c93a3ef93f7061985c3edd7e17 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 17 Mar 2020 20:54:32 -0700 Subject: [PATCH 1738/3965] fix: make sure to include peer in dial error --- p2p/net/swarm/swarm_dial.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 475a7e250e..f2b21abf0a 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -387,7 +387,7 @@ func (s *Swarm) dialAddrs(ctx context.Context, p peer.ID, remoteAddrs <-chan ma. // use a single response type instead of errs and conns, reduces complexity *a ton* respch := make(chan dialResult) - err := new(DialError) + err := &DialError{Peer: p} defer s.limiter.clearAllPeerDials(p) From 5c496af242bf32a9bec8d9c47cae7e72ea5de466 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 18 Mar 2020 13:17:04 -0700 Subject: [PATCH 1739/3965] Add option to force nat into a specified reachability state --- p2p/host/autonat/autonat.go | 60 +++++++++++++++++++++++++------- p2p/host/autonat/autonat_test.go | 17 +++++++++ p2p/host/autonat/options.go | 20 ++++++++--- p2p/host/autonat/svc.go | 12 ++++++- p2p/host/autonat/svc_test.go | 4 +-- 5 files changed, 92 insertions(+), 21 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 6156d0cbc5..f0bde8d407 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -47,6 +47,13 @@ type AmbientAutoNAT struct { emitReachabilityChanged event.Emitter } +type StaticAutoNAT struct { + ctx context.Context + host host.Host + reachability network.Reachability + service *autoNATService +} + type autoNATResult struct { network.Reachability address ma.Multiaddr @@ -54,10 +61,11 @@ type autoNATResult struct { // New creates a new NAT autodiscovery system attached to a host func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { + var err error conf := new(config) conf.host = h - if err := defaults(conf); err != nil { + if err = defaults(conf); err != nil { return nil, err } if conf.addressFunc == nil { @@ -65,13 +73,31 @@ func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { } for _, o := range options { - if err := o(conf); err != nil { + if err = o(conf); err != nil { return nil, err } } + emitReachabilityChanged, _ := h.EventBus().Emitter(new(event.EvtLocalReachabilityChanged), eventbus.Stateful) + + var service *autoNATService + if (!conf.forceReachability || conf.reachability == network.ReachabilityPublic) && conf.dialer != nil { + service, err = newAutoNATService(ctx, conf) + if err != nil { + return nil, err + } + } + + if conf.forceReachability { + emitReachabilityChanged.Emit(event.EvtLocalReachabilityChanged{Reachability: conf.reachability}) + return &StaticAutoNAT{ + ctx: ctx, + host: h, + reachability: conf.reachability, + service: service, + }, nil + } subAddrUpdated, _ := h.EventBus().Subscribe(new(event.EvtLocalAddressesUpdated)) - emitReachabilityChanged, _ := h.EventBus().Emitter(new(event.EvtLocalReachabilityChanged), eventbus.Stateful) as := &AmbientAutoNAT{ ctx: ctx, @@ -83,20 +109,13 @@ func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { subAddrUpdated: subAddrUpdated, emitReachabilityChanged: emitReachabilityChanged, + service: service, } as.status.Store(autoNATResult{network.ReachabilityUnknown, nil}) h.Network().Notify(as) go as.background() - if conf.dialer != nil { - var err error - as.service, err = newAutoNATService(ctx, conf) - if err != nil { - return nil, err - } - } - return as, nil } @@ -115,7 +134,7 @@ func (as *AmbientAutoNAT) emitStatus() { func (as *AmbientAutoNAT) PublicAddr() (ma.Multiaddr, error) { s := as.status.Load().(autoNATResult) if s.Reachability != network.ReachabilityPublic { - return nil, errors.New("NAT Status is not public") + return nil, errors.New("NAT status is not public") } return s.address, nil @@ -225,7 +244,7 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { if currentStatus.Reachability != network.ReachabilityPublic { // we are flipping our NATStatus, so confidence drops to 0 as.confidence = 0 - if as.service != nil && !as.config.forceServer { + if as.service != nil { ctx, cancel := context.WithCancel(as.ctx) go as.service.Enable(ctx) as.serviceCancel = cancel @@ -337,3 +356,18 @@ func shufflePeers(peers []peer.AddrInfo) { peers[i], peers[j] = peers[j], peers[i] } } + +func (s *StaticAutoNAT) Status() network.Reachability { + return s.reachability +} + +func (s *StaticAutoNAT) PublicAddr() (ma.Multiaddr, error) { + if s.reachability != network.ReachabilityPublic { + return nil, errors.New("NAT status is not public") + } + addrs := s.host.Addrs() + if len(addrs) > 0 { + return s.host.Addrs()[0], nil + } + return nil, errors.New("No available address") +} diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index c6bca8dd34..abeb810151 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -232,3 +232,20 @@ func TestAutoNATObservationRecording(t *testing.T) { } } + +func TestStaticNat(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) + s, _ := h.EventBus().Subscribe(&event.EvtLocalReachabilityChanged{}) + + nat, err := New(ctx, h, WithReachability(network.ReachabilityPrivate)) + if err != nil { + t.Fatal(err) + } + if nat.Status() != network.ReachabilityPrivate { + t.Fatalf("should be private") + } + expectEvent(t, s, network.ReachabilityPrivate) +} diff --git a/p2p/host/autonat/options.go b/p2p/host/autonat/options.go index 5ed6ee7578..6f7d729466 100644 --- a/p2p/host/autonat/options.go +++ b/p2p/host/autonat/options.go @@ -12,9 +12,10 @@ import ( type config struct { host host.Host - addressFunc AddrFunc - dialer network.Network - forceServer bool + addressFunc AddrFunc + dialer network.Network + forceReachability bool + reachability network.Reachability // client bootDelay time.Duration @@ -52,13 +53,22 @@ var defaults = func(c *config) error { // make parallel connections, and as such will modify both the associated peerstore // and terminate connections of this dialer. The dialer provided // should be compatible (TCP/UDP) however with the transports of the libp2p network. -func EnableService(dialer network.Network, forceServer bool) Option { +func EnableService(dialer network.Network) Option { return func(c *config) error { if dialer == c.host.Network() || dialer.Peerstore() == c.host.Peerstore() { return errors.New("dialer should not be that of the host") } c.dialer = dialer - c.forceServer = forceServer + return nil + } +} + +// WithReachability overrides autonat to simply report an over-ridden reachability +// status. +func WithReachability(reachability network.Reachability) Option { + return func(c *config) error { + c.forceReachability = true + c.reachability = reachability return nil } } diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index d55beacc37..41afafd56d 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -6,6 +6,7 @@ import ( "math/rand" "net" "sync" + "sync/atomic" "time" "github.com/libp2p/go-libp2p-core/helpers" @@ -29,6 +30,7 @@ type autoNATService struct { config *config // rate limiter + running uint32 mx sync.Mutex reqs map[peer.ID]int globalReqs int @@ -46,7 +48,7 @@ func newAutoNATService(ctx context.Context, c *config) (*autoNATService, error) reqs: make(map[peer.ID]int), } - if c.forceServer { + if c.forceReachability { go as.Enable(ctx) } @@ -212,6 +214,14 @@ func (as *autoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { // Enable the autoNAT service temporarily until the associated context is canceled. func (as *autoNATService) Enable(ctx context.Context) { + alreadyRunning := atomic.SwapUint32(&as.running, 1) + if alreadyRunning > 0 { + return + } + defer func() { + atomic.StoreUint32(&as.running, 0) + }() + as.config.host.SetStreamHandler(AutoNATProto, as.handleStream) timer := time.NewTimer(as.config.throttleResetPeriod) diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 260ce64d87..f38d4041a6 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -21,7 +21,7 @@ func makeAutoNATConfig(ctx context.Context, t *testing.T) *config { dh := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) c := config{host: h, dialer: dh.Network()} _ = defaults(&c) - c.forceServer = true + c.forceReachability = true return &c } @@ -195,7 +195,7 @@ func TestAutoNATServiceStartup(t *testing.T) { h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) dh := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) - an, err := New(ctx, h, EnableService(dh.Network(), false)) + an, err := New(ctx, h, EnableService(dh.Network())) if err != nil { t.Fatal(err) } From 7b280dd74dc58f65e312b3bdb8fbde6b857af1a9 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 18 Mar 2020 21:31:43 -0700 Subject: [PATCH 1740/3965] chore: update yamux On, final, time. --- go.mod | 2 +- go.sum | 25 +++++++------------------ 2 files changed, 8 insertions(+), 19 deletions(-) diff --git a/go.mod b/go.mod index 423fa4303a..2bdfa7c385 100644 --- a/go.mod +++ b/go.mod @@ -24,7 +24,7 @@ require ( github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 - github.com/libp2p/go-libp2p-yamux v0.2.4 + github.com/libp2p/go-libp2p-yamux v0.2.5 github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 diff --git a/go.sum b/go.sum index a47f524de7..00f276a2ce 100644 --- a/go.sum +++ b/go.sum @@ -70,6 +70,7 @@ github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmv github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= @@ -87,8 +88,6 @@ github.com/ipfs/go-cid v0.0.4/go.mod h1:4LLaPOQwmk5z9LBgQnpkivrx8BJjUyGwTXCd5Xfj github.com/ipfs/go-cid v0.0.5 h1:o0Ix8e/ql7Zb5UVUJEUfjsWCIY8t48++9lR8qi6oiJU= github.com/ipfs/go-cid v0.0.5/go.mod h1:plgt+Y5MnOey4vO4UlUazGqdbEXuFYitED67FexhXog= github.com/ipfs/go-datastore v0.0.1/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= -github.com/ipfs/go-datastore v0.1.0/go.mod h1:d4KVXhMt913cLBEI/PXAy6ko+W7e9AhyAKBGh803qeE= -github.com/ipfs/go-datastore v0.1.1/go.mod h1:w38XXW9kVFNp57Zj5knbKWM2T+KOZCGDRVNdgPHtbHw= github.com/ipfs/go-datastore v0.4.0/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= github.com/ipfs/go-datastore v0.4.1/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= github.com/ipfs/go-datastore v0.4.4/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA= @@ -96,10 +95,8 @@ github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8= github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaHzfGTzuE3s= -github.com/ipfs/go-ds-badger v0.0.7/go.mod h1:qt0/fWzZDoPW6jpQeqUjR5kBfhDNB65jd9YlmAvpQBk= github.com/ipfs/go-ds-badger v0.2.1/go.mod h1:Tx7l3aTph3FMFrRS838dcSJh+jjA7cX9DrGVwx/NOwE= github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc= -github.com/ipfs/go-ds-leveldb v0.1.0/go.mod h1:hqAW8y4bwX5LWcCtku2rFNX3vjDZCy5LZCg+cSZvYb8= github.com/ipfs/go-ds-leveldb v0.4.1/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50= @@ -191,8 +188,6 @@ github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMg github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HBb6u0MOvdJ7YIsmcwtY= github.com/libp2p/go-libp2p-peerstore v0.1.3 h1:wMgajt1uM2tMiqf4M+4qWKVyyFc8SfA+84VV9glZq1M= github.com/libp2p/go-libp2p-peerstore v0.1.3/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= -github.com/libp2p/go-libp2p-peerstore v0.1.4 h1:d23fvq5oYMJ/lkkbO4oTwBp/JP+I/1m5gZJobNXCE/k= -github.com/libp2p/go-libp2p-peerstore v0.1.4/go.mod h1:+4BDbDiiKf4PzpANZDAT+knVdLxvqh7hXOujessqdzs= github.com/libp2p/go-libp2p-peerstore v0.2.0 h1:XcgJhI8WyUOCbHyRLNEX5542YNj8hnLSJ2G1InRjDhk= github.com/libp2p/go-libp2p-peerstore v0.2.0/go.mod h1:N2l3eVIeAitSg3Pi2ipSrJYnqhVnMNQZo9nkSCuAbnQ= github.com/libp2p/go-libp2p-pnet v0.2.0 h1:J6htxttBipJujEjz1y0a5+eYoiPcFHhSYHH6na5f0/k= @@ -219,12 +214,8 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07q github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 h1:5EhPgQhXZNyfL22ERZTUoVp9UVVbNowWNVtELQaKCHk= github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSoMuyojOdjx73Q47cYD7s5+gZOlns= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= -github.com/libp2p/go-libp2p-yamux v0.2.1 h1:Q3XYNiKCC2vIxrvUJL+Jg1kiyeEaIDNKLjgEjo3VQdI= -github.com/libp2p/go-libp2p-yamux v0.2.1/go.mod h1:1FBXiHDk1VyRM1C0aez2bCfHQ4vMZKkAQzZbkSQt5fI= -github.com/libp2p/go-libp2p-yamux v0.2.2 h1:eGvbqWqWY9S5lrpe2gA0UCOLCdzCgYSAR3vo/xCsNQg= -github.com/libp2p/go-libp2p-yamux v0.2.2/go.mod h1:lIohaR0pT6mOt0AZ0L2dFze9hds9Req3OfS+B+dv4qw= -github.com/libp2p/go-libp2p-yamux v0.2.4 h1:SoIR1z7kjjncCPliuchPptd4n/xTxePzWtmR2vIQLwM= -github.com/libp2p/go-libp2p-yamux v0.2.4/go.mod h1:Qq6yW6EfI9vNeL4J8H1l8yCcBoZnJiljJGsbZgfC/08= +github.com/libp2p/go-libp2p-yamux v0.2.5 h1:MuyItOqz03oi8npvjgMJxgnhllJLZnO/dKVOpTZ9+XI= +github.com/libp2p/go-libp2p-yamux v0.2.5/go.mod h1:Zpgj6arbyQrmZ3wxSZxfBmbdnWtbZ48OpsfmQVTErwA= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.5 h1:CW3AgbMO6vUvT4kf87y4N+0P8KUl2aqLYhrGyDUbLSg= github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= @@ -259,12 +250,8 @@ github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfj github.com/libp2p/go-ws-transport v0.2.0 h1:MJCw2OrPA9+76YNRvdo1wMnSOxb9Bivj6sVFY1Xrj6w= github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzlHoKzu0yY9p/klM= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= -github.com/libp2p/go-yamux v1.2.3 h1:xX8A36vpXb59frIzWFdEgptLMsOANMFq2K7fPRlunYI= -github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= -github.com/libp2p/go-yamux v1.3.0 h1:FsYzT16Wq2XqUGJsBbOxoz9g+dFklvNi7jN6YFPfl7U= -github.com/libp2p/go-yamux v1.3.0/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= -github.com/libp2p/go-yamux v1.3.2 h1:9ABGfqAAQB5hlBHWk0bAbGnS4dDaAJLSF0C7TPYia+0= -github.com/libp2p/go-yamux v1.3.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-yamux v1.3.3 h1:mWuzZRCAeTBFdynLlsYgA/EIeMOLr8XY04wa52NRhsE= +github.com/libp2p/go-yamux v1.3.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= @@ -365,6 +352,7 @@ github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFSt github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= @@ -501,6 +489,7 @@ google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiq gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= From ed4646f711b1b3323fcae43e3d236944b47babb7 Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Thu, 19 Mar 2020 19:05:15 +0530 Subject: [PATCH 1741/3965] local addr updated event --- p2p/host/basic/basic_host.go | 56 ++++++++++-- p2p/host/basic/basic_host_test.go | 139 ++++++++++++++++++++++++++++++ p2p/host/relay/autorelay.go | 2 +- 3 files changed, 188 insertions(+), 9 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index f8bf279cff..cae41a9e4e 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -88,6 +88,7 @@ type BasicHost struct { lastAddrs []ma.Multiaddr emitters struct { evtLocalProtocolsUpdated event.Emitter + evtLocalAddrsUpdated event.Emitter } } @@ -141,6 +142,9 @@ func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHo if h.emitters.evtLocalProtocolsUpdated, err = h.eventbus.Emitter(&event.EvtLocalProtocolsUpdated{}); err != nil { return nil, err } + if h.emitters.evtLocalAddrsUpdated, err = h.eventbus.Emitter(&event.EvtLocalAddressesUpdated{}); err != nil { + return nil, err + } h.proc = goprocessctx.WithContextAndTeardown(ctx, func() error { if h.natmgr != nil { @@ -150,6 +154,7 @@ func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHo h.cmgr.Close() } _ = h.emitters.evtLocalProtocolsUpdated.Close() + _ = h.emitters.evtLocalAddrsUpdated.Close() return h.Network().Close() }) @@ -295,24 +300,59 @@ func (h *BasicHost) newStreamHandler(s network.Stream) { go handle(protoID, s) } -// PushIdentify pushes an identify update through the identify push protocol +// CheckForAddressChanges determines whether our listen addresses have recently +// changed and emits an EvtLocalAddressesUpdatedEvent & a Push Identify if so. // Warning: this interface is unstable and may disappear in the future. -func (h *BasicHost) PushIdentify() { - push := false - +func (h *BasicHost) CheckForAddressChanges() { h.mx.Lock() addrs := h.Addrs() - if !sameAddrs(addrs, h.lastAddrs) { - push = true + changeEvt := makeUpdatedAddrEvent(h.lastAddrs, addrs) + if changeEvt != nil { h.lastAddrs = addrs } h.mx.Unlock() - if push { + if changeEvt != nil { + err := h.emitters.evtLocalAddrsUpdated.Emit(*changeEvt) + if err != nil { + log.Warnf("error emitting event for updated addrs: %s", err) + } h.ids.Push() } } +func makeUpdatedAddrEvent(prev, current []ma.Multiaddr) *event.EvtLocalAddressesUpdated { + prevmap := make(map[string]ma.Multiaddr, len(prev)) + evt := event.EvtLocalAddressesUpdated{Diffs: true} + addrsAdded := false + + for _, addr := range prev { + prevmap[string(addr.Bytes())] = addr + } + for _, addr := range current { + _, ok := prevmap[string(addr.Bytes())] + updated := event.UpdatedAddress{Address: addr} + if ok { + updated.Action = event.Maintained + } else { + updated.Action = event.Added + addrsAdded = true + } + evt.Current = append(evt.Current, updated) + delete(prevmap, string(addr.Bytes())) + } + for _, addr := range prevmap { + updated := event.UpdatedAddress{Action: event.Removed, Address: addr} + evt.Removed = append(evt.Removed, updated) + } + + if !addrsAdded && len(evt.Removed) == 0 { + return nil + } + + return &evt +} + func (h *BasicHost) background(p goprocess.Process) { // periodically schedules an IdentifyPush to update our peers for changes // in our address set (if needed) @@ -329,7 +369,7 @@ func (h *BasicHost) background(p goprocess.Process) { for { select { case <-ticker.C: - h.PushIdentify() + h.CheckForAddressChanges() case <-p.Closing(): return diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index ed0370b8a4..dd3593a894 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -507,6 +507,145 @@ func TestAddrResolutionRecursive(t *testing.T) { } } +func TestHostAddrChangeDetection(t *testing.T) { + // This test uses the address factory to provide several + // sets of listen addresses for the host. It advances through + // the sets by changing the currentAddrSet index var below. + addrSets := [][]ma.Multiaddr{ + {}, + {ma.StringCast("/ip4/1.2.3.4/tcp/1234")}, + {ma.StringCast("/ip4/1.2.3.4/tcp/1234"), ma.StringCast("/ip4/2.3.4.5/tcp/1234")}, + {ma.StringCast("/ip4/2.3.4.5/tcp/1234"), ma.StringCast("/ip4/3.4.5.6/tcp/4321")}, + } + + // The events we expect the host to emit when CheckForAddressChanges is called + // and the changes between addr sets are detected + expectedEvents := []event.EvtLocalAddressesUpdated{ + { + Diffs: true, + Current: []event.UpdatedAddress{ + {Action: event.Added, Address: ma.StringCast("/ip4/1.2.3.4/tcp/1234")}, + }, + Removed: []event.UpdatedAddress{}, + }, + { + Diffs: true, + Current: []event.UpdatedAddress{ + {Action: event.Maintained, Address: ma.StringCast("/ip4/1.2.3.4/tcp/1234")}, + {Action: event.Added, Address: ma.StringCast("/ip4/2.3.4.5/tcp/1234")}, + }, + Removed: []event.UpdatedAddress{}, + }, + { + Diffs: true, + Current: []event.UpdatedAddress{ + {Action: event.Added, Address: ma.StringCast("/ip4/3.4.5.6/tcp/4321")}, + {Action: event.Maintained, Address: ma.StringCast("/ip4/2.3.4.5/tcp/1234")}, + }, + Removed: []event.UpdatedAddress{ + {Action: event.Removed, Address: ma.StringCast("/ip4/1.2.3.4/tcp/1234")}, + }, + }, + } + + currentAddrSet := 0 + addrsFactory := func(addrs []ma.Multiaddr) []ma.Multiaddr { + return addrSets[currentAddrSet] + } + + ctx := context.Background() + h := New(swarmt.GenSwarm(t, ctx), AddrsFactory(addrsFactory)) + defer h.Close() + + sub, err := h.EventBus().Subscribe(&event.EvtLocalAddressesUpdated{}, eventbus.BufSize(10)) + if err != nil { + t.Error(err) + } + defer sub.Close() + + // host should start with no addrs (addrSet 0) + addrs := h.Addrs() + if len(addrs) != 0 { + t.Fatalf("expected 0 addrs, got %d", len(addrs)) + } + + // Advance between addrSets + for i := 1; i < len(addrSets); i++ { + currentAddrSet = i + h.CheckForAddressChanges() // forces the host to check for changes now, instead of waiting for background update + } + + // drain events from the subscription + var receivedEvents []event.EvtLocalAddressesUpdated +readEvents: + for { + select { + case evt, more := <-sub.Out(): + if !more { + break readEvents + } + receivedEvents = append(receivedEvents, evt.(event.EvtLocalAddressesUpdated)) + if len(receivedEvents) == len(expectedEvents) { + break readEvents + } + case <-ctx.Done(): + break readEvents + case <-time.After(1 * time.Second): + break readEvents + } + } + + // assert that we received the events we expected + if len(receivedEvents) != len(expectedEvents) { + t.Errorf("expected to receive %d addr change events, got %d", len(expectedEvents), len(receivedEvents)) + } + for i, expected := range expectedEvents { + actual := receivedEvents[i] + if !updatedAddrEventsEqual(expected, actual) { + t.Errorf("change events not equal: \n\texpected: %v \n\tactual: %v", expected, actual) + } + } +} + +// updatedAddrsEqual is a helper to check whether two lists of +// event.UpdatedAddress have the same contents, ignoring ordering. +func updatedAddrsEqual(a, b []event.UpdatedAddress) bool { + if len(a) != len(b) { + return false + } + + // We can't use an UpdatedAddress directly as a map key, since + // Multiaddr is an interface, and go won't know how to compare + // for equality. So we convert to this little struct, which + // stores the multiaddr as a string. + type ua struct { + action event.AddrAction + addrStr string + } + aSet := make(map[ua]struct{}) + for _, addr := range a { + k := ua{action: addr.Action, addrStr: string(addr.Address.Bytes())} + aSet[k] = struct{}{} + } + for _, addr := range b { + k := ua{action: addr.Action, addrStr: string(addr.Address.Bytes())} + _, ok := aSet[k] + if !ok { + return false + } + } + return true +} + +// updatedAddrEventsEqual is a helper to check whether two +// event.EvtLocalAddressesUpdated are equal, ignoring the ordering of +// addresses in the inner lists. +func updatedAddrEventsEqual(a, b event.EvtLocalAddressesUpdated) bool { + return a.Diffs == b.Diffs && + updatedAddrsEqual(a.Current, b.Current) && + updatedAddrsEqual(a.Removed, b.Removed) +} + type sortedMultiaddrs []ma.Multiaddr func (sma sortedMultiaddrs) Len() int { return len(sma) } diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 0c3f40b53f..484c7dbb9f 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -125,7 +125,7 @@ func (ar *AutoRelay) background(ctx context.Context) { ar.cachedAddrs = nil ar.mx.Unlock() push = false - ar.host.PushIdentify() + ar.host.CheckForAddressChanges() } select { From 3ec0ac08503ce0efc6cdb0cae8613c9abd40f71a Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 19 Mar 2020 10:52:39 -0700 Subject: [PATCH 1742/3965] simplify service enable/disable interface --- p2p/host/autonat/autonat.go | 15 +++++---------- p2p/host/autonat/svc.go | 35 +++++++++++++++++++++++++---------- 2 files changed, 30 insertions(+), 20 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index f0bde8d407..6b03170db3 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -42,7 +42,6 @@ type AmbientAutoNAT struct { subAddrUpdated event.Subscription service *autoNATService - serviceCancel context.CancelFunc emitReachabilityChanged event.Emitter } @@ -245,9 +244,7 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { // we are flipping our NATStatus, so confidence drops to 0 as.confidence = 0 if as.service != nil { - ctx, cancel := context.WithCancel(as.ctx) - go as.service.Enable(ctx) - as.serviceCancel = cancel + as.service.Enable() } changed = true } else if as.confidence < 3 { @@ -274,9 +271,8 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { // we are flipping our NATStatus, so confidence drops to 0 as.confidence = 0 as.status.Store(observation) - if as.serviceCancel != nil { - as.serviceCancel() - as.serviceCancel = nil + if as.service != nil { + as.service.Disable() } as.emitStatus() } @@ -294,9 +290,8 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { log.Debugf("NAT status is unknown") as.status.Store(autoNATResult{network.ReachabilityUnknown, nil}) if currentStatus.Reachability != network.ReachabilityUnknown { - if as.serviceCancel != nil { - as.serviceCancel() - as.serviceCancel = nil + if as.service != nil { + as.service.Disable() } as.emitStatus() } diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 41afafd56d..8d88252ee2 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -6,7 +6,6 @@ import ( "math/rand" "net" "sync" - "sync/atomic" "time" "github.com/libp2p/go-libp2p-core/helpers" @@ -25,7 +24,9 @@ const P_CIRCUIT = 290 // AutoNATService provides NAT autodetection services to other peers type autoNATService struct { - ctx context.Context + ctx context.Context + instance context.CancelFunc + instanceLock sync.Mutex config *config @@ -49,7 +50,7 @@ func newAutoNATService(ctx context.Context, c *config) (*autoNATService, error) } if c.forceReachability { - go as.Enable(ctx) + as.Enable() } return as, nil @@ -212,16 +213,30 @@ func (as *autoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { return newDialResponseOK(ra) } -// Enable the autoNAT service temporarily until the associated context is canceled. -func (as *autoNATService) Enable(ctx context.Context) { - alreadyRunning := atomic.SwapUint32(&as.running, 1) - if alreadyRunning > 0 { +// Enable the autoNAT service if it is not running. +func (as *autoNATService) Enable() { + as.instanceLock.Lock() + defer as.instanceLock.Unlock() + if as.instance != nil { return } - defer func() { - atomic.StoreUint32(&as.running, 0) - }() + inst, cncl := context.WithCancel(as.ctx) + as.instance = cncl + go as.background(inst) +} + +// Disable the autoNAT service if it is running. +func (as *autoNATService) Disable() { + as.instanceLock.Lock() + defer as.instanceLock.Unlock() + if as.instance != nil { + as.instance() + as.instance = nil + } +} + +func (as *autoNATService) background(ctx context.Context) { as.config.host.SetStreamHandler(AutoNATProto, as.handleStream) timer := time.NewTimer(as.config.throttleResetPeriod) From 42a3e6f451584ca8bae7268473e5290f4aebf4aa Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 19 Mar 2020 10:56:55 -0700 Subject: [PATCH 1743/3965] switch enablement to not happen in svc constructor --- p2p/host/autonat/autonat.go | 5 +++++ p2p/host/autonat/svc.go | 4 ---- p2p/host/autonat/svc_test.go | 1 + 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 6b03170db3..f0c98d0645 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -88,6 +88,11 @@ func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { if conf.forceReachability { emitReachabilityChanged.Emit(event.EvtLocalReachabilityChanged{Reachability: conf.reachability}) + + // The serice will only exist when reachability is public. + if service != nil { + service.Enable() + } return &StaticAutoNAT{ ctx: ctx, host: h, diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index 8d88252ee2..ed8fadcdce 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -49,10 +49,6 @@ func newAutoNATService(ctx context.Context, c *config) (*autoNATService, error) reqs: make(map[peer.ID]int), } - if c.forceReachability { - as.Enable() - } - return as, nil } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index f38d4041a6..749c7b9343 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -30,6 +30,7 @@ func makeAutoNATService(ctx context.Context, t *testing.T, c *config) *autoNATSe if err != nil { t.Fatal(err) } + as.Enable() return as } From 685ba00833c3436be748c60962331582b15192f7 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 19 Mar 2020 13:27:48 -0700 Subject: [PATCH 1744/3965] update to current autonat interface --- config/config.go | 42 ++++++++++++++++++++----------------- go.mod | 2 +- go.sum | 2 ++ options.go | 23 ++++++++++++++++---- p2p/host/relay/autorelay.go | 4 +--- 5 files changed, 46 insertions(+), 27 deletions(-) diff --git a/config/config.go b/config/config.go index 380086db47..b18c294b9c 100644 --- a/config/config.go +++ b/config/config.go @@ -41,6 +41,8 @@ type NATManagerC func(network.Network) bhost.NATManager type RoutingC func(host.Host) (routing.PeerRouting, error) +// AutoNATMode defines the AutoNAT behavior for the libp2p host. + // Config describes a set of settings for a libp2p node // // This is *not* a stable interface. Use the options defined in the root @@ -78,7 +80,8 @@ type Config struct { Routing RoutingC EnableAutoRelay bool - EnableAutoNAT bool + Reachability network.Reachability + AutoNATService bool StaticRelays []peer.AddrInfo } @@ -212,7 +215,6 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } } - hop := false // Note: h.AddrsFactory may be changed by AutoRelay, but non-relay version is // used by AutoNAT below. addrF := h.AddrsFactory @@ -222,6 +224,7 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return nil, fmt.Errorf("cannot enable autorelay; relay is not enabled") } + hop := false for _, opt := range cfg.RelayOpts { if opt == circuit.OptHop { hop = true @@ -254,25 +257,26 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } } - if cfg.EnableAutoRelay || cfg.EnableAutoNAT { - autonatOpts := []autonat.Option{autonat.UsingAddresses(func() []ma.Multiaddr { - return addrF(h.AllAddrs()) - })} - if hop { - dialerStore := pstoremem.NewPeerstore() - dialerHost, err := cfg.newHost(ctx, dialerStore) - if err != nil { - h.Close() - return nil, err - } - - autonatOpts = append(autonatOpts, autonat.EnableService(dialerHost.Network(), false)) - } - - if _, err = autonat.New(ctx, h, autonatOpts...); err != nil { + autonatOpts := []autonat.Option{autonat.UsingAddresses(func() []ma.Multiaddr { + return addrF(h.AllAddrs()) + })} + if cfg.AutoNATService { + dialerStore := pstoremem.NewPeerstore() + dialerHost, err := cfg.newHost(ctx, dialerStore) + if err != nil { h.Close() - return nil, fmt.Errorf("cannot enable autorelay; autonat failed to start: %v", err) + return nil, err } + + autonatOpts = append(autonatOpts, autonat.EnableService(dialerHost.Network())) + } + if cfg.Reachability != network.ReachabilityUnknown { + autonatOpts = append(autonatOpts, autonat.WithReachability(cfg.Reachability)) + } + + if _, err = autonat.New(ctx, h, autonatOpts...); err != nil { + h.Close() + return nil, fmt.Errorf("cannot enable autorelay; autonat failed to start: %v", err) } // start the host background tasks diff --git a/go.mod b/go.mod index a8ed941a9f..eb4daa0cfb 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/jbenet/goprocess v0.1.3 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 - github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200317183318-4b2cc5830d44 + github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200319175655-901286ad641d github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 github.com/libp2p/go-libp2p-core v0.5.0 diff --git a/go.sum b/go.sum index f76c2aa36f..302d3ba406 100644 --- a/go.sum +++ b/go.sum @@ -155,6 +155,8 @@ github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200316221154-f1cd6ba0fd67 h1:3UM/ github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200316221154-f1cd6ba0fd67/go.mod h1:yoF5B6yi+Wjxci55xdLlJo7gfyHwG80df4B9hiO2A88= github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200317183318-4b2cc5830d44 h1:xfmItLrKVdinHJJouT4ocZFWK+Y0WaEONku6aYN92+s= github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200317183318-4b2cc5830d44/go.mod h1:yoF5B6yi+Wjxci55xdLlJo7gfyHwG80df4B9hiO2A88= +github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200319175655-901286ad641d h1:iA6ZxnrWVFW4Q4QlmKYctSapkGKbcbalnHygPVRCCfc= +github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200319175655-901286ad641d/go.mod h1:yoF5B6yi+Wjxci55xdLlJo7gfyHwG80df4B9hiO2A88= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= diff --git a/options.go b/options.go index 12b34ad6e1..107e38db46 100644 --- a/options.go +++ b/options.go @@ -10,6 +10,7 @@ import ( "github.com/libp2p/go-libp2p-core/connmgr" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/metrics" + "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/pnet" @@ -278,11 +279,25 @@ func DefaultStaticRelays() Option { } } -// EnableAutoNAT configures libp2p to monitor its NAT status and track when -// the local host is externally reachable. -func EnableAutoNAT() Option { +// WithReachability overrides automatic reachability detection to force the local node +// to believe it is either unreachable or reachable externally. +func WithReachability(reachable bool) Option { return func(cfg *Config) error { - cfg.EnableAutoNAT = true + if reachable { + cfg.Reachability = network.ReachabilityPublic + } else { + cfg.Reachability = network.ReachabilityPrivate + } + return nil + } +} + +// EnableNATService configures libp2p to provide a service to peers for determining +// their reachability status. When enabled, the host will attempt to dial back +// to peers, and then tell them if it was successful in making such connections. +func EnableNATService() Option { + return func(cfg *Config) error { + cfg.AutoNATService = true return nil } } diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index e5187109d1..3dc096b772 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -103,9 +103,7 @@ func (ar *AutoRelay) background(ctx context.Context) { } ar.mx.Lock() - if ar.status != evt.Reachability && evt.Reachability != network.ReachabilityUnknown { - push = true - } else if update { + if update || (ar.status != evt.Reachability && evt.Reachability != network.ReachabilityUnknown) { push = true } ar.status = evt.Reachability From d49806efc92fdc0f996c047d56f0652e3ef3966e Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 20 Mar 2020 10:04:13 +0530 Subject: [PATCH 1745/3965] emit address change evt --- p2p/host/basic/basic_host.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index cae41a9e4e..bce9e580e8 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -356,7 +356,7 @@ func makeUpdatedAddrEvent(prev, current []ma.Multiaddr) *event.EvtLocalAddresses func (h *BasicHost) background(p goprocess.Process) { // periodically schedules an IdentifyPush to update our peers for changes // in our address set (if needed) - ticker := time.NewTicker(1 * time.Minute) + ticker := time.NewTicker(10 * time.Second) defer ticker.Stop() // initialize lastAddrs From 891bba6566ec41e8964df938ec354d2a872d1580 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 22 Mar 2020 16:00:16 -0700 Subject: [PATCH 1746/3965] fix: minimal autonat dialer Currently, the AutoNAT dialer spins up a full host with identify, ping, etc. this switches to a blankhost. This also avoids running a relay, listening on transports, etc (regression). --- config/config.go | 136 +++++++++++++++++++++++++++++++---------------- 1 file changed, 90 insertions(+), 46 deletions(-) diff --git a/config/config.go b/config/config.go index b18c294b9c..bd843cac5d 100644 --- a/config/config.go +++ b/config/config.go @@ -13,6 +13,7 @@ import ( "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/pnet" "github.com/libp2p/go-libp2p-core/routing" + "github.com/libp2p/go-libp2p-core/transport" "github.com/libp2p/go-libp2p-peerstore/pstoremem" bhost "github.com/libp2p/go-libp2p/p2p/host/basic" @@ -28,6 +29,8 @@ import ( logging "github.com/ipfs/go-log" filter "github.com/libp2p/go-maddr-filter" ma "github.com/multiformats/go-multiaddr" + + blankhost "github.com/libp2p/go-libp2p-blankhost" ) var log = logging.Logger("p2p-config") @@ -85,7 +88,11 @@ type Config struct { StaticRelays []peer.AddrInfo } -func (cfg *Config) newHost(ctx context.Context, store peerstore.Peerstore) (*bhost.BasicHost, error) { +func (cfg *Config) makeSwarm(ctx context.Context) (*swarm.Swarm, error) { + if cfg.Peerstore == nil { + return nil, fmt.Errorf("no peerstore specified") + } + // Check this early. Prevents us from even *starting* without verifying this. if pnet.ForcePrivateNetwork && len(cfg.PSK) == 0 { log.Error("tried to create a libp2p node with no Private" + @@ -106,95 +113,110 @@ func (cfg *Config) newHost(ctx context.Context, store peerstore.Peerstore) (*bho return nil, err } - if err := store.AddPrivKey(pid, cfg.PeerKey); err != nil { + if err := cfg.Peerstore.AddPrivKey(pid, cfg.PeerKey); err != nil { return nil, err } - if err := store.AddPubKey(pid, cfg.PeerKey.GetPublic()); err != nil { + if err := cfg.Peerstore.AddPubKey(pid, cfg.PeerKey.GetPublic()); err != nil { return nil, err } // TODO: Make the swarm implementation configurable. - swrm := swarm.NewSwarm(ctx, pid, store, cfg.Reporter) + swrm := swarm.NewSwarm(ctx, pid, cfg.Peerstore, cfg.Reporter) if cfg.Filters != nil { swrm.Filters = cfg.Filters } + return swrm, nil +} - h, err := bhost.NewHost(ctx, swrm, &bhost.HostOpts{ - ConnManager: cfg.ConnManager, - AddrsFactory: cfg.AddrsFactory, - NATManager: cfg.NATManager, - EnablePing: !cfg.DisablePing, - UserAgent: cfg.UserAgent, - }) - - if err != nil { - swrm.Close() - return nil, err +func (cfg *Config) addTransports(ctx context.Context, h host.Host) (err error) { + swrm, ok := h.Network().(transport.TransportNetwork) + if !ok { + // Should probably skip this if no transports. + return fmt.Errorf("swarm does not support transports") } - - if cfg.Relay { - // If we've enabled the relay, we should filter out relay - // addresses by default. - // - // TODO: We shouldn't be doing this here. - oldFactory := h.AddrsFactory - h.AddrsFactory = func(addrs []ma.Multiaddr) []ma.Multiaddr { - return oldFactory(relay.Filter(addrs)) - } - } - upgrader := new(tptu.Upgrader) upgrader.PSK = cfg.PSK - upgrader.Filters = swrm.Filters + upgrader.Filters = cfg.Filters if cfg.Insecure { - upgrader.Secure = makeInsecureTransport(pid, cfg.PeerKey) + upgrader.Secure = makeInsecureTransport(h.ID(), cfg.PeerKey) } else { upgrader.Secure, err = makeSecurityTransport(h, cfg.SecurityTransports) if err != nil { - h.Close() - return nil, err + return err } } upgrader.Muxer, err = makeMuxer(h, cfg.Muxers) if err != nil { - h.Close() - return nil, err + return err } tpts, err := makeTransports(h, upgrader, cfg.Transports) if err != nil { - h.Close() - return nil, err + return err } for _, t := range tpts { err = swrm.AddTransport(t) if err != nil { - h.Close() - return nil, err + return err } } if cfg.Relay { - err := circuit.AddRelayTransport(swrm.Context(), h, upgrader, cfg.RelayOpts...) + err := circuit.AddRelayTransport(ctx, h, upgrader, cfg.RelayOpts...) if err != nil { h.Close() - return nil, err + return err } } - return h, nil + + return nil } // NewNode constructs a new libp2p Host from the Config. // // This function consumes the config. Do not reuse it (really!). func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { - if cfg.Peerstore == nil { - return nil, fmt.Errorf("no peerstore specified") + swrm, err := cfg.makeSwarm(ctx) + if err != nil { + return nil, err + } + + h, err := bhost.NewHost(ctx, swrm, &bhost.HostOpts{ + ConnManager: cfg.ConnManager, + AddrsFactory: cfg.AddrsFactory, + NATManager: cfg.NATManager, + EnablePing: !cfg.DisablePing, + UserAgent: cfg.UserAgent, + }) + + if err != nil { + swrm.Close() + return nil, err } - h, err := cfg.newHost(ctx, cfg.Peerstore) + // XXX: This is the only sane way to get a context out that's guaranteed + // to be canceled when we shut down. + // + // TODO: Stop using contexts to stop services. This is just lazy. + // Contexts are for canceling requests, services should be managed + // explicitly. + ctx = swrm.Context() + + if cfg.Relay { + // If we've enabled the relay, we should filter out relay + // addresses by default. + // + // TODO: We shouldn't be doing this here. + oldFactory := h.AddrsFactory + h.AddrsFactory = func(addrs []ma.Multiaddr) []ma.Multiaddr { + return oldFactory(relay.Filter(addrs)) + } + } + + err = cfg.addTransports(ctx, h) if err != nil { + h.Close() return nil, err } @@ -261,13 +283,35 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return addrF(h.AllAddrs()) })} if cfg.AutoNATService { - dialerStore := pstoremem.NewPeerstore() - dialerHost, err := cfg.newHost(ctx, dialerStore) + // Pull out the pieces of the config that we _actually_ care about. + // Specifically, don't setup things like autorelay, listeners, + // identify, etc. + autoNatCfg := Config{ + Transports: cfg.Transports, + Muxers: cfg.Muxers, + SecurityTransports: cfg.SecurityTransports, + Insecure: cfg.Insecure, + PSK: cfg.PSK, + Filters: cfg.Filters, + Reporter: cfg.Reporter, + + Peerstore: pstoremem.NewPeerstore(), + } + dialer, err := autoNatCfg.makeSwarm(ctx) if err != nil { h.Close() return nil, err } - + dialerHost := blankhost.NewBlankHost(dialer) + err = autoNatCfg.addTransports(ctx, dialerHost) + if err != nil { + dialerHost.Close() + h.Close() + return nil, err + } + // NOTE: We're dropping the blank host here but that's fine. It + // doesn't really _do_ anything and doesn't even need to be + // closed (as long as we close the underlying network). autonatOpts = append(autonatOpts, autonat.EnableService(dialerHost.Network())) } if cfg.Reachability != network.ReachabilityUnknown { From 5f2357c2c15912632dd752d78435dd51ca3533d6 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Sat, 14 Mar 2020 11:38:25 +0700 Subject: [PATCH 1747/3965] use a stateless reset key derived from the private key --- p2p/transport/quic/conn_test.go | 56 ++++++++++++++++++++++++++++++--- p2p/transport/quic/listener.go | 2 +- p2p/transport/quic/transport.go | 18 ++++++++++- 3 files changed, 70 insertions(+), 6 deletions(-) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index f7d7253c9c..d70af9f6e4 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -8,12 +8,14 @@ import ( "io/ioutil" mrand "math/rand" "net" + "sync/atomic" "time" ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" tpt "github.com/libp2p/go-libp2p-core/transport" filter "github.com/libp2p/go-maddr-filter" + quicproxy "github.com/lucas-clemente/quic-go/integrationtests/tools/proxy" ma "github.com/multiformats/go-multiaddr" . "github.com/onsi/ginkgo" @@ -52,9 +54,9 @@ var _ = Describe("Connection", func() { runServer := func(tr tpt.Transport, multiaddr string) tpt.Listener { addr, err := ma.NewMultiaddr(multiaddr) - Expect(err).ToNot(HaveOccurred()) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) ln, err := tr.Listen(addr) - Expect(err).ToNot(HaveOccurred()) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) return ln } @@ -183,13 +185,13 @@ var _ = Describe("Connection", func() { Expect(err).ToNot(HaveOccurred()) // make sure that connection attempts fails - quicConfig.HandshakeTimeout = 250 * time.Millisecond + clientTransport.(*transport).config.HandshakeTimeout = 250 * time.Millisecond _, err = clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).To(HaveOccurred()) Expect(err.(net.Error).Timeout()).To(BeTrue()) // now allow the address and make sure the connection goes through - quicConfig.HandshakeTimeout = 2 * time.Second + clientTransport.(*transport).config.HandshakeTimeout = 2 * time.Second filters.AddFilter(ipNet, filter.ActionAccept) conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) @@ -256,4 +258,50 @@ var _ = Describe("Connection", func() { Eventually(done, 5*time.Second).Should(Receive()) Eventually(done, 5*time.Second).Should(Receive()) }) + + It("sends stateless resets", func() { + serverTransport, err := NewTransport(serverKey, nil, nil) + Expect(err).ToNot(HaveOccurred()) + ln := runServer(serverTransport, "/ip4/127.0.0.1/udp/0/quic") + + var drop uint32 + serverPort := ln.Addr().(*net.UDPAddr).Port + proxy, err := quicproxy.NewQuicProxy("localhost:0", &quicproxy.Opts{ + RemoteAddr: fmt.Sprintf("localhost:%d", serverPort), + DropPacket: func(quicproxy.Direction, []byte) bool { + return atomic.LoadUint32(&drop) > 0 + }, + }) + Expect(err).ToNot(HaveOccurred()) + defer proxy.Close() + + // establish a connection + clientTransport, err := NewTransport(clientKey, nil, nil) + Expect(err).ToNot(HaveOccurred()) + proxyAddr, err := toQuicMultiaddr(proxy.LocalAddr()) + Expect(err).ToNot(HaveOccurred()) + conn, err := clientTransport.Dial(context.Background(), proxyAddr, serverID) + Expect(err).ToNot(HaveOccurred()) + str, err := conn.OpenStream() + Expect(err).ToNot(HaveOccurred()) + + // Stop forwarding packets and close the server. + // This prevents the CONNECTION_CLOSE from reaching the client. + atomic.StoreUint32(&drop, 1) + Expect(ln.Close()).To(Succeed()) + time.Sleep(100 * time.Millisecond) // give the kernel some time to free the UDP port + ln = runServer(serverTransport, fmt.Sprintf("/ip4/127.0.0.1/udp/%d/quic", serverPort)) + defer ln.Close() + // Now that the new server is up, re-enable packet forwarding. + atomic.StoreUint32(&drop, 0) + + // The new server doesn't have any state for the previously established connection. + // We expect it to send a stateless reset. + _, rerr := str.Write([]byte("foobar")) + if rerr == nil { + _, rerr = str.Read([]byte{0, 0}) + } + Expect(rerr).To(HaveOccurred()) + Expect(rerr.Error()).To(ContainSubstring("received a stateless reset")) + }) }) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index ed2fb731c3..422f7be96c 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -36,7 +36,7 @@ func newListener(rconn *reuseConn, t *transport, localPeer peer.ID, key ic.PrivK conf, _ := identity.ConfigForAny() return conf, nil } - ln, err := quic.Listen(rconn, &tlsConf, quicConfig) + ln, err := quic.Listen(rconn, &tlsConf, t.config) if err != nil { return nil, err } diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index a60b28f65a..534984abd4 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -2,9 +2,13 @@ package libp2pquic import ( "context" + "crypto/sha256" "errors" + "io" "net" + "golang.org/x/crypto/hkdf" + logging "github.com/ipfs/go-log" ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/peer" @@ -85,6 +89,7 @@ type transport struct { localPeer peer.ID identity *p2ptls.Identity connManager *connManager + config *quic.Config } var _ tpt.Transport = &transport{} @@ -107,12 +112,23 @@ func NewTransport(key ic.PrivKey, psk pnet.PSK, filters *filter.Filters) (tpt.Tr if err != nil { return nil, err } + config := quicConfig.Clone() + keyBytes, err := key.Raw() + if err != nil { + return nil, err + } + keyReader := hkdf.Expand(sha256.New, keyBytes, []byte("libp2p quic stateless reset key")) + config.StatelessResetKey = make([]byte, 32) + if _, err := io.ReadFull(keyReader, config.StatelessResetKey); err != nil { + return nil, err + } return &transport{ privKey: key, localPeer: localPeer, identity: identity, connManager: connManager, + config: config, }, nil } @@ -135,7 +151,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - sess, err := quic.DialContext(ctx, pconn, addr, host, tlsConf, quicConfig) + sess, err := quic.DialContext(ctx, pconn, addr, host, tlsConf, t.config) if err != nil { pconn.DecreaseCount() return nil, err From 805048ad5378e310a08a063a465bfb90366e9e1f Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 23 Mar 2020 14:10:16 +0700 Subject: [PATCH 1748/3965] use hkdf.New instead of hkdf.Expand for stateless reset key derivation --- p2p/transport/quic/transport.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 534984abd4..55e7a1bf8d 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -117,7 +117,7 @@ func NewTransport(key ic.PrivKey, psk pnet.PSK, filters *filter.Filters) (tpt.Tr if err != nil { return nil, err } - keyReader := hkdf.Expand(sha256.New, keyBytes, []byte("libp2p quic stateless reset key")) + keyReader := hkdf.New(sha256.New, keyBytes, nil, []byte("libp2p quic stateless reset key")) config.StatelessResetKey = make([]byte, 32) if _, err := io.ReadFull(keyReader, config.StatelessResetKey); err != nil { return nil, err From 2559e4ba47cf17eec7f6227f36679a4ba3a8cacc Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 23 Mar 2020 14:17:28 +0700 Subject: [PATCH 1749/3965] use minio/sha256-simd instead of standard library SHA256 --- p2p/transport/quic/transport.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 55e7a1bf8d..7b3484b203 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -2,11 +2,11 @@ package libp2pquic import ( "context" - "crypto/sha256" "errors" "io" "net" + "github.com/minio/sha256-simd" "golang.org/x/crypto/hkdf" logging "github.com/ipfs/go-log" From 6ca3f72b62c8e0b6d2a2e3354e313687e2c4181e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 23 Mar 2020 13:59:56 -0700 Subject: [PATCH 1750/3965] fix: make close work again Reference pstoremem components by pointer to: 1. Avoid copying locks around on construction. 2. Avoid copying these around when calling `weakClose`. 3. Ensures that they all implement the `io.Closer` interface (it's implemented on the pointer, not the value). Technically, we could have just taken a reference when calling `weakClose`, but this is cleaner. --- p2p/host/peerstore/pstoremem/peerstore.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/peerstore.go b/p2p/host/peerstore/pstoremem/peerstore.go index f16887de22..113b1d7261 100644 --- a/p2p/host/peerstore/pstoremem/peerstore.go +++ b/p2p/host/peerstore/pstoremem/peerstore.go @@ -11,20 +11,20 @@ import ( type pstoremem struct { peerstore.Metrics - memoryKeyBook - memoryAddrBook - memoryProtoBook - memoryPeerMetadata + *memoryKeyBook + *memoryAddrBook + *memoryProtoBook + *memoryPeerMetadata } // NewPeerstore creates an in-memory threadsafe collection of peers. func NewPeerstore() *pstoremem { return &pstoremem{ Metrics: pstore.NewMetrics(), - memoryKeyBook: *NewKeyBook(), - memoryAddrBook: *NewAddrBook(), - memoryProtoBook: *NewProtoBook(), - memoryPeerMetadata: *NewPeerMetadata(), + memoryKeyBook: NewKeyBook(), + memoryAddrBook: NewAddrBook(), + memoryProtoBook: NewProtoBook(), + memoryPeerMetadata: NewPeerMetadata(), } } From 24eeef583316876b8a8afe82a98755c4b1b71a9b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 23 Mar 2020 14:16:16 -0700 Subject: [PATCH 1751/3965] test: check for goroutine leaks when closing the peerstore --- p2p/host/peerstore/pstoremem/inmem_test.go | 33 ++++++++++++++++++---- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/inmem_test.go b/p2p/host/peerstore/pstoremem/inmem_test.go index 4e34756349..7491a9339c 100644 --- a/p2p/host/peerstore/pstoremem/inmem_test.go +++ b/p2p/host/peerstore/pstoremem/inmem_test.go @@ -5,34 +5,57 @@ import ( pstore "github.com/libp2p/go-libp2p-core/peerstore" pt "github.com/libp2p/go-libp2p-peerstore/test" + + "go.uber.org/goleak" ) +func TestFuzzInMemoryPeerstore(t *testing.T) { + // Just create and close a bunch of peerstores. If this leaks, we'll + // catch it in the leak check below. + for i := 0; i < 100; i++ { + ps := NewPeerstore() + ps.Close() + } +} + func TestInMemoryPeerstore(t *testing.T) { pt.TestPeerstore(t, func() (pstore.Peerstore, func()) { - return NewPeerstore(), nil + ps := NewPeerstore() + return ps, func() { ps.Close() } }) } func TestInMemoryAddrBook(t *testing.T) { pt.TestAddrBook(t, func() (pstore.AddrBook, func()) { - return NewAddrBook(), nil + ps := NewPeerstore() + return ps, func() { ps.Close() } }) } func TestInMemoryKeyBook(t *testing.T) { pt.TestKeyBook(t, func() (pstore.KeyBook, func()) { - return NewKeyBook(), nil + ps := NewPeerstore() + return ps, func() { ps.Close() } }) } func BenchmarkInMemoryPeerstore(b *testing.B) { pt.BenchmarkPeerstore(b, func() (pstore.Peerstore, func()) { - return NewPeerstore(), nil + ps := NewPeerstore() + return ps, func() { ps.Close() } }, "InMem") } func BenchmarkInMemoryKeyBook(b *testing.B) { pt.BenchmarkKeyBook(b, func() (pstore.KeyBook, func()) { - return NewKeyBook(), nil + ps := NewPeerstore() + return ps, func() { ps.Close() } }) } + +func TestMain(m *testing.M) { + goleak.VerifyTestMain( + m, + goleak.IgnoreTopFunction("github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine"), + ) +} From 22232e0d6d8511368e1170ddba1362b9f66a6f2c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 23 Mar 2020 14:25:38 -0700 Subject: [PATCH 1752/3965] fix(mock): fire disconnect notifications async like we do in the swarm --- p2p/net/mock/mock_conn.go | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/p2p/net/mock/mock_conn.go b/p2p/net/mock/mock_conn.go index 882522b2ea..d0c28c628b 100644 --- a/p2p/net/mock/mock_conn.go +++ b/p2p/net/mock/mock_conn.go @@ -62,11 +62,13 @@ func (c *conn) teardown() error { } c.net.removeConn(c) - c.notifLk.Lock() - defer c.notifLk.Unlock() - c.net.notifyAll(func(n network.Notifiee) { - n.Disconnected(c.net, c) - }) + go func() { + c.notifLk.Lock() + defer c.notifLk.Unlock() + c.net.notifyAll(func(n network.Notifiee) { + n.Disconnected(c.net, c) + }) + }() return nil } @@ -92,11 +94,13 @@ func (c *conn) removeStream(s *stream) { } c.Unlock() - s.notifLk.Lock() - defer s.notifLk.Unlock() - s.conn.net.notifyAll(func(n network.Notifiee) { - n.ClosedStream(s.conn.net, s) - }) + go func() { + s.notifLk.Lock() + defer s.notifLk.Unlock() + s.conn.net.notifyAll(func(n network.Notifiee) { + n.ClosedStream(s.conn.net, s) + }) + }() } func (c *conn) allStreams() []network.Stream { From 223cd4225131293b8c7d7582b35d636a5566c651 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 23 Mar 2020 14:26:16 -0700 Subject: [PATCH 1753/3965] fix(mock): wait till we add the connection to set it up Otherwise, we can try to remove it before we add it and panic. --- p2p/net/mock/mock_conn.go | 14 +++++++++----- p2p/net/mock/mock_peernet.go | 4 ++++ 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/p2p/net/mock/mock_conn.go b/p2p/net/mock/mock_conn.go index d0c28c628b..495b2b24a4 100644 --- a/p2p/net/mock/mock_conn.go +++ b/p2p/net/mock/mock_conn.go @@ -30,14 +30,15 @@ type conn struct { link *link rconn *conn // counterpart streams list.List - proc process.Process stat network.Stat + pairProc, connProc process.Process + sync.RWMutex } func newConn(p process.Process, ln, rn *peernet, l *link, dir network.Direction) *conn { - c := &conn{net: ln, link: l, proc: p} + c := &conn{net: ln, link: l, pairProc: p} c.local = ln.peer c.remote = rn.peer c.stat = network.Stat{Direction: dir} @@ -47,13 +48,16 @@ func newConn(p process.Process, ln, rn *peernet, l *link, dir network.Direction) c.localPrivKey = ln.ps.PrivKey(ln.peer) c.remotePubKey = rn.ps.PubKey(rn.peer) - - c.proc.AddChild(process.WithTeardown(c.teardown)) + c.connProc = process.WithParent(c.pairProc) return c } func (c *conn) Close() error { - return c.proc.Close() + return c.pairProc.Close() +} + +func (c *conn) setup() { + c.connProc.SetTeardown(c.teardown) } func (c *conn) teardown() error { diff --git a/p2p/net/mock/mock_peernet.go b/p2p/net/mock/mock_peernet.go index ff549873e0..9216db500c 100644 --- a/p2p/net/mock/mock_peernet.go +++ b/p2p/net/mock/mock_peernet.go @@ -192,6 +192,10 @@ func (pn *peernet) addConn(c *conn) { defer c.notifLk.Unlock() pn.Unlock() + // Call this after unlocking as it might cause us to immediately close + // the connection and remove it from the swarm. + c.setup() + pn.notifyAll(func(n network.Notifiee) { n.Connected(pn, c) }) From 5403811df94b68a643313dceb2051ed6d0a58f9c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 23 Mar 2020 14:27:04 -0700 Subject: [PATCH 1754/3965] fix: make sure to close the mocknet's context when canceled --- p2p/net/mock/mock_net.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_net.go b/p2p/net/mock/mock_net.go index 5b75c399f4..9ed89df8a9 100644 --- a/p2p/net/mock/mock_net.go +++ b/p2p/net/mock/mock_net.go @@ -46,11 +46,14 @@ type mocknet struct { } func New(ctx context.Context) Mocknet { + proc := goprocessctx.WithContext(ctx) + ctx = goprocessctx.WithProcessClosing(ctx, proc) + return &mocknet{ nets: map[peer.ID]*peernet{}, hosts: map[peer.ID]*bhost.BasicHost{}, links: map[peer.ID]map[peer.ID]map[*link]struct{}{}, - proc: goprocessctx.WithContext(ctx), + proc: proc, ctx: ctx, } } From 29b355faaa6bd4c1da500544f9236910aad5165b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 23 Mar 2020 14:27:16 -0700 Subject: [PATCH 1755/3965] fix(mock): close the mocknet's peerstore --- p2p/net/mock/mock_peernet.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_peernet.go b/p2p/net/mock/mock_peernet.go index 9216db500c..e03f2334e6 100644 --- a/p2p/net/mock/mock_peernet.go +++ b/p2p/net/mock/mock_peernet.go @@ -64,7 +64,7 @@ func (pn *peernet) teardown() error { for _, c := range pn.allConns() { c.Close() } - return nil + return pn.ps.Close() } // allConns returns all the connections between this peer and others From c99a5bfa25e5bb1b61e41bf1628bafff48df83e5 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 23 Mar 2020 14:27:41 -0700 Subject: [PATCH 1756/3965] test(mock): close the mocknet when fuzzing --- p2p/net/mock/mock_test.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_test.go b/p2p/net/mock/mock_test.go index ce207a78ab..2fa4185d9e 100644 --- a/p2p/net/mock/mock_test.go +++ b/p2p/net/mock/mock_test.go @@ -598,7 +598,9 @@ func TestFuzzManyPeers(t *testing.T) { peerCount = 100 } for i := 0; i < peerCount; i++ { - _, err := FullMeshConnected(context.Background(), 2) + ctx, cancel := context.WithCancel(context.Background()) + _, err := FullMeshConnected(ctx, 2) + cancel() if err != nil { t.Fatal(err) } From 1ce9b9980afb8685647257a12fd2c89abf1a220e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 23 Mar 2020 14:27:55 -0700 Subject: [PATCH 1757/3965] chore(dep): update go-libp2p-peerstore fixes psmem.Close() --- go.mod | 2 +- go.sum | 31 ++++++++++++++++++++++++++----- 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index fbf5766be2..f1f71d4252 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/libp2p/go-libp2p-mplex v0.2.2 github.com/libp2p/go-libp2p-nat v0.0.5 github.com/libp2p/go-libp2p-netutil v0.1.0 - github.com/libp2p/go-libp2p-peerstore v0.2.0 + github.com/libp2p/go-libp2p-peerstore v0.2.1 github.com/libp2p/go-libp2p-secio v0.2.1 github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 diff --git a/go.sum b/go.sum index 8c4f4d41a3..8e40e26b97 100644 --- a/go.sum +++ b/go.sum @@ -27,6 +27,7 @@ github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3Ee github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE= github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davidlazar/go-crypto v0.0.0-20170701192655-dcfb0a7ac018 h1:6xT9KW8zLC5IlbaIF5Q7JNieBoACT7iW0YTxQHR0in0= github.com/davidlazar/go-crypto v0.0.0-20170701192655-dcfb0a7ac018/go.mod h1:rQYf4tfk5sSwFsnDg3qYaBxSjsD9S8+59vW0dKUgme4= @@ -36,6 +37,7 @@ github.com/dgraph-io/badger v1.6.0/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6ps github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98= github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= @@ -43,13 +45,16 @@ github.com/gogo/protobuf v1.3.0/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXP github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0= +github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg= github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= @@ -62,6 +67,7 @@ github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= +github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/huin/goupnp v1.0.0 h1:wg75sLpL6DZqwHQN6E1Cfk6mtfzS45z8OV+ic+DtHRo= github.com/huin/goupnp v1.0.0/go.mod h1:n9v9KO1tAxYH82qOn+UTIFQDmx5n1Zxd/ClZDMX7Bnc= @@ -115,8 +121,10 @@ github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d h1:68u9r4wEvL3gYg2jvAOgROwZ3H+Y3hIDk4tbbmIjcYQ= github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= +github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/libp2p/go-addr-util v0.0.1 h1:TpTQm9cXVRVSKsYbgQ7GKc3KbbHVTnbostgGaDEP+88= github.com/libp2p/go-addr-util v0.0.1/go.mod h1:4ac6O7n9rIAKB1dnd+s8IbbMXkt+oBpzX4/+RACcnlQ= @@ -130,11 +138,8 @@ github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UG github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= github.com/libp2p/go-flow-metrics v0.0.3 h1:8tAs/hSdNvUiLgtlSy3mxwxWP4I9y/jlkPFT7epKdeM= github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= -github.com/libp2p/go-libp2p v0.6.0/go.mod h1:mfKWI7Soz3ABX+XEBR61lGbg+ewyMtJHVt043oWeqwg= github.com/libp2p/go-libp2p v0.6.1/go.mod h1:CTFnWXogryAHjXAKEbOf1OWY+VeAP3lDMZkfEI5sT54= github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= -github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200319175655-901286ad641d h1:iA6ZxnrWVFW4Q4QlmKYctSapkGKbcbalnHygPVRCCfc= -github.com/libp2p/go-libp2p-autonat v0.1.2-0.20200319175655-901286ad641d/go.mod h1:yoF5B6yi+Wjxci55xdLlJo7gfyHwG80df4B9hiO2A88= github.com/libp2p/go-libp2p-autonat v0.2.0 h1:Kok+0M/4jiz6TTmxtBqAa5tLyHb/U+G/7o/JEeW7Wok= github.com/libp2p/go-libp2p-autonat v0.2.0/go.mod h1:DX+9teU4pEEoZUqR1PiMlqliONQdNbfzE1C718tcViI= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= @@ -170,6 +175,8 @@ github.com/libp2p/go-libp2p-peerstore v0.1.0/go.mod h1:2CeHkQsr8svp4fZ+Oi9ykN1HB github.com/libp2p/go-libp2p-peerstore v0.1.3/go.mod h1:BJ9sHlm59/80oSkpWgr1MyY1ciXAXV397W6h1GH/uKI= github.com/libp2p/go-libp2p-peerstore v0.2.0 h1:XcgJhI8WyUOCbHyRLNEX5542YNj8hnLSJ2G1InRjDhk= github.com/libp2p/go-libp2p-peerstore v0.2.0/go.mod h1:N2l3eVIeAitSg3Pi2ipSrJYnqhVnMNQZo9nkSCuAbnQ= +github.com/libp2p/go-libp2p-peerstore v0.2.1 h1:u+gOfsKgu73ZkGWhvckRm03z9C+iS9TrLqpANweELGs= +github.com/libp2p/go-libp2p-peerstore v0.2.1/go.mod h1:NQxhNjWxf1d4w6PihR8btWIRjwRLBr4TYKfNgrUkOPA= github.com/libp2p/go-libp2p-pnet v0.2.0 h1:J6htxttBipJujEjz1y0a5+eYoiPcFHhSYHH6na5f0/k= github.com/libp2p/go-libp2p-pnet v0.2.0/go.mod h1:Qqvq6JH/oMZGwqs3N1Fqhv8NVhrdYcO0BW4wssv21LA= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= @@ -189,7 +196,6 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07q github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 h1:5EhPgQhXZNyfL22ERZTUoVp9UVVbNowWNVtELQaKCHk= github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSoMuyojOdjx73Q47cYD7s5+gZOlns= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= -github.com/libp2p/go-libp2p-yamux v0.2.2/go.mod h1:lIohaR0pT6mOt0AZ0L2dFze9hds9Req3OfS+B+dv4qw= github.com/libp2p/go-libp2p-yamux v0.2.5 h1:MuyItOqz03oi8npvjgMJxgnhllJLZnO/dKVOpTZ9+XI= github.com/libp2p/go-libp2p-yamux v0.2.5/go.mod h1:Zpgj6arbyQrmZ3wxSZxfBmbdnWtbZ48OpsfmQVTErwA= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= @@ -221,7 +227,6 @@ github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfj github.com/libp2p/go-ws-transport v0.2.0 h1:MJCw2OrPA9+76YNRvdo1wMnSOxb9Bivj6sVFY1Xrj6w= github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzlHoKzu0yY9p/klM= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= -github.com/libp2p/go-yamux v1.3.0/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.3 h1:mWuzZRCAeTBFdynLlsYgA/EIeMOLr8XY04wa52NRhsE= github.com/libp2p/go-yamux v1.3.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= @@ -290,10 +295,12 @@ github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXS github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.12.0 h1:Iw5WCbBcaAAd0fpRb1c9r5YCylv4XDoCSigm1zLevwU= github.com/onsi/ginkgo v1.12.0/go.mod h1:oUhWkIvk5aDxtKvDDuw8gItl8pKl42LzjC9KZE0HfGg= github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= +github.com/onsi/gomega v1.9.0 h1:R1uwffexN6Pr340GtYRIdZmAiN4J+iw6WG4wog1DUXg= github.com/onsi/gomega v1.9.0/go.mod h1:Ho0h+IUsWyvy1OpqCwxlQ/21gkhVunqlU8fDGcoTdcA= github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= github.com/opentracing/opentracing-go v1.1.0 h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU= @@ -302,6 +309,7 @@ github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/9 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= @@ -322,6 +330,7 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+ github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= @@ -343,6 +352,8 @@ go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8= go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.uber.org/atomic v1.4.0 h1:cxzIVoETapQEqDhQu3QfnvXAV4AlzcvUCxkVUFw3+EU= go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= +go.uber.org/goleak v1.0.0 h1:qsup4IcBdlmsnGfqyLl4Ntn3C2XCCuKAE7DwHpScyUo= +go.uber.org/goleak v1.0.0/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A= go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI= go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM= @@ -364,6 +375,8 @@ golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs= +golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -380,6 +393,7 @@ golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAG golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -407,8 +421,11 @@ golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20191108193012-7d206e10da11/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191216052735-49a3e744a425 h1:VvQyQJN0tSuecqgcIxMWnnfG5kSmgy9KZR9sW3W5QeA= golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= @@ -420,12 +437,16 @@ google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZi google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d/go.mod h1:z+K8VcOYVYcSwSjGebuDL6176A1XskgbtNl64NSg+n8= gopkg.in/src-d/go-log.v1 v1.0.1/go.mod h1:GN34hKP0g305ysm2/hctJ0Y8nWP3zxXXJ8GFabTyABE= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I= gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= From 09d3ae44501c5ecfc8d4e21f52b0d2042aac6990 Mon Sep 17 00:00:00 2001 From: Will Date: Mon, 23 Mar 2020 16:49:48 -0700 Subject: [PATCH 1758/3965] skip dial attempts to non-public peers (#59) skip dial attempts to non-public peers fix https://github.com/libp2p/go-libp2p-autonat-svc/issues/33 fix #59 --- p2p/host/autonat/autonat.go | 55 ++++++++++++++++++++++++++++++++ p2p/host/autonat/autonat_test.go | 1 + p2p/host/autonat/options.go | 2 ++ p2p/host/autonat/svc.go | 28 ++-------------- p2p/host/autonat/svc_test.go | 25 ++------------- 5 files changed, 63 insertions(+), 48 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index f0c98d0645..331b9c67c9 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -154,6 +154,44 @@ func ipInList(candidate ma.Multiaddr, list []ma.Multiaddr) bool { return false } +// skipDial indicates that a multiaddress isn't worth attempted dialing. +// The same logic is used when the autonat client is considering if +// a remote peer is worth using as a server, and when the server is +// considering if a requested client is worth dialing back. +func (c *config) skipDial(addr ma.Multiaddr) bool { + // skip relay addresses + _, err := addr.ValueForProtocol(ma.P_CIRCUIT) + if err == nil { + return true + } + + if c.allowSelfDials { + return false + } + + // skip private network (unroutable) addresses + if !manet.IsPublicAddr(addr) { + return true + } + candidateIP, err := manet.ToIP(addr) + if err != nil { + return true + } + + // Skip dialing addresses we believe are the local node's + for _, localAddr := range c.host.Addrs() { + localIP, err := manet.ToIP(localAddr) + if err != nil { + continue + } + if localIP.Equal(candidateIP) { + return true + } + } + + return false +} + func (as *AmbientAutoNAT) background() { // wait a bit for the node to come online and establish some connections // before starting autodetection @@ -336,6 +374,23 @@ func (as *AmbientAutoNAT) probeNextPeer() { if proto, err := as.host.Peerstore().SupportsProtocols(p, AutoNATProto); len(proto) == 0 || err != nil { continue } + + goodAddr := false + for _, a := range info.Addrs { + if !as.config.skipDial(a) { + goodAddr = true + // if a public IP of the peer is one of ours: skip the peer. + aIP, _ := manet.ToIP(a) + aHost, _ := manet.FromIP(aIP) + if len(manet.AddrMatch(aHost, as.host.Addrs())) > 0 { + goodAddr = false + break + } + } + } + if !goodAddr { + continue + } addrs = append(addrs, info) } // TODO: track and exclude recently probed peers. diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index abeb810151..5e82fa5bed 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -55,6 +55,7 @@ func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, A h.Peerstore().AddAddrs(ash.ID(), ash.Addrs(), time.Minute) h.Peerstore().AddProtocols(ash.ID(), AutoNATProto) a, _ := New(ctx, h, WithSchedule(100*time.Millisecond, time.Second), WithoutStartupDelay()) + a.(*AmbientAutoNAT).config.allowSelfDials = true return h, a } diff --git a/p2p/host/autonat/options.go b/p2p/host/autonat/options.go index 6f7d729466..f7f664c9cb 100644 --- a/p2p/host/autonat/options.go +++ b/p2p/host/autonat/options.go @@ -16,6 +16,8 @@ type config struct { dialer network.Network forceReachability bool reachability network.Reachability + // Don't try to skip dials that might be within the lan + allowSelfDials bool // client bootDelay time.Duration diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index ed8fadcdce..ca1e391d38 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -20,8 +20,6 @@ import ( manet "github.com/multiformats/go-multiaddr-net" ) -const P_CIRCUIT = 290 - // AutoNATService provides NAT autodetection services to other peers type autoNATService struct { ctx context.Context @@ -112,7 +110,7 @@ func (as *autoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me // add observed addr to the list of addresses to dial var obsHost net.IP - if !as.skipDial(obsaddr) { + if !as.config.skipDial(obsaddr) { addrs = append(addrs, obsaddr) seen[obsaddr.String()] = struct{}{} obsHost, _ = manet.ToIP(obsaddr) @@ -125,7 +123,7 @@ func (as *autoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me continue } - if as.skipDial(addr) { + if as.config.skipDial(addr) { continue } @@ -154,28 +152,6 @@ func (as *autoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me return as.doDial(peer.AddrInfo{ID: p, Addrs: addrs}) } -func (as *autoNATService) skipDial(addr ma.Multiaddr) bool { - // skip relay addresses - _, err := addr.ValueForProtocol(P_CIRCUIT) - if err == nil { - return true - } - - // skip private network (unroutable) addresses - if !manet.IsPublicAddr(addr) { - return true - } - - // Skip dialing addresses we believe are the local node's - for _, localAddr := range as.config.host.Addrs() { - if localAddr.Equal(addr) { - return true - } - } - - return false -} - func (as *autoNATService) doDial(pi peer.AddrInfo) *pb.Message_DialResponse { // rate limit check as.mx.Lock() diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 749c7b9343..ed326bbc3d 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -2,7 +2,6 @@ package autonat import ( "context" - "net" "testing" "time" @@ -13,7 +12,6 @@ import ( swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" - manet "github.com/multiformats/go-multiaddr-net" ) func makeAutoNATConfig(ctx context.Context, t *testing.T) *config { @@ -22,6 +20,7 @@ func makeAutoNATConfig(ctx context.Context, t *testing.T) *config { c := config{host: h, dialer: dh.Network()} _ = defaults(&c) c.forceReachability = true + c.allowSelfDials = true return &c } @@ -48,6 +47,7 @@ func TestAutoNATServiceDialError(t *testing.T) { c := makeAutoNATConfig(ctx, t) c.dialTimeout = 1 * time.Second + c.allowSelfDials = false _ = makeAutoNATService(ctx, t, c) hc, ac := makeAutoNATClient(ctx, t) connect(t, c.host, hc) @@ -66,9 +66,6 @@ func TestAutoNATServiceDialSuccess(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save := manet.Private4 - manet.Private4 = []*net.IPNet{} - c := makeAutoNATConfig(ctx, t) _ = makeAutoNATService(ctx, t, c) @@ -79,17 +76,12 @@ func TestAutoNATServiceDialSuccess(t *testing.T) { if err != nil { t.Fatalf("Dial back failed: %s", err.Error()) } - - manet.Private4 = save } func TestAutoNATServiceDialRateLimiter(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save1 := manet.Private4 - manet.Private4 = []*net.IPNet{} - c := makeAutoNATConfig(ctx, t) c.dialTimeout = 1 * time.Second c.throttleResetPeriod = time.Second @@ -120,17 +112,12 @@ func TestAutoNATServiceDialRateLimiter(t *testing.T) { if err != nil { t.Fatal(err) } - - manet.Private4 = save1 } func TestAutoNATServiceGlobalLimiter(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save1 := manet.Private4 - manet.Private4 = []*net.IPNet{} - c := makeAutoNATConfig(ctx, t) c.dialTimeout = time.Second c.throttleResetPeriod = 10 * time.Second @@ -161,8 +148,6 @@ func TestAutoNATServiceGlobalLimiter(t *testing.T) { if !IsDialRefused(err) { t.Fatal(err) } - - manet.Private4 = save1 } func TestAutoNATServiceRateLimitJitter(t *testing.T) { @@ -191,12 +176,10 @@ func TestAutoNATServiceStartup(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - save := manet.Private4 - manet.Private4 = []*net.IPNet{} - h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) dh := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) an, err := New(ctx, h, EnableService(dh.Network())) + an.(*AmbientAutoNAT).config.allowSelfDials = true if err != nil { t.Fatal(err) } @@ -223,6 +206,4 @@ func TestAutoNATServiceStartup(t *testing.T) { if an.Status() != network.ReachabilityPublic { t.Fatalf("autonat should report public, but didn't") } - - manet.Private4 = save } From 82be3ee46fa62980e93dcb741009f4f3b4c2bc3b Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Wed, 25 Mar 2020 00:10:23 +0530 Subject: [PATCH 1759/3965] signal address change --- p2p/host/basic/basic_host.go | 67 ++++++++++++++++++++----------- p2p/host/basic/basic_host_test.go | 48 +++++++++++----------- p2p/host/relay/autorelay.go | 2 +- 3 files changed, 67 insertions(+), 50 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index bce9e580e8..8b0a0fd98b 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -90,6 +90,8 @@ type BasicHost struct { evtLocalProtocolsUpdated event.Emitter evtLocalAddrsUpdated event.Emitter } + + addrChangeChan chan struct{} } var _ host.Host = (*BasicHost)(nil) @@ -130,12 +132,13 @@ type HostOpts struct { // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHost, error) { h := &BasicHost{ - network: net, - mux: msmux.NewMultistreamMuxer(), - negtimeout: DefaultNegotiationTimeout, - AddrsFactory: DefaultAddrsFactory, - maResolver: madns.DefaultResolver, - eventbus: eventbus.NewBus(), + network: net, + mux: msmux.NewMultistreamMuxer(), + negtimeout: DefaultNegotiationTimeout, + AddrsFactory: DefaultAddrsFactory, + maResolver: madns.DefaultResolver, + eventbus: eventbus.NewBus(), + addrChangeChan: make(chan struct{}, 1), } var err error @@ -230,6 +233,7 @@ func New(net network.Network, opts ...interface{}) *BasicHost { } h, err := NewHost(context.Background(), net, hostopts) + h.Start() if err != nil { // this cannot happen with legacy options // plus we want to keep the (deprecated) legacy interface unchanged @@ -300,24 +304,13 @@ func (h *BasicHost) newStreamHandler(s network.Stream) { go handle(protoID, s) } -// CheckForAddressChanges determines whether our listen addresses have recently -// changed and emits an EvtLocalAddressesUpdatedEvent & a Push Identify if so. +// SignalAddressChange signals to the host that it needs to determine whether our listen addresses have recently +// changed. // Warning: this interface is unstable and may disappear in the future. -func (h *BasicHost) CheckForAddressChanges() { - h.mx.Lock() - addrs := h.Addrs() - changeEvt := makeUpdatedAddrEvent(h.lastAddrs, addrs) - if changeEvt != nil { - h.lastAddrs = addrs - } - h.mx.Unlock() - - if changeEvt != nil { - err := h.emitters.evtLocalAddrsUpdated.Emit(*changeEvt) - if err != nil { - log.Warnf("error emitting event for updated addrs: %s", err) - } - h.ids.Push() +func (h *BasicHost) SignalAddressChange() { + select { + case h.addrChangeChan <- struct{}{}: + default: } } @@ -366,10 +359,36 @@ func (h *BasicHost) background(p goprocess.Process) { } h.mx.Unlock() + // emit an EvtLocalAddressesUpdatedEvent & a Push Identify if our listen addresses have changed. + go func() { + for { + select { + case <-h.addrChangeChan: + h.mx.Lock() + addrs := h.Addrs() + changeEvt := makeUpdatedAddrEvent(h.lastAddrs, addrs) + if changeEvt != nil { + h.lastAddrs = addrs + } + h.mx.Unlock() + + if changeEvt != nil { + err := h.emitters.evtLocalAddrsUpdated.Emit(*changeEvt) + if err != nil { + log.Warnf("error emitting event for updated addrs: %s", err) + } + h.ids.Push() + } + case <-p.Closing(): + return + } + } + }() + for { select { case <-ticker.C: - h.CheckForAddressChanges() + h.SignalAddressChange() case <-p.Closing(): return diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index dd3593a894..c0dee8e571 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -6,6 +6,7 @@ import ( "io" "reflect" "sort" + "sync" "testing" "time" @@ -518,7 +519,7 @@ func TestHostAddrChangeDetection(t *testing.T) { {ma.StringCast("/ip4/2.3.4.5/tcp/1234"), ma.StringCast("/ip4/3.4.5.6/tcp/4321")}, } - // The events we expect the host to emit when CheckForAddressChanges is called + // The events we expect the host to emit when SignalAddressChange is called // and the changes between addr sets are detected expectedEvents := []event.EvtLocalAddressesUpdated{ { @@ -548,8 +549,11 @@ func TestHostAddrChangeDetection(t *testing.T) { }, } + var lk sync.Mutex currentAddrSet := 0 addrsFactory := func(addrs []ma.Multiaddr) []ma.Multiaddr { + lk.Lock() + defer lk.Unlock() return addrSets[currentAddrSet] } @@ -563,46 +567,40 @@ func TestHostAddrChangeDetection(t *testing.T) { } defer sub.Close() + // wait for the host background thread to start + time.Sleep(1 * time.Second) // host should start with no addrs (addrSet 0) addrs := h.Addrs() if len(addrs) != 0 { t.Fatalf("expected 0 addrs, got %d", len(addrs)) } - // Advance between addrSets + // change addr, signal and assert event for i := 1; i < len(addrSets); i++ { + lk.Lock() currentAddrSet = i - h.CheckForAddressChanges() // forces the host to check for changes now, instead of waiting for background update + lk.Unlock() + h.SignalAddressChange() + evt := waitForAddrChangeEvent(ctx, sub, t) + if !updatedAddrEventsEqual(expectedEvents[i-1], evt) { + t.Errorf("change events not equal: \n\texpected: %v \n\tactual: %v", expectedEvents[i], evt) + } + } +} - // drain events from the subscription - var receivedEvents []event.EvtLocalAddressesUpdated -readEvents: +func waitForAddrChangeEvent(ctx context.Context, sub event.Subscription, t *testing.T) event.EvtLocalAddressesUpdated { for { select { case evt, more := <-sub.Out(): if !more { - break readEvents - } - receivedEvents = append(receivedEvents, evt.(event.EvtLocalAddressesUpdated)) - if len(receivedEvents) == len(expectedEvents) { - break readEvents + t.Fatal("channel should not be closed") } + return evt.(event.EvtLocalAddressesUpdated) case <-ctx.Done(): - break readEvents - case <-time.After(1 * time.Second): - break readEvents - } - } - - // assert that we received the events we expected - if len(receivedEvents) != len(expectedEvents) { - t.Errorf("expected to receive %d addr change events, got %d", len(expectedEvents), len(receivedEvents)) - } - for i, expected := range expectedEvents { - actual := receivedEvents[i] - if !updatedAddrEventsEqual(expected, actual) { - t.Errorf("change events not equal: \n\texpected: %v \n\tactual: %v", expected, actual) + t.Fatal("context should not have cancelled") + case <-time.After(2 * time.Second): + t.Fatal("timed out waiting for address change event") } } } diff --git a/p2p/host/relay/autorelay.go b/p2p/host/relay/autorelay.go index 3ef741b08a..a1691efb26 100644 --- a/p2p/host/relay/autorelay.go +++ b/p2p/host/relay/autorelay.go @@ -119,7 +119,7 @@ func (ar *AutoRelay) background(ctx context.Context) { ar.cachedAddrs = nil ar.mx.Unlock() push = false - ar.host.CheckForAddressChanges() + ar.host.SignalAddressChange() } } } From fa575b4d32958ce330c2c7ccef483f23c6fe1181 Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Wed, 25 Mar 2020 00:35:26 +0530 Subject: [PATCH 1760/3965] changes as per review --- p2p/host/basic/basic_host.go | 46 ++++++++++++++---------------------- 1 file changed, 18 insertions(+), 28 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 8b0a0fd98b..3de9984c2f 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -359,40 +359,30 @@ func (h *BasicHost) background(p goprocess.Process) { } h.mx.Unlock() - // emit an EvtLocalAddressesUpdatedEvent & a Push Identify if our listen addresses have changed. - go func() { - for { - select { - case <-h.addrChangeChan: - h.mx.Lock() - addrs := h.Addrs() - changeEvt := makeUpdatedAddrEvent(h.lastAddrs, addrs) - if changeEvt != nil { - h.lastAddrs = addrs - } - h.mx.Unlock() - - if changeEvt != nil { - err := h.emitters.evtLocalAddrsUpdated.Emit(*changeEvt) - if err != nil { - log.Warnf("error emitting event for updated addrs: %s", err) - } - h.ids.Push() - } - case <-p.Closing(): - return - } - } - }() - for { select { case <-ticker.C: - h.SignalAddressChange() - + case <-h.addrChangeChan: case <-p.Closing(): return } + + // emit an EvtLocalAddressesUpdatedEvent & a Push Identify if our listen addresses have changed. + h.mx.Lock() + addrs := h.Addrs() + changeEvt := makeUpdatedAddrEvent(h.lastAddrs, addrs) + if changeEvt != nil { + h.lastAddrs = addrs + } + h.mx.Unlock() + + if changeEvt != nil { + err := h.emitters.evtLocalAddrsUpdated.Emit(*changeEvt) + if err != nil { + log.Warnf("error emitting event for updated addrs: %s", err) + } + h.ids.Push() + } } } From e17a6f8bf2c9faba8c74dde2be214cf1fe43703e Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Wed, 25 Mar 2020 00:52:25 +0530 Subject: [PATCH 1761/3965] remove lastaddrs as member variable --- p2p/host/basic/basic_host.go | 37 ++++-------------------------------- 1 file changed, 4 insertions(+), 33 deletions(-) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index 3de9984c2f..f64e1840f8 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -4,7 +4,6 @@ import ( "context" "io" "net" - "sync" "time" "github.com/libp2p/go-libp2p/p2p/protocol/identify" @@ -84,9 +83,7 @@ type BasicHost struct { proc goprocess.Process - mx sync.Mutex - lastAddrs []ma.Multiaddr - emitters struct { + emitters struct { evtLocalProtocolsUpdated event.Emitter evtLocalAddrsUpdated event.Emitter } @@ -353,11 +350,7 @@ func (h *BasicHost) background(p goprocess.Process) { defer ticker.Stop() // initialize lastAddrs - h.mx.Lock() - if h.lastAddrs == nil { - h.lastAddrs = h.Addrs() - } - h.mx.Unlock() + lastAddrs := h.Addrs() for { select { @@ -368,13 +361,11 @@ func (h *BasicHost) background(p goprocess.Process) { } // emit an EvtLocalAddressesUpdatedEvent & a Push Identify if our listen addresses have changed. - h.mx.Lock() addrs := h.Addrs() - changeEvt := makeUpdatedAddrEvent(h.lastAddrs, addrs) + changeEvt := makeUpdatedAddrEvent(lastAddrs, addrs) if changeEvt != nil { - h.lastAddrs = addrs + lastAddrs = addrs } - h.mx.Unlock() if changeEvt != nil { err := h.emitters.evtLocalAddrsUpdated.Emit(*changeEvt) @@ -386,26 +377,6 @@ func (h *BasicHost) background(p goprocess.Process) { } } -func sameAddrs(a, b []ma.Multiaddr) bool { - if len(a) != len(b) { - return false - } - - bmap := make(map[string]struct{}, len(b)) - for _, addr := range b { - bmap[string(addr.Bytes())] = struct{}{} - } - - for _, addr := range a { - _, ok := bmap[string(addr.Bytes())] - if !ok { - return false - } - } - - return true -} - // ID returns the (local) peer.ID associated with this Host func (h *BasicHost) ID() peer.ID { return h.Network().LocalPeer() From aa30b1d124237ebee8b9244af64151692d3c03a0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 24 Mar 2020 14:23:42 -0700 Subject: [PATCH 1762/3965] fix: update goprocess to fix a test race --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index f1f71d4252..d5f125b0ad 100644 --- a/go.mod +++ b/go.mod @@ -7,7 +7,7 @@ require ( github.com/ipfs/go-ipfs-util v0.0.1 github.com/ipfs/go-log v1.0.2 github.com/jbenet/go-cienv v0.1.0 - github.com/jbenet/goprocess v0.1.3 + github.com/jbenet/goprocess v0.1.4 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 github.com/libp2p/go-libp2p-autonat v0.2.0 diff --git a/go.sum b/go.sum index 8e40e26b97..11e6861d3e 100644 --- a/go.sum +++ b/go.sum @@ -110,6 +110,8 @@ github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2/go.mod github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsjFq/qrU3Rar62tu1gASgGw6chQbSh/XgIIXCY= github.com/jbenet/goprocess v0.1.3 h1:YKyIEECS/XvcfHtBzxtjBBbWK+MbvA6dG8ASiqwvr10= github.com/jbenet/goprocess v0.1.3/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= +github.com/jbenet/goprocess v0.1.4 h1:DRGOFReOMqqDNXwW70QkacFW0YN9QnwLV0Vqk+3oU0o= +github.com/jbenet/goprocess v0.1.4/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= From 1359013f3955035d1e7b4f3198cd48ce73415c7e Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 24 Mar 2020 16:48:50 -0700 Subject: [PATCH 1763/3965] fix: set the private key when constructing the autonat service Otherwise, it the node fails to build. --- config/config.go | 8 ++++++++ libp2p_test.go | 9 +++++++++ 2 files changed, 17 insertions(+) diff --git a/config/config.go b/config/config.go index bd843cac5d..f491f92dff 100644 --- a/config/config.go +++ b/config/config.go @@ -2,6 +2,7 @@ package config import ( "context" + "crypto/rand" "fmt" "github.com/libp2p/go-libp2p-core/connmgr" @@ -283,6 +284,11 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return addrF(h.AllAddrs()) })} if cfg.AutoNATService { + autonatPrivKey, _, err := crypto.GenerateEd25519Key(rand.Reader) + if err != nil { + return nil, err + } + // Pull out the pieces of the config that we _actually_ care about. // Specifically, don't setup things like autorelay, listeners, // identify, etc. @@ -294,9 +300,11 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { PSK: cfg.PSK, Filters: cfg.Filters, Reporter: cfg.Reporter, + PeerKey: autonatPrivKey, Peerstore: pstoremem.NewPeerstore(), } + dialer, err := autoNatCfg.makeSwarm(ctx) if err != nil { h.Close() diff --git a/libp2p_test.go b/libp2p_test.go index d1099c647d..9a8e4b8010 100644 --- a/libp2p_test.go +++ b/libp2p_test.go @@ -77,6 +77,15 @@ func TestInsecure(t *testing.T) { h.Close() } +func TestAutoNATService(t *testing.T) { + ctx := context.Background() + h, err := New(ctx, EnableNATService()) + if err != nil { + t.Fatal(err) + } + h.Close() +} + func TestDefaultListenAddrs(t *testing.T) { ctx := context.Background() From 96a91cce5f6b4d2309cd5fd37506b15095727f76 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Wed, 25 Mar 2020 16:52:49 +0700 Subject: [PATCH 1764/3965] move the info used for stateless key expansion to a constant --- p2p/transport/quic/transport.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 7b3484b203..fa54bd8c54 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -36,6 +36,8 @@ var quicConfig = &quic.Config{ KeepAlive: true, } +const statelessResetKeyInfo = "libp2p quic stateless reset key" + type connManager struct { reuseUDP4 *reuse reuseUDP6 *reuse @@ -117,7 +119,7 @@ func NewTransport(key ic.PrivKey, psk pnet.PSK, filters *filter.Filters) (tpt.Tr if err != nil { return nil, err } - keyReader := hkdf.New(sha256.New, keyBytes, nil, []byte("libp2p quic stateless reset key")) + keyReader := hkdf.New(sha256.New, keyBytes, nil, []byte(statelessResetKeyInfo)) config.StatelessResetKey = make([]byte, 32) if _, err := io.ReadFull(keyReader, config.StatelessResetKey); err != nil { return nil, err From 5b578f5921692220bcd497f51a474f0181242dd8 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Wed, 25 Mar 2020 11:21:17 +0000 Subject: [PATCH 1765/3965] build(deps): bump github.com/ipfs/go-log from 1.0.2 to 1.0.3 Bumps [github.com/ipfs/go-log](https://github.com/ipfs/go-log) from 1.0.2 to 1.0.3. - [Release notes](https://github.com/ipfs/go-log/releases) - [Commits](https://github.com/ipfs/go-log/compare/v1.0.2...v1.0.3) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index d5f125b0ad..fcc862cdeb 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ require ( github.com/ipfs/go-cid v0.0.5 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 - github.com/ipfs/go-log v1.0.2 + github.com/ipfs/go-log v1.0.3 github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.4 github.com/libp2p/go-conn-security-multistream v0.1.0 diff --git a/go.sum b/go.sum index 11e6861d3e..7385d181e8 100644 --- a/go.sum +++ b/go.sum @@ -96,8 +96,12 @@ github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyB github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= github.com/ipfs/go-log v1.0.2 h1:s19ZwJxH8rPWzypjcDpqPLIyV7BnbLqvpli3iZoqYK0= github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= +github.com/ipfs/go-log v1.0.3 h1:Gg7SUYSZ7BrqaKMwM+hRgcAkKv4QLfzP4XPQt5Sx/OI= +github.com/ipfs/go-log v1.0.3/go.mod h1:OsLySYkwIbiSUR/yBTdv1qPtcE4FW3WPWk/ewz9Ru+A= github.com/ipfs/go-log/v2 v2.0.2 h1:xguurydRdfKMJjKyxNXNU8lYP0VZH1NUwJRwUorjuEw= github.com/ipfs/go-log/v2 v2.0.2/go.mod h1:O7P1lJt27vWHhOwQmcFEvlmo49ry2VY2+JfBWFaa9+0= +github.com/ipfs/go-log/v2 v2.0.3 h1:Q2gXcBoCALyLN/pUQlz1qgu0x3uFV6FzP9oXhpfyJpc= +github.com/ipfs/go-log/v2 v2.0.3/go.mod h1:O7P1lJt27vWHhOwQmcFEvlmo49ry2VY2+JfBWFaa9+0= github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc= github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= From 73266e7dc91b5e29d163e3fc93cf1d484fbd5b0e Mon Sep 17 00:00:00 2001 From: Vibhav Pant Date: Wed, 25 Mar 2020 17:34:36 +0530 Subject: [PATCH 1766/3965] peer: Add (*PeerRecord).ToProtoBuf to get protobuf equivalent struct --- core/peer/record.go | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/core/peer/record.go b/core/peer/record.go index c45efbea7e..f21b0dace6 100644 --- a/core/peer/record.go +++ b/core/peer/record.go @@ -187,6 +187,19 @@ func (r *PeerRecord) Equal(other *PeerRecord) bool { return true } +// ToProtobuf returns the equivalent Protocol Buffer struct object of a PeerRecord. +func (r *PeerRecord) ToProtobuf() (*pb.PeerRecord, error) { + idBytes, err := r.PeerID.MarshalBinary() + if err != nil { + return nil, err + } + return &pb.PeerRecord{ + PeerId: idBytes, + Addresses: addrsToProtobuf(r.Addrs), + Seq: r.Seq, + }, nil +} + func addrsFromProtobuf(addrs []*pb.PeerRecord_AddressInfo) []ma.Multiaddr { var out []ma.Multiaddr for _, addr := range addrs { From e2dd6fd569ee35b3f2fffa00f173265736f4f344 Mon Sep 17 00:00:00 2001 From: Vibhav Pant Date: Wed, 25 Mar 2020 17:42:49 +0530 Subject: [PATCH 1767/3965] (*PeerRecord.MarshalRecord): Use ToProtobuf() to get protobuf struct --- core/peer/record.go | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/core/peer/record.go b/core/peer/record.go index f21b0dace6..60373c36d6 100644 --- a/core/peer/record.go +++ b/core/peer/record.go @@ -153,16 +153,11 @@ func (r *PeerRecord) UnmarshalRecord(bytes []byte) error { // This method is called automatically when constructing a routing.Envelope // using Seal or PeerRecord.Sign. func (r *PeerRecord) MarshalRecord() ([]byte, error) { - idBytes, err := r.PeerID.MarshalBinary() + msg, err := r.ToProtobuf() if err != nil { return nil, err } - msg := pb.PeerRecord{ - PeerId: idBytes, - Addresses: addrsToProtobuf(r.Addrs), - Seq: r.Seq, - } - return proto.Marshal(&msg) + return proto.Marshal(msg) } // Equal returns true if the other PeerRecord is identical to this one. From 0d304cb76928aa483cc8f9d604d32205ff102fdd Mon Sep 17 00:00:00 2001 From: Will Date: Wed, 25 Mar 2020 11:11:55 -0700 Subject: [PATCH 1768/3965] factor out dial policy (#62) factor out dial policy --- p2p/host/autonat/autonat.go | 56 +----------- p2p/host/autonat/autonat_test.go | 2 +- p2p/host/autonat/dialpolicy.go | 95 +++++++++++++++++++ p2p/host/autonat/dialpolicy_test.go | 137 ++++++++++++++++++++++++++++ p2p/host/autonat/options.go | 3 +- p2p/host/autonat/svc.go | 4 +- p2p/host/autonat/svc_test.go | 6 +- 7 files changed, 242 insertions(+), 61 deletions(-) create mode 100644 p2p/host/autonat/dialpolicy.go create mode 100644 p2p/host/autonat/dialpolicy_test.go diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 331b9c67c9..04ec7a1a46 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -63,6 +63,7 @@ func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { var err error conf := new(config) conf.host = h + conf.dialPolicy.host = h if err = defaults(conf); err != nil { return nil, err @@ -154,44 +155,6 @@ func ipInList(candidate ma.Multiaddr, list []ma.Multiaddr) bool { return false } -// skipDial indicates that a multiaddress isn't worth attempted dialing. -// The same logic is used when the autonat client is considering if -// a remote peer is worth using as a server, and when the server is -// considering if a requested client is worth dialing back. -func (c *config) skipDial(addr ma.Multiaddr) bool { - // skip relay addresses - _, err := addr.ValueForProtocol(ma.P_CIRCUIT) - if err == nil { - return true - } - - if c.allowSelfDials { - return false - } - - // skip private network (unroutable) addresses - if !manet.IsPublicAddr(addr) { - return true - } - candidateIP, err := manet.ToIP(addr) - if err != nil { - return true - } - - // Skip dialing addresses we believe are the local node's - for _, localAddr := range c.host.Addrs() { - localIP, err := manet.ToIP(localAddr) - if err != nil { - continue - } - if localIP.Equal(candidateIP) { - return true - } - } - - return false -} - func (as *AmbientAutoNAT) background() { // wait a bit for the node to come online and establish some connections // before starting autodetection @@ -375,21 +338,8 @@ func (as *AmbientAutoNAT) probeNextPeer() { continue } - goodAddr := false - for _, a := range info.Addrs { - if !as.config.skipDial(a) { - goodAddr = true - // if a public IP of the peer is one of ours: skip the peer. - aIP, _ := manet.ToIP(a) - aHost, _ := manet.FromIP(aIP) - if len(manet.AddrMatch(aHost, as.host.Addrs())) > 0 { - goodAddr = false - break - } - } - } - if !goodAddr { - continue + if !as.config.dialPolicy.skipPeer(info.Addrs) { + addrs = append(addrs, info) } addrs = append(addrs, info) } diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 5e82fa5bed..7e45862a2d 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -55,7 +55,7 @@ func makeAutoNAT(ctx context.Context, t *testing.T, ash host.Host) (host.Host, A h.Peerstore().AddAddrs(ash.ID(), ash.Addrs(), time.Minute) h.Peerstore().AddProtocols(ash.ID(), AutoNATProto) a, _ := New(ctx, h, WithSchedule(100*time.Millisecond, time.Second), WithoutStartupDelay()) - a.(*AmbientAutoNAT).config.allowSelfDials = true + a.(*AmbientAutoNAT).config.dialPolicy.allowSelfDials = true return h, a } diff --git a/p2p/host/autonat/dialpolicy.go b/p2p/host/autonat/dialpolicy.go new file mode 100644 index 0000000000..4c35680f34 --- /dev/null +++ b/p2p/host/autonat/dialpolicy.go @@ -0,0 +1,95 @@ +package autonat + +import ( + "net" + + "github.com/libp2p/go-libp2p-core/host" + ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" +) + +type dialPolicy struct { + allowSelfDials bool + host host.Host +} + +// skipDial indicates that a multiaddress isn't worth attempted dialing. +// The same logic is used when the autonat client is considering if +// a remote peer is worth using as a server, and when the server is +// considering if a requested client is worth dialing back. +func (d *dialPolicy) skipDial(addr ma.Multiaddr) bool { + // skip relay addresses + _, err := addr.ValueForProtocol(ma.P_CIRCUIT) + if err == nil { + return true + } + + if d.allowSelfDials { + return false + } + + // skip private network (unroutable) addresses + if !manet.IsPublicAddr(addr) { + return true + } + candidateIP, err := manet.ToIP(addr) + if err != nil { + return true + } + + // Skip dialing addresses we believe are the local node's + for _, localAddr := range d.host.Addrs() { + localIP, err := manet.ToIP(localAddr) + if err != nil { + continue + } + if localIP.Equal(candidateIP) { + return true + } + } + + return false +} + +// skipPeer indicates that the collection of multiaddresses representing a peer +// isn't worth attempted dialing. Addresses are dialed individually, and while +// individual addresses for a peer may be worth considering, there are some +// factors, like the presence of the same public address as the local host, +// that may make the peer undesirable to dial as a whole. +func (d *dialPolicy) skipPeer(addrs []ma.Multiaddr) bool { + localAddrs := d.host.Addrs() + localHosts := make([]net.IP, 0) + for _, lAddr := range localAddrs { + if _, err := lAddr.ValueForProtocol(ma.P_CIRCUIT); err != nil && manet.IsPublicAddr(lAddr) { + lIP, err := manet.ToIP(lAddr) + if err != nil { + continue + } + localHosts = append(localHosts, lIP) + } + } + + // if a public IP of the peer is one of ours: skip the peer. + goodPublic := false + for _, addr := range addrs { + if _, err := addr.ValueForProtocol(ma.P_CIRCUIT); err != nil && manet.IsPublicAddr(addr) { + aIP, err := manet.ToIP(addr) + if err != nil { + continue + } + + for _, lIP := range localHosts { + if lIP.Equal(aIP) { + return true + } + } + goodPublic = true + } + } + + if d.allowSelfDials { + return false + } + + return !goodPublic +} diff --git a/p2p/host/autonat/dialpolicy_test.go b/p2p/host/autonat/dialpolicy_test.go new file mode 100644 index 0000000000..d18385caf7 --- /dev/null +++ b/p2p/host/autonat/dialpolicy_test.go @@ -0,0 +1,137 @@ +package autonat + +import ( + "context" + "errors" + "net" + "testing" + + blankhost "github.com/libp2p/go-libp2p-blankhost" + "github.com/libp2p/go-libp2p-core/peer" + "github.com/libp2p/go-libp2p-core/transport" + swarmt "github.com/libp2p/go-libp2p-swarm/testing" + "github.com/multiformats/go-multiaddr" +) + +func makeMA(a string) multiaddr.Multiaddr { + addr, err := multiaddr.NewMultiaddr(a) + if err != nil { + panic(err) + } + return addr +} + +type mockT struct { + ctx context.Context + addr multiaddr.Multiaddr +} + +func (m *mockT) Dial(ctx context.Context, a multiaddr.Multiaddr, p peer.ID) (transport.CapableConn, error) { + return nil, nil +} +func (m *mockT) CanDial(_ multiaddr.Multiaddr) bool { return true } +func (m *mockT) Listen(a multiaddr.Multiaddr) (transport.Listener, error) { + return &mockL{m.ctx, m.addr}, nil +} +func (m *mockT) Protocols() []int { return []int{multiaddr.P_IP4} } +func (m *mockT) Proxy() bool { return false } +func (m *mockT) String() string { return "mock-tcp-ipv4" } + +type mockL struct { + ctx context.Context + addr multiaddr.Multiaddr +} + +func (l *mockL) Accept() (transport.CapableConn, error) { + select { + case <-l.ctx.Done(): + } + return nil, errors.New("expected in mocked test") +} +func (l *mockL) Close() error { return nil } +func (l *mockL) Addr() net.Addr { return nil } +func (l *mockL) Multiaddr() multiaddr.Multiaddr { return l.addr } + +func TestSkipDial(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + s := swarmt.GenSwarm(t, ctx) + d := dialPolicy{host: blankhost.NewBlankHost(s)} + if d.skipDial(makeMA("/ip4/8.8.8.8")) != false { + t.Fatal("failed dialing a valid public addr") + } + + if d.skipDial(makeMA("/ip6/2607:f8b0:400a::1")) != false { + t.Fatal("failed dialing a valid public addr") + } + + if d.skipDial(makeMA("/ip4/192.168.0.1")) != true { + t.Fatal("didn't skip dialing an internal addr") + } + + s.AddTransport(&mockT{ctx, makeMA("/ip4/8.8.8.8")}) + err := s.AddListenAddr(makeMA("/ip4/8.8.8.8")) + if err != nil { + t.Fatal(err) + } + if d.skipDial(makeMA("/ip4/8.8.8.8")) != true { + t.Fatal("failed dialing a valid host address") + } +} + +func TestSkipPeer(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + s := swarmt.GenSwarm(t, ctx) + d := dialPolicy{host: blankhost.NewBlankHost(s)} + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/8.8.8.8")}) != false { + t.Fatal("failed dialing a valid public addr") + } + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/8.8.8.8"), makeMA("/ip4/192.168.0.1")}) != false { + t.Fatal("failed dialing a valid public addr") + } + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/192.168.0.1")}) != true { + t.Fatal("succeeded with no public addr") + } + + s.AddTransport(&mockT{ctx, makeMA("/ip4/8.8.8.8")}) + err := s.AddListenAddr(makeMA("/ip4/8.8.8.8")) + if err != nil { + t.Fatal(err) + } + + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/8.8.8.8"), makeMA("/ip4/192.168.0.1")}) != true { + t.Fatal("succeeded dialing host address") + } + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/8.8.8.8"), makeMA("/ip4/9.9.9.9")}) != true { + t.Fatal("succeeded dialing host address when other public") + } + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/9.9.9.9")}) != false { + t.Fatal("succeeded dialing host address when other public") + } +} + +func TestSkipLocalPeer(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + s := swarmt.GenSwarm(t, ctx) + d := dialPolicy{host: blankhost.NewBlankHost(s)} + s.AddTransport(&mockT{ctx, makeMA("/ip4/192.168.0.1")}) + err := s.AddListenAddr(makeMA("/ip4/192.168.0.1")) + if err != nil { + t.Fatal(err) + } + + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/8.8.8.8")}) != false { + t.Fatal("failed dialing a valid public addr") + } + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/8.8.8.8"), makeMA("/ip4/192.168.0.1")}) != false { + t.Fatal("failed dialing a valid public addr") + } + if d.skipPeer([]multiaddr.Multiaddr{makeMA("/ip4/192.168.0.1")}) != true { + t.Fatal("succeeded with no public addr") + } +} diff --git a/p2p/host/autonat/options.go b/p2p/host/autonat/options.go index f7f664c9cb..d7c77f9e96 100644 --- a/p2p/host/autonat/options.go +++ b/p2p/host/autonat/options.go @@ -13,11 +13,10 @@ type config struct { host host.Host addressFunc AddrFunc + dialPolicy dialPolicy dialer network.Network forceReachability bool reachability network.Reachability - // Don't try to skip dials that might be within the lan - allowSelfDials bool // client bootDelay time.Duration diff --git a/p2p/host/autonat/svc.go b/p2p/host/autonat/svc.go index ca1e391d38..481de16d82 100644 --- a/p2p/host/autonat/svc.go +++ b/p2p/host/autonat/svc.go @@ -110,7 +110,7 @@ func (as *autoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me // add observed addr to the list of addresses to dial var obsHost net.IP - if !as.config.skipDial(obsaddr) { + if !as.config.dialPolicy.skipDial(obsaddr) { addrs = append(addrs, obsaddr) seen[obsaddr.String()] = struct{}{} obsHost, _ = manet.ToIP(obsaddr) @@ -123,7 +123,7 @@ func (as *autoNATService) handleDial(p peer.ID, obsaddr ma.Multiaddr, mpi *pb.Me continue } - if as.config.skipDial(addr) { + if as.config.dialPolicy.skipDial(addr) { continue } diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index ed326bbc3d..195a523438 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -20,7 +20,7 @@ func makeAutoNATConfig(ctx context.Context, t *testing.T) *config { c := config{host: h, dialer: dh.Network()} _ = defaults(&c) c.forceReachability = true - c.allowSelfDials = true + c.dialPolicy.allowSelfDials = true return &c } @@ -47,7 +47,7 @@ func TestAutoNATServiceDialError(t *testing.T) { c := makeAutoNATConfig(ctx, t) c.dialTimeout = 1 * time.Second - c.allowSelfDials = false + c.dialPolicy.allowSelfDials = false _ = makeAutoNATService(ctx, t, c) hc, ac := makeAutoNATClient(ctx, t) connect(t, c.host, hc) @@ -179,7 +179,7 @@ func TestAutoNATServiceStartup(t *testing.T) { h := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) dh := bhost.NewBlankHost(swarmt.GenSwarm(t, ctx)) an, err := New(ctx, h, EnableService(dh.Network())) - an.(*AmbientAutoNAT).config.allowSelfDials = true + an.(*AmbientAutoNAT).config.dialPolicy.allowSelfDials = true if err != nil { t.Fatal(err) } From 31e2e45f7fe726390d79cb76f9940a3664f4c9bc Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 25 Mar 2020 12:12:32 -0700 Subject: [PATCH 1769/3965] chore: update deps * Update logging to fix logging on windows. * Update autonat to improve filtering of potential AutoNAT peers. --- go.mod | 4 ++-- go.sum | 9 +++++++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index d5f125b0ad..4044698945 100644 --- a/go.mod +++ b/go.mod @@ -5,12 +5,12 @@ require ( github.com/ipfs/go-cid v0.0.5 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 - github.com/ipfs/go-log v1.0.2 + github.com/ipfs/go-log v1.0.3 github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.4 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 - github.com/libp2p/go-libp2p-autonat v0.2.0 + github.com/libp2p/go-libp2p-autonat v0.2.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.1.4 github.com/libp2p/go-libp2p-core v0.5.0 diff --git a/go.sum b/go.sum index 11e6861d3e..a154322477 100644 --- a/go.sum +++ b/go.sum @@ -96,8 +96,12 @@ github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyB github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= github.com/ipfs/go-log v1.0.2 h1:s19ZwJxH8rPWzypjcDpqPLIyV7BnbLqvpli3iZoqYK0= github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= +github.com/ipfs/go-log v1.0.3 h1:Gg7SUYSZ7BrqaKMwM+hRgcAkKv4QLfzP4XPQt5Sx/OI= +github.com/ipfs/go-log v1.0.3/go.mod h1:OsLySYkwIbiSUR/yBTdv1qPtcE4FW3WPWk/ewz9Ru+A= github.com/ipfs/go-log/v2 v2.0.2 h1:xguurydRdfKMJjKyxNXNU8lYP0VZH1NUwJRwUorjuEw= github.com/ipfs/go-log/v2 v2.0.2/go.mod h1:O7P1lJt27vWHhOwQmcFEvlmo49ry2VY2+JfBWFaa9+0= +github.com/ipfs/go-log/v2 v2.0.3 h1:Q2gXcBoCALyLN/pUQlz1qgu0x3uFV6FzP9oXhpfyJpc= +github.com/ipfs/go-log/v2 v2.0.3/go.mod h1:O7P1lJt27vWHhOwQmcFEvlmo49ry2VY2+JfBWFaa9+0= github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc= github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= @@ -141,9 +145,12 @@ github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZ github.com/libp2p/go-flow-metrics v0.0.3 h1:8tAs/hSdNvUiLgtlSy3mxwxWP4I9y/jlkPFT7epKdeM= github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= github.com/libp2p/go-libp2p v0.6.1/go.mod h1:CTFnWXogryAHjXAKEbOf1OWY+VeAP3lDMZkfEI5sT54= +github.com/libp2p/go-libp2p v0.7.0/go.mod h1:hZJf8txWeCduQRDC/WSqBGMxaTHCOYHt2xSU1ivxn0k= github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= github.com/libp2p/go-libp2p-autonat v0.2.0 h1:Kok+0M/4jiz6TTmxtBqAa5tLyHb/U+G/7o/JEeW7Wok= github.com/libp2p/go-libp2p-autonat v0.2.0/go.mod h1:DX+9teU4pEEoZUqR1PiMlqliONQdNbfzE1C718tcViI= +github.com/libp2p/go-libp2p-autonat v0.2.1 h1:T0CRQhrvTBKfBSYw6Xo2K3ixtNpAnRCraxof3AAfgQA= +github.com/libp2p/go-libp2p-autonat v0.2.1/go.mod h1:MWtAhV5Ko1l6QBsHQNSuM6b1sRkXrpk0/LqCr+vCVxI= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= @@ -232,6 +239,7 @@ github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZ github.com/libp2p/go-yamux v1.3.3 h1:mWuzZRCAeTBFdynLlsYgA/EIeMOLr8XY04wa52NRhsE= github.com/libp2p/go-yamux v1.3.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= +github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= @@ -315,6 +323,7 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= +github.com/smola/gocompat v0.2.0 h1:6b1oIMlUXIpz//VKEDzPVBK8KG7beVwmHIUEBIs/Pns= github.com/smola/gocompat v0.2.0/go.mod h1:1B0MlxbmoZNo3h8guHp8HztB3BSYR5itql9qtVc0ypY= github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= From c64258a5d407ed9b8635e892d761aaf2d776d265 Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Thu, 26 Mar 2020 17:27:09 +0530 Subject: [PATCH 1770/3965] put signed peer record in address change event --- core/event/addrs.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/core/event/addrs.go b/core/event/addrs.go index 42d3c7d878..cb02835226 100644 --- a/core/event/addrs.go +++ b/core/event/addrs.go @@ -55,6 +55,11 @@ type UpdatedAddress struct { // If the event producer is not capable or producing diffs, the Diffs field will // be false, the Removed list will always be empty, and the Action for each // UpdatedAddress in the Current list will be Unknown. +// +// In addition to the above, EvtLocalAddressesUpdated also contains the updated peer.PeerRecord +// for the Current set of listen addresses, wrapped in a record.Envelope and signed by the Host's private key. +// This record can be shared with other peers to inform them of our listen addresses in +// a secure and authenticated way. type EvtLocalAddressesUpdated struct { // Diffs indicates whether this event contains a diff of the Host's previous @@ -70,13 +75,8 @@ type EvtLocalAddressesUpdated struct { // Removed contains addresses that were removed from the Host. // This field is only set when Diffs == true. Removed []UpdatedAddress -} -// EvtLocalPeerRoutingStateUpdated should be emitted when a new signed PeerRecord -// for the local peer has been produced. This will happen whenever the set of listen -// addresses changes. -type EvtLocalPeerRecordUpdated struct { - // Record contains the updated peer.PeerRecord, wrapped in a record.Envelope and - // signed by the Host's private key. - Record record.Envelope + // SignedPeerRecord contains the updated peer.PeerRecord for the Current set of listen addresses, + // wrapped in a record.Envelope and signed by the Host's private key. + SignedPeerRecord record.Envelope } From 93bcd2ccd1713de836302bf4ae221846875fa3cb Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 27 Mar 2020 00:04:57 +0530 Subject: [PATCH 1771/3965] Update event/addrs.go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Raúl Kripalani --- core/event/addrs.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/event/addrs.go b/core/event/addrs.go index cb02835226..2de6159e3b 100644 --- a/core/event/addrs.go +++ b/core/event/addrs.go @@ -76,7 +76,7 @@ type EvtLocalAddressesUpdated struct { // This field is only set when Diffs == true. Removed []UpdatedAddress - // SignedPeerRecord contains the updated peer.PeerRecord for the Current set of listen addresses, + // SignedPeerRecord contains our own updated peer.PeerRecord, listing the addresses enumerated in Current. // wrapped in a record.Envelope and signed by the Host's private key. SignedPeerRecord record.Envelope } From 5d92d970be23c18fe93d34a9eb6f546aaa0d3ea4 Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 27 Mar 2020 00:05:03 +0530 Subject: [PATCH 1772/3965] Update event/addrs.go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Raúl Kripalani --- core/event/addrs.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/event/addrs.go b/core/event/addrs.go index 2de6159e3b..28bc373941 100644 --- a/core/event/addrs.go +++ b/core/event/addrs.go @@ -58,7 +58,7 @@ type UpdatedAddress struct { // // In addition to the above, EvtLocalAddressesUpdated also contains the updated peer.PeerRecord // for the Current set of listen addresses, wrapped in a record.Envelope and signed by the Host's private key. -// This record can be shared with other peers to inform them of our listen addresses in +// This record can be shared with other peers to inform them of what we believe are our diallable addresses // a secure and authenticated way. type EvtLocalAddressesUpdated struct { From eaba2efe4a620729356dea52bc643671bcfe9e5a Mon Sep 17 00:00:00 2001 From: Vibhav Pant Date: Fri, 27 Mar 2020 11:24:40 +0530 Subject: [PATCH 1773/3965] peer: Add PeerRecordFromProtobuf. --- core/peer/record.go | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/core/peer/record.go b/core/peer/record.go index 60373c36d6..3638c337f3 100644 --- a/core/peer/record.go +++ b/core/peer/record.go @@ -108,6 +108,23 @@ func PeerRecordFromAddrInfo(info AddrInfo) *PeerRecord { return rec } +// PeerRecordFromProtobuf creates a PeerRecord from a protobuf PeerRecord +// struct. +func PeerRecordFromProtobuf(msg *pb.PeerRecord) (*PeerRecord, error) { + record := &PeerRecord{} + + var id ID + if err := id.UnmarshalBinary(msg.PeerId); err != nil { + return nil, err + } + + record.PeerID = id + record.Addrs = addrsFromProtobuf(msg.Addresses) + record.Seq = msg.Seq + + return record, nil +} + // TimestampSeq is a helper to generate a timestamp-based sequence number for a PeerRecord. func TimestampSeq() uint64 { return uint64(time.Now().UnixNano()) @@ -138,14 +155,13 @@ func (r *PeerRecord) UnmarshalRecord(bytes []byte) error { if err != nil { return err } - var id ID - err = id.UnmarshalBinary(msg.PeerId) + + rPtr, err := PeerRecordFromProtobuf(&msg) if err != nil { return err } - r.PeerID = id - r.Addrs = addrsFromProtobuf(msg.Addresses) - r.Seq = msg.Seq + *r = *rPtr + return nil } From 394a5e7d4b0c41474c4240e999ac76675ebb7901 Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Fri, 27 Mar 2020 16:18:53 +0530 Subject: [PATCH 1774/3965] fix memory store signed peer record bug --- p2p/host/peerstore/pstoremem/addr_book.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index c0db4f5d8e..b883902775 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -252,7 +252,7 @@ func (mab *memoryAddrBook) addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du // } // if we've expired all the signed addresses for a peer, remove their signed routing state record - if len(addrs) == 0 { + if len(amap) == 0 { delete(s.signedPeerRecords, p) } } From 82d957e31daa8731f94901b5c40bab128f9cc47c Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Fri, 27 Mar 2020 11:21:34 +0000 Subject: [PATCH 1775/3965] build(deps): bump github.com/libp2p/go-libp2p-yamux from 0.2.5 to 0.2.6 Bumps [github.com/libp2p/go-libp2p-yamux](https://github.com/libp2p/go-libp2p-yamux) from 0.2.5 to 0.2.6. - [Release notes](https://github.com/libp2p/go-libp2p-yamux/releases) - [Commits](https://github.com/libp2p/go-libp2p-yamux/compare/v0.2.5...v0.2.6) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 4044698945..4b0f660b50 100644 --- a/go.mod +++ b/go.mod @@ -24,7 +24,7 @@ require ( github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 - github.com/libp2p/go-libp2p-yamux v0.2.5 + github.com/libp2p/go-libp2p-yamux v0.2.6 github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 diff --git a/go.sum b/go.sum index a154322477..094ac646da 100644 --- a/go.sum +++ b/go.sum @@ -207,6 +207,8 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSo github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= github.com/libp2p/go-libp2p-yamux v0.2.5 h1:MuyItOqz03oi8npvjgMJxgnhllJLZnO/dKVOpTZ9+XI= github.com/libp2p/go-libp2p-yamux v0.2.5/go.mod h1:Zpgj6arbyQrmZ3wxSZxfBmbdnWtbZ48OpsfmQVTErwA= +github.com/libp2p/go-libp2p-yamux v0.2.6 h1:BXl4fEzzZwLUjoqoRJNx4co1UhvOwXaM+Qg2twEqEFU= +github.com/libp2p/go-libp2p-yamux v0.2.6/go.mod h1:YbnSVt8RYb3t6YBGPD9rLUT0vmmvcr/waaQ79PINpOM= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.5 h1:CW3AgbMO6vUvT4kf87y4N+0P8KUl2aqLYhrGyDUbLSg= github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= @@ -238,6 +240,8 @@ github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzl github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.3 h1:mWuzZRCAeTBFdynLlsYgA/EIeMOLr8XY04wa52NRhsE= github.com/libp2p/go-yamux v1.3.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-yamux v1.3.4 h1:CptmtNfQomI9RpM7LFJhVBFkwp2zwEXIskEfWwwtoHg= +github.com/libp2p/go-yamux v1.3.4/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= From 0134a890ee986f0dcd4acb9ed010f16454776ace Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Sat, 28 Mar 2020 16:17:10 +0530 Subject: [PATCH 1776/3965] fix add addrs bug --- p2p/host/peerstore/pstoremem/addr_book.go | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index b883902775..cb7340f34b 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -240,21 +240,6 @@ func (mab *memoryAddrBook) addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du } } } - - // // when adding signed addrs, make sure only the addrs from the input list remain. - // if signed { - // for k := range amap { - // _, ok := addrSet[k] - // if !ok { - // delete(amap, k) - // } - // } - // } - - // if we've expired all the signed addresses for a peer, remove their signed routing state record - if len(amap) == 0 { - delete(s.signedPeerRecords, p) - } } // SetAddr calls mgr.SetAddrs(p, addr, ttl) From 3639cf81a136c029613050c667cf83f4d29dc932 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sun, 29 Mar 2020 21:00:59 -0700 Subject: [PATCH 1777/3965] feat: handle no addresses If we aren't listening on any addresses, we should return no addresses, not an error. --- p2p/net/swarm/swarm_addr.go | 24 ++++++++++++++---------- p2p/net/swarm/swarm_addr_test.go | 20 +++++++++++++++++++- 2 files changed, 33 insertions(+), 11 deletions(-) diff --git a/p2p/net/swarm/swarm_addr.go b/p2p/net/swarm/swarm_addr.go index 650fb79d13..88bc626bd2 100644 --- a/p2p/net/swarm/swarm_addr.go +++ b/p2p/net/swarm/swarm_addr.go @@ -34,7 +34,7 @@ func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { isEOL := time.Now().After(s.listeners.cacheEOL) s.listeners.RUnlock() // RLock end - if ifaceListenAddres != nil && !isEOL { + if !isEOL { // Cache is valid, clone the slice return append(ifaceListenAddres[:0:0], ifaceListenAddres...), nil } @@ -46,16 +46,20 @@ func (s *Swarm) InterfaceListenAddresses() ([]ma.Multiaddr, error) { ifaceListenAddres = s.listeners.ifaceListenAddres isEOL = time.Now().After(s.listeners.cacheEOL) - if ifaceListenAddres == nil || isEOL { + if isEOL { // Cache is still invalid - - var err error - ifaceListenAddres, err = addrutil.ResolveUnspecifiedAddresses( - s.listenAddressesNoLock(), nil) - - if err != nil { - s.listeners.Unlock() // Lock early exit - return nil, err + listenAddres := s.listenAddressesNoLock() + if len(listenAddres) > 0 { + // We're actually listening on addresses. + var err error + ifaceListenAddres, err = addrutil.ResolveUnspecifiedAddresses(listenAddres, nil) + + if err != nil { + s.listeners.Unlock() // Lock early exit + return nil, err + } + } else { + ifaceListenAddres = nil } s.listeners.ifaceListenAddres = ifaceListenAddres diff --git a/p2p/net/swarm/swarm_addr_test.go b/p2p/net/swarm/swarm_addr_test.go index 5009946c6d..baeac46203 100644 --- a/p2p/net/swarm/swarm_addr_test.go +++ b/p2p/net/swarm/swarm_addr_test.go @@ -8,6 +8,8 @@ import ( "github.com/libp2p/go-libp2p-core/test" ma "github.com/multiformats/go-multiaddr" + + swarmt "github.com/libp2p/go-libp2p-swarm/testing" ) func TestDialBadAddrs(t *testing.T) { @@ -37,8 +39,10 @@ func TestDialBadAddrs(t *testing.T) { } func TestAddrRace(t *testing.T) { - ctx := context.Background() + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() s := makeSwarms(ctx, t, 1)[0] + defer s.Close() a1, err := s.InterfaceListenAddresses() if err != nil { @@ -53,3 +57,17 @@ func TestAddrRace(t *testing.T) { t.Fatal("got the exact same address set twice; this could lead to data races") } } + +func TestAddressesWithoutListening(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + s := swarmt.GenSwarm(t, ctx, swarmt.OptDialOnly) + + a1, err := s.InterfaceListenAddresses() + if err != nil { + t.Fatal(err) + } + if len(a1) != 0 { + t.Fatalf("expected to be listening on no addresses, was listening on %d", len(a1)) + } +} From f8e12fcb8e4ffc5eb2c36f945510f434e76d9afd Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 30 Mar 2020 11:20:22 +0000 Subject: [PATCH 1778/3965] build(deps): bump github.com/libp2p/go-libp2p-peerstore Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.1 to 0.2.2. - [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases) - [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.1...v0.2.2) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 4b0f660b50..2b318b7db1 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/libp2p/go-libp2p-mplex v0.2.2 github.com/libp2p/go-libp2p-nat v0.0.5 github.com/libp2p/go-libp2p-netutil v0.1.0 - github.com/libp2p/go-libp2p-peerstore v0.2.1 + github.com/libp2p/go-libp2p-peerstore v0.2.2 github.com/libp2p/go-libp2p-secio v0.2.1 github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 diff --git a/go.sum b/go.sum index 094ac646da..149f4e2005 100644 --- a/go.sum +++ b/go.sum @@ -186,6 +186,8 @@ github.com/libp2p/go-libp2p-peerstore v0.2.0 h1:XcgJhI8WyUOCbHyRLNEX5542YNj8hnLS github.com/libp2p/go-libp2p-peerstore v0.2.0/go.mod h1:N2l3eVIeAitSg3Pi2ipSrJYnqhVnMNQZo9nkSCuAbnQ= github.com/libp2p/go-libp2p-peerstore v0.2.1 h1:u+gOfsKgu73ZkGWhvckRm03z9C+iS9TrLqpANweELGs= github.com/libp2p/go-libp2p-peerstore v0.2.1/go.mod h1:NQxhNjWxf1d4w6PihR8btWIRjwRLBr4TYKfNgrUkOPA= +github.com/libp2p/go-libp2p-peerstore v0.2.2 h1:iqc/m03jHn5doXN3+kS6JKvqQRHEltiXljQB85iVHWE= +github.com/libp2p/go-libp2p-peerstore v0.2.2/go.mod h1:NQxhNjWxf1d4w6PihR8btWIRjwRLBr4TYKfNgrUkOPA= github.com/libp2p/go-libp2p-pnet v0.2.0 h1:J6htxttBipJujEjz1y0a5+eYoiPcFHhSYHH6na5f0/k= github.com/libp2p/go-libp2p-pnet v0.2.0/go.mod h1:Qqvq6JH/oMZGwqs3N1Fqhv8NVhrdYcO0BW4wssv21LA= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= From d4146fbf230ffc7c46cc165761908501cb5db8c1 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 30 Mar 2020 14:05:40 -0700 Subject: [PATCH 1779/3965] feat: switch to go-netroute This library makes the correct syscalls instead of shelling out to a binary. --- gateway.go | 17 +++++++++++++++++ go.mod | 2 +- go.sum | 17 +++++++++++++---- nat.go | 4 +--- natpmp.go | 35 +++++++++++++++++------------------ 5 files changed, 49 insertions(+), 26 deletions(-) create mode 100644 gateway.go diff --git a/gateway.go b/gateway.go new file mode 100644 index 0000000000..a87e46a892 --- /dev/null +++ b/gateway.go @@ -0,0 +1,17 @@ +package nat + +import ( + "net" + + "github.com/libp2p/go-netroute" +) + +func getDefaultGateway() (net.IP, error) { + router, err := netroute.New() + if err != nil { + return nil, err + } + + _, ip, _, err := router.Route(net.IPv4zero) + return ip, err +} diff --git a/go.mod b/go.mod index 3b3c622235..605525ba0c 100644 --- a/go.mod +++ b/go.mod @@ -2,9 +2,9 @@ module github.com/libp2p/go-nat require ( github.com/huin/goupnp v1.0.0 - github.com/jackpal/gateway v1.0.5 github.com/jackpal/go-nat-pmp v1.0.2 github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d + github.com/libp2p/go-netroute v0.1.2 ) go 1.12 diff --git a/go.sum b/go.sum index 547800b2b7..70ef62784e 100644 --- a/go.sum +++ b/go.sum @@ -1,15 +1,24 @@ +github.com/google/gopacket v1.1.17 h1:rMrlX2ZY2UbvT+sdz3+6J+pp2z+msCq9MxTU6ymxbBY= +github.com/google/gopacket v1.1.17/go.mod h1:UdDNZ1OO62aGYVnPhxT1U6aI7ukYtA/kB8vaU0diBUM= github.com/huin/goupnp v1.0.0 h1:wg75sLpL6DZqwHQN6E1Cfk6mtfzS45z8OV+ic+DtHRo= github.com/huin/goupnp v1.0.0/go.mod h1:n9v9KO1tAxYH82qOn+UTIFQDmx5n1Zxd/ClZDMX7Bnc= github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150/go.mod h1:PpLOETDnJ0o3iZrZfqZzyLl6l7F3c6L1oWn7OICBi6o= -github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc= -github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= -github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= -github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d h1:68u9r4wEvL3gYg2jvAOgROwZ3H+Y3hIDk4tbbmIjcYQ= github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= +github.com/libp2p/go-netroute v0.1.2 h1:UHhB35chwgvcRI392znJA3RCBtZ3MpE3ahNCN5MR4Xg= +github.com/libp2p/go-netroute v0.1.2/go.mod h1:jZLDV+1PE8y5XxBySEBgbuVAXbhtuHSdmLPL2n9MKbk= +github.com/libp2p/go-sockaddr v0.0.2 h1:tCuXfpA9rq7llM/v834RKc/Xvovy/AqM9kHvTV/jY/Q= +github.com/libp2p/go-sockaddr v0.0.2/go.mod h1:syPvOmNs24S3dFVGJA1/mrqdeijPxLV2Le3BRLKd68k= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1 h1:Y/KGZSOdz/2r0WJ9Mkmz6NJBusp0kiNx1Cn82lzJQ6w= golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190405154228-4b34438f7a67 h1:1Fzlr8kkDLQwqMP8GxrhptBLqZG/EDpiATneiZHY998= +golang.org/x/sys v0.0.0-20190405154228-4b34438f7a67/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= diff --git a/nat.go b/nat.go index d38ad52c12..a2606794fe 100644 --- a/nat.go +++ b/nat.go @@ -8,8 +8,6 @@ import ( "math/rand" "net" "time" - - "github.com/jackpal/gateway" ) var ErrNoExternalAddress = errors.New("no external address") @@ -101,7 +99,7 @@ func DiscoverGateway() (NAT, error) { case 1: return nats[0], nil } - gw, _ := gateway.DiscoverGateway() + gw, _ := getDefaultGateway() bestNAT := nats[0] natGw, _ := bestNAT.GetDeviceAddress() bestNATIsGw := gw != nil && natGw.Equal(gw) diff --git a/natpmp.go b/natpmp.go index a31a499988..495d42b49e 100644 --- a/natpmp.go +++ b/natpmp.go @@ -5,7 +5,6 @@ import ( "net" "time" - "github.com/jackpal/gateway" "github.com/jackpal/go-nat-pmp" ) @@ -16,24 +15,24 @@ var ( func discoverNATPMP(ctx context.Context) <-chan NAT { res := make(chan NAT, 1) - ip, err := gateway.DiscoverGateway() - if err == nil { - go func() { - defer close(res) - // Unfortunately, we can't actually _stop_ the natpmp - // library. However, we can at least close _our_ channel - // and walk away. - select { - case client, ok := <-discoverNATPMPWithAddr(ip): - if ok { - res <- &natpmpNAT{client, ip, make(map[int]int)} - } - case <-ctx.Done(): - } - }() - } else { - close(res) + ip, err := getDefaultGateway() + if err != nil { + return nil } + + go func() { + defer close(res) + // Unfortunately, we can't actually _stop_ the natpmp + // library. However, we can at least close _our_ channel + // and walk away. + select { + case client, ok := <-discoverNATPMPWithAddr(ip): + if ok { + res <- &natpmpNAT{client, ip, make(map[int]int)} + } + case <-ctx.Done(): + } + }() return res } From 904165fd3170186b45ad044b9143533b7603c1ce Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 31 Mar 2020 08:57:02 -0700 Subject: [PATCH 1780/3965] fix: set teardown after storing the context. Otherwise, we can modify the context after/while the process is shutting down. fixes #189 --- p2p/net/swarm/swarm.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index f85ef5234b..fe4e1ee978 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -112,9 +112,13 @@ func NewSwarm(ctx context.Context, local peer.ID, peers peerstore.Peerstore, bwc s.dsync = NewDialSync(s.doDial) s.limiter = newDialLimiter(s.dialAddr) - s.proc = goprocessctx.WithContextAndTeardown(ctx, s.teardown) + s.proc = goprocessctx.WithContext(ctx) s.ctx = goprocessctx.OnClosingContext(s.proc) + // Set teardown after setting the context/process so we don't start the + // teardown process early. + s.proc.SetTeardown(s.teardown) + return s } From 7fff4588422de5b757f082939ca9a6e271c0ecdd Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 31 Mar 2020 12:08:27 -0700 Subject: [PATCH 1781/3965] feat: remove relay discovery and unspecified relay dialing This was a misfeature that looks great in demos but isn't useful in practice. First, random relays we discover are unlikely to support _active_ relaying. This means that picking a random relay is usually going to fail because the random relay won't already be connected to the target. Second, there are two use-cases for relaying: 1. Connecting to undialable nodes. 2. Relaying all outbound traffic. For the first case, we only want to use the relays specified by the target peers. For the second case, we'd need to modify either libp2p's dialer logic (in the swarm) to prefix all addresses with the specified relay. The logic _here_ covers neither use-case. --- .../internal/circuitv1-deprecated/dial.go | 46 +++------------ .../internal/circuitv1-deprecated/notify.go | 54 ------------------ .../internal/circuitv1-deprecated/relay.go | 30 +++++----- .../circuitv1-deprecated/relay_test.go | 57 ++++++------------- .../circuitv1-deprecated/transport_test.go | 18 ++---- 5 files changed, 42 insertions(+), 163 deletions(-) delete mode 100644 p2p/protocol/internal/circuitv1-deprecated/notify.go diff --git a/p2p/protocol/internal/circuitv1-deprecated/dial.go b/p2p/protocol/internal/circuitv1-deprecated/dial.go index 1b6e9db0f4..a81715a5ee 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/dial.go +++ b/p2p/protocol/internal/circuitv1-deprecated/dial.go @@ -3,7 +3,6 @@ package relay import ( "context" "fmt" - "math/rand" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/transport" @@ -30,6 +29,13 @@ func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (*Conn, err return nil, fmt.Errorf("%s is not a relay address", a) } + if relayaddr == nil { + return nil, fmt.Errorf( + "can't dial a p2p-circuit without specifying a relay: %s", + a, + ) + } + // Strip the /p2p-circuit prefix from the destaddr. _, destaddr = ma.SplitFirst(destaddr) @@ -38,11 +44,6 @@ func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (*Conn, err dinfo.Addrs = append(dinfo.Addrs, destaddr) } - if relayaddr == nil { - // unspecific relay address, try dialing using known hop relays - return r.tryDialRelays(ctx, *dinfo) - } - var rinfo *peer.AddrInfo rinfo, err := peer.AddrInfoFromP2pAddr(relayaddr) if err != nil { @@ -51,36 +52,3 @@ func (r *Relay) Dial(ctx context.Context, a ma.Multiaddr, p peer.ID) (*Conn, err return r.DialPeer(ctx, *rinfo, *dinfo) } - -func (r *Relay) tryDialRelays(ctx context.Context, dinfo peer.AddrInfo) (*Conn, error) { - var relays []peer.ID - r.mx.Lock() - for p := range r.relays { - relays = append(relays, p) - } - r.mx.Unlock() - - // shuffle list of relays, avoid overloading a specific relay - for i := range relays { - j := rand.Intn(i + 1) - relays[i], relays[j] = relays[j], relays[i] - } - - for _, relay := range relays { - if len(r.host.Network().ConnsToPeer(relay)) == 0 { - continue - } - - rctx, cancel := context.WithTimeout(ctx, HopConnectTimeout) - c, err := r.DialPeer(rctx, peer.AddrInfo{ID: relay}, dinfo) - cancel() - - if err == nil { - return c, nil - } - - log.Debugf("error opening relay connection through %s: %s", dinfo.ID, err.Error()) - } - - return nil, fmt.Errorf("Failed to dial through %d known relay hosts", len(relays)) -} diff --git a/p2p/protocol/internal/circuitv1-deprecated/notify.go b/p2p/protocol/internal/circuitv1-deprecated/notify.go deleted file mode 100644 index 813daed4d9..0000000000 --- a/p2p/protocol/internal/circuitv1-deprecated/notify.go +++ /dev/null @@ -1,54 +0,0 @@ -package relay - -import ( - "context" - "time" - - inet "github.com/libp2p/go-libp2p-core/network" - peer "github.com/libp2p/go-libp2p-core/peer" - ma "github.com/multiformats/go-multiaddr" -) - -var _ inet.Notifiee = (*RelayNotifiee)(nil) - -type RelayNotifiee Relay - -func (r *Relay) notifiee() inet.Notifiee { - return (*RelayNotifiee)(r) -} - -func (n *RelayNotifiee) Relay() *Relay { - return (*Relay)(n) -} - -func (n *RelayNotifiee) Listen(net inet.Network, a ma.Multiaddr) {} -func (n *RelayNotifiee) ListenClose(net inet.Network, a ma.Multiaddr) {} -func (n *RelayNotifiee) OpenedStream(net inet.Network, s inet.Stream) {} -func (n *RelayNotifiee) ClosedStream(net inet.Network, s inet.Stream) {} - -func (n *RelayNotifiee) Connected(s inet.Network, c inet.Conn) { - if n.Relay().Matches(c.RemoteMultiaddr()) { - return - } - - go func(id peer.ID) { - ctx, cancel := context.WithTimeout(n.ctx, time.Second) - defer cancel() - - canhop, err := n.Relay().CanHop(ctx, id) - if err != nil { - log.Debugf("Error testing relay hop: %s", err.Error()) - return - } - - if canhop { - log.Debugf("Discovered hop relay %s", id.Pretty()) - n.mx.Lock() - n.relays[id] = struct{}{} - n.mx.Unlock() - n.host.ConnManager().TagPeer(id, "relay-hop", 2) - } - }(c.RemotePeer()) -} - -func (n *RelayNotifiee) Disconnected(s inet.Network, c inet.Conn) {} diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index fcb1b5e262..10816233bb 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -4,7 +4,6 @@ import ( "context" "fmt" "io" - "sync" "sync/atomic" "time" @@ -45,15 +44,11 @@ type Relay struct { ctx context.Context self peer.ID - active bool - hop bool - discovery bool + active bool + hop bool incoming chan *Conn - relays map[peer.ID]struct{} - mx sync.Mutex - // atomic counters streamCount int32 liveHopCount int32 @@ -72,9 +67,14 @@ var ( // this will only relay traffic between peers already connected to this // node. OptHop = RelayOpt(1) - // OptDiscovery configures this relay transport to discover new relays - // by probing every new peer. You almost _certainly_ don't want to - // enable this. + // OptDiscovery is a no-op. It was introduced as a way to probe new + // peers to see if they were willing to act as a relays. However, in + // practice, it's useless. While it does test to see if these peers are + // relays, it doesn't (and can't), check to see if these peers are + // _active_ relays (i.e., will actively dial the target peer). + // + // This option may be re-enabled in the future but for now you shouldn't + // use it. OptDiscovery = RelayOpt(2) ) @@ -94,7 +94,6 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. ctx: ctx, self: h.ID(), incoming: make(chan *Conn), - relays: make(map[peer.ID]struct{}), } for _, opt := range opts { @@ -104,7 +103,10 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. case OptHop: r.hop = true case OptDiscovery: - r.discovery = true + log.Errorf( + "circuit.OptDiscovery is now a no-op: %s", + "dialing peers with a random relay is no longer supported", + ) default: return nil, fmt.Errorf("unrecognized option: %d", opt) } @@ -112,10 +114,6 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. h.SetStreamHandler(ProtoID, r.handleNewStream) - if r.discovery { - h.Network().Notify(r.notifiee()) - } - return r, nil } diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go index b23c2e0d44..220249d5f8 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay_test.go @@ -69,7 +69,7 @@ func TestBasicRelay(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) + r1 := newTestRelay(t, ctx, hosts[0]) newTestRelay(t, ctx, hosts[1], OptHop) @@ -143,7 +143,7 @@ func TestRelayReset(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) + r1 := newTestRelay(t, ctx, hosts[0]) newTestRelay(t, ctx, hosts[1], OptHop) @@ -201,7 +201,7 @@ func TestBasicRelayDial(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) + r1 := newTestRelay(t, ctx, hosts[0]) _ = newTestRelay(t, ctx, hosts[1], OptHop) r3 := newTestRelay(t, ctx, hosts[2]) @@ -263,13 +263,12 @@ func TestBasicRelayDial(t *testing.T) { } } -func TestUnspecificRelayDial(t *testing.T) { +func TestUnspecificRelayDialFails(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) - defer cancel() hosts := getNetHosts(t, ctx, 3) - r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) + r1 := newTestRelay(t, ctx, hosts[0]) newTestRelay(t, ctx, hosts[1], OptHop) @@ -281,36 +280,22 @@ func TestUnspecificRelayDial(t *testing.T) { time.Sleep(100 * time.Millisecond) var ( - conn1, conn2 net.Conn - done = make(chan struct{}) + done = make(chan struct{}) ) defer func() { + cancel() <-done - if conn1 != nil { - conn1.Close() - } - if conn2 != nil { - conn2.Close() - } }() - msg := []byte("relay works!") go func() { defer close(done) list := r3.Listener() var err error - conn1, err = list.Accept() - if err != nil { - t.Error(err) - return - } - - _, err = conn1.Write(msg) - if err != nil { - t.Error(err) - return + _, err = list.Accept() + if err == nil { + t.Error("should not have received relay connection") } }() @@ -320,19 +305,9 @@ func TestUnspecificRelayDial(t *testing.T) { defer rcancel() var err error - conn2, err = r1.Dial(rctx, addr, hosts[2].ID()) - if err != nil { - t.Fatal(err) - } - - data := make([]byte, len(msg)) - _, err = io.ReadFull(conn2, data) - if err != nil { - t.Fatal(err) - } - - if !bytes.Equal(data, msg) { - t.Fatal("message was incorrect:", string(data)) + _, err = r1.Dial(rctx, addr, hosts[2].ID()) + if err == nil { + t.Fatal("expected dial with unspecified relay address to fail, even if we're connected to a relay") } } @@ -347,7 +322,7 @@ func TestRelayThroughNonHop(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) + r1 := newTestRelay(t, ctx, hosts[0]) newTestRelay(t, ctx, hosts[1]) @@ -384,7 +359,7 @@ func TestRelayNoDestConnection(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) + r1 := newTestRelay(t, ctx, hosts[0]) newTestRelay(t, ctx, hosts[1], OptHop) @@ -419,7 +394,7 @@ func TestActiveRelay(t *testing.T) { time.Sleep(10 * time.Millisecond) - r1 := newTestRelay(t, ctx, hosts[0], OptDiscovery) + r1 := newTestRelay(t, ctx, hosts[0]) newTestRelay(t, ctx, hosts[1], OptHop, OptActive) diff --git a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go index 960ffaf8ce..2d81c6bc32 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/transport_test.go +++ b/p2p/protocol/internal/circuitv1-deprecated/transport_test.go @@ -26,7 +26,7 @@ var msg = []byte("relay works!") func testSetupRelay(t *testing.T, ctx context.Context) []host.Host { hosts := getNetHosts(t, ctx, 3) - err := AddRelayTransport(ctx, hosts[0], swarmt.GenUpgrader(hosts[0].Network().(*swarm.Swarm)), OptDiscovery) + err := AddRelayTransport(ctx, hosts[0], swarmt.GenUpgrader(hosts[0].Network().(*swarm.Swarm))) if err != nil { t.Fatal(err) } @@ -121,7 +121,7 @@ func TestSpecificRelayTransportDial(t *testing.T) { } } -func TestUnspecificRelayTransportDial(t *testing.T) { +func TestUnspecificRelayTransportDialFails(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -137,17 +137,9 @@ func TestUnspecificRelayTransportDial(t *testing.T) { hosts[0].Peerstore().AddAddrs(hosts[2].ID(), []ma.Multiaddr{addr}, peerstore.TempAddrTTL) - s, err := hosts[0].NewStream(rctx, hosts[2].ID(), TestProto) - if err != nil { - t.Fatal(err) + _, err = hosts[0].NewStream(rctx, hosts[2].ID(), TestProto) + if err == nil { + t.Fatal("dial to unspecified address should have failed") } - data, err := ioutil.ReadAll(s) - if err != nil { - t.Fatal(err) - } - - if !bytes.Equal(data, msg) { - t.Fatal("message was incorrect:", string(data)) - } } From bf52b405f378fe279d48e01bd4374d2dcc539382 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 31 Mar 2020 12:18:52 -0700 Subject: [PATCH 1782/3965] feat: functional options Introduce functional options. This should make it easier for us to add additional options in the future (e.g., known relays, relay policies, etc.). --- .../internal/circuitv1-deprecated/relay.go | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 10816233bb..320533e259 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -55,18 +55,24 @@ type Relay struct { } // RelayOpts are options for configuring the relay transport. -type RelayOpt int +type RelayOpt func(*Relay) error var ( // OptActive configures the relay transport to actively establish // outbound connections on behalf of clients. You probably don't want to // enable this unless you know what you're doing. - OptActive = RelayOpt(0) + OptActive RelayOpt = func(r *Relay) error { + r.active = true + return nil + } // OptHop configures the relay transport to accept requests to relay // traffic on behalf of third-parties. Unless OptActive is specified, // this will only relay traffic between peers already connected to this // node. - OptHop = RelayOpt(1) + OptHop = func(r *Relay) error { + r.hop = true + return nil + } // OptDiscovery is a no-op. It was introduced as a way to probe new // peers to see if they were willing to act as a relays. However, in // practice, it's useless. While it does test to see if these peers are @@ -75,7 +81,13 @@ var ( // // This option may be re-enabled in the future but for now you shouldn't // use it. - OptDiscovery = RelayOpt(2) + OptDiscovery = func(r *Relay) error { + log.Errorf( + "circuit.OptDiscovery is now a no-op: %s", + "dialing peers with a random relay is no longer supported", + ) + return nil + } ) type RelayError struct { @@ -97,18 +109,8 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. } for _, opt := range opts { - switch opt { - case OptActive: - r.active = true - case OptHop: - r.hop = true - case OptDiscovery: - log.Errorf( - "circuit.OptDiscovery is now a no-op: %s", - "dialing peers with a random relay is no longer supported", - ) - default: - return nil, fmt.Errorf("unrecognized option: %d", opt) + if err := opt(r); err != nil { + return nil, err } } From 193a3191a480bdf62832eb6eac71b6669f0cace2 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 1 Apr 2020 14:39:49 -0700 Subject: [PATCH 1783/3965] change backoffs to per-address --- p2p/net/swarm/dial_test.go | 19 ++++++----- p2p/net/swarm/swarm_dial.go | 65 +++++++++++++++++++++++++------------ 2 files changed, 54 insertions(+), 30 deletions(-) diff --git a/p2p/net/swarm/dial_test.go b/p2p/net/swarm/dial_test.go index 73bd02c674..49c21fd6d3 100644 --- a/p2p/net/swarm/dial_test.go +++ b/p2p/net/swarm/dial_test.go @@ -200,7 +200,7 @@ func TestDialWait(t *testing.T) { t.Error("> 2*transport.DialTimeout * DialAttempts not being respected", duration, 2*transport.DialTimeout*DialAttempts) } - if !s1.Backoff().Backoff(s2p) { + if !s1.Backoff().Backoff(s2p, s2addr) { t.Error("s2 should now be on backoff") } } @@ -337,10 +337,10 @@ func TestDialBackoff(t *testing.T) { } // check backoff state - if s1.Backoff().Backoff(s2.LocalPeer()) { + if s1.Backoff().Backoff(s2.LocalPeer(), s2addrs[0]) { t.Error("s2 should not be on backoff") } - if !s1.Backoff().Backoff(s3p) { + if !s1.Backoff().Backoff(s3p, s3addr) { t.Error("s3 should be on backoff") } @@ -407,10 +407,10 @@ func TestDialBackoff(t *testing.T) { } // check backoff state (the same) - if s1.Backoff().Backoff(s2.LocalPeer()) { + if s1.Backoff().Backoff(s2.LocalPeer(), s2addrs[0]) { t.Error("s2 should not be on backoff") } - if !s1.Backoff().Backoff(s3p) { + if !s1.Backoff().Backoff(s3p, s3addr) { t.Error("s3 should be on backoff") } } @@ -451,7 +451,7 @@ func TestDialBackoffClears(t *testing.T) { t.Error("> 2*transport.DialTimeout * DialAttempts not being respected", duration, 2*transport.DialTimeout*DialAttempts) } - if !s1.Backoff().Backoff(s2.LocalPeer()) { + if !s1.Backoff().Backoff(s2.LocalPeer(), s2bad) { t.Error("s2 should now be on backoff") } else { t.Log("correctly added to backoff") @@ -464,8 +464,9 @@ func TestDialBackoffClears(t *testing.T) { } s1.Peerstore().AddAddrs(s2.LocalPeer(), ifaceAddrs1, peerstore.PermanentAddrTTL) - if _, err := s1.DialPeer(ctx, s2.LocalPeer()); err == nil { - t.Fatal("should have failed to dial backed off peer") + if c, err := s1.DialPeer(ctx, s2.LocalPeer()); err == nil { + c.Close() + t.Log("backoffs are per address, not peer") } time.Sleep(BackoffBase) @@ -477,7 +478,7 @@ func TestDialBackoffClears(t *testing.T) { t.Log("correctly connected") } - if s1.Backoff().Backoff(s2.LocalPeer()) { + if s1.Backoff().Backoff(s2.LocalPeer(), s2bad) { t.Error("s2 should no longer be on backoff") } else { t.Log("correctly cleared backoff") diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index f2b21abf0a..40821644bd 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -101,7 +101,9 @@ type DialBackoff struct { lock sync.RWMutex } -type backoffPeer struct { +type backoffPeer map[ma.Multiaddr]*backoffAddr + +type backoffAddr struct { tries int until time.Time } @@ -113,14 +115,18 @@ func (db *DialBackoff) init() { } // Backoff returns whether the client should backoff from dialing -// peer p -func (db *DialBackoff) Backoff(p peer.ID) (backoff bool) { +// peer p at address addr +func (db *DialBackoff) Backoff(p peer.ID, addr ma.Multiaddr) (backoff bool) { db.lock.Lock() defer db.lock.Unlock() db.init() bp, found := db.entries[p] - if found && time.Now().Before(bp.until) { - return true + if found && bp != nil { + ap, found := (*bp)[addr] + // TODO: cleanup out of date entries. + if found && time.Now().Before(ap.until) { + return true + } } return false @@ -145,25 +151,36 @@ var BackoffMax = time.Minute * 5 // BackoffBase + BakoffCoef * PriorBackoffs^2 // // Where PriorBackoffs is the number of previous backoffs. -func (db *DialBackoff) AddBackoff(p peer.ID) { +func (db *DialBackoff) AddBackoff(p peer.ID, addr ma.Multiaddr) { db.lock.Lock() defer db.lock.Unlock() db.init() bp, ok := db.entries[p] if !ok { - db.entries[p] = &backoffPeer{ + bp := backoffPeer(make(map[ma.Multiaddr]*backoffAddr)) + db.entries[p] = &bp + bp[addr] = &backoffAddr{ + tries: 1, + until: time.Now().Add(BackoffBase), + } + return + } + // todo: cleanup out of date entries. + ba, ok := (*bp)[addr] + if !ok { + (*bp)[addr] = &backoffAddr{ tries: 1, until: time.Now().Add(BackoffBase), } return } - backoffTime := BackoffBase + BackoffCoef*time.Duration(bp.tries*bp.tries) + backoffTime := BackoffBase + BackoffCoef*time.Duration(ba.tries*ba.tries) if backoffTime > BackoffMax { backoffTime = BackoffMax } - bp.until = time.Now().Add(backoffTime) - bp.tries++ + ba.until = time.Now().Add(backoffTime) + ba.tries++ } // Clear removes a backoff record. Clients should call this after a @@ -210,12 +227,6 @@ func (s *Swarm) dialPeer(ctx context.Context, p peer.ID) (*Conn, error) { return conn, nil } - // if this peer has been backed off, lets get out of here - if s.backf.Backoff(p) { - log.Event(ctx, "swarmDialBackoff", p) - return nil, ErrDialBackoff - } - // apply the DialPeer timeout ctx, cancel := context.WithTimeout(ctx, network.GetDialPeerTimeout(ctx)) defer cancel() @@ -268,10 +279,6 @@ func (s *Swarm) doDial(ctx context.Context, p peer.ID) (*Conn, error) { log.Debugf("ignoring dial error because we have a connection: %s", err) return conn, nil } - if err != context.Canceled { - log.Event(ctx, "swarmDialBackoffAdd", logdial) - s.backf.AddBackoff(p) // let others know to backoff - } // ok, we failed. return nil, err @@ -318,10 +325,18 @@ func (s *Swarm) dial(ctx context.Context, p peer.ID) (*Conn, error) { return nil, &DialError{Peer: p, Cause: ErrNoGoodAddresses} } goodAddrsChan := make(chan ma.Multiaddr, len(goodAddrs)) + nonBackoff := false for _, a := range goodAddrs { - goodAddrsChan <- a + // skip addresses in back-off + if !s.backf.Backoff(p, a) { + nonBackoff = true + goodAddrsChan <- a + } } close(goodAddrsChan) + if !nonBackoff { + return nil, ErrDialBackoff + } ///////// // try to get a connection to any addr @@ -402,6 +417,10 @@ dialLoop: active-- if resp.Err != nil { // Errors are normal, lots of dials will fail + if resp.Err != context.Canceled { + s.backf.AddBackoff(p, resp.Addr) + } + log.Infof("got error on dial: %s", resp.Err) err.recordErr(resp.Addr, resp.Err) } else if resp.Conn != nil { @@ -429,6 +448,10 @@ dialLoop: active-- if resp.Err != nil { // Errors are normal, lots of dials will fail + if resp.Err != context.Canceled { + s.backf.AddBackoff(p, resp.Addr) + } + log.Infof("got error on dial: %s", resp.Err) err.recordErr(resp.Addr, resp.Err) } else if resp.Conn != nil { From ac37d0dd17212a9c17adaaa28076471f64561210 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 1 Apr 2020 15:19:38 -0700 Subject: [PATCH 1784/3965] add background cleanup task --- p2p/net/swarm/swarm.go | 1 + p2p/net/swarm/swarm_dial.go | 57 ++++++++++++++++++++++++++++--------- 2 files changed, 45 insertions(+), 13 deletions(-) diff --git a/p2p/net/swarm/swarm.go b/p2p/net/swarm/swarm.go index fe4e1ee978..198d88a72b 100644 --- a/p2p/net/swarm/swarm.go +++ b/p2p/net/swarm/swarm.go @@ -114,6 +114,7 @@ func NewSwarm(ctx context.Context, local peer.ID, peers peerstore.Peerstore, bwc s.limiter = newDialLimiter(s.dialAddr) s.proc = goprocessctx.WithContext(ctx) s.ctx = goprocessctx.OnClosingContext(s.proc) + s.backf.init(s.ctx) // Set teardown after setting the context/process so we don't start the // teardown process early. diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 40821644bd..9c2886fcb3 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -97,7 +97,7 @@ const DefaultPerPeerRateLimit = 8 // * It's thread-safe. // * It's *not* safe to move this type after using. type DialBackoff struct { - entries map[peer.ID]*backoffPeer + entries map[peer.ID]backoffPeer lock sync.RWMutex } @@ -108,9 +108,23 @@ type backoffAddr struct { until time.Time } -func (db *DialBackoff) init() { +func (db *DialBackoff) init(ctx context.Context) { if db.entries == nil { - db.entries = make(map[peer.ID]*backoffPeer) + db.entries = make(map[peer.ID]backoffPeer) + } + go db.background(ctx) +} + +func (db *DialBackoff) background(ctx context.Context) { + ticker := time.NewTicker(BackoffMax) + for { + select { + case <-ctx.Done(): + ticker.Stop() + return + case <-ticker.C: + db.cleanup() + } } } @@ -119,10 +133,9 @@ func (db *DialBackoff) init() { func (db *DialBackoff) Backoff(p peer.ID, addr ma.Multiaddr) (backoff bool) { db.lock.Lock() defer db.lock.Unlock() - db.init() bp, found := db.entries[p] if found && bp != nil { - ap, found := (*bp)[addr] + ap, found := bp[addr] // TODO: cleanup out of date entries. if found && time.Now().Before(ap.until) { return true @@ -154,21 +167,18 @@ var BackoffMax = time.Minute * 5 func (db *DialBackoff) AddBackoff(p peer.ID, addr ma.Multiaddr) { db.lock.Lock() defer db.lock.Unlock() - db.init() bp, ok := db.entries[p] if !ok { - bp := backoffPeer(make(map[ma.Multiaddr]*backoffAddr)) - db.entries[p] = &bp - bp[addr] = &backoffAddr{ + db.entries[p] = backoffPeer(make(map[ma.Multiaddr]*backoffAddr)) + db.entries[p][addr] = &backoffAddr{ tries: 1, until: time.Now().Add(BackoffBase), } return } - // todo: cleanup out of date entries. - ba, ok := (*bp)[addr] + ba, ok := bp[addr] if !ok { - (*bp)[addr] = &backoffAddr{ + bp[addr] = &backoffAddr{ tries: 1, until: time.Now().Add(BackoffBase), } @@ -188,10 +198,31 @@ func (db *DialBackoff) AddBackoff(p peer.ID, addr ma.Multiaddr) { func (db *DialBackoff) Clear(p peer.ID) { db.lock.Lock() defer db.lock.Unlock() - db.init() delete(db.entries, p) } +func (db *DialBackoff) cleanup() { + db.lock.Lock() + defer db.lock.Unlock() + now := time.Now() + deletePeers := []peer.ID{} + for p, e := range db.entries { + good := false + for _, backoff := range e { + if now.Before(backoff.until) { + good = true + break + } + } + if !good { + deletePeers = append(deletePeers, p) + } + } + for _, p := range deletePeers { + delete(db.entries, p) + } +} + // DialPeer connects to a peer. // // The idea is that the client of Swarm does not need to know what network From 390b8ca72567672433d8c1065b6da57bdfb5bca1 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 2 Apr 2020 09:15:44 -0700 Subject: [PATCH 1785/3965] simplify data structure --- p2p/net/swarm/swarm_dial.go | 28 +++++++++++----------------- 1 file changed, 11 insertions(+), 17 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 9c2886fcb3..75c9ad26ba 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -97,12 +97,10 @@ const DefaultPerPeerRateLimit = 8 // * It's thread-safe. // * It's *not* safe to move this type after using. type DialBackoff struct { - entries map[peer.ID]backoffPeer + entries map[peer.ID]map[string]*backoffAddr lock sync.RWMutex } -type backoffPeer map[ma.Multiaddr]*backoffAddr - type backoffAddr struct { tries int until time.Time @@ -110,17 +108,17 @@ type backoffAddr struct { func (db *DialBackoff) init(ctx context.Context) { if db.entries == nil { - db.entries = make(map[peer.ID]backoffPeer) + db.entries = make(map[peer.ID]map[string]*backoffAddr) } go db.background(ctx) } func (db *DialBackoff) background(ctx context.Context) { ticker := time.NewTicker(BackoffMax) + defer ticker.Stop() for { select { case <-ctx.Done(): - ticker.Stop() return case <-ticker.C: db.cleanup() @@ -134,9 +132,8 @@ func (db *DialBackoff) Backoff(p peer.ID, addr ma.Multiaddr) (backoff bool) { db.lock.Lock() defer db.lock.Unlock() bp, found := db.entries[p] - if found && bp != nil { - ap, found := bp[addr] - // TODO: cleanup out of date entries. + if found { + ap, found := bp[string(addr.Bytes())] if found && time.Now().Before(ap.until) { return true } @@ -165,20 +162,21 @@ var BackoffMax = time.Minute * 5 // // Where PriorBackoffs is the number of previous backoffs. func (db *DialBackoff) AddBackoff(p peer.ID, addr ma.Multiaddr) { + saddr := string(addr.Bytes()) db.lock.Lock() defer db.lock.Unlock() bp, ok := db.entries[p] if !ok { - db.entries[p] = backoffPeer(make(map[ma.Multiaddr]*backoffAddr)) - db.entries[p][addr] = &backoffAddr{ + db.entries[p] = make(map[string]*backoffAddr) + db.entries[p][saddr] = &backoffAddr{ tries: 1, until: time.Now().Add(BackoffBase), } return } - ba, ok := bp[addr] + ba, ok := bp[saddr] if !ok { - bp[addr] = &backoffAddr{ + bp[saddr] = &backoffAddr{ tries: 1, until: time.Now().Add(BackoffBase), } @@ -205,7 +203,6 @@ func (db *DialBackoff) cleanup() { db.lock.Lock() defer db.lock.Unlock() now := time.Now() - deletePeers := []peer.ID{} for p, e := range db.entries { good := false for _, backoff := range e { @@ -215,12 +212,9 @@ func (db *DialBackoff) cleanup() { } } if !good { - deletePeers = append(deletePeers, p) + delete(db.entries, p) } } - for _, p := range deletePeers { - delete(db.entries, p) - } } // DialPeer connects to a peer. From 035e5dcada370ef5d01ea77e81d32afbac52d895 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Apr 2020 11:36:55 -0700 Subject: [PATCH 1786/3965] chore: slightly simplify backoff logic --- p2p/net/swarm/swarm_dial.go | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 75c9ad26ba..970cf0105e 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -131,15 +131,9 @@ func (db *DialBackoff) background(ctx context.Context) { func (db *DialBackoff) Backoff(p peer.ID, addr ma.Multiaddr) (backoff bool) { db.lock.Lock() defer db.lock.Unlock() - bp, found := db.entries[p] - if found { - ap, found := bp[string(addr.Bytes())] - if found && time.Now().Before(ap.until) { - return true - } - } - return false + ap, found := db.entries[p][string(addr.Bytes())] + return found && time.Now().Before(ap.until) } // BackoffBase is the base amount of time to backoff (default: 5s). @@ -167,12 +161,8 @@ func (db *DialBackoff) AddBackoff(p peer.ID, addr ma.Multiaddr) { defer db.lock.Unlock() bp, ok := db.entries[p] if !ok { - db.entries[p] = make(map[string]*backoffAddr) - db.entries[p][saddr] = &backoffAddr{ - tries: 1, - until: time.Now().Add(BackoffBase), - } - return + bp = make(map[string]*backoffAddr, 1) + db.entries[p] = bp } ba, ok := bp[saddr] if !ok { From 240d2fe66fc18a25673e10cefebda89253bf5721 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 2 Apr 2020 11:53:58 -0700 Subject: [PATCH 1787/3965] don't expire backoffs until 2x backoff period --- p2p/net/swarm/swarm_dial.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/swarm_dial.go b/p2p/net/swarm/swarm_dial.go index 970cf0105e..cae4110c01 100644 --- a/p2p/net/swarm/swarm_dial.go +++ b/p2p/net/swarm/swarm_dial.go @@ -196,7 +196,11 @@ func (db *DialBackoff) cleanup() { for p, e := range db.entries { good := false for _, backoff := range e { - if now.Before(backoff.until) { + backoffTime := BackoffBase + BackoffCoef*time.Duration(backoff.tries*backoff.tries) + if backoffTime > BackoffMax { + backoffTime = BackoffMax + } + if now.Before(backoff.until.Add(backoffTime)) { good = true break } From 0c848886fc608fa782ea07bbc2a95116d3c06761 Mon Sep 17 00:00:00 2001 From: Will Date: Thu, 2 Apr 2020 13:20:01 -0700 Subject: [PATCH 1788/3965] Use Netroute (#25) Use OS routing table as a hint when dialing --- p2p/net/reuseport/multidialer.go | 126 ++++++++++++---------------- p2p/net/reuseport/transport_test.go | 13 +-- 2 files changed, 61 insertions(+), 78 deletions(-) diff --git a/p2p/net/reuseport/multidialer.go b/p2p/net/reuseport/multidialer.go index c7d388c144..edb200f8df 100644 --- a/p2p/net/reuseport/multidialer.go +++ b/p2p/net/reuseport/multidialer.go @@ -5,12 +5,15 @@ import ( "fmt" "math/rand" "net" + + "github.com/libp2p/go-netroute" ) type multiDialer struct { - loopback []*net.TCPAddr - unspecified []*net.TCPAddr - global *net.TCPAddr + listeningAddresses []*net.TCPAddr + loopback []*net.TCPAddr + unspecified []*net.TCPAddr + fallback net.TCPAddr } func (d *multiDialer) Dial(network, addr string) (net.Conn, error) { @@ -24,87 +27,64 @@ func randAddr(addrs []*net.TCPAddr) *net.TCPAddr { return nil } +// DialContext dials a target addr. +// Dialing preference is +// * If there is a listener on the local interface the OS expects to use to route towards addr, use that. +// * If there is a listener on a loopback address, addr is loopback, use that. +// * If there is a listener on an undefined address (0.0.0.0 or ::), use that. +// * Use the fallback IP specified during construction, with a port that's already being listened on, if one exists. func (d *multiDialer) DialContext(ctx context.Context, network, addr string) (net.Conn, error) { tcpAddr, err := net.ResolveTCPAddr(network, addr) if err != nil { return nil, err } - - // We pick the source *port* based on the following algorithm. - // - // 1. If we're dialing loopback, choose a source-port in order of - // preference: - // 1. A port in-use by an explicit loopback listener. - // 2. A port in-use by a listener on an unspecified address (must - // also be listening on localhost). - // 3. A port in-use by a listener on a global address. We don't have - // any other better options (other than picking a random port). - // 2. If we're dialing a global address, choose a source-port in order - // of preference: - // 1. A port in-use by a listener on an unspecified address (the most - // general case). - // 2. A port in-use by a listener on the global address. - // 3. Fail on link-local dials (go-ipfs currently forbids this and I - // figured we could try lifting this restriction later). - // - // - // Note: We *always* dial from the unspecified address (regardless of - // the port we pick). In the future, we could use netlink (on Linux) to - // figure out the right source address but we're going to punt on that. - ip := tcpAddr.IP - source := d.global - switch { - case ip.IsLoopback(): - switch { - case len(d.loopback) > 0: - source = randAddr(d.loopback) - case len(d.unspecified) > 0: - source = randAddr(d.unspecified) - } - case ip.IsGlobalUnicast(): - switch { - case len(d.unspecified) > 0: - source = randAddr(d.unspecified) + if !ip.IsLoopback() && !ip.IsGlobalUnicast() { + return nil, fmt.Errorf("undialable IP: %s", ip) + } + + if router, err := netroute.New(); err == nil { + if _, _, preferredSrc, err := router.Route(ip); err == nil { + for _, optAddr := range d.listeningAddresses { + if optAddr.IP.Equal(preferredSrc) { + return reuseDial(ctx, optAddr, network, addr) + } + } } - default: - return nil, fmt.Errorf("undialable IP: %s", tcpAddr.IP) } - return reuseDial(ctx, source, network, addr) + + if ip.IsLoopback() && len(d.loopback) > 0 { + return reuseDial(ctx, randAddr(d.loopback), network, addr) + } + if len(d.unspecified) == 0 { + return reuseDial(ctx, &d.fallback, network, addr) + } + + return reuseDial(ctx, randAddr(d.unspecified), network, addr) } -func newMultiDialer(unspec net.IP, listeners map[*listener]struct{}) dialer { - m := new(multiDialer) +func newMultiDialer(unspec net.IP, listeners map[*listener]struct{}) (m dialer) { + addrs := make([]*net.TCPAddr, 0) + loopback := make([]*net.TCPAddr, 0) + unspecified := make([]*net.TCPAddr, 0) + existingPort := 0 + for l := range listeners { - laddr := l.Addr().(*net.TCPAddr) - switch { - case laddr.IP.IsLoopback(): - m.loopback = append(m.loopback, laddr) - case laddr.IP.IsGlobalUnicast(): - // Different global ports? Crap. - // - // The *proper* way to deal with this is to, e.g., use - // netlink to figure out which source address we would - // normally use to dial a destination address and then - // pick one of the ports we're listening on on that - // source address. However, this is a pain in the ass. - // - // Instead, we're just going to always dial from the - // unspecified address with the first global port we - // find. - // - // TODO: Port priority? Addr priority? - if m.global == nil { - m.global = &net.TCPAddr{ - IP: unspec, - Port: laddr.Port, - } - } else { - log.Warning("listening on external interfaces on multiple ports, will dial from %d, not %s", m.global, laddr) - } - case laddr.IP.IsUnspecified(): - m.unspecified = append(m.unspecified, laddr) + addr := l.Addr().(*net.TCPAddr) + addrs = append(addrs, addr) + if addr.IP.IsLoopback() { + loopback = append(loopback, addr) + } else if addr.IP.IsGlobalUnicast() && existingPort == 0 { + existingPort = addr.Port + } else if addr.IP.IsUnspecified() { + unspecified = append(unspecified, addr) } } - return m + m = &multiDialer{ + listeningAddresses: addrs, + loopback: loopback, + unspecified: unspecified, + fallback: net.TCPAddr{IP: unspec, Port: existingPort}, + } + return } diff --git a/p2p/net/reuseport/transport_test.go b/p2p/net/reuseport/transport_test.go index 942019b0b4..876f71d53f 100644 --- a/p2p/net/reuseport/transport_test.go +++ b/p2p/net/reuseport/transport_test.go @@ -38,11 +38,12 @@ func init() { } } -func acceptOne(t *testing.T, listener manet.Listener) <-chan struct{} { +func acceptOne(t *testing.T, listener manet.Listener) <-chan interface{} { t.Helper() - done := make(chan struct{}) + done := make(chan interface{}, 1) go func() { defer close(done) + done <- nil c, err := listener.Accept() if err != nil { t.Error(err) @@ -50,6 +51,7 @@ func acceptOne(t *testing.T, listener manet.Listener) <-chan struct{} { } c.Close() }() + <-done return done } @@ -72,7 +74,7 @@ func dialOne(t *testing.T, tr *Transport, listener manet.Listener, expected ...i return port } } - t.Errorf("dialed from %d, expected to dial from one of %v", port, expected) + t.Errorf("dialed %s from %v. expected to dial from port %v", listener.Multiaddr(), c.LocalAddr(), expected) return 0 } @@ -127,10 +129,12 @@ func TestGlobalPreferenceV4(t *testing.T) { t.Skip("no global IPv4 addresses configured") return } + t.Logf("when listening on %v, should prefer %v over %v", loopbackV4, loopbackV4, globalV4) testPrefer(t, loopbackV4, loopbackV4, globalV4) + t.Logf("when listening on %v, should prefer %v over %v", loopbackV4, unspecV4, globalV4) testPrefer(t, loopbackV4, unspecV4, globalV4) - testPrefer(t, globalV4, unspecV4, globalV4) + t.Logf("when listening on %v, should prefer %v over %v", globalV4, unspecV4, loopbackV4) testPrefer(t, globalV4, unspecV4, loopbackV4) } @@ -142,7 +146,6 @@ func TestGlobalPreferenceV6(t *testing.T) { testPrefer(t, loopbackV6, loopbackV6, globalV6) testPrefer(t, loopbackV6, unspecV6, globalV6) - testPrefer(t, globalV6, unspecV6, globalV6) testPrefer(t, globalV6, unspecV6, loopbackV6) } From ec2907ed95b4bf47045a671000d059bd7e8c0cb1 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Thu, 2 Apr 2020 16:42:40 -0700 Subject: [PATCH 1789/3965] disable write coalescing --- go.mod | 4 ++-- go.sum | 12 ++++++++---- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index 2b318b7db1..43fc931cda 100644 --- a/go.mod +++ b/go.mod @@ -16,7 +16,7 @@ require ( github.com/libp2p/go-libp2p-core v0.5.0 github.com/libp2p/go-libp2p-discovery v0.2.0 github.com/libp2p/go-libp2p-loggables v0.1.0 - github.com/libp2p/go-libp2p-mplex v0.2.2 + github.com/libp2p/go-libp2p-mplex v0.2.3 github.com/libp2p/go-libp2p-nat v0.0.5 github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.2.2 @@ -24,7 +24,7 @@ require ( github.com/libp2p/go-libp2p-swarm v0.2.2 github.com/libp2p/go-libp2p-testing v0.1.1 github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 - github.com/libp2p/go-libp2p-yamux v0.2.6 + github.com/libp2p/go-libp2p-yamux v0.2.7 github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 diff --git a/go.sum b/go.sum index 149f4e2005..9b06ec580c 100644 --- a/go.sum +++ b/go.sum @@ -175,6 +175,8 @@ github.com/libp2p/go-libp2p-mplex v0.2.0/go.mod h1:Ejl9IyjvXJ0T9iqUTE1jpYATQ9NM3 github.com/libp2p/go-libp2p-mplex v0.2.1/go.mod h1:SC99Rxs8Vuzrf/6WhmH41kNn13TiYdAWNYHrwImKLnE= github.com/libp2p/go-libp2p-mplex v0.2.2 h1:+Ld7YDAfVERQ0E+qqjE7o6fHwKuM0SqTzYiwN1lVVSA= github.com/libp2p/go-libp2p-mplex v0.2.2/go.mod h1:74S9eum0tVQdAfFiKxAyKzNdSuLqw5oadDq7+L/FELo= +github.com/libp2p/go-libp2p-mplex v0.2.3 h1:2zijwaJvpdesST2MXpI5w9wWFRgYtMcpRX7rrw0jmOo= +github.com/libp2p/go-libp2p-mplex v0.2.3/go.mod h1:CK3p2+9qH9x+7ER/gWWDYJ3QW5ZxWDkm+dVvjfuG3ek= github.com/libp2p/go-libp2p-nat v0.0.5 h1:/mH8pXFVKleflDL1YwqMg27W9GD8kjEx7NY0P6eGc98= github.com/libp2p/go-libp2p-nat v0.0.5/go.mod h1:1qubaE5bTZMJE+E/uu2URroMbzdubFz1ChgiN79yKPE= github.com/libp2p/go-libp2p-netutil v0.1.0 h1:zscYDNVEcGxyUpMd0JReUZTrpMfia8PmLKcKF72EAMQ= @@ -209,8 +211,8 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSo github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= github.com/libp2p/go-libp2p-yamux v0.2.5 h1:MuyItOqz03oi8npvjgMJxgnhllJLZnO/dKVOpTZ9+XI= github.com/libp2p/go-libp2p-yamux v0.2.5/go.mod h1:Zpgj6arbyQrmZ3wxSZxfBmbdnWtbZ48OpsfmQVTErwA= -github.com/libp2p/go-libp2p-yamux v0.2.6 h1:BXl4fEzzZwLUjoqoRJNx4co1UhvOwXaM+Qg2twEqEFU= -github.com/libp2p/go-libp2p-yamux v0.2.6/go.mod h1:YbnSVt8RYb3t6YBGPD9rLUT0vmmvcr/waaQ79PINpOM= +github.com/libp2p/go-libp2p-yamux v0.2.7 h1:vzKu0NVtxvEIDGCv6mjKRcK0gipSgaXmJZ6jFv0d/dk= +github.com/libp2p/go-libp2p-yamux v0.2.7/go.mod h1:X28ENrBMU/nm4I3Nx4sZ4dgjZ6VhLEn0XhIoZ5viCwU= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.5 h1:CW3AgbMO6vUvT4kf87y4N+0P8KUl2aqLYhrGyDUbLSg= github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= @@ -218,6 +220,8 @@ github.com/libp2p/go-mplex v0.0.3/go.mod h1:pK5yMLmOoBR1pNCqDlA2GQrdAVTMkqFalaTW github.com/libp2p/go-mplex v0.1.0/go.mod h1:SXgmdki2kwCUlCCbfGLEgHjC4pFqhTp0ZoV6aiKgxDU= github.com/libp2p/go-mplex v0.1.1 h1:huPH/GGRJzmsHR9IZJJsrSwIM5YE2gL4ssgl1YWb/ps= github.com/libp2p/go-mplex v0.1.1/go.mod h1:Xgz2RDCi3co0LeZfgjm4OgUF15+sVR8SRcu3SFXI1lk= +github.com/libp2p/go-mplex v0.1.2 h1:qOg1s+WdGLlpkrczDqmhYzyk3vCfsQ8+RxRTQjOZWwI= +github.com/libp2p/go-mplex v0.1.2/go.mod h1:Xgz2RDCi3co0LeZfgjm4OgUF15+sVR8SRcu3SFXI1lk= github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-msgio v0.0.4 h1:agEFehY3zWJFUHK6SEMR7UYmk2z6kC3oeCM7ybLhguA= github.com/libp2p/go-msgio v0.0.4/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= @@ -242,8 +246,8 @@ github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzl github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.3 h1:mWuzZRCAeTBFdynLlsYgA/EIeMOLr8XY04wa52NRhsE= github.com/libp2p/go-yamux v1.3.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= -github.com/libp2p/go-yamux v1.3.4 h1:CptmtNfQomI9RpM7LFJhVBFkwp2zwEXIskEfWwwtoHg= -github.com/libp2p/go-yamux v1.3.4/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-yamux v1.3.5 h1:ibuz4naPAully0pN6J/kmUARiqLpnDQIzI/8GCOrljg= +github.com/libp2p/go-yamux v1.3.5/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= From 79f13c024f7827d2d6f0a1faf1e92948f65088a5 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Apr 2020 17:04:28 -0700 Subject: [PATCH 1790/3965] chore: update the rest of the deps --- go.mod | 5 ++--- go.sum | 6 ++++++ 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 43fc931cda..8d2e95ab95 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.2.2 github.com/libp2p/go-libp2p-secio v0.2.1 - github.com/libp2p/go-libp2p-swarm v0.2.2 + github.com/libp2p/go-libp2p-swarm v0.2.3 github.com/libp2p/go-libp2p-testing v0.1.1 github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 github.com/libp2p/go-libp2p-yamux v0.2.7 @@ -29,10 +29,9 @@ require ( github.com/libp2p/go-stream-muxer-multistream v0.2.0 github.com/libp2p/go-tcp-transport v0.1.1 github.com/libp2p/go-ws-transport v0.2.0 - github.com/miekg/dns v1.1.28 // indirect github.com/multiformats/go-multiaddr v0.2.1 github.com/multiformats/go-multiaddr-dns v0.2.0 - github.com/multiformats/go-multiaddr-net v0.1.3 + github.com/multiformats/go-multiaddr-net v0.1.4 github.com/multiformats/go-multistream v0.1.1 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 ) diff --git a/go.sum b/go.sum index 9b06ec580c..f23c9ad963 100644 --- a/go.sum +++ b/go.sum @@ -199,6 +199,8 @@ github.com/libp2p/go-libp2p-secio v0.2.1/go.mod h1:cWtZpILJqkqrSkiYcDBh5lA3wbT2Q github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= github.com/libp2p/go-libp2p-swarm v0.2.2 h1:T4hUpgEs2r371PweU3DuH7EOmBIdTBCwWs+FLcgx3bQ= github.com/libp2p/go-libp2p-swarm v0.2.2/go.mod h1:fvmtQ0T1nErXym1/aa1uJEyN7JzaTNyBcHImCxRpPKU= +github.com/libp2p/go-libp2p-swarm v0.2.3 h1:uVkCb8Blfg7HQ/f30TyHn1g/uCwXsAET7pU0U59gx/A= +github.com/libp2p/go-libp2p-swarm v0.2.3/go.mod h1:P2VO/EpxRyDxtChXz/VPVXyTnszHvokHKRhfkEgFKNM= github.com/libp2p/go-libp2p-testing v0.0.2/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.3/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= github.com/libp2p/go-libp2p-testing v0.0.4/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= @@ -209,6 +211,7 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07q github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 h1:5EhPgQhXZNyfL22ERZTUoVp9UVVbNowWNVtELQaKCHk= github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSoMuyojOdjx73Q47cYD7s5+gZOlns= github.com/libp2p/go-libp2p-yamux v0.2.0/go.mod h1:Db2gU+XfLpm6E4rG5uGCFX6uXA8MEXOxFcRoXUODaK8= +github.com/libp2p/go-libp2p-yamux v0.2.2/go.mod h1:lIohaR0pT6mOt0AZ0L2dFze9hds9Req3OfS+B+dv4qw= github.com/libp2p/go-libp2p-yamux v0.2.5 h1:MuyItOqz03oi8npvjgMJxgnhllJLZnO/dKVOpTZ9+XI= github.com/libp2p/go-libp2p-yamux v0.2.5/go.mod h1:Zpgj6arbyQrmZ3wxSZxfBmbdnWtbZ48OpsfmQVTErwA= github.com/libp2p/go-libp2p-yamux v0.2.7 h1:vzKu0NVtxvEIDGCv6mjKRcK0gipSgaXmJZ6jFv0d/dk= @@ -244,6 +247,7 @@ github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfj github.com/libp2p/go-ws-transport v0.2.0 h1:MJCw2OrPA9+76YNRvdo1wMnSOxb9Bivj6sVFY1Xrj6w= github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzlHoKzu0yY9p/klM= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= +github.com/libp2p/go-yamux v1.3.0/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.3 h1:mWuzZRCAeTBFdynLlsYgA/EIeMOLr8XY04wa52NRhsE= github.com/libp2p/go-yamux v1.3.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.5 h1:ibuz4naPAully0pN6J/kmUARiqLpnDQIzI/8GCOrljg= @@ -297,6 +301,8 @@ github.com/multiformats/go-multiaddr-net v0.1.1/go.mod h1:5JNbcfBOP4dnhoZOv10JJV github.com/multiformats/go-multiaddr-net v0.1.2/go.mod h1:QsWt3XK/3hwvNxZJp92iMQKME1qHfpYmyIjFVsSOY6Y= github.com/multiformats/go-multiaddr-net v0.1.3 h1:q/IYAvoPKuRzGeERn3uacWgm0LIWkLZBAvO5DxSzq3g= github.com/multiformats/go-multiaddr-net v0.1.3/go.mod h1:ilNnaM9HbmVFqsb/qcNysjCu4PVONlrBZpHIrw/qQuA= +github.com/multiformats/go-multiaddr-net v0.1.4 h1:g6gwydsfADqFvrHoMkS0n9Ok9CG6F7ytOH/bJDkhIOY= +github.com/multiformats/go-multiaddr-net v0.1.4/go.mod h1:ilNnaM9HbmVFqsb/qcNysjCu4PVONlrBZpHIrw/qQuA= github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA= github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= From 29a922ff928ea1d19490a3d8739006b754bf5aab Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Apr 2020 20:44:44 -0700 Subject: [PATCH 1791/3965] fix: don't drop bytes in the insecure transport gogo's varint reader buffers internally. If you use it then throw it away, you'll drop data. This commit reverts to using msgio, but uses the varint reader/writers instead. --- core/sec/insecure/insecure.go | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/core/sec/insecure/insecure.go b/core/sec/insecure/insecure.go index b9a939407f..1f40924937 100644 --- a/core/sec/insecure/insecure.go +++ b/core/sec/insecure/insecure.go @@ -9,10 +9,9 @@ import ( "io" "net" - gogoio "github.com/gogo/protobuf/io" - "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/sec" + "github.com/libp2p/go-msgio" ci "github.com/libp2p/go-libp2p-core/crypto" pb "github.com/libp2p/go-libp2p-core/sec/insecure/pb" @@ -183,27 +182,34 @@ func (ic *Conn) runHandshakeSync() error { // read and write a message at the same time. func readWriteMsg(rw io.ReadWriter, out *pb.Exchange) (*pb.Exchange, error) { - const maxMsgSize = 1 << 16 - r := gogoio.NewDelimitedReader(rw, maxMsgSize) - w := gogoio.NewDelimitedWriter(rw) + const maxMessageSize = 1 << 16 + + outBytes, err := out.Marshal() + if err != nil { + return nil, err + } wresult := make(chan error) go func() { - wresult <- w.WriteMsg(out) + w := msgio.NewVarintWriter(rw) + wresult <- w.WriteMsg(outBytes) }() - inMsg := pb.Exchange{} - err := r.ReadMsg(&inMsg) + r := msgio.NewVarintReaderSize(rw, maxMessageSize) + msg, err1 := r.ReadMsg() - // Always wait for the write to finish. + // Always wait for the read to finish. err2 := <-wresult - if err != nil { - return nil, err + if err1 != nil { + return nil, err1 } if err2 != nil { + r.ReleaseMsg(msg) return nil, err2 } - return &inMsg, err + inMsg := new(pb.Exchange) + err = inMsg.Unmarshal(msg) + return inMsg, err } // LocalPeer returns the local peer ID. From d0bf192109b7ec35b2207a783e01ba78ba85c151 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Apr 2020 22:12:24 -0700 Subject: [PATCH 1792/3965] fix: restrict dials to IP + TCP That is, forbid DNS. See https://github.com/libp2p/go-libp2p/issues/841 --- p2p/transport/websocket/websocket.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/p2p/transport/websocket/websocket.go b/p2p/transport/websocket/websocket.go index 7e1c900f27..ddd4141090 100644 --- a/p2p/transport/websocket/websocket.go +++ b/p2p/transport/websocket/websocket.go @@ -18,7 +18,7 @@ import ( var WsProtocol = ma.ProtocolWithCode(ma.P_WS) // WsFmt is multiaddr formatter for WsProtocol -var WsFmt = mafmt.And(mafmt.TCP, mafmt.Base(WsProtocol.Code)) +var WsFmt = mafmt.And(mafmt.TCP, mafmt.Base(ma.P_WS)) // WsCodec is the multiaddr-net codec definition for the websocket transport var WsCodec = &manet.NetCodec{ @@ -28,6 +28,10 @@ var WsCodec = &manet.NetCodec{ ParseNetAddr: ParseWebsocketNetAddr, } +// This is _not_ WsFmt because we want the transport to stick to dialing fully +// resolved addresses. +var dialMatcher = mafmt.And(mafmt.IP, mafmt.Base(ma.P_TCP), mafmt.Base(ma.P_WS)) + func init() { manet.RegisterNetCodec(WsCodec) } @@ -44,7 +48,7 @@ func New(u *tptu.Upgrader) *WebsocketTransport { } func (t *WebsocketTransport) CanDial(a ma.Multiaddr) bool { - return WsFmt.Matches(a) + return dialMatcher.Matches(a) } func (t *WebsocketTransport) Protocols() []int { From 503a78e2800a6a010c40371b1fdffe47e0575a07 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Apr 2020 22:20:21 -0700 Subject: [PATCH 1793/3965] fix: don't allow dialing DNS addresses See https://github.com/libp2p/go-libp2p/issues/841 --- p2p/transport/tcp/tcp.go | 4 +++- p2p/transport/tcp/tcp_test.go | 21 +++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/p2p/transport/tcp/tcp.go b/p2p/transport/tcp/tcp.go index 1483d67e69..6c204109f3 100644 --- a/p2p/transport/tcp/tcp.go +++ b/p2p/transport/tcp/tcp.go @@ -70,10 +70,12 @@ func NewTCPTransport(upgrader *tptu.Upgrader) *TcpTransport { return &TcpTransport{Upgrader: upgrader, ConnectTimeout: DefaultConnectTimeout} } +var dialMatcher = mafmt.And(mafmt.IP, mafmt.Base(ma.P_TCP)) + // CanDial returns true if this transport believes it can dial the given // multiaddr. func (t *TcpTransport) CanDial(addr ma.Multiaddr) bool { - return mafmt.TCP.Matches(addr) + return dialMatcher.Matches(addr) } func (t *TcpTransport) maDial(ctx context.Context, raddr ma.Multiaddr) (manet.Conn, error) { diff --git a/p2p/transport/tcp/tcp_test.go b/p2p/transport/tcp/tcp_test.go index a4a85bdf04..8dcad45248 100644 --- a/p2p/transport/tcp/tcp_test.go +++ b/p2p/transport/tcp/tcp_test.go @@ -36,6 +36,27 @@ func TestTcpTransport(t *testing.T) { envReuseportVal = true } +func TestTcpTransportCantDialDNS(t *testing.T) { + for i := 0; i < 2; i++ { + dnsa, err := ma.NewMultiaddr("/dns4/example.com/tcp/1234") + if err != nil { + t.Fatal(err) + } + + tpt := NewTCPTransport(&tptu.Upgrader{ + Secure: makeInsecureTransport(t), + Muxer: new(mplex.Transport), + }) + + if tpt.CanDial(dnsa) { + t.Fatal("shouldn't be able to dial dns") + } + + envReuseportVal = false + } + envReuseportVal = true +} + func TestTcpTransportCantListenUtp(t *testing.T) { for i := 0; i < 2; i++ { utpa, err := ma.NewMultiaddr("/ip4/127.0.0.1/udp/0/utp") From e8ccc0b534cea5acbaa35680eaa603c5776a2335 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Apr 2020 22:36:47 -0700 Subject: [PATCH 1794/3965] fix: avoid dialing/listening on dns addresses See https://github.com/libp2p/go-libp2p/issues/841 --- p2p/transport/quic/transport.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index fa54bd8c54..fcb6c82ff1 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -190,9 +190,12 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp }, nil } +// Don't use mafmt.QUIC as we don't want to dial DNS addresses. Just /ip{4,6}/udp/quic +var dialMatcher = mafmt.And(mafmt.IP, mafmt.Base(ma.P_UDP), mafmt.Base(ma.P_QUIC)) + // CanDial determines if we can dial to an address func (t *transport) CanDial(addr ma.Multiaddr) bool { - return mafmt.QUIC.Matches(addr) + return dialMatcher.Matches(addr) } // Listen listens for new QUIC connections on the passed multiaddr. From b980ae80f0d129f026035fef6749791cb7a81aeb Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Apr 2020 23:06:40 -0700 Subject: [PATCH 1795/3965] Revert "feat: functional options" This reverts commit bf52b405f378fe279d48e01bd4374d2dcc539382. Unfortunately, we're relying on the fact that we can inspect and compare these options in go-libp2p. --- .../internal/circuitv1-deprecated/relay.go | 34 +++++++++---------- 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/p2p/protocol/internal/circuitv1-deprecated/relay.go b/p2p/protocol/internal/circuitv1-deprecated/relay.go index 320533e259..10816233bb 100644 --- a/p2p/protocol/internal/circuitv1-deprecated/relay.go +++ b/p2p/protocol/internal/circuitv1-deprecated/relay.go @@ -55,24 +55,18 @@ type Relay struct { } // RelayOpts are options for configuring the relay transport. -type RelayOpt func(*Relay) error +type RelayOpt int var ( // OptActive configures the relay transport to actively establish // outbound connections on behalf of clients. You probably don't want to // enable this unless you know what you're doing. - OptActive RelayOpt = func(r *Relay) error { - r.active = true - return nil - } + OptActive = RelayOpt(0) // OptHop configures the relay transport to accept requests to relay // traffic on behalf of third-parties. Unless OptActive is specified, // this will only relay traffic between peers already connected to this // node. - OptHop = func(r *Relay) error { - r.hop = true - return nil - } + OptHop = RelayOpt(1) // OptDiscovery is a no-op. It was introduced as a way to probe new // peers to see if they were willing to act as a relays. However, in // practice, it's useless. While it does test to see if these peers are @@ -81,13 +75,7 @@ var ( // // This option may be re-enabled in the future but for now you shouldn't // use it. - OptDiscovery = func(r *Relay) error { - log.Errorf( - "circuit.OptDiscovery is now a no-op: %s", - "dialing peers with a random relay is no longer supported", - ) - return nil - } + OptDiscovery = RelayOpt(2) ) type RelayError struct { @@ -109,8 +97,18 @@ func NewRelay(ctx context.Context, h host.Host, upgrader *tptu.Upgrader, opts .. } for _, opt := range opts { - if err := opt(r); err != nil { - return nil, err + switch opt { + case OptActive: + r.active = true + case OptHop: + r.hop = true + case OptDiscovery: + log.Errorf( + "circuit.OptDiscovery is now a no-op: %s", + "dialing peers with a random relay is no longer supported", + ) + default: + return nil, fmt.Errorf("unrecognized option: %d", opt) } } From d14fdba1efc50340783b98f514e6e4af71fcbe79 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 2 Apr 2020 23:02:10 -0700 Subject: [PATCH 1796/3965] chore: update deps again One final time. --- go.mod | 14 +++++++------- go.sum | 30 ++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 7 deletions(-) diff --git a/go.mod b/go.mod index 8d2e95ab95..422aa17511 100644 --- a/go.mod +++ b/go.mod @@ -12,23 +12,23 @@ require ( github.com/libp2p/go-eventbus v0.1.0 github.com/libp2p/go-libp2p-autonat v0.2.1 github.com/libp2p/go-libp2p-blankhost v0.1.4 - github.com/libp2p/go-libp2p-circuit v0.1.4 - github.com/libp2p/go-libp2p-core v0.5.0 - github.com/libp2p/go-libp2p-discovery v0.2.0 + github.com/libp2p/go-libp2p-circuit v0.2.1 + github.com/libp2p/go-libp2p-core v0.5.1 + github.com/libp2p/go-libp2p-discovery v0.3.0 github.com/libp2p/go-libp2p-loggables v0.1.0 github.com/libp2p/go-libp2p-mplex v0.2.3 - github.com/libp2p/go-libp2p-nat v0.0.5 + github.com/libp2p/go-libp2p-nat v0.0.6 github.com/libp2p/go-libp2p-netutil v0.1.0 github.com/libp2p/go-libp2p-peerstore v0.2.2 - github.com/libp2p/go-libp2p-secio v0.2.1 + github.com/libp2p/go-libp2p-secio v0.2.2 github.com/libp2p/go-libp2p-swarm v0.2.3 github.com/libp2p/go-libp2p-testing v0.1.1 github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 github.com/libp2p/go-libp2p-yamux v0.2.7 github.com/libp2p/go-maddr-filter v0.0.5 github.com/libp2p/go-stream-muxer-multistream v0.2.0 - github.com/libp2p/go-tcp-transport v0.1.1 - github.com/libp2p/go-ws-transport v0.2.0 + github.com/libp2p/go-tcp-transport v0.2.0 + github.com/libp2p/go-ws-transport v0.3.0 github.com/multiformats/go-multiaddr v0.2.1 github.com/multiformats/go-multiaddr-dns v0.2.0 github.com/multiformats/go-multiaddr-net v0.1.4 diff --git a/go.sum b/go.sum index f23c9ad963..88747dc217 100644 --- a/go.sum +++ b/go.sum @@ -56,10 +56,14 @@ github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8l github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/gopacket v1.1.17 h1:rMrlX2ZY2UbvT+sdz3+6J+pp2z+msCq9MxTU6ymxbBY= +github.com/google/gopacket v1.1.17/go.mod h1:UdDNZ1OO62aGYVnPhxT1U6aI7ukYtA/kB8vaU0diBUM= github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/websocket v1.4.1 h1:q7AeDBpnBk8AogcD4DSag/Ukw/KV+YhzLj2bP5HvKCM= github.com/gorilla/websocket v1.4.1/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc= +github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= @@ -106,6 +110,8 @@ github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= +github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= +github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/jbenet/go-cienv v0.0.0-20150120210510-1bb1476777ec/go.mod h1:rGaEvXB4uRSZMmzKNLoXvTu1sfx+1kv/DojUlPrSZGs= github.com/jbenet/go-cienv v0.1.0 h1:Vc/s0QbQtoxX8MwwSLWWh+xNNZvM3Lw7NsTcHrvvhMc= github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA= @@ -156,6 +162,8 @@ github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUA github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= github.com/libp2p/go-libp2p-circuit v0.1.4 h1:Phzbmrg3BkVzbqd4ZZ149JxCuUWu2wZcXf/Kr6hZJj8= github.com/libp2p/go-libp2p-circuit v0.1.4/go.mod h1:CY67BrEjKNDhdTk8UgBX1Y/H5c3xkAcs3gnksxY7osU= +github.com/libp2p/go-libp2p-circuit v0.2.1 h1:BDiBcQxX/ZJJ/yDl3sqZt1bjj4PkZCEi7IEpwxXr13k= +github.com/libp2p/go-libp2p-circuit v0.2.1/go.mod h1:BXPwYDN5A8z4OEY9sOfr2DUQMLQvKt/6oku45YUmjIo= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.4/go.mod h1:jyuCQP356gzfCFtRKyvAbNkyeuxb7OlyhWZ3nls5d2I= github.com/libp2p/go-libp2p-core v0.2.0/go.mod h1:X0eyB0Gy93v0DZtSYbEM7RnMChm9Uv3j7yRXjO77xSI= @@ -166,9 +174,14 @@ github.com/libp2p/go-libp2p-core v0.3.1/go.mod h1:thvWy0hvaSBhnVBaW37BvzgVV68OUh github.com/libp2p/go-libp2p-core v0.4.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= github.com/libp2p/go-libp2p-core v0.5.0 h1:FBQ1fpq2Fo/ClyjojVJ5AKXlKhvNc/B6U0O+7AN1ffE= github.com/libp2p/go-libp2p-core v0.5.0/go.mod h1:49XGI+kc38oGVwqSBhDEwytaAxgZasHhFfQKibzTls0= +github.com/libp2p/go-libp2p-core v0.5.1 h1:6Cu7WljPQtGY2krBlMoD8L/zH3tMUsCbqNFH7cZwCoI= +github.com/libp2p/go-libp2p-core v0.5.1/go.mod h1:uN7L2D4EvPCvzSH5SrhR72UWbnSGpt5/a35Sm4upn4Y= +github.com/libp2p/go-libp2p-crypto v0.1.0 h1:k9MFy+o2zGDNGsaoZl0MA3iZ75qXxr9OOoAZF+sD5OQ= github.com/libp2p/go-libp2p-crypto v0.1.0/go.mod h1:sPUokVISZiy+nNuTTH/TY+leRSxnFj/2GLjtOTW90hI= github.com/libp2p/go-libp2p-discovery v0.2.0 h1:1p3YSOq7VsgaL+xVHPi8XAmtGyas6D2J6rWBEfz/aiY= github.com/libp2p/go-libp2p-discovery v0.2.0/go.mod h1:s4VGaxYMbw4+4+tsoQTqh7wfxg97AEdo4GYBt6BadWg= +github.com/libp2p/go-libp2p-discovery v0.3.0 h1:+JnYBRLzZQtRq0mK3xhyjBwHytLmJXMTZkQfbw+UrGA= +github.com/libp2p/go-libp2p-discovery v0.3.0/go.mod h1:o03drFnz9BVAZdzC/QUQ+NeQOu38Fu7LJGEOK2gQltw= github.com/libp2p/go-libp2p-loggables v0.1.0 h1:h3w8QFfCt2UJl/0/NW4K829HX/0S4KD31PQ7m8UXXO8= github.com/libp2p/go-libp2p-loggables v0.1.0/go.mod h1:EyumB2Y6PrYjr55Q3/tiJ/o3xoDasoRYM7nOzEpoa90= github.com/libp2p/go-libp2p-mplex v0.2.0/go.mod h1:Ejl9IyjvXJ0T9iqUTE1jpYATQ9NM3g+OtR+EMMODbKo= @@ -179,6 +192,8 @@ github.com/libp2p/go-libp2p-mplex v0.2.3 h1:2zijwaJvpdesST2MXpI5w9wWFRgYtMcpRX7r github.com/libp2p/go-libp2p-mplex v0.2.3/go.mod h1:CK3p2+9qH9x+7ER/gWWDYJ3QW5ZxWDkm+dVvjfuG3ek= github.com/libp2p/go-libp2p-nat v0.0.5 h1:/mH8pXFVKleflDL1YwqMg27W9GD8kjEx7NY0P6eGc98= github.com/libp2p/go-libp2p-nat v0.0.5/go.mod h1:1qubaE5bTZMJE+E/uu2URroMbzdubFz1ChgiN79yKPE= +github.com/libp2p/go-libp2p-nat v0.0.6 h1:wMWis3kYynCbHoyKLPBEMu4YRLltbm8Mk08HGSfvTkU= +github.com/libp2p/go-libp2p-nat v0.0.6/go.mod h1:iV59LVhB3IkFvS6S6sauVTSOrNEANnINbI/fkaLimiw= github.com/libp2p/go-libp2p-netutil v0.1.0 h1:zscYDNVEcGxyUpMd0JReUZTrpMfia8PmLKcKF72EAMQ= github.com/libp2p/go-libp2p-netutil v0.1.0/go.mod h1:3Qv/aDqtMLTUyQeundkKsA+YCThNdbQD54k3TqjpbFU= github.com/libp2p/go-libp2p-peer v0.2.0/go.mod h1:RCffaCvUyW2CJmG2gAWVqwePwW7JMgxjsHm7+J5kjWY= @@ -196,6 +211,8 @@ github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBg github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-secio v0.2.1 h1:eNWbJTdyPA7NxhP7J3c5lT97DC5d+u+IldkgCYFTPVA= github.com/libp2p/go-libp2p-secio v0.2.1/go.mod h1:cWtZpILJqkqrSkiYcDBh5lA3wbT2Q+hz3rJQq3iftD8= +github.com/libp2p/go-libp2p-secio v0.2.2 h1:rLLPvShPQAcY6eNurKNZq3eZjPWfU9kXF2eI9jIYdrg= +github.com/libp2p/go-libp2p-secio v0.2.2/go.mod h1:wP3bS+m5AUnFA+OFO7Er03uO1mncHG0uVwGrwvjYlNY= github.com/libp2p/go-libp2p-swarm v0.1.0/go.mod h1:wQVsCdjsuZoc730CgOvh5ox6K8evllckjebkdiY5ta4= github.com/libp2p/go-libp2p-swarm v0.2.2 h1:T4hUpgEs2r371PweU3DuH7EOmBIdTBCwWs+FLcgx3bQ= github.com/libp2p/go-libp2p-swarm v0.2.2/go.mod h1:fvmtQ0T1nErXym1/aa1uJEyN7JzaTNyBcHImCxRpPKU= @@ -230,6 +247,10 @@ github.com/libp2p/go-msgio v0.0.4 h1:agEFehY3zWJFUHK6SEMR7UYmk2z6kC3oeCM7ybLhguA github.com/libp2p/go-msgio v0.0.4/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= github.com/libp2p/go-nat v0.0.4 h1:KbizNnq8YIf7+Hn7+VFL/xE0eDrkPru2zIO9NMwL8UQ= github.com/libp2p/go-nat v0.0.4/go.mod h1:Nmw50VAvKuk38jUBcmNh6p9lUJLoODbJRvYAa/+KSDo= +github.com/libp2p/go-nat v0.0.5 h1:qxnwkco8RLKqVh1NmjQ+tJ8p8khNLFxuElYG/TwqW4Q= +github.com/libp2p/go-nat v0.0.5/go.mod h1:B7NxsVNPZmRLvMOwiEO1scOSyjA56zxYAGv1yQgRkEU= +github.com/libp2p/go-netroute v0.1.2 h1:UHhB35chwgvcRI392znJA3RCBtZ3MpE3ahNCN5MR4Xg= +github.com/libp2p/go-netroute v0.1.2/go.mod h1:jZLDV+1PE8y5XxBySEBgbuVAXbhtuHSdmLPL2n9MKbk= github.com/libp2p/go-openssl v0.0.2/go.mod h1:v8Zw2ijCSWBQi8Pq5GAixw6DbFfa9u6VIYDXnvOXkc0= github.com/libp2p/go-openssl v0.0.3/go.mod h1:unDrJpgy3oFr+rqXsarWifmJuNnJR4chtO1HmaZjggc= github.com/libp2p/go-openssl v0.0.4 h1:d27YZvLoTyMhIN4njrkr8zMDOM4lfpHIp6A+TK9fovg= @@ -238,14 +259,22 @@ github.com/libp2p/go-reuseport v0.0.1 h1:7PhkfH73VXfPJYKQ6JwS5I/eVcoyYi9IMNGc6FW github.com/libp2p/go-reuseport v0.0.1/go.mod h1:jn6RmB1ufnQwl0Q1f+YxAj8isJgDCQzaaxIFYDhcYEA= github.com/libp2p/go-reuseport-transport v0.0.2 h1:WglMwyXyBu61CMkjCCtnmqNqnjib0GIEjMiHTwR/KN4= github.com/libp2p/go-reuseport-transport v0.0.2/go.mod h1:YkbSDrvjUVDL6b8XqriyA20obEtsW9BLkuOUyQAOCbs= +github.com/libp2p/go-reuseport-transport v0.0.3 h1:zzOeXnTooCkRvoH+bSXEfXhn76+LAiwoneM0gnXjF2M= +github.com/libp2p/go-reuseport-transport v0.0.3/go.mod h1:Spv+MPft1exxARzP2Sruj2Wb5JSyHNncjf1Oi2dEbzM= +github.com/libp2p/go-sockaddr v0.0.2 h1:tCuXfpA9rq7llM/v834RKc/Xvovy/AqM9kHvTV/jY/Q= +github.com/libp2p/go-sockaddr v0.0.2/go.mod h1:syPvOmNs24S3dFVGJA1/mrqdeijPxLV2Le3BRLKd68k= github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= github.com/libp2p/go-stream-muxer-multistream v0.2.0 h1:714bRJ4Zy9mdhyTLJ+ZKiROmAFwUHpeRidG+q7LTQOg= github.com/libp2p/go-stream-muxer-multistream v0.2.0/go.mod h1:j9eyPol/LLRqT+GPLSxvimPhNph4sfYfMoDPd7HkzIc= github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= github.com/libp2p/go-tcp-transport v0.1.1 h1:yGlqURmqgNA2fvzjSgZNlHcsd/IulAnKM8Ncu+vlqnw= github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfjg2pWP97dFZworkY= +github.com/libp2p/go-tcp-transport v0.2.0 h1:YoThc549fzmNJIh7XjHVtMIFaEDRtIrtWciG5LyYAPo= +github.com/libp2p/go-tcp-transport v0.2.0/go.mod h1:vX2U0CnWimU4h0SGSEsg++AzvBcroCGYw28kh94oLe0= github.com/libp2p/go-ws-transport v0.2.0 h1:MJCw2OrPA9+76YNRvdo1wMnSOxb9Bivj6sVFY1Xrj6w= github.com/libp2p/go-ws-transport v0.2.0/go.mod h1:9BHJz/4Q5A9ludYWKoGCFC5gUElzlHoKzu0yY9p/klM= +github.com/libp2p/go-ws-transport v0.3.0 h1:mjo6pL5aVR9rCjl9wNq3DupbaQlyR61pzoOT2MdtxaA= +github.com/libp2p/go-ws-transport v0.3.0/go.mod h1:bpgTJmRZAvVHrgHybCVyqoBmyLQ1fiZuEaBYusP5zsk= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.0/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.3.3 h1:mWuzZRCAeTBFdynLlsYgA/EIeMOLr8XY04wa52NRhsE= @@ -430,6 +459,7 @@ golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190405154228-4b34438f7a67/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= From d7e97470198da7c72a252a2dcb97a06260c62cf8 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 16 Mar 2020 18:46:35 +0700 Subject: [PATCH 1797/3965] test that CanDial fails on /dns addresses --- p2p/transport/quic/transport_test.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/p2p/transport/quic/transport_test.go b/p2p/transport/quic/transport_test.go index 8dfece43ba..111b702c8c 100644 --- a/p2p/transport/quic/transport_test.go +++ b/p2p/transport/quic/transport_test.go @@ -24,6 +24,12 @@ var _ = Describe("Transport", func() { Expect(t.CanDial(validAddr)).To(BeTrue()) }) + It("says that it cannot dial /dns addresses", func() { + addr, err := ma.NewMultiaddr("/dns/google.com/udp/443/quic") + Expect(err).ToNot(HaveOccurred()) + Expect(t.CanDial(addr)).To(BeFalse()) + }) + It("supports the QUIC protocol", func() { protocols := t.Protocols() Expect(protocols).To(HaveLen(1)) From bdbf17697e4484ebea7ab2466e2b1946b68e7e6e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 26 Mar 2020 21:58:20 +0700 Subject: [PATCH 1798/3965] export qlogs when the QLOGDIR env variable is set --- p2p/transport/quic/buffered_write_closer.go | 25 ++++++++ .../quic/buffered_write_closer_test.go | 26 ++++++++ p2p/transport/quic/conn_test.go | 4 +- p2p/transport/quic/listener.go | 2 +- p2p/transport/quic/transport.go | 62 +++++++++++++++---- 5 files changed, 103 insertions(+), 16 deletions(-) create mode 100644 p2p/transport/quic/buffered_write_closer.go create mode 100644 p2p/transport/quic/buffered_write_closer_test.go diff --git a/p2p/transport/quic/buffered_write_closer.go b/p2p/transport/quic/buffered_write_closer.go new file mode 100644 index 0000000000..aeeef0035a --- /dev/null +++ b/p2p/transport/quic/buffered_write_closer.go @@ -0,0 +1,25 @@ +package libp2pquic + +import ( + "bufio" + "io" +) + +type bufferedWriteCloser struct { + *bufio.Writer + io.Closer +} + +func newBufferedWriteCloser(writer *bufio.Writer, closer io.Closer) io.WriteCloser { + return &bufferedWriteCloser{ + Writer: writer, + Closer: closer, + } +} + +func (h bufferedWriteCloser) Close() error { + if err := h.Writer.Flush(); err != nil { + return err + } + return h.Closer.Close() +} diff --git a/p2p/transport/quic/buffered_write_closer_test.go b/p2p/transport/quic/buffered_write_closer_test.go new file mode 100644 index 0000000000..949e211c22 --- /dev/null +++ b/p2p/transport/quic/buffered_write_closer_test.go @@ -0,0 +1,26 @@ +package libp2pquic + +import ( + "bufio" + "bytes" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +type nopCloser struct{} + +func (nopCloser) Close() error { return nil } + +var _ = Describe("buffered io.WriteCloser", func() { + It("flushes before closing", func() { + buf := &bytes.Buffer{} + + w := bufio.NewWriter(buf) + wc := newBufferedWriteCloser(w, &nopCloser{}) + wc.Write([]byte("foobar")) + Expect(buf.Len()).To(BeZero()) + Expect(wc.Close()).To(Succeed()) + Expect(buf.String()).To(Equal("foobar")) + }) +}) diff --git a/p2p/transport/quic/conn_test.go b/p2p/transport/quic/conn_test.go index d70af9f6e4..61320343ca 100644 --- a/p2p/transport/quic/conn_test.go +++ b/p2p/transport/quic/conn_test.go @@ -185,13 +185,13 @@ var _ = Describe("Connection", func() { Expect(err).ToNot(HaveOccurred()) // make sure that connection attempts fails - clientTransport.(*transport).config.HandshakeTimeout = 250 * time.Millisecond + clientTransport.(*transport).clientConfig.HandshakeTimeout = 250 * time.Millisecond _, err = clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).To(HaveOccurred()) Expect(err.(net.Error).Timeout()).To(BeTrue()) // now allow the address and make sure the connection goes through - clientTransport.(*transport).config.HandshakeTimeout = 2 * time.Second + clientTransport.(*transport).clientConfig.HandshakeTimeout = 2 * time.Second filters.AddFilter(ipNet, filter.ActionAccept) conn, err := clientTransport.Dial(context.Background(), ln.Multiaddr(), serverID) Expect(err).ToNot(HaveOccurred()) diff --git a/p2p/transport/quic/listener.go b/p2p/transport/quic/listener.go index 422f7be96c..9c77d24bac 100644 --- a/p2p/transport/quic/listener.go +++ b/p2p/transport/quic/listener.go @@ -36,7 +36,7 @@ func newListener(rconn *reuseConn, t *transport, localPeer peer.ID, key ic.PrivK conf, _ := identity.ConfigForAny() return conf, nil } - ln, err := quic.Listen(rconn, &tlsConf, t.config) + ln, err := quic.Listen(rconn, &tlsConf, t.serverConfig) if err != nil { return nil, err } diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index fcb6c82ff1..6707aa64b2 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -1,10 +1,15 @@ package libp2pquic import ( + "bufio" + "compress/gzip" "context" "errors" + "fmt" "io" "net" + "os" + "time" "github.com/minio/sha256-simd" "golang.org/x/crypto/hkdf" @@ -87,11 +92,12 @@ func (c *connManager) Dial(network string, raddr *net.UDPAddr) (*reuseConn, erro // The Transport implements the tpt.Transport interface for QUIC connections. type transport struct { - privKey ic.PrivKey - localPeer peer.ID - identity *p2ptls.Identity - connManager *connManager - config *quic.Config + privKey ic.PrivKey + localPeer peer.ID + identity *p2ptls.Identity + connManager *connManager + serverConfig *quic.Config + clientConfig *quic.Config } var _ tpt.Transport = &transport{} @@ -125,13 +131,17 @@ func NewTransport(key ic.PrivKey, psk pnet.PSK, filters *filter.Filters) (tpt.Tr return nil, err } - return &transport{ - privKey: key, - localPeer: localPeer, - identity: identity, - connManager: connManager, - config: config, - }, nil + t := &transport{ + privKey: key, + localPeer: localPeer, + identity: identity, + connManager: connManager, + serverConfig: config, + clientConfig: config.Clone(), + } + t.serverConfig.GetLogWriter = t.GetLogWriterFor("server") + t.clientConfig.GetLogWriter = t.GetLogWriterFor("client") + return t, nil } // Dial dials a new QUIC connection @@ -153,7 +163,7 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp if err != nil { return nil, err } - sess, err := quic.DialContext(ctx, pconn, addr, host, tlsConf, t.config) + sess, err := quic.DialContext(ctx, pconn, addr, host, tlsConf, t.clientConfig) if err != nil { pconn.DecreaseCount() return nil, err @@ -190,6 +200,32 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp }, nil } +func (t *transport) GetLogWriterFor(role string) func([]byte) io.WriteCloser { + qlogDir := os.Getenv("QLOGDIR") + if len(qlogDir) == 0 { + return nil + } + return func(connID []byte) io.WriteCloser { + // create the QLOGDIR, if it doesn't exist + if _, err := os.Stat(qlogDir); os.IsNotExist(err) { + if err := os.MkdirAll(qlogDir, 0777); err != nil { + log.Errorf("creating the QLOGDIR failed: %s", err) + return nil + } + } + now := time.Now() + t := fmt.Sprintf("%d-%02d-%02dT%02d-%02d-%02d-%06d", now.Year(), now.Month(), now.Day(), now.Hour(), now.Minute(), now.Second(), now.Nanosecond()/1000) + filename := fmt.Sprintf("%s/log_%s_%s_%x.qlog.gz", qlogDir, t, role, connID) + f, err := os.Create(filename) + if err != nil { + log.Errorf("unable to create qlog file %s: %s", filename, err) + return nil + } + gz := gzip.NewWriter(f) + return newBufferedWriteCloser(bufio.NewWriter(gz), gz) + } +} + // Don't use mafmt.QUIC as we don't want to dial DNS addresses. Just /ip{4,6}/udp/quic var dialMatcher = mafmt.And(mafmt.IP, mafmt.Base(ma.P_UDP), mafmt.Base(ma.P_QUIC)) From 6e90057f87afef38295631521a8450c4009e028e Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 6 Apr 2020 12:32:53 +0700 Subject: [PATCH 1799/3965] use RFC3339Nano date format for qlog file names --- p2p/transport/quic/transport.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 6707aa64b2..7c9b959797 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -213,8 +213,7 @@ func (t *transport) GetLogWriterFor(role string) func([]byte) io.WriteCloser { return nil } } - now := time.Now() - t := fmt.Sprintf("%d-%02d-%02dT%02d-%02d-%02d-%06d", now.Year(), now.Month(), now.Day(), now.Hour(), now.Minute(), now.Second(), now.Nanosecond()/1000) + t := time.Now().Format(time.RFC3339Nano) filename := fmt.Sprintf("%s/log_%s_%s_%x.qlog.gz", qlogDir, t, role, connID) f, err := os.Create(filename) if err != nil { From 4710507714496fea3a6f03727ccb3573f5d509e2 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 6 Apr 2020 12:34:16 +0700 Subject: [PATCH 1800/3965] simplify the mkdir for the QLOGDIR --- p2p/transport/quic/transport.go | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 7c9b959797..059b9f3685 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -207,11 +207,9 @@ func (t *transport) GetLogWriterFor(role string) func([]byte) io.WriteCloser { } return func(connID []byte) io.WriteCloser { // create the QLOGDIR, if it doesn't exist - if _, err := os.Stat(qlogDir); os.IsNotExist(err) { - if err := os.MkdirAll(qlogDir, 0777); err != nil { - log.Errorf("creating the QLOGDIR failed: %s", err) - return nil - } + if err := os.MkdirAll(qlogDir, 0777); err != nil { + log.Errorf("creating the QLOGDIR failed: %s", err) + return nil } t := time.Now().Format(time.RFC3339Nano) filename := fmt.Sprintf("%s/log_%s_%s_%x.qlog.gz", qlogDir, t, role, connID) From 615f125342dda78f74282f60db57c27f478f22be Mon Sep 17 00:00:00 2001 From: Aarsh Shah Date: Mon, 6 Apr 2020 11:34:45 +0530 Subject: [PATCH 1801/3965] Remove goprocess from Host (#865) * remove goprocess from Host --- go.mod | 1 + p2p/host/basic/basic_host.go | 71 +++++++++++++++++-------------- p2p/host/basic/basic_host_test.go | 11 +++++ p2p/protocol/identify/id.go | 35 ++++++++++----- p2p/protocol/identify/id_test.go | 31 ++++++++++---- 5 files changed, 97 insertions(+), 52 deletions(-) diff --git a/go.mod b/go.mod index 422aa17511..a0ae86d2ae 100644 --- a/go.mod +++ b/go.mod @@ -33,6 +33,7 @@ require ( github.com/multiformats/go-multiaddr-dns v0.2.0 github.com/multiformats/go-multiaddr-net v0.1.4 github.com/multiformats/go-multistream v0.1.1 + github.com/stretchr/testify v1.4.0 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 ) diff --git a/p2p/host/basic/basic_host.go b/p2p/host/basic/basic_host.go index f64e1840f8..ca721eb54e 100644 --- a/p2p/host/basic/basic_host.go +++ b/p2p/host/basic/basic_host.go @@ -4,6 +4,7 @@ import ( "context" "io" "net" + "sync" "time" "github.com/libp2p/go-libp2p/p2p/protocol/identify" @@ -21,8 +22,6 @@ import ( inat "github.com/libp2p/go-libp2p-nat" logging "github.com/ipfs/go-log" - "github.com/jbenet/goprocess" - goprocessctx "github.com/jbenet/goprocess/context" ma "github.com/multiformats/go-multiaddr" madns "github.com/multiformats/go-multiaddr-dns" @@ -68,6 +67,13 @@ const NATPortMap Option = iota // * uses an identity service to send + receive node information // * uses a nat service to establish NAT port mappings type BasicHost struct { + ctx context.Context + ctxCancel context.CancelFunc + // ensures we shutdown ONLY once + closeSync sync.Once + // keep track of resources we need to wait on before shutting down + refCount sync.WaitGroup + network network.Network mux *msmux.MultistreamMuxer ids *identify.IDService @@ -81,8 +87,6 @@ type BasicHost struct { negtimeout time.Duration - proc goprocess.Process - emitters struct { evtLocalProtocolsUpdated event.Emitter evtLocalAddrsUpdated event.Emitter @@ -128,6 +132,8 @@ type HostOpts struct { // NewHost constructs a new *BasicHost and activates it by attaching its stream and connection handlers to the given inet.Network. func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHost, error) { + hostCtx, cancel := context.WithCancel(ctx) + h := &BasicHost{ network: net, mux: msmux.NewMultistreamMuxer(), @@ -136,6 +142,8 @@ func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHo maResolver: madns.DefaultResolver, eventbus: eventbus.NewBus(), addrChangeChan: make(chan struct{}, 1), + ctx: hostCtx, + ctxCancel: cancel, } var err error @@ -146,28 +154,12 @@ func NewHost(ctx context.Context, net network.Network, opts *HostOpts) (*BasicHo return nil, err } - h.proc = goprocessctx.WithContextAndTeardown(ctx, func() error { - if h.natmgr != nil { - h.natmgr.Close() - } - if h.cmgr != nil { - h.cmgr.Close() - } - _ = h.emitters.evtLocalProtocolsUpdated.Close() - _ = h.emitters.evtLocalAddrsUpdated.Close() - return h.Network().Close() - }) - if opts.MultistreamMuxer != nil { h.mux = opts.MultistreamMuxer } // we can't set this as a default above because it depends on the *BasicHost. - h.ids = identify.NewIDService( - goprocessctx.WithProcessClosing(ctx, h.proc), - h, - identify.UserAgent(opts.UserAgent), - ) + h.ids = identify.NewIDService(h, identify.UserAgent(opts.UserAgent)) if uint64(opts.NegotiationTimeout) != 0 { h.negtimeout = opts.NegotiationTimeout @@ -242,7 +234,8 @@ func New(net network.Network, opts ...interface{}) *BasicHost { // Start starts background tasks in the host func (h *BasicHost) Start() { - h.proc.Go(h.background) + h.refCount.Add(1) + go h.background() } // newConnHandler is the remote-opened conn handler for inet.Network @@ -343,7 +336,9 @@ func makeUpdatedAddrEvent(prev, current []ma.Multiaddr) *event.EvtLocalAddresses return &evt } -func (h *BasicHost) background(p goprocess.Process) { +func (h *BasicHost) background() { + defer h.refCount.Done() + // periodically schedules an IdentifyPush to update our peers for changes // in our address set (if needed) ticker := time.NewTicker(10 * time.Second) @@ -356,7 +351,7 @@ func (h *BasicHost) background(p goprocess.Process) { select { case <-ticker.C: case <-h.addrChangeChan: - case <-p.Closing(): + case <-h.ctx.Done(): return } @@ -805,14 +800,26 @@ func (h *BasicHost) AllAddrs() []ma.Multiaddr { // Close shuts down the Host's services (network, etc). func (h *BasicHost) Close() error { - // You're thinking of adding some teardown logic here, right? Well - // don't! Add any process teardown logic to the teardown function in the - // constructor. - // - // This: - // 1. May be called multiple times. - // 2. May _never_ be called if the host is stopped by the context. - return h.proc.Close() + h.closeSync.Do(func() { + h.ctxCancel() + if h.natmgr != nil { + h.natmgr.Close() + } + if h.cmgr != nil { + h.cmgr.Close() + } + if h.ids != nil { + h.ids.Close() + } + + _ = h.emitters.evtLocalProtocolsUpdated.Close() + _ = h.emitters.evtLocalAddrsUpdated.Close() + h.Network().Close() + + h.refCount.Wait() + }) + + return nil } type streamWrapper struct { diff --git a/p2p/host/basic/basic_host_test.go b/p2p/host/basic/basic_host_test.go index c0dee8e571..bc6df3edee 100644 --- a/p2p/host/basic/basic_host_test.go +++ b/p2p/host/basic/basic_host_test.go @@ -22,6 +22,7 @@ import ( swarmt "github.com/libp2p/go-libp2p-swarm/testing" ma "github.com/multiformats/go-multiaddr" madns "github.com/multiformats/go-multiaddr-dns" + "github.com/stretchr/testify/require" ) func TestHostDoubleClose(t *testing.T) { @@ -80,6 +81,16 @@ func TestHostSimple(t *testing.T) { } } +func TestMultipleClose(t *testing.T) { + ctx := context.Background() + h := New(swarmt.GenSwarm(t, ctx)) + + require.NoError(t, h.Close()) + require.NoError(t, h.Close()) + require.NoError(t, h.Close()) + +} + func TestProtocolHandlerEvents(t *testing.T) { ctx := context.Background() h := New(swarmt.GenSwarm(t, ctx)) diff --git a/p2p/protocol/identify/id.go b/p2p/protocol/identify/id.go index a0e1ae25c3..5df656f60c 100644 --- a/p2p/protocol/identify/id.go +++ b/p2p/protocol/identify/id.go @@ -7,7 +7,6 @@ import ( "sync" "time" - "github.com/libp2p/go-eventbus" ic "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/event" "github.com/libp2p/go-libp2p-core/helpers" @@ -17,6 +16,7 @@ import ( "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/protocol" + "github.com/libp2p/go-eventbus" pb "github.com/libp2p/go-libp2p/p2p/protocol/identify/pb" ggio "github.com/gogo/protobuf/io" @@ -71,7 +71,12 @@ type IDService struct { Host host.Host UserAgent string - ctx context.Context + ctx context.Context + ctxCancel context.CancelFunc + // ensure we shutdown ONLY once + closeSync sync.Once + // track resources that need to be shut down before we shut down + refCount sync.WaitGroup // connections undergoing identification // for wait purposes @@ -94,7 +99,7 @@ type IDService struct { // NewIDService constructs a new *IDService and activates it by // attaching its stream handler to the given host.Host. -func NewIDService(ctx context.Context, h host.Host, opts ...Option) *IDService { +func NewIDService(h host.Host, opts ...Option) *IDService { var cfg config for _, opt := range opts { opt(&cfg) @@ -105,13 +110,15 @@ func NewIDService(ctx context.Context, h host.Host, opts ...Option) *IDService { userAgent = cfg.userAgent } + hostCtx, cancel := context.WithCancel(context.Background()) s := &IDService{ Host: h, UserAgent: userAgent, - ctx: ctx, + ctx: hostCtx, + ctxCancel: cancel, currid: make(map[network.Conn]chan struct{}), - observedAddrs: NewObservedAddrSet(ctx), + observedAddrs: NewObservedAddrSet(hostCtx), } // handle local protocol handler updates, and push deltas to peers. @@ -120,6 +127,7 @@ func NewIDService(ctx context.Context, h host.Host, opts ...Option) *IDService { if err != nil { log.Warningf("identify service not subscribed to local protocol handlers updates; err: %s", err) } else { + s.refCount.Add(1) go s.handleEvents() } @@ -143,14 +151,19 @@ func NewIDService(ctx context.Context, h host.Host, opts ...Option) *IDService { return s } +// Close shuts down the IDService +func (ids *IDService) Close() error { + ids.closeSync.Do(func() { + ids.ctxCancel() + ids.refCount.Wait() + }) + return nil +} + func (ids *IDService) handleEvents() { sub := ids.subscription - defer func() { - _ = sub.Close() - // drain the channel. - for range sub.Out() { - } - }() + defer ids.refCount.Done() + defer sub.Close() for { select { diff --git a/p2p/protocol/identify/id_test.go b/p2p/protocol/identify/id_test.go index 234dbe4891..e6d50c95c4 100644 --- a/p2p/protocol/identify/id_test.go +++ b/p2p/protocol/identify/id_test.go @@ -38,8 +38,10 @@ func subtestIDService(t *testing.T) { h1p := h1.ID() h2p := h2.ID() - ids1 := identify.NewIDService(ctx, h1) - ids2 := identify.NewIDService(ctx, h2) + ids1 := identify.NewIDService(h1) + ids2 := identify.NewIDService(h2) + defer ids1.Close() + defer ids2.Close() testKnowsAddrs(t, h1, h2p, []ma.Multiaddr{}) // nothing testKnowsAddrs(t, h2, h1p, []ma.Multiaddr{}) // nothing @@ -253,9 +255,14 @@ func TestLocalhostAddrFiltering(t *testing.T) { Addrs: p2addrs[1:], }) - _ = identify.NewIDService(ctx, p1) - ids2 := identify.NewIDService(ctx, p2) - ids3 := identify.NewIDService(ctx, p3) + ids1 := identify.NewIDService(p1) + ids2 := identify.NewIDService(p2) + ids3 := identify.NewIDService(p3) + defer func() { + ids1.Close() + ids2.Close() + ids3.Close() + }() conns := p2.Network().ConnsToPeer(id1) if len(conns) == 0 { @@ -291,8 +298,12 @@ func TestIdentifyDeltaOnProtocolChange(t *testing.T) { h2.SetStreamHandler(protocol.TestingID, func(_ network.Stream) {}) - ids1 := identify.NewIDService(ctx, h1) - _ = identify.NewIDService(ctx, h2) + ids1 := identify.NewIDService(h1) + ids2 := identify.NewIDService(h2) + defer func() { + ids1.Close() + ids2.Close() + }() if err := h1.Connect(ctx, peer.AddrInfo{ID: h2.ID(), Addrs: h2.Addrs()}); err != nil { t.Fatal(err) @@ -404,8 +415,10 @@ func TestIdentifyDeltaWhileIdentifyingConn(t *testing.T) { defer h2.Close() defer h1.Close() - _ = identify.NewIDService(ctx, h1) - ids2 := identify.NewIDService(ctx, h2) + ids1 := identify.NewIDService(h1) + ids2 := identify.NewIDService(h2) + defer ids1.Close() + defer ids2.Close() // replace the original identify handler by one that blocks until we close the block channel. // this allows us to control how long identify runs. From 9d18160c9f40e3ef6c5a8d6543b9dc9fd1a817e3 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 2 Jul 2019 11:44:57 -0700 Subject: [PATCH 1802/3965] fix: handle nil peer IDs Feels like Java all over again. fixes #87 --- p2p/host/peerstore/pstoreds/addr_book.go | 8 +++++ p2p/host/peerstore/pstoreds/metadata.go | 6 ++++ p2p/host/peerstore/pstoreds/protobook.go | 20 +++++++++++ p2p/host/peerstore/pstoremem/addr_book.go | 42 ++++++++++++++++++++++ p2p/host/peerstore/pstoremem/metadata.go | 6 ++++ p2p/host/peerstore/pstoremem/protobook.go | 20 +++++++++++ p2p/host/peerstore/test/peerstore_suite.go | 33 +++++++++++++++++ 7 files changed, 135 insertions(+) diff --git a/p2p/host/peerstore/pstoreds/addr_book.go b/p2p/host/peerstore/pstoreds/addr_book.go index 12cb0814e8..88082102bb 100644 --- a/p2p/host/peerstore/pstoreds/addr_book.go +++ b/p2p/host/peerstore/pstoreds/addr_book.go @@ -207,6 +207,9 @@ func (ab *dsAddrBook) Close() error { // // If the cache argument is true, the record is inserted in the cache when loaded from the datastore. func (ab *dsAddrBook) loadRecord(id peer.ID, cache bool, update bool) (pr *addrsRecord, err error) { + if err := id.Validate(); err != nil { + return nil, err + } if e, ok := ab.cache.Get(id); ok { pr = e.(*addrsRecord) pr.Lock() @@ -421,6 +424,11 @@ func (ab *dsAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multi // ClearAddrs will delete all known addresses for a peer ID. func (ab *dsAddrBook) ClearAddrs(p peer.ID) { + if err := p.Validate(); err != nil { + // nothing to do + return + } + ab.cache.Remove(p) key := addrBookBase.ChildString(b32.RawStdEncoding.EncodeToString([]byte(p))) diff --git a/p2p/host/peerstore/pstoreds/metadata.go b/p2p/host/peerstore/pstoreds/metadata.go index 4d285af789..bf7655231c 100644 --- a/p2p/host/peerstore/pstoreds/metadata.go +++ b/p2p/host/peerstore/pstoreds/metadata.go @@ -41,6 +41,9 @@ func NewPeerMetadata(_ context.Context, store ds.Datastore, _ Options) (*dsPeerM } func (pm *dsPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { + if err := p.Validate(); err != nil { + return nil, err + } k := pmBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))).ChildString(key) value, err := pm.ds.Get(k) if err != nil { @@ -58,6 +61,9 @@ func (pm *dsPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { } func (pm *dsPeerMetadata) Put(p peer.ID, key string, val interface{}) error { + if err := p.Validate(); err != nil { + return err + } k := pmBase.ChildString(base32.RawStdEncoding.EncodeToString([]byte(p))).ChildString(key) var buf pool.Buffer if err := gob.NewEncoder(&buf).Encode(&val); err != nil { diff --git a/p2p/host/peerstore/pstoreds/protobook.go b/p2p/host/peerstore/pstoreds/protobook.go index 27e70f556d..84d546b912 100644 --- a/p2p/host/peerstore/pstoreds/protobook.go +++ b/p2p/host/peerstore/pstoreds/protobook.go @@ -39,6 +39,10 @@ func NewProtoBook(meta pstore.PeerMetadata) *dsProtoBook { } func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { + if err := p.Validate(); err != nil { + return err + } + s := pb.segments.get(p) s.Lock() defer s.Unlock() @@ -52,6 +56,10 @@ func (pb *dsProtoBook) SetProtocols(p peer.ID, protos ...string) error { } func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { + if err := p.Validate(); err != nil { + return err + } + s := pb.segments.get(p) s.Lock() defer s.Unlock() @@ -69,6 +77,10 @@ func (pb *dsProtoBook) AddProtocols(p peer.ID, protos ...string) error { } func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { + if err := p.Validate(); err != nil { + return nil, err + } + s := pb.segments.get(p) s.RLock() defer s.RUnlock() @@ -87,6 +99,10 @@ func (pb *dsProtoBook) GetProtocols(p peer.ID) ([]string, error) { } func (pb *dsProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { + if err := p.Validate(); err != nil { + return nil, err + } + s := pb.segments.get(p) s.RLock() defer s.RUnlock() @@ -107,6 +123,10 @@ func (pb *dsProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, } func (pb *dsProtoBook) RemoveProtocols(p peer.ID, protos ...string) error { + if err := p.Validate(); err != nil { + return err + } + s := pb.segments.get(p) s.Lock() defer s.Unlock() diff --git a/p2p/host/peerstore/pstoremem/addr_book.go b/p2p/host/peerstore/pstoremem/addr_book.go index cb7340f34b..48f7f44996 100644 --- a/p2p/host/peerstore/pstoremem/addr_book.go +++ b/p2p/host/peerstore/pstoremem/addr_book.go @@ -196,6 +196,11 @@ func (mab *memoryAddrBook) ConsumePeerRecord(recordEnvelope *record.Envelope, tt } func (mab *memoryAddrBook) addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration, signed bool) { + if err := p.Validate(); err != nil { + log.Warningf("tried to set addrs for invalid peer ID %s: %s", p, err) + return + } + // if ttl is zero, exit. nothing to do. if ttl <= 0 { return @@ -244,12 +249,22 @@ func (mab *memoryAddrBook) addAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du // SetAddr calls mgr.SetAddrs(p, addr, ttl) func (mab *memoryAddrBook) SetAddr(p peer.ID, addr ma.Multiaddr, ttl time.Duration) { + if err := p.Validate(); err != nil { + log.Warningf("tried to set addrs for invalid peer ID %s: %s", p, err) + return + } + mab.SetAddrs(p, []ma.Multiaddr{addr}, ttl) } // SetAddrs sets the ttl on addresses. This clears any TTL there previously. // This is used when we receive the best estimate of the validity of an address. func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) { + if err := p.Validate(); err != nil { + log.Warningf("tried to set addrs for invalid peer ID %s: %s", p, err) + return + } + s := mab.segments.get(p) s.Lock() defer s.Unlock() @@ -287,6 +302,11 @@ func (mab *memoryAddrBook) SetAddrs(p peer.ID, addrs []ma.Multiaddr, ttl time.Du // UpdateAddrs updates the addresses associated with the given peer that have // the given oldTTL to have the given newTTL. func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL time.Duration) { + if err := p.Validate(); err != nil { + log.Warningf("tried to set addrs for invalid peer ID %s: %s", p, err) + return + } + s := mab.segments.get(p) s.Lock() defer s.Unlock() @@ -310,6 +330,11 @@ func (mab *memoryAddrBook) UpdateAddrs(p peer.ID, oldTTL time.Duration, newTTL t // Addrs returns all known (and valid) addresses for a given peer func (mab *memoryAddrBook) Addrs(p peer.ID) []ma.Multiaddr { + if err := p.Validate(); err != nil { + // invalid peer ID = no addrs + return nil + } + s := mab.segments.get(p) s.RLock() defer s.RUnlock() @@ -336,6 +361,11 @@ func validAddrs(amap map[string]*expiringAddr) []ma.Multiaddr { // given peer id, if one exists. // Returns nil if no signed PeerRecord exists for the peer. func (mab *memoryAddrBook) GetPeerRecord(p peer.ID) *record.Envelope { + if err := p.Validate(); err != nil { + // invalid peer ID = no addrs + return nil + } + s := mab.segments.get(p) s.RLock() defer s.RUnlock() @@ -356,6 +386,11 @@ func (mab *memoryAddrBook) GetPeerRecord(p peer.ID) *record.Envelope { // ClearAddrs removes all previously stored addresses func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { + if err := p.Validate(); err != nil { + // nothing to clear + return + } + s := mab.segments.get(p) s.Lock() defer s.Unlock() @@ -367,6 +402,13 @@ func (mab *memoryAddrBook) ClearAddrs(p peer.ID) { // AddrStream returns a channel on which all new addresses discovered for a // given peer ID will be published. func (mab *memoryAddrBook) AddrStream(ctx context.Context, p peer.ID) <-chan ma.Multiaddr { + if err := p.Validate(); err != nil { + log.Warningf("tried to get addrs for invalid peer ID %s: %s", p, err) + ch := make(chan ma.Multiaddr) + close(ch) + return ch + } + s := mab.segments.get(p) s.RLock() defer s.RUnlock() diff --git a/p2p/host/peerstore/pstoremem/metadata.go b/p2p/host/peerstore/pstoremem/metadata.go index 7ded769192..7bd0fb3d72 100644 --- a/p2p/host/peerstore/pstoremem/metadata.go +++ b/p2p/host/peerstore/pstoremem/metadata.go @@ -35,6 +35,9 @@ func NewPeerMetadata() *memoryPeerMetadata { } func (ps *memoryPeerMetadata) Put(p peer.ID, key string, val interface{}) error { + if err := p.Validate(); err != nil { + return err + } ps.dslock.Lock() defer ps.dslock.Unlock() if vals, ok := val.(string); ok && internKeys[key] { @@ -49,6 +52,9 @@ func (ps *memoryPeerMetadata) Put(p peer.ID, key string, val interface{}) error } func (ps *memoryPeerMetadata) Get(p peer.ID, key string) (interface{}, error) { + if err := p.Validate(); err != nil { + return nil, err + } ps.dslock.RLock() defer ps.dslock.RUnlock() i, ok := ps.ds[metakey{p, key}] diff --git a/p2p/host/peerstore/pstoremem/protobook.go b/p2p/host/peerstore/pstoremem/protobook.go index 04d8ec47ae..af44c6fe38 100644 --- a/p2p/host/peerstore/pstoremem/protobook.go +++ b/p2p/host/peerstore/pstoremem/protobook.go @@ -67,6 +67,10 @@ func (pb *memoryProtoBook) internProtocol(proto string) string { } func (pb *memoryProtoBook) SetProtocols(p peer.ID, protos ...string) error { + if err := p.Validate(); err != nil { + return err + } + s := pb.segments.get(p) s.Lock() defer s.Unlock() @@ -82,6 +86,10 @@ func (pb *memoryProtoBook) SetProtocols(p peer.ID, protos ...string) error { } func (pb *memoryProtoBook) AddProtocols(p peer.ID, protos ...string) error { + if err := p.Validate(); err != nil { + return err + } + s := pb.segments.get(p) s.Lock() defer s.Unlock() @@ -100,6 +108,10 @@ func (pb *memoryProtoBook) AddProtocols(p peer.ID, protos ...string) error { } func (pb *memoryProtoBook) GetProtocols(p peer.ID) ([]string, error) { + if err := p.Validate(); err != nil { + return nil, err + } + s := pb.segments.get(p) s.RLock() defer s.RUnlock() @@ -113,6 +125,10 @@ func (pb *memoryProtoBook) GetProtocols(p peer.ID) ([]string, error) { } func (pb *memoryProtoBook) RemoveProtocols(p peer.ID, protos ...string) error { + if err := p.Validate(); err != nil { + return err + } + s := pb.segments.get(p) s.Lock() defer s.Unlock() @@ -130,6 +146,10 @@ func (pb *memoryProtoBook) RemoveProtocols(p peer.ID, protos ...string) error { } func (pb *memoryProtoBook) SupportsProtocols(p peer.ID, protos ...string) ([]string, error) { + if err := p.Validate(); err != nil { + return nil, err + } + s := pb.segments.get(p) s.RLock() defer s.RUnlock() diff --git a/p2p/host/peerstore/test/peerstore_suite.go b/p2p/host/peerstore/test/peerstore_suite.go index 8a629624be..5797297aec 100644 --- a/p2p/host/peerstore/test/peerstore_suite.go +++ b/p2p/host/peerstore/test/peerstore_suite.go @@ -279,6 +279,29 @@ func testPeerstoreProtoStore(ps pstore.Peerstore) func(t *testing.T) { if !reflect.DeepEqual(supported, protos[2:]) { t.Fatal("expected only one protocol to remain") } + + // test bad peer IDs + badp := peer.ID("") + + err = ps.AddProtocols(badp, protos...) + if err == nil { + t.Fatal("expected error when using a bad peer ID") + } + + _, err = ps.GetProtocols(badp) + if err == nil || err == pstore.ErrNotFound { + t.Fatal("expected error when using a bad peer ID") + } + + _, err = ps.SupportsProtocols(badp, "q", "w", "a", "y", "b") + if err == nil || err == pstore.ErrNotFound { + t.Fatal("expected error when using a bad peer ID") + } + + err = ps.RemoveProtocols(badp) + if err == nil || err == pstore.ErrNotFound { + t.Fatal("expected error when using a bad peer ID") + } } } @@ -309,6 +332,10 @@ func testBasicPeerstore(ps pstore.Peerstore) func(t *testing.T) { if !pinfo.Addrs[0].Equal(addrs[0]) { t.Fatal("stored wrong address") } + + // should fail silently... + ps.AddAddrs("", addrs, pstore.PermanentAddrTTL) + ps.Addrs("") } } @@ -355,6 +382,12 @@ func testMetadata(ps pstore.Peerstore) func(t *testing.T) { continue } } + if err := ps.Put("", "foobar", "thing"); err == nil { + t.Errorf("expected error for bad peer ID") + } + if _, err := ps.Get("", "foobar"); err == nil || err == pstore.ErrNotFound { + t.Errorf("expected error for bad peer ID") + } } } From 8ffd6a94e23d9478f50363501b9c93010d77b5ca Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 6 Apr 2020 11:22:03 +0000 Subject: [PATCH 1803/3965] build(deps): bump github.com/stretchr/testify from 1.4.0 to 1.5.1 Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.4.0 to 1.5.1. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](https://github.com/stretchr/testify/compare/v1.4.0...v1.5.1) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index a0ae86d2ae..c77785fd44 100644 --- a/go.mod +++ b/go.mod @@ -33,7 +33,7 @@ require ( github.com/multiformats/go-multiaddr-dns v0.2.0 github.com/multiformats/go-multiaddr-net v0.1.4 github.com/multiformats/go-multistream v0.1.1 - github.com/stretchr/testify v1.4.0 + github.com/stretchr/testify v1.5.1 github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9 ) diff --git a/go.sum b/go.sum index 88747dc217..061ac251ef 100644 --- a/go.sum +++ b/go.sum @@ -383,11 +383,14 @@ github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnIn github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= github.com/src-d/envconfig v1.0.0/go.mod h1:Q9YQZ7BKITldTBnoxsE5gOeB5y66RyPXeue/R4aaNBc= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.1.1 h1:2vfRuCMp5sSVIDSqO8oNnWJq7mPa6KVP3iPIwFBuy8A= github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4= +github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1/go.mod h1:8UvriyWtv5Q5EOgjHaSseUEdkQfvwFv1I/In/O2M9gc= From a9e9654f0ab6aaad70d381f465224ca2e2ba9352 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Tue, 7 Apr 2020 11:20:37 +0000 Subject: [PATCH 1804/3965] build(deps): bump github.com/libp2p/go-libp2p-peerstore Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.2 to 0.2.3. - [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases) - [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.2...v0.2.3) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index c77785fd44..66c6bed1ff 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/libp2p/go-libp2p-mplex v0.2.3 github.com/libp2p/go-libp2p-nat v0.0.6 github.com/libp2p/go-libp2p-netutil v0.1.0 - github.com/libp2p/go-libp2p-peerstore v0.2.2 + github.com/libp2p/go-libp2p-peerstore v0.2.3 github.com/libp2p/go-libp2p-secio v0.2.2 github.com/libp2p/go-libp2p-swarm v0.2.3 github.com/libp2p/go-libp2p-testing v0.1.1 diff --git a/go.sum b/go.sum index 061ac251ef..d0d13b4264 100644 --- a/go.sum +++ b/go.sum @@ -3,6 +3,7 @@ github.com/AndreasBriese/bbloom v0.0.0-20180913140656-343706a395b7/go.mod h1:bOv github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/Kubuxu/go-os-helper v0.0.1/go.mod h1:N8B+I7vPCT80IcP58r50u4+gEEcsZETFUpAzWW2ep1Y= +github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32/go.mod h1:DrZx5ec/dmnfpw9KyYoQyYo7d0KEvTkk/5M/vbZjAr8= @@ -18,6 +19,7 @@ github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVa github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc= github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY= github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs= +github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk= @@ -34,6 +36,8 @@ github.com/davidlazar/go-crypto v0.0.0-20170701192655-dcfb0a7ac018/go.mod h1:rQY github.com/dgraph-io/badger v1.5.5-0.20190226225317-8115aed38f8f/go.mod h1:VZxzAIRPHRVNRKRo6AXrX9BJegn6il06VMTZVJYCIjQ= github.com/dgraph-io/badger v1.6.0-rc1/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= github.com/dgraph-io/badger v1.6.0/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= +github.com/dgraph-io/badger v1.6.1/go.mod h1:FRmFw3uxvcpa8zG3Rxs0th+hCLIuaQg8HlNV5bjgnuU= +github.com/dgraph-io/ristretto v0.0.2/go.mod h1:KPxhHT9ZxKefz+PCeOGsrHpl1qZ7i70dGTu2u+Ahh6E= github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= @@ -92,8 +96,10 @@ github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46U github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8= github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaHzfGTzuE3s= github.com/ipfs/go-ds-badger v0.2.1/go.mod h1:Tx7l3aTph3FMFrRS838dcSJh+jjA7cX9DrGVwx/NOwE= +github.com/ipfs/go-ds-badger v0.2.3/go.mod h1:pEYw0rgg3FIrywKKnL+Snr+w/LjJZVMTBRn4FS6UHUk= github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc= github.com/ipfs/go-ds-leveldb v0.4.1/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s= +github.com/ipfs/go-ds-leveldb v0.4.2/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50= github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc= @@ -135,6 +141,7 @@ github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d h1:68u9r4wEvL3gYg2jv github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= @@ -205,6 +212,8 @@ github.com/libp2p/go-libp2p-peerstore v0.2.1 h1:u+gOfsKgu73ZkGWhvckRm03z9C+iS9Tr github.com/libp2p/go-libp2p-peerstore v0.2.1/go.mod h1:NQxhNjWxf1d4w6PihR8btWIRjwRLBr4TYKfNgrUkOPA= github.com/libp2p/go-libp2p-peerstore v0.2.2 h1:iqc/m03jHn5doXN3+kS6JKvqQRHEltiXljQB85iVHWE= github.com/libp2p/go-libp2p-peerstore v0.2.2/go.mod h1:NQxhNjWxf1d4w6PihR8btWIRjwRLBr4TYKfNgrUkOPA= +github.com/libp2p/go-libp2p-peerstore v0.2.3 h1:MofRq2l3c15vQpEygTetV+zRRrncz+ktiXW7H2EKoEQ= +github.com/libp2p/go-libp2p-peerstore v0.2.3/go.mod h1:K8ljLdFn590GMttg/luh4caB/3g0vKuY01psze0upRw= github.com/libp2p/go-libp2p-pnet v0.2.0 h1:J6htxttBipJujEjz1y0a5+eYoiPcFHhSYHH6na5f0/k= github.com/libp2p/go-libp2p-pnet v0.2.0/go.mod h1:Qqvq6JH/oMZGwqs3N1Fqhv8NVhrdYcO0BW4wssv21LA= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= @@ -373,6 +382,7 @@ github.com/smola/gocompat v0.2.0/go.mod h1:1B0MlxbmoZNo3h8guHp8HztB3BSYR5itql9qt github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572/go.mod h1:w0SWMsp6j9O/dk4/ZpIhL+3CkG8ofA2vuv7k+ltqUMc= +github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= From 3ec3559c03b70344c07e471a7656eaa824126136 Mon Sep 17 00:00:00 2001 From: Will Date: Tue, 7 Apr 2020 07:51:20 -0700 Subject: [PATCH 1805/3965] switch local route binding to use netroute (#134) switch local route binding to use netroute --- p2p/transport/quic/libp2pquic_suite_test.go | 2 +- p2p/transport/quic/netlink_linux.go | 10 --- p2p/transport/quic/netlink_other.go | 9 --- .../quic/{reuse_base.go => reuse.go} | 39 ++++++++--- p2p/transport/quic/reuse_linux_test.go | 42 ------------ p2p/transport/quic/reuse_not_win.go | 68 ------------------- p2p/transport/quic/reuse_test.go | 33 ++++++++- p2p/transport/quic/reuse_win.go | 30 -------- p2p/transport/quic/transport.go | 11 +-- 9 files changed, 64 insertions(+), 180 deletions(-) delete mode 100644 p2p/transport/quic/netlink_linux.go delete mode 100644 p2p/transport/quic/netlink_other.go rename p2p/transport/quic/{reuse_base.go => reuse.go} (81%) delete mode 100644 p2p/transport/quic/reuse_linux_test.go delete mode 100644 p2p/transport/quic/reuse_not_win.go delete mode 100644 p2p/transport/quic/reuse_win.go diff --git a/p2p/transport/quic/libp2pquic_suite_test.go b/p2p/transport/quic/libp2pquic_suite_test.go index d6ae1510fd..5905763c07 100644 --- a/p2p/transport/quic/libp2pquic_suite_test.go +++ b/p2p/transport/quic/libp2pquic_suite_test.go @@ -32,7 +32,7 @@ var ( func isGarbageCollectorRunning() bool { var b bytes.Buffer pprof.Lookup("goroutine").WriteTo(&b, 1) - return strings.Contains(b.String(), "go-libp2p-quic-transport.(*reuseBase).runGarbageCollector") + return strings.Contains(b.String(), "go-libp2p-quic-transport.(*reuse).runGarbageCollector") } var _ = BeforeEach(func() { diff --git a/p2p/transport/quic/netlink_linux.go b/p2p/transport/quic/netlink_linux.go deleted file mode 100644 index a5f6ffe6e1..0000000000 --- a/p2p/transport/quic/netlink_linux.go +++ /dev/null @@ -1,10 +0,0 @@ -// +build linux - -package libp2pquic - -import "golang.org/x/sys/unix" - -// We just need netlink_route here. -// note: We should avoid the use of netlink_xfrm or netlink_netfilter has it is -// not allowed by Android in his base policy. -var SupportedNlFamilies = []int{unix.NETLINK_ROUTE} diff --git a/p2p/transport/quic/netlink_other.go b/p2p/transport/quic/netlink_other.go deleted file mode 100644 index ccc073da0b..0000000000 --- a/p2p/transport/quic/netlink_other.go +++ /dev/null @@ -1,9 +0,0 @@ -// +build !linux -// +build !windows - -package libp2pquic - -import "github.com/vishvananda/netlink/nl" - -// SupportedNlFamilies is the default netlink families used by the netlink package -var SupportedNlFamilies = nl.SupportedNlFamilies diff --git a/p2p/transport/quic/reuse_base.go b/p2p/transport/quic/reuse.go similarity index 81% rename from p2p/transport/quic/reuse_base.go rename to p2p/transport/quic/reuse.go index 053b777395..f566de2fd0 100644 --- a/p2p/transport/quic/reuse_base.go +++ b/p2p/transport/quic/reuse.go @@ -6,6 +6,7 @@ import ( "time" filter "github.com/libp2p/go-maddr-filter" + "github.com/libp2p/go-netroute" ) // Constant. Defined as variables to simplify testing. @@ -51,7 +52,7 @@ func (c *reuseConn) ShouldGarbageCollect(now time.Time) bool { return !c.unusedSince.IsZero() && c.unusedSince.Add(maxUnusedDuration).Before(now) } -type reuseBase struct { +type reuse struct { mutex sync.Mutex filters *filter.Filters @@ -63,15 +64,15 @@ type reuseBase struct { global map[int]*reuseConn } -func newReuseBase(filters *filter.Filters) reuseBase { - return reuseBase{ +func newReuse(filters *filter.Filters) *reuse { + return &reuse{ filters: filters, unicast: make(map[string]map[int]*reuseConn), global: make(map[int]*reuseConn), } } -func (r *reuseBase) runGarbageCollector() { +func (r *reuse) runGarbageCollector() { ticker := time.NewTicker(garbageCollectInterval) defer ticker.Stop() @@ -110,17 +111,37 @@ func (r *reuseBase) runGarbageCollector() { } // must be called while holding the mutex -func (r *reuseBase) maybeStartGarbageCollector() { +func (r *reuse) maybeStartGarbageCollector() { if !r.garbageCollectorRunning { r.garbageCollectorRunning = true go r.runGarbageCollector() } } +func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { + var ip *net.IP + if router, err := netroute.New(); err == nil { + _, _, src, err := router.Route(raddr.IP) + if err == nil && !src.IsUnspecified() { + ip = &src + } + } + + r.mutex.Lock() + defer r.mutex.Unlock() + + conn, err := r.dialLocked(network, raddr, ip) + if err != nil { + return nil, err + } + conn.IncreaseCount() + r.maybeStartGarbageCollector() + return conn, nil +} -func (r *reuseBase) dialLocked(network string, raddr *net.UDPAddr, ips []net.IP) (*reuseConn, error) { - for _, ip := range ips { +func (r *reuse) dialLocked(network string, raddr *net.UDPAddr, source *net.IP) (*reuseConn, error) { + if source != nil { // We already have at least one suitable connection... - if conns, ok := r.unicast[ip.String()]; ok { + if conns, ok := r.unicast[source.String()]; ok { // ... we don't care which port we're dialing from. Just use the first. for _, c := range conns { return c, nil @@ -152,7 +173,7 @@ func (r *reuseBase) dialLocked(network string, raddr *net.UDPAddr, ips []net.IP) return rconn, nil } -func (r *reuseBase) Listen(network string, laddr *net.UDPAddr) (*reuseConn, error) { +func (r *reuse) Listen(network string, laddr *net.UDPAddr) (*reuseConn, error) { conn, err := net.ListenUDP(network, laddr) if err != nil { return nil, err diff --git a/p2p/transport/quic/reuse_linux_test.go b/p2p/transport/quic/reuse_linux_test.go deleted file mode 100644 index 6fe77dbbf6..0000000000 --- a/p2p/transport/quic/reuse_linux_test.go +++ /dev/null @@ -1,42 +0,0 @@ -// +build linux - -package libp2pquic - -import ( - "net" - - . "github.com/onsi/ginkgo" - . "github.com/onsi/gomega" -) - -var _ = Describe("Reuse (on Linux)", func() { - var reuse *reuse - - BeforeEach(func() { - var err error - reuse, err = newReuse(nil) - Expect(err).ToNot(HaveOccurred()) - }) - - Context("creating and reusing connections", func() { - AfterEach(func() { closeAllConns(reuse) }) - - It("reuses a connection it created for listening on a specific interface", func() { - raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") - Expect(err).ToNot(HaveOccurred()) - ips, err := reuse.getSourceIPs("udp4", raddr) - Expect(err).ToNot(HaveOccurred()) - Expect(ips).ToNot(BeEmpty()) - // listen - addr, err := net.ResolveUDPAddr("udp4", ips[0].String()+":0") - Expect(err).ToNot(HaveOccurred()) - lconn, err := reuse.Listen("udp4", addr) - Expect(err).ToNot(HaveOccurred()) - Expect(lconn.GetCount()).To(Equal(1)) - // dial - conn, err := reuse.Dial("udp4", raddr) - Expect(err).ToNot(HaveOccurred()) - Expect(conn.GetCount()).To(Equal(2)) - }) - }) -}) diff --git a/p2p/transport/quic/reuse_not_win.go b/p2p/transport/quic/reuse_not_win.go deleted file mode 100644 index 57097a3098..0000000000 --- a/p2p/transport/quic/reuse_not_win.go +++ /dev/null @@ -1,68 +0,0 @@ -// +build !windows - -package libp2pquic - -import ( - "net" - - filter "github.com/libp2p/go-maddr-filter" - - "github.com/vishvananda/netlink" -) - -type reuse struct { - reuseBase - - handle *netlink.Handle // Only set on Linux. nil on other systems. -} - -func newReuse(filters *filter.Filters) (*reuse, error) { - handle, err := netlink.NewHandle(SupportedNlFamilies...) - if err == netlink.ErrNotImplemented { - handle = nil - } else if err != nil { - return nil, err - } - return &reuse{ - reuseBase: newReuseBase(filters), - handle: handle, - }, nil -} - -// Get the source IP that the kernel would use for dialing. -// This only works on Linux. -// On other systems, this returns an empty slice of IP addresses. -func (r *reuse) getSourceIPs(network string, raddr *net.UDPAddr) ([]net.IP, error) { - if r.handle == nil { - return nil, nil - } - - routes, err := r.handle.RouteGet(raddr.IP) - if err != nil { - return nil, err - } - - ips := make([]net.IP, 0, len(routes)) - for _, route := range routes { - ips = append(ips, route.Src) - } - return ips, nil -} - -func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { - ips, err := r.getSourceIPs(network, raddr) - if err != nil { - return nil, err - } - - r.mutex.Lock() - defer r.mutex.Unlock() - - conn, err := r.dialLocked(network, raddr, ips) - if err != nil { - return nil, err - } - conn.IncreaseCount() - r.maybeStartGarbageCollector() - return conn, nil -} diff --git a/p2p/transport/quic/reuse_test.go b/p2p/transport/quic/reuse_test.go index 6a24d5d575..1072bd00be 100644 --- a/p2p/transport/quic/reuse_test.go +++ b/p2p/transport/quic/reuse_test.go @@ -4,6 +4,7 @@ import ( "net" "time" + "github.com/libp2p/go-netroute" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) @@ -32,13 +33,19 @@ func closeAllConns(reuse *reuse) { Eventually(isGarbageCollectorRunning).Should(BeFalse()) } +func OnPlatformsWithRoutingTablesIt(description string, f interface{}) { + if _, err := netroute.New(); err == nil { + It(description, f) + } else { + PIt(description, f) + } +} + var _ = Describe("Reuse", func() { var reuse *reuse BeforeEach(func() { - var err error - reuse, err = newReuse(nil) - Expect(err).ToNot(HaveOccurred()) + reuse = newReuse(nil) }) Context("creating and reusing connections", func() { @@ -85,6 +92,26 @@ var _ = Describe("Reuse", func() { Expect(err).ToNot(HaveOccurred()) Expect(conn.GetCount()).To(Equal(2)) }) + + OnPlatformsWithRoutingTablesIt("reuses a connection it created for listening on a specific interface", func() { + router, err := netroute.New() + Expect(err).ToNot(HaveOccurred()) + + raddr, err := net.ResolveUDPAddr("udp4", "1.1.1.1:1234") + Expect(err).ToNot(HaveOccurred()) + _, _, ip, err := router.Route(raddr.IP) + Expect(err).ToNot(HaveOccurred()) + // listen + addr, err := net.ResolveUDPAddr("udp4", ip.String()+":0") + Expect(err).ToNot(HaveOccurred()) + lconn, err := reuse.Listen("udp4", addr) + Expect(err).ToNot(HaveOccurred()) + Expect(lconn.GetCount()).To(Equal(1)) + // dial + conn, err := reuse.Dial("udp4", raddr) + Expect(err).ToNot(HaveOccurred()) + Expect(conn.GetCount()).To(Equal(2)) + }) }) Context("garbage-collecting connections", func() { diff --git a/p2p/transport/quic/reuse_win.go b/p2p/transport/quic/reuse_win.go deleted file mode 100644 index 14ea1babfa..0000000000 --- a/p2p/transport/quic/reuse_win.go +++ /dev/null @@ -1,30 +0,0 @@ -// +build windows - -package libp2pquic - -import ( - "net" - - filter "github.com/libp2p/go-maddr-filter" -) - -type reuse struct { - reuseBase -} - -func newReuse(filters *filter.Filters) (*reuse, error) { - return &reuse{reuseBase: newReuseBase(filters)}, nil -} - -func (r *reuse) Dial(network string, raddr *net.UDPAddr) (*reuseConn, error) { - r.mutex.Lock() - defer r.mutex.Unlock() - - conn, err := r.dialLocked(network, raddr, nil) - if err != nil { - return nil, err - } - conn.IncreaseCount() - r.maybeStartGarbageCollector() - return conn, nil -} diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 059b9f3685..97add03c07 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -49,14 +49,9 @@ type connManager struct { } func newConnManager(filters *filter.Filters) (*connManager, error) { - reuseUDP4, err := newReuse(filters) - if err != nil { - return nil, err - } - reuseUDP6, err := newReuse(filters) - if err != nil { - return nil, err - } + reuseUDP4 := newReuse(filters) + reuseUDP6 := newReuse(filters) + return &connManager{ reuseUDP4: reuseUDP4, reuseUDP6: reuseUDP6, From 10fc36520602a3d5404dae91a909124e6e22ad01 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 7 Apr 2020 10:55:15 -0700 Subject: [PATCH 1806/3965] Run Autonat Service while in unknown connectivity mode This supports local LAN connectivity / discovery --- p2p/host/autonat/autonat.go | 10 +++++----- p2p/host/autonat/svc_test.go | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index 04ec7a1a46..aa355d3308 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -46,6 +46,7 @@ type AmbientAutoNAT struct { emitReachabilityChanged event.Emitter } +// StaticAutoNAT is a simple AutoNAT implementation when a single NAT status is desired. type StaticAutoNAT struct { ctx context.Context host host.Host @@ -85,15 +86,12 @@ func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { if err != nil { return nil, err } + service.Enable() } if conf.forceReachability { emitReachabilityChanged.Emit(event.EvtLocalReachabilityChanged{Reachability: conf.reachability}) - // The serice will only exist when reachability is public. - if service != nil { - service.Enable() - } return &StaticAutoNAT{ ctx: ctx, host: h, @@ -297,7 +295,7 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { as.status.Store(autoNATResult{network.ReachabilityUnknown, nil}) if currentStatus.Reachability != network.ReachabilityUnknown { if as.service != nil { - as.service.Disable() + as.service.Enable() } as.emitStatus() } @@ -362,10 +360,12 @@ func shufflePeers(peers []peer.AddrInfo) { } } +// Status returns the AutoNAT observed reachability status. func (s *StaticAutoNAT) Status() network.Reachability { return s.reachability } +// PublicAddr returns the publicly connectable Multiaddr of this node if one is known. func (s *StaticAutoNAT) PublicAddr() (ma.Multiaddr, error) { if s.reachability != network.ReachabilityPublic { return nil, errors.New("NAT status is not public") diff --git a/p2p/host/autonat/svc_test.go b/p2p/host/autonat/svc_test.go index 195a523438..e1df6c8a8c 100644 --- a/p2p/host/autonat/svc_test.go +++ b/p2p/host/autonat/svc_test.go @@ -188,8 +188,8 @@ func TestAutoNATServiceStartup(t *testing.T) { connect(t, h, hc) _, err = ac.DialBack(ctx, h.ID()) - if err == nil { - t.Fatal("autonat should not be started / advertising.") + if err != nil { + t.Fatal("autonat service be active in unknown mode.") } sub, _ := h.EventBus().Subscribe(new(event.EvtLocalReachabilityChanged)) From 1f18d0f3e293476bbfd51531e85cf6a69087181f Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 7 Apr 2020 12:34:00 -0700 Subject: [PATCH 1807/3965] Clearer naming of nat override options --- options.go | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/options.go b/options.go index 107e38db46..8efe9a834b 100644 --- a/options.go +++ b/options.go @@ -279,15 +279,20 @@ func DefaultStaticRelays() Option { } } -// WithReachability overrides automatic reachability detection to force the local node -// to believe it is either unreachable or reachable externally. -func WithReachability(reachable bool) Option { +// ForceReachabilityPublic overrides automatic reachability detection in the AutoNAT subsystem, +// forcing the local node to believe it is reachable externally. +func ForceReachabilityPublic() Option { return func(cfg *Config) error { - if reachable { - cfg.Reachability = network.ReachabilityPublic - } else { - cfg.Reachability = network.ReachabilityPrivate - } + cfg.Reachability = network.ReachabilityPublic + return nil + } +} + +// ForceReachabilityPrivate overrides automatic reachability detection in the AutoNAT subsystem, +// forceing the local node to believe it is behind a NAT and not reachable externally. +func ForceReachabilityPrivate() Option { + return func(cfg *Config) error { + cfg.Reachability = network.ReachabilityPrivate return nil } } From 70b242b685e4022a6a9be066087350d93550e141 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Thu, 9 Apr 2020 11:20:22 +0000 Subject: [PATCH 1808/3965] build(deps): bump github.com/libp2p/go-libp2p-autonat Bumps [github.com/libp2p/go-libp2p-autonat](https://github.com/libp2p/go-libp2p-autonat) from 0.2.1 to 0.2.2. - [Release notes](https://github.com/libp2p/go-libp2p-autonat/releases) - [Commits](https://github.com/libp2p/go-libp2p-autonat/compare/v0.2.1...v0.2.2) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index c77785fd44..56e640596d 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/jbenet/goprocess v0.1.4 github.com/libp2p/go-conn-security-multistream v0.1.0 github.com/libp2p/go-eventbus v0.1.0 - github.com/libp2p/go-libp2p-autonat v0.2.1 + github.com/libp2p/go-libp2p-autonat v0.2.2 github.com/libp2p/go-libp2p-blankhost v0.1.4 github.com/libp2p/go-libp2p-circuit v0.2.1 github.com/libp2p/go-libp2p-core v0.5.1 diff --git a/go.sum b/go.sum index 061ac251ef..f42c89c28d 100644 --- a/go.sum +++ b/go.sum @@ -152,11 +152,14 @@ github.com/libp2p/go-flow-metrics v0.0.3 h1:8tAs/hSdNvUiLgtlSy3mxwxWP4I9y/jlkPFT github.com/libp2p/go-flow-metrics v0.0.3/go.mod h1:HeoSNUrOJVK1jEpDqVEiUOIXqhbnS27omG0uWU5slZs= github.com/libp2p/go-libp2p v0.6.1/go.mod h1:CTFnWXogryAHjXAKEbOf1OWY+VeAP3lDMZkfEI5sT54= github.com/libp2p/go-libp2p v0.7.0/go.mod h1:hZJf8txWeCduQRDC/WSqBGMxaTHCOYHt2xSU1ivxn0k= +github.com/libp2p/go-libp2p v0.7.4/go.mod h1:oXsBlTLF1q7pxr+9w6lqzS1ILpyHsaBPniVO7zIHGMw= github.com/libp2p/go-libp2p-autonat v0.1.1/go.mod h1:OXqkeGOY2xJVWKAGV2inNF5aKN/djNA3fdpCWloIudE= github.com/libp2p/go-libp2p-autonat v0.2.0 h1:Kok+0M/4jiz6TTmxtBqAa5tLyHb/U+G/7o/JEeW7Wok= github.com/libp2p/go-libp2p-autonat v0.2.0/go.mod h1:DX+9teU4pEEoZUqR1PiMlqliONQdNbfzE1C718tcViI= github.com/libp2p/go-libp2p-autonat v0.2.1 h1:T0CRQhrvTBKfBSYw6Xo2K3ixtNpAnRCraxof3AAfgQA= github.com/libp2p/go-libp2p-autonat v0.2.1/go.mod h1:MWtAhV5Ko1l6QBsHQNSuM6b1sRkXrpk0/LqCr+vCVxI= +github.com/libp2p/go-libp2p-autonat v0.2.2 h1:4dlgcEEugTFWSvdG2UIFxhnOMpX76QaZSRAtXmYB8n4= +github.com/libp2p/go-libp2p-autonat v0.2.2/go.mod h1:HsM62HkqZmHR2k1xgX34WuWDzk/nBwNHoeyyT4IWV6A= github.com/libp2p/go-libp2p-blankhost v0.1.1/go.mod h1:pf2fvdLJPsC1FsVrNP3DUUvMzUts2dsLLBEpo1vW1ro= github.com/libp2p/go-libp2p-blankhost v0.1.4 h1:I96SWjR4rK9irDHcHq3XHN6hawCRTPUADzkJacgZLvk= github.com/libp2p/go-libp2p-blankhost v0.1.4/go.mod h1:oJF0saYsAXQCSfDq254GMNmLNz6ZTHTOvtF4ZydUvwU= From 03fa6d81e4d9e5b4ba7eda1b3d77c577dca6fd13 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 9 Apr 2020 10:59:58 -0700 Subject: [PATCH 1809/3965] doc(routing): document count Count == 0 now means that we search for an unbounded number of providers. --- core/routing/routing.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/core/routing/routing.go b/core/routing/routing.go index c7078054a6..807f968ec0 100644 --- a/core/routing/routing.go +++ b/core/routing/routing.go @@ -30,6 +30,9 @@ type ContentRouting interface { Provide(context.Context, cid.Cid, bool) error // Search for peers who are able to provide a given key + // + // When count is 0, this method will return an unbounded number of + // results. FindProvidersAsync(context.Context, cid.Cid, int) <-chan peer.AddrInfo } @@ -37,7 +40,6 @@ type ContentRouting interface { // This can be implemented by a simple lookup table, a tracking server, // or even a DHT. type PeerRouting interface { - // Find specific Peer // FindPeer searches for a peer with given ID, returns a peer.AddrInfo // with relevant addresses. FindPeer(context.Context, peer.ID) (peer.AddrInfo, error) From 97be7d6b7b5c4ea65cb497d9bf03c98e0c2a5ef2 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Fri, 10 Apr 2020 12:19:52 -0700 Subject: [PATCH 1810/3965] Expose option for setting autonat throttling --- config/config.go | 24 +++++++++++++++++------- options.go | 13 +++++++++++++ 2 files changed, 30 insertions(+), 7 deletions(-) diff --git a/config/config.go b/config/config.go index f491f92dff..9b818caec4 100644 --- a/config/config.go +++ b/config/config.go @@ -4,6 +4,7 @@ import ( "context" "crypto/rand" "fmt" + "time" "github.com/libp2p/go-libp2p-core/connmgr" "github.com/libp2p/go-libp2p-core/crypto" @@ -83,10 +84,12 @@ type Config struct { Routing RoutingC - EnableAutoRelay bool - Reachability network.Reachability - AutoNATService bool - StaticRelays []peer.AddrInfo + EnableAutoRelay bool + Reachability network.Reachability + AutoNATService bool + AutoNATThrottling bool + AutoNATThrottles [2]int + StaticRelays []peer.AddrInfo } func (cfg *Config) makeSwarm(ctx context.Context) (*swarm.Swarm, error) { @@ -280,9 +283,16 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { } } - autonatOpts := []autonat.Option{autonat.UsingAddresses(func() []ma.Multiaddr { - return addrF(h.AllAddrs()) - })} + autonatOpts := []autonat.Option{ + autonat.UsingAddresses(func() []ma.Multiaddr { + return addrF(h.AllAddrs()) + }), + } + if cfg.AutoNATThrottling { + autonatOpts = append(autonatOpts, + autonat.WithThrottling(cfg.AutoNATThrottles[0], 1*time.Minute), + autonat.WithPeerThrottling(cfg.AutoNATThrottles[1])) + } if cfg.AutoNATService { autonatPrivKey, _, err := crypto.GenerateEd25519Key(rand.Reader) if err != nil { diff --git a/options.go b/options.go index 8efe9a834b..23cf2b461f 100644 --- a/options.go +++ b/options.go @@ -307,6 +307,19 @@ func EnableNATService() Option { } } +// OverrideNATServiceThrottling changes the default rate limiting configured in helping +// other peers determine their reachability status. When set, the host will limit +// the number of requests it responds to in each 60 second period to the set +// numbers. A value of '0' disables throttling. +func OverrideNATServiceThrottling(globalThrottle, perPeerThrottle int) Option { + return func(cfg *Config) error { + cfg.AutoNATThrottling = true + cfg.AutoNATThrottles[0] = globalThrottle + cfg.AutoNATThrottles[1] = perPeerThrottle + return nil + } +} + // FilterAddresses configures libp2p to never dial nor accept connections from // the given addresses. FilterAddresses should be used for cases where the // addresses you want to deny are known ahead of time. From 5473f0ea0ed915f71b9bc2bc84856e3ef8b9deee Mon Sep 17 00:00:00 2001 From: Will Scott Date: Fri, 10 Apr 2020 13:12:19 -0700 Subject: [PATCH 1811/3965] clarify nat options --- config/config.go | 33 ++++++++++++++++++++------------- options.go | 20 +++++++++++--------- 2 files changed, 31 insertions(+), 22 deletions(-) diff --git a/config/config.go b/config/config.go index 9b818caec4..36372622b4 100644 --- a/config/config.go +++ b/config/config.go @@ -46,7 +46,15 @@ type NATManagerC func(network.Network) bhost.NATManager type RoutingC func(host.Host) (routing.PeerRouting, error) -// AutoNATMode defines the AutoNAT behavior for the libp2p host. +// autoNATConfig defines the AutoNAT behavior for the libp2p host. +type AutoNATConfig struct { + ForceReachabilityPublic bool + ForceReachabilityPrivate bool + EnableService bool + ThrottleGlobalLimit int + ThrottlePeerLimit int + ThrottleInterval time.Duration +} // Config describes a set of settings for a libp2p node // @@ -84,12 +92,9 @@ type Config struct { Routing RoutingC - EnableAutoRelay bool - Reachability network.Reachability - AutoNATService bool - AutoNATThrottling bool - AutoNATThrottles [2]int - StaticRelays []peer.AddrInfo + EnableAutoRelay bool + AutoNATConfig + StaticRelays []peer.AddrInfo } func (cfg *Config) makeSwarm(ctx context.Context) (*swarm.Swarm, error) { @@ -288,12 +293,12 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { return addrF(h.AllAddrs()) }), } - if cfg.AutoNATThrottling { + if cfg.AutoNATConfig.ThrottleInterval != 0 { autonatOpts = append(autonatOpts, - autonat.WithThrottling(cfg.AutoNATThrottles[0], 1*time.Minute), - autonat.WithPeerThrottling(cfg.AutoNATThrottles[1])) + autonat.WithThrottling(cfg.AutoNATConfig.ThrottleGlobalLimit, cfg.AutoNATConfig.ThrottleInterval), + autonat.WithPeerThrottling(cfg.AutoNATConfig.ThrottlePeerLimit)) } - if cfg.AutoNATService { + if cfg.AutoNATConfig.EnableService { autonatPrivKey, _, err := crypto.GenerateEd25519Key(rand.Reader) if err != nil { return nil, err @@ -332,8 +337,10 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { // closed (as long as we close the underlying network). autonatOpts = append(autonatOpts, autonat.EnableService(dialerHost.Network())) } - if cfg.Reachability != network.ReachabilityUnknown { - autonatOpts = append(autonatOpts, autonat.WithReachability(cfg.Reachability)) + if cfg.AutoNATConfig.ForceReachabilityPublic { + autonatOpts = append(autonatOpts, autonat.WithReachability(network.ReachabilityPublic)) + } else if cfg.AutoNATConfig.ForceReachabilityPrivate { + autonatOpts = append(autonatOpts, autonat.WithReachability(network.ReachabilityPrivate)) } if _, err = autonat.New(ctx, h, autonatOpts...); err != nil { diff --git a/options.go b/options.go index 23cf2b461f..70b7901e0b 100644 --- a/options.go +++ b/options.go @@ -6,11 +6,11 @@ package libp2p import ( "fmt" "net" + "time" "github.com/libp2p/go-libp2p-core/connmgr" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/metrics" - "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/pnet" @@ -283,7 +283,8 @@ func DefaultStaticRelays() Option { // forcing the local node to believe it is reachable externally. func ForceReachabilityPublic() Option { return func(cfg *Config) error { - cfg.Reachability = network.ReachabilityPublic + cfg.AutoNATConfig.ForceReachabilityPublic = true + cfg.AutoNATConfig.ForceReachabilityPrivate = false return nil } } @@ -292,7 +293,8 @@ func ForceReachabilityPublic() Option { // forceing the local node to believe it is behind a NAT and not reachable externally. func ForceReachabilityPrivate() Option { return func(cfg *Config) error { - cfg.Reachability = network.ReachabilityPrivate + cfg.AutoNATConfig.ForceReachabilityPrivate = true + cfg.AutoNATConfig.ForceReachabilityPublic = false return nil } } @@ -302,20 +304,20 @@ func ForceReachabilityPrivate() Option { // to peers, and then tell them if it was successful in making such connections. func EnableNATService() Option { return func(cfg *Config) error { - cfg.AutoNATService = true + cfg.AutoNATConfig.EnableService = true return nil } } -// OverrideNATServiceThrottling changes the default rate limiting configured in helping +// AutoNATServiceRateLimit changes the default rate limiting configured in helping // other peers determine their reachability status. When set, the host will limit // the number of requests it responds to in each 60 second period to the set // numbers. A value of '0' disables throttling. -func OverrideNATServiceThrottling(globalThrottle, perPeerThrottle int) Option { +func AutoNATServiceRateLimit(global, perPeer int, interval time.Duration) Option { return func(cfg *Config) error { - cfg.AutoNATThrottling = true - cfg.AutoNATThrottles[0] = globalThrottle - cfg.AutoNATThrottles[1] = perPeerThrottle + cfg.AutoNATConfig.ThrottleGlobalLimit = global + cfg.AutoNATConfig.ThrottlePeerLimit = perPeer + cfg.AutoNATConfig.ThrottleInterval = interval return nil } } From c08993b4e8521d087d05183f7e557c53bf876632 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Mon, 13 Apr 2020 10:23:22 -0700 Subject: [PATCH 1812/3965] switch bool to enum --- config/config.go | 17 +++++++---------- options.go | 9 +++++---- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/config/config.go b/config/config.go index 36372622b4..f67c117c6c 100644 --- a/config/config.go +++ b/config/config.go @@ -48,12 +48,11 @@ type RoutingC func(host.Host) (routing.PeerRouting, error) // autoNATConfig defines the AutoNAT behavior for the libp2p host. type AutoNATConfig struct { - ForceReachabilityPublic bool - ForceReachabilityPrivate bool - EnableService bool - ThrottleGlobalLimit int - ThrottlePeerLimit int - ThrottleInterval time.Duration + ForceReachability *network.Reachability + EnableService bool + ThrottleGlobalLimit int + ThrottlePeerLimit int + ThrottleInterval time.Duration } // Config describes a set of settings for a libp2p node @@ -337,10 +336,8 @@ func (cfg *Config) NewNode(ctx context.Context) (host.Host, error) { // closed (as long as we close the underlying network). autonatOpts = append(autonatOpts, autonat.EnableService(dialerHost.Network())) } - if cfg.AutoNATConfig.ForceReachabilityPublic { - autonatOpts = append(autonatOpts, autonat.WithReachability(network.ReachabilityPublic)) - } else if cfg.AutoNATConfig.ForceReachabilityPrivate { - autonatOpts = append(autonatOpts, autonat.WithReachability(network.ReachabilityPrivate)) + if cfg.AutoNATConfig.ForceReachability != nil { + autonatOpts = append(autonatOpts, autonat.WithReachability(*cfg.AutoNATConfig.ForceReachability)) } if _, err = autonat.New(ctx, h, autonatOpts...); err != nil { diff --git a/options.go b/options.go index 70b7901e0b..7dc5cdff92 100644 --- a/options.go +++ b/options.go @@ -11,6 +11,7 @@ import ( "github.com/libp2p/go-libp2p-core/connmgr" "github.com/libp2p/go-libp2p-core/crypto" "github.com/libp2p/go-libp2p-core/metrics" + "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" "github.com/libp2p/go-libp2p-core/peerstore" "github.com/libp2p/go-libp2p-core/pnet" @@ -283,8 +284,8 @@ func DefaultStaticRelays() Option { // forcing the local node to believe it is reachable externally. func ForceReachabilityPublic() Option { return func(cfg *Config) error { - cfg.AutoNATConfig.ForceReachabilityPublic = true - cfg.AutoNATConfig.ForceReachabilityPrivate = false + public := network.Reachability(network.ReachabilityPublic) + cfg.AutoNATConfig.ForceReachability = &public return nil } } @@ -293,8 +294,8 @@ func ForceReachabilityPublic() Option { // forceing the local node to believe it is behind a NAT and not reachable externally. func ForceReachabilityPrivate() Option { return func(cfg *Config) error { - cfg.AutoNATConfig.ForceReachabilityPrivate = true - cfg.AutoNATConfig.ForceReachabilityPublic = false + private := network.Reachability(network.ReachabilityPrivate) + cfg.AutoNATConfig.ForceReachability = &private return nil } } From 573c9c6e19340a16820d67001528abac4419e45f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 13 Apr 2020 10:42:57 -0700 Subject: [PATCH 1813/3965] fix: use correct reachability type --- core/network/network.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/network/network.go b/core/network/network.go index 7b61c50b9e..43fd011a17 100644 --- a/core/network/network.go +++ b/core/network/network.go @@ -59,7 +59,7 @@ type Reachability int const ( // ReachabilityUnknown indicates that the reachability status of the // node is unknown. - ReachabilityUnknown = iota + ReachabilityUnknown Reachability = iota // ReachabilityPublic indicates that the node is reachable from the // public internet. From eb76a02251aa09f6cfbb7ad39b90eb13e21ec675 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 13 Apr 2020 11:15:24 -0700 Subject: [PATCH 1814/3965] feat: support TLS by default But don't make it the default per https://github.com/libp2p/go-libp2p/pull/710#pullrequestreview-382182995. --- defaults.go | 6 +++++- go.mod | 1 + go.sum | 3 +++ 3 files changed, 9 insertions(+), 1 deletion(-) diff --git a/defaults.go b/defaults.go index aff515c5bc..6ccc71be6c 100644 --- a/defaults.go +++ b/defaults.go @@ -9,6 +9,7 @@ import ( mplex "github.com/libp2p/go-libp2p-mplex" pstoremem "github.com/libp2p/go-libp2p-peerstore/pstoremem" secio "github.com/libp2p/go-libp2p-secio" + tls "github.com/libp2p/go-libp2p-tls" yamux "github.com/libp2p/go-libp2p-yamux" tcp "github.com/libp2p/go-tcp-transport" ws "github.com/libp2p/go-ws-transport" @@ -19,7 +20,10 @@ import ( // // Useful when you want to extend, but not replace, the supported transport // security protocols. -var DefaultSecurity = Security(secio.ID, secio.New) +var DefaultSecurity = ChainOptions( + Security(secio.ID, secio.New), + Security(tls.ID, tls.New), +) // DefaultMuxers configures libp2p to use the stream connection multiplexers. // diff --git a/go.mod b/go.mod index 56e640596d..6e41899de1 100644 --- a/go.mod +++ b/go.mod @@ -23,6 +23,7 @@ require ( github.com/libp2p/go-libp2p-secio v0.2.2 github.com/libp2p/go-libp2p-swarm v0.2.3 github.com/libp2p/go-libp2p-testing v0.1.1 + github.com/libp2p/go-libp2p-tls v0.1.3 github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 github.com/libp2p/go-libp2p-yamux v0.2.7 github.com/libp2p/go-maddr-filter v0.0.5 diff --git a/go.sum b/go.sum index f42c89c28d..fea6c1b194 100644 --- a/go.sum +++ b/go.sum @@ -227,6 +227,8 @@ github.com/libp2p/go-libp2p-testing v0.0.4/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MB github.com/libp2p/go-libp2p-testing v0.1.0/go.mod h1:xaZWMJrPUM5GlDBxCeGUi7kI4eqnjVyavGroI2nxEM0= github.com/libp2p/go-libp2p-testing v0.1.1 h1:U03z3HnGI7Ni8Xx6ONVZvUFOAzWYmolWf5W5jAOPNmU= github.com/libp2p/go-libp2p-testing v0.1.1/go.mod h1:xaZWMJrPUM5GlDBxCeGUi7kI4eqnjVyavGroI2nxEM0= +github.com/libp2p/go-libp2p-tls v0.1.3 h1:twKMhMu44jQO+HgQK9X8NHO5HkeJu2QbhLzLJpa8oNM= +github.com/libp2p/go-libp2p-tls v0.1.3/go.mod h1:wZfuewxOndz5RTnCAxFliGjvYSDA40sKitV4c50uI1M= github.com/libp2p/go-libp2p-transport-upgrader v0.1.1/go.mod h1:IEtA6or8JUbsV07qPW4r01GnTenLW4oi3lOPbUMGJJA= github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 h1:5EhPgQhXZNyfL22ERZTUoVp9UVVbNowWNVtELQaKCHk= github.com/libp2p/go-libp2p-transport-upgrader v0.2.0/go.mod h1:mQcrHj4asu6ArfSoMuyojOdjx73Q47cYD7s5+gZOlns= @@ -468,6 +470,7 @@ golang.org/x/sys v0.0.0-20190228124157-a34e9553db1e/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190405154228-4b34438f7a67/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190526052359-791d8a0f4d09/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= From 81302e977af6468a4bbb8972fe0596aa7a970ee6 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Tue, 14 Apr 2020 13:37:45 -0700 Subject: [PATCH 1815/3965] remote mock addresses won't be unspecified. deprioritize those --- p2p/net/mock/mock_conn.go | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/p2p/net/mock/mock_conn.go b/p2p/net/mock/mock_conn.go index 495b2b24a4..6bde1ea2dc 100644 --- a/p2p/net/mock/mock_conn.go +++ b/p2p/net/mock/mock_conn.go @@ -9,6 +9,7 @@ import ( "github.com/libp2p/go-libp2p-core/network" "github.com/libp2p/go-libp2p-core/peer" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr-net" ) // conn represents one side's perspective of a @@ -44,7 +45,15 @@ func newConn(p process.Process, ln, rn *peernet, l *link, dir network.Direction) c.stat = network.Stat{Direction: dir} c.localAddr = ln.ps.Addrs(ln.peer)[0] - c.remoteAddr = rn.ps.Addrs(rn.peer)[0] + for _, a := range rn.ps.Addrs(rn.peer) { + if !manet.IsIPUnspecified(a) { + c.remoteAddr = a + break + } + } + if c.remoteAddr == nil { + c.remoteAddr = rn.ps.Addrs(rn.peer)[0] + } c.localPrivKey = ln.ps.PrivKey(ln.peer) c.remotePubKey = rn.ps.PubKey(rn.peer) From f6bfa824aa61c3a93b287b1c97f5173af2916554 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 25 Mar 2020 15:55:44 -0700 Subject: [PATCH 1816/3965] react to incoming events --- p2p/host/autonat/autonat.go | 85 ++++++++++++++++++++++---------- p2p/host/autonat/autonat_test.go | 24 +++++++++ p2p/host/autonat/client.go | 18 ++++--- p2p/host/autonat/notify.go | 13 ++++- p2p/host/autonat/proto.go | 1 + 5 files changed, 104 insertions(+), 37 deletions(-) diff --git a/p2p/host/autonat/autonat.go b/p2p/host/autonat/autonat.go index aa355d3308..ebf713470b 100644 --- a/p2p/host/autonat/autonat.go +++ b/p2p/host/autonat/autonat.go @@ -40,8 +40,7 @@ type AmbientAutoNAT struct { lastInbound time.Time lastProbe time.Time - subAddrUpdated event.Subscription - service *autoNATService + service *autoNATService emitReachabilityChanged event.Emitter } @@ -100,8 +99,6 @@ func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { }, nil } - subAddrUpdated, _ := h.EventBus().Subscribe(new(event.EvtLocalAddressesUpdated)) - as := &AmbientAutoNAT{ ctx: ctx, host: h, @@ -109,8 +106,6 @@ func New(ctx context.Context, h host.Host, options ...Option) (AutoNAT, error) { inboundConn: make(chan network.Conn, 5), observations: make(chan autoNATResult, 1), - subAddrUpdated: subAddrUpdated, - emitReachabilityChanged: emitReachabilityChanged, service: service, } @@ -159,17 +154,24 @@ func (as *AmbientAutoNAT) background() { delay := as.config.bootDelay var lastAddrUpdated time.Time - addrUpdatedChan := as.subAddrUpdated.Out() - defer as.subAddrUpdated.Close() + subAddrUpdated, _ := as.host.EventBus().Subscribe(new(event.EvtLocalAddressesUpdated)) + addrUpdatedChan := subAddrUpdated.Out() + defer subAddrUpdated.Close() + + subIDOccured, _ := as.host.EventBus().Subscribe(new(event.EvtPeerIdentificationCompleted)) + IDChan := subIDOccured.Out() + defer subIDOccured.Close() defer as.emitReachabilityChanged.Close() timer := time.NewTimer(delay) defer timer.Stop() timerRunning := true + var peer peer.ID for { + peer = "" select { - // new connection occured. + // new inbound connection. case conn := <-as.inboundConn: localAddrs := as.host.Addrs() ca := as.status.Load().(autoNATResult) @@ -188,6 +190,13 @@ func (as *AmbientAutoNAT) background() { } } + // peer identification occured. + case idev := <-IDChan: + peer = idev.(event.EvtPeerIdentificationCompleted).Peer + if s, err := as.host.Peerstore().SupportsProtocols(peer, AutoNATProto); len(s) == 0 || err != nil { + peer = "" + } + // probe finished. case result, ok := <-as.observations: if !ok { @@ -204,18 +213,21 @@ func (as *AmbientAutoNAT) background() { if timerRunning && !timer.Stop() { <-timer.C } - timer.Reset(as.scheduleProbe()) + timer.Reset(as.scheduleProbe(peer)) timerRunning = true } } // scheduleProbe calculates when the next probe should be scheduled for, -// and launches it if that time is now. -func (as *AmbientAutoNAT) scheduleProbe() time.Duration { +// and launches it if that time is now. peer is an an optional hint +// for prioritizing a specific peer in the probing queue. +func (as *AmbientAutoNAT) scheduleProbe(peer peer.ID) time.Duration { // Our baseline is a probe every 'AutoNATRefreshInterval' // This is modulated by: - // * recent inbound connections make us willing to wait up to 2x longer between probes. - // * low confidence makes us speed up between probes. + // * if we are in an unknown state, or have low confidence, that should drop to 'AutoNATRetryInterval' + // * recent inbound connections (implying continued connectivity) should decrease the retry when public + // * recent inbound connections when not public mean we should try more actively to see if we're public. + // * if we find out in connecting to a peer that it supports the autonat service, we should opportunistically consider probing early. fixedNow := time.Now() currentStatus := as.status.Load().(autoNATResult) @@ -228,11 +240,22 @@ func (as *AmbientAutoNAT) scheduleProbe() time.Duration { untilNext = as.config.retryInterval } else if currentStatus.Reachability == network.ReachabilityPublic && as.lastInbound.After(as.lastProbe) { untilNext *= 2 + } else if currentStatus.Reachability != network.ReachabilityPublic && as.lastInbound.After(as.lastProbe) { + untilNext /= 5 + } + if as.confidence < 3 && peer != "" { + untilNext /= 5 } + nextProbe = as.lastProbe.Add(untilNext) } if fixedNow.After(nextProbe) || fixedNow == nextProbe { - go as.probeNextPeer() + if err := peer.Validate(); err != nil { + peer = as.getPeerToProbe() + } + if peer.Validate() == nil && as.tryProbe(peer) { + as.lastProbe = fixedNow + } return as.config.retryInterval } return nextProbe.Sub(fixedNow) @@ -302,6 +325,17 @@ func (as *AmbientAutoNAT) recordObservation(observation autoNATResult) { } } +func (as *AmbientAutoNAT) tryProbe(p peer.ID) bool { + info := as.host.Peerstore().PeerInfo(p) + // TODO: reject if recently probed / on backoff + + if !as.config.dialPolicy.skipPeer(info.Addrs) { + go as.probe(&info) + return true + } + return false +} + func (as *AmbientAutoNAT) probe(pi *peer.AddrInfo) { cli := NewAutoNATClient(as.host, as.config.addressFunc) ctx, cancel := context.WithTimeout(as.ctx, as.config.requestTimeout) @@ -321,13 +355,13 @@ func (as *AmbientAutoNAT) probe(pi *peer.AddrInfo) { } } -func (as *AmbientAutoNAT) probeNextPeer() { +func (as *AmbientAutoNAT) getPeerToProbe() peer.ID { peers := as.host.Network().Peers() if len(peers) == 0 { - return + return "" } - addrs := make([]peer.AddrInfo, 0, len(peers)) + candidates := make([]peer.ID, 0, len(peers)) for _, p := range peers { info := as.host.Peerstore().PeerInfo(p) @@ -337,23 +371,20 @@ func (as *AmbientAutoNAT) probeNextPeer() { } if !as.config.dialPolicy.skipPeer(info.Addrs) { - addrs = append(addrs, info) + candidates = append(candidates, p) } - addrs = append(addrs, info) } // TODO: track and exclude recently probed peers. - if len(addrs) == 0 { - return + if len(candidates) == 0 { + return "" } - shufflePeers(addrs) - - as.lastProbe = time.Now() - as.probe(&addrs[0]) + shufflePeers(candidates) + return candidates[0] } -func shufflePeers(peers []peer.AddrInfo) { +func shufflePeers(peers []peer.ID) { for i := range peers { j := rand.Intn(i + 1) peers[i], peers[j] = peers[j], peers[i] diff --git a/p2p/host/autonat/autonat_test.go b/p2p/host/autonat/autonat_test.go index 7e45862a2d..a9629b1bb2 100644 --- a/p2p/host/autonat/autonat_test.go +++ b/p2p/host/autonat/autonat_test.go @@ -176,6 +176,30 @@ func TestAutoNATPublictoPrivate(t *testing.T) { } } +func TestAutoNATIncomingEvents(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + hs := makeAutoNATServicePrivate(ctx, t) + hc, ani := makeAutoNAT(ctx, t, hs) + an := ani.(*AmbientAutoNAT) + + status := an.Status() + if status != network.ReachabilityUnknown { + t.Fatalf("unexpected NAT status: %d", status) + } + + connect(t, hs, hc) + + em, _ := hc.EventBus().Emitter(&event.EvtPeerIdentificationCompleted{}) + em.Emit(event.EvtPeerIdentificationCompleted{Peer: hs.ID()}) + + time.Sleep(10 * time.Millisecond) + if an.Status() == network.ReachabilityUnknown { + t.Fatalf("Expected probe due to identification of autonat service") + } +} + func TestAutoNATObservationRecording(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() diff --git a/p2p/host/autonat/client.go b/p2p/host/autonat/client.go index b86f17a874..e2ab8a3df2 100644 --- a/p2p/host/autonat/client.go +++ b/p2p/host/autonat/client.go @@ -14,8 +14,8 @@ import ( ma "github.com/multiformats/go-multiaddr" ) -// AutoNATError is the class of errors signalled by AutoNAT services -type AutoNATError struct { +// Error wraps errors signalled by AutoNAT services +type Error struct { Status pb.Message_ResponseStatus Text string } @@ -71,30 +71,32 @@ func (c *client) DialBack(ctx context.Context, p peer.ID) (ma.Multiaddr, error) return ma.NewMultiaddrBytes(addr) default: - return nil, AutoNATError{Status: status, Text: res.GetDialResponse().GetStatusText()} + return nil, Error{Status: status, Text: res.GetDialResponse().GetStatusText()} } } -func (e AutoNATError) Error() string { +func (e Error) Error() string { return fmt.Sprintf("AutoNAT error: %s (%s)", e.Text, e.Status.String()) } -func (e AutoNATError) IsDialError() bool { +// IsDialError returns true if the error was due to a dial back failure +func (e Error) IsDialError() bool { return e.Status == pb.Message_E_DIAL_ERROR } -func (e AutoNATError) IsDialRefused() bool { +// IsDialRefused returns true if the error was due to a refusal to dial back +func (e Error) IsDialRefused() bool { return e.Status == pb.Message_E_DIAL_REFUSED } // IsDialError returns true if the AutoNAT peer signalled an error dialing back func IsDialError(e error) bool { - ae, ok := e.(AutoNATError) + ae, ok := e.(Error) return ok && ae.IsDialError() } // IsDialRefused returns true if the AutoNAT peer signalled refusal to dial back func IsDialRefused(e error) bool { - ae, ok := e.(AutoNATError) + ae, ok := e.(Error) return ok && ae.IsDialRefused() } diff --git a/p2p/host/autonat/notify.go b/p2p/host/autonat/notify.go index a444df62db..44c80fd425 100644 --- a/p2p/host/autonat/notify.go +++ b/p2p/host/autonat/notify.go @@ -9,11 +9,19 @@ import ( var _ network.Notifiee = (*AmbientAutoNAT)(nil) -func (as *AmbientAutoNAT) Listen(net network.Network, a ma.Multiaddr) {} -func (as *AmbientAutoNAT) ListenClose(net network.Network, a ma.Multiaddr) {} +// Listen is part of the network.Notifiee interface +func (as *AmbientAutoNAT) Listen(net network.Network, a ma.Multiaddr) {} + +// ListenClose is part of the network.Notifiee interface +func (as *AmbientAutoNAT) ListenClose(net network.Network, a ma.Multiaddr) {} + +// OpenedStream is part of the network.Notifiee interface func (as *AmbientAutoNAT) OpenedStream(net network.Network, s network.Stream) {} + +// ClosedStream is part of the network.Notifiee interface func (as *AmbientAutoNAT) ClosedStream(net network.Network, s network.Stream) {} +// Connected is part of the network.Notifiee interface func (as *AmbientAutoNAT) Connected(net network.Network, c network.Conn) { if c.Stat().Direction == network.DirInbound && manet.IsPublicAddr(c.RemoteMultiaddr()) { @@ -24,4 +32,5 @@ func (as *AmbientAutoNAT) Connected(net network.Network, c network.Conn) { } } +// Disconnected is part of the network.Notifiee interface func (as *AmbientAutoNAT) Disconnected(net network.Network, c network.Conn) {} diff --git a/p2p/host/autonat/proto.go b/p2p/host/autonat/proto.go index cf5be81ad9..b29a53c3fa 100644 --- a/p2p/host/autonat/proto.go +++ b/p2p/host/autonat/proto.go @@ -8,6 +8,7 @@ import ( ma "github.com/multiformats/go-multiaddr" ) +// AutoNATProto identifies the autonat service protocol const AutoNATProto = "/libp2p/autonat/1.0.0" func newDialMessage(pi peer.AddrInfo) *pb.Message { From 92a3a76d88fc9e10af809afe7322581d4b81be96 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Wed, 15 Apr 2020 19:43:28 -0700 Subject: [PATCH 1817/3965] clarify dialPolicy comment --- p2p/host/autonat/dialpolicy.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/p2p/host/autonat/dialpolicy.go b/p2p/host/autonat/dialpolicy.go index 4c35680f34..a9f47e92db 100644 --- a/p2p/host/autonat/dialpolicy.go +++ b/p2p/host/autonat/dialpolicy.go @@ -52,10 +52,9 @@ func (d *dialPolicy) skipDial(addr ma.Multiaddr) bool { } // skipPeer indicates that the collection of multiaddresses representing a peer -// isn't worth attempted dialing. Addresses are dialed individually, and while -// individual addresses for a peer may be worth considering, there are some -// factors, like the presence of the same public address as the local host, -// that may make the peer undesirable to dial as a whole. +// isn't worth attempted dialing. If one of the addresses matches an address +// we believe is ours, we exclude the peer, even if there are other valid +// public addresses in the list. func (d *dialPolicy) skipPeer(addrs []ma.Multiaddr) bool { localAddrs := d.host.Addrs() localHosts := make([]net.IP, 0) From 31d6fba8368ed7ebc8db8146916a1f38264ac3ab Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Thu, 16 Apr 2020 12:19:29 +0700 Subject: [PATCH 1818/3965] write qlogs to a temporary file first, then rename them when done --- p2p/transport/quic/buffered_write_closer.go | 25 ------ .../quic/buffered_write_closer_test.go | 26 ------ p2p/transport/quic/qlog.go | 83 +++++++++++++++++++ p2p/transport/quic/qlog_test.go | 83 +++++++++++++++++++ p2p/transport/quic/transport.go | 32 +------ 5 files changed, 168 insertions(+), 81 deletions(-) delete mode 100644 p2p/transport/quic/buffered_write_closer.go delete mode 100644 p2p/transport/quic/buffered_write_closer_test.go create mode 100644 p2p/transport/quic/qlog.go create mode 100644 p2p/transport/quic/qlog_test.go diff --git a/p2p/transport/quic/buffered_write_closer.go b/p2p/transport/quic/buffered_write_closer.go deleted file mode 100644 index aeeef0035a..0000000000 --- a/p2p/transport/quic/buffered_write_closer.go +++ /dev/null @@ -1,25 +0,0 @@ -package libp2pquic - -import ( - "bufio" - "io" -) - -type bufferedWriteCloser struct { - *bufio.Writer - io.Closer -} - -func newBufferedWriteCloser(writer *bufio.Writer, closer io.Closer) io.WriteCloser { - return &bufferedWriteCloser{ - Writer: writer, - Closer: closer, - } -} - -func (h bufferedWriteCloser) Close() error { - if err := h.Writer.Flush(); err != nil { - return err - } - return h.Closer.Close() -} diff --git a/p2p/transport/quic/buffered_write_closer_test.go b/p2p/transport/quic/buffered_write_closer_test.go deleted file mode 100644 index 949e211c22..0000000000 --- a/p2p/transport/quic/buffered_write_closer_test.go +++ /dev/null @@ -1,26 +0,0 @@ -package libp2pquic - -import ( - "bufio" - "bytes" - - . "github.com/onsi/ginkgo" - . "github.com/onsi/gomega" -) - -type nopCloser struct{} - -func (nopCloser) Close() error { return nil } - -var _ = Describe("buffered io.WriteCloser", func() { - It("flushes before closing", func() { - buf := &bytes.Buffer{} - - w := bufio.NewWriter(buf) - wc := newBufferedWriteCloser(w, &nopCloser{}) - wc.Write([]byte("foobar")) - Expect(buf.Len()).To(BeZero()) - Expect(wc.Close()).To(Succeed()) - Expect(buf.String()).To(Equal("foobar")) - }) -}) diff --git a/p2p/transport/quic/qlog.go b/p2p/transport/quic/qlog.go new file mode 100644 index 0000000000..73c9a4e6e4 --- /dev/null +++ b/p2p/transport/quic/qlog.go @@ -0,0 +1,83 @@ +package libp2pquic + +import ( + "bufio" + "compress/gzip" + "fmt" + "io" + "os" + "time" +) + +var qlogDir string + +func init() { + qlogDir = os.Getenv("QLOGDIR") +} + +func getLogWriterFor(role string) func([]byte) io.WriteCloser { + if len(qlogDir) == 0 { + return nil + } + return func(connID []byte) io.WriteCloser { + // create the QLOGDIR, if it doesn't exist + if err := os.MkdirAll(qlogDir, 0777); err != nil { + log.Errorf("creating the QLOGDIR failed: %s", err) + return nil + } + return newQlogger(role, connID) + } +} + +type qlogger struct { + f *os.File // QLOGDIR/.log_xxx.qlog.gz.swp + filename string // QLOGDIR/log_xxx.qlog.gz + io.WriteCloser +} + +func newQlogger(role string, connID []byte) io.WriteCloser { + t := time.Now().UTC().Format("2006-01-02T15-04-05.999999999UTC") + finalFilename := fmt.Sprintf("%s%clog_%s_%s_%x.qlog.gz", qlogDir, os.PathSeparator, t, role, connID) + filename := fmt.Sprintf("%s%c.log_%s_%s_%x.qlog.gz.swp", qlogDir, os.PathSeparator, t, role, connID) + f, err := os.Create(filename) + if err != nil { + log.Errorf("unable to create qlog file %s: %s", filename, err) + return nil + } + gz := gzip.NewWriter(f) + return &qlogger{ + f: f, + filename: finalFilename, + WriteCloser: newBufferedWriteCloser(bufio.NewWriter(gz), gz), + } +} + +func (l *qlogger) Close() error { + if err := l.WriteCloser.Close(); err != nil { + return err + } + path := l.f.Name() + if err := l.f.Close(); err != nil { + return err + } + return os.Rename(path, l.filename) +} + +type bufferedWriteCloser struct { + *bufio.Writer + io.Closer +} + +func newBufferedWriteCloser(writer *bufio.Writer, closer io.Closer) io.WriteCloser { + return &bufferedWriteCloser{ + Writer: writer, + Closer: closer, + } +} + +func (h bufferedWriteCloser) Close() error { + if err := h.Writer.Flush(); err != nil { + return err + } + return h.Closer.Close() +} diff --git a/p2p/transport/quic/qlog_test.go b/p2p/transport/quic/qlog_test.go new file mode 100644 index 0000000000..dc28d50bd8 --- /dev/null +++ b/p2p/transport/quic/qlog_test.go @@ -0,0 +1,83 @@ +package libp2pquic + +import ( + "bytes" + "compress/gzip" + "fmt" + "io/ioutil" + "os" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" +) + +type nopCloser struct{} + +func (nopCloser) Close() error { return nil } + +var _ = Describe("qlogger", func() { + var origQlogDir string + + BeforeEach(func() { + origQlogDir = qlogDir + d, err := ioutil.TempDir("", "libp2p-quic-transport-test") + Expect(err).ToNot(HaveOccurred()) + fmt.Fprintf(GinkgoWriter, "Creating temporary directory: %s\n", d) + qlogDir = d + }) + + AfterEach(func() { + qlogDir = origQlogDir + }) + + getFile := func() os.FileInfo { + files, err := ioutil.ReadDir(qlogDir) + Expect(err).ToNot(HaveOccurred()) + Expect(files).To(HaveLen(1)) + return files[0] + } + + It("saves a qlog", func() { + logger := newQlogger("server", []byte{0xde, 0xad, 0xbe, 0xef}) + file := getFile() + Expect(string(file.Name()[0])).To(Equal(".")) + Expect(file.Name()).To(HaveSuffix(".qlog.gz.swp")) + // close the logger. This should move the file. + Expect(logger.Close()).To(Succeed()) + file = getFile() + Expect(string(file.Name()[0])).ToNot(Equal(".")) + Expect(file.Name()).To(HaveSuffix(".qlog.gz")) + Expect(file.Name()).To(And( + ContainSubstring("server"), + ContainSubstring("deadbeef"), + )) + }) + + It("buffers", func() { + logger := newQlogger("server", []byte("connid")) + initialSize := getFile().Size() + // Do a small write. + // Since the writter is buffered, this should not be written to disk yet. + logger.Write([]byte("foobar")) + Expect(getFile().Size()).To(Equal(initialSize)) + // Close the logger. This should flush the buffer to disk. + Expect(logger.Close()).To(Succeed()) + finalSize := getFile().Size() + fmt.Fprintf(GinkgoWriter, "initial log file size: %d, final log file size: %d\n", initialSize, finalSize) + Expect(finalSize).To(BeNumerically(">", initialSize)) + }) + + It("compresses", func() { + logger := newQlogger("server", []byte("connid")) + logger.Write([]byte("foobar")) + Expect(logger.Close()).To(Succeed()) + compressed, err := ioutil.ReadFile(qlogDir + "/" + getFile().Name()) + Expect(err).ToNot(HaveOccurred()) + Expect(compressed).ToNot(Equal("foobar")) + gz, err := gzip.NewReader(bytes.NewReader(compressed)) + Expect(err).ToNot(HaveOccurred()) + data, err := ioutil.ReadAll(gz) + Expect(err).ToNot(HaveOccurred()) + Expect(data).To(Equal([]byte("foobar"))) + }) +}) diff --git a/p2p/transport/quic/transport.go b/p2p/transport/quic/transport.go index 97add03c07..d71b366d7f 100644 --- a/p2p/transport/quic/transport.go +++ b/p2p/transport/quic/transport.go @@ -1,15 +1,10 @@ package libp2pquic import ( - "bufio" - "compress/gzip" "context" "errors" - "fmt" "io" "net" - "os" - "time" "github.com/minio/sha256-simd" "golang.org/x/crypto/hkdf" @@ -134,8 +129,8 @@ func NewTransport(key ic.PrivKey, psk pnet.PSK, filters *filter.Filters) (tpt.Tr serverConfig: config, clientConfig: config.Clone(), } - t.serverConfig.GetLogWriter = t.GetLogWriterFor("server") - t.clientConfig.GetLogWriter = t.GetLogWriterFor("client") + t.serverConfig.GetLogWriter = getLogWriterFor("server") + t.clientConfig.GetLogWriter = getLogWriterFor("client") return t, nil } @@ -195,29 +190,6 @@ func (t *transport) Dial(ctx context.Context, raddr ma.Multiaddr, p peer.ID) (tp }, nil } -func (t *transport) GetLogWriterFor(role string) func([]byte) io.WriteCloser { - qlogDir := os.Getenv("QLOGDIR") - if len(qlogDir) == 0 { - return nil - } - return func(connID []byte) io.WriteCloser { - // create the QLOGDIR, if it doesn't exist - if err := os.MkdirAll(qlogDir, 0777); err != nil { - log.Errorf("creating the QLOGDIR failed: %s", err) - return nil - } - t := time.Now().Format(time.RFC3339Nano) - filename := fmt.Sprintf("%s/log_%s_%s_%x.qlog.gz", qlogDir, t, role, connID) - f, err := os.Create(filename) - if err != nil { - log.Errorf("unable to create qlog file %s: %s", filename, err) - return nil - } - gz := gzip.NewWriter(f) - return newBufferedWriteCloser(bufio.NewWriter(gz), gz) - } -} - // Don't use mafmt.QUIC as we don't want to dial DNS addresses. Just /ip{4,6}/udp/quic var dialMatcher = mafmt.And(mafmt.IP, mafmt.Base(ma.P_UDP), mafmt.Base(ma.P_QUIC)) From 28f45ac904e20307c8e6f1cc0b08990f432ea1f2 Mon Sep 17 00:00:00 2001 From: Edgar Aroutiounian Date: Thu, 16 Apr 2020 19:25:48 -0700 Subject: [PATCH 1819/3965] [discovery] missing defer .Stop on ticker --- p2p/discovery/mdns.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/p2p/discovery/mdns.go b/p2p/discovery/mdns.go index 2da6618910..bd46438f61 100644 --- a/p2p/discovery/mdns.go +++ b/p2p/discovery/mdns.go @@ -119,8 +119,9 @@ func (m *mdnsService) Close() error { } func (m *mdnsService) pollForEntries(ctx context.Context) { - ticker := time.NewTicker(m.interval) + defer ticker.Stop() + for { //execute mdns query right away at method call and then with every tick entriesCh := make(chan *mdns.ServiceEntry, 16) From e13a3fe3a31bb0fefb56a687d18405ee31d9c5c0 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Fri, 17 Apr 2020 11:31:17 -0700 Subject: [PATCH 1820/3965] fix: reduce log level of a noisy log line This will trigger whenever we're, e.g., offline. --- p2p/discovery/mdns.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/p2p/discovery/mdns.go b/p2p/discovery/mdns.go index 2da6618910..a3d890df3e 100644 --- a/p2p/discovery/mdns.go +++ b/p2p/discovery/mdns.go @@ -140,7 +140,7 @@ func (m *mdnsService) pollForEntries(ctx context.Context) { err := mdns.Query(qp) if err != nil { - log.Error("mdns lookup error: ", err) + log.Warnw("mdns lookup error", "error", err) } close(entriesCh) log.Debug("mdns query complete") From 9b04f0ff285b6fb1747b43ba025481e5a97c8f18 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 20 Apr 2020 17:16:28 +0700 Subject: [PATCH 1821/3965] add command line client and server --- p2p/transport/quic/cmd/client/main.go | 69 +++++++++++++++++++++++ p2p/transport/quic/cmd/server/main.go | 79 +++++++++++++++++++++++++++ 2 files changed, 148 insertions(+) create mode 100644 p2p/transport/quic/cmd/client/main.go create mode 100644 p2p/transport/quic/cmd/server/main.go diff --git a/p2p/transport/quic/cmd/client/main.go b/p2p/transport/quic/cmd/client/main.go new file mode 100644 index 0000000000..8d391c9786 --- /dev/null +++ b/p2p/transport/quic/cmd/client/main.go @@ -0,0 +1,69 @@ +package main + +import ( + "context" + "crypto/rand" + "fmt" + "io/ioutil" + "log" + "os" + + ic "github.com/libp2p/go-libp2p-core/crypto" + peer "github.com/libp2p/go-libp2p-core/peer" + libp2pquic "github.com/libp2p/go-libp2p-quic-transport" + ma "github.com/multiformats/go-multiaddr" +) + +func main() { + if len(os.Args) != 3 { + fmt.Printf("Usage: ./main ") + return + } + if err := run(os.Args[1], os.Args[2]); err != nil { + log.Fatalf(err.Error()) + } +} + +func run(raddr string, p string) error { + peerID, err := peer.Decode(p) + if err != nil { + return err + } + addr, err := ma.NewMultiaddr(raddr) + if err != nil { + return err + } + priv, _, err := ic.GenerateECDSAKeyPair(rand.Reader) + if err != nil { + return err + } + + t, err := libp2pquic.NewTransport(priv, nil, nil) + if err != nil { + return err + } + + log.Printf("Dialing %s\n", addr.String()) + conn, err := t.Dial(context.Background(), addr, peerID) + if err != nil { + return err + } + str, err := conn.OpenStream() + if err != nil { + return err + } + const msg = "Hello world!" + log.Printf("Sending: %s\n", msg) + if _, err := str.Write([]byte(msg)); err != nil { + return err + } + if err := str.Close(); err != nil { + return err + } + data, err := ioutil.ReadAll(str) + if err != nil { + return err + } + log.Printf("Received: %s\n", data) + return conn.Close() +} diff --git a/p2p/transport/quic/cmd/server/main.go b/p2p/transport/quic/cmd/server/main.go new file mode 100644 index 0000000000..68eda3de6a --- /dev/null +++ b/p2p/transport/quic/cmd/server/main.go @@ -0,0 +1,79 @@ +package main + +import ( + "crypto/rand" + "fmt" + "io/ioutil" + "log" + "os" + + ic "github.com/libp2p/go-libp2p-core/crypto" + "github.com/libp2p/go-libp2p-core/peer" + tpt "github.com/libp2p/go-libp2p-core/transport" + libp2pquic "github.com/libp2p/go-libp2p-quic-transport" + ma "github.com/multiformats/go-multiaddr" +) + +func main() { + if len(os.Args) != 2 { + fmt.Printf("Usage: ./main ") + return + } + if err := run(os.Args[1]); err != nil { + log.Fatalf(err.Error()) + } +} + +func run(port string) error { + addr, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/0.0.0.0/udp/%s/quic", port)) + if err != nil { + return err + } + priv, _, err := ic.GenerateECDSAKeyPair(rand.Reader) + if err != nil { + return err + } + peerID, err := peer.IDFromPrivateKey(priv) + if err != nil { + return err + } + + t, err := libp2pquic.NewTransport(priv, nil, nil) + if err != nil { + return err + } + + ln, err := t.Listen(addr) + if err != nil { + return err + } + fmt.Printf("Listening. Now run: go run cmd/client/main.go %s %s\n", ln.Multiaddr(), peerID) + for { + conn, err := ln.Accept() + if err != nil { + return err + } + log.Printf("Accepted new connection from %s (%s)\n", conn.RemotePeer(), conn.RemoteMultiaddr()) + go func() { + if err := handleConn(conn); err != nil { + log.Printf("handling conn failed: %s", err.Error()) + } + }() + } +} + +func handleConn(conn tpt.CapableConn) error { + str, err := conn.AcceptStream() + if err != nil { + return err + } + data, err := ioutil.ReadAll(str) + if err != nil { + return err + } + log.Printf("Received: %s\n", data) + if _, err := str.Write([]byte(data)); err != nil { + return err + } + return str.Close() +} From 82c90ce038b3b90d45d950023fccefdb45b219c3 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 21 Apr 2020 10:20:35 +0700 Subject: [PATCH 1822/3965] apply @Stebalien's suggestions from code review Co-Authored-By: Steven Allen --- p2p/transport/quic/cmd/client/main.go | 2 +- p2p/transport/quic/cmd/server/main.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/p2p/transport/quic/cmd/client/main.go b/p2p/transport/quic/cmd/client/main.go index 8d391c9786..37ede0c338 100644 --- a/p2p/transport/quic/cmd/client/main.go +++ b/p2p/transport/quic/cmd/client/main.go @@ -16,7 +16,7 @@ import ( func main() { if len(os.Args) != 3 { - fmt.Printf("Usage: ./main ") + fmt.Printf("Usage: %s ", os.Args[0]) return } if err := run(os.Args[1], os.Args[2]); err != nil { diff --git a/p2p/transport/quic/cmd/server/main.go b/p2p/transport/quic/cmd/server/main.go index 68eda3de6a..3a5fe24281 100644 --- a/p2p/transport/quic/cmd/server/main.go +++ b/p2p/transport/quic/cmd/server/main.go @@ -16,7 +16,7 @@ import ( func main() { if len(os.Args) != 2 { - fmt.Printf("Usage: ./main ") + fmt.Printf("Usage: %s ", os.Args[0]) return } if err := run(os.Args[1]); err != nil { From 4fa1ab45d6e809cd56d2e5bf1fb5873310067b6f Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Tue, 21 Apr 2020 14:46:32 +0700 Subject: [PATCH 1823/3965] improve the error message returned when peer verification fails --- p2p/security/tls/crypto.go | 6 +++++- p2p/security/tls/transport_test.go | 3 ++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/p2p/security/tls/crypto.go b/p2p/security/tls/crypto.go index 14e1db02b5..e6d6d5fdab 100644 --- a/p2p/security/tls/crypto.go +++ b/p2p/security/tls/crypto.go @@ -93,7 +93,11 @@ func (i *Identity) ConfigForPeer(remote peer.ID) (*tls.Config, <-chan ic.PubKey) return err } if remote != "" && !remote.MatchesPublicKey(pubKey) { - return errors.New("peer IDs don't match") + peerID, err := peer.IDFromPublicKey(pubKey) + if err != nil { + peerID = peer.ID(fmt.Sprintf("(not determined: %s)", err.Error())) + } + return fmt.Errorf("peer IDs don't match: expected %s, got %s", remote, peerID) } keyCh <- pubKey return nil diff --git a/p2p/security/tls/transport_test.go b/p2p/security/tls/transport_test.go index 901c85e1ac..2b5be8f2a9 100644 --- a/p2p/security/tls/transport_test.go +++ b/p2p/security/tls/transport_test.go @@ -184,7 +184,8 @@ var _ = Describe("Transport", func() { }() // dial, but expect the wrong peer ID _, err = clientTransport.SecureOutbound(context.Background(), clientInsecureConn, thirdPartyID) - Expect(err).To(MatchError("peer IDs don't match")) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("peer IDs don't match")) Eventually(done).Should(BeClosed()) }) From f650f4b3dfde3620b243832dd9723fe83d07d217 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Tue, 21 Apr 2020 17:21:30 +0000 Subject: [PATCH 1824/3965] build(deps): bump github.com/ipfs/go-log from 1.0.3 to 1.0.4 Bumps [github.com/ipfs/go-log](https://github.com/ipfs/go-log) from 1.0.3 to 1.0.4. - [Release notes](https://github.com/ipfs/go-log/releases) - [Commits](https://github.com/ipfs/go-log/compare/v1.0.3...v1.0.4) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 6e41899de1..f4fec05e90 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ require ( github.com/ipfs/go-cid v0.0.5 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ipfs-util v0.0.1 - github.com/ipfs/go-log v1.0.3 + github.com/ipfs/go-log v1.0.4 github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.4 github.com/libp2p/go-conn-security-multistream v0.1.0 diff --git a/go.sum b/go.sum index fea6c1b194..8192a5cc90 100644 --- a/go.sum +++ b/go.sum @@ -58,6 +58,7 @@ github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/gopacket v1.1.17 h1:rMrlX2ZY2UbvT+sdz3+6J+pp2z+msCq9MxTU6ymxbBY= github.com/google/gopacket v1.1.17/go.mod h1:UdDNZ1OO62aGYVnPhxT1U6aI7ukYtA/kB8vaU0diBUM= +github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/websocket v1.4.1 h1:q7AeDBpnBk8AogcD4DSag/Ukw/KV+YhzLj2bP5HvKCM= @@ -102,10 +103,14 @@ github.com/ipfs/go-log v1.0.2 h1:s19ZwJxH8rPWzypjcDpqPLIyV7BnbLqvpli3iZoqYK0= github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= github.com/ipfs/go-log v1.0.3 h1:Gg7SUYSZ7BrqaKMwM+hRgcAkKv4QLfzP4XPQt5Sx/OI= github.com/ipfs/go-log v1.0.3/go.mod h1:OsLySYkwIbiSUR/yBTdv1qPtcE4FW3WPWk/ewz9Ru+A= +github.com/ipfs/go-log v1.0.4 h1:6nLQdX4W8P9yZZFH7mO+X/PzjN8Laozm/lMJ6esdgzY= +github.com/ipfs/go-log v1.0.4/go.mod h1:oDCg2FkjogeFOhqqb+N39l2RpTNPL6F/StPkB3kPgcs= github.com/ipfs/go-log/v2 v2.0.2 h1:xguurydRdfKMJjKyxNXNU8lYP0VZH1NUwJRwUorjuEw= github.com/ipfs/go-log/v2 v2.0.2/go.mod h1:O7P1lJt27vWHhOwQmcFEvlmo49ry2VY2+JfBWFaa9+0= github.com/ipfs/go-log/v2 v2.0.3 h1:Q2gXcBoCALyLN/pUQlz1qgu0x3uFV6FzP9oXhpfyJpc= github.com/ipfs/go-log/v2 v2.0.3/go.mod h1:O7P1lJt27vWHhOwQmcFEvlmo49ry2VY2+JfBWFaa9+0= +github.com/ipfs/go-log/v2 v2.0.5 h1:fL4YI+1g5V/b1Yxr1qAiXTMg1H8z9vx/VmJxBuQMHvU= +github.com/ipfs/go-log/v2 v2.0.5/go.mod h1:eZs4Xt4ZUJQFM3DlanGhy7TkwwawCZcSByscwkWG+dw= github.com/jackpal/gateway v1.0.5 h1:qzXWUJfuMdlLMtt0a3Dgt+xkWQiA5itDEITVJtuSwMc= github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1 h1:i0LektDkO1QlrTm/cSuP+PyBCDnYvjPLGl4LdWEMiaA= @@ -371,6 +376,7 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= github.com/smola/gocompat v0.2.0 h1:6b1oIMlUXIpz//VKEDzPVBK8KG7beVwmHIUEBIs/Pns= @@ -416,12 +422,19 @@ go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8= go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.uber.org/atomic v1.4.0 h1:cxzIVoETapQEqDhQu3QfnvXAV4AlzcvUCxkVUFw3+EU= go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= +go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk= +go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= go.uber.org/goleak v1.0.0 h1:qsup4IcBdlmsnGfqyLl4Ntn3C2XCCuKAE7DwHpScyUo= go.uber.org/goleak v1.0.0/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A= go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI= go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= +go.uber.org/multierr v1.5.0 h1:KCa4XfM8CWFCpxXRGok+Q0SS/0XBhMDbHHGABQLvD2A= +go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= +go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM= go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= +go.uber.org/zap v1.14.1 h1:nYDKopTbvAPq/NrUVZwT15y2lpROBiLLyoRTbXOYWOo= +go.uber.org/zap v1.14.1/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= @@ -429,6 +442,7 @@ golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnf golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= @@ -441,6 +455,7 @@ golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvx golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs= golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -487,6 +502,9 @@ golang.org/x/tools v0.0.0-20181130052023-1c3d964395ce/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191108193012-7d206e10da11/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191216052735-49a3e744a425 h1:VvQyQJN0tSuecqgcIxMWnnfG5kSmgy9KZR9sW3W5QeA= golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= @@ -505,6 +523,7 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d/go.mod h1:z+K8VcOYVYcSwSjGebuDL6176A1XskgbtNl64NSg+n8= @@ -516,3 +535,4 @@ gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I= gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= From ee27334a62c9166ad2b673f120f7de22715fec36 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Tue, 21 Apr 2020 23:17:45 -0700 Subject: [PATCH 1825/3965] fix: add read/write locks We keep running into races so I figured we might as well just make this all thread-safe. It should have no effect on performance. The actual _bug_ was setting the write deadline while writing. But we might as well just have read and write locks. fixes https://github.com/libp2p/go-libp2p-swarm/issues/205 --- p2p/transport/websocket/conn_native.go | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/p2p/transport/websocket/conn_native.go b/p2p/transport/websocket/conn_native.go index 6be4697ec6..7a0daeacaf 100644 --- a/p2p/transport/websocket/conn_native.go +++ b/p2p/transport/websocket/conn_native.go @@ -17,9 +17,14 @@ type Conn struct { DefaultMessageType int reader io.Reader closeOnce sync.Once + + readLock, writeLock sync.Mutex } func (c *Conn) Read(b []byte) (int, error) { + c.readLock.Lock() + defer c.readLock.Unlock() + if c.reader == nil { if err := c.prepNextReader(); err != nil { return 0, err @@ -67,6 +72,9 @@ func (c *Conn) prepNextReader() error { } func (c *Conn) Write(b []byte) (n int, err error) { + c.writeLock.Lock() + defer c.writeLock.Unlock() + if err := c.Conn.WriteMessage(c.DefaultMessageType, b); err != nil { return 0, err } @@ -113,10 +121,18 @@ func (c *Conn) SetDeadline(t time.Time) error { } func (c *Conn) SetReadDeadline(t time.Time) error { + // Don't lock when setting the read deadline. That would prevent us from + // interrupting an in-progress read. return c.Conn.SetReadDeadline(t) } func (c *Conn) SetWriteDeadline(t time.Time) error { + // Unlike the read deadline, we need to lock when setting the write + // deadline. + + c.writeLock.Lock() + defer c.writeLock.Unlock() + return c.Conn.SetWriteDeadline(t) } From dde9d4747387761f97c755e1c107caf62812f9f2 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Wed, 22 Apr 2020 11:20:36 +0000 Subject: [PATCH 1826/3965] build(deps): bump github.com/libp2p/go-conn-security-multistream Bumps [github.com/libp2p/go-conn-security-multistream](https://github.com/libp2p/go-conn-security-multistream) from 0.1.0 to 0.2.0. - [Release notes](https://github.com/libp2p/go-conn-security-multistream/releases) - [Commits](https://github.com/libp2p/go-conn-security-multistream/compare/v0.1.0...v0.2.0) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index f4fec05e90..1da207343e 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,7 @@ require ( github.com/ipfs/go-log v1.0.4 github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/goprocess v0.1.4 - github.com/libp2p/go-conn-security-multistream v0.1.0 + github.com/libp2p/go-conn-security-multistream v0.2.0 github.com/libp2p/go-eventbus v0.1.0 github.com/libp2p/go-libp2p-autonat v0.2.2 github.com/libp2p/go-libp2p-blankhost v0.1.4 diff --git a/go.sum b/go.sum index 8192a5cc90..ae8606a200 100644 --- a/go.sum +++ b/go.sum @@ -150,6 +150,8 @@ github.com/libp2p/go-buffer-pool v0.0.2 h1:QNK2iAFa8gjAe1SPz6mHSMuCcjs+X1wlHzeOS github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM= github.com/libp2p/go-conn-security-multistream v0.1.0 h1:aqGmto+ttL/uJgX0JtQI0tD21CIEy5eYd1Hlp0juHY0= github.com/libp2p/go-conn-security-multistream v0.1.0/go.mod h1:aw6eD7LOsHEX7+2hJkDxw1MteijaVcI+/eP2/x3J1xc= +github.com/libp2p/go-conn-security-multistream v0.2.0 h1:uNiDjS58vrvJTg9jO6bySd1rMKejieG7v45ekqHbZ1M= +github.com/libp2p/go-conn-security-multistream v0.2.0/go.mod h1:hZN4MjlNetKD3Rq5Jb/P5ohUnFLNzEAR4DLSzpn2QLU= github.com/libp2p/go-eventbus v0.1.0 h1:mlawomSAjjkk97QnYiEmHsLu7E136+2oCWSHRUvMfzQ= github.com/libp2p/go-eventbus v0.1.0/go.mod h1:vROgu5cs5T7cv7POWlWxBaVLxfSegC5UGQf8A2eEmx4= github.com/libp2p/go-flow-metrics v0.0.1/go.mod h1:Iv1GH0sG8DtYN3SVJ2eG221wMiNpZxBdp967ls1g+k8= From 0dd75aff91745a3bd2fce45bdba72a77ba03f349 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Wed, 22 Apr 2020 11:20:59 +0000 Subject: [PATCH 1827/3965] build(deps): bump github.com/libp2p/go-stream-muxer-multistream Bumps [github.com/libp2p/go-stream-muxer-multistream](https://github.com/libp2p/go-stream-muxer-multistream) from 0.2.0 to 0.3.0. - [Release notes](https://github.com/libp2p/go-stream-muxer-multistream/releases) - [Commits](https://github.com/libp2p/go-stream-muxer-multistream/compare/v0.2.0...v0.3.0) Signed-off-by: dependabot-preview[bot] --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index f4fec05e90..5695e9cf61 100644 --- a/go.mod +++ b/go.mod @@ -27,7 +27,7 @@ require ( github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 github.com/libp2p/go-libp2p-yamux v0.2.7 github.com/libp2p/go-maddr-filter v0.0.5 - github.com/libp2p/go-stream-muxer-multistream v0.2.0 + github.com/libp2p/go-stream-muxer-multistream v0.3.0 github.com/libp2p/go-tcp-transport v0.2.0 github.com/libp2p/go-ws-transport v0.3.0 github.com/multiformats/go-multiaddr v0.2.1 diff --git a/go.sum b/go.sum index 8192a5cc90..d54c92e386 100644 --- a/go.sum +++ b/go.sum @@ -276,6 +276,8 @@ github.com/libp2p/go-sockaddr v0.0.2/go.mod h1:syPvOmNs24S3dFVGJA1/mrqdeijPxLV2L github.com/libp2p/go-stream-muxer v0.0.1/go.mod h1:bAo8x7YkSpadMTbtTaxGVHWUQsR/l5MEaHbKaliuT14= github.com/libp2p/go-stream-muxer-multistream v0.2.0 h1:714bRJ4Zy9mdhyTLJ+ZKiROmAFwUHpeRidG+q7LTQOg= github.com/libp2p/go-stream-muxer-multistream v0.2.0/go.mod h1:j9eyPol/LLRqT+GPLSxvimPhNph4sfYfMoDPd7HkzIc= +github.com/libp2p/go-stream-muxer-multistream v0.3.0 h1:TqnSHPJEIqDEO7h1wZZ0p3DXdvDSiLHQidKKUGZtiOY= +github.com/libp2p/go-stream-muxer-multistream v0.3.0/go.mod h1:yDh8abSIzmZtqtOt64gFJUXEryejzNb0lisTt+fAMJA= github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= github.com/libp2p/go-tcp-transport v0.1.1 h1:yGlqURmqgNA2fvzjSgZNlHcsd/IulAnKM8Ncu+vlqnw= github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfjg2pWP97dFZworkY= From 5c4f2c7df774aa6e12d1201d4a1d423e48c18062 Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Wed, 22 Apr 2020 18:52:08 -0400 Subject: [PATCH 1828/3965] add tests for backoff factory reuse --- p2p/discovery/backoff/backoff_test.go | 68 +++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/p2p/discovery/backoff/backoff_test.go b/p2p/discovery/backoff/backoff_test.go index 655d720622..063978fd91 100644 --- a/p2p/discovery/backoff/backoff_test.go +++ b/p2p/discovery/backoff/backoff_test.go @@ -1,6 +1,8 @@ package discovery import ( + "fmt" + "golang.org/x/sync/errgroup" "math/rand" "testing" "time" @@ -123,3 +125,69 @@ func TestFullJitter(t *testing.T) { t.Fatal("jitter increased overall time") } } + +func TestManyBackoffFactory(t *testing.T) { + rng := rand.New(rand.NewSource(0)) + concurrent := 10 + + t.Run("Exponential", func(t *testing.T) { + testManyBackoffFactoryHelper(concurrent, + NewExponentialBackoff(time.Millisecond*650, time.Second*7, FullJitter, time.Second, 1.5, -time.Millisecond*400, rng), + ) + }) + t.Run("Polynomial", func(t *testing.T) { + testManyBackoffFactoryHelper(concurrent, + NewPolynomialBackoff(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, rng), + ) + }) + t.Run("Fixed", func(t *testing.T) { + testManyBackoffFactoryHelper(concurrent, + NewFixedBackoff(time.Second), + ) + }) +} + +func testManyBackoffFactoryHelper(concurrent int, bkf BackoffFactory) { + backoffCh := make(chan BackoffStrategy, concurrent) + + errGrp := errgroup.Group{} + for i := 0; i < concurrent; i++ { + errGrp.Go(func() (err error) { + defer func() { + if r := recover(); r != nil { + err = fmt.Errorf("panic %v", r) + } + }() + backoffCh <- bkf() + return + }) + } + if err := errGrp.Wait(); err != nil { + panic(err) + } + close(backoffCh) + + errGrp = errgroup.Group{} + for b := range backoffCh { + backoff := b + errGrp.Go(func() (err error) { + defer func() { + if r := recover(); r != nil { + err = fmt.Errorf("panic %v", r) + } + }() + + for i := 0; i < 5 ; i++ { + for j := 0; j < 10; j++ { + backoff.Delay() + } + backoff.Reset() + } + return + }) + } + + if err := errGrp.Wait(); err != nil { + panic(err) + } +} \ No newline at end of file From 412b885e3f74712a6aa23867dd1e50e446ed69ef Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Wed, 22 Apr 2020 18:06:28 -0400 Subject: [PATCH 1829/3965] backoff strategy factory constructors take random seeds instead of random sources --- p2p/discovery/backoff/backoff.go | 28 +++++++++++++++++++--- p2p/discovery/backoff/backoff_test.go | 6 ++--- p2p/discovery/backoff/backoffcache_test.go | 4 ++-- 3 files changed, 29 insertions(+), 9 deletions(-) diff --git a/p2p/discovery/backoff/backoff.go b/p2p/discovery/backoff/backoff.go index ecd9470017..d7393afbcd 100644 --- a/p2p/discovery/backoff/backoff.go +++ b/p2p/discovery/backoff/backoff.go @@ -3,6 +3,7 @@ package discovery import ( "math" "math/rand" + "sync" "time" ) @@ -90,7 +91,8 @@ func (b *fixedBackoff) Reset() {} // timeUnits are the units of time the polynomial is evaluated in // polyCoefs is the array of polynomial coefficients from [c0, c1, ... cn] func NewPolynomialBackoff(min, max time.Duration, jitter Jitter, - timeUnits time.Duration, polyCoefs []float64, rng *rand.Rand) BackoffFactory { + timeUnits time.Duration, polyCoefs []float64, rngSeed int64) BackoffFactory { + rng := rand.New(&lockedSource{src: rand.NewSource(rngSeed)}) return func() BackoffStrategy { return &polynomialBackoff{ attemptBackoff: attemptBackoff{ @@ -138,7 +140,8 @@ func (b *polynomialBackoff) Delay() time.Duration { // jitter is the function for adding randomness around the backoff // timeUnits are the units of time the base^x is evaluated in func NewExponentialBackoff(min, max time.Duration, jitter Jitter, - timeUnits time.Duration, base float64, offset time.Duration, rng *rand.Rand) BackoffFactory { + timeUnits time.Duration, base float64, offset time.Duration, rngSeed int64) BackoffFactory { + rng := rand.New(&lockedSource{src: rand.NewSource(rngSeed)}) return func() BackoffStrategy { return &exponentialBackoff{ attemptBackoff: attemptBackoff{ @@ -173,7 +176,8 @@ func (b *exponentialBackoff) Delay() time.Duration { // NewExponentialDecorrelatedJitter creates a BackoffFactory with backoff of the roughly of the form base^x where x is the attempt number. // Delays start at the minimum duration and after each attempt delay = rand(min, delay * base), bounded by the max // See https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ for more information -func NewExponentialDecorrelatedJitter(min, max time.Duration, base float64, rng *rand.Rand) BackoffFactory { +func NewExponentialDecorrelatedJitter(min, max time.Duration, base float64, rngSeed int64) BackoffFactory { + rng := rand.New(&lockedSource{src: rand.NewSource(rngSeed)}) return func() BackoffStrategy { return &exponentialDecorrelatedJitter{ randomizedBackoff: randomizedBackoff{ @@ -204,3 +208,21 @@ func (b *exponentialDecorrelatedJitter) Delay() time.Duration { } func (b *exponentialDecorrelatedJitter) Reset() { b.lastDelay = 0 } + +type lockedSource struct { + lk sync.Mutex + src rand.Source +} + +func (r *lockedSource) Int63() (n int64) { + r.lk.Lock() + n = r.src.Int63() + r.lk.Unlock() + return +} + +func (r *lockedSource) Seed(seed int64) { + r.lk.Lock() + r.src.Seed(seed) + r.lk.Unlock() +} diff --git a/p2p/discovery/backoff/backoff_test.go b/p2p/discovery/backoff/backoff_test.go index 063978fd91..b1eaf12e5a 100644 --- a/p2p/discovery/backoff/backoff_test.go +++ b/p2p/discovery/backoff/backoff_test.go @@ -39,8 +39,7 @@ func TestFixedBackoff(t *testing.T) { } func TestPolynomialBackoff(t *testing.T) { - rng := rand.New(rand.NewSource(0)) - bkf := NewPolynomialBackoff(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, rng) + bkf := NewPolynomialBackoff(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, 0) b1 := bkf() b2 := bkf() @@ -59,8 +58,7 @@ func TestPolynomialBackoff(t *testing.T) { } func TestExponentialBackoff(t *testing.T) { - rng := rand.New(rand.NewSource(0)) - bkf := NewExponentialBackoff(time.Millisecond*650, time.Second*7, NoJitter, time.Second, 1.5, -time.Millisecond*400, rng) + bkf := NewExponentialBackoff(time.Millisecond*650, time.Second*7, NoJitter, time.Second, 1.5, -time.Millisecond*400, 0) b1 := bkf() b2 := bkf() diff --git a/p2p/discovery/backoff/backoffcache_test.go b/p2p/discovery/backoff/backoffcache_test.go index 5d7e5dd6ea..e88f566ac4 100644 --- a/p2p/discovery/backoff/backoffcache_test.go +++ b/p2p/discovery/backoff/backoffcache_test.go @@ -67,7 +67,7 @@ func TestBackoffDiscoverySingleBackoff(t *testing.T) { d2 := &mockDiscoveryClient{h2, discServer} bkf := NewExponentialBackoff(time.Millisecond*100, time.Second*10, NoJitter, - time.Millisecond*100, 2.5, 0, nil) + time.Millisecond*100, 2.5, 0, 0) dCache, err := NewBackoffDiscovery(d1, bkf) if err != nil { t.Fatal(err) @@ -101,7 +101,7 @@ func TestBackoffDiscoveryMultipleBackoff(t *testing.T) { // Startup delay is 0ms. First backoff after finding data is 100ms, second backoff is 250ms. bkf := NewExponentialBackoff(time.Millisecond*100, time.Second*10, NoJitter, - time.Millisecond*100, 2.5, 0, nil) + time.Millisecond*100, 2.5, 0, 0) dCache, err := NewBackoffDiscovery(d1, bkf) if err != nil { t.Fatal(err) From a618d46b21dedd53ff1eb5ae74437c178afda54e Mon Sep 17 00:00:00 2001 From: Adin Schmahmann Date: Wed, 22 Apr 2020 19:13:10 -0400 Subject: [PATCH 1830/3965] backoff strategy factory constructors take sources instead of seeds for randomness --- p2p/discovery/backoff/backoff.go | 12 ++++++------ p2p/discovery/backoff/backoff_test.go | 14 +++++++------- p2p/discovery/backoff/backoffcache_test.go | 5 +++-- 3 files changed, 16 insertions(+), 15 deletions(-) diff --git a/p2p/discovery/backoff/backoff.go b/p2p/discovery/backoff/backoff.go index d7393afbcd..384a95cf6f 100644 --- a/p2p/discovery/backoff/backoff.go +++ b/p2p/discovery/backoff/backoff.go @@ -91,8 +91,8 @@ func (b *fixedBackoff) Reset() {} // timeUnits are the units of time the polynomial is evaluated in // polyCoefs is the array of polynomial coefficients from [c0, c1, ... cn] func NewPolynomialBackoff(min, max time.Duration, jitter Jitter, - timeUnits time.Duration, polyCoefs []float64, rngSeed int64) BackoffFactory { - rng := rand.New(&lockedSource{src: rand.NewSource(rngSeed)}) + timeUnits time.Duration, polyCoefs []float64, rngSrc rand.Source) BackoffFactory { + rng := rand.New(&lockedSource{src: rngSrc}) return func() BackoffStrategy { return &polynomialBackoff{ attemptBackoff: attemptBackoff{ @@ -140,8 +140,8 @@ func (b *polynomialBackoff) Delay() time.Duration { // jitter is the function for adding randomness around the backoff // timeUnits are the units of time the base^x is evaluated in func NewExponentialBackoff(min, max time.Duration, jitter Jitter, - timeUnits time.Duration, base float64, offset time.Duration, rngSeed int64) BackoffFactory { - rng := rand.New(&lockedSource{src: rand.NewSource(rngSeed)}) + timeUnits time.Duration, base float64, offset time.Duration, rngSrc rand.Source) BackoffFactory { + rng := rand.New(&lockedSource{src: rngSrc}) return func() BackoffStrategy { return &exponentialBackoff{ attemptBackoff: attemptBackoff{ @@ -176,8 +176,8 @@ func (b *exponentialBackoff) Delay() time.Duration { // NewExponentialDecorrelatedJitter creates a BackoffFactory with backoff of the roughly of the form base^x where x is the attempt number. // Delays start at the minimum duration and after each attempt delay = rand(min, delay * base), bounded by the max // See https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ for more information -func NewExponentialDecorrelatedJitter(min, max time.Duration, base float64, rngSeed int64) BackoffFactory { - rng := rand.New(&lockedSource{src: rand.NewSource(rngSeed)}) +func NewExponentialDecorrelatedJitter(min, max time.Duration, base float64, rngSrc rand.Source) BackoffFactory { + rng := rand.New(&lockedSource{src: rngSrc}) return func() BackoffStrategy { return &exponentialDecorrelatedJitter{ randomizedBackoff: randomizedBackoff{ diff --git a/p2p/discovery/backoff/backoff_test.go b/p2p/discovery/backoff/backoff_test.go index b1eaf12e5a..1d0f0fc8af 100644 --- a/p2p/discovery/backoff/backoff_test.go +++ b/p2p/discovery/backoff/backoff_test.go @@ -39,7 +39,7 @@ func TestFixedBackoff(t *testing.T) { } func TestPolynomialBackoff(t *testing.T) { - bkf := NewPolynomialBackoff(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, 0) + bkf := NewPolynomialBackoff(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, rand.NewSource(0)) b1 := bkf() b2 := bkf() @@ -58,7 +58,7 @@ func TestPolynomialBackoff(t *testing.T) { } func TestExponentialBackoff(t *testing.T) { - bkf := NewExponentialBackoff(time.Millisecond*650, time.Second*7, NoJitter, time.Second, 1.5, -time.Millisecond*400, 0) + bkf := NewExponentialBackoff(time.Millisecond*650, time.Second*7, NoJitter, time.Second, 1.5, -time.Millisecond*400, rand.NewSource(0)) b1 := bkf() b2 := bkf() @@ -125,17 +125,17 @@ func TestFullJitter(t *testing.T) { } func TestManyBackoffFactory(t *testing.T) { - rng := rand.New(rand.NewSource(0)) + rngSource := rand.NewSource(0) concurrent := 10 t.Run("Exponential", func(t *testing.T) { testManyBackoffFactoryHelper(concurrent, - NewExponentialBackoff(time.Millisecond*650, time.Second*7, FullJitter, time.Second, 1.5, -time.Millisecond*400, rng), + NewExponentialBackoff(time.Millisecond*650, time.Second*7, FullJitter, time.Second, 1.5, -time.Millisecond*400, rngSource), ) }) t.Run("Polynomial", func(t *testing.T) { testManyBackoffFactoryHelper(concurrent, - NewPolynomialBackoff(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, rng), + NewPolynomialBackoff(time.Second, time.Second*33, NoJitter, time.Second, []float64{0.5, 2, 3}, rngSource), ) }) t.Run("Fixed", func(t *testing.T) { @@ -175,7 +175,7 @@ func testManyBackoffFactoryHelper(concurrent int, bkf BackoffFactory) { } }() - for i := 0; i < 5 ; i++ { + for i := 0; i < 5; i++ { for j := 0; j < 10; j++ { backoff.Delay() } @@ -188,4 +188,4 @@ func testManyBackoffFactoryHelper(concurrent int, bkf BackoffFactory) { if err := errGrp.Wait(); err != nil { panic(err) } -} \ No newline at end of file +} diff --git a/p2p/discovery/backoff/backoffcache_test.go b/p2p/discovery/backoff/backoffcache_test.go index e88f566ac4..7b0621d343 100644 --- a/p2p/discovery/backoff/backoffcache_test.go +++ b/p2p/discovery/backoff/backoffcache_test.go @@ -2,6 +2,7 @@ package discovery import ( "context" + "math/rand" "testing" "time" @@ -67,7 +68,7 @@ func TestBackoffDiscoverySingleBackoff(t *testing.T) { d2 := &mockDiscoveryClient{h2, discServer} bkf := NewExponentialBackoff(time.Millisecond*100, time.Second*10, NoJitter, - time.Millisecond*100, 2.5, 0, 0) + time.Millisecond*100, 2.5, 0, rand.NewSource(0)) dCache, err := NewBackoffDiscovery(d1, bkf) if err != nil { t.Fatal(err) @@ -101,7 +102,7 @@ func TestBackoffDiscoveryMultipleBackoff(t *testing.T) { // Startup delay is 0ms. First backoff after finding data is 100ms, second backoff is 250ms. bkf := NewExponentialBackoff(time.Millisecond*100, time.Second*10, NoJitter, - time.Millisecond*100, 2.5, 0, 0) + time.Millisecond*100, 2.5, 0, rand.NewSource(0)) dCache, err := NewBackoffDiscovery(d1, bkf) if err != nil { t.Fatal(err) From 9b8dbb79824f7ec75c8c10e79ec552caa4c36565 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 22 Apr 2020 17:50:50 -0700 Subject: [PATCH 1831/3965] fix: avoid calling AddChild after the process may shutdown. --- p2p/net/swarm/testing/testing.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/p2p/net/swarm/testing/testing.go b/p2p/net/swarm/testing/testing.go index 5e39602426..10de2ace09 100644 --- a/p2p/net/swarm/testing/testing.go +++ b/p2p/net/swarm/testing/testing.go @@ -73,7 +73,9 @@ func GenSwarm(t *testing.T, ctx context.Context, opts ...Option) *swarm.Swarm { ps.AddPubKey(p.ID, p.PubKey) ps.AddPrivKey(p.ID, p.PrivKey) s := swarm.NewSwarm(ctx, p.ID, ps, metrics.NewBandwidthCounter()) - s.Process().AddChild(goprocess.WithTeardown(ps.Close)) + // Call AddChildNoWait because we can't call AddChild after the process + // may have been closed (e.g., if the context was canceled). + s.Process().AddChildNoWait(goprocess.WithTeardown(ps.Close)) tcpTransport := tcp.NewTCPTransport(GenUpgrader(s)) tcpTransport.DisableReuseport = cfg.disableReuseport From fac8488ec4e15e866872dbfb151d5df6f5d95199 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 4 Mar 2020 11:47:21 +0000 Subject: [PATCH 1832/3965] remove ETHBerlin pdf. --- .../noise/go-libp2p-noise-ethberlin-1a.pdf | Bin 657719 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 p2p/security/noise/go-libp2p-noise-ethberlin-1a.pdf diff --git a/p2p/security/noise/go-libp2p-noise-ethberlin-1a.pdf b/p2p/security/noise/go-libp2p-noise-ethberlin-1a.pdf deleted file mode 100644 index 062183b1fef6f74f0e4f045f625a5e0b1ded9c61..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 657719 zcmb@tXIN8f*Dea7_l{V=03y9Nl^`g+NKFWa-V`FeNJ0~kjvydSx^zNELJ>r&AVqpn z>Am+NeNS9#z5Dz2`?rrDEAoS35S-vBEPVJC@Dwl!tTOa-d;vu z>sL}Sun^e697zg+fCbg9?4P?@gT;YMI$%LX8(UW^XW-q|%+*TX%EH0Y3jFXPsf(+# zm6;tW3O6oBO*24cL#Auwc^s1}RQxJKbE?8# z|ANVWn2OgNCk2*!oIR&^q+(Z|k0lpne){M+@h#rp5`=j)VT6((R|Re>d3xoqi8eG8 zk6HPu^S8;jvP-;Nw!a^ISlFmZz>MU)y%yrR9%v@!#{S(>FE>;qjVnt=ry}6>j`(}s zX03{z#IH4c+@-8tWUf!^lTXOK1>6cxa52nC4(;PynzYsO+s4OXgea}0>a+YH9I2d} zIe7ECA!_Il#-|Z!dqp8Od+r-^Qy!Kw-!hd#2fc9?-oLXYwX(PTTUWrRD-~bq_3B;Z z-uX3zh;I$PPhf<^yALdVL*!Ohvi$^|U(*9AETdspCl7w{E& z08~$`EN#plJD|V@Lcj%KArUdKq{KZVQUI>Mf9UGLe_+;fcCgU3as?Xzzb3B;7KB=% zT){?QK{*Fo2WMSJGYc#5l{Lt@2!kd6$ba|{EU18TRnm0@EaA_0N-iR#B7fV7l8Y!< z3Xu73mMi-9h~g|^9Nq9?jRKg3qJm%<9qjo!2G}}goH)F_ay+u{cn7Puk;T9qU7x0 z=J;o~x`2M2&Fozq%>YC#(12QXNk#r%jSAS+*$qHj4gAc^*5%5({+okQbbg@p1=Gv+ zZ^FlKJ2sM?EHftGsj$G8)HXZ0blxq%=fQ-veG5ppdEXKE!&&VZ{pCEHS^9o-8lSPY z!}UO@&o-@@Mh?=hE|0E{-fE|bmNq|MgguxQ`@BcSYME;YYNJrLD zHVcs}9Qm}J`&Fro*X(?6R=oQRssQ3%VduGgOG5iA+PlP?O8(YzF;RGi@7-Tql58y< zcBffO2}JEp;Rv(D*KRjHSp?#~SCPY~-9#c=cH&oK4)@MG@&&!+(}tn&!{u@4#zwY}V z)5QO^OG(_9`~#n`khti-s)L zB3B#zN6r6P1pkE0e?3tC2&epzU#zRrrxSl+^=5WkdOHE5(V( zCPQ9}Kx_g#zOfVgD=EG7C?{j0A^nilssHzvI=wgT4`#NvMY@QTd|7S&9X%lO|AmyW zu;f4J{n_O|D*g|n_YZaWFDy;?@09yL(X{_-{}a6rgii7MU##YyhFE-N||3(gP)80Nw+pvCq710L|9h{ z=P`Sfhni>xi(25kA%SAnAV zKEIwyqE8PWIr2_Z5&PDv{zmb-r6?D1fYDbDZKb)dHB#$LWEBIcvpE+<|i!CvXfDR1B4 zT*nmwoxSlLZ9Iw*VQy#`*;3DeuW*#Axjn3-s`HV^<&}BP`JDAH5x3d5`<0kS&eJ!3 z@RX8?pKHXU2fAu`w#dXML*ILkPfRV&J(#~`iQhpSMbq~Jf3dq7;_)4?8KZ0RDLUNRhaMy?Ws;Sy8W6oGwSw;Q+o0@QsM@;db2fulmK4%tCrTlBX zzIyC^#W$5 zT)6Rngtnz3z{p1)`p*4y8Xf1WmYO1%V;-ufp0`c?7efV*xLZxMe}Cr%_me?;2@bzNo0$>y`nA!{1Z zp$P#x;YtTs_P#8oVDaR24LHi^%>5>uf$Wv5519b&HN@wBN&(f2Xc{m8f@~p8;)9P$ zP6YB)bbMf2VqEv<;EUf@8t8sdUz4`Jc=Gbmah=88m2YH{qoMfE38GD;O22&$POU~3Th>{I@5&y+tH|HC1;t+9w z;3|5Tjc3x_Rfoad(yX_oA*;kk_nSP+yOT8zldoQijqc9X=k11H8j$0_9Z1&RmSLak zfswa8---Ax?mgFNDCmFdn=tC>_^WwNrY5jf>-=i@r|93C5saxi8X}XQ(l*7?vA5CgR7-xso8dTvsKXxac|zYKb*Fm>St! zW~;fpF}rIpcnXM;S8S|N%qVR);QkSrAu0zmo7{4Lcb~O2W%_}^r-83EKDG0MF*HK& zX?z5ys~jT8Lk+Zjca;A6!`E=AL?w&5{*$}kBt zID-g~u1@Xa@~N-!C-8xh-FQRYlbGNb+2aWhr-rr?*yXCiXxi|9zIT}W|rEB8( zsGrA$(yjG;)0(Xpc0_i-vJ)P3_Ca*TCx32A^nN!~$+RP6YV#~lZo-`IhR+vU8wo=W1j4? zm*Z7W4_hivdSX%m=NKT~-<&Y}@pB4((SERM^tO~+_M2Ibd~wBtQBfGnau7#(yk8fZ z`OCUT#dy{O(fQz&3eLGbmM7oNz6qupBS6l+h`gZE>-#=}x+m;+rUKvKB+zW9pEP~p zNoB>YADYh%<4%H?fo{Ft-%c{H2_zz^B&WA#E_Ec(kGmG}xlU+yCtzIZU4^wecj=kT zmp2KIqFA0iqaxnw_ft<=+R6D@F#8vBMOk5cuNzSZYK}fV=~WtKyE>O-oLf;e zLL)T#HRB*_mxB^1)g7p8>p4dpi*b>Ql+lVGJwF?A$+!ctrYfXk zu;&27HugS$MA31;tVI1oP8F%KK9nb?beG*+vuLg3I!?g|JAA^6ozGvkwu~-6FD=|G z%KRh}UQ=qJ9gbqD(O#3k?+iFFS~pQTVy|uJ@sZd zG$I9$kZcT3LIqyd7T*NUkEtw`6IIx)Xo5OitTDkgFDY)Djdr>_XQQeQla3=BZwH+T zSmb$cNWWc8-aHFg?c|$npd8ACP~}@;Y9C6s>I}c9OP8=S1{K_1ULDtNnM>0=`)C}v zUbCB5{Hki>d4nY3@gnU_T%~qOnsSL@$%%{e2)`U2yL=d7T5#8*WOQ&0M#E^kA?e;+ zd*7W{oEwUm*+}e?-~CI9&hw=*tMj^dp%B_Sx1O!JZX=}A=G5&`AI?~44n@U$<_L$& z`;OPy7}Dy_Gw!42>a{t{?02?EPo1>jGo_2C5wyYqk_rv(+h0#U(9vD{xzbUxlxFBGQK=}!reE$lxqFD4mAXWYd} z`?L&uC$iCA7Qi{z3UC$C4F%K5SFd+>iMlWI)vonlBz+5f;(_0s;3@+<7KY&}I+lVW zn5E%d$kb9NUh!QKUofmmr`%%TvtSil0aC<;fN2wl#;5mkDIHfd_tkHgPSvHIyWUe# zf}H!rZ_;<`KDXcMue`yfSHvlt?Rbv^nJWH06rzv$v_1S>5?+F!2PW0ij=-Yw1Jw8NwN~PzLc=go?*EQaFkjwrhIT$%yd*j^_E^Q-nX)p)x zW|aVHIXx(x;if-pbNWYdp{1qiGda1-T@^{CcXh!X*J#c0GfGE3aTSJL`a&1XECGLt z)m~xcG3IgV#jC*x)HSyFCsh_4wajuXz^2LUr_sLq)M>1DPXd0pmC?;=rWm$B6-^PS zYPy7!rlOJ$`oiua%4kgUBhrJ8O5R= zcsJ;bh5cg4>PR^Fr-QV*JYYU5D`*z2@>!mQy|xznqNfDn!x2W9Sq*-G;}yXqoCi^A z4ssXe?_6%~5ZBaxPLAEv?y+;%SC_!GY`+%A!q$r?V2nw;SEREe8fo_CVnr=)?Fx~-$qmQv=fsv?QW*AQ@RP^ z_EEQm9nS`ZEPVj+=kGZZj-rJuJwkXJQ>4O9kycJCg;qECW-s^=c+RFLGv8(kPha&V zzm<9AOqq4vFU?!3-RhU##1OVLK7^d+Vt^h?K2`=RC}kW5J=C0~!>OeUI+AYM!}*Ak zjd*)R`ElLUM(pV3afZj6#;}9EUVS$`)SW7|__#33DVxbGhsSNw)>OU+J zamK0$AX`UuHYriN5PC1E`!;6j{T|~)^I8ROW&8H++dESlVX%H%+Ce0F6UU*8P8!j8 z8LamK6=xK5L8oE#D7h{9DWWl%(2RVCKaEh4nDT{5!TDd8kS>0x?2yO3pvr_{v{ zq8qZ%32LmquoT6cStFi4Q19z{v%RhA*FC&=smJH_d^w2$GzO&sUs{{MygW=h5!mrLO8G5gIm3r}pL6gNiqnC9*gr99< zMDWM8T1KY*Pj>oeZbwXey^3Pa5dqDe*WIdH#?QtGMUV>V+-1V-Wo$!)+~%W?hJVT^ z{%$}7fnUe={-907D<4^*xs+P=9{BZ`htK@-mi;Rf@1<6Hq5DLJemVCLCPt)SxmPhK zcxkODV|M*Rs%I^mn?;=krm}Om(3=MCk!MQnc2(821v|p6M zjUo);;-$wfKcLcr`>hs=>#})cQJk-1HGg-=(1D7#?s5vgyv07tuOd)vkIy#hTCelM zWUqL1^vRtfTCAaH$(|zv1Mh_y;ucS#h9S&nC9^RCIfl3jHh1;W1$9mxFa4CJdtZ5c zTzT9MKa%=fb1;&c%Musy{o0MjO4)o47A&|A`Lb5$!P#aKHs5k5;Fk8Lp0hTSpOVjt z4AJ}yEWM>g)&H4)1g*f`@g#?4SCqT*GrAl_Pzr`KylI zwH=yn)9FtA^x8B0hi7GSk?RzVLxrBHh>O>z3WEGS%zg-}q^Y_QYm<>_V?_`@%{LO} zR{Q($qjloXf~ASoyu{+WDd@R`y&fy*iQ_(1XgYq%8i-~kD5DS99FyFTJ)6(g_Escu zNslRPBtF~9oPrds5u3r*B1RK4(SdM8@|3mRCaSd70A?{oM!^z*Fb_(4QgY2f3aYpN zrIrGb^bB%?ivJ10C%c~Q$8XOEaP@jF-kT6CSq0NC>SQ<9H_NztH+XLCpC`AUrCU%9 z{Aw$U1rXNZdR2P5vpI z4v|Gbp0H0`Z?MIW?jD1_3kwx1yP986mHA+xjY z_-rbZ~NCh(JWUUmjCW6HLXYKo+pFX-zCzEC!uO-}(P*u53eS2r|)s~jd zup5>HOu_Y)p@V{Tp*joZd2euk%dqP|X{r3Q}8OLlRtFEy?6=oUXN^ZcE*MJ>IzJI>7PK$K?{ixy@6i4142iK2F@>@3@w{xEU9r zai@=#?kPoyRgVA|2{U5ydOk-wi|*s^Y}Og3xa^(<^T7(9;0!mBI$U}xHR%_97koZ<+)6rF_9X{oL za$NFK2j=;p;FrHN6;-m-sKiga3Sw+)He+_MB=ZcO*ka%X6XRQX{x9T_)y3C071dw~ zAm^7nXolEtS-lTXEMVzw;*J{&$`Gm>)Ie6#sctx)eaYE3Y~UJ)84|{vzYJXZ@WL&N zeFqW}%I6p&{$amtwxTy;IZ&3hb5Axu2lK+L05c-JX<{^iFIs#zViO*Gf}`XSFLrm*h-iAY!_->%Sa9HW{js;3v!e5@CI@qzK)+L8 zeYLp&$oVZiZ8A{tdWNkGl}>%->uR~+SC-(*CBdm1s+d_yfpv7v)S1r{3kR*rkflf6 z`ed5(wTnMe9;}%;hW+qhF$GP_eRSh+sQNgTbz12469EC{ zs$%cG;~tHS!a{YNc?~AHb&^a*h+O@z@pI=aI>1up8Br{~S!;Dgh$zvejgKZ)r>Rei8Y5(!iE6jVJ|Ya<}*1jaZ@wFpWAA-&-UWArrfDWMR1* z{Nz7`C!UY2o&IngRis?ef`Ljf)fb)yTH2}X1+nf~(o}BHRH9ilHO~_v?)?D`KSI z(g*9G4a+3u_VMTHZxG4wi$y28T^{0TjQP@A`z`+^T*~@kOYQ#Y#~^;EfXa}nAt_)9R}DsS@swx?>MM+Ihq})hf~b=b{7HBCwoUHRd{62oUB*cFwB&_uP(7)j z{mY>U2HN{!hY&IGAB_}h`5YqH#y!W0m7fhNGG67WNgeMB`kU`GF+1wJwaa|G!zBne z2gq#3LNKF$KEd~3(_Yg#g^>_VsTAqHmEqTsg4iM)b{bh>A9D5L+sRXPst~G7wVkm& zt+oExhjWjVMQE{ap0iO!t-d!op}M%fl*12m%41ZY{~>iQdzSrMbK_0--c_fXRAbPdRsKi{I5gD-J$2hw(c=IjEgGk6u<8kaKd8t9h`1b}YItaWXm3 z1Kkyl9FB^(tt+yfrMrytG;J;yNc+L0AH* zQ08>VoWfon<7R5Pj5OFKW$<1BD_J&%u5KzIk@R4AAop*kzmERRf@Zk|xSb`ERlDoM ztm>CFCzn^GjcG)~JVc;iV}0nCtpr?eEa|;#yo(_Op<6)N`AU7N+I#%)LgG9}mOoMN z-2og^$_QA2j1IRhy(f#kvEulvZb>_P$Ks=V1ed7eVv`C1C$i-3*SvTmknE#cnXy`!$e zVlAb`gnVRDoFz4R#sE;)BK`8%r>*B7&3f_d$5gL;t52QsYJ_#aSi7^d6Ewv)$*{X>b0aw zcx{A+Crnyb+@UZ~Py{k6Kva|FvNpYIzfn^dltXh|O!#MNS!WrXc66*eMK1>xq&@bh zrAH9bmy%O@ub+M%bYX*uzkok|%!>SMFlDI(c!Dkv&612(-3i9iIpzw#<8i=pjDi~w zUnOw0qE_jbs8@R~zT!V>NY^86#t!SG3-G7Rw* zx;hpjP34~C36;19^dCxb*@6fjPnI2=MM|_jr z3KRyHxM{(Yj0ny7@{em#vb2vt|17Z{JJE1?Cq7=5W769aD$n^3b7d#<+v zbjj@J#XR@725;{l1>*hHEejURW{93Hsa6;Vej!kvO@A6Cr!Y8;2`V;c1N0x24}1FK za0*;vO!2VQIXLDCWHS*TL~-=E`Y?S)m10`{9s(10s0$!Nr+fp2^nx zIeb|s3sE2^hC-+SN4E}T&CRJdTl+?Hq*tVr799Emj5O1c-d8K7t}E3$J^dL_Jsk>> zdhE`B=IR8nLu+J?g4;QSZQ)XTQF;QRa>!M&gKcQ`+vmo(FzS)VPNyf;2C8*c>mR+eEq{EP z+O@YS?R+6{EomYnG_IpLlIa9$h7Mf4_iTDv=9#!kr0R*T6 z^U}6=a?+(H%Xsbrt%%vlFV9>Lo+Wfm-Am!V8THxgQGn*a-J>Dp^j6~ZfS!jt{c`U_ zbLVj?a12iZNaywKlyH;@7MMluF0@jqg`bN({m!e-Ph6>?kBxWt9@%FksjMMX{uE7d zsrg~|bNnnz2=tV?AT~iS1bc`$gW4LwSBS$HX|UwsJ&L29Qju2JO;1bRYRZU->* zczQ>Ah(d`^sp*zFaUpWj?*4^YmoYdFQlt~a@cF_koq%sDwXCqjDQLg!O?EJ0hbFK6 z1K7CZ33BP)kNF;1OwW0YwR%C3YzD)9A=Yn_c=oc7`6swqceubXdsE}iMDj-H`m`mO z!I%jZI3TAw^uVIudBccD`i>6db|fDH^I(F3hQxGOB3I{#7E~OM7hm1`6FyW+s~z4h zyQcGzEC9x-NvT3+QuU49>fQ4&(u|*9T32SOnpAAwRm`LT*3fZ#v#u$Bk>FZ(L~jhm z69uz4f2eJJvmX!_ix7f7V*|4C{GZW*Ihcwg1Io|wm?2z%IrAZ-=~B)on_AyC5Q3LqzhuJaj;DQ& zEFPMKR>%wH^sLf*35;Et8TjwO+-z&dYjYCWU{Zh8GiZ%agbd5aTGF)OyFg{KGh?&B zKcw>9BMm^m`|dK$fKWUL8PLLmMqOoP)sZ!^=87=`4| z+@<~lb20*k=0B8X(8}NTyEs-@-Me|RF}8{YswT$$D>F{(Xw*&VB{q%X!?&PvDC{E5 z9vEH6j*XglvhdvM0@%bvR*T|jm-{V2n z%0_2T=c?fIX+*Pv-puxXaNM)iV6wX%cjYTQ8KBcyduqy5)#U2xhdlv8uR{~5k&rT*XF=txIZv^v~ONs$QEpZ%pi^GF9qtqgmQlYVGS5T_Nk zle1ah+z?hf;Rq3$rTZlt)dK7ueC1uW#4Y6t zl6ulJ71+3DApcHz%OIc`33vO`#+J(JX-*m&x{A3(L)aT@^YMJaQXy=gd|FyDgVs`0 z_ubt-onbR0L)fiH*~9i@d)w3b?>$(icW#$`;=1p68nSWncbmR*FDg;!fwW+Ux-?a3 z4yM;wv(U_vCL!W%J4^#0x{hrS2$entvpMceuLd*hSw%D{Eu}u{2N|tR78t|e+Nne? z+YsouV@b{MQAdL?iQif@yeXNe6a*w1q&qEV%DhA2(f_o_s>zx{o zINlvX8Y^!I6(*#e-?5mXEra?Y+EVZMZ-h!y3Crn<Z8eq)tBtAkZgd_4is*EKVdhx6z zUf{xq;X2WOJe;zY6(V~!Q)Ab|F!;U$@5gNjJ|t1<{>yd7J3!;Mu#x7j|Hj=T$QyC- z-q(hhq?lwOf1(0aYvcaR5u5P%LZA#-`}&?mM+epw=WY<`_Phwz_MLR4J{(As{97qi zrA6YKvpM|r1#XG9SEap7TY;bIfS3JQwl z^bK)l`MDRh5WEIqvhoO_ZJa(ED%ZkQ+{2euBz2myZuFvV3A+F&s<8-dFWz0)paK69 zcY&&QQKHoK?E2@taV!rZ<}Mq~CbIEPT4buelP~!SU3kN&CM6&quUXs}DRB47 zJw97@-PXC0z`;T*i=t0_D9mZkBv;I{Nk7|==UCr74MGBaQrb;BzZ?u{AT&pbfQei8 z%4=-!+tjp4`jR!;JiX@5BrGfPdd_KG zdGbLR0{l1pu-43xLsssUcR6e5j2w(5y4+k&1*S8@Kw0Svw2-*gMgUcA6y74PC*bsT z+nR1b+y%q9x(KC%(DefUIr{%hY?2d8;z}6j?qs zmwO~|2pJbFFE02>VPFEkKW$9?wp6-+HbdZ4S|#*?_@hlG2aH2W6ikTss!tQ!+FG%8;c?4dRsLbAi0 zfYDhqqg;I~DG=R&kZwJu`ea@>Dj+b2M}Z9WG3L)av87GzHcNR)v3wd)zDXy%q}`DP zW3sXH=6CY41 zV^)Wpwc8sI<6G{7V{{CH;*KqP82tRf5B@t9q4`EO@-igDuqZILidSx1c+KjKy+)z(6It#D|gLm@1iA-lk3j*n6(PXHR1_UT0bcdF9fq~hY zxtg%I<)h;Hjj91}tRop}E?tiuLb%|@k{?v8^+e627Z1o82K`iVILxwfI0q^yZ?zkO z>H#P7r3(BPm_b=k2%>QfaLQ<*Gy7OEfDx{YUzNx~>xCSU2lDl7hmbRr`V887U~*6P zNg*ytA=iD=UmGl z#jx`f2KP1Rn{;6Bn@*9TK*vIWVuclGvK3m7ksbtJzvG%aP92By`A0QsYx%b<#;9QB zNg)kSgU;ej<(L78#C3E zGOCfGnDfnRY^Ax!QFB)@9%P|O{MNYX>Qfhr)#m&4AY^JKa>8Lx|JjjP`wdu;{|z#>iL&`mLZE3h3SD*@ z;o93>{$ivy&tX-GjkmwZN*gbRRXSy@0=YN6u_IT&LM?~S4{wvAdJ+H+F7ktyK~yTd z^{fMv>}Zgoq#FsrbzkmTYSRcm@Ugt3_2MIy@KX}iVt%|B9K9}Sc7KoF>lPbdnqSau zhq-$Yzj{141525(&=a?F}1H>7RLIuVqcVKRDSx);dH-&to;YU+m941FX$>eJzYRE40=)Wmn6sFR(K*{*R$(s&wgb5 zBm<@oabtp%2fr6Oh}=IKx(D))EvHsAb=64y4C!Gz1oP9FW1c*$5H!?NUuBROk+AkR z;^GT{4}^98+zZRq%bfTKc|ku%cz8%>D6|fQA z5pbHCc90nym2j*#A`W7hiK{dXJa`%RJ1i3G=#glCYRjoYrJxZ392v@~;f1tFS3fiF z?%Mz(6{xkt-yTcpV-Ooyjv+IFR%jH?Kz1|lorW03ilH{RUBQ7^3*qoH&dm6HzmdJy zV5Il9#iMk&8PTSVvPn95{$dLf=MZyzFs}NKX*#Jc8FGy358$rsO-jOv~&91G%E`jzC=;F8qUMB=F8oO1Fx8j0qNih<*tLCPpA(s~^& zfPIkyjNi-gGv6lR^ru_*+03B$I}k@si95b!e0yWJ#ylmY3X~smDL6X4DgBuntT|wt zz5z_YI)<7vN9LrfIO7TJ&gBWyh;(XG84Sk?%j}ZoKz#*2)nGi)fPT2QRcWmK2xe{W z(qQEk87)HuxwO}etGd!`_9SR}@jhk*XY*|H1k~{*TT8-Hp*d-An$}kw3@>eNn)LfI z7B%9v;@9Q1?twAHZxsM!+%9Wt3_jaxodk$eTLaHCTLXpDEN!)$&!MQ-(;ObRdc?}- z(CtooRA4j#9lCL^tR)O0-MUP@!wDuU@iE)@J!bYRWC_=kLC-@WX#W^w&J#(JL5lUV z^VdT;s52!Aob6wqEahTY%!lj_uI4=4VPJlttx=q}BEjz&lF#8B^!wJ6`QzzFW?Q9; z8o^90hSF3WMGXxm-RgN9+v}Qz_)*)c1Nw7;a3(`p{&n=28s83^?pmAKJi6xnV%+T- zS3DF;w++zS=E;0@<~0g?7neJnuN88tMr#68){tG*N5z~fFM$o4>nWSuc(zl2Z~QaQ z>c?#1WT5@>>^5C`$Lr6eCLXr4%dqLk{LQ>{OAP$T9NIt7`;vLGq|yIKK`egR7}S4v z03nyv#2H07P`IS@P87TQ-sVDy92>_XVCFWE!5E7hKoml;gD1*aZ2T5Nr6?0ky5!u8{tdA zeoKBl#+`1tLK`jg4g^mMpN~nj!Cur?1>Nr+L?bAqV)Z9E>e=c&&dF&s=ULrY?p#^M?}w#Z4NqsL z`3T_kC{hOgT-E~8dQz8^1dz`Dd%c#&cMhw4;a&A3?~Y&Ve2B;CC0Zd0T-BRiG4r4J zXb7BS0IjetpsiJx)1|HAYSM^03!}=YRegWH;KJ>syd^zFV9bADsD@HN9X#j+Mb6{} zS1$Rv&eMp`f_m`+3>K2Vwt817pnCDr^yGjvR|NMX>m)ACe$2D(lknI}q1}kI7tdyl zv?8X4YzX~(L3uR&<-vnbT&`V>s1PihH$FAC3=UAk&RTq~YwOGPR5J`Su;h*~pic!F zdQ4#e%wS-7G^gJg5^l~K1I=8m)J2V;U*G!_I`&K(A5i6kx}T-j2pYy~uICt$Xr(@F zU62JVjhEu8lF$wpvyCGxbOIXb%O46XM&&#@Ja=^Nz&^apS^HJZyWmg%dQ%SASX(Gu zmOq6hGpHEovv#bQt>LP1oUXbF&+O%7R-(tCPrag691 zsD<^Hx7qu=4>+L4O+#!H1{~BcfYaOOUcq=S&m0a_21-%fF&MEbUU9DQnv|4Q_&{dL zJ~{`GwWG5QS!ZukQ;5W7e8+1=^r$?n_33hZF-9uP5Q}3p{mG-kJCUn|MA3SgXkx=dX{>1H#%GINzC)gEUOeg)j1*o>D0R=hJ(V#F<_j*m(J%u!X)m>a}@ zk7NN22BLF0BWBNW{&n5clDZZ~+kFkQT?n`!3<>{a`0Hg3h&uF~Hn+9#8fQ^9LHeQd zr?329Gfl9)6b1TZPv%(xm4MsiT1FDuZjXHRNs_Qnb#Bp{hb^bqBqHHRdLUjlsE5J@ zH4QF3zMcds-c?2uWi0|3h^<$ymoeW^H7p^lU0@sHn?GDpx>PGpXAxu%bjB&m5IH7w zIt(7fm$WEsD|C@6a%E~~t&*iwVC+l>X;+fGCpCD4t&UdrR2@tuhyad>&r4PO+7JRD8e6KTf^l%sG7hK8~nPu4iOM_VxC4MI` z>oVnNVm9bv!y$b^fqeYp+g9*EStk7M`A+fk=XF_Jg{`Q3E06k2g<`Sf=XJ^+T=ZB< z{VK)-ukw~ft2Ac5(wDTxp+Vf!ECDd?7MJOzd#B&TGGgHuXL*$3mmXOQ&)P?Rx~Hda z?3^VY?9o~%Q$2}+hslQTLa4R78aqHb8*9hrc`48Uczv+S&$R8**Sg>e(w8ZdIcnFe z6ldJh-%6zbgN(v0sq?cZdCz-N(|A02mQXGR|!Mfp#dDbzI|cAr~5 z9LNPD{av?;6uPkP+zS|XbNDHNeDR`2aY7wL=7%m~LVcg_St_Di^m)aDZ>Kb z%^4~SJo*^b8*ynH=|tI`RB`Q4v4U;VP6|1)kGb|)SwjH6kPiy6XsL=J z^-jg%X(1&hy!EI3l=`TNe?h!-TTWKC?^{}NoKc$im%#kthn@cXi3sc$JLzC)DsvfG zF@vkHGLHsR7z}HNzlH*VS1^36moPiV4=Qd9!|2>)D9Rca;*xQ*y_-U@^rkGclUJUf zN16|7U4Y)Yo?|Gi3(%4h-=#;k+Fjj1zmL}s#qyH}L9Ms((W^o2?E=nr}s4~lW_+4&)ISZ z9)*E>qPGKCEi^E9Sg0bTOHsFzXsK4&_xUL>O*R54tH69}N!SOd3kVBu ztPk%3q2{f@U!$HXLT!KtKcsKplSKuhfj=tb*(|Q2`&2OItqI&a$ddf}W*ykq`Ef{J zmY51`n!ZgZ&RJif^u;vbHM;Tquv-u+xRi_xJIX^DmDA0e1oWCZBt)+!u0J*YYS;p? zqHG;0XvD zzVe?BLB+?$+DPPkA;C%Hu_=vm9 zXf0)!V5JjGI5X6VJ+!n2N8ai-uuX0v6`1NE@+X+A4g=bYX}9#v)Bqa%8g0LI+~3Gf zw|?8hp+nQB5k<`R+YvSMM*QJyM1$62VY$QBi8vj~teZDUN|{1G*k{E+bG{W)8%9Y` zX7j*G{XdEFI*g{m8yWeKPDcb}|-;>7geZ2#mIwb+?0tdeVBFedq`3_wT zzX5en!X22VCysZ>@5D8bs>?4S=ZbUtj0$XFRsHZdEm-kM z{O=%l(}uSx&YuEx-Y5Zydq%){?LFC+L|btGjYAX@oIA7gQ0|7D*ElJ^jJ>t9%d89WXTxJJeyA8%EDDXsuzF zK8T7WV3K{5?T92t_jDn^0g3v72%s3Iwd2@8ea-&&Y-YCn^d{_hhB@ZEd{5Z};NO#67I{x3KAL(FlDq|Kah& zujXxMnV~zjfYF8e84`hE5#N22NT8s6MNFo1%bRZ=DzJ@GN`v1^f0BO-Q4!Ga67hBW z;_YMnWi}@O?y(z2S(#z>B1p``K#K&tS=dHQhbuvAjAbRXg5Z?WUn?L@fuE?s>`F?u zqi=)(I{;1<5v)S;-H{(2P@SMzh)Yp33{UtIJJ>UqX0XuVT!RQ4U!B?j?j4?Sw}OXz zsnJkzF>&T=yn3Rz)Hh2qwn4EbI78M87;|~LImMRUFh=b61M7k|I&>mW0*_aomb+TQ z1F8V{Rh`A}1f05oGicJ`z$F~cmVtFjzuw)NS8WB2W0oJ+)oN)Xu+4w{|t7A zIVJ10bg&8BB)Dp4>Pj03x&VjFDQ;BFKA-{uM*uu%I*zR`6ZivIDp6-aTb(wp2A3vI zvGKBYVHnLl4|_2aW9QiZa5G zPwaS|!!raAUm_La+wPy?-dGER@_UTgWJiYh~|yjcdHU6xKfnL_qwed~b3Dd5=W$-=@Mjjl5dfNpv?o0+RpNZ(G!lSe&^Pzs?acbY|Fi{a zftw#h{HiqRz?wol0qJ4<7_glD50aF6zLWbK?Vxj*W&m{PDD(xr%@Nf|Z|Z+@5S8{uRt-@`xLJZ&eT6ckLC+7 zy|bt=d6MwITPyUX;2`52%!J|K(rDRQ{s(DycPBPT)uR}JvpFs?w zXUN?&7^0xkf*YWt8fXZ%V)O3&S#tnm0bl&wTVC zK!L8}?PbbVs3A(W_g4Fu=#}gjsNt1_R;+^OgXu0!(eZcgs@F}wWzfJbM4qdP`nU~> zkB`I*>o(Ko3(dHH8*xL{17s1Dc4fWrs{sCC0fNy$ZVlmPu{VGjuSYwE`-~^?e|8Vr zF`(iy>O2Qk!$niAu(y={zxP{&Xfs1R_#EX`(MFHoP#CoYwMJ}JUn)~6|3PYJS!PIv<NMEF|x=%PS8aSmb4PE;6bvst@$9jh=WwSa;kP-i7;!IGn(zJ)~dd*v2 zyZS4%k0Y_JROgN#c-T|wqdMfS6SR8zF=J zDMAxJYp4aBu-kUr6D_ee^r>7ds<{??rAUc!W~GE}?k(!ym4gD=y@9NJ8c1?#HfViH z^qF<9g30PN=>ffPateQL(zw%F*& z=Dh(<#s@iP8I>6qqU3d3amop3yIeUkIA_2{&8Hw@5&W*Wmj|QLrJV&Vxf(cnfH18T z1ZRr|{T_MvHjLt;F5pobSAlw^`psFNU6rOxDp5EqU)92HG>R|w>-#*9Rwy4@_NPSq zn0#}e?_^R2t8G#yf!=XV^a=tJhs!pi46%8aD26Bb&<+Q#RtKB9D{y9U=)d9nq~+E5 zIUf|z7w|xm>frOSulrSZLAoi9%E-_NBjrGuE%4Hf6~8~-$jVzsH>bCK z{ggHtMe&~^!Jq!y+T9Q30vgWL*ce?m2cP;;@#>+WK<3b6MzfZe7zp|VdV*)xI z#ZdvmU?L)#LT?RHH6QbPB0#E0qX7W~$P49as|BZU#~lY{>Ub%NyilH_6+1)d%$l13 z+AE&{1pimF5DvKiCD-Dy_^udPG~^xwvfBUhL+>HSS6@;#V`m3ca){W9d5}S=tRo7D z{GZ=hSX_-DX+x1M>&oW}T9cc`H=#l6`qLFWCSMaoO&Mv%L9BVICK`GB#uA9nD_OLu zv~fyaX2X-yWCxza&MX@6)Z06t448$lDEb_iXeJo6ZNe!Km_K0TK{~;l43g?tDs4J@ z7Liin6bJ%nxmMl6`Z;_Ji5?AEzZcJFmNJ)CbQcF=96pUZE#&C77IH}*ap;wRW7iF}R)8wR;r`NQk;MkLLa^dtLyw_hKcpji336q%a9Nd`rtQJ-NG>7KU zEWGC~jN&|Sd?b&3+f`%zkyI?Sg$^_yc=btTyz%hFTKV|vXT-QR{|w736#u>=gJTk%`luuFgL$eZ)0rti z>`Cg@2UW--@`e;X+T;lYz^0fDgPCHEvL@JdQ<#XbFV_p@cku0mx~0z)1;iL;y|rxf4Q5174ph+7)o?-jDi>_4eX+F z!(c*-Xek=x#_CVPG<3?xe0^N4>UEALPG*UdWU<}euK@B)GH7vX-tfspHueS?lGn!W z>6kt5_#n^4_xVCuL`T7ucw9(~gRFP>nppo;0^2*#{JZ^zt;YgdhbxccMKG zFXBic)uZO-hzSM^*=U7>qqV*NHFw zyewPlsY&gdcMkrS%vDnN@1{KRIQiKTu7BP<&iGGK?VpT4=9KE_2kaF5=^%gjRLaZ> z2|#S^KbWLKWYheE8B) zfS@4}+Nw-$Ic)7#z?yu}En>x@g9x4U;V4%B+&1W=Z=b2Z5+hL1xw))1$opUfey0ezv=2*Dx{W&0&eIJuj?4Sf z*E!8zktJ1QaEGZKVk;v=Y=4B+Cc)lqFUz*QZ%Qrob!+Xe!O|SNoj1E!u*9aU_y9Nz z(2Dsf4Ijq`P3!EO)el2%l-wDRy|)bErjB^|nBbJ{ygDkm_L#5fi)BT-p@cDAsO2DW zMDouqJiS;_xnf+`a_emc_4YR!UuvF*ZC+r#*mzCW$?@N2IKDq%|U1)m`KvAE{>Jgd-yx3m8~{Dd?v& zR7iQlDVxyjS1bMeyXBt6TV>wAE5G*`f8n-Iq`+TyYzW_~ecI8^d_mpI`NQ5%fvCcj zBao)~M9`--zNUC`g_pbFgGKR2H=nC)Pc_6v;(v##{6Z(sF44>8WU?(nKH=!S;+}c4 zaW+i3y$p0xyL1&x$a*ZbUaT`__9qk_nZ$pQ(bGxPXDk}Rs$pxtHWNa8est}5&C+$6 z;(kAC7x{Iw-?o@r<&DpzgVZ1M4>$YFwQJp5eIldgSWXTx4 z#L8W7_ao9$hKSha{Imnp3`c;O|HO?d3(idqEqHgECce$!%?nIz(NE{gJkOVYy_i$T zN!xO!H+_qke`^*^A;i?9eelRjB`und*Xw#4Jzn8FH^#4ubl8DyihXrdyQsQwuHr!y zqMUQGP394$e~&|#W0dZXVH%5Os3vo$hQ`zT`raI4zb-7Iqu(0q2!O01+H)W6W5h;% zR^vGhmR=+BZ0w+R;r{6DC?-qZr{WcS7wu)gF#B3WlDFhSNZOZ-?s8nXaZ`8iYh)9u zsKV<b9EdtPZSK^YGd~-=L@Ntqse-kzH8Xrv2m&iJq~x z?>QeVp2y}dRX_Q?J)V(n-w+GnS$aa?`WDgqvnn?0>+@*}BS_>Ypgu815LBqnLG{S- zKlB*Yvv-&%%vh0O(fcB)%2e1Y=3~o$!ezDv0~=a2mQ-UJ*|&T@qkq}CPs?>xR>wGP zb*^2`9sPORvT>

#I#v!^wP2&MWMbNzl<6uG*zZ#?=*trK^d5IY)`+vl5u+e39a* zyG*xgSCF4d>xTXYnocXwpKjYo0!Mj6}g4QZii1cA$gPi#`&>7 zLUVJxc@XQBYvDQviH*j@SNkR!qa}UM?jIM3?QdochtzXmDNJ_+v75z(|MI2^FtWrp z{_ew^Md~nto<(#pwK$k|qw%>=>^j4uPGiA%YVkp*^zp}|Dq26`mlSz%W~sELC?LaD zxKsS=aryMshesOullK;-M>sVpo;3R%VKs4qG&i*W2zSv2=Wq(Zr)IN#jz?|WkLSb5 zh1&6?wWuch*EX*TU#yJJOgwT_zj@4MdE%nIu`ygX21o=ND4oYl$B6QE+`qLw?im}M zI)d1$#2v>(Iy}cJzouoc&KTmb3-^+Cnpie2UU z`a4HAW$4*dej8!FJz6jgorv()Gs78u;b_o933A~xLvuUMgM#-m`lkzJ^jcZZC4Wf6 z-=l?EreX#o=dWH(LmNHeC*I}*gyOZ3^+|CRPJbP4(_JsC<0zKrym(C?*X~~8M;R&@$=!J3e6)6x&4h3fpH7DQES&lix;W!IR zQf^|jDL(4uJm)8{`6I5tDGOz2JCb+IygJaYF=S^R<8J7P#EJ1%-K^D&Lyu^Zow@dL zWR3e(L!Wev$q_&{xX0GIdrT{nOO7kgsEIq=uKX5_pf~KX`zcdhxt`U|HZb6UKtT2PUL*Z zxPN$XVShU)t(y?&aXQCs@$!$?H+a>qRO2SC*?F6vf2>z+&s8A#w)llAza=8VcMUet z?+g@xI)g9Q+5F|rr(9foj2mbvDZDLJ`cgUUC^=h1waVEp9tV?q?vY$R?q$Ui0hpL4 zw{K#Txix9-*#7euCr)uvVR4h%@SBDHCJ>32y}L{blfo1+^tOPOcm*K6Ni(0T|9I#t zhTj3g0~KxbX5qn~(XYZsMwxM%sQ}fd0dOuL#c%Aur0NA8SqUs8UtL&!VQp}goVy0& zaukS~L(weR_*#Y_RDXeepo# zM`_9(^ia>sn{MLr8!A+bgIAbpv<%h^rEctme)4h=^I2)ZFKw9MUX$};Xl#r8HN zdqC;W%;)^wS&R2Jpce(yMlX28QTqN zyqm7O<^c)x=OuS0ItAc%M%M1B#%s4B6eCK7nb;9NgNT?&_YD?w zLak*Bdc z+W(uj|6K6mTKwRtdt7AOfJ);%LB|&C7=K$LPxr(>FD*G{X50=kfnJlny_q|P} zhu#gpR9YlT9qwN$s(X>Agb7!4kA}+>>o_3#qsy*N3-G00nOaM($rQQF;fkB3kwGT1 zL5BYcbHyL@)~cv0O-+zd^FZyLe`m~9;!HTqf1?GUMRfS#ZTc5^YT|hv1!1N&&qlJlG2UQv1j; z8thHZ4ZKvhzeS02|03GIHdLgtS}*>-viz#^_0tuP_f6N8 zSd21<$5*Xgb6oubjmD)CCpNUX$AO-YA1qdZKls=QL{)k4RV+ayOucXUoZB546oZSKyT}FWW3_`#4g}{Q z3R$H3+msKE)J*5G=D50*n+vyN05hT+dC`1FY4Zgb5?T7~A}c40=HuJMDtVE=p;Sh7 z_#)!jkPaPC0y*-}EZWtg6?r;Gd@LrDdBrQaDwTGV03^abVTX<#0j1hqY3Yk6Z!O@n zIE&f=CDg4gBM4Zjo7nZNXdD}}dkjvG9~=H1n2%h?yr0TwZxrUN9CakYrp~GZJ^5Er z8DH2)RP`y}C4br@cauvfSz3&UlrZKNL#W!V6m7YpVvUZe#&Xdlj+ZA52}7e+td}^_24rDs;l&;e@01X6E9|m z@?5NPT!a9smh7^hd-&n$Mgfg9sJoXXDidv0PFMQcWfP5=fo^?Mvb=VeL=+fS%D^{bfq%-z0|L#8!}Q{^@8TrWL`%4@vZwbR&6*@)LD3-EWkTVKjY7iWK(#4~o9Hav-vZ%pBZ!I18S6F-- z?8zRTKcE6559V7h^X_FRG7rcmt-*$8l+EC;@UId!JwW8#B|?;de)8$d9a!X#vQ67> zoy6_YHJ&P&9eN$OT-h|DyVZ>Wm6NRx$GKZpb}2pnAa%NxSJPu}Wj^en3u0&9wX|$$ zfqQgIoA_b1HIa7hU~C00H(y*ff~M%BHklG_;5b^zW&j}Keib;m5eQd4QN?v~2iw@l zXM+@4S(n_5Uh0~8qMA51_K%bopUG)acpu#m!$wV;&Pfs*a-6pz z)(a3gNwK#gFcMH7p9X5!4t-j~O-a$1l|<$hk%f zTH~(=Mn09pbpTQ`ReLpqwwZju$ojzK*)_;d&E3~x@Fm76kt9@Eh|;@+=*rU5)tTVkux$bI3PnhNJwNDljBO{#9T#M#d=KO^;< zeZmkA@fUfGrN02p1(Me|zX2v~vHPpvF7uvWTtnyAGt)SJFnKzoywuhO^a@X|{HI^> zJa{+;NdgW`mZmA;ypb^GZBu39Btf*I*{3o+JGJg%$4Xc>f#&a}qHqYAMlg zU@DqsFPQB|r5f~|gMA67Z{13wx1kZ*`G#ElyCknaD$i^y0sjRzaHQ7wluA31 zWn3JapELX;kOZy>DD-)%ihsTZto$}CpG5yH&DPY@vOF<|26k1hebW9sfyz*k&!YLL z)3W)7Haz-l>uU_~-dg@tNB*UCvGF_@dJlg}f?YG}&{=I9?1t7JB+wDAC`WbRaBWnH{6Z4 zVJ{lRU;6Rr(8!gQ>-)Vi^ms*zbbnw1eg0B~?VWf|QsfkiQx^6#l9*FxO;|QB2>T0Y z!y{J~f^Wc5)g*wYQPdM1W-wAof^i_0{~5^7j%@EE*Q>7(*x22GlL#A_Gu#pNH_6fc ze=oiOkZS`5l%$0C?6WW^oSOzOby$+$@v{WJiOzRvbahyWq4r6=@A*dH%?AFK{gub* zF&CeBtA_@Pf~gvY16OPd7D zqAVnB2iJpNKkp~>Yl7!Tn~D_cY{i%FP(}&uVRs7x&!OXLKUM0-iw|9Haw7vwXc{mK zhfdEanFs4$Ws7R`Zv!(A{dYrcBOpzT`dywmcr@@wb?l|LhriNrFu)2PzmV z>%$iq2{_@ENCoy;j1j{@j#OT5cx9;!IJf0y(pT*craL8(jIVC}N0vKgRxot)UxLZL zu9jB!KK5tdqdKLJ=Hw3?L8<+(P_i!rzQ1K@g~nylhWJLr;Tm`52dVh7MK=x|w{`e8 zpWR@wV|267WS*;F9g$-jQC2*I>`LN$YK0Y_X_|KcOp#Lg-T3xirr(R`tecPYn%=B{`tr??D5&=+;@ zlw?vj%lN>bciBsZ0Ue46n*feI^XXD}FlW1q3Fz*S*j5uHs5e8q*GUx_rxW~RfOhPwxm z1|}_0DQ5@$5K`G(hxdR!C6#tG4CV$LW%kFnhqs0f1BjHZ;&$$e5(s`kHj0PbHYg*6 zA6>hH^=tKNBGAzhr=nzAGrwK;YOrym7L%xl{`@YIa!YonVnc0pL32U)=1q+=O3G&} zTsyvphZJ=$@veAulS#&2y4Di`@TmEXw6ER=YCNZ1t*Y&P;!Ew1h#*#MrpB|k&*gt^ zxu*Qs7Gf#1R~djFhyU{VaI?}%5Dm3hOsWYs}w;n zV_D-7@xZFJb?1ghFn=eR1=Ww%!$QB{5a{}x%E zaL;I@x&6tIXH;k78EFh{j0sK2j)(?Z+@m@5h=aCwCelX~0N<>mREy2>2U;KsCQGno z2C1R3PIB52)ry+#tOKVq!W)*1Zs{Nttf_msnze3Y}wp z1#P&Q8k&sjfvc0-flm2<8~qq*Xf#swan)EZ_TUmOCtI`@arX{oI{-rd@%-40@y9*~ z!ZJoEo6V{gbLnZ?=>GMinQV|(g%a6=tyA`?{yZR-g1CL!B3=Jj&T18s90Ao^)IKFQPV!Dhik zry!qLZpBCd61byZYldMLgZtxtn9X5D*xpv#WTW zGseB50U#}cBEzHuyX$>HxgkYBI^$*Lx;j%JP;a8mevHBFLj|{cLZl_mUo(&&yr9;8zLr z|E@8p5VYVyF~q(ZWz@6EoiokgSh>T|yu}+Sp2hX+SF|ZavZ%hMfzhkGJ1c)bmJlgH z(_&b}%ste$Xo2fh-S|FubwF9D9!UHzot&29i~+nUeGc}>4rXmChLvutg2uBe*ly$Z zChA~56^ok^$lwz6FLfc%TSWtc|Hh?jZ?3Raie}hB!7PUfk-r{?E3>B@&T)9ZTCB>_ z;ePIa?-_EMGjHj4ZbwK2i*#)Yt_e{9>x$t=mw=+l@8{Ls><{hLCrjU zA?g-U_srw%aZs)NcQ?<@L6&fn|I8LM>alTXj8}I;C&{W zG?u*3{t^g`S?VJ#=cF-?K&8$%e{pk5+4vRo*9}ePjG>zm#l@RoknBHUZLJ5|4TmVg zT$XY3|BmqY=nR$}WPv5RBVG)oW(`MH4FkkZz)`&S6SjLuUGb zp5{Lt$~WOLHsoab8}hO;XO?sb>4p(^zsfCTE`v$ zn5%#IPCxEWZ=~Em@`Ki-ga+a7HjQwJd3_1)Kc{utGlk(JBgD@hR+tS3hQXu@8^l!x zv(PG@N$59{1XUNf3_aBRHn{@Mu1)0mdlS*BWJ5 zXd{1tUjm5BH88ag?mNsA)s}UJ7GR*_13x@~B#+j85fbZ1S>O8Uj(XdU=s-o%)d&Cd zgH4tUT<;^S%dh<9v~wAmByc z9z==12U(RKNI)14+_Z>xdQ8Jm(-?|dDl}Xx1tUWwpl}3w3r1~2zaaHHNGDboTI^fkZ> zy!}HjkZMPs7oqaDeITyOWciUuiBn95ae69G@aBYSpDq=@TlTiaH-eEPrE+u*o)hTe z_eKV}Z+r1+q6C{aUS!7u$2$9QTno8>*LW0@6yBi2*Q3a+d3s4ZS@=EePbrQ$PLe%U zdYn(?E_h-nAk#%2oH-gFC#RR%K{dA(dh~8CA60(Y^RERB^Syb^h00hozw3GW*4wIMQn-^_6(uz;TD$^?aYvL0boQ2++ z9i6p*92o1VLfA>+mAfri>-TUt;ajw>@XacJ&lySJ&gBoR?c2}cBQIQ~@^c5+h=I@5 z)bDKiXujfG@C$hBhA&$|(P%(IvLM4g$V8O-Dz)Wxh2-92IJm2cd$?LK(hQcu6F5i+ z1BI$*L#fS&9hg-V!Tko-)a6%O_Tihb@61)!h1=7q&T>Afy5koBo4gQ(FVxU4y`(s? z5b@6N&MeN+R2bfp%rlHAh&KM8Tf|$$m-8%h%F24o8K~9QxL)dAR%3}R6?~rT&mQ}K z&zAI&4kzESs|$ zEu@dmd^XLx0hZ`N5Z^-|hICG=Bruaji|B@mczBi-pF|ss+?G)K@qGE-k%Is`KKg^I!Eh*1x!5hfEYyJPXduE<>BG&QKMDK$B!h-=3iP|zRYss$m6fE zS8T0jXd*1~dxMPHMl0$RC<;a}avHoZTop}-=$hi0fu}WLQ-h6b<)|VbuIx~9j-L3V zIgkae3QY$0zSJG3y*jmJR-Q!X|95v}KK|ZR-&1YE;mnuIrlvEuihoSCD~SWF5CLZC zEV-FHtE_5eb1J$PTvLwhOahG>ImX(i7d=*RI?ldZBteeKYwPG}CGh4v(gRQ36O7ez z_cIE>Nm(}~*WpJ+W`oRtn|M~3n?8`L`-3mE$zWr{eCCpb39_&%J^uyjQ$g!fV5TY6OWPi8ArInL5UhpYf9{y zxG^vrZGSgFyVL5!t$Jdrs~Kk)RT1|f8K#Lrm}VVc$2wvx_vo%D*P8&PxEN~Vzv}-a z6>s=Fi|ZHZY*NlPP_(Re$|6JPM=>Fdp@r{YUH`) z$mYzmEXEo69WN3Bp0}H;?zwHP7sia2)vZ2ChtRsp)@M2N2fSUp^W7(9+Rx zO!U|{DOvmNml6_nlIE96<5-kmC)lm%)O#WLfF3n+<9D+4AzFp=@4BVI9C{_GPK5zA zvNi63KOX5dOx!EKFrm*7AAVj^1W52jKWo|f$mO7U3kR;jxl8^h{3a0Om$>c;R(-sDOeBcp=5g7VC)DKn2HsPYBG)(Gh_z;;tM7o z7Kc@(-9oUX(sq&P%YZ-diIPqh%oBr+$1VC)USb3qa0(!)Yvn?FwDOZG9%UD*p6yWhO2z8~E^}5FdwB|P}}qJ1M#MyoadkA#x(Lz z;hFtEuTL(^Ju>aS&kc@+S_F*W(TOi)WS`M$SRggDvq1SQF>UDD-=A^n;y-pAO~u^c z;=1P~4n)hs9w@y(ai&BD=l<-M_bE0_QE8xm(QK?KYFm|=_=KB)$7ImoDDsnzYHRdiY=L( z&=A^kf{a1W$zo7+L`t-q6iekH@ZYV#R*_)ifrxAq*_VbYJ48F;mT`{gf`cru`dH;$ zbS&ked=bbu)ztA4dG~Z2To_(1B!AGa$~s*Xn~@Qrjq_AG`r4b?WM>$$y~?SJu=c4sq&(F*n->*2wX+7tI)4AqPtW^JybHcxUcil1#fu)$)>@wLQMsw!e+zy?@&n}NMYRYeJR`gKO!%G( z33k&UmG;p`BcIgJ;(<^-cOrT=5@CYv_Ut+wKoi6MT-JtvQS&!nxos_pDd;df11tDj z`icl!#59_WaF(Es-nspZyV@D9|G=z4a23hSQQ1#!9#-oyLUQM;kC9y)6|dij;^G_M zBsWu@F8!aLz_ReG#jgx`Z6n2LSKvxV+A^x+frpzNxj_zf%h2HoOjnBz8xskvD!QcwG=%wA=Zk>Gh%@5x>r=AB``}~b5^JOcWJDOT<}y} zjKb<{&To^YGlEwO?Vby#{Z&#}+D_{y+fILrW(BJZToG(ZFh)XjsMt}k60>YI6)-+^ zV&+~aH;dslv5otcL0D_d5dE=g+a`9z? zox48r?p+Pj|5xax7t7vOEGpzM zQZ0F+pNlU#5vt#R!gEV9$9m{d93mtPeAH+dB$~zK9h>-el{8u*lsX1?1pyVw{>$@? znpOh-TNWze6c<|%uJrWIGNh71s*(T7(@I!H6m9Fl7VWPXwyjrBfB&QX96y+7U)lnt zt{lef-?v$L=l`6f@r!i0oYPO7Yduac{M+KiceV@a1L)_ZfjhE`=(t#14Y-+M0sjR13VnK&k@XFSE7i8sGv9pK~?%&CYA9yTTL*6fhe0x9LkBWM1 znNDoss=PgmvCE{93Ci7oe#AB{##uESk|&Y@y*aKwdIeOhO{ z(O@3+pc!grO8pX->p$+nM+Jos5Py8_;ug{WNBEWJ_b@F~F5Dt{yIV!vtdspOMx>8= z^)j#&siq!cW_vua3+pFhe9f8EnlJmGJ9w6D@}!WEYeT_hI#X24rf~-5li*q+2qi;Z z#0s&0gruI2)o&F%me>t#VyS+$a~|oMeQ6$*?kD2K$2dHnfH>&kD7?nLW~qxi9fj;wDJaC8X)-vO#|^jQHb$yLskn}1pEpWi2hM|#ZEPJ z%Y(ZTjLw2NsLv$&{G7h3{!t~y8)HQNvPOji*UG1gkny_LKm4~jmr=h(seB(Tj}I+h zon@-E*dg>dGd>&Gin)utTg+Jlj-*qgj=x71awRSNo#p2P@e$%&*zkp&@$l0_?11!| z3!7NZv9nP+*DWdq1Vb&*&XFX}q7O4{Mh}FkQoBb4dK{xozJPpY+XLv&9NaI;v|{6T zRVM23Iie#|lJqTEm_I|xIe$_1mZxDfDIU2d;Itcw`SgxIoZgv9aiUQ7r_&VKHWp!k z69C^(`>o!5Q4-$bmVtnqsYcbU-D1@6rpQ*kHuQrr6)|J4RXqB z{aGJ02U(?H=dj7h{9GVN&tdM$`sjj~PK~ed5i#Du`jZo~8?bm2Sr%TZr@$p zyPPvhh(&-j3)~PA7@c$Gr`sHXDdq}>7%-3d3D5C@P@a@y&!Oy1AGlXqWE_K2`k?J= zxTp9MrqNuwMnw+&cAJasue>ZmxoPDpXv?GZ$VPlS;EaJ4thEv^x3zPiP9jht z;BRcbYquhqc{R#lh3bt*qz-7z=Idv7tAg_tnZ1|6t;{KQmfJ!D>)qzQclaZzeTMNt zqT_p^S_|Ep!Vp1?-!nz#$WE^POF%$a=%)Wqz5`&%B+LbnIqF9?M0kix<>ltv?mMTNFY>+^{9hsqkfK zbfna&i7x^>2gwkO*gT4#1Ig-`az+0gWnM;K5Oz5SPedGA6@~DftSO8aCOiscYKNAX z_*~>lTR4{fT5I8bg)W|$o+$H3bx85xx549pqI>g@D+Mq3ndzJxv>}y;in|R$@T)aA zP4-9MM&`MUL)sd?L+wo&i*d%diOUmj-0ryNco%AEad&O>X;yVS`UvBr|9}Yjs>(Kg z2*2NMIl-BYr&4RS@m`0E7M(*;Lk~!q%J?gJXO~TWCne}x2W3BC{%lqM`LmFGwfD0= zd}7}}iNLR_0jFC&NT`tf5g1MHUMb=E$8_pSt;KlZ?KJhTfexa3+oC*~j?d4CX_;v3 zh$DK0KqS5^%br&V^Mv>+KDdA)8{H*`d%>=$i@N~JmS*-$zu(qk@I)axxvj|ya?d{~ zA87Kv`KF-lM0{T07XEbuuW~Tock7mN=bDd*RbNm)`bmO4*y3l>D2N1l{JBvJ6a9f7 zhx#0x7;afwnpi%Au0BDvjCmW_YENzsf0aDLGcI}_xr)uXUL)G~RY>aAo3q!hC|cmJ zyBIEx-1%nJ8d@4gv^5D}otu*m#`Gs?xPI5^Xoo&&kzK@WJhk~9{xBu8rpwK|?3-$I zt%XKt)w>g}n;v^1UtZYpNBP5;JE1?ATtDl*P4ID48qL4%+9OuQgAPbtx@bFsvrMHOT9)l*K>@xeeyYc}m z{a)bN>$v(YpXt=%RIVRQK(p_l z1mBD#<(n5s+}fDeC4=GEo$d;%aJCdeewT<{+P91KaIIL&ts<+m-3?Y<3vidpo0L@ZnLHpyoiss zm+1(FoMDPMQKxy#GS1`2LuGjq^Ya+-`9VcA7^a!IPLYHA^fc;k&j;y?uHKnc8W1i; z1+Pa#J}Y`LquYUx9QpCSH!NtAUT7|C*>5njNgKAm({Qp=X>Gerj-w!HASN6)b)HNI#!P&D<=(P0cJVlWOb@Nco4HU!N=}Z1q(){K^JD%fAhlS; zB%pGx3)ET^ht(z&bhO@XvzE#Am$`YfcnxaS88bkZi{RY3f(KJ@D-_1QZHo1Sn{DQi z)9{+BCjPoKfqCDNe!qK*mH#R<#5@NUCW0vBU0SZ|@FlBP3H6`8e`eT!PhXPROzTuF zRDE(zJbv{ia6h9<1WntTSr+-3gN&S<>2ske5z*#GD)(CN&4A2`-lhZ>WH^ECI{sY& zuA)Qs&V`qiD#LtPLcFa{`f z_sVb&y@H`9SMR&~?9Xl+Q1(K&G6vc?;DNf(1(6EMzv^4z_mSIah-*}hgzql*`Ia5s zlywTYvk@6Le}9Ocp53JE2H2y?nEOf#cSy-N}uwU-p^!_#h=IR12mqn$P{t z>J_C}OXiMvqA)YY@Ct_Gw6fVG*oh4O&m*ng^A4uo*}ZOa7C)wiP>)UgCF!qYx>u$K zt@Tgj3v$NT(NE~K#x(@GMns+5w8q8B?dPSuxg}A2-UiK*GBU90(ftSUb%*@D`0=wJ z{~W8*CM?KJ8ApVk(VuPG2q5~?lME=oc1bIRUi{mu2!|`ONL0Q(hG`~2W=OYDzSlKC zwf%NhWLuwdKPL>=P3v1}4bX{u*>MJF8U~6C6ZrTnOU#uRhXYTWS?&xfZ#WnyDg9kC zlc?8}^fgd5js;DVVL<^U{+<9wX^cOB*x^ZYsZ|)gi4FLw`Ww4Nqb44(jf;Ne@1`rx zBXd>y`VAFut9yazpOmt`*~qCLYtDf>ekpH zhc1f_y+8Fo^z$!Nfl(V;-3l(K+qH-k+QdZ(Q;mPFCh~{|+6b%>DkzuW0&hV&;k7HkBcz zG<0??Z|YN)5)%Wu<`=mcj9_4K=Bb_?zW*Gh{lZW#`TnsSTs)WNYKe@X2b|ADiwF}y zLeRa2aI((*rY@Kx$&QCWqTd3`S1$8>>by9~p6@yn%b3g(L-WzKICtlKn}lnPcL?d>e5^i>g7wE0VeTkMFohiABa?ITSvd40iawnxzuA($ro$!l zJTr+p)5C|aa1o-`98@4mZjXMa^&*A!A0g55Q%*Qfceqh;9hz_!5r^u>2> zRj-G8-kZsLLsi53pXbG0xBdUvd+)HOn(tpUAWacb5Ky{Q0Rz&NDk4n`s3^TiCxj5` z0RkupNEZ+g5Kwxr2}tiC(p!K~69l9uC?P_CoA>?x?(dxEJok@t&;9qVBzw;@v$FPH zGn1LM=CiYAgJUz@jN(d97aps8!m$lZyC(A;dy-n#Qn_z7{96efoV47c!$$*9{yFvw zD!f-ZBuY(QcFGrx-ZdHIoFm8<1H@mUs4HWVU%XsSbc=~5DxOiyZWbz=iJhcR_8aPb zTY@8vb%DAlwRDRLAN{C< z!^+PxF}l$__Iu2~Jbi38ltS_XH8M{VBbF;hl58v4SC4U5DEIQ_8Lh%k-etr(XI^tp z)>mpu@CbSb5b@e2M|TG=Z%;>C=szi#o$Do8dH%cn|0ZM=6lMQA_wIk?s;KZwJ#ln* z_IBZym8Rr9=a+in|JspX%GA)#)zRLYU&_eG&imgSJ*cfSrN9L2Kw*}VzsoPB>j-vs z@#dGlf1h7U&)wV6^EJ%P*4y!tx2F$U69#<^bEoiLy2mf24R)i5rkwwg{m9WC=HSS$rgrJS z)Pz&*S~0@ercr0p;?cTnHKE^&80v55y4-%bSCNZ)ByQ6Cu;BV>W!KA6)3wsutuL*0p7;bM(lxJvn#793WN{=+C zs02IYx4gNv!d0q#nrNMRPk}1$Rh(SEvx?%rn%z&xyIXI?db~8BB@fjMylrHk_~5G5 zp;-xc>xnnV!b;gh2NsqYUdOA6dzI?8U>-|a$F+-@S)q<<@~k~fSh`F&9LX85S0#Ji zG-hrqCUzoiOB%neb3YyTToAo~%tW<4Z*IQ87M?skd)U);68AZ(0tOI>BOG2KU|IF~ z&wn}vaeQNV;&#M%ZYfX#?z*b0)V)UxM_ufGDswv^oOb_tC-C;c>DbZgtFN_kOT${K*g#MaS3XSLO%&CBDokM`a2}v?5BAckPxg%t*c4eKu#%H8^-g{Rj>LB}uXrzsJTe z?~zkz?KSM0{T;O=IjEwv=|b@`_u>Y+Vvcw(*9Y!e`L1OcGjm_=nKszces7yFP^}yU zYVpfCV!PPe(@A`();_Ce6J#4`>6SLz91<9NN+rll>OLzC3z70mnrT71tRw2f^6W3Y z+;ks$%N!MwcV;Y{1D_DN?@XR2i%SD}BYuT)XYYr-1e7tL4SJ{HVGpki%yeStAk5{K zhkj+pPpJoN8l!u&UDHFP;0Ndy_FU~t!Qng5gWy=cueX#} zz1>c!jeXsgt86`$fPuSL4vguV7T!lVZAouJ$eaXvje~nvLALO=wi7Fv^-MM?;^34uPw^2@r-fJU6;r4L} z<{~WhcCFOau{)(xiz+R3ACk>x8@co7R-@NHoke>Vfuk3PN8`N^G-`q z&C`tYP!F~w_$&SQVwe8TWfMer;YNf|!b0Po;%)3}ehCBmfeL50hYpRNJ#`F1_lUYq z>-^9+?t45TPp!R8-z85CYf*w5au#U8B-CW-lbv=%UKDz@F>kAD?LC+0UyS;QR)Xfy zk&MKT&u*I3Pto&$AJ~{iYRbyV8ouMU&ff{V@wPPLtcyX@CP>|yr-bWCB!Wx*(4eR* zUwXP8*TMX2nI1IS`7-U*2&DR8@VCE$-T9@K)VExf^}o4gzX!i@@iYgYiDtj~^Y$J| zwcO5>-tB08@nW%wPMX2y;L+4@K?XP6V^On%cx>?g7E_+Pu&>@G^!ws_#>}k z&Xei3Bl&!q;r{pP>H2S|=l73DKH&M?jo_x<>B9WIcj{cw=+nZEzU963v?=p`uSk0@ zmDnjCdu#8K)m?hWt$GZfzn9pHxs1gM7Enx5K2rQ(l$qh(r-j=ba&5dCO}vkmm3>x2 zmNgo0U%53b@R@K&?6FO_jNpe@?uiZQ6I!F)hegXN>u3_U+vgZG$9(hAazU{HtuL#n z#X)*)I5Ondgc$JjJF)7=MW*zWl5+0nNjq?^w>@=Ph{1lcpHJO(_Pgovu1%r)jUe%L z`)Ny+bLO9S8~>(rWdr*%rBc^@C!)TovZ;j=+2gDXENP{65GLyHv`w-ZfvGRXF?EN45bqTg7C>RIuf`c3=f4*IdO zO!U{OL#MUM$nb#ALwcJ$vvpK)v0_Cg{kIjP!osUBz3^AeH?LiV(y#;{?p_|4ZZ$pi z&>d|5@dE4ZOOyP{!HKG{yH)-D30_-g|A22*>lmGs*mCrKK+>1d-sL$pPLrinNTkLlzhT{ zKdED3xQ{u;!&sH6vz!nOs~q?J-nWQAoW&a}Uo$nlL>O!W{L?48W(|_FBRO)qaAv;O3*?Ry6RcI`XjGd^uNB*q zLDteg+vwZdoV;+)h9hq7xqVwU>#9K~)N{B^A~ zzrX1#VANjgX+}I)rf2x5#7nS-OpGEwTn^w^t@HN{j8rdZaaX!5_dWd6X(`=TjUYFh z^3=l!LR<#DG~Twtf350c!tjkei%aP?vE$vy5hrEW7DW9(dyFIgvB2t|L+t@<2nO@`b^{;qFJOBK7V%%efO(bGh-` z`Cc?XXo0pPvVKQ))BVMoqJ?f2_vVSOAIH3k8rr1*`P7mv9t}t5C53Ta$MN%5oh)M0 z;$Hp%N@-K|Uny@XGji;#h-FyP;g8=oXiLPN?7Y^&KIZnJ%a{AA0>6B;R50K4DkFn; zP3!;{eq|e#OV1lqzLtd6&b> z*Y2IlW&O0VG_O<7ZB8%2ee^*K`18KK=XS!$Z5HY1)!8pOqRho8JN@8bxjO#Uy4>Q( zq(LXAK6Ht?TxW=ftT? z6_i+;%`Qb=93N6{J+&j7&}TstB)3v+*{$TwoPkLj*BHK9(#~W(8SHL}YErG=1=go| zI>I{U)Mxrdx#3D4LE&q_~b3D)GmU9R1+)L&>-C|iN-E19x^erxsi-K>(!`0eD z@ie)9Th3|wUuZMMS`))w)h1zwLj9C5IOfJ%>*>d4_DaN|$!U*w2=CLdk=@a(m+fCJ ztd>g8BglvD6_W*do^dYSRy;;ec^vi|T;PRS`4-}ZPG8;)iVVMME8%M|FB71sSKYl@ zN%)BpQ5Dep<+*ZoZ@(lH=_tLFk?WVK@Q^nxG==y+vYwUHd~8^qcJcq}zN(Y{qgsl|k>}(}%xw77 z74zTN=0V;sj4`bt2`zCh<4+)MNrio-PDgXm_H*C5Okdsqb3O&^*r^k-GNbZ+*Xnwz z0_pO7IhHm2u4bUA@}Ca#@j8MmK0*I#R#108fmy^}y+7hIi;@UNh;A+J9`dHi&B+Ju zC&!DhK>4*3Eg@6e@2i0vpCh>?qkfvo$L0#1Q>pw}cCY+KXmIV&Pi>O_`Wj$(LxSlZ z(9yZqwOJ|gkt2`z-Lu4FY2HLBj$e>9{Spl;6$FRp{W_!E8+J+W4ya=9^49vA!>yuE zWtfzMS^RbQp}G&-99Ask?$XuyEtMW$*IX`KbUIME^EuC>%!|4$z3ThU`jb;t(z@p7 zp@H$*-K?6e$`@4g_71Smu8wPiAt8btcL&`@7P%Fi62H^bW*Lwks`89#qpM;LW3JD2 zT}BOXH~pa3h{)v1S+8?exWBk6B4h>5ZOE%2Jbo*BUU(cn;1D0`C-73W&1B=c3(EQ?wKHb7` znf;z0qvPM+8t8)gPx%Vhbm!z2d*MXAid4`0S7b=t23V&-g9dupzLjis+Y7*ux0epS zS@pdvbJvI{J{JG0?TWcRZ-RAUPi>DpX^J#C;W1E~DDXICN-Dj|nmyynVe=*z=-T?Y zcgL=arJPLdGo(lj5WLqXgq@4x?vfCxI(f2T2*%EfN35dX(mq>H_UrvIV0*6y*JkX| zD;`9g|M2)nyv^0>$O0k5Hc-nDq1ZylgdFNs9e4Ss*VTA}P*p8q^|{>BYi>F)G!5H* zk6kahvk$qE(l8MNCSmg6sCbvx?H{3^Qb(Tfy34uqExH@~Ffy*Zrfk#JFy@j{a!k0% zDV&IS_h#5(;?vOAFz6iNwb%B99w;Fx?uuryQEYRwYl@_UpeCY!jX8!t2Z+F@8f@#@ z-vsA}J70P6s_5REXO4!UWF3nqf!3`OeH9%ZjHOhxe`~Et@_&ZgP~~^vy0u~t)ed!a z*V>0(pr@tt$rkVZid@5Gx9ud-4rE;XE%hx@LkeqCprpOvj1LExOJpAk%bb2b9Qn@0`b>9j3^u>GQVIPSAuiH)SLz7=djIaL=e-(%7 z`s94w#DD9Xn};sM_J>Q>LSo-|*G4fnyvkNy_e^!mR}&Jl;07_i)v{H)FF_Ms_wi7& zGZFYneVDiM0L$pNI>hN@&PGi3 zNchlA-mo$K>79MxZf z&ab&08|XDIB?j6CVJ3}}iCVL@fXvPGOx85-7Qb9Coh+^P!0B4soO?ovl9d1PdPB4h z)Tw>PMhOzS7?kh(^<>4JMxi!l{#CG+(~PXM-|fzx#HL}yt&YBM*SGA3l!$N@i8yZ~ ztxGi2=+N49k%JDev1--r8xp5WGMKmSz3Sh za7)L_qcK1~)@U)s>qnr7BuCI$sBeXIpg{as%|`iC{u0ULqU?3I6mjP&w}!j7G@U(9 z<$q}1N(bezRW=M~UdRZ>y@o{oO=|!oX3< z4zm{lCyCC(l~`x93%8!HTqVs#-J63O@+^~KRrKrPJrDXV9=cm-FJE3*OQF-!dc#^6 z>G_dsHQta(K11jU_j`_`?$}S`KgH(DWS^b@-H0pc=N}}b`QQ#Kc@Np@!nnOYJm>%0k*irv^TV$ ze$>bV%^2F%6z)Be_{A_Ab178yaSGgi>^`^23H6|_uY-i7(4+feR~f#9Z`_#(&Hv^* z@%=IX)w^9L)Kq#&FBoVd@OnPe`86rp9l`OV?P11BTifD`-g=+%8Aub~2g%QpYUQbh z>uo?Q9D<`%B`O(-Zy?z_X0wCfmKH#^4{LJvju7be2~==jV`NhQT&WWnC*1tDRXCQ9 z(LfDdWAk{Gr>jn6(3NTEgcRVAjx3cYL69rv*5XgHXNbp@>l^lP^_HXPA?xYe1lOAC z3BT>>pLgr)_4abDsw4w^buf8b6_B!pZUGuTDxR`(5%vD$&eS8Pl8yFY@t`Uvyi^_X z?scj_^Jp94mG6n_7{mrlI95H7DP+kyZ=U$PB|E-q{mYx-pE2|a5q7r%^&%c@M&@vb z89sixVpGI;FJzb(tD^8L@9VPOuQ|5G=N>00(%|q4hg19NiG9$Eb0PaYxrS(lC(oJd zHh`T|HSjOW>;wrU*`bK$yo{kt=tzi~L%tV7dt0RYX69N#&LWl(O;Ql|hxa7U=!?J9 zqA_pi;VJnh!DdKio9wsi=eYOd=;wYz`=>jZlt!Sj&T(T63{X1rO&I=bFvEWspyJEC zbR%w5R+y8ox-4sF7#wo{(v2ujm79$BGpq3+2NicOwL{LIN2&;JZArXn?C43zT`~WVTk35+#MV{cehLqowJ`3Ww;4hgSBP^ zt(0zZ@8F3G8nreL!67OP4!;Mhy`4w{4-G2sFE`{Siemp34hPNVuo_zDN-$9T*?DE0 z&EyXAbqsMK>z?3l-G}xI73TiU#OKNy{JbmeQh?g-SJa_Xm2m*KUw@m$iz>50YXwaQ-*{eYs9fk}~{Ed5g zZ`wwrRYA??nIj@ z-#sQe{qRV0-8PYys$JkNY^fztQ0EGt8e(gz(BEp?Zr#9R1*sN1b!5%q_Dn*qL?kWMvfy3&zRPKN zx&7VwhNGb5e5%j~;S6K-%)BaaGKmm$LaccD%-|BW#)DU5Uo=KlkGM7EKpAou9|q<# zg?13GevzFlF)CUG5BO{FGccH@_I`g7!ww7`y8#a4r({rvzWM)8|DUVi>LZ#nzEiFT zZ2u1k!#}veKL~@u-Me@H8(@(6ugw1$V37UKBLDvuz(5i9-vEPxg7p76!0^993jF^Q zr0_`d#eV>WOaB8SM- zc@Kb(a!p#Qf9c<^Q4Unp7iehd=ov0vVx%yjt^%m3E?l6dxj;+%&t<5BDCGc}YqTu4 zq&4VRpV`t2cwCoxoA{MM@L^3S+w*a}kgT0&=*3Iy95*Nyw20_!F>yJ01;u;!l{6n| zY3u0fJvMsr(%8h*%-r6=(aG5b?CRz1F@DB zdQs~0KWb5q|IzG!(d!yTFKQZ^3pDiq=tV{C_mAS&XlQRq)3Ingqqp^76_9z$aQ$K8 z*P70Yg0j!?Y<8aGm)M2mu)>6YRQp%6|2@S*|Bp2LZ^izHUh@D=0QJ9gftvaP?S%^$ zXz6GvM8`n)4>4R~_?IsI7hV3BuKYvH|4ruIIzo@+hac_AmM2I>2qFpDiMW^kf97e& zGmejWf+x_u!85pvuyV=1B0rDoPpw^jlR9QN;sj7}i$$r!n@wbE!FG~kpieo-mLT|S z(#>h2MG?pIOv+7CMFR(Xd=JH?C)gOuq&E7b2GSdK$DzJ4Lu`+|f3P)l4+A_0a6y!g zVkOTmFOgppA_{$$uaVqU9dZ}{9xR%ZazJJd@q6Q_xj%F3h*S zZc~B>$*N1Wt0qgz#KBbrRnaA@C!?2sEzh~R`P92C1>){c>l0=*cbLr9-o z#%Qv2GsK|s3>7Kp+njq7EeY5;k_Q+|iRHt0gDjQ@kI^RZy>HP_OH_=Omzx42m^Q?% zh6`NvmG!`?n8BbBKc!`>`Nf5K^Spt2A{Rsgra{=t!L?)f`gt#nw|J~5+i=ZARjSxJ z2bFCe7e&~oAh%PH7WeMB=Mva{m@aA*rSJ0}}CTha-y_~h3*sJg+owy7ysY16qmNRng`M2h%S7scpQ-GJl`N#T4G zKp#$qt?{CxbS-ELHTA>PJl3b%32s-(=kVof$cfYO2_T-=8N0xi7#qblo6`M`I zHMn~!ettns#JzR(S{?z+uu!^F^V_yTW!zq$zg;MwVi94v$1uiPfROXqMSM%ZFqhH|jIupPa=haUL9fc}b73r)mgawj)ef-bK zMbxmS-sGRz3Sdeq?KFB%0vtTx%=7s-2D7ku4gl+kVs=DMgaKY^aww8)!H60?+zIp$ zx6yOFv!veCJS`7yQHK>SKdGoZ4SSLH_(Y)q>N|${OB|&qZ7!LL5S=; z2e80%u$ArDb^utGZ#100Tr9; zHmGlU3;;*l!D{k&j;?)rj0;BeT>)X{U9bctYT{zn6sKFQCH(6-p!Rv-r@dU$4CPki zv%V4Bx(%}5u0clc^0j{1ph|eK0~&@r2e56w-U?x=KWy|E^1=j)UgfL+y z_7Die_4ky;S#+t>Ldk-5M;Yu@I3D{KX2@A%vwhL_$qnOFo+n`9IL1n2O@B?#dj@Yc z7y><^1UQK7Hbcl0a%9rgzu%ii3c`KF^BUG4K z&JzOvI8dZv+P)SQVI{UJx6x#X++rUFoJdxz+Zejk*eowu_RHP%$U&;^Rbi#BlQeM! zS$#F~=YSaBUs6|UoD~YLlmuZLo#W->G3X;hJ@xumC;FzH&NR=g>U6l|3+jtPxX%F( z;gQ(NoO8fMogVd@B+Xt3*1rW7=v>$Cd@eV+r{#f(3JnvMmFa%h zkHbzgxK#BY$6wsruXO+Or!n5JRsnr-m%jZM2G%_qqqJ7g9@`g&vq<3CI^s0MkcDxx zK1@o3Itd2ZbI4Yu$w9q>gFE$|LAA7j&&n2;Fd+9j1OC4WT{+hf(s-i|4r{(i6(~-X zy(2k^uTLdLBaQ1B*rApOV>o_EojG&VSw$4~d z-GyePdHsmNNBF8x6=e^*jCU6Ub0yPqXGtMXT@FJoTj~h9vNr`CuA!!qmJQm#J_;GN zn`fH6>s$BuF16iTB*nEcBg_b>qUbY$t$=0?=_xW-kXJc4m~SdC^C|TCeI%WkvwpI3 z5RHiiHVujziYY`PebL+e^tTFwhxxeO$bR^;E~SwY-=>kYlI^?ZrgK|Oy=(1ZNRx(8 z2B}+9P!`5*2-kr3d@;~%Z!aXsQED#lTj;L4ls!?|FjnmnDH)&Ksdf(^ky!3>MxO?X zpMQ=|Y5VF^65;}acopi1MpWFn6}`?XJ2%@abPe=Vcs<9<>m=M_esG)Fn)Hky+gY#1 zM_BF-4E!aDxVECIhO;3(@)^|rosUmxZCzjq%GiI=e7X~C*AMYv`zbl3|*)q9H`^Pd4j75+Zw0EWE+C=sNnf!~kvVKVrE z*?tD{-$kA*%X6;z+;`MBtnMOaxFv#4`u^~}+sB7lGnpwB%CF}l zeGengL&)s@D?&KB!9y_;X+;TWg>5|RpDLP4?Gr<&%ROgI(PXs>%JAnd9u*9jjR34IS4W0sGus(hD>)T4l8l%X2kHtBtmY08dXzwKS6upW3@ zu+_>y@6H?L<#22HraFA^Tibki5-fUr%iIwi#+DN$sn!oGcJU1eIL*-A*>k8y8c$Wp zGhM+5O}U)|kT+q%^L~MjLL889OY->0PF0&HEi(vF#csk%*0svrKV_$)-KjN8dPSoO zSr9=;hCP?^O?k%lYl4Q?gDrh4)(52KDkQC{R8IPu&H>$we5|Cfm9+RXKK<3I8>D)W z!h8=0XZ+XV&4gmtnGAF?Pwq169Y_5*xTa!=LqP_(HZzNZRm*e=k0vWhV6#kgQF!1& z`why*uiz%8s`v7YQ1u?p{Bf%#NQ_Ux95@L)qbB6yURjQtLc=TVr>0pB_VtY$>+30X z0%v=msz|IFC+p!75_m5Y4BwK)99KiQ;db}t$kJ-4a!o(pm@1pWLP?ZEW_cPYbq*?E z(o`Ss8){L7d{SC(JR>tk@*r?@b0|%tmU~ApY7}mpBrm$84)T7L&xsNtD4Cj)f)CxK zeD0eSeHbl7B`txKc4EuCpr>Cs2R5v_$4P0r1ks)>&F(f8LPNM*7W={Pm+37a_Dug` zg+>patiElCvX~rY&;yw~9k&3rEpk~}EzSdxe)Yuh1-;WOYefR<*;SV|E-=RDBO@VT zP!lCjzPH~Z&C?&*Hd)j!lIGp;y5P6rY2LMOD4DHmL4{H~W|Ek|R+%@l2%l0N zLIC6(zLRk4c_bB3#=K zuFxnj3raR>Y@4(_;Yk^~>))W*?i$J#`e+h*CrD=g@ic-5uY(265Dg>AW~;k=J-^LJ z@$*_ujgc>>q$=!^h(zV)R^s#B-E%t$gfYmwzKIajxoPu5*$Sz6MXBJ`AA-Y8R_8+PPF?}X;Qyoa$oK}vcgljg5!CT7+IgU z)k|PrU#+ZS-ezt~4Em~8c1P(1O&s+7SP%4x*W`Z13&D?^5W{#G2*AiN z+i5jEgpwL-WT!bOEpj!%eELraTesUxMPRu`UU!HW#A_W%Y>MmgO`U-fW6U@)H`9Eo zvPjm<5l1;!uv2RME+1U8MW%ME1jdFkpb?3M3oc4*<;T`v^JgJVk`2Vhm9X#nHxHQ} z9&G=pacD}ND`zLJ1;^ow;#$BszBp_3F%?zpso`^i>^!)sJYc7c;$PhOGgu-=Y&2vx zgU27-T$@~*%FT4lnyk!m8W5DV>Gy2%awmCp`ze{ko(bvaS}iSa^Rk6PWwGagH%=|n z@)lg|_{G#j$AX1n@J2V3&gUX7)u!RWVDrEfv@&>@(Rzq#81afEg-`2KV$awc7$LfFE^prq-P{7m-4{mR8Nb4H0G`jE)q;X)Zu0 zqbQNZY8%AW={XAhDVY`MCG_;rnsrjerz&_nhJb40UbF%?Ks`J&9%Zy8<3JWqCl9ql zkMx@2H%9*%_|U7mfz1|RcefQ}voS>CBuC!u_Jtq2e+;8CoO^7jtPsKYXt-J+GaZEe zc)hH2QL}u&uf$>=Z0^{5x`>qdk+GItKobz0jb=RiG`VC@GqA1&Rr)K{9c12NX^oI3 zi3K{YCU9>rW0Znw#`b{7eWq?)5Z{w2*f{|8s#ampuGZ~rC=ApVitgR3^d3xaA$|Fs zv9T9Nzj73VWWI`cI-w>7x866hh_Qkh;!WbD%AqP=9`;3xI>iR{C%hOdhf>VKUv12> z{wH_GXk8YqlP7nFe}}wBNZaFQ`j+G;d`54WflY#M*R&zWd@`$cRrQA^5b8|34KXO5 z=M(qjt+-xH71mGgwkzqZecu;!La6-vR{elMV-jE`0YYBC|(HyY+sHDm{3tX zQHDo8nsj%Us}PygOL*DPKE?iHedYy7Wbk=&24>VK-b`MdlrmamV{I^|^3)XMBZ3upXpO-$*@U~16g!5MNQa&E08%C=NFZ>jF~;K!}|K6lLnKVU0TfzjryPp8uqjd z2bRA_)z&uMY(M7U{jtadMi);~SZ_Q1TrfL4T3=Oxp~dP8SgepNmw`UmDKpor0^?PF z_lV87H% zd*{gD`iT2KpxT5#i;wj|pdVX<&PwIaJLQMpeQ-@vl&-f4|2gpoc9E=)uPi3!Tp~+h zC2k-LQ7bLPgneG&^Nrg*<@9)l(S~ttG;(<@AB_nf6m%XSkW+D3cGJB4#ooy z#GWPKSD)keiU?l%1Xz@c7O>zjA_1DarF58@fNjoHHPp|*?9Ekq&mBcflF><4#(^$U z#_TqKT-Fmhw7V%S^{&MXueNS;*)`8#48_bbc>TCC1)IryY4h0CMHZP+uJn;9NXXe| ztyC?y0?M1?!KDK&2LADT#hVitLXd>jlOnMRQ7U&DvM)EZ;T5Wi7T~_B^p>ZmyN88u z3&D^gYv68tW{{T`gtj3k$aOp@o(q+{-e`PibDfmg+r~TY6T;&{vAoEjRb?inAH=C; z+VS<86O-8r`8XwgNS1_8d>YheJwGm_0(|tk*DbEiLbt%U`8Vtu;t9#OokddZE<%?O z%VDnY75K9j?$_Or5!4^iqJeGB(*E6*I-`VymdYb18*=P{!-u$DYAqIw_}>qYYkt9( ziB<$tQX@xy!rVFFSsgz79FVZuREG?AnD(xp#!hq2TQ;4|I@V+#O!&{7*qYlYTm;r{ z)^v=6crgIn87H`%nd}i%Kn9%_s?e5@tgYN7hA(Yfwkmpwg-9_kF4$Pl&Tfyp1n_KanZu$@y6ZU< z5(i=6La@i6T3WG=W1-oi5HTP)^NyzuQ7S#9vc&)`l~4`WG)lwKN9~fCv8{A}MJcO9 zMrPkynoJL^RsBfRCmx19t?b73_W(Ki)vu)K zPxwGN*C!uOp?+1}A?su3q4s(fR_teeUHfL!Rc?~z7OhU}gC^1Izn{i$oXEiQznQhX zCsUq0>_U-5aXFk2zYM#hBSox>q_! zuj(N=nl|QgYwDX1DxKRWdm`wz%cLX* zB$rh%)Wjg8|HTAZ5M1oql+xRi&5eAo1C^^*sj4y86+vj~bFyD%jOX^#sS zeWPMg4zC{t5on9L4Vl3KqY+5;i>8}?OfW=)ViLyY7prx7eVhaO^#e2|0N(HKsYB=w z{uwE#ZOfdCWZ42sm(|B`AcAML6|gNsYVSOT#wLT!cLG463-w`OwvWfXS#2p7;Boei z4G|-Ondx&y@obISuD`3;H-m3YwXhcTe_5D?952W1RH7K)V&|J@DXWU9$)`DXO|%i| zZkR{4W|UPDfllGrW+`CO`W^2_-4{ylr$7WxT(X1iReZ{S} zPW$WvQQLa{pvOPez6AJPrd);oNECPfyRZgC_$+~_c}GAxJi@=rOOXccDKm-G@_ zg;5p~?~#iS@9iKmf0a|_W6}DiRs36dO{x)doa(ZNspMv05IgV(A$UPTT`&4Y3<%!saS}Cbs(F_`0%;jHi)??>8RQ zBga&ARD)JGGg%1A#id*Q*>-rBzU z4`%CzwTf9#sR1<}f@s$X(8h9{%!E5RRmA#*g`gz=WZCFW-9n=*)U$)=pP5-Nt&2wP z1PFO9{_@nP-8~`(wPS8Ir!s<(^#EMkjhy$vd}9^vD#ADhwYydjmTiWCe6#s(T>Irs zPrDy`N8J~>A>5RnK}^E*s^tW!rJsJmo0{oJB~4hK?8m8NB?-$3Y7*pU(`4xoEHmAf z^^KN@T|4~nQsk;GHi<`O`W@zH`&^TSmc43C^Iw!jL7HQ6k1O31d@y@l)15;PU1jn;Dml{RP+46OhIuKO3 z8{gl3jIM7p$j)Z1$A|wU*3T$e6`TVi4^^dfAOkxnu2u;>Y1vJX-})BN<22H}-7QJ0 zcDk@W6~p*}Sp@&*HQc1z6K-sC)$|wMRo{m$suC`8(A76GdJ?=2mz0N`WU)`o564e1 zDV!0^mA~IvLymv7m zGK<=-!r7L^Egn>MrZ6!ZR*+(j30$YFu5W;}C+o(}Zg&zS`zXr+elrTgWh4*8ODHfW zlpJS&#!?pdI7qRFGa?Ub=7hG`!K9{m8m|u*fgK%(C{FP3b+mDIJ3$I38iCr6Y>KEL z80zAWB2;Yc?Z0d_Fte@ger_86lWMHkCOqN8Bgu91P`ft9Fn)5E_*=t|%M{6^{t>-{ zUg6g1K2_}GyAD+0D=dmtvA}LuJbvb1?wZk>9UkOS?$RbcVLvIS41KP1HXY9OtiGl` z!Lh5{&j?dzIj`gARk-m1PsT8%y}hF*jj$u^rR9?!;4#CHqD^Acf*-ocxEZG_+8k*f zBtyJ?$9WniZ?@;yuy}{lnFs;)m`K4~8sxWmxhC?)57n>04NdS%DFmbL|8y*+WcD{ytiirN;1UVnQVUf9n?7=N*zjGrrSjbOn-(*h*Z%Jn@|?q0l86w17)tmRqZTZ^&j$rUD*Z(6`m$RMNVFzQMQGpy{ZL$Db7%+mmhB=SSS?+_Dlm6*MZ#7$qySUNfp?A z77|1Yy}K_ztOn!D($=rRjk3(ynEx_vxHhP|6c+LSZzW2kVX7k@C2G%V1B z3iNl3vw{^i>t*a&%vGMKbHLM#ny1NDI3pa-^Nrr(rj~1l_QjGp9z`U>=VtrvD-n#d zK%m^f4wU0b($@CVrqF|l-vhrJEJB>;Twn1i{ld@{t)5zR^IdK+!_iH)ahh#6EHT&r z_R-A*)l45U6wFWtU}o)FiY2G01Vt6JR_2s|kB*G1fD4J8G_PAf1_sBIN}OLcnY{Pr zVzdGk5=JA-)AaE|L>ai#Ip8_I=-v8M)z69Y${m$nCMBzKKHKQ&D&zja2Fm2Gm){!M zO|~iB4B=5R!G^Gsnb9jX1no@RDfg!Dr0F(O@pc>ExLdMuEyz~Ngc8p4gxk;8Xu;L@ zwX5&A_K{3JnvrOzS&iQa^;QNlKMuLD8Mi`le72MtqAOZVgqLFkC;yxTqSHWe$hB;- za#&h7aYo&tdFoG<+HFDP1jZ?OC>G$yOEVKJqIdEV?kkPUw4nq74Bz#+ztK2dX`(k( znV)E9vqRaQFxQ5ZwvhL?+B8f3W%hU$bXRgWA0o~xgkNu63As+PTYWk19Y^02tlC^N zO}6zfaK&^}_R^)eLz_-k;z36G=@r0SWMisgD0+S5Sj>I599R@>RzWe(QxIaLLBeEb z06!t8kNLvZ0G9a*yy$7@(M$c?sEcx9h0~~XpKWyH<0$)F+sj=gGml9V_?X5^}MTr$vZ`G#rU32j1^#!lT8Zf&p zMqjrkv+sJ~5PR5-9^@cSR|@^$JU1X83$mq%m5z6@y&4Rk$J;M{+YVhZfB zVM?9v$z~)d4SSU+zD?HumZMFeD*#_fr?_A|+BYkl&!&>}H}@-jEVgY~&5$;7m&%8z zHiFggYH@hq{mu-vx8U-v5*#r5+T;%65t<{TJ(RM}%_&vWUoh%l-zKQ9jbJOZn38%j4>;#AVl~;18C)US$jQm56hI!i2I5T0YSs zc~z>1nahGqOE}9UB@dbRK19MCiAwUW zkKcX~zlRn(EE_OCM&Gux5F?{1tGcuK2{H&{G}$Vsh!kTrDL4G^;7BK|lc`|Pzu4ti zZ&GNYa(!B;*<#mP^?Bcj=~}c!S${ix&pv+XrrfkEz&W->Dt{|m5#P|W|6YMtl%W1L zZlnrFUq9)itD=WJ)qFlxNLi}s0!qECM<(pdNIyU~c@w~c#LYuFQy94?)FS&QS!Jxh zM~V%+eBmRktO)py(A`(Dn&{mxUV)xfP7ygs%dH3jQ1C5q2iFi$Nkf)+x%x*lXDc* z%{KnG-c$&3Z)R69TPrS~*`6xQU`mp`Z`EkChh$UFlUiLm(+Ja*g{b)^i;#{nPg zVrdPZd5;Q`3}UfT*nX~uf-&~5gTa?yFSs5T7WvcV;-y20nAj22(8mE zZS*AdGZy`JE=aKQ1ftcV8E&fzphyZ4k!bsPW)=}Nv%;@D09tdxoQO*9@HJ5 zmz)oehulWz^nF8jcmFmeEw)Vt<&x6OCdqmgo5XssN5STqK;y(APsx`{t%2>V#6)o%_UzE99qS^8i zJp1UyBuvs6ayTu|#7+)zapiQSJgj9h2H{{=cWRsG{RO2=b!)#vSOy4+{YqveM?$9N zlnMyPcNzY8uy-8puB^)U0xEKF&u5f*09X)F8ea?yR&C}O6Y=!d>`nrvWUb~fpw42~&USQvyB zSPqCe#)G6o`~_zH_}CE^1TWoY0;N(=IpNiiGjT-@rN{`m)+@*B6I@*1ckQNvCv5|{ zGhWWCv;`NaB$;+#BC2GTF~M6*s`(!-q~ja9mbnpDXg7qdR{=yxJyb;pyK8uMZBV%p zAyF_#syT#;>_P7Bx$U&9H#EkARFC#wwS`wY{$CWGeLU0q`^P&+<#Z%P=nyL56thzq z9k)l4w6t#BY-W|*%$l8I3}fG;B81|w;;`gCX4s9~F84>0Y)m#bn^JDZR?K2{exKjN z|NF=L^L}5~^?JTuS8FR2vlR_0oLEvF17v_K5kr6ZksbCp$jqm($2Z=tpuIMKH>)O+ zXjbc^|AwZ>FA1wvOVm6}Xrk-y20YZ!@Y&=eJh$J+Iw0XfQ!aB8bwwmM>sBAgy_VV& zf1&UNtgIh*qk=gkYzT>~#u(a46WBHZ?oLYMy% z9#Ep?9=!$Q_amLriiaxWFtjA|cf7WJ_{>Ps(0^O>jvV4jUvRN?w>Azd6qJw+8&K5Y zKc@uVioydr;mSKz#IwrJ5|>bgqQBSmC%MnxWCQi#X2wQ{B6YKBym7AOD*RUNygcpZ zNaey37wR&TJVStQmQIU zGG#i`TWlwHLfi9N&KHm#ylomx;u-KgGZ&>%vDI;gH?3G>i=VQNj1_=#FZ^KEwX9FP zCCe|DSKu;?NNGv7#FL|>#+xW<`(-WTR_e_@1!ug+D73KjyL7J{GPNQDQEOa^1PHH6 zn^C@e`Sc$5@-8xW+{DEU(SYVmuvheDWENk>FzOE!R;Ujy)Fmmha(xy8!#Q z%$BmPpiK^a8Ta3oo5%*?_?r$5B}i^QP41$ekchg5w-fHYZV_ceeu)+}(%}P%Vm6cT z(%Q$iD`c#k(bCKa4~0H8Pxm)3o&_rlAXAxL{n%Apm)lOXA6vs*xkG+vf*2+#oH`>{ zxmSMS7kp`9HDp?ZwihoUjpPNZ6>-5VG(HWexLt1DlfHTtUhwh9{ywvZ00BYo7dt8O7eG2zao z3M*cRiLyo8tk2X$W{@Wya_ABotH){tvLjvD9USaNuIzMq z^YS|_!ExR=a#K?>oOzh^u!*uKOtw@K9Q?^6q-H(|lFSF$j(4Hv!1$f~K>Mz*1ibr8 zcl`$$=bB>4yg`$<7gB62b4NVJ<&YryY97RRX-Y7PD#AA-C7O393xA!V(v^|5!U zyqeG2QOyG0OS!Kfch{i{?#DDawmI(ilTV(j&<+i#dx8Dk_BuejmRT;#D{*{viX)$ zZ~H+^xR@bbDT19*`bbW;9hu%V8tyA~AEg?&obcdAQ|-OYN0~z;&(T|~j4Rg)j4>wA ztFKtl=zM?b2*`~?%si(!y9Ls*Uv-?bN}1rT1<8Ed!(`okLL<3rC9P-ty}$z__8-4W z_GY&>bFPOmav&`3a(=&bC`xhozb$97)&^M{r{tyvN*xI}wI!=W-2U_MhH*DVm*{p! zKf4>NDfQ!Ej3Ak83M6l-p%z_0TtBf{d6_)i+?H1!PU{%>xf8-{bto4JiLRu3rbZk) zC|gHK={Mbm;k7r*E%M&-#&z(O!hm@Z#t67hwpEnGgxc*d-IC;OifMrS`Nn51PSh`shISRQg}<` z+C_s1sL^0`zy$v_?GtixaetAWa=TYpWw#10>|j+2@M4r?hD6ogcATHD1m{ z(W4cOgv#DWb{iOa`NX`@7@*zE{{~I4Q)nedYnnkJ7K#%D>?D!Qy5Q&G=R(1HnJ!X92KBVrvOPC+04WeE2Idx@N z50lR;fT97WE!O0tAxjjoac$@k`F&i>Z;1uy%FM{+CaHS25B4{Ju}{xs}!;`>%42i`ndz_={P#eGo*Ma6^m)D3R_dzn+3ohbGfQ>fzD4w%?8V zEfiF6I^M*CWjI@lBtJ zMLK*!vb}7Yag#$R~6J-(PhzQ(YfT zGL6u-dI&-7Qrpmbrm>W>1U4GRmB0z9u#xi7+aGhgPayq0EC$g@Cak||Mc99ITIQ`! zCqtRs&rJw-8`b9wHXwo;DM>T=Oc`D53}1WnJm(X&k}Hj@?HH>Vpq_uj6%5tF%{}9S zE%H7M)jj+3Tc^uE3$gnf&^%$_AEr(-MS8D0oGH*v@=Eqr%=+ht&gQbEuMp5qQ^Qea z=C(+2y==2m07MXlqK;b+sCWr3gU1KUnF;~cjjarjAJ^HTocVVYOT1P&i6UWw7DyBE zjsR9v6TbepE<)7tSE;~GLE^6`Rr!9~xX|~zs=Qv{R(@ zzYP^j>ot717>Pd_$t=$f3a*4B0>9@>P7Va4T~=Qr?2Hnr5A&UV6>OXn4DxImlQfh| zZu$WNg&M`zCOa>XAg}-AHG00a0Opyg%wr%TL$unfHsfyM<|==q|X%?vXlN8*$w?sXT|~ zaZ!dbYsGt**7{vR{K)3~KI^BS5?qeea)y}J&jfFu)$tH4ig{PDof0bn>k*@<8yCXcHOHyd$`fR&APlBQ*~~{dp~Ort324r;vcBhI+9$3JD^$T zmaeuUy2#8P5U+`|e8` zOt&Vh^>gEJQjKb5X?K#DOjv}Unc6-RSa@zcHI;@pPM8Wkx!mUTCva^ql^&J{4(PKA zE9Y1P2LH%|V4D?#7F^^M9JGmN?{+mOIC!ySP4`qzAv@!P+AI0W)@!5AFhwS&J9+8d z^2X6w>%w;5wX@A_9rWS^d>$m@Z^ShQBg;B`IcZq;tJz-H-Sck7D)5Yn)pRCsATvmc zK_?9IoY1c$%*f~8j{^Oxxo3ub-%<0m zn{kA2_h!+`RaDwq3$vrBvVW&zUM)7HZb}%$-#1~$JNxW**y!RCSjs$E4v-(^Yd`3!u@W|3KC}t27yYbBGbDOtV=P#52$Qdp7`a^D_e%Tno*U9kfjlb|>y(!I( zUe*aIE68V>o}RGt(YK|Aj5CV1P97xh)5N!4iHfK9Rs8kS z;rfo8IOY$B7ow)k_PG8&g&B4w8&InVq|T&YIjc|~tswTur!lGOQdnh?0c=-*6W_Mv zr8oMsaB%x`e)8mIr8Ktwcz0Bk0=h-#0N3?|%0{k}g$L#Xk{bH1%DtS7YEm8uu?JcC)puU21I_nD`6uUwIMy*5i-H_SLFza}8O z6rCG|?4!L=hKeJXAcBl+$0dpNYStLdG8>vTva0pVP&cv@^7aWDyMHc3JI<84VIv~+ z_p*cso8;}ndnEu(KBBr-Gw0V5foUmMR9d8zhUlV@ffh{i_uG zeK+G^oO!X2eV=&gV}e2YzAcWp^!3lZ%PC0!eDs{)sIxO%y_dnm4^kE}f`dt-iFMQa z<3kmJ3|&6o8q0y4uMI%fQeEv`GJLPAnar<|Uct(XlhG6@@1X+$TrAGY{oBSYa z4pAd&7v+w41Cx0H+WI|z*2J;1Zh{8U!Xy-(4cSysYsv8vtB;Fpku_j+xU}AlMxse8{+c( zp1ttTZD0*$hQz8Xp-=fgyxGAOKzW%rLOvirRJowV*yARc%UOtahf^aseAUY4xx}MR zt-mV}we#;gZ7I7mZpZ4nqh7D~0UL6cJOA6Ve|w=z-!n`i6;jduav=@>LWH^0UniI! zXBqzK6~ltxroU-K#rd)_q6_h77pWk-8b&T`<_G1#%<`5Y@!oL=$W0Pi=$cE4l1SaU zj2I_2Q+$4o;oe}Scf-^?`Zro-t9Ot+`Cma2ouL(FO12#DqH2hfo{5L`zo2v&EM)+d z=r$(Biyk(*--#6?7DB@zkhpTjSxBLE;l4vx*$wsS3Z!eIsLlWRl*zx_{@b#h5GBzb z?$OaAKzl{{saeiucWUQ>MWkW{u&sU7lM~Y;L6DPerJz9xl!h^ia z(OfS_W_5i#^3*kQmC}2|C4#NfhDO(^ph}7@Rs2!r%dB#bm1rd(UFY8|t@$%}R%6s! zT6f%Keqg`JCxIwniCK&Y=Oc<*v8tI1avgw#rWmMPrm21xd681*Q^kizXL*ZQywAZ~ zh27+DfP{awiRcrYXm1*jX9%65|^i zq_52CFwYI*Che}ENic%~EW{a8h4C{iaNQ7K*!uqOSa*c}ALw&$$S1sJU8v>#xpsPd z1Y%v(h08Rm8e{5YxkBEnpmhWtn7o(`BB!llFcUV>w4a2)<~(1&7(R+0gvZ%SZ)RNF z&|F%XFgjw9V)2zb>}pNu?*9$ll2jt0=!SEV^L~%(o@v;Y&ww@OgRFSh8$jDi5nk2v z*4s-;gCdS4CQF>ow$=RW+KID1wSI349<_6o{ooFP)T7=vqC!T=4ZDOmg|SEphpp5k z-VJRGZ5ts%(A$_rrSRcxyc&$lgNm}RHiog7n8s5hf${>s-v72F$D(wx*d#OeurlhC z_o0tJ2&d-NW?Vma^?Pn3ES)3)u*B#LC>( zUqn<4IfL(DUCX01$p94ccR3mRe^_l_96BqL|4 z_Y=+KyB^4jKO>R4TGhTFSL=!t$++)CS`^pdnu{jn_-RB=KY?1AdY z^WHlD?u47s{_)pR<1tL{_V&|-ytvOQyX#QDTa{$MGgf7ν-;sByG_+2~Q!%;};z$(KPz!aT$3454?Zb?@p2CXMNY|F^FzQ#X%cPPrWYE9cI@Ll5;3(=xjc25FZ zl#0PW1b%ruZ@^#8N~1QVPmkJ0lhxxbDX2y35cb&Zko*OhmeX8*%@LsdP6x>4v1e^x zHc)A5H@R6KDY@8HMoRhIq0GQb0&ZXU+gYOq0jWI_hP}zJ$QK@HViahhea}|5Y+n~@ z#BO_g(S4(jaQ%C~gjHHafW1y~l9lJv#F^k8UTh3qN%5DBWxgFZima|f4pT9FW^MeV z%fqpt`Vn30@Rfz$aEs)GW_7RPZ2n->DdoKc8RC@8HB~aW=Y_JhkFuMwkWmBD{W|MZ zL(=&m+)VeeTU04_SNU zY8+HoBH1^?7|p9C9>bN_Z6>{0mM*wVHkDAN9o;WPHZgi9p4qN!oMGi~xWXiDPcbRA zrLKlb_$E0|(OW#kHD}4U6_=LH!;zdIOSH z`sA$5GAxY<+i+IaNZ6T7Y!c}g+dTVraUu1W=%+Yrbk(Abq*?uLW7}*c{ansBKv=H3 zUhD)|6vler$~-h+g&U_;9`fvTe2Owr7T@0uw=WcHJ6dwYuK@R@##LEsfG3r>wq_KM z2%K~L+cD|`fG+}!45`2D>m+3AZ(tFh<}t~#J}T+`9RlTavU;P(k(SZst}p?gG0fqk zWB4H}zPVov>8H|S zizRx>fM-NEV*OgbBCEWzTVk9Uzb1|U8W?dG-4aMg`1dsN-i>s|Ms20sYHDpt=X(Z? zcTqIk@Z#kRM^U#(Yu@NghWN-bXXAXwsB~qa+Q!+tY(7WyA>16=IBK4d;ql@`#M#_g zl56Z(6GBM8{kcu2lK8t^$aZ1(nCYTTO>=xSVYz2`#2vv^{O}l>I6;7g!P&`6>OSld zET~}6s{uhSj=t5H1;K3d^^pOQF?Jd5h%X$2bq-gPh$tgUJEIKv{TfPK-08FiC;)f)p zpMxyOeWM$IMB=K+z12kLe9Tf?QxV>yON-ALqiuo#kZXT}#nO z%;5H+%?wu(d>p#dB%-Y>ifKv7)ZE3UH0R$-7YcSguAczxGe2!+R+LAT&n0} zFuZqGtG5`@%=1isuQ9v`zQK$B?tDqENDMnS3JT?{7;mtpneb3qMi3GePLJ5*?Meqd zGVtXuJqfdkJu!)#9B^Tb-+oe?pP@G)0FSA7RtV)3rL zY_n11)lAYEf@G#oI2+my?7c9k|JHBtZc`(Yky+_X6tY9A34h^5sq8Iw8^eZ#&)xCu zV~X&mlV%VgkczIIiLR9 z68*y$%HA2dWmCyygqw-VymmfHX&&%ulJ3r4*Vl|K1}2%nAjGk8}kC zTX8!L9mw{A;QHp)V)Z^^XIsc~Lbie+Q>|zBAV?Xm78s@K&>~1|M6@5WE_87P>Itu8e^4F#WFF{$ zCh|jqtH5n4@{#Y#lUN%|4f>X^Qlvpg!Vk-Ua)~((iHpX0og?TJ1;2&ldP5_eR!A%Meyg&FJSFI+E0GAM@9)h9fJ3qK9ik zl?1XMoqSHN-{a7fD~i}NfoSZk_NqYkZZ;(*;zE8#lMPrby1-krNPl9qt)h9$w`HxN zGU6q9jmkk;6NfVoz!;_mZ6rkS^Jp0m6HPcPSpYMPe^5*b=Us#n-Y4cCk^Vo}{xS18 zw`1U7%ICMPhk&q`upTL<$JJO_giXX*rPtQ3?693?bDgd6r62y~r!bx48Qk6~NLFDL zI%170*oOF0%AIaxr<0C52}JHPS}!RJSBZASWkwH~yK{-wVv6#dtqK})$F+e5|FUM3 zSr(tINED^HF--CF-vUuE=RNbj;|7E1ZH)A<@W6p@UCQD90`0Xl=eX(i<7V34l07$@ z+;$sx@v)DRjO8&6b@8pk|H+QDoljt&mN7FQ6n&=9=8bkPL|K;JNoq3K<5}sxzo~cp zJcCud^u=3|P$-(51YGgkiJsFSvvW#M2?!j(S!(m1M!}fw{b>HAS`!9ngFN7&QB3GuK8dl$CPQrF&hL zOM^i&3(n+(PesJ#Ks3pv_irv7sm!t-G#tC?{mJ59?_N}EbFu%e4TI_$iJJz|T}hYC zTwp3Qt57Ou`EYOMnS~8hPmIu@*w`-70p6H>h7n-oa|jd6`pJTOc7Jl`8ctqc(T zODRW8bBl4c0y!MdixEsd$wM3*U;VYDH9O#~~*r=q&V z&S+?8b*ASrTxcs8e8!VTV)xWZ7|(T}%EWTw)iITY93>(3dC7fJhIb)8EWj(MC37+H zx8H|1Lhn=|`2i8W+)3Ey2@!I$c-zRLwAd=2|EXZLSW!a=p4Lor-H%Zp9E&Dx@p4GF zE5fDD; zo_{^qPML5HPufkil6?)=shykyqbh0Ft4U1JR+cKFx#qGlz=2C$6%1;hv2M3mg^Kn<-W6HA zy&L2Ywf{RKs>&-kNtjeh40z6bP=NG|l(2J)00s#%!(MmL*#TH{+GPCz&p1{t%RZ~D zmVmc@AmsKGJ93G!brFu21^n>%GYu!Yal6_f9p&Po8#l&4zf4nLzd3g#t`<#goUv*J zJV$4P8A0FF`a06;|F(Egm(>O=Gga4n&HqN$T19Cw>g^2K=ZCIZlYy!$Y=A`tAS8Ve zvKRjXGJ0&Yh_N6*x?P~axXy#X65fnLV+>|*mh3y3gwEQYwUzub>+>mPO{LP(Y04Zi z8f})Y$De~2Wi?Y#^Q`sD{bm0BRtoRGTt|l>jHY~PbW=>@lxI{_wU53!rz6_TQ2kbY zKpxxEiqrgWOIN)TftBQzpjko#QB*nln=;J72u!Z3D5;tZ&*wjQ5w}M6BWLJ;`b;@l zEEo)FPmXgPk?Ev6ACZ54?HJp=3A>tmR_>R&s+H;XM*}A`w@a#6aB2!ye4$2dHIu`7 z_5W$~pQH{Ay%3y;i`_kpb@`&1s~q(u0Qb@Q(8TrF@cB(BcJwmX+@EC<9vMi`rO*F9U)vQ)2?{wlTi%MvH)|$?T)SV z6D`G6V@hRS-!99di^&|rs~JAyt4Le9G*li_6IIb4g}zY{o-1Fdf!t4UquXY20_W^6 z4`af%D#r62$j3U%V7Ivg2IT!?elvL~4`zIP*VG2IoTzcvkD(@iUk|- zDw<^Ni*(&OS{p4GG`^2LwSaBn6F0Dq&<`ELDR93i%~7)pwJ+O@U>r1}oqDe@5cTjf z0pZTf4oK}O=q)q25)mJ`{;2qLhv!PmT@hRm_+5VDbdxmdbJ8*srArLrst=%r+REnJ z!lQC_vU)-@75{OBdK(Xl@^5L+zt+~s5LFJ`sq78b&!m3+FmJ!0WtUM^yBCTE#vtkF9qG!UN*(qvL>}&wgCZpfzxIM6i5LwovvoXjhoAtWH}!QiTDiT z`y6JX>-=f%b@do#c~NBF_toX-_N)Zv*MkKoCkg}&^8Q{*Dti|LbmqS;eeB&!(LfXV zg-jMlV?!4sT3D?LM8bkMw(SExrv?qlEE+4SH5@@)z#~G6=PKI21^=V*=$@pnTr#M$ zTXn)RF-p>rLB$w4Rw|NOU71)Q)`u#LP5wdiF5V1U#cR7_@2;EJ80OWjv=$U(&|CvSA!Z+J#mhZfWCS>r{5@83MGt z)-Rv?W<=n>tR7?J&9t-9o~J!)P1HKCye13obv2d;6=F%Wg`ku4aI=0+jeEsB{9MG7 z2qR|uyV94FB5RLrj||Bdp?E!)cX@j+3L=hs8x0@W?$)!MO44Pz_+6A8uAfA=L{@Gu zsq%VB@L9!=*%kh~Ok&dTC+(2X%lCl7@m=}0o+ihuSEdB!lV8$i&yECw8N^d^HBGWZ zs`W_wCkfr5R>j{vs6FTlryMJCH$$> zmj(Bj9ve}c&j~Hx<2P?%fH=J`3QaHT%3Mg@@xQ)6)Z?FHN{p}shn1KCfoeUJUCz+y!XaoMi*M%`S+dhm z*SnfP?sD)m_;cPdOi|IWkpZ1@eFyyor#pV7m>&nW4cc*eiNqAdnU(-t6Nao^P4K#y z{$|`m7R273@q$ovdps!h)L5MXhg46ObYvrhChC{o>V!= z*#CQ%Azn@u=@Eg%rW^Op*5`VUw=Fwem>*)W%yPhCJ`Y%sj2T{B8+*ZVLaavzKL2 z`CP;Etb{_WOXTu>-*qZ80~U3grAwP&?knZ4XX_A+WUkFKY@pd%jy}=`j+6WLal81q z)YX%;>pTe301Cx`A`>r64&I3~2Tk$gzIG~5N?H0R8ZbY=G&`@oZo#EQ_sA&S83<&s z=lBXrpG7IPB3CB1__$q`@8xqN%w*Za+Zek+%tAiYdCzJ3#CI8LYOcm4h3vA$K@8Zn zXn&y??M;i_GLQcJ<4VmgLtcCxwG{O|^W~QUoA^Mz2nX#d;fG=DexnDIgP@+>*ta=e zia6|PGDs7Nme05=pNMy~>B(C!aa2z7;)R~v^8D0_xiz&D5`0mndNO?Yv3o^rt^i4W zq`tx||N3yBvkcPyX3Epha7O1r3$LD^-TS*Uz}ke*_;JQhjfG^N8h_4yg}v~tTW`WH z-g7NA7$}%(4egKM_ijvYm^2EIZ-@5&={}k?d@;P(ZnIo%PQ0OnPJhYu2uzXqEb{zpVY!Vos%ChBdWK`j*EVLYwri$2Fz21#fr~ zG_G*%U_0kqmsMA^0*56D*yC^J&31d&A{ns`jxLNC?^r;~K*C5=_pQ4ThwFh^sa^hp z{=3SBmekiS;M3f?HWF3Iy`;>Taj`z1(dB5F7aLnKpZ`R_p9mD7`acTX_ZGAdyIPYM3<bX+2 zEjm6a%|4q(fD_;@9zH3?g?VGQqY>@#NY{0HR!8Dt5OeaOBE~+3LkG}CFUkWM=`XO# zWcs1{2r7KtFV*=HW*QrIR<2Vh=2j7|Bt3!5@oFLa7)6(t-f#vAw#8kiGm-ndB?8HJ((-X0U(#xa$EWY9=)sPucA` z^ScVPi-LJRSMVuw#GRX!Ns=y zYa8v2zeZW~QPb3}QvGm3>P6fGpfsKZP&Qyf#~j{^(Ib#9yswi|nIQY+UVf24nY4DL zV~jEQ)2H@g!lB~0=jlS>!eG9FkDq++iTxUSe{cOsfhe{CJBU75-x;lKhm6aEe=@NA~1BdTv=iq0A1F=ww=dflN$ zI2KbSXy=4!C`FsuxZn_9z7+Z%2eC%9A2sCQgy%o>AM&_S=836^Gl)x$DdzeV1i&$9 zFws0jEHXu*<2AEu549^ZCCJEMRZevnIIETxHm#h-G+T@Z7q2byfk|n|#4siboEKGQ zG4_T=asJK5kGN+h<4cFR(!ode5<&c336bdhI2RSz8sVp1<5fNx-Z)qZ>D}znRv<9H zQo%2n>AdwF=EY7$=VG3Oggn*gR~_e*7OU&#c|j)1P~s)3(&~zOpq@vIEMX?QIKF`v zJ~f*7#A-7I7J-;zL^@29) zQUPY+UzLj_uZ6o>!;-1iF8}{O9^ouY%+lJHJb0BXb+A9~ISa+EnC1a9U4pUV9sm{K zaX-(AJ`}tC9f9JvfFHJbIiwlz_WI~LjM*AE{C#si=VE2tJ7&%_H8l61F*c>_{Av*mQo$O{D8 zk}$mys2a{Ro_71IO-sVoi0Bjb01qx?5Dl41@Z;SYV>8PNVXeqh*q<>IOW0%7fjC7P zh%r_y^vs!!s{_5r5qgeQtM1BanQb_{#6`oK#~9Y0J2vT2d%ENJRM??8#|Qbjq*?+d zfZa+6<6>PW@WG0w2Qd@|5ETReaOOP%9$0Jkhq6?>+$ENzsr3K@E>7iEMKJDMvv(Zo zPqABR%5ht%(Z2Zs)&G1TMRdk7`5JTWmQW>r>?pD$g8W-)d0*Tk|&qwQ)jgE3Pq)-6u*l z0FqTeedXd-AH7bsu1v4#Au#|j2YQ2@J@r{+Md6Swzsq=H$)=9+$=%}7l|H$D_My*< zxtrJxt5d+JebI z^k7I_D8EwicOjU028x;eX1fdrLdz`;wtS`Ox}^fK+kd(KrW7WycPgF4ZH|E`jI6J0 z+#vqyad-qAFPgX?dtxF0y_OcTt!UJsDN5*`7|P3kGU-jy-t1fMGHxa9MG`^Eyy*ta zG#coZg3%Zn(9B1m8!^BUe6~zmsVU6fZAalpiBO4+pKh|q7dPyr*!_9~pJAk}u8#80 z+0Gi3DzrSGS{sn*_8P7zNsP#)kuke6C0ZNpj{MITvb)IXKBQ`|OcN(KCc@45!hGjgWv5|6-p zrPKpWSl6N8Z%gna&I0d)ru6V*@`B?k2jVPqiTtvH z2X7Vf1%6BsCN8|`(uHxi+xcgWv_cStd8`eWslIAs;+ovgLoT{+DIf2pMxAMrle35* z%ktm~r4S*tH1Y6xB~w2Lt~%NY^iMkUz9xr1k@HRmJd2$|Up2FG{6D2&*w}_wIpDM) zS0z{{HKdI#`xu`%?RuDw+q+Nl z`@;vS(2Ihk!6Qcw1-igNMhIjmIx&#hrawB7Uhm)QFG zmZVyi`e>v2HCep}!2_5ZACA;{*?jnwx9`%YgJ`^EF%id-(S+)ix#;BeWwmYS6~+Bh%ei zV;-zSz`S-f;F-&W<&tcDLAWEkdt)o0$F=457XQLyGtEa2HK+P;DbQDqIsP}2t*Z{u zS!st^s40I>w8`Y3!bk1rXbTVEp+3t?@e4A}o1XW5ngu2ga2u72N9 zod*!qZkuD>FO=677nvq{uwsFAkJ% zPgBaA8HqMKWf`|J6*dTP5_0>8tf&aS-$Tf;8?8PaQ8faT`yIVDc$<9K;@xWT>dhQf z#4GuxuvymBr#`66lxy_0fK!~m+wwWU-*68+sGc=k6NtXk{W8Et76}kz^J6zRuExg) zv;hk(o^!d(&VA<%%7!Q#_NIxN{zK0m4Scw$FTQMpC= z5H*k42>#@nT@YG!ZOsy=HQGwz(sJZ4hy(w3uc;s5H5Ct~LY@U%788?#7+%nSgO3 zpd-gFX3W`r_QRZm;fsk~HL)+}6$N)4o?-PF27`y`-~aU4>-W3S{djO@l8Ne|vR4k3 zJouo@3nA6oK4gDb|D1YT!#(h4aw*3xwd1q;Vj{PPTR5IYR@|+qsJ0yfg_AW z@a6X_3iRr~HU&^%A*meB?@QMw>T?P9?-_Gkj+!jLhFYwib3}4_~PG}u;pCyR{tA+@bd=@xIpIZv{dhsJSGEc z_+Ukpl4LIjWtF_}2#$+vLH@&Y43ycE%jYQ*{G<|eVudKK0F?Pzib_p`0Rw%)c_ z%f(2Qxp;Vs$295z#mxCmH37)4?urxCem0TS+=czBaxB7!x-@Lmvmr zdTY=;gS4nR$XhI^gA%p%;=~dce5(oZArwulfyyl#fZ*fK4q%zU6|3|%j~T36x9{;F zW$1#Ze1E~Jg7Q*0-?!$?!G}GQ9^g)a50Ec5=ks~|&$Fo)GH;-)>6@jb@B~^)lHQoD zD@5t$r;L;6JS>Q0R|1YpMYSnaEO;uN7R81{VatXio_JM3-uf7nJsKZ~b6&7ZyAfIA zhLhCLw3HeW0HiekVkXNBh=@O~tS<^gb>bzzn=?qPc^({J^k$;7QaI7+Om6U0n4HoY z>O%;g!e^nak9RrE4BunIG{2bI%h&El*0x!cDHD3MHfu=Kxpi8qt#iOj5A4p&YM|Qe zatkMqkN6KLo=uZs29KdpHsiNzOj3(T0@ZG1fMm#UdKY;*+en!WmH({N>w)`bR4Sh= z{U1kXAJ6puzwyqgbmK@7idg0D6ysEGPVRQpk)*X!xtnpiaWk{_EoQ@Tx}yjOU)yQE zNNz@qu$wSb7)g?CwrrS9l$)`MS%&lboZo-`;(^cJpV#|(UDxxflaipXxC`Nor?$5A z{^P5_Pc}W35U1Jy>~6}iQNA%v-*cC!_<%B=o}lsdnHVteNw#1g9Rk>qSAcerg09?9 zH5P-epCf6s6d})p7!|d{z5rJ^7QOSk(VfuuHO4RjGSOI_%m%3;UWyZ)(g4gH^+J!# z_%(R#B&Fxh>Mv>UWT>Z~*_t~(NvKG)D(km{eDFV;lF&$t|AhRB|5G^t5Hnbz<{yEB zYEN`xTDnqeQGAgmxP^6{jn4@|?E}-+&$#|vc(Ijli!SDX>#d>w-M^NPB5_Hsfx-5r z_?5XSO~3c) z)WYpB3pe?}AGorGC%jbTzytOn<(JN|ZTQPwqWK)^r|9Ax4a7r~p33Q#p`w~Gm& zi!epbXo%Re8X+%sVlEa)BQv2%QS%d5|Jjt5s7TMLXTi30>0JLXbJm(1wx@8^`E*0h z>6g5@`f}baLck47>=KQ-Z#wK#!^ye>{Hn9^;6st{*BZFc6k=yt7}_$sVB>mRwW zN_LksiZBlLdyskvbs48ctAWzk4^Ul*1(PAeqwwjL_qRpQm_^q2Kuca4?-Dn^1V~Pb zQOd{7XVqbV3pKciQBe+X35|?eQqz}pJhTCHiNGR5NpO@(0lMQHcjkSgL%@$ihG(h5 zw>iD!ef4&1nnJ63B9MSiPV}=i~&(4-GOn_1h3A?X*{efwnMr{XxeJH- zV)~fpA;yqnRIzBn_=-RD=)0CDHRvc2`q8$(364y)M5a3&`)_yx_~f+>$?2++AEz1r zc{4#hhcY1fHAAg0wcZUj`HF?ZoaT?nZ!Hv0X|JE*CftO3)xq9?XhsjI8+z(udo3Vz z*hM}dqLEtYUB2suz!f0@{o`mzG|!VVI&f5s1QvVz81y*c@sPXac=$Fu9%%_ncL^`} z(jn1ab5tWL1MvaD^x|*<#!Pb|os+UJtF)ov@g$Gs{sNc?qBC)KRFaq?FowAG6^;1L zr%t5Zep;EvoLJaNxS1oRbS2OMV}&o!`3ou1F3N-D#m_gPXafA>;!i zw!9a2!9j@i6gR$^(YjQh?v$$QATh$1OaRmP ziAzHphmjt~o8Uf7xP8V=P528j9!W{}X7E1q%!lNb83H)eKX7jCQ$a+=N$UdA1En=5 zY}ePYZEGlv7XhaB3V$Bmz6UV%Ke#S}9-T1>vPg%fO{qpB_;ja@ftH5iciRoMn(i&~ z{1y$ueTewEgPUduVEaxu=+DF+xvr|{G|;YcUnZ6(T~a!(q|3`5Iu`kM#Scb@dbP~x zjeOrD4)(rrhn!M2iGcfBq)y|WpS}K-?MAq!jL&G_#R`}HO}95yi-TK+SSM1^NJ8~R0aZfuW2A^%Y5p9!9nO3;`eK=`dj;Wj>6B>#EAb=WZQV5eG zn12#XD5L5_0EIHzkZq#4+Ruw3I*ZztbZuM0N7z>wp_WW$p0>O=CcghTJb{TyKvu1{ z{po?vxu^vGlmvU3PUIEEZ|#c8+-uY^d@k!`c&$2vQ+w5JS&KnljWRFV5W3+$C}2sJ zqLIF84sLVUv=a$*?rWG4Az|DmOn(fH?DSIlsfoZh3J?MuXQ!OKB!6;p|3J|nBvJZm z+#WA<IAi z!sTTjn7^E!oc0e+A)#ZnN+XDuqT@e;>2ppiN|6ngZ2=eu`oO3Fwfh9T-IbHLK}o3T zcMHb1vG`B&|Jj5_P;2+F+w=lo_r4g2ESoVjxkb4eg##j#YtPKKDa2mNk}MDXF^M|K zi-CERSd;n(vIX5;f!$RiE7=8=f<39mu1SEZ4L&?SSzm7pMx&oI^tu@RKWWDZoy0)q z58L`T0aU3L`V7(Scdwny3HV5eHLc90yl22`fB(;zI5abJHgGOA35UUgIUgFTgX!}H zRe%S2irQtc4B;&N0AS_9c9yF#4&-8ngS)f8(kr{QwpaDDo86c`qbcLVREMj!L53CM zzarWtc#s_LA+}y9Djr)H2R%qkedw|O>uOGV%3f5ef{bsHrc^tXg_nMfVW=9PmvYtPby_Yh;Dk8ga*&tF=>xMdKTLy+u#hZ>!G@{&$FPjzaCTWuP?Tbr zQvse7JZC%cwk-D4Arp&>ROns5T5{_HF5~iM--^+etUI0-dZXe5-L)h3qMnV+`mU0- z1AseuZ6shg#a5VNymp+oeEN%zv?Q%$`y3cTUuvg!zCZ~`#e0=EjXX@5_`29?FoU8? zj&ajG1rv8%Duah3zI{&ZoQ8zMR*eYPfHnvx&|1lFpntflATLD#rzm=%JN2if63_|! zhh$6$YJd?<6bA638u9mGaNqB=)T2CTa}(UCMH9g%hK+ht!0)xEkOFn62V0F*Ztjp6 zTBw03?ec2?TbwWfy`>3Z@dUqKqx0$87vkdDTG7{~Hy&32lK-!k4gAB|C#^H)<7ijK zNTxDV7N1+8ez=fgs7^Y=k3-nNz7clF&~}Twi&2#=zEDfr!mhy7D93t{G;gpD44A41 z+i4y>IEp9I+$S~>53rWQ(mW2~T{}R#fyG>jpfnMn7pCKe!C{?ME`8OmeP(9@KG{T8%F>YSrGN|)e z%uo2>P8h2B^Q$4plk<*N$$no37*S?6eFpcijMl8{jG)i$OnUX&31xTM2Bc&>GKY*g zpx(LaJZKB*aB(kD-TWZ7NgN>D{{C0U+x{Bw zB%E$?#aGZCf{((PGqy+QaR8q-@H*J4!i+hn-lgM%hBLP7WrR|TR6G6Nn#uk+ z6nPIGxC}ns1&ZMvX25%hSfq1;e2bxAm~Eqmh1aW7ebrE)i*G{@r4!ac)=iJAfgV1i z7;CE0{V_0PhIly$mK0g-85^0LaJ*gh#7`vlWb7j*J>6fHpP}d<$YGnVT^B5^qddvV zYYaHpaOEochL?cl{e09r88;QwTQ=NN4SUgJ)nL2&!n5K+L!<4=ao`dH?jP{RBmg6N zssR9C{I{fG?PPpjNdGuOy9@nnWVokpYV|<%bZyJ^ z^7%$u<7v0Qpbo?U)FSiAW!e187GV})k5yR(FrtXc5MGsUP$tpEySU3fjtxmk760nF ztHHtE$>_>4#EI$#HakJ?_Wcl1EC0&Mzk?O-f+D>GHERFb!;HMx$=NVq0yQlqy3^zf z&zOceIN(sr-@4Ay13J)J7(~RGA@Fmaxicyu94L_b4=nC~)(_kw6=7k68=KekihCy% z@pJ>h{c^F-3wSX8vVV|6FWAvUgScRNRD7jnD4ZcM9y@E1(QU*x#^b>6;r|0RIQH2o zcCQ53b?;M31iL|cN5olq5aDu#WDI+{b1aHM44_Ii{W8(Fs~PcJE}+ygerxD4<$Rz_ zS~49K3q4~$t0OfoNSB{dwsK8bu`lqGSQG+xvGE%jYKN5DL8OOBm)Sm!KGusBulNg8 zVa)C`?-<4BUI<+i6lJoUH{YbGJJkWw|Fs*~ZEPQldWm#CQa(^AnrRI2*`GE4g}?0k zYAT~2Q*S1i8Ma1yESE=Q!WtiTm$d=qg58p)jb?>JM#%wS(dtN!%Xxr#9KfF%L_2B^ z{1hEb_NjySqA3rOX8*1a!s^(``fnhIDA`lP7!0hkX8i=`HiCnA0;8pyh(;84am69>b@@8)(K==k|AOIWqVyU^PFO;Q6QS9)FU?x_|yfDp68!JW{u0> zp*C(W(LVCJ*nGEJn6~i z7ZKY!7y@Bz&}l$28NuMtROJ4Krn^|M-@ebyz+h^FKZuGTb+ZhT6sa98ac3kP;n@B{ z4rsT-=(!zpj+wgp(ZOQxoo{r-w_+Tq$yjLU*!(cvpO=rO(poB-5@u~Yna%33am+5Y zw(Q!_+9he5z4U)%*q?OpDfzrbE1%$d(mGB3U2-3RyxDk{{%I?K=n!x zelN8N)4XYQ^Y2W4%3ife&`0rEvaB;;C=RH-g!e)8W(`~FniG<%9gfxI)J5O!eU7NC zKk$>i1>Y_6Y2>|FH>fRo+d#23P?43`!2ER zL%$~a7gA?`L6QPmA|BU)pR5=IGjT)oU^Grh6lJ_69SiZ@CW_3~_F~Y;sDwW~qSbIk z;_REmu&(-a$aZ@`v;wXDqV&&wj=UI*e{6S6QCBTHY3DF{933IPl2tcXE^)}QFF;oH zBz@^oos}D(Jy_t5l10=5raOtgV`}#djA67{Q@n%bN|I}S<;a`(#`VW_qv0z1UJ{#rx81Oi#lbxMGcKSpFp-2M(B9RYm!^kuki{1Z`F8tI0HcYSZMVZ$OPSKKtLS z_SU5Y7Q7afs%z52gVw6eVQ`x^KNYY`xj0xqR6j4c`ig%YKD=eoH=&F!Yq{`pWTXb3 z+#bg9IKClL_e%S5oKO(g%@G)DLHCy1_xmJAJ_kY>h+>Lm0hQ4#H~k(xZ$fR95jDuVVV3N_e!Hea?BlL(x&$iXUH^Rp;0F7I0-S zK_lkLnyY)sy`}baiO#&GqUSN)^h8&v9jjG+yV{9a6YZZ^D^EhlB9m}PtC}i@lABGM zD~l53?NBFilw}d?)xmiJG+YD4cXqj?r_V2E`M%Ldf`2+43T&dVfQ)ZTc5$i?$5`b9 zp{i(-(CGv%Ag8*r1Rcu;Eg`4i_HLU$q*$nLbnhlyRo@3Jl*jN4h%p{0annP}r)Y&2 zxekQAvj@lt zfL`pFd>XL+5=_T)be!rd@WINplTMkp&8WB=NB~O3pdu;a5I8(6xdzL86q)Xqn>Th- zjI(@yJISa^cN?Cfx(js2PANf|Mo?u+b`~Vfjv9w*P1T)c29Rlb$U*ps=oX8SdvMS= zsX1xwFMH%+YL^iJ3fF_hc8IioAQ!wuG4+`IFzVLRd8 zzy=d^?X2NEzBEreh_G&croHuM(9KsFV_zOp*{(1pPIjwFsVmz>l13;t8ik=?)*xCg z_;y7BL<>elUVB~6Iv)eRfTupEt>6-is4tO+h`jlk!ca2>uKS-&`v}JTlwE{hhAgZi3?S2B_=xFd{hD2VKhiB+KT`19yXL1T2$n)2i6^ zW?{D)j9%s-d^?D%p#{K7`Gc>!k)Hqy4qg>4oPL=h>Q>+J@`pW2v1F=qq@ysIN7i$ENvFrjpn)~>4a@8WKfwf3X`fp)6HhiBEx$Z%X2F{i zywk2Swm*)|e9O!4dzH8S=D)d~$Ds$*o3sjI74tf%u9lY0XEWr=f{K|$&|~b`1zMZk zY>bGn^a9KRmY>n?D=I(ORG-}^fgE!qN3KaUdM(lae_GG(jl%>TB|p?cU^K_x#eJAX zNp}Q0NjjAuRua35&|An4*Z)Tje>c{IR5{;)!_Wu(iM+EfZO#tS+sxl=7_UFW&eogZ z<0Tx%;0(HYMoZ#D7I5NHjo;`EdmDM`oRRD@Q{H_EM_?M_wRC){8|OHh=8g zFi}VKjgRU|siPb!hnYqQkm=xH3Ss)PC6F=Em|AsMk~&-XhW_M&iILjNJ`@R<6&6t_ z&x6KVj+Mr431BzK00ZT5P*e8fiJNTRtu^QIbr;D6@(01Z`{`5MIWU?sXiXlIv&9tP zvZ`&0H1;(i4e5P(i?WGp%}p5!VRqK*L(&2F5`g=PJ;zpf-v<ffY*GGH;^hYD3~wMY*-4958$jy*K5GCVFyEjlC!13Gya*OhxH8mpwP{S zy0%rl=ea$Hs@;{}2K;lSP)})E2V_s!$c#-Tv$zszz@7q)sxyNB8^=q{FVhD!iEEHAJ3*7e%1j zRI!{YhmYFun%RLlpeS8}6gn#>YeUwp72mpDU>x*jPN{SZ0VGuCo(WW0-4L>OgWJA)&8sX3FB}g+X)#|+rTb)7C+NvN1PL5XOxm~3gxcQg z&-dp@!hgpVsZw}=4TdtQS z#E%_9Ng<}{q=^fKI0fmf0M0qCjtn53;)Cn!MUcILnVIP8gV=BZ^?9W}muFvZy}Y%{sia7dfsmul25x;M>mwg37CIrHd zT+T7>H=S&pHZl$^y!`?my^&lNiWr_=v2g)w67*7J3CsIpuC3Z|66>=ald2qZo_K)N z>8QHd(4ScigGQ%c@h;?+*X3gOIu(APf-7@lO($m)$WTFPWe;I70ktPGo13u7BUv5T zaKD8|2 zQtvW#;UVkTc#+k9!e)g^zJ|G0AFWgRCJNB{*gIQ>#IV7Xfp%!l)q5OE-a^^XPkwJ= z_iDmhi!D;^9KLr5-{3^;=}V)@EN{r4^!8oRp~j6|f~}|^J-;^E4AkC=idvABAXp_A z^WcTPg-NbXEwo(AyXf*_yqu`otF}?Bbf)Z84CJf@NZoC^%=BKd()~1O2H=-w)>ooVI`?37)zMJB#&dP>ZURhu7VkFbFj#xGsfA~ru_q}&;k+*4~V4t9#(5GU`giMnPNgy zIN!te@A?K4teMx${3HxcA-)_IBlE?dLO8feaVWJw=GNlo&;ETU@VFy=EJ*R8A-LGJ z%gVb$bHL{Fa)83R3nh>AZ&m6r+7h&=rNgs%Q7sO{;`ma_(PhTr;$)AB^NY2i5vT}8 zwyRBibK)#G#e7=0y3tlnFj8t|RnE4o7Z=ogif;K(qLWipiLclAEX}UlIdI2`NT~oD zSXL0JxndU58vYb>>sHj}`#?C{cOq4*5VL)TZ2x$H26`e$Ov|bPDCOWk?{_k&KM7V= zxMQ5E=!>FrM4p#n!Csx?fagkdn$l~~|BmB8I3Nl!i*vEd@ z(geNt2XbTv06fx31|k7CS=&Hns?=6#k+^k0L^E}gTuj644*M6?1e6meX_3wOkfxZp z@AEM!?-Xf3v^N`$H2;=DQ3$(W+wnP_q6nsfcXluxNFrgSRE*!o(8s!Kub>gD3%|D| z)ePH$%NvSUlfdUhf#GAtUi=?c`gdLsTvkK6Qg^zYP}E`mG&>2`R1pknk!D+g-Rco* zq##O8@`Bel_{18E!r+cHj4hD-r8gIpXXF3)c9RkI=nvqjTj*}rb%6buvW;=3jXOSM z=OEDvL?&V<3xp5m>5m;40x-QsT;yICH#JzCw9ojY?XpeptzdcOqE$9_)AGqu<(DV_ zvQ9Qfo9+C6o`zB9O0abn_6Xd+g~^|D0N5=5Z0cAwtPlc)yoDtMIBAo4FS zHzVHYiD57IKIr7=fHnk50bJ>k86JCPnm&b$Opj@iI$W&p7M^PwqJQ`3teAyF&}c2Z zxTGSJ0ao#XJm6Sxm?NznX&C?RNsYT?m=!C7J@(kMwo9qgwr~$y2s6URFL0cyky?ex zILc_0kZsG8U%z*DUKm8@i3V9N!2K{t#>mjNKe}&ic^G?S)2O1j0;5>r2>#i$i=MMb zox%%KP)EC|xa#3)=lGV1z8kTW=>P}@2xPUo_L_7B-1!7U16D$-(#iGGyt9)uMq^+8 zYsN~+-nC;!_gSu3pkebSP;X^6t6g?0r%R_gdXf7n(Q@I9J^rVC1THPV%XtZYjKbx5LPM=P-EU2CY2=mrU z!QkM81$*H20lpb@h!|U0N40(!hyrb18h9v5?_9|AF!@Mx*Y6}b8}^skWjXL1**WMN zh83~ZA;B=M^lgvm_8x-mc7A8YP*i=>I+YTU_9qyYprD*qhe<&I1>;@MMa6u+IZ94Z zr<`3zC;5l%=qCOZf^_*Gx?+R_E&zU)Ka!!gbLa|MN$K=)(77{@Bw*@-rO`4Df;Xz}}=wim2o^ho~L*a%>} zd`mDJl=$73N-|Oo;Qs~pi$V|6vPH5^`Q^4r=KWiBnB{$k8u{>0CJD-5qHIv69tq(* zz3*<*YnkivEub?5h~hIF%;u&TuHl(n_Hq2mC4rUM!REkHv{rYIukYXfR?Ylzq6|sU zw4F)1bYn}{bnfWz8I{0a-+T8qh!pe4IZP)-M_tp!-lQm-alzbO4qWOhME}_1_3@f; zSq{JSs1psy+sm}z2QQ&$=6T*I#^T^!;i}{_r!kyqVERniGhgtho4vsqI!|Qw3$} zLMfWrQ50~2520DN@Xb1Su#a;DYl^RPx_#xtn2zix)NVr1Cl*v@k&+U~z) z^i9akLzPB3GsuK-(H# zHR{(JU}{cWY{9$?nejk8>9anhoaqSyN+Q5EzX{h+;gzV8Eo+c6yH+WxwGXNbGiJ~! z!NB}>J{LJ_=T{dMC|*nY%uF?oIieM3{JIk4`m}RLxnLG>6c5+>zB7AmwfsnpdhE3cne>i zmixC}ZF7EN7YAq)P#~hftE!n*0Ri&QoE_PIVDpt-Yib+*SGBeldZtd5W z_`*M0=&ay}?6RoAPf-K9{}#quGXhDMTKNYx4w~?`#-ORcH=FgH0d%}a1ca=Yg#rP- z58bX{hgHlYNAbP_8h(LpqGsXt&_2gtUnrCTfco?EtKK9 z++8$MRdwa`n28dWMmKk&5p8a~m^B0Ou~{G=Ud&Z&zxO49g9g*&gFN{%=oWp4L0O>1 zmDbkB6|`1mVJ0L!%z$82J_FDTC03^`Wsr1UP$7UCUC5W+T z(S?SXI)O{MCD<}Ul`q`lT&>Ll1GPCzua_=IBSPxoh6nVokHlEZ6D1`Qh{VghpD;ftsx!$_@w@ zYm!fnuWNU(4=ErSbjcHs9c(7oEn3jBtG;GrBq&;#7hSx)z4_(K692}t?WxJJZia&# zbcCpY-ajz!Rxlz#en~MToKw<7?Og+pf5E>HK(vvWIA_4gzXyN6%Oy4ax#s80(>y8` z1q~jI3_5u8>3oIeZ1&<}!jAyChEn*)591&mTJnWO-OX#K2HlKg1L>hVgv&{pqw%uT zrA+tJzVwG!6vkj6sTz=u3fBv(gL?=>+_#Y)>ypDF-}t7`9!R7#N2Hg!ckQ@v4Xcvu zwYvK}+U0!W=u}0`Cw!7WL$N-@D8QMORrgYS(XInk^NV0Rs_i_bxVs-sExR{Ak0LJR zgO zXGAFE7qP>guk&hP{NAb3-2zkQyg8DpSp9Zs@DOvbteHo3MAGVY#UY3&wwdz6^QIut zdY-Q@eE6zG@#58zGt7cwUt$LG+NSq8Z&n#}D!%=-^D5)66p<>!JaTX?y0(jVlM|=7_#6H@@QRdWl377vj8G>5)gvw)CvHd z!khj!BM2Z{l|28XzPE}xa;;ej{$IeIf#kvxij0y;I5G|s z9mS+UcC+lrZOkk;AIt>WZkQ&X9ky!a)-H)*v>=bCs8l=kVHz=Z_g#ykHTAp&gO+^~ z3)!p=Q{cJ)c&D=xBqenre18`%r|Jd5Ly=}$FA+`x$whYhrDT_Rl}r-AjT7%HE4l^{n|HyJV|hq$_g$XX=1Az_A3ztnIH}JFOV4yk7&*uNNrO>F0`n zUDg!Y^+q<|Q~4|bMD3-Ma)-XweR537fCcC#Y27%z8<|e% zTZ?uqYbT$+qTC!f_NiRRbctxFeBHRcEa&{aWVE9|Er_0G;xKqRHeacQSB_z%)UvmVY=p` z`h}sZ_g@w}2KrYPoF9;ON>lg^r*?ipo0rfyeOx6?TQ5 zXy)hpa6bzrKHtFhn59{;%a?Pk=9hds_DEB2y)XKJj@t>bW0My>AS6{!eWTOtXRe#B zY)NR#3Xo#-Q7#4~Rco~IO_3lQU?PjSUvWE>GJPFE*Pu z|5?Bm9|&-1?8B_Lfkjkm{a@QkZp4#bo_RS&+t(8C8g>9bAp_lQN4M+ZOMDdzPdp9= ziNA`Hq6K?;s~vFudLlUf-agV*MnlQP;%f%$M+l~sGh?AuO}`19L?HpUVFzM4S|wPEmSjfF zmODpBYq`Et*VUSyuwC@5o%3SZ4uxfLkyK6Sjf10*wS*(LgoV^+T)IgS^#w-;hnk!% zu$|2Gp%?cG=f=VoZrr<2LDN*Hm;eUn-Eq@#ZIIs-)ZUEMfeRn&fyY$(6Hx7PxVa7G z8fBaSym3Fj7oM`&Kotdqb~fFlLC}ulFIM z7_!}#eBd&`XwS@Yta!-|tYFZan*aM-jt($HEg1O`zsNV4C6oh5W2htp3w0frXEUYA z1`FsE^M5wQAVkv_d$Aa+EOhu@pb6?l^rdoU(`hYA^~e|kVV+tR_9h9HNKMlB6VmE4 z+7}ICooT8O_uAi$tFlA0EuxGgnyPxP)u5PUcR`-u1RNUPf`D8Ez})gJ zI>^YqICK?kOX&H$)3}rHWhU5+YPsC6 z%0=5X+p4A#=EsY|u8_N z_4i7?(d$Juu_N#%Kw>vZ(zNan4)EN(UU*tkCprC}S0A>07BscB%{s(1c=h&7lKmm&Y2H=>w4 z^_`#zT61G_kO*(_?Pg2<`3x-Icx%+g`-U(tgKD}&5AQ5vK5{!r9j!I9RHn7vs25GM zf9A`9Pl=1<(%{0JgUJp@xYxZ!wa@ws#xbO@Kaw)wF>FVex`$V?ZShInp+^+k$rHf> zw>`3L0t`K=eV0#7XW8_M6IJHVD^D8@<#o;`A#pmRBejwk8Y|Ct*0PP5>cI_k3A-O^ zs7}9zk>xxbRQHzV%lflet9H%LK6#^VKc8N8;38eJ${8P1!+W-l6{6RRB6xL0Xsk~E zyV}2cb?$?!fU*V=s3YewvV7!|#8+w4K@Q6#@26Sw9A3q}k}`R8ogc97)qo}x!_6~6 z8VaUnFJbWq1N{@~Q{}eTC87YPiy;l8?9l+(ZL^btO}|}@tN@Z zp-S)*6;WOpZS)^673TQQb8>VY#;y9`9^06{SU$c_%6tvU@~|UyZ8#`5Xsgf)Mp5ld zsQ$MF%%P-*g%|l4v;oj3jIq9lbYNs88-8%b9HFgq#kR`dUv5gkb$liGQr0f;H;nOc z&E+d3j;KI16YpZ55iEcjeh>yWYLeyaBiE>tLlaG8(ZJ=jvFYWlc;l&)pV61M3oYxs)$=zZIr933D^p6Izw z^xd+eqQ3)7GBV0z5A8KiWk6?GIwO%fU&VG+OfHWVmD!a%N488gzwkmQ#cqyXiCOtY z$A59vT4Un4G>f|75E>W13usCB28F#b1nC3+=uVIkD|RBxRDJze4R|ErPH^<@SzNIm zpXyNPo3)*Tw0)V1^g<`-#{g1whyo71(_i%ltlq{@>%%^lD$lfQ&Z~1S=Jc59wGILa zUe7wb1i_2oVNG1!&&?+@x;r?rXzbyZqb5QBY`U!~Nbh3+S?VRCa&5M zC#Ff4dvHmy7L1LXxb`Si$&V~txaPMR0oN* z5SZRmir^Qu!rSixIt^aZnPOWA&$&3nHjLJfP7HhGKq>T8 zY;h(5E*oU!Q%xG17guv8W1dq{a=lIL0FP*GhW{;m4|a2{2_YN>@@434vNlx(b$Uae zy9TLLd;2K8Dwt{Zjxehv9X%9s{u&LiBKnH=hqD_BC?wpxxc}aEdEE*DR2BVP(ix^h z8{OtNc#SB5pZ(BE5g&&4lymlyenI{x*w(`uB_`#G5s_FLyB(1G@dq2Tr+m=Amgo>p zvH<@uSo^Y~)>&STY*7`D83`TaPVdG{`VH#is_FZv#+|Ut0q$p-$1S;wwj;c7b_nnx zWaURYH*89jyuKuBPd8>~0z=~|wgPi+;~0Lm+jW}Y?om?fi7`u$jQ{9aeOgshLYIX4 zY|$S8XYp0Md7{tfY!Q~9BKd4Qx8bN>n9v1kVb3b(9|F~|`Kd7+J-cdZSaf@M!%Gs3 z#+4t8vctq+3(~lYK3_17^t1NX1!%er3P~lkd|dO^{7!I3t^5D8sf&?vm_Nz2vzHHK zvDUYBU7XQ(eAC=~&qh2uYIN@2ha;HsfZf(2IZgbCN~O>;<@8>l*g}-^uW8C0pugyD z8~|`{nVF{Eh_qGv-Lu>TNc3I1KLqAn`M)myWREG!7Fv*up3KMzM}wZm?)W&d=@YQ3 zK`6fbPKFuIkXfNe)j3M;FKVvyak?oRM>4hcCv z*}6Vfl7vb6^8G(yM;*KJyp2TGNq^fK#tB*0R5fsfz%mxxjxMTaPXUz1Xr`|_hr{#1 zNHt*C`Byw^M(aJaAIj@-Wdl2~@DhBs$a6tXio>dPaMr+hgH6{8mY)Q01d(XAU03WRqryU&XyN!AKQeZ;esuoUY zC!4%B4)HHG9$Y(#xWU)yOOFHOdqzfk zlOV&-T9juptJJAihX`KJ>tDAU;NiY-JqCk}80s#Ou3KM#xA{VyUEG*0_oh|F(Ic2l zjpcB}%Z%Y?WGwmI##RNo{s|ziP*bH7^bLwKTkOkjBy|Z^m>)-=71P-QKkrl|vU1jm z=*0DkVUUZK#Yt*9hvEbljbd6vB_I=R8L^BXoBF_TO~Ts7F=v!PAdclNF|x{1zqvi4bUL1 zE$O`A`NOp8x*&CaXd$#sFp0M<)3K`tf62AX8k#EGq6a+`5#=GugZrSB;L-Lj$UVVD zAWJB%hAQo4R%Op-LXE?;rlEI)X7K<%db>QYr@F^9Ynit4*Bg8T%&uBh(RP)~_`WdN z_-FE6lrrG;m#kI0Dx)@{Rr2uKv@r>InYsX86?HSyL@v31$gwY@Hebb zV-Nt*QHyu|9f&kq+ucQ4K5@QHf2b-;Kzo!~==WAeuaJM*gJDFZzYA-UeAlC8A#eTg zPQ4wUG2vYzGveH@noT*XuCJD;-)S#w>jf-R{-qXXKbAln2V*R9I+b4KBT#)Hx z>J2!Jkteqh35I6ObQC?0SSNrS=dQKil@TrfY}y-_Y%D^1cP@HL+q~R-tD?zaX|?4A za(de@Ug+5M7Y!~=At91ZoDR)t5P17K=T5Wt4z2y}7CR11-gxe8M}HN5EZH?cNQ3MT z?ktNI*|sc?$qjqF$?kU6thW1_L)UXLW2v)}SsAB=I*2ZCa zkSt7omT2nTL6xP>O*?1`pBmKK&YvkjMghzL_u!xB>d@v*Xsa_sdppFho12>G>2J(j zUMVbBG95H+VO^puz!BhV97nA_{X#VBf=DZ=o)P5Pm6SOGh>#oi`*Oh!-#5bSeO|S; zHlv#Hg_u9tCjd$x!AGh3CN+GVG?COQ(S=sx?`)T6BK?D}rWLC8IrLR9hORU6Yzj!{ zn)Bb@wLB|O?RWV43~ADBi>=os0B}@LS_`!Pk;CYTH*D(+{JfiQBo2k0XajYo+$yR` zi1)eKmlS)WJehKy)wD=lT-CZ;cXKpG0G(bn$^bIe8SS*lG;rBRM|<5d*o|^gP$>T0 z2OgX&9f~utDLTW|2AToM^I~|G9j1;|?Ff`jnJ<1O`|RK7{mqop3E7QW1CSa-s- zL)U&)3cDo6_(H%poM!Kk>^}~VCKJxD)@NX6t19ka&FmZ;34@h}^uK^ra&Fs|sP<3Z zn>y1NCChuq#fsE8C+4+rxa5%@;dk`55JkW6Q82)LU}nUyJk6>rM4#Zu22(THSsK*} z?o;o$-Oet&r0*LR9%cB?CW4sED^&~EYb9p1)SuMu=p~3=0%^=n@%|X-xJGm1yug~) z-gxgF-gbVYSN5yfaJ^gg+??u7Y@*Ql;(A@eYbQ4^EWB69W>-#jy8k zq!#np+EwzG*D20JEZCjVC&{N?IzBy2?Qsm({_$6ceQeU5I z#5W3$(P`WVJ2wc4%#_^U&{t3%v35|Wi+9s`3j|MeDIOP`-<>MDoElx*H0>1Nl#8JD zXE+h5gc=+=atp~RFQi^HnVs~9NnFg(9ocC^uhO!;koMALZ3-(j8uu3aF6{_KmDaFm{xU`TkMuIu@HUx6bxN1xND(@OjxH+3DYujBQAy6yO69VdosyWDoepCd zs}vy|oH%hoOiY6g>I@^X=mw{fLd1T7NiC{2({XpVt-%xe_*)MZ z-@3T@aVu40@`4^mTgT(|&!r*Mrx~LAnMZ)TA8+Lde-C3qv>xXoPSzFA33{C4W+RT(3@Wu&UeR-bV#1A zYB_kbR{^iq4YhqkLpc9&Lrq=Yq^P_=|MldB#GQ2O!zi@I&|llXxYk-#0%4c$;7Hw$ zDQ`@-4KCGPzTpQ%UeLT05cMg!Y$ST-^{5&8yw>@IJPqoEBbZ8hQ8}P2x{S_OYOb&Q ziT$2Ld!zag-vYQW!q2Vjq%#5Y@qeC0={ppfl$kySpafR3Ri_sX5k&Q^2sAS_jU3pa zQ8757@BQ6tkHh!zo8qDExTwpD1&rD~U~^f4z=WUwk9=J2vPb4@hQc%_L(82 z{*D6)J9&=wwaz0hZ`;N=f&Fo@pu*E%33CoKEovkIw)0y^fyKc(2{uG)sMp^%^@edZ>qP5a#F$F}$k4-I2 zFo)K{_gZeh{I|^$b#UTe-b-UYWqz6*)rrun!~1E?G}NKEj`XKg(50@<*2vQG@$q?I zngf4@4-T|zd*2)y8|yaCjbm}NUiE~vKD#e)pAFpjZ(GBL8xTpQ41G86dP?5=FgA9R zfdn-?0jgE)E9!ApQua>(dop10*H#KAgeRQ2T&4!C7~%{31i`+eNxYpzcl^;eh(lpz zJ&mfhErY{Jhnlz%1hIZgYL8qn37yZpU;P0t9R?Ub;X}6GufUeRNlC@0Cn&X%bceH^ zkz?lpk3VACvs<(MyD&hR#n_4tY0JQb18^Kv;FASM-IT9pj;>utt``eS<92Eura740 z=n3710bu`X$ujjx)49ZUg7-KOHMp1_$bJ<#XtUHg__kXX9x_FE z+CEPA0Xqdu?MSYN(L5VGBMF)CvQsDz||Kw{OzNFz(n;xwZ+{1iE+C^ zCuHoIfvP}}v%!y+87>Q3ZSri1_iQ&Y(;?``VdnSfDaO!ugwBhFycF9_H_z75a)n7z zEKsTW{zVDKPFjP;hP>lbhepZ_!b%oL%#8PaIF*^uO*nuz z1+aVb7ga9g3!ok$IkaZ8w%gHh+_N&=cg@wdr36>Mkx?}M|D?lT16C1S?b9LI3t8J2 z9Iuv=FDT;Sp}vZ?VlK5@TH@3qCl!(XVd2nn28Itr7|agYeziMU#z8x{kTc2ZiSXBP zZp@~KhI=+*xatnK=7J6v!!;!i5TnEsL8rXkdVbXMNr251M-pCRSb`H}CYkons{;9f z9Z@Id-{f;=_`B#d^^Yy>2n}|bVyQcO8{S5Nt$b$1vc^UBxsS~%_GQ+nXfv8Gj;8+7 zoadJ-kzptU;Gf9#Sg$%G83r_}&-wb_HefKmMWu7r8fcD7$GP!lSxN{lPT= z#N2)3{}m~i?pVaGq(gxXk0~x!7Xi;1n3k*dNbLc>@V~7m0m5iV*5WwU!U<*k%Q;)+ z))CHbz#*wAVyvS1xP zNnR*)tycbYG)uE6^WILN!nDDe#u1J)Z@7IJM_ciY7-)O;1D90sP-n4c4}vn(gGguT zDs~!5rSt}5t>9u~7@ccT8lI^HU#^FN16DP`b1UPPOGXj;LOVGjZv%m)Nccb%%&JtUPl-YiW#uhYa8H=; z+e7*@4DF{S#;P_AZrbh-n2q3I`?N~W@}b&rV6yW*?@f<%F4o^Og@On5E=sxnAIDaA z6{E?)Z%`ks&d2RZ|qyVnj)%DQ544W?ADe)3BOO{$#3K(Jc7QbwbimY_-ZY&o#M0)zG&-0|1*%wWKDR zawU0+xMwbQ$)qKrQV1I&g`syAIE7{paPFY?8eMr1t`&QtHsL!R_@J#agWDm27^rmN z3e(gLg7~D?Qu2mT8Lp&v(NjJ)J5FgcF+!eTaO#dMn{$BimK3luZ_h}T4VsQOtbFD= zePz8=V7e2mb#e4A;pq{(CE2D{ompO2l-3=W-TrV$&+ADD+A=<}Qj1np?Qw#*l~G2i zwg5>6H9G0*@WGe6^Z9rKLMlRDU2lh6wKfh!8k%IFD<%g>gQjV=ZMqPIO<)$t>irrx zm#vW&aVNmX<-)bi4do3of9N}JiD+9nFX;!VNd9!5_03kec+PyvuTYxd3E(6|0FE(d_*Z1|~ei`#U2 zo25wyw+4otkCNt14ib9|_^#wSOMXJ&Krk9({nReO;15g3+^+_{79n*WFC=$$-L*u2 zZtwS0B#W22<`Z{I3Np41e^4c9YeDB*v02HlCScyLCc~QGbHKl~K8%*vR;Tl|z+!j7 zPwAW?V)VTb7R>1T4Yow^%ZRHP7%e8F#EUt=;d7BQ&8u@g0Y}lHi64bWM#v(clDFPT zv$EF^jlT~-2kyNNUO3xLkJU0?x2ssvf&&B=SZ6NMO`Dp%2ZlF!GrY7Sa4Lc)bZ6Kx z{|E&34^(EW^IFq96#aZz>JU?qYpXB3>C5#MUW}#NSYs;Y2nPXC`>7vd@753V>*lH0 z4F9_5TWB|ke5S3sPyFU%t2!yKYI1UJ9sA6bZ=Y9R7K+iUJv&D`YnvYw|JLf~v#C#Z zr*C+g_P_~Ug^(_b;JAvpR+K7&Twj3|54w3hsar*vbF5#~ZwMIkF2|3sk0V_N;;c%V zCI*#Wr6QmZ5hyEGOZ>VM30GA2C6hKc+01hoQW)Ri=~(one%K153ZU-DDrwF3d6app z(j5)T`tgN&;i39`&2fSE{MKm|6k~w1?jU}-I^4!$<7fx6JrJWvFm~V z6O|)TlZGO%yz?CL33XC+0~C23QcehMl4=<@d;o@uy%Gk>0+%Fm*>?MLYZ(kh*vrzy zF)n|JSke4AdauAw5mbPwtxGzMv?Mg+!QzX}ZC_?vGp?lz6P_gFNuTOJmat((RKy83 z-i3t2bQlV$aioa>0N`F7X3vdBPm&5yp;Ncmbod0Z30e4jfs2YF!x3H8f~sj zsc;mfk@AFZ_2L!K*E6acN5j_7zl*EqVc} zi2f2ile*AIdlh;K2)AfKqF8Km`R$-lPrT_^a%JY23m>~}7N8WIcp%}Fm$c1`a zwUT#y7^+-++vlA-=uJ-kr^#z)l;^<%q6d|0W73K}?RI*(wkCQX| z!S|@S?2Z7mRRY|h4{Dy8bcT;?eO{1tf-ypgqhuO%yE=@OaadiILg$QH)Vhx<+56Et zY(C7nwrDo2%rO)%{(e?0PM$EVD565?)E!A*r}GE<{Qu;HO9BK5l{*h}Rc=s?%- zhvR>)HTKuR!VMpPSNOQv)-dMzZXyP?uaIt^bEOvP2Gj%Y(Uxfd^@XY3LmQ|5AK!Vu z)dSzd`&X|6C#6d!wYAduD$j%Z zBLPF`IOpXhSajyg)ES5>Bzru0>nTrW4m`|UfS2~~%J%<8*D+mg-`RN^i9H`#;*U!# zFo))o%<>?ujfRhNKHn#$Hw~EcqZym}Jjht|n`iNq-@0kCs*XFD!ma@n#2&lsmXTye zRSadg9>w{3Yi{kIgk0qq`Ee*;H_l;fQ1IX)qE_iFq{#9fY(>X zj93*PAAXqB7vjRO8{I!Kfm?gb^s3iR6KONyQ>Ljw=Bm``P0gDl=xEVFRkCjzlkTnA zomu1ldx4H?k;c1Lzs1GHXqo$+7K@!Z1NUu%5K7dxc*L`*F}%|hK*kJxd<@8=FWsgC z)j`hYQ~w6DI1AAqu!1VAAL@PFQQUtxkpOXx< zKZW+CQ%>Sts>1mtFAGNtTHiUY_CEa1PkCl;uyUfFh{gsd9$ud4sX*x`Kzj-HY|VRN zlp$}`Q3YT&Fz5hlKl*PQg$HQ3-i>1$ZWUqI&jVHghZN@_ZTC!vK(O*T(6i^iSaNa+ z#6NXKXYuyzg%HyNbsK(iUfPCHjq1`mU({82sT5T@SNKGkQASG@Pn!F2WR=O+vu^^P zXSc|bJP2Tfhg3@S^O}($Sv(r*Mk6d!njOz>6|V3 z2d1Z%vaT`kb5mfSMjfbfqGF6c0S*&|a6*~Ob*vFIW>*BX8Y9kLW#hQCdRI=Z9j&s~&J;o{n{FULJ?0Q*U+kwwr4IRwSpFtNiZO$&CP-B zN|$2Q=jS@tr@{Z^UHfYXd+R8U)9HvUNiT8XU=Qhn!+I8<$4a!2zGyigaBGpIytRDC zTkq@#cWUXq`_gyJAQl*eLP(8f7|H?VKX%<=9H{^bvy$dmz)y9*>#Yc52fW{g!w3X?4^8xyz}>qxqZ$) z@3y}U!0BjiI!J_!`zFoh|3{k^LBtFc==UyBzc6F05Yr5vL=#gApB1#uK4RL)XZm$};!iv*pL0NoCw<;eV)Y6?)!U9W zC!kJ(tU{mv>N{z!gbzM1v*qSGMm#q znOsiy&FA-)({+&RnWSPpijL)SoewpAKq&}2hI&D!ZS4;y>~pq>+MT_nAAm9h$PuO= z`W({g>X;4^7qB{9vgM7~($VxcxWUXK*e`zhCF3otHSO3V7#J{U7tK%}?8uduRM+AX zrmfRl&u!e?8V8`53WJ{ipHMKW!u0Vy^-jlVoj#N;|Lzi!e)+4c7;ALsCBvf-+a40& zYlmynuYhyK>w`St^Nn0WK5 zy2t*HF!xBy)quKSxsYs3Do&6j2nysIF$MgZ4oAMbE(E>9(^=E<`ak{CUVv~i?Z0J) zBX&T~s|qg4g+(T>L*8%}ot+ZP1)`z4$$7&VV%27$G(n#Q`>AlVXu)Bx%Y-R9<@gpC z4tJXd;~|*ualpI~nK3QX6a)KHDp1qlrGRHu##RMgXb7j1D~3LL7_SbN=c6rse{+v} zW#ef|)TkJGVapwu{cl@ycanuFTcY_Q$+!bKRczi3#Wtkr_jxmOgPo&djTroBeb$!~xmZ7y`*sO3dHNp~p-$Sa8x5$IpNTt`rZ^k- zLluZ_lOt?AY*!U;9$<}t%*N}=+aJL!(7@<%%eV21P>2P4$@2L>*@~Lr&9_vsQ>I26 zk@7KKrt@w+VkWiyvGACDV%I55bjquTv8=5FG{u&!&+Xi+YlhwO5lb4R=K{)JN zBYr;Z`D$qTjNrl>2wmt>seIlv(WS9q4_Kd*63g*tSaR*hv%HPH!S`R{7>L|?FZpU} z29@Cr_HPD0rRl%IKsp1m^`O$w;IJH+ng1kp;t#9RMD$@I@{bSiT1@xKoNeMH0H^B4 zbY+jG3qnlnYO^1n`|Uy;YF{bTPaPgO>S)8*rZ zv)$R_(@?gno(ge+LnZ9T9~gD^vOL@N2ZPNuYS5=-07Mb}5)<^@zsCWQpb>#Kv&4`wK7q1@}sYA5@gC^-Lint$tv^ofXjYC0U<=!RK zGU*qo;n-l?Ct_Z?z_S|3&%nj3vq63`L(bd?oJ{fhrY9<*HiD-WrT_P)(?2X5swx?9 zA9*V#%eRiFUT;OA4ZMK{qjp^{;6AyPO*bEsBXvu0 zY~99J5Yv->z?*Q3e8z$~&YCFk(6^uCv!6yr6TcPoZKhU@3p^9{6&SBel{bU{a37gL ziu_8x3r2aLyQhYM&#oF8j}?DxHyG*w1#Ea!N_F743qG7x0qG5k&Ba-gZT}5T?P>Bn z{%*b|P}+bBwg44N0tXJ$uMd7_Ke&LHAncRuW>$9rZDcLi{sX*SE0*Z9J?cSkg!d+wECY>2&>8QYx%Ex%5LW?5Ons36r zQKcL^>kkeF`V+ofeU70F-dS28e|H^R1@k~{QbWTTgle>RlCn7D5WF zTZ&4o?5P^6NNhUg8-O{AQLQi@@wHzwJq@~+o(Oj-E;Wh)8-P7oEHVNq->m-`bXsky zeA4r`L3~f103oXLIof0ieXKM7%Lvpd+`z)jJAxd)UDB?PcPE72u9X_@eu%fLn#838 z?KJgx<6~=l>ohF+PJ#!ALfpTZ?&6X(s7IDwY?4C%_Kp1mPJa6#@O#&OWI=3Y7_e{{ z(!*a>?U<^z-T;fU_s-E>zqdadDCHV{syp-Hv%GIJ!-Rx|72QSYsZGjXdxSb#^Y3QG zP$-N-x5uIb8&K;NLIz>46;?YvrsN3xLbomyW%B<)B^ow?DmaUb$0(VZ+T(Uc^Zn6W z4n^$*3#(iCsynbb^SPmrpalznevVJ=!a2tsG&xdqvzf&uG#wjWJVxc(B9Eim52^~l zxyy0DAlO7BhDJHn*Ni}7KYdjO0bu962PmBGKl(|QCf2XzWLlosR5P5*Ix6p3dt%vO zHfm*i>TVn7?pQ2HSl)PnR0cjBK8#Ew{UkCSXYl|!&Zc`D<{K4EY>VD^AKXuOK1VLifF)#GqFHI`l4pJdn8!=-{fKCN1>NY)S`)~cKlRamF zq6W*{yxVh?OI~0wm?EDCm6RdTthtcUNOb;Q*MHkl`EO@t=HlPn7uvwnK2QN>9wIa3E|Z zaXIl;&%mBbFGQV-1IpjVxNrGO5FjocRxXgTjHwD>YnD)Bsy|>)c;x?*AVH2l1_ zfoELhoMFN^>e}b_501BJ8MklOpkDWD&~mCQ^F)C4&*vZ-TSKk8 zv+%KG^UkHmj`S`DWG}8BFe;N5K`#@5NqWT=s9%P1BI^r5*97p1jwrMeR_%+!_ZMvf zcqo2Qs(elPYXmqQ^20#oykq7kN@I5|;bd`n{->qHj4lo?+Y?yYA~M?bfh3=2$CiP-qy$TLaHqnL zcCB-h<8~I=Iice6`C48jf`yp8IbPcC@!+i41ukrH!C~8P+`Zy{JMDer!fgTKJ-Q5rzkE8?v0j3QSJe^+P59E~*OzZG3?dXx8;>bktLPt?4Y z(!e}kujO=!u6kJI@UU*xD1(#@8GBiBy-W}^Ag*W{jUHd!WpEwr(l$6&F#$Yz&qJ;s zj(@i1TBgv^-ihn_hj@TxKECl2lh+WPy??@GEd>j*KePb~trk$<-Iv;2X3xb$^&4+o zvtm@mtta;!j{Om6fol3HD^M78nCwHOCTYn-|B-FC`;0#aTI!ngu4mp69!lW??LNoa z4umF$IM^pv;Dn)7!XOW2^(vH2kPvghwblLAYN<{S-x z83y91{h2w$xZ}QxsEYA9dsnl2EsT<`3yu-%xqJwz_G^=+7uqt%vj$Su@S@TyN-3Qu z>M_rRly4k;{bzWPJymhyNoJK(#)?4HG@!>K4MPqRP^>SJ z&M$)W)K~rn(6Xu}Nnhseu`NAZ5)ydj4$c)K(y%8%cI2pwxq2xlWoF9=aGrM-%G-w@ zX5Kt%9uT=v3;uAIQj%~oTJWp3Bd=k@<&{b~)Tv+M0DNu$c=>S!wqWCEk_qrzAU6h2 z9AZn^$|cWS_#OU$-{|VzmtTNj4SLEV+$}icdO&h?(#g>zJuTm^srMC%J$({lM$oA@ zV=HY4AA0r;M&a_jXA38~?ao_|dyn4hT_;VqTjeiS))yVAWD27?8X-xmne)i0)7))`g0>}hAoM7iZrw_PrnzEY840cmzR9B)73+GzbpzhC$~OmTMfxvJ zUzQm^c}}Xl^GA>KK45V5fs@O;tAZLzRES>Up@^+RstgI?fwQMo+C62^fqbu-*5Q@s zBewJTSz#yOCQjsCgBAXFGJu=#mT2>=#UcHXdz@8sxZ{7@db0I4aAVGiQ-su@v+~D5 z`w+SUgnb@Tf~vLFfvFh2Bir)_y4Jh->2u4};^y~Ud$bw7D9%yTXE*VTPF@JwncN{f zqR4-us8L|GGaM^FL^8c<219E_$Tg1FVc)eFt=*}|IV%J3^Zo|jBeGXv%#=3{_I^Bl z@L}53mxh(f6gmCD)=^d2lz9)x;AcvaSQ4%)U3W;4Ey$vmbUqX0Sp&zJqVd+}e?Q}N z1RG~Qjvvk`Isg2D zKPA6ZJ!QIYXxb&TE8Za$SSDY|kbqF0hNK6)pu+&^rIhlT#hHR%KTh2IRy~B6xPadM zysAs10d1PK5hpQC2{KXaPFbK@sEPYKaQujp(-c~3{@Op+Pvq#A**BYzW8}^SXOOa4 zieU<|a_C?N*7gsfFu=pjx`O{%q#pu6=O=x>bn{(P<+JV_j5DNsT=mKrstz2sn-8I> zHS&w6)TrkzjcI7gr%6NpE=qA_?w(IAFxomzIz4Uh04uIjX?G5z&N!rD!JpXWxwGfr zpbpeZWBi&1BQvWJ@G`ixzq-eDeBmR_H>|GCN?#=V)#TTRc4R}U<5jwxAx=%3ZEJl) zD^v%tUXPFpr@P4JX|w4rP$9?n4XUcjTwv)1GO;JDfwXg_Vw2KKDA!Dp9C0w*w`r@4J z{EW!@uQkG*tR+Yu5Lq=0!W*mF(Nlwua|F}ri`pSm(%F=0Nv<#J7*Q{0A%V5(l8rN zl|!MVa0dtb$PZ>XE5U?x7JpeP2^%E~L)YcFs-1`M4zW zLT_QF@VOWb?xN+Q;s|9}%dezi!XxOJo(1g#?GbLIHo=97Sc3q6{ z)<p+Lk(PJlODbh0zAN**3=9 zU8SOZ4GE~nM8h%$PBg)mD1FCt-hO${6<$2^A+e$!RJLx_C7FFnQaJ&3nTJ@6aae11 zx)t8wM247~eT{3UI>ufkjULPIzf-c zU6GoCKQNW)B%!B8t`3?komV|5Ec4vSJ@WyKtdKg=T0(y3`8oJRsEWD`l8nZfr>f^2S(zlMFO6=)2iIM$)h23) z#`V$g$oewoqxJW3R&@X&2=+Jw^EYlu95U2%pyqQ-Yf@%qSY~LME48BG5xU+kbGQ0N zmli5Cb~CSBx@O4eC;iKI4-JjmB`~Gi=1|}1UmMW&N>uoH%daw~MH%gSOiRp$p3CZ| z75>HRb0b|9{&VpioWg-XPQJyz!o^vb+M8PxWxO&Um3WX{hNLY^y@RN)ROXWGXH0Ji z;ag;@ov>zIpBQ`pWqUS-;kB@GZ?rs1pWRNiio|C6evw(|tr*tJ`lBL7sDEX5jk$yl}rFv`Z?eE8;GQ7sw(K60?>C8C2Pgbh;b=d<(Ty{;fjH>0;QA=}W0OL6 z@^d>zy{lHA24vLX=M#M{wIpK%ksRk1RdsWiPkRkRc)15(lO6qOhXvq zHruYCab&}$!3bw1lToXpm8eiZW@u~6YZl^64!Ing!gnGT;crjL-1>yp8E_u{J~DPn zWH_gT^nQiQJtQ$P;8wD+D7EN-v^AukcqRK91?@O~JS?UkilLznD+57v^_5di)(9S&i1476HO*=lkAtt19YEGJnHlgYf5pd~F}5 zDj?kZ8!4MgR~9QvfBnM}rOP}puu~Q$yxG{$vSGbhYKPnKkFfohc@ZBv4Ua+Ve7UK3 zY%_sGX{XAj*ii3tM>8%pXC34&5yrmyvIAu$=bVeaE|5=t1H=||2%S#iMEkdDKY3P* zc)-~DJU2vPQjaEM?b=e7ZnZt6dS89373G=BU%EwUxy4dLCj_?5k;v7(uUbl$p0dmd zo%}X7fHoZJTz}`_$QI0fnPV9!p#8}EsH(up@xTwLMganbRKCsY{lnwFvZCPHi?Z}R zTc|dBYs*o92bwKOO#}R1&>6Q~bAG}-94pGCeV1Qjf;ls{4_Mo!c_;fL2Fc{r-kg(6 zSjRE?nG;1-J%q5q;}Zs&3ppD)3WLtfdtgFp{K}R!o7Y8iJ*rvRw}YO#O4Jh68CvG& z|2DDN>TToK_$cx0#mvZt`d=TSKepvSy$}U9W7b>S{D&9 zRt}p3lvVRDvdU!&Bc$m1Xhh|<6}XX@-u2%$eGv)_1dpZhS8L1v;TArjtoLRnmV4F) zA#(|xH_&G<95cY&e%Vpwr4-bXJ!MWy4TaygQ|i6ntx<~sy@cP;g*lisyk#(4Cf_ME zQ|Of&-YhW!T5RU*=3P8})c6=B|6UQc@*n=+|7~OBzn@9y4f|Dn#o>2hFAp_2XUL^= ziDxRy5qy=|eD^vx4M2bo#u=~9kZa-44pJxG1`0%=1Z2Bg59{#)9GP4nGBUF)9t~zZ ztoQ=ijyF<#6gszU=2J$>tRFjn=X}GN?^}u~mHGjJlOO@))j!+<2VU%1LEu;4sEN93 zVKqzYIV-3vKW3XqZXi1&r*|L@W5M2r=o_#%47pTArUIX}>%IK{wk{fkNgtSjo} zd~6ItVCb^2UYuA#9jDPZJLX%OZgKa$VDHgHc?$7v7pDId{_tlL^<6SFaD?6-SX4;ov zGbpW@zF1D=W3`M)cBlbzTS+JJDDK7qiS@V*goC$1ikJ3>|0a!pNB8CB+dF#C)Yf># zR`lN{d6*D?=WOP3KMA>?1L?(g@4gu0U5~(|0|U?lt1VO9jv#Kd>}VH46Jv0NyWv^+ zPb)l=RZ1%R`DhboieKR8cu}(oLUz6}?iC-(pk|pi2lktdX$h20kOBQk_&TkIfMVpd(o zdSCWO)PWk2zzdUD1@i|U%*vB#(8Yu|!RiY^A{BY9t7z$ayvE3iC~JxI8JOdU67OJW z@&$;tRZ)$!P)UDTpW8$&sU-xX!`8h&eXO0K4biLX#;7`xHSsk z`uDo_n`BnBjb~Yelq8s{l+OuqXy*qe5d9W~r+3Sn|6>!h5x8J+`;&WZFBYi4Sa?38 ztCgO9gflaWCO2UCMs}Qq+?kUR(D!hikT!1W&8F03nC;y8JTG1r1F@V-vue zNUBp?=!M4Stm)49#1e81d|m?!c?~KHk45j?i;6nv^*dl)rRV*d=;QR|i*NBXr~S(1J?lqPPNVd`xsMh=Emho8+c>I|{ zzGyw7z$TF%L>*%tX>JAhg%!xqf*ZvC=9bL-gtTKVx!$MnMO`)<&Zgfv z>j*97eJKm8Hj)5mFUiRUraKyA_mO+QgX|Rr&?Th&I>X0#H8q<}Gjky*@N6Q0z*JogzG52R$~?P!&uS8+5gjh;@D6HQ{RJD2e+Pwa)WAMnC|Ndp(ZObe!7K z(pqAKs&XBS7@6qPdor*s!L$eQlt_tep3-vb+|p1XUQXgn)Ls0;O1WogxMS|?!Wk!z z_H@KP;UF?pqv`a7z?~!v_ZvC?+S^cuG~^Uy1hC`_@!5}AmQeRfoWn)hnDcHD?%wy6 zqYbPflhpfePa_v27nfd+9%0{YMIEOUN|%jpHln;1k5GJR?a9wnE;V>-nyR)`#hwmq zk$V&<-&1Vxrpd#BpJfd3@ z%MDJp-_4p~2miR6u8J8@2FvMPnu01!dy_5Zz~nst#6t=CJPhsoUgXUI!cL8#+{MM= zK5P{(@ydwhwQv?XJ??%-)b9uCg09Ye%FSe{-5fR7RngC;nPSpe_s@#INyZ6n+E!!Ev%k9@RkJ%Yfc&o`thFJ zwEGPnB(eThCGrKpK4jIMG0Ioxp9(TfUXV81hgzs(4OA_6pb`d(3#be5^$={O9e%gH zR=mVd8jB{I0%SM9BG+d)_`G=VZ)L&taK7EvllZ2AmwopuuW+NJYk;5&V<2P)%q9ge zPd!RbKl$o#_)i4=4G>bk)M@gAJgYP@PE0TfPttiqXEqNugoqz|=#PXtX8#1ufN z$pI`z34$`O?N|qnI!d@V^zp09?qnt=SnU?xuFXsUPFjwuDMD4~zcT5LUD6SNfRaO1 z<&9B0(;ou(d?Sz(gfc>h`nOSTm$xk6zT1)_i`>ZWv3JvQ{a#f#3&hlWwYB``N)7`q z69CDu{&+3Mu!r3fu|_MTjPS#9Tsq0hzxX%lV07i&?~}|NVNHig$pe!yF?zR;Dnotw zb2HANXFm=iK|BE&F7-l#1FsH}faAg=L=7-f<~U(U7HOalXG zUkdSUTzw=JbuQu-)#7vh&;oDR!})SkK;XhCh&8-@fd1ekkK*FyrTSs&oVe+peDuOK z1!(nsbpz6o@D!xAIa3`{)_?F1_svqv!C?ISeWelOuS%GT)vxs~Ia|?7Bum?M?1rA& z`^oWF6}Q?S7x@()wSV&JGg%x*%?x74*#b%gRJwNW{mkBy0Qr1a{`&!~m3)pr%=)3~ zf+XolbpTvNzuv~#vPSV!8mrw>$XFPxN;bo_)kI&_GWRZY5-^C=J|&pMd}N{50$_Ak z_Z)3!(DN_XTWw7$hMr+f9}~0zsW5x0*hjhsVYDeo&1K_Zt((f!`?YW|@_zY%i?CXG zinrkC-JEu2_lFhrp8@6clo|e8_OJVl0vM^{)Gn_ri=?3+C#3nU`Nj8|=9 z_^sPzl_8^Q5aZ|R4)>8t{;f~XizQ6@Y2MimtLSR5J=E!lNSS4} z@PhMtpeexhWY7BwQr`{`5U4HS#@m-hSt~JrEp3)*tQHGj(QB2cB4k6PZr6bP8}(%# z1(UH?zZy`sdst>;Oy4gJmF?`8@9f@Ekv9x>8RO*)Lj{5*Q}Vf1D~H@I9?gizc)yGexRsn1V$=jQ>rP{yC> zZuSe7KJd3AwUKWgE$sNf#7?>Fo;s1&F!v+Q`WGJS*ysD<2ze@w4(Cp_vjXZPtTuk; z_m-Jd@htB2TM=W1!SsJyMvt@=3~*jDa%w#xs?6G}uIAyQ<}T!x)f*1pIF1DEpAw{w z?aQd~3~L^THuci}@TwHpfa=rp<3rvnVK0v5@;yhPP?7GhoDJh?**%VUrno^Bdx^V} zX?=|@UrZ%=*5Q%^Tie^($MhHB_A_c3|55=d=)Jn|7<(30{n5YvcG$1jS!d5`+-csG zme*qdmSGx^;`<{UW$(yK1NiCnOwaOp6C)thi!}~E16XKQVhYh8<<7JLV~SM7XI}s@ z4|HjK&~S;D*;;>#15fYn8cManJE>Abs=97VF#A=7^Orb9UZc)_N;r@NwbN`_hzy41 z(=yTQSlXkX3~qqK&`>mY|F8I~@{e1N%@p2-^%UH%)8x>b<}s|1cc?7ZG=e>Y0?y(g zJrO3(@(HaXF|JBPuCKJs*x#1t!`b41CdTGW_v2j@8UK`+%Mo3oXaARK^AhSS=#L|# zM+m{CR1yCedRHf#<3)j?U1#3=TiSb&J?aE;2{S2|)Xs8<0C)5}YvfhlbbXl^asbDg z?*T{E&+}JV6$y!Vt!37A(=>{__erJKtSCSTCIRNm`;OgY(jLQv>aw#xztm%*rnR+z zmnC!|LyXw#j}tn%od0QSz^%9%NgQttszR@Pw&IT5X<4@AC0~7{T%Y+KZI-3UK$^F; z(Q^T^GE$7%+_1a3WPsU6vsb0zAw|*h(($W1}6Y)CZ*RA zcKM@azRP*8(`DhUN1CW_7i@?2Y>gzHOqYnJsoVmrq``Ek9wHvpu|7ac13I_Chrivr zFj)ETl74|+lS^$tUz85Q(r-Djaxg{NW$icYtgC33EE3B%Y~@xt@}=GqBL-%l)DW#C zo)%eUU@&Og+S-mvB%9xdrm@+Y2e&xt6BQ&#Y>4qYonP>OXbmO%0WWI3HNZ4c4c!7d zg$M@Wg@2yuNWeumnyJ#?AvNP$jVH96Oeuljs5*BIa{bpxs#Y`k$LXb`a)6@R$k4@y2w;_VGG8G<0U#_l!30+_PRLOO#Wxy$<< zxBDT}JXm4f39M)}rZpK}%paYdss}1?rd{!g^rzA7+Vf~My87$7i%<$@O1I!`d#*eh1Oq&6`X$Fo!H+HSbt%bNCW zD^XulUv7D{&bh*#&EkU`VKa|TSz%cHX9tn@tEAzq{@st6=HeomX-E9N3YWI_bP5^Z z?LNs)JH~ZRY-k&kN@Hctjp53X9%E5KEPD3CSLJKFRhbmHKX}EHMM=Z!llAn5IibFs zR;!(yWUj(=t(lx*lg5Jcx71z<>XtigB>+JOdAAX`0O=eC?yD;U_Ww>TpP)R9urvKe z9-D?a&SQcS#6t$ElK)Y3?tx72{~xc@?R0#lQgjGiP#wmpj1#k+lBC6olgnmQ5;LG{+`)F$=q7*m6I$B*vDFVam0!h1qf0`F(!>`7_x*pZELqdOjZyQKuA& z1qy#g_tv!c^w&nm41(!O?R=1;WlnuzV{ z4VLjTt-%>Kl7BrWJK{+%4*rGzepX`wWD!=fz1?!A1pj2oy=^1xR6|Q6yr*%nHSoKI zY(rUAFZK--4CIq^#CGGN4nAz}4A2a9+zGm%9X@$IM@h43#y$thX>}`(S3+=s>*#^a z!mEwcYfsr?8vr2J-0*B;GZAUZ%3V`;{Fhrl{N*rnrFEZm?| zz;S*49!lafmw}0hE5YhL0h&srsLSAk+!Pc>Y7L#zsyabg^4PM3I54~cR%l^V^U6qg z`h=#jTFr4g?o|J}!0Gisi<8Y0>=WdiZR}c$UVq}lo9jPSjGjRO?|;+WucAiUC`;7A z(<#OOa@abm3O049K)^m5fGhwv1GpjZ~uczxbt+|QIUZ%m?hCd z&MGBmhHDj8FRIOI_q9c$|Np$FqfY2lA#pwJ_Xb!af!wtDbt};9ew()t$23H}J;rJ*=p#ul2Q~r4SSIvP(g4T67ppm{ zw4&`O41u0WcS%_r*cUs2cDUrbF$1z`4Q*fOFK)=kF=|kJQl`ZrG2EIt`(7|I;`f^} z`AuTXY7cp}Q<7+a}bvFEMu#p~vV%Yk|P%ifK6rq7t~ zSb!f!9=Y>R@$G-F`HzggD?W^P9EI357hn|76QE0m>8c|;HOF&s1`^G$t0dd&pL|ih z$>ncVf2}-^o&jrHbs#9+!xWV{iyF;vOWuCNE=6O`%ON_yM9&2517-|!K)z6bgPhi@ zW5<$I_GN$qqprG2Z0Rwp3;?q-fZOZR*E9U>`?dSQ~hj>gYc$}jaAm2(#h3nIQ-_hs8PSp& zx|ZuC8rJhOxc6aeDj`%XK37Uv^N~?@-6wVTY~Har?)E>}72?TPgGr_%0GgBckddrC z3VdzWk{|7RQLh;Qh|=amrkS|2>j>xO9GM=M(zCj{DzXoRL%eNk!k!-IQ($u+~WYEI%iBZo^-Ptp4)r2dO4jy(O8u1f+mn&HN z#%)pX>wi56lUZ`?!A<#ly;a<-L3VCV(w5MIr0oj!qZMl)?b-hGh@USODN0)&!X~TX zri`ZM2Ek_rlU2{f*(wdDzuLxL1M?AIT=4Pa+hX)03d7-2<7{|anTdv16r}3gW0HG> z2qxmtoJh}prStl{0dTVP0SSx6FUM+GENTezMp2kp<`v!W4Y$R<@6j5)!&%Xn5!|BG zJT2Y`d`&@r2h;Z}y3hm9V2s|KzOLz_`eb=Dxq<7v`*e|QRUL6gwbiU)NEJxX2mO(# zqkd}LNo`c~T!E<8#JuFM@Tea*ZoYb3iw36vY!zXVEK7nr;lXNRkp>oKze*Q^Mz@m` zK5pyt&vrdg10D_ZIRob+JAA@VOa&u;>QJDBajjR(o0J@_NL0$$E0H&P$A04+#uE#++iiZMtI8nOZ- z_DjT=c9x$ib3Xj<;&6e0%IQDH7$UK6AHzFzQEIEsfD1X|%y1HdnBRf}R&3 z1Qq$TeU`;2MB>Ev#M2Rq$sX~0|NZ?$y;T3f7L&;FO)puOfzd0bw7d-Id;m$w=ImZb z65QlCKlV;>C@gfn({$tUwqD&g6kT-GM}h2(;b!q5{;LU*W};jefV<2|YOkw$4DBq(aPMOPz4C#l5rC zTSH!WwVT&Y9+Qy;Z*9tPyXQ4<(8{W~K4o_xwt1`p758)N*a*c`ouE9i?lR$>wQ*~r zW_MVUiC>6H3lqCD1KV4&_8Pet?w5kMYNjp0>AwE}=nJbErn!ccIP(w5^@2DkyOQ4?P@uC4j(>W5jOL5->mL z+mV8InW#gxUse2DQ*GKU{{ucS#NZ(a5trmZv{#;EwhET%`v{j3oGs|s0%X~mZT3qW z0Jgcj4!3Za{|ih#1D}nF)8DuF{0j;wzEtW@s)-3T1mq$@T}=<=)Mbx@t_ZbvE_M4G z4c=iZZR}GM>FxuIcNzrZg{b;mw@f!5pcRR2&X<>cyUT9iq2jGl*PRH*kpgf)R>5Q& zY}ZCm(2IY>|1-d1$k05g7e(aRIy$(-PRECZNAC`oPsa%pMlW}=+a*F!+J5Je?nUkGsjtliWN*kEDh0&ngVJ2bmc@XOE zl4%-WZuCIBlR^xZ`2a&-Jp%S>VGo&-PXq38K}xf3W)-2t;hfd-Vm%V&g8PnN)lG{$ zTpyxudq!+FTF%hf@QB})!ay7V(ux6lAUkAoi!Q*wX9un*BsRcnQN6X8bkD)b)|NlU zFdPaw<7HV*1cCG5yJ-)vJgg)On}Bng($U|5s)r^Vs1MidQd@qNnhc>jkfaw85oR08 zjZ0*h-UwD7w_`KEsC8v!IX*0}Yx*5Im|Wlix6#IHo;p_T;t+Pprl>NWmU?krr=x)o z)8ozIw0prVoQRB_!+O zH9E7>Eh4C;nzr#{V$CReo(+IPEg7x15&9T$8a2^6;|ufkpd5Sx5G0TLn&O;qX9~5U z2}J{qP6l1Dgkap{SO&}(n0BulIsB^Jl}kK_q3dY#s{))u@r+IDPJKO=(4d0U`1S;l zb9o9;u_Jt|TSIEeaLMjT@+smP{tpFl&&P1630j@%+Ncc9y6wXR9Y>^nbxll63<7hh zjzFfe+$bxARx?swgT(6Dio@7vM(Atc7M1Yb)?inKcXzH)Mk~D zXrsV-#j_<*{8=la0|vHs=F>2V$;y*`n7`iqO-^o~IyLz^?a;z$B5ZPT4{+^x<9}2= z8u*f#U5IJ0j5n{eZIZ!@P;N~Hzp`F84}GNvQm{ak2ky@GiJFhf`V%*39$$vSj)&m=gjvD@)M$o7GOuv zZ4sR7krH#Kg;n@I|5oa{hb@Xng~ATLQuwnKh#MaP6o$?&UumenV_NT>=8s|F5W5#6 zeV`9HN`SZvD9M~v4gl?A<~#AFr*ltQdVd=JfxOt0yJIHc`;7qeqAF=$;7fI+H^M;e zwdhIpP@3|@y~cmO>Wz-Cam=YK2`z}2mEzT<-6CxCEo?jyx5G--aT`ms{Qb3nnaJ&j zxxB-ISfQd^u zx{D$tC#y^YlyWon6Hp;uyvQ9hBK!$EK(tU4xCx?l9?eJ42~mFRG;j<+tCB8t#3xa! zB}fF#=@zCcw0_nLh`D?D-ynZNl>Y14ndcM~FBFaiZ&}7?0@-d@h|ci(c#Ra zMOtp~cDW-xi?*Eqac4n!sKGr6=qnP<_*O2nC53&JH6)7O$&c6AQ6p!OI$5Ubjs)3m zK)M9KUg)7H8m$TCn4}P431#R|_TIKcZ827>%34xO53da?7tn-gS7;xYNgFGgM&P#l zM?_5Krb`{w_kmkaXgwimvIA1sLkw!;{YbO}3bfB~ zYf*{VpONJDXCvxgdPgKJPI>`-dQaux;nRQohA-vag4|}TbeX3M99i4c z4~q%8egihS`b3X`J3Ily4S#h&Zj)tbEkB7r!f?AF@EJNH3P%dRrz+aKMN)Ss9^KcXxF2^>(nOgS0j8=+bFgli4G-wxMD(IrykdgCt_Cg<82%R5luL7 zv^=j3vKVVzgFE=;y{9b|v*)c4oHHDl?Oiai60l7@){|tdey+s2yIKLQ^NpimZjH=A z%g6Ww0LP_u^S(G~iEBAG8Ze7X(({K`h4E4S{GDU44+_{1K&;j>;D0Ma;ZV)hJ zjg3EGug%V+ui&=5S&L@G1SoRfzU-JpZ1wNK#p;o1#m+m)Y~es>_BK9X4~vW5v1C`M z3c2`lD@~yvAWX8A0z6;3^QbjuP(e_hsTw;xHKuBXBJ~ll2McoTs%vx3l&6$a> z3uds^aStrass3;4W|Q|EQS4zVdnWFtZ#Vv4NZbZIUUr*2xWmC&83ybabB(NiuOeF& z?(E^dW|O`-V0xFY(K`CpyN2x;^lBJ9V;~w9v^%mDXs0*&J{B)d_M~p7Ldb}`nC}n0JHAg;!*!f z8L71R)5iHmH5l~ass2NV6TofvKlsdx$>I#klt$gHnBuWdPBn3MPp^Q@^5U$>2v+Fs z=@I_R^T%Gk`Jey_IQYq(Y@MFmM--5d;q|<*T}jwyJ_I>ko{Z_x=H@2lzI2H_$hpiJ zdfy;%HS|X_cANfQyZU<~rH>B@3@V+CcmDsIp3cVk0mi7^mDl94ONxR=cYt7{L>}v# zqQb6g6}s6DEOaK{9D^V1Vo|E2e;O%iJI08$_siTmLfj5WafvV`B|Ub0M}W!3UZu40 zEa8?|hJ*PGA`K&B&(hM$Ibt)+ok4v0Q%?l;x%@dPghp;}Pb%ms@PZyzP*au>hpnkHOFD5MnfVOu4{YwuXci;;izBy_8=6$eL$e^|WxF5-cOfqwi=2Fn;rSC29f z>44##^}q&ZL_9snD7w3<(XE+fx}9w&6_Aicnxg+jo`*8;)rx+0Im`>YyjVx}(c8gz z$|!GGf$ao-?IAUG{{~p;oyBli+=Co;r4eFQJ7e||ZsEZ=*j3!6t+C~PW`DE9R*y37 z3Y>!uFSWt)HBS#*8xbS75LaYHC*0~cOBU$Rt!n6M2kf9V7*J!X#c_1pWdu@Q}X0(9nSW9D8s2e?%f7%bJ zauXxyKFOFlv3mxH5jjU9*~SR@j*M3r<)v2=mk8Bgf>>2<3wk3Apf@N^w3Dx3u(Cv5 zmwNJ!L2!uNrfBEdfq~oP0mn|l=YBfGr7TpnW=<=a)2_TE6hmu5`e!_t4|v$j=*Dm9&yWAadS+;2 zv*o`n+5Y;2Eq-K8y38c~c)RqVBB$t9$&8%=w>u%^_z2bcr#CbWhafLd?E69@NI z?0ja*itq`QmyiA=9(lXxdC90c=`BOlZ{?2~g|;W~9ipC>WKIou3&?osmM{RGL9TpU z&P>Dh8laDRDRrGD)Hm76yXiFcW4z7tXgF1#5ER@xu>3C=?gRD=-ZlIzw4>^6Bxj%M zH6%!hofzvFz3mv;Wo%flcBw7@kQH4y8z26P177)%fr~uI%@b7@b1QeY21V}pTP3~d zmB+8F0dYE1m_iS9Zbk#aWnxcigmVA4#BBiz{oDeN%H(RL|nFx@)Ou!rL7n;-ZZpqH8|P11PZX%5U$57+>y7GNG)6g6zzec9S(6x*JKGk)=+ z&5W5hXQ#?Tzdbrwtm$jAu&u0Q{Uv@7(Fe>B7RJfT+h(oPdHabjySfpF&R2Cz;t!~? zU0f^0%2dAb!)mves1TKX_RCS=_OauA3|oAo?3CH+y6o(4S*fKjL!U-CXAd^*lm+TO2{pyv&OzKjkwN^NTuDBn-B^;UJk zq$$z0)|6ASmR6MO_%~=L55TRSu--KK^}j7|`hNIt%TnO4CjSXn@Y{&4tj%KRnNi{| z%G|VDtpKnjUbxgn;+S;|^R4u>y->jrKu zzz#wU#OnGL-Vel!fP`~@i~6D7bF1viJ{#7mi8$|kVg;zRyXJrTKX*l{%$V$inG;RV z>nhj50{>6&Ter?ajBUErbl_x=hD>XI*^#K|yQok-tA$c0n%%C5JbAe;ZU`5lC9Slv z`I(zNktM?KEHYqH#qYJh+pLiJOn;6-JQ{1Hu-7dKsaF-M8iGv?Pv>oUi1@|>aZV`> zdcWFkxR!QYV7+jsXc)ifd4&WwB&A@+GtHY)eo5-Rvvrnh@mf7xPMB_QB-*KgV_bml zhI2%88IFld5(~S$w1zUpD#G!c1c&)MoEFY~rs1Wu(}$721kr zYQ<##Jo#^3@o-mkMNjhK3u4hbub|5%O|LW0hi&C4db*SLDpaKr<9$1NBaEA(WxiaK zS#jd$$jyR|HQQ%KWl8AnDp8(a?b=07i|`eE$G60aZ|h&ZB*p8d!j;-2BkUSI*QN<_ z8aPn3kL|Y#%EP%wU~52iz*SUca_dv5ifeU$y*0Y*tb+LljbnV1mB0O)5mtD4|7rG3HB0hPsS<|z-3=y`bw zEi75$x?FupoeeV&aO_m(T(eBG~wIRG&WtRlp&CtOSG zyAop9xj*#GTioW0+8O&WtR;(wxomAEjtjoC-vtOOpxUUx0|67TPT)eD-fe8I+?4@= za&7F(Dng=6_Vi#QAqL4}=}Pe+Jt|92X~y?I8`{8EUCe<$Tt86rDz^*BNoRxyD_7h5 zjAl7|ieAjSsb-hQ^rdf$#bbmd)dIM6h(Y*FxzV7!lWn?sfF7FGJ06{OKY(URt(OBS2XbmBVS?LuM%6ZSQ^5zT%YX~bkxu_|WX zuR7VxfZYMuWLy;^5uTKnmBUnvPyg1Ou(ROm8qELBfUQDXSMpjur zjs{@V`v`0}XK1={&?>5RrH-{V>~-y8(w;69sG`tQ8^tvFBSTHM6A%y@3PrA+{F zRq!T;D5Wy%L&URLom8D@Hx~H=9bid{d$n_Dmhl>P+SCr6Yn?XZ@K>>I}*3PJqBtpMue$@mDDbZxjKEKhVM&sh@j2c(=$x# z^KiyL(!)fZ-uBEgTsb_sA~esb>U^---%H)RyO%*4L2~5wD{l;`hd?a=Ry_^spl+^I zH=u3D+$eH}q1(N+J|7GirX%q;b+gp-zZ0MKA`nE9+-b`5E%p)qx5cBWmHf`oLdQl( zJT>7MK95qkq}u#bYZHl|P%|wMduK3}ewa1TrUU=~x`qPYUEGgWLaUD^0 zL3MqTyKvH~oVKW!)Ne-?{b3olKx;uO=3x8h*2<*EhE_&B*I}I`d#fzlgSCwH%Z475 zUo2zc*-$`i&&epfmN%ta?Fu!?;RR=a$1=5Lu&i3HouCDh)K;n1z4{kM48c}@0&k`j z9!BB{`M+~4k&mT?lQz}?_u8Gjfxav`#cLL4A6hR8VgnE6Pw?co!`P0~Fpo|K9{+l+i zlRSM43%RI5q*CTS7`nweuis|92!838nCr=0cb~aQ0D1U=f(!m3NGCl4*eF2F63sKf z*bCJ7%Ziwc@(u8ymwFV1b}E@Eh>5>?9mB6SQx!p0V=T;yYU{M>dr~q;dNoKDF=^gyE{;zeD=mdD9a%-McXYXjo z&KGezYoJv7^AMAndw$a4xL8{$+}m;dzb#sR>Y5(0U8crdeQ&eHRVGP5MJt_E>=<%E zD)XOz@@U)zFHYQ+teZPCvf9?03F{_8I3!QFzjukA6Kd!mO|?9OgK4Q)5PdUx)OC{DoXhAc`h+4iauK;YQoWRRxQFM>9&R z`FUBoUriZ*L9OcNWm3&c_XEJNiI%x9BqjaoqfY!W#=-ocqa6WuN$7@ zy=ObKCl1vCp8J(nTLJLa=F}Bb_;IPmO_dmN^oYVLsEL>LwKr6H<4f7^mjwR3*h9Pc z#)lR80G=-}RhCwRFILCP)~|6|n^Mm=zHSl$@fd{9>#R;&+kLa%_79|fNJ&o>v%SwE zZg-e)J*k_0(`h{6pJ)?ffjZo8B{NC^BD|RiVE7zHU->5VNB7W3;Qm z+Y6iZ`JbPHDhv78Q_w!y?m9xGM%MutWde1(r5TN`Is-Qi=*q9DV>D6+xuI(_R);kM zC`Qn375sq)GEx%K5d4+adX{|z0Y~NdtmI3AVO)kRO^*!!w~ph z?@}YJGh=y&9+SN@g9pIFz(UMJ5VdK(Bqu>lBw!Xr6@P6K0 zRCXPFSk#~UKJTQJbw-kjQ$}xbsLrCuw7I=}((`n^H91QD;*~j&> zpbxe6NI%Se=bOh~H#Us|^ufHu7fFJ1&AxZGtS`B%&y6qMhw7*^WJbwXo79C7!0e4{ z+#j@VRFQ2H!}&pMq;r|U&HRNP1gy3zGCaNw_Hh=(r}c*yZl#I2c%`OF^SvOf4g#8R zbdkl>%X;xW_JCC0dO;rDi$JWqea`dsn?(kdDcd4n!Ylf2vt2R&Z844yOKzzT%3XKp zBc2;ibn*59psDRbfuCSSXWtB`L~JTfqQwQ?GD+t#h+yNw>b@#9w)3%~mi{?TbMXo4 z;=4vmJy9}VuCXFScCn!*AHl7LU8g_6vKVPpG(!n(GGeD&+2CB4GooFsz%(n+QEL|o z@w|O!4M_eZGl3=G_C|gYpWNA#7S-#bdfcdFx3?yD*=f%7ub8uM2*TtmMS*Tiy;r_- zEH8eIw%0TwpYCM{V?)*$Zrfj>C7*a3SChv4DgdHtH+3(>imTn9WUjs>>VP^3ZY7>y zDY^jLGsj|aM8nq6S?j>+Be7y&XVD!JoR4U{B8;_(F5!pen77zmV zYK)1$bJSUQMS^UXivHL}`#9iNXKW^WG5aU_>!8CC6O!F`^42|ATuj|U_%ja@+J4y| z|1>NpWhdpN`Qz`{U5k$jl!Vmo0IjIrfNL%IgDAg%(4;dh{q#V9o-na_wnOjCNGl&+ z*>yNFxwCujvshfh(2f!Q!+I7_OL{hAFYuowC#P#{Lf2Ni9Pw>l-=Nv}3fF4!toTn# zP(=|qf3voqRa!fWI)C`tydek={ilHQuuRo9$x3rW?8heWjmEiLQ4ICnFICXnXj)fz ztt~)vFw#@;o=tXdo#<|AQpkEDzhJ(N`0BTi%v_0uhd_EJXAwTD{4d65Vf!?vHjD?# zGi5z*^si%~M?wJ5!F8p3R@p2SlfHh&ob9us?`AM1_F0IUwG3Xr@uzO>)2?G3$(V!c z9eA;~t1_v`uA^@sA@@zJJtB)}QAR+wAOO$Rx;D%ekj%-wEH-b+>D~ZmQhs8vjP)Bu z`#jCoKVSiiW1khlm8oy^{dcGhW!5`FR{ej_^sol4>Q;L1pT;`^84(o^KsC^ zTe#R#S9|vpDT(KNi<`NbI-_vN*-zF(nnP-{$~IN}46dk&!Zd!!-2`cgvi=o?@Z70=aQUNhZ)IBKazW z{QC{Z*b?9OBJNLKOhNfz*pU=#W?O7E?CEy6S$1zDTnmYLWm8~x^Dv@^t~aCkPMQ87 z8Q9GAwB?0Tb|@S->FFypU33m!dj=UKEO&hPSsNRaBy_tyw_g}TBET94d~x*a$S>wk ziPSdUz9Z^e<0o5;!o|~%S^~xss-i*GaCL^Zro2K8b7M9z2EPo3GkVV=0Iv(qdez{L z=ewtGG+4i3n;1a*ZNl3K0G1{roD?>5PBPVK*Aq((zD095KwgSE%KC$Ik1 z{UcaM?rZShmK&^k#^vd>Ukt`Je}E5VYEdr^ff3u1yIv{jxmF9rI9ayG+Ps&*B@4eG z7I=_B$2~YKw-c`Y;%Rr%0$=Z%?TeraR~(EwHOyFzeED%FutfBXQ^t0S@tHOfYdtmA zM@Tzu{$hd_>Uc$hV2bFs@k~bK`NRWCeiMy5An8BWnEJIJnfUB_mdj`E5%t`{qAVtB zoM)^qnQW)%@P-x_!do0lXW6!Ef1ZAOkA_L!9`iPKdTw5}d%B+bbpSCH&K?S*6^Bd! zHN4Fi8V5q3%)MNBqTOID4PrL|r5Cr$2`_JfkNH;(*Gz`r>rlY9Fx zDI!<&`Q_JMjz~h?S5IHzV&3Jv8>AjrLEX({=sP{isr=GpfU-)+Tt^8E0y1wq7#X(P z6Yc%W!AmZ|(byW7*d31>u}J2Z$MG(i;Zc$m4h8>pcHx2E80HVx1522OL|@<#+byP) zR})Gn4)~u&?3r2RgjG){V-oGZ=N}Rq{Tk(wPQ+ zg?QF$&3;}iQe}^JSBUR~ucf_+OvK;mcb)FT81`~x2^OCsqp1&{z{eAvRIv(5Hb|6d z1KdJ=2O#i3Gj|^jN-9E>~YqM&AC2}VX16w%yiZs<$H!IawpH$t7gOe zW6QQ0b)+omdG&jU8)h;<7waW1r%=l!zVqTe1*JWp-tu`ZZ17NYqQ|ag0ojGx+rr>9 z<$my47^lXj5wEj_fT8{p7p8NhUQIVIuj$#D4}5K^SF};^K^mSO#K}58+Z*LzOxdNd2$WAmpvvU1K?r+Fx9_Ys3ypdAl* z97{kj?HykEM`1?Y7igbLI|O z$@LQ8*=IoK<(6fv9K%8&p3zV)_~YD|K9YT(XsDQUN z0Tm0)7=uXyXieCfe}NS<*gjMrXGR?^!(u}!a^>|aQiq{uvJ>6Zbnc!eCAU@9_fUFd zFdA{#Nw0EZ)7jJ~xskR)J7cXgC$x2~0uQ=@mLzfdr{M>#CuZb7#vbgmR_ZNw18H3R zc%MEIKG@<^<{zpj&Y^%H=Su*6vz^E*7KY~(Ezkd^1m!<&1Zu8f8Dy-@*llYZx6AC| z`{bKa1)`%>zXJ+g0(&Nx5S5OhV&!jFAZb>skCm17W@`X`kA6ViO@te*}S*Zdfr2e;ojh@-Jtq1KHOBQLQ`Q zSJMw{XY|TL=Bpt{09E=Fvd=j-Ftq&6x2e3=6BiQklhAbSe0n+q_9#HXTK;FnfqKxF zRT)^)6QOb_#J1`qo~uGXSU0s3ekwEtXQw-m?h&W_mmB8=4VX{4|6 zeNAj6up(9d%wt4-rgn=Fk1bmcLC>ih;e%1Ycd*s7Gp(<)Kt!^4&mpnxZ+v|PS@HvY zE_edS!p^#G=iJYmU%!lm?*M{g;SwnXG>J$v+1rAf=)LC(eH*PaYM4+s zk$%6TX&^|QU)=M;E-lJIIQVrr!7uedox$7dQH{3L?J+e| zW&drNDexyoxJ4e09t9-PV+SD4$@tGKnY>z10};Svl@ZLK$BY%u`XK?5?S#}!s|RW>Fnq&7N0p6%Qc zF!6884O}f<6;=?-aq^A3hMsc^#%?b-ea4TLuKIHxXA0l4D=<*0`^G-D^S>?W!nJ@@ znE*JuChWIgfQ2w;=^8MQlDJM(w0*pm1 zo(_bgpXM|2ldATTnm^qiA* zVcx5YHtb7lK^&k7*WeaH(&`9F-fP%Ru~6D&PGHD^o3J-L9WZ^Le@}6% zlpB3Hs>lPcEB2WevAIFnK(8orYEIiS!k;$yGqCEgIpg|!jn1&cIS|wUSkD)9;C0ph zz&2`}XA&whKBF{#T57q3l+GrZw6cb95JZI-?bk4?w0;^4`tUqF&z)s=&*jTovy_%t zFwR0oZ%Y`x9{r=uX0XSQ^#b(xtbt1wGjV_4)imXfR%!h$T#(ID9|jWnwmMWRa>t-h zWq}V%5}ZXOPX)~L?^okv`&&7!EW$LxKW0t13NZu4nbX!5XrWQe_W z8jk2d5Qo2hB?N>1TS4|s)Z`ked)kGu zi$u=D+%E02o`M(LA=986cbkTHK4ASG8snAH7yi69v5}wo3vs@X5MJ)@JyDQl13QF8 z-d2xpoSih$BeW-?EcsvfgunE2RKXiw`asE#|mN#6Zs-R2vCv)%y+4-jcYPoNBKKb;CZ9{;EI>h6*k z54D_&s|)489!?Cbls|J%G2}I(-?7d(@GT3@NSt~h^*+|!2c`1)&ztXI}tNJCJiNpG^QJ<{8LMIVYYuuBzAtC`uSuJ|BUIl5~VU07&O_*9Nb`lR_yDMoWk zRkK5ZX}+MT$Jsf60ye~ELPlXt7S2lRcGN%J)rqm& zS@zB&o|v^eBoQf+)AyOUg!;)3_9r2-O?p^4oq4ogAuT#sF0BlXtsO?s!sBZFQJAD7 z{VNv%*&?IILs-mh$@O^VDJbRu!XWw0OW(#@nlZmdhmMm6or-gx}EyCi*an3ioh?t$IJrWyh z|CoCF8_&o(74X^7FDX`gTJ3k2Dt11Ps3p9E_gcFTngh?zv$ZM_#_=oCo$kmk+&*uq zLRzZqEUNt^7stL0oQlqUq_IN)>12hdv#cND#0H9dS!Gx|WJO_Zs6MIDbq|RH(JA8B zyuIUjB~R?E>C4k=#uj;~EJ*ZcEm}2mYtnjI-*)xJZIaTby~Wa?cXHPzFtn zs~IDnPjGbT)T%@KjLrchC`{2$j+YWIgp4GZej3QO=h@>**bp7}a-Z=-hBc3rPze6k-)#7o9Qvy1DgwSnu|P zo8P!$>PW>JPUpJP|=DRGEw5QGQOo`bjoyH5t2xVq1OZD}>xqcPox{7BqAN&Wxa2WUL-((KBboW*%j9K6nT}m0eeMzq|mE;6p3^Ru?V@kxVWD3zOi@{8nNd@B>Nm z-nEECmv(%+WdjC9ry=j-S}_I7q>0R|B2R6T_oP24TM6{V_==O zz>pMTLE(l8iLo7aqH3#}2{anSJY^MVtd}ztAb%TvB6BV-XrSdG{Oy;tbGQ=)vAt_L zZ%lOFj`McSCD|*X)Y91`GsVP58|{RS>z_j7!&8H?6{9`L2S-O6+Bnnl*jSH%k+#@K z?ayw>5SH-iFi2j|a$eDaQrnqiCK-(QN{FWmWf0Mm5a=-li)NrE%jUXQ7*$pa%SA$A z?_Cc1+ZcZb>66WJ=F}IxcXwGRyt%rKu3o*y7DSD0R|8i!z;E+lNiugWq$VPvm;iiE zJCRnkz0h0}3MWk0bT#JL2evniL-HX>I|7lR?AooukeIWC_MVBfVyRy0g@Izk|7JD1 zh`ph1u_gR6LUjq%w}GYk#L7c8zm97C6_Xt%PHC&U*x-uU9;>H?!_5Iea4?*@2WX+R z0S{?f<&A9SX!w%PV5w$=t<*l`TaA>jv9E;;dr-A9Y%k&h=>BddUicZ?okTv(L5`Sc3v)|B!QrCaB;U^#k;2p8q z@NPhBRx>~qvkJ-Dja^lKA2|L+TJBq`Wx#2d5P4N;m&tY92pHIio^XKe4ON72jWzv3 z1JR9-}5fx3%rJ!3Z4@{38Qv6+I}t@F@6CR+)+ z)G$CijR`L8C=KCYhnZjAUpt>{)XS`Rkr-U&OkfK}P{}<*loz)pv8cGmT*M?VjcYl$ z;ascfqnyEp5Vd56+9nsX+WfbzBj?|X9Z>L~s!>)`@JWjJdm68i$9ndT#mNU9Z_X2+ z9Hu#D{if_mO}_hdVit0JywytNZGV>vut0k0s`9T_tPQ+btXG38KU&gS9!(2mjadks zd$;wO#8Vnc1gBjr{}s~M4coq-$_SMhipve-xd2Ak+K*$2;HdM{-J0 zgf65GvvbNgxolmIq@`8LWpk+{X3b798{0VDD8h-;cACQyGq(|T$!#mwO2WpL4KvDp zY+@GU{64>b{N*n`m-pxOdOjbIh1L>ktkkJHGXD33y_tu|$CQk%pHCEi$C9k)+TFG% zMz|%0-8)mks@AOa(jG1ky=&~*L=92@LtJk9YkYNGYETy$H`L(N%z}KGGgW+ViXYbVt}^d{BA*2=0MlbjhSl_>c@!rpC=L z9|mLVkKKMb=s#sknj>Z)`>o5B){YwM6#mmc*Q0~8bu~d(#S}iGix9yzj z7|44bxO()*h)}yp6X}IXsdUvz%AvqcWiDQ75Cl|=tafRfQVF4j71|xXRLQS_=jm7K z5ES0R4IVz=%ff5H;qhFfbn}YjpAaM9N64-A`Mr((Zy> zmE_Xz=cP|4;>l_5wQy{CV!``6OM<+u&n>k7@aYfdwL-utMaockebm*=C$1s?7GiKp zdDy`*B-u<$3#%Z8`*m~d{AV~FCq?ejm+y(H6u=6D@A;_`&!{dklfK~? zMdxX;tCu@L9-68qrStuTV_b%b7wicc!X}<=ZEd=k?fV=DE)?OydQ?nCU>#@frKkeO z&Dlqv`=q&kPqX)NKIV`E`Gw)EO#D{G((iq(HhQ?P22KKzE^9V$iwFA9RG~U+vTMTS z*-UOV66N2s0!bGQXQQ^&w*#g<6K96dXQDCr5WSxpb=fXx@|Wk zPKC#9(S@I@<_S$Uv9Bef*%Q&8CBL^j-U&hjJs#9qo_FHypJ&p00dt%qVux_WiT=vm^MP@CUcJA;hjhva81L=`aX!iPMYEIW}!AozE_H-|)dC$V}{sLhXhG z{U{Rqq)u;_{a$6|^dTbrplev5)wT$My?Ud{R_Psj3VXB-`b)K#8-(;Jmdl8E!N3;A z%?9Cpuc~kOy5}>h{XHsQ5@z>ME_k{ytbqjP@Li3XJM6Rs%^*viF(ORHPq~BJ25mFt zX9eos-ldMBpdb%sXG!o1-wO+!z0^RvkhuX{8*_U7auBoN+v(D(3@POI;E(7SgVsl?DDqY#j0GHSw}H`OyB}m)D(((x!l*tpWId+!XWZFT61N%{AW? zJA=tSx>?OpA4%E1s%|dZvyni}40>zBt8_}nGN+_2zQVN;+dZ(w7&GhwH)|KD!f~ns zeBoJW#=DzON9K6+_BZKKjv1RHMZpnftVfO&Kk5<^`4iB6sZG*L0d43qyR)?kW#xmH z&BFd=7Xg*moxIc?P|C+@50;uS^81nlO=H_kSly}qI@W|UkDMwR(B&RGmKE#{Cb}E6*J0qlQkvx)IXK+JRZOC@_gwU9S##2z3BI)O!Ic9Qbhd)Gx%U zM$l=Z{d{VWFA7$b7%?ULTKB0MO?8fc)?5X@D(Zi){XF)VaXzbk&BUsOoS2ztq98{} z^Xj`f?TTHI&?CJwWY22=)x70zmo#1uT3O1d3i&PDNAw~hD(0ObQ~Inps`4%nbgc7R z;*Wq^k3j~&$5Y?_HSQ;{)GVZD4IAv@&TeO>Ek1tPu+sY}`C8t990AB@FqM64J)NJK zX2+J5@P8D0e(P*b{E^r+3~K?-=Ua65l^N?IT_iM6y_x%y+C5bJ50bwkA73jy5sfW9 zfw{pmk+uALsZSB?VI#L|G0M4i^u(8VR*TgSdPm93zHU7*hJ%_($E4g{jtx*dhDui` zvi(#^Ni1xw49@1g7TJ)n2vm>5@3%j^8VK;(La1u~;fsQ^@gCHWnC&l=>g~#=1pB24 zdrq(;Yd<|ntfn1W0(7p8(2dC_ zb)n1W+ViGZjV=W;=u=icsJUB5Xk5C|sm=Yj_JOmj4Nt79*uQ*5!g9#_4N$7^fGDT) zoq$tm>bO-0{bilS2o~1@wL(0m#={;L($4hkPldZcE3x4}Hx+-R{~@vs`B?*Cb|+Zk zve6!glbvDKrEV8CPm#}YbYaWC6NlTBV;S`sYlxjq@yUAY+502mK;V&j> zs}{YE@?)Iu?EKf=3+|XJ)u8QtZEmkT2nVc=EMKWhF&8??r~NFSvL;>l_sQD@KS+QZ z7v<0G^?Tzoy+p|=EUYcei$gO9CQA#0r~XI6UAX}hTR%_OUTunXK%aNlUZdBf7mw{ejDCn?PVrxJb zgZA@HhZDcnm6x}lRBvG2E?X7lZhtL~K?QkwYfb|$W)$d7;}V7vJ1p5`HXL{DKBBT4 zqB;dk&sZMBrR!DSZ+}lKYI6J2!l@GAOj)h3M<^?jtu}Gnjf_2U4%|C3pYdO(F0_rn z6M6lgy^GuO`uYK6#~IPv9lj<`M^@;qBls!M1K(#Yy5J9;ZxXjzOUTid*#vb((-J2a zrrD{|vh#sYixZgP0Nij@4M}HW{WQF~R|SxopK-a&4sLCWsciHnuWp8$@ss|fZXXNP zuahHCl<2dSBGcuddKlK4mup@r?7<>uy~G@11>5m^Oe$~*jJ-zBs9x|^JJZjYQO^2@ zzSrGVk7x^ep}Tao%N`QQI?aD;QXFEYGK?vYZ0kdv$c!a*j;W~Vi&G*vbgIU8y85i`Ixa#BwF>(Wx z>c}^L@eJ8@C2QEw3Rzv}58^)hf_0G1Ob-N4Io%GD1;eBG1au+i7a%>^vGZf;xK#L~ zqwpS(b@LmUDYdMDW{n(k^MmrSRu4GiI5)e|$RktYDO0`*7i}oj{;s8#Mr1X-&;5P* zJpEgvZ}*$+O;>ba4b`smL9mtgsU}|TF^$Q++LpA=2PTp^?EgA!?vh>q&`RP%GI@SZ z&Q8B_wnrMvJg{!7BvezZ#DCRVb0fi4?OQxN?iTPucYYDoWr{#y2`#>Z`;$F;Aj^;G zqoSF_ZSM|XKxYlGE^`NV<_y&Hd`*FZ?9^)3`i{<8-C~Q-2rDqGNPhy|t8rk&-%@do zLE200EKZJE@dGA(wZMyDIwXoZcB#_ehBvo+>}KU@D#`u&F;)c?!q_?$4{M&?Gy&U2 zrVJxC>CWQCcoKw8ar)LA-yiS5isvC)E8V9}u%R0mJE*l~lB2Rt%ig-W+=C4n7jlk6 zpR4(RO+$ZC#D#m`Yz1#2@2VEx!WK#s2C(_Muownl1J5|04(10R?{Kf@X`|%`OlY)O zR4%USd@Y_sB$!<$0CC-9mM&9%Q3^@ipvyW1C{f5Rs0)A+vyNnt<9YtI5QUEUejQT` z@uV6&1)2l5kt^k2s`ytpA9O6-FfLheM0aH*WaosIhIH@cKX%O6HXeMoKM@Pb`J4AG z$EVpe8u)@&^u^|4%tv{dbMW8HW_C$ny1xe{ZsvIj?Tjt;D7C<#U1${0NpPyPee{V2 zr7`;fHk%ysx8=>WvI!HO>;XSMrEi6KqcUW_82j;8e-E4uK#NWj7!s@f)WRfSO<)v> zE=Il2Kh3|7>dFqN!L%2O*f(?CkD{`dPaZlb+9%)(xyyI@tIl+M?FZ_n`sL-5`T(7W zZEMD6y={%=FTgGsn^?s?oj$2unmU)^pJceN9Cxi}8t3>}4-j8>)F$N9!qz)RUh%brAt_3+hUb=);8?oL)se)e)0C9e^&rd>3FGJE>J$R|- z>T_iwoQ`{%y8qVX*(*+|LCmzV2e1f^X01Ola(Ve^Gr+M|uAG=w;l0m;oMvXU zqj+=__FC;+c-0x|ub_@HDvd@$%67f^d6FhLqyy$E05?9>&74I{!2K|_RPi9e0)1)! zWO!gV+4&D(D4R=L-yvY2i%>yLl!khF`+sW&-fwU2Iu$Pn%*7S!FX^yMGE!`vtBZ6w z&1FxRxi3D)6{0Y{mIu`jCnOMWm(>sQX*`fbIiLGlUR!}3W2%y6O8kSo@WO5mxkX)xp~plNfF+4F^4BRW8x>0|nX zA-jj}Ep;GW4+=47ep(yirBjsvRR`l?u%SZLo&9#HoXW2#UMcLn1h?9Tf(+&yWZF<) z@PEM^Ij8n6?pGw<^zR?^0;HCY7QNyH$29O0nulBRJV2A zq5RC7J>`&2qw`Wi{Iiv3{JxjD@44apx>UqdkZ>3i29w3U;dRY{hDr@XheH}?*OAF; ziZ+9!wA(e`=lBmt$NdeT;}mq-JR6gP=B%80hdTDPx{lh`q{ZM(ac;6%%|H{D+o;{+5H!!gi)JO#8|M z7QtC-y~7BJ(WAlUJ@po4`bf>zY|Z^P`p!|7ddjcIVWHtbQ!P$=rDdS1KWD8M8jQ8M z*SF#69<86#Ccd+*%($Mat{P~rgsk|yVeG0m9yYu_r8G^UD_SlQp_9O%&T^}GKBjA7 z<&t@7&Uy1XUhR>fL3rFP?Ed?R9^Vlp-TUOlqrX%&pu`5lmeVNtj}KyaoT_^i7(!4j5ZwDT=VhtA(y;vdAijf^b@hh{tDQ6Rh;{mP zwHii-KBTatGz4?IA5CY`Sm;>6fdG?M(O7Wz5C(9YHj<8kakK0w$CjtTDppT^N(A`& zUG6;PK(pl!YE{KFUk!hoh>?1(JU^AZHUZKEB>#3+kx!*8Em`e| zlnwNLR)+y8(FA(%%i5VVG{q+E@A!vL0M5rv_OR#J%Th?E?7uZ}!EyP(KWiKGfBH4F zw$&XPqX$r%$#}~=O3C{~R@tgGFIIv#0bo84D*S5GvM=4Uvv83upTCCH7Vo~Y*s;HD zzeh9KnXh)JHYe{;^E2ixE-g2q%Z{RJ6YgIt?1xC0WpHK_)}gW#{cXZjcFt!i{vlu| zVXSq$qvBO%dYQ=tz3Nfm_>?4jHu|^hq4eH@;i=$WaG0>S564+vahr&L`0Rdge}AOO z=!}0}HwedE{?)V_6?d_}+L6CFw}w1sa(5&R{5An*wjr-5BH_q$F5B2U0+R7lB(mjS zYIJa)Q*p_-8D2UYj5;>|*knc|nGCBD1YHWtbn*!_N){{2mK=(QQ>``=n;K=A*bQtX5TSz7%$xQTM~IKx+DZ&Sm7VC~SVbAf@Vw%3 zePznwOR`OQNfJ$hv=%PcmDMGtTp*gB(>-JAz^yXd^!dcCK_0A$rB4+F#MAS{n-Moe zDmI65^sIj&9lmRC&+dcV`_lx1Q|;enyl-zh?B3h9wu6&^td~`D=g#q9lbW7+^N^n+ z;Dpd&L*#6e3*-Uj1)A6O`Lo==Wqr#7zAUP!Hle*JbDhC~$n9kpJ=f(n4V7A@x@5HC z*0U=fI9hFult}4f-O{M6bNTuQi<2MRp>XuHZKi*iIQzo=O9HCQCN?bS}=iJ%Et;ypahl4#_kzFAzN8+n_%yH_Cfh|1^xE43rLS5ROc8oL(Dn{T*%fYW1QzzIW{GpgV_pKRyL!e9GlCfK%F{-_g<`0_N1Z-b)R+AN)Q zc;!Pe=g`;sOj7a80{q*N$ynN4B#8B<@-s8-Bix=(qai7secYSRqpoKj-%1;HZq9g= zV9)~s>yL$ohQIK#E-geOlOI{!>@9aZ1OhL#lj_^HC!89i%Y-Tf?x% z6n!T^!6T^ItpF?&TTZmq-memf%ekPitC~;61(C9r#;I6(gDRJOJAvnGmXOW*9-q~Y zFLeBtjSl7U=rS2bv5!54@~PB49R^zv+b-xRN;u984c$|he#UX&h-)d2l>55Tw$`nQMo^3VLh`&ZkVBVtMnz)2W4)*^U-RDr3LPxX|g*` ziFNkr<0~SSwK9YDcfVAS! zixZzX`>j!Bo8+j1wFFzi1gT%8eeEzU{TX8RytcOXt(eL>BBIRP`3D10h4kWXb2lRFWytNY+ZUwgs0 zYW(#s{{ea6N+FRIu;5y(9}la^q0Xgx+>nhNa$U~IUfSASs0)dC*G58yz=!>>|W3i;jSc5LLgk9ERyJK;2N$4d_FOPMdZi8SA;qks6G6hrU9bX;f zS>$OSE8=Apb}~Ik8266WoWhVGWZzpY?y?MDV z*D&#NH*gD?V4dMTcq=ZqS;ot=C}iF;zfhkm{eZ!Ekp2Ppl}~ft%__5StpczvR83NT z${?vThHalK$21#WEZNLH#>YIZC@*-(wZ?7fFZ0+bS-wDR{dT42M#SE4<=BPa1^0d% zda@SPoM6`lJ*TTI%ge9cTrE?~BH- zqXEXW_t^z*5CV~F_s*8O+1b0HOO>`nX3s7+lB4&a&fp6BlFg&e0 zTA=?`7bC2`0ksf{91kzdPOqG853KD*D%yUTbgs>gcvAcC;_Vi!y)Zn4+fbOh^x4jAqC1(=(El9jD#m$FOD|5#o=mtSSQ4nxtMEqPQ;f{lHE{j zUm{h9fpihD9pt41g3Mb0xaaD=Nv)H!>0jj25|gP$_K+&n3wx*YGQCm$vF895>DoT! z+Wxl~g{T&wQkU_y%SJ%GfJs6%xc;dwznhO{nABShSD2z*9MaU7VLh_^={048*-~3T zeKmDXJZ(m$*QP!Da_QjTQgMF+$BY#CrutKgget9iQ~75sV;e9jXOW%Nw0n5RTCuG< zha<4*nWuOL@Vf9FzO9oH=I{n#&$W7B#?}KsSH)pHVLkg#V_uir@X!(yROdd#U1UZo z6cT7bI;Gh^HMqC>bvI`K=uAL1JN}dK9;+;ExLCGtwX2%3?6e%l{P5qJC|=(Bk|y_N ztlOnvewp5b^BN$eGqw;j{B^VvL}EEKzd(1uujc{J+0TRkuwyN=D1=+o?hsTTasVkG zbSQOrDJnJ*cW5hsljO~|R$bxL5G`%hl*m$o|8(0HJ`CQfGj2KRfyHI^8e(gR)vT)Qwf{-Z&gvNQe`WvG%h~m4c6$AASt2% zIHjUlsMLkmy<|^T;P|)xNvGHhAh7WB-g=aUpUO@faW$on4x@C%Jk~=%i)~hh0JUEC ztZY$F~o3u3E(?Bm=5o(hoeCl2Oz|raI>eTEh zI{;b{2fmuc)ftpcxZoZLEcIqO3d=tPc!$-kl;RRgbgawu;`urY7*a%*=AHovPCXD` z$yNU${;RONH)?p9FsMvB8u!M1`Mf%;A}PT|AEA3Z^bVph)t{t$n>Nv^`jH)g*@?$G z$gl=@NMUZb{(8^q4Bf*&ykpUY98U`RRQ>&F8-5P=7I^u)^$4Kkv-Ki3k73HGHnaD= zd)3U&6gT^i*HdoRq@xh3RNVsUefaX$H1~fE<4nf4IY_6Y1GycwE9N6c#zeualJ&qb zal1Wou-YhT-rM>2TA-X#I05(?DloouE=@k)UIW1pK~z;4HcUK26_Jv7xijl|iB2(`PWA_BML|m{g6jP#04AfOuGj^=gj zFn#Zkx1L=aSYYhc@aHk`<3EfZm?h9m@41=UKGDOKebb1?R|lAZ5o#UEJ7B&Yj(N1t zSn>O}@#qiO{GR>wIWh97!dy~~<5#+PP`^(T`ZjmlN*)(v+@>`HhzlwM`=b-J0~}f& z-SDhr6E^1a%=|xlE9>CqC{nCh8n>U#>MV-8=yha;w}i0L!HXwC4D`RM6+`Tn)|XNl zhptH0d+ji7FCV9weYA?G$!hYOD)9`>cVLzdJimUmO#FSD$GYe^ztY4YSgTRrhxX%| zu~Yys0c`9DLOK0mZ(0jCH<=w0X@Y#1TYS3=qWU{F@w0TUVZQQscccMPa{?+qGw2Pt z?%(|{BMri^AkzGF;Uo>FNqY~z5^H6syMLN-Va`|H?2@*^S-efDB7*6&-U4_w(WZOm z`oI^|iD;G=l(9BMdfVtBp1t;juBlXS(W7^-0KOY#!)=#c?tD&c^d{IeaGasmRbueG z`^GbLyK$R3-cLbp@hK3+U)3kDzecoKL~9IufOgg+a@+`F2fA|zWe7iBSE(jwhx>rh z5t5r;IwsfAM-~3nqIm|FtI(gjUQM0ZjLlX#{~F*dFUVm6I<#THr=a7%QEbugD;-~& zP0}A@jdue#123bLSnT%eivcGFsX$J7el=Z;x_cw&((ooh|KuWkU6$M%lKl4%pUCF~ z(ibaKK&lL6b%FrZ1kiT$2|}C7o;{HuM`?IW91atWCt>p@B;s+2jNP9R$Kn_d*c2p? zwQ%=l=T{$v*?vFdt23l=}md7mQJ#-?`%l)*`(*k#A|A)#8NSerOr3h?sp z)RX>O^RWZSXLT)Qd8;%2gjx~-2>%}PW}I>*b_BpdDSi@i@eCDg%fv7RbCzd%v@h}q zx;CGu+uMUPKSxy|53$f@n@u*2CT9Q+fbddXI*E})cdH$w?tshyEB*Yu1PI zWWnh@ftuWrY1zxqX5lSDbG@;;H$6?6u@Oa-Hv=1&21JWXr{0Dp<;6F5dk~^G zJ}Qs#FIPRUXXPexnu`11j|R#SHZQHeHQnB`n@)I>6D1y6e5^mF&TjS#23*3;*+SjP z{lIJI=HWabw2&@ZTz1m##;*2sk9Q3YsCp!JrTkpPk>MFtA!fzpxKU6g;FH=m49ERQ z?1ZZ~Fh&~UJAjz9;Ut}=)v_tr^1y13=+Xd}NN;;np37ya%fEjitnbJpqb5tL+$bTB zS1miEH7SZqy$SoL7DdGEGFRc|h^2Cx8M`A*P|#5dEH~H^=J4TL@^ieJCmuVZRjAFO zb?6!2)q3Hu^BZt7!)VoV5@=pz1AHXqS5}g0Ar02-!bqCcpTLKXMP@`3u92{%LH+-& z$v?3c#h7mY$Em$25Tf7a2`LcxVTKfsCPo+!1g7fQ`;ZLdK#W%ySpcf90U>MU`sth$AH|=*9P9p%D17lS8@_Mw$MP}9-%SFF=Qw4 z>wJ#U@WHhF0b-MX*L>=@n|DC~cf=ttYN@Ty78omi&?NG%aw^4^ojxWVAm)>&@5kOr zwL2-({ChGINon=V*CwJ6y2D&^1uU=dvH=#845w$Ae9NTL>sXY8r2A=Ppu zDnRx|``7uscikA=NmD|sulJgbRb&8_*)*H99cZH%>kX8KIBzwqg`Ie!+YQDlk?W!n z0oFB@+^oPX#Qh5^6Fob(``70<84aM0yx;L_|9kfq#NAcXZhRcu@J+VtKHmx0UwEQrf~Z zYW`RBH@Rmm{CO1WLjRCC!_ahCm|lNHliw82f8^$c+juv|tHPf1-) zoD)yaBELpSFQn`$$Bw0Cf2~=zCV)#&r*{0JLg~b9xmMk)CKxl*^p{d{Xpt^=v~gxU zV3L-JcHuG01JH}I*!&IEMV`e?FFnFZ|6@Pr-CsFku=`6*dce@P5K?i3%(EHLs^H$N zZjj8c++04%?~$IUe=g(zsr=0WZ{4J7Ps=PfKZ%Prk8&sc(WIZxTaco!ki-o5Jn7uC}KJe{vOdP&on;kKSM!qR2rqYVyLCB+6+V6dOrVo6p zewI5J*ClQy3OuB>I!->b{@GxD zm=@~{oE_xttoAa8DF{R5RJuMtXo4rnd4eQNza1``1uq~>bZKE65f-?GWM8pmNZi&& zad|c~5rZ~DznCyfDAwXpzqw6eFjH~*v4F7Guj+VfzA%PyjWF3#T+9mYxW@Dm#44Fi zEhd!MKCEz%`UtuctDi@W$Z^nOG$p(!lIBcxlJI1V=UBAM_w69Q4B4<Fq*@LT*rhbx8#QoNI%s{{{P@z*I=3WTzgDm0i>XLSwGM-i;b8=-E z{lAGOLY?`W3krW7>SW^f_c-tq zdwi!+KRT91rAH6+1y3M0K7Hv{L{*VS2TzQ}X^vFMh(WAYia}HS`i!PD<3`0+j^TA9 zgCGna88%46vE|&>y!FkMXsRMig8EREt%~m*2|Bux)PA=9Zo%BkT<^hrB_OB=pxvsC zjfqA$F2!s_DWcGaJ!;K*63nx_>+Z>$WLy54c>4WP%MQZPw&Hgm`5@y#nFHU^M3y2g zkIV&HSFRd>Mf>Y#^rF@heA;+&30N0i<;RDlq;U(GCFsE$T8Zzzg%&XboAY%+Y$KJ4 zZ&h=u|BO~_Ts=nie688Jz<9X2Cz8J58O9x$yo6h_m;(dEd%l{Vxot~s&Q^Wf+c(g) zQaUm75V(-YW*^|*OB}R^f;C4^i|Y>n59lC%vymx3Eng#jfG;Y?H~4s=>rb%Il>w{HUC9p$_yh`OK+(d!0xWCG7t?68g z4n8d9tl0Tgcj2vM8&7J^4eqL{tLoAFndDg|mvcyc+3h_GX*=IGdH#n<_-~(z}@^sj}_U6h7Bdm3%LUmg+wRYjO^0DWO~HP`u^sBoV`Pbd z9PB65fQIXaVo|J2GEwjx(!uDylzZpBNfY$}zv3M!iBSxB4hI~NNc3cCA5rjEt zojOIVgf->Bm2$m3;B;l`XurSi;+hw6F>&P(Qpd^D!dLus5hiCJ-WvfIqUuP#yv(>+ z5VpSXE}LtP&Rc>jZ&r^4g^v`h{-nFDt^t9p`#Km4Mw>z4 z#L_8nU)k)-+zHbQ{$xp2dfeC-h*gs1KGA&V&VKy`9Y4Q6SXX+UU)}4dD|4_P?Wrik zE}*$}-lq|~h`INy{to5<{2Crl!2Yi+8ig{)9{b9xSu?FS(rj$2mEK$3gN~m`BMe(T zTHV`)$A-R*XI7&Dx{v!4lhloAp*oDuua?DQD6opTvR(6&sICnlD zY2>d|ysh&R+QkHD%&b#G}r*pHzV> zY^wt!?}-NTd(5<-Th`@lpwbd|JA|#!q5H;Ns7&QleoSr55@ke(I8({TmTrB#^I^Rm z`i=^{&a8#Yl`?>UTYo#3ncL+R`@_WQZXlxq5XeH_E*U^tOAEqoChGYb`6&Muy=UuFCEwo&-ZAb_HhVcEan>aD;@VI;)yWK z%Yp1YY_wf}T=PE+Or0KrUb4*4#yOa^`Bdgkx>NRdL*T-B&`G3$|D! zD?&XV>%qfsJ}#)EBVHti&zgtmFKUcZ4u}^xy8qTF6-FWQ zN!gyIhq=l5PFY346#7gfp$dMe{Mi$~tPe8AhZ)Ht+Tz~3N($?X9BfR!sNp>)s0l5D zpx*jurC3{GLab=caOBysAkEPO9$^>~B&RcZ=%zUM(^4Bns=eh0C82^!opK0RxsIjG zefkpUGW$K!!_;rdusG!-Hf}}_E9|$u>*iI~fr$&(9!wWSKme!AKl^KVcca+cJj&=` zpezbi^M&`qkft$h&arK^+^NgXpZX;Ncm>%zuKWNIZVicCb#=yA0w2K9TI{83MNv0-~=lM0Kd* z29Mh5ovGpii3NGb38Z69PA;)LJVUh&^H()0&So%mI49a$7gj(9`P-_;nsW|*TisTd zxXt-#9I!SuFK36b0)pfxU@2jJHmRKstt9^!bUV0RjSU8)nSlaNjWXf^CCZOy>~b3ZtR(dSgbhVJ#HPw}B8ohvjb(>nJ?JPK~3+kpGi&9el4 zP5rISWijO%x2oF_;NUsm`5Z7<&H&|$P7nytKP8_(p*Xk%GXaFdnnsLaLpVq)5<68tYigLyG zN_x8w)?*B{zK+)=>9G792+WUG3TI>};I}zfGc(ebHa=PCOqK{YkbixF=38!+)*s$t z>as=(0^R6ZFqP0nGfn6GMV;_o0CIUY2)bzz_|alh-(mQvcn^Q?Hu8pKfGHUfm0(78 z?Z!8aG;BK=apb*4QP5f+QB1aOKD+KSmMYVJbqLE{M>*fqZtv%mMkqo-2C62FBXfW2 zB_E>DFL66bqZd}Zpkx3D17~`}Ohz;_2yM(3f6x;6 zV5Y2P-J(Q<3VbInd2f}hAm%EDcBfB`EH+`!P9Y^sKBosu*Fc|g7ghE&YAo7@)nVY7nDE2&QurIhgAf4098Q`AHZPTRJ#4&cYy0Ur&eS{W zWDj-DHw2mh{a1F&xEkdVT8seXg4Ro!oTv7pp#F9%IckF;Y-9s-M1}I~!P}*wnpf>L z@JYHM?GgJr?Xr4fa^`<)mZ2zZRT8NNoR9g;7Q9KRo+!0p^4Pbnyt|QG*-#vW5CqcAOKx! zvmmM2(>>0IuB~Bon0Y^J=;(N4qL7(LeF5?{Q95%__Vd&0(vJR9oVkSTyhfhR3r>Ihg8<&#eu5M8~dw!L&Y=mQ`vO^ z(=%5_YSI-eHc?kIKCYn!YNCmVldshk53usLd}K#Ia*)RS&NI<=K{Da7Y}3zb-t; z^*PSo<3wi;=0UTsi#*~?o|6AtgRRCvM`z5j@Q-JKgeH4L@c?GDP+^H1D`w4cLWzU zg1`^MbXN$=t&-gke}KYat|tMb=ITdckFQ1B6liMGtK$ z#fh}jzaZrR)_rc-wh10RG6l1o2&u-ZCUxIo z3EVlCSg$L;4)m-#KAK_)x6S4;dE`TUwe4fK#NlhoWurZxCMabC{koFH6PVvddKSz< zpbGr!Fu#y!R-HF|CyCNHj4nC${%(x}k6;W$RQ>v_*uVek18R~3L z%Y0loWdzlM+zI zX1y|+{(!S5zpMMZb@_n;(amM|s`r43nmL&$qj|?+i-WM=ur|P6UHqLY#o7YatD%}1 zpyrWb=nZ9VO|?5wGO^@_p%4naUrzeEB$e)*5mJi#;(lJdJz4cR(xus#)(!&-2f*lp z-D#{b8h${B<_|}3!X#wyh|qWtCCC}xM3o0=w$fisMs52!~4&)mFOfGEn@ z<=#LU660zAhcBb8(}cJ@Z2Qpsjbf`?6c9RDN1vfFBV3Y`Ap;JrOCy)7{2^DWY9$b0 zbK+IMd{*vJ*ug?rj|<3BA-0AWVY`O4L%5|35_EQ6*V(K9ASw^$6hK>Bu%|jXgew9j z4IDPaHFXoSYrAji=}Hu4-2SQ*>(t7>l<<`E(vMyo=cm*pg;~^b8@Ewz4sIMQ*S)jINE0X8{tqbM)h@u# zb%|RYcr>Vh=DgRGTS|#;TT04NVpQvmy>yh}gCPe_WcWxq9Y0RCHrnZK?dPR<40%Me*d6I|?2$pZj=fK8N6dO-xmUfyY*Vdg6H}Xh8DD}uxwWAB85Dg*0-BFq zHVBE#-J9Ea?lozYBbh0uk8Y^M9%G6#53j>QWc8|@ibp`&2c%Xp-sXUtoCe+0Gxza@ zk(N7q^jE@HGD>yvt>_n2yVvXm11}(M!gYB$ApxF1ncOGcCyWCev)aPI(aWcU$-V+z zTzA(<}v zt`&a&iN`eS`y3J+Acw~pqfqbnTxz>!PF@+Ba{p28^Gb&f!O@feddTj0{7#uixOG+lg@3WmpIyt49XQL8tlCG$6Vl2cD-P{V z@eX|a@Furk!>=7U0%Fchxg=#l2AR>JOnGE%?E_OJyI>i|_qzI_`u!d0uYcl;ol;F| z!b)3);TDJy%e1gkFG7L;gww@IDZMv|MtKq7@41r7?)S#It`x#cr?5~0pXW~<^$-Pn z+0-ntnvk>ip+|I#b86@2<*Dp_1j}L&?5c0!S5P(V z0))ynf=#tMJ+~TB+tBSs8csAO=4Ro%`Q)CK}NQ>pyQi4A@K!_1?W?_hH3j;K+IFWzd0u zth0hDQ8}AqE;Q0*2lvEuc+ARFxc#{S7pa5m>+hksIUZ~-7FAns~S8F~?R!&hS}-z!Lo4z+(3Y2xFxj&=lf zx(~^30k4Tf;tSd8cDTsVJ5jkosEnJpfSrr9sv=GO^Cea07BCFkJ#j}50~oAu^R!Ftr-0B~DJ#u^jIe0VjNo4iNdU@TTm$&c zIUGkPw8{0 zNt`8S00}j^#Q%sBGrAb^wWAs^O|qFC9Sa|ydOBtyk&(YFC^n74)(f^pz_vmy4sw+h z+uF(0)jf>hu6fn=;W^1^UPHBLL0c0SM*U>afgh@=uTJ?4l!SOd+QvIq<_hG(H73dG zBO_XG&AQy9HH<*@-?IIaP7)+GZS;A#GR^NY?_NyLeA-bTGh8;B5>FG$`T=L!mK_tZ z+c>o7CK!6zNmr%THd6{4KXt-udO=^+ zOXFxatChgPuk0hMe3h%@R~a4W?o%(XRy=)Yi0OBEP5(`BQ&_i7tlZ zvH@8b=H+0-ofG9DlPs}B;u4p_{ z>yIhTJDiVad-1cIoTzl%V?4YR1fS00ei#_94U{;!VCVlhu_4Ez;+eIuwgK~x`~&UF z>{hhmM}64x`3Z?)gTsuQEmOaz3u1M^+XbWo2(~v<@p0PJhO-|)k!^l(UJY=Fz(W0P z>G+lT$~e;mCh{#ibvdu4z;E<8Sb z?{^&z|KI*tle|0u^phXnUD^VaksotBMOh%~#!Q}*PBv1(r8t5+lgS3wA`H^yrZ?Q0 z{GZL}ez$WIJo3Sm3-h}LeQygT`z|nT385IXPX5k6ZID{Hg!UvoZZycp%O1~9%+9Rn z5D9nB>p6WQB{hT=O!R-DT|3Xe347p2gzEgeKFAou4L7g-5tKi7za!I6&NdO|ru4^aE&ctr0?T1Sy? zms&@0OQe+wsHiQwqo%C`to7^eTf#|TtlkS+h-P9U7+4H9oR=w$gFB-iMs!RLJ8bpoPSZ- zGl%h)`Ag(V0m_b9H|Z9}$4Kb;nL(xe%gU%e2SG^9lx8KPNPF4~g$ZNYLbkhaXij2lOy7`WVry6Cy4^X-cNNH;oQYv5nymTV<*&%*PSJ7W}qwQq>cg)l4 z$UB@_1{~~pvXPd%WZtypTNZLUGX&t%x}=Gvy*$e5@<9a{EM3Zag#uSC*xSss#_0+$ zC*QQTGK$I?X{{?^SXvd?^rWp! zD>bTZ3*M1c=;N$c*xs}^(>m$&AP`^q5>0{MQeG*nyf1*#;Am3)_-ji4M-(7n`hV9j z8Z()bpkQmFL9cyB6{S#hcN>$zVuM^7k~s3i>+^Hi z$f{4*x3@h-Ol9vL{_&wKM{;V4`XgW*3tI1-qc0zvZ^Aq;0FcQGA(GFzkR!LEiX3i? zl(yC7e*45v_*%1On^29WqnGO;;XGj1^-_#KDOLA~t^QJ(PxWaUIL4x-&D{ZE#k-Yf z$L2O5tTsY|$^SOUWCpp?vu-yZj_yNK&n`pL1M|Vq6O(cS_+4m<(Gh_0Jdok+D6>jy z@u}1^L>r#ul7}ro05s3!WjzMgjq!r9O?`4V zcvq8WR&>0+T-9=t<-8l%!yt>m(1R1BA%@04;S{KbqJADqmC}iMb71_qrSIE z!PxH-S>`wCy5j22|Av%y;=noKp7Lj+=bTAg{5b9$Lgv&U2z5SbYr6s9S4M_c=Iv{V z)rqYyCvdR2in=UnQ67uU+uwmQ`b5%H*~#?YuXh@0a&A6@zhQUhmEq5;M++gz*9tpN z^CI0)n)Ht`XN4&HkyTrk2Pygh(zGs~W6g_=JvSOn+8YfR=I!jE=_+WA>S`Oe#i|+| zF)<#>|J~MGPB6{1cs7bIU=#xINiv@I3v68IkNO)Dw4?AS=_|;I_Qytf;m#-J&-`UQ z4^?j1k{lg0MU3Mk6`s`NBl!;LrksMIlZ^xk4hX7;Fd+c;Ep**nc(2qc4VHTGE3_S6diG;ddJw zni@m!$^?b6hR9{V#G3RIPM%2NcJo+)1Z> zk6JD6UGuGI%=hJPH4V22508?Ew~1offCbMXT-#+C^NAJJ8syeFX*v>K#;rX;Yj`wp zvz{IP8Dizlsi8X{z?ZVkA!pwwH17__zHClq4-Xh0?6Y-_98&MagNBlRddKHI+3&br zW_N)lgFd#RQ}+l*CqtHaf6%F;l&I>jHBpr|CJ$kse!Df*(#V8B%WAj?^Iwxg9TeMeTsTeeC8v9A-T&3%mhX9tgg`sSEv`bc>6XgkhFw4$rxP z!`2v&u9!g~YT@9&by!ZmS+U@H_7}`CM!oV7_sL+?W5+WhOhU)x5g^;QO4tRTLf$)5t-yC5M6{%D4fHTp(b%F3BVBC-4sei_K zK-l!hc-<+X-dqgT^ck>)+EKudOUDaX$fL9^7oTlGiKc1hB(e|Xg!|w zl&*JbrPa;dz}=n1?!)o?WtNmOn;D|rsP_Fl%5pH{|Ci=I;MTXp!O3b9Jq(Qqi7<|j z-RMIl%EeZ@a3_%?QB{s!wJ4yepi`A~-jCwcMGarY}n?x@4-re0_<*7qEUJOhi+D6|-4gmBP>h%j%G( z=XI{H+ASnx_&^*c+M@Up4Pty?OI*%l)>+?iT#Qoe((%^OM(bgVtBCS|0PycnF$FWP zm^vDz{^JVtjW%o%uKMhmPjO;BR!VfDR+s}i&Kf?Udn5@K|P z9f<8xQqx?1bT(I=Z|14s@*OQLEhB5jbWSt|w(&`S&N6&z$a=XUbOjEms9Yg`9hw*+ z|IHh8@gnV8Ao^k0)(&5V7VtClKAmNtLQ7{;*81U>hoYc6J~k1OvBu1X0=9Ufb`2nx z?#Txme0o{QZbz#D@brO!gfHXce*S%AS%1)*Mn$HqSbxI&eL-6q-j0mJdJfhCZZoY= zu6;mqpkq_)V@cHG;-z0_znQH}d0{D(mUah}>|KB4PU33mgrfPK@X%tCp2}K5s>Hzr z7vJR7hS41$Hnc<_ZKIR#g#|WirRvPn(Pb1}6iIY09-c_F0 z_~ylVYhc9%lhOE=HNe-#^u_B?d3;QfxX9shfl^XzSLjDeJ#wTwV`fP5W3>co{-;4mtK^iHu8jm)wX@}I()>(dQYM) zCKL$g6egODMuQQjeQ0$I8sq6p@KzIuIbhot5*C z(6#oO32Yn^S_7};!f_I=3B1suajH=;E6Lctva~dFzi}!$%jF7Z6?3rDP+ws+t(oBc zW(e}X{qub7i5@v3XVffT1LAQ3alOZ5SfvC{+zEsq2}IZpBwvPmeh!(st#wZEP`rs0 znLS~djN@W`xTZW>77{RfY!K5_PS(sxeU2MSF6V<-X;sj2-@b|tgZEwPN5mmSU-X>> zm$4b}ahDQR$Ovl>-Z001#H@~|)>$b2g`0E?A@Fxi3O`;5r*iW(5mG|Y!#}>ek{b<$j>+*+z6^MJWTpfti05KIXx!M=w9-)*_8`-3u}roJlY~H z#Fav&R-Julghx++c~de}wq2OUAE6Nr!zTGCrksCSud5S=VS5>TD%}P-)P{kF6!4Ek z8`WbnzqB3sT0rV}OWQm!+uf9y(+Rn*^kvF+WLD4fxph<=^Xed`rc}E2f&4tCAg{rH zUJuG&ea9ocg}1GG-OrAp)$;$_(h+Z$ZsR~m4={ut4c)lH6g%W-EAFm$cvC77m=PO4 zo8KgQaxIA_2=lUqty*P=gADSQz+Ow15=Z4%$rdMY2aJ&E82EAl(45w=yLwU8Xc#Z; zt}LKcgu~R@=mWsNQztQQpO{>rC&ANZ+X{D{{VCzs+S_f7jZEcbo?Is&qE z?^nK+e7DkoEyNfaGy@0Gdyxuwt{^eV?LK_#B!8DXIoBZYE5o;eNhCeIp)vsF1Hxh} z;G5aqy|)nzaHNw;hsc_Kr8?b8lbYj=fbpgC>k`XYb&2zGD2YREOQYIo^f_H*hhW!& z8o9G--f&cPS@uZy)w2_E45P2+2Bc?4ZyP?WKpXzXb=JcP#9bflS?Mwz5b|bY6MYcx z$S3DFZss`AS)>rlsDCV?;eS1{%ihIK=mS3|N$LW+xm=FGy#oimQl}^t_gSU5Xkw#V z^+k4>QO$_!!b>_$@5kNI8vYERGs(gCDH)>|-N=50IR|Y#Ge$4)IH6WwWMmSicQB=; z-7fnvK$$b}tGsT@au~E&<5Y~{0-W3l_0d)b=GJ5Fet_I?FHU3!WZ8aO2y425fNo!- zk?u|y%rmv6ydL$AVb!(h=`^08i!Xm#hO*NOlg>}Zlwx0nB%S6}v?X6}gIxGz`e*=( zzSmg%O}46hF`erE?CR#g|F*tU+_E;>hFj?g*v_^#h;3(WliH<&;699DinkEVwQ}Zj zbvD(5ns@J2l3=y#_o)X-Xi5I0K*9q}EgrmnFw z{bh5KJ0;^5dSLfj#HRrlejszA2KjH(b0+^D{MG)GzHTmmD*~^|tTF(ir3cC58H`u6 zs+1h;nmn8=iARtgI}c%+@xkCId=@&j& zg!u7VF0UiLoEc08QufnV05)Ju)%~{H!_WK~7~3xTZVL#bH1SkpOG}1d0V*6sJ&t^H zY-fLx!$=gkS<^#AVWf-Pc4R(iLOzTFC9Pj$HfF0gcCQN1bJm*P33IhU9P7%OdrFWz z=itQnfMfm+KeKddBIy=G=JS_IL%QzAvr$5Ppt8%*GCu{7TyoapL{S%8@2}N|uf zZeDnnX&=V8R1l`iwqC|48*4=wbEALl0Nb$KHDZ4%7U|^ zLqJiLs#Cx%lyydVza%;#jv7GkqQxc|(=Zo;GM3;P)o(2?&@`MrP6GNCY)cTd5{CCs z%D6G1QwWP#O;aH9s!F8Lmd|yy(2=(nV zd{B+V+=l$|Y8(NB9tSyy04mNFMjK{<;CyFQNpPv?O2QT34g* zeT$js4V%1osTvc9GU$W(!hOA}EDZIlab}m^%C2O%h>idX;A^JpIO;6A%jL%I{NB$sq+deor^3Z|a2;-qwl!-rg($Gm;hMkSSw(XBL(n96y(PuLp z&8;3=fFxy&$(^up8Z(lX<@A>h+m$Tz9f+(HjE%J?ZX^1fN@CXg_uEYLR}2t#;LW8l zEnOj?ARiY)u>Lw~6ZG;P)%nIjGhQ@`?nQAN*f$4jE&pm6{p*+He~U+5!-x@^0RdQJgea&_{a(OZ7@7TD1GMc$mB<1 z?oNgdp!fKk^NQRo*}m*fc!>we2GXKjGMx{|c2rjq-os-nLko&a^*j<^y`I-BpSOSh zY`*yTP;Jy&J^Msm9r#v=#pm>>?{^Tqy58`YhCUP^8Ap+=|9Mvd(9ltTC_-obng>^# zjdsz_xFweNzY67d=?Y!#%m|(07kL1JKz^7UEM}ec{j-H z{<^JGY@PvZkYW!k5WN}kmr3^?P7en*BCgLuIeg_h)02C)o6U!WP2NDmQhGe3vF~F7 z3}lYQ$7RbM1g-hW6YRAK4`@q66J5<8iE4JzS?cBkryQg;XvEP2Kk2@>w>lPcD8TiR zJH~eG4qC10d>F;WZjz#<3y6`@tf;JT3H^l$kM6^inY&_4+p2K^Ae)*tIwmYv{qNbt z@)`KQ1qlN1TPC7=>$rl1l&02A0r{itgR1}}z{&O#uS+kFQTG}gCk7K%wdzCb`M&-L zl%YxPuYl@=J$D0G`T_k+?rG8W?Wov+BzDp^1yba)Jx;hUTI;hXzeTg@NKqUf&2X%=-OHun!%akZ*86a zZUar9JVTedWIAfa7aL%9cG%xswvOHSGD2Zc%=j1^E32D~6?`+s2Z{yc7F5^$T)6Ld z;zANn$D8}4U>cYi-?#k;T;2Ce2i`@D;?K)03quRS^lk!Im09^rGNuxwBCgjH?tdln z>^0{%E5ev(X+)CU7^Uz-BhsL0coh>WysXgGA(MeySlnx9oK`wyt@xRUlJHK}QzG!a z$j_53lK`ud{f+SteTViV;uv;kdQJ#n{xFXTocF*51zEiO1Kv#0%{m^^5~1Bckjh_D zPpkJUGtEIdT&|qY)&O0Zi9z@4&0Ob!UJKzCl$jnoQe>s?u^$~#Q&=NhaCg8_MV3r& z88DvgW$l}bZ>=pX3)M1h#Juh)pZ8%tgJXPOB`c2~gp3$-bd?@o*q^;%K%^qH7as4@ zfr-7kS+r#16xr?5wj+=e4T|@*TGfPM%-+W*_(RaQ2W`sut3E$hMb7jFV@~=~C?6km zMFi>E@|=Th1{S{ul_>he%k8Z<+8I&>3JZ))`+K&JU+vh;j4%}L^93RltpJs7zV9f} zb{~oupH(B9yz+r*(Pu!1%Gv^?Wh}d4=cE39^DtzVqcJbP=_BlXV>Lg$T$y8VPS^aASa5#tB!HkgEMq?UR3%!NaJVS@)O<%Q+!ug0 zuVAD{)DVJAKEe{*DCh&d&M#t}(-Ta-JP8Qwt>A;TyT5Q=DNPpD2F@0|wZ2Trj~$>& zpevz(ksLIwkGOA=b92}R?4X3dT?4@!cBzyGJYi7rdtVy^@a1GraDMhErHSuI*oC|< zV`d?rXn0tuz0_#M$5Ntr`P&;` zP{KdwuP@#xizS3;{<1UGO=ctyk$H4m?ZY2XY>G7?KWG`w$<-%D5TCz=E!A3?5wFwU z3`e7t2DURub4c?oI{Y67{G!wiBZpjk?@&m(JEGE_veLqRdiq;SPa=&n(lq!-n#)?yvh+HyUq;Har3lu#A z9Mv_0?BojJJmVMtc^H9M6Xz2dFdv`Vs6Efe-Q5j*cROICa950lH%Ab9;9|L!T`QNq zsU_BZ449sn@`}N5yrX_t%q+c758bxxRu+vhcUx@>zCi?}a;~f%U1mSSzCt#->#Rm0*}3uSjpr z5apD;NNqpV4@;Y2l+s6X&;`v43BLtHPP2bDhWB}7fADT?%i6rA@)HB0<=+d8-a|DM zh12C(Q<({bJ%YKpd%Y#&Fw`H=a}e$M2`>!jL)UhCX^YN{dL)XsPa^?E_Z<7v5h^Du zW+`aGejBYVV5*;W0nJAS2Y$-D$S@RtkFCpXx@>!Sp{Dv((|vN_>nl=U*Escy)8~0t zC(81P=Gg7jQuO2@g%Vwjzph9C#KEs``Y02j`-SK;AyNnBo9ISQ_o` zi#5!vbT!yqjOD>&#+GtXTu}hToz?$-L8t|Q$Kf4@2e_=`I37Y~nJFsl=HNX7#RJ;F z9yb)I*xtY!@w4t2Ryd_F$0#*PBl4(WU|A=-2XvEHAbJyi-C5o%{h(>}KAzXHd6Eue1WRQTP@ zex>7^9)@|_4Ib6;6iSTNZ`1poe_HXVN4=kPsC{P&P2xpQwB-tZA6hdoSGu`oI4BF- zB$~HC&LhDVI)l*r+@aBPzq7)Sd*}mWOXg7R(0KNI0rg>cpbWw-l_MOynfC@&;2<)@ zW@0>dUi0;QbsbUSa~(F|cz=dJjma-&T>r4Og-6gx`T_4C9a5C&cESg?;ZHTS5IdOm z>BADLI0Iw&;Ts!O8&-IU!T8hWuX zc%)emXbqGPY^B+4luaINYD%>_@u8zLrl3Y%U|PMm8e@-yI{pEvnkaivKb}Yq%rv{j zR#^cRK_SwfCwOdJwPLmtHt^U+^T3SLxtmNDJIn?H4`2(FIteH|WE;`;3HOh#3JT?0 zyWAT-b}}2%;DeM_i-U7u>s>%LP;Gl83W$!Zr_wPGw6=afH}?yAKQ7)g`Xa9usF<_+ zuo7)>MM6?uFh?J`Tg+KMWxrYaVN0X9-&D}mGfLjD-^0!;o|p} z_j$ycwAV8El{Fv)*-drZh)B9v>RIvak~BgRXw8lW3nPGc{o~Wc-XNdCNJrQm{14 z0u8GzLN<)s^p+1I0pGI6MOJw2 z9p#DssJ~t1gIZBcpiEhTZnqEavdY|em-&v!G~CanMgec8*EajEa&jCnyFHkhzRI$ z@{1CvbH(oE$rY}V=e(k^ffTnx5^MCHrspxzlU^7dVyqadxxhLm#s1a(j0}@#Q zmF`*h1gB8V3zK{tg)ZF3pVzUAPwBI4077K{tM3E#bva2k2VQ8Fk?s#E(x>~7`$qBF zGD3!hwaeosLPm|s<)z}`OGdWXiQqRZ(Y@L5MyWkhT?YH>8(vUIkifmsewLDA{*JiW zZ-4%ZJc7I1_pXVUV(F+WzB#Nm#Z`L@y?hJ6AJLWL0>FVWU$D8H{v}sa!!zfqGODR< z3|m%ovJq~Zv6~qE{DAwR?@8x)j?k&)Erj>2Z={D{;*OW08w05(LuIY9XjVxj(%1D> zkF`rE+yOmO>=eIy*#x~iY%JwtVeK+D7;fd4I|)rT(v5aGiW|bTL{1xJFyk-(jk2YJ zIs;A=OfCr)b>$!U_?Uu3X4>N7m*yT4Zb&~5-7M)tMYwF0_J{~w(4Bas;_39jzEE5B zZ-wtRh)--lmSA%Sr6hjfIvQlC58uHYF=M=K&Pa5kATecKFK&D4Ui%gvl<~(d@!;)Y z$Kb@E8dQ_*3z@z0eN)<#)jkES8T&)%I-mMD{J>;~$1t9B?#9EH&CMC(6L7R`)`M^K zG7>u1q0zj7@Wka%w2QWX$mBkC_sD9iz{~_Of)4qwO+F!hWQp57Eq71DRdrL=_o@lWfzR z$_ojBs5^iI%vx-N77ltLbxD?mK| z<2*A)jvRZ|o*?QC166=MwJpQKf`+%RaZteJowP`t+mWXc@Tb2buts4gs(9#lPVBNZ z6qeJmc|&aW*d}GTHrmO^Hw`SM#ax^B0eoESLyz=-=3OFPYZOGpZ7YbU>8?kb0uojx z5{TO0DQ}gK{+!`V%W@u-aiWVZCV~Spy#cQlxU8AzmHMlQM)V`zKzPsWUgRQvYond? zYKsqF!(0pBJU1OSc~Z9XFHkgf%RG^9u>R+ZWn;kQf3w(OK!cG>o&ol)tz> zvm){Fxs=;piAo9u>hO9u|Hs3dEfOQAgnQ5d+hrad)e~moy`(q;HEG)FP{(V3f$g&z z+>5CGOWIGP&Hn*zP8F=-BNRNBt~&_>%qL#6FwcfR@e7p;q#x9lxEt+;HbZx%GYzuk zPPJjv*Cq(Qn*}Oevn)q1 z8Cerd|WR1XYrY#mc_}VnlPcQlis}QUTHZLZ+(j7p#UE7Sl zUPXCJJxu8RwESnPX<0?PeGFNQ)f^WJ&?M~^c*z_;agro1blLA~*$6x@-%+AX+jZsX zi{8T~O~c21ty~hCFipgwO4#nzh1*({Nk(2JI4${OY1^W58rZ4YU+5Eoj5-*6r}9#> zF@QTrL!N%mH?Y9~%-`Q^NJ4-A8fNaq#25k&v#IYnn%ZcUG5NF1?6Ko-I^=R(%tnxu zRRRpLn$&cgI65Y2-Ft8i<8~R~1;1*B>!YG1$g7V!Pk0Y>uE(xPQZf-t#AnrqFrJQD%fy z-X6C5OdH*-)4c7^Vb0Uh-tzW1@HGENB1cRAz;M9xfg@p6B}?SxcQL~ z+}7OY2}Bf(GtU<n|YFU}MTGYz+m+-MgbI5rrzeC#0&d9+JBaY55^DR$TgVV$UY^7jKnPyRp0N-nAak-+zbPi$+USM$;>7JHR{or+70KeOpBnBVZS*W@B>`Er>hr&6+ z4=vaTF~&udme%t=6L4Wq;xfU*s{#jvQ?%9|-}y7}aRG!*gFmYXWz_jjx+kqiV9?d0 zKG!`lOCl;*;bNInj^+rzsx>iYVx1Ick`CEvwdq#Icqr4Sp6`E||G$Hw3|>O*tono3 zvaS;T?=$agjpogZ4*wjmN{dg*L6rfea7o6}vVfBu`~`k-X=L^29Je4ftrDqtc9nxa zXF_~Y>$;;vo&Sucj_(bY|Kp4*^v$acCZ+@3Y}C=|ZrFMn4f>F8!pw98zyV}JskN;= zA;kuCWqy1f8kSWRlD{vpU{-BX7|499+y<$b_|EBxJn=nr>&FH4PqJ9x+Eg1t(lTZ5 zWtJ)k>QfZ^4i5%riT*`;`45<9-GcYiP$CfZm@l{h6z1GbB%A#*5XRI+GteRbOVYw& zePy%Bjuw+9^pTd`i6y?k^ki4zi;6Ewp=L~TjYn*-nmu9%klA8BcZh$5XlHJrbGe4W zixd`O?RN>F@A8SlZ7t9$MfbGm3o$DSdl2{=b1>}VHW6sh#(Y83;Q}XY?Mq{0^DyRB z`ci;Xb329d5oo3Iv)PmFg`l)2QLO_ApsI7mPNtk z;MWb~7c1*RFH3t**7&$s0o&J2A~0MR;Ee$}uPgSyExm`f{>PqA(!{libd&K~(u!%9 ztkpK1(GD^4P)}?{D6F-+bbkggYEGtPfgtQYMD;n5`x5VxSNB|ZpPmAcRpY+=fy2%|ftI-o$H{P@{ zLROUq&NCakjWGt{uWkVmZ`?ExQzyRRCY_m^n@UeQ;`cTF6RA?=FhW@d?%F_1c{oZE z@ClxYJk17fb^)`>>!1_LfqYq*aR_@XTH_+a8+EdY6Y^uf+NCF^O>1?*Ve^8(O_3sU zfLL{GTx3>;#50mHq6B!g0PbZd0tFGlY_kKWX?l`Oo1)Ku`19fA{~nAT3NTgpy+)~> zWLDYXFWEMvt%^&NM_SHM{TG-GLaWUDShhEK^yH(BZgl^tSU6~HrYJUsge>CmL<>#lwvy4609A?^zy^?w>!B}%av#-EC z`hl#ytQXZ3Vl^GUAnpK{;$c!=h6*XNI*7A}=yvLj%#g%Imo_kj4(>15=a(0`O+>*7 z2;wehiouqTz5dj{4eSVs#Kdf!B5ZHDRs~GvZMrOrD{zR^kVHY?lD%7)Z)uDX(0$$_ zEmIQM@f7WIZXk)5X`C5)8}9I1aLXer#Xo$AvQkcOOH10NfJ+Pgfew8?!0MreyTG>k z33%MONk(wk#Qbt}wclpL&EW$^jCr&Jf3u+d)~n>J7gwe-L}iQ&pNw_ZW>E{qHb-Rc z0M%DPInf$^@b^hr)d^TXojtdLV(2{Mhr|>_7n1?jx@BIaAOJqcePBzLzlDwA-_E8A z9!h&Ua#E1Gj(R{x6xf%uW_1AVQ(e{NToU{Rflvf?*=Mbf()GP8XW5HE&0aQw7gGQl zH|Swm_HVRhCWTJBttqYO!0M_j6wK)fEB_UGH5_3D83B;XXLhDLT2+Rg^M@vqdlMp! zY=WM< z!iiqMT;FXCg5ManK?Fd~L0hw;Qu5I{n#NTI^t|Ws&(HSg0!36`MmdgF!BXOTXP{2R z=$|i5(oQ{RVuWqUo}ofA+tDq)fc}IS5vsEh{x_HVjodnYNr)nC9?$>qerYyaeX4`1 z(nan#0jn)4L-GI8`w7?JMwc>E*9sl7wxPKEFxclawKuz)-CN$l%1E`AL=1=%rQc}+b^5L3rJI<^lmTPFtEritZRx>rKC`&D_rX@~Fcg*YB z(yGbOgTVSIh!ZAp6lh*`4P(N}dLte8A>P-Uv}V6h{yN8BZ!dJn*^b-U&fi}9;&r>& z>qM%Pn{Od3SKn)ubP#5iXYx_k&W`BK&urOnuUQ%M@4;$Vuwn6gi-4p4;Rdo}`SIT> zd(IlX?GS?+k$|QMdv>H(g>vJIo8=&nOJ^tHp?byDhNN@f^-{8{1LGni9siE=UgGJ+ zSMGe=^HTBtEB>r3r?U{S<_U!O-&m<1J_&sgmp%wP0MlW1-j0d{`3S@XIt;Q_F6fUa z_xSwc=jRovnMeP)V3X1i|0%yI?Js9*oQIo2AJ~3&6>CVX<2zhqE6}<3OG={KR(`d((BmHDqJvF z%2b&Q&L!z9BH~1?U@Me~fvZ7GiRr%))djJ5YXf>XY7KPrt(ARB$ zSh|4LTXohcSod%~oU*LrY4WDAb;N~ct^5@rS;0?PfLOigyLWS!ui9R-i0}c4p^u#p z%YrjoZis|`Ia>`nMph_<>H|S~Hp6y$`K_C+i5N$->$FsPx9{iTfh&v_4hNE(0EgCt z8o-f9?PRCpdEM9@iX1s^XgPWyO?}{5CqYx?CI0y~$&H8@#&Fyr(G@ow6`lFAHlpFAum>n(%>pE=8$I!R zUx=Y81r-@s8+5B{IeRq$VEN2p4DwhdF%1Ia>%AxDxRuTvTwT|W5}O`=0Q(&nHPg1| zm7-4CGaXI+vqJhY=`3$ab)KIhls}35ogb59_QJNb8f8!W+jp)hRnQvmXNk`Qqu&<$ zE$Zy16daMT2-2IA=VW(BRX;TIY{0FipZYoiv>T`cVD@R87?)1>$4ILMEJka962!>#sG^^slC1cj z@n>S=stE6YqaZ>SX6KSl-XX}*MLz$sUj*haT#KTO7&DS3vEOPU$mFPE-NEQq2hqjG zYDKeHPpEQA?dwEou2`j)Aufe1VXY;1@E)(=zN7s|8%75XesUg>Kent18qpnCiQWxo zw~2gkD^uiDigZAPO}2J93K)DK45o}xXq1k>%9oRCaDX$XAzOL1_=70w-)Z##*vFG@ z<|~*<`w&!|@YeUoF1(ranZLqyDl1^VYTK>|2q*qY+z z%W9%|TAm~>QY=d&LqS#*G&IIGL4yJr$?s;z8I&q8B|fOuP;8CZ^~O9dm(lJ3Uvt$*(t$O z6X2fxD#wu}tOfoIb~-T<`jqxHko7j%8x8suhLX+IkIw0Qk~4y@rH;Y8mXmr-3$ZAR z16vX%266}}f?+_9%;Q~?wGP9(XAuyaNx;6c0b^yHtD(WMjt8ysn)1iELB2L1oo={O zz1JVy>#oMvRd=KdmlYUk+H_K88Sdcwg1~V<(-N%FBbuNej;QCTAg0wQIf@Pno&2yw zu!S2~&TN?L7rHBrD+Pl!%W9!4VdJ#1VanXXaxD!+?Y1Vc^@g0zdUDnk;}i)wT;Yxm6l=2Ja}f%n^f5 z^Ue=jHMvrkt~BTIFDGZsd<&y`8sno%blb`9}K`U;QRCwe$sE zCF6wPanxMd+~0meR6*h=Vg;khG8qk`?H%%P`Fq>7a2)>cpOQ>u!8uVtGhb=Pf&T}> zNjAkua10fzJ;L#uH#Cp47CdJrhI4KL@Y+85n}<==xBhWl*cQnsL1BY1vo2aTSUp9u zJe%j}YT5YOD5dzsh$`~+SZJ})uuUo_dfDMh&Sl1Ztn6T z=NKkB1YV;^2BL{SwOCay82$>hbQx@RvOrv8*^aF8Q8b^%v=5fw%C0iZEE}&u2RoNy zcD&?wULyVthtBjzK6v`YrF2k^ha&Ed;?{2@D%#Tv~CPNq&fF6MZ? z$z}H~Hu%$@d;bEBjzEKo8s>C33sxt_DITTX7>#31R*oshIgYL*$K$cnC5P#|7-r3c z`s7Z=0mvaD5c=H|fXeoWV+PqS4 zxt-{I>@BC|nsc+}H@B$1s`>X4?za##I-U6qaXVe-6W^w-(UZGm^&(0Fo!08a^389t zS-TFmXv)lQNf(<&CMjB()|>E)Jj&ufV<8(<{#`cJaOSt9iRhU*HXm;GO#AuY^agjm z%AI6r32vt-Hf`0kje}wJOF6F62FnsL!fw$%Wu%Lg`M%FO^5f9N-0nyVOZhLz33Ao| z$%yzFHim4trl3biL!)wfGO<%RtU=c$jOo%H~rB*_2=+b}v9dJaD4tqFW5 z_uHl6&9T)X@zJzrbKIeS3zDeY`};zOyCZ@vz#tk#*<2W6^||#AO^}u=H|eCxO1dhf z1!vMoKLN>&y+wQTB5UT5()6*huisGE*KomHlOal) zv$I}NztRcBz3zyO(vu9N6D%S*lry)H^SH%DtQmNXIW&{w%Uj~X_7mO*m5;$(O+Nl4 zi`&2X_^c3OT-VFIf@%py4$>gtcky}Ih0_pye@4PvEFNBs%YZ|E7NTr7$ZO;EC#2f->fnmA-*84%lIoTER zhYG5+d!WPls0@Z;+2>rXOQ(%lD>M5YIt-)YK|+oYR=LEx_Jdn;nYe5j?mfk`4>o8M|sD|&~vEjXia_tsws_+ zsgcl>Ro*{;@d^(sRNhdV46PAp2x6)u#4vLj`d0&r;cUS1#eIVJU*}GH(MPN|A8d5r zZzzUfuW~RpzLpo6uLn-o)Rh^=&((5C=)CZEXPkS3D&2sxzP+W0w z8zlf2nU3JSf-HFECjD~KHkrT=>57iK6CL>`l*gUzZ~Sk|WrospGG!`J|7H^l$V>H# z+X&jB%*m5HmXX5yH=D=HH;@1YbN3DJ;!iVPrX)9jKufy%=+oG3wC}Wca{+&d693

o z;AofIdnb*15AlwIZzx)#gVIC2 zA8`XBv#6(M^^Z%Uw~>=L7BIqwO+U?ikB{v*G85RPZ0d`;0V>~~dHc={b#fI9_LT_3 zZ-7zuF;(SXuM2F`Dgq{ZN?JAQIqd%NVBaBbexKajGh9TS za@mhtzo2N26k%f68U`_%=icSVPpm9l>t6APjfdQk=)1CO!z*at^ftcsfL<^-N0wC0 z|C;@|N&eF@u+xZU)sdrx2#x2xZ`+T9Lq_P(-LM7zUP?6|5P1=}Nbit)Z@4p=YLhnU z(}3~G^VW0Ti5JES5`qfkZYC|7VVgavcsnZIgV*9Ga>={_v+EkNZ6m|HHea$4eu4>i zr#TLrf9h`9O?GYVG3EUiWV8g+)L*NM>7ti{?OrKdI#}BP7;LDanIihNUC|Wg@c8Mk zuvfqz_w^l}r^M|{sa^dlA&(rVH-;QQXBsX3TT%C{pF;^M=0odUznoog)+bI8Gh5us zdS8Yl=`d*Mb5*W!wiIl6NIm+TFJgR&HT=&cNM3S?7~t;>GJ1TB!#AfZ-B)epf9k4i z#H3X9Pydgj-(i{t&w0=)F7Lk%q) zEWVMlc&~UNWBC6#Iuo}f)Aw&T?PX@0TAJGAR=F^hOS#~*nh{x6E}&AG3nEe}ii$Gn zxTLASbdm|bV&I-`Dl|oacmWEE)e` zf|?G*J+LDOmB)V(FKW&%+Eflq+wt>Fin_(h4e)no&UdngrO$TJ9G?sP69DA;p9%-t zGrgRLFcbhxy2!zHS!vAw4FV9SSK#vp4zG02IvTV_->Z{@f=Gxy;5 zYrXIzr9N8$QiwdpONb_mJh{i1&#Uf}gGgRRX`c~9gLbE#0EK?7F9rWVIyu(fuuv$; zo$ZctFzPZwKkz$Bv1&S;g+3#48~oOoKd&V~`2RSTx^$B6pMN7D>04Yfc+a4NMg^qO zYeeC9QMwqvz6C3W-V}A*hT{mW^Vldtxd{BG!J_cY1$5O+PDD+YVbs3vz@^J|PrFpl zUr%%cF_+OjgDYuhgrv7ZntPF(1DyA=)K76;*euMRX`=0|g7}3`j_!-D@#?-3PxG$< zwOXUz?)$u)BPy&nh1uQ9Y&Q^Uv_>PI;g7sk+-U(4?JOs8?e^Mj{QSLz1hF!)j@SOG zc)o;)W1N}R_h)>gGg2&ga#$XF=-3k4GHOg4cSL2pej=7If&a07g;^)PK|U@kDYt$h zukb4iT6IDuefo6RUF{09#`AkMoBOSyy8c>G4(ck6ym%YgF!qeF7p$-1Aa=~-&&kay ztW~6bgs6ofB=>R{rFa5Z;HedF}M2px+7?XN?pOE+ zzYq3RAH@2=TLom}Y5@iRg|27gFVio0HwnPs!xi@kOIob&BLY1XqeJ$`|RxMZ<#-Ojer}8JKn$1QaVOwnm zC_OzROrNDl@Nurn?90rk{YLIvx^yueYlhN2#NdY?Sd6dyc;RX4!_~DMV-T`C zxCB?=5%Xk1?~Fb0UcAF7Et!F(j-sKUdvxV-TGM4+Q}Vfq42fti`~7BIH;_`_r#cLb zuj5V*f=`Wslabq**e9h8*P1K_>Cl^!#j7=N6m&1#nrAMU7d{lZTC_US+it^uUqdVw zBsfVht<*j1mQsLJm(nFp0}LxopbX$+0a;o+tS)m+I|-}tB8LiKIa>5ja_HB5XF4x6HOSHziuHO6 z3g|@<+K~*Bo$s_x>?dFNS%i@^>XuJ4vx^y!{b|!D#orD-vapGqLTQKSZles{NGVMj z7f@&;r$aoT(u~?b<06v7Y}^2lEohlsh0^nHu;7)w#$cZ78vrDwO*wSW6{n;EXd+x)oCtuVOB` zs4y7%36;orQ+Vbs|M^0}imz#`5lW zNK)Saw1NdTPlcBOfML-As9dUHFPo%JEFK0fLc}zcngY{iHuivtIzPG8^4BKiBvs{MQ&o&|F}(v`y{X<4Oc*{TM zdp9bFd4K0RLR#8M6r9BcPS_Mns9HN$pcj{xAdaSYv!R(S4aLK0R?#6k*smpOQ+bc- z{xBZwkxXm%9eNMlt3+Y6WTx#y|DW-HJFYj!J(`fVJhpeXX(y94h~1_H9B=1Kc_1q< zOMjbgw7=!c?f8245)dol05mD&H#dG|Z!=q6i(RU-#E#Fbh zZMZWh(P7$jz$wWbuki65yQgq1!zDw3#y=*MWf3M;{u4@fu}16*CebrWDc(YuPap79 z2`g<`y}AJZ$M*LutNxF(dTnkqIIvS9d*8ah*j2-u#b^S{pd+0r4zAntuwmu4uR{lB z-S>IV=D0fztrEUuciqn!_#~t4w`QFlrX2vP9X|wV>(DCvWrb#G-j>VUPr^WoT#q&*;`}^=`%R`&&s0S3a@_V#_ za0_QimdIO?i}(Q7bM>empkllAZuU86UI!2;mH`Yjm_I^O*1!4T7IV#rfuS>s4^Z=? z2i8hIy1qc{1DiO<$eh}uld2~cH}!;JO^9J{k#^))R_cLeZT_M|PhlH1?UGtJ6W69q zG}9eHW)Js)zt9b=WZFPoUkmTnZ;hy%A|I3XzEkOOh_2GM9Fj$P5~XlfAh5B!3qGoZ zo$ZP(Gc-TgIN?}QfR0!-Cq)V=6&bR+PN#i%m@J(1J>Gi2uQ+I_wa%9h_w^JwH_*~H zKsOl~+^g{&VMu$I*w|C2+O^6LJ)U*H-4inKM`V+hw9Lovo}zu6VsnkM4k<_SZSDk< zuJl%8L)jugwvsqnU5<4fn!G--$m{>;5#`h^;hD_VCR zr>3Y*NORKO;5-#%IO2#x>xcLw=Q}5mV@0sz2gWb@FgfyO)SGr=+E$p0^5>}i2UhNO z<=N)2?*?1qbfyrRDm$DuU_)=R7*-@e`ioS80h2#Qhh}v0@+!kDtBpl7(cyKr?gNP6 zTb^D(Lcsn?$ZGXQah1C`XiJl{E?#w5h%0Pk1B%eS)Hl;Wt&W2u_`(AQt!B?#Mfhz0 zwD9kyPc+idL(VboeO|bz2g*v!yiIl;wYuY-ZwOqCfm62*l--eSWmSaAq|+SOh8m`R=7vnw5th@Us;jI0@b^Le}8O z98b)02bp7W={nES!S#t)-LF+(`^4EAz)m=qK|=Z|?Slg0#Cs;nCVl~@S>i8k3~N~h zTB7^QaHZoPc@76Bh|>^MyJ5eO;AZ4%tG-s=O#*CvH3_`uP}c1w|9g|Ik@d zTxwTsQ#@vCfsLkxW10G$w{j*II% zch0H#KM3j{j@q#f%6vgs4q|HqWWjM9uro>*5X55#kHb9R?m zJ{NDmW_|ep4DdoxaNNU>b;QSBO(OVI$mq#~b8b<#5&Pg5G{=b3|8BZ3jojEm2-?__vc`%ads!_4bp{=Ds-?!Xeyc^;?=Uy|qVL3*rC+0t=) z{gm`urNQ2@OEORz9+p;)`(&?Y3Zn^%`MGX(?x0}h)Vzz zMR!`icb?vJwI`0Nl^ryIxXJK1^Tib%%2TF|X{}q+bK6PUY2gN1`FlLQ(ooPqmj-D^ zL5Xeh+@JZu^)yCp9rOLtv3gd0!nJ2_y_P*K-E~nvv1-f@zdIz~pbf?u;Z_lxb;k+N zj#fX(5mmc*sFHQtB>!!zZGNO!nQnS5URU?9?j4`qI1332rFZ?>+YI4lgo$dXK)L0z zB1{JAfN8$~3K$!vqmc)`TIPe-@&l%zax;Icnge2067|MWGg}(np zjM09B1p1$aqOE3QC)*e>U@dd@qn*QAM0wSezB*@lL*r774`dETP4=s6k`5MegE_c_ zD@p2Z_Gm{a2TIsFy_JaBVmv}`7>+uo4+4GMupM8_1cmFJve!F4(dxVZE_-m1t_GU# zvNljB1Tr!uy9xzV{0A#4*chejV@If_$u1zvy)UO3HxfZ&YC=D-6PF|rl~yzQzvh1= z>od7jD~vi+{FA2Hz+XRE+TcTAU1E1*%YP56pP**%9y<(Xixk(3?3QY%L6m}bkY2f{_DT{LC-9Y|w5a7ces zKj2g|&njCfX{h%f_%b{^^l7Re_!*&}CN5mao6@FXkl&#KEk8DSCLwfPI z3LL>x_HZ@~eu?5`MKvi}a@ndC?RCvGbyuimgm+h%USf1dvb1Kgg-|OmtWqdHHUy?& zMe0tyz=D?*rC(64cTv)LoveiN-li_}3ct1oJ0C z8-KSSXV}qjUBSF79g(w(r+>pSJy3st<6^HFxWK=Wx~~!84H<|!0Ih@G_BA4+&$yu| zhkTbIyuX8l^OL9doAigrqB>RPEGZj^+Gqk4Wlh;ON}=cyg|@qar_(xP!=b;pGxBCB zr(;xN;RtS9duyHTiHz8AN{O?Y0^8Ima%EdC*7I1Xk|-UTO>je8wslo5wn*qRBP=fN zsEG8V#?2~i)+RgJi*jP@U;Fd?fEAqyppJIo;vv95{4oALpm9Z{5`(L85sOgfz`@ov zO;s}1&w7ycG-wUt>19$Z*WD?As=ma;-dx(X{^eK}JZPR`LmN9R;bPfGaS82_Yx`@Y z#6()AcHj0|aYe-mbN3m(`%5)io7rr&~_3G-}+`qj&)P z_rHuG$(t3gr1ijFZA0T?f-{XnrO(40Z6N2ur!h3N{bw($G?WMs$94NN5ohPSH{LTX zC`9$L&X4$`g1Ey$cFyfbs;a_uW1VhsSI{7(-!&cxV-=;v$7VZoNvcI)FL1=_OEJ)^ z>QEk}zTT>4lc$U1&XBplO29wA!)YfDV91^I#8D-0k&!(THT&tRaL~b?!Y+!%XUewh zU~QYa(<1K{nb!pzXIlN~fa}(Qq0X<~VI<<*&xu@iJOIImj8AjLgAG$prkAG3ze-D! z=;;Bv6ldoI=Oq(1JO8H7YBvfKT=uE9+woK9(+Uhj?e{$>_70rdN0mM1&{RWW=buzd zx`?9F2Sq;s={0kZXK=nA zBX74YZQV_Q8h^x(I}|rCw$BpHQWMAOwyq`q8GWIVAvGxvuG>fNO)G&z>WwoN)G#)D z9D$BL?(U`IbZVF78{+r8n9B-hai1nIpE770L3l`QldRRypc|>6{m_kai5i8UFCwls zWV{YL=7&w8MVQy|;Q4yC(WmHM!H3QIl@`I*===Pr&4s z^gL*|$NZ2U8sv1)R1qcZYX?e^PKl#NdQNn=YSK`=l!1ZC z&zgFB@Bc>4q($Ik0Ffc$NadVg9gmJ0(r!WA?{^!(ueNEpX;7J zX46b+$3NiDzlABdt@D;7ISiXq5C}-XT`(H3e=IFlY-Jup=5*9~zEXKy1RdaipWjXO z3qxuwTFs^itv1PNie_}h`pJd?R-^%maM#~>x6Kcx;8VOw3k4hwxm&i;`>zkCG- zf({n%@F!$D5&)j&SmxPWr+p~$qnAkEV+}r9sP4k;)G|flYr3A(r)PN4Bnoa5tJCWu zSk1cmzPZdtK58mpU8n?#P<5@2x~;3iz(0V3fErF4ASjnR@<}<-m?XTjF-_-xw}VH0 z3r=};`RO=gu?K_Q0cd?1`?OP}j;tY;no0%h;*Ut#e}p(#$j-Kz?ah5Ry>~4hHVX?O z>ymrVNd{EhZJ#3|1l2)qg_VnXpI2?QZMC)BUjl47gG&=J|87cSYl3ZXw&Jlid(god z@@{%f6K@nUZ!$oopY!WS)1g;6lW2P=qmjm7G!cQz`Z-!h9`#7EOU|0`ll^@S{eG6Z z(~IyZ2Lh^SG-wUG`bU)&%HTkEnGLJy2nTN|w=;NIG9rZdm(qJic#%C(W)Sp4Mi@op zdei`u6n6(MTU|H#chlBI2kf}(e%zfFn})aq@yLPMR$=TQCD+&2H#pMbL11~OV+}6i znt#35N?Hyi$T)R7^L>5lIpZws8n?5_UUB4!7}A$vqtb8EH|JU0Xs~^;NZ%=Fsl|;! zGjB;%uN@+Y4UHo&vdu|^+^x#c9N~_P$w{0bd~nKlue3Cib$k2^u=CRM6}Z*QCKHPR zjq+Jv4iGt*)h7LAOM1AGdv|Ghw){A=-lHbf40r#L{|~>K0P;4n{Qh>u#nxL=jm!WA z77)?l5}HSiGJt0aG5LY22UyG2A|06NSJmwnkO6$Wb6n3k-oKmTI3m}2ugYJUv?U9N zRp_9@4w;kWuQ)?Kb_#!mf|I!FPUnyWiIzsZDogX_6KJ|FRI9b`9s>|e))VJ*Jkk#7 zqo(^uI(yc$ApamUqbQ2lw+b~$ptSV)vIZUZ9l}{ReuI0enr1n^gCV7{uS1Lyt}35) zhaIZR5qb1@9%Cr+C(C2A4mV(S;v=M0g~}`KWka~*EwLW=ps?-nL$nd+)-%p!XDg$t zh!V{t)HJ#(^kKg8sxIo<+uO$T!~IcwofpJoI&sJJjkqc!T!q9SQDqh^Tv<(;FN8P- zs9)sFUg{Tfmz7L2;MFgxOM{TV-(@D;$i2fQnQ6tx86*7>5_yIgriwJZKNTIKd z`e(pa%*6Y$o9U7gj^E4WJU|^+pLt^dP5nlmFQhZ7l~-TAWx{+a%xw-{?*C$y2WAt{ zu~KEb>X!`lKp>pOphk&NDH2lbb=O*7J-0>Ant{;8qWXH4ARMAOIR7R`9NxjClebuD zBaO5AW_?cp^e^p?(V!$&n7?GiR2*BH__jU`LTrVhRrSGW^n9fs4y$3=YwdFio_Ep4 z-&2CVe$tA`jQNn;^+w~Uv@VU_3hazAKma#0&?Y(4PNQ+6q-5#xdKc zz`tIbr?Bj|Rb}tZtFjU0ntj0t=1<5!SD;fUZ}Q#hc}8$ZlzHI?^4BdSX>W?kMi!Z@ zGD4j!!S!8{y5}CPM)CK}#QoO5S$|M6?7flG4Pj=)xb6>jw>^&KeWN#M-!j+5m#cb= z!a7UE1}Q@4PJ`XkO~%rIv;u{pyX@r$I)m5-OPnNDT%!KWp5dznH5tGI5?n{BtY0@F z(V>e(x6Ga&Mx+y%83nt;j+mfwz~#!{$~5QfMqZGLDW!=eDCBaZzWM>(6Xw72NAEE+ zglj{AGSFWFj>F=&N_IfvgV#medRKMN^_v2gL4i-u07~Dyfj?@Wse!u$P$=XU)KnJm z4h(Ylcnn7}BGJVnk!^15Df1tnvxmA~qv;0Lq3A!%70M3qL5?Ytw-U8}MXZi$17acZ zGH*!H%IK^w}FZy=(kyg^XI7 z(tj{RVM>NtYO(*_bTLUy?#@1F0MBmrJt^%dew(j0%{YyV_-4qEjuH!JVI8#xk@C}B z#vPQ&%alEC@0!a;k*P1?UIhQH$U`=qKU`%Otew*Ap01x&)JZw3Z5|}mt2bShjDo`U z%3!h<&ph;g{P|G$1>*IXd9Vy)>Ceb_&HtS0I&ZwhdR0%O{TNH}HIji6Ri?q(PiLb0 zS}MjgXFNuKT{2(Zc(?>?%=`VwWA6MF6cM7!MBBJSkXA7sOA1r?U6*ao3q#AV3M?M*+%Wq`r|lv|`b)bE=H;$_7C^4%mv}X_ zma~@tV==y%l&_8k`U_qTuBAixCxCA<@IyT0GF)RWcecmLg_&_uELFWim$?=Vw=qr! zEpDxD4s?2n4$~PP?L_Nsx&VMGU%((x+72ay#0nElmkGrZbgKV00cLdTj zbB=%8XS&Yz$*fHd!|p2mz|HE9@_taBG4Z8SSScPPl~aH{sCU0Z;JCZ1CT>Qq2Y~@; zb|8;4pV=}S#0kkd0w9Z(?gf^w$|*%qy*{FOq^-^i5$w}g%R?EVMUTOUoR9;-t#zDEK`QOK1DZ5hT{bhJ&zl+K1&Fx|6&7DWj zc$ym^H_g!v5=gGi^M396U;sO-X9$)0G$y^=jjVetazT|&3G<&XV9zg~{92Eg3tCvf zRH22SQ|I5%;x&$=m{2L|pestIagk5A?=cnUw~BH-mq!jlV?>SnOJpX+<7{n8@%(2y zi-l~{*G3JYx%CKF1|*24M7wF2g`AB_M|NMD*dXtab*A9&0e8-QLYx|$i=pdofIY(( zQ|!E>tE+t;fV704OldSaNxt^|)?cHJr9y%P?#O!S86}0sfacXE(qk@1@aQ7#T50>_ zOEGHkz_7^IrC8C|N{klBlZgId{@6%@X8Mb`XmeMF>hd$vd}&fpRluu1%(~Fpt*#ol zZf`ahi}=&dK`v7yeFE)-DS}09+7WTOxp!DhMq@mOk26ejO`D78`##*YwR(f3^*YpY zA*5)@jIdpy3*>l00Zr@(E~#b0O&L*+Ekr>i_2!`#aU+i3yZo)*6OfRw11!14woDOy zYjtCN77Becpq>k`RuKBC12BNM?ws5NQV`q+@wVktI;Cl$)=)!OBNrWNrG!7PEaa|G zOinI;F|Ehcs-JEeIdHS~r;fL*?^|pMx^G7i(LmEWJ@AU+fkpy=(c9840Bg!_)|dag z=|NQ{0D-n8Nko@xW0N47S(ELw9>^_Lef_Y1{VNKmuQImjMf^{ysKO_xgJRT}XH|5)`3Ux@aTJDAgkQ#iQ+S|8}{-k^jGb z&yiZjv?adxA!m}HwZ5IM2(}*6RtVaJozE?;Q3{=znK3VV9rAmwOVXkn*@nTPhEx?Y z(zOO3(o}+4QWv9LtI*2#1QrK2om%TiuM|3^;7`kaoH~8a@>d{wq5RQB7sgdoZmvJi z!Zc%gk@hnbO;c7JrXd|h97)@fgt{m0ZfU5c|F;p)xGi1RKa~wbq{Qh9t!XAl-4bzMQL!|Z_|(xq3dG?REq{4a!~qOoLM+2zIF@& z{jiZwLu)(p$6TcCRnKIQo3F#smRwi&q`tOj2OZ0NW-By|(&wNi`W~R{%q4C5MO+DZ=DGz&7jK2kFYvK@2`>gD$=-u#SnsqPojM zR-mT_!@KJj$1X5a7d}mPMEro?xSZi4K-1@kiMD4V`obhYY8a*SDuXEfFUoLq-9~l> zX_r%gDH#ix>AW3(9)1fId%VNm#X9e}&-oDjbCZl~78J|Z#Wg<`9xj9S`{1F9W=Wsg zaNkBTw&Kg|05#bfdJ>CrJm>Iq}3f*1!@84iOq`cTd%{-o=?6{T(GLBi1pPtb0U#GTC6!HkIi^x^%6Pe z2lrPx{C2?~SP~y4(l&xY&)G-)xp=IYVhhlc?@Yj{`{BWBP4(ri?}F2CI%a0-mRwls z`H&Z3!$_xK2E5Ed|0-iifc!djh(YyoT$o-*flAoMOQO~xZiW?jmWc1>wOl6zOk z7_zpW)Nxp+6+%Oya=RZ%=WaVvCg1WIly^6|pE8cezYC5t6|DG}PPo2M83f%5vdd%3 zSBeGCzp1!f>}|iO@)_Ir-Y|<&BKWyqku%1)=oD=Qz9Qgh2t^B_koshG>J%F~-Fi2KB(b!v5LG7q^Epm;;_1v#; z_%U<^J%!i@(3S+^%v6WK82{Lfg7PoZaeJ79_)Dnq4o=mITKevm#e0K^JWjCAZ_N6j z%*sUc4Xa@^(rNUrYq=mF+E^SlmA5DTKZwgj^T;=1J5VgZtnBHeJjd zu!cIh*Vi+pZgGN0%lGo%5kwzI@kfU{q~BcB?MO z%r@&+c@U)vt@}B*Z$c-3{#O8EhiF9{3`e1Nbk;==W>2rJ%1cugwbUS50ZDv_l3MHC z9f-C}h&{v*xGglsh?qcKTC*^h%*7+b7N~LU%qBGj%3GR~fT}|LtRq!6nG9Bj07T1f zcOTTfdN!G}u;6oq{z}!G`mO3AyeB^ZMbEl6QNL-0AT*9UhWC-017Epy+c(ye^_X|T z`nvgEgOAc6^bdpd+`r#2({6IYXTLYr4Lt}8Bj}9`vU(fto{7pw*Rem4T#1(Kps_TA zEmr?-I)Y<&5O3qHWUHB9xeKDJoMSb-xk;X-Eod&hw(yDd40r*Xsa}pi?a$HH>g;bd~RA)n2A2Q_jkR5l9X8%0> zMc9olv(A*?jX@W1*GtkL)w%xRD3{9JG~WoSCuge1iwpz%knQB0f_Ggtlk z3Yr>1F@jpgw5Q$ES{t13wJ9(nK81fV+I4t)M>7>)xUYMB1 zL4_IR$f~9vE^VA`(LXFZ)O4nQC1qI;E?%z_2yK(rPrt(zuUlg($AWFmkVid852q`h ze1~=WMU^7;W9o+PK;v~G=zjGJf1FX7Pv!4kKaG)S9K!qN-jkgL62Ct1>p1IgJo;?9 zQ%kD{7oGK~WpdZDGJo#KBUJ9+I!j(&Fvy5scLeHWfpoDgFS4C*B<>9Ii6Z7ID`>@U ze3fHCOIN4=6Aql|E`&uIQ)sfT$dIMn4~v!)(lBJQ&KfQ1^~$@ETDwYsrm9>v<21b3Fdri|CV8hm?K zX`EBJ8s_ZqyAI)?6~SO#uR&1<6yhcYo&$7Ds`jeNWMH`h?Y0QbhLT)09O8au^H+3V zYdbS&-0wU<7!>33)eEF`!a=@dsI<{ewZa(8`kph;T+Y;-B5HxRGqT$b7J{;=2;?f4 zQH*?0FspyrBVtr5bDl6XTX_Wi7O@Oe@tTiDNSAd?>Lx(XIf`h|(A!C$UFdnd1yF-V zkIeDioGH$z?K+yW0Ar@FY%@UeQn1NDv|8$&~eiT;1HqQG>%pUrE6eXmL%qPE$_+=qZ zN2bx@0sv7!Tg}%>%8L~VZoH~+ryUCRCH|ufwX_YhIbwrWgg~WbKIPeJFS29m8smkR@7YTln@X0lO!a=fAaiO+iLOL@qJqx}xr`hfMdsf^Rn6t^3V_UJ>{G z6YWO~0MI+!q=&!!`mK~jwrUvL%dntT9JWRr3idTt-pn{_fh`S8A>~!H|@i3uBlTztHQBc>&V7}Vn!|6k#(VH*M(+Ym&!x)Clt*;)r(qgud^+Cfqh&yead(`yLS>McE}nO~Ai!Thl}y8k za{oMbN;p+xx{g$&lfEkNEw4nrt7&U+e<84YI4tUu>(vzxn!jZPHO4?T3cm{+tfjN# zt8=S`sL0XngW))Ton@Oy*{~0w-~Q0x{(-L%QsY?k@(bL)b#q8w&XU1#2A*58j!)e0P(oo;ssZ5*#M-YzF5W z2t!!Re7mb6=|vn@{z`GaaNSP!{W<=3z7W8+4M8IN#}HmXLgo>JysOfP_PoFg?=eAWVW%S0R7an%&RcaZw`e7lGZ{G!phTN|t#BvpnFc`% za18dILP{&z)WsEg7QfciF?aybkP-p^GRdr~r^C^flQA@<`D`h?4_DK!N0j@}azu}o zslO4-k2|#TMSImR+rl?!Qnq_Vae|BGQO7|35b$ zkZ`Ctb<)-3ocla=leI%wF_N&qigg&g<8NzfLjG>+b9fgCLUB1fMwu4Vw+}gEk=5F2 z7xKjFb{gB}D#$O^+@`k=Yu|e2XIYPrs=+FJIryY1x*Oa`tn0&K9O6_jLfzyT@K7zcn`7!CZzY#(WQ zeo|a}m33nxxwy2aaM~`G@wfc;Y(#@M<80rzsMU9M5!tF8fOTss5(QlLf?WgbMm9r_s%FxilM5TXIYdgx{>cvoqVk!)!O~Nrvy1i{l4jQ zKNj8-CXv@luXnc1qQGpVM*JMj>sK~2+_%0}0T>E*?fv_{0E^L7qydlV(`7;t5{8ce z;`;t0V}O#ovae{QssI(X?YkUWnfXAo=zQLN<+eieC(#=72glj^xMx;`xxPf0RxSm% zeYJHh>A8NULg@@27qs%z)OqR;jAszBrQMKj7jsGCH^%!c$z+rlzOZ%dprInF z%oI3I4Iw(5M7{@;=hHT73#V7@7Ukw7`g#sO2NsZsiA5freX#wd@z+|mJaI%qZ(nOR z*VcY|?U(FwMZCJfxKj#nn%DzFt?jWT{R!5TZVW#ShK1%nkf>GUB43B6YiV()RGl%8 zRKSRYrcG5*^bs}iFTO(;jBaDKR!M>Jac1lnP1}an?fNZP29-lX+^8eu`asB{gZf^I z75}BVP7f^_UG#nKa~C!wd$@bsFe1=v!7sx7JBaI%+}xzbR~>WwYOpsPZW7;l3rN8b z)PvI{F5sRA(LH)SpLIgB%cuIirx&*}2K2W3+`Dn>=b0;CqrW%z?Z%&$x+c-6zpS6k zzNH|Fb@NcaTgDOk^+U#h0 zldh0WFsWKo0}5hu!)U!pW?3*SBX6UiGU&Qci(Wc5Z8LVC$)PxuvQU%x;qT$DTO-)$ zM_Hd1q86hJlB0v9$a9dpxl?JR{<&W?ilc- zw69D1&|iIgW^Ci|pb){?mHf?grzF@|Qutk2T^oBJPTA39%AtfG!5$IJYn>LN^mS^P z?3#Q0Lc0*zg+n{;)W1UWW8Ys<|94ZKL({!QXPau&gNby@Zpj{Po$a;7otm^Y%D0SU zl*DQ0wAN{+*UWQdNJB|(Sc%;~4DO8CfmT0XE*Xsar=SM$*@PLC9 zlN@KIxYJJho@mnVaKUdv&-M~YX0ZSmQWt4Rx*1#q(&&8fsA06YbQsqAe^*LBRmhKc zxYpDoD_I}RmbZLta%3s8Com~0o!Tt4E|ekrbJTz>bNo}#029^BX!Z9zD8CdHdGuhg zNB8D0UuJaf{upjPBo{s4vaxD-tkPoLwN6nuUV(W2y}2K^DA|mw4k-rMp61p@{h-Av zi>e8kddIUOXFn?xR)ds17F*(i-vFyqepo&JBQ{EEfd`eK`lO~e6}3B_<83QNHq?sg zrej=83DC(TqV^UXuzAlh+668SXGHG5F0e4PVvxOcUeV`A`p!!-(^XosSzNwu#H?EScP!)!#sJ+Z@40~YDkLKIkk>)riZ?jI?P}no%HQH5QGbfEPfN=? zzKvUH*_?@&Z|h`KNo(eFjAcTxmxA1(ZtErsd?lf1 zD`9O4-=8#5>)oqRa_@) z0+maj0u=1CoRr5G&0lIu(|i#fe&@(Kz_j!`bZ}@+>GTq5Hwve@A+wP`Dxxw!#`;1x zk2N<;rj^k<|7<$pI|B6=13_P39^F+77XcBSr@K6C;W)q62=+lMFTAz1yS3?;x_(59T~mG~Vzx=MI#=RBeq=D zq7mT&bP1hVow_89(m7F_+a15NvWO8coZvcmO&BB!H_WMYY`IW(T zag@@T$U2ol=EV;jm?_3ad5=+F8}AZ`a~GnnWCe-IkezQ72{OM+(uLBZY4ZMLp9Wy0 z2iE(*S=qHr*xZHD`&Lr?0Qq*Bgh;U}K{Xyf&xy%9U3xDH|7D(Zz9`|2`=({8I#QcN z1m=OeaoV@ovZ{p004yrkC-8Cu7gObM0b2r+xBsvFqP+ljYFWXB@QLlWXLWYOv##v;Qps{wXGPyz$tbS6geqeq%q%2V z9AsX>PC-FUpL_HGqvEzc1C3O?j|_wW94@$)X;70sDnbZOFY_LTA}eu(_BGnsh7l^z z+py&eV;~Zj`;7cs%A@bCAi6MwC^X4_?3Q5t^!Q?G{!XkHTr4krgOkfge_Y_~EwDP> zvm=FtMMqj}{KFaNt(fJV-yB?nKs19ECukz0m)9BYlgBiEmF2f<=qNmHCbtRYuQy&6 z;>_lKPjl(l7AUzbamQ7{tWHn%hlc(~gQ2EPq~68b=~i9lD<{!xx>0gS|E2mu`F|3} zeGY09fPwJbe^?bHJtoaQk8s@wa)C!T9{ja>kV;3K`9}Ky7Wh3Wc70k^b6t>di|lDc zvk}-HSP&_~Cg5hEB-Lp}eOb3e3n;qoaOG94oLp(tGa01Luc-aWqXQz#ZQKc^w-&#v z#5#}mf6m1|A}d?7FgYsM~ z!%Z+(Y#QHWqxi3*7{XmxaZS^4xD)Kb7m~BB$cKP=Lk;s@^1alF*2gY8gn@J=o>6xT z`3-*W1M_{BOTde+FYAY8J2FC-Q3k(UDE$)r12OHZO%qfa`Ix$LSJZ9vEMsy8)}7Qw zN80gqHlGHJKQmC6NIkj&di`7ucW#p&FFGSz!Y#uaD*gZ-03AOds(ub#V`p;_q63IhudZrnYkfLm!)=5`S)|XGx~BA1LI>Eq#bQ1!6tx~MDvBdnbha= z0B}Ec9mUEay63Ob@~Npn8f}X z;^6jwb;-*j-A0c~)xd%N+hXbh{VFXHYfrEPoDSb%=MV^L> zE1*r&PU>;SD8%NbqmHX>x=yj=oM}Jnyjbtz*uD3BmhIl$2a!hE zu4Mr4uzB16p6Z?h(Ti(YKdjX+?x~dLM>`^(8Gb})fDhtsidk30n4LbThK}S3J@+La z$VI?xqhAoc#>jsgoF#-I>rPj5e7(>PA8a5_d3&eAzbPHoWHU`C6+mzgSVy*HCa-}~ znhqmwsaM6{R-JUtPoGi{*9+dpWd;35rnP?ym}o!3CUZ+>8b;H_=^A&iSsiFQ zY|!!u9DT&Zdy|erAV#EW+f~)c0MO#E$Le3Nq~ESwu-?vW8xMlj(_S z*`o&I5=O1(DA3pOo6(AlCamAITnONYTF0I`iB&sn{w?FgoAeyDAWKum4|D6_O>1tE zKxVMWxm+cHg@ZTAA~La$ooEhWROAoX%u|=*+N)?HG!1H3O#dtLc<=0uXK1UR{DsBO zsNDS1@9ckq^{$8!dPp7ltG%zoDsVZd5yVABW)!kCP4CpKPyDpCSurH&cq2Ci`}%uM z$AI;DKp;u}rnTVm6%|yU|AL@P+c>k>rsKGpWc9i#06;VVN@JA|pmm`6JUnc{g~=Qf zd+1FClPeGQYTMba-={?&#~vRjbwgwsS3GdVaqAO0>K)k_1r?0 z9=tbfiQO@`NqutPc%U=phT*t#Aa=U|v{VGj$_d{KJIuF)`h7c(X7VSX7jqug=UC_< z9(&Y6KwQrCh<(Vd7ni?-M6QArw)F}R?4g)ORyP~~qsfc9xeT3QL3dNZ=oQ}0z7O1-CdsO;EH>$C>7n6j zf<5`}10~fh^%MdtaiDruTlb?jRO^dPk}Op~v7M%-nEq@03}c2FDPIS=1Km?FO*u=c zRo|xRs7~LBuAftylH-ezCN8aXgnkmFad2&7(d&iI;ODXJXNCOfdA{wtu~%eN03>kh zcU9FXj&OOm!Yx~LQC_@1l|Ye|22PoxL62$_7y zcbp%>wnJAUz65ivPM}hX!^0|OJ%JY6GCwd~rDcUI2i0NCmcDtD{LwDpqY0gNrMxsC z0A<@-sG1&uhVAr?(Ycc1D~Bc^L`N*g;GfWGfd~lh8dih!xo)btSVTz%m+-|Q{;ef! zZ=<#xJg;k+zf0o>g>CWzfHzv(hO{#r*&)kxR0RSUI8?=B-=MYM!lDT%n-^j8{5bCk z@9&xDd!Zn`{+z00*>1_{ zXwuC#+Hf7#E?ndez%`*3UDEdbr2!2{U=x@zmx-~hI}dFR0APo_2)V}MNV&p+t$+pA zddu}XSg~dEO@hl|<)d@qF^^wZQ~9BUND#HgxG^{yb!Oh$t+x{Nk9JW&k$cMN3U;Gh zb;<$|anrh?g;a`0dPL&-0^-G$A{+VE6MNxF41#v>bj8F0qhmbS5QF?4qy^p$tr=TKUgg*`!ciA>D$| zt4`*5GLA!3^AL#sA~fsmt7b;JqdkM&)E>6aI#N_(OqG8VVpARLB)k_1D)lj|JvwpU zD%()DUVUMwa%p%SkgOFjP!CaAB2#~-hIB*_p{Zfw{#RNa<`JcB3r3~rqyY|DNT9HG zLR0>Fqr>MoUGf#ruWZ0o$5?l6R(K31&}deL4>$7W_cc`JW50f%UDoXnx0CkWHOaIT zd&=@9?8J{Yyr+>D0kL35<7JVBxv#iiJ$PKX3_~J35!8mUc-*l+eIOOCA-w3{eWBs| zlnG%@$5me|4oMiE3J6rFaD8D|0>+;au33+Ym)~YA1kz=knEJ{k@9}05w^+hp^1}QJ$%1zkO zKEG;Mbe+CU2LgJz+v?FgHLrrUF`&hrJ)XjLHEvv~H@E*ZA!3T1lfuhl@_Gxk04fUI z%_0S~ZY8MdBgij(n!~S5+h#&;<^;ZcO-^=ta*MNmRP4Z_PK)}*)&U)VINX<})J#o` z=B!Pu;ra@*e|vNG$mhf3D4<-m<`CpOVI z*8BuD%cW4@nKi#t|6A4PqOD7N>M=7OeAe^0jg(2)kBaNSA}#~_^Z*#pK4EZGv+z3p zm1U#2l#Kh;UP0C^aVN&UeD1+Zywkqr)P0TB`AFm3Z~kXzGC;AGOkHhk!A}S)pZE&_ zMKXcD4p=GQ9=@NM;O1|fo$d>5O1;_q%#6+LiN)gi)~#8#MXjb+8|x=rF~A zT!vEXzjDacp_G2TOwYby^RJ~a&coax#z&e}OTrT1HPPwU%;DdqR7k#)I$Mi9g%(}S z4h9MpNos1Op{D#A)M28F)`I(L`mkk)O9YAW%H>TNl z^drCDcHS_wF&Q&IB(NA#o|FW(BZ1zwkJj*7E7Gtd4)fR0tiuu98R(f$8@HwEfOJ9+ z%&j#jp;;e7z~@mFmb(7z8!}!z7e?+ot_qe~Ii{AJ9g1kRNb@<%NO}w>$O`J(tL91{ zNX8!V(~` z%u*!po>co4t6q-RgE;QL8|NX?0iBAnNE>jHm))M~`y{%tVu+O;TKa~PUkSy}T1t6w&`Kn~|0|xA^8h;)fHK>{`CKLT+Muy2mJr|+qJL`gCAN3=`JN!KsYX2+($nx1P(tj z>?M{PRccoFS6n!fING{}Dd~r4v^9>KmmgDJ+0elk`FUGB$ck4iAEflAg!r=))chxt z4$G5zbf3okWT5#{ti2Mz5mFJm+Bn#?Er5mjcjz*KL1)9E=XNhYY)s#CqP3-2IrzCz zV$dLDK0m<|7Idm%j=-;HR|d6I3{=i-7A{z~Bu~u!nUSWq<*<4kco{u7Um4_FjyN2C zai7g!X%|#8KQF)yfjyC`d|60+{3e(I=8~BQFIR5Y<9s22HA!UFLdg`98&Fi=1-+ZO z77pX9Zq1C`fx19y;V8YCSihmv3~rip_@AV=D?7zu|1f=Tf}>HPH6Y{Bm9+Jc{D^>x z>(N|OJ>h=Y1j^@1PIonEr5pDO;p6;!1)1@nq5MR$A7FbwzhGxwWa_8kO$fSR7FFuQxN0^rPHjirImA z#Cl(HkfTTvf=+r;n6%RFXdRFrXuq9Y;wWQH)a*E+~Si+j*NI`Pc&aSS!1Lp_mrm7z!X{o>)m%o^==0zi#bsQlVa^cSTDp=iZ-v@y4H? z)P9*jd3xl0l+{ytQ5)9iaN~g`qDBtV;u%F9E5tWQ_y4!!z3$Rq6Nl(%KfemO+ck32 z6W5dv+@)_JL}Fp*4Rm#2tE8ePk)IB3TdN8FGpXX@(L$>a@`7r)iO`&)ldRkygYERy zG#avV)&Fwa*D?+#D*&d0#+5}fn1*1C?{sVvU!zI$;Kn^9)Xx9~#oQ0mzy5fa{D(oa zRPWA{juy`>rLF=gv6h`GBD67|+gXQAZjC(<^>9H39iPTKa92SN@rH}FSgX;JA>)!y zU44NL-|wUSna$GDXH?%~3I4CFD8f>;&+@a1^N}K`X$M?Z=67y|kgZ|ma3BqmfS#`Jg?odDssFGtKH));Ealh9)3LT* z90eFy!C{cX0cw$OJx5Bks_5Z#E#jl=f}*DJ{HvTy8=_G@UgOE%h9g9q%ImwyN%z;I@;%hdKIq@~4%6oO-+S@ejGQ z7U&0*wi_P~H~Zb|%XYa7rwVao=3VkpQJ%OF@dpkf)Yvy`sA0)cOJvFLE&3Br$=fz7 zy}$7yNR4069-ApewFm?r2ZVuhfE}8TT2bq;N3yNJqh!g{-~Y<@xM|juMrW~!8<5OM zI)GxF;YI41#zW7|xQzHq?X1Voa6!k@XS&Y8!4(BzzUF^%X6#*N!tyX$u7KTX^xe;g>a_8CgJ94Q*J5c+0gLj&j({8V2P2G$dY z##OS}_D=5nPd&?b0-r-Fy)%lM`FpxHWO$E)M8GUz5InJ)*RDD_zAR{eV-I>slWg{6kEEQj_^O z!M$>p_q*K9>+OhWXx-lW#(8L^7i?yJ3;OM8%N0p?ENIx~x(MPc-qn`tg3&%n>=P)o zfvEvu>{Ncu4;6|7!wKj2HM)u*DkCR30q;2A3rnGE@gwJFdn{K&{f5DM8G2~+I|tTD z(1d`$vbghbr_XtLskEy5{2V6;*gA`(i7%^RRr1nH@386vQ|JARGRlI(rsIzuXOu!H zbYR_MjJFD;wV_Xk5C#9P7^+&lNIV*3WA#{)aYuTC>|peH)^q#^K!;@Y0TVY?a7%ju z6HJqoFEtGEzZ<*%h5GZNGPkMMfOzcpZAWR^3F*XB1!NL?P;r?HkJS+vHa~(MR zT0>fzRzb#Rw~N6+Gkdl(1N|aooB1y`F~!4M$-cz?zHq}Iqw7A-u8bdRl?Q6)6M(+XCR2v{ho@~rl2sO?%KOXtcgN3OcdANRHT}*hCCZl3&iI|U*`|cdjxv6axHN;W_*&jy0lqlIadlJ zi(CPZ=V-T)sd^K#SM|Iodv~z%ykvc;R0J6?=t)vtDEuc(?K|CG)T;kOIS<1mM;@u5 zP{|_&i|AOt>Mvh{&-KlG_dL7sOGjg`-S)1gsSE*0Lm+pY^zHJBL>f%Mv!TrmfQ2u~ z0!HVN*3~TbQwU4{yakn{J^J`33~d@hZAd?A+2=D4G3|e=;m4VgOr$y-L#=wDHwsPu zNYrnS44l#{-5=D_n=(T|V;mt_?e&<-_<;cL2kc_d*Rf_lYIG~Q6;i8qO|reyjjhyE z*bg#}DNLA!ot?^`0QwEUCv%Lg@acU$hO*Y7{yEkT8u&!0hN3_((72h=&#Y+8ji~6i z(4C>^*eC^ZQo}I8YcL{%Dn%sRk~$_U922t3=HsHW)yLTFVaD|hnCN5tt0X^_mHEwt zJ${ZwMxLiJW@l?-b#ya(O{Sn~!juB3#trG&j4i{?@uP#w+7nkKOKERVGKP_akX5El z*;Rujp+525GrU=5IzuqD8N^I%$X{Ay@Qf*c?&|Fjv;w!*nA+|*T37bIy2L$gL*NXE zy`d{-(F|BdJhetDN3%{b+bAV`zAD;rABszDc^aJ^z-0zxCZZBQw}oa)WIW+U7u2z{ zbXVjqiFG=WqJBznlPwm{&>L3nw@{kV{;8K5ABTouh$QU6`t>`>({Vh$QPa=rvy7=^k8o_uU3J7y%uI2R; zolfVPdW#yED;4jWb4ovwcWmf~i+oeoOo0?RKvn0ZF4HgDSEVBEN>v6vbl!h84>@)K z;nQB#Tkl)v7~nbnHu&d;Z(;-;OJe?WJ-(3Gcrh1!S7z ze)?$_aveic)NEaAQ?nG z1X=1)RK=>cBG$+QFxtCl!x)g|nD(5AcJ!ko}0B6B~d zGj2SmdSOE2xTG`<@xDYaoiF2fjGU5cmlmIM0lOHr(m4Sm>+1G!4y?IE4S_yll>ZO~ z0?aU_3x>~hU;Gk-Xx}s-8K%^OSIaty@OBaMvI~RmD|VgS(i$L)|IO}?omsGo;^}yT z4s1n^eewJqc5KMJ)bL>24owiQcSlD7xnbxulgPUvGSu@lttmX-re>eHwN~arHlq)Zje-E zDII<1OZO#2i_~C8<%U-?oLhL7`u1R=NtZ|NcKvkEzn>T+vlDP$TcBjT&HgHOEG?ce2^$X2w%j79)z*M@)hb?~C$l@Zkc2i`Q^*Ch9sauYYOxlHP-sXCV6730?%+)_UD+(!fC>kSgpIT_XW0G!pkD- zr9}c@SkVQLC-AvlLqOHssLhd0$uaBhDW*lm9%_hYQ9AD2C@CSOkd1&Hm1H$c7Na{0 z&nfl}8Vb|rTRfjD|Mv{|w#d4m9OPX~oCulte2NdzKf#9B#$SMPNoDQRy?QW}kKD|o zvk3?ND^SdR5zA>x$2=FLZx`Qad>8P$5w5jGwYDG+(F#X-5Vcq_@uoLWe?3$5o*TvU z?0=-ZVh&U}Pu~3_YExz6k9!O$h?j{m1H42nTb?Zv(o^t-#Wlfb;VPUd079Hvukah! z+W;p>6+NatlxBJBi>v=n2kNZx1G5f>G&)yd0o1^bCY)&)VjPJ{5^@`AD5NrVpe$dfJ3=)lW*@@H?Lb= z2e>L7bI=b?JVv9ulcY%k2%|O6>Eabfd!}CGOKpQKrQesd+;Me}2vn z6CSSq#1~Ks#Mhv#Jn*`jZ}>!Ne%087FTB?B_jxt5KfM9X`bz9ssHESQod=O$1|BmR zEgCP@Sq3PEI=c135*eM+GE7<`t!))iiUT;JbhS1|cMdR{xJEDeIsYN~UCH#6_z~oY zs$-qTo>`@ayh)Vlu%OibF)1k6$3SXkWwn`=Ptd>g2^4>8gp8-IvZwi`)Zvn!Nyd!f zDdiv|V^Jop`{gPFHdUJKp!qVXD4G-Uv`*{ouPx6#T3Il1Bkc=)cr79#qONuA?(_?u?8JKrP)gM+{L0x`oXNzv8=sJEw&)L2%4 z5zf-xqm$}T!G+UR)tB#!4r5Tlt=N!HqhZ6xbL|-%qkq>>l`z4U7jx`^+Y@FAWyOPqU3AW*qs9urNSZ3~gfemyu zm{$PzNvri8wJV4#vh(CWjdpNanb+>wqm8zG_m!DbaHEyEW15b^O!+RW~fJ&t`bIi>>f{weIcq_Q2FvNfd>RWRPkl)Q1H4lwU zJPmAo2F-!J$!uVWxqkjNLk$K+?6WG*{K1aE; z@-4XoddYRh*BPi%0^}R*;KW~elRP^o;TCJ}Kn=oWT$d{wsPm11uU>0pmOyQ9m#ji0 z$|9{VQpg#gO+kIJp>xP7ghLb?J~#M@#}wYX2;gDIFEEFbgcc0)V8ob5WR`_Se_&El zUGJOOXj-N%#9l^vkj1m&MsahTVlnqM&mqou8{IS5BPr@iI+n1L1QkwX7DnY%!Q~!m zK}+}2xl`EGkOWhI^D2KreK}}C1|9oU>KAQZUg^~2*_`X-jpa}Li^#wR4w07)USLpl<*^Eyfl8nTvFFDXB^&|&J`*l=UQGfESB zqs1dx5{nJjB$>&q_ZX9bGgM8HW@pPw}Rag6^Z7_R<|kzxGo z&Fgs^5MksK5aWOJWeyGyY=G5@$H+JQKm{6d8b4Y-=dd@y8c5r2yTGLWk;OaRc=z9x zx&Ys!@kQ~!E`w&cp=H+5O1%Ook~&a(cl(Wy2g~rzb%FOs{N6UOoD);kL321Fe^6p7 zvCg1mt9FYPCNy6o%;2$Hvo#E$aXWxSZ*0D(_bY{#sQqjdK+jzk|FWG^FaFk+u5Zx* zv?P=C<^7^=+LL>dU5_WNv1;B)7~e0{QfnD*(KBiQ6}`^`Z*>ddgKFs2BFRKv-Qhb? z%lZ{91^nTiNusU^z3b-`2x{{nS*n{7EFT5@h zhrU*_R%h_5u$9nyqQGlG(iLQsa)=_Ntz@ zvz@rHZG3aMCOqVjtF5z6sekn08*iP%47*oiN}VJ|iO1znL?FEm=8<(^0cH&Y9^--= z!{&^!>R|q}ZjPlaY`Y3ZZAnMCtfh=#kB>ZPErNOB*=#izZCBIb2(?T{;Y_E+frX@f z3FxXhtHxkiblm&q;d)ZNo%m9FM0`=}_qDmTW*^&Gvu`8h%p39j4~LEE+5KdojqCc? zPL^G4Ebv%3>;sEO0UIE->P{e&`=GtvC%8cXoPIm(h2D7R#R|4x)3BvnOuUac8XBVK zbkI3rBPzpC_Tr$E1x;?{YZy~{OHyApiMh(@6Z1DJ?QI;flPoE=6HyL)s&ZZZmM_4A$QYlz1P){hi}tcI1~%<1TDyl zbFubf<>=kZ^|;2L&2Etl;`M%ZH%~}=*eK9oDRm@CX$DkcdzR3ErFqCC(U-UY+ zK!Uy`VRbOo6ZSXDp(3?01EBuH4=t~^Y9R1~4+vP*o#l05<3QdD6W%y>f`NcqAv274 zphts170q6?v*e&8slD{bY>5fia@jE0$5hgfxLB4gccPyaAssmIK@4e0&jY{YQUFWg zEGErH4esIge7dj!`r;u0RS^^yQ#%ZG$l3c*6NlH)7zb8|<3V8-2+|26LIW#(5dW?Q zTj?}*R>NVK>Wkq&HCU0tLmG&b;b<#q_sqw^9f1i)DAJ@(+n0)qQ+*Z+80j-QXR!p> z;r28WoU6?SRMY**h5cKei=6%$BF;bwN{Lk8Ka4W1My;R=6SQwT)uHn$tjxK$S1&kExM9i$gp0Se_le(rz?B8P;QkJi%{^o&Sx(sc(>Q2)Oa|jZ`!9le!+0B0w$g5h|_qHM6ocp!$|)@gX-rtpA^~P(VUM;<5p`qa_ok>Em9s?mdG>QYo|JinxQt~ z=NGz4jE0xjzvsaQL(Jj)Wfsz5dLzM8@HHSH{kxEwJRX@MRAg8b0_7af+d~RJN%_=H zd5eZljNc`01WX-{K=aLRE+~&~?L`&5{frQJQ@#T5>`s4BX4mfL+~`MRW8Ynsd3N^< zFC`8-?pFB45@wTh$|#%!pj%QE{CiC46*h_P9Of52WlQ8IG>LjvTxXI^R@d59M*~W1 zqY!(BFnXUj2?C+{OW_ak!Zy#2gD9@y-UQWhUzEYG!)2VnJlJq~3VY@<=IRgnO$c8^ z@Wue>)2preS6f#3{`QL!h{m6EjZWmX{`ani-TGG0z|M)?06z-;DfQLZXEg5&@!WXt z{d@h!U7NX|=ehh-C7kUy(-7MW<3~U*L(0j80bP=(<3|B|K-*!+@WZ@zgCBkO&Ym4j z>bP9RhXNOu{i)?i!R>^38(ErFMH)C4^Zf2zP~B1k9M}ACNc=4;=)L_if(u&ThImPP zdx;3=BH$VyslH1?<3^~t!mXlH#9Zu26?<-2TQAs;QHxHhp|hLnP|>E}S7Xl^3qTWT za6nOKpf8E;#O`!VHIZmM@oQ_9jxf@yMq?6t1TOT}6-o;2Z5yI>0YbFCmb(J-;EuLnIa_hvjMdEq)ena}N*acyN&)X;w$0YVNz@dK9^9s~9pd}|2mn`;g4=p=JzLeNm8 zM%t!HSFB|%>0nIL;-*tK8UFO_1b8~5@-9m#C$kQIIrlGrVBl-c*k|Gz;FN(U+#HB^ zd?~})SnSZ(`2S)^b_c;em&RZgaHD>^Xd-0zezqqhyp-@cwPVt zIg#O$#aGd8fS&2AhQpo1jKaW#ihJ5-4u@j0+Y50inN{*6R|bvN*pdz0-crEc;2zGB z(J+tmOLy&m3CK{lD+!aVeF@G5 zJ?iS8+SJPd=vf_zm`~^`Jhs|rMpF682N4Q`&q1Qku&R5bF+mJu5Rh%j6K3`hCNq<| zxB!C+#%O67xvo4jVx4pkWp&Hz3ZF9k3vztULkC~LvzDR2iWNj~p1=-dmph<&OeOn? zE=%~2p)SQxXU^x0Jgr>*1U?v99LLuiuo5{*{a<%KU}zUX*!4cv0Gl!H1z-jVpJs)J z+l7luqW=DomCn%p5>#R>p)m4s0C5jKUl@Nks9<~_GQ3l!+xr|0aECNBt~YuiU?D>+ zzmpfi6w1n&lgh~U;`xx2UkhN9l$k%)lT>3qEZmjYC~SCjM01_)7APsz4rHCW20=$s zsihv29L%c{ulk6#OPi2A=7IEdm&Eag?xbUA{ay~^Bu^13!XA_#?w!~J0Bee^JGFbg zlR;P6Gl2gzK|fjzahJvipmnC?=GK+tEgZP*-&G8c++W4IkkeO6t$Ibwu}Su;1uo=ar~464*d{KJg&RA!Cyl%QDmJA7&} ztP^7d*I$e4dCldXwWpMiU$k$U829cBSA+V4D}EQJXY@T&7!zSs zZbp5(O5vBeID9D)=obaa@VUM$qqroQzo`6kuQ9i@sz(-m)>&l%R}`cA4`BWVGgEZJ z5^UsQLeM4F6t}2q!BeTv`8HTr*3`x(q{o{cJ%n!4yOt}sNllIXTHpvAOK-UF*-Kal zFN_(ZRC=Rm3y`U8z@#bVqBP2~_FZ}6lv4o=(D-oIE#wejM`&mFmW9Dq&DU;|A~C;% zNR!)NZ+};_ER3A^lY>CY1!jE5o(m4A<$4~#k>Ag!p>0smop=K{eIF|7dzyo5=?ks* z-?}EiutA9eTwvA3pQ3CMU-K~0;3lt&?-R`A^$sRp-tydqu0oF@sfH+^raC6^QsFad z8u0!;d+|P2f4YHy5I{(F4P%{=(oaPw){pJnq9;oa$JB4K z7di`{rPTgJEjiS-O1CW|{#8rp-zEgf3QxV11@X6T4`^x*RJR`olOeB>qn{EI=5tP-9~~;(v?V^ z&Nxg~Z8h9<;gfki#y`kBL2z0(Ry^$u`pw>1G|_4Cquf(iQZCs7szQe%0UcfYtC;eR z6?sxexm%IgiWZlk&FQVYY`;jOS?7Fs(bR)ajB7VLh)e$`App@#S<8fU8E6bD?qkZ;^nA7E9r**XZ-{kj4{4j3o#Xt=hkTD%JbZWg zjO$lRo5Vbv*2oyGNsf3w(e&Sr&gHa;eZ83p#v<4eIaLItDv18%A}AWd;F7dW^M~*A zh;_GZBy-f(jDcfLdll~0Jy7dHt|tlyo9%nhibott{2LBYOM?sm==~%2y+%+pU{%Y^V|# zJozG!*JiGbfK3q#__tg~VCvln-+es$MsAOL8kA6RNx&*})#fXwm)!5jYk*-3pi8lN z&|Rmvn%(KyvhT1wc;QaaL!{NhZI^ou6h?~|7zyc=<@TRM*-$E?qs^Nvqp@I02k8&^ zg5y3i_POH$1!BuOnCox|_%yUd!1nt>38Tap4o3rqH_+FH1i=U3#uR)Wdb&R{h}|D< z>NEZs*@;0#ao-z0_=1$P+6~p~m3qK=Ix90`+@37W1g6bz(7LiV{f&$=fN&f&f#KG9 zs8DlvTbLg;zmCE^8u%to{nVN_g99lXVt{y%=};Z?5m29M*P?}$$(ts9J4%zsb%wz>LntuI6nHNj z47TF6@KWh*2<0JL+yq)`P!5oYfZRr?69yGa2@Ry@8v8%47r$y9n@u`sN!s@r4g^#g z@A!r<9FF(|7xtszD#OX-XW^AK3L|t!866h_mov@GzA~*ebZHNeqJpe z6C1Vhdn~%zXEc`dI8jgGzWvg&)r+e%1*vxP3hgUN<#U_Qwj~C7v=sYOkS#S+b!!;6 zjCaDFBMvMXIqKis;$4}p;=bhwmaKLH7Z70DwqM>W^nJ3Ncn)21loKF%>RDiOOgDMr za*d_-MqEmB_=A@Gk5PE;qjr_bzlM6>@JGK7LUvv>%G3xGVKc@J0G~E`C9vN!oyI4s~MK~+Lz*bp3=QSKlsO)S;heg zw!>oQxN^wntkk}nkkw2?1nc_$x8wOkp3dEL5$7Xw>+ahRf`^i(PV+v|3NxvyKMb>% z6|Ej?onp6V%Ua2@A!K)z9y@Axay!)^LqycO%w8gzo6x<=b2&Fs?7^k(Puxn~8KoX2 zrln%r)?$ZIj8>>PrzoH|gC2We(A@6sYD{!bqHgL9HV`IAn-1^>FW6rLYyq6eEu6=m z$lZ$1L_GS!(MZUGE$Pf|A}Q43Ucs&0HsB}Z{$+^+6PgVD80_RfC`mkwyA)b0v9vqNz6=C{ySx za;A>jg->o3%`Qhw?Cn4HV0#VV04_pW<8h8RnyDPJpcSIVIx{5pnfAzCzSNvN9{Uad ztL5@({I_4Tk0^Xb?9Q1htgFVyN^+dOLrmZiuIn21SiEHs~k5CMbUo9%z;pI<7 zONi?wPC)>bsesj-+X3;+wJ)l5JnF5mlcpz_rXVBjV6lGishrG#u-kJYcgYZN$u8{~ z6$?t;hE~pEq53u8ZWaytaddoCBkx+gx9r~r*OPZYGSSZ(61C*ky}+)%=~SvjPZZh! zRPFYm#!YE#bGx%>b+Szfv#K{*#%w~NFOZv8CQEkP%>|aQ5yr~T=}Q;BzJ2aI6mxz` zC&S@{0>!ZuI|~PT)2|lS&e2M=*p$-4&X$E%0MrT##U5xkvdLVrDtm*%82S3Ad%eyv z4{R$zy~+;=+K#^|2ZWcb9>)U_zN#?=U2$5ng35tP!GcX{`u}K6lEhNtZ!MJz@w?42 zOKTaRGxp~!nl7IBjR!GpKeb?te!c@FzYIL|b|#*RDG)~Z?p8oW7H`oI!wfO#6vkZE z3(eh(wFt<*HEe{XI0=DT{tYp)XaF{>Zx?{??ru>$~VsauQlBHVg9cp1gsgAHMKI}0pUVjFJ%0AdbHU>~Q z;{p8ldoJ_^j1}M2+%E^@$l(E--|lzH-HEIgTE~Ao3;}rU=0hKy$8}wC`2x5+$t(5t zU?9hg{MjgF$>|-9y86mHLV%bq0jEGX(Xrtw9~-xH{!rY%FAbOxhCUW%AB`!v)XLb3 z94}KLj%WMF>yy&+ZjKPm*Im|TsBiOb4ugnx@)PNwrV(C7&4uoWm-pIIr%9u|?GA?} z*(qP;P!fvq3)d2>BG0-IxqD4~mSU>!u+3n%5bzX#;qv!A`j;cSrfA=G}I}#$32Wdpml1r`r+mzqs^zQ zfL4MmQF9^b@^>lORImHwr9_in5HKZ8JSCl#FsjPEfhNFiWvU3^7>_<}c$W=jz8y}u z!gU*~Q?K*^%S!dXHaLe{R&p?#^QNe{FWB}ZZMos(Mi$+@4po?0^#=v{AbtUP^%KA6 zJfKYSs(e{KXT>eiNrsGFgFx&r;jRfn3vfoak#HDwd2xB)P3!l}-R6nfz#Ik8k*jqn zO=ui3-4>^jVrij;G_B!Q3=;r>m#rzjp{Zerkms4~yEw~Ua>XY6TQBRlpZ&Y4vji(* zJzgI$>2X6Rnn;gPxdw+lo5@o5ESdTWffO zaE{6$jecEDu$PwiI%?X*@w1J%W6`sF+Hp*jp0k!f)!7vp%li4E3baEzZIRzqLY zIFnQ$j=}jxeuCNjP>ODD91H*EbDF&dJRlvl5>L(YMLsbbNX~=Q>rKOvj<#J96@;-ZQo%_Qe%W;3LyJIkxy;$g#zsa&r;C*fB~eu3gd3zF3Z+y6P{k>KJx z+>h#>-P{%~lV}t^UX(=n((SjMY0r8Y{-!&G8|E?y?==>@3~c;kr^$QxOuEA_jB+WY zi^gN_vGW0I(7a-LmOC4~__N|;v_d;s;aUL(RW(n;69{Y!t*TV>HUfGKV$|{?YV5`{ zl4AdeV<7Ydr;9&bBWb$GDQl4yZ)Xwp$H{Dz!>MKs{o##<-eb}1Ycw$it<_a9crDje zYn!k**njNUOqQMP)`hvGAN?iF$Bfg6Vl~l}bCo=*`+n=NC73o+o&kM}^V{-Rn)j*? zDd!Bg+m694)hiqM@X(Yc{?rqB{Pr`{eIYT)o7pT6Cn#>xk!P_X^%R{E^CZ!w&7v{+ z)elPcX^z~yC+o#>u zPbe3rvsOf_H=8&lf)_>q!HV~~3TG)2fU~1Pt7f-*KJpe&J6)?E0UATTq2yNf=HWzb zp!yGl3;x?Noz)6E46qNv=iwIbf2c8UiX_bGUd<}l$l@C`7?6Zp#pdlHAxzPIjQyvV z#)PH?9h)nL^zE-YzklfkiT~=G0IoEe3yy0?Qvchbq6A=E(_Fx4$ld&a{Yl{?<>p9l zs6lsn(*UM~dDaZ@?TzA1(h#=xx78hDBl-oc>aaUZj+jVr9|lhAc%Inwm|?)#x`y`* z-2Lcq;w&vmi!!#mMIL_>c^L*YbY55*0~qO)uMIiCql|(WyS;=26I{h)xdB8laP?=S zRG6|aH(tWN)n}N#_1Utgm605O)QtOyAA4_@+fAguTM8Yi%q-ur>DyIeUD~ePo!7o8t1XAm zZt@f-HUT6?>-J@@hOMZEHz+`>LXp+fjGB|Gs7*ii28I)ST~-lrQ?o%NOy(-|!S?DX z9i=)7)6~4)xW&yB!sNCaWU!|pu=-+6U!;D;L8t234FfIY?U~#86!jy)wA}v4@X(M@ zF}+P0{O@@0#BT= zIaQ@XDfbQaa%9KGE&@~w7Gwwrs>yFCU{erN?5^(nQ|fBYBbMSv*u~RCfhX4Hf}PxA znPO^NO^%QM_ZtUWg*RbRv!+KXy#6pn8}6gupA^T7e7h!g$piRXqHTOeV6@0rQ;vX^ zuvySy-pxk*^aKpxejFK)RmjpJagSQ>rOPZDO*^Sdr{+bhr7WWd9>U*eI)&WXxxq8g zPfptJ8m~;rOFpWZGmdyW~H(t8Jhi_)0G{!-Y7&ETcB6vV=ow*|$v_>8)h67dpmJJ!N? z%x0X4neOdQ(y*L3BroCJiQP{T{Y?LL3S`pdb9{*BoU$ai%X!6_ZfF=j-rCxdy!!-1 z3w>lM)UV`&i+Jc&^N!Uj;I23$!4PQe7CTT5CPNXKXS|7Y*9tF-C!yrAOPioeop%_@ zp_irC2jSl^uZKZ|VlEc6NW_1D_JyMsY)VK3IikJf7!#%jBxTuCjqIt?_Bh`RCJk(= z?|f%S$wOOzCcX;NWydgvg1K>a2jTU9^Uf2A;tC?he;~m@!h32LD1o_+0Qvp}riQt( zL|IzP4{n^r^Y=w0H9WNJD$j7y~I<#B< za`lJFBrWV(#+Yd*NUC#*vutNv@2b%yc1A+L03(pu+(_O!?^HDoemyqY_h#Jbo(Cc$ zU<*9fAN=QKyBzfGu|mfu_(W?C*<#O}4%cB1v(;BSCcJCTs;xrcJ(qYU)s2H8_uy^s z%EQbf@!9_ta(5H!*FL{qRs6TZ6zEjAB;1g=Fa5V;SHdA++qTttgQ*pg;BT;{hjW8* z>Jn^2nJ7+K?tF)rIiOp7sdqy;NKN}mC%o@>W-H1XB6QTlFbrbyI(!e*6L`z?gOMVD zkB;-aDosrxTaVa*h1>D)0ypOHl?`>3v2WypeI-sr!pIj{LGE|Rp)s;)&=1Q%%;&y; znOt{DcutP>lqg3K&_quA?*hOc!GE@qwOD$+)xVi%oEdql;Ux|3q?z%J%=H*|v%LPl zwPHv&KPz7)wM#=^k7#&B>R$eU7xZiC1Y61XkUp=e^G}`#*}#$C2s$kK^6-`MajH~mb{Fv)XXf350 zwphRXn3;XcuVL&`2;t&vXAb4}uMzem%yxcP@@r$VvDt+D7+YzU&F=fTzW}q(=kxx& zUeD*_0YJVJfBMH+?;U{j+BRISESmPU>0z~uSkcNxN9&eL8b*(d{)L9+Sh@g@!y==u z({Hhd5mi+sT7&;3Xgd?stWJP~BP;*|;7j%d!gNL=_cSpnZI zLMrk{=DF(mgyXG4mNRz!l?eW%zc3^k3v4@$x}PN+eMF%_&)+~0^8!$?9qOm|QKpCj z$M*_khCxaS-n&wnE(V^qX@z`07(rPt=!sn){}g4ncJ;i;^9mF{S30MN&VSe>L~*-~ z-qV46*uR@B$$nC%^s&T9M*_hSA#t8N(L9lktyUXqThHJ6 zFQ>8FvOO)X1lP^BHRKWb<9D({x>#*PR$y)1te_jIYs-EL)*dd+0{R>?tjuYT3^uKd zF%ls)hC%~Jrx67wIdhj^HOI}b3g+v=Fb2~Dfv`->SqTp(pFzRuf@97Bv=#>X^QBv`ATWTrF zq%q?;m+37oIv}zuGf(C507A^4SFMU zB#|!^r2(Yj6bZP^ynrCFemrK7ToVvkvq(%Z&AK|`yv&zu~>WLJUW$94Ai- zeB{6e5%d)T1%L6B-K95$m%>eS#$)72M0~llSonp~pCLH+L%UHa2GqM11FZKiVIWfj z`O_KKI|;j(P;u6-1)(lWp8Iz+9QlN*dK1J>HerZFzNPifr7g*Yya#1#`6z>K+Xb~!Z70UEUP`u(L{ycxj8R2qDyLC?9uqZoxU`>h1mn-MFk zMPPzxa9jS54nWhK|LXx`Alf!H&kU;gfV1v&Na>Oy|A{T9JP5g%gOTV+-|$CQrZb%r zgw7BBcn&qFP=-n9BT8}WBP;bcDc{#|1!zM4GIUtR%9{2}r--w(L&&bPkM&dQ`@4vi zO)w-nmJ?!7cW7URPRN_Dkwh3$1e`7Yn8V{;Aw(Evfb^ zG$x`-pfaTz_Qi)-?^>{D=pxW#E}^uBHouF2>9y~vWifYoS*#iMxifBT(n&B-TX0xL zq7e4>pfI~#jo7&BILB=9xy=|}J!55M>zF@Jt!Z=dC|YWn zxY`YBggSnUf&s>}(Ru$Ll@BOtXjTWI{J#~U<8)Wjxa&_|hrIEvl@yBrw z2VIl@-E=i^dGS^i@gU*NS-gV3$ zQLi`8awK^@zCHprM8buhwaWBP^6yPK-44kV%pomI`3FCtzo&)ANguaz#1#X_$^saF zPUYaUL^IxeO_@=ywo-y5$gOH;j_8W=kz~WX%V8j&5O7bifrerpN;pC^q%TYA&8rwG z0~D_A5K`iagmH8W8f!KDmxnkN4D5C!gj9`Ae99#tWV~b*D!8oS%J(nl)Z*at&m-R6 z=9YZsNdt0MYJCrL!j2BB%QHLpQqHU!aWLhpP72FVHdDYOO&924cTJ$=$`F;GypT*h zwOI2&M=@gHn)II7w=JbKWVh*v4$3;uh*lbP-4|{xxZT>kY3b{#v88h7l!*ZzAsc&2 z)iL*fKv}Z&v%{PTS@RX5!djv+o=#8eZe+AUIv|570_V zA#8WDkgc7M)6qeg{D_8u4<~X^HUbc&J2v!`R*;@S(r9UHI<0!)mv;=pr`TNTo(%@3 zQQ(1KeA??rUj#CWk~| zn4c)1Vj3(s5}l6DMK(}M8s)%Rv9o?3Dgh*$E%@&x?~Rjn_)EAl!EohA*I9%w0^!pi zZdQXi+1j^}Tlx0x0L9VA4U64t^V$8oDfVv7Zsm1x!Q5jtnlPPC*rhBIw>9FtdW}u| z&Uf;|S}Wnce!>v4f9}gIX#?xxu%PUpY!CR>g~4^KtU;LZsbSVH3mSka`o#A25`NDV zI~g#c86V|foDdSkq=5v|cVcNJY?gGL(OeLjM_fkvU$->&y48mQar+ZUQCDO9Yb04p zghDEYw3%Gt%wX(2)3{v*RXVL84^ZdaiT7Qt(mh>o$y0TqaJ}`H|$3~SAt|D~zDHgYCM%!8w zd5%WEeI~rxC{+4EPP-37LGNXL9ZfObNqGBdBa^A9CoE%`qG_7F2Ul(({o!rcvX%N; z**sK2kd9A}E#H65l~M)G0jN+9q&uDU$eRaa?QGBs z$AwKpfXSF~A(p(@#jhg;RnHUVV{h#hTIJrun3F07);LH^tB-yk*)br8=j1%4t#d_2 z>iY9ZNVxiZ&P|zmy-k#izgB0fcxhb75kPeEz-6GXU*PAqz2Nc@Mzxjv2>t7#agXrb z+qP#A7ztb2#n6x#3U+zL>0ETR(%hZ(TGN>nT2DK`>Mp4Xu}`lahiuJy9{7jPyv&Cm zU-gom3Axys-N5vd4Nm>Z)(ezt9>ucG3?dMb~yguG1W zlDrJ8N}^}b&l+Q3EfLpTkpnsclm2IDj|{sPWo0^l^^;f zgsmUyQ?XNILlV#|jog}bgGOa06Rgg>KpUM4JKE4t=zI=t7aZ`%W^@;8Hy{F}X!Vla z+I7+t_6B8>iYD=jHUK2h-e))d`i2JvDbJ^lOuX#?6;@4I;X&c{9d&)7K;4il}NY|;&MduCf>-s=U=_?IF z#JOwwr!XOaRIAtaJ>%qZIQ><;jmJWX>;4N%?p)bwuEGVZ5l@%H#KO0cjel$ zmz30a#~${F*pv_DBj#D>Tj|Xs@Y#p_q>tV?12y4JuOogM|3fvT>-rETyE+Alfjmk{Z#|^VV^yNi5(m8;CmS^AP(9~{89OK`k zD*8WqhSG=Hkq-DfGW&m;$~WRt9Dpf{rL!S4FU`o&f-H?a4 z1RiAL^k~@ONuwFjaz#_3q4m;6f>xa18`+nk-YyAd9+Z1u=K^BSBgR|h0-VdG;q0%p zRu?VG3@fL!Q8Y;r{!Qeu#}$2!8`TKIsyRj)zr!b@JDa<CRba}lZ1y;hoS7xDLn~MZC+TYHdV`V2bW4tBDLaNfC$|q zTNLmsh$Nm*m(&m$Py6(sKSO@8z7i7nYGKddT|RuGO}g52VLhVCDeB~aDtmp|u^Buz z*I&H}H`rmVs}548i05QQOa0T76Sd1Wv!l0V%MCz4r4!SVxw)FJD4&rFT2AC4rsu&$t#p!sX+WBw+s zr0Y$Y;`>~TwGH&*GE|R03YWiVZ7B3S!j#(hF6!H#hP&x2o?vps0?|(8<4%|XyQjbk zNHZA?tytSSJDj!EDl$?=dkja6)67R~E%Gam=hfZL@qF}xTQ&-Ze2W}#Wp5dRi zYsuaI0Xxey&C@`)(8nj(b4nV;pTsh+2i%JeiU;Pedlw7tN)~&J4`lM^Jekdj`=bi1 zsc#BBBDJEDAs6|?$w9oTS=vXj-nm=^0T&d5E+bD9)bOq zSnTHcmhj{MZVE}1j4y@3(e869VEjdCGwcviZA^5SXfs9ynx%{VVF8X|)mM+hC(onH zKdplbHnLJEE{!YGtq6+`xva_mjcK73Rj?hFW@-G3)I?+Z9+MrKU)~mvBXB%|4p44k z{UnFrLR@}8#cr#*gQ`K*A5x@-$0;6|JfG2uLmvI?KBpOI;7$QN8fLs)^gCWK=pHhkah^gENAZRMwC0k3TmW@hFjrWdzT{H|XPt!8ofDAP9>aIiS z+`FlTA1BB8EeH>cT28-aVLZK?xgd}!&t?NAfKWNbG#33i#-{$LE4?}SYi{61H=Cw3 zm$dzDylbP}X=jbgQ>JgF5ROaWAS#*GB|?t}l#uHP&?Ivg1K{94k(zcjTro$_aHg`) zTCa#d-ye|wyQv4uck6JVd>!kl*{S|Tmfu}y!C>ci`yZg^cf#pq+KdmFBi!V?$m0h5 zws&xwht(}Y4AySv>?>OE-0I4>$S>;xH?oND)!7q8Wa+cz=1;96wF+Xa!zd)r$RgO7 zlnKJnj%S-tKhXMV{Y}7HDTgNglVpIm`1O#qx5HXn9o&-A0d$7Y#%8Z%vy}41mcV?Y z*+eJ%N;Ev`*|AnCy_L%4bH*b+8G1H0Ry&DUOgWjxBor-bhqNKijB#>&RMI!hx}-R6GVeu zMe=)M(BNZcz_eI`)yH4JF1g#j(e$AsZ2mZyq;YJP=_iKb?J*(YtWlchuJP`Q z`o|G8Hiy8y+d4ua4ButMn>nTevjzHO+usj+?fFY6T_v|;FB-W`a}dYq28;-u;)C8^ zT&6{MWbIo_v{~DOQZ%#{@E(JnTx=8285N?wILw$_tY_^hA%9)sJ6} z2_DxWdn@re%Cc$a8wmVOWwgy*>sL5b{nr)yTPpvTW(|kd?CL87ZU%s3TNGC0k+!2^ z1XMv+U4ZHR{N}sWq6{WRHqP85=|etS*W_a5+qAz|hpf|SFW$b$IN#21NFtj)g86x? zz{4~gWr~K#N4K{9)=yD?`1s|ciOqr+Zfl7w2zdL3cZ*}d26b(+w-J4Pggr{W#7@2i zg5I~L^Zh5Tz%?GT4^__GcC$H6T9SB1Z)1_UUmbJX63>F_=33i^@6Zcz*)z*COvC~5_%SN^-JU7dzA?N@B`*3m<;_~< z;%{8F4)9RiKELjrDH)FVlSj>;&AL1q`Q{}#4psNje#mqGM}`+l2!Hq7Gkl|L>jaTE zv?+GEK&Dxl#sa~D6qfUtI;Kt;BnDSm<;PVCPz-XJBFiS4sHOijqDoJzd4{_h%zQ#m zLsVs3oN%Ez^}l)>Trp~MFv0|wno+kuN;su-(AlBfsn*A!iv_5bXHs;HKkgwUzt6%W zf_|UesdAWwwC<2rG}5qOUvt{yYoCcO)QE-!kv>=#W&AxaIj?v=cA#i#G2Z<8h^;Ff zm_KJ72(jZ4A8r&c^IZ$i*c2`O**qF{f%A>{z`buv(S6^kZ*Hy}YL+%`5D3OpPv8hqEsciHa}>$)UEN2AA1wPEj7)^x(zsZMvMv<87jObaQA%4-^POQYP@CtwBD%8+*- zikmP2Eh87n^y`+=AFsDjvGr3$sR;)*svm$$MU83ARe^<{#*N`FHDa-9qxJPx_uGeN8Tpt0PqgSE+og)Vb$8z{7>r{ z&!W(!8#F483Td50K+zcJQs_Ou?H_%9@P#Abo>jK*hd}QH1K)9R6#xY&Lq-Gy#nf$C zVrd{R-f9IVnNNQ%TCU6nDgeFBeY4~P9l8`C}#pCl@rp^W+1;rs};XWTtD+MHK_3OjMuPs zTYqC?6Fejzd~$dR%Q`WMD-rjvf)B7qZC12A#jTmoj+oO_|+z>V|7`7Yao|KKYTz6|Pc z-83gc>nTPUw{iY*>kXz#J88r_waQm`Za=*sJjkgCOXSQj7s&}=nlf4^14yiBTeq*5 zt)$vQ+N|w_1_rUXLdeQ)pIQY>k3@0_8TGzOJi9tN;?=W!rVaz-hz_AQhYY`6T@pIU z?Q*7gjcu~Fzf^IM5+%V~MtulQuLy?sQ+Lnn!EMzKP+s;$Q4xQc-e^fYYg$l2Z%cMO zQNn$lwN+`-ZbGa02f0`6D>Hc-#IuFo>++Nso!lW!=#42AqN4ROFae-4R?U`c3tIQ? zY<=RL8WY$tN_6MzH8+z#Zk~%b?1cv<;qQs94b|rxviT;N5Bp2)JL|QrQ^G*J0mC+z z%d#2j9}8AB@nu3dYV9QXNdRVl5CD-;*%uwpi7OG|E#;YvGNpy$kqny@$<`WQ;Noo_ zU!}IN8C>ybot2fC`ikphYajylK4)$}GFHTd@{i|i?2Vxi&2z@Urt=!du@d=UhWZ4> zO#=8DuHneDb8d3GhH`d_p?Bj24!rNi=l#qK2O;MK_GnX306E5}p5KAH31puI?Z(ID z$j%M8^ReF8Ny^$2OCXMfzC3aaxl@VtO6kl#heGXNP6^@J3vUkOniZ1&WS~AG;Qkjl z$Syka?jM_zl)cvJz@j`e)-%n#fHt`u*35MRNucN3_aYqL8fU%m8I04`G}pLAZk?fj z3o9=a+$62>ct!G4fr&qvGS*hM0y$DDmqv+QyeRil!gO?T>79QJN-)4G1;z=D*tipb z$WJ}i_PD=!#Dx|xilLaI9rN?D2}}Kux7mt=#}=3eB_=6g6TqV*8GN#PUTBdN*}oru z;^lmyAv&)X1E{0{=pRz)W{aWb--2f=GJ(jLQ@Abf~tR!-?d;m!a#5a;5m|fiKEWnn_?-p}sTGHb5Zz5l{ z&%$GV?j~UNlU^yqJ9;Ha{Y4%9ZteW8Xo3!2|Ak-@1^0gX72+OI@SC`Hc{LcjT4lyL z)~xBzk%G2%(`%e4`vI_Me3?}7ZGXkmc(wXIcXj0{dr#ez+&cMw)>Q;@A8-SvzRK?m zp@0U{?#iw5T0KcxRjx5XcZ}Nl>IzBry1n6t8W8T|V+M2G)|>qCXO+*^x0O}lP2}5C zBd~fOgk~>|ctxekzzdoDK4>#Sfw8}UJ~5ZL>~om@`FJ3*$&XJ6bUXTcxbeZFF1~p> zO~*K4&>^{rP}Lo9%(Zke79`9^$G@!02`cry6&xFt9v&_+<&KwRyeLEj&y5v^>- zhw~52U*ZAKT8q3sQ+-pX!_m+rC94=02yiGm9qNE?sX>_{=Rqkku^;R7dU>@Ejp9u% zm*twK2~;O%-6MB^yA`pk*!>e~%Q3)SMd;u&hB3C2$j&vmDZjsLqnh8Vf*^yzlQqgj zKIj;f4$LE{->?<*=0t&4qO{JH`iCFTOFs8XUfwF#GyH(pML&>V_mpW0jFV!@5pSz5 zGRPTiRPR`b586baAA3vpTLYxZv*v9iNJ{ncL4=_d-+n)9@)%wr^{i9^pF?17@p&<9 zG%&--qt3rKAX1Xya?5u*an#D+HLYh|F!#kNiPVpAiR{jvAM%^G)9S;YmDhJy;$7nF z@F(TQ3?TFDS&br_72Vs`6tqFALLmJG1Z;OyW61Mc@CiPMRsW#ENrYcxyv@1cl6BekXcmhmyd6R;z*Q4S?OihHT_?xgc)|I(PySfizh8qsgm z5h?8K(dJgY|G73=teo5umya=?+xXrCwCMzHuTB!nWu%r6L`XOOk=ab@)UzkEpME9i zF0|m^RwMZ1MrLdNZOwFMTOnjIj!m1F|H>HDD>AgnoHuPN-LqOGdqZ;O z(Bi#4l<8(gl#q38b~4tc<}3|uXo$IFST7A;jk}Sk8U)f&zLGCI5o&fy`1JL$^jLO+oM zT-1-U{C5-kGeH}#r+iT7r_Ai_m0LZR3*0%DK$M2_lVu&X)o~GI8mfnJ5O{ET%hJH% zBk0v3R0+` zVA{K_3}WvEHxfYKNrP_9M*Ky5$A7m$wp)A&EM;uuDIJ4pq%pW&TV&a%vBI<52;|Nf zz4dH?P??~mr2;u{f$-}bi@WS)&uSvN!yt|r`=vd}-ibdoeKoo+4g?JEF;1>1u{5X? zT4}@gOo@byTiFnPR3;7w`Q7ciH7terpjh!#eGWwxPv0>^XOCPf^(~=e;xF}#DG0nU zr%3VO=J9zAz}?0~SDuTZEWSeX>yuYoGrrft9yJUcDbC)@FDJJ|QnemLDu|E|$6hx+ z$}QW(sL%kR4_&YYD&dsuJgx>R3$_g<$BMvT0D3e(0-GY{&NP3zh+3ulr zcN<5n!c1E^M4!XVarE)aGEj-1^fSM~in)2Jz6rgS~Ex{ z;`M>-z1}*W-2FUz7Id2HK^?W?oY3&Nw(Eli8)?pZ0z$uYsWX(3f!B(W_h1m6*Kphp z*2z(5|3D^xf}Gov<+QW9Yb+K|C#~sGO8pkrPQ&}Q))!%PdpAh4r$G}$UG>{q*D938 zfCoN-L*7j9Hr}m7ZGZOl%<(ZAx&ixg!9ZA2bqj@_J5RubgUb~|wL04Gp99+|Ud@)z z9=}*dDT=P)>U)fj#zCp%Hyj`&(H9 z09h%xz9Mm50fAceU$e9dyPbKn5U4}G=Q;`gnl#J6mP7TI)+CsRq7MIqW1v-ZfwyNf zm#6J{GYSn2>4b6A=BCvjs+kIs!{v2Y5z4a0F^1F^-&s#rhWjb=;^dpV4NMfQbcwT> z^DmtS36H@H#T%H7eEYa5_7$%6N>kzj&UPoJo@epxZfD3I6y?Y4TW^n_k{9hP8D&~e zX@_?rKNK3$a=MK7$&cG@_hfOtuIxsW*y9c+`QhM95jyR&>T1KINMz#&5hUn;jlD)H zn_5N~Fqbn+z+R@U>z`Uj=KEdm(kp=uWA5>& z{a~#)$o&2EC34c}$l`aSK{=s2w`j)&4-gCHk~Mo1hPrSBs9cm3`t{9HBxGk;N~``Mt=u+U7F0Fe&ci=DEzq%1BQS>i3iuo zQNPRY=mNJB?(&p68pw_4KLRTCGw53)g6~c&cx@!TnS-^`vJ2+Vog983-qPJ1W`rHe=%&1SsM7~U^Pf~#Q2C<7T4?kxSLvDG%0&ZsdM zKj0$=39wGOh3=sVz1u2Kdj~6_N8Jn|$Hz?E&P}IVYc05#VC!w2e{=Lqp~I7N;_H;@ zh&IMvKYzgg4qG$1k6cyl&;EF&Fffz&ze|)*-ql+YyT(nQR}67K2#{lup8GxlxA|3T zlLazV^Sl~b))zKL+6{n$t_LW5>%M}FeV6WO_^KPvMSka;3o@E~FHk`51`nNv$C`I| z>MF->%C@Hgl7p}&V^dW?KQF1;SAe33*TOHK7Jaqaqp(Qc3rzJpW=%)K02~oOI=x-JVm9rd1S6! zEmSSV=0IkJ?u!-O$FN+1#$|f_%d~mEb%*Ex{)jRa2#(n`72~G8~(G zsIju8by$T&N9kVvx&iNIPkAuZJ1EmTBq=syb&<8xdgK1extML&J|s3>7s?_zFR(v zYcgEI@|@~aZD=h|paSodt}gynLOzlAqVTO5SrS8PrpZ@2tPlF*-Q?wu!mDs&-S!5= zoO?C~_iic6d!adQ>=9bMb_f(xQtR@`s`y|YrVP`s$+BNZ^6%#Z`yhi5e|_coy7p;Q zZGb$h2Z)mW?@Z2Bb_?bg&lF%f(cpt42FOiEd&qqJy`|{q2>ATxVoNN2=026&!52ug zwkZ2tLYin%s<|de{V_pw2x5`_A(I{WDO+HC2 zv|y4hjbT>w*5{Y4B8|*D4g3VR=IQTYz0eCRl|3JDOF-&ZQKp#R8f~>r#Pv@R?!y|F zNu;3m;i7Kui^YT^#Kq1IG1Vtx^6r^AI+g4@J6v}kc(-p>L1MH(%cWy%RLZr)51T8L zZ{>TDDgOKAdd>1*d`YV9y~LKUvq@Fw2;sF#$D5pG2~mu3Y8}rj9u1;?o_p*W?FXJ) z65VfnO-Beu9gL4Dql?w|MrjyF1U=X#h_V^ND}opK`7A|3U63Qi%n_Qi&-|JUVoH@N zJ_S7;M58SY;pmHLJ+Sm^-~J;qUjY&%KjX2=q!{xoGtP>WZE-;~jI+$NFZbyzLjia& zu_xI6XIgoBwD#yLF2;u4(XZ(3)Idmg!@09HzfPIAJ2 z&;2&h4&Z#kbn=+B?b#Q8SAy}LRqcva3qy~U+U2{c*n}~sqCm71ipd4P;dbh$;syqP z9aWzfBd}Y7HF~n>rwIo<0sx_2C+ln`f0h@2zRO;^4?=i}f+^n_2mC7(9jdaVmr}6^&}Iw=6AkoJoUXq>d6RfTv&J0NLnHexez&lqtS;SNae24D zh|twB9ac<7Mcr*}{`O+xZ7j=IJecwT=iL*_cor!#T%Z>o8Md01Z4|R84_+w!C=owV zXa|-Kk5Stk_`ORo7(eV4=?^o~Ue{mxG4?q&|MQ#J6lHQqb7*jiuxXzC9e!&u?&5;x zOD7pnuQOj+QR<=C61ayRsZ!W!GH$RAsF6c4cfHYuO~4xiBeV1226^Q@_#_^-QOMF# zCb%4G-R8=;pmJ5#c*?z>Ef;miUMLD`qc=S#`4x)XM()2Z^3=4C;75T>?IliJMo5p} z+50T+6cKSCAls#>*LLs+?H>pR{<#0)qW#%&Yr=52>Sp6fvx6EF#NL*!Ke03U6*4QtRrq*7=(VAVH8A5g^vdTR_a%Id`)$7~uw?#Nqq3wDy?*g!on}b0 zOXgb91v-Dd3E{a*Nfx3a;;pzIO*vk?5`;1=a>woL0sb14Ug&qTBGKYaeJ(})X7tW$ z$qGQlcTm>I3N5tbrpmHXzy?4F%h-&`u~iBHUY`+HdIW`?<{nR1S)G}&w_|v@#e+t? zYR4R{Fvqw7nO}cDOV9)Qx@8rQX>IVIxDxNKo3a76d8YaVxB*)&vW%^9Cxdf;ZbYBG z8@v>`6Bxw)_Gk~e$A>CvvMUQW6(XfaBrdx?A@_V7!bBo>;dE}u;=6gz_m?W79}+@4 zPqtT|i>p+YQHTn=#D#VKdYtoFfSEI+5tzP@V^b|#7PEG$f92t$Z4Ih)oy06;L|1FtJpDWx8d47*@mk zZr<2XXps-&i!Ftr}R~b!^}k1{=z@1(CF|= zgzLu3eP~NefWo5H{m2ySxQEJjsS?^)TS8#z%p7 zP$I_BZ-jZyIOjo3x&L5X`BF13dv+O;`xq9!waJzmH5MDep*}>$r2I2^Z%UJAl9XVw z;6UjGz#9{b#{Y$bCD5Z1$fQe`!@ej{kiCT@z3g$nj~x{@4EH>=X90_P7z%c!1JNXx zCk|~ZX|{t0ZmI2LS4twqMf(=X6W$UaxlhYC%-ZP-lqT1ib}jGP3Mi`#-ZRR!szwch zn7n-bxE|*l@+{tsXT7AhMwQ|1khzIVbBXJLN2 zOFxTh_3fkVzX%bx285Ue$iG79F7g~z=b7_%j3Upr0FjHS12x~Sd8L1jTg6Qo*JPFX#A4@ zj$4w|9<4EurKB;nr?8oYhd8!BIa$Pu**Z!$U-_+uc+=m zZV**urfL=ZRYrgP`D5nSdkYcR$H?6-0aRh3Qe>R|D1Y|WkZvUb9ug$twWx1m(DNYYw^vWM! z=Mkt8fOL4b^6{?-N1a`D4R#Hv@E4N{W6^sjJ;H-9sD}%-hUfzM{FnJJ%hWWzSC1ZO z^zi4EgTRR{H5_jzBRsAL%9pDJ-kLogU%eX=d1Dz4|&|0(dHMp;Hied_Yv#=QF}kP%xnR?l+1i(EE2 z`pmOs8Ij-m@v+CvV4iR*UFCE-Oy0K<-sM1gM$6iRF5D**K9+AjSL8c-cr-!~XMr^6 zjK^Z51Ac7it?;MOYk*Lz-}t-?PMfG5L)+zs8>yc|D=W810rYc`p0(4ds==6IrA3#< zMYf!xAZ!_oXt8=(L5*#r@{{_@)3MI9_rwm7Dq+4(BV=&e0gHu-=xgAKRld9jY{qxs z_m6;o!h;)|0gK3S*?sW7l=H2;sRry4V0%>!WNNSbmoi+&iEo|1(%u8)l2DJZ`w7faM=Uc3@ zVPww_hHv>zEB)!Rmpk^(w8lyKCjdE81Fmgtyu+xIzD!Ov(LG-!+93J14M$k-79nDO zp7;cb-tsfT71TeuT!aDT`&!PMI6aPS{@wtFobPKqQ_@V&p1UD(+I>p} z$J;1Nt?w^g(?pjNf3jL8 zdYbXqR92E%WD0U;akkD*yopTMfDx}8oM$w;1KwvWHJ+v03VdwzkHF`!^cT$qjDnr0 z7kD8aRoSSPCE2qxo`TTdZ8&qi30iVuPBF^Zz#c0$dN|e@E!;iy#N-}}!D!>qezJxB z;sFWduFgH8FW8@++y=pYz40jWX&vv3)G4dG-J%;p zGJs}_+9qO?ChLlu-d6EF0`v+n+xcd3kg6F5`Jffe5PiaWgVf_fGn#uU>Sq6dw?K!- zRU?$=RY%t(rftu7YsN=es?$%P2K>=`Fp|Tc-sCu{gpbJoEA$0!%!}5RV;i}W4AvdK zFAJ-Xo808mvW##KqZq6)(31LGwZo#fU6kIQ-tyzoB3aJc8cKC~6$kM~k|=mb)zcgM z+48ng{qIvF>VsnEyM*s?LA|VF$Q>w_31Qdb^Ey(s$nB*lxB=A9$~*pWem(wk_twUL zHwu8ymrYTpOWGhcYiQ4dM zb+-fp8zPtTJ{soWt#B>qPQOA|3?UF6m_w(7)gWcGF-4vrnqKQsHAl92i|_~LaoR(H4on1gl1qaC{D9E4VJ$ITNvWk$ z^x?zBuOaA-5FY9U<}T1>rw@NM9Oq%Yw0~Qix7? zo?{#o_LF%lmE;Mv_D-7}IH9fzTLx4b3T@NeK@Zh=+?YhIv6Ic_$++FIN%fs5rb)4~ z=vp%%KVF~iBwvo!GJT&7fWSwt`h!Nx{>*NgSkSlsrB=UT{8lY#x8>BKluYTWqqb)^ z9jT`_tyV6mB)9l>`4nY2xAWl#(YfM9E^wf631%2Xv{oc9pH;hTD?>ZXubra3r(iK# zEBJSF+j1n<{~&dhBC&p!%PXh2n2Ha6=K*dWA!tPZJU3$A#&5IAYEH_zLQY)xUwpZ@ zpFeCQfOrN(91k-uY`+`k4TJ>2Cdzb4IU~GAZOK*tfWNk_xl9N^^m!m#Yw1pdIn~jf zwcmujD9qU)X?Yj&<#YVkm>j_kekf_{TZLVlXJ6wFT`?z7})*%bT3q+eXAkvPG!8|L@rP zr5f$Yb~z)t4rFy-e%YeNM;Z0ShfBIep1bjSB;a)`i*ywV=bxz+$AsSfk1cSt(9^;E zGk~8rGyJKepsj zc%lINvmfc(RrmqqebsAJq_8u5TJg4S>s5uwLUg)udfZIXQl%~mar5oeu z%XJEny8`#%zAiX$f7MR|XgVXoR@b(`c=m;++UmcCeu-E8S^k8=>b0j@4v68=TbDcaf;z=Wd`0!8w zy$ITtw)0>eqVB-eTMVW$9s)V@7 zgh^E)263pUb@h5FkEgc=D)SGHre~vA7t6M2W4G)y%L|W;yZ!!^-$t!i9j^?hs81?s z;=%N*I8_(frAz0~g02!}irh8NNR5xN&qWur^Re;w_UTU4YWU1g0y>5-e zkZDdq(>h6P+QlsK3UHv*UEMhO1?Iu@jRDrW-VX8Xr#87G7`@KRX&qW_X$Lj^cc&fe z>+7?(aQTWe<>~hXx8&xDwktp4=DNWIeQx>vNBmewgK+#gIK47VryWy9KtdO`qKyS zNSk%Ld=#eE_cA#%^$>7j%%^L2UdVFpUyTjw*PY1E&qo*jsH}LYwD28PACYVrsn1c= z4vqOp&e5T;h-2antQRjrxiUqMd#YX}7QuNhnnxli;L-i>HWFt-*~7UQhWb1CQEumn zQg%AiUOWkoixcTAf{zT@5*xXp8WVqy=<@1`@!+-fC+PBah2gw8qz_3Q1iuW|2}(QhP}T;0N0j~{3XFM$R-xy(ZJiWNh>gjvDFpzUB@1pA~iT#+oRc*+h~ zIiO5Z?<~dv)5gvO%+s;3g#OA~h{=$`{WtSrWM?$?x#fC6={2GCAAk6Jx{$)g)*ONG z79YG@eN%~*5&kJOmgn-=m@*XWaO(*7lJP#+n9YYVNYg0xek%`j1fumV6rTYN3yWAZ z1emH}x4+=qz>rkX^s?IIg)&F1-_DRnKcOtI-f(C27YE0>Jz2&^ZqtWHQrGOFu62XU zz@+H8`RDBqN!b(skpE-mJ&8s~VvN{Xi~P(6Irpagu2R3F3FUG?Z5v*!%(}^8hJ{(1 zH4W7tMo*r<)P&x-90~nx;9{#!7_}3#znbs<2k41I`nMdFPLf71_}@)NYQrjE$Mx@~ zL?^c?;7LFPpPDeT26!?WWBv*-{Mp8^codHK6ZJ=o?d3(^rjOzFn z`D_z2Z&g)Fhmqa~y#BnB^q-gMEK1he3$x?QFxM@H_vYWb^c*k>HIcIvciIEIFl{tN+^L~rC94|YO5 zYQfua^uca^QAWfDL_C{YgFAY0II2AZtWKu^ z;F7v}zQrly`c-AL3^Gl6yoyYX79Jfw#{KGkhf`M}k1xx798l%kXWFl(!;ksj8b`(6 zh;^Nadb_p1V`3w1Rw(3;R-o$8M5XBiaeYeRei<@JeN1ff!~ZEd`*5cB|BrXR-8dbg zI~1WCa*A;SCR!)1&l8OuE=WEcP8$GO)Cs65kIDLLip z{IWu9=j%0kvZYeoh9U3BVITa>$yYcr%jZ%+?cd4x6aFYU zxl5~J;YmPy1?maE(rKya2G%6Jjf&PHdi%ADhTum^Yw7azyEIwCQB5z9F<4o7+dsuObTl-hdsY!p{6;u{bimeN%+`7Q)!s${GiHd z8tS~l=WRqJ>dlZ^M_D zU=H~*1H-jSQ6@LhW1)Qw?#ww!NC(`gtP69jD;9qFiA(m<5#_1x@rTF(k7IXMR`!&C zFfOYYCDgOEBG6SIZqip~au9v?#7U^`L*P(y81*YaCN9%>w5s!9aNhVwMO>_~Uuz_E zAxMaz?De#L7~ODXsM<8Y^KZ9G*Zs`95OnO!{I_DB3m?65xtcqQ;~%?!WGMI1NdXr3Sy65jI%a_#ydV3?jkhffpv8K?ZrO4l@Eum z%n9xcAFInjJTg8bYHWCNGt@k1drkfoE)4)|_!ep@dTP+{YWvF+DqInJRe6RJWwLmJ zmLPA-0|OVT-u{O3DCbN}dDyQILnPse=xIn6pDh|}IpuN?Zz0urV6z$YuGJ)=ZSROv zRqMktF@w3WfNnyK#omzh?fX>MQxUd1EGE`#zCIu@Fzly+@%PMMK!Qi}ftK7a0Ps=i zq6(I#l$GN$E=eVmwdkNCzuHG@0e4_~A?0Fgs#7wmDcmRLEgbU9c;gzbFE1^KQk z+#*?X001du>voCGwE5#B$zC$FXPLO(T@Jd`510LtGvk3jk19uw^8?y%H)kM@Kh@3r zxXJjj+6oS4ep6qO9fuY6;%L)$=-FjEpKC~~%K>jqkcyyvQqtabF z8AJH7D>jIC}xh)Yp+AvN$w8KkFmKaIw*ra{q{nwoj70<+D& zY6D0>H858l2<$_J#NB!AW}R<7#;P*3KBe?qksL~}>jK1`M9st>QO^I9t!@C(8Th<` z%h+n)9VL{rIOaNtr8T>?UBJ;szT&u=zPRI*_vwhzv<^d| z)8P}}lN1qMrPnY_2YFI?5Z8~uKIn~G-9!l6Ql8@ zbiBK0d7G1(PMsiPak+n_o@d0BaLL%8{O4D2IRO_xCd?p0!4^NR&JIV-K+apl$w_y| zZ0@yVz$&6OO6{ariJTVfRs{J;pUO)rfFfk;{-G$_4DaD++Mp$7@iA##9yw=6A|XdN zQYy-dAeQjwu^i@T*|+GOcIT0Qs33!?)1u`=sAF9Q#zi-Yug3jwA5^a8V)E_S!h>&8 z&cq|2gI#fgU*MLfbSC@$;G28 zCz0Ld^ZM|JqRo=K!ml|?B6)(v9obe7Ym8_JG}YL^;mbgI94P5y-p?UOZAFmubZ0y= z#d59I2p5QA9b?P_mStWFeMIF3gpyL|<-NhgQZXA&c)F^fV|H28YM~}=@`Zix8snm;PuR3Xz!nj`{zdLX66Y~gEcV`jjmZd`&@{gy z$v_*3IpV(U^j?4KL1Uoe6eOzEKqwk|u88&2_SwEX+K5eoU0{7HOR9=#vYPPP(Tr}} z<%ljDFZPZ7$h31f7?gK{w5gZyi}vZa@Uk?=gBlN1QM|yz%Ilmk{SS9VeQ`z7Xl_Z3 zpyyBCy~#O7u6T|YA1;_13&z-LHvjtqFi5RS8r?d%umF1EogvYUGthAYyf>)?I zN1}+LG16Iq!?s6G1mc*T_GldXLY-6v6t&~9n`!3qD0p5GM%W9J5Ugy`y>q6@o!98~ zD6Ve(P|b~`zs_hH9+58l3=u;$t*@?GoBJ^~k z*770LKtl(pHt$@BROFAG=SR(4joUutO&}*kajo?wiA27dV^;i2n9p@&lG+?+(qlkM zQkjX;458BHzeLQT=qiv4HXRg%_+Yh~0Ma`XC8iT>{lIQONbDNu1tAs16ZZ}4}& z{lhA$o48|PL7MN~q(BLwzL%J;10-b+53*UD7<#ivouLuA@eY*xrV;ZdZr<$USJ2V6q1i6u}RCR?%bF zU-6@5l)x4^xjijrG~_O7#dG|w^AmWPwXFy67d9ztNWPwy9fTzXdS6lb$lcJ=d9D;O zGS7c1r`)SYdL|ryw)v0xaLTuDGdd^FSYuk*tpT4b?%HVGJl{?Zc@W-u2xliu?bO=W zop@AcLMfgPG~umU>xkv+aI0KQnHUOQJQYUj#e|?me&IDn1{eP*^o*d^+{l~J{P*@2 zxmJkCIk`sV)x@SrAYJK&mIG_6(x>_K37`O99&tR=)Q8sC7`k*Vq70qyknd2X7vMlF zkYi36qVp+7-zeg$RX-0qB8Ap}@img1594MRZR>IIMpo)|PcFq{aA}sk2AO~@uK()a zU5W6F13lh{=)3dPnRh4=eqO$P&Uv58SXz^;JSk8>-n#4w0VwtadFH8&vG$GM?H{(j z#6}ZsljYy32DU?G^hOT@g!mW$TYT-&bZuO3u=2yW3a60We1$(FGkKI~uBCsDXJf4) zK>v;Nw_2-EeI2&KxV&Dood_v`Z6C5{Y3>jlvBDT!D~O=x-+TW9q&w0dPZ(YWFuWn} z$m&dB{50>I+UaA(^MJeY`*RPbSlLy&jj{F%fh@;qVIWypJWrN!_@~WueYLaHEekLk2>)hF-1Ad_P8G z=IW^Re)k}&g2ukN*dQHL<&QRHb~rK|e{hMv=nsHx#&jZeNWr@QP- zJfLz_1d9S78-c+X0>*ud-)ccWn9YlHRj z>Z2Q!D-;0F_ac3#PQmM8VbeKvZm-{Q(}ev}u&P9D7Y4Tx1R@u!CInzFJ^^*(7&lB265e=9#M zQGbY5zDCrV*U!r^uI|I^|FiO4ntTC`!kW|n>MyZst7}bI$hI=sttcJMvw)uS19Fsw{t?xf5LPQXRNYr@nd=!| zqga#eQ}4-d@dh?YhIGT0L^j)&H5_&S%k^`w=l?`~8+EMbKGA4Nc|05!Qw25w1}rxZ|8dncjce0ouTWf;7MS`GW3 z=k4|NzG$blnu5C_d);Syf=zXZhty(eb0>0-DrnkT3rFb;v+4f{=kra(UGLFTgHb6t9?pG7&nFx^ z584vP`>PtSHC4`Xko_-?=JJ!mr}rI(5c1-`EkQ<>{+kWnZ#ZP4IxAZABxd4h+hKcOzU0Qx}oZwQvEE}PHNa74%TIGyGe`rnC`i@v&Q;3x2!ow z3FlIeH?p1IZG@ZLOxIvEv(ghNN5S8|q##;qlQ&F~qL9Y;-=u%=hx@x*AYRg$+|v2P zU+C<^L($Mn#62k6KjHk-o~4tnAo3WhDq7y0-~T$B;chfP-rF?8ij{lFmTHx!#0F)5 zwf+h`awPsXc}&v-vVY3_GBo9Lr_u*wuby0oWqB+vVr(|g&7Z}DS?`8nqRVHvko_L27wZT*dfT@fp-zz^{qF2Emu)Oq^fu>HT$Za) z7m!f5=%j&aYe1_b@CmR*+6(q=Zt$?)hD9@a;hOSVkNmkwL;?iFWncHRef~p+3@PHJ zJ8DQ{#=2fjYNA$Mww^}?iA=VIN!u|suBG#EWa^q5@JXu2=Pg3xqOJa=2H)MUiEAj1 zA^iypXX8PNjMxg5%aGbo%_eGFvYouIH@+FLS=LtEtZt)$Fz{KooS9BYW4gzhL*|jZ zEjE&i9VaB!H?rhzO*y95<`cK!eBTCcmG-x1KXlPsmDq_cDLxBnRSTGlJQxd*vn?90 z|K6(pgmIqzBN!kb7)B@bW?<`3>oIJ|Hp4GToGk3aNr%m{wW_|Lx4w7b3nYg9111~L z7v?&r5=2@CE0tO^DU98b1M=VP4{7EXQ>z$Z=WdiP%q zfMD*H(nyInE3I;+Z5@D2+FZu#&Il@3UFCrR@(zZo@~rY-`1@R|Nki6&Yqf36rDf9; z*bz*1^g&R{!oOxHN6OF8DjcN)Hg!aKLFF{a3VHRBoboiN0s&W`7oAN5t^#aK=um{J z86mNQHUWCCFEmKGp2di21W-%ev_bEWnKC|7ZP!tS&EmH1m+5pk#Ul-I&sN1WDp09< zs6rn)jZ0ecSgW=Ac*fe%$*Q5sagX&sPpHLAYxB{);LvETf@AIc#0F9Jk2sy4_=8dg zGc(3VyVa|sda?cOdjwW_SM&52(B|mH}GnI8m$`^Ve{$Nd1(d)LRiG;|&uLM!R z*6Gv-)@qy3#KY3rPA3b`Aute-WE#X!&DOrayN9#dFhuYI+VLK_dXvTf>MN;-NBpK;CMN(AP@N2eK#Kk&yy5Y-X%h{&zXI9y3ohv$ z!3uNQw|NYOxzSr@;XgJPYj?8g-dOh?Gy0=|jQkT*L*SjSB!J8}wY$(-kvrwWH_PLk z&*&SV+H77#oGoSJf8)XLP(>C9h==vfrsg{tpBgw+Krm>Co-JHbpSdA0jTyWJQ2gV0 zodHB)>|%GB$^d%4C(r~)dDGi?&iz0#ENGHmydG=I<=qu;v{EuXqfnz|bvQB-%#^ew zEs30CVT$|0HeI0dNPLUkhrh^@UTzlk@0<6YBjs3lIv3oW`T$hk!6thq;0VVE^#7W>U)} zWVwfpVG86&Zze@Y}jTNZ^e1 z&_?M&*SZn-JA>={=K?%t!J!24YTYOH=jHP5uy6cRVaV`;d@f$25(qG%mFHEz08SP> zjN^zD!FJ=yfb$%)o>7!Pbqqb+n4$+L^FHEAkhTf9(ICvF=;p>O9s0Z@>9JaO zvg|P!YNG;PBn@T28d#?D2&}a)uvMm2u>5efZMUn#GpnfgOSc)cB(%eaRsra13dIx? zC3Q`}*vMCY`(Z(^ZRc&NRYnBf-K>R-hHJGYsdSC{3Q1^nnr_*no9yn-l-hufrj&+z zq(2F-bm~VWIdkol2Kq+UUUi2Osz{io@QL!WHa5rx;Mn=<8C)wZ!4%K$MIr>aduQ|l z>;YtFqJbm&aMtpBe_XvN-IQlK z zs|cu&ZXZ^;-W54e?Ke8Fu*fLz&vXZi6B2qk;!30-qnGKd#T>4=!rS220i z@4qdj0h>>}5~^}%H_6S`dN{*&X9FoE6Ynowapi`u>7+#B`rQZjb&~(#7nG&Yt#!vI z{6ADIHl&-;)7b}Jio9KLAvg==uch@O>Cz{s1&{6m@shm9g`KdXt5yo6}R^AvZf6Da4s!>xxDZBsrC zGW~0cn^ohXF?R5$jEqP-Iq^{$hcEhQZM7r@IcD1VUi-$R7BgZ5tilQWd(2x$03d-f zvg_n?tEB6mvS>;d$h&=a*rXA~HMf9fHQ zr(BO0nEYE0(@(O*V`No6iZ+>s*RZ2Swb8EdiTsW6d zk9hMUeTk1YM`W;CuxYg4B)D+IA44yOY`$Acte0jodLIUNGxLA4d;4VJXb-T;{Q>bC zcUnOdDvhi!4Xv(AZ}*j<+HSU`Z$JvnZZmj6K)n^=HDN@b{D+haH$6Q8yrr;aeLeNS z?%rTT@%McE_ID*xmyWBGI4<`cN1|tdDd^(r#Yfd9B*+zO1Y7P7*6X2h!_Zc?tg!_W zcN23iUZw=Dv`_FSq;?t3Hfj!Bx9u-T>8 zrEzGsMR_66CqEiY%MA!)k_0GRwPYG`VtDmStxYQ6qmdn_h_FT~kTmcm&`|!3Gb7D< zA~bS}SVd}~mTl{Tj28Ex$pI4u4Po93r>iW)E4BQ^YVz6XEdSvm`2=7TUuvL+k?m4x z`C8|uJK(bq5hAO?-cm6$&}?Wat7<w5NWl0}frTOOvgEnTvhCo(wOWt(*u#sNC0{;9W|c$1IviN8)hNu8mWPV^*lH+JP^ zMmi>YwT>2#L0(P3Ok>cFeONEU6WP}Spg6rsBqL3Vz&U!yu|%=EJn^d>%H?#+jz{of>4ZE zLt{fBpG~BTy*G3C+lH;phxD%?P8@4&{K_%`t`sslr9=4#*XF-1x_BFD#QmaBANU`V zSD%0?p8lBRg*|~wi7c5I^Pl%<3-{1n&gLKe2O6BPU#lXZ)>|||Xay@0gcl|cDO}7m zE;mm5$aYkfjlq#ddL}xOvAetYDOAt{t>X=jyV17tdTO<=6_-7U@H8!q9?@u4otFit zrV>N@f50P{3J4J>kiyx%o%YVnBFW{kPWcts(fL^0EBp(uKQEMrP5x2$vA%oPYs8Gn>=y)*oOW?LQoakk(_DGwi7WGT4ZZ;pj(SL;% zT5~P@Mvi7xa3jsHI3Yvlx}s5dWI7?Aa(_dovu*2Gm^V->N5S1DDHU-%IOcKurSDZl zj5cZ827x~O3K%kr`290I7{X7T*1ei`|JWrVy@>h>{T}rX(&Dz=t;cZyzQU9tGwWMe z;vO52)R_BYZ>D;;B$x#_$3or>iB=+_`mKJo97;_cyKqR>`B5GvG$IEJRnZUs}= znrmClmuGC|$u|h^N=(V6MbURL)#HAfH30^G)ggk&m`U`+DDaAI5sW^oRT#q_#cpJg zigDlieVIUQf)3k-2xYEZ=q*Qc;dcsRH=mUY#@&Oe!ua8EYCoTJHA@0Imc);n_9x@k z&MW~7$~z`W{MjVdt=0ov76d>XV6Q1{t4dI>i9j|39e|5;AF2teD^znnr7XjeG|%2( za6~|H^q`(UwU+z!CWBVLsjfB-e|STXm8bR$i9}f(>kICYymJbxEOlc zH0Ks~b%4wlE4i>^O3zoVD+V#m|G&R??7zU|07K=Z{to-&X&T)iNEbJ8chY}b+P%g> zwW+FKD}fbZ4E%NuKa9(>@6VS51aMhF5IP_2r&}Y@C9-*WQf8M+CUH;Vp#{Kc8ZC)@ zXRdDMicKey8BfCf?mIxqinq5R|7B1*Tj z!9ZK+a;9(LDWEl=O+`IZ7Dn&QIcnE z@iqNt_Ke{2TKA2u0FyU9clW-2N$M1g-trf85bgt)0 z0;0g>G8-^m!#^ej_}>O1^zXy1s>7!k(RVV7Xg)1D9ZkzR3cD`n(+UE>l3Vs|P{d9b z=zHaQG!5HW?-}W=f^R zW3;S-%YT%-W~<4|16j7~kmwuBHWxOKm^x7hLT*rnX_oN)S%LhiS-R%Q$!TyeQE8~TLBtxu9Cq(m@gA&NF3_HmOuwfMRs@^|Ue z(zX&mMFLc|t^Oi^h@3oW_KB@^OX<~I*Z3TP9BEmrAum^NR?aL3Sr8cf9*K@@>vTw0 z+!5JwXF1(k1JI}ZW{@-z23N(KkAbVsTC*lhyb%ErAF5{^BR1vL)26}Di4kN;a`*ke^< zRdjFNE_0jjRV(hlTQe2GMX16Pihk+3u(eU9*>0>QGwD!ZZ)0^-;NcpZwZd=9K7jN9 ziav4GC-k6=!eD7E4);_4Ogm$Sk47W+EvZiezFl}olJoy2M^(ukHO{5c~d*f3!19&!(#*F`Rn2;L!e#7$OdLLFnP zh6}w#zKoSy7$a) zj=f#2CJKUGXQzzvTSkwhOfY}S{qAf-SRjZ=o}bau8-oCCmVd5J(8!4;y8R=f4Db9! zd&;Qwmz1ymj#<54z_lzzz0V(4INj}RI-nv<>lSig?6rIec5wqJGF#PPEdesKVH#)+ zWC6|c41%7LMO4R7C!Uj(vnft$-<&akjI!8hLqs#K zwk8^dS@OM<+!X@^(PB?cxMSkt2(23lp=6NHSs(u23hdoV^u|&?Mt8Q$$w7p?4@iqd zJE!Tn-i>dDKc^M_x8>%1Bq84$u)04JJuxNq#jUP3?EStHNQF1fgWr(aLK0)&URC!qN@aMu%r`Xd9ZL zjIQ>z|3k~F;9lCybypxD)>qRRo#r| zTo=Qm^C9vJPmDXPX`{U{=8XXZW9=9@tWdRG1WPSE-6AD*EI%e+Obsd)8xYfJ=w{40 zG{I}Eowtzy{Zr7`486qW&-}c&M(hb8}=m%yIYYj zax2S9v{LP9qI8<x#;QeOvMyx5sx$~Ht=a9 znw65K@(~GRP#15KTgsWXfe`kKgflg{l3!?bi0g^)=~6y zOrC9)mdKj8i1zhd1gK*K=TGy5wv>)N150cr`tK1=wlU8k7^nK4d(C8ZGd4q@wc7uX zYs=#=CtF+H=F-szm%OiVR(lOibi}VG%qAYH?0cRWXUh9*`L>+H^kJ-7S_tv@$1|&q z25jZH^FaXNm*Fd=blxqW=fQQWHy)sZ%SYOcs_poiutR*}sATU6Td_W9NALSGiW}tF z@stnY$Dzm1Y&lCY3I$4+>52Nd(>^Swv^III?MEC6Tdi_2s~j6`H=D^Xn-9E`ze$ND zl)GB)Lv2t-OKJ5=gzNTkTQ=fS-8@KEtFp3w+_q{999$H?DmcjB#Jr;JrkT`o#YtJB zF^<@2;G3;FTdgSkN`i1Lsc>8N>clO1uV=D5qX9s&S#kMX{`Xsr0z-G*+Ks(}+Du|c zGsi`23UFF~1uV#4B|wO4aHF@I9P%*HTv6|m3KN=1@6d%tXQMDq_!G6Z$)b$ZdiHEH zy&l*s^+Hgz(1_`$RI_y7bVQYhdCml;h|;^f3fkI5CMMe$g$O5Gj40`erk;0e0_U4B zEQ1+(C2Ax79iLG0c>TXEZ$f#HJ)>?_^Iq=8aN9QwCFZ{^ORTM(lfN*ei-3wWWGY+V zL_;y#*{aL(l}^Z;V@*;qHPZ!jLkp-|aZB5f9_T2S9hb=2zlT5X(=qA7=n0P8P9p&~ zHVyx*o_30fv)yQC1!8o5Bk-kFezSKQy^gaET5730Z(J#lFU>c9vd#BIDcn>N%CMv! z8id7u{mU~*q@@8%pPuSl%C8Nx(-cBA;0;n>KtwMMGo&q`9Ujq57@db2#HbKUJ#Lk5 z&C96w+7moW>={Y?J7fC!Q4+z0L_3J-eA(aHXcQWZQky;h}F6yWDB z$}~0t`OI^!$&7D9ZV)X@-h649(-$(F+X5I&)0Yj88U@u4Hs# zL|WR)ciK$jeS^y^0uZ}DTHgX_5|4`x9IH!$_{}sAv19$YW?975vCtb=w2!#~#7cm; zN_nh;D$j(qo*0aF*M-9pvt(hL38Jo`t z#%v#_{5~2AkIgLML>;tQ(1Hm2lXEW)Thm|)cWiXY&=N>;l)VN{NXXU((I3kxmVd&< zS6q)lfN6~DwtARcI|HKV6Sk#2K{p99mS`Cyc1U9#AT9~p0J->bgx4~>c@fyvs>bje z*XsCE1k%CzsRiO-tq}dj6nU1j*3a+h-dLkDv+#DrsWZO5(wUcfbx~ZtAZ1x=yopdk zBaUv!UzWl`_63Nm?&b9!YWf7U9=HthMZ~}^;sH1SRG@QI*MzXN4ItXwsks2eW`hDw z8oCzEU^62Mhs3wy&RHh>SbKJ{nmwT;*j{;;4QhGyF9yywFjB zl`V@j9)L}fk$ZJj!7mU!#i)(w$+U7bf3376u6TZ7wzI~wiGOz4%#Z$KaWn4=6SrYzdg?mD&0?KlD8K~u;4C%}n1>RqQ zMIj4@y2QRdhO{JM%iBSHeGjYash>Dic2DlH2}q@@%4OKmvr=_J zTGn+91M%TnC4O-F&JG^VK=fgP%FuB5JU^g-G(x&kNAb>$VkBOFOg)ni^E>93ctmk? zT1R^^OlVVUd=3?Cboy2FO$u_64r$#*vMCY;seo>ZtJ<8SG&+MSBdl!w4JrddToP07hDCracKK(r% zkdlw)N!`;?NgHg$ZAKyA`V^5l8fM%n>^D*9cQr-DTakoRAWJwgS%9t$Gd?s-oz8Wy zlV;9PgE#5rD3YCAkJ}qmiPExcdhmlNIM0PVYNh5Ji_4=Z&&-=_bv)#_vsR&m$tZ6n z2)|^A%n|;OUO!rsTT1V%=AE9mCIoFSm}ps*`qx`F%o|o?zC0u!aA|n8rEf^3A=7u2 z&7_JZg4#*WmbvemNGembaI74zgbey{MHsYq3rAAgs1|VG2}56t2LdCE5LTYfJ8L4YbDBv^L&SxUmvC zyt;{aKLQM)6Fladsi2x52BRfANLW(XBF*h`*^f7@`4A#9F@a5k+ykYXl4|}nd2K0T z!m5Ob!VtAz@d?G6k^d9G&4f9R3$HIL1PdR@4uwTYKG3Z-^p0*_Ia%NsHM*XbG$!kV{ zDbDpjKMU8ctyW~qAIZ!mJ73^d&b?tTz2M-Ts>c=f;>Fb#aO;Oyf^$V|LV2&=Q;}wY zaaXm4`7-`ZvGVu4J>dGW-+!pRDa-oU+dH!M?>}ZEv#RG*^umVo<%89%ZnkbmN@I6`bJvPMW3DwVKgyvQzE5CA`zA$(L^$EN0=S9c;7Vj7#=vJyUg>~nBuEJfrB%Q~sYE+}%R zDVcl6+(gU&6Whioi0x}1*|eO#m|#)6!xCz#S$1-ci;1uEj66Dt^$f>@MI+~Uyx72iB~q}X zME9}F?Hsnpfq?3DP|!?-ewEp{$t{6&HW{2ji0y_yNOTzT7V#cfu^h^ARsQurc2ho{ z3m*QI(|d=zajo=8Ed*^gkrs&U`tC z+WSY~$$?go>URZDx&MoNxz@T<+48?0CGskB2;3{!tC?5Vb|)#wxU30CP2~1{d;AJu zV)3_8?`KARdn5`aAH5Y|{YkA|n}SI^ByQ89kJ8Uz!7Znj5+wSaXT67gDlc4`M{nz9vtPKIo+-QMk-S)PX8Y&!+QwoHJ%?r(%P;(!S!H#;)GW}o4 zD+bz6Z}#5#uRVvH{}{jt9weSZ0Vg zEYQw1DjCnS&84fDobQ_Y&oC%p;WOsCjr7wG65@}>)9-B8eGFV4+HAVK)7KJadamX} zHM}#I!J3BuT9M#wGMDFEW;~&jt73;t^pr_9y&0BNCF-uq*KrGFV(uw=J-jNlsh$=9 zyXrqf>tH3p_{;L7A&EI#0zN|6W?z#`*sRj!rWYDs60VG61*@Q zv0;~nkdztrXPkL<GgNxLfse3N8kVedrG8CKbpT8(KxlKa=#fRukb1#LHv`EO!6=HYe8r<^X z32`hJ<463!WRd@IrB-Q4rC+7{ekrW%Me}Qd{g;5U+~AYr-C9dUfj6a<3+;L8?TP~F zgyVTE)m?TeF51aLIArj2VtBYUyTplJy`Uu_jQ|eoR<_s9a7+;+&iD6BxF%ZViTOpt zTNdIa-DMZ3DPwj_Xo(ber7M?P>Ib65;WpgC@M!3*B~1s+Ehx+nk!y;npn12f{>1we z{LlKn&R7g_|L|OFq2^GCqF@SZgj#W(c6nr7+BO1qLSD%_TO`|^eD%Rt+!OJJ4RFBz z@vM>ur+#(ecl`s;|J14q!K>*jrI$=5G@Za)8m>&x(w4zj*`snT-hgZAD4qH|4Sg3xrCHnTwjXoU z+?BVcpWzaU*>^=@A+tNz_J$4o`znvH5W?Es?m@edH(Bgy`hY3+)bf>_xMIkY2ZB`oliR9z^L{W1jZyZE;C! z?TXE+V{e;8)NzJyUt~$>)YeVw%AL*FeCZ!w_o;!c4+fv^b>Lfb0 z6ue^D7+@}P&v5jYW(e<8h(K07J@z#%74QW{9yce95g+)1Aw!WFb@3a>l*7-jf523p zMnH7C>5G5|sz2dtJ8BoBs&Tq@zv`@vlyYP^@ic#7(sI~RWujdhR5m~Oi3w*uqG#lr z9P{K9)4thk298Exs;|CE{;iRt5*(jF9W2Q?yT!hLtGHG3je=gI(D0N&vmYhy3T-{a zji5%K0Aoz9vZIEFf%IXyQLEZf8zitBijw=JidfF^{2YVK^s{EIt{H=Sd9cf$9 z)k66}4Co2$Z~RdO`bFXgmpzFFD3{%)JRms@G-J)Wvyu`=R-E&TsQ+X;=hXiuq@P&~ zft<;g{EG1F%IP6swli~yW?SOq4`rn3_4I0IL;O`mTmqUZw%7d>AKjB~y;LL>q(;zi zH?x-gUemC`FE5A6cxlB=dL366P#W3;^i?*#Vq zF_XunKynBzborkOAwx;m(~K8j%}=Vw0$b~b*8NdQa)S3q`ctc`i|Wpx{b>9;HY64k z9N+Xj{q>ee%AOmFvR9s!<6?jy2k&KGMpsb52 zfsIPXm_HqL%6BNYSA&!#Lkgj^zcbKM36uN}e-$`UqJKrhZIi2jSmE3^zlB^rzaJH; z{jxIDuDPvYE#T-`_RRd(TNo!vS)=Fb8^);Tgdg5nmYO=|t}v0lgZg@;1`Ok2ufmwl zCi`m2UNu<4>HqNTOMq(E#+|1GLS#eV8@Bn-(ZUlil4j06t62rtZdaY+vGil9eZV88 zwgntFV0LtD;l@WpqT<>0(xOTt+AbA*V(IW8l@56`op-{#5OKijOHOchnBeUXQv%IZ zJ;nX$>_;XhG{Fu4_Ft$kBR&XUrcIpn`Ha;o$a7| zBt||OHGUrbWnC?kLbPBp+XuWRP0z=v9oHUUT1k=b;Jwc=v&FW2Y2 zC&CPNUG22oa;Y7MGDaeN8klc&%vJ4o)}xl>f-6A`$0!x@Imuk%SI1 z&22xlQL>nLn1cbOWZK9suiGAnNKSh2m*OzRD}tb|Q+S91I@JOZrY0SR>ESOc5;jRx z#P#bvU&AoYRAGzvC6m#6HhQlW>Fdy>&z@Ijll5(%&>K!pj4fT+)e_llVBTGug%AS>Q=H*7pX|V&#AWTfA8csbjDj(aK!o%9sZCgfjPSndXJo zi#^1fBH+)hsYv!S(_{3EL49FG6{*98^#Ppr)5hwmAO?~SEkdAh@9<6od@+eIWAdwSCG`cVdl^U?G4zebXMv9M3A~R?@7Y~h?K`e*;++CvfF9)E z)jlt*j-u_#fRL6#KqJZiv}^{_?|lPoKOiCY?~-n{_peT#YnR$$GI_;eb zNw-KFj${Js&W+dO6SlM?etGZuKVmLJ{y|AI}OTF+7hnnNF!Zim%K1zC^zL*fy&4@&^-~-_8EP9I9b%W-nTUiu1VeTQuUW zbfy$=|3AT88@4$>E#)2?j2c6?90>3%DJ^X` z;~1PBzOlmRd?Nq+mopxCZ8^w#;fmkoO(|Kex{Bm-TY6la$YBA20_!NsmT$do83ut? zjZQ0ll)iNTKt6XLC4C;o)6Hi<9jo-o3&z!g@M7viSljhmE; z`P=|cvy^{YP4YU>p&;{CMnW^@Cm;W)vavY5GL4*pagmcrdQ;7P{mkRCz|1M)_PK`2 z+H^~SFNY2Rd1)#Mw@wrd=dst72Xo{bn$UBQuCA-qKEFUK=7~4H$v0x?G6y(2D!^Qr znu-0A-!Zv|(E!Zpb?5c@@b5h7X0wNSq@^*{w{VGT+sas(UNPlF_#~i$vNmh+&6

|Q0re~-5KKZ?%ApXvR7<8?aSoa0C-6rodYI)`y8qufj# zN35k)%FX7cl9(B%7{gdmZo@%sC%&lM$1Ln7%p`Y7l5MtZ?lvJeV-vG%c7C7Vf3U~) z+2{Rwzpm?gUf7hgd3cT6<^i+WZ%pp$qYF9nhQFGj)5Gh5P{)B_r$woqr_91LS8VmXR;91FJSCja z!DV^2=60?3sMYV$fQjYWC#;5shwVW4RPKdl3mpLZs9S)Sxe#{DnHRiAXB?eXo0_#z zaO<5X@z-dti-3^nXQl_C+E0K>HRDq_=ZBNW7T^ij`@Ic8;A)%A#t-EYGY~0z3^4TptpJq!S0G$HzRVDP;uUQpT&5G6S1HU0S3a4VT=>+ zue%@N9(j*MW#2LJeg!?SqiS*XPPt;%RrWu~5iS|Ob&8N?YXK^0)-9?HjV;tlTcYhH zyHqUb#y{GF=nlews~*$u1j4m_zVkkPtZwuhTwvKrt2%NuQR+WPI4&+?I#~lw9k9%^ zULLkwdIJtu#ZARwC(@Iw6u`Ah^JnX=VMzy+JLRjF24aJN760(O?}1fGe1|FVW6#oj zTx5MvODi!U)HG#LUWA!>O*Gz)hjW>BV&|;@wf0Xx)o%*JG6mwV_R_9zNFh0gj6UfW z?4%66a@MRAYx{hgTdbk{`F7g3&3aCMj{0<2xWT4zkAC=&%0ZC}EaZEqSO9jorb59p zzyAoliVkxxU4(*CVdt*vPbz1h&xK#Xm0*Pos)oH730`S*K{ z=s?*ZU?k3g=*s@5K0wVCTfgN4vT8P}0^v9R$NWB5ED9`PW;p(hOEoB(p{1HZ!6(sp zhI6nN^_3FoR=qZ{GwDDWVHa@e$=un!0itJAd=5)?a_|P4a6lTc{8L^WD3LAbN3a^v zi6K|+pYI^MV(t5!MoVRXL|WgXV#lnzHfG-%#=cdX$gY*D%yfSew^$ouhV`758zr;!!t8ojI3R$_o#mpvuB9R+f5=ThHcteKwTVqiuM zgIVaZ*b$B=MGFZ7#y;&s5w{Xh5z%R+0Kd2$AXxM|fCsRF?t8P9BDLtA*t42opdo{PN3YK9wKd|qJqD*yv4FA++BaT~aN-Q|>@utXnF1h9n>s}?n zS~FMP??hP*FW(v(iVhTXc1-NCc08LXUGDPMRci_j%d>@A%l< zNkB;a9XWnY)=)zv zsY(lCu$a8t_S38VCveHL65DuNUeoZclBJ;~Og$AjeY==!;zb-6z}y>DsHutVs$)Y; zJJ}S#_5yBuG;5=dM03I73!>7EC;7s;T_`YgxkT(a>qMu-p+X=}tKfiCp2-Rk9HqN9 z=2B#oR8`bezo)c73>wc90}TpZLt}~v!Ta8e(&_r=B4MH%{He{R)%-Y8c)NksoO1qE zgRmA6aE{MP9H92;&=P-&dWkE&BGb#jhf}Hsq${1*sIu*)>EwEfN$V^A(<+Xshv&tX z;MWZ)uw+=E1bK_~s%sCt6#x~|_iTxKxUH4N0cRO}UyRsyIn6CcfVq9GrDWp$6LS6GTu*F@2gY8b2wFQ&`AK2-mm;M` zI_3{f-GwRUxQy2I~rf3f*71-(@uqKGQN+Ys2#3KjmBFldU>9f&~liBX7 ziOHnrQ<{O|@g@MWxna`MQ}+MXMeDm8J4AOaapR#&7WqjexriQ zu?bL-Rd!+=R;PQDMqjx0lw8O=4MhCnMnl9*+X?ZYT zaj=F5=3n*K_yOhyA?O2t-8h`ME6)z{wbc9V!0P2P=ImX1WQSs4tM(5~-0eK>f+v43(p$}DaKb<6iZ(Mlg8YPFpCAyg! zTXt)82$r*f!~Pp!o|D2=NA=;kK|HL*Ku8B7+&=y*eIsIU4>YP8m+Qmf!-tD}i8f6{ z*Jh6+J}*YeCU+^Mhq9xdio}{o=w=kl#?HJE@3Xj$7E$P*x$CfeWc$H&}ZQX=8|?w85txc>{5FX zvW**k&|1*b+jY{d7Ir7zgh#d+sbq33*DT(Vb&@zyVJ*SsO<>g?zwd{Q&nQ{}Hu9>| zJ`aGYsB98#YF*2H8zV?!hzza0hD7WL?0~-TPqsj#f8s>KA32R})8*}`;>#Y3va!!D z=m7X+`;Px7h#4q4dnXR44^Wa>2EJ#Fo{M)q%uSCts#)I^;{I}(#KYMXi@iFUeFf||R ze=gGOCnVxNX>g=U=wd7!!B9Uqk94{#ew7_PSF-SSWA~HEqTDn@MNeLZqVz&$qTM1x ze|q4NTvAEv;{VYHdA9h07klD+3!$XYW1w6Hbrx)K`eEkdR96zXNjlRi#)PWeq_gRu z$Z!G3u3OjMgZI>^Mh*%Se;ij^P~Fgp@iT)1bmr^KV_t0WX$femyL~iJN>(}lx5XJT zsMZ2ItIt#VA|D2r46CfA@x~g8%QnsAXvy6TsT+iH!^U<6S`~=P5u*Vkaac@LU*!@f zq$vwJ^uc94*5L9|HT)j1mnmvfiz5InT!q>Y)OT4y{@5^+V5g!Ky)A`<~KBo0fA8cUvOv zSG!`X;g!9ZAd0||V4J56rpDU-x8-_<$@){UrfFG|V!L?2;k-0)_#x9ytwZNfK^G$# zR^+>pWy6?NC*5TaE_JYx*7gCTGr%Ycy8G%q_P5|s&Xy3EtRR5LI^-V~VAI$#yavk( zTx@mYVLeP89i}SUG1~7XFB+S;*dJ-czb9@$7gYn0p{1FU*zC&Jnbr`QbLT{c7qQ`M z5>Tr4jFMh(-Xj#j%G1kMj?G~mnbWGAcNP#418Op^|H=+SgG^44#E$1VevMVqkY#rey zuX->NUEA-kzjA;aO?L2OOX{y3DY38+X4pW3HzhBSKGsBj6d`03SJf>93c6Z5h^#$~ z((_!JhI4uo?m5#TD2QyAZexL)Qgd%nua)#(LMn@>T>T*vpD#>Wv(nFaghyQ2ED$dJ zDosme8cX{pa{&r4c2Z+Gt;FdQ{c{q3(|%$kD+ zEpyqAE*+wmO<)%|yWaOD6d^2%2`!jH7lWc}VpOg-rf zgF&qYx!vCZzF+x;cgJvd`@7h2`Z=`VC~PnWgW1;|8yB|nR9>f#q3ZpHkz>v|o%#Wu z0r^!%>Ixj0>MpT-%>@o~{GXf~%D@a8%_X66MVS?DGCioey=`UVH__L-Vs;{H< zjok&}9YE%UN^ffgdU|y^9>~XN%|Aa{xEDgrTlbE^&B~k@pIVY0%2GSKh=iSsz=W=8 ze3^%#P?#w5v?{5J)KdStCA?h#yJ3N#4L@pvZ5GF2qGByCxndGE8BeR;(Y+pRZIwlx zm05R1ZvD#orP&@*THT*mjmtOII%_(SPHnq_fvBC_0_ekJs{e^yMM~T6toPMG%2tVX za63|8W@GmU@GA;|q^GlJT7G|u@RI^q)@ZF+7#|opD~4+-5VKvjdj!3~m7mX^v_w1D z7U6-*BdTqteg$yuxc&S+>ySpIWwBFkdbs+Jep!^S;$6wlucegE$UPX@C~w@L#3S|+ zOIly^E5}3$ZVj{#9lV{jziG7M!^1sjjDCTu-r8J!Fy}Y_XBkKQ(15H?2}64s=h{1*YjO!eFXvNb1>nv;i^j=MBrRX%l+!bHpT=8jezjljgg<0$Q^bkd)AZZwDd3H{KSK3a z<~<@>D%w_r{qeWWQ>m%Ioa7_I&tNW#^Q%PB3loM&$&gsFO zKJX|CEgIDD&$bX1uXku3bwv`i$B?wy3`d8mfG}RtLD`O){k;Acg4;+lqy><=U{ho5 z_RPsTW=4f^q`d;~VpV{Y^^`mpa9H6zWr&1H-QH>%eGBkE2m)@+jh$rMtj*_Ol@^4A zs@lxsg_=Zy!m+y^Yy?xA&RQ+R@U}qiZx*u*@J{?XOFTiu{e(ewHGVD*b2{2d&%aPG z7e75vrm357Yzde+A~BZ3re4m$^d=-PbUETnTh@KqV21htP>MsAGvcOpfrUlsR_)X^ zqI6wl3P3ER%Q=zw0jthyt!RmS1Cz}(^3(kU3)$Zuo{SUcJJae)!4q>yj%lKgRkqEx z86jHPj&>q{04cYhOQ9y}x##=;`4jM>#ds_=o7Ec79!oHW+}ynh2kI@4APGk=q9f}7 zEh()dcHA&+sx;XVCL-*>+ct46Rq(1gsGabdu(Qx(l@yP8Ah-$`N?D$^WtM3d=&a_< z*+K5F#TD+N^lcv@dwmo~oRe7x)4Cexjm-J=RAr+6d*Rs}Eo~R0*2!$WtK8@4M7N8> z%cd#op51Z>2f$s^oba!{&rwF|{IvIw5XLMib)-eT8H}+8hk`qP`A+s&oQ53S-fC&;w5Ip6j95ip3K)p2<3wRE(fQA<( z1X35SqeFp~D}J#3dUpn#>%!EaPg9TPjDBdLN&&z2X|vpC69K4_Bds)G+7uZKNtP=( zb+=P4bDov-EO8_!{2Be02H{u(%oX`ANo6%HSD@i5J34_s{e(8=?5ns>_6UAipZzQ` z_cArL(-mm;9e(%%*@|rbwcvc*;uq@h5=Gm~2jBYe6&l8(xN;qjQUHQjl9hC9HOWz$ zU6GUZ5=THb@T`^{Jwgn^1OsEEwqAlqEe}`OP&7t|UZtzBijX{OO+7G;4sN`+O-p`H zwpJH#BH6)z2k=EsXj@U~Y$g;#C>ssasKbR>e79cxK1rpvu?nu-&Q<=z`}HLbZ2)oskG+$Ku_`m9@v6cH<)sa99crVSlX25*3ZU+H`mqTiat04?S~*7?Fv{Mf z)P!;Tt;p5Kn6J_3;iiI4#62n}7OGQ~mrfyv2o0>C5{~@3USV6*hJK+oHj)OHDMuA) zWc*zzcreG(GP-nc2Hs;*=@fgRE!Ri=fotZ24P64>MUern?4@OD7P=2ahsAwybKE;t z_eqk<3=~J+hy1AeU0Rj47HFtQ^3dyqY~g~**Vg^QY-3?{qaKE1OC%A!E#)t2vfw8k zZ>49}qd7U@_2O(QTL8CRE8>!qj^Y0dG(@ea>}2cdj%d@bMHfKiuUVCIvbd-=>b}6w zP*kwf7kXiZ7!d#~t6=;?)}9aWL}=0O#m&BP5tHGf2LXFzH&JDxB}O7(uxY0;{SBvk zN-INGP#m-;tTs;1wq=b@G3{!aNGv&4Ck7-M^VS^bU&q7nU@q41%70q|6wQ+pyKux| zFQiAv)x^k4?Q|~Fvt}+2?e%tq&_vcY5U7*~jwgYKXe$DRpfkkn`xm16;^pKY5r+LHki+?A=-a&ErKHn@xSw^N`RvpxN8A%E=%kTSR*ULN)BT`VU|`~pRPH42WK_UvME;4u%a4J z-Bf16pp_8Yrpia2dAXG$+tx9px+n{ItC{fTq;pqL2&VcEs_d7cFwLsDlEa+3a^?1q zWmOj&8eLJ0fx`c`D820HE&0&Gr8uh6-{oBa08%3e7N$O!t$~d-=L5Dj(Hdlj0)BrD zwE}XoBXCRtCbq1zBJ6L!*(>x1t|oF+rSIUOEB_|4 zb-ligcZ6=00QO~2EsC%Yyqxa51)Hcp=BQKlL2?6Sn-vlqQaKmXA4|TgX1Fa=mjdMEfPXc}6g5(8+4(-WetUX~Q`->NQ@f(2e0IrY1fWbqs zAMVBd2+@;?tO!BsxR>M z{%{RZzjQ!9NoK<#5mnu+Et}7^dNU$diyeCOyEhb(ZjC(FP{mjMr0iO&(wZZEbtuyD z(NSMbuhRkkFgTUDVxD{(g_wPJ?Nw196;c4)!`{*bi_^b|?sOAUOq%C7@dtI!lAl&EVgt?`-_N z2(MtVmb`ypcVgwO4)2!MT}*006O{^!b3;r|nYZ$H`EsXPhgEbn7tUGLgrHk(Ipo#E zgZx1%_4UeXvDb=VMeuQHV473nP91J+&GR~KS(ehd);>~uUNKyOzwnUhBHG*uSKO&o zq;%HCzL<*-NsTtNUHIva8Hw<7e;sJ-!HF01l%sw7TM`HQZNei|?^ZwkQoK*}q_<-* zOAXi!VHlUI`5xvSEq3YD1s|L08w`f@%ipM2xqx1Ei^mr{`LZ``Z|}{lxbR!e3`TQ% zHo30$B~9@j+dmNNB|GRB3nuG*Hs@(+o zewp4#j{2`cDBGY2^XCk-B2&(!yp{JGO9I7hcB0aBP&5o~=; zOk_G~>1uu75yaPV16cs=#T?`iYj{lqbZ!cYF%x>k>3rc@AAaAxT~c=!wGR44+m_gm zGB>DKM2XD1H-oxLAPT1{v^|aT)*w65y;VA}^t8r9>~zHtK`{5D=F3f5xxc2aOA2VE z!YMS3nCPi?BAOQVHvo%bwE9OGArm0NDzC61#^}QAWFa&PV_;NRY(9Gwx_RwZMBAQ;awU4t>tO={pVu8Hs)H%72eXTOSG=->%4>R0igp7za{S zx%WZKhuOdgXvmHpY?wEo)KL>J&djX4<}7JC)z{Cos)w60X9NzX1s%`7MGW7L^p>vo zpzz>wkC(u!%<(e%x5P4B*ano0J%GH@A|0rF1~!EIq z&8=tsUfx1pAE;kwc};7kr^CTq!fkrR7Mvua#VKi&lb_hEvbQm28B zd8Ptav?bmCaTeC`RS%G;f1+|mE$E4>x~&f=G~5(!Qj5;^5p#FCba)jp{w&D%5B{C)+r7B^SNydlfFDzhTnrBLd^ zZ!)-r{s^8^0U`>BP(oC2xmnb&FE zEoFGClnb%|G8Txm7-T68o#DAxNFJF`ntNRcyMFByh+MzA@b1yRiub@O#zq5$!CaR= z-a01$s*q%}kA5Xi^yYo_YL8%y)cDV>-L;wX9nK&?1`+F74 zp?mmw+PkN*cCQqr=Z$|!(jQS>;4p)J2=oV#(2_j7+cy6&?*+5$N=Etwvb~ifq3r@m zKy|{!WgRkY3QC`vX}%Wtuj(3FG?6uafQLVemfeq*=sKNrYm)s!t}$&52te7-R1>vr z+t8KpN)!S^ZNcYOKtrm|nl8Mnz1uJlrS)`aTwf7ak1o+80sF+5u$Xq87Xa_VWqkZh zOMPh^?v((E3%ZDW3aUHOlCv@1d$A>XerL)R+^y72%v7@A{TZ#jQgI;ONg3NUk?s|s z%T-xSd7R%^_T{eb0vme1-AJA%{bokfe6lg`dB*r^3!PK9REi{B;Ei4HWYHFOl zJrnuVvHz);@xzIIw00dFT#lq9KTH7KaiFWR< zGNStym!I);p{B+45s`?c*>}m2`@b6R!o$mP2J#Rdc;IJaL$p2BK8aH4r-*mu263W= z$Li|Cfs4p$>g`O6>P5?7SW$m(_H|={Zo!ceAMop)FfpKQ7<96Z$Rb*07VfW8dWp`v z!A>@3ccxiNKuffq*Iv4pP=$}}_ze388`zKJAk^J|oXU}r^;5VduO){t5p~R=!&zIc ziBRZ?0nV1np3U?O78?{BTB=N=Li9}9Cys6*@0 zMoFJSOXl#_MWppGEjK!<9BpLU$UDN09<%2e2L0Lk2@|d_I1stW8nXy%1jJKwiB-mq zm;d-yJnXa>753IxOMZLli{o+pWN!DtUMy(|t>$xEa%dhP+Pb%tr(I&Pu}-6b=NHGuje1rKNhkjsZ7cmlqU=?lOYC*v`&^$+F6<2Log=Z&|;DI zYEF=tlPPfeV7E{MLw`x6FU_2mB#qSBeEaEEGu5j(8C8D+2$C*3qmb5H@NUw|Y7*<^D5_7)EjOwOy7Nr$7*8j zg}9yFk-PnHu=6Zw)SQtvRZ$fj46Fi;4h{To-9D+fOhH_l>4bX~1l*ry9`~tW_d))* z@}2A`3m9R>z?`<+`ESb)vztW_U6rE@&}2}K;40NAW|RI*YKehNn>zFD?a446y;Nq) zDO29A`WDb>RczY7+0@re^dIU+AJY8?8K{TjbQtdiQ}&|THJO;rHbZ$uH40(5?upia zz+)ZE@A)abyoCea!TN@^hFH|LlD&K-`T`4$P!R2yxRK&1GZ{Zw+EbX;6`5=VE;<8T zZD8&)onpZP@1!4c&Ynn#+FmXQUMh=dc~bF6|Cd(?|GK>b*PSC}zN&>HbXM!<=Jp3I zA8W+T0Hy`x@&-{3(1Lz5PXhoy&KF+iDE&Fu+HPzyrk;7|kJK=4^YZe;dz};bm zpv@2b6XelAfwbe0%cWv*JNIxa2UBPfYP58&+4nRV2FoT+HBZ*cbx;de&|MgjruA_3Am5K22xW(Sl8|TUg$DcS z`CggkO+ZNBp DuVMY`l8Mq|UY)Ec6$fUqj^p8M{LyEj8z0+5pCcWes|R#51s=Tn zk&!`Sn>HG^9z1xwT~A^vhxVvd)89S1Z}^~Ok_?dTRenH?+c61>0}UsWI{ zzk_|gG`9mLI1IMN5PgMflht<4$yL{NcnPnpAxBE^@2nv_l5YN4jm^w2+xUSU-{v6u z3%W=FjziC+yeAG}#(S^{3kk?-;T6=Hg=T^e2g9g~g1sE94*V)_2ncQLcGM^Iz3vV^ zB&dlK=Vs~`e{PU=56YXYDk>m(I~szk5dz#p=9#8Ip}*j>>Hk)~<_+SB>SNz9BlY40 z=yTY=lew28%kZAErnfBP*Syuz11hK1G`HLfJcG-@Y=TIi3&O0SvzoyM7<-g8e#~or zMnc0;P7EE{3foil`r&8l}LT8G8} z?qz$1BSA)->K@k|!dvVoF$8$vwQD+?F~iF7H}=o}2rNTK$yQBZ;`7Za(+?${@Ix`# zKWfc-d~wzS17}ynytV#9>Gnk|CBv3JZpKkL?-VZSxv{uwxw~Bw{@`M(Z$It})}dYk zNUFy#0x847Ud-&MD+`9&;?yqQ*(tFT{nQ{!Z7KOf7Mp{u9sAK~skj#5GqSSlA=?HL_G%P>{IQpcTxfjs3YI zP(o9kykRhv=dvqsHwRs%WTxO@@ngo_vnJm@_f{hX^O$nj&2=e&GiIEB zx^8LxW6U$7G>ZCT(jSWZ;%wuoVSe`*tow4VmMy6+P=PoF60NMhK0+h9A zv*CYk#rO!1;5^IzvT`_&I900nKB;AhcOFum1@0d$k@v52XNMaa+UA8=D?h@k*?%gPp3L%}5~RcX4~0{WjOI;yr;1JO9%xn1&Mb!!pOK`yPj zm$cQbPW6W@o0TFeLI(nsDjsZ0pmu>rm4~u~H4}4oE!gqaSQA~y6~FHQ{n^!Wk#Qny z&j|PddSZ@cp`cA#r=z6#N^u*#jS{NMBSi8Yz3;sIJc6>nNHQIrm^lO9uT_?iy zA|CU>d?Xa_^=JF5CIW>8PCW`ju4v zv<4@3Z5d|5F?Yyr6gRTZeBo7NbYnBK9d?UrSt1cd}oM2iYVnATz+b!WNA6r$aF7+10iINkq12_fRAXa z1*dopt^l0g#hu!U4vNLv&>833nQb|eGC%_1s!z^4?~bGn{;aq)F&}RdxCejMJZ+f- z<6r}|6ol^hvF`G^@o;6t%@mK}@fDoW%z@eF<~&hz=8VtkOga(EAT1Rza#S97bLC^k{`zsWYLAV_nLci9$ zYJFhjD-dj4b=FQA3ec5)F*1}r_ISAdP~ABN*q%S8#nlGd*3>t7Xk}T3 zFV1B4UnL3$AAZT-*yu6b9swVS(yfh+;ru?{t(mXTQ(5MIj#cNvri|5^v{JPvzg98Z z)vmV|$NCj-|0yjk8uOAcc=80nsg^YX>) z6hwryy~TDNW4=sRbXA{GN{rfq!@N=1S(f!1HSvl&Wi5TRXfsTCtwbOyi<_o1drE(R z8#Z{S3Y-RJARvX4ti6{_$zH6-tZC`HMn%@X5t#6gzuR~mA6!D&omK;_NBReReTA6U z*et6O_|^Kn|Ipyy5o|*oICCf$7)(_zq|3vOKS`TpWVm$l-{8P(=MXga?U={7=H| zANG*zh2yyM#y`&y!Z`(ADPAOxl7wB$mm|y@l(r>}wIhMC07$x5>4g;s6!NH;nt*l5}z!ALb8HepIM$Pg-AhO>3 zbtF*Fy&#&+(4XjJX)s=Axv`|n4{RhJzlLHQ9i9uu#ssJnzJ&0TFajQ6GCIaGh8$cE zk@mRiw0+Hyg>IX1Ho$=ZG;prCG6Own06Hpm8h&gWU`A~pHfQAzUg9s7_&Y^ltrwS@ zY%?m6P9MuxU`Jt_&qGk?(23hrc2^$$caEc-w5+p`pAB}-fS5}sfPSYF^YL%(sf7qd zS{Fd7N431;VkjU0j^P@0q-lh?IB1z06;&Nj_lhynQVj2n|B+>;elM%)YK=Ugs4h0| zlJgJ=u9NSu7?(FOg%4Gfy1yc@CX801JLr15|3%H?M{)$t7VAW#3pjl~FlDJsnOv3; zuJsH@foNWmaA!I|bzf*LvY{p;_hRHv4T%6=3mg&LUk?>MS6rY%ufF*}HjuwDQly_sg|hM>o5N(~4m8%%FCe-lIrL z9cIyNIEi14Vmz!PU2daN={FL{mN5uob_m9@X`~}A$X@6_O8!t9sUF(0Qmi!v?`b9P z2I`yaWnfm;Bw5RSNaa~Kp=f#QZ#5(5ee-En_o1`CM?^QTX?joxAi%Zvq-|+p*1whQ zkzAHoYRr8} zyv$aim-2;?YqH|d#tiC zcN|c9`MAtZw?_WaWW(M5x}cMu1(#cp7_S{A8i5V;;!1y?zNpHib;$`!(zUe7!F{7> zqrCvEO^mq$x+~}=_5-#<|MQ*Qdhvl>4sn#|1x+LNF5!79a~R+Mdbw~T!KtpsBtKxZ z!?aCOwNo_PqR^}eP5MO3ha2f7Yyrp9H_|HaS+i2JIJ4H*FF;_&i!aSo z9ltT<3DkzE&JC-74-5Kh*wMi#Jr^69h{lq^l)S2mfhxqgXb@n6@XR$1XXbZfv`y)m zob?kPJM2Uywj7 zGJ-HC*n~xSA@l_@PV`JmN0wA|AoZKOA}d(rZ$35qC<3$OXjeDk8!)$S$7`Wte%>rL z2YOQgOlAZnM&`TZVS6rs!x4_=zj$yPAf(o zie;V)ZX*R z4h*;iil3mnN(has@KZQ@=g*$c2WhL-{R1h=gC-uvmuSP(8CY*<`TyR?+P+{LVnqr4 z#yODRxJ*z85QCWL3e3C{>K$*^R1p}ex2t7){+yWa!;P>Gp9LDD3;jp|nAAMesv^fb zmIb4mS&MIse3kZ}7bdh=N1!!f5Sw8M*yC`24I=b!-+%?!OO#+HV>>OCNi@=jhEr_P zD`=U!Ru?>8eWu>&X(_Xe8M@Fv@uTWWE2nC74X z+#eKaqQT7e(Ci%+7ETA5XNaGS_aGIiANf5=0FBqd-}(x(WXVBc4oMtd)bn~z7u@o! zMbfJBOz1pPd?KR(XFj<`$&G$Z?H~KIA1)=`2g4Pv61+p5g5=31!8us8a^UxG&^Dku zPHS#Vbu zO>V}xb|~|XD0CHs>{J{3>jF7ft+Z(6NEy(*g#(lW2RIb`m<{&Pdxtp*&*6c1h_oe) zrOomQhr(6L!$Pe`j;7Mu9)wYGWHqk$4c*>wEhjIe2E@ozdD|A$pm^?>Ish~6#es7q zj?I=PDHn5O^|bA&z7i%q)XUjHTAHOgIaT^d?H~>8Y0tDPEbRjH^wJ%u!iQXlVSaZ6 z{#?_$Axdg_@-$v~aJs;qb&L3Y*lN~Sf0tu#3U^6;3|Ai&MUWBB%7DIiZ_JGvGSE#V zFG>Y^lP7a#eYmE1YtQZEE`f$nlg!@e<*-5rWLx<{S^uf+?K>5uD>8V_s<8e3I0R66 zdTL$@Kf@Ey)%`D7MU`>r8wse*=Xa_RXJeutGuoLPXKHj2o%t!G_P{|3L$$ zN^2T`WHEj`Aitd>Fm#ALw^ay zrK##)Rew@ns|RwslMZuN4Q+d75ubvSH{=c(-j|rIC4!0foY^ez61eqypDA?c(qwYv zX=yI7B10fu>v5&xTSZZT^9+p^EO5`eYgTD#aN^)aBRBvEX=X8}=OqBU%d4b*g@xNK(KF_iE~RsNk( zn|U?9a~lcJ?H`{z0ZGRJN#~h<6JoZ;u^r&?1}S~LBj9%hA3dq*(xrB8&4lNFT%!R~ zc$as*ASY`xb>en{MYBz7LQy4yr92scK)rHxV#eBsOFbZ3-2)DOQX^8C>aZ+AV(lRv zkbHxRqe5!^`YLb^olz*poq(@$VqO5!?WHB+g+K5GohgeF;Ifqq$c7llA=Ok<3T~E8%@EUF}!9cnXa%b2NS2u*n z&m#g`N`(c|4RgKyb|Lgudv27-I^7b#ZK^%Pe)erYpRYn}a!yTIMxO)c0k>({<=Ma#;92mKRm z0hTak%bU2Q3}~)CvtTSws%#;a=lPEzi;%_eB#I6Ng5d8k_MJm5Pa8BYlOx zXw$j-uA&T@b*{OcyBH3_8;dn>_IfZ|{Ot`e+Djh0J^oC1Y8jjzYfic$PZ##L`p&j) zS!>_fZAF{P)khoea|Bf?wsjbp@v-iQW&2=txCB48sK7^=*fy|6r(rCQ57yCWj1lYs zUEzBrZ~%^W#%_PS_YUDF-~#l0*-CAJw<;tUDpI;OQi~w%34mww?)=4-V~vFqu6cOp z8?o{<8t-}{VmabpsUKXsFR5{p?yd~X46vq3%^!tFtlmhh|C*%B;IGy?YvXuJ@{b@R zSi;CM-cz@@{&-@*{0t17b)gZRaC-LY{V(+9$RlgFX4QIQih@pCRHdS`vxV`f=xvKz z=km`WV0ZDD)84Q|zVVd$Bv1ieGYH1qwvb%>34v`Q?z-+H@DVFvb%%`#CjZ;A8&}X( zNSk=5@>QOJ-*|yqZPOpoh|D6%5`M zUQZr(%byN6T}76r5Y>mJD+;tHg9peYln(oT6hw!shhRttO4Ic&k(6e@)Oanw_Y&}* zGXR0=F#gw(t*zkHI34fH`KMM?Fwthf`dh@g3alurcxUHE^xk>IgB9fX=)mNd#hF)} z2#>bu`?8-0pHrI#jjD7ibMd5FObF`GKmLruhAnX+-CwQon$jXS1y*V{>&{+Ha083rv$^goV}iT+WGq>LaPYEl&P!Y5c%6AvE=pp()D&5G-w zQAP|(A^=uTs=Z_We-xdKKa>0a$2+I{BS|-$h;>L#F-~Qan>pR0otbjbwopmTj8pE0 zvC|zPoH)&C4$JKlv#^`YaNH*e8yj10HbrhWo10n8zQ61DAMC;QxjxtX{dzs08Z*um z?TBL*CF;mtB(a^Rdv1y^gC+u6B=Pl)H*m<~$3TL3sUw3%@Mwe@zU_ya5o{Pjrpe{D zD@mR>kLkMg9~B!+$`c~f8TJw*N@CWEJmRBp6`#bAl`n(vitFO@Qy$i=%oTa6l$aW5 zlx#Hv($gR&Z1;^hJdVL|J&I?^kM4z%Saqk$G^a*qc*HxNb>KbW5Nqz+yK^Dm0PffvK zZV!+lgWx7XL-+zDZ2^2_(0YMJK5g#4JU1T6R7OyghV7!j$4zGVC?BReQ-bXc=(=9V zV`k4M1xSq(cQSFNqLFBSPoD67J*%GhrXwmo%Go%g$fq75Gr6pHTAcw^gU&?7 z0bz&RwHhwmFo4O;`Z)R9p!3=4Sy3!A&VYL0!%obd*PT}SDs(65i%9!kr+#t}^~dhb z6PNNPY&Xw{gdNRk4so;zn9Knsi1~+bF=SiOQ5seKG0_CNv#ui_|zU6^{%mD?bPlV)H67$@`SsvS<|}*uR+ndLi9{N<0pv{3EUwoe84E zepxDivu5j~6w2VQ2{-u~o5Dl9=*-ytisfgsQy-HIOi+^>PF?^gC8BUmhP=hU_{mE5 zDeANS^>@PLBu07@;)CnXCSz?Mst}kVs<%54TZt0lNx^iRV_sUg^)l^NCy?}>*R_1& z?UC{`Jv=-FV_7(YeEPH(Ym6xGDYpY8G1WcAJ%Syl>qU8Pu55wPE9wn1O))qW7Es`77!UATv+!QU_1? zn|ty3=Y%#ZRx?mBEpS#;_Bm2_l$3GgFGNWEwqXFogER7}Z5GcgJuo*J@(1oM?m>!QR zYuCHX+o84=%tUz50SE&HN^DmB@d%#fJTwSlSa_*B9lQhr=DZ>*4I3VA&D4`Kz$GE`WVAQfcm~ zEG!8w4?`r;km@3?i|%L9-z68{z|ZZk$3ymKIfVe{7J4dvx(1C22`>Cn{H>>r6Vd`u zQ2j>$RWu++f#f4LovoS;yo2*J9=&Re&Y<11@~}4cD16sNg#}xQ#2rCpnAJe+bHGZl zIxirT7VfB(6G8>6rXe&dUO6a4P{9UXtFJ>dKPYG=q~ zB4uajnrt;?CDYrZo)?+i_ww5C@U_`{{vvT(5Uw1Jz`eeZCv;4|xb4N~S*yE=TLyY= zqAwH32x+0-^4LSEZ95S-hWnjeO<(A>yPyC@5ic@3qf8x= zbTcS~`sw_BPQ^|xVcKMWUiEalq`&WqdQ>vKmxs=re_Y%>L^qqQJPhbZs~ z1iZHgrjU&lR2yd;Jd)~#gSyL@m5E;%#*T{A4h5WR8X%7xZI9qvx0lf<{G5Hv8q0^fdFFxilLWr%5&7*GHI%7_;q=)5-3;uDjm zW={=^NBmo&4s_&EoEu)mtJGftL`^Q9(bGm#%O)>8EjKXl(OQ`2kkM`0om8s|NwTL6wYx&ho$pT3v)6P?&95 zu~lC*UU!J7-_6|mbge4$E}j;*yK=hMs8v|!KWlF|(6&*Kp#+$GM%ywb-zFR3sc3=L zW=8Y?Ym|ryFh>5QQzAb9>OAJv*Dtpp$b;f(9f{yU-`4}PsHbt*LtY|J)=QJ>$LqHE zlK+fSGO7@-Pz-Dvr{5&V&hVhW9!-^oWV_>ykb94od*nZb`>MTe}VyuH*8`? zf&KjTt`s0!8kf8SxP!auQnrm@$-*{qZ&Zf^Cn@_8)Ao&c963j>@_jZw!^-{k*G|ou zB!KFTMz_B~IIB2iwATM^DyvGSOxS|_)fPC7E5O3siHM)f*V`6m3aZO*`)e2<=r(9A zO})xt`Zc(vuMJe+>5nn2sb`4quy|duHGr$^b32Km3q1y0CKY}eTFbx|DOl%<62SAt zl0%+o&c;K!#1UqV4=?vG$3teo7VIV#tI1KFByyhUp6-JpwpsF{H|fwFAP}>&9C64B=Lp~8fUL_keaSKU7qFP zoZY!bN?Sig@aF~^VMR9il9Sq>2k6!)<7R!NxJ&wise4PH(+H0E|3Qf5{dvtW>1R6K zlK(kfbOvmMPWcMaOA&QFO|}UA-=-~1mg>Su4@GsINzM%W4sYtcpv_kyY6;S%q`v$c zNvldbG=ec}W9t9f9KU=lf_p%U_$~l5y{_ZWBAW!B z7gWjQcJ=h~bY06f5GMN~pYO$=3KhHO&YSi9Mz)GH!y_ZC>Dr-u1-Y-aX|EcnZIgHB zmVZswQ7ohwUgauefDR&7c`8L7+{dA1i%v zdL8O6pRA?$K9IV;ECcu)1)C*L?Z=0q7G%90*8nSlEGzd_7-y@B4aj@CH0A?yr+Q0f zPK|r!&iDpm3e1|hC&cl`Ss%vlG~U>K%10e3fk#O?^8QQI+eL8WG5S&yqMywgX;_Xi z!l#k6=%J)rZGgfx^LD{u^swsyeeV(e=R5=&hMZn1$>AU#nc4w*M2%rRP$H6TiC!?A zq3Ch9hHS7)L)U_})M#=@-VtAN(ir9L@r%??T_MLCdqzdS3g3KuFTTk?TEOY`>k8aO z_zojQh&KPKh*9N9H}3eGO5eA2aFX-N9a1NA1{i|3<4$N86;15oS^EhE&jS6s?0mhq zHY}BWcE-`v3vEDP5GILM9$;t0;9QY#L!^)ZkAIC+6R&G|zBs>m0dAuxNaeKeHbr#z ztnJzneXPf!gzJEgID5H|Ubo)+C4a$Rdr4`kAwc-LCfo zHle&m3jgCPdWzX3guD~T`7nPd1n07F)NqLN8_U@kAvC@EU-Ur^u$`2{;*_NxK=wUN$a7pp1Ycf*3qE$fW6oJ^kg_`3)-d9r<~6QMqUY zvBjJ9QkX&9RuV$BHu$JWDwjJQNyOQD@{^3(+7qUqtk~|;91++sv$q4FxBp0}@KrzZ zb)(qVoGfM{8-=ZRhJ5xYORC-`e*KX(cBja>=2Khwr%$xf9FGF3x~Iiem#bfcp&0*!ow~ z7fPEY$s>b+t$dBHwCzhbyN5&xuU1?4GE-HkQkW6uVLPvIbH2nuP zx?NPBFEN%mCV^m0IFNUS_xm%G(SRS-yM&|?in<1LO_Djim-p}J`YIG4S}}F zL#b<)oi3~$5ESX4;rz&AN~dpZ*qd2$`zq^$s9+)Gk#4ZdvN7!?x8N4K;T^rWft_M< zMpDn7)tY?L4g?+~z#5VifG5Y z)aB;g>IiH%1nwy1iT&3D^DqX_anz1GeL>D~_Etx7UR#PIaK)RHI=Fk&ZR0zkhk;xpJ9+Dx=_X&}-F6x}o!b1Y=rgVmTHm0a0s=}pAQ}cTKdRG!Qfw|apuTc87YsBfg9NSx544n4%P zy^*!%TPR3`)yq&iEb^JvPhJ11_k<=qBLAdNpw{2Q!iRX=5WD6Iwj;l7G*3BMWi6T^C@SYVTTpsY`4q(xmgSe!}_4lAhu~Y~HDv%0nX1^U?q`Vhh-@C%NzV^*+pjIgee>7=pYB z{Hz}#G%4Ss@hHBn&}S#rHFzHi;6Ft;* z17!&2kd&(eoJ|C{gp)3iTIm3haY)_L=iBv#@nG4Jw*s3D&*?VIBw|0{^2%!$_ik^$ ztB#c>`hO1qv^wM&cPsG^#-?;4^=A@d%Q-$(|J(FcL?4#lYQ9+CRF6NBA#*U3Yn^Pb z2$I3q)cp2@B?zBNp;fH(dwtA(_t3x!)5h@5MX?*vh> zSBQk#UsZm@JKwC%s`JY5Xle7)8$vGn6&7St<^3x;3D#khFef4;_>+g#sVJWG!PpnD zo%kQf*80dXhHva$Q57Z%V_T5$5_AKD0zH;H>q+QY+7rTR&6UZ2%&#O>jg2s$uQX+t zl$GiS&am?Kb(fE#Fsr@gVR;VTy-8(f@2VQa@dsYS0%^L(b1#8NogffMU-d|U!^F6H zv(!@dT`@lBzruqob=s@y8l{dTC8HS7TC0=qY_P*(HuB|Yta;>P?-z5flWDgnrf_}7hpmizY3G}$UBDU_}N zOAb%-lQhZF3%Z{NwrlvYw1FS75;|flE8x-$#u4Z<(s&*p_xDm2J<4Z=xEIgvBKv0C ze660o)XzNz5YWd*#(u;)HGjNeY>5sgqPTHDbee&o z4&BLeJ{zTadyxBNp|~+?eBFNFzh5*l(!fmZPl|01sp>h;T7^Tq?f#jZxiQc+l9qpU z{UX1`rZRY2Uit8>@RX?KWCygUbgXP-(5xQ&!n9`|j>9c*`sP;^3*JqG1QUfSNH1fB zSo7&q^$V)F?d6E*>mrk@`wHb=H&g4`kL;peIFc@R!oL07Sj#WE-AI3udO|9n}3Y6N@~uER!{Ov2NzZXw1x!;+-3$DI~(ymuk_R)U!s~ zyhpvcm18oUZs+23brh>=2;AcVem?cgrha6K6gVoUmU}7D!a_))sO~-$gGkfQ(MfdWN;OL%EyOqpp zl*f%l^ZHGbhMsEnc*Ko!YO%D$G; z1@jM(x*Nn>aI?@jM|5P(n8zWx_QF#IP-@iwq|TInb4E(ApcEhjr%hF?ODIq0akmC( zmP31$)zOuuSwdN%=|H5cDx#{SUnaxJ%Q%HrPFRa||Gr^c=h6Z${cL}90k#dgOD&#& z_5G;v19G>JM#Fhi1Lc0juYy(6gD2+%P5!DP@Dk>ymrFL`n)S*T0IZRU81uL%9M}{$ z_KP}m-R`TYp<=92qIaqzkNxZ#KGLlr)icL<;SG1Q4==iM9UJHGjlg*>9*c-Z(;Pfm zXpA{4N2FJK*65QuIYVjQu9yqltvNr@_k`1&#&FptY+dsX3?(P2ippkDEBrt_0I$FJ z7*Nt+jWOowB9TuGFVY4a4s`z9rGZdhAY6He_DY0ra98iFgyu^#@2sEI_9da=P0bu- znea*8)!Zxfe1?CR2Wm~nz6bqzH!1R@Mkod^8Xf;Ywl7js2I3|!0 z`ILK6nKCCaY~8q}QsYnTevT!^$CS53(W~iD%4=4QY3>$-J>$q3mVI8BpXd1)7v=|D z+!6Ra;UBBxySUqxxozXS!Q3;7bswb-#UX_N!M+M5Ul7Ujx0OXw$84x7u>Dle?|fTs zWp{E1wXB`};(^GoKEpFd2aCZ`D`SBN{*tMEp3Mr6@fuRi@DnhcV8ZX$&AFj`unzDKYF?KbR$4AoUZRhi3u>cWl6Hq$?ehP-k zd$0md;A7}PeB$Ni#8E>ee?^`7u|R^;2~cR!o{fUite(pax$Y_MGUzy=uIYtq&z2uV z>1~)pM7SE%9;lrGr_9=FW<`t4i~~vcyqqdX;uf$_`J>^U`2*1iuf4nsP38^|MAlt%7^x`xir*Q zH*)tgU`ZEx77GE&XNPMz49?(Tj(ZuC#Z2`Xu_gV;4Nm zuwZwUIN6S&tRIMg^gSgSLS%;L)X1QiYDfuiCAd9j;NWX$l#;E@JgR!uG>UO_E4@C3 zWn+^ZKk!--K?R3Sqlgck{o<6{JI78bFNclo8Dg-6a{hl!`a_J{ft{fDK_4zk$Bh_s7nevK5gjeSZE+;ne<6GTJDnYZ?wdjB{TT^uaiMg^ z4xl}ir&Ct1@>Yljs81$Q9&eikRF6nYZ9gYxhYbxifU`)ewgo6A&#UK4WOjW3|Kfnw zOugD(XwO?=DA-T9=B0Q_+gj!)4q@#x6vSbW{jV+#lwxJJ7Z|rwV8)@gw#AD!A^hqC z^4sY@-)>lG*3;N2;=0^5_L-A3`|%C~{m|sY{;qAPj=vhT9X!$EQ1n06%iU&KeQ==L zf@gQc-|GR3&k5PTpf3pzn;s}4_|t~@rwd|VE z%dn>V^{ifb-TRxu}B zRrJQZ)HrK-Kwod+$dGjm{mrM5(d)AY{r>MroZt_#Lw?!uQ&oluj|WLY``oiQd7-^f zx>3U2r)X#gddLtcbt;UEoiV&O>q`ds3u<2&Iv4&wGo<4>1y(flo&qhP7zpAu zS$tKn8slH~#so%|1JT_ciA9J)ZBp4S5V2DNl#=q$Cm&{V5 zyM-MuNgcuUnJ;|{!f8(as4i)0>sMQb-pKlYB;c_er$lsuM2INE4Eb`Pz7=|gR9PV+ zmiA#@*^~-n9lU$Axxng=Q=c`v71w(`&#h z`#iz#721vz=RG!?adT^`HaM z`3hgSexY{YC!uXLano>5sp|te&DOBZ)JCDHEy8VgivcVnva-SO8+^+AU6EPWp=Qjb zP$PUYX^2iZAl>-?o`Bsy3xt~$c*l?LZ|7Ypyrj^kVUu(lKNMQjGwK^o*3!zJv&uOL z@zI7~t*N0n8~JSx-BKN?80$j}5Zsh<*L~__7$psQ2)@^;vhp+p=WuOAGi*@IjV6?hxx;0lW( zH_GCVzAo=_ORHk`UNv?vE!25po`{@qH?JRr9NRl(^(aLyiu^(Cp{PX|Uk6a6oS*s8 z2px?MZ<3MK%k?e~L-{urmfDh5y&L?kt}IRk+TTg~t;Z!jpryRF3a%0`_{+A-nL2)o zg(pAd0b(q5$SeH!v*HdTgPgTXdBLS_bTvm7y`Xpf)6CjsNbll*JzFZak3@tpLaXPg zV9US^;*L!Ag9WZ1fan7THJpb0^GG{?gBV6_?$rW2 zxc+=}0~RsHAZJyC)`{H(z5`;&D5Gg(qECGlNR!-E%(ZfuWvqG$;>l|AF69Brlil@I zSVM~9AJ5gRZi%)9yZ(Z7d=XF%>HZ;DyPb1ksalw2^^c4*zDu)NjhrCrT?AAc0h>v% zk>$*&D{qZfRZq>`2AQvH?c#yovy|J2h80v`EGe%n7$nx4(QAlAm&T6w3nGHG`idgN zLs_=k@pBBw;1Ws1PF-oh8dumFb+su*+>-le#x8!oI%9NznfW2xrr-~{Em<|t25xh^ zX)(P0pNp=k+8ucBtLmU)lR#f(XN9EK8-{M)Gz=%wKQ%ul>FtJjiZ~-TkrA+U(qjaVPDA>&>I&7tFBIA&3XWUp1u>`B-ax; z_+)qDiUaXEuqjFa;AzM}C*39VUYc?_$y;(CG~?C{9?2Og%j%eZPRs9Rr{8&#{%*5~ z>w}Xb4AhZc!epqr64@}|>=a4IJ9M(WkfZrDgBc0(NaGuxY#G*YV!P_O@+4*EUR`JG7qGmU$;!oPQ; zek(&KI?o|9%9?%R(m+oH`FY}W)`DGuLz{N{6$StDU|{>AxbfpNX@*m_p}O|RsoJ3x z-BVtTS*j_g(Y0s;p|zlD_0&7yzWSyXuPGxF3k+2#Wx;9}1w=TP(I(pX5a8G}$hE8x zHI|X`U8d4pQs~%Mvjd}Dw^+wqmPIkN-b$(PL!qRo!H3b1BA?h?!B2KVjbq1vB(cVP z?M{IdWZIEfITO?pN1}Sw`1^`Qp8E6xve(Smuw9QHBzQG|K+)lJeZ`DzYga-XB;+_tzQgC?j!!lXz1HU@D*4n4A_&~S@jM4 zKzQ!&c3drcVFR6Uv1I@gmML#2wt zwAN}7xaoLRCO`~xBksE;6gIVMnH{9-ma?D?tyQS>0d!N9i z7rE|HS6@p)h@RK$K1P0@jLzMzXg;f=;&gAw6S zagq4j+T*zkoZojw)vm5CzOSLl`vqS3vV}Bf;Cgs~3s+#bh3UwowI@6uB|LRz8Y_1q~jz05U^yL3{(wEv6BD#|*7c50)&TA^C+o{Cv2>iVbpfF6 z{eYS!ey=rE6WtY4w4jN|J?boxe<0u-awcazuEfVz(dCkL2qSM`?r!g9n6>^0X3m1) zeb&zDRyu0ac)#|j`jixuPPkmSUHKPB9X~1C)0rwz8oh42{)w36CBI zP*+EQZ3vTlhqVX}ZhX5j8K3-aTzAEGFG7MaP3|*2tPrNi?OaazC{|O8>IXu2b`6;k zv+vj9A$oc(r$&)Uq*#iqFossT*tfOPP1Q~N{I9QR*-Ov?%XU`-_1bi2vQ>YR=@>he z#~^0+*Z*4ED>@qN;5z@V}a^p@)dn(MZ;CTq@K@g-m~!{WJpsn zk>|De_v~>N4;@B&FX-M)=6js~l?0CRR2*{D(zZX)*d_)i0{Q4Ycvp`HKSt+gUiS4z ziGRy~&&JasU{PivrChwx;I&A84`3+K>jz8gSDd~wMlY{+IG92xt>Xt7{*LPxmCkEp zsfQ9Ap7PyO1c?Peo5*`mR?d%JhgBjNBS~4)GJ~;*d=pK2^IVXZp1s0$eq%j|-~yB| zldaX1cEgjJ^E|>CDXgdj1H!&ts&>ARGn?TIHtVG_rL&4lY{%pm`+w-!f6slXs`9|l zxGuah47z`y2|Qbfw={r*XX(}5k7eZa^hqb=Y&W!dzj>KMeKueENN?I{ZkT9n6-%_FFP`9Dqja&H^> zKDdOn8#1e_xetUdNK@}c{+zoL@|(P9y*;7Y?;7o}jGp>Ilj=y);^MKObYnms z1Ts{{WKMT9o2cI}bfEO^YY(U!e&d$|9TGbOwy8Nlzlr`s#LaPI5>!>SmQ?Z3TM|5j))6hkpk|9gJdA8Ly4 z(!@~A6r=IQ%PbdHpZfG)DB)Nq;|5nII}h?&8@Fti%cH;DL^nJ+ zr4p%{A2yqEEod6(TLtkU)BrVOgh&g@zM@R(9c>H1tob+9m;1wmD#HfGV3FW2%_GsH z9By7Nk}3E?9;F90q$B@yhPfF{x5^v^N$1-QM3LFi!0S61D?>@@LOQ!YK z4@G<_JiYg}|9Kyh2f$%w<-gD2IR!7xL$h+FU7g0`AYHdMPS+Er>n=?&dFgUL1TkMf z#q5`m-}9!=HuHZ<5%G@kdxokbnA-*oj$WI;&0Z?c^+ejp2+qK&t1peX1!{y@Yff>t z`iv*n&$6-1!6zg<;jWEDT3lJbcHxyZPjGT4P7!(7J68QD5tb zpS-9>2T2jBi5iSlPp7Zqzouh~jg^O(pmwX5>?biMH_g$f8H0<(gG?YxiMB8N7ROxz z#NOc_zLSA=YV|ZH6!aF`hpc2PY_nb%Z`k&!k-z=gd)}O<>Ba24xI`dnDK^GfTLXKu zD@`QzFR&h1=$&;!?)gJwF^cs`y+;6Ri{H8y`DBH53dU;}T^vu%14Lmq5WTGNCFzn44xXHrS<4XP&Cq}k4{A2ALs8HJ3 ze&->j@0j%#qY+YI?>pV`N4Fi^EUDZl<8pm%hKxg_svfQSo)WtYe?j}t(lFK;n>V>( z6&p3Gq48mCe zy<3=}+}uvSR8b`w4!eS}2VO{=fnHuAt+=gpRQ{Gxo871Vz^Q9MH(+;ETT2IR;{3(G z4!~X5&n=5IcLfYMTC&A;mpuH=fU{fq*f@Inircf>S#s6R>fbkqNLDkM8*%Q$6qe}J zOx$YJL5PoHP=pQhh3d1HB&qW#&&uF<@WAvCu3Z!p z?pJbs zRt^u(xaTlDSgkt^Wf<_MSVNZ3;W%&8-GE=zC)R!N=I7Pa5aL7usLSE}`Rw2LIEKs; z!5oN@8FaqaNlYA{{~CTfP*!k!N(Tm5q`u_Xr^6a>HWt@bui~jb3d0OSAZY&EuUFL2iJ z-q?(H_bXs^&}Ty}pEa6)w|bn{b%eh`9(cKQI5%$dBq)tM-hsQ<)yUUPb2;GTA`O=9} ze_Ki9LmK3ZGMQPD^sjq{B=U@17c=LJZy;mTqEvx4fo^SjAaF0t`bhs41Ir}k=#RGq@TD6I_gTggmdeY=2HW&8nLfuALHA*Vl5xrhu{hev#e$*#? z+=YIEUL|mHQy<#VT;2Qu+*(XL*Xbz5qPJ95jRsIrVdRI`x>flFX4~#ZHUpZ%=4;OR z=sQ`FH+eXqxq3mt)K! z?z)O@);}g&u(R6!(}FiLaRiH(vm9JGQ(^_iq6dNDkOY)^tzKxed>=eo zon6#*iwzI57_I$S86T!FM;)O%IV{|KI2qWrkY@Dhl$tsbqrJa9)h>-CcB5hS)j2NP zuU@z|P;Gq_9&DM194K*n9^I`OHjZ-k>RcNA6SR2+k*+oGC)iH7rjh;SC6j&n7nORR z=_)a@p2Jm?%{%Nv&S)cEwSK7z^4>pS2<+dvzLin8OllT>K zV95-qwOjieEl3?Tw+114o6o*(WupoTvhtOV|J!u-jerihqdfE%B0=EYbM)bXv4ZcB z$E)68)^g8MyR82L&WO*i|HKi_C=qFl>>Um*VA#4rlAjypG7mDYqkd-1utH;OsRKtE zU{;ZOGpbeMjX@^(GJ4xMHi`P_HQ;u{ZJWc+H*X0gf=bm0sa0R5)-3RldaqnJ5>YGM z>(t<%=%t{wTNU-iZIngf;;AAuVpnXO;;i*wZc%2y;W0X1*4Wsdd)L%Tt)q16gzo2S z4--KA=pSz>7t&TLkJ%L%zE}D^F0UJ8f@%78S|kDWZ9%<4&CX1Y39 zWwt4Bw>q*d&!EXlG1e>vsjnyRS5LnFdW9I$Q65Gns;1a?ONVEBqK@%o;WTGgZIjEn zMdEU|jrFZ>CF}6AE{cfQ_7-YN9N?${J>qV^bmBK}KkawpXw8Kojd5_rtYcQ5B+|5_ zVm82(M_-`jx3Slf{{00R#yPG|&=C)%kIu{4XR*+dKLmO9QSV43lCm4;`l(sb{l86h z+c)%qLKAV~0Wj9d1pJ6YGg=VAFDVR2hWo|(K@X7Dk=pk|z)og&V+sECWT&?$`BJ$k znM~qDl*I)n;VeZFj1_G2va4C3%k8%~$G(=x6xVIs6U7<_p7Z(Q8O~~|t@)5G&>Y3N zx(j~d!i+P}#+EF84jho0HB``2Uff%=V+&8o8S7yiOwQuS569|C5ukklvKE0Iabb=R+V$L*#bB@4-adtKj1nffU0lRax5IZ!P{ zW7GEj>8#g6G;P`yit^dWd}`jdFN~Et!*-lbAJO(y&yve(n=&xcjGwQriQukRSuf~E z2Br0Bmh~ClZ!>RX$?w!J**+=UmD(ZlO{Hu0DqPs9)+nDks}d|)K?dqpG}>8QK|n39%4@B2|WuY2jyc)X4`VWknjo8PF!y}1*q-f%DUoeTmN*_jqA)K z9(3MWrn#i16k(don0YV&hX{7NFlaZoLA)q5AId#HxV*5N_|8@qUTLix6rJt*{M8QB z8HmC9wt3L@uRP#tTmyuXO^p?`VZKe)irkBpvwJZ1l&^8VOYaqdBO@4cFRN}HppksQ zKzrlkr-#2K#4Ic0{q)twgtG!j^Zqc?t?KN!L@CYBQ`tM7I{rI0Q8&>m#iW{CMtO1S zf+{(IQ8V1K#q<(eb>Kb&uMy8-$8RW_foDWY;V!NadLXoEKj}T<2h1O$q#R(pPcSE9 zclGyGT`WQ3Q~dZ5xbUGH4#GT91FFY$qrLhY4Gy?hfpiw=D}AXB3F9dm z&&7Z+C1RsQ($JxNUS&TJw&qgSgW)`$yhx~8Gok0--FIrE`L`_MF~--R&H-oKR1{v*>W)q`#{C)hc6XP@N z%0&6t(Sas?ENE|Z_Et5ljl&c^=DPt6*@;qxU#g6u4v(ykODju+c)&umWJoX?3G6| zC0|Lf)6fpdPDn-ulMnbGwF1S)XsO>{=W(x3x!oft^_KOG(4GsreYL6Gz_Sj+fjz2hrbo4kTV5&SLcaH{ZgT z9B>^zuOl)Ov~@vi7}~sxyR(GyWDSw^u?Sia{BorFei7CvCBHo2>>=+FH{4C<>;nrU zFS9)j7Xz#fMAsECm-hUSO>3=l2sOg?QFOM6=CV?^4>%8Zg&KVX!Y1}7!^F{}NGcEF zPF3z0q8Yf$iHwr^NOqw3*E(^aHIH2oH|0S7O4;M=Q>pn?@l4ocfC4>1(B?QtDlMx2 zS-UMrNUk{0`?R|n`O6(fKQerI8X4q^p8K2hvkiS7`XL!Npfws++Yq0qRL{cLW2Qo( zr$S?DCgh$kKTjvNIuPwdvD{jex%J#<49l7Yw4(!jQdPK(Pn27vgEXr-SCL+;=@A8s zHsSiz^q>N;SddlKd45gDc>4kvwagka4&-&`Sp;=(`3r`j;toZk*MV#xlUNFK?ypNK z`_&oS6&tWmakOG0AMVzv#P7Sd=i1jmeY^Ecl=e%s6@3pRtD&OXWQrE|dV~bXq$$p6Ab;XO_=Cz96bj1B1nqvV8u2Y0!mT6OcAM3LbGg*`$ZkQ z@H_KuMoS*xwm3>D`>zf%B>Q_Ui3yOASnG%kFYeNjB_Qx@MSU2{t`$uzLw{)$r?kM#@?^wwm(ghEQuJ2A9&GeY_ z@KFqeqCXb^5(1XC7#$UB5VxXh4&!mp#BY@8sYVx`MB0(kL#HU`>z8m$Ov&3^G)d=9 z;RV*?m>=c79xLJSNSUK+ruN{lTG|7!8q=Ugm%1e#OV(iKT(3iGT?;WP=+&$*+@gSy zDWOt&898!3JftBj^5n%r_vi-V4z3fRxexf6CtC=w(i|(ABR6uBcPioXG6Tx10JJ(L zB)3h7ipZAgS_BW;l+*}+;_rQ1bD+ub_i^e-eD`D{jQOz&{iax@OriumPAFV?$oA$Y;>B-l^<9`^@C+!z%4k=XwMY_8TNxeB7N zhO8$>l8IKSCwwuprpCc0E7}u+SUJtx#Sc6ILXsdYT?d{D&<{l9hKG`^Q zFOUGSROPs5V80RL(f}Gg-dQIZOp~@94D5^D$_XOG#5n#AE-&wkkB|7yKar2chlgGd(?>bwFk2)==EBB<|6T!#>S=Zk}K> znMTv~tQD-1isCLmE3M-aW68N+F5J17A@wx59k>K8+@pYYTV;XI3e;z+RWaQg?~?(< zQNa_MqfPnTRl&*_wcKDl|I>128CPc9oWZ-AkMP{ITMkaaH(;V`#}eB900gPsYUd^% zA!Pj0mrr-)#Z@7afU};q(?Cu9W9Tgb<9<`F=bqbi`f1OWct8$2B<^>7`2Jo}$I{Z} zmo2T&zZ(Ld!T%_K#E)z-hic9v4ksRe(E-^Y`V*;G4RR{JRRJ#VBk2vz>5f}bFo zod~f<#sBK4FaI5M%-LCG)PQO(T7N3)Ri?|Yh(C*Z|4%ChrdRQgQLOxrIu7 ziZ{=F)H%=Kqc)a7{Qgwd3#yFeagap=seYcu=ub~nv>jc|**_|pK^$K_E_Y^oG#>~M_Owjr90a~|l@s-qiX>Uiw%&qt z+1r*#T6cH)G~{^7mE{rmfgfPVxFb(}BAWdG3W_ox-t%S6z}A^TVgJvNEJcXyIydZ+ z*K$&^!P2br-N5?4$z(lkH943}X{CG;kWH3a_gA&~Fj#PO%Yg8Fe^sw0Wi2ykv2W*4 z-#)@Y@L{#1+(-=~q}Y#?;T*aIfk)-z%Smb%&D_r{@Ab@`%~@NbxnjKDxmV~vLCXHi z6m5ddJ$f-jj5KfKC;q6utFo(9k-D}W>T>_v;jPLMRHYO9}_}JMP}{x zwt1BkT}mC8(|C3==79k%-)z+S7rhJN0;Qtx_DFa+!N#mM?J64MB_sCM5e?3nixC-y z7IOSX+hfX_MvubPB}`;6Lg{IJ)lNZ0Hy3A?)1wEF779OcAe4fD9EamcA3kOoUC%30 zMF9dlY-DwC58g^n175wFXHh!2a~l`ui5vMJZDaJV`fM219qGw(l83I{s!P7!!udiB zIJM+qT>sw=B2iT6{50_(4%!ViRo#>l3~|ZrI&lLPG-aJZab^_u4RTwDClE;;ii{TH zDoS=9m{SgX3kllA5(P6`=`Bg-nU_m{`=NO@6&TW8ryD}IPb+q(AOThW(3~gm1imkq zz99K0-(<5g683cVmABV;>MH+ch6@Xy=AIig=trTB-T!aL|M*7YCC20)l|AA+prodG zToroFcw3;5&cm=zoFyjp@K9l3E5@74jY%@Uq(cZ@!8a`}h0#YgEj*j>R^B74K=2N=3_yLLpBTzkOhi=IHqb7aMNDfQ(;U_JliNTzsyNm^Gy?!Ml^*Uk-xr zWte@XTpqbn6Qi>-(Zl}+-LrdrYPlP+8#fDdn33P0ws&y&7Qja65GKf5KvBC?dy>z& znzRlqRaIIR0j(0_?2YIb55igJ-!+IWR(gi~Z>q60{I3f4UNj`&{>n85-{vr@8R-9N zM9UDJK845Q%+kXXdkx-<_rDRj#e55N8&gYSDr4I$7a(YK3yNb|r!tTRVcy_U0}5bq zR!^7KpjsaB23a2q6J-2??UNA->XqAx$FF=@qL0aaF{NIk6}yw&~+zV+6A(X9KU z-?D^@43j866u#1?~l=CpdELS><2W zvS@+xp?SH>>v{WIg;yKqlm*dt=imYV7JKNP``R$@ag5z>YL|&WBPltiDrg7LxDTW) zgBgV(86Vlr_TewygvJS;w%}hxX@Jf{NKg-;8gV3&zcQ8nvDxo*#*F@b0FQQJzZpc8 zFbzyj0EeGbO|^Xiz@kzqKk+~ywNC>1v*-TeW?&yG{(!oUiYBHeMr_% z$TL_hj$;7uE8y`MuqMJY|JVuMRI#r#iFb{Y^!^iOgyfa-<97x&XOMCDc z^fHutLSaftz{DG}YEXvyxk~I=Gz6n1vjc2D=)-$AtMx1B&#>$Tu~^A$$yk5ScD4ex zLG2az8w&3_`D}08k>aK|II@c#hAoII2Npv@rY7iE7Mv}t+7WTlK3r+uEIT8+@)W1-kXmCi%HvZbSV-IY}S2P zwa~BvJ1qGvrIMPDMp|5lJU~`iM+-sG-w4$RPecz#6QPL;G~rXhCg@^AKHDZ187h1* zzfZ)-ny5mWmhZDa@-*)GLEo>hG(vc?awbA%Qx9*wq5aDzRZJ<~Z8#A$2W-UGpnLr~ zgQv_P4y}l;+y!o@MCRo%Zu;r~4*$h*Bq^GcvRKvxH-VlE4-Z+yYf1(W ztHA#f)!y zi6lKDO%YNZ6XcUpO+dM_10+@z6~4$^`__Cs;^I)^+MDJ z$oQ5vum5((=Zy`1yu`aUWf&x}(nY9U6@r{;t0zG1l5Tn!`1rgs@zf>y5OxhtxnVpk zSaGMC+Ovr3a`by!=}+#=R=DW0IayPRuzpq0;RtEa_IAWsMc7Dt_jUds46Nb_$5k@S-M4fWmzC`=y%==Jh-@gE2@QcD~6oG)pRl1^bC3= z_Q*AN__xqu(u7w2VskK^X|!hv{L@^bfl46_aPA4*q1g4@9w3^_uB>zbWtsNC;r0+c zN~OMH7QZ(SeRq|KDu}}?5b$5?EWAw0?z=1EdV66LFQy!_4Ag**esJ!~oKCYzw}29* zlUd~tLn_xjV&bVD38CwvsH_Fp{6W~9{o{f9ng8Q$)doz6LCd?;->a78Ep7x2q_arR zl37S8&athMwD=h}kX!`DygNw_lXQAq${O*{VU(?{@EPiNR9wQF&U3v6jKLR2cc*nZ z14*&{)<#Dc2=d3MK=XP&{!rxr8OQ5t4$n7wf-Up_v4Kq}q&m}_835+KpD{;bU&a4I zR9HxP-g`kcCiLWdHlH0){TXbnaPI*8@IHFG+MZCTHYJCFPZ}x6oA#(kif$U#mtm@q z-nv9s0%KrcW^V0!ofwZ%&R|U3z+Itl%T}3Ve1hZ`^-WBT>LygMDlWP-7@*i0GojDP z75mqL{Vn~MiEsO_bAM~xJR*+c;VN^n(E@}M+IXIu=|@n zLy|woc*bJUQ57ifLQ7IYu!Zv65}=x3DEGWrymqvKj8WDf`sdK@-3^sRt{#f}WT|-( zTC$Y2Cp+qM(0=VjG$d6tU ze=%h?5@-9zI{>FicdLUki)#%%qV_A1n&uxFY?p=nudTM^ zMnez3gh4g`77)dC-XYyk{vUB$x9Tq32{08dqQ5H3~VgIk*+Fy($<{r zo_VQ)8VIu9pbrgDLgQFz49{!ZJ@IHLbFd1RPJ7)nGIbfuYwM$qaMfYoQi4k zZnlS{igQwh?x0l-z8O3Bxsb2?d2ahe-t7?+W@{tAjC6$;<)l5F{|cv%fFI@XIS|m; z?_w81ylLLl@{>aMtZ79aLY3ELv@bFf1D?IeQ+rCQEAInmne}a=jM^=R;;O%O!2-45 zz`lP~+`D z5Sn!CK2@2okf=V@Up+xU!6qi=&f$7*aicUw0UhYA!;^9ko2+YEwa{1HJ8y1pyuDv? zJ-aVthXSgta=x}zlBn;8i}}Z1nK@0+S|G*jmwBbb52_Kqp>d$KPs^n0 z?)WXTvf9~(61myq7gV3}ng|U7T8yfdPa9PQ)3Wg8?nHh5X}REqR)&m@aqrR@eDFX* z>Pb4L^C`>{?!j!0H?Y-ou{vF_7|os6Y<4!gK91}MUjLIRj>y+|AWcH3!sT6*Pd@$& zX@9>hwgxF;R@Di5uhaQ60!DSqYu=|3(c25FbF!B}(s_V#WK2l7n`?AO)i9Ow*hqt; zkMogUN+!>9^ttTwr`o3`_w|y;fVxov8m%q%_4BBxi6fTax&G}~r*6SttOsl<0jwMi z=R$w_u7hKZ%j-NesXnP9Q|FYTo~FvspXYog^cDBZUw!C%mC)h_me*^dl+TIEENrk42N#E9P)xO-!SX<-klGMPi z3{xYc&N4>Amxz#}OWd!PxgO-SMFEe2FPPdo9jF-L*>&aUp}dpoR{a5vP*Eyz zScz2*#}S*N7$$o=23&pewsm$FIL*+4f=UA>ita>N{V+fITU()dD@-)A zH*$0~Y4mOprkwaGxFuEUAXbGhS^ms96M(;^2LMHZA zpkaYUJBjjMY#HDY$nB2UB0I5R@qw{+e_B@7!V_Kv^Wnjcnp1;Xs&gwBf!EgK9Letd z^7#R@q{$kqx^idO$31o8t&-=}_Vu@dE4PwlQo;F%5QDD%c;hcyZ+=|D>~L&VyGg5Q z8u)->=##khbh5lH!zNSnU?2o@vm17;DL%B&f^~`qWz$nfgR!zuYj{uu`OUw>U%l7Y z?QEZQFZ{RTC>tTc>Zp?b+o23s^~=A^c5g6~b&BGxJ+}qy_)sQ zPa-ii%q+kb-e;kEgD1(NRX_K1mYU{lTH<_q!50*9xSF|*r^UuRhRl8O00MzzUcV-~ zR*u=K!_s0S#K`D2k=@TOK}TluEpn@f8xD=j`1KwptmFyjgeqBL?IT)Fwg;*Z~rMFGy(m`+{Zd~GNz zQ5&SE%Dk+?5Iyw1{5q+QH;2nJgL zNpK$J2&=$lm7snTdP@}6mROFKVzqID9mVJ5lHpQS{L8rwU&P_bKvgE9b;QRbxA(Su z>wOt8jK?|Ln7oIfaPN)|v{%6;h#znO(Z>f`hoo1nRy48-rx$aPaTdRO=wuJi z{ps$M3Ae6yG3fy1`3ArTZ*J?9WU-wjZAUw$yA-`?rt-9O(AvLIgd?FFPJ*WWg#}Dv z;AuXlPy>0lSfByA`02NJeLt??dro}YsTtAmjti3+6E=KCP8v;dvt7sp8xKd`7S^ae z$fF1(4;ONiJ+#fMwk&%+$FnBpp$Z@0WWM2A7u5P$Xm7omPy*d!q`!fa`6ha5+_gt< zFg?R-+D0?%fi$)?2tSVDz(^BlJ)`b)Ji z@!Wi%-3T3;<}6QEp)}8WqI;t2aEXurS9gVhpxL>BQUFoxTW4*AS6g@@Q8bf*F9{X6 zGUn)}DqKe?PL|vCWMR8Ry;*xQX6xlkFR)XS`*CETHsC%6!~VH^qP?=6Ha3Q~zxJkp zWgBgWK6a_CDVRfV^!0g1lKs3RF7cT>qeFj>oXGfH$vMcSy#`V@Nvq=#?zw35c*?cy znKNui<@mguT$9K!_|;h*ILpJ9Ydb^y(_ zX1>_0UVfzJydR~7fVvo^Z%$%9nqjgdnG80P8ce+Xcjw_O4EPnBCi_mV+o=N#72UKX zuq`W1{f!*}qQZN`+Kn>wt3)ki#1Vb z$!=?hEFg;egzYC4n=fEf>w&tfQ-b%N!>UxCCo9R|VuCrOCE)%H8l7OteG9+M*ji;x zqKhwRJ7hu_JNf~KUygIHt9O&!wiF3y$6n6*p$o6n%_&=1H9EHTjt^{!f}a@zR&Y7g zV#B$>ptv8h3m%-SK?Ght0=-S?Y#IYS7jB*4+6gutifr*?fb8E296~$$fw*5khb6$; zz?;_2teS?F=Z6Ppn4iYRpGKY^>o+ULkst*iCGWkc+7Fks#b?{k|?`mTKoiTG5_BV`O}4F ze#5vp6l_pDv+-7mt|pPPTmv(2{+&4O)fxi#rP+Z7vPlskq3h=HF&N|D4AxTtkiZni zsJldS`_u&Jwe!&^S;$)^hif8N=1C1uD>6}dDH)QWqoMQWz`E9V= zC^ZtB27pdS@UAtem~;KuX?Ug+#J91y^Hxbfs(xd^W&2a7z23oIEMvmSNy2n6NXvC zO`DjT`?6v!tIb`aFj3!{&`;WVgjT-*x*8@ATQ5K#6ZWuO%rYI1nHFHm=bBr~8D%(l z2SqZ&74`#FT?mf^i(D~fRg*B=XQ3_by>qVJW}@u-JwDQw<>cZW3T!GKZ47hIux>lH zkZ9n0as;L!gI%ZzVh8BL>i8BEfHE;>mefP21AScAp0MWJIt0h$PQ)28_H;Vwdj?Iz z4J>;*;;ua}(3{LE(|I_6eW3ZJ5A%Rc<@aeO|yDy;g11#W}A? zNR#a;bCEbF$NeoLbexH7H3K?X?(^FiAbP(o3Cazw<+Gh_{%OL5+|wScI5lWL)@$?v zKusb&Kx(I@{2H}|idQO9K_#@#dvJqmUxZL5bJ(^M?Ns!h3~pn&lDd8dwZ6MCNS@Cf z2X2+OI1E7GaJ* z_m|KAiObb!)f+CF%}p@B;g*xY5(&7WubP;^6-bf?7_Ffj#)t;O_9351wP#R5ixk=w ztV&4$qI3W4@KB{qX+3%9-%0)uS|{>&-4~42oS|c=mWws)8UD6Vzf1HOMps@(#gUxq z2c77{!u~e-D>`ArUC97YT}#z~(E@O43D(Df-JQAWBvag7d}6!Pbfdp(W9E@*mHm7n z?9JSR4>l7`54RI$NXjN)?qk4b^L$pM>33=wnf*i!Q@oqPhL+;^&9n@K*gI!UkEA4t_bOoH#*mN`h2`vU67Tl#mn9kujU(~0>W1F($p zDQRYA4^ad-J?Ahpp1C(CUU7sxoge{%`R=EI}8+e^nUk-7inr%99oa z9-~;dKuGY`E4wm|5--yS&Mgl#{pveH4Jn46#s=WP+W0_ucS%QC(XXRF3$5q>f z)s|1xnNZu5GKg`zwhFSAJQQ*?(_C>Wdt(kM9d3&_8jWy?`AZp&Th+Yx_U^ zNEq7>sffZKbQ&2aK9fKwFkx4rzdbZ_G2HqX$&<-viV?4Ee{yp`vi_tbAriMJwAtLr z0XXI#!@UaE&JqpBkZH~MY26fjuq5XPC^qE0|0FKv7FT_d&v@zhYi)y(eM5ttpO|VpCz+X%eTPe~cQI znoM)d(nENWTXqnx74Q)03DrkWBwugj-fN_1Bcsg4P8Y>d18bE2EvhZL^^yaW-e+w) zb#y{s1(HHjh2O4ELA%BwU4eGnqr*#U4rv0bsP19P3=xvq0=g>#;@e#U-gV;4mrZCi z&svG$T@j*<>)h;2s!_UaJC|OoHi4R$8}r5AkFaAaa&%F}$46u&ci-t2F1Z+MD%g&- z^c-l$pl{V;pb`EFv>UC=txv%J({FOV7^pDrY#QwC&m)lMyygl-?irEJr#(lenE9Y6 zq#3i7FFVCAekIyF;pZbcwhCw(TVs0gnCj;AJrCp*F7mL#ImPjU zvl8^aseID!bIl4N=!3h-qPGI+Eds>+`%U*79)*8ue45bkE<1RKVls`Rl?M={)3?v4 z9a!|+!;EOU{FVlwHeAbz#Y$5*10Jt8#9nu~|66j8SKpC(D=`^U>lb^@<)M(XK1EBJ z-0QekVbx90O8oxwgHW)IB89Ov${8+e8=-stz`LINo_B+pyVMOxK1gh2br&NgoYihp zlXD2(u$9gU8reP#ERY_JfKQDA5F9iz5zvjI67CD)TTkMnV)p>sFVmhc9ai+fc`u}L zQfvk7v6L5AeZ;YK)u+SWNZ(S`_|qobpQ20Q~xJ z1(1!i6hnD5>hf!(mA3;a^h9ybdqot1(YL2yrZdKP>S*E~h3*qaPr&b?c4$OB~1|%fQ+K@Yz)vh$$lJj>sNav9HC{z9%*7n4b?q@?dV442_ zsMpjpV12MIIne%U6mG3+SAE>moo7%xB%l@X zoO0O%9^ihRPbZ>Zh#*!6pb=+*cnm9*X0cioEHEos#2Tabi~=z559+)wAhXq))Absb*7v`j~9IK!u-5#O(g8v&YbZ?CT2rF?nF4Z$iT=isEv zR9juI^55mZd|{s8RI4M@T&c+S&{Yj&C1k7G5jEWLjIOIHWLCYwL$-=AYNVoK!zln3 zV0B4x`+X_kdh@n_&VoHh+$G)M-+4x<;JyONEGP_3Fr+JI^CIYqd1wgT!@Pal%6-}@ zt+%~x7yF?6;ai6UeQ$uv!pP^E@@mw#t5L-B(3m%_M~YuK$~31f{tWoF@ah*%KDqxG z@)W)0%bgvogcJaFQ}B=T9$y>+B7ZkwsBllR2wN>ebS9qRh45c1v@&Y$*SBT&UnKrQ z9yJ+Qdpmf42&HwT;fk>O^}o4Wl*IfIYAS0AvX>Uw@06$vc(^HH=1U+LYxkfqF3MbU ztGt3PI{uJK(jj4Db}h<%zaoMk-SyB}5?AT~zNW zYo1$~pv+Jzka52fy;o2TbLYqYkq26Eeg%woAvH#Ki8I!|g8gY-#SX*Zodb8KKoB32 zt17t$x8xMKH)?VW?PGYf~aj)r(uUr0>QOie7ln1nf~@JGJ(z_H;TQ0RMWUb_mWclDcBq8V=5DBtD_Y_fIHgK|WpNfbYqes8ZNngg>NCyw9!K(CW z=x9OL4YArKejzJv*v4U$vXotbjG?)ZVV}MHKY-qSnQ$@bQcU=|?Odr@{$XOPG@SIe zP%It%2by+WWv3(%4&Z_=3x6vNwRUp7Bs3j5&E~Z6By{d->!r)5N9bR%tZ}$CfRpK_ z;`{%S)qusw2m^t9cU{bz04HjhU8e!MYvWRR)k5+`1t|5AK~`HDHmJ6i8@@#Ei^9FE zACjHqq%TW(l;83&HHYXi4?T~BEdX;Wc6xGuC}9`ptP1^)=(SFwS+aWo#Jz4PY>6`x zp&Zw`Az##ndF*-@c)!&2FV@2r%6J3nkgvIi$vf_ja8=x7cL6~gv;D2wSs|bRAcoI{ zX-U?b1)C@jiMZdUiOK!c!n0sFpf7NQKqZS%M|iGhWsBU`5|t!^Yh2z=6eL=z$XDg2 zEeazEITMq3c8;*0hPSEieEGReY!u9`Sqk#>p26IVE*wI!?7~vppzke}GAHs##E4ThbM1(9k-fqLjK z^6iW4r}#I3#_2hyIwMZ0#}&J+zCp)Lq&RodRtI%)T%^@Yx_qsbm=p907j5uxW~pNx z##cM-FW(2G!(ac%5=O@{{$4nKy^raTUwCQ9>hhDsL#>M5&T3~-@yaa$(9ufuObEtG zF7Yha9LDeV-fC&xyk8_4WR7(z(GFWCub`|CA9|z1KwxpbZTw&6f|@cO=MZkzOMV_c zKLKVB$xhRA5l0q==eEx(e&~NdM)BH_(F;N3`NG8f5M_D@oKzL;jRYNa2?oy%rUAP? zBk2rTVw7HDc)F>f0LH^wS2~&-r2-6X6c6?x(1mt-cP!<^h_so`4`N*|kYOsm62G`V zFnNav0d`8soMVbx5x@^fkw=ehgrc*lq-~F6VuLpMRgk z1+MrB|JzYtvuqhgjf@k<=x zQE-wr=-a2#Jh`x5U;b)q>`68nsIQ5~?swTuqb|ejYJa)+LFi{u_OXRs1S5S%%Hs#8aii!C`d_Co9 zbCPHG_2cia9|Uz;uLCE#XG!xIbN z4tB=Pl$gJ38|8JEb~E4aUA%jnw^Vq$A;*K6flhcjQ4|bJv4z;Keyu+U8aA3S7v&?} zSw*s}EY{;F&D~$jt7ED6Q%9pfaP(y0xESVN-%l`Lyq1M%`1e~AM7F{c+6$8h)Na$X z(G}_K!4$f)CWP`AmG&~zgVDksVPIu6tJ0aa`lD*zjk`9ArE7 z_2_0b&`%m(7dKFYf@Q6jRC`o}Hca}r2ierHWMK+OghkAGaf5sCN1qaMuT@6#_KWG| z-Jd75=$TN!%zsMZ$;f9tv9LoG3UbgfF*`-!*@e+r;QXivjA^Me2{fR5S#>^eET9bT zm}|NxE2oq8gku67HJ9#OZw6%f1qVq*AT-vZkn#6JL^}akbqMOp1|G8F9vj)_qwk9D zt!oE5(Y;3fYhwlC`7Cqa9H~|3Y5A;l5{R7~|164T8Io%HzZEY8o-Ul9ces_i6|U=F z4?yfSZ8uDb8b)Au&Zb#E+R^Sod*U}=%(^!Rzr_h5a`zsqy`M!BWn{c7d&JR4n&3V2 z1$xjbm9AuR>X|JgVb+b7K}5mmY(6L~yyzv+3k1KEt=l|eK95K@ItmmiEKiA$&E1nJ zC0&VTj{q3P11svP<}ORBr3roWc6Ql={iVS%psY^-SSRgSf=;RN-Whl7&S&SK9ieT8 zw8ZamR}8-E%5HX4G`o`*z$ZC!KG>$NUP!%@`{gah79bchE@^*2K%K@K+D*UOXed{n zMG6-1Hzh{GJfou>R^_$=w3)7+gS+S?fYSgto&|E=Iu`V(dc1{@8bB+1*tc%C+ zR=5q*y#1wjkZ<}71SLj!m1veItgZPOmP+2l&_O2kfC2Wwwzhn|eOVV*ATj%+t$|{b z^H{(PJjGvanLOaO^TFJIJ4TsGu6s;;MeJDt5$sJu$0l&*N1`r75$Lk)3#QCWz~q&a zeID;Bd^N=0TN!F}#L)%JIyZ?U+-wXu9hle2^%KuOn;1h6JIdX64qXx*>ZnK7>3^1) zWk~}rp822CWS~5S)~%(D6uobHEy6|540hQQ%Ohxk;w&!*hkO2yj)NY1uT~Vc`Ct$H+3MkPBUeJi)?U zd7$^DHK?{xRzI0S&UYGED;yj9F#l{IhB+}aW4b2O7E9W?jI!nG3##&}(6+jF!97Q}5hxNeyEOPGYsg(cW~UR>hc2FfWi*H;{16UK;pNIgsgoc50cF!$+ZZhhD7 zAz8380P3Nfin#MW4Q?-SY2lT1D4{t9K-Yf1^ekd#($7WdhyQRo!P9Nw#c5XdFCV`9 zo6R>7+IYGp{`h~jhP{9r#gqgH$I(=M7ve(u|Icp}3esqBI_xc=;l^Vv%m3SPjR=2c zR`9T{Jxq|vR|z`^MnEZ<5T?o_i$70*6}AnffN67h-divk%#A|YppRMimRyE6eX9GF z&Kr0GKvgx-7E0TdNxFHk$>u}n!<^%2J0M6Ksjq2wlY3Gglf1fFwX`;=H7J>*Sr^^B z-CED5djkQua^~Y|@G8Bz*yvPW_-`%ZeL_1>ct5au$O3&>REo%w(|FGksJ zxUE%GTP*{Lnt0=zFM^s;N{E*UVt#*6X*GLZ92d#9g;G0woC_B?28WM6oo$HxruAd^pByF8Vvl^@etPTDz#EXL+wFYtso;i-Wfl_CNl8OoUqTU(gefqOc$5hfvXa;JlsM{%?hK zE$7Aq1~FC8ou0}cbKi5{YE{tGPE=Ie0jt|;=Qs2AC*-Z!rD(~xgS(%RhhV!0ewCi) zx80NNsY!Q$zYv-K!kzeXWYcE6K44Dp%jh`b~53gZG@ovN~dok*e zt19>d`I3|;=>_EfRIMQCYpCwz2eiG@r=Dk3eUhxGs~eGaUd()JqAu6nn8`$lK&ZPd zT#lc}_KQjRG&XhnIO2~Or6?zpAklqhWj%Nl9k-N+ek1;@Q<&=kY+E~A%NCluvrEC` z*t3uzrbfV}y~~AX^o|idX8d*--J7sci4}%Rn=bMJPUe(s_YsynAmyWTSn9MllLyk7 zG1})C@fn0c0=``ik3Dd4S3*{0XV{8+3#4yk731E%eP*OBwBI~SUuw%LUC0Ym5jyO_ zjMu$(DR|rA@i_#tjURQ?M&k)3Ed1VZ!@4aMZ)wlEL_=GCIngiC&*+pjg5oF5v z7~(!Nw5{Pp?hdUBsz*CF2GT?vO%&~TF z4J0E`cLaWGCmIqsPS<(>%HO7SLG*ydgHCKH*A7boxNtxh1Znx0`dcPPK2ueRZVqfH zN5O0;{g!p*3mKyBrBZ`;g@Ng>`~KSQKO~xj8i@+`p5zBN6*_@lZCvS{+RiUiLmL(M z8={OdWlKwzCIa>Oc*)l_nOTR@z9P>?ww9V@P?3MgJt5{)>rhA;QsU z5((oH;C%`GV~oq}wvMVq3i6Q2i!hRRWAO)AZ&m+q1C9+}qlK7bD@G3z`kyXZ=KivFxlc zHQkjcO>?MIR;J4lK$+b5Z%6SN{UAxpvIgG0W$SgK2C^CcVQvMNIM{V)61uB6o$MdV zWW|97%0dN6^Rt}_x4T3B48F{1rZBIZEq4#yD0v20UbsQSvx@8%DOLjs-ajI3;oCs| zlG$Xoo}tSO-#Wao!vBu3cJi2I7cPhWey)92#mjyR;-PUygY~;_V!Vot4lGO_S;7ZR zL5~9ayzIA~*Xx8bs@2*t$B*dGHDNWDk#PNgBfnaAb9|{~^o&^j!R8g>_NZB7Q;*77 zz9Q{;IRR>HB?2_;RI-$I1YrD?4~&mzjDCPw2kI1V$0wLyYhY&HtYz|~kY5=sbfR?c zGXBNOt!YDm`X5kHMyLYF7bi>n71$g+Au4wI%pD|<>zaUgdq1K#`Vku7Hz8**jM&xw z=WL=>1Kr5bDa3iPAzf{|*(Z^kHaH*2e>*zk+rGtB{{z+!z^GlM;s6tx&8H*@l@cD+ zh#}M4AfXgM(WM%CZ1c(2H~0PS>+gCR-upkp)M8^*u;K)6a;lb<0~64pT9Yo7?IEux z=JmW8h9mq^UOBUP7BT{44HFm>rAO%N1wJ}FNY(C^T!Qnz%K%5~^WY{BHycJ3B&pPiZZ_QECqYL@o4Sfa^c@^WGCiDlsu^WI#{NIL&0SjVoAbt32Ntz7r z^9B`Dka%neXah)}h96x$=G>r;z$Vz1qD}tW@q~7%JR%xTLjOf6ufJa)EA%dni3t^& zKNOy$mW7SoZ`ku_+i(u%vp5lW!N#nX(KSViT&Ur|fEI<6;bi?OuhuJ|!JixRZIh__6VAnlit}sfQ0+_at?uJGMIOp}Kb^Rm^Edu{pk8rOH41}Xty0%QC z&uGjcx*!*RN$@|zwJD0B*k2Guc+zau9b_?K!UT3e$}CeO?RlMM>a)`JgqVokd4vaq zA4{%tSV`WwaA$DB&)dU#bYVRt#?D+pn+8+j6&f8J4eUM_sgk%NU>K1Qv(fiNU-<9YU19rIKboZ`2gn-%@h?Ya!=>0}Kv{ z`2CxKeu6x+eIQ0+KDfVIq2E(-01tUkt|E8F-A>H`SIwdYqH@ug81++M`Pf%2@;tP` z>-oPW_q?;pYI7ka=G`_j7{;nc0REAoN-1-z!dr~6p%VCBYYoKWhivrhi=i$`*^2KVrp4rcnb{iS*S3H}I9IC@kFH#p&{Sa47+R?u=avyieTkEE zr@dC%yXF+xta6+2ppuibE2Xu(f+88WoewqTg%m5OCeQ03W-pq>Z!lsvz@vNL%R<@XnTdHI< zuU&ip3a>ILXovdL&K3}^+8E!DQ}s+nK~R60@P`dG%sr2 zxI0Ydg)N265+M%V1#h`0k#|}>D1SKV-%vH!k`>#|y8e!XQyPSQ?>MKp)S%Rv-flUL z*aO)z@))|ue}Rpfo8{}@fIAJtU5+1Ql+zE!<|U9zR3z#0<2dJkwv28h`{f~qz{Ydy z#fJe@vke?V8gMD)u=o@z0LFe8;IatY@VHkTZ{3pJN70eSpV4`Hj5b@pY5?e$x%hZt zq|qbo_Hs*#Gyxoop}@FvcNmSsA+-hm!>iSvXEy@CSlh40CD#GeDEG^cGUj+6CWc|R z40Zr|bz=bEn-OxuUyqP##Q?>-(cyy!Tuid-_UBidyo+O`DYI$l5YPzExV|qoR{P4A zh5_ZM{$lO?ABF%Fe6I+vJ9L_FgH1_1Azx0@nRVyz7+VqDdJ+D<$OdFI5fbyTx$B_c zXaI8ER&;0kZS%8d%lWX8kmRd>xXn$ zK3P!dGM%@OOikBo4TTe$x&S?PWPnOC4*Y){osB=!`~UwtpDxsqqg0$GbU~eBR=GOI z)s7@-X-c`+uu5Xq?3Alvv(p7aC=N~>mRw(qu!~$xF2<1<8(TJ4n~;mKiCN6f@BRJ# z2e#XLZ?D($`FPy#IAIk616%X;5bqhm9dcg;I0@=b0pba%WMbQY+t`qJ1BHe-O@$8H z{vUq>If+XQ-(-*QSrb(4r&95f4OGuHO1x(^_c))Q!xvb9J*KO2MjnJT&o<;=Hvf)V zJV)wmC~Vxsn8!L%(snBjAP2*2Y6Sk1!yoYxNb>(i7Et$ER)90S=BTq?VryN{1EZ&# zHUI~Ct%f-<9sFo3ix+ioG-Do$rpqeR4v*t&ST8Xsb(;Uf$%%wUl+uW6l`;c*me6T^ z@7z3Mm0Q6oH_AY0z~D%Boe=y<#}8ZbTCa{x3f1uhNcm~)D{~%jese9)}Jsysf zW-&y)klhRW6mK_X;ThMkm+UNm$&%sTyhmADJW+=)mMI&i@^Pc1+oBnZ&rL0`v2v`M z+i;k#+ngYdkp!uEka>&=6|D1eV4ocUT!i2;1%Lq=C;ZBKTF{UCE+3TN4fpTWT@A_3Y6Y3SNSA z!D+SmYtQlBeJPqR>qn&q`)vj8(1N2Yv zm+#%L=Y{3=tVLnzPElGTpT~S04Igl+p~$mp4r1I6{M-Mwi7gYPxNdZyOzoz$tGEnK z*@_}@%THLX#pn4SFalNi($bTq+h*~F@-$;f_+D1&a@-6|smq^)sa2#mZ@ z$6JH$ZAnuaM&@Yemdu}61WgyuloO8|U(UTn{eC>l0<#m7sI#7~bdZ+R1QeeU;B7iG z@V?cnH4;~M&jE|HpU{hfkeuws8}kteq;}dBPX4sF&h^I>Ci{Q@#YTq)92#DZnPkT? z&ypR0P$@jkT5ml`*dG1Y>0$wTgE*dsE$J>ELDvfA>FH>G+5;_tSMke}74+=%qIGrb z4nM?apFba=>RaQPd*`gXO1mFXV$j#T+MMzAm0)p5cWber*9*26Qnn?;0A#iUlgX|B-{Q z4hMj$!+{s+Pp{MH%!d4+LSk=ZiIx0;`-biKUTJn*){>q9tELF>RjBkxs}gmOrycC7 zxkliJ?1LmG$xm{_7HHht4GrmCGwiFhCqa}OfJmLQd)Z^5QURGlso@!7Ttciaa34LZ z@DzjNb8*$d2uFv9mJOA5Ls=>#B-jPAx>jrUxhK@-dqgm|&^+U_Y4-kn&Y$P0kOb0x zuxcBSiHS^ET-C7Z0#m&3*L1NrX1$~fK_F%-Bif>b@>6TEZ&mFDx-q8HgZ%!Lyxxax z82YevPst!)_%)KMB{P5NjVe}b7snT9#sQEMVW*X4V{CERJa7CSD5X$1@0s(A0v^qn zTbDz^fDmq$OStam+c~;9ZPN9QvA}M*!M<4)n`80sGR4$f;@HXjjoJF1JH=gvYJSZU zB+Uc8q^BhRU{6G5W(ub#6e@q==Je2T;UMp2Hv}cqc6IDf?FMX3>SDEaqS*)3zA&2> zt4*9wBSt-Cd#7~>xzqt2Bj023OO4?Z!3xKjd;5vHixE16GL;cZq#fJ#1-Ay>t(aof ztNN@|x&V$-LF(v%B)oYtiy*gq=0M}acS%)+q6K6nrYSCSIDA(!tSQSlV#u;h`>S9& zM?GFiOuiybt*yYvEfNBzY?hyGT&gV!&Oa(UK?*h{m!r^;(2*5K19@EP^Fi3%+TPe2 zoS9D%U7PcPzaEMO^4m%vQyYhscHfXqBsZV2fI*Ke(Epg1`&rb{rA3yiUxxn46@E8N zXPsz_*F9|CT?WV8PhfM6nl?{mMuvtSY~5!|oLqiv|68k!GM-5c4-N+!v;rG znUt2XqB&9-Fn6b5wSpJqUxpDY_g$|B5$4AymV!{HF)_) z%pU(TKLV_Eh{lp-;45z`h$ZvDVofk1&#!89s`$O}Ov}5}k$YM!|3G>2YjN`RG>LUg zl}-h-Tbe%wOeU#2fl=G?K|He;Dc$?uHf`MwkmsvA=GJk%0_|eanDs@vcR+nZeco+) z`b>CuW(gAZx*7+j
    _Rd{lz>X6j3d=ffR*`5QJCnPK#8@lbQ4jXFDgSl|l9Px}` zxwd9ZsUpc&chlG%`PH;)_WTw3KuA+Vx)}CmFMB&Vk@?F&tX^#YxB)|^(}vSaVF0pp zkW z{dF)*a$~j=4vZO&C~_pt;YD8&0R1RgsAsXX_QRb3aY>qUu6y4n`PuQSy!3HPa;UBB zb>S#L&RhfKlPkZ*s_|{@3JAyR>)%VKE9WJXKZDnBT8_nj_;i6-E(kz=)`92nR^|t( zxyi2QSy`-*km)4V7~+eY8nHyt>mj?wkOq450RMoF`_+M9)rpLDfgU4~62K2J^rn8j zp*nSsYa2g2%PVSrfBT?|Dlt5w9&X6=j6zTOB0m685!^GSK@IS4=^~%xZd5`>3iaW zG`>8l451)hs?8UpZQ=Tl!*375N?al>`iqGLh_gds-LR;ozygr2F0y zzw)M>s!OT<{WrGSSX;Rwv^^b#97E?DO7q4u%7E7~!`gMMQt?jV_;1GbZ>yWRHYx7z zFm>lF%OTtw+DK(_>Bx>yI303a0h3+CCka{hHnOhziqDJ=3FI19_m=)Ko~@HXJPie0 zW{`qwJS!Wi$h;1!frfhZfkxfoYVOfPI#P!(3y zvsHTV{8-0>dZE*~C{c`NO6*Zi;E>7)pX4C&{So;CG$D-Wf8+jxwhCCderd^EV{9%a z^x);5;;s1p(>2F853aOGXq?Y3(>f?EunH18NZQXlj2>6xTM%u!tgLZy#e>RAT%m9j z`FVt1aDRgeq8fy7{qqeZ$Vy#`y(}>K-_%IX7cvz6k*BoMOtp0uLtfA^h4X*lFK};4 zmuOpcf`1l58vSyy)kwA&Z~nEAIRwtE3gnJh9RBjkA9JyZo^j*J(w;xbd_now*7;)lvi_p!83`^!2amn7wYdX|MhLqI)urQu-Jb1U;E%qUv~XwjW5UO%`oI5z-&OLW=R0A2<3porYF3Tc6mt#%x}j zKH1RFP*+RizD<`Hk`uh!+}n@+ne~q&s*8i)!SvpaUSr#BBSev=*asD*;sxb3YAu&= z?EjX0v5x`Z&Q4y#JwaA{$#TTo+PQ}ltXUL9ZlWC#2^G*Gk9vbDLnGz60c$mXPH5Qb z7UFw5$p;8Vkz&aSk#>uJ@B;dv-|>3B#U68dBZvAv?DL7NStC3o^{U0+uK`cwPB9W*R($=V|2Rewp(|n-RCVe<3w>DgH1`;PJyIauRxu%$vLOVA47|)efcI4;d@}7(D z00A;M!%^=qOKr1uMn?U3vyK6-f!4GM$u*;SQ@+L?<1`i;epQipIyd4=4m;Wr+<+4wHO%$T-B zl#c&03{kS7ZaJ3QRwhPJ9jZNwVVq3YM|LXJXj3naR$U2TtLG!roy0=BbD-()EV9R@ zPvkydfZpDQd8j?jo$R9lY88#~g(hqM;r@0;0xb;QYFo-9QMb}^HH^MaM#S{)mHT8&3uC)TeCYrTo<*Sc?1v!Cn|jb zTwrIu$q;povs&|0Utn1fJ(vgbOGZcOeTpU)e?xek%NmbbK1}#t4H8_QxwY}*rYP%m zA>nr!O0q{JiM%2U_(wp=uiR<~mBS&hS}%bT;k#h{h+S?`yc(TXR~7%u(Rwa8vwD1{ z{Sdu%o2*7##U*JqYfc#hjKv^r(thmH8d>A3`-nYb=gi-Yssckq>JVp1Zw&tmlwm1r_j9JJu?M%$YsOm} zJSY561d#e5|JTdL#;~#sg#7V?RH^z{PjOF?kCSe_0L}5f_70)llm%wOa zv3}mHpjTt9Zfs;ZHQ>qMiE6VeGOI4~e!M4u>G|1KZR+yL7?gTnjL1M;Fpdb$*1}K% z&74E9U}|8Jr7QIE5EcN$VyV;U8%T8=xTv|MW#dGb6-ZuC^v{Sa#~6y9ghq55rbP1iUEzH2U;m{nk&?j;!eJclhFpA$3zV#gp90$ZPD zw~OoMWhAsyW*eY4cW%Z2CtQjDWPGm7%3Bd5h93C}aDO=xf^&J$I;g@2v)vDPT!^kU zHFuwS)p)18{=$se)5ElzEQj*j+3%3+E3SAMr4=@CvVj!i-j5p4PX zt(Rn0Pgh+LzBrb>e#TWDk3mJq9zlPPe{1H)-n~4t4y*|!G(w857x>XWX^>1FDNza+ zk-;wfrS;Bn6WPfE5fR>=Wle11y~YMNn!cvcnqBXZY_`%|c^smR8;6R)B(5f7(6{OI zdXJ_g3-rCT1Vtoq)0SW{9BZdA=#Dj%IwvesNcBs1Io0M3rd45z2VtzbmVlv)wc{9E zs=hrh8pV>NI$4ZA5=s?(Oa8SV{_TRKjpXUrQ1VsNVTcW52T1BM?iW4Bt*4c6pgiMg-o@&(`~1}ru9KoV`8tMOVZ zntM;?)+my1$c&ybFrj7D4gQC6kx26-4hn%7L&Pq?bMTVm^p(<0(udZ4Dh;-DEl@Eg zf@&%Oc5NIDqFJ3CNSV0MIubq2OF)5KJ2dg;)F@Dh6lojE{Fth6Svma1_ z=VOgzwEoAy4dB2Sq_gD*LI^)QR*QqfhCx?POAKHc*IE(*SP`&e1WHHbT8i9IhaXCP zSfL#80AQmesZMwFdOVe_g_!X994v?}74K=@6DC0T&L8j_>lgew5IZWk^f%o@aF*zK zqcQ)Kik)=1bZfdZ+x&+Eqd?EjoOxcV*#Xh^Qd-Ksn{^!UL;ViOTWhLDXnHv0&|BUC zx2Dt?A$$m2tyHYpl!x2~I^tTC;=W?W08seoYPs9ZBJX$*o_5G@E;_!cQoufS0(PW7sO;VR()z{l3c8;b*zS0#`R z@*WEf14Q#TY97Q-x|WiseAs3YM~i=(m2EkI4Au(XhlU5*jBlmx0o&G`3tNWvzEy-y zww&znmXecJv4N`n!ritm*sWJ08#{j=nsf%h??ku!CHIin>R4u5395@r+a57;D(cR( zojCK^uK4L$wa*B4ho%MrJYcI{^jY+ShF=EowW=8-h5|`JKnVqyMylkPNcHR1~K9O;99CwC# zUs@ceq{Fsyi$~x#80`T=9m`~c5kTMUHG{gMn3MiAC7!9ABu39K-{hLwYTSK^C*GkY zdjO*})~@cWn1)&GHSg&rj_qq4=)1RvRl54K^l8dTZ&gr+Xj!5ue%^cs$Twl{ZCKVu zzE#wC4VKn54TqOm=$20o&Bp6izCSKq=`MFInHaO`HT}EssY_}7rMhnLHgs&?goZr` z6Z+Lzw3Pei^TK)bA9GcLy&zT1$(I%ykX!k^?lEKckv0{XN5GFZNBl5#;Z4QsJcBg?Nf^ia&f&IK!PL z(V>|Ji!X5=Takj9@gBt1C6DWRBDWk)lvGW+UNbos<@6ZCTE2RZ(;!u z%+}oRI)!^NA;{&ftmugG4;vw~4-{J2rEw&lf zckdhmk>omp>!Q?Y%87Kl`mL(%$aa)EP<5+O!lmQ{+Y>qogtN?}_07Yo-XHPFz8P+{ zE6X7d%(oz}ssVZu*kX;vg{151JTVTypaYsV!_vKi6!~-E%gL3ZmJ8oH5cgX|xd21R z)?EztcN|{3%wy$x$$kR3eidFOgU%&An5fe4*AZKBrl?Qx#+vWOSl1OOacZV0_cRrv zt!RXAH+aslo{1kfeA2IBS2|=E&@@c*#$pdRH&B_}TZY_f+?wtV;c@Ta{Z4xiOmy}& zaO}4M`QATm4{Tlx0wpLGhfgd;eieM2X(wKb>Fxb$ z=@b*XQawWjw6_3aw~}y78k;8W^H_!}P{3;Nbd~Kbqr9%PP_cRoY;hyL;HT`ec?2u} zb?FFl#G&>9Dg2LmpdJ#zbqeYYhbAj6rKE%~=%EDz4f7PUi~Lxvg^dUK=j)z&=CEOd z5_8tAMp`R=Z8s>APCPHX{I0P%<7yK?hEd^bCi9JpbBlrPW?%3W3n9;b;|ob47|9I0 zxdGhes&K%vT3Oj==7aTUx538zOWibjv2Hf(CJeHkA zu-R+yLsheb%sXLcfS4E)I`AhWii|!W6{{yLh|=D6<%S) zAgSpeS%9!`eD@t{JO@>54%z?pmGaIo^Auj2k8F^s#nI-dM4aPbu3H+salkOw%^OmU z^&hzVWMxL^#u%=qBR{GFrr3KU*R^2Spz#wj1MxMobXnB;+}3FG*Ezlwh)9lGT&8L0 zc0|*uy>IBL6BueM_K|UGEx;W{5c57&y#wbLmm9uxY5LTJ#;#b9cHu*TGh2u2sSc*P z^>9~>*f`nsH$&-a@?^>tT%8L*`=6^DK?M(Zd-q_fm<3}dNbZDcPU3!4IxGAu6?6;5w?f~qOv3uh1wnj9!JJwBXy2XCJ6i=+qw`JG z_cm=|QICM#K!#7(J2dkmSnA6708MPf#fJc8nOxif##8h6VaBLKRnUNe)dmW}HAZE0 z4(T=Wd5&>@PMad16XzS}ojyioB#EjcuT9thN9J8A6rY-|!>g3QE_*|1REEbuAV4A< z7kASvt8wDrBrjVV5&Ovh zuec`wivvLZ-Tg#mI)B5yuw4D(s6srK=Jr&rtKuU$yO*{oR1DinY!KLtcSXWkm8$ol&x1i8DUS$#6WMSu`8)8)G z99$&{dMEr{4m5UytO{LaI-Lk(g-{gvXHBz3j5K|@!$3H0ko_SvD1vl{o(goE<8%9X zVbf1KB(;9#br;9)k9=8K``Pr^*5_T&&P^i3n?nE&k^1#AA1k`xRw|_skGc4+j_Z_K zFFG7TfX0*pO){zP$JXW~{`G8^SEVp7+vm%l4vZjO#IXkyuYQ(g#}f|7VC{a#xxiBo ztM|3JF%Tw%_4zvwRUg5tFCA%j5p0=HIQ@cejwyeDGKLr^EX~9bglR%eT{)ot`XdQtN{wA|X2lY+uOcB0YM6B|!IH zPh;%jxT%y^N>SeD*dJ0}{8sWlY7H<9B~IZa2Ky1g(S%6QrIupf5rnsczkb0H(#T48 znfbcSjR z5LN(r1}UIs*}(u@!$`@~unp34&^HPHY+K}3K_KL{9yL;{0~;#lClSXv?+orsqpc-@5t9(q^*w+<+{d(7r7*Ar@B0Bvp z?lY0L;r_=8eKfEwCNQ398YJk-+o5kVyw(J)UwIbbQ;6hi3>p+~voBhlw0dfIggrCs=0{;&$dLlAs4^TlDM# zj5gyD-0ss2Dk3(kHDs7o=jY%*bMYlRX0`(?JV|fe{7o6eHSI5)T;Aj3tSXsE7>0CrA*$434ru5(@5} za2&QkwqQ-G7xTL>G{xK5NY28rT0)Clizn2S=_@%LfyMnwazf3Yas0 z3>%B_9D+~*Z>s%iVZ{PsIWFT?qh(HfK_n(rqseCkT->+(aw9W8Lyr9I6q75d(PxGD zr_46sfQDftaT2<7jt<0JY)=p$mMTLVu*Z>OF|M32Ac_faQn!Xekw^r}Br>tw4;~KW z!mP5`g`;(SvkvzV9XZn04y?`8u0r7ZVvDFC(qqY5ufg8==#V(#PF1P+@~ZDor_I&J z%&yCR^cp$8m2T4`n7x)|{Q5x%&-p0^ER3`yfe5E%d>1NU$9jHSdNQqWYy?>0u16Cr zr6USxd(J;{k9CHIQ9ij0{&wL!xKMDB&jGfEH2yz2V9A_ptEHL(6WZr<+Dbptvp!OwPZjmz zQ_oi`B|05BRxdKTROX5R|4Ph&B7@aG`FLMN#a1;n)`XeoPw^vclMZ)L#AVZ5ldusbSnPrg+fx6;842WO zxV)!xPI0j`wdS05RQ&`s0;0E8{47fd?8FDQt6x))#Cw#EB3$A(Z%nHeR?v@AGuY>< zH@~2R7!v%GJ+N?YtmlY)BQl=Z&nd2^lpHp_O>4rw;8su#E|A)*6e)mnBN?Z(Qvr~b zFK0Rc2$aG8`TVmo3jf93p;nQuE6y(i4&)yb7Q-cKUJTCKWo*ubyQcVh7(J#}Ht|HwQSqO8uZbomI{bl)He7kO55rnS==^-WqD;+jVJY ztG2CMBlNN9s_tx#-Hj>?w9)B==z9Ub^6RCFw3!d^dt;d%mz8Tx6M2 zjCRL>_(E{aGQpt|&6hdc9ds)5n?MM?=O2PA$In+;y_*=L^acsM2{!#r)v?3*zrgn? z&&nd@Tm9LU3lG1_8J-@EahcVn*h-}N1~#ia(vWT@7)hQ4O^Hx+0DS$u*3QXdu8C++ zdAj_oJd!TZDB@!ZN4BcF-Rgr%adl&Y34qdM*Pm&qEB!r{`O zvkQLOz>Ercg`+$yB5V5upG7w=6%>!S7R={sp(MI3*Kc!8!=AtWOX&0cz?__S-mA+W zi)1&XqRia4O)-B)M>}`6KHD%zW*(XcAd=5nlOyJAz3$N4bdaH?HAxv0%<-`(vc8mU zT4|dlpywR~Jole-jSqQejf4nhi0r+J?Olj8)+;G_a!~+O(tS?~JG#CT9Lv32j41s0 z#j3Ec)wc)laH{W+-Kh8J>E1?JF`iCOrepgvf*PL@XFF|r_0&7;_e*uUh7I;9uTM!( zeoaFF7fd@YoW(1Y&I-ak2R+nBK=>&R&HIIi<+$%B=vU^giIKvVhLodRdu<^5-Mzt@ zNLC&P92;hDO(OC&hb_?6i-@lI@n094?d)3q%K2;mzweAzH3HS-1dW}2eSloeF=lUF z@KODSN!)X5cmm(2wIQmA(|owNyUGRpf*+ZGlt0Df(|U>O7xRhNPz++lh$QExPO#qxRk{lDb*W~1W$Qr)QCb4RlujQ%EG=G^-D4O+y!r^cel)=1M^)mLbt{?*DMD&pT{md)_jMq{_|Er|t1mF=r>Al>?AMVk0Z(A87|f--yGa=1 z32K{0YkVv_RtH#A;->8M=euT6$shc5o2C4CiY=<*!r&<0-6tp<1A$&{Tmc~)z<|}= z`yN*k;nj9$29gtGI!IY zr=FQhpkmwJif(Z(47cAG5$tC!yLukwQlmg{mK|F#EywLH^j>Ms*R{Eqf;*A9gWhY- zh_vjl1hc_~N(dlF3&Ps#C*DJF6bJ*_l8V0487e_zByvRGB~72Xv_Z}FIF!&R#UqC%=U1}NHEWO zhNbFg8eBykocHA6c4bDAj3#n1GiXmMo5^=*$Xm=#DrXXxqj{&EkL-wKEQoq&K*V!{ zoJ{?Wm_-g*)M$<&7C?1T9$D^Qb1I!w@ad4{yy6DIpv#$2mfdK&V_B6Me!QVbk<~E; zQXJQFcGq5q1?o4Id36`gt)Nv$d0=4T%sWBlgxn@y<%Q2Az(F?~pS`GPjGOP#XIuWr zQS?rcekSbMdTCBh@jK0@R5J|~b<>nA`Lf%P(pA)yoWi%xj8o+XoGb|S$C_6L?CgvC z6hS!{Y97p!54fBf`Cwsd_Evdk>u)7gFC~49aAO#l<|m&7qLYyweC==Oh{!(wd+*Tc zRPB5Si&N@;9=11O>>TO+9QEls{L`a>I`twXCF~8y8KSP2P~|3!s)`# zL4S-^MLRl_RrWj*_;zXqb*+5%VpsT%(*CJuVGYPo0f7kNpJGiH z#*JDOF;hJ?wt6z1_KjhHem1uD&%^F18<;NmKF1E29HMu4T1BQ5`Bb=+i$W$-5SM|M zGPwHwU)!J84l0QN1qPJ%Css5BY5vV1Zuy8}hN$V4c9Xvb*@sVC>&{ZGwyFG(jD(SpAgo=oLO;^H2dd9C;X9 zjkU15Guyc7Dk&lR1BJ#fjWy)3d_p?w-e8|aW(!5Fousoyac-a3r@T`NOfwEP|Au}G z7lUEm9)~!}rfeyB05c@B{}CjMs%Vd1!m`4p)uFZvBNE_LIeCdkgYN z>4(VJqm6vq7K08s;(mR}*Ns*3J+8R1C4sFHFjef~qj}iU{H|Yf`oUI0b`nGlN6cmhzt%bRPjy*G6sNxJ$-``K?u{STwa9v-_*Rd+R5%V{Mu*k@**MU12 z85NQzzZ@OIosvf$5JB|)lrWh&QzH$iWw(lK2g5aPbVbs$of|gV7bT%rYO+o^b^!K95I#!b3Z6JFBoe`@6J6JufS z#5*s`;;omEyL1@xYwJ78k5LSrbA9NTh(><`S6Ts9>&z*AYHK0NyTbn9cShP+uVYWzH@-VaVX&CF04$+<{}l7IQ37ymSHAlMDt9;(6{ z|1K*;BVGuuWd^)ekgg0tYZvu>VV$6m`bhbX`X+916V3O`vUjRw(x>?>%ZT#`?~CNG zo|R2m3VxWtH4XrY_F1ta1PPs(UJYpH{)KZR-ofgaigEOYgf9<|n=byB*^nhbEkYN& zn(sFxv9KVnTQdB9QcdT?e&y~49>m^Ap?*WA`4ks>$OVHIWArXyk!>Ji{L=_ID1dq% z5kx;XujNwagx(>__w2$BlWkESZJfxePd!nVG5IZu$40ksHLc&N0oaR2ifTs`Zr@ z$vp9n!EZJ$z$CvJ^ZIIkv=#|N7jUq;Zck3D0{#qo)E1+4f#GhUB%=c`_d+}$1@v;+<@haAROSvxvP*KZAlZ9zO<7^(8f(ljB=uDBj8m)KTiF|xr}?2(DU zc<4oI##7Y^0`mQ1nz6x7v}LT(VD*}-<~1O>7f956PCGs8tK-&$gT)Z$}-m(Yz=VMliCG4B1O<<1S)l!8}zr7bz& zE82FSqv*!V7}|p2aS%JW3i%j;Ro0c8UAdZNT<3vhbAc z&S|c7X^7T__cZdR{pRJmTCDv>m(Af2PxpK2#=k5XQeWYh_v4>_SXZ5w?PSESy4>NJ zgUvcXrSiDffTh@yte;k5ZtYjDpQnZ1fR7xd^CU(?km}N`-LJnSUqjVZ)%6SLgi7`;A(v(UcI88>Sp(c6yS{{ z%yFeRMPkb5E!C@nvulYI$}6KyaI(`Em+iJM&9l~jvo7r*+OWoV19N(nh7xdTh1^eY zOBJ%BX-l={`+bl|%=+1}wn|cvm%z`pY!nIZ&$R2*@;kppDN6KzvIIGyFh+~h5;poX zWF&dKOe3qNNI17K)~9+MiYdpZV~7V__@`c1FBTE)Y9xRp8%F{kScuZ4UV}JYM*mi zFn?oRZkkbp&W`R{*v#|JCqmt`gsxAFXE;<=&>IXLz}bfaanx0VKi5)XK6ou*Y1lOgPtatGixT`OH;tU}d9om~yYr!RMlMtC##=QI0k=QX9rP z@yF)~8dCRmL|lGcXu%k*^IVXi;`0JOrzo;LM$fwcWK&(zx2r8_|7Pv33Lipa1E^Ur zV^@7-l&MM4wvGH{7IaU5j1+fu{K()EH3gwA)Z8=I&vnIBIb(Lx7A>uvm_>5SvyQeN z*5E$Mr+qMgjZdDIs$jUeJNpZ1XV`1Nke1M48EYb|2&$A$+~hPLOnZ2jUXKBkAbq^# z;*qg7?Yx7Gl+fbQ-l+TTHv?*)#JGfjO+Bnyn#dx6d(j1~=EwCEuS8&v%T)B5OS5i3 zq1QNS0c7wLF%}r08kEYj{ogRMbOMT8NYj0#Rj*tlmd1S|MqeO!$;8SD!0k(FgDae? z2`5$FKz6g`*cZYbD)tM6n%|yh@btSzL+ftS&DT2Sq&r7(``2} zXjTspP3)^qEF$>GEQ)6+)j@(o)#X1W2NJ`f^7|G1cby=pA-wJdpptLC_N=$t+PcIP*PJ0G(O;+XeMZMLGWlEK}nfNGU+O~Q`9XInz#B| zx-3CVN8Ys5X6y9#u2@!@Ve@TZBevVc&kTOBKkXtW?VgjvMLYT7y*)})We~!n08{BT zpT)=$gsod;dm7xOBQFi$YIHqG$C0{L4d|jMzonfm+a%2G!5)b94_}o#nA^BC_8za& z1k*JDI2r^X0R`)WD3oLAR(gLgJRf+qoDV7lPCs^9yHM)A$RkN;4~I2DXyiHuTdK?3X#Wq z@*Fg2{7AF@k5Uqgyi-xY?3l_$(_T|;Ds?p++v{A+d&Eu2y<|pPiQz*hY@n^1?SI?u zy;=%LipcpGRkKvGVla{+hPF-otV9Eq++E86&SP00|3X1_m9n4RX=q97h}6;du`YcM zS)*L)g*DS*udxxZpz5Rn_?NAm8Jo;o+x`WjKKw50h)mf59|cjAV&gs#Mn0kdmdT4f zRQv;K>)v^uAhsodgga=4#zfSoy<_WrT=3vWlSY8@V)OgiCs(%u#i=bdc&Fx=DTo?U zk=}i2uy9z3eyf{EQ-5etb=EZRtx$k)u;V=~K{=F`G}!ojt~&7X-H300?f0adiQ7_v z&qP`xx1=(lWPZI>6B|MSWvh_J&lfxzCWG}8snOICG1qh8^9}@(RYouKrg3_V16ZB@z>^|fi;#mY|>`HxG z-L3TNJLaN$V<{GrmulG<0Br~46e&M_LuGgDS6;vgbgum0J5~wzzk`dZGcVfAa^D7DA1{tCzf%?LZkxq; z!S^r|P~~#z#X(2SoMouoq*Cm6RyvVHtx4d?=ce?o^I9bYrsCVYBBfXm%CprAo%eFl$yLG zH)10;&HicWn9EOv)xjGH2_Okofiorap3E!#c0Bb|mU>LGVk-Yq;-6NlY|z5DF}$v5 z)o(XXd6N?)p#PQC8)Fr33w)fD;+{J}%OTqRGau(is>63AUqshjDa0)1_y{)sQ(fC6miA)?VYN9Nd|}JbN*V0;C=JI zruui#v^3W@8xDKVS=zuGh2jVy6bj`5&3NXuCwtolL3DcgEwF#%KlBq;%fI`sDi#6te&IQ)Y zUsuq18m6uon5sXIQd~cz+)AGymY)h6dVO~&mNWiHd4o@QI4Nw^#+!-mdDtJ41p-?y z(MjGCvi*irwIaziVtz-$O^zfNJ2vD7i#(M2mecjp_j{<|5}5Lq+T2vX4tWyo$zo-6 zpftna2ZA@`aVJwS<)iav3tE4Nt4p9B+$(-x*Q1McBCRaJ4>DNLEDyn#Gwegf&vba{ zqu=ZgNvFrg;uEAOr0isksHtx|fRqfe4#_SN3g_0-ea#lq{b~}h=E>meH#m67(WH%? z+!<->_UW_)1CY*Tg+mb@_91$#v=f+E0>F_X9$>7A-|W;d)XupoNXeA+ zWiA!WjLy;P+koEb0b6(4QfYwh#oE!W|F%Wl5gF_!Tp48ksJQg=r1Wr%?v|q?e|5~+ zi`u|wdMeREKb*@p>(wV8s;efSDH`yqPJ@gbx|?Hn{cg=(zO8|nt)aX+um+FU*Wxxm z46njJ>YDAKKkqHUTfu>iU_b-cpKF1leh_?Vz0X~YYy;PB!Vp3~i@s;tFInj28^M4T zSsiQtr=Y|E@v++)q3p4=JK1~-7%*cf-6;=_0S$y*577Z=f4eajy&M~z-HV$R zOnjHR5Oqg=`>+9el8H#Ef9Lp_rzojks|@DwUK;zc zKC{4AVb@M=nst;`ex7TL8TI}s_^UCj))D+Aw4ZZ1gz!%DaI3;h)XP>|nApBLv12}3 zKWALe*Ab32bHG^E`(6;7UJ^eA_{`Y;g4v!OT65Wh^Ps!IIs?QcXuWxIXlQ6dg;HG+ z7m1NQ?Mx0V8TavCu;MtpaVhN)>%ninADqmZa!Q)@%JyV7C28E~j|qa9%qIP!`j_Ky z(4+Se7e0jma%sNOZqZiS3k&d%-7LQ|@BSY}=iFP+jaw43P>Kw+mGEU5P zB%z(P(#d7RDv6nKirFxnLkQvEnA05Qeu)uw$-P`F$z@D7#wO%4wlJTJo!{s8Cw$oZ z{eE4Z&&N|voUOV0oRZwB6Pm1FaG;{n-}Q^Xj`3w4=NYl`{{SP6@MX^a{7 z=m=h!RD+lY>0ojr#A{aQCroOHz$-ywYz@Yo^`JqqYo}u4fyjQ46s8(#8`~he3}AoZ zzT;2cscRE(Hi)6$^nA96(sF-Ul8ukns^sE_+)I`IjD80BaKvBA+Amz}-!u|%+Ih6MDI?g| zXBRSfyO^U$F*oj^Ty&Q?EOnIlffEM^t!a9~x|=uONXt4?7C|iOD*(vXT86Wx2dk*| zy8Q|Q(MtZfI@>wAp#LEmwi#e{l(z{=G#zC6AP$%Mh#|`X(f&T=*q%aU@uC(V;#0V2 zA&5__=u>FGBYZft_OI364KG{sjFFC7>#WZKh8#cRtD3t$tIp;r(V)i`Yb&Qq1SsQM zF>^?^+j?^3v!bbJk9r3jAP-L-m6rnz}dXPOyAC^ zBP^NCXk64W>QNceof0eIgM=J1v0?#rk0bCPbG?30y=jmo`{&()_OU^^LgzJlC}XY{ zhJ@YpMD2%ri(701;irK>jqg^vz4kNOt;q4k!+}54)(r-M1YheW!ntM}t+Xm93vRHF z5qIo9P2CgioP=HcCuivv`cbWAng8OHrsksl_rNpXX`8^wc71F4Z7ExkPaoFTuGI-; zBesm_{%=V>l~@n32Emu(kcYiMY*dIZ%>s+IcwM+-2ymC@uBbRg1%LUhb0s(fpTfo%BJtX9XZgYh|YTRP!3n{G|`*F>mw>Y zp5>ja$!Bh%rp?s+=yP#Vy1;kDvt0&M&#v zB&aJO`{06N6V!b2-GS_D{+9`B#fCj@m>ob#e9%y8xjgZp(d zp}}FlO;2t-?2x!W0n9vh41*Lw;}_06*;xmj2NPfegn7ssUAR zp70j91#6@HkZa+749pxIcagxYrH~w6` zr`DxvRz+sVChY|i!u>dvb{pY6vOLl{vFjnG>#QPx*Q@dVBgYa9JMFLV09IcM`V-7= z$Itzea64wYgS=VxalPdpZtaN1_PF-%IMctl7NahY<}+U7!FT6Ok+p3Qe0~n_k!5xm zXqu-4T?TB$(Hek&?+7wNI-?BO!)9xGNY{~oppy7^=GuWAu9ZguiZ(dam}+|zAF0Zz zU~$_BJEqNGk@;OR-YX8-%QX8MU{D3zbXn}w(s}StV`_F4-jW;apB;KQvMzdQ82Oob zDs73+r46DL#*s!t?vPI&pd$&>u$>2Ul{2~w4(utBW4pi<9J*JnB=X+X!8L8ZiG&kp zI8qZd#oqUSNCQk;#v;p_CG0Uu6gR3$dMY?w} zT4HRYuNp-YUFIC()tUOsk&nPcsI6S98K_(4te@7bc0(@FB%5z8eKgk-bxs})5)nR5 zQr$*OoZAW35eAkWjq7>YWqri*ZixFQo(selLp-p#eLO@mdbsa|vj^m{#f=^0sDq=b zlO4~~z;w2QU{92GprdSp@Yjb8I%=e)^RJI~j|9aUkoS#VafB0^@MlC2(NpKy^=Q;m zQ(2kJ|BC1kaFVBkPE1hmjx@XFogHii6Rqi`IcAB=kP_BlGbH#f*olWpb=_3G4P5>@ zvQ93vC@@L?en`EE-*lfBxC~Q8KY^bdHQ9=So4qXp<4CrkM2m*9&-GaFzT{^SstnyM za!R{MM|JX^f)&sZM3Vg$H0!z-XIxTw+SjNUK$sZvP-ToaP=je!Ex@&8ipu<2pN#!0vm}53b(u zvhA_I;O{|?9%}-x_u-NL;IjF}ipc8d^Cy$vW&DZ&E;oZFR>uXIuS#?4;(t5dgnaYg z;2>}eOdT+w>H){krkN9Uv2sxO1x?w*?{B9cl8uZrggPh?JF9pG?NwLvO{AMi!>1mgs$>(qHO6lM&gOW$s$t*h~_jYk755R7UA)zZ*n?^?+4t@>|pJi8I!4Fb+ z<<+`PdfA9-g7yEI40MOkem29bkOa)W9vUU7r_<|jW8%Hxz}6FsBEy(SXK>XU^pH^H zzNl|De3$jIsc9(a&Tx%j2nC=0V7G@6sj#h_U-3EG7Nk6yu8gmphIn6z?VzW1fpj>k zSt%e{L1%Yx=pFv9WS!FNLfgvEmxmmeo#o5+qJIave9PMy<(?DFaRvWNp9cHoZUa3b zr60LnI66HUzhs`K)>H0K>F*LMf;ObtHx4fs4I1*8>?`p$w?5?cU*lZO`=Y*xbzdTy z)?m&VHDw35!p~Fe96&3%`V`}d)di1*4yRV$8Jt=Vyx^hu9BH;Tv$-l zalS4S^O4AM0f9hZD=*jVE{z%_Z^q4a5=aelQg@pC+B-b;d@a#7w1~-yX13}MBF(Yu^?oa#yPl1O+og(5F4e*x5RAI|q9_L6Ird0PsqND#7JCW>*5=Q3+taVM3(ETtm}gmQ)GHI&sv_9ur5aq3$;_2VK7 zw|>U$02<#3zRcYO$@rbW+eH>ky7vSQc9aVJ!}F4ZOF4y{6f3)xieq?lm;1 z`i4h*!JxP33xtR!Ki7bkfVRVH=y>{ELX_s#qk7rgAI)$M-D3!kfqoLevr0q@_qvJK zMfjqO<=suZbv6E`=8Lb|F4ipNKg&^X$>koH9TD-_VvwAknk7ecTY}*?Q?gIL1F@LV z8Gc)Rh}3S0BlG8vhDXJo7;V+1UAhndZMB)!yQ{)z=~h^t=bJAVrx*!3ny+)BpU}Kc z4Xg=MTvntiJvr@p2<`|iJNk7G71+NVSpEn(*b2qt=Ix zYarUnugc{%rzzEO>D*4+EoiWek2p+yB-4CS15MjW@qRChlMS(zErCMR;FahHtlXl( zpsJa8UI&I+X>4B?c^O%;;#q9(2EDe1JFs;euqkUTlN)z*ul@!)a^ocGo(>(GrHmjl zL6UNluZBI_K4WwH}~kyTd`iyn)?Fqd1+H!xil-7nDFj57OUeUBH(=l1N& z5y!_7#nOA=kyZU9bIGcr#G*!6ZQumc8QNlo%Rw`2+XhxX?+5ny-&eKcvN9(>Ta&2l|f zqh4N)H|X>xpo8yaTdlK9;)?#PrgCD&T$Yg|O--4`ucFRniZ`AFs?GyxJ>f=KdD&Hs zl%)T5fMN7Ri!aFIfkFKWc)`J1pQh%)=BrkuxXo=K8+|$Y2 z`->bQwHEgpXaGFNB);c0Eb1~5`5W&pm+fBT-Sk4fhjHp-RrFmYiTy9}k>K$SFP28| z(G46?9Aya7)jlxG2#s@H{d)m$FGoI@ z#dhY2tZ!r84eLOxmE$Mg$Cj30E*YDe<+#wj(2mUF!q*w_o}~!2D{{M zEi_18Qk3b!9=a`r)<*Xv_ns9Q@0nNv)N ztO}@lis`1KJ6x1=GJvEx#!-z%8Kdw-G6%kBJ-Xf=O_ zb}k+FSh|k=$Rt6$-5SBvjT!b*n}5SFH0AKn>^XfKXV2?yQ;o#Smn02fps3S>m@2m? zTe#{MQ~M^J0g0@+@%I0oSfjlw5IH7f!7#4)ttd1cmJ&u;h<=0IElZ%9xX66`LBInmzuvdgAZVfuJ8h)FZ^x_WZMOrpNvHhtOp>Ps)2r_48vjfiVI z91@hI15N0tw7J^Tk$0F+%C%Tj9;4oVj~h9$+g1mR+#c$u6rp->dph z1Uvq>0|rzgxu8Cw2065a204|lbgm}^drKyMT6G@`3~F}L&-NwZA8KXZh4T9k0D*#c zPGk=>;o&Z=0XT#Bs5QX#;lSPsiVYO^#DWw4ARA5Ai**ty4}|MArn>M z$lV}o%eul9qTl~E!&{l4UK|C`7_n$@Qa_5=^wH7>7GxNIl%Ftn+^06$8w z(T{POEGfbZNS^+lWF552%U{b?&AWJ>RwqQ4&SOwnoPPvV^8Xy8d}+}Tmu}JAjRrSW zUr^vT^_KChTnCBipO%SRXp0)L2ElnaH6=K>#-RwHBa>@by`>&_Oy?17Zw3B_ZMjii zUb@$KJ-q(Z_*cl@bnSmAZJn;5(qk(~mN)4!EC!P0v>vCHFEHC8PjHvv=bxv?b#3Ai z_1j$=h&R_?nm6V6rXQEoHZ>@(kLEf0^ShJJin>RQZSZ0z51n` z#iK{iQ|lwll!-E7^-A@cl0!oI50*BZ?<)IhSvN7?c7J}&OdmjqZ}-H*F9PH)oxpVQ zKOrr)Fk`XaUM-w%Oqe>~xh) zb|M6ZlIs=IQVV3-VIXye*lg<^S-g;mSA;`alM$JBi|4a1xGfxBaJ9lWxHQuS1F_V_ zviVg3bW0uz|CXV^@R!Z|)4cW#v-*&3vfs`VL?$nXnV&}4E#4@l)-!WK3RjU=PYNUA zORm*&u{mH#0*=DCPP08^sUvDaW{ z$3ZEy&5)+l_8-r-ruZmNXyA={g-)ksTf|Drmqjt73-JU~2Ga8nqGdJh9QdgL5{UHg z^%rpe1^;Byn)^e6LB6cK*>3lx>t%X<%x6utv@o4_Kf{_z8g?kmj&5B6R;JM@3b-a< zSh=WwqETkw&Cmg9!!4gC{kgo5|8`iYI)0<6mahy$cWIj?-pLTP-)?w8pT-ap&9_?} z*J16ve?lY3l_l>e%f9?4SGr54b3@6 zo_Z&TO~9R&O^PxvcJc>&d^zzJu``(j?PS>9GB6o@B>`qMRio!W1U!l-_mxJ!q`FT%o5#*H}(ORnG zi*SQ&2F$?wVWgy-xuyob4Z5HM``T)UT;x}IhF$6&n2l=4ZS5%aU~;o}z3}*Y?S$gmPmYmoh4C3AYH_{zdvKt|JA^|yYI<6gHNe=AAT)6;gA!wY<> zg#MU>$EIxS*bg#N~9b(&3pR9j+ z0r2j`X+U$$y6C8qw!~%aU){|k4>++k9DdETPii-lYMAmJe`v+pT%t$*2ve5dd)eX# zOHMS-l*h598>w3aqteRycG2EpnESLb1c4l!1M#%YgQlF19(@%PLVvo4bMoW?k>Irr zFA#Z2N0;PwM!Ea1y=9b+jvEB9FPgq9pUn{)BwE>Q#8Q#dk?;lnAPWqkV*9a3$}?3P zR)L=j2&l{55xyg)Q@s8~3KP>H>thEe7^3NC`!*0fG= z!eGe}MBB1a;vBfQO2jS_ZUYrs+4y|G&YsNyiVEgkL+pR_Fa5wbF(@|PcADDA2H<-2QAh|Q!kH3dK^GUYmWZ>9EkS~@Ra zwyQJ2y5L^3>Kz-IgEG7w>kTeS3A=|NY(bdTrn3O||N6Uj7lhuH=QJ*X{C2v{0Jt}_ zAiODSQ8X1zdJr(F3_WFz(|GXodOeebJo$+zLCk~Yo5HMB}QR#Wz z_}r1c#*}-+tNfu&;iI?Oanv6rz5u%jIMQJ?mpCo+nM|N}mE>6i>&$!ow;P9dF9}Be zriBND+m9dl%C%^$Qr(`I%(mbU&*nKwn=7ZduOdOzqE?G+u4nT=+;gDpN^{%LmX7@M zMyr0B$B+s1J|CKB`XKvSYf=v?Nb2%p%=a+Xv*%&3U*AK6v9hN-g)pSj4kb>6J(OrB zw3@^a^ujf+VS5^hrx)mF*k$t@=i9lBq+I&Afcvoudozrr$?+)RFd&IM+REFwhWR4- zvC@|OMIDFab(`o;_{k=&Y%+E#?YDGnr`N$315Ea%mE}NJzy2fR(E=oJhOwk#%l*--HgDaNX5yb@~OrJ3;bZLA3c`R zW;>ix*rWInyvmO!d75433A)eSiA6izl$JC#a#yp=barPJ1Kt(Dr0WZQvmxldL=HCJ zpa$hHrWBjz^Q~xks!L4-Wv-RT=F!mEx>g3NxpiVYs{*WMLFxU~`j+>j2uGmlwa8|~ zq?&zct;rn@6X1_{&SO>t%<3FuE4P)SEyKEF(7RXXAF53^*5jjuR%0R*5w`b(>O+eR z@vYa#ErP6@Es5SmJ;Ug-x}IIx`b^v)tr~UW>Yo@4#7x_Hp7rq2zqtd6KorhX`Kz5_e=yD(%Vq4V2rlCwFJq}4u zT4DUng_%34s%60io}GRrLwxsQe=%B3Z^-97QP!0O6>Dte%$g9(6UNnhh zwV+;%wTSb1sK~)X^vBlMVotxTMzlxVgyE!%&TXxwt) z-5oK59%nVI^+vEbMh`hnF$nI^BPa z(xmX(F5SAQA2&T=?8vJ!J0t7GO<*0$Oz40A{uS+A_WD42nE>N|&DkMa1q;d$sKSm1 zdjFuO*@ZIfp>abu)1I~(dQhzz2SUoBZMx$WC}~z{k$n0v9{K1+j&T+^q4;tZB&>s~ zs(T!6hS~oxD4YJ=&%XmjV#~Dk?O+Vpklktz6)>K-lO7LMvzJM`4$ld_E6g35}Xn|jYu$$sMtbBKa@1RG@)LUOkA5o zHI5+an2_rA(ln>KnAsWl=p#Yg$Vjoyz3s9O1iX}H;paWue74G8_0tO=OGfCVkzMI+ zPx<{gt1D68@pNoX-ma;++aNdLDCmz3UHQZip;MsN92$QwZ&tHz%!ys}WM_Blu`w`D zIzq1bE7;LxZK3fsdYAjrybLL8QR^l5WgCr%cCy$s%{h}Ej3=L|EJ6D|wCU7(#9aS7 zlpa?$R=GT>2Dq%M4sCxo{knB;Jx|)~S|P%>8pn^ig5L!S>_?qd@0(PEIJ3UihGC{o zm|%NK8zfKvZ{g8bbraGmH;x)1KdXoXo=q{er`KaCgpTNAu)Pw3nI znSF}mOe~%kLPRMK;Td3-QCN(MxTAkAZ5&vI{(mLpH4e8Z^1YLBNyYmm~)9{ zATwc#KMY-rwDcMxf|GrR$Y7$+nP+$PawSH6i`%JgXR~y!VAUIZ>BrKsZ+Zmqqm-R4 za&j7baWPG2L1ZRB6744fDJ#{P5(vWKi^->FcVB`%C*!s!T>VX1g!RX4bOTKkZ6R2e zEYI~+!N}8$81XR#-Ast zTjNi!dN(j`DK9W5vbkFpK%Z>#8u~CLRC%jaKKCcpU7~S$sOLJ{?72|z8AHn~ZCOq} z){-T}zYyZR78jE`RT&%X{LW2H#S|Q3gOTXo4a!Ef$?vEwY7q ztC2kRf!uNndZUf%pUIDc$g-Ro|AV@3v_3=*EofUz2G!NMZ&l+IzOvDdTta))F@QJ? zxwq;d%S+E+(1}l|5#s;b@v*UFc*x;QS#Y`kMGP9D_)Q1OX@Ia?M#p%|LrUp+_Iu0o zKTZ29tjwOSCD$Z~bU{71_An{;NV`af-5R0B@{8MNzm8SYWK4?wYi&P7&n$hN;FJ|C#-9x%#`;)Au zK-ZM_fMK2N!JiUvcKljAnJW4g4~;x)OOb{syN6axr-!J?3^NqAYZzkwy%w}KN$q|U zkFO?9B`^$^{?H2pd?O2h=Vl3D^iQQ(jA{Nr@fWD@O+e{O*)lV@Xad}56gzMhgtkc< z7c@Bpb$ofx_kI7=5{$znnAh9sEauMk@M`w-rI03+nIvV&>M?4X_~7SH^&DQvIg_c; znswrZIoI0Zk+P9H%@d^HU9Q$;S99Jb()Y=!_Oc+!Tn7p%=5{wJiFTb1U0MilxhbO> zDD*tD+UZJb3^5kJH5Dg~-LL20d63?UWA%6|jy2J@Ph*oeHw-kX;lTX!|HprTpPqz4 z#I3+}f!N<#>Ua=VpR*7e-nsTnWGxx3wz`vd>>C0eByAgB{bn8{E%{1*Y`sLa z9D)*FPi-Tv5`JVU-F@o#Ybc)(YvyGz!*=9kf=e>;VOYq0`?MQFRRTXlQw^Mu>J!wqSb^z2ILV6>p8GeEj5K2dm7k*<7lhnMH~*% zReJdO!^<-xvSjT?ZE*eua$8nFk+?4hd*otQ!jqa8Dw80}p{2q!pH z`77vMBkwD;(zWtTgYw*^LDhW9T7yMTFn$*M3;)_TpF{?LcseasWim7k_)KYPP?X4M z%^pebjEp!eWqJw6t-A1f(zL4gQe=Kv;vCiTAsZ$o$ zx`V0@oR?o%LCUWeCX>P54HDepwP)xVbul7Mvs>_>q7n;4A$vd9I4cBWff-ko*DGei z7Bx4i*ykH*Foeh$+A0Mk*S*vL?N7jDxzigxWiX zA^j8`$&zFzyF6@fS+Q~ zoD5kxPMqW$pm1o3$n<^5a%J=@?{icd3iahjX9tsP`d9y8(7bJs@6 z^eT*g&Rxqy;#rWJs^ZnA_@=f?KDe>I&izTBeM@8-#+Ek?u*~Ic;I*d_)p2ptGBgrs8`vB*=Dq!dJFE7j(t72 z7}|EQdh0qwTEvwpohtnWDS6pv7~)}hXhj7le*jt~-4zw!BD*FF%$&_|gKFh#{=Id( z0{_TQjR{glM>qxIv>je}88_h*FQlT`x)9V0eH{QjP_8D80WZQfvr;}RN)3Khp58E8 zLP&nE+>@)-(=OJIQGMrE3m|a}a5H+0iX#^k+c`EWz1_wZXqoPm&|VKtM(^z;$WO;f zZW$9K>u07jb#61MK6wHS3tdpWB%&h^%N7=rT;_P)tFH5=E$**H2$@TccAue;pBX1;<4OMR>Uf;Q9d~j zgSK`eLerJ@f4AKLvb%LU5%qD5j+*#Qhb<$k+n>V82_Zzwu5gysiHrCRdVm#7$CFc1 z=Fw9M#6$otUsfzO6E-jYe6P(fW}ogm>TQV{Oam->b!H}0Lcmma(7d8bk-RM8ujG#w z@R`($cfjj1C_5tf=y4?5V3Rg{YjfC^67TH}d!<=Sy;~X8Co&s`wjJ3DO))WR zK7IGeS4-`SWsjw#WC9YYZrtQTchf<+sBf7WaO1&jDnBk_v=dJ@zO1i(neEW@fHO=Y-ZMtN zoN1{46TuQd8;1Ek_i4^{c$=>6p4V-%(x?;B{r~M);(-#2vx#kAE&f3KH?+ChHF>1j z2Q(;2G~ZD$;wa}Y!%h!%!KADuuvgmAtFRyPU#mrTZFqW0zz zza1A`ZsvBU9RNVD!0a>7UTF`O2>#&bdF#%t3L+#=USNaMjfgdX73iuC46#^FrXX+&+H9`G3a`?}-mZ;)5*G ze^>NcjemF%#`)3TtZHrh0?x|qQ<~E1<5yglSv>B0mFY(yty`d<63}`|<~i{x0ckmk zF=`fcF(--~cK6Mc79HCb39&z~99C_jU{an!D|#I1{A`Gugq{Hn31^{+!rL5G79p5F)p1t$8R=$u7e3cLpjSC!_K5hZUQ z?)vM^z?MFriYxL9kM0bLhacp_?#lOeyI#bLjAYZEpA{RMk)V=pmgu~1g~YrR5>)r@ zza7L?l2+r3R(wea4@|tq1~gm7tbqoCX-Is@bbq4GQEk^rH5jDAW-d|6Hg13+BIiUY z_Rg0+@$NS{3EYEu*D1~C z*A>}9#_3c>GeLI*8*Rr{^~=_9nl&txq1^zJdcLB%WxIwsfh?lj+xT_R!~|yJ^KgH5 zc*$xn@%MYU7Yo0e@PyHYVT!1$<*Va=+~!fwR#5g%xjdUjs6QdHgZ9sd8^7RdUzby@ zJoEnIhBLcbMz(JGXa2Xtm;E%{r?YT`1%O!d2@5mMI7oSYTT+O@QIaLQ`?#@C$TcFe z!!+9l+H!Y>M}57t2uk zg9M}0ECS)V8wR3EJFQ_fQNqd3rms17VqRmV_l_I-BnefgH`x6Lw}JsG<9nD*6(v+t z1Gw026`iErE})eZ_7L`4sB~YZGg-KL6P?HECDn;U5^0w<-KWXp%?46E@LBbzp%Lo& zYCkXd3jnA@Bv7rVY;=`|*BDE!`Z2l?|4Fyn)D1L;z`W_p;T1Jtm1;!aVP|!}t=>p0 zg`=5w%pX7Z)*P9S-q#E&8P#`6EDAIH+cfheJxMZDJq@<&rbKL_(W-r=OHvbIEI-)|h5hAMYNJ!|XXz0?vBR#zogHA=swPE`k$_ z+i7H@#RYAH2ej8Kr(-D9bE*5Jrb|E2700Dj`JYrrx3qvi3qGrM3~cQVemIqu2>(t- zT>m@#(FFcHOHkYgiIYx%75`|uDP#8*-|{>NY=>q4Sk|kb&$T4nXAe_^ul4tB7b3*^ z09$Ps4gnA@$}@jP<3YG5JlU666-KtYf3dWvsF!#yE0Akjep)^F*eA^63+|A2SnMzP z`kKXl9L1!1tARgbC%Yx+mBG<$_tgF_IT-K5>OGUsJwIQ?q4Cpg^Izl16&Sm3uW@Oi z7eQ=d`H45obo~(8(zXYYc4F7g6gYZ60dB#=^#7`KVH17vnu&9)*k9{p-Oq~s zHXi;0?rMXeM3}JRl>0rpL?H~Eyin7QOUZ6K(u^zyw7Xoregu%E^P=;uGJfUm?uy{& zb`P2An)SlZGJrOB8(&Pn!fAN$1!57HhcJ8Fri5$_yd_K#Bc^I%w$O6d4vWU@WH4jv z(8mR}qco7e!tDruJm5LGV1ca0&)781aYOYaPp8tq?OR$h3vYaz1baGfw#awo2^35drpio$GKL2yNp&mZ&O4# z9L)f8O*OiU@Q=nJqg{x&)}7#r@(I;Yz851K);9v~HM$fPJ6L3ryiGGpSlPz;7?1vP zAo@mD;99SZvzk=rXZl?Kidlj};#&BmeN|MO`Wqx! z6}{zY_ydN4q6*wlVBO8rUk%!WsDiP-&}0+;+6Cj_JKtRP0eWR+_6%`CH_BqMr!ll^ zq>#fNS^0!J<}WDWGfHjqt2Rqi`Mrhi{pI$0JTE7|QU5{i*qL~1e&eT7U{!1&IMNlR zlY!7Bw5h|$lF1$Z2OTPzBPCT2p9ed`_= z)n5mky&j7k*|H&&eQvdu8;Np)^B3r!ast-(1rU1@q%eo=Y;~-^mPL3Ldwr($Ss9Xx zc11=+gh5U>#P+}4E?U>nw3V4Q+?Ax9#SU(f?vk05l^ZM~W*Rkc#DyELi#jzM>AxUe zUVeY`t9prT3Hoq9>{^}Btq$gruWzZmvUL+EI*=tXOM>eSP>sbc&Pu8N>DcZ@7p6^D z9*f@^l#qSBZiKch;3=hMLjK7!9lcc7%`=h$n@BgT&w%WljvKE&z)l z?YMP_v2!{JA+>lpxUP;q+@)3f+~2^rRyRZCB7Y# zhqQl{X_VH8aZmMaMp#WX-wTlU1dvX+_NrE}1j&h+^*lxBy~_E!rH|8lX|9nbShOD?oAOk9WU)TFxSM+NuNKdNj1Rivl1WV9|-|n8Mt~geVDS5v+ z!30Z~fP7AE(@Ubi#-V#xhgr#tHNxWYyG^a>fx_OD2X4)k;{GlJ%`Lc$mYuH}#3%|1 z@%rY;$W-NxVQq(DbKJp+NvxYwMCc~FCvTLW`3s+uQCR5^z;%5nL_fTHpR4m=Zz|PD zyEE=t+<02s^bVD<73*`H$G`-gHhiARr2 zOo9^j`%J%Y;9SiV&~1+ur==qNKbz}WzLNXvc(x{&dM;>65L^t($D}d7qS6awHxwfg zVRmC`ekOBG6OiJ*W8>-(%rUA<+G~BA|EEzu0%fpKKQi2gk6hzPr{yIjBQ0Ge z6Mw6&5O=>PN00r5cR7r+FY&kNgU=WEweq0Fkv~ueAGc!wk;`m0(F8C@sk55<&ny8QDrVe zbTBkDV-Ona#<+OZ?qOz7l{ABR)}sl}F69ulux52=o1_o*y(e7!8G;%^^mavbz$orP z-qOxV>W{Kb&s*I(p3Al3=!2i7Qe@m{Ol(WI)- z-8V@fph(&K=2GXR+UILO!O+w0K~$W$?SP-iv`N-o6jtaCVA|(AM%E%P8|e7eBQ;7H z(Wjz)Ceanh3a{SZ|3a_!c<+5yj{epml%L%iEU^ySN4+ACe)0*ZB$+l;`45^a=%6D-<$orv>a^Fr|5`6!59`(FDLh(>` zy+9nKMq&IW-(_4jzi9$D7zsKlKOR5%%m2pk41tToyYxc0mE}80VSK^qM4mJv@RcQ% zAkS>q>3N;pb`S_^_c|i$PhHn==+LsVNH-D{Nvy-nBjW1Q?C7t}3w$Dq#!Xx^M0idu z)`41a3^am>y!%g6dqA;F0u6pqHW^P^8H??5aizm+YwSv@b1Vw*66Q6VTP_@iGoJ~m zS$A47$p|~Oy6M?@fB+bRcXA-%THz3Ma!)pJ-fxRYYKHu2L8`<&@=kjYB*og{%fq?yCw zV2?Kj9imwGL~8_9vo#-!4CLfgDzOkfrJRRZKVJaz)3vxd*@{Zj3ZuT}pa7W@$r|Ht z;7xGy^{aW%l820VHv6)$?P$~iid8qnK|C2B;t1CApLImpKA?fMw`|_aY%nAz7cO`g zF>wo1|0R(6B%kqXl4zzlZ%?j^4PP4(l-$1p)13N81sZYoVf-e?{@bDPfbRmaZVxqP z0zS>C>4-|yIlN`P)7_)=tei2YAjK1tJ9$zq>_3LeQ(a}4Yu}7+8OpU^8E8|Y9pqUn zPeF?ULA4WY{9EkRx=g7Cp#UHLKI(_2SmgmMMOqaobZ%>?Z>`sLnMtRIl~Af6FW{H_ zR2N?6IBgeD)uvO(UL!2-h&f|P@5qf2!@-b`0+D*d52I`-!Ma6vY1CT#rfMLoq~S}! z1LB2aJ})*6#MvFVIDOp_^A^{svwyG&W3ZFHkp8%8AD(2~JR;?hu0y9kA?iq4qW0U81q=PNBd}jaj}TnW zdGe~~IPSV& zqzLEMPpN%<7-__8%_S0vM*Z3U+gnVjZ`YdC+_EJ}4D^5xb7JOfV$lK}!q_?$rN;|F zVZ8=;*LbGO_jq*N#id??$qOz$&ws30=V>3ANIcbZZ)^oLWJKQrT+b%Jc30p4*PhkI zhahAP`Y&DAGPm5z>$47tQ}o<4areL4H~cZlVJ$ zEr&-&gK^hGkW${dx~}J5)L;m5{L%v*8%SU>bXP*Kwx zRnYJTNN&$cEgZ?@n0SqYc&Ajap_HbZq()t|Q?Z~%#&IPGe-XT+<6iZ4*_}{qOD>McHXg zNzq#y){z=U>jy%EA-gS(PRX7wOKf(Pec^RSnGZ(h&=|MAZ0c1ltSMd*f# zaZb6MT(;wsTT3g-Wy30onb}v2v9Z&Q5Kd$}agba_%)&Nd$o)7HHnwcJY;rd?F^k#x zeSZG|HlO$V^Ljm>k4HQ9>6j)5)T|b2JJwX;D7!;+6PuzXp`Vu9Yi0*$z*(f;z_Xyw_UIZ0UK1MOnXJ}xmlwGhR;q*fOj~r6)rr};RMCRQ&HiVC1fh)>Q*5x zBMG9WBoxmz_{y+Jw>iN6b01m|KL!j_ep*V#mHDGK^r!P6pRp4u2$ZVZv{%3Bsqi$V z2NSj+2XWiQxf;8bzc)(PJN@^6`>@gcLIjg6??;gZPbX2FP`LjX?d_a(j?+_e2K5u) zD@OPHs3Vo3ZeR1IBMq~+X51+@(h>L+VI#}8xK_*CQ8_IsN_}Z$hxeH75H(*f&#;3` zejxN&IOt5V=FG8Tay;fep@f*o%EPs;zU=H@66Ck{w;}ha{xMJkKw_~m(uF=LIEnhj zXC4MXY||u_(UAxz+kUYfY>|=qZIKdp>#OJ1Tg0m{y7oo$K;L5IJCrZHl`w&UOn{SU zev00XjplkPkmhUwhHdnP90r-J+bpaYKj3m0ehO7E647buMD2}A;j=iuzvV~Ryqak_ zpu_+SO|NFKqEVD_R8iOM)ofgiXb*l9fKOOz+i=%5;^6e}x-{0ZeI7Fs?y(x~*_ht{ z(-ax|i@O%H1!3^GL_zA}tL%B~NK7RWpenkXlaA4!72M2f&tWxTsGnWHX#_$fkgV|N zIlVZov9@Q&LQG`Yr;42=?{=4>1?f*qmA>z#yB|}fl7de4z$-OFR`JkITZhrg^B`W6 zTrGFhHkz{O>;@je?9gUuBF)SSM$+y)s1}NmKW+WJX1K#imK&hpPGAt5zc8T=9uUL+ zU{Di3iAMraHY42|z48hlOKWdN9B4eye)OvhRebnT=Ka zExh1dXJcb(Nt)%iz=)Ou!1@#7jXj39o#+3Syu|(m@{ndubTQ&RE=S@s1!$o5Bp4J8 z!8bNsnyb;H9-LUQ4_mR`zJvTH-X+yuO9Xi|wu3of4-+Bs%pJh$ZbXnsV!h&k?6Nhe ztwMfg{4%GT-Pb42uLdmfegL-FxApBwgj)o5A4QF1me4JgYorI*@DePDy3KsRSKB4f zN*mhvR8)+n=w$lNQ@p$iKF<}wLv8vn*ueN>4Y_7m+kg9yc^|SSkKaq&roNmQpe4;u z282c3?FDN16tykGuAFVQpJD>gQMSn2GvyBc%8TT95n*}jyaXX z@nYo#P9&Uh!igYz0t_?a09l>kJ}boX{#hFvIC)%s-pkgY;C9`zae2hsbI64lPG`Jb ze7YLyp6_I)c#yH>=ym2-SW1AQB0`4km~6`dniB<`y)8Nl=x^Ac<_`Edm5#ZW`_I{A z^^$@JRabIe!tajEy}LF>T<7)1N1D3p4}S@(fIL%PpewT4;&vDuSCTtD)x=6v#6NEm-OpB$;>uR6$J!@%KLU>MD=6jea)qX-%}R0YVS183>dY@fDP^eg&Ir-PQy(i_l6oA zYdqDeGJ&e~UD?5Ug!;O92Mwip!}7m*ZYFHz_WIs$X4gLq7iFHohFgb!$`(~PxVY8F z-jl9(8SeE_+*e0UHuB6wb({%=+wGdc)fdGs=E;%rf?8Cab+nw@v^J)AgGLLltYqki z0(-ERiiM77cvp2rNd9rzVIcm1%%~G~S12BJjeBs{VI}Z*DzpYj?#XTs1>NV6C3Jeu zztA(e{XV8;TuTt4y+zFmmI0}aZ!Va~+up9y@9KJA-#l zG5e;AV$T(;$C&zm-@WUSyccG zk;nHX7~)ByB#DMWL0M??9u8w!vlB#=PBjLxMRRvN>>7!+q(g7!U?oR-Gg%ZEpiN%5 z!nMkchk9b1CZ(9PMjy5(9L1N8IPR+7A@_RcTUBtR+G&OSZ*1xNbPIEAk`*>Fz!>0D z^unHQBqY7YKX$Cmhwc3mYGBLcMD<aIHx@9}lTm1-O$nx!tEe&v`Ye~kl z^cexrZD=BlC8MSg^{eK_Yj<~{aH(lV%y_S5#N!SfUUa!U&BbHLJn&^mmQyLJCZqPk zR_(3J%;wD~8sIRu{_~Q~aQ!@bzaECw9B;r1<@o2EcX7p}9U-Wf_<#B1Tm zXH;Cnyw})8E*nJy&@!UNzm}9xGaBZU`I{AiQ{-2h7dz9&3bEzTslL51hc6!tgi;q# z(nF@dr;CCn%CpwcwUnY%#)E7dY^XpzyEHF&Xa9(mTD-ZK`%f)fHPAeCmm>k#7IuH> zNlFQ+?@gL8vpD+G7Om!!K>FIGvmo&xTDP=B)2J68y#g(f*~j@yKINi+A0bb6;n#+1&n4g!ZrXisHs zlfhD~R@E8-bu?W2+tqkMX>lr{tl~@Oxo*FNJW{C=BD>!>*%#bF-Y}pf2m%Txbfm+p z-|%$-(lJ_Wym?kB3hp!vM4tt*a+wnq^t2@RQwoaHrviiKaA0-#TeGR`9z3$!;4i-Q z4Mb~csOHFr?cjpL|2w_n8-71iFnNu>bF0vPdwD2n|Jt)v34EbDPP;{n*(E^J47;=` z6u6&lTwmFIj=zdo=_VIuvru0%)x20Kf-V6E=3Cyi-SH|ern zdW@0j_zf(sUKGEeEyEq0r{#kTp>fNQ1x{V>$|He;>Awa4 zS1QyU?N=?ENt9StENDr$@>bzt2PlliahtoF}UZPyj@~F=I^iwc*raEP$kd@Ti zln()eO|F0IOr1-9dRV9!nrot6^gUSa}^M&@|hJan;RX}|q* z`APobV8G|^I?<9Ki11$iOx|PBc(O_l2(oslY-FJ2_JTu-;Kc`4QXYs=LDK#_7gs5@ zE@v&@V&8l9dg#y;jjqk%xCeauI1CmZ`&hjVT81aw;DGh!R2R5dI`MA9`oq%ZpXVaR z=im<6%_SwUX|xx7<{9oVcM*3sI6oFkFHM=ep1H@q{a!9T_&qSvG}y^j#7|^w!-ubT zMmL+uj7oC@#S0ihJ3s~zJWZ@Ht|wHu&|Uc(G)7F^;e-C8Lbv$0HLZuUx+C;z{roGD zh%uC{O&X1D6DZ}vhRjcLh{2_-A`CrLH@%!b+dyzEyA)VuokvHE`=jbEc~JUU*jklp zM+&q)ka3~XP!Df%Q+6Bh4Z}I5v1gmNeFmPrii?5sLjeNM(VHBAxU^8CX@S54Y!oN(n8Q0HR)*41D`v|7V}xm`EYZ-Xo(=v;DjBkccb-ohws5Mr7 zI>*A!o$=@P$3x(qN(ZSXDyREHUg&1KJYS!r?bjD?^NdN`?_kl+Y6z?SAP6cE`p4I~ zs<@e5(<|S#fh;r$|E>>Xk+nP7U51v2Dd?L{*XjmQUIv;{O;VfYR@tBP{l!3^#LbGJ zmU#bNJ;LsAhQwN~t%D5Nw2VMHbdy`^Sv8wT5v7x!+I+@_Z!JipcM3;$cj%H|VYEvA z;M~KU8;e!;3ZLB@)`U@TQ<)`Jj6x%>0x1L zfF555NCPwky1Z^TNvxCzezoe;33!LE%m?XZ^eUmrp&7&t`CqzhMNgg+u8NV5hgcamZDuzY$@%0fO+H7 z+!=|DjAnl{*e(5PU?h!c-yGkqX!IW9xtkkH)2ky&nzDhV=W4t*eiGeF05D*yoGkbB zcq^Qp?_C0y3cMUs&I2Varg$(52$_KE>BNgeRtV(N0$rNbFgJ>%DQw+eT`ZAq`l4I>q%o-=voIw#s+crU z#6U*o%Kc~IN%HI(Arzs%c; z-AgJN7hb95zZ?Dk5jz_u&U%fC93Jmz+MSykbGY724A_^^ z_}={L&%Q2)feOkSTIX8vLB;*L7%?evd^g+uEifM9FtO_H6Yfr{hI>Ncs}yg<&7QCZ z0RI-+_rUOSJ;zM%7F6$5_$aYHtS~4M+G2OfXQ26bNQ=HD$h#xKF4slzN3~P={B;Bg zbp#HRICyzI5-iPgHvtl`E>s(eR1j}GMEnR97W71=IX|)Ks zLs2E*-coU;{=*v;;q@4_fBwv^#tipHLL+4u+*|5A9{xCP^gEBtsVullpM)J*2+)zO zrvdG9pzkce_jK zqW&_t=b?1|+r&&ezWKC@57{#{FezPs0;)QN)lgjU2*Ip@%;%?HByl`hpqlFCt(fl&SpU|;za zj&QP!1e9??D5;F0Isc1e>HN=~@}G+H?lYcm=dbdl^hGK0qSUiPQ-Ib~w6_B(*GB8Y zk~xmz&BnKzhwF-Pp`J41F>HQ+DlI}-qc3S~lr>1vG3DyH(rC@CAlB%OKh6M;=!`b96q^pvlWk@DUqgaNgImXifOOp`o4sL93s!&r zZRFfmLCg#LI@1!QW9-)uRt$D+SM_|MT$_t$M>YcoHs^haCdxGtF&<LYe3_(Tx4cmEL)@ILgUmsX8=*L&zoTgbBWPAr`Hb+NNU!~Q|6&_7AeSf47=IIm7fNCB04`8k2$EUXiZc%2feHky94^mYRc8h`OJMeT;kW4#Oj&mqHfgRDz-{2sUE_z$q7eG0 zQLu9Om(TOwe6`!vJq9#03XA-r2wEvvY9RAs57Ov@rQPVCssdL?&mem#B4lzwTNZF` z!KA=O9~^3HTp&i{sWrI(j)j=_yXna8Z0g84;K0{?sY@ydx}ACX1qtO=pCbv}HUrjA z_-D*OHl9eY!s&xI_-(ptIbR?tK{;~lA+&9CNpKLVY`k@gQsyTA%RDga)-6N{o0T?& zZbZ?egsg|5vxfW4qc&mB(;`KE$PQlf5gr42^nQ(W)15k>Q7mwbMFGE9+eunbIWzBh z1d1o88y7w}IucGCuBjikpR?*ew4gdZY>%<{0NK|*>3-oV0~2^%W{|c&wPX$qN{?M; z^gu*L)jn0hJWST2fLZ`b`M`frw_%lO3J<6X;wUr`21f_iAPVP0hWlE}dk!g99IPiV zGdcI)1C3|xhWvLF!B|O&vy369xXJC@FGV+_hc5%+DEq#_3-yO5uRa+t<%m-1I=q|F>;h zl%Xd6v1oV=5SMKiN0f$o+djfS;(86|Ud@2kCtV!?Ea_S94~2_qmR_Dwtn}ac;{@9$}VOu?ETzp4nR3F15b6wn;}GQ_WNjc-LUo)UX(t2trw= zDJ=pU78V4vf)9GB)Xr2q$#FOFkT%tIf-sEn6v!@S9$d8qIkb>DI=g{pZhei8!{=W6 z=Ftvgr`|$wZ?XNgFjZ0Ona<~7k&&1OKNLyts@y&OI3ln;pN^yJp(NB<+tKwWx$fy= ztw(lS2J;RO&mm*XeRZ&EiO^$llJTDH)((py?9W#PjR%)2QpMgW%2Q%bnVqQX8Lw_V zhgg??qQFfxGzQ3rSvPEkCPl`+8 z*F8k8wfk1z0S3LYkfERdA@<4xGD*R@pKp3DKQ8AdSW6& zTb0{nGi~YsGr4mVeUmw`N(=0Qe~mFPXhHrsudYs}9ECQW zTN?LiX^EAvHywU?liFX$naV%2bUzWT7P}PH!&?KgLo|&KK+*74V!5aZA#yH^zppaq zN=^)|r9Q1^4xa?vXH~*bYt8TS-PgHSb^mscjEs$eEC+dIt$(ZgOKHxH!tPR@RXA=s z<2?=*!DJfO>zFd{RBD5YoAYUojIq1f!gxQ7F(G#?&5n)&z#^L$C_{Phi&vAG7tL3} zI(W)u8Mt056!DniO?NP&@7%JxlviFJaH)7KbmOJ&)E0wuA4tT6Zmcf~5^{HBwcf5M zgIfvdpBdl9jK&t<-yGCL)r*s_Y=)-3Brh?Zp_y)<`IbKf^0fV1{240HC=F0;50JW4)KMh<#*nW zxCM*V5Anv9+5~btjI*3`aRh4~_8i_c{O|W_tCt7Ne*V&97ml|PXCyC^f|0sJEOYqy z;Az>~)HG1Ft-uD0q48=RSGe;62W3Nc=Ngk)10;%DsXo5KQ76L}Q4Pd=kIFQ8y4ogs zBD!2!;Z-NYEkDEA(Y4SN-3s#;MIkZ$v``AX!?Ood;%-a<<*j;Cc)jw9i@38>qopnw zl#wK}_}d=@Nn$w_4tyHT+OxU=$-&YsKxD6kL~fN3sEP#r-8Ua0()J5|7%MdMF|HS| z?qlx3?P`4Fenk>ZTX+>#`QAFoLxC+FR}fnBcggONZ|H|EEqlsvfAep+L^XUW*D5V9 z`9kZ;j)y#B`Ya`MV-LIQ4^>3oQIT8jV;O6JsubI1_#^T1fXWab*=k_OP?X35p|!lV zDWIef7ykh2#=3?L)AMl+g68_)y3fx4A%3Scu`^r?K66(HP-|=X9oXaoMg*%?)>Tm& zfCug)Yr~b~;9J9%(LkiM*7sYb;iknk{1jKP?960Ct(+R3Imy@Ra~FA6e>+dq4-q83 zuxRgpX0RXmBk}yO@?v=i+$B*7+bSyVeiN$A(k)pV?Uz4qHY|XBBVP>n09`X>kP%ww zRAytNS6UoNlL%JCic7NE5+K~FJAl9^^+T#C1PH8<;rj~#lgx_0B4HDjakFLUpvR8) z&bL@(BbS1hJjjUoA&1{f3npTlkK+l>(roH;rDLI@5yp~!Zy2)5^WuCuYn7&RE61+M z+^(DJ%M5-TkbWj#mILe!WB%+3p&kAP|02lh8aqc3dp~3UGI_>4w^P1hT~LhtZ(FCP zyn=LqG3f7$_8cF9nW>jKM(6zg1+K1*KrLa2bX(K+jFCXbHn2JI!Y~eUbb-ekJ336f zXGR27xuHtt!Dm#w_vBCf_CbUjCZZS-3bm9X7ff~kl@D48=E-1!-#>LfZ)IvcC26yb zWSjCpAZRp`$c9ZgobT_bZY}(*xOB4~b46rMYwpErE_iz&eqZUv*D#wHSs$tyy11Iw zb3G{aF_-5SQHFaJYEXXj9v@){TCDM`!B#}8k8Hd%Qg^jtFSL{FE-?l;mK&p#dQt~w9kkW~r)&0%icRoOkRl|*{ zL&REX>_xC%89XlQZqTY9T63_r!`9Aiq(C+OJMBhXD{vn_aQsP(UV+Vw`0wPD0^ghGQy?{sA5{- zsWM0k`_$`Em^l8@wMK?iRZd%=)OqD7WQY1K$vHc6%zD++@)&STo_Ne=q-67)i&6gw zm?KCT%9R)Mn@=k5$ikUY?&YTO3H{%@p#*5IGe_yl+k+|(_2!WU;l&hExW8M}8vZuJ z-&*DD>NV4wICB5`enWQ!PAY5@`&tkIJQo;}XzBnF@9ghX_jGH_5r5jy`}H2;neL1S z8Fn6H@M+!@wdA#n_5bdI_xD4-)~r7R|2Y4Pi->07Iz(}CH@Q}uXQ>Y=LQJuxx8BUj z`b!09Uhl5LNcpQ%&qDdFw>Qs!#yq?aMJw1OF|xDwC=mI|01!i4iwU|iSRw;4dz9(S zTU>R}@p&O=_sj1&{%F&+F>yovBDIcdLFr!+OpjEBTh19}lmR{SjY)Tzhtk#G#f@)& zNqm2ZQPJnc8#@A@Maf@u@pzB^n{JU&?rX&z-)UD9!obNbKT_{cTs6qg)+JNZquRvMIcH$yLA4AsgFM@Eq>Z>F-0(i z@QHv2oEc!7O0zn8WOZWdbE^?rR7Dvvjvp7DTdatdRT*^o@ZE<3hn7gpWl+W|P|c{9_yyz**O>>+pBNk5AVuH0ScQSX@t!^qxDslJq4s8oI{{ zC0MzLL-mp7cs<%w2>kOabS6j{DlMQcIP;soVRr-XU7an%>qQ@$I{q20%}KNVZ<_~laki?8 z@9K-wsb>tW9l@41rojpI#yM@a+eDaYsbu{L7MQR+^F%ePN$Wb!YFq3_H%KI(%J(;! zBsY(M-wUnKdAbCL^Z$N|6*KRraAv|dy{Jh0oF~-*Q{Pob_?W0)qsE`yP@uc=4TeCf z6R4csiHl)XjDAoF)B43+-1C@-PQXzSSZZDPbxw7wv=E~;${KEYS|w?^Bv!puK?*Lj zY83urjibGjlD%W%&BYh9U)KYES~ol_(&3| z1yE9Y&deWG_P9x)x}*GgI9CF-cosNRY3YNjXU-i@QXSm<(L5ip?Q&+WCYxLSLmPda zITk|AjHMzL{@8y`V}}!A(|v^aD~E_ci!%%qCj}vEY9)h9i%QQbe6Yu(E~m#nyp!o8 zG^VNP;T0P0%#6^kM(D@P`qUdafKTlT(3WB!6K_mHo+jw>%_h0_c0m(oPD%g$G6w$= z5PHZFU1FV7-e1~|=X=`{qbtlSIa|4*bE(JRMIhE#OdG3~s18wQYXQGGN_CunTq@5J z(bO*2W?kbMH}H*{RD#Dob@j5{^51)ld(@w%6+NIwlcfs-9QqeU2xEY4Zo}sR>eLk5^41Dou6|?Nj!~dH$qnXe6l|C zx*9-b!U6G}jSlb;DbYY*mR>DR)2B)?t2usx%UR>dIe0X!qA$LL_%U#}IE820m9iAY zPQSmBi}|8$P&e_68IOc2u(HitK-4XM|K<(p614=i5FZm?(ym>5FUR32vm}Xf(zlKx z1{y;lGl?b?*rO>-QMx$jarRcO6$aUEpe31*&g>t>rTnETi7Iz07vm^S&cZ#)q^EYt1B5SfG>AVsubr?&N?Nq8uA8fW# zbOJPFmY#m>2z(%1b(X1*btT^Aulz$(7;^G}3S-i00= zt_)Jg?4ZX)(-FWs;-lsRii|dh^mU<>mh{9z8T%R$!U+hmu0{PXG@=uGD75#`Xg3Ko z3>b;?R_r&Ln9R}VHI89peswLVm)rAXq}D#`<+eEB(FX+D-U@vck&`V7mo>c#1Q71v z_fD}Y_8{C=eVC28XeaLYwYpl2t!YM$0IGUU&M@keu#&o=SdDRfjGF=_%k3n4*;1YZ z1LY?Hn}~GIa+_{lvb~Eu6?)P9E-U`z=K8_8J2Zr6N=!A`?iKCmokaYYKJv$qS|9wQ z31UKOSF@Fj|HrE7Lf`(8%AjgMkcq}KB#u!Fxo2m9W9Fx-VIIN#E~1y*=(EIuHmbR4 zCX#!_)JNt_td=qkxeIR&+5cIA^yXS_fiD`P5j2Y{^(%my#Q@N5ln-XjJ459aA%H=!;F45j?hwcS8~E*jO_h2G(Kby^``MMyf(Ubb4d9 zPH<1)&Nv`+G$`B=m(5GV%m^+EXjx-duJZI~%KW-EhsUe?(%w}Ied0BitffP(Uk4a4CfcJbJ}b~ZGOxcgedhQ!IuC5JgAVi0 z3$04{f;Xymc0Tt(z(`vS)QohOo7+~(Q5geU7ZiXgxZ1*5WSq5v9YmnLy>!VGvA+Qz zTij!{X9<^VM&3Z!{c}ooB-7%Z(6!-7Gm*@8wyt0lS9Cl^|m2hHE2} z7f+s#-*1OF%gC=fT_lL`!@WN&wcqamlkQ}(TcJV0yBKE;8*eE zl7j96E#e&!X>oj{qBRHEKKG`+wxQsrB1zN7FC^gvPc=$)JErT01~Idh3vxoXYwa|v zM6U;Ra=gdNP#tXV?^fxnsE~M%A6`pSXJR0}1pT9CMArd@H_WU4DPzPEfhvw3#=U$9 z64yU``x272w_-MD&1l#>yl1U zZVX_|aRiAQ?tkCDj0mCHJtyD$9!G*qvIf}^G^LviraYN{ic7pu7MgH;a4Cqx4xEe+ z-zA-98fS>U8}W|#!tNN=DNh|7hcjL@%%2T3ZI8!n_DuSkPco~N#!^a>Jq|Tt@fZlG zJy*nZz}H?j>kU>~782Cy$-t~aKOcxU!49Kz{#USildRWc#P;`=*jQUk#l~gUQOq+W zn!trGsd@V_+g-NoODNF9dwOFJhX7{E(Z6u-6E|-S675P^sWGG=t+l`150lUF_t){l zE5~P-BfqjQ{NZM_-VUJIqsKK9_CZl3jQCeekmv8i60;?Sdtk5h_nty@*R=k%GxisM zY%OvlmVL@0-l%;WpO^bjUTn*cc)rh|N*~|tA$9s_i+o~Fs0cte8_CG271jkbgb<0C zt3_P0%n?1lR4v=N0_enHMhvO&B<*AL3sA?wbF<1TQs)<9`tCW>L9j-^-2d=8zt`qo z9=tsOBqGE544SYH8g7vp>v50gW32y#IWWJnPF=eRSJ0(GX9br| zJgiXh51`j`ZdGNA&`l;D?}wP`pzumY@-pc7=FJ;0hgEMLAa_i7GXQh=TK)Zijqo0Ey;0Xo$-{ee}mkZ}MI)UkQjoIc{ zzvnF$qRaw3P+{~DYV_Z>@{pJW3t4cAp0jLshW5<$tgft(X?&D(k_Vb*<2Pi&%rsK^!pr(Dq1ni|Gd8Kh8_CIOk;hb*!i3BBOePQwM(TdE7cjJ(WhnZ@vtgoUMZHNPz+y(-?Tac4KgFNq zDiOeHMysC$Sz%%-s-Fj(1v16uf6Al;b4c+H~Q~HppNC_vE?$C5Z&-9^hlm-z$zr`ZIfcZv$S;?8UCiI%!|Lza2g!rvalhJtGloAqME zXDbb+&+IyeWPCq9TMsmwA{J+nL~?cv^|q^9QCR8i@_G1>d=Zqtd3Y4YtC$ z71wdV^^|kBwWK#@SH1Qt(qYX#;lxD1bl1$+*>HcqI|EQ@wK3J-!euCYtqDHHS^aLR+l@7T5o-$XIpo%? zeqTh6Zt_b{XI_h+6_Jz+N$Xs#$ z?jfri@mnR}5x|7fg0%Qz=?$!FEskXD_g7x#T3v2h!L>qCac3Cuz*FqooDPc&XO{5t ziLa)B!lae)r@#4G{|xvsKgJlgT;FH$lOn4#K|>KJ9hb)_2$6xJ+G3NZ5-Wms(siJq zgUdLC;@i}1;FLBERr$b_J;drCG=}rmg*E-FvdLK;e9^(PEoc7=S#T$iF;|!c7JTS# zvw5MI>C&;@&@XApxWnP>WH=v~R|gkt|0kA2YB8OREI%pcw>swfYW*qff4-CzgY+pynA?TkG6S>R~z#nYhr2vO3|jiDN5-SbKSrB752XwX9 z^z4wu?G&#^K1+eyzLQ`8P;(t%7ZMLnLJa}T`oNmT-r1G8^JsZB5*_I#?oACE?5mTL z3nQtcg5`$Mvi>r}R+<$89rVrGrDOyV2SD+E-v6|S=qJ;f%6Jae2~`U$8MS%V|5(Bp zwQ=EMfjA@K*HTUp1$GPJ9v!0*)@uqbFe<_zzCJRv=UI+!2O1$7i}mG-x6|J~bfF!< zU3Sn4C`KzaAf6b@I+#Y#>TvfUMr3wZxGV1Hu!}#jO(wRJzLTqQ|9u3EV!0s zPqBTX^Ra=p(iwZ7$6-)n0D=?ZGR@>~;u*y)tl>k|$@yw}@0Y*4HfEr8F?IO%R@oW(WAAo-C;@jOrjC1}+ua(U{Z zQGrX0&PP6AXX#H%FdYNfKvH>~bh699ENIB9-LVeE9#@;QK7jJ#>l!Wo=^KD*sN5h^ z8v}cT^=Y-Pf(`>%Po$(pi;mNHOr}Y5uMOis7T$jWkP8?yRKW*!|2MR8<=+`G>>(R9(P#GoY?s^B z5I}#c&~$I$S{j;fLbI+_16Aq9O;<}{uGc`A$Jq%sJH-?wxp!f1p37u@l+X1mLkHLB zn)sv+Ta8qE$m<2eBWw?jOKC`4Q#+Cr7ScfUdI9&F`_qb5xRrV*Cc4(4?=eHiOoi!} zGt};J=$Cf4I~=u)kLduw3Z8ElHlD-PhZ^pg zcDeTpJfXR+r}L)?@vxh5Y7|KeY(z9{k!iO?29@K-^zs}e zL{FN(#piG=?kv6d9d{LSZi92gq+U0EYidg9v-RGxre_8o|Kk1l;|BHhc3NPY>L~C- z&~H`C%(Kd7ZSSeaAB(nqI{erPQ#cC9$}4?awbCi452D(}rrjZu^J73Vr<`C}7o7!L zKmrG2+NDti<+3y*NkrN=7of{$jCwkAbN=n_i0hk4^WcV#T#BH6sgbbRb?ASz@fNz% z(|bx8uizXzRf2=&2dO_kOXSrEPAkEm?C;8`5QdP)_6>z89Z&zSZwHy*a20Hr1!LEy zoYh+Whk5jN9<|}|gZ^{=PN*ZhVc1Oz<AJjg8n4#75uMoi3N`r_{nc-ul??jVzx7eb?0sCPc{*?OI z%5cwTcxX%Lu-THu@yxcBNua-%tv@Ay&vl74VwanF_MS3zYc?e`qW@A-+c&6f8JY^{ zWm!wIJp`@s5cqo-l0t-`l$@@T!Z8eYv6xjibTTjMnF-$t^ZWFt`4v0}U@BC>IeHYk zqo9U96H$M%Sb;cZV9b|*wdUb^zM~6#AVZ^{EziJ3yba~{l$F+-ym||8Y;fiNzMYuU ztiJzM`&H`JwgN8JiDTEOVvF5wVwjeq@i+D2h2{f1M2GHD-s`d0xF-dH+&eWg$VimF zotKYg1>5DVqdAqxsYl&Ls0m-0V~UgJ`{0I0fO}?$%7V*A#7rg=tpY`xY2ICh7zV*N z46~N2?|E~@{4@dMP-zjh9-XzQchlwWyYVmnJzu*G)kMpH!A6=;J87t;bd-6Z3J!)_ zAzZ~$dHTChjvtJ4OqVK;E-DSN4H_1_o!hucwvNkd_xIa+-v8iT`${L$PLaqL_0j*p zjsK_w;^uTZ-bOUKn1k)e^YjkUxjxLcEBWI0iN|rF+F#|ujbF-n7C5v0iME4N+eI>&2X=+^~LK7ul`9<`CGGaF#dbuxm1a13_sV!N(!`Ts*jzNkd^I zmJ)(PX~HelcEExybs&K-$xmH7SxG_{U{ZN?pd~(2bUfpFj%ao3+SIGNz@6|a zN|dTD_Txo32C_BB(4vVIVTy^(dVDx2-qW|NK_0oI4C~SpaA{Vl?I`l zhKEH9h%Aec8b`uC8o}u4wg=U-q}@@HQ(bZVbgMi^1}xduJ9%7nNCQy5_{AlZ1l`%H zHK4^9v0nCI%a9H!PX8N6>13L%n*pYWKLvi>n3*_}?t&li+q-~@+CJG*9k@{(AGvug zJ`-i@OaU0tYcF&5HFW4^$ZvWHs$QYXK@Aq!ROhZ+^+R8iuBTi1th14_X-Epd?t(ot zILF`hF!0Ui_W!o+1-57Pxz{!|lC5$V&@CE%E(GoJJ3EUiV-5zyZr)|4MG2?!r#T$g z$q?E?PC%0=6|xU7mNaOjgcU2AA|ph(Fpb&sJ^=xE+XWZ`;BD7PnCST5)jGH;VaceO z{O154p%dzKV@qu?Q+Vb_U|+v0x77IMMsduF4?gLNj0f=Y_b%V$GSlv^$o=6fBc-s~ z5%|1!&A1CX-mK;Jicvj_^ZFL*UTyl>`i$L5l7cX~o2R@p+2k*nyTegJW5lqdzD=}( zrxwNUb-S{tb2$+Cq+jo@lHk%3PAk>#%JRuJF7xAq_?%wPiT)0aJ&M|HO3QAd(N{dI zb7j066HFcZ;#M}VL&>p>89ULKf@jD>M3dJGe3gbY{u8at7do+-yIBEJ|)zLfEn)9<~6(HH#VU_AaI8lKc0Zy{o4JYuX#m2ss>6$m_`Jw{vJfPKOY zS*0^a7X+CjAp@VXh}A-rmjDQ7mfC*em_>a5rM*pgT45tH)O#Yc>DK!P*VwPUz+dOi z%e{3m{3qbv{q>PzwMJ8DJv}nMcnsjsE+KZM6LOCR*4MvJKe7~K<5Ebxrr4r6iToq4 zbJv!CS}-`>B;8yPV0b0bkJ`PN%?t+c9tgr&xo|Dj!;hHXBd=&M{d!|hAzb-~5L=nE z+d1|>7U(K*(xR(U?=F4gX{i4>PS-RcckA4Y2cOYrK65EoVi5F+S`we@J3Qk%#zP?7 z&m+$ng7sb%*Jt}SBc!`ifn}g!it3yKROYJy1OK`}_AD=4?qJ*Q_MQdsueeo>d**dxRCvjp4mAB*?d4s2f>9HL{C`w^`CRwR=5z(FOnDW1%yH8O1-~`+Y9iU* zuF~CMwEvNaTby+@=c47PfnF^k;;q?R9#AN+3GFTYcMi4PzbEn=;@S_>6(XR?uc&9A zm?RxjWVsGNM7xq&b@Q93ibI@FpXSAUcr#*r|0S~ney#iFLVKy5+Aleskq!DYD0Jhww<&%PDMA~O~|=p_*M4+e>=%6Dv* zHsljpYCpru{o7BIpbG`sO3yMkI?Uq3D8wnzSz1LEy9OLN)y$QIP1J6NS?BzX9En6A zQQEew>_|)m@f|1?cs0(hU%|Wy5@UJaM8O?;$UQ{oboJo2h|&3tPryd^n&E{muefPi z`S&LM~$y zvl!?1{r&Id<;8rzpU?Aop7#Tg`kPHsEwTX;R=GUfMi&unf4h;I%N0+%Jf`%Wzcp++ zb7*GO{}BM@@q4c|+&Vct_VM{0_?I6p;Knc}WV4NJ8eAMRKx<&7;xH{et-w~NYIuvz z#nq{d>T*dB=&c;n5$w-3i{&1RMHX0U3LLtbx9v_ZAs8SwK@@7z)7PavZ`IWBl~zAv zOUAwx{4yM{yxyy7W^E>V(A2;MDlg7ZX~|bImVR+v=$^%1ox1Qf!_w!??O12a{`)*>rk+XeIj zg=P(1K_K4SEldz#oDb#HSz%@HDGX^l;;9KsWH1s)2~u2<1ZAQyCT7r<98;49h`exz zFgKfr`?_{3V4x3sR8sm;_I?0A(N-VxK>*^KrrzwN$5;*m?Q~CpIEJ&U*l&K>83QxR z%}x)l$iHZeyVcap#W`f?Ha0z;i1le&(tqQzavWlF%0U+AJb<;jI8NG)CtWfNTcd!+ zzFJ`XL5SkD_J!3t74iEq+XuRvyN0TKXia?VJsn{Jv}udg_s#-! z#a1Z*$+SpJSE`5~srjjCY0r`Lgi}M4=mm)l2Uj69$@4nrpV<|> z&TU;k3>#6T-=WB8jo~G;HhNNSinu?HOB(gmFh)PGP$HP{<6<8lZeI zzZ)$~H!>UYqV@tIm?D1!F;=FXXfWR)PDM>!)5+LVe}wq0G1B|0<-{BqU5v8^0#TtJT?e6EcdL~l=+f)1#HFU@t4cH4Td2La@V zt~Ej|MDuuvJL5bitd@t4@TvAKci3pTu=&|z`7=ZlGt7i~n3(oj9{<|HR~_q=&*_at z?kttD#Dr&%v2Aj1DIRuQ!LWHu2ADI$WM7V4@o08)rcU|pXb6593vun{O=|e^l%t}I z4`ylN-O9!`ojsd+-vnD3(p~Y&gB5sSV?(f%G|%d) zcXW$Cn?u~VVGn<8tH1_PoDHH&m6sK9N0~W0(jP;1v8`^u5~AHU4*-hhF z2o8I6nvuqVo%FB&JoDF`AFD5VR7(!sslMFox9cUIET16wz7XK+)sg~b!rvSC5Q-r%OdJ4rvK{}T^#Wee@8a~>nA3Qe z6FVgOCr*zmC?Q{bLmG~3 zStOr&ET$pBwKv~Iw%W}4WL&7V;X%B&Eaj!D=85}gENNw_!dV8pO6<~B8^$thMLt6T z|81#X#!Dc-;CBBh&-gIt;n9CDNMr3+TH)yq^aEPVB6@HsuF;n`di&tnb>s2G_5b zW{MJWL_vu$$GF5lT4kiC)P7Y3ZRFN)k?fhp%H&s2G_#sOJ2RbZB(K62ojx%XU^2s44b{ zX+ZWpaYfbh6Tf{V^`3d=U9RvEo~mj#b%9*e*piezI~$h4yK=}ftF>X0LV89zY;ztT zxp}yMFMp0#KrDn-D2R7}Vs)HorbZHFEmnadve+x?!=;Z;a~8G1&H*dxKu08nqq%9V z&0E+>#(ES|n1K3V&}DN{=947eaD>yD)FB=h5PCqg*0R}hXLj-H!Ik5s%xug%7u7Sm z$!j9@E0ZXRUc%-clli~T+tx=p$=++YgS>_9K~LvI>(vCc8CKws8&sN+Jv8we$UMD< z7DI96GgniBeXVHcdo-PbA24yvU+yW-$V_byD=kIoDbO-BAQ%7AQ;Ltf%$+24$l;fX z6^J=_YacmlUSn16de+If@GIfL+l)mY%=;3+H-jZ$%?s7iQQLqqK{!gkQZLhZ5@lbA zXD-|xDE-YW%lxTc=G(B)?C&2&)*hgIS~feKh2HBS2TMP?@_k_Xq2V}|pREq2V2kVo zLCqBD#88c>)Mtq;qs@-WAO)#Y{v#GubM>Dw*#qyZd!|3F8uR{eT|zHzulsBDET3<8 zY*)6O)MveYLbO5#*g=ZJ_EKh+*3hcUoInQ&2jJMN4Z_)h&v!2OmO?8%%t-g-In(UZ z>wku3bVOFPoro+;5!*CMc|qFmWw43Vg6iVAhxkju!MKg|;39wMdAq9jr8MfVqcf2U zL8gkL#w=4w;HWP%4SK9-{q%W99$zu%+SDQoi?uny2Y-qegCiwj7E^`3#eE{-i7%)- ze812P^1lx)`j37pjP`Gq^d;T8Qj_ghOluj+wOE7N4@I4ysMAnPPqdQB(0?K1GVB z;#kteL2+sXr+rP3h%4Cxpaq?dj9otIzo5aGU2wYf5pwkC!iEx59B(2pLf`je?u zclo**p78d+E!$RLTa-4Os2eg3cj=diQhK~;OW!H>n&6YOzpbPi>#}-H27dMzqG2O* z-OPuMKJ9Zj{H#4dojDP0)>JXKx#i!eKNZ0>il8oSy}^)_6bY{zey2X>etL%OoX#* ziB;bU&Op--T%u?EC@>f*OTW5)f(S2r;ZLa5oHjLD zT%S}%qc)+=?7Xn z+MlP_0-p_%_g#Wk0~lYQ@4B4Aj01bM4HRLr!`J9R>1%FDivc)R{cLVRAv#RN=SQnt zjk`U>9oNKK-l4>fnt)ASA-adOovj~WETQ?A1s4JraK{vWHo8&Ximelo=_&}nM+)9t zXM|^7L?32y#$tZ(+W@-0OKj~0!_bEo7V2uf5mzYhaVk1AE0?!}rXxYLf>f#FEd^^L z6-X&RwmKgzme{Pe(LqBR^yVQAfYAGt;j|B=0`RUQ4YlmIs=fue%AGz5QF#K`VdO38mdcs|oy#EOqrs z2>&{uJtz&(q(Whl5A{3xmCy)+V zRW8qYv%w+jsbRdAwrK62Q;?CCLugtA`wMa`B~KCvHT-dM2)9R4Gg`jc z-{fJ8zy9Csw>VYw-xHsCsZA;u@_1Ps5vX!hik#qRC5;^8*nGN!@GIb zO91sRJ}W(E2{@H^_Q{*b;vc!wvCHYtM%LsdR@-DWnksi^_3^8HqSv&uc7Xuj9w;9;@-z`l3FMQC_rNSc6|0_Ku_Q`%LmjFQ$RPYhVuf$6CbEo&=lAg<~&%Uc;byFqg@4q6b`Zw@s!GC0P8 z)C++D5^dgXj9)`+eH9D-_XCE83ucAa*hhvRx5{7Xx!VvUZTPMIIEh|l1A)cyNBJOv+vN?zN? z+DGa05zhhOZ_fO(#QIf%D9W<9M_jQ;)%<1opUw8%%vp;-QdRinPuppf=tDcOml0QV z*G*Li06LKxQDpnnfz*w7A7vo21))%=^M0!_C)0!-%yY48JWJ=x&Lh7ee=q~+Xk5ic z=__I{ttIZ$SioD*fTLJbElyyx`SH!5(MsLX(Vi+r(&f~W_c+m(-HU}#hyzM*ob(WCRIc^YCu`eF(#3Au*K=@##E{ej9+cd& zY%beLjtMtK8w}Mf1m0^v3H%yA4M6p}N5N)}*fzY`hwtNMc`{fRD4`VSt4aig#Y-r8 z-;D0N13t8q38Jg86QKA6VjW^eJ=p)?j4b?x`$n$n4fsWU_iQuzpQNX7Qo3?$7&+=1 z6?>V}(rl+(|GS*sRfkDp!0bBZ64{~WAN6?SnAntR6sQ_jmFCZwqUqZu!W$CwMe*hh&?sQpDO0kqnFGs`6asVx2!avt0>lbP|f;QyUG9+1K z4qNBd|Yc_S|j#z?qlh<8Z{CLo8gu@nYgUmO9 z;aH>CCU2Cb_LYv65xqX~JZvIf_eu3SJolHU5$cW1H7B*K?5Nu=#(co0d+}V`0TH z$7VKnv^2=BQAL^#Qn!v&1?2QX~?hg&d zMuzEiQRm-0pHr>GH+T2n2JEbAN1|-su!UXCmkUqB9i#PXOD8v&(e^G)oEqZgMq1-g ziT7qdKOF088Q2u}k%I*V9s6cupfD8ihot54ThdmSCCu= zZ{A`oMcLLO{6XKrRXqRm;o7&UPeOMtU1vLh`A!dLDkm_;CY{2)o_^ZLfZg_O-G1dw z8plRk>Da_x9y5TkTG#E7zL}#_vX4w>?qeHo>QK@LJ zN;|o?#}q2t2DN!0CDpP`{hY$Ei|DyiT9>j&%?ZKi?4E`Dl#9Waj`_5n0;``r%Y@Y^LU985V?%3$Ey_+(E4)}+j zYv*<6S1CNFuQ5qihu|uCrLh<1`H~x7#!OcmoP#5$>vHjKn@4-P)Sfj*2^$yFYog8q zzmK*(WK2Xt?i^O1B;t=5J701(lTh;g`=L#-AArE~!Zf~M@_y3`>}_%pFvF);4U+!c z$X~QNjnn!2Wa15J;ES4Bkj|5x^_Z^!IsdS^yhom0uiKpZ(aD%tg%;d2BKytu6C$zY zmkOfk=yBU8au6{|iIDcR=U%4LM*2v9$o8j07UTIaMD+f_5Nk_#%fC*Q_R*!Eg$~%D z1F;D7-2xzaTi%)`xxb}LrcbSovTBtD^}vqQ<^|ON{uSV3gv9)1SACgok&szsWae^5 zIFso$Ba?Tq!7sCpms4&;|XnY<^lF{ zs!1EuXb93Pk6qJ*V64sihN)$g*DWnvF4!8tCLm$!Hp7P`rLj&DNpbcuZ`umrg~6l5 z-m|01NTBhv)j%9WzjtI1$aP5@=CUKo3QPGNQ>JU!kH&&KOZquypbw}G91{XXW1hGk zwBYq-%MYPAD4r21twyqyn&)OV^`wh`Gk&f?U|vt*q@OAuimVMK&8Ph%ta*AR&3%33 zl7n}E?(~L@$1d{E2&2%|-y*)+?q}>OpA(#rGAtGvf%PGmbNd-SaLt|TtkVI9cZk*_ z(FSUqJ#`)G#i8>p)J{BL$RO?&_9={x<=nh1o`X*?=9MqAO?!8#OmZ`|GZ zm29Ma!v*Sr%eU>whX0cZZ5}oRt2)()*b3zJ;&yOiB{GeU#w5%XqP+*>@^^w45n3_@%WnD%7Y7yQ3UVdY{ z<2B#x%du0(X=A;SS|*30&T*L*{{M%cbf=UlE@`>BNk0-p4LU2saYeH{Ti zrMQv9?&)XKQJB2z*=IcN6%3MD4O6o8WTx(ms2c>0q!n9LK)&lwoX08&6)69iQfe)| zNW!V1YLd;bat6t@K_@f1$Tfjfl@@d;vfmr@rJLnfOB)gQcLF&~Z8Jr>%zwmLN~4P0 zO<9si^z`~q9k}q|^7d@?%r=wE(=m(Jh61N8a`~oiT%Kq7X}c2|pyVxZiM*f$9|X+$ej+3Vf%RYe4of{<&=-ih1}F}K?ou!@P3rputcpsCKRFF`0+BvFvEP} zo7cK-SOhIIc!N|63*JrxTOhymVz)0j?#u9quG=t!rTyQMzfocP?bRknN8#v?QVw3< z_200(PE{}m1TUus>#VkEu{v~i`w5OfM4l!3)imh#(6}H7di`*R1MxUOx@E$Oc zp7r5P;4izM9)!Cjw@}k zt3p5@@v&G`UtDkQg2Ur%wuKaSysvr=Qw>+Bjk6LMmgV@e4p{m+wC#Cc`Do90UM~h5 z?Apq$3q;r@L!pz8u{Fx6M%N{(A?)IV^e7Svj)#h`y#PLLfXKTaV`JDbw1`7oP}+G% z7tRD<;RO(LZ__4oP4Ncb$8SU5hg6<+uendjmIFX3cayhP=JF=RE)hxLnw$7+c{f3% zDN9TYghcOxZGHwM-iC?H_JGEjSS^c-25hf%7}DN_=7Y#`fAt-7EM}x^BUz<@B+}@% zY|%38)ILL#Eyj{+6Ds@H7$Nr2-0l*28F%St9oc{wVlUh=Kg_|w;&u^Wnk{p z7X)8E+_Gdbd3UkLeBw*m?q5q=~P5ZShX7nC~a9sV6wTP^@cF{nu{@_NtxAk=$O-auF=|f7xBgmrv_TUvtd+bw)P#&)gC~;O zlGa!`k2hnSt^OTc64cOfD!SIPnJk7K!&nfyP)8Inm~(Ku_Nf*{CK-6yOwv;UBj+## z)2ip2U^|b}*97Qf4^o_x0v4vzE@&h0dQI+|oEx)q<0*5A(tWvF-C453JBj&ek)VjAZ{m+ZAOi^2ignGtaV#9zPK&A-`Sk%k+TN6qL|fgYcXsnp>ofNyfV4bb+?n&u}p8J;`Nb zIRy+<91X7O=zLX#8OMoCjw#IDvBz)PHLyDcnrsH-p-mn-kVe#3HNPD$0y(eBeTu;! zT~dIj8Ktv@kp@}fiWxx@Mod!$3XnlVk;aihT59TwM@T{!A+VK7;Xl0y=Cj9;^o1WK?d6s!Fy+VwWz$yd~x}f|?(q-9>zFrhsHv5PY z@p6QgIeRt_5Y#W;9lG08;L7wkYD?#9^S2!s*;9zQii5TU8ibY`4q7Z`Gqs&xVO9zI zsbzy!TSbHfnz_IjOS zsCTVYY;zwnZ8ZVxXqf; z4$1EU78*yA6|kZiX?Stm@8<&1=qn0EnxJ5G2q~D?|ECv150Wu{UqyO}twQ2BY4U<9{w_zp%_QUX|DsR=yvv)5ibA z>4-3RoY(nN(AK%!K3cUG0HK^Uag8pf(_2lk3wWGwc{KpXdN(I!>sh?t}`}I`o@r!JjfWs(f>n zt1U5>?@Vfa<6JX#WKnnP(~=i}+|&8GVNg(;&Vi^4GI~-CW0#n8;A#{EI0I@r@8mc` zlYXYkV#w9}OU(~x8Q*SVDL!m(PBHy+f5dg^H? zZOz)qdTB=YJdvT2&0aJWHjMT8-m9|L6SP43I>UIl`pg>?*OvUzySF3Pp%1qFSX@_m z>-np!t`mSZ=jtRXZ-|6j@R^&3HX`uGjivk_o|k5ufnLxVip`LRB;2DP(85vp3cfJ_Kk!A>n!x)Q%YMe}cc8T9DH~&ph38#h0=(fh6aEiq{Bj7dm!R?}ND}^nM7)+iYn7b62YG({Tjhl^8 z2LjSMi7`LRrr$a-N2SGWI%-z7wV39X%R=6&2IyL)9;Gr^G@2VS1iGTLZpWfbVP1Y0 zer%%CeG=L^`urO*^SD>sC{3BEr^3ENF3^^f_V9}H zZQrzL#60&Pbh}EVT-bPraxmK9m4dl}W!x44AbwkESz(o5Zg-YV>UgyT-UDSe60f}` z&S&>djyBeo)T0f-lt-Ohf#V{XipX=g{r9%7U@x(o#6xDb*h##hxNm(}yer;HvqX_; zPa`^VOswo8xb%pZDv#v3S*@l0B&?iB%R1S3`#T$&r=OCU6K!32XlIu)t2r1?m77M__5`lds}74(+DyDWg5#!|W=S z&UIPXb|MJLsAG`b3o*Jg<`%e0${<63^)S+N=HaYcL~VXp057vET;8kR{L&8GI#x#v zJ35w6E$S<4|F|_tM9}!+r@cjH(4*?JKsZv~i8_EO0MWT7)L#An8!9OLB$JYLWYz{i zq@zT5p>);3h6q0R<6m%dIpSKSD$k^|?kBXiC4g_Y7pwg=(>z8@3?xTE{u1mR`uWW; z80UP>ux@{C%CXv_PW zz&oI->L%J1AGN#Sl;XYk%(TorW%K;EX38Ox#1$1J!jUSiPhd*~o2#9mc*;nqp=xuN z_1Ybt0rAmmd%4g0VaLH4o2>vvz=z}3FbqjoLE*38!3T8!jf6PwvTwm(M&ElCdGejn z=$+N2?rk-*tCnI)v<37{d1Ho$9`7|DfnR*vtD}i0wOi~@m(I0WBtDMPWgQIJt?4vW zOyP{UG!L$r58(mu-0_|W7TG_K(P+1SA-YN*??r_ zPmGl}ynV-8w4m(|9eoKEX!42GRT?z7N`R~h&G>NxOmR!u?&|p`oInJ%{kF?owia#a zEkW+#Xt-ywl-S1=0aqheLMY0S6{{l(sd@bVUS z{n$kVc_671-O(!s8qkG28?!6R%XimVKg!IB^ktG?y-Jds45M%VO4gU9kIVenn@R7o+0 z7)m^6U&?owMA}up4}5`y%w$YUI6qW$z}`ieY|q^srJi(!(ibNBlwWBtHVitBj%Xhh z;_EusqPt3R|6MYx#Gh(yjhp8&%MZiAeJK!X#s|q!%HX@LIa0T`8rrG~r$}JmximzM zau}C@hK)i}^;WDcZ^=vr$D4*1?&i7DUih{|o8PYIj5ku8J;C+ifals(-m7b9w7C_a zpL=l`yGj%j*9XB$)gpXchYNl$qSc3YvF_&~*OA*ToTzoItA}2FL&mYZYn+#!=kL_! zZ)OI2ZR^u_d#j~Rcx-!!p;PRqtu=CT1TtoAyr{=Q!1gI31nY1OH?u*YN-r=EYxSuA zOl<$Pc`D7*O;np%ILF#;N=VY#%CLP^l3f>V`#H?ML_vb5J2ed7 zrQ|ugOCU}Q035q_-Ic9+>5Omee_O6)u8utH*++Y;Wdj%~c13{{u}mX{n-)D@dG}N0 zJ+j>^}p9aO?_k%Z_CQfW#r_+A-!4G3mERh8Rc=4}9wUBVIs!=wyvx;3abm^t_}Gss)*pa= zYIJg>he5Kn@$PY2Am_TwAjN8rG_%KIAx}EH@M*PuO}`FZHY3HD0sU4-65E?Yvy4;1 z^of=I+XSL)>pwh^?q$_mmQMORvmP-82Opv`)3;Lp2BP03Qy>;`AxRJ(JZU7 zO)N|p+#RKXm6!RsUIgE7NSK~S@kp4_U|dQ65!L8u5Uh-$eh6v$*AKITi_%79i{iG7 zev{-ez#F;VGPFXr{XYOm`8)4c82kh=vp3S}c|Z0Q;o`N4V7;KEy!0ETGpM(egD)qd6<0weB5Jq;K(06Pp!^Rz)ti}*y`Zv?aWgg zt9ux+s;ii=A2o+DPXY->u4aFCOz+4X`^1%zYpZzwuHnW~;{dubh8m0Pg-{op&^D@E z#t!iq)B&)lTx1--SLN^6p;Cg505W-ODZ}hdiH)iT9m>j4grDA5&fCeO7%1F1Hf|ez_x(?37 zoE(_fZEE4xNth#*v+5=lGqm8DLA0C#nhFqmLm1xBM%{!Q?UQJ@%&El!w1T8JTACZh}|NO6rba-D;? zG24#wN?Qfe%c~k4{LMnE#KIG_@Mt$R^%2tY<}d?OG~0j=;iLaw{>Ow2 z&M(dbGTF@*CS`cM1(6f;UK6;xT$?+g&PGZo1*6$cf9pzN?cpJ+Fjp@TOAQZggVQQ8 zl|aY|1c@Kxwp6%(0ouJ)$ZHQ z6%Zk{Jsfiy0aO8i1{ee5%`Cbr1B2C{As4b21APj>d!ace&0j#nk5Ufw8E5x`=YfmI zMnY>v)D@CAsKa(A!ULiF)nRZ!`E$4`UtkMWs91w_9au|iYwWi(wE9p78Uiu| zRzI#TpDdE4ClOhyG5eiFump_)mGof!|9fuK3j*4cgL znLA$o_1jN=o4cenfsn+F|F=AC)3=kKp#i3>2X`NiD4T6>LN8u2_`$~#8Qza9Rjsj@LmkMgeAib z1@~H+ZGk(9))#Fh#BB4v@FUqZTyDnu3hP;C`Jbtp&J;ybAwFe7m6TvA+*}F+4e>TY@PEwtN2rRz)DN3 zlLY^2;n=ZZ-eHkYSc{1<7+{#&{}+ z^mCWEIXVXcXNExs>j7s4%2MYmzPIO^Bw=eAcHZGY(ig&d>Yf5cKz|%|sBdB(7COic zHMOkX2jehaSb3i5<7Y9af%Z9_Bep(4DHbM)DAD9d$HuO1kaqqn=b{-9xWs}&(ia3%zuoiDM1HkBEDjd9RAV;`QQY4OS47C@>T-8&<1JOcQOkrqi(QQw04Pg zo>533yvyp1H-k9)U~I*x0!zWibk+IusD14-Gw7pVnj<=SHlr@PPx-z{cWKQp$!UzT z^qycPqmDxQk1FiDY}7ec_OdgkWQ|UB5H~YSI}dlS=q>z5$UmCib;);!Pi-jOGCnl& z(2sgR6asooLrvPzVRK->%F#;5CO7sv0uLF7!*hF-lvvJsBKx*(msCPl3d11l06VQ1)Swp$-sUQM>V9=}LE12&=p z0k?gq*}qWk!kX2Hm+JN5YZO(>D{};I;0Vd{Wo-Fog^^vG6LrVQG&)#YO&hB)s6EtZ z+bel@*wYQKGuAMv+K4M8cXhZf+{o49gbM@zChI;e@>mRKGPT+yAO8UsE8NDoIPu>W zH8g-V>w&3YLJQ6gD?PS@hUGaGqdn!dPu5DaHa_KfMVABDwnje*ZfYSX!}9WX?+kS@ z!_rq~5pC}z>btczN4zs0m^$KE>dF(Qqg28k#5Z#F9LN;FkT!IASAA>#L`PSQ`Ukt$y$BD} zqdt@MhWqn1dYj&$A=UghZ+BIqwT)z`x9MBI38l~R-097ue?_Z*cOG7>D{`M-F=Zhl zg?o%iD%QbHF;*U15Xr()>X}BpXJZO zu(G`~Jn@D3ko9$p^R@12XZC0|&$iJnX7vEfG_!x#=i1ZB3vzE_DVq{>=yxk9K(Y7z zyBN0H9}Xm8E*9~?faaJG%Y|G=BIM*kQ6R7-?xkZ}LI*!yAwrw7kP0smitJl9~x6h7buf04!)+TxfbS>t! zXjvQ38&j^6uABemPI*zspFi%o`td9CAj+S#Hw0{ zLg=5neaYdW*!sL=reub@>dMv>(l2;`v)L+s*_?Oo%)V*XZsl#D?C&rwt8TV=j9D#} zgru~hy53N7uP`7bGc>~Jqp|*)l4gV+w~#=nr7*EFoqJP5)No*Hb3|rYf$X!gx2!FNNpXl6@+;IzgY6sh(Uft7sxTvd~*HO`1&&YQwA@vUb zZ57n5qXuUAu6xxegn(|?=z(rDNv5<&RIUw%v_><|aV^ruET-)_z8)d#c`d!+<8CzI zvUibqE2k)KUb?21O6^xuK_!j|)#jDX;UXujl}XhM9Q-oapXjVl3Xn^kM22by_#SsY zHJgSSTJ{DckJ$1sVKJE=m$+`$jQ_T1t+0+Lw@K@iKaa3IdTMPd#d$w=JzZWmtWX-> z116sU56;lz%7d$GyW5FG# zKi)i!PZYZ);8ezbn(JzgAJRTGth{OdbioLyd&&!6bd_VivLPx>rc%0vC&@QZ5WO&! zhX4B0e_Q4^>N*Xf(T5?s|>yxGENdQbA)? zlogOWQSK?jBgs~dm3`Xhf8_7iqbI^v&QjLtr`(bQ; zq4?!)REO8uyWwr(y1T0T4NH*YY%^oa~lnuTN&rA)AGu? zRJ-O~2fKoP`unfu{&*{-h_DqO)U|1#hz=8zgIxyrmt;>g;4jbN3D>aZ>#n^w%p(f& zv8EVYHKqCA##wCe^o-jh@!EnRG^dC-#?+6p98sdfCt4P@@sExPwhx%*o7@<|@;l-B z>Bc#F8&$npR3$v7knGu~UT5*SL$318pmmABo_djM83dELCXM&UtyvNF+nmF|1#}TX z{TnYnYi#kUm0MVle~SJTIHE7i`h(GnF0^lSXf17U><-`YUI2^d_OMo`Bj|{yGP+(G zPwzCe-LGi0FZghACE9Fu4j%ttgnDHW6282SwY(m_+~)&{wL}Mmjh*BF&cZ%+0z6hIVmlR!nv&jW{3S*UClYONQ0I ztjv%XZQA5x2GE?09_M6BZ`#19F^$(74EnM`L$4gewtgSyR!do3!rdV{afAbsnoZ59 z3sQM%dGCbIuDM6i*_wzkIM&bTr|3(ej5vu){P2tNCJ$^8P>*PyFQ?)EzF1Kx1J>j% ziZqy7lq0=~8)dEc3|^ONODo|nqts^p;v@TmtNqpRv zz-nnppgvyje7cM8Fa6%fIk` zqp_XG_*y!+)=oI2=H?imNeYV$GvQ-KY%~=&+ied;T^vu3g|8-vq~!eUM4b84C_~`K zeOTdGqR73tfUU8m=y3wR?qzpoB0al)g&ti=ZnGkqpAVniIxPeN*0uk(?7|2S#8~4g zbDpZiHZ@Ta!=Og!K6}==Qon>AMh|u6@~Eq$L3QeelFQr+CG|^5Rp^gZ0v&o;<{&x}xVMqI!BH?14usSQKAmC`=3JS6ZUojGiC@wB%>zkp;5m5z6rQcJV@PTrK6&RI8z6Z^ z$|hxsbZN01OA?uMm>itt#{4`2EJOSbnc~zK&d_Wy(6kAMbeaAWV3M%K%#2Bm8%)%E zjevawl^GNP-~ zT%PIvmJ5yymRIMQ4xOYWpDBFZ^w>N5O4R0e)JTjvADx@nN!l&I2ip(V{ycQ}N;-^V zCO56t5(%tJoWcVS%14@X3Z_`q&2pca*thnP2E9Lov>^p3-Z&T(leLFj@<&-sXz(xg5 zfxa}3iSaY|zZhN3kW;Vi|6=ya%mLzJx(JN9CWHOGJY_hps(F(pl?W1j2{?nqdyUC# z{Aq;Yn2-7J${tDax)wt&x=;MDt|wv8*i{}|eDS@2b%L6&(I(u(2(H8jdvaEgzs|Wg z03btQSwafc(pH}vk*y@y6EVo1{2u&72alB=0^pau zCd!{K+k%SlVSfPG8Rhx^adbBROz;02@0?Pngi=w2Q@J_CIF*}|+oKX%S~n**8+Ify zGdsl?Han>Z;lwegIVksHgx%zBJMNE!Yz)I@lbc~|#Vnhh-{<@L2j;Q)+`M0}>$;xT zYi!ER#`=-<^ec!}i~$(ROxhreE$_0AXl+Fyme0Ez4;ie;#>9QJ7eeOms=XZCu{ZL& z{T)8MV?@Zx*l6jPO@0i^@f;~&C9jzvbI!AX;vWZ4GdLC4)Z^Tz{6ZG!$+GS^qinkf zggaD*&G#EwzfilPG`CZpYSzralCrze4hU}=`_h9d{L7%^JuM?PCywjLHl44$u}mCO z`|}U5$;ymj0DzD^e^mk$duQ!hjuLA|v$@ z^QwS$=y(F&?!+9rV_uwap)2S&3M97Zh@YJ8<}S zeS3Q6T3USQBeCiLR`)U+g^P-|frw)yvClX96WVmUc*ZYlQ8wrU1G3ygSeW&qo=>5C zx<4vvZW88x+zmJV4yCfff#r5WX0nTh_0d)*%dNkw}?=}?0GcKO6yeG)feag8zF|9Qyz9C!q)J8R3FA)CSh47)bY4>8ly z7FXlwSviTcgJ#TPk5E%zCwFxAvn%S0AobfU6fHaKS6bTFpq4_bGlw}lE}(bdteYiS zo!ggytf11xY5i?@pf_2!bAlqi)!C(p900(DR9n@_v*cwG@-;} z28MD7^$(=z?pUz^{UjdUZJ^afw?`GF*r0rRZS7)JCmD@W{eb@ce~9dKp!mVUhN1HD zA>{4?YaJHfiN{}W3T^V^R|IxeW9>ZI4D&EjT_#filjq!E^97JGE&Xn<@jiyLL>H(#O5P%*P^Br>;L z6crfGvE!EK9M4Fof{VPNPx9EkF?CgZ3!l>lA7n+`KjPT#2F1Te+nWJt-F@PXtTl*)pewj}81!KVVYkPe2 zsnQ3iR(?FXF>t$U3!X+nnipt6>gfzgA^8C&j?5!OTn`SzxxXIoewE7x)h?ud9|A)d z7i~K)U>dzbddG2L@(bn4vr@A(XDhjSC*!;nlICr2cdX;fbU!)#Aaq;=svNg1Az16$ zKkX+32R6E4Ib%0mPkv~Gb1xZtbDTPG7o-)bHt2oW|5@MN0F|K{!j81wz6G@RNddo+ zJ~rh&vn>;42s!he&37nZ2@_$&MIDw6Pq#M@qfUhE_)-AtsmFP-BO+Z8d<}%TQI|eb z?rEh)OVLa>`IDzE`2z1NNBTUptbT(+wG`i>DXul+`9*4zm&_6G#R;=6UGS_iok9~i zDy~9)seLOkU(lz72w{_R)cUp&OBaC=QM^IEq7T+LV^I8rX%i8vD;wYG4KDPB;TBKS z-AaK;T(SU^KVgSz|59^a1_eOXxwhIA^P%DP7s7(WCd+Bqp>7QXMLP&kr%xH=lVcMKC&*?13wc0 zaw(yUo0%WvoL8kESM#fCh=W4a>P#$H$O7+75y6MOb^hpHH8i4+DM3Q3yl74cW{lEuN%s}Yfw{Wm5H*=!sJaFxX zNFgbtfDC@-iqc51tqwrno(1Y=1xU8($y6t+YjBMvw&(8r;^nPnh+*MojQ!=?u2J7! z8K7ZjvLJw1%<;PLvlCMGyVNjC($Z;(Zax4!tgR-^wv{Le%G5`~8QoUE~Q*|U$zP(j_Z=F5eq z!@#E4MgT+~;!$Ok-xi}L7fFLJV>KVytN>Ni(Dv9Ww@ObGexBTd`bD8P-R)24f*({y z<;n6gH6n*P&2Cc0YFk~lQy-Y6`1neBqkrqYIuZ#0o@+#0F%ncRfD z%)z)_dYBKJE{J6F|8`iOW}x=~Yfi#5o{^MMCrRnbqpK{~J{P6V`M_~_ANIUYTYxcw zd!00dDwn^vL5A{WV6i=y8;vRS8cvk1G z#!1o__e=*5N0$dj%5vBDnos|?^C2zXbr;80!Aj*RYc)#QCMvedo zSr=uEid48uy(?Z(iTSSvq-Ix#tp}_+y(+Px{=?c6u+F?@f8L!BoA##p`;4QiMf7B# zFxQ)ZoN$jbg4-z_n5QuEmkz>ifpoXEH7p=&&Qh^y|vMaV7rxr z)kaI)&iGGR5Mv$mMu}VptCeL)qiOx@bf~x7?+sY;o&`T9r}uKNq&eOWGU(C@uA5w5 zE)@|sqbB19bW>~`6y|e8jgN5}pLjKr=}r_X<{tz1r$aqIBpS(GnW6AK@`Qh!bxG#f zXmEZUl?XV16j?NQZoXW!(zsX;gD;?y>HwZhU zgEkswUc-);N+x;OCos4t=0SAxk^gqcyk6)z(zsVL@Cc23SIZo|MbYu1dn9qx5Pq^0~qf4gCUsuDo%lpMS2=9M~!coO( zTWb)aRs-xSarmnO7u+3qSiUT^6MB~wtA-PeMqBR)wf17s;^0o`ae%kCr9*#!5rY3+ z=bL-8c?ko~6UDXz$D_I<*?kqgQG}CaDWn~ay#oB$b3ky&_fsYa_;PQ!x}&ZzAz8(v zWoqaT-5*6;=ev#Sy5%YKdN74tu)g@p;0VOR(5umdsCym~u=a#9beOAjmTIh0h3d@j zmgvbny>Q8Kc*whYHYT-VL|miSvv7^Q0Rmskb*xq=?~9@2#$2kHfcXc$rYX!yMgQ$M z!q6X3xPK_TkU6Lk)onK*Y_lD8bRm$vs@zNdZARtT{#}g+mDdID&adhM5}eQL(KLkE z_?^P5thTM9(<+lY)a@FO^Q-l?duK1)>b?5-Y8#2S6}S3pLuz6l z5=*rG9N2k&kOqAM*xs_^e%SfG)m^%K!Jk#eVRTlD;+>01!W!j|g=!TUmMGF-UocB- zvnzSlqX~%$UmEm7mr~J|To!DU2}7Fa=QhEJxa;Y?hY&efe!JnZ6#p@+yHeAV2SAad z8)d}u`;od_ogu}o{+rDK@LSaEp{~uWQ{oIz_WW=;3uQZmFkz+{X)*K@2aF7;1%!VJ zt%^GBufN8l6SniqMcni-l^*gZ6b7ldpV{Fs_fBzlyuC8ew_PoLJ9C$x)e8AjM&wtB zn#e_~4xl!k62LQsqZIP3!Fi{hw>Sgay!^_Gj8m|>gHYeV=_OrM6llKArme4u#O>)3 z&YORFG?;JkP}Hy^Ht|PxBrZVO*u^vDP(fzB^+1KYMl00pDaFC-9A^xi?=AHv(Y4*H zFI56*dUF39I)tGxD#EdAG@ObD$t7hWt`sxP|}KXJOEzksu=i&+RlUt761`^D#Bv@>h&`OsXfp6dOwSY&sNe|7Hx-ip zOSI8en_78e&n@-kbeg0hS?N*?E?aD-Pg+mvBi6jxo-kjF^{=N!CH)qv?mnU2?0gy@eo97~0Vi z_N!(D*DoUf-b2_WL=}&xd~X#E&@$zDQ28m$Ox7V7QVO$4rVF;uUkAipmUVz@bhW}t z<5KN7co(?T(=o43f1q-rhp6t92ly*2%2WQ`;HLoRzdhYYclT;lZb=kXtGg-KgTN+(12d)lYHAqQ{{bb#_@ zU}#kjbeTcfxI5%YL?f2q`q2XCs#dT{-XH%rEk1)nA70B|0>mN;%Z!FC&G5FN`L3%B z7ga8;=qJwKupDpJ8QAs22Nnr0SXqD7u_lTsBGiVLA38L(_SYoD3EJrQ6Nd}&;!m8d zPdEPC5&m=ah8YkYaeOih$h7gHnJOy5qq z%bYmpA?SsLK_2BA94Q1OWBKOBjkzNmiWqxiCS zO!Jf9BJ}bm5{GvgVklpHci<21=>i&i6X~;P1BC6NZb^SXgQbL2_6yd;1rFYsV7|DV zl;Cyd>>QA{ywBr-$eXz*r^|-8Ruj1vrOdBZ-6Y5p=dB!Z#PeB#&dy) z2kkVNZ|%(E%|8$yx66Mzm3~(%Bk_B=yHu;2XDQE-!mhF`lwr*YxSLR`u~(BgaDQyfyX9mvjrM{$J9odZwXvgc=cu!}e72`izO#p+F4Zg%j*L>r zDbssyBzJ)d<|__s`^pO_F!M=+n=|aW`xK%=Y%c4IoXys_GKovyO$Myl+@ciAtd3ra z9AAos+!({)p}k#axp3df+Kie&Koz)M?jN>wy>`I(#l*DR+%~Zf>SYuBws1pShb!G7 z14cmL>c83|j};oJ@vwwD$zvYFN*6)(vUtx;g`?dK9v9{tciw5E)Q)Sq>{$mQe*Qd< zt>v*2JXb%pG^9Z^v(jGek5;o}(!Zzgow}Z8wm8RMwPXTi4Fn4(; zL1u8}O9x?+RM3+z{EIT}03Lb$aizAjsiCkxPyvSOgZsO`z9>;|77oYI%Hn=h14yMy zbjG)IUDUkU4Tixy}q4p+npH6+_vjR|coi7fVyg7`k71k^@*Y*qR6H_8qU-1${E3;=FtYO{-Wt7!IVbks92$t{9jXaoVaQPG>d)h73v9a_3 zfYUe505~G}b_qe)`&4D2w3J2%ev`HkAT8ZZO}_UVr73O77U72pVHuDfrkXNTGB0{9Wy-_@3#n#VzIkp`=?8n% z-+}(xai85okY$0Qflyg8Q8DnsIJy5lrQp@iT36ty1fVW^lqwGI?IuYjdC#nDj8@jZ zLvCT_spX8S%DxcHuPB`XG*!dpav?A7)R!kQ?5VG|18S=U-|~7#lmNm0)uM|QhM#pV zA8sV#Y8Jc+6pzl1Sy@DKKf7^28&>rQWJt;M_|$(pUU}jV>)moYu0lMR`*)*6w^v{!+5mRp=;b7{>xHHQkgB7F6`2HY=)b$y)+3%?*l;$~bs1 zG6L0_x-Jq&yh=y~fnHkPeZ0M6XR_2Mv-MaRJxysNS;Fu03Kgt9_33TEMs-Shq8)sr zb;Hcd+I_E>CUe7AmAy;v&;invS>rKgWC$NcS?Duy`D}M|hV;E|< zbaO8=%&9HO9Cfhx!W|)U`r$^MOM~Tx1Gv03>t;+r1jj>1Z89jz-zSZ62v9`YC1}`) z1 z94MIt{yNY>&3H@wxRE~q>87hrws>7-yN#&Y%lkmYd_qllJf7STXj+S^i7bAJ%8$rE zA#6&;_P@tjbr~fyb^`jxYvm~kI(1|1m?)m@M9d3)4fXdA@D(h`iNt4<=T%f{i;0T-Ly22)IkS+6EU-dh?Q zrS^Ysz6}Te^pSju@fmcB>tXq88cTlzSSF5{TX~krZ~v1Bl6TH9pBY-pPgKdrx*Hr* z@w4XB=WnyD$4zO>G|q4Z3k@nM^vN9DjMg4PIJh?a#8qL%wHx}^<2yTfCnz))sMV-g z-f{;FW*)}I-jF)Ak(1+DrT9 zTpeR>>HolUxH_Mbj)}8e2cAMq?RB!Hs%LxvQ6kD~R?94VJp~)@+Yil4smSJhq&_(i zP@AVli6K|TL7!z#ikrKn+ayb)nGc40*h;5abefC0<4`J7V|K(+#}6aEI@#IGloh3{ zSOJi627Mka8*UQQ*yd~ z^r~Dov`OZG6(}fg(&vElYZS7gJE6SwqN@X9Q-`%pQ~zrkCZ0KjRs2tLUeEu_uxHi_ zh?&fQ71clBT*|aofc)8$D}4#o2ztE5AN~_92o`l+LMsTanxaxmWnb)c8dpEG=GMCx20Jcf$)0=1o*F}(MZ8OXR^7PJT?_3n9dD(g22xzq~8i+-vJHh%} zSjW8QISV+Svo#+E4#%K^E5R{!pNfto>L3YwYaA0Kiq7gPY+7$OLko4vkHTsNgQnLD z9i{h%OmYV+GopgN{iiDENZq6{l~!~UnVaUZcJE#GPyd{dw@B_!UKwr;mH5aoIbL~P zdJHm%myRg4ZZ>RsE&pTL&c5ppilP6_NhqYum%IHEsKBA;M!$;`fB!;B+OXQUXsFdB zPXOX_r27?b200TV)`!4{VuHKONm>JJgLpg{i(zag+SLg(|NnAVS_uxEVax zFe$64dr-Cd7}7$^8)yp4zxFa5x1WtWBncLj)1`0NH;X&~d`iUx9NUqA1ZXDvp|x*= z*y#0Nvqf16QM;ltuKrNS85~*x7%5Mx&%3JSl75dj>3y!;D|P9Dw#n*>F9X?0g#e85 z`K4jZc49m4isX;`Z<#G5b^VsVKvdHf57QAe5RjL_Qh_{naDe^oM^%eOn$9{{hS7f9 zaVhP&#cS1+WKDTWW_JUd&^4CFm*i4Mle2;>3Wgg58WWb54-E;Rd(L{17T}u!fey~ z;x{z`sn$m#4IBXBLFRk1@Tb*Z6-P>cH>|3b2miAL>0C?k0m|X55jb#p#j`bQmjgV1 zZjD`(UCQLGvC)&j^5EzO-s(;Z{mX{>_LXIY6OB6Q8e&C~-Eo8qhrSgRo_j2;=w3St zo-xqvg6_8u%s`Z5S!*h7bG&WXCvY3|f{uA;3(=Hi@yYK~!oM48Uv%`_ASx41uBJe1 z8+5W2p6`4NYWB})n_OcC;Ho6Kq%jXDP4Hw-LDQ>Ylz2VHZN`2yw@s=| zX8e`XP6E1NCq;VhDbpyBsqR*noy1Ka=2_|b?GQ$nTR0Kv&3*3`K%%S6uqmDPv8k<| zp$hPJe!J>0tCq6OT}vhd{V>Twp6YYRi0Z@EJ<=zi*yfbUT~Y1kq6`#1raqt#qu(F% z4OKGK$v+qk^vJ~)Y1ka47(IT0eK>ZVg@fcX_rKwXLu)o|p_v7*0>ahNPUOF{x#a%v zyAEwvjPVV8lTP;spX}BIkUh%NlGTbCaF(vW8$lnAz0@AFH75I$1R1ySpcdGad6=Yl zJuOSxgnT}9JHT61!FZeWLnJQiv0NBOUdB+$c)n20p5mAL>!<2LN=!r>l=& z%_}#6gm=BoDR|li^yfCED?J=Jw z#Y7B~+0>93ztVcF7We)Ce*ObE4=G)SS}mAO&)10h#hu#naqYfjuwoCl!j7irRnUpF zpf^WW7$Y5c7NpQ3W9}0gYs%9FG5gL@O=@bjkJ-Ub;Tpb13SIPoV`hhIr5}_(pjlMa zdzzLFurOunLGg6E=8!q=GaZTZ*7s6|H8Sl0+; zLS;C(*kAP9G8dIu75dZ2Lg!ho{LlzAHYL>@nqa_zGYR(Wdm~Y~A?; zqi{TJ^f3ti{&F`$un^=hn?X@oN}a20)15Y+lM?Hy2q-8vC>nq60_fKk)%%v#FDT0y zMJQJ_fs8#eFhHNEoo#7*?yfkJee@>?DVTfw_`NA33I^4;9b#Vx&J1Z0~l^C!2n`Ww-fbH-qK3fcm-BD46a2 zmFz;;|G+P@=;{B8_hi6+xAqsyzV#&R+VFX~8P8UCP15KgASo}Vx zA;;#t4y}XO#4m!CLaWwMYxSXnz>L5+P8nUpq^>tnw6y)a&?S@p*kF~(u&c)32Q1zK z4Mj)RvnL7R7K%WYI)g2+N>&+%^S(hRtDNMqy1$F5>HGsrfqIP}R~#Wh!PVF_?ixfd zv98v0XU6@W@2rkev>r;o{*fs2AUk)5v37V$%mj(Q|Dp6~m1jsI#pqW11}HL~kX+Kn zhQZ%Sl-_x^1I|xFusowea6VZ6vc~u8t*benDA{7|Js2dWewbk<=eq2|0@bleRP_FE zr~y@07F=^A-+zSCWP|juxey#CEOcl}OhC*urAE}qz$UbO6e_9$Ujvyx8^+(M0#a@r zh1sj-MZ3ZSjXvMLo%tDSwGOg93gUOzQ>RA#vX_g^+-LSi_tChNoXgKg<0AhxwRF2)V&6$Ev4FZJyaP95BSU+~t_*5WS)3Ew7%s@IMHMLFmGbw$HW36l7T9 z`)`1F%_GN47{M=6&BsE3-4X5p-z#EjMQjwESVhc9P4WJV?wm8-)i(T z>%LD}D>3OP*=bY_nQWh>k43i!YRW^=Jhp7g!vnD_Uc|H9o zM(aRe_(6!{gP~ePd>(g|I^eeP)w(I-PBH8I`T#Hh<%iuEvHN;+yVy1Zx8P z@HG~+cuRSOE_H4S#-M7$n3|u)>ekO1Jp;biA?KD@P+^sA#RGxDlpQ&n_L(*BhcRxY zr@vR2`|Zu1Bfj5V{|EfgQY>731Mkw!JNdXnbtwAF%8`h5C+ET0&gJ78LF6#qh5%8$ zA1d47JL{*}2>_Vqr9eU8dV>+Dx4%l69n+9b!#&%T5o<1jZzWofXQVdCsa2vtT|nrV zW#s!l%D;sziD_!2jI^et&cJ+)xtlQqiHwjIk8g(g*hrYy&$tJGd9>{|uZwCW~h&Y&yWU=azKE6Ou5Hy}nDo z`7QE}4X>g>5f~4f@2n>%6BZ})-9~U%>X+inWCaQ`*eCn6uoYA_)sL%=iu{6LiU@NHo2th`=t^8wmC z<$B7^*j5`{=V?yB46yQ~Y~ETq1b zDhZh5PktJXjIP!Kj8JcV7**&Ki z>C4%-CB!;J#kkzvE9;h|r>Q#J+`)f(;1Db>FqF$31@ z$m3hgt~CuFR36$>Xx$7OGT<)h&HWjxqq;jiVM#vH{pZ1nYG`O)odMiRGl}R&d1<^34)QFS0%%3Q-#NP*n75 z+GhushW>fiZ~O2;#@_ucb&|{M+ZI&9Y!K%Kt zY_D60Io*Rjw7vG=pbC*HgxhWMAtxP-fK_30gk$LxEkilFoh{Gtlotp$rN?DH&PMfY z3Vl@Pr0mvs= zDd^Vo!KH|1%Rf}Q?C5mVEshEw3T>+&sX>wqtgufaxnG`Fexi`=Ng_rRZud>Z_`IRQ7^@Ud310@Agjd(E(Sc=wBv z-rrS*%8Qv}JzHJPp`9&w7ibcAdE1t2VATb6F6AbsC!p>7D|JCY|1=4vejs&Kp#R?O;M<-L5X2jlwng% zIO!gcp5^o>EQqSq247G~Ju1(+?73XBX-47n??38a{-{)!EOc)f(xuggP^9!nhzx+uzz8vJNmPB#yk8{}QWw+YOjNX&XO!Z{9iTG>jg zigSzckxyUycUD0c4UK3cK`j*cG}m$X>t_6KX&i8d&k#V4N#^hXQZcI4{SwC!GNbjx zo{+^1(@ImF9%USJFPpM0Mz5o(0pYq(Dpi&e#&$f2wEtx*d=lzvLh*{Su=DXDpxa)y zE{BDVFy$_i_%$K9_Zh_~g;=OH;^h^0Jca-=pam7@dWanrhp>3>0V&dqVIA@ z1jRGbU(r)$V1BROQWRdL77{;1^V88{(KW>^oZ2wi8+#ljVD4&tP~tWV(FCd&`h_|p zszV|UiSSg%>TsA2O$N|=zkdsRSvN;gVUI;(AQbZ4+=qj32a7Re)Ts$~MCH^rL<071 zl^S*Kma_%?{qoG&TOaeft~UTyq(!?MOZ8J;_&0uzBvcM8jy~+Ek;b$9i}Mg%8TXt2 zz^Mta>}M^z?R=Fcwm8gM>loA@>2hFwGV7o(pcBOpHU-sk!6d*@;zz<@)_5dqx zNm@WYWCd>%M4^sSy-Zg*K%lgKYKk9DOk;l~hrKn2#IxPCWTjC6!L|wu6-5v1`kzZC zF$dVc?+@1?Px@lhSVrH;SwfG<&=z{rTM8{f9YJ<{XSs1fOm95^mc^YYd>=;o{U`o2 z7kwpK0{;0eM`=)n;91F$TF+K$S6eTkgDXYxjLZg`zrFL~w~2Yq#O5daQsc_Vq{+^R zY~Oa&`R!uUhQPIdRQioltFCF?UT3xB1&DM+Wh~Wsgmt^=VYShC4wPKzqHb%$C+32O!5x8r_aW9bCa7iZGXI2ZDT2oW~59+4c z%~H%#+?z&CbE7@X!d6~(k-Tcl2W5#N^cTlt#QTkWS?Qnbi;qRUO%Co3h zt;5hqp;GtmX8uzuC|=Q-XFv4JBDD2g@|bu_H>Jk9y1+2NlTrjI#usCWc-S>q)h|x` zaXRV+-PvCGgQVh_(hZ}yBlo$(j@H<7wMi12-g>Gs{N|Fay)D6VBseW3LRnC8v`PzbXwsAWllRVszBPl=Z95FEFxW=yY`I-tt2Cy!0M8u=a-cz*?gj@@BU)Byo(3*F@$Ypsd3&dr!cc~VE` z)yB5MB>?c(1JRs33%=4e94S6U+>3#s?mF}@Ul5nt<()Q%25o+Mrht58oSC82PK)KE_HGMm`>{JI3HVDwtw$V*CyaeKvFkH(PN7I z0i~XoSFCiZ2UP2G-tTGQvbd+p#{m^Weau2CHMpmA`FB1Xs;X&F<*}&0oyNA_c z-PggLEjEpr6Lq!GJtHP9#0d`UFO1QdZ6t@g*~zn((`N`z`3ID) zVGY=dJJKV}$E|L=8xu@{6N_PqVUwWG2IaMo_WAR4>5tk$V2AouE|5t@COQQv~F`Rim0F>aHP< zJ~R@rUyq&Z8YuUlv?A66L^!V~;}7w)4;>VGn@3;CMOn z&5~YBUeyvfGzeeD%-aPFC8JHBYrF^dYKuT+N$QBT$Oz&P+c7}WGAmx_C_1D(LU@L7 zJhaFnbhr|%xK>>2ty635}Mie_fU`kL1C)RLabl7AyN9cBOJW`>FC{ZmR2 zrC0XT7@vZEZfMaeuQ#pGdTOBz5-!6W<|X)E8I^K&nd|cGgq@6dYbie1rikmK(7g>g zgLfk5t$h4%N5$Iq_8LhwcwSy4`%{iGYe#3=|F?s*1zpNU#yd&g-LHpDSQ&?LzuasY zG5UouOvJk!@UKk2rnu%3;kPDu^lPI~f7_v_xjai6vB)6}wWBV=;v7#4`qrCHWP2#B zLSl_lW|OoU%w%}#Dj@0+ttdke>GlD66D67R9`|+t2*<=da=7%6V4ZhCURDXiI&^MX zzD7n2k8?#C22a~+UdUqu3t{Hzo|qpjgX`0;*C(l?oYm4kiRfgt(wwPDk%}$^1q1QE zk#&fs(s_g(Acs_WR@Bn#!bl2)tT`kz-?<9;Aa-}288;eiq&4Xp6txo;&}psr-2B_G zV|GZ3`Y8BLXI=UG*DBsl#q~XHk5F~@-|_EJSMBNDL)@L6gaa6ePPgQ}qDObjM1j@1N^N?FGW`OjHov4T z{W!&g!rB+AMRjqQ?c8rCJrQxcay;qa>7#ot=>jO+*#NkYbRpAv?Ep3yB7tF@QF{V8 z2kUhbXQXJ3=|?+ORr!lQI$}9^VqGoj}?YqcPAk%4A{kqvU!E* zTfsin3`nI78FxtEt?Y2{au67$JK9D&>Gv=-C~F7YghY06ql8z2ESr>HAMz(@(WYwi zOZzGVwljaQ5ZCFf&QI%Lnu!Tzw9#VXPZq63T`r9SfK#ItHQS=4kG-XiRv!>{7aBmk z>+Q=^EMgjbA%bZa@Y{wYbE0pAA4NthJT1sA&hcbNqRL0_hX9lfr^mwkWPw5>s$);5 z%MnKxkTTIgLafFM^+C>q>JH;n^(Ljw1CzNj>fC5X#YBX-xE@F|d;2cDyq};(32WQz zszVCewE5jzF^`K8u{_--$;#%BkmC%4j}ej9#CkS5dOKe0+!m|vRobfhw*CCxkT2TY z_ihR=#!D3F-#z*Ay3Y9B`O@1xf#%yUac712LpPgg=%d)&Jok}>9B83K)yzBKLo|># z>S6!de21!XO$_x3R9a^8)XB zGJjcbV6$sGvSYSerrUs}X#(ik;u7rLfcpzxL{vbXm2uN$8k^E2{ERsk30tZD zrjgQAAO`6SdvE-DEO57+fHBESb-A~;7iaspJ5du@yI{=!4SCn{(FU#MKj2hQUP!M$ z2M5urc9G-aocg~qBO8Aqay>1`uN&$R^1e!?W*a({pdO8c1^9G1+vHsvsZ$+?Agd8z zOx%N7KR5HGJlT+=?N-JBe|2T;WCAQsCjkK@uhzItc>@AyqZxO^OjEh$EBg0;~ zWvbXK{hy$*ghQv~d#c8J$=F$+mX8>wo((Ov#4O$VC{;freyCHT|89M41*W(%V%CV9 zM;Lo&PTpv031g{eH26k-<+iQ{O z(rB$EbOLPyMrv5btvlp9EW0t1mJA$j!)WBWpL;}ZVr7K}3!%kS<{FaY^Zl(fe;Ri}TL76A>Ee3Q_Og3rp zLjhKoL2h5F zxYOmnpI!$yAVHzK;$wTj)u0g;9?QLU031(yYwfKK?L69+cERZ<<&W|xv~;MmIk^fY zRM$FqiQewh={@2?gZ&b4lC|vL#-u#<9CN~osAJuRY z5?0SYGZW6I81zsD1NyS~CP{gT5dV|kS$qson#Z8VS!es1g9en|T$}zSZ@bu7)9+Yd zaQ^8#;xeSw+ko-7nd4U7i>-TR(WyLjVCb=clyGsLlEj?ZqC-SUR3;`WAb#?a2L$O&XM@C{O&@3dQR>0(A2~k*2!mZN*}3$o?=RNu#6D%(kroeCA44 zxn~WR@5EZtLoGa^w16&72GS&RlvrDy_nR?Yt01R!?Kspw% zI-1?)>Yl9|eT${C{MH|7=Ge(-d5K@_djC=x692e?8eP(C$~wGZ6m070&`+pA$o!x1k@JzNQox*sG2h z^eceHPEA@&wcM`7eVb0yL+KKRDxs-d{)}`sV%f(`6kuQI^>yXvk5dRi$KGu(d{{BF z1jk`rTQfccaRe-A;3=Nh!elTXOcWlMMx)XnqRtleH#IIg+yL&4-Ki7@JLk}Jn49y? zNOR1Y>oP%Q)$G79Gv!+&=_7zFF7MW0<(=wORrfC1*dvY`eseSVdtd}StIm(?mh~s$ z&8ic4UQD53z2Ky@a#r^w+d{Tk^v3&zAJp^}LeS z^t;MC5`uxNywsiN;FF`alBa&p%g?BkbjzNNt$v~eYcYxdWt6DD5*l6Q}tX}aK=lUbOk-h=Q zYWzj}LYEspYKo0Ff0|e3wFLQ=v}sMWhsFWl70)Ib!A=gnVyteFAq*u2B zOCmRV9dBW}B_Dt*Jh?G8?K?uDU1xa>F+NfiO{OHpB)<3SRkFj7r% zqMLPqS&s^%03nG$c`7b(GH?tNeQ4?2gY~}L*ZywvRqHtxh+%4^eo*LwgCpQjJmbnU zmL<2xIO(~9p0f^Ap$v`k$A?rh>B1_a?Rfb}G z-g*Nnzn@;EY-w9_+L8Ru6SsV>uCbtN+=M>hMfe%%Zj)DP4Sb=*U??WgOk0(&m!*^C z3ajQ1#HT`hv>5q()hMMoAXgF|S>e-D-VQ9DlY`{e=}F}PK;D3oGYX5Vm`f|O9Hvs} znwP$j{3+!q&1h_hpa_);J{t$pM(OppQGW=)q1!f%)4B{TeeSn3*AG+-mdd?~W3WNh zj5=V1TbMij{BT`Mc2xetz@84zP|YZY-0hJFomnPJe$29!^#8eDl(=~a_pUal4UXX? z(+T=$yV}f3+@ra6qSFPv+_ETPLU}kO4+X@03hG5Frzu#&?`zTM)Bn!@@Xilaq~+jl z)xfK?4uC}|>Z|tLxQMXZEiJXP|=T>CK)%cJ1K`8OtUm8;myn6J{5PFi* znmbmO7X97mht2Ky3mZia$KK#!1NdL_*cKRFPu{|+MqMM{I^-+Y6#Evn91?AN%Kd0( zVBmpQS{0$2HuVb3$WuRwx1VN@HMrxsi*c41b z2Yf$o#uJi*HBH}M7cTz3s`;;A=E$p_NJtfYDhln&yGuNt2Wxx{@^#XeIkyvroqpql z#%~3Jy0(AX1bm0?dA)ADy(^$nS)$q6@Q z)HdW#$}ZK!)uK|-h?xT4+*dO&M(3nhPqt*+gZ98m`?nYD_de~E64~7Dk?peza!eQk zFxYw`3jm-USv{=0;R(cTTsq;4>8l4R0xox=W3NZ?TsW@a6HFWK8!$e0E~sJ@UM&pc z)HS7qVIej1$6AOBXDP+V^R042W{onKAwQPGi;Gc&)e0v zf0sN4L+&>Bb63QnHO&d}0D0vVkl=72e71cjQl1C43`_NYI{)`sWqFB5S-hi6p15fD z>Dh#{X6Vq3??l6W8B@4L+K5flkX3bcL7e9F=1{UvTHkWQu*1*LNRy8`y_h(wHC@9% zM?qQ{-hw4x%yjw{QYfYnuE-R*WphNkvGF`p3I!GGD9s(~nQ--5tyOVQgX~ICWKjq_P z7P^KHI9Tk-r4vMJIs>gW$TnB@f4x{NPXHfXyUqjSp%Ua?FoPa%&4v5BkpqOGu~#VQ zAj!@kuF2!Obq#O%9Y!eB*gf|e%A%A~`eN%KA*2Ab2fvpKVIR>7Sl>fYrb+V|m9NMA zX4bscjctD&%gtzuE4-}ba)Xg-5)+IDfmh;0S6nCyAzJJ3LN6z;K*W~ zN=eA7^MP3><7UAG7f8@XDlwb}FJJxZs7c!6^2AR&Z|7ZszNAo&H1QS)4bzCL{cVN^ zWqF;29tpcbNxJm_z7AU&UtwLBgjHU|ma<4|*R#zv&7Trymv*ie+W24X1b!7(Y_7s$ zar5yZt)5VkKFePoIN|(^9nRO?Rc89-)TIdY)8}I8|axL8fui^MeV!j{RR2!{*D;DDZ(lHZ4P(P7kmUz$W__ zT!DBSRUG}|h}5=PLdED6c>fymuz_ zuDoXFQz2WY0bKKr8i(Udf)<2c-KQyWPF1@a4TII7;t<7pRFO?)4WZLnaiNv}TF34w zF`q2NQ=FE}|0k{RUDVxl3d6<+O1j&7HxO;`|F&eTB9j%Us+58Y_v;#cYaQH7TOCO+ z9T&1?z?iWc^(Iu8dI7D5vJPZIyL?CLv&3X{aFn)!ZR`KtW%Whjk!bclBATqmw>>wTG zv;4)lTD7~Rb^CIM%_+2vaDglip;}t5^yxx*o?Ma5`Xrr82+}2f5~6Wox0h=sNuUNd z6te2V*ccMmx39LQ%E@!2|{{7Rf1B=D0SzO&aa1RRRf}7G|Qe)VzA^?B1z%_oc&w z)*4MMDy`n(aVzY@2fc(h|0o2ie@wV|Ez?r081!?W7`tySz*07pf;gA4DVV8=xHw4r zSArO(qvmNGdVf%Kl#qpr5=0N!h1wD@Bmp`JdYl##Dau%5@06K6W*37q`tTHt98kH)1k9{L66sGfJ=z#xQz9i()(P11NSZT z#?gOIl_HMsIHS<---RRJ04DL5lK;Go0YQliNHecN2-qPITj_pB@_wbFG5JPA>!5Q3 zZR88pk|=N=_Od5ty+-#Y&9Tk_pXcA@(}@#uAP=<9yK#dL1{i*Oy{# zG&?JBYm3jHxv%nPNNp?CUUX!@4Y`eSTGQSiU13`C=E*!?Zb z0B~B_{WYxUH2%1fP(vIjD{|T7jeigotAgd@Q_TDfj5bfo4?b$)k@7&d%qrLCIc57BTgf`ap@i# za}_2{!$6i*QGNMK_C45q^Y7E7sfA0$Z~3OK~n@@8>I)+d~o#e#S>z zmp$w)L+=f(4`G{S==WBmELWjxj0Q$44{W{DfA3@vi|Bg`GXV^EB7VHB?cHXypRGM3 z(jqr24_1uNazmn>4#t^(uW0qPY7OjMFYG-D_eBYWt;B+84RT*>J{}zyY26El-WGwM z#ddexlP#3XR2|NGo4-JVB*-o8gzPGl z>_Rng>5OjvR4HLF>AYY$c(QPeQ7BfmntgcU{X_-rZGQ40w}hVU6MUMY7TrgN=6$_b zTB_8%CIhyZ*ypTU2-6~?ORR6g-x#g zPT$TQ|D^v^e4o~mKTIj;Dbz8@p2-t6(^zkpE$=73tEgYEmVnw8wL%OHd2ERCj+p>V zq4yuaWvbBE>o|Py17MTZ^wLvksk~&>IKMLYej&(x@|GQDepjCN;{ILrwmZsE(S}=O z*fhME+_o!WuWTskZVQ{wipsYS_%6cKH8zHt7f%4rhe$a^2o zLnC2D?$Hcz-rBPNNXHMn9=Q8qB-ewKn&+$aO4D@;&b~jrj{o$)oa=_UQ;-UkLqSUZt zg&Sm0N_{CyGjc9Vl=V=CsF_uSBial2LDF_2{#Vf;)j#t~EXn zZUh=I%EMDUg*c4PiK$mZ6uggeI1?2PhvbGF<5)}$+-_Dcen_#pqDM-rL@oItLc6_H9^+*xxFnEyA4;cv zz7I9!uMC#1Mi=9M;IYn#LCFEf>y;RS@p32(c(hcFFJIvb{oRRiP%XRT7G8+ffiLBw zC{$EILf^wUN8oSZ`Z0vCm*R9h6n-&T1}x{eqIS4fT{WutX`gAi#PR$(?;4xE`9-#x z{G3C($IA4~3M!OJNd~T1FuO1M&F?@Q2P$ zPu^`B%KKk6W)?$F}$i zK8<{2CiiA|2Z;$Nst_vCP2z|_0yTIk2517ft|$RSKka_G)@ zgI99*cvZ`Lot+R?V`<|hW6ZgD-5c_YDHD!o+Q+vltARO3Z?|aIgB5>7$MqPyicEC* zg?c0+)UM)X`OIemie0HywJGrpO#F#8Vy?1mi0M7_=ONov^{kd^IIYf~E^fLmb06j{j?yfJdpzMDbpLnZv)`9R9yL$%6 z|Fw0>-JVH?=$X&*hmvd6u9CR^?LFj;26yfaSUul3^^(*gxXpck^9$r%%w^-mf#PgJ zy?-|cMZFyj! zWJqy6)Du=Ud)?I{XQVTM_pjVtd-31++n~RWDDRLrd3LrP27zL=W#EpdwcY_-tB?gq zMbeUF_V^u|{3`WMTwv8$s8)^8x7YGZ(8Z>Z(#*9pj5kA+m9VA>*uhYOEJIm4(y{JE zMt%K86u|^6Yf3c%jbfbom`4Gil+d#lI>?4A&4nLE=k%V2`+X-N*GJF=1=wHtUzK+z zj1s+Ui)j5RY~7x4ijs3fezh8fskLrRUrQEBd?3~ahCCOQN$00>vtfs76eNFX8e4d% zz9zdDWmJVrY`-Bx3jf;@0^X-c<6}2}R-VYe9;idrDc-0A2yCh7)5hNQv$_P~zMk-t z-&5C*R5sFDkyl(>4tkz2 za%bE1*&!RRY(OQU*0IQUtu1u?6WxDO(H38~kv)a9Qud(AaLeKs0jCG>=SBOy;T&RD z*J9`Tv(Ad5*MzY5%&6wE4&7J`r|_r{9$z)}Y49+e#+~%l`w(~Herr>8Wcb9SGVUu! zyK}-4ghYw1ZU#U9ER<-v;=61DfoxsUxChrW6^nM2Zr=Z5;#VNm;&p!Xc%NNRnE8bNC|GKkL zw$Taac;FJA!!V{K?PGfMiYO`JkWoW_$4(kY(^qZpLAbzHU8RccG4`Gr6PF9FwD>Mg z4q`D8W0v#qSF!Oe^o&A-(sPzM#!Cq6-uQw}R?Gx&hMgOQrnvuEmeI0*F)8Vi z552>sGe~ncj{;{=PGq|Q#-@iwJczK&>Me;4FBU;w^kB$BrO#l`1 zoFKsKx7vBn=+jqb42QF~-|T*KdLtsBB044)Q->SeqPiho!VU!KqxhboPk ziwVMPuME_nx5X^?#|Q51HUW4sWm4omwVo2AaNPQiv{!DM7}@^=Fun{On;{)c;@u=@ zD!B?M2sGLgUW51&pOw?!itb+_W6s;(Te1KHvm$+z@vE(~mO?W>V3{FGA3v?ktDTcT zDMl9?niwQ4Q^&g=HnX4BU?eR=wEN`W=ww^Gg;iPieKP8XZN=5E zU6e0AXP-bXVvT!Re$U=!i zMp{`2(Prqx(H>(3=23mGDsAL`hT=DH*_LTm3`$ZbcY_bod{>J0hK^gwMiaD}xU4(1 zz5%!j3{XNO)A$n^nqcS!UZ5{TFr#zaTJvRKA%=2C=lZ3AcY|tsUOWCS*U3$C-j^RD=_hJI`XVl^C~Io4n3YQEO0|1n-Yt~p`b~d(kGhup$$K;o1C4$O!Smkj z4Y4uWW7QXaQ~ojGC{OM0lvAc8+$uElFzbFh7YDY^B8aE*_3eqgMeEwT&nUj|DaF*Z zbcivO#S|DG1xia|`(8^U<(znXmSiroUP+9<+zpleO=Lc%L(}vfFGe7s(kS85e_L)c zLWsx9U6*hBZt~($hr+x5F|xrSJ><-1ca=l3wciEvkyOprid^yHk6O;%w#~_k_@Y!( z(#ZK%e{y;l!G?`8aiS6Mo;Q5$PFl`ld6EGEYmb#R^u35j65{l?$yV@-T3 z3i@h#@)jM_H!1)i5MRt3aH0rqPnBd^ytIOX;7o!kR^fAc~f9l4fo5FIe! zv^zvM7%JoIc}z}@>cv;IPGFoqU~JC~SqMM;);VwI`1jAK&|bsL{jh1*5cwDSyrymL zHsL?p*yfCl(#?~h4fuV)(8CeY4iEUBz%JoQB}(WQJInGc8HUz`dfOGQc6dxV4S@6g z#;*$Rt>@C`!!4>~UlFr{Wm_k#$r*TS*_rn6IXGnr_oS`4qBZ)>C~7?~HGc@^>jc2F zi0_4`YwEvzfnpzgs&l=Y=M+$2P-^>9oL@+!l;DkvBs;oZMjns!aBkB}AY-UA3c?7n zG9FS+_CbUe$+Bjz!@{>WILz7gI=yP>SikG*agS$`OMN*Dx6N1)fuO3hJzoo>-^C`s zU|GH%b0g!TuL5AsRm=T-laS)O`O?3^<>ws73dj1_EkN31=Mg6$dJ3J#e{Y@9_xJt^ z)K|cEMq2Ku*I$&RHD4~>e;=|Tg>^@g9y&uCgV+c2R?h=@;(Z9%$)y?c(!p=pZ3-zq z1ww>vRSs5udH0p`V<>Rr4=Qt&80g;OA8FyP4om6f&Hm>X-|IPk{254o)gb-aR? zz@S0+TkQi$hUvhu$jmEi>DW$T(LatZn+NVwCD6G{gE{#9atUA)0CKsDwZ)&U3Wg_R z#MjBQ$U7OwU-mHyAukO{dMJ#P3YWs&{{@07aR4b3^9aaVWW21wHC)SVGdwm_16ZlC zz4`8+JEwaCx0eRSn7h~%p~Ec_b7uq6AT(!8wA$RF_@aW zjO@jBrrpzrr^|Fs_gQ#^)P*VpHY7jZXtm4X9o$W`L8qOYpA$@!K}JtjwVXZWriI%} zB}F?sVRwbfJnJlsW zXHk2OwEeS#`RiWlI3a9itrtxJ`khbN9-GC{lz{l8>1#A}F9vos^%9wHh-3FSXIY&Z z=@tg~D^(NLNzWtzeNz;Gx?U6OoyuVn>Q@)F8!k47(a{G)$-1yQ|A46l#CMv?zV9H1 zaL6m?oZ0%-!HEZW59ego*-8-*pZfIIj6@zRKdtJpw8;^BB2;*lTNu+s*QC!|Yl>9S5>Pt@@FaAuXl|5->)|pl4n}&It}iXO!{_s& zo44gT9TV(T`hOV?^}gl{_nYDR_fB5oU983f4Ic(qi46a0qbvpdT1p;VHk1kw<&c*V zoc$xXeDpzT^kG#7@@PAZ@1M6GAw7IFxuI4g;wxJ?HqG!PpdCC%27X$mu9? z7ZA>tZNaRHQh{MlcNY>g@9?X`|0_z@pDmTIC0R~1w|L8c@cBks^b%M_^R4dNL}ebJ z^PzMpC9!>Pw`nRQe&~T5wq#_-#St{55mqpD|1gdCw&Bo@2RRGjB>x(k%zb%)Q@8p@Co7)5t`eTST zptZj~S?@cOJp=!LNp@|hbJ)N_#7v3mivGPkFaAnzj~?a(yFK1G?~%n{qmd>>By?$j#hml|q+|o*`&ldJW(dHPN!4lO$K9GqUxVHom zy!U}y=M4pbo0TJ!0Hs~O#Oq1**obZVeQxbN+4?ADA;VG-#pEPPgg+g#3;i-!B+Y4@(S!B}VhTkRl3RTXwN{LbGRKf-&UaEMD9-3dzK76vRcxnZC5b zREFx>8CMBe%+JvM;UmD)xx}%AQvZHPk8Milt@H!2eI53Ge1P1sza8yw1d|PtsNp&< zO$Jwl?t5pVyZ}qj_|5PTnB{OZh4lxko?Sx?BFESMCf>po;PvJEB(U!Iz;+{zRhrUL zqIJ3Jzb)5?tQb7uxXYW~H^Vdeg36${Z{W$XLTTA$xr0ZAYq*_owEGAP__^~WdjK^` zWXdzeYn_A?XW5^ABW_|7E(;mfK^&6 zcrqo}mdG9LH6&FH*$4(`***^wjx@`e9nk({bJSNT=+JkL)UsJGfG$H@A7P zCBNGfI)~WU4x3MVt`7sSQT8ggEaRq(Wp@sT?uybIuDzd9&S;@!+?vE6QJqo$CUDRQ z6g&TjoYm7VF2UUZN?&x;N{eH-S3U2pz`AD@ti1+BpeygT+;2wff6?*jDn9D5EvoN8 z`B+&#aCB-&fQzqwJut9&7Pb(vVcy|%LIq1Y%xWf0K)G9ekcjpYRJafpB+*2-5Rr74 z!JDTRPC?I@BfL|)5`(RV{fV}E>~2fPO8 z+42Wiaued9Vc!Eu)6(nuOY<}$?{Ld)EwN3p7`anf;Lcs7nS*vIwbuYf6*V$SbZ4^z zz&G~as^>qzgU`$m1zxdLR@m^+OmvK5W>K607*R@JfcZ^0m`8sNu$p&oeM|%!4eSQK$_dw5OojDaUiJ4ThMi~QG_MVfrDa%L(=rY+*a>Vx)0WXr6I0M1oF!;M} z7Ui5f2^@mDo}$L&>-!duk>U!Zj=X*^x{sbYjNX=w@7wXhijyv)Ij30? zQ0b5^o_IQ0C{o$puq3HF$Z6BQ;_nGxtLV5qh^xoS=B>A^8g<_Q1a!FRtD^k6 z-zy(R$TnUkbbf;+tO701?B4*n08$tCMBM!h?;+6u>_4RbE4067y@lIbDF4P@Ex!{- zKA#>_Ju?>`zc)0ydNHcdspp>H`WI-x%W7qkR$#zqhv+IokUuUPq~9&k?3MmF2c!m` zdVBCBJX(W8UMzDYLWKgp@x^+DkkS?s=xy(|GeR)ur3ESBtc=hWANad~6{mvXu$SdF zDP(do{%njyvmh>gpBJ}Q%z5I$z?x1?ov`{}ua`s$;BZDK>mpKzi~lOYki3}2mPNH_rwDVnP}kBx7i=> z3`GU1_Ft(!JsbXR8t$EhkFIbp+vEbcHCj64;~Z0%oC2R+SS`iugK;qqZe@P$6qHCf z<5aM%Ii9%)z6-@V2{oZmXvkDc1~LS&jgE8!Ig=aZvv5eEq@nz~H>GBDR&> zarqZYjRhp44=oMw(`oU(nkroK$`<}q#t&}g5u$|2uhE;o%4*5gxNylSWp%WS(-DsP zwL=)dG)tmWmF`w%zF~p<1q`ic8WkaTNVj0_8SOkkDf0%*zWY7|KSFGW3gKu8A6$_lKQ?`K78?{mGp3;FTfO*^Rt#t_I#YV$YcspatBJHMMOBV4 zhmFxN;5_o7zCyD@GQ*d5E(gh-H-k zv8;i0&_8VZ7WxqW0PyQQ>bw&VW67kF(eDYzWh>EZxL)zBcDnew(6((p!K54 zIH>IDbF$f*CBi+kzfBSyxqiJ<7VJv7nIwrS4|FeR3)s)I1|=Epw_!UqwF)K&i*9lk zC?9-nIMI{70jFp8Ab_S_C>XVWr#X0DhlDt{OZ0A^QTJMr4$Gt(l^we7nAzSBnt{X5 zNp`YYwa1MPijZSSNGrAmS{8a8l?OD2jia-BOlK$%>T8Ct&Zbw^?~c8a$m#Bv4*CE@ zX19L`NoQykd;=z1dtuedpI5t) zdu!hduV|Kz4*So>-l`>+ZBz_K(HiK5v_HD)+QS{T7?9^HAIe80LA5Ot{ZGdAklG=J zMyamKgy!kO(ivSHrqO*~NwU{wf4*eGR|Uxm#C-F6-#0n>@7osVylsE|Q*_A5LVZ{n z9Y5g#y1)_~9gHx~)bZ;U13@~Qgk9b&?mfgsMhkGjKSUYObZos(xrLK^wk?e{k`^wi z4jG6#Ecc<=`;*<}bY{9WXZK8OUh1mlzHY$r1WOdGNp-Jn6rATSL<|ae*NRkE*FVip zK`$L0iw`&H7f!s*GR?L@dl&s&i|k0ymOUk!$A}H|`p?OY*@jR223F%9R*a7q?Y4kJ z(hc~6*a!9RPJUu|d-}u3qWE$zA?V1>t&aiqcKAPRTR@gYGkUh})^V#bU)3>wdhkxG zhs(9>&1KJw56f56$vmgNc*ag^{XhOfDaIc>5`FPKEfNOM`}#G3n4Xqf~4=FN<{(!2YSACMje zER~YMF`4p-{3Ne)y(o=|hC3%)Yt8}a{@H*dzNJrY*0-Vt!K6!I_^JK602JVm^uXWO z{~bWML0;-UoUq*MH^XQymLJ6WNCoH3p<~EnSy7_85l! z;b25B;vkqkSmACdRM3FCOvsk64$){y>ls+Xz(T^o5cVmy4Dwj)HH0_8O7sCq@1Cle zkp-(&?j**0p4c`e{Iv6^N{b(z6o+wL5a;uxQDcUjHzDJvH(t@9gfzR=DVBq6xs{e! z^fTaiINy(ywPv)Uj^WBHWQrmks+FfJrhdpX4hg=FpE;I&qJl(7XYW?!Z7{G+4IH>-6@ zOD$@a`;Yxzt$%#Spx&n@S<5+DaEG&DXn=X0jMJW}IkOJQs!*1mD?qRWK)0_n&5I;0 zSW9bOTKI1ZtSQ?ky?-Qcl7FvY(ud`{iUpzckT2Lec%&S>N~(5x)%5OkNu`4ov_RGxD*q4!W57?NJ(dV0EMuo9M13hn6NZSV=IB2PC zXM}kj=}yisNI$S;io&Wdb3zCyYbJ_x*f#dhrku#9J8p1Gxbyh}V;15j!W<3$%T41$ zH|p>XBCer5pDOi`jjJ{RBU3wd*G!QXOBW@vl6icT3sDTGBaQM~edBrMwXfK2dE_y~ zm6rSy!Cs{0`94QFE4$s1>@TNw#xAzU^{B1Cs^Vi%U6I?4)0&#Qq^A? zU{N>sE8zh;(=m5&tTt))dTox$Mt^SLBua#eOBq{^NpLU!B z>TQ6j&JlS{nJIa*^4m%)veQPeZdy$ZjWsOSnod%u9Voa#W8_?7*EO*Ocaz+`+~FrM zrHR6y2m?n$OnQgd7U~1aXjv63MYM};;D;vKkyE7;GMn;1V0wv5_LOh0H_2BlsxB*p z@;qru$X~x4j_c0i3|RkiJBlRKMo*BRvYqiaTKQJa+U>CpfnVdpuQiZ1Rj|#`fe5K> zz>^ztgUMInZU)jZQ4zVPY-ahKVte`(C~C(`jnS-Lx@8yVx#ygXU1kdHqHSiNX|kjM zdk_`tRdnZJpvlumvv5FDSXrCwJPfplCb28h%a%4jnMT$X&EIgl{lgQv=dyVE3Ht~x z`s>gW+MCD?m*0n2uu2L!`5p!`*9K<_t*ToM?;(a)eX>7I&0av(cNz_c1(U4PVA-1o z=-9hTPH9MU@Sm$bRF;XZ+6Oj&92{05c*(La`SMt-D%da(`rK)04WX1E8uR0wJ(mv# z;zpzv&3L3U;`=%_1@8jna-SRO$Uo%)e*k>Lf!FRx7<>z|+*OgRom1-Kp!ZD=fnqiw zF7iGMamP(Qz0N|M6uJc^2zaA$I_dJG!wHAbdPAL19X?xMSu(!l5ytcQe`ymWv&zcu z0xvY2X6L83JL*%td(Gv(K#RRSQ{%(XXoJPvUv_9n%te244q6+fj+-j?h*9c&62iXF zfMcJHy6cpQj=Js5Pf&LEyltaxnNhq;1MSBy56x)2+C3z?x4htcTZPSOkO#LZHjMz{ zLA?28*}`M8uliaOlS`qAbWqJsX%CH~Zj|~dV`qx%Sd1osLwKcu(Lku{aV`uG5$L^; zovDt1KOi0U$BL(#fM+WLhxZSBC|=yA%z?4wXK=j%^upT6m)NeBAWT&!>Wy*OnSG#f zp%D58rU*&s*)T#Xp*u}8mr7aHaVsavV%uRBUBB5xu>D?-77d_#z(@K;#uK1%} z2kDwV)&?MW1`xRO;8V(*Vk~Z>(Us%EpB5HtZ)fP({_4JqS%BkofvIW-<(bGg%G*k6 zmmVutyWoyqkI@s_$l|=eE2l0jf=)}5dRYRH?>CU#hv827HN=K~jjHPSY;3rkQc*c! zj4JHf3}M-@$~(SmooR z=MIe6{0%%_cLD!#@Yl;iz4j`nukpjMRxN!eChF-tsoMQv1*7TB`1t6oT}8PVyV{JN zy~5~mR>XNM|HJLI`}l*=Q}vk&G)G@of}K~%>OskyIs^GeAXqIf#%kpKDjL!EsOR2N zM9T?rc3;?vB1vu?U(EB!_SHE%SW}Ui%2k`i$jI$Oq!Y4>xhRRjHk@FtgJ{QYZmw;u z7)1ua012xvxizt0W!I&70o_P1-l(lJD`I+%j{9#?yOD{Tnoc_M;7;UrpnSgHs~|oG zQvpXY*5=~N=YFn~ZI)s+h^65g=_@yC_P{r?kr3W{nm^>ja7F9#Tw-#Yv+gFShcbe9 zkz0!Z@`vB7Frkcnq*dV5S2728e){dn6&{s7Y(o0Uq>g1fCA!8;RjJNf`#TChEclw` zUjw0);y3Dha%Ouxw4E5c(ceBpD?BPO((Vy&Km#~@4vl8u8Ru~~Ze*~L-qgaAf-NMd z1&GyJjN$sJ2Sc_ASEBz=Ci^R0w<8-$SmOCKYp~>Z{|w@@UR` z5toMY&=^#>Qw}-4;=e6dLL}J}^F*8M_CUoi7G*`(lIMqhwLx~o+jctvtShZ@9JwRl z?2!6k`S+1+A9KHx0ayj5hdPI5iN|gUw#v+msFrZleygq(xMfex%5Mx z>I`mEXqn=4TG?5KCgvraAfRP!BQMj?hQB%pByQJ zf&2Q0r&|zFJZhNJbKoMksFmHdyNjng#*RkwM8*DlcdF?jRm&pfs*j&FqDw|gY-;Q z_`9klTgB7$d*0*z=^t%0R_4liB}*&+TGo{MEF5n96EH1Hk99d8RJ(kQ2W07|02Ci^ zee_Y1rZ2Z7j$BFe90UWye|KXB-_{~W6v7GN0S5FZNW35+zAUM2B(&!j4Ooiv?oAE# zQ5pLQf$Bs7qQAR*MdP??H#P!M=U*a z!6>xMBBy7M$DeQ0k>Jm%9Pu{FcM_@BoENEI;!%+&`JaP-6jASmuzoaTow#9oaS8u4 z3ocR}DMi~GzyhKBE(<;p6>w;bHM1kr;QQv0A^i37(D7ee<3T@kk9@#y3CytK=z*OCExxmTGszg zuePv!ZmXx9QC@}8b1DtQQbMCW&}GAO`MWv~<>!`9b&KA>`_T@B!0f5}%)tYt*=tm} z{fwP%Dy`_bF3FrDtIgV>Ha)Gp(UFf$CKp@)@DrC>`H|><;qmbGsi2gaqs@;8EH+1u zXUBysd?mSyoy-6|plqBjh7C{rgVc8F7h~>!@tti@!l5s9^m~o8Ji>Q-5HM^lH zV^}oY`$!>pbV-O2jt*bjVg5#aR&JCTu@iqv7T9KJns!N!d~W6VXyUw}j31%FjLK73X`jDZ>LE_g?Jo)mgH~l{|8;@}dBA436l*4f1frk%^+D1Y>1d zXVHEZM^}ADp77Ejh06-9zvQKY(fJTWFBbxt$2%Wlc$9;uO6^C>Yvf(L8Rj1F=pPWv z`9a-ewlPXSh#RSekmRLVMuHJH**xiknuxhfnoGQGGTrikuO_rb{{$rAW1g4OO?YCr z#UBCHXF^Y>z1{YC(^`4i`9~!~Bwk%qICX?oIUKFHkhTOI9JdjEoHw$UzHw!P&u+oAC-@>}0X zM#_@>usp-Un(iu!pa@XA5pM6|q@ z7B@B}N0A+xmef{CGhjfrKAjF=qnP=e4*yHzxK9fU7#s8Bn1vcC5Y^r5RyOxJK709Q zUE);yDY~XT5?DhCcZ$;@(vCFcA?lSmF{%@|>37JV9imHdCDQ@2!H@NRxCLoM+?ylm z?9TG>?y~XsAGB6vt_SF=gM+EvCp#Yr2xt{jsKRc?`;q`asiR3GFfH!(ivfU>$|t@o z(kwa#9Ff!`oJV7v`l! zaMJbsfp?gF74=Ej0(Avt4b`&sXv(61DUb(rO*Gi{x_{pxs=37(AOD_Xu&wSc0UPBG z0P7Egm=o=>Ld0S8+hLladg(-B^U#Zkjq?+}jR2|7-%c1(@_9rWYrLmSH}eh9OuD1B zC{&ETZ(=w(D1%UPhCm`-1?`{l))n2B30_vrW-EiQ@$Pd?2Qicgd-MBs+=I0%UlxLW zF=14S4I=a}M2z=GMt1+IZ9Czoxp7@!x*G*b$&Rl#uv7G3cpOonxnTgd77QM4%Kp@? z@;WZC)E2@~ieC}d^pDq&F`f&l3kr$G7Q;v~8Yp_E$f7%nwmTX7!I4VFYGk2N*B+%O zcM5ZL38D1}#1IMHi+$otj@K+in&(07rM8iy^=I6TiG7aJp-=0CPSFANZ(O{+7`si4 z<+Dw{!&h5Z#&!4>^X%@2CVV{$FQehQYMAp#?FvvsYk6`dTk`;mZ;-nnP}n?f?VYeq z26&C+>+f;{Ek&-c$cK^Xk6jZ8kI79zFUh!u+5IL+waAPad? z-ss3_|MTszCm($+@x=NqGLA5~%Lcyv8F1bCp5Hpokq~zHjQNj^jaVpRJt$Ml^Ia#rmVI`T7G~zlIR+_SD=mty*E-N2uEI!Ad`8sBPCC^(f zm5%3Bhcb6!cBI~VNxx5O8<|fF#Is;A;M{N5EYh%nVO&AC^oVS+)A=XXIgK$QHM|OM zr!oDFo7G%V!_@)5qCH?b?;K>V#ZjSqx{G)2(SVZE1$Rrs zt`O2Lygx23S}_hC757AHKOqP08)cQy*K$2N(m6VsugiaSp~GxO)qod*RlT ze#Vw0qo>u9g8#MvQK9yP(|ndmprl1~0HhVrSA_`89`j+B$0cODV503t=WOz_tHwjM zKgeK#;eos50W@fpPYkI5Xt^9xgJovv_&xH83u$e7u&47@8&Urjv$*qN4(<|j19K0A zhmK~!B{G-BIr3mwTG9~be+#1vYht9_cmF?%&c&bU{qf_SZt4h?;zTHs(_x&-IJs=c zaU@AA%4IXF*RhUhOuR1+Js!j<}!=f`F+0s!1j2s z&*lBPJYUnfCBsd-zAWuxWQ!<|RfcqFQd&l0lWoPf=(sc0gA0q+TUO^2`B?Z8s%}W4 z?O6*vZmbsO+wu4M@DUMrciwfy=)3!zFE3Fdxk|S$Er&5172JkD?ZQ8Ut<+Hyg#7+4 zG4dpUE>4(nQ?=(_B#Y18Syh0rt6w+I2;6N7&Wtt1e%&@7?(@a96_%`Lm9&=U>^b0Y z{2h`|)d2@Ho^xDHU$!Vu==j_p&Oh$Hv4~a)?0z=KW;ydh$sLWsj=cz!X5l>8h4ThPP^Yf?^%bj22zO@ zv}SX+oS&*3n$J%!^REvZc|MYy={rj1o%vZUkFeCXG0_3d5blDd6!=RulVrc}SK8mh zqBS~(G;{+7rp>rZcu9OHbUr}R}(iyleRR4*KhXaF&g z^=ndh{#k_^sVr`oc9{#lI~hSCP-Id?ZelURn`VvTSQ^$xv|WH8mXH99L!hVPwSx&tCF_MfxH zG(*0Q2?P9l3Dza`)#TR7lbGC?myOV;lmjd+!gf-8htPNsAtt|&!NeXJkMLB>Wt;?q zK=j%7rBw>tl3*`A<%iSFLn|hI$cudbN0P+;Vqb>T%e!7tF1w0%^lG=Yi>odIj;C2V z^uj-K;^mjAdSNY?5hK@7-M;FEe3G}i{Tek{G<+JcuG#@F?(q!)Rv5U&XfyN(;ff*hN|Mp$}w=$D;Mif`Oe`2&^{f1$* zNGlJ+^9p{4IBpmHGrbY~%LE zhq{!-(M#3e|H;0be@eM&B{7d#1i)eCly;P+gQ-YsJFs(1Ni_x3kB2~P3=g$_x3YxeoTBeP!n*q{3P_ynnz?wHU+qn@Wh{jfeqQpL0_CAmQFjo1rFW z1{jc~z*It>L(tta=StXpW`t{e$pkv5BK36k)r9_ddH@36Yf~0F*FxA)C)G^>Qywx6 z&1F^H|7~N2)(C(J9ltTV$^v{r)1XsCSy;~SeT!2Xsbdpv7|_>cv{tlf&A&BBC+rl; z!|G*N+^<_XP0>Jf1J@2!$51{qY5?Y5%I2R^ITeyx(h6Sd@C2Yej&}TRi7gb}jOK1c za^h*nFznhh%zQ3)oYLCo1DAQo0Yp(U{BJ=2%Fin4e*4WPn`{$c;EvJ`!|v*@$i~HD zPH4DyF#mG@#RV5)QjLDb1RXsa&_(?!+@`RdwAa^qA~cs#(v}JokEAveZa`RZ-{|EA zhv?@5;XK#SkY;-i;d;1(In;wI>AL+h1vq)2PIK2xp6@`lvQ8=7#FQthQ@}W@(c}BW znPYC6MK?v`plmoME%0|vMgF3Go{>xD^ibCt-kiL7iw#WN^~@l)7+T;Q(`5obi*=Is1}K_bE+FO)%TgS2{v;V+gyiO26`KeKxKaG5_Jk^E3Ig z$<(pg^?MiBwJXkqr3N|NZ6bY)_YzlWD5jAbs}nyWoXQEg*G8?&~MG->p@r!vYLYngm;<0no?F>KO2Js#4{II!bu`2=5A{xqt^)vx0joZ2nVb?}YQ(b^N-}I^1JU!jdobQ-BO#Yo8C8moR4&|Q|YNCLbqN-;B zKg&|kaybAW{H$Yu+k|!S?EIh4oe8s97Lb?{Ti*ITMPirRpu~!dr!^uPF}dcs=Y8{M z8V89unNR12S_j}_UA2||0PzpkWt$adM{4*NYbjZs$6$UcTBefs&ODTa z>-`KtflAWiIKQbS63-4fSc#UB0KvrJx`oP;N8A;JDHQev^^O8`Us1EN5i(>No09T*Q=Lo`Y*3&fwD%$?HcKy=f^En|}Das;vAagNcmMeEIq1j%z0UMri@X@CBg#Y4=*JdW-G-1PKPkQC!Rlz!X%Tc!_uFxlAGIR~)F*!v z9fY}Qj~Jt`{JA{ViH|RJvN*urU39PSukxGJ$<2)Y3j0g4l1|j&iOyn0V0#WE<87$W zRO-?ve1r*n4Q%wn;oz*wG*kLGZT&3i&Coggb&l~%eosLoBSSJqP3ovhvZEwb;afDr zsKxsD5pavf7%=HA%7q>XU#1DTBVc=h`u9K13P$~t@t7Vg-QfJRy}3LSfd)|@I7VL? zp#Qf`S0UVNj|$M?V&i@RjL3jt2dHc}+mBPFQJGz=NUJDLizszF##!yf2yoB~WYd&s z810V%SIF#qvNXHBS+_IaTvnM}KtTP@p&P(DWpf+Xg>UXg>r@3=*6x&qlvUNMGxQC7Gi)Q^Z4pO;5g@54Dv7;_u@Hks^CC<0=0WSLB5r@>H(ku3BunS)QGDkqBboE`r?ca1;9h+$0e8l zoyQuk=gFJJzj*@5@%zLlLrNQ}!-@dKw#wZ_Z+R*2xoYN4;C@E97!RnW04&$8_yY^d zpw|jrF*zA6QrFT)XM2TJhv{`RtaZJyp+-l<2ECorC`Kp9FD#%Wve}$#KEJxWJ2`)0 z(T^M(V`uer!(gThwJi*FKxs}z=NJ}Xqho@^S_Xt$U4WoOi=i=Tor@(!bJ2_l{#Ifq zPGaU$pTBnOrQT;!j(nQm6gBB9O{CrpoMb1Vj)1KalHV~NmLX?{;)Buhdts{E_cWY6 z(-BXN^ZAP(-NGrKxK@>Tk0n*@rs->3kgT#q{3kGzNIfe8PxZgi_{uWFZJ#qdJ~Elf z9U`6N!_1?yZf-}0b?XvPQxaL5&__#cR`aT_ z&sq>7y146dhEMr{f2BzPGaqP_ss~C#l4w=c^Fk9Y{G6q5PKEmk{hZ&onHS-mfpM*v zQ{TnRwtTVkogZy8*EI3M>*n{I2dBfUl(09mP+g*JVJJkep^-OLE4@q+VSjWtO~9Ht z(VDV=9?qzpcSf0OM8rL6X#f)7L8%$&fkFb47}Cn2htIk;^iq{P(gDddSyYWCB{8&fIU(j zB%DVjwP-@Dh!-gx4d?^+c~3)EpnAglG2Drf=a%H%4NV_QN&mMf9SJxD#XLULGlM*0OJ(YzQ>*ECphF< z0V6D=r#nW7hh=7>zNOB;W$ZTOZ3J4c(8ul&ypW0&8__UX0o^xEn6;29%h%l0=!haG zV}Y~&C(0;QKC$pEa9Q;M_}O6OZ-6-QlP?oW6TFowbbfhWUaELFY4sGPqtXo^nsdN= z?tl^Q?8Bj_*+uk`$}(KYRG3ZnhDXAj#v%SEL*(?^irzqn7arJ~Q5Cg4`$j3!0V78D z)^5lzz)MN)`~S90(YM9_H?$7Z4Ex_l60)uW-eWd9m-m-LXuB}{g}dC4qWIMGxQ4^7Em>4@eXRHS!z z-4Y-Mk27=RM=G%bcfZB2q160ozFWoilNGzIE zj|F*;-X3c5vCbivX-uca?ix{*Irv21`dI*DsXkJ!4d?g0&hi?Mz};q^!r~k#=%$GS zptxTZRTpIJht>wse-7g>gw)apZX(S(?dUT}#Q*`mQC>exHxH%T+v^8nc6GdKC9mLH zvgDzBw`~yX_X^i?;<;fW=u6)T(eMjC@%SL9YDO0IXscT6=iv0t54P(QKR6E;D*JO@ zHyOLPj$`#x+``y+7e6v}#`}EQBwLe+ml>t58Xcay8+KOd-qg@E2ns?-F7i552GWY0 z#DcHR=+GL}=WdeD8;htR_Exc+X~R4%aKc6h1?_~)O9id>jh<9A9hPn!S+j>ev+Jrg zF*IgmcVydoH*_mcQ{rv=bIhMITy0N@#?8AoT;$C% zGB|UN*ADOOsuDG_>G5qIAwW;6Qm|_%{5B(tJ&X@(OzHDHh1XwN9Y1LvI>o^j*3|vq z4w{q$SKb0R(5qTI2(hgonFqc(0Ar6qY=UL3iI&G%DO!>~*9!@;?&y8Grj~)2vHgeNzyFW%0Tv;?YTw#oiTNplj_#jtdEi-5n~E!KOdM zu~e|RLic)_z_-s1!ZlF}GwC$8J%B{6dIT9Ykp< zbUlXySwOkOsV}eNzJ6FvaamDkNYuInuz^E;1;U?NYS(@%NZG41^NBn+_!1r|&Co*w zl7tcXCzVxLqO*vtqwu+wpgyt9pUy2QMF@S77CF19UGHZ#OEaB{{d*x4!z`?0+QNG` zy;`tjNTxbAz<#1K{7hI83tyV0v5W}zzsqc17$r+AOAw8p@xTrw(RrUv%jgPhiyGB( z^96P$<$Q{pS>rHHV#$@wy3qaoLqJ9P?5FE;pJ1?DI&G`tm@b#wULz$UL^4v=embvvz|`6QR2RXYGcDf(#( z(|ST14tdJEUydG{=-8EB$q~Sd+GaPNtWxEHN9RHO@5~KKpCP0{#w~1!@7N2`m}Tzo zi7w(RdM^X_kek#u%my8mK_jCUX*Uu%ety}{3T&MKvIji$FO6g^``(np-poy?ui{nV zY_^?cJ>Ux-A(VJan}8~+mQ0@nl=ESU6JB*>bHvMPTK#7HQz(5jlAc`QeaiAWj=ZSf zh&HWn&PtaRX8{LgXs6b~c)aL2qIXHVy?k~j6w*6$dQ-KyOjYfAu4samG=%IO?s9!@HY#-s9q5X7@1z>Li zei~z3aG`-*5<|k52KbNii|8K_dlCQU^KBYqFV$*Nq|7d%jp7YJP1Fm2GYQp3>%}p_ zjPf#Hl;4=&Eb!2+Nf`RjGc#}Zah`XH_dM*cJuD7sEAObF62dzi%X?Bu)UEJU{-}V7 zU>ZWgx*H(h7qV)r=&Uz^cQHR2SEhphMjKPt(%-!M+zEZPbN&7ux8T|fU565v)wJ-W)5Kic}vkC3p^Nq)vb=X;E04++q`h6JR`PBs?v56JXk z3vmMbe>*1ABHVdPaeHPdJ8_Xw(zquj6IM?xL#nGAMkK{K!ZsAmKP+e861w;sAKXvh z^@%YUrfc)u)j?g}Z240`W2RBCq5v@Ggu7+gZ8$@sNz0I>@`CkX(X8r6nMWf=F0Ucn z+p{`lcZvHVY&|O44!lt9UW)J?Tg*}Dze%6 z;%P?G&DV3$dKeh!t&o)KGBQI4o)uV&J2Uvrqx%C8?l_!@KDOz}@b)}(BTJ`is!p0VL=|+02PIaDVU;gBk z>U?uY$Kms=_zjbZTS}yi-HB;AFbmyd1u;S}6ODP;MI0Wg?vBUmhplI0;7@H`QSuu)WAl8q zA@I(9owru6jD*7Y$)QT;nid48m^bf@>3=> z21jW<*uM3Sg%r>7Bhe9hO}w}9)wW;OA6Qv;4@;^RXnD9!z34;-aJkU!3)cPez)~UF z(sRisCnV8u8ozvJ?G$zqA4<)fp6+Dzl?bMT^S|z>Ymi2Fd02JHDca#pKGBoqpxCxc zFI>z~z#LrIzyzWqdtHi9#CmQX<;Ok8tIy|p)P65F?Uv&U;{My_*67#dBb{%bFoOJ~ zxYhm#O_BN~S09O{*Wq&oMp~cEp5(S|+=d(Fv$6Of9f0-Uy@x&Cj)I<0et4~5@IEYwsx%?VrSpb|^_sgo%)ePdi>A zd<`2!Fznn+0`HRVLSZw49_sz8=6Yw85uj&r#nSE$M6<0%icnzhdl2{jbU@yxUNW`0kEs`eO#$SP}h%h%F*ScQwSPQE$t_lP0&k+I@P< zl~CIwP3gZCAWmg_4Rg8b^Ny$v&(W(( ze$;bC`bNX;cmnHRxQb|SAyqKfh zE~VsE5|K?ZSR#JnVCcG@ZStxD42tN`)wBGk*Rjz#qH3%&q6UogJQc(l3E;=}M81L` zV5^PKcPdaiRnlCPOe;}ks|fc%bQ+#WG)nvnl3c8a(f3wefHv{Y8zWIJ?7>;MxjI)de8OU`0r@ z55$a9nC%e?j(z3u{R^fMoO06lYfF8maPV#R2PG&H@?$9K;A;HnhPRZLN@pJwjE3n^ zl9cu*sSL*-r`Io$%r0>BTxeNW(x};%px`K()zjCxo3U*^SKki`C3v7lZ9=)8Nhsuq32}Wp5N>yva^1Ua>&DVmtz8iN69|C$5ct^vw_2+-6a9LIqhFB|u=u)AOYdsYnly$dF7%H;6 zSo++e1_WlFq7JKaEt%cuuz`-m}_ghlz@`s$bm&LPa9V+C_E}A6I_G*l2dRP0UI^eWWqo>~XCk>B?|{RHrt=lAeq^U$ST~An0$jFpyWPH zxUcRj0sB#lFul1M3$u5ql&ike*W)drXVpajvB~PG9i?6CCq-eHp`PNVsE?$70mw6< z>GUdf)|)bacMaA{Z1F2g_+CpmP~c|x1su6ipAu8?I>Sqj+c^u}LDZVWVyz$KGzNm)sXZ}pTo(iKc7gHoPVW4~O zVe<2gZOiAHkm4T-#@wZw;w|P=FyJtU#hqlahF>RFTGD&^ge)uC?39J|4r?0Vw zyM)7^*Nwl)Xvnyjel7juM}WF^ypa~8oFxiAth06u1K+zB9dP`SH=v1E(ek`}SknBw z;-0PSf;lMPE|!$Z@m{8V=T56ny#J6Bl@u0!gBxuz+l?6J^VgRB) zrpRhD+=EjmJn|j3uAMwa-a!6e&f~aa4Vx~i6nkOZJq{{eE1@?- zO8c7uF{x(^Pi)y^*H#bq7?0D=0L9R|!oAYz60uggvp66L)ct4y-}syA7>ch5^sV6l zGRBF-&RZCd^ByNZ3a)UgK;Wn#KMRBF*rLs=e&LIj^7m>~qMJu-M*|R3Pa=AW-h6Q% z512KsMId#rRsYD^(ob+VBIpL9vlPyLLzLMKm~Bup=xAFC3#ej|g$^_K>N^qvS zFBE?Rh?60^d!QN)(UZ`9^C~qds~J=B&4y(Oj9bBEm*nuvZ%BbLCu~whTRJ~ke;t}k z=v6_e)Wr&Jyu|!mH~^@A|D-agW*bFp+nGkf5l@Tkeb*W(J3_C7HXQtR?_a(RrF@>d zVI+Sk%qVqrEN+hfWqj0Q7eyZQsk#>=usBdO#v=8#l=X-yi5o`;(K!>qEwFr+-;%Uw zMI)(o?5#k#QxWB^u|>>8x_)h;^#;csRJe3BEjkwGkctg*tjc6x0m6SJZ6Gyq{!3|# zNchMs5!Eg<%y6n>u;`pM+(EzGkR~34KLGAqWt5}e#dvl`$y3pw-u9`JZw~i8=_Ko> zs*V9YlKdYBS=O^cV}?vC1J?unyVr_~zVpM>eVtTbzny3S6t`h!9a|oFlwBY-c>i~ z<1t{dh!bbZhxp$?H=1cF9%ts3V{Q=i&RL2wLl^H<9A9nOrTl#e4AXW3P(8EppY6P} zFeP)GZ#j<7qCk}{q@csX2i)u^?X9Gb2?h|yvyu=m8{0E3;J=?gy-f@H9(_gjuj23* zn7$&l-R;eso(1A}$Cd_h6*}u60J}Rwn#2rN-g;INzV% z`xlA^d)#yYnk;GBAvOn^wz8I|v`r1IE1mN4GNi@5iwv^eebk)ouh^6!_pNhPZ|N)= zYdCFaV$9{}k6a!L0AF}<7ib2%tx{Pg4K>4fti4|X+r4ImzDMcIyj07~Zv&bSFK8Kz zaUrt6C44?)SFP(k?jWdWtM6qW@uuuiCsY%n2D5qw0R!cu{+E-`!-Vr#jPvO0IlkqK ziU7blwG&+AT2RBoZ(!!JWic^32YV0I_U`ob7k~anUTtAhnVsV*0c^#s(Z?P6)b~KIVZE!T`cK_Rp#t_4#_t%uH<}_^V@rC>e5Gj z{KK8hU;VeL zZ!McOjv7?q3l&z^2XQAA=Smcrd@M$_zuC*Jh6m~^jLwnQ9$kMi=Jv+V?XJFMRZw)` zVR<=$|L$MT_jvS^_(8LpZC_b-O3y*M{q7-XD0F`hPH*-e8Xk~4HTxoWzL5%Tr=YX0 zk~cgw3DfbC@|O+t7uIW=dRvvht#0P}nw6GtmCb5EZ2*r95Qsw39Zp39oIAPDn3?O$ z!BHc7uhrNEzYG%FBTR-sU|)$tkM_l3Jpy?V+?>}`I(aTfm@0#bqMzWVWi9PnUSixs zA~t%fCAOws5$`*28By8Ip=e+#V8EVM~~FEm{&qu(n%`=?5a^+ zrbk$>+7huhIOF<@DYAZTx6cNw!{~|jH|}2qOlQ329sJ40>VaEy9vBp+V*Az%4h_wO zQRy7Em3(32$<_*D&XeyEu`~7X_yQhSw?&p1?Z^DLZMQ|!36UU{oy6RDlJtzO;qH`a zr0$FZL@~hOne)(??}^=2RQv+#$Xv24pe=v(uo2*2j_Jz%cZzF^9b&SbbC7oFme-zFQIw+c4qd?ZNW{IalD3axK7UaL+?zlq#KD0p*MgELn zcucV+ZRw^QY&x~+Hv60a=X0FXw;aF~bNol&?iaHLx}$F%HH@N%+}sJj4;~R9)lcEf z-13K9lfu4xOd0KJI{QXx!x6@!_8=8yR@UY52j`=NKMuyf2UBe}`uU3*SA$o;h!9%y z3NQaYr{@aR{>11a^UNhw}uyHGU_j!E2 z!RAryJ5zCJ^}kjjkGy!w^Mj+gFVjy=_;PT^=qudmn92b4)c>|kMBMkMXyRLrqJ?|v zUN!lVDm*Geqk;P@)f-bKaegBl#mIH|%>{ECAFY>De=EuK3lHe(?^Xfaqa_bS1LU*< zR}w({ITjiJAv^AnEU3R$Y(B2_C->%iR#Y8A0Mq)$@Rsf=95So%1FIuKEx|!~0GjdN z1>J={jXvhxnAZTH>A!7{=q*X*;y!36uRCqWNU7_|87=K_NlAWtnnWe#vF(-st*dD zSw-zvOmsqbG@TNoo-|ODCjyLQrW}7h8bInoj2onX^J(g$3@rp=qR$+d%4TL`Bg2C@ z;!Wh+r5Z-c_+Gtlk!VtDk0f%ZJJIhYK6&eiF+-61quIDd7 z+>_NTRwFHCbV)+T+MSvrF{?WbdX|sp+KZDOGzAtC&ovJNSPiNp(LgtiKHCJS86Z4X zr`L-Gv@3ZE@P_B1#jp5ze5sSivsT;#z9$kEBRT-EfhSpvfIM4mRC(Klp zF(F#iPvf7=rYbTn4Ijkm&$9qLz@Vs(HcX|ishe+-P@4vIXppzPSes*QkLlR^4Y)m*??SxGyAh!<_<*-f_FozE2+9YVt`ferY*^(+j)RLF2j0cJT77C(x5V~x~?}}s*%E-FK@T5$w zBh9D-ZuC@AMP~aZ0ej0#j`Ky%R>>vg#i*4N96sSKE8iD%y^!-Zub?r<{qyhV#t)b& z*1o)g{H$2IiZUd)RBCs4LyTm*noa}5oU_2OHp29A;v(M>!j)rUJ3yQXiBag({&!Tjcx>TawfR%}2q-ZXS%BrpkL!*N&BL5AaYN7cx zS%0b`MI7B91|de-MA@Zvc>gT3KvY!ndhl%ZS=AnyTJrw{xF7(ug)N=sVbnVEuHNK0 z?AS`|#lLNx4j<0vf4IoAIY|ANj=iX!8^j4B@5+}vu+IY|JTob;YwI8g)}Di%c?$V0 zY_@0)Ce~65^CDVg>Fk;`HN<^MMI<}qvr!P7N^AFF$C z(H1Bpw2jb`fzpiocS5_#{$%wR1D!>W?YMmtLSF^kksp`3)zm=UT6}SrG1Hg-6?Tuz z881q!P&fUg+mO~rldRz$0X+Va3*5>kze)S@K163Pt^xt)W5_kO$qwg^ouje1gQq-QS}LkR}Q6%w{m{FGwLLp?Bf3h0EQko zAPVPl7GK_tC5fyKE0W;cCWrD6oIKCY9=Lt++3c)qF{c(| zJN4g{6O8)w-xL{pLs3RV64n-w@ZU^?L4F%r8@Gh+S7am8`(3I>05OomkX>iIc!$0! zh>@awPXA8s!S8;9-qDT*BJc=w>5jh@Oez5(w*n$F7OTzMH`~wnTJUVh{5)=Msc*MK zLsu0^G_MiiN*@kkk-42AtbFxN1cfZR`RZ*waD6f7iE5K+aUPB;wJ^KKc2a300O-ec zY&?TXDRX+~sdtO#^u7XH<%6F;tPie|wDf_x{m{O;*`8dPO-JxqX z*&aidBX8bk3cQ#J8V9HC>yBW}^2I`Nl$^3nd3JDW>;F_I39)b~X`;hJtJ7{JCe3mE za}&B08DIj>ua0?p7Br^YJLUU%;!-oU64>y-5+Bb(SMXyJfPcj;tI1)0__G(KVqa|wSHChsqy}&$e|GE1d1{0LU zUOB#Pwh1pLVY*xBhXS@`h&8g?VcK9`l64yL&Qc-r_i^-J>?(mv!TGHC5`_D%tv>2- z1O1szxOFa7l06p=v~;il~5TkG=1T&)FQm^jPneXd7( z@5NMD-InUj)<4WMsffadH|nw?pSzksL38)x8mVsh0sER?#^4p5gco*AMYK+c!CSek}&^j z>uc5+25ek;mRQ3IA+>F*Qn-XKI)HcQfx2GZ-O4T!qUFzlvLd9O>f0)(7@LpcQ_C?J z@=73?f9^}VSUwXK(uD-}I$eQ+Z>`9u6{o>2RSoj*?7jPLKAPDJqD8|QdC}-nwt=ub zM!y>*EUIuZ@zK}eFRt`3R!;tJS1Pp#m{o1B9#!fq#=2W6dbGxRFvc^ay=VS< zM|Pjs5%1%kFoaL8i2OE{vaE`YO85=5hz;mnn85v|<<-&zpq-RgvC_>iYf~-lxPxID zqLzJOvT={Vs#&{?M5=}4cw?%&u;J53E9=-`vN`5y5dwSm6Zzx0w=XM+xvIlB3I4xr zF7{n0#Sb#||F&h$bBQjKNFz~n280uhrvf{L?LH%PEp3F8NMiKAu#_F!xg{aiym52f z(!JWD;poA)tr{aEV^J^IyEq8fuLwW?`$QXWQAlzjtg|8W)_A_|N8 z-xzgJZQHa7sckCbA?s90Y=@zVbd{f-Sh2d*;9XxDFd82SkdiL)ns!%7}1^+|}+QkrmBhi8E~;Aa(Js+V8-?9F;<-Uy>=UyeVM0rv+^_->_6 z_`qL!Lq8?kzpY^I4Z6|0UD;VqD265H|H5^B(qvyS58s_z5-K#4vLAyTIn|EmkDx zx;Dh`c9sDRs@XH~!KgW2B_>w_H2*08BE1_18tdvmEgkft$aZzH%&IV(*O2n*<4^X5ynOiukSJan@ zmO0@?^FD~#eBp1!zEhc=URSq?9ZV)PD=NMMe597W-IG}RjtHg_Mkjf-#aYT6a`G!$ zLZo-@eG>Y6jg1Mw!;IOxI&bzJfZgefmj79|3I1on;YgLjs{NF#u^TI@I-;<0V@Ney zNOxWc|5qc;W|Dy*1pw8uoe1{AnT|XzzgNxgerJ(-zn^EbDLf>dZV~r}V#Af75<;bZ z{N9OKJ8MIeJq?g%FyZ-XQLgf5b`P@iqSJ5QF|awUm&Zq=JxfMt-eVK2LRH@L$ z+tMA?on%!VxSa?^hBpm}crQKDdu+a|ufOPBYj8tG6326ewrMhAUGAZ|>}YwR5SSf0 z&)PJt%3uFQFhT@CQNV&zb9{%P!5|PCnH06kpg?|pukMq=+StAeipbgf&uoB1nl#PT zLe^nc-`LQ&X_}r}>RfXHjQP6!xgClZW$*(#CJqAYe_rAV-Q(4RGt9<%Y5GevpmFQC z6m?~dqXYm?eXZz2jjz|g%dhjVGxSIwef?444}DNmjb(y!7JgCqGkT5 zar2rxV4s`Hn2#O&CiR^vmE zHaWh^)pR21?a0}PicZo!=3S@Y98agvn&zx<#Ozo2wkd0kv+XUKkuj{Fm48VylB~;! z1`+Qr49wN~mb(><`N9czD~jp{BlAKwFRCN|!1Su85{eipeWs71DA*moaD=nW>XO2| zLXi)B(BPUDIquvR_ap8AvQ@OX2Tw&@L+$S-k#NPXHW|d6%rItD^pFtdfV#s_4Wzi_ z5Fo8+qgOq|TMBxJ*dw}?q6sndA61$uz~cDBv(+XI*seTLqc{D%P7axhKiU!;3K=c& zuh7sbz?6O0wSSZT(Y$uXJz!9#^`(m{Giv9f9^GC!`eI;11x;^HmdbzbK(8TMQS|0H*lS2f_PFz={i5!1fY6IY&bvGI@ zM#&o+g$+k|m8gqD5^Pgv3ig*#>r#^xU<%Gv;7BXOmVxMx*phq=1xmKOGzskl#<&$| z+!`12)H>tP+{PUnC_jhvt~aOzjvDiG-Hvj?JDhPLg>77l#p>zPhY|mBY-4nC#dt-p zs3jr(uoC)Z>+0&Jer|>)97fv7kIEGVap`*_?L$}D)$ZsSMm~>w+L4!&)zA2g^x{Sx z@}j$zuMFC*VF)!uXE(4PS~?(K6EAbbnS$%A&~W-Gde=~)jg8Um5n8<^UOiV|KNy3? z`+=;duCL%GRf8$+uO2vX+EqF-+XaDi(=E$?xJs#ew|Lxd-Yz?BbwV?=!7_zmH%0&3 zmN5WqJUc~z#!jQD+KBjIT+HF%+>4p(3)$ud>@(0q7_OpflYEN{4NZ+sD48ycio7&R%tT*OBw2772H~*8m-zVXngVePSI~^t@FP zhU36TBAc0^znXh&ldX51#QGAfb-%;cN z_9PJY0wfx>R;V`$i18+=y);ujdgfrVKLjev3?WG$QL3%MRf4w z)m!gIZy{QYr}YVR{Rf(yNed%gYa_Lc#~UEVLvVM;&$uSr7_ok8DG0v@Ou}8v6eG6- zUF8w_{6A}I>0SLgD*@B$b+;MJgjgLqGv6a)uhOMSwofm*3%cE1g!@T>yEs^YUSn$u zTMK6m4P?}5=+$4rewa>jxFYNXZT(y^TNs{#%>z*=hc%)wobN^YQ` z4pxDi53V>ha`5hKHGs(6jxWBPHQTk+&U4d!QHGpA`ojf+f|^doM|(xw!0k}mI1}dz z!33Y^H&|5xtn%!gX=b!Ea3;}<*&cPnH~8u2BO2#P)F#>5N`w@ehVNw`5W*G4$lV))@P{@O9JQ;)6=!k(@Tcefl2VH-JICcw zet@fP&crvK&VX->RT40roFOLlr^#|#DF-4Apk4QQ0?>&;u;~C^-5zN;*>M*=*rN~n z(~&Vt%q@36Q2;dBium?B(?5@@NZqrJ?YSm2sm3;zf4QH>PL zFM*T|TSfF2%S~UiJ!)#ne08W@Y=#XVpZ?K)@BBxllax8B^~^!7a5RpSuL0Xh*wL;! z!NIDG6mC`)c=P%4Gz6Yb&YnSzZxt`9qfcb8Yh$VQk=kry_C9ZK>7PH;PwtH~Ep8U4 znST&!4iFtlqVbDsL?1agVc9ow{NB*wIQBLt0AwS$*2E1D+3(BWdZ;%|I_r@scb|7# zs2s^R9a=L=4S>k_KuKKY^JEch-)lhNv)}rkXBo+M7ZuHw6$Z?o0r;fN!hb*`yPEqt zymbaU-UC1JnEfDn&sxhBS3u;@BV9;~iPt@Sj^M7e$rj`090^UH3hP3Ux9`7B7Y* zi>!7Nw>7^9I7IE4m6} zbn`>CQF!Fpp3Uv!N1Dz_t2!DCfPPs9$oe$&=^a|jepit2MPp}e%)nfBzS-dFCg~4E z?rLI8fsXDV5~+xCGesZfFhquLess{7*6}e)jqFl6ONSJY#7ALXc2`GJ(`;bi6Bvn8cgxtx-inH@2Pu?|HD2Zx>J zYfCO8M%X0`G2=+CV>0&%xr{B$vf254e*gD}9%i4<`~7~sp3gV(^A#UX!rBNY@xMS# zSCe*iM3T=)%DHC>%0J7f_s~0qOU^t#?;ap>9iauUmjNBKv!?=$Q+h6m(iS0O4huEF z%pvSebGK#-z+8$}XW=6Jd7AQ7*}oRAUJr2FD+Wq3R4XD#W6}AA@JX5|zobwKu=ZxI z3DX1W)q2pjH)C(eaV3eJ#v6KFZrA;({$+Ky@gU(bBhh1B7V^QaYR0aL(=1w5!C$XsG`<&G7WmO{ zgzJp=wcnQ4|8aNWi_D4!{8^fPmdT}vE0JO0lY$8DX=QgEsvLTD7z3RGAHr#ONS_WV zPstW1udwqt@ZYqmgWRlrQ~V^9{E`LhgT!w>+m%sYlJD#X{O>=bUd^TmJ&GSAA$zj# z^mKLTsf&F_@LE$cI8$+h``dGaN-TDwBm9FxZH1Z7c1F` zHNJGXhlh`BzNJh51pd~5QXlUk^PRx}o_T<*$jrog*j_4iHhHc%XFUF9B(Qz5m1SB6 zTFL!Z_#6D(48qg!{aRc*f1u^}9~KEetovq|PEo~0O7#J}e;;PqdL_?$GJd+X@bAD_BNGts$%Wij2W!p*iwZbh6Dg!A20}7|@^6kRVP`v^n z2JAM(b|~t7Sljvz`3DTe6ky3J`38tQ0^JFXF8z1J=eXn$u}pyyhm<)aS(G9=q{HOX z%svG~it0)Wf_T_M{VR1wDlIN8$MGW3y5t4=2lFrVZyV=IZX=Df*t?2xg_oPe`_kxO z-KFSsqLAkB|5+?OBPFky0f@(6Bj;Jt%#u zZ-d(lfT)~u{}e9y9x?81UETjxkT5wO$Oop?SY3VrwWV-@wTV8k;C=)MBs#sjI=C#V z2uNGszSA-i^5rqv;ZHzXhIs?LGt2i5GfuEwrqNdbLc&~aOaPTOTOOpvymclS8x!_+ zY!Qy_dAliiw#9h)4g4yCr=o&%|CGMDxO!9(*T1?Ua&Fsl<=08BxD1IvbNvN89hkR{ z$FQvdE6ZynaO;NH^=;=7Hn86tT%RsIM*KLAJ*Els!3gz_pvY&%cW_r`dQ(M^-aQI{ ziNAOYr45W>4+Z0(2Im74v&6ga`e`OmZ@n8{f{KO6@-9%(UtZMDr91VPC)1;0=_^(C zx(m^601u+uV0cQnE#nsdKYlP?Ar$x)_y zwf7}J(wrS2OwCAE4Vy;8-0G6qHngN+`ak$9hZxhe>rhwSic`r|RJ6f#gfn{A=U!>D z?RvWIWp84WI;I$ZwNCb`aMJfH`BdMAtI5q;*{gle=z+g8YQ1$}?`wE@PO&zfdLiX| z(_&(&!O1kX{2IOA7HXT-q1$}?vjSW%$w*S5$z%bMHj zeP3jOeB>MnVgX)4jZ$TnJ(8-{4?s@B}?+aB;*UTOnNn^ zkPdvHNqd9VtCEx&`Ue056?OVdF%~$~pIFp8(loGPx6f6gmr0emqQjTU9x1tXyvlHq zN$~KkAYx?109|vSo#)-&*NOUrH+u7ju8h67>%Iv-C&bQeB5m)Z|LtfIgd|CJ4qv4+ z-YdKME46MgN3fNu4+Br76%U~yI-bG&tFKh9h)fN&=p3QmM25d5B}RxyYc>Yb>ONxmNC>#U!!e}{!}Euu41S1 zRgIFkf8cNRVOBB9vq86dD_Y8ywlWecFf{hB;oBz)-wj}@jC1%r*Fi(8GDGE@qJ=ur zY3jU#{;hW>D?x#pPJ?yWTMXg%m*Djzd(*bl0*(>k)aUD6w3{eLLK*kX0d$G zIo*TN2~KF`o1U{qDN(>R_|&d3y0`N+KBRZ2;sGP!I*)xHfSX$1EI%MP%N(qvcf~@` zaNNX->gCw$Bf+y7ege;hT1Rx)>3aFz?JF3NSPKY49J0MqF6$6>Gl@d*uE2W6VD!FJ zg~{8Nm^e~#d)b@7m%!6<^OjjPs1^nU+MqhX$G0;EvN~cEI66x+oWZn;5K}&N;`a;I zk)Njp> zKs@q31%l4hrDwMi=xWaxK>o6RlMB-V$l-c@=>;@1QsVRNvMNsDqBfWDiBA%?JP6%0 zK@1ec`~&OdZ&pW!mRb`I+p7}j-i}?oZ_3$umj}m)xcfRdl)i#0Oey9YjQ8i zDX(jI6^M>^?`@5$V5r2v4tdkE{;h$PjYBrx^n&s~(HfXn2T zOV&Bf&oQsXlP~D2I~KH{Rxi%*mPM0p$_{eOB)vCy^U2uQQT_J2C7IPgFZ_1|Z9Qx874c z^KrtBSUTdgJ8VHSsM(KrojtO1lu!ie$=)vfa%*1!(%}#xpc^yQL^*j!uqePtf9jmXlz0su2+Om!or0uRS z(z{<=Sq4rzlVXzwz5!fb>gD$ar)!vUBkkB)NAn`$Fey^{KX^R7w6E9DInw@BjLD1U z*3|=weUiU{1ZbL22f~oW@$G-orX+(Ns2q#{ES1yclnYKOr9o-M6VAp7_61>n$k`~% zT)r!$ySB1OU(?}n^HZY3VJs|J&!C}JvYMXT%c*ZpHrq%ALEGxRO#P?=izB0+y^)br zMrv$r7!>qH`Iu}}SJyyBIQ_k|BYEBTPrm+f;1mzjz+aF=WJq?5F*D8UGlTS02So16 z%wg-2GD3xY;VIw*3Kn@Dg~pWqCU&y8W5Jh=)aSkZ@>{te)dX6HakMUW55FYgIklPh^*H&z*_nh(&0GIP8u#Bcl3<)q& zmf@T^94_f*MxhLK%}W2l5PNAU9}FjPZq*FBU1UTBf0CH(>n0CeBH*^XR&4}T`3je2 z*@LLP*#JL9?{sR5Tij{NWl3UH%-iiZTfzys7th@2T}`^3<>vL~p^6Hn8*lZLI}!Fi zIalCvO8x}-TK|r$&_&G83q{Kv0L2XQ75;}kY(YqZ#vJ9?`307At0yC-?CV;mp%yzH z!E}YmJ&jVIh&(2QxVgL+veX%!Dvbq)BXK4;KR8pCeF}ZWvA>una_excXQ$Zd-0fS= zRkp6IEXL%aC=Af^p09cG>vKZE{$-P_(NAmL?pFu(y=4jVi#X@D&;?bkXB*x6jtRH2 zMqvXXhGZ$o<|tBN2i!|&qLqR&*zQQEpP8#N?NEOyusZIZZa!oiDrQYLGJ2AMG}*gx z)Z`Cul3GV;+$Co5cE%s|0~|d@Sn=quO->tW9n#174qM;pGSCB(O+;h%?H%i@hk{73 zr;49h@Y%|olBGC<=IqM@H1@fG7^%Y{9L1$E#c}#y5eYDcfY55>st=4ao9Dn^}So(tjEa>v0VCD{D zJEGf_`3g9XM25a?@$)N1=0btx(!~{zrJs$9Ki&D**K4;UD&VQo1W5k7xA`V-Riy%D zUimIgRKk6_B%->^r3z)GXEc;|11PLOx~~p&$RX>$F+Bm?ULT^HiXv}|iO=wWD*gBZ50_<1 z(MDST78$)ld^^rLlu{NbV7u;_@#>X-tH(0P#W>q4M)|+>|5N5j1~%`ADXHuG@nIE& zU>wnz2u-~=w6dWo9wXhPKp3-J0XLcyM=V)P!|o;}<^;s?MrbcL-4?V3-d$l~;P|+H zt7eV3J$P+d!rlxjD_(IysHXb0j}QQOE;n;G1wLg+bO9}AbZl%wfQ)7)qxWT-ReLQr zdH%kp)t#oGzF)H-E11BZd@~BJ;2+7x^Q!dAfgHy1km) zu)Q8?{@Q!--h@)5@?2lj=>=4GnE|mG)e4e9Txb{(L&!WZCW?K zNaHmwE?%D7y7s!B$zmiZjg`Th7dj&P2>)8^9~f1f67v~T;GOvEC{+~K{;cK&4)9L( zHSW+LL=_6nxXIs8IsSlIs~iTpf(sgVVq%)lBN$A2U}4HO&31832qd^qNPra2!8kLV z-%YUt$JNklZNsMfF4xEKJ&Qe5*xcBVh`P!*9B8KF(|qm|Z{_^n^k)|C4B2CbnfBLC zz&p7_Rd`7-X<*xn6s;-?plpX^y17&k0=rpm8Blgz_HB{}`yw(OY&=xfC0V=4Z!*Ta@bH&nXt+o(!jGe}yUHV}-Nu*;hp4+ouWk+1j3rhe#u!+n2Y8x7WlvG`ptR{tT@8q>^X;uZqzL7;j z`5+qDe=y`rf_7HNnc_Wkioldw6bg6Q^Rb6#rkS~d*OM((;%tOx0N41;zGT|(;>{QS zJjEr*dam}~Z0$vU^{9x+FG0MhT;5Qz3B;@v#mBsQ_^ndYIAvhN{pR1hxZ!qSXGv0O z_&tgD9ObP7E+!_evQI}w30)YbRVKHR?hrX{qLaL~{>81Ub)`^ScN_Y5z5b9D4TegT zA)Q%*x2@YQo|0jwRt0J;k)+DJGFd82ygj>K&;62tF7tH`-TNnM5BGTwxqWNiEBS6` zt^lBAq<)ke5oss?x)Kw)FPVLAN?o|R6Ap-mea?w*<`v0q#@I|AhgCqQ%hSvQD}vpj z;!0LI!m-p7?)^bp0~s$I6_~{Z-zg=F{_&P$xM}--6>p|J;6VtQF2ZrcHnMIbkT)2y zg+hk*yRKjlG5!13jtKnbHFEH1- z418Roo(PhN$WZI3ZMMS^+b9VThgbVEm}-A0sjc&k4*+BI;z0metgqT4*@vRrlUwbv z2o2NS*w~;|=ApLBwMJ>uaqd3B?~l_*yVyW?&{5_nra1B;+hI4w{u|U67WszqD_7YF zf+)=r3ar-@SM;cn0c9i1XI_`{!YqN4OzkY{;6-l+(# zKmpx)v7D>?NfJ_5qOf4~{##$$i zAVKl47JNhZo;XS&VT;NnQ>p~jNMVVAdGyGlx4&0pI>4~e!u2WfR7Ra0b9>(fmF zUKbBO$6vX_S_|si4g)?+`ctc0mogIdzn}YWM_2uh{{}Z`*Vj4#MA;RguD$M+Ho_6W z^W}POU8MC}AGXe{l2KLd)wsPJcMACVGzW%nc?wa~gzG8Cm*;_vXM;!4HE`&%QfEoI z$5$73rEQmEKU=MGp%zKPN)w7Yh0s_VH+CL^Iam>tm+ejDz5i8o(K}SKE(2+2i@&PH)92h!}ifG z=Babg$dhR&+dqnQx@2m?c@FHk-VgeR6z0vJCv_NAhh*4cuK`BB0UGR8!z04zweeBh z_|KJ@S|jA$xtnblI?v=E3-xMg^7!Jp!@gUh+MXmiFwxtpuR#UWBv_i8&DA0GdwuD} zCB0?6Zz0^AA~a|@42Vm5X}j{%nKx=21VfeBt1k0Zf5#oP|MO5`DgLrGVi8=7yTY+D zrZHSz4`3j%XP^!5H@MY9fXQssJcn9D!Z(i$2tF_;gXl5Fi38WnL*i)L1qx)W+};r- zTzG^?J9A5rTTMktT4F*WJvkLfI6iu|IHv)DY?#lAK_zRfnkkaCRo)k-+;5JbpPoD> z15Z;E8JK6l;p-#Lfuj{s<6OM>1hFKD%PwAucHMkoM5oJdh)NbC1kb*Wjg-`xUeTaR zAE7UtZ!!M#R4;WJw(}ny-(jpnJQ*nWI;bTDV?a|4VugUYdIT3HiZ+Xwfess{Y<(Mf zJ;b|3Ut48w!~;Rau$mDXt?>`VO1#%DsB)FzPxj=@5_%Wsb(GOscA`8ul&rwC8HFl#2L|Hn6W?JP?D7Le0x3_ zQX&aRnF3B|N7VLR$LQ<4wjb=6m_BC{6cf zQj;~@ zmdq!cO%0C$j7Y4GBVP z__j5v4#i!7iaW7YI=E~YOO{-Od)Twp>Uv(YV>B3xzTm>h9oPg|4LQrgp^y;%e(HZa zt|bmUPA;HRy6lhOU@gE{*JPkDkY#{V*Q<_p3-1=R2-S`(kQlR!q0JYzy88JK&^AF# zLL;w*>&rzV{Q;Z=<3CDXh-~ViEz5&A#DcwyF^f{0AyOg?hD@ zH~4;l--|ogulx~)nmMa*nsL4B$hUQ(PAtNGlc#|h5$*0Zgsucpnf^_HPcGqpVZV`!(Buj;U-@MZTEHYqlkrna>;;6AYEdfnGXidA6{$RdO8{o`L-yxT4tvbu_)_49(NQFckegP0Q6 zk})JJ(kf!us>X%dBXBCI)1|S+9==YoPbiO>B!@GE+8~YUy%dS*w21oL!tONQSuBV3 zS&W0PiW$qKzb&8OCOzLLT^)Fs8*9EPaw7%gAOq#WtA@F=eq_XZn)aMP2w6`MPjV*r z<0(62KOuUxI?u#2|B@Y*ZgBv&&*>qDT2V=}vL*iFne>lyFlKB?Os+p~#b++C;*6*E zQ`^lc&F28}@fTb#Cv!2*Xy7rgY6GSwi~nwBk!Q2OFov3{_`d77Na-OP_fNg!6*usf z1DN0P(V$f`KcwBRX?-q5C9cosI3^`JokkrC@-fbG*f;9EsCUxC*5Mebl^YsE_4({v z4v>W-|AKpd_|_ZUw>Rq}e-F@8oXVb~q%3Od?uT>+T9 z3~81A%RJ{?7^ecoG`UGl66lhkks=vbBs)M&8$`gXOo}nHX*ZSDH`}L+@(zbbea8(U|Y`cn$ znJHf~^<4iBObeq8qd1GIqb}$u>UeCqO9S155d%GwvSR0h5xJ~jQH;!~CP0nz`&t12 zWXUHS#@`3jw)Q(IFREeF<8V;W#qiZ1f>MvlU@qy<2=c0H{5*7MMac;k1P8N7k80by z0JoLT!%pLz?qSEN^5$gneu+KhkI7@$JYtlwJf1^zgn-pcv96?wIA2q{@)HAh6f}w zpcpWl`UU1T(DNVmwr;vx4~eKF1$DWma{YJ3Bp34n$ZUQ|ajb>gFvuuX6-_z3NKs>3 z0p%!7Syo4TlX>;~)RwKdvbRXf`ezf6T#?22`AsODImlSGoD=X}n+4s+q(5&nB0n~~ zZbYQBeicW4pqU7D0X>xtkaUk@NcMST0a6oJ;8g&L7#9eBtDno>rUk@8K53PIX(^rD z&YAEGsP(7{QIzKySNyl*2;QN-&lWR-*$rT=hc)B&^%V_(){*NY2KDko1px&kR|mL$ zPg2Xr7MTM45p&U*Z+2ULUa?u%urHe9ewb&u!3k-KC;&v86iIrXnxQ87f_JHNXzOX-fUqt@fKi(rr|O3ZXV?Q<;EGweY8W?ni`Fd2yD~_1tKmH(Izy#RurJEwyv2}TlHs(C$C-Ke z@;(8og=x_JhxA0tnwn~!e;wkVH{PcftzvHbv64g=wkua^6tUB4DN?1@d-2Mv=@wec z)T&NkEe_RAw7i4MxWnip*%=6JCe=I^a(^*3+X+GaJC(@i-nRcl3CS=Ud7^+k$H3?j?p?ZE`PCAE}9e7C39q_EOO~s~r zDE@tF*CF(usKN?qk7LI5@4c67rcTK%StUSW@7y5U4v0_ZHfXH>>9KhJ-4s2M8f!Ux(qEG)xfW z#rkKuTl_bjw3~MJ9+d&|nBF4Cz;6}IP0jD}tCZa;-zZ$F#C0_ll5Y+>SoM;g2U8a? zdXvp9C^A>G2a=P8*D@85C(ZRu2{32_gFz-3Gd!nK1e{KijoO)~cKc>J+EKK9&i#rL zg^p%=$7M4!s!=Pec^q^JM>W5x+C2L$!goRYo@CK|n8ywK0<-JQF}#{fzIQmt2i@IV ziSjY-SDsLulk6?5#9thyi`4JM!X<3e8MTuW9xkE9vg+mVW`|93Y^^islgPYKcVjZY z7#3zE0`Lt#uYARah6+`EInAm~AWu}YIOVIA{i&hC!Hw8mZSeC8)~;;9I_szpBB&2p z?PNWkU(>7kruk7_idwBV_e`}ho!YHDjT9&xPU5dj6Ltm)0y0kbLB{vWqqT@O~3#EMeX5NVNZBY14Jcl4>uRDT^A?4@R!9{YGU)_G<5a{A1kpN zgX#reB5{k`tS`$*QWaI}k3x=X4(N3w>eYltl{vLxaOK`4*T8$6dlKUCEKcJqS$~^^ zzcg*P@7jQtmt-PMj}oO@%w-I%q(xQ0CqDZw@-?$6R0$fhe@AYH=3lVV0-<#oMZ@*J zS$uM@#JU3xERnb%l<-j9bfW`&0EGP)?`ZrbxAtqKCR1t^5wSK#2*SmmE^^9PhUz@3 zb-~NqM@z3n`lElUAp8U*^EL!Rx;@|}yJP(r;*!{?bDo4x>yM!XMT4JY*nk&4`VcMIZ5t0!=mfsW&J z>{XrAGpxGj1&O~6nbbV4M)q+i?p3+Em z2M9cC!Xpv_$~{KAj5!yaG(Neh3y^=oX|++AgaC!8$hfT=2t5N`^?yD*kLQT+XHMzB;y{zU2IA{Ff% zCjBVZkH=0v7t8!^iyvu5`w(~GSjwZiFkkPmwzH)C?5pT6%y^d#-K*a&f+`;+!@mPh z5>o(CgT4$!MZJEpYnPMkf%r2{pef722wem24J%-3K#c`};Sh7-Z2~B+RZVoUGFbm= zu8NJT{k(f!f)yyJ=7f~N8Zk-By_wNk?+?6JS6Za02+bK&GLTXa_Yr-?4Y=<~x2;;6 zblsVQRqHCrVC+TP@e=^YuWoz!zyw~a&q+RdriNvoWSK%?=s#lzIHcQZp@nGu7VR8b zdPcdxD@ePT+n6G_VH8N0*FDOVl_@n2 zTaY{Du3lK(Z;!RNEXZ3ZGW2>Y#%MhgmjH)ooBjU}abmaVK)Cfdvqoo@2jM#jrq%1@j2MiQ%}Vp($~P$|BBYQVzI+o3ibEOTaq{Q^!# zZqjxOh%aUuNC?lh@=CKIG z$JW8gp?K@(=y30w(r>*TCT_!!)P_9N%7SveDzA9mnZZ5tZ<36!W(a7eH>7pWL_NOhs8P;>#yM=*~WRQ$Ny)q5FYu0e~mWH3Os5AS2 zklsh4#}F1|a+{~!PtCHqRxw3Jl)XvWFPNL9iDurvR{H^{wA&vCo>&rDti7QxkmBou zp#AO-Zd}7KbR9gzajeKE;wu^@32zC=$*zh?pUzN?f_&SrN5PgO1UePfFy{N5a=7U{ z?gcL9o+dncMAmz!S}d)dQ8m0p=Pf_tjo!_&k^hqiBkjM9g?8yWEr(F6Q@%_FedD0) zYB~C&(zwGe05dPT^3NH}juE)RN1~sI)vddK_`W3BuIZ$w3RTJC#U@P;ATRaS-h^5Geju-% zbVItyg2fyy$(tT}&W{1vBV4as&s`k7-OK{wuiwtXk7Iu<%tAER5CCxNL0Q&+J3i>| zWpw$f1H&&oj%Jk-_sN_jm_XlVd+y#6m6W`5sSGl3uknQ#KxYT3kCA6c;}MKc`238g z?9)?|8aU_aIHF|IJHulD?^)`-Ak~V#uPl4(T0VvpJTD$2V|)Lnd=$}B|16EQi~@Zk zhP4>yQXd?==s=HrW&u_KGPuCE7Gv7=NC~h`sbeJEmH<1D&$hod>GP=Zy5`?FccFox z!w7F!)ONK)9-8{8)-_^s;k?VmJ>GXc`FLH)1MK!*U9jtbExHT`h}Bm=E8>{eez55w z6f#_-r)u9mhg>giszcj<>wzA-m*q=Wlp8{xuWC1T;jfB`K+y4^qhFfUK*RE8O#aDsEeW)o zI9K~8RyD>k4T50SG_sJCUUJM53hH4MTbHY;>U~Y}9aGLf-sVBZ7 z!*KNvu~7j)e{V0&@Q%dzza1|v>e%i@0^`BVn_{OXHG>|bYyX?1(O7>;I;E%xHlm_2 zZv2bvc^PU6=q6CyC0(eAQ~h||*AuGtq1%-MIKurrZp2UZxj6w*4_RsGJ@dq!E>cvp z=v>@D6Er%Yttz=mT^v_Xtp9@mDcz#}S%HDfRaZ-VQiZzau9{bQ5UjHE-QcTLvI+`% z9Q@`TEA?91o&EOQC+`!H*7}JPw!AnnAkQU3WHaSJ_9sKU2^)IFMwCx_;h^_wlRTL8 zcL}#y@7}dp2W!Vy$}u9F(kgSfd&4^f?st!~yYl-O;2+@%hko{@X>WUgFl6f4Bsl?4 z;X@lUMncR&9@Mag;d66#mZPrd102^ucK(GMY-R%@@bxPL=D0FpGV-;)=K?5HOu54BFO(tQ%GBUuKCJ@BaT&ln6f?fmlN&J) zbSwDc5oCQx@$P7+S!PM9QBBYjKtGe^(c+MaJXYR^L$&%zW_^fS=* z^Y_^CuAx3{AHwtUQe)vR@Yy?3L?C?*r*&O9a1D~WgatM!pejQ#ubpfwnQZlTzl=$a z7>+wKXbXEV{bd-$hLv_<1F)523+^h&qKOV*H;nYLqemW%O^@j+WtWx2pgx;07@l9g z&&*J{4$t&YtOEf^hog8uOX4)mL4Lj`Y9r%_vtFGapcRe0#}lYL=B0qL{Yd%sjxdWfwfr6X-s9ASu*An~h72Kxb>o_W?M<0);Q0w8^NBYy+J7~b0b_Pj zL92o@Og1wYVZ_VbKV6@$l7C()h&od zXmJLp`P%pJeoO3;&O3`|ll~#~yk4wNFZ18aFfO6^y)EYV|HJ`({4Ou5mb43a(-}|k zT4>sm!1*~t^#b=OAvh>bmrVzr`D%`dbBOzOa;+qwJV}p8ds`vh)Vk)^Kwq0@COT(H zWBhk)sb7&zi`_j3*p2}+4NK@qPPBzPX`*i5i@E-F+TI!H5Z2Tt8KQ#T%MDo#@SMM2 zH{iB)&OAWNb$(nu5@R&sL`5hZB@~W+BGIbJMv|LUskk*{cB-D)z)DB^dH1m$pKL}X za=iKlmMN-xdt$GyZ7#mHQpAopB0-GmMYA|V_)@b9M5I0~5@jR=E45r=%SRRk5MXwt z$Nsd~aedIRCdE7Y(hArl`LzdFQ)s%Ina*{;mT$C1>Q?kl8>bT`>QbV3GJ%1{v_;P` zW(cZ;R;WX=IbN-3WHAj@@fBrZ^-h+kQ)g0>-oE0N-YpX4{hO29LM2`NJ@A8_%6*Bl zP@K?0*|*YUulSh{V*4E`5ayfQtK&_*e?L?*WP8O(>rTm^cF!n&P|;&3gV>#dW50OQ z%%o>sB6gKIAT!eJwOEowsu1wo)IV+%*ZK_m@lG}5F2auB9_wD=FfFab2wym&aow`F zjL}2&`-2wPtDl1;oZN}PSfl>HS!qCClR~ck{8Msk#T2S~vB6#u)(7pU3Gd z-f*4I7P0WNl^gTbgWBgF=C-RqU7CkRJisHg93nUFw3zi=X%;kjNO5zR>M3D;UZ**l zgX!KNV9+_#t=gLmG0le$nC77_g#=R@n(D}s@a-$E^~}`eqb^39k;@;{eFODQihTgP zkneH8#5;LBeB0C$*d9(MRR^L=gS6bNh0Nk|mx{pKTxyGTS0n+LHf>cjUWyS#Pw1_P z13d$X)@F^uq$%(RS-71|8(CBQh2Q73^oI&9qJ09Rwh--y2ogkoURmxbGEZXw_b*$U zhdjk7>Ty)=MsS^J##6!pTmRQCenNdfWpz@#DZQVjWSJNE`OkQS&+#2<*lnrY3p5e! z%_x#}4z)Xzi;noqQDgfP&jTU201(qpYy`4TOgA6K!8kmZ>9g!TZ;{sI9ICg%rH>9A zdf2zAi)rP~z#3osnRK&zyTGPdfH7Kg-!^ESyP!F;!`eX&|A*|pTB5>l8i0fh(H&06 z=C7r;5t~f1mVV7TA-IUn5C6ia*U@>9mv?&vkFqsw}D_k+2fc%HaX<1`gL97X2^-+a;T9BU?#AQm1HVhT#MJtu@Em+JSJ~!|dDXQXSD9Q3&=Q zgF(aRKG47AR{6VC${>>?1=xA^F0?+!_jz^PXKP~ygVNTUq8)6C#HzmJJB08VX?IrJ z&VOAhnh@nmz(wEa|5m*w&qXA3X8kx7f5QCuE&)-fa8L)<#tA|vGWEhFZs%H?-tklp z=|+S*d@di-m|F%GperylS1Imfn`5YReyCD*d0qp!8k`doW708UAWNbJ=^t4q>{A@= zW9tf6k9v9sZYge$AR{Pse0qc&lJmFK6X!R-)*pmpKOIJro`2CgZ$ZrLpt^>hui%p> z#RI@Eq;MUHhXR)?C6K8b2Rl_a57{~u_yjiT)Hl`ok0Zlu#lH>2k~&1z8KVZs-|39H zc<43t+jY#whU{^X!mmu$%rTYtx;7I%aJKrPm5Wyq|KiVgJ*dD%4_p&w31_Q9g8l02 z87>bg*Vy#x^lO#yDDqs__Ss#naca0h!~#JBj}pSPClpu2Ok$}AD+yb59G4b8iUWRy zMjjSr;AQi0T;+UqV$>IQ{3XQ;BJ7(0_-b<-vdsWDi1k0XKrn}KCNuj|fPeo`X4AG+ z+_*t5z*a42GYj5SQ37cdDDCvZy*Xvbt&jqL6m`@|>f)c`q4UVs&L2x|BH?q{Vvoxdn;Y9r266>lK>x-wt^E@$e|W zhF=(rZ{<;BPa!4+IQ-}efx#UAvn)hvX^kgq)(?sTnw0z@i_uGLo0#3vstV?jC9B8b zv0&Hudv(m}Tx=~O)`jH(4c+s8-vEGQWV=gZrjxiT%`b)p#5Bk`$bPJ@0q-UcXu>Y3 z#Hdep$CSWZ4pGch4G@kqi6ve-}* zh?@NH+h|B$R4Gsfx&kP7B2y1;ruX~6N(p{&x^xohGVGdujl|RBEjJy{#0LOXPIyjl zg2EaPdxLB~T=}SG;7t5C@bX6LjmG%7fpd_#MAx4#-P!7ivL8277kIVt<%My)4wWg8 zN*j<9iTg1~Ku*I%(ICvULY(_OgsVLYNKy2LC z1n4JzM6C69&})iw^%!lwqU%A~hGaDVOf|E(9NBQ>toi- z6<1|B6{EdHxXkb_4?1LU9Z{b5kNFqL!I(G9OnSLibws?t1UH*LVw5X1?Y0iz0f