Skip to content

Commit 5633bb0

Browse files
imposerenblag
imposeren
authored and
blag
committed
added test project
1 parent 9194025 commit 5633bb0

27 files changed

+517
-20
lines changed

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
*.pyc
22

33
*.egg-info
4+
.tox
45
build
56
dist
67
cities/data
78

89
# WebDAV file system cache
910
.DAV/
1011

12+
13+
__pycache__

‎.travis.yml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
language: python
2+
services:
3+
- postgresql
4+
python:
5+
- "2.7"
6+
- "3.3"
7+
- "3.4"
8+
- "3.5"
9+
env:
10+
- DJANGO=1.7.11
11+
- DJANGO=1.8.14
12+
- DJANGO=1.9.9
13+
- DJANGO=1.10
14+
matrix:
15+
exclude:
16+
- python: "3.5"
17+
env: DJANGO=1.7.11
18+
- python: "3.3"
19+
env: DJANGO=1.9.9
20+
- python: "3.3"
21+
env: DJANGO=1.10
22+
install:
23+
- pip install -q tox tqdm psycopg2
24+
- pip install -q Django==$DJANGO
25+
before_script:
26+
- psql -U postgres -c "create extension postgis"
27+
- psql -c 'create database django_cities;' -U postgres
28+
- psql -c 'CREATE EXTENSION postgis;' -U postgres -d django_cities
29+
script:
30+
- PYTHONPATH=. python test_project/manage.py test test_app --noinput

‎CHANGELOG.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Change Log
2+
All notable changes to this project will (may?) be documented in this file.
3+
4+
## [Unreleased][unreleased]
5+
### Changed
6+
- added ``cities.plugin.reset_queries.Plugin`` that calls reset_queries randomly (default chance is 0.000002 per imported city or district). See CITIES_PLUGINS in Configuration example for details
7+
- It's now possible to specify several files to be downloaded and processed. See Configuration example for details.
8+
9+
## [0.4.1] - 2014-07-06
10+
11+
- Last version without changelog.

‎README.md

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,6 @@ needs to be applied:
6565
```shell
6666
./manage.py migrate cities 0001 --fake
6767
```
68-
6968
### Configuration
7069

7170
There are various optional configuration options you can set in your ```settings.py```:
@@ -111,8 +110,27 @@ CITIES_PLUGINS = [
111110

112111
# Import cities without region (default False)
113112
CITIES_IGNORE_EMPTY_REGIONS = True
113+
114+
# This setting may be specified if you use 'cities.plugin.reset_queries.Plugin'
115+
CITIES_PLUGINS_RESET_QUERIES_CHANCE = 1.0 / 1000000
114116
```
115117

118+
119+
### Writing plugins
120+
121+
You can specify import path of any class in ``CITIES_PLUGINS``. to affect import it should have one of following methods:
122+
123+
* ``FOO_pre(self, parser, item)``;
124+
* ``FOO_post(self, parser, model_instance, item)``,
125+
126+
where "FOO" should be one of: "country", "region", "subregion", "city", "district", "alt_name" or "postal_code".
127+
128+
Arguments passed to hooks:
129+
130+
* ``parser``: instance of "cities" management command;
131+
* ``item``: dict instance with data for row being processed;
132+
* ``model_instance``: instance of model that was created based on ``item``.
133+
116134
### Examples
117135

118136
This repository contains an example project which lets you browse the place hierarchy. See the ```example``` directory. Below are some small snippets to show you the kind of things that are possible:
@@ -161,6 +179,26 @@ These are apps that build on top of the ``django-cities``. Useful for essentiall
161179

162180
* [django-airports](https://github.com/bashu/django-airports) provides you with airport related model and data (from OpenFlights) that can be used in your django projects.
163181

182+
### Running tests
183+
184+
1. install postgres, postgis and libgdal-dev
185+
2. create django_cities database:
186+
187+
sudo su -l postgres
188+
# Enter your password
189+
createuser -d -s -P some_username
190+
# Enter password
191+
createdb -T template0 -E utf-8 -l en_US.UTF-8 -O multitest django_cities
192+
psql -c 'create extension postgis;' -d django_cities
193+
194+
3. Run tests:
195+
196+
POSTGRES_USER=some_username POSTGRES_PASSWORD='password from createuser step' tox
197+
198+
# if you have changed example data files then you should push your changes to github and specify commit and repo variables:
199+
TRAVIS_COMMIT=`git rev-parse HEAD` TRAVIS_REPO_SLUG='github-username/django-cities' POSTGRES_USER=some_username POSTGRES_PASSWORD='password from createuser ste' tox
200+
201+
164202
### Notes
165203

166204
Some datasets are very large (> 100 MB) and take time to download / import, and there's no progress display.

‎cities/conf.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@
66
__all__ = [
77
'city_types', 'district_types',
88
'import_opts', 'import_opts_all', 'HookException', 'settings',
9-
'NO_LONGER_EXISTENT_COUNTRY_CODES'
9+
'CONTINENT_DATA', 'NO_LONGER_EXISTENT_COUNTRY_CODES',
10+
'CITIES_IGNORE_EMPTY_REGIONS',
1011
]
1112

1213
url_bases = {

‎cities/data/UA.txt

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# truncated version of http://download.geonames.org/export/dump/UA.zip
2+
# please read http://www.geonames.org/export/zip/readme.txt for license info
3+
697796 Pechersk Raion Pechersk Raion Pechers'kij rajon,Pechersk,Pechersk Raion,Pecherskij rajon,Pecherskiy Rayon,Pechersky,Печерский район,Печерський район 50.41902 30.56602 A ADM2 UA 12 697796 131127 168 Europe/Kiev 2014-02-01
4+
703448 Kiev Kiev Chijv,Civ,Cív,Gorad Kieu,IEV,Kaenugardur,Keju,Kiebo,Kief,Kieu,Kiev,Kiev osh,Kievi,Kievo,Kiew,Kiiev,Kiiv,Kijev,Kijeva,Kijevas,Kijew,Kijow,Kijuw,Kijv,Kijów,Kijůw,Kiova,Kiovia,Kiyev,Kiyiw,Kiëf,Kjiv,Kueyiv,Kyev,Kyiiv,Kyiv,Kyjev,Kyjiv,Kyjiw,Kyèv,Kænugarður,Kíev,Kîev,Küyiv,ji fu,kheiyf,kiefu,kiv,kiva,kiyebha,kiyepeu,kyf,kyiva,kyyf,qyyb,Κίεβο,Горад Кіеў,Кейӳ,Киев,Киев ош,Київ,Кијев,Кыив,Кыйив,Кꙑѥвъ,Կիև,קייב,קיעוו,كىيېۋ,كييف,کیف,کیێڤ,کی‌یف,कीव,क्यीव,কিয়েভ,கீவ்,കീവ്,เคียฟ,ཀིབ།,ကီးယက်မြို့,კიევი,ኪየቭ,キエフ,基輔,키예프 50.45466 30.5238 P PPLC UA 12 2797553 187 Europe/Kiev 2013-07-08
5+
710734 Chernihivs’ka Oblast’ Chernihivs'ka Oblast' Chernigivs'ka Oblast',Chernigivs'ka oblast',Chernigov Oblast,Chernigovskaja oblast',Chernigovskaya Oblast',Chernigovskaya Oblast’,Chernihiv,Chernihivs'ka Oblast',Chernihivs’ka Oblast’,Черниговская область,Чернігівська Область,Чернігівська область 51.33333 32 A ADM1 UA 02 1156609 112 Europe/Kiev 2015-02-05
6+
710735 Chernihiv Chernihiv Cernigau,Cernigiv,Cernigivo,Cernigovas,Cernigău,Cernihiv,Cernihivo,Cernihiw,Cernihov,Cerniqov,Chernigiv,Chernigov,Chernihiv,Chernígov,Csernyihiv,Czernihovia,Czernihow,Czernihów,Gorad Charnigau,Tchernihiv,Tjernihiv,Tschernigow,Tschernihiw,Tsernigiv,Tsernihiv,Tsjernihiv,Txernihiv,Txerníhiv,Tšernigiv,Tšernihiv,cheleunihiu,chernigovi,cherunihiu,chrnyhyf,qie er ni ge fu,tshrnyhyf,z'rnyhyb,Çernigiv,Çerniqov,Ĉernigivo,Ĉernihivo,Černigiv,Černigovas,Černihiv,Černihiw,Černihov,Горад Чарнігаў,Чернигов,Чернігів,Чернїгів,Չեռնիգով,צ'רניהיב,تشرنيهيف,چرنیهیف,چرنیہیف,ჩერნიგოვი,チェルニーヒウ,切尔尼戈夫,체르니히우 51.50551 31.28487 P PPLA UA 02 307684 127 Europe/Kiev 2013-05-12
7+
690791 Ukraine Ukraine 'Iukuleini,An Ucrain,An Úcráin,Ikerene,Ikrɛni,Lukrayaen,Lukrayän,Oekraine,Oekraïne,Okraina,Orileede Ukarini,Orílẹ́ède Ukarini,Oucrinne,Oukrania,Owkraina,U-crai-na,U-crai-na (Ukraine),Ucraegna,Ucraina,Ucraina - Ukraina,Ucrania,Ucrayena,Ucraína,Ucraína - Україна,Ucraïna,Ucràina,Ucrânia,Ucrægna,Ukereen,Ukraina,Ukraine,Ukraine nutome,Ukraini,Ukrainia,Ukrainian Soviet Socialist Republic,Ukrainio,Ukrainskaya Sovetskaya Sotsialisticheskaya Respublika,Ukrainë,Ukrajina,Ukrajna,Ukrajno,Ukrania,Ukranya,Ukrayina,Ukrayn,Ukrayna,Ukraîne,Ukreina,Ukren,Ukreni,Ukrêni,Ukɛrɛni,Wcrain,Wcráin,Yukaran,Yukreini,Yukurayine,awkranya,awkrayn,i'ukre'ina,i'ukrena,i-Ukraine,prathes yukhern,ukeulaina,ukrain,ukrena,ukuraina,ukuraina gong he guo,wu ke lan,yukhern,yukra'in,yukren,yukrena,ywkrayn,ywkryn,Úkraína,ʻIukuleini,Ουκρανία,Украина,Украйна,Украіна,Україна,Украјина,Ѹкраина,Ուկրաինա,אוקראינה,אוקריינע,אוקרײַנע,أوكرانيا,ئۆکرانیا,ئۇكرائىنا,اوكرانيا,اوکراين,اوکراین,یوکرائن,یوکرین,ܐܘܟܪܢܝܐ,उक्रेन,युक्रेन,यूक्रेन,ইউক্রেইন,ইউক্রেন,યૂક્રેન,ୟୁକ୍ରାଇନ୍,உக்ரைன்,యుక్రెన్,ಉಕ್ರೈನ್,ഉക്രൈന്‍,യുക്രെയിന്‍,යුක්රේනය,ประเทศยูเครน,ยูเครน,ຢູເຄຼນ,ཡུ་ཀྲན།,ယူကရိန်း,უკრაინა,ዩክሬን,ᏳᎬᎳᎢᏅ,អ៊ុយក្រែន,ウクライナ,ウクライナ共和国,乌克兰,우크라이나 49 32 A PCLI UA 00 45415596 147 2013-05-24

‎cities/data/admin2Codes.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# truncated version of http://download.geonames.org/export/dump/admin2Codes.txt
2+
# please read http://www.geonames.org/export/zip/readme.txt for license info
3+
UA.03.686734 Sokyryans’kyy Rayon Sokyryans'kyy Rayon 686734
4+
UA.07.686892 Zolochivs’kyy Rayon Zolochivs'kyy Rayon 686892
5+
UA.04.687050 Zhovtnevyy Rayon Zhovtnevyy Rayon 687050

‎cities/data/countryInfo.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# one line from http://download.geonames.org/export/dump/countryInfo.txt
2+
# please read http://www.geonames.org/export/zip/readme.txt for license info
3+
UA UKR 804 UP Ukraine Kiev 603700 45415596 EU .ua UAH Hryvnia 380 ##### ^(\d{5})$ uk,ru-UA,rom,pl,hu 690791 PL,MD,HU,SK,BY,RO,RU

‎cities/management/commands/cities.py

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -34,14 +34,22 @@
3434
from django.core.management.base import BaseCommand
3535
from django.template.defaultfilters import slugify
3636
from django.db import transaction
37+
from django.db.models import ForeignKey
3738
from django.contrib.gis.gdal.envelope import Envelope
38-
39-
from ...conf import *
40-
from ...conf import CITIES_IGNORE_EMPTY_REGIONS
41-
from ...models import *
39+
from django.contrib.gis.geos import Point
40+
41+
from ...conf import (city_types, district_types, import_opts, import_opts_all,
42+
HookException, settings, CONTINENT_DATA,
43+
NO_LONGER_EXISTENT_COUNTRY_CODES,
44+
CITIES_IGNORE_EMPTY_REGIONS)
45+
from ...models import (Continent, Country, Region, Subregion, District, City,
46+
PostalCode, AlternativeName)
4247
from ...util import geo_distance
4348

4449

50+
# Only log errors during Travis tests
51+
LOGGER_NAME = os.environ.get('TRAVIS_LOGGER_NAME', 'cities')
52+
4553
# TODO: Remove backwards compatibility once django-cities requires Django 1.7
4654
# or 1.8 LTS.
4755
_transact = (transaction.commit_on_success if django.VERSION < (1, 6) else
@@ -54,9 +62,9 @@ class Command(BaseCommand):
5462
else:
5563
app_dir = os.path.normpath(os.path.dirname(os.path.realpath(__file__)) + '/../..')
5664
data_dir = os.path.join(app_dir, 'data')
57-
logger = logging.getLogger("cities")
65+
logger = logging.getLogger(LOGGER_NAME)
5866

59-
option_list = BaseCommand.option_list + (
67+
option_list = getattr(BaseCommand, 'option_list', ()) + (
6068
make_option(
6169
'--force',
6270
action='store_true',
@@ -82,14 +90,14 @@ def handle(self, *args, **options):
8290

8391
self.force = self.options['force']
8492

85-
self.flushes = [e for e in self.options['flush'].split(',') if e]
93+
self.flushes = [e for e in self.options.get('flush', '').split(',') if e]
8694
if 'all' in self.flushes:
8795
self.flushes = import_opts_all
8896
for flush in self.flushes:
8997
func = getattr(self, "flush_" + flush)
9098
func()
9199

92-
self.imports = [e for e in self.options['import'].split(',') if e]
100+
self.imports = [e for e in self.options.get('import', '').split(',') if e]
93101
if 'all' in self.imports:
94102
self.imports = import_opts_all
95103
if self.flushes:
@@ -132,7 +140,7 @@ def download(self, filekey, key_index=None):
132140

133141
uptodate = False
134142
filepath = os.path.join(self.data_dir, filename)
135-
if web_file is not None:
143+
if web_file is not None and 'last-modified' in web_file.headers:
136144
web_file_time = time.strptime(web_file.headers['last-modified'], '%a, %d %b %Y %H:%M:%S %Z')
137145
web_file_size = int(web_file.headers['content-length'])
138146
if os.path.exists(filepath):
@@ -141,9 +149,6 @@ def download(self, filekey, key_index=None):
141149
if file_time >= web_file_time and file_size == web_file_size:
142150
self.logger.info("File up-to-date: " + filename)
143151
uptodate = True
144-
else:
145-
self.logger.warning("Assuming file is up-to-date")
146-
uptodate = True
147152

148153
if not uptodate and web_file is not None:
149154
self.logger.info("Downloading: " + filename)

‎cities/migrations/0002_continent_models_and_foreign_keys.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -72,17 +72,17 @@ class Migration(migrations.Migration):
7272
name='alt_names',
7373
field=models.ManyToManyField(related_name='cities_continents', to='cities.AlternativeName'),
7474
),
75+
migrations.RenameField(
76+
model_name='country',
77+
old_name='continent',
78+
new_name='continent_code',
79+
),
7580
migrations.AddField(
7681
model_name='country',
7782
name='continent',
7883
field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.CASCADE, related_name='countries', to='cities.Continent'),
7984
),
8085
migrations.RunPython(add_continents, rm_continents),
81-
migrations.RenameField(
82-
model_name='country',
83-
old_name='continent',
84-
new_name='continent_code',
85-
),
8686
migrations.RunPython(add_continent_fks, rm_continent_fks),
8787
migrations.RemoveField(
8888
model_name='country',

0 commit comments

Comments
 (0)