Igor Moiseev Applied mathematician, Web Developer

An opinionated take-away from JSConf 2017, Verona

This is my opinionated take-away from JSConf 2017, Verona. Here bellow I’ll collect some of the most interesting slides, videos and tweets from JSDay. But at first please meet the awesome speakers

Uber with Dustin Whittle

Dustin Whittle had shown us the whole story behind the Uber platform

Lessons learned building at scale with JS at @UberEng. “Enables you to move fast, but allows for sloppy code.”

  • Latency is too high for ulta performant backed systems (99th percentile for max latency)
  • Early on it made it quick to iterate, but as the size of the team scaled the developer velocity started to slow down
    • Microservices enforce a tight interface so having static typing enables large teams to catches issues earlier. It really has an impact with 100+ devs.
  • Quick to learn, but easy to write poor quality code
    • Enables you to move fast, but allows for sloopy code

Uber uses universal javascript with React and Express.

And here is the full slides

Surviving guide with Ceph and Proxmox

After you’ve followed the familiar Proxmox Ceph Server manual youl’ll have the brand new Ceph clutser up and running, here I’m collecting the basic tasks and command you’ll need to manage Ceph cluster.

Ceph structure info

Disk structure

Ceph automatically configure and creates the block device in /dev/rbd/<pool-name>/

root@node1:~# ls -la /dev/rbd/<pool-name>/
total 0
drwxr-xr-x 2 root root 560 mar 26 15:05 .
drwxr-xr-x 3 root root  60 mar 23 13:29 ..
lrwxrwxrwx 1 root root  11 mar 26 12:39 vm-101-disk-1 -> ../../rbd0
lrwxrwxrwx 1 root root  11 mar 26 13:42 vm-102-disk-1 -> ../../rbd1
lrwxrwxrwx 1 root root  11 mar 23 19:34 vm-103-disk-1 -> ../../rbd2
lrwxrwxrwx 1 root root  11 mar 24 08:31 vm-103-disk-2 -> ../../rbd3

It is a normal block device that can be mounted in a regular way

root@node1:~# fdisk -l /dev/rbd0

Disk /dev/rbd0: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
Disklabel type: dos
Disk identifier: 0x53332713

Device      Boot    Start      End  Sectors   Size Id Type
/dev/rbd0p1 *          63 41110334 41110272  19,6G 83 Linux
/dev/rbd0p2      41110335 41929649   819315 400,1M  5 Extended
/dev/rbd0p5      41110398 41913584   803187 392,2M 82 Linux swap

so

root@node1:~# mount /dev/rbd0p1 /media
root@node1:~# df -h
Filesystem                          Size  Used Avail Use% Mounted on
/dev/rbd0p1                          20G   17G  1,4G  93% /media
root@node1:~# ls -la /media/
total 2,1G
drwxr-xr-x  24 root root 4,0K mar 23 13:43 .
drwxr-xr-x  22 root root 4,0K mar  9 18:06 ..
drwxr-xr-x   2 root root 4,0K apr 10  2012 bin
drwxr-xr-x   3 root root 4,0K ago  8  2012 boot
lrwxrwxrwx   1 root root   11 apr  9  2010 cdrom -> media/cdrom
drwxr-xr-x   2 root root 4,0K set  3  2012 dev
drwxr-xr-x 109 root root  12K mar 23 13:43 etc
drwxr-xr-x  11 root root 4,0K mag 21  2013 home
drwxr-xr-x   2 root root 4,0K ott 26  2009 initrd
...

Config files structure

Benchmarking

BLOCK WRITE 4KB

root@node1:~# /usr/bin/rados -p rbd bench -b 4096 300 write -t 32 \
  --no-cleanup -m 10.0.4.1:6789 -n client.admin --keyring \
  /etc/pve/priv/ceph.client.admin.keyring --auth_supported cephx df

Total time run:         300.015640
Total writes made:      614977
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     8.0071
Stddev Bandwidth:       3.35268
Max bandwidth (MB/sec): 11.9766
Min bandwidth (MB/sec): 0.03125
Average IOPS:           2049
Stddev IOPS:            858
Max IOPS:               3066
Min IOPS:               8
Average Latency(s):     0.0156087
Stddev Latency(s):      0.0442384
Max latency(s):         1.51347
Min latency(s):         0.00396259

Convert your sitelinks to https with `sed`

I’ve wrote before the small post on how to find and substitute the string in all files with sed command in GNU/Linux. Now I’d like to show the real use case.

You’ve bought a new SSL certificate and configured your web server. After you fire it, you can make an unfortunate discover when you open the browser on your brand new https://example.com, that it says “Some parts of this page are not secure”.

So what is insecure? Simply you may have used the insecure contents like images or extenal JS libraries loaded from CDN.

To help with this, Linux has a small and powerful command to find and substitute the old_phrase with the new_phrase in all files and directories recursively - it is sed command on GNU/Linux

find . -type f -print0 | xargs -0 sed -i 's/old_phrase/new_phrase/g'

Attention! The previous command finds files also in the hidden folders and if you’re working with Subversion or GIT you’d like to skip them. The following keys -not -path '*/\.*' makes the trick

find . -not -path '*/\.*' -type f -print0 | xargs -0 sed -i 's/http:\/\//https:\/\//g'

After this command your links like

<img src="http://example.com/dot.png">

will be converted to

<img src="https://example.com/dot.png">

Happy coding!

Find and substitute the string in all files with `sed` command on GNU/Linux

Small and powerful command to find and substitute the old_phrase with the new_phrase in all files and directories recursively with sed command on GNU/Linux

find . -type f -print0 | xargs -0 sed -i 's/old_phrase/new_phrase/g'

Attention! The previous command finds files also in the hidden folders and if you’re working with Subversion or GIT you’d like to skip them. The following keys -not -path '*/\.*' makes the trick

find . -not -path '*/\.*' -type f -print0 | xargs -0 sed -i 's/old_phrase/new_phrase/g'

Oracle fixed length CHAR field in Django

When you’re developing the Django app based on the legacy Oracle database you’ll find this type of model

from django.db import models

class Product(models.Model):
    type_product = models.CharField(max_length=3)
    code_product = models.CharField(max_length=15)
    price = models.DecimalField(max_digits=8, decimal_places=2)

    class Meta:
        unique_together = (('type_product', 'code_product'),)

where the primary key is made by the tuple ('type_product', 'code_product'). The field code_product usually is filled by some characters and the rest is padded with spaces to respect the Oracle’s CHAR(15) type. You may wish to trim those spaces in your REST web-service, but still wish to filter lists and to join on foreign keys.

Here is my hack of this problem, the CharFieldPadding class register the new character field in Django that call handy ljust function to pad the string with spaces and respect the max_length parameter.

from django.db import models

class CharFieldPadding(models.CharField):
    def __init__(self, max_length, *args, **kwargs):
        kwargs['max_length'] = max_length

        super(CharFieldPadding, self).__init__(*args, **kwargs)

    def get_prep_value(self, value):
        return value.ljust(self.max_length, ' ')


class Product(models.Model):
    type_product = models.CharField(max_length=3)
    code_product = CharFieldPadding(max_length=15)
    price = models.DecimalField(max_digits=8, decimal_places=2)

    class Meta:
        unique_together = (('type_product', 'code_product'),)

Everything works smoothly in Django Rest Framework and you have nice looking URLs.

Happy coding!